Generative Fractional Diffusion Models
Authors: Gabriel Nobis
Maximilian Springenberg
Marco Aversa
Michael Detzel
Rembert Daems
Roderick Murray-Smith
Shinichi Nakajima
Sebastian Lapuschkin
Stefano Ermon
Tolga Birdal
Manfred Opper
Christoph Knochenhauer
Luis Oala
Wojciech Samek
Venue: NeurIPS 2024 poster (2024)
Keywords: diffusion models
fractional brownian motion
fractional noise
generative modeling
Primary area: diffusion_based_models
TL;DR: We generalize the continuous time framework for score-based generative models from an underlying Brownian motion to a Markov-approximate fractional Brownian motion.

Abstract: We introduce the first continuous-time score-based generative model that leverages fractional diffusion processes for its underlying dynamics. Although diffusion models have excelled at capturing data distributions, they still suffer from various limitations such as slow convergence, mode-collapse on imbalanced data, and lack of diversity. These issues are partially linked to the use of light-tailed Brownian motion (BM) with independent increments. In this paper, we replace BM with an approximation of its non-Markovian counterpart, fractional Brownian motion (fBM), characterized by correlated increments and Hurst index $H \in (0,1)$, where $H=0.5$ recovers the classical BM. To ensure tractable inference and learning, we employ a recently popularized Markov approximation of fBM (MA-fBM) and derive its reverse-time model, resulting in *generative fractional diffusion models* (GFDM). We characterize the forward dynamics using a continuous reparameterization trick and propose *augmented score matching* to efficiently learn the score function, which is partly known in closed form, at minimal added cost. The ability to drive our diffusion model via MA-fBM offers flexibility and control. $H \leq 0.5$ enters the regime of *rough paths* whereas $H>0.5$ regularizes diffusion paths and invokes long-term memory. The Markov approximation allows added control by varying the number of Markov processes linearly combined to approximate fBM. Our evaluations on real image datasets demonstrate that GFDM achieves greater pixel-wise diversity and enhanced image quality, as indicated by a lower FID, offering a promising alternative to traditional diffusion models

Generative Fractional Diffusion Models
Gabriel Nobis Fraunhofer HHI
Maximilian Springenberg Fraunhofer HHI
Marco Aversa Dotphoton
Michael Detzel Fraunhofer HHI
Rembert Daems Ghent University FlandersMake-MIRR
Roderick Murray-Smith University of Glasgow
Shinichi Nakajima BIFOLD, TU Berlin
RIKEN AIP
Sebastian Lapuschkin Fraunhofer HHI
Stefano Ermon Stanford University
Tolga Birdal Imperial College London
Manfred Opper
Christoph Knochenhauer
Luis Oala Dotphoton
Wojeiech Samek Fraunhofer HHI TU Berlin BIFOLD
TU Berlin
Technical University of Munich
University of Potsdam University of Birmingham
Abstract
We introduce the first continuous-time score-hased generative model that leverages fractional diffusion processes for its underlying dynamics. Although diffusion models have excelled at capturing data distributions, they still suffer from various limitations such as slow convergence, mode-collapse on imbalanced data, and lack of diversity. These issues are partially linked to the use of light-tailed Brownian motion (BM) with independent increments. this paper, we replace BM with approximation of its non-Markovian counterpart, fractional Brownian motion (fBM), characterized by correlated increments and Hurst index H E (0, 1), where H 0.5 recovers the classical BM. To ensure tractable inference and learning, we employ a recently popularized Markov approximation of fBM (MA-IBM) and derive its reverse-time model, resulting in generative fractional diffusion models (GFDM). We characterize the forward dynamics using a continuous reparameteri- zation trick and propose augmented score matching efficiently learn the function, which is partly known in closed form, at minimal added cost. The ability drive our diffusion model via MA-fBM offers flexibility and control. H 0.5 enters the regime rough paths whereas H > 0.5 regularizes diffusion paths and invokes long-term memory. The Markov approximation allows added control by varying the number of Markov processes linearly combined to approximate fBM. Our evaluations on real image datasets demonstrate that GFDM achieves greater pixel-wise diversity and enhanced image quality, as indicated by a lower FID, offering promising alternative traditional diffusion models!.
Introduction
Recent years have witnessed remarkable leap in generative diffusion models [1, 2, 3], celebrated for their ability to accurately learn data distributions and generate high-fidelity samples. These models have made significant impact across a wide spectrum of application domains, including the generation of complex molecular structures [4] for material [5] or drug discovery [6], realistic
"oorreppnndnng author gabriel.nobis fri.fraunhofrrdee
'The implementation of our framework available
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

dX,- jlt)x,dt o())B
dY, -Y,dt 1dB,
Xo
XT Yo
Yr
2 at
Xo dz.- F( G()G V. log (Z:) dt G()BB,
Xr Yo dY. uY-=1-7X 1k Vy log (Y (]de + 1dB;
YT
Knowa guiding function
Figure 1: Each data dimension transitions to known prior distribution through a forward process that approximates a fractional diffusion process. The Hurst index H on the LHS interpolates between the roughness of a Brownian driven SDE and the underlying integration in PF ODEs. The driving noise process is linear combination of the correlated processes on the RHS, all driven by the same Brownian motion. The score function of these augmenting processes is available in closed form and serves as guidance for the unknown score function.
audio samples [7, 8). 3D objects [9, 10] or textures [11], medical images [12], aerospace applications [13], and DNA sequence design [14, 15]. Despite these successes, modern score-based generative models formulated in continuous- time [16] face limitations due to their reliance
simplistic driving noise, the Brownian motion (BM) [17, 18, 19). As a light-tailed process, using BM can results in slow convergence rates and susceptibility to mode-collapse, especially with imbalanced data [20]. Additionally, its purely Markovian nature may also make it hard to capture the full complexity and richness of real-world data. All these attracted number of attempts for involving different noise types [20, 21J. In this paper, we propose leveraging fractional noises, particularly the renowned non-Markovian fractional BM (IBM) [22. 23] to drive diffusion models. fBM extends BM to stationary increments with a more complex dependence structure, i.e., long-range dependence vs. roughness/regulariyy controlled by Hurst index, measure of "mild" or "wild" randomness [24]. This all comes at the expense of computational challenges and intractability of inference, mostly stemming from its non-Markovian nature. Moreover, deriving reverse- time model poses theoretical challenges, as fBM is not only non-Markovian but also not semimartingale [25]. To overcome these limitations, we leverage recent works in Markov approximations of fBM (MA-IBM) [26. 27] and establish framework for training continuous-time score-based generative models using an approximate fractional diffusion process, as well as generating samples from the corresponding tractable reverse process. Notably, our method maintains the same number of score model evaluations during both training and data generation, with only a minimal increase in computational load. Our contributions are:
We derive the time-reversal forward dynamics driven by a Markovian approximation of fractional Brownian motion in way that the dimensionality of the unknown part of the score function matches that of the data.
We derive an explicit formulae for the marginals of the conditional forward process via a continuous reparameterization trick.
We introduce a novel augmented score matching loss for learning the score function in our generative fractional diffusion model, which can be minimized by score model of data-dimension.
Our experimental evaluation validates our contributions, demonstrating the gains of correlated-noiss with long-term memory, approximated by combination of number of Markov processes, where the amount of processes further control the diverstiy.
Differentiation from existing work. Yoon al. [20] generalizes score-based generative models from an underlying BM to a driving Lévy process, a stochastic process with independent and stationary increments. A driving noise with correlated increments not included in the framework of Yoon et al. [20]. Conceptually, every Lévy process is a semimartingale [28]. Since fBM is not a Lévy process, is not included in the framework of Yoon et al. [20]. The closest work to ours Tong et al. [29] constructing a neural-SDE based on correlated noise and using the neural SDE as forward process of a score-based generative model. Our framework with exact reverse-time model is based on the integral representation of fBM derived in Harms and Stefanovits [26] and the optimal approximation coefficients of Daems et al. [27], while the fractional noise [29] is sparsely approximated by a linear combination of independent standard normal random variables without exact reverse-time

---

3 and only compatible with the Maler-Maruyama sample schema [30] while our framework up numerical stability applicable for any H E (0, 1) and compatible with any suitable SDE or ODE solver. To the best of our knowledge, we are the first to build a framework for continuous-time score-based generative models that includes driving noise processes converging to non-Markovian processes with infinite quadratic variation.
model. Moreover, the framework of Tong et al. [29] is limited to H >
2 Background
Modeling the distribution transforming process of score-based generative model through stochastic differential equations (SDEs) [16] offers a unifying framework to generate data from an unknown probability distribution. Instead of injecting finite number of fixed noise scales via Markov chain, infinitely many noise scales tailored to the continuous dynamics of the Markov process X = (X1) (ejor] are utilized during the distribution transformation, offering considerable practical advantages over discrete time diffusion models [16]. The forward dynamics, transitioning from data sample Xạ Po to a tractable noise sample XT ~ Pr are specified by a continuous drift function f and a continuous diffusion coefficient These dynamics define a diffusion process that solves the
SDE
(1) driven by multivariate BM B. To sample data from noise, a time model is needed that defines the backward transformation from the tractable noise distribution to the data distribution. Whenever X = (X, (e is stochastic process and g is function on [0, T), we write X, X for the reverse- time model and g()) g( t) for the reverse-time function. The marginal density of the stochastic process X at time t is denoted by Pt throughout this work?. Remarkably, an exact reverse-time model to the forward model in eq. (1) is given by the backward dynamics [31, 32, 33]
dX, f(x,, t)dt (())dB, Xo Po
(2) where the only unknown is the score function Vx log Pt. inheriting the intractability from the unknown initial distribution Po- In addition to the stochastic dynamics, the reverse-time model provides deterministic backward dynamics via an ordinary differential equation (ODE) by the so called probability flow ODE (PF ODE) [16]
dx, tt) t) g" (t) pr [X)] dt + g(t)dB,, X. = XT Pr.
dx.- = [(xe, t)
1 2 log lo dt, 2
XT ~ pT.
Stochasticity is only injected into the system through the random initialization Xr ~ Pr, implying a deterministic and bijective map from noise to data [16]. Conditioning the forward process on data sample Xo Po results for linear f(., E) in a tractable Gaussian forward process with conditional score function Vx log por (x xo) in closed form. To approximate the exact reverse-time model, this tractable score function is used to train a time-dependent score model So via score matching [34, 35). Upon training, any solver for SDES or ODEs can be utilized to generate data from noise by simulating the stochastic or deterministic backward dynamics of the reverse-time model with Sg Vx log p.
Simulation error of the reverse-time model. The two main sources of error when simulating the reverse-time model are the approximation error due So only approximating Vx Volog p. and the discretization error, which arises from transitioning from continuous- Simulating the PF ODE with the Euler method over N N equidistant time steps results in a global error of order N-1 [36]. In contrast, the expected global error for simulating the SDE using the method is of a lower order - i, indicating larger error for the same number of steps [30. 36]. From this perspective it is reasonable that sampling from the PF ODE requires fewer steps. Yet, the source of qualitative differences between sampling from the ODE and the SDE [16] remains unclear.
to discrete steps.
A pathwise perspective on sampling. The roughness of a path can be measured by its Hölder exponent 0 < < [37]. For example, BM as the integrator in the backward dynamics eq. (2) has 8-Hölder continuous paths for any 0 < 8 < 1. whereas the integrator / + L of the PF ODE eq. (3) can be regarded as a Hölder continuous path with exponent = 1. Therefore, from a pathwise perspective, we move away from rough path when we sample using the PF ODE. An unexplored topic in score-based generative models is the interpolation between the SDE and the PF ODE in terms
"See Appendix for the notational conventions of this work.
3

---

of the Hölder exponent. It remains to be examined whether there is, to some extent, an optimal degree of Hölder continuity in between, or if an even rougher path with 8 < could yield an advantageous data generator.
The process that naturally arises from this line of thought is fBM [22, 23] with Hurst index H E (0, 1), where almost all paths are Hölder continuous for any exponent ò < H. controlled by H. In terms of roughness, the Hurst index interpolates between the paths of Brownian driven SDEs and those of the underlying integration in PF ODEs, while also offering the potential for even rougher paths. Motivated by these observations, we define novel score-based generative model with underlying dynamics that approximate a fractional diffusion process.
3 Fractional driving noise
Before describing the challenges in defining a score-based generative model with control over the roughness of the distribution transforming path, we introduce fBM. The literature distinguishes between "Type I" fBM and "Type II" fBM [38] having stationary and non-stationary increments, respectively. The type II [BM, also called Riemann-Liouville fBM, possesses smaller deviations from its mean, potentially an advantageous property for a driving noise of score-based generative model, since large deviations of the sampling process to the data mean can lead to sample artifacts [39]. Here and in the experiments we focus on type II fBM. However, our theoretical framework generalizes to both types as detailed in Appendix A. The empirical study of a score-based generative model approximating a fractional diffusion process driven by type I fBM is dedicated to future work. We begin with the definition of Deemann-Liouvile IBM [22]. generalization of BM permitting correlated increments.
Definition 3.1 (Type II Fractional Brownian Motion [22]). Let B = (B:)120 be standard Brownian Motion (BM) and the Gamma Sannnaffnnee function. The centered Gaussian process
( (t 8) -) t t20,
B
I(H +2
uniquely characterized in law by its covariances
1
min(t,,s)
E [BB.B..1 = T2(H
([I-u)(s - u)) - ddu, & E (0,
(5)
is called type II fractional Brownian motion (fBM) with Hurst index H 10.
BM being the unique continuous and centered Gaussian process with covariance minft,s) s} is recov- ered for H = 0.5, since r(1)= = 1. In comparison to the purely Brownian setting with independent increments (diffusion), the path of Rd becomes more smooth for H 0.5 due to positively correlated increments (super-diffusion) and more rough for H < 0.5 due to negatively correlated increments (sub-diffusion). These three regimes are reflected in the Hölder exponent of 5 < H for almost all paths.
Generalization challenges. The most challenging part in defining score-based generative model driven by fBM is the derivation of reverse-time model. Due to its covariance structure, fBM is not a Markov process [40] and the shift in the roughness of the sample path leads changes in its quadratic variation: from t in the purely Brownian (diffusion) regime to zero in the smooth regime, and to infinite in the rough regime [30]. For that reason fBM is neither a Markov process nor a semimartingale [25] for all H * 0.5. Hence, we cannot make use of the Markov property or the Kolmogorov equations (Fokker-Planck) that are used to derive the reverse-time model of Brownian driven SDES [31, 32, 33]. See Appendix H for a more illustrative view of the problem. The existence of a reverse-time model can be proven in the smooth regime of fBM [41]. However, due to the absence of an explicit score function in Darses and Saussereau [41] it does not provide a sufficient structure to train a basre-based generative model.
To overcome this difficulty we follow [26, 27] and define the driving noise of our generative model by a linear combination of Markovian semimartingales. The approximation is based on the exact dnimite-dimensionn Markovian representation of fBM given in Theorem A.2.
Definition 3.2 (Markov approximation of fBM [26, 27]). Choose K E NN Ornstein-Uhlenbeck (OU) processes
y* = 1- nutt 1: EEN, t2 0,
(6)
30
4

---

with speeds of mean reversion "K and dynamics dy Yx + dB,, Given a Hurst index H E (0, 1) and geometrically spaced grid * = pk-n with r > 1 and n K+1 we call the process
B : HE (0,1), 120,
Markov-approxinate fractional Brownian motion (MA-BBM) with approximation coefficients Wi...., WK E IR and denote by B (BH, y.... BB) the corresponding D-dimensional process where B and Bf are independent for + inheriting independence from the underlying standard BMs B, and By.
Our framework is conceptually independent of the specific choice of spatial grid and approximation coefficients. To achieve strong convergence rates with a high polynomial order in K for H < 0.5 in the driving noise to fBM, one may follow the approach outlined in Harms [42]. Consequently, our framework includes driving noise processes that converge to non-Markovian processes with infinite quadratic variation. For computational efficiency, we instead follow the approach of Daems et al. [27] to choose the (P) optimal approximation coefficients for a given K, achieving empirically good results in approximating fBM, even with small number of OU processes.
Proposition 3.3 (Optimal Approximation Coefficients [27]). The optimal approximation coefficients
(W1,, ..... wk) E IR for given Hurst index H E (0, 1), terminal time TT >> 0 and fixed geometrically spaced grid to minimize the eerror
10² E B
T
-B"-Bj -B,1 2
E(w)
d
(8)
are given by the closed-form expression Aw b with
(7 1r
As, = 2T +
T
H + H+ %
3 Tx
41%
b
H 2. nT) H+
P
H +
(9)
Y/s 1177 73
7k
and where P(2,x)= ,tay So e dt is the regularized lower incomplete gamma function.
MA-IBM serves as the driving noise of our generative model, replacing BM in the distribution transforming process solving eq. (1), approximating a fractional diffusion process. See Figure 1 for an illustration of the underlying processes.
A score-based generative model based on fractional noise
In this section, define continuous-time time soore-based generative model driven by MA-fBM. A detailed treatment of the theory can be found in Appendix A. We begin with the forward dynamics, transitioning data to noise.
Definition 4.1 (Forward process). Let B be D-dimensional MA-fBM with Hurst index H E (0, 1). For continuous functions μs : [0. T] - T and X (X))eE(OTT] of a generative fractional diffusion model (GFDM) by
[0, - we define the forward process
dX; dt + g(t)d X, = Xo Po. t <0,
(10)
where po is the unknown data distribution from which we aim sample from.
Considering both the forward process as well as the OU processes defining the driving noise BB we have for every data dimension an augmented vector of correlated processes (X, y ..... YK), driven by the same BM, approximating the time-correlated behavior of one-dimensional fractional diffusion process [27]. We denote the stacked process of the /) augmented vectors as Z (Z.) and refer the resulting D(K + 1) -imeensiioa process as the augmented forward forwaad process. Rewriting
the dynamics of the forward process we observe that the augmented forward process Z solves a linear SDE. Hence, Z Z|x the augmented forward process conditioned on data sample X, Po- is linear transformation of BM. Thus Z|xo is Gaussian process and so is X|xo [43]. For each dimension 1 S d < D, we have system of K + trajectories that transform Xo.t according the augmented forward process with D 1, following the dynamics
dz, F(t) + G(t)dB,,
(1I)
5

---

where all K + 1 processes are driven by the same one- mne-dimnnsional BM B with matrix valued functions F and G defined in Appendix A.2. To efficiently sample for every (0. 7] from the conditional augmented forward distribution during training, we characterize its marginal statistics.
Derivation of marginal statisties. The marginal mean [x] X, exp( u(s)ds) of the conditional forward process is unaffected by changing the driving noise to MA-fBM, and the mean of the augmenting OU processes is zero. See Appendix A.2 for a detailed derivation of the marginal statistics of the augmenting processes. The missing components in the marginal covariance matrix E, of the conditional augmented forward process Z Z|xo are the marginal variance of the forward process and the marginal correlation between the conditional forward process and the augmenting processes. We derive by reparameleriziation an explicit formula for the marginal variance of the conditional forward process. This generalizes the formula for the perturbation kernel Pou (x(x²) N(x; c(t)xu, e (t)o² (t) given in Karras et al. [44] to a driving MA-fBM and is reminiscent of the reparameterization trick used in discrete time.
Proposition 4.2 (Continuous Reparameterization Trick). The forward process X of GFDM condi- tioned on Xo E RD admits the continuous reparameterization
X, =(t) (x00)xx+) xx+
L a(t. alt.s)dB.) ((c(t))x,,&(tt)((
(12)
with c(t) = exp (Io ((s)ds) and o2 = -So (t, s)ds where a is given by
alt, 8)
Σ Wk
k
g(s)
Th S: fa(u,
fu(u.s)du s)du). fu(u,s)
g(u) clu) e 1a (u-s)
(13)
c(s)
Sketch of Proof. Reparameterization of the forward dynamics in eq. (10) and the Stochastic Fubini Theorem yields the Gaussian process X, = c(t)(xo + fo att, s)dB.) with variance V [X,]
(1) So (lds by Itô isometry. See Theorem A.3 for the full proof.
By the above defimition of a, we retrieve the perturbation kernel of the purely Brownian setting given in Karras al. [44, Equation 12] for K = 1. 71 0 and W1 1. When, depending on the choice of forward dynamics, fod (x(1, s)ds is not accessible in closed form, E, can described by an ODE and solved numerically as described in Appendix B. Thus our method admits any choice of forward dynamics in terms of μ and g.
Explicit fractional forward dynamics. Although our framework is not bound to any specific dynamics, this work's 's empirical evaluation focuses on Fractional Variance Exploding (FVE) dynamics given by
dx.- = G minn
mmaz
2log
maa aß
E [0,
(14)
T Jmin with (amin: mmaxx) (0.01, 50) and Fractional Variance Preserving (FVP) dynamics given by
mmin
dX,- - 3(1xx, VB(t)BB, E [0,7]
(15)
with 3()) = 3(t) = Bmin + (Bmax- - .min) and (Bmin, 3mar) = (0.1, [16]. Leveraging the continuous reparameterization trick we derive in Appendix the conditional marginal covariance matrix of FVE in closed form. To the best of our knowledge, the integral in eq. (13), needed to compute a in the setting of FVP dynamics, is not accessible in closed form. Therefore, we use numerical ODE solver to estimate this quantity for FVP dynamics. See Appendix B for details on the computation of the marginal variances and an illustration of the resulting variance schedules.
The reverse-time model. We observe that the augmented forward dynamics of GFDM are already encompassed in the general framework presented in Song et al. [16, Appendix A], although they differ from the Variance Exploding (VE), Variance Preserving (VP), and sub-VP dynamics discussed therein. To simplify notation, we use P: here to denote the marginal density of both Z, and X,. The specific density referred to will be clear from the context. By the significant results of [31, 32, 33], the reverse- teerse-time model of GFDM is given by the backward dynamics
dz, Z(t) (t)) V log Dy [Z)) dt G())dB. t [0,7].
(16)

---

However, a direct application of [16] would require to train a score model with input and output dimension of D(K + 1). By proposing augmented score matching below, we show that learning a score model with input and output dimension D) is sufficient, enabling the use of the same highly curated model architecture as in traditional diffusion models to approximate the score function.
Augmented score matching. We condition the score function Vx log Pt on a data sample X ~ Po and additionally on the states of the stacked vector Y [K] To train our time-dependent score model so we propose the augmented score matching loss
(Y):.., Y of augmenting processes.
( Be { F vK()) (X, Xo) |I
Σ
Th
Vx Pos YY, Xo)I
(17)
The weights 1h ,..., no arise from conditioning [x/x; Y!K and the time points t uniformly sampled from 00, 7). We show in the following that the optimal Sq w.n.. the augmented score matching loss is the o.ptott approximation of the score function of our reverse-time model. Proposition 4.3 (Optimal Score Model), Assume that 5e is optimal w.r.t. the augmented score matching loss L. The score model
So (Z., 1) == so (X - th ΣnEYt,,, - 4i so E Stxt,.)..... - (X.
Evtt.
(18)
k
yields the optimal (2(P) approximation of Vz log (Zz) via
So (Z, t) + V2 log ge ( [kly
~ Vz lo (Z.).
(19)
Sketch of Proof. Using the relation Vx PogPo- Poe = v/ y Por por and the independence of X, and y!!] yields the claim. See Appendix A.3 for the full proof.
In addition to the result that score model of data dimension D minimizes the proposed augmented score matching loss, Proposition 4.3 also implies that GFDM requires the same number of score model evaluations during sampling from the reverse-time model as traditional diffusion models. This because, for a given time point i, we only need to evaluate sp(., () once at X, - E. of Y to compute So (Z., 1) according to eq. (18), and Sp is all that is required to approximate the reverse-time dynamics described below. We provide a thorough quantitative evaluation of compute time in seconds for GFDM in Appendix F, validating the theoretical reasoning in this section that GFDM incur only minimal additional computational cost.
Sampling from reverse-time model. Once trained score model So via augmented score matching, we simulate the reverse-tinns time model backward in time and sample from the reverse-time model via the SDE
dz.- { F(t - ((t) (o(( + V, (x))) de G(t)dB, G((DB) ee [0,T] (20)
or the corresponding augmented PF ODE [16]
dz = F(t)z - 1 (()) Bo(zz + V. V. 7F
dt, tE [0,
(21)
where we initialize in both cases the reverse dynamics with the centered (non-isotropic) Gaussian Zo with covariance matrix Er. To traverse backward from noise to data, we may deploy any suitable SDE or ODE solver. In both cases, for each data dimension, we have K+1 1 trajectories that transform the Gaussian initialization into an approximate sample of the data distribution. The PF ODE enables in addition negative log-likelihoods (NLLs) estimation of test data under the learned density [16]. See Appendix for the computation details of NLLs.
Remark 4.4. We showed in this section that it suffices to approximate D-dimensional score to reverse the D(K + d)ddimensio MA-fBM driven SDE with unknown starting distribution. Since this holds for any fixed K E N an interesting task is to examine the behaviour of the reverse-time model as K + co and potentially link it to the dynamics of a reverse-time time model of true JBM. To the best of our knowledge, existence of such a reverse-time model is not known for H < 0.5 and the drifi of the reverse-time model for H > 0.5 lacks sufficient structure to train a score score-based generative model [41].
7

---

Table 1: Effect of aug- menting processes on conditional image gen- FVR(H 0.5)
FID 4
NLLa Tsst
VS,1
EVP(R 0.5)
RDD
NLLATes
VS, T
VE (ntmrnll
10.82
2.73
24.20
VP (retmiide))
1.44
2.38
23.64
10.30
3.90
24.15
4.57
9.74
24.42
3.51
7.02
eration on MNIST for
11:25
1.10
24.54
1.88
5.71
24.50
FVE and FVP dynamics.
25.01
3.94
23.08
R 5
4.89
7.09
24.56
MNIST
0.0
11 = 0.7
11 0.5
If = 01
CIFARIO
= 0.9
=
=
0.5
/f =.1
TDL
15.1
BD4
15.1
nD. We1
701
V5.
deiven
driven
VE (ntramed)
10,82
(nerained)
5:30
3.01
VP (ntmined))
23.66
(retmined)
4.85
3.38
MA HEM driven
MA-IIM driven
EVP PVP H.
H. R
2,80
23.56
3.11
23.78
2.51
23.60
2.92
23.50
50
FVR
1)
4.79
5.53
0.50
5.84
4.19
3.99
4.00
3.46
1.98
24.00
2.30
21.82
2.52
20.63
2.56
23.55 R2
FVR
5.77
5.60
4.17
5.35
4.85
4.04
5.77
3.45
EVP H
0.73
24.18
2.67
21.96
8.51
20.78
4.87
20.00
EVP K
30
14.27
3.38
6.12
339
6:32
3-49
3.10
3.51
EYP
1:2
24.76
0.58
24.39
1.56
24.50
6.25
20.80 89
EVPE K
29.72
5.07
8.30
324
8.85
3.05
6.00
0.36
PVP H.
5)
2.17
25.15
1.30
24.63
4.59
24.56
9.07
20.70
EVPUL K 6)
09.06
0.61
15.01
5.20
56.54
7.30
7.38
3.11
(a) Conditional image generation on MNIST.
(b) Conditional image generation on CIFAR10.
Table 2: FID and pixel-wise diversity v of GFDM compared to the original setting of purely Brownian driven VE and VP. In bold the scores that are better than both purely Brownian driven dynamies. The overall best scores within the experiment are boxed in, indicating that the highest scores on both datasets are achieved in the super-diffusive regime for H = 0.9.
5 Experiments
We conduct experiments on MNIST and CIFARIO to evaluate the ability of GFDM to generate real images. First, we measure the quality and the pixel-wise diversity of the generated images across different numbers of augmenting processes and various Hurst indices, showing that the super-diffusive regime with H >> 0.5 yields better performance compared to the purely Brownian driven dynamics. Second, we further evaluate the best performing models in terms of class-wise image quality and class- wise distribution coverage. We measure image quality by the Frechét Inception Distance (FID) [45] and the Inception score (IS) [46], pixel-wise diversity by the pixel Vendi Score (VS,) [47] and class- wise distribution coverage by improved recall (Recall) [48]. See Appendix D for the implementation details and additional experimental results. We begin with the empirical evaluation of how the augmenting processes affect performance on MNIST.
Effect of augmentation on MNIST. To isolate the effect of the augmenting processes on MNIST while minimally adapting the driving noise distribution, we fix H = 0.5 so that the weighted sum of the augmenting processes approximates BM, rather than fBM. We observe an increase of the pixel-wise diversity VS, for both FVE and FVP dynamics, with increasing K. In Table we can observe that VS, increases from 24.20 to 24.54 for FVE dynamics and from 23.64 to 24.56 for FVP dynamics. The enhanced pixel-wise diversity on MNIST comes at the cost of reduced likelihood of test data under the learned density, indicated by a higher NLLs for more augmenting processes.
Quality results across different Hurst indices. On both, MNIST and CIFARIO, we obtain the best performance in terms of FID and VS, in the super-diffusive regime with = 0.9 and FVP dy- namies. On MNIST achieve state of the art FID of 0.72, compared to an FID of 1.44 with the purely Brownian VP dynamics (Table 2a). Comparing FVP to the best-performing purely BM driven VP dynam- ics, we observe not only an improvement quality but also an increase in pixel-wise diversity from 23.64
FyPti 0S,K-
PVPty KC 20
Purtly frownan VP
Puraly Brown V2
18
to 24.18, as measured by VS,- In Table 2b we ob- serve the same behaviour on CIFAR 10. The best performing configuration in terms of FID and pixel- wise diversity is achieved for FVP H = 0.9, K = 2) with an FID of 3.77 instead of 4.85 and an VS, of 3.60 instead of 3.28. Additionally, in Figure 7, we show the FID evolution of the super-diffussive regime for various numbers of augmenting processes, showing similar pattern that either that K = 2 or
252
750
1000
Figure Comparison of the super-diffusive regime and purely Brownian dynamics in terms of average FID over three rounds of sampling plotted across different NFEs.
8

---

Denamies
airplane
frog
ship
10.74
14.18
10.32
15.37
10.04
Recall 1
VP
0.6814
0.6186
0.6860
0.6466
0.7002
0.6730
0.6768
0.0392
0.6468
0.5082
PVP PVPER
7.7K
0.0833
0.6436
0.0870
0.6712
0-71,0
0.0844
0.0922
0.6764
0.8350
0.6308
0.0, K = 2)
0.7038
0.8614
0.7188
0.6842
0.7284
0.7096
0.7104
0.6806
0.6772
0.6852
Table 3: The class-wise image quality and class-wise distribution coverage of the super-diftusive regime FVP(H 0.9, K compared to the purely Brownian VP dynamics.
Figure 4: Visual compari- son of PF ODE samples. (LHS) Purely Brownian VP dynamics. (RHS) Super- diffusive regime FVP 0.9, =). 2).
== yields the best performance across different datasets and dynamics. Evaluating the perfor- mance with different number of sampling steps in Figure shows that the super-diffusive regime with K saturates already at 500 number of function evaluations (NFEs) on a lower level than both purely Brownian driven dynamics VP and VE. See Figure 2 in Appendix D) for the exact FID values.
Class-wise distribution coverage. We evaluate the capability to generate samples from different classes in terms of FID and class-wise distribution coverage, measured by Recall, comparing the best-performing purely Brownian driven dynamics to the super-diffusive regime with Table 3 we observe that the super-diffusive regime with K = outperforms in both FID and Recall, where = 0.7 and H = 0.9 achieve better class-wise wise FID for all but two and one class, respectively (deer and dog for H 0.7, bird for H 0.9). Additionally, the super-diffusive regime shows improved class-wise distribution coverage, as indicated by a higher Recall across classes. Overall, both H 0.7 and H 0.9 perform significantly better in terms of distribution coverage than VP dynamics, H = 0.9 being the best performing model.
2. In
Sampling with the augmented probability flow ODE. We compare the performance of sampling via the PF ODE for the best performing models from above. For MA-fBM driven dynamics, we have K + deterministic trajectories for each pixel, traversing from noise to data. As shown in Figure 3, the PF ODE asso- ciated with purely Brownian dynamics outperforms the super-diffusive regime in terms of FID, while the super- diffusive regime achieves the overall highest pixel-wise diversity of VS, 4.89 confirmed mildly perceptually in Figure 4. See Appendix for additional visualization of the generated data.
vS,
Sampled with SDE
VE (retrained) VP (retrained)
5.20
9.60
3-12
4.55
9.64
3.28
FVPCH 0.7.K FVPCH 0.9. =
4.17
9.51
3.35
3.77
9.41
3.60
Sampled with PF ODE
(retrained) VP (retrained)
9.22
5.63
9.23
3.91
FVPCH K=
12
9.73
EYP(H 0.9. K
12.26
9.55
4.89
Our experiments show that, compared to purely Brownian dynamics, the super-diffusive regime of MA-IBM yields higher image quality with fewer NFEs, improved pixel- wise diversity and better distribution coverage.
Figure 3: Quantitative performance com-
parison of SDE and PF ODE sampling.
6 Related work
Diffusion models in continuous-time. The seminal work of Song et al. [16] offers unifying framework modeling the distribution transforming process by stochastic processes in continuous- time with exact reverse-time model. Extensive research has been carried out to examine [44, 49, 50] and extend [39, 51, 52, 53, 54, 55] the continuous-time view generative models through the lens of

---

SDEs, including deterministic corruptions [56] and blurring diffusion [57]. While critic on this view question the usefulness of the theoretical superstructure [58], others extend in line with our work the theoretical framework to new types of underlying diffusion processes [59]. Conceptually similar to our work, Yoon et al. [20] generalizes the score-based generative model from an underlying Brownian motion to a driving Lévy process, thereby dropping the Gaussian assumptions on the increments. In contrast to our work, the framework of Yoon et al. [20] does not include correlated increments. Importantly, every Lévy process is semimartingale, which means that fBM is not a Lévy process.
Fractional noises in machine learning. Recently, Hayashi and Nakagawa [60] considered neural- SDEs driven by fractional noise. Yet they do not study diffusion models. The closest work to our work, Tong et al. [29] approximated the type-Il fBM with sparse Gaussian processes constructing a neural SDE forward process of a score-based generative model, without exact reverse-time dynamics. Unfortunately, they are also limited to Fuler-Maruyama solvers and to the case of H > 1/3, while our framework is up to numerical stability applicable for any H E (0. 1) and compatible with any suitable SDE or ODE solver. Daems et al. [27]. who inspired our Markov-approximar noise, includes more elaborate discussion well as variational inference framework for MA-IBM.
Rough path theory. The pathwise analysis of SDEs driven by processes with Hölder exponent less than 0.5, including fBM for H < 0.5 and BM, encompassed by rough path theory [37]. Rough path theory is applied in machine learning in several ways including (i) deriving stability bounds for the trained weights of a residual neural network [61], (ii) enabling rough control of neural ODEs [62], and (iii) modeling long time series behavior via neural rough differential equations [63, 64). In finance the famous Black-Scholes model [65] driven by BM, while more recent continuous-tims models employ fractional noise to model price processes [66, 67] or rough volatility [68, 69] to more closely mimic real-world behavior.
7 Conclusion
In this work, we propose a generalized framework of continuous-time score-based generative mod- els, introducing a novel generative model driven by MA-fBM with control over the roughness of distribution transformation paths via augmenting processes. Despite the increased dimensionality of the forward process, learning score model with the dimensionality of the data distribution, guided by the marginal known score of the augmenting processes, is sufficient. Consequently. both training and sampling efficient. Our experimental results show that the super-diffusive regime of our MA-IBM driven dynamics achieves superior performance in terms of FID and pixel-wise diversity. Additionally, the FID saturates at a lower level with fewer function evaluations compared to purely Brownian driven dynamics. The super-diffusive regime also improves class-wist distribution coverage, as measured by Recall. Based on these results, GFDM offers promising alternative to traditional diffusion models for generating data from unknown distribution.
Limitations and future work. Several practical and theoretical questions remain open. While draw conclusions from experiments conducted on MNIST and CIFARIO, generalizing the observed behavior to other datasets and data modalities may not be valid. In future work, we aim to empirically and theoretically determine the optimal degree of correlated noise, and thus the optimal Hurst index, for training and sampling across different data modalities. Beyond image data, a particularly interesting modality could be the generation of rough time series data using dynamics of the aab-diffusive regime. A theoretical open question is the limiting behavior of GFDM's 's reverse dynamics with infinitely many augmenting processes and whether this limit is connected to the reverse time model for true fBM. An intriguing extension would be to adapt the dynamics of our framework to switch between two unknown distributions. This adaptation would enable the use of MA-fBM driven dynamics in the sciences to model real-world evolution between two states of unknown distributions. This is promising direction, as the assumption of independent increments in real-world noise processes is often too strong.
Broader impact. Our contribution advances generative modeling by introducing a specific driving noise process to improve the learning of an unknown distribution. This conceptual work aims to support impactful applications of generative modeling, such as molecular structure generation, medical imaging, drug discovery, and DNA sequence design. However, we acknowledge that generative models can reflect biases in the datasets they are trained on and may pose risks, including misuse for human impersonation and the spread of fake content.
10

---

Acknowledgements
We would like to give special thanks to Thorsten Selinger for his support utilizing Fraun- hofer HHI's GPU cluster. We also thank the anonymous reviewers for their constructive feed- back, which helped improve our work. This work was supported by the Federal Ministry of Education and Research (BMBF) as grants [SyReal (011S21069B)). R.M-S. & M.A. acknowl- edge funding from the QuantTC Project funded by EPSRC Quantum Technology Programme (grant EP/MO1326X/1 EP/T00097X/1), and dotPhoton AG. R.M-S acknowledges funding from EP/R018634/1, OPPT021020//
CPY0029178/1 and Google. SN acknowledges funding from the German Federal Ministry Education and Research under the grant BIFOLD24B. RD acknowledges funding from the Flemish Government under the "Onderzoeksprogrammn Artificièle Intelligentic (AI) Vlaanderen" " programme and from Flanders Make under the SBO project CADAIVISION. TB was supported by a UKRI Future Leaders Fellowship [grant number MR/Y0185 18/1]. MO has been partially funded by Deutsche Forschungsgemeinschatt (DFG) Project - ID 318763901 SFB1294. WS acknowledges financial support by the German Research Foundation (DFG) Research Unit
KI-FOR 5363 (project ID: 459422098).
References
[1] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2256-2265. PMLR, 2015.
[2] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pages 6840-6851. Curran Associates, Inc., 2020.
[3] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
[4] Emiel Hoogeboom, Víctor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3D. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 086778887, PMLR, 2022.
[5] Matteo Manica, Jannis Born, Joris Cadow, Dimitrios Christofidellis, Ashish Dave, Dean Clarke, Yves Gaetan Nana Teukam, Giorgio Giannone, Samuel C Hoffman, Matthew Buchan, et al. Ac- celerating material design with the generative toolkit for scientific discovery. npj Computational Materials, (()):69, 2023.
[6] Gabriele Corso, Hannes Stärk, Bowen Jing, Regina Barzilay, and Tommi S. Jaakkola. Diff- dock: Diffusion steps, twists, and turns for molecular docking. In The Eleventh International Conference on Learning Representations, 2023.
[7] Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Defossez. Simple and controllable music generation. In Advances in Neural Information Processing Systems, volume 36, pages 47704 47720. Curran Associates, Inc., 2023.
[8] Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. In The Eleventh International Conference on Learning Representations, 2023.
[9] Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeie, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3D shape generation. In Advances in Neural Information Processing Systems, volume 35, pages 10029- Curran Associates, Inc., 2022.

---

[10] Zhiying Leng, Tolga Birdal, Xiaohui Liang, and Federico Tombari. HyperSDFusion: Bridging Hierarchical Structures in Language and Geometry for Enhanced 3D Text2Shape Generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 19691-19700, 2024.
[11] Simone Foti, Stefanos Zafeiriou, and Tolga Birdal. UV-free texture generation with denoising and geodesic heat diffusions. In Advances in Neural Information Processing Systems, 2024.
[12] Marco Aversa, Gabriel Nobis, Miriam Hägele, Kai Standvoss, Mihaela Chirica, Roderick Murray-Smith, Ahmed Alaa, Lukas Ruff, Daniela Ivanova, Wojeiech Samek, Frederick Klauschen, Bruno Sanguinetti, and Luis Oala. Difflnfinite: Large mask-image synthesis via parallel random patch diffusion in histopathology. Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[13] Miguel Espinosa and Elliot Crowley. Generate your own scotland: conditioned on maps. NeurIPS 2023 Workshop Diffusion Models, Aug 2023.
Satellite image generation
[14] Pavel Avdeyev, Chenlai Shi, Yuhao Tan, Kseniia Dudnyk, and Jian Zhou. Dirichlet diffusion score model for biological sequence generation. Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 1276-1301. PMLR, 2023.
[15] Chenyu Wang. Masatoshi Uehara. Yichun He, Amy Wang, Tommaso Biancalani, Avantika Lal, Tommi Jaakkola, Sergey Levine, Hanchen Wang, and Aviv Regev. Fine-tuning discrete diffusion models via reward optimization with applications to dna and protein design. arXiv preprint arXiv:2410. 13643, 2024.
[16] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[17] Robert Brown. XXVII. A brief account of microscopical observations made in the months of June, July and August 1827, on the particles contained in the pollen of plants; and on the general existence of active molecules in organic and inorganic bodies. The Philosophical Magazine, (21):161-173, 1828.
[18] Albert Einstein. Über die von der molckularkinetischen Theorie der Wärme geforderte Be- wegung von in ruhenden Flüssigkeiten suspendierten Teilchen. Annalen der Physik, pages 549-560, 1905.
[19] Norbert Wiener. Differential-space Journal of Mathematics and Physics, 2:131-174, 1923.
[20] Eunbi BI Yoon, Keehun Park, Sungwoong Kim, and Sungbin Lim. Score-based generative models with Lévy processes. In Advances in Neural Information Processing Systems, volume 36, pages 40694-40707. Curran Associates, Inc., 2023.
[21] Hengyuan Ma, Li Zhang, Xiatian Zhu, and Jianfeng Feng. Approximated anomalous diffusion: Gaussian mixture score-based generative models, 2023. URL httts::: net/ forum?ideyc3xen7Bkka
[22] Paul Lévy. Random functions: general theory with special reference to Laplacian random functions. University of California Publications in Statistics, 1:331-390, 1953.
[23] Benoit B. Mandelbrot and John W. Van Ness. Fractional Brownian Motions. fractional noises and applications. SIAM Review, 10(4):422-44 437,22224471111 1968.
[24] Jerry Stinson. The (mis) behavior of markets. Journal of Personal Finance, 4(4):99, 2005.
[25] Francesca Biagini, Yaozhong Hu, Bernt Øksendal, and Tusheng Zhang. Stochastic Calculus for Fractional Brownian Motion and Applications. Springer- Verlag London Limited 2008, 2008
doi: 10.1007/978- 1110077978--8888 1005-18444 144228-797-55
[26] Philipp Harms and David Stefanovits. Affine representations of fractional processes with applications mathematical finance. Stochastic Processes and their Applications, 129(4):
1 1185-1228 2019. ISSN 0304-4149.
12

---

[27] Rembert Daems, Manfred Opper, Guillaume Crevecoeur, and Tolga Birdal. Variational inference for SDEs driven by fractional noise. In The Twelfth International Conference on Learning Representations, 2024.
[28] Philip E. Protter. Stochastic Integration and Differential Equations. Stochastic Modelling and Applied Probability. Springer Berlin, Heidelberg, 2nd edition, 2013. ISBN 17883-662-10000
[29] Anh Tong. Thanh Mguyen-Tang Toan Tran, and Jaesik Choi. Learning fractional white noises in neural stochastic differential equations. In Advances in Neural Information Processing Systems, volume 35, pages 77660-37675 Curran Associates, Inc., 2022.
[30] Samuel N. Cohen and Robert J. Elliott. Stochastic Calculus and Applications. Probability and Its Applications. Birkhäuser, New York, NY, 2st edition, 2015. ISBN 978-1 1178114939-28666
[31] R. L. Stratonovich. Conditional Markov Processes, Theory of Probability & Its Applications, 5 (2):156-178, 1960.
[32] Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-322) 32,,66, 1982. ISSN 0304-4149.
[33] Hans Föllmer. Time reversal on Wiener space. Stochastic Processes - Mathematic and Physics, volume 1158 of Lecture Notes in Math, page 119-129, 1986.
[34] Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, ((24):695-709 2005.
[35] Yang Song, Sahaj Garg. Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach density and score estimation. In Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, page 204, 2019.
(36] Christian Bayer, Antonis Papapantoleon, and Raul Tempone. Computational Finance. Technical University of Berlin, 2021. URL https://www beaabberrinndee www.d//popll//aarrreeeeeeee lecture. pdf. Lecture notes.
[37] Terry J. Lyons. Differential equations driven by rough signals. Revista Matemática Iberoameri- cana, 14(2):215-310, 1998.
[38] James Davidson and Nigar Hashimzade. Type I and type fractional Brownian motions: A reconsideration. Computational Statistics & Data Analysis, 53(6):2099-2066 2009. ISSN 0167-9473. The Fourth Special Issue on Computational Econometrics.
[39] Aaron Lou and Stefano Ermon. Reflected diffusion models. In Proceedings of the 40th Inter- national Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 22675-22701. PMLR, 2023.
[40] Dang Huy. A remark on non-Markov property of a fractional Brownian motion. Journal of Mathematics, 31, 01 2003.
Vietnam
[41] Sebastien Darses and Bruno Saussereau. Time Reversal for Drifted Fractional Brownian Motion with Hurst Index H > 1/2. Electronic Journal of Probability, 12(none): 181 1211, 2007. doi: 10.1214/EJPv12-439,
[42] Philipp Harms. Strong convergence rates for Markovian representations of fractional processes. Discrete and Continuous Dynamical Systems B, 26(10)5556-5557, 2021. ISSN 1531-3492.
[43] Simo Särkkä and Aro Solin. Applied Stochastic Differential Equations, volume 10. Cambridge University Press, 2019.
[44] Tero Karras, Mila Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Advances Neural Information Processing Systems, volume 35, pages 26565-26577. Curran Associates, Inc., 2022.
[45] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge a local Nash equilibrium. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
13

---

[46] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training GANs. In Advances Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
[47] Dan Friedman and Adji Bousso Dieng. The Vendi Score: A diversity evaluation metric for machine learning. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
[48] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[49] Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. In The Eleventh International Conference on Learning Representations, 2023.
[50] Raghav Singhal, Mark Goldstein, and Rajesh Ranganath. Where to diffuse, how to diffuse, and how to get back: Automated learning for multivariate diffusions. In The Eleventh International Conference on Learning Representations, 2023.
[51] Bowen Jing, Gabriele Corso, Renato Berlinghieri, and Tommi Jaakkola. Subspace diffusion generative models. In Lecture Notes in Computer Science, pages 274-289. Springer Nature Switzerland, 2022.
[52] Dongjun Kim, Byeonghu Na, Se Jung Kwon, Dongsoo Lee, Wanmo Kang, and Il-chul Moon. Maximum likelihood training of implicit nonlinear diffusion model. In Advances in Neural Information Processing Systems, volume 35, pages 32270-32283 -22244. Curran Associates, Inc., 2022.
[53] Chin- Wei Huang, Milad Aghajohari, Joey Bose, Prakash Panangaden, and Aaron C Courville. Riemannian diffusion models. In Advances in Neural Information Processing Systems, vol- ume 35, pages 2750-2761. Curran Associates, Inc., 2022.
[54] Charlotte Bunne, Ya-Ping Hsich, Marco Cuturi, and Andreas Krause. The Schrödinger bridge between Gaussian measures has closed form. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2023.
[55] Yang Song, Prafulla Dhariwal, Mark Chen, and llya Sutskever. Consistency models. In Proceed- ings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 32211-32252. PMLR, 2023.
[56] Giannis Daras, Mauricio Delbracio, Hossein Talebi, Alex Dimakis, and Peyman Milanfar. Soft diffusion: Score matching with general corruptions. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
[57] Emiel Hoogeboom and Tim Salimans. Blurring diffusion models. The Eleventh International Conference Learning Representations, 2023.
[58] Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. In Advances in Neural Information Processing Systems, volume 36, pages 41259-41282. Curran Associates, Inc., 2023.
[59] Severi Rissanen, Markus Heinonen, and Arno Solin. Generative modelling with inverse heat dissipation. In The Eleventh International Conference on Learning Representations, 2023.
[60] Kohei Hayashi and Kei Nakagawa. Fractional SDE-Net: Generation of time series data with long-term memory. In 2022 IEEE 9th International Conference on Data Science and Advanced Analytics (DSAA), pages 1-10, 2022.
[61] Christian Bayer, Peter K. Friz, and Nikolas Tapia. Stability of deep neural networks via discrete rough paths. SIAM Journal on Mathematics of Data Science, 5(1):00-7,, 2023.
[62] P Kidger. On neural differential equations. PhD thesis, University of Oxford, 2021.
14

---

[63] Shujian Liao, Terry Lyons, Weixin Yang, and Hao Ni. Learning stochastic differential equations using RNN with log signature features. arXiv preprint arXiv: 08286, 2019.
[64] James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster. Neural rough differential equations for long time series. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1829-7838. PMLR, 2021.
[65] Fischer Black and Myron Scholes. The pricing of options and corporate liabilities. The Journal of Political Economy, 813)663776654 1973.
[66] Christoph Czichowsky, Rémi Peyre, Walter Schachermayer, and Junjian Yang. Shadow prices, fractional Brownian motion, and portfolio optimisation under transaction costs. Finance and Stochastics, 22:161-180, 2018.
[67] Paolo Guasoni, Zsolt Nika, and Miklós Rásonyi Trading fractional Brownian motion. SIAM journal on financial mathematics, 10():769-788 2019.
[68] Christian Bayer, Peter Friz, and Jim Gatheral. Pricing under rough volatility. Quantitative Finance, 1666):877-904, 2016.
[69] Jim Gatheral, Thibault Jaisson, and Mathieu Rosenbaum. Volatility is rough. In Commodities, pages 659-690. Chapman and Hall/CRC, 2022.
[70] Olaf Ronneberger, Philipp
Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 234 241. Springer, 2015.
[71] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, December 2014. arXiv:1412.6980 [cs.LG].
[72] Leslie N. Smith and Nicholay Topin. Super-convergence: very fast training of neural networks using large learning rates. In Defense Commercial Sensing, 2018.
[73] Marjorie G. Hahn, Kei Kobayashi, and Sabir Umarov. Liatee--llanck-ollmmgee equations associated with time-changed fractional Brownian motion. arXiv: Mathematical Physics, 139: 691-705, 2011.
15

---

Appendix
The mathematical framework generative fractional diffusion models
17
A.1 A Markovian representation of fractional Brownian motion
17
A.2 The forward model
18
A.3 Estimating the score via augmented score matching loss
Forward sampling
23
Implementation details
26
D Additional experiments
26
Illustration of generated data
28
F Computational cost of augmenting processes
30
Likelihood computation
32
H Challenges in the attempt to generalize
32
Notational conventions
34
16

---

A The mathematical framework of generative fractional diffusion models
In this section we provide the mathematical details of the score-based generative model defined in the main paper. The driving noise of the underlying stochastic process is based on the affine representation of fractional processes from Harms and Stefanovits [26] and further simplified by the closed form expression to determine optimal approximation coefficients of Daems et al. [27].
A.1 A Markovian representation of fractional Brownian motion
We begin with the definition of type fractional Brownian motion, defined on the whole real line, possessing correlated increments that are in contrast to type II fractional Brownian motion stationary,
Definition A.1 (Type Fractional Brownian Motion [23]). Let (9,, F, IP) be a complete probability space equipped with complete and right continuous filtration {Fe} and I' the Gamma function. For two standard independent [Jr)-Brownian motions (BMs) B and B the centered Gaussian process
W = (W DAER with
W." T(H+ +22
1
[ ((t ((t-s)"
"- (-s))" +
1
[(ee (t (((--)"-- -)"-tdB (22)
T(H 2)
uniquely characterized in law by its covariances
E [ " = 2 2 + 2H (t- s)2" 128>0
(23)
is called type I fractional Brownian motion (fBM) with Hurst index H E (0.
Type II fBM from the main paper is retrieved by setting the additionally defined BM B on the negative real line to zero. Therefore, the difference to type II fBM is the stochastic integral w.r.t. B that yields stationary increments and a non trivial distribution at O. For H = 0.5, the process is a BM and has thus independent increments. For Hl E (0, 1) } the process possesses correlated increments and, compared to BM, smoother paths for > 0.5 due to positively correlated increments (super- diffusion) and rougher paths for H < 0.5 due to negatively correlated increments (sub-diffusion). These three regimes reflect for type I fBM in the same change of quadratic variation from t to zero quadratic variation in the smooth regime and to infinite quadratic variation in the rough regime [30]. To prepare the approximation of the Markovvaan and non semmmmrrratee fBM [25] via Markovian semimartingales, define for every 7 E (0, 20) the Ornstein-Uhlenber process y7 given by
Y = Yoe to + fee-xtt-sada dB. t t20, Yo = dB.
(24)
with speed of mean reversion 7 and non trivial starting value in contrast to the OU processes defined in eq. (6) of the main paper. By Itô's product rule [30], the process Y solves the same SDE
ar? = - dB,, Yo
en aB.
(25)
with different starting value. According to Harms and Stefanovits [26] we represent fBm by an integral over the predefined family of Ornstein- Srrein--Uhlebeckk processes.
Theorem A.2 (Markovian representation of fBM [26, 271). The non-Markovian process W permits the infinite-dimensionall Markovian representation
Jo 00 (Y, -Y²)
H
W.B
(26)
So O. (Y (1))2(7)d
H >
where
7- 2
7- (1-
vi(r)= I(H
and 12(7) I(H + 4 H)
(27)
H)
Note that we follow Daems et al. [27) in replacing the process Zi = 2ie-t 4 + +5- (1-
from the original theorem throughout this work by 2 -2,Y,' + (0,10" + e This is
17

---

justified by Harms and Stefanovits [26, Remark 3.5] and simplifies for H the approximation of fBM and the definition of our generative model, since we only have to reverse the Y processes instead of the pairs (Y1, 27). For You 0 eq. (26) yields an infnite-dimensionall Markovian representation of type II fBM [27]. The MA-fBM from Definition 3.2 in the main paper becomes for type fBM
K Wk (Y, -Y)) H E (0.1), 22 0
B
(28)
k=1
with non trivial Y, (%) ...., Yo that is centered multivariate Gaussian with covariances E | 1/((% + m2) [27]. Theorem 3.3 holds true for type fBM well with optimal approxi-
mation coefficients given in Daems et al. [27, Proposition 5]. For more details on the properties and distinction of type and type II fBM we refer the reader to Daems al. [27].
A.2 The forward model
We define in the following a score-based generative model approximating a fractional diffusion process driven by type fBM. For the remainder of Appendix A we assume Y = - d for all - < K where the setting from the main paper with type fBM is recovered by choosing y = instead. Let B be a D-dimensional MA-fBM with Hurst index H E (0, 1). For continuous
functions y [0, T) IR and g: [0, 7] IR we define the forward process X (X (xeeoo by
dx,- d,t + (t)
X
Xo ~ Po.
t E [0,
(29)
where po is an unknown data distribution from which we aim to sample from. Using eq. (25) we note
K
K wndBe
aBm = Σ
+
(30)
where B (B, Ba) is multivariate BM. With + = E K forward process as
we rewrite the dynamics of the
K Σ wkmn Y dt + @g(t)d. t [0,
dx, = [(t)xx g(t)
(31)
k=1
Taking into account the dynamics of the OU processes, we define the augmented forward process Z (Z,7) by
Z: = (Xr,1, Y. Y,K Xe,2, Y22 ... Y following the dynamics
1. .. X,D, V.D.... X,D) E D(K-1)
(32)
dz. = F()) G(()dB,
(33)
with F()) diag diag(R((), (R(1), R()) E D((K+))
R(t)=
(u(t)
-9(()mm
-g(()KrK
E ERK+IKHI
(34)
0K
-iag((n,... 7K)
and
G(t)= = (T ID ID)" E D((++)),
(35)
For each dimension 1 < d < D. the dynamics of the process transforming Xo,d reduce to those of the augmented forward process with D 1, given by
dz. Z( + G(L)dB,,
(36)
where the K + 1 processes that transform Xy.d are all driven by the same one-dimensional BM B. The augmented forward process Z conditioned on yo..., tb....yy and data sample Xo ~ Po is linear transformation of BM and hence a Gaussian process and so is X [43]. Since the integral w.r.t BM has
zero mean, the mean vector of the augmenting processes is E Y & mean of the conditional forward process is the solution the ODE
f O₂ for all 1 < k < K and the
O,E [xo] ((t)EE X [x]]
(37)
18

---

and hence the marginal mean
(% ((s)dds
E X = dt)xx
with ctt exxp
(38)
is not affected by changing the driving noise to MA-IBM. The marginal covariance matrix Σ of the conditional augmented forward process can be approximated numerically by solving an ODE, see Appendix B for details. In addition we present a continuous reparameterization of the forward process, resulting for some forward dynamics in a closed form solution of the marginal covariance matrix. Our result generalizes the explicit formula for the perturbation kernel Poe (x(xo) = N(x; x(t)xo, c² (t) given in Karras al. [44].
Proposition A.3 (Continuous Reparameterization Trick). Let Xy be a fixed realisation drawn from pa- The forward process X (x,7) conditioned Xo admits the continuous reparameterization
xo + S' all,s)dB.) c(t)
g(s) c(s)
X. = c(t) =c()
Σ Wk Th
ds
(39)
k=
=0 for type 1I (BM since Y =0
with c(t) = exp (fo (ss)ss) and
K
Wk7k [" g(() --rala--- du w g(s)
alt, s)
Σ
(40)
c(u)
c(s)
such that X, N (c(1)xa, (2(())2(1) +o (1) La) Gaussian random vector for all (0, 7) with
a² = / a at
(41)
and
g(s)
ox (t) Σ
Wk
du
(42)
2
c(u)
k=
(t)
W
+
1 g(s)
c(s)
e
27 S
g(()
c(s)
om ds
(43)
vanishing for underlying type 11 fBM.
Proof. By continuity, the functions and are bounded. Moreover, the processes Y .... Y posses continuous, hence bounded, paths and thus
L In(w)]du co, 1'" (u)du
and / Σ wann Yt
P a.s., (44)
where the last integral is understood entrywise. Hence, by Cohen and Elliott [30, Theorem 16.6.1], the unique solution of the SDE eq. (31) is given explicitly as
xo - -
1/" g(u) c(u)
K
du + S g(u)
X, = c(t) =(t)
Wo nh y
dB
(45)
c(u)
with e(t) = exp (Io (())ds). Define
(Y(K] (t) Σ
wk7k ( g(()) ((s)c=op) - ds
(46)
c(s)
19

---

and by the definition of g in (24) we calculate using the Stochastic Fubini Theorem [26]
J.¹
g(w) du)
!,"' /
g(u) du)
Σ WA
Y
du Σ Yk
(u- dB,du + J(Y(K) .)
(47)
K
g(u) c(u)
---(ume dudB, + (x
WkTH
(48)
and hence
g(u)
du i / g(u) dB
x, c())
X0
Wktk Y
(49)
c(u)
d())
K
7k { g((u) e -x dudB + [ c(u)
/ g(u)
dB - -J(--( (1)
=((
X0
(())
k=1
XD + !
Wh?k {" g(u) e - du + g(s) Σ
K
dB, - J 3(x!!1
=c(t)
c(u)
c(s)
km
c(t)xo + dt) f'" !'" a(t,s)dB, c()] (x!
(50)
with
K
/ g(u) (u(--) du + 5 g(}) c(u)
att, =-
wkTk
(51)
c(s)
k=1
Since at,, -) is continuous for every fixed t E [0, we have fo at(t, s)ds < co. Using that the integral of a bounded deterministic function w.r.t. Brownian motion is Gaussian process we have by Itô's isometry
S. al,,, ()),, ~ N (0),) with 0²(1) = 02(1)= S a2(1,
(52)
Therefore, conditional on Xo, the random vector X, is Gaussian with mean vector
m c(t)=o + ( Xn exp ( ((s)dss)
(53)
Moreover, B, and Bj corresponding to the entries of B (Bi,... Ba) and B (Bi, Ba) are independent by Theorem A. resulting in the entrywise variance
E ct) S(" a² (t,ts))s + (t)
(54)
with xk(1) = (J(Y!
= =c((1)
7kk 2
Wk
g(s)
du c(u)
2
(55)
k=
Wk S" g(()
8 ds 1' g(())
2x²
-
- -dss. ds,
(56)
Nh n c(s)
c(s)
kel
where we used again Itô's isometry to calculate
1 vo = Sudd S T T a =
dd
(57)
7h + 71
Since the entries of B independent, find the covariance matrix
=x = (c(tto²())) + o (t) La-
(58)
20

---

The preceding proposition generalizes the 'reparameterization trick from discrete time to continuous-timg the sense that
X,,
Vo, Xo + v1 11-001 -0,, 6~ NOs. Ia)
(59)
used discrete time [2] with time steps = <.. <IN is replaced by our continuous-time reparameterization
X, =(() Xo + / all,, a().s)) + +c(1)
Σ
K
Wk7k /
g((s) c(s)
( 07 ds Y
(60)
k=1
enabling directly sample X, Xx ~ ~ (cc(t))x + (t) x Xo and time point T), in case that (1) and (1) have closed form solution. For complete characterization of the marginal covariance matrix E, of the conditioned augmented forward process calculate by Itô isometry with X Xj and y yj for all 1 S D, <IS K and any t [0,
Ib) for given data sample
E [X,Y]] = c(t) I' alt, o)t,s)e ()t,t))-nllool ol + c(t)
K
Wk
e nt S
g(s)
e ds c(s)
(61)
7k
1=1
and
E [ e-(u+t))
1 e- (matra)
(62)
Yk + 7k n
"ik + 78
reducing for type 1BM
E [XYY c(t) I att, s)e - and E [Y,Y
- e (m++)
(63)
Tkk + 7
We denote in the following the stacked vector of the augmenting processes by
[K] (Y2,,, y.,...., Y,,, Yi,,, Vi,,..., Y -... Yi,,, Yi,D..., ... Ygb) E R D(K+1),
(64)
The random vector Y y!K] is centered Gaussian process with covariance matrix
A = dia EX) E D.K.D.K EY E RRK
| = [y]x
(65)
where ΣY does not depend on the dimension SJ D and we write de for the multivariate Gaussian density of y!KK] Since we know the distribution of y we can directly calculate the corresponding score function by
Inl log qi (Y (x! A, Y K
(66)
A.3 Estimating the score via augmented score matching loss
Conditioning Ze on Xo ~ Po and realisation y!K] of the stacked augmenting processes Y!!] defined in eq. (64) fixed time & [0, T] results in the Gaussian vector X, N(m, Σ,) with mean
K
m c())xo +
wkyt where o
E [X,V]]
(Σ (E))-1)
(67)
k=1
1=1
and covariance
K Σ, - (()) La where 7? = >>>ma X,Y:A
n
(68)
Am1
We denote with Vx log Por the conditional score function of X, and calculate for the gradient w.r.t.
x (z,, TDD E D
Vx (x [K] xn) = -E, -mm) =
(x - m.
(69)
(c) - -,²)
'See petts:///iin iawwelgiawwngggg.. in discrete time.
07 oo-.aao--------ffffffsoo---od-oo for the derivation
21

---

and for the gradient w.r.t. yt = -yf .... .b.B// E R
y log (xly!() [K] xn)
2 vy+ [(x
1
" (x-- mi)
(70)
=- Vx log po: (xly) [x] xx))
(71)
Deploying this relation of Vx log Por and yy log Por we derive the augmenting score matching loss that reduces the dimensionality of the score model we have to learn to the dimensionality of the data
distribution and results in a score model guided by the the known score function Vytn log qr-
Proposition A.4 (Optimal Score Model). Assume that sg is optimal w.r.t. the augmented score matching loss C. The score model
So 1) so(X: Inivi,o) (n s (X, Σ
sx Envi,tt
k
k
yields the optimal (P) approximation of Vx log (Z+) via
So (Z..t) t) + V. log (K)) Vz logg
(72)
Proof. Fix & [O, We write pi aug for the density of Z₁ Por for the conditional density of Z, on Xo. Por for the density of X, and for for the conditional density of y!! on Xo. First note that Y Y!K] and Xo are independent by assumption and hence 9: = gor- By direct calculations we find
Vx log Pt aug (Z,) = E (X.
Vx logPut
Put
ang Xo)]
(73)
Vx log PPoe (X, Y Y
Xo) (Y)))o(Y(!A)) (x!(
= E
xx.,..
(74)
Vx Por Xx/
E
[K] Xo) + Vx log at Y ( [k])
(75)
=0,
E [xx Xx M!M)
Vx Vx pue Pxlogpue(X)) (X, Y [A] Xu)
(76)
X₁ Ex Y - -((t)xo 2 (1) - T?
(69)
E X!!')
(77)
Hence the best L aappoxxiitti of Vx logp! g:00 (Ze) is a minimizer of the augmented score matching loss by
X, Ex c(1)Xo)
Vx log ang (Ze)
(77)
Xx x(!
(t)) -
(78)
K
X, E nh Y c(t)x,,
min
E
E (x..
(x!
(X, Σ
n Yk t)
(Xo.
(t)) - T?
40
km
(79)
K Vx log (x,PoeeX,lYi X [K] .xo)
(09)
arg min E
(x, y! E (x,, Xx)
so
Σ
m
Y
Ap
(80)
Assume now that sg is a minimizer of the augmented score matching loss. Similar to the calculation above we have
y log Pt aug (Z:) F. (xx x. {k))
V y log (Z, Xo)]
(81)
x log (pus (X,) Y! !K] Xx) (x( [Xx)
E (X, X, (())
(82)
yu log (X, Y!!! X²) + V log q: Y ! [3]]
-xxxo x. (k))
(83)
(70) -, (X.1X, x!Ml)
Vxlog log Poe X, Y!!! [K] Xo)] + Vy log 10 k!!l)
(84)
22

---

and hence sX,, Y + vy, log lo (Y!K1) is the best approximation of y log vu (Z,) in (P) and the score model
(o == so(Xx E ti t), -) so E We Y t), ni so Σxxi,t)
yields the best aaapppoaooo of V log pt via So (e, () + Vx lo (Y! R V2 log
(85)
B Forward sampling
We assume throughout this section type fBM. Given the marginal covariance matrix E, of Z we uniformly sample first a time point t E (0, T] and second Z₁ ~ (2,, Στ) with
2, = (c(t)xo,1, ...,, 0, (, O,... 0,.............................. ........ x(t)xx, 0,...0) E D(K++))
(86)
where we use E X Xo] = x()xx and E Y = OD. In the following we characterize further the entries of the marginal covariance matrix E, The calculations in this section are straightforward; nevertheless, we present them in full detail to facilitate easy understanding for the interested reader. We begin with rewriting o² from Proposition 4.2 given by
0² = (t)) L." a² (t,
(87)
where
st, 3 g(8)
Σ
Wk [:
g(u) c(u)
(m du
(88)
c(s)
k=1
K
g(s)
g(u)
Wk
c(s)
7k
c(u)
(u- du
(89)
1=1
=ian(taa)
With
fu(u,s) = g(u)
(((-s) and ls(t,)) = L: fulu, s)du
(90)
c(u)
have
of - ( (1,)s)
(91)
=t(()] !
Σ
K
Wk
g(s)
7k [ falu, s)due)
) ] ds
(92)
c(s)
A==1
= = /
wxaa(t,s) ds
(93)
= =1( /''
Σ
wraeyas(1, s)ar(1,, s)ds
(94)
i=15=1
K
wrwye? (1) S
Σ
(95)
i=,j=1
K
e ('(2(s) c(()
(g(s)
(g(s) ()4,(t,s) ds
- rd((tss)
(96)
c(s)
=1,==1
K
varr(()) - ((t) /
((())
Σ
wiwl
(m)l((t,)) + (tts(t, 5)) w)s44(t,)44,(tt.. ds
ds).
((s))
1-=1
(97)
23

---

where
var (())) S/" g2(s) c(s)
( ds
(98)
corresponds to the purely Brownian marginal variance, explicitly calculated for VE and VP in Song et al. [16]. Using the above derivation, we derive the closed form variance schedule for FVE dynamics.
Fractional Variance Exploding Fix max > Omin > 0 and define r == mmin [16] we set
Following Song et al.
(ut)) = and g(t) = ar with a = -minn (log(r)
(99)
such that =(t)= exp(O) and calculate
st) J. fu sud [. ar" -oall-nal-odaa du F(t) F(s)
(100)
(nn())e-nttuu - cdu(r)o)) ( an
- -r)),
(101)
In(r) 7k
since the derivative of F(u) " is given by 7a(uus)
du du (apr"e -uu(uo)) = Ogr" In(r)e ((lr))))-((s))) mu(u=)) + axr " ()
d F())
d
(102)
(0n((((r)) (In(r) - 7x)(r" (m((a))) ar ralu-s)
(103)
In(r)- 9i
We calculate for the variance of X,\x0
wHw waa (var a7s / sdds(t -07s / r21
[Xe]xo]
Σ
(104)
J((0)
12(()
+ 707 / (1,
(105)
(1,(t))
with
[. [-' rd (fte-mut-o)) -1²) ds as /- p ds 0k 1 2d
J=(1) as
(106)
2 - 3et 2 1
ax [F((t) Fi(0)] a4 (12 F2(()) as
(107)
In (n 7k
21n(r)
since
d (8) (r(+*In(r)e
ln(r)e-ullt- (((-6)
+ ( ((tte- r e
(108)
ds
In(r) + 7k
d ds (2)) ds 21](r)
d 22 20 In(r)2 22.
(109)
2 2l((r)
Finally
(t) a,az 1² (r'e-nll-9)) - pr) (1'c->)(t-s)) - ds - - (1-e-(((stt))) (1-e-((otts)) (((ott)))))
(110)
Jo
26
2 - -
7.2 754 2
+ 21n(r)
(111)
7/8 7'
7/ + In(r)
7/ In(r)
24

---

Analytcal Approsimated
Amaltical Appucimated
Var X.]
1500
Vull?
1000
500
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.8
1.
(a) Variance schedule of the forward FVE process.
(b) Variance schedule of the augmenting processes.
Figure 5: Analytical solution (blue) used by our method for FVE dynamics with K = 5 and H = 0.5
compared to the approximated solution (dashed red) resulting from solving ODE (119).
We calculate the covariance of X, X, and Y!
V₁) = d(1) / a(l, -1(t-00)-))
(112)
K
g(s)
7/ S- Salu, s)duy e -ilee-s)
Wk
(113)
c(s)
r - (x((0))) ds 7h J. fulu, sidue - -ult-))) o)
W/
(114)
km
K
pt r¹ ( Taa (-" (r'o-wut-s)) - re) - -i(t-
wk
k=1
(115)
K
ae vt S// e - aaa / (f'e- -10) e rcl-
A=1
(116)
K
(r'-e-711) ah (1-e-t(aatn))) 7/ In(r)
(a + ax7k)
(117)
7k
Fractional Variance Preserving To the best of our knowledge, there is no closed form solution for Sr (u, s)du for the dynamics of FVP. In this case, we numerically solve an ODE to determine the marginal covariance matrix of the conditional augmented forward process.
General Dynamics. The covariance matrix of the conditional augmented forward process with dynamics
dz. F(t) + G(t)dB,,
(118)
solves the ODE
0,E, = F()= EFF(1) (AF())*+(()GGI)) +((GG(()
(119)
lacking in general closed form solution [43] in contrast the setting of Song al. [16]. This approach applicable for any choice of and in the forward dynamics, but depending on the choice of drift and diffusion function it might not yield numerically stable solution. We empirically observe in Figure that the analytical solution for FVE and the numerical approximation of the variance schedule, determined by solving eq. (119) do not differ significantly.
Variance schedules. We normalize the variance schedule of FVE and FVP dynamics such that the variance = 0 and = T equal to the variance used the purely Brownian setting of VE and VP dynamics. For both FVE and FVP dynamics we calculate w according to Proposition 3.3 and determine o and define w / 0/66. to weight the OU-processes. By doing so, the terminal variance remains the same throughout different choices of H, empirically confirmed in Figure 6.
25

---

1.2
# 0.5
1.0
Var Xi]
1500
1000
500
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.5
1.
(a) Variance schedules of the forward FVE process.
(b) Variance schedules of the forward FVP process.
Figure 6: Normalized variance schedules for K = over time. (a) Variance schedules of FVE dynamics, calculated in closed form according to the derived formulas. The shape the schedule is preserved throughout different values of H. (b) Variance schedules of FVP dynamics numerically approximated. The shape of the schedule is shifted for different values H.
In Figure we observe for FVE dynamics that not only the terminal variance is the same across different choices of but also the shape of the variance schedule. For FVP dynamics, the shape of the variance schedule shifts with different values of H, approaching nearly linear schedule for H 0.1, while H 0.9 offers decreasing variance towards the end near T.
C Implementation details
We used for all experiments conditional U-Net [70] architecture and the Adam optimizer [71] with PyTorchs OneCylce learning rate scheduler [72]. On MNIST trained without exponential moving average (EMA) while on CIFARIO we conducted experiments with and without EMA.
Set up MNIST. We used attention resolution of [4, 2]. 3 resnet blocks and channel multiplica- tion of 2, 2, 2, 2] and trained with a maximal learning rate of 10 for 50k iterations and a batch size of 1024. For all MNIST training runs used one A100 GPU per run, taking approximately 17 hours.
Set up CIFAR10. We used attention resolution of [8]. 4 resnet blocks and channel multiplica- tion of [1, 2, For the experiments without EMA, we used the same setup as with MNIST, but trained the models in parallel
A 1100 GPUs for 300k iterations with effective batch size of 1024. When training with EMA, we followed the set up of Song et al. [16] using an EMA decay of 0.9999 for all FVP dynamies and an EMA decay of 0.999 for all FVE dynamics. contrast to Song et al. [16] we used Py Torchs OneCycleLR learning rate scheduler with maximal learning rate of 2 10 and trained only for Imii iterations instead of the 1.3mio iterations Song et al. [16].
D Additional experiments
In addition to the experiments presented in the main part, provide additional results here, including a full evaluation of FVE dynamics on MNIST, as well as training on CIFARIO without EMA.
Evaluation of different Hurst indices of FVE dynamics MNIST. In Table 4 provide the evaluation of FVE dynamics. For the ease of comparisan, we include the quantitative results on FVP dynamics already presented in the main part. For FVE dynamics both, the super-diffusive regime and the sub-diffusive regime achieve a higher FID as the purely Brownian dynamics for K 1, throughout all tested Hurst indices and for K H = 0.9 with a higher pixel-wise diversity in the sub-diffusive regime of H < 0.5.
throughout all tested Hurst indices except for
Training on CIFAR10 without E.M.A As Song et al. [16] point out, the empirically optimal EMA decay rate for VP dynamics differs from that for VE dynamics. Since we do not have the computational resources to optimize the EMA decay rate for every configuration of our framework, we evaluated it in line with Song et al. [16] using consistent EMA decay rate of 0.999 across all configurations FVE dynamics and 0.9999 across all confi infiguratiions FVP dynamics. Nevertheless, because the optimal EMA decay rate appears to depend on the dynamics of the underlying stochastic
26

---

MNIST
12 N.w07
N: 05
#7 0.
VINST
d 0.
= 0.7
= 05
10. 14.7
1914 W5.
15.1
70. 15.1
15.7
IM driven
HM dehsen
VP Uenoined
1.46
23.4
VE trennissd:
10.82
24.50
FORUE
2.56
23.56
301
11.75
2.81
53.00
2.09
39.50
10.00
21.05
9.65
24.24
10.30
24.23
8.08
21.31
P
24.00
2.30
21.83
8.00
33161
2:56
09.87
INELE
2
9.83
24.00
9.73
26.38
9,80
28.16
9,42
24.26
287
IVEIZ
1101
9.74
INP
2.15
24.63
21.56
33.70
FVE
(a) Conditional image generation on MNIST with FVP. (h) Conditional image generation on MNIST with FVE.
Table 4: FID and pixel-wise diversity scores of GFDM compared to the original setting of purely Brownian driven dynamics VE and VP. In bold the scores that are better than both purely Brownian driven dynamics VE and VP. The overall best scores within the experiment are boxed in.
CIRARI
#
#1 0.5
Table 5: Quantitative results for FVE dynamics and varying Hurst index on CIFARIO trained without EMA. In bold the scores that bet- ter than both purely Brownian driven dynamics VE and VP. The overall best scores within the experiment are boxed in.
FID
VS,T
FID
V5,7
FID.
V5,
BM deincs
VT titraisss
9:38
0.91
Oecuaned)
17:29
2:24
MA-IMM drives
13
9.52
1:222
9-46
3.22
K.05
3.26
PVELM, M,
5.00
3:24
62
3.29
10.25
3.00
PVE M,
16.67
9:175
13.41
2:04
2.62
process, we also evaluate our framework without EMA. In Table 5 we observe that the best performing configuration in of FID is FVP( 0.9, K 2) with an FID 8.99 and FVP( 0.1, K 1) with an FID of 8.93 compared to the purely Brownian dynamics VE with an FID of 9.38 and VP with an FID of 17.29. Due to limited computational resources we only compared the best purely Brownian dynamics (VE) with the performance of corresponding augmented FVE dynamic of GFDM. As to be expected, using EMA for training of GFDM results improved performnace w.r.t. image quality measured by FID obervable in Table 2b.
Effect of the number of augmenting processes in the super-diffusive regime. Additionally, in Figure 7, we show the FID evolution of the super-diffusive regime for various numbers of augmenting processes, showing a similar pattern that either that K = 2 or yields the best performance across different datasets and dynamics.
MNIST Generation
MNIST Generation
CIFAR10 Coneration without EMA
CIFAR1D B Generation with EMA
3.0
Vi
PVP - 0.9)
15-
VE
FVEIH
AP FVPUY - 0.5)
2.5
PVPe - 0.71
FVPMY - 0.7)
10.6
30.4
0.7
1.5
12
10.0
1.0
9.8
Muugmentingg processes
Eaugmentiin
Kaagmeniing
Figure 7: Dynamies driven by MA-IBM with super-diffusive Hurst index H 0.9 and K 0.7 perform in all four experiments we conducted better than the original purely Brownian driven dynamics, where either K K yields the best performance.
FID
Table 6: Averaged FID values for different NFEs of the sufer-diffusive regime compared to purely Brownian dynamics.
250 NFES
NELS
750 NFES
NFES
BM driven
VE
5.652002
5.28
5.2010.2
5. 4.791om1
VP
15.1210m1
3.8610am
4.79.o.11
MA-BM driven
FYP 0.7.
15.442000
4.47a0.m
4.13
4.12.0.03
FVP 0.9, K
12.44
3.71
3.7010ca
3.70.oou
27

---

E
Illustration of generated data
Visual comparison of generated CIFAR10 images sampled with SDE dynamics.
(a) Purely Brownian VP sample.
(b) Super-diffusive regime of MA-IBM with H 0.9.
Figure 8: (LHS) Images generated with the purely Brownian driven VP dynamics sampled with SDE dynamics, a FID of 4.85 and a pixel-wise diversity of 3.42. (RHS) Images generated with FVP(H 0.9, K 2) dynamics sampled with SDE, FID of 3.77 and pixel-wise diversity of 3.60.
28

---

Visual comparison of generated CIFARIO images sampled with PF ODE dynamies.
(a) Purely Brownian sample.
(b) Super-ditfusive regime MA-IBM with
Figure 9: (RHS) Images generated with the purely Brownian driven VP dynamics sampled with
PF ODE, a FID of 5.63 and pixel-wise diversity of 3.91. (LHS) Images generated with FVP(H 0.9, 2) dynamics sampled with PF ODE, FID of 12.36 and pixel-wise diversity 4.89.
29

---

Visual comparison of generated MNIST images sampled from SDE,
0
0 2 3 4 5
6
7 8
9
01 2 3 4 S
6 7
8
9
(a) PVP(K = 3. H. = 0.9) with FID = 0.72 and vs,
24.18.
0 4 6 8
9
O
2 3 4 5 6 7 8 9
(b) EVP(K = 4, = 0.9) with FID = 1.22 and vS, 24.76.
2 3 9 5
6
9 8 9
0 1 2 3 4 6 4 7 8 9
(c) FVP(K 5, H 0.9) with FID 2.17 and vs, 25.15.
Figure 10: Diversifying effect of the augmenting processes with FVP dynamics MNIST. The super-diffusive regime with !/ = 0.9: For K 5 instead of K 3 augmenting processes the pixel VS increases from 24.18 25.15.
F Computational cost of augmenting processes
In this section we compare the computation time of GFDM to the purely Brownian setting of traditional diffusion models. For a given Hurst index H E (0, 1) and given K, the optimal coefficients w1, Wk are calculated only once before training. For completeness of our quantitative compute time evaluation, we provide the average computation time in seconds, needed to compute wi.... WK on a GPU Tesla V100 with 32 GB RAM. We randomly sample 1000 times H 20[0.1, 0.9] for given K E {1, 21, 4, 5) and report the average computation time in Table 7.
K
K
K 3
4
K
time [s]
0.0043
0.0003
0.0003
0.0003
0.0003
Table 7: Averaged time in seconds needed before training to calculate for given K the optimal approximation coefficients using the approach of Daems et al. [27].
Computation time during training The computational difference during training consists of the computation of the covariance matrix Σ, instead of the marginal variance and sampling from multivariate Gaussian instead of a univariate Gaussian. Note however, that we only need to calculate E, for D = and also sample only once for a given time t and a given data point. In Table 8 and Table we report the average time of one training step measured in seconds calculated over 1000 training steps on CIFAR10. 10. The underlying conditional U-Net has 58.7mio and EMA is applied. The batch size is 128 and all computation have been carried out on a GPU Tesla V100 with 32 GB RAM.
We observe that the computation time depends only minimaly increases when switching from the original model to the augmented system and increases across FVE and FVP dynamics by at most 11/1000 seconds, while the choice of the Hurst index H has no effect on the computation time.
Computation time during sampling Since the augmented system depends for fixed K only on the approximating coefficients W1, WK it would suffice to report the average sampling time for FVP and FVE dynamics for varying K. Nevertheless, Table 11 the average time to sample a batch of 1000 images over 1000 discretization steps of the reverse-time SDE over 10 trials. We observe that the average time in seconds for sampling step in the reverse dynamics of FVE and FVP dynamics increases for < 4 by at most 2/100 seconds. Only for K 5 observe significant increase of average sampling time of roughly 4/10 seconds.
report for H E (0.9, 0.5, 0.1) in Table 10 and
30

---

training step time [s]
H 0.9
= 0.5
H = 0.1
average
VE
0.0478=t0.100
0.0478
FVE(H, K =
0.0489
0.0483
0.0485+0..0pg
0.0486
FVE H. K 2)
0.04861000944
0.0484 0484
0.0485=10.000
0.0485
FVE H. K
0.0487:0.090
0.0484 0502
0.0493
0.0488
FVE H. K
0.0492+0.09s8
0.0484=00.888
0.04877=0.00m
0, 0488
FVE H. K=
0.0987
0.0488
0.04166:0.0033
0.0487
Table 8: Average time in seconds for one training step with FVE dynamics on CIFARIO with a batch size of 128, conditional U-Net with 58.7mio parameters and EMA.
training step time [s]
H = 0.9
= 0.5
= 0.1
average
VP
0.0478#0.180
0.0478
K
0.0475+0.0809
0.0475=0.085
0.0475+
0.0475
2
0.0176+0.0007
0.0477=t0.092
0.0481 000007
0.0478
K
0.0483+0.0037
0.0477
0.0477
0.0479
K = 4
0.047622.0800
0.0479 0916
0.0484
0.0480
K 5
0.0484+0.0942
0.0479 0925
0.0479#0.0030
0.0481
Table 9; Average time in seconds for one training step with FVP dynamics on CIFARIO with a batch size of 128, a conditional U-Net with 58.7mio parameters and EMA.
sampling step time [s]
H = 0.9
H = 0.5
H = 0.1
average
VE
2.3092
2.3092
1
0.3125+
2.3269 1069
2.3261 100.
2.3218
K = 2
2.3095
2.3297 a. 1077
2.3107 1593
2.3166
3
2.3071
2.3133 1083
2.3063++0.
1068
2.3089
K 4
2.3322
2.3323 1007
2.3156+0.1122
2.3267
2.6515++0.0000
2.6560 20.1007
2.6510+0.006:
2.6528
Table 10: Average time in seconds for one sampling step in the reverse dynamics of FVE to generate data of dimension (3, (3,32,32) with batch size of 1000 using a conditional U-Net with 58. 7mio and EMA.
sampling step time [s]
= 0.9
= 0.5
H = 0.1
average
VP
2.3013
2.3013
K =
2.3036
2.3120 #0.1002
2.3031 40.113
2.3062
= 2
2.3139 1161
2.3070 -0.77000.1100 1102
2.3154+o.100
2.3121
K 3
2.3134+0.1248
2.3168 +0.
2,3309 4-0.
2.3204
= 4
2.3199
2.3091 =0. 1332
2.3210+o.
2.3167
K 5
2.6568
2.6603
2.669240.0073
2.6621
Table 11: Average time in seconds for one sampling step in the reverse dynamics of FVP to generate data of dimension (3,32, 32) with batch size of 1000 using conditional U-Net with 58.7mio parameters and EMA.
31

---

G
Likelihood computation
Given the approximate PF ODE corresponding to the augmented forward process
dz: = F(1)z= G(t)G(()) (o((,,)) t) + Va log (e
dt, LE [0,
(120)
(R..t)
we estimate according Song al. [16] the ligllikll of data zo under the learned density soug Po
via
log sau (zo) = log goug (ZT) + /'' Vig (zr. t)dt.
(121)
According to Song et al. [16], we integrate over [e, 7) rather than T], using the same value of 10 which has been empirically shown to yield the best performance when simulating the SDE. For & and type II fBM we need to adjust the starting value of the augmenting processes from zero to a jointly sampled vector ye (ve,..,, )) NOx,, Ae) with
(Ae)a.= = B [v"u']] L." e (1 ds 1 e 1-e-(rati) t7)+
(122)
7h + TV
Using the exact likelihood of ye and the independence of ye and Xo we have auy (z.) = (oggo(Xo) + log ge (ye) log Po
(123)
where Po is the learned density of Xo corresponding to Hence in total
(121)
log snn (zr) + {" - log (y.))
log (xxo
(124)
and we define the negative log- liekliiiihodds NLs of test data xo under the learned density by
NLLs(Xx; 0) - llg v0 (2o) (ye)).
(125)
H Challenges in the attempt to generalize
In this work, we seek determine the extent to which the continuous-time framework of score-based generative models can be generalized from an underlying BM to an underlying 1BM. For a 1BM W is not straightforward to define the forward process
X, = Xo + 1.' ((X,,s)ds + + "'xx,,)))." te [0,
(126)
driven by IBM, since 1BM is neither a Markov process nor semimartingale [25], and hence Itô calculus may not be applied, to define the second integral. However, definition of the integral w.r.t. IBM is established [25, 73] such that the remaining problem is the derivation of the reverse- time model. Following the second and more intuitive derivation of the reverse-time model for BM from Anderson [32], the conditional backward Kolmogorov equation and the unconditional forward Kolmogorov equation are applied. Starting point of the derivation is rewrite p(ze,t, 2s, p(z,, s 2/ t))(,,, t) with Bayes theorem to calculate with the product rule
t,, Øt
2pp(xs,8)xie p(xr,t) Öp(at, t) Øt
Ōt p(z.,)(Ze,t) 8 s2t.
(127)
Replacing ap(rs.t) with the RHS of the unconditional forward Kolmogorov equation and (p(...))x...) St with the RHS of the conditional backward Kolmogorov equation one derives an equation that only depends on the joint density p(rr, s,,,, Using Bayes theorem again leads to conditional backward Kolmogorov equation for p(#,, l|.,, s) that defines the dynamics of the reverse process by the one-to-one correspondence between the conditional backward Kolmogorov equation and the
8t
reverse-time time SDE [32]. Following these steps for IBM, starting from eq. (127) and deploying the
32

---

one-to-one correspondence of fBM and the evolution of its density [73], we could replace pp(a..)) (127) by the RHS of
in
8t
Öp(2, 1) 0t
Op(t, dr; + H Σ (r, t)
pp(,,)) dz,0x,
Σ fi(t, x)
(128)
The missing part is however analogous to the conditional backward Kolmogorov equation to replace @p(a..s) #v.44 in eq. (127). The derivation of such equation is to the best of our knowledge yet unsolved problem and hence the limiting factor in the generalization of continuous-timg score-based generative models from underlying BM underlying fBM.
33

---

I Notational conventions
[0, T
Time horizon with terminal time T > 0
X (Xt)
Stochastic forward process taking values in IR
DEN
Data dimension
X
Vector valued stochastic forward process X (Xi)ee(o.)) with X, (Xe,1 Xt,))
X
Reverse time stochastic process with X; X
f
Vector valued function f RD [0, T R Functions 1. [0. T
4.9
F
Reverse time function with f(x, t) = f(x,T - -)
1.9
Reverse time functions with ji()) 1x( 1) and g(1) g(T 1)
Po
Data distribution
Marginal density of (augmented) forward process at E [0,
Pt
B
Brownian motion (BM)
H
Hurst index H (0,1)
W
Type fractional Brownian motion (fBM)
BH
Type II fractional Brownian motion (fBM)
y === (YE))
Orastein-Uhlenbeek (OU) process with speed of mean reversion 7 E Tk.
KEN
Number of augmenting processes
M1... KK
Geometrically spaced grid
W1
Approximation coefficients
Optimal approximation coefficients = (wi .... wx)
Sum of optimal approximation coefficients
Markov-approximate fractional Brownian motion (MA-fBM)
k
k: E N with 1 < k < K
yk
OU processes yk Y
Y! YK
Augmenting processes with Y -Y ,Y,,..., Y*)
F.G
Vector valued functions F, G j0, - RR (.(+++))
G.G
Reverse time vector valued functions with F(t) = F(T t) and G(t) G -)
By Y1 YK augmented forward process
Y(K]
Stacked vector of augmenting processes
Marginal density of YKk) at E [0, 7]
Weight vector of neural network
34

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We summarize the contribution of our work in the introduction and in the
abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We include section on limitations of our work where we discuss the limita- tions of our results.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
35

---

Answer: [Yes]
Justification: We give a complete proof for our own theoretical results and refer to complete proofs for he theoretical results of others.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We give implementation details the appendix revealing the used model architecture and training procedures.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
36

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We release our code upon publication. Together with the implementation details given in the paper our results can be reproduced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
wwweimissssooioiiiiii onPolicyl for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //niss wwtppclcccc////ccccccccccccc..lt wtwp/////////eeeee/eeeecoe.sseet https//oooSSuuiisiiieeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe the hyperparameters in our section on implementation details in the appednix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Unfortunately we don not have the computational resources to run all experi- ments a sufficient number of times to provide statistical certainty.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
37

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We reveal the hardware specification we use and report the number of hours of training in our section on implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics attps://neut cc/public/Ettiie senbbiicEthicssuudellleeee
Answer: [Yes]
Justification: Our work fully confirms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We include a broader impact statement at the end of our paper discussing potential misuse.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
38

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We only feature experiments on small scale dataset up to size 3 x 32 x 32. The models trained in this work do not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly cite all research works we build on and use the code of others only according to its license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
39

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
[Yes]
Justification: We release our code upon publication alongside a proper documentation under the MIT license.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: our work does not involve crowdsourcing nor research with human subjects.
16. Depending the country in
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
40

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
41


Reviews:
[{"review_id": "TqG3QvfT4o", "reviewer": "Reviewer_jifC", "summary": "The paper proposed diffusion models where Brownian motions are generalized by fractional Brownian motions - a correlated noise process. Due to the non-Markovian property of fractional Brownian motions, the paper suggests approximating it using a linear combination of semimartingale processes. The key insight of inducing fractional noise to diffusion models is to allow a control over the diversity sampling.  Experiments consider different setups of Hurst exponent parameters, demonstrating good performance in terms of inception score and Vendi score.", "questions": "1. Can the Markovian approximation of fractional Brownian motion be extended to multifractional Brownian motions where the Hurst index varies with respect to time?\n2. Can you discuss further potential applications of Fractional Diffusion Models? While the paper suggests they are promising for generating time series, especially financial time series, do you think this model could be applied to molecular design, given that atomic interactions follow fractional Brownian motions?\n\nMinor points:\n\nMissing table reference in Line 722.", "limitations": "Please see above.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written, providing necessary background and come up the solution of variance exploding and variation preserving. \n- The idea of fractional diffusion models is explored in-depth compared to Tong et al. 2022.", "weaknesses": "- I find the discussion in the experiments in the main text does not give much insight. What is the main message that the paper is trying to convey? What is the implication of the numbers? \n- Missing the computational trade-off between Fractional Diffusion Models and traditional Diffusion models. The approximation using $K$ different semimartingale processes requires to sample these processes. How does it affect the training and inference?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback and truly insightful and thoughtful questions, which highlight the potential contributions of our work. We address specific comments and questions below: \n\n>I find the discussion in the experiments in the main text does not give much insight. What is the main message that the paper is trying to convey? What is the implication of the numbers?\n\nWe agree with the reviewer that we did not clearly highlight the advantages of our framework and presented too many numbers that should have been included in the appendix alongside Table 5 and Table 6. We replace Table 1 and Table 2 in our paper with the attached Table 1, where we retrained our model on CIFAR10 on all configurations that we already investigated on MNIST. Comparing the two tables, we see a better performance of the super-diffusive regime ($H>0.5$) across the two datasets, reaching in both experiments the best FID for $H=0.9$, with either $3$ or $2$ augmenting processes. In Figure 3, it is seen that the best performance in terms of FID of the purely Brownian dynamics is dominated by the super-diffusive regime of our method, saturating on a lower FID level with aleary $500$ NFE. In addition, the pixel-wise diversity of the best performing configuration on CIFAR is higher than the pixel-wise diversity of the two baseline dynamics. \n\n> Missing the computational trade-off between Fractional Diffusion Models and traditional Diffusion models. The approximation using different semimartingale processes requires to sample these processes. How does it affect the training and inference?\n\nWe commented on the computation time of GFDM in lines 229-232, arguing that the computational load increases only minimally. We show that the same number of score model evaluations is sufficient, where that the score-model has the same dimensionality as the input data and do not depend on the dimensionality of the augmented system. Since the score-model evaluation takes up the most compute time, augmenting the system with additional process increases the computation time only very little. In addition to this commentary, we now provide a quantitative evaluation of the computational trade-off between GFDM and traditional Diffusion Models. \n\nTable 3 in the attached pdf tabulates the average time GFDM needs to perform one training step for different $H$ and $K$. \nThere is a mild increase of average compute time when switching from the original model to the augmented system, while no significant change between different Hurst indices $H$ or $K\\leq5$.\n\nTable 4 in the attached pdf shows the average time GFDM needs to perform one sampling step. For $K\\leq4$ the average compute time in seconds differs at most by 0.02 seconds, while there is a significant increase switching from $K=4$ to $K=5$. This is presumably due to the inversion of the covariance matrix of the augmenting processes needed to simulate the reverse time model.\n\n>Can the Markovian approximation of fractional Brownian motion be extended to multifractional Brownian motions where the Hurst index varies with respect to time?\n\nIt is certainly conceivable that such an extension is possible. The form of the Markovian representation of fractional Brownian motion (Theorem A.2) suggests that this could be achieved by making the densities $\\nu_1$ and $\\nu_2$ depend on time. While a formal derivation and proof is beyond the scope of the work, this is a very interesting extension to be explored in the future. We now discuss this in our future work.\n\n>While the paper suggests they are promising for generating time series, especially financial time series, do you think this model could be applied to molecular design, given that atomic interactions follow fractional Brownian motions?\n\nThis is a really great suggestion that we have in mind for our future work. Combining our framework with a Brownian Bridge model to switch between different unknown distributions might be a promising direction. In particular the reviewer's suggestion to use our framework for molecular evolution might be very promising, since noise increments being independent posits a strong assumption.\n\nWe hope that we have sufficiently addressed the weaknesses and questions raised by the reviewer. \n\n\n\nWith best regards, \n\nThe Authors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SD4DIWoBW1", "reviewer": "Reviewer_kgmF", "summary": "The paper replaces Brownian motion in diffusion models with fractional Brownian motion.", "questions": "If the authors really want to make some contributions to image generation, I suggest they spend more energy understanding the structure of image data, instead of focusing only on plugging in a fractional BM to replace BM.", "limitations": "yes", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "The mathematical description of fractional Brownian motion is clear.", "weaknesses": "1. The paper has very thin numerical experiments. The results do not adequately justify why replacing BM with fractional BM is preferred. \n\n2. In order to show that fractional BM is better than BM, the experiments should focus on exactly showing the marginal improvement while controlling everything else, which the paper does not do. \n\n3. The paper does not have strong methodology contribution, in additional to the replacement of fractional BM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gog7U2kwIB", "reviewer": "Reviewer_HxVk", "summary": "The work proposes a theoretical framework for training score-based diffusion models based on fractional brownian motion.\n\nAuthors provide an approximation framework where fBM being approximated by Markovian noise to derive score-matching in Markov approximation.\n\nTheoretical derivation is supplemented by experiments showing some gains w.r.t. score-matching driven by standard brownian motion.", "questions": "If the process driven by fBM is adapted, can we use Gyöngy theorem to reduce the process to BM one and connect this score-matching procedure to BM driven score matching? If so, would it imply that gains are coming from having different path-integrator, rather than from just using fBM? \n\nCan authors add pvalues to the tables?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "An interesting idea of extending score-matching to non-Markovian setting and approximating otherwise intractable fractional BM with colored brownian motions.", "weaknesses": "It feels that motivation for why we should use fBM is not well elaborated. \n\nMany different settings evaluated without sensitivity analysis of $H$ and $K$ on resulting quality. It is rather hard to interpret the findings whether the gains are from fBM or from multiple comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer HxVk, \n\nWe thank the reviewer for their valuable feedback.  Our detailed responses follow:\n\n> It feels that motivation for why we should use fBM is not well elaborated\n\nHaving control over the diffusion trajectories are crucial for obtaining a trade-off between generation quality and diversity. Moreover, there is no true support for why the noise increments should be independent. Our framework provides two hyperparameters, the number of processes and the Hurst index to have control over diffusion processes and characterize the nature of trajectories, \\emph{e.g.} roughness vs. mild randomness. Our results indicate that in both aspects, quality and diversity, the diffusion models driven by Brownian motion lead to suboptimal results and the best outcome are obtained when $K\\geq 2$ and $H>0.5$. We now mention this in the introduction of our main paper. \n\n> Many different settings evaluated without sensitivity analysis of $H$ and $K$ on resulting quality. It is rather hard to interpret the findings whether the gains are from fBM or from multiple comparisons.\n\nWe agree that the tables in our main paper could appear cluttered. \nTo foster comparability of our experiments, we retrained our model on CIFAR10 on all configurations that we already investigated on MNIST and replace Table 1 and Table 2 in our paper with the attached Table 1. Comparing the two tables, we see a better performance of the super-diffusive regime ($H>0.5$) accross the two datasets, reaching in both experiemnts the best FID for $H=0.9$, with either $3$ or $2$ augmenting processes. In Figure 3 you see that the best performance in terms of FID of the purely Brownian dynamics is dominated by the super-diffusive regime of our method, saturating on a lower FID level with aleary $500$ NFE. In addition, the pixel-wise diversity of the best performing configuration on CIFAR is higher than the pixel-wise diversity of the two baseline dynamics. \n\n> If the process driven by fBM is adapted, can we use Gyöngy theorem to reduce the process to BM one and connect this score-matching procedure to BM driven score matching? If so, would it imply that gains are coming from having different path-integrator, rather than from just using fBM?\n\nThanks for this very insightful question. Fractional Brownian motion does not fit into the setting of Gyöngy's theorem directly since it is not a semimartingale so that the forward process is not given in terms of a stochastic differential equation driven by a Brownian motion. The lack of the semimartingale property also makes the question of whether it is possible to match the marginal distributions of the forward process a formidable one which is far beyond the scope of this paper. Importantly, replacing fractional Brownian motion by an alternative Brownian motion model with the same marginal distributions would result in an entirely different dependence structure of the marginals of the forward process, hence possibly leading to entirely different results. This would certainly be very interesting to check but, unfortunately, once again beyond the scope of the article.\n\n> Can authors add pvalues to the tables?\n\nUnfortunately this is beyond our computational resources. One training on CIFAR with 2 GPUs Ampere A100 40 GB RAM takes roughly 50 hours, making it unfeasible to train multiple times. Following the evaluation set up of Song et. al[1] and Lou et. al[2] with a smaller number of trainable parameters we trained for 1 million iterations on CIFAR10 and evaluated one checkpoint every 100k iterations, to ensure that the result is not affected by overfitting. This ensures to some extend that the FIDs we report on CIFAR10 are not a lucky pick. \nNevertheless, we report for the best four configuration in Figure 3 a) the average FID over three rounds of training with the corresponding standard deviation. Please note that we need to sample $3 x 50k$ samples to report one of these scores  with $1000$ NFE which takes 1 GPU  Ampere A100 with 40 GB RAM approximately 16 hours. \n\nWith best regards,\nThe Authors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "83SOtkDQcV", "reviewer": "Reviewer_kZkj", "summary": "The authors introduce a new type of continuous score-based generative models relying on fractional diffusion, a type of diffusion where Brownian Motion BM is replaced by its fractional counterpart fBM, where noise increments or either positively correlated (Hurst index $1 > H > 1/2$) or negatively correlated ($0 < H < 1/2$). The authors use a Markovian approximation of fBM, consisting in the addition of $K$ correlated OU processes as the underlying dynamics guiding the diffusion. This lets them obtain tractable learning and inference, using the augmented process (noised data, OU processes) and the corresponding score. The augmented score matching loss introduced makes it possible to use a single D-dimensional score model to optimally approximate the full score in $\\mathbb{L}_2$, where D is the data dimensionality. The authors then design a set of experiments on MNIST and CIFAR10 to validate the performance of the new method and the effect of varying the hyper-parameters ($K, H$), and outline its advantage over classical score-based diffusion.", "questions": "Overall I am willing to raise my score if my concerns for the empirical study are addressed. Here are some additional questions and remarks.\n\n- With $K \\gg 1$ performance deteriorates. Does it means true fractional dynamics does not work well? What is happening? How does $K$ impact the approximation of true fBM (you cite a reference on Line 184, can you give something quantitative?)\n\n- Why for $H=1/2$ do we see improvement with the augmented dynamics? Because of the correlation structure?\n\n- Table 1 and effect of $K$ for $H = 1/2$: Not much difference, until FID deteriorates, where then $\\textrm{VS}_p$ increases. If the sample quality heavily deteriorates and that the model produces random out of distribution samples, then it is not a very interesting situation and the metric loses meaning. Moreover no discussion on why choosing too high of a $K$ eventually leads to bad samples. Numerical instabilities? Training loss? Difficulty learning the augmented process?\n\n- The Vendi score thus does not seem to be such a good choice as we cannot understand the tradeoff in quality and diversity. It would be better to use some precision/recall metric (or density/coverage) where diversity relates to covering the true data distribution. One could devise some summary statistic like an associated $F_1$ score to appreciate the tradeoff and see if a better 'Pareto optima' is attained. \n\n- It is also not clear what is the effect of $H$ on the samples. It seems rough path $H < 1/2$ have larger jumps but the authors advocate for $H>1/2$ and on line 284 invoke heavy-tailedness. And what is meant by heavy-tailedness here, as the variance is finite?\n\n- The effect of varying the hyperparameters does not seem very consistent. Will the observed remarks for $(K, H)$ hold with different datasets? Will this scale? It would also be nice to have multiple runs with the associated standard deviation to assess statistical significance here.\n\n- In the abstract, limitations of diffusion models are mentioned, in particular (i) slow convergence and (ii) mode-collapse on imbalanced data, which are not addressed in the experiments and discussed properly. For (i), there is no mention of the number of diffusion steps/number of function evaluation in the experiments. For (ii), maybe try a run on CIFAR10\\_LT. \n\n- Table 3: no discussion on why the ODE has bad performance. \n\n- Line 259: 'we evaluate GFDM on [..] test distribution coverage': this seems not to be done anywhere, as Vendi score does not relate to coverage in this sense?\n\n- The diffusion is run on $[\\epsilon, T]$ but the value of $\\epsilon$ is nowhere to be found it seems, and no experiment to see its effect on  GFDM's performance?\n\n- Line 32-34: why would BM's lack of control over sampled trajectories matter? The correlation structure is independent of the model, depending on the data it could also increase mixing time or actually make it harder to learn the score. Is it because the augmented process makes it possible for the model to incorporate better correlation structure? What are the basis of these claims?\n\n- Line 83-84: $\\hat f$ seems not to be properly introduced\n\n- Line 168-169, eq (9): typo in the denominator of the last fraction?\n\n- Line 210: $\\Sigma_t$ seems not to be properly introduced\n\n- Line 255-258: Why would using EMA on the models interfere with the SDE dynamics?\n\n- Line 282: Did you not mean SOTA FID of 0.72, K = 3 and H = 0.9? What are you basing your subsequent interpretation of easier to learn smoother sample paths on?\n\n- Line 291: Did not you mean super-diffusion here with $H = 0.9 > 1/2$? idem in caption of Figure 2, with $H=0.7$?", "limitations": "Properly addressed.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "In terms of theory, the paper is quite satisfying. It is well contextualised in the field of research of fractional Brownian Motion, and leverages a good set of techniques to achieve its goals, and overcome the limitations of previous similar approaches with this noise regime. These aspects are treated efficiently and with clarity.\n\n- Tractable learning and inference\n- D-dimensional score model even though one has to consider the $D\\cdot (K+1)$-dimensional augmented process\n- Marginal statistics for faster training, and explicit formulaes for the VE noise schedule", "weaknesses": "While the paper tackles the extension of diffusion to fractional Brownian motion very satisfyingly, it seems to me that it suffers from an insufficient and unclear empirical study. \n- Performance gains are mildly convincing and the effect of the varying hyper-parameters are not that clear. The general idea seems to say that choosing $H>1/2$ and $K=3$ yields good results, with higher $H$ providing some smoothing beneficial to learning the diffusion and higher $K$ bringing in more diversity, but to be honest I am not convinced by the interpretation of the results and the general setup.\n- It could be beneficial to include a more exhaustive comparison with other approaches for diffusion with different noise regimes, and the different advantages/limitations, especially since the mentioned limitations for classical diffusion models (slow convergence, mode-collapse on imbalanced data, and lack of diversity) are not properly tackled. See, e.g., [1] [2], and even consider some more detailed experimental comparison with [3].\n\n\n[1] Eliya Nachmani, Robin San Roman, and Lior Wolf. Denoising diffusion gamma models, 2021\n\n[2] Jacob Deasy, Nikola Simidjievski, and Pietro Liò. Heavy-tailed denoising score matching, 2022.\n\n[3] Eunbi Yoon, Keehun Park, Sungwoong Kim, and Sungbin Lim. Score-based generative models with Lévy Processes. In Thirty-seventh Conference on Neural Information Processing Systems, 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer kZkj, \n\nThanks for the insightful comments, valuable feedback and very precise and thorough engagement with our work. See below for detailed answers to your questions and concerns: \n\n>Performance gains are mildly convincing and the effect of the varying hyper-parameters are not that clear. The general idea seems to say that choosing $H>1/2$ and $K=3$ yields good results, with higher $H$ providing some smoothing beneficial to learning the diffusion and higher $K$ bringing in more diversity, but to be honest I am not convinced by the interpretation of the results and the general setup.\n\nInline with the reviewer's feedback, we now provide a more interpretable evidence. To this end, we retrained on CIFAR10 with all the configurations we investigated on MNIST before to better detect similarities. We find that on both datasets $H=0.9$ yields the best results with $K=3$ and MNIST and $K=2$ on CIFAR10, directly followed by $H=0.7$ with $K=4$ on MNIST and the $K=2$ on CIFAR10. The super-diffusive regime performs on MNIST $K\\geq 3$ in $4$ out of $6$ configurations better than both purely Brownian driven baseline dynamics and on CIFAR10 for $3$ out of $4$ configurations with $K\\leq2$. The attached Figure 2 indicates that there are dataset and dynamics specific configuration clusters, where GFDM performs better than the purely Brownian driven models. \n\n>Why for $H=1/2$ do we see improvement with the augmented dynamics? Because of the correlation structure?\n\nYes, our hypothesis is that the known part of the score function is guiding the data generating process towards the direction of the true data distribution, since the starting distribution of the augmenting processes is known and all processes are driven by the same random path realization of Brownian motion. In the forward process $\\mathbf{Y}_t$ does not depend on $\\mathbf{X}_t$, but in the reverse model $\\mathbf{Y}_t$ does depend on $\\mathbf{X}_t$ since it depends on every entry on the score model evaluation $s_{\\theta}(\\mathbf{X}_t,t)$. Assume that $\\mathbf{Y}_t$ does not leave its true reversed trajectory, in this case the trajectory of $\\mathbf{Y}_t$ might serve as a corrector for the trajectory of $\\mathbf{X}_t$.\n\n>The diffusion is run on $[\\epsilon, T]$ but the value of $\\epsilon$ is nowhere to be found it seems, and no experiment to see its effect on GFDM's performance?\n\nThanks for noting. We defined $\\epsilon$ in the Appendix (lines 730-731), but the definition should, of course, be in the main part of the paper. We have added this.\n\n>Line 83-84: $\\hat f$ seems not to be properly introduced\n\nThanks for the detailed review. Since we have $\\mathbf{f}(\\mathbf{x},\\cdot):[0,T]\\to\\mathbb{R}^{D}$ for fixed $\\mathbf{x}\\in\\mathbb{R}^{D}$ we intended to define $\\mathbf{\\overline{f}}(\\mathbf{x},t):=\\mathbf{f}(\\mathbf{x}_t,T-t)$\nas well in line 80-81: \n\nWhenever $\\mathbf{X}=(\\mathbf{X}_{t})_{t\\in[0,T]}$ is a stochastic process and $g$ is a function on $[0,T]$, we write $\\mathbf{X}_{t}=\\mathbf{X}_{T-t}$ for the reverse time model and $\\bar{g}(t)=g(T-t)$ for the reverse time function.\n        \nTo make it more clear, we replaced line 80-81 by:\n\nWhenever $\\mathbf{P}=(\\mathbf{P}_{t})_{t\\in[0,T]}$ is a stochastic process and $f$ is a function on $[0,T]$, we write $\\mathbf{P}_{t}=\\mathbf{P}_{T-t}$ for the reverse time model and $\\bar{f}(t)=f(T-t)$ for the reverse time function.}\n\n>Line 168-169, eq (9): typo in the denominator of the last fraction?\n\nThanks or finding this typo, which we incorporated.\n\n>Line 255-258: Why would using EMA on the models interfere with the SDE dynamics?\n\nSong et. al 2021 point out at that \"For models trained with VE perturbations, we notice that $0.999$ works better than $0.9999$, whereas for models trained with VP perturbations it is the opposite. We therefore use an EMA rate of $0.999$ and $0.9999$ for VE and VP models respectively.\" Due to this empirical observation, the EMA decay rate seems to have a different effect on different underlying dynamics. Since we do not have enough compute to investigate the optimal EMA decay rate for every configuration $(H,K)$ we investigated instead on CIFAR10 both settings, training with and without EMA. \n\n> Line 282: Did you not mean SOTA FID of 0.72, K = 3 and H = 0.9? What are you basing your subsequent interpretation of easier to learn smoother sample paths on?\n\nThis is absolutely right, thank you for noting. Our hypothesis here is that we have a smaller discretization error from continuous to discrete time, when the distribution transforming path is smoother. This is based on the observation that the discretization error of approximating the (rough path driven) SDE with Euler-Maruyama method results in a discretization error of order $N^{-\\frac{1}{2}}$, while the discretization error of the Euler-method to simulate the (smooth path $t\\mapsto t$ driven ) ODE is of order $N^{-1}$. We describe this in more detail in lines 95-112. We have changed the paragraph in lines 280-284 and rephrased it as a conjecture:\n\nBy varying the Hurst index we observe in Table 2a that $H>0.5$ with FVP dynamics performs better in terms of FID achieving a SOTA FID of $0.72$ for FVP with $(H,K)=(0.9,3)$. Moreover, the configurations $(H,K)=(0.9,4),(0.7,4),(0.7,5)$ in the regime of $H>0.5$ perform all better than the purely Brownian dynamics VE and VP. We conjecture that this is due to the super-diffusion regime, smoothing the sample paths, making the dynamics easier to learn. The augmenting processes increase the pixel-wise diversity in terms of $VS_p$ and $VS^{min}_p$ in Table 2a as well for $H\\in\\{0.9,0.7\\}$ compared to the original VP dynamics, again at the cost of a higher NLLs for more augmenting processes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers and Respected Area Chair,

We would like to express our gratitude for your feedback. 

We appreciate the reviewers' acknowledgement of the novelty of our method for the controlled use of fractional Brownian motion in diffusion models. We are also grateful for the appreciation reviewers shared for the clarity of our mathematical framework and the techniques underlying our method, such as the continuous reparameterization trick or augmented score matching to efficiently learn the score-function, that lead to tractable learning and inference, a D-dimensional score model, and marginal statistics for faster training, among other things.

Reviewers across the board requested a clearer concentration of the empirical takeaways. To address this concern, we restructured our section on experiments to give a complete overview of the ablations and configurations that we ran to benchmark the performance of our framework. 

We replace Table 1 and Table 2 in our paper with Table 1 from the attached pdf, where we retrained our model on CIFAR10 on all configurations that we already investigated on MNIST. Comparing the two tables, we see a similar behaviour of GFDM accross the two datasets: 

In the super-diffusive regime ($H>0.5$) we observe for $H=0.9$ on MNIST and and CIFAR10 the best performance in terms of FID providing on CIFAR10  as well a higher pixel-wise diversity compared to the purely Brownian driven dynamics.

In the attached Figure 1 we observe a similar pattern across different datasets and dynamics, indicating that more than one augmenting process yields the best performance. 

Evaluating the performance with different number of sampling steps in Figure 3a) shows that the super-diffusive regime of MA-fBM saturates at 500 NFE on a lower level than the purely Brownian driven dynamics.

Additionally we give a quantitative evaluation of the computational trade-off comparing the average compute per training and sampling step of traditional diffusion models an GFDM ind Figure b) and Figure d).

Finally, answers to detailed questions can be found in the responses to each of your reviews.

We thank you again for your suggestions. Please let us know if you have any further questions.

Meta-review: The paper proposes an extension of diffusion models where the Brownian motion driving the noise process is replaced by fractional Brownian motion. However, this introduces several challenges due to the non-Markovian nature of fractional Brownian motion. To address this problem, the paper suggests approximating this process using a linear combination of Ornstein–Uhlenbeck processes driven by the same Brownian motion. Experiments on standard image dataset benchmarks illustrate the method's behavior for different Hurst exponent parameters and demonstrate good performance.

Most of the reviewers and myself initially found the paper borderline. However, the authors addressed most of the concerns raised by the reviewers during the rebuttal. I trust they will incorporate the reviewers' feedback and the clarifications provided during the rebuttal into the final version of the paper.
Decision: Accept (poster)