Generative Fractional Diffusion Models

Authors: gabriel  nobis
maximilian  springenberg
marco  aversa
michael  detzel
fraunhofer  hhi
roderick  murray-smith
nakajima  shinichi
t u bifold
riken  berlin
  aip
sebastian  lapuschkin
stefano  ermon
tolga  birdal
manfred  opper
christoph  knochenhauer
luis  oala
dotphoton  wojciech
samek  fraunhofer
hhi  tu
berlin  bifold
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 00:38:03

Abstract: We introduce the first continuous-time score-based generative model that leverages fractional diffusion processes for its underlying dynamics. Although diffusion models have excelled at capturing data distributions, they still suffer from various limitations such as slow convergence, mode-collapse on imbalanced data, and lack of diversity. These issues are partially linked to the use of light-tailed Brownian motion (BM) with independent increments. In this paper, we replace BM with an approximation of its non-Markovian counterpart, fractional Brownian motion (fBM), characterized by correlated increments and Hurst index $H \in (0,1)$, where $H=0.5$ recovers the classical BM. To ensure tractable inference and learning, we employ a recently popularized Markov approximation of fBM (MA-fBM) and derive its reverse-time model, resulting in *generative fractional diffusion models* (GFDM). We characterize the forward dynamics using a continuous reparameterization trick and propose *augmented score matching* to efficiently learn the score function, which is partly known in closed form, at minimal added cost. The ability to drive our diffusion model via MA-fBM offers flexibility and control. $H \leq 0.5$ enters the regime of *rough paths* whereas $H>0.5$ regularizes diffusion paths and invokes long-term memory. The Markov approximation allows added control by varying the number of Markov processes linearly combined to approximate fBM. Our evaluations on real image datasets demonstrate that GFDM achieves greater pixel-wise diversity and enhanced image quality, as indicated by a lower FID, offering a promising alternative to traditional diffusion models

Main paper: 
 introduction: 
 Recent years have witnessed a remarkable leap in generative diffusion models [1, 2, 3] , celebrated for their ability to accurately learn data distributions and generate high-fidelity samples. These models have made significant impact across a wide spectrum of application domains, including the generation of complex molecular structures [4] for material [5] or drug discovery [6] , realistic Figure 1 : Each data dimension transitions to a known prior distribution through a forward process that approximates a fractional diffusion process. The Hurst index H on the LHS interpolates between the roughness of a Brownian driven SDE and the underlying integration in PF ODEs. The driving noise process is a linear combination of the correlated processes on the RHS, all driven by the same Brownian motion. The score function of these augmenting processes is available in closed form and serves as guidance for the unknown score function.audio samples [7, 8] , 3D objects [9, 10] or textures [11] , medical images [12] , aerospace applications [13] , and DNA sequence design [14, 15] . Despite these successes, modern score-based generative models formulated in continuous-time [16] face limitations due to their reliance on a simplistic driving noise, the Brownian motion (BM) [17, 18, 19] . As a light-tailed process, using BM can results in slow convergence rates and susceptibility to mode-collapse, especially with imbalanced data [20] . Additionally, its purely Markovian nature may also make it hard to capture the full complexity and richness of real-world data. All these attracted a number of attempts for involving different noise types [20, 21] . In this paper, we propose leveraging fractional noises, particularly the renowned non-Markovian fractional BM (fBM) [22, 23] to drive diffusion models. fBM extends BM to stationary increments with a more complex dependence structure, i.e., long-range dependence vs. roughness/regularity controlled by a Hurst index, a measure of "mild" or "wild" randomness [24] . This all comes at the expense of computational challenges and intractability of inference, mostly stemming from its non-Markovian nature. Moreover, deriving a reverse-time model poses theoretical challenges, as fBM is not only non-Markovian but also not a semimartingale [25] . To overcome these limitations, we leverage recent works in Markov approximations of fBM (MA-fBM) [26, 27] and establish a framework for training continuous-time score-based generative models using an approximate fractional diffusion process, as well as generating samples from the corresponding tractable reverse process. Notably, our method maintains the same number of score model evaluations during both training and data generation, with only a minimal increase in computational load. Our contributions are:• We derive the time-reversal of forward dynamics driven by a Markovian approximation of fractional Brownian motion in a way that the dimensionality of the unknown part of the score function matches that of the data. • We derive an explicit formulae for the marginals of the conditional forward process via a continuous reparameterization trick. • We introduce a novel augmented score matching loss for learning the score function in our generative fractional diffusion model, which can be minimized by a score model of data-dimension. Our experimental evaluation validates our contributions, demonstrating the gains of correlated-noise with long-term memory, approximated by a combination of a number of Markov processes, where the amount of processes further control the diverstiy.Differentiation from existing work. Yoon et al. [20] generalizes score-based generative models from an underlying BM to a driving Lévy process, a stochastic process with independent and stationary increments. A driving noise with correlated increments is not included in the framework of Yoon et al. [20] . Conceptually, every Lévy process is a semimartingale [28] . Since fBM is not a Lévy process, it is not included in the framework of Yoon et al. [20] . The closest work to ours is Tong et al. [29] constructing a neural-SDE based on correlated noise and using the neural SDE as a forward process of a score-based generative model. Our framework with exact reverse-time model is based on the integral representation of fBM derived in Harms and Stefanovits [26] and the optimal approximation coefficients of Daems et al. [27] , while the fractional noise in [29] is sparsely approximated by a linear combination of independent standard normal random variables without exact reverse-time model. Moreover, the framework of Tong et al. [29] is limited to H > 1 3 and only compatible with the Euler-Maruyama sample schema [30] while our framework is up to numerical stability applicable for any H ∈ (0, 1) and compatible with any suitable SDE or ODE solver. To the best of our knowledge, we are the first to build a framework for continuous-time score-based generative models that includes driving noise processes converging to non-Markovian processes with infinite quadratic variation. 
background: 
 Modeling the distribution transforming process of a score-based generative model through stochastic differential equations (SDEs) [16] offers a unifying framework to generate data from an unknown probability distribution. Instead of injecting a finite number of fixed noise scales via a Markov chain, infinitely many noise scales tailored to the continuous dynamics of the Markov process X = (X t ) t∈[0,T ] are utilized during the distribution transformation, offering considerable practical advantages over discrete time diffusion models [16] . The forward dynamics, transitioning from a data sample X 0 ∼ p 0 to a tractable noise sample X T ∼ p T are specified by a continuous drift function f and a continuous diffusion coefficient g. These dynamics define a diffusion process that solves the SDE dX t = f (X t , t)dt + g(t)dB t , X 0 ∼ p 0 (1) driven by a multivariate BM B. To sample data from noise, a reverse-time model is needed that defines the backward transformation from the tractable noise distribution to the data distribution. Whenever X = (X t ) t∈[0,T ] is a stochastic process and g is a function on [0, T ], we write X t = X T -t for the reverse-time model and ḡ(t) = g(Tt) for the reverse-time function. The marginal density of the stochastic process X at time t is denoted by p t throughout this work 2 . Remarkably, an exact reverse-time model to the forward model in eq. ( 1) is given by the backward dynamics [31, 32, 33 ]EQUATIONwhere the only unknown is the score function ∇ x log p t , inheriting the intractability from the unknown initial distribution p 0 . In addition to the stochastic dynamics, the reverse-time model provides deterministic backward dynamics via an ordinary differential equation (ODE) by the so called probability flow ODE (PF ODE) [16] dx t = f (x t , t) -1 2 ḡ2 (t)∇ x log pt (x t , t) dt, x T ∼ p T .Stochasticity is only injected into the system through the random initialization x T ∼ p T , implying a deterministic and bijective map from noise to data [16] . Conditioning the forward process on a data sample x 0 ∼ p 0 results for linear f (•, t) in a tractable Gaussian forward process with conditional score function ∇ x log p 0t (x|x 0 ) in closed form. To approximate the exact reverse-time model, this tractable score function is used to train a time-dependent score model S θ via score matching [34, 35] .Upon training, any solver for SDEs or ODEs can be utilized to generate data from noise by simulating the stochastic or deterministic backward dynamics of the reverse-time model with S θ ≈ ∇ x log p.Simulation error of the reverse-time model. The two main sources of error when simulating the reverse-time model are the approximation error due to S θ only approximating ∇ x log p, and the discretization error, which arises from transitioning from continuous-time to discrete steps. Simulating the PF ODE with the Euler method over N ∈ N equidistant time steps results in a global error of order N -1 [36] . In contrast, the expected global error for simulating the SDE using the Euler-Maruyama method is of a lower order N -1 2 , indicating a larger error for the same number of steps [30, 36] . From this perspective it is reasonable that sampling from the PF ODE requires fewer steps. Yet, the source of qualitative differences between sampling from the ODE and the SDE [16] remains unclear.A pathwise perspective on sampling. The roughness of a path can be measured by its Hölder exponent 0 < δ ≤ 1 [37] . For example, BM as the integrator in the backward dynamics eq. ( 2) has δ-Hölder continuous paths for any 0 < δ < 1 2 , whereas the integrator t → t of the PF ODE eq. ( 3) can be regarded as a Hölder continuous path with exponent δ = 1. Therefore, from a pathwise perspective, we move away from a rough path when we sample using the PF ODE. An unexplored topic in score-based generative models is the interpolation between the SDE and the PF ODE in terms of the Hölder exponent. It remains to be examined whether there is, to some extent, an optimal degree of Hölder continuity in between, or if an even rougher path with δ ≪ 1 2 could yield an advantageous data generator.The process that naturally arises from this line of thought is fBM [22, 23] with Hurst index H ∈ (0, 1), where almost all paths are Hölder continuous for any exponent δ < H, controlled by H. In terms of roughness, the Hurst index interpolates between the paths of Brownian driven SDEs and those of the underlying integration in PF ODEs, while also offering the potential for even rougher paths. Motivated by these observations, we define a novel score-based generative model with underlying dynamics that approximate a fractional diffusion process.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We include a section on limitations of our work where we discuss the limitations of our results.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
fractional driving noise: 
 Before describing the challenges in defining a score-based generative model with control over the roughness of the distribution transforming path, we introduce fBM. The literature distinguishes between "Type I" fBM and "Type II" fBM [38] having stationary and non-stationary increments, respectively. The type II fBM, also called Riemann-Liouville fBM, possesses smaller deviations from its mean, potentially an advantageous property for a driving noise of a score-based generative model, since large deviations of the sampling process to the data mean can lead to sample artifacts [39] . Here and in the experiments we focus on type II fBM. However, our theoretical framework generalizes to both types as detailed in Appendix A. The empirical study of a score-based generative model approximating a fractional diffusion process driven by type I fBM is dedicated to future work. We begin with the definition of Riemann-Liouville fBM [22] , a generalization of BM permitting correlated increments. Definition 3.1 (Type II Fractional Brownian Motion [22] ). Let B = (B t ) t≥0 be a standard Brownian Motion (BM) and Γ the Gamma function. The centered Gaussian processEQUATIONuniquely characterized in law by its covariancesEQUATIONEQUATIONis called type II fractional Brownian motion (fBM) with Hurst index H ∈ (0, 1).BM being the unique continuous and centered Gaussian process with covariance min{t, s} is recovered for H = 0.5, since Γ(1) = 1. In comparison to the purely Brownian setting with independent increments (diffusion), the path of B H becomes more smooth for H > 0.5 due to positively correlated increments (super-diffusion) and more rough for H < 0.5 due to negatively correlated increments (sub-diffusion). These three regimes are reflected in the Hölder exponent of δ < H for almost all paths.Generalization challenges. The most challenging part in defining a score-based generative model driven by fBM is the derivation of a reverse-time model. Due to its covariance structure, fBM is not a Markov process [40] and the shift in the roughness of the sample path leads to changes in its quadratic variation: from t in the purely Brownian (diffusion) regime to zero in the smooth regime, and to infinite in the rough regime [30] . For that reason fBM is neither a Markov process nor a semimartingale [25] for all H ̸ = 0.5. Hence, we cannot make use of the Markov property or the Kolmogorov equations (Fokker-Planck) that are used to derive the reverse-time model of Brownian driven SDEs [31, 32, 33] . See Appendix H for a more illustrative view of the problem. The existence of a reverse-time model can be proven in the smooth regime of fBM [41] . However, due to the absence of an explicit score function in Darses and Saussereau [41] it does not provide a sufficient structure to train a score-based generative model.To overcome this difficulty we follow [26, 27] and define the driving noise of our generative model by a linear combination of Markovian semimartingales. The approximation is based on the exact infinite-dimensional Markovian representation of fBM given in Theorem A.2. Definition 3.2 (Markov approximation of fBM [26, 27] ). Choose K ∈ N Ornstein-Uhlenbeck (OU) processesEQUATIONwith speeds of mean reversion γ 1 , ..., γ K and dynamics dY k t = -γ k Y k t dt + dB t . Given a Hurst index H ∈ (0, 1) and a geometrically spaced grid γ k = r k-n with r > 1 and n = K+1 2 we call the processEQUATIONMarkov-approximate fractional Brownian motion (MA-fBM) with approximation coefficients ω 1 , ..., ω K ∈ R and denote by BH = ( BH 1 , ..., BH D ) the corresponding D-dimensional process where BH i and BH j are independent for i ̸ = j inheriting independence from the underlying standard BMs B i and B j .Our framework is conceptually independent of the specific choice of spatial grid and approximation coefficients. To achieve strong convergence rates with a high polynomial order in K for H < 0.5 in the driving noise to fBM, one may follow the approach outlined in Harms [42] . Consequently, our framework includes driving noise processes that converge to non-Markovian processes with infinite quadratic variation. For computational efficiency, we instead follow the approach of Daems et al. [27] to choose the L 2 (P) optimal approximation coefficients for a given K, achieving empirically good results in approximating fBM, even with a small number of OU processes. Proposition 3.3 (Optimal Approximation Coefficients [27] ). The optimal approximation coefficients ω = (ω 1 , ..., ω K ) ∈ R K for a given Hurst index H ∈ (0, 1), a terminal time T > 0 and a fixed geometrically spaced grid to minimize the L 2 (P)-errorEQUATION)are given by the closed-form expression Aω = b withEQUATIONand where P (z, x) = 1 Γ(z)x 0 t z-1 e -t dt is the regularized lower incomplete gamma function.MA-fBM serves as the driving noise of our generative model, replacing BM in the distribution transforming process solving eq. ( 1), approximating a fractional diffusion process. See Figure 1 for an illustration of the underlying processes.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: We release our code upon publication. Together with the implementation details given in the paper our results can be reproduced.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
a score-based generative model based on fractional noise: 
 In this section, we define a continuous-time score-based generative model driven by MA-fBM. A detailed treatment of the theory can be found in Appendix A. We begin with the forward dynamics, transitioning data to noise.Definition 4.1 (Forward process). Let BH be a D-dimensional MA-fBM with Hurst index H ∈ (0, 1).For continuous functions µ : [0, T ] → R and g : [0, T ] → R we define the forward process X = (X t ) t∈[0,T ] of a generative fractional diffusion model (GFDM) byEQUATIONwhere p 0 is the unknown data distribution from which we aim to sample from.Considering both the forward process as well as the OU processes defining the driving noise BH , we have for every data dimension an augmented vector of correlated processes (X, Y 1 , . . . , Y K ), driven by the same BM, approximating the time-correlated behavior of a one-dimensional fractional diffusion process [27] . We denote the stacked process of the D augmented vectors as Z = (Z t ) t∈[0,T ] and refer to the resulting D(K +1)-dimensional process as the augmented forward process. Rewriting the dynamics of the forward process we observe that the augmented forward process Z solves a linear SDE. Hence, Z|x 0 , the augmented forward process conditioned on a data sample x 0 ∼ p 0 , is a linear transformation of BM. Thus Z|x 0 is a Gaussian process and so is X|x 0 [43] . For each dimension 1 ≤ d ≤ D, we have a system of K + 1 trajectories that transform x 0,d according to the augmented forward process with D = 1, following the dynamicsEQUATIONwhere all K + 1 processes are driven by the same one-dimensional BM B with matrix valued functions F and G defined in Appendix A.2. To efficiently sample for every t ∈ (0, T ] from the conditional augmented forward distribution during training, we characterize its marginal statistics.Derivation of marginal statistics. The marginal mean E[X t |x 0 ] = x 0 exp( t 0 µ(s)ds) of the conditional forward process is unaffected by changing the driving noise to MA-fBM, and the mean of the augmenting OU processes is zero. See Appendix A.2 for a detailed derivation of the marginal statistics of the augmenting processes. The missing components in the marginal covariance matrix Σ t of the conditional augmented forward process Z|x 0 are the marginal variance of the forward process and the marginal correlation between the conditional forward process and the augmenting processes. We derive by reparameteriziation an explicit formula for the marginal variance of the conditional forward process. This generalizes the formula for the perturbation kernel p 0t (x|x 0 ) = N (x; c(t)x 0 , c 2 (t)σ 2 (t)I D ) given in Karras et al. [44] to a driving MA-fBM and is reminiscent of the reparameterization trick used in discrete time. Proposition 4.2 (Continuous Reparameterization Trick). The forward process X of GFDM conditioned on x 0 ∈ R D admits the continuous reparameterizationEQUATIONwith c(t) = exp t 0 µ(s)ds and σ 2 (t) = t 0 α 2 (t, s)ds where α is given byEQUATIONSketch of Proof. Reparameterization of the forward dynamics in eq. ( 10) and the Stochastic Fubini Theorem yields the Gaussian process By the above definition of α, we retrieve the perturbation kernel of the purely Brownian setting given in Karras et al. [44, Equation 12] for K = 1, γ 1 = 0 and ω 1 = 1 . When, depending on the choice of forward dynamics, t 0 α(t, s)ds is not accessible in closed form, Σ t can be described by an ODE and solved numerically as described in Appendix B. Thus our method admits any choice of forward dynamics in terms of µ and g.X t = c(t)(x 0 + t 0 α(t, s)dB s ) with variance V [X t ] = c 2 (t)Explicit fractional forward dynamics. Although our framework is not bound to any specific dynamics, this work's empirical evaluation focuses on Fractional Variance Exploding (FVE) dynamics given byEQUATIONwith (σ min , σ max ) = (0.01, 50) and Fractional Variance Preserving (FVP) dynamics given byEQUATIONwith β(t) = β(t) = βmin + t βmaxβmin and ( βmin , βmax ) = (0.1, 20) [16] . Leveraging the continuous reparameterization trick we derive in Appendix B the conditional marginal covariance matrix of FVE in closed form. To the best of our knowledge, the integral in eq. ( 13), needed to compute α in the setting of FVP dynamics, is not accessible in closed form. Therefore, we use a numerical ODE solver to estimate this quantity for FVP dynamics. See Appendix B for details on the computation of the marginal variances and an illustration of the resulting variance schedules.The reverse-time model. We observe that the augmented forward dynamics of GFDM are already encompassed in the general framework presented in Song et al. [16, Appendix A], although they differ from the Variance Exploding (VE), Variance Preserving (VP), and sub-VP dynamics discussed therein. To simplify notation, we use p t here to denote the marginal density of both Z t and X t . The specific density referred to will be clear from the context. By the significant results of [31, 32, 33] , the reverse-time model of GFDM is given by the backward dynamicsEQUATIONHowever, a direct application of [16] would require to train a score model with input and output dimension of D(K + 1). By proposing augmented score matching below, we show that learning a score model with input and output dimension D is sufficient, enabling the use of the same highly curated model architecture as in traditional diffusion models to approximate the score function.Augmented score matching. We condition the score function ∇ z log p t on a data sample x 0 ∼ p 0 and additionally on the states of the stacked vector Y[K] t := (Y 1 t , ..., Y K t ) of augmenting processes. To train our time-dependent score model s θ we propose the augmented score matching lossEQUATION)The weights η 1 t , ..., η K t arise from conditioningZ t |x 0 on Y [K] tand the time points t are uniformly sampled from U[0, T ]. We show in the following that the optimal s θ w.r.t. the augmented score matching loss is the L 2 -optimal approximation of the score function of our reverse-time model. Proposition 4.3 (Optimal Score Model). Assume that s θ is optimal w.r.t. the augmented score matching loss L. The score modelEQUATIONyields the optimal L 2 (P) approximation of ∇ z log p t (Z t ) viaEQUATIONSketch of Proof. Using the relation ∇ x log p 0t = -η k t ∇ y k log p 0t and the independence of X 0 and Y[K] t yields the claim. See Appendix A.3 for the full proof.In addition to the result that a score model of data dimension D minimizes the proposed augmented score matching loss, Proposition 4.3 also implies that GFDM requires the same number of score model evaluations during sampling from the reverse-time model as traditional diffusion models. This is because, for a given time point t, we only need to evaluate s θ (•, t) once at X tk η k t Y k t to compute S θ (Z t , t) according to eq. ( 18), and S θ is all that is required to approximate the reverse-time dynamics described below. We provide a thorough quantitative evaluation of compute time in seconds for GFDM in Appendix F, validating the theoretical reasoning in this section that GFDM incur only minimal additional computational cost.Sampling from reverse-time model. Once we trained our score model S θ via augmented score matching, we simulate the reverse-time model backward in time and sample from the reverse-time model via the SDEdZ t = F(t)Z t -G(t)G(t) T S θ (Z t , t) + ∇ z log q t (Y [K] t ) dt + G(t)dB t , t ∈ [0, T ] (20)or the corresponding augmented PF ODE [16] EQUATIONwhere we initialize in both cases the reverse dynamics with the centered (non-isotropic) Gaussian Z 0 with covariance matrix Σ T . To traverse backward from noise to data, we may deploy any suitable SDE or ODE solver. In both cases, for each data dimension, we have K + 1 trajectories that transform the Gaussian initialization into an approximate sample of the data distribution. The PF ODE enables in addition negative log-likelihoods (NLLs) estimation of test data under the learned density [16] . See Appendix G for the computation details of NLLs. Remark 4.4. We showed in this section that it suffices to approximate a D-dimensional score to reverse the D(K + 1)-dimensional MA-fBM driven SDE with unknown starting distribution. Since this holds for any fixed K ∈ N an interesting task is to examine the behaviour of the reverse-time model as K → ∞ and potentially link it to the dynamics of a reverse-time model of true fBM. To the best of our knowledge, existence of such a reverse-time model is not known for H < 0.5 and the drift of the reverse-time model for H > 0.5 lacks sufficient structure to train a score-based generative model [41] . MNIST H = 0.9 CIFAR10 H = 0.9 Table 2 : FID and pixel-wise diversity VS p of GFDM compared to the original setting of purely Brownian driven VE and VP. In bold the scores that are better than both purely Brownian driven dynamics. The overall best scores within the experiment are boxed in, indicating that the highest scores on both datasets are achieved in the super-diffusive regime for H = 0.9.EQUATIONEQUATION 
experiments: 
 We conduct experiments on MNIST and CIFAR10 to evaluate the ability of GFDM to generate real images. First, we measure the quality and the pixel-wise diversity of the generated images across different numbers of augmenting processes and various Hurst indices, showing that the super-diffusive regime with H > 0.5 yields better performance compared to the purely Brownian driven dynamics. Second, we further evaluate the best performing models in terms of class-wise image quality and class-wise distribution coverage. We measure image quality by the Frechét Inception Distance (FID) [45] and the Inception score (IS) [46] , pixel-wise diversity by the pixel Vendi Score (VS p ) [47] and class-wise distribution coverage by improved recall (Recall) [48] . See Appendix D for the implementation details and additional experimental results. We begin with the empirical evaluation of how the augmenting processes affect performance on MNIST.To isolate the effect of the augmenting processes on MNIST while minimally adapting the driving noise distribution, we fix H = 0.5 so that the weighted sum of the augmenting processes approximates BM, rather than fBM. We observe an increase of the pixel-wise diversity VS p for both FVE and FVP dynamics, with increasing K. In Table 1 we can observe that VS p increases from 24.20 to 24.54 for FVE dynamics and from 23.64 to 24.56 for FVP dynamics. The enhanced pixel-wise diversity on MNIST comes at the cost of a reduced likelihood of test data under the learned density, indicated by a higher NLLs for more augmenting processes.Quality results across different Hurst indices. On both, MNIST and CIFAR10, we obtain the best performance in terms of FID and VS p in the super-diffusive regime with H = 0.9 and FVP dynamics. On MNIST we achieve state of the art FID of 0.72, compared to an FID of 1.44 with the purely Brownian VP dynamics (Table 2a ). Comparing FVP to the best-performing purely BM driven VP dynamics, we observe not only an improvement in quality but also an increase in pixel-wise diversity from 23.64 to 24.18, as measured by VS p . In Table 2b we observe the same behaviour on CIFAR10. The best performing configuration in terms of FID and pixelwise diversity is achieved for FVP(H = 0.9, K = 2) with an FID of 3.77 instead of 4.85 and an VS p of 3.60 instead of 3.28. Additionally, in Figure 7 , we show the FID evolution of the super-diffusive regime for various numbers of augmenting processes, showing a similar pattern that either that K = 2 or Table 3 : The class-wise image quality and class-wise distribution coverage of the super-diffusive regime FVP(H = 0.9, K = 2) compared to the purely Brownian VP dynamics. Class-wise distribution coverage. We evaluate the capability to generate samples from different classes in terms of FID and class-wise distribution coverage, measured by Recall, comparing the best-performing purely Brownian driven dynamics to the super-diffusive regime with K = 2. In Table 3 we observe that the super-diffusive regime with K = 2 outperforms in both FID and Recall, where H = 0.7 and H = 0.9 achieve better class-wise FID for all but two and one class, respectively (deer and dog for H = 0.7, bird for H = 0.9). Additionally, the super-diffusive regime shows improved class-wise distribution coverage, as indicated by a higher Recall across all classes. Overall, both H = 0.7 and H = 0.9 perform significantly better in terms of distribution coverage than VP dynamics, H = 0.9 being the best performing model.Sampling with the augmented probability flow ODE. We compare the performance of sampling via the PF ODE for the best performing models from above. For MA-fBM driven dynamics, we have K + 1 deterministic trajectories for each pixel, traversing from noise to data. As shown in Figure 3 , the PF ODE associated with purely Brownian dynamics outperforms the super-diffusive regime in terms of FID, while the superdiffusive regime achieves the overall highest pixel-wise diversity of VS p = 4.89 confirmed mildly perceptually in Figure 4 . See Appendix E for additional visualization of the generated data.Our experiments show that, compared to purely Brownian dynamics, the super-diffusive regime of MA-fBM yields higher image quality with fewer NFEs, improved pixelwise diversity and better distribution coverage. 
related work: 
 Diffusion models in continuous-time. The seminal work of Song et al. [16] offers a unifying framework modeling the distribution transforming process by a stochastic processes in continuoustime with exact reverse-time model. Extensive research has been carried out to examine [44, 49, 50] and extend [39, 51, 52, 53, 54, 55] the continuous-time view on generative models through the lens of SDEs, including deterministic corruptions [56] and blurring diffusion [57] . While critic on this view question the usefulness of the theoretical superstructure [58] , others extend in line with our work the theoretical framework to new types of underlying diffusion processes [59] . Conceptually similar to our work,Yoon et al. [20] generalizes the score-based generative model from an underlying Brownian motion to a driving Lévy process, thereby dropping the Gaussian assumptions on the increments. In contrast to our work, the framework of Yoon et al. [20] does not include correlated increments. Importantly, every Lévy process is a semimartingale, which means that fBM is not a Lévy process.Fractional noises in machine learning. Recently, Hayashi and Nakagawa [60] Rough path theory. The pathwise analysis of SDEs driven by processes with a Hölder exponent less than 0.5, including fBM for H < 0.5 and BM, is encompassed by rough path theory [37] . Rough path theory is applied in machine learning in several ways including (i) deriving stability bounds for the trained weights of a residual neural network [61] , (ii) enabling rough control of neural ODEs [62] , and (iii) modeling long time series behavior via neural rough differential equations [63, 64] . In finance the famous Black-Scholes model [65] is driven by BM, while more recent continuous-time models employ fractional noise to model price processes [66, 67] or rough volatility [68, 69] to more closely mimic real-world behavior.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We describe the hyperparameters in our section on implementation details in the appednix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 In this work, we propose a generalized framework of continuous-time score-based generative models, introducing a novel generative model driven by MA-fBM with control over the roughness of distribution transformation paths via augmenting processes. Despite the increased dimensionality of the forward process, learning a score model with the dimensionality of the data distribution, guided by the marginal known score of the augmenting processes, is sufficient. Consequently, both training and sampling is efficient. Our experimental results show that the super-diffusive regime of our MA-fBM driven dynamics achieves superior performance in terms of FID and pixel-wise diversity. Additionally, the FID saturates at a lower level with fewer function evaluations compared to purely Brownian driven dynamics. The super-diffusive regime also improves class-wise distribution coverage, as measured by Recall. Based on these results, GFDM offers a promising alternative to traditional diffusion models for generating data from an unknown distribution.Limitations and future work. Several practical and theoretical questions remain open. While we draw our conclusions from experiments conducted on MNIST and CIFAR10, generalizing the observed behavior to other datasets and data modalities may not be valid. In future work, we aim to empirically and theoretically determine the optimal degree of correlated noise, and thus the optimal Hurst index, for training and sampling across different data modalities. Beyond image data, a particularly interesting modality could be the generation of rough time series data using dynamics of the sub-diffusive regime. A theoretical open question is the limiting behavior of GFDM's reverse dynamics with infinitely many augmenting processes and whether this limit is connected to the reverse time model for true fBM. An intriguing extension would be to adapt the dynamics of our framework to switch between two unknown distributions. This adaptation would enable the use of MA-fBM driven dynamics in the sciences to model real-world evolution between two states of unknown distributions. This is a promising direction, as the assumption of independent increments in real-world noise processes is often too strong.Broader impact. Our contribution advances generative modeling by introducing a specific driving noise process to improve the learning of an unknown distribution. This conceptual work aims to support impactful applications of generative modeling, such as molecular structure generation, medical imaging, drug discovery, and DNA sequence design. However, we acknowledge that generative models can reflect biases in the datasets they are trained on and may pose risks, including misuse for human impersonation and the spread of fake content.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Unfortunately we don not have the computational resources to run all experiments a sufficient number of times to provide statistical certainty.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We include a broader impact statement at the end of our paper discussing potential misuse. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We only feature experiments on small scale dataset up to size 3 × 32 × 32.The models trained in this work do not pose such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We properly cite all research works we build on and use the code of others only according to its license.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?[Yes] Justification: We release our code upon publication alongside a proper documentation under the MIT license. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this section we provide the mathematical details of the score-based generative model defined in the main paper. The driving noise of the underlying stochastic process is based on the affine representation of fractional processes from Harms and Stefanovits [26] and further simplified by the closed form expression to determine optimal approximation coefficients of Daems et al. [27] .We begin with the definition of type I fractional Brownian motion, defined on the whole real line, possessing correlated increments that are in contrast to type II fractional Brownian motion stationary. Definition A.1 (Type I Fractional Brownian Motion [23] ). Let (Ω, F, P) be a complete probability space equipped with a complete and right continuous filtration {F t } and Γ the Gamma function. For two standard independent {F t }-Brownian motions (BMs) B and B the centered Gaussian processW H = (W H t ) t∈R with W H t := 1 Γ(H + 1 2 ) 0 -∞ ((t -s) H-1 2 -(-s) H-1 2 )d Bs + 1 Γ(H + 1 2 ) t 0 (t -s) H-1 2 dB s (22)uniquely characterized in law by its covariancesEQUATION)is called type I fractional Brownian motion (fBM) with Hurst index H ∈ (0, 1).Type II fBM from the main paper is retrieved by setting the additionally defined BM B on the negative real line to zero. Therefore, the difference to type II fBM is the stochastic integral w.r.t. B that yields stationary increments and a non trivial distribution at t = 0. For H = 0.5, the process is a BM and has thus independent increments. For H ∈ (0, 1) \ { 1 2 }, the process possesses correlated increments and, compared to BM, smoother paths for H > 0.5 due to positively correlated increments (superdiffusion) and rougher paths for H < 0.5 due to negatively correlated increments (sub-diffusion). These three regimes reflect for type I fBM in the same change of quadratic variation from t to zero quadratic variation in the smooth regime and to infinite quadratic variation in the rough regime [30] . To prepare the approximation of the non-Markovian and non-semimartingale fBM [25] via Markovian semimartingales, define for every γ ∈ (0, ∞) the Ornstein-Uhlenbeck process Y γ given byEQUATIONwith speed of mean reversion γ and non trivial starting value in contrast to the OU processes defined in eq. ( 6) of the main paper. By Itô's product rule [30] , the process Y γ solves the same SDEEQUATIONwith different starting value. According to Harms and Stefanovits [26] we represent fBm by an integral over the predefined family of Ornstein-Uhlenbeck processes. Theorem A.2 (Markovian representation of fBM [26, 27] ). The non-Markovian process W H permits the infinite-dimensional Markovian representationEQUATIONwhereEQUATION)Note that we follow Daems et al. [27] in replacing the process Z γ t := Z γ 0 e -tγ + t 0 e -(t-s)γ Y γ s ds from the original theorem throughout this work byZ γ t = -∂ γ Y γ t + (∂ γ Y γ 0 + Z γ 0 ) e -tγ. This is justified by Harms and Stefanovits [26, Remark 3.5] and simplifies for H > 1 2 the approximation of fBM and the definition of our generative model, since we only have to reverse the Y γ processes instead of the pairs (Y γ , Z γ ). For Y γ 0 = 0 eq. ( 26) yields an infinite-dimensional Markovian representation of type II fBM [27] . The MA-fBM from Definition 3.2 in the main paper becomes for type I fBMEQUATIONwith non trivial [27] . Theorem 3.3 holds true for type I fBM as well with optimal approximation coefficients given in Daems et al. [27, Proposition 5] . For more details on the properties and distinction of type I and type II fBM we refer the reader to Daems et al. [27] .Y 0 = (Y 1 0 , ..., Y 1 0 ) that is a centered multivariate Gaussian with covariances E Y k 0 Y l 0 = 1/(γ k + γ l )We define in the following a score-based generative model approximating a fractional diffusion process driven by type I fBM. For the remainder of Appendix A we assume Y k 0 = 0 -∞ e sγ k d Bs for all 1 ≤ k ≤ K where the setting from the main paper with type II fBM is recovered by choosing Y k 0 = 0 instead. Let BH be a D-dimensional MA-fBM with Hurst index H ∈ (0, 1). For continuous functions µ : [0, T ] → R and g : [0, T ] → R we define the forward process X = (X t ) t∈[0,T ] byEQUATIONwhere p 0 is an unknown data distribution from which we aim to sample from. Using eq. ( 25) we noteEQUATIONwhere B = (B 1 , ..., B d ) is a multivariate BM. With ω := K k=1 ω k we rewrite the dynamics of the forward process asEQUATIONTaking into account the dynamics of the OU processes, we define the augmented forward process Z = (Z t ) t∈[0,T ] byEQUATIONfollowing the dynamicsEQUATIONwithEQUATIONandEQUATIONFor each dimension 1 ≤ d ≤ D, the dynamics of the process transforming x 0,d reduce to those of the augmented forward process with D = 1, given byEQUATIONwhere the K + 1 processes that transform x 0,d are all driven by the same one-dimensional BM B.The augmented forward process Z conditioned on y 1 0 , ..., y K 0 and a data sample x 0 ∼ p 0 is a linear transformation of BM and hence a Gaussian process and so is X [43] . Since the integral w.r.t BM has zero mean, the mean vector of the augmenting processes is E Y k t = 0 d for all 1 ≤ k ≤ K and the mean of the conditional forward process is the solution of the ODEEQUATIONand hence the marginal meanEQUATIONis not affected by changing the driving noise to MA-fBM. The marginal covariance matrix Σ t of the conditional augmented forward process can be approximated numerically by solving an ODE, see Appendix B for details. In addition we present a continuous reparameterization of the forward process, resulting for some forward dynamics in a closed form solution of the marginal covariance matrix. Our result generalizes the explicit formula for the perturbation kernel p 0t (x|x 0 ) = N (x; c(t)x 0 , c 2 (t)σ 2 (t)I d ) given in Karras et al. [44] .Proposition A.3 (Continuous Reparameterization Trick). Let x 0 be a fixed realisation drawn from p 0 . The forward process X = (X t ) t∈[0,T ] conditioned on x 0 admits the continuous reparameterizationEQUATIONwith c(t) = exp t 0 µ(s)ds andEQUATIONsuch thatEQUATIONandEQUATIONEQUATIONvanishing for an underlying type II fBM.Proof. By continuity, the functions µ and σ are bounded. Moreover, the processes Y 1 j , ..., Y K j posses continuous, hence bounded, paths and thusEQUATIONwhere the last integral is understood entrywise. Hence, by Cohen and Elliott [30, Theorem 16.6.1], the unique solution of the SDE eq. ( 31) is given explicitly asEQUATIONwith c(t) = expEQUATION)and by the definition of Y k j in (24) we calculate using the Stochastic Fubini Theorem [26] t 0 g(u) c(u)EQUATIONEQUATIONand henceEQUATIONwith EQUATIONTherefore, conditional on x 0 , the random vector X t is Gaussian with mean vectorEQUATIONMoreover, Bj and B j corresponding to the entries of B = ( B1 , ..., Bd ) and B = (B 1 , ..., B d ) are independent by Theorem A.1 resulting in the entrywise varianceEQUATIONwithEQUATIONEQUATIONwhere we used again Itô's isometry to calculateEQUATION)Since the entries of B are independent, we find the covariance matrixEQUATIONThe preceding proposition generalizes the "reparameterization trick"3 from discrete time to continuous-time in the sense thatEQUATIONused in discrete time [2] with time steps 0 = t 0 < ... < t N = T is replaced by our continuous-time reparameterizationEQUATIONenabling to directly sample X t |x 0 ∼ N (c(t)x 0 + c 2 (t)σ 2 (t) + σ 2 K (t) I D ) for a given data sample x 0 and time point t ∈ (0, T ], in case that σ 2 (t) and σ 2 K (t) have a closed form solution. For a complete characterization of the marginal covariance matrix Σ t of the conditioned augmented forward process we calculate by Itô isometry with X = X j and Y l = Y l j for all 1EQUATIONandEQUATION)reducing for type II fBM toEQUATIONWe denote in the following the stacked vector of the augmenting processes byEQUATIONThe random vector Y[K] t is a centered Gaussian process with covariance matrixEQUATION)where Σ y t does not depend on the dimension 1 ≤ j ≤ D and we write q t for the multivariate Gaussian density of Y[K] t . Since we know the distribution of Y[K] 0 , we can directly calculate the corresponding score function byEQUATIONA.3 Estimating the score via augmented score matching loss Conditioning Z t on x 0 ∼ p 0 and a realisation y[K] tof the stacked augmenting processes Y[K] t defined in eq. ( 64) at fixed time t ∈ [0, T ] results in the Gaussian vector Xt ∼ N ( mt , Σt ) with meanEQUATIONand covarianceEQUATIONWe denote with ∇ x log p 0t the conditional score function of Xt and calculate for the gradient w.r.t.EQUATION)and for the gradient w.r.t.EQUATIONDeploying this relation of ∇ x log p 0t and ∇ y k log p 0t we derive the augmenting score matching loss that reduces the dimensionality of the score model we have to learn to the dimensionality of the data distribution and results in a score model guided by the the known score function ∇ y [K] log q t . Proposition A.4 (Optimal Score Model). Assume that s θ is optimal w.r.t. the augmented score matching loss L. The score modelS θ (Zt, t) := s θ (Xt - k η k t Y k t , t), -η 1 t s θ (Xt - k η k t Y k t , t), ..., -η K t s θ (Xt - k ηY k t , t)yields the optimal L 2 (P) approximation of ∇ z log p t (Z t ) viaEQUATIONProof. Fix t ∈ [0, T ]. We write p aug t for the density of Z t , p aug 0t for the conditional density of Z t on X 0 , p 0t for the density of Xt and q 0t for the conditional density of Y[K] t on X 0 . First note that Y [K] tand X 0 are independent by assumption and hence q t = q 0t . By direct calculations we findEQUATIONEQUATION)Hence the best L 2 (P)-approximation of ∇ x log p aug t (Z t ) is a minimizer of the augmented score matching loss byEQUATIONAssume now that s θ is a minimizer of the augmented score matching loss. Similar to the calculation above we haveEQUATION) (70) = -η k t E (X0|Xt,Y [K] t ) ∇ x log p 0t (X t |Y [K] t , X 0 ) + ∇ y k log q t (Y [K] t ) (84) and hence -η k t s θ (X t - k η k t Y k t ) + ∇ y k log q t (Y [K]t ) is the best approximation of ∇ y k log p aug t (Z t ) in L 2 (P) and the score modelS θ (Zt, t) := s θ (Xt - k η k t Y k t , t), -η 1 t s θ (Xt - k η k t Y k t , t), ..., -η K t s θ (Xt - k ηY k t , t)yields the best L 2 (P)-approximator of ∇ z log p t viaEQUATIONWe assume throughout this section type II fBM. Given the marginal covariance matrix Σ t of Z t |x 0 we uniformly sample first a time point t ∈ (0, T ] and second Z t ∼ N (ẑ t , Σ t ) with ẑt = (c(t)x 0,1 , 0, ..., 0, c(t)x 0,2 , 0, ..., 0, ..., ..., ..., c(t)x 0,D , 0, ...0) ∈ R D(K+1) (86) where we use E [X t |x 0 ] = c(t)x 0 and E Y k t = 0 D . In the following we characterize further the entries of the marginal covariance matrix Σ t . The calculations in this section are straightforward; nevertheless, we present them in full detail to facilitate easy understanding for the interested reader. We begin with rewriting σ 2 from Proposition 4.2 given byEQUATIONwhereEQUATIONWithf k (u, s) := g(u) c(u) e -γ k (u-s) and I k (t, s) := t s f k (u, s)du (90)we haveEQUATIONwherevar B (t) = c 2 (t) t 0 g 2 (s) c 2 (s) ds (98)corresponds to the purely Brownian marginal variance, explicitly calculated for VE and VP in Song et al. [16] . Using the above derivation, we derive the closed form variance schedule for FVE dynamics.Fractional Variance Exploding Fix σ max > σ min > 0 and define r := σmax σmin . Following Song et al. [16] we set µ(t) ≡ 0 and g(t) = ar t with a = σ min 2 log(r) (99) such that c(t) = exp(0) = 1 and calculateEQUATIONsince the derivative of F (u) = a k r u e -γ k (u-s) is given byEQUATION)We calculate for the variance ofX t |x 0 V [X t |x 0 ] = K i,j=1 ω i ω j {var B (t) -aγ i t 0 r s I i (t, s)ds Ji(t)-aγ j t 0 r s I j (t, s)dsEQUATIONEQUATION)withEQUATIONEQUATION)EQUATION)FinallyJ i,j (t) = a i a j t 0r t e -γi(t-s) -r s r t e -γj (t-s) -r s ds (110) We calculate the covariance ofEQUATIONX t |x 0 and Y l t cov(X t |x 0 , Y l t ) = c(t) t 0 α(t, s)e -γ l (t-s) ds (112) = t 0 K k=1 ω k g(s) c(s) -γ k t s f k (u, s)du e -γ l (t-s) ds (113) = K k=1 ω k a t 0 r s e -γ l (t-s) ds -γ k t 0 t s f k (u, s)due -γ l (t-s) ds (114) = K k=1 ω k a t 0 r s e -γ l (t-s) ds -γ k a k t 0 r t e -γ k (t-s) -r s e -γ l (t-s) ds (115) = K k=1 ω k ae -γ l t t 0r s e γ l s dsγ k a k t 0 r t e -γ k (t-s) -r s e -γ l (t-s) dsEQUATION)Fractional Variance Preserving To the best of our knowledge, there is no closed form solution for t s f k (u, s)du for the dynamics of FVP. In this case, we numerically solve an ODE to determine the marginal covariance matrix of the conditional augmented forward process.General Dynamics. The covariance matrix of the conditional augmented forward process with dynamicsEQUATIONsolves the ODEEQUATIONlacking in general a closed form solution [43] in contrast to the setting of Song et al. [16] . This approach is applicable for any choice of µ and g in the forward dynamics, but depending on the choice of drift and diffusion function it might not yield a numerically stable solution. We empirically observe in Figure 5 that the analytical solution for FVE and the numerical approximation of the variance schedule, determined by solving eq. ( 119) do not differ significantly.Variance schedules. We normalize the variance schedule of FVE and FVP dynamics such that the variance at t = 0 and at t = T is equal to the variance used in the purely Brownian setting of VE and VP dynamics. For both FVE and FVP dynamics we calculate ω according to Proposition 3.3 and determine σ2 T and define ω = ω/σ 2 T to weight the OU-processes. By doing so, the terminal variance remains the same throughout different choices of H, as empirically confirmed in Figure 6 . In Figure 6 we observe for FVE dynamics that not only the terminal variance is the same across different choices of H but also the shape of the variance schedule. For FVP dynamics, the shape of the variance schedule shifts with different values of H, approaching a nearly linear schedule for H = 0.1, while H = 0.9 offers a decreasing variance towards the end near t = T .We used for all experiments a conditional U-Net [70] architecture and the Adam optimizer [71] with PyTorchs OneCylce learning rate scheduler [72] . On MNIST we trained without exponential moving average (EMA) while on CIFAR10 we conducted experiments with and without EMA.Set up on MNIST. We used an attention resolution of [4, 2] , 3 resnet blocks and a channel multiplication of [1, 2, 2, 2, 2] and trained with a maximal learning rate of 10 -4 for 50k iterations and a batch size of 1024. For all MNIST training runs we used one A100 GPU per run, taking approximately 17 hours.Set up on CIFAR10. We used an attention resolution of [8] , 4 resnet blocks and a channel multiplication of [1, 2, 2, 2, 2]. For the experiments without EMA, we used the same setup as with MNIST, but trained the models in parallel on two A100 GPUs for 300k iterations with an effective batch size of 1024. When training with EMA, we followed the set up of Song et al. [16] using an EMA decay of 0.9999 for all FVP dynamics and an EMA decay of 0.999 for all FVE dynamics. In contrast to Song et al. [16] we used PyTorchs OneCycleLR learning rate scheduler with a maximal learning rate of 2 • 10 -4 and trained only for 1mio iterations instead of the 1.3mio iterations in Song et al. [16] .In addition to the experiments presented in the main part, we provide additional results here, including a full evaluation of FVE dynamics on MNIST, as well as training on CIFAR10 without EMA.In Table 4 we provide the evaluation of FVE dynamics. For the ease of comparisan, we include the quantitative results on FVP dynamics already presented in the main part. For FVE dynamics both, the super-diffusive regime and the sub-diffusive regime achieve a higher FID as the purely Brownian dynamics for K = 1, 2 throughout all tested Hurst indices and for K = 3 throughout all tested Hurst indices except for H = 0.9 with a higher pixel-wise diversity in the sub-diffusive regime of H < 0.5.Training on CIFAR10 without EMA. As Song et al. [16] point out, the empirically optimal EMA decay rate for VP dynamics differs from that for VE dynamics. Since we do not have the computational resources to optimize the EMA decay rate for every configuration of our framework, we evaluated it in line with Song et al. [16] using a consistent EMA decay rate of 0.999 across all configurations of FVE dynamics and 0.9999 across all configurations of FVP dynamics. Nevertheless, because the optimal EMA decay rate appears to depend on the dynamics of the underlying stochastic MNIST H = 0.9 MNIST H = 0.9 Table 4 : FID and pixel-wise diversity scores of GFDM compared to the original setting of purely Brownian driven dynamics VE and VP. In bold the scores that are better than both purely Brownian driven dynamics VE and VP. The overall best scores within the experiment are boxed in.EQUATIONEQUATIONCIFAR10 H = 0.9 H = 0.5 process, we also evaluate our framework without EMA. In Table 5 we observe that the best performing configuration in terms of FID is FVP(H = 0.9, K = 2) with an FID of 8.99 and FVP(H = 0.1, K = 1) with an FID of 8.93 compared to the purely Brownian dynamics VE with an FID of 9.38 and VP with an FID of 17.29. Due to limited computational resources we only compared the best purely Brownian dynamics (VE) with the performance of corresponding augmented FVE dynamic of GFDM.EQUATIONAs to be expected, using EMA for training of GFDM results in improved performnace w.r.t. image quality measured by FID obervable in Table 2b .Effect of the number of augmenting processes in the super-diffusive regime. Additionally, in Figure 7 , we show the FID evolution of the super-diffusive regime for various numbers of augmenting processes, showing a similar pattern that either that K = 2 or K = 3 yields the best performance across different datasets and dynamics. Visual comparison of generated CIFAR10 images sampled with SDE dynamics. In this section we compare the computation time of GFDM to the purely Brownian setting of traditional diffusion models. For a given Hurst index H ∈ (0, 1) and a given K, the optimal coefficients ω 1 , ..., ω K are calculated only once before training. For completeness of our quantitative compute time evaluation, we provide the average computation time in seconds, needed to compute ω 1 , ..., ω K on a GPU Tesla V100 with 32 GB RAM. We randomly sample 1000 times H ∼ U[0.1, 0.9] for a given K ∈ {1, 2, 3, 4, 5} and report the average computation time in Table 7 .K = 1 K = 2 K = 3 K = 4 K = 5time [s] 0.0043 0.0003 0.0003 0.0003 0.0003Table 7 : Averaged time in seconds needed before training to calculate for a given K the optimal approximation coefficients using the approach of Daems et al. [27] .Computation time during training The computational difference during training consists of the computation of the covariance matrix Σ t instead of the marginal variance and sampling from a multivariate Gaussian instead of a univariate Gaussian. Note however, that we only need to calculate Σ t for D = 1 and also sample only once for a given time t and a given data point. In Table 8 and Table 9 we report the average time of one training step measured in seconds calculated over 1000 training steps on CIFAR10. The underlying conditional U-Net has 58.7mio and EMA is applied. The batch size is 128 and all computation have been carried out on a GPU Tesla V100 with 32 GB RAM.We observe that the computation time depends only minimaly increases when switching from the original model to the augmented system and increases across FVE and FVP dynamics by at most 11/1000 seconds, while the choice of the Hurst index H has no effect on the computation time.Computation time during sampling Since the augmented system depends for fixed K only on the approximating coefficients ω 1 , ..., ω K it would suffice to report the average sampling time for FVP and FVE dynamics for varying K. Nevertheless, we report for H ∈ {0.9, 0.5, 0.1} in Table 10 and Table 11 the average time to sample a batch of 1000 images over 1000 discretization steps of the reverse-time SDE over 10 trials. We observe that the average time in seconds for one sampling step in the reverse dynamics of FVE and FVP dynamics increases for K ≤ 4 by at most 2/100 seconds.Only for K = 5 we observe a significant increase of average sampling time of roughly 4/10 seconds.Given the approximate PF ODE corresponding to the augmented forward processEQUATIONwe estimate according to Song et al. [16] the log-likelihoods of test data z 0 under the learned density paugEQUATIONAccording to Song et al. [16] , we integrate over [ϵ, T ] rather than [0, T ], using the same value of ϵ = 10 -3 , which has been empirically shown to yield the best performance when simulating the SDE.For ϵ ̸ = 0 and type II fBM we need to adjust the starting value of the augmenting processes from zero to a jointly sampled vector y ϵ = (y 1 ϵ , ..., y K ϵ ) ∼ N (0 K , Λ ϵ ) withEQUATION)Using the exact likelihood of y ϵ and the independence of y ϵ and x 0 we haveEQUATION)where p0 is the learned density of x 0 corresponding to θ. Hence in total log p0 (x 0 )= log paugEQUATIONand we define the negative log-likelihoods N LLs of test data x 0 under the learned density by N LLs(x 0 , θ) := -log paug 0 (z 0 ) + log q ϵ (y ϵ ).H Challenges in the attempt to generalizeIn this work, we seek to determine the extent to which the continuous-time framework of score-based generative models can be generalized from an underlying BM to an underlying fBM. For a fBM W H it is not straightforward to define the forward processEQUATIONdriven by fBM, since fBM is neither a Markov process nor a semimartingale [25] , and hence Itô calculus may not be applied, to define the second integral. However, a definition of the integral w.r.t. fBM is established [25, 73] with the RHS of the conditional backward Kolmogorov equation one derives an equation that only depends on the joint density p(x t , t, x s , s). Using Bayes theorem again leads to a conditional backward Kolmogorov equation for p(x t , t|x s , s) that defines the dynamics of the reverse process by the one-to-one correspondence between the conditional backward Kolmogorov equation and the reverse-time SDE [32] . Following these steps for fBM, starting from eq. ( 127) and deploying the one-to-one correspondence of fBM and the evolution of its density [73] , we could replace The missing part is however an analogous to the conditional backward Kolmogorov equation to replace ∂p(xs,s|xt,t) ∂t in eq. ( 127). The derivation of such an equation is to the best of our knowledge yet unsolved problem and hence the limiting factor in the generalization of continuous-time score-based generative models from an underlying BM to an underlying fBM. k k ∈ N with 1 ≤ k ≤ K Y k OU processes Y k = Y γ k Y 1 , ..., Y K Augmenting processes with Y k = (Y k , ..., Y k ) F, G Vector valued functions F, G : [0, T ] → R D•(K+1)F, G Reverse time vector valued functions withF(t) = F(T -t) and G(t) = G(T -t) Z By Y 1 , ..., Y K augmented forward process Y [K]Stacked vector of augmenting processes Justification: We summarize the contribution of our work in the introduction and in the abstract.q t Marginal density of Y [K] at t ∈ [0, T ]Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.See Appendix I for the notational conventions of this work.See https://lilianweng.github.io/posts/2021-07-11-diffusion-models/ for the derivation in discrete time. 


Decision: The paper proposes an extension of diffusion models where the Brownian motion driving the noise process is replaced by fractional Brownian motion. However, this introduces several challenges due to the non-Markovian nature of fractional Brownian motion. To address this problem, the paper suggests approximating this process using a linear combination of Ornstein–Uhlenbeck processes driven by the same Brownian motion. Experiments on standard image dataset benchmarks illustrate the method's behavior for different Hurst exponent parameters and demonstrate good performance.

Most of the reviewers and myself initially found the paper borderline. However, the authors addressed most of the concerns raised by the reviewers during the rebuttal. I trust they will incorporate the reviewers' feedback and the clarifications provided during the rebuttal into the final version of the paper.

Reviews: review: paper_summary: paper_summary: The paper proposed diffusion models where Brownian motions are generalized by fractional Brownian motions - a correlated noise process. Due to the non-Markovian property of fractional Brownian motions, the paper suggests approximating it using a linear combination of semimartingale processes. The key insight of inducing fractional noise to diffusion models is to allow a control over the diversity sampling.  Experiments consider different setups of Hurst exponent parameters, demonstrating good performance in terms of inception score and Vendi score.
strength_weakness: strength_weakness: - I find the discussion in the experiments in the main text does not give much insight. What is the main message that the paper is trying to convey? What is the implication of the numbers? 
- Missing the computational trade-off between Fractional Diffusion Models and traditional Diffusion models. The approximation using $K$ different semimartingale processes requires to sample these processes. How does it affect the training and inference?
questions: questions: 1. Can the Markovian approximation of fractional Brownian motion be extended to multifractional Brownian motions where the Hurst index varies with respect to time?
2. Can you discuss further potential applications of Fractional Diffusion Models? While the paper suggests they are promising for generating time series, especially financial time series, do you think this model could be applied to molecular design, given that atomic interactions follow fractional Brownian motions?

Minor points:

Missing table reference in Line 722.
limitations: limitations: Please see above.
paper_summary: paper_summary: The paper replaces Brownian motion in diffusion models with fractional Brownian motion.
strength_weakness: strength_weakness: 1. The paper has very thin numerical experiments. The results do not adequately justify why replacing BM with fractional BM is preferred. 

2. In order to show that fractional BM is better than BM, the experiments should focus on exactly showing the marginal improvement while controlling everything else, which the paper does not do. 

3. The paper does not have strong methodology contribution, in additional to the replacement of fractional BM.
questions: questions: If the authors really want to make some contributions to image generation, I suggest they spend more energy understanding the structure of image data, instead of focusing only on plugging in a fractional BM to replace BM.
limitations: limitations: yes
paper_summary: paper_summary: The work proposes a theoretical framework for training score-based diffusion models based on fractional brownian motion.

Authors provide an approximation framework where fBM being approximated by Markovian noise to derive score-matching in Markov approximation.

Theoretical derivation is supplemented by experiments showing some gains w.r.t. score-matching driven by standard brownian motion.
strength_weakness: strength_weakness: It feels that motivation for why we should use fBM is not well elaborated. 

Many different settings evaluated without sensitivity analysis of $H$ and $K$ on resulting quality. It is rather hard to interpret the findings whether the gains are from fBM or from multiple comparisons.
questions: questions: If the process driven by fBM is adapted, can we use Gyöngy theorem to reduce the process to BM one and connect this score-matching procedure to BM driven score matching? If so, would it imply that gains are coming from having different path-integrator, rather than from just using fBM? 

Can authors add pvalues to the tables?
limitations: limitations: Yes
paper_summary: paper_summary: The authors introduce a new type of continuous score-based generative models relying on fractional diffusion, a type of diffusion where Brownian Motion BM is replaced by its fractional counterpart fBM, where noise increments or either positively correlated (Hurst index $1 > H > 1/2$) or negatively correlated ($0 < H < 1/2$). The authors use a Markovian approximation of fBM, consisting in the addition of $K$ correlated OU processes as the underlying dynamics guiding the diffusion. This lets them obtain tractable learning and inference, using the augmented process (noised data, OU processes) and the corresponding score. The augmented score matching loss introduced makes it possible to use a single D-dimensional score model to optimally approximate the full score in $\mathbb{L}_2$, where D is the data dimensionality. The authors then design a set of experiments on MNIST and CIFAR10 to validate the performance of the new method and the effect of varying the hyper-parameters ($K, H$), and outline its advantage over classical score-based diffusion.
strength_weakness: strength_weakness: While the paper tackles the extension of diffusion to fractional Brownian motion very satisfyingly, it seems to me that it suffers from an insufficient and unclear empirical study. 
- Performance gains are mildly convincing and the effect of the varying hyper-parameters are not that clear. The general idea seems to say that choosing $H>1/2$ and $K=3$ yields good results, with higher $H$ providing some smoothing beneficial to learning the diffusion and higher $K$ bringing in more diversity, but to be honest I am not convinced by the interpretation of the results and the general setup.
- It could be beneficial to include a more exhaustive comparison with other approaches for diffusion with different noise regimes, and the different advantages/limitations, especially since the mentioned limitations for classical diffusion models (slow convergence, mode-collapse on imbalanced data, and lack of diversity) are not properly tackled. See, e.g., [1] [2], and even consider some more detailed experimental comparison with [3].


[1] Eliya Nachmani, Robin San Roman, and Lior Wolf. Denoising diffusion gamma models, 2021

[2] Jacob Deasy, Nikola Simidjievski, and Pietro Liò. Heavy-tailed denoising score matching, 2022.

[3] Eunbi Yoon, Keehun Park, Sungwoong Kim, and Sungbin Lim. Score-based generative models with Lévy Processes. In Thirty-seventh Conference on Neural Information Processing Systems, 2023
questions: questions: Overall I am willing to raise my score if my concerns for the empirical study are addressed. Here are some additional questions and remarks.

- With $K \gg 1$ performance deteriorates. Does it means true fractional dynamics does not work well? What is happening? How does $K$ impact the approximation of true fBM (you cite a reference on Line 184, can you give something quantitative?)

- Why for $H=1/2$ do we see improvement with the augmented dynamics? Because of the correlation structure?

- Table 1 and effect of $K$ for $H = 1/2$: Not much difference, until FID deteriorates, where then $\textrm{VS}_p$ increases. If the sample quality heavily deteriorates and that the model produces random out of distribution samples, then it is not a very interesting situation and the metric loses meaning. Moreover no discussion on why choosing too high of a $K$ eventually leads to bad samples. Numerical instabilities? Training loss? Difficulty learning the augmented process?

- The Vendi score thus does not seem to be such a good choice as we cannot understand the tradeoff in quality and diversity. It would be better to use some precision/recall metric (or density/coverage) where diversity relates to covering the true data distribution. One could devise some summary statistic like an associated $F_1$ score to appreciate the tradeoff and see if a better 'Pareto optima' is attained. 

- It is also not clear what is the effect of $H$ on the samples. It seems rough path $H < 1/2$ have larger jumps but the authors advocate for $H>1/2$ and on line 284 invoke heavy-tailedness. And what is meant by heavy-tailedness here, as the variance is finite?

- The effect of varying the hyperparameters does not seem very consistent. Will the observed remarks for $(K, H)$ hold with different datasets? Will this scale? It would also be nice to have multiple runs with the associated standard deviation to assess statistical significance here.

- In the abstract, limitations of diffusion models are mentioned, in particular (i) slow convergence and (ii) mode-collapse on imbalanced data, which are not addressed in the experiments and discussed properly. For (i), there is no mention of the number of diffusion steps/number of function evaluation in the experiments. For (ii), maybe try a run on CIFAR10\_LT. 

- Table 3: no discussion on why the ODE has bad performance. 

- Line 259: 'we evaluate GFDM on [..] test distribution coverage': this seems not to be done anywhere, as Vendi score does not relate to coverage in this sense?

- The diffusion is run on $[\epsilon, T]$ but the value of $\epsilon$ is nowhere to be found it seems, and no experiment to see its effect on  GFDM's performance?

- Line 32-34: why would BM's lack of control over sampled trajectories matter? The correlation structure is independent of the model, depending on the data it could also increase mixing time or actually make it harder to learn the score. Is it because the augmented process makes it possible for the model to incorporate better correlation structure? What are the basis of these claims?

- Line 83-84: $\hat f$ seems not to be properly introduced

- Line 168-169, eq (9): typo in the denominator of the last fraction?

- Line 210: $\Sigma_t$ seems not to be properly introduced

- Line 255-258: Why would using EMA on the models interfere with the SDE dynamics?

- Line 282: Did you not mean SOTA FID of 0.72, K = 3 and H = 0.9? What are you basing your subsequent interpretation of easier to learn smoother sample paths on?

- Line 291: Did not you mean super-diffusion here with $H = 0.9 > 1/2$? idem in caption of Figure 2, with $H=0.7$?
limitations: limitations: Properly addressed.

Comments: comment: The reviewer's comments include two parts: (1) pointing out the paper's need to exactly showing their marginal contribution by controlling other factors, without which the paper is purely a plug-in paper without verifiable contributions and (2) pointing out the authors' inconsistency with themselves and double standards regarding what is subjective. 

The authors responded to none of the reviewer's comments, and the authors again left a subjective message. It is unfortunate that the authors chose to run away from discussions when their own inconsistency and double standards are revealed.
Thank you for the clarification. I believe that the paper is technically solid. Therefore, I maintain the original score.
Thank for the reply, I'll increase my score
As authors, we find it difficult to comprehend the perspective of this reviewer, and we believe that further engagement in what appears to be a non-scientific discussion would not be productive. We have brought this matter to the attention of the Area Chair and respectfully request that the other reviewers form their judgments independently of this review, which we believe to be biased.
I would like to immensely thank the authors for their dedication in answering my questions in such details, and for the additional experiments they have run to address the issues that were raised.

I acknowledge their answer, and will now reflect and discuss with the other reviewers before taking a final decision.

With best regards
It is interesting that the authors claim the review to be subjective, while they themselves argue in a subjective way with strong assumptions and vague arguments. 

As an example, the review pointed out that "the experiments should focus on exactly showing the marginal improvement while controlling everything else, which the paper does not do." The authors claim that they "exactly do so", by saying that "our method closely recovers the original model". This not exacting showing the marginal improvement. The controlling should be done way more precisely, say, on training-related parameters and hyper-parameters, all the randomness involved, etc. Just having "method closely recovers the original model" does not mean much, if anything. The authors just throw in a more complicated model and method and claim they "exactly do so" and controlled everything. 

This is an example of the authors being inconsistent. The authors do not understand what it means by "exactly showing the marginal improvement", while at the same time accusing the reviewer's understanding of the paper and "overlooked some of the important parts in our paper." 

Regarding other points in the authors' response:

Methodological contribution: the point that the authors raised about replacing BM with fBM would be a non-trivial (subjective opinion of the authors, which others would find rather trivial) task does not merit strong methodological contribution. The work is simply replacing the BM with fBM, where the authors take "non-Markovian" to be a big point. Unfortunately, there is nothing fancy about non-Markovian, except for how long and how much it can depend on the past. 

The authors on one hand uses "strength of a contribution is a subjective matter" to defend themselves and on the other hand accuse the reviewer of not acknowledging the work's strengths. It is fairly convenient use of self-contracting standards when they see fit for themselves. 

The review suggested "focus more on understanding the structure of image data rather than the noise", which the authors accuse of being subjective. This review comment suggests the authors give more focus on how the fBM noise provides more understanding of the structure of image data when they are learned from denoising fBM noise. This is about providing more understanding of why fBM is essential, instead of focusing too much on the technical details of how to handle fBM. The authors took a very defensive and accusing attitude toward the review, with their own subjective understanding.
Dear Reviewer HxVk,

We sincerely apologize for the the missing references:

[1] Yang Song and Jascha Sohl-Dickstein and Diederik P Kingma and Abhishek Kumar and Stefano Ermon and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations, International Conference on Learning Representations, 2021.

[2] Aaron Lou and Stefano Ermon. Reflected Diffusion Models, International Conference on Machine Learning, 2023.

We hope that we have sufficiently addressed the weaknesses and questions raised by the reviewer. 

With best regards,
The Authors
We truly appreciate the reviewers detailed engagement with our work and thank the reviewer for the valuable feedback.

We hope that we have sufficiently addressed the weaknesses and questions raised by the reviewer. 

With best regards,
The Authors
>Line 83-84: $\hat f$ seems not to be properly introduced

Thanks for the very detailed review. Since we have $\mathbf{f}(\mathbf{x},\cdot):[0,T]\to\mathbb{R}^{D}$ for fixed $\mathbf{x}\in\mathbb{R}^{D}$ we intended to apply the definition for  $\overline{g}$ as well to $\mathbf{\overline{f}}(\mathbf{x},t):=\mathbf{f}(\mathbf{x}_t,T-t)$ in line 80-81. In order make it more clear that the notation applies to any process $\mathbf{P}$ and any function $f$ we replace line 80-81 by:

"Whenever $\mathbf{P}=(\mathbf{P}_t) _{t\in[0,T]}$ is a stochastic process and $f$ is a function on $[0,T]$, we write $\mathbf{\overline{P}} _{t}=\mathbf{P} _{T-t}$ for the reverse time process and $\overline{f}(t)=f(T-t)$ for the reverse time function."

>Line 168-169, eq (9): typo in the denominator of the last fraction?

Thanks for finding this typo. We cut of the lower incomplete gamma function in the second part of $\mathbf{b}_{k}$. We replaced it by the correct term:

$\mathbf{b} _{k} := \frac{T}{\gamma^{H+1/2} _{k}}P(H+1/2,\gamma _{k}T) - \frac{H+1/2}{\gamma _{k}^{H+3/2}}P(H+3/2,\gamma _{k}T)$

>Line 210: $\Sigma_t$ seems not to be properly introduced

Thanks for worrying about the details. In this case we already defined it in  line 195 - 198: "The missing components in the conditional covariance matrix $\Sigma_t$ of the augmented forward process are the conditional marginal variance of the forward process and the conditional marginal correlation between the forward process and the augmenting processes."

>Line 291: Did not you mean super-diffusion here with $H = 0.9 > 1/2$? idem in caption of Figure 2, with $H=0.7$?

Yes, thanks for noting. This is a typo and should be super-diffusion.
>The effect of varying the hyperparameters does not seem very consistent. Will the observed remarks for $(K,H)$ hold with different datasets? Will this scale? It would also be nice to have multiple runs with the associated standard deviation to assess statistical significance here.

We agree with the reviewer's point that our previous presentation of our findings was not very clear. We hope to adress this issue in our attached rebuttal PDF:

* in Table 1 we observe that the super-diffusive regime $(H>0.5)$ yields the best performance on MNIST and CIFAR10 in terms of quality and an increasing number of augmenting processes yields higher pixel-wise diversity. 
* in Figure 1 we observe that the quality evolution w.r.t. the number of augmenting processes evolves similar across dynamics and dataset. 

We also agree with the reviewer that it would be favorable to report the associated standard deviation across different run. However, this is beyond our computational resources. One training on CIFAR with 2 GPUs (Ampere A100 40 GB RAM) takes roughly 50 hours, making it unfeasible to train multiple times. Following the evaluation setup of Song et. al[1] we trained for 1 million iterations on CIFAR10 and evaluated one checkpoint every 100k iterations, to ensure that the result is not affected by overfitting. This ensures to some extend that the FIDs we report on CIFAR10 are not a lucky pick. Nevertheless, to adress the reviewers concerns we report in Figure 3c) the average FID and the associated standard deviation for a certain number of sampling steps over three rounds of sampling. Please note that to report one of these scores for $NFE=1000$, we need to sample $50k$ images on a single GPU (Ampere A100 with 40 GB RAM), which takes 16 hours.

>In the abstract, limitations of diffusion models are mentioned, in particular (i) [...]. For (i), there is no mention of the number of diffusion steps/number of function evaluation in the experiments

We thank the reviewer for pointing out that we did not mention the number of function evaluations (NFE). To address the reviewer's point, we compare the performance of the super-diffusive regime to the purely Brownian-driven dynamics in Table 1 of the attached rebuttal PDF. Evaluating the performance for different numbers of NFE of sampling steps in Figure 3a), we observe that the super-diffusive regime of MA-fBM saturates at $500$ NFE on a lower level than the purely Brownian driven dynamics.

>(ii) mode-collapse on imbalanced data, which are not addressed in the experiments and discussed properly.[...] For (ii), maybe try a run on CIFAR10_LT. 

Unfortunately, this was not feasible given the short time period from review release to rebuttal. Based on the reviewer's suggestion, we will consider experiments on CIFAR10_LT in our future work.

>Line 259: 'we evaluate GFDM on [..] test distribution coverage': this seems not to be done anywhere, as Vendi score does not relate to coverage in this sense?

Our evaluation of negative log likelihood (NLL) in Table 1 and Table 2 was intendet to measure test distribution coverage, since NLL estimates the negative log-likelihood of test data under the learned density. 

>The diffusion is run on $[\epsilon, T]$ but the value of $\epsilon$ is nowhere to be found it seems, and no experiment to see its effect on GFDM's performance?

Thanks for noting. We defined $\epsilon$ in the Appendix (lines 730-731), but the definition should, of course, be in the main part of the paper. We used throughout all experiments $\epsilon=1e-5$ for training and $\epsilon=1e-3$ for sampling according to Song et. al[1].

>Line 32-34: why would BM's lack of control over sampled trajectories matter? The correlation structure is independent of the model, depending on the data it could also increase mixing time or actually make it harder to learn the score. Is it because the augmented process makes it possible for the model to incorporate better correlation structure? What are the basis of these claims?

The Hurst index $H$ in our framework provides control over the diffusion processes trajectory in terms of roughness vs. mild randomness. Our results indicate that the more regular and correlated trajectories are preferred, which is not possible to achieve with BM. As we pointed out in our answer above, a smoother trajectory might decrease the discretization error from continuous time to discrete time steps.
Dear Reviewer kZkj, 

We sincerely apologize for the formatting issues in our previous response and the missing reference. Here is the corrected version, along with additional answers to the questions we couldn't cover previously due to the character limit:

[1] Yang Song and Jascha Sohl-Dickstein and Diederik P Kingma and Abhishek Kumar and Stefano Ermon and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations, International Conference on Learning Representations, 2021.

>Why for do we see improvement with the augmented dynamics? Because of the correlation structure? 

Yes, our hypothesis is that the known part of the score function is guiding the data generating process towards the direction of the true data distribution, since the starting distribution of the augmenting processes is known and all processes are driven by the same random path realization of Brownian motion. In the forward process $\mathbf{Y}_t$ does not depend on $\mathbf{X}_t$, but in the reverse model $\mathbf{Y}_t$ does depend on $\mathbf{X}_t$, since it depends in every entry on the score model evaluation. Assume that $\mathbf{Y}_t$  does not leave its true trajectory when we simulate the reverse time model, in this case the trajectory of $\mathbf{Y}_t$ might serve as a corrector for the trajectory of $\mathbf{X}_t$.

>With $K>>1$ the performance deteriorates. Does it means true fractional dynamics does not work well? What is happening? 

We believe that the most important factor for chosing the optimal number of augmenting processes is the dataset. In Table 1 of the attached Rebuttal PDF, we observe that on MNIST, $K=4,5$ works well in the super-diffusive regime, while on CIFAR10, the performance degrades in the same regime. In Figure 2 of the attached rebuttal PDF, we observe that for the same FVP dynamics, there are different well-performing regimes for the number of processes. (K=3,4,5 on MNIST and K=1,2 on CIFAR10).

>How does $K$ impact the approximation of true fBM (you cite a reference on Line 184, can you give something quantitative?)

The integrated $L_2$ error over time $[0,T]$ for different choices of $K$ is visualized in Figure 7 of Daems et. al[2]. The higher the value of $K$, the better the approximation. However, after a certain number of augmenting processes, the error saturates, depending on $H$.

[2] Rembert Daems, Manfred Opper, Guillaume Crevecoeur, Tolga Birdal. Variational Inference for SDEs Driven by Fractional Noise. The Twelfth International Conference on Learning, 2024.

> Table 1 and effect of $K$ for $H=1/2$: Not much difference, until FID deteriorates, where then increases. If the sample quality heavily deteriorates and that the model produces random out of distribution samples, then it is not a very interesting situation and the metric loses meaning. Moreover no discussion on why choosing too high of a eventually leads to bad samples. Numerical instabilities? Training loss? Difficulty learning the augmented process

We agree with the reviewers point that a low-quality image with increased pixel-wise diversity is not a favorable situation. In Table 1 of the attached rebuttal PDF, we observe that the pixel-wise diversity increases with a higher number of augmenting processes on MNIST and CIFAR10. However, the quality seems to depend on the Hurst index and the dataset, as we observe on MNIST for larger $K$:


| K | H=0.9 | H=0.7 |H=0.5 |H=0.1|
|---|-------|-------|------|-----|
| 4 | 1.22  | 0.86  |1.86  |6.25 |
| 5 | 2.17  |  1.36 |4.89  |9.57 |

while on CIFAR10 we observe a clear degradation in quality:

| K | H=0.9 | H=0.7 |H=0.5 |H=0.1|
|---|-------|-------|------|-----|
| 4 | 29.72  | 8.45  |8.85  |5.02 |
| 5 | 69.06  | 35.91 |96.54  |7.38 |

Our hypothesis here is that $K$ controls the pixel-wise diversity, while the optimal number of augmenting processes for quality depends on the dataset and the dynamics (FVE or FVP).
We appreciate the reviewer's comments and feedback. However, we believe the strong criticism is excessively harsh and unjustified. The absence of a summary or acknowledgment of our method's strengths raises concerns about the reviewer's full understanding of our work. Below, we address the specific comments and questions they posed:

>  thin numerical experiments.

We now provide a set of additional experiments and report the computational considerations in the attached rebuttal PDF. We have included all of these in a further revision. We have also included a motivation for exploiting fractional Brownian motion (fBM) instead of Brownian motion (BM):

Having control over the diffusion trajectories are crucial for obtaining a trade-off between generation quality and diversity. Moreover, there is no true support for why the noise increments should be independent. Our framework provides two hyperparameters, the number of processes and the Hurst index to have control over diffusion processes and characterize the nature of trajectories, \emph{e.g.} roughness vs. mild randomness. Our results indicate that in both aspects, quality and diversity, the diffusion models driven by Brownian motion lead to suboptimal results and the best outcome are obtained when $K\geq 2$ and $H>0.5$. This indicates that more regular and correlated trajectories are preferred while a single process is usually not sufficient to capture the entire complexity. 

> ... the experiments should focus on exactly showing the marginal improvement while controlling everything else,

In fact, we exactly do this. We have two parameters, the number of Markov processes ($K$) and the Hurst index ($H$), that differentiate our method from a standard diffusion model. When $K=1$ and $H=0.5$, our method closely recovers the original model driven by Brownian motion. Our evaluations cover a range of $K$ and $H$ values, clearly demonstrating the improvements from our contributions. We have to admit that we suspect the reviewer might have overlooked some of the important parts in our paper. 

> The paper does not have strong methodology contribution, in additional to the replacement of fractional BM

We kindly yet strongly disagree that *our method does not have a strong methodology contribution*. Replacing BM with fBM in diffusion models is definitely a non-trivial task due to the non-Markovian nature of fBM. To achieve this, we have made significant contributions, including deriving a memory and computation-efficient reverse process, proving the optimality of the score model, and providing explicit formulas for the marginals of the conditional forward process. Besides, strength of a contribution is a subjective matter and the reviewer does not mention on which aspects our contribution is not strong.

> focus more on understanding the structure of image data rather than the noise

Regrettably, this is another subjective comment. We agree that there are various ways to improve diffusion models. In this work, we specifically focuses on the driving noise, which, as shown by our experiments, has a significant impact in quality and diversity. Our work should be evaluated within this context. We find it surprising that the reviewer recommends a different, vague research direction instead of providing constructive feedback on our work.

In the light of these and the feedback of other reviewers, we kindly ask the reviewer to reconsider their initial assessment.