Title: Dimension-free Score Matching and Time Bootstrapping for Diffusion Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion Models

Score Matching

Empirical Risk Minimization

Generalization

Markov Chain

Learning from Markov Dependent Data
Abstract: Diffusion models generate samples by estimating the score function of the target distribution at various noise levels. The model is trained using samples drawn from the target distribution, progressively adding noise. Previous sample complexity bounds have a polynomial dependence on the dimension $d$, apart from $\log({|\mathcal{H}|})$, where $\mathcal{H}$ is the hypothesis class. In this work, we establish the first (nearly) dimension-free sample complexity bounds, modulo any dependence due to $\log( |\mathcal{H}|)$, for learning these score functions, achieving a double exponential improvement in dimension over prior results. A key aspect of our analysis is to use a single function approximator to jointly estimate scores across noise levels, a critical feature in practice which enables generalization across timesteps. We introduce a novel martingale-based error decomposition and sharp variance bounds, enabling efficient learning from dependent data generated by Markov processes, which may be of independent interest. Building on these insights, we propose Bootstrapped Score Matching (BSM), a variance reduction technique that utilizes previously learned scores to improve accuracy at higher noise levels. These results provide crucial insights into the efficiency and effectiveness of diffusion models for generative modeling.

Dimension-free Score Matching and Time Bootstrapping for Diffusion Models

Abstract

Diffusion models generate samples by estimating the score function of the target distribution at various noise levels. The model is trained using samples drawn from the target distribution, progressively adding noise. Previous sample complexity bounds have a polynomial dependence on the dimension d , apart from log ( |H| ) , where H is the hypothesis class. In this work, we establish the first (nearly) dimension-free sample complexity bounds, modulo any dependence due to log( |H| ) , for learning these score functions, achieving a double exponential improvement in dimension over prior results. A key aspect of our analysis is to use a single function approximator to jointly estimate scores across noise levels, a critical feature in practice which enables generalization across timesteps. We introduce a novel martingale-based error decomposition and sharp variance bounds, enabling efficient learning from dependent data generated by Markov processes, which may be of independent interest. Building on these insights, we propose Bootstrapped Score Matching (BSM), a variance reduction technique that utilizes previously learned scores to improve accuracy at higher noise levels. These results provide crucial insights into the efficiency and effectiveness of diffusion models for generative modeling.

1 Introduction

Score-based diffusion models [43, 17] are generative models that have transformed image and video generation [38, 40, 37, 36], enabling foundation models to produce photorealistic and stylized images from text prompts. Their adaptability extends diverse domains such as audio [24, 10], text [13, 15, 29, 49], molecule [18, 19], and layout generation [21, 28]. They generate additional samples given m i.i.d. samples from a target distribution ( π ) using a trained neural network that learns the score function π at different noise levels. In contrast, the classical Markov Chain Monte Carlo (MCMC) methods which seek to sample from a distribution given access to underlying density function. While MCMCmethods can be slow for multi-modal distributions, diffusion models can sample efficiently with minimal assumptions, provided the score functions are learned accurately [6, 1].

Given m i.i.d. samples from the target distribution, the first step (called the forward process) obtains noised samples from a noising Markov process converging to the Gaussian distribution at various noise levels. The second step estimates score functions of the distribution at each noise level using Denoising Score Matching (DSM) [51]. This approach relies on learning from dependent data from

multiple trajectories of a Markov process in contrast to learning with i.i.d. data in traditional settings. Prior works [3, 14] provide theoretical guarantees for score function approximation separately at each noise level using the same samples. However, in practice, a single function approximator is commonly used at all noise levels, which is considered by [16]. [4] show that despite the problem of distribution estimation suffering from the curse of dimensionality [5, 35], the existence of low-dimensional structures allows neural networks to learn the score functions. All of these existing bounds exhibit polynomial dependence on the dimension, d . We establish that under suitable smoothness conditions for a given function class, score matching with a single function approximator jointly across all timesteps achieves a nearly dimension-free sample complexity that depends on the smoothness parameter and grows only as log log( d ) .

1.1 Our Contributions

(1) We analyze the sample complexity of denoising score matching across noise levels using a single function approximator, achieving a double-exponential reduction in dimension dependence .

(2) We present a novel martingale decomposition of the error, which allows us to bound the error despite being composed of samples from multiple trajectories of dependent data .

(3) We use second-order Tweedie-type formulae to obtain a sharp bound on the error variance , crucial for establishing almost dimension-free convergence rates.

(4) Inspired by the above, we present Bootstrapped Score Matching . Here the score at a given noise level is learned by bootstrapping to the learned score function at a lower noise level, achieving variance reduction. This shows improved performance compared to DSM in simple empirical studies.

1.2 Related work

Score matching and diffusion models: Score Matching was introduced in the context of statistical estimation in [20] with an algorithm now called Implicit Score Matching (ISM). Diffusion models are trained using Denoising Score Matching (DSM) introduced in [51], and is based on Tweedie's formula. Several algorithms have been introduced since, such as Sliced Score Matching [44] and Target Score Matching [7]. Prior works have analyzed the complexity of Denoising Score Matching in various settings [5, 35, 14, 3, 12]. We consider the setting in [14, 3], where the score functions of the given distribution can be accurately approximated by a function approximator class (ex: neural networks), instead of the worst case non-parametric analysis in [12]. These bounds can then be used with the discretization analyses in [1, 6, 27] to guarantee the quality of the generated samples.

Learning from dependent data: Learning with data from a markov trajectory has been explored in literature in the context of system identification, time series forecasting and reinforcement learning [8, 42, 34, 25, 48, 53, 2, 26, 46] Many of these works analyze the rates of convergence with data derived from a mixing Markov chain, when the number of data points available is much higher than the mixing time, τ mix . In our context, the Markov chain contains ˜ O ( τ mix ) data points created by progressively noising samples from the target distributions, where ˜ O hides logarithmic factors. This is similar to the setting in [48], which considered linear regression and linear system identification.

2 Problem setup and preliminaries

Notation: We use [ n ] to denote { i ∈ N | i ≤ n } . I ∈ R d × d represents the d -dimensional identity matrix. We use N ( µ, Σ ) to denote the multivariate normal distribution with specified mean, µ and covariance matrix Σ . ∥ . ∥ 2 denotes the ℓ 2 euclidean norm for vectors and ∥ . ∥ op denotes the operator norm for matrices. E [ X ] denotes the expectation of the random variable X and Cov ( X ) denotes its covariance matrix. For a, b ∈ R , we write a ≲ b if and only if there exists an absolute constant C > 0 such that a ≤ Cb. ˜ O, ˜ Ω represent order notations with logarithmic factors. We also define a coarser notion of subGaussianity used subsequently in our proof sketch,

Definition 1 ( ( β 2 , K ) -subGaussianity) . A mean-zero random variable Y is said to be ( β 2 , K ) -subGaussian if it satisfies P ( | Y | > A ) ≤ e K exp( -A 2 2 β 2 ) .

Ornstein-Uhlenbeck process: Consider a target distribution π over R d . Suppose x 0 ∼ π and x t solve the following Stochastic Differential Equation (SDE):

where B t is the standard Brownian Motion over R d . An application of Ito's formula demonstrates that x t = x 0 e -t + z t where z t ∼ N (0 , σ 2 t I ) is independent of x 0 and σ t := √ 1 -e -2 t . This is the forward noising process, which progressively noises the initial sample into a standard Gaussian vector. Ito's formula also relates x t , x t ′ for any timesteps t > t ′ ≥ 0 to obtain, x t = x t ′ e -( t -t ′ ) + z t,t ′ where z t,t ′ ∼ N (0 , σ 2 t -t ′ I ) is independent of x t ′ and σ t -t ′ := √ 1 -e -2( t -t ′ ) . For t ∈ [0 , T ] , let p t be the probability density function of x t . Given ¯ x 0 ∼ p T and a standard R d Brownian motion ¯ B , then the denoising process is:

It is the time reversal of the noising process which implies ¯ x T ∼ π [45].

Score matching: Given i.i.d. data points x (1) , . . . , x ( m ) from the target distribution π , diffusion models learn the score function s ( t, x ) : R + × R d → R d defined as s ( t, x ) ≡ s t ( x ) := ∇ log p t ( x ) via denoising score matching (DSM). Tweedie's formula states that

Let H be a finite class of functions which map R + × R d to R d with functions ( t, x ) → f ( t, x ) ≡ f t ( x ) . Let T = { t 1 , . . . , t N } ⊆ [0 , T ] . Let x ( i ) t denote the solution of Equation (1) at time t with x ( i ) 0 = x ( i ) and define z ( i ) t := x ( i ) t -e -t x ( i ) . We consider the joint DSM objective to be:

Thus, optimizing (4) is a regression task with noisy labels. The two sources of error are: i) By (3), while -E [ z t /σ 2 t | x t ] = s ( t, x t ) , z t /σ 2 t is noisy conditioned on x t and ii) x t ∼ p t itself is random. The empirical risk minimizer is defined as ˆ f = arg inf f ∈H ˆ L ( f ) . The results established in [1] states that the error in sampling arising from using the estimated score function ˆ f is given by:

Our goal is to bound this error. For simplicity, we consider t i = i ∆ for some step size ∆ ∈ (0 , 1) .

3 Main results

We operate under the following smoothness assumption on the function class, H .

Assumption 1 (Smoothness of function class) . Let the true score function, s ∈ H .

∇ log p t ( · ) is continuously differentiable for every t ∈ R + .

Lipschitzness : For all t ∈ T , x 1 , x 2 ∈ R d , f ∈ H : ∥ f ( t, x 1 ) -f ( t, x 2 ) ∥ 2 ≤ L ∥ x 1 -x 2 ∥

Local Time Regularity : There exists a set B δ,t such that p t ( B δ,t ) ≥ 1 -δ , ∀ t ≥ t ′ ∈ T , x ∈ B δ,t , ∀ f ∈ H

Assumption (1) is a standard Lipschitz continuity assumption followed in the literature (see e.g. [3]) 2 . Assumption (2) assumes Hölder continuity with respect to the time variable. This is a natural assumption because Lemma 10 shows that Assumption 1-1 implies Assumption 1-2 for the true score

function, s ( t, x ) . We also note that the assumption s ∈ H can be relaxed to assume that ∃ ¯ s ∈ H with sufficiently small ℓ 2 error, similar to [14]. While we assume a finite function class, H , we can extend it to infinite classes by a standard covering argument in learning theory or consider H to be the finite class of floating point quantized models such as neural networks.

Equation (1) demonstrates that x t forms a Markov chain, leading to the noise random variables, z t , being dependent. Additionally, (1) is typically iterated for T = ˜ O ( τ mix ) timesteps, until p T is close to a gaussian distribution. This setup falls outside the scope of conventional analyses of learning from dependent data (see Section 1.2). Such analyses usually assume a significantly larger number of datapoints, where datapoints separated by τ mix in time are approximately independent, and the convergence rates align with their i.i.d. counterparts, adjusted for an effective sample size reduced by a factor of τ mix . In contrast, our setting involves substantially fewer datapoints. To address this challenge, we propose a novel martingale decomposition (stated in Lemma 2 and proved in Lemma 20) of the error and establish sharp concentration bounds to account for these dependencies.

Recall the DSM objective in (4). As explained before, there are two sources of noise: (1) due to -z t /σ 2 t conditioned on x t , (2) due to x t ∼ p t . We demonstrate the effect of fluctuations in z t | x t in Theorem 1 and then deal with the random fluctuations due to x t in Theorem 2.

(a)

(b)

parameter

10-4

Errorperparametervs.d

EmpiricalL2Error

9 ×10-5

Fit:slope=-0.02

8×10-5

per

7 × 10-5

error

6 ×10-5

2

Empirical I

5 × 10-5

10

20

3040

75

125

200

Dimensiond

parameter

Errorperparametervs.d

0.010

EmpiricalL2Error

Errorwithlinearddepedence

0.008

per

0.006

error

0.004

0.002

0.000

10

30

75

125

200

Dimensiond

Our first result in Theorem 1 provides a dimension-free bound on the empirical squared error, wherein we show how to control the noise due to z t , conditioned on the data, x t .

Theorem 1 (Empirical L 2 Bound) . Let Assumption 1 hold. Fix δ ∈ (0 , 1) . For all j ∈ [ N ] , let t j := ∆ j and γ j := ∆ . Let B := C log ( ( L +1) dmN log ( 1 δ ) / ∆ ) for an absolute constant C > 0 , and let ∆log 3 ( 1 ∆ ) d 3 log 3 (2 d ) log 3 ( 2 Nm δ ) log 3 ( B |H| δ ) ≤ 1 and N ∆ ≤ C log( 1 ∆ ) . Then for

with probability at least 1 -δ ,

Remark 1. All prior works such as [3, 14, 12] have at least a linear dimension dependence apart from the log ( |H| ) . In contrast, modulo log ( |H| ) , our bounds are nearly dimension-free ( log log ( d ) dependence due to B ), and instead depend on the smoothness parameter L . Therefore, we bring down the complexity from poly ( d ) log( |H| ) to log( |H| ) which is meaningful in high dimensions.

To put our results in context, prior work [12] shows that score matching suffers from exponential dependence on d in the worst case. In contrast, we show that when the target distribution admits a

suitable class of estimators with mild smoothness assumptions, score estimation becomes sampleefficient. This closes the gap between theoretical results and empirical findings in diffusion models, where global optimization reliably learns accurate score functions for natural images, despite the worst-case guarantees for training neural networks.

In Figure 1 (see Appendix Section F for details of experimental setup), we train a 2-layer neural network with a fixed hidden dimension and sample size, to sample from N (0 , Σ) , and measure the L2 error across timesteps, described in (4) , scaled inversely by log ( |H| ) log log ( d ) . Here we use the fact that log ( |H| ) scales linearly with the number of parameters for neural networks (with quantization). Figure 1 is consistent with the dimension-free bounds in Theorem 1.

Remark 2. Theorem 1 requires the time-discretization ∆ = ˜ O (1 /d 3 ) . However, this is not a problem during training due to the use of stochastic optimization algorithms which obtain minibatches of datapoints at randomly sampled times. We illustrate this further by distinguishing between the statistical and optimization questions underlying our results.

Statistical question : The standard loss formulation in both theory and practice has time discretization ∆ = 0 , corresponding to an integral/expectation over time. See Equation (17) in [17] and Equation (7) in [43]. We consider a fine approximation of the integral with ∆ = O (1 /d 3 ) due to purely technical reasons since a log log (1 / ∆) term appears in the error bound. Note that ∆ can be made much smaller than O ( 1 /d 3 ) due to dependence. In fact, due to assumption 1, the difference between the integral (i.e, ∆ = 0 ) and our loss is O ( √ ∆) . Therefore, our framework allows for considering the standard integral formulation of the loss as well.

Optimization question : The integral or the large sum in the loss function is computationally intractable to optimize directly. Hence stochastic optimization algorithms such as SGD or Adam are used. Here random batches of datapoints are drawn to evaluate the stochastic gradients. When the times are sampled randomly, we obtain unbiased estimators for the gradients of this loss, even when (or when it is very small). Therefore, practical training can be performed efficiently even when ∆ = O (1 /d 3 ) .

We further note that this does not adversely affect inference since Theorem 3 shows that one can use larger step sizes during inference without deteriorating quality.

Remark 3. The parameter B arises from our martingale-based concentration analysis, which involves subGaussian random variables whose subGaussianity parameters are themselves random. We show that with high probability, these parameters are uniformly O (exp( B )) via a union-bound, leading to the log ( B ) factor. Refer to Lemma 17 in the Appendix for the detailed argument.

Theorem 1 is the first step in proving the expectation bound in Theorem 2 and may be of independent interest. Theorem 2 deals with the noise arising from the data x t ∼ p t . Our next assumption, called 'hypercontractivity', controls the 4 th -moment of the error bound with respect to the 2 nd -moment, which can be used to prove the generalization of the score function in L 2 error. This is a mild assumption, standard in statistics and learning theory under heavy tails [30, 23, 33].

Assumption 2. Let κ > 0 be a fixed constant. Then, for every f ∈ H and x t ∼ p t , we have:

κ 4 can be bounded (up to multiplicative constants) by the kurtosis of f ( t, x t ) -s ( t, x t ) . Assumption 2 follows from the smoothness and strong convexity of neural networks in the parameter space (not x t ), as shown in Lemma 40 in the Appendix. Recent work [32, 52] shows that near the global minimizer of the population loss, many smooth non-convex losses exhibit local strong convexity.

Under Assumptions 1 and 2, we state our main result in Theorem 2. In this result, we use Theorem 1 and handle the noise due to x t ∼ p t in the DSM objective.

Theorem 2 ( L 2 Error Bound) . Let Assumptions 1 and 2 hold. Fix δ ∈ (0 , 1) . For all j ∈ [ N ] , let t j := ∆ j and γ j := ∆ . Let B := C log ( ( L +1) dmN log ( 1 δ ) / ∆ ) for an absolute constant C > 0 , and let ∆log 3 ( 1 ∆ ) d 3 log 3 (2 d ) log 3 ( 2 Nm δ ) log 3 ( B |H| δ ) ≤ 1 and N ∆ ≤ C log( 1 ∆ ) . If

then with probability at least 1 -δ ,

Remark 4. In addition to the sample complexity of Theorem 1, the sample complexity for the generalization bound in Theorem 2 additionally has a factor of κ 2 due to the local strong convexity assumption formalized in Lemma 40.

We note that Theorem 2 considers small values of time discretization ∆ , which is not an issue during training (see Remark 2). However, we can accelerate inference by using a larger timestep-size to discretize the diffusion process, as shown in Theorem 3 and proved in Theorem 5.

Theorem 3 (Fast Inference) . Under the same assumptions as Theorem 2, partition the timesteps { t j = ∆ j } j ∈ [ N ] into k disjoint subsets S 1 , S 2 , . . . , S k , where each subset S i contains timesteps of the form t j = ∆( i + nk ) for n ∈ N . Define γ ′ j := k ∆ for all j in any subset S i . Then, there exists at least one subset S i such that, with probability at least 1 -δ .

The subsets S i allow for a much coarser discretization with differences being k ∆ instead of ∆ . While the error due to discretization of the SDE might become worse, as shown by the bounds in [1], Theorem 3 demonstrates that the score estimation error does not degrade.

Comparison with prior work: [3] and [14] analyze the DSM objective (4) by independently bounding the error at each timestep and applying a union bound over all t ∈ T . [3] assume bounded support over an ℓ 2 ball of radius R and L -Lipschitz score functions, and derive (up to logarithmic factors) the following per-timestep bound using Rademacher complexity:

where R n ( H ) is the Rademacher complexity of H . When using a uniform step size ∆ , this leads to sample complexity scaling as 1 poly (∆) , with at least linear dependence on d , especially since R = O ( √ d ) in practice.

[14] improve the dependence of sample complexity for Wasserstein error, removing smoothness assumptions on the score function and relaxing the ℓ 2 error. They show that learning f ∈ H satisfying the relaxed criterion for each t suffices for sampling, with a per-timestep sample complexity of m ≳ d log ( |H| δ ) /ϵ 2 , again using a union bound over time. Their bound avoids dependence on 1 σ t , but retains linear scaling with d .

[16] study gradient descent for optimizing (4) when s is L -Lipschitz and the target distribution is within an ℓ 2 ball of radius R . They model time as an input and use a kernel regression perspective to jointly learn across timesteps. While conceptually similar to our approach, their sample complexity (Theorem 3.12) still exhibits polynomial dependence on d .

[12] analyze the non-parametric setting under the assumption that the score belongs to a Hölder class with β -smoothness. This leads to exponential sample complexity in d unless β = Ω( d ) , aligning with worst-case lower bounds. In contrast, our work avoids this curse of dimensionality by making more pragmatic assumptions inspired by empirical diffusion model performance: (a) H only needs to approximate the target's score function; (b) mild time regularity; and (c) second-order differentiability of the log-density. These assumptions, also common in prior work [50, 3, 6, 35], enable nearly dimension-free generalization bounds.

4 Technical results

In this section we describe our proof techniques and key technical results. Figure 2 summarizes the key results in this section and how they work together to lead to Theorem 1.

For ease of exposition, we introduce some additional notation. For all timesteps { t j } j ∈ [ N ] , wherever its clear from context, we denote σ t j ≡ σ j and for all samples i ∈ [ m ] , we denote x ( i ) t j ≡ x ( i ) j and

Lemma 1:

Linear-to-Quadratic Reduction

Martingale Decomposition

Lemma 3:

Conditional Sub-Gaussianity

Lemma 4:

Lemma 5:

Lemma 6:

ℓ

∞

→

ℓ

2

Conversion

Theorem 1:

Empirical

ℓ

2

Error Bound

z ( i ) t j ≡ z ( i ) j . For all appropriately defined function f, g , we denote the empirical expectations as ̂ E x j [ f ( x j )] := 1 m ∑ i ∈ [ m ] f ( x ( i ) j ) and ̂ E x j ,z j [ g ( x j , z j )] := 1 m ∑ i ∈ [ m ] g ( x ( i ) j , z ( i ) j ) . For any random variable y , we denote the conditional expectation as E i,j [ y ] := E [ y | x ( i ) j ] .

We start by bounding the empirical squared error in terms of a linear form in Lemma 1. This relates the empirical error, ˆ L of the minimizer ˆ f with the true score function, s . While we assume s ∈ H for simplicity, it can be relaxed to assume that ∃ s ∈ H with sufficiently small ℓ 2 error, similar to [14].

Lemma 1. For f ∈ H ,

Let ˆ L be as defined in (4) . If s ∈ H then for f = arg inf f ∈H ˆ L ( f ) , we have

Let ˆ f be the minimizer of ˆ L ( f ) . Lemma 1 (proved in Lemma 35) bounds L ( ˆ f ) , the loss of ˆ f against the true and unknown score function with H ˆ f . We will show a high probability bound on H ˆ f defined in (7) to control L ( ˆ f ) . Interestingly, as shown in Lemma 2 (and proved in Lemma 20), for a fixed f it is possible to decompose H f as a martingale difference sequence.

The martingale difference decomposition of H f , exploiting the Markovian structure of (1), has terms of the form Q i := ⟨ G i , Y i -E [ Y i |F i -1 ] ⟩ adapted to the filtration {F i } i ∈ [ n ] , where G i is a F i -1 measurable random variable. The proof primarily uses the fact that for t 1 ≤ t 2 ≤ t 3 , E [ x t 1 | x t 2 , x t 3 ] = E [ x t 1 | x t 2 ] due to the Markov property.

Lemma 2. Let ζ := s -f m for any f ∈ H . Define

and define R i,k as

and R i,k = 0 for k = 0 . Define t 0 = 0 . Consider the filtration defined by the sequence of σ -algebras, F i,k := σ ( { x ( j ) j : 1 ≤ j < i, j ∈ [ N ] } ∪ { x ( i ) j : j ≥ N -k } ) , for i ∈ [ m ] and k ∈ { 0 , . . . , N } , satisfying the total ordering { ( i 1 , j 1 ) < ( i 2 , j 2 ) iff i 1 < i 2 or i 1 = i 2 , j 1 < j 2 } . Then,

For k ∈ [ N -1] , G i,k +1 is measurable with respect to F i,k -1 , and ¯ G i is F N -1 -measurable.

For i ∈ [ m ] , k ∈ { 0 }∪ [ N ] , { R i,k } ( i,k ) forms a martingale difference sequence with respect to the filtration above.

H f = ∑ i ∈ [ m ] ∑ k ∈ [ N ] R i,k , where H f is defined in Lemma 1

In the above Lemma 2 (and proved in Lemma 20), R ik denotes the martingale difference sequence arising from the Doob decomposition (see e.g. [9]). Our aim is to bound H ˆ f by bound H f uniformly for every f , using martingale concentration. In the next lemma, we show that conditioned on F i -1 , Q i is subGaussian. To gain intuition into how subGaussianity comes into play in our context, we note that Lemma F.3. in [14] shows that the score function, s ( t, x t ) , is 1 /σ t -subGaussian. We develop a more fine-grained argument exploiting the smoothness of the score function to show subGaussianity (Definition 1) for our sequence. The proof is provided in Lemma 30.

Lemma 3. Fix δ ∈ (0 , 1) . Consider R i,k and F i,k as defined in Lemma 2 and let ∆ := t N -k +1 -t N -k . Under Assumption 1, following the definition in Definition 1, conditioned on F i,k -1 , R i,k is ( β 2 i,k ∥ G i,k ∥ 2 , W i,k ) -subGaussian where β i,k , W i,k are F i,k -1 measurable random variables such that W i,k ≤ log ( 2 δ ) with probability at-least 1 -δ and

However, the subGaussianity parameters in Lemma 3, depend polynomially on the data dimension, d along with G i and the step size, ∆ . Solely relying on this leads to a dimension-dependent bound. To further refine our analysis and show a dimension-free bound, we evaluate the variance of Q i conditioned on F i -1 . As shown in the next Lemma (Lemma 4) (and proved in Lemma 29), the variance depends only on the smoothness parameter, L , along with G i and ∆ .

Lemma 4 (Variance bound for martingale difference sequence) . Consider the martingale difference sequence R i,k and the predictable sequence G i,k +1 with respect to the filtration F i,k from Lemma 21.

where C > 0 is an absolute constant and ν 2 i,k = 0 for k = 0 .

The proof of Lemma 4 is involved when h t ( x ) := ∇ 2 log ( p t ) ( x ) is not assumed to be Lipschtiz in x . Starting with the martingale difference sequence defined in Lemma 2, an application of the second-order tweedie's formula (see Lemma 22), reduces the problem to bounding the operator norm Cov ( s ( t ′ , x t ′ ) | x t ) for t -t ′ = ∆ > 0 , i.e, the conditional covariance matrix of the score function given the future. Exploiting the smoothness assumption on the score function, an application of the mean value theorem reduces our problem to bounding the operator norm of:

for x t ′ , ˜ x t ′ i.i.d conditioned on x t and y t ′ = λx t ′ + (1 -λ )˜ x t ′ , λ ∈ (0 , 1) . Notice that y t ′ | x t is dependent on x t ′ , ˜ x t ′ | x t , which does not allow the use of Tweedie's second-order formula (Lemma 22) to bound E [ ( x t ′ -˜ x t ′ )( x t ′ -˜ x t ′ ) ⊤ | x t ] and derive variance bounds that are dimension-free. To approximately allow this argument, we decompose h t ′ ( y t ′ ) into two components:

The first term, h t ′ ,ϵ ( y t ′ ) , represents a hessian after being smoothed with an appropriately chosen distribution, which we show satisfies Lipschitz continuity. This allows us to approximate h t ′ ,ϵ ( y t ′ ) ≈

h t ′ ,ϵ ( e ∆ x t ) and bound the variance with Tweedie's second order formula. The second term, which represents the deviation between the original and mollified Hessians, is bound using Lusin's theorem (Lemma 27) to provide approximate uniform continuity for h t ′ , as developed further in Lemma 28.

Putting together Lemma 3 and Lemma 4, we provide a general concentration tool for martingale difference sequences with bounded variance and subGaussianity in Lemma 5. We follow a similar proof strategy via a supermartingale argument as in the proof Freedman's inequality (see for e.g. [47]), but diverge in dealing with subGaussianity instead of almost surely bounded random variables.

Lemma 5. Let M n = ∑ n i =1 ⟨ G i , Y i -E [ Y i |F i -1 ] ⟩ , M 0 = 1 and the filtration {F i } i ∈ [ n ] be such that G i is F i -1 measurable and

⟨ G i , Y i -E [ Y i |F i -1 ] ⟩ is ( β 2 i ∥ G i ∥ 2 , K i ) sub-Gaussian conditioned on F i -1 (where β i , K i are random variables measurable with respect to F i -1 )

Pick a λ > 0 and let A i ( λ ) = { λJ i ∥ G i ∥ β i √ K i ≤ c 0 } for some small enough universal constant c 0 . Then, there exists a universal constant C > 0 such that

Observe that the concentration result developed in Lemma 5 (and proved in Lemma 16) has two parts. Optimizing over the choice of λ , it can be shown that the bound on M n depends on two terms: (1) an ℓ 2 term, ∑ i ∈ [ n ] ν 2 i ∥ G i ∥ 2 and (2) an ℓ ∞ term, sup i ∈ [ n ] J i ∥ G i ∥ β i √ K i . When applied in our context, these two terms in turn depend on norms, ∥ f -s ∥ 2 and ∥ f -s ∥ ∞ . This is where the time-regularity assumption in Assumption 1 plays a crucial role in our analysis. Specifically, it enables us to bridge the ℓ ∞ and ℓ 2 norm bounds derived from the martingale concentration results in Lemma 5. The proof of Lemma 6 leverages this assumption to relate ∥ f ( t + k ∆ , x t + k ∆ ) ∥ 2 to ∥ f ( t, x t ) ∥ 2 , as shown by:

Exploiting this property over a carefully selected range of k values allows us to relate ℓ ∞ and ℓ 2 norm bounds as we show in the following Lemma.

Lemma 6. Under Assumption 1, with probability 1 -δ , for a universal constant C > 0 the following holds uniformly for every f ∈ H :

Lemma 6 establishes that the simultaneous analysis of all timesteps uses the smoothness across time. In the absence of this approach, the smoothness assumption in the x t -space would lack dependence on ∆ and could grow as large as the Lipschitz constant L . This is essential for establishing nearly dimension-independent bounds. The proof of this result can be found in Lemma 39 in the Appendix.

5 Bootstrapped score matching

In Section 4, we used time regularity and could prove nearly d -independent bounds. Learning with the same function class across timesteps and Assumption 1 was critical to our proof.

We now attempt to exploit the dependence across timesteps explicitly and reduce variance in estimation. Using the Markovian nature of (1), we show that for any t ′ < t and α t ∈ R , s ( t, x t ) = E [˜ y t | x t ] for ˜ y t := -z t σ 2 t -α t ( s ( t ′ , x t ′ ) --z t ′ σ 2 t ′ ) . This shows that ˜ y t can also be used to construct a learning target for the score function. This is in contrast to the target y t := -z t σ 2 t used in (4). The advantage of ˜ y t over y t is in the lower variance of ˜ y t , as shown in Lemma 7 (proved in Lemmas 42, 43).

To compare with y t = -z t σ 2 t , we note that an application of the second order tweedie's formula along with Assumption 1 shows the variance ∥ ∥ E [( y t -s ( t, x t ))( y t -s ( t, x t )) ⊤ | x t ] ∥ ∥ op to be of the order O ( L +1 σ 2 t ) . Therefore, although both y t and ˜ y t are unbiased, the variance of ˜ y t has an additional step size ( ∆ ) factor in the numerator (see Lemma 7)

The BSM algorithm (described in detail in Appendix G) operates sequentially over a discretized time horizon 0 = t 0 < t 1 < · · · < t N = T and builds upon the principles of DSM while introducing a novel bootstrapping mechanism to mitigate the increasing variance of the DSM loss in later timesteps. Given a dataset D = { x ( i ) 0 } i ∈ [ m ] sampled from the data distribution, the perturbed samples at timestep t k are generated as x ( i ) t k = x ( i ) 0 e -t k + z ( i ) t k , z ( i ) t k ∼ N (0 , σ 2 t k I ) where σ 2 t k = 1 -e -2 t k . The task at each timestep t k is to estimate an approximate score function ˆ s t k ( x ) to optimize E x t k [ ∥ s ( t k , x ) -ˆ s t k ( x ) ∥ 2 2 ] . For the initial timesteps t k with k ≤ k 0 , the algorithm employs DSM.

The score function ˆ s t k is obtained by solving ˆ s t k = arg min f ∈H k ∑ i ∈ [ m ] 1 m ∥ ∥ f ( t k , x ( i ) t k ) --z ( i ) t k σ 2 t k ∥ ∥ 2 2 . For later timesteps t k with k > k 0 , the algorithm transitions to BSM. At each timestep, the algorithm constructs bootstrapped targets ˜ y ( i ) t k by combining the DSM target -z ( i ) t k σ 2 t k with the previously estimated score ˆ s t k -1 . Specifically, the targets are defined as:

where α k = e -γ k √ 1 -e -2 t k -1 1 -e -2 t k , with γ k = t k -t k -1 . Given access to the true score function, s ( t k -1 , . ) , then ˜ y ( i ) t k would form an unbiased target with lower variance, as shown in Lemma 7. However, since we only have access to the estimated score function, ˆ s t k -1 at the previous timestep, ˜ y ( i ) t k is a biased target, and the parameter α k weighs between the biased and unbiased targets. The score function, ˆ s t k , is then learned as: ˆ s t k ← arg min f ∈H k ∑ i ∈ [ m ] 1 m ∥ ∥ f ( t k , x ( i ) t k ) -˜ y ( i ) t k ∥ ∥ 2 2 .

6 Conclusion

To our knowledge, this is the first work to establish (nearly) dimension-free sample complexity bounds for learning score functions across noise levels. We show that a mild assumption of time-regularity can significantly improve over previous bounds which have polynomial dependence on d . We achieve this with a novel martingale-based analysis with sharp variance bounds, addressing the complexities of learning from dependent data generated by multiple Markov process trajectories. Furthermore, we introduce the Bootstrapped Score Matching (BSM) method, which effectively leverages temporal information to reduce variance and enhance the learning of score functions. While we provide theoretical insights into the training of diffusion models, several open questions still remain. One potential direction is extending our framework to flow-matching models where such bounds could yield further insights. Additionally, while BSM is a compelling algorithm, establishing rigorous theoretical and empirical performance guarantees is an open problem which we leave for future work.

Acknowledgments and Disclosure of Funding

We gratefully acknowledge NSF grants 2217069, 2019844, and DMS 2109155. Additionally, part of this work was carried out while Syamantak was an intern at Google DeepMind.

References

Joe Benton, Valentin De Bortoli, Arnaud Doucet, and George Deligiannidis. Nearly d-linear convergence bounds for diffusion models via stochastic localization. In The Twelfth International Conference on Learning Representations , 2024.

Jalaj Bhandari, Daniel Russo, and Raghav Singal. A finite time analysis of temporal difference learning with linear function approximation. In Conference on learning theory , pages 16911692. PMLR, 2018.

Adam Block, Youssef Mroueh, and Alexander Rakhlin. Generative modeling with denoising auto-encoders and langevin sampling. arXiv preprint arXiv:2002.00107 , 2020.

Nicholas M Boffi, Arthur Jacot, Stephen Tu, and Ingvar Ziemann. Shallow diffusion networks provably learn hidden low-dimensional structure. arXiv preprint arXiv:2410.11275 , 2024.

Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. In International Conference on Machine Learning , pages 4672-4712. PMLR, 2023.

Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint arXiv:2209.11215 , 2022.

Valentin De Bortoli, Michael Hutchinson, Peter Wirnsberger, and Arnaud Doucet. Target score matching. arXiv preprint arXiv:2402.08667 , 2024.

John C Duchi, Alekh Agarwal, Mikael Johansson, and Michael I Jordan. Ergodic mirror descent. SIAM Journal on Optimization , 22(4):1549-1578, 2012.

Rick Durrett. Probability: theory and examples , volume 49. Cambridge university press, 2019.

Zach Evans, CJ Carr, Josiah Taylor, Scott H. Hawley, and Jordi Pons. Fast timing-conditioned latent audio diffusion. In Forty-first International Conference on Machine Learning , 2024.

Gerald B. Folland. Real Analysis: Modern Techniques and Their Applications . John Wiley & Sons, 1999.

Hengyu Fu, Zhuoran Yang, Mengdi Wang, and Minshuo Chen. Unveil conditional diffusion models with classifier-free guidance: A sharp statistical theory, 2024.

Ishaan Gulrajani and Tatsunori B Hashimoto. Likelihood-based diffusion language models. Advances in Neural Information Processing Systems , 36, 2024.

Shivam Gupta, Aditya Parulekar, Eric Price, and Zhiyang Xun. Improved sample complexity bounds for diffusion model training. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.

Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. Ssd-lm: Semi-autoregressive simplexbased diffusion language model for text generation and modular control. arXiv preprint arXiv:2210.17432 , 2022.

Yinbin Han, Meisam Razaviyayn, and Renyuan Xu. Neural network-based score estimation in diffusion models: Optimization and generalization. arXiv preprint arXiv:2401.15604 , 2024.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

Emiel Hoogeboom, Vıctor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International conference on machine learning , pages 8867-8887. PMLR, 2022.

Chenqing Hua, Sitao Luan, Minkai Xu, Zhitao Ying, Jie Fu, Stefano Ermon, and Doina Precup. Mudiff: Unified diffusion for complete molecule generation. In Learning on Graphs Conference , pages 33-1. PMLR, 2024.

Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research , 6(4), 2005.

Naoto Inoue, Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi. Layoutdm: Discrete diffusion model for controllable layout generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10167-10176, 2023.

Chi Jin, Praneeth Netrapalli, Rong Ge, Sham M Kakade, and Michael I Jordan. A short note on concentration inequalities for random vectors with subgaussian norm. arXiv preprint arXiv:1902.03736 , 2019.

Adam Klivans, Pravesh K Kothari, and Raghu Meka. Efficient algorithms for outlier-robust regression. In Conference On Learning Theory , pages 1420-1430. PMLR, 2018.

Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations , 2021.

Suhas Kowshik, Dheeraj Nagaraj, Prateek Jain, and Praneeth Netrapalli. Near-optimal offline and streaming algorithms for learning non-linear dynamical systems. Advances in Neural Information Processing Systems , 34:8518-8531, 2021.

Syamantak Kumar and Purnamrita Sarkar. Streaming pca for markovian data. Advances in Neural Information Processing Systems , 36, 2024.

Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence of score-based generative modeling for general data distributions. In International Conference on Algorithmic Learning Theory , pages 946-985. PMLR, 2023.

Elad Levi, Eli Brosh, Mykola Mykhailych, and Meir Perez. Dlt: Conditioned layout generation with joint discrete-continuous diffusion layout transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2106-2115, 2023.

Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. 2023.

Shahar Mendelson and Nikita Zhivotovskiy. Robust covariance estimation under l_4-l_2 norm equivalence. 2020.

Chenlin Meng, Yang Song, Wenzhe Li, and Stefano Ermon. Estimating high order gradients of the data distribution by denoising. Advances in Neural Information Processing Systems , 34:25359-25369, 2021.

Tristan Milne. Piecewise strong convexity of neural networks. Advances in Neural Information Processing Systems , 32, 2019.

Stanislav Minsker. Sub-gaussian estimators of the mean of a random matrix with heavy-tailed entries. The Annals of Statistics , 46(6A):2871-2903, 2018.

Dheeraj Nagaraj, Xian Wu, Guy Bresler, Prateek Jain, and Praneeth Netrapalli. Least squares regression with markovian data: Fundamental limits and algorithms. Advances in neural information processing systems , 33:16666-16676, 2020.

Kazusato Oko, Shunta Akiyama, and Taiji Suzuki. Diffusion models are minimax optimal distribution estimators. In International Conference on Machine Learning , pages 26517-26582. PMLR, 2023.

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 , 2023.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 1(2):3, 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

Walter Rudin. Principles of Mathematical Analysis . McGraw-Hill, 1976.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems , 35:36479-36494, 2022.

Daria Schymura. An upper bound on the volume of the symmetric difference of a body and a congruent copy. Advances in Geometry , 14(2):287-298, 2014.

Max Simchowitz, Horia Mania, Stephen Tu, Michael I Jordan, and Benjamin Recht. Learning without mixing: Towards a sharp analysis of linear system identification. In Conference On Learning Theory , pages 439-473. PMLR, 2018.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning , pages 2256-2265. PMLR, 2015.

Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence , pages 574-584. PMLR, 2020.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

R Srikant. Rates of convergence in the central limit theorem for markov chains, with an application to td learning. arXiv preprint arXiv:2401.15719 , 2024.

Joel Tropp. Freedman's inequality for matrix martingales. 2011.

Stephen Tu, Roy Frostig, and Mahdi Soltanolkotabi. Learning from many trajectories. Journal of Machine Learning Research , 25(216):1-109, 2024.

Harshit Varma, Dheeraj Nagaraj, and Karthikeyan Shanmugam. Glauber generative model: Discrete diffusion models via binary classification. arXiv preprint arXiv:2405.17035 , 2024.

Santosh Vempala and Andre Wibisono. Rapid convergence of the unadjusted langevin algorithm: Isoperimetry suffices. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation , 23(7):1661-1674, 2011.

Mingyang Yi, Ruoyu Wang, and Zhi-Ming Ma. Characterization of excess risk for locally strongly convex population risk. Advances in Neural Information Processing Systems , 35:2127021285, 2022.

Ingvar Ziemann and Stephen Tu. Learning with little mixing. Advances in Neural Information Processing Systems , 35:4626-4637, 2022.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We provide a clear description of the scope of our result and a detailed comparison with prior work.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We provide a descriptions of our limitations and a scope for future work as part of the conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide detailed proofs of all claims made in our work.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide a clear description of the experimental setup in the Appendix for reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our primary contributions are theoretical and our experiments are on synthetically generated data for simple data distributions which we describe in detail in the Appendix. We provide code for our experiments in the supplement.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide a clear description of the experimental setup in the Appendix for reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: In our first experiment, we overlay a best-fit linear regression whose slope is virtually zero, thereby substantiating the statistical significance of our near-dimension-free claim.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Our experiments are provided on a single Google Colab CPU. We provide this detail in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our contribution is theoretical, where we show how to analyse the sample of diffusion models to obtain a dimension-free bound.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our contribution is theoretical, where we show how to analyse the sample of diffusion models to obtain a dimension-free bound for the well-known DDPM training algorithm. As such, we do not introduce any new societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release any new models. Our contribution is theoretical.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: There are no new assets introduced in the paper. Our contribution is theoretical.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: There are no new assets introduced in the paper. Our contribution is theoretical.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: There is no human subject study conducted.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: There is no human subject study conducted.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The paper does not use LLMs apart from writing and editing.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

The Appendix is organized as follows:

Section A provides some utility results which will be useful in subsequent proofs.

Section C provides variance calculation for the martingale decomposition.

Section B analyzes concentration properties for martingales with bounded variance and subGaussianity, which may be of independent interest.

Section D analyzes convergence of the empirical squared error by providing the martingale decomposition and exploiting the results developed in Sections C and B.

Section E provides generalization bounds to achieve guarantees for the expected squared error.

Section F provides details for the experiment conducted in Figure 1 in the manuscript.

Section G provides details about the Bootstrapped Score Matching Algorithm described in Section 5.

A Utility Results

Definition 2 (norm subGaussian) . We will call a random vector X ∈ R d to be σ norm subGaussian if E X = 0 and

Definition 3. We will call a random vector X ∈ R d to be σ subGaussian if E X = 0 and for every v ∈ R d and λ ∈ R we have:

Lemma 8. Let X ∼ N ( 0 , σ 2 I ) . Then, X is 2 σ norm subGaussian.

Proof. Consider the random variable y := ∥ X ∥ 2 2 σ 2 . Then, y ∼ χ ( d ) follows the chi-squared distribution with d degrees of freedom. Therefore, for any t < 1 2 ,

Setting t = 1 4 d , we have

Lemma 9. For all t > 0 , x 1 , x 2 ∈ R d , consider any function u : R d → R d satisfying ∥ u ( x 1 ) -u ( x 2 ) ∥ 2 ≤ S ∥ x 1 -x 2 ∥ 2 , where S > 0 is a fixed constant. For timesteps 0 ≤ t ′ < t , consider the random variable

where x t is defined in (1) . Then, q t,t ′ is ϕ √ d norm subGaussian for

where ∆ := t -t ′ .

Proof. We first note that

Using Lemma 1 from [22], we show that E x t ′ ,x t [ exp ( ∥ q t,t ′ ∥ 2 2 ϕ 2 d )] ≤ 2 . Let x ′ t ′ be an iid copy of x t ′ , conditioned on x t . Then, we have,

Note that using (1), x t = e -∆ x t ′ + w t,t ′ = e -∆ x ′ t ′ + w ′ t,t ′ , for w t,t ′ , w ′ t,t ′ ∼ N ( 0 , σ 2 t -t ′ I d ) . Therefore, from (9),

where the last inequality follows since w t,t ′ , w ′ t,t ′ ∼ N ( 0 , σ 2 t -t ′ I d ) marginally (but not necessarily conditionally).

Lemma 10. Fix δ > 0 . Let t > t ′ . Then, under Assumption 1-(1), with probability at least 1 -δ over x t ,

where σ 2 t -t ′ := 1 -e -2 ( t -t ′ ) ≤ 2 ( t -t ′ ) .

Proof. Using Corollary 2.4 from [7],

Using (1),

Where z t,t ′ is independent of x t ′ . Let y t,t ′ := e -( t -t ′ ) s ( t, x t ) -s ( t ′ , e ( t -t ′ ) x t ) . Then,

Note that since z t,t ′ ∼ N ( 0 , σ 2 t -t ′ I ) ,

Therefore, with probability at least 1 -δ over x t :

Using Jensen's inequality,

The result then follows by taking log on both sides.

Lemma11. Let w t,t ′ := z t,t ′ + σ 2 t -t ′ s ( t, x t ) for t > t ′ ≥ 0 . Then, w t,t ′ is ν t,t ′ √ d norm subGaussian for ν t,t ′ := 4 σ t -t ′ .

Proof. Notice that x t = e t ′ -t x t ′ + z t,t ′ Using Tweedie's formula, s ( t, x t ) = -E [ z t,t ′ σ 2 t -t ′ ∣ ∣ ∣ ∣ x t ] . Therefore,

Applying Lemma 9 with u ( x ) = -e t ′ -t x (which is e t ′ -t Lipschitz), we conclude the result.

Lemma 12. Suppose Assumption 1-(1) holds. Let v t,t ′ := E [ x 0 | x t ] -E [ x 0 | x t ′ ] for t > t ′ ≥ 0 . Then, v t,t ′ is ρ t,t ′ √ d norm subGaussian for

Proof. Using Tweedie's formula, for all t > 0 ,

Note that x t ′ = e t -t ′ ( x t -z t,t ′ ) . Furthermore, note that

Therefore, we have

Using Lemma 11, T 1 is 4 e t σ t -t ′ √ d norm subGaussian. Using Lemma 9, T 2 is 4 Le t -t ′ e t ′ σ 2 t ′ σ t -t ′ √ d = 4 Le t σ 2 t ′ σ t -t ′ √ d norm subGaussian. Therefore, the result follows using the sum of subGaussian random variables.

Lemma 13. Let ∆ > 0 and ∆ < c 0 for some universal constant c 0 . Then,

Proof. Let us start with the first bound. We have,

Consider the function f ( x ) := 1 1 -e -2∆ x . Then, f ( x ) is positive, convex and decreasing. Therefore,

which completes the first result. Now for the second result,

Consider the function, g ( x ) := e -2∆ x (1 -e -2∆ x ) 2 . For x > 0 , g ( x ) is a positive, decreasing and convex function. Therefore,

which completes the proof. Finally for the third result, consider the function h ( x ) := e -∆ x 1 -e -2∆ x . For x > 0 , h ( x ) is a positive, decreasing and convex function. Therefore,

B Martingale Concentration

Lemma 14. Let Y be a ( β 2 , K ) -subGaussian random variable following definition 1, with ( K ≥ 1) . Then, for any integer k > 0 and some universal constant C > 0 :

Proof. By Definition 1, for any A > 0 ,

Using the tail-integration representation of moments, we have

Make the change of variables t = x 2 k so that dt = 2 k x 2 k -1 dx . Then

Let x 0 = √ 2 β 2 K

In the second step we have used the fact that whenever x ≥ x 0 , we must have K -x 2 2 β 2 ≤ -( x -x 0 ) 2 2 β 2 . In the third step we have used the fact that x 2 k -1 ≤ 2 2 k -2 [( x -x 0 ) 2 k -1 + x 2 k -1 0 ] whenever x ≥ x 0 .

A standard Gamma-function integral yields

and for integer k , Γ( k ) = ( k -1)! . Substituting this to the equation above, we conclude that for some universal constant C 1 , we have:

We then conclude the result using the fact that K ≥ 1 and k ≤ 2 k .

Lemma 15. Let Y be a ( β 2 , K ) -subGaussian random variable following definition 1, such that K ≥ 1 , E [ Y ] = 0 and E [ Y 2 ] ≤ ν 2 . Then, for a sufficiently small universal constant c 0 > 0 such that, λβ ≤ c 0 , and any arbitrary A > 0 , we have:

Proof. For some λ > 0 , consider:

Now, using Lemma 14, consider

Here, we have used the fact that (2 k )! ≤ 4 k ( k !) 2 . Plugging this back in Equation (12), we conclude that whenever λβ ≤ c 0 for some small enough constant c 0 , we have:

Theorem 4. Let Y be a ( β 2 , K ) -subGaussian random variable following definition 1, such that K ≥ 1 , E [ Y ] = 0 and E [ Y 2 ] ≤ ν 2 . Set A ≥ β √ 4 log( βK ν ) + β √ 2 K and λ ≤ c 0 A for some small enough constant c 0 > 0 . Then, there exists a constant C such that:

Proof. The result follows from Lemma 15 substituting the values of λ and A .

Lemma 16. Let M n = ∑ n i =1 ⟨ G i , Y i -E [ Y i |F i -1 ] ⟩ , M 0 = 1 and define the filtration {F i } i ∈ [ n ] such that:

G i is F i -1 measurable.

⟨ G i , Y i -E [ Y i |F i -1 ] ⟩ is ( β 2 i ∥ G i ∥ 2 , K i ) sub-Gaussian conditioned on F i -1 (where β i , K i are random variables measurable with respect to F i -1 )

Pick a λ > 0 and let A i ( λ ) = { λJ i ∥ G i ∥ β i √ K i ≤ c 0 } for some small enough universal constant c 0 . Then, there exists a universal constant C > 0 such that:

exp( λM n -Cλ 2 ∑ n i =1 ν 2 i ∥ G i ∥ 2 ) ∏ n i =1 ✶ ( A i ( λ )) is a super-martingale with respect to the filtration F i

Proof. Let L n := exp( λM n -Cλ 2 ∑ n i =1 ν 2 i ∥ G i ∥ 2 ) ∏ n i =1 ( A i ( λ )) . Then we have,

The second result follows from a standard Chernoff bound argument.

Lemma 17. Under the setting of Lemma 16, let λ ∗ := √ α ∑ n i =1 ν 2 i ∥ G i ∥ 2 and λ min := c 0 sup i J i ∥ G i ∥ β i √ K i . Let B ∈ N be arbitrary and consider the event: B = { e -B ≤ min( λ ∗ , λ min ) ≤ max( λ ∗ , λ min ) ≤ e B } . Then, for some universal constant C 1 > 0 and any α > 0 ,

Proof. We apply union bound over λ ∈ Λ B := { e -B , e -B +1 , . . . , e B } . Using Lemma 16 along with a union bound,

Consider the following events:

Event 1: E 1 := { max( λ ∗ , λ min ) > e B }

Event 2: E 2 := { min( λ ∗ , λ min ) < e -B }

Event 3: E 3 := { e -B ≤ λ ∗ < λ min ≤ e B }

Event 4: E 4 := { e -B ≤ λ min < λ ∗ ≤ e B }

In the event E 4 , almost surely there exists a random ¯ λ ∈ Λ B such that ¯ λ/λ min ∈ [ 1 e , e ] and such that the event ∩ n i =1 A i ( ¯ λ ) holds. Thus, we have:

Similarly, under the event E 3 , there exists a random ¯ λ ∗ ∈ Λ B such that: ¯ λ ∗ /λ ∗ ∈ [ 1 e , e ] , such that the event ∩ i A i ( ¯ λ ∗ ) holds. Therefore, we must have:

Notice that λ ∗ is chosen such that

Combining these equations, we conclude that for some constant C 1 > 0 , we must have

Noting that B = E 3 ∪ E 4 , we conclude the result.

C Martingale Decomposition and Variance Calculation

In this section, we will consider the quantity similar to H f in Lemma 35, decompose it into a sum of martingale difference sequence, and then bounds its variance using the Tweedie's formula. In this section, assume that we are given ζ : R + × R d → R d and consider the quantity:

We suppose that ζ ( t, x ( i ) t ) has a finite second moment. Where γ t > 0 is some sequence. When ζ = s -f m , this yields us H f as we show in Lemma 19. We define the sigma algebras: σ -algebra F j = σ ( x ( i ) t : 1 ≤ i ≤ m,t ≥ t N -j +1 ) for j ∈ [ N ] and F 0 is the trivial σ -algebra. We want to filter H through the filtration F j to obtain a martingale decomposition. To this end, define:

Lemma 18.

such that

Proof. By Tweedie's formula, notice that y i t -s ( t, x ( i ) t ) = E [ z ( i ) t | x ( i ) t ] -z ( i ) t σ 2 t . This shows us that H f = H when we pick ζ = s -f m . The proof follows due to Lemma 18 once we note that H 1 = 0 almost surely

Lemma 20. Define ¯ G i := ∑ N j =1 γ j e -( t j -t 1 ) ζ ( t j ,x ( i ) t j ) σ 2 t j , G i,k := ∑ N j = N -k +2 γ j e -t j ζ ( t j ,x ( i ) t j ) σ 2 t j and R i,k as

̸

Proof. 1. Using the fact that x ( i ) t forms a Markov process and that ( x ( i ) s ) s ≥ 0 , ( x ( j ) s ) s ≥ 0 are independent when i = j , we have via the Markov property:

In the second step, we have used the tower property of the conditional expectation. Now, z ( i ) t = x ( i ) t -e -t x ( i ) 0 . By the Markov Property, we have: E [ x ( i ) 0 | x ( i ) t , x ( i ) t j -N +1 ] = E [ x ( i ) 0 | x ( i ) t ] . Plugging this in, we have:

Notice that z ( i ) = x ( i ) -e -t x ( i ) . Clearly, x ( i ) is measurable with respect to F j

t t 0 t . Therefore, ⟩

Now, consider the fact that x ( i ) 0 , x ( i ) t 1 , ... is a Markov chain. Therefore, the Markov property states that x ( i ) 0 | x ( i ) s : s ≥ τ has the same law as x ( i ) 0 | x ( i ) τ . Therefore, we must have: E [ x ( i ) 0 |F j ] = E [ x ( i ) 0 | x ( i ) t j -N +1 ] . Plugging this into the display equation above, we conclude the result.

We connect the quantity H defined above to the quantity H f related to the excess risk.

Lemma 19. Let y ( i ) t := -z ( i ) t σ 2 t , f ∈ H and

Suppose we pick ζ = s -f m in the definition of H . Then,

Let t 0 = 0 . Consider the filtration defined by the sequence of σ -algebras, F i,k := σ ( { x ( j ) t : 1 ≤ j < i, t ∈ T } ∪ { x ( i ) t : t ≥ t N -k } ) for i ∈ [ m ] and k ∈ { 0 , . . . , N } , satisfying the total ordering { ( i 1 , j 1 ) < ( i 2 , j 2 ) iff i 1 < i 2 or i 1 = i 2 , j 1 < j 2 } . Then

For k ∈ [ N -1] , G i,k +1 is measurable with respect to F i,k -1 and ¯ G i if F N -1 measurable.

For i ∈ [ m ] , k ∈ { 0 } ∪ [ N ] , ( R i,k ) i,k forms a martingale difference sequence with respect to the filtration above.

Proof. 1. We first note that for 1 ≤ k ≤ N -1 , σ { x i t : t ≥ t N -k +1 } ⊆ F i,k -1 . Therefore, G i,k +1 is measurable with respect to F i,k -1 . Furthermore, if k = N , then ¯ G i is measurable with respect to F i,k -1 .

First note that R i,k is F i,k measurable.

The case of R i, 0 is straightforward.

This follows from Lemma 19.

Lemma 21. Consider the setting of Lemma 20. Define:

Let Σ i,k := E [ V i,k V ⊤ i,k |F i,k -1 ] . Then, we have:

Proof.

This follows from a straightforward application of Lemma 20.

Let U be any random vector over R d independent of V ∼ N (0 , σ 2 I d ) . Let W = U + V and let p be the density of W , s = ∇ log p and h = ∇ 2 log p . Then, second order Tweedie's formula states (Theorem 1,[31]):

Lemma 22. Let s τ : R d → R d be continuously differentiable for every τ > 0 . Let t ′ < t and x t = e -( t -t ′ ) x t ′ + z t,t ′ where z t,t ′ ∼ N ( 0 , σ 2 t -t ′ I d ) , as defined in Section 2. Then,

where h t ( x t ) := ∇ 2 log ( p t ( x t )) .

Proof. Applying second order Tweedie's formula:

By Markov property, we must have for any measurable function g :

Applying this to (26):

Now, note that x t = e -t x 0 + e t ′ -t z t ′ + z t,t ′ . Taking y 0 = e -t x 0 + z t,t ′ , we have: x t = y 0 + e t ′ -t z t ′ . Therefore, applying the second order Tweedie's formula again, we must have:

That is : E [ z t ′ z ⊤ t ′ | x t ] = e 2( t ′ -t ) σ 4 t ′ s ( t, x t ) s ( t, x t ) ⊤ + e 2( t ′ -t ) σ 4 t ′ h t ( x t ) + σ 2 t ′ I . Substituting this in Equation (27), we have:

Lemma 23. Let s τ : R d → R d be continuously differentiable for every τ > 0 . For t > t ′ > 0 , let v t,t ′ := E [ x 0 | x t ] -E [ x 0 | x t ′ ] , then,

where h t ( x t ) := ∇ 2 log ( p t ( x t )) is the hessian of the log-density function.

Proof. Using Tweedie's formula, for all t > 0 ,

Note that x t ′ = e t -t ′ ( x t -z t,t ′ ) . Furthermore, note from Tweedie's formula and Corollary 2.4 [7] that:

Therefore, we have

Then, using Lemma 22 and the fact that ( a + b )( a + b ) ⊤ ⪯ 2 aa ⊤ +2 bb ⊤ :

we adopt a strategy of partitioning the interval [ t ′ , t ] into smaller subintervals. Specifically, we divide [ t ′ , t ] as t ′ = τ 0 < τ 1 < · · · < τ B -1 < t = τ B , where B ≥ 1 . By leveraging the smoothness of the score function s τ ( x ) over each subinterval [ τ i , τ i +1 ] , we express the deviations between s τ i and s τ i +1 in terms of the Hessian, h τ ( x ) := ∇ 2 log p τ ( x ) . This decomposition allows us to quantify the overall deviation of the score function across the interval [ t ′ , t ] in terms of contributions from each subinterval, controlled by the Hessian, h τ ( x ) . The following lemma formalizes this approach, establishing an upper bound for the given operator norm in terms of the Hessian and a carefully constructed decomposition. This result will serve as the foundation for subsequent analysis.

Lemma 24. Let s τ : R d → R d be continuously differentiable for every τ > 0 . Let B ∈ N and let τ 0 := t ′ < τ 1 < τ 2 < · · · < τ B -1 < t := τ B for B ≥ 1 and define ∀ t, h t ( x t ) := ∇ 2 log ( p t ( x t )) . Then,

where ˜ x τ i is an independent copy of x τ i when conditioned on x τ i +1 . λ i is uniformly distributed over [0 , 1] independent of the random variables defined above and x τ i ,λ i := λ i x τ i +(1 -λ i ) ˜ x τ i .

Proof. Let ∀ i ∈ [0 , B -1] , ∆ i := τ i +1 -τ i . Then,

Therefore,

̸

Therefore,

Note that E [ s ( τ i , x τ i ) | x τ i +1 ] = e -( τ i +1 -τ i ) s ( τ i +1 , x τ i +1 ) . Therefore,

Using the fundamental theorem of calculus, for x τ i ,λ i := λ i x τ i +(1 -λ i ) ˜ x τ i , λ ∈ (0 , 1) , we have,

Substituting in (29) and using the fact that c i ≤ 1 completes our proof.

We aim to derive a sharp bound on the quantities stated in the previous lemma. Since the Hessian is not assumed to be Lipschitz continuous, directly bounding these quantities can be challenging. To address this, we employ a mollification technique. Mollification smooths a function by averaging it over a small neighborhood, effectively regularizing it to ensure desirable continuity properties. This approach is particularly useful when dealing with functions that may not be smooth or Lipschitz continuous, as it allows us to derive meaningful bounds by working with the mollified version of the function.

In our case, the Hessian is mollified by integrating over a uniformly distributed random variable on a small ball of radius ϵ . This process ensures that the mollified Hessian exhibits controlled variation, enabling us to bound the difference between its values at two points x and y . The following lemma formalizes this construction and provides a bound on the operator norm of the difference between the mollified Hessians at x and y .

Lemma 25. Let h : R d → R d × d such that ∀ x ∈ R d , ∥ h ( x ) ∥ op ≤ L . Let z be uniformly distributed over the unit L 2 ball. For ϵ > 0 , define h ϵ ( x ) := E z [ h ϵ ( x + ϵz )] . Then, for all x, y ∈ R d ,

Proof. Define B ( a, R ) be the ball of radius R around a . Define the set B ( x, ϵ ) ∩ B ( y, ϵ ) = S and denote dµ ϵ to be the lebesgue measure over B (0 , ϵ ) . Then,

Using Theorem 1 from [41], we have

We have for B (0 , ϵ ) , Surf ( B (0 ,ϵ )) Vol ( B (0 ,ϵ )) = d/ϵ

which completes our result.

Therefore,

Lemma 25 demonstrates that the mollified Hessian h ϵ becomes Lipschitz due to the smoothing introduced by the uniform averaging over the ball z , even though the original Hessian h does not have this property. This insight is crucial when dealing with expressions such as

which arise from Lemma 24.

When t and t ′ are close, one would hope to exploit the smoothness of the Hessian h t with respect to time. Specifically, if h t were smooth in the time parameter, this would allow the expectation to move inside, enabling the use of Tweedie's second-order formula (Lemma 22) to derive variance bounds that are dimension-free and independent of strong assumptions on the Hessian.

However, directly imposing such strong assumptions on the Hessian is restrictive. To address this, we decompose the Hessian h t ′ ( x t ′ ,λ ) into two components:

Here, the first term, h t ′ ,ϵ ( x t ′ ,λ ) , leverages the Lipschitz continuity of the mollified Hessian and can be analyzed by conditioning on x t . The second term, which represents the deviation between the original and mollified Hessians, requires a finer analysis that draws upon Lusin's theorem, as developed further in Lemma 28.

The decomposition allows us to systematically address each term: - The Lipschitz property of h t ′ ,ϵ helps bound the first term cleanly. - The second term is bounded using probabilistic arguments based on the regularity properties introduced by mollification.

The following lemma formalizes this decomposition and provides the necessary bounds to proceed with the analysis.

Lemma 26. Suppose Assumption 1-(0) and (1) hold. Let t > t ′ > 0 and define the following quantities:

Let ˜ x t ′ be an independent copy of x t ′ when conditioned on x t .

Let λ ∼ Unif (0 , 1) independent of the variables above.

Let x t ′ ,λ := λx t ′ +(1 -λ ) ˜ x t ′ , ˜ z t,t ′ := x t -e -( t -t ′ ) ˜ x t ′ .

Let h t ′ ( · ) := ∇ 2 log ( p t ′ ( · )) .

For z be uniformly distributed over the unit L 2 ball and ϵ > 0 , define h t ′ ,ϵ ( x ) := E z [ h t ′ ( x + ϵz )] .

Then, there exists a random d × d matrix M such that ∥ M ∥ op ≤ 2 Ld ϵ ∥ (1 -λ ) z t,t ′ + λ ˜ z t,t ′ ∥ 2 and

Proof. By assumption, we have ∀ x ∈ R d , ∥ h t ( x ) ∥ 2 ≤ L . Note that conditioned on x t , we have

Where ˜ z t,t ′ ∼ N (0 , σ 2 t,t ′ I d ) marginally. Therefore,

Using Lemma 25,

Then,

Let q t := E λ,x t ′ , ˜ x t ′ [ h t ′ ( x t ′ ,λ ) ( x t ′ -˜ x t ′ ) ( x t ′ -˜ x t ′ ) ⊤ h t ′ ( x t ′ ,λ ) ⊤ | x t ] and g t ′ ,ϵ ( x t ′ ,λ ) := ( h t ′ ( x t ′ ,λ ) -h t ′ ,ϵ ( x t ′ ,λ )) . Then, using the fact that ( a + b + c )( a + b + c ) ⊤ ⪯ 3( aa ⊤ + bb ⊤ + cc ⊤ ) for arbitrary vectors a, b, c ∈ R d , we have:

Let's first deal with T 1 . We use the fact that x t = e -( t -t ′ ) x t ′ + z t,t ′ = e -( t -t ′ ) ˜ x t ′ + ˜ z t,t ′ along with first order and second order Tweedie's formula in Lemma 22

Now, for T 2 , we have

and similarly for T 3 ,

which completes our proof.

Lemma 27 provides a corollary of Lusin's theorem (see for e.g. [11]) to assert that any measurable function, such as the Hessian h t ( x ) = ∇ 2 log p t ( x ) , can be approximated uniformly on a compact subset G γ ⊆ [ t ′ , t ] × F , where the excluded measure is arbitrarily small. This result ensures that h t ( x ) is uniformly continuous on G γ , with its continuity quantified by a modulus of continuity ω γ ( · ) depending only on γ . See [39] for Heine-Cantor theorem which implies uniform continuity due to compactness.

Lemma 27 (Corollary of Lusin's Theorem) . Let F be a convex, compact set over R d and Λ be the Lebesgue measure. Let h t ( x ) = ∇ 2 log p t ( x ) be measurable. For any γ > 0 , there exists a compact set G γ ⊆ [ t ′ , t ] × F such that Λ([ t ′ , t ] × F ) \ G γ ) < γ and ( t, x ) → h t ( x ) is uniformly continuous over G γ . Let us call the corresponding modulus of continuity as ω γ () , which depends only on γ .

Building on Lemma 27, Lemma 28 aims to bound the fourth moment of the operator norm of the difference h τ i ( x τ i ,λ ) -h τ i ,ϵ ( x τ i ,λ ) , which arises from the deviation between the Hessian and its mollified counterpart. To achieve this, the interval [ t ′ , t ] is partitioned into smaller subintervals τ 0 , τ 1 , . . . , τ B , allowing the analysis to proceed incrementally. The lemma exploits the uniform continuity of h t ( x ) on G γ to tightly control this difference using the modulus of continuity ω γ ( ϵ ) . Contributions from outside the compact subset G γ are accounted for separately using indicator functions, with their impact controlled by the boundedness of the Hessian, ∥ h t ( x ) ∥ op ≤ L . The resulting bound consists of two key terms: a primary term proportional to Bω γ ( ϵ ) 4 , capturing the uniform continuity of the Hessian on G γ , and a residual term proportional to the probability of h t ( x ) lying outside G γ , which is effectively managed by the boundedness assumption. This decomposition is crucial for controlling the variance of the Hessian and ensuring the residual terms remain small.

Lemma 28. Fix a B ∈ N . Let τ 0 := t ′ < τ 1 < τ 2 < · · · < τ B -1 < t := τ B . Let Assumption 1(0),(1) hold. Let h t ( x ) , h t,ϵ ( x ) be defined as in Lemma 26. Let Z be uniformly distributed on the unit L 2 ball in R d , independent of everything else. Then for any γ > 0 :

where x τ i is an i.i.d copy of ˜ x τ i conditioned on x τ i +1 and x τ i ,λ := λx τ i +(1 -λ ) ˜ x τ i for any given λ ∈ [0 , 1] and ω γ , G γ are as defined in Lemma 27.

Proof. Let us consider Lusin's theorem (Lemma 27) over [ t ′ , t ] × F endowed with the Lebesgue measure Λ . By Assumption 1-(0),(1): we have ∥ h t ( x ) ∥ ≤ L for every t almost everywhere under the Lebesgue measure on R d . We denote E λ ∼ Unif (0 , 1) as E λ and only in the set of equations below, we denote expectation with respect to x τ i , ˜ x τ i , Z conditioned on x τ i +1 by ¯ E :

Therefore, we must have:

The following lemma consolidates the results and arguments developed so far to provide a variance bound for a martingale difference sequence. Our goal is to bound the variance of the terms in the sequence R i,k , which is determined by both the predictable sequence G i,k +1 and the smoothness properties of the score function and its Hessian. To achieve this, we build on several key results:

Lemma 28, which establishes bounds for the difference between the Hessian and its mollified counterpart by leveraging the compactness provided by Lusin's theorem.

Lemma 26, which shows how the mollified Hessian can be used to control variance terms using its Lipschitz properties.

Lemma 24, which provides a decomposition of the conditional variance in terms of contributions from smaller subintervals.

The argument proceeds by partitioning the time interval [ t N -k , t N -k +1 ] into smaller subintervals and analyzing the contributions to the variance over each subinterval. Using mollification and uniform

continuity on compact subsets, we control the deviations arising from the lack of Lipschitz continuity in the Hessian. Furthermore, the variance bounds incorporate the contributions from outside the compact subset, which are managed via Lusin's theorem. By carefully summing these contributions and leveraging smoothing techniques, we arrive at a sharp variance bound that scales with the parameters ∆ (the interval size) and L (the bound on the Hessian)

The final result, formalized in Lemma 29, also uses the second-order Tweedie formula to handle the special case of the last time step ( k = N ) in the martingale sequence. This lemma serves as a culmination of our efforts, combining mollification, decomposition, and smoothness assumptions to derive a practical variance bound that is essential for analyzing the concentration of the martingale difference sequence.

Lemma 29 (Variance bound for martingale difference sequence) . Consider the martingale difference sequence R i,k , predictable sequence G i,k +1 with respect to the filtration F i,k as considered in Lemma 21. Define ∆ := t N -k +1 -t N -k

Proof. Consider the case k ∈ { 1 , . . . , N -1 } . For the sake of clarity, we let t = t N -k +1 , t ′ = t N -k . Then, ∆ = t -t ′ and let B ∈ N . We decompose [ t ′ , t ] as follows:

For ∀ i ∈ [ B ] , τ i ∼ Unif ( I i ) , J ∼ Unif ( { 1 , . . . , B } ) . Given τ i , define the random variables Z, λ, x τ i ,λ , ˜ x τ i ,λ , x τ i as in Lemma 28 and with Z, λ, ( x s ) s ≥ 0 indepenent of ( τ i ) i , J . Define the random variable τ ∗ := τ J , X = x τ ∗ ,λ , X ϵ = x τ ∗ ,λ + ϵZ . Notice that T is uniformly distributed over [ t ′ , t ] .

Let r i := τ i +1 -τ i ≤ ∆ B . Using Lemma 26 along with the Cauchy-Schwarz inequality, we have

Using Lemma 24 along with (34) and Cauchy Schwarz inequality, we have

Using (35) and the observation that E λ i ,x τ i , ˜ x τ i [ ∥ x τ i -˜ x τ i ∥ 4 op | x τ i +1 ] 2 = O ( σ 2 r i d ) = O ( ∆ d B ) , we have

Using Lemma 28,

Notice that none of ω γ , G γ , distribution of T, X depend on B . Therefore pick ϵ → 0 and B →∞ such that 1 Bϵ 2 → 0 and ω γ ( ϵ ) → 0 . ( T, X ϵ ) → ( T, X ) almost surely as ϵ → 0 . Then, we take γ → 0 and argue via continuity of the law of ( T, X ) with respect to Lebesgue measure that P (( T, X ) ̸∈ G γ ) → P (( T, X ) ̸∈ [ t ′ , t ] × F ) . Since F is arbitrary compact convex set, we let F ↑ R d to conclude the following:

Using Lemma 23, we have

The result for k < N then follows due to Lemma 21.

Now, consider the case k = N . Recall Σ i,k defined in Lemma 21.Then by second order Tweedie formula (Lemma 22) we have Σ i,k = σ 4 t 1 h t 1 ( x t 1 ) + σ 2 t 1 I d ≲ ∆ 2 L + ∆ . Combining this with Lemma 21, we conclude the result.

We state a useful corollary which is subsequently useful for time bootstrapping and is implicit in the above proof.

Corollary 1. Let t ′ < t and ∆ := t -t ′ . Then, under Assumption 1,

Proof. The proof is implicit due to (36).

Lemma 30. Fix δ ∈ (0 , 1) . Consider R i,k and F i,k as defined in Lemma 2 and let ∆ := t N -k +1 -t N -k . Under Assumption 1, following the definition in Definition 1, conditioned on F i,k -1 , R i,k is ( β 2 i,k ∥ G i,k ∥ 2 , W i,k ) -subGaussian where β i,k , W i,k are F i,k -1 measurable random variables such that W i,k ≤ log ( 2 δ ) with probability at-least 1 -δ and

Proof. We have,

Since G i,k +1 is measurable with respect to F i,k -1 , set λ := 1 ∥ G i,k +1 ∥ 2 2 ρ 2 k d for ρ k defined in Lemma 11,

Therefore,

Note that Lemma 11 shows that E [ x ( i ) 0 | x ( i ) t N -k +1 ] -E [ x ( i ) 0 | x ( i ) t N -k ] is ρ k d norm subGaussian

Therefore, using Markov's inequality, with probablity atleast 1 -δ ,

Pluggint these equations above, we conclude that with probability at-least 1 -δ , for every α > 0 , we have: P (∣ ∣ ∣ 〈 G i,k +1 , E [ x ( i ) 0 | x ( i ) t N -k +1 ] -E [ x ( i ) 0 | x ( i ) t N -k ] 〉∣ ∣ ∣ ≥ α |F i,k -1 ) ≤ 2 δ exp( -λα 2 ) , which proves the result for k ∈ [ N -1] .

For k = N , we similarly use the definition of ν k Lemma 12,

we have,

The conclusion follows by a similar argument as (37).

Based on the bounds established in Lemma 29 and Lemma 30, we establish the following results. Lemma 31. For j ∈ [ N ] , Let t j := ∆ j and γ j = ∆ . Then, for some universal constant C > 0 the following equations hold:

and

Proof. Define g 2 0 := ( L ∆ 2 +∆+ L 2 ∆) . Applying Lemma 29, we conclude:

Recall β i,k , W i,k as defined in Lemma 30. Applying these results along with the union bound we conclude with probability 1 -δ , the following holds every i, k simultaneously:

We will specialize the setting in Lemma 17 with M n being given by H , the filtration being F ik and the martingale decomposition given in Lemma 20. Similarly, β i corresponds to ( β i,k ) i,k , K i corresponds to ( W i,k ) i,k given in Lemma 30. ν 2 i corresponds to the upper bound on E [ R 2 i,k |F i,k ] in Lemma 29. Therefore, J i corresponds to max(1 , C W i,k log( β 2 i,k W i,k ν 2 i,k )) satisfies J i ≤ C log(2 d ) for some constant C . In this case, the quantity ∑ n i =1 ν 2 i ∥ G i ∥ 2 as given in Lemma 17 corresponds to ∑ E [ R 2 |F i,k -1 ] and it can be bound using Lemma 31:

i,k i,k

Similarly, we adapt λ min , λ ∗ be the random variables defined in Lemma 17 to our case for some arbitrary B ∈ N , α > 1 . This lemma demonstrates the concentration of the quantity H conditioned on the event B := { λ min , λ ∗ ∈ [ e -B , e B ] } . It remains to deal with the following cases:

First, consider the case max( λ min , λ ∗ ) > e B .

Lemma 32. Assume γ t = ∆ , ∆ < c 0 for some universal constant c 0 . Then max( λ min , λ ∗ ) > e B implies

Proof. Using the fact that α > 1 , we note that

By defining G i, 0 = 0 and , we note that σ 2 t N -k +1 e t N -k +1 ( G i,k +1 -G i,k ) = ζ ( t N -k +1 , x ( i ) t N -k +1 ) for k < N and σ 2 t 1 ( ¯ G i -e t 1 G i,N -1 ) = ζ ( t 1 , x t 1 ) . Using the fact that σ 2 t k ≤ 1 for some universal constant c 0 , we conclude that

We now consider the event min( λ ∗ , λ min ) < e -B .

Lemma 33. Assume γ t = ∆ , t j = j ∆ , ∆ < c 0 for some universal constant c 0 , α > 1 . min( λ ∗ , λ min ) < e -B implies:

Proof. It is easy to show that min( λ ∗ , λ min ) < e -B implies:

This implies that there exists i, k such that

We then conclude the result using the fact that σ 2 t k ≥ c 0 ∆

Lemma 34. Assume N ∆ > 1 , ∆ < c 0 for some universal constant c 0 . Assume t j = ∆ j and γ j = ∆ . Let α > 1 and B ∈ N . Let L 2 2 ( ζ ) := ∑ i ∈ [ m ] ,t ∈T ∥ ζ ( t, x ( i ) t ) ∥ 2 , L ∞ ( ζ ) := sup i ∈ [ m ] ,t ∈T ∥ ζ ( t, x ( i ) t ) ∥ . Let σ max := log( 1 ∆ ) log(2 d ) √ d ∆sup i,k W i,k . Then with probability 1 -(2 B +1) e -α , at least one of the following inequalities hold:

1.

2.

Proof. As considered in Lemma 17, define the event B := { λ min , λ ∗ ∈ [ e -B , e B ] } . Applying Lemma 17 to our case with the martingale increments as defined in the discussion above, along with bounds for the quantities ∑ n i =1 ν 2 i ∥ G i ∥ 2 and sup i J i β i √ K i ∥ G i ∥ as developed in Lemma 31, we conclude that:

Almost surely

Define the events B 1 := { max( λ min , λ ∗ ) > e B } , B 2 := { min( λ min , λ ∗ ) < e -B } , A = { H L +1 > C √ αN ∆ 2 L 2 2 ( ζ ) + Cα L ∞ ( ζ ) σ max } . By Lemma 32, the event { L 2 2 ( ζ ) > CNmα ∆ e -2 B } ⊆ B ∁ 1 . By Lemma 33, the event:

Therefore consider complement of the event of interest in the statement of the lemma:

Clearly, P ( B ∪ B 1 ∪ B 2 ) = 1 . This implies P ( B ∁ ∩ B ∁ 1 ∩ B ∁ 2 ) = 0 . Therefore, using the above inclusions along with Equation (41) we conclude:

D Convergence of Empirical Risk Minimization

Lemma 35. For f ∈ H , let y ( i ) t := -z ( i ) t σ 2 t and

If s ∈ H then for ˆ f = arg inf f ∈H ˆ L ( f ) , we have

where ˆ L is defined in (4) .

Proof. Let y ( i ) t := -z ( i ) t σ 2 t . We have, for any f ∈ H ,

where ̂ L ( s ) := ∑ i ∈ [ m ] ,j ∈ [ N ] γ j ∥ ∥ ∥ s ( t j ,x ( i ) t j ) -y ( i ) t j ∥ ∥ ∥ 2 2 m . Since ˆ f is the minimizer, ̂ L ( ˆ f ) ≤ ̂ L ( s ) . Therefore,

which completes our proof.

We will first demonstrate a very crude bound, which will be of use later to derive a finer bound based on Martingale concentration developed in previous sections.

Lemma 36. Fix δ ∈ (0 , 1) and let y t := -z t σ 2 t , ∀ t ∈ T , γ t := ∆ < 1 . Furthermore, assume a linear discretization, i.e, t j = ∆ j . For L , ̂ L as defined in Lemma 35 and ˆ f := argmin f ∈F ̂ L ( f ) , we have almost surely:

we have with probability atleast 1 -δ ,

Proof. Using Lemma 35 and the Cauchy-Schwarz inequality,

which completes the first part of the proof. Next, we have

Clearly, since y ( i ) t is marginally Gaussian , we conclude that it is 4 √ d σ t norm subGaussian (see Definition 1). Using the fact that s ( t, x t ) is the conditional expectation of y ( i ) t , Lemma F.3. in [14] shows that s ( t, x t ) is 4 √ d/σ t -norm subGaussian. Therefore applying a union bound over all ∥ s ( t, x ( i ) t ) ∥ , ∥ y ( i ) t ∥ ≳ √ d log( |T | δ m ) σ t , with probability at-least 1 -δ the following holds:

Now, note the fact that σ t ≥ c 0 min(1 , t ) for some universal constant c 0 . Therefore, ∑ t ∈T 1 σ 2 t ≲ N + log( 1 ∆ ) . Plugging this into the equation above, we conclude the result.

∆

Lemma 37. Recall y ( i ) t := -z ( i ) t σ 2 t for all t ∈ T . Let for f ∈ H ,

Then, for ϵ > 0 ,

where L , ̂ L , ˆ f are defined in Lemma 35.

Proof. From Lemma 36, we must have L ( ˆ f ) ≤ ˆ L ( s ) . Therefore:

Lemma 38. Let f ∈ H and suppose Assumption 1 holds. For any fixed τ 0 > 0 , with probability 1 -δ , the following holds for every f ∈ H :

Proof. Let g ( t, x ) := f ( t, x ) -s ( t, x ) . Note that x t + τ 0 = e -τ 0 x t + z t,t + τ 0 . By Assumption 1, g is 2 L Lipschitz in x and with probability 1 -δ over x t + τ 0 , and every f ∈ H :

We conclude the second inequality with a similar proof.

Lemma 39. Under Assumption 1, with probability 1 -δ , for a universal constant C > 0 the following holds uniformly for every f ∈ H :

Proof. For the sake of clarity, we will denote g = f -s . Using Lemma 38, via the union bound for every t = t j , τ 0 = | t j -t k | along with Gaussian concentration for z ( i ) t,t + τ 0 , we conclude that with probability 1 -δ the following holds uniformly for every f ∈ F , i ∈ [ m ] and j, k ∈ T with | j -k | ∆ ≤ 1 for some universal constant C, c 0 > 0 :

Squaring both sides and using the AM-GM inequality,

Now, let ( i ∗ , k ∗ ) ∈ arg sup i ∈ [ m ] ,k ∈ [ N ] ∥ f ( t k , x ( i ) t k ) -s ( t k , x ( i ) t k ) ∥ 2 . Now, for any j such that | ( j -k ∗ ) | ∆ ≤ 1 , the Equation (45) implies:

This implies the following inequality from which we can conclude the result.

Theorem 1 (Empirical L 2 Bound) . Let Assumption 1 hold. Fix δ ∈ (0 , 1) . For all j ∈ [ N ] , let t j := ∆ j and γ j := ∆ . Let B := C log ( ( L +1) dmN log ( 1 δ ) / ∆ ) for an absolute constant C > 0 , and let ∆log 3 ( 1 ∆ ) d 3 log 3 (2 d ) log 3 ( 2 Nm δ ) log 3 ( B |H| δ ) ≤ 1 and N ∆ ≤ C log( 1 ∆ ) . Then for

with probability at least 1 -δ ,

Proof. Consider L ( f ) defined in Lemma 35, H f as defined in Lemma 2. Let ˆ f be the empirical risk minimizer. Then, by Lemma 35, we have: L ( ˆ f ) ≤ H ˆ f almost surely. Then, using Lemma 37, we have: L ( ˆ f ) ≤ ˆ L ( s ) almost surely.

As per Lemma 36, we pick UB = C ( N ∆+log( 1 ∆ )) d log( mN δ ) for some large enough constant C and conclude that

Let f ∈ F be arbitrary. We consider the martingale H developed in Appendix C with ζ = s -f m . In this case we can identify H f = H . Considering the notation given in Lemma 34, we have: L 2 2 ( ζ ) = 1 m ∆ L ( f ) . Let α = log( 10 |H| (2 B +1) δ ) . By Lemma 34, we conclude P ( A 1 ( f ) ∪ A 3 ( f ) ∪ A 3 ( f )) ≥ 1 -(2 B +1) e -α where:

1.

2.

3.

Taking a union bound over all f ∈ H , we conclude that ˆ f satisfies:

Since α = log( 10 |H| (2 B +1) δ ) , we conclude:

By Lemma 30, we conclude that with probability 1 -δ 4 , sup i,k W i,k ≤ log( 8 Nm δ ) . Now, consider

Now, consider the event A 2 ( ˆ f ) . It is clear from our choice of B that following inclusion holds:

Now, consider the event A 1 ( ˆ f ) . Define the following events for some large enough constant C .

Lemma39, we have P ( C ) ≥ 1 -δ 8 . By Lemma 30, and union bound we have sup i,k W i,k ≤ log( 8 Nm δ ) with probability 1 -δ 8 . Therefore, P ( D ) ≥ 1 -δ 8 . Under the event A 1 ( ˆ f ) ∩ C ∩ D we have:

Using the choice of ∆ being small enough as stated in the Theorem, as well as our choice of α , we conclude that under the event A 1 ( ˆ f ) ∩ C ∩ D , for some large enough constant C ′ :

Therefore, under the events ( A 1 ( ˆ f ) ∩ D ∩ C ) ∪ A 2 ( ˆ f ) , the guarantee for L ( ˆ f ) stated in the theorem holds. It now remains to show that P ( ( A 1 ( ˆ f ) ∩ D ∩ C ) ∪ A 2 ( ˆ f ) ) ≥ 1 -δ . We begin with Equation (47):

This demonstrates the desired result.

E Generalization error bounds

Lemma 40. Let all f ( t, x ) ∈ H , be parameterized as g ( t, x ; θ ) for θ ∈ Θ ⊆ R D and θ ∗ be such that h ( t, x t ; θ ∗ ) = s ( t, x t ) . Suppose ∃ λ, µ ≥ 0 such that ∀ θ ∈ Θ ,

Then, all f ∈ H satisfy Assumption 2 with κ = λ µ .

Proof. The proof follows by squaring the second inequality and comparing with the first inequality.

Lemma 41. For timestep t ≥ 0 , let x t be defined as in (1) . Consider function f : R × R d → R d such that ∃ κ ≥ 1 satisfying,

Let X = { x ( i ) t } i ∈ [ m ] be iid samples. Then, with probability atleast 1 -exp ( -m 8 κ 2 ) there exists a set G ⊆ [ m ] such that | G | ≥ m 8 κ 2 and

Proof. Using the Payley-Zygmund inequality, for any i ∈ [ m ] , ∀ θ ∈ [0 , 1] ,

Define the iid indicator random variable { χ i } i ∈ [ m ] as,

Then, using (51), P ( χ i = 1) ≥ 1 4 κ 2 . Let µ := ∑ m i =1 E [ χ i ] ≥ m 4 κ 2 . Using standard chernoff bounds for Bernoulli random variables,

The result then follows by setting ϵ := 1 2 .

Theorem 2 ( L 2 Error Bound) . Let Assumptions 1 and 2 hold. Fix δ ∈ (0 , 1) . For all j ∈ [ N ] , let t j := ∆ j and γ j := ∆ . Let B := C log ( ( L +1) dmN log ( 1 δ ) / ∆ ) for an absolute constant C > 0 , and let ∆log 3 ( 1 ∆ ) d 3 log 3 (2 d ) log 3 ( 2 Nm δ ) log 3 ( B |H| δ ) ≤ 1 and N ∆ ≤ C log( 1 ∆ ) . If

then with probability at least 1 -δ ,

Proof. Using Theorem 1, we have with probability at least 1 -δ ,

Using Lemma 40 and 41, if m ≳ κ 2 log ( N δ ) then, using a union-bound, for all particular timesteps { t j } j ∈ [ N ] with probability at least 1 -δ ,

Adding over all timesteps { t j } j ∈ [ N ] ,

The result then follows by setting the RHS smaller by ϵ 2 .

Theorem 5 (Accelerated Inference) . Under the same assumptions as Theorem 2, partition the timesteps { t j = ∆ j } j ∈ [ N ] into k disjoint subsets S 1 , S 2 , . . . , S k , where each subset S i contains timesteps of the form t j = ∆( i + mk ) for m ∈ N . Define γ ′ j := k ∆ for all j in any subset S i . Then, there exists at least one subset S i such that:

with probability at least 1 -δ .

Proof. From Theorem 2, we have with probability 1 -δ :

where γ j = ∆ . Partition the N timesteps into k disjoint subsets S 1 , . . . , S k as described. Each subset S i contributes:

Summing over all k subsets gives the original total:

Now scale each subset's step size by k (i.e., γ ′ j = k ∆ ). The contribution of subset S i becomes:

Summing over all subsets with the scaled γ ′ j , we get:

We conclude that at least one subset S i must satisfy:

since otherwise all k subsets would contribute more than ϵ 2 , leading to a total exceeding kϵ 2 , which contradicts the scaled bound kϵ 2 .

F Dimension Free Experiments

In this section, we describe our experimental setup for the experiments conducted in Figure 1. We implement DSM on samples drawn from N (0 , Σ) using an Ornstein-Uhlenbeck schedule with ¯ α j = exp( -2 θt j ) ( θ = 1 . 0 , T = 5 . 0 ) and a two-layer MLP of hidden size H = 1000 that concatenates the noisy sample x t ∈ R d with a scalar time embedding t j / ( N -1) ∈ [0 , 1] to predict noise z pred . A random covariance Σ = Q Λ Q ⊤ is generated from a GOE matrix with eigenvalues Λ ii ∼ Uniform(1 , 2) . We train for E = 200 epochs on m train = 1000 samples (batch size 1000 , learning rate η = 10 -3 ) and evaluate on m test = 1000 held-out samples over N = 100 timesteps. For each dimension d ∈ { 10 , 20 , 30 , 40 , 75 , 100 , 125 , 150 , 175 , 200 } (each averaged over R = 5 runs) we compute the per-step MSE E j = (1 /m test ) ∑ m test i =1 ∥ -Σ -1 t j x ( i ) t -z ( i ) pred / √ 1 -¯ α j ∥ 2 , average E j over j = 2 , . . . , N and runs to obtain a time-averaged error, define the scaled error ˜ E ( d ) = ( mean time-averaged error ) / (#params( d ) · loglog d ) , and plot ˜ E ( d ) versus d on a log-log axis alongside a best-fit linear curve. Our experiments were performed on a single Google Colab CPU.

G Bootstrapped Score Matching

Lemma 42 (Bootstrap Consistency) . For some α > , let

Proof. Note that by Tweedie's formula,

Therefore, using the Markovian property, we have

Finally, the result follows using another application of Tweedie's formula which shows that s ( t, x t ) = E [ -z t /σ 2 t | x t ] .

Then, under Assumption 1,

Proof. Using Tweedie's formula,

Using the Markov property,

Therefore, E [ h t,t ′ | x t ] = 0 . Let v t,t ′ := s t ( x t ) -αs ( t ′ , x t ′ ) and r t,t ′ := z t σ 2 t -α z t ′ σ 2 t ′ .

First consider r t,t ′ . We have using (1), z t = e -( t -t ′ ) z t ′ + z t,t ′ where z t,t ′ ∼ N (0 , σ 2 t -t ′ ) . Then,

Next, for v t,t ′ again using Tweedie's formula,

Therefore, using (55) and (54),

Therefore,

which implies,

G.1 Experimental Details

In this section, we provide some preliminary experiments (Figure 3) with the Bootstrapped Score Matching algorithm described in Section 5. The formal pseudocode has been provided in Algorithm 1.

(a) L2 error for a multivariate Gaussian

density

(b) Empirical density for a mixture of

Gaussians

Gaussian:L2errorvstimesteps

DSM

0.004

BSM

0.003

Error

0.002

0.001

0.000

0

200

400

600

800

1000

Timesteps

GMM:EmpiricalvsTrueDensity

0.30

DSM

BSM

True GMM Density

0.25

0.20

Density

0.15

0.10

0.05

0.00

-10

-5

O

5

10

15

Value

In the first experiment, we study the accuracy of different score estimation methods in the context of learning the score function of a Gaussian distribution under the variance-reduced Bootstrapped Score Matching (BSM) objective. We compare BSM with DSM to evaluate their relative performance in estimating the true score function across different timesteps. Our target distribution is a d -dimensional Gaussian distribution with covariance matrix Σ ∈ R d × d , constructed as Σ = 5 MM T + 5 vv T where M ∈ R d × d and v ∈ R d × 1 are sampled from a standard normal distribution. We generate m = 10000 samples from the target distribution. Note that since the target density is gaussian, the density at all intermediate timesteps, p t , also follows a gaussian distribution. The time evolution follows an non-linear decay model, with N = 1000 discrete timesteps sampled as: t i = linspace (0 . 001 , t max , N ) 2 , where t max = √ 5 . The noise covariance scaling factor follows σ t = √ 1 -e -2 t . The bootstrap ratio for BSM is adaptively chosen as 1 -( σ t / ( σ t -t ′ + σ t )) , where t ′ represents the previous timestep. The score function is estimated using the standard least-squares regression solution on account of the simple target distribution which implies a linear score function of the form s ( t, x ) := A t x for some matrix A t . We run 5 training epochs for the first few timesteps ( t ≤ 3 ) and 1 epoch thereafter. We plot the squared error of the learned score matrix, ˆ A t against the true score matrix, A t at all timesteps.

In the second experiment, we move away from the Gaussian density, which is unimodal, to a Gaussian Mixture model (GMM), which is multimodal. We fix the dimensionality of the data as d = 1 for ease of visualization, and generate a mixture of two gaussians with means ± 5 and mixture weights 0 . 7 and 0 . 3 respectively. We generate m = 10000 samples from the GMM. The time evolution is linear with N = 1000 timesteps. We train a 3 layer neural network with hidden layer dimensions of 10 each, separately for DSM and BSM. We train the neural network for 100 epochs, with an initial learning rate of 0 . 05 , using the AdamW optimizer, along with a cosine scheduler to manage the learning rate schedule. The number of warmup steps of the scheduler are chosen to be 10% of the total training steps. When training the BSM network, we start bootstrapping after k 0 = 250 timesteps and 90 epochs. The bootstrap ratio is fixed at 0 . 9 . Once training is completed, we sample 10000 points using the learned score functions to plot and compare the empirical density. Our experiments were performed on a single Google Colab CPU.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the sample complexity of score matching and establishes the first nearly dimension-free bounds under certain regularity conditions. Specifically, assuming Gaussian noise and a Lipschitz condition on the score function, the authors show that the number of samples required to achieve an $\\varepsilon$-accurate score estimate is $O(L^2 \\log(B|H|/\\delta)/ \\varepsilon^2)$. This bound holds for both empirical error and the expected $L_2$-norm error. The paper also introduces a bootstrapped score matching (BSM) method to improve empirical performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nThe primary contribution of this work lies in establishing a sample complexity bound for score matching that is almost independent of the data dimension. This may address a key challenge in high-dimensional generative modeling. The paper also verify the theory with numerical experiments, and demonstrate the performance of the proposed BSM method.\n\n**Weakness:**\n\n1. The analysis assumes optimization over a specific function class, which may not be available in practical settings.\n\n2. The analysis relies on a hypercontractivity assumption, which may not hold in real-world data distributions.\n\n3. The theoretical results are derived only for a fixed step size $t_j = \\Delta j$, which limits their generality.\n\n4. There is a minor typo in Remark 1: \"at least\" should replace \"atleast.\"\n\nQUESTIONS:\n1. Could the authors provide concrete application scenarios where the function class H is explicitly available?\n\n2. If the function class H is defined by a neural network class, what is the relationship between $\\log(H)$ and the number of network parameters?\n\n3. Even if the function class is parameterized by a one-dimensional contious parameter in $R$, will $\n\\log(H)$ tend to infinity?\n\n4. Do previous works [3] and [14] also rely on Assumption 2 to establish generalization bounds for score function learning?\n\n5. Definition 1 seems to be used only in the proof. The authors may consider moving it to Section 4.\n\n6. In Theorem 1, the bound is said to hold with high probability. Is the randomness here due to both the Gaussian noise in $x_t$ and the randomness in the samples $x^{(i)}$? Or conditioned on the data $x_t$?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper establishes dimension-free sample complexity bounds (modulo dependence due to the hypothesis class) for diffusion models. A single function approximator is used to jointly estimate the score for different noise levels. The proof techniques include marginal-based error decomposition and sharp variance bounds. Additionally, the paper proposes bootstrapped score matching with variance reduction techniques to improve the performance.

STRENGTHS AND WEAKNESSES:
**Strengths**:
This paper achieves a double-exponential reduction in dimension dependence by leveraging a single function approximator to estimate the score across noise levels. The proposed martingale error decomposition is theoretically novel, offering a clear path toward tighter guarantees. Given the widespread adoption of diffusion models in modern generative modeling, addressing the issue of dimension dependence is impactful, especially for understanding and improving their performance on high-dimensional data.

**Weaknesses**:
I am not aware of significant weaknesses in the current version.

QUESTIONS:
In this paper, a uniform time discretization is considered. When $T$ is large enough, the marginal distribution $p_T$ approaches the Gaussian distribution $N(0,I)$, implying that the score function should have a simpler form. Could the authors clarify how this affects the sample efficiency? And how to optimize the time discretization?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the sampling error of the score function estimated by a score-matching objective in the context of a diffusion model. The  authors provide high-probability bounds for the error. The claimed contributions are 1) the bounds are nearly dimension-free, 2) the use of a novel martingale-based decomposition, and 3) a bootstrapped version of the score estimate with lower variance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The bootstrapped score estimator looks interesting and I am slightly surprised by its performance.\n\n\nWeaknesses:\n\n- I am not convinced by the claim that the provided results are dimension-free:\n1) line 128 as well as subsequent theoretical results involve a condition that is very hard to intrepet, and notably has d^3 log^3(2d) dependence. As Remark 2 says, for this to be truly dimension free, this would require the discretisation step to be Delta = O(1/d^3), which is hard to reason about in practice. I'm not sure I understand why randomly sampled times, as argued in Remark 2, would address this theoretical issue.\n2) The log |H| dependence together with s in H feels prohibitively strong as to introduce in the conditions. One would expect H to also have dimension dependence (often exponential in d), unless a very informative class of H is chosen.  It would be helpful for the authors to provide a few concrete examples of score functions that satisfy Assumption 1 with concrete examples of H, to convince the readers that these are actually dimension-free results.\n\n- I am also not convinced that Theorem 3 provides a way to obtain faster inference, as I take that k is fixed in Theorem 3. Can the authors clarify whether k is allowed to grow, and whether the bound in Theorem 3 has hidden implicit dependence on k? In particular, I am not convinced about the claim in Remark 2 that Theorem 3 is a way to answer the Delta = O(1/d^3) requirement.\n\n- While I find the bootstrapped score matching section interesting, not enough details are given to illustrate why the bootstrapped version would be better. Both variance computations are upper bounds and not exact. Also, some limited experiments are included in the appendix but not in the main text. To illustrate the validity of this method, and also in view of the context of this paper, I think the authors need to perform experiments with a larger variety of settings, e.g. multimodal distributions beyond d=1 and non-Gaussian data.\n\n- Section 5: Can the authors clarify whether, the additional term in tilde y_t, should be interpreted as some sort of estimation of the noise? If so, some interpretation around this will be nice. There also seems to be a tradeoff in terms of decreasing Delta to improve the bound in Lemma 7 and the need to get good estimation of hat s in an earlier time step -- it would be great if some more interpretations can be given for this tradeoff. At the moment, the discussion mostly focuses on an explanation of how the method works but not really on why it works and what might be potential barriers for the method. I think a better illustration of the latter would enhance the message quite a lot.\n\nQUESTIONS:\nMost of my questions have been asked in the weakness part of the above section.\n\nLIMITATIONS:\nVery restrictive limitations were provided in the Conclusion. I think there are technical limitations that are unclear from the current exposition, and I have included them in the Weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work obtains sample (and time) complexity bounds for estimating the score. Assuming that the score may be approximated in $L^2$ and that the score is both spatially and temporally Lipschitz with sufficiently nice parameters, they are able to show that the dependence in dimension is only doubly logarithmic. Namely, the best score estimator in a sufficiently regular and expressive class has $\\varepsilon^2$ error if the number of samples used is $O(\\log \\log d)$.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The work purports to obtain dimension-free statistical rates for learning a Lipschitz score under relatively standard hypotheses, \n- There are significant technical contributions: leveraging the temporal regularity via a martingale argument, which could prove useful in other domains.\n- The BSM algorithm is interesting as a method for variance reduction.\n\n**Weakness**\n\n- It seems that a large price ought to be paid in time complexity ($N \\sim d^{3}$) in training, compared to the standard analysis. Is this trade-off reasonable? It is not really clear to me how Theorem 3 fits into this picture in terms of the concrete guarantees.\n- The paper is very difficult to parse and should be thoroughly checked for typos. I have highlighted some in the main text below, but have not thoroughly checked the appendix.\n- Spatial Lipschitzness appears in a major way and also justifies the temporal regularity. However, a more reasonable hypothesis might be time-varying regularity (in both space and time).\n- The dependence on $\\log \\lvert{\\mathcal H}\\rvert$, while standard, means that it is difficult to achieve these dimension-free rates for practical classes.\n\nQUESTIONS:\n**Major**\n\n- The step-size condition in Theorem 1 is unreadable. The factor $d^3$ appears to be outside the logarithm, but this is very unclear. Remark 2 appears to clarify that this does not affect the statistical rate.\n\n**Minor**\n\n- In related work, Markov should always be capitalized\n- subGaussian should be sub-Gaussian\n- Ito should be \\\\^Ito.\n- Equation (3) is strange, the $-$ sign should be outside the fraction.\n- L. 114: gaussian -> Gaussian\n- L. 115: datapoints -> data points\n- Figure 1 caption: atleast -> at least, similarly elsewhere.\n- L. 232: F.3. has an extra period.\n- L. 250: tweedie -> Tweedie. Also appears elsewhere.\n- L. 258: hessian -> Hessian\n- L. 265: proof Freedman -> proof of Freedman\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-04"}