Title: Understanding Generalizability of Diffusion Models Requires Rethinking the Hidden Gaussian Structure
Venue: NeurIPS 2024
Accepted: yes
Keywords: diffusion models

inductive bias

generalization

memorization
Abstract: In this work, we study the generalizability of diffusion models by looking into the hidden properties of the learned score functions, which are essentially a series of deep denoisers trained on various noise levels. We observe that as diffusion models transition from memorization to generalization, their corresponding nonlinear diffusion denoisers exhibit increasing linearity. This discovery leads us to investigate the linear counterparts of the nonlinear diffusion models, which are a series of linear models trained to match the function mappings of the nonlinear diffusion denoisers. Surprisingly, these linear denoisers are approximately the optimal denoisers for a multivariate Gaussian distribution characterized by the empirical mean and covariance of the training dataset. This finding implies that diffusion models have the inductive bias towards capturing and utilizing the Gaussian structure (covariance information) of the training dataset for data generation. We empirically demonstrate that this inductive bias is a unique property of diffusion models in the generalization regime, which becomes increasingly evident when the model's capacity is relatively small compared to the training dataset size. In the case that the model is highly overparameterized, this inductive bias emerges during the initial training phases before the model fully memorizes its training data. Our study provides crucial insights into understanding the notable strong generalization phenomenon recently observed in real-world diffusion models.

Understanding Generalizability of Diffusion Models Requires Rethinking the Hidden Gaussian Structure

Abstract

In this work, we study the generalizability of diffusion models by looking into the hidden properties of the learned score functions, which are essentially a series of deep denoisers trained on various noise levels. We observe that as diffusion models transition from memorization to generalization, their corresponding nonlinear diffusion denoisers exhibit increasing linearity. This discovery leads us to investigate the linear counterparts of the nonlinear diffusion models, which are a series of linear models trained to match the function mappings of the nonlinear diffusion denoisers. Interestingly, these linear denoisers are approximately the optimal denoisers for a multivariate Gaussian distribution characterized by the empirical mean and covariance of the training dataset. This finding implies that diffusion models have the inductive bias towards capturing and utilizing the Gaussian structure (covariance information) of the training dataset for data generation. We empirically demonstrate that this inductive bias is a unique property of diffusion models in the generalization regime, which becomes increasingly evident when the model's capacity is relatively small compared to the training dataset size. In the case where the model is highly overparameterized, this inductive bias emerges during the initial training phases before the model fully memorizes its training data. Our study provides crucial insights into understanding the notable strong generalization phenomenon recently observed in real-world diffusion models.

1 Introduction

In recent years, diffusion models [1-4] have become one of the leading generative models, powering the state-of-the-art image generation systems such as Stable Diffusion [5]. To understand the empirical success of diffusion models, several works [6-12] have focused on their sampling behavior, showing that the data distribution can be effectively estimated in the reverse sampling process, assuming that the score function is learned accurately. Meanwhile, other works [13-18] investigate the learning of score functions, showing that effective approximation can be achieved with score matching loss under certain assumptions. However, these theoretical insights, grounded in simplified assumptions about data distribution and neural network architectures, do not fully capture the complex dynamics of diffusion models in practical scenarios. One significant discrepancy between theory and practice is that real-world diffusion models are trained only on a finite number of data points. As argued in [19], theoretically a perfectly learned score function over the empirical data distribution can only replicate the training data. In contrast, diffusion models trained on finite samples exhibit remarkable generalizability, producing high-quality images that significantly differ from the training examples. Therefore, a good understanding of the remarkable generative power of diffusion models is still lacking.

In this work, we aim to deepen the understanding of generalizability in diffusion models by analyzing the inherent properties of the learned score functions. Essentially, the score functions can be interpreted as a series of deep denoisers trained on various noise levels. These denoisers are then chained together to progressively denoise a randomly sampled Gaussian noise into its corresponding

clean image, thus, understanding the function mappings of these diffusion denoisers is critical to demystify the working mechanism of diffusion models. Motivated by the linearity observed in the diffusion denoisers of effectively generalized diffusion models, we propose to elucidate their function mappings with a linear distillation approach, where the resulting linear models serve as the linear approximations of their nonlinear counterparts.

Contributions of this work: Our key findings can be highlighted as follows:

Inductive bias towards Gaussian structures (Section 3). Diffusion models in the generalization regime exhibit an inductive bias towards learning diffusion denoisers that are close (but not equal) to the optimal denoisers for a multivariate Gaussian distribution, defined by the empirical mean and covariance of the training data. This implies the diffusion models have the inductive bias towards capturing the Gaussian structure (covariance information) of the training data for image generation.

Model Capacity and Training Duration (Section 4) We show that this inductive bias is most pronounced when the model capacity is relatively small compared to the size of the training data. However, even if the model is highly overparameterized, such inductive bias still emerges during early training phases, before the model memorizes its training data. This implies that early stopping can prompt generalization in overparameterized diffusion models.

Connection between Strong Generalization and Gaussian Structure (Section 5). Lastly, we argue that the recently observed strong generalization [20] results from diffusion models learning certain common low-dimensional structural features shared across non-overlapping datasets. We show that such low-dimensional features can be partially explained through the Gaussian structure.

Relationship with Prior Arts. Recent research [20-24] demonstrates that diffusion models operate in two distinct regimes: ( i ) a memorization regime, where models primarily reproduce training samples and ( ii ) a generalization regime, where models generate high-quality, novel images that extend beyond the training data. In the generalization regime, a particularly intriguing phenomenon is that diffusion models trained on non-overlapping datasets can generate nearly identical samples [20]. While prior work [20] attributes this ' strong generalization ' effect to the structural inductive bias inherent in diffusion models leading to the optimal denoising basis (geometry-adaptive harmonic basis), our research advances this understanding by demonstrating diffusion models' inductive bias towards capturing the Gaussian structure of the training data. Our findings also corroborate with observations of earlier study [25] that the learned score functions of well-trained diffusion models closely align with the optimal score functions of a multivariate Gaussian approximation of the training data.

2 Preliminary

Basics of Diffusion Models. Given a data distribution p data ( x ) , where x ∈ R d , diffusion models [1-4] define a series of intermediate states p ( x ; σ ( t )) by adding Gaussian noise sampled from N ( 0 , σ ( t ) 2 I ) to the data, where σ ( t ) is a predefined schedule that specifies the noise level at time t ∈ [0 , T ] , such that at the end stage the noise mollified distribution p ( x ; σ ( T )) is indistinguishable from the pure Gaussian distribution. Subsequently, a new sample is generated by progressively denoising a random noise x T ∼ N ( 0 , σ ( T ) 2 I ) to its corresponding clean image x 0 . Following [4], this forward and backward diffusion process can be expressed with a probabilistic ODE:

In practice the score function ∇ x log p ( x ; σ ( t )) can be approximated by

where D θ ( x ; σ ( t )) is parameterized by a deep network with parameters θ trained with the denoising score matching objective:

In the discrete setting, the reverse ODE in (1) takes the following form:

where x 0 ∼ N ( 0 , σ 2 ( t 0 ) I ) . Notice that at each iteration i , the intermediate sample x i +1 is the sum of the scaled x i and the denoising output D θ ( x i ; σ ( t i )) . Obviously, the final sampled image is largely determined by the denoiser D θ ( x ; σ ( t )) . If we can understand the function mapping of these diffusion denoisers, we can demystify the working mechanism of diffusion models.

Optimal Diffusion Denoisers under Simplified Data Assumptions. Under certain assumptions on the data distribution p data ( x ) , the optimal diffusion denoisers D θ ( x ; σ ( t )) that minimize the score matching objective (3) can be derived analytically in closed-forms as we discuss below.

Multi-delta distribution of the training data. Suppose the training dataset contains a finite number of data points { y 1 , y 2 , ..., y N } , a natural way to model the data distribution is to represent it as a multi-delta distribution: p ( x ) = 1 N ∑ N i =1 δ ( x -y i ) . In this case, the optimal denoiser is

which is essentially a softmax-weighted combination of the finite data points. As proved in [24], such diffusion denoisers D M ( x ; σ ( t )) can only generate exact replicas of the training samples, therefore they have no generalizability.

Multivariate Gaussian distribution. Recent work [25] suggests modeling the data distribution p data ( x ) as a multivariate Gaussian distribution p ( x ) = N ( µ , Σ ) , where the mean µ and the covariance Σ are approximated by the empirical mean µ = 1 N ∑ N i =1 y i and the empirical covariance Σ = 1 N ∑ N i =1 ( y i -µ )( y i -µ ) T of the training dataset. In this case, the optimal denoiser is:

where Σ = U Λ U T is the SVD of the empirical covariance matrix, with singular values Λ = diag ( λ 1 , · · · , λ d ) and ˜ Λ σ ( t ) = diag ( λ 1 λ 1 + σ ( t ) 2 · · · λ d λ d + σ ( t ) 2 ) . With this linear Gaussian denoiser, as proved in [25], the sampling trajectory of the probabilistic ODE (1) has close form:

where u i is the i th singular vector of the empirical covariance matrix. While [25] demonstrate that the Gaussian scores approximate learned scores at high noise variances, we show that they are nearly the best linear approximations of learned scores across a much wider range of noise variances.

Generalization vs. Memorization of Diffusion Models. As the training dataset size increases, diffusion models transition from the memorization regime-where they can only replicate its training images-to the generalization regime, where the they produce high-quality, novel images [17]. While memorization can be interpreted as an overfitting of diffusion models to the training samples, the mechanisms underlying the generalization regime remain less well understood. This study aims to explore and elucidate the inductive bias that enables effective generalization in diffusion models.

3 Hidden Linear and Gaussian Structures in Diffusion Models

In this section, we study the intrinsic structures of the learned score functions of diffusion models in the generalization regime. Through various experiments and theoretical investigation, we show that

Diffusion models in the generalization regime have inductive bias towards learning the Gaussian structures of the dataset.

Based on the linearity observed in diffusion denoisers trained in the generalization regime, we propose to investigate their intrinsic properties through a linear distillation technique, with which we train a series of linear models to approximate the nonlinear diffusion denoisers (Section 3.1). Interestingly, these linear models closely resemble the optimal denoisers for a multivariate Gaussian distribution characterized by the empirical mean and covariance of the training dataset (Section 3.2). This implies diffusion models have the inductive bias towards learning the Gaussian structure of the training dataset. We theoretically show that the observed Gaussian structure is the optimal solution to the denoising score matching objective under the constraint that the model is linear (Section 3.3). In the subsequent sections, although we mainly demonstrate our results using the FFHQ datasets, our findings are robust and extend to various architectures and datasets, as detailed in Appendix G.

3.1 Diffusion Models Exhibit Linearity in the Generalization Regime

Our study is motivated by the emerging linearity observed in diffusion models in the generalization regime. Specifically, we quantify the linearity of diffusion denoisers at various noise level σ ( t ) by jointly assessing their 'Additivity' and 'Homogeneity' with a linearity score (LS) defined by the cosine similarity between D θ ( α x 1 + β x 2 ; σ ( t )) and α D θ ( x 1 ; σ ( t )) + β D θ ( x 2 ; σ ( t )) :

where x 1 , x 2 ∼ p ( x ; σ ( t )) , and α ∈ R and β ∈ R are scalars. In practice, the expectation is approximated with its empirical mean over 100 samples. A more detailed discussion on this choice of measuring linearity is deferred to Appendix A. Linearity across NoiseLevels(EDM-VE,FFHQ)

Following the EDM training configuration [4], we set the noise levels σ ( t ) within the continuous range [0.002,80]. As shown in Figure 1, as diffusion models transition from the memorization regime to the generalization regime (increasing the training dataset size), the corresponding diffusion denoisers D θ exhibit increasing linearity. This phenomenon persists across diverse datasets 1 as well as various training configurations 2 ; see Appendix B for more details. This emerging linearity motivates us to ask the following questions:

To what extent can a diffusion model be approximated by a linear model?

If diffusion models can be approximated linearly, what are the underlying characteristics of this linear approximation?

1.00

0.95

Cosine Similarity

0.90

datasetsize=7ooo0

datasetsize=35000

0.85

datasetsize=8750

datasetsize=4375

datasetsize=1094

0.80

dataset size =137

datasetsize=68

0

10

20

30

40

50

60

70

80

NoiseVariance

Investigating the Linear Structures via Linear Distillation. To address these questions, we investigate the hidden linear structure of diffusion denoisers through linear distillation . Specifically, for a given diffusion denoiser D θ ( x ; σ ( t )) at noise level σ ( t ) , we approximate it with a linear function (with a bias term) such that:

where the weight W σ ( t ) ∈ R d × d and bias b σ ( t ) ∈ R d are learned by solving the following optimization problem with gradient descent: 3

If these linear models effectively approximate the nonlinear diffusion denoisers, analyzing their weights can elucidate the generation mechanism.

While diffusion models are trained on continuous noise variance levels within [0.002,80], we examine the 10 discrete sampling steps specified by the EDM schedule [4]: [80.0, 42.415, 21.108, 9.723, 4.06, 1.501, 0.469, 0.116, 0.020, 0.002] . These steps are considered sufficient for studying the diffusion mappings for two reasons: (i) images generated using these 10 steps closely match those generated with more steps, and (ii) recent research [30] demonstrates that the diffusion denoisers trained on similar noise variances exhibit analogous function mappings, implying that denoiser behavior at discrete variances represents their behavior at nearby variances.

Generation Trajectories  (                     )  for Various Models

)(+

#

; -(.))

ScoreFieldApproximationError

80.0

42.415

21.109

9.723

4.066

1.502

0.47

0.117

0.02

0.002

0.35

Gaussianvs.EDM

EDM

Linear vs.EDM

0.30

Multi-DeltaVs.EDM

0.25

Multi-Delta

RMSE

0.20

0.15

0.10

0.05

Linear

0.00

0

10

20

30

40

50

60

70

80

NoiseVariance

After obtaining the linear models D L , we evaluate their differences with the actual nonlinear denoisers D θ with the score field approximation error, calculated using the expectation over the root mean square error (RMSE):

where d represents the data dimension and the expectation is approximated with its empirical mean. While we present RMSE-based results in the main text, our findings remain consistent across alternative metrics, including NMSE, as detailed in Appendix G.

We perform linear distillation on well trained diffusion models operating in the generalization regime. For comprehensive analysis, we also compute the score approximation error between D θ and: (i) the optimal denoisers for the multi-delta distribution D M defined as (5), and (ii) the optimal denoisers for the multivariate Gaussian distribution D G defined as (6). As shown in Figure 2, our analysis reveals three distinct regimes:

High-noise regime [20,80]. In this regime, only coarse image structures are generated (Figure 2(right)). Quantitatively, as shown in Figure 2(left), the distilled linear model D L closely approximates its nonlinear counterpart D θ with RMSE below 0.05. Both Gaussian score D G and multi-delta score D M also achieve comparable approximation accuracy.

Low-noise regime [0.002,0.1]. In this regime, only subtle, imperceptible details are added to the generated images. Here, both D L and D G effectively approximate D θ with RMSE below 0.05.

Intermediate-noise regime [0.1,20]: This crucial regime, where realistic image content is primarily generated, exhibits significant nonlinearity. While D M exhibits high approximation error due to rapid convergence to training samples-a memorization effect theoretically proved in [24], both D L and D G maintain relatively lower approximation errors.

Qualitatively, as shown in Figure 2(right), despite the relatively high score approximation error in the intermediate noise regime, the images generated with D L closely resemble those generated with D θ in terms of the overall image structure and certain amount of fine details. This implies (i) the underlying linear structure within the nonlinear diffusion models plays a pivotal role in their generalization capabilities and (ii) such linear structure is effectively captured by our distilled linear models. In the next section, we will explore this linear structure by examining the linear models D L .

3.2 Inductive Bias towards Learning the Gaussian Structures

Notably, the Gaussian denoisers D G exhibit behavior strikingly similar to the linear denoisers D L . As illustrated in Figure 2(left), they achieve nearly identical score approximation errors, particularly in the critical intermediate variance region. Furthermore, their sampling trajectories are remarkably similar (Figure 2(right)), producing nearly identical generated images that closely match those from the actual diffusion denoisers (Figure 3). These observations suggest that D L and D G share

Correlation Matrices

!

012345

6

!

‖"

!(#)

-$	& '

! #

$

2

‖

&

/‖$ 7 Λ

! #

$

2

‖

&

EDM-ADM(FFHQ)

=80.0

0=42.415

0=21.109

0

0=9.723

0=4.066

1.0

0=80.0

20

20

20

20

20

0 = 42.415

40

40

40

40

40

0.8

0 = 9.723

0 = 21.109

60

60

60

60

60

0 = 4.066

80

80

80

80

80

0.6

0 = 1.502

0=0.117

0=0.47

0

20

0=1.502

40

60

80

0

0=0.47

40

60

80

0

20

0=0.117

40

60

80

20

=0.02

40

60

80

20

=0.002

40

60

80

0.4

0 =0.02

0

0

0

0=0.002

20

20

20

20

20

0.2

40

40

40

40

40

60

60

60

60

60

0

20

40

60

80

100

80

80

80

80

80

Epochs

20

40

60

80

0

40

60

80

0

20

40

60

80

20

40

60

80

0

20

40

60

80

similar function mappings across various noise levels, leading us to hypothesize that the intrinsic linear structure underlying diffusion models corresponds to the Gaussian structure of the training data-specifically, its empirical mean and covariance. We validate this hypothesis by empirically showing that D L is close to D G through the following three complementary experiments:

Similarity in weight matrices. As illustrated in Figure 4(left), W σ ( t ) progressively converge towards U ˜ Λ σ ( t ) U T throughout the linear distillation process, achieving small normalized MSE (less than 0.2) for most of the noise levels. The less satisfactory convergence behavior at σ ( t ) = 80 . 0 is due to inadequate training of the diffusion models at this particular noise level, which is minimally sampled during the training of actual diffusion models (see Appendix G.2 for more details).

Similarity in Score functions. Furthermore, Figure 2(left, gray line) demonstrates that D L and D G maintain small score differences (RMSE less than 0.05) across all noise levels, indicating that these denoisers exhibit similar function mappings throughout the diffusion process.

Similarity in principal components. As shown in Figure 4(right), for a wide noise range ( σ ( t ) ∈ [0 . 116 , 80 . 0] ), the leading singular vectors of the linear weights W σ ( t ) (denoted U Linear ) align well with U , the singular vectors of the Gaussian weights. 4 This implies that U , representing the principal components of the training data, is effectively captured by the diffusion models. In the low-noise regime ( σ ( t ) ∈ [0 . 002 , 0 . 116] ), however, D θ approximates the identity mapping, leading to ambiguous singular vectors with minimal impact on image generation. Further analy-

Image 1

Image2

Image3

Image4

Image5

EDM

Linear

sis of D θ 's behavior in the low-noise regime is provided in Appendices D and F.1.

Since the optimization problem (9) is convex w.r.t. W σ ( t ) and b σ ( t ) , the optimal solution D L represents the unique optimal linear approximation of D θ . Our analyses demonstrate that this optimal linear approximation closely aligns with D G , leading to our central finding: diffusion models in the generalization regime exhibit an inductive bias (which we term as the Gaussian inductive bias) towards learning the Gaussian structure of training data. This manifests in two main ways: ( i ) In the high-noise variance regime, well-trained diffusion models learn D θ that closely approximate the linear Gaussian denoisers D G ; ( ii ) As noise variance decreases, although D θ diverges from D G , D G remains nearly identical to the optimal linear approximation D L , and images generated by D G retain structural similarity to those generated by D θ .

Finally, we emphasize that the Gaussian inductive bias only emerges in the generalization regime. By contrast, in the memorization regime, Figure 5 shows that D L significantly diverges from D G , and both D G and D L provide considerably poorer approximations of D θ compared to the generalization regime.

Clean Image

!

Noise

" ∼ .(/, & '

#

1)

# = ! + "

)

!

("; &('))

)

"

("; &('))

(70000)

)

"

("; &('))

(35000)

)

"

("; &('))

(1094)

)

"

("; &('))

(68)

)

!

(#; &('))

)

"

(#; &('))

(70000)

)

"

(#; &('))

(35000)

)

"

(#; &('))

(1094)

)

"

(#; &('))

(68)

Denoising Outputs  for

! "

= 4

Memorization

Generalization

(a)

(b)

ScoreFieldApproximationError（FFHQ)

0.25

Gaussian vs. EDM (Memorization)

Gaussian vs. EDM (Generalization)

Linear vs. EDM (Memorization)

0.20

Linear vs. EDM (Generalization)

Linear vs. Gaussian (Memorization)

Linear vs. Gaussian (Generalization)

0.15

RMSE

0.10

0.05

0.00

0.00

0.02

0.12

0.47

1.50

4.07

9.7221.11 42.42 80.00

NoiseVariance(logscale)

3.3 Theoretical Analysis

In this section, we demonstrate that imposing linear constraints on diffusion models while minimizing the denoising score matching objective (3) leads to the emergence of Gaussian structure.

Theorem 1. Consider a diffusion denoiser parameterized as a single-layer linear network, defined as D ( x t ; σ ( t )) = W σ ( t ) x t + b σ ( t ) , where W σ ( t ) ∈ R d × d is a linear weight matrix and b σ ( t ) ∈ R d is the bias vector. When the data distribution p data ( x ) has finite mean µ and bounded positive semidefinite covariance Σ , the optimal solution to the score matching objective (3) is exactly the Gaussian denoiser defined in (6) :

The detailed proof is postponed to Appendix E. This optimal solution corresponds to the classical Wiener filter [31], revealing that diffusion models naturally learn the Gaussian denoisers when constrained to linear architectures. To understand why highly nonlinear diffusion models operate near this linear regime, it is helpful to model the training data distribution as the multi-delta distribution p ( x ) = 1 N ∑ N i =1 δ ( x -y i ) , where { y 1 , y 2 , ..., y N } is the finite training images. Notice that this formulation better reflects practical scenarios where only a finite number of training samples are available rather than the ground truth data distribution. Importantly, it is proved in [25] that the optimal denoisers D M in this case is approximately equivalent to D G for high noise variance σ ( t ) and query points far from the finite training data. This equivalence explains the strong similarity between D G and D M in the high-noise variance regime, and consequently, why D θ and D G exhibit high similarity in this regime-deep networks converge to the optimal denoisers for finite training datasets.

However, this equivalence between D G and D M breaks down at lower σ ( t ) values. The denoising outputs of D M are convex combinations of training data points, weighted by a softmax function with temperature σ ( t ) 2 . As σ ( t ) 2 decreases, this softmax function increasingly approximates an argmax function, effectively retrieving the training point y i closest to the input x . Learning this optimal solution requires not only sufficient model capacity to memorize the entire training dataset but also, as shown in [32], an exponentially large number of training samples. Due to these learning challenges, deep networks instead converge to local minima D θ that, while differing from D M , exhibit better generalization property. Our experiments reveal that these learned D θ share similar function mappings with D G . The precise mechanism driving diffusion models trained with gradient descent towards this particular solution remains an open question for future research.

Notably, modeling p data ( x ) as a multi-delta distribution reveals a key insight: while unconstrained optimal denoisers (5) perfectly capture the scores of the empirical distribution, they have no gen-

Memorization

Generalization

(a)

(b)

Score Field Approximation Error (Gaussian vs.EDM)

0.40

dataset size =68

68

137

1094

8750

35000

70000

dataset size = 137

0.35

dataset size =1094

EDM

dataset size=8750

0.30

datasetsize=3500o

RMSE

0.25

dataset size = 70000

0.20

0.15

0.10

0.05

0.00

0

10

OE

40

50

09

70

08

eralizability. In contrast, Gaussian denoisers, despite having higher score approximation errors due to the linear constraint, can generate novel images that closely match those produced by the actual diffusion models. This suggests that the generative power of diffusion models stems from the imperfect learning of the score functions of the empirical distribution.

4 Conditions for the Emergence of Gaussian Structures and Generalizability

In Section 3, we demonstrate that diffusion models exhibit an inductive bias towards learning denoisers that are close to the Gaussian denoisers. In this section, we investigate the conditions under which this bias manifests. Our findings reveal that this inductive bias is linked to model generalization and is governed by (i) the model capacity relative to the dataset size and (ii) the training duration. For additional results, including experiments on CIFAR-10 dataset, see Appendix F.

4.1 Gaussian Structures Emerge when Model Capacity is Relatively Small

First, we find that the Gaussian inductive bias and the generalization of diffusion models are heavily influenced by the relative size of the model capacity compared to the training dataset. In particular, we demonstrate that:

Diffusion models learn the Gaussian structures when the model capacity is relatively small compared to the size of training dataset.

This argument is supported by the following two key observations:

Increasing dataset size prompts the emergence of Gaussian structure at fixed model scale. We train diffusion models using the EDM configuration [4] with a fixed channel size of 128 on datasets of varying sizes [68 , 137 , 1094 , 8750 , 35000 , 70000] until FID convergence. Figure 6(left) demonstrates that the score approximation error between diffusion denoisers D θ and Gaussian denoisers D G decreases as the training dataset size grows, particularly in the crucial intermediate noise variance regime ( σ ( t ) ∈ [0 . 116 , 20] ). This increasing similarity between D θ and D G correlates with a transition in the models' behavior: from a memorization regime, where generated images are replicas of training samples, to a generalization regime, where novel images exhibiting Gaussian structure 5 are produced, as shown in Figure 6(b). This correlation underscores the critical role of Gaussian structure in the generalization capabilities of diffusion models.

Decreasing model capacity promotes the emergence of Gaussian structure at fixed dataset sizes. Next, we investigate the impact of model scale by training diffusion models with varying channel sizes [4 , 8 , 16 , 32 , 64 , 128] , corresponding to [64 k , 251 k , 992 k , 4 M , 16 M , 64 M ] parameters, on a fixed training dataset of 1094 images. Figure 7(left) shows that in the intermediate noise variance regime ( σ ( t ) ∈ [0 . 116 , 20] ), the discrepancy between D θ and D G decreases with decreasing model scale, indicating that Gaussian structure emerges in low-capacity models. Figure 7(right) demonstrates that

(a)

(b)

Score Field Approximation Error (Gaussian v.s. EDM)

scale=4

scale=8

scale=16

scale=32

scale=64

scale=128

0.30

Scale=128(70000 images)

Scale =128 (1094 images)

EDM

0.25

Scale = 64 (1094 images)

Scale =32(1094 images)

Scale = 16 (1094 images)

0.20

Scale = 8 (1094 images)

RMSE

Scale = 4 (1094 images)

NN

0.15

scale = 128 (70000 images)

0.10

0.05

0.00

0

10

20

OE

40

50

09

70

08

Generalization

Memorization

(a)

(b)

Score Field Approximation Error (Gaussian vs.EDM)

epochs=187

epochs = 841

epochs = 2293

epochs = 9173

epochs =32105

epochs = 64210

0.40

sufficient training(7oooo images)

64210epochs(1094 images)

EDM

0.35

32105epochs(1094 images)

9173 epochs (1094 images)

0.30

2293 epochs (1094 images)

841 epochs (1094 images)

RMSE

0.25

187epochs(1094 images)

468epochs(1094images)

0.20

scale = 128 (70000 images)

0.15

0.10

0.05

0.00

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

this trend corresponds to a transition from data memorization to the generation of images exhibiting Gaussian structure. Here we note that smaller models lead to larger discrepancy between D θ and D G in the high-noise regime. This phenomenon arises because diffusion models employ a bell-shaped noise sampling distribution that prioritizes intermediate noise levels, resulting in insufficient training at high noise variances, especially when model capacity is limited (see more details in Appendix F.2).

These two experiments collectively suggest that the inductive bias of diffusion models is governed by the relative capacity of the model compared to the training dataset size.

4.2 Overparameterized Models Learn Gaussian Structures before Memorization

In the overparameterized regime, where model capacity significantly exceeds training dataset size, diffusion models eventually memorize the training data when trained to convergence. However, examining the learning progression reveals a key insight:

Diffusion models learn the Gaussian structures with generalizability before they memorize.

Figure 8(a) demonstrates that during early training epochs (0-841), D θ progressively converge to D G in the intermediate noise variance regime, indicating that the diffusion model is progressively learning the Gaussian structure in the initial stages of training. Notably. By epoch 841, the diffusion model generates images strongly resembling those produced by the Gaussian model, as shown in Figure 8(b). However, continued training beyond this point increases the difference between D θ and D G as the model transitions toward memorization. This observation suggests that early stopping could be an effective strategy for promoting generalization in overparameterized diffusion models.

5 Connection between Strong Generalizability and Gaussian Structure

A recent study [20] reveals an intriguing 'strong generalization' phenomenon: diffusion models trained on large, non-overlapping image datasets generate nearly identical images from the same initial

Early Stopping

Decrease Scale

Non-overlapping datasets with size 35000, model scale 128

Generated Images from Gaussian Models (size 35000)

Generated Images from Gaussian Models (size 1094)

Non-overlapping datasets with size 1094, model scale 128

Early Stopping at Epoch 921

(a)

(b)

(c)

Strong generalizability under small dataset size (1094)

Gaussian ef 51

Gaussian of S2

ZS jo ueyssneg

Nearest Neighbor in S1

Nearest Neighbor in S1

EDM trained on 51

EDM trained on S1

EDM trained on S2

EDM trained on S2

Nearest Neighbor in S2

Nearest Neighbor in S2

EDM (51)

EDM (S2)

EDM (S1)

EDM (S2)

noise. While this phenomenon might be attributed to deep networks' inductive bias towards learning the 'true' continuous distribution of photographic images, we propose an alternative explanation: rather than learning the complete distribution, deep networks may capture certain low-dimensional common structural features shared across these datasets and these features can be partially explained by the Gaussian structure.

To validate this hypothesis, we examine two diffusion models with channel size 128, trained on non-overlapping datasets S1 and S2 (35000 images each). Figure 9(a) shows that images generated by these models (bottom) closely match those from their corresponding Gaussian models (top), highlighting the Gaussian structure's role in strong generalization.

Comparing Figure 9(a)(top) and (b)(top), we observe that D G generates nearly identical images whether the Gaussian structure is calculated on a small dataset (1094 images) or a much larger one (35000 images). This similarity emerges because datasets of the same class can exhibit similar Gaussian structure (empirical covariance) with relatively few samples-just hundreds for FFHQ. Given the Gaussian structure's critical role in generalization, small datasets may already contain much of the information needed for generalization, contrasting previous assertions in [20] that strong generalization requires training on datasets of substantial size (more than 10 5 images). However, smaller datasets increase memorization risk, as shown in Figure 9(b). To mitigate this, as discussed in Section 4, we can either reduce model capacity or implement early stopping (Figure 9(c)). Indeed, models trained on 1094 and 35000 images generate remarkably similar images, though the smaller dataset yields lower perceptual quality. This similarity further demonstrates that small datasets contain substantial generalization-relevant information closely tied to Gaussian structure. Further discussion on the connections and differences between our work and [20] are detailed in Appendix H.

6 Discussion

In this study, we empirically demonstrate that diffusion models in the generalization regime have the inductive bias towards learning diffusion denoisers that are close to the corresponding linear Gaussian denoisers. Although real-world image distributions are significantly different from Gaussian, our findings imply that diffusion models have the bias towards learning and utilizing low-dimensional data structures, such as the data covariance, for image generation. However, the underlying mechanism by which the nonlinear diffusion models, trained with gradient descent, exhibit such linearity remains unclear and warrants further investigation.

Moreover, the Gaussian structure only partially explains diffusion models' generalizability. While models exhibit increasing linearity as they transition from memorization to generalization, a substantial gap persists between the linear Gaussian denoisers and the actual nonlinear diffusion models, especially in the intermediate noise regime. As a result, images generated by Gaussian denoisers fall short in perceptual quality compared to those generated by the actual diffusion models especially for complex dataset such as CIFAR-10. This disparity highlights the critical role of nonlinearity in high-quality image generation, a topic we aim to investigate further in future research.

Data Availability Statement

The code and instructions for reproducing the experiment results will be made available in the following link: https://github.com/Morefre/Understanding-Generalizability-ofDiffusion-Models-Requires-Rethinking-the-Hidden-Gaussian-Structure .

Acknowledgment

We acknowledge funding support from NSF CAREER CCF-2143904, NSF CCF-2212066, NSF CCF- 2212326, NSF IIS 2312842, NSF IIS 2402950, ONR N00014-22-1-2529, a gift grant from KLA, an Amazon AWS AI Award, and MICDE Catalyst Grant. We also acknowledge the computing support from NCSA Delta GPU [33]. We thank Prof. Rongrong Wang (MSU) for fruitful discussions and valuable feedbacks.

References

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning , pages 2256-2265. PMLR, 2015.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations .

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems , 35:26565-26577, 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨ orn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. In International Conference on Learning Representations , 2023.

Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. Transactions on Machine Learning Research , 2022.

Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence for score-based generative modeling with polynomial complexity. Advances in Neural Information Processing Systems , 35:22870-22882, 2022.

Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence of score-based generative modeling for general data distributions. In International Conference on Algorithmic Learning Theory , pages 946-985. PMLR, 2023.

Yuchen Wu, Yuxin Chen, and Yuting Wei. Stochastic runge-kutta methods: Provable acceleration of diffusion models. arXiv preprint arXiv:2410.04760 , 2024.

Gen Li, Yuting Wei, Yuejie Chi, and Yuxin Chen. A sharp convergence theory for the probability flow odes of diffusion models. arXiv preprint arXiv:2408.02320 , 2024.

Zhihan Huang, Yuting Wei, and Yuxin Chen. Denoising diffusion probabilistic models are optimally adaptive to unknown low dimensionality. arXiv preprint arXiv:2410.18784 , 2024.

Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. In International Conference on Machine Learning , pages 4672-4712. PMLR, 2023.

Kazusato Oko, Shunta Akiyama, and Taiji Suzuki. Diffusion models are minimax optimal distribution estimators. In International Conference on Machine Learning , pages 26517-26582. PMLR, 2023.

Kulin Shah, Sitan Chen, and Adam Klivans. Learning mixtures of gaussians using the ddpm objective. Advances in Neural Information Processing Systems , 36:19636-19649, 2023.

Hugo Cui, Eric Vanden-Eijnden, Florent Krzakala, and Lenka Zdeborova. Analysis of learning a flow-based generative model from limited sample complexity. In The Twelfth International Conference on Learning Representations , 2023.

Huijie Zhang, Jinfan Zhou, Yifu Lu, Minzhe Guo, Liyue Shen, and Qing Qu. The emergence of reproducibility and consistency in diffusion models. In Forty-first International Conference on Machine Learning , 2024.

Peng Wang, Huijie Zhang, Zekai Zhang, Siyi Chen, Yi Ma, and Qing Qu. Diffusion models learn low-dimensional distributions via subspace clustering. arXiv preprint arXiv:2409.02426 , 2024.

Sixu Li, Shi Chen, and Qin Li. A good score does not lead to a good generative model. arXiv preprint arXiv:2401.04856 , 2024.

Zahra Kadkhodaie, Florentin Guth, Eero P Simoncelli, and St´ ephane Mallat. Generalization in diffusion models arises from geometry-adaptive harmonic representation. In The Twelfth International Conference on Learning Representations , 2023.

Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art or digital forgery? investigating data replication in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6048-6058, 2023.

Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Understanding and mitigating copying in diffusion models. Advances in Neural Information Processing Systems , 36:47783-47803, 2023.

TaeHo Yoon, Joo Young Choi, Sehyun Kwon, and Ernest K Ryu. Diffusion probabilistic models generalize when they fail to memorize. In ICML 2023 Workshop on Structured Probabilistic Inference {\ & } Generative Modeling , 2023.

Xiangming Gu, Chao Du, Tianyu Pang, Chongxuan Li, Min Lin, and Ye Wang. On memorization in diffusion models. arXiv preprint arXiv:2310.02664 , 2023.

Binxu Wang and John J Vastola. The hidden linear structure in score-based models and its application. arXiv preprint arXiv:2311.10892 , 2023.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4401-4410, 2019.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8188-8197, 2020.

Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 , 2015.

Hyojun Go, Yunsung Lee, Seunghyun Lee, Shinhyeok Oh, Hyeongdon Moon, and Seungtaek Choi. Addressing negative transfer in diffusion models. Advances in Neural Information Processing Systems , 36, 2024.

Mallat St´ ephane. Chapter 11 - denoising. In Mallat St´ ephane, editor, A Wavelet Tour of Signal Processing (Third Edition) , pages 535-610. Academic Press, Boston, third edition edition, 2009.

Chen Zeno, Greg Ongie, Yaniv Blumenfeld, Nir Weinberger, and Daniel Soudry. How do minimum-norm shallow denoisers look in function space? Advances in Neural Information Processing Systems , 36, 2024.

Timothy J Boerner, Stephen Deems, Thomas R Furlani, Shelley L Knuth, and John Towns. Access: Advancing innovation: Nsf's advanced cyberinfrastructure coordination ecosystem: Services & support. In Practice and Experience in Advanced Research Computing , pages 173-176. 2023.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems , 34:8780-8794, 2021.

DP Kingma. Adam: a method for stochastic optimization. In Int Conf Learn Represent , 2014.

Alfred O. Hero. Statistical methods for signal processing. 2005.

Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning , pages 1096-1103, 2008.

Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation , 23(7):1661-1674, 2011.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18 , pages 234-241. Springer, 2015.

Sergey Ioffe. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 , 2015.

Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml , volume 30, page 3. Atlanta, GA, 2013.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4195-4205, 2023.

Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Sreyas Mohan, Zahra Kadkhodaie, Eero P Simoncelli, and Carlos Fernandez-Granda. Robust and interpretable blind image denoising via bias-free convolutional neural networks. In International Conference on Learning Representations .

Hila Manor and Tomer Michaeli. On the posterior distribution in denoising: Application to uncertainty quantification. In The Twelfth International Conference on Learning Representations .

Siyi Chen, Huijie Zhang, Minzhe Guo, Yifu Lu, Peng Wang, and Qing Qu. Exploring lowdimensional subspaces in diffusion models for controllable image editing. arXiv preprint arXiv:2409.02374 , 2024.

Contents

Appendices

A Measuring the Linearity of Diffusion Denoisers

In this section, we provide a detailed discussion on how to measure the linearity of diffusion model. For a diffusion denoiser, D θ ( x ; σ ( t )) , to be considered approximately linear, it must fulfill the following conditions:

Additivity: The function should satisfy D θ ( x 1 + x 2 ; σ ( t )) ≈ D θ ( x 1 ; σ ( t )) + D θ ( x 2 ; σ ( t )) .

Homogeneity: It should also adhere to D θ ( α x ; σ ( t )) ≈ α D θ ( x ; σ ( t )) .

To jointly assess these properties, we propose to measure the difference between D θ ( α x 1 + β x 2 ; σ ( t )) and α D θ ( x 1 ; σ ( t )) + βD ( x 2 ; σ ( t )) . While the linearity score is introduced as the cosine similarity between D θ ( α x 1 + β x 2 ; σ ( t )) and α D θ ( x 1 ; σ ( t )) + βD ( x 2 ; σ ( t )) in the main text:

it can also be defined with the normalized mean square difference (NMSE):

where the expectation is approximated with its empirical mean over 100 randomly sampled pairs of ( x 1 , x 2 ) . In the next section, we will demonstrate the linearity score with both metrics.

Since the diffusion denoisers are trained solely on inputs x ∼ p ( x ; σ ( t )) , their behaviors on outof-distribution inputs can be quite irregular. To produce a denoised output with meaningful image structure, it is critical that the noise component in the input x matches the correct variance σ ( t ) 2 . Therefore, our analysis of linearity is restricted to in-distribution inputs x 1 and x 2 , which are randomly sampled images with additive Gaussian noises calibrated to noise variance σ ( t ) 2 . We also need to ensure that the values of α and β are chosen such that α 2 + β 2 = 1 , maintaining the correct variance for the noise term in the combined input α x 1 + β x 2 . We present the linearity scores, calculated with varying values of α and β , for diffusion models trained on diverse datasets in Figure 10. These models are trained with the EDM-VE configuration proposed in [4], which ensures the resulting models are in the generalization regime. Typically, setting α = β = 1 / √ 2 yields the lowest linearity score; however, even in this scenario, the cosine similarity remains impressively high, exceeding 0.96. This high value underscores the presence of significant linearity within diffusion denoisers.

We would like to emphasize that for linearity to manifest in diffusion denoisers, it is crucial that they are well-trained, achieving a low denoising score matching loss as indicated in (3). As shown in Figure 11, the linearity notably reduces in a less well trained diffusion model (Baseline-VE) comapred to its well-trained counterpart (EDM-VE). Although both models utilize the same 'VE' network architecture F θ ( x ; σ ( t )) [2], they differ in how the diffusion denoisers are parameterized:

where c skip is the skip connection and c out modulate the scale of the network output. With carefully tailored c skip and c out, the EDM-VE configuration achieves a lower score matching loss compared to Baseline-VE, resulting in samples with higher quality as illustrated in Figure 11(right).

B Emerging Linearity of Diffusion Models

In this section we provide a detailed discussion on the observation that diffusion models exhibit increasing linearity as they transition from memorization to generalization, which is briefly described in Section 3.1.

B.1 Generalization and Memorization Regimes of Diffusion Models

As shown in Figure 12, as the training dataset size increases, diffusion models transition from the memorization regime-where they can only replicate its training images-to the generalization regime, where the they produce high-quality, novel images. To measure the generalization capabilities of diffusion models, it is crucial to assess their ability to generate images that are not mere replications of the training dataset. This can be quantitatively evaluated by generating a large set of images from the diffusion model and measuring the average difference between these generated images and their nearest neighbors in the training set. Specifically, let { x 1 , x 2 , ..., x k } represent k randomly sampled images from the diffusion models (we choose k = 100 in our experiments), and let Y := { y 1 , y 2 , ..., y N } denote the training dataset consisting of N images. We define the generalization score as follows:

where NN Y ( x i ) represents the nearest neighbor of the sample x k in the training dataset Y , determined by the Euclidean distance on a per-pixel basis. Empirically, a GL score exceeding 0.6 indicates that the diffusion models are effectively generalizing beyond the training dataset.

B.2 Diffusion Models Exhibit Linearity in the Generalization Regime

As demonstrated in Figure 13(a) and (d), diffusion models transition from the memorization regime to the generalization regime as the training dataset size increases. Concurrently, as depicted in Figure 13(b), (c), (e) and (f), the corresponding diffusion denoisers exhibit increasingly linearity. This phenomenon persists across diverse datasets datasets including FFHQ [26], AFHQ [28] and LSUNChurches [29], as well as various model architectures including EDM-VE [3], EDM-VP [2] and EDM-ADM [34]. This emerging linearity implies that the hidden linear structure plays an important role in the generalizability of diffusion model.

C Linear Distillation

As discussed in Section 3.1, we propose to study the hidden linearity observed in diffusion denosiers with linear distillation. Specifically, for a given diffusion denoiser D θ ( x ; σ ( t )) , we aim to approxi-

Linearity across Noise Levels (FFHQ)

Linearity across Noise Levels (LSNU-Churches)

Linearity across Noise Levels(AFHQ-V2)

1.000

1.000

1.00

0.995

S66°0

660

Cosine Similarity

Cosine Similarity

0660

Cosine Similarity

0660

0.985

0.98

0.985

0.980

α² = 0.99

a²=0.87

α² = 0.74

0.980

0.975

α²=0.99

α²=0.87

α² = 0.74

0.97

α² = 0.74

α² = 0.99

α² =0.87

0.975

α²=0.62

0.970

α²=0.62

0.96

α²=0.62

α² = 0.50

α²= 0.50

α²=0.50

0

10

20

30

40

50

60

70

80

0.965

0

10

20

30

40

50

60

70

08

0

10

20

30

NoiseVariance

40

50

60

70

80

Generation Trajectories  (                     )  for Various Models

((*

!

; ,(-))

Linearity across Noise Levels(FFHQ)

80.0

42.415

21.109

9.723

4.066

1.502

0.47

0.117

0.02

0.002

1.00

EDM-VE

0.95

Cosine Similarity

0.90

0.85

0.80

0.75

0.70

baseline-ve

0.65

edm-ve

Linear

0

10

20

30

40

50

60

70

80

NoiseVariance

70000 images

Generated

NN

Generated

NN

Generated

NN

Generated

NN

Generated

NN

Generated

NN

4375 images

1094 images

50000 images

12500 images

782 images

FFHQ

CIFAR-10

(a)

(b)

(c)

(d)

(e)

(f)

0

Q

0

0

20

20

20

20

20

20

40

40

40

40

40

40

60

60

60

60

60

60

0

25

50

0

25

50

0

25

50

0

25

50

0

25

50

0

25

50

0

0

0

0

0

0

10

10

10

10

10

10

20

20

20

20

20

20

30

30

30

30

30

30

0

20

0

20

0

20

0

20

0

20

0

20

mate it with a linear function (with a bias term for more expressibility):

for x ∼ p ( x ; σ ( t )) . Notice that for three dimensional images with size ( c, h, w ) , x ∈ R d represents their vectorized version, where d = c × w × h . Let

We train 10 independent linear models for each of the selected noise variance level σ ( t ) with the procedure summarized in Algorithm 1:

In practice, the gradients on W σ ( t ) and b σ ( t ) are obtained through automatic differentiation. Additionally, we employ the Adam optimizer [35] for updates. Additional linear distillation results are provided in Figure 14.

FFHQ

CIFAR-10

(a)

(b)

(c)

(d)

(e)

(f)

Generalization Score w.r.t.DatasetSize (EDM-VE,FFHQ)

Linearity across Noise Levels (EDM-VE, FFHQ)

1.00

dataset size=7oooo

0.7

0.6

datasetsize=35000

0.6

0.95

0.5

dataset size =8750

dataset size =4375

GL Scores

S°0

0.4

Cosine Similarity

0.90

datasetsize=7oooo

MSE

Normalized

0.3

0.4

dataset size=1094

dataset size = 137

dataset size=68

0.2

E'0

0.85

dataset size=4375

datasetsize=35000

dataset size=8750

0.2

0.1

0.80

dataset size = 137

dataset size =1094

0.1

0'0

dataset size =68

0.0

0

10000

20000

30000

40000

50000

60000

70000

0

10

20

30

40

50

60

70

80

0

10

20

30

40

50

60

70

80

Dataset Size

Generalization Score w.r.t.DatasetSize (EDM-VE,CIFAR-10)

Linearity across Noise Levels (EDM-VE, CIFAR-10)

Linearity across Noise Levels (EDM-VE, CIFAR-10)

1.00

0.8

dataset size =50000

0.6

0.7

dataset size =25000

0.95

dataset size = 12500

0.5

9'0

dataset size=3125

Scores

0.4

Similarity

0.90

Normalized MSE

0.5

dataset size = 782

dataset size =196

7

0.3

Cosine

0.85

dataset size=50000

0.4

0.2

0.80

datasetsize=250oo

dataset size =12500

0.2

0.3

0.1

0.75

dataset size =3125

dataset size =782

0.1

0.0

0.70

dataset size =196

0.0

0

10000

20000

Dataset Size

30000

40000

50000

0

10

20

30

40

50

60

70

80

0

10

20

30

40

50

60

70

80

D Diffusion Models in Low-noise Regime are Approximately Linear Mapping

It should be noted that the low score difference between D G and D θ within the low-noise regime ( σ ( t ) ∈ [0 . 002 , 0 . 116] ) does not imply the diffusion denoisers capture the Gaussian structure, instead, the similarity arises since both of them are converging to the identity mapping as σ ( t ) decreases. As shown in Figure 15, within this regime, the differences between the noisy input x and their corresponding denoised outputs D θ ( x ; σ ( t )) quickly approach 0. This indicates that the learned denoisers D θ progressively converge to the identity function. Additionally, from (6), it is evident that the difference between the Gaussian weights and the identity matrix diminishes as σ ( t ) decreases, which explains why D G can well approximate D θ in the low noise variance regime.

We hypothesize that D θ learns the identity function because of the following two reasons:

(i) within the low-noise regime, since the added noise is negligible compared to the clean image, the identity function already achieves a small denoising error, thus serving as a shortcut which is exploited by the deep network.

(ii) As discussed in Appendix A, diffusion models are typically parameterized as follows:

where F θ represents the deep network, and c skip ( σ ( t )) and c out ( σ ( t )) are adaptive parameters for the skip connection and output scaling, respectively, which adjust according to the noise variance levels. For canonical works on diffusion models [2-4, 34], as σ ( t ) approaches zero, c skip and c out converge to 1 and 0 respectively. Consequently, at low variance levels, the function forms of diffusion denoisers are approximatly identity mapping: D θ ( x ; σ ( t )) ≈ x .

This convergence to identity mapping has several implications. First, the weights W σ ( t ) of the distilled linear models D L approach the identity matrix at low variances, leading to ambiguous

Algorithm 1 Linear Distillation

Require:

(i) the targeted diffusion denoiser D θ ( · ; σ ( t )) ,

(ii) weights W σ ( t ) and biases b σ ( t ) , both initialized to zero,

(iii) gradient step size η ,

(iv) number of training iterations K ,

(v) training batch size n ,

(vi) image dataset S .

Randomly sample a batch of training images { x 1 , x 2 , . . . , x n } from S .

Randomly sample a batch of noises { ϵ 1 , ϵ 2 , . . . , ϵ n } from N ( 0 , σ ( t ) I ) Update W σ ( t ) and b σ ( t ) with gradient descent:

.

"

! #

-"

! #

%

'

/ "

! #

'

(a)

(b)

AssymetricMeasure

0.8

0.6

1.0

+

0 = 4.066

0 = 1.502

0=80.0

0 = 42.415

0 = 21.109

0 = 9.723

0 = 0.47

0.10

0.06

0.02

0.08

0.12

0.00

Q

2000 4000 6000 8000 1000012000

Singular Value Index

=80.0

！

Linear

Value

0.15

0.20

0.05

0.30

0.25

0.00

2000 4000 6000 8000 1000012000

Singular Value Index

0 =42.415

Linear

0.5

0.4

0.3

0.1

0.6

0.0

0

2000 4000 6000 8000 1000012000

Singular Value Index

0=21.109

Linear

Singular Value

0.6

0.4

0.2

8'0

0.0

2000 4000 6000 8000 1000012000

Singular Value Index

=9.723

Linear

0.4

1

0 = 0.117

0 =0.02

0=0.002

Singular Value

0.6

1.0

0=4.066

jeaun

Singular Value

0.8卡

0.6

1.0

0 =1.502

Linear

Singular Value

0.8

0.6

1.0

=0.47

Linear

Singular Value

0.6

1.0

0=0.117

jeaun

0.2

0.4

0.2

0.4

0.2

0.2

0.4

0.2

0.4

0.0

0.0

0.0

0.0

5

10

15

20

O

2000 4000 6000 80001000012000

Singular Value Index

0

2000 4000 6000 8000 1000012000

Singular Value Index

0

2000 4000 6000 8000 1000012000

Singular Value Index

0

2000 4000 6000 8000 1000012000

Singular Value Index

Epochs/5

FFHQ

(c)

LSUN-Churches

First 25 Singular Vectors of the Linear Model

First 25 Singular Vectors of the Gaussian Model

(d)

singular vectors. This explains the poor recovery of singular vectors for σ ( t ) ∈ [0 . 002 , 0 . 116] shown in Figure 4. Second, the presence of the bias term in (8) makes it challenging for our linear model to learn the identity function, resulting in large errors at σ ( t ) = 0 . 002 as shown in Figure 4(a).

Finally, from (4), we observe that when D θ acts as an identity mapping, x i +1 remains unchanged from x i . This implies that sampling steps in low-variance regions minimally affect the generated image content, as confirmed in Figure 2, where image content shows negligible variation during these steps.

First 25 Singular Vectors of the Linear Model

First 25 Singular Vectors of the Gaussian Model

Normalized MSE between

𝑫

𝜽

𝒙; 𝜎

𝑡

and

𝒙

Cosine Similarity between

𝑫

𝜽

𝒙; 𝜎

𝑡

and

𝒙

(b)

(a)

(c)

(d)

1.0

0.200

1.0

EDM-VE (FFHQ)

1.000

0.175

EDM-VP (FFHQ)

0.995

Normalized MSE

0.8

0.6

3SW

pazij

0.150

0.125

Aruei(s

0.8

9'0

EDM-ADM (FFHQ)

Baseline-VP (FFHQ)

Aues

0.990

0.985

0.2

0.4

EDM-VE (FFHQ)

EDM-VP (FFHQ)

EDM-ADM (FFHQ)

Normal

0.050

0.100

0.075

EDM-VE (FFHQ)

EDM-VP (FFHQ)

EDM-ADM (FFHQ)

Cosine

0.4

0.2

0.980

0.975

0.970

EDM-VE (FFHQ)

EDM-VP (FFHQ)

EDM-ADM (FFHQ)

0.025

0.0

0.965

0.0

0

10

20

30

40

50

Baseline-VP (FFHQ)

60

70

80

0.000

0.00

0.02

0.04

90°0

0.08

Baseline-VP (FFHQ)

0.10

0.12

0

10

20

30

40

50

60

70

80

0.960

0.00

Baseline-VP (FFHQ)

0.02

0.04

0.06

0.08

0.10

0.12

E Theoretical Analysis

E.1 Proof of Theorem 1

In this section, we give the proof of Theorem 1 (Section 3.3). Our theorem is based on the following two assumptions:

Assumption 1. Suppose that the diffusion denoisers are parameterized as single-layer linear networks, defined as D ( x ; σ ( t )) = W σ ( t ) x + b σ ( t ) , where W σ ( t ) ∈ R d × d is the linear weight and b σ ( t ) ∈ R d is the bias.

Assumption 2. The data distribution p data ( x ) has finite mean µ and bounded positive semidefinite covariance Σ

Theorem 1. Under Assumption 1 and Assumption 2, the optimal solution to the denoising score matching objective (3) is exactly the Gaussian denoiser: D G ( x , σ ( t )) = µ + U ˜ Λ σ ( t ) U T ( x -µ ) , where Σ = U Λ U T represents the SVD of the covariance matrix, with singular values λ { k =1 ,...,d } and ˜ Λ σ ( t ) = diag[ λ k λ k + σ ( t ) 2 ] . Furthermore, this optimal solution can be obtained via gradient descent with a proper learning rate.

To prove Theorem 1 , we first show that the Gaussian denoiser is the optimal solution to the denoising score matching objective under the linear network constraint. Then we will show that such optimal solution can be obtained via gradient descent with a proper learning rate.

The Global Optimal Solution. Under the constraint that the diffusion denoiser is restricted to a single-layer linear network with bias:

We get the following optimizaiton problem from Equation (3):

where we omit the footnote σ ( t ) in W σ ( t ) and b σ ( t ) for simplicity. Since expectation preserves convexity, the optimization problem Equation (16) is a convex optimization problem. To find the global optimum, we first eliminate b by requiring the partial derivative ∇ b L ( W , b ; σ ( t )) to be 0 . Since

we have

Utilizing the expression for b , we get the following equivalent form of the optimization problem:

The derivative ∇ W L ( W ; σ ( t )) is:

Suppose Σ = U Λ U T is the SVD of the empirical covariance matrix, with singular values λ { k =1 ,...,n } , by setting ∇ W L ( W ; σ ( t )) to 0 , we get the optimal solution:

where ˜ Λ σ ( t ) [ i, i ] = λ i λ i + σ ( t ) 2 and λ i = Λ [ i, i ] . Substitute W ⋆ back to Equation (20), we have:

Notice that the expression for W ⋆ and b ⋆ is exactly the Gaussian denoiser. Next, we will show this optimal solution can be achieved with gradient descent.

Gradient Descent Recovers the Optimal Solution. Consider minimizing the population loss:

Define ˜ W := [ W b ] , ˜ x := [ x 1 ] and ˜ ϵ = [ ϵ 0 ] , then we can rewrite Equation (28) as:

We can compute the gradient in terms of ˜ W as:

Since E ϵ (˜ ϵ ) = 0 and E ϵ (˜ ϵ ˜ ϵ T ) = [ σ ( t ) 2 I d × d 0 d × 1 0 1 × d 0 ] , we have:

Since E (˜ x ˜ x T ) = [ E ( xx T ) E ( x ) E ( x T ) 1 ] , we have:

With learning rate η , we can write the update rule as:

where we define A := I -2 η [ E x ( xx T ) + σ ( t ) 2 I µ µ T 1 ] for simplicity. By recursively expanding the expression for ˜ W , we have:

Notice that there exists a η , such that every eigen value of A is smaller than 1 and greater than 0. In this case, A t +1 → 0 as t →∞ . Similarly, by the property of matrix geometric series, we have ∑ t i =0 A i → ( I -A ) -1 . Therefore we have:

where we define B := E x ( xx T ) + σ ( t ) 2 I for simplicity. By the Sherman-Morrison-Woodbury formula, we have:

Therefore, we have:

from which we have therefore

Therefore, gradient descent with a properly chosen learning rate η recovers the Gaussian Denoisers when time goes to infinity.

E.2 Two Extreme Cases

Our empirical results indicate that the best linear approximation of D θ is approximately equivalent to D G . According to the orthogonality principle [36], this requires D θ to satisfy:

Notice that (51) does not hold for general denoisers. Two extreme cases for this to hold are:

, which implies

Similarly, we have:

Since E x [ xx T ] = E x [( x -µ )(( x -µ ) T ] + µµ T , we have:

Case 1: D θ ( x + ϵ ; σ ( t )) ≈ x for ∀ x ∼ p data , ϵ ∼ N ( 0 , σ ( t ) 2 I ) .

Case 2: D θ ( x + ϵ ; σ ( t )) ≈ D G ( x + ϵ ; σ ( t )) for ∀ x ∼ p data , ϵ ∼ N ( 0 , σ ( t ) 2 I ) .

Case 1 requires D θ ( x + ϵ ; σ ( t )) to be the oracle denoiser that perfectly recover the ground truth clean image, which never happens in practice except when σ ( t ) becomes extremely small. Instead, our empirical results suggest diffusion models in the generalization regime bias towards Case 2, where deep networks learn D θ that approximate (not equal) to D G . This is evidenced in Figure 5(b), where diffusion models trained on larger datasets (35000 and 7000 images) produce denoising outputs similar to D G . Notice that this similarity holds even when the denoisers take pure Gaussian noise as input. The exact mechanism driving diffusion models trained with gradient descent towards this particular solution remains an open question and we leave it as future work.

F More Discussion on Section 4

While in Section 4 we mainly focus on the discussion of the behavior of diffusion denoisers in the intermediate-noise regime, in this section we study the denoiser dynamics in both low and high-noise regime. We also provide additional experiment results on CIFAR-10 dataset.

F.1 Behaviors in Low-noise Regime

We visualize the score differences between D G and D θ in low-noise regime in Figure 16. The left figure demonstrates that when the dataset size becomes smaller than a certain threshold, the score difference at σ = 0 remains persistently non-zero. Moreover, the right figure shows that this difference depends solely on dataset size rather than model capacity. This phenomenon arises from two key factors: (i) D θ converges to the identity mapping at low noise levels, independent of training dataset size and model capacity, and (ii) D G approximates the identity mapping at low noise levels only when the empirical covariance matrix is full-rank, as can be seen from (6).

Since the rank of the covariance matrix is upper-bounded by the training dataset size, D G differs from the identity mapping when the dataset size is smaller than the data dimension. This creates a persistent gap between D G and D θ , with smaller datasets leading to lower rank and consequently larger score differences. These observations align with our discussion in Appendix D.

F.2 Behaviors in High-noise Regime

As shown in Figure 7(a), while a decreased model scale pushes D θ in the intermediate noise region towards D G , their differences enlarges in the high noise variance regime. This phenomenon arises because diffusion models employ a bell-shaped noise sampling distribution that prioritizes intermediate noise levels, resulting in insufficient training at high noise variances. A shown in Figure 17, for high σ ( t ) , D θ converge to D G when trained with sufficient model capacity (Figure 17(b)) and training time (Figure 17(c)). This behavior is consistent irrespective of the training dataset sizes (Figure 17(a)). Convergence in the high-noise variance regime is less crucial in practice, since diffusion steps in

ScoreField ApproximationError(Gaussianvs.EDM)

ScoreField Approximation Error(Gaussianv.s.EDM)

0.40

dataset size = 68

0.30

Scale = 128 (70000 images)

0.35

1

dataset size = 137

dataset size = 1094

0.25

Scale =128 (1094 images)

Scale =64(1094 images)

dataset size = 8750

Scale =32(1094 images)

0.30

datasetsize=35000

Scale=16(1094images)

datasetsize=70000

0.20

Scale=8(1094images)

0.25

Scale=4 (1094 images)

RMSE

RMSE

0.20

0.15

0.15

0.10

0.10

0.05

0.05

0.00

0.00

0.0

0.2

0.4

0.6

0.8

1.0

0'0

0.2

0.4

0.6

0.8

1.0

NoiseVariance

Denoising Outputs  for 𝜎 𝑡 = 60 (PSNR = -29.5 dB)

Effect of Model Scale (1094 training images)

Effect of Training Epochs (1094 training images)

Effect of Dataset Size

𝒚 = 𝒙 + 𝜎 𝑡

∗ 𝝐

𝐷

!

(𝒚; 𝜎(𝑡))

(68)

𝐷

!

(𝒚; 𝜎(𝑡))

(1094)

𝐷

!

(𝒚; 𝜎(𝑡))

(35000)

𝐷

!

(𝒚; 𝜎(𝑡))

(70000)

𝐷

𝜽

(𝒚; 𝜎(𝑡))

(68)

𝐷

𝜽

(𝒚; 𝜎(𝑡))

(1094)

𝐷

𝜽

(𝒚; 𝜎(𝑡))

(35000)

𝐷

𝜽

(𝒚; 𝜎(𝑡))

(70000)

𝐷

#

(𝒚; 𝜎(𝑡))

(68)

𝐷

#

(𝒚; 𝜎(𝑡))

(1094)

𝐷

#

(𝒚; 𝜎(𝑡))

(35000)

𝐷

#

(𝒚; 𝜎(𝑡))

(70000)

𝐷

𝜽

(𝒚; 𝜎(𝑡))

(4)

𝐷

𝜽

(𝒚; 𝜎(𝑡))

(8)

𝐷

𝜽

(𝒚; 𝜎(𝑡))

(64)

𝐷

𝜽

(𝒚; 𝜎(𝑡))

(128)

𝐷

𝜽

(𝒚; 𝜎(𝑡))

(187)

𝐷

𝜽

(𝒚; 𝜎(𝑡))

(841)

𝐷

𝜽

(𝒚; 𝜎(𝑡))

(9173)

𝐷

𝜽

(𝒚; 𝜎(𝑡))

(64210)

(a)

(b)

(c)

)

!

("; &('))

)

"

("; &('))

(187)

)

"

("; &('))

(841)

)

"

("; &('))

(9173)

)

"

("; &('))

(64210)

)

!

(#; &('))

)

"

(#; &('))

(187)

)

"

(#; &('))

(841)

)

"

(#; &('))

(9173)

)

"

(#; &('))

(64210)

!

Clean Image

# = ! + "

Denoising Outputs  for

! "

= 4

Memorization

Generalization

(b)

(c)

)

!

("; &('))

)

"

("; &('))

(4)

)

"

("; &('))

(8)

)

"

("; &('))

(64)

)

"

("; &('))

(128)

)

!

(#; &('))

)

"

(#; &('))

(4)

)

"

(#; &('))

(8)

)

"

(#; &('))

(64)

)

"

(#; &('))

(128)

Memorization

Generalization

(a)

Noise

" ∼ .(/, & '

#

1)

this regime contribute substantially less than those in the intermediate-noise variance regime-a phenomenon we analyze further in Appendix G.5.

F.3 Similarity between Diffusion Denoiers and Gaussian Denoisers

In Section 4, we demonstrate that the Gaussian inductive bias is most prominent in models with limited capacity and during early training stages, a finding qualitatively validated in Figure 18. Specifically, Figure 18(b) shows that larger models (channel sizes 128 and 64) tend to memorize,

Memorization

Generalization

(a)

(b)

ScoreField Approximation Error(Gaussian vs.EDM)

dataset size =196

196

782

3125

12500

25000

50000

dataset size=782

0.25

dataset size=3125

dataset size = 12500

EDM

0.20

datasetsize=250oo

dataset size=5oooo

RMSE

0.15

0.10

0.05

NN

000

0

10

20

30

40

50

60

70

80

Generalization

Memorization

(a)

(b)

Score Field Approximation Error (Gaussian vs.EDM)

scale=4

scale=8

scale=16

scale=32

scale=64

scale=128

Scale =4(782 images)

Scale =8(782 images)

EDM

0.4

Scale =16 (782 images)

Scale =32(782 images)

Scale = 64 (782 images)

0.3

Scale = 128 (782 images)

RMSE

NN

0.2

scale = 128 (70000 images)

0.1

0.0

0

10

20

30

40

50

60

70

80

directly retrieving training data as denoising outputs. In contrast, smaller models (channel sizes 8 and 4) exhibit behavior similar to D G , producing comparable denoising outputs. Similarly, Figure 18(c) reveals that during early training epochs (0-841), D θ outputs progressively align with those of D G . However, extended training beyond this point leads to memorization.

F.4 CIFAR-10 Results

The effects of model capacity and training duration on the Gaussian inductive bias, as demonstrated in Figures 19 to 21, extend to the CIFAR-10 dataset. These results confirm our findings from Section 4: the Gaussian inductive bias is most prominent when model scale and training duration are limited.

G Additional Experiment Results

While in the main text we mainly demonstrate our findings using EDM-VE diffusion models trained on FFHQ, in this section we show our results are robust and extend to various model architectures and datasets. Furthermore, we demonstrate that the Gaussian inductive bias is not unique to diffusion models, but it is a fundamental property of denoising autoencoders [37]. Lastly, we verify that our

Generalization

Memorization

(a)

(b)

Score Field Approximation Error (Gaussian vs.EDM)

205 epochs (782 images)

epochs=205

epochs = 921

epochs=2509

epochs = 10035

epochs=70246

0.25

921 epochs (782 images)

2509 epochs (782 images)

EDM

0.20

10035 epochs (782 images)

70246 epochs (782 images)

RMSE

0.15

NN

0.10

scale = 128 (70000 images)

0.05

0.00

0

4

6

8

10

(a)

(b)

(c)

Normalized MSE

EDM-VE(FFHQ)

EDM-ADM(FFHQ)

EDM-VP(FFHQ)

1.0

=80.0

1.0

=80.0

1.0

0=80.0

0 = 42.415

0 = 42.415

0 = 42.415

MSE

E

0.8

0 = 21.109

0 =9.723

0 = 21.109

0 = 9.723

MSE

E

0.8

0 = 21.109

0 = 9.723

Normalized

0.6

0 = 4.066

0 =1.502

0=0.47

0 = 1.502

0=4.066

0=0.47

Normalize

0.6

0=0.47

0=1.502

=4.066

0.4

0=0.02

0=0.117

1

0=0.117

0=0.02

0.4

1

0 = 0.117

0 =0.02

0=0.002

=0.002

0=0.002

0.2

0.2

0.2

0.02.5 5.07.5 10.012.515.017.520.0

0

20

40

60

80

100

0

20

40

60

80

100

Epochs/5

Epochs

Epochs

conclusions remain consistent when using alternative metrics such as NMSE instead of the RMSE used in the main text.

G.1 Gaussian Structure Emerges across Various Network Architectures

We first demonstrate that diffusion models capture the Gaussian structure of the training dataset, irrespective of the deep network architectures used. As shown in Figure 22 (a), (b), and (c), although the actual diffusion models, D θ , are parameterized with different architectures, for all noise variances except σ ( t ) ∈ { 0 . 002 , 80 . 0 } , their corresponding linear models, D L , consistently converge towards the common Gaussian models, D G , determined by the training dataset. Qualitatively, as depicted in Figure 23, despite variations in network architectures, diffusion models generate nearly identical images, matching those generated from the Gaussian models.

G.2 Gaussian Inductive Bias as a General Property of DAEs

In previous sections, we explored the properties of diffusion models by interpreting them as collections of deep denoisers, which are equivalent to the denoising autoencoders (DAEs) [37] trained on various noise variances by minimizing the denoising score matching objective (3). Although diffusion models and DAEs are equivalent in the sense that both of them are trying to learn the score function of the

EDM-VE

EDM-VP

EDM-ADM

noise-mollified data distribution [38], the training objective of diffusion models is more complex [4]:

where x ∼ p data , ϵ ∼ N ( 0 , σ ( t ) 2 I ) and σ ∼ p train . Notice that the training objective of diffusion models has a few distinct characteristics:

Diffusion models use a single deep network F θ to perform denoising score matching across all noise variances while DAEs are typically trained separately for each noise level.

Diffusion models are not trained uniformly across all noise variances. Instead, during training the probability of sampling a given noise level σ is controlled by a predefined distribution p train and the loss is weighted by λ ( σ ) .

Diffusion models often utilize special parameterizations (13). Therefore, the deep network F θ is trained to predict a linear combination of the clean image x and the noise ϵ , whereas DAEs typically predict the clean image directly.

Given these differences, we investigate whether the Gaussian inductive bias is unique to diffusion models or a general characteristic of DAEs. To this end, we train separate DAEs (deep denoisers) using the vanilla denoising score matching objective (3) on each of the 10 discrete noise variances specified by the EDM schedule [80.0, 42.415, 21.108, 9.723, 4.06, 1.501, 0.469, 0.116, 0.020, 0.002], and compare the score differences between them and the corresponding Gaussian denoisers D G . We use no special parameterization so that D θ = F θ ; that is, the deep network directly predicts the clean image. Furthermore, the DAEs for each noise variance are trained till convergence, ensuring all noise levels are trained sufficiently. We consider the following architectural choices:

DAE-NCSN : In this setting, the network F θ uses the NCSN architecture [3], the same as that used in the EDM-VE diffusion model.

DAE-Skip : In this setting, F θ is a U-Net [39] consisting of convolutional layers, batch normalization [40], leaky ReLU activation [41] and convolutional skip connections. We refer to this network as 'Skip-Net'. Compared to NCSN, which adapts the state of the art architecture designs, Skip-Net is deliberately constructed to be as simple as possible to test how architectural complexity affects the Gaussian inductive bias.

DAE-DiT : In this setting, F θ is a Diffusion Transformer (DiT) introduced in [42]. Vision Transformers are known to lack inductive biases such as locality and translation equivariance that are inherent to convolutional models [43]. Here we are interested in if this affects the Gaussian inductive bias.

Generation Trajectories  (                    )  for Various Models

)(+

#

; -(.))

(a)

(b)

ScoreFieldApproximationError

0.14

0.12

0.10

0.08

RMSE

0.06

0.04

0.02

0.00

0.00

0.02

0.12

0.47

1.50

4.07

246

21.1142.42 80.00

80.0

0.002

DAE-Linear : In this setting we set F θ to be a linear model with a bias term as in (8). According to Theorem 1, these models should converge to Gaussian denoisers.

The quantitative results are shown in Figure 24(a). First, the DAE-linear models well approximate D G across all 10 discrete steps (RMSE smaller than 0.04), consistent with Theorem 1. Second, despite the differences between diffusion models (EDM) and DAEs, they achieve similar score approximation errors relative to D G for most noise variances, meaning that they can be similarly approximated by D G . However, diffusion models exhibit significantly larger deviations from D G at higher noise variances ( σ ∈ { 42 . 415 , 80 . 0 } ) since they utilize a bell-shaped noise sampling distribution p train that emphasizes training on intermediate noise levels, leading to under-training at high noise variances. Lastly, the DAEs with different architectures achieve comparable score approximation errors, and both DAEs and diffusion models generate images matching those from the Gaussian model, as shown in Figure 24(b). These findings demonstrate that the Gaussian inductive bias is not unique to diffusion models or specific architectures but is a fundamental property of DAEs.

G.3 Gaussian Structure Emerges across Various datasets

As illustrated in Figure 25, for diffusion models trained on the CIFAR-10, AFHQ and LSUN-Churches datasets that are in the generalization regime, their generated samples match those produced by the corresponding Gaussian models. Additionally, their linear approximations, D L , obtained through linear distillation, align closely with the Gaussian models, D G , resulting in nearly identical generated images. These findings confirm that the Gaussian structure is prevalent across various datasets.

G.4 Strong Generalization on CIFAR-10

Figure 26 demonstrates the strong generalization effect on CIFAR-10. Similar to the observations in Section 5, reducing model capacity or early stopping the training process prompts the Gaussian inductive bias, leading to generalization.

G.5 Measuring Score Approximation Error with NMSE

While in Section 3.1 we define the score field approximation error between denoisers D 1 and D 2 with RMSE ( (10)), this error can also be quantified using NMSE:

As shown in Figure 27, while the trend in intermediate-noise and low-noise regimes remains unchanged, NMSE amplifies differences in the high-noise variance regime compared to RMSE. This amplified score difference between D G and D θ does not contradict our main finding that diffusion models in the generalization regime exhibit an inductive bias towards learning denoisers

LSUN-Churches

Generation Trajectories  (                    )  for Various Models

+(-

#

; /(0))

Final Generated Samples

Generation Trajectories  (                    )  for Various Models

+(-

#

; /(0))

Final Generated Samples

AFHQ

(a)

(b)

(c)

(d)

Final Generated Samples

CIFAR- 10

Generation Trajectories  (                    )  for Various Models

+(-

#

; /(0))

image 1

image 2 image 3

image 4 image 5

(e)

(f)

image1image2image3image4image5

EDM

Multi Delta

Linear

80.0

0.02

0.002

EDM

Multi-Delta

Linear

80.0

21.109

4.066

0.02

0.002

EDM

Multi-Delta

Linear

image1image2image3image4image5

EDM

MultiDelta

Linear

80.0

42.415

21.109

9.723

4.066

1.502

0.47

0.117

0.02

0.002

EDM

DAE

Linear

EDM

MultiDelta

Linear

approximately equivalent to D G in the high-noise variance regime. As discussed in Section 3.2 and appendices F.2 and G.2, this large score difference stems from inadequate training in this regime.

Figure 27 (Gaussian vs. DAE) demonstrates that when DAEs are sufficiently trained at specific noise variances, they still converge to D G . Importantly, the insufficient training in the high-noise variance regime minimally affects final generation quality. Figure 25(f) shows that while the diffusion model (EDM) produces noisy trajectories at early timesteps ( σ ∈ { 80 . 0 , 42 . 415 } ), these artifacts quickly disappear in later stages, indicating that the Gaussian inductive bias is most influential in the intermediate-noise variance regime.

Notably, even when D θ are inadequately trained in the high-noise variance regime, they remain approximable by linear functions, though these functions no longer match D G .

Early Stopping

Decrease Scale

Non-overlapping datasets with size 25000, model scale 64

Generated Images from Gaussian Models (size 25000)

Generated Images from Gaussian Models (size 782)

Non-overlapping datasets with size 782, model scale 128

Early Stopping at Epoch 921

(a)

(b)

(c)

Strong generalizability under small dataset size (782)

Gaussian ef 52

Gaussian of 52

Nearest Neighbor in S1

Nearest Neighbor in S1

EDM trained on S1

EDM trained on S1

EDM trained on S2

EDM trained on S2

Nearest Neighbor in S2

Nearest Neighbor in S2

EDM (S1)

EDM (52)

EDM (S1)

EDM (S2)

(a)

(b)

(c)

(d)

ScoreField Approximation Error(CiFAR-10)

Optimalvs.EDM

+

Gaussianvs.EDM

Linear vs.EDM

0.25

0.20

RMSE

0.15

0.10

0.05

0.00

0

10

20

30

40

50

60

70

80

NoiseVariance

ScoreField Approximation Error(FFHQ)

0.35

+

Gaussianvs.EDM

Linear vs.EDM

0.30

Multi-DeltaVs.EDM

0.25

RMSE

0.20

0.15

0.10

0.05

0.00

0

10

20

30

40

50

60

70

80

Score Field Approximation Error(CIFAR-10)

1.0

Optimalvs.EDM

Gaussianvs.EDM

Linear vs.EDM

0.8

Normalized MSE

0.6

0.4

0.2

0.0

0

10

20

30

40

50

60

70

80

ScoreField ApproximationError(FFHQ)

0.6

Linear vs. EDM

0.5

Multi-DeltaVS.EDM

Normalized MSE

0.4

0.3

0.2

0.1

0.0

0

10

20

30

40

50

09

70

80

H Discussion on Geometry-Adaptive Harmonic Bases

H.1 GAHB only Partially Explain the Strong Generalization

Recent work [20] observes that diffusion models trained on sufficiently large non-overlapping datasets (of the same class) generate nearly identical images. They explain this 'strong generalization' phenomenon by analyzing bias-free deep diffusion denoisers with piecewise linear input-output

mappings:

where λ k ( x t ) , u k ( x t ) , and v k ( x t ) represent the input-dependent singular values, left and right singular vectors of the network Jacobian ∇D ( x t ; σ ( t )) . Under this framework, strong generalization occurs when two denoisers D 1 and D 2 have similar Jacobians: ∇D 1 ( x t ; σ ( t )) ≈ ∇D 2 ( x t ; σ ( t )) . The authors conjecture this similarity arises from networks' inductive bias towards learning certain optimal ∇D ( x t ; σ ( t )) that has sparse singular values and the singular vectors of which are the geometry-adaptive harmonic bases (GAHB)-near-optimal denoising bases that adapt to input x t .

While [20] provides valuable insights, their bias-free assumption does not reflect real-world diffusion models, which inherently contain bias terms. For feed forward ReLU networks, the denoisers are piecewise affine:

where b x t is the network bias that depends on both network parameterization and the noisy input x t [44]. Here, similar Jacobians alone cannot explain strong generalization, as networks may differ significantly in b x t . For more complex network architectures where even piecewise affinity fails, we consider the local linear expansion of D ( x t ; σ ( t )) :

which approximately holds for small perturbation ∆ x . Thus, although ∇D ( x t ; σ ( t )) characterizes D ( x t ; σ ( t )) 's local behavior around x t , it does not provide sufficient information on the global properties.

Our work instead examines global behavior, demonstrating that D ( x t ; σ ( t )) is close to D G ( x t ; σ ( t )) -the optimal linear denoiser under the Gaussian data assumption. This implies that strong generalization partially stems from networks learning similar Gaussian structures across non-overlapping datasets of the same class. Since our linear model captures global properties but not local characteristics, it complements the local analysis in [20].

H.2 GAHB Emerge only in Intermediate-Noise Regime

For completeness, we study the evolution of the Jacobian matrix ∇D ( x t ; σ ( t )) across various noise levels σ ( t ) . The results are presented in Figures 28 and 29, which reveal three distinct regimes:

High-noise regime [10,80]. In this regime, the leading singular vectors 6 of the Jacobian matrix ∇D ( x t ; σ ( t )) well align with those of the Gaussian weights (the leading principal components of the training dataset), consistent with our finding that diffusion denoisers approximate linear Gaussian denoisers in this regime. Notice that DAEs trained sufficiently on separate noise levels (Figure 29) show stronger alignment compared to vanilla diffusion models (Figure 28), which suffer from insufficient training at high noise levels.

Intermediate-noise regime [0.1,10]: In this regime, GAHB emerge as singular vectors of ∇D ( x t ; σ ( t )) diverge from the principal components, becoming increasingly adaptive to the geometry of input image.

Low-noise regime [0.002,0.1]. In this regime, the leading singular vectors of ∇D ( x t ; σ ( t )) show no clear patterns, consistent with our observation that diffusion denoisers approach the identical mapping, which has unconstrained singular vectors.

Notice that the leading singular vectors of ∇D ( x t ; σ ( t )) are the input directions that lead to the maximum variation in denoised outputs, thus revealing meaningful information on the local properties of D ( x t ; σ ( t )) at x t . As demonstrated in Figure 30, perturbing input x t along these vectors at difference noise regimes leads to distinct effects on the final generated images: (i) in the high-noise regime where the leading singular vectors align with the principal components of the training dataset,

&("

; $(%))

Generation Trajectories

(f)

#

(a)

Correlation Matrices                       across Various

$(%)

)

%

)("

#

)

(b)

)

(

)

)

)

*

(c)

*

+

("

,

)

across Various

$(%)

(d)

(e)

*

&

("

,

)

across Various

$(%)

*

-

("

,

)

across Various

$(%)

80.0

43.92

22.794

16.022

11.054

4.931

1.979

0.693

0.201

0.002

0

0

Q

Q

0

0

0

Q

10

10

10

10

10

10

10

10

10

10

20

20

20

20

20

20

20

20

20

20

30

30

30

OE

30

30

30

30

30

30

0

20

0

20

0

20

0

20

0

20

0

20

0

20

0

20

0

20

0

20

80.0

43.92

22.794

16.022

11.054

4.931

1.979

69'0

0.201

0.002

Q

0

0

0

0

Q

Q

0

0

0

5

5

u

5

5

u

u

5

10

10

10

10

10

10

10

10

10

10

15

15

15

15

15

15

15

15

15

15

20

20

20

20

20

20

20

20

20

20

0

10

20

0

10

20

0

10

20

0

10

20

D

10

20

0

10

20

0

10

20

10

20

0

10

20

10

20

0

10

10

10

20

20

20

30

30

30

20

0

20

0

20

80.0

43.92

22.794

16.022

11.054

4.931

1.979

0.693

0.201

0.002

O

O

O

0

0

0

0

10

10

10

10

10

10

10

10

10

10

20

20

20

20

20

20

20

20

20

20

30

OE

30

30

30

0E

OE

30

OE

30

0

20

0

20

0

20

0

20

0

20

0

20

0

20

D

20

0

20

0

20

80.0

43.92

22.794

16.022

11.054

4.931

1.979

0.693

0.201

0.002

0

Q

0

0

0

0

0

0

Q

10

10

10

10

10

10

10

10

10

10

20

20

20

20

20

20

20

20

20

20

30

30

30

OE

30

30

30

30

OE

30

0

20

0

20

0

20

0

20

0

20

0

20

0

20

20

0

20

0

20

80.0

43.92

22.794

16.022

11.054

4.931

1.979

0.693

0.201

0.002

Q

0

0

0

0

0

0

0

0

Q

10

10

10

10

10

10

10

10

10

10

20

20

20

20

20

20

20

20

20

20

30

OE

30

OE

30

30

OE

30

OE

30

0

20

0

20

0

20

0

20

0

20

20

0

20

0

20

0

20

0

20

perturbing x t along these directions leads to canonical changes such as image class, (ii) in the intermediate-noise regime where the GAHB emerge, perturbing x t along the leading singular vectors modify image details such as colors while preserving overall image structure and (iii) in the low-noise regime where the leading singular vectors have no significant pattern, perturbing x t along these directions yield no meaningful semantic changes.

These results collectively demonstrate that the singular vectors of the network Jacobian ∇D ( x t ; σ ( t )) have distinct properties at different noise regimes, with GAHB emerging specifically in the intermediate regime. This characterization has significant implications for uncertainty quantification [45] and image editing [46].

(

(a)

)

)

("

+

)

across Various

$(%)

(b)

)

*

("

+

)

across Various

$(%)

(c)

)

(

("

+

)

across Various

$(%)

(d)

+

80.0

42.415

21.109

9.723

4.066

1.502

0.47

0.117

0.02

0.002

0

0

口

10

10

10

10

10

10

10

10

10

10

20

20

20

20

20

20

20

20

20

20

0

10

20

OT

20

0

10

20

0

10

20

0

10

20

0

10

20

10

20

10

20

0

10

20

10

20

80.0

42.415

21.109

9.723

4.066

1.502

0.47

0.117

0.02

0.002

O

O

O

0

0

0

0

0

10

10

10

10

10

10

10

10

10

10

20

20

20

20

20

20

20

20

20

20

30

OE

30

OE

30

0E

OE

30

OE

30

0

20

0

20

0

20

0

20

0

20

0

20

0

20

D

20

0

20

0

20

80.0

42.415

21.109

9.723

4.066

1.502

0.47

0.117

0.02

0.002

0

O

0

0

0

0

O

0

0

10

10

10

10

10

10

10

10

10

10

20

20

20

20

20

20

20

20

20

20

30

30

30

OE

30

30

30

30

OE

30

0

20

0

20

0

20

0

20

0

20

0

20

0

20

0

20

0

20

0

20

80.0

42.415

21.109

9.723

4.066

1.502

0.47

0.117

0.02

0.002

O

D

O

10

10

10

10

10

10

10

10

10

10

20

20

20

20

20

20

20

20

20

20

30

OE

30

OE

30

30

30

30

30

30

0

20

0

20

0

20

0

20

0

20

D

20

0

20

20

0

20

0

20

Correlation Matrices                       across Various

$(%)

%

(("

)

! "

= 22.79

! "

= 1.979

! "

= 0.002

+"

*

($

+

)

-"

*

($

+

)

+"

*

($

+

)

-"

*

($

+

)

+"

*

($

+

)

-"

*

($

+

)

(a)

(b)

(c)

I Computing Resources

All the diffusion models in the experiments are trained on A100 GPUs provided by NCSA Delta GPU [33].

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While '[Yes] ' is generally preferable to '[No] ', it is perfectly acceptable to answer '[No] ' provided a proper justification is given (e.g., 'error bars are not reported because it would be too computationally expensive' or 'we were unable to find the license for the dataset we used'). In general, answering '[No] ' or '[NA] ' is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist' ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We mention the main contribution in both the abstract and introduction section, listed in the 'Contributions'.

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations in the 'Discussion' section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide the assumption and proof in Section 3.3 and Appendix E.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We include the experiment details in Section 3, Section 4 and Section 5. We will also release our code upon publication.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We include the experiment details in Section 3, Section 4 and Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: For various plots we specify that the numbers are calculated over 100 random samples. However, since the standard deviations are quite small, the error bar is unnecessary.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We include the computating resources in Appendix I.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research doesn't have potential harm cause by research process and negative societal impact.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work is an empirical and theoretical work on the inductive bias of the diffusion models. It has more scientific contributions rather than societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work is an empirical and theoretical work on how the diffusion model learns the data distribution. It doesn't have a high risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use the public released dataset and models, which properly credited the license.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our work is an empirical and theoretical work on how the diffusion model learns the data distribution. We don't release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper compares diffusion models trained on natural image datasets with their Gaussian approximations. It evaluates the quality of this approximation in both the memorization and generalization regime by studying the influence of the training set size, model capacity, and training time.\n\nSTRENGTHS:\nThis paper makes several novel and interesting empirical observations:\n- according to the authors' linearity measure, denoisers are surprisingly well-aproximated by linear functions (but more on this below),\n- reducing model capacity or training time can bring diffusion models into the generalization regime even with very small training sets, though at the expense of image quality.\n\nIt is also clearly written, and studies the very important problem of characterizing the inductive biases of diffusion models/network denoisers which allow them to generalize.\n\nWEAKNESSES:\nI have two main issues with this paper:\n- First, a large part of its content (most of section 3) is rather obvious for people with a signal/image processing background. Indeed, Theorem 1 is well known, usually under the name of \"Wiener filter\" (which is simply a consequence of linear regression): see, e.g., Theorem 11.3 from _A Wavelet Tour of Signal Processing, Third Edition: The Sparse Way_, Stephane Mallat, Elsevier/Academic Press, 2009. It immediately implies that differences between the \"linear\" and \"Gaussian\" denoisers can only come from suboptimality of the network denoiser (or optimization errors of the linear denoiser), and are thus irrelevant for this study. A second observation that the authors did not make is that any linear denoiser leads to Gaussian generated images (as is evident from equation (4) since $x_{i+1}$ is then a linear function of $x_i$ and $x_0$ is Gaussian). Further, the Gaussian denoiser produces samples from the Gaussian distribution with the same mean and covariance as the training set, which is thus a very simple (but crude) model for which samples can easily be produced without any diffusion procedure.\n- The main claim of the paper then boils down to whether Gaussian distributions are a good approximation of natural image distributions (and in particular, a better approximation than the empirical distribution of the training set). These two examples are interesting to contrast, as they lie on the two extremes of the quality-diversity tradeoff (Gaussian distributions maximize entropy but have very low quality, while the a sum of delta functions at the training images has perfect quality but essentially no diversity). However, Gaussian models are extremely crude and have been extensively studied in the past, and we therefore have few things left to learn from them. It is not surprising that good diffusion models are capable of capturing the covariance of their training data, but this is well understood. As the authors acknowledge in the discussion, what we do not understand is the rest (higher-order moments captured with non-linear denoisers). The visual similarity between faces and Gaussian images with matched first and second moments results from the fact that the faces are centered, so that they are well-approximated with a PCA, also known as \"eigenfaces\": _Low-dimensional procedure for the characterization of human faces._ L. Sirovich; M. Kirby (1987), Journal of the Optical Society of America A. 4 (3): 519–524.). This similarity breaks down for more complex datasets such as LSUN-churches which are more translation-invariant, leading to stationary Gaussian samples which look like textures. I therefore disagree with the main claim that \"the image contents generated with such Gaussian denoisers [...] closely resemble those produced by well-trained diffusion models.\" For the same reason, the inductive biases of network denoisers cannot be reduced to the Gaussian structure only, as they are capable of learning much more complex structures despite the curse of dimensionality. \n\nSpecific points:\n- The sentence at lines 164-166 is wrong, as shown in Figure 1: the denoiser is linear for very small $\\sigma$ (close to identity) and very large $\\sigma$ (close to its Gaussian approximation as noted in [19]), but less linear in the middle range.\n- Root mean square error in equation (8) should have the square root outside the expected value.\n- The RMS plots in Figures 2, 6, 7, and 8 could be improved in several ways. First, they are difficult to interpret due to the lack of a reference point. I suggest normalizing them by the expected RMS norm of the network denoiser, so as to have values in $[0,1]$. Second, I suggest to use a continuous colormap when a single parameter is being varied (such as the size of the training set or number of epochs), in order to facilitate visual analysis. Third, why are the numerical values for these varied parameters chosen arbitrarily and irregularly? It would be easier to see trends with regularly (e.g., linearly or logarithmically) spaced parameters, chosen at round-ish numbers.\n- The authors should mention in section 3.3 that the Gaussian denoisers compared in Figure 6 use the empirical covariances of the smaller training sets, this is implicit and confused me for a while.\n- Two observations relative to memorization and generalization made by the authors are straightforward to me. First, any generative model which perfectly optimizes its training loss will only reproduce its training set, so it is clear that trained network denoisers are suboptimal for their training loss, _and in a specific way_ that enables them to generalize \"correctly\". Second, a Gaussian model will strongly generalize in the sense of [15] when the two training sets have similar first and second-order statistics, which then happens quite rapidly (and can be studied precisely with random matrix theory analyses).\n\nMinor: \n- Typo line 286: \"diffsuion\"\n- Typo caption of Figure 9: \"Gausisan\" (twice)\n- Typo line 316: \"covaraince\"\n- Equations (9) and the first of Appendix C should are missing $c_{\\rm out}(\\sigma(t))$\n- line 504: footnote -> subscript\n\nQUESTIONS:\n- Given the points above, the surprisingly high cosine similarities reported in Figure 1 look suspicious to me. If denoisers were truly linear, then they would generate Gaussian images, and we know that they learn much better models than this. As noted by the authors in appendix A, the linearity measure $\\texttt{LS}(t)$ evaluates the denoiser out-of-distribution since linear combinations of natural images are not natural images (they are superpositions of two unrelated images). Doesn't this mean that these cosine similarities are not really meaningful and deceiving? Another factor that could influence these high values is that they are computed for the denoiser function $\\mathcal D_\\theta(x)$, as opposed to the denoiser residual $x - \\mathcal D_\\theta(x)$ (or equivalently the score). As noted in Appendix A, the denoisers are very close to the identity function at low noise and therefore are expected to be close to linear. It would be more interesting to evaluate the linearity of the score, which I suspect is much smaller (I predict it would only decrease as the noise level decreases).\n\nLIMITATIONS:\nAs stated above, the main limitation of this paper is that it studies the Gaussian case which is well-understood. As a result, a large fraction of its results are straightforward (and some even long known), and it misses the point which resides entirely in the non-Gaussian/linear structure learned by these models. I encourage the authors to study the literature in denoising (e.g., Chapter 11 of the aforementioned book by Mallat) and image modeling (for a brief introductory review, see, e.g., Simoncelli, Eero P. _Statistical modeling of photographic images._ Chapter 4.7 of Handbook of Video and Image Processing 9 (2005).)\n\nHowever, these topics are unfortunately not well-known by the broader machine learning community. I thus think that this paper could provide a service to the community by reviewing what is known about the Gaussian case (after major changes, e.g., to remove the linear denoiser and only mention the Gaussian denoiser). It also makes several novel and interesting observations that are valuable to the community (see strengths).\n\nIn the current state of the paper, I recommend to reject, and encourage the authors to resubmit their work after another iteration.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

comments: SUMMARY:
The work analyzes the behavior of a diffusion-based generative model from the perspective of ``Gaussian structures''. In particular, it checks the linearity of scores at single-time steps and compares scores against a Gaussian model.  The analysis shows that the Gaussian structure plays a main role in image generation.

STRENGTHS:
The work has compared diffusion-based generative models against a Gaussian model. It has discovered convincing evidence that linear score plays an important role in image generation. 

It has done extensive experiments with different configurations. By controlling irrelevant variables, the study highlights factors that affect generalization. In particular, it discovers that the linear score mimicking a Gaussian model plays an important role.

WEAKNESSES:
The study is only limited to a few generative models and a face dataset. The observation might be different on different datasets. The face dataset has a more obvious Gaussian distribution. For example, the mean of all faces is a face, and correlations between pixels are relatively consistent. Therefore, a Gaussian distribution is somewhat reasonable for such a dataset. It is unknown whether a Gaussian distribution is still a reasonable choice for a set of diverse images. In such a case, does the diffusion model still need to mimic the Gaussian model? 


The current analysis still could not explain all observations. For example, the behavior in noise levels between 20 and 80 is not well explained in Figure 6. These observations might be worth more in-depth discussion.

QUESTIONS:
Have you considered using standard Gaussian distribution as the convergent distribution of the diffusion process?

LIMITATIONS:
The scope of the study could be wider.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the generalization properties of diffusion models by examining the learned score functions, which are denoisers trained on various noise levels. It shows that nonlinear diffusion denoisers exhibit linearity when the model can generalize, leading to the idea of distilling these nonlinear mappings into linear models. The findings suggest that these linear denoisers closely align with optimal denoisers for a multivariate Gaussian distribution, indicating an inductive bias towards capturing Gaussian structures in the training data. This bias becomes more pronounced with smaller model sizes and might provide insights into the strong generalizability seen in real-world diffusion models.\n\nSTRENGTHS:\n1. The paper is well-motivated and written. \n2.  The Gaussian structure hypothesis for generalization in diffusion models is interesting.\n\nWEAKNESSES:\n- Although the paper's focus is on generalization in diffusion models, it lacks any quantitative measure of generalization within the experiments presented.\n- Similarly, no quantitative measure of memorization is reported. \n- The work primarily investigates the linearity of the learned score functions across various noise levels. However, the connection between these experiments and the necessity of linearity for generalization in diffusion models is unclear and not well substantiated.\n- The term \"inductive bias\" is frequently used throughout the paper, but it is never clearly defined. It remains ambiguous whether this bias pertains to the model architecture, the parameterization of the forward diffusion process, or the denoising score matching loss.\n- The paper lacks detailed information on the experimental setup, including training procedures and the hyperparameters of the architectures used.\n\nQUESTIONS:\n1. Can the authors explain how from their set of experiments we can conclude that the emergence of Gaussian structure leads to strong generalizability? \n1. The paper shows how the linear (distilled) model resembles the optimal denoiser for a multivariate Gaussian distribution. Would the same phenomena be observed if we would linearize the trained model instead of distilling it? Have the authors conducted any experiments in this direction?\n1. Could the authors provide a more precise definition of inductive bias in their work? Specifically, is it a property of the architecture, the input-output mapping parameterized by the network, the forward diffusion process, or the denoising score matching loss?\n1. The authors refer to edm-ve, edm-vp, and edm-adm as different \"*model architectures*\". Could they clarify this terminology? My understanding is that all these models use a U-net based architecture to parameterize the score function, and the difference lies in the parameterization of the forward process SDE.\n1. Do VE and VP stand for variance exploding and variance preserving? How do they differ from ADM?\n1. Can the authors provide experiments that measure generalization compared to memorization in their setup?\n1. In Figure 1, the authors show that when the model is memorizing (yellow dashed curve), the score function is not approximately linear (at least for some noise levels). However, there is no evidence reported that the yellow curve corresponds to memorization compared to the other curves. Could the authors provide empirical evidence to support this?\n1. Can the authors provide details of their experimental setup, including training and architectures?\n1. Could the authors better discuss the comparison with [1]  in more detail? They claim, \"*We propose an alternative explanation: the networks capture certain common structural features inherent across the non-overlapping datasets*\". How is this claim in contrast with the hypothesis that due to the inductive bias of the neural network, the score and therefore the density are learned? I fail to understand how learning \"common structural features\" differs from learning the density. Could the authors elaborate on this?\n\nLIMITATIONS:\n- The description of the experimental setup lacks essential details, making it difficult to replicate the results.\n- The experiments exclusively focus on the RMSE as the performance metric across various noise scales. Alternative performance measures are neither reported nor discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}