Understanding Generalizability of Diffusion Models Requires Rethinking the Hidden Gaussian Structure
Authors: Xiang Li
Yixiang Dai
Qing Qu
Venue: NeurIPS 2024 poster (2024)
Keywords: diffusion models
inductive bias
generalization
memorization
Primary area: diffusion_based_models

Abstract: In this work, we study the generalizability of diffusion models by looking into the hidden properties of the learned score functions, which are essentially a series of deep denoisers trained on various noise levels. We observe that as diffusion models transition from memorization to generalization, their corresponding nonlinear diffusion denoisers exhibit increasing linearity. This discovery leads us to investigate the linear counterparts of the nonlinear diffusion models, which are a series of linear models trained to match the function mappings of the nonlinear diffusion denoisers. Surprisingly, these linear denoisers are approximately the optimal denoisers for a multivariate Gaussian distribution characterized by the empirical mean and covariance of the training dataset. This finding implies that diffusion models have the inductive bias towards capturing and utilizing the Gaussian structure (covariance information) of the training dataset for data generation. We empirically demonstrate that this inductive bias is a unique property of diffusion models in the generalization regime, which becomes increasingly evident when the model's capacity is relatively small compared to the training dataset size. In the case that the model is highly overparameterized, this inductive bias emerges during the initial training phases before the model fully memorizes its training data. Our study provides crucial insights into understanding the notable strong generalization phenomenon recently observed in real-world diffusion models.

Understanding Generalizability of Diffusion Models Requires Rethinking the Hidden Gaussian Structure
Xiang Li', Yixiang Dai', Qing Qu'
'Department of EECS, University of Michigan,
forkobe umich. yixiang@ yixiang@umich.edd umich.edu, qingqu @omiih.uuiiihiic
Abstract
In this work, we study the generalizability of diffusion models by looking into the hidden properties of the learned score functions, which are essentially a series of deep denoisers trained on various noise levels. We observe that as diffusion models transition from memorization to generalization, their corresponding non- linear diffusion denoisers exhibit increasing linearity. This discovery leads us investigate the linear counterparts of the nonlinear diffusion models, which are a series of linear models trained to match the function mappings of the nonlinear diffusion denoisers. Interestingly, these linear denoisers are approximately the optimal denoisers for a multivariate Gaussian distribution characterized by the empirical mean and covariance of the training dataset. This finding implies that dif- fusion models have the inductive bias towards capturing and utilizing the Gaussian structure (covariance information) of the training dataset for data generation. We empirically demonstrate that this inductive bias is unique property of diffusion models in the generalization regime, which becomes increasingly evident when the model's capacity is relatively small compared to the training dataset size. In the case where the model is highly overparameterized, this inductive bias emerges during the initial training phases before the model fully memorizes its training data. Our study provides crucial insights into understanding the notable strong generalization phenomenon recently observed in real-world diffusion models.
Introduction
In recent years, diffusion models [1-4] have become one of the leading generative models, powering the state-of-the-art image generation systems such as Stable Diffusion [5]. To understand the empirical success of diffusion models, several works [6-12] have focused on their sampling behavior, showing that the data distribution can be effectively estimated in the reverse sampling process, assuming that the score function is learned accurately. Meanwhile, other works [13-18] investigate the learning of score functions, showing that effective approximation can be achieved with score matching loss under certain assumptions. However, these theoretical insights, grounded in simplified assumptions about data distribution and neural network architectures. do not fully capture the complex dynamics of diffusion models in practical scenarios. One significant discrepancy between theory and practice that real-world diffusion models are trained only on a finite number of data points. As argued in [19], theoretically perfectly learned score function over the empirical data distribution can only replicate the training data. In contrast, diffusion models trained on finite samples exhibit remarkable generalizability, producing high-quality images that significantly differ from the training examples. Therefore, a good understanding of the remarkable generative power of diffusion models still lacking.
In this work, we aim deepen the understanding of generalizability in diffusion models by analyzing the inherent properties of the learned score functions. Essentially, the score functions can be interpreted as a series of deep denoisers trained on various noise levels. These denoisers are then chained together to progressively denoise randomly sampled Gaussian noise into its corresponding
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

clean image, thus, understanding the function mappings of these diffusion denoisers is critical to demystify the working mechanism of diffusion models. Motivated by the linearity observed in the diffusion denoisers of effectively generalized diffusion models, we propose to elucidate their function mappings with linear distillation approach, where the resulting linear models serve as the linear approximations of their nonlinear counterparts.
Contributions of this work: Our key findings can be highlighted follows:
Inductive bias towards Gaussian structures (Section 3). Diffusion models in the generalization regime exhibit an inductive bias towards learning diffusion denoisers that are close (but not equal) to the optimal denoisers for a multivariate Gaussian distribution, defined by the empirical mean and covariance of the training data. This implies the diffusion models have the inductive bias towards capturing the Gaussian structure (covariance information) of the training data for image generation.
Model Capacity and Training Duration (Section 4) We show that this inductive bias is most pronounced when the model capacity is relatively small compared to the size of the training data. However, even if the model is highly overparameterized, such inductive bias still emerges during early training phases, before the model memorizes its training data. This implies that early stopping can prompt generalization in overparameterized diffusion models.
Connection between Strong Generalization and Gaussian Structure (Section 5). Lastly, we argue that the recently observed strong generalization [20] results from diffusion models learning certain common low-dimensional structural features shared across non-overlapping datasets. We show that such low- now-dimensional features can be partially explained through the Gaussian structure.
Relationship with Prior Arts. Recent research [20-24] demonstrates that diffusion models operate in two distinct regimes: (i) memorization regime, where models primarily reproduce training samples and (ii) generalization regime, where models generate high-quality, novel images that extend beyond the training data. In the generalization regime, a particularly intriguing phenomenon is that diffusion models trained on non-overlapping datasets can generate nearly identical samples [20]. While prior work [20] attributes this "strong generalization" effect the structural inductive bias inherent in diffusion models leading to the optimal denoising basis greometry-adaptioe harmonic basis), our research advances this understanding by demonstrating diffusion models" inductive bias towards capturing the Gaussian structure of the training data. Our findings also corroborate with observations of earlier study [25] that the learned score functions of well-trained diffusion models closely align with the optimal score functions of a multivariate Gaussian approximation of the training data.
2 Preliminary
Basics of Diffuusin Models. Given data distribution Pdita (d), where æ Rd diffusion mod- els [1-4] define series of intermediate states p(æ; (t)) by adding Gaussian noise sampled from the data, where (t) is predefined schedule that specifies the noise level time t t [0, T). such that at the end stage the noise mollified distribution p(æ; (T)) is indistinguishable from the pure Gaussian distribution. Subsequently, a new sample is generated by progressively (T) (T)) to its corresponding clean image zy. Following [4], this forward and backward diffusion process can be expressed with a probabilistic ODE:
(O (t)
denoising random noise IT NO0,
dx à (1) log p(æ;; (I))dt.
(1)
In practice the score function logp((;; can be approximated by
V log o (Do(æ:; (t)) =))/
where Da (t)) is parameterized by a deep network with parameters O trained with the denoising score matching objective:
min Em Ee (oo()+)oo(()))))))) ==1]]
+
(3)
In the discrete setting, the reverse ODE in (1) takes the following form:
o(t)) xz + (t: t++1) Do Do (2x
a(t))
0:41 f (1 (t: +++1) (14)
(4)
o(44)

---

where Zo o Notice that at each iteration i, the intermediate sample 2;+1 is the sum of the scaled x; and the denoising output Do (ax;; (t,)). Obviously, the final sampled image is largely determined by the denoiser Do ( (()). If we can understand the function mapping of these diffusion denoisers, we can demystify the working mechanism of diffusion models.
Optimal Diffuusion Denoisers under Simplified Data Assumptions. Under certain assumptions on the data distribution Paata (a)), the optimal diffusion denoisers Do (z; (t)) that minimize the score matching objective (3) can be derived analytically in closed-forms as we discuss below.
Multi-delta distribution of the training data. Suppose the training dataset contains a finite number of data points (y1. U2. .... UN J. natural way to model the data distribution is to represent it as multi-delta distribution: p(x) + E 8(x y:). In this case, the optimal denoiser is
DM( Σ 1=
(5)
Σ
which is essentially a softmax-weight combination of the finite data points. As proved [24], such diffusion denoisers (x; (t)) can only generate exact replicas of the training samples, therefore they have no generalizability.
Multivariate Gaussian distribution. Recent work [25] suggests modeling the data distribution Paaa (x) as multivariate Gaussian distribution p(æ)) = Nu, E), where the mean μ and the co- variance E are approximated by the empirical mean μ / Σi y, and the empirical covariance Σ= v Ee=(0 (y: py -p) =x)" of the training dataset. this case, the optimal denoiser is:
(a (()) = μ + UA U" - 14),
where Σ = UAUT is the SVD of the empirical covariance matrix, with singular values A diag (A1,... -.a,),))) Ad) and Ao(t) = diag
A A Astole)2) With this linear Gaussian denoiser, proved in [25], the sampling trajectory of the probabilistic ODE (1) has close form:
(t)² + Nr uf(T(r (T)+
(xT u
where u, is the Thh singular vector of the empirical covariance matrix. While [25] demonstrate that the Gaussian scores approximate learned scores at high noise variances, we show that they are nearly the best linear approximations of learned scores across much wider range of noise variances.
Generalization vs. Memorization of Diffuusio Models. As the training dataset size increases, diffusion models transition from the memorization regime-whert they can only replicate its training images- to the generalization regime, where the they produce high-quality, novel images [17]. While memorization be interpreted mechanisms underlying the generalization regime remain less well understood. This study aims to explore and elucidate the inductive bias that enables effective generalization in diffusion models.
overfitting of diffusion models to the training samples, the
3 Hidden Linear and Gaussian Structures in Diffusion Models
In this section, we study the intrinsic structures of the learned score functions of diffusion models in the generalization regime. Through various experiments and theoretical investigation, we show that
Diffusion models in the generalization regime have inductive bias towards learning the Gaussian structures of the dataset.
Based on the linearity observed in diffusion denoisers trained in the generalization regime, we propose to investigate their intrinsic properties through a linear distillation technique, with which we train a series of linear models to approximate the nonlinear diffusion denoisers (Section 3.1).
Interestingly, these linear models closely resemble the optimal denoisers for a multivariate Gaussian distribution characterized by the empirical mean and covariance of the training dataset (Section 3.2).
This implies diffusion models have the inductive bias towards learning the Gaussian structure of the training dataset. We theoretically show that the observed Gaussian structure is the optimal solution to the denoising score matching objective under the constraint that the model linear (Section 3.3). In the subsequent sections, although mainly demonstrate results using the FFHQ datasets. our findings are robust and extend to various architectures and datasets, as detailed in Appendix G.
3

---

3.1 Diffusion Models Exhibit Linearity in the Generalization Regime
Our study is motivated by the emerging linearity observed in diffusion models in the generalization regime. Specifically, we quantify the linearity of diffusion denoisers at various noise level ((t) by jointly assessing their" "Additivity" and "Homogeneity" with linearity score (LS) defined by the cosine similarity between Do(acr + Baz; (1)) and (Da( (3Da((z;;
[oo (| [S(()) (app:o0)))()) KK
(arz; a (1))ll2 IaDo (~1: 3Do m2: a (1))l2 I| Do Do(cx + (32;
aDo(ær; + 8Do
where 21,22 pl:; (()), and a E R and 3 E are scalars. In practice, the expectation is approximated with its empirical mean over 100 samples. A more detailed discussion on this choice of measuring linearity is deferred to Appendix A.
Linearity Noise Levels (EDMAVE, FFHQ)
Following the EDM training configuration [4], we set the noise levels (t) within the contin- uous range [0.002,80]. As shown in Figure 1, as diffusion models transition from the mem- orization regime to the generalization regime (increasing the training dataset size), the corre- sponding diffusion denoisers Da exhibit increas- ing linearity. This phenomenon persists across diverse datasets' as well as various training con- figurations ; see Appendix B for more details. This emerging linearity motivates to ask the following questions:
1.00
0.95
Cosine Sirrilarity
0.30
dataset size 70000
dataset size 35000
0.05
dataset 8750
4375
cataset
dataset size 137
dataset size
68
30
50
80
To what extent can a diffusion model be ap- proximated by a linear model?
Naise Variance
Figure 1: Linearity scores of diffusion denoisers. Solid and dashed lines depict the linearity scores across noise variances for models in the general- ization and memorization regimes, respectively, where a 1/V2.
If diffusion models can be approximated lin- early, what are the underlying characteristics of this linear approximation?
Investigating the Linear Structures via Linear Distillation. To address these questions, we investigate the hidden linear structure of diffusion denoisers through linear distillation. Specifically, for given diffusion denoiser Da(æ; (1)) noise level (l), we approximate it with linear function (with bias term) such that:
DL(x; = W W bo(t) = (t)); Vo p(x;
(8)
where the weight W Wa(t) ott) Rddd and bias bo(t) E IRd are learned by solving the following optimiza- tion problem with gradient descent:"
min boco
(e Bew W (1) (1: E) + bo(n) ( (()))1
If these linear models effectively approximate the nonlinear diffusion denoisers, analyzing their weights elucidate the generation mechanism.
While diffusion models are trained on continuous noise variance levels within [0.002,80], we examine the 10 discrete sampling steps specified by the EDM schedule [4]: [80.0, 42.415, 208, 9.723, 4.06, 1.501, 0.469, 061 0.020, 0.002]. These steps are considered sufficient for studying the diffusion mappings for two reasons: (i) images generated using these 10 steps closely match those generated with more steps, and (ii) recent research [30] demonstrates that the diffusion denoisers trained on similar noise variances exhibit analogous function mappings, implying that denoiser behavior at discrete variances represents their behavior at nearby variances.
1 For example, FFHQ [26], CIFAR- 10 [27]. AFHQ [28] and JSUN-Churces [29]. For example, EDM-VE EDM-VI VP and EDM-ADM.
$ Foor the following. the input the vectorized version of the noisy image and the expectation is approximated using finite samples of outptt pairs (æ, + e:, Do (z. + e, (())) with = 1,.., N (see distillation details in Appendix C).

---

Generation Trajectories ((53) Various Models
Score Fiedd Approsimation Error
10.0
21.150
4,000
1.302
0.47
0.02
Gausstan ys. EDM
Linear EDM
1.0
Mult-Delta Vs,
Linear vs. Gaussian
Conrolan 14 Cett-
044
Noton Vartance
Figure 2: Score field approximation error and sampling Trajectory. The left and right figures demonstrate the score field approximation eгтоr and the sampling trajectories D(æ; at) of actual diffusion model (EDM). Multi- Delti- model, linear model and Gaussian model respectively. Notice that the curve corresponding to the Gaussian model almost overlaps with that of the linear model, suggesting they share similar funciton mappings.
After obtaining the linear models DL. we evaluate their differences with the actual nonlinear denoisers Do with the score field approximation error, calculated using the expectation over the root mean square error (RMSE):
DL - -
(t)
Score-Difference(t) Ee---aad(m)e---00000
(10)
RMSE of of randomly sampled and
where d represents the data dimension and the expectation is approximated with its empirical mean. While present RMSE-based results in the main text, our findings remain consistent across alternative metrics, including NMSE, as detailed in Appendix G.
We perform linear distillation on well trained diffusion models operating in the generalization regime. For comprehensive analysis, we also compute the score approximation error between Da and: (i) the optimal denoisers for the multi-delta distribution DM defined as (5), and (ii) the optimal denoisers for the multivariate Gaussian distribution DG defined as (6). As shown in Figure 2, our analysis reveals three distinct regimes:
High-noise regime [20,80]. In this regime, only coarse image structures are generated (Fig- ure 2(right)). Quantitatively, as shown in Figure 2(1eft), the distilled linear model DD, closely approximates its nonlinear counterpart Do with RMSE below 0.05. Both Gaussian score DG and multi-delta score DM also achieve comparable approximation accuracy.
Low-noise regime [0.002,0.1]. In this regime, only subtle, imperceptible details added to the generated images. Here, both Dt, and DG effectively approximate Da with RMSE below 0.05.
Intermediate-noiee regime 1,20]: This crucial regime, where realistic image content is primarily generated, exhibits significant nonlinearity. While DM exhibits high approximation error due to rapid convergence to training samples- a memorization effect theoretically proved [24], both Dr and DG maintain relatively lower approximation errors.
Qualitatively, as shown in Figure 2(right), despite the relatively high score approximation error in the intermediate noise regime, the images generated with DL closely resemble those generated with Do in terms of the overall image structure and certain amount of fine details. This implies (i) the underlying linear structure within the nonlinear diffusion models plays a pivotal role in their generalization capabilities and (ii) such linear structure is effectively captured by our distilled linear models. In the next section, we will explore this linear structure by examining the linear models Dr..
3.2 Inductive Bias towards Learning the Gaussian Structures
Notably, the Gaussian denoisers DG exhibit behavior strikingly similar to the linear denoisers DL. As illustrated Figure 2leeft), they achieve nearly identical score approximation errors, particularly in the critical intermediate variance region. Furthermore, their sampling trajectories are remarkably similar (Figure 2(right)), producing nearly identical generated images that closely match those from the actual diffusion denoisers (Figure 3). These observations suggest that Dr and DG share
5

---

Figure 3: Images sampled from vari- ous Models. The figure shows the sam- ples generated using different models starting from the same initial noises. Correlation Matrices
U
EDM-ADM (FPHQ)
0- $0.0
42.415
o 21,105
o 9,723
G. 4.005
IWop -0,,,00 1,,000
0-0
0-40.77 010
0.0007
0 0.000
0
20
40 60
80 100
Epochs
Figure 4: Linear model shares similar function mapping with Gaussian model. The left figure shows the difference between the linear weights and the Gaussian weights w.r.t. 100 training epochs of the linear distillation process for the 10 discrete noise levels. The right figure shows the correlation matrices between the first 100 singular vectors the linear weights and Gaussian weights.
similar function mappings across various noise levels, leading to hypothesize that the intrinsic linear structure underlying diffusion models corresponds to the Gaussian structure of the training data data-sseciffiallly, its empirical mean and covariance. We validate this hypothesis by empirically showing that Dj. is close to DG through the following three complementary experiments:
Similarity in weight matrices. As illustrated in Figure 4(left), Wo(e) progressively converge towards UA U throughout the linear distillation process, achieving small normalized MSE (less than 0.2) for most of the noise levels. The less satisfactory convergence behavior at (t) 80.0 is due to inadequate training of the diffusion models at this particular noise level, which is minimally sampled during the training of actual diffusion models (see Appendix G.2 for more details).
Image
Image
Image 3
Image
Image
Similarity in Score functions. Furthermore, Figure 2(left, gray line) demonstrates that DD. and DG maintain small score differences (RMSE less than 0.05) across all noise levels, indicating that these denoisers exhibit similar func- tion mappings throughout the diffusion process.
EDM
Linear Caussian Multi-Dela Detra
Similarity in principal components. As shown in Fig- ure 4(right), for wide noise range (a(t)= E [0.116, 80.0)) the leading singular vectors of the linear weights Wa())
(denoted Linear) align well with U, the singular vectors of the Gaussian weights.* 4 This implics that U, representing the principal components of the training data, is effectively captured by the diffusion models. In the low-noise regime (t))) (0.002, 0.116)), however, Do approximates the identity mapping, leading to ambiguous singular vectors with minimal impact on image generation. Further analy- sis of Dø's 's's behavior in the low-noise regime is provided in Appendices D and F.I.
Since the optimization problem (9) is convex w.r.t. W and bo(e). the optimal solution DL represents the unique optimal linear approximation of Dø. Our analyses demonstrate that this optimal linear approximation closely aligns with DG, leading our central finding: diffusion models in the generalization regime exhibit an inductive bias (which we term as the Gaussian inductive bias) towards learning the Gaussian structure of training data. This manifests in two main ways: (i) In the high-noise variance regime, well-trained diffusion models learn Do that closely approximate the linear Gaussian denoisers De; (ii) As noise variance decreases, although Do diverges from De. DG remains nearly identical to the optimal linear approximation Dr, and images generated by De retain structural similarity those generated by De.
Finally, we emphasize that the Gaussian inductive bias only emerges in the generalization regime. By contrast, in the memorization regime, Figure 5 shows that Dr significantly diverges from DG, and both DG and DL provide considerably poorer approximations of Dø compared to the generalization regime.
'For (t) [0.116, 80.0], the less well recovered singular vectors have singular values close to 0, whereas those corresponding to high singular values well recovered.

---

Approximation (FFHQQ
Donoising Outputs
022
COM Connriiooinn
Uinou Gossoan (Peresnddn
Liaen Calouen (Caness pittcel
0.10
0.02
Woise Variance Dog scale!
Coneniiiaaiio
Memorization
(a)
(b)
Figure 5: Comparison between the diffusion denoisers in memorization and generalization regimes. Fag demonstrates that in the memorization regime (trained on small datasets of size 1094 and 68), Dr. significantly diverges from Da, and both provide substantially poorer approxima- tions of DD compared to the generalization regime (trained on larger datasets of size 35000 and 1094). Figure(b) qualitatively shows that the denoising outputs of Da closely match those of Da only the generalization regime-a similarity that persists even when the denoisers process pure noise inputs.
3.3 Theoretical Analysis
In this section, we demonstrate that imposing linear constraints on diffusion models while minimizing the denoising score matching objective (3) leads the emergence of Gaussian structure.
Theorem 1. Consider a diffusion denoiser parameterized as single-layer linear network, defined as D(#;; (()) W Wott) Do(t). where W oot)) E Iddd is linear weight matrix and boit) Rd is the bias vector. When the data distribution Páana (a:) has finite μ and bounded positive semidefinite covariance Σ. the optimal solution to the score matching objective (3) is exactly the Gaussian denoiser defined in (6):
Da(= = U" - 11) + Me.
with W (o()) U U U and bo(t) - UA) μ.
The detailed proof postponed to Appendix F. This optimal solution corresponds to the classical Wiener filter [31], revealing that diffusion models naturally learn the Gaussian denoisers when constrained to linear architectures. To understand why highly nonlinear diffusion models operate near this linear regime, is helpful model the training data distribution the multi-delta distribution p(x) = + Σi= 8(x y:), where (y1 ... yN } the finite training images. Notice that this formulation better reflects practical scenarios where only finite number of training samples available rather than the ground truth data distribution. Importantly, it is proved optimal denoisers DM in this case is approximately equivalent to DG for high noise variance a (t) and query points far from the finite training data. This equivalence explains the strong similarity between De and DM in the high-noise variance regime, and consequently. why Da and DG exhibit high similarity in this regime- deep networks converge to the optimal denoisers for finite training datasets.
[25] that the
However, this equivalence between DG and DM breaks down at lower (()) values. The denoising outputs of DM are convex combinations of training data points, weighted by softmax function with temperature at) As (t)) decreases, this softmax function increasingly approximates an argmax function, effectively retrieving the training point y; closest to the input Learning this optimal solution requires not only sufficient model capacity to memorize the entire training dataset but also, shown in [32], exponentially large number of training samples. Due to these learning challenges, deep networks instead converge to local minima Dø that, while differing from DM. exhibit better generalization property. Our experiments reveal that these learned Dø share similar function mappings with DG. The precise mechanism driving diffusion models trained with gradient descent towards this particular solution remains open question for future research.
Notably, modeling Paana (ar) as a multi-delta distribution reveals a key insight: while unconstrained optimal denoisers (5) perfectly capture the scores of the empirical distribution, they have gen-
7

---

Memorization
Generalization
Score Flel matrox mation Emar (Gaussian us. EDMI
68
137
1094
B750
35000
79000
330
dalamst
ditasst sig
1004
NN Galuasiini EDM
dataner 1179
350099
425
642
No Venance
(a)
(b)
Figure 6: Diffusion models learn the Gaussian structure when training dataset is large. Models with fixed scale (channel size 128) are trained across various dataset sizes. The left and right figures show the score difference and the generated images respectively. "NN" denotes the nearest neighbor in the training dataset to the images generated by the diffusion models.
cralizability. In contrast, Gaussian denoisers, despite having higher score approximation errors due to the linear constraint, can generate novel images that closely match those produced by the actual diffusion models. This suggests that the generative power of diffusion models stems from the imperfect learning of the score functions of the empirical distribution.
4 Conditions for the Emergence of Gaussian Structures and Generalizability
In Section 3, we demonstrate that diffusion models exhibit an inductive bias towards learning denoisers that are close to the Gaussian denoisers. In this section, we investigate the conditions under which this bias manifests. Our findings reveal that this inductive bias linked to model generalization and is governed by (i) the model capacity relative to the dataset size and (ii) the training duration. For additional results, including experiments on CIFAR-10 dataset, see Appendix F.
4.1 Gaussian Structures Emerge when Model Capacity is Relatively Small First, we find that the Gaussian inductive bias and the generalization of diffusion models are heavily influenced by the relative size of the model capacity compared to the training dataset. In particular, we demonstrate that:
Diffusion models learn the Gaussian structures when the model capacity is relatively small compared to the size of training dataset.
This argument is supported by the following two key observations:
Increasing dataset size prompts the emergence of Gaussian structure at fixed model scale. We train diffusion models using the EDM configuration [4] with fixed channel size of 128 on datasets of varying sizes (68, 137, 1094, 8750, 35000, 70000) until FID convergence. Figure 6(left) demonstrates that the score approximation error between diffusion denoisers Dg and Gaussian denoisers Do decreases as the training dataset size grows, particularly in the crucial intermediate noise variance regime (a(() E [0.116, 20).. This increasing similarity between Dø and DG correlates with transition in the models' behavior: from memorization regime, where generated images are replicas of training samples, a generalization regime, where novel images exhibiting Gaussian structure" are produced, as shown in Figure 6(b). This correlation underscores the critical role of Gaussian structure in the generalization capabilities of diffusion models.
Decreasing model capacity promotes the emergence of Gaussian structure at fixed dataset sizes. Next, investigate the impact of model scale by training diffusion models with varying channel sizes [4, 8, 16, 32, 64, 128), corresponding to 664k, 251k, 992k, 4M, 16M, 64M) parameters, on a fixed training dataset of 1094 images. Figure 7(left) shows that in the intermediate noise variance regime (a(t) E [0.116, 20), the discrepancy between Dø and DG decreases with decreasing model scale, indicating that Gaussian structure emerges in low-capacity models. Figure 7(right) demonstrates that
'We use the term "exhibiting Gaussian structure" to describe images that resemble those generated by Gaussian denoisers.
8

---

Generalization
Memorization
Score fleld Aggrocmation Error (Gazzllnn v.b. ED1O
scare
Scabe 1s
scale
scale 125
Scale 12554
Scile 12004 magao
Scoee 11884 maaees
010
(a)
(b)
Figure 7: Diffusion model learns the Gaussian structure when model scale is small. Models with different scales are trained on fixed training dataset of 1094 images. The left and right figures show the score difference and the generated images respectively.
Generalization
Memorization
Score fiidd Approsimut on Erar (Curzalan va. EDNI
medtc CTEEN
suffieett mainng (EAA Imagedt
64219 spocts images
9123
2/04 oporte (1994 Incrgen
541 spocts (3054 Imaged!
spects 11004 imaggo
Stendin 20000 merpon
Conten Imarct
(a)
(b)
Figure 8: Diffusson model learns the Gaussian structure in early training epochs. Diffusion model with same scale (channel size 128) is trained using 1094 images. The left and right figures shows the score difference and the generated images respectively.
this trend corresponds to transition from data memorization to the generation of images exhibiting Gaussian structure. Here we note that smaller models lead to larger discrepancy between Do and De in the high-noise regime. This phenomenon arises because diffusion models employ a bell-shaped noise sampling distribution that prioritizes intermediate noise levels, resulting in insufficient training at high noise variances, especially when model capacity is limited (see more details in Appendix F.2).
These two experiments collectively suggest that the inductive bias of diffusion models is governed by the relative capacity of the model compared to the training dataset size.
4.2 Overparameterized Models Learn Gaussian Structures before Memorization
In the overparameterized regime, where model capacity significantly exceeds training dataset size, diffusion models eventually memorize the training data when trained to convergence. However, examining the learning progression reveals a key insight:
Diffuusinn models learn the Gaussian structures with generalizability before they memorize.
Figure 8(a) demonstrates that during early training epochs (0-841), Do progressively converge to DG in the intermediate noise variance regime, indicating that the diffusion model is progressively learning the Gaussian structure in the initial stages of training. Notably. By epoch 841, the diffusion model generates images strongly resembling those produced by the Gaussian model, as shown in Figure 8(b). However, continued training beyond this point increases the difference between De and DG the model transitions toward memorization. This observation suggests that early stopping could be an effective strategy for promoting generalization in overparameterizee diffusion models.
Connection between Strong Generalizability and Gaussian Structure
recent study [20] reveals an intriguing "strong generalization" phenomenon: diffusion models trained on large, non-overlapping image datasets generate nearly identical images from the same initial

---

Sounsan Hodkl 1000
Seoong gor or zabiliy under small dotasct (1094)
Farly Stopping Epoch 921
Decrease the Scale to
(b)
[c)
Figure 9: Diffusion models in the strong generalization regime generate similar images as the Gaussian models. Figure(a) Top: Generated images of Gausisan models; Bottom: Generated images of diffusion models, with model scale 128; S1 and S2 each has 35000 non-overlapping images. Figure(b) Top: Generated images of Gausisan model; Bottom: Generated images of diffusion models in the memorization regime, with model scale 128; SI and S2 each has 1094 non-overlon non-overlapping images. Figure(c): Early stopping and reducing model capacity help transition diffusion models from memorization to generalization.
noise. While this phenomenon might be attributed to deep networks' inductive bias towards learning the "true" continuous distribution of photographic images, we propose an alternative explanation: rather than learning the complete distribution, deep networks may capture certain low-dimensional common structural features shared across these datasets and these features can be partially explained by the Gaussian structure.
To validate this hypothesis, we examine two diffusion models with channel size 128, trained on non-overlapping datasets SI and S2 (35000 images each). Figure 9(a) shows that images generated by these models (bottom) closely match those from their corresponding Gaussian models (top), highlighting the Gaussian structure's ' role in strong generalization.
Comparing Figure (())tto)) and (b)(top), observe that DG generates nearly identical images whether the Gaussian structure is calculated on a small dataset (1094 images) or a much larger one (35000 images). This similarity emerges because datasets of the same class can exhibit similar Gaussian structure (empirical covariance) with relatively few samples- Given the Gaussian structure's 's critical role in generalization, small datasets may already contain much of the information needed for generalization, contrasting previous assertions in [20] that strong generalization requires training on datasets of substantial size (more than 10y images). However, smaller datasets increase memorization risk, as shown in Figure 9(). To mitigate this, as discussed in Section 4, we can either reduce model capacity or implement early stopping (Figure 9())). Indeed, models trained on 1094 and 35000 images generate remarkably similar images, though the smaller dataset yields lower perceptual quality. This similarity further demonstrates that small datasets contain substantial generalization-reeewee information closely tied to Gaussian structure. Further discussion on the connections and differences between our work and [20] are detailed in Appendix
hundreds for FFHQ
Discussion
In this study, we empirically demonstrate that diffusion models in the generalization regime have the inductive bias towards learning diffusion denoisers that are close to the corresponding linear Gaussian denoisers. Although real-world image distributions are significantly different from Gaussian, our findings imply that diffusion models have the bias towards learning and utilizing low-dimensional data structures, such as the data covariance, for image generation. However, the underlying mechanism by which the nonlinear diffusion models, trained with gradient descent, exhibit such linearity remains unclear and warrants further investigation.
Moreover, the Gaussian structure only partially explains diffusion models' generalizability. While models exhibit increasing linearity as they transition from memorization to generalization, a substan- tial gap persists between the linear Gaussian denoisers and the actual nonlinear diffusion models, especially in the intermediate noise regime. As a result, images generated by Gaussian denoisers fall short in perceptual quality compared to those generated by the actual diffusion models especially for complex dataset such as CIFAR-10. This disparity highlights the critical role of nonlinearity in high-quality image generation, topic we aim to investigate further in future research.
10

---

Data Availability Statement
The code and instructions for reproducing the experiment results will be made available in the following link: btttps://gtthu /ttp////weeeeee/////aataaaaaaane Diffusion- Requuie -nn teapgg-tthh---eeeee -ane--iiddn---ussiaa------- BeaussannSStuuttie
Acknowledgment
We acknowledge funding support from NSF CAREER CCF-2143904, NSF CCF-2212066, NSF CCF- 2212326, NSF IIS 2312842, NSF IIS 2402950, ONR N00014-22-1-25299 gift grant from KLA, an Amazon AWS AI Award, and MICDE Catalyst Grant. We also acknowledge the computing support from NCSA Delta GPU [33]. We thank Prof. Rongrong Wang (MSU) for fruitful discussions and valuable feedbacks.
References
[1] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages
PMLR, 2015.
[2] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances neural information processing systems, 33:6840 6851, 2020.
[3] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations.
[4] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of ceafuusion-bseed generative models. Advances in Neural Information Processing Systems, 55:26565-266577 2022.
[5] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[6] Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru Zhang. Sampling is easy as learning the score: theory for diffusion models with minimal data assumptions. In International Conference on Learning Representations, 2023.
[7] Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. Transactions Machine Learming Research, 2022.
[8] Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence for score-based generative modeling with polynomial complexity. Advances in Neural Information Processing Systems, 35:22870-22882. 2022.
[9] Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence of ccore-based generative modeling for general data distributions. In International Conference on Algorithmic Learning Theory, pages 946-985, PMLR, 2023.
[10] Yuchen Wu, Yuxin Chen, and Yuting Wei. Stochastic runge-kutte methods: Provable accelera- tion of diffusion models. arXiv preprint arXiv:2410.04760 2024.
[11] Gen Li, Yuting Wei, Yuejie Chi, and Yuxin Chen. A sharp convergence theory for the probability flow odes of diffusion models. arXiv preprint arXiv:2408.00 02320, 2024.
[12] Zhihan Huang, Yuting Wei, and Yuxin Chen. Denoising diffusion probabilistic models are optimally adaptive to unknown low dimensionality. arXiv preprint arXiv:2410.1876 187844 2024.
[13] Minshuo Chen, Kaixuan Huang. Tuo Zhao, and Mengdi Wang. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. International Conference Machine Learning, pages 4672-4712. 12. PMLR, 2023.

---

[14] Kazusato Oko, Shunta Akiyama, and Taiji Suzuki. Diffusion models are minimax optimal distribution estimators. In International Conference on Machine Learning, pages 26517-26582. PMLR, 2023.
[15] Kulin Shah, Sitan Chen, and Adam Klivans. Learning mixtures of gaussians using the ddpm objective. Advances in Neural Information Processing Systems, 16:19666-1944, 2023.
[16] Hugo Cui, Eric Sanden-Eijnden Florent Krzakala, and Lenka Zdeborova. Analysis of learning a flow-based generative model from limited sample complexity. In The Twelfth International Conference Learning Representations, 2023.
[17] Huijie Zhang, Jinfan Zhou, Yifu Lu, Minzhe Guo, Liyue Shen, and Qing Qu The emergence of reproducibility and consistency in diffusion models. In Forty-first International Conference on Machine Learning. 2024.
[18] Peng Wang, Huijie Zhang, Zekai Zhang, Siyi Chen, Yi Ma, and Qing Qu. Diffusion models learn low- din-dimeniional distributions via subspace clustering. arXiv preprint 24X.::4409.02226. 2024.
[19] Sixu Li, Shi Chen, and Qin Li. A good score does not lead to good generative model. arXiv preprint arXiv:2401.04883 044566 2024.
[20] Zahra Kadkhodaie, Florentin Guth, Eero P Simoncelli, and Stéphane Mallat. Generalization in diffusion models arises from geometry-adaptive harmonic representation. In The Twelfth International Conference on Learning Representations, 2023.
[21] Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffu- sion art or digital forgery? investigating data replication diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6048 -058, 2023.
[22] Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Understanding and mitigating copying in diffusion models. Advances in Neural Information Processing Systems, 36:47783-47803, 2023.
[23] TacHo Yoon, Joo Young Choi, Schyun Kwon, and Ernest K Ryu. Diffusion probabilistic models generalize when they fail to memorize. In ICML 2023 Workshop on Structured Probabilistic Inference &) Generative Modeling, 2023.
[24] Xiangming Gu, Chao Du, Tianyu Pang, Chongxuan Li, Min Lin, and Ye Wang. On memorization in diffusion models. arXiv preprint arXiv:2110.0266,, 2023.
[25] Binxu Wang and John Vastola. The hidden linear structure in score-based models and its application. arXiv preprint arXiv:2311. 1..0022 2023.
[26] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IÉEE/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[27] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[28] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1188-8197, 2020.
[29] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 0336,0 2015.
[30] Hyojun Go, Yunsung Lee, Seunghyun Lee, Shinhyeok Oh, Hyeongdon Moon, and Seungtaek Choi. Addressing negative transfer in diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
12

---

[31] Mallat Stéphane. Chapter 11 denoising. In Mallat Stéphane, editor, A Wavelet Tour of Signal Processing (Third Edition), pages 535-610 Academic Press, Boston, third edition edition, 2009.
[32] Chen Zeno, Greg Ongie, Yaniv Blumenfeld, Nir Weinberger, and Daniel Soudry. How do minimum-norm shallow denoisers look in function space? Advances in Neural Information Processing Systems, 36, 2024.
[33] Timothy J Boerner, Stephen Deems, Thomas Furlani, Shelley Knuth, and John Towns. Access: Advancing innovation: Nsf's advanced cyberinfrastructure coordination ecosystem: Services & support. In Practice and Experience in Advanced Research Compuling, pages 173-176. 2023.
[34] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 1188880-8744 2021.
[35] DP Kingma, Adam: method for stochastic optimization. In Int Conf Learn Represent, 2014.
[36] Alfred O. Hero. Statistical methods for signal processing. 2005.
[37] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Antronn Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096-1103, 2008.
[38] Pascal Vincent. connection between score matching and denoising autoencoders. Neural computation, 23(7):1661-1677 2011.
[39] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. Medical image computing and computer-assisted intervention-MICCA. 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. pages 234-241. Springer, 2015.
[40] Sergey loffe. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint FrXXiv15022.01611 2015.
[41] Andrew Maas, Awni Y Hannun, Andrew Y Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. ieml, volume 30, page 3. Atlanta, GA, 2013.
[42] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023.
[43] Alexey Dosovitskiy. An image worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 2020.
[44] Sreyas Mohan, Zahra Kadkhodaie, Eero P Simoncelli, and Carlos Fernandez- bust and interpretable blind image denoising via bias-free convolutional neural networks. In International Conference Learning Representations.
Semandez-Granda. Granda. Ro-
[45] Hila Manor and Tomer Michaeli. On the posterior distribution in denoising: Application certainty quantification. In The Twelfth International Conference on Learning Representations.
[46] Siyi Chen, Huijie Zhang, Minzhe Guo, Yifu Lu, Peng Wang, and Qing Qu Exploring low- dimensional subspaces in diffusion models for controllable image editing. arXiv preprint arXiv:2409.02374, 2024.
13

---

Appendices
Contents
Introduction
2
Preliminary
3
Hidden Linear and Gaussian Structures in Diffusion Models
3.1
Diffusion Models Exhibit Linearity in the Generalization Regime
3.2
Inductive Bias towards Learning the Gaussian Structures
3.3
Theoretical Analysis
4
Conditions for the Emergence of Gaussian Structures and Generalizability
4.1
Gaussian Structures Emerge when Model Capacity is Relatively Small
4.2
Overparameterizee Models Learn Gaussian Structures before Memorization
Connection between Strong Generalizability and Gaussian Structure
Discussion
10
A Measuring the Linearity of Diffusion Denoisers
15
Emerging Linearity of Diffusion Models
16
B.1 Generalization and Memorization Regimes of Diffusion Models
16
B.2 Diffusion Models Exhibit Linearity in the Generalization Regime
16
C Linear Distillation
16
Diffusion Models in Low-noise Regime are Approximately Linear Mapping
18
E Theoretical Analysis
20
E.1 Proof of Theorem
20
E.2 Two Extreme Cases
22
More Discussion on Section
23
E1
Behaviors Low-noise Regime
23
F2 Behaviors in High-noise Regime
23
F33 Similarity between Diffusion Denoiers and Gaussian Denoisers
24
F.4
CIFAR-10 Results
25
Additional Experiment Results
25
G.1
Gaussian Structure Emerges across Various Network Architectures
26
G.2
Gaussian Inductive Bias as General Property of DAEs
26
14

---

G.3 Gaussian Structure Emerges across Various datasets
28
G.4 Strong Generalization CIFAR-10
28
G.5 Measuring Score Approximation Error with NMSE
28
H Discussion on Geometry-Adaptive Harmonic Bases
30
H.1 GAHB only Partially Explain the Strong Generalization :tt.:iiaaaaai...................
30
H.2 GAHB Emerge only in Intermediate-Nooise Regime
I Computing Resources
33
A Measuring the Linearity of Diffusion Denoisers
In this section, we provide a detailed discussion on how to measure the linearity diffusion model. For diffusion denoiser, (æ o()), to be considered approximately linear, it must fulfill the following conditions:
Additivity: The function should satisfy Do(xi x2:0(()) Do(z);a(t) + Do (z(zz;
Homogeneity: It should also adhere to Da(ax;; = aDo(
To jointly assess these properties, we propose to measure the difference between D3 (ax (g(ax) 3az; (t)) and aD (æ);; ((t)) + (zx::(((()) While the linearity score is introduced as the cosine similarity between De (0xr1 + 33z; a (t)) and (eDa((1io(1)) 3D(x (()) in the main text:
KK
Do + Say; (t))
3Do
LS(()
E - x2xp((x0))
(oo + (a2;; (t)) (2 (()]]
(11)
it can also be defined with the normalized mean square difference (NMSE):
[Do + Sarz; or (t))
(Do())
(12)
De + fxz; (tt)ll
where the expectation is approximated with its empirical mean over 100 randomly sampled pairs of (æ1, 22). In the next section, we will demonstrate the linearity score with both metries.
Since the diffusion denoisers are trained solely on inputs æ ~ p(æ; a(t)), their behaviors on out- of-distribution inputs be quite irregular. To produce denoised output with meaningful image structure, critical that the noise component in the input æ matches the correct variance (t) Therefore, our analysis of linearity is restricted to in-distribution inputs æ and 22, which are randomly sampled images with additive Gaussian noises calibrated to noise variance o1) We also need to ensure that the values of a and correct variance for the noise term in the combined input az, + Ba2.. We present the linearity scores, calculated with varying values of a and for diffusion models trained on diverse datasets in Figure 10. These models are trained with the EDM-V VE configuration proposed in [4], which ensures
chosen such that a² B = 1, maintaining the
the resulting models are in the generalization regime. Typically, setting = B = 1//2 yields the lowest linearity score; however, even in this scenario, the cosine similarity remains impressively high, exceeding 0.96. This high value underscores the presence of significant linearity within diffusion denoisers.
We would like to emphasize that for linearity to manifest in diffusion denoisers, crucial that they well-trained, achieving low denoising in Figure 11, the linearity notably reduces a less well trained diffusion model (Baseline-Ve comapred its well-trained counterpart (EDM-VE). Although both models utilize the same 'VE' network architecture Fo(;; (t)) [2], they differ in how the diffusion denoisers are parameterized:
matching loss indicated in (3). As shown
Do(æ: (x: (t)) == (a (t)) + Cou (Fo a (t))),
(13)
15

---

where Caup is the skip connection and Cout modulate the scale of the network output. With carefully tailored Skkp and Cout- the EDM-VE VE configuration achieves a lower score matching loss compared to Baseline- VE, resulting in samples with higher quality as illustrated in Figure (right).
B Emerging Linearity of Diffusion Models
In this section we provide a detailed discussion on the observation that diffusion models exhibit increasing linearity as they transition from memorization to generalization, which is briefly described in Section 3.1.
B.1 Generalization and Memorization Regimes Diffusion Models
As shown in Figure 12, as the training dataset size increases, diffusion models transition from the memorization aggime-where they can only replicate training images-tto the generalization regime, where the they produce high-quality, novel images. To measure the generalization capabilittes of diffusion models, it is crucial to assess their ability to generate images that are not mere replications of the training dataset. This can be quantitatively evaluated by generating a large set of images from the diffusion model and measuring the average difference between these generated images and their nearest neighbors in the training set. Specifically, let {z1, Ø2. 2x represent & randomly sampled images from the diffusion models (we choose h: = 100 in our experiments), and let Y := {y1. V2: .... yN } denote the training dataset consisting of N images. We define the generalization score as follows:
<2 (Ny(a)) 22112
GL Score
(14)
where NNy (xx) represents the nearest neighbor of the sample a; in the training dataset Y, determined by the Euclidean distance on a per-pixel basis. Empirically, GL score exceeding 0.6 indicates that the diffusion models are effectively generalizing beyond the training dataset.
B.2 Diffusion Models Exhibit Linearity in the Generalization Regime
As demonstrated in Figure 13(a) and (d), diffusion models transition from the memorization regime to the generalization regime as the training dataset size increases. Concurrently, as depicted in Fig- ure 13(b), (c), (e) and (f), the corresponding diffusion denoisers exhibit increasingly linearity. This phenomenon persists across diverse datasets datasets including FFHQ [26], AFHQ [28] and LSUN- Churches [29], as well as various model architectures including EDM-VE [3], EDM-VE VP [2] and EDM-ADM [34]. This emerging linearity implies that the hidden linear structure plays an important role in the generalizability of diffusion model.
C Linear Distillation
As discussed in Section 1. we propose study the hidden linearity observed in diffusion denosiers with linear distillation. Specifically, for a given diffusion denoiser Do(x; (t)), we aim to approxi-
Lineerity across Mess (fFiO)
Uneenrty across Moise LSM.
Unear ty across Leve's (APHO-V21
3.00
2200
037
Figure 10: Linearity scores for varying a and J. The diffusion models are trained with the edm-ve configuration [4], which ensures the models are in the generalization regime.
16

---

Generation Trejectories (R(R)e(t)) for Various Modiels
innerrity
COWN
Chitte Sinbartyy
Graakin Docee
Inselie-ve
Figure 11: Linearity scores and sampling trajectory. The left and right figures demonstrate the linearity scores and the sampling trajectories D(æ;; (t)) of actual diffusion model (EDM-VE and Baseline- Bas) Multi Delta model, linear model, and Gaussian model respectively.
FFHQ
70000 images
4375 images
1094 images
0-
20
40
25
50
25
25
50
50
25
50
Genercted
NN
Generatedd
NN
Gemerated
NN
(a)
(b)
(c)
CIFAR-10
50000 images
12500 images
782 images
30
50
20
20
20
20
Generated
NN
Generated
NN
Generated
NN
(d)
Figure 12: Memorization and generalization regimes of diffusion models. Figures(a) (c) show the images generated by diffusion models trained on 70000, 4375, 1094 FFHQ images and their corresponding nearest neighbors in the training dataset respectively. Figures(d) (f) show the images generated by diffusion models trained on 50000, 12500, 782 CIFAR-10 images and their corresponding nearest neighbors in the training dataset respectively. Notice that when the training dataset size is small, diffusion model can only generate images in the training dataset.
mate with linear function (with bias term for more expressibility):
(a (1)) W Wo(1) + bott) R D)(a;
for æ p(=; (t)). Notice that for three dimensional images with size (c, h, w), x E Rd represents their vectorized version, where d = x h. Let
C(W, b) b)
Σ 1=1
Wo - 12< + +e;) + bocojkk - 1 - Do (a; + +
We train 10 independent linear models for each of the selected noise variance level (t) with the procedure summarized in Algorithm
In practice, the gradients on Wot and bocy are obtained through automatic differentiation. Addi- tionally, we employ the Adam optimizer [35] for updates. Additional linear distillation results are provided in Figure 14.
17

---

FFHQ
Gene ulltton Decaset Siz (C3MME, (TRi))
Lipearity acrous
Levels (COMAT (FHO)
Linearity Levets (IDN-NC,
calxct520 7000
cotemtt
Merrater PIEE
cxtawt& 1094
catsont 2e
12003
catauet 2.0
Dinoeet-i 15003
ditrecti00 6753
ditaretaze
5003 3000
1040
1000
1000
2506
(a)
(b)
(c)
CIFAR-10
aloodonn
colourt 20 Sto0e
colmatt De 25000
ratuort
Seekkkk
312
delauet SC002
Aurrse On 25000
dutrect 00 12303
ditasetai 3023
Canmer:
13
(d)
(e)
(f)
Figure 13: Diffusion model exhibit increasing linearity as they transition from memorization to generalization. Figure(a) and (d) demonstrate that for both FFHQ and CIFAR-10 10 datasets, the generalization score increases with the training dataset size, indicating progressive model generaliza- tion. Figure(b), (c), (e), and (f) show that this transition towards generalization is accompanied by increasing denoiser linearity. Specifically, Figure(b) and (e) display linearity scores calculated using cosine similarity (11), while Figure(c) and (f) show scores computed using NMSE (12). Both metrics reveal consistent trends.
D Diffusion Models in Low-noise Regime are Approximately Linear Mapping
should be noted that the low score difference between Dg and Do within the low-noise regime (a<() E [0.002, 0.116)) does not imply the diffusion denoisers capture the Gaussian structure, instead, the similarity arises since both of them are converging to the identity mapping as (t) decreases. As shown in Figure 15, within this regime, the differences between the noisy input and their corresponding denoised outputs (æ (t)) quickly approach O. This indicates that the learned denoisers De progressively converge to the identity function. Additionally, from (6), it evident that the difference between the Gaussian weights and the identity matrix diminishes as o (t) decreases, which explains why De can well approximate Dg in the low noise variance regime.
We hypothesize that Do learns the identity function because of the following two reasons:
(i) within the low-noise regime, since the added noise is negligible compared to the clean image, the identity function already achieves small denoising error, thus serving as a shortcut which is exploited by the deep network.
(ii) As discussed in Appendix A, diffusion models are typically parameterized as follows:
Da(æ; (()) : Cakip (a(()) + Coun (Fo( (())),
where Fo represents the deep network, and (o)()) and coml (t)) are adaptive parameters for the skip connection and output scaling, respectively, which adjust according to the noise variance levels. For canonical works diffusion models [2-4, 34], as (t) approaches zero, Cskp and Cout converge to 1 and 0 respectively. Consequently, at low variance levels, the function forms of diffusion denoisers approximatly identity mapping: (x;; a (l)
This convergence to identity mapping has several implications. First, the weights Wo(() of the distilled linear models Dt, approach the identity matrix at low variances, leading ambiguous
18

---

Algorithm Linear Distillation
Require:
(i) the targeted diffusion denoiser Po(;a((1))
(ii) weights Wo and biases (o()). both initialized to zero,
(ii) gradient step size 77,
(iv) number of training iterations K,
(V) training batch size n.
(vi) image dataset S.
for to K do
Randomly sample batch of training images (2,, L2,.... In from S. Randomly sample batch of noises (Ei, E2.....CC Cn from ((,
Update Wotg) a(t) and (o) with gradient descent:
Watg(k) = W (a( - 1} VV (k- (KW,,
bo(t) [k: (o()) (k
v b b)
end for
Return (o))) (K). (1)
Assymetric Measure
1.0
80.0
[[ Wiall_/WWaoo ,
62.113
21-21.100
9.123
4.566
0.6
1.502
1,502
0.47
o- 0.117
0.4
Seates
0.2
10
15
10.0
100
500
Sinpller onl
Epochs/5
(a)
(b)
FFHQ
Churchhs
2e
25 insorct Covoden
(c)
(d)
Figure 14: Additional linear distillation results. Figure(a) demonstrates the gradual symmetrization of linear weights during the distillation process. Figure(b) shows that at convergence, the singular values of the linear weights closely match those the Gaussian weights. Figure(c) and Figure(d) display the leading singular vectors of both linear and Gaussian weights at ((t) 4 for FFHQ and -UUN-hhrchess datasets, respectively, revealing a strong correlation.
singular vectors. This explains the poor recovery of singular vectors for (t) [0.002, 0.116) shown in Figure 4. Second, the presence of the bias term in (8) makes it challenging for our linear model to leam the identity function, resulting in large emoor (t) = 0.002 shown in Figure 4(a).
Finally, from (4), we observe that when Dg acts as an identity mapping, 4;+1 remains unchanged from 2;, This implies that sampling steps in low-variance regions minimally affect the generated image content, as confirmed in Figure 2, where image content shows negligible variation during these steps.
19

---

Normelized MSE between oo(x:(()) and x
Similaray between (2(x; (tt) and
27161
JTFC)
(d)
Figure 15: Difference between Da(a; (()) and x: for various noise variance levels. (a) (a) and (c) show the differences between Pa(2;a((1) and across ((l) (0.002,80], measured by normalized MSE and cosine similarity, respectively. Figures(b) and (d) provide zoomed-in views of (a) and (c). The diffusion models were trained on the FFHQ dataset. Notice that the difference between Do (z; a (f)) and quickly converges to near zero in the low noise variance regime. The trend is consistent for various model architectures.
E Theoretical Analysis
E.1 Proof of Theorem 1
In this section, we give the proof Theorem (Section 3.3). Our theorem based on the following two assumptions:
Assumption 1. Suppose that the diffusion denoisers are parameterized as single-layer linear net- works, defined as D(a;; Wott) bolty where Wo(t) E RRddd is the linear weight and bi(t) E TRd the bias.
Assumption 2. The data distribution Paata has finite mean μ and bounded positive semidefinite covariance Σ
Theorem 1. Under Assumption I and Assumplion the optimal solution to the denoising score matching objective (3) is exactly the Gaussian denoiser: 'D (()) p. + U U (æc p.), where Σ U represents the SVD of the covariance matrix, with singular values ((k=1,.... and Ba(t) = diaglx. (oyz) Furthermore, this optimal solution can be obtained via gradient descent with proper learning rale.
To prove Theorem 1, we first show that the Gaussian denoiser the optimal solution the denoising score matching objective under the linear network constraint. Then we will show that such optimal solution can be obtained via gradient descent with proper learning rate.
The Global Optimal Solution. Under the constraint that the diffusion denoiser is restricted to a single-layer linear network with bias:
D(z;o(t)) = Wa(o) + bo()))
(15)
We get the following optimizaiton problem from Equation (3):
W b' arg min L(W, b. ot)) == Em L N(... W (a E) + b b-
(16)
W.b
where omit the footnote (t) in Wait and batt) for simplicity. Since expectation preserves convexity, the optimization problem Equation (16) is a convex optimization problem. To find the global optimum, we first eliminate b by requiring the partial derivative VoL(W, b; (t)) to be Since
VoL(W,b (())
= 2 Ex--ya (00) - -Ix We
(17)
= =2 x Exerypan (W I)x + b)
(18)
((W 1)p2 b),
(19)
we have
b" = (I - W H.
(20)
20

---

Utilizing the expression for b, we get the following equivalent form of the optimization problem:
W arg min ((W;o(()) Be Peoooo μ E) (x --)1 (21)
The derivative V Vw[(W;o(()) is:
Vw((W;o(t)) = * E. (W (2 - - (( (a- u)(2 - μe +
- x)
(22)
2 * Ex (W -I) (= 1n² (1) +o())
(23)
2 W(E o) 2 E.
(24)
Suppose E UAUT is the SVD of the empirical covariance matrix, with singular values A(k==....... np. by setting Vw ((W;o(()) to O we get the optimal solution:
W = UAA + -
(25)
U UOO
(26)
where {o = x and X Ali, Substitute W back to Equation (20), we have:
b' -I- U U Jr.
(27)
Notice that the expression for W and b* is exactly the Gaussian denoiser. Next, we will show this optimal solution can be achieved with gradient descent.
Gradient Descent Recovers the Optimal Solution. Consider minimizing the population loss:
L(W, N,,,b;oo ot)) == Egepan (e- + e)
x: x=12.
(28)
Define W W b]. 0 and ē
then can rewrite Equation (28) as:
0
((Wo((t) E Eawpan Be- W (Ō + + x
(29)
We can compute the gradient in terms of W as:
VL(W) = * Ex,e [W @ + (z + +2) x( + ))²)
(30)
2 * Ex,e (2x" + -ē + tēx" + Eēe - -x:² - xē²
(31)
((()2Iaxce
0axi
Since (E) = 0 and Ee(EET) =
we have:
01xd
[
0ax xxx" -
VL(W)= = 2 E.x (xx
(32)
0 0xxd
[(x
B(x)
Since E(xλT)
we have:
)(x²))
Ex (xx T +
(t)
44]
VC(W) 2W
- Ex(xxT 14)
(33)
n²
1
With learning rate 77, write the update rule
W = (()( =(((1)=((((11 - 2n
(xxxx
(t)22 11"
1
+ 2m [xe( 14]
(34)
( - 22) + 2n (x
14].
(35)
(xxxx" + =
11)
where we define A = 2m
for simplicity. By recursively expanding
11"
1
the expression for W. we have:
W W(0 + 2n (x (xTx)
μ4] Σ A
(36)
7=0
21

---

Notice that there exists 1,, such that every eigen value of A is smaller than and greater than O. In this case, A A 0 as t + oc. Similarly. by the property of matrix geometric series, we have Σi= A + (I ---))-
A)- - Therefore we have:
En (xx")
+ It
1
W - (5a""x" x))
ml]
(37)
u
B P 12
( x:)
(38)
where define B (xxx ) + (t)) I for simplicity. By the
Coorison-Woodboee
formula, we have:
B E
11
[ -ppp)) B-!
-(B nh )
)
(39)
(1
p² μ1)
B-
(1- 1 B 11)
Therefore, we have:
W +
[xxx" upp )==
1y B-1 1- tp
Ex 1xx [ uppe") -1
T-
B 1
(40)
from which we have
1ep1" B
W - Ep [x 1
[xxx
(B )
(41)
1 1 B B-1p 1
- - Ee xx
[T - -111 "-1 1 n¹ B B-I μ
1
(42)
Since En [=:=. E. [(x 1)((22 11)²1 ptt have:
W = Σ(Σ + + "41" ( pup1") 11 B
(43)
11" B B- μ
Applying Sherman-Morrison Formula, we have:
B - --p²)-1- B-1 + B- up T B-
(44)
1 μ² B-1
therefore
mm T B- n²= B-1
mm B up T B-1
phi T B 4 B- 4
p - mu1 )
(45)
11 B B-1 m
- urº B-1
T B p
(ul T B-1
1111 B B-1)
(46)
n B-
B-1p
0
(47)
which implies
W + E(E + It)
(48)
= UU
(49)
Similarly, we have:
+ (I- U /
(50)
Therefore, gradient descent with properly chosen learning rate y recovers the Gaussian Denoisers when time goes infinity.
E.2 Two Extreme Cases
Our empirical results indicate that the best linear approximation of Da is approximately equivalent to De. According the orthogonality principle [36]. this requires Da satisfy:
Exr (z) (o=
+ et)())) (x - (z) + e 1 ~ O.
(51)
Notice that (51) does not hold for general denoisers. Two extreme cases for this to hold are:
22

---

. Case 1: Do(æ + (t)( = a for Va ~ Pdata: ~ ((
Case 2: Do(x + e; (()) ~ (a +E; for Vo Pdata; 6 ~ ((0,,
Case I requires (o ( to be the oracle denoiser that perfectly recover the ground truth clean image, which never happens in practice except when (t) becomes extremely small. Instead, our empirical results suggest diffusion models in the generalization regime bias towards Case 2, where deep networks learn Dg that approximate (not equal) to Dg. This is evidenced in Figure 5(b), where diffusion models trained on larger datasets (35000 and 7000 images) produce denoising outputs similar to Dg- Notice that this similarity holds even when the denoisers take pure Gaussian noise as input. The exact mechanism driving diffusion models trained with gradient descent towards this particular solution remains an open question and we leave it as future work.
F More Discussion on Section 4
While in Section 4 we mainly focus on the discussion of the behavior of diffusion denoisers in the intermediate- regime, in this section study the denoiser dynamics in both low and high-noise regime. We also provide additional experiment results on CIFAR-10 10 dataset.
F1 Behaviors in Low-noise Regime
We visualize the score differences between DG and De in low-noise regime in Figure 16. The left figure demonstrates that when the dataset size becomes smaller than a certain threshold, the score difference
0 remains persistently non-zero. Moreover, the right figure shows that this difference depends solely on dataset size rather than model capacity. This phenomenon arises from two key factors: (i) Dy converges to the identity mapping at low noise levels, independent of training dataset size and model capacity, and (ii) Da approximates the identity mapping at low noise levels only when the empirical covariance matrix is full-rank, can be seen from (6).
Since the rank of the covariance matrix upper-bounded by the training dataset size, DG differs from the identity mapping when the dataset size is smaller than the data dimension. This creates persistent gap between De and Dg. with smaller datasets leading to lower rank and consequently larger score differences. These observations align with our discussion in Appendix D.
F.2 Behaviors in High-noise Regime
As shown in Figure 7(a), while a decreased model scale pushes Do in the intermediate noise region towards DG. their differences enlarges in the high noise variance regime. This phenomenon arises be- cause diffusson models employ a bell-shaped noise sampling distribution that prioritizes intermediate noise levels, resulting in insufficient training at high noise variances. A shown in Figure 17, for high (t), Da converge to DG when trained with sufficient model capacity (Figure 17(b) and training time (Figure 17(c)). This behavior is consistent irrespective of the training dataset sizes (Figure 17(a)). Convergence in the high-noise variance regime is less crucial in practice, since diffusion steps in
Score Rele Asproximation Eror (Caussian vs. EDM)
Score Rell Aporoximation Enro (Couss ior v.s. EDM
dataget dbc 66
Scale 120 (70000 imagesi
1.1
dataset die 1054
daluset sicu 12f
Scale 11098 images)
Stale 128 (LOM4 imagaal
dataset dize 8750
Scale 32 (3034 images)
ditaset diz 22000
Scale 13024 Images)
135
dotaset dize 70000
3.7
Stalle (11914 images)
Scale- (L994 imagesi
A.3
8.0
6.3
500
Figure 16: Score differences for low-noise variances. The left and right figures are the zoomed-in views of Figure 6(a) and Figure 7(a) respectively. Notice that when the dataset size smaller than the dimension of the image, the score differences are always non-zero at a = 0.
23

---

Denoising Outputs for (t) = 60 (PSNR -29.5 dB)
Effect of Datseet Size
Effect Model Scale (1084 training images)
Putr (1294)
DugCorrO 29000
Burr otro CoOES
Iepoaeei 841
Pegroete 113
Puly 0100 d9
Auosettll nobe
Pubrett 15300
Rebr m00) 22300
(b)
Effect of Training Epochs (1 training images)
ApOsecon
Svor #00 (o94
4udr-a00) peaad
Rutnaato poacos
Rudrabio
Audkectt
(a)
(c)
Do converge to DG with no overfitting for high noise variances. Figure(a) shows the denoising outputs of DM. De and well-trained (trained with sufficient model capacity till convergence) Do. Notice that at high noise variance, the three different denoisers are approximately equivalent despite the training dataset size. Figure(b) shows the denoising outputs of Do with different model scales trained until convergence. Notice that Dg converges to De only when the model capacity is large enough. Figure(c) shows the denoising outputs of Da with sufficient large model capacity different training epochs. Notice that Da converges to Dg only when the training duration is long enough.
Figure 17:
Denoising Outputs (()) =
Cam mage Now -VL 907)) 2-264
(a)
Varying Madel Scales
Varying Training Epochs
Pytcetto)
0411
Ogkk dr72
exx0 (s-11
ote)
Aadeectioss st
Aphoneri
Avorotto mon
Puosacti s410
Avcretti (172)
Generalization
Memorization
Generallzation
Memorization
(b)
(c)
Figure 18: Denoising outputs of DG and De at 4. Figure(a) shows the clean image (from test set), random noise e and the resulting noisy image y. Figure(b) compares denoising outputs of Da across different channel sizes 8, 64, 128] with those of De. Figure(c) shows the evolution of Do outputs at training epochs [187, 841, 9173, 64210) alongside DG outputs. All models are trained fixed dataset of 1,094 images.
this regime contribute substantially less than those in the intermediate-non -oise variance regime- phenomenon we analyze further in Appendix G.5,
F.3 Similarity between Diffusion Denoiers and Gaussian Denoisers
In Section 4, we demonstrate that the Gaussian inductive bias is most prominent in models with limited capacity and during early training stages, finding qualitatively validated in Figure 18. Specifically, Figure 18(b) shows that larger models (channel sizes 128 and 64) tend to memorize.

---

Memorization
Generalization
Score Field Approcmmaiion Error (Gaussian vs, EDM)
deaeet
196
782
3125
12580
25300
53000
0135
12320
de.ewe ste 25320
53030
NN Conostin
(a)
(b)
Figure 19: Large dataset size prompts the Gaussian structure. Models with the same scale (channel size 64) are trained on CIFAR-I0 datasets with varying sizes. Figure(a) shows that larger dataset size leads increased similarity between De and Do. resulting in structurally similar generated images as shown in Figure(b).
Generalization
Memorization
Score Rees Approcmation Enor iGausdin vs. EDNI
Scale (762 Imapes)
Scale (782 imppes]
Scaie mmger]
ineges
Noae Namencelo
(a)
(b)
Figure 20: Smaller model scale prompts the Gaussian structure. Models with varying scales are trained on fixed CIFAR-10 10 datasets with 782 images. Figure(a) shows that smaller model scale leads to increased similarity between DG and Da in the intermediate noise regime (a E [0.1, 10],, resulting in structurally similar generated images as shown in figure(b). However, smaller scale leads to larger score differences in high-noise regime due to insufficient training from limited model capacity.
directly retrieving training data as denoising outputs. In contrast, smaller models (channel sizes 8 and 4) exhibit behavior similar De, producing comparable denoising outputs. Similarly, Figure 18(c) reveals that during early training epochs (0-841), Do outputs progressively align with those of DG- However, extended training beyond this point leads memorization.
F.4 CIFAR-10 Results
The effects of model capacity and training duration on the Gaussian inductive bias, as demonstrated in Figures to 21, extend to the CIFAR-10 dataset. These results confirm our findings from Section the Gaussian inductive bias is most prominent when model scale and training duration are limited.
G Additional Experiment Results
While in the main text we mainly demonstrate our findings using EDM-VE diffusion models trained on FFHQ, in this section we show our results are robust and extend to various model architectures and datasets. Furthermore, we demonstrate that the Gaussian inductive bias not unique to diffusion models, but it is fundamental property of denoising autoencoders [37]. Lastly, we verify that our
25

---

Generalization
Memorization
Score Field Apprssmatiinn Emor (Saurtiin SEMO)
Pactt de
raccho Not
TEN
723 esedhs (182 imagest
2500 upc (782 magess
10020 epocis imapee
No Varlance
(a)
(b)
Figure 21: Diffusion model learns the Gaussian structure in early training epochs. Models with the same scale (channel size 128) are trained on fixed CIFAR-10 datasets with 782 images. Figure(a) shows that the similarity between Dg and Do progressively increases during early training epochs (0-921) in the intermediate noise regime (a E [0.1, 10)), resulting in structurally similar generated images as shown figure(b). However, continue training beyond this point results in diverged De and Do. resulting in memorization.
EDMVV (!FHQ)
EDM-DDM (FTHO)
EDMVV (FFHO)
1.0
1.0
800
1.0
800
2- 42433
GG412
4G.455
No cormalzzdd MSE
2-
21,00
21,00
0.6
1.0
0.6
0.1
0.
0-0
0.4
2.00
0.2
0.2
0.2
0.0
2.5
5.0
7.5
10.0
0
20
40
60
80
100
0
20
40
60
80
100
Epocha/5
Epachs
Epachs
(a)
[b)
(c)
Figure 22: Linear model shares similar function mapping with Gaussian model. The figures demonstrate the evolution of normalized MSE between the linear weights Dt and the Gaussian weights DG w.r.t. linear distillation training epochs. Figures(a), (b) and (c) correspond to diffusion models trained on FFHQ, with EDM-VE, EDM-ADM and EDM-VP network architectures specified in [4] respectively.
conclusions remain consistent when using alternative metrics such as NMSE instead of the RMSE used in the main text.
G.1 Gaussian Structure Emerges across Various Network Architectures
We first demonstrate that diffusion models capture the Gaussian structure of the training dataset, irrespective of the deep network architectures used. As shown in Figure 22 (a), (b), and (c), although the actual diffusion models, Dø. are parameterized with different architectures, for all noise variances except (t) E {0.002, 80.0]. their corresponding linear models, DL. consistently converge towards the common Gaussian models, DG. determined by the training dataset. Qualitatively, as depicted in Figure 23, despite variations in network architectures, diffusion models generate nearly identical images, matching those generated from the Gaussian models.
G.2 Gaussian Inductive Bias as a General Property of DAES
In previous sections, we explored the properties of diffusion models by interpreting them as collections of deep denoisers, which are equivalent to the denoising autoencoders (DAEs) [37] trained on various
noise variances by minimizing the denoising score matching objective (3). Although diffusion models and DAEs are equivalent in the sense that both of them are trying to learn the score function of the
26

---

image!
image 2
image 3
image
image
image
Gaussian
EDMAVE
EDM-VP
EDM-ADM
Figure 23: Images sampled from various model.The figure shows the sampled images from diffusion models with different network architectures and those from their corresponding Gaussian models.
noise-mollified data distribution (38), the training objective of diffusion models is more complex [4]:
min Err,,,r,,, Cour (o)² Fa + 6, 0
Coar (a) xx - Cskip Cskip (x) + +)) l1a)
(52)
linear combination
where a ~ Pdata e ~ (NO, (t) and a ~ Pirii- Notice that the training objective of diffusion models has few distinct characteristics:
Diffusion models use single deep network Fo to perform denoising score matching across all noise variances while DAEs are typically trained separately for each noise level.
Diffusion models not trained uniformly across all noise variances. Instead, during training the probability of sampling a given noise level a is controlled by a predefined distribution Puain and the loss is weighted by X(o).
Diffusion models often utilize special parameterizations (13), Therefore, the deep network Fg is trained to predict a linear combination of the clean image æ and the noise e, whereas DAEs typically predict the clean image directly.
Given these differences, we investigate whether the Gaussian inductive bias unique to diffusion models general characteristic of DAEs. To this end, we train separate DAEs (deep denoisers) using the vanilla denoising score matching objective (3) on each the 10 discrete noise variances specified by the EDM schedule [80.0, 42.415, 21.108, 9.723, 4.06, 1.501, 0.469, 0. 11,,, 0.020, 0.002), and compare the score differences between them and the corresponding Gaussian denoisers Dç. We special parameterization that D, Fo; that is, the deep network directly predicts the clean image. Furthermore, the DAEs for each noise variance are trained till convergence, ensuring all noise levels are trained sufficiently. We consider the following architectural choices:
DAE-NCSN: In this setting, the network Fo uses the NCSN architecture [3], the same as that used in the EDM-VE diffusion model.
DAE-Skip: In this setting, Fo is a U-Net [39] consisting of convolutional layers, batch normalization [40], leaky ReLU activation [41] and convolutional skip connections. We refer to this network as "Skip-Net". Compared to NCSN, which adapts the state of the art architecture designs, Skip-Net is deliberately constructed to be simple possible to test how architectural complexity affects the Gaussian inductive bias.
DAE-DiT: this setting, Fo a Diffusion Transformer (DiT) introduced [42]. Vision Transformers are known to lack inductive biases such as locality and translation equivariance that are inherent to convolutional models [43]. Here we are interested if this affects the Gaussian inductive bias.
27

---

Score Field Approximation Error
1322
E312
DAE Linear Caussinn
0.4
DAE Skip Gaussian
Gaussian
DAE-NO AIE-NCSW vs Gaussian
EUM Eokanoiinn
041
0865
8.4
800
Kose Varlance (ag scak)
(a)
(b)
Figure 24: Comparison between DAES and diffusion models, Figure(a) compares the score field approximation error between Gaussian models and both (i) diffusion models (HDM vs. Gaussian) and (ii) DAES with varying architectures. Fbg illustrates the generation trajectories of different models initialized from the same noise input.
DAE-Linear, In this setting we set Fo to be a linear model with bias term as in (8). According Theorem these models should converge to Gaussian denoisers.
The quantitative results shown in Figure 24(a). First, the DAE-linear models well approximate De across all 10 discrete steps (RMSE smaller than 0.04), consistent with Theorem 1. Second, despite the differences between diffusion models (EDM) and DAEs, they achieve similar score approximation errors relative to Dc for most noise variances, meaning that they can be similarly approximated by Dg. However, diffusion models exhibit significantly larger deviations from Da at higher noise variances (a E 442.415, 80.0)) since they utilize a bell-shaped noise sampling distribution Prain that emphasizes training on intermediate noise levels, leading to under-traiee meddi-n high noise variances. Lastly, the DAEs with different architectures achieve comparable score approximation errors, and both DAEs and diffusion models generate images matching those from the Gaussian model, as shown in Figure 24(b). These findings demonstrate that the Gaussian inductive bias is not unique to diffusion models or specific architectures but is fundamental property of DAEs.
G.3 Gaussian Structure Emerges across Various datasets
As illustrated in Figure 25, for diffusion models trained on the CIFAR-IO, AFHQ and LSUN-Churches datasets that are in the generalization regime, their generated samples match those produced by the corresponding Gaussian models. Additionally, their linear approximations, DL, obtained through linear distillation, align closely with the Gaussian models, DG, resulting nearly identical generated images. These findings confirm that the Gaussian structure is prevalent across various datasets.
G.4 Strong Generalization on CIFAR-10
Figure 26 demonstrates the strong generalization effect CIFAR-10. Similar to the observations in Section 5, reducing model capacity or early stopping the training process prompts the Gaussian inductive bias, leading generalization.
G.5 Measuring Score Approximation Error with NMSE
While Section 3. we define the score field approximation error between denoisers D1 and Dz with RMSE ( (10)), this error can also be quantified using NMSE:
| +c) - D2( +c)l2
Score-Difference() == Be--puale)).-WW0000001
(53)
(D) +
As shown Figure 27, while the trend in intermediate-noise and low-noise regimes remains unchanged, NMSE amplifies differences in the high-noise variance regime compared to RMSE. This amplified score difference between DG and Dg does not contradict our main finding that diffusion models in the generalization regime exhibit an inductive bias towards learning denoisers
28

---

LSUN-Churches LSUN- Churches
Final Generated Samoles
Generation Trajectories (PCr,atC))) Various Models
innaee
image
irtege
imtage
Emaege
40/155
21.101
9,772
4.00
1.50
0:17
6.17
6.22
0LEOP
EDM
Canaaan NAR Dena
Uhev
(8]
(b)
AFHQ
Final Generated Samples
Goneration Trajectorios (P(r,)tE)) for Various Models
image
image
Image
image image
80.9
42415
23.18
723
690
1.522
647
0.177
0.2
0.002
Geroola sue
Uneer
(c)
CIFAR-
Generation Trajectories (P()))0(E)) Various Models
Generated Samples
Image:1
imaga
imagie image image
ONE
DAb
Geooder
(e)
Figure 25: Final generated images and sampling trajectories for various models. Figures(a), (c) and (e) demonstrate the images generated using different models starting from the same noises for LSUN-Churches, AFHQ and CIFAR-10 respectively. Figures(b), (d) and (f) demonstrate the corresponding sampling trajectories.
approximately equivalent to DG the high-noise variance regime. As discussed in Section 3.2 and appendices F.2 and G.2, this large score difference stems from inadequate training in this regime.
Figure 27 (Gaussian vs. DAE) demonstrates that when DAEs are sufficiently trained at specific noise variances, they still converge to De. Importantly, the insufficient training in the high-noise variance regime minimally affects final generation quality. Figure 25(f) shows that while the diffusion model (EDM) produces noisy trajectories early timesteps (a E {80.0, 42.415)), these artifacts quickly disappear in later stages, indicating that the Gaussian inductive bias most influential the intermediate-noise variance regime.
Notably, even when D, are inadequately trained in the high-noise variance regime, they remain approximable by linear functions, though these functions no longer match DG-

---

ppeneral
Early Stopping Epoch
Nox costoginn DINAAE Codd52
Our Seying CYCME cosiicads
Caty Enoppis
Deprease the Model Scale to
(b)
16)
Figure 26: Strong generalization on CIFAR-10 dataset. Figure(a) Top: Generated images of Gausisan models; Bottom: Generated images of diffusion models, with model scale 64; S1 and S2 each has 25000 non-overlapping images. Figure(b) Top: Generated images of Gausisan model; Bottom: Generated images of diffusion models in the memorization regime, with model scale 128; $1 and S2 each has 782 non-overlapping images. Figure(c): Early stopping and reducing model capacity help transition diffusion models from memorization to generalization.
Score Field Approdmaatio Emar ICIFAR-10;
Score Field Approximatiion Error (CIFAR-10)
Optimal
Optimal EDM
Gaussian
Gaussiinn
Lineer EDM
Linear vs. EDM
Nomoalizd IISE
4.28
Gaussian vs. DAE
Gaussian vs. DAE
0.15
0.30
Notze Variance
holee Variznc2
(a)
(b)
Score Field Approximation in Error (FTHO]
Score Fleid Approsimation frror (PTHO)
0.3
Gaussian EDM
Gaussian vs. EDM
Lineer
Multi-Delta vs. EDM
Muti-Delta vs. EDM
Lineer Gaussian
Linear Gaussian
Normatued MSE
Geussien vs. LAE
C4
Geussien vs. UAE
1.28
0.25
0..2
Nolse Variance
Varionce
(c)
(d)
Figure 27: Comparison between RMSE and NMSE score differences. Figures(a) and (c) show the score field approximation errors measured with RMSE loss while figures(b) and (d) show these errors measured using NMSE loss. Compared to RMSE, the NMSE metric highlight the score differences in the high-noise regime, where diffusion models receive the least training.
H Diseussion on Geometry-Adaptin Harmonic Bases
H.1 GAHB only Partially Explain the Strong Generalization
Recent work [20] observes that diffusion models trained on sufficiently large non-overlapping datasets (of the same class) generate nearly identical images. They explain this "strong generalization" phenomenon by analyzing bias-frec deep diffusion denoisers with piccewise linear input-output
30

---

mappings:
D(zz = (D(a
(54)
E (((
(55)
where As (ate; ug (are), and ny (24) represent the input-dependent singular values, left and right singular vectors of the network Jacobian (z( (t)). Under this framework, strong generalization occurs when two denoisers D1 and D2 have similar Jacobians: (2r s VD; a (t)). The authors conjecture this similarity arises from networks" inductive bias towards learning certain optimal (a (t)) that has sparse singular values and the singular vectors of which are the geometry-adaptive harmonic bases nGa--i)--arr-opiimal denoising bases that adapt to input et.
While [20] provides valuable insights, their bras-free assumption does not reflect real-world diffusion models, which inherently contain bias terms. For feed forward ReLU networks, the denoisers are piecewise affine:
D( (t)) (D(a at)) + ba.
(56)
where bæ, is the network bias that depends on both network parameterization and the noisy input 2e [44]. Here, similar Jacobians alone cannot explain strong generalization, as networks may differ significantly in bz, For more complex network architectures where even piecewise affinity fails, we consider the local linear expansion of (( (f)):
D(x, + A (t)) VD(z;; (t)) D(x,;a(t)),
(57)
which approximately holds for small perturbation Az. Thus, although VD(x;; characterizes (t))'s local behavior around x,, does not provide sufficient information on the global properties.
Our work instead examines global behavior, demonstrating that D(æ;; (t)) is close to Da(a;; (t))- optimal linear denoiser under the Gaussian data assumption. This implies that strong generalization partially stems from networks learning similar Gaussian structures across non-overlapping datasets of the same class. Since our linear model captures global properties but not local characteristics, it complements the local analysis in [20].
H.2
GAHB Emerge only in Intermediate-Noise Regime
For completeness, we study the evolution of the Jacobian matrix oD(z;;((t) across various noise levels oft). The results are presented in Figures 28 and 29, which reveal three distinct regimes:
High-noise regime [ [10,80] In this regime, the leading singular vectors of the Jacobian matrix VD(x;; (()) well align with those of the Gaussian weights (the leading principal components of the training dataset), consistent with our finding that diffusion denoisers approximate linear Gaussian denoisers in this regime. Notice that DAES trained sufficiently on separate noise levels (Figure 29) show stronger alignment compared to vanilla diffusion models (Figure 28), which suffer from insufficient training at high noise levels.
Intermediate- regime [0]1 In this regime, GAHB emerge singular vectors of VD(x;; (t)) diverge from the principal components, becoming increasingly adaptive to the geometry of input image.
Low-noise regime /0.002,0.1). In this regime, the leading singular vectors of VD(a,;; a (()) show no clear patterns, consistent with our observation that diffusion denoisers approach the identical mapping, which has unconstrained singular vectors.
Notice that the leading singular vectors of (D(x;; (t)) are the input directions that lead to the maximum variation in denoised outputs, thus revealing meaningful information on the local properties of D(x;; (t)) at Iy As demonstrated in Figure 30, perturbing input x; along these vectors at difference noise regimes leads distinct effects on the final generated images: (i) in the high-noise regime where the leading singular vectors align with the principal components of the training dataset,
'We only care about leading singular vectors since the Jacobians in this regime are highly Jow-rank. The less well aligned singular vectors have singular values near O.
31

---

Generation Trajectories DXK, a(t)
$6.23
cen
2.1
DARZ
0221
5.00
(a]
Correlation U(x,)
Various
0.43
22:704
30422
11:04
100g
CO0
KSCI
C80
(b)
U22
U,
(c)
(kg))) across Various (())
(d)
(x)) across Various ((t)
A00
14.07
1501
EXO
(e)
(x,) acroes Various (1))
4343 22.54 18:32
4902
25
30
25 20
(f)
Figure 28: Evolution of VD(x;; across varying noise levels. Figure(a) shows the generation trajectory. Figure(b) shows the correlation matrix between Jacobian singular vectors (x;,) and training dataset principal components U. Notice that the leading singular vectors of U Uxx)) and U well align in early timesteps but diverge in later timesteps. Figure(c) shows the first three principal components of the training dataset while figures(d-f) show the evolution of Jacobian's 's first three singular vectors across noise levels. These singular vectors initially match the principal components but progressively adapt to input image geometry, before losing distinct patterns at very low noise levels. While we present only left singular vectors, right singular vectors exhibit nearly identical behavior and yield equivalent results.
perturbing æ, along these directions leads to canonical changes such image class, (ii) in the intermediate-noise regime where the GAHB emerge, perturbing æ; along the leading singular vectors modify image details such colors while preserving overall image structure and (iii) in the low-noise regime where the leading singular vectors have no significant pattern, perturbing æ; along these directions yield no meaningful semantic changes.
These results collectively demonstrate that the singular vectors of the network Jacobian VD( (()) have distinct properties at different noise regimes, with GAHB emerging specifically in the intermedi- ate regime. This characterization significant implications for uncertainty quantification [45] and image editing [46].
32

---

Correlation U*U())
across a(C)
(a)
Wy (t,) pcross Various caom
0.43
3,00
6211
OC2
(b)
My (K2) acrosa Varioua d(t)
BITY
(c)
M2 (42) across Varioua e(t)
0.455
25 204
1500
1/
6101
022
9642
(d)
Figure 29: Evolution of ( across varying noise levels for DAEs. We repeat the experiments in Figure 28 on DAEs that sufficiently trained on each discrete noise levels. Notice that with sufficient training, the Jacobian singular vectors (xx,) show better alignment with principal components in early timesteps.
(t)) 22.79
1.999
(t) 0.002
Ry (x²)
(X)
(z,)
(x)
(x,)
(X)
(a)
(b)
(c)
Figure 30: Effects of perturbing x; along Jacobian singular vectors. Figure(a)-(c) demonstrate the effects of perturbing input along the first singular vector of the Jacobian matrix (æ, + Au, (zz)) on the final generated images. Perturbing æ; in high-noise regime (Figure (a)) leads canonical image changes while perturbation in intermediate-n noise regime (Figure (b)) leads to change in details but the overall image structure preserved. At very low noise variances, perturbation has no significant effect (Figure (c)). Similar effects are observed in concurrent work [46].
Computing Resources
All the diffusion models in the experiments trained A100 GPUs provided by NCSA Delta GPU [33].
33

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable "[No] ", it is perfectly acceptable answer "[No] provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" "we were unable find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased binary way. we acknowledge that the answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendíx. If you answer [Yes] a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We mention the main contribution in both the abstract and introduction section, listed in the "Contributions".
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the "Discussion" section. section..
34

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
We provide the assumption and proof in Section 3.3 and Appendix E.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We include the experiment details in Section 3, Section 4 and Section 5. We will also release code upon publication.
Guidelines:
The answer NA means that the paper does not include experiments.
35

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. https/ccppblcccgiideeeeee Ctteeee//ooooSeeeeeeeeeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
36

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We include the experiment details in Section 3, Section 4 and Section 5.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: For various plots we specify that the numbers are calculated over 100 random samples. However, since the standard deviations are quite small, the error bar is unnecessary.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We include the computating resources in Appendix I.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
37

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https://neuripp: http:://aourrs..ccpplll//////ai
ines?
Answer: [Yes]
Justification: Our research doesn't 't have potential harm cause by research process and negative socictal impact.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is empirical and theoretical work on the inductive bias of the diffusion models. has more scientific contributions rather than societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly. harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system lears from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Our work an empirical and theoretical work on how the diffusion model learns the data distribution. It doesn't have high risk of misuse.
38

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
We use the public released dataset and models, which properly credited the license.
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, saperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work is an empirical and theoretical work on how the diffusion model learns the data distribution. We don't release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
39

---

Answer:
[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
40


Reviews:
[{"review_id": "Nqto5OGBq3", "reviewer": "Reviewer_6yNJ", "summary": "The paper investigates the generalization properties of diffusion models by examining the learned score functions, which are denoisers trained on various noise levels. It shows that nonlinear diffusion denoisers exhibit linearity when the model can generalize, leading to the idea of distilling these nonlinear mappings into linear models. The findings suggest that these linear denoisers closely align with optimal denoisers for a multivariate Gaussian distribution, indicating an inductive bias towards capturing Gaussian structures in the training data. This bias becomes more pronounced with smaller model sizes and might provide insights into the strong generalizability seen in real-world diffusion models.", "questions": "1. Can the authors explain how from their set of experiments we can conclude that the emergence of Gaussian structure leads to strong generalizability? \n1. The paper shows how the linear (distilled) model resembles the optimal denoiser for a multivariate Gaussian distribution. Would the same phenomena be observed if we would linearize the trained model instead of distilling it? Have the authors conducted any experiments in this direction?\n1. Could the authors provide a more precise definition of inductive bias in their work? Specifically, is it a property of the architecture, the input-output mapping parameterized by the network, the forward diffusion process, or the denoising score matching loss?\n1. The authors refer to edm-ve, edm-vp, and edm-adm as different \"*model architectures*\". Could they clarify this terminology? My understanding is that all these models use a U-net based architecture to parameterize the score function, and the difference lies in the parameterization of the forward process SDE.\n1. Do VE and VP stand for variance exploding and variance preserving? How do they differ from ADM?\n1. Can the authors provide experiments that measure generalization compared to memorization in their setup?\n1. In Figure 1, the authors show that when the model is memorizing (yellow dashed curve), the score function is not approximately linear (at least for some noise levels). However, there is no evidence reported that the yellow curve corresponds to memorization compared to the other curves. Could the authors provide empirical evidence to support this?\n1. Can the authors provide details of their experimental setup, including training and architectures?\n1. Could the authors better discuss the comparison with [1]  in more detail? They claim, \"*We propose an alternative explanation: the networks capture certain common structural features inherent across the non-overlapping datasets*\". How is this claim in contrast with the hypothesis that due to the inductive bias of the neural network, the score and therefore the density are learned? I fail to understand how learning \"common structural features\" differs from learning the density. Could the authors elaborate on this?", "limitations": "- The description of the experimental setup lacks essential details, making it difficult to replicate the results.\n- The experiments exclusively focus on the RMSE as the performance metric across various noise scales. Alternative performance measures are neither reported nor discussed.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-motivated and written. \n2.  The Gaussian structure hypothesis for generalization in diffusion models is interesting.", "weaknesses": "- Although the paper's focus is on generalization in diffusion models, it lacks any quantitative measure of generalization within the experiments presented.\n- Similarly, no quantitative measure of memorization is reported. \n- The work primarily investigates the linearity of the learned score functions across various noise levels. However, the connection between these experiments and the necessity of linearity for generalization in diffusion models is unclear and not well substantiated.\n- The term \"inductive bias\" is frequently used throughout the paper, but it is never clearly defined. It remains ambiguous whether this bias pertains to the model architecture, the parameterization of the forward diffusion process, or the denoising score matching loss.\n- The paper lacks detailed information on the experimental setup, including training procedures and the hyperparameters of the architectures used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Although the paper's focus is on generalization in diffusion models, it lacks any quantitative measure of generalization within the experiments presented.**\n\nA1: The memorization and generalization can be clearly observed in our experimental results by comparing the generated image with the nearest neighbor (NN) in the training dataset (see Figure 6 to Figure 9). To measure the memorization and generalization quantitatively,  we can empirically define the generalization score as follows\n\\begin{align}\n    \\text{GL Score} := \\frac{1}{k}\\sum_{i=1}^k\\frac{||x_i-\\text{NN}_Y(x_i)||_2}{||x_i||_2}\n    %\\mathcal{L}(\\mb x_k,\\text{NN}_Y(\\mb x_i)),\n\\end{align}\nwhere $[x_1, x_2, ..., x_k]$ are sampled images from the diffusion models, and $Y:=[y_1, y_2, ..., y_N]$ denote the training dataset. We used this metric to assess generalization versus memorization, with the results presented in Figure 1 of our global response. As shown in Figure 1 (d) to (g), diffusion models start to exhibit memorization behavior when the number of training images becomes smaller than 8750 (GL Score around 0.5). Therefore we choose 0.6 as the threshold to distinguish between generalization and memorization. As shown in Figure 1(a) and (b), the diffusion denosiers exhibit increasing linearity as the diffusion models shift from memorization to generalization.\n\n**Q2. The work primarily investigates the linearity of the learned score functions across various noise levels. However, the connection between these experiments and the necessity of linearity for generalization in diffusion models is unclear and not well substantiated.**\n\nA2. Please refer to A.3. of our global response.\n\n**Q3. The term \"inductive bias\" is frequently used throughout the paper, but it is never clearly defined.**\n\nA3: Here we give a precise definition of the inductive bias in our work: “When the model size is relatively small compared to the training dataset size, training diffusion models with the score matching loss (Equation (3)) results in diffusion denoisers that behave similarly to the linear Gaussian denoisers (though with certain amount of difference especially in the intermediate noise variance region). Furthermore, even when the model is overparameterized, such similarity emerges in the early training epochs.\" Our finding is consistent across various architectures including VE, VP and ADM (see Figure (13)).\n\nThough in the paper, we mainly test on EDM configuration, which specifies specially designed forward process, and denosier parameterization (Equation (9)), we expect the inductive bias will manifest for other forward process and parameterization as well since recent work [18] shows that diffusion models trained with different forward process and parameterization generate almost identical images starting from the same random seed, which suggests that even in those cases, the diffusion denoiser share similar function mappings. \n\n**Q4. The paper lacks detailed information on the experimental setup, including training procedures and the hyperparameters of the architectures used.**\n\nA4: In the revision, we will include more experimental details (e.g., the model architectures, hyperparameters, training procedures) in our paper for reproducibility of our results. \n\n**Q5. Can the authors explain how from their set of experiments we can conclude that the emergence of Gaussian structure leads to strong generalizability?**\n\nA5: Firstly, based upon our studies in Section 3 and 4 we find strong correlation between generalization and Gaussian structures. Furthermore, in section 5 we show that decreasing model scale and early stopping are able to prompts strong generalization. Remember those two activations prompts the emergence of the Gaussian structure.\n\n**Q6. Could the authors better discuss the comparison with [15] in more detail? I fail to understand how learning \"common structural features\" differs from learning the density.**\n\nA6: Our work offers complementary explanations based on the common Gaussian structures from non-overlapping datasets and the inductive bias towards these structures. In contrast, [15] hypothesized that the models learn the same underlying distribution. Notice that  learning the ground truth distribution is only a sufficient condition not a necessity for the strong generalization. Just because two models produce the same images doesn’t mean they learn the underlying distribution. It only implies certain common structures of the two datasets are captured but such a common structure might not be the optimal. Our experiment shows that such a common structure is highly related to the Gaussian structure.\n\n**Q7. The paper shows how the linear (distilled) model resembles the optimal denoiser for a multivariate Gaussian distribution. Would the same phenomena be observed if we would linearize the trained model instead of distilling it?**\n\nA7: Yes, our experiment results show that directly training a linear diffusion model based on the denoising score matching loss results in the Gaussian denoisers.  However, this is not the main point of the paper since what is interesting is that diffusion models behave closely to the Gaussian denoisers even without this explicit linear constriant.\n\n**Q8. The authors refer to edm-ve, edm-vp, and edm-adm as different \"model architectures\". Could they clarify this terminology?**\n\nA8: The EDM paper [4] proposes a novel diffusion configuration (a specially designed forward process, time schedule, network parameterization). This configuration can be adapted to various network architectures. EDM-VE [3], EDM-VP [2] and EDM_ADM [23] are all trained with the EDM configuration but with different network architectures. Here VE stands for the architecture proposed in the paper [3], VP stands for the architecture proposed in the paper [2] and ADM corresponds to the architecture proposed in [23]. We refer the reviewer to the EDM [4] paper for more detail.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mShrkIhKOV", "reviewer": "Reviewer_VCtP", "summary": "The work analyzes the behavior of a diffusion-based generative model from the perspective of ``Gaussian structures''. In particular, it checks the linearity of scores at single-time steps and compares scores against a Gaussian model.  The analysis shows that the Gaussian structure plays a main role in image generation.", "questions": "Have you considered using standard Gaussian distribution as the convergent distribution of the diffusion process?", "limitations": "The scope of the study could be wider.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The work has compared diffusion-based generative models against a Gaussian model. It has discovered convincing evidence that linear score plays an important role in image generation. \n\nIt has done extensive experiments with different configurations. By controlling irrelevant variables, the study highlights factors that affect generalization. In particular, it discovers that the linear score mimicking a Gaussian model plays an important role.", "weaknesses": "The study is only limited to a few generative models and a face dataset. The observation might be different on different datasets. The face dataset has a more obvious Gaussian distribution. For example, the mean of all faces is a face, and correlations between pixels are relatively consistent. Therefore, a Gaussian distribution is somewhat reasonable for such a dataset. It is unknown whether a Gaussian distribution is still a reasonable choice for a set of diverse images. In such a case, does the diffusion model still need to mimic the Gaussian model? \n\n\nThe current analysis still could not explain all observations. For example, the behavior in noise levels between 20 and 80 is not well explained in Figure 6. These observations might be worth more in-depth discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. The study is only limited to a few generative models and a face dataset. The observation might be different on different datasets. The face dataset has a more obvious Gaussian distribution. For example, the mean of all faces is a face, and correlations between pixels are relatively consistent. Therefore, a Gaussian distribution is somewhat reasonable for such a dataset. It is unknown whether a Gaussian distribution is still a reasonable choice for a set of diverse images. In such a case, does the diffusion model still need to mimic the Gaussian model?**\n\nA1: Although the Gaussian structure is the most evident on face images, the same linear phenomenon persists in other datasets as well. As shown in Figure 15 in the appendix, for more complicated dataset such as AFHQ and LSUN-Churches, there is still a clear similarity between the generated images from the diffusion models and those from the Gaussian model. Furthermore, we conducted extra experiments on Cifar-10 during the rebuttal, with results in Figure 2 (b) of our newly uploaded PDF. We can observe that the generated images for Cifar-10 dataset exhibits similarity with Gaussian models.\n\n**Q2. The current analysis still could not explain all observations. For example, the behavior in noise levels between 20 and 80 is not well explained in Figure 6. These observations might be worth more in-depth discussion.**\n\nA2: For the noise variance region between 20 and 80, the behavior of the learned diffusion denoisers are relatively stable. From Figure 6, we see that the difference between the diffusion denoisers and the Gaussian denoisers in the high noise region does not change as much compared to the intermediate noise variance region when varying the dataset size. Furthermore, from Figure 7 we see that the difference between the diffusion denoisers and the Gaussian denoisers seems to decrease consistently as the model scale increases. These observations indicate that with sufficient training, the diffusion denoisers for the high noise variance region converge to the Gaussian denoisers without overfitting. This observation might be attributed to the fact that for the high noise variance region, the Gaussian denoisers are approximately the global minimizer of the denoising score matching objective under finite training dataset assumption (see Equation (5)); this is proved in [19]. Therefore, we should expect that with a large enough model capacity, deep networks converge to such a minimizer. Since it is a global minimizer, we observe less overfitting. \n\nWe will include this discussion in our revised paper and make those points clear. Nevertheless, this region doesn’t influence the final generated images as much as the intermediate noise regions, which is why  we mainly focus on the discussion of the intermediate noise region.\n\n**Q3. Have you considered using standard Gaussian distribution as the convergent distribution of the diffusion process?**\n\nA3: We start the sampling process from a standard Gaussian distribution with std = 80. Since the image lies in the range of [-1,1], the noise magnitude is much larger, which means the convergent distribution is roughly a standard Gaussian distribution with std equals to 80.\n\n**We thank the review for the insightful comments, please let us know if you have further questions.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rGWvzE8maP", "reviewer": "Reviewer_N1bC", "summary": "This paper compares diffusion models trained on natural image datasets with their Gaussian approximations. It evaluates the quality of this approximation in both the memorization and generalization regime by studying the influence of the training set size, model capacity, and training time.", "questions": "- Given the points above, the surprisingly high cosine similarities reported in Figure 1 look suspicious to me. If denoisers were truly linear, then they would generate Gaussian images, and we know that they learn much better models than this. As noted by the authors in appendix A, the linearity measure $\\texttt{LS}(t)$ evaluates the denoiser out-of-distribution since linear combinations of natural images are not natural images (they are superpositions of two unrelated images). Doesn't this mean that these cosine similarities are not really meaningful and deceiving? Another factor that could influence these high values is that they are computed for the denoiser function $\\mathcal D_\\theta(x)$, as opposed to the denoiser residual $x - \\mathcal D_\\theta(x)$ (or equivalently the score). As noted in Appendix A, the denoisers are very close to the identity function at low noise and therefore are expected to be close to linear. It would be more interesting to evaluate the linearity of the score, which I suspect is much smaller (I predict it would only decrease as the noise level decreases).", "limitations": "As stated above, the main limitation of this paper is that it studies the Gaussian case which is well-understood. As a result, a large fraction of its results are straightforward (and some even long known), and it misses the point which resides entirely in the non-Gaussian/linear structure learned by these models. I encourage the authors to study the literature in denoising (e.g., Chapter 11 of the aforementioned book by Mallat) and image modeling (for a brief introductory review, see, e.g., Simoncelli, Eero P. _Statistical modeling of photographic images._ Chapter 4.7 of Handbook of Video and Image Processing 9 (2005).)\n\nHowever, these topics are unfortunately not well-known by the broader machine learning community. I thus think that this paper could provide a service to the community by reviewing what is known about the Gaussian case (after major changes, e.g., to remove the linear denoiser and only mention the Gaussian denoiser). It also makes several novel and interesting observations that are valuable to the community (see strengths).\n\nIn the current state of the paper, I recommend to reject, and encourage the authors to resubmit their work after another iteration.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper makes several novel and interesting empirical observations:\n- according to the authors' linearity measure, denoisers are surprisingly well-aproximated by linear functions (but more on this below),\n- reducing model capacity or training time can bring diffusion models into the generalization regime even with very small training sets, though at the expense of image quality.\n\nIt is also clearly written, and studies the very important problem of characterizing the inductive biases of diffusion models/network denoisers which allow them to generalize.", "weaknesses": "I have two main issues with this paper:\n- First, a large part of its content (most of section 3) is rather obvious for people with a signal/image processing background. Indeed, Theorem 1 is well known, usually under the name of \"Wiener filter\" (which is simply a consequence of linear regression): see, e.g., Theorem 11.3 from _A Wavelet Tour of Signal Processing, Third Edition: The Sparse Way_, Stephane Mallat, Elsevier/Academic Press, 2009. It immediately implies that differences between the \"linear\" and \"Gaussian\" denoisers can only come from suboptimality of the network denoiser (or optimization errors of the linear denoiser), and are thus irrelevant for this study. A second observation that the authors did not make is that any linear denoiser leads to Gaussian generated images (as is evident from equation (4) since $x_{i+1}$ is then a linear function of $x_i$ and $x_0$ is Gaussian). Further, the Gaussian denoiser produces samples from the Gaussian distribution with the same mean and covariance as the training set, which is thus a very simple (but crude) model for which samples can easily be produced without any diffusion procedure.\n- The main claim of the paper then boils down to whether Gaussian distributions are a good approximation of natural image distributions (and in particular, a better approximation than the empirical distribution of the training set). These two examples are interesting to contrast, as they lie on the two extremes of the quality-diversity tradeoff (Gaussian distributions maximize entropy but have very low quality, while the a sum of delta functions at the training images has perfect quality but essentially no diversity). However, Gaussian models are extremely crude and have been extensively studied in the past, and we therefore have few things left to learn from them. It is not surprising that good diffusion models are capable of capturing the covariance of their training data, but this is well understood. As the authors acknowledge in the discussion, what we do not understand is the rest (higher-order moments captured with non-linear denoisers). The visual similarity between faces and Gaussian images with matched first and second moments results from the fact that the faces are centered, so that they are well-approximated with a PCA, also known as \"eigenfaces\": _Low-dimensional procedure for the characterization of human faces._ L. Sirovich; M. Kirby (1987), Journal of the Optical Society of America A. 4 (3): 519–524.). This similarity breaks down for more complex datasets such as LSUN-churches which are more translation-invariant, leading to stationary Gaussian samples which look like textures. I therefore disagree with the main claim that \"the image contents generated with such Gaussian denoisers [...] closely resemble those produced by well-trained diffusion models.\" For the same reason, the inductive biases of network denoisers cannot be reduced to the Gaussian structure only, as they are capable of learning much more complex structures despite the curse of dimensionality. \n\nSpecific points:\n- The sentence at lines 164-166 is wrong, as shown in Figure 1: the denoiser is linear for very small $\\sigma$ (close to identity) and very large $\\sigma$ (close to its Gaussian approximation as noted in [19]), but less linear in the middle range.\n- Root mean square error in equation (8) should have the square root outside the expected value.\n- The RMS plots in Figures 2, 6, 7, and 8 could be improved in several ways. First, they are difficult to interpret due to the lack of a reference point. I suggest normalizing them by the expected RMS norm of the network denoiser, so as to have values in $[0,1]$. Second, I suggest to use a continuous colormap when a single parameter is being varied (such as the size of the training set or number of epochs), in order to facilitate visual analysis. Third, why are the numerical values for these varied parameters chosen arbitrarily and irregularly? It would be easier to see trends with regularly (e.g., linearly or logarithmically) spaced parameters, chosen at round-ish numbers.\n- The authors should mention in section 3.3 that the Gaussian denoisers compared in Figure 6 use the empirical covariances of the smaller training sets, this is implicit and confused me for a while.\n- Two observations relative to memorization and generalization made by the authors are straightforward to me. First, any generative model which perfectly optimizes its training loss will only reproduce its training set, so it is clear that trained network denoisers are suboptimal for their training loss, _and in a specific way_ that enables them to generalize \"correctly\". Second, a Gaussian model will strongly generalize in the sense of [15] when the two training sets have similar first and second-order statistics, which then happens quite rapidly (and can be studied precisely with random matrix theory analyses).\n\nMinor: \n- Typo line 286: \"diffsuion\"\n- Typo caption of Figure 9: \"Gausisan\" (twice)\n- Typo line 316: \"covaraince\"\n- Equations (9) and the first of Appendix C should are missing $c_{\\rm out}(\\sigma(t))$\n- line 504: footnote -> subscript", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. First, most of section 3 is rather obvious for people with a signal/image processing background; e.g., Theorem 1 is well known.**\n\nWe sincerely appreciate the review for pointing us to the Wiener filter. We agree that Theorem 1 is well studied (we will add citations on the Wiener filter to acknowledge this), but we believe many findings of our work are not obvious as the reviewer suggested. Please see A.2. in the global response.\n\n**Q2: A second observation that the authors did not make is that any linear denoiser leads to Gaussian generated images. Further, the Gaussian denoiser produces samples from the Gaussian distribution with the same mean and covariance as the training set, which is thus a very simple model for which samples can easily be produced without any diffusion procedure.**\n\nFirstly, in this work, we are interested in the mapping from the noise space z to the image space x. An arbitrary linear denoiser does not produce similar images as the actual diffusion models therefore it is worth no interest. Furthermore, we cannot compare diffusion mapping with the Gaussian mapping if we directly sample from the Gaussian distribution without the reverse diffusion procedure. \n\n**Q3. It is not surprising that good diffusion models are capable of capturing the covariance of their training data**\n\nAgain, this point is not well understood as well. First of all, we not only show that diffusion models capture the covariance of their training data but more importantly, we show that the obtained diffusion denoisers share similar function mapping with the Gaussian denoisers. The first observation doesn’t explicitly imply the second. Since even the optimal denoisers under the multi-delta distribution assumption captures the covariances, after all, they are able to reproduce the whole training dataset. On the contrary, it is not well understood how gradient descent on deep networks arrive at a denosier that is close to the Gaussian denoisers. To understand this requires rigorous analysis on the gradient dynamics of the deep network and to our best knowledge, no such analysis exists in the current literature.  Furthermore, in the current literature on diffusion models, this problem is circumvented by either assuming an infinite number of training data or directly restricting  the architectures of the diffusion models.\n\n**Q4. The similarity breaks down for more complex datasets. The inductive biases of network denoisers cannot be reduced to the Gaussian structure only, as they are capable of learning much more complex structures despite the curse of dimensionality.**\n\nOur results in Figure 15 in the appendices and Figure 2 (b) of our newly uploaded PDF demonstrate that this similarity also persists in more complex datasets (e.g., LSUN-Churches, AFHQ datasets and Cifar-10 dataset). This similarity indicates that the first and second order statistics of the finite training dataset is heavily utilized by diffusion models. Otherwise, we should not be able to observe the similarity. However, we do agree the generation power cannot be reduced to the Gaussian structure only, but it indeed plays a critical role.\n\n**Q5. The surprisingly high cosine similarities reported in Figure 1 look suspicious to me.**\n\nWe conduct experiments to measure the linearity of the score functions as the reviewer suggested. The results are shown in Figure 3 of our newly uploaded PDF, where we see that measuring linearity of score is not very meaningful. This is because the noise magnitude can be much higher compare to the denoising output (in the range of [-1,1]), therefore subtracting the denoising outputs from the noisy image does not change the noisy image significantly except in the low noise variance regime. For this reason, we always see a high linearity of the scores for most of the noise variances. In the figure, denoised_img_1 and dneoised_img_2 correspond to $D(x_1|\\sigma_t)$ and $D(x_2|\\sigma_t)$, denoised_img_1+denoised_img_2 corresponds to $\\frac{1}{\\sqrt{2}} D(x_1|\\sigma_t)+\\\\frac{1}{\\sqrt{2}}D(x_1|\\sigma_t)$ and denoised_image_additive corresponds to $D(\\frac{1}{\\sqrt{2}}x_1|\\sigma_t)+D(\\frac{1}{\\sqrt{2}}x_2|\\sigma_t)$. $x_1$ and $x_2$ are two randomly sampled noisy image.\n\n**Q5. Two observations relative to memorization and generalization made by the authors are straightforward to me...**\n\nFirst of all, this point is not well perceived by the current literature. Current theorems normally assume you can directly sample from the ground truth training data distribution instead of just a finite number of training data. In this case, exactly minimizing the score denoising matching loss indeed results in the ground truth score function. However, less work focuses on the finite training setting, in which minimizing the score denoising matching loss results in overfitting. As you mentioned, the trained network is suboptimal in a specific way and enables them to generalize correctly. Our results indicate that this specific way is to be close to the Gaussian denoiser and why this happens is not well understood.\n\nSecondly, our discussion on strong generalizability focuses on the actual diffusion model rather than the Gaussian model. Yes, a Gaussian model will strongly generalize in the sense of [15] when the two training sets have similar first and second-order statistics. We use this fact to explain why diffusion models exhibit strong generalizability. Since the diffusion models learn similar function mappings as their corresponding Gaussian denoisers in the generalization regime, strong generalization of the Gaussian models leads to the strong generalization of the actual diffusion models. This is supported by the fact that we can direct the model towards strong generalization by either early stopping or decreasing the model scale. Remember these two actions prompts the emergence of the Gaussian structure, which further highlights the necessity of the Gaussian structure in diffusion model’s generalizability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
To all Reviewers:

We thank all the reviewers for their insightful and constructive comments. Most of the reviewers find our work well written (6yNJ, VCtP, N1bC), well-motivated (6yNJ, N1bC),  interesting and novel (6yNJ, N1bC, VCtP) with convincing evidence (VCtP).

We summarize our main findings as follows:

**Inductive bias towards Gaussian structures.** Here, through linear distillation, we demonstrated that when the model generalizes, practically learned diffusion models exhibit an inductive bias toward Gaussian/linear structures without any explicit linear constraints. In contrast, the linear/Gaussian structures are less prominent in the memorization regime. Furthermore, we study how training dataset size, model capacity and training time effect such inductive bias. These insights are neither obvious nor well understood in previous works.

In this response, we address the common concerns and questions raised by most reviewers and we will include the revisions into our manuscript. Unless otherwise specified, all reference numbers refer to those in the bibliography of the NeurIPS submission. 

**Q1. Does the phenomenon appear on more complex datasets?**

A1. Although the Gaussian structure is the most evident on face images, the same linear phenomenon persists in other datasets as well. As shown in Figure 15 in the appendix, for more complicated dataset such as AFHQ and LSUN-Churches and Cifar-10 (Figure 2 (b) of our newly uploaded PDF), there is still a clear similarity between the generated images from the diffusion models and those from the Gaussian model. This similarity implies that though natural images distributions are far from the Gaussian distribution, the Gaussian structure of the finite datasets play an important role in the generation ability of diffusion models in the generalization regime.

**Q2. The main limitation of this paper is that it studies the Gaussian case which is well-understood/most of section 3 is rather obvious for people with a signal/image processing background.**

A2. Our work is not studying the Gaussian case itself, but to study the relationship between generalization and the Gaussian structures. We demonstrate that in the generalization regime, diffusion denoisers share high similarity with the Gaussian denoisers. This result suggests that the Gaussian structure (first and second order statistics) of the finite training dataset plays an vital important role in the generation process. More specifically, notice in section 3, our linear models are not trained using the score matching loss, rather, it is trained to regress the input and output pairs of the actual diffusion networks (we made this clear in line 150-155 of the main text and also in the appendix B). For this reason, the high similarity between the linear model and the Gaussian model is not trivial. It requires the deep network to be regularized. In this case, this similarity can be attributed to the fact that for diffusion models in the generalization regime, the corresponding diffusion denoisers share similar function mappings as the Gaussain denoisers (though with a certain level of difference especially in the intermediate region). We refer the review to the Figure 2 (a) of our newly uploaded PDF, where we directly visualize the denoising outputs of the Gaussian model and the linear model. Notice that when the diffusion model generalizes (model scale = 4 or 8), the denoising outputs of the Gaussian model and the diffusion model are quite similar. This similarity in denoising outputs holds even if you directly input the noise to the denoisers. However, for diffusion models that memorize models (scale = 64 or 128), the similarity between Gaussian denoisers and the actual diffusion models breaks. In this case, as shown in Figure 1(c) of our newly uploaded PDF, the linear model trained with the linear distillation can no longer approximate the diffusion models as well as in the case of generalization and the similarity between the linear model and the Gaussian model breaks as well. The reason behind the similarity between the diffusion denoisers in the generalization regime and the Gaussian denoisers is not well understood.

**Q3. The work primarily investigates the linearity of the learned score functions across various noise levels. However, the connection between these experiments and the necessity of linearity for generalization in diffusion models is unclear and not well substantiated.**

A3. Our work has shown strong connections between the necessity of linearity and generalization in the following aspects.

*In Section 3*, in the generalization regime we demonstrate that diffusion models produce images that are highly similar to those generated by the distilled linear denoisers (as well as the Gaussian denoisers) when sampled starting from the same random noise. This means that in the generalization regime, the learned diffusion denoisers share similar function mappings as the Gaussian denoisers. On the contrast, this similarity between the linear model and Gaussian model breaks when diffusion model memorizes, meaning that the Gaussian linear structure is ubiquitous in the generalization regime. This shows the necessity of linearity and generalization.

*In Section 4*, we demonstrate that as the diffusion models transition from memorization to generalization, the similarity between the diffusion denoisers and the Gaussian denoisers increases as well. This strong correlation implies the necessity of the Gaussian structure in the diffusion model’s generalizability.

*In Section 5*, As shown in Figure 9 of our paper, we can bring memorized models into strong generalization by either decreasing the model capacity or applying early stopping. Remember from section 4, these two actions basically prompts the diffusion denoisers to get closer to the Gaussian denoisers. This once again highlights the necessity of Gaussian structure in the generalization of diffusion models.

Meta-review: The work analyzes the behavior of a diffusion-based generative model from the perspective of hidden Gaussian structures. The work makes a novel finding, which would be useful in understanding diffusion generative models. There is still a room for futher explaining this finding.
Decision: Accept (poster)