Towards a Mechanistic Explanation of Diffusion Model Generalization
Authors: Matthew Niedoba
Berend Zwartsenberg
Kevin Patrick Murphy
Frank Wood
Venue: ICML 2025 spotlightposter (2025)
Keywords: Diffusion Models
Generalization
Diffusion
Generative Modelling
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: Empirical patch-based denoisers approximate the generalization behaviour of image diffusion models

Abstract: We propose a simple, training-free mechanism which explains the generalization behaviour of diffusion models. By comparing pre-trained diffusion models to their theoretically optimal empirical counterparts, we identify a shared local inductive bias across a variety of network architectures. From this observation, we hypothesize that network denoisers generalize through localized denoising operations, as these operations approximate the training objective well over much of the training distribution. To validate our hypothesis, we introduce novel denoising algorithms which aggregate local empirical denoisers to replicate network behaviour. Comparing these algorithms to network denoisers across forward and reverse diffusion processes, our approach exhibits consistent visual similarity to neural network outputs, with lower mean squared error than previously proposed methods.

Towards a Mechanistic Explanation of Diffusion Model Generalization
Matthew Niedoba !2
Berend Zwartsenberg'
Kevin Murphy'
Frank Wood 123
Abstract
Empiiricnl
NCSN
DDPM
PSPc.Plec
We propose simple, raining-firee mechanism which explains the generalization behaviour of diffusion models. By comparing pre-trained dif- fusion models to their thcoretically optimal em- pirical counterparts, we identify a shared local inductive bias across a variety of network architec- tures. From this observation, network denoisers generalize through localized denoising operations, as these operations approxi- mate the training objective well over much of the training distribution. To validate our hypothesis, we introduce novel denoising algorithms which aggregate local empirical denoisers to replicate network behaviour. Comparing these algorithms to network denoisers across forward and reverse diffusion processes, our approach exhibits consis- tent visual similarity to neural network outputs, with lower mean squared error than previously proposed methods.
hypothesize that
1=33
L=06
1. Introduction
Diffusion models "Sohl-Dickstenn et al., 2015; Ho et al., 2020; Song et al., 2021) have become the de facto stan- dard for modelling image (Rombach et al., 2022) and video (Harvey et al., 2022) data due their high sample quality and generalization abilities. When properly tuned, diffusion models produce samples that distributionally similar to their training set, but are not exact copies of training data (Zhang et al., 2023).
Figure I. Denoiser outputs given shared reverse process noisy in- puts from CIFAR-10. Column 1: Optimal empirical denoiser, i.c. what a "perfect" neural network denoiser would output ap- propriately parameterized and trained achieve minimal loss the diffusion denoising loss (Equation (5)). Columns 2-4: Outputs from various denoising neural networks. For 3.3 all net- works deviate from the optimal denoiser in similar ways. Column 5: Our learning-free patch posterior composite denoiser pro- duces qualitatively similar outputs to the neural network denoisers, suggesting that neural networks may generalize part patch denoising and composition.
This behaviour remarkable, as linear increases in data di- mensionality require exponentially more training samples to model the data density (Bellman, 1966). Avoiding this curse of dimensionality requires inductive biases that enable gen- eralization from sparse examples (Goyal & Bengio, 2022). Recently, research has found that diffusion models produce
near-identical samples despite differences in architecture, optimization, or diffusion hyperparameters (Zhang al., 2023), suggesting the presence of inductive biases common to all image diffusion models.
Uniieersity of British Columbia IIverted AI Alberta Ma- chine Intelligence Institute. Correspondence to: Matthew Niedoba amniedoba@cs.ube...
Diffusion models generate samples through an iterative pro- cess which noise is progressively removed via repeated denoising operations. Crucially, at each step of this process, there exists an optimal denoiser function that can be ex-
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Towards Mechanistic Explanation of Diffusion Model Generalization
pressed as a simple weighted average of the training dataset (Vincent, 2011; Karras et al., 2022). However, using this op- timal function in the denoising sampling procedure results in exact replication of the training dataset without gener- alization (Gu et al., 2023). The remarkable generalization capabilities of diffusion models therefore emerge from re- peated neural network approximation errors relative to this optimal denoiser. These deviations from optimality, accu- mulated over the sampling procedure, compound to produce diverse samples.
where f(z, () and g(() are known as the drift and diffusion functions and dw is the standard Wiener process (Song et al., 2021).
At every point t E (0, T1, Equation (1) produces marginal latent variable distributions pe pz(z) = f Px (x)x))(xxxdxx a E Rd, With appropriate f(z, t) and g(t), Pe. (z|x) is Gaussian distribution with closed form mean and variance. Generally, these g and are also selected such that pr T(z). a simple Gaussian prior.
The aim of diffusion models is to learn the reversal of Equa- tion (1), described by a matching SDE (Song et al., 2021)
In this work, we study diffusion model inductive biases through analysis of these network denoiser approximation errors. Using this approach, we find that irrespective of neu- ral network architecture, denoisers make similar approxima- tion errors in both magnitude and quality. Through analysis of the gradients of these denoisers, find evidence of shared local inductive bias across image diffusion models.
dz = [f(2, () - vv log log)((z)) di + g(()dw (2)
Starting from any pr(z) R T(z), every marginal distribu- tion of the solutions to Equation (2) match those of Equa- tion (I). Notably, the reverse time SDE has a corresponding probability flow ODE (PF-ODE) which also shares this property
From this, we hypothesize that neural diffusion model gen- eralization arises in part through locally biased operations. We establish evidence for this hypothesis by approximat- ing these operations with patch-based empirical denoisers. Using these patch estimators, we demonstrate that for large portions of the forward diffusion process, local denoisers are equivalent to regions of the optimal denoiser. Further, we find that over the portion of the sampling procedure in which network denoisers deviate from optimal denoisers, patches of network outputs are closely approximated by patch empirical denoisers.
1 2 f(x,t) -20---915 =0(()) log dt.
(3)
Although multiple choices of f(z, t) and g(t) are possible, Karras et al. (2022) demonstrate that many such choices are equivalent. We therefore adopt their parameteriza- tion with f(z, t) = O and g(t) = 22 resulting in tran- sition distributions pz x) = N (x, LL) and prior ा (()) = N II.). Note that for these choices, the stan- dard deviation of the added noise (t) = t.
Finally, we propose our Patch Set Posterior Composites (PSPC) denoiser which aggregates patch empirical denoisers across varying spatial locations to approximate the hypoth- esized local mechanism of denoiser generalization. Com- paring our fully training-free, empirical denoiser to network denoisers, we find PSPC and network denoisers are more similar to each other than to the optimal denoiser (Figure 1). Furthermore, samples produced using our denoiser share structural similarities to those produced by neural-network parameterized diffusion models. These findings provide strong evidence to conclude that patch denoising and com- position comprise a significant portion of the generalization behaviour of image diffusion models. Our open-sone open-source im- plementation of PSPC and other denoisers is available at bttps://githhn cwwcpcoaloaooo wawpllagruppppp/eeeeeeeee pspc.
Solving Equation (2) or Equation (3) requires estimation of Vz z log Pz(z), known as the score function. For our choice of diffusion process, the score has the form
E x
2, 1 -z
Vx log
(4)
t2
From Equation (4), score estimation is equivalent to esti- mating the posterior mean E [x|2, t), an operation referred to as denoising. As the analytic form of p(x) is generally unknown, exact computation of the posterior pa (x z) and therefore E [x|z, t] is intractable. Instead, diffusion mod- els use neural-network denoisers to approximate E [x|z, t). These denoisers are trained using an empirical data dis- tribution PD (x) = 1 ExMED 6(x - x()), with dataset D = (x(1) (N) x(() ~ p(x)), by minimizing
2. Background
A(t) x -Do(z, t)
E x( xx toup()))
Diffusion models are based on forward diffusion process that gradually adds Gaussian noise to a data distribution p(x), x e ddd. This forward diffusion process can be de- scribed through stochastic differential equations of the form
where A(t) is weighting parameter. The minimizer of Equation (5) and optimal denoiser for any (z. 1) is the em- pirical posterior mean (Vincent, 201 I; Karras et al., 2019)
E x-pp
[x|z, t|
Σ (x( (2)x()
(6)
dz f(2, t)dt + g())dw,
X
2

---

Towards Mechanistic Explanation of Diffusion Model Generalization
Network Approximation Errors
DiT
NCSN++
U-VIT
DDPM+
Empirical
on Forward Process Samples
MSE vs. Optimal Denoiser
0.10
30
DIT
t=
0.08
+
DDPM+
0.06
V-T
3
t=
0.04
0.02
= 00.3
0.00
t
0.03
0.1 0.3
3
10
30
Figure 2. Left: Mean squared error between empirical and network denoisers for three architectures CIFAR-10. Right: Comparison of network and empirical denoiser for shared is ~ pt (x(x) at three values. Network estimators have low error for small and large I, but large emors around 3. At this point, each network varies substantially from the empirical denoiser in the way.
which is simple average over the images of the dataset D values of E [0.01, 100). For each I, we compute MSE weighted by their posterior probability
over 10,000 pr(z)x)pD(x) drawn from the forward process. Across all architectures, haviour to Niedoba et al. (2024), Figure 3 that network denoisers exhibit low MSE for both small and large values of f, but substantial error for Ł E [0.3, 10). From the the right portion of Figure 2, only the middle t = row has significant differences between the optimal and network de- noiser outputs At this t, observe that all three networks make qualitatively similar approximation errors, despite significant differences in architecture and training hyperpa- rameters (see Appendix A). This observation builds upon those of Zhang et al. (2023), suggesting that the similarity of samples produced by generalizing diffusion models is the product of corresponding similarities in denoiser outputs throughout the diffusion process. These denoiser output sim- ilarities further imply that denoiser approximation errors not random optimization artifacts, but the result of shared inductive bias common to all image network denoisers.
observe similar be-
Pe (z)x(()
pr (x
Ex Pa (z)x(0))
Hereafter, we refer to the denoiser of Equation (6) as the optimal denoiser.
3. Inductive Biases of Network Denoisers
Although Equation (6) is the optimal solution the diffu- sion score matching objective, sampling using this denoiser can only reproduce exact copies of training data (Gu et al., 2023). The presence of generalization in image diffusion models beyond their training set therefore implies that de- noiser networks make approximation errors relative to the optimal empirical denoiser. Further, the similarity of dif- fusion samples across several confounding factors (Zhang et al., 2023) suggests shared class of approximation bias. To understand the generalization of diffusion models, we must understand and characterize these approximation er- rors.
To focus our analysis and avoid redundancy, we hereafter re- strict our attention to DDPM+ as a representative network denoiser. Additional quantitative and qualitative results for NCSN++ DiT, and U-ViT denoisers. confirming the consistency of results across diverse architectures, provided in Appendix D.
To begin, we simply compare the denoiser outputs of four unconditional diffusion models trained on CIFAR-10 (Krizhevsky et al., 2009) to the optimal denoiser of that
dataset. We evaluate models parameterized by NCSN++ (Song et al., 2021; Karras et al., 2022), DDPM++(Song et al., 2021; Karras et al., 2022), DiT (Peebles & Xie, 2023) and V-V (Bao et al., 2022) architectures!.
3.1. Network Denoiser Gradients
One potential inductive bias (Goyal & Bengio, 2022) of network denoisers is local inductive bias, where denoiser outputs are more sensitive to spatially local perturbations of
Figure plots the mean squared error (MSE) between net- work and optimal denoisers, evaluated over 150 discrete
"The noise in U-VIT's output for 30 is due the amplifi- cation problem for e-predictor networks (Karras al., 2022).
For architecture details, see Appendix A.
3

---

Towards Mechanistic Explanation of Diffusion Model Generalization
Gradient Concentration vs. Patch Size
Gradient Heatmaps
Square Patch Side Length [px]
33
50%
735
95
%
t=0.33
29
25
21
17-
13
t=0.3
t 1030
9
=0.003
0.01
0.03
0.1
0.3
Figure 3. Left: Comparison of average square patch sizes required capture 50%, 75%, and 95% of the total gradient sensitivity heatmap. As / increases, larger patch sizes are required to capture a fixed percentage of the total gradient sensitivity heatmap. Right: Gradient sensitivity heatmaps for DDPM++ denoiser CIFAR-10 for output pixel (15.15) across varying
than distant ones. We investigate potential local inductive bias in network denoisers by measuring the sensitivity of the DDPM++ denoiser (Karras et al., 2019) each input pixel through their gradients. For each I, we define the gradient sensitivity heatmap as
with While 13 13 patch is required to capture 50% of Eig G y, t) 30, the same percentage can be accounted for in 3 patch at t = 0.03.
Figure 3 provides preliminary evidence that network de- noisers are predominantly locally sensitive to changes in z. This is somewhat surprising, as the optimal denoiser by definition globally sensitive to any changes z. Since pt (x()Iz) x P: (z(xx()) and (z is Gaussian, the pos- terior probability of any (() related to the squared dis- tance between every pair of pixels in x() and z.
G(r,y,
E xe-pi (2c(s),gg
Σ
Do, (),t))mm
(8)
Here, (o t).. indicates the output of the network de- noiser at spatial position (I, 3) and channel c where where E I,..., w. y f1....,hi h), and e E 21, for an RGB image of height h and width w.
4. Local Denoising Mechanisms
Section 3 presents strong evidence that diffusion model de- noisers deviate substantially and similarly from the optimal denoiser. Moreover, Section 3.1 finds that network denoiser gradients show evidence of local inductive bias. However, Figure 2 also shows that network denoisers accurately es- timate the empirical posterior mean for most despite the inherent global sensitivity of this quantity. To resolve this apparent contradiction, we hypothesize that network denois- ers perform local computations whose combined result equivalent to the optimal denoiser for most values of
Gæ y, captures the channcl-averaged absolute gradient of the network denoiser output at pixel (z, y) with respect to z, a measure of the sensitivity of an output pixel of the network denoiser to each input pixel. In practice, we evalu- ate the expectation of Equation (8) using 1,000 z samples drawn from the forward process per t. We plot four such heatmaps in Figure 3.
The right portion of Figure 3 confirms that network denois- ers demonstrate strong local inductive bias. At 0.03, the network denoiser is almost exclusively sensitive to the same spatial location as the output pixel. As increases, so does the area over which the input gradient concentrated.
One example of such local computation is denoising over patches of the input z. Formally, we denote cropping ma- trices as C E {0, 1 1xx Then, for any such C, which produces patches x Cx(1) and zc Cz, we define (0)
We quantify this local inductive bias by measuring the erage side length of square patch centered at pixel (Ir y) required to capture a fixed percentage of Σr, G(r, 4) We plot this quantity in the left panel of Figure 3. For all t, the majority of the input gradient is concentrated within a square 15 15 pixel region around the output pixel. Further, the strength of the local inductive bias inversely correlated
the patch posterior with pz x = NN PI,,) as
(1)
Pt
2c
x
px(x (zc)
ExtrED 2c xx
(1)

---

Towards Mechanistic Explanation of Diffusion Model Generalization
Patch Posterior Mean Optimal Denoiser
Patch Posterior Mean Network Denoiser
Forward Process Samples
Reverse Process Samples
0.20
Mean Squared Error
0.15
0.10
0.05
0.00
0..000
0.01
0.03
0.1
0.3
10
30
0.003
0.01
0.03
0.1
0.3
30
3x3
5x5
7x7 9x9
11x11
15x15 19x19
23x23
27x27
Optimal (32x32)
Figure 4. Left: Comparison of patch posterior means with varying patch sizes corresponding patches of the optimal denoiser over forward process samples. For < 1, relatively small square patch posterior means exactly match the optimal denoiser. As / increases, larger patch sizes required to exactly estimate patches of the optimal denoiser. Right: Comparison of patch posterior means with varying patch sizes to DDPM++ denoiser patches on z drawn from the reverse process. Patch posterior means estimate the network denoiser patches better than the optimal denoiser for all < 3. As decreases, so does the patch size which hest estimates network denoiser patches.
and the empirical patch posterior mean
ilarly, becomes large, both posteriors will approach uniform distribution over D. We note that these two cases correspond to the regions of Figure 2 in which network estimators accurately estimate the empirical posterior mean.
Xx %c.4 1] PD
( Σ Pt x x
2c
(i) x
(10)
The left portion of Figure empirically confirms these cases of similarity. It plots MSE between 18xx &z. t and CE(x(' z, t), averaged across E C, and 10,000 ~ Pt (2, x()) for varying and patch sizes s. For both 0.1 and large patch posterior means are similar optimal denoiser patches, regardless of patch size. Further, as t increases, larger patch sizes required to accurately estimate the optimal denoiser. This matches the correlation between local sensitivity and & observed in Figure 3. No- tably, the region which patch posterior means are poor estimators the optimal denoiser is similar to the regions of Figure 2 in which network denoisers do match the optimal denoiser.
Like the empirical denoiser of Equation (6), the patch posterior mean is a simple average over dataset elements, weighted by posterior probabilities. However, unlike the empirical denoiser which uses full images x(+), the patch posterior mean is computed over a spatial subset of the dataset where each image is cropped by C.
It is particularly convenient to work with the set of square cropping matrices. We define C(r, y, s) as square crop- ping matrix such that C(x, y. x)x(1) (i) is the 8 x pixel patch of (0) with upper left corner at pixel (x, y)- For a patch size s and a square image of spatial size
h, we denote }}} as the set of all such cropping matrices. Notably, C, does not include cropping matrices which would require any padding of x(1 at the image boundaries.
C. [C(z, s) x {O....,At
s) E (O...., .....
If network denoisers utilize localized denoising mechanisms to approximate the optimal denoiser over the forward pro- cess, we would expect this mechanism to be used for reverse process samples as well. Although patch posterior means approximate optimal denoiser patches well over the forward process, from the right subplot of Figure this is not the case for sampled from the reverse process. Instead, patch-based denoisers estimate network denoiser patches well for all < 5 and especially for 3 x patch denoisers when t < 0.3. This provides further evidence that network denoisers may utilize local denoising operations such as patch posterior means to approximate the optimal denoiser.
We have hypothesized that local inductive biases in network denoisers are the result of local denoising operations which approximate the optimal denoiser. However, in general, because pa (x zc)+ *
Epp |xc kc,4] 4] A CEp Xx
find
(x(( z). Why then would we expect network denoisers to use local posterior mean estimates to estimate the global posterior mean?
when these distributions similar. For sufficiently small & and z drawn from the for- ward process, (x zc) Rs pe(x 5(x(0). Sim-
Critically, there
5

---

Towards Mechanistic Explanation of Diffusion Model Generalization
Patch Posterior Means
Patch Set
G(z.y.t) v.t)
Co. A=0.3
Co. A=0.5
Co. A=0.7
Posterior Composite
Input Image
Figure 5. Our PSPC denoiser. First, is decomposed into patches using a set of cropping matrices. For each patch, we compute the patch posterior mean via Equation (9). Resulting means then combined into one image and normalized by the by the number of patches that overlap each pixel. Although square patches visualized, PSPC can be used with any set of cropping matrices.
Figure 6. CIFAR-10 gradient sensitivity maps and corresponding PSPC-Flex cropping matrices for varying À at t = 3. Coloured re- gions indicate areas cropped by Ca. Unlike PSPC-Square, PSPC- Flex patches are adaptive to average network sensitivity.
5. Patch Set Posterior Composites
Figure 4 demonstrates that patch posterior means of varying sizes approximate patches of diffusion network denoisers well over both forward and reverse processes. This finding motivates our proposed methodology, which combines patch posterior means at multiple spatial locations.
the network denoiser. Patch size schedules can be found in Appendix B.
5.2. PSPC-Flex
Formally, given an arbitrary set of cropping matrices (Ci....: C which denote these spatial locations, we de- fine our Patch Set Posterior Composite (PSPC) method
While square patches mimic the square receptive fields of convolutional U-Nets, Figure 3 suggests that denoisers gra- dients are not perfectly square. To enable more flexible, non-square patches of varying shapes and sizes, we intro- duce PSPC-Flex, which utilizes adaptive patch sets based on average gradient sensitivity maps.
1(2,4,)
((
Σ ECTO) CEC
c CEC
zc.
(11)
We precompute sensitivity maps using Equation (8), averag- ing DDPM++ denoiser gradients over 1000 forward process samples for each from the EDM sampling schedule. Then, for each G(x, y, t), we construct a flexible cropping matrix Cc (x. y, f, A) by greedily selecting pixels at po- sition (i. in descending order of Gz y, t)x,, until the cropped region defined by Ca y, L, A) contains fixed portion X E [o, 1] of the total gradient
The patch set posterior composite Equation (11) estimates the patch posterior mean for every patch C E C of The output is produced by summing each of these patch posterior means together before normalizing by (EcTC) number of patches which overlap each pixel. This process is described in Figure 5. In practice, we estimate patch poste- rior means using the fast nearest-neighhour score estimators of Niedoba et al. (2024).
the
Σ (Ca A)
E G(I.y,t)..
The performance of Equation (11) is dependent on the choice of patch set C. We introduce two possible choices of patch set below which represent two variants of our method- ology.
(12)
Figure 6 demonstrates how adaptive cropping matrices better capture the average sensitivity shown in the gradient sensitivity maps. We construct the patch set of PSPC-Flex using these adaptive patches CG = [Ca(2, y, 1, A) x E (O,..., w 1} y E {O...., h 1}}- Similarly to PSPC-Square, we utilize a time varying thresh- old function (tt) which is described in Appendix B.
5.1. PSPC-Square
natural choice patch set Cs, the set of overlapping square patches of spatial size s defined in Section 4. However, Figure shows that the optimal patch size varies substantially across the reverse diffusion process. To adapt our method to this observation, we define patch size sched- ule s(t): g + .iI...,,,, The square Patch Set Posterior Composite (PSPC-Square) is defined by Equation (9) with = Cu(t)- In practice, we set s(t) according to Figure 4, choosing patch size each & with the lowest error versus
6. Results
We evaluate our method on three image datasets - CIFAR- 10 (Krizhevsky et al., 2009), FFHQ 64x64 (Karras et al., 2019), and AFHQv2 64x64 (Choi al., 2020). For each
6

---

Towards Mechanistic Explanation of Diffusion Model Generalization
CIFAR 1 10 32x32
FFHQ 64x64
AFHQ 64x64
0.04
Forward Processs
0.03
MSE vs. DDPM++
0.02
6:01
0.0
0.01
0.03
0.1
0.3
10
30
0.02
0.03
0.1
0.3
3
0.01 0.05
0.1
033
10
30
Reverse Process
0.0
Optinal Denoiser
Gaussian Denoiser
CFDM
PSPC- -Squre (Ouss)
PSPC-Flex (Ours)
Figure 7. Comparison of various denoisers against DDPM++ over forward and reverse processes. For every dataset, PSPC denoisers consistently provide better estimates of DDPM++ outputs than the optimal denoiser. In most cases, PSPC-Flex yields the best estimate of network denoisers, outperforming the Gaussian and CFDM denoisers. For comparisons against other network baselines, see Appendix D.
dataset, we generate an evaluation set of samples drawn from the forward and reverse processes. For both sets, we use the values defined by the EDM sampling procedure (Karras et al., 2022), with 18 steps for EDM and 40 steps for both FFHQ and AFHQ. To produce our reverse process evaluation set, we utilize pretrained + EDM models with their default Heun sampler to generate solutions to Equation (3). For each f, we draw 10, 000 from each process for CIFAR-10 and 2000 z for FFHQ and AFHQ.
Qualitatively, we compare the output of PSPC-Flex to sev- eral CIFAR-I0 network denoisers and the empirical denoiser in Figure 1. For every t, PSPC-Flex outputs are visually sim- ilar to the outputs of the various network denoisers, suggest- ing that local denoising operations comprise a significant portion of the generalization mechanism of diffusion models. Additional denoiser outputs can be found in Appendix E.
6.2. PSPC Sampling
Inspired by the remarkable similarity between network de- noisers and PSPC-Flex evidenced by Figures I and 7, investigate the efficacy of PSPC-Flex as fully training-free, completely non-neural diffusion model.
6.1. Network Denoiser Comparison
To evaluate the similarity of PSPC to network denoisers, we compare various empirical denoisers against the EDM DDPM++ denoiser outputs for each dataset. In addition to our PSPC-Square an PSPC-Flex methods, we evaluate the empirical denoiser of Equation (7), the Gaussian de- noiser decribed by Wang & Vastola (2024); Li al. (2024), and the Closed-Form Diffusion Model (CFDM) denoiser of Scarvelis et al. (2023).
Figure compares PF-ODE sampling trajectories using DDPM++ and PSPC-Flex denoisers, starting from shared (2) for each row. PSPC-Flex samples are remarkably similar in structure those of the diffusion model. For example, the FFHQ samples for both denoisers resemble brown-haired subject on blue background. However, the sample quality of PSPC-Flex is consistently worse than those of DDPM++. Although the outputs of both denoisers is generally quite similar at each point every trajectory, the errors made by PSPC-Flex starting in the middle of each trajectories appear compound negatively. This leads to substantial visual artifacts in the final samples of PSPC-Flex.
Figure plots the MSE of each of the denoisers against the neural network denoiser. We find that for both forward and reverse processes, both PSPC variants have substantially lower MSE against the network denoiser outputs as com- pared to the empirical and CFDM denoisers. Compared to methods generally have better methods, the flexi- bility of the PSPC-Flex patch set produces better estimates than those of PSPC-Square. Over the reverse process evalu- ation sets on every dataset, PSPC-Flex generally estimates the network denoiser better than PSPC-Square and all other methods.
the Gaussian denoiser,
MSE for f E [0.3,3 3). Comparing
Despite these artifacts, Figure 9 quantitatively demonstrates that PSPC-Flex samples are still significantly more similar to network denoiser samples than the samples produced by the optimal, CFDM, or Gaussian denoisers. We measure similarity using the cosine similarity of SSCD descriptors

---

Towards Mechanistic Explanation of Diffusion Model Generalization
Network Denoiser Sampling
PSPC-Flex Sampling
CIFAR-I1
EDM
Ours
Ours
60.09
10.35
1.55
1.30
0.30
3.56
1.09
CL34
FPHQ
Ours
Ours
AFHQ
Ours
Ours
11:29 3.54
11:3 5.38 7.35
Figure 8. PF-ODE sampling trajectories starting from the same initial z, comparing the DDPM++ score estimator (Left) with our PSPC-Flex estimator (Right). At each we display the noisy image along with the output of the DDPM+ denoiser and PSPC-Square on that z. In all cases, minor differences between DDPM++ and PSPC-Square outputs only occur over intermediate t. Sampling using PSPC-Flex yields samples which are similar in content the network samples. However, compounding errors lead suboptimal final samples.
(Pizzi al., 2022), embedding model used to detect image copies (details in Appendix C) Notably, across all net- work architectures, PSPC-Flex similarity scores approach 0.6, the threshold used to determine copying by Zhang et al. (2023).
SSCD Cosine Similarity
Optimal
1.00
0.23
0.22
0.35
0.35
0.32
0.30
CFDM
0.23
1.00
0.30
0.23
0.25
0.23
0.22
0.36
0.37
Gaussian
0.22
0.30
1.00
0.34
0.38
0.33
0.33
0.70
0.68
7. Related Work
NCSN++
0.34
0.23
0.34
1.00
0.80
0.90
0.79
0.55
0.46
Diffusion generalization There is broad literature consid- ering diffusion generalization. Comparing variety of net- works, Zhang et al. (2023) finds diffusion models produce consistent samples despite differences in architecture and training. Both Niedoba that diffusion models only deviate from empirical denoisers for intermediate Further, Niedoba et al. (2024) identify that errors in this region primarily responsible for dif- fusion generalization. Similarly, Yi et al. (2023) attributes diffusion generalization to "slight differences" between the network and empirical denoisers. Kadkhodaie et al. (2024) suggest that generalization stems from geometrically adap- tive harmonic bases. However, they do consider how trained models deviate from empirical denoisers.
DIT
0.35
0.25
0.38
1.00
0.79
0.59
0.51
DDPM
0.35
0.23
0.33
0.50
7.9
1.00
0.81
0.53
0.46
U-VIT
0.33
0.22
0.33
0.79
0.72
0.81
1.00
0.32
0.45
(2024) and Xu al. (2023) find
PSPC-Flex
0.32
0.36
0.55
0.59
0.55
1.00
0.88
PSPC-Square
0.30
0.37
0.68
0.46
0.51
0.46
0.45
0.88
1.00
Optimal
CFDM
Gaussian
NCSN
DIT DDPM+
U-VIT
PSPC-Fles SSPC-Square
Figure 9. SSCD cosine similarity of CIFAR-10 PF-ODE samples produced with varying denoisers from shared z initial conditions. Average PSPC-Flex similarity neural-networks (μ 0.55) significantly higher than the similarity of optimal (μ 0.34). CFDM (μ+ 0.23), Gaussian (μ 0.34) denoisers.
Several other methods have been proposed reproduce diffusion generalization. Both Wang & Vastola (2024) and Li et al. (2024) find strong correlations between network de- noisers and optimal denoisers under the simplistic model of Gaussian p(x). Closed-fore form diffusion models (Scarvelis
8

---

Towards Mechanistic Explanation of Diffusion Model Generalization
al., 2023) bias the empirical denoiser with a spectral bias (Rahaman et al., 2019) to produce generalization mecha- nism.
approximate the optimal denoiser. Approximating such local operations with patch posterior means, we find that these means are excellent approximators of optimal denoiser patches over the majority of the forward diffusion process.
Concurrent to our work, Kamb & Ganguli (2024) propose a empirical, patch-based denoiser which leverages empir- ical patch posterior means and an increasing schedule of square patch sizes to approximate network denoiser outputs. While their method shares similarities with PSPC-Square, work differs in several key ways. First, PSPC estimates patch-posterio means exclusively over localized patch sets. We found this to be particularly beneficial on datasets such as FFHQ, where image features are correlated to specific image positions. Second, while Kamb & Ganguli (2024) focus their analysis on convolutional diffusion denoisers, our work extends to both U-Net and DiT architectures. This broader evaluation is essential to our conclusion that local inductive biases form an important component of diffusion model generalization, regardless of architecture.
By spatially aggregating local patch denoisers, we find the resulting PSPC denoisers are remarkably similar to network denoisers when evaluated at identical (z, t) inputs. Addi- tionally, PSPC samples share structural similarity to those produced by diffusion models when sampling is identically initialized. We believe that the performance of PSPC is strong empirical evidence that a significant portion of the generalization behaviour of image diffusion models arises from a mechanism which is substantially similar to simple time-varying patch denoising and compositing operations.
Our work has numerous applications. Understanding the mechanisms of diffusion generalization helps determine the cases when models fail to generalize, mitigating claims of "digital forgery" (Somepalli et al., 2023). Specifically, patch posterior probabilities (Equation (9)) are a promising signal for identifying specific training images which influ- ence the generation a sample, with copyright and training- data licensing implications. In addition, patch- dif- fusion generalization mechanisms may be exploited to im- prove training and sampling efficiency, as demonstrated by the preliminary exploration of Wang et al. (2023). Fi- nally, further improvements to empirical denoisers such as PSPC-Flex may result in fully training-free models of simi- lar quality to neural diffusion models, eliminating the fiseal and environmental costs of diffusion training.
Patch-based Methods Patch-based methods have long been used in image processing. Classical image denoising meth- ods such as Field of Experts (Roth & Black, 2005) model images as Markov Random Fields learn patch-based pri- ors. Methods such expected patch log likelihood (Zoran & Weiss, 2011 1) and half quadratic splitting (Elad& & Aharon, 2006; Zoran& & Weiss, 2011 1; Friedman & Weiss, 2021) lever- age such patch-based priors to enable maximum a priori image reconstructions. Importantly, these methods typically focus on recovering clean images rather than estimating posterior means, and typically operate on lower levels of ad- ditive Gaussian noise than is typical in the diffusion setting.
Patch-based methods have also been used in image genera- tion. classical single image generation methods synthesize images by ensuring similarity between patches of the in- put and output images (De Bonet, 1997; Efros & Leung, 1999; Barnes et al., 2009; Simakov et al., 2008) while more recent, deep learning approaches leverage patches to train GANs(Shaham al., 2019) and diffusion models (Nikankin et al., 2023; Wang et al., 2025). In the area of unconditional image generation, Ding et al. (2023) and Wang et al. (2023) utilize patch-based approaches to improve diffusion model training and performance.
A Acknowledgements
We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), the Canada CIFAR AI Chairs Program, Inverted AI, MITACS, and Google. This research was enabled in part by techni- cal support and computational resources provided by the Digital Research Alliance of Canada Compute Canada (al- liancecan.ca. ca), the Advanced Research Computing at the University of British Columbia (arc. auc..ubc.ca,, and Amazon.
Impact Statement
8. Conclusions
This paper presents work whose goal is to advance the field of Machine Learning. There many potential societal consequences of our work, which we discuss here. As generative models become available and are used by the general public, understanding the generalization behaviours is more important than ever. Sentiment towards generative image models has been negatively influenced by instances of dataset reproduction in open-source models. Understand- ing the mechanisms by which these models generalize is key stepping stone to preventing such copies. Further, ex-
Our work investigates the generalization mechanisms of image diffusion models. By comparing network denoisers and the optimal denoiser, we find strong evidence that a persistent local inductive bias in network denoisers results in denoiser approximation errors which are both qualitatively and quantitatively similar across all evaluated networks. We hypothesize that this local bias is the result of network denoisers employing local denoising operations to partially
9

---

Towards Mechanistic Explanation of Diffusion Model Generalization
plicit mechanisms such those presented in work may serve as invaluable tools in assigning attribution to generated samples to the source images which inspired them.
Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA,, USA, November 28 December 9, 2022, 2022.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
References
Bao, F., Li, C., Cao, Y., and Zhu, J. All are worth words: vit backbone for score-based diffusion models. In NeurlPS 2022 Workshop on Score-Based Methods, 2022.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion prob- abilistic models. In Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, De- cember 6- 12, 2020, virtual, 2020.
Barnes, C., Shechtman, E., Finkelstein, A., and Goldman, D. B. Patchmatch: A randomized correspondence algo- rithm for structural image editing. ACM Trans. Graph., 28(3):24, 2009.
Kadkhodaie, 7., Guth, F., Simoncelli, E. P., and Mallat, S. Generalization in diffusion models arises from geometry- adaptive harmonic representations. In The Twelfth Inter- national Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024.
Bellman, R. Dynamic programming, Science, 153(3731); 14-37, 1966.
Choi, Y. Uh, Y. Yoo, and Ha, J.-W. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Kamb, M. and Ganguli, S. An analytic theory of creativ- ity in convolutional diffusion models. arXiv preprint arXiv:2412.20292, 2024.
De Bonet, J. S. Multiresolution sampling procedure for analysis and synthesis of texture images. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques. pp. 361-368, 1997.
Karras, T., Laine, S., and Aila, T. A style-based generator architecture for generative adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 4401-4410. Computer Vision Foundation IEEE, 2019.
Ding, Z., Zhang, M., Wu, J., and Tu, Z. Patched denoising diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Repre- sentations, 2023.
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
Efros, A. A. and Leung, T. K. Texture synthesis by non-
parametric sampling. In Proceedings of the seventh IEEE international conference on computer vision, volume 2, pp. 1033-1038. IEEE, 1999.
Elad, M. and Aharon, M. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image processing, 15(12)33736-3745 2006.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In 3rd International Conference on Learn- ing Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Friedman, R. and Weiss, Y. Posterior sampling for image restoration using explicit patch priors. arXiv preprint arXiv:2104.09895, 2021.
Krizhevsky, A., Hinton, G., al. Learning multiple layers of features from tiny images. 2009.
Li, X., Dai, Y., and Qu, Q. Understanding generalizability of diffusion models requires rethinking the hidden gaussian structure. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NewrlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
Goyal, A. and Bengio, Y. Inductive biases for deep learn- ing of higher-level cognition. Proceedings of the Royal 2022.
Society A, 478(2266):0011000)
Gu, X., Du, C., Pang, T., Li, C., Lin, M., and Wang, Y. On memorization diffusion models. arXiv preprint arXiv:2310.02664. 2023.
Niedoba, M., Green,
D., Naderiparizi, S., Lioutas, V., Lav- ington, J. W., Liang, X., Liu, Y., Zhang, K., Dabiri, S., Šcibior, A., Zwartsenberg, B., and Wood, F. Nearest neighbour score estimators for diffusion generative mod- els. In Forty-first International Conference on Machine
Harvey, W., Naderiparizi, S., Masrani, V., Weilbach, C., and Wood, F. Flexible diffusion modeling of long videos. In Advances in Neural Information Processing Systems 35:
10

---

Towards Mechanistic Explanation of Diffusion Model Generalization
Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024.
CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, 2023.
Nikankin, Y., Haim, N., and Irani, M. Sinfusion: Training diffusion models on a single image or video. In Inter- national Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, 2023.
Song, Y., Sohl-Dickstein, J., Kingma. D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In 9th Interna- tional Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195-4200 2023.
Vincent, P. connection between score matching and de- noising autoencoders. Neural computation, 23(7): 166) 1674, 2011.
Pizzi, E.,, Roy, D.D Ravindra, S. N., Goyal, P., and Douze, supervuper descriptor for image copy detection. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 14532-14542, 2022.
Wang, B. and Vastola, J. The unreasonable effectiveness gaussian score approximation for diffusion models and its applications. arXiv preprint wrXiv:2412.09722 2024.
Wang, W., Bao, J., Zhou, W., Chen, D., Chen, D., Yuan, L., and Li, H. Sindiffusion: Learning a diffusion model from a single natural image. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
Rahaman, N., Baratin, A., Arpit, D., Draxler, F., Lin, M., Hamprecht, F., Bengio, Y., and Courville, A. On the spec- tral bias of neural networks. In International conference machine learning, pp. 5301-5311 PMLR, 2019.
Wang, Z. Jiang, Y., Zheng, H., Wang, P., He, P., Wang, Z., Chen, W., and Zhou, M. Patch diffusion: Faster and more data-efficient training of diffusion models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent difffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp- 10684-10695, 2022.
Roth, S. and Black, M. Fields of experts: A framework for learning image priors. In 2005 IEEE Computer Society Conference Computer Vision and Pattern Recognition IEEE, 2005.
Xu, Y., Tong. S., and Jaakkola, T. Stable target field for reduced variance score estimation diffusion models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023.
(CVPR'0S) 05), volume 2, pp.
Scarvelis, C., Borde, H. d. O., and Solomon, J. Closed- form diffusion models. arXiv preprint arXiv:2310.123955 2023.
Yi, M., Sun, J., and Li, Z. On the generalization of diffusion model. arXiv preprint arXiv:2305. 14712, 2023.
Shaham, T. R., Dekel, T., and Michaeli, T. Singan: Learn- ing generative model from a single natural image. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4570-4580, 2019.
Zhang, H., Zhou, J., Lu, Y., Guo, M., Wang, P., Shen, L., and Qu, Q. The emergence of reproducibility and con- sistency in diffusion models. Forty-first International Conference on Machine Learning, 2023.
Simakov, D., Caspi, Y., Shechtman, E., and Irani, M. Sum- marizing visual data using bidirectional similarity. In 2008 IEEE conference on computer vision and pattern recognition, pp. 1-8. IEEE, 2008
Zoran, D. and Weiss, Y. From learning models of natural image patches to whole image restoration. 20/1 in- ternational conference on computer vision, pp. 479-486. IEEE, 2011.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. International conference on machine learning, pp. 2256-2266222 2265. PMLR, 2015.
Somepalli, G., Singla, V., Goldblum, M., Geiping, J., and Goldstein, T. Diffusion art or digital forgery? investigat- ing data replication in diffusion models. IEEE/CVF Conference on Computer Vision and Pattern Recognition,

---

Towards Mechanistic Explanation of Diffusion Model Generalization
A. Network Denoiser Architecture Details
A.1. CIFAR-10
Our work analyzes diffusion generalization across diverse set of network architectures, A summary of some of the differences between the networks compared in Figures and is given in Table
Table 1. Differences in hyperparameters between evaluated CIFAR-10 Network Denoisers
+
NCSN++
DiT
U-VIT
Network Architecture
U-Net
U-Net
Vision Transformer
Vision Transformer prediction
Output
EDM Residual
EDM Residual
EDM Residual
Diffusion Process
EDM
EDM
EDM
Variance Preserving
Patch Size
N/A
N/A
4
2
Hidden Size
varies
varies
768
384
Noise Encoding
positional
fourier
positional
token
For the NSCN++ and DDPM++ architectures, we utilize the pretrained unconditional model checkpoints provided by (Karras al., 2022). For the Diffusion Transformer, we adopt the code of (Peebles & Xie, 2023), utilizing the EDM preconditioning scheme and data augmentation pipeline. We trained DiT-B/4 network from scratch 200 million examples using the Adam (Kingma & Ba, 2015) optimizer and the hyperparamters given in Table 2
Table 2. Hyperparameters for DiT training CIFAR-10
Hyperparameter
Value
Batch Size
512
Learning Rate
0.0001
B1
0.9
82
0.999
1E-8
Patch Size
4
# Heads
12
Hidden Size
768
Transformer Blocks
12
Dropout Ratio
0.12
Augmentation Rate
0.12
For U-VIT (Bao al., 2022), we utilize their pretrained checkpoint. Since U-VIT is an prediction network, with variance preserving diffusion process, convert predictions from their network into predictions through the equation
z - -(t()) s(t)
(13)
In general, the diffusion models analyzed in this work produce high quality samples, as measured by Frechet Inception Distance (FID) (Heusel al., 2017). The 50k FID scores for each model are given below in Table
Table 3. FID Scores for CIFAR- diffuusion models Model
FID
DDPM+ (Karras et al., 2022)
1.97
NCSN+ (Karras et al., 2022)
1.98
DiT++ (Peebles & Xie, 2023)
9.08
U-ViT (Bao al., 2022)
3.08
A.2. FFHQ and AFHQ
For all FFHQ and AFHQ experiments, we utilize a pretrained DDPM+ EDM checkpoint (Karras et al., 2022).
12

---

Towards Mechanistic Explanation of Diffusion Model Generalization
32x32 Patch Schedule
CIFAR-10
27
5-
0.012
001
0,10
0.1
0.3
30
30
Figure 10. Patch Schedule for 32x32 Datasets
B. Patch Composite Algorithm Details
For each dataset, tuned individual schedule for the patch size and gradient threshold for PSPC-Square and PSPC-Flex respectively. Schedules were tuned by minimizing the average path posterior mean error over forward process samples for each dataset. Figures 10 and 1l shows the patch schedule s(t) used for all PSPC-Square results while Figure 12 shows the threshold schedule for all PSPC-Flex results.
C. Experimental Details
C.1. SSCD Details
To compute the SSCD cosine simiilarities of Figure official SSCD github repository (Pizzi et al., 2022). SSCD is self-supervised method trained which produces image descriptor which is similar among copied images.
the wwl.limaaeeett.. checkpoint provided by the
For our application, we started from a shared set of 1000 z ~ π (z) and then used each of the denoisers listed Figure to draw samples from those initial points. We used deterministic PF-ODE sampling with an Euler solver and the EDM sampling schedule (Karras al., 2022). We then encoded each denoiser's samples using the SSCD encoder and computed cosine similarities between each of images
sxc)(x1) (xc)
DsscD(X1, x2)
(14)
[scd( ( sxcd
We averaged the cosine similarities across the 1000 images per denoiser to obtain the results in Figure
D. Additional Network Baselines
Figure compares four network denoisers on CIFAR-10 10 and finds that their outputs are qualitatively similar, and that they have similar mean squared error when compared against the optimal denoiser. Due to the similarity between networks, for clarity of presenation we utilized DDPM+ in Figures 3, 4 and 7 and did not present additional results using NCSN++, DiT or U-VIT network denoisers. For completeness, have included additional versions of these figures below
13

---

Towards Mechanistic Explanation of Diffusion Model Generalization
64x64 Patch Schedule
FFHQ AFHQ
27
11
Figure 11. Patch Schedules for 64x64 Datasets
Gradient Threshold Hyperparameter Schedules
1.-
CIFAR-10
FFHQ
AFHQ
0.5
0.7
0.1
0.5
0.4
0.5
Figure 12, Gradient threshold schedules for varying datasets.
14

---

Towards Mechanistic Explanation of Diffusion Model Generalization
D.I. Gradient Concentration
Figure 13 plots the average side length square patch required to capture a fixed percentage of the gradient concentration heatmap for DDPM++, NCSN++ and DiT network denoisers. Although there some differences in the gradient concentrations, the general trend is consistent that network denoisers exhibit more locally concentrated gradient for small t, and less concentration as f increases.
Gradient Concentration vs. Patch Size
Square Patch Side Length [px]
29
25-
DDPM++
21-
NCSN
DIT
17-
50%
75%
13
95%
9-
=0.003
0.01
0.03
0.1
0.3
3
10
30
Figure Comparison of the Gradient concentration DDPM++, NCSN+ DiT networks across varying diffusion time, All three network denoisers display the same anti-correlation between gradient concentration and diffusion time.
D.2. Patch Errors
Figure 4 compares patch posterior means to patches of DDPM++ denoiser outputs for drawn from that network's 's reverse process. In Figure 14, we produce similar plots for DDPM++, NCSN++, DiT and U-VTT network denoisers. Across each denoiser, patch posterior means have similar errors when compared to patches of the network output. At both small and large f, appropriately sized patch posterior means have low MSE against network denoiser patches, with poorer MSE over intermediate Additionally, for each network we observe a similar trend that as increases, network denoiser patches are best estimated by patch posterior means of increasing spatial size.
Notably, for U-ViT, find that patch posterior means do not estimate high denoiser outputs well. This is symptom of the amplification problem for e-predictor networks, described by (Karras et al., 2022).
D.3. Denoiser Errors
Figure 7 visualizes the mean squared error between variety of proposed denoisers and DDPM+ network denoiser across the forward and reverse processes of three different datasets. However, other choices of network denoiser baseline are possible. Figure 15 plots denoiser MSE against four different network baselines. As checkpoints are only available on CIFAR-10 for DiT and U-ViT, no comparison is performed on FFHQ and AFHQ for these baselines. In addition to the denoisers shown in Figure 7, Figure 15 also plots MSE between pairs of network denoisers.
Examining Figure 15, PSPC-Flex consistently has lower MSE versus each network baseline than any other non-network denoiser. However, our method is consistently outperformed by the outputs of other network denoisers. We note that the gap between our approach and the performance of DiT when compared against DDPM++ and NCSN+ baselines is remarkably close. We believe closing this gap is promising of future research.
E. Additional Denoiser Outputs
E.1. CIFAR-10
We present additional examples of denoiser outputs for the same values presented Figure 1. All z are drawn from the reverse process of the DDPM+ network denoiser.
15

---

Towards Mechanistic Explanation of Diffusion Model Generalization
Reverse Process Patch Errors DDPM++
Reverse Process Patch Errors DTT
0.125
Mosn Sqioeal Earor
0.100
0.100
0.075
0.079
0.050
0.050
0.025
0.025
0.000
(-0.003
0.01
0.03
0.1
0.3
10
30
1-0.003
0.01
0.03
0.1
0.3
30
Reverse Process Pasch Erroes NCSN++
Reverse Process Patca Errors UVIT
Monn Squard Eroer
0.100
0.100
0.075
0.075
0.050
0,050
0.025
0.025
0.000
0.00
0.01
0.00
0.1
0.3
10
30
100000
0.01
0.00
0.1
0.3
10
30
5x5
060
16x15
18x10
23323
27x27
Optiinal (32x32)
Figure 14. Comparison of patch posterior means of varying size against patches of network denoiser outputs for shared z drawn from each network's reverse process CIFAR-J0 10. Across each network, observe that patch posterior means are better estimators of network output patches than the optimal denoiser.
F. Additional PSPC-Flex Samples
We provide additional PSPC-Flex samples for CIFAR-10, FFHQ, and AFHQ datasets in Figure 24. For each row, the images in the left subplot has been generated from identical latent seed from the corresponding image in the right subplot. In addition, Figure 25 compares samples from a larger set of denoisers, including PSPC-Flex, PSPC-Square, NCSN++ Gaussian, optimal, and CFDM.
16

---

Towards Mechanistic Explanation of Diffusion Model Generalization
CIFAR-10 10 32x32
FFHQ 64x64
AFHQ 64x64
Powward Process
010
MSE vs. DDPM +++
0.00
Revere Process.
0.00
0/0
6.0
Optinal Denoiser
CEDM PSPC-Squate (Oun)
PSPC-Fles Flex (Our)
DiT UVIT
Consziin Denoiser
NC8N++
CIFAR-10 32x32
FFHQ 64x61
AFHQ 64x64
Forwor Procees
0.04
MSE vs. NCSN +++
0.00
0.0
0:11
010
31
13
20
0.11 1825 01
03
DEC D43
01
05
10 31
Reverse Proiess
0.00
0m
Optinnal
PSPC-Flex (Ouns)
PSPC-Square (Ours)
DDPM++
CIFAR- 10 32x32
FFHQ 64x64
AFHQ 64x64
Forward Process
0.0
MSE vs. DYT
0.0
Reveree Process
0.0
0.2
Optiral Denoiee
CFDM PSPC-Squan (Ouss)
PSPC-Flox (Oum)
NC
Ganssian Denoiser
DDPM++
DVT
CTFAR- 32x32
FFHQ 64x64
AFHQ 64x64
Forward Process
MSE vs. UVIT
2002
0.0
DG3
03 03
30
DO1 010
D1
03 1.0 185
30
01
03
Roverse Process
500
Optimal Denosser
CFDM
PSPC-Flex (Ouns)
Eepurre (Ouns)
DDPM+
Figure 15, Comparison of network and empirical denoisers against varying network denoiser baselines. For DIT and UVTT, no comparison performed FFHQ and AFHQ. Against each baseline, PSPC-Flex consistently outperforms Gaussian, CFDM, and optimal denoisers. However, PSPC performance consistently lags behind neural network performance.
17

---

Towards Mechanistic Explanation of Diffusion Model Generalization
Empirical
NCSN+4
+
DIT
Ours
Figure Additional CIFAR-10 10 denoiser outputs = 0.3
Empirical
NCSN +++
+
DIT
Ours
Figure Additional CIFAR-10 10 denoiser outputs
0.6
18

---

Towards Mechanistic Explanation of Diffusion Model Generalization
Empirical
NCSN
+
DIT
Ours
Figure Additional CIFAR-10 denoiser outputs
1.1
Empirical
NCSN +++
+
DIT
Ours
Figure Additional CIFAR-10 10 denoiser outputs
1.9
19

---

Towards Mechanistic Explanation of Diffusion Model Generalization
Empirical
NCSN
+
DIT
Ours
Figure 20, Additional CIFAR-10 denoiser outputs
3.3
Empirical
NCSN+4 +++
+
DIT
Ours
Figure 21. Additional CIFAR-10 10 denoiser outputs
5.3
20

---

Towards Mechanistic Explanation of Diffusion Model Generalization
Empirical
NCSN
+
DIT
Ours
Figure 22, Additional CIFAR-10 denoiser outputs
8.4
Empirical
NCSN
+
DIT
Ours
Figure Additional CIFAR-10 denoiser outputs for 12.9
21

---

Towards Mechanistic Explanation of Diffusion Model Generalization
PSPC-Flex CIFAR-10 Samples
DDPM+ CIFAR-10 Samples
PSPC-Flex FFHQ Samples
DDPM+ FFHQ Samples
PSPC-Flex AFHQ Samples
DDPM AFHQ Samples
Figure 24, Additional PSPC-Flex PF-ODE samples on 10,AR-10, FFHQ and AFHQ. Samples in the left column are generated by
PSPC-Flex, while those right column are generated by DDPM+ network denoiser, starting from the same initial
22

---

Towards Mechanistic Explanation of Diffusion Model Generalization
CIFAR-10 Samples - Shared Initial Conditions
DDPM+ +
PSPC-Square
PSPC-Fles Flex
Ganssian
Empirical
CFDM
Figure 25. Comparison of PF-ODE CIFAR-10 samples generated using varying denoisers, starting from shared initial
23


Reviews:
[{"review_id": "jU1nc7P4JN", "reviewer": "Reviewer_Se8t", "summary": "That real diffusion models generalize their training data, rather than memorize it, is not obvious: the optimal solution to a typical denoising score matching objective is the score of the (empirical) data distribution, which can only reproduce training examples. Why do diffusion models generalize, and what inductive biases determine how they generalize? The authors of this paper propose that denoising happens locally, in 'patches', rather than globally, and that this is at least partly responsible for generalization. They validate this hypothesis through a variety of numerical experiments, which show that their patch denoiser looks more like real denoisers than the 'optimal' one.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **General Comment**\nWe'd like to thank the reviewer for their thoughtful consideration of our work. Below, we have responded to specific components of your review. If you have additional questions, we would welcome further discussion.\n\n### **Relation To Broader Scientific Literature:**\n>This paper makes a major contribution to the study of diffusion model generalization/memorization, which is itself a fairly major subfield of diffusion models. It also contributes meaningfully to literature on how generative models generalize and produce 'creative' output. Their main claim is kind of similar to that of Kamb and Ganguli, which they mention, but the details of their approach are a bit different, and it is likely that the two works developed somewhat independently.\n\nAlthough our method shares many similarities to Kamb & Ganguli, our work is entirely concurrent and independent. There are some differences between our methodologies and experimental results which we will highlight here, but plan to elaborate on in our camera ready revision. \n\nThe primary difference between our works is that Kamb & Ganguli utilize shared patch distributions whereas we utilize patch distributions which are localized to specific image locations. We found that this localization was essential for FFHQ & AFHQ where the subject is centered within the image. In addition, Kamb & Ganguli utilize square patches exclusively, while PSPC-Flex uses localized, adaptive patch shapes and sizes. Experimentally, while Kamb & Ganguli restrict their analysis to only convolutional networks, we find similar generalization patterns exist even in convolution free architectures such as DiT and UViT.\n\n### **Other Strengths And Weaknesses:**\n>As a very minor point, it could be helpful to include some additional discussion of what these findings may mean. This inductive bias ('locality', to use Kamb and Ganguli's terminology) seems to be true for at least U-net- and vision-transformer-based diffusion models trained on images. What about other kinds of architectures or data sets?\n\nPlease see our response to this point in the Questions for Authors section\n\n### **Other Comments Or Suggestions:**\n>line ~158: \"plot four such heatmaps in Figure 3\" needs a period\n>\n>line ~359: \"can be found in Appendix D\" needs a period\n\nWe'd like to thank the reviewer for identifying these items. We will fix both in the camera ready revision.\n\n### **Questions For Authors:**\n> What kind of architectures does one expect to exhibit this inductive bias? If one trained a fully-connected MLP-like architecture to denoise images, would it exhibit this bias?\n\nThis is an interesting open question in our opinion. Although more rigorous study is required, we would guess that the inductive biases are probably due to interplay between the diffusion process and the correlations in the data. For example, in datasets where the features are not locally correlated, (ie shuffling the data dimensions), we would not expect this local inductive bias to be useful.\n\nOur observations in this paper seem to suggest that network architectures do not significantly affect the generalization behaviour. We would therefore expect MLPs to have similar properties to DiT & U-Net architectures when trained on the same data. This hypothesis is somewhat supported by prior art. For example, Li et al. (2024) show that the optimal linear denoiser is the Gaussian denoiser which produces outputs which are quite similar to our method.", "claims_and_evidence": "The paper has a clear hypothesis and collects an impressive set of empirical results to validate it.", "methods_and_evaluation_criteria": "The methods by which the authors test their hypothesis are reasonable, and their evaluation criteria (a mix of comparing denoisers and looking at generated samples) make sense.", "theoretical_claims": "The authors make no major new theoretical claims.", "experimental_designs_or_analyses": "The experiments the authors run are well-designed and appear sound.", "relation_to_broader_scientific_literature": "This paper makes a major contribution to the study of diffusion model generalization/memorization, which is itself a fairly major subfield of diffusion models. It also contributes meaningfully to literature on how generative models generalize and produce 'creative' output. Their main claim is kind of similar to that of Kamb and Ganguli, which they mention, but the details of their approach are a bit different, and it is likely that the two works developed somewhat independently.", "essential_references_not_discussed": "No references come to mind.", "other_strengths_and_weaknesses": "This paper is well-written, clear, and has well-made figures. It was a joy to read. \n\nAs a very minor point, it could be helpful to include some additional discussion of what these findings may mean. This inductive bias ('locality', to use Kamb and Ganguli's terminology) seems to be true for at least U-net- and vision-transformer-based diffusion models trained on images. What about other kinds of architectures or data sets?", "other_comments_or_suggestions": "line ~158: \"plot four such heatmaps in Figure 3\" needs a period\n\nline ~359: \"can be found in Appendix D\" needs a period", "questions_for_authors": "What kind of architectures does one expect to exhibit this inductive bias? If one trained a fully-connected MLP-like architecture to denoise images, would it exhibit this bias?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HQTNCi6TcQ", "reviewer": "Reviewer_HwKw", "summary": "This paper attempts to explain the mechanism behind the ability of diffusion models to generalize beyond the training data. It starts by pointing out that this ability is due to the neural denoiser deviating from the optimal empirical denoiser (the optimal denoiser for the training set). It then shows that the function learned by neural denoisers is more similar to local empirical denoisers, namely ones which operate on small-sized patches rather than on the whole image. The paper therefore concludes that it is this tendency of neural denoisers to learn local operations that enables diffusion models to generalize well.\n\n## update after rebuttal\nThe rebuttal has answered my questions. I keep my original score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **General Comment**\n\nWe'd like to thank the reviewer for taking the time to read and review our paper. Furthermore, we appreciate the additional references that the reviewer has brought to our attention. Below, we have responded to several items of your review. If there are items which you feel have not been sufficiently addressed, we would welcome additional discussion on these matters.\n\n### **Experimental Designs or Analyses:**\n\n>There are, however, several missing details that I think are important to mention and discuss:\n\nWe have clarified both of the details below. Although we believe that our definitions of $C_s$ on line 199, $C_{G(t, \\lambda)}$ on line 316, as well as equations 9,10 and 11 address these details, we will update the text in the camera ready revision to improve clarity on these items\n\n> - What was the dataset of patches used to construct each empirical patch denoiser (both for PSPC-patch and for PSPC-square)? For example, when constructing the denoised patch at location (i,j), does the computation involve all the overlapping patches extracted from all the training images, or only the patches at location (i,j) extracted from all the training images (namely, only one patch per training image)? \n\nFor both PSPC-Flex and PSPC-Square, we construct an empirical patch set for each patch posterior mean. In the case of PSPC-Flex, this corresponds to one patch set for each pixel in the input image (ie $32^2$ patch sets for CIFAR-10 and $64^2$ patch sets for AFHQ & FFHQ). For PSPC-Square, we create a dataset for each possible square patch location with no padding and a stride of one. The total number of patch sets created for PSPC-Square varies depending on patch size. In general, we create $(h - s + 1)^2$ patch sets, where $h$ is the height of the image in pixels and $s$ is the height of the square patch in pixels. For both PSPC-Flex and PSPC-Square, each patch set is created by applying the same cropping matrix to each image in the training set. This results in patch sets with one spatially localized patch per training set image. \n\n> - Were denoised patches constructed only for the patches that are fully contained within the 64x64 image, or did patches at the boundaries also contain zero-padded regions?\n\nWe did not use padding in our computations.\n\n### **Essential References Not Discussed:**\n>The paper draws connections to classical patch-based image restoration methods. But there were also quite a few patch-based image generation methods. Most of them in the context of learning from a single image. Starting from the classical texture-generation paper:\n> - J. De Bonet, \"Multiresolution sampling procedure for analysis and synthesis of texture images\", SIGGRAPH'97.\n> \n>To more recent papers, like:\n> - N. Granot, et al. \"Drop the GAN: In defense of patches nearest neighbors as single image generative models\", CVPR'22.\n>\n>These are also related to methods that learn patch statistics using GANs (e.g. SinGAN) or using diffusion models (e.g. SinFusion, SinDiffusion, SinDDM).\n\nWe thank the reviewer for drawing our attention to these highly relevant references. We will include the references mentioned in the camera ready version of the paper.\n\n### **Other Strengths And Weaknesses:**\n>A weakness is that the paper does not explain, and does not provide supporting experiments, how local processing induces coherent global structure. If when denoising a patch, the denoiser doesn't know from where in the image that patch was extracted, how does the denoiser know whether this patch is more likely to contain sky (if it was extracted from the upper part of the image) or grass (if it was extracted from the lower part of the image)?\n\nAs mentioned in the experimental design, we use spatially consistent patches for each patch posterior mean. This means that for example, when denoising patches in the upper portion of the image, the patch dataset contains patches which are more likely to contain sky than grass. We found this detail to be especially important for localized datasets such as FFHQ where facial features (noses, eyes, mouths etc.) are located in specific spatial regions of the image.\n\nAnother key finding of our work is that the ideal patch size and therefore the degree of locality is anti-correlated with the level of noise. When generating a sample with PSPC-flex or PSPC-square, we initially denoise using large patches before moving to smaller patches. The larger patches therefore condition the posterior means of the later, smaller patch denoising operations. We believe this large to small patch hierarchy is important for ensuring coherent global structure\n\n### **Other Comments Or Suggestions:**\n>Typo: It seems that in Eq. (8), the input to $D_\\\\theta$ should be $\\\\mathbf{z}$ rather than $\\\\mathbf{x}$ \n\nThank you for this observation. You are correct, the input to $D_\\\\theta$ is $\\\\mathbf{z}$. We will correct this mistake in the camera ready.", "claims_and_evidence": "The claims are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The methods and evaluation criteria make sense.", "theoretical_claims": "The paper doesn't present any formal theoretical claims in the forms of theorems or lemmas.", "experimental_designs_or_analyses": "I checked the experimental settings for all the experiments. In general, all experiments seem valid to me.\n\nThere are, however, several missing details that I think are important to mention and discuss:\n- What was the dataset of patches used to construct each empirical patch denoiser (both for PSPC-patch and for PSPC-square)? For example, when constructing the denoised patch at location (i,j), does the computation involve all the overlapping patches extracted from all the training images, or only the patches at location (i,j) extracted from all the training images (namely, only one patch per training image)?\n- Were denoised patches constructed only for the patches that are fully contained within the 64x64 image, or did patches at the boundaries also contain zero-padded regions?\n\nThese points are important for understanding how the sampling process with PSPC manages to generate sky at the top, grass at the bottom, etc. If when denoising each patch, the computation involves patches extracted from all spatial locations within the training images,  and there are no cues from padding, then the statistics for each patch is the same. Namely, patches at the upper part of the generated image should not necessarily favor sky, and patches at the bottom should not necessarily favor grass. All patches are obtained using the same operation applied to the input.\n\nIt would be great if the authors can clarify those points.", "relation_to_broader_scientific_literature": "Understand generalization in diffusion models is a topic of vast interest, both theoretically, and with empirically. \n\nThis paper presents convincing evidence that at least part of the generalization capabilities of diffusion models is associated with the tendency of neural denoisers to learn local processing. Interestingly, the tendency to learn local processing is not a feature of a particular architecture. Even DiTs, whose architecture doesn't promote this implicit bias, learn local processing. These observations are certainly interesting to the community and may draw follow-up works that attempt to explain this tendency.", "essential_references_not_discussed": "The paper draws connections to classical patch-based image restoration methods. But there were also quite a few patch-based image generation methods. Most of them in the context of learning from a single image. Starting from the classical texture-generation paper:\n- J. De Bonet, \"Multiresolution sampling procedure for analysis and synthesis of texture images\", SIGGRAPH'97.\n\nTo more recent papers, like:\n- N. Granot, et al. \"Drop the GAN: In defense of patches nearest neighbors as single image generative models\", CVPR'22.\n\nThese are also related to methods that learn patch statistics using GANs (e.g. SinGAN) or using diffusion models (e.g. SinFusion, SinDiffusion, SinDDM).", "other_strengths_and_weaknesses": "As mentioned above, this paper's main strength is that it provides convincing experiments that point to a plausible explanation for generalization in diffusion models.\n\nA weakness is that the paper does not explain, and does not provide supporting experiments, how local processing induces coherent global structure. If when denoising a patch, the denoiser doesn't know from where in the image that patch was extracted, how does the denoiser know whether this patch is more likely to contain sky (if it was extracted from the upper part of the image) or grass (if it was extracted from the lower part of the image)?", "other_comments_or_suggestions": "Typo: It seems that in Eq. (8), the input to $D\\theta$ should be $z$ rather than $x$.", "questions_for_authors": "See comments in the sections above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "feGArarfTX", "reviewer": "Reviewer_wsKg", "summary": "The paper studies the inductive bias of diffusion models that enable generalization. The authors attribute such inductive bias to the locally denoising capability of diffusion models, which is supported by the observation that the network outputs are sensitive to the local perturbation of its input. \n\nBased on this intuition, the authors then propose to reproduce the generalization of diffusion models with a patch-based local denoiser. They find that the resulting denoiser generates similar outputs to those of the real diffusion models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **General Response**\n\nWe'd like to thank the reviewer for taking the time to read and review or work. Please find our responses to your review below. If there are any items which you believe have not been addressed, we welcome this feedback.\n\n### **Other Comments Or Suggestions:**\n> Are the labels in Figure 8 correct? Which row corresponds to DDPM++ ?\n\nThe labels in Figure 8 are correct, the middle row of each subplot corresponds to DDPM++ (An EDM architecture) evaluated on the $\\mathbf{z}$ value in the top row. The left column of subplots show sampling trajectories using the DDPM++ denoiser, while the right column shows sampling trajectories using the PSPC-Flex denoiser. \n\n### **Questions For Authors:**\n\n> 1. Please clarify the gap between denoising ability and generalization ability. Why the proposed patch denoiser well-matches the diffusion models in Figure 13-20 but diverges significantly in Figure 21? Such gap is not adequately addressed in the current paper. This raises concerns on how much the locally denoising mechanism contributes to the generalization ability.\n\nThe reviewer is correct that the samples produced by PSPC are significantly different to those produced by network denoisers. Although we tried our best to match the output of network denoisers as closely as possible, our method is an imperfect approximation to their behaviour. There is a significant difference between our fully empirical method and a deep neural network trained with gradient descent over millions of $\\mathbf{z}$ samples. We believe the remarkable similarity of our method our denoiser presents compelling evidence that network denoisers may employ similar generalization mechanisms.\n\nFigure 1 & 6 illustrate that significant differences remain between PSPC and network denoisers, especially for intermediate $t$. When sampling with PSPC-Flex, these differences in denoiser outputs compound to result in PSPC-Flex PF-ODE trajectories which slowly drift from the PF-ODE trajectories of network denoisers. We discuss this briefly in section 6.2, line 371. \n\nThis process of error accumulation is visualized in Figure 8 which illustrates the denoiser outputs of PSPC-Flex and DDPM++ on shared $\\mathbf{z}$ inputs drawn from  DDPM++ (left) and PSPC-Flex (Right) PF-ODE trajectories. Comparing the network and PSPC denoisers, we can see that for all cases and $t$, the outputs are highly similar. However, differences in two, which are most pronounced in the middle of the trajectory, result in substantially worse samples in the right column of figure 8 (PSPC-Flex) than the left column (DDPM++). Despite the artifacts present in PSPC samples, we highlight that the structure of the final samples in the right column of Figure 8 are similar to the final network samples in the right column.\n\nNotably, even as sample quality degrades (ie right column, t < 0.5) both denoisers produce similar denoiser outputs. These degraded $\\mathbf{z}$ samples are obviously outside the training distribution $p_t(\\mathbf{z})$ and are clear examples of neural network generalization. The fact that PSPC-Flex outputs are similar to network denoisers in this case provides further evidence that a local denoising mechanism may be partially responsible for the generalization of diffusion denoisers.\n\n> 2. Please compare the proposed denoiser with more (classical patch based) denoisers.\n\nIn the methods & evaluation criteria section of your review, you specifically mention Closed-Form Diffusion Models (Scarvellis et al. 2023) as a missing baseline method. We have therefore implemented this method and added it as a baseline to Figure 6 [here](https://drive.google.com/file/d/1esmDa7DP2rPNjPShzRCh8UTPUUMzauPe/view?usp=sharing). We used hyperparameters $\\sigma=0.1, M=2$, which they report in Appendix section C2. for their CelebA results. However, we did not find that CFDM outputs were significantly different to the optimal denoiser with these settings.\n\nIf there are other baselines you believe are necessary, please cite them explicitly and we will try our best to implement them by the end of the discussion period.\n\n> 3. Why PCSC shares the highest similarity with the Gaussian denoiser rather than the diffusion model (Figure 9)?\n\nWe are unsure as to why PSPC-Flex samples are more similar to the Gaussian denoiser. Recently, several methods have been proposed to explain diffusion generalization, including our work, the optimal Gaussian denoiser, and the geometrically adaptive harmonic bases of Kadkhodaie et. al (2023). Understanding the similarities between these methodologies is an interesting direction for future research.", "claims_and_evidence": "The paper claims that the PSPC-Flex samples are remarkably similar to those of the diffusion model. However, Figure 8, 21 and 22 shows the generated images of PSPC-Flex is significantly different from those generated from the diffusion model, especially for Figure 21. This implies the locally-denoising inductive bias cannot well explain the inductive bias of real diffusion model.", "methods_and_evaluation_criteria": "The authors demonstrate the similarity between the proposed patch denoiser and the real diffusion models by comparing the denoiser outputs, which might not be equivalent to the generalization capability. For example, although Figures 13-20 demonstrate PSPC can generate similar denoiser outputs as those of the diffusion models, PSPC fail to generate high quality samples (Figure 21). This suggests there exists a huge gap between the generalization ability of the models and the denoising ability of the models. Due to this gap, the authors should propose alternative metrics for measuring the generalization ability. \n\nAnother issue is the authors only compare PSPC with the Gaussian denoiser. More baselines, such as the closed-form diffusion models and other types of patch-based diffusion models in the paper should be included.", "theoretical_claims": "The theoretical claims have no obvious issues.", "experimental_designs_or_analyses": "I have doubt on measuring the generalization ability of different models using the MSE of denoiser outputs. The detailed reason can be found in the \"Methods And Evaluation Criteria\" section.", "relation_to_broader_scientific_literature": "Previous works have shown that diffusion models have certain inductive bias that enable generalization. The authors try to characterize the properties of such inductive bias. They attribute such inductive bias to the locally denoising operation. The findings are novel.", "essential_references_not_discussed": "I am not aware of any essential references that are not discussed.", "other_strengths_and_weaknesses": "No comments.", "other_comments_or_suggestions": "Are the labels in Figure 8 correct? Which row corresponds to DDPM++ ?", "questions_for_authors": "1. Please clarify the gap between denoising ability and generalization ability. Why the proposed patch denoiser well-matches the diffusion models in Figure 13-20 but diverges significantly in Figure 21? Such gap is not adequately addressed in the current paper. This raises concerns on how much the locally denoising mechanism contributes to the generalization ability.\n\n2. Please compare the proposed denoiser with more (classical patch based) denoisers.  \n\n3. Why PCSC shares the highest similarity with the Gaussian denoiser rather than the diffusion model (Figure 9)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rU8cAxCLsE", "reviewer": "Reviewer_6EJx", "summary": "In this paper, the authors analyze inductive biases of diffusion and relate these to the generalization capabilities of diffusion models. The authors start by examining the network denoiser approximation error, where the approximation error is defined as the deviation of the prediction from the optimal denoiser, where the optimal denoiser equals the weighted average over the images of the dataset. The authors analyze how this difference behaves for different architectures and find similar behavior across them. The authors proceed to show that the generalization of these models arises as a product of locally biased operations, showing this by approximating the operations using patch-based empirical denoisers. Finally, the authors propose PSPC, a training-free denoiser and show that PSPC and the other denoisers exhibit larger similarity between each other than that to the optimal denoiser, as well as that the samples produced with PSPC exhibit larger structural similarities.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## General Comment\nWe'd like to thank the reviewer for their thorough review. We have summarized and responded to the key points of your review below in the limited available space. If we have missed any of your points, we welcome further discussion.\n## Methods & Evaluation\n**Network errors for  $t>30$**\n\nThe reviewer is right. U-ViT & NCSN++ MSE is higher for $t> 30$. As U-VIT predictis $\\epsilon$, we compute $x$ as $x = z - t \\epsilon_\\theta(z, t)$. This scales $\\epsilon_\\theta$ errors by $t$ (see L162). NCSN++ errors fluctuate near $t=40$, but are ~4x smaller than peak errors at t=3. We believe this is a training artifact. Further, Fig. 9 implies high-$t$ errors have little impact on generalization. Both U-ViT& NCSN++ have SSCD similarities > 0.7 with every model, indicating near-copy samples.\n\n**\"Averaging effect\"**\n\nBeyond MSE, Fig. 1, 2, & 13-20 provide evidence that network outputs are qualitatively similar for the same input. To quantify this output similarity, we have measured output cosine similarity at $t=3.2$ for shared forward process inputs [[Link](https://drive.google.com/file/d/1hk6b-m7meFDw6qjw70_579REIpevgLIr)].\n\n**Lack of other networks baselines past Fig. 2**\n\nThis assertion is incorrect. Past Fig 2, Fig 9 and 13-20 all compare various models. However, for transparency and completeness, we will include the following modified versions of Fig 3, 4, and 6 to the appendix:\n- [Fig 3](https://drive.google.com/file/d/1U1x3id-8ECHJugKNPVMouBRyHho4tpsB/) All models have a similar trend between gradient concentration and $t$.\n- [Fig. 4](https://drive.google.com/file/d/1qLbefgMe_FI_dGph9LRPLSbZLcXGamHn/) Patch posterior MSE is similar for all models, except high-$t$ U-ViT.\n- [Fig. 6](https://drive.google.com/drive/folders/1_296T4B96x_r7xAzczftO5YB9PlLZ0F8). We compare our methods against each network baseline. For UViT & DiT, we only have CIFAR checkpoints. \n\nWe are happy to provide additional figures upon request.\n\n**On the impact of attention**\n\nWe do not believe attention layers are the cause of our observations. To test this, we trained an attention-free DDPM++ model on CIFAR. We find the MSE of PSPC is mostly unaffected[[Link](https://drive.google.com/file/d/1H6UrOp4looPt9m8FAVtcuvn4YlGW0Tu9)]. Qualitatively, the attention-free model's outputs are similar to other models [[Link](https://drive.google.com/file/d/1fnjmlGHC77uiXUYaJkOJI2cMkkDmSyO1)]. Similar to our work, Kamb & Ganguli find their patch-based denoiser performs **better** against attention free models.\n\n**On the inclusion of U-Net and BFCNN**\n\nThis paper's aim was to investigate generalization in modern image diffusion models, which overwhelmingly utilize attention layers. In addition to the lack of attention, the networks of Kadkhodaie et al. (2023) have unique properties such as no bias terms, and no $t$ conditioning. Due to these architectural quirks, we believe that analyzing commonly used U-Net and DiT architectures is of greater value to the community than BFCNN.\n\n## Other Comments\n**Q1**\n\nNiedoba et al. (Fig. 3) finds network denoisers are biased for $t \\in [0.3, 10]$, which matches our Fig 2 findings. For clarity, we suggest the change:\n>Across all architectures, we observe similar behaviour to Niedoba et al. (2024), Figure 3 - network denoisers exhibit low MSE for both small and large values of $t$, but substantial error for $t \\in [0.3, 10]$. \n\n**Q2**\n\nWe use \"local inductive bias\" to describe a preference for denoising functions where outputs at a spatial position (x,y) are more influenced by nearby input pixels than distant ones. Goyal & Bengio (2022) define inductive bias as a tendency for learning algorithms to favour solutions with certain properties. Kamb & Ganguli (2024) provide a more formal definition of the bias, calling it \"locality.\" However, since Sec 3.1 mainly motivates our hypothesis on generalization via local denoising, we felt a formal definition was unnecessary. We propose the following changes:\n- Replacing “local bias” with “local inductive bias” throughout\n- Rephrasing the intro of Sec. 3.1 (L142) as: “One potential inductive bias (Goyal & Bengio 2022) of network denoisers is local inductive bias, where denoiser outputs are more sensitive to spatially local perturbations of $z$ than distant ones”\n\n**Q3**\n\nThe networks analyzed have high performance but are not SoTA. Here are their FID scores\n|Model|CIFAR|FFHQ|AFHQ|\n|-|-|-|-|\n|DDPM|1.97| 2.39|1.96|\n|NCSN|1.98| 2.53|2.16|\n|DiT|9.08|||\n|UViT|3.11|||\n\n**Q4**\n\nPixel (x,y) is the output pixel at spatial location x,y where $x \\in \\\\{1, ..., w\\\\}, y \\in \\\\{1, ..., h\\\\}$ for image height $h$ & width $w$. We'll update this notation for clarity\n\n**Q5**\n\nWe'll fix this error.\n\n**Q6**\n\nThe subscript x,y,c denotes indexing at position (x,y) and channel $c \\in \\{1, 2, 3\\}$. We'll reword this to improve clarity.\n\n**Q7**\n\nFor each $t$ value, we draw 10k samples from the forward process $z \\sim p_t(z | x^{(i)}) p_D(x)$, with $p_t(z | x^{(i)})$ defined on L75.", "claims_and_evidence": "I have several significant concerns regarding the paper's claims and methodology. Firstly, I found that many of the authors' assertions are unsupported by evidence or lack clear definitions. One of the primary motivators of the work is the concept of \"local bias,\" but the authors fail to define or reference this term. This omission makes it challenging to understand the paper's main arguments and results.\n\n\n\n\n\n\nIn the following section, the authors do not really provide sufficient motivation and the \"local bias\" is not properly introduced or referenced. Furthermore, as the main claim of the paper is that this inductive bias is similar across various architectures and noise schedulers, it is disappointing that the authors have performed the remaining experiments only using DDPM++. I did not find any analyses corresponding to other architectures in the supplementary material either. \n\nThe biggest concern I have is that many of the effects which the authors analyze (e.g., gradient heatmaps and local behaviour) might be coming from the fact that all four chosen architectures use attention mechanisms. In a related work (which the authors correctly point out) by Kadkhodaie et al. (2023), it is suggested that generalization occurs as the inductive biases of networks interplay well with geometrically adaptive harmonic bases.", "methods_and_evaluation_criteria": "I believe that the problem of understanding generalization in diffusion models requires either more rigorous approaches (from a theoretical perspective) that can be backed up by numerical experiments, or in the case of pure numerical experiments (such as in this paper), much larger and stronger set of evaluations.\n\nFor example, the methodological issues in Figure 2 raise concerns: the authors display the MSE vs . Optimal Denoiser only for time up to t=30, but it is visible that for both U-ViT and NCSN++ the differences start to increase after t=30. Additionally, the averaging of 10,000 samples might suppress differences between the networks. To address this, the authors could focus on averaging paths that are semantically similar, such as averaging only 100 paths that generate an image from a same class, where the class has been obtained using a pre-trained classifier. This would help reduce the \"averaging effect\" and provide a more accurate representation of the similarities of the models' behavior.\n\nFurthermore, the following section lacks sufficient motivation for the \"local bias\" concept, and the authors do not provide adequate analysis or references to support their claims. Moreover, the main claim of the paper – that the inductive bias is similar across various architectures and noise schedulers – is not adequately supported by experiments, as all experiments past Figure 2. only use DDPM++. \n\nAlso, the effects analyzed in the paper (e.g., gradient heatmaps and local behavior) might be attributed to the fact that all four chosen architectures use attention mechanisms. A related work by Kadkhodaie et al. (2023), which the authors correctly cite, suggests that generalization occurs due to the interplay between inductive biases and geometrically adaptive harmonic bases. In my opinion, this paper provides a stronger and cleaner argument towards the inductive biases. But, more importantly, the authors performs analysis using a UNet architecture, as well as BF-CNN, a version of DNCNN network, neither of which use attention. In my opinion, it would be crucial to include either of these architectures (or both) , in order to be able to argue that the methodology in sections 4 and 5 can actually be attributed to generalization and not just attention.", "theoretical_claims": "There were no proofs or theoretical results in this paper.", "experimental_designs_or_analyses": "Please see above.", "relation_to_broader_scientific_literature": "Although the phenomena of diffusion's ability to generalize is of great interest, I find that the contribution of this paper lacks, as well as the validity of their claims.", "essential_references_not_discussed": "I did not find any references to be missing, although I believe that the work by Kadkhodaie et al. (2023) should've been referenced much earlier as they provide an important and significant contribution to explanation of the generalization phenomena in diffusion models.", "other_strengths_and_weaknesses": "Please see above.", "other_comments_or_suggestions": "1. It is unclear why the authors included the sentence \"Corroborating findings of Niedoba et al…\": what are the findings that the authors are referring to here and why is this paper relevant to the problem at hand? The authors should point this out when citing it.\n2. As mentioned previously, in Section 3.1, the authors analyze \"local bias\" without defining it or giving a reference to the definition. I am not aware of what do they mean by this, and this really dampens the strength of the motivation for the section, as well as the subsequent ones. \n3. I think that the authors should include what does \"near SOTA\" mean, and provide some performance of their trained models (e.g., please provide FID) to strengthen the validity of their experiments.\n4. What do the authors mean in line 152 when they write “of the network denoiser output at pixel (x,y)”? What is pixel \"(x,y)\"?\n5. a) In equation (8), shouldn’t the first argument of the network D_theta be z and not x? \n5. b) In same equation, what is the notation x,y,c (in the subscript) supposed to represent? Also, why is there y in equation (8), as from their introduction of the network they use is unconditional and does not take into account information y?\n6. Finally, what do the authors mean by \"drawing 10000 z samples from the forward process\" (around line 150)? Do they mean sampling 10000 z samples and running forward process of 150 steps each time?", "questions_for_authors": "Please see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper advances our understanding of why diffusion models do not simply memorize training data but instead exhibit genuine generalization. Specifically, the authors propose that diffusion denoisers exploit a “local inductive bias,” whereby outputs at each spatial position in an image are primarily shaped by nearby pixels. This hypothesis is intriguing, and the evidence the authors provide is convincing. The consensus among the majority is that the paper’s novel insights, careful empirical investigation, and thorough rebuttal responses justify acceptance.

One reviewer, however, remains unconvinced, expressing a desire for more rigorous theoretical grounding and more extensive tests to confirm that “locality” truly drives the observed generalization.
Decision: Accept (spotlight poster)