Locality in Image Diffusion Models Emerges from Data Statistics
Authors: Artem Lukoianov
Chenyang Yuan
Justin Solomon
Vincent Sitzmann
Venue: NeurIPS 2025 spotlight ()
Keywords: Diffusion models
generative models
ideal denoiser
Primary area: deep_learning
TL;DR: Locality is an important factor in generalization of diffusion models, but it is unclear where it comes from; we show that it comes from pixel correlation patterns in the dataset.

Abstract: Recent work has shown that the generalization ability of image diffusion models arises from the locality properties of the trained neural network. In particular, when denoising a particular pixel, the model relies on a limited neighborhood of the input image around that pixel, which, according to the previous work, is tightly related to the ability of these models to produce novel images. Since locality is central to generalization, it is crucial to understand why diffusion models learn local behavior in the first place, as well as the factors that govern the properties of locality patterns. In this work, we present evidence that the locality in deep diffusion models emerges as a statistical property of the image dataset and is not due to the inductive bias of convolutional neural networks, as suggested in previous work. Specifically, we demonstrate that an optimal parametric linear denoiser exhibits similar locality properties to deep neural denoisers. We show, both theoretically and experimentally, that this locality arises directly from pixel correlations present in the image datasets. Moreover, locality patterns are drastically different on specialized datasets, approximating principal components of the data’s covariance. We use these insights to craft an analytical denoiser that better matches scores predicted by a deep diffusion model than prior expert-crafted alternatives. Our key takeaway is that while neural network architectures influence generation quality, their primary role is to capture locality patterns inherent in the data.

Locality in Image Diffusion Models Emerges from Data Statistics
Artem Lukoianov
Chenyang Yuan Toyota Research Institute chenyang. ywanyuui.ngegeeee global
Massachusetts Institute of Technology artem1@mit. edu
Justin Solomon
Vincent Sitzmann Massachusetts Institute of Technology sitzmann@mit edu
Massachusetts Institute of Technology jsol wellomon@mit.com
https://locality: lukoianov..... com
Abstract
Recent work has shown that the generalization ability of image diffusion models arises from the locality properties of the trained neural network. In particular, when denoising particular pixel, the model relies on a limited neighborhood input image around that pixel, which, according the previous work, is tightly related to the ability of these models to produce novel images. Since locality central to generalization, it is crucial to understand why diffusion models learn local behavior in the first place, as well as the factors that govern the properties of locality patterns. In this work, diffusion models emerges as a statistical property the image dataset and is not due the inductive bias of convolutional neural networks, as suggested in previous work. Specifically, we demonstrate that an optimal parametric linear denoiser exhibits similar locality properties deep neural denoisers. We show, both theoretically and experimentally. that this locality arises directly from pixel correlations present the image datasets. Moreover, locality patterns are drastically different on specialized datasets, approximating principal components of the data 's covariance. We use these insights craft an analytical denoiser that better matches scores predicted by deep diffusion model than prior expert-crafted alternatives. Our key takeaway that while neural network architectures influence generation quality, their primary role is to capture locality patterns inherent in the data.
the
present evidence that the locality in deep
Introduction
Denoising diffusion models 27] have achieved state-of-the-art results for generative modeling, especially for domains involving continuous data distributions such as images, videos, and audio. They are trained to predict a clean image from one corrupted by varying levels of Gaussian noise. Analysis of the diffusion model training objective leads to an apparent paradox: The objective admits unique, analytical, non- non-aarmetriic, and closed-form solution that only function of the training dataset. However, this so-called optimal denoiser does not empirically match the outputs of deep diffusion models and in fact can only produce images the training set, exhibiting perfect "memorization" and failing to generate novel images.
Recent work investigates this paradox and proposes changes the optimal denoiser close the gap between theory and practice [12, 19, 25, 26]. Kamb and Ganguli [12] hypothesize that inductive biases of the neural network
parietuurrrar shift equivariance and locality biases of
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

convolutional neural networks- prevent the model from converging to the global optimum of the loss function and thus allow for generalization. Locality here means that during denoising, any pixel in the denoised output will only be sensitive to local neighborhood around that pixel in the noisy input. They demonstrate that adding locality and equivariance constraints to the closed-fore optimal denoiser yields model that generates images that closely resemble those generated by a simple U-Net diffusion model. However, their theory has a key limitation: it cannot predict the degree of locality from first principles. Instead, their method relies on measaring the receptive field of trained U-Net diffusion model and estimating a patch size for each diffusion timestep that is then fed as a parameter to their analytical model. Further, other neural network architectures, such as transformers, can similarly generate novel images, but they lack either explicit locality or equivarqqiv inductive biases; even U-Nets generally have receptive field that covers the complete image.
In this work, we demonstrate that locality in image diffusion models is not a property of the neural network architecture and instead can be derived directly from the training dataset via simple statistical analysis. Specifically, we analyze the principal components of the data-in in particular, their signal- to-noise (SNR) ratio-and show that learned sensitivity fields for different architectures closely approximate projection operators onto principal components with high SNR. On CIFARIO [16], a dataset with high self-similarity across pixel locations, simple locality pattern emerges. However, on CelebA-HQ [13]. a dataset of centered human faces, sensitivities are nonlocal, aligned with correlations of pixels across, for instance, the eyes. We relate this behavior to the optimal linear denoiser known as the Wiener filter, establishing a connection to prior work that observed linear behavior of diffusion models [17, 29]. We provide further evidence that sensitivity fields are learned to match statistical properties of the training set by showing that we can achieve arbitrary, nonlocal patterns in model's sensitivity field by imperceptibly editing the pixel statisties of the training set.
We rigorously benchmark recent analytical models by how well they match generations by a trained deep diffusion model. Surprisingly, we find that simple Wiener filter oulperforms all recent analyt- ical methods based on modifications of the optimal denoiser. Integrating our analytically-derive sensitivity fields into the model of Kamb and Ganguli [12], however, yields the best- pest-performing analytical diffusion model to date across multiple datasets, including CIFAR [16], AFHQv2 [4], and CelebA-HO [13]. This observation provides evidence that capturing pixel correlations across dataset plays a major role in the performance of denoising diffusion models.
In summary, our contributions are as follows:
We demonstrate that local sensitivities in trained image diffusion models are learned property of deep diffusion models and not just an inductive bias of the model architecture.
We analytically derive the spatial sensitivity of an optimal linear filter as function of the training data and show empirically that it closely matches that learned by a denoising neural network, yielding both local and nonlocal sensitivities depending on the training data statistics.
We establish a quantitative benchmark to measure how well an analytical diffusion model explains predictions made by a U-Net and demonstrate that, surprisingly, prior optimal denoiser-based methods are outperformed by a simple optimal linear filter.
We incorporate our maalyticall-commuue
locality into the optimal denoiser-based model proposed by Kamb and Ganguli [12] and show that it outperforms alternative analytical models while eliminating a previously auurisiiallly-etermiii hyperparameter and the need to measure from pre-trained neural networks.
2 Preliminaries and Related Work
We discuss preliminaries and related work for our primary line of inquiry, building analytical models of deep diffusion networks.
Denoising diffusion models. Score-based image generative models [8, 27, 28] learn to reverse the process of adding Gaussian noise to clean data. During training, we sample a data point zi from the training data distribution X, a noise level / from the interval [0, 1J. and Gaussian noise direction e (0, I); noise schedule or chosen such that og = and 0 = We then add noise to zi to obtain 2t Vaizo + vI - aye. The training objective for an image diffusion model f(z t), also
2

---

known as score-matching, aims to predict zo given ng:
min
[| + at -2o =0012 co [-10,1]
E
(I)
2 X
Recent studies explored the generalization capabilities of diffusion models, highlighting the contra- diction between their theoretical propensity for memorization and their empirical ability to generate novel samples. Yoon et al. [34] introduce the memorization neenoriaaiionneeeraaazzaainn dichotomy, positing that diffusion models generalize when they avoid memorizing training data. Yi et al. [33] formalize generalization through mutual information metrics, demonstrating that trained diffusion models can generalize beyond the empirical optimal solutions. Gu et al. [7] further investigate this phenomenon, revealing that factors such as dataset size and conditioning can influence the extent of memorization in diffusion models.
Analytical difftusion via the optimal denoiser. Multiple works [5, 14, 23, 25] identify the optimal denoiser as a promising analytical model for the behavior of deep diffusion models. This optimal denoiser (ar, 1) minimizing (1) can be written conditional expectation:
f(z, t) B|ro I x]
When the data distribution is approximated with a finite empirical distribution X = {20 [N and due to the fact that we have an analytic form for density p(xx) = N (Varzo, (1 (1-aa) [)) the optimal denoiser is available in closed-form [5, 14, 23, 25]:
(1)
N
f(x,t) = Σ (i t)) w,(x,t) t) = softmax
(3)
20
Vag
JE[N]
where of = (1- denoiser as a kernel-weite weighted average over the training set, and clarifies the key limitation of the optimal denoiser as an appropriate model for deep neural networks: as the noise level approaches zero, the softmax term in effect picks the nearest neighbor in the training set. As result, the optimal denoiser will always generate an image in the training set and never generate a novel image.
and softmax, (ag) E exp(a;) This expresses the optimal
exp(a.)
Improving the optimal denoiser via smoothing. To promote generalization, Scarvelis et al. [25] propose smoothing the score function to generate novel samples different from the training data. Niedoba et al. [18] derive an efficient nearest neighbor search to support the implementation of the analytical score models. Separately, Shah et al.[26] propose smoothing the empirical training data distribution by adding Gaussian noise encourage neural network models to generalize in the small-dat regime. Aithal et al. [1] argue that hallucinations in diffusion models happen due to smooth interpolation between modes of the data distribution. Simply smoothing the score function, however, leads to pixel-space interpolation between training images and does not explain the high-quality, sharp novel images observed in practice.
Adding inductive biases to the optimal denoiser. Kadkhodaie et al. [11] observe that the inductive biases of neural networks align well with the data density, effectively projecting data onto low- dimensional basis adapted to the image structure. Based on this insight, Kamb and Ganguli [12] and Niedoba et al. [19] suggest that the gap between the optimal denoiser and deep diffusion models stems from the inductive bias of the deep neural network used to approximate the denoiser. Specifically, they assume that the structure of the convolutional U-Net [24] imposes constraints of locality and/or shift equivariance on the function f in the score score-mater matching objective eq. (1):
min zomxX
IIf( 100
E
+ vI - /I-ace, t) - xoll2
s.t. ((M E [Q] fgoo,, - go f(x Vgg E T(2),
(locality) (equivariance)
'In practice, often linear combination of and zu is predicted. but these are all equivalent (1) for the purposes of deriving an optimal denoiser and for our theoretical analysis. Without loss of generality, through the rest of the paper, we will assume that all models are trained to predict xq. We provide additional details on the effects of parametrization in Appendices A. and B.4
3

---

where f is pixel q (out of Q total pixels) of the output, Mž is a masking operator for each time-step t selecting patch around pixel q. and (2) is the 2D translation group acting on z E R" through o. They show that for general Mf, the solution for the constrained score-matching objective eq. (4) has a similar form compared to eq. (3), but with weights specific to each pixel & such that
(x,11) = Σ " wig (2, (r ob);, wi (1, 1) = softmax 1.g
{
M2
x ------ h o o VaL
20
JEIN] (2)(2)
N 9ET(2)
where M, is binary mask, and we end up again with an isotropic multivariate Gaussian projected to subspace defined by the mask. Effectively, this model splits each training image into patches of size M,, forgets about their location (equivariance), and denoises each input pixel by taking the average of the center pixels in the ground truth patches weighted with the distances to them from the patch centered at q. Kamb and Ganguli [12] obtain patches for each noise level t by iterating over all possible square binary patches and choosing the one that yields the best correlation with trained diffusion model. Nicdoba et al. [19] relax the constraint on the shape of the patches and allow them to be of arbitrary shape but compact and averaged across all pixels. Then they measure the average sensitivity of a trained U-Net and binarize it to get the masks. Both works fit masks MD to the petiiiialll-obberreeeee
locality fields of the trained models and average all the masks for each pixel i.e. M and Mf" are identical up to a translation.
Linear denoisers. Recent studies have uncovered that diffusion models exhibit strong linear behavior. Wang and Vastola [29, 30] provide theoretical and empirical evidence that, at high noise levels, the learned score functions of well-trained diffusion models closely align with those of linear models. Linear denoisers are well-studied [2, 20, 31], and one can show that the optimal denoiser constrained to be linear has the same form as the optimal denoiser Li et al. [17] demonstrate that, particularly in the generalization regime and for high levels of noise diffusion, denoisers approximate the optimal denoiser for multivariate Gaussian distribution characterized by the empirical mean and covariance of the dataset.
Gaussian dataset [15].
3 Deriving Denoising Sensitivity from Dataset Statisties
In this section, explore the relationship between generalization and locality in patch-based optimal denoisers [12, 19] and link it to the observed linearity of diffusion models [17, 29, 30]. Unlike the optimal denoiser in eq. (3), trained diffusion models exhibit "pass-through" behavior, retaining input information along high signal-to-noise (SNR) ratio data directions. We hypothesize that by constraining the locality in patch-based optimal denoisers, previous works effectively adopt the "pass-through" behavior from linear denoisers and thus are capable of producing novel images.
Sampling voids. The optimal denoiser f(x () is well-defined for all values of x E RR" and I E (0,1), I), but the distribution of (xr, 1) that neural network denoiser is trained on has low-density regions that will be sparsely sampled throughout the training process. For example, when 1. - 0 (low noise regime), regions in R" far away from training data will be undersampled in training the diffusion model with score- mote-iat objective in eq. (1), This is illustrated in Figure I (left), where the regions of small noise near the test images are not covered with any training samples. We will refer to the part of IR. x [0, 1] that is not covered by the empirical samples in eq. (1) as sampling void regions. On one hand, the behavior of the denoiser in those regions is critical for generalization. On the other hand, the optimal denoiser is not a good model of trained diffusion model in these regions, as there were no empirical samples in this part of the space during training. In the next sections, we will build up intuition on how to reason about the behavior of the trained diffusion models in the sampling void regions.
"Pass-throughh denoisers. When the optimal denoiser in eq. (3) is presented with test image outside of the training dataset and the amount of noise small, the softmax becomes more selective, and the optimal denoiser will predict zn to be the closest image in the dataset (see Figure 1, middle). This behavior prevents generalization, as any novel image will be "teleported" to its closest neighbor in the training dataset 0, + 0.

---

(test)
(2)
0
0
0
10
MSE to Xg
10
(1)
Fee'
10-3
Unot
Nose
(test) Z
10
Optimal
10
10
10
10'
Figure I: Left: We visualize the distribution of zt for two training data points 20 and 20 as wegh-probbbility-
(1))
(2)
density "cones", as a function of spatial dimension x: and noise level t. Note how for a new testing point (tea)
there exists noise level t' such that noised versions of 2; are outside of any of the training "cones" and thus the behavior of the denoiser there is undefined. Middle: We take CIFAR 10 test images (top) and add noise frr (2nd
(lest)
row). With single denoising step, trained diffusion model "passes through" most of the coarse structure of the input image, and thus the output image is visually similar to the input (3rd row). Optimal demoiser f instead "teleports" the image to the closest data point in the training dataset (4th row). Right: We compare MSE error of single-step denoising of fe (U-Net) and f (Optimal). At low noise levels, fo removes noise from x. (test)
(les)) but f" predicts a different image from Tb At high noise levels, the outputs of fo and f are similar.
Inspired by the observations in Figure 1, we develop an intuition about "pass-throughh properties of denoisers. For small noise levels, lot of information in the image not destroyed by the added noise. It is natural to assume that "good"" denoiser, unlike the optimal denoiser, will retain this information in its estimation of ro- Which part of Ie is not affected by a small amount of noise? Informally, for natural images, low frequencies "survive" after adding small amounts of noise. We formalize this intuition by observing that 1) adding noise preserves the higher principal components of data, and 2) these principal components correspond to low- www-frequency features in natural images.
The principal components of the data come from the cigendecomposition of the covariance matrix Cov(X)= = U U'diag Xi Xg..... ......., X2 ) where U is matrix of the eigenvectors and X are eigenvalues. The noise's s covariance matrix of also isotropic this basis, so that the signal-to-nois ratio (SNR) along the i-th principal component is ///of. It is well-known in classical image processing literature [10] that for natural images, this eigenbasis approximates the Fourier basis, and thus the highest variance components (equiv. high SNR) correspond to low-frequency features. Thus, intuitively, the "pass-through" projection resembles a low-pass filter. Generally, however, it is not the case, and for more specific datasets, the "pass-through" projection is not just low-pass filter. For instance, as we will show in Section 4, for datasets such as centered and normalized human faces (i.e., Celleb-HO/FFFHE patterns observed in trained denoisers are not translation equivariant, nor isotropic.
the eigen-basis is very different from Fourier basis, and thus the locality
Connection to Gaussian data and linear denoiser. The intuition in the previous paragraphs was built on an assumption that the dataset is well-described with a single covariance matrix (assuming Gaussian data distribution). In this case, one can craft a simple denoiser by just projecting the input noise images to their high-SNR principal components:
W
Udiag
UT.
(6)
Vot
This denoiser, optimal under a Gaussian dataset assumption Zo (O, Σ) [15]. is known as the Wiener filter [31]. At the same time, the Wiener filter also the optimal linear denoiser minimizing eq. (1) under a linear constraint f(xx) A [31]. As we can see from eq. (6), it projects its input to the data's principal components, shrinks these projections according to their SNR, and projects them back to the data space. As reported in multiple previous works [17. 29], trained difffusion models exhibit linear behavior and can be surprisingly well-approximated with a Wiener filter. Later in this work, we extend these observations and demonstrate that the Wiener filter performs par with or better than existing patch- analytical denoisers [12, 18].
Locality and sensitivity. So far, we built up the intuition that "good" denoiser should "pass- through" high-SNR components of the input the sampling void regions-pare of space where optimal denoiser analysis is no longer effective, but critical for generalization. According to [12, 19].
5

---

locality of the denoisers' sensitivity fields plays a crucial role for its generalization. But how does it relate the ppass-through intuition above?
To show the relationship, we return to the notion of locality. By locality, we mean a limited, typically compact sensitivity field of a neural network. Formally, the sensitivity field of denoiser [xx is its input-oulput Jacobian (x, t) = Of /) Kamb and Ganguli [12] approximate learned locality patterns in diffusion models with square patches, assuming them to be compact, roughly isotropic, and constant with respect to both the output pixel position and the input image. As we will demonstrate in Section 4, none of these assumptions is universal, and while they are reasonable for diverse datasets of natural images, the sensitivities of neural diffusers on more specialized datasets can violate any of the assumptions above.
For our analysis, we retain the assumption of independence of the model's sensitivity field with respect the input image x and lift all other assumptions, allowing locality patterns take arbitrary shapes and depend on the output pixel. Under the assumption that the sensitivity S (z., t) is constant and takes the form far, t) A, x + B,, where A is the sensitivity and B, is a bias term. Recall that a solution to eq. (1) under a linear constraint is the Wiener filter and thus As We and B, assuming the dataset is centered. For each individual output pixel q, the sensitivity takes the form:
w.r.t. x, the denoiser is linear
SNR, (SNR, + )
)
A?
$7 -) = W."
Udiag
where SNR,
Vau
In other words, with the assumption that locality patterns are shared for all input images, the sensitivity of the denoiser is identical to the high-SNR projection operator to the principal components of the covariance matrix. A key observation about the form of eq. (7) is that as + 0 the signal-to-noise ratio for each component SNR; + inf and thus S (az t) + 1g. the indicator function pixel q; i.e. the sensitivity field of the locally linear denoiser shrinks with smaller noise levels.
Under the local lincarity assumption, the exact shape of those sensitivity fields is a function of the data and not of the model's 's architecture. In the subsequent sections, empirically show that different architectures of denoising diffusion models learn sensitivity fields similar to those of linear denoisers, effectively approximating the projection operator to the high-SNR data's principal components.
Recall that previous work [12, 19] adopts the learned sensitivity fields from the trained diffusion networks, which play a crucial role in their generalization. As we reveal the connection of the sensitivity field to the local data statistics, we obtain valuable insights into the performance of the analytical models, especially for specialized datasets, where the data's principal components are far from being local and equivariant.
4 Validation
In this section, we perform extensive validation to support our claims. As the main backbone for diffusion model, we use the DDPM U-Net [8] with removed self-attention to follow the protocol of [12]. As we show in Appendix B.2, removing the self-attention does not drastically affect performance, and our insights are valid for both architectures.
We begin by showing that different neural network architectures learn similar sensitivity fields, which in turn match projection operators to high-SNR principal components, or equivalently, the Wiener filter. We continue by demonstrating that the learned sensitivity fields are property of the dataset and thus by manipulating the statistics of the data, we can force the diffusion model's sensitivity to take any shape, including being nonlocal for low noise levels.
Finally, to support our claim that the locality properties of trained diffusion models come from data statistics, we suggest a simple modification to previous patch-based analytical models [12, 19]. Instead of measuring locality from trained diffusion models, we limit the analytical model only use high-SNR principal components. We benchmark this modification on five datasets and show that while being more interpretable, this algorithm also explains trained diffusson models better than other baselines. Additionally, we ablate our model and present comparisons in Appendix B.I.
Locality pattern is shared across architectures. We compare the locality patterns throughout the denoising process across architectures (U-Net [8] and diffusion transformer (DiT) [21]) trained on

---

950
800
650
500
350
200
50
10
DiT
0.8
0.6
Unet
0.4
High SNR pro).
002
0.0
Comparison of sensitivity fields of deep denoisers and the projection operators high-SNR data's components (i.c. the Wiener filter) on CIFAR-10 dataset. Sensitivity is measured at the center pixel w.r.t 2o prediction and throughout 1000-step DDIM denoising process. Each image is averaged across 32 samples and normalized to [0.1].
Figure 2:
the CIFARIO 10 dataset. Although there is architectural bias for locality in U-Nets due to convolutional layers, the self-attention layers DiTs are global in scope, where every patch can attend any other patch. Nevertheless, Figure 2 shows that U-Nets and DiTs share similar sensitivity fields. Surprisingly, these fields are similar the shapes of high-SNR projector operators, i.e., the sensitivity of a Wiener filter. This provides evidence that the main reason for the diffusion models to exhibit locality properties the correlation of the pixels between the images the dataset. Our observation is that although the neural network architectural choices are important to accurately capture data statistics, they are not the main cause locality patterns in diffusion models.
Learned sensitivity fields are not always equivariant. As we saw in the previous experiment, the sensitivity fields learned by diffusion models similar across architectures and align with the data's 's principal components. In Figure 2, these fields appear roughly isotropic and equivariant mennariinttmmmannngg the sensitivity s" (xx) has the same shape for each pixel q. up translation. This form of sensitivity is well-captured by the square-shaped patches [12].
This behavior arises due to the high correlation among neighboring pixels the translation equivariance inher- ent in the CIFARIO 10 dataset, which consists of diverse natural images.
However, for more specialized datasets, the principal components-ann consequently the learned sensitivity helds-take drastically different shapes. In particular, CelebA-HQ is dataset of uniformly-scaled and centered human faces. The lack of translation equivariance and unique pixel correlation patterns result in structured, location-dependent sensitivity fields.
Average sensitivity fields of trained DDPM the CelebA-HQ dataset. The top corresponds
Figure 3:
In Figure observe the complex structure that arises in the sensitivity fields of diffusion model trained on CelebA-HQ. Notably, the pattern of sensitivity is now highly dependent on the pixel's location. This experiment highlights the need for more flexible representations of sensitivity fields, especially for specialized datasets, than those in prior work.
output pixel located near the left cye; the bottom row responds to an output pixel near image center. Left to right: different noise levels corresponding to i of 600, 400, 200.
Manipulating the sensitivity field. We show that by editing the dataset's statistics, we can manipu- late the sensitivity fields of neural denoisers and make them take any shape. With this manipulation.

---

U-Net-based denoisers can learn sensitivity fields that are not local, thus suggesting that the locality properties learned denoisers emerge from dataset statisties.
In our experiment on CIFAR-10, we generate modified dataset:
So To + 7cs,
c Unniform((11111))
where zo E Rd is training image, 8 {0, 1] is fixed binary mask in the shape of the letter "W", and e E IR's is random RGB vector (single color per image) with y > 0 controlling signal strength. Note that this transformation does not change the first-order moments of the data [[[cs] 0.
With this, we train a new DDPM U-Net from scratch on the modified CIFARIO dataset and then study its sensitivity fields. Let AE be the variance of the perturbation. We choose 7 that A. 0s, for some intermediate t., i.e., the variance of the added signal matches the variance of the noise. As we see from Figure 4, the "W""shapeds sensitivity field emerges for all t where de < \w- Crucially, this demonstrates that any desired pattern can be induced in the sensitivity of a trained neural network by embedding the pattern into the data covariance.
950
800
650
500
350
200
50
++.
s sW
Figure 4: We slightly manipulate pixels' correlations across the CIFAR-10 dataset such that a desired pattem emerges in the sensitivity of a trained diffusion model. In particular, a DDPM diffuusio model trained on the CIFAR-10 10 dataset (sample on the top left) has coarse-to-fing sensitivity field (top row, noise level decreases from left right). For each image the dataset, cdit pixel correlations by adding the desired pattern with random color and weights y manipulated datasets exhibit the pattern in their sensitivity fields. We underscore the time-steps for which SN Rw > 0.1, i.c. Xiy > 0.1o?. This supports our claim that the locality in diffusion models arises not from the inductive bias (i.e. usage of convolutional layers) but from the data statistics.
(middle row) ? 0.5 (bottom row), DDPM models trained on those
Our model. Previous work uses rectangular binary masks fitted to the sensitivity fields of trained denoisers. In this paper, demonstrated that the sensitivity fields of the trained denoisers emerge from the data covariance. We consider constant sensitivity fields, i.e., A7 is not a function of x. If the sensitivity field constant, the denoiser is linear, and know that the optimal linear denoiser is the Wiener filter. Using this intuition, we consider a generalized notion of locality and show that the locality property is equivalent to subspace projection, which can be written as an orthogonal change of basis followed by a masking operator. We provide a detailed derivation in Appendix A.3.
Replacing the measured sensitivity field as in [12, 19] with the projection operator to high-SNE -NR components (sensitivity the optimal linear denoiser) performs on par with or better than the patch-based optimal denoisers. More formally, suggest using the following analytical model:
wf softmax {-
f(x,t) Σ (I t)a6
1
W
(10, V
(8)
200
JEIN]
where W is a q-th row of the Wiener matrix binarized with threshold - 0.02 (we are using = 0.02 relative to the max value in the row unless stated otherwise). For MNIST and Fashion
2To aid visualization, we apply the square root to the sensitivity field and plot it with "turbo" color map. In the rest of the paper, the color map applied the raw signal.

---

CalchA-IIQ
CIFARIO
AFHOO
MNIST
FashionMNIST
8
0
NN
8
Ours
g
Wiener
Komb
8
Dataeet
9
Figure 5: Qualitative comparison. In this figure, we compare our analytical model (3rd row) with multiple
baselines: Wiener filter (4th row), Kamb and Ganguli [12] analytical model (5th row). All images are generated
with the same initial noise sample with 10 steps of DDIM [27]. In the top row, we provide the results of generation with two trained neural networks, NNI and NN2 both are instances of the same DDPM U-Net [8], but trained with different seeds. The distance in Table is measured with respect to NNI. In the last row provide the nearest image from the dataset for our final generation w.r.t. L2 distance.
MNIST we use r = 0.005. We provide a detailed derivation of this formula and the ablation of T in Appendices A.2 and B.1. Key differences of our model compared to [12] and [19] are:
1. Enhanced interpretability. Instead of fitting patch sizes and/or shapes to trained models, we obtain them analytically from dataset statistics.
No equivariance. While prior work claims equivariance as an important property of diffusion models [12, 18), instead find that introducing the translation group integral to Equation (8) does not improve performance but increases inference time.
3. Locality specific to each pixel in the image. Unlike prior work, we do not enforce the shape of the patches to be shared across all pixels of the image, instead relying on the dataset's 's statistics. This is particularly important for datasets with nonlocal covariances, such datasets of faces.
Our model nonlinear and does not assume that the dataset Gaussian. Rather, it approximates only the locality fields with the second-order statistics of the dataset. We benchmark our analytical model by measuring the -cecofffii of determination and gqean-suurred error (MSE) between the predictions of the analytical model and trained DDPM [8] given the same starting noise.
For comparison, we chose five datasets with diverse sets of statistics: CIFAR10 [16], a dataset of diverse 32 x 32 natural images; CelebA-HQ [13] and AFHQv2 [3], datasets of centered faces and animals in 64 64; and MÄIST [6] and FashionMNIST [32], datasets of binary centered images in 28 x 28 resolution. We compare against the vanilla optimal denoiser in eq. (3), the Wiener filter [31], and the patch-based optimal denoising algorithm by Kamb and Ganguli [12]. Additionally, to capture the variance in signal-to-noiss to--ossee mapping of diffusion models, we compare with another trained diffusion model with the same architecture and dataset, but with different weight initialization.
Table and Figure 5 show that our analytical model outperforms all of the baselines, with the Wiener filter being almost always in second place. Kamb and Ganguli's is model qualitatively performs worse on CelebA-HE HQ due to patch-base locality erasing eyes and blurring out facial features, while other models using dataset-dependent locality retain these features. This experiment confirms our hypothesis that the modeling of correct locality in the analytical models is key to explaining trained diffusion models and that those localities come from the dataset statistics. We provide additional quantitative results for AFHQv2 and Fashion-MNIS" in Appendix B.7.

---

Table I: We provide quantitative comparison that measures how well each analytical model explains the trained image diffusion models. All metrics are averaged over 128 samples. Best results are highlighted in green and second best maroon.
CIFARIO
CelebA-HI
MNIST
Method
22
MSEI
MSE
MSEJ
Optimal
-0.549100.774
0.19110.004
0.400.10.298
0.101
0.187:10.204
0.231110.036
Wiener (linear)
0,408=0.092
0.032=0.004
0,818#0.039
0.031 004
0,469=0.066
0.161#0.014
Kamb [12]
0.303=0.126
0.065=0.017
0.831=0.073
0.028=0.000 0.00
0.402=000022
0.188=+0.039
Ours
0.589=0.078
0.028=0.008
0.902::0.022
0.016=0.0006
0.491=0.051
0.153+00.015
Another DDPM
0.852:10.113
0.023++0.002
0.981+-0.007
0.004:10.001
0.969:10.082
0.007+L0.019
Conclusion, Limitations, and Future Work
In this work, we demonstrated that locality in diffusion models emerges from dataset statistics rather than architectural inductive biases. Through theoretical analysis and empirical validation, we showed that both U-Nets and Transformers learn sensitivity fields that closely align with projections onto high-SNR principal components of the training data. This intuition links the behavior of the diffusion models to linear denoisers, equivalently, the Wiener filter. Using theoretical insights, show that by editing the dataset's ss statistics, we are able to manipulate the sensitivity fields trained diffusion models and can make them take arbitrary shapes, including highly nonlocal ones. Finally, our analytical model, based on dataset statistics, outperforms previous approaches in approximating trained diffusion models across multiple datasets.
This work addresses a critical gap in understanding how diffusion models generalize rather than memorize, highlighting the ppass-through behavior where high-SNR components preserved. Limitations of our approach include a focus on simpler architectures and reliance on second-order statistics, while deep diffusion networks can capture higher-order statistics of the data. In particular, we are making a strong assumption that the locality fields are constant with respect to the input images. Studying non-linear regimes of the neural diffusers deepen our understanding of the mechanisms of image generation. Future work on complex architectures, higher-order statistics, and conditional generation has the potential to further explain the
gap in diffusion models.
Acknowledgments
The authors thank Christopher Scarvelis, Ishaan Chandratreya and Vitor Guizilini for their thoughtful insights and feedback.
Vincent Sitzmann was supported by the National Science Foundation under Grant No. 221 1259, by the Singapore DSTA under DSTM00C122030223 (New Representations for Vision, 3D Self-Supervised Learning for Label-Efficient Vision), by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/ Interior Business Center (DOI/IBC) under 140D0423C0075, by the Amazon Science Hub, and by the MIT-Google Program for Computing Innovation.
The MIT Geometric Data Processing Group acknowledges the generous support of Army Research Office grant W91INF2110292 of National Science Foundation grants ISS2355992 and OAC2403239, from the CSAIL Future of Data and FinTTchAAI programs, from the MIT-IBM Watson AI Laboratory, from the Wistron Corporation, from the MIT Generative AI Impact Consortium, from the Toyota- CSAIL Joint Research Center, and from Schmidt Sciences.
10

---

References
[1] Sumukh Aithal, Pratyush Maini, Zachary Lipton, and Zico Kolter. Understanding hallu- cinations in diffusion models through mode interpolation. Advances in Neural Information Processing Systems, 37:1 34614-13466444 2024.
[2] Robert Grover Brown Brown and Patrick YC Hwang. Introduction random signals and applied Kalman f with MATLAB exercises fourth ed. Wiley & Sons,, 2012.
[3] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. StarGAN v2: Diverse image synthesis for multiple domains. In 2020 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR). IEEE, 2020. doi: 10. 1009///pp4220000200000022 URL
attp://dx doi. .oo.orgg///ooooooooooo.o.o...... 10. 1109/ wwt.:1111.veepeee cvpr42600..... 2020 00821.
[4] Yunjey Choi, Youngjung Uh, Jacjun Yoo, and Jung-Wor Ha. StarGAN v2: Diverse image synthesis for multiple domains. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 6 2020. doi: 10.1 119/cypp42600.2222000088 http://ás doi
URL
00821.
[5] Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. arXiv preprini 20Xiv:2208.053344 2022.
[6] Li Deng. The MNIST database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 2916):411114,, 11 2012. ISSN 1053-5888. doi:
10.1 10.1109/msp.22 20122.2 1477. URL http: hoi://xxdoooio.. org/10. 10. 1109/msp..... 2012. 22111777
[7] Xiangming Gu, Chao Du, Tianyu Pang, Chongxuan Li, Min Lin, and Ye Wang. On memorization in diffusion models. arXiv preprint arXiv:2310.02664, 2023.
[8] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 118800-651,,, 2020.
[9] Hugging Face. Diffusers library documentation. s..ffuses/eeninnee 2024. Accessed: 2025-05-23.
[10] Aapo Hyvärinen, Jarmo Hurri, and Patrick O Hoyer. Natural image statistics: A probabilistic approach to early computational vision., volume 39. Springer Science & Business Media, 2009.
[11] Zahra Kadkhodaie, Florentin Guth, Eero Simoncelli, and Stéphane Mallat. Generalization in difffusion models arises from geometry-adaptive harmonic representations. arXiv preprint arXiv:2310.02557, 2023.
[12] Mason Kamb and Surya Ganguli. An analytic theory of creativity in convolutional diffusion models. arXiv preprint arXiv:2412.20292, 2024.
[13] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017.
[14] Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35: 26565-26577, 2022.
[15] Steven M Kay. Fundamentals of statistical signal processing: estimation theory. Prentice-Han Hall, Inc., 1993.
[16] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[17] Xiang Li, Yixiang Dai, and Qing Qu Understanding generalizability of diffusion models requires rethinking the hidden gaussian structure. Advances in neural information processing systems, 37:57499-57538. 2024.
[18] Matthew Niedoba, Dylan Green, Saeid Naderiparizi, Vasileios Lioutas, Jonathan Wilder Lav- ington, Xiaoxuan Liang, Yunpeng Liu, Ke Zhang, Setareh Dabiri, Adam Šeibior, et al. Nearest neighbour score estimators for diffusion generative models. arXiv preprint arXiv:2402 08018, 2024.
11

---

[19] Matthew Niedoba, Berend Zwartsenberg, Kevin Murphy, and Frank Wood. Towards a mecha- nistic explanation of diffusion model generalization. arXiv preprint arXiv:2411.19339, 2024.
[20] Alan V Oppenheim and George C Verghese. Signals, systems inference. Pearson London, 2017.
[21] William S Peebles and Saining Xie. Scalable diffusion models with transformers. 2023 IEEE. In CVF International Conference on Computer Vision (ICCV), volume 4172, 2022.
[22] Frank Permenter and Chenyang Yuan. Interpreting and improving diffusion models from an optimization perspective. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 40461 10661144443..0. PMLR, 21-27 Jul 2024. URL https: //proceedings.. mlr. wwwppprrrsssseeeeeeeee /ttp//////eeeeee22/5555 html.
[23] Martin Raphan and Eero P. Simoncelli, Least squares estimation without priors or supervision. Neural Computation, 23(2):374-420. 2 2011. ISSN 1530-888X. doi: 10.1162/neco_a a a_006666._. URL http://dx.doi. doi. org 10. wweee111eeeeeee 110.11110///eco_000777.
[24] O. Ronneberger, P.Fischer, and T. Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical Image Computing and Computer-Assisted In- tervention (MICCAI), volume 9351 of LNCS, pages 234-241. Springer, 2015. URL http://lmb/ inf informmiik.....mm uni- maai-frebuurg... de//ublications//n0 200.aaaaiioos/22055//5/555.... (available on arXiv:1505.04597 [cs.CV]).
[25] Christopher Scarvelis, Haitz Sáez de Ocáriz Borde, and Justin Solomon. Closed-form diffusion models. arXiv preprint arXiv:2310.1223 12395,, 2023.
[26] Kulin Shah, Alkis Kalavasis, Adam Klivans, and Giannis Daras. Does generation require mem- orization? creative diffusion models using ambient diffusion. arXiv preprint arXiv:2502.21228. 2025.
[27] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[28] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[29] Binxu Wang and John Vastola. The hidden linear structure in score-based models and its application. arXiv preprint arXiv:2311. 2023.
[30] Binxu Wang and John J Vastola. The unreasonable effectiveness of gaussian score approximation for diffusion models and its applications. arXiv preprint arXiv:2412.09726, 2024.
[31] Norbert Wiener. Extrapolation, Interpolation. and Smoothing of Stationary Time Series: With Engineering Applications. The MIT Press, 8 1949. ISBN 9780262257190. doi: 10.7551/ mitpress/2946.001 .0001. URLhttpp: //xx. doi
http:////ttttts///.e 001p://///226000000.0.0 001.0001.
[32] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: arXiv:1708.077:0 0778.07777777 2017.
[33] Mingyang Yi, Jiacheng Sun, and Zhenguo Li. On the generalization of diffusion model. arXiv preprint wwX.iv:2305.17710 2023.
[34] TaeHo Yoon, Joo Young Choi, Schyun Kwon, and Ernest K Ryu. Diffusion probabilistic models generalize when they fail to memorize. ICML 2023 workshop on structured probabilistic inference & generative modeling, 2023.
[35] Bohao Zou. Denoising diffusion probabilistic model (DDPM) implementation. https: / cessed: 2025-05-23.
2022. Ac-
12

---

Partl I
Appendix
Table of Contents
A Derivations and proofs
14
A.1 Optimal denoiser: derivation, equivalence of e and To parametrization
14
A.2 Patch-based optimal denoiser: formal derivation
15
A.3 Ours: why binarize the sensitivity field
17
A.4 "Pass-through" denoisers: detailed analysis of SNR
A.5 Manipulating the sensitivity field: variance of the perturbation
20
B Additional Experiments and Ablation
21
B.1 Ablation of our model
21
B.2 Self-attention layers in denoising U-Nets
21
B.3 Low rank projection of the covariance matrix
.er.............................
22
B.4 How to reproduce reported sensitivity fields
23
Sensitivity field the optimal denoiser
24
B.6
Generation dynamics
24
B.7
Quantitative results for AFHQv2 and Fashion-MNIS" ................................
25
B.8 Quantitative measure of novelty of samples
25
B.9 Additional generation results
26
Implementation details
27
Sampling.
................................
27
C.2 Training DDPM Model
................................
27
C33 Our analytical model
27
C44 Baseline implementation details
28
C.5 Algorithmic Complexity
28
C.6 Computational resources and runtime
29
13

---

A Derivations and proofs
In this section, we provide detailed derivations and proofs for the background and the claims made in the paper.
A.1 Optimal denoiser: derivation, equivalence of e and To parametrization
We begin by defining the optimal denoiser for the zo parameterization use in the paper.
Definition A.1. The optimal denoiser fxx t) for a data distribution X at a particular noise level is the minimizer of the loss function
min
[[(Var vi-oue, -00l2
ZaNX c~N(O.1)
Recall that of 1-0e or
Proposition A.2. When X = (#0) LE[N] is finite empirical distribution, the optimal denoiser (xx(t) has the following analytical expression:
1
f(x,t)
Σ x softmax
2
7
(10)
20
vae
Proof. We first write down the objective (9) in terms of the random variable
vagi0 vI a,e:
[|| [[I(var + v1 a,c,t) =moll²]
E
xp~ e(.1)
1lf t) xool
N -a
exp (-1100-
E x
- (Va)(1-ar))
=2
x
1207 [1((x,))
xall dx
Voa
We then minimize the integral coordinate-win wise for each get the optimal f(x,
0 = X
exp
ro
x
120
(f(I,t)
=2o)
Va
Σe 2b exp(
x Vac
x
/20
fx,
Σxexp(=
x Vai
120
Using the definition of softmax, {astsE[N]
exp(as) Ejen exp(a,)"
we get (10).
Definition A.3. The optimal denoiser ε(2, t) for data distribution X at particular noise level t is the minimizer of the loss function
min
[[[(Var + -1- 1) - -el12
(11)
X coN(O.)
Proposition A.4. When X = {=o) Ve(N] is a finite empirical distribution, the optimal denoiser for
the e-parameterization ē(x, t) can be written in terms of that of the z-paaramterrizttion f(z, t):
(x,
8(x, 1)
(12)
vI
14

---

Proof.
We follow the same proof as Proposition A.2, with the main difference being the following
step:
[|[(va vI -cc, -c11]]
2a coN(),1
va,20
E
[(x,t)
xo B=N[yarzo.(1aaa)
v1 ar
Remark A.5. Another way to prove Proposition A.2 and Proposition A.4 is to show that the optimal solutions are of the form E(no æ] and clear that the two expressions are linearly related to each other.
and TEle z:}, where N (Vvatzo (1 a.) Then becomes
A.2 Patch-based optimal denoiser: formal derivation
We now turn to the patch-based denoiser, incorporating both locality and equivariance constraints into the optimal denoising problem as suggested in [12], repeating the derivations in the notations of this manuseript. Let { be finite empirical distribution of images, and let
M2 IR dxd dxd
denote the operator that masks out p patch centered at pixel setting the rest of the pixels to 0.
As suggested in [12], impose constraints on each patch-wise function, f
Locality:
(xx,t) (3MM x, 1).
Equivariance: For every 2D translation g T()),
f(g x, t) g f(x,
(0002, (xx,
i.e. denoising commutes with the action of T(2) and relocates patches accordingly.
Definition A.6. The patch-based optimal denoiser f(x, for data distribution X at particular noise level is the minimizer of the loss function
min 1 z0~X, (~N(0,)), (~100,1 |I
E
ai (Var
v1 cre, 1) -xol22
s.t. (z, = (M] t), q q=.....,Q.
(locality)
(13)
f 1) = o f(r,t; (g)
(equivariance)
Proposition A.7 (Patch-based optimal denoiser). Under the empirical distribution X = fard) the minimizer { eq. (13) given, for each patch location q, by
softmax (-
))))
(z, Σ
Σ
(g xg)
(14)
207
Vat
gET(2)
and the full- full-image denoiser is obtained by reconstructing the final image from the pixels above.
Proof of Patch-based optimal denoiser. We prove this result in three steps: (1) decomposition into per-pixel optimization, (2) equivalence of equivariance constraint and data augmentation, and (3) derivation of the local form.
Step 1: Decomposition into per-pixel optimization. Let Pg Rdxd _ denote the operator that extracts pixel from an image, and define Σ9 Pg Pyp where p3 places a scalar value at pixel q and zeros elsewhere. Since pixels are disjoint, P.P. for o j. making (P,P?)) orthogonal projections with Ed P.P.T 1.
15

---

By orthogonality of pixel projections:
Q (f(()) - xo)
fI(()) - 2012
Σ
9=1
Σ [P)f(x) - Porol2
Q Σ (x)
1=1
Therefore, the original minimization problem decomposes as:
min Niin + vI- -zoll
min Bap. ( + v1 - (1)e)e)= - =x/]
Each pixel can be optimized independently.
Step 2: Equivariance constraint equals data augmentation For the q-th pixel problem with equivariance constraint:
minf1 B.0-X.o~N(0.1) [f + -I-aat)) =2]]
(15)
s.t. (g)o)) = (g0 =( f())) Vg T(2)
The equivariance constraint implies that for any translation g: (g = f0 (x)x
Now consider the data-augmented problem:
min Beg B.g. [f o (Vaiso + vI at)) - (g 20))
(16)
We want prove that (15) is equivalent to (16). To do so, first show that the optimal solution of (15) does not change with data-augmentation, then show that the optimal solution (16) satisfies the equivariance constraint.
Constrained optima invariant under data-augmentation Let x Voizo VT- agc. Since
translation commutes with noise addition and (g xo)² = = "9 (15) under aata-augmentation becomes:
min P.oo.x..--N00118Bxx110 ( o x) = '01² f
f Vg T(2)
fo satisfies the equivariance constraint, then (g fa 1a (z), so:
arg min B.o. ()) - 0) = arg min E Exprg
(x) - -x0 "0)
fa
fa
Q
arg min Ezg,e
Σ (f(())223)
1=1
arg min E Exg, ())))18))
16

---

Data-augmented optima is equivariant We can solve the data- augmented problem (16) and check that its optimal solution fo is equivariant under the transformation - h o for any h E (2)):
N
x)) =
(g o 2 softmaxi,g
c x go 2i
20
i=1 gET(2)
N
(g o ) softmaxi,g g
x
1
#0
20,
Vaa
(ET
N
=
Σ
Σ
(ho o' softmax:.
-0,
Vai
g o 2i
i= g'ET(2)
f())))
where g' -h------ ogg.
Step 3: Local form derivation. From the data augmentation equivalence, the optimal denoiser for pixel minimizes:
[ xo) + v1 - aic) (g o zo x0)
Let / aag o vn
vI are. Writing the objective as an integral over æs we get:
ate.
ai)) a=)) exp (- (-190-0
BxgxX (2T(1 a1))
x 04
/20 (g dz
With the locality constraint f (x) = (M))x) we can "integrate out" the coordinates of x that are masked out by M1 (as fr does not depend on them), get:
-x (2:)) exp (-
M xo (
2 1202 (f9(M))) - (g xo) d(M}z)
-
Vat"
Solving for the optimal fr, we get:
|a MR - M2 =
(1
=
exp
X gT2(22) 6~ (O.T)
20,
Rearranging:
((M)) EN E Eger(2) (0 o/)3 (- Mºx M2 o 26))13/200?
va:
Σ/ EnET(2) exp (-1 M2x M2 (h x/)) 12/03
Using the softmax notation:
(x, Σ Σ (g (goz/)) softmaxi.g
-
1
M³r Mt (g ag)) m3)
20
Vai
i=l gET(2)
This completes the proof.
A.3 Ours: why do we binarize the sensitivity field
In this section, provide justification for our algorithm provided in the main paper. In particular, we formally justify why makes sense to binarize the sensitivity fields into a mask zeros and ones.
At first, we generalize the patch-based optimal denoiser by relaxing the locality constraint. Instead of restricting to patch extraction operators M", we consider linear operators A Rddd -> ddd that can capture more complex spatial relationships.
17

---

Definition A.8 (Generalized masked optimal denoiser), The generalized masked optimal denoiser far 1) for data distribution X at noise level is the minimizer of:
min
E X e~ [1 [V To + -00 t) zoo12
(17)
s. (a, = [ 1), q= q=1,...,,2
(generalized locality)
[1g o 10 - (x, Vg E T(2)
(equivariance)
where A7 IRdxd + Rddd is linear operator.
Note that when Ag invertible, f is effectively unconstrained (i.e. there is locality). On the other hand, when A? = 0, f does not depend on æ. The following result interpolates between these two extreme cases, showing that the optimal denoiser depends only the row-space A,-
Proposition A.9 (Generalized masked optimal denoiser). Following the decomposition and data augmentation equivalence from the previous section, the optimal denoiser for pixel q under the generalized locality constraint is:
(9(x,
Σ GET(2)
N
(go xh)a softmax, ,
{-1 20
Pd
(Var
1
g o
)rr)
(18)
where P. = (A?)* (A)())))) Ag is the orthogonal projection matrix onto the row space of Aq.
Proof. The objective (17) can be written as
/ (2m(1 - ax)) -d cxp (-1100 g 20
x var
200
(f*(x) x0)9) dr.
#o.9
We SVD write A T AV, and notice that P. VAIVT, where At is diagonal matrix with or diagonal entries depending if the corresponding diagonal entry of is nonzero. Then we impose the constraint (x) fo (AAxr) and perform change of variables y Vx.
(-110e00
1
(2m(1 az)) exp
x
1207
(f9(Atz) - (g 20))) dr
20.9
Vot
= -f
E (2 az)) dd exp 20.9
(-)
1 Vav
/202 (F((N)) - (g 20) dy,
where have absorbed U into 1 is invertible matrix. Next, we integrate the coordinates of y corresponding to the zero-values in A, and absorb non-zero entries of A into f", to get
E (2 as))) exp (-|N (-IN(vloom)) (vloo zo) 20.9
1) (PM(TT)) - (o o xx) dy.
1 Vae"
/20?
We then solve for the optimal (y) and change variables to get (x)::
1
(g =1)² softmax
(y V o z661
20
Va
1=1 9ET(2)
Σ
(g o x
9
softmaxi.g
A
tv
(x.
20
vat
1=1 9ET(2)
1
9 s/ |1}
Σ
(g 2 softmax,,0
pd
207
var
9ET(2)
18

---

Corollary A.9.1 (Diagonal operators and mask binarization). When A = diag(a?) is further constrained be diagonal matrix with entries a.. the optimal denoiser simplifies to:
N
xg)) softmax, {
-
()x,1))= = Σ Σ
(gox))) gET
13 Φ Vaa
- g good)
(19)
20²
where Ba is the binary mask with B = if a + 0 and B = otherwise, and @ denotes element-wiss multiplication.
Proof. We begin with the generalized optimal denoiser from equation (18) and substitute the diagonal operator A7 diag(a;, az..... ........... .g)).. Then P. (A4)² T (A] mask because oefficients a, cancel out completely when aj * The actual values of non-zero a} do not affect the optimal denoiser-only whether a² = or a o 0 matters. Then we can apply Proposition A.9 to get
(A4)7)) A B", We get binary
N (x, Σ 2g))))) softmax 2=1 92T(2)
(1_
)
B o
y xz
20
Vor
This completes the proof, showing that the optimal denoiser depends only on the binary support of the diagonal operator, not on the specific non-zero values.
Connection to patch-based denoiser: When af O, the corresponding pixel j is effectively removed from the optimization, it contributes zero to the distance metric. This is precisely the locality constraint from the patch-based denoiser: pixels outside the patch (where a² 0) do not influence the denoising of pixel q.
Remark A.10 (Justification for binary masks). In summary, we showed Proposition A.9 that the only interesting generalized locality matrices are binary masks. In particular, when the masking operator Ai has diagonal structure, the specific values of the non-zero entries cancel out the softmax computation. This means that:
1. The optimal denoiser depends only on the pixels that included in the mask (the support), not their relative weights.
Binary masks (0, 1} are expressive as any diagonal weighting scheme for this optimization problem.
This theoretical result justifies our practical choice of binary masks in the main paper, as more complex weighting provides no additional benefit for the optimal denoiser.
A.4 "Pass-through" denoisers: detailed analysis of SNR
In this section, we provide detailed analysis of the signal-to-mn to--sosee ratio in the principal components of the data, extending section "Pass-through" denoisers in the main paper. Let's consider the data matrix X = [ .......) x E RdxN Doing singular value decomposition, and assuming N 2 get X U diag(A1, (X1, .. >.)V where X; sorted in the descending order of their absolute values. Covariance of the dataset, assuming that the mean of the dataset zero:
1 N
Σ XXT = Udiag 12, A.)UT. N
where U are the principal components of the data and X is the variance of the data along those components. We can now compute the signal-to-noise ratio along each of the principal components of the data:
19

---

B (UT var.00) )²
SNR,
Be-N((.)))
(U7 VI oxte)
0.¹ EU:
(1-ar)- UU?Up
-2/NN
1 Ott.
X
Not
When A > No;, the intrinsic data variance is much larger than the relative noise level, and the signal was not "destroyed" by noise. Note, the analysis above does not have be performed on the entire dataset, but rather on the most relevant set of neighbors to the image that is currently being denoised. In that case, the high SNR projections will more precise and specific to each particular image as long as SVD is well defined. Due to computation constraints and to keep the analysis simple from now on we will assume that the covariance matrix computed on the entire dataset.
A.5 Manipulating the sensitivity field: variance of the perturbation
In this section, we provide the derivation for the variance the added perturbation Xw in the section "Manipulating the sensitivity field" of the main paper. Denote by tycs the signal vector; then the empirical covariance of the modified data is
Emod = _[
B[( + (2o + rcs)²1
=
E[ro=]] + y Bloos + =11 + B[css" cT)
Eoris + 72 E[ce 1ss""
Eorig + + 1
3 Is 2 Ss"
where used F[(oo] 0, EEc] 0, and for Uniform([-1,) 1J3), we have B[co²"] T. For the noisy observations ft vario /1 aye, the covariance becomes:
Σ m
= 2
= 2 T + (1)
agEnod (1 an)l
= az + arr 3 Is 2 ss" + -1-60 (1-
Assuming the RGB perturbation affects each color channel independently and focusing on a single channel, the second term contributes a rank-1 perturbation with eigenvalue Xiy 1133/ss1///3
By the Wiener filter analysis of Section the learned sensitivity along the new "W" principal component is
A X2y + (1 oc)) (m)(1)
2072118|12/3
? |||///3 (1 04 21882
a x| 3(1 (4))
7/11/122
7 1/ + 30
where (1 ax))) is the noise variance at timestep t.
20

---

B Additional Experiments and Ablation
B.1 Ablation of our model
The analytical model proposed in this paper has a single hyperparameterr: T- the threshold of the sensitivity field binarization. In Appendix A.3 we formally justify binarization of the sensitivity fields for our analytical model. Here, demonstrate the effect of choosing different binarization thresholds. In particular, from Figure we can see that higher threshold values (i.e., smaller patch sizes) correspond to sharper, but "patchier". On the other side, small threshold values (i.e., bigger patch sizes) cause the generated image to be over-smoothed. We report the y² and MSE metrics of correlation with the trained diffusion model for different threshold values in Table 2.
Table 2: Comparison of 7 and MSE metrics across datasets for different binarization threshold values. Best values are highlighted in bold.
Threshold
CIFARIO
CelebA-HO HQ
MNIST
Fashion MNIST
MSE
yd
MSEJ
y2
MSE
y²
MSE
0.005
0.396
0.059
0.786
0.038
0.492
0.151
0.563
0.115
0,010
0.520
0.046
0.865
0.023
0.441
0.165
0.517
0.122
0.020
0.672
0.031
0.897
0.017
0.418
0.176
0.406
0.144
0.050
0.773
0.021
0.894
0.017
0.214
0.255
0.072
0.211
0,070
0.771
0.022
0.879
0.020
0.214
0.255
-0.192
0.264
0.100
0.737
0.026
0.852
0.024
0.214
0.255
-0.407
0.311
0.150
0.641
0.036
0.799
0.033
0.214
0.255
-0.209
0.270
0.005
0.01
0.02
0.05
0.07
0.1
0.15
4
7
9
9
9
Figure Ablation of the binarization threshold т.
B.2 Self-attention layers in denoising U-Nets
Across our experiments, are using trained DDPM model with removed self-attention (SA) layers following [12]. In this section, we demonstrate that removing the self-attention layer brings the FID score 6.04 from 4.12 with SA. Qualitatively, the generated images look similar with and without SA, and thus our analysis in the main paper can be extended to U-Nets with SA layers.
In particular, we train U-Net without self-attention and compare it with baseline U-Ner trained with self-attention. Using the gradient-estimation sampler from [22], we report the FID scores for both models, and in Figure 7, we compare sample results.
21

---

DDPM w/o SA (FID 6.04)
DDPM with SA (FID 4.12)
Figure 7: Samples from trained DDPM U-Nets without (left) and with (right) self-attention layers. The initial random noise is the same for both sets of images.
B.3 Low rank projection of the covariance matrix
In this section, we study whether projecting the covariance matrix to a lower rank can help eliminate the high-fregue fighffrequency artifacts observed in the Wiener Filter in Figure 5.
First, on Figure 8, we analyze the singular values of the covariance matrices for each of the datasets. We observe that most of the total energy is contained in the first 200 singular values for all datasets, leaving long tail of low-energy principal components.
Singuiar Value Spectra Comparison
First 100 Singuiar Values (Detalled View)
32
33
FF-0
FF-40
No Narmatood Eoguler Ware thn srail
rggut Volue (oggcci)
3944
Singular Value Indix
Singular Value Index
Figure 8: Analysis of the absolute values of the singular values in the covariance matrices across different datasets. Left: across the full set of the singular values. Right: across the first 100 singular values.
Next, we zero out the smallest singular values of the covariance matrix based on different thresholds of total energy. Both for Ours and for the Wiener filter, measure the difference between predictions of the analytic models and a trained DDPM model, as done the paper. The results are averaged across 16 generations and presented in Table below. Qualitative results are presented in Figure 9. A 0% SVD Energy Cutoff corresponds to using the full-rank covariance matrix.
For the Wiener filter, low-rank approximation leads to visually smoother outputs; however, it reduces the correlation with outputs from trained diffusion models. Our algorithm does not benefit from using a low-rank projection of the covariance matrix and still demonstrates higher correlation than
22

---

Table Comparison of our method vs Wiener filter across different SVD energy cut-off thresholds.
Results show Ra scores (the higher the better) for each dataset and energy cut-off combination. Best Ra score for each dataset is shown in bold.
SVD Energy Cutoff
0%
0.01%
0.05%
0.10%
0.50%
1.00%
5.00%
10.00%
20.00%
50.00%
Ours CIFARTO
0.773
0.501
0.312
0.245
0.176
0.176
0.192
0.194
0.200
-0.248
Ours CELEBA HQ
0.897
0.759
0.702
0.695
0.685
0.668
0.652
0.650
0.607
0.607
Ours MNIST
0.492
0.197
0.156
0.158
0.351
0.341
0.361
0.384
0,389
0.368
Wiener CIFARTO
0.674
0.677
0.661
0.649
0.538
0.460
0.223
0.168
0.127
-0.048
Wiener CELEBA HQ
0.818
0.817
0.805
0.797
0.766
0.739
0.649
0.625
0.548
0.548
Wiener MNIST
0.469
0.468
0.468
0.467
0.453
0.445
0,421
0.394
0,353
0.292
0.005%
0.01%
0.05%
0.1%
0.5%
1%
5%
10%
50%
Ouss
CIFARIO
Wiener
CelebA-HO
Wiener
9
MNNST
Wiener
9
Figure 9: Qualitative comparison between generations of the analytical models when the covariance matrix is projected to lower rank corresponding to different energy cut-off thresholds.
the Wiener filter. This indicates that low-rank structure alone does not account for the performance of learned denoisers. We believe this result further supports the distinctiveness and relevance of proposed model.
B.4 How to reproduce the reported sensitivity fields
In this section, we provide the technical details and intuition needed measure the sensitivity fields of diffusion models. All the results are reported for CIFAR 10 dataset. Recall that the optimization problem is invariant under the change of variables from the initial image To the noise sample e; see Appendix A.1 for details. Consequently, one can measure the sensitivity field in either the noise parameterization, de (ar, /),rr, or the image parameterization, 0xro /)/Or. Although the choice merely a theoretical convenience, in practice the model's 's behavior is highly sensitive to it.
The top row of Figure 10 shows the sensitivity fields of DDPM model trained predict xo and then re-parameterized with linear transform to predict e; here we plot De(z,t)/0z. The middle row depicts the same model, but the sensitivity evaluated in the image parameterization. iee. @xo(x, t)/Or. As we can see, simple linear reparameterization applied to the model output drastically alters the result. These observations are intuitive. From the optimal-denoiser analysis, we know that, in the high-noise regime, the model predicts an image close to the dataset mean. Thus, predicting the added noise sample e for each pixel y almost equivalent to outputting q minus that mean, so the poise-parameterizee sensitivity field appears highly local. Because this visualization is not very informative, we chose to plot ar (x. 1)/Or throughout the paper, as captures the actual structure of the sensitivity field.
23

---

Figure 11 illustrates the effect of training U-Net and DiT models in the two parameterizations. Recall that zy /a, Fn + vI- 0z For large where a O, image 2n ill-defined given and ry; conversely, for small where a, + 1, is ill-defined given and Tt- Hence, while theory predicts identical results (up to re-parameterizatioon) numerical errors lead different behavior at and high noise levels. The top two rows of Figure 11 show (m t)/Oz for models trained in the noise parametcrization, revealing a pronounced shrinkage of the sensitivity fields in the high-noise regime. We hypothesize that this is numerical artifact and therefore plot, in the bottom two rows, the fields obtained from models trained directly in the image parameterization. For clarity, all DDPM examples in the main paper trained in that setting.
Finally, the middle and bottom rows of Figure compare two normalization strategies. In the middle row, each sensitivity field is normalized independently to [-1, 1I: in the bottom row, the images are normalized jointly, preserving relative scale. Joint normalization makes the field appear less local while preserving its overall mass. Throughout the paper, we adopt per-image normalization, as it more faithfully reflects the binarization assumed in our analytical model.
Summary of visualization choices
Train the model to predict the image 20 (not the noise c).
Visualize the sensitivity of the image prediction, i.e. Oxp (x, 0)/Or.
Apply per-image normalization.
955
800
6.50
500
350
200
50
Diff w.r.t. E
Image norm.
Global norn.
Figure 10: Top: sensitivity field of the noise prediction Öe(x, l)/Ox. Middle: sensitivity field of the image prediction axo (x, 1) with per-image normalization. Bottom: the same field with joint normalization across images.
B.5 Sensitivity field of the optimal denoiser
In this section, we provide visualization of the sensitivity fields of the optimal denoiser on the CIFARIO dataset. As shown Figure 12, the sensitivity of the optimal denoiser closely resembles that of the trained models only in the high-noise regime. At intermediate noise levels, the sensitivity field begins diverge, and in the low-noise regime, it ultimately "explodes".
B.6 Generation dynamics
Here provide additional results demonstrating the dynamics of image generation. In Figure 14 numerically compare Zy predictions through the generation process.
In Figure 13 we demonstrate how the dynamics of image generation of our analytical model compares with that one of a trained DDPM model. Note that the trained model produces noisy single-step predictions for high noise levels (1 2 850). We explain this behavior by the fact that the model was trained to predict and later re-parametrized to output Tg for the visualization. Since a - 0 for high noise level, 2g becomes ill-defined and thus noises the outputs.

---

950
800
650
500
350
200
50
U-Net
DiT E
U-Net .50
DFT 200
Figure 11: Sensitivity fields (x t)/ for U-Net (left) and DIT (right). Top two rows: models trained to predict noise c. Bottom two rows: models trained to predict the image Zu- The shrinkage observed at high noise in the noise-parameterinn models is likely due to numerical instability.
950
800
650
500
350
200
50
Optinal
Figure 12: Sensitivity field of the optimal denoiser.
B.7 Quantitative results for AFHQv2 and Fashion-MNIST
In addition to Table in the main paper we report quantitative results for AFHQv2 and Fashion- MNIST Table All the values calculated across 128 samples.
Table 4: Comparison of methods across AFHQv2 and Fashion-MNIST. All metries are averaged over 1 128 samples. Best results are highlighted in green and second best in maroon.
AFHQv2
Fashion MNIST
Method
2
MSEL
2
MSEJ
Optimal Denoiser
-1.239
0.371
0.180 + 0.023
-0.137 0.344
0.254 0.077
Wiener (linear)
0.601
0.072
0.025 + 0.003
0.449
0.068
0.137 0.018
Kamb & Ganguli [12]
0.429 = 0.081
0.041 1004 0.006
0.342 0.183
0.186
0.019
Ours
0.759 = 0.026
0.019 + 0.004
0.523 = 0.042
0.125 + 0.011
Another DDPM
0,928 + 0.019
0.050 + 0.001
0.950 + 0.020
0.015 0.005
B.8
Quantitative measure of novelty samples
In this work, we focus on the ability of the trained diffusion models to generate novel samples that contrast with the behavior the optimal denoiser. Therefore, the ability the analytical model to generate novel samples is paramount. In figure 5 of the main paper (as well as in Appendix B.9) we report the nearest neighbors from the training dataset each sample generated with our analytical model. To quantify these results, report the average L2 distances between samples generated with
25

---

950
850
750
650
550
450
350
250
150
DDPM
Ours
DDPM
Ours
Intermediate generation results of trained DDPM model (rows I and 3) and ours (rows and 4). The figure displays single-step estimations of Tg from each xy along a sampling trajectory of 10 steps.
Figure 13:
CAM6A-HQ
115
Denotiing strgt
Desnsing
Figure 14: Mean Squared Error (MSE) between the baseline's predictions and trained DDPM model. The MSE is calculated on æ prediction from each iy point along a 10-step generation trajectory. The results presented on the CIFARIO and CelebA-HQ datasets. Mean and standard deviation values were calculated across 128 samples.
each of the baseline models and the closest image in the dataset in Table 5. Additionally, report the dynamics of the "novelty" measure the generation process in Figure 15.
Table We numerically quantify the ability of analytical models to produce images outside of the training dataset. In this table, we provide the average L2 distance between images generated with the basclines and the corresponding closest image the training dataset. Results are averaged over 128 samples.
Method
CIFARIO
CelebA-HQ
AFHQv2
MNIST
Fashion MNIST
Optimal Denoiser
0.000 + 0.000
0.000 + 0.000
0.000
0.000
0.000 + 0.000
0.000 0.000
Wiener (linear)
0.091 0.006
0.104 1 0.006
0.112
0.005
0.177 + 0.015
0.133 0.013
Kamb & Ganguli [12]
0.094 1 0.005
0.089 1 0.006
0.136
0.007
0.355 1 0.061
0,218 1 0.032
Ours
0.040 1 0.00S
0.063 1 0.004
0.063
0.004
0.204 1 0.023
0.131 1 0.027
Another DDPM
0.079 = 0.014
0.087 + 0.010
0.095
0.013
0.103 + 0.023
0.067 = 0.015
B.9
Additional generation results
We present additional generation results similar to fig. 5 of the main paper in Figures 16 to 20.
26

---

CIRARIO
ARRON
CeleM-HiQQ
MINIST
Fastion MINSST
300
8
Denaing
Figure 13: L2 distance between 2o prediction and the closest image in the training dataset reported along a 10-step generation trajectory for 5 datasets.
C Implementation details
C.1 Sampling.
In all of the generations in this paper, we are using diffusers' [9] implementation of the DDIM [27] sampler with 10 sampling steps. We discretize the noise time scale for t [0, 1000] where f 0 is no noise and t 1000 is full noise. The scheduler is linear with og 10 and 010000 0.02.
C.2 Training DDPM Model
We train Denoising Diffusion Probabilistic Model (DDPM) U-Net using third-party pytorch implementation [35]. We adopt the U-Net model architecture based on the input image resolution:
MNIST/FashionnMNIST (img_size = 28): 3 downsampling levels with channel_mu
[1,
2, 2], base channel width 64. CIFARIO (img_st size 32) and CelebA-HQ/AFHQ (img size 64): 4 downsampling levels
with channel_mult
[1, 2, 3, 4]. base channel width 128.
The number of residual blocks per level is fixed with self-attention modules included. Dropout set to 0.15 throughout the network. The model is trained for 200 epochs with a batch size of 32. We the Adam optimizer with learning of 10-4 over 1000 diffusion steps. Training and evaluation use fixed random seeds for reproducibility.
C.3 Our analytical model
Below, we provide detailed description of the implementation of our analytical model. A key component of this implementation is the weighted streaming softmax (wssm) that accumulates the product zg softmax (...) over batches of training images.
27

---

Algorithm Single denoising step of the proposed analytical model.
Require:
Noisy image 3tz
Timestep
Precomputed covariance S of the data Masking threshold 7
Dataset X
Schedule of an and of 1-ae
a
Ensure: Estimated clean image do
1: UAU = S
SVD of the covariance matrix
2: W, U
A?
UT
Current Wiener matrix
/ou 3: M, Binarize(WE,77
x7+
Construct the projection matrix > Initialize weighted streaming softmax
4: wssm.init(
5: for each batch
10
(A) from X do
(Ix - vario (k) 27
6: D3 - stack
b Distance to 2t for each image in the batch b Each row of M, serves as a mask
7: Dm, Da
8: wssm.updale
(-Dm,/2 -cel, 40
(k)
b. Add the distances and the value
9: end for
10: Co wasm..ealue()
11: return Z0
C.4 Baseline implementation details
Wiener filter. To implement the Wiener matrix, we first center each dataset to a zero mean. Then we pre-compute the covariance matrix of the dataset. Note that this is part of "training" and these computations were not included in the runtime report. On sampling, use the PyTorch implementation of SVD compute the principal components and the corresponding singular values. Finally, we are using eq. (7) from the main paper to implement W, Note that we are using the Wiener filter as denoiser, and when generating the images, we are still using a 10-step DDIM sampling, effectively applying the Wiener filter 10 times to the initial noise.
Kamb & Ganguli model. We implemented the analytical model suggested by Kamb & Ganguli in code base. Then we fit the patch sizes M, of the analytical model to our trained DDPM U-Nets, maximizing the 72 between the scores on each step of generation. For the CelebA-HQ dataset we are using (non-equivariant) Local Score (LS) machine, for other datasets we used Equivariant Local Score (ELS) machine. Below are the patch sizes that we obtained:
CIFARIO 32 x 32: [32, 32, 32, 29, 25, 17, 13, 9, 7,
CelebA-HQ 64 64: [ 64, 64, 45, 25, 17, 17, 9, 7, 7,
AFHQ 64 x 64: [64, 64, 45, 33, 25, 17, 17, 9, 9,
MNIST 28 28: [28, 28, 23, 17, 13, 13, 13, 9.
Fashion MNIST 28 x 28; [28, 25, 23, 17, 17, 13, 13,999.55
C.5 Algorithmic Complexity
In this section, provide an analysis of the algorithmic complexity of our method and the baselines. For small-resolution images, the Wiener filter remains the most efficient at ((m)), where m the flattened image resolution. Both our model and Kamb&Ganguli's 's require dataset pass per inference step, leading to scaling linear in n, where n is the dataset size.
Kamb& Kamb&Ganguli assume translation equivariance, so for each pixel. its surrounding patch (size pa at denoising step /) is compared to every patch in the dataset, resulting in O(npmm²) complexity. With approximate vector search (e.g., using & clusters), this reduces to O(pem" /k). Our model forgoes translation equivariance as we did not observe any difference quality. Additionally, we are using distinct per- vexel mask pattern. This leads to the algorithmic complexity of our algorithm being
28

---

((ppem). For larger datasets, we can match the mmppmm/k) complexity by indexing masks per timestep and pixel. We provide the summary in Appendix C.S.
Table 6: Algorithmic complexity of the baselines. Here, m denotes the flattened image resolution, n the dataset size, Pt the patch size denoising step 4, and k the number of clusters used in approximate nearest-neighbor search.
Method
Wiener
Kamb (exact) Kamb (approx)
Ours (exact)
Ours (approx)
Complexity
(
((npmm²
apmm
(mppmm)
O m
C.6
Computational resources and runtime
All the experiments were performed on a server machine with Ubuntu 20.04. The machine has 1008GB RAM, 128 CPU cores and 8x NVIDIA RTX A6000 GPUs with 49/40MB VRAM. We note that all the baselines could be run with fewer computational resources. In Table 7 we provide the average run times for each baseline.
Table 7: We demonstrate the computational efficiency of each method by displaying the total sampling time for each of the baselines over 10 denoising steps. None of the methods are optimized for runtime, and the comparison provided only as rough reference. Results show times averaged over 64 samples.
Method
CIFARIO
CelebA-HQ
AFHQv2
MNIST Fashion MNIST
Optimal Denoiser
7.90
18.90
10.01
0.63
0.64
Wiener (linear)
0.11
3.10
3.08
0.07
0.07
Kamb & Ganguli [12]
44.44
349.68
181.08
4.40
4.49
Ours
21.25
70.23
314.55
22.39
22.97
Another DDPM
0.57
0.65
0.65
0.61
0.63
DDPMI
DDPM2
Ours
Wiener
Kamb
Dataset
Figure 16: Additional generation results for all baselines and ours the CelebA-HQ dataset.
29

---

DDPMI
DDPM2
Ours
Wiener
Kamb
Dataset
Figure 17: Additional generation results for all baselines and ours on the AFHQ dataset.
DDPM2 DDPMI
Ours
Wiener
Kamb
Dataset
Figure 18: Additional generation results for all baselines and ours on the CIFAR10 dataset.
30

---

DDPM2 DDPMI
4
8
5
1
8
2
5
1
Ours
4
3
36
4
$
t
Wiener
2
5
2
3
Kamb
6 9 7
2
5
0
8
3
1
Dataset
0
0303755
0
0
0
3
5
4
4
9
9
Figure 19: Additional generation results for all baselines and ours on the MNIST dataset.
DDPM2 DDPMI
Ours
Wiener
Kamb
Dataset
Figure 20: Additional generation results for all baselines and ours on the Fashion MNIST dataset.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: the main claims in the abstract and introduction are backed up with detailed and rigorous experiments, well theoretical justifications. Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have discussed this the main paper before the conclusion
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model well-specification,
asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We have provided all assumptions and proof details, either in the main paper or appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
32

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes]
Justification: We have included details in the paper and appendix, as well as code and instructions required reproduce these experiments in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, if the contribution is a specific model and empirical evaluation, it may necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily architecture clearly and fully.
model architecture, the paper should describe the
(c) If the contribution is new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g.. with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases. in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g.. to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
Open access data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will include the code required to reproduce all of our experiments in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines huttps://nipp httcpp:://ips.cccppbll./
guides/ Muides/Cooeeuule SodeSubbissi wwwsssssssliiiioooo icy for more details.
While we encourage the release code and data, we understand that this might not be possible,
"No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed the results. See the NeurIPS code and data submission guidelines (https: / //nips/i/////e/// /ttps.cccpublll..... cc/pubicc/ quides/CodeSuuee Cuuee/mddSuubmaa ssi ssionPoiicy) for more details.
reproduce
33

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reprodueible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended. but including URLs data and code is permitted. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g.. data splits. hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have included important details the paper, and additional details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is
necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report standard deviation values for numerical results in Table I in the appendix. We verify that our claims that our method outperforms baseline are statistically significant.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors). It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experi- ments?
Answer: [Yes]
Justification: We include in the supplementary materials details about the computational resources needed to reproduce experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
34

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't 't make into the paper).
Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code Ethics https: http:////eaaappp..p.ppppttthhhaa
Answer: [Yes]
Justification: Yes, the research conducted this paper conform to the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code Ethics.
If the authors answer No, they should explain the special cireumstances that require a deviation
from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consideration due
to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work consists of basic research aiming to better understand how diffusion models generalize, and is not tied to any potential applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to par- ticular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, is legitimate to point out that improvement in the quality of generative models could be used generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse. mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question: Does the paper describe safeguards that have been put place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: We not release any such data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere
usage guidelines or restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
35

---

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yss]
Justification: We cite the datasets, codebases and pre-trained models used in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should included for each asset. For scraped data from particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's
creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We do not release any new assets.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/moodl as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We do not perform any studies involving human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human
subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail possible should included in the main paper.
According to the NeurlPS Code Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We do not perform any studies involving human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
36

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly
state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non- standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: We do not use LLMs important, original, or non-standard components in this work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttttp://neurppss wewparrisscc/ooooeeeeeeeoee wct///oofeernnc////2///////// for what should or should not be described.
37


Reviews:
[{"review_id": "WTl7MMpCgy", "reviewer": "Reviewer_8kUK", "summary": "This paper is a nice response to the recent paper Kamb Ganguli, which propose generalization comes from local equivariant inductive bias. This paper showed that the local receptive field is dataset dependent and universal across architectures, suggesting it’s a property of dataset than architecture. They further leveraged the Gaussian linear score solution / Wiener matrix to analytically approximate the receptive field and use it to derive another closed form diffusion sampler.  They showed empirically that their analytical diffusion better approximates actual diffusion model than the Gaussian linear one and local equivariant one.", "questions": "### Questions\n\n- Though it’s not the main point of this paper, is there a comparison of efficiency / runtime of your method and Kamb Ganguli method & Wiener filter? In other word Is this new method practical? Wouldn’t the softmax easily run into computational out of memory problems when the dataset goes up?", "limitations": "N.A.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strength\n\n- I agree and am convinced the data statistics place a larger role in determining the sensitivity field. The manipulation of dataset statistics and the UNet vs DiT comparison are very convincing. This is a important and nice point to make given the influential Kamb Ganguli paper.\n- Comparison to related works are carefully made. \nThe qualitative and quantitative benchmark in Table 1 and Figure 5. are nice. Previous work Kamb & Ganguli esp. lack such quantitative and systematic comparison with the Wiener filter results (Wang Vastola 2023, 2024), so it’s good to show quantitatively the linear case works well still.   Using another trained DDPM as reference is also very nice control.\n\n### Weakness\n\n- Maybe in the full paper some of Appendix A and more intuition should be put into main paper to support the Eq. 9? I think the intuition is the using the binarized corresponding qth row of Wiener matrix as the “receptive field” mask.\n- Theoretical justification (Supplementary section A.3) it’s a bit confusing to follow.\n    - The notation in the  is sometimes not consistent. in line 119 A_t^q is d by d but in the definition in L96 it’s dxd by dxd.\n    - The generalized locality definition A.8 does not make much sense for arbitrary linear operators. It cannot be true for a general A_t^q\n    - I imagine there is a clearer and more intuitive way to write this section.", "was_revised": true, "final_justification": "We strongly recommend this paper as a worthy addition to the literature. \nIt will clarify some mis-understanding created from previous influential papers and guide development in future works \nThe new derivation made the method more intuitive, the comparison of computational efficiency, and comparison with Wiener filter with lower rank covariance is interesting and made the work much stronger. \n\nWe highly recommend this work for acceptance!", "rebuttal": "We thank Reviewer 8kUK for their thoughtful review and for describing our paper as “a nice response” to recent work. We’re glad you found our experimental comparisons “very convincing” and appreciated the “quantitative and systematic comparison with the Wiener filter.” Reviewers 8EDr and Ws6m also found our model to significantly outperform prior analytical approaches, and 4nQU highlighted the clarity and accessibility of our paper.\n\nWe also appreciate reviewers' suggestions on how to further strengthen the paper. In the rest of this reply, we first address suggestions shared by multiple reviewers and then will specifically address comments in this review.\n\n## Computational Efficiency\n\nReviewers 8EDr and 8kUK suggested including a comparison of runtime between the baselines. The runtimes and hardware used for all baselines are reported in Section C.5 of the appendix, and we provide additional context below. Our algorithm is more computationally efficient than Kamb & Ganguli's model both asymptotically and in practice, as we are not assuming translation equivariance. As reviewer 8kUK correctly mentioned, creating an efficient analytical denoiser is not the main focus of this paper; however, we agree that additional analysis of computational efficiency will further strengthen the paper.\n\nBelow, we provide a detailed analysis of the algorithmic complexity for the Wiener filter, Kamb & Ganguli's analytical model, and ours.\n\n### Algorithmic Complexity and the Softmax\n\nFor small-resolution images, the Wiener filter remains the most efficient at $O(m^2)$, where $m$ is the flattened image resolution. Both our model and Kamb & Ganguli’s require a dataset pass per inference step, leading to scaling linear in $n$, where $n$ is the dataset size.\n\nKamb & Ganguli assume translation equivariance, so for each pixel, we compare its surrounding patch (size $p_t$ at denoising step $t$) to every patch in the dataset, resulting in $O(n p_t m^2)$ complexity. With approximate vector search (e.g., using $k$ clusters), this reduces to $O(n p_t m^2 / k)$. Our model forgoes translation equivariance as we did not observe any difference in quality (reported in the main paper). Additionally, we are using a distinct per-pixel mask pattern. This leads to the algorithmic complexity of our algorithm being $O(n p_t m)$. While this makes approximate search harder, our implementation (Section C.5) is faster than even Kamb & Ganguli’s approximate version on benchmark-scale datasets. For larger datasets, we can match their $O(n p_t m / k)$ complexity by indexing masks per timestep and pixel.\n\n**Softmax**: As Reviewer 8kUK noted, computing softmax across a dataset is challenging. We address this with a numerically stable streaming implementation requiring just one pass and constant memory (see `src/utils.py` in the appendix files).\n\n**Algorithmic complexity table**:\n|Method|Wiener Filter|Kamb & Ganguli (exact)|Kamb & Ganguli (approx)|Ours (exact, implemented)|Ours (approx, not implemented)|\n|-|-|-|-|-|-|\n|Complexity|$O(m^2)$|$O(n p_t m^2)$|$O\\left(\\frac{n p_t m^2}{k}\\right)$|$O(n p_t m)$| $O\\left(\\frac{n p_t m}{k}\\right)$|\n\n\n## Choice of $A_t^q$: intuition, other alternatives and formatting\n\n### Motivation in Section A.3 (8kUK, Ws6m)\n\nWe agree with reviewers 8kUK and Ws6m that the main paper will benefit from some part of the motivation of our algorithm from the appendix to the main paper. We will update the main paper with a clearer intuition behind our algorithm: Kamb and Ganguli use rectangular binary masks fitted to the sensitivity fields of trained denoisers. In our paper, we show that the sensitivity fields of the trained denoisers come from the data covariance. In this paper, we are considering constant sensitivity fields, i.e., $A_t^q$ is not a function of $x$. If the sensitivity field is constant, the denoiser is linear, and we know that the optimal linear denoiser is the Wiener filter. Thus, the sensitivity field under the constant constraint is that of the Wiener filter.\n\nWe also revised the derivations for why the sensitivity fields should be binary:\n\n* **Generalized locality definition** We first consider a generalized notion of locality, with the constraint $f^q(x) = f^q(A^q_t x)$, where $A^q_t$ can be any matrix. Please note that indeed, when $A^q_t$ is invertible, $f^q$ is effectively unconstrained, i.e., there is no locality. When $A^q_t = 0$, $f^q$ does not depend on $x$.\n \n* **Generalized locality imposes subspace projection** Next we prove that imposing the generalized locality constraint is equivalent to adding a projection operator $\\mathbf{P_t^q}=(A_t^q)^T (A_t^q (A_t^q)^T)^\\dagger A_t^q$ (projection onto the row-space of $A^q_t$) to the optimal denoiser: $\\hat{f}^q(x, t) = \\sum_{i=1}^N (x^i_0)^q \\; \\text{softmax}_i\\left(-\\frac{1}{2\\sigma_t^2} \\|\\mathbf{P_t^q}(x - \\sqrt{\\alpha_t}x_0^i)\\|^2\\right)$\n\n* **Projection = Orthogonal change of basis + Masking** All projection operators can be written as an orthogonal change of basis $V_t^q$ followed by a masking operation $M^q_t$ (i.e., a diagonal 0-1 matrix): $\\mathbf{P_t^q} = (V_t^q)^TM^q_tV_t^q$\n\nThis explains why the only interesting generalized locality matrices are binary masks. We will move motivation for our method and additional intuition from Appendix A.3 to Section 3 of the main paper, in case if the paper is accepted. We will update our paper's appendix with proofs of the above statements.\n\n### Low-Rank Covariance Matrix\nIn response to Reviewer Ws6m’s suggestion, we conducted an additional comparison between the Wiener filter and our algorithm under low-rank projections of the covariance matrix. For the Wiener filter, low-rank approximation leads to visually smoother outputs; however, it reduces the correlation with outputs from trained diffusion models. Our algorithm does not benefit from using a low-rank projection of the covariance matrix and still demonstrates higher correlation than the Wiener filter. This indicates that low-rank structure alone does not account for the performance of learned denoisers. We believe this result further supports the distinctiveness and relevance of our proposed model. Please refer to our detailed qualitative and quantitative analysis in the personal response to Reviewer Ws6m.\n\nThank you for your thoughtful review, and please let us know if you have any additional suggestions on how we can further improve our paper."}, {"review_id": "LH6hlJmDDW", "reviewer": "Reviewer_4nQU", "summary": "The paper challenges the current understanding of the generalization capabilities of diffusion models. Specifically, the authors analyze the gap between the optimal denoiser and approximate denoisers learnt by neural networks. It is demonstrated, theoretically and empirically, that the locality of the learnt denoisers can be explained by data statistics, not by inductive biases of the neural network architectures.", "questions": "1. Is the difference between the proposed method and the baselines in Table 1 statistically significant in all experiments?\n2. I think that Figure 5 should be improved. My suggestion would be to remove the last row unless the authors can convince me that it is relevant to that experiment (in which case, I believe that it should also be presented for other baselines)", "limitations": "The limitations are properly discussed.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. This paper is very well-written. It reads somewhat like a tutorial, which I appreciate. The reader is guided through the problem statement, current problem understanding and approaches, experiment design, and results.\n2. I believe that the research problem is important, especially given that this work challenges the understanding of the machine learning community and provides an alternative explanation of the generalization capabilities of denoising diffusion models.\n3. The authors provide the code to facilitate the replication of results. The codebase is neatly organized.\n4. Experiments are clearly explained and convincingly support the claims made by the authors.\n5. Proofs in the appendix are readable and easy to follow, given the explanations throughout.\n\n**Weaknesses:**\nI did not spot any major weaknesses. Minor weaknesses/suggestions:\n\n1. The README in the codebase could be written better. There are typos, and not immediately obvious which scripts should be run. This should be improved upon acceptance.\n2. Figure 5 - at a first glance, the \"Ours NN\" label can be interpreted (at least I initially did) as \"Ours - Neural Network\", as in \"We built a Neural Network model using our theoretical insights\". Perhaps it can be relabeled somehow to instantly be read as \"nearest neighbor in the data\"?\n    * On second thought, I am not sure I understand what is the purpose of this last row. Why show this only for the proposed method, and not other baselines?\n3. Table 1 - would be nice to have some error bars/statistical significance tests. Although the difference sometimes seems quite large. \n    * I can see that the standard deviations are reported separately in a table in the appendix, but that is not very readable to me, having to go back and forth looking and the values table and the standard deviation table", "was_revised": true, "final_justification": "I maintain my support for the paper. I chose not to give a higher score, because I am not an expert in this area.", "rebuttal": "We sincerely thank Reviewer 4nQU for highlighting the strength of our approach, in particular, evaluating it as “very well-written”, as well as your remarks on the importance of the problem and the clarity of our experiments and proofs. We’re also grateful that other reviewers, including 8EDr and 8kUK, found the benchmarks and comparisons both novel and convincing.\n\nWe also appreciate reviewers' suggestions on how to further strengthen the paper. In the rest of this reply, we first address suggestions shared by multiple reviewers and then will specifically address comments in this review.\n\n# Shared comments\n\n## Computational Efficiency\n\nReviewers 8EDr and 8kUK suggested including a comparison of runtime between the baselines. The runtimes and hardware used for all baselines are reported in Section C.5 of the appendix, and we provide additional context below. Our algorithm is more computationally efficient than Kamb & Ganguli's model both asymptotically and in practice, as we are not assuming translation equivariance. As reviewer 8kUK correctly mentioned, creating an efficient analytical denoiser is not the main focus of this paper; however, we agree that additional analysis of computational efficiency will further strengthen the paper.\n\nBelow, we provide a detailed analysis of the algorithmic complexity for the Wiener filter, Kamb & Ganguli's analytical model, and ours.\n\n### Algorithmic Complexity and the Softmax\n\nFor small-resolution images, the Wiener filter remains the most efficient at $O(m^2)$, where $m$ is the flattened image resolution. Both our model and Kamb & Ganguli’s require a dataset pass per inference step, leading to scaling linear in $n$, where $n$ is the dataset size.\n\nKamb & Ganguli assume translation equivariance, so for each pixel, we compare its surrounding patch (size $p_t$ at denoising step $t$) to every patch in the dataset, resulting in $O(n p_t m^2)$ complexity. With approximate vector search (e.g., using $k$ clusters), this reduces to $O(n p_t m^2 / k)$. Our model forgoes translation equivariance as we did not observe any difference in quality (reported in the main paper). Additionally, we are using a distinct per-pixel mask pattern. This leads to the algorithmic complexity of our algorithm being $O(n p_t m)$. While this makes approximate search harder, our implementation (Section C.5) is faster than even Kamb & Ganguli’s approximate version on benchmark-scale datasets. For larger datasets, we can match their $O(n p_t m / k)$ complexity by indexing masks per timestep and pixel.\n\n**Softmax**: As Reviewer 8kUK noted, computing softmax across a dataset is challenging. We address this with a numerically stable streaming implementation requiring just one pass and constant memory (see `src/utils.py` in the appendix files).\n\n**Algorithmic complexity table**:\n|Method|Wiener Filter|Kamb & Ganguli (exact)|Kamb & Ganguli (approx)|Ours (exact, implemented)|Ours (approx, not implemented)|\n|-|-|-|-|-|-|\n|Complexity|$O(m^2)$|$O(n p_t m^2)$|$O\\left(\\frac{n p_t m^2}{k}\\right)$|$O(n p_t m)$| $O\\left(\\frac{n p_t m}{k}\\right)$|\n\n\n## Choice of $A_t^q$: intuition, other alternatives and formatting\n\n### Motivation in Section A.3 (8kUK, Ws6m)\n\nWe agree with reviewers 8kUK and Ws6m that the main paper will benefit from moving some part of the motivation of our algorithm from the appendix to the main paper. We will update the main paper with a clearer intuition behind our algorithm: Kamb and Ganguli use rectangular binary masks fitted to the sensitivity fields of trained denoisers. In our paper, we show that the sensitivity fields of the trained denoisers come from the data covariance. In this paper, we are considering constant sensitivity fields, i.e., $A_t^q$ is not a function of $x$. If the sensitivity field is constant, the denoiser is linear, and we know that the optimal linear denoiser is the Wiener filter. Thus, the sensitivity field under the constant constraint is that of the Wiener filter. \n\nWe also revised the derivations for why the sensitivity fields should be binary:\n\n* **Generalized locality definition** We first consider a generalized notion of locality, with the constraint $f^q(x) = f^q(A^q_t x)$, where $A^q_t$ can be any matrix. Please note that indeed, when $A^q_t$ is invertible, $f^q$ is effectively unconstrained, i.e., there is no locality. When $A^q_t = 0$, $f^q$ does not depend on $x$.\n \n* **Generalized locality imposes subspace projection** Next we prove that imposing the generalized locality constraint is equivalent to adding a projection operator $\\mathbf{P_t^q}=(A_t^q)^T (A_t^q (A_t^q)^T)^\\dagger A_t^q$ (projection onto the row-space of $A^q_t$) to the optimal denoiser: $\\hat{f}^q(x, t) = \\sum_{i=1}^N (x^i_0)^q \\; \\text{softmax}_i\\left(-\\frac{1}{2\\sigma_t^2} \\|\\mathbf{P_t^q}(x - \\sqrt{\\alpha_t}x_0^i)\\|^2\\right)$\n\n* **Projection = Orthogonal change of basis + Masking** All projection operators can be written as an orthogonal change of basis $V_t^q$ followed by a masking operation $M^q_t$ (i.e., a diagonal 0-1 matrix): $\\mathbf{P_t^q} = (V_t^q)^TM^q_tV_t^q$\n\nThis explains why the only interesting generalized locality matrices are binary masks. We will move motivation for our method and additional intuition from Appendix A.3 to Section 3 of the main paper, in case if the paper is accepted. We will update our paper's appendix with proofs of the above statements.\n\n### Low-Rank Covariance Matrix\nIn response to Reviewer Ws6m’s suggestion, we conducted an additional comparison between the Wiener filter and our algorithm under low-rank projections of the covariance matrix. For the Wiener filter, low-rank approximation leads to visually smoother outputs; however, it reduces the correlation with outputs from trained diffusion models. Our algorithm does not benefit from using a low-rank projection of the covariance matrix and still demonstrates higher correlation than the Wiener filter. This indicates that low-rank structure alone does not account for the performance of learned denoisers. We believe this result further supports the distinctiveness and relevance of our proposed model. Please refer to our detailed qualitative and quantitative analysis in the personal response to Reviewer Ws6m.\n\n# Individual comments\n\n## RE: the README in the code\nThank you for pointing this out. We updated our README with detailed installation and execution instructions. We will publicly release a cleaned-up and better-documented codebase together with our camera-ready version. \n\n## RE: Formatting of Figure 5\nSince the main goal of the work is to study generalization mechanisms of trained diffusion models, we believe it is important to report the nearest image (\"OursNN\") in the dataset. By showing the closest image, we allow the readers to judge how novel the generated samples are and if the proposed analytical model indeed generalizes. To report the closest images for each baseline while preserving the space in the paper, we plan to change the format of Figure 5 and place a miniature version of the closest neighbor images at the top-right corner of each generated image. We will be happy to follow any recommendations on better formatting of Figure 5. We further agree that the label \"OursNN\" might be confusing and will rename it to \"Dataset\" by the camera-ready version.\n\n## RE: Table 1, moving variance to the main paper\nWe will be happy to move the variance for Table 1 from the appendix to the main paper. To comply with NeurIPS formatting, in the camera-ready version of the main paper, we are planning to provide the metrics only on 3 datasets out of 5 (CIFAR10, Celeba-HQ, and MNIST) with the respective variances, while moving the other two datasets (AFHQ and FashionMNIST) to the appendix. Please find the proposed formatting of Table 1 below:\n\n|Method|CIFAR10 r² ↑|CIFAR10 MSE ↓|CelebA-HQ r² ↑|CelebA-HQ MSE ↓|MNIST r² ↑|MNIST MSE ↓|\n|-|-|-|-|-|-|-|\n|Optimal Denoiser|-0.549±0.774|0.191±0.044|0.400±0.298|0.101±0.023|0.187±0.204|0.231±0.036|\n|Wiener (linear)|0.408±0.092|0.032±0.004|0.818±0.039|0.031±0.004|0.469±0.066|0.161±0.014|\n|Kamb & Ganguli (2024)|0.303±0.126|0.065±0.017|0.795±0.081|0.033±0.006|0.283±0.110|0.227±0.045|\n|Niedoba et al. (2024)|0.523±0.137|0.049±0.004|0.733±0.092|0.043±0.005| 0.301±0.077|0.214±0.022|\n|**Ours**|0.589±0.078|0.028±0.008|0.902±0.032|0.016±0.006|0.491±0.051|0.153±0.015|\n|Another DDPM|0.852±0.113|0.023±0.002|0.981±0.007|0.004±0.001|0.969±0.082|0.007±0.019|\n\n## RE: statistical significance\nOur results are statistically significant. To elucidate this, we calculate the p-value by performing a two-sample t-test against the hypothesis that \"Ours\" is better than the other baselines. All the results are presented for the $r^2$ coefficient of determination. From the table below, with high statistical significance, our method explains trained diffusion models better than any of the baselines.\n\n|Dataset|vs Wiener|vs Kamb|vs Niedoba|\n|-|-|-|-|\n|CIFAR10|$3.4*10^{-8}$|$8.0*10^{-10}$|$0.0351$|\n|CelebA-HQ|$1.9*10^{-5}$|$6.8*10^{-5}$|$2.9*10^{-9}$|\n|AFHQv2|$5.0*10^{-4}$|$1.1*10^{-6}$|$2.2*10^{-8}$|\n|MNIST|$0.0016$|$8.9*10^{-9}$|$8.0*10^{-4}$|\n|Fashion MNIST|$2.1*10^{-6}$|$9.3*10^{-8}$|$2.6*10^{-8}$|\n\nPlease let us know if you have other suggestions on how to format it better."}, {"review_id": "FjjuoZShaj", "reviewer": "Reviewer_8EDr", "summary": "This study challenges the prevailing assumption that CNN inductive biases primarily explain the gap between optimal diffusion denoisers and practical applications, and demonstrates that locality arises from underlying statistics of image datasets. The authors develop an analytical denoiser based on principal component analysis based on these statistical principles, which yields results that are closer to the true diffusion model behaviour than previous expert-designed approximations. It shows that local sensitivity in diffusion models arises from learned data statistics rather than architectural inductive biases, and identifies that analytically derived optimal linear filters-dependent only on training data, accurately predict both local and non-local spatial sensitivities observed in trained denoising networks, unexpectedly outperforming previous analytical models based on optimal denoisers on new quantitative benchmarks.", "questions": "Refer to weaknesses part.", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This work demonstrates that local sensitivity in diffusion models arises from learned data statistics rather than architectural inductive biases. It is demonstrated that analytically derived optimal linear filters which depend only on training data can accurately predict local and non-local spatial sensitivities observed in trained denoising networks, and surprisingly outperform previous analytical models based on optimal denoisers on new quantitative benchmarks. ​​ ​​The authors explore the mechanism of locality in image diffusion models from the perspective of data statistics and provide detailed visual proofs. \n\nWeaknesses: The author gives a very detailed discussion about his proposed strategy, but it seems to increase some computational efficiency, so I am curious about the experimental results of this. Although the method performs significantly, a key practical consideration is the computational overhead. Could the authors provide the above results? I am not very familiar with this field; therefore, if there are enough reviewers, please ignore my comments.", "was_revised": true, "final_justification": "The authors' ​​response​​ has addressed my concerns, so ​​I will maintain my rating.​​", "rebuttal": "Thank you for the thorough review of our paper, highlighting that it “surprisingly outperforms previous analytical models,” and for appreciating our “new quantitative benchmarks,” “detailed visual proofs,” and “very detailed discussion.” We're also encouraged that other reviewers, such as 8kUK and 4nQU, similarly praised the clarity, rigor, and completeness of our experiments and comparisons.\n\nWe also appreciate reviewers' suggestions on how to further strengthen the paper. In the rest of this reply, we first address suggestions shared by multiple reviewers and then will specifically address comments in this review.\n\n## Computational Efficiency\n\nReviewers 8EDr and 8kUK suggested including a comparison of runtime between the baselines. The runtimes and hardware used for all baselines are reported in Section C.5 of the appendix, and we provide additional context below. Our algorithm is more computationally efficient than Kamb & Ganguli's model both asymptotically and in practice, as we are not assuming translation equivariance. As reviewer 8kUK correctly mentioned, creating an efficient analytical denoiser is not the main focus of this paper; however, we agree that additional analysis of computational efficiency will further strengthen the paper.\n\nBelow, we provide a detailed analysis of the algorithmic complexity for the Wiener filter, Kamb & Ganguli's analytical model, and ours.\n\n### Algorithmic Complexity and the Softmax\n\nFor small-resolution images, the Wiener filter remains the most efficient at $O(m^2)$, where $m$ is the flattened image resolution. Both our model and Kamb & Ganguli’s require a dataset pass per inference step, leading to scaling linear in $n$, where $n$ is the dataset size.\n\nKamb & Ganguli assume translation equivariance, so for each pixel, we compare its surrounding patch (size $p_t$ at denoising step $t$) to every patch in the dataset, resulting in $O(n p_t m^2)$ complexity. With approximate vector search (e.g., using $k$ clusters), this reduces to $O(n p_t m^2 / k)$. Our model forgoes translation equivariance as we did not observe any difference in quality (reported in the main paper). Additionally, we are using a distinct per-pixel mask pattern. This leads to the algorithmic complexity of our algorithm being $O(n p_t m)$. While this makes approximate search harder, our implementation (Section C.5) is faster than even Kamb & Ganguli’s approximate version on benchmark-scale datasets. For larger datasets, we can match their $O(n p_t m / k)$ complexity by indexing masks per timestep and pixel.\n\n**Softmax**: As Reviewer 8kUK noted, computing softmax across a dataset is challenging. We address this with a numerically stable streaming implementation requiring just one pass and constant memory (see `src/utils.py` in the appendix files).\n\n**Algorithmic complexity table**:\n|Method|Wiener Filter|Kamb & Ganguli (exact)|Kamb & Ganguli (approx)|Ours (exact, implemented)|Ours (approx, not implemented)|\n|-|-|-|-|-|-|\n|Complexity|$O(m^2)$|$O(n p_t m^2)$|$O\\left(\\frac{n p_t m^2}{k}\\right)$|$O(n p_t m)$| $O\\left(\\frac{n p_t m}{k}\\right)$|\n\n\n## Choice of $A_t^q$: intuition, other alternatives and formatting\n\n### Motivation in Section A.3 (8kUK, Ws6m)\n\nWe agree with reviewers 8kUK and Ws6m that the main paper will benefit from some part of the motivation of our algorithm from the appendix to the main paper. We will update the main paper with a clearer intuition behind our algorithm: Kamb and Ganguli use rectangular binary masks fitted to the sensitivity fields of trained denoisers. In our paper, we show that the sensitivity fields of the trained denoisers come from the data covariance. In this paper, we are considering constant sensitivity fields, i.e., $A_t^q$ is not a function of $x$. If the sensitivity field is constant, the denoiser is linear, and we know that the optimal linear denoiser is the Wiener filter. Thus, the sensitivity field under the constant constraint is that of the Wiener filter.\n\nWe also revised the derivations for why the sensitivity fields should be binary:\n\n* **Generalized locality definition** We first consider a generalized notion of locality, with the constraint $f^q(x) = f^q(A^q_t x)$, where $A^q_t$ can be any matrix. Please note that indeed, when $A^q_t$ is invertible, $f^q$ is effectively unconstrained, i.e., there is no locality. When $A^q_t = 0$, $f^q$ does not depend on $x$.\n \n* **Generalized locality imposes subspace projection** Next we prove that imposing the generalized locality constraint is equivalent to adding a projection operator $\\mathbf{P_t^q}=(A_t^q)^T (A_t^q (A_t^q)^T)^\\dagger A_t^q$ (projection onto the row-space of $A^q_t$) to the optimal denoiser: $\\hat{f}^q(x, t) = \\sum_{i=1}^N (x^i_0)^q \\; \\text{softmax}_i\\left(-\\frac{1}{2\\sigma_t^2} \\|\\mathbf{P_t^q}(x - \\sqrt{\\alpha_t}x_0^i)\\|^2\\right)$\n\n* **Projection = Orthogonal change of basis + Masking** All projection operators can be written as an orthogonal change of basis $V_t^q$ followed by a masking operation $M^q_t$ (i.e., a diagonal 0-1 matrix): $\\mathbf{P_t^q} = (V_t^q)^TM^q_tV_t^q$\n\nThis explains why the only interesting generalized locality matrices are binary masks. We will move motivation for our method and additional intuition from Appendix A.3 to Section 3 of the main paper, in case if the paper is accepted. We will update our paper's appendix with proofs of the above statements.\n\n### Low-Rank Covariance Matrix\nIn response to Reviewer Ws6m’s suggestion, we conducted an additional comparison between the Wiener filter and our algorithm under low-rank projections of the covariance matrix. For the Wiener filter, low-rank approximation leads to visually smoother outputs; however, it reduces the correlation with outputs from trained diffusion models. Our algorithm does not benefit from using a low-rank projection of the covariance matrix and still demonstrates higher correlation than the Wiener filter. This indicates that low-rank structure alone does not account for the performance of learned denoisers. We believe this result further supports the distinctiveness and relevance of our proposed model. Please refer to our detailed qualitative and quantitative analysis in the personal response to Reviewer Ws6m.\n\nPlease let us know if you have any additional suggestions on how we can further improve our paper."}, {"review_id": "2fuLQGN4Wt", "reviewer": "Reviewer_Ws6m", "summary": "This work demonstrates that the locality in image diffusion models arises from the structure of data statistics (covariance structure) rather than the architectural inductive bias of the deep networks. They support this claim by showing  linear diffusion models consisting of Wiener filters which are purely determined by the data statistics outperform previously proposed analytical diffusion models based on architectural inductive bias. Building upon this finding, they replace the sensitivity fields used in previous patch-based analytical diffusion models with the binarized version of the sensitivity files of the linear diffusion model, which leads to improved performance.", "questions": "1. What are other choices of $A_t^q$? What do you think is the limit of the patch based analytical denoiser, suppose the optimal  $A_t^q$ can be found? What would be the property of the optimal $A_t^q$?\n\n2. Can you do ablation study on the effects of first doing low-rank approximation on the covariance matrix and then build the Wiener filter or the proposed method? Can low-rank approximation further improve the performance?", "limitations": "See the \"Strengths and Weakness\" section.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. This finding—that locality in image diffusion models derives from the data’s covariance structure rather than from the networks’ architectural inductive biases—has important implications, correcting the common belief that the models’ creative power primarily originates from network architecture [1].\n\n2. The proposed method improves upon previously proposed analytical diffusion models, hence contributing to the understanding of the generalization mechanisms of diffusion models.\n\nWeaknesses:\n\n1. The proposed model in equation (9) is kind of abrupt. While I understand this choice exploit the second order statistics of the dataset and also integrate the binary threshold as inspired by proposition A.10, it is still simple and fail short at  capturing more complex data structures. A natural question to ask is what is the limitation of tuning $A_t^q$? Can you make objective (8) in A.3 differentiable and tuning the sensitivity field end to end? I understand the Mahalanobis distance makes implementation difficult but I am curious if we can make some approximations.\n\n2. From my experience, building the linear denoisers with low-rank approximation of the full covariance matrices can bring significant improvement, visually similar to the results of (ours) presented in Figure 5. I encourage the author to document the performance of linear denoisers under different low-rank approximations. Similarly, the proposed method might also benefit from low rank approximation. Intuitively, many tail principal components can correspond to noise rather than informative images structures. Get rid of those can potentially improve the denoising performance.\n\n3. Section A, especially the part you motivate the proposed method should be put in the main text.\n\n4. There is a typo in line 171 of the appendix.\n\nKamb, Mason, and Surya Ganguli. \"An analytic theory of creativity in convolutional diffusion models.\" arXiv preprint arXiv:2412.20292 (2024).", "was_revised": true, "final_justification": "The rebuttal provides extra experiments and have addressed most of my questions. I believe the paper has make progress towards understanding the mechanism of generalization of diffusion models.", "rebuttal": "We thank Reviewer Ws6m for recognizing the broader implications of our findings, especially in “correcting the common belief” about generalization in diffusion models and for noting that our method “improves upon previously proposed analytical diffusion models.” Other reviewers, such as 8EDr and 8kUK, also appreciated our benchmarks, experimental rigor, and the clarity of our comparative analysis.\n\nWe also appreciate reviewers' suggestions on how to further strengthen the paper. In the rest of this reply, we first address suggestions shared by multiple reviewers and then will specifically address comments in this review.\n\n# Shared comments\n\n## Computational Efficiency\n\nReviewers 8EDr and 8kUK suggested including a comparison of runtime between the baselines. The runtimes and hardware used for all baselines are reported in Section C.5 of the appendix, and we provide additional context below. Our algorithm is more computationally efficient than Kamb & Ganguli's model both asymptotically and in practice, as we are not assuming translation equivariance. As reviewer 8kUK correctly mentioned, creating an efficient analytical denoiser is not the main focus of this paper; however, we agree that additional analysis of computational efficiency will further strengthen the paper.\n\nBelow, we provide a detailed analysis of the algorithmic complexity for the Wiener filter, Kamb & Ganguli's analytical model, and ours.\n\n### Algorithmic Complexity and the Softmax\n\nFor small-resolution images, the Wiener filter remains the most efficient at $O(m^2)$, where $m$ is the flattened image resolution. Both our model and Kamb & Ganguli’s require a dataset pass per inference step, leading to scaling linear in $n$, where $n$ is the dataset size.\n\nKamb & Ganguli assume translation equivariance, so for each pixel, we compare its surrounding patch (size $p_t$ at denoising step $t$) to every patch in the dataset, resulting in $O(n p_t m^2)$ complexity. With approximate vector search (e.g., using $k$ clusters), this reduces to $O(n p_t m^2 / k)$. Our model forgoes translation equivariance as we did not observe any difference in quality (reported in the main paper). Additionally, we are using a distinct per-pixel mask pattern. This leads to the algorithmic complexity of our algorithm being $O(n p_t m)$. While this makes approximate search harder, our implementation (Section C.5) is faster than even Kamb & Ganguli’s approximate version on benchmark-scale datasets. For larger datasets, we can match their $O(n p_t m / k)$ complexity by indexing masks per timestep and pixel.\n\n**Softmax**: As Reviewer 8kUK noted, computing softmax across a dataset is challenging. We address this with a numerically stable streaming implementation requiring just one pass and constant memory (see `src/utils.py` in the appendix files).\n\n**Algorithmic complexity table**:\n|Method|Wiener Filter|Kamb & Ganguli (exact)|Kamb & Ganguli (approx)|Ours (exact, implemented)|Ours (approx, not implemented)|\n|-|-|-|-|-|-|\n|Complexity|$O(m^2)$|$O(n p_t m^2)$|$O\\left(\\frac{n p_t m^2}{k}\\right)$|$O(n p_t m)$| $O\\left(\\frac{n p_t m}{k}\\right)$|\n\n\n## Choice of $A_t^q$: intuition, other alternatives and formatting\n\n### Motivation in Section A.3 (8kUK, Ws6m)\n\nWe agree with reviewers 8kUK and Ws6m that the main paper will benefit from some part of the motivation of our algorithm from the appendix to the main paper. We will update the main paper with a clearer intuition behind our algorithm: Kamb and Ganguli use rectangular binary masks fitted to the sensitivity fields of trained denoisers. In our paper, we show that the sensitivity fields of the trained denoisers come from the data covariance. In this paper, we are considering constant sensitivity fields, i.e., $A_t^q$ is not a function of $x$. If the sensitivity field is constant, the denoiser is linear, and we know that the optimal linear denoiser is the Wiener filter. Thus, the sensitivity field under the constant constraint is that of the Wiener filter.\n\nWe also revised the derivations for why the sensitivity fields should be binary:\n\n* **Generalized locality definition** We first consider a generalized notion of locality, with the constraint $f^q(x) = f^q(A^q_t x)$, where $A^q_t$ can be any matrix. Please note that indeed, when $A^q_t$ is invertible, $f^q$ is effectively unconstrained, i.e., there is no locality. When $A^q_t = 0$, $f^q$ does not depend on $x$.\n \n* **Generalized locality imposes subspace projection** Next we prove that imposing the generalized locality constraint is equivalent to adding a projection operator $\\mathbf{P_t^q}=(A_t^q)^T (A_t^q (A_t^q)^T)^\\dagger A_t^q$ (projection onto the row-space of $A^q_t$) to the optimal denoiser: $\\hat{f}^q(x, t) = \\sum_{i=1}^N (x^i_0)^q \\; \\text{softmax}_i\\left(-\\frac{1}{2\\sigma_t^2} \\|\\mathbf{P_t^q}(x - \\sqrt{\\alpha_t}x_0^i)\\|^2\\right)$\n\n* **Projection = Orthogonal change of basis + Masking** All projection operators can be written as an orthogonal change of basis $V_t^q$ followed by a masking operation $M^q_t$ (i.e., a diagonal 0-1 matrix): $\\mathbf{P_t^q} = (V_t^q)^TM^q_tV_t^q$\n\nThis explains why the only interesting generalized locality matrices are binary masks. We will move motivation for our method and additional intuition from Appendix A.3 to Section 3 of the main paper, in case if the paper is accepted. We will update our paper's appendix with proofs of the above statements.\n\n# Individual comments\n\n## RE: What are other choices of $A_t^q$\nThank you for your thoughtful comments and suggestions on the choice of $A_t^q$. We provided extended motivation earlier in this reply. We also attempted to perform gradient-based optimization of the matrix $A_t^q$ pre-binarization; however, in the limited time of the rebuttal, we were not able to arrive at conclusive results.\n\nOverall, our intuition is that for constant sensitivity fields (i.e., not a function of $x$), the denoiser is linear, and we know that the optimal linear denoiser is the Wiener filter. To further close the gap between analytical and trained denoisers, we will need to allow the sensitivity fields to depend on the input image, i.e., $A_t^q(x_t)$ should be a function of the noisy image. We agree that this is a very interesting direction to explore and plan to address it in future research.\n\n## RE: Low-Rank Covariance Matrix\nWe agree that a low-rank projection of the covariance matrix is a good method to prevent high-frequency artifacts in the generations produced by the Wiener filter. We performed additional experiments to see if removing the weakest principal components in the covariance matrix can help the analytical models better explain trained diffusers. For the Wiener filter, low-rank approximation leads to visually smoother outputs; however, it reduces the correlation with outputs from trained diffusion models. Our algorithm does not benefit from using a low-rank projection of the covariance matrix and still demonstrates higher correlation than the Wiener filter. This indicates that low-rank structure alone does not account for the performance of learned denoisers. We believe this result further supports the distinctiveness and relevance of our proposed model. Please refer to our detailed qualitative and quantitative analysis below.\n\n### Singular Values\nFirst, analyzed the singular values of the covariance matrices for each of the datasets. We observe that most of the total energy is contained in the first 10% of the singular values for all datasets, leaving a long tail of principal components. We plan to include the plots in the appendix of the final version if accepted.\n\n### Quantitative Comparison\nWe zero out the smallest singular values of the covariance matrix based on different thresholds of total energy. Both for ours and for the Wiener filter we measure the $r^2$ between predictions of the analytic models and a trained DDPM model, as done in the paper. The results are averaged across 16 generations and presented in the table below. A 0% SVD Energy Cutoff corresponds to using the full-rank covariance matrix.\n\n|SVD Energy Cutoff|0%|0.01%|0.05%|0.10%|0.50%|1.00%|5.00%|10.00%|50.00%|\n|-|-|-|-|-|-|-|-|-|-|\n|**Ours \\| CIFAR10**|**0.588**|0.501|0.312|0.245|0.176|0.176|0.192|0.194|-0.248|\n|**Ours \\| CELEBA HQ**|**0.897**|0.759|0.702|0.695|0.685|0.668|0.652|0.650|0.607|\n|**Ours \\| MNIST**|**0.492**|0.197|0.156|0.158|0.351|0.341|0.361|0.384|0.368|\n|**Wiener \\| CIFAR10**|**0.410**|0.408|0.409|0.382|0.338|0.260|0.223|0.168|-0.048|\n|**Wiener \\| CELEBA HQ**|**0.818**|0.817|0.805|0.797|0.766|0.739|0.649|0.625|0.548|\n|**Wiener \\| MNIST**|**0.469**|0.468|0.468|0.467|0.453|0.445|0.421|0.394|0.292|\n\n### Qualitative Comparison\nAdditionally, we generate visual comparisons between all the baselines across the different energy cut-off thresholds. We observe that removing a small portion of the singular values (e.g., 0.01% in total energy) indeed smooths out the images generated by the Wiener filter and makes the results more visually appealing. However, as we can see from the table above, this does not lead to a better correlation with trained diffusers. For our method, projecting the covariance matrix to a low rank does not seem to visually improve the results—it only blurs the images. We plan to include the visual comparisons in the final version of the paper if accepted. *Please note that we cannot attach the images to this response due to the new NeurIPS guidelines*.\n\n### Conclusion\n*In conclusion, based on both qualitative and quantitative comparisons across multiple energy cut-off thresholds, we did not find evidence that low-rank projection of the covariance matrix in analytical models leads to a better explanation of trained denoisers.*\n\n### RE: typo\nThank you for highlighting the typo. We fixed it.\n\nThank you for your thoughtful review, and please let us know if you have any additional suggestions on how we can further improve our paper."}]

Author rebuttal:

Meta-review: This paper received three accept ratings and one borderline accept rating. The reviewers found several weaknesses, which were addressed by the rebuttal comments. The paper is well-written, addresses an important topic and the experiments are convincing, which justify an accept decision on behalf of the AC.
Decision: Accept (spotlight)