Title: Locality in Image Diffusion Models Emerges from Data Statistics
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion models

generative models

ideal denoiser
Abstract: Recent work has shown that the generalization ability of image diffusion models arises from the locality properties of the trained neural network. In particular, when denoising a particular pixel, the model relies on a limited neighborhood of the input image around that pixel, which, according to the previous work, is tightly related to the ability of these models to produce novel images. Since locality is central to generalization, it is crucial to understand why diffusion models learn local behavior in the first place, as well as the factors that govern the properties of locality patterns. In this work, we present evidence that the locality in deep diffusion models emerges as a statistical property of the image dataset and is not due to the inductive bias of convolutional neural networks, as suggested in previous work. Specifically, we demonstrate that an optimal parametric linear denoiser exhibits similar locality properties to deep neural denoisers. We show, both theoretically and experimentally, that this locality arises directly from pixel correlations present in the image datasets. Moreover, locality patterns are drastically different on specialized datasets, approximating principal components of the data’s covariance. We use these insights to craft an analytical denoiser that better matches scores predicted by a deep diffusion model than prior expert-crafted alternatives. Our key takeaway is that while neural network architectures influence generation quality, their primary role is to capture locality patterns inherent in the data.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work demonstrates that the locality in image diffusion models arises from the structure of data statistics (covariance structure) rather than the architectural inductive bias of the deep networks. They support this claim by showing  linear diffusion models consisting of Wiener filters which are purely determined by the data statistics outperform previously proposed analytical diffusion models based on architectural inductive bias. Building upon this finding, they replace the sensitivity fields used in previous patch-based analytical diffusion models with the binarized version of the sensitivity files of the linear diffusion model, which leads to improved performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. This finding—that locality in image diffusion models derives from the data’s covariance structure rather than from the networks’ architectural inductive biases—has important implications, correcting the common belief that the models’ creative power primarily originates from network architecture [1].\n\n2. The proposed method improves upon previously proposed analytical diffusion models, hence contributing to the understanding of the generalization mechanisms of diffusion models.\n\nWeaknesses:\n\n1. The proposed model in equation (9) is kind of abrupt. While I understand this choice exploit the second order statistics of the dataset and also integrate the binary threshold as inspired by proposition A.10, it is still simple and fail short at  capturing more complex data structures. A natural question to ask is what is the limitation of tuning $A_t^q$? Can you make objective (8) in A.3 differentiable and tuning the sensitivity field end to end? I understand the Mahalanobis distance makes implementation difficult but I am curious if we can make some approximations.\n\n2. From my experience, building the linear denoisers with low-rank approximation of the full covariance matrices can bring significant improvement, visually similar to the results of (ours) presented in Figure 5. I encourage the author to document the performance of linear denoisers under different low-rank approximations. Similarly, the proposed method might also benefit from low rank approximation. Intuitively, many tail principal components can correspond to noise rather than informative images structures. Get rid of those can potentially improve the denoising performance.\n\n3. Section A, especially the part you motivate the proposed method should be put in the main text.\n\n4. There is a typo in line 171 of the appendix.\n\nKamb, Mason, and Surya Ganguli. \"An analytic theory of creativity in convolutional diffusion models.\" arXiv preprint arXiv:2412.20292 (2024).\n\nQUESTIONS:\n1. What are other choices of $A_t^q$? What do you think is the limit of the patch based analytical denoiser, suppose the optimal  $A_t^q$ can be found? What would be the property of the optimal $A_t^q$?\n\n2. Can you do ablation study on the effects of first doing low-rank approximation on the covariance matrix and then build the Wiener filter or the proposed method? Can low-rank approximation further improve the performance?\n\nLIMITATIONS:\nSee the \"Strengths and Weakness\" section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study challenges the prevailing assumption that CNN inductive biases primarily explain the gap between optimal diffusion denoisers and practical applications, and demonstrates that locality arises from underlying statistics of image datasets. The authors develop an analytical denoiser based on principal component analysis based on these statistical principles, which yields results that are closer to the true diffusion model behaviour than previous expert-designed approximations. It shows that local sensitivity in diffusion models arises from learned data statistics rather than architectural inductive biases, and identifies that analytically derived optimal linear filters-dependent only on training data, accurately predict both local and non-local spatial sensitivities observed in trained denoising networks, unexpectedly outperforming previous analytical models based on optimal denoisers on new quantitative benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nThis work demonstrates that local sensitivity in diffusion models arises from learned data statistics rather than architectural inductive biases. It is demonstrated that analytically derived optimal linear filters which depend only on training data can accurately predict local and non-local spatial sensitivities observed in trained denoising networks, and surprisingly outperform previous analytical models based on optimal denoisers on new quantitative benchmarks. ​​ ​​The authors explore the mechanism of locality in image diffusion models from the perspective of data statistics and provide detailed visual proofs. \n\nWeaknesses: The author gives a very detailed discussion about his proposed strategy, but it seems to increase some computational efficiency, so I am curious about the experimental results of this. Although the method performs significantly, a key practical consideration is the computational overhead. Could the authors provide the above results? I am not very familiar with this field; therefore, if there are enough reviewers, please ignore my comments.\n\nQUESTIONS:\nRefer to weaknesses part.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper challenges the current understanding of the generalization capabilities of diffusion models. Specifically, the authors analyze the gap between the optimal denoiser and approximate denoisers learnt by neural networks. It is demonstrated, theoretically and empirically, that the locality of the learnt denoisers can be explained by data statistics, not by inductive biases of the neural network architectures.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. This paper is very well-written. It reads somewhat like a tutorial, which I appreciate. The reader is guided through the problem statement, current problem understanding and approaches, experiment design, and results.\n2. I believe that the research problem is important, especially given that this work challenges the understanding of the machine learning community and provides an alternative explanation of the generalization capabilities of denoising diffusion models.\n3. The authors provide the code to facilitate the replication of results. The codebase is neatly organized.\n4. Experiments are clearly explained and convincingly support the claims made by the authors.\n5. Proofs in the appendix are readable and easy to follow, given the explanations throughout.\n\n**Weaknesses:**\nI did not spot any major weaknesses. Minor weaknesses/suggestions:\n\n1. The README in the codebase could be written better. There are typos, and not immediately obvious which scripts should be run. This should be improved upon acceptance.\n2. Figure 5 - at a first glance, the \"Ours NN\" label can be interpreted (at least I initially did) as \"Ours - Neural Network\", as in \"We built a Neural Network model using our theoretical insights\". Perhaps it can be relabeled somehow to instantly be read as \"nearest neighbor in the data\"?\n    * On second thought, I am not sure I understand what is the purpose of this last row. Why show this only for the proposed method, and not other baselines?\n3. Table 1 - would be nice to have some error bars/statistical significance tests. Although the difference sometimes seems quite large. \n    * I can see that the standard deviations are reported separately in a table in the appendix, but that is not very readable to me, having to go back and forth looking and the values table and the standard deviation table\n\nQUESTIONS:\n1. Is the difference between the proposed method and the baselines in Table 1 statistically significant in all experiments?\n2. I think that Figure 5 should be improved. My suggestion would be to remove the last row unless the authors can convince me that it is relevant to that experiment (in which case, I believe that it should also be presented for other baselines)\n\nLIMITATIONS:\nThe limitations are properly discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is a nice response to the recent paper Kamb Ganguli, which propose generalization comes from local equivariant inductive bias. This paper showed that the local receptive field is dataset dependent and universal across architectures, suggesting it’s a property of dataset than architecture. They further leveraged the Gaussian linear score solution / Wiener matrix to analytically approximate the receptive field and use it to derive another closed form diffusion sampler.  They showed empirically that their analytical diffusion better approximates actual diffusion model than the Gaussian linear one and local equivariant one.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n\n- I agree and am convinced the data statistics place a larger role in determining the sensitivity field. The manipulation of dataset statistics and the UNet vs DiT comparison are very convincing. This is a important and nice point to make given the influential Kamb Ganguli paper.\n- Comparison to related works are carefully made. \nThe qualitative and quantitative benchmark in Table 1 and Figure 5. are nice. Previous work Kamb & Ganguli esp. lack such quantitative and systematic comparison with the Wiener filter results (Wang Vastola 2023, 2024), so it’s good to show quantitatively the linear case works well still.   Using another trained DDPM as reference is also very nice control.\n\n### Weakness\n\n- Maybe in the full paper some of Appendix A and more intuition should be put into main paper to support the Eq. 9? I think the intuition is the using the binarized corresponding qth row of Wiener matrix as the “receptive field” mask.\n- Theoretical justification (Supplementary section A.3) it’s a bit confusing to follow.\n    - The notation in the  is sometimes not consistent. in line 119 A_t^q is d by d but in the definition in L96 it’s dxd by dxd.\n    - The generalized locality definition A.8 does not make much sense for arbitrary linear operators. It cannot be true for a general A_t^q\n    - I imagine there is a clearer and more intuitive way to write this section.\n\nQUESTIONS:\n### Questions\n\n- Though it’s not the main point of this paper, is there a comparison of efficiency / runtime of your method and Kamb Ganguli method & Wiener filter? In other word Is this new method practical? Wouldn’t the softmax easily run into computational out of memory problems when the dataset goes up?\n\nLIMITATIONS:\nN.A.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}