Title: An Analytical Theory of Spectral Bias in the Learning Dynamics of Diffusion Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Spectral bias

analytical theory

learning dynamics

diffusion

flow matching

deep linear network
Abstract: We develop an analytical framework for understanding how the learned distribution evolves during diffusion model training. 
Leveraging the Gaussian equivalence principle, we derived exact solutions for the gradient-flow dynamics of weights in one or two layer linear or linear convolutional denoiser settings with arbitrary data, where linear networks converge along principal components, and convolutional networks converge along Fourier modes. 
Remarkably, these solutions allow us to derive the generated distribution in closed-form and its KL-divergence through training. 
These analytical results expose a pronounced \emph{spectral bias}, i.e. for both weights and generated distributions, the convergence time of a mode follows an inverse power law of its variance.  
Empirical experiments on both Gaussian and natural image datasets demonstrate that the power-law spectral bias—remain robust even when using deeper or convolutional architectures. 
Our results underscore the importance of the data covariance in dictating the order and rate at which diffusion models learn different modes of the data, providing potential explanations of why earlier stopping could lead to incorrect details in image generative model.


Peer reviews:
comments: SUMMARY:
This manuscript delivers a rigorous analysis of diffusion model training dynamics under a linear denoiser assumption. Assuming the data follow a Gaussian distribution, the optimal denoiser becomes a linear operator whose convergence rate is governed by the eigenvalue spectrum of the data covariance matrix. While the theoretical results hold exactly for linear denoisers, the authors demonstrate through experiments that nonlinear neural networks exhibit qualitatively similar convergence behaviour.

STRENGTHS AND WEAKNESSES:
**Strengths**

- Provides a sound theoretical framework linking covariance eigenvalues to denoiser convergence speed.
- Empirical validation supports the theoretical predictions and reveals analogous trends in nonlinear denoisers.
- Section 6.2’s experiments highlighting distinct CNN behaviors enrich the manuscript.
- This work will be important in helping researchers understanding the generalization power of diffusion models.

**Weaknesses**

- The analysis is limited to linear denoisers. However, the authors do acknowledge this limitation and I wouldn't consider it a major drawback, as the empirical results suggest that the theory is still relevant for nonlinear denoisers.

QUESTIONS:
1. In line 94, does $s_\theta$ denote the score function and $D_\theta$ the denoiser? Please clarify to avoid confusion.
2. Equation (6) defines $W_\sigma=\Sigma(\Sigma + \sigma^2 I)^{-1}$, but Equation (3) suggests $W_\sigma=(\Sigma + \sigma^2 I)^{-1}\Sigma$. Which form is correct?
3. In Section 5.2, Proposition 5.3 uses a Fourier basis, whereas Proposition 5.4 relies on principal components of local patches. What justifies this change given that the filter width is the only differing parameter?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a theoretical analysis of the learning dynamics in diffusion models, focusing on the simplified setting of linear denoisers trained on data equivalent to a single anisotropic Gaussian distribution. The authors derive a closed-form solution for the weight dynamics under gradient flow (Equation 7) and, assuming commutativity of the weight matrices across noise scales, provide an analytical expression for the generated data distribution at any point during training (Proposition 4.2).\n\nThe central implication of such is the existence of a \"spectral bias\", where the time required to learn an eigenmode is inversely proportional to its variance ($\\tau_k\\propto \\lambda_k^{-1}$). This results in a coarse-to-fine learning trajectory, with high-variance modes being learned much faster than low-variance ones. The analysis is extended to deeper models, such as a two-layer symmetric linear network (Proposition 5.1), and to linear convolutional networks, where a similar spectral bias is predicted in the Fourier domain (Propositions 5.3 and 5.4).\n\nTo validate these theoretical predictions, the authors conduct empirical studies using MLP and CNN-based denoisers. The results show that the learning dynamics of MLPs align well with the theory. In contrast, CNNs show a different behavior where many modes are learned near-simultaneously, suggesting that the bias shifts from the global image covariance to the patch-level covariance, a phenomenon the authors attribute to the inductive bias of local convolutions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The paper provides a thorough and rigorous gradient-flow analysis for the learning dynamics of linear denoisers, with extension to two-layer and convolutional cases.\n* The theoretical claims are supported by empirical validation on both synthetic and real-world image datasets (Figures 3 and 9).\n\n\nWeaknesses:\n* The primary limitation is the reliance on strong simplifying assumptions, a linear denoiser and data drawn from a single Gaussian distribution which may not fully capture the complexity of practical diffusion models.\n* The extensions to symmetric linear and convolutional networks feel less developed. For example, while the authors attribute the faster convergence of CNNs to the theoretical \"N-fold speed-up\" from weight sharing, this connection is not rigorously established. The empirical speed-up could be influenced by many confounding factors in a modern U-Net architecture and optimization.\n* The presentation could be streamlined to better emphasize the central theoretical contributions. For instance, the key result for weight dynamics (Equation 7) is presented as an intermediate step rather than a standalone proposition or theorem. The frequent use of subsections for brief interpretive points also fragments the narrative flow.\n\nQUESTIONS:\nCan you elaborate on Figure 17~20? While I agree they resemble the increase-decrease separation in Figure 2, the tailing eigenmodes seem not to show strong incremental learning, also the scatter points seem not to agree with the fitted line (both increase fit and decrease fit).\n\nLIMITATIONS:\nWhile I commend the authors for their effort in building an analytical theory, the paper offers limited practical insight due to its restrictive core assumptions. Contemporary work is beginning to offer precise insights into phenomena like the effect of early stopping (https://arxiv.org/abs/2505.17638), but this paper's impact is constrained by its idealized setting.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an analytical study of how diffusion models learn the data distribution over time during training. Focusing on linear denoisers trained with full-batch gradient flow, the authors derive closed-form expressions for both the model parameters and the generated distribution. They find that modes with higher variance are learned faster, leading to what they describe as a spectral bias. This inverse-variance effect is extended to deeper linear networks and linear convolutional architectures. The authors also run experiments with MLP and convolutional UNet models on synthetic and image datasets to compare against the theoretical predictions.\n\nSTRENGTHS AND WEAKNESSES:\n**Quality**\n\nThe analytical results are clearly stated under well-specified assumptions. The paper links training dynamics to a mode-wise description of learning speed, which is insightful. The experimental validation is well done, spanning both synthetic and natural image data. \n\nA limitation is that the theory relies on idealized conditions (linear networks, full-batch gradient flow, and aligned initialization) which restricts its direct applicability to practical settings. The experiments with CNNs show deviations from the theory, which are acknowledged.\n\n\n**Clarity**\n\nThe paper is generally well-written and well-structured. Assumptions are stated clearly, and the figures effectively illustrate the main points. The appendix is very detailed, and it's clear that the authors have put significant effort into it.\n\n**Significance**\n\nUnderstanding how diffusion models learn the data distribution over time is an important problem, and the paper provides theoretical insights on this front. The proposed framework gives a nice picture of how different parts of the data are learned at different speeds, which could be relevant for understanding failure modes or improving training strategies.\n\nThat said, the practical impact is somewhat limited by the idealized assumptions behind the theory, though qualitative features of the analysis are noted in more practical settings.\n\n\n**Originality**\n\nThe work offers an interesting perspective on spectral bias in diffusion models, focusing on the training phase rather than the sampling process. The mode-wise view of convergence time and its link to inverse-variance scaling seems novel. Some elements of the setup, like the linear score approximation and Gaussian assumption, are standard, but the way they’re combined here leads to new observations about the learning dynamics in this context.\n\nQUESTIONS:\n**1.** Is there any difference between the EDM and DSM formulations, or are they equivalent in the context considered here?\n\n\n**2.** Do the authors consider feasible and useful to carry a similar analysis with a random features model parametrising the score? While still linear in the parameters, the model is non linear in the data.\n\n\n**3.** I believe there is a typo in Line 130: it should refer to Appendix C.4, not C.3.\n\n**4.** In the discussion, the authors mention local convolutional structure. Could they elaborate on how one might, in principle, build a theoretical framework to handle this?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the spectral bias in denoising score matching. They analyze the learning dynamics of diffusion models, focusing on how the generated distribution evolves throughout training under some strong assumptions to ensure analytical traceability. Additionally, they also empirically test their insights extracted from the theoretical analysis in a more realistic setting. They provide both theoretical and empirical evidence for spectral bias i.e., weights converge faster along the eigenmodes or Fourier modes with high variance, and the learned distribution recovers the true variance first along the top eigenmodes.\n\nSTRENGTHS AND WEAKNESSES:\n# Strenghts\n1. This paper studies an interesting problem with a very sensible methodology. They first start by considering the simple toy setting with linear denoisers where everything is analytically tractable, and then empirically validate the results in a more realistic setting.\n2. The presentation is clear. And the paper is well structured.\n# Weaknesses\n1. It seems that the experimental results deviate more and more from the predicted theory as we move towards the more realistic settings\n -  Even before introduction of Convolutions in Experiment 2 with MLPs on natural images the observed power-law exponent is significantly attenuated(from the theoretical prediction of -1 to -0.48)\n -  In the most realistic experiment(experiment 3) the observed power-law exponent is approximately 0, meaning that the spectral bias is absent.\n\nIf in realistic settings the purported phenomenon(spectral bias) is no longer observable then I am not sure what is the upshot of the analysis in this paper. Is it just an idiosycracy of simplistic setting studied in section 4? Or is there reason to believe that it is a more general phenomena for all diffusion models? I think a more detailed discussion connecting the analysis connecting the analytically tractable setting and the realistic setting would be highly beneficial, especially since the empirical analysis is inconclusive.\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nLimitations are discussed in Section 7.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}