Title: Spectral Analysis of Diffusion Models with Application to Schedule Design
Venue: NeurIPS 2025
Accepted: yes
Keywords: Spectral analysis

Designing scheduling

Diffusion models
Abstract: Diffusion models (DMs) have emerged as powerful  tools for modeling complex data distributions and generating realistic new samples. Over the years, advanced architectures and sampling methods have been developed to make these models practically usable. However, certain synthesis process decisions still rely on heuristics without a solid theoretical foundation. 

In our work, we offer a novel analysis of the DM's inference process, introducing a comprehensive frequency response perspective. Specifically, by relying on Gaussianity assumption, we present the inference process as a closed-form spectral transfer function, capturing how the generated signal evolves in response to the initial noise. We demonstrate how the proposed analysis can be leveraged to design a noise schedule that aligns effectively with the characteristics of the data. The spectral perspective also provides insights into the underlying dynamics and sheds light on the relationship between spectral properties and noise schedule structure. Our results lead to scheduling curves that are dependent on the spectral content of the data, offering a theoretical justification for some of the heuristics taken by practitioners.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work analyzes how to schedule the noise based on the data features from a spectral perspective. To achieve this goal, this work first presents a spectral analysis on the reverse process based on the Gaussian target distribution. Then, based on the spectral analysis, they propose an optimization problem to obtain the noise schedule specific to the given data feature. The simulation and real-world experiments (including VP and VE-based models) also support their theoretical results.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1.\tThe spectral analysis for the noise schedule design is novel and interesting.\n\n2.\tThe simulation and real-world experiments also support their theoretical results.\n\nWeakness:\n\n1.\tThe analysis of this work is limited to the Gaussian distribution, which does not match the real-world data distribution. The high-dimensional Gaussian assumption will also lead to a time-consuming optimization process for the given optimization problem. \n\nFor the above problems, some works [1][2] assume the target data distribution admits a low-dimensional linear subspace and the latent distribution is Gaussian. [2] further prove that under the linear subspace assumption, the optimization process of diffusion models is equivalent to PCA.\n\nAs mentioned in the limitation part of this work, PCA is a promising method, and it will be interesting to consider this modeling in future work. \n\n[1] Chen, Minshuo, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. \"Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data.\" In International Conference on Machine Learning, pp. 4672-4712. PMLR, 2023.\n\n[2] Yang, Ruofeng, Bo Jiang, Cheng Chen, Baoxiang Wang, and Shuai Li. \"Few-shot diffusion models escape the curse of dimensionality.\" Advances in Neural Information Processing Systems 37 (2024): 68528-68558.\n\nQUESTIONS:\nPlease see the Weakness part.\n1.\tIt would be better to discuss the technique challenge when considering GMM target distribution (it has a closed-form score function, but does not have a closed-form denoiser).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDiffusion model has a strong ability to model complex data distributions and generate realistic new samples. However, its generation process depends on a heuristically chosen noise schedule without a solid theoretical basis. In this paper, assuming the target distribution is Gaussian, the authors demonstrate the inference process as a closed-form spectral transfer function, showing how the generated signal develops in response to the initial noise. They then use this analysis to determine an optimal noise schedule that effectively matches the data's characteristics.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Based on the Gaussianity assumption, a novel theoretical spectral analysis of the discrete diffusion reverse process is presented, along with a closed-form expression for its spectral transfer function.\n- Formulates an optimization problem to find an optimal noise schedule that matches data characteristics, regardless of the number of diffusion steps.\n- A well-structured paper with high-quality writing.\n\nWeaknesses:\n- The Gaussianity assumption is too strong and unrealistic.\n- The performance of the identified noise schedule does not appear to outperform other schedules.\n\nQUESTIONS:\n1. The Gaussianity assumption about the target distribution is somewhat unrealistic because it is unlikely to occur in real datasets. This reduces the effectiveness of the proposed method when applied to actual data, as shown in Section 6.3. Figure 4 does not demonstrate the significant performance difference of the learned noise schedule compared to heuristic baselines.\n\n2. Even in the synthetic case where the assumption holds, Figure 1.3 indicates that the performance gap (Wasserstein distance) of the learned noise schedule relative to others is minimal. This raises the question of whether a complex \"trained\" noise scheduler is truly necessary. Is the cost of tuning hyperparameters for heuristic noise schedules (e.g., cosine, sigmoid) higher than the cost of discovering the \"trained\" noise schedule? If not, what is the main reason for developing this \"trained\" noise schedule?\n\n3. The statement in Lines 300-301 about the mean drift, “Additionally, as the depth of the diffusion process increases, the bias value tends to grow, regardless of the selected noise schedule.”, diminishes the motivation behind this paper.\n\n4. Other minor points:\n- In Eq 14, what are $\\epsilon_0$, $\\epsilon_S$? Additional explanations are needed.\n- Visualizations of some figures (e.g., Figures 1b,c, and 4a, b, c) could be improved by combining the legends horizontally and adding color to make certain lines more distinguishable.\n\nLIMITATIONS:\nPlease see the above questions", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAn important aspect of diffusion model sampling is the choice of time/noise discretization, or \"noise schedule\". A bad noise schedule can yield poor sample quality, or at least make sampling require more steps for good sample quality than with a better schedule. Given that noise schedules tend to be chosen on an empirical basis, the authors of this work propose to use theoretical insight to design better noise schedules.\n\nThe key idea is that Gaussian/linear model sampling dynamics is solvable, even when time/noise is discretized. The authors use this fact to express the final sample in terms of the noise schedule and initial noise seed, and then to compute final sample quality (in terms of the Wasserstein-2 distance to the 'true' Gaussian distribution) in terms of the noise schedule. They can then compare their \"spectral recommendation\" to other schedules on real and toy data sets.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths.** I'm sympathetic to the idea of using theoretical insight to inform the design of things like noise schedules. The idea to exactly solve the (discrete-time) sampling dynamics associated with the Gaussian model, and use this result to determine the theoretical 'goodness' of noise schedules by computing Wasserstein-2 distances (and KLDs), is interesting. I have done similar math before, and do not doubt that the authors' math is correct.\n\n\n\n**Weaknesses.** My main concern is that neither the theoretical analysis nor empirical analysis appear to be major contributions. The central theoretical result of the paper is the solution of the Gaussian model's sampling dynamics for an arbitrary $\\alpha_t$, but this has been done many times now, as this paper explicitly mentions (see, e.g., line 92). Given this result, it's easy to compute other downstream things like Wasserstein-2 distances and KLDs. The main difference, as far as I can tell, is that this paper's result is in *discrete* rather than continuous time. Although this paper appears to view this as a major technical distinction between its calculations and prior work (see lines 215-216), I kind of disagree. Linear recurrence relations are essentially as easy to solve as linear ODEs. The calculation may be valuable, but it's extremely closely related to work that's been done before.\n\nOn the empirical side, the authors use a few data sets to illustrate that their method gives reasonable results. But the evaluation criteria are a little weird, since the authors are mainly concerned with the extent to which their method reproduces the *Gaussian approximation* of a given data set, rather than returns high quality samples. The most interesting plot is Fig. 4a, since it depicts FID scores (a measure of sample quality) for CIFAR-10. According to the plot, the EDM-recommended noise schedule does as good as their method. Since this is the only sample-quality-related plot I can see in the main text, I'm not convinced that the authors' proposed method is particularly good.\n\nA subtle point which the authors don't mention is that even coming up with an 'optimal' noise schedule requires fitting a Gaussian to a data set, which is not always possible for extremely high-dimensional data sets, and is (at least to my knowledge) essentially impossible for text-to-image conditional models. If the EDM schedule performs about as well, but doesn't require this step, that's worth discussing explicitly. \n\nI find all of the figures kind of hard to parse, since the text and labels are very small. It would really help the paper if the sizes of axis labels, numbers, and so on were greatly increased.\n\nA few minor nitpicks. The authors mention the usefulness of a \"spectral\" perspective on diffusion models, which has parallels to signal processing. But I find this link really misleading. In signal processing settings, spectral analysis involves transforming the time domain to a frequency domain; here, since the data (e.g., an image) may not naturally have a time component, the eigendirections of the covariance matrix do not necessarily have any kind of frequency-like interpretation. Also, the aforementioned papers studying the Gaussian model also do this diagonalization trick, so the spectral idea is far from new. \n\nThe authors should probably cite some foundational papers in their first paragraph, where they write \"Diffusion Models (DMs) have become ...\" There are a number of typos, like misspelled words and missing periods (see, e.g., line 270). These should be fixed.\n\nQUESTIONS:\n1. Is it possible to define a measure of sample quality for the audio data set? It seems important to have a sense for how good the schedule is doing at generating high quality samples, not just at reproducing the data set's Gaussian approximation.\n2. Have the authors identified a case where their recommendation is *particularly* effective? In the case of CIFAR-10, it seems like the EDM schedule does about as well, and does not require a Gaussian fit to use. \n3. How feasible / computationally costly is it to do a Gaussian fit on various high-dimensional data sets? Can the authors discuss this point explicitly?\n\nLIMITATIONS:\nI think the extent to which their proposal yields meaningfully higher sample quality and/or efficiency should be discussed in more detail, and ideally the authors identify one or more cases where their approach yields a large improvement as opposed to a small one. The authors should also discuss the point that their method requires a Gaussian fit, which is difficult or impossible for large conditional diffusion models.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel spectral analysis of the discrete diffusion reverse process. This leads to a closed-form expression for the spectral transfer function, which characterizes how the final generated signal results from a linear transformation of the initial Gaussian noise (Theorems 3.5 and 3.6).\n\nBuilding on this analysis, the paper introduces an optimization problem to compute a data-dependent noise schedule that aligns with the spectral properties of the target data. This approach allows for the derivation of an optimal noise schedule without being constrained by heuristic bounds or fixed step counts.\n\nThe experimental results—spanning synthetic setups and real-world data under Gaussian assumptions—demonstrate that the proposed spectral schedule outperforms standard heuristics (e.g., linear or cosine). Specifically, results on CIFAR-10, MUSIC, and SC09 datasets show that the optimal schedules adapt to the unique spectral characteristics of each dataset, highlighting the limitations of one-size-fits-all heuristics.\n\nSTRENGTHS AND WEAKNESSES:\nNote: I have not verified the theoretical proofs in full detail, and my comments are based on my best understanding. I am open to adjusting my evaluation in light of other reviews.\n\nStrengths\n\n•\tThe idea of a data-dependent, adaptive noise schedule is intuitive and well motivated.\n\n•\tThe method is inference-time only—it does not require retraining the denoiser.\n\n•\tThe method is designed for discrete processes (DDIM/DDPM) rather than continuous-time formulations (SDE/ODE), which aligns better with practical implementations.\n\nWeaknesses\n\n•\tAs acknowledged in the paper’s limitations section, computing the optimal spectral schedule can be slow—scaling with the number of diffusion steps and dataset size—taking several minutes for large setups.\n\n•\tThe assumption that the target data is drawn from a Gaussian distribution may be too strong for real-world applications. While the experiments include pretrained diffusion models, CIFAR-10 is a relatively small and low-resolution dataset and may not strongly violate this assumption.\n\nQUESTIONS:\n-\tIn Section 6.3 (lines 338–340), the authors use a pretrained denoiser trained with a continuous noise schedule for the CIFAR-10 dataset. Why not use a pretrained model with a standard discrete schedule, such as this one: https://huggingface.co/google/ddpm-cifar10-32 \n\n-\tThe current approach involves an iterative optimization process that scales with the number of diffusion steps. Is it possible to approximate or bypass this optimization to accelerate computation?\n\n-\tHow would the results differ if the denoiser were trained from scratch using the optimized spectral schedule, rather than applying it only at inference time?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}