The Diffusion Process as a Correlation Machine: Linear Denoising Insights
Authors: Dana Weitzner
Mauricio Delbracio
Peyman Milanfar
Raja Giryes
Venue: Accepted by TMLR ()

Abstract: Recently, diffusion models have gained popularity due to their impressive generative abilities. These models learn the implicit distribution given by a training dataset, and sample new data by transforming random noise through the reverse process, which can be thought of as gradual denoising. In this work, to shed more light on the evolution of denoisers in the reverse process, we examine the generation process as a ``correlation machine'', where random noise is repeatedly enhanced in correlation with the implicit given distribution. 
To this end, we explore the linear case, where the optimal denoiser in the MSE sense is known to be the PCA projection. This enables us to connect the theory of diffusion models to the spiked covariance model, where the dependence of the denoiser on the noise level and the amount of training data can be expressed analytically, in the rank-1 case.
In a series of numerical experiments, we extend this result to general low rank data, and show that low frequencies emerge earlier in the generation process, where the denoising basis vectors are more aligned to the true data with a rate depending on their eigenvalues. This model allows us to show that the linear reverse process is a generalization of the prevalent power iteration method, where the generated distribution is composed of several estimations of the given covariance, in varying stages of convergence.
Finally, we empirically demonstrate the applicability of our findings beyond the linear case, in the Jacobians of a deep, non-linear denoiser, used in general image generation tasks.

Published in Transactions on Machine Learning Research (10/2025)
The Diffusion Process as Correlation Machine: Linear De- noising Insights
Dana Weitzner Tel Aviu University
danawettaee&@maaltaaccc.
Mauricio Delbracio Google
mdelhrra@goonle...o
Peyman Milanfar Google
milanfar@ggoggle.com com
Raja Giryes Tel Aviu University
raja@tauer.ter tau. ac.
Reviewed on OpenReview: https: // openreview. net/ forum? id- FGDJOc27rs
Abstract
Recently, diffusion models have gained popularity due to their impressive generative abilities. These models learn the implicit distribution given by a training dataset, and sample new data by transforming random noise through the reverse process, which can be thought of gradual denoising. In this work, to shed more light the evolution of denoisers in the reverse process, examine the generation process a "correlation machine", where random noise is repeatedly enhanced in correlation with the implicit given distribution. To this end, we explore the linear case, where the optimal denoiser in the MSE sense is known to be the PCA projection. This enables us to connect the theory diffusion models to the spiked covariance model, where the dependence of the denoiser on the noise level and the amount of training data can be expressed analytically, in the rank-1 case. In series extend this result to general low rank data, and show that low frequencies emerge earlier in the generation process, where the denoising basis vectors are more aligned to the true data with rate depending their eigenvalues. This model allows us to show that the linear reverse process is a generalization of the prevalent power iteration method, where the generated distribution is composed of several estimations of the given covariance, in varying stages of convergence. Finally, we empirically demonstrate the applicability of our findings beyond the lincar case, in the Jacobians of deep, non-lincar denoisers, used in general image generation lasks.
of numerical experiments,
1 Introduction
Recently, diffusion models have gained much popularity as very successful generative models, showcasing impressive performance in image generation lasks (Dhariwal & Nichol, 2021 Ho al. 2020 Song Ermon, 2019 Song et al.) 2021c). These models learn the implicit distribution given by training dataset and sample new data by transforming random noise inputs through reverse diffusion process, which can be thought of as gradual denoising. More formally, it has been shown in Kadkhodaie et al. (2024) that learning the underlying distribution equivalent to optimal denoising at all noise levels.
In order shed more light onto the mechanism behind the success of diffusion models, in this work we analyze the behavior of denoisers in the context of image generation, where pure noise gradually processed into a sample from given (implicit) distribution by gradual denoising. Unlike other works, e.g. Kadkhodaie

---

Published in Transactions on Machine Learning Research (10/2025)
et al 2024). we focus the denoisers throughout the generation process, and not only on the final generated data.
To this end, we suggest the following simple model to illustrate our point. Consider the class of linear denois-
ets, where the optimal denoiser in MSE sense has a dosed-form solution. We explore two linear denoising trajectories, corresponding to the DDPM (Ho et al., 2020) and DDIM (Song et al.) 2021a) approaches of sampling. To simulate the diffusion generation process, we learn a series of projections onto noisy data different noise levels, and use them to transform pure noise into samples from the underlying distribution. Given this simple model we can inspect the evolution of eigenvectors spanning gradual projections with decrcasing noise levels, as well as the distribution of the generated data samples.
We show that the correlation of the noisy basis eigenvectors with their clean version decays as the noise level increases, with a rate determined by the eigenvalues and the size of the training dataset. In other words, we show that low frequencics, corresponding to large cigenvalues, emerge carlier in the reverse process, as empirically observed in Ho et al. (2020), ; and analyze how more training data contribute to generalization [Kadkhodaie et al., 2024b. Analytically, this corresponds to the spiked covariance model (Johnstone 2001 in which we bound this decay to the leading eigenvector (corresponding to the largest eigenvalue).
Next, we demonstrate the applicability of our findings to more general, non-linear deep denoisers. Although the network is not linear, its application can be written as a linear operation of the Jacobian calculated on the input image. We empirically show that the aforementioned decay of eigenvector correlations is prevalent also in the Jacobians of deep denoisers, in the final stages of image generation, thus showing the relevance of our analysis in a broader context, and not just in simplified linear case.
2 Background and Related Work
Since their introduction in Sohl--ickses et al. (2015), diffusion models have been vastly used in image
generation tasks (Dhariwal & Nichol 2021 Ho et al. 2020 Song & Ermon 2019 Song et al. 2021c), more
general computer vision tasks (Amit et al. 2021 Baranchuk et al. 2022 Brempong el al. 2022 Cai et al.
2020 and in other domains such as natural language processing (Austin et al. 2021 | Hoogeboom et al.
2021 Li et al.] 2022 Savinov et al.] 2022 Yu et al. 2022) and temporal data modeling (Alcaraz & Strodthoff
2023 Chen et al. 2021 Kong et al. [021 Rasul et al. 2021 Tashiro et al. 2021). On top of their practical
success, different flavors of training and sampling have risen based interesting theoretical reasoning, e.g.,
considcring the statistical propertics of the intermediate data (Song et al. 2021a Sohl-Dickstein et al. 2015
or by framing the problem in the form of stochastic differential equations (SDEs) (Karras el al. 2022 Song
let al.) 2021b c [Chen et al.) (024) or score based generative models (Song & Ermon 2019 2020). In this
work, we look at diffusion models in the context of iterative denoising, and focus on the propertics of the
learned denoiser (Milanfar & Delbracio 2024).
Recently, the work in Kadkhodaie al. (2024) showed that the learned denoising functions are equivalent to shrinkage operation in a basis adapted to the underlying image. In this sense, the diffusion denoiser is adaptive filter (Milanfar) 2013 Talebi & Milanfar 2014 2016). While they focus on the analysis of the nonlinear denoiser at the point of the final generated data, we are interested in the evolution (adaptation) of the denoiser throughout the generation process, and its dependence on the noise level. To this end, we suggest a simple linear denoising model, presented in Section 3 In this case, the (optimal) denoiser does not depend on the underlying image, and its dependence on the noise level can be traced analytically, as we show hereafter.
Due to their phenomenal empirical success, some attempts have been devoted towards providing theory supporting the sample and iteration complexity of diffusion models. The current body of work can be generally parted to attaining itcration complexity bounds assuming approximately accurate scores (Li ct al. 2024b a Chen et al. 2023b: Huang el al. 2024; Benton et. al. 2024), and to assessing the sample complexity to learn the score functions (Chen et al. 2023a Block et al. 2020 Biroli & Mézard 2023). Among these
works, many assume a low dimensional data distribution (Bortoli 2022: Li & Yan 2024 Oko et al.; 2023) Chen et al. 2022aa Wang et al. (024), which is a reasonable assumption in practice (sce e.g., Pope et al. (. Yet, it might particularly explain the gap between the current iteration bounds and the much lower
2

---

Published in Transactions on Machine Learning Research (10/2025)
complexity apparent in practice (Li & Yan 2024.. In our work, we consider linear models and deduce linear sample complexity bound associated with learning the score function in Sec. and discuss the trade-offs of the synthesis conversion rate in Sec. 4.1 The previous works mentioned above mainly develop bounds assuming specific samplers and scaling details, which differ from our setting. In addition, they generally bound the Total Variation distance (under varying assumptions on the larget distributions), which is not trivial to translate to the generated covariance matrix that we focus on even the linear Gaussian case (Devroye et al. 2018). The difference in our setting enables to connect the theory of diffusion models to broad body of work concerning the spiked covariance model (Johnstone 2001], and supports the analysis of denoising diffusion as correlation machine, which is the main purpose of this paper.
In the setting of Statistical Mechanics, the work in Biroli & Mézard (2023) analyzes diffusion models in very large dimensions, focusing on the Curie Weiss model of ferromagnetism. As an introduction to their work, they also discuss a simple linear score model, in the context of the sample complexity of learning the score function. They focus their discussion on the case of Gaussian data, where the cigenvalues of the covariance matrices be lypically characterized. Relatedly, the work in Wang & Vastola (2024) recently showed that the learned neural score is dominated by its Gaussian approximation for moderate to high noise scales, and supply both thcoretical and empirical arguments to support this claim. Compared to these works, we consider data that reside in a low dimensional subspace, with no specific distribution, described in Sec. 4 We limit the denoiser to be linear and focus on two stochastic sampling trajectories, which give rise to the spiked covariance model.
Power iteration is a fundamental algorithm for approximating the dominant eigenvalue and eigenvector of matrix. It relies iteratively multiplying initial vector by the matrix, where its convergence rate is proportional to the ratio of the largest and second-largest eigenvalues. The method's simplicity and scalability have made it cornerstone in various fields, including numerical linear algebra, machine learning, and graph theory. For the ease of reading, we include formal presentation of the method and discuss its convergence in Appendix A In this work, we shall show how a linear denoising chain converges in mean to the celebrated power iteration method.
3 Linear Diffusion - Problem Setup
For our analysis, we define the following simple iterative linear generation model. First, define the standard diffusion model. Let qo denote the natural data distribution and let œg ~ qD be a sample from the natural data (Io Rd). The forward (diffusion) process is defined (Ho et al. 2020 by
(( = NN(1 3,r 1,
(1)
for some fixed noise schedule {3 and xo ~ qD. It can be shown that
q =
(1 l),
(2)
where or B, and a, - The reverse (generation) process is defined using a parameterized distribution model pe, generally defined by the Markov process
Ot.-
Pe( = ((()-11))) & N(po f), Eo (0 t)):
(3)
(4)
where p(æT)= = 10,1). By choices of parametrization and loss manipulations (see (Ho et al. 2020 ) one generally learns to estimate the error (x(x,t),, whore
_(
_
Be
po(x,,()
(5)
Vai
-I-ai at
En (x( = I,, and er is a designed schedule. Thus, the reverse process can be expressed as a denoising chain
1
Br
De(xz) = Vat
Iz
c0 (2()))))) + e22,
(6)
VI
α
3

---

Published in Transactions on Machine Learning Research (10/2025)
where 2 ~ (0, I) and 21 0. This is stochastic denoiser which preserves the Markovian property of the forward process. Later versions suggested similar (non-Markovian) deterministic denoisers, e-g.: DDTM (Song et al. 2021a), or more general stochastic denoiser chains, for a continuous forward model (InDI (Delbracio & Milanfar 2023)). ).
Now, we turn to define our linear setting within the diffusion context. For our simplified model, consider the process (without scaling),
= -N(xo_1,
=
(7)
This implies that = It-1 + co,, where ea, ~ ~N(0, I2I) for some fixed noise schedule {or jo.)=1 We discard
the scaling to comply with previous analysis of the spiked covariance model (Nadler 2008 (more details
in Section This corresponds to the "Exploding Variance' formulation, used with Langevin dynamics to
sample data as a variant of score based diffusion models (Song & Ermon 2019 Song et al. 2021c Song & Ermon 2020 We choose to present the 'standard' diffusion models in the setting of denoising dilfusion (Ho et al. ) and not using the score-based approach entirely, as focus our discussion on the qualities of the denoiser. In our case, we restrict the denoisers to be a lincar function of 2r. For the reverse process, we shall now define two linear denoising trajectories, corresponding to different approaches of diffusion models.
Recall that at each time step I₁ = 2t-1 to, where co. ~ ~N(0, (Equation to Gaussian, we can write T₁ = zo + E where
7). Since the noise is assumed
(8)
where 6,, ~ (N i.e., oe is the accumulated noise at time t. Let E, be the covariance of the noisy data at time f
Σ, EE00o + A Eo + I
(9)
Considering an intermediate denoising step from + 1 - L, the optimal linear denoiser Di+1-se in the lo sense is the minimizer of the loss
441-1 = Eng DDe41 + ta.-1) xx xx112
(10)
given by
Di+1-1 = (E, + 02+11)-1E)
(11)
(full derivation in Appendix Dh. Notice, that in the limit of diminishing de.
An Aotof
D Di+1-t Us
U! 01-100
U.U
DiCA
(12)
Ar-1
t
where U, is the diagonalizing basis of Σ, with the time dependent spectrum {A:), where we omit an explicit temporal dependency in the notation for brevity (more on that in Sec. 4 Our sampling process is based on the sequential application of DPCA. which is the projection on perturbed principal components with respect the clean data distribution, followed by the addition of noise with variance d, This generation path is in the spirit of (Ho et al.) (220),, with its gradual temporal denoising. However, DICA deterministic denoiser given the sampling of training data and noise, which does not depend either d, Zp- In Section we analyze the change in the finite sample approximation of D'cA over time, to study its evolution along the generation trajectory.
Given a similar &2 loss (to Equation in the essence of Song et al. (2021a). The correspouding loss is thus
10 an alternative denoising chain can use multiple estimations zo.
4 Eryce, De-sa(20 + <a.) 20
(13)
4

---

Published in Transactions on Machine Learning Research (10/2025)
S
5
5
5
5
Figure 1: Digit generation from pure noise (class conditioned).
The reverse process runs from left to right.
where oz is the overall added noise (defined in Equation S The adequate denoising chain in this case the application of Di to estimate Zo, followed by the addition of noise with the appropriate variance of-1- before the iterative application of Di-1-00 In this case, the optimal denoiser given by
Di (Eo + o
(14)
Despite the different approaches the two paths represent, their resulting denoising chains exhibit similar properties in both cases, the appearance of frequencies in the generated images is gradual, where low frequencies are first to emerge.
Empirical Demonstration of a Linear Diffusion Model. To illustrate the forward and backward processes in the linear case, we perform a numerical simulation using the MNIST dataset, which is simple enough to be estimated via a lincar model. We start here with the training and generation procodures, and use the same setting and trained denoisers to demonstrate our findings throughout the paper.
In the following experiment simulate the process described above using the MNIST dataset (we use the default train / test splits). In the class conditioned case, we learn (finite sample approximation of a) PCA denoiser with 30 components for each time step where Z₁ = 2t-1 + Cor, x t, and T = 65 iterations. Figure 1 shows a (decimated) example of digit generation from pure noise, where we apply the sequence of (finite sample approximation of) denoisers D'CA In order to understand the reverse process, we now turn to analyze the gradual change in the denoisers, that might be expressed by the angle between the clean and noisy components time.
Notations. We use At to denote the matrix A at time t, and ot to denote the ith column of Ar.
4 Linear Diffusion as Basis Perturbation
We now turn to analyze the linear model presented above and show how the generation process can be seen as a kernel "correlation machine". Specifically, we are interested in the temporal (i.e., noise level) dependence of the finite sample approximation of D'rCA throughout the generation process. Assume that the data distribution is such that its population covariance is given by
Σ. = Exguo
Σ Xuugu
(15)
1=0
where r- 1 < d, i.e., the data reside in a low dimensional subspace (which is generally true for natural data). Thus, the population covariance at time t is given by
Σ, =
Exunt -0'1.
(16)
This data model is known as the "spiked model" (Johnstone 2001] with vast body of work covering the distribution and identifiability of the spikes spectrum (e-g., (Naller 2008); for more information, please see
Appendix B Throughout the paper, the eigenvalues X; are ordered largest to smallest.
the term "index" to refer to to the index i in Equation 16 where
Given n samples concatenated as columns in the matrix X, and n noise vectors in the columns Eg,, at each time step we learn the PCA basis associated with X, = Xo + Es,, by the diagonalization of the sample covariate matrix
- = -
1 (XxX] XoE5 + Xx + Ea, Et.) U,S.01.
(17)
5

---

Published in Transactions on Machine Learning Research (10/2025)
sn Sen MNST
10
0.-
0.6
0.5
0.2
0.0
Figure 2: The sine of the angle between the clean principal components and their noisy versions, colored by the order of the cigenvalues (the darkest being largest eigenvalue). Low frequencies emerge carlier in the generation process (at higher noise levels). This motivates Assumption 4.1 that extends Equation 19 to higher ranks.
Thus, during the reverse process, at each time step we apply the projection
P. a C,0,
(18)
In order to understand the generation process, we analyze the decay of the product (f), us) over time, where ip is the ith column of Ü, Note, that there are two drivers of change in the perturbation of u, to af. The first being the added noise, i.e., |Σ - -ool- This is the key in the diffusion process and our main focus. The second, is in the finite sample approximation | This source of error is interesting in the context of sample complexity,
encompasses the approximation of the denoiser learned from finite dataset, the equivalent of the sample complexity of learning the score function (Chen et al. 2023a Block et al, 2020 Biroli & Mézard 2023). For the rank-1 case, Nadler (2008) presented a finite sample theorem which holds with high probability for the closeness between the leading cigenvalue and cigenvector of sample and population PCA under a spiked covariance model similar to Equation 16 They bound the angle between the leading empirical eigenvector and its population counterpart with approximately ((d) sample complexity, and a lincar dependence on the noise level. Their bound can be approximately expressed by
ŌL X
Esin OPCA = EV (f
(19)
n
where o, is assumed to be small and >>>>>> 1 (for the full derivation, please see Appendix E This result shows that the leading eigenvector rotates at rate proportional to the noise level. Our experiments on the MNIST dataset (detailed in Section 4.1 show that this is good approximation in practice, also for the rank-r case (Fig. 2) We chose to only consider the leading part of the bound in Nadler (2008), as it captures its essence while significantly simplifying the writing. This approximation is justified since it was also shown to be sharp [Nadler 2008 Corollary 1), and empirically by our simulations (and specifically by Fig. 2
Notice that in Equation 19 the angle inversely linked to the eigenvalue, inferring a slower change with higher eigenvalues. In the reverse process, we gradually move from pure noise or high noise levels to smaller noise variance. Given the lower slope of the components corresponding to larger eigenvalues, we interpret the result in Fig. 2 as the earlier emergence of low frequencies in the generation process. The first component to be visible in the generated image is the one with the largest cigenvector, as it is the first one that shows a correlation in high noise levels. Throughout the generation process, when the noise level decreases, the next components take presence, by the order of their associated eigenvalue from the larger to the smaller. Finally, the components with the smallest cigenvalues appear when the noise level low. In other words, we show that low frequencies, corresponding to large eigenvalues, emerge earlier in the reverse process, as empirically observed in Ho et al. (2020). This phenomenon resonates with recent findings in the context. of

---

Published in Transactions on Machine Learning Research (10/2025)
Ning al. 2023 who describe the diffusion process as inherently two-staged: initially pushing samples toward the data manifold, then committing to individual modes. For more context regarding the connection between the principal component index and frequency, please see Appendix C
exposure bias (Li et al. 2024c
In the linear case, Equation 19 shows that the diffusion model's sample complexity is determined by the sample complexity of PCA, with a linear dependence on the dimension of the data. To further enhance our understanding of the relationship between the amount of training data and generalization, we repeat the experiment with varying datasets sizes. Figure 3 shows the angle to noise profile for selected principal components, with the indices 0, 5, 10 (left to right: index corresponds to the largest in list of ordered cigenvalues, that was covered by the result in Nadler (2008)). Incrcasing the amount of training data improves robustness to noise and enables the emergence of higher frequency components at higher noise levels, thereby capturing more nuances in the generated data. Thus, the two right plots in Fig. 3 show that the data behavior in Equation 19 is a good approximation also in higher indices.
dith Bec WMSSTrrdoe 0
an8 HHSTTnnoxx
2ie MMST.irdos
100
06-
02
360
02
3000
3003
5000
5059
5530
Figure 3: Effect of dataset size. The plots show sin OPCA at different noise levels when trained on datasets with incrcasing size (lighter color). Each plot is of different component index, for indices 0, (left right; index 0 corresponds the largest eigenvalue). Increasing the amount of training data improves the robustness to noise, and allows the appearance of high frequencies at higher noise levels, hence capturing more data nuances in the generated data and better generalization.
4.1 The Generated Distribution
We now turn to discuss the distribution of the generated output, and how it relates to the natural data distribution. We shall start from the first sampling path, considering the PCA based denoising (Equation and then describe the second generation trajectory, using estimations of xo (Equation 14 Given our linear model, the generated output is given by
f Io = Po ... P. + + Po5o
(20)
for &. ~ N((, ot). For the ease of writing, define P. = IT; Pr, and so 2 = P
Other than the visual acsthetic of the generated images, we are interested their distribution, and how well represents the natural distribution of training images. Thus, we would like compare the generated
covariance Rist to the natural covariance Eg. We start analysis by focusing the first summand comprising 2,
P PrET.
(21)
In this context, a natural comparison is the power itcration (PI) method, which may be used to estimate the leading eigenvector of a matrix. This can be seen as another ilerative form of generating data from random vectors. Unlike our projection, in Pl we 'project' a random vector onto the entire matrix, i.e. including the cigenvalues. In this case the denoiser would be Di, = Eg where we ignore the normalization and focus on the direction of the final vector, since there is no normalization constraint for generated data in diffusion models.

---

Published in Transactions on Machine Learning Research (10/2025)
0
0r
O++1
10p
u
02 d X
Figure 4: Schematic illustration of the basis perturbation, per index.
We now turn to show how the reverse process performed by a repeated denoising as in Equation 21 converges in mean to PI. To this end, we make the following assumptions, that are expansions of the result 19 from Nadler (2008) to the rank-r case. The first describes the correlations of the same index with noisy versions, and the second describes the cross index correlations (that are relevant due the noise in the finite sample). Assumption 4.1. Assume that Equation
19 holds for all eigenvectors, i.e.,
Tt X,
E 1
(6 ux)
R
n
(22)
for r
This assumption the extension of Equation 19 to higher ranks, and is motivated by our simulations (Fig. D.. In addition, we make the following assumption regarding the cross products of components of different indices, at consccutive time steps.
Assumption 4.2. For each inder i there exisls time Tx, where for 1 S Ti and S
Bía aftt) 0.
(23)
In addition, Ti > Tj. for i <.
This assumption is supported by our simulations in Fig. 5 and will be further discussed hereafter. Assumptions 4.1 4.2
extension of Nadler (2008) to higher ranks. We leave their explicit derivation to future work, and focus their implications to linear diffusion.
We are now ready to state our main result.
Theorem 4.3 (Convergence to Power Iteration). Let o. = 4, t= .,..., T. Assuming 4.1 T 00, 00,
4.2 in the limit
E x Gna
(24)
where iy = PTET, as defined in Equation 21
Proof. Let us analyze the product in Equation 21 to show how relates the power method. The linear operator representing the reverse process can be written
Pr 0o11 (0)001)00
(25)
The matrix product 0f0 can be analyzed using the extension of Equation 19 to higher ranks. Given 4.1 the expected inner product with the natural population data component, is given by
5 d X? n
E u.)
(26)
The evolution of this product over time is depicted in Figure We are interested in the projection of û onto ût, which is the cosine of the angle A0 o4n 0,. This angle is tractable for small noise levels, so
8

---

Published in Transactions on Machine Learning Research (10/2025)
divide our analysis to two parts: < 7 and 7 St S where the choice of 7 will soon be motivated. In addition, we assume adequate sample complexity, such that B(6), ut) x 0.
First, we inspect the limit of t + (0 S t T). For small angles, we can write
AA arcoos (1 x1
0, d
arccos (1 o.
Xan - - -,)
o
(27)
X2 n
n
X2n
since arccos f ~ 2 - 0 and oī = Σ += o² The diagonal clements in Ult4+1 are then given by
B(i 11-1) = cos
(28)
X2n
where the off-diagonal elements are negligible, since
E(a, aft") ~ B(i,, atj = 0,
(29)
which holds for / S Tr- by Assumptiio 4.2 Notice, that in small angles, (0) (0/-at=+ .u) =of=1d)((A9n) - 0, so the vectors of are co planar, as depicted in Figure 4 Thus, the time point basis correlations 0 0/411 form aind on the diagonal, where c; > ej for j. We eliminate the dependence of o on by choosing the constant schedule or 1/T Vt, to simplify the proof. However, many schedules can be used, as long as Ci Cj,a remains correct. Define the partial linear diffusion operator until time by IEP, = Pt. Then
an approximately diagonal matrix with the fraction c; cos
x
c
0 = Ooco (" (2)
co
EP, = Co
0
- T-T
Uo
0
(30)
where the diagonal elements decay as grows larger, since &: > Cj for j. Similarly to power iteration, the convergence rate depends on the ratio ca/co. The convergence rate might not be fast enough for the process to converge while the small angles approximation still holds. Thus, continue with the second phase of our analysis, showing the convergence of the full reverse process.
We now turn to analyze the phase where о SLS T. In high noise levels, the correlation with the natural basis is low, and the products 0 010++1 are not exactly diagonal. However, the correlation 'leaks" to a close neighborhood of the original component and the temporal products are still somewhat concentrated around their diagonal. This process happens in accordance with Equation 4.1 where the spreading of the diagonal elements happens for high indices in lower values of t (less noise is needed to spread the correlation). This leads to Assumption 4.2 claiming that for each index i there exists time after which the small angle approximation does not hold; T, Ty for i < j. This is apparent in practice, and depicted in 5 (left image per duo). However, given the decaying diagonal structure of the partial operator Pr, we will now show that 4.2 is sufficient for the total operator to converge as desired.
Suppose we added one more matrix multiplication to our former analysis, i.e. observe
EP,Ū = Docg =toco
(2)
0:0.n.
(31)
Assumption 4.2 guarantees 010,41 is diagonal just enough not spoil the diagonality of the next partial operator EP,+1. To see this, let us inspect some intermediate index i, where the entries in j > i are already practically wero. Thus, we have
EP, Ur+t = 7++1
= ++
C
0
SC+,
9

---

Published in Transactions on Machine Learning Research (10/2025)
404 no4ph
Ug
Mg
50
16-
10-
40
13
23
20
20
Figure 5: The time point basis correlation matrices Üţū,-1 (left per pair), together with the partial product oO, U441) (right per pair) at different time points. This justifies Assumption 4.2. and shows that the total projection (bottom right image, for T == T) converges to the first eigenvector, similarly to the power method.
where O a block of zeros and A a block matrix the same size as O, that can have nonzero entries, by Assumption 4.2 Since the elements of the partial product Cr; decay faster with than any single product U1- Cr,+1 is also diagonal. Overall, the final product is a diagonal matrix with a spectrum that converges to be concentrated around the first eigenvalue, where we can control the distribution of the generated data by the choice of the diffusion parameters. Figure 5 shows our simulation of the process, where 0+0++1 (the left in each duo) is approximately diagonal until a certain index, and the total product (right in each duo) is also approximately diagonal (with less indices than its counterpart). This supports both assumption and the result stated by this theorem.
Thus, the generated output is a combination of a (purcly) noisy image that was repeatedly correlated to converge to uo (as shown above), with generally lower noise levels that are "lightly" correlated, although to the cleaner projection operators. The generated output can thus be seen as a combination of three conceptual parts, with different balance of the noise level and the portrayed components.
The first eigenvector The first part of the sum in Equation 20 is Po...PPeT the estimation of the eigenvector with the largest eigenvalue, as shown above theoretically in Equation 30 and empirically in the rightmost matrix in Fig. The 'strongest' noise is repeatedly correlated to be concentrated around the first eigenvector.
The entire (clean) spectrum The last part in Equation 20 is Počo, weak noise level that is spread across all components. This noise is very lightly and not repeatedly correlated, although to a clean version of the natural data basis.
In between The third part consists of all the intermediate products =1 The product operators of preserve varying parts of the natural spectrum, according to t grows, the total projection tends to retain only the components associated with larger eigenvalues. This can be seen in Fig. The right matrix in each pair shows the product - for varying values of t. The total projections range from the entire spectrum (left) to only the leading eigenvalue (right). In between, the products are diagonal matrices where the entries the indices of the smaller eigenvalues have already diminished, in a similar way to the convergence described in Equation 30
Thus, we get a combination of a solid estimation of the leading eigenvector, together with a more uniform and week sampling of the components with low eigenvalues in the natural data basis. In between, the intermediate projections are at different levels of convergence to the leading cigenvector, hence tend to be more concentrated on components with large eigenvalues as t + T. The freedom in choice of schedule {G:) allows control of the spread of the final distribution on the natural data components.
The sample trajectory we consider in Equation 20 is stochastic. However, our analysis also covers a deter- ministic sampling path, where the denoisers are repeatedly applied, with no added noise, This deterministic sampling in fact portrayed by the first summand in the generated output Equation 20 which show to
10

---

Published in Transactions on Machine Learning Research (10/2025)
converge in mean to the first cigenvector, similarly to the power iteration. Under the limitation of the linear model, this also implies that this deterministic sampling is insufficient to capture the training data.
1.0
Moue FC sutc
10
Meas TC ESDBAC
1.0
1940 PC derrbcee
1-60
Naturl data
06-
30
08-
Norekll caveasee
100
152
121
Horee knd combttue
66-
06
Hotoral
30
34
30
22
25
Figure 6: The empirical distribution of generated images over the natural principal components, with (mid-
dle) and without (left) injected noise. On the right the best configuration with the generated standard deviation (sce Soc.
To inspect this, we plot the empirical distribution of generated images over the clean PCs, given by
P: = 1
(0,, [ l12
(32)
where û, is the clean principal component with index i (defined in Equation 16 and A is a generated sample, out of examples. Figure G shows the empirical distribution of generated images over the clean principal components. On the left, plot the distribution without injected noise (i.e., a Pr for various values of T. As we show above, the distribution tends to be concentrated on the first eigenvector as T' increases. The center plot shows the distribution of the process including the injected noise in the intermodiate denoising steps. While in the low indices the dominant behavior is similar to the former case, the higher orders do not converge to zero and maintain their presence in the generated distribution. We note, that more sophisticated nonlinear deterministic samplers might not require the injection of noise order to converge to the natural data distribution (e.g. (Lu al. (022)).. However, given linear model, it is natural to accept added stochasticity in the lack of nonlinearity (more that in Section 5). On the right, we picked the best configuration (T= = 65 in this case) to approximate the natural distribution. Notice, that the final generated distribution depends on the choice of parameters, where one can control the mean of the generated spectrum (this might be a feature for some applications, such as segmentation via diffusion, etc.). It might be interesting to derive the optimal parametrization for the convergence of the linear model we leave this for future work. In addition to the convergence in mean, we included the standard deviation of the natural and generated samples, resulting in a decent fit to the target distribution.
The form of projection in Equation 32 can also help illustrate the appearance of higher frequencies during (stochastic) sampling. In Figure 7 we plot the mean distribution of the projection of generated samples over the clean principal components. The color is linked with the time- lighter colors as t + O. We can see that components with higher indices are apparent towards the end of the reverse process, indicating that higher frequencies appear later in the generation process.
We now turn to analyze the second sampling procedure, considering the loss defined in Equation 13 and repeated estimations of zo. The generation starts from the denoising of &r by Di (defined in Equation 14 to obtain the first estimate of Zu, D7 The next denoiser is optimal considering the noise level ōT-1: so prior to its application, we add the next noise instance, --- Thus, the iteration in this denoising chain is given by
xt-1 = Do + Ei-1,
(33)
where again & NN(0 of.). similarly to the former case, the final generated output f can be expressed as
f ET Ex IT.
D Di-o"" Di +... + Do
(34)
11

---

Published in Transactions on Machine Learning Research (10/2025)
Moan dver time
0.7
Nomatted coreeation
0.4
0.3
82-
0.0
Component
Figure 7: The empirical distribution of generated images over time lighter colors for larger t. The high indices (i..., high frequencies) appear at the later stages of sampling.
The difference between the generation path in Equation 34 and the one described in Equation 20 is in the applied denoisers, where the former utilizes the denoiscr defined in Equation 14 and the latter employs the finite sample PCA denoiser (defined in Equation 18p. In addition, the accompanying noise schedules should match the denoiser: {o,} for the PCA denoiser and }or} considering Equation 14
Notice, that in this case as well, if we inspect the first element in Equation 34 i.c., D5- -o the dominant direction is concentrated in the first eigenvector of Eg. This be seen by looking at the diagonalization of Di-00
Di = (Eo + +0211)
(35)
Ag Aotoz
= Un
U
Ar-1 1
since is monotonically increasing for æ, a 2 0. Thus, similarly to the case described in Equation 20 the generated output can be interpreted as sum of high noise levels that were repeatedly correlated to estimate the leading data eigenvector, and lower noise levels that sample the entire data spectrum, in accordance with our discussion in Section
5 Empirical Extension to Deep Denoisers
In the linear case described above, the optimal denoiser is given by PCA projections. These denoisers are computed with the training data, and their principal components do not depend the input noise in the reverse process. When the denoiser is nonlinear, and might be implemented using deep neural network, its input-output mapping can be locally expressed via the network Jacobian estimated at the inference point, Dneet(Fr), by
Dnnt (x:)) DDeet(E:)E V.A.Vi..,
(36)
where V,A,V, denotes the eigen decomposition of the Jacobian calculated at xG. For simplicity, we assume that the Jacobian is symmetric and non-negative (which is approximately true (Mohan et al. 2020 ). Note that in this case, the denoising base depends also on the input image (on top of the noise level). Although the network is non-linear, we can follow the generation path in the sampling process and inspect the basis of the network Jacobians calculated at the intermediate sampled points Zt. We can then trace sin 0,
/1 /1- (nf, of 0y² where the subscript 'J' stands for Jacobian, of is the ith column in V, defined in Equation 36 in similar way our simulations of the linear case (Figure 2). This can be calculated per generation path, where 20 the final generated image, and Vo is the basis of the Jacobian calculated at this final point.
12

---

Published in Transactions on Machine Learning Research (10/2025)
knn (Galdb4)
(CFFA11))
66-
68
04-
64.
02
30
25
Figure 8: Image generation - the sine of the angle between Jacobian eigenvectors at the final generated image (t = 0) and intermediate iterations (t > 0). The diffusion model includes a UNet-based denoiser trained on CelebA (left) or CIFAR1O (rignt). Color by index (the darker the color the lower the index, referring columns of the Jacobian basis V2). The Jacobians of the nonlinear denoiser conform the behavior of the linear model.
Figure shows sin f calculated using the Jacobians of a UNet based diffusion model, described in Ning ct al. (003)) This model was simply chosen as the -tate-of-the-art in the task of image generation considering the CelebA dataset at the time of writing this paper. We used the default settings and calculated the Jacobians at the final iterations. We plot the results for the leading 300 Jacobian eigenvectors, where the color is assigned by the index darker colors for lower indices i. We repeated the experiment sampling images from the CelebA dataset (left) and CIFAR 10 (right). Even though the denoising model is far from linear, the decay of the angle between the denoising basis in high noise levels and the natural denoising basis is similar to the decay in the linear case (compare to Figure 2 In this case as well, the correlation of the low indices (and hence low frequencies) withstands higher noise levels, thus appearing first in the generation process. As this is the basis of our analysis comparing the reverse diffusion process to power iteration, this experiment shows that our analysis is relevant in a broader context and not just in the simplified linear case. For more experiments considering other architectures, please Appendix F
This analysis focuses on the local behavior of the nonlincar denoiser at the end of the generation process, demonstrating its similarity to a linear denoising chain. Each plot represents a single generation path, not the overall distribution of generated outputs.
While linear diffusion models are easy to analyze, they may struggle to generate complex datasets. Nonlinear models, on the other hand, can navigate diverse set of lincarized regions during the generation process (as illustrated in Figure 8) This allows them to generate diverse data even without added noise, unlike linear models which ultimately converge in mean to single point (Theorem 4.3) and therefore require noise injection for diverse outputs, This contrasis with some deterministic nonlinear samplers (e.g., (Lu et al., 2022)) that do not rely on added noise, The linearization approach we use is accurate at the sampled points in the generation trajectory, we cannot express the sampling procodure as strictly matrix multiplication (but rather as the composition denoisers). However, if the denoiser is near linear in the sense that it has limited curvature around these sampling points, then our result might be approximately applied. We believe this is an interesting direction and leave its exploration for future work.
6 Conclusion
In this paper, discuss simple diffusion model with lincar denoiser and normalization free sampler, that allows cast the diffusion problem noisy PCA, and make the connection to the spiked covariance model assuming that the natural data distribution resides in low dimensional subspace. This enables us to show that in the linear case, the generation process acts as a "correlation machine", where initial random noise is repeatedly correlated to noisy estimations of the natural data basis, to finally embody the true distribution,
http:://ppprruutthcccecccccc///p htt.:///ee..cooeoo//onnnnnnnnnrr www.ea-eeeaeeeeee 64x64
13

---

Published in Transactions on Machine Learning Research (10/2025)
in manner similar to the power iteration method. We show that in this process, low frequencies emerge earlier, and more data contributes to richer representation per the same diffusson configuration. Finally, we demonstrate the relevance of our analysis also in a deep, non-linear diffusion denoiser.
We acknowledge the limitation admitting a linear model, with its lack of ability to represent the complex data and often expected of diffusion models. In addition, studying whether a similar correlation-basen struc- ture persists in latent-space diffusion, where the manifold hypothesis may no longer apply, an interesting direction for future work. While theoretical setting is modest, empirically demonstrate how our obser- vations deduced from simple linear model and classic theory (Johnstone) 2001 Nadler 2008 are relevant to more general models and datasets. This enables us to shed light on the internal mechanism powering this technology, and connect it to rich pool of theory and prevalent methods such as power iteration.
Acknowledgments
The authors extend their sincere gratitude to Giulio Biroli, Gabriel Peyré, Mojtaba Ardakani, Tom Duerig, Tom Tirer and Shady Abu-Hussein for their valuable feedback. RG and DW were partially supported by the Israeli Innovation Authority, the Israeli council of higher education and the center for Al and Dala Science at Tel University. RG and DW are also thankful for KTA and Google for their support.
References
Juan Lopez Alcaraz and Nils Strodthoff. Diffusion- based time series imputation and forecasting with struc- tured state space models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL
https://openreeiiv.. hww.://wwwweeettttrtt/hhhhaaa///
Tomer Amit, Tal Shaharbany, Eliya Nachmani, and Lior Wolf. Segdiff: Image segmentation with diffusion probabilistic models. arXiv preprint arXiv:2112. 0rXio:2112.00390, 2021.
Stephen Andrilli and David Hecker. Chapter 9 numerical techniques. In Stephen Andrilli and David Hecker (eds.), Flementary Lânear Algebra (Sirth Eddtionn) pp. 401-439. Academic Press, sixth edition edition, 2023. ISBN 978-0-12-822978-1. doi:
/tt.://////0.00000//00---0-00-00 https://wwez http:/wwww.ciiencddreet... httpp//dce...otoooo.....oo/////a
URL
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffiusio models in discrete state-spaces. Advances in Newral Information Processing Systems, 34:17981-17993, 2021.
Jinho Baik, Gérard Ben Arous, and Sandrine Péché. Phase transition of the largest eigenvalue for non-null complex sample covariance matrices. Annals of Probubilily, 33(5):164331688 2005.
Dmitry Baranchuk, Andrey Voynov, Tvan Rubachev, Valentin Khrulkov, and Artem Babenko. Tabel-efficient semantic segmentation with diffusion models. In International Conference Learning Representations, 2022. URL
Florent. Benaych-Georges and Raj Rao Nadakuditi. The eigenvalues and eigenvectors of finite, low rank perturbations of large random matrices. Advances in Mathematics, 227(1):494 521, 2011.
Joe Benton, Valentin De Bortoli, Arnaud Doucet, and George Deligiannidis. Nearly Sd$-lineat convergence bounds for diffusion models via stochastic localization. In The Twelfth International Conference on Learn- ing Representations, 2024. URL
Giulio Biroli and Mare Mézard. Generative diffusion in very large dimensions. Journal of Statistical Me- chanics: Theory and Experiment, 2023(9) 0093402, 2023.
Adam Block, Youssef Mrouch, and Alexander Rakhlin. Generative modeling with denoising auto-encoders and langevin sampling. arXiv preprint ar 2000:20002.01177 2020.
14

---

Published in Transactions on Machine Learning Research (10/2025)
Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. Thansac- lions on Machine Learning Research, 2022. TSSN 2835-8856. URL cectp:://peeneeiie..eeee////// fet/oommmidd MhK5aXo3gB Expert Certification.
Emmanuel Asiedu Brempong, Simon Kornblith, Ting Chen, Niki Parmar, Matthias Minderer, and Mo- hammad Norouzi. Denoising pretraining for semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4175-4186, 2022.
Ruojin Cai, Guandao Yang, Hadar Averbuch-Elor, Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pp. 364-381. Springer, 2020.
Defang Chen, Zhenyu Zhou, Can Wang, Chunhua Shen, and Siwei Lyu. On the trajectory regularity of ODE- based diffusion sampling. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 21-27 Jul 2024. URL htttp:://proceedigg:: mlr grress/v25//ore / html
PMLR,
Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang- Score approximation, estimation and dis- tribution recovery of diffusion models on low-dimensional data. In International Conference on Machine Learning, pp. 4672-4712. PMLR, 2023a.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In International Conference on Leorning Representations, 2021. URL
Sitan Chen, Giannis Daras, and Alex Dimakis. Restoration-degradattie beyond linear diffusions: A non- asymptotic analysis for ddim-type samplers. In International Conference on Machine Learning, pp. 4462- 4484. PMLR, 2023b.
Mauricio Delbracio and Peyman Milanfar. Inversion by direct iteration: An alternative to denoising diffusion for image restoration. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https: Featured Certification.
//openreview http///www....oooooooo////////as
Lue Devroye, Abbas Mehrabian, and Tommy Reddad. The total variation distance between high-dimensional gaussians with the same mean. arXiv preprint r=Xi::180000869 2018
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in newral information processing systems, 34:8780-877 8794-8994420 2021.
David T, Donoho, Matan Gavish, and Tain M Johnstone. Optimal shrinkage of eigenvalues in the spiked covariance model. Annels of Statistics, 46(4):1742-177 2018
Clf Grenander and Gábor Szegö. Toeplitz forms and their applications. University of California Press, 1958.
Peter J Hancock, Roland J Baddeley, and Leslie S Smith. The principal components of natural images. Computation in Neural Systems, 3(1):61-70, 1992. doi: 10.1088 00.1088/0954-8987 3 008.
Network:
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 2020.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 34:12454 12465, 2021.
Daniel Zhengyu Huang, Jinoyang Huang, and Zhengjiang Lin. Convergence analysis of probability flow ode for score-based generative models. arXiv preprinl ar 00X4:2244.099730, 2024.
lain M Johnstone, On the distribution of the largest eigenvalue in principal components analysis. The Annals of statistics, 29(2):295 -222955--77660 2001.
15

---

Published in Transactions on Machine Learning Research (10/2025)
lan T. Jolliffe. Principal Component Analysis. Springer Series in Statistics, New York, 2nd edition, 2002.
Zahra Kadkhodaie, Florentin Guth, Eero P Simoncelli, and Stéphane Mallat. Generalization in diffusion models arises from geometry-adaptive harmonic representations. In The Twelfth International Conference on Learning Representations, 2024.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- generative models. Advances in neural information processing systems, 14-26565-26577 2022.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021. URL https: //openreview.net
Gen Li and Yuling Yan. Adapting to unknown low-dimensional structures in score-based diffuusion models. arXiv preprint arXio:2405. 14861, 2024.
Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Towards non-asymptotic convergence for diffusion-based generative models. In The Twelfth International Conference Learning Representations, 2024a. URL
https://openrevier. http:://ppppareeeeeeeeeeeooootar
Gen Li, Yuting Wei, Yuejie Chi, and Yuxin Chen. A sharp convergence theory for the probability flow odes of diffusion models. arXiv preprint ar arxiv:2488. Xxi:2240.00220, 2024b.
Mingxiao Li, Tingyu Qu, Ruicong Yao, Wei Sun, and
Moens. Alleviating exposure bias in diffusion models through sampling with shifted time steps. In The Twelfth International Conference on Learning Represenlations, 2024c. URL https /ttp:/::/p////i///ee/e wwwp:/reiiiii.nefforrieee net www..//oouum...c2220M.ooooo..
Xiang Li, John Thickstun, Ishaan Guirajani, Percy S Liang and Tatsunori Hashimoto. Diffusion-Im improves controllable text generation. Aduances in Newral Information Processing Systems, 35:4328-455 4343, 2022.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5779 2022.
V. Marchenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics 144):457-483, 1967. doi: 10. BLL00//SSAD00700000AABBB33004 URL/attps://aoi.
of the http:/00.0000/111100000000000000 org/10.
Peyman Milanfar. A tour of modern image filtering: New insights and methods, both practical and theoret- ical. IEEE Signal Processing Magazine, 30(1):106-128, 2013. doi: 10.1109/ 10.1109/MSP.2011.. MSP.2001.229992222
Peyman Milanfar and Mauricio Delbracio. Denoising: A powerful building- for imaging, inverse prob- lems, and machine learning. arXiv preprint arXiv:2409.00 06219, 2024.
Sreyas Mohan, Zahra Kadkhodaie, Eero P. Simoncelli, and Carlos BernandezGrannde Robust and inter- pretable blind image denoising via bias-free convolutional neural networks. In International Conference on Learning Representations, 2020. http:a::://opeerrooi..eeeee/e//e
Boaz Nadler. Finite sample approximation results for principal component analysis: A matrix perturbation approach. The Annals of Statislics, 36(6):2791 2817, 2008
Mang Ning, Enver Sangineto, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Input perturbation reduces exposure bias in diffusion models. In Proceedings of the 40th International Conference on Machine Learning, ICML'2 '23. JMLR.org, 2023.
Kazusato Oko, Shunta Akiyama, and Taiji Suzuki. Diffusion models are minimax optimal distribution estimators. In International Conference on Machine Learning, pp.
PMLR, 2023.
16

---

Published in Transactions on Machine Learning Research (10/2025)
Debashis Paul. Asymptotics of sample cigenstructure for large dimensional spiked covariance model.
Statistica Sinica, 17(4):1617-1642, 2007.
Phil Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact on learning. In International Conference on Learning Regresentations, 2021. URL
Kashif Rasul, Abdul-Saboor Sheikh, Ingmar Schuster, Urs M Bergmann, and Roland Vollgraf. Multivariate probabilistic time series forecasting via conditioned normalizing flows. In International Conference on Learning Representations, 2021. URL httts::/ wwtp://eeeetoot////////////////
Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski, Erich Elsen, and Aaron van den Oord. Step- step-unrolled denoising autoencoders for text generation. In International Conference on Learning Representations, 2022. URL httpl///w//pppoeeeoeeeee/e//rrrp
Lawrence Sirovich and Michael Kirby. Low-dimensional procedure for the characterization of human faces. Journal of the Optical Society of America A, 4(3):519- 524, 1987.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathann, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusson implicit models. International Con- ference on Learning Representations, 2021a. URL htttps://penrrevier. http://www...////aeee/////ee///r http://ww....eeoooooo//tttttaara
Yang Song and Stelano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33:12438 12448, 2020.
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in neural information processing systems, 14:1415-4425 2021b.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021c. URL https: http:/ppwreeeeeeeeeeeerrrrraaaaa wwtp:/orrrnidd-x111......
Hosscin Talcbi and Peyman Milanfar. Nonlocal image editing. IEEE Transactions on Image Processing, 23 (10):4460- 4473, 2014. doi: 11010//TTP.011.23488858
Hossein Talebi and Peyman Milanfar. Fast multilayer laplacian enhancement. IEEE Transactions on Com- putational Imaging, 2(4):496-50X 2016. doi: T0.1109/TCT.22166
Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi: Conditional score-based diffusion models for probabilistic time series imputation. Advances in Neural Information Processing Systems, 34: 24804-24816, 2021.
Binxu Wang and John Vastola. The unreasonable effectiveness of gaussian score approximation for diffusion models and its applications. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL
https://openrevieew http://wwww.....oooo///taotoaaaa
Peng Wang, Huijie Zhang, Zekai Zhang, Siyi Chen, Yi Ma, and Qing Qu. Diffusion models learn low- dimensional distributions via subspace clustering. or preprint X4i0:2409.022266 2024.
Poiyu Yu, Sirui Xie, Xiaojian Ma, Baoxiong Jia, Bo Pang, Ruiqi Gao, Yixin Zhu, Song- Song-Chun Zhu, and Ying Nian Wu. Latent diffusion energy-based model for interpretable text modeling. In Proceedings of International Conference Machine Learning (ICML), July 2022.
17

---

Published in Transactions on Machine Learning Research (10/2025)
A Power Iteration and its Convergence
Power Iteration is simple algorithm used to compute the dominant eigenvalue and its corresponding eigen- vector of matrix. It iteratively refines an initial random vector by multiplying it by the matrix, which gradually aligns with the eigenvector corresponding
the largest eigenvalue. For a thorough rotroiutio the method, see, e.g. Andrilli & Hecker (2023). Given a square matrix A E IR*x* the goal is compute the dominant eigenvalue λ and its corresponding eigenvector vi. The Power Iteration algorithm is defined follows:
Algorithm Power Iteration Algorithm
Input: Matrix A R"x", initial vector vo E R" number of iterations k
Output: Approximate dominant eigenvector us Normalize the initial vector: || for each iteration = ..... do
20
Vi f A0 Normalize end for return 0k
"
The algorithm starts with an arbitrary vector up, which is normalized to ensure numerical stability. In cach iteration, the vector oi updated by multiplying it by the matrix A, followed by normalization. After k iterations, the vector Up is expected to be close to the eigenvector corresponding to the largest eigenvalue of A.
A.1 Convergence Analysis
Let A be square matrix with eigenvalues A2........ , where the eigenvalues ordered such that Arl > |al 2 ... |a.l. Denote the corresponding eigenvectors by U2...., where U1 is the eigenvector corresponding to the dominant eigenvalue \
An,
The key idea behind Power Iteration is that, after sufficient iterations, the sequence of vectors es converges the eigenvector associated with A₁, under certain conditions.
Let to be the initial vector, which can be expressed lincar combination of the eigenvectors of A:
n
Up Σ
0
where as are scalar coefficients. After applying the matrix A in each iteration, we obtain the sequence of vectors:
U; Av A
Σ 0g
Σ anxi
Thus, the i-th iteration amplifies the component of to along the direction of the eigenvector corresponding to the eigenvalue λı, while the other components decay at rate proportional to the magnitude of their respective eigenvalues. As the iterations proceed, the contribution of the eigenvectors associated with smaller cigenvalues diminishes, and the vector becomes incrcasingly aligned with U1, the cigenvector corresponding to A
Formally, we express the evolution of v as:
" Xane Xba222 .... Xgannn
18

---

Published in Transactions on Machine Learning Research (10/2025)
The relative influence of the cigenvectors corresponding to X2 A,,...,,A, decays exponentially as - because Å > [22| 2 ... 2 |all- Specifically, the error in approximating decreases at a rate proportional to LAzl [As]' leading to the following convergence result:
0/ Meall
122 A1
CC
101
for some constant C, where || is the vector norm (usually the Euclidean norm).
Therefore, the Power Iteration algorithm converges to the dominant eigenvector v1 at a rate determined by the ratio of the magnitudes of the first and second largest eigenvalues, p [xi] If A2 is much smaller than A1; convergence is fast. However, if A2 is close to Å1, convergence can be slow, requiring more iterations to achieve a satisfactory approximation. The convergence is linear, with the error decaying exponentially as the number of iterations increases. For a matrix A with well-separated dominant cigenvalue X (i.c., Aul > [aal); Power Iteration converges quickly, typically in O(log(e) log(()) iterations to achieve an error of size c.
Daal
B
The Spiked Covariance Model
The spiked covariance model is a widely studied statistical framework that describes the behavior of the sample covariance matrix when a low-rank signal is embedded in nigh-dimensional noise. Originally intro- duced by Johnstone (2001), the model captures scenarios where a small number of principal components (the "spikes") are distinguishable from an otherwise isotropic Gaussian noise background.
Formally, let X E Rex= be a data matrix with i.i.d. observations drawn from multivariate normal distribution:
X, ~(0, E), E= TI D + Σ U
where o² 0 is the noise variance, p is the number of spikes, 02 0 are the spike strengths, and i E R are orthonormal eigenvectors associated with the signal. The goal is to understand the spectral properties of the empirical covariance matrix E xx", especially in the high-dinensional regime where p, + with p/n + E oo).
A key insight of the spiked covariance model that in the high-dimensional limit, the bulk of the spectrum of E follows the Marchenko Pastur distribution (Marchenko & Pastur 1967). while sufficiently strong spikes result in outlier eigenvalues. [Biik et al. (2005) discovered a phase transition now known as the BBP phase transition characterizing when spike separates from the bulk spectrum. Specifically, for single spike A, the corresponding sample eigenvalue detaches from the bulk if 0 > o² V7, and the associated eigenvector becomes asymptotically correlated with the true signal direction.
Subsequent work extended these results to multiple spikes (Paul 2007 non non-Gaussian settings Benaych-
Georges & Nadakuditi] 2011). and Bayesian and optimal shrinkage estimation Donoho et al. 2018 Compu- tationally, algorithms such as PCA and its variants remain the primary tool for extracting low-rank signals, although care must be taken in interpreting components near the detection threshold.
The spiked covariance model has profound implications in both theoretical and applied contexts. It provides rigorous foundation for principal component analysis (PCA) in modern regimes where the dimension is comparable to or larger than the sample size. Applications span signal processing, wireless communications, genomics, and finance anywhere that. latent low-dimensional structures are inferred from noisy data.
Furthermore, the model serves as testbed for understanding fundamental limits in statistical estimation, in- cluding the detectability of weak signals, the performance of eigenvalue shrinkage methods, and the reliability of inference under model misspecification.
19

---

Published in Transactions on Machine Learning Research (10/2025)
C Frequency Characteristics of Principal Components
Principal Component Analysis (PCA) is cornerstone technique in statistical data analysis and dimension- ality reduction. Given data matrix X E I=** (with each column representing an observation), PCA identifies an orthogonal basis of directions U1, #2...., (the principal components or PCs) such that the projection of the data onto each captures succossively less variance.
When applied to natural images, PCA components often exhibit a hierarchical structure, with lower-index components capturing broad, frefueeuency features, and higher-index components capturing finer, high- frequency details. This phenomenon particularly evident when analyzing small image patches (e.g., 8x8 16x16 pixels) extracted from natural images. The first few principal components often correspond to smooth, low-frequency patterns, while the later components capture high-frequency structures such as edges and textures.
This pattern is consistent. with the fact that PCA decomposes data using the eigenvectors of the sample covariance matrix Σ = - x When X arises from smooth processes (e.g., discretized Gaussian processes, Brownian motion, or autoregressive time series), the covariance matrix has a Toeplitz or smooth kernel structure. Its eigenvectors, under suitable conditions, converge sinusoidal Fourier-like modes (Grenander & Szegö 1958 Jolliffe 2002). This behavior can be theoretically justified under models where the data- generating process is smooth governed by differential operator. In natural images, the covariance matrix lypically exhibits a smooth, spalially invariant structure. This translation-invarinnc implies that the eigenvectors of the covariance matrix, i.e., the principal components, tend to be sinusoidal in nature. As the component index increases, these sinusoidal modes become increasingly oscillatory, corresponding to higher spatial frequencies, often visualized as "Fourier-like" PCA modes.
This behavior has been observed in various studies. For instance, Hancock et al. (Hancock et al. 1992 found that the principal components of natural images resemble derivatives of Gaussian operators, similar to those found in the visual corlex and inferred from psychophysics. In addition, eigenvalues in facial recognilion exhibit smooth lighting and shading variations in early PCs, and finer texture or edge-like patterns in later PCs (Sirovich & Kirby, 1987).
D PCA Optimality And Other Linear Denoising Chains
In the main text we discuss a gradual denoising chain, where noise is iteratively projected onto cleaner PCA bases (as defined in [8]. In the following, we will clarify the sense in which PCA is optimal, and present another linear denoising scheme, which will help to frame the subject of this work.
The optimal linear denoiser at time in the Eg sense is the minimizer of the loss
4++1+ = Ex.. [De=1-ee(Ee + a,w) - xell
(37)
where w I). For the brevity of writing, we locally denote D++1- by D., until Eq. 41 minimized by deriving the expected loss
This can be
Ep, De(re + )rw) xıll2 = Er, [.,)) -
D.,, of -x, Daw, + +
Er,,w Tr (D. -D + De D.w +mal1 Tr - 20 11 E.).
(38)
where we have used the fact that w. has zero mean. To derive the optimal linear denoiser, we have
di = 2D,E, - -2, + 20 = 0, dD,
(39)
and so
D, (E4 + -15
(40)
20

---

Published in Transactions on Machine Learning Research (10/2025)
Notice, that in the limit of diminishing dz.
Ap Aoto
D. = Up
U. +
U.U & DeCA-
(41)
ac-0
Ar-1 to-
Alternatively, this can be seen the minimizer when we average also on the input noise variance. In this work, focus on the iterative application of D4+1-1 and use the theory regarding noisy PCA (Nadler 2008 to analyze the convergence properties of this chain.
Given a similar & loss, one might suggest an alternative denoising chain, using multiple estimation of zo- The corresponding loss is thus
4-0 - DD-so( F,w) - xo 12:
(42)
where og is the overall added noise (sce Section 4p. The adequate denoising chain in this case is the application of De-o to estimate 20, followed by the addition of noise with the appropriate variance of-1, before the iterative application of D.-1- In this case, the optimal denoiser is given by
D2 = (Eo +
(43)
An
Aotor
Ug
Up-
Ar-
In order to generate data, this denoiser is applied on scries of noises Wt, where we ~ (0, ofTl) for some schedule (ō.) The generation starts from the denoising of wT by Dr-so, to obtain the first estimate of æo. WT. The next denoiser is optimal considering the noise level 0r-1, so prior to its application, we add the next noise instance, WT-1- Thus, the iteration in this denoising chain given by
It-1 De-soat + w1-1,
(44)
where again we (((0 Due to the linearity of the denoisers, the final generated output Tg can be expressed as
Zg = T -0 = Do DT + .... Dowe.
(45)
Notice, that in this case as well, if we inspect the first element, i.e., Do Dr-owr, the dominant direction x is monotonically increasing for æ, 0). Thus, is concentrated in the first eigenvector of Eg (since z+a similarly to the case described above, the generated output be interpreted that were repeatedly correlated to estimate the Icading data cigenvector, and lower noise level that sample the entire data spectrum, in accordance with our discussion in Section 4.1
of high noise levels
E The Derivation of Perturbation Bounds from Nadler (0008)
In our analysis, we rely on nonasymptotic bounds on the angle between the empirical principal component vector and the true population component direction, as derived by Nadler 2008). These results pertain to the spiked covariance model in a finite-sample regime.
Consider the spiked covariance model:
+ of,
where u E IR is a scalar latent variable with zero mean and unit variance, v E RP is a fixed signal direction, ~~>((0,) 1p) is Gaussian noise. Given n i.id. samples (o we compute the sample covariance matrix Sn, and denote denoted by 8PCA, with
leading eigenvector as OPCA- The angle between UPCA and the true dircction e/|le|l is
sin OPCA V1 (EPCA, w/|lo|1?2
21

---

Published in Transactions on Machine Learning Research (10/2025)
Assuming ||e||e and defining the signal strength K ||l||s, where sy high-probability upper bound:
Er - Nadler provides
p-1- 1 1+
20s1
(1 1+
$2
4v20²
p
sin Sin 1
K
n
vp
n 1
2
which holds with probability at least -1 62 E3 - E, where $1, 82 are parameters controlling the tail probabilities of the underlying Gaussian and chi-squared distributions (Nadler 2008 Eq. (2.13)), and
= exp
p
e1 = Pr (1, 81};
2(v5 2)2
82 Pr
Xp-1
82
E3 = Pr {xi 833
p
vp
This bound shows that the misalignment of the empirical principal direction with the true signal direction scales approximately as
σ, P = X n
sin OPCA |
with additional correction terms involving higher-order noise and concentration effects, where the r.h.s. is the equivalent notation in the main lext. In addition, Nadler (2008) show in their Corollary that this bound is sharp, in the sense that it is also the the expected value for sin PPCA- This relationship justifies the dependence on sigma, X, n, d in our own estimate and supports the analysis used in Section of this paper.
F Additional Non-linear Diffusion Model Simulations
In this section we provide additional experiments to support Section 5 We follow the generation path in the sampling process of a single image and inspect the basis of the network Jacobians calculated at the intermediate sampled points Fy. We then trace sin 0, = v - {wt, of where the subscript "J" stands for Jacobian, of is the jth column the Jacobian diagonalizing basis V, defined in Equation 36 in similar way to our simulations of the linear case (Figure 2 Each plot represents the generation path of a single image, where the color is by index the darker the color, the lower the index.
We reproduce the experiment held in Section using two popular implementations Karras et al. 2022) of two different model architectures: *DDPM++ cont. (VP)" and "NCSN++ cont. (VE)" models by Song et al. (2021c), with the default parameters and 36 generation iterations. We repeat the experiment in conditional cifar-10 models, reported in Figure g In these cases as well, we see that lower indices preserve correlation the generated image in higher noise levels, towards the end of the generation process.
sind; www.cchrrggee..ooooo vend
drd cinettru.ccaszccrdo
48-
634
Figure 9: Image generation - the sine of the angle between Jacobian eigenvectors at the final generated image (t= 0) and intermediate iterations (t > 0), for "DDPM++ cont. (VP)" (right) and "NCSN+ cont. (VE)" (left) models. Color by index (the darker the color the lower the index, referring to columns of the Jacobian basis V.). The Jacobians of the nonlinear denoiser conform the behavior of the linear model.
22


Reviews:
[{"review_id": "cuUjwBJLKN", "reviewer": "Reviewer_MRyv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n - The analysis presented is rigorous and well founded in general. It is straightforward to follow and understand (but for a few minor issues, check Requested Changes)\n - The presented experimentation validates the observations derived.\n - I believe the analysis opens up avenues for further understanding diffusion models, and the results derived could be of value to the broader community. For example, it might be interesting to see if the learning of low frequencies first comes as a consequence of the 2-norm formulation of the losses, and if there could be similar results derived for other denoising (and consequently, different noise) models.\n\n**Weakenesses**:\n - The experimental extension to non-linear models is good, but maybe not entirely comprehensive. While it does seem that the proposed insights translate to the Jacobian’s eigenvalue and eigenvector analysis, it might’ve been more convincing to try it out on atleast a few more model/architecture variants. For one, given Ning et al., [35] (that this paper considers), one could consider the follow-up Li et al., ICLR 2024. Furthermore, could there be some correlation between the analysis carried out in the proposed approach, and in these works that deal with exposure bias in diffusion models (wherein the main claim is that the diffusion process is “two-stage” — first in moving generally towards to the data distribution, and second, in latching onto a single image that gets generated). Of course, I understand the constrains of doing full-data PCA on larger-dimensional data, so the authors could restrict themselves to the CIFAR-10, CelebA-64 scale. \n - Tying in to the point above, though I don’t expect significant experimentation here (but it would be a plus), is on how this analysis fares when dealing with latent-space diffusion, wherein the manifold hypothesis of data no longer holds, and the low-to-high-frequency distribution does not exist in the sense that the paper considers. Might be an interesting direction to address in the future at the very least, though I would then hope that the authors discuss it in the future scope. \n\n[1] Li et al., ALLEVIATING EXPOSURE BIAS IN DIFFUSION MODELS THROUGH SAMPLING WITH SHIFTED TIME STEPS, ICLR 2024", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a novel approach to interpreting and understanding diffusion models, albeit primarily in a simplified linear setting. The resulting analysis brings about a better understanding of denoising process in diffusion as one of projections onto principal directions of the data model. The paper also shows experimentation on some real-world data to correlate the observations — both by applying the linear model to MNIST data, and by analyzing the Jacobian through the same analysis on CelebA/CIFAR10.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Requested Changes**\n- Some diversity of experimentation on the non-linear model would help solidify the paper’s claim that the insights generalize to realistic settings as well. The authors consider two datasets, but at least 2-3 architecture choices, or models trained using varying strategies also conforming to the proposed analysis would be strongly supporting of this work\n- Adding discussions on the extension to latent spaces, or to non-$\\ell\\_2$ losses/models would be interesting\n \nThe rest of my concerns are relatively minor. In general I found the notations to sometimes get overloaded, causing the reading experience to get a tad bit jarring sometimes, but these are easily addressable concerns:\n - The notation is equation (9) felt confusing to me. Would’t it be better to differentiate between the function $D\\_t(\\cdot)$ in Eqn (8) and the the optimum $D\\_t^*(\\cdot)$ defined in Eqn (9). \n - Similarly in Sec. 3, the use of $q$ to denote the data distribution, with $q(x\\_t | x\\_{t-1})$ denoting the transition probability. Why not call the data model $q\\_d$? \n - Lastly, in Eqn (24), $D(x) = \\nabla D(x) x$ is confusing notation. It would help to differentiate the Denoiser $D$, and maybe the Jacobain of the neural-net as $\\nabla D\\_{Net}$ or some such notation. \n-In Fig. 6, it would help to have the scale of the y-axis aligned, since the primary goal is to bring to attention the fact that the correlations saturate to higher value in the middle figure. This insight is visually lost because of the scaling of the axes. Let all three figures have Y-axis in [0,1]. Please add Y-axis labels to Fig. 7 \n - It feels weird to not use grayscale to be plotting MNIST in Fig. 1, since that is the standard approach in the literature. Maybe the authors to fix their plotting function to not print MNIST figures on a heat-map color gradient. (Unless there is some particular reason to use it here that I might have missed).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NkAmdfbJUw", "reviewer": "Reviewer_3BeG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- it is an interesting idea to look at the spectral structure of linear denoisers. This is also insightful to understand the emergence of data structures during the generation process (e.g. early low frequency elements), which can lead to insightful developments.\n\nWeaknesses:\n\n- I think the paper has more weaknesses than the strengths. Overall I found the explanations confusing and complex, despite the paper essentially is in the \"linear setting\". There are multiple typesetting issues (pointed out below in requested changes) around referencing. Also I think figure captions do not read well, as well as the general message in the abstract. The paper would benefit lots from clarification both in text and explanations (see my requested changes).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers diffusion models in a setting where the reverse process is driven by a linear (learned) drift function - and attempts at bringing some understanding to general behavior of diffusion models by analyzing/demonstrating the properties of the learned score.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1) References are poorly formatted. At the moment, equations and references uses the same formatting. E.g. when the authors mention (19), it is unclear if the equation is referred to or a citation.\n\n2) What is $\\Sigma_t$ in eq. (9), please clarify.\n\n3) While I appreciate the full derivation is in Appendix, lines following eq. (9) and (10) are quite unclear. First $D_t$ is given as the expression in (9) without clarifying $\\Sigma_t$, then in the limit $\\sigma_t \\to 0$, some expression is given as the PCA denoiser.\n\n- What does the PCA denoiser mean?\n- What do you mean by \"the cleaner version of the training dataset\"?\n\nplease clarify.\n\n4) Fig. 1 caption would use some clarification. The first line is the denoising process, the second line is the target image? please clarify.\n\n5) Could you make the derivation of the approximation in eq. (17) self-contained please - moving from [34] the relevant derivation?\n\n6) In general, the results could be stated better and notation is confusing. I will compile some examples here, but this generalize and makes things unreadable.\n\nFor example, $x_{0:T}$ is used for diffusion model trajectory. But the training set is denoted $X = \\{x_1, \\ldots, x_n\\}$! Further more, before eq. (11), $X_{t-1}$ notation is introduced, \"cleaner version\" (as referred above), left unclear. $\\bar{\\sigma}_t$ is not defined when it is first used in (13), deferred, making things unclear. in Theorem 4.3, still define $\\hat{x}_T$ to make the statement self-contained. Please give a proofread to the paper.\n\n7) How are your results impacted by several approximations you are making? Why are they okay, how are they justified? Please add a remark about these.\n\n8) Can anything be said in Section 5, that is more interesting for general denoisers? If a denoiser \"near-linear\", can most of your results be applied using Jacobians?\n\n\n\n\"Assuming 4.2, 4.1 -> Assuming 4.1, 4.2\"", "broader_impact_concerns": "no concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "r8awnNGkNs", "reviewer": "Reviewer_6Q5P", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper presents an intriguing topic by theoretically analyzing the well-known phenomenon in diffusion models where coarse, low-frequency features are generated first, followed by finer, high-frequency features.\n\nThe linearization strategy and power method approach offer valuable insights without being overly simplistic.\n\nHowever, the conclusions drawn in the last three paragraphs on page 9 (The first eigenvector, The entire spectrum, and In between) are weak. To make the paper more compelling, the authors could explore additional findings, such as how the frequency of features generated varies with t, the differences between stochastic and deterministic samplers in this context, and the impact of varying the number of training samples on t.\n\nMoreover, the paper is not easy to follow because of its presentation style. (See below)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors analyze the generation process, considering a linear model as the score estimator or denoiser. In this case, the optimal denoiser is $D_t=(\\Sigma + \\sigma^2 I)^{-1} \\Sigma$, where $\\Sigma$ represents the data covariance and $\\sigma$ is the noise level. \n\nConsequently, the diffusion generative process simplifies to the sum of noise projected by the trained linear denoiser: $D_0 \\cdots D_T \\xi_T + \\cdots + D_0 \\xi_0.$\n\nThis leads to a generalization of the power iteration method.\n- The first part $D_0 \\cdots D_T \\xi_T$ aligned with the first eigenvector.\n- The last part $D_0 \\xi_0$ closely resembles the clean data.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Explain following related material in this paper.\n- correlation machine\n- spiked covariance model\n- low frequency feature in terms of linear space\n\nSection 3 of \"Problem Setup\" is confusing.\n- Begin with clear definitions of the practical setups in literature and theoretical ones used in this paper, then explain the differences between them.\n- Differentiate the definitions and derived properties.\n- $D_t$ has different meanings in equations (7), (9), and (13); use distinct notation.\n- $\\Sigma_t$ in equation (9) is undefined before.\n- Define $\\lambda_i$ and indicate $t$ in the notation since $\\lambda_i$ depend on $t$.\n- Clarify the definition of $X_t$ in equation (11).\n- What $\\bar{\\sigma}$ is and why both $\\sigma$ and $\\bar{\\sigma}$ are used.\n\nSection 4 is incomplete and lacks explanation.\n- Equation (15) is incorrect. Replace $U_t$ with the approximated of basis $\\hat{U}_t$ as noise can also perturb eigenvectors along with eigenvalues.\n- What does assumption, eq.(17) imply?", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)