Whitened Score Diffusion: A Structured Prior for Imaging Inverse Problems
Authors: Jeffrey Alido
Tongyu Li
Yu Sun
Lei Tian
Venue: NeurIPS 2025 poster ()
Keywords: diffusion models
inverse problems
computational imaging
stochastic differential equations
score-based diffusion models
bayesian inverse problems
Primary area: deep_learning
TL;DR: Whitened Score diffusion models enable stable training with arbitrary Gaussian noise by avoiding covariance inversion, improving performance on inverse problems with structured noise.

Abstract: Conventional score-based diffusion models (DMs) may struggle with anisotropic Gaussian diffusion processes due to the required inversion of covariance matrices in the denoising score matching training objective \cite{vincent_connection_2011}. We propose Whitened Score (WS) diffusion models, a novel framework based on stochastic differential equations that learns the Whitened Score function instead of the standard score. This approach circumvents covariance inversion, extending score-based DMs by enabling stable training of DMs on arbitrary Gaussian forward noising processes. WS DMs establish equivalence with flow matching for arbitrary Gaussian noise, allow for tailored spectral inductive biases, and provide strong Bayesian priors for imaging inverse problems with structured noise. We experiment with a variety of computational imaging tasks using the CIFAR, CelebA ($64\times64$), and CelebA-HQ ($256\times256$) datasets and demonstrate that WS diffusion priors trained on anisotropic Gaussian noising processes consistently outperform conventional diffusion priors based on isotropic Gaussian noise.

Whitened Score Diffusion: A Structured Prior for Imaging Inverse Problems
Yu Sun Lei Tian' Department of Electrical and Computer Engineering, Boston University, Boston, MA 02215, USA 2 Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21205, USA {jalido, tongyuli, lei Leitian)(br edu, ysun2148jh, edu
Jeffrey Alido' Tongyu Li!
Abstract
Conventional score-based diffusion models (DMs) may struggle with anisotropic Gaussian diffusion processes due to the required inversion of covariance matrices in the denoising score matching training objective [58]. We propose Whitened Score (WS) diffusion models, novel framework based stochastic differential equations that learns the Whitened Score function instead the standard score. This approach cireumvents covariance inversion, extending score-based DMs by enabling stable training of DMs on arbitrary Gaussian forward noising processes. WS DMs establish equivalence with flow matching for arbitrary Gaussian noise, allow for tailored spectral inductive biases, and provide strong Bayesian priors for imaging inverse problems with structured noise. We experiment with riety of computational imaging tasks using the CIFAR, CelebA (64 x 64), and CelebA-HQ (256 256) datasets and demonstrate that WS diffusion priors trained on anisotropic Gaussian noising processes consistently outperform conventional diffusion priors based on isotropic Gaussian noise. Our code is open-sourced
Introduction
Diffusion models (DMs) a powerful class of generative models that implicitly learn complex data distribution by modeling the (Stein) score function [48] 49 then plugged into a reverse denoising process described by an ordinary differential equation (ODE) or a stochastic differential equation (SDE) to generate novel samples from noise. Typically. the forward noising process defined by adding different levels of isotropic Gaussian noise a clean data sample, which enables simple and tractable denoising score matching (DSM) objective (58]. However, the DSM objective exhibits instability when the forward diffusion noise covariance ill-conditioned or singular, as its computation requires inverting the covariance matrix.
18 13
26 25 The score function
Flow matching (FM) [33 35 65 is alternative generative modeling paradigm that reshapes arbitrary known noise distribution into a complex data distribution according to an implicit probability path constructed by the flow. For the isotropic Gaussian case, 34 [2 established that FM and DMs are equivalent up to a rescaling of the noise parameters that define the SDE and probability paths. However, for anisotropic Gaussian noise, there exists gap between score-based DMs and FM, where score-based DMs cannot be as easily trained for arbitrary Gaussian forward noising processes due to the necessary inversion of the covariance matrix in the conditional score (58]
denoising DM capable of denoising structured, correlated noise is desirable in many scientific inverse problems, especially in imaging, as it may Imaging through fog, turbulence and scattering [ 61644 31,, wide-field microscopy 39 diffraction
rich Bayesian prior 44)
16.
[3] 28)
tomography [32 30 optical coherence tomography (OCT) 2]], interferometry (56) and many other
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

imaging modalities have an image formation process cortupted by structured, spatially correlated noise
[62] 6 in contrast the widely assumed additive isotropic (white) Gaussian noise. Conventional
DMs are trained on isotropic Gaussian noise, which may render them practically insufficient Bayesian priors for realistic use cases with correlated noise.
Motivated by FM's ability to model arbitrary probability paths and the expressiveness of dif- fusion priors for inverse problems, we propose Whitened Score Diffusion, a framework for learning DMs based on arbitrary Gaussian nois- ing processes. Instead of learning the (time- dependent) score function, Xx, log p(x,), we learn G, G. Vx, log px.)). with G, the diffu- sion matrix in the forward diffusion process (Fig. D. We term our framework Whitened Score (WS) DMs, after the whitening transfor- mation that transforms the score vector field into an isotropic vector field. This extends the current SDE framework for score-based DMs as it avoids the computation of the inverse co- variance for any anisotropic Gaussian noise in DSM objective, enabling an arbitrary choice of Gaussian probability paths, similar to FM. We elaborate on the equivalence of our framework to FM and draw connection the reverse- time diffusion process derivation by (8) where G G, V log (x,) is predictable process of the stochastic term in a reverse-timg -etereettmeeeeeee SDE.
SDDE with arbitrary Gauasian paths
ferwed tine SDIl
F.xdl G.dw
|Pax
G.G, V. kxg Gydw
vhlinnel
imaging system with additive structured noise
b
imaging problems
inleskk difracón
detbociie
This work presents an extension of score-based DMs to arbitrary Gaussian forward processes, bridging a gap between DMs and FM. Our framework enables a principled construction of denoising generative priors that incorporate spectral bias aligned with correlated measure- ment noise, leading to improved performance in inverse problems with structured noise. Empiri- cal results CIFAR-10, CelebA (64 x 64), and ColebA-HQ (256 x 256) across several imaging tasks show consistently higher peak signal-to- noise ratio (PSNR) reconstructions compared conventional models trained with isotropic
affecstial decscm
Our framework enables arbitrary Gaus- sian diffusion processes, allowing us to train a de- noising DM on diverse set of structured noise. The WS DM applies to a variety of imaging in- verse problems corrupted with correlated, struc- tured noise.
Figure 1:
noise. Our contributions are: (i) framework for training DMs that supports arbitrary Gaussian prob- ability paths, (ii) theoretical insights and connection FM, and (iii) demonstration of effective priors for imaging inverse problems under structured noise.
Background
2.1 Score-based diffusion models
Score-based DMs are class of generative models that estimate probability density function p(xn) by reversing a time-dependent noising process. continuous time, the forward noising process described by an Itô SDE in the form of,
dx; Fxx, + G.dw.
F. E m the drift coefficient, w R standard Wiener process (Brownian motion), and G, E mm is the diffusion matrix that controls the structure of the noise. The noise level is indexed by time 1 [0, T) such that Xo pxx)), XT ~ (O,T)) and x, ~ p(x, xo). probability transition Gaussian kernel defined by Eq.

---

The corresponding reverse-time time SDE for Eq. Dis:
dx,- [x; G, xx, log px,)] dt + G, dw,,
(2a)
and the deterministic ODE, also known as probability flow, with the same time-marginals is:
1 G.G. Vx. logpe(x.) de,
dx
FxX: 2
(2b)
where w is the reverse-time standard Wiener process and Xx, log (xx) is the Stein score function.
Sampling from p(Xo) requires solving Eq. 2 and thereby knowing the score function, Vx, log pa (xt).
48 49 approximated the score function with neural network sx(x,,)) t) by optimizing the DSM
objective [58]
i arg min (0, ()xx)_(())))))) (x)))xx)))))xxxxx(())))))
(x(, Vx, log Pi xo)]lā},
(3)
where the conditional score function has a closed form expression given by
(4) with ft and Σ, the mean and covariance of the Gaussian transition kernel p(x; xo). f4, and E, functions of the drift coefficient and diffusion matrix in Eq. attainne by solving the ODEs in Eqs. 5.50 and 5.51 in (44 creating linearly proportional relationship 14, Xo and E, G.G.T. This leads the transformed score function term, G, Vx, log p. (x,) in Eq. 2b to always be isotropic, as the covariance will multiply with its inverse, regardless of the diffusion coefficient, G₁-
Xx, log (x xu) E, (H4, (x)),
2.2 Structured Forward Processes in Diffusion Models
Conventional score-based based diffusion models (DMs) typically employ uncorrelated white Gaussian noise, corresponding to diagonal diffusion matrix G, 49 However, this formulation constrains the learned score function Vx, log p(x,) ) to isotropic noise settings. Extending beyond diagonal G2 poses numerical challenges, as the inversion of the covariance matrix in Eq. can become unstable for ill-conditioned or singular cases.
Recent studies have demonstrated that controlling the spatial frequency content of the forward noise can influence the model's inductive spectral bias and enhance generative flexibility (23]. Yet, a unified framework for training diffusion models under arbitrary noising processes remains lacking. Our
work addresses this gap through the lens of score-based DMs in the SDE formulation, establishing principled foundation for frequency-comtrooled and structured forward processes that can be adapted to diverse DM tasks.
Several prior approaches have introduced structured forward processes improve generative expres- sivity. CLD [15] and PSLD [40] extend the state space by incorporating velocity variables, injecting noise in phase space to simplify score estimation, albeit at the cost of auxiliary dynamics. MDMs 445 and Blurring Diffusion [19 employ anisotropic or spatially correlated noise but require inversion of dense covariance matrices. Flexible Diffusion [17] parameterizes the forward SDE to allow adaptive noise scheduling, increasing model complexity and training cost.
These advances collectively underscore the importance of moving beyond isotropic Gaussian noise while revealing practical limitations related to stability and computational overhead. Motivated by this, our proposed WS model enables arbitrary Gaussian forward processes without covariance inversion, offering a simple, stable, and general mechanism for structured generative modeling.
2.3 Flow matching
Flow matching (FM) [33 35 I is another paradigm in generative modeling that connects a noise distribution and data distribution with an ODE
(xx( u. (dbt
5
dt
for FM vector field u, (x:)) and initial condition do (Xo) Xo. Noise samples are transformed along time into sample from the data distribution using neural network that models the conditional FM vector field
ur
xx) Ex(x) (xx) mx(x))) + (x())))
(6)
3

---

where H4 (xx) and [x(x)) are the mean and covariance of the probability path Pt, and f" denotes the time derivative of f. Because Σ is proportional to Σ, up to scalar coefficient, multiplying by the
inverse to yield identity [54), the functional form of u, (x; xx) allows simple and stable training of
FM models with arbitrary Gaussian probability paths. This diagonal matrix- -ieediii multiplication currently lacks in score-based models due to the necessary inversion of the covariance matrix in Eq. I
We note that WS aligns with FM in the sense that both frameworks aim to enable arbitrary prob- ability paths. In Section 3.2 we present a formal connection between WS and FM. Nevertheless, FM may require new approaches to incorporate the measurement likelihood for solving inverse problems [65] whereas our WS framework can be readily combined with existing techniques for enforcing measurement consistency (see Section 2.4 for a review).
2.4 Imaging inverse problems with diffusion model priors
Reconstructing an unknown signal Xo E I from a measurement y E IR' given a known forward model y N (Ax;, - with Σy Rex" the covariance of the additive Gaussian noise and
A E R2xx the measurement forward model-is a central challenge in computational imaging and scientific problems. Recent advances employ DMs as flexible priors [14, using plug-and-play
schemes [66 67 60]. likelihood-guiin guided sampling via posterior score approximations [22] 49 9 I1]
47 27 Markov Chain Monte Carlo (MCMC) techniques [38 61 63 53 55], variational methods
16 37..
and latent DM frameworks
[3 [2 46].
Here, we adopt methods that approximate the posterior. The posterior score be factored into the prior score and the likelihood score using Bayes's ' rule to arrive at a modification of Eq. 2 for the stochastic reverse diffusion
dx = Fix, - G.G.T (Vx, (xx) + Vx, log xx))] di + G. dw.,
(7a)
and the deterministic reverse diffusion
1 G. (Vx, logpo + Vx, log xx)) dt.
dx;
Fix:
(7b)
The prior score is approximated by the denoising DM. However, the measurement likelihood score is intractable due to the immedeeennence. Methods in
50 make simplifying assumptions about the prior distribution, while those in [22 10 28 treat the likelihood score approximation as an empirically designed update using the measurement as a guiding signal. All these likelihood score approximations can thus be plugged into Eq. 7 to solve the inverse problem.
major gap in current research on imaging inverse problems is the consideration of additive structured noise. Most research on DM priors for imaging inverse problems has largely focused on scenarios with isotropic Gaussian noise, employing corresponding isotropic Gaussian denoising DMs. Recent work by (20) explored structured priors for imaging inverse problems using stochastic restoration priors achieving superior performance over conventional denoising DMs trained on isotropic Gaussian noising processes in cases involving both correlated and uncorrelated noise. However, a formal treatment of structured noise diffusion-based frameworks lacks, which we seek address.
3
Whitened Score Diffusion
We define our forward-time SDE with non-diagonal diffusion matrix as,
1
dx;
3r x,t + B.KK dw,
(8)
2
==
adopting from the aariance-preserviie (VP) SDE 49 In experiments, we constrain K to be in the class of circulant convolution matrices due to their ability to be implemented with the fast Fourier transform (FFT). However, method generalizes to any K that is positive semidefinite. When K I, we recover exactly the VP-SDE. The corresponding probability transition kernel of Eq.
p(x: Xo) = N (x: a,x0, (1 - a KK ))
'The mean and covariance of the transition kernel solved in Eqs. 5.50 and 5.51 in [4

---

COIV
Oure
mads
CDOM
7.044
18.45
20.36
8,63
RE:0
2957
3.73
4.38
1187
19.02
24
9b:6
214
2725
7.22
$299
12n5
CIFARIO
ColebA
Figure 2: Denoising correlated noise on CIFARIO and CelebA (64 64). We benchmark our WS DM trained on anisotropic Gaussian noise with the conventional DM (conv) trained on isotropic Gaussian noise. Left: results with fixed SNR of 0.26; Right: measurements with decreasing SNR from 1.4 to 0.12 using additive grayscale noise filtered by a Gaussian kernel of std 2.5 and 5 pixels for CIFARIO and CelebA, respectively. The PSNR is labeled in white.
where ar e 8,ds In general, a₁ is defined the integral of the drift coefficient from 0 to t, Og l F, F,ds. By leveraging the parameterization trick for Gaussian distributions, we may rewrite Eq. as the following continuous time system:
0o + V VI- oz 2~ ~(01)
(10)
Note that may other drift and diffusion matrices, such the variance-exploding (VE) SDE, ending up with scalar multiples of Xa and G,G, for the mean and covariance, respectively, given the initial conditions of H4 Xg and Eg 0. Specific to our SDE in Eq. g we define the signal-to-noise ratio (SNR) to the ratio a a?.
3.1
Whitened Score matching objective
From Eq. 9 the conditional score to solve Eq. is
(11) and inverting the matrix may often lead to instability the score computation. For example, the condition number of Gaussian convolution matrix grows the Gaussian kernel K widens, amplifying high spatial frequency features, leading to poor model training for the DSM objective in Eq. 3
Xx, log p(x; xx) ((1 ( KK (axX0 xx),
To mitigate these numerical instabilities in the score computation during training, we apply whitening transformation to the score by naturally multiplying it with G,G, where G. x Σ,, the forward diffusion process covariance. Similar DSM, for our SDE in Eq. & we approximate G. Vx, x log (x.) as G.G. Vx, log,(xx, xo) which has the following closed-form expression after canceling E, with G.G.
G. Vx, log xo) = Br 0g X
(12)
1-a of
train a model ng (xx, 1) using the following denoising WS matching loss:
Ex~U/(0 pxx, xare(() { [ t) G.G. Vx, log xo)
(13)
with proof in Appendix A This objective accounts for varying levels of spatial correlation in the noise to enable our model to denoise arbitrary Gaussian noise.
This objective defines a new learning target within the broader landscape of diffusion model losses. Our approach can be seen as generalization of noise prediction [18] to the setting of correlated Gaussian noise, where the preconditioning term G. captures the noise structure. Unlike conven- tional noise prediction, which assumes isotropic noise, our formulation enables stable training under
5

---

1
K12=119
second principal subspace
Figure 3: Vx, log p(x;
xo) vector field (white) and G, G,G.T Vx, log xo) vector field (red)
for increasingly anisotropic 2D Gaussian probability transition kernel p(x; xx)). The covariance amplifies the magnitude of the conditional score field by its condition number s<E), and additionally rotates the direction towards the first principal subspace where there is higher density, while the
G,G, Vx, log p(x, Xn) field remains stable in magnitude and directionally isotropic pointing towards the mean μ,, of the probability path.
arbitrary Gaussian forward processes. Furthermore, Eq. 13 reveals that the conditional FM vector field is linear combination of our conditional WS function and the drift term (see Appendix B highlighting that both WS and FM avoid covariance inversion by preconditioning the score function with G, G, This shared property enables principled modeling of flexible Gaussian probability paths.
3.2 Interpretation of WS
Concurrently with 44. 8 derived identical results for the reverse-time SDE, Eq. 2a by decomposing the diffusion term of a reverse-time SDE into a unique sum of zero-mean martingale and predictable process n,, given as
A E Gf I)G (x,, n2 ii=1 DE
(14)
(x(x))
When G; independent of the state Xs, n, simplifies to G. Vx. log Pr This process is conditionally deterministic with respect to the filtration of the reverse time flow, motivating modeling the complete predictable process instead of the score function in isolation.
Furthermore, multiplying the score with G. whitens its vector field, as seen in Fig. 3 leadinng to a two-fold effect. Firstly, the original score vector field is numerically unstable; its values highly sensitive to small errors in the residual, characterized by the condition number A( (E). This leads to unstable model training, as there often noise amplified by the condition number. Multiplying the field with GG a scalar multiple of the transition kernel's covariance, preconditions the field.
Secondly, the score field rotates in the direction towards the major principal axis that contains most of the density for the noise transition kernel p(x; Xn). For anisotropic Gaussian transition kernels, the score does not point towards the data distribution, but rather towards the major principal axis of the (correlated) noise from the forward-time SDE. Eq. [ naturally re-orients the field towards the data mean, providing motivation for modeling the complete predictable process instead of solely the score function. Furthermore, by learning the predictable process, we enable more general scheme for SDE-bbaed DMs developing model that will always have isotropic reverse-time sample paths without needing to specify the diffusion matrix during sampling.
Connection to FM To connect WS DMs with FM and explain why training models with arbitrary Gaussian probability paths achieved, we re- frame FM with the SDE framework and rewrite the conditional FM vector field expressed in terms of the VP-SDE variables in Eq. 8
(x xx) F2(2x, 0; G,G, Vx. logp(x, xx).
(15)
Eq. 15 reveals that the conditional FM vector u, is a linear combination of our conditional WS function and the drift term (see Appendix B for derivation). The key property shared by FM and WS

---

meas
conv
OUTS
conv
24.65
24.16
25.13
25.19
18:85
19.73
22.59
22 28
13.37
21:71
16 16.47
28 .33
9.71
16.82
20.13
Figure 4: Measurement noise with different covariance matrices shown in the bottom left of the measurement. The PSNR is shown in white text for a sample image in the CIFAR 10 validation dataset and the CelebA (64 64) validation dataset. Compared to DMs trained only isotropic Gaussian noise, our WS model is able to denoise correlated noise with superior PSNR. For uncorrelated noise (first row), our model has similar performance conventional DMs.
DMs is that they avoid inverting the covariance matrix in the score by preconditioning it with G, G,G.T enabling flexible modeling of arbitrary Gaussian probability paths.
3.3 WS diffusion priors for imaging inverse problems
We solve the imaging inverse problem using
Eq. with our WS diffusion prior and an approx- imation of the measurement likelihood score. Recall from Section 2.4 the myriad of methods developed to approximate the measurement like-
Algorithm WS diffusion priors for imaging in- verse problems
Require: T. A. y. {B,)7 ng
1: Initialize Xy ~ (0,1))
lihood score, all of which follow the template,
2: for = to do
3: x (2 VI B,A + ()A
Vx, logply x() = Ey r(x)) (16)
4: X X BAA Axi)
where (x,) is the gradient of the residual function that guides the update x, towards re- gions where the observation y is more likely.
5: end for
6: return X
The reverse-time SDE framework aids inverse problems with correlated noise as the diffusion matrix G.G. preconditions the inverse measurement covariance in the likelihood score, when G. is designed to proportional to the covariance matrix.
G.G. Vx, logp(y(xx) = G.G. E vx,r(x.).
(17)
In designing our diffusion process, we set G.G.T = K.K + where KKT encompasses a large of measurement noise covariances, and 72 drawn uniformly between and in order to encourage the model to learn finer detailed features that reside in high spatial frequency subspaces.
In practice, regularization term X is important to balance the generative prior with the data likelihood. For proof-of-conceppp we experiment with the likelihood-guides -guided sampling via posterior score approximation in [22] due to functional simplicity Vx, log x.) Σy E-1AH( (y Ax,). We also the deterministic sampler of Eq. 7b The resulting algorithm is shown in Algorithm
7

---

meas
CoTV
Ours
meas
conv
OurS
gt
10.47
13:01
14.47
15.17
CIFAR
12.42
12:90
11.16
13:74
10.61
12:56
15:11
11.10
13.39
motion deblur
lens deblur
Figure 5: Motion and lens deblurring on CIFARIO dataset with additive spatially correlated grayscale Gaussian noise of std 2.5 pixels. WS diffusion prior consistently removes correlated noise resulting in higher PSNR compared to DMs trained solely on isotropic Gaussian noise.
4 Experiments
4.1 Training details
For each dataset, we train two attention UNet models based on the architecture in (18) with three residual blocks in each downsampling layer, where one is for the conventional isotropic Gaussian SDE, and the other our anisotropic Gaussian SDE. We set the learning rate to 3e with a linear decay schedule. For CIFARIO (32 32), the batch size is 128, for CelebA (64 x 64), the batch size is 16, and for CelebA-HQ (256 x 256), the batch size is 4. Models were trained on single NVIDIA L40S GPU with 48GB of memory for two days. Our model trained on the training sets of CIFAR- 10 29).. CelebA (64 x 64) [36] and CelebA-HQ (256 x 256) [24] where K is 2D Gaussian convolutional matrix characterized by an std. For CIFAR, the sld that characterizes
is uniformly distributed between 0.1 and 3, between 0.1 and 5 for CelebA (64 64), and between 0. and 20 for CelebA-HQ (256 x 256) where std 0.5 equals the 2D delta function. The noise is also randomly grayscale color with 0.5 probability.
4.2 Imaging inverse problems with correlated noise
is well-established that natural image spectra exhibit exponential decay [57], indicating the dom- inance of cow-frequecyy components in representing images. When additive measurement noise occupies the same spectral subspace, especially at low frequencies, the computational imaging task becomes fundamentally more challenging. We show that our framework is beneficial as generative prior for solving inverse problems with such structured noise by experimenting with a variety of computational imaging modalities that are known to be affected by structured noise.
The measurements in experiments are corrupted by additive grayscale structured noise, designed to mimic real-wor world conditions frequently encountered in both computational photography such as fog. haze, and atmospheric turbulence-ane and computational background, laser speckle, and detector noise. We use Algorithm with T = 1000 and 8min = 0.01 and Bmar = 20 so that the SNR decays at T. For results with our WS prior, Xr was drawn from (O KKT) with std and std for CIFAR and CelebA, respectively with grayscale color (all color channels have the same value). For conventional DM prior results Xy was drawn from (0, I). evaluation was performed on unseen validation dataset sample images picked uniformly at random. The regularization parameter À scales the magnitude of the likelihood step to be proportional to the magnitude of the prior step as was done by [22]. Line search was used to find an optimal À that yielded reconstruction with the highest PSNR, where PSNR is defined as
fluorescence

---

ours
gt
conv
meas
measurement noise support
CIFAR
X
conv ours
ColebA
Figure 6: Effect of changing regularization parameter À for denoising. The top figures the average power spectral density of the images in CIFAR and CclebA, with a dotted red circle to denote the frequency support of the additive correlated noise. Changing the regularization weight À for denoising affects the final reconstructions using our WS diffusion priors (top) and conventional diffusion priors (bottom). When À is O, it is equivalent sampling form p(x). As À increases, the generative modeling effect overpowered by the measurement fidelity term, that the reconstruction resembles the measurement y.
PSNR 20 logio (MSE) where MSE is the mean squared error between the reconstruction and the ground truth.
Our results demonstrated variety of computational imaging tasks such imaging through fog, motion deblurring, lens deblurring, linear inverse scattering, and differential defocus. More details are Appendix C
Denoising correlated noise To demonstrate the capabilities our model as a generative prior for measurements corrupted by correlated noise, explore the denoising problem and compare the results with that of a conventional score-bas basee diffusion prior that was trained only on isotropic Gaussian forward diffusion. Fig. shows the results CIFAR and CelebA (64 64) test samples across a range of SNRs, where color is faithfully restored from fog-like corruption and likeness to the dataset is maintained due to the generative prior.
Generalize different noise structures Our model generalizes to different measurement noise covariance matrices with varying Gaussian noise distributions. Fig. reveals that measurements corrupted by different distributions spatially correlated Gaussian noise are restored with higher PSNR compared to conventional DM priors (conv). Conventional basre-based priors change the higher level semantic features of the measurement, due to the model's inability to distinguish noisy features from target image features based Fourier support. Specifically, the added correlated noise's low frequency support overlaps with that the visual features the data, seen in Fig. 6 This makes the reconstruction task more difficult.
Spectral inductive bias WS diffusion priors more effectively distinguish structured noise from target features compared to conventional diffusion priors trained on isotropic Gaussian noise. As shown in Fig. [ standard DMs tend to suppress high-frequency components in the measurement, assuming they originate from noise-a valid assumption only when the noise Fourier support extends beyond that the data. For CIFAR, whose average signal spectrum extends beyond the noise's ss this misclassification leads to undesired attenuation of image features. In contrast, for CelebA (64 x 64), where the average image spectrum lies within the noise support, conventional models better preserve image features. WS DMs, trained on ensembles of Gaussian trajectories, learn to identify structured noise beyond simple spectral heuristics. This enables selective removal of low-frequency noise even when it spectrally overlaps with signal content, yielding improved denoising performance in the presence of correlated noise.

---

Computational imaging Using WS diffusion prior to solve inverse problems with non-identity forward operators outperforms traditional score-based diffusion priors in PSNR. Noticeably, our diffusion prior is able to maintain fidelity to the color distribution for restoring measurements corrupted by grayscale fog-like noise, while conventional score-based diffusion priors fail to remove the noise, as seen in Figs. 5 and 8 for deblurring inverse problems. Additional results for other imaging inverse problems on CIFAR, as well as on the CelebA (64 64) and CelebA-HQ (256 x 256)
datasets, are presented in Appendix and and Figs. 7
8
10 and 11
5 Conclusion
We introduced WS diffusion, a generalization of score-based methods that learns the Whitened Score, G,G, Xx, log (x,). This avoids noise covariance inversion, enabling arbitrary anisotropic Gaussian forward processes and bridging connections to FM. We demonstrate WS diffusion as robust generative priors for inverse problems involving correlated noise, common in computational imaging. Experiments consistently showed superior PSNR and visual reconstructions compared to conventional diffusion priors trained on isotropic noise, particularly in accurately handling structured noise while preserving image features. WS diffusion provides principled approach for developing effective generative models tailored structured noise, advancing their utility in computational imaging applications.
Limitations and future work primary limitation of our approach lies in the computational cost of sampling. The current time discretization of the reverse- evversettme SDE necessitates approximately 1000 denoising steps, which may be prohibitive for certain practical applications. Reducing the number of denoising steps through model distillation represents promising direction for future work [44 51
Another limitation concerns the absence of an explicit mechanism to estimate the measurement noise covariance, which directly influences the specification the diffusion matrix G, A natural extension of this framework would involve parameterizing and learning jointly with the model parameters. Such an approach would allow the diffusion process to adaptively capture data-dependent or task- specific noise structures, thereby enhancing the model's flexibility and representational capacity. This line of work connects to recent advances in vector-valus and multivariate diffusion models, which have demonstrated improved performance in scenarios characterized by complex or structured noise.
Finally, while our model exhibits strong performance as a denoising prior, additional research is needed for WS DMs to achieve competitive results in unconditional conditional generation tasks. Promising directions include latent diffusion formulations and related techniques for improving generative efficiency and expressiveness 42 41)
10

---

Acknowledgements We are grateful for grant from 5022 Chan Zuckerberg Initiative DAF, an advised fund of Silicon Valley Community Foundation. We also thank the Boston University Shared Computing Cluster for computational resources. J.A. acknowledges funding from the NSF Graduate Research Fellowship Program (GRFP) under Grant No. 2234657.
References
[1] Michael S. Albergo and Eric Vanden -ijnnen.. Building normalizing flows with stochastic interpolants, 2023. URL https:t//axxiii wwwpxiii.rgg//ba.. wwwg/b//////eeeeo/oo 2209. 15571
[2] Emma Alexander, Qi Guo, Sanjeev Koppal, Steven Gortler, and Todd Zickler. Focal Flow: Measuring Distance and Velocity with Defocus and Differential Motion. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision ECCV 2016, pages 667-682, Cham, 2016. Springer International Publishing. ISBN 978-3-319-46487-99
[3] Jeffrey Alido, Joseph Greene, Yujia Xue, Guorong Hu, Yunzhe Li, Mitchell Gilmore, Kevin J. Monk, Brett T. DiBenedictis, Ian Davison, and Lei Tian. Robust single-shot 3d fluorescence imaging in scattering media with simulator-trained neural network. Opt. Express, 32(4):6241- 6257, Feb 2024. doi: 10.133//0E.51400
URL httt:hhtt//ppptt /ttttpp://pp.ooptia... owg/oe/////eoooooooeeooeeeoo/t// abbatraact
cfm?URI=0e-33 101-1---0-22----00 4-641
[4] Brian D. O. Anderson. and their Applications, 12(3):313-32, May 1982. ISSN 0304-4149. doi: 10.1016/
Reverse-time diffusion equation models. Stochastic Processes
0304-4149(82)90051- URL https://www..r wwtp:w/wccicncdiec..... com comm/ciiecce/article.
p11/0804444982900555
[5] Benjamin Boys, Mark Girolami, Jakiw Pidstrigach, Sebastian Reich, Alan Mosca, and Omer Deniz Akyildiz. Tweedie moment projected diffusions for inverse problems, 2024. URL haatts/s//ppprrevvveeeeeeerrrrra
[6] Coleman Broaddus, Alexander Krull, Martin Weigert, Uwe Schmidt, and Gene Myers. Remov- ing Structured Noise with Self-Supervised Blind-Spot Networks. In 2020 IEEE 17th Interna- tional Symposium on Biomedical Imaging (ISBI), pages 159-163, April 2020. doi: 10. 109/ ISB145749.2090
URL https://ieeeepplore ieee. httgg///occmee//99830030
ISSN: 1945-8452.
[7] Gabriel Cardoso, Yazid Janati el idrissi, Sylvain Le Corff, and Eric Moulines. Monte carlo guided denoising diffusion models for bayesian linear inverse problems. In The Twelfth Inter- national Conference on Learning Representations. 2024. URL https::/ net/ forum?iddii id=nhnsshHnEEEH
[8] David Castanon. Reverse-time diffusion processes (Corresp.). IEEE Transactions on Infor- mation Theory, 2866)955- November 1982. ISSN 1557-9654, doi: 10. 10.1//9//T..99 1056571. URL ence Name: IEEE Transactions on Information Theory.
ieeeeerreeeeeeeeeeeeeeeeeeeeeeee ieee.
Confer-
[9] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models, 2021. URL https://
arxiv. .rw. abs/2105 02938
[10] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models. In 2021 IEEE/CVF International Conference Computer Vision (ICCV), pages 14347-14356. October 2021. doi: 10.11099ICCV4499222222.000
URL URL htlp::tttt:://eeppl.oo ieee org/ document/
9711284
ISSN: 2380-7504.
[11] Hyungjin Chung, Jeongsol Kim, Michael T. Mccann, Marc L. Klasky, and Jong Chul Ye. Diffusson Posterior Sampling for General Noisy Inverse Problems, May 2024. URL http
//arxiv.oo orwilv..rrg/ee httg/bb//22000 14687 MrXiv:2209.1466 [stat].
[12] Hyungjin Chung. Jong Chul Ye, Peyman Milanfar, and Mauricio Delbracio. Prompt-tuning http:/perreiiie.ee/ffouunmll net/forum?
latent diffusion models for inverse problems, 2024.
id=ckzglrAMsh

---

[13] Giannis Daras, Mauricio Delbracio, Hossein Talebi, Alexandros G. Dimakis, and Peyman Milanfar. Soft Diffusion: Score Matching for General Corruptions, October 2022. URL
http://arxiv: www.aaiwii....raaaee.eeeeeoeeeee a2gg9 2205//022900042 arXiv:2209.05442 [es].
[14] Giannis Daras, Hyungjin Chung, Chieh-Hsin Lai, Yuki Mitsufuji, Jong Chul Ye, Peyman Milan- far, Alexandros G. Dimakis, and Mauricio Delbracio. Survey on Diffusion Models for Inverse Problems, September 2024. URL http::tttt/::te org/ ab//abbs/.1... 2410.00083 arXiv:2410.00083 [es].
[15] Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically-damped langevin diffusion. arXiv preprint arXiv:2112.07068, 2021.
[16] Berthy T. Feng, Jamie Smith, Michael Rubinstein, Huiwen Chang. Katherine L.. Bouman, and William Freeman. Score-basee based diffusion models as principled priors for inverse imaging. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10520-10531, October 2023.
[17] William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffiusion modeling of long vidcos. Advances in neural information processing systems, 35:27953-27966 2022.
[18] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models, December 2020. URL //at:://arxivv org/ abaaabas/s00/ 11239 arXiv:2006. 1239 [cs].
[19] Emiel Hoogeboom and Tim Salimans.
Blurring diffusion models.
arXiv preprint
arXiv:2209.05537,200 2022.
[20] Yuyang Hu, Albert Peng, Weijie Gan, Peyman Milanfar, Mauricio Delbracio, and Ulugbek Kamilov. Stochastic deep restoration priors for imaging inverse problems, 2025. URL https
//openreiie net./forumiddd02aio222..
[21] David Huang, Eric A. Swanson, Charles P.
Lin, Joel S. Schuman, William G. Stinson, Warren Chang, Michael R. Hee, Thomas Flotte, Kenton Gregory, Carmen A. Puliafito, and James G. Fujimoto. Optical Coherence Tomography. Science, 17885 181, November 1991. doi: wee.1////cieneee199070
|ttttts::// wwwence.ecc..c..c...o.... Publisher: American Association for the Advancement of Science
wraeaceeerggee org/ owt//oo///////ooo 0o1/10. 1126/
1957169
[22] Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G. Dimakis, and Jonathan I. Tamir. Robust Compressed Sensing MRI with Deep Generative Priors, December 2021. URL 01368 arXiv:2108.0133 [cs].
[23] Thomas Jiralerspong, Berton Earnshaw, Jason Hartford, Yoshua Bengio, and Luca Scimeca. Shaping inductive bias diffusion models through frequency- noise control, 2025, URL https://arxiv org/ aba/ 10236
[24] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017.
[25] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the Design Space of Diffusion- Based Generative Models. Advances in Neural Information Processing Systems, 35: 26565 26577. December 2022. URL wtt.://ssstttttttttttttt /ttpatpss//ppocceennnssn
htt.://nww...uurrss..ceeeeaeapep erence.
html
[26] Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and Improving the Training Dynamics of Diffusion Models, March 2024. URL http://arxiv.orz wttpa///rrrra/b///////a/ ab//23/2.22.22.. 02696) arXiv:2312.02696 [cs].
[27] Bahjat Kawar, Gregory Vaksman, and Michael Elad. SNIPS: Solving Noisy Inverse Problems Stochastically, November 2021. URL http://arxii: http://aarxi..oo/rb///////// 14951 arXiv:2105.1 14955 [cess].
(28) Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models, 2022. URL https: httee://axxvvvvvo org. /ttps//20..1177
12

---

[29] Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images.
[30] Tongyu Li, Jiabei Zhu, Yi Shen, and Lei Tian. Reflection mode diffraction tomography of multiple-scateringg samples on reflective substrate from intensity images. Optica, 12(3):406- 417, Mar 2025. doi: 10. 10.1364/PPICA
URL https:///// opt.t/pp..pppiiaaaapp. httpeiic..rg//gooe wtt.ooot.....oootiicaaaa.....aa. optica/
abstract.cff cbstrratt.ccfPPi.pppppaa---2---2
[31] Huakang Lin and Chunling Luo. Demonstration of computational ghost imaging through
fog. Optics & Laser Technology, 182:1 192:11200 12075, April 2025. ISSN 0030-3992. doi: 10.1016
Loptllastec.2024.111 12075. URL htttps:/wwwz http://wwww.iiiiieeeeeeeteeeeeee httpp//eeeeeeeceeeeeeecc/rrrtttt
p11/$0030339224013330
[32] Ruilong Ling, Waleed Tahir, Hsing-Ying Lin, Hakho Lee, and Lei Tian. High-throughput intensity diffraction lomography with a computational microscope. Biomed. Opt. Express, 9()): 2130-2141, May 2018 doi: 10. 10.136//0E.900021330 URL httpp:///pppt optica. org/boe/
abstract. Siapraact.ccmmUUR--oe------- 9-1-
[33] Yaron Lipman, Ricky T. Q. Chen, Heli Ben- Maximilian Nickel, and Matt Lc. Flow Matching for Generative Modeling, February 2023. URL http::/arxiv.. org/ wwwp:/ogg///b//////// 02747
arXiv:2110002747 [cs].
[34] Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky T. Q. Chen, David Paz, Heli Ben- Hamu, and Itai Gat. Flow matching guide and code, 2024. URL htt.::tttt///aatttte
org/ ab////b///// 26264.6662...
[35] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow, 2022. URL http::ttp//// arxiv wwwpsiiivrrr/.ooooooo abs/ 020ps//229..o0 2200/22290000
[36] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[37] Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat. A variational perspective on solving inverse problems with diffusion models. In The Twelfth International Conference on Learning Representations, 2024. URL /tattps://openreviiee
meetfforumidd-11
[38] Naoki Murata, Koichi Saito, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Yuki Mitsufuji, and Stefano Ermon. Gibbsddrm: A partially collapsed gibbs sampler for solving blind inverse problems with denoising diffusion restoration, 2023. URL https:/tt:///att //apxii////xvvv..// wwwrxiv.or//ooe www.oorg////aaaeeee..eee abs/2301 12686
[39] Leonhard Möckl, Anish R. Roy, Petar N. Petrov, and W. E. Moerner. Accurate and rapid background estimation in single-molecule localization microscopy using the deep neural net- work BGnet. Proceedings of the National Academy of Sciences, 117(1):60-67 January 2020. doi: 10. 10.1073/pnas... 10073/pnas.9112199111 1916219117. URL https: ://www pnas, org/ hts//di//// mooo/fful1/10.... 1073/ 1916219117 Publisher: Proceedings of the National Academy of Sciences.
[40] Kushagra Pandey, Maja Rudolph. and Stephan Mandt. Efficient integrators for diffusion generative models. arXiv preprint arXiv:2310.07894, 2023.
[41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-Resolution Image Synthesis with Latent Diffusion Models, April 2022. URL http // //arxiv. hwwgg/abs/212.... 10752 arXiv:2112.10752 [cs].
[42] Litu Rout, Yujia Chen, Abhishek Kumar, Constantine Caramanis, Sanjay Shakkottai, and Wen- Sheng Chu. Beyond First-Order Tweedie: Solving Inverse Problems using Latent Diffusion, December 2023. URL http://:
org/ abs/2312. 2312.00852 00852 trXiv:2312.0088 [cs].
[43] Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alex Dimakis, and Sanjay Shakkottai. Solving linear inverse problems provably via posterior sampling with latent diffusion models. In Seirth-seventh Conference on Neural Information Processing Systems, 2023. URL http::://openreviee.nettt/ (t-torrrmmdd-XXXFFfddo....
[44] Tim Salimans and Jonathan Ho. Progressive Distillation for Fast Sampling of Diffusion Models, June 2022. http:///axxvvvvee wwtpab//222........m.// 00512 arXiv:2202.00552 [cs].
13

---

[45] Raghav Singhal, Mark Goldstein, and Rajesh Ranganath. Where to diffuse, how to dif- fuse, and how get back: Automated learning for multivariate diffusions. arXiv preprint arXiv:2302.07261, 2023.
[46] Bowen Song. Soo Min Kwon, Zecheng Zhang, Xinyu Hu, Qing Qu, and Liyue Shen. Solving inverse problems with latent diffusion models via hard data consistency. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreeial htte/foruum1d--jBdRRQUUUl
[47] Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. sesloiinvers-guuddee diffusion models for inverse problems. In International Conference on Learning Representations, 2023.
URL https:://pperreieeee wwt.:/neeeffummmmommmmmm 9_
[48] Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribution, October 2020. URL attp://arxiv.. wwt///g////////// abs/190/ 0049007056600 arXiv:1907.05600
[cs].
[49] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations, February 2021.
wtw.:o////aa.................... 13456 arXiv:2011.13456 [es].
[50] Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medi- cal imaging with score-based generative models. In International Conference on Learning Representations, 2022. URL
[51] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency Models, May 2023. http//bb//33033 2001//1630300009999 01469 arXiv:2303.0146) [es].
URL hwt.:http:/aaat
[52] Peng Sun, Yi Jiang, and Tao Lin. Unified continuous generative models, 2025, URL https //arxiv. org/ ptrgg/bss/20e abs/ 07447
[53] Sun, Zihui Wu, Yifan Chen, Berthy T. Feng, and Katherine L. Bouman. Provable Probabilistic Imaging Using Score-Based Generative Priors. IEEE Transactions on Computational Imaging, 10:1290-1305 2024. ISSN 2333-9403.
doi: 10. 1109/TTC...22.44499144 URL https
//ieeexplore ieee. ort.
[54] Simo Särkkä and Arno Solin. Applied Stochastic Differential Equations. Institute of Mathematical Statistics Textbooks. Cambridge University Press, Cambridge, 2019. ISBN 978-1-316-51008-7
doi: 10.101779781108118777
URL https://wwa
cambridge org/ ctte//oreeeeeeeeeee bwo.s//ooosssssooo/eoooooo..ae 1B8118800119F8012211AAA.. 6EAACC88C2EAA
[55] Brian L. Trippe, Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi S. Jaakkola. Diffusion probabilistic modeling of protein backbones in 3d for the motif- scaffolding problem. In The Eleventh International Conference on Learning Representations, 2023. URL attg:s//pppaaeeeeeeeeee/o/oooora net.fooruridd-TxxxxaPM.
[56] Takafumi Tsukui, Satoru Iguchi, Ikki Mitsuhashi, and Kenichi Tadaki. Estimating the statistical uncertainty due to spatially correlated noise in interfcrometric images. Journal of Astronomical Telescopes, Instruments, and Systems, 9(1):01800) 2023. doi: 10.1117/L.JAIIS... 1.0000.. URL https: org/ 111..111111..111111.1.. 9. 018001 Publisher: SPIE.
[57] A. van der Schaaf and J.H. van Hateren. Modelling the power spectra of natural images: Statistics and information. Vision Research, 36( (7)::7999-27700 1996. ISSN 0042-6989, doi: http:///oooog///0000011000226669
URL /tt.htts:://www.cieeeee httt:///wwwwcciicceeerreeeee. com/
[58] Pascal Vincent. A Connection Between Score Matching and Denoising Autoencoders. Neural Computation, 23(7):1661-1674, July 2011. ISSN 0899-7667, 1530-888X. doi: 16.//O2O
14

---

[59] Laura Waller, Lei Tian, and George Barbastathis. Transport of Intensity phase-amplituee imaging with higher order intensity derivatives. Optics Express. 18(12): 1211111522-111,,, June 2010. ISSN 1094-4087. doi: 10. 1364/OF. 18.012555. Rtt.ttps: https://opg.
optica. org/ oe/ Publisher: Optica Publishing Group.
18- 0fo 12552
[60] Hengkang Wang, Xu Zhang, Taihui Li, Yuxiang Wan, Tiancong Chen, and Ju Sun. Dmplug:
A plug-in method for solving inverse problems with diffusion models, 2024. URL https 7/arxiv. org/ abs 16749
[61] Zihui Wu, Yu Sun, Yifan Chen, Bingliang Zhang, Yisong Yue, and Katherine L. Bouman. Principled probabilistic imaging using diffusion models plug-and-play priors, 2024. URL
https://arxiv org/ a405 18782
[62] Ali Zafari and Shirin Jalali. Bayesian Despeckling of Structured Sources, January 2025. URL http://arxiv org/ abaaabss///00/ 11860 arXiv:2501. 1160 [es].
[63] Bingliang Zhang, Wenda Chu, Julius Berner, Chenlin Meng, Anima Anandkumar, and Yang Song. Improving Diffusion Inverse Problem Solving with Decoupled Noise Annealing, Decem-
ber 2024. URL httt:://axixl http://xxxx.oorgaa///a/a/aaaaaa/ /400 015221 urXiv:2407.0133 [cs].
[64] Xingguang Zhang, Zhiyuan Mao, Nicholas Chimitt, and Stanley H. Chan. Imaging Through the Atmosphere Using Turbulence Mitigation Transformer. IEEE Transactions Computational Imaging, 10:11 2024. ISSN 2333-9403. doi: 00011099/CC12224:33554111 URL https //ieeexplore ieee. org/ abstract/
(locument/104009225 //t.e//..e...a.aaaataaaaaaat///a
[65] Yasi Zhang. Peiyu Yu, Yaxuan Zhu, Yingshan Chang, Feng Gao. Ying Nian Wu, and Oscar Leong. Flow Priors for Linear Inverse Problems via Iterative Corrupted Trajectory Matching, January 2025. URLhttp: htttp::/aaziii..g//bb//////
18816 1XXi:22405.18810 [cs].
[66] Hongkai Zheng, Wenda Chu, Bingliang Zhang, Zihui Wu, Austin Wang, Berthy T. Feng, Caifeng Zou, Yu Sun, Nikola Kovachki, Zachary E. Ross, Katherine I.. Bouman, and Yisong Yue. Inversehench: Benchmarking plug- and-play diffusion priors for inverse problems in physical sciences, 2025, URL
org/ abs 2t1.//500...1333331
[67] Yuanzhi Zhu, Kai Zhang, Jingyun Liang, Jiezhang Cao, Bihan Wen, Radu Timofte, and Luc Van Gool. Denoising diffusion models for plug-and-ple image restoration, 2023. URL wttp:///wwwww///////000003330
15

---

A Denoising Whitened Score matching
Lemma A.1. (Generalized Tweedie formula for non-diagonal covariance). Let:
X 01 + 2,
(18)
where Xo p(x) and z ~ (0, E). Then
ax.Exx xt] = + EV log):(Xx)
(19)
Proof.
Vx.Pe(x,)
Vx, log (x(e))
(x()
1
Vxe / (x(xee xo) dxo
(de-man marginaaiilz joint distribution)
pr(x))
1
/ Px(xx (x)) dxo
Xx.
(factor joint via Bayes rule)
((()))
/ Vx.Pe(xe xx)) dxo
(move gradient inside integral)
P((x))
1
/ Pc(xse Vx, log (x (x))o(XX)) dxo
(use identity ff fv log f)
Px(x))
/ po(xo xx) -- -x,) dxo
(Gaussian conditional score)
Σ- (ar E(xo x1] xx)
(expectation under posterior)
Theorem A.2. (Denoising Whitened Score matching)
Our Whitened Score matching loss function, Eq. [3 copied here as:
10 xo {1lmo (x,,t) G.G. Vx, log xo)
(20)
is a denoising objective that uses the conditional probability to estimate G.G. Vx, log.:(x.). Here we prove that our loss function results in an estimator for G.G, Vx, log (x.))
Proof. Let p(x, xx) denote the Gaussian probability transition kernel associated with the forward-
time SDE in Eq. For linear SDE in x,, the covariance E, of the transition kernel is scalar multiple of twice the diffusion matrix, G.G. (154) provided the initial conditions of (a( = Xo
and E()) 0 for pxx, xo):
Σ, cG,
(21)
The Minimum Mean Squared Estimator (MMSE) E[xo xx] is achieved through optimizing the least squares objective:
min Expoe (xe xxxx)())))
(xx,)
x
(22)
such that ho- (x.) E[xo x.). for optimal network parameters 0" Tweedie's formula from Lemma A.1 gives us that,
ag (xx) X2 Vxx, log (xx)).
(23)
Parameterizing he (x:) as
hx(x:) X₁ + cx,
(24)
with scalars c, ag E lR and (x(x),() () our G.G.T Vx, logpt((;) model, the MMSE objective in Eq.
22
becomes
min F
[||x + (x, t) 0: l
(25)
which is equivalent to our objective in Eq. 20 through the closed form expression of G. Vx, log xo) given in Eq. 12 Finding the optimal 0* implies
16

---

X, + eng (x,,t) + VV x, log Pa (x.)
hg. (xx)
E(xo xx]
(26)
ar
which proves that our model ng (x 8) learns G,G, Vx, logp((x)) with objective Eq. 20
B
Flow matching in SDE
Consider the probability path
Ps Nix (x(x)), Ee Ex(x)))
(27)
and the corresponding continuous normalizing flow:
(xxx) (x()) + Eij (x0),
(28)
where we define Σ (xo) such that Cov(o, (x,) = Σ (xn) (xx)) and the initial
condition (xx) = This probability path is equívalent to the probability transition kernel in Eq. defined by linear SDE Eq. with drift coefficient F, and diffusion matrix G,, Therefore we may
attain the time derivatives of the mean and covariance functions using Fokker-Planck (see Eqs. 6.2 in
54
expressed as
Ex(x)) = 2F (3, + G.G.,
(29)
(x(())) (xo).
(30)
The FM conditional vector field for Gaussian probability paths is
un (X:
Xg)
Σx(x,) E (xx) (xoo)) + (xxo)
(31)
Plugging in Eqs.
30 and
29 into Eq. 31 have
(x
xo)
E E, (xx) 14, (x0)) + (x((x))
(32)
= =2F, + G,G, E-- (xx) - axx1)) + F Fx14,(x))
(33)
=2F,x, F,0,x0 + G.G, Σ, (xo) 0x80)
(34)
=F,(2x xxx)) G.G. Xx, log p(x, xo).
(35)
C Imaging inverse problems
Imaging through fog/turbulence We simplify imaging through fog/turbulence denoising
problem for correlated noise which we achieve by setting the imaging system A = I. Specifically, demonstrate our model for grayscale low-pass filtered white Gaussian noise, where K is 21) Gaussian kernel characterized by std.
In our method, the model is trained to denoise correlated noise, and is able to distinguish target image features from anisotropic Gaussian noise features, even though both may share similar spatial frequency support. In contrast, the conventional DM trained to denoise only isotropic Gaussian noise, removes the images features for low enough X, as mistakes the correlated additive noise for the target features. As seen in Fig. 66 there more image features average outside the additive noise support for CIFAR, and for CelebA, the support of the image features are more closely overlapped with the noise support. Additional result on the CelebA-HQ (256 x 256) dataset is shown in Fig. 7
Motion deblurring Motion blur is a common image degradation in computational photography. We experiment with a spatially invariant horizontal motion blur kernel of five pixels for CIFAR and seven for CelebA. The additive correlated noise is Gaussian-filtered grayscale WGN with a circular kernel of std 2.5 pixels for CIFAR and pixels for CelebA, each with SNR = 0.493. The result
shown in Figs. 5 and
17

---

meas
conv
ours
gt
meas
conv
ours
gt
5.2
6,3
5.6
13.4
4.1
6.3
7.0
16.1
4.5
15.6
5.9
20.4
3.7
20.
4.6
Figure 7: Denoising correlated noise CelebA-HQ (256 256). We benchmark WS DM trained on anisotropic Gaussian noise with the conventional DM (conv) trained on isotropic Gaussian noise. Results for measurements, y with additive grayscale Gaussian noise of std 5 pixels.
meas
conv
ours
gt
meas
cony
ours
gt
10.35
14.67
11.38
CelebA
10.58
14.50
14.30
19
10.78
15:33
19.59
11.11
1469
11.21
19
90
motion deblur
lens deblur
Figure 8: Motion and lens deblurring CIFAR10 dataset with additive spatially correlated grayscale Gaussian noise of std 2.5 pixels. Our diffusion prior is able to consistently remove correlated noise resulting in superior PSNR compared to DMs trained solely on isotropic Gaussian noise.
18

---

meas
conY
ours
DDT
gt
8,65
9.25
rDD
9.45
14.37
#DT
10.84
12.49
rDD
11.05
real
fourier
real
fourier
real
fourier
Figure 9: Linear inverse scattering CIFAR
Lens deblurring Lens blur is the loss of high spatial frequency information result of light rays being focused imperfectly due to the finite aperture size, causing rays from point source to spread region in the image plane rather than converging modeled as a convolution between circular Gaussian kernel and the clean image. In Figs. 5 and 8 demonstrate our WS diffusion prior on lens deblurring with Gaussian blur kernel of STD 0.8 and 1.0 for CIFAR and CelebA, respectively. The additive correlated noise is cauussian-fiteeeee grayscale WGN with a circular kernel of std 2.5 pixels for CIFAR and pixels for CelebA, each with SNR = 0.810.
single point. This be effectively
Linear inverse scattering Inverse scattering is prevalent direction in optical imaging, to recover the permittivity field from measurements under angled illumination. Intensity diffraction tomography (IDT) is powerful computational microscopy technique that can recover 3D refractive index distribution given of 2D measurements. The model be linearized using the first Born approximation (322):
u(r) = (x(r) + (((r) -')derrrrr'''''''
r')dr',
(36)
for the field at the measurement plane u(r) and incident field u, ur) The scattering potential V(r) = I= Ae(r) with permittivity contrast Ac(r) = <(r) - between the sample <(r) and surrounding medium and wavenumber ko for illumination wavelength X. Green's 's function
G(r) exp(vkir]] where & = Veokn-
When the illumination is transmissive. referred as transmission intensity diffraction tomography (tIDT), meaning that the light passes through the sample, the linear operator, A, results in a mask in the shape of cross section of torus that attenuates Fourier coefficients as seen in Figs. and 10 leading to the well-known "missing cone" problem.
In reflection IDT (rIDT), placing the sample object specular mirror substrate causes light to reflect towards the camera, enabling the capture additional axial frequency components and partially filling the missing cone, shown in Figs. and 10
19

---

meas
conY
ours
DDT
gt
6,50
9:38
rDT
8.35
10.33
HDT
14
7.39
rDDT
8.84 real
11.53
real
fourier
fourier
real
fourier
Figure 10: Linear inverse scattering CelebA (64 x 64)
meas
tikhonov
conv
ourS
meas
Nkhonoy
conv
ourS
gt
15.1
13.32
15.68
2.27
10
transfer function
8282
14.98
94
10:
12.32 94
8-11
1186
9004
CIFAR10
CelebA
Figure 11: Laplace imaging, Transport of Intensity (TIE)
We experiment with both uDT and rIDT with the measurements corrupted by low-pass filtered grayscale WGN mimic background noise common in microscopy. The grayscale noise similarly caussian-filtered with std = 2.5 and 5, for CIFAR and CelebA, respectively with SNR 0.632.
Differential defocus Differential defocus is computational imaging technique that aims to recover the depth map from a series of defocused measurements ((2)) The linear operator can be realized with 2D Laplacian kernel, bandpassing mid-frequency components. In computational microscopy, this is also known as transport of intensity imaging (159)) to recover the phase and amplitude of an object.
demonstrate our framework on the differential defocus problem in Fig. II The noise again grayscale WGN filtered with Gaussian kernels of sid = 2.5 and 5, for CIFAR and CelebA, respectively with SNR 12.91. We also compare with Tikhonov regularization, which is L2 norm prior the object to constrain the energy of the reconstruction.
20

---

164.2
148.3
FID
88.3
40.0
18.5
isotropic
s.6
sdF2..
s.F-3.0
sdF44,0
5.0
XT
Figure 12: FID scores for different WS DMs trained on different sidmar-
D Generative modeling
We also train 6 different models where we vary the std the maximum value the Gaussian blur kernel, K, from a delta function (isotropic Gaussian noise) to stdmaa pixels. During training, K varies uniformly from std = 0.1 to std = stdmaa-
Novel samples are produced by solving the probability flow ODE, replacing G, Xx, log Pe (x)) with our optimized model (x, t) using Euler-Maruyama discretization with T 1000 and
8min 0.01 and Smaz 20. The initial noise condition, Xy (O, Kstd K std.a The
Fréchet Inception Distance (FID) scores decrease as the spatial correlation range increases as seen in Fig. 12
While WS DMs perform well as generative denoising priors for inverse problems, we leave to future work further investigation on their generative capabilities.
E
Forward Consistency Loss
The model (x f) is trained to approximate the scaled noise component introduced in the forward stochastic differential equation (SDE) that perturbs X, to yield x,. Accordingly, the score function G.G. xx, log p(x; Xo) in Eq. I2 can be substituted with the model prediction. To enforce consistency with the forward diffusion process, we introduce an auxiliary loss term defined as:
Bix + no)) (Xe, t)
L2 = (x
Xo
(37)
B,a
The term inside the expectation represents reconstruction of based on the noisy sample xt and the model prediction ng. Minimizing L2 encourages the model to remain faithful to the generative process defined by the forward SDE.
Empirically, we find that including L2 as an auxiliary jnjective-weiggtee equally with the primary loss term Leads to improved training stability and faster convergence.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly state the core innovation-leaming whitened score handle anisotropic Gaussian processes-and claims align well with the theoretical development and experimental validation presented in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss its suboptimal performance in generative tasks.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer: [Yes]
Justification: We provide proofs in the appendix that clearly state assumptions and step-by- step proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All details related to training and hyperparameter selection are mentioned in the Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
23

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
A link to the repository is provided and the datasets are public datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipsscc//pill wttpcccppppppppeeeeeeeeeeeeeeeee httae//ggiieee/eeeeubbissssssss /Cc////oddiiisssseeeeeee ssiiiisiicyiiyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: All training and test details are described in the Experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: Experiments conducted are image reconstruction, which cannot include error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The details of the compute resources are described in the Experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Yes, the research adheres the NeurlPS Code of Ethics, as it presents methodological contribution without involving human subjects, sensitive data, or foreseeable societal harms.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: Not applicable-thh paper presents theoretical and algorithmic contribution in diffusion modeling without direct deployment implications, and does not discuss societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
25

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not applicable, as the paper does not involve the release of models data with high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: CIFAR-I0 and CelebA datasets are cited.
Guidelines:
The answer NA means that the paper does
existing assets. The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets are introduced.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not use crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper doess not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: There are no human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs are not used
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy pettts://neurre http//rrppscccomofeeeeeeee/////p cc//oonferences/22255/LI. for what should or should be described.
28


Reviews:
[{"review_id": "9oEbZ9yUSd", "reviewer": "Reviewer_j1Re", "summary": "The authors propose an alternate loss target for diffusion model score matching: instead of score s, they model GG^\\top s where G is the diffusion matrix. This eliminates a $[G G^\\top]^{-1}$ that shows up in the original loss. And for inverse problems, this approach also removes the matrix inversion in front of the gradient of the likelihood.", "questions": "Intro \n\n- \"bridging the gap between DMs and FM\". Careful with the word \"the\" gap. Is this the only gap? Is it \"a\" gap?\n\nBackground\n\n- equation 1. assumes linearity of drift. maybe state that? \n- [line 98] \"So far, all implementations of score-based DMs use uncorrelated white Gaussian noise\". I'm not sure this is the whole story here. While there might be less work with noise correlated across data dimensions, there are other work that add auxiliary coordinates and have correlated noise in small blocks across those coordinates (CLD, PSLD, Multivariate Diffusion Models). See the works I listed in Weaknesses.\n\nTraining details\n\n- how did you choose the std's for drawing the elements of K ? They are different for the two datasets\n- How did you decide to do random color vs grayscale noise?\n\nAdditional question that is not about clarity, but more just exploritory: have you considered learning the choice of correlated noise? I believe there are some existing tools for this from VDMs (https://arxiv.org/abs/2107.00630) and Multivariate Diffusion Models (https://arxiv.org/abs/2302.07261).", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths \n\n- clear presentation of just the essentials of existing diffusion/flow methods \n- clear presentation of their proposed learning target, the whitened score \n- convincing experiments that are well motivated: image data has low and high frequency components yet we independently introduce additive noise to all features, causing some things too decay much faster than others. \n\nWeaknesses:\n\n- lacking a little discussion of other methods that change the noising processes. For example,\n  -  CLD (https://arxiv.org/abs/2112.07068)\n  - PSLD (https://arxiv.org/abs/2303.01748)\n  - Multivariate Diffusion Models (https://arxiv.org/abs/2302.07261)\n  - Blurring diffusion (https://arxiv.org/abs/2209.05557)\n  - flexible diffusion (https://arxiv.org/abs/2206.10365) \n  - etc \n- Somewhat lacking a little more discussion on how they are changing the loss target, and how alternative loss targets have also been studied in depth for diffusion: replacing score prediction (Song et al) with noise prediction (Ho et al) or v prediction (Salimans et al), and how flow matching is just another choice here (velocity, related but distinct from v prediction). Likewise the whitened score is another choice of loss target.", "was_revised": true, "final_justification": "My positive opinion of the work remains unchanged.", "rebuttal": "**W1&Q3**: We appreciate this important clarification and agree that our original statement was too strong. While most mainstream models (e.g., DDPM, SDE-based score models) assume independent Gaussian noise, recent works have explored correlated or structured noise, particularly in the context of auxiliary dimensions or learned coordinate transformations. \n\nRecent advances in diffusion modeling have introduced a range of structured forward processes and auxiliary-variable formulations to improve generative performance and flexibility. CLD and PSLD augment the state space with velocity variables and introduce noise in phase space, simplifying score estimation. Multivariate Diffusion Models generalize auxiliary-variable training across arbitrary linear inference processes. Blurring Diffusion incorporates frequency-aware noise to unify blurring and diffusion, enhancing synthesis. Flexible Diffusion parameterizes the forward SDE via geometric constraints, allowing adaptive noise shaping while preserving tractability. Together, these works extend diffusion models beyond standard isotropic Gaussian frameworks, enabling more expressive priors and principled inference across domains.\n\nWe will revise the text by adding an additional section to the Background to acknowledge these developments and cite the relevant references.\n\n\n\n**W2**: Thank you for this insightful comment. We agree that the choice of loss target plays a central role in diffusion modeling and appreciate the opportunity to clarify this connection. While different loss formulations have been proposed in the literature, many are mathematically equivalent or closely related. For instance, score prediction (as in Song et al.) and noise prediction (as in Ho et al.) are formally equivalent under the reparameterization of the forward process, differing only in the parametrization of the denoising network. Similarly, v-prediction (Salimans et al.) can be viewed as a linear combination of the data and noise vectors, effectively interpolating between data and noise prediction. Flow matching (as in Lipman et al.) generalizes this by directly modeling the velocity field that transports the data distribution toward the base distribution. In this context, our proposed whitened score formulation introduces yet another valid target that remains consistent with the diffusion framework while offering practical advantages under structured noise. We will revise the manuscript to make these relationships more explicit and situate our method within this broader landscape of loss design.\n\nAdditionally, we compared our method to the velocity-prediction loss target in flow matching and show that the vector field learned by flow matching naturally includes the whitened score function (Eq. 15). This connection explains why flow matching, like our approach, is capable of supporting arbitrary Gaussian diffusion paths. We view this as reinforcing the broader theme of designing loss targets that align with the structure of the generative field, and we will elaborate on these connections in the revision.\n\n**Q1**: We agree with the reviewer’s suggestion and will revise the phrasing to “a gap” to avoid implying exclusivity. Our intent was to highlight a specific challenge related to differing loss formulations, not to claim this is the only open question between diffusion and flow models.\n\n**Q2**: Thank you for the suggestion. We will explicitly state that Eq. (1) assumes a linear drift for clarity and consistency with the standard formulation of variance-preserving SDEs used in DMs.\n\n**Q4**: The standard deviations used to generate the noise correlation matrix $\\mathbf{KK}^{\\top}$ were selected empirically to produce structured noise resembling real-world degradations such as fog and scattering. To maintain perceptual consistency across datasets, we adjusted the parameters based on image resolution. Specifically, for CelebA (64×64), we doubled the standard deviation of the Gaussian kernel compared to CIFAR-10 (32×32) to account for the twofold increase in spatial scale. This ensures that the correlated noise exhibits a similar spatial extent relative to the image dimensions, preserving realism in the forward noising process. Additionally, we compared the statistics of our synthetically noised images with real foggy and hazy image datasets to validate the plausibility of the simulated degradations. We will include these comparisons in the appendix.\n\n**Q5**: Our initial focus was on the imaging task of dehazing and visibility restoration through fog, for which we trained our diffusion model using purely grayscale noise. However, we empirically observed that this led to unnatural deviations in the color channel distributions when applied to natural RGB images. To address this, we adopted a mixed noising strategy that combines grayscale and color (RGB) noise in a 1:1 ratio. This hybrid approach effectively restores image structure while preserving color fidelity, enabling robust denoising across both luminance and chromatic components.\n\n**Q6**: We thank the reviewer for this insightful suggestion. Our current models are trained on a wide range of diffusion matrices G to ensure generalizability to different measurement noise structures (see Section 4.1 for experimental details); however, we are actively exploring extensions where G is parameterized and learned jointly with the model. We are aware of relevant efforts in vector-valued diffusion models (e.g., VDM, Multivariate DMs) and will include a forward-looking discussion on this direction in the conclusion."}, {"review_id": "LVJ1vfcR2r", "reviewer": "Reviewer_492K", "summary": "Authors suggest learning the whitened score function of a diffusion process with non-isotropic Gaussian noise to avoid the instability issues associated with ill-conditioned matrix inversion.", "questions": "See above.", "limitations": "Authors don't have an explicit limitation section.", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n- The authors motivate the problem relatively well, and the writing is clear and concise.\n\n**Weaknesses:**\n\n* Doesn’t the null space of the whitening operator create issues when\n  the covariance of the noise is ill-conditioned? While I understand\n  that there is no **explicit** matrix inversion, the premise of\n  avoiding instability by not inverting the covariance matrix is\n  somewhat undercut since the authors are now implicitly inverting a\n  linear operator (whitening operator), which has a **squared\n  condition number**. This typically leads to **worse conditioning and\n  reduced numerical stability**, potentially slowing convergence\n  significantly.\n\n* The authors motivate the use of non-isotropic Gaussian noise by saying certain applications might benefit from it. Can you further elaborate on this? It is mentioned that using isotropic noise can render the prior learned insufficient, but it is not clear how this is the case. If you end up training a model with isotropic noise successfully, then you have a prior. What advantage does non-isotropic noise provide?\n\n* The authors should describe the trade-off between the computational complexity of the method and the performance gain achieved by using non-isotropic noise. It is not clear how much additional computational cost is incurred by the method, and whether this cost is justified by the performance gain.\n\n* Algorithm 1 is not really a tractable method to sample from the posterior distribution of inverse problems with computationally costly forward operators. I understand this sampling strategy is not the main contribution of the paper but plays a major role in motivating the problem. So this limits the applicability of the overall method. The authors should discuss this limitation and suggest alternative sampling strategies that could be used in practice.\n\n* Unless I missed it, I don’t see any discussion of the computational complexity of the method. Based on my arguments above regarding the condition number of the whitening operator, I suspect convergence will be slow for ill-conditioned noise covariance. Could the authors show training loss curves for different, increasingly ill-conditioned noise covariance matrices to demonstrate the convergence speed of the method?", "was_revised": false, "final_justification": "", "rebuttal": "**W1**: We appreciate the reviewer pointing out a potential confusion.\nTo clarify, Figure 3 in the manuscript illustrates that when the forward noising process uses a covariance matrix with a high condition number, the score estimated by conventional methods becomes unstable. In contrast, our proposed whitened score remains stable and isotropic under the same conditions. The following paragraphs provide a more detailed explanation of our approach.\n\nOur method does not apply a whitening transformation to the data, nor do we perform any matrix inversion—explicitly or implicitly—at any stage of the diffusion process or during training. Instead, we construct the forward diffusion process using non-isotropic Gaussian noise with covariance $ \\boldsymbol{\\Sigma} = \\mathbf{G} \\mathbf{G}^\\top $. The noise is added directly in this structured form, and our model is trained to estimate the whitened score, defined as $\\mathbf{GG}^\\top \\nabla_{\\mathbf{x}_t} \\log p(\\mathbf{x}_t)$, which represents the structured noise added in the forward noising diffusion process.  \nImportantly, this score is well-defined even when $ \\boldsymbol{\\Sigma} $ is ill-conditioned or low-rank, and does not require computing $ \\boldsymbol{\\Sigma}^{-1} $ or inverting $ \\mathbf{G} $.\n\nThis stands in contrast to traditional denoising score matching (DSM), which estimates $ \\nabla_{\\mathbf{x}_t} \\log p(\\mathbf{x}_t) $ and then multiplies it by $ \\boldsymbol{\\Sigma}^{-1} $, leading to significant amplification of signal components along low-variance directions—precisely the pathology we aim to avoid. Our approach instead keeps the learning target in the same metric as the structured noise used in the forward process, ensuring numerical stability and robust training dynamics even under extreme anisotropy or degeneracy in $ \\boldsymbol{\\Sigma} $.\n\nThus, we respectfully note that no operator with a squared condition number is being inverted or applied at training time. The learning process does not suffer from the ill-conditioning effects that plague DSM, which we demonstrate both analytically and empirically. We hope this clarifies the distinction and addresses the concern.\n\n**W2**: We thank the reviewer for the opportunity to clarify this important point. In fact, a central contribution of our work is to demonstrate that training diffusion models with anisotropic (non-isotropic) Gaussian noise leads to more effective priors for inverse problems with structured degradations. When measurements are corrupted by spectrally biased or non-Gaussian noise—such as blur or scattering—models trained with isotropic noise struggle to recover lost high-frequency content due to a mismatch between the learned prior and the measurement process. Our WSD framework addresses this by training on anisotropic noise, which induces a spectral bias aligned with the structure of real-world corruptions. This alignment enables more reliable recovery in challenging inverse problems, as we demonstrate empirically across multiple settings.\n\n**W3**: We thank the reviewer for this suggestion. Training the WS DM model on CIFAR10 takes approximately 20 hours for 1.7k iterations, which is comparable to standard diffusion models. For CelebA64×64, both WS DM and standard DMs require roughly 15.5 hours for 1k iterations. The reason is because our method does not introduce any additional computational cost compared to standard diffusion models. The forward process adds structured Gaussian noise with covariance $ \\boldsymbol{\\Sigma} = \\mathbf{G} \\mathbf{G}^\\top $, and the model learns to predict this structured noise component—referred to as the $\\mathbf{GG}^\\top \\nabla_{\\mathbf{x}_t} \\log p(\\mathbf{x}_t)$—at each time step $ t $. There is no matrix inversion, no additional per-step computation, and no increase in network complexity or sampling time.\n\nIn effect, the only change lies in the choice of noise structure during training. As shown in our experiments, this structured noise yields improved performance in inverse problems without incurring computational overhead. We will clarify this point in the revision.\n\n**W4**: We thank the reviewer for this thoughtful comment. In our case, the forward operators $ \\mathbf{A} $ are linear and circulant, allowing both $ \\mathbf{A} $ and its adjoint $ \\mathbf{A}^\\top $ to be implemented efficiently using fast Fourier transforms (FFTs), as detailed in Section 3. There are other recent works that use a precomputed singular value decomposition (SVD) of the A matrix to enable computational efficiency (see SNIPS and DDRM). As such, Algorithm 1 remains computationally tractable in our experimental settings. Nonetheless, we will include this consideration in the limitations section to acknowledge that the computational efficiency of Algorithm 1 relies on the structure of the forward model $ \\mathbf{A} $, and may not hold for general, non-circulant operators.\n\n\n**W5**: We thank the reviewer for raising this point. To clarify, ill-conditioning of the noise covariance does not affect training dynamics in our framework. As noted earlier, our method avoids explicit matrix inversion by directly learning the structured noise component via the $\\mathbf{GG}^\\top \\nabla_{\\mathbf{x}_t} \\log p(\\mathbf{x}_t)$ parameterization. As such, optimization proceeds similarly to conventional DMs, and we observe stable convergence across a range of structured noise covariances, including highly ill-conditioned ones.\n\nAt inference time, convergence of the reverse diffusion process is unaffected by the conditioning of the noise covariance. This is because sampling follows a fixed-length trajectory (e.g., 1000 steps) using predefined noise schedules, with no iterative solver or adaptive update that would otherwise depend on the condition number.\n\nFinally, we note that our framework is explicitly designed to train on a broad family of structured noise covariances—including increasingly ill-conditioned ones—within a single WSD model. This joint training strategy ensures robustness and generalizability across a range of degradations without requiring separate models or retraining. As such, reporting separate training loss curves for individual covariances would not meaningfully reflect the convergence behavior of the method. Instead, we demonstrate that a single model can accommodate diverse noise structures with stable training dynamics and consistent performance at test time.\n\n\n**Limitations**: Thank you for this suggestion. We agree that a dedicated limitations section would strengthen the manuscript by clarifying the scope and potential extensions of our method. In the revision, we will include such a section that incorporates insights from this and other reviewer comments, highlighting practical considerations and avenues for future work."}, {"review_id": "om0rh1fGAL", "reviewer": "Reviewer_3afH", "summary": "The authors propose Whitened Score Diffusion, a framework for learning the whitened score function which enables training on arbitrary Gaussian forward noising process without the need for noise covariance inversion. The approach also establishes theoretical connections to flow matching for general Gaussian noise and proves particularly effective as a Bayesian prior in inverse imaging problems with correlated noise. The authors present results on tasks involving CIFAR and CelebA.", "questions": "How does the model perform on inverse problems with isotropic noise? Does its performance degrade compared to methods specifically designed for such settings?\n\nIs the approach scalable to larger datasets or higher-resolution settings? This seems challenging given the large number of possible K variants the model would need to account for.\n\nHow does this framework extend to latent space generative models? Can the same principles be applied effectively in that context?", "limitations": "Unfortunately, the manuscript does not explicitly discuss any limitations. However, several become apparent based on the concerns outlined above. These include limited experimental scale, potential performance degradation with broader noise structures, lack of evaluation on standard uncorrelated inverse problems, and unclear generalization to more complex or higher-dimensional settings.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe paper is well-motivated and clearly written, providing a solid theoretical foundation for the proposed method. The authors articulate the motivation behind learning a whitened score and how this circumvents challenges with inverting structured noise covariances in traditional diffusion frameworks.\n\nWeaknesses:\nHowever, the work has several notable weaknesses. The experimental validation is limited to very small-scale datasets, which raises concerns about the method’s scalability and robustness. Moreover, the results show that the model’s FID score worsens as the kernel’s standard deviation increases. This suggests that the learned prior performs poorly with broader noise structures, resulting in lower reconstruction quality.\n\nIn addition, the paper does not evaluate the method on standard uncorrelated inverse problems. Without comparisons to conventional priors in these settings, it remains unclear whether the proposed approach offers a meaningful advantage. Demonstrating similar performance in such standard cases would make the claims more convincing.\n\nFinally, the absence of qualitative comparisons makes it difficult to assess the visual fidelity of the generated samples or to evaluate any perceptual improvements the method may provide.", "was_revised": true, "final_justification": "The rebuttal addresses my concerns in a satisfactory manner.", "rebuttal": "**W1**: We thank the reviewer for this thoughtful feedback. To address the concern about scalability, we conducted additional experiments and trained a new WS DM model at 256×256 resolution on the CelebA dataset. Due to time constraints, we focused on the dehazing task, where we found that WS DM priors achieved consistently higher PSNR compared to conventional DMs across multiple examples:\n\n| Example | WS DM PSNR (dB) | Conv DM PSNR (dB) |\n|---------|------------------|--------------------|\n| 1       | 15.9             | 5.2                |\n| 2       | 13.4             | 5.6                |\n| 3       | 17.1             | 6.3                |\n| 4       | 15.6             | 4.5                |\n| 5       | 20.7             | 3.7                |\n| 6       | 17.7             | 6.3                |\n| 7       | 17.2             | 4.1                |\n| 8       | 16.1             | 7.0                |\n| 9       | 20.4             | 5.9                |\n| 10      | 21.3             | 4.6                |\n| 11      | 20.3             | 4.2                |\n| **Avg** | **17.79**        | **5.22**           |\n\nThese results, which will be included in the revised manuscript, demonstrate that our method scales effectively to higher-resolution settings and maintains stable performance under structured noise.\n\nRegarding the observation on FID degradation with increasing kernel width: we agree that FID is a useful but limited proxy for generative quality. However, a worse FID score does not necessarily imply degraded performance on inverse problems, which often benefit from stronger structural priors even when sample realism (as captured by FID) decreases. In fact, our inverse problem results—especially for highly ill-posed scenarios—demonstrate that the learned prior remains effective despite broader noise structures. We will clarify this distinction in the revision and include both the new experiments and a more nuanced discussion of evaluation metrics.\n\n\n**W2 & Q1**: We thank the reviewer for this observation, and we would like to clarify that our method is, in fact, evaluated on a standard uncorrelated inverse problem—namely, denoising under white Gaussian noise. This experiment is shown in Figure 4, first row, where we compare our approach against a conventional diffusion model trained with isotropic Gaussian noise. As shown, our method achieves comparable performance in this baseline setting, demonstrating that it does not degrade under unstructured noise.\nWe will revise the manuscript to make this comparison more explicit and easier to locate. We appreciate the reviewer’s suggestion and agree that establishing parity with conventional models in standard settings is an important baseline to report.\n\n**W3**: We appreciate this suggestion and agree that qualitative examples are important for assessing perceptual quality. In the revised manuscript, we will include unconditional generated samples in the appendix to illustrate the visual fidelity of our method across different noise structures.\n\n**Q2**: We appreciate the reviewer’s observation and agree that scaling to higher resolutions naturally introduces a broader range of possible blur kernels KKK that the model must learn to handle. Our method addresses this by training over a continuous range of kernel standard deviations, which increases with resolution: from 0.1–3.0 at 32×32, to 0.1–5.0 at 64×64, and 0.1–10.0 at 256×256. This structured noise is baked into the forward process, and the model learns to denoise samples corrupted by blur-induced covariances drawn from this range.\n\nWhile this formulation scales well in practice, we acknowledge that covering a larger space of degradations makes the learning task more challenging. We will highlight this trade-off as a limitation in the revised manuscript and discuss potential extensions such as conditioning on kernel parameters or leveraging amortized inference.\n\n**Q3**: We appreciate the reviewer’s insightful question. In latent space generative models, clean images are first encoded into a latent representation, where the diffusion process is then applied, and the final samples are decoded back into image space. Extending our framework to this setting would involve introducing correlated (non-isotropic) noise within the latent space rather than the pixel space.\n\nThis is a natural and promising extension, though the effect of structured noising in latent space remains an open question. Its impact likely depends on how the autoencoder transforms and encodes noise structure, which may distort or align with the underlying image-space priors in complex ways. Understanding how structured diffusion in latent space influences generative quality or inverse problem performance is an interesting direction for future work, and we will include this point in the revised manuscript.\n\n**Limitations**: We thank the reviewer for highlighting this. While we do discuss the impact of broader noise structures on generative performance in the appendix, we agree that an explicit limitations section would help frame the scope of our contributions more clearly. In the revised manuscript, we will add a dedicated limitations section that synthesizes relevant concerns—including those raised here and by other reviewers—and outline directions for future work, such as improved generalization across degradation models and broader-scale evaluation."}, {"review_id": "SXNjuHRJbl", "reviewer": "Reviewer_nvZa", "summary": "The paper introduces Whitened Score Diffusion, an extension to score-based diffusion models that enables stable training over arbitrary (including anisotropic) Gaussian noising processes. The key innovation is to learn the product $\\mathbf{G}_t\\mathbf{G}t^{\\top}\\nabla{\\mathbf{x}_t}\\log p_t(\\mathbf{x}_t)$, rather than just the score, which avoids problematic covariance inversions when handling structured, non-isotropic noise. The method is motivated as a structured Bayesian prior for imaging inverse problems involving correlated noise. Experiments on CIFAR10 and CelebA demonstrate that the proposed method outperforms conventional DMs—especially under correlated/structured noise—across several computational imaging tasks.", "questions": "1. How sensitive are the improvements to the exact choice of spatial covariance? Could the authors provide results or analysis for highly non-circulant or non-Gaussian structures in $\\mathbf{K}$?\n2. Could more challenging or realistic noise models be evaluated? For example, astronomy, or non-additive processes common in optics/microscopy.\n3. How robust is sampling to mis-specification of $\\mathbf{G}_t$ during inference? Is there a drop in performance if the test noise structure mismatches the training structure, and can WS DMs be tuned or regularized for increased generalization?\n4. Could the method be extended to non-linear/inverse problems outside the linear Gaussian setting? What would be required from an algorithmic or mathematical perspective?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- This work addresses a significant limitation in current score-based diffusion models by making training feasible and stable for arbitrarily structured Gaussian noise processes. This is well-motivated for imaging scenarios where noise is rarely isotropic, as detailed in the introduction and Section 2.3.\n- The visualizations in Figure 3 are straightforward, which effectively illustrate how the WS transformation preconditions the score field to be magnitude-stable and isotropic, even for highly anisotropic covariances.\n- Empirical results consistently show that WS DMs provide higher PSNR and improved perceptual quality over conventional DMs in various inverse imaging tasks with structured noise. \n- The theoretical analysis reveals a clear connection drawn between the new \"whitened score\" objective and flow matching. The relationship both justifies the method and indicates its generality.\n\n\n**Weaknesses**\n- The empirical validation scope is somewhat narrow: Only CIFAR10 (32×32) and CelebA (64×64) are used in experiments. While these are standard, more diverse datasets or higher-resolution tasks (e.g., FFHQ 256×256 or real-world scientific/medical applications) would strengthen the generalizability claim. This is particularly relevant as the method is framed as benefiting scientific imaging.\n- Baseline selection should be expanded. Only a conventional DM trained with isotropic noise is used as a baseline. Previous work[17] on handling structured/correlated noise is referenced, but not directly compared against. There should also be flow matching-based baselines or other Bayesian generative priors.\n- There is no explicit ablation on the impact of the whitening transformation versus simply adjusting training objective scaling, nor an exploration of how the choice of covariance $\\mathbf{K}$ (e.g., various correlations or non-circulant structures) affects results.\n- Realistic noise modeling is missing. The extent to which this matches real scientific imaging noise statistics (e.g., turbulence, CCD noise) is not quantitatively validated.\n\nOverall, I like the idea of considering arbitrarily structured Gaussian noise processes for real-world DM-based inverse solvers, but the empirical results are insufficient in the current version. I would like to increase the score if more results are provided.", "was_revised": true, "final_justification": "This is a well-motivated and theoretically sound paper proposing a method to train diffusion models with arbitrary Gaussian noise for inverse problems. While I appreciate the new 256x256 results, the paper would be more convincing with additional results on downstream imaging tasks. Overall, I lean positive to this paper.", "rebuttal": "**W1**: We thank the reviewer for this insightful comment. We agree that broader empirical validation is crucial, especially given the method’s potential for scientific and real-world imaging. To address concerns about generalizability and scalability, we conducted new experiments training a WS DM on CelebA at 256×256 resolution—substantially higher than the original 64×64—while maintaining the same architecture and training setup.\n\nDue to time constraints, we focused on the dehazing task, where structured degradations are particularly relevant. As shown below, WS DM outperforms the conventional DM baseline by a wide margin in PSNR:\n\n| Example | WS DM PSNR (dB) | Conv DM PSNR (dB) |\n|---------|------------------|--------------------|\n| 1       | 15.9             | 5.2                |\n| 2       | 13.4             | 5.6                |\n| 3       | 17.1             | 6.3                |\n| 4       | 15.6             | 4.5                |\n| 5       | 20.7             | 3.7                |\n| 6       | 17.7             | 6.3                |\n| 7       | 17.2             | 4.1                |\n| 8       | 16.1             | 7.0                |\n| 9       | 20.4             | 5.9                |\n| 10      | 21.3             | 4.6                |\n| 11      | 20.3             | 4.2                |\n| **Avg** | **17.79**        | **5.22**           |\n\nThese results, to be included in the revision, demonstrate that our approach scales effectively to higher resolutions and remains robust to structured noise. While we focused on CelebA256 due to compute limits, extending to scientific and medical datasets is a natural next step.\n\n\n**W2**: We thank the reviewer for this suggestion. While broader comparisons are valuable, prior work [17] addressing structured noise does not release code or pretrained models, complicating direct evaluation. Flow-matching and Bayesian priors follow orthogonal paradigms beyond this paper’s scope, which focuses on extending score-based diffusion to arbitrary Gaussian covariances. We believe comparisons with a standard isotropic diffusion model offer a fair and interpretable baseline for isolating the impact of structured noise.\n\n\n**W3**: We thank the reviewer for this thoughtful comment. The term “training objective scaling” is unclear in this context. If it refers to heuristic loss reweighting or adjusting the score magnitude, we note that such approaches are not equivalent to our method. The whitening transformation we employ is a principled reparameterization that enables stable and well-conditioned training of score-based diffusion models under arbitrary Gaussian noise structures. It acts as a preconditioning mechanism that standardizes the training dynamics and ensures that the learned score model is properly adapted to non-isotropic and potentially correlated noise.\n\nMoreover, we emphasize that the family of diffusion covariances used during training is carefully designed to include the noise covariance present in the target inverse problem. This ensures that the learned prior is not only generalizable but also task-relevant, particularly in scientific imaging applications where noise is often structured and nontrivial.\n\nWe agree that a systematic exploration of different covariance structures—especially non-circulant and spatially correlated forms—is an important and valid research direction. While our current work does not fully explore this space, it provides a foundation for doing so in future studies.\n\n\n**W4**: We appreciate the emphasis on realistic noise modeling. To validate our synthetic structured noise, we compared it against real-world hazy images from the **Dense-Haze CVPR 2019 dataset**. By adding synthetic noise to clean images and analyzing Fourier spectra and intensity histograms, we show qualitative and statistical similarity. Results will be included in the appendix.\n\nThat said, modeling temporal fluctuations, sensor noise, or turbulence is important for future work. Our goal here is to establish a generalizable Gaussian-based framework that can flexibly incorporate such complexities moving forward.\n\n**Q1**: We thank the reviewer for this thoughtful question. Our framework supports arbitrary Gaussian noise structures through the whitening transformation, and in principle allows training under any symmetric positive definite covariance $ \\mathbf{KK}^{\\top} $, including non-circulant forms. However, we observe that generative modeling performance tends to degrade as $ \\mathbf{K} $ becomes broader—that is, as its spectrum suppresses higher spatial frequencies—since more fine-scale information is attenuated during the forward process, making recovery inherently more difficult. This aligns with the intuition that broader noise corrupts more of the signal bandwidth, leading to a harder learning and sampling problem.\n\nMoreover, performance is sensitive to the alignment between the training diffusion covariances and the noise present in the measurement process. If the assumed noise model in the forward process is significantly mismatched to the actual corruption in an inverse problem, the learned prior may fail to effectively denoise or reconstruct. To mitigate this, we explicitly select the diffusion covariance schedule to include the noise statistics observed in the target imaging task, ensuring compatibility between training and deployment.\n\nWe agree that further exploration of highly non-circulant or non-Gaussian structures would be valuable, and view this as a promising direction for extending the generality of our approach.\n\n**Q2**: We appreciate the suggestion. Our current formulation handles _additive Gaussian noise_, which is common in scientific imaging. However, many real systems exhibit non-additive corruptions (e.g., Poisson or multiplicative noise). Extending our framework to such settings would require rethinking both forward diffusion and training objectives. Prior work on generalized score matching may offer a path forward, and we will clarify this scope and limitation in revision.\n\n\n**Q3**: We thank the reviewer for raising this important point. We evaluated robustness under mismatch between training and test-time noise. Models trained with identity covariance performed poorly on structured noise. In contrast, WS models—trained with a range of structured covariances—generalized much better. This reflects the benefit of learning in a preconditioned space aligned with the noise geometry. While explicit regularization is an interesting avenue, our results show that diversity in training covariances alone confers strong robustness.\n\n\n**Q4**: We thank the reviewer for this excellent question. Our WS diffusion model is trained purely as a prior over clean data under Gaussian corruptions, agnostic to downstream inverse problems. For such extensions, Diffusion Posterior Sampling (DPS) allows adaptation to arbitrary forward models, including non-linear and non-Gaussian cases, by combining the learned score with a task-specific likelihood. This modularity is a strength of our framework and supports broad applicability in scientific imaging.\n\n\n**Limitations**: In response to this and related comments, we will include a brief limitations section in the revised manuscript to clarify the scope of our current work and outline promising future directions."}]

Author rebuttal:

Meta-review: Overall, reviewers find the paper theoretically solid, clearly written, and addressing an important limitation of current diffusion models. The main contribution—the whitened score formulation—is novel and well-justified, and the rebuttal successfully clarified misconceptions about stability and computational cost. However, the empirical validation remains narrow and could be strengthened by additional baselines, higher-resolution experiments, and more realistic noise modeling.

The AC recommends acceptance, encouraging the authors to broaden experiments, compare with related methods, and discuss limitations in the final version.
Decision: Accept (poster)