Removing Structured Noise using Diffusion Models
Authors: Tristan Stevens
Hans van Gorp
Faik C Meral
Junseob Shin
Jason Yu
Jean-luc Robert
Ruud Van Sloun
Venue: Accepted by TMLR ()

Abstract: Solving ill-posed inverse problems requires careful formulation of prior beliefs over the signals of interest and an accurate description of their manifestation into noisy measurements. Handcrafted signal priors based on e.g. sparsity are increasingly replaced by data-driven deep generative models, and several groups have recently shown that state-of-the-art score-based diffusion models yield particularly strong performance and flexibility. In this paper, we show that the powerful paradigm of posterior sampling with diffusion models can be extended to include rich, structured, noise models. To that end, we propose a joint conditional reverse diffusion process with learned scores for the noise and signal-generating distribution. We demonstrate strong performance gains across various inverse problems with structured noise, outperforming competitive baselines using normalizing flows, adversarial networks and various posterior sampling methods for diffusion models. This opens up new opportunities and relevant practical applications of diffusion modeling for inverse problems in the context of non-Gaussian measurement models.

Published in Transactions on Machine Learning Research (03/2025)
Removing Structured Noise using Diffusion Models
Tristan S.W. Stevens'
t.s. W...eeensstuue
Hans van Gorp'
h.v.gorp@lue.nl
Faik C. Meral
con.memläphhlips.com com
Jun Seob Shin
junseob.shin @philiisssppppp...ooocmm
Jason Yu?
jason. yu @phillpps.cmm
Jean-Luc Robert12
jean-luc. robert Sphilips.oops com
Ruud J.G. van Sloun
r.j.g. v. www.nnuuunun.....m
Department of Electrical Engineering, Eindhoovnn University of Technology, The Netherlands Philips Research North America, Cambridge MA, USA
Reviewed on OpenReview; https: // oppnrevvee.. net/ forum? id- BoKYsaDVEn
Abstract
Solving ill-posed inverse problems requires careful formulation of prior beliefs over the sig- nals of interest and an accurate description of their manifestation into noisy measurements. Handcrafted signal priors based e.g. sparsity increasingly replaced by data-driven deep generative models, and several groups have recently shown that state-of-the-art score- based diffusion models yield particularly strong performance and flexibility. In this paper, we show that the powerful paradigm of posterior sampling with diffusion models can be ex- tended to include rich, structured, noise models. To that end, we propose a joint conditional reverse diffusion process with learned scores for the noise and dignal-generatinn distribution. We demonstrate strong performance gains across various inverse problems with structured noise, outperforming competitive baselines using normalizing flows, adversarial networks and various posterior sampling methods for diffusion models. This opens up opportu- nities and relevant practical applications of diffusion modeling for inverse problems in the context of non- Gans-iaasia measurement models
1 Introduction
Many signal and image processing problems, such as denoising, compressed sensing, or phase retrieval, can be formulated as inverse problems that aim to recover unknown signals from (noisy) observations. These ill- posed problems are, by definition, subject to many solutions under the given measurement model. Therefore, prior knowledge is required for meaningful and physically plausible recovery of the original signal. Bayesian inference through posterior sampling incorporates both signal priors and observation likelihood models. Choosing an appropriate statistical prior is not trivial and dependent on both the application as well as the recovery task.
!Code: httpp: http:////tttb.cooo/sssaarrraaast

---

Published in Transactions on Machine Learning Research (03/2025)
In these image recovery tasks, the choice of noise prior is often assumed to be Gaussian Poisson due to their mathematical tractability and ease of modeling. Corruptions in many applications, however, are often highly structured and spatially correlated. Therefore, besides accurate knowledge of the signal distribution, it is crucial to model the noise effectively. While it is often challenging to derive analytical modcls for these structured noise distributions, samples can be practically obtained through simulation or by isolating noise in the absence of the signal of interest. Relevant examples of structured noise include speckle, haze or interference. In medical imaging, for instance, ultrasound images are often corrupted by speckle noise, which limits contrast and complicates diagnoses (Yang et al. 2016). In computer vision, haze, fog and rain are highly correlated across neighboring pixels and can significantly degrade the quality of images. (Berman et al. 2016; Ren et al. 2019). Another example is the presence of interference in radar, which can lead to severe artifacts in the reconstructed mange-Doppler maps (Uysal 2018).
A popular approach for solving such problems involves Bayesian inference and inverse modeling, which requires the design of suitable priors. Before the advent of deep learning. sparsity in some transformed domain has been the go-to prior, such as iterative thresholding (Beck & Teboulle 2009) or wavelet. decompasition (Mallat 1999 At present, deep generative modeling has established itself as strong mechanism for learning such priors for inverse problem-solving. Both generative adversarial networks (GANs) (Bora et al. 2017)
and normalizing flows (NFs) (Asim et al. 2020 Wei et al. 2022) have been applied as natural signal priors for inverse problems in image recovery. These data-driven methods are more powerful compared to classical methods, as they can accurately learn the natural signal manifold and do not rely on assumptions such as signal sparsity or hand-crafted basis functions.
Recently, diffusion models have shown impressive results for both conditional and unconditional image gen- eration and can be easily fitted to target data distribution using score matching (Song et al. 2020). These deep generative models learn the score of the data manifold and produce samples by reverting a diffusion process, guiding noise samples toward the target distribution. Diffusion models have achieved state-of- the-arr performance in many downstream tasks and applications, ranging from statc-of-the-art text-to-image models such as Stable Diffusion (Rombach el al. 2022) to medical imaging (Song el al.
2021b
Jalal el al. 2021a
Chung& & Ye 2022). Furthermore, understanding of diffusion models is rapidly improving and progress the
field is extremely fast-paced (Chung et al. 2022b Bansal et al. 2022: Daras et al.
2022a Karras et al. 2022
Luo
2022). The iterative nature of the sampling procedure used by diffusion models renders inference slow compared to GANs and VAFs. However, many recent efforts have shown ways to significantly improve the sampling speed by accelerating the diffusson process, from improving the sampling process itself (Salimans
& Ho 2021 Daras et al.) 2022b Chung et al. 2022c; Stevens et al. 2025 Park ct al.] 2024), to exccuting
the diflusion in some reduced (latent) space (Jing el al.. 2022: Vahdal el al. 2021 Rombach et al. 2022).
Despite this promise, current score-based diffuusion methods for inverse problems are limited to measurement models with unstructured noise. In many image processing tasks, corruptions however highly structured and spatially correlated. Nevertheless, current conditional diffusion models naively assume that the noise follows some basic tractable distribution (e.g. Gaussian or Poisson). Diffusion Posterior Sampling (DPS) (Chung et al. 2022a), Diffusion Model Based Posterior Sampling (DMPS) (Meng & Kabashima 2022). and Pseudoinverse-guiiee Diffusion Models (IIGDM) (Song et al. 2023). all have a different take on posterior sampling with diffusion models. Namely, they seek to approximate the intractable noise- noise-perture perturbed likelihood score, usually involving Tweedic's formula, in various ways. RED-diff sidesteps the challenge of posterior score approximation using variational inference (Mardani et al.
2023). resulting in simple gradient update rule that resembles regulaariatiooo----dddooiinnss Denoising Diffusion Restoration Models (DDRM) take another approach altogether by performing the diffusion trajectory in the spectral space, tying the measurement noise to the diffusion noise (Kawar et al. 2022). Albeit still under the Gaussian assumption. Denoising Diffusion 2022) opt for a different decompasition by projecting samples to the null-space of the forward operator of noiseless and noisy (Gaussian) inverse problems. Finally, Deep Equilibrium Diffusion Restoration (DeqIR) rethinks the sampling process by modeling it as fixed point system, achieving faster parallel sampling (Cao et. al. 2024). To summarize, all these methods improve upon incorporating measurements into the diffusion process. Nonetheless, they limit their scope to classic inverse problems such as denoising (Gaussian), inpainting, super-resolution, deblurring, etc., and do not address problems with structured noise.
Null-Space Models (DDNM) (Wang et al.
Luo et al. (2023) propose a general- image restoration framework
2

---

Published in Transactions on Machine Learning Research (03/2025)
y
Sp (x,, 1)
Sp (n..t)
Xt
nr
ply x.
Mx
M
X
nlo
R d
6
IR
Figure 1: Overview of the proposed joint posterior sampling method for removing structured noise using diffusion models. During the sampling process, the solutions for both signal and noise move toward their respective data manifold M through score models s9 and sg. At the same time, the data consistency term derived from the joint likelihood plyla,, n,) ensures solutions that are in line with the (structured) noisy measurement Az + n.
for arbitrary degradations. Unfortunately, this requires clean-noisy sample pains for training, leading to models that are task-specific, need retraining, and are more vulnerable to out-of-distribution data.
Beyond the realm of diffusion models, Whang et al. 2021] extended normalizing flow (NF))based inference structured noise applications. However, compared to diffusion models, NFs require specialized network architectures, which are computationally and memory expensive.
Given the promising outlook of diffusion models, propose to learn models for both the noise and the desired signal and perform joint inference of both quantities, coupled via the observation model. The resulting sampling scheme enables solving wide variety of inverse problems with structured noise.
The main ribtiiooiooss of this work are as follows:
We propose a novel joint conditional approximate posterior sampling method to efficiently remove structured noise using diffusion models. Our formulation is compatible with many existing iterative sampling methods for score-based generative models.
We show strong performance gains across various challenging inverse problems involving structured noise compared competitive state-of-tles ofeattt methods based on NFs, GANs, and diffusion models.
We provide derivations for and comparison of three recent posterior sampling frameworks for diffusion models (IGDM, DPS, projection) as the backbone for our joint inference scheme.
We demonstrate improved robustness on a range of out-of-distributio signals and noise compared to baselines.
2 Problem Statement
Many image reconstruction tasks be formulated as an inverse problem with the basic form y Aæ + n. where E Rm is the noisy observation, E Rd the desired signal or image, and E Rm the additive noise. The linear forward operator A E Rmxd captures the deterministic transformation of æ. Maximum posteriori (MAP) inference is typically used to find an optimal solution âMAp that maximizes posterior density yxly (x)y()ly):
MAP = arg max log (aly)= (ax) max [logPyix [o)px(x))],
(1)
where (2)()) is the likelihood according to the measurement model and logxx(#) the signal prior. Assumptions on the stochastic corruption process n are of key importance too, in particular for applications for which this process is highly structured. However, most methods assume i.i.d. Gaussian distributed noise, such that the forward model becomes (y)x) N(Ax, I). This naturally leads the following simplified problem:
3

---

Published in Transactions on Machine Learning Research (03/2025)
1
MAP arg min 1/ Ax logx 20 2
(2)
However, this naive assumption can be very restrictive as many noise processes are much more structured and complex. A myriad of problems can be addressed under the assumed measurement model, given the freedom of choice for the noise source n. Therefore, in this work, our aim is to solve more broad class of inverse problems defined by any arbitrary noise distribution n ~ pN(n) + A and signal prior x ~ px(x), resulting in the following, more general, MAP estimator:
MAP = arg max log PN Ax) + logPx(x))
(3)
In this paper, we propose to solve this class of problems using flexible diffusion models. Moreover, diffusion models naturally enable posterior sampling, i.e. x ~ Pxly (æly), allowing us to take ad advantage of the benefits thereof (Jalal et al. 2021b Kawar et al. 2021 Daras et al. 2022a) with respect to the MAP estimator which simply collapses the posterior distribution into single point estimate.
2.1 Background
Score-based diffusion models have been introduced independently as score-based models (Song & Ermon 2019] 2020) and denoising diffusion probabilistic modeling (DDPM) (Ho et al.) (020).. In this work, we will consider the formulation introduced by [Song et al. (020)) which unifics both perspectives on diffusion models by expressing diffusson as a contimuuus-lime process through stochastic differential equations (SDE). Diffusion models produce samples by reversing a corruption (noising) process. In essence, these models are trained to denoise their inputs for each timestep in the corruption process. Through iteration of this reverse process, samples can be drawn from learned data distribution, starting from random noise.
The diffusion process of the data {Ia E Rd)
[e is characterized by a continuous sequence of Gaussian perturbations of increasing magnitude indexed by time 1 LE [0. 1]. Starting from the data. distribution at O, clean images are defined by 2o ~ p(az)) = p(x). Forward diffusson can be describod using an SDE as follows: dæ, = (() + g()ddw, where
Rd is a standard Wiener process, f(() : [0, 1] - R and g(t): : [0, + R are the drift and diffusion coefficients, respectively. Moreover, these coefficients so that the resulting distribution p(æ1) at the end of the perturbation process approximates predefined base distribution p(æ,) se ((x)) Furthermore, the transition kernel of the diffusion process can be defined in one step as q( an) ~ ~N(x (arlaa (0, III), where a₂ and 3r can be analytically derived from the SDE.
chosen
Naturally, we are interested in reversing the diffusion process, so that we can sample from mg ~ p( The reverse diffusion process is also diffusion process given by the time SDE (Anderson 1982 Song
et al. 2020
dæ; = (f(())=; g(t)² Va, logp(a;) dt + g(t)
score
where w. is the standard Wiener process in the reverse direction. The gradient of the log-likeld of the data with respect to itself,, a.k.a. the score function, arises from the reverse-time SDE. The score function is a gradient field pointing back to the data manifold and can intuitively be used to guide random sample from (xD) >
the base distribution (2x) to the desired data distribution. Given a dataset. X' = x , .0 ..... 20
(1)
(2)
p(æo), scores can be estimated by training a neural network 8p (æx, t) parameterized by weights 0, with score matching techniques such as the denoising score matching (DSM) objective (Vincent 2011
8" = arg min Em../0 (m [1150 Va. log (ogg((z))o))) [x))
(5)
Given a sufficiently large dataset X' and model capacity, DSM ensures that the score network converges to sy(2,, t) ~ Va, log p(x:). After training the time-dependent score model Sg, it can be used calculate the reverso-time diffusion process and solve the trajectory using numerical samplers such the Euler- Maruyyaa algorithm. Alternatively, more sophisticated samplers, such ALD (Song & Ermon 2019), probability flow
4

---

Published in Transactions on Machine Learning Research (03/2025)
ODE (Song et al. 2020). and Predictor- Correccor sampler (Song et al. 2020 can be used to further improve sample quality.
These iterative sampling algorithms discretize the continuous time SDE into a sequence of time steps {0- = to. f1...., tr = 1}, where a noisy sample ét, is denoised to produce a sample for the next time step
_ The resulting samples ( constitute an approximation of the actual diffusion process (0t))
3
Method
In this section, we outline our approach to solving inverse problems under structured noise. Section 3.1 introduces our joint posterior sampling framework, leveraging joint diffusion processes for signal and noise. Section 3.2 discusses data consistency rules, detailing different methods to ensure alignment with observa- tions. Additionally, we demonstrate compatibility of our method with common posterior sampling strategies.
3.1 Joint Posterior Sampling under Structured Noise
We are interested in posterior sampling under structured noise. We recast this a joint optimization problem with respect to the signal and noise n given by:
(x, n) ~ px,N (z, n/y) x (y)x. n) px (x)) (N)),,
(6)
where we assume the signal and noise components be independent. Solving inverse problems using diffusion models requires conditioning of the diffusion process on the observation y, such that we can sample from the posterior Pxjy (z, n y)) Therefore, construct joint conditional diffusion process {ar, nely)eejo.11 in turn producing a joint conditional reverse-time SDE:
d(æ,, n2) = {f(()) n.) VVtt)Va,.,, m, logp(æ,, n nely) + g())dw.
We would like to factorize the posterior using our learned unconditional score model and tractable mea- surement model, given the joint formulation. Consequently, we construct two separate diffusion processes, defined by separate score models but entangled through the mcasurement model Py py n.. In addition to the original score model sa f), we introduce second score model (a(n),,) = Vn, (ogp(((n)) parame- terized by weights é, to model the expressive noise component These two score networks can be trained independently on datasets for æ and n, respectively, using the objective in equation 5 This is a significant differentiator, as our method climinates the need to collect samples of signals and noise together with corre- sponding ground truth. Self-supervised generalive modeling on isolated signals and noise measurements is sufficient, thus relaxing the difficulty of curating signal and noise datasets. The gradients of the posterior with respect to æ and n, used in equation 7 are now given by:
V m, log,(x nely)
50 + X logp(y a n.) (n..) <V n, log pyyppyy,,.. p., me) ]
V Vænn, log p(x,, nely) =
(8)
nn, logp(z;; n1ly))
which simply factorizes the joint posterior into prior and likelihood terms using Bayes' rule from equation for both diffusion processes. Following the literature classifier-(fre) diffusion guidance (Dhariwal & Nichol 2022) and diffusion for inverse problems (Song et al. 2020 Chung et al. 2022a Song et al. [202)], two Bayesian weighting terms, À and k, are introduced. These terms are tunable hyper- parameters that weigh the importance of following the prior, *3 1) and 8 (n,, !), versus the measurement
2021
Ho & Salimans
model, Vn, logp(y/a;, nt). A conceptual overview of the proposed method is shown in Fig.
3.2 Data Consistency Rules
The resulting Irue noise-perturbed likelihood p(y|æ,, n,) is generally intractable, unlike p(y pp, no). Different approximations have been proposed in recent works (Song et al. 2020 Chung et al.) 2022a Song et al. 2023 Meng & Kabashima 2022 Fong et al.) 22023] Finzi et al. (023). Our method agnostic to the type of data-consistency rule employed. To study its effect on the final output, we will implement three strong approaches proposed in literature, namely, Pseudbinverse-Cuidee Diffusion Models (IGDM) (Song et al.
5

---

Published in Transactions on Machine Learning Research (03/2025)
Table 1: Parameter choices for the Gaussian model of the noise-perturben likelihood function in equation
IIGDM Song al.
2023
DPS Chung et
2022a
Projection
Song
2020
Y
y Awojj rooje (r?AAT + of
Arrott Toj
Aas nt
p21
p²
A
Xp²// (a(t) 12
x /g(t)² < <9p2/
x/q,, xat)2
k/p²/
(g(t) ly 142
2023), Diffusion Posterior Sampling (DPS) (Chung al. 2022a), and projection (Song al.) 2020 and see how they can be leveraged for our joint posterior sampling framework. In all methods, to ensure traccability of pyy [ ne), it is modeled Gaussian, namely:
p n.) xe N(7 Ms, E.),
(9)
where the three different methods employ different approximations for the parameters of the Normal distri- bution. In all three methods, the covariance E, is not a function of a or nh, and we can thus write the noise-perturber likelihood score as:
Vz.n, log poyp(y n,) = [V Ho E, (n pc).
(10)
We will now specifically derive the sampling procedure for our joint diffusion process using IIGDM as basis. Additionally, we provide derivations for DPS and the projection method in Appendix A Finally, Table [ shows overview of the choices made for each parameter in the different methods.
Similar to the original ITGDM paper, we start with an approximation of æt, toward To. no, which then allows the usage of the known relationship of p(y|oo, no). Since y, 3t, and ny are conditionally independent given ty and no, we can write:
/ p(zu x (no n.)p( Do, 20
plyl n.)
Øo no no)dngdoo.
(11)
which is a marginalization over vn and Now, we have substituted the intractability of computing plyle,, ne), for the intractability of computing (scores of) p( (x,) and p(no (ms). 11GDM then estimates p(zp) (2)) using variational inference (VI), where models the reverse diffusion steps as Gaussians, which we exlend here to the noise as well:
{
p(æolæ:) Rs (a ri p(nolm:) = N(n oz T),
(12)
where of and ri represent the uncertainty error made in the V1. The means of the Gaussian approximations (Zoje no e) are calculated using Tweedie's formula, which can be thought of as a one-step denoising process using our trained diffusion model to estimate the true and no:
St + B? V u, log p(oe)
x. + (3,5
Zojt = E[zg|211
(13)
a1
with an analogous equation for Roj-- Here, ag and Bg can be derived from the SDE formulation as mentioned in Section 2.1 Substitution of the VI estimate (equation 12 into equation 11 then results in an approximation of the nersu-perturbed likelihood:
7t y
plul n.) N N(76 Ei)
1r Awuji + noj
(14)
Σ, = r AAT + o
Subsequently, we derive the following estimated noise- peiite-prttrbed likelihood scores:
Væ, logp(y/< ne) (V Zpis) AT (y - Awojt - rejte)
(15)
Vn, log logp(y)z 22, ne)
(V (nn
Σ, (g Ax Rojo))

---

Published in Transactions on Machine Learning Research (03/2025)
where Va, and Vn, Roje are the Jacobians of equation 13 which can be computed using automatic differentiation methods. In IIGDM, the Bayesian weighting terms À and K are not fixed scalars, rather these are chosen to be equal to the estimated VI variances, rf and qZ. Additionally, the diffusion coefficient g(t)? gets cancelled out in the weighting scheme. Lastly, this work, we introduce the additional explicit scalars X and n', to bring in line with the other data consistency rules. Note that introducing these scalars is the same scaling r and qf by a fixed amount for all timesteps.
Song et al. (2023) provide recommendations for choosing the variance of the VI, namely r² = 9a-1 when the noise model is a known tractable distribution, which we adopt. Additionally, since we here introduce the notion of modeling n using a different diffusion model, we also set the variance of the VI estimate of p(nolm)) to of r?, as it is subjected to similar SDE trajectory.
Algorithm 1: Joint posterior sampling with IIGDM for score-based diffusion models
Require: T, so, sq. A, K. r?. 92 y
1 Ør ((x), n1 (n), At 4
// Unconditional diffusion stepa 3t-Ar At + 2, - f(1)x;A/
13
2
3 for i= T to 0 do
14
It At + x1- + g()) (z A)
(0,
4
( -
1/ Data consistancy stops
16
T at t It- + g(t) Atz
Foft + (ætt + B3 ((2, ar
17
6
Nui: + -(n+ 8 (Tn, f))/at
18
ni nu f(())
7
1 t Azolt + Raje
19
T4-A: At f -Tt-1: g(())² st (n t)
8
Er + rÂAA + of-
20
2 (0,
9
Et <- Dt - Xr? V 20jz)
A Σ (y- - 14)
21
74- t nt-At + g(t) Vtzz
The f nae Vn, mooje)
Σ (y 1b)
22
end
return: To
11
4 Related Work
In this section, we discuss alternative approaches for tackling inverse problems with structured noise using deep generative models, namely normalizing flows (NF) and generative adversarial networks (GAN). These methods, along with three widely used diffusson posterior sampling methods, serve as baselines in our experi- ments evaluate the performance of the proposed diffusion-based denoisor. Importantly, while the diffusson methods included in the comparison do not explicitly model the noise prior, our approach is the first to tackle structured noise in inverse problem settings. Direct application of existing diffusion posterior sam- pling methods without the proposed joint-sampling framework fails to effectively remove structured noise, as shown in our experimental results. That being said, we do show the compatibility of our method with current state-of-the-arl guided diffusion samplers. Finally, the NF- and GAN-based methods discussed in the following section rely on MAP estimation, see Section whereas we perform posterior sampling.
Normalizing Flows: Whang et al. (2021) propose to use normalizing flows to model both the data and the noise distributions. Normalizing flows are a special class of likelihood based generative models that make use of an invertible mapping G: Rd - IRd to transform samples from a base distribution pz(z) into a more complex multinodal distribution x G(z) px()). The invertible nature of the mapping G allows for exact density evaluation through the change of variables formula:
logpx = logp2 (x) + log det (x))
(16)
where J is the Jacobian that accounts for the change in volume between densities. Since exact likelihood computation is possible through the flow direction G-1, the parameters of the generator network can be optimized to maximize likelihood of the training data. Subsequently, the inverse task is solved using the MAP estimation in equation 3
a = arg max flog Pc (v Ax) + log PGx (z))
(17)
7

---

Published in Transactions on Machine Learning Research (03/2025)
where GN and Gx are generative flow models for the noise and data respectively. Analog to that, the solution can be solved in the latent space rather than the image space as follows:
2 arg max (og (y A(Gx(z))) + Alogpax
(18)
Note that in equation 18
smoothing parameter À is added to weigh the prior and likelihood terms,
also done in Whang et al. (0221). The optimal & or & can then be found by applying gradient ascent on respectively.
equation 17 or equation 18
Generative Adversarial Networks:
Generative adversarial networks are implicit generative models that can learn the data manifold in an adversarial manner (Goodfellow et al. 2020). The generative model is trained with an auxiliary discriminator network that evaluates the generator's performance in a minimax game. The generator G(z) R' Rd maps latent vectors z E RC NV(,, I) the data distribution of interest. The structure of the generative model can also be used in inverse problem solving (Bora et al, 2017). The objective can be dcrived from equation | and is given by:
arg min {1l AGx + A|2211)
(19)
where À weights the importance of the prior with the measurement error. Similar to NE, the optimal é be found using gradient ascent. The & regularization term the latent variable is proportional to negative log-likelihood under the prior defined by Gx, where the subscript denotes the density that the generator is approximating. While this method does not explicitly model the noise, it remains an interesting comparison, as the generator cannot reproduce the noise found in the measurement and can only recover signals that are in the range of the generator. Therefore, due to the limited support the learned distribution, GANs can inherently remove structured noise. However, the representation error (i.e. observation lies far from the range of the generator (Bora. al. 2017)) imposed by the structured noise comes at the cost of recovery quality.
5 Implementation Details
Aulomatic hyperparameter tuning for optimal inference was performed for the proposed and all baseline methods on a small validation of only 5 images (depending on the experiment as detailed in section 6
All parameters used for training and inference be found in the provided code repository linked in the paper. A summary of the most important hyperparameters for each method can be found in Appendix C The peak signal-to noise ratio (PSNR), structural similarity index (SSIM) and perceptual similarity metric
(LPIPS) (Zhang et al.) 2018) are used to evaluate our results and inspect both ends of the perception- distortion tradeoff Blau & Michaeli 2018
5.1 Proposed Method
Given the two separate datasets, one for the data and one for the structured noise, two separate score models can be trained independently. This allows for casy adaptation of our method, since many existing trained score models can be reused. Furthermore, this ensures the same two prior networks can be used in a variety of different tasks. For both the score models, we use the NCSNv2 architecture as introduced in Song & Ermon (2020). The two priors are combined only during inference through the proposed sampling procedure as described in Algorithm D using the adapted Euler-Maruyama sampler. We use the following SDE: [() = 0, g(t) = of with 25 define the diffusion trajectory. During each experiment, we run the sampler for T = 600 iterations.
5.2 Baseline Methods
As starting point, we compare our method across all experiments with three common diffusion postcrior sampling approaches. Unlike our proposed framework, these methods rely on a Gaussian noise prior and do not utilize explicitly learned noise model.
8

---

Published in Transactions on Machine Learning Research (03/2025)
Gronn Thin
Inpal
Ours
FLOW
G
HGGDO
5
(a) CelebA with MNIST noise
(b) OoD data
(c) OoD noise (TMNIST)
Figure 2: Qualitative results the removing MNIST digits (noise) from CelebA (signal) experiment,
paring our joint posterior sampling method to the basclines:?: -IIGMM, FFLOW, !GAN, BBM3D.
The closest to our work is the flow-based noise model proposed by Whang et al. (2021) ) discussed in Section 4 which will serve as our main bascline. To boost the performance this bascline and to make more competitive, we moreover replace the originally used RealNVP (Dinh el al. 2016) with the Glow archilecture (Kingma & Dhariwal 2018). We use the exact implementation found in Asim al. (2020), with a flow depth of K = 18, and number of levels L 4, which has been optimized for the same CelebA datasct used in this work and thus should provide fair comparison with the proposed method.
Additionally, GANs, as discussed in Section | are used as comparison. We train DCGAN (Radford cct al. 2015), with a generator latent input dimension of 100. The generator architecture consists of 4 strided 2D transposed convolutional layers, having 4 x kernels yielding feature maps of 512, 256, 128 and 64. Fach convolutional layer is followed by a batch normalization layer and ReLU activation.
Lastly, depending the reconstruction task, classical non-data-driven methods used comparison. the block-matching and 3D filtering algorithm (BM3D) (Dabov et al,
For denoising experiments, 2006), and in compressed sensing experiments, LASSO with wavelet basis (Tibshirani 1996. Except for the flow-based method of Whang et al. (2021), none of these methods explicitly model the noise distribution. Still, they are a valuable baseline, as they demonstrate the effectiveness of incorporating a learned structured noise prior rather than relying on simple noise priors.
6 Experiments
We subject method to variety of inverse problems such denoising, compressed sensing, deraining, and dchazing, all with an element of additive structured noisc. To test the method's robustness, we repeat the experiments on both oul-of-distribution (OoD) data and OoD noise in Section 6.2 To show the capabilities of our method in a variety contexts, we evaluate the coint-conditonal diffusion method on different datasots, such as CelebA (Section 6.1
6.2 6.3) ImageNet (Section 6.2). FFHQ (Soction 6.4), and medical ultrasound dataset (Section Lastly, we compare the methods' computational performance in Section 6.6 The proposed method outperforms the baselines both qualitatively and quantitatively in all experiments.

---

Published in Transactions on Machine Learning Research (03/2025)
6.1 Removing MNIST digits from CelebA
For comparison with Whang et al. (2021) we recreate an experiment introduced in their work, where MNIST digits are added to CelebA faces. The corruption process is defined by y= 0.55 CCeebA +0. MANITT
Setup:
In the experiment, the signal score network se is trained on the CelebA dataset (Liu et al. 2015) and the noise score network s on the MNIST dataset, with 10000 and 27000 training samples, respectively, Images are resized to 64 x 64 pixcls. We test on randomly selected subset of 100 images.
Results: A random selection of test samples is shown in Fig, Za for qualitative analysis. Additionally, Fig. 3a shows a quantitative comparison of our method against all basclines. Both our proposed diffusion method and the flow-based method have an explicit noise prior and are able to recover the underlying signal, with the diffusion method preserving more details. While the GAN method effectively removes the digits, struggles to accurately reconstruct the faces, as it fails to project the observations onto the range of the generator. Both the BM3D denoiser as well as the diffusson method without structured noise prior (TGDM) fail to recover the underlying signal, confirming the importance of prior knowledge of the noise.
6.2 -itt-f-ddistribution data and noise
In real-world applications, both signal and noise are often subject to distribution shifts with respect to the original training data, making it challenging train reliable models. While in many practical cases both signal and noise components can be measured in isolation or simulated, the resulting data may not perfectly match the true underlying distributions. This motivates the need to evaluate the robustness of models under out-of-distribution (OoD) conditions.
Setup:
To this end, we extend our previous experiments in Section 6.1 to include OoD scenarios for both signals and noise. Specifically for the signal case, we test with (1) random data from ImageNet and (2) synthetically generated data from the Stable Diffusion text-lo-image model (Rombach et al., 2022). To explore the robustness to shift in noise distribution, we introduce two OoD noise variants: (1) samples drawn from the TMNIST-Alphabet dataset, which features different characters, and (2) random translations applicd to the noise (digits). Importantly, we use the exact same hyperparameters and models as in the original non-OoD experiments.
Results: Qualitative results for the OoD data and noise experiments are shown in Fig 2b and Fig 2 respectively. Consistent with prior findings (Asim et al.) 2020 Whang et al.) 2021). the Row-baser method shows robustness to OoD data, unlike the GAN. We empirically show that the diffusion method is also resistant to OoD data and noise in inverse tasks with complex noise structures and demonstrates superior performance over the baselines. Quantitative results for the OoD data experiment are shown in Fig while we refer the reader to Appendix [.1] for extended results on the OoD noise experiments. Among the OoD noise variants, random translations proved more challenging than TMNIST-Alphabet characters, with our method maintaining its competitive edge.
6.3 Compressed sensing with structured noise
Setup: In this experiment, the corruption process is defined by y = Ax + nsine with random Gaussian measurement matrix A md and noise source with sinusoidal variance o% x exp(sin( 2xx) for each pixel k, which d/m. Additionally, we include an experiment with the special case A = I and 2D sinusoidal noise pattern, where h is now each row in the image.
use to train 8g. The subsampling factor is defined by the size of the measurement matrix
Results: In Fig. 4a the results of the compressed sensing experiment and the comparison with the basclines are shown for an average standard deviation of a = 0.2 and subsampling of factor d/m- = 2. The proposed method demonstrates robust recovery under structured noise and distribution shifts distiriitt (OoD) cases. In contrast, the flow-basod mothod underperforms when subjected to the OoD data, see Fig. 4b A qualitative analysis is shown in Appendix B.2 Interestingly, DPS (diffusion without explicit noise model
2+Our Song et al. 2023 Whang et al. 2021
Bora al. 2017 Dabov al. 2006
Tibshirani 1996
10

---

Published in Transactions on Machine Learning Research (03/2025)
1.0
1.0
30
30
0.8
0.8
PSNR
20
0.6
LPPPS
PSNR
20
0.6
LPIPS
0.4
0.4
10
10
0.2
0.2
0.0
0.0
BM3D
I1DD
GAN
FLOW
Ours
BM3D
HIGMM
GAN
FLOW
Ours
(a) CelebA
(b)
Out-o--distribution data
Figure 3: Quantitative results using PSNR (green) and LPIPS (blue) for the removing MNIST digits exper-
iment of the (a) CelobA and (b) ont-of-distribution datasets.
Ground Truth
Truh
Iqpa:
Outs
Oa
DVS
DIS
FLOW
FLOW
GAN
GAN
LASSO
BMGD
(a) CelebA with structured noise
(b) Out-of-distribation data
Figure 4: Results on the compressed sensing with structured noise experiment, comparing diffusion-based method to the baselines.
Figure 5: Results the 2D sinusoidal noise experiment.
performs relatively well in this CS experiment, which is likely due to the random mapping of the Gaussian noise pattern through the measurement matrix reduces the structure of the noise. The necessity of a learned noise prior becomes more apparent in Fig. 5 where DPS is unable to obtain an accurate estimate of the signal. For detailed results see Appendix B.2
6.4 Deraining FFHQ
In this experiment, we address the problem of deraining, which involves removing rain streaks from images significantly occluding objects of interest. We employ the 256 x 256 FFHQ dataset to assess our method's performance on high-resolution images. In this setup, the signal diffusion model is trained on the FFHQ dataset, whereas the noise model is trained using rain simulator.
Setup:
Results: We compare our method diffusion posterior sampling without an explicit noise model in Fig. The proposed method scores PSNR 23.97, 0.82, LPIPS 0.18. Unsurprisingly, observe that
11

---

Published in Transactions on Machine Learning Research (03/2025)
y
Cimund
Noisy
0.08
ft
1101
0.70
Imput
Ours
X
0.64
f 1155
0.68
DPS
DRS
Figure 6: Comparison of diffusion posterior sampling methods with explicit noise model (Ours) and without (DPS) on the task of dehazing ultrasound data. The gCNR metric is given for each example.
Figure 7: Deraining experiment on the FFHQ 256 256 dataset com- paring DPS with ours.
diffusion models without explicit modeling of the noise distribution (DPS) fail to accurately reconstruct the images under heavy structured noise, it scores PSNR 19.70, SSIM 0.67, LPIPS 0.40.
6.5 Medical Ultrasound Reconstruction
Sctup: To evaluate the proposed method in a realistic medical imaging context, we address the inverse problem of dehazing in cardiac ultrasound, aiming to reconstruct a clear depiction of the anatomy from hazy observations. The haze artifact arises from multipath scattering between the probe and tissue of interest. We train our diffusion model on log compresed beamformed IQ data, and model the observed data as = x+n,, where æ represents signals originating from tissuc and n corresponds to the multipath scattering. The signal dataset is constructed using clean images minimally impacted by haze, while the noise dataset is acquired by capturing data from ultrasound probe scanning medium with high scattering.
We evaluate the method using the unsupervised generalized contrast-to-noise ratio (gCNR t) metric yielding values of 0.58, 0.62, and 0.65 for the noisy input, DPS, and the proposed method, respectively, across a lest set of 100 images. Fig. G presents qualitative results, including noise estimates from both meth Unlike the proposed mothod, DPS leaves residual signal in its noise estimate, effectively "eating away" at the clean signal, which is often unacceptable in a clinical context. In contrast, the proposed method produces noise estimates that closely resemble the haze, effectively suppressing the hazy regions without distorting the underlying anatomy, resulting in clearer images.
Results:
cardiac ultrasound dataset, acquired with Philips X51-c probe,
et al 2020)
6.6 Performance
To highlight the difference in inference lime between our method and the baselines, benchmarks are performed on a single 12GBytes NVIDIA GeForce RTX 3080 Ti, see Table in Appendix B A quick comparison of inference times reveals 4x (UGDM) or 10x (Projection) difference in speed between ours and the flow-based method. All the deep generative models need approximately an equal amount of iterations (T 600) converge. However, given the same modeling capacity, the flow model requires substantially more trainable parameters compared to the diffusion method. This is mainly due to the restrictive requirements imposed the architecture lo ensure tractable likelihood computation. It should be noted that in this work no improvements are applied to speed up the diffusion process, such as distillation (Salimans & Ho 2021) or improved initialization (Chung et al. 2022c leaving room for even more improvement future work.
7 Discussions
Inverse problems are powerful tools for inferring unknown signals from observed mcasurements and have been at the center of many signal and image processing algorithms. Strong priors, often those learned through deep generative models, have played crucial role in guiding these inferences, especially in the context. of
12

---

Published in Transactions on Machine Learning Research (03/2025)
high-dimensional data. While complex priors on the signal are commonly employed, noise sources are often assumed to be simply distributed, drastically reducing their effectiveness in structured noise settings.
In this work, we address this limitation by introducing novel joint posterior sampling technique. We not only leverage deep generative models to learn strong priors for the signal, but we also extend our approach to incorporate priors on the noise distribution. To achieve this, we employ an additional diffusion model that has been trained specifically to capture the characteristies of structured noise. Furthermore, we show the compatibility of our method with three existing posterior sampling techniques (projection, DPS, IIGM)). We demonstrate method natural and ofttt---iitrittti data and noise and achieve increased performance over the state-of-the-art and established conventional methods for complex inverse tasks. Additionally, the diffusion-based method is substantially easier to train using the score matching objective compared to other deep generative methods that rely on constrained neural architectures or adversarial training.
While our method shows considcrable improvements speed and effectiveness at removing structured noise compared to the flow-based method, it is not yet suitable for real-time inference and still lags behind GANs and classical methods in terms of inference speed. Fortunately, research into accelerating the diffusion pro- cess is on its way. In addition, although a simple sampling algorithm was adopted in this work, many more sampling algorithms for score-based diffusion models exist. Future work should explore this wide increase in design space to understand the limitations and possibilities of more sophisticated sampling schemes in com- bination with the proposed joint posterior sampling method. Additionally, our method assumes independent noise and linear measurement models. Extending to a broader family of possibly non-linear or dependent cases is an interesting direction for future work. Lastly, the connection between diffusion models and con- timuous normalizing flows through the neural ODE formulation (Song et al.) (021a) is not investigated but is of great interest given the comparison with the flow-based method in this work.
8 Conclusions
In this work, we presented framework for removing structured noise using diffusion models. The proposed joint posterior sampling technique for diffusson models has been shown to effectively remove highly structured noise and outperform baselines in both image quality and computational performance. Additionally, it exhibits enhanced robustness in out-of-distribution scenarios. Our work provides an efficient addition to existing score-based conditional sampling methods by incorporating knowledge of the noise distribution, whilst supporting a variety of guided diffusion samplers. Future work should focus on accelerating the relatively slow inference process of diffusion models and further investigate the applicability of the proposed method outside the realm of natural images.
References
Brian DO Anderson. Reverse-time diffusson equation models. Slochastic Processes and their Applications, 12(3):313-38) 026,,32222222 1982.
Muhammad Asim, Max Daniels, Oscar Leong, Ali Ahmed, and Paul Hand. Invertible generative models for inverse problems: mitigating representation error and dataset bias. In International Conference on Machine Learning, pp. 399-409. PMLR, 2020.
Arpit Bansal, Eitan Borgnia, Hong- Min Chu, Jie S Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. arXiv preprint arXiv:2208.008 093922 2022.
Amir Beck and Mare Teboulle. A fast itcrative shrislagg-thresooldinng algorithm for linear inverse problems. STAM journal on imaging sciences, 2(1):183-2P) 202020022222202 2009. Dana Berman, Shai Avidan, et al. Non-local image dehazing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2016.
Yochai Blau and Tomer Michacli. The perception tradeoff. In Proceedings of the IEEE conference on computer vision and poltern recognition, pp. 6228-6237, 6237, 2018.
13

---

Published in Transactions on Machine Learning Research (03/2025)
Ashish Bora, Ajil Jalal, Eric Price, and Alexandros G Dimakis. Compressed sensing using generative models. In International Conference on Machine Learning, pp. 537-546. PMLR, 2017.
Jiezhang Cao, Yue Shi, Kai Zhang, Yulun Zhang, Radu Timofte, and Lue Van Gool. Deep equilibrium diffusion restoration with parallel sampling. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2824-2834, 2024.
Hyungjin Chung and Jong Chul Ye. Score- diffusion models for accelerated mri. Medical Image Analysis, pp. 102479, 2022.
Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Marc L Klasky, and Jong Chul Ye. Diffusion postcrior sampling for general noisy inverse problems. arXiv preprint arXiv:2209. 1 14687, 2022a.
Hyungjin Chung, Bycongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. Xin preprinl. or arXiv:2206.0000 00941, 2022b.
Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Cone--loser-ddiffus----ssss faster: Accelerating condit diffusion models for inverse problems through stochastic contraction. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 12413-12422, 2022c.
Kostadin Dabov, Alessandro Foi, Vladimir Katkovník, and Karen Egiazarian. Image denoising with block- matching and 3d filtering. Image processing: algorithems and systems, neural networks, and machine learning, volume 6064, pp. 354 365. SPIE, 2006.
Giannis Daras, Yuval Dagan, Alex Dimakis, and Constantinos Daskalakis. Score- intermediate level optimization: Fast langevin mixing for inverse problems. In International Conference on Machine Learning, pp. 4722 4753. PMLR, 2022a.
Giannis Daras, Mauricio Delbracio, Hossein Talebi, Alexandros G. Dimakis, and Peyman Milanfar. Soft diffusion: Score matching for general corruptions, 2022b.
Prafulla Dhariwal and Alexander Nichol. Diffusson models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8799 2021.
Laurent Dinh, Jascha Sohl-Dickstoin, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605. 08803, 2016.
Berthy T Feng, Jamie Smith, Michael Rubinstein, Huiwen Chang, Katherine L Bouman, and William T Freeman. Score-based diffusion models as principled priors for inverse imaging. arXiv preprint arXiv:2304.11751, 2023.
Marc Anton Finzi, Anudhyan Boral, Andrew Gordon Wilson, Fei Sha, and Leonardo Zepeda-Núñez, User- defined event sampling and uncertainty quantification in diffusion models for physical dynamical systems. In International Conference on Machine Learning, pp. 10136-10155 PMLR, 2023.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commumications of the ACM, 63(11): 139-144, 2020.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXio preprint 2xx:22207.12500 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6857 2020.
Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. Robust compressed sensing mri with deep generative priors. Advonces in Newral Information Processing Systems, 34:14938-1499 14954, 2021a.
Ajil Jalal, Sushrut Karmalkar, Alex Dimakis, and Eric Price. Instance-optimal compressed sensing via posterior sampling. In International Conference Machine Learning, pp. PMIRR 2021b.
14

---

Published in Transactions on Machine Learning Research (03/2025)
Bowen Jing, Gabriele Corso, Renato Berlinghieri, and Tommi Jaakkola. Subspace diffusion generative models. arXiv preprint arXim:2205.01 01400, 2022.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. orXw preprint 2iv: 2r0622200.00364 2022.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. Snips: Solving noisy inverse problems stochastically. Advances in Newral Information Processing Systems, 34:21757-21769 2021.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Newral Information Processing Systems, volume 35, pp. 23593 23606. Curran Associates, Inc., 2022.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. Advances in neural information processing systems, 31, 2018.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Calvin Luo. Understanding diffusion models: unified perspective. arXiv preprint ar Xiv:2200.. Xfx0:2208.11970, 2022.
Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas B Schön. Image restoration with mean-reverting stochastic differential equations. arXiv preprint arXiv:2301.11699 2023.
Stéphane Mallat. A wovelet tour of signal processing. Elsevier, 1999.
Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat. A variational perspective on solving inverse problems with diffusion models. arXiv preprint arXi::2305.0433 2023.
Xiangming Meng and Yoshiyuki Kabashima. Diffusion model based posterior sampling for noisy linear inverse problems. arXiv preprint ar arXiv:2211.2234 2022.
Geon Yeong Park, Sang Wan Lee, and Jong Chul Ye. Inference-time diffusion model distillation. arXiv preprint arXiv:2412.8877 08877111, 2024.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolu- tional generative adversarial networks. arXio preprinl urXiv:1511.06434 2015.
Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhn, and Deyu Meng. Progressive image deraining networks: A better and simpler baseline. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 3937-394 197-3994, 2019.
Alfonso Rodriguoz-Molares, Ole Marius Hoel Rindal, Jan D'hooge, vein-Erik MåsPy, Andreas Austeng, Muyinatu A. Lediju Bell, and Hans Torp The generalized cont rast-tto--nise ratio: A formal definition for lesion detectability. IEEE transactions ultrasonies, ferroelectrics, and frequency control, 67(4):745-75) 759, April 2020. ISSN 0885-3010. doi: 10. (0.100//UFFC..2011226666
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esscr, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 10684 10684-10695, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference Learning Representations, 2021.
Jiaming Song, Arash Vahdat, Morteza. Mardani, and Jan Kautz. Psendoiinerss-guidee diffussoo models for inverse problems. In International Conference on Learning Representations, 2023.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
15

---

Published in Transactions on Machine Learning Research (03/2025)
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33:12438-1222 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference Learning Representations, 2020.
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of acore-based diffusion models. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vanghan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 1415 1428. Curran Associates, Inc., 2021a.
Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with scoro-based generative models. In International Conference on Learning Representations, 2021b.
Tristan S. W. Stevens, Oisín Nolan, Jean-Luc Robert, and Ruud J.G. van Sloun. Sequential Posterior Sampling with Diffusion Models. In 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Hyderabad, India, 2025.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Slatistical Sociely: Series B (Methodological), 58(1):267-25) 288, 1996.
Faruk Uysal. Synchronous and asynchronous radar interference mitigation. IEEE Access, 7:5846 58526655522--0 2018.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 34:11287 11302, 2021.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23 (7):1661-1674, 2011.
Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. arXiv preprint Xrxi::2212.00. 00490. 2022.
Xinyi Wei, Hans van Gorp, Lizeth Caraaai,, Daniel Freedman, Yonina C Eldar, and Rund JG van Sloun. Deep unfolding with normalizing flow priors for inverse problems. IEEE Transactions on Signal Processing, 10:2962-2971, 2022.
Jay Whang, Qi Lei, and Alex Dimakis. Solving inverse problems with flow based noise model. In Interna- tional Conference on Machine Learning, pp. 11156-11157. PMLR, 2021.
Jian Yang, Jingfan Fan, Danni Ai, Xuehu Wang, Yongchang Zheng, Songyuan Tang, and Yongtian Wang. Local statistics and non-local mean filter for speckle noise reduction in medical ultrasound image. Nearo- computing, 195:88-95, 2016.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
16

---

Published in Transactions on Machine Learning Research (03/2025)
A Derivation of Data Consistency Steps
The proposed joint posterior sampling framework for removing structured noise is versatile and compatible with various existing diffusion posterior sampling methods. In Section 3.2 we established the foundation for jointly sampling from two distributions (signal and noise), given a corrupted observation. Specifically, a derivation was given for the 1IGDM data consistency method. The following section presents two additional examples of leveraging popular guidance methods for diffusion models to remove structured noise, namely DPS and projection. A comparison of all methods is shown in Appendix B.4
A.1 DPS
Diffusion Posterior Sampling (DPS) CChng et al. 2022a also leverages Tweedie's formula in order estimate Poj and noje. However, unlike IIGDM, DPS does not leverage VI with Gaussian posteriors. Instead, Gaussian error with diagonal covariance and variance p² is assumed, which again can adapt to our
problem as such:
Yt = y
plyla,, ne) N Me, Ex)
μr = Azo + Noje
(20)
Ee I²I,
resulting in the following scores:
V ae log logp(y/z,,, ni)
V x. Doje) V Mojee) (y AxoIe mojot)
A A Axolt - nooie)
2
(21)
V n. log puy aa, mz):
Note the difference between equations 15 and 21 The former employs non-diagonal covariance matrix, while the latter uses a simple diagonal approximation. In other words, DPS does not take into account how the variance of the estimation Toit gets mapped to y, in the case of a non-diagonal measurement matrix A. The authors of DPS (Chung al.] (022a)) then propose to rescale the noise, or step size, of the noise-perturbed likelihood score by fixed scalar divided by the norm of the noise-perturbed likelihood. Additionally, the diffusion coefficient gt))² gets canceled out in the weighting scheme. Again, achieve that here by choosing À and appropriately.
A.2 Projection
The projection method (Song et al. 2020) takes another approach altogether in comparison with IIDM and DPS. Instead of relating æt, toward Zo no, it relates y to ye and then uses the following approximation:
plyle n.) 20 ((0:/2r. nz),
(22)
where Ú: is a sample from p(y ly), and (ye W hejo, [) the observation along the SDE trajectory together with x; Note that in the case of linear measurement p(y:ly) is tractable, and we can easily compute ÿ. = a,y + B,Az, using the reparameterization trick with z E Rd (N, I), see follow-up paper of the same group; Song al. (2021b). In contrast to the case where we DPS and IIDM, which perform the data consistency using noiscless estimates at diffusion time / = 0, the projection method projects the observation to the current diffusion step l. Consequently, we cannot sample the noise vectors z independently anymore, but should reuse them for the forward diffusion of signal, noise and observation.
is an additional stochastic process that essentially corrupts
We then use the measurement model which is normally only defined for time = 0, and apply it to the current timestep f. In this approximation, we assume that we make a Gaussian error with diagonal covariance and standard deviation 0² as:
7t = 1. Mi = Ax; + n2
plyla,, ne) ~ N Ee)
(23)
Σ, = p²I.
17

---

Published in Transactions on Machine Learning Research (03/2025)
Calculating the score of equation 23 with respect to both æt and n, then results in:
V æ, poyl(y) n²))
p A Ax n.)
]
~
(24)
V n, log p(yla,, n²)
(8:
Ax: n2)
Similar to DPS, we reweigh the scores in order to cancel out both g()) and 1/p², using À and K, see Table 1
B Extended results
In the following section, complement the experiments outlined in Section with further analysis. Addi- tionally, there are other experiments, such as comparison of data consistency methods used in combination with the proposed joint-sampling framework; see Appendix B.4
B.1 Out-of-distribution data and noise
Table 2: Results for the OoD signal and noise (TMNIST or translation) experiments Section 6.2
Problem: y 0.5 #CelebA voD + 0.5 RTMNNIST translation
Dataset
TMNIST
translation SSIM (1)
PSNR (t)
SSIM (t)
LPIPS (1)
PSNR (t)
LPIPS (L)
Noisy
CelebA
12.26 + 2.0
0.633 + 0.02
0.305 = 0.11
12.14 + 2.0
0.634 = 0.02
0.279 + 0.07
*Ours
CelebA
25.94 + 2.4
0.851 + 0.04
0.159 + 0.06
23.63 + 4.1
0.893 + 0.04
0.150 + 0.07
'FLOW
CelebA
22.61 + 1.1
0.826 + 0.05
0.179 + 0.06
22.96
+ 1.1
0.837 + 0.05
0.167 0.05
Noisy
OoD
11.50 + 1.3
0.634 + 0.01
0.251 + 0.08
11.49
+ 1.3
0.641 + 0.01
0.203 + 0.08
*Ours
OoD
22.59 +: 2.4
0.858
0.06
0.197 1 0.08
21.55
1 3.0
0.895 + 0.05
0.167 0.09
1FLOW
OoD
20.06
0.831 0.08
0.177
20.54
1 2.1
0.08
0.06
B.2 Compressed sensing
Table 3: Quantitative results for compressed sensing experiments outlined in Section 6.3
Problem: y Ax + Msine A E Rmxxd d/m = 2, Naine (O0 of), exp(sin 25r
)) for each pixel k.
16
CelebA SSIM (t)
OoD
PSNR (1)
LPIPS (4)
PSNR (1)
SSIM (1)
LPIPS (L)
"Ours
25.51 + 1.0
0.823
0.04
0.042 0.02
22.90 1.6
0.823 = 0.08
0.059 = 0.02
*PS
25.34 + 1.0
0.820
0.05
0.052 0.03
22.79
1.6
0.818 + 0.09
0.069 0.04
FFLOW
24.96 + 2.3
0.779
0.08
0.105
0.07
19.85
+ 4.8
0.608 + 0.18
0.266 = 0.16
1GAN
18.90 + 1.3
0.529
0.08
0.136 + 0.06
12.39 + 1.7
0.159 + 0.07
0.518
0.14
LASSO
12.93 + 1.8
0.284 0.04
0.645 0.08
11.62
1.5
0.336 + 0.06
0.493 0.10
Table 4: Quantitative results for the structured sinusoidal noise experiments as outlined in Section 6.3 )) for each row k.
Problem: y = Dsine Aesine ~ NO1 of), ax x exp(sin( 2nk
Celeb SSIM (t)
PSNR (t)
I.PPSS (J)
Nolsy
15.60 + 0.3
0.434 + 0.06
0.458 + 0.11
"Ours
18.61
1.0
0.772 + 0.05
0.054 + 0.02
DPS
22.31
1.0
0.716 t 0.06
0.074 t 0.04
FLOW
18.11
0.800 + 0.05
0.070 =
GAAN
21.15
1.5
0.632 + 0.07
0.097 = 0.04
BM3DD
23.12 1.0
0.695 1 0.06
0.209 1 0.06
2+Ours, Song al. 2023
Chhng al.
2022a
Whang
2021
1
2017
Dabov
2006
Tibshirani
18

---

Published in Transactions on Machine Learning Research (03/2025)
B.3 Performance
A summary of the performance of the proposed methods and baselines as discussed in Section 6.6 is listed in Table 5
Table 5: Inference performance benchmark for all methods.
Model
# trainable parameters
Inference time
[ms]]
*Ours
(Proj-) (DPS)
8.9M
5605
16818
(TIGDM)
16094
FFLOW 1GAN
25.8M
61853
3.9M
59
BB3D
29
B.4 Comparison Data Consistency Methods
The proposed joint sampling framework outperforms any of the baselines mentioned in Section
regardless of
which of the three diffusion-based data consistency methods are used as basis, see Section 3.2 IIGM), and Appendix (DPS, projection). Nonetheless, we investigate how the specific data-consistenccy rule used affects the performance of our hethod in the task of removing structured noise. As shown in Fig. 8a IDGM as basis for our framework provides the most consistent results with lower variance between samples. Empirically, this trend continues to be seen in the out-of-distribution datasots; see Fig. 8b This not surprising as IDDGM has a sophisticated approximation for the noise-perturben likelihood score compared to DPS and the projection method. A visual comparison is shown in Fig. DG Note that in all these experiments the samplers are used in combination with proposed joint sampling framework. Straightforward inference without learned model for the noise distribution is unable to effectively remove structured noise as seen in Fig 7
Eicound Trush
1.0
Input
0.8
0.8
PSNR
25
5500
PSNN
RGDM
0.4
0..4
0.2
0.2
DPS
Dm9.
DPS
TGDM
Pmn.
DPS
TGDD
Phoj.
(a) CelebA + M
(b) Out-of distribution data
(c) CelebA + MNIST
Figure 8: Comparison of the proposed joint posterior sampling framework with different data consistency methods basis (projection, DPS and IIGM)). Qualitative (c) and quantitative results shown using PSNR (red) and SSTM (orange) for the removing MNIST digits experiment images of the (a) CelebA and (b) out-of-distribution datascts.
19

---

Published in Transactions on Machine Learning Research (03/2025)
C Hyperparameters
For extensive list of all hyperparameters used, consider looking the configuration files for each experiment in the online codebase. A more compact summary can be found in Table 6
Table 6: Hyperparameters for training and inference (CelebA + MNIST and related OoD experiments).
Hyperparamnters Diffusion
Flow
GAN
Architecture
NCSNv2 VE-SDE
Glow
DCGAN
(levels)
[= 100
f(1) 0, 9() o' = =2
(latent dim size)
K 18 (depth)
x kernel
=1
= 5 (gradient clip norm)
[512, 256, 128, 64]
(channels generator)
8: (0 -1)
[64, 128, 256, 512] (channels discriminator)
Training
Ir
0.0005
0.0001
0.0002
Adam
0.9, 82 0.999
81 0.9, 82 0.999
81 0.5, 82 0.999
epochs
150
300
100
Inference
DC rule
IIGDM
DPS
proj.
gradient ascent (MAP)
gradient. ascent (MAP)
step size
1/T
1/T
1/T
0.005
0.05
T
600
600
600
600
600
0.93
12.7
0.5
0.03
0.9
0.88
16.7
0.5
2
(B3 1)
q6
20


Reviews:
[{"review_id": "jVBqnrcnl1", "reviewer": "Reviewer_Z8d2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of this paper is the novelty of modelling the additive noise with a diffusion model. This is an interesting extension for diffusions for solving inverse problems as it should in theory generalise the applicability of diffusion models for inverse problems. \nGenerally, the paper is well presented.\n\nCurrently I do not understand how the score for the noise $n$ is trained. I believe there is an assumption that we can sample from the noise generating process, that is we have access to samples $(x,n)$. This is a current limitation for applying the method to realistic scenarios, as in practice we do not have these samples. I believe it would be highly beneficial if the authors could give an example in their computations section where the true noising process is not known and they can demonstrate an increase in performance with their method over assuming additive Gaussian noise. \n\nAnother limitation is the authors assume that $x$ and $n$ are independent. One realistic scenario for non-Gaussian noise $n$ could be that the data also contains heteroskedasticity. If the authors could comment if such an extension would be possible, this would make the current work a step closer to more applied use.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a generalised method for solving the inverse problem \n$$ y = Ax + n ,$$\nwhere we now assume that $n$ is possibly not Gaussian noise. \nThis is novel to current literature which currently assumes either Gaussian or Poisson noise models. \n\nTheir model of the noise is to assume that the diffusion noise on $x$ and the observation noise $n$ are independent. \nBy doing so we have the relation \n$$ p_{X,N} ( x,n|y) \\propto  p_{Y|X, N} (y|x,n) p_{X}(x) p_{N}(n) $$\nwhere we can model both the signal variable $x$ and noise variable $n$ with a diffusion model. \nThe authors derive the joint SDE for the reverse sampling process, where the score along the diffusion path must be estimated. \nThe authors show how three current methods for score estimation along the diffusion path may be used with their method. \nExperiments on Celeb-A with MNIST noise are given, along with other synthetic data examples.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I request that the authors add baselines of comparing their method to just assuming that $n$ is Gaussian. It would be interesting if the authors could identify a non-synthetic example of where the standard assumption of Gaussianity on $n$ is a poor approximation, and the added flexibility from their methodology could improve performance on this. \n\nI also request that the authors add additional experimental details in their appendix. Currently I do not understand how either score network was trained, the training schedule used, amount of sampling steps taken, the sampling schedule used in their diffusion process or the early stopping parameters used. For instance, for diffusion inverse problems with Gaussian noise, the guidance strength is currently a sensitive hyper-parameter. In this work, there is a guidance strength in both the $x$ and $n$ directions. I cannot currently find these parameters in the text, but are required to run Algorithm 1.", "broader_impact_concerns": "none.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XN9Q3HPT33", "reviewer": "Reviewer_u795", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed extension is simple and straight-forward. \n- The experiments on out-of-distribution data/noise are interesting. \n- The comparison with non-Diffusion methods look convincing. \n\nWeakness:\n- The proposed method needs samples from noise distributions, which may be a constraint.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to extend the diffusion priors of signals to structured noises in inverse problems such as denoising and compressed sensing. This idea is formulated as jointly inferring the signals x and noises n from the (corrupted) observation y from the posterior p(x, n|y) \\propto p(y|x, n)p(x)p(n). After constructing a reverse-time SED for the joint conditional diffusion process {x_t, n_t|y}, the focus is shifted to the score function \\nabla_{x_t, n_t} \\log p(x_t, n_t|y), which can be decomposed based on the factorization of p(x_t, n_t|y). This formulation extends the previous ones with tractable noise distributions. The authors then apply this extension to three existing methods for inverse problems: Pseudo-Guided Diffusion Models, Diffusion Posterior Sampling, and projection, each has their own way of approximating p(y|x_t, n_t). Due to the parallelism between x and n, all three extensions are quite straight-forward. Experiments include (1) removing MNIST digits (as corruption) from CelebA signals, (2) sinusoidal noise with compressed sensing, (3) removing sinusoidal noise, (4) generalizing to out-of-distribution data/noise, (5) deraining FFHQ. The proposed method outperforms GAN, Flow-based method, block-matching and 3D filtering algorithm, and LASSO.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I wonder if the authors would like to provide a comparison with the original Pseudo-Guided Diffusion Models, Diffusion Posterior Sampling, and projection in the same dataset i.e. CelebA, with both simple and richly structured noises.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hTwoYNnguH", "reviewer": "Reviewer_DXv6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The studied problem is interesting and relevant, extending the diffusion-based posterior sampling approach to structured noise models definitely seems like a good contribution to the field.\n- The problem and proposed approach are introduced well in the Abstract, Introduction and Figure 1.\n- The proposed method seems to work quite well on the current experiments.\n\n\n\n\nWeaknesses:\n- The paper could be more well-written overall. It contains some typos etc, and the methods and experiments/results are somewhat difficult to follow in Section 3 and 6, respectively.\n- The proposed method is only applied to inverse problems with _synthetic_ structured noise. Even in the \"Deraining FFHQ\" experiment, synthetic rain streaks are added to images, and the noise model is trained on the same(?) type of simulated rain. It is unclear to me how the proposed method could be applied to real-world data with actual rain, fog or haze. Therefore, it is also unclear to me how useful this method would be in practice.\n\n\n\n\nOther questions/suggestions:\n- In eq. (8) you give separate expressions for $\\nabla_{x_t}$ and $\\nabla_{n_t}$, but in eq. (7) you should have $\\nabla_{x_t, n_t}$?\n- Why PSNR and SSIM as the only evaluation metrics, would it not be relevant to also include perceptual metrics such as LPIPS and FID?\n- Is there really no diffusion-based method that could be relevant to include as a baseline? For example, what is the key difference between the proposed method and \"Image Restoration with Mean-Reverting Stochastic Differential Equations\" (Luo et al., ICML 2023)?\n- Why are there no quantitative results reported for the experiments in Figure 4 or Figure 5? \n\n\n\n\nMinor things:\n- The formatting of Figure 4 and 5 is perhaps not ideal, maybe combine into sub-figures of a single figure?\n- Figure 1 is never mentioned in the text?\n- Eq. (1) is missing a \"=\"?\n- Should it not be \"+\" instead of \"-\" in eq. (3)?\n- I think it could be helpful to add a short paragraph to the start of Section 3, clarifying what will be covered in the following subsections.\n- Inconsistent use of e.g. \"in equation 7\" and \"in (9)\" (Table 1 caption) when referencing equations. Also, \"Substitution of the VI estimate (12) into equation (11)\" in Section 3.2.1.\n- \"Song et al. provide recommendations for choosing the variance of the VI Song et al. (2023)\" in Section 3.2.1, odd citation formatting.\n- Section 4, \"Whang et al. propose to use normalizing flows to model both the data and the noise distributions Whang et al. (2021)\", also odd citation formatting.\n- Multiple \"(Song et al., 2020)\" citations should actually be \"(Song et al., 2021b)\", for the projection method, or?\n- Figure 2 caption, \"to the baselines2: †FLOW, ‡GAN, and §BM3D\", but there are no footnotes anywhere?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a framework for removing structured noise using diffusion models, extending the posterior sampling approach to inverse problems with non-Gaussian noise models.\n\nThey do so by introducing a joint posterior sampling technique, utilizing an additional diffusion model which is trained to capture the characteristics of the structured noise.\n\nThey apply their method to inverse problems with synthetic structured noise, mainly based on the CelebA and MNIST datasets (e.g., adding MNIST digits to CelebA images and then recovering the clean CelebA images).\n\nThey compare their method mainly with normalizing flow and GAN models.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The studied problem is interesting. This could definitely be relevant for the TMLR audience.\n\nHowever, I have some concerns regarding the experimental evaluation, and the real-world applicability of the proposed methods requires some clarifications, se \"Weaknesses\" above.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: See above.
Decision: Accept (poster)