Removing Structured Noise with Diffusion Models
Authors: Tristan Stevens
Ruud Van Sloun
Jean-luc Robert
Faik C Meral
Jason Yu
Junseob Shin
Venue: Submitted to ICLR 2023 (2023)
Keywords: inverse problems
diffusion models
score-based
generative models
structured noise

Abstract: Solving ill-posed inverse problems requires careful formulation of prior beliefs over the signals of interest and an accurate description of their manifestation into noisy measurements. Handcrafted signal priors based on e.g. sparsity are increasingly replaced by data-driven deep generative models, and several groups have recently shown that state-of-the-art score-based diffusion models yield particularly strong performance and flexibility. In this paper, we show that the powerful paradigm of posterior sampling with diffusion models can be extended to include rich, structured, noise models. To that end, we propose a joint conditional reverse diffusion process with learned scores for the noise and signal-generating distribution. We demonstrate strong performance gains across various inverse problems with structured noise, outperforming competitive baselines that use normalizing flows and adversarial networks. This opens up new opportunities and relevant practical applications of diffusion modeling for inverse problems in the context of non-Gaussian measurements.

Under review as a conference paper at ICLR 2023
REMOVING STRUCTURED NOISE WITH DIFFUSION MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Solving ill-posed inverse problems requires careful formulation of prior beliefs over the signals of interest and an accurate description of their manifestation into noisy measurements. Handcrafted signal priors based on e.g. sparsity are increas- ingly replaced by data- deep generative models, and several groups have recently shown that state-of-the score-based diffusion models yield particu- larly strong performance and flexibility. In this paper, we show that the powerful paradigm of posterior sampling with diffusion models can be extended to include rich, structured, noise models. To that end, propose a joint conditional reverse diffusion process with learned scores for the noise and aggnalgenerating distribu- tion. We demonstrate strong performance gains with structured noise, outperforming competitive baselines that use normalizing flows and adversarial networks. This opens up new opportunities and relevant practical applications of diffusion modeling for inverse problems in the context of non- non-Gaussian measurements.
various inverse problems
INTRODUCTION
Many signal and image processing problems, such denoising, compressed sensing, phase trieval, can be formulated as inverse problems that aim to recover unknown signals from (noisy) observations. These ill-posed problems are, by definition, subject many solutions under the given measurement model. Therefore, prior knowledge is required for a meaningful and physically plau- sible recovery the original signal. Bayesian inference and maximum posteriori (MAP) solutions incorporate both signal priors and observation likelihood models. Choosing an appropriate statistical prior not trivial and often dependent on both the application as well as the recovery task.
Before deep learning, sparsity in some transform domain has been the go-to prior in compressed sensing (CS) methods (Eldar & Kutyniok (2012), such as iterative thresholding (Beck & Teboulle, 2009) wavelet decomposition (Mallat 1999). At present, deep generative modeling has estab- lished itself strong mechanism for learning such priors for inverse problem-solving Both gen- erative adversarial networks (GANs) (Bora et al. 2017) and normalizing flows (NFs) (Asim et al.) 2020 Wei et al., 2022) have been applied as natural signal priors for inverse problems in image recovery. These data-driven methods more powerful compared to classical methods, they can accurately learn the natural signal manifold and do not rely on assumptions such as signal sparsity or hand-crafted basis functions. Recently, diffusion models have shown impressive results for both conditional and unconditional image generation and can be easily fitted to a target data distribution using score matching (Vincent) 2011; Song al. 2020). These deep generative models learn the score of the data manifold and produce samples by reverting a diffusion process, guiding noise sam- ples towards the target distribution. Diffusion models have achieved srate-ofthhe-aee performance in many downstream tasks and applications. ranging from statc-of-the -fht---rrtt--tttt text-to-image models such as DALL-E (Ramesh al. 2022) to medical imaging (Song et al. 2021b) Jalal al., 2021a| Chung & Ye 2022). Furthermore, understanding of diffusion models is rapidly improving and progress in the field is extremely fast-paced (Chung et al. 2022a Bansal et al. 2022 Daras et al.) 2022a Karras et al.) 2022 Luo 2022). The iterative nature of the sampling procedure used by diffusion models renders inference slow compared to GANs and VAEs. However, many recent efforts have shown ways significantly improve the sampling speed by accelerating the diffusion process. In- spired by momentum methods in sampling, Daras et al. (2022b) introduces a momentum sampler for diffusion models, which leads increased sample quality with fewer function evaluations. CChng

---

Under review as a conference paper at ICLR 2023
et al. (2022b) offers sampling strategy. namely Come- Jome-Closer-De Mloser-Diffuse-FFi (CCDF), which leverages the conditional quality of inverse problems. The reverse diffusion can be initialized from the observation instead of sample from the base distribution, which leads to faster convergence for conditional sampling. Salimans & Ho (202)) proposes a progressive distillation method that augments the training of the diffusion models with student-teacher model setup. In doing this, they were able to drastically reduce the number of sampling steps. Lastly, many methods aim to execute the diffusion process in a reduced space accelerate the diffusion process. WhileJing al. (2222) restricts diffusion through projections onto subspaces, Vahdat et al. [2021) and Rombach al. 2022) run the diffusion in the latent space.
Despite this promise, current score score-based based diffusion methods for inverse problems are limited to mea- surement models with unstructured noise. In many image processing tasks, corruptions are however highly structured and spatially correlated. Relevant examples include interference, speckle, or haze. Nevertheless, current conditional diffusion models naively assume that the noise follows some basic tractable distribution (e.g. Gaussian or Poisson). Beyond the realm of diffusion models, Whang al. 202T) extended normalizing flow (NF)-based inference to structured noise applications. However, compared to diffusion models, NFs require specialized network architectures, which are computa- tionally and memory expensive.
Given the promising outlook of diffusion models, propose to learn models for both the noise and the desired signal and perform joint inference of both quantities, coupled via the observation model. The resulting sampling scheme enables solving wide variety of inverse problems with structured noise.
The main contributions of this work follows:
We propose novel joint conditional posterior sampling method to efficiently remove struc- tured noise using diffusion models. Our formulation is compatible with many existing iterative sampling methods for score-based generative models.
We show strong performance gains across various challenging inverse problems involving structured noise compared competitive state-of-the-an methods based on NFs and GANs.
We demonstrate improved robustness on -iito---istriuttion signals compared to baselines.
PROBLEM STATEMENT
Many image reconstruction tasks can be formulated as an inverse problem with the basic form
y Ax n,
where E I is the noisy observation, additive noise. The linear forward operator A E Rexx captures the deterministic transformation of Maximum posteriori (MAP) inference is typically used to find an optimal solution *MAP that maximizes posterior density Px|y (xly):
d the desired signal or image, and E I the
MAP = arg max PXly (xly) x
(2)
arg max [logrix(y|>)) + (x)],
x
where Py|x (ylx) the likelihood according to the measurement model and log px (x) the signal prior.
Assumptions on the stochastic corruption process n are of key importance too, in particular for ap- plications for which this process is highly structured. However, most methods assume i.i.d. Gaussian distributed noise, such that the forward model becomes PyJx (ylx) M(Ax, o I). This naturally leads to the following simplified problem:
MAP = arg min
Ily
Ax 1 lx
(4)
20
However, as mentioned, this naive assumption can be very restrictive as many noise processes are much more structured and complex. A myriad of problems can be addressed under the formulation
2

---

Under review as a conference paper at ICLR 2023
of equation given the freedom of choice for the noise source Therefore, in this work, aim is to solve more broad class of inverse problems defined by any arbitrary noise distribution Pv (n) # M and signal prior estimator proposed by Whang et al. 2021
Px (x), resulting in the following, more general, MAP
SMAP arg max logp.((s Ax)
(5)
In this paper, we propose to solve this class of problems using flexible diffusion models. Further- more, diffusion models naturally enable posterior sampling, allowing us to take advantage of the benefits thereof (Jalal et al. 2021b Kawar et al. [202]] Daras et al. 2022a).
2.1
RELATED WORK
2.1.1 NORMALIZING FLOWS
Whang et al. 12021 propose to normalizing flows (NFs) to model both the data and the noise distributions. Normalizing flows are a special class of likelihood-based generative models that make of invertible mapping G Rdd - RRd transform samples from base distribution pz(z) into a more complex multimodal distribution x G(z) px(x). The invertible nature of the mapping G allows for exact density evaluation through the change of variables formula:
log (x) = Pz))) + log det Ja
where is the Jacobian that accounts for the change in volume between densities. Since exact likelihood computation is possible through the flow direction G-1, the parameters of the generator network can be optimized to maximize likelihood of the training data. Subsequently, the inverse task is solved using the MAP estimation in equation
x = arg max (logPoN (y - Ax) + log log)
where GN and Gx are generative flow models for the noise and data respectively. Analog that, the solution can be solved in the latent space rather than the image space as follows:
à arg max (lo pc (y A(Gx(z))) + Alogpax (Gx(
(8)
Note that in equation a smoothing parameter À is added to weigh the prior and likelihood terms, also done in Whang al. (202T.
2.1.2 GENERATIVE ADVERSARIAL NETWORKS
Generative adversarial networks (GANs) implicit generative models that can learn the data man- ifold in an adversarial manner (Goodfellow et al.) 2020). The generative model is trained with an auxiliary discrininator network that evaluates the generator's performance in a minimax game. The generator G(z): R' + Rdd maps latent vectors z E RR ~ No I) to the data distribution of interest. The structure of the generative model can also be used in inverse problem solving (Bora al. 2017).
The objective can be derived from equation 3 and is given by:
arg min {lly AGx(2)) x||=112)
where À weights the importance of the prior with the measurement error. The & regularization term on the latent variable is proportional to negative log-likelihood under the prior defined by Gx, where the subscript denotes the density that the generator is approximating. While this method does not explicitly model the noise, it remains an interesting comparison, as the generator cannot reproduce the noise found in the measurement and can only recover signals that are in the range of the generator. Therefore, due to the limited support of the learned distribution, GANs can inherently remove structured noise. However, the representation error (i.e. observation lies far from the range of the generator (Bora et al. 2017)) imposed by the structured noise comes at the cost of recovery quality.
3

---

Under review as a conference paper at ICLR 2023
2.2 BACKGROUND ON SCORE- DIFFUSION MODELS
One class of deep generative models is known as diffusion models. These generative models have been introduced independently score-based models (Song & Ermon 2019 2020) and denoising
diffusion probabilistic modeling (DDPM) (Ho et al. 2020). In this work, we will consider the formulation introduced in SSng et al. (220), which unifies both perspectives on diffusion models by expressing diffusion as a continuous process through stochastic differential equations (SDE). Diffusion models produce samples by reversing corruption process. essence these models are networks trained to denoise its input. Through iteration of this process, samples can be drawn from a learned data distribution, starting from random noise.
The diffusion process of the data {x+ E 2e(o. is characterized by continuous sequence of Gaussian perturbations of increasing magnitude indexed by time t E [0, 1]. Starting from the data distribution at t = 0. clean images are defined by Xu p(xo) p(x). Forward diffusion can be described using an SDE as follows:
dx, [()xx,t g())ww
(10)
[0. + R are the drift and diffusion coefficients, respectively. Moreover, these coefficients are chosen so that the resulting distribution pi (x) at the end of the perturbation process approximates predefined base distribution pi (x) As Tx)). Furthermore, the transition kernel of the diffusion process defined as 2(x, ~ (() 3 (L)T), where a()) and (()) can be analytically derived from the SDE.
where w E Fd is standard Wiener process, f(t) : [0, 1] + R and g(t)
Naturally, we are interested in reversing the diffusion process, so that we can sample from Xp ~ Po(Xo). The reverse diffusion process is also diffusion process given by the reverse-time SDE (Anderson 1982; Song et al. 2020);
dx, [f(t)x g())² xx, log ((x)) dt + g(t)dw,
(11)
score
where W, is the standard Wiener process in the reverse direction. The gradient of the likeliieldoood of the data with respect to itself, a.k.a. the score function, arises from the reverse-time SDE. The score function is gradient field pointing back to the data manifold and can intuitively be used to guide a random sample from the base distribution xxx) to the desired data distribution. Given a dataset X' = [x(),x ",,),,,,,,,..),,,
x(xx)) } ~ p(x), scores can be estimated by training neural network
(x(x,, t) parameterized by weights 6, with score- score-mater matching techniques such as the denoising score matching (DSM) objective (Vincent 2011):
0 = arg min (..1) I [2xxxxx)))xxxx(xx. x) [1so(xi,)) - Vx, log (xx, [x)]]
(12)
Given a sufficiently large dataset X' and model capacity, DSM ensures that the score network con- verges to so (x,, t) ~ Xx, log p(x,). After training the deeedeepndentt score model sp, it can be used to calculate the reverse-time diffusion process and solve the trajectory using numerical sam- plers such as the Maler-Maruyama algorithm. Alternatively,
sophisticated samplers, such as ALD (Song & Hrmon 2019), probability flow ODE (Song et al., 2020). and Predictor-Corrector sampler (Song et al. 2020), can be used to further improve sample quality.
These iterative sampling algorithms discretize the continuous time SDE into a sequence of time steps {0 to .1,...,45 lT 1}, where noisy sample &t, is denoised produce a sample for the next time step 1. The resulting samples (Ke, constitute an approximation of the actual diffusion process [
METHOD
3.1 CONDITIONAL POSTERIOR SAMPLING UNDER STRUCTURED NOISE
We are interested in posterior sampling under structured noise. We recast this as a joint optimization problem with respect to the signal x and noise n given by:
(x, n) (x, n|y) Py)x,N(y)xx,)) px (x) (n)
(13)
4

---

Under review as a conference paper at ICLR 2023
Solving inverse problems using diffusion models requires conditioning of the diffusion process the observation y, such that we can sample from the posterior Pxjy (x. n Therefore, we construct a joint conditional diffusion process (x,, n,ly) m (e(o.)) in turn producing a joint conditional reverse- time SDE:
d(,, n.) = [f(t) g(t)) x.. log,(x, ngly)] + g(t)dw;
(14)
We would like factorize the posterior using our learned unconditional score model and tractable measurement model, given the joint formulation. Consequently, we construct two separate dif- fusion processes, defined by separate score models but entangled through the measurement model Py|x,N (y)x,v(y/xx, n). In addition to the original score model sg(x, f), we introduce second score model 86 (ne, () = Vn, (oggx((n) parameterized by weights é, to model the expressive noise component n. These two score networks can trained independently on datasets for x and n, respectively, The gradients of the posterior with respect to and n are now
using the objective in equation
given by:
vx, logp(xi, n.ly) = Vx, (x:)(x:)= + Vx, (X, ne)
sp- (xx, 1) Vx,
ne),
(15)
nn logp(x;, n.ly) n. log p(,) + n,, log (( n.) Ser (nt, t) + Vn, log p( ne),
(16)
where y; is a sample from p(y:ly), and (y: he is an additional stochastic process that essentially corrupts the observation along the SDE trajectory together with x,. As p(y, (y) is tractable, easily compute yr = (tl + ((t)Az, using the reparameterization trick with z E Rd ~ N(0, I). see [Song al.) (20215). Subsequently, the approximation in equation 15 and equation T6 can be
substituted for the conditional score in equation [4] resulting in two entangled diffusion processes:
{ dx; = [f(t)x gg()² {so- (xe, f) + V x; 1og |x, nz)) Jt g dn; [f(t)n: g(t)² {sy. (n, t) + Vn, log ( n:)3 dt + g(f)
g(t)dwxx
(17)
which allows us to perform posterior sampling for both the signal, such that x = xo ~ (xol)) as well as the structured noise, such that n ng ~ PNIY (noly).
To solve the approximated joint conditional reverse-time SDE, resort the aforementioned iterative scheme Section 2.2. however, now incorporating the observation via a data-consistency step. This is done by taking gradient steps that minimize the & norm between the true observation and its model prediction given current estimates of x and n. Ultimately, this results solutions that are consistent with the observation y and have high likelihood under both prior models. The data-consistency update steps for both x and n are derived as follows:
& = X; Vx, log ni;)
á; = h. Vn, log logg(yr/kk, ii.) = h. Vn. - (Ax; + i
-; Vx,liS - (A&, + it)ll
X; - AATAAR P. fz), (18)
n, u(Ax, ý: ni); (19)
where the time difference between two steps At 1/T and À and are weighting coefficients for the signal and noise gradient steps, respectively. An example of the complete sampling algorithm is shown in Algorithm [ which adapts the Euler-Maruyama sampler (Song et al. 2020) to jointly find the optimal data sample and the optimal noise sample while taking into account the measurement model in line and 8 using the outcome of equation 18 and equation [9 respectively. Although show the Euler- Euler-Maruy Maruyama method, our addition is applicable to a large family of iterative sampling methods for score-based generative models.
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Joint conditional posterior sampling with Fuler-Maruyama method
Require: T, s0: SΦ. A, y.y
X1 π (x), in (n), At +-
10
- Al t X1 f(t)x,Lt
X f RI- + g(1) (8), f)
11
for: to t i+1
12
2 2~ a~,((
13
X,- F XI- g(t)
yz po, (yrly)
át + -n -t))itt 15
11 Data consistency steps
á -i g(t) (i,,
21- f 8; (A 81 fi:)
z z~ (0,1)
D f ñ. MMAA: -ÿ: in)
ár - gt) Atz
return: S0
19 end
3.2 TRAINING AND INFERENCE SETUP
For training the models, 2020 in combination with the Adam optimizer and a learning rate of 5e-4 until convergence. For simplicity, exponential moving average (EMA) filter on the network weights is applied. Given two separate datasets, one for the data and one for the structured noise, two separate score models can be trained independently. This allows for easy adaptation of our method, since many existing trained score models can be reused. Only during inference, the two priors are combined through the proposed sampling procedure as described in Algorithm [ using the adapted Fuler- Maruyama sampler. We use the variance preserving (VP) SDE (3o = 0.1, 31 = 7.0) (Song et al., (020) to define the diffusion trajectory. During each experiment, we run the sampler for = 600 iterations.
the NCSNv2 architecture introduced in SSnn & Ermon
4 EXPERIMENTS
All models are trained on the CelebA dataset (Liu et al. 2015) and the MNIST dataset with 10000 and 27000 training samples, respectively. We downsize the images to 64 x 64 pixels. Due to computational constraints, we test on a randomly selected subset of 100 images. We use both the peak signal-to noise ratio (PSNR) and structural similarity index (SSIM) to evaluate results.
4.1 BASELINE METHODS
The closest to our work is the flow-based noise model proposed by Whang et al. (2021). discussed in Section 2.1.1 which will serve to make it more competitive we moreover replace the originally-used RealNVP (Dinh et al. 2016) with the Glow architecture (Kingma Dhariwal, 2018). Glow is widely used flow model highly inspired by RealNVP, with the addition of 1 x convolutions before each coupling layer. We use the exact implementation found in Asim al. 2020), with flow depth of K 18, and number of levels L = 4, which has been optimized for the same CelebA dataset used in this work and thus should provide a fair comparison with the proposed method.
main baseline. To boost the performance of this baseline and
Secondly, GANs as discussed in Section 2.1 are used comparison. We train DCGAN (Rad- ford et al. 2015), with a generator latent input dimension of 1 100. The generator architecture consists of 4 strided 2D transposed convolutional layers, having 4 4 kernels yielding feature maps of 512, 256, 128 and 64. Each convolutional layer followed by a batch normalization layer and ReLU activation.
Lastly, depending on the reconstruction task, classical non-data-driven methods are used as a com- parison. For denoising experiments, (Dabov et al. 2006), and compressed sensing experiments, LASSO with wavelet basis (Tibshi-
the block-matching and 3D filtering algorithm (BM3D)
rani 1996).

---

Under review as a conference paper at ICLR 2023
11.57
1333
17.5
0,49
22.91 0.83
27.26 0.87
1.0
30
10:45
027
13:07
0.22
19.98
0.82
22,85
0.84
1.0
0.8
0.8
PSNR
SSIM
20
0.4
0.4
0.2
0.2
10
0.0
BM3D
GAN
FLOW
DIFFUSION
BM3D
GAN
FLOW
DIFFUSION
(a) CelebA
(b) Of -fitoribtititttionn data
Figure I: Quantitative results using PSNR (green) and SSIM (blue) for the removing MNIST digits
experiment on 64 x 64 images of the (a) CelebA and (b) out-of-distribution datasets.
Except for the flow-based method of Whang al. (021), none of these methods explicitly model the noise distribution. Still, they are a valuable baseline, as they demonstrate the effectiveness of incorporating learned structured noise prior rather than relying simple noise priors.
Automatic hyperparameter tuning for optimal inference was performed for baseline methods on small validation set of only images. For both GAN and flow-based methods, anneal the step size during inference based on stagnation of the objective.
4.2
RESULTS
4.2.1
REMOVING MNIST DIGITS
For comparison with Whang et al. 2021 we recreate an experiment introduced in their work, where MNIST digits are added CelebA faces. Moreover, the experiment casily reproducible as both CelebA and MNIST datasets are publicly available. The corruption process is defined by y = 0.5 XCelebA + 0.5 DMNIST. In this experiment, the score network 8g is trained on the MNIST dataset. Fig. Ta shows quantitative comparison of our method with all baselines. Furthermore, random selection of test samples is shown in Fig. for qualitative analysis. Both our method and the flow-based method are able to recover the data, and remove most of the structured noise. However, more details are preserved using the diffusion method. contrast, the flow-based method cannot completely remove the digits in some cases and is unable reconstruct some subtle features present in the original images. Furthermore, we observe that for the flow-based method, initialization from the measurement necessary to reproduce the results in Whang et al. (2021) since random initial- ization does not converge. The GAN method is also able to remove the digits, but cannot accurately reconstruct the faces as it is unable project the observation onto the range of the generator. Simi- larly, the BM3D denoiser fails to recover the underlying signal, confirming the importance of prior knowledge of the noise in this experiment. The metrics in Fig. support these observations. See Table the extended results.
Additionally, we expose the methods in a similar experiment duttof-disttibution (OoD) data. The images from this dataset not found in the CelebA dataset, which is the data used for training the models. In fact, the out-of-distribution data is generated using the stable-diffusion text-to-image model Rombach et al. (2022). We use the exact same hyperparameters as during the experiment on the CelebA dataset. Quantitative and qualitative results are shown in Fig. [b and Fig. respectively. Similarly to the findings Whang et al. (2021); Asim et al. (2020), the flow-based method is robust to OoD data, due to their inherent invertibility. We empirically show that the diffusion method also resistant to OoD data in inverse tasks with complex noise structures and based methos. Unsurprisingly, the GAN method performs even more poorly when subjected to OoD data. More experiments, covering different inverse problem settings can be found in Appendix A
outperforms the flow-

---

Under review as a conference paper at ICLR 2023
Ground Truth
Noisy Ingui
DIFFUSION
FLOW
GAAN
BMBD
Figure 2: Results for diffusion-based method compared to the baselines; FLOW (Whang al,
2021]. GAN (Bora et al. 2017). and BM3D (Dabov et al. 2006) on the removing MNIST digits
experiment 64 x 64 images of the CelebA dataset.
Ground Truth
Imput
DIFFUSION
FLOW
GAN
S
BMED
Figure 3: Results for our diffusion- based method on the removing MNIST digits experiment on an
of distribution dataset, generated using stable diffusion (Rombach et 2022).
8

---

Under review as a conference paper at ICLR 2023
4.2.2 PERFORMANCE
To highlight the difference in inference time between our method and the baselines, benchmarks performed on single 12GBytes NVIDIA GeForce RTX 3080 Ti, see Table Appendix [B.2] Although this is not an extensive benchmark, a quick comparison of inference times reveals a 50x difference in speed between ours and the flow-based method. All the deep generative models need approximately an equal amount of iterations (I' xs 600) to converge, However, for the same model- ing capacity, the flow model requires substantial higher amount of trainable parameters compared to the diffusion method. This is mainly due to the restrictive requirements imposed on the architec- ture to ensure tractable likelihood computation. It should be noted that no improvements to speed up the diffusion process, such as CCDF (Chung et al.) (022b) are applied for the diffusion method, giving room for even more improvement in future work.
DISCUSSION AND CONCLUSIONS
In this work, we present framework for removing structured noise using diffusion models. Our work provides an efficient addition to existing score-based conditional sampling methods incor- porating knowledge of the noise distribution. We demonstrate our method on natural and out-of- distribution data and achieve increased performance over the state-of-the-art and established con- ventional methods for complex inverse tasks. Additionally, the diffusion based method is substan- tially easier to train using the score matching objective compared to other deep generative methods and furthermore allows for posterior sampling.
While our method considerably faster and better in removing structured noise compared to the flow-based method (Whang al. [022], it is not ready (yet) for real-time inference and still slow compared to GANS (Bora et al.) 2017) and classical methods. Luckily, research into accelerating the diffusion process are well on their way. In addition, although a simple sampling algorithm was adapted in this work, many more sampling algorithms for score-based diffusion models exist, hyperparameters. For example, the predictor-corret correcto (PC)
each of which introduces a new sampler has been shown to improve sample quality (Song et al., 2020). Future work should explore this wide increase in design space to understand limitations and possibilities of more sophisticated sampling schemes in combination with the proposed joint diffusion method. Furthermore, the range of problems to which we can apply the proposed method, can be expanded into non-linear likelihood models and extend beyond the additive noise models.
Lastly, the connection between diffusion models and continuous normalizing flows through the neu- ral ODE formulation (Song et al. 202 is not investigated, but greatly of interest given the com- parison with the flow-based method in this work.
REPRODUCIBILITY STATEMENT
All code used to train and evaluate the models as presented in this paper can be found at https: sci scciennne//clr/223-j0000
// //anonymous . www.o..cc.eeeeeeeoooooo.c.o
ffusion Essentially, the codebase in
https://github. www.waoo-ooo//eee wwwp:/yangoooog///eee score_ aede_pytorrhhh of [Song et all (020 is used to train the acore-based diffusion networks, for both data and structured noise, independently. To implement the proposed inference scheme, the lines in Algorithm [ should be adapted create a sampler that
includes both trained diffusion models. Details regarding the training and inference settings used to
reproduce the results in this work can be found in Section 3.2
REFERENCES
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Ap- plications, 12(3):313-326, 1982.
Muhammad Asim, Max Daniels, Oscar Leong, Ali Ahmed, and Paul Hand. Invertible generative models for inverse problems: mitigating representation error and dataset bias. In International Conference on Machine Learning, pp. 3999-9999999 PMLR, 2020.

---

Under review as a conference paper at ICLR 2023
Arpit Bansal, Eitan Borgnia, Mong-Min Chu, Jie S Li, Hamid Kazemi, Furong Huang, Micah Gold- blum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. arXiv preprint
2022. !
Amir Beck and Mare Teboulle. A fast iterative thenshol-t algorithm for linear inverse problems. SIAM journal on imaging sciences, 2(1):183 2009. [
Ashish Bora, Ajil Jalal, Eric Price, and Alexandros Dimakis. Compressed sensing using genera- tive models. In International Conference Machine Learning, pp. 537-546. PMLR, 2017.
15
Hyungjin Chung and Jong Chul Ye. Score-based diffusion models for accelerated mri. Medical Image Analysis, pp. 102479, 2022. [
Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. arXiv preprint orXiv:2206.00941 2022a.
Hyungjin Chung. Bycongsu Sim, and Jong Chul Ye. Come-closer-iifus---aae ditional diffusion models for inverse problems through stochastic contraction. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22413-12422
Accelerating con-
2022b. 19
Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising with mlock-matching and 3d filtering. In Image processing: algorithms and systems, neural networks, and machine learning, volume 6064, pp. 354-365. SPIE, 2006. 6 15
Giannis Daras, Yuval Dagan, Alex Dimakis, and Constantinos Daskalakis. Score-guided interme- diate level optimization: Fast langevin mixing for inverse problems. In International Conference on Machine Learning, pp. 4722-4753, PMLR, 2022a. 3
Giannis Daras, Mauricio Delbracio, Hossein Talebi, Alexandros G. Dimakis, and Peyman Milanfar.
Soft diffusion: Score matching for general corruptions, 2022b. |
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint (605:06600008803 2016. [6]
Yonina C Eldar and Gitta Kutyniok. Compressed sensing: theory and applications. Cambridge university press, 2012. D
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, (3(11):139-14) 2020. 3
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851 2020.
Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. Robust compressed sensing mri with deep generative priors. Advances in Neural Information Processing Systems, 34:14938-14955 2021a. [
Ajil Jalal, Sushrut Karmalkar, Alex Dimakis, and Eric Price. Instance-optimal compressed sens- ing via posterior sampling. In International Conference on Machine Learning, pp. 4709-4720.
PMLR, 2021b. 3
Bowen Jing, Gabriele Corso, Renato Berlinghieri, and Tommi Jaakkola. Subspace diffusion gener- ative models. arXiv preprini arXiv:2205.01440) 2022. 2
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-
based generative models. arXiv preprint Stti:2200.003666 2022. []
Bahjat Kawar, Gregory Vaksman, and Michael Elad. Snips: Solving noisy inverse problems stochas- tically. Advances in Neural Information Processing Systems, 34:21757-21769, 2021.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions.
Advances in neural information processing systems, 31, 2018. 6
10

---

Under review as a conference paper at ICLR 2023
Ziwei Liu, Ping Luo, Xiaogang Wang. and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015. 6
Calvin Luo. Understanding diffusion models: A unified perspective.
arXiv preprint
arXiv:2208. 11970, 2022. 0
Stéphane Mallat. wavelet tour of signal processing. Elsevier, 1999.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.0643: 2015. 6
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06123 2022. [
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the JEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
8 13
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2021. 2
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score- woreebbaed generative models.
Advances in neural information processing systems, 33: 13:12438-1244 2020. 6
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. Interna- tional Conference on Learning Representations, 2020.
5
Yang Song. Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score- based diffusion models. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 1415-1428.
Curran Associates, Inc., 2021a. g
Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with score-based generative models. In International Conference on Learning Representations, 2021b. 5
Robert Tibshirani.
Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological),
1996. 15
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 34:1 14411128--11000 1302, 2021.
Pascal Vincent. connection between score matching and denoising autoencoders. Neural compu- 2011.
lation,
Xinyi Wei, Hans Gorp, Lizeth Gonzalez-Carabane Daniel Freedman, Yonina C Eldar, and Ruud JG van Sloun. Deep unfolding with normalizing flow priors for inverse problems. JEEE Transactions Signal Processing, 70:2962-2971, 2022. [
Jay Whang, Qi Lei, and Alex Dimakis. Solving inverse problems with a flow-based noise model. In International Conference on Machine Learning, pp. 11146-11151 PMLR, 2021. 3
6
15
11

---

Under review as a conference paper at ICLR 2023
12.03
0.38
18.
0.53
2406 0078
25.52 0,82
1.0
30
1.0
0.8
25
0.8
25
PSNR
20
SSM
PSNR
20
0.6
0.4
0.4
0.2
0.2
10
LASSO
GAN
FLOW
DIFFUSION
LASSO
GAN
FLOW
DIFFUSION
(a) CelebA
(b) Out-of-distribun data
Figure 4: Quantitative results using PSNR (green) and SSIM (blue) for the compressed sensing with
sinusoidal noise experiment 64 64 images of the (a) CelebA and (b) ant-of-distribution datasets.
A ADDITIONAL EXPERIMENTS
The following section explores additional inverse problems with compressed sensing and structured noise. The goal is to show the performance of the proposed method in a variety of settings.
A.1 STRUCTURED NOISE WITH COMPRESSED SENSING
The corruption process is defined by y Ax + nsne with a random Gaussian measurement matrix A E Rgdd and noise with sinusoidal variance σы exp(sin 16 )) for each pixel k. The subsampling factor is defined by the size of the measurement matrix d/m. In this experiment, the score network 8y is trained on a dataset generated with sinusoidal noise samples Naine In Fig. 5 the results of the compressed sensing experiment and the comparison with the baselines an average standard deviation σ = 0.2 and subsampling of factor d/m = 2. Given the same hyperparameter settings, we repeat the experiment on the out-of-distribution (OoD) dataset, shown in Fig. 6 Similar to the results found in Section 4.2.1 the diffusion method is more robust to the shift distribution and is able to deliver high quality recovery under the structured noise setting. In contrast, the flow-based method under- mnder-performs when subjected to the OoD data. Quantitative results on both CelebA and OoD are found Fig, 4 as well as Table and Table respectively.
shown for
A.2 REMOVING SINUSOIDAL NOISE
The corruption process is defined by y = Nsie where the noise variance 0 exp(sin( 2mr 16
follows sinusoidal pattern along each row of the image k. In this experiment, the score network & trained dataset generated with 1D sinusoidal noise samples Maine- See Fig. [ a comparison of our method to the flow-based method for varying noise variances. Both methods perform quite well, with the diffusion method having a slight edge. A visual comparison in Fig. however, reveals that the diffusion method preserves more detail in general.
12

---

Under review as a conference paper at ICLR 2023
Ground Truih
DIFFUSION
FLOW
GANN
LASSO
Figure 5: Comparison results from our diffusion method compared to the baselines on the com-
pressed sensing with sinusoidal noise experiment with d/m 2, oN 0.2 on 64 64 images of the CelebA dataset.
Ground Trulh
DIFFUSION
FLOW
GAN
LASSO
Figure 6: Results for our diffusion based method on the compressed sensing with sinusoidal noise experiment on an out-of-distribution dataset, generated using stable diffusion (Rombach et al. 2022).
13

---

Under review as a conference paper at ICLR 2023
Ground Truih
Noisy Imput
DIFFUSION
FLOW
GANN
BMBD
Figure 7: Comparison of results from our diffusion method compared the baselines on the remov-
ing sinusoidal noise experiment with = 0.2 on 64 x 64 images of the CelebA dataset.
FLOW
24
DIFFUSION
22
PSNR
18
16
14
0.1
0.2
0.3
Avg. Noise dy
Figure 8: Comparison of PSNR values for varying sinusoidal noise variances. Shaded areas repre- sent the standard deviation the metric.
14

---

Under review as a conference paper at ICLR 2023
B EXTENDED RESULTS
B.1
METRICS
Table 1: Results for the experiments and different methods the CelebA dataset. *Ours,
Whang et al. (2021). Bora et al. 2017)
Dabov et al. (2006
Tibshirani (996).
MNIST
CS + sine noise
PSNR
SSIM
PSNR
SSIM
"DIFFUSION
27.26 + 1.925
0.865 0.039
25.51 1.040
0.823 + 0.044
FFLOW
22.90 # 1.214
0.827 0.051
24.96 2
0.779 - 0.082
GAAN
17.50 # 1.404
0.486 0.099
18.90 1.343
0.529 + 0.084
BBM3
11.56 # 1.879
0.326 + 0.059
LASSO
12.93 1.819
0.284 + 0.037
Table 2: Results for the experiments and different methods on the out-of auttof-distribution (OoD) dataset.
*Ours, Whang al. (2021). Bora et 2017). Dabov et al. 20006)
Tibshirani [996]
MNIST
CS
sine noise
PSNR
SSIM
PSNR
SSIM
DDIFFUSION
22.87 + 4.581
0.842 + 0.110
22.90 + 1.568
0.823 + 0.082
FFLOW
19.98 + 1.946
0.824 + 0.081
19.85 4.840
0.608 + 0.176
2GAN
13.06 # 1.788
0.218 0.088
12.39 1.693
0.159 = 0.070
8BM3D
10.44 # 1.446
0.274 0.069
LLASSO
11.62 + 1.473
0.336 + 0.057
B.2 COMPUTATIONAL PERFORMANCE
Table Inference performance benchmark for all methods.
*Ours, Whang et al. 2022)) Bora et al. 2017). Dabov et al. 2006
Tibshirani 1996).
Number of trainable parameters
Inference time image [ms]
*DIFFUSION
8.9M
1292
[2.153 / it]
FLOW
25.8M
61853
[103.1 / it]
GAN
3.9M
59
[0.059 / it]
BM3D
28.5
15

---

Under review as a conference paper at ICLR 2023
C PSEUDO-CODE
In this section, we provide pseudo-code for the proposed joint conditional diffusion sampler with the Euler-Maruyama sampling algorithm as basis. Furthermore, we use the SDE formulation for the diffusion process which is denoted as an sde object with dri ft, diffusion and marginal_proo methods. The latter computes the mean and standard deviation of the diffusion transition ker- nel at certain time t. Lastly, there are two trained score networks (NCSNv2) score_data and score noesen for the data and structured noise respectively.
def joint_cond Seint_cond_diifuueee cetffusion_ssmpleeee
lambda_coeff, mu_coeff, num_steps):
dt 1/num_steps
x random... normal (y. shape)
n random... normal (y. shape)
for t in linspace 0, num_steps): corrupt observation along the diffusion process mean, std = sde marg (y, t)
y_hat = mean std random normal (y. shappe
# data consistency step for (data)
x lambda_coeff A.T (A x hat n)
# data consistency step for n (noise)
mu_coeff * (n y-hat A ( x)
reverse diffusion step z random. random.normal (x shape) shapp)
x (data)
x_hat sde dt)
* dt
<_hat hat sde. diffusion(t)*** **) acore_data(K, t) dt
x_hat x_hat sde, diffusion(t) * sqrt(dt) 2
x x_hat
# reverse diffusion step for (noise) z random.normal (n. shape)
hat = n sde. (t) n dt
n_hat = n_hat sde. diff
score_noise (n, t) dt
n_hat n_hat sde. diffuifusi (t) sdrt(dt)
n_hat
return the denoised sample x return x
16


Reviews:
[{"review_id": "Vhz3Q_6u5gd", "reviewer": "Reviewer_Kqyc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use a score-based model for the task of removing structured noise. Specifically, the method uses a pretrained (unconditional) NCSNv2 model as the signal prior and uses modified update rule during sampling that incorporates the score of a separate noise model.  Experimental results show that score-based models achieve superior performance to existing methods based on GAN and normalizing flow.", "summary_of_the_review": "This paper proposes to combine two score-based models -- signal prior and noise prior -- for the task of removing structured noise.  The proposed method outperforms existing deep generative priors on the task of MNIST digit removal, and while the approach is simple, this discovery has some value.  That said, the experiments and analysis are lacking.   ", "strength_and_weaknesses": "**Strengths**\n* Clear exposition and motivation.\n* Problem is relevant to the field as inverse problems encompass a large family of problems and is an active area of research.\n\n**Weaknesses**\n* Experimental comparison is done on a single dataset -- especially when compared to the main baseline (Whang et al.), which considered various types of noises.\n* Unclear whether the benefits are due to the difference in the model class, or other factors (e.g. hyperparameters, network size, etc).  While I don't doubt that score-based prior could outperform GAN/flow priors, it would have been nice to see some effort on trying to decouple the effects of model class vs. other architectural/implementation details.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5h8wKliWO36", "reviewer": "Reviewer_kwKg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new method to solve a large class of inverse problems using diffusion models. The approach is to jointly learn both a source data diffusion model and a noise diffusion model, from only a training set of noisy observations. It is assumed that the noise is added to the source signal in a known manner, but a highlight here is that the noise is allowed to be structured and not assumed to be i.i.d.\n\nA method to jointly learn the two diffusion models is proposed. The method is then tested by using CelebA as the source signal and then adding MNIST digits as a noise model. In experiments, the diffusion model is shown to outperform alternative solutions that use GANs and and flow models. ", "summary_of_the_review": "This is exciting work, and an excellent contribution to ICLR.", "strength_and_weaknesses": "Strengths:\n- The paper is clearly written and well-organized.\n- The paper addresses a broad class of problems. Scientists in other fields (e.g. astronomy, physics) could find this very useful. \n- The method proposed is not obvious, and a significant contribution. \n- The experiments are well done. \n\nWeaknesses:\n- None", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "4u9xVJVRBXI", "reviewer": "Reviewer_uwBZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes an approach for solving the inverse problem via diffusion models. Particularly, the work considers structured noise \nthat contaminates the observations. To solve the problem, another line of the diffusion process that describes noise is used except \nfor the image generative process. The score models in both image generation and noise generation are pretrained, and then applied\nto two reverse stochastic differential equations to sample images and noises. During inference, a data-consistency step is added.\nExperiments about removing MNIST digits and compressive sensing are presented to verify the effectiveness of the proposed\nmethods.\n", "summary_of_the_review": "The work is relatively easy to follow and the idea is clear. However, the clarity to embed the structural noise is not clear. Additionally, the necessity of using the diffusion model is not convincing.", "strength_and_weaknesses": "Strength: \n1. The work additionally takes the noise structure into consideration and proposes a noise diffusion process together with an image \ndiffusion process to solve the inverse problem. \n2. The method that samples image and noise from joint conditional posterior is rational and interpretable, and also seems easy\nto implement.\n\nWeakness\n1. The right side of Eq. (14) does not contain $n_t$.\n2. In algorithm1, step 7 and step 8 seem to be covered by step 10 and step 15, respectively.\n3. The necessity of using the diffusion model to describe the structural noise is not convincing. Why is it better than the other generative model, such as GAN, VAE and Flow?\n4. How the structural noise is embedded into the diffusion model? This point is not clear.\n5. It is not clear what the training data is when training the second score function $s_{\\phi}(n,t)$. \n6. I wonder how the proposed method behaves compared with non-generative methods for removing MNIST digits and compressive sensing, and what is the superiority of the proposed method.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "1T1JxuUWdH", "reviewer": "Reviewer_o14W", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to remove structured noise from noised images using a diffusion model that describes the joint distribution of the clean data samples and the structured noise. Using an additional score model for the noise, an approximate score for the joint distribution (for all time $t$) is derived. Experimental results show that the proposed method outperforms the previous benchmarks based on normalizing flows and GANs.", "summary_of_the_review": "In conclusion, I have raised several concerns regarding the technical soundness and empirical results of the proposed method, which prevents me from giving this paper a higher rating at this time. I will consider increasing my rating if the authors address my concerns and clarify my misunderstandings.", "strength_and_weaknesses": "Strength:\n\n- Removing structured noise is an interesting task and the idea of the paper is fashionable.\n- The experimental results look promising.\n\nWeaknesses:\n\n- Technical soundness & Clarify: I found the technical part of the paper hard to follow and a little bit misleading.\n    * At the beginning of Section 3.1, it seems that this work tries to build a conditional distribution $p(\\mathbf{x},\\mathbf{n}|\\mathbf{y})$ so that the inversion problem can be solved by just sampling from this distribution or taking its MAP estimation. And then to build this sampling process (or to model this distribution), a diffusion process is imposed on the \"joint space\" of $(\\mathbf{x},\\mathbf{n})$, while everything is conditioned on the noise observation $\\mathbf{y}$. However, in Eq.(14), the score is conditioned on $\\mathbf{y}_t$, which is further explained below in Eq.(16) as a corrupted version of $\\mathbf{y}$. What is $\\mathbf{y}_t$ here? Why does the conditioning depend on time $t$? What is the exact definition of $p(\\mathbf{y}_t|\\mathbf{y})$ and why is it related to the diffusion process of $\\mathbf{x}$ and $\\mathbf{n}$?\n    * In Eq.(15&16), why does the second term of RHS use a sample $\\hat{\\mathbf{y}}_t$? Is it a good approximation?\n    * In page 3 it mentioned $s_{\\phi}(\\mathbf{n},t)\\approx\\nabla_{\\mathbf{n}}\\log p_N(\\mathbf{n})$. Is the score model $s_{\\phi}(\\mathbf{n},t)$ actually independent with time $t$?\n    * It is not very clear how are the two score networks trained. Is $s_{\\phi}(\\mathbf{n},t)$ trained jointly with $s_{\\theta}(\\mathbf{x},t)$?\n- Experiments: It is mentioned in Section 4.1 that the baseline results are improved to be more competitive. However, in Whang et al. (2021) the results seem to be better than what is reported in this paper (see Figure1&2 in Whang et al. (2021)). The inconsistency seems to make the results here questionable. Can you please provide more details on this?\n\n\nMinors:\n- In Eq.(13) the “proportional to” is not correct as the RHS uses log probabilities.\n- In Eq.(14) the drift term should be $f(t)(\\mathbf{x}_t, \\mathbf{n}_t)$.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject