Diffusion Posterior Sampling for General Noisy Inverse Problems
Authors: Hyungjin Chung
Jeongsol Kim
Michael Thompson Mccann
Marc Louis Klasky
Jong Chul Ye
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Diffusion model
Inverse problem
Posterior sampling

Abstract: Diffusion models have been recently studied as powerful generative inverse problem solvers, owing to their high quality reconstructions and the ease of combining existing iterative solvers. However, most works focus on solving simple linear inverse problems in noiseless settings, which significantly under-represents the complexity of real-world problems. In this work, we extend diffusion solvers to efficiently handle general noisy (non)linear inverse problems via the Laplace approximation of the posterior sampling. Interestingly, the resulting posterior sampling scheme is a blended version of diffusion sampling with the manifold constrained gradient without a strict measurement consistency projection step, yielding a more desirable generative path in noisy settings compared to the previous studies. Our method demonstrates that diffusion models can incorporate various measurement noise statistics such as Gaussian and Poisson, and also efficiently handle noisy nonlinear inverse problems such as Fourier phase retrieval and non-uniform deblurring.

DIFFUSION POSTERIOR SAMPLING FOR GENERAL NOISY INVERSE PROBLEMS
Hyungjin Chung"),2 Jeongsol Kim"!, Michael Mccann'', Mare L. Klasky² & Jong Chul Ye! KAIST, 2 Los Alamos National Laboratory
jeongsol, jong. ye) @kaist.
{mecann, mklasky @eaann..... gov
ABSTRACT
Diffusion models have been recently studied as powerful generative inverse problem solvers, owing their high quality reconstructions and the ease of combining exist- ing iterative solvers. However, most works focus on solving simple linear inverse problems in noiseless settings, which significantly under-represents the complexity of realdworld problems. In this work, we extend diffusion solvers to efficiently han- dle general noisy (non)linear inverse problems via approximation of the posterior sampling. Interestingly, the resulting posterior sampling scheme is a blended ver- sion of diffusion sampling with the manifold constrained gradient without a strict measurement consistency projection step, yielding a more desirable generative path noisy settings compared the previous studies. Our method demonstrates that diffusion models can incorporate various measurement noise statisties such as Gaus- sian and Poisson, and also efficiently handle noisy nonlinear inverse problems such as Fourier phase retrieval and non-uniform deblurring. Code is available at https posterior -sarrirrr--ampp)p ing
7/github. wttp://cccccmm/2/222/2/22
INTRODUCTION
Diffusion models learn the implicit prior of the underlying data distribution by matching the gradient of the log density (i.e. Stein score; V. log p(x)) (Song et al. (0216)) The prior be leveraged when solving inverse problems, which aim to recover æ from the measurement y. related through the forward measurement operator AA and the detector noise When know such forward models, one can incorporate the gradient of the log likelihood (i.e. v log p(y|z)) in order to sample from the posterior distribution p(æly)) While this looks straightforward, the likelihood term is in fact analytically intractable in terms of diffusion models, due to their dependence on time t. Due to its intractability. one often resorts projections onto the measurement subspace (Song et al.] 20216] Chung et al., 2022b Chung & Ye, 2022 Choi et al. (021). However, the projection-type approach fails dramatically when 1) there is noise in the measurement, since the noise is typically amplified during the generative process due to the ill-posedness of the inverse problems; and 2) the measurement process is nonlinear.
One line of works that aim to solve noisy inverse problems run the diffusion in the spectral do- main (Kawar et al. 2021 2022) so that they can tie the noise the measurement domain into the spectral domain via singular value decomposition (SVD). Nonetheless, the computation of SVD is costly and even prohibitive when the forward model gets more complex. For example, Kawar et al. (2022) only considered seperable Gaussian kemels for deblurring, since they were restricted the family of inverse problems where they could effectively perform the SVD. Hence, the applicability of such methods is restricted, and it would be useful to devise method solve noisy inverse problems without the computation of SVD. Furthermore, while diffusion models were applied to various inverse problems including inpainting (Song al.
2021b Chung et 2022b Kawar et
2022,Chung et al. 2022a), super-resolution (Choi et al.
2021 Chung et al. 2022b Kawar et al.
2022). colorization (Sogg et al. [2021]] Kawar et al. 2022
Chung et al. 2022ap. compressed-sensing
MRT (CS-MRI) (Song et al. 2022; Chung & Yel 2022) Chung et al. 2022b), computed tomography
(CT) (Song et
Chung et al. 2022a), etc., to our best knowledge, all works so far considered
linear inverse problems only, and have not explored nonlinear inverse problems.
"Joint first authors

---

Linear
(4) Impainting
Caussian deblur
(0) Buper-reoolutio
Non-linear
Phase retrivaal
Non-ariform debiur
Figure 1: Solving noisy linear, and nonlinear inverse problems with diffusion models. Our recon- struction results (right) from the measurements (left) are shown.
In this work, devise method to circumvent the intractability of posterior sampling by diffusion models via a novel approximation, which can be generally applied to noisy inverse problems. Specifically, we show that our method can efficiently handle both the Gaussian and the Poisson measurement noise. Also, our framework easily extends to any nonlinear inverse problems, when the gradients can be obtained through automatic differentiation. We further reveal that a recently proposed method of manifold constrained gradients (MCG) (Chung et al.) (202aa) is special case of the proposed method when the measurement is noiseless. With geometric interpretation, we further show that the proposed method is more likely to yield desirable sample paths in noisy setting than the previous approach (Chung et al., 2022a). In addition, the proposed method fully runs on the image domain rather than the spectral domain, thereby avoiding the computation of SVD for efficient implementation. With extensive experiments including various inverse cubblems-inpainting super-resolution GGaussian/motion/non unnifrrm) deblurring, Fourier phase retrieval-we show that our method serves as a general framework for solving general noisy inverse problems with superior quality (Representative results shown Fig.
BACKGROUND
2. SCORE-BASED DIFFUSION MODELS
Diffusion models define the generative process the reverse of the noising process. Specifically, Song et al. (2021b) defines the Itō stochastic differential equation (SDE) for the data noising process (i.e. forward SDE) (t), t [0, T), (t) E Idd Vt in the following form
3(t)
dæ
xdt + B(t)dw,
(1)
2
where B(t) R. - R is the noise schedule of the process, typically taken be monotonically increasing linear function of (Ho et al., (020). and w is the standard d-dimensional Wiener process. The data distribution is defined when
i.c. =0) Pdata, and a simple, tractable distribution (e.g. isotropic Gaussian) is achieved when t = T, i.e. (T) ~ I).
Our aim is to recover the data generating distribution starting from the tractable distribution, which can be achieved by writing down the corresponding reverse SDE of (I) (Anderson) 1982):
3(L) 2
dx
V(t) m. log (x.) dt + B(t)dw.
(2)
In this work, consider the variance preserving (VP) form of the SDE (Song al. 2021b) which is equivalent Denoising Diffusion Prohabilistic Models (DDPM) (Ho et al. 2020).
2

---

where dt corresponds to time rumning backward and dw the standard Wiener process running backward. The drift function now depends on the time-dependent score function V Va, log Pa 2+ which is approximated by neural network 8, trained with denoising score matching (innent 201
0" = arg min (et.dfteeeaaaa()ppp((())00000000
(())) Vv, logp(z(() (0) (3)
where 0 is a small positive constant. Once 0* is acquired through one can use the ap-
proximation Va, log Pr())) = Sg. (æ,, , t) as a plug-in estimate to replace the score function in (2).
Discretization of (2) and solving using, e.g. Euler-Maruyama discretization, amounts to sampling from the data distribution p(æ:), the goal of generative modeling.
Throughout the paper, adopt the standard VP-SDE (i.e. ADM of Dhariwal & Nichol (2021) 1 or
Denoising Diffusion Probabilistic Models (DDPM) (Ho et al., 2020).. where the reverse diffusion variance which we denote by (t) is learned as in Dhariwal & Nichol (2021). ) In discrete settings
(e.g. in the algorithm) with N bins, we define a; x(tT/N), B - B(tT/N). and subsequently Og
3r, a; _ I[ ar following Ho et al. (2020).
2.2 INVERSE PROBLEM SOLVING WITH DIFFUSION MODELS
For various scientific problems, we have a partial measurement y that is derived from æ. When the mapping
is many-to-one, we arrive at an ill-posed inverse problem, where we cannot
exactly retrieve x. In the Bayesian framework, one utilizes p(x) as the prior, and samples from the posterior p(zly). where the relationship is formally established with the Bayes' rule: p(xly)
p(y (p)) /p(y)- Leveraging the diffusion model as the prior, it is straightforward to modify 2 to arrive at the reverse diffusion sampler for sampling from the posterior distribution:
3(t) (()) lox):(x;) + Vx, log log dt + v 3(Fddw 2
dæ
(4)
where we have used the fact that
Vz, log (x Va, lo,).(x.)) + Va, (y)xx)
(5)
In (4), we have two terms that should be computed: the score function Va, log (x.;), and the likelihood Va, lo (g/ (z.) To compute the former term involving Pt (z), we can simply use the pre-traine tre-trained score function sø-. However, the latter term is hard to acquire in closed-form due to the dependence on the time t, as there only exists explicit dependence between y and Tn-
Formally, the general form of the forward mode I can be stated as
A(xx) + n,
E R", x E Rd.
(6)
where A(-) Rdd IR is the forward measurement operator and n is the measurement noise, In the case of white Gaussian noise, n ~ (O, I²I). explicit form, p(yla)) ~ (N(y)A(xx)) o However, there does not exist explicit dependency between and æ,, as can be seen in the probabilistic graph from Fig. where the blue dotted line remains unknown.
In order to circumvent using the likelihood term directly, alternating projections onto the measurement subspace a widely used strategy (Song et al. [222b] [Chung & Ye 2022] Chung et al. 20226). Namely, one disregard the likelihood term in (4), and first take unconditional update with (). and then take projection step such that measurement consistency can be met, when assuming n O. Another line of work Jalal et al. 2021) solves linear inverse problems where A (x) & Ax
and utilizes an approximation Va, log Pe x) A (y-Aw), which is obtained when assumed to be Gaussian noise with variance o². Nonetheless, the equation is only correct when t while being wrong at all other noise levels that are actually used in the generative process. The incorrectness counteracted by a heuristic of assuming higher levels of noise as - T, such that
Va, log Pe (y)z))
A Aw) where {T) hyperparameters. While both lines of works aim +
'The approximation error comes from optimizzatiom/prraaettrzzzziooii error of the neural network. 'To be precise, when we have signal- dependent noise model (e.g. Poisson). we cannot write the forward model with additive noise. We shall still write the forward model with additive noise for simplicity, and discuss which treatments are required when dealing with signal-dependent noise later in the paper.
3

---

to perform posterior sampling given the measurements and empirically work well on noiseless inverse problems, it should noted that 1) they do not provide means to handle measurement noise, and 2) using such methods to solve nonlinear inverse problems either fails to work or is not straightforward to implement. The aim of this paper is to take a step toward a more general inverse problem solver, which can address noisy measurements and also scales effectively to nonlinear inverse problems.
DIFFUSION POSTERIOR SAMPLING (DPS)
3.1 APPROXIMATION OF THE LIKELIHOOD
Recall that no analytical formulation for p (x) exists. In order to exploit the measurement model p(y (20),, we factorize p(y (z:)) as follows:
plylar)
x
/ ply æo, x (o:
p xc))
p(an/2r))
/ p ao) z0)p((2) )dzo.
y
0t
where the second equality comes from that y and xa are conditionally independent on Te as shown in Fig. [ Here, p( (x:)) as was shown with blue dotted lines Fig. 2 is intractable in general. Note however, that for the case of diffusion models such as VP-SDE or DDPM, the forward diffusion can be simply represented by
Figure 2: Probabilistic graph. Black solid line: tractable, blue dotted line: in- tractable in general.
Ee Va(t) + /1 a())z)
N,, I),
(8)
so that we can obtain the specialized representation of the posterior mean as shown in Proposition through the Tweedie's approach (Efron 2011 Kim & Ye 2021). Detailed derivations can be found in Appendix [A]
Proposition 1. For the case of VP-SDE DDPM sampling. p(zolæx,) has the unique posterior mean al
to = E
Va(t) (z. + (l-((tt)) log (x.))
(9)
Remark 1. By replacing Va log p(x:) the posterior mean from p( xx) as:
in 9 with the score estimate sp- (xt),, we can approximate
xz + (1 -(())))) (xt, 1)).
(10)
Va(t)
In fact, the result is closely related to the well established field of denoising. Concretely, consider the problem of retrieving the estimate of clean Do from the given Gaussian noisy &t. A classic result of
Tweedie's formula [Robbins] 1992 Stein 1987 Efron 2017 Kim & 2027) states that one can retrieve the empirical Bayes optimal posterior mean Ty using the formula in (10
Given the posterior mean to that can be efficiently computed at the intermediate steps, our proposal to provide tractable approximation for p(y((x;) such that one can use the surrogate function to maximize the likelihood -iieldiiiing approximate posterior sampling. Specifically, given the
interpretation p(ylar; (r.)) = (aa-p((a)e..) [p(z)(=)) from (7), we use the following approximation:
(()l;)) ~ p(y)))), where ig : E|=, x.] = (mo-p((a)) [xg]
(1I)
implying that the outer expectation of p(y) over the posterior distribution is replaced with inner expectation of Do. In fact, this type of the approximation is closely related to the Jensen's 's inequality,
so we need the following definition to quantify the approximation error:
Definition 1 (Jensen gap (Gao et al. 2017 Simic 2008)). Let æ be random variable with dis- tribution p(z). For some finction that may or may not be convex, the Jensen gap defined
as
I a ~ p(x)) = B[[(=)) ((E(x)))
(12)
where the expectation is taken over p(m)
4

---

Algorithm DPS Gaussian
Algorithm DPS Poisson
Require: N, y: [Gi) {or)
Require: N. y. [C.) {
I: æv (N,,
I: 2v ~ (0, I)
2: for i = .V
do
2: for i N to to do
- so( i)
3:
+ (o
4: to yar
c + (1 - (,)
4:
to f Jol
0z + (1 ā,)8)
(0,
z ~ (0,
s/-1 f- -
mi 18t
20
+0:2
6:
x/-1 y
ii 19
i-n;
1-n,
#; +- -i-1 1-ai
1-a;
7: 2;-1 + أ;ت 4 mi
11
A(
7:
V 1/
A(
8: end for
8: end for
9: return
9: return Xu
The following theorem derives the flosed-form upper bound of the Jensen gap for the inverse problem from (6) when n NO, o² I):
Theorem 1. For the given measurement model 6 with n ~ (0, I²1), we have
p(y( = p(y(zo),
(13)
where the approximation error can be quantified with the Jensen gap, which is upper bounded by
J
d
1/20² [Va((z)]]
(14)
2
where | A=x()]]] max, V (x.((r)) and m1 == =I (ollp(mro(x.) dan-
Remark 2. Note that A(x)] is finite in most of the inverse problems. This should not be confused with the ill-posedness of the inverse problems, which refers to the unboundedness of the inverse operator A Accordingly, if m, is also finite (which is the case for most of the distribution in practice), the Jensen gap in Theorem 7 can approach to 0 as a - oc, suggesting that the approximation error reduces with higher measurement noise. This may explain why our DPS works well for noisy inverse problems. In addition, although we have specified the measurement distribution be Gaussian, we can also determine the Jensen gap for other measurement distributions (e.g. Poisson) in an analogous fashion.
By leveraging the result of Theorem we can use the approximate gradient of the log likelihood
an logp(y)a:)) (x)) V.,, poy (0))
(15)
where the latter is now analytically tractable, as the measurement distribution is given.
3.2 MODEL DEPENDENT LIKELIHOOD OF THE MEASUREMENT
Note that we may have different measurement models p (z)) for each application. Two of the most common cases in inverse problems are the Gaussian noise and the Poisson noise. Here, we explore how our diffusion posterior sampling described above can be adapted to each case.
Gaussian noise. The likelihood function takes the form
- A 1
p(y)
exp
20t
where n denotes the dimension of the measurement y. By differentiating p(y]a:)) with respect to ats using Theorem and 15). get
VR, (
v., ly A A())
where we have explicitly denoted to (o to emphasize that &o function of æ;. Con- sequently, taking the gradient Va, æ, amounts to taking the backpropagation through the network. Plugging in the result from Theorem Ioo (5) with the trained score function, we finally conclude that
V u. log (x) ~ sg. (æ,, t) Vva.
æ. ly A(ta)
(16)
where p _ 1/02 is set as the step size.
5

---

M
M3
M4
Mz
y=
M
y A(x)
M2
M,
M,
Mr
Mc
x
xt'
X
X
Vgllly
proj
5g"
Vxllly (R,))
(a) Geometry of Chung (2022a)
(b) Geometry of DPS
Figure 3: Conceptual illustration of the geometries of two different diffuuson processes. Our method
prevents the sample from falling off the generative manifolds when the measurements are noisy.
Poisson noise. The likelihood function for the Poisson measurements under the i.id. assumption is given as
ply (m)) II [A(x))), exp [[-A(
(17)
yj
where indexes the mcasurement bin. most cases where the measured values are not too small, the model can be approximated by Gaussian distribution with very high accuracy Namely,
(y) - [A(xx))l))
p
+
II
exp
(18)
2n [A(xo))s
2[A(xx));
J
(y) [A( (A(())));i" )1;)
exp
(19)
Bry,
2y;
j=1
where we have used the standard approximation the shot noise model [A(xz)]; = yj to arrive at the last equation (Kingston 2013). Then, similar to the Gaussian case, by differentiation and the use of Theorem we have that
Va, logp(y/æz.) PV æ: Alz
[A]s - 1/2y,,
(20)
where ||all a" Aa, and we have included to define the step size in the Gaussian case. We can summarize our strategy for each noise model as follows:
Væ, log (e sa (Ie, t) pv æ. ly A(
(Gaussian)
(21)
RR, log Pe (g) ~ 80- (a+,, t) V x, ly A ll
(Poisson)
(22)
Incorporation of (16) (2)) into the usual ancestral sampling (Ho al., 2020) steps leads
Algorithm 1 Here, we name our algorithm Diffusion Posterior Sampling (DPS), as we construct our method order to perform sampling from the posterior distribution. Notice that unlike prior methods that limit their applications linear inverse problems A(x) & Aa, our method is fully general in that we can also use nonlinear operators A(-). To show that this is indeed the case, in experimental section take the two notoriously hard nonlinear inverse problems: Fourier phase retrieval and non-uniform deblurring, and show that our method has very strong performance even in such challenging problem settings.
Geometry of DPS and connection to manifold constrained gradient (MCG). Interestingly, our method in the Gaussian measurement case corresponds to the manifold constrained gradient (MCG) step that was proposed in Chung et al. (2022a). when setting W from Chung et al. (022a)).
For y; > 20, the approximation holds within 1% of the (Hubbard 1970
"I the discrete implementation, we instead use (, to express the step size. From the experiments, we observe that taking C' /llly A( A()o(z))) with Ç" set constant, yields highly stable results. See Appendix D for details in the choice step size.

---

SR (x-)
Inpaint (box)
Inpaint (random)
Deblur (gauss)
Deblur (motion)
Method
FID J
LPIPS 4
FID L
LPIPS $
FID 4
LPPPS 1
FID 4
LPPS
FID 4
LPIPS J
DPS (ouns)
39.35
0.214
33.12
0.168
21.19
0.212
44,05
0.257
39.92
0.242
DDRM
Kawar et al.
2022
62.15
0.294
42,93
0.204
69.71
0.587
74.92
0.332
MCG Chung et al. 2022a
87.64
0.520
40.11
0,309
29:26
0.286
101.2
0.340
310.5
0.702
PaP-ADMM Chan ot al. 2016
66.52
0.353
151.9
0.406
123.6
0.692
90.42
0.441
89.08
0.405
Score-DDE (Song
(ILVR (Chor al 2021
al. 2021h
96.72
0.563
60.06
0.331
76.54
0.612
109.0
0.403
292.2
0.657
ADMM-TV
110.6
0.428
68.94
0.322
181.5
0.463
186.7
0.507
152.3
0.508
Table I:
Quantitative evaluation (FID, LPIPS) of solving linear inverse problems on FFHQ 256x256- Ik validation dataset. Bold: best, underline: second best.
However, Chung et al. (2222a) additionally performs projection onto the measurement subspace after the update step via (16). which can be thought of as corrections that are made for deviations from perfect data consistency. Borrowing the interpretation of diffusion models from Chung et al.) (202a)), we compare the generative procedure geometrically. It was shown that in the context of diffusion models, a single denoising step via sg- corresponds to the orthogonal projection the data manifold, and the gradient step V 1ly - A(to)ll 2 takes a step tangent to the current manifold. For noisy inverse problems, when taking projections on the measurement subspace after every gradient step as in Chung et al. (2022a), the sample may fall off the manifold, accumulate error, and arrive at the wrong solution, as can be seen in Fig. 33a due to the overly imposing the data consistency that works only for noiseless measurement. On the other hand, our method without the projections on the measurement subspace is free from such drawbacks for noisy measurement (see Fig. 3b). Accordingly, while projections the measurement subsspae that that[Chung et al. (2022a) tries to solve, they fail dramatically for noisy inverse problems that we try solve. Finally, when used together with the projection steps the measurement subspace, it was shown that choosing different W for different applications was necessary for MCG, whereas our method is free from such heuristics.
useful for noiseless inverse problems
4 EXPERIMENTS
Experimental setup. We test our experiment on two datasets that have diverging characteristic FFHQ 256x256 (Karras et al. 2019), and Imagenet 256x256 (Deng al., 2009), on 1 validation images each. The pre-trained diffusion model for ImageNet was taken from Dhariwal & Nichol (202I) and was used directly without finetuning for specific tasks. The diffusion model for FFHQ was trained from scratch using 49k training data (to exclude IK validation set) for 1M steps. All images are normalized to the range [0, 1]. Forward measurement operators are specified as follows: (i) For box-type inpainting, we mask out 128 x 128 box region following Chung et al. (2022a), and for random-type we mask out 92% of the total pixels (all RGB channels). (ii) For super-resolution bicubic downsampling is performed. (iii) Gaussian blur kernel has size 61 x x6 with standard deviation of 3.0, and motion blur is randomly generated with the coda with size x 61 and intensity value 0.5. The kernels convolved with the ground truth image to produce the measurement, (iv) For phase retricval, Fourier transform is performed to the image, and only the Fourier magnitude is taken as the measurement. (v) For nonlinear deblurring, we leverage the neural network approximated forward model as in Tran et al. (202T). ). All Gaussian noise is added to the measurement domain with 0.05. Poisson noise level set to À 1.0. More details including the param--asseeters can be found in Appendix D
We perform comparison with the following methods: Denoising dif- fusion restoration models (DDRM) (Kawar et al.] (022). manifold constrained gradients (MCG) (Chung et al.) 2022a). Plug-and-play alternating direction method of multiplirrs (DMM-ADMM) (Chan et al. 2016) using DnCNN Zhang et al. (017) in place of proximal map- pings, total-variation (TV) sparsity regularized optimization method (ADMM-TV), and Score-SDE (Song et al., 20216). Note that [Song et al. 202 only proposes a method for inpainting, and not for gen-
Method
FID J LPIPS
DPS(ours)
55.61 0.399
OSS
137.7 0.635
HIO
96.40 0.542
ER
214.1 0.738
Table 3: Quantitative evalu- ation of the Phase Retrieval task (FFHQ).
https://githhn hitp:/lvviiiiioo htt.h//..coo/eeeeeooooeeeeeeeeee
7

---

Wessuroment
DDRM
Ground Truth
Measurement
PrP ADWM
Ground Trush
Caussian blur(a = 0.05)
Super-resolutien (x4, a = 0.05)
Motion blur(a = 0..05)
Figure 4: Results on solving linear inverse problems with Gaussian noise (a 0.05).
SR (x:4)
Inpaint (box)
Inpaint (random)
Deblur (gauss)
Deblur (motion)
Method
FID J.
LPIPS
FID J
LPIPS
FID J
LPIPS L
FID
LPIPS
FTD L
LPIPS
DPS lours)
50.66
0.337
38.82
0,262
35.87
0.303
62.72
0.444
56.08
0.389
Kawar 20222
45.95
0.665
0.427
MCG Ching al. 2022a
144.5
0.637
39.74
0.330
39.19
0.414
95.04
0,550
186.9
0.758
PaP-ADMM Chan 2016
97.27
0.433
78.24
0.367
114.7
0.677
100.6
0.519
59.76
0,483
Score-SDy Sine et
(LVV Choi 04 2041
170.7
0.701
54.07
0.354
127.1
0.659
120.3
0.667
98.25
0.591
ADMM-TV
130.9
0.523
87.69
0.319
189.3
0.510
155.7
0.588
138.8
0.525
Table 2:
Quantitative evaluation (FID, LPIPS) of solving linear inverse problems on ImageNet Ik validation dataset. Bold: best, underline: second best.
eral inverse problems. However, the methodology of iteratively applying projections onto convex sets (POCS) was applied in the same way for super-resolution in iterative latent variable refinement (ILVR) (Choi et al., (021), and more generally to linear inverse problems in Chung et al. (202b)); thus we simply refer to these methods as score-SDE henceforth. For fair comparison, we used the same score function for all the different methods that are based on diffusion (i.e. DPS, DDRM, MCG, score-SDE).
For phase retrieval, we compare with three strong baselines that are considered standards: oversam- pling smoothness (OSS) (Rodriguez et al. 2013). Hybrid input-output (HIO) (Fienup & Dainty 1987). and error reduction (ER) algorithm (Fienup) 1982). For nonlinear deblurring, compare against the prior arts: blur kernel space (BKS) styleGAN2 (Tran et al. 202T), based on GAN priors, blur kernel space (BKS)- generic (Tran al. 2021), based Hyper-Lapplciaa priors, and MCG. Further experimental details are provided in Appendix[D. [ For quantitative comparison, we focus on the following two widely- used perceptual metrics Fréchet Inception Distance (FID), and Learned Perceptual Image Patch Similarity (LPIPS) distance, with further evaluation with standard metrics: peak signal- to-noise-ratie ratio (PSNR), and structural similarity index (SSIM) provided in Appendix F.
SR
Motion deblur
Messturment
Noisy linear inverse problems. We first test our method on diverse linear inverse problems with Gaussian measurement noises. The quantitative results shown Tables 2 illustrate that the proposed method outperforms all the other comparison methods by large margins. Particularly, MCG and Score-SDE (or ILVR) methods that rely on projections ment subspace, where the generative process is controlled such that the measurement consistency is perfectly met. While this
DIP
Ground trudth
the measure-
Figure 5: Results on solving lin- ear inverse problems with Poisson noise (A = 1.0).
8

---

Measurement
Ours
Phase Retrieval
MCG
BKS generic
BKS - stylegan2
Non-uniform deblurring
Figure 6: Results solving nonlinear inverse problems with Gaussian noise (a 0.05).
is useful for noiseless (or negligible noise) problems, in the case where we cannot ignore noise, the solutions overfit to the corrupted measurement (for further discussion, see Appendix C.1 In Fig. 4 we specifically compare our methods with DDRM and PnP-ADMM, which are two methods that are known to robust measurement noise. Our method able provide high-qualit, reconstructions that are crisp and realistic on all tasks. On the other hand, we see that DDRM performs poorly on image inpainting tasks where the dimensionality of the measurements are very low, and tend to produce blurrier results on both SR, and deblurring tasks. We further note that DDRM relies on SVD, and hence only able to solve problems where the forward measurement matrix can be efficiently implemented (e.g. separable kernel in the case of deblurring). Hence, while one can solve Gaussian deblurring, one cannot solve problems such as motion deblur, where the point spread function (PSF) is much more complex. Contrarily, our method is not restricted by such conditions, and can be always used regardless
the complexity. The results of the Poisson noisy linear inverse problems presented in Fig. 5 Consistent with the Gaussian case, DPS is capable of producing high quality reconstructions that closely mimic the ground truth. From the experiments, we further observe that the weighted least squares method adopted in Algorithm works best compared to other choices that can be made for Poisson inverse problems (for further analysis, see Appendix C.4).
Nonlinear inverse problems. We show the quantitative results of phase retrieval in Table and the results of nonlinear deblurring in Table 4 Representative results are illustrated in Fig. We first observe that the proposed method is capable of highly accurate reconstruction for the given phase retrieval problem, cap- turing most of the high frequency details. However, we also ob- serve that we do not always get high quality reconstructions. In fact, due to the non-uniqueness of the phase-retrieval under some conditions, widely used methods such HIO are also dependent on the initializations (Fienup 1978), and hence it is considered standard practice to first generate multiple reconstructions, and take the best sample. Following this, when reporting our quanti- tative metrics, we generate 4 different samples for all the methods, and report the metric based on the best samples. We see that DPS
Method
FID J LPIPS
(PS(s
41.86 0.278
BKS-styleGAN2
63.18 0.407
BKS- -KS-ggneric
141.0 0.640
MCG
180.1 0.695
Table 4: Quantitative evaluation of the non-uniform deblurring task (FFHQ).
outperforms other methods by a large margin. For the case of nonlinear deblurring, again see that our method performs the best, producing highly realistic samples. BKS-styleGAN2 (Tran et al.) 202T) leverages GAN prior and hence generates feasible human faces, but heavily distorts the identity. BKS-generis utilizes the Hyper-Laplace prior (Krishnan & Fergus 2009),, but is unable to remove artifacts and noise properly. MCG amplifies noise similar way that was discussed Fig.
CONCLUSION
In this paper, proposed diffusion posterior sampling (DPS) strategy for solving general noisy (both signal dependenn/independent inverse problems in imaging. Our method is versatile in that it also be used for highly noisy and nonlinear inverse problems. Extensive experiments show that the proposed method outperforms existing state-of-th of-te-of-theartt by large margins, and also covers the widest range of problems.

---

ACKNOWLEDGMENTS
This work was supported by the National Research Foundation of Korea under Grant NRF- 2020R 2020RJA2B5B0300 BA235003001990, by the Korea Medical Device Development Fund grant funded by the Korea government (the Ministry of Science and ICT, the Ministry of Trade, Industry and Energy, the Ministry of Health & Welfare, the Ministry of Food and Drug Safety) (Project Number: 1711137899, KMDF_PP 20200901_001 50015), and by the KAIST Key Research Institute (Interdisciplinary Research Group) Project.
REFERENCES
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-326, 1982.
Thilo Balke, Fernando Davis, Cristina
Michael McCann, Luke Pfister, and Brendt Wohlberg. Scientific Computational Imaging COde (SCICO). Software library available from https://g hww..coo./ooooooo.coooooomo...ot ann scico, 2022.
Yochai Blau and Tomer Michaeli. The perception distortion perception-distortion tradeoff. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 0228-6237, 2018.
Yu M Bruck and LG Sodin. On the ambiguity of the image reconstruction problem. Optics communications, 30(3):304 1979.
Stanley H Chan, Xiran Wang, and Omar A Elgendy. Plug-and-play admm for image restoration: Fixed-point convergence and applications. IEEE Transactions on Computational Imaging, 3(1): 84-98, 2016.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. ILVR: Conditioning method for denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Hyungjin Chung and Jong Chul Ye. Score-based diffusion models for accelerated MRL. Medical Image Analysis, pp. 102479, 2022.
Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. arXiv preprint arXiv:2206. 00941, 2022a.
Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Soine-Closer-Dfffuuse Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contraction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In A. Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association. 106(496);1602-1614 2011.
C Fienup and J Dainty. Phase retrieval and image reconstruction for astronomy. Image recovery: theory and application, 231:275, 1987.
James R Fienup. Reconstruction of an object from the modulus of its fourier transform. Optics letters, 3(1):27-29, 1978.
James R Fienup. Phase retrieval algorithms: comparison. Applied optics, 21(15):2758-277 1982.
Xiang Gao, Meera Sitharam, and Adrian E Roitberg. Bounds on the jensen gap, and implications for mean-concentrated distributions. arXiv preprini orXiv: 1712 2017.
10

---

MHMH Hayes. The reconstruction of a multidimensional sequence from the phase magnitude of its fourier transform. IEEE Transactions on Acoustics, Speech, and Signal Processing, 30(2): 140-154, 1982.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851, 2020.
WM Hubbard. The approximation of a poisson distribution by gaussian distribution. Proceedings of the IEEE, 5897),533331333 1970.
Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. Robust Compressed Sensing MRI with Deep Generative Priors. In Advances in Neural Information Processing Systems, volume 34, pp. 14938-14954, 2021.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. Snips: Solving noisy inverse problems stochas- tically. Advances in Neural Information Processing Systems, 34:21757-21769, 2021.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In ICLR Workshop on Deep Generative Models for Highly Structured Data, 2022.
Kwanyoung Kim and Jong Chul Ye. Noise2score:
Tweedie's s approach to self-supervised image denoising without clean images. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https:// openreview net/ wtte:/orrmmdddd00eee
Robert Hildreth Kingston. Detection of optical and infrared radiation, volume 10. Springer, 2013.
Dilip Krishnan and Rob Fergus. Fast image deconvolution using hyper- priors. Advances in neural information processing systems, 22, 2009.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models manifolds. In International Conference Learning Representations, 2022. URL https //openreview.o net www.mmidduiip.ooo.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps. arXiv preprint arXiv:2206.00992 00927, 2022.
Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3883-3891, 2017.
Herbert Robbins. An empirical bayes approach statistics. In Breakthroughs in statistics, pp- 388-394. Springer, 1992.
Jose A Rodriguez, Rui Xu, C-C Chen, Yunfei Zou, and Jianwei Miao. Oversampling smoothness: an effective algorithm for phase retrieval of noisy diffraction intensities. Journal of applied crystallography, 46(2):312-318, 2013.
Slavko Simic. On global upper bound for jensen's inequality. Journal of mathematical analysis and applications, 343(1):414 2008.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In 9th International Conference on Learning Representations, ICLR, 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In 9th International Conference Learning Representations, ICLR, 2021b.

---

Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with score-based generative models. In International Conference on Learning Representations. 2022.
Charles M Stein. Estimation of the mean of a multivariate normal distribution. The annals of Statistics, pp. 1135-1151, 1981.
Phong Tran, Anh Tuan Tran, Quynh Phung, and Minh Hoai. Explore image deblurring via encoded blur kernel space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. (1956-11955 2021.
Pascal Vincent. A connection between matching and denoising autoencoders. Neural computa- tion, 1377)16611-1174 2011.
Kai Zhang. Wangmeng Zuo, Yunjin Chen, Deyu Meng. and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE transactions on image processing. 26(7): 3 3142-3155, 2017.
12

---

A PROOFS
Lemma (Tweedie's formula). Let p(ynn (n) belong to the exponential famity distribution
ply n) py) exp(n) TTy) - (nn)),
(23)
where n, is the canonical vector of the famity, T(y) is some function of y. and ((n) is the cumulant generation function which normalizes the density, and po(y) the density up the scale factor when
= 0. Then, the posterior mean vj = EE(nyy) should satisfy
(V,T(y) n = V ply) - V log Po
(24)
Proof. Marginal distribution p(y) could be expressed as
p(y) plylm)p(m)dn
(25)
f poly) exp (n T(y) - ((n)))(()dn
(26)
Then, the derivative of the marginal distribution ply) with respect becomes
V Vyp(D) = (p)o(() / exp (n p(n))p(n)dn+ + / (VyT(y)) (poly) exp (n TT)) =((n))p(n)dd
vuPo(y)
np(yln)o(n)dn
Poyy)
Vypo(y) p(v) + (V+T(y)) / nply. (o
Therefore,
Vup(y) p(y) Po(y)
Vypo(U)
/ np(ngg)dn
+ (V (V)(()))
(27)
which is equivalent to This concludes the proof.
(v)(())) EEnnll V, logp(v)
Vy log
(28)
Proposition 1. For the case of VP-SDE or DDPM sampling. p(zol=z)) has the unique posterior mean
io = E[ [x]]
(at + -ā(t)) log (a.))
(9)
Vā(l)
Proof. For the case of VP-SDE and DDPM forward sampling in X we have
I aa(())=ll1
p(z,|zo) = (2x(1 d(t)))d/2 exp
(29)
2(1 a(t)
which is a Gaussian distribution. The corresponding canonical decomposition then given by
p(x;(x0)) po(x;) exp (a. T(x:) p(xo))
(30)
where
2(1 ax ||
po(ar)) (2T( att))
exp
a(t))
Vā(t)
T(x:)= 1- ā(t) 2t
p((r)) 2(1 a(t))
13

---

Therefore, using
24 have
Vā(t) ā(l)
to = V.. logp((())) + 1 a(1) 2t
which leads to
3o vā(t)
(at, + (1 (1- a(t) (xt))
(31)
This concludes the proof.
Proposition (Jensen gap upper bound (Gao al. 2017)). Define the absolute cenetered moment as Assume that for a O, there exists positive number K such that for any x E R. (f(x) - f(x)) - K| 11l.. Then,
my == </E[ p. and the mean as μ [E[]]
[[((X) - S(E|])]I < / [I(()) ((())dp(()
(32)
S {K sK S Mm
(33)
Lemma 2.
Let <6-) be univariate Gaussian density function with mean and variance o². There exists constant L such that Vx, y R.
(x(x) x(y)1 L|r y/-
(34)
where L
v2=0²
exp
20
Proof. As o' is continuous and bounded, we use the mean value theorem get
V(z, y) IR?? (Ø(x) < = 1 oloo |x - y/- Since L is the minimal value for (4). we have that s || Taking the limit y + x gives [x(xx) L, and thus || I L. Hence
(35)
x p
L = 10 =
r2 6z))llo0
(36)
Since the derivative of ø' is given as
0"(x)=o-- 22 (x n) (zx)
(37)
and the maximum is attained when r 1 + 0²H,, we have
= 1101|00 = c 21200
(38)
2TT22
Lemma 3. Let ((-) be an isotropic multivariate Gaussian function with mean μ and variance
I I. There exists constant L such that Vz, y RRd.
[10(x))
<(y) S L
x - //1
(39)
where L V220=
1/20²
Proof.
[((()) ا(y))1 max V2((2)]1 || yl
(40)
d exp (-
202 -| bll
(41)
2
where the second inequality comes from that each element of V=Φ(2) is bounded by V2 exp ( (- (-2z²)
14

---

Theorem 1. For the given measurement model 6 with n ~ (0, o²I), we have
(x)) = ply (o))
(13)
where the approximation error can be quantified with the Jensen gap, which is upper bounded by
J S V
1/20² [vA(<))]
(14)
where |=A(z)] = max Vax(x)) and m1 == f ||z0 - to (oolppaxll (x)) day-
Proof.
p(y(ar)) = p(y (y (2o ) p0 ddao
(42)
[m mm. [ (2x)]
(43)
Here, f(-) = h(A(-)) where is the forward operator and h(x)) the multivariate normal distribu- tion with mean and the covariance I²I. Therefore, we have
J( p(xoo:))) = [x](()))] - /(B[==))] [B[[(x))] - s(oo) (B(((A(z))) - h(A(()))
(44)
(45)
[h(A(20)) ((A((a))
(46)
I7 IA IAS
/ ||A(zo) A(do))ldP(aa)))))))
11202
(47)
2
d
/
11/20²
/eA(z))
|00 2o @olld((z))zz)
(48)
22T02
d
1/20"
V
(49)
2
where da (a+x) p(ao dmo, (b) is the result of Lemma theorem, and (d) is from Proposition
(c) is from the intermediate value
B INVERSE PROBLEM SETUP
Super-resolution. The forward model for super-resolution is defined as
~ 22, y ~ D(z
(Gaussian)
(50)
(Poisson)
(51)
where L E x represents the bicubic downsampling block Hankel matrix with the factor and P denotes the Poisson distribution with the parameter X.
Inpainting. For both box-type and random-type inpainting, the forward model reads
y ~ Pay I), y ~ P(y Px; A),
(Gaussian)
(52)
(Poisson)
(53)
where E {0, 1 is the masking matrix that consists of elementary unit vectors.
Linear Deblurring. For both Gaussian and motion deblurring, the measurement model is given
y " o y ~ Ply A);
(Gaussian)
(54)
(Poisson)
(55)
where tt E R the block Hankel matrix that effectively induces convolution with the given blur kernel v/.
15

---

0.0
0.01
0.02
0.03
0.04
0.05
MCG
Figure 7: Failure cases of MCG (Chung al. 2022a) noisy inverse problems due to noise amplification.
Nonlinear deblurring. We leverage the nonlinear blurring process that proposed in the GOPRO dataset (Nah et al., (2017), where the blurring process is not defined as convolution, but rather an integration of sharp images through the time frame. Specifically, in the discrete sense the measurement model reads
M Σ a
y
i= T,
(56)
M
where b(x) 2/2.22 is the nonlinear camera response function, and T' denotes the total time frames. While we could directly use (56) as our forward model, note that this is only possible when we have multiple sharp time frames at hand (e-g. when leveraging GOPRO dataset directly). Recently, there was an effort to distill the forward model through a neural network (Tran et al.) (2021). Particularly. when have a set of blurry-sharp image pairs {(x,, y:)) one can train neural network to estimate the forward model as
arg min
Σ y - Fa(x,, Go y.))ll.
(57)
where Go (z; ys) extracts the implicit kernel information from the pair, and Fg takes in 2;, Go yi) to generate the blurry image. When using Fe at deployment to generate new synthetic blurry images, one can simply replace Go(mi, y:) with a Gaussian random vector k. Consequently, our forward model reads
N F(y, II), KERE & (0,
(Gaussian)
(58)
Ply k); A), KER", & of.
(Poisson)
(59)
where k is the dimensionality of the latent vector k and of is the variance of the vector.
Phase Retrieval. The forward measurement model is usually given as
~ Fzol o I), y P(yl Fzol:
(Gaussian)
(60)
(Poisson)
(61)
where F denotes the 2D Discrete Fourier Trasform (DFT) matrix. In another words, the phase of the Fourier measurements are nulled, and our aim is to impute the missing phase information. As the problem is highly ill-posed, typically incorporates the oversampling in order to induce the
16

---

<- 0.001
Cauzsian (a =00000 0.00)
Poisson (2== 1.00
Figure Effect of step size Ç' on the results
uniqueness condition
Hayes 1982
Bruck & Sodin 1979). usually specified as
N F Pxol, o² #I),
(Gaussian)
(62)
y P FPyol A)
(Poisson)
(63)
where P denotes the oversampling matrix with ratio k/n.
Poisson noise simulation. simulate the Poisson noise, we assume that each measurement pixel is source of photon, where the number of photons is proportional to the discrete pixel value between and 255. Thus, we sample noisy measurement values from the Poisson distribution with the mean value of the clean measurement values. Here, the clean measurement is A(zu), which is an image after applying the forward operation. Then, we clip the values by [0, 255] and normalize to [-1, 1].
C ABLATION STUDIES AND DISCUSSION
C.I NOISE AMPLIFICATION BY PROJECTION
As discussed the experiments, methods that rely on projections fail dramatically when solving inverse problems with excessive amount of noise in the measurement, Even worse, for many problems such as SR or deblurring, noise gets amplified during the projection step due to the operator transpose A being applied. This downside is clearly depicted in Fig. where show the failure cases of MCG (Chung et al., 2022a) on noisy super-resolution. In contrast, our method does not rely on such projections, and thus is much more robust to the corrupted measurements. Notably, find that MCG also fails dramatically in SR even when there is no noise existent, while it performs well on some of the other tasks (e-g. inpainting). We can conclude that the proposed method works generally well across a broader range of inverse problems, whether or not there is noise in the measurement.
C.2 EFFECT OF STEP SIZE Ç'
There one hyper-parameter in our DPS solver, and that is the step size. As this value is essentially the weight that is given to the likelihood (i.e. data consistency) of the inverse problem, expect that the values being too high or too low will cause problems. In Fig. [ we show the trend of the reconstruction results when varying the step size G. Note that instead see that with low values of ç' 0.1, achieve
the notation
ç' & Gily A (a;))))) for brevity. Here, results that are not consistent with the given measurement. On the other hand, when we crank up the values too high (C' 5), we observe saturation arfiacts that tend to amplify the noise. From our experiments, we conclude that it is best practice to set the Ç' values in the range [0.1, 1.0] for best results. Specific values for all the experiments are presented in Appendix D
C.3 OTHER STEP SIZE SCHEDULES
While the proposed step size schedule in yields good results, there can be many other choices that one can take. In this section, we conduct an ablation study to compare against other choices. Specifically, we test 100 images for Gaussian deblurring (Gaussian noise, 0.05) FFHQ, and
17

---

Schedule:
decay
decay
Messurmment
Ground Iruh
Figure 9: Ablation study on the choice of step size schedule for DPS. (a) Measurement, (b-c) exponential decay with initial values 0.3, 1.0, (d-e) linear decay with initial values 0.3, 1.0, (f) x 1/o² (g) ours, (h) ground truth.
Strategy
Constant
Linear decay
Exponential decay
Initial value
1.0 (Ours)
M/
0.3
0.3
LPIPS +
0.247 + 0.045
0.727 = 0.038
0.287
0.045
0.251 + 0.044
0.421 = 0.065
0.442 + 0.108
Table Ablation study on step size scheduling. Bold: best, underline: second best.
compute the average perceptual distance (LPIPS) against the ground truth. We compare against
the following three choices: 1) Linearly decaying steps Ç = Syminit x (1 x) 2) exponentially
decaying steps Ç = Cimi y², with 7 = 0.99, 3) directly using step size proportional to 1/a² as in eq. [6
present qualitative analysis Fig. From the figure, it clear that the proposed schedule produces the best result that most closely matches the ground truth in terms of perception. For decaying step sizes, we often yield results that are coarsely similar to the ground truth, but varies in the fine details, the information about the measurement is less incorporated in the later steps of the diffusion. From Fig. 9 we see that taking step sizes proportional to 1/o², motivated by direct derivation from the gaussian forward model, yields poor results. We see similar results with the quantitative metrics presented in Table.
C.4 POISSON INVERSE PROBLEMS
For inverse problems corrupted with Poisson noise, more care needs to be taken compared to the Gaussian noise counterparts, as the noise is signal-dependee and therefore harder to account for. In this section, we discuss the different choices of likelihood functions that can be made, and clarify the choice (20) used in all our experiments. One straightforward option is to directly use the Poisson likelihood model without the Gaussian approximation. From (T).
we have that
log logp(y//o))
Σ log[A(xo)l; [A(xo)] log(y;!)
(64)
Va, logp(yy(2o) -a m
log (og(A((xo))s (A(ar))ss
(65)
18

---

Measurement
Polsson-exact
Polsson-Gausalan
Polsson-LS
Proposed
Ground truth
Gaussian deblur
Motion deblur
SR
Figure 10: Differences in the reconstruction results when using different choices for imposing data consistency for Poisson linear inverse problems.
which we refer to as Poisson-direct. Moreover, one can use the Gaussian approximated version of the Poisson measurement model given in 18
(y) [A(ax)))))
logp(yy)aroo)
log [A(za())
(66)
2[A((zo))l
(y) [A(xx)))))
Va, logp(y/20)) = V
log [2 [2x[A()))]]]
+
(67)
2
2[A(xx))
which we refer to as Poisson-Gausee Gaussian. Next, we can use our choice 19 to arrive at 20 which the proposed method. Finally, while irrelevant with the noise model, same least sqaures (LS) method used for Gaussian noise (we refer to this method as Poisson-LS), as due the central limit theorem, Poisson noise nearly Gaussian in the high SNR level regime. In Fig. 10 we show representative results achieved by using each choice. From the experiments, we observe that Poisson-direct unstable due to the log term in the likelihood. hence often diverging. We also observe that the residual A(ăo) fails to converge, hinting that the information from the measurement not effectively integrated into the generative process. For that the weighting term of the MSE is problematic, and this term prevents the process from proper convergence. Both the proposed method and Poisson-LS are stable, but Poisson-LS tends to blur out the relevant details from the reconstruction, while Poisson-shot preserves the high-frequency details better, and does not alter the identity the ground truth person.
also still the
we see
C.5 SAMPLING SPEED
As widely known in the literature, diffusion model-based methods are heavily dependent on the number of neural function evaluations (NFE). We investigate the performance in terms of L.PPS
19

---

LPIPS
0.5
Method
Wall-clock time [s]
DDRM
Score SDE (Song et al. 2021b
36.71
MCG
DDRM (Kawar Euze
2,029
0.4
MCG Chnng dr 103344
80.10
Score SDE
PnP-ADMINT (Cisa mi 2016
3.631
DPS (ours)
BK5 -t.5-styeGGAN
er
891.8
BKS generic [Tai et al 2021
93,23
0.3
ER (Fienup
HIO (Feeni & Dainty 1987
6,317
OSS Roiiggue
15.65
Ours
78.52
0.2
(b) Runtime for each algorithm in Wall-clock time: Computed with a single GTX 2080Ti GPU.
NFR=20
50
100
250
500
1000
(a) LPIPS vs. NFE
Figure II: Ablation studies performed with SRx4 x 4 task on FFHQ x56x256 data, and the runtime analysis of the different algorithms.
with respect to the change in NFEs in Fig. Ila For the experiment, we take the case of noisy SRx 4, which is a problem where DDRM tends to perform well, in contrast to other problems, c.g. impainting. In the high NFE regime (2 250), DPS outperforms all the other methods, whereas in the low NFE regime (S 100), DDRM takes over. This can be attributed to DDIM (Song et al. 2021a) sampling strategy that DDRM adopts, known for better performance in the low NFE regimes. Orthogonal to the direction presented in this work, devising a method to improve the performance of DPS in such regime with advanced samplers (e.g. Lu al. (2022) Liu al. (022) would benefit the method.
C.6 LIMITATIONS
Inheriting the characteristics of the diffusion model-based methods, the proposed method is relatively slow, as can be seen in the runtime analysis of Fig. [Tb However, we note that our method is still faster than the GAN- optimization methods, as we do not have to finetune the network itself. Moreover, the slow sampling speed could be mitigated with the incorporation of advanced samplers. Our method tends to preserve the high frequency details (c.g. beard, hair, texture) of the image, while methods such DDRM tends produce rather blurry images. In the qualitative view, and in the perception oriented metris (i.e. FID, LPIPS), our method clearly outperforms DDRM. In contrast, in standard distortion metrics such as PSNR, our method underperforms DDRM. This can be explained by the perception-distoriie frequency details may actually penalize the reconstructions from having better distortion metrics. Finally, we note that the reconstruction quality of phase retrieval is not as robust compared to other problems- linear inverse problems and nonlinear deblurring. Due the stochasticity, we often encounter failures among the posterior samples, which can be potentially counteracted by simply taking multiple samples, as was done in other methods. Devising methods to stabilize the samplers, especially for nonlinear phase retrieval problems, would be promising direction of research.
tradeoff phenomena (Blau & Michaeli, 2018). where preserving high
D EXPERIMENTAL DETAILS
D. IMPLEMENTATION DETAILS
Step size. Here, we list the step sizes used in our DPS algorithm for each problem setting.
Linear inverse problem
Gaussian measurement noise
* FFHQ
Super-resolution: Ç, 1/1ly A
Inpainting: Ç; = 1/1ly A(& (x: ))
Deblurring (Gauss): = 1/lly A(ăn (x:)))
Deblurring (motion): G: 1/1ly (x;)())))
20

---

ImageNet
Super-resolution: Ç = 1/1ly A( (x))]
Inpainting: Ç; 1/1ly (x;))
Deblurring (Gauss): Ç = 0.4/| A(āb(z;)))
Deblurring (motion): Ç = 0.6/|ly y- A()n(()))
Poisson measurement noise
* FFHQ
Super-resolution: Çi = 0.3/ 1yy - A())((;)))
Deblurring (Gauss): (. = 0.3/ (z))(
Deblurring (motion): Ç 0.3/ A())(x)))
Nonlinear inverse problem
Gaussian measurement noise
FFHQ
Phase retrieval: = 0.4/lly Al (x;)))
non-uniform deblurring: G 1.0/1 - A(Co (2:)))
Score functions used. Pre-trained score function for the FFHQ dataset was taken from Choi et al. (2021 and the score function for the ImageNet dataset was taken from Dhariwal & Nichol (02
Compute time. All experiments were performed on single RTX 2080Ti GPU. FFHQ experiments take about 95 seconds per image (1000 NFE), while ImageNet experiments take about 600 seconds per image (1000 NFE) for reconstruction due to the much larger network size.
Code availability.
Code is available
at
s.ffuusion-poteerior-
D.2 COMPARISON METHODS
For DDRM, MCG, Score-SDE, and our method use the same checkpoint for the score functions.
DDRM. All experiments were performed with the default setting of y/8 = 1.0, = 0.85, and leveraging DDIM (Song et al. 2021a) sampling for 20 NFEs. For the Gaussian deblurring experiment, the forward model was implemented by separable ID convolutions for efficient SVD.
MCG. We set the same values of a that used in our methods (DPS). At each step, the ad- ditional data consistency steps are applied as Euclidean projections onto the measurement set = {a ((x;)) Yi Y; p(y: yo)].
Score-SDE. Score-SDo solves inverse problems by iteratively applying denoising followed by data consistency projections. As in MCG, we apply Euclidean projections onto the measurment set C.
PnP-ADMM. We take the implementation from the scico library (Balke et al., 2022). The parameters are set as follows: X = 0.2 (ADMM penalty parameter), maxiter= For proximal mappings, we utilize the pretrained DrCNN Zhang et al. (2017) denoiser.
ADMM-TY. We minimize the following objective
min -
A(zo) Dzo 12.1.
(68)
where D = Dr., Dy) computes the finite difference with respect to both axes, À is the regularization weight, and 112,1 implements isotropic TV regularization. Note that the optimization is solved with ADMM, and hence we have an additional parameter p. We take the implementation from the scico library (Balke et al. 2022). The parameters X, p were found with grid search for each optimization problems. We the following settings: (X,p) (2.7c - 2, 1.4e 1) for deblurring, (X, p) = (2.7e 2, 1.0e - 2) for SR and inpainting.
ER, HIO, oss. For all algorithms, initialize real signal by sampling from the normal distribution as the problem statement of (Fienup) 1982). For the object domain constraint, we apply both the
Unconditional version
21

---

non-negative constraint and the finite support constraint. We set the number of iterations to 10,000 for sufficient convergence. To mitigate the instability reconstruction depending on initialization, repeat each algorithm four times per data and report the best one with the smallest mean squared eroor hetween the measurement and amplitude of the estimation in the Fourier domain. In the case of HIO and OSS, set 3 to 0.9, which yields best results.
E FURTHER EXPERIMENTAL RESULTS
We first provide quantitative evaluations based the standard PSNR and SSIM metrics in Table 6 and Table 7
(x4)
Inpaint (box)
Inpaint (random)
(gauss)
(motion)
Method
PSNR
SSIM
PSNR
SSIM
PSNR 1
SSIM
PSNR
SSIM
PSNR
SSIM f
DPS (cuns)
25.67
22.47
0.873
25.23
24.92
0.859
DDRM Kawer
2022
25.36
0.835
22.24
U,869
9.19
0.319
23.36
0.767
MCG
Chung et al. 2022a
20.05
0.539
19.97
0.703
21.57
0.751
6.72
0.051
6.72
0.055
PaP-ADMM
26.55
0.865
11.65
0.642
8.41
0.325
24.93
0.812
24.65
0.825
Score-SDE
(ILVR
Seng et of bb2
17.62
0.617
18.51
0.678
13.52
0.437
7.12
0.109
6.58
0.102
ADMM-TV
23.86
0.803
17.81
0.814
22.03
0.784
22.37
0.801
21.36
0.758
Table 6: Quantitative evaluation (PSNR, SSIM) of solving linear inverse problems on FFHQ 256x256-
Ik validation dataset. Bold: best, underline: second best.
SR (x4)
Inpaint (box)
Inpaint (random)
Deblur (guuss)
Deblur (motion)
Method
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM 1
PSNR
SSIM 1
DPS (ouns)
23.87
0.781
18.90
0.794
22.20
0.739
21.97
0.706
20.55
0.634
24.96
0.790
18.66
0.814
14.29
0.403
22.73
0.705
MCG
302
13.39
0.227
17.36
0.633
19.03
0.546
16.32
0.441
5.89
0.037
PaP-ADMM
Chan et al 2016
23.75
0.761
12.70
0.657
8.39
0.300
21.81
0.669
21.98
0.702
Scone-DDg
(LVR Coor or 2021
0.256
16.48
0.612
0.120
ADMM-TV
22.17
0.679
17.96
0.785
20.96
0.676
19.99
0.634
20.79
0.677
Table 7: Quantitative evaluation (PSNR, SSIM) of solving linear inverse problems on ImageNet 256x validation dataset. Bold: best, underline: second best.
Further experimental results that show the ability of our method to sample multiple reconstructions are presented in Figs. 12
15 16 [7 (Gaussian measurement with o = 0..55,, and Fig. [8]
(Poisson measurement with
1.0).
22

---

Meosurement
Oura
Ours 2
Ground Trush
Mossurement
Oura
Ours
Ground Trush
Figure 12: SR (Left x8, Right 16), results on the FFHQ Karras al. 2019 256 256 dataset.
23

---

Measurement
Oura
Ours 2
Ground Truth
Messurement
Oura
Ours
Ground Trush
Figure 13: SR (Left x8, Right 16), results on the ImageNet (Deng al. 2009) 256 256 dataset.
24

---

Messurement
Ours
Ground Truth
Measurement
Ours
Ground Truth
Figure 14: Inpainting results (Left 128x 128 box, Right 92% random) the FFHQ Karras et al. 2019) 256 256 dataset.
25

---

Messurement
Ours1
Oura
Ground Truth
Measurement
Oura
Ground Truth
Figure 15: Inpainting results (Left 128x 128 box, Right 92% random) on the ImageNet 2009) 256 256 dataset.
Deng et al.
26

---

Meosurement
Truth
Nonsurement
Ours
Ground
SH
FU
MATT
Figure 16: Deblurring results (Left Gaussian, Right motion) on the FFHQ
Karras et al. 2019
256 x 256 dataset.
27

---

Meosurement
Ours
Ground Truth
Nonsurement
Ours
Ground
Pntn
Figure 17: Deblurring results (Left Gaussian, Right motion) on the ImageNet (Karras et al. 2019
256 x 256 dataset.
28

---

Measurement
Ground
Palls
Polic Exch
Pad
Poli
Figure 18: SR (Left x8, Right x 16) with Poisson noise (A = 0.05), results on the FFHQ KKrrrs
al. 2019 256 256 dataset.
29

---

Moosurement
Ours
Ground Truth
Nonsurement
Ground
Figure 19: Deblurring results with Poisson noise (A = 1.0) (Leff Gaussian, Right motion) on the FFHQ Karras et al. 2019) 256 x 256 dataset.
30


Reviews:
[{"review_id": "tJ_H30PDiD", "reviewer": "Reviewer_ooZZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a framework for general Bayesian noisy inverse problems using diffusion probabilistic models. The unknown data distribution is expressed as a posterior distribution. The updated reverse diffusion process now involves a generally intractable likelihood term. The authors propose the use of Laplace approximation of the likelihood term to circumvent the intractability issue. Results are presented that show the usefulness of the proposed framework.", "summary_of_the_review": "The paper is overall interesting, the issue is the novelty factor which is low. The paper could still appeal to a sufficiently large audience. I'm not convinced on accepting the paper, but I'm not convinced on rejecting it either. I'm looking forward to reading the rebuttal on the novelty factor.", "strength_and_weaknesses": "The main selling point of the paper is that it proposes a rather straightforward framework for solving general Bayesian noisy inverse problems using diffusion probabilistic models. The numerical results show that by using the framework one can obtain high quality results for a diverse set of problems. It is worth mentioning that the proposed method works on top of existing diffusion probabilistic models.\n\nIts main weakness is that overall, it brings little to the table in terms of new knowledge. The paper is interesting, however, and with the risk of oversimplifying a bit the contributions, it is a collection of existing models and/or techniques. While the use of the Laplace approximation is clever in allowing a straightforward approximation of the likelihood term in the reverse diffusion, it is arguably a standard technique in a statistician and ML practitioner's toolbox.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YcM_5YDJuwc", "reviewer": "Reviewer_3uLY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers an important problem in denoising diffusion models (DPM), that is, how to accurately conduct posterior sampling given a pre-learned score function and an inverse problem. Previous works address this problem by circumventing the calculation of $p(y|x^t)$ and resorting to projections onto the measurement subspace. However, these works either achieve unsatisfactory performance or are not elegant in terms of theoretical interpretation. Different from prior methods, diffusion posterior sampling (DPS) proposes to approximately compute $p_t(y|x^t)$ for each diffused image $x^t$, and conduct posterior sampling using the discretized version of the following equation\n$$dx=[-\\frac{\\beta(t)}{2}x-\\beta(t)(\\nabla_{x_t}\\text{log} p_t(x_t) + \\nabla_{x_t} \\text{log} p_t(y|x_t))]dt + \\sqrt{\\beta{t}} d\\bar{w}$$\n\nIn summary, the contribution of this work is:\n1. It proposes a feasible approach to compute $p_t(y|x_t)$ by using Tweedie's formula and Laplace's method. In particular, \n$$\\nabla_{x_t} \\text{log} p_t(y|x^t) \\sim \\nabla_{x_t} \\text{log} p_t(y|\\hat{x}_0),\\quad\\text{where}\\quad \\hat{x}_0 = \\text{TweedieFormula}(x_t)$$\n2. It experimentally shows that DPS achieves better results than the state-of-the-art algorithms on multiple inverse problems", "summary_of_the_review": "I think this paper makes concrete contributions to the literature on diffusion models with application to inverse problems. The method DPS is novel, and the theoretical interpretation in the work is insightful.", "strength_and_weaknesses": "*Strength*:\n1) The manuscript is well-written and easy to follow.\n2) The figures are illustrative and useful for readers to understand the problem and the approach.\n3) The proposed DPS is novel and provides a new way to compute $p_t(y|x^t)$ \n4) Clear experiments with sharp results\n\n*Weakness & Questions*:\n1. The connection between Proposition 2 and Theorem 1 is not clear in the text. Please provide more explanation on how Proposition 2 helps derive Theorem 1. I notice the proof in the appendix, what I suggest is to include some high-level explanation.\n2. Fig. 3 is very illustrative. However, it can mislead readers to interpret that DPS sticks to each manifold $\\mathcal{M}_n$ perfectly, which I don't think is true as multiple layers of approximation are applied. I suggest the authors emphasize in the caption that this is just a conceptual illustration of DPS.\n3. In the experiments, performance is only evaluated by using perception loss rather than PSNR/SNR/SSIM. Is it because diffusion models generally have bad performance under these metrics? It will be interesting to the PSNR/SSIM comparison because reconstruction accuracy is much more important than perceptual quality in imaging inverse problems.\n4. What is the denoiser of PnP-ADMM? If it is not a deep denoiser, I suggest comparing DPS with a deep denoiser.\n\n*Minors:\n1. Eq. 11 and Alg. 1 & 2 are not consistent.\n2. Please clarify the definition of $\\simeq$.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YaCBoP2AYo", "reviewer": "Reviewer_dAym", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a framework based on denoising diffusion model for solving inverse problem in image processing. The main contribution is the characterization of the forward process for non-linear models which include many well known image formation models. The proposed method has the specificity to both be easier to use than previous models (e.g no SVD) and relies on automatic differentiation for complex computation. The results shows great improvement compare to others methods based on diffusion models. ", "summary_of_the_review": "This paper proposes a denoising diffusion process for inverse problems. The main contribution is on the forward process that is well described and easy to implement. The experiments show promising results for image processing problems.", "strength_and_weaknesses": "## Strength\n\nThe proposed methods has several interesting contributions,\n1. The forward process is easy to implement and don't ask for heavy operations like SVD.\n2. All the framework is well motivated and the explanation are clear.\n3. Results are impressive compared to state-of-art.\n4. The full framework seems very extensible and then may interest a large part of image processing community.\n\n## Weaknesses\n\nThere is no clear weakness. My only regrets is there is no comparison against more classical methods (e.g image denoising with sparse prior and ADMM as solver).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "A9BCfVNHdjG", "reviewer": "Reviewer_VKPQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "this submission deals with posterior sampling from diffusion models for inverse problems. For general inverse problems with nonlinearity and measurement noise, sampling the conditional posterior needs the conditional score \\log \\nablap(y|x_t), that is very complicated and can be quite far from Gaussian, at a generic time step t. The idea in this submission is to (first-order) approximate it around the optimal MMSE estimate at each time t, namely \\hat{x}_0. Several experiments, provided for linear and nonlinear inverse tasks, show significant quality (FID & LPIPS) improvement over SOTA with this simple approximation.   ", "summary_of_the_review": "This paper deals with an important and timely problem. The idea is original, and the experimental results are solid. However, the main result in Theorem 1 seems to be quite restrictive to a small class of distributions that limits the scope of the theorem. ", "strength_and_weaknesses": "Strength\n- extensive evaluation results and good improvements over SOTA methods\n- addressing an important and timely problem\n- simple approach but seems effective\n- well written for the most parts\n\n\nWeakness\n- The main result in Theorem 1 includes quite strict assumptions that may not hold true for distributions with scores with order beyond linear such as Gaussian. This can really limit the scope of Theorem 1. \n\n- The result in Theorem 1 seems to be a simple first order approximation, but it’s been derived in a complex manner. The Laplace approximation to end up with the result in Theorem 1 seems not necessary.\n\n- The approximation can be very loose for large noise levels at the later steps of the diffusion process because MMSE estimator will be quite far from the ground-truth x_0\n\n\nQuestions and comments\n- The assumptions in Theorem 1 seem to be quite restrictive, which does limit the scope of the results and they don’t even include Gaussian distribution. First of all, what's the intuitive meaning of the assumptions? Second, the authors need to elaborate when these assumptions are satisfied and how limited the scope of the distributions satisfying these assumptions would be.\n\n- in section 3.2. it is not quite clear, how p(y|x_0) is related to p(y|\\hat{x}_0)? This seems to be coming from proposition 1, but it's not clear since x_0 and \\hat{x}_0 could have completely different distributions. Is that exact or is there an additional hidden assumption here? This needs to be clarified in the paper. \n\n- The choice of step size seems to be very specific in this work. Did you try other step sizes such as the constant or diminishing step size rule? it seems that the step size rule adopted here is the key to suppress noise? what if one uses the step size coming from eq. 12, namely 1/sigma^2? an ablation on the choice of step size would be needed to clarify the sensitivity of the algorithm to step size.\n\n- better to move tables/algorithms to the top part of each page\n- page 3, last paragraph, line 6, is there a typo in p(x|y)? shouldn’t it be p(y|x)?\n- page 3, last line, remove “the” from “.... both lines of the works …”\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%