DMPlug: A Plug-in Method for Solving Inverse Problems with Diffusion Models
Authors: Hengkang Wang
Xu Zhang
Taihui Li
Yuxiang Wan
Tiancong Chen
Ju Sun
Venue: NeurIPS 2024 poster (2024)
Keywords: Inverse problems
diffusion models
Primary area: diffusion_based_models

Abstract: Pretrained diffusion models (DMs) have recently been popularly used in solving inverse problems (IPs). The existing methods mostly interleave iterative steps in the reverse diffusion process and iterative steps to bring the iterates closer to satisfying the measurement constraint. However, such interleaving methods struggle to produce final results that look like natural objects of interest (i.e., manifold feasibility) and fit the measurement (i.e., measurement feasibility), especially for nonlinear IPs. Moreover, their capabilities to deal with noisy IPs with unknown types and levels of measurement noise are unknown. In this paper, we advocate viewing the reverse process in DMs as a function and propose a novel plug-in method for solving IPs using pretrained DMs, dubbed DMPlug. DMPlug addresses the issues of manifold feasibility and measurement feasibility in a principled manner, and also shows great potential for being robust to unknown types and levels of noise. Through extensive experiments across various IP tasks, including two linear and three nonlinear IPs, we demonstrate that DMPlug consistently outperforms state-of-the-art methods, often by large margins especially for nonlinear IPs.

DMPlug: A Plug-in Method for Solving Inverse Problems with Diffusion Models
Hengkang Wang' Xu Zhang Taihui Li
Yuxiang Wan Tiancong Chen'
Ju Sun
DDepartment of Computer Science and Engineering, University of Minnesota (wang9881, 1ixx5027 , wan01530, chen6271, jusuun @umnn edu Ammzzoncom Inc., spongezhang@gmail..
Abstract
Pretrained diffusion models (DMs) have recently been popularly used in solving inverse problems (IPs). The existing methods mostly interleave iterative steps in the reverse diffusion process and iterative steps to bring the iterates closer satisfying the measurement constraint. However, such interleaving methods struggle to pro- duce final results that look like natural objects of interest (i.e., manifold feasibility) and fit the measurement (i.c., measurement fcasibility), especially for nonlinear IPs. Moreover, their capabilities deal with noisy IPs with unknown types and levels of measurement noise are unknown. In this paper, we advocate viewing the reverse process in DMs as function and propose novel plug-in method for solving IPs using pretrained DMs, dubbed DMPlug. DMPlug addresses the issues of manifold feasibility and measurement feasibility in principled manner, and also shows great potential for being robust to unknown types and levels noise. Through extensive experiments across various IP tasks, including and three nonlinear IPs, we demonstrate that DMPlug consistently outperforms state-of-the-an art methods, often by large margins especially for nonlinear IPs. The code is available at whttts://// / githtb...b......g com/ sun uun- DmmnDu
linear
Introduction
Inverse problems (IPs) are prevalent in numerous fields, such as computer vision, medical imaging. remote sensing, and autonomous driving (14). The goal of IPs is to recover an unknown object from noisy measurements y = A(xx) + n. where A is a (possibly nonlinear) forward model and n denotes the measurement noise. IPs often ill-posed: typically, even y is noiseless, æ cannot be uniquely determined from y and A. Hence, incorporating prior knowledge on is necessary to obtain a reliable estimate of the underlying æ.
Traditionally, IPs are solved in the regularized data-fitting framework, often motivated as performing the Maximum a Posterior (MAP) inference:
ming A(x)) + ((x)
A(x))(x)) data-fitting loss, 2(x:) regularizer.
Here, minimizing the data-fitting loss promotes y R A(æ), and the regularizer encodes prior knowl- edge on x. Recently, the advent of deep learning (DL) has brought about a few new algorithmic ideas to solve IPs. For example, given training set of oeasurement-object pairs, i.e., {(y,. æ=));==...... one can hope to train a DL model that directly predicts æ for a given y [3] However, such hopes be shattered by practical challenges in collecting massive and realistic paired training sets,
especially for complex IPs 17 18] Even such challenges can be tackled, one may need to collect training set and train
DI. model for every new IP [15] overlooking potential shared priors on æ across IPs. An attractive alternative family of ideas combine pretrained priors on æ and
* "This work is not related to Dr. Xu Zhang's position at Amazon.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Messurement
Reference
DMPlug (ours)
DPS
Reaample
Nonlinear IPs
DID
BID with turbulence
Measurement
Reference
DMPPugg (ouns)
BRWDP
Noninaaa deburing
Noninear
Btlpfomme
Figure 1: Visualization of sample results from our DMPlug method (Ours) and main competing methods (DPS 19 and Resample (20) for super-resolution inpainting. and nonlinear deblurring; BlindDPS [21] and Stripformer [6] for blind image deblurring (BID) and BID with turbulence) on IPs we focus on in this paper. All measurements contain Gaussian noise with a 0.01.
regularized data-fitting in Eq. (1) For example, they first model the distribution of æ using deep generative models, such as generative adversarial networks (GANs) and diffusion models (DMs), based training sets of the form (a=:)i.......,, and then encode these pretrained generative priors when solving Eqq. (1) In this way, pretrained priors on æ can be reused in an off-the-shelf manner in different IPs about the same family of structured objects.
In this paper, we focus on solving IPs with pre- trained DMs. DMs have recently emerged as a dom- inant family of deep generative models due their relative stability during training (c.g.. GANs)
Ours
DPS
Residual Iy Abolli Log
Resample
and their strong capabilities to generate photorealis- tic images (and, depending on applications, other structured objects) once trained 22 27) These strengths of DMs have motivated ideas to pre- trained DMs to solve IPs, such as denoising, super- resolution, inpainting, deblurring, and phase re- Most
trieval 28 of these ideas interleave iterative reverse diffusion steps and iterative steps (sometimes projection steps, especially for linear IPs) move closer to the feasi- bility set {æly = A(x)} (see Fig. 3 and Fig. 4 (1)). However, they typically cannot guarantee the final convergence of the iteration sequence to either the feasible (i.e., measurement feasibility). or the object manifold M captured by pretrained DMs (i.e.,
33 26
Progress (%)
Figure 2: Evolution of the data-fitting loss |y - A(x)] of our DMPlug method vs. SOTA methods over percentage progress, for noiseless nonlinear deblurring on the CelebA dataset. Here, the percentage progress is calculated with respect to the total number of iterations taken by each method. The shadow regions indicate the ranges of the loss over 50 instances.
manifold feasibility), as they have modified both iter- ative processes (see detailed arguments in Section 2).
Fig. shows visible artifacts produced by these ideas on several IPs, highlighting (Issue 1) insuf- ficient manifold feasibility. To quickly confirm (Issue 2) insufficient measurement feasibility,

---

Figure 3: Interleaving methods (left) vs. our DMPlug method (right) for solving IPs using pretrained DMs. While interleaving meth- ods cannot ensure the feasibility of the final estimate for either the object manifold M the feasi- ble set {æly = A(x)), our DMPlug method ensures the manifold feasi- bility while promoting y = A(x) through global optimization.
Rasiblle Sat
(xly Abx)
(xly Aix))
M
Bood Delettry
Vaglor AA(R665))
we experiment on noiseless instances of the nonlinear deblurring problem (a nonlinear IP) follow- ing [20 19 and find that statc-of-the (SOTA) methods fail to find an æ that satisfies A(æ), as shown in Fig. 2. Furthermore, most SOTA DM-based methods for IPs assume known noise types (e.g., often Gaussian) and known, often very low, noise levels, casting doubts on their performance when faced with unknown noise types and levels, i.e., (Issue 3) robustness to unknown noise types and levels, as we confirm in Table 4
Our contributions In this paper. we propose a novel plug-in method, dubbed DMPlug, to solve IPs with pretrained DMs to mitigate all of the above three issues. DMPlug departs from the popular and dominant interleaving line of ideas by viewing the whole reverse diffusion process as function R(-), consisting of multiblock stacked DL models, mapping from the seed space to the object manifold M. This novel perspective allows us to naturally parameterize the object to be recovered x R(z) and then plug this reparametrization into Eq. (1) leading to unified optimization formulation with respect to the seed z. Conceptually, since R(z) probably produces feasible point the object manifold M and global optimization of the unified formulation encourages y R A(R(2)). ,, the optimized z could lead to an x R(z) that enjoys both manifold and measurement feasibility, i.e. (tackling Issues 1 & 2). Fig. 3 and Fig. 4schematically illustrate the dramatic difference between intericaving methods and our plug-in in method (DMPlug), and Figs. 1 confirmms DMPlug's 's strong capability in finding feasible solutions. For noisy IPs with unknown noise types and levels, we observe that our DmPlug enjoys benign rearly-learning-hher -he--therfnt-ttinggg (ELTO) property: the quality of the estimated object climbs first to a peak and then degrades once the noise is picked up. as shown in Fig. (2). This benign property, in combination with appropriate early-stopping (ES) methods that stop the estimate sequence near the peak, allows our method to solve IPs without the exact noise information, i.e., (tackling Issue 3), as shown in Table 4). Fortunately, we find that an ES method, ES- WMV 35 (co-developed by a subset of the current authors), works well for this purpose (see Table 4).
Our contributions can be summarized as follows. (1) In [eection 3.1, we pioneer a novel plug- in method, DMPlug, which is significantly different from the prevailing interleaving methods, to solve IPs with pretrained DMs. Then we make the proposed method practical in terms of computational and memory expenses by leveraging one key observation; (2) In [Section 4 and Appendix El we perform extensive experiments on various linear and nonlinear IPs and show that our method outperforms SOTA methods. both qualitatively and guantitatively-ofii margins, especially on nonlinear IPs. For example, measured PSNR, our method can lead SOTA methods by about 2dB and 3 ~ 6dB for linear and nonlinear IPs, respectively. Moreover, our method demonstrates flexibility in employing different priors and optimizers, as explored [Section 4.3, (3) In Section 3.3 we observe an early-learning- weaiiinggttenoovvrrre tvvr-tivg (ELTO) property. i.e., that our DMPlug tends to recover the desirable object first and then overfit to the potential noise. By taking advantage of this benign property and integrating the ES method ES-WMV [35), our method is the first to achieve robustness to unknown noise types and levels, leading SOTA methods by about 1dB and 3.5dB in terms of PSNR for linear and nonlinear IPs, respectively.
by large
2 Background and related work
Diffusion models (DMs) The denoising diffusion probabilistic model (DDPM) 23 is a seminal DM for unconditional image generation. It gradually transforms 2o ~ Pama into total noise TT ~ (O, I) (i.c., forward diffusion process) and then learns to gradually recover zo from 27 through
3

---

incremental denoising (i.e., reverse diffusion process). The forward process can be described by stochastic differential equation (SDE), dæ - zdt + /5,dw, where 3, is the noise schedule and uo is the standard Wiener process. The corresponding reverse process is described by
(Reverse SDE for DDPM) da - 3. /2/2 + V po di + Bidw.
Here, w is the time-reversed standard Wiener process, Pr (x) is the probability density at time t, and v log (z) is the (Stein) score function, which is approximated by DL model (x) via score matching methods during DM training [36 37.. For discrete settings, given time steps / E (1....; T., a variance schedule ,1....,81 ,T, ag = 1 3e with oT + O, and āg = II C'g. the DDPM has the forward process De = /11 - -1,00 + /B2z = and the reverse process
(t) (DDPM) 0t-1 = 1/ (at - B.-B///1 ar Eo (2:)) + VBZz
where z ~ (O. As the DDPM has a slow reverse/sampling process, (4 proposes the denoising diffusion implicit model (DDIM) to mitigate this issue. With the same notation as that of the DDPM, the DDIM makes a crucial change to the DDPM: relaxing the forward process to be Markovannn by making Ft depend on both Zo and - This simple change allows skipping iterative steps in the reverse process, without retraining the DDPM. This leads to much smaller numbers of reverse steps, and hence substantial speedup in sampling. The reverse process is now defined as
(DDIM) x -i==va=-10000 + / ā.- (0) (xx))
(4)
where (2:4) [a:: v1
(t) (xx v
Vār is the predicted 2o with Zt.-
Pretrained DMs for solving IPs Ideas for solving IPs with DMs can be classified into two categories: supervised and zero-shot (8). The former trains DM-based IP solvers based on paired training sets of the form {()). æ.)} and is hence not our focus here (see our arguments in Section l}. The latter makes use of pretrained DMs as data-driven priors: (I) Most of the work in this category considers modeling pe (zyy)) directly and replaces the (unconditional) score function Va log (a) in
Eq. (2) by the conditional score function Va log Pt (xly)
V log Pe (x) + V log Pogpeeyyaa leading
to the conditional reverse SDE
dx = 1-3./2
log + V log (y [)]] dt + B.dw.
(5)
Here, while v. pog) can be naturally approximated by the pretrained score function e (x). V log Pe (y)())) is intractable as y does not directly depend on at Ideas to cireumvent this difficulty include approximating pt (z()) by Pr(3) (3) [f))), where 2 (t)] is implemented
as (o(x;) of [Eq. (4) in discretization 19 21). and interleaving unconditional reverse steps of
Eq. (3) or Eq. (4) and (approximate) projections onto the feasible set {æly A(z)) to bypass the
20] (II) An interesting alternative is to recall that the MAP framework (see also Eq. (1) involves maxę log pggp(y y x) + log p(x). ). and first-order methods, especially proxinal-gradient style methods, to optimize the MAP formulation typically only need to access p(x) through V_ log p(x), i.c., the score function the central object in DMs! So, one can derive IP solvers by wrapping pretrained DMs around first-order methods for (approximately)
likelihood Pe (pyl x)) 39. 30
33
32
40
41
optimizing the MAP formulation, see, e.g., 27 28).
Despite the disparate conceptual ways of utilizing pretrained DM priors, most of the methods under (I) and (II proposed in the literature so far follow a single algorithmic template, i.e., Algorithm I There, Lines 3-5 are simply a reverse iterative step in the DDIM (Bq. (4) obviously, one could replace this by reverse iterative step in other appropriate pretrained DMs. Line 6 helps move the iterate closer or onto the feasible set x y = A(x)). In other words, these methods interleave iterative steps move toward the data manifold M defined by the pretrained DM and iterative steps to move toward the feasible set {xly = A(x)) }. i.... as illustrated in Fig. 3 (left). However, it is unclear, priori, whether such interleaving iterative sequence will converge either, leading concerns about (Issue 1) insufficient manifold feasibility and (Issue 2) insufficient measurement feasibility. In fact, our Figs. 1 andd] 2 confirm these issues empirically, echoing observations made in several prior papers (20 42 30 In principle, Issue 2 can be mitigated by ensuring that Line 6 finds a feasible
'Reaall that in the continuous formulation of diffusion processes, æ is a function of t, i.e. 2(1) where the continuous variable E [0,1] is often omitted.

---

Algorithm 1 Template for interleaving methods Algorithm Proposed plug-in method, DMPlug
Input: # Diffusion steps T, measurement
Input: # Epochs E, # diffusion steps T, y
1: at ~ ~N( I)
Initialize seed 27 (O(0,))
2: for i = T 1 to 0
do
2: for e = to E - do
3:
$ -t= E (z;
(5)
3:
for i = T
to do
- Eg [1) (2{)
4:
io f Va, D; VI
4:
2/_1 < DDIM reverse with By and $
5:
26 Va. (2
6:
-i-1 f
(Approximately) Projec-
6:
DDIM reverse with 2g, $
tion [39 30 33
32
40
41
34 or gradient
7:
end for
update [20 28
19
29
27 26 with æo
8:
Update ze+1 2T
from 24 via gradient update
and - 1 to get closer to a y A(a))
for Eq.
7: end for Output: Recovered object z0
9: end for Output: Recovered object R(= (zF-1)
Figure 4: Comparison of prevailing interleaving methods and our plug-in method
20 However, this is possible only for easy IPs (c.g., linear IPs where can perform closed-form projection) and is difficult for typical nonlinear IPs as hard nonconvex problems are entailed. Moreover, although most of these methods have considered noisy IPs alongside noiseless ones (39 30
2;-1 for {aly y = }(x)) in each iteration 33 30
28 19] 21 29) their common assumption about known noise types (often Gaussian) and levels (often low) is unrealistic: there many types of measurement noise in practice- the noise levels are usually unknown and hard to estimate [44], raising concerns about (Issue 3) the
33
32
20
hybrid from multiple sources [43], and
robustness of these methods to unknown noise types and levels; see (Section 4.2)
3 Method
In this section, we propose a simple plug-in method, DMPlug, to solve IPs with pretrained DMs to address Issues 1 & 2 in /Section 3.1] discuss its connection to and difference from several familiar ideas in [Section 3.2] and finally explain how to integrate an carly-stopping strategy, ES- WMV [35, into our plug-in in method to address Issue 3 in SScction 3.3] leading to an algorithm, DMPlug+ES-Wa summarized in Algorithm 3) that solves IPs with pretrained DMs under the regularized data-fitting framework Eq. (1) even in the presence of unknown noise.
3.1 Our plug-in method: DMPlug
Reverse process as a function and our plug-in method Interleaving methods discussed above can have trouble satisfying both manifold feasibility and measurement feasibility because they interleave and hence modify both processes, Although projection-style modifications can improve measurement
feasibility [39] 33 3
40 41 20, their application to nonlinear IPs seems tricky, and a single "projection" step could as difficult and expensive as solving the original problem due to the typical nonconvexity induced by the nonlinearity in A (28 20] In contrast,
32
propose viewing the whole reverse process as a function R(-) that maps from the seed space to the object space (or the object manifold M). Mathematically, if we write a single inverse step as a function g that depends on (4) i... 9 for the i-th reverse step that maps 2++1 to I, then
R = 9 o 9:00) O .... 9 .... o ---r-11
(o means function composition)
(6)
Note that for the DDPM, and those DMs based on SDEs in general, R. is a stochastic function due to noise injection in each step. To reduce technicality, we focus on DMs based on ordinary differential equations (ODEs), in particular, the DDIM, due to their increasing popularity [45] [6), resulting in
deterministic R's. This conceptual leap allows to reparametrize our object of interest as R(z)
and plug this reparametrization into the traditional regularized data-fitting framework in Eq. yielding the following unified optimization formulation:
(DMPlug) z" E arg ((y, A(R())) + ((R(z)),
xx' =²=R(2²))
We stress that here the optimization is with respect the seed variable 2, given the pretrained
reverse process R. Since we never modify the reverse diffusion process R, we expect R(z) to
5

---

produce an object on the object manifold M. i.e., enforcing manifold feasibility to address Issue Moreover. optimizing the unified formulation Eq. (7) expected to promote y A(R(zz)) inherent in the regularized data-fitting framework, i.e., promoting data feasibility to address Issue 2.
When there multiple objects of interest in the IP under consideration, i.e., y A(21, axx) + with & objects (e.g., in blind image deblurring image are objects of interest, where A(k, x) & æ;; see also Appendix C.4), different objects may have different priors that should be encoded and treated differently. Our unified optimization formulation Eg. (7) facilitates the natural integration of multiple priors. Another bonus feature of our unified formulation lies in the flexibility in choosing numerical optimization solvers. We briefly explore both aspects in [Section 4.3
A(k,, æ) + n. both the blur kernel & and sharp
Fast samplers for memory and computational efficiency When implementing DMPlug with typical gradient-based
25
79
11
15
30
solvers, e.g., ADAM [47], the gradient calculation in each it- erative step requires forward and a backward pass through the entire R. i.c., T blocks of the basic DL model the given
25
d 15
DM. For high-quality image generation 251. T is typically tens or hundreds, resulting in prohibitive memory and computa- tional burdens. To resolve this, we the DDIM, which allows skipping reverse sampling steps thanks to its non-Markovian property, the sampler R. We observe, to our surprise and also in our favor, that a very small number of reverse steps, such 3, is sufficient for our method to beat SOTA methods on all IPs we evaluate (see Fig. 5 and [Section 4). and further increasing the number does not substantially improve the per- formance (actually even slightly degrades it perhaps due to the numerical difficulty caused by vanishing gradients as more steps are included). So, we default the number of reverse steps to 3 unless otherwise stated. This number is not sufficient for generating high-qu nigh-quality photorealistic images with existing DMs, but is good enough for our method. The discrepancy suggests fundamental difference between image generation and image "regression" involved in solving IPs leave this for future work.
10
Resample (SOTA)
0.2
0.4
0.6
0.8
Time
Figure 5: PSNR (dB) vs. per- iteration wall-clock time (s) running on NVIDIA A100, for various reverse steps in R. Experiments on CelebA for 4x super-resolution; solver: ADAM; maximum itcrations: 6, 000
3.2 Seemingly similar ideas
(A) GAN inversion for IPs Our plug-in method is reminiscent of GAN inversion to solve IPs 48 for pretrained GAN generator Gg. GAN inversion performs a similar reparametrization x G (z)
and plugs it into Eq. (I) The remaining task is also to minimize the resulting formulation with respect to the trainable seed z, and produce an x = Gø(=*) through solution x". However, there are numerous signs that GAN inversion does not work well for IPs. For example, [50] also finetunes
the generator Go alongside the trainable seed to boost performance, and [33 32 report superior performance of DM-based methods compared to GAN-inversion-bel basedd ones for solving IPs; (B)
Diffusion inversion (DI) Given an object æ. DI aims to find a seed z so that R(z) reproduces æ,
an important algorithmic component for DM-based image and text editing [5153 24
54 55 A
popular choice for DI to modify DDIM 5
24
54 555 Although we cannot use DI to solve
general IPs, DI can be considered as an IP and solved through minz f(x., R(=)) using our plug-in method. (C) Algorithm unrolling (AU) The multiblock structure in R also resembles those DL models used in AU, popular family of supervised methods for solving IPs [56] [6]. In AU, the trainable DL model also consists multiple blocks of basic DL models, induced by unfolded iterative steps to solve SolveEE..c (1) While in AU the weights in these DL blocks trainable, the weights in DL blocks are fixed. More importantly, as a supervised approach, AU requires paired training sets of the form {1), x;) ) in contrast to the shot nature of our method here.
3.3 Achieving robustness to unknown noise
The www.-llaninngthenn--vrirriiii (ELTO) phenomenon When y contains noise, solving Eq. (7) can promote measurement feasibility, i.e., R A(R(=*)), but A(R(z*)) may also learn the noise, i.e., overfitting to noise. Interestingly, our method seems to favor desirable content and resist noise: (A) our method converges much faster when used to regress clean natural images than random

---

ynlage imagge shuffed
0.30
Random nose
0.25
MSE
0.20
PSNR
125
0.75
0.5
10F
10a
10'
10a
Iteration in Log Scale
Reration
(1)
(2)
Figure 6: The recovery performance of our method on image regression with (1): (a) clean natural image, (b) the same image with pixels randomly shuffled, (c) random noise iid sampled from Uniform(O, 1), and (2) noisy natural image with Gaussian noise at a 0.08. Here, image regression means, given any image æ, performing ming f(x, R(z)).
noise Fig, (I), suggesting that our method shows high resistance to noise and low resistance to structured content, and (B) when performing regression against noisy image x 20 + n, our method, although powerful enough to overfit the noisy image æ ultimately, picks up the desired image content first and then learns the noise, leading to a hallmark "early-learning then-ovveffittigg" (ELTO) phenomenon so that the recovery quality climbs to peak before the potential degradation due to noise (Fig. (2)). We stress that similar ELTO phenomena have been widely reported in the literature on using deep image priors (DIPs) solve IPs (17) although this is the first time this phenomenon has been reported for DM-based methods. Inspired by related studies in DIP, we also perform spectral analysis of intermediate recovery and reveal that our method has a spectral bias toward low-frequency components during learning, similar to DIP methods; see Appendix B
Achieving robustness via early stopping (ES) One may wonder how widely this ELTO phe- nomenon occurs when using our DMPlug to solve IPs. Besides extensive additional visual confirma- tion aeeeAAppeddi, F). in Table 4 4e show that our method, without using the noise information, leads the SOTA methods terms of peak performance during iteration particulaary,, by large margins on non-linear deblurring. The quantitative results suggest that the ELTO phenomenon is likely widespread, especially across noise types and levels. Hence, if we can perform proper early stopping (ES) to locate the peak performance, we tackle Issue 3. Deriving an effective ES strategy here is nontrivial, as in practice we do not have groundtruth images to compute any reference- based perfor- mance metries such PSNR. Fortunately, in the DIP literature, [35 discovers that for DaPed methods for IPs, the valleys of the running-variin variance curves of the intermediate reconstructions are well aligned with the performance peaks. Based this crucial observation, they propose an ES strategy, ES- WW, that can accurately detect performance peaks with small performance loss on various IP tasks. Inspired by their success, we integrate the ES-WMV strategy into our plug-in method and find that ES- WMV is highly synergetic with our method and performs reliable ES with negligible performance loss (see Table 4). Details of the entire algorithm can be found in Algorithm
4 Experiments
In this section, we evaluate our plug-in method, DMPlug, and compare it with other SOTA methods on two linear IPs, including super-resolution and inpainting, and three nonlinear IPs, including nonlinear deblurring, blind image deblurring (BID), and BID with turbulence. Following 20].
we construct the evaluation sets by sampling 100 images from CelebA (22, FFHQ (63) and LSUN-
bedroom [64), respectively, and resizing all images to 256 x 256; we measure recovery quality using three standard metrics for image restoration, including peak signal-to-noise-ratii (PSNR), structural similarity index (SSIM) and Learned Perceptual Image Patch Similarity (LPIPS) (55 with the default backbone. We describe in detail the five IPs tested and the formulations we use for them in Appendix C. we provide implementation details of method and the competing methods in
7

---

Appendix D
we include comparisons of computational costs, along with more quantitative and qualitative results in Apppendix E
Table (Nonlinear IP) Nonlinear deblurring with additive Gaussian noise (a = 0.01). (Bold: best,
under; second best, green: performance increase, red: performance decrease)
CelebA 62 (256 x 256)
FFHQ
63
(256 x 256)
LSUN
64 (256 x 256)
LPIPS.L
PSNR1
SSIM1
LPIPS
PSNR1
SSIM
I.IPS...
PSNR*
SSIM1
BKS sKylstA
1.047
22.82
1.051
0.620
0.987
20.90
0.538
BKS generic 66
1,051
21.04
0.591
1.056
20.76
0.583
0.994
18.55
0.481
MCG E0
0.705
13.18
0.675
0.167
14.28
0.188
ILVR 41
0.335
21.08
0,586
0.374
20:40
0.556
0.482
18.76
0.444
DPS D
0.149
24.57
0.723
0.130
25.00
0.759
0.244
23.46
0.684
ReSample
201
0.104
28.52
0.839
0.104
27.02
0.834
0,143
26.03
0.803
DMPlug (ours)
0.073
31.61
0.882
0.057
32.83
0.907
0.083
30.74
0.882
Ours Best compe.
00.031
+3300
+0.043
-0.017
+-579
+00073
-0.060
+4.71
+0.079
4.1 IP tasks and experimental results
Linear IPs Due to space constraints, our experimental results on Super-resolution & inpainting are included in (Appendix E.2, Our DMPlug can lead the best SOTA methods by about 2dB in PSNR and 0.02 in SSIM on average, respectively.
Nonlinear IPs Nonlinear deblur- ring We use the learned blurring operators from [66] with known Gaussian-shaped kernel and Gaus- sian additive noise with a 0.01, following 200 [9]. We compare our DMPlug against several strong baselines: Blur Kernel Space (BKS)- styleGAN2 (66] based on GAN priors, -Ks-generic [66] based on Hyper- Laplacian priors [68]. and DM-based methods that can handle nonlinear IPs, including MCG, ILVR, DPS and Re-
Table 2: (Nonlinear IP) BID with turbulence with additive Gaussian noise (o = 0.01). (Bold: best, under: second best, green: performance increase, red: performance decrease)
CelebA 62 (256 x 256)
FFHQ (63 (256 x 256)
LPIPS
PSNR1
SSIM
LPIPS
PSNR1
SSIM
DeBlurGANv2 S
0.356
24.17
0.739
0.379
23.52
0.723
Stripformer 0
0,318
24.97
0.745
0.343
24.30
0.725
MPRNet 7
0.696
0.399
22.28
0.682
ISR-WGAN 67
0,304
23.09
0.732
0.333
22.56
0.715
ILVR 4]
0.337
21.25
0.589
0.375
20.24
0.554
BlindDPS 21
0.137
25.45
0.730
0.165
24.40
0.712
DMPlug (ours)
0.146
28.34
0.790
0.164
27.91
0.812
Sample. Despite the advancements made by the recent ReSample (20] enhance the DPS method, from Table is evident that our DMPlug can still significantly outperform the SOTA ReSample by substantial margins. Specifically, our method improves LPIPS, PSNR, and SSIM by 0.05, 4.5dB, and 0.07, respectively, across the three datasets on average. In addition, our DMPlug delivers more faithful and precise restoration of details, as shown in Fig.
Ours Best compe.
-0.009
+2.89
+0.045
-0.001
+-3.5
+0.087
BID & BID with turbulence BID is about recovering a sharp image x (and kernel k) from y = k x + where denotes the linear convolution and the spatially invariant blur kernel k is unknown; for BID with turbulence which often arises in long-range imaging through atmospheric turbulence, we model the forward process as simplified "tilt-then-burr" process, following [69] y = k T (ar) +n, where (-).) is the tilt operation parameterized by unknown & (see more details in Appendix C). We mainly compare our DMPlug two DM-based models, including ILVR and BlindDPS 22.. which is an extension of DPS. In addition, we choose two classical M-P-based methods, i.e., Pan-Dark Channel Prior (Can-DCP) (70] and Pan-Co (71). based SelfDeblur (72), and four methods that are based on supervised training on paired datasets, including DeBlurGANs [5), Stripformer [6]. MPRNet 7 and -SRN [67]. It is important mention that BlindDP [21] use pretrained DMs not only for images but also for blur kernels (and tilt maps), lending it an unfair advantage over other methods. Although our method is flexible in working with multiple DM priors, as shown in /Sectionn 4.3] we opt to only use the pretrained DMs for images to ensure fair comparison. Tables 2) and show that our method, despite using fewer priors than BlindDPS, can surpass the best competing methods by approximately 0.03, 4.5dB, and 0.1 in terms of LPIPS, PSNR, and SSIM, respectively, on average. In the reconstructions of our method look sharper and more precise than those of the main competitors.
8

---

Table 3: (Nonlinear IP) BID with additive Gaussian noise (a 0.01). (Bold: best, under: second best, green: performance increase, red: performance decrease)
CelebA 62 (256 x 256)
FFHQ
63 (256 x 256)
Motion blur
Gaussion blur
Motion blur
Gaussian blur
I.PIPS..
PSNR1
SSIM
I.PIPS..
PSNRT
SSIM
LPIPS.,
PSNR1
SSIM1
P.PSS.
PSNR
SSIM
SeltDeblur
72
0.568
16.59
0.417
0.579
16.55
0.423
0.628
16.33
0.408
0.604
16.22
0.410
DeBlurGANv2
20.56
19.67
0.726
Stripfommer
0
0.287
22.06
0.644
0.316
25.03
0.747
0.324
21.31
0.613
0.139
24.34
0.728
20.53
0.375
22.33
0.685
Pan-DCP 70
0,606
15.83
0.483
0.653
20.57
0.701
0,616
15.59
0,464
0.667
20.69
0.698
Pan-fo X
15.16
0.470
0.654
20.49
0.675
0.642
14.43
0.669
20.34
0.671
ILV 31
0.398
19.23
0.520
0.338
21.20
0.588
0.445
18.33
0.484
0.375
20.45
0.555
BlindDPS
0.164
23.60
0.185
21.77
0.693
DMPlug (ours)
0.104
29.61
0,825
0.140
28.84
0.795
0.135
27.99
0.794
0.169
28.26
0.811
Ours vs. Best compe.
-0.060
+-6.01
+0
+0.1413
-0.033
+3.69 69.69
+000048
-0,050
46622
40.166
-0.024
+-392
+0.088
4.2
Robustness to unknown noise
Measurement
Reference
DMPlug (ours)
ReSample
Figure 7: (Robustness) Visualization of sample results from our DMPlug and main competing methods for 4x super- resolution (top) and nonlinear deblurring (bottom) with low-level Gaussian noise.
For robustness experiments, we choose super-resolution and nonlinear deblurring represent linear and nonlinear IPs, respectively. To simulate scenarios involving unknown noise, we generate measurements with four types noise: Gaussian, impulse, shot, and speckle noise, and across two different noise levels: low (level-1) and high (level-2), following 43 (see details in Appendix D.I). but we use the same formulation and code for each designed for mild Gaussian noise, regardless of the actual noise type and level. Table 4 and Fig. 7 clearly show that (1) most current IP solvers, except for DPS, suffer from the robustness issue, corroborating the hypotheses made in Section 2 and (2) the peak performance of our DMPlug can lead SOTA methods by around 1dB and 3.5dB in check the compatibility of our method with ES-WMV [35),
PSNR for the two tasks, respectively. measure the detection performance via PSNR gaps, i.c., the absolute PSNR difference between the peak and the detected ES point following [35] 59, Table findicates that the detection gaps in the two exemplary tasks are nearly negligible, with PSNR gaps smaller than 0.5dB and 0. 0.2dB, respectively. This suggests that the proposed method is highly synergetic with ES-WMV.
4.3 Ablation studies
We conduct ablation studies demonstrate the flexibility of our method in terms of using multiple and non-DDIM DM priors and using alternative optimizers. (Flexibility of using DM priors) First, we explore the possibility of using different types of DMs. For super-resolution, we show in Table 5) that latent diffusion models (LDMs) (74] (5) are also synergetic with our method. Next, we study the potential of using multiple DMs together, taking BID with turbulence as an example. Using extra pretrained DMs for blur kernels and tilt maps from (1I,, our method can achieve even better reconstruction results. (Flexibility of using alternative optimizers) Here, we test the built-in ADAM [4] and L-BFGS [75] optimizers in PyTorch, with several different learning rates to solve [iq. (7) As shown Table 6, the best ADAM and L-BFGS combinations can lead to comparable performance for our DMPlug. We choose ADAM as the default optimizer because in PyTorch, optimizing multiple groups of variables with different learning rates as for the case of BID (with turbulence)- is casy to program with ADAM but tricky for L-BFGS.

---

Table 4: (Robustness and ES) Super-resolution and nonlinear deblurring on CelebA 62 with different types and levels of noise. We only show PSNR and PSNR Gapl, to save space. (Bold: best, under: second best, green: performance increase, red: performance decrease)
(Linear) Super-resolution (4x)
(Nonlinear) Non-uniform image deblurring
Gaussian
Impuise
Shot
Speckle
Gaussian
Impulse
Shot
Speckle
Low/Higg
LowiHigh
Low/iigh
Low/High
Law/High
Low/High
Low/High
Low/High
ADMM
20.17/17.97
14.28/14.52
19.97/17.82
19.42/18.11
DMPS AD
7.56
18.78/16.05
19.96/16.74
20.77/18.73
N/A
N/A
N/A
NIA
DDRM
15.45/14.79
14.82/14.14
15.31/14.59
15.46/15.03
N/A
MCG 30
17.43/15.83
16,39/15.07
17.19/15,49
17.44/16.43
12.88/12.85
13.16/13.04
13.21/13.13
13.24/13.07
TLVR
21.08/21.03
20.93/20.00
11/19/21.12
20.96/20.88
21.70/21.43
21.43/21.00
21,56/21 .24
1..53/21.36
DPS h0
25.51/24.58
24.89/23.92
25,47/24.27
25.69/24.97
23.97/23.35
23.74/23 18
24.32/23.56 58
23.45/23.61
ReSample
20
3..30/13.04
15.56/13.48
14.38/12.87
15.64/14:23
17/10/5
20.69/18.91
22.94/20.11
23.59/21.66
BKS-stylGGAN
66
N/
N/A
NIA
N/A
22.61/22.53
22.64/22.34
22.96/22.79
22.70/22.56
BKS-generic 50
N/A
N/A
N/A
N/A
16.85/15.09
14.86/13.44
16.69/14.74
17.04/1599
DMPlug (ours)
26,49/25.29
26.01/24.76
26.34/26.34
26.81/25.81
27.58/26.60
27.22/26.13
27.71/26.55
27.68/26.96
Ours vs. Best compe.
0.98/0.11
1.12/0.84
0.87/2.07
1.12/0.84
3.61/3.25
3.48/2.95
3.39/2.97
4.23/3.35
PSNR Gap.
0.36/0446
0,38/0.00
0.25/0.49
0,200.21
0.158.112
0.14/0.33
0,10/.19
0.12/0.09
Table 5: (Flexibility) Ablation on different priors for æ, and Φ on 50 cases from CelebA 662 for super-resolution (SR), nonlinear deblurting (ND),
Table 6: (Flexibility) Ablation on different optimizers and learning rates on 50 cases from CelebA 62 for super-resolution
and BID with turbulence.
Learning rate
LPIPS
PSNRT
SSIM
Prior for æ DM
Prior for ls
Prior for &
PSNRT
L-BFGS ADAM
0.1
0.375
20.22
0.479
SR
N/A
N/A
31.51
0.01
0,066
31.51
0.879
LDM
N/A
N/A
31.10
0.001
0.093
31.45
0.879
Turbelence ND
DM
NIA
N/A
31.61
1.0
0.254
28.53
0.786
LDM
N/A
N/A
30.64
0.1
0.173
31,89
0.888
DM
Simplex
None
28.21
0.225
DM
Simplex+DM
None
29.33
DM
Simplex
DM
28.35
Simplex+DM
DM
28.91
5
Discussion
In this paper, we focus on solving IPs with pretrained DMs. To deal with (Issue 1) insufficient manifold feasibility and (Issue 2) insufficient measurement feasibility of the prevailing interleaving methods, we pioneer novel plug-in method, DMPlug, and make practical in terms of computation and memory requirements. Taking advantage of benign ELTO property and integrating an ES method ES- (35], our method the first to achieve robustness to unknown noise (Issue 3). Extensive experiment results demonstrate that our method can lead SOTA methods, both qualitatively and quantitatively often by large margins, particularly for nonlinear IPs. As for limitations, our empirical results in SSection 3.1 suggest a fundamental gap between image generation and regression using pretrained DMs, that we have not managed to nail down. Also, our work is mostly empirical, and we leave solid theoretical understanding for future work.
Acknowledgements
Wang H. is partially supported by UMN CSE DSI PhD Fellowship. Wan Y. is partially supported by UMN CSE InterS&Ections Seed Grant. This research part of AI-CLIMATE:"AL Institute for Climate-Land Interactions, Mitigation, Adaptation, Tradeoffs and Economy." and is supported by USDA National Institute of Food and Agriculture (NIFA) and the National Science Foundation (NSF) National AI Research Institutes Competitive Award no. 2023-67021 1300022..99,, The authors acknowledge the Minnesota Supercomputing Institute (MSI) at the University of Minnesota for providing resources that contributed to the research results reported in this article.
10

---

References
[1] J. Janai, F. Güney, A. Behl, and A. Geiger, "Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Art," Mar. 2021, 1rXiv:1004.05519 [es]. [Online]. Available:
http://anni...rgabbb//77....5
[2] R. Szeliski. Computer Vision: Algorithms and Applications, ser. Texts in Computer Science.
Cham: Springer International Publishing, 2022. [Online]. Available: butps//innk.ppinger.omm.......
178-3-030-34372-9
[3] R. Olsen and R. Olsen, "Introduction to Remote Sensing." Jan. 2007, book Title: Re- mote Sensing from Air and Space ISBN: 9780819462350 Publisher: SPIE. [Omline] Available: howeooooo-oeeeeenngnliiiile
Sensenn--nnnnnnnnnn from- Air- and SpacSeae//Chattrrl
Introductioon-o-- to- to-Remote------eee-e-e
[4] J. Sylvester and C. L. Epstein, "Introduction to the Mathematics of Medical Imaging," in The American Mathematical Monthly, vol. 112. May 2005, p. 479, iSSN: 00029890
Issue: 5. [Online). Available:
htps://www.jsoororr/stbbbblll/l /ttpa//ee0000000/000555554440000 (4rii///111oorggniooossrrrr
[5] O. Kupyn, T. Martymiuk, J. Wu, and Z. Wang, "DeblurGAN-v2: Deblurring (Orders-of Maggnntud) Faster and Better," Aug. 2019, arxiv: wttXivv19080000 1908.03886.66 [cs]. [Online]. Available: http://axivvorrr/bbba. http///as///0...... 03826 [6] F.J. Tsai, Y.-T. Peng. Y. Lin, C.-C Tsai, and C.- W. Lin, "Stripformer: Strip Transformer for Fast Image Deblurring," Jul. 2022, arXiv:2204,044 (4472 [cs]. [Online]. Available:
S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, "Multi- Stage Progressive Image Restoration," Mar. 2021, arXiv:2102.02800 [cs]. [Online]. Available: http://araix.orr//b////100 /00p/a/ni..rrraaas2220000002
[8] M. Delbracio and P. Milanfar, "Inversion by direct iteration: An altemative to denoising diffusion for image restoration." Transactions on Machine Learning Research, 2023, featured Certification, [Online]. Available: httpss//peenrviieeee/e/oomm uat
[9] D. Gilton, G. Ongie, and R. Willett, "Deep Equilibrium Architectures for Inverse Problems in Imaging," Jun. 2021, arXiv:2102.07944 [cs, eess]. [Online]. Available: http://arxivoorg/bb//11...001
[10] Z. Luo. F. K. Gustafsson, Z. ZZao,, J. Sjölund, and T. B. Schön, "Image Restoration with Mcan-Reverting Stochastic Differential Equations," May 2023, arXiv:2301.1 1699 [cs]. [Online]. Available:
http://arxiv.org/abb/2500 11699)
[11] I.. Guo, C. Wang. W. Yang, S. Huang, Wang, H. Pfister, and B. Wen, "ShadowDiffusion: When Degradation Prior Meets Diffusion Model for Shadow Removal," Dec. 2022, arXiv:2212.04711 [cs].
[Online]. Available:
[12] J. Whang, M. Delbracio, H. Talebi, C. Saharia, G. Dimakis, and P. Milanfar, "Dehlurring via Stochastic Refinement," " Dec. 2021, arXiv:1112.02477 [cs, eess). [Online]. Available: http://arkiv.org/ab/211112
[13] S. Gao, X. Liu, B. Zeng. S. Xu, Y. Li. X. Luo. J. Liu, X. Zhen, and B. Zhang. "Implicit Diffusion Models for Continuous Super-Resolution. Sep. 2023, (.L.:2303.1699) [cs]. [Online]. Available:
http://ariiv.orr/bss2333...
[14] O. Kupyn, V. Budzan, M. Mykhailych, D. Mishkin, and J. Matas, "DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks," Apr. 2018, arXiv:1711.07064 [es]. [Online]. Available:
http://aaii..or//ab////1....7
[15] G. Ongie, A. Jalal, C. A. Metzler, R. G. Baraniuk, A. G. Dimakis, and R. Willett, "Deep Learning Techniques for Inverse Problems in Imaging," May 2020, arXiv:2005.06001 [es, eess, stat]. [Online].
Available:
[16] V. Monga, Y. Li, and Y. C. Eldar, "Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing," Aug. 2020, arXiv:1912.10557 [cs, eess]. [Online]. Available:
http://arxiv.org/abba htwp://worg/bb//911... 1912.
[17] K. Zhang, W. Ren, Luo, W.-S. Lai, B. Stenger, M.-H. Yang, and H. Li, "Deep Image Deblurring: Survey," May 2022, arXiv:2201.
[cs]. [Online]. Available: hhttp://arxiv.rrg http:/aaxii.orr/abs/20...00
[18] J. Koh, J. Lee, and S. Yoon, "Single-image deblurring with neural networks: A comparative survey." Computer Vision and Image Understanding,, vol. 203, p. 103134, Feb. 2021. [Online]. Available: https:/linkinguubeeeeeeeerrc httaaevveecooomeeeeeeee/////////
[19] H. Chung, J. Kim, M. T. Mccann, M. L. Klasky, and J. C. Ye, "Diffusion Posterior Sampling for General Noisy Inverse Problems," Feb. 2023, arXiv:2209.1 1468777 [es, stat]. [Online). Available:
http://arxiv.org//bb/2209999 14687
[20] B. SSng. S. M. Kwon, Z. Zhang, X. Hu, Q Qu, and L. Shen, "Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency," Oct. 2023, arXiw:2307.08123 [cs]. [Online]. Available:
http://arxiv.or//ab//337..777
[21] H. Chung, Kim, S. Kim, and J. C. Ye, "Parallel Diffusson Models of Operator and Image for Blind Inverse Problems." Nov. 2022, arXiv:221 [cs, stat]. [Online]. Available: hhttp//arxii.orr/aas/2]] 106666
[22] P. Dhariwal and A. Nichol, "Diffusion Models Beat GANs on Image Synthesis," " Jun. 2021, arXiv:2105.05233 (cs. stat]. [Online]. Available: http://ariivoor///ss2210...22.2
[23] J. Ho, A. Jain, and P. Abbeel, "Denoising Difffusion Probabilistic Models," Dec. 2020, arXiv:2006.1 1239 [cs, stat]. [Online]. Available: http://arrivvorrghb///00001.1111

---

[24] J. Song. C. Meng. and S. Ermon. "Denoising Diffusion Implicit Models," Oct. 2022, arXiv:2010.02502
[cs]. [Online]. Available: http://arii..rr/abb/22000.22222.
[25] R. Rombach, A. Blattmann. D. Lorenz. P. Esser, and B. Ommer. "Liih-Resolution Image Synthesis with Latent Diffusion Models," Apr. 2022, arXiv:2112.10552 [es]. [Online]. Available: http://arii..ogabbs2111..0
[26] Y. He, N. Murata, C.-H. Lai, Y. Takida, T. Ucsaka, D. Kim, W.-H. Liao, Y. Mitsufuji, J. Z. Kolter, R. Salakhutdinov, and S. Ermon, "Manifold Preserving Guided Diffusion," " Oct. 2023. [Online]. Available:
https///peeeeviio.ee///ooomm Booo
[27]
X. pling for Plug-and and-Play Image Reconstruction." [Online]. Available:
Xu and Y. Chi, "Provably Robust Score-Based Diffusion Posterior
Sam-
https://www
semanticscholar.com www.c/ooaaarrg/pppp////oaabaaa
Serbbuut--cor----eee -asee-aaeddd-ee Di%EF& WCt.sAsoonnooooo.... SOusson-Posterior- for Xu-Chi Chi
4713ac4d6336/6699a (wG.)388666677771666666007000060
source=alert umail&&um conlent-iibaryFoodder
utm www..aaaaiggnllttttaallsssssss. &attaaallllaallAAAAttttaaattaa &
haci:uuuttmmmmmmmm-me--- medium=
[28] Y. Zhu, K. Zhang, Liang, Cao, B. Wen, R. Timofte, and L. Van Gool, "Denoising Diffusion Models for Plug-and-Play Image Restoration," May 2023, arXiv:2305.0899 20.2990000 [cs, eess]. [Online]. Available:
http://aniiv..r///b//2.........
[29] X. Meng and Y. Kabashima, "Diffusion Model Based Posterior Sampling for Noisy Linear Inverse Problems." Jan. 2024, arXiv:2211.1343 [cs, math, stat]. [Online]. Available: http
Varxiv.og//bb/2211.1222
[30] H. Chung, B. Sim, D. Ryu, and J. C. Ye, "Improving Diffusion Models for Inverse Problems using Manifold Constraints," Oct. 2022, arXiv:2206.0090 009041 [cs, stat]. [Online]. Available: http://arxw..rr//h//2220
[31] X. Li, Y. Ren, X. Jin, C. Lan, X. Wang. W. 7eng, X. Wang, and Z. Chen, "Diffusion Models for Image Restoration and Enhancement A Comprehensive Survey," Aug. 2023, arXiv:2308.093: [es]. [Online]. Available: http://aaxxx..rr//bb///33 09388
[32] B. Kawar, M. Elad. S. Ermon, and J. Song. "Denoising Difftusion Restoration Models," " Oct. 2022,
arXiv:2201.1199: [cs. eess]. [Online]. Available: http://arxii.orr/abb/22....111
11793
[33] Y. Wang. J. Yu, and J. Zhang, "Zero-Shot Image Restoration Using Demoising Diffusion Null-Space Model," Dec. 2022, arXiv:2212.00490 [cs]. [Online], Available: htpp:/arxiv..r/aas/221111
[34] G. Liu, H. Sun, J. Li, F. Yin, and Y. Yang. "Accelerating Diffusion Models for Inverse Problems through Shortcut Sampling," May 2024, arXiv:2305. 16965
[es, eess). [Online). Available:
http://arkiv.or//bbs//// 16965
[35] H. Wang. T. Li, Z. Zhuang, T. Chen, H. Liang, and J. Sun, "Early Stopping for Deep Image Prior," Dec. 2023, arXiv:21 12.1102.0444 [cs, eess). [Online]. Available: http://aaxiv.orraas22212..2
[36] A. Hyvärinen. "Estimation of Normmaame Statistical Models by Score Matching," Journal of MMachine Learning Research. vol. 6. no. 24. pp. 695-709, 2005. [Online]. Available: htwl:j/l.orr//agesss////aaaaaaie
[37] Y. Song and S. Ermon, "Generative Modeling by Estimating Gradients of the Data Distribution," Oct. 2020, arXiv:1907. .05600 [cs, stat]. [Online]. Available: http://arxiv..r//abs// 1907.056000
[38] J. Song, C. Meng, and A. Vahdat, "Denoising diffusion models: A generative learning big bang," Nov.
2023, CVPR 2023 Tutorial. [Online]. Available: https::/ewpr2223-uuuur tutorrall diffusioo--mddeee.giiee models.gitubb.ii/
[39] Z. Kadkhodaie and E. Simoncelli, "Stochastic Solutions for Linear Inverse Problems using
the Prior Implicit in a Denoiser." in Advances in Neural Information Prucessing Systems. vol. 34. Curran Associates. Inc., 2021. pp. 13242-13 254. [Online]. Available: htips: Abstract.htm! [40] Y. Song. J. Sohl- Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, Secore-Basee Generative Modeling through Stochastic Differential Equations," Feb. 2021, arXiv:2011.13456 [cs, stat]. [Online]. Available: httpp:/aarii..rr/////201111....1
[41] J. Choi, S. Kim, Y. Jeong, Y. Gwon, and S. Yoon, "ILVR: Conditioning Method for
Denoising Diffusion Probabilistic Models," Sep. 2021, NtV::2008002933 [es]. [Online]. Available:
http://axxiv.or//abs///l 02938
[42] Y. Sanghvi. Y. Chi, and S. Chan, "Kernel diffusion: An alternate approach to blind deconvolution."
arXiv preprint arXiv:2312.02331 2023.
[43] D. Hendrycks and T. Dietterich, "Benchmarking Neural Network Robustness to Common Corruptions and Perturbations." Mar. 2019, arXiv: 12261 (es, stat]. [Online], Available: http://arxiv.orr/abs//
[44] F. Li, F. Fang, Z. Li, and T. Zeng. "Single image noise level estimation by artificial noise," Signal Processing. vol. 213, P. 109215, Dec. 2023. [Online]. Available: https
[45] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li. and J. Zhu, "DPM-Solver: A Fast ODE Solver for Diffuusion Probabilistic Model Sampling Around 10 Steps." " Advances in Neural Information Processing Systems,
vol. 35, pp. 5775 5787, Dec. 2022. [Online]. Available: httpss//proceeedings.. wwwpdiinnnnnuuuis..oo wwt.rip..ccppprrr.iiiie.e 2022/hash/260al Maroo0alaacceaaaaaa Baddeaaddddddddddeeeeeeeeee
Conference.htmm)
[46] K. Zheng, C. Lu, J. Chen, and J. Zhu, EDPM-Solver.s Improved Diffusion ODE Solver with Empirical Model Statistics." Advances in Neural Information Processing Systems, vol. 36. pp.
12

---

53502-55 Dec. 2023. [Online]. Available: http://occeeddnngs.euuuurr..opoe filesspppee//////2/2233 http://aaakk99994b44///b///ee/ee
[47] D. P. Kingma and J. Ba, "Adam: A Method for Stochastic Optimization," Jan. 2017, arXiv:1412.6990 [cs].
[Online]. Available: http://aaxiv.org/ab///111
[48] A. Creswell and A. A. Bharath, "Inverting The Generator Of A Generative Adversarial Network," Nov.
2016, arXiv:1611 611.66611444 [es]. [Online]. Available: http://arxiv.or//bb///dd 1.06444
[49] J.-Y. Zhu, P. Krähenbühl, E. Shechtman, and A. A. Efros, "Generative Visual Manipulation on the Natural
Image Manifold," Dec. 2018, arXiv:1609. 00552 [cs]. [Online]. Available: http://rrxiv http:///xiiv.rg//bb/////// wwtp///////0////// 1109666333333 005552
[50] X. Pan, X. Zhan, B. Dai, D. Lin, C. Loy, and Luo, "Exploiting Deep Generative Prior for Versatile
Image Restoration and Manipulation." " Jul. 2020, arXiv:2003. 13659 [es, eess]. [Online]. Available:
http://arrix.or//aa/2000... 13659
[51] X. Su. J. Song. C. Meng. and S. Ermon, "Dual Diffusion Implicit Bridges for Image to- Im-age Translation." Mar. 2023, arXiv:2203. 0800000888222111 [es]. [Online]. Available: http:://axiiv wttp:/www..ggg/////../3..
0083882220
[52] A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen- Or. Srropp-to-Prompe
Image Editing with Cross Attention Control," Aug. 2022, arXiv:2208.0166 [cs]. [Online]. Available:
http://arxivvorr//baee wwtp:///////....ooooo 01626
[53] G. Kim, T. Kwon, and J. C. Ye, "DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation." Aug. 2022, arXiv:2110.02711 [es]. [Online]. Available: http://arxiv.org/bb//11100000000
[54] R. Mokady, A. Hertz, K. Aberman, Y. Pritch, and D. Cohen-Or, "Null-text Inversion for Editing
Real Images using Guided Diffusion Models," " Nov. 2022, 1xXxx22211009990 [cs]. [Online]. Available:
http://arxxx.or//abb//22 1.09994
[55] B. Wallace, A. Gokul, and N. Naik, "EDICT: Exact Diffusion Inversion via Coupled Transformations." "
pp. 22 232-222 541, 2023. [Online]. Available: https://openaccees..toe hwwpew/w.oom/cooooc//////e/////e
Wallace_EDIC" Bee__IDCTTFEatttt___ _ifffutiffuiion______ Diffusion_Inversionn via vii__vvaa__ppeeee Coip_oo_u__daaaee Trannsoomattinns_ccco SatpiiormaatttocccccP.e SUPPR_2033_p___ paper.html
[56] K. Gregor and Y. LeCun, "Learning fast approximations of sparse coding." in Proceedings of the 27th
International Conference on International Conference on Machine Learning, ser. ICML' 10. Madison,
WI, USA: Omnipress, Jun. 2010, pp. 399-406.
[57] D. Ulyanov, A. Vedaldi, and V. Lempitsky, "Deep Image Prior," International Journal of Computer Vision, vol. 128, no. 7, pp. 1867-1888, Jul. 2020, arXiv:1711.10225 [cs, stat]. [Online]. Available:
http://axxiv.orr/aha 1wt.b/////111.......... 10925
[58] T. Li, Z. Zhuang. H. Liang, L. Peng, H. Wang, and Sun, "Self-Validation: Early Stopping for Single-Instance Deep Generative Priors," " Oct. 2021, 10. 12271 [es, eess]. [Online]. Available:
hhtp://arkiv.or//ab/// 12271
[59] T. Li. H. Wang, Z. Zhuang, and J. Sun, "Deep Random Projector: Accelerated Deep Image Prior," pp.
18 176-18 185, 2023. [Online]. Available: httpss//openaccess..t htt.ee/...oomooeeeee////////000a
Deep_ BesdpR_andmm____ Projector Accelerated Deep_ Image Cihor__ 222__222_________ paeees_ppp.hhm
[60] Z. Shi. P. Mettes, S. Maji. and C. G. M. Snock, "On Measuring and Controlling the Spectral Bias of the Deep Image Prior," Dec. 2021, arXiv:2107.01125 [cs, eess]. [Online]. Available: http://arriv.onaaaa/1111...11111
[61] Z. Zhuang. T. Li. H. Wang, and J. Sun, "Blind Image Deblurring with Unknown Kernel Size and Substantial Noise." International Journal of Computer Vision, vol. 132, no. 2. pp. 319-348, Feb. 2024, 09483
arXiv:2208. 0998809993380 [cs, eess]. [Online]. Available: hatt://aaxiv.orr//bs//22
[62] 7. Liu, P. Luo, X. Wang. and X. Tang, 'Deep Learning Face Attributes the Wild," Sep. 2015, arXiv:1411.7766 [cs]. [Online]. Available: http://anniv.or//b///1111..1
[63] T. Karras, S. Laine, and T. Aila, "A Style-Based Generator Architecture for Generative Adversarial
Networks," Mar. 2019, 1r1iv:1812.00000 www::111204445 [cs, stat]. [Online]. Available: http:///anxiv.rr//ba//ir 11.00998
[64] F. Yu, A. Seff, Y. Zhang. S. Song. T. Funkhouser, and J. Xiao, "LSUN: Construction of a Large-scale Image Datasct using Deep Learning with Humans in the Loop," Jun. 2016, 1r06v:1506.00m 1506.0336 (cs). [Online]. Available: hhttp://axxivorr//art wwt.:or//////...03...33
[65] R. Zhang. P. Isola, A. A. Efros, E. Shechtman, and O. Wang, "The Unreasonable Effectiveness
of Deep Features as a Perceptual Metric," Apr. 2018, 1001.0080000392 [cs]. [Ouline]. Available: http://arxiv.or//bbaa http://.orr/abb////...323/ 080.03392
[66] P. Tran, A. T. Tran, Q Phung, and M. Hoai, "Explore Image Deblurring via Encoded Blur Kernel Space." "
pp. 11 956-11 963, 2021. [Online]. Available: https://openaccess..e wwwpppennnaeesss.eee...ooooooooe wttpeeee...ooooon//////eera I/tm//
Tran Explore_Image Deblurring via Encoded Blur Kernel Space CVPR 2021 2021__aperrh____ paper.html
[67] D. Jin, Y. Chen, Y. Lu, J. Chen, P. Wang, Z. Liu, S. Guo, and X. Bai, "Neutralizing the impact of complex scene imaging via deep learning," " Nat. Mach. Intell., vol. 3, no. 10, pp. 876-884, 2021. [Online]. Available: https:///oii. org/10. wttp//...0.///222222.2 00014422222-2100--999
atmospheric turbulence
[68] D. Krishnan and R. Fergus, "Fast Image Deconvolution using Laplacian Priors." in Advances in Neural Information Processing Systems, vol. 22. Curran Associates, Inc., 2009. [Online]. Available: httppsscccppppeeells//pr////////
1d016ffcbf3 I htwph/hhccccee--taaa.....t. Abstract.html [69] S. H. Chan, "Tilt- Blur or Blur-then- then--Tt?t Clarifying the Atmospheric Turbulence Model," IEEE
https:/pppprr..mit wwtps//ppppppcnen..//epesm
Signal Processing Letters, vol. 29, pp. 1833-1837 2022, arXiv:2207.0633 [eess]. [Online]. Available:
http://arkii.orr/aa////77....
13

---

[70] J. Pan, D. Sun, H. Pfister, and M.-H. Yang, "Deblurring Images via Dark Channel Prior," IEEE Transactions Puttern Analysis and Machine Intelligence, vol. 40, no. 10. pp. 2315-2328, Oct. 2018. conference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence. [Online]. Available: https///iceeplorriicc.c www.:iic..og/oommmee/////s.....
[71] J. Pan, Z. Hu, Z. Su, and M.-H. Yang. "L_0 -Regularized Intensity and Gradient Prior for Deblurring Text Images and Beyond," IEEE Transactions Pattern Analy- and Machine Intelligence, vol. 39, no. pp. 342-355, Feb. 2017, conference
Name: IEEE Transactions on Pattern Analysis and Machine Intelligence. [Online]. Avail-
able:
[72] D. Ren, K. Zhang. Wang. Q. Hu, and W. Zuo. "Neural Blind Deconvolution Using Deep Priors." Mar.
2020, arXiv:1908.02: 00219999002197 [es]. [Online]. Available: hntp://arxiv.oorgabb/////// 1900///9990000997777
[73] S. H. Chan, Wang, and O. A. Elgendy, "Plug-and-Play ADMM for Image Restoration: Fixed
Point Convergence and Applications." Nov. 2016, arXiv:1605.017
[cs]. [Online]. Available:
[74] A. Vahdat, K. Kreis, and J. Kautz, "Score-based generative modeling in latent space," Advances in Neural
Information Procexsing Systems. vol. 34, pp. 287-11 302, 2021.
[75] D. C. Liu and J. Nocedal, "On the limited memory BFGS method for large scale optimization," "
Mathematical Programming, vol. 45, no. 1, pp. 503 528, Aug. 1989. [Online]. Available:
[76] H. Chihaoui, A. Lemkhenter, and Favaro, "Zero- shot Image Restoration via Diffusion Inversion," Oct. hww.oruummmm////gggggggoM
2023. [Online]. Available: http:://openneviee.ee/ffrrmmm
[77] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, "Gans trained by a two time-scale update rule converge local nash equilibrium," Advances in neural information processing systems, vol. 30, 2017.
[78] P. Chakrabarty and S. Maji, "The Spectral Bias of Deep Image Prior." Dec. 2019, 1:X.00912.00800 [cs]. [Online]. Available: http://xxiv..rr//aaaee (ttpps/9912.099555
[79]
D. L. Ruderman, "The statistics of natural images," Network: Computation in Neural Systems, vol. 5, no. 4,
pp. 517-548, Jan. 1994, [Online]. Available: https:/wwww.anndoollnnc.ccm http:///oiiic.cooooo/////////// 1088/0954 898X
$_4_006
[80]
Z. Dou Y. Song, "Diffusion Posterior Sampling for Linear Inverse Problem Solving: Filtering Perspective," Oct. 2023. [Online]. Available:
[81] L. Rout, N. Raoof, G. Daras, C. Caramanis, A. G. Dimakis, and S. Shakkottai, "Solving Linear Inverse
Problems Provably via Posterior Sampling with Latent Diffusion Models," Jul. 2023. [Online]. Available:
https:/arki..oggddd//333......11
14

---

A Remarks on the concurrent work (76
We became aware of the unpublished concurrent work, SHRED [76.. when we were finalizing the details of our method in mid Jan 2024 (We started the current project in Oct 2023). Although the core idea of SHRED is the same as that of our DMPlug, there are a few crucial differences between (76] and the current paper. (1) Motivation: [76] aims only at addressing manifold feasibility, while the current paper targets manifold feasibility, measurement feasibility, and robustness to unknown noise, simultaneously: (2) Integral components: Since achieving robustness unknown noise is part of our goal, the ES-WMV ES strategy is integral to our method, besides the unified optimization formulation Eq. (7) shared with [76]: (3) Key hyperparameters: To address the computational and memory bottleneck induced by reverse steps, we use only 3 reverse steps for all IPs we test, vs. the 10 or more reverse steps used in 76.. So, our current setting makes the method much more practical; (4) Flexibilityy: Our framework allows for the use of more than one pre-traie vre-trained DM prior when available, as shown in Section 4.3] while [6] uses only single DM prior as proposed in their Eqs. (17) to (19); (5) Experimental evaluation: 76) focuses linear IPs, including inpainting, super-resolution, compressive sensing, and one nonlinear IP-blind deconvolution, i.e., blind image deburring (BID). We focus our evaluation nonlinear IPs, including nonlinear deblurring, BID, and BID with turbulence, besides linear IPs (inpainting and super-resolution) Moreover, [76] measures performance only by perceptual metrics, LPIPS (65] and FID [77]. while measure performance by both the perceptual metric LPIPS and the classical metrics PSNR and SSIM. In addition, [76] does not even compare their method with clear SOTA methods that use pre- pre-trained DM, e.g., DPS 19). which they obviously were aware of (cited in the paper), whereas our comparison more comprehensive.
B Spectral bias of our DMPlug
1.0
frequency (1.jower0
Frequency band (2)
0.8
Frequency band (3)
band
Frequency band highest)
Iteration
(1)
(2)
Figure 8: (1) The spectral bias of DMPlug with the ADAM solver 47 DMPlug with the L-BFGS solver (75). The IP we experiment with here is regression against a noisy image with Gaussian noise at 0.08, i.e., the same denoising problem in Fig. (2).
(2) The spectral bias of
One may wonder why the ELTO phenomenon occurs. Here, we borrow the ideas of spectral biases and spectral analysis from the DIP literature 78 60 59 61 shed some light on this. The theory of spectral biases for DIP states that low-frequency components are learned much faster than high- frequency components during DIP learning. Spectral biases in DIP lead to the ELTO phenomenon in DIP: because natural images are typically wowffrequency dominant (79., the different learning paces imply that DIP learns mostly the desired image content (low-free frequency image content plus the low-frequency part of noise) in the early stage, but gradually picks up the high-frequency part of noise the late stage, resulting in performance degradation after a certain quality peak. Spectral analysis provides quantitative visualization of spectral biases.
Here, we perform a similar spectral analysis of our DMPlug learning process to demonstrate its spectral biases, which causes the ELTO phenomenon. To measure spectral biases, we follow (59 61 and use frequency band errors (FBEs). For an groundtruth image æ and its estimate ă the calculation of this metric goes as follows. First, calculate the pointwise relative error pointwise in the Fourier
15

---

domain, i.e., [F(x) - F()) [()) Then, we divide the Fourier frequencies into five radial bands, compute the bandwise mean errors, Le., the frequency-bane errors (FBEs).
We visualize the evolution of FBEs of DMPlug over all five frequency bands in Fig. The disparate learning paces across the frequency bands are evident: the lowest-frequency band is learned much more rapidly than the other bands, which is consistent between two different optimization solvers. With spectral biases similar those in DIP, we can explain the ELTO phenomenon in DMPlug following the argument above for DIP.
C Setup details of the inverse problems we test
For super- nuper-resolution, inpainting. and nonlinear deblurring, we use the forward models from two of main competing methods [19 20 for BID and BID with turbulence, we follow the forward
models from BlindDPS 22 the SOTA DM-based method. Moreover, following [19] (0), all the
measurements contain additive Gaussian noise with 0.01. Our loss é by default is the MSE loss.
C.1 Super-resolution
In noisy image super-resolution, the goal is to reconstruct clean RGB image æ from noisy down- sampled version = D(x) + n. where D()) [0. 1| + [0, 1 |jxxXx is downsampling operator that resizes an image with dimensions t/d by tW by the factor and n models additive noise. We set f = 4 in SSection 4 To ensure fair comparison, we do not include any explicit regularization terms in the formulation:
(Super-resolution) z" = arg ming fly. D(R(=))),
=== =R=")=²)..
(8)
C.2 Inpainting
In noisy image inpainting, clean RGB image x [0, 1|xxHX is only partially observed and then contaminated by additive noise n, described by the forward model y m @ x + n. where m {0, x is binary mask and denotes the Hadamard product. Given y and m. the goal is to reconstruct æ. Following [20), the masks for the three channels of m are identical, and 70% of the mask values are randomly set O. To ensure fair comparison, we do not include any explicit regularization terms in the formulation:
(Inpainting) arg ming E(y, m @ R(z)).
x'= = =R()).
C.3 Nonlinear deblurring
We follow the setup in 19 which is inspired by [66]. Recently, (66] has proposed to learn data-driven blurring models from paired blurry-sharp training of the form {(),, 2x.) J= through
a², , = arg min B
Σ
y - Fa (2;; Ga(,,, y:))1l,
(10)
where Gg(..)) predicts the latent blur kernel associated with the input blurry-sharp image pair, and ( models real-world nonlinear blurring process given the input smage-kernel pair. To study the performance of its DPS method on nonlinear IPs, [9] proposes the following nonlinear deblurring problem with known Gaussian-shaped kernel:
y Far - (x, g) + n, where g R64x04 Gaussian-shaped with o 3.0.
(11)
The task here is recover from y and the forward model Fa (+.g). Our formulation follows [19
and does not include extra regularizers:
(Nonlinear deblurring) z" arg ming E(y.Fa- (R(2),
x" = R(z').
(12)
C.4 Blind image deblurring (BID)
BID is about recovering a sharp image x from y = k + n where * denotes the linear convolution and the spatially invariant blur kernel is is also unknown. It is nonlinear IP because the forward
16

---

model A(k, x) = k x: is nonlinear. In solving BID under the regularized data-fitting framework in Eq. (1) l or ε data-fitting loss, and the sparse gradient prior to z enforced by (z( = ||
or (x) = || |V>]| 2|2 are typically used. In addition, because of the scale ambiguity in the
forward model, i.e., k * x (ak) * (2x) for any a > O, the scale of & is often fixed by requiring k
to be on the standard simplex (i.e., & 2 0, 1 = 1) or on the sphere (i.e., ||k||| = 1). [18 61] provide detailed coverage of these priors and regularizers. For our experiments here, we follow the settings BlindDPS [9]] all the blur kernels are simulated with the size of 64 x 64; the standard
deviation of the Gaussian kernels is set to 3.0 and the intensity of the motion blur kernels is adjusted to 0.5. It is important to mention that BlindDPS [21] uses pretrained DMs not only for images but also for blur kernels, giving it an unfair advantage over other methods. But to ensure fair comparison with other competing methods that only use data-driven image priors, we only use pretrained DMs for the image, plus the typical simplex constraint that is also used in BlindDPS:
BID)= z", k' arg ming, Eyy, (k) R(z)),
x² R(z"),
(13)
where SoftMax (k) leads to kernel estimate that lies on the standard simplex.
C.5 BID with turbulence
BID with turbulence often arises long-range imaging through atmospheric turbulence. The forward
imaging process can be modeled as a simplified "tilt-then-blur" process, following [69]: y = k * To (x) + n, where the tilt operator (-) applies the spatially varying vector field Φ to the image a so that pixels of x are moved around according to the vector field Φ. i.c. To(x) (z) pp: + o. = x p.] where P; 's are the pixel coordinates. In BID with turbulence, none of the k. x. o is known and the task is to jointly estimate them from the measurement y. So, it is clear that this BID variant is strictly more difficult than BID itself. We follow BlindDPS (21) for data generation: the blur kernel comes from the point spread function (PSF) and takes a Gaussian shape with standard deviation 3.0; the tilt maps are generated as iid Gaussian random vectors over the pixel grid. Similar to the BID case, DPi [21] use pretrained DMs for all three objects: image, kernel, and tilt map, unfair to other methods. We again use pretrained DMs for the image only. For the kernel, we again only impose the simplex constraint through Max activation. For the tilt map, inspired by the fact that the tilt vectors are Gaussian distribution with a small standard deviation and set an extremely small learning rate. Our final formulation for this task is
random vectors, we initialize the map from a zero-mean
(BDD with turbulence) z", k arg min Ely, SoftMax(k) To(()=))), x' =R(z"). (14)
D More implementation details
D.1 Noise generation
we simulate four types of noise, with two intensity levels for each type. The detailed information is as follows. Gaussian noise: zero-mean additive Gaussian noise with variance 0.08 and 0.12 for low and high noise levels, respectively; Impulse noise: also known as salt-and-pepper noise, replacing each pixel with probability p E [0, 1] in a white or black pixel with half chance each. Low and high noise levels correspond to p = 0.03 and 0.06, respectively; Shot noise: also [0, 1]. the noisy pixel is Poisson distributed with the rate
Following 43
known as Poisson noise. For each pixel, Az, where is 60 and 25 for low and high noise levels, respectively; Speckle noise: for each pixel [0, 1]. the noisy pixel is z(1 + e), where is zero-mean Gaussian with variance level 0.15 and 0.20 for low and high noise levels, respectively.
D.2 Additional implementation details of our method
We employ the following setup for our methods across all IP tasks. For R(-), we take the standard
pretrained DMs from 22 Fl and [41 3 and use the standard DDIM (24) sampler with only 3 reverse
steps based on Fig. also use pretrained latent diffusion models (LDMss) [25 to obtain the
httpp///hthabbbcom////////t////s -veeee------- oile
17

---

We use the pretrained DMs for blur kernels and tilt maps from 21 to obtain the results reported in Table For ((-). we choose the standard MSE loss. For 2(.), we use the typical explicit regularizers for each task to make the comparisons fair. The default optimizer is ADAM and the learning rate (LR) for is x -2: for BID (with turbulence), the LRs for blur kernel and the tilt map are 1 x 10-1 and 1 10-7, respectively. For the maximum numbers of iterations, we set 5, 000 and 10, 000 for linear and nonlinear IPs, respectively, which empirically allow good convergence. We perform all experiments NVIDIA A100 GPUs with 40GB memory each.
results reported in Table 5
Algorithm 3 DMPlug+ES- WMV for solving general IPs
Input: diffusion steps T, y. window size W, patience P. empty queue Q. iteration counter
0, VARmin 00
1: while not stopped do
2: for i = T to do
3:
$ + (2
4:
26 - Va. [2
VI a,
5:
z DDIM reverse with 25,
6:
end for
7:
Update z from 27 via gradient update for Eq. (7)
8:
push R ((2tt) to Q. pop queue if [Q] W
9:
1| W then
10:
compute VAR of elements in @ via Eq. (15)
if VAR < VARRmin then VARmin < VAR, x² 2T
12:
13:
end if if VARmiu stagnates for P iterations then stop and return
14:
15:
16:
end if end if
17:
18:
e= e +++1
19:
end while
Output: Recovered object R(=*)
Early stopping (ES) using ES-WMV The "early-leaminngthen-- -hve-oovrrfiitting (ELTO) phenomenon has been widely reported in the literature on deep image prior (DIP) [57] 35 and ES-WMV (35) an ES strategy that achieves the SOTA ES performance for DIP applied various IPs. ES-EMV, the running variance (VAR) is defined as:
W
W-1
VAR(t) W
Σ 0
|
1/W
Σ 1=0
(15)
where W the window size and x denotes the recovery at iteration
335 observes that the first major valley of the VAR curve is often well aligned with the peak of the PSNR curve. Based on this, (35] introduces an online algorithm to detect the first major valley of the VAR curve: the minimal VAR does not change over P consecutive steps, i.e., the VAR does not decrease further over P consecutive steps, the iteration process is stopped. The combined algorithm. ES- ZS-WWM-iintegatted
DMPlug, described in Algorithm For implementation,
use the official code of WM For super-resolution with unknown noise, we set its patience number as 100 and its window size as 10; for nonlinear deblurring with unknown noise, we set its patience number as 300 and its window size as 50. Table 4iidiictess that the detection gaps in the two exemplary tasks are nearly negligible, with PSNR gaps smaller than 0.5dB and 0.2dB, respectively. This suggests that our DMPlug highly synergetic with ES-WMV.
D.3 Implementations of competing methods
We use the default code and settings of each competitor's official implementation, listed below.
httt::://gittubb.com/muuuuuuuu http://suuuuun/E/aaayaa for
18

---

ADMM-PaP 13
DMPS 29
DDRM 32
DPS
19
MCG
ILVR
41
ReSample 20:
BKS
66]
SelfDeblur http:///ghthhhccoo///caa/////bbt
DeBlurGANv2 http://///////tooaoo///////o/aaa
Stripformer http:://shhhaaa..oo////0///////0
MPRNet 7
Pan-DCP 70
Pan-fo
BlindDPS htgp:
TSR- 67 http:://coeecceeccccccc///opsoes
80
https:
DifPIR
ht https: htta:///gthhb..com/yyaaahhhh
E
More experiment results
E.1
Computational Efficiency
find that the memory usage of our method is in the same order that of most other algorithms. When it comes to speed, our implementation using L-BFGS is significantly faster than the ADAM version. Although our method with L-BFGS remains somewhat slow, it surpasses the performance of ReSample. We will explore further acceleration of our method in future work.
As shown in Table 7
Table 7: Wall-clock time (seconds) and memory usage (GB) of various algorithms for super- resolution on the CelebA 62 dataset, tested on a single NVIDIA A/00 GPU. (Ours-A: ours with ADAM, Ours-L: ours with L--FFG)
ADMM-PUP
DMPS
DDRM
MCG
DPS
DDNM
FPS
DifPIR
ReSample
Ours-A
Ours-L
635
Memory 0.42
5.10
4.99
2,80
2.79
5.11
20.63
1.44
4.87
6.59
6.74
E.2 Quantitative results
to generate measurements, we use bicubic downsampling for super-resolution (4x) and random mask with 70% missing pixels for inpainting: all measurements contain additive Gaussian noise with a 0.01. We compare our method with Plug- and-Play using ADMM (ADMM-Pai) PPPP) [7 and several SOTA DM-based methods: Diffusion Model based Posterior Sampling (DMPS) [29]. Denoising Diffusion Destoration Models (DDRM) [32]
Super-resolution & inpainting Following 20 19
Manifold Constrained Gradients (MCG) 330, Iterative Latent Variable Refinement (ILVR) 41]
Diffusion Posterior Sampling (DPS) 19). and ReSample (20]. The quantitative results are reported in Tables to I0 with qualitative results visualized in Fig. I It is clear that our DMPlug consistently outperforms all competing methods on the three datasets, both quantitatively and qualitatively. Specifically, while there is significant improvements in terms of LPIPS, the proposed method can lead the best SOTA methods by about 2dB in PSNR and 0.02 in SSIM on average, respectively.
E.3 Qualitative results
F
More early stopping results
19

---

Table 8: (Linear IPs) Super-resolution and inpainting with additive Gaussian noise (o 0.01).
(Bold: best, under: second best, green: performance increase, red: performance decrease)
Super-resolution (4x)
Inpainting (Random 70%)
CelebA
62
(256 x 256)
FFHQ 63
(256 x 256)
CelebA
62
(256 x 256)
FFHQ
63 (256 x 256)
LPIPS.
PSNR
SSIM
L.IPS..
PSNR1
SSIM
LPIPS]
PSNR
SSIM1
LPIPS.
PSNR
SSIM
ADMM PnP 28
0.217
26.99
0.808
0.229
26.25
0.794
0.091
31.94
0.923
0.104
30.64
0.901
DMPS
20
0.070
28.89
0.848
0.076
28.03
0.843
0.297
24.52
0.693
0.326
23,31
0.664
DDRM
12
0,226
26.34
0.754
0.282
25.11
0.731
0.185
26.10
0.712
0.201
25.44
0.722
MCG 30
0.725
19.88
0.323
0.786
18.20
0.271
1,283
10.16
0.049
1.276
10.37
0.050
IVR
0.322
21.63
0.603
0.360
20.73
0.570
0.447
15.82
0.484
0.483
15,10
0.450
DPS 19
0.087
28.32
0.823
0.098
27.44
0.814
0.043
32.24
0.924
0.046
30,95
0.913
ReSample 220
0,080
28.29
0.819
0.108
25.22
0.773
0.039
30.12
0.904
0.044
27.91
0.884
DMPlug (ours)
0.067
31.25
0.878
0.079
30.25
0.871
34.03
0,038
33.01
0,931
Ours vs. Best compe.
-0.003
+22.6
+0.030
+0.003
- -2,2
+-00088
-0. -0,000
+1.79
+0.012
-0.006
+2.00
+00011
Table 9: (Linear IPs) Super-resolution and inpainting on LSUN-bedroom 64 with additive
Gaussian noise (a = 0.01). (Bold: best, under: second best, green: performance increase, red: performance decrease)
Super-resolation (4x)
Inpainting (Random 70%)
LPIPS
PSNRT
SSIM
LPIPS
PSNRT
SSIM
ADMM-PaP
7A
0.358
24.09
0.728
0.223
27.75
0.842
DMPS
0.172
0.320
DDRM 52
0.415
22,73
0.644
0.273
23,66
0.673
MCG 30
1.069
15.00
0.176
1.271
11.15
0.055
ILVR 41
0.476
18.89
0.445
0.581
14.79
0.407
DPS
0.195
25.10
0.737
0.094
28.95
0.868
ReSample
20
0.137
24.72
0.762
0.069
26.43
0.861
DMPlug (ours)
0.136
26.93
0.801
0.064
30.15
0.905
Ours Best compe.
-0.001
+1.39
+-0.029
-0.005
+1.20
+0.0037
Table 10: (Linear IPs) Super-resolution and inpainting on CelebA 62 with additive Gaussian noise (a 0.01). (Bold: best, under: second best, green: performance increase, red: performance decrease)
Super-resolution (1x)
Inpainting (Random 70%)
LPIPSL
PSNRT
SSIM
LPIPS}
PSNR1
SSIMT
FPS X8
0.149
29.12
0.858
0.064
32.06
0.924
DiAPIR 28
0.203
31.55
0.857
0.219
31.22
0.866
DDNM 32
0.193
29.21
0.836
0.224
27.89
0.799
PSLD 3
0.243
26.45
0.682
0.213
27.65
0.785
DMPlug (ours)
0.067
31.25
0.878
0.039
34.03
0.936
Ours vs. Best compe.
0.0822
- 0.30
+00020
0.025
+1.97
4000012
20

---

Measurement
DMPlug (ours)
ReSample
Bedroom
CelebA
FFHQ
Figure 9: (Linear IP) Visualization of sample results from the plug-in method (Ours) and competing methods for 4x super-solution. All measurements contain Gaussian noise with a 0.01.
21

---

Measurement
Reference
DMPlug (ours)
ReSample
Bedroom
CelebA
FFHQ
Figure 10: (Linear IP) Visualization of sample results from the plug- in method (Ours) and competing
methods for inpainting (random 70%). All measurements contain Gaussian noise with o 0.01.
22

---

Measurement
Reference
DMPlug (ours)
DPS
ReSample
ILVR
BKS- BKS-styleGAN
BKS generie
Bedroom
CelebA
FFHQ
Figure 11: (Nonlinear IP) Visualization of sample results from the plug-in in method (Ours) and competing methods for nonlinear deblurring. All measurements contain Gaussian noise with = 0.01.
23

---

Measurement
Reference
DMPlug
BlindDPS
Stripformer
DeBlurGANv2
MPRNet
F Fare S
F ff
CelebA
FFHQ
Figure 12: (Nonlinear IP) Visualization of sample results from the plug-in method (Ours) and competing methods for BID (motion). All measurements contain Gaussian noise with o 0.01.
24

---

Measurement
Reference
DMPlug
BlinDDPS
Stripformer
DeBlurGANv2
MPRNet
CelebA
FFHQ
Figure 13: (Nonlinear IP) Visualization of sample results from the plug-in method (Ours) and competing methods for BID (Gaussian). All measurements contain Gaussian noise with a 0.01.
25

---

Measurement
Reference
DMPlug (ours)
BlindDPS
Stripformer
TSR-WGAN
DeBlurGANv2
MPANet
CelebA
FFHQ
Figure 14: (Nonlinear IP) Visualization of sample results from the plug-in method (Ours) and competing methods for BID with turbulence. All measurements contain Gaussian noise with = 0.01.
26

---

Measurement
Reference
DMPlug (ours)
DPS
ReSample
ADMM-PunP
DMPS
Gaussian
Impulse
Shot
Speckle
Figure 15: (Robustness) Visualization of sample results from the plug-in method (Ours) and compet- ing methods for 4x super-resolution. We generate measurements with four types of nousenGaussian. impulse, shot, and speckle noise across two different noise levels: low (level-1) 1) and high (level-2), following 43 (top: low-level noise; bottom: high-level noise)
27

---

DMPlug (ours)
ReSample
BKS-stylGGAN
BKS-generic
Gaussian
Impulse
Shot
Speckle
Figure 16: (Robustness) Visualization of sample results from the plug-in method (Ours) and competing methods for nonlinear deblurring. We generate measurements with four types of noise- Gaussian, impulse, shot, and speckle noise-acma two different noise levels: low (level-1) and high (level-2), following 43 (top: low-level noise; bottom: high-level noise)
28

---

Gaussian
Impulse
Shot
1ir S 111
Speckle
Figure 17: (Early stopping) Our DMPlgu with ES-WMV 35 for 4x super-resolution with different types and levels of noise. (top: low-level noise; bottom: high-level noise). Red curves are PSNR curves, and blue curves are VAR curves. The green bars indicate the detected ES point.
29

---

Gaussian
EOS SELERRR
Impulse
Shot
Speckle
Figure 18: (Early stopping) Our DMPlug with ES-WMV [35 for nonlinear deblurring with different types and levels of noise. (top: low-level noise; bottom: high-level noise). Red curves are PSNR curves, and blue curves are VAR curves. The green bars indicate the detected ES point.
30

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We clearly claim the scope and contributions of this paper in both abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this work in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
31

---

Justification:
All the formulas are numbered and cross cross-referenced.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide all the experiment details
Section
Appendix
C and Ap-
pendix D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
32

---

Answer: [Yes]
Justification:
We provide all the essential code for this paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https::/niis.cc.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes]
Justification: We provide all the experiment details in Section Appendix C and pendix D
Ap
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: First, we conduct extensive experiments in this paper and report the mean results. We do not anticipate significant fluctuations in the results. Second, we are afraid that adding the statistical significance of the experiments will mess this paper up because our tables are already very dense, but we are willing to provide them if needed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
33

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the experiment compute resources in Appendix D.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:////wrrpppppcctcuttttttttta
Answer: [Yes]
Justification: This paper is strictly with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the socictal impacts in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
34

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper only uses existing pretrained models for zero-shot tasks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We provide all the code and models we have used in Appendix D
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
35

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: The new assets in this paper well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "4xDro2IUnU", "reviewer": "Reviewer_yL1C", "summary": "The authors propose a new framework, called DM-Plug, to solve inverse problems with pre-trained diffusion models. Most of the prior works on this topic propose approximations for the conditional score function. The authors propose an alternative approach that is closely related to techniques for solving inverse problems with GANs. Namely, the authors consider the latent space that emerges in diffusion models by projecting images through the deterministic sampler and perform the optimization in this latent space.\n\n\nThere are two challenges with this approach: 1) one needs to backpropagate through the whole sampling chain to perform this optimization, so this approach can be really computationally intensive, and 2) for every update of the latent, one needs to run the whole sampler, so this approach can become really slow. The authors mention that they can circumvent these two issues by only running 3 steps of the sampler and that this suffices for their considered inverse problems.", "questions": "1) It seems to me that one of the main weaknesses of this work is the propagation over the sampling chain. Since all that is needed is the solution of the ODE, is it possible to use this method with Consistency Diffusion Models?\n2) There are many techniques to solve inverse problems with GANs. One of these techniques is Intermediate Layer Optimization, where the optimization happens in some intermediate latent space. This idea seems directly relevant here and could lead to improvements in performance and cost. Namely, the authors could optimize in the latent space that corresponds to some time other than $t=T$. Could the authors ablate this?\n3) I would like to see the number of required steps needed in the sampling chain as the difficulty of the diffusion problem increases. Namely, the authors could fix an inverse problem, let's say random inpainting, and ablate the number of required sampling steps for good performance as the corruption probability increases.", "limitations": "The authors have addressed the limitations of their work.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1) The topic of this research paper is interesting. There is a growing literature on approaches for solving inverse problems with diffusion models. This paper proposes a fresh idea in this space.\n2) The experimental results are strong, providing evidence that this is a promising approach.\n3) Since the approach here is very similar to solving inverse problems with GANs, there is a variety of ideas from this space that can be potentially leveraged to further improve the results of this approach.", "weaknesses": "1) The proposed method is a maximum-likelihood method and does not offer posterior samples. For certain inverse problems, being able to sample from the posterior is really important, for uncertainty quantification, diversity, etc. \n2) The presentation of the paper could be improved. For example, one idea would be to directly optimize in the space of clean images, since diffusion models also offer the score there. This idea wouldn't work because the estimation of the score for $t=0$ is usually poor. The authors do not explain this properly in their manuscript. \n3) The authors also mention that for conditional tasks, 3 steps are enough. However, this should truly depend on the level of corruption. At the limit of extreme corruption, it can't be true that $3$ sampling steps are enough since we know for a fact that diffusion models need more steps to achieve good generation results.\n4) From Figure 6.1., it looks like the MSE is minimized at around 1000 iterations of the proposed algorithm. Given that each iteration requires 3 sampling steps, this is equivalent to 3000 sampling steps. Most methods for solving inverse problems with diffusion models work with much fewer sampling steps.\n5) The authors could have used better baselines in their experimental evaluation, including ΠDGM, Red-diff and PSLD.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer yL1C for the detailed review and insightful comments!\n\n### RE Weakness 1:\n\nWe agree that allowing posterior sampling for uncertainty quantification and other purposes would be great. This is, unfortunately, not what the MAP framework can offer. We follow most current DM-based methods for inverse problems, which do not provide posterior sampling or uncertainty quantification. We will acknowledge this as a limitation of the current work, and leave it for future research. \n\n### RE Weakness 2:\n\nWe will definitely revise the manuscript thoroughly after the rebuttal to improve the presentation! \n\nOptimizing in the space of the clean image directly reduces things to the typical MAP formulations without using the pretrained DM priors. Our consideration is not about whether the score functions are estimated accurately or not; ours is more geometric: we view the whole reverse process as a learned function that characterizes the image manifold. Then, the question here is really about how many steps we need in the pretrained DM model, so that approximate DM priors are incorporated. About this choice, we have tried to address it in Sec 3.1 as well as our GR1 above. \n\n### RE Weakness 3 & Question 3:\n\nWe totally agree with the intuition, and our choice of 3 steps is a hyperparameter that works reasonably well in typical settings, but in no sense optimal. \n\nWe conduct the ablation study as you suggested. We test 40 cases from the CelebA dataset for random inpainting, and try different numbers of reverse steps with the increase of the mask ratio. The table below shows that our method with 7 steps performs slightly better than it with 3 and 11 steps, as you expected. We will add this ablation study in our future version and recommend users ablate the number of steps when facing challenging IPs.\n\n| Inpainting (CelebA) PSNR | Mask ratio 90% | Mask ratio 94% | Mask ratio 98% |\n|--------------------------|----------------|----------------|----------------|\n| 3 Steps                  | 25.96          | 25.68          | 24.87          |\n| 7 Steps                  | **26.48**          | **26.36**          | **25.83**          |\n| 11 Steps                 | 25.84          | 25.67          | 25.31          |\n\n### RE Weakness 4:\n\nPlease refer to GR2. \n\n### RE Weakness 5:\n\nPlease refer to GR3. \n\n\n### RE Question 1:\n\nThis is a very good idea! We noticed a recent paper [1] (after our submission) that successfully combines optimizing the input of diffusion models and consistency models. We will probably also incorporate this in our future version.\n\n> [1] Inverse Problems with Diffusion Models: A MAP Estimation Perspective\n\n### Question 2:\n\nThis is an interesting idea and definitely worth exploring! Using our 3-step method for super-resolution on 100 cases from CelebA, we experiment with different latent spaces to optimize when t=T, t=⅔ T, t=⅓ T, respectively. The results below show that optimizing the latent space when t<T can lead to unsatisfactory performance, which shows that it is essential to optimize the latent space when t=T to fully take advantage of the pre-trained prior information.\n\n| SR (CelebA) | PSNR  | SSIM  | LPIPS |\n|-------------|-------|-------|-------|\n| t = T       | **31.25** | **0.878** | **0.067** |\n| t = ⅔ T     | 26.35 | 0.587 | 0.237 |\n| t = ⅓ T     | 11.39 | 0.096 | 1.162 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Uo9JVe28K7", "reviewer": "Reviewer_wUnV", "summary": "This paper proposes an optimization-based method for optimize the initial noise for data consistency. This method is evaluated on a variety of linear inverse problems and non-linear inverse problems, and achieves SOTA results. The method also show robustness for unknown noise levels with a technique called ES-WMV.", "questions": "See weaknesses. \n1. memory and computing time\n2. clearer description of the pretrained model\n3. experimenting on more challenging forward models like CT operator or phase retrieval\n4. more LDM results", "limitations": "Yes, limitations are adequately addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method looks novel and straightforward to implement. \n2. The paper is well written and easy to understand\n3. diffusion inverse solvers struggle in nonlinear problems since that restricts the usage of SVD or null space decomposition. This paper improves the SOTA on nonlinear inverse problems by a significant margin\n4. The Early stopping/deep image prior phenomena reported in this paper is interesting.", "weaknesses": "1. the R(.) function looks quite long since it involves many reverse passes. Also the optimization may take up to 5000 iterations. Does that require additional memory and computing time? I cannot find the memory and time cost in this manuscript except in Fig.5\n2. the results on LDMs are limited, it would be more desirable to include LDM results on more nonlinear problems and linear problems besides super-resolution.\n3. the backbone model should be stated more clearly in the paper: what is the pretrained model? How is it trained? How many NFEs are used after optimizing the initial noise?\n4. Authors could provide more intuitions on why 3 ddim reverse steps are sufficient since it is challenging to generate high-quality images with 3 steps and the measurement function A(.) is applied on the $E[x_0|x_t]$ and may not reflect the ground truth signal. I think deblur or superresolution works since 3 DDIM reverse steps give blurry images which is quite consistent to measurement, but have the authors experiment on more challenging forward models like CT operator (Radon Transform) or phase retrieval?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer wUnV for the detailed review and insightful comments!\n\n### RE Weakness 1 & Question 1:\n\nPlease refer to GR2. \n\n### RE Weakness 2 & Question 4:\n\nPlease refer to GR4. \n\n### RE Weakness 3 & Question 2:\n\nThis is a very good suggestion! We use the standard DDIM/DDPM and LDM models. The pre-trained DDIM models of CelebA, FFHQ are from [this link](https://github.com/jychoi118/ilvr_adm?tab=readme-ov-file); the pre-trained DDIM model of LSUN-bedroom is from [this link](https://github.com/openai/guided-diffusion); the pre-trained LDM model of CelebA is from [this link](https://github.com/CompVis/latent-diffusion). We recommend the original papers for the training details. After optimizing the initial noise, we apply the same number of NFEs in R(.) as the training stage, i.e., 3 in most of the experiments of this paper. We will add an appendix to include these details in the revision. \n\n### RE Weakness 4 & Question 3:\n\nThanks for this great suggestion! \n\nPlease refer to GR1 for our discussion of the seemingly fundamental difference between inverse problems and conditional image generation. \n\nIn our paper, we have tried three challenging nonlinear IPs, including nonlinear deblur, BID and BID with turbulence, but we still want to try the tasks you suggested. For CT reconstruction, we follow the settings in [1] since they provide pre-trained DM for CT images. However, we are still waiting for the data access approval from TCIA on [this link](https://www.cancerimagingarchive.net/collection/ldct-and-projection-data/). In phase retrieval (PR), the goal is to recover a 2d or 3d image from the oversampled diffraction pattern (squared magnitude) of its Fourier transform. Although we can definitely test our method on PR, we have decided not to do it here, as the setting in the DPS paper [2] and follow-up work may be physically wrong, as Fourier PR never involves color images (Fourier involves only 2d or 3d single-channel images) [3,4,5,6]—in fact, their setting leads to a more difficult, yet unrealistic, problem; we leave this study as future work. \n\n>[1] Improving diffusion models for inverse problems using manifold constraints\n\n>[2] Diffusion posterior sampling for general noisy inverse problems\n\n>[3] The numerics of phase retrieval\n\n>[4] SiSPRNet: end-to-end learning for single-shot phase retrieval\n\n>[5] Three-dimensional imaging of strain in a single ZnO nanorod\n\n>[6] What is Wrong with End-to-End Learning for Phase Retrieval?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "giI66a3Tje", "reviewer": "Reviewer_y52g", "summary": "This paper presents a plug-and-play method DMPlug for solving inverse problems with diffusion models. DMPlug utilizes a pre-trained diffusion model as a deterministic function that generates images from latent seeds and solves MAP problems by optimizing the seeds. Experiments show that their method beats current SOTA methods across different inverse problems and is robust against unknown perturbations.", "questions": "- DMPlug seems more space/time-consuming than existing methods since it requires gradient backpropagation through the diffusion model multiple times per step, while ReSample does not backpropagate through the diffusion model. Depending on the actual number of iterations used for DMPlug, the time cost for DMPlug might be prohibitively high.  I am curious to see some discussion and comparison of DMPlug with existing methods in terms of time cost.", "limitations": "The authors address the limitations and provide possible future directions in their paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed algorithm DMPlug is concise and novel. \n- The authors observe an intriguing early-learning-then-overfitting property of their algorithm. They integrate an early-stopping strategy in DMPlug to enhance the robustness of DMPlug against unknown data corruption, which is illustrative and useful.\n- The experimental results shown in the paper look promising. The empirical robustness analysis is also interesting and informative.", "weaknesses": "- The paper is missing many baseline methods on plug-and-play inverse problem solving with diffusion models, such as DDNM[1], DiffPIR[2], RED-Diff[3], FPS[4]. These methods are claimed to perform much better than DPS. The experimental result is less convincing without comparison with these recent results. \n- Moreover, the major recent baseline considered in the paper, ReSample, uses pre-trained latent diffusion models, different from the pre-trained model used in this paper, so a direct comparison does not make sense. This weakness is partly mitigated by the ablation study in Table 5.\n\n[1] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. ICLR, 2023.\n\n[2] Yuanzhi Zhu, Kai Zhang, Jingyun Liang, Jiezhang Cao, Bihan Wen, Radu Timofte, and Luc Van Gool. Denoising diffusion models for plug-and-play image restoration. CVPR, 2023.\n\n[3] Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat. A variational perspective on solving inverse problems with diffusion models. ICLR, 2024.\n\n[4] Zehao Dou, Yang Song. Diffusion Posterior Sampling for Linear Inverse Problem Solving: A Filtering Perspective. ICLR, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer y52g for the detailed review and insightful comments!\n\n### RE Weakness 1:\n\nPlease refer to GR3.\n\n### RE Weakness 2:\n\nWe totally understand this concern that using different DMs may lead to some unfairness concerns, but it is not clear which model the unfairness pertains to. Actually, even the Resample paper [1] compares their LDM-based methods with DM-based methods in most of their experiments. We are happy that our ablation study in Table 5 partially mitigated this concern, so we further add more LDM experiments this time. Please refer to GR4 for using our method with LDMs on inpainting and nonlinear deblurring tasks.\n\n> [1] Solving inverse problems with latent diffusion models via hard data consistency\n\n### RE Question 1:\n\nThis is indeed a valid question! Please refer to GR2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KmIsSSYz6e", "reviewer": "Reviewer_uJfQ", "summary": "This paper is devoted to the use of diffusion models to solve inverse problems, arguing for treating the inverse process in a diffusion model as a function and proposing a new plug-in approach called DMPlug. DMPlug addresses the issues of feasibility and measurement feasibility of manifolds in a principled manner, and shows great potential for robustness to unknown types and levels of noise.", "questions": "1. Although the authors point out that the use of 3-step DDIM can significantly mitigate computational consumption, the memory required to save the computational graphs of 3-stepDDIM is still huge for some larger diffusions, as far as I know. Can the authors report a comparison with the time consumption and GPU memory used by other methods, such as DPS.\n\n2. Can the author provide some intuition or explanation as to why the 3 steps work.\n\n3. Can the author show some qualitative results of DMPlug in more difficult cases, such as inpainting of box masks or the imagenet dataset?      \n\n3. Can the proposed DMPlug method with 3 steps be used for other controllable generation tasks rather than just inverse problems such as, Classifier/Clip guidance in [2].\n\n[2] End-to-End Diffusion Latent Optimization Improves Classifier Guidance\n\n\nIf the author can address my concerns, I am open to changing my score during the discussion phase.", "limitations": "The authors briefly discuss for the limitations of their approach.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The article is very comprehensive in its review of existing relevant technologies, which is more friendly to readers unfamiliar with the field.\n\n2. The article describes the proposed method in a clearer and more detailed way, which is easy to understand.\n\n3. The qualitative results on celeba and FFHQ datasets presented in the figures of the experimental sections look very effectual and better than the baseline method.\n\n4. This paper provides a comprehensive exploration of the robustness of the proposed method under various noises.", "weaknesses": "1. *Contributing issues:* The proposed method is very similar to the existing work [1,2] in its approach to condition generation by optimizing the noisy latent variables of a diffusion model. The main difference in this paper (and why it works is also very surprising to me) is the empirical evidence that good results can be achieved with only 3 backward steps. It is well known that 3 backward steps usually fail to generate meaningful images for diffusion models. Although the authors do not provide a theoretical basis, they should also provide an intuitive insight into why it works.\n\n2.  *Experimental issues:* The experimental section states that their metrics are calculated primarily on 100 images, which is too few, and much of the existing work is 1000-image tests. In addition, the paper selects Celeba and FFHQ which have somewhat similar patterns despite being two datasets.\n\n[1] D-Flow: Differentiating through Flows for Controlled Generation\n[2] End-to-End Diffusion Latent Optimization Improves Classifier Guidance", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer uJfQ for the detailed review and insightful comments!\n\n### RE Weakness 1:\n\nWe appreciate this insightful comment! Indeed, the two papers have similar ideas to ours from a high level, and we will definitely add and discuss them in our revision. However, we still want to highlight several crucial differences between their methods and our method.\n\n* [1] computes the gradient using the adjoint method and the gradient checkpointing trick to mitigate the memory issue, but the two techniques will further slow down the optimization process. In contrast, our method, with only a few steps, can achieve SOTA results.\n* [1] only considers linear IPs, but our paper contains three nonlinear IPs and achieves much better results than the current SOTA methods (typically 3-6dB in terms of PSNR).\n* Our paper observes the early-learning-then-overfitting and spectral bias phenomena when solving IPs. With the two phenomena, our method can achieve robustness to unknown types and levels of noise.\n* [2] also uses the gradient checkpointing trick to mitigate the memory issue, slowing down the whole process.\n* Although IPs can be considered as conditional generation problems, the condition is much stronger than the classifier/CLIP guidance. Please refer to GR1 for more information about IPs and conditional generation with the classifier/CLIP guidance.\n\n>[1] D-Flow: Differentiating through Flows for Controlled Generation\n\n>[2] End-to-End Diffusion Latent Optimization Improves Classifier Guidance\n\n>[3] Scalable gradients for stochastic differential equations\n\n>[4] https://pytorch.org/docs/stable/checkpoint.html\n\n### RE Weakness 2:\n\nWe mainly follow the experiment setting of the recent work—Resample [5] (ICLR’24 Spotlight), which also uses 100 images for each task and dataset, but we totally understand this concern. Hence, we try to add more experiment results in the rebuttal period as you and other reviewers suggested (please check General Response). Besides the two datasets, we also try a more complex dataset—LSUN-bedroom, following [5]. Also, we add more qualitative results for ImageNet as you suggested in GR5. \n\n>[5] Solving inverse problems with latent diffusion models via hard data consistency\n\n### RE Question 1:\n\nThis is indeed a valid question! Please refer to GR2. \n\n\n### RE Question 2:\n\nWe appreciate this insightful suggestion! Please refer to GR1. \n\n\n### RE Question 3:\n\nSure, please refer to GR5. \n\n### RE Question 4:\n\nWe appreciate this very interesting comment! Please refer to GR1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their thoughtful and constructive comments about our manuscript!

### GR1: Difference between solving inverse problems and condition generation with the classifier guidance (CGwCG) 

As suggested by Reviewer uJfQ and hinted by other reviewers, we ablate the number of reverse steps for [1], i.e., CGwCG, in the attached pdf file. It is clear that 3-step CGwCG cannot generate meaningful objects.

This suggests that CGwCG can be substantially more difficult than typical inverse problems. We suspect the reason is that in typical inverse problems, the measurement y provides much stronger “guidance”/information than the label/text guidance in conditional generation: the former, together with additional priors (say from pre-trained DMs), typically enables reliable estimation of the ground truth, whereas the latter can have numerous, or even infinite, solutions. 

> [1] End-to-End Diffusion Latent Optimization Improves Classifier Guidance

### GR2: Time and memory consumption of our method

In response to the valuable suggestions from several reviewers, we have provided a comparison of our method with other competitors in terms of time consumption and memory usage below (we report the time to get the peak for ours). We acknowledge that currently, our method is often slower than other competing methods. But we also want to highlight the superior performance of our method for solving IPs, particularly nonlinear IPs (typically 3-6dB boost in terms of PSNR). The user has the flexibility to choose the right balance between recovery quality and computation cost, based on their own priorities and constraints.  We will add this point into limitations and leave accelerating our method for future work.

|             | ADMM-PnP | DMPS | DDRM | MCG  | DPS  | ReSample | FPS   | DiffPIR | DDNM | Ours |
|-------------|----------|------|------|------|------|----------|-------|---------|------|------|
| Time (s)    | 6        | 42   | 30   | 43   | 43   | 14       | 62    | 4       | 367  | 635  |
| Memory (GB) | 0.42     | 5.10 | 4.99 | 2.80 | 2.79 | 5.11     | 20.63 | 1.44    | 4.87 | 6.59 |


### GR3: Comparison with more SOTA methods

Although we have compared our method with the very recent work—Resample [2] (ICLR’24 Spotlight) for most of the experiments, we still would like to compare with more SOTA methods as reviewers suggested. We have tried our best to implement three methods in the short rebuttal period. We run super-resolution (SR) and inpainting for FPS, DiffPIR and DDNM on 100 cases of CelebA since they only consider linear IPs in their papers. Our method and DiffPIR lead other methods for SR, and our method can lead all competitors for inpainting.

> [2] Solving inverse problems with latent diffusion models via hard data consistency

| SR (CelebA) | PSNR  | SSIM  | LPIPS |
|-------------|-------|-------|-------|
| FPS         | 29.12 | 0.858 | 0.149 |
| DiffPIR     | **31.55** | 0.857 | 0.203 |
| DDNM        | 29.21 | 0.836 | 0.193 |
| Ours        | 31.25 | **0.878** | **0.067** |

| Inpainting (CelebA) | PSNR  | SSIM  | LPIPS |
|---------------------|-------|-------|-------|
| FPS                 | 32.06 | 0.924 | 0.064 |
| DiffPIR             | 31.22 | 0.866 | 0.219 |
| DDNM                | 27.89 | 0.799 | 0.224 |
| Ours                | **34.03** | **0.936** | **0.039** |

### GR4: More comparison with methods based on latent-diffusion models (LDMs)

We thank Reviewer wUnV for the suggestion to include more LDM results. We have provided additional results of our method using LDM models, as shown below. These results exhibit a similar trend to the SR results in Table 5, indicating that our method performs slightly better with DMs compared to LDMs. Regardless of whether DMs or LDMs are used, our method consistently achieves comparable or superior results to SOTA methods for these tasks.

| Inpainting (CelebA) | PSNR  | SSIM  | LPIPS |
|---------------------|-------|-------|-------|
| Best competitor     | 32.24 | 0.924 | **0.039** |
| Ours (DM)           | **34.03** | **0.936** | **0.039** |
| Ours (LDM)          | 33.10 | 0.923 | 0.048 |

| Nonlinear deblur (CelebA) | PSNR  | SSIM  | LPIPS |
|---------------------------|-------|-------|-------|
| Best competitor           | 28.52 | 0.839 | 0.104 |
| Ours (DM)                 | **31.61** | **0.882** | **0.073** |
| Ours (LDM)                | 30.64 | 0.861 | 0.108 |

### GR5: More complex settings for inpainting and super-resolution

We follow the valuable suggestion from Reviewer uJfQ to add more qualitative results for more complex settings. Please check the visual results of box inpainting on CelebA and super-resolution on ImageNet in the attached pdf file.

Meta-review: This paper introduces an optimization-based approach to refine the initial noise for data consistency. The method is tested on a range of both linear and non-linear inverse problems, achieving state-of-the-art (SOTA) results. 

Most reviewers find the problem is important and the work is timely. The idea is interesting and novelty, and the demonstrated results are strong and evaluated on both linear and nonlinear inverse problems. For the camera ready, we suggest the authors to incorporate the reviewers' feedback into the final version.
Decision: Accept (poster)