Title: DMPlug: A Plug-in Method for Solving Inverse Problems with Diffusion Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Inverse problems

diffusion models
Abstract: Pretrained diffusion models (DMs) have recently been popularly used in solving inverse problems (IPs). The existing methods mostly interleave iterative steps in the reverse diffusion process and iterative steps to bring the iterates closer to satisfying the measurement constraint. However, such interleaving methods struggle to produce final results that look like natural objects of interest (i.e., manifold feasibility) and fit the measurement (i.e., measurement feasibility), especially for nonlinear IPs. Moreover, their capabilities to deal with noisy IPs with unknown types and levels of measurement noise are unknown. In this paper, we advocate viewing the reverse process in DMs as a function and propose a novel plug-in method for solving IPs using pretrained DMs, dubbed DMPlug. DMPlug addresses the issues of manifold feasibility and measurement feasibility in a principled manner, and also shows great potential for being robust to unknown types and levels of noise. Through extensive experiments across various IP tasks, including two linear and three nonlinear IPs, we demonstrate that DMPlug consistently outperforms state-of-the-art methods, often by large margins especially for nonlinear IPs.

DMPlug: A Plug-in Method for Solving Inverse Problems with Diffusion Models

Abstract

Pretrained diffusion models (DMs) have recently been popularly used in solving inverse problems (IPs). The existing methods mostly interleave iterative steps in the reverse diffusion process and iterative steps to bring the iterates closer to satisfying the measurement constraint. However, such interleaving methods struggle to produce final results that look like natural objects of interest (i.e., manifold feasibility) and fit the measurement (i.e., measurement feasibility), especially for nonlinear IPs. Moreover, their capabilities to deal with noisy IPs with unknown types and levels of measurement noise are unknown. In this paper, we advocate viewing the reverse process in DMs as a function and propose a novel plug-in method for solving IPs using pretrained DMs, dubbed DMPlug. DMPlug addresses the issues of manifold feasibility and measurement feasibility in a principled manner, and also shows great potential for being robust to unknown types and levels of noise. Through extensive experiments across various IP tasks, including two linear and three nonlinear IPs, we demonstrate that DMPlug consistently outperforms state-of-the-art methods, often by large margins especially for nonlinear IPs. The code is available at https://github.com/sun-umn/DMPlug .

1 Introduction

Inverse problems (IPs) are prevalent in numerous fields, such as computer vision, medical imaging, remote sensing, and autonomous driving [1-4]. The goal of IPs is to recover an unknown object x from noisy measurements y = A ( x ) + n , where A is a (possibly nonlinear) forward model and n denotes the measurement noise. IPs are often ill-posed: typically, even if y is noiseless, x cannot be uniquely determined from y and A . Hence, incorporating prior knowledge on x is necessary to obtain a reliable estimate of the underlying x .

Traditionally, IPs are solved in the regularized data-fitting framework, often motivated as performing the Maximum a Posterior (MAP) inference:

Here, minimizing the data-fitting loss promotes y ≈ A ( x ) , and the regularizer encodes prior knowledge on x . Recently, the advent of deep learning (DL) has brought about a few new algorithmic ideas to solve IPs. For example, given a training set of measurement-object pairs, i.e., { ( y i , x i ) } i =1 ,...,N , one can hope to train a DL model that directly predicts x for a given y [5-16]. However, such hopes can be shattered by practical challenges in collecting massive and realistic paired training sets, especially for complex IPs [17, 18]. Even if such challenges can be tackled, one may need to collect a new training set and train a new DL model for every new IP [15], overlooking potential shared priors on x across IPs. An attractive alternative family of ideas combine pretrained priors on x and

Measurement

Reference

DMPlug(ours)

DPS

Resample

NonlinearIPs

BID

BIDwith turbulence

Super-resolution

Measurement

Reference

Inpainting

DMPlug (ours)

Nonlinear IP

Stripformer

regularized data-fitting in Eq. (1). For example, they first model the distribution of x using deep generative models, such as generative adversarial networks (GANs) and diffusion models (DMs), based on training sets of the form { x i } i =1 ,...,N , and then encode these pretrained generative priors when solving Eq. (1). In this way, pretrained priors on x can be reused in an off-the-shelf manner in different IPs about the same family of structured objects.

In this paper, we focus on solving IPs with pretrained DMs . DMs have recently emerged as a dominant family of deep generative models due to their relative stability during training (e.g., vs. GANs) and their strong capabilities to generate photorealistic images (and, depending on applications, other structured objects) once trained [22-27]. These strengths of DMs have motivated ideas to use pretrained DMs to solve IPs, such as denoising, superresolution, inpainting, deblurring, and phase retrieval [28, 29, 20, 30-33, 21, 26, 27, 34]. Most of these ideas interleave iterative reverse diffusion steps and iterative steps (sometimes projection steps, especially for linear IPs) to move closer to the feasibility set { x | y = A ( x ) } (see Fig. 3 and Fig. 4 (1)). However, they typically cannot guarantee the final convergence of the iteration sequence to either the feasible set (i.e., measurement feasibility ), or the object manifold M captured by pretrained DMs (i.e., manifold feasibility ), as they have modified both iterative processes (see detailed arguments in Section 2).

100

Ours

DPS

Resample

10-1

102

103

10-4

20

40

60

08

100

Fig. 1 shows visible artifacts produced by these ideas on several IPs, highlighting (Issue 1) insufficient manifold feasibility . To quickly confirm (Issue 2) insufficient measurement feasibility ,

FeasibleSet

FeasibleSet

{x|y=A（x)}

{x|y=A（x)}

X

0x

x'

M

ImageTrajectory

XT-2

XT-1

SeedTrajectory

ZT

Vxlly-A（x)11²

XT

ZT

we experiment on noiseless instances of the nonlinear deblurring problem (a nonlinear IP) following [20, 19], and find that state-of-the-art (SOTA) methods fail to find an x that satisfies y = A ( x ) , as shown in Fig. 2. Furthermore, most SOTA DM-based methods for IPs assume known noise types (e.g., often Gaussian) and known, often very low, noise levels, casting doubts on their performance when faced with unknown noise types and levels, i.e., (Issue 3) robustness to unknown noise types and levels , as we confirm in Table 4.

Our contributions In this paper, we propose a novel plug-in method, dubbed DMPlug, to solve IPs with pretrained DMs to mitigate all of the above three issues . DMPlug departs from the popular and dominant interleaving line of ideas by viewing the whole reverse diffusion process as a function R ( · ) , consisting of multiblock stacked DL models, mapping from the seed space to the object manifold M . This novel perspective allows us to naturally parameterize the object to be recovered as x = R ( z ) and then plug this reparametrization into Eq. (1), leading to a unified optimization formulation with respect to the seed z . Conceptually, since R ( z ) probably produces a feasible point on the object manifold M and global optimization of the unified formulation encourages y ≈ A ( R ( z )) , the optimized z could lead to an x = R ( z ) that enjoys both manifold and measurement feasibility , i.e., (tackling Issues 1 & 2) . Fig. 3 and Fig. 4 schematically illustrate the dramatic difference between interleaving methods and our plug-in method (DMPlug), and Figs. 1 and 2 confirms DMPlug's strong capability in finding feasible solutions. For noisy IPs with unknown noise types and levels, we observe that our DmPlug enjoys a benign 'early-learning-then-overfitting' (ELTO) property: the quality of the estimated object climbs first to a peak and then degrades once the noise is picked up, as shown in Fig. 6 (2). This benign property, in combination with appropriate early-stopping (ES) methods that stop the estimate sequence near the peak, allows our method to solve IPs without the exact noise information, i.e., (tackling Issue 3) , as shown in Table 4). Fortunately, we find that an ES method, ES-WMV [35] (co-developed by a subset of the current authors), works well for this purpose (see Table 4).

Our contributions can be summarized as follows. (1) In Section 3.1, we pioneer a novel plugin method, DMPlug , which is significantly different from the prevailing interleaving methods, to solve IPs with pretrained DMs. Then we make the proposed method practical in terms of computational and memory expenses by leveraging one key observation; (2) In Section 4 and Appendix E, we perform extensive experiments on various linear and nonlinear IPs and show that our method outperforms SOTA methods , both qualitatively and quantitatively-often by large margins, especially on nonlinear IPs. For example, measured in PSNR, our method can lead SOTA methods by about 2 dB and 3 ∼ 6 dB for linear and nonlinear IPs, respectively. Moreover, our method demonstrates flexibility in employing different priors and optimizers, as explored in Section 4.3; (3) In Section 3.3, we observe an early-learning-then-overfitting (ELTO) property , i.e., that our DMPlug tends to recover the desirable object first and then overfit to the potential noise. By taking advantage of this benign property and integrating the ES method ES-WMV [35], our method is the first to achieve robustness to unknown noise types and levels , leading SOTA methods by about 1 dB and 3 . 5 dB in terms of PSNR for linear and nonlinear IPs, respectively.

2 Background and related work

Diffusion models (DMs) The denoising diffusion probabilistic model ( DDPM ) [23] is a seminal DMfor unconditional image generation. It gradually transforms x 0 ∼ p data into total noise x T ∼ N ( 0 , I ) (i.e., forward diffusion process) and then learns to gradually recover x 0 from x T through

incremental denoising (i.e., reverse diffusion process). The forward process can be described by a stochastic differential equation (SDE), d x = -β t / 2 · x dt + √ β t d w , where β t is the noise schedule and w is the standard Wiener process. The corresponding reverse process is described by

Here, w is the time-reversed standard Wiener process, p t ( x ) is the probability density at time t , and ∇ x log p t ( x ) is the (Stein) score function, which is approximated by a DL model ε ( t ) θ ( x ) via score matching methods during DM training [36, 37]. For discrete settings, given time steps t ∈ { 1 , . . . , T } , a variance schedule β 1 , . . . , β T , α t . = 1 -β t with α T → 0 , and ¯ α t . = ∏ t s =1 α s , the DDPM has the forward process x t = √ 1 -β t x t -1 + √ β t z and the reverse process

where z ∼ N ( 0 , I ) . As the DDPM has a slow reverse/sampling process, [24] proposes the denoising diffusion implicit model ( DDIM ) to mitigate this issue. With the same notation as that of the DDPM, the DDIM makes a crucial change to the DDPM: relaxing the forward process to be non-Markovian by making x t depend on both x 0 and x t -1 . This simple change allows skipping iterative steps in the reverse process, without retraining the DDPM. This leads to much smaller numbers of reverse steps, and hence substantial speedup in sampling. The reverse process is now defined as

where ̂ x 0 ( x t ) . = [ x t - √ 1 -¯ α t ε ( t ) θ ( x t )] / √ ¯ α t is the predicted x 0 with x t .

Pretrained DMs for solving IPs Ideas for solving IPs with DMs can be classified into two categories: supervised and zero-shot [31, 38]. The former trains DM-based IP solvers based on paired training sets of the form { ( y i , x i ) } and is hence not our focus here (see our arguments in Section 1). The latter makes use of pretrained DMs as data-driven priors: (I) Most of the work in this category considers modeling p t ( x | y ) directly and replaces the (unconditional) score function ∇ x log p t ( x ) in Eq. (2) by the conditional score function ∇ x log p t ( x | y ) = ∇ x log p t ( x ) + ∇ x log p t ( y | x ) , leading to the conditional reverse SDE

Here, while ∇ x log p t ( x ) can be naturally approximated by the pretrained score function ε ( t ) θ ( x ) , ∇ x log p t ( y | x ) is intractable as y does not directly depend on x ( t ) 2 . Ideas to circumvent this difficulty include approximating p t ( y | x ( t )) by p t ( y | ̂ x (0)[ x ( t )]) , where ̂ x (0)[ x ( t )] is implemented as ̂ x 0 ( x t ) of Eq. (4) in discretization [19, 21], and interleaving unconditional reverse steps of Eq. (3) or Eq. (4) and (approximate) projections onto the feasible set { x | y = A ( x ) } to bypass the likelihood p t ( y | x ) [39, 30, 33, 32, 40, 41, 20]; (II) An interesting alternative is to recall that the MAP framework (see also Eq. (1)) involves max x log p ( y | x ) + log p ( x ) , and first-order methods, especially proximal-gradient style methods, to optimize the MAP formulation typically only need to access p ( x ) through ∇ x log p ( x ) , i.e., the score function-the central object in DMs! So, one can derive IP solvers by wrapping pretrained DMs around first-order methods for (approximately) optimizing the MAP formulation, see, e.g., [27, 28].

Despite the disparate conceptual ways of utilizing pretrained DM priors, most of the methods under (I) and (II) proposed in the literature so far follow a single algorithmic template, i.e., Algorithm 1 . There, Lines 3-5 are simply a reverse iterative step in the DDIM (Eq. (4); obviously, one could replace this by a reverse iterative step in other appropriate pretrained DMs. Line 6 helps move the iterate closer or onto the feasible set { x | y = A ( x ) } . In other words, these methods interleave iterative steps to move toward the data manifold M defined by the pretrained DM and iterative steps to move toward the feasible set { x | y = A ( x ) } , i.e., as illustrated in Fig. 3 (left). However, it is unclear, a priori, whether such interleaving iterative sequence will converge to either, leading to concerns about (Issue 1) insufficient manifold feasibility and (Issue 2) insufficient measurement feasibility . In fact, our Figs. 1 and 2 confirm these issues empirically, echoing observations made in several prior papers [20, 42, 30]. In principle, Issue 2 can be mitigated by ensuring that Line 6 finds a feasible

Algorithm 1 Template for interleaving methods

Input:

# Diffusion steps

T

, measurement

y

1:

x

T

∼ N

(

0

,

I

)

2:

for

i

=

T

-

1

to

0

do

3:

ˆ

s

ε

(

i

)

θ

(

x

i

)

4:

ˆ

x

0

1

√

¯

α

i

(

x

i

- √

1

-

¯

α

i

ˆ

s

)

5:

x

′

i

-

1

DDIM reverse with

ˆ

x

0

and

ˆ

s

6:

x

i

-

1

(Approximately)

Projec-

tion [39, 30, 33, 32, 40, 41, 34] or gradient

update [20, 28, 19, 21, 29, 27, 26] with

ˆ

x

0

and

x

′

i

-

1

to get closer to

{

x

|

y

=

A

(

x

)

}

7:

end for

Output:

Recovered object

x

0

Algorithm 2 Proposed plug-in method, DMPlug

Input:

# Epochs

E

, # diffusion steps

T

,

y

1:

Initialize seed

z

0

T

∼ N

(

0

,

I

)

2:

for

e

= 0

to

E

-

1

do

3:

for

i

=

T

-

1

to

0

do

4:

ˆ

s

ε

(

i

)

θ

(

z

e

i

)

5:

ˆ

z

e

0

1

√

¯

α

i

(

z

e

i

- √

1

-

¯

α

i

ˆ

s

)

6:

z

e

i

-

1

DDIM reverse with

ˆ

z

e

0

,

ˆ

s

7:

end for

8:

Update

z

e

+1

T

from

z

e

T

via a gradient update

for Eq. (7)

9:

end for

Output:

Recovered object

R

(

z

E

-

1

T

)

R

x i -1 for { x | y = A ( x ) } in each iteration [39, 30, 33, 32, 40, 41, 20]. However, this is possible only for easy IPs (e.g., linear IPs where we can perform a closed-form projection) and is difficult for typical nonlinear IPs as hard nonconvex problems are entailed. Moreover, although most of these methods have considered noisy IPs alongside noiseless ones [39, 30, 33, 32, 20, 28, 19, 21, 29], their common assumption about known noise types (often Gaussian) and levels (often low) is unrealistic: there are many types of measurement noise in practice-often hybrid from multiple sources [43], and the noise levels are usually unknown and hard to estimate [44], raising concerns about (Issue 3) the robustness of these methods to unknown noise types and levels ; see Section 4.2.

3 Method

In this section, we propose a simple plug-in method, DMPlug, to solve IPs with pretrained DMs to address Issues 1 & 2 in Section 3.1, discuss its connection to and difference from several familiar ideas in Section 3.2, and finally explain how to integrate an early-stopping strategy, ES-WMV [35], into our plug-in method to address Issue 3 in Section 3.3, leading to an algorithm, DMPlug+ES-WMV, summarized in Algorithm 3, that solves IPs with pretrained DMs under the regularized data-fitting framework Eq. (1), even in the presence of unknown noise.

3.1 Our plug-in method: DMPlug

Reverse process as a function and our plug-in method Interleaving methods discussed above can have trouble satisfying both manifold feasibility and measurement feasibility because they interleave and hence modify both processes. Although projection-style modifications can improve measurement feasibility [39, 30, 33, 32, 40, 41, 20], their application to nonlinear IPs seems tricky, and a single 'projection' step could be as difficult and expensive as solving the original problem due to the typical nonconvexity induced by the nonlinearity in A [28, 20]. In contrast, we propose viewing the whole reverse process as a function R ( · ) that maps from the seed space to the object space (or the object manifold M ). Mathematically, if we write a single inverse step as a function g that depends on ε ( i ) θ , i.e., g ε ( i ) θ for the i -th reverse step that maps x i +1 to x i , then

Note that for the DDPM, and those DMs based on SDEs in general, R is a stochastic function due to noise injection in each step. To reduce technicality, we focus on DMs based on ordinary differential equations (ODEs), in particular, the DDIM, due to their increasing popularity [45, 46], resulting in deterministic R 's. This conceptual leap allows us to reparametrize our object of interest as x = R ( z ) and plug this reparametrization into the traditional regularized data-fitting framework in Eq. (1), yielding the following unified optimization formulation:

We stress that here the optimization is with respect to the seed variable z , given the pretrained reverse process R . Since we never modify the reverse diffusion process R , we expect R ( z ) to

produce an object on the object manifold M , i.e., enforcing manifold feasibility to address Issue 1 . Moreover, optimizing the unified formulation Eq. (7) is expected to promote y ≈ A ( R ( z ∗ )) , inherent in the regularized data-fitting framework, i.e., promoting data feasibility to address Issue 2 .

When there are multiple objects of interest in the IP under consideration, i.e., y = A ( x 1 , · · · , x k )+ n with k objects (e.g., in blind image deblurring y = A ( k , x ) + n , both the blur kernel k and sharp image x are objects of interest, where A ( k , x ) = k ∗ x ; see also Appendix C.4), different objects may have different priors that should be encoded and treated differently. Our unified optimization formulation Eq. (7) facilitates the natural integration of multiple priors. Another bonus feature of our unified formulation lies in the flexibility in choosing numerical optimization solvers. We briefly explore both aspects in Section 4.3.

Fast samplers for memory and computational efficiency

When implementing DMPlug with typical gradient-based solvers, e.g., ADAM [47], the gradient calculation in each iterative step requires a forward and a backward pass through the entire R , i.e., T blocks of the basic DL model in the given DM. For high-quality image generation [23-25], T is typically tens or hundreds, resulting in prohibitive memory and computational burdens. To resolve this, we use the DDIM, which allows skipping reverse sampling steps thanks to its non-Markovian property, as the sampler R . We observe, to our surprise and also in our favor , that a very small number of reverse steps, such as 3 , is sufficient for our method to beat SOTA methods on all IPs we evaluate (see Fig. 5 and Section 4), and further increasing the number does not substantially improve the performance (actually even slightly degrades it perhaps due to the numerical difficulty caused by vanishing gradients as more

steps are included). So, we default the number of reverse steps to 3 unless otherwise stated . This number is not sufficient for generating high-quality photorealistic images with existing DMs, but is good enough for our method. The discrepancy suggests a fundamental difference between image generation and image 'regression' involved in solving IPs-we leave this for future work.

3.2 Seemingly similar ideas

(A) GAN inversion for IPs Our plug-in method is reminiscent of GAN inversion to solve IPs [48, 49]: for a pretrained GAN generator G θ , GAN inversion performs a similar reparametrization x = G θ ( z ) and plugs it into Eq. (1). The remaining task is also to minimize the resulting formulation with respect to the trainable seed z , and produce an x = G θ ( z ∗ ) through a solution z ∗ . However, there are numerous signs that GAN inversion does not work well for IPs. For example, [50] also finetunes the generator G θ alongside the trainable seed to boost performance, and [33, 32] report superior performance of DM-based methods compared to GAN-inversion-based ones for solving IPs; (B) Diffusion inversion (DI) Given an object x , DI aims to find a seed z so that R ( z ) reproduces x , an important algorithmic component for DM-based image and text editing [51-53, 24, 54, 55]. A popular choice for DI is to modify DDIM [53, 24, 54, 55]. Although we cannot use DI to solve general IPs, DI can be considered as an IP and solved through min z ℓ ( x , R ( z )) using our plug-in method. (C) Algorithm unrolling (AU) The multiblock structure in R also resembles those DL models used in AU, a popular family of supervised methods for solving IPs [56, 16]. In AU, the trainable DL model also consists of multiple blocks of basic DL models, induced by unfolded iterative steps to solve Eq. (1). While in AU the weights in these DL blocks are trainable, the weights in our DL blocks are fixed. More importantly, as a supervised approach, AU requires paired training sets of the form { ( y i , x i ) } i =1 ,...,N , in contrast to the zero-shot nature of our method here.

3.3 Achieving robustness to unknown noise

The early-learning-then-overfitting (ELTO) phenomenon When y contains noise, solving Eq. (7) can promote measurement feasibility, i.e., y ≈ A ( R ( z ∗ )) , but A ( R ( z ∗ )) may also learn the noise, i.e., overfitting to noise . Interestingly, our method seems to favor desirable content and resist noise : (A) our method converges much faster when used to regress clean natural images than random

0.35

Ipmage

1e-5

2.00

'imageshuffled

PSNR:

0.30

Randomnoise

Variantce

1.75

30

ES-WMV

0.25

1.50

MSE

0.20

PSNR

25

1.25

1.00

0.15

20

0.75

0.10

0.50

0.05

15

0.25

0.00

0.00

100

101

102

103

0

500

1000

1500

2000

2500

IterationinLogScale

Iteration

(1)

(2)

noise (Fig. 6 (1), suggesting that our method shows high resistance to noise and low resistance to structured content, and (B) when performing regression against a noisy image x = x 0 + n , our method, although powerful enough to overfit the noisy image x ultimately, picks up the desired image content first and then learns the noise, leading to a hallmark 'early-learning-then-overfitting' ( ELTO ) phenomenon so that the recovery quality climbs to a peak before the potential degradation due to noise (Fig. 6 (2)). We stress that similar ELTO phenomena have been widely reported in the literature on using deep image priors (DIPs) to solve IPs [57-61], although this is the first time this phenomenon has been reported for DM-based methods . Inspired by related studies in DIP, we also perform a spectral analysis of intermediate recovery and reveal that our method has a spectral bias toward low-frequency components during learning , similar to DIP methods; see Appendix B.

Achieving robustness via early stopping (ES) One may wonder how widely this ELTO phenomenon occurs when using our DMPlug to solve IPs. Besides extensive additional visual confirmation (see Appendix F), in Table 4 we show that our method, without using the noise information, leads the SOTA methods in terms of peak performance during iteration-particularly, by large margins on non-linear deblurring. The quantitative results suggest that the ELTO phenomenon is likely widespread, especially across noise types and levels. Hence, if we can perform proper early stopping (ES) to locate the peak performance, we tackle Issue 3 . Deriving an effective ES strategy here is nontrivial, as in practice we do not have groundtruth images to compute any reference-based performance metrics such as PSNR. Fortunately, in the DIP literature, [35] discovers that for DIP-based methods for IPs, the valleys of the running-variance curves of the intermediate reconstructions are well aligned with the performance peaks. Based on this crucial observation, they propose an ES strategy, ES-WMV, that can accurately detect performance peaks with small performance loss on various IP tasks. Inspired by their success, we integrate the ES-WMV strategy into our plug-in method and find that ES-WMV is highly synergetic with our method and performs reliable ES with negligible performance loss (see Table 4). Details of the entire algorithm can be found in Algorithm 3.

4 Experiments

In this section, we evaluate our plug-in method, DMPlug, and compare it with other SOTA methods on two linear IPs, including super-resolution and inpainting , and three nonlinear IPs, including nonlinear deblurring , blind image deblurring (BID) , and BID with turbulence . Following [20], we construct the evaluation sets by sampling 100 images from CelebA [62], FFHQ [63] and LSUNbedroom [64], respectively, and resizing all images to 256 × 256 ; we measure recovery quality using three standard metrics for image restoration, including peak signal-to-noise-ratio (PSNR), structural similarity index (SSIM) and Learned Perceptual Image Patch Similarity (LPIPS) [65] with the default backbone. We describe in detail the five IPs tested and the formulations we use for them in Appendix C; we provide implementation details of our method and the competing methods in

4.1 IP tasks and experimental results

Linear IPs Due to space constraints, our experimental results on Super-resolution & inpainting are included in Appendix E.2. Our DMPlug can lead the best SOTA methods by about 2 dB in PSNR and 0 . 02 in SSIM on average, respectively.

Nonlinear IPs Nonlinear deblurring We use the learned blurring operators from [66] with a known Gaussian-shaped kernel and Gaussian additive noise with σ = 0 . 01 , following [20, 19]. We compare our DMPlug against several strong baselines: Blur Kernel Space (BKS)styleGAN2 [66] based on GAN priors, BKS-generic [66] based on HyperLaplacian priors [68], and DM-based methods that can handle nonlinear IPs, including MCG, ILVR, DPS and ReSample. Despite the advancements

made by the recent ReSample [20] to enhance the DPS method, from Table 1, it is evident that our DMPlug can still significantly outperform the SOTA ReSample by substantial margins. Specifically, our method improves LPIPS, PSNR, and SSIM by 0 . 05 , 4 . 5 dB, and 0 . 07 , respectively, across the three datasets on average. In addition, our DMPlug delivers a more faithful and precise restoration of details, as shown in Fig. 1.

BID & BID with turbulence BID is about recovering a sharp image x (and kernel k ) from y = k ∗ x + n where ∗ denotes the linear convolution and the spatially invariant blur kernel k is unknown; for BID with turbulence which often arises in long-range imaging through atmospheric turbulence, we model the forward process as a simplified 'tilt-then-blur' process, following [69]: y = k ∗T ϕ ( x )+ n , where T ϕ ( · ) is the tilt operation parameterized by unknown ϕ (see more details in Appendix C). We mainly compare our DMPlug to two DM-based models, including ILVR and BlindDPS [21], which is an extension of DPS. In addition, we choose two classical MAP-based methods, i.e., Pan-Dark Channel Prior (Pan-DCP) [70] and Panℓ 0 [71], DIP-based SelfDeblur [72], and four methods that are based on supervised training on paired datasets, including DeBlurGANv2 [5], Stripformer [6], MPRNet [7] and TSR-WGAN [67]. It is important to mention that BlindDPS [21] use pretrained DMs not only for images but also for blur kernels (and tilt maps), lending it an unfair advantage over other methods . Although our method is flexible in working with multiple DM priors, as shown in Section 4.3, we opt to only use the pretrained DMs for images to ensure a fair comparison. Tables 2 and 3 show that our method, despite using fewer priors than BlindDPS, can surpass the best competing methods by approximately 0 . 03 , 4 . 5 dB, and 0 . 1 in terms of LPIPS, PSNR, and SSIM, respectively, on average. In Fig. 1, the reconstructions of our method look sharper and more precise than those of the main competitors.

4.2 Robustness to unknown noise

Measurement

Reference

DMPlug (ours)

DPS

ReSample

For robustness experiments, we choose super-resolution and nonlinear deblurring to represent linear and nonlinear IPs, respectively. To simulate scenarios involving unknown noise, we generate measurements with four types of noise: Gaussian, impulse, shot, and speckle noise, and across two different noise levels: low (level-1) and high (level-2), following [43] (see details in Appendix D.1), but we use the same formulation and code for each IP designed for mild Gaussian noise, regardless of the actual noise type and level. Table 4 and Fig. 7 clearly show that (1) most current IP solvers, except for DPS, suffer from the robustness issue , corroborating the hypotheses made in Section 2, and (2) the peak performance of our DMPlug can lead SOTA methods by around 1 dB and 3 . 5 dB in PSNR for the two tasks, respectively. To check the compatibility of our method with ES-WMV [35], we measure the detection performance via PSNR gaps, i.e., the absolute PSNR difference between the peak and the detected ES point following [35, 59]. Table 4 indicates that the detection gaps in the two exemplary tasks are nearly negligible, with PSNR gaps smaller than 0 . 5 dB and 0 . 2 dB, respectively. This suggests that the proposed method is highly synergetic with ES-WMV .

4.3 Ablation studies

We conduct two ablation studies to demonstrate the flexibility of our method in terms of using multiple and non-DDIM DM priors and using alternative optimizers. (Flexibility of using DM priors) First, we explore the possibility of using different types of DMs. For super-resolution, we show in Table 5 that latent diffusion models (LDMs) [74, 25] are also synergetic with our method. Next, we study the potential of using multiple DMs together, taking BID with turbulence as an example. Using extra pretrained DMs for blur kernels and tilt maps from [21], our method can achieve even better reconstruction results. (Flexibility of using alternative optimizers) Here, we test the built-in ADAM [47] and L-BFGS [75] optimizers in PyTorch , with several different learning rates to solve Eq. (7). As shown in Table 6, the best ADAM and L-BFGS combinations can lead to comparable performance for our DMPlug. We choose ADAM as the default optimizer because in PyTorch , optimizing multiple groups of variables with different learning rates-as for the case of BID (with turbulence)-is easy to program with ADAM but tricky for L-BFGS .

5 Discussion

In this paper, we focus on solving IPs with pretrained DMs. To deal with (Issue 1) insufficient manifold feasibility and (Issue 2) insufficient measurement feasibility of the prevailing interleaving methods, we pioneer a novel plug-in method , DMPlug, and make it practical in terms of computation and memory requirements. Taking advantage of a benign ELTO property and integrating an ES method ES-WMV [35], our method is the first to achieve robustness to unknown noise (Issue 3) . Extensive experiment results demonstrate that our method can lead SOTA methods, both qualitatively and quantitatively-often by large margins, particularly for nonlinear IPs. As for limitations, our empirical results in Section 3.1 suggest a fundamental gap between image generation and regression using pretrained DMs, that we have not managed to nail down. Also, our work is mostly empirical, and we leave a solid theoretical understanding for future work.

References

J. Janai, F. Güney, A. Behl, and A. Geiger, 'Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Art,' Mar. 2021, arXiv:1704.05519 [cs]. [Online]. Available: http://arxiv.org/abs/1704.05519

R. Szeliski, Computer Vision: Algorithms and Applications , ser. Texts in Computer Science. Cham: Springer International Publishing, 2022. [Online]. Available: https://link.springer.com/10.1007/ 978-3-030-34372-9

R. Olsen and R. Olsen, 'Introduction to Remote Sensing,' Jan. 2007, book Title: Remote Sensing from Air and Space ISBN: 9780819462350 Publisher: SPIE. [Online]. Available: https://spiedigitallibrary.org/eBooks/PM/Remote-Sensing-from-Air-and-Space/Chapter1/ Introduction-to-Remote-Sensing/10.1117/3.673407.ch1

J. Sylvester and C. L. Epstein, 'Introduction to the Mathematics of Medical Imaging,' in The American Mathematical Monthly , vol. 112, May 2005, p. 479, iSSN: 00029890 Issue: 5. [Online]. Available: https://www.jstor.org/stable/10.2307/30037514?origin=crossref

O. Kupyn, T. Martyniuk, J. Wu, and Z. Wang, 'DeblurGAN-v2: Deblurring (Orders-of-Magnitude) Faster and Better,' Aug. 2019, arXiv:1908.03826 [cs]. [Online]. Available: http://arxiv.org/abs/1908.03826

F.-J. Tsai, Y .-T. Peng, Y .-Y . Lin, C.-C. Tsai, and C.-W. Lin, 'Stripformer: Strip Transformer for Fast Image Deblurring,' Jul. 2022, arXiv:2204.04627 [cs]. [Online]. Available: http://arxiv.org/abs/2204.04627

S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, 'MultiStage Progressive Image Restoration,' Mar. 2021, arXiv:2102.02808 [cs]. [Online]. Available: http://arxiv.org/abs/2102.02808

M. Delbracio and P. Milanfar, 'Inversion by direct iteration: An alternative to denoising diffusion for image restoration,' Transactions on Machine Learning Research , 2023, featured Certification. [Online]. Available: https://openreview.net/forum?id=VmyFF5lL3F

D. Gilton, G. Ongie, and R. Willett, 'Deep Equilibrium Architectures for Inverse Problems in Imaging,' Jun. 2021, arXiv:2102.07944 [cs, eess]. [Online]. Available: http://arxiv.org/abs/2102.07944

Z. Luo, F. K. Gustafsson, Z. Zhao, J. Sjölund, and T. B. Schön, 'Image Restoration with Mean-Reverting Stochastic Differential Equations,' May 2023, arXiv:2301.11699 [cs]. [Online]. Available: http://arxiv.org/abs/2301.11699

L. Guo, C. Wang, W. Yang, S. Huang, Y. Wang, H. Pfister, and B. Wen, 'ShadowDiffusion: When Degradation Prior Meets Diffusion Model for Shadow Removal,' Dec. 2022, arXiv:2212.04711 [cs]. [Online]. Available: http://arxiv.org/abs/2212.04711

J. Whang, M. Delbracio, H. Talebi, C. Saharia, A. G. Dimakis, and P. Milanfar, 'Deblurring via Stochastic Refinement,' Dec. 2021, arXiv:2112.02475 [cs, eess]. [Online]. Available: http://arxiv.org/abs/2112.02475

S. Gao, X. Liu, B. Zeng, S. Xu, Y. Li, X. Luo, J. Liu, X. Zhen, and B. Zhang, 'Implicit Diffusion Models for Continuous Super-Resolution,' Sep. 2023, arXiv:2303.16491 [cs]. [Online]. Available: http://arxiv.org/abs/2303.16491

O. Kupyn, V. Budzan, M. Mykhailych, D. Mishkin, and J. Matas, 'DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks,' Apr. 2018, arXiv:1711.07064 [cs]. [Online]. Available: http://arxiv.org/abs/1711.07064

G. Ongie, A. Jalal, C. A. Metzler, R. G. Baraniuk, A. G. Dimakis, and R. Willett, 'Deep Learning Techniques for Inverse Problems in Imaging,' May 2020, arXiv:2005.06001 [cs, eess, stat]. [Online]. Available: http://arxiv.org/abs/2005.06001

V. Monga, Y. Li, and Y. C. Eldar, 'Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing,' Aug. 2020, arXiv:1912.10557 [cs, eess]. [Online]. Available: http://arxiv.org/abs/1912.10557

K. Zhang, W. Ren, W. Luo, W.-S. Lai, B. Stenger, M.-H. Yang, and H. Li, 'Deep Image Deblurring: A Survey,' May 2022, arXiv:2201.10700 [cs]. [Online]. Available: http://arxiv.org/abs/2201.10700

J. Koh, J. Lee, and S. Yoon, 'Single-image deblurring with neural networks: A comparative survey,' Computer Vision and Image Understanding , vol. 203, p. 103134, Feb. 2021. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S1077314220301533

H. Chung, J. Kim, M. T. Mccann, M. L. Klasky, and J. C. Ye, 'Diffusion Posterior Sampling for General Noisy Inverse Problems,' Feb. 2023, arXiv:2209.14687 [cs, stat]. [Online]. Available: http://arxiv.org/abs/2209.14687

B. Song, S. M. Kwon, Z. Zhang, X. Hu, Q. Qu, and L. Shen, 'Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency,' Oct. 2023, arXiv:2307.08123 [cs]. [Online]. Available: http://arxiv.org/abs/2307.08123

H. Chung, J. Kim, S. Kim, and J. C. Ye, 'Parallel Diffusion Models of Operator and Image for Blind Inverse Problems,' Nov. 2022, arXiv:2211.10656 [cs, stat]. [Online]. Available: http://arxiv.org/abs/2211.10656

P. Dhariwal and A. Nichol, 'Diffusion Models Beat GANs on Image Synthesis,' Jun. 2021, arXiv:2105.05233 [cs, stat]. [Online]. Available: http://arxiv.org/abs/2105.05233

J. Ho, A. Jain, and P. Abbeel, 'Denoising Diffusion Probabilistic Models,' Dec. 2020, arXiv:2006.11239 [cs, stat]. [Online]. Available: http://arxiv.org/abs/2006.11239

J. Song, C. Meng, and S. Ermon, 'Denoising Diffusion Implicit Models,' Oct. 2022, arXiv:2010.02502 [cs]. [Online]. Available: http://arxiv.org/abs/2010.02502

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, 'High-Resolution Image Synthesis with Latent Diffusion Models,' Apr. 2022, arXiv:2112.10752 [cs]. [Online]. Available: http://arxiv.org/abs/2112.10752

Y. He, N. Murata, C.-H. Lai, Y. Takida, T. Uesaka, D. Kim, W.-H. Liao, Y. Mitsufuji, J. Z. Kolter, R. Salakhutdinov, and S. Ermon, 'Manifold Preserving Guided Diffusion,' Oct. 2023. [Online]. Available: https://openreview.net/forum?id=o3BxOLoxm1

X. Xu and Y. Chi, 'Provably Robust Score-Based Diffusion Posterior Sampling for Plug-and-Play Image Reconstruction.' [Online]. Available: https://www. semanticscholar.org/paper/Provably-Robust-Score-Based-Di%EF%AC%80usion-Posterior-for-Xu-Chi/ a713ae4d638ef679a7165fbf7301a6c045e2588d?utm_source=alert_email&utm_content=LibraryFolder& utm_campaign=AlertEmails_DAILY&utm_term=LibraryFolder&email_index=0-0-0&utm_medium= 33840575&citedSort=relevance&citedQueryString=Denoising%20Diffusion%20Models%20for% 20Plug-and-Play%20Image%20Restoration

Y. Zhu, K. Zhang, J. Liang, J. Cao, B. Wen, R. Timofte, and L. Van Gool, 'Denoising Diffusion Models for Plug-and-Play Image Restoration,' May 2023, arXiv:2305.08995 [cs, eess]. [Online]. Available: http://arxiv.org/abs/2305.08995

X. Meng and Y. Kabashima, 'Diffusion Model Based Posterior Sampling for Noisy Linear Inverse Problems,' Jan. 2024, arXiv:2211.12343 [cs, math, stat]. [Online]. Available: http: //arxiv.org/abs/2211.12343

H. Chung, B. Sim, D. Ryu, and J. C. Ye, 'Improving Diffusion Models for Inverse Problems using Manifold Constraints,' Oct. 2022, arXiv:2206.00941 [cs, stat]. [Online]. Available: http://arxiv.org/abs/2206.00941

X. Li, Y. Ren, X. Jin, C. Lan, X. Wang, W. Zeng, X. Wang, and Z. Chen, 'Diffusion Models for Image Restoration and Enhancement - A Comprehensive Survey,' Aug. 2023, arXiv:2308.09388 [cs]. [Online]. Available: http://arxiv.org/abs/2308.09388

B. Kawar, M. Elad, S. Ermon, and J. Song, 'Denoising Diffusion Restoration Models,' Oct. 2022, arXiv:2201.11793 [cs, eess]. [Online]. Available: http://arxiv.org/abs/2201.11793

Y. Wang, J. Yu, and J. Zhang, 'Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model,' Dec. 2022, arXiv:2212.00490 [cs]. [Online]. Available: http://arxiv.org/abs/2212.00490

G. Liu, H. Sun, J. Li, F. Yin, and Y. Yang, 'Accelerating Diffusion Models for Inverse Problems through Shortcut Sampling,' May 2024, arXiv:2305.16965 [cs, eess]. [Online]. Available: http://arxiv.org/abs/2305.16965

H. Wang, T. Li, Z. Zhuang, T. Chen, H. Liang, and J. Sun, 'Early Stopping for Deep Image Prior,' Dec. 2023, arXiv:2112.06074 [cs, eess]. [Online]. Available: http://arxiv.org/abs/2112.06074

A. Hyvärinen, 'Estimation of Non-Normalized Statistical Models by Score Matching,' Journal of Machine Learning Research , vol. 6, no. 24, pp. 695-709, 2005. [Online]. Available: http://jmlr.org/papers/v6/hyvarinen05a.html

Y. Song and S. Ermon, 'Generative Modeling by Estimating Gradients of the Data Distribution,' Oct. 2020, arXiv:1907.05600 [cs, stat]. [Online]. Available: http://arxiv.org/abs/1907.05600

J. Song, C. Meng, and A. Vahdat, 'Denoising diffusion models: A generative learning big bang,' Nov. 2023, CVPR 2023 Tutorial. [Online]. Available: https://cvpr2023-tutorial-diffusion-models.github.io/

Z. Kadkhodaie and E. Simoncelli, 'Stochastic Solutions for Linear Inverse Problems using the Prior Implicit in a Denoiser,' in Advances in Neural Information Processing Systems , vol. 34. Curran Associates, Inc., 2021, pp. 13242-13254. [Online]. Available: https: //proceedings.neurips.cc/paper/2021/hash/6e28943943dbed3c7f82fc05f269947a-Abstract.html

Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, 'Score-Based Generative Modeling through Stochastic Differential Equations,' Feb. 2021, arXiv:2011.13456 [cs, stat]. [Online]. Available: http://arxiv.org/abs/2011.13456

J. Choi, S. Kim, Y. Jeong, Y. Gwon, and S. Yoon, 'ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models,' Sep. 2021, arXiv:2108.02938 [cs]. [Online]. Available: http://arxiv.org/abs/2108.02938

Y. Sanghvi, Y. Chi, and S. H. Chan, 'Kernel diffusion: An alternate approach to blind deconvolution,' arXiv preprint arXiv:2312.02319 , 2023.

D. Hendrycks and T. Dietterich, 'Benchmarking Neural Network Robustness to Common Corruptions and Perturbations,' Mar. 2019, arXiv:1903.12261 [cs, stat]. [Online]. Available: http://arxiv.org/abs/1903.12261

F. Li, F. Fang, Z. Li, and T. Zeng, 'Single image noise level estimation by artificial noise,' Signal Processing , vol. 213, p. 109215, Dec. 2023. [Online]. Available: https: //www.sciencedirect.com/science/article/pii/S016516842300289X

C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu, 'DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps,' Advances in Neural Information Processing Systems , vol. 35, pp. 5775-5787, Dec. 2022. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/ 2022/hash/260a14acce2a89dad36adc8eefe7c59e-Abstract-Conference.html

K. Zheng, C. Lu, J. Chen, and J. Zhu, 'DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statistics,' Advances in Neural Information Processing Systems , vol. 36, pp.

55502-55542, Dec. 2023. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2023/ hash/ada8de994b46571bdcd7eeff2d3f9cff-Abstract-Conference.html

D. P. Kingma and J. Ba, 'Adam: A Method for Stochastic Optimization,' Jan. 2017, arXiv:1412.6980 [cs]. [Online]. Available: http://arxiv.org/abs/1412.6980

A. Creswell and A. A. Bharath, 'Inverting The Generator Of A Generative Adversarial Network,' Nov. 2016, arXiv:1611.05644 [cs]. [Online]. Available: http://arxiv.org/abs/1611.05644

J.-Y. Zhu, P. Krähenbühl, E. Shechtman, and A. A. Efros, 'Generative Visual Manipulation on the Natural Image Manifold,' Dec. 2018, arXiv:1609.03552 [cs]. [Online]. Available: http://arxiv.org/abs/1609.03552

X. Pan, X. Zhan, B. Dai, D. Lin, C. C. Loy, and P. Luo, 'Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation,' Jul. 2020, arXiv:2003.13659 [cs, eess]. [Online]. Available: http://arxiv.org/abs/2003.13659

X. Su, J. Song, C. Meng, and S. Ermon, 'Dual Diffusion Implicit Bridges for Image-to-Image Translation,' Mar. 2023, arXiv:2203.08382 [cs]. [Online]. Available: http://arxiv.org/abs/2203.08382

A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-Or, 'Prompt-to-Prompt Image Editing with Cross Attention Control,' Aug. 2022, arXiv:2208.01626 [cs]. [Online]. Available: http://arxiv.org/abs/2208.01626

G. Kim, T. Kwon, and J. C. Ye, 'DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation,' Aug. 2022, arXiv:2110.02711 [cs]. [Online]. Available: http://arxiv.org/abs/2110.02711

R. Mokady, A. Hertz, K. Aberman, Y. Pritch, and D. Cohen-Or, 'Null-text Inversion for Editing Real Images using Guided Diffusion Models,' Nov. 2022, arXiv:2211.09794 [cs]. [Online]. Available: http://arxiv.org/abs/2211.09794

B. Wallace, A. Gokul, and N. Naik, 'EDICT: Exact Diffusion Inversion via Coupled Transformations,' pp. 22 532-22 541, 2023. [Online]. Available: https://openaccess.thecvf.com/content/CVPR2023/html/ Wallace_EDICT_Exact_Diffusion_Inversion_via_Coupled_Transformations_CVPR_2023_paper.html

K. Gregor and Y. LeCun, 'Learning fast approximations of sparse coding,' in Proceedings of the 27th International Conference on International Conference on Machine Learning , ser. ICML'10. Madison, WI, USA: Omnipress, Jun. 2010, pp. 399-406.

D. Ulyanov, A. Vedaldi, and V. Lempitsky, 'Deep Image Prior,' International Journal of Computer Vision , vol. 128, no. 7, pp. 1867-1888, Jul. 2020, arXiv:1711.10925 [cs, stat]. [Online]. Available: http://arxiv.org/abs/1711.10925

T. Li, Z. Zhuang, H. Liang, L. Peng, H. Wang, and J. Sun, 'Self-Validation: Early Stopping for Single-Instance Deep Generative Priors,' Oct. 2021, arXiv:2110.12271 [cs, eess]. [Online]. Available: http://arxiv.org/abs/2110.12271

T. Li, H. Wang, Z. Zhuang, and J. Sun, 'Deep Random Projector: Accelerated Deep Image Prior,' pp. 18176-18185, 2023. [Online]. Available: https://openaccess.thecvf.com/content/CVPR2023/html/Li_ Deep_Random_Projector_Accelerated_Deep_Image_Prior_CVPR_2023_paper.html

Z. Shi, P. Mettes, S. Maji, and C. G. M. Snoek, 'On Measuring and Controlling the Spectral Bias of the Deep Image Prior,' Dec. 2021, arXiv:2107.01125 [cs, eess]. [Online]. Available: http://arxiv.org/abs/2107.01125

Z. Zhuang, T. Li, H. Wang, and J. Sun, 'Blind Image Deblurring with Unknown Kernel Size and Substantial Noise,' International Journal of Computer Vision , vol. 132, no. 2, pp. 319-348, Feb. 2024, arXiv:2208.09483 [cs, eess]. [Online]. Available: http://arxiv.org/abs/2208.09483

Z. Liu, P. Luo, X. Wang, and X. Tang, 'Deep Learning Face Attributes in the Wild,' Sep. 2015, arXiv:1411.7766 [cs]. [Online]. Available: http://arxiv.org/abs/1411.7766

T. Karras, S. Laine, and T. Aila, 'A Style-Based Generator Architecture for Generative Adversarial Networks,' Mar. 2019, arXiv:1812.04948 [cs, stat]. [Online]. Available: http://arxiv.org/abs/1812.04948

F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao, 'LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop,' Jun. 2016, arXiv:1506.03365 [cs]. [Online]. Available: http://arxiv.org/abs/1506.03365

R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, 'The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,' Apr. 2018, arXiv:1801.03924 [cs]. [Online]. Available: http://arxiv.org/abs/1801.03924

P. Tran, A. T. Tran, Q. Phung, and M. Hoai, 'Explore Image Deblurring via Encoded Blur Kernel Space,' pp. 11 956-11 965, 2021. [Online]. Available: https://openaccess.thecvf.com/content/CVPR2021/html/ Tran_Explore_Image_Deblurring_via_Encoded_Blur_Kernel_Space_CVPR_2021_paper.html

D. Jin, Y. Chen, Y. Lu, J. Chen, P. Wang, Z. Liu, S. Guo, and X. Bai, 'Neutralizing the impact of atmospheric turbulence on complex scene imaging via deep learning,' Nat. Mach. Intell. , vol. 3, no. 10, pp. 876-884, 2021. [Online]. Available: https://doi.org/10.1038/s42256-021-00392-1

D. Krishnan and R. Fergus, 'Fast Image Deconvolution using Hyper-Laplacian Priors,' in Advances in Neural Information Processing Systems , vol. 22. Curran Associates, Inc., 2009. [Online]. Available: https://papers.nips.cc/paper_files/paper/2009/hash/3dd48ab31d016ffcbf3314df2b3cb9ce-Abstract.html

S. H. Chan, 'Tilt-then-Blur or Blur-then-Tilt? Clarifying the Atmospheric Turbulence Model,' IEEE Signal Processing Letters , vol. 29, pp. 1833-1837, 2022, arXiv:2207.06377 [eess]. [Online]. Available: http://arxiv.org/abs/2207.06377

J. Pan, D. Sun, H. Pfister, and M.-H. Yang, 'Deblurring Images via Dark Channel Prior,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 40, no. 10, pp. 2315-2328, Oct. 2018, conference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence. [Online]. Available: https://ieeexplore.ieee.org/document/8048543

J. Pan, Z. Hu, Z. Su, and M.-H. Yang, 'L_0 -Regularized Intensity and Gradient Prior for Deblurring Text Images and Beyond,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 39, no. 2, pp. 342-355, Feb. 2017, conference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/7448477?casa_token=BY5p4dV5BVwAAAAA: LaPxWMHRFDbrXQYcJmp247D-oOPlvPitreoe4MXKkd53Ku96bdoLz_-JE7lug9H9Cm-CxJlpxkI

D. Ren, K. Zhang, Q. Wang, Q. Hu, and W. Zuo, 'Neural Blind Deconvolution Using Deep Priors,' Mar. 2020, arXiv:1908.02197 [cs]. [Online]. Available: http://arxiv.org/abs/1908.02197

S. H. Chan, X. Wang, and O. A. Elgendy, 'Plug-and-Play ADMM for Image Restoration: Fixed Point Convergence and Applications,' Nov. 2016, arXiv:1605.01710 [cs]. [Online]. Available: http://arxiv.org/abs/1605.01710

A. Vahdat, K. Kreis, and J. Kautz, 'Score-based generative modeling in latent space,' Advances in Neural Information Processing Systems , vol. 34, pp. 11 287-11 302, 2021.

D. C. Liu and J. Nocedal, 'On the limited memory BFGS method for large scale optimization,' Mathematical Programming , vol. 45, no. 1, pp. 503-528, Aug. 1989. [Online]. Available: https://doi.org/10.1007/BF01589116

H. Chihaoui, A. Lemkhenter, and P. Favaro, 'Zero-shot Image Restoration via Diffusion Inversion,' Oct. 2023. [Online]. Available: https://openreview.net/forum?id=ZnmofqLWMQ

M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, 'Gans trained by a two time-scale update rule converge to a local nash equilibrium,' Advances in neural information processing systems , vol. 30, 2017.

P. Chakrabarty and S. Maji, 'The Spectral Bias of the Deep Image Prior,' Dec. 2019, arXiv:1912.08905 [cs]. [Online]. Available: http://arxiv.org/abs/1912.08905

D. L. Ruderman, 'The statistics of natural images,' Network: Computation in Neural Systems , vol. 5, no. 4, pp. 517-548, Jan. 1994. [Online]. Available: https://www.tandfonline.com/doi/full/10.1088/0954-898X_ 5_4_006

Z. Dou and Y. Song, 'Diffusion Posterior Sampling for Linear Inverse Problem Solving: A Filtering Perspective,' Oct. 2023. [Online]. Available: https://openreview.net/forum?id=tplXNcHZs1

L. Rout, N. Raoof, G. Daras, C. Caramanis, A. G. Dimakis, and S. Shakkottai, 'Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models,' Jul. 2023. [Online]. Available: https://arxiv.org/abs/2307.00619v1

A Remarks on the concurrent work [76]

We became aware of the unpublished concurrent work, SHRED [76], when we were finalizing the details of our method in mid Jan 2024 (We started the current project in Oct 2023). Although the core idea of SHRED is the same as that of our DMPlug, there are a few crucial differences between [76] and the current paper. (1) Motivation : [76] aims only at addressing manifold feasibility, while the current paper targets manifold feasibility, measurement feasibility, and robustness to unknown noise, simultaneously; (2) Integral components : Since achieving robustness to unknown noise is part of our goal, the ES-WMV ES strategy is integral to our method, besides the unified optimization formulation Eq. (7) shared with [76]; (3) Key hyperparameters : To address the computational and memory bottleneck induced by reverse steps, we use only 3 reverse steps for all IPs we test, vs. the 10 or more reverse steps used in [76]. So, our current setting makes the method much more practical; (4) Flexibility : Our framework allows for the use of more than one pre-trained DM prior when available, as shown in Section 4.3, while [76] uses only a single DM prior as proposed in their Eqs. (17) to (19); (5) Experimental evaluation : [76] focuses on linear IPs, including inpainting, super-resolution, compressive sensing, and one nonlinear IP-blind deconvolution, i.e., blind image deburring (BID). We focus our evaluation on nonlinear IPs, including nonlinear deblurring, BID, and BID with turbulence, besides linear IPs (inpainting and super-resolution). Moreover, [76] measures performance only by perceptual metrics, LPIPS [65] and FID [77], while we measure performance by both the perceptual metric LPIPS and the classical metrics PSNR and SSIM. In addition, [76] does not even compare their method with clear SOTA methods that use pre-trained DM, e.g., DPS [19], which they obviously were aware of (cited in the paper), whereas our comparison is more comprehensive.

B Spectral bias of our DMPlug

1.2

1.2

1.1

1.0

1.0

0.9

11

Frequencyband (1,lowest)

E0.8

FBE

0.8

B

Frequency band(3)

0.7

二

Frequency band(4)

Frequencyband (5,highest)

0.6

0.6

0.5

0.4

0.4

0

500

1000

1500

2000

2500

0

100

200

300

400

500

Iteration

Iteration

(1)

(2)

One may wonder why the ELTO phenomenon occurs. Here, we borrow the ideas of spectral biases and spectral analysis from the DIP literature [78, 60, 59, 61] to shed some light on this. The theory of spectral biases for DIP states that low-frequency components are learned much faster than highfrequency components during DIP learning. Spectral biases in DIP lead to the ELTO phenomenon in DIP : because natural images are typically low-frequency dominant [79], the different learning paces imply that DIP learns mostly the desired image content (low-frequency image content plus the low-frequency part of noise) in the early stage, but gradually picks up the high-frequency part of noise in the late stage, resulting in performance degradation after a certain quality peak. Spectral analysis provides a quantitative visualization of spectral biases.

Here, we perform a similar spectral analysis of our DMPlug learning process to demonstrate its spectral biases, which causes the ELTO phenomenon. To measure spectral biases, we follow [59, 61] and use frequency band errors (FBEs) . For an groundtruth image x and its estimate ̂ x , the calculation of this metric goes as follows. First, we calculate the pointwise relative error pointwise in the Fourier

domain, i.e., |F ( x ) -F ( ̂ x ) | / |F ( x ) | . Then, we divide the Fourier frequencies into five radial bands, compute the bandwise mean errors, i.e., the frequency-band errors (FBEs).

We visualize the evolution of FBEs of DMPlug over all five frequency bands in Fig. 8. The disparate learning paces across the frequency bands are evident: the lowest-frequency band is learned much more rapidly than the other bands, which is consistent between two different optimization solvers. With spectral biases similar to those in DIP, we can explain the ELTO phenomenon in DMPlug following the argument above for DIP.

C Setup details of the inverse problems we test

For super-resolution, inpainting, and nonlinear deblurring, we use the forward models from two of main competing methods [19, 20]; for BID and BID with turbulence, we follow the forward models from BlindDPS [21]-the SOTA DM-based method. Moreover, following [19, 20], all the measurements contain additive Gaussian noise with σ = 0 . 01 . Our loss ℓ by default is the MSE loss.

C.1 Super-resolution

In noisy image super-resolution, the goal is to reconstruct a clean RGB image x from a noisy downsampled version y = D ( x ) + n , where D ( · ) : [0 , 1] 3 × tH × tW → [0 , 1] 3 × H × W is a downsampling operator that resizes an image with dimensions tH by tW by the factor t and n models additive noise. We set t = 4 in Section 4. To ensure a fair comparison, we do not include any explicit regularization terms in the formulation:

C.2 Inpainting

In noisy image inpainting, a clean RGB image x ∈ [0 , 1] 3 × H × W is only partially observed and then contaminated by additive noise n , described by the forward model y = m ⊙ x + n , where m ∈ { 0 , 1 } 3 × H × W is a binary mask and ⊙ denotes the Hadamard product. Given y and m , the goal is to reconstruct x . Following [20], the masks for the three channels of m are identical, and 70% of the mask values are randomly set to 0 . To ensure a fair comparison, we do not include any explicit regularization terms in the formulation:

C.3 Nonlinear deblurring

Wefollow the setup in [19] which is inspired by [66]. Recently, [66] has proposed to learn data-driven blurring models from paired blurry-sharp training sets of the form { ( y i , x i ) } i =1 ,...,N through

where G β ( · , · ) predicts the latent blur kernel associated with the input blurry-sharp image pair, and ψ β ( · , · ) models real-world nonlinear blurring process given the input image-kernel pair. To study the performance of its DPS method on nonlinear IPs, [19] proposes the following nonlinear deblurring problem with a known Gaussian-shaped kernel:

The task here is to recover x from y and the forward model F α ∗ ( · , g ) . Our formulation follows [19] and does not include extra regularizers:

C.4 Blind image deblurring (BID)

BID is about recovering a sharp image x from y = k ∗ x + n where ∗ denotes the linear convolution and the spatially invariant blur kernel k is also unknown. It is a nonlinear IP because the forward

model A ( k , x ) = k ∗ x is nonlinear. In solving BID under the regularized data-fitting framework in Eq. (1), ℓ 2 or ℓ 1 data-fitting loss, and the sparse gradient prior to x enforced by R x ( x ) = ∥∇ x ∥ 1 or R x ( x ) = ∥∇ x ∥ 1 / ∥∇ x ∥ 2 are typically used. In addition, because of the scale ambiguity in the forward model, i.e., k ∗ x = ( α k ) ∗ ( 1 α x ) for any α > 0 , the scale of k is often fixed by requiring k to be on the standard simplex (i.e., k ≥ 0 , 1 ⊺ k = 1 ) or on the sphere (i.e., ∥ k ∥ 2 = 1 ). [18, 17, 61] provide detailed coverage of these priors and regularizers. For our experiments here, we follow the settings in BlindDPS [19]: all the blur kernels are simulated with the size of 64 × 64 ; the standard deviation of the Gaussian kernels is set to 3 . 0 and the intensity of the motion blur kernels is adjusted to 0 . 5 . It is important to mention that BlindDPS [21] uses pretrained DMs not only for images but also for blur kernels, giving it an unfair advantage over other methods . But to ensure a fair comparison with other competing methods that only use data-driven image priors, we only use pretrained DMs for the image, plus the typical simplex constraint that is also used in BlindDPS:

where SoftMax( k ) leads to a kernel estimate that lies on the standard simplex.

C.5 BID with turbulence

BID with turbulence often arises in long-range imaging through atmospheric turbulence. The forward imaging process can be modeled as a simplified 'tilt-then-blur' process, following [69]: y = k ∗ T ϕ ( x ) + n , where the tilt operator T ϕ ( · ) applies the spatially varying vector field ϕ to the image x so that pixels of x are moved around according to the vector field ϕ , i.e., T ϕ ( x )[ p i + ϕ i ] = x [ p i ] where p i 's are the pixel coordinates. In BID with turbulence, none of the k , x , ϕ is known and the task is to jointly estimate them from the measurement y . So, it is clear that this BID variant is strictly more difficult than BID itself. We follow BlindDPS [21] for data generation: the blur kernel comes from the point spread function (PSF) and takes a Gaussian shape with standard deviation 3 . 0 ; the tilt maps are generated as iid Gaussian random vectors over the pixel grid. Similar to the BID case, BlindDPS [21] use pretrained DMs for all three objects: image, kernel, and tilt map, unfair to other methods . We again use pretrained DMs for the image only. For the kernel, we again only impose the simplex constraint through a SoftMax activation. For the tilt map, inspired by the fact that the tilt vectors are small-magnitude random vectors, we initialize the map from a zero-mean Gaussian distribution with a small standard deviation and set an extremely small learning rate. Our final formulation for this task is

D More implementation details

D.1 Noise generation

Following [43] 3 , we simulate four types of noise, with two intensity levels for each type. The detailed information is as follows. Gaussian noise: zero-mean additive Gaussian noise with variance 0 . 08 and 0 . 12 for low and high noise levels, respectively; Impulse noise: also known as salt-and-pepper noise, replacing each pixel with probability p ∈ [0 , 1] in a white or black pixel with half chance each. Low and high noise levels correspond to p = 0 . 03 and 0 . 06 , respectively; Shot noise: also known as Poisson noise. For each pixel, x ∈ [0 , 1] , the noisy pixel is Poisson distributed with the rate λx , where λ is 60 and 25 for low and high noise levels, respectively; Speckle noise: for each pixel x ∈ [0 , 1] , the noisy pixel is x (1 + ϵ ) , where ϵ is zero-mean Gaussian with a variance level 0 . 15 and 0 . 20 for low and high noise levels, respectively.

D.2 Additional implementation details of our method

We employ the following setup for our methods across all IP tasks. For R ( · ) , we take the standard pretrained DMs from [22] 4 and [41] 5 and use the standard DDIM [24] sampler with only 3 reverse steps based on Fig. 5; we also use pretrained latent diffusion models (LDMs) [25] 6 to obtain the

results reported in Table 5. We use the pretrained DMs for blur kernels and tilt maps from [21] 7 to obtain the results reported in Table 5. For ℓ ( · ) , we choose the standard MSE loss. For Ω( · ) , we use the typical explicit regularizers for each task to make the comparisons fair. The default optimizer is ADAM and the learning rate (LR) for z is 1 × 10 -2 ; for BID (with turbulence), the LRs for blur kernel and the tilt map are 1 × 10 -1 and 1 × 10 -7 , respectively. For the maximum numbers of iterations, we set 5 , 000 and 10 , 000 for linear and nonlinear IPs, respectively, which empirically allow good convergence. We perform all experiments on NVIDIA A100 GPUs with 40 GB memory each.

Algorithm 3 DMPlug+ES-WMV for solving general IPs

Early stopping (ES) using ES-WMV The 'early-learning-then-overfitting' ( ELTO ) phenomenon has been widely reported in the literature on deep image prior (DIP) [57-60, 35], and ES-WMV [35] is an ES strategy that achieves the SOTA ES performance for DIP applied to various IPs. In ES-EMV, the running variance (VAR) is defined as:

where W is the window size and x i denotes the recovery at iteration i . [35] observes that the first major valley of the VAR curve is often well aligned with the peak of the PSNR curve. Based on this, [35] introduces an online algorithm to detect the first major valley of the V AR curve: if the minimal VAR does not change over P consecutive steps, i.e., the V AR does not decrease further over P consecutive steps, the iteration process is stopped. The combined algorithm, ES-WMV-integrated DMPlug, is described in Algorithm 3. For implementation, we use the official code of ES-WMV 8 . For super-resolution with unknown noise, we set its patience number as 100 and its window size as 10 ; for nonlinear deblurring with unknown noise, we set its patience number as 300 and its window size as 50 . Table 4 indicates that the detection gaps in the two exemplary tasks are nearly negligible, with PSNR gaps smaller than 0 . 5 dB and 0 . 2 dB, respectively. This suggests that our DMPlug is highly synergetic with ES-WMV .

D.3 Implementations of competing methods

We use the default code and settings of each competitor's official implementation, listed below.

ADMM-PnP [73]: https://github.com/kanglin755/plug_and_play_admm

DMPS [29]: https://github.com/mengxiangming/dmps

DDRM [32]: https://github.com/bahjat-kawar/ddrm

DPS [19] & MCG [30]: https://github.com/DPS2022/diffusion-posterior-sampling

ILVR [41]: https://github.com/jychoi118/ilvr_adm

ReSample [20]: https://github.com/soominkwon/resample/tree/main

BKS [66]: https://github.com/VinAIResearch/blur-kernel-space-exploring

SelfDeblur [72]: https://github.com/csdwren/SelfDeblur

DeBlurGANv2 [5]: https://github.com/VITA-Group/DeblurGANv2

Stripformer [6]: https://github.com/pp00704831/Stripformer-ECCV-2022-

MPRNet [7]: https://github.com/swz30/MPRNet

Pan-DCP [70]: https://jspan.github.io/projects/dark-channel-deblur/index.html

Panℓ 0 [71]: https://jspan.github.io/projects/text-deblurring/index.html

BlindDPS [21]: https://github.com/BlindDPS/blind-dps

TSR-WGAN [67]: https://codeocean.com/capsule/9958894/tree/v1

FPS [80]: https://github.com/ZehaoDou-official/FPS-SMC-2023

DiffPIR [28]: https://github.com/yuanzhi-zhu/DiffPIR

E More experiment results

E.1 Computational Efficiency

As shown in Table 7, we find that the memory usage of our method is in the same order as that of most other algorithms. When it comes to speed, our implementation using L-BFGS is significantly faster than the ADAM version. Although our method with L-BFGS remains somewhat slow, it surpasses the performance of ReSample. We will explore further acceleration of our method in future work.

E.2 Quantitative results

Super-resolution & inpainting Following [20, 19], to generate measurements, we use bicubic downsampling for super-resolution ( 4 × ) and a random mask with 70% missing pixels for inpainting; all measurements contain additive Gaussian noise with σ = 0 . 01 . We compare our method with Plugand-Play using ADMM (ADMM-PnP) [73] and several SOTA DM-based methods: Diffusion Model based Posterior Sampling (DMPS) [29], Denoising Diffusion Destoration Models (DDRM) [32], Manifold Constrained Gradients (MCG) [30], Iterative Latent Variable Refinement (ILVR) [41], Diffusion Posterior Sampling (DPS) [19], and ReSample [20]. The quantitative results are reported in Tables 8 to 10, with qualitative results visualized in Fig. 1. It is clear that our DMPlug consistently outperforms all competing methods on the three datasets, both quantitatively and qualitatively. Specifically, while there is no significant improvements in terms of LPIPS, the proposed method can lead the best SOTA methods by about 2 dB in PSNR and 0 . 02 in SSIM on average, respectively.

E.3 Qualitative results

F More early stopping results

Bedroom

CelebA

FFHQ

Measurement

Reference

DMPlug (ours)

DPS

ReSample

ADMM-PnP

DMPS

DDRM

Measurement

Reference

DMPlug (ours)

DPS

ReSample

ADMM-PnP

DMPS

DDRM

Bedroom

CelebA

FFHQ

CelebA

FFHQ

Measurement

Reference

DMPlug (ours)

DPS

ReSample

ILVR

BKS-styleGAN

BKS-generic

Bedroom

29

2

CelebA

FFHQ

Measurement

Reference

DMPlug(ours)

Stripformer

ILVR

DeBlurGANv2

MPRNet

CelebA

FFHQ

Measurement

Reference

DMPlug(ours)

Stripformer

ILVR

DeBlurGANv2

MPRNet

CelebA

FFHQ

Measurement

Reference

DMPlug (ours)

Stripformer

TSR-WGAN

DeBlurGANv2

MPRNet

Measurement

Reference

DMPlug(ours)

DPS

ReSample

ADMM-PnP

DMPS

DDRM

Impulse

Shot

Speckle

Measurement

Reference

DMPlug (ours)

DPS

ReSample

ILVR

BKS-styleGAN

BKS-generic

Impulse

Shot

Speckle

1.50

XVI

Impulse

X

3

15.0

12.5

0.0

.50

Shot

2.5

10

yM

18

M

Speckle

15.0

日日

15.0

日日日日

Impulse

日日日日

25.0

15.0

Shot

日日日

150

6

Speckle

15.0

日日日日

日月

15.0

15.0

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly claim the scope and contributions of this paper in both abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of this work in Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All the formulas are numbered and cross-referenced.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide all the experiment details in Section 4, Appendix C and Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide all the essential code for this paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all the experiment details in Section 4, Appendix C and Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: First, we conduct extensive experiments in this paper and report the mean results. We do not anticipate significant fluctuations in the results. Second, we are afraid that adding the statistical significance of the experiments will mess this paper up because our tables are already very dense, but we are willing to provide them if needed.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the experiment compute resources in Appendix D.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This paper is strictly with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the societal impacts in Section 5.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper only uses existing pretrained models for zero-shot tasks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We provide all the code and models we have used in Appendix D.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The new assets in this paper are well documented.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is devoted to the use of diffusion models to solve inverse problems, arguing for treating the inverse process in a diffusion model as a function and proposing a new plug-in approach called DMPlug. DMPlug addresses the issues of feasibility and measurement feasibility of manifolds in a principled manner, and shows great potential for robustness to unknown types and levels of noise.\n\nSTRENGTHS:\n1. The article is very comprehensive in its review of existing relevant technologies, which is more friendly to readers unfamiliar with the field.\n\n2. The article describes the proposed method in a clearer and more detailed way, which is easy to understand.\n\n3. The qualitative results on celeba and FFHQ datasets presented in the figures of the experimental sections look very effectual and better than the baseline method.\n\n4. This paper provides a comprehensive exploration of the robustness of the proposed method under various noises.\n\nWEAKNESSES:\n1. *Contributing issues:* The proposed method is very similar to the existing work [1,2] in its approach to condition generation by optimizing the noisy latent variables of a diffusion model. The main difference in this paper (and why it works is also very surprising to me) is the empirical evidence that good results can be achieved with only 3 backward steps. It is well known that 3 backward steps usually fail to generate meaningful images for diffusion models. Although the authors do not provide a theoretical basis, they should also provide an intuitive insight into why it works.\n\n2.  *Experimental issues:* The experimental section states that their metrics are calculated primarily on 100 images, which is too few, and much of the existing work is 1000-image tests. In addition, the paper selects Celeba and FFHQ which have somewhat similar patterns despite being two datasets.\n\n[1] D-Flow: Differentiating through Flows for Controlled Generation\n[2] End-to-End Diffusion Latent Optimization Improves Classifier Guidance\n\nQUESTIONS:\n1. Although the authors point out that the use of 3-step DDIM can significantly mitigate computational consumption, the memory required to save the computational graphs of 3-stepDDIM is still huge for some larger diffusions, as far as I know. Can the authors report a comparison with the time consumption and GPU memory used by other methods, such as DPS.\n\n2. Can the author provide some intuition or explanation as to why the 3 steps work.\n\n3. Can the author show some qualitative results of DMPlug in more difficult cases, such as inpainting of box masks or the imagenet dataset?      \n\n3. Can the proposed DMPlug method with 3 steps be used for other controllable generation tasks rather than just inverse problems such as, Classifier/Clip guidance in [2].\n\n[2] End-to-End Diffusion Latent Optimization Improves Classifier Guidance\n\n\nIf the author can address my concerns, I am open to changing my score during the discussion phase.\n\nLIMITATIONS:\nThe authors briefly discuss for the limitations of their approach.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a plug-and-play method DMPlug for solving inverse problems with diffusion models. DMPlug utilizes a pre-trained diffusion model as a deterministic function that generates images from latent seeds and solves MAP problems by optimizing the seeds. Experiments show that their method beats current SOTA methods across different inverse problems and is robust against unknown perturbations.\n\nSTRENGTHS:\n- The proposed algorithm DMPlug is concise and novel. \n- The authors observe an intriguing early-learning-then-overfitting property of their algorithm. They integrate an early-stopping strategy in DMPlug to enhance the robustness of DMPlug against unknown data corruption, which is illustrative and useful.\n- The experimental results shown in the paper look promising. The empirical robustness analysis is also interesting and informative.\n\nWEAKNESSES:\n- The paper is missing many baseline methods on plug-and-play inverse problem solving with diffusion models, such as DDNM[1], DiffPIR[2], RED-Diff[3], FPS[4]. These methods are claimed to perform much better than DPS. The experimental result is less convincing without comparison with these recent results. \n- Moreover, the major recent baseline considered in the paper, ReSample, uses pre-trained latent diffusion models, different from the pre-trained model used in this paper, so a direct comparison does not make sense. This weakness is partly mitigated by the ablation study in Table 5.\n\n[1] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. ICLR, 2023.\n\n[2] Yuanzhi Zhu, Kai Zhang, Jingyun Liang, Jiezhang Cao, Bihan Wen, Radu Timofte, and Luc Van Gool. Denoising diffusion models for plug-and-play image restoration. CVPR, 2023.\n\n[3] Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat. A variational perspective on solving inverse problems with diffusion models. ICLR, 2024.\n\n[4] Zehao Dou, Yang Song. Diffusion Posterior Sampling for Linear Inverse Problem Solving: A Filtering Perspective. ICLR, 2024.\n\nQUESTIONS:\n- DMPlug seems more space/time-consuming than existing methods since it requires gradient backpropagation through the diffusion model multiple times per step, while ReSample does not backpropagate through the diffusion model. Depending on the actual number of iterations used for DMPlug, the time cost for DMPlug might be prohibitively high.  I am curious to see some discussion and comparison of DMPlug with existing methods in terms of time cost.\n\nLIMITATIONS:\nThe authors address the limitations and provide possible future directions in their paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an optimization-based method for optimize the initial noise for data consistency. This method is evaluated on a variety of linear inverse problems and non-linear inverse problems, and achieves SOTA results. The method also show robustness for unknown noise levels with a technique called ES-WMV.\n\nSTRENGTHS:\n1. The method looks novel and straightforward to implement. \n2. The paper is well written and easy to understand\n3. diffusion inverse solvers struggle in nonlinear problems since that restricts the usage of SVD or null space decomposition. This paper improves the SOTA on nonlinear inverse problems by a significant margin\n4. The Early stopping/deep image prior phenomena reported in this paper is interesting.\n\nWEAKNESSES:\n1. the R(.) function looks quite long since it involves many reverse passes. Also the optimization may take up to 5000 iterations. Does that require additional memory and computing time? I cannot find the memory and time cost in this manuscript except in Fig.5\n2. the results on LDMs are limited, it would be more desirable to include LDM results on more nonlinear problems and linear problems besides super-resolution.\n3. the backbone model should be stated more clearly in the paper: what is the pretrained model? How is it trained? How many NFEs are used after optimizing the initial noise?\n4. Authors could provide more intuitions on why 3 ddim reverse steps are sufficient since it is challenging to generate high-quality images with 3 steps and the measurement function A(.) is applied on the $E[x_0|x_t]$ and may not reflect the ground truth signal. I think deblur or superresolution works since 3 DDIM reverse steps give blurry images which is quite consistent to measurement, but have the authors experiment on more challenging forward models like CT operator (Radon Transform) or phase retrieval?\n\nQUESTIONS:\nSee weaknesses. \n1. memory and computing time\n2. clearer description of the pretrained model\n3. experimenting on more challenging forward models like CT operator or phase retrieval\n4. more LDM results\n\nLIMITATIONS:\nYes, limitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new framework, called DM-Plug, to solve inverse problems with pre-trained diffusion models. Most of the prior works on this topic propose approximations for the conditional score function. The authors propose an alternative approach that is closely related to techniques for solving inverse problems with GANs. Namely, the authors consider the latent space that emerges in diffusion models by projecting images through the deterministic sampler and perform the optimization in this latent space.\n\n\nThere are two challenges with this approach: 1) one needs to backpropagate through the whole sampling chain to perform this optimization, so this approach can be really computationally intensive, and 2) for every update of the latent, one needs to run the whole sampler, so this approach can become really slow. The authors mention that they can circumvent these two issues by only running 3 steps of the sampler and that this suffices for their considered inverse problems.\n\nSTRENGTHS:\n1) The topic of this research paper is interesting. There is a growing literature on approaches for solving inverse problems with diffusion models. This paper proposes a fresh idea in this space.\n2) The experimental results are strong, providing evidence that this is a promising approach.\n3) Since the approach here is very similar to solving inverse problems with GANs, there is a variety of ideas from this space that can be potentially leveraged to further improve the results of this approach.\n\nWEAKNESSES:\n1) The proposed method is a maximum-likelihood method and does not offer posterior samples. For certain inverse problems, being able to sample from the posterior is really important, for uncertainty quantification, diversity, etc. \n2) The presentation of the paper could be improved. For example, one idea would be to directly optimize in the space of clean images, since diffusion models also offer the score there. This idea wouldn't work because the estimation of the score for $t=0$ is usually poor. The authors do not explain this properly in their manuscript. \n3) The authors also mention that for conditional tasks, 3 steps are enough. However, this should truly depend on the level of corruption. At the limit of extreme corruption, it can't be true that $3$ sampling steps are enough since we know for a fact that diffusion models need more steps to achieve good generation results.\n4) From Figure 6.1., it looks like the MSE is minimized at around 1000 iterations of the proposed algorithm. Given that each iteration requires 3 sampling steps, this is equivalent to 3000 sampling steps. Most methods for solving inverse problems with diffusion models work with much fewer sampling steps.\n5) The authors could have used better baselines in their experimental evaluation, including ΠDGM, Red-diff and PSLD.\n\nQUESTIONS:\n1) It seems to me that one of the main weaknesses of this work is the propagation over the sampling chain. Since all that is needed is the solution of the ODE, is it possible to use this method with Consistency Diffusion Models?\n2) There are many techniques to solve inverse problems with GANs. One of these techniques is Intermediate Layer Optimization, where the optimization happens in some intermediate latent space. This idea seems directly relevant here and could lead to improvements in performance and cost. Namely, the authors could optimize in the latent space that corresponds to some time other than $t=T$. Could the authors ablate this?\n3) I would like to see the number of required steps needed in the sampling chain as the difficulty of the diffusion problem increases. Namely, the authors could fix an inverse problem, let's say random inpainting, and ablate the number of required sampling steps for good performance as the corruption probability increases.\n\nLIMITATIONS:\nThe authors have addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}