Title: Principled Probabilistic Imaging using Diffusion Models as Plug-and-Play Priors
Venue: NeurIPS 2024
Accepted: yes
Keywords: Computational imaging

Inverse problems

Diffusion models
Abstract: Diffusion models (DMs) have recently shown outstanding capabilities in modeling complex image distributions, making them expressive image priors for solving Bayesian inverse problems. However, most existing DM-based methods rely on approximations in the generative process to be generic to different inverse problems, leading to inaccurate sample distributions that deviate from the target posterior defined within the Bayesian framework. To harness the generative power of DMs while avoiding such approximations, we propose a Markov chain Monte Carlo algorithm that performs posterior sampling for general inverse problems by reducing it to sampling the posterior of a Gaussian denoising problem. Crucially, we leverage a general DM formulation as a unified interface that allows for rigorously solving the denoising problem with a range of state-of-the-art DMs. We demonstrate the effectiveness of the proposed method on six inverse problems (three linear and three nonlinear), including a real-world black hole imaging problem. Experimental results indicate that our proposed method offers more accurate reconstructions and posterior estimation compared to existing DM-based imaging inverse methods.

Principled Probabilistic Imaging using Diffusion Models as Plug-and-Play Priors

Abstract

Diffusion models (DMs) have recently shown outstanding capabilities in modeling complex image distributions, making them expressive image priors for solving Bayesian inverse problems. However, most existing DM-based methods rely on approximations in the generative process to be generic to different inverse problems, leading to inaccurate sample distributions that deviate from the target posterior defined within the Bayesian framework. To harness the generative power of DMs while avoiding such approximations, we propose a Markov chain Monte Carlo algorithm that performs posterior sampling for general inverse problems by reducing it to sampling the posterior of a Gaussian denoising problem. Crucially, we leverage a general DM formulation as a unified interface that allows for rigorously solving the denoising problem with a range of state-of-the-art DMs. We demonstrate the effectiveness of the proposed method on six inverse problems (three linear and three nonlinear), including a real-world black hole imaging problem. Experimental results indicate that our proposed method offers more accurate reconstructions and posterior estimation compared to existing DM-based imaging inverse methods.

1 Introduction

Inverse problems arise in many computational imaging applications, where the goal is to recover an image x ∈ R n from a set of sparse and noisy measurements y ∈ R m . The relationship between x and y can be described by

where A ( · ) : R n → R m is the forward operator (linear or nonlinear) and n is the random measurement noise in R m . Since the sparsity and noisiness of y often lead to significant uncertainty in x , it is preferable to sample the posterior distribution p ( x | y ) over all possible solutions based on some prior distribution p ( x ) , rather than finding a single deterministic solution. Traditional posterior sampling methods often rely on simple image priors that do not reflect the sophistication of real-world image distributions. On the other hand, diffusion models (DMs) have recently emerged as a powerful tool for modeling highly complex image distributions [33, 63]. Nevertheless, it remains a challenge to turn DMs into reliable imaging inverse solvers, which motivates us to develop a principled Bayesian method that leverages DMs as priors for posterior sampling.

Diffusion models generate samples from a distribution by reversing a diffusion process from the target distribution to a simple (usually Gaussian) distribution [33, 63]. In particular, it estimates a clean image x 0 from a noise image x T by successively denoising noisy images, where x t ∼ p t is the intermediate noisy image at time t ∈ [0 , T ] . Reversing diffusion requires one to estimate the time-varying gradient log density (score function) ∇ log p t ( x t ) along the diffusion process, or ∇ log p t ( x t | y ) in the case of sampling the posterior p ( x | y ) .

2 3 4

Mean image

Posterior samples

Official image by EHT

Diffusion model image prior

Real-world imaging system

* Experiment is performed with

real data for the M87 black hole

Interferometry

N-S frequency (v)

E-W frequency (u)

Fourier spectrum

PnP-DM

Proposed

method

40

μ

as

40

μ

as

To design generic DM-based inverse problem solvers, most existing methods attempt to approximate the time-varying gradient log density ∇ log p t ( x t | y ) [17, 73, 84, 62, 39, 58, 60, 44, 15, 77, 18, 55, 8]. In particular they first apply Bayes' rule to separate the forward operator from an unconditional prior over the intermediate noisy image x t :

By instead aiming to evaluate the right hand side, one can leverage the existing pre-trained DMs for the unconditional term ∇ log p t ( x t ) . However, the main challenge in this case is that ∇ log p t ( y | x t ) is intractable to compute in general, as p t ( y | x t ) involves an integral over all possible x 0 's that could give rise to x t [17]. Various methods have been proposed to circumvent the intractability and can mostly be categorized into two groups. One group of methods explicitly approximate ∇ log p t ( y | x t ) by making simplifying assumptions [62, 17, 60, 8]. However, even for arguably the finest approximation to date proposed in the recent work [8], it is exact only when the prior distribution p ( x ) is Gaussian. For general prior distributions beyond Gaussian, these methods do not sample the true posterior p ( x | y ) . The other group of methods do not make explicit approximations but instead substitute ∇ log p t ( y | x t ) with empirically designed updates where y is treated as a guidance signal [73, 84, 39, 58, 44, 15, 77, 18, 55]. Although these methods may have strong empirical performance, they have deviated from the Bayesian formulation and no longer aim to sample the target posterior. In summary, these existing DM-based inverse methods should be best viewed as guidance methods , where the generative process is guided towards the regions where the measurement y is more likely to be observed, not as posterior sampling methods [8]. We also note that some recent work considered combing DMs with Sequential Monte Carlo to ensure asymptotic consistency in posterior sampling [11, 23], but the investigation has been limited to linear imaging inverse problems.

Our contributions In this work, we pursue a different path towards posterior sampling with DM priors by proposing a new Markov chain Monte Carlo (MCMC) algorithm, which we call Plug-andPlay Diffusion Models (PnP-DM). It incorporates DMs in a principled way and circumvents the approximation required when taking the approach in (2). The proposed algorithm is based on the Split Gibbs Sampler [71] that alternates between two sampling steps that separately involve the likelihood and prior. While the likelihood step can be tackled with traditional sampling techniques, the prior step involves a Bayesian denoising problem that requires careful design. Importantly, we identify a connection between the Bayesian denoising problem and the unconditional image generation problem under a general formulation of DMs presented in [37] (which is referred to as the EDM formulation hereafter). This connection allows us to perform rigorous posterior sampling for denoising using DMs without approximating the generative process and enables the use of a wide range of pretrained DMs through the unified EDM formulation. We present an analysis on the non-asymptotic behavior of PnP-DM by establishing a stationarity guarantee in terms of the average Fisher information. We further demonstrate the strong empirical performance of PnP-DM by investigating three linear and three nonlinear noisy inverse problems, including a black hole interferometric imaging problem involving real data that is both nonlinear and severely ill-posed (see Figure 1). Overall, PnP-DM outperforms existing baseline methods, achieving higher accuracy in posterior estimation.

2 Preliminaries

Split Gibbs Sampler (SGS) is an MCMC approach developed for Bayesian inference [71]. It is also related to the Proximal Sampler [42, 14, 25, 79] and serves as the backbone for the Generative Plug-and-Play (GPnP) [6] and Diffusion Plug-and-Play (DPnP) [78] frameworks in computational imaging. The goal of SGS is to sample the posterior distribution

where f ( x ; y ) := -log p ( y | x ) and g ( x ) := -log p ( x ) are the potential functions of the likelihood and prior distribution, respectively. The dual dependence of (3) on both the likelihood and prior makes it nontrivial to directly sample from it in general. Instead, SGS leverages the composite structure of the posterior distribution by adopting a variable-splitting strategy and considers sampling an alternative distribution

where z ∈ R n is an augmented variable and ρ > 0 is a hyperparameter that controls the strength of the coupling between x and z . We denote the x - and z -marginal distributions of (4) as π X ( x ) := ∫ π ( x , z )d z and π Z ( x ) := ∫ π ( x , z )d x , respectively. As ρ → 0 , π X converges to the target posterior p ( x | y ) in terms of total variation distance [71], so one can obtain approximate samples from the target posterior by sampling (4) instead.

SGS samples (4) via Gibbs sampling. Specifically, SGS starts from an initialization x (0) and, for iteration k = 0 , · · · , K -1 , alternates between

Note that the two conditional distributions separately involve f ( · ; y ) and g ( · ) . The likelihood and prior are decoupled so that these two steps can be designed in a modular way. A similar variablesplitting strategy is also adopted in optimization methods such as the Half-Quadratic Splitting (HQS) method [31] and the Alternating Direction Method of Multipliers (ADMM) [30, 7]. In fact, SGS can be viewed as a sampling analogue of HQS. SGS is a principled approach to posterior sampling if the two sampling steps are rigorously implemented.

Existing works related to SGS Several works have designed algorithms for solving imaging inverse problems based on SGS [53, 19, 6, 27, 78]. The key distinction among these methods lies in their approaches to the prior step. For instance, the works [53, 6, 27] applied Langevin-based updates for sampling π X | Z = z such that the prior information is encoded by either traditional regularizers or off-the-shelf image denoisers. The work [19] tackled the prior step by heuristically customizing a diffusion model (i.e. DDPM [33]) for sampling π X | Z = z . A concurrent work [78] improved the implementation by devising two diffusion processes that rigorously solve the prior step. Our method differs from [78] by connecting the prior step to the EDM formulation [37]. This connection allows us to seamlessly integrate state-of-the-art DMs as expressive image priors for Bayesian inference through a unified interface, eliminating the need for additional customization for each model and leading to better empirical performance. We also note the recent work [43] that adopted the optimizationbased variable-splitting formulation of HQS and utilized general DMs as image priors. We instead considers the SGS formulation from a Bayesian posterior sampling standpoint. Additionally, while SGS-based methods theoretically accommodate general inverse problems, empirical evidence on real-world nonlinear inverse problems remains scarce in the literature. In this work, we demonstrate our method on three nonlinear inverse problems, including a black hole imaging problem. For a more comprehensive review of related works, see Appendix E.

3 Method

A schematic diagram for the proposed method is shown in Figure 2. Our method, dubbed PnP-DM, builds upon the SGS framework with rigorous implementations of the two sampling steps and an annealing schedule for the coupling parameter ρ . We start with our implementations of the first step for solving both linear and nonlinear inverse problems.

<latexi sh

1_b

64="

Bj+TS

L

NgEnD37

qwoK

V

>A

CW

c

ZH

I

F

8

pzM5GQu rY

O

J

m

0

v

R

y

k

/

f

2

d

9U

X

P

<latexi sh

1_b

64="cuHI

XTrqVOy

A

mw

P

95Bj8

>

C

3

LSgM

FE

d

Y

G

K

0

QNzW

7

/o

f

Z

U

n

v

J

D

k

p

2

+

R

<latexi sh

1_b

64="

Bj+TS

L

NgEnD37

qwoK

V

>A

CW

c

ZH

I

F

8

pzM5GQu rY

O

J

m

0

v

R

y

k

/

f

2

d

9U

X

P

<latexi sh

1_b

64="fBIXyK

Lp

0RT

o58

OrMw

>A

C

H

c

Z

g

F

q

k

N

uS

E

2

v

V

z

3PY

Jn+/

Q

W7

d

D

G

j

m

9

U

<latexi sh

1_b

64="

Bj+TS

L

NgEnD37

qwoK

V

>A

CW

c

ZH

I

F

8

pzM5GQu rY

O

J

m

0

v

R

y

k

/

f

2

d

9U

X

P

<latexi sh

1_b

64="59oD

08mYK

JNgwBIF

E

pU

fV

>A

CZ3

c

HLS

M

d

2

R

zW

r

X

y

/

P

G

n

v

k

u

+

q

O

j

T

7

Q

<latexi sh

1_b

64="jc nvTXVG

oS+Z8

H

RL

w

>A

CW

gM

FI

p

52

K7

E

JU

3

d

r

0Q

P

9

DO

Yf

u

/

N

m

y

B

k

<latexi sh

1_b

64="w

O0/D9vX

F

R5

Y

E

qnH

k

g

>A

C

3

c

V

LS

M

d

I

y

G

K

rQNzW

7

P

o

f

Z

U

J

8

T

p

j

2

+

m

B

<latexi sh

1_b

64="jc nvTXVG

oS+Z8

H

RL

w

>A

CW

gM

FI

p

52

K7

E

JU

3

d

r

0Q

P

9

DO

Yf

u

/

N

m

y

B

k

<latexi sh

1_b

64="j fqKXB3w

D

V

pU/QcAm

>

C

H

Z

LTgI

F

o

ORk

M8

0N

uS

5E

2

v

z

PYr

y

W7

d

G

9

<latexi sh

1_b

64="jc nvTXVG

oS+Z8

H

RL

w

>A

CW

gM

FI

p

52

K7

E

JU

3

d

r

0Q

P

9

DO

Yf

u

/

N

m

y

B

k

<latexi sh

1_b

64="59oD

08mYK

JNgwBIF

E

pU

fV

>A

CZ3

c

HLS

M

d

2

R

zW

r

X

y

/

P

G

n

v

k

u

+

q

O

j

T

7

Q

<latexi sh

1_b

64="7qI

X

jHr

y

C

cv pPok

>A

n

Z

NSgM

F

TU

W

3

B

z

V

K

Edw

2

0

J

Y

G5

8

L

RO

+

u

Q

D

9

/

m

f

<latexi sh

1_b

64="BcPp y5

MF

0u

79

qEd

/

g

>A

CXn

ZHLS

I

T8V

r

2

v

kQ3

YW

RKO

m

JG

+

w

z

N

j

o

D

U

f

<latexi sh

1_b

64="7qI

X

jHr

y

C

cv pPok

>A

n

Z

NSgM

F

TU

W

3

B

z

V

K

Edw

2

0

J

Y

G5

8

L

RO

+

u

Q

D

9

/

m

f

<latexi sh

1_b

64="cFYBE

r

JNgMpX

9

A

>

CWn

ZHLS

T8

7

q

U

R

u

w

2

O

Gk

m

/

V

D

z

f

d

3

v

K

5

y

0

Q

P

o

iteration 0

iteration 1

iteration K-1

Arbitrary

initialization

p

(

x

;

ρ

0

)

p

(

x

;

ρ

1

)

p

(

x

)

p

(

x

)

General reverse diffusion under EDM

ρ

0

ρ

0

ρ

1

ρ

1

{

ρ

k

}

Likelihood step:

enforcing

data-consistency

Annealing schedule

General reverse diffusion under EDM

ρ

K

-

1

ρ

K

-

1

Prior step:

denoising with

noise level

ρ

k

start from     at

noise level

ρ

0

t

∗

0

start from     at

noise level

t

∗

1

ρ

1

3.1 Likelihood step: enforcing data consistency

For the likelihood step at iteration k , we sample

Linear forward model and Gaussian noise We first consider a simple yet common case where the forward model A is linear and the noise distribution is zero-mean Gaussian, i.e. A := A ∈ R m × n and n ∼ N ( 0 , Σ ) . In this case, the potential function of the likelihood term is f ( x ; y ) = 1 2 ∥ y -Ax ∥ 2 Σ (up to an additive constant that does not depend on x and y ) where ∥ · ∥ 2 Σ := ⟨· , Σ -1 ·⟩ . It is then straightforward to show that

where Λ := A T Σ -1 A + 1 ρ 2 I and m ( x ) := Λ -1 ( A T Σ -1 y + 1 ρ 2 x ) . The problem of sampling from Gaussian distributions has been systematically studied [72]. We refer readers to Appendix C.1 for a more detailed discussion.

General case For general nonlinear inverse problems, the likelihood step is not sampling from a Gaussian distribution anymore. Nevertheless, since we have access to π Z | X = x in closed form up to a multiplicative factor, we can use Monte Carlo methods based on Langevin dynamics to draw samples from it as long as the likelihood potential is differentiable. Specifically, we first set up the following Langevin SDE that admits π Z | X = x as the stationary distribution

We then initialize the SDE at z 0 = x and run it with Euler discretization. The pseudocode is provided in Appendix C.1.

3.2 Prior step: denoising via the EDM framework

For the prior step at iteration k , we sample

A closer examination of (6) reveals that this prior step is essentially to draw posterior samples for a Gaussian denoising problem, where the 'measurement' is z ( k ) , the noise level is ρ , and the prior distribution is p ( x ) ∝ exp( -g ( x )) .

We tackle this denoising posterior sampling problem within SGS using DMs as image priors. In particular, we leverage the EDM framework [37], which was originally proposed to unify various

formulations of DMs for unconditional image generation. To see the connection of the EDM framework to (6), consider a family of mollified distributions p ( x ; σ ) given by adding i.i.d Gaussian noise of standard deviation σ to the prior distribution p ( x ) , i.e. x + σ ϵ ∼ p ( x ; σ ) . The core idea of the EDM framework is that a variety of state-of-the-art DMs can be unified into the following reverse SDE:

where ¯ w t is an n -dimensional Wiener process running backward in time, σ ( t ) > 0 is a pre-defined noise level schedule with σ (0) = 0 , s ( t ) is a pre-defined scaling schedule, and ˙ σ ( t ) , ˙ s ( t ) are their time derivatives. As shown in [37], the defining property of (7) is that x t /s ( t ) ∼ p ( x ; σ ( t )) for any time t . Therefore, solving this SDE backward in time allows us to travel from any noise level σ ( t ) to the clean image distribution at t = 0 . This means that we can use (7) to solve (6) with arbitrary noise level ρ as long as ρ is within the range of σ ( t ) . Indeed, the distribution of x 0 conditioned on x t is

We highlight that the last line exactly matches (6) when x t = s ( t ) z ( k ) and σ ( t ) = ρ . Therefore, we can naturally design a practical algorithm that samples (6) by following these three steps: (1) find t ∗ such that σ ( t ∗ ) = ρ , (2) initialize at x t ∗ = s ( t ∗ ) z ( k ) , and (3) solve (7) backward from t ∗ to 0 by choosing the discretization time steps and integration scheme. Through this unified interface, any DMs, once converted to the EDM formulation, can be directly turned into a rigorous solver for (6).

Leveraging the connection with EDM, our prior step implementation comes with a large design space that encompasses a variety of existing DMs, such as DDPM (or VP-SDE) [33], VE-SDE [63], and iDDPM [52]. In our experiments, we conduct posterior sampling with all these different models within our framework and all of them provide high-quality samples. The pseudocode of our implementation and more details on the EDM formulation for the prior step is given in Appendix C.2.

3.3 Putting it all together

The pseudocode of PnP-DM in complete form is presented in Algorithm 1. PnP-DM alternates between the two sampling steps with an annealing schedule { ρ k } for the coupling parameter. We find that the annealing schedule on ρ accelerates the mixing time of the Markov chain and prevents the algorithm from getting stuck in bad local minima for solving highly ill-posed inverse problems. This is a common practice in both Langevin-based [40, 34, 65] and SGS-based [6, 78] MCMC algorithms to improve the empirical performance in solving inverse problems.

Our work shares some similarities with PnP-SGS [19] but contains three main key differences. First, as demonstrated in our experiments, we investigate three nonlinear inverse problems, while nonlinear inverse problems are beyond the scope of [19]. Our experiments show that PnP-SGS struggles with challenging nonlinear inverse problems such as Fourier phase retrieval. Second, we adopt the EDM formulation to ensure that the prior step of PnP-DM is a rigorous mapping from the image manifold with the desired noise level to the clean image manifold, aligning with the theory of SGS. In contrast, the prior step of PnP-SGS [19] is heuristic (which is also pointed out by [78]) and not rigorously designed to sample (6). Third, unlike PnP-SGS [19] that uses a constant ρ , we consider an annealing schedule { ρ k } for the coupling parameter, which is important for highly ill-posed inverse problems.

3.4 Theoretical insights

We provide some theoretical insights on the non-asymptotic behavior of PnP-DM. We start with the following definitions. For two probability measures µ and ˜ µ such that µ ≪ ˜ µ , the Kullback-Leibler (KL) divergence and Fisher information (or Fisher divergence) of µ with respect to ˜ µ are defined, respectively, as

˜ ˜ Both divergences are equal to zero if and only if µ = ˜ µ . KL divergence is a common metric for quantifying the difference

ν

X

0

ν

Z

0

ν

X

1

ν

Z

K

-

1

ν

X

K

Non-stationary process

Stationary process

Likelihood step

Prior step

π

X

π

Z

π

X

π

Z

π

X

ν

X

K

-

1

π

X

τ

= 0

τ

=

T

Time from initialization (   )

τ

Prior step with error

of one distribution with respect to another. Fisher information has been used for analyzing the stationarity of sampling algorithms [3, 66].

We analyze PnP-DM via a continuous-time perspective, leveraging the interpolation techniques introduced for Langevin Monte Carlo [68, 3, 66]. We assume that the likelihood step (5) can be implemented exactly and the prior step (6) involves running the reverse diffusion process (7) with an approximated score function s t ≈ ∇ log p t := ∇ log p ( · ; σ ( t )) . Let ν X 0 be the distribution of the initialization x (0) . Let ν Z k and ν X k +1 be the distributions of z ( k ) and x ( k +1) at the k th iteration. Recall that the stationary distributions are π X and π Z . Our analysis is concerned with two continuous-time processes: (1) the non-stationary process from ν X 0 , a non-stationary initialization, to ν X K where (7) is run with the approximated score function s t and (2) the stationary process that alternates between stationary distributions π X and π Z . These two processes are the interpolation PnP-DM in non-stationary and stationary states and define continuous transitions over discrete iterations. A conceptual illustration of the two processes is provided in Figure 3 with the exact formulations in Appendix A. Now we present our main result:

Theorem 3.1. Consider running K iterations of PnP-DM with ρ k ≡ ρ > 0 and a score estimate s t ≈ ∇ log p t := ∇ log p ( · ; σ ( t )) . Let t ∗ > 0 be such that σ ( t ∗ ) = ρ and δ := inf t ∈ [0 ,t ∗ ] v ( t ) where v ( t ) := s ( t ) √ 2 ˙ σ ( t ) σ ( t ) . Define ν τ and π τ as the distributions at time τ of the non-stationary and stationary process, respectively. Then, for over K iterations of PnP-DM, or equivalently over τ ∈ [0 , T ] with T := K ( t ∗ +1) , we have where we assume that the score estimation error ϵ score := ∫ t ∗ +1 1 v ( τ ) 2 E π τ ∥ s τ -∇ log p τ ∥ 2 2 d τ < ∞ .

The proof is provided in Appendix A. This theorem states that the average distance (measured by Fisher information) of the non-stationary process with respect to the stationary process over K iterations of PnP-DM goes to zero at a rate of O (1 /K ) under certain conditions up to the score approximation error. Note that our theory only requires L 2 -accurate score estimate under the measure π τ , which is a relatively weaker condition than the common L ∞ -accurate score estimate assumption in prior analysis of sampling methods involving score estimates [5, 66]. This result resembles the first-order stationarity for Langevin Monte Carlo [3]. Unlike the non-asymptotic analysis in [78], we utilize the average Fisher information instead of the total variation distance, enabling us to obtain an explicit convergence rate. Here δ is the infimum of the diffusion coefficient along the reverse diffusion in (7); see further discussions on the role of δ in Appendix A.3. Our theory shows that the accurate implementations of the two sampling steps lead to a sampler that provably converges to the stationary process that alternates between the two target stationary distributions.

GT prior

Std. deviation (     )

SD

Mean

GT posterior

DPS

Ours

(EDM)

Compressed sensing

0.15

-1.2

1.2

GT image

std. dev.

mean

0.05

<latexi sh

1_b

64="Qj5M

Kg7+FTRc8

u2

9

P

p

>A

CWn

ZHLS

I

d

k

BV

q

UY

X

w

O

m

y

G

z

f

o

3

J/v

DN

r

0

E

<latexi sh

1_b

64="m0DJ

V

pgMOU

9X

k

yc

+I

>A

CW

ZHLT

F

Y7

w3

G8jSz

R

or

u

N

Q

2

q

B

/

f

v

P

E

5n

d

K

<latexi sh

1_b

64="fKDp

7jC

X

RYFk3

Sv

>A

M

H

c

Z

9

2

V

d

L

J

B

zyN

g

QE

U

I

r

GP

0n

m

O

/

8

+u

T

q

w

5

W

o

<latexi sh

1_b

64="o7

QL

PKN

Bc+

w

UH5v

>A

CW3

Z

TgI

F

V

jORk

M8

0

uS

E

p2

fz

X

Yr

Jn

/

y

9

q

m

d

<latexi sh

1_b

64="z8IA

Z3yHrM

cQ

+F

upod

7G9

>

CWX

NSg

U

f

L

O

K

R

2Yj

n

B

P

m

5

J

V

D

v

q

E

/

T

k

w

0

<latexi sh

1_b

64="cw

S

n

/B

0D

8M2

Q

y+o

>A

CW3

ZHL

g

FIYzU

9

v

u

q

K

R

Tj

Jr mf

N

G7

OV

5

E

X

d

k

p

P

<latexi sh

1_b

64="m0DJ

V

pgMOU

9X

k

yc

+I

>A

CW

ZHLT

F

Y7

w3

G8jSz

R

or

u

N

Q

2

q

B

/

f

v

P

E

5n

d

K

<latexi sh

1_b

64="

H/uM38N

jg

GDqPm+VR

y

Y

>A

CW

c

Z

LT

I

F

O

k

K0

S

5Ewp2

B

v

fz

X

r

Jn

7Q9

U

o

d

<latexi sh

1_b

64="

87Ozy

IomJprH

5

E

2Cc0

>A

X3

Z

LSgM

F

T

V

K

W

v

j

kq wY

UB

u

D

f

9

P

/

d

G

Q

R

n

N

+

<latexi sh

1_b

64="m0DJ

V

pgMOU

9X

k

yc

+I

>A

CW

ZHLT

F

Y7

w3

G8jSz

R

or

u

N

Q

2

q

B

/

f

v

P

E

5n

d

K

<latexi sh

1_b

64="RmYI

wjAC

5v

B

Q0g8

oVM

>

E

X

cr

PL

N

FJ3

p

U

d

OySK

f

2

/

nGD

Z

H

+

9

z

7

W

T

u

q

k

<latexi sh

1_b

64="FY3EcdL

oIZz

T0n

WU

>A

CX

H

SgM

8V

r

pK

w

Rk

GN

B

mJ

y5

O

j

Q

7

u

q

9

PD

+

f

/

v

2

<latexi sh

1_b

64="z8IA

Z3yHrM

cQ

+F

upod

7G9

>

CWX

NSg

U

f

L

O

K

R

2Yj

n

B

P

m

5

J

V

D

v

q

E

/

T

k

w

0

<latexi sh

1_b

64="DG+

FKQ

3BNH5k

m

j

Euc

o

>A

CWn

Z

LTgI

OP

dS

M

X

zp

Y

U

/

J

R

7

y

f

r

2

8

w

V

9

0

vq

<latexi sh

1_b

64="m0DJ

V

pgMOU

9X

k

yc

+I

>A

CW

ZHLT

F

Y7

w3

G8jSz

R

or

u

N

Q

2

q

B

/

f

v

P

E

5n

d

K

<latexi sh

1_b

64="7

X502wRBvCrGyQAD3o+T

NZI

>

W

c

H

g

F

L

jKm

u

S

P

M

J

n

9

8

zk

Y

E

O

d

f

U

p

/

q

V

Method

Motion deblur

Super-resolution (4

×

)

PSNR (

↑

)

SSIM (

↑

)

LPIPS (

↓

)

PSNR (

↑

)

SSIM (

↑

)

LPIPS (

↓

)

PSNR (

↑

)

SSIM (

↑

)

LPIPS (

↓

)

PnP-ADMM [13]

26.88

0.7855

0.3472

26.55

0.7655

0.3600

26.61

0.7634

0.3766

DPIR [80]

28.74

0.8348

0.2677

29.97

0.8529

0.2404

28.75

0.8378

0.2577

DDRM [39]

27.05

0.7819

0.2570

-

-

-

29.47

0.8437

0.2322

DPS [17]

28.83

0.8212

0.2330

27.87

0.8035

0.2542

29.45

0.8379

0.2274

PnP-SGS [19]

27.46

0.8356

0.2445

28.98

0.8447

0.2190

28.30

0.8349

0.2160

DPnP [78]

29.24

0.8360

0.2098

30.21

0.8527

0.2010

29.32

0.8407

0.2127

PnP-DM (VP)

29.46

0.8215

0.2202

30.06

0.8336

0.2099

29.40

0.8238

0.2219

PnP-DM (VE)

29.65

0.8399

0.2090

30.38

0.8547

0.1971

29.57

0.8431

0.2108

PnP-DM (iDDPM)

29.60

0.8383

0.2203

30.26

0.8507

0.2103

29.53

0.8404

0.2213

PnP-DM (EDM)

29.66

0.8411

0.2170

30.35

0.8547

0.2062

29.60

0.8435

0.2191

Motion deblur

DPIR

DPS

Ours:

PnP-DM (EDM)

PnP-SGS

Measurement

Ground truth

Ours:

PnP-DM (VE)

4 Experiments

4.1 Validation with ground truth posterior

We first demonstrate the accuracy of PnP-DM for posterior sampling on a simulated compressed sensing problem with a Gaussian prior where the posterior distribution can be expressed in a closed form. The mean and per-pixel standard deviation of the prior are visualized on the bottom left of Figure 4. The linear forward model A ∈ R m × n is a Gaussian matrix ( m = n/ 2 ), i.e. A ij ∼ N (0 , 1) . A test image is randomly generated from the prior (see top left of Figure 4), and the measurement is calculated according to (1) with n ∼ N ( 0 , 0 . 01 2 I ) . We compare our method with the popular DM-based method DPS [17]. We draw 1,000 samples and visualize the empirical mean and per-pixel standard deviation for both algorithms. Compared with the true posterior (second column), we find that the both methods accurately estimate the mean. However, the standard deviation image estimated by DPS significantly deviates from the ground truth. In contrast, our standard deviation image matches the ground truth in terms of both absolute magnitude and spatial distribution. These results highlight the accuracy of our method over DPS by taking a more principled Bayesian approach.

4.2 Benchmark experiments

Dataset and inverse problems We test our proposed algorithm and several baseline methods on 100 images from the validation set of the FFHQ dataset [38] for five inverse problems: (1) Gaussian deblur with kernel size 61 × 61 and standard deviation 3.0, (2) Motion deblur with kernel size 61 × 61 and intensity of 0.5, (3) Super-resolution with 4 × downsampling ratio, (4) the coded diffraction patterns (CDP) reconstruction problem (nonlinear) in [10, 51] (phase retrieval with a phase mask), and (5) the Fourier phase retrieval (nonlinear) with 4 × oversampling. We add i.i.d. Gaussian noise to all the simulated measurements y . In particular, i.e. n ∼ N ( 0 , σ 2 y I ) . For all problems except for Fourier phase retrieval, the noise standard deviation is set as σ y = 0 . 05 . Due to the severe ill-posedness of Fourier phase retrieval, we consider a smaller noise standard deviation σ y = 0 . 01 .

Baselines and comparison protocols We consider four variants of DMs as plug-in priors for our method, namely VP-SDE (VP) [33], VE-SDE (VE) [63], iDDPM [52], and EDM [37]. We compare our method with various baselines, including (1) optimization-based methods: PnP-ADMM [13], DPIR [80]; (2) conditional DMs: DDRM [39], DPS [18]; and (3) SGS-based method: PnP-SGS [19], DPnP [78]. For fair comparison, we use the same pre-trained score function checkpoint for all DM-based methods. Since the pre-trained score function was trained with the DDPM formu-

lation (VP-SDE) [33], we convert it to the EDM formulation by applying the VP preconditioning [37]. We use the Peak Signal-to-Noise Ratio (PSNR), the Structural Similarity Index Measure (SSIM), and the Learned Perceptual Image Patch Similarity (LPIPS) distance for quantitative comparison. For each sampling method, we draw 20 randoms samples, calculate their mean, and report the metrics on the mean image. More experimental details are provided in Appendices B, C, D.

DPS

PnP-SGS

|error|/SD = 3

0.45

0

0.15

std

|

¯

x

-

x

|

0

|error|/SD = 3

0.45

0

0.15

std

|

¯

x

-

x

|

0

3.80 %

96.20 %

11.23 %

88.77 %

Abs. error (            )

|

¯

x

-

x

|

Std. deviation (      )

0.0

3.0

Ours:

PnP-DM (EDM)

0.0

0.15

0.0

0.3

|error|/SD = 3

0.45

0

0.15

std

|

¯

x

-

x

|

0

2.54 %

97.46 %

std

vs.

|

¯

x

-

x

|

std

|

¯

x

-

x

|

/

std

Results: linear problems A quantitative comparison is provided in Table 1. PnP-DM generally outperforms the baseline methods and that the VE and EDM variants consistently outperform the other two variants on these linear problems. Figure 5 contains visual examples for the motion deblur problem (see Appendix F.2 for the other two linear problems). PnP-DM provides high-quality reconstructions that are both sharp and consistent with the ground truth image. We also provide an uncertainty quantification analysis based on pixel-wise statistics in Figure 6. In the left three columns, we visualize the absolute error ( | ¯ x -x | ), standard deviation ( std ), and absolute z-score ( | ¯ x -x | / std ). In the third column, red pixels highlight locations where the ground truth pixel values are outliers of the 3-sigma credible interval (CI) under the estimated posterior uncertainty. The fourth column contains scatter plots of | ¯ x -x | versus std for each pixel of the reconstructions, where red boxes show the percentages of outliers (outside of 3-sigma CI) and gray boxes indicate the percentages within the 3-sigma CI. Similar to the synthetic prior experiment, DPS tends to have larger standard deviation estimations, as shown by the less concentrated distribution of gray points around the origin. Compared with baselines, especially PnP-SGS, our approach captures a higher percentage (97.46%) of ground truth pixels than the baselines (96.20% and 88.77%). If the true posterior were truly Gaussian, 99% of the ground-truth pixels should lie within the 3-sigma CI; however, as the posterior is not Gaussian with a DM-based prior, we do not necessarily expect to reach 99% coverage.

Results: nonlinear problems We provide a quantitative comparison in Table 2. For the CDP reconstruction problem, PnP-DM performs on par with DPS but outperforms other SGS-based methods. We then consider the Fourier phase retrieval (FPR) problem, which is known to be a challenging nonlinear inverse problem. One challenge lies in its invariance to 180 ◦ rotation, so the posterior distribution have two modes, one with upright images and another with 180 ◦ -rotated images, that equally fit the measurement. To increase the chance of getting properly-oriented reconstructions, we run each algorithm with four different random initializations and report the metrics for the best run, following the practice in [18]. We find that PnP-DM significantly outperforms the baselines on this highly ill-posed inverse problem. As shown in Figure 7 (a), our method can provide high-quality reconstructions for both orientations, while the baseline methods fail to capture at least one of the two modes. We further run our method for a test image with 100 different random initialization and collect reconstructions in both orientations that are above 28dB in PSNR (90 out of 100 runs). The percentage of upright and rotated reconstructions are visualized by the pie chart in Figure 7 (b). With a prior on upright face images, our method generate mostly samples with the upright orientation. Nevertheless, it can also find the other mode that has an equal likelihood, demonstrating its ability to capture multi-modal posterior distributions.

4.3 Experiments on black hole imaging

Problem setup We finally validate PnP-DM on a real-world nonlinear imaging inverse problem: black hole imaging (BHI) (see Appendix B for more details). A visual illustration of BHI is provided

<latexi sh

1_b

64="

9RNo qLMH

v

0G

3U

kS

A

>

cjZ

f

2E

V7

X

FuyPQzD

g

T

J

n

p

I

Y

K

+

5

/

O

W

8

r

w

m

C

d

B

<latexi sh

1_b

64="

R

X

5o

/

>

CY

c

VF

S8

E

3

r

up

L

k

J

w

m

KT

I

q

U

g

z

y

Z

9

f

On

Bj

W

0

+

<latexi sh

1_b

64="VHO

N8pL

o+Q2

SjAWZ

P

Y

>

C

c

F

T

E

UJKI

RB

d

G

Xn

q

0

rz wv

u9my

7f

/

k

5

D

g

3

M

<latexi sh

1_b

64="

9RNo qLMH

v

0G

3U

kS

A

>

cjZ

f

2E

V7

X

FuyPQzD

g

T

J

n

p

I

Y

K

+

5

/

O

W

8

r

w

m

C

d

B

<latexi sh

1_b

64="

9RNo qLMH

v

0G

3U

kS

A

>

cjZ

f

2E

V7

X

FuyPQzD

g

T

J

n

p

I

Y

K

+

5

/

O

W

8

r

w

m

C

d

B

<latexi sh

1_b

64="

9RNo qLMH

v

0G

3U

kS

A

>

cjZ

f

2E

V7

X

FuyPQzD

g

T

J

n

p

I

Y

K

+

5

/

O

W

8

r

w

m

C

d

B

<latexi sh

1_b

64="

9RNo qLMH

v

0G

3U

kS

A

>

cjZ

f

2E

V7

X

FuyPQzD

g

T

J

n

p

I

Y

K

+

5

/

O

W

8

r

w

m

C

d

B

<latexi sh

1_b

64="

R

X

5o

/

>

CY

c

VF

S8

E

3

r

up

L

k

J

w

m

KT

I

q

U

g

z

y

Z

9

f

On

Bj

W

0

+

DPS

Ours:

PnP-DM (EDM)

PnP-SGS

Ground truth

81.1%

Rotated

(a)

(b)

Upright

18.9%

Rotated

Upright

Put PSNR and show average psnr Add flipped to the corner in Figure 8 (a). This BHI inverse problem is severely ill-posed. Even with an Earth-sized telescope, only a small fraction of the Fourier frequencies of the target black hole can be measured (region within the red box); in reality, this region is further subsampled with a highly sparse pattern (black lines). Additionally, the atmospheric noise causes nonlinearity of this BHI problem that sometimes results in a multi-modal posterior distribution of the reconstructed image [64]. Here we demonstrate the effectiveness of PnP-DM in capturing a multi-modal posterior distribution. For brevity, we restrict our choice of diffusion models in PnP-DM to EDM and use DPS as the baseline.

Results on simulated data We use the simulated data from [64] where the measurements are generated assuming that the ground-truth black hole image were at the location of the Sagittarius A ∗ black hole. Figure 8 (b) visually compares the results obtained by PnP-DM and DPS. We use the t-SNE method [67] to cluster the generated samples (100 for each method) and identify two modes in the samples generated by PnP-DM and three modes in those generated by DPS. We visualize the mean and three samples for each image mode. A metric for quantifying the degree of data mismatch is labeled on the top right corner of each image. As illustrated by both the mean and sample images, PnP-DM successfully captures the two modes previously identified for this dataset [64]. Note that PnP-DM generates high-fidelity samples from both modes with sharp details of the flux ring, and its samples from 'Mode 1' align well with the ground truth image. In contrast, two out of the three modes sampled by DPS fail to exhibit a meaningful black hole structure and do not correspond with the observed measurements, as indicated by the significantly larger data mismatch values.

Results on real data Finally, we apply PnP-DM to the real M87 black hole data from April 6 th , 2017 [21], with the results shown in Figure 1. By leveraging an expressive DM-based image prior, PnP-DM generates high-quality samples that are both visually plausible and consistent with the ring diameters observed in the official EHT reconstruction. These results highlight the robustness and effectiveness of our method in tackling a highly ill-posed real-world inverse problem.

5 Conclusion

We have introduced PnP-DM, a posterior sampling method for solving imaging inverse problems. The backbone of our method is a split Gibbs sampler that iteratively alternates between two steps that separately involve the likelihood and prior. Crucially, we establish a link between the prior step and a general DM framework known as the EDM formulation. By leveraging this connection, we seamlessly integrate a diverse range of state-of-the-art DMs as priors through a unified interface. Experimental results demonstrate that our method outperforms existing DM-based methods across

<latexi sh

1_b

64="

5MGNr

S

J/w2Y

A

>

Vn cjZ

f

9

E

7

v

RI

H

B

d

T8O

mLz

y

koq

CP

+

p

3

F

u

U

g

W

D

K

Q

<latexi sh

1_b

64="ypQ

0mojgKSM35Id

/+

8

>A

Vn

c

Z

f

9

2E

7JNv

X

R

H

B

r

T

O

G

Lz

k

CP

w

Y

U

q

F

u

W

D

<latexi sh

1_b

64="5Xk/Epw

Mrd+

z

S

JoA

8N

Q

>

gn cjZ

2

fV

u

v

7

O

C

F

m

W

T

G

H

I

Y

K

0

yR

U

L

D

3

B

9

q

P

︸

︸

︷︷

︸

︷︷

︸

Mode 2

Mode 1

9e-3

0.0

DPS

3.24

41.17

788.44

Mode 3

19.79

45.79

15.67

5.58

1087.77

550.14

3.24

3.23

6.19

Mode 2

Mode 1

2.60

2.83

2.92

4.19

3.12

2.60

3.22

2.38

Ground truth

u

v

Fourier spectrum

Zoom-in on low-freq.

(Reconstruction from

fully sampled low-freq.)

(a)

Data-fidelity (

)

↓

(b)

Ours: PnP-DM (EDM)

Mean image

Data mismatch (

)

↓

60

μ

as

60

μ

as

60

μ

as

60

μ

as

Posterior samples

both linear and nonlinear inverse problems, including a nonlinear and severely ill-posed black hole interferometric imaging problem.

Limitations PnP-DM can be further improved in the following two aspects. First, PnP-DM currently requires evaluating the likelihood and prior steps for the entire image at a time. This potentially poses computational challenges in solving large-scale inverse problems (e.g. 3D imaging) or those with expensive likelihood evaluation (e.g. PDE inverse problems). Second, the current theoretical analysis does not consider the approximation error introduced in the likelihood step for general nonlinear inverse problems when running Langevin MCMC for finite iterations. Explicit incorporation of this error would offer further insights into the empirical performance of PnP-DM.

Broader impacts We expect this work to make a positive impact in computational imaging and related application domains. For many imaging problems, there is a need to facilitate image reconstruction with expressive image priors and quantify uncertainty, which could lead to better imaging systems that enables further understanding of the imaging target. Nonetheless, as we are introducing DMs as priors into the imaging process, it is inevitable to inherent the potential bias of these models.

6 Acknowledgments

The authors thank Charles Gammie, Ben Prather, Abhishek Joshi, Vedant Dhruv, and Chi-kwan Chan for providing the black hole simulations. The authors also thank the generous funding from Schmidt Sciences and the Heritage Medical Research Fellowship. Z.W. was supported by an Amazon AI4Science Fellowship. Y.S. was supported by a Computing, Data, and Society Fellowship. B.Z. was supported by a Kortschak Fellowship.

<latexi sh

1_b

64="

MZpK

T

jWY7+NgJz

09

d8O

E

>A

n

c

/

f

UqCkB

w

I

R

H

ou

L

X

3

F

P

G

r

5

m

S

D

2

V

Q

y

v

References

Rizwan Ahmad, Charles A. Bouman, Gregery T. Buzzard, Stanley Chan, Sizhuo Liu, Edward T. Reehorst, and Philip Schniter. Plug-and-play methods for magnetic resonance imaging: Using denoisers for image recovery. IEEE Signal Processing Magazine , 37(1):105-116, 2020.

Brian. D. O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications , 12:313-326, 1982.

Krishna Balasubramanian, Sinho Chewi, Murat A Erdogdu, Adil Salim, and Shunshi Zhang. Towards a theory of non-log-concave sampling:first-order stationarity guarantees for langevin monte carlo. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 2896-2923. PMLR, 02-05 Jul 2022.

Georgios Batzolis, Jan Stanczuk, Carola-Bibiane Schonlieb, and Christian Etmann. Conditional image generation with score-based diffusion models. ArXiv , abs/2111.13606, 2021.

Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications to score-based generative modeling. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Charles A. Bouman and Gregery T. Buzzard. Generative plug and play: Posterior sampling for inverse problems, 2023.

Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning , 3:1-122, 01 2011.

Benjamin Boys, Mark Girolami, Jakiw Pidstrigach, Sebastian Reich, Alan Mosca, and O. Deniz Akyildiz. Tweedie moment projected diffusions for inverse problems, 2023.

Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations , 2019.

Emmanuel Candès, Xiaodong Li, and Mahdi Soltanolkotabi. Phase retrieval from coded diffraction patterns. Applied and Computational Harmonic Analysis , 39, 10 2013.

Gabriel Victorino Cardoso, Yazid Janati, Sylvain Le Corff, and Éric Moulines. Monte carlo guided diffusion for bayesian linear inverse problems. ArXiv , abs/2308.07983, 2023.

Andrew A. Chael, Michael D. Johnson, Katherine L. Bouman, Lindy L. Blackburn, Kazunori Akiyama, and Ramesh Narayan. Interferometric imaging directly with closure phases and closure amplitudes. The Astrophysical Journal , 857(1):23, apr 2018.

Stanley Chan, Xiran Wang, and Omar Elgendy. Plug-and-play admm for image restoration: Fixed point convergence and applications. IEEE Transactions on Computational Imaging , PP, 05 2016.

Yongxin Chen, Sinho Chewi, Adil Salim, and Andre Wibisono. Improved analysis for a proximal algorithm for sampling. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 2984-3014. PMLR, 02-05 Jul 2022.

Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 14347-14356, 2021.

Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11472-11481, June 2022.

Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Marc Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh International Conference on Learning Representations , 2023.

Hyungjin Chung and Jong Chul Ye. Score-based diffusion models for accelerated mri. Medical Image Analysis , page 102479, 2022.

Florentin Coeurdoux, Nicolas Dobigeon, and Pierre Chainais. Plug-and-play split gibbs sampler: embedding deep generative priors in bayesian inference, 2023.

Regev Cohen, Yochai Blau, Daniel Freedman, and Ehud Rivlin. It has potential: Gradient-driven denoisers for convergent solutions to inverse problems. In Neural Information Processing Systems , 2021.

The Event Horizon Telescope Collaboration. First m87 event horizon telescope results. iv. imaging the central supermassive black hole. The Astrophysical Journal Letters , 875(1):L4, apr 2019.

The Event Horizon Telescope Collaboration. First m87 event horizon telescope results. v. physical origin of the asymmetric ring. The Astrophysical Journal Letters , 875(1):L5, apr 2019.

Zehao Dou and Yang Song. Diffusion posterior sampling for linear inverse problem solving: A filtering perspective. In The Twelfth International Conference on Learning Representations , 2024.

Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association , 106:1602-1614, 12 2011.

Jiaojiao Fan, Bo Yuan, and Yongxin Chen. Improved dimension dependence of a proximal algorithm for sampling. In Gergely Neu and Lorenzo Rosasco, editors, Proceedings of Thirty Sixth Conference on Learning Theory , volume 195 of Proceedings of Machine Learning Research , pages 1473-1521. PMLR, 12-15 Jul 2023.

Zhenghan Fang, Sam Buchanan, and Jeremias Sulam. What's in a prior? learned proximal networks for inverse problems. In The Twelfth International Conference on Learning Representations , 2024.

Elhadji C. Faye, Mame Diarra Fall, and Nicolas Dobigeon. Regularization by denoising: Bayesian model and langevin-within-split gibbs sampling, 2024.

Berthy T. Feng, Jamie Smith, Michael Rubinstein, Huiwen Chang, Katherine L. Bouman, and William T. Freeman. Score-based diffusion models as principled priors for inverse imaging. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 10520-10531, October 2023.

James R. Fienup. Phase retrieval algorithms: a comparison. Applied optics , 21 15:2758-69, 1982.

Daniel Gabay and Bertrand Mercier. A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Computers & Mathematics With Applications , 2:17-40, 1976.

Donald Geman and Chengda Yang. Nonlinear image recovery with half-quadratic regularization. IEEE transactions on image processing : a publication of the IEEE Signal Processing Society , 4 7:932-46, 1995.

Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as plug-andplay priors. ArXiv , abs/2206.09012, 2022.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 6840-6851. Curran Associates, Inc., 2020.

Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. Robust compressed sensing mri with deep generative priors. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 14938-14954. Curran Associates, Inc., 2021.

Zahra Kadkhodaie and Eero P Simoncelli. Stochastic solutions for linear inverse problems using the prior implicit in a denoiser. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Ulugbek Kamilov, Charles Bouman, Gregery Buzzard, and Brendt Wohlberg. Plug-and-play methods for integrating physical and learned models in computational imaging: Theory, algorithms, and applications. IEEE Signal Processing Magazine , 40:85-97, 01 2023.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4396-4405, 2018.

Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In Advances in Neural Information Processing Systems , 2022.

Bahjat Kawar, Gregory Vaksman, and Michael Elad. SNIPS: Solving noisy inverse problems stochastically. Advances in Neural Information Processing Systems , 34:21757-21769, 2021.

Rémi Laumont, Valentin De Bortoli, Andrés Almansa, Julie Delon, Alain Durmus, and Marcelo Pereyra. Bayesian imaging using plug & play priors: When langevin meets tweedie. SIAM Journal on Imaging Sciences , 15(2):701-737, 2022.

Yin Tat Lee, Ruoqi Shen, and Kevin Tian. Structured logconcave sampling with a restricted gaussian oracle. In Proceedings of Thirty Fourth Conference on Learning Theory , volume 134 of Proceedings of Machine Learning Research , pages 2993-3050. PMLR, 15-19 Aug 2021.

Xiang Li, Soo Min Kwon, Ismail R. Alkhouri, Saiprasad Ravishankar, and Qing Qu. Decoupled data consistency with diffusion purification for image restoration, 2024.

Jiaming Liu, Rushil Anirudh, Jayaraman J. Thiagarajan, Stewart He, K Aditya Mohan, Ulugbek S. Kamilov, and Hyojin Kim. Dolce: A model-based probabilistic diffusion framework for limited-angle ct reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 10498-10508, October 2023.

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV) , December 2015.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019.

Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat. A variational perspective on solving inverse problems with diffusion models. In The Twelfth International Conference on Learning Representations , 2024.

Ségolène Martin, Anne Gagneux, Paul Hagemann, and Gabriele Steidl. Pnp-flow: Plug-and-play image restoration with flow matching, 2024.

Tim Meinhardt, Michael Möller, Caner Hazirbas, and Daniel Cremers. Learning proximal operators: Using denoising networks for regularizing inverse imaging problems. 2017 IEEE International Conference on Computer Vision (ICCV) , pages 1799-1808, 2017.

Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations , 2021.

Christopher A. Metzler, Philip Schniter, Ashok Veeraraghavan, and Richard Baraniuk. prdeep: Robust phase retrieval with a flexible deep network. In International Conference on Machine Learning , 2018.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 8162-8171. PMLR, 18-24 Jul 2021.

Marcelo Pereyra, Luis A. Vargas-Mieles, and Konstantinos C. Zygalakis. The split gibbs sampler revisited: Improvements to its algorithmic structure and augmented target distribution. SIAM Journal on Imaging Sciences , 16(4):2040-2071, 2023.

B. Remy, F. Lanusse, N. Jeffrey, J. Liu, Jean-Luc Starck, K. Osato, and T. Schrabback. Probabilistic mass-mapping with neural score estimation. Astronomy & Astrophysics , 672, 12 2022.

Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alex Dimakis, and Sanjay Shakkottai. Solving linear inverse problems provably via posterior sampling with latent diffusion models. In Thirtyseventh Conference on Neural Information Processing Systems , 2023.

Ernest K. Ryu, Jialin Liu, Sicheng Wang, Xiaohan Chen, Zhangyang Wang, and Wotao Yin. Plug-and-play methods provably converge with properly trained denoisers. In International Conference on Machine Learning , 2019.

Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence , PP:1-14, 09 2022.

Bowen Song, Soo Min Kwon, Zecheng Zhang, Xinyu Hu, Qing Qu, and Liyue Shen. Solving inverse problems with latent diffusion models via hard data consistency. In The Twelfth International Conference on Learning Representations , 2024.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations , 2021.

Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guided diffusion models for inverse problems. In International Conference on Learning Representations , 2023.

Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming-Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. Loss-guided diffusion models for plug-and-play controllable generation. In Proceedings of the 40th International Conference on Machine Learning , ICML'23. JMLR.org, 2023.

Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with score-based generative models. In International Conference on Learning Representations , 2022.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations , 2021.

He Sun and Katherine L. Bouman. Deep probabilistic imaging: Uncertainty quantification and multi-modal solution characterization for computational imaging. In AAAI Conference on Artificial Intelligence (AAAI) , 2021.

Yu Sun, Brendt Wohlberg, and Ulugbek Kamilov. An online plug-and-play algorithm for regularized image reconstruction. IEEE Transactions on Computational Imaging , PP:1-1, 01 2019.

Yu Sun, Zihui Wu, Yifan Chen, Berthy T. Feng, and Katherine L. Bouman. Provable probabilistic imaging using score-based generative priors. IEEE Transactions on Computational Imaging , 10:1290-1305, 2024.

Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research , 9(86):2579-2605, 2008.

Santosh S. Vempala and Andre Wibisono. Rapid convergence of the unadjusted langevin algorithm: Isoperimetry suffices. In Neural Information Processing Systems , 2019.

Singanallur V. Venkatakrishnan, Charles A. Bouman, and Brendt Wohlberg. Plug-and-play priors for model based reconstruction. In 2013 IEEE Global Conference on Signal and Information Processing , pages 945-948, 2013.

Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation , 23:1661-1674, 2011.

Maxime Vono, Nicolas Dobigeon, and Pierre Chainais. Split-and-augmented gibbs sampler-application to large-scale inference problems. IEEE Transactions on Signal Processing , 67(6):1648-1661, 2019.

Maxime Vono, Nicolas Dobigeon, and Pierre Chainais. High-dimensional gaussian sampling: A review and a unifying approach based on a stochastic proximal point algorithm. SIAM Review , 64(1):3-56, 2022.

Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. The Eleventh International Conference on Learning Representations , 2023.

Luhuan Wu, Brian L Trippe, Christian A. Naesseth, David M Blei, and John P Cunningham. Practical and asymptotically exact conditional sampling in diffusion models. arXiv preprint arXiv:2306.17775 , 2023.

Zihui Wu, Yu Sun, Jiaming Liu, and Ulugbek Kamilov. Online regularization by denoising with applications to phase retrieval. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) , pages 3887-3895, 2019.

Zihui Wu, Yu Sun, Alex Matlock, Jiaming Liu, Lei Tian, and Ulugbek S. Kamilov. Simba: Scalable inversion in optical tomography using deep denoising priors. IEEE Journal of Selected Topics in Signal Processing , 14(6):1163-1175, 2020.

Bin Xia, Yulun Zhang, Shiyin Wang, Yitong Wang, Xinglong Wu, Yapeng Tian, Wenming Yang, and Luc Van Gool. Diffir: Efficient diffusion model for image restoration. ICCV , 2023.

Xingyu Xu and Yuejie Chi. Provably robust score-based diffusion posterior sampling for plug-and-play image reconstruction, 2024.

Bo Yuan, Jiaojiao Fan, Jiaming Liang, Andre Wibisono, and Yongxin Chen. On a class of gibbs sampling over networks. In Gergely Neu and Lorenzo Rosasco, editors, Proceedings of Thirty Sixth Conference on Learning Theory , volume 195 of Proceedings of Machine Learning Research , pages 5754-5780. PMLR, 12-15 Jul 2023.

Kai Zhang, Yawei Li, Wangmeng Zuo, Lei Zhang, Luc Gool, and Radu Timofte. Plug-and-play image restoration with deep denoiser prior. IEEE Transactions on Pattern Analysis and Machine Intelligence , PP:1-1, 06 2021.

Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE Transactions on Image Processing , 26(7):31423155, 2017.

Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. Learning deep cnn denoiser prior for image restoration. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2808-2817, 2017.

Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In The Eleventh International Conference on Learning Representations , 2023.

Yuanzhi Zhu, Kai Zhang, Jingyun Liang, Jiezhang Cao, Bihan Wen, Radu Timofte, and Luc Van Gool. Denoising diffusion models for plug-and-play image restoration. In IEEE Conference on Computer Vision and Pattern Recognition Workshops (NTIRE) , 2023.

A Theory

A.1 Interpolation of PnP-DM

In this section, we formally introduce the interpolation of PnP-DM. We consider the case where the coupling strength ρ is constant, i.e. ρ k ≡ ρ and make the following assumption.

Assumption A.1. There exists a unique t ∗ such that σ ( t ∗ ) = ρ .

This assumption is satisfied for common diffusion models. Popular choices of the noise level schedule include σ ( t ) = t or σ ( t ) = √ t , which are monotonically increasing functions of t . We first present two propositions showing that the two steps in SGS can be implemented by running two SDEs.

Proposition A.2 (Brownian bridge for the likelihood step) . For iteration k with iterate x ( k ) , the likelihood step of SGS is equivalent to solving the following SDE from t = 0 to t = 1 :

Proof. This proposition is due to the Brownian bridge construction presented in Lemma 4 of [79]. This SDE satisfies that p ( x 1 | x 0 ) ∝ exp ( -f ( x 1 ; y ) -1 2 ρ 2 ∥ x 0 -x 1 ∥ 2 2 ) . Therefore, solving (9) from t = 0 to t = 1 is equivalent to taking a likelihood step.

Proposition A.3 (EDM reverse diffusion for the prior step) . For iteration k with iterate z ( k ) , the prior step of SGS is equivalent to solving the following SDE from t = t ∗ to t = 0 :

d x t = [ u ( t ) x t -v ( t ) 2 ∇ log p t ( x t ) ] d t + v ( t )d ¯ w t (10) where x t ∗ = s ( t ∗ ) z ( k ) , u ( t ) := ˙ s ( t ) s ( t ) , v ( t ) := s ( t ) √ 2˙ σ ( t ) σ ( t ) , and p t is the distribution of s ( t ) x + s ( t ) σ ( t ) ϵ with x following the prior distribution p ( x ) ∝ exp( -g ( x )) and ϵ ∼ N ( 0 , I ) .

Proof. First note that (10) is exactly (7) written in terms of u ( t ) and v ( t ) . We know that the (10) is the reverse SDE of the following SDE

where x 0 ∼ p ( x ) and p t is the marginal distribution of x t . As we showed in the main text, it holds for (11) that

As (10) is the time-reversed process of (11), they share the same path distribution and thus the same conditional distribution p ( x 0 | x t ) . So, if we set x t ∗ = s ( t ∗ ) z ( k ) , we have that

which is the desired conditional distribution of the prior step. Therefore, solving (10) from t = t ∗ to t = 0 is equivalent to taking a prior step.

Due to Proposition A.2 and Proposition A.3, the SDEs (9) and (10) implement the two desired conditional distributions in SGS. In PnP-DM, the prior step involves a network that approximates the score function of the prior distribution, i.e. s t ≈ ∇ log p t , so the continuous-time process for the actual update is d x t = [ u ( t ) x t -v ( t ) 2 s t ( x t ) ] d t + v ( t )d ¯ w t . (12) We can then interpolate PnP-DM by considering a dynamic that alternates between running (9) and (12).

Since each likelihood step takes 1 unit of time and each prior step takes t ∗ unit of time, the total time of the interpolating process for K iterations of PnP-DM is T := K ( t ∗ +1) . We use τ to denote the time that has elapsed from initializing PnP-DM with x (0) . We define { ν τ } and { π τ } as the distributions at time τ of the non-stationary process initialized at x (0) ∼ ν X 0 (Figure 3 top) and the stationary process initialized at x (0) ∼ π X (Figure 3 bottom), respectively. Therefore, we have

A.2 Proof of Theorem 3.1

Before proving our main result, we present a key lemma for our analysis, which quantifies the time-derivative of the KL divergence in terms of the Fisher information along a pair of general diffusion processes.

Lemma A.4. Given the following pair of diffusion processes

d ˜ x t = ˜ b ( ˜ x t , t )d t + c ( t )d w t (14) where b ( · , · ) : R n × R → R n , ˜ b ( · , · ) : R n × R → R n , and c ( · ) : R → R . Let µ t be the distribution of x t initialized with x 0 ∼ µ 0 for (13) , and let ˜ µ t be the distribution of ˜ x t initialized with ˜ x 0 ∼ ˜ µ 0 for (14) . Then we have

Proof of Lemma A.4. Writing b ( · , t ) as b t and ˜ b ( · , t ) as ˜ b t , by the Fokker-Planck equations of (13) and (14), we have that

Now we are ready to prove Theorem 3.1.

Proof of Theorem 3.1. We first consider the likelihood steps over K iterations of PnP-DM. Applying Lemma 2 of [79] to the likelihood steps (9) of the non-stationary and stationary processes, we have that

for τ ∈ [ k ( t ∗ +1) , k ( t ∗ +1) + 1] with k = 0 , ..., K -1 . Integrating both sides over τ ∈ [ k ( t ∗ + 1) , k ( t ∗ +1) + 1] , we get

for k = 0 , ..., K -1 .

Then, applying Lemma A.4 to the prior steps (12) with

we have that for τ ∈ [ k ( t ∗ + 1) + 1 , ( k + 1)( t ∗ + 1)] with k = 0 , ..., K -1 . Integrating both sides over τ ∈ [ k ( t ∗ +1) + 1 , ( k +1)( t ∗ +1)] , we get

where

Finally, combining (16) and (17) for k = 0 , ..., K -1 , we obtain

The proof is concluded by dividing T = K ( t +1)

A.3 Discussion

To facilitate the discussion, we first present the following proposition.

Proposition A.5. Define a weighting function λ ( τ ) over τ ∈ [0 , T ] such that for k = 0 , ..., K -1 ,

∗ on both sides.

Then, under the same settings of Theorem 3.1, we have

Proof. With the definition of λ ( τ ) , we can apply Lemma 2 of [79] to the likelihood steps and obtain where ϵ score := ∫ t ∗ +1 1 v ( τ ) 2 E π τ ∥ s τ -∇ log p τ ∥ 2 2 d τ .

for k = 0 , ..., K -1 . Similarly, we can apply Lemma A.4 to the prior steps and obtain where

Together, for τ ∈ [0 , T ] . We can then get (18) by combining (19) and (20) for k = 0 , ..., K -1 and dividing by T := K ( t ∗ +1) .

Unlike Theorem 3.1, this proposition calculates the weighted average of the Fisher information along the two processes with the weighting function λ ( τ ) . The bound in Theorem 3.1 on the unweighted average of Fisher information can be obtained by further lower-bounding the left hand side of (18) using the infimum of λ ( τ ) over τ ∈ [0 , T ] . Given this observation, we can see the role of δ in Theorem 3.1. With a strictly positive δ , the weighting function λ ( τ ) is always strictly positive, so the (unweighted) average Fisher information must converge to 0. This is precisely the case for the VP- and VE-SDE [63]. On the other hand, if δ = 0 , the Fisher information FI ( π τ || ν τ ) may be increasingly large as λ ( τ ) gets closer to 0. For iDDPM and EDM, this could happen near t = 0 in the reverse diffusion at v (0) = 0 for these diffusion processes. Nevertheless, we can instead consider a slightly adjusted diffusion coefficient ˜ v ( t ) := v ( t ) + ϵ with ϵ > 0 . Using the relation between scores and diffusions div( p ∇ log p ) = ∆ p , we get the following reverse SDE which has the same law as (7) at each t :

In this case, ˜ v ( t ) = s ( t ) √ 2 ˙ σ ( t ) σ ( t ) + ϵ is strictly positive, so the convergence on the unweighted average Fisher information is also guaranteed.

B Inverse problem setup

Data usage We list the data we have used for our experiments:

For the synthetic prior experiment, we took images from the CelebA dataset [45], turned them into grayscale, rescaled them to [ -1 , 1] , and resized them to 32 × 32 pixels for efficient computation. We then found the empirical mean and covariance of the images to construct the Gaussian image prior. The test image was randomly drawn from this Gaussian prior.

For the benchmark experiments, we used the first 100 images (index 00000 to 00099) in the FFHQ dataset [38]. For all linear inverse problems, the test images were in RGB and normalized to range [ -1 , 1] . For all nonlinear problems, the test images were in grayscale and normalized to range [0 , 1] .

For the black hole experiments, we used the simulated data used in [64] and the publicly available EHT 2017 data 1 that was used to produce the first image of the M87 black hole.

Gaussian and motion deblur The forward model is defined as

where B ∈ R n × n is a circulant matrix that effectively implements a convolution with kernel k under the circular boundary condition. For the Gaussian deblurring problem, we fixed the kernel k as a Gaussian kernel with standard deviation 3 . 0 and size 61 × 61 . For the motion deblurring problem, we randomly generated the kernel k for each test image using the code 2 with intensity of 0 . 5 and size 61 × 61 . For fair comparison, the blur kernel for each test image was set the same for all compared methods.

Super-resolution The forward model is defined as

where P f ∈ R n f × n is a matrix that implements a block averaging filter to downscale the images by a factor of f . Specifically, we set f = 4 and used the SVD implementation from the code 3 of [39].

Coded diffraction patterns (CDP) CDP is a measurement model originally proposed in [10]. The target x is illuminated by a coherent source and modulated by a phase mask D . The light field then undergoes the far-field Fraunhofer diffraction and is measured by a standard camera. Mathematically, the forward model of CDP is defined as

where F denotes the 2D Fourier transform. We followed [75] to set D as a diagonal matrix with entries drawn randomly from the complex unit circle.

Fourier phase retrieval We adopted a similar setting as [18]. In particular, the forward model is defined as

where P denotes the oversampling matrix that effectively pads x in 2D matrix form with zeros. We considered a 4 × oversampling ratio for grayscale images of size 256 × 256 , so Px has a size of 512 × 512 .

Black hole imaging We adopted the same BHI setup as in [64, 66]. The relationship between the black hole image and each interferometric measurement, or so-called visibility , is given by

where a and b denote a pair of telescopes, t represents the time of measurement acquisition, i is the imaginary unit, and F t a,b ( x ) is the Fourier component of the image x corresponding to the baseline between telescopes a and b at time t . In practice, there are three main sources of noise in (21): gain error g a and g b at the telescopes, phase error ϕ t a and ϕ t b , and baseline-based additive white Gaussian noise η a,b . The gain and phase errors stem from atmospheric turbulence and instrument miscalibration and often cannot be ignored. To correct for these two errors, multiple noisy visibilities can be combined into data products that are invariant to these errors, which are called closure phase and log closure amplitude measurements [12]

where ∠ computes the angle of a complex number. Given a total of M telescopes, there are in total ( M -1)( M -2) 2 closure phase and M ( M -3) 2 log closure amplitude measurements at time t , after eliminating repetitive measurements. In our experiments, we used a 9-telescope array ( M = 9 ) from

the Event Horizon Telescope (EHT) and constructed the data likelihood term based on these nonlinear closure quantities. Additionally, because the closure quantities do not constrain the total flux (i.e. summation of the pixel values) of the underlying black hole image, we added a constraint on the total flux in the likelihood term. The overall potential function of the likelihood is given by

In this equation, y flux is the total flux of the underlying black hole, which can be accurately measured. We use y := ( y cph , y logcamp , y flux ) to denote all the measurements and c , d as the indices for the closure phase and log closure amplitude measurements. Parameters σ cph , σ logcamp were given by the telescope system and σ flux was set to √ 2 in our experiments to constrain the total flux. The data mismatch metric reported in Figure 8 is defined as the sum of the reduced χ 2 values for the closure phase and log closure amplitude measurements, which are calculated using the ehtim.obsdata.Obsdata.chisq function of the ehtim package 4 . Both χ 2 values should ideally be around 1 for data with high signal-to-noise ratio (SNR). Therefore, a data mismatch value around 2 to 3 is considered as fitting the measurements well.

C Technical details of PnP-DM

C.1 Likelihood step

Linear forward model and Gaussian noise As we showed in the main paper, in case of linear forward models and Gaussian noise, the likelihood step is

where Λ := A T Σ -1 A + 1 ρ 2 I and m ( x ) := Λ -1 ( A T Σ -1 y + 1 ρ 2 x ) . The bottleneck here is that both the mean and the covariance involve the matrix inverse Λ -1 , which can be prohibitive to compute directly for high-dimensional problems. Nevertheless, the computational cost can be significantly alleviated when the noise is i.i.d. Gaussian, i.e. Σ = σ 2 y I , and A can be efficiently decomposed. For example, if one can efficiently calculate the SVD of the forward model A , i.e. A = USV T , one can find the Cholesky decomposition of Λ -1 as

Since S is a diagonal matrix, the second term can be calculated with only O ( n ) complexity. Then, leveraging the property of multivariate Gaussian distribution, we can sample η ∼ N ( 0 , I ) and calculate z = m ( x )+ Lη as a sample that exactly follows the target Gaussian distribution N ( m ( x ) , Λ -1 ) . An analogous derivation with Fourier transform can be done when A is a circulant convolution matrix.

Nonlinear forward model We provide the pseudocode of the LMC algorithm for sampling the likelihood step with general differentiable forward models in Algorithm 2.

Algorithm 2 Langevin Monte Carlo for the likelihood step under general A Input: state x , coupling strength ρ > 0 , likelihood potential f ( · ; y ) with measurements y Hyperparameter: step size γ > 0 , number of iterations J > 0 1: u 0 ← x 2: for j = 0 , · · · , J -1 do 3: u j +1 ← u j -γ ∇ f ( u j ; y ) -γ ρ 2 ( u j -x ) + √ 2 γ ϵ j where ϵ j ∼ N ( 0 , I ) 4: end for 5: return u J

Table 3 summarizes the hyperparameters we used for solving the nonlinear inverse problems considered in this work.

C.2 Prior step

The EDM framework We formally introduce the EDM formulation [37] using our notations. The forward diffusion process is defined as the following linear Itô SDE

where u ( t ) : R → R , v ( t ) : R → R are the drift and diffusion coefficients. The generative process is the time-reversed version of (23). According to [2], it is another Itô SDE of the form where p t ( x t ) is the marginal distribution of x t . There also exists a reverse probability flow ODE

which shares the same marginal distributions as (24). Based on (23), we have

where s ( t ) := exp ( ∫ t 0 u ( ξ )d ξ ) and σ ( t ) := √ ∫ t 0 v ( ξ ) 2 s ( ξ ) 2 d ξ . We also have x t /s ( t ) ∼ p ( x ; σ ( t )) where p ( x ; σ ( t )) is the distribution obtained by adding i.i.d. Gaussian noise of standard deviation σ ( t )) to the prior data. The idea of the EDM formulation is to write the reverse diffuison directly in terms of the scaling and noise level of x t with respect to x 0 , which are more important than the drift and diffusion coefficients. With the relations between u ( t ) , v ( t ) , p t and s ( t ) , σ ( t ) , p ( · ; σ ( t )) , we can rewrite (24) and (25) as and

Note that (26) is precisely the SDE (7) we considered for the prior step. Finally, due to the Tweedie's formula [24], we can approximate ∇ x t log p ( · ; σ ( t )) by a denoiser [ D θ ( · ; σ ( t )) - · ] /σ ( t ) 2 trained to minimize the ℓ 2 error of a denoising objective. Substituting the score function by the approximation with the denoiser and using the chain rule, we can further rewrite (26) and (27) as and

Pseudocode We provide the pseudocode for our prior step in Algorithm 3. Note that the update rule is precisely the Euler discretization of (28) and (29). The discretization time steps { t i } N i =0 , scaling schedule s ( · ) , and noise schedule σ ( · ) , are kept the same as in Table 1 of [37]. For all experiments, we set the total number of time steps to 100, i.e. N = 100 . We note that this does not imply that each prior step has a number of function evaluations (NFE) equal to 100. Since ρ is to a small value as the algorithm runs, the number of steps in later iterations of the algorithm are much fewer than 100. The prior step is similar to the image synthesis process in SDEdit [50] that starts from the middle of the reverse diffusion process. We used the pf-ODE solver for the CDP problem and the SDE solver for all other problems. Part of our code implementation is based on the repository 5 .

̸

Algorithm 3 EDM for the prior step (Bayesian denoising with noise level ρ ) Input: noisy image z ∈ R n , assumed noise level ρ > 0 , pretrained model D θ ( · ; · ) that approximates ∇ log p ( x ; σ ) with ( D θ ( x ; σ ) -x ) /σ 2 Hyperparameter: discretization time steps { t i } N i =0 (monotonically decreasing to t N = 0 ), scaling schedule s ( · ) , noise schedule σ ( · ) , solver ( SDE or pf-ODE ) 1: i ∗ ← smallest i such that σ ( t i ) ≤ ρ ▷ Find the starting point of the reverse diffusion 2: v i ∗ ← s ( t i ∗ ) z ▷ Initialize at time t i ∗ 3: for i = i ∗ , · · · , N -1 do 4: λ ← 2 if solver is SDE else 1 5: d i ← ( λ ˙ σ ( t i ) σ ( t i ) + ˙ s ( t i ) s ( t i ) ) v i -λ ˙ σ ( t i ) s ( t i ) σ ( t i ) D θ ( v i s ( t i ) ; σ ( t i ) ) 6: v i +1 ← v i +( t i +1 -t i ) d i ▷ Drift 7: if i = N -1 and solver is SDE then 8: v i +1 ← v i +1 + s ( t ) √ 2 ˙ σ ( t ) σ ( t )( t i -t i +1 ) ϵ i where ϵ i ∼ N ( 0 , I ) ▷ Diffusion 9: end if 10: end for 11: return v N

Model checkpoint For experiments with FFHQ color images, we used the pretrained checkpoint from [16] available at the repository 6 . For experiments with synthetic data, FFHQ grayscale images, and black hole images, we trained our own models using the same repository. The model network is based on the U-Net architecture in [52] with BigGAN [9] residual blocks, multi-resolution attention, and multi-head attention with fixed channels per head. See the appendix of [16] for architecture details. Specifically, we changed the input and output channels to 1 and 2, respectively, to accommodate grayscale inputs, and reduced the number of down-pooling and up-pooling levels in the U-Net for smaller images. We trained all models until convergence using an exponential moving average (EMA) rate of 0.9999, 32-bit precision, and the AdamW optimizer [46]. Here is a list of training data we used for each model:

For the Gaussian prior model, we randomly generated images from the constructed Gaussian prior distribution.

For the FFHQ grayscale model, we used the images with index 01000 to 69999 in the FFHQ dataset.

For the black hole model, we used 3068 simulated black hole images from the GRMHD simulation, which stands for general relativistic magnetohydrodynamic simulation [22]. See Figure 9 for some example training images. We applied data augmentation with random flipping and resizing, so that the flux spin rotation and the ring diameter vary from sample to sample.

Preconditioning Since the checkpoints we used are all trained based on the DDPM (or VP-SDE) formulation [33], we converted them to the denoiser D θ under the EDM formulation via the VP preconditioning [37]. Specifically, if we denote the pretrained model as F θ ( · ; · ) , the model we used for Algorithm 3 is

is the inverse of the VP-SDE noise schedule defined as σ VP ( t ) := √ e 1 2 β d t 2 + β min t -1 with β d = 19 . 9 and β min = 0 . 1 . This adaption allows us to make a fair comparison with other DM-based methods using the same pretrained models. One can also incorporate DMs trained with other formulations into PnP-DM by properly setting the preconditioning parameters.

Connection to DDS-DDPM in [78] A concurrent work [78] introduced a rigorous implementation of the prior step, called DDS-DDPM , by converting the DDPM [33] (or VP-SDE [63]) sampler into a reverse diffusion based on the VE-SDE [63]. The diffusion process after the conversion can be used to solve (6) rigorously by properly choosing the starting point. In fact, our formulation admits DDS-DDPM as a special case with the VP preconditioning and reverse diffusion based on the VE-SDE. Here we explicitly show this connection. For the VE-SDE, we have s VE ( t ) = 1 , σ VE ( t ) = √ t , u VE ( t ) = 0 , and v VE ( t ) = 1 . So (28) becomes

Applying the VP preconditioning (30) to (31), we obtain

We can then rescale the time range from [0 , 1] to [0 , 1000] , discretize (32) backward in time over the time steps { τ t } from [78], and apply the exponential integrator [83] to the drift term, resulting in the following update rule:

Based on the definition τ t := ¯ α -1 t -1 = σ VP ( t ) 2 in DDS-DDPM , we get

̂ ̂ ̂ This is exactly the update rule of DDS-DDPM with F θ ( · ; t ) denoting the noise estimate ̂ ϵ t ( · ) of DDPM. One can also verify that the initialization in DDS-DDPM is equivalent to ours by checking that ¯ α t ≥ 1 η 2 +1 is equivalent to τ t = σ VP ( t ) 2 ≤ η 2 where η ≡ ρ is the assumed noise level in (6). As one can see, DDS-DDPM is equivalent to our prior step by choosing the VP-preconditioning, VE reverse diffusion, and a particular integration scheme. In fact, our prior step allows for more general definitions of diffusion processes and includes both the ODE and SDE solvers.

C.3 Others

Annealing schedule for ρ In this work, we considered an exponential annealing schedule for the coupling strength ρ . We note that the schedule can be more general than exponential decay and we leave the investigation of other decay schedules in future work. Specifically, we specified a starting level ρ 0 , decay rate α , and a minimum value ρ min. Then we set

for k = 0 , · · · , K -1 . Table 4 summarizes the annealing hyperparameters that we used for all the inverse problems considered in this work.

Initialization For the linear inverse problems, we used the zero initialization, i.e. x (0) = 0 ∈ R n . For the CDP and Fourier phase retrieval problems, we used the Gaussian initialization, i.e. x (0) ∼ N ( 0 , I ) . For black hole imaging experiments, we used the uniform random initialization between 0 and 1 for each pixel. We found that PnP-DM, as an MCMC algorithm, is insensitive to the initialization. Except for the black hole experiments where we found the negative values would cause problems, any reasonable initialization would lead to comparable results. This observation corroborates our convergence result, which holds for any initialization ν X 0 .

Number of iterations We ran 500 iterations for the synthetic prior experiments, 200 iterations for the black hole experiments, and 100 for all other experiments. The numbers were chosen so that the algorithm was fully converged.

Sample collection To collect multiple samples using our method, there are two main approaches: (1) Run a single Markov chain and collect samples after a certain number of iterations, known as the burn-in period, to ensure the chain has converged. (2) Run several independent Markov chains and collect one sample from each chain after convergence. The first approach is more efficient, but the collected samples are not entirely independent and thus may have a small effective sample size. The second approach ensures all samples are fully independent but takes longer to run. In our experiments, we used the first approach for all tests involving 256 × 256 images to enhance efficiency. Specifically, we set the burn-in period to 40 iterations and collected 20 random samples from the remaining 60 iterations (one every 3 iterations). For other experiments, due to the smaller image sizes, we employed the second approach to obtain fully independent samples.

Compute All experiments were performed on NVIDIA RTX A6000 and A100 GPUs. The runtime per image depends on several factors, such as the choice of GPU, the total number of iterations and the coupling strength schedule { ρ k } (as it takes more network evaluations for larger ρ for our EDM-based denoiser). In our actual experiments, we ran each image for at least 100 iterations to ensure convergence, which took around 1 minute for a single Markov chain. Here we present a comparison of computational efficiency with the major baselines on a linear super-resolution and a nonlinear coded diffraction patterns problem in Table 5. The clock time in seconds and number of function evaluations (NFE) are calculated for each method to measure its computational efficiency. All hyperparameters are kept the same for each method as those used for Table 1 and Table 2 in the manuscript. As expected, DM-based approaches (DDRM & DPS) generally yield shorter runtimes due to their lower NFEs. Nevertheless, our PnP-DM method significantly outperforms these methods while achieving comparable runtimes with DPS ( ≈ 1 . 5 × ), despite its larger NFEs ( ≈ 3 × ). This is primarily due to two factors: 1) PnP-DM avoids running the full diffusion process by adapting the starting noise level to ρ k at each iteration, and 2) the runtime is further reduced by using an annealing schedule of ρ k . We also note that the runtime reported for DDRM and DPS below is the time it takes to generate one sample. For the linear inverse problem experiments, where we generated 20 samples for each sampling method, PnP-DM was faster than DPS because we took 20 samples that PnP-DM generated along one Markov chain of batch size 1 (hence same runtime as below, around 50 seconds) but DPS requires running a diffusion process with batch size 20, which was significantly slower (around 330 seconds).

D Implementation details of baseline methods

PnP-ADMM We set the ADMM penalty parameter as 2 and ran for 500 iterations to ensure convergence. We used the pretrained DnCNN denoiser [81] available at the deepinv library 7 .

DPIR We followed the annealing schedule in [80] and ran for 40 iterations. We used the pretrained DRUNet denoiser [82] available at the deepinv library.

DDRM We ran all the experiments with the default parameters: η B = 1 . 0 , η = 0 . 85 , and 20 steps for the DDIM sampler [59]. For the Gaussian deblur problem, we used the SVD-based forward model implementation based on separable 1D convolution. We ran it with an additional batch dimension to collect multiple samples.

DPS We followed the original paper to use a 1000-step DDPM sampler backbone. For the linear inverse problems, we used the step size given in [17], i.e. ζ ′ = 1 . For the nonlinear inverse problems, we optimized the step size ζ ′ by performing a grid search, which led to ζ ′ = 3 for CDP and Fourier phase retrieval and ζ ′ = 0 . 001 for black hole imaging. For the synthetic prior experiments, we also optimized the step size and used ζ ′ = 0 . 1 for compressed sensing and ζ ′ = 1 for Gaussian deblur. We ran it with an additional batch dimension to collect multiple samples.

PnP-SGS We performed a grid search for the coupling parameter ρ and found that ρ = 0 . 1 worked the best for all problems. We followed the practice in [19] to have a burn-in period of 20 iterations during which the reverse diffusion is early-stopped. We ran the algorithm for 100 iterations in total and collect 20 samples in the 80 iterations after the burn-in period.

DPnP We implemented the DDS-DDPM sampler for the prior step. For fair comparison, we used the same annealing schedule for the coupling strength (denoted as η k in [78]) as PnP-DM. We ran it for the same number of iterations for each inverse problem with the same way of collecting samples as our method.

HIO We set β = 0 . 7 and applied both the non-negative constraint and the finite support constraint. To mitigate the instability of reconstruction depending on initialization, we first repeatedly ran the algorithm with 100 different random initializations and chose the reconstruction that has the best measurement fit. Then we ran another 10,000 iterations with the chosen reconstruction to ensure convergence and report the metrics on the final reconstruction.

E Additional related works

Image reconstruction with plug-and-play priors Plug-and-Play priors (PnP) [69] is an algorithmic framework that leverages off-the-shelf denoisers for solving imaging inverse problems. Recognizing the equivalence between the proximal operator and finding the maximum a posteriori (MAP) solution to a denoising problem, PnP substitutes the proximal update in many optimization algorithms, such as ADMM [13, 56] and HQS [82, 80], with generic denoising algorithms, particularly those based on deep learning [49, 82, 80]. The PnP framework enjoys both convergence guarantees [65, 56] and strong empirical performance [76, 1] due to its compatibility with stateof-the-art learning-based denoising priors. Recent works have also proposed learning-based PnP frameworks that have direction interpretations from an optimization perspective [20, 26]. See [36] for a comprehensive review on the theory and practice of PnP.

Posterior sampling with MCMC and learning priors Learning-based priors have also been considered in the Bayesian context [35], where one seeks to sample the posterior distribution defined under a learned prior. An important technique is denoising score matching (DSM) [70], which connects image denoising with learning the score function of an image distribution. Based on DSM, prior works have incorporated deep denoising priors into MCMC formulations, particularly focusing the Langevin Monte Carlo and its variants as they involve the score function of the target distribution [40, 34, 41, 66, 54]. Recently, methods based on SGS have also gained increasing popularity [53, 19, 6, 27, 78]. Unlike PnP methods based on optimization, these sampling methods possess the ability to generate diverse solutions and quantify the uncertainty of solution space.

Solving inverse problems with diffusion models The remarkable performance of diffusion models [33, 63] on modelling image distributions makes them desirable choices as images priors for solving inverse problems. One popular approach is to leverage a pretrained unconditional model and modify the reverse diffusion process during inference to enforce data consistency [73, 18, 17, 60, 39, 61, 62, 8, 44, 84, 58]. Despite of the promising performance of these methods, they usually involve approximations and empirically driven designs that are hard to justify theoretically and may lead to inconsistent sample distributions. Another line of work learns task-specific models, which achieves higher accuracy at the cost of re-training models for new problems [4, 57, 44]. Methods based on Particle Filtering and Sequential Monte Carlo are also considered to ensure asymptotic consistency [11, 23, 74]. Diffusion models have also been considered as a prior for variational inference [28, 47] and pluy-and-play image reconstruction [32, 48].

F Additional experimental results

F.1 Synthetic prior experiment

In addition to the compressed sensing experiment presented in the main paper, we show another comparison on a Gaussian deblurring problem in Figure 10. Here, the linear forward model A ∈ R m × n is a 2D convolution matrix with a Gaussian blur kernel of size 7 × 7 and standard deviation 3 . 0 . Similar to the compressed sensing experiment, both methods yield accurate reconstructions of the mean. However, in terms of the posterior standard deviation, DPS exhibits a notable difference from the ground truth, whereas our method achieves a significantly more accurate result.

F.2 Linear inverse problems

We provide visual comparisons for the Gaussian deblur and super-resolution problems in Figure 11.

Super-resolution

GT prior

Std. deviation (     )

SD

Mean

0.15

-1.2

1.2

GT image

std. dev.

mean

GT posterior

DPS

Ours

(EDM)

0.05

DPIR

DPS

PnP-SGS

Measurement

Ground truth

Ours:

PnP-DM (VE)

Additional visual examples are provided in Figure 12 (Gaussian deblurring), Figure 13 (motion deblurring), and Figure 14 (super-resolution).

F.3 Nonlinear inverse problems

We provide visual comparisons for the CDP reconstruction problem in Figure 15, where we visualize one sample for each method. As shown by the red zoom-in boxes, PnP-DM can recover fine-grained features such as the hair threads that are missing in the reconstructions by the baselines. Additional reconstruction examples are given in Figure 16 for the CDP reconstruction problem.

We then show some additional reconstruction examples with comparison to DPS in Figure 17. For each method, we visualize the best reconstruction out of four runs for each test image according to the PSNR value. While DPS failed on around half of the test images, our proposed method provided high-fidelity reconstructions on almost all test images. This comparison highlights the better robustness of our method over DPS.

<latexi sh

1_b

64="fKDp

7jC

X

RYFk3

Sv

>A

M

H

c

Z

9

2

V

d

L

J

B

zyN

g

QE

U

I

r

GP

0n

m

O

/

8

+u

T

q

w

5

W

o

Ours

(EDM)

Measurement

Ground truth

Ours

(VE)

Ours

(EDM)

Measurement

Ground truth

Ours

(VE)

Ours

(EDM)

Measurement

Ground truth

Ours

(VE)

Ours

(EDM)

Measurement

Ground truth

Ours

(VE)

corsh

F.4 Black hole imaging

Finally, we present visual examples from the black hole imaging experiments. Samples generated by PnP-DM (EDM) and DPS using the simulated data are shown in Figure 18, with the data mismatch metric labeled at the top right corner of each sample. Consistent with the results in Figure 8, DPS can only capture one of the two posterior modes. DPS samples from Mode 2 and Mode 3 significantly deviate from the measurements and lack the expected black hole structure. In contrast, PnP-DM successfully samples both posterior modes and consistently produces samples that fit the measurements well. Additionally, Figure 19 presents more samples obtained by applying PnP-DM to the real M87 black hole data. The generated samples are not only diverse but also fit the measurements

Ours

(EDM)

Measurement

Ground truth

Ours

(VE)

Ours

(EDM)

Measurement

Ground truth

Ours

(VE)

Coded diffraction patterns

DPS

PnP-SGS

DPnP

Ground truth

Ours:

PnP-DM (EDM)

Ours:

PnP-DM (iDDPM)

well with data mismatch values around 2. These samples exhibit a ring diameter consistent with the official EHT reconstruction in Figure 1 and share a common bright spot location at the lower half of the ring.

F.5 Further analysis

Sensitivity analysis on the annealing schedule { ρ k } In Figure 20, we present a sensitivity analysis on the annealing schedule { ρ k } . In particular, we show the PSNR curves of x k with different exponential decay rates α (left) and minimum coupling levels ρ min (right) for one linear (super-resolution) and one nonlinear (coded diffraction patterns) problem. We have the following conclusions based on the results. First, different decay rates lead to different rates of convergence, which corroborates with our theoretical insights that ρ plays the same role as the step size. The final level of PSNR is not sensitive to different decay rates, as all curves converge to the same level. Second, as ρ min decreases, the final PSNR becomes higher. This is as expected because the stationary distribution of the x k , π X should converge to the true target posterior, p ( x | y ) , as ρ decreases.

Convergence curves with intermediate visual examples In Figure 21, we show some visual examples of intermediate x k and z k iterates (left) and convergence plots of PSNR, SSIM, and LPIPS for x k (right) on the super-resolution problem. As ρ k decreases, x k becomes closer to the ground truth and z k gets less noisy. Both the visual quality and metric curves stabilize after the minimum

Ours

(EDM)

Ours

(VE)

Ground truth

Ours

(iDDPM)

Ours

(EDM)

Ours

(VE)

Ground truth

Ours

(iDDPM)

Ours

(EDM)

DPS

Ground truth

Ours

(iDDPM)

Ours

(EDM)

DPS

Ground truth

Ours

(iDDPM)

coupling strength is achieved. Despite being run for 100 iterations in total, our method generates good images in around 40 iterations, which is around 30 seconds and 1600 NFEs.

G Licenses

We list the licenses of all the assets we used in this paper:

Data

-CelebA [45]: Unknown

<latexi sh

1_b

64="5Xk/Epw

Mrd+

z

S

JoA

8N

Q

>

gn cjZ

2

fV

u

v

7

O

C

F

m

W

T

G

H

I

Y

K

0

yR

U

L

D

3

B

9

q

P

<latexi sh

1_b

64="

MZpK

T

jWY7+NgJz

09

d8O

E

>A

n

c

/

f

UqCkB

w

I

R

H

ou

L

X

3

F

P

G

r

5

m

S

D

2

V

Q

y

v

Mode 2

Mode 1

DPS

2.82

12.48

660.61

Mode 3

21.28

39.95

37.42

842.39

505.21

654.74

4.03

3.23

4.47

Mode 2

Mode 1

2.83

3.84

3.02

2.75

3.24

3.08

2.43

2.91

Indices used in teaser: 5, 11, 32, 50, 59, 88

Indices used here: 9, 14, 25, 29, 52, 54, 64, 74, 86, 99

2.05

1.82

1.79

2.02

1.96

1.92

2.08

1.98

2.15

1.90

-FFHQ [38]: Creative Commons BY-NC-SA 4.0 license

-Simulated and real black hole data: Unknown

Code

-The license for each code repository that we have used is listed in the footnote after the repository link.

Pretrained model

-FFHQ model by Choi et al. [16]: MIT license

<latexi sh

1_b

64="R

QHXdpUwCVg

O

+

r

I

>A

c

Z

PT

EM

7K/5D

N

2

G

m

F

JL

yW

n

q

B

f

980k

3

u

Y

o

j

z

v

S

<latexi sh

1_b

64="u

HZIAN9

ELM

cBSwd

y

gq

Y

>

CXn

T

o

r

G

KJ7

D

Qv

R

W

2

V

z

3

U

0

f

+

8

/

p

5

P

m

k

<latexi sh

1_b

64="

JP

kzCd

OK

p

cn

8

9

>A

X3

ZH

j

u

v

wG

V

M

mQ

L

2F

D

r

5

f

0

U

+

T

W

R

q

o

y

/

<latexi sh

1_b

64="LgY8

X

Z

f

Q

K

HGWp+E

wq

>A

C

3

c

T

I

F

Broy

S

dk

j

N0

RM

5

D

2J

O

zu

v

7

/

9n

V

U

m

P

<latexi sh

1_b

64="I

O2V

8

M5W

9z

Umj

7TywcE

>A

CXH

Z

P

g

K/

D

d

N

Q

G

R

F

+

JL

n

q

B

f

0k

r

3

u

Y

o

p

v

S

<latexi sh

1_b

64="

EOX0pFBr

U

vM

PTw8gG

I

o

>A

C

H

c

Z

LS

d5

q

j

n

V

k

R

u

2

m

W

z

3

J

7

/

y

fY

N

K

Q

D

9

+

<latexi sh

1_b

64="g

YT

NR

PHV75W2q

Q/o

0

>A

CX3

c

Z

I

F

L+

O

urK

J

y

G

S

m

k

U

j

w

vf zE

M

9

d

n

B

D

p

8

<latexi sh

1_b

64="

3

W

AUmuQ5NOyd2E

M

DoX

g

>

C

c

ZHLT

I

F

Br

YS

k

0

G

K

n

z

q

p

fR

J

8v

7

/

V

9

w

j

+

P

<latexi sh

1_b

64="

yAV

7BIo

NCcXJ3

m

9f

Q

>

Wn

ZHLTg

F

OP

dS

E

M

G

5j zp

U

u

q

w

k

0+

v

r

/

D

K

R

2

<latexi sh

1_b

64="u

HZIAN9

ELM

cBSwd

y

gq

Y

>

CXn

T

o

r

G

KJ7

D

Qv

R

W

2

V

z

3

U

0

f

+

8

/

p

5

P

m

k

<latexi sh

1_b

64="

JP

kzCd

OK

p

cn

8

9

>A

X3

ZH

j

u

v

wG

V

M

mQ

L

2F

D

r

5

f

0

U

+

T

W

R

q

o

y

/

<latexi sh

1_b

64="LgY8

X

Z

f

Q

K

HGWp+E

wq

>A

C

3

c

T

I

F

Broy

S

dk

j

N0

RM

5

D

2J

O

zu

v

7

/

9n

V

U

m

P

<latexi sh

1_b

64="WcLu

Z7r

A8dO2MHm+

Yqv

30

w

>

C

X

TgI

F

E

R

SN

y

k

o

G

J

Q

U

D

P

f

z

n

/

j

5

V

9

B

K

p

<latexi sh

1_b

64="p

r

A

IfgN7JLO

/

9C

3

G

Q

>

ZX

c

H

S

M

F

T8V

5w

B

q

y

R

m

W

2

uY

K

z

dj

v

E

+

0

D

n

P

o

U

k

<latexi sh

1_b

64="

0z7/8G+J53qST

OIZ

QF

Y

>A

C

X

c

H

gNBE

j

w

L

op

U

P

29

m

k

v

d

r

W

f

u

K

V

M

y

n

D

R

<latexi sh

1_b

64="K/w

Wyvnq9MkdpU

2T

I

Gc

g

>A

CX3

ZHLS

F

8V

r

NYB

R

E

u

Jmf

j

5

+

7

0

D

O

P

o

Q

z

<latexi sh

1_b

64="

0z7/8G+J53qST

OIZ

QF

Y

>A

C

X

c

H

gNBE

j

w

L

op

U

P

29

m

k

v

d

r

W

f

u

K

V

M

y

n

D

R

<latexi sh

1_b

64="K

7Z

fAj

/

>

X

c

HJ

NBEIY

F

8

w

L

d

om

O0

2

9

v

W

u

3

R

q

PMU

n

p

T

y

<latexi sh

1_b

64="

+/

y

q

5

kB3

Y

>A

CZX

c

HLS

M

FI

8

9

w

W

E

GN

p

K0D

n

R

z

m

7

O

Q

rU

J

u

PSNR (dB)

8

32

8

32

8

35

8

35

PSNR (dB)

Super-resolution

Coded diffraction patterns

Different minimum coupling strength

ρ

min

Super-resolution

Coded diffraction patterns

0

100

Iteration (

)

k

0

100

Iteration (

)

k

0

100

Iteration (

)

k

0

100

Iteration (

)

k

α

= 0

.

85

α

= 0

.

9

α

= 0

.

95

α

= 0

.

85

α

= 0

.

9

α

= 0

.

95

ρ

min

= 0

.

6

ρ

min

= 0

.

3

ρ

min

= 0

.

9

ρ

min

= 0

.

9

ρ

min

= 0

.

5

ρ

min

= 0

.

1

Different decay rate

α

PSNR (dB)

8

32

0

100

Iteration (

)

k

0.2

1

SSIM

LPIPS

0

0.8

x

0

z

90

z

30

z

20

z

10

Convergence for

iterates

x

Initialization

z

0

x

1

x

11

x

21

x

31

x

91

<latexi sh

1_b

64="NW

MPZm0GBVc/UY

kQ

A

p

>

CX3

HLTgI

F

roy

SE

d

u

K

n

2z

q

j

5O

w

f

v

7

+

D

R

J

8

9

<latexi sh

1_b

64="dw2N

/zu

RS

q

OjUDG

+KgY

>A

CX3

c

ZHL

M

FI

T8V

r

W

Bk

y

J

Q

7

m

v

5

Po

p

0

9

n

f

E

<latexi sh

1_b

64="3YEX/FRC

IDnjP

9

cHoz

>A

Z

LSgM

T8V5v

f

r

U

J

pK

m

W

d

G

2

OB

u

Q

y

7

w

k

N

+

0

q

<latexi sh

1_b

64="QSU

3rjW

ABG

pH/7D

g

o

>

CX

c

Z

L

I

F

RE

V

wkJq

0

Y

K

n

2zM

O

f

N

d

8v

y

5

+

9

P

m

<latexi sh

1_b

64="20EL

Hf/Md

9Pq+o

F

YpvIk

>A

CX3

c

Z

T8

J

u

Vy

G

m

nB

5U

W

D

w

z

Q

K

7

r

N

O

R

j

<latexi sh

1_b

64="J

fRZo m7

Iz2K

C

D

LY+/

U

>A

X3

c

H

SgM

F

T8d

V

vB

r

q

y

N

n

E

5

u

W

9

P

w

Q

O0

G

j

k

p

<latexi sh

1_b

64="CFG/n+

BDyug7m

WIQ

r8

>A

c

ZHLS

M

T

dqp

3

Rk

V

w

2

z

0

Y

J

O

9

K

X

U

E

5

f

o

N

j

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have made sure that the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have added a paragraph on the limitations of our method to the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have presented our theoretical result in the main paper with the proof in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided all the necessary details to reproduce the main results in our paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have included the code for reproducing the main experimental results of our work in the supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have provided all the training and test details in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Due to the limited space and constraints of time and computational resources, we were unable to include error bars for our experiments over multiple runs. Alternatively, we evaluated our method and baselines based on multiple samples and various metrics. We conducted a uncertainty quantification analysis where we calculated the per-pixel statistics and confidence intervals. We also extensively included visual examples to better demonstrate the performance of the methods we considered in this paper. All of these efforts aimed to better reflect the significance of our improvement over existing methods.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have provided details on our compute resources in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We confirm that our research conform with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have added a 'Broarder Impacts' section in the main manuscript.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work does not involve models that have a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have properly credited the owners of all the assets used in the paper. Their licenses are listed in the footnotes and Appendix G.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We have documented the code we provide in the supplement.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our research does not involve crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our research does not involve human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIt is proposed to solve image inverse problems with diffusion priors by sliced Gibbs sampling (SGS). To sample from the product of prior and likelihood distributions, SGS assumes a pair of variables $x,z$ coupled by a Gaussian and alternates conditional sampling steps: on $x$ using the prior density multiplied with the conditional Gaussian and on $z$ using the likelihood multiplied with the conditional Gaussian. When the prior is a diffusion model, the two steps can be implemented by modifying the given pretrained score model and by Langevin dynamics (or exactly in the case of a linear problem), respectively. This procedure is evaluated on a few standard linear and nonlinear IPs and on a black home imaging problems, and the reconstructions found by the proposed method are typically visually and quantitatively better than the ones found by algorithms from prior work.\n\nSTRENGTHS:\n- The writing is mostly clear; I have no complaints on the paper organization or exposition.\n  - In particular, the exposition of SGS in the general case followed by its application to DM priors is helpful.\n- As far as I know, this is an original way of solving the diffusion posterior sampling problem.\n- Strong results (if we are to accept that they support the claims) and application to a real imaging problem.\n- Code is provided.\n\nWEAKNESSES:\n- On fairness and strength of experimental comparisons:\n  - The results do not come with confidence intervals, which makes it hard to assess significance.\n  - I may have missed/misunderstood this, but are all methods using the same pretrained prior? Appendix D seems to suggest otherwise, which would be a problem (it isn't fair to compare posterior sampling methods with different priors). \n    - Similarly, different methods seem to be assuming different numbers of sampling steps and otherwise incompatible choices. How were hyperparameters, such as the $\\rho_k$ annealing schedule, chosen?\n  - Computation cost is not sufficiently discussed.\n    - The method is run for at least a hundred Gibbs iterations in each experiment, and each one involves diffusion sampling initialized from some intermediate noise level. This makes the number of model evaluations quite large. It would be necessary to compare the number of function calls, as well as wall time, for all the methods.\n- Convergence rate:\n  - Theorem 3.1 tells us the rate of convergence to the stationary process for a fixed $\\rho$, but not how close the stationary marginal is to the true posterior. They are not equal, in general, for positive $\\rho$, and the rate of convergence shown has a $\\frac1\\rho$ factor, which means the convergence guarantees with small $\\rho$ are weaker.\n  - To understand the method, it would be important to show the dependence on the number of iterations, both by showing examples at different iterations in an illustrative case and by tracking the convergence of some metric at a function of the number of steps.\n- Please also see questions below.\n\nQUESTIONS:\n- After equation (1), writing $y=A(x)+n$ where $n\\in\\mathbb{R}^n$ means that $y$ depends deterministically on $x$. In fact $n$ should be a random variable taking values in $\\mathbb{R}^n$, which makes $y$ also a random variable.\n- I am surprised not to see any reference to the paper whose title is a suffix of this one's, [\"Diffusion models as plug-and-play priors\", NeurIPS'22], also abbreviated \"PnP\" in some subsequent work. There a stochastic optimisation is used to find modes of the posterior.\n  - Related to this, \"PnP-DM\" is not a very informative name for the proposed algorithm when so many of the baseline methods already have \"PnP\" in the name in some other configuration of letters.\n\nLIMITATIONS:\nYes, but it would be good to compare computation costs, see above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel posterior sampling algorithm for solving inverse problems with Diffusion models. The proposed algorithm is based on the Split Gibbs sampling scheme and consists in, first introducing an extended distribution $\\pi_\\rho(x, z)$ that admits the original posterior $\\pi$ as $x$ marginal after letting $\\rho$ tend to $0$. Then, for a fixed $\\rho$ this joint distribution can be sampled via Gibbs sampling; the form of the joint distribution $\\pi_\\rho(x, z)$ allows sampling sampling both $X$ and $Z$ easily. For $Z$, this can be done exactly in the case of linear inverse problems, or with Langevin Monte Carlo otherwise. For $X$, the particular structure of the joint distribution is exploited and as it turns out, it can be sampled by simply running the learned backward Diffusion. Notably, this is the only step that requires using the learned denoiser. The theoretical properties of the ideal sampler are inverstigated and extensive experiments are considered.\n\nSTRENGTHS:\nI highly enjoyed going through this paper and I found it to be interesting. The strengths are: \n- The method is original and clever; the joint distribution introduced, which stems from the SGS approach, allows almost exact \nsampling of both the conditional distributions. Furthermore, the considered framework allows having actual theoretical guarantees for the sampler (although in an ideal case). This is still quite informative and contrasts with existing approaches (besides the SMC-based ones which also allow more straightforward theoretical guarantees). The main perk of this method wrt to the latter is that it has theoretical guarantees while not requiring absurd memory requirements due to the storage of a large number of particles. \n\n- The writing is extremely clear. The paper is written in a way that It is straightforward to understand while still being quite detailed. This is a significant perk (Still, i think that section 3.4 could be improved as I had to go through the supplementary material to actually understand what the proof technique was). \n\n- Finally, the experiments are very good; the authors first consider a toy example in which the posterior has a closed form. This serves as a nice introduction to the experimental section and showcases the perks of the present method in comparison with DPS. Next, the imaging experiments are rigorously executed, with a precise look at the uncertainty quantification. Finally, an original black hole experiment is considered and I found it to be very interesting.\n\nWEAKNESSES:\n- **Computational cost**:  I believe that the proposed method is much slower than existing alternatives. This is not that much of an issue for me (but of course this depends on the reader and practitioner). In my opinion the compute time is not properly addressed; although the authors discuss it in the bottom of page 21, I would like to see a proper runtime comparison with existing methods. I would like to emphasize that  this is only a minor weakness.\n\n- **Originality**: I believe that the proposed method is related to SDEdit [1]; it can be thought of as a variant of SDEdit. Indeed, to see why this is the case, when the present algorithm is applied to inpainting, the first iteration, the likelihood step, will result in a noisy image with the observation already present In it. This image is then denoised using the backward process during the prior step. This is in fact very similar to sdedit which starts with some initial image, say the pseudo-inverse of the observation, noises it to some given noise level and denoises it using the backward Diffusion process. See Algorithm 3, [1]. I believe that the main difference with the algorithm presented in your paper is the use of the decreasing noise schedule, which makes sense theoretically. For a fair treatment I suggest you mention this in your paper. \n\n[1] Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.Y. and Ermon, S., 2021. Sdedit: Guided image synthesis and editing with stochastic differential equations.\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nsee above", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a method to address Bayesian inverse problems in computational imaging. It leverages the generative capabilities of diffusion models (DMs) to sample the posterior distribution over all possible solutions from noisy and sparse measurements. The method combines a Markov chain Monte Carlo (MCMC) algorithm with a general DM formulation to perform rigorous posterior sampling, effectively integrating state-of-the-art DMs as expressive image priors. The method deviates from current methods that rely on approximating the intractable posterior via separating the forward operator from an unconditional prior over the intermediate noisy image. The approach is validated on six inverse problems, demonstrating superior accuracy and posterior estimation compared to existing DM-based methods.\n\nSTRENGTHS:\n- The paper presents a novel combination of MCMC and diffusion models to solve inverse problems. This use of DMs as priors in a Bayesian framework can be integrated into SOTA diffusion models as plug-n-play expressive image priors for Bayesian inference.\n   \n- The presented method is rigorous. The use of the Split Gibbs Sampler and the EDM formulation for the prior step is well-executed. The theoretical insights provided, including the stationarity guarantee in terms of average Fisher information, add depth to the method.\n\n- The paper is well-structured and supported by diagrams and pseudocode to enhance understanding. The distinction between existing methods and the proposed approach is clearly articulated.\n\n- Experiments show the efficacy of the proposed method compared to existing methods on a diverse set of real-world problems, highlighting its potential broader impact on computational imaging applications.\n\nWEAKNESSES:\n- While the method demonstrates superior performance, the computational cost and efficiency are not thoroughly discussed. A comparison of computational resources required compared to other methods would be beneficial.\n\nThe impact of various parameters, such as the annealing schedule for the coupling parameter ρ, on the method's performance is not deeply investigated. A sensitivity analysis could provide insights into the method's robustness.\n\nQUESTIONS:\n1. Can the authors compare the computational resources and time required for PnP-DM versus existing DM-based methods?\n      \n2. How sensitive is the performance of PnP-DM to the choice of the annealing schedule for the coupling parameter ρ? Is there an optimal range or strategy for selecting these parameters?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper proposes a Markov Chain Monte Carlo algorithm for posterior sampling in both linear and non-linear inverse problems. The core of the proposed method is based on a Split Gibbs Sampler that alternates between two steps: one involving the likelihood and the other the prior. Additionally, the paper connects the Bayesian denoising problem with unconditional generation using Diffusion Models. The proposed method is validated on a range of linear and non-linear inverse problems, with an additional real-world application in black hole imaging.

STRENGTHS:
- The proposed method outperforms DPS, with satisfactory evidence provided by the authors.
- Unlike some previous works, the proposed method is effective for both linear and non-linear inverse problems.
- The paper provides sufficient theoretical analysis to support the proposed method.

WEAKNESSES:
There are no apparent weaknesses. However, I am curious about the reconstruction speed comparison (seconds/image) between DPS and the proposed method. It would be practically very attractive for the community to use this algorithm if a rigorous comparison of reconstruction speed is made.

QUESTIONS:
No questions.

LIMITATIONS:
No limitations other than those mentioned in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors treat the problem of sampling from posterior of image inverse problems. Their formulation is based on the Split Gibbs Sampler (SGS) algorithm, which alternates between sampling from Moreau regularized versions of the prior and of the likelihood. The main contribution of the paper is to use the EDM formulation of diffusion models for sampling the prior term. The EDM formulation was \nproposed to unify various diffusion models in a common general formulation.\n\nSTRENGTHS:\nFirstly the paper is well-written, provides clear and concise explanations of the methods and contributions. The authors have also included some theoretical insights that guarantee the convergence of each sampling step of the algorithm to the right stationary distributions when the number of iterations tends to infinity.  \n\nThe strength of the paper is mainly its experimental section, which covers a range of both linear and non-linear inverse problems. Additionally, I find interesting the inclusion of a toy experiment with a Gaussian prior, which allows comparing to ground-truth posterior distributions.\n\nWEAKNESSES:\n1. **Significance of Contribution**: My primary concern is the overall significance of the contribution. When compared to the works of Coeurdoux et al. (2023) and Xu & Chi (2024), the advancements appear to be relatively minor. Specifically, the contribution seems to involve using a more general formulation of diffusion models (EDM) instead of the more specific DDPM or DDIM. Are there additional conceptual differences? Furthermore, given this similarity, what accounts for the significant performance disparity with these methods?\n\n2. **Algorithm Efficiency**: The algorithms seem to require considerable computational time, necessitating the diffusion process to be run at each iteration. While this is mathematically sensible, it appears excessive. Given the annealing process chosen, how many calls to the denoiser are typically made in practice? Additionally, how does this number compare to methods that perform a single diffusion using an approximation of p(y|x_k), such as DPS? Including a comparative table in the paper would be valuable.\n\n3. **Theoretical Results**: The theoretical results presented, while interesting, are somewhat limited as they assume a fixed rho parameter, which is not the case in practical applications. Additionally, the algorithm is presumed to run with the true score, which is also not realistic. In addition, although it may be beyond the scope of the current paper, providing theoretical insights into the convergence of the proposed Gibbs Sampler (with approximate score) to the true posterior would be beneficial.\n\nQUESTIONS:\n- In Figure 4, the diffusion process is represented in a closed-form. Did you train a denoiser, or did you use the closed-form score instead?\n\n- How did you tune the hyperparameters for your method and for each method in the comparison? Did you use their default hyperparameters? For example, DPIR provides hyperparameters fine-tuned for various inverse problems. However, the SR x4 task is not one of the inverse problems for which DPIR's authors fine-tuned their hyperparameters, likely making the default hyperparameters sub-optimal for this task.\n\n- Did you employ any regularization parameters to balance the data-fidelity and regularization terms?\n\n- For the backward diffusion process, did you use the same number of iterations for all values of  $\\rho_k$ ? Specifically, is the step size of the diffusion process consistent, or is it adjusted according to the value of $\\rho_k$ ?\n\nLIMITATIONS:\nNo limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}