A Variational Perspective on Solving Inverse Problems with Diffusion Models
Authors: Morteza Mardani
Jiaming Song
Jan Kautz
Arash Vahdat
Venue: ICLR 2024 poster (2024)
Keywords: diffusion models
score matching
variational approximation
regularization by denoising
inverse problems
Primary area: generative models

Abstract: Diffusion models have emerged as a key pillar of foundation models in visual domains. One of their critical applications is to universally solve different downstream inverse tasks via a single diffusion prior without re-training for each task. Most inverse tasks can be formulated as inferring a posterior distribution over data (e.g., a full image) given a measurement (e.g., a masked image). This is however challenging in diffusion models since the nonlinear and iterative nature of the diffusion process renders the posterior intractable. To cope with this challenge, we propose a variational approach that by design seeks to approximate the true posterior distribution. We show that our approach naturally leads to regularization by denoising diffusion process (RED-diff) where denoisers at different timesteps concurrently impose different structural constraints over the image. To gauge the contribution of denoisers from different timesteps, we propose a weighting mechanism based on signal-to-noise-ratio (SNR). Our approach provides a new variational perspective for solving inverse problems with diffusion models, allowing us to formulate sampling as stochastic optimization, where one can simply apply off-the-shelf solvers with lightweight iterates. Our experiments for various linear and nonlinear image restoration tasks demonstrate the strengths of our method compared with state-of-the-art sampling-based diffusion models. The code is available online \footnote{\url{https://github.com/NVlabs/RED-diff}}.

Published as a conference paper at ICLR 2024
A VARIATIONAL PERSPECTIVE ON SOLVING INVERSE PROBLEMS WITH DIFFUSION MODELS
Morteza Mardani, Jiaming Song, Jan Kautz, Arash Vahdat
NVIDIA Inc.
mma rdani, ji ami ngs, jkautz, wwwwdattevvddddeaaaaaooo
ABSTRACT
Diffusion models have emerged as a key pillar of foundation models in visual domains. One of their critical applications is to universally solve different down- stream inverse tasks via single diffusion prior without re-training for each task. Most inverse tasks can be formulated as inferring a posterior distribution over data (e.g., a full image) given measurement (e.g.. masked image). This is however challenging in diffusion models since the nonlinear and iterative nature of the diffusion process renders the posterior intractable. To cope with this challenge, we propose a variational approach that by design seeks to approximate the true posterior distribution. We show that our approach naturally leads to regularization by denoising diffusion process (RED-diff) where denoisers at different timesteps concurrently impose different structural constraints over the image. To gauge the contribution of denoisers from different timesteps, we propose a weighting mechanism based on signal-to-noise- to-noise-ratio (SNR). Our approach provides a new variational perspective for solving inverse problems with diffusion models, allow- ing us formulate sampling as stochastic optimization, where one can simply apply off-the-shelff solvers with lightweight iterates. Our experiments for vari- ous linear and nonlinear image restoration tasks demonstrate the strengths of our method compared with state-of-the-art sampling-based diffusion models. The code is available online!
INTRODUCTION
Diffusion models such as Stable diffusion (Rombach al. 2021 are becoming an integral part of nowadays visual foundation models. An important utility of such diffusion models is to use them as prior distribution for sampling in various downstream inverse problems appearing for instance in image restoration and rendering. This however demands samplers that are (i) universal and adaptive to various tasks without re- re-training for each individual task, and (ii) efficient and easy to tune.
There has been a few recent attempts to develop universal samplers for inverse problems; (Kawar al.
2022a Song et al., [023] Chung al.,
2022a Kadkhodaie & Simoncelli 2021 Graikos et al.
2022) to name a few. DDRM (Kawar et al. 2022a) was initially introduced to extend DDPM (Ho ett al. 2020) to handle linear inverse problems. It relies on SVD to integrate linear observations into the denoising process. DDRM however needs many measurements
work. Later on, IIGDM was introduced (Song et al. 2023) to enhance DDRM. The crux of HIDMM is to augment the denoising diffusion score with the guidance from linear observations through inversion. In a similar vein, DPS (Chung et al., 2022a) extends the score modification framework to general (nonlinear) inverse problems. The score modification methods in DPS and IGDDM, however, heavily resort approximations. In essence, the nonlinear and recursive nature of the backward diffusion process renders the posterior distribution quite intractable and multimodal. However, DPS and IIGDM rely on a simple unimodal approximation of the score which is a quite loose approximation at many steps of the diffusion process.
To sidestep the challenges for posterior score approximation, we put forth fundamentally different approach based on variational inference (Blei et al. 2017]] Ahmed et al.) (2012) Hoffman et al.) 2013)) Adopting the denoising diffusion model as our data prior and representing the measurement model as a likelihood, we use variational inference to infer the posterior distribution of data given the observations. Our method essentially matches modes of data distribution with a Gaussian distribution
https://gi NV1 abs RRD0 ff

---

Published as a conference paper at ICLR 2024
y/
34 = C:jl oie
t ~~ [ N(O,1)
t 100
= 500
t 900
Mcasurement matching loss |1v - (1))
Score matching regularization (sg(co(#z; () c]) /
Denoiser co( t)
The schematic diagram of our proposed variational sampler (RED-diff). The forward denoising diffusion process gradually adds noise the estimate μ.. The denoisers of the backward diffusion process apply score- mooee-matching regularization to the measurement matching loss. The refined estimate using optimization is then fed back to the forward process and the process repcats.
Figure 1:
using KL divergence. That leads to simple (weighted) score-matching criterion that regularizes the measurement matching loss from observations via denoising diffusion process. Interestingly, the score-matching regularization admits interpretable form with simple gradients; see Fig. [
This resembles the regularization-by-or by-noo----ddnoiinggg (RED) framework by Romano et al. (2016), where denoisers at different stages of the diffusion process impose different structural constraints from high-level semantics to fine details. This is an important connection that views sampling as stochastic optimization. As a result, one can simply deploy the rich library of off-the-shelf optimizers for sampling which makes inference efficient, interpretable, and easy to tweak. We coin the term RED- diff to name our method. is however worth noting that our framework differs from RED in several aspects: (i) we derive our objective from a principled variational perspective that is well studied and understood, and (ii) our regularization uses feedback from all the diffusion steps with different noise levels while RED uses a single denoising model.
For the success of the score matching regularization, denoisers different timesteps need to be weighted properly. To do so, we propose weighting mechanism based on densoing SNR at each timestep that upweights the earlier steps in the reverse diffusion process and dewn-weights the later timesteps. To verify the proposed idea, we conduct experiments and ablations for various linear and nonlinear inverse problems. Our main insights indicate that: (i) RED-diff achieves superior image fidelity and perceptual quality compared with state-of-thee -ft----rttt--ttt-- samplers for image inverse problems; (ii) RED-diff has lightweight iterates with no score Jacobian involved as in DPS and IIGDM, and as result, it is more memory efficient and GPU friendly: (iRt) Our ablation studies suggest that the optimizer parameters such as learning rate and the number of steps are suitable knobs to tweak the trade-off between fidelity and perceptual quality.
Contributions. All in all, the main contributions of this paper are summarized as follows:
We propose, RED-diff, a variational approach for general inverse problems, by introducing framework that mitigates the posterior score approximation involved in recent IIGDM (Song et al. 2023) and DPS (Chung et al., 2022a)
a rigorous meniimu--likelibee
We establish a connection with et al. 2016), which allows to treat sampling as stochastic optimization, and thus enables off-the-shelf optimizers for fast and tunable sampling
(RED) framework (Romano
We propose a weighting mechanism based on denoising SNR for the diffusion regularization We conduct extensive experiments for various linear and nonlinear inverse problems that show superior quality and GPU efficiency of RED-diff against state-of-the-as samplers such IIGDM and DPS. Our ablations also suggest key insights about tweaking sampling and optimization to generate good samples.
2 RELATED WORKS
Our work is primarily related to the following lines of work in the context of diffusion models.
Diffusion models for inverse problems: There are several recent works to apply diffusion models in plug-and-play fashion to inverse problems in various domains such natural images (Kadkhodaie
2

---

Published as a conference paper at ICLR 2024
& Simoncelli
2021:
Jalal et al.; 2021 Kawar al. 2022a Song al. 2023 Chung et al. 2022a| by
Graikos et al.
2022:
Chung et al., 2023a b). medical images (Jalal et al. 20211, and audio processing
(Kong et al. 2020). We primarily focus images, where these works primarily differ in the way that
they handle measurements. As some of the early works, Kadkhodaie & Simoncelli (202T) Jalal et al. (202T) adopt Langevine dynamics for linear inverse problems and integrate the observation guidance via either projection (Kadkhodaie & Simoncelli (021), or gradient of the least-squares
fidelity (Jalal et al. 2021). Some other works adopt DDPM (Ho al. 2020) diffusion and alternate between diffusion denoising and projection steps (Choi et al. [021]] Chung et al. 2022c). The iterations however can accumulate error that pushes the trajectory off the prior manifold, and thus MCG method (Chung et al. 2022b) proposes an additional correction term inspired by the manifold constraint to keep the iterations close to the manifold. DDRM (Kawar et al. 2022a) extends DDPM to solve linear inverse problems using matrix SVD, but it fails for limited measurements.
To address this shortcoming, recent methods aim to provide guidance by differentiating through the diffusion model in the form of reconstruction guidance (Ho et al. (022). which is further extended
in DPS (Chung et al. 2022a to nonlinear inverse problems. HIGDM (Song et al. (022)) introduces pseudoinverse guidance that improves the guidance approximation by inverting the measurement model. Its scope is however limited to linear and certain semi- linerr tasks (such as JPEG (Kawar
al. 2022b)) However, both ITGDM and DPS heavily rely posterior score, which is quite crude for non-small noise levels at many steps of the diffusion process.
approximation of the intractable
Note also that, a different method has also been recently proposed by Graikos et al. (2022) which regularizes the reconstruction term of inverse problems with the diffusion error loss. This is similar to
the traditional plug-and-play prior (P3) approach for inverse problems /Venkatakrishnan al. 2013) that roots back to ADMM optimization (Boyd et al. 2011). Our method is however closer in spirit to the RED framework, which offers more flexibility for optimizer and tuning; see e.g.. (Romano et al., 2016 Cohen et al., 2021).
Diffusion models for 3D: few recent works have adopted distillation loss optimization to generate 3D data from 2D diffusion priors, which is related to our view of treating sampling as optimization. For instance, DreamFusion (Poole et al., 2022) and ProfilicDreamer (Wang et al., 2023) adopt a probability density distillation loss as the criterion for text-to-3D generation. Followup works include SparseFusion (Zhou & Tulsiani 2022) that generates 3D given a few (e.g. just two) segmented input images with known relative pose, and NeuralLift-366 (Xu et al.) 2022) that lifts single 2D image to 3D. All these methods use a distillation loss, that bears resemblance with our (unweighted) denoising regularization. However, they aim optimize for parametric 3D NeRF model that is fundamentally different from our goal.
3 BACKGROUND
In this section, we first review diffusion models in Section 3.1 and we discuss how they are used for solving inverse problems in Section 3.2
3.1 DENOISING DIFFUSION MODELS
Diffussio models (Sohl-Dickstein et al. 2015 Ho et al., 2020; Song et al. 2021b) consist of two processes: forward process that gradually adds noise input images and reverse process that learns to generate images by iterative denoising. Formally the forward process can be expressed by the variance preserving stochastic differential equation (VP-SDE) (Song al., 2021b) dx +1,(()) / dW for t e [0, 7] where (() Punin + (Bmax BBmin) rescales the time variable, and dw is the standard Wiener process. The forward process is designed such that the distribution of standard Gaussian distribution (i.e., Ø7 ~ NN I)). The V.. log p(xx)) + /B(t)do where Vz, log p(æ))
JT at the end of the process converges
reverse process is defined by dr 1 (t) ((t) x. is the score function of diffused data at time t, and dē is the reverse standard Wiener process.
Solving the reverse generative process requires estimating the score function. In practice, this is done by sampling from the forward diffusion process and training the score function using the denoising score-matching objective (Vincent 201T). Specifically, diffused samples are generated by:
0,.00 + oee, NO0 T), LE [0,T]
where To ~ Paaaa is drawn from data distribution, o, = 1 - e 8s)dds and a: = =V1- 11- of Let's denote the parameterized score function (i.e., diffusion model) by co 1) -a, V., log ((.)( with
3

---

Published as a conference paper at ICLR 2024
parameters 8, train (0(2);t) with a mixture of Euclidean losses, such as
min Ezo (")
|||c ca =0( t)
Other loss- loss-weighting functions for can be used as well. Given trained score function, samples can be generated using DDPM (Ho et al. 2020). DDIM (Song et al. 2020), or other solvers (Lu et al. 2022
Zhang & Chen 2022
Dockhorn al. 2022
3.2
SCORE APPROXIMATION FOR INVERSE PROBLEMS
Inverse problems can be formulated as finding from (nonlinear and noisy) observation:
= f(xa) + v, v ~~ N=(
(2)
where the forward (a.k.a measurement) model f is known. In many applications, such as inpainting. this is severely ill-posed task that requires a strong prior to find a plausible solution. Our goal is to leverage the prior offered by (pretrained) diffusion models, in a plug and-play fashion, efficiently sample from the conditional posterior. Let's denote the prior distributions imposed by diffusion models as p(rr). The measurement models can be represented by p(y (m)) := (f(o))) The goal of solving inverse problems is sample from the posterior distribution p(roly).
As we discussed in the previous section, diffusion models rely on the estimated score function to generate samples. In the presence of the measurements y. they can be used for generating plausible To ~ p(zoly) as long as an approximation of the conditional score for p(z, [g] over all diffusion steps is available. This is the idea behind IIGDM (Song et al.) (2023) and DPS (Chung al. 2022a).
Specifically, the conditional score for p p(xx/y) based on Bayes rule is simply obtained as
Vx log p(re(( = V. log p(y/ V p(x,)
The overall score is a superposition of the model likelihood and the prior score. While V. log p(x)) easily obtained from a pretrained diffusion model, the likelihood score is quite challenging and intractable to estimate without any task-specific training. This can be seen from the fact that p(y( = J p(y p( )dzg- Although p(yl(z)) takes a simple Gaussian form, the denoising distribution p( azx) can be highly complex and multimodal (Xiao al. 2022 As result, p(y( can be also highly complex. To sidestep this, prior works (Song et al.
2023 Chung et al. 2022a
Kadkhodaie & Simoncelli [02] Ho et al. 2022) resort Gaussian approximation of p [t
around the MMSE estimate
Io [ = at
(x. (x),))
(4)
VARIATIONAL DIFFUSION SAMPLING
In this section, we introduce our variational perspective on solving inverse problems. To cope with the shortcomings of previous methods for sampling the conditional posterior p(zoly), we propose variational approach based on KL minimization
min K T (a(roll))ll((
q
where q NH, o² is variational distribution. the data distribution that matches the observations. It is casy to show that the KL objective in Eq. 5 can be expanded as K KL 20 ( p(zolu))
The distribution q seeks the dominant mode in
- -Bg(zol)) [ logg((/zo))))) K (g(2ol)) (( + p(y)
(6)
temm
where term (i) is the variational bound that often used for training variational autoencoders (Kingma & Welling 2013 Rezende al. 2014) and term (ii) is the observation likelihood that is constant w.r.t. q. Thus, to minimize the KL divergence shown in Eq. 55 w.r.t. q, it suffices to minimize the variational bound (term (i) in Eq. w.r.t. q. This brings to the next claim.
Proposition 1. Assume that the score is learned exactly, i.e., ea(Ie;) = -o,Vo v., logp(x:). Then, the KL minimization w.r.t q in Eq. 5 equivalent to minimizing the variational bound (term (i) in Eq. 6. that itself obeys the score matching loss:
|/y f 12
S w( ((t)
min E Bg(zoll) (a.0)
V logg((11))))) - Ve..ogg((z>)))dd (7)
20
4

---

Published as a conference paper at ICLR 2024
where q(zely) Maau, lafo² o)) produces samples 2, by drawing zo from q(Zulz) and applying the forward process in Eq. and (t) = B(t) is a loigwweighting term.
Above, the first term is the measurement matching loss (i.c., reconstruction loss) obtained by the definition of p(y)z)). while the second term is obtained by expanding the KL term in terms of the score-matching objective a weighting based on maximum likelihood (the proof is provided in the supplementary material). The second term can be considered as a acore-matching regularization term imposed by the diffusion prior. The integral is evaluated on a diffused trajectory, namely 2x g( for E [O, 7]. which is the forward diffusion process applied (zolly) Since ( admits a simple Gaussian form, we can show that g( (1) is also Gaussian in the form g = (aju, (afo² + r) (see (Vahdat al. 2021)) Thus, the score function Vz, log ((arely) can be computed analytically.
shown in /aahdat et al. 2021 Song al. 2021a), and (+t) /3())22
Assuming that the variance of the variational distribution is small constant value near zero (i.e., 0), the optimization problem in Eq. can be further simplified to:
min
ly - 1(1]
2 + e (2.())(()) (1: 1) -
(8)
recon
reg
where Cuff dee. In nutshell, solving the optimization problem above will find an image that reconstructs the observation y given the measurement model f. while having a high likelihood under the prior imposed by the regularization term.
Remark [Noiseless observations]. If the observation noise o, = 0, then from equation 6 the reconstruction term boils down to hard constraint which can be represented as indicator function (y-f(p)) that is zero when y = f(u) and infinity elsewhere. In practice, however we can still use equation 7 with a small o, as approximation.
4.1 SAMPLING AS STOCHASTIC OPTIMIZATION
The regularized score matching objective Eq, alloos to formulate sampling optimization for inverse problems. In essence, the ensemble loss over different diffusion steps advocates for stochastic optimization suitable sampling strategy.
However, in practice the choice of weighting term w(t) plays a key role in the success of this optimization problem. Several prior works training diffusion models (Ho et al., 2020 Vahdat et al. 2021 Karras et al. 2022 Choi et al., 2022) have found that reweighting the objective over t plays key role in trading content vs. detail at different diffusion steps which we also observe in our case (more information in Section 4.3) Additionally, the second term in Eq. g marked by
"reg" requires backpropagating through pretrained score function which make the optimization slow and unstable. Next, we consider a generic weighting mechanism (t)) = (1)) for a positive-valued function wt),, and we show that if the weighting is selected such that (0) = the gradient of the regularization term can be computed efficiently without backpropagating through the pretrained score function.
Proposition 2. If wf)) and a = 0, the gradient of the score matching regularization admits
V vgge(()) - -c)
where X : 2T dw((t)
First-order stochastic optimizers. Based on the simple expression for the gradient of motehnn regularization Proposition 2, we can treat time as uniform random variable. Thus by sampling randomly over time and noise, we can easily obtain unbiased estimates of the gradients. Accordingly, first-order stochastic optimization methods can be applied to search for J.. We list the iterates under Algorithm 1. Note that we define the loss per timestep based on the instantancous gradient, which can be treated as a gradient emphasize that score is not differentiated during the optimization. The ablations in Section [D.4] show that (descending) time stepping from t = T to t = 0, as in standard backward diffusion samplers such DDPM and DDIM, performs better than random time sampling in practice.
linear loss. We introduce the notation (sg) as stopped-gradient gradient to
Remark [Non-zero dispersion]. Note that Proposition 2 derives the gradient for no dispersion case (i.e., σ 0) for simplicity. The extension to nonzero dispersion is deferred to the appendix (A.3).
5

---

Published as a conference paper at ICLR 2024
Algorithm 1 Variational sampler (RED-diff)
Input: y. f()), ov. L, (as, Ae,
Initialize: μ40 for l L....,
[]
~ Ln)
3: 0zl oGE
loss = ly + (s(ss(oo(ass) =))"M
n - OptimizerStep(loss)
end
Return:
4.2 REGULARIZATION BY DENOISING
Note that variational sampler strikes resemblance with the regularization by denoising (RED) framework (Romano et al. 2016). In essence, RED is a flexible way to harness a given denoising engine for treating general inverse problems. RED regularization effectively promotes smoothness for the image according to some image-adaptive Laplacian prior. To better understand the connection with RED, let us look at the loss per timestep of our proposed variational sampler. From the gradient expression in Proposition 2, it is useful to form the loss at timestep t as
y f + >e (sg) <)) -()
This regularization term resembles RED. A small regularization implies that either the diffusion reaches a fixed point, namely ( t) = e, or the residual only contains noise with no contribution left from the image. It should however be noted that there is no need for Jacobian symmetry, or the assumptions needed in the original RED Romano et al. (2016) since gradient of is naturally ( (note the stopped gradient operation sg). Having said that, there are fundamental differences with RED including the generative nature of diffusion prior, and the fact that we use the entire diffusion trajectory for regularization Nonetheless, we believe this is an important connection to leverage RED utilities for improved sampling of diffusion models in inverse problems. It is also worth commenting that the earlier work by Reehorst & Schniter (2018) also draws connections between RED and score matching based on a single (deterministic) denoiser.
4.3 WEIGHTING MECIIANISM
In principle, timestep weighting plays a key role in training diffusion models. Different timesteps are responsible for generating different structures ranging from large-scale content in the last timesteps to fine-scale details the earlier timesteps (Choi et al., 2022). For effective regularization, is thus critical to properly tune the denoiser weights {A.) in our Algorithm | We observed that the regularization term in Eq. is sensitive to noise schedule. For example, in the variance-preserving scenario, it drastically blows up as t approaches zero.
To mitigate the regularization sensitivity to weights, it is more desirable to define the regularization in the signal domain, that is compatible with the fitting term as
My - f()) + A(sgl - TT.) μ.
(10)
where is a hyperparameter that balances between the prior and likelihood and jag is the MMSE
predictor of clean data. Here, we want the constant À to control the trade-off between bias (fit to
observations) and variance (fit to prior). order to come up with the interpretable loss in equation 10 one needs to rescale the noise residual term ea ( ()
Recall that the denoiser at time observes He
arTio + o,e. MMSE estimator also provides denoising
1 - (x. orea(a)()) ()).
A. E[u[az]
(11)
ai
Thus, one can show that
# レー = (o/a)))(()((
where we define SNR, a or the signal-to-noise ratio. Accordingly, by choosing X, = A/SNR, we can simply convert the noise prediction formulation in equation 9 to clean data formulation in equation 10

---

Published as a conference paper at ICLR 2024
DDRM Figure 2: Comparison of the proposed variational sampler with alternatives for inpainting representative ImageNet examples. Each sampler tuned for the best performance.
Input
RED-Diff ПIDM
DPS
EXPERIMENTS
In this section, we compare our proposed variational approach, termed RED-diff, against recent state-of-the-ar techniques for solving inverse problems on different image restoration tasks. For the prior, we adopt publicly available checkpoints from the guided diffusion mode that is pretrained based on 256 x 256 ImageNet (Russakovsky et al. 2015);; see details in the appendix. We consider the unconditional version. For the proof concept, we report findings for various linear and nonlinear image restoration tasks for 1k subset of ImageNet (Russakovsky et al. 2015) validation datase Due to space limitation, we defer more elaborate experiments and ablations to the appendix. Next, we aim to address the following important questions:
How does the proposed variational sampler (RED-diff) compare with state-of-the-aa art meth- ods such as DPS, IIGDM, and DDRM in terms of quality and speed?
What a proper sampling strategy and weight-tuning mechanism for the variational sampler?
Sampling setup. We adopt linear schedule for 3e from 0.0001 to 0.02 for 1, 0.000 timesteps. For simplicity, always use uniform spacing when we iterate the timestep. For our variational sampler we adopt Adam optimizer with 1, 000 steps, and set the momentum pair (0.9, 0.99) and initial learning rate 0. 1. No weight decay regularization is used. The optimizer is initialized with the degraded image input. We also choose descending time stepping from T to as demonstrated by the ablations later in Section [.3.2 Across all methods, we also use batch size of 10 using RTX 6000 Ada GPU with 48GB RAM.
Comparison. For comparison we choose state-of-the-art techniques including DPS (Chung al. 2022a), IIGDM (Song et al. 2023), and DDRM (Kawar et al., 2022a) as existing alternatives for sampling inverse problems. We tune the hyper- parerr-prameters follows:
DPS (Chung et al. 2022a): 1, 0000 diffusion steps, tuned 0.5 for the best performance:
1IGDM (Song al.
2023): ) 100 diffusion steps, tuned 1) 1.0, and observed that HIGDM
does perform worse for 1000 steps;
-DDRM (Kawar et al.) 2022a):: tested for both 20 and 100 steps, and set 7) 0.85, 77 1.0.
DDRM is originally optimized for 20 steps.
For evaluation, we report metrics including Kernel Inception Distance (KID, (Binkowski et al. 201 LPIPS, SSIM, PSNR, and top-1 classifier accuracy of pre-trained ResNet50 model (He et al. 2015).
5.1
IMAGE INPAINTING
For inpainting evaluation, adopt 1k samples from the ImageNet dataset and random masks from Palette (Saharia et al., (022). We tune À = 0.25 for the SNR- -based denoiser weight tuning discussed in Section 4.3 A few representative examples
shown in Fig. For fair comparison, we choose relatively hard example in the first row, and an easier one in the bottom row. It is evident that RED-diff identifies the context, and adds the missing content with fine details. JIGDM however
httpss//qithub..comm// /quieeaiiiiuiiii ded- / ffusion
https///bit..ly/valllpppp
7

---

Published as a conference paper at ICLR 2024
Table 1: Performance of different samplers for ImageNet inpainting using pretrained unconditional guided diffusion model. For RED-diff we set 0.5. For time per step (for each sample) we use the maximum batch size that fits the GPU memory. All methods run on a single NVIDIA RTX 6000 Ada GPU with 48GB RAM.
Sampler
PSNR(dB) 1
SSIM
KID
LPIPS
top-1 [
time per step (sec) 1
max batch size
DPS
21.27
0.67
15.28
0.26
58.2
0.23
15
HIGDM
20.30
0.82
4.50
0.12
67.8
0.24
DDRM
20.72
0.83
2.5
0.14
68.6
0.1
25
RED-diff
23.29
0.87
0.86
0.1
72.0
0.05
30
Table 2: Performance of different samplers for nonlinear tasks based on ImageNet data.
Task
Phase Retrieval
HDR
Deblurring
Metrics
DPS
RED-diff
DPS
RED-diff
DPS
RED-diff
PSNR(dB) f
9.99
10.53
7.94
25.23
6,4
45.00
SSIM
0.12
0.17
0.21
0.79
0.19
0.987
KID
93.2
114.0
272.5
1.2
342.0
0.1
LPIPS
0.66
0.6
0.72
0.1
0.73
0.0012
top-1
1.5
7.2
4.0
68.5
6.4
75.4
fails to inpaint the hard example, and DPS and DDRM inpaint blurry contents. More examples are provided in the appendix.
Quantitative results are also listed in T One can see that RED-diff consistently outperforms the alternative samplers across all metrics such KID and PSNR with significant margin. This indicates not only more faithful restoration by RED-diff but also better perceptual quality images compared with alternative samplers.
Finally, note that RED-diff iterations are quite lightweight with only forward passing to the diffusion score network. In contrast, DPS and IIGDM require score network inversion by differentiating through the diffusion denoisers. This in turn is source of instability and renders the steps compu- tationally expensive. Likewise, DDRM involves SVD calculations that are costly. We empirically validate these by comparing the time per-step and GPU memory usage in Table [
5.2 NONLINEAR INVERSE PROBLEMS
For various nonlinear inverse problems we assess RED-diff on ImageNet data. We choose DPS as the baseline since IIGDM and DDRM only deal with linear inverse problems.
High dynamic range (HDR). We choose the nolinear HDR task as candidate to verify RED-diff. HDR performs the clipping function fzx clip(2z, 11 1) on the normalized RGB pixels. Again, we choose À = 0.25 and Ir 0.5, and 100 steps. For DPS we choose Ç; = 0.1 search over the nominator. While RED-diff converges to good solutions, DPS struggles to find decent solution even after tuming. The metrics listed under Table demonstrate the gap.
y A()o(,;)]] after grid
Phase retrieval. We test on phase retrieval task as well. The task deals with reconstructing the phase from only magnitude observations in the Fourier domain. is a difficult task especially for ImageNet the weight À 0.25 and dataset with diverse details and structures. Again, for RED-diff While both methods face with 0.4
Ir = 0.5, while for DPS we optimize the step size Ç; yy (x::((
challenges for recovering faithful images, RED-diff performs better and achieves higher scores for most of the metrics; see Table 2 Note that, phase retrieval from arbitrary measurements known to be a challenging task. Thus, for a better assessment of RED-diff one can use supported measurement models (e.g.. to Gaussian or coded diffraction patterns) that lead to better quality Metzler et al. (2018). That would however need separate study that we leave for future research.
We also test another nonlinear scenario that deals with nonlinear deblurring. adopt the same setup in DPS (Chung et al., 2022a) with the blur kernel adopted from a pretrained UNet. For RED-diff we choose 0.25 and Ir 0.5. For DPS also after grid search over the coefficients DPS struggles for this nonlinear tasks. In general, DPS is
Deblurring,
we end up with A
1.0
8

---

Published as a conference paper at ICLR 2024
KIC
PSNR
denoberr meightt
10'
Uier
apttllinne square(Liser
10t2
corat
800
600
timestep
400
200
10g
welght-ft1/SNii
sqrt
Square
Figure 3: Ablation for denoiser weight tuning. Left: denoiser weight over timesteps (reversed); right: KID and PSNR vs. for different monotonic functions of inverse SNR.
sensitive to step size and initialization, while RED-diff is not sensitive and achieves much better scores as listed in Table 2
5.3 ABLATIONS
We provide ablations to verify the role of different design components the proposed algorithm such as denoiser weight tuning, and sampling strategy.
5.3.1 DENOISER WEIGHTING MECHANISM
As discussed in Section 4.2 the variational sampler resembles regularization by the denoising diffusion process. When sampling in descending order, namely from / = T to = 1, each denoiser regularizes different structures from high-level semantics to low-level fine details, respectively. To effect prior at different image scales, each denoiser needs to be tuned properly. We proposed inverse SNR (i.e., 1/SNR,) as the base weight per timestep in Section 4.3 To validate that choice, we ablate different monotonic functions of SNR to weight denoisers over time. The weights are plotted in Fig. (left) over timesteps. The corresponding KID and PSNR metrics are also shown in Fig. (right) for Platte inpainting for different weighting mechanisms. It is observed that the square root decay (1/VSNR) and linear schedule (1/SNR) are the best strategies for KID and PSNR, respectively.
5.3.2 TIMESTEP SAMPLING
We consider five different strategies when sampling the timestep t during optimization, namely: (1) random sampling: (2) ascending; (3) descending; (4) min-batch random sampling; and (5) mini-batch descending sampling. We adopt Adam optimizer with 1, 000 steps and choose the linear weighting mechanism with X 0.25. Random sampling (1) uniformly selects a timestep t [1, 77, while ascending and descending sampling are ordered over timesteps. is seen that descending sampling performs significantly better than others. It starts from the denoiser at time t = T, adding semantic structures initially, and then fine details are gradually added in the process. This appears generate images with high fidelity and perceptual quality. We also tested batch sampling with 25 denoisers per iteration, for 40 iterations. It is observed that batch sampling smears the fine texture details. See appendix for more details.
CONCLUSIONS AND LIMITATIONS
This paper focuses on the universal sampling of inverse problems based on diffusion priors. introduces a variational sampler, termed RED-difff that naturally promotes regularization by the denoising diffusion process (DDP). Denoisers at different steps of DDP concurrently impose structural constraints from high-level semantics to low-level fine details. To properly tune the regularization, propose a weighting mechanism based on denoising SNR. Our novel perspective views sampling as stochastic optimization that embraces off off-thh-sheeff optimizers for efficient and tunable sampling. Our experiments for several image restoration tasks exhibit the strong performance of RED-diff compared with state-of-the-art alternatives for inverse problems.
One of the limitations of our variational sampler pertains to the lack of diversity. It mode-seeking in nature and promotes MAP solutions. We will investigate methods that encourage diversity e.g., by tuning the optimizer, introducing more expressively in the variational distribution, or modifying the criterion by adding dispersion terms in Stein variational gradient descent Liu & Wang (2016). Additionally. more extensive experiments for 3D generation tasks will solidify the merits of our variational sampler.

---

Published as a conference paper at ICLR 2024
REFERENCES
Amr Ahmed, Moahmed Aly, Joseph Gonzalez, Shravan Narayanamurthy, and Alexander Smola. Scalable inference in latent variable models. In Proceedings of the fifth ACM international conference on Web search and data mining, pp. 123-132, 2012.
Mikotaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv:1801.01 401, 2018.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 1125188)839-877,, 2017.
Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends@ in Machine learning, 3(1):1 2011.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. ILVR: Con- ditioning method for denoising diffusion probabilistic models, arXiv preprint arXiv:2108.0200 02.00298, August 2021.
Jooyoung Choi, Jungbeom Lee, Chachun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11472-11481 2022.
Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh International Conference on Learning Representations, 2022a.
Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. Advances in Neural Information Processing Systems, 35:25683-25696, 2022b.
Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Come--loser-ddffuse-fassss Accelerating conditional diffusion models for inverse problems through stochastic contraction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12413-12422, 2022c.
Hyungjin Chung, Jcongsol Kim, Schui Kim, and Jong Chul Ye. Parallel diffuusio models of operator and image for blind inverse problems. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6059-6069, 2023a.
Hyungjin Chung, Dohoon Ryu, Michael T McCann, Marc Klasky, and Jong Chul Ye. Solving 3d inverse problems using pre-trained 2d diffusion models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 22542-22551, 2023b.
Regev Cohen, Michael Elad, and Peyman Milanfar. Regularization by denoising via fixed fixed-point projection (red-pro). SIAM Journal Imaging Sciences, 14(3): 2021.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794 2021.
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. GENIE: Higher- Denoising Diffusion Solvers. In Advances in Neural Information Processing Systems, 2022.
Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as plug-and-play priors. arXiv preprini arXiv:2206.0900 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03338 December 2015.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. ç. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Jonathan Ho. Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv preprint arXiv:220..03558 2022.
10

---

Published as a conference paper at ICLR 2024
Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. Journal of Machine Learning Research, 2013.
Samuel Hurault, Arthur Leclaire, and Nicolas Papadakis. Gradient step denoiser for convergent plug-and arXiv preprint arXiv:2110.03220, 2021.
Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jonathan I Tamir. Robust compressed sensing mri with deep generative priors. Advances in Neural Information Processing Systems, 2021.
Zahra Kadkhodaie and Eero Simoncelli. Stochastic solutions for linear inverse problems using the prior implicit in denoiser. Advances in Neural Information Processing Systems, 34: 14:13222-13254 2021.
Ulugbek S Kamilov, Charles Bouman, Gregery T Buzzard, and Brendt Wohlberg, Plug-and-play methods for integrating physical and learned models in computational imaging: Theory, algorithms, and applications. IEEE Signal Processing Magazine, 40(1):85-977 2023.
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Proc. NeurIPS, 2022.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. arXiv preprint arXiv:2201. 1793, 2022a.
Bahjat Kawar, Jiaming Song, Stefano Ermon, and Michael Elad. Jpeg artifact correction using denoising diffusion restoration models. arXiv preprint arXiv:2209.11888, 2022b.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 1rXiv:1312.6114, 2013.
Florian Knoll, Kristian Bredies, Thomas Pock, and Rudolf Stollberger. Second order total generalized variation (tgv) for mri Magnetic resonance in medicine, 65(2):480-491, 2011.
Erich Kobler, Teresa Klatzer, Kerstin Hammernik, and Thomas Pock. Variational networks: con- necting variational methods and deep learning. In Pattern Recognition: 39th German Conference, GCPR 2017, Basel, Switzerland, September 12-15, 2017, Proceedings 39, pp. 281-293 Springer, 2017.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.0977 2020.
Rémi Laumont, Valentin De Bortoli, Andrés Almansa, Julie Delon, Alain Durmus, and Marcelo Pereyra. Bayesian imaging using plug & play priors: when langevin meets tweedie. SIAM Journal on Imaging Sciences, 15(2):701-737 2022.
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances neural information processing systems, 29, 2016.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around steps. arXiv preprint wrXiv:2206.00922 2022.
Christopher Metzler, Phillip Schniter, Ashok Veeraraghavan, and Richard Baraniuk. prdeep: Robust phase retrieval with flexible deep network. In International Conference Machine Learning, pp. 3501-3510. PMLR, 2018
Frank Ong, Shreyas Amin, Shreyas Vasanawala, and Michael Lustig. mridata. org:: An open archive for sharing mri raw data. In Proc. Intl. Soc. Mag. Reson. Med, volume 26, 2018.
Batu Ozturkler, Morteza Mardani, Arash Vahdat, Jan Kautz, and John M Pauly. Regularization by denoising diffusion process for mri reconstruction. In Medical Imaging with Deep Learning. short paper track, 2023.
11

---

Published as a conference paper at ICLR 2024
Jean-Chine Chn-Chhistoppe Pesquet, Audrey Repetti, Matthieu Terris, and Yves Wiaux. Learning maximally monotone operators for image recovery. SIAM Journal on Imaging Sciences, 14(3):1206-133 1237, 2021.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
Javier Portilla, Vasily Strela, Martin J Wainwright, and Eero P Simoncelli. Image denoising using scale mixtures of gaussians in the wavelet domain. IEEE Transactions on Image processing. 12
(tl::133--355/1 2003.
Edward T Reehorst and Philip Schniter. Regularization by denoising: Clarifications and new interpretations. IEEE transactions on computational imaging, 5(1):52-67, 2018.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference machine learning, pp. 1286. PMLR, 2014.
Yaniv Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by denoising (RED). arXiv preprint arXiv:1611.02862 November 2016.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg. and Fei-Fei. ImageNet large scale visual recognition challenge. International journal of computer vision, 115(3):211-22 doi: 10.1007/s 12600015-0--------
December 2015. ISSN 0920-5691, 1573-1405.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. ACM SIGGRAPHI 2022 Conference Proceedings, pp. 1-10, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International Conference on Machine Learning, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guidded diffusion models for inverse problems. International Conference on Learning Representations, 2023.
Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 34:1415-14288 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-basee generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021b.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-basee generative modeling in latent space. In Neural Information Processing Systems (NeurIPS), 2021.
Singanallur V Venkatakrishnan, Charles Bouman, and Brendt Wohlberg. Plug-and-play priors for model based reconstruction. In 2013 IEEE Global Conference on Signal and Information Processing, pp- 945-948. IEEE, 2013.
Paseal Vincent. A connection between score matching and denoising autoencoders. Neural computa- tion, 2367):16611167, 2011.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao. Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelit and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305. /62/3, 2023.
12

---

Published as a conference paper at ICLR 2024
Kaixuan Wei, Angelica Aviles-Rivens Jingwei Liang, Ying Fu, Hua Huang, and Carola-Bibiana Schönlieb. Tipnp: Tuning-free free blug-and-play proximal algorithms with applications to inverse imaging problems. The Journal of Machine Learning Research, 23(1):699-746, 2022.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion GANs. In International Conference on Learning Representations (ICLR), 2022.
Dejia Xu, Yifan Jiang, Peihao Wang. Zhiwen Fan, Yi Wang, and Zhangyang Wang. Neurallift- 360: Lifting an in-the-wild 2d photo to 3d object with 360 (Adeg] views. arXiv preprint arXiv:2211. 16431, 2022.
Xiaojian Xu, Yu Sun, Jiaming Liu, Brendt Wohlberg, and Ulugbek S Kamilov. Provable convergence of plug-and-play priors with mmse denoisers. IEEE Signal Processing Letters, 27:1280-1288 2020.
Jure Zbontar, Florian Knoll, Aditya Sriram, Chloe Murrell, Steen Moeller, Riccardo Lattanzi, Peter Batchelor, Michael T. McCann, Diederik P. Kingma, Manoj Mani, al. fastmri: An open dataset and benchmarks for accelerated mri. arXiv preprint
2018.
Kai Zhang, Yawei Li, Wangmeng Zuo, Lei Zhang, Luc Van Gool, and Radu Timofte. Blug-and-pas play image restoration with deep denoiser prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, (4(10:6660066) 6376, 2021.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204.13900 2022.
Zhizhuo Zhou and Shubham Tulsiani. Sparsefusion: Distilling view-conditioned diffusion for 3d reconstruction. arXiv preprint arXiv:22 2022.
Yuanzhi Zhu, Kai Zhang, Jingyun Liang, Jiezhang Cao, Bihan Wen, Radu Timofte, and Luc Van Gool. Denoising diffusion models for plug-an blug-and-ple and-play image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1219-1229 2023.
13

---

Published as a conference paper at ICLR 2024
A PROOFS AND DERIVATIONS
A. PROOF OF PROPOSITION
As discussed in section 4, using Bayes rule, one can re-write the KL objective in equation 5 as
([(g(zul))lppzaall -=Eq(=olv) [ogp(y)z))) (((((ol))))p(( + p(y)
term(()
term
For minimization purposes, term(ii) is constant, and we can ignore it. The term(i) however has also two parts. The first part is simply the reconstruction loss. Based on the measurement model equation 22 of the main paper, since we assumed noise is i...d Gaussian, the first part simply derived as
E (ogssl)
IF pooc f(xr)) [3] 2x?
Using Theorem in Song et al. (2221a) assuming that the score is learned exactly, namely eo of v. log p(z)), under some mild assumptions on the growth of log q(2 ly) and p(z,) at infinity, we have
KL(g(zol[\lp(xo)) = [-' 3(t) E ol v)
vx, log g(<el)) Va, logp(xz)) dt
(12)
over the denoising diffusion trajectory {xr} for positive values (B(t)). This essentially implies that weighted motee-mat over the continuous denoising diffusion trajectory is equal to the KL divergence. In practice, however, we are often interested a reweighting of r.h.s. in Eq. [2 that leads other divergence measures (Saug al. (022aa).
A.2 PROOF OF PROPOSITION 2
The regularization term is essentially the score matching loss in the r.h.s. of equation 12 In practice, we often use a weighting scheme different than 3(1)/2 that corresponds to maximum likelihood estimation [Song et al. (2021a). For re- re-weighting, it is useful to recognize the following Lemma, which leverages integration by part, with the complete proof provided in Song et al. (2021a).
Lemma 1. The dinedverivative the KL divergence at timestep t obeys
d.K ((((selw)lll()) dt
((1) E (g(()l)) (w)
V.r, log g(x+()) Vz, log
(13)
This Lemma is intuitive based on equation 12 Now, to allow a generic weighting, one can simply weight equation 13 with wt).. Then, under the condition (0)) = 0, the r.h.s. of equation 12 can be written (see e.g.. Song al. (2021a))
B(t) 9h 2
T
V log (<<el))) Va, logr(z;))*] Jd
(1) E ((g (x)
T w(t)
dk I (KL(g(rlw))lp(()) dt
dt
(() + S. ((KKl((()bblbl)t))))
w E(t)) log ((<rl))) p(xx)
where w'()) = de(()) db
The equality (a) holds because (=)) K (qg (())))))))))))))))))))))))
is zero at / =
and = T. This is because w(t) = 0 by assumption at = 0, and By becomes a pure Gaussian noise at the end of the diffusion process which makes p(xT) q (y) and thus K (1((7r ()((r 0.
14

---

Published as a conference paper at ICLR 2024
For simplicity, let us suppose 0 in the variational distribution, and thus μ is deter- ministic. The forward diffusion process then 2ς = axj + dGe for e NO,, 1). Applying the ce-parameterization trick, the gradient w.r. i be simply written as
dz du
V (ege(()) = 20th
w' d'
(V log q(((()))))) - V.. log (ogp(2)))
dt
J0
P = 20 w' w'tt) W()))E- (- so(zzit))) (- ( (az1)
de
where g<z.lv) N(auu, and V., (e (2, a/H)/o7 -e/oz- Note, the gradient was exchanged with the expectation since both gradient terms exist and are bounded. Finally, can rearrange terms to arrive at the compact form
V Vareg(u) = I (w((t
T
20 at E E. -))
T
= -Ba-4
for A. (o Note that we can ignore the second term inside the expectation since has a zero mean.
A.3 ADDING DISPERSION TO VARIATIONAL APPROXIMATION
The results in Proposition 2 presented for Dirac distribution with dispersion, namely g( ly) 6(zg 14). We however can easily extend those to optimize for Gaussian dispersion as well. The interesting observation that the gradient w.r.t. the mean μ remains the same, and the gradient w.r.t. is simple and tractable.
In this case, we have q(Foly) N(,,o11) The diffusion signal at timestep can then be represented a compact form using reparameterization trick as Dt deftt + Vařa2 + ofe. Let's define :- (1 + o²( (2(e))) )2)1/2 so that d2 = atp + moc. Note, for no dispersion case 1. Then, gradient w.r.t. the mean is obtained as
w (()) (Vz, log (F<l/)))) Va, log logp(z:)) T dIt de Jat du
V (reelu,
J0
Varo2 to, (- =o(veell))) {)(())((()))))))))))){) dt w (())Ee at co(zeit) - mi netedet
((t))B/a
ot
Ee, ( t)]
Similarly, the gradient w.r.t. the dispersion is found
dra dt
Voreg(p,o) = ((())_,Noo.)) (V log (x(o))))) Va, logp(ze))
do
w' d'(()))
e Va,o² + of - ) (=((artt))) claroz + -2)-1/220 jat
= a.E >:215 ( (eo(())) - ne n
The dispersion gradient has close form. However, believe the Gaussian dispersion is not the right choice to add stochasticity and diversity to natural images since simply perturbing an image with Gaussian noise does not lead another legitimate image. In essence, needs proper dispersion on the image manifold that needs more sophisticated dispersion models such as in Wang et al. (2023).
Finally, it is also useful to note that believe framework can be extended to more complex variational distributions such as hierarchical, multimodal, or normalizing flowbbased distributions as in VAEs, following the recipe in the previous work such Vahdat et al. (2021)
15

---

Published as a conference paper at ICLR 2024
input
RED-Diff
ΠGDM
DPS
DDRM
Figure 4: Comparison of the proposed variational sampler with alternatives for superresolution of representative ImageNet examples. Each sampler is tuned for the best performance.
Table Performance of different samplers for ImageNet 4x superresolution. We adopt unconditional guided diffusion model for the score function. We choose Adam Ir 0.25.
Sampler DPS
PSNR(dB)
SSIM
KID
LPIPS | top-1 1
24.83
0.71
10.01
0.16
71.5
IIGDM
25.25
0.73
10.9
0.15
71.02
DDRM
25.32
0.72
14.0
0.23
63.9
RED-diff
25.95
0.75
10.0
0.25
66.7
B ADDITIONAL EXPERIMENTS
B.I PRETRAINED DIFFUSION MODEL
We adopt the function from the pretrained guided diffusion model, which class conditioning. It is trained on 256 x 256 ImageNet dataset. Architecture details are listed in section of Dhariwal & Nichol (002))
B.2 IMAGE SUPERRESOLUTION
We perform 4x superresolution on 1k subset of ImageNet dataset. Bicubic degradation is applied to create the low-resolution inputs. After tuning À 0.25, Adam iterations are run for 1, 000 steps. few samples are illustrated in Fig. where one notice that RED-diff strikes good balance between image fidelity and perceptual quality. One can play with this trade-off by tuning the Adam learning rate. Smaller learning rates lead higher fidelity, while larger learning rates give rise to higher perceptual quality. See supplementary material for more examples.
Quantitative results are also listed Table One can see that RED-diff significantly outperforms the alternative samplers in terms of PSNR and SSIM. Table however indicates that RED-diff is not as good as other alternatives terms of the perceptual quality. We want to note the trade off we observe between fidelity (e.g., PSNR) and perceptual quality (e.g.. LPIPS). It seems that one can achieve better perceptual quality at the expense of lower fidelity by tuning the regurlization weight that control the bias- variance trade off. Other hyperparamaters such step size and number of steps can be tuned alternatively. Fig. 5 depicts the trade off between OSNR-LPIPS where the red-co red-cross denotes the point reported in Table
16

---

Published as a conference paper at ICLR 2024
0.27
67.25
0.24
86.50
6.22
0.21
Figure 5: LPIPS vs. PSNR (dB) trade-off for 4x superresolution.
B.3 MORE EXAMPLES FOR COMPARISONS
We provide additional examples compare RED-diff with alternative methods for inpainting and superresolution. The inpainting examples are shown in Fig. 6 Superresolution examples are also shown in Fig. For both tasks, the same setup discussed the main paper used for each scheme.
B.4 DIVERSITY
To verify the sample diversity for RED-diff, Fig. dillustrates different samples for the ImageNet inpainting task when the seed for e is changing. We choose Adam optimizer with Ir = 0.25, and 1, 000 steps. From the examples, we observe that RED-diff samples are sufficiently diverse. To further enhance the diversity, one can play with the optimizer parameters, for example by choosing larger learning rate for Adam, or using a smaller number steps.
B.5 DIFFUSION EVOLUTION
To understand the restoration process with diffusions, we plot the evolution of the diffusion model over timesteps in Fig. 9 We visualize fi as the outputs from every 100 steps over 1, 000 timesteps. generated, the
To gain further insights into the generation process and how the image structures evolution of diffusion steps is also plotted in the frequency domain. Fig. [0 illluuutttees the magnitude and phase every 100 steps over 000 timesteps. can be observed that the earlier denoisers add high frequency details, while the later denoisers generate low frequency structures.
C ADDITIONAL SCENARIOS
We include additional tasks to assess RED-diff for noisy inverse problems. We consider noisy inpainting and compressed sensing MRI.
C.I COMPRESSED SENSING MRI
We adopt the pretrained diffusion model from Jalal al. (2021), and sample via RED-Diff. The pretrained diffusion has been trained based on fastMRI brain data. We test sampling for both in- domain brain, and out-of-domain knee data. The performance compared with CSGM-Langevin Jalal et al. (202T) for which the codebase is publicly available, and serves as state-of-the-as for complex- -alpue medical image reconstruction. We use the multi-coil fastMRI brain dataset /honnar et al. 2018) with 1D equispaced undersampling, and the fully-sampled 3D fast-spin echo multi-coil knee MRI dataset from Ong et al. (2018) with 2D Poisson Disc undersampling mask, as in Jalal al. (2021). We used 6 validation volumes for fastMRI, and 3 volumes for Mridata by selecting 32 middle slices from each volume. We use exactly the same tuning parameters as for the inpainting task. The results for different undersampling rates (R) shown in Table 4 Obviously, Riff outperforms CSGM method (with Langevin sampling) consistently in terms of PSNR, that is the measure of interest in MRI reconstruction. For more details see Ozturkler et al. (023). Representative
17

---

Published as a conference paper at ICLR 2024
input
RED-Diff
NGDM
DPS
DDRM
Figure 6: Comparison of the proposed variational sampler with alternatives for inpainting representa- tive ImageNet examples. Each sampler is tuned for the best performance.
18

---

Published as a conference paper at ICLR 2024
input
RED-Diff
ПGDM
DPS
DDRM
Figure 7: Comparison of the proposed variational sampler with alternatives for superresolution of representative ImageNet examples. Each sampler is tuned for the best performance.
19

---

Published as a conference paper at ICLR 2024
11
1
1
1
Figure 8: Sampling diversity for ImageNet inpainting. A few random realizations are shown, where the first column is the masked input image, and the last one is the reference image.
Figure 9: Evolution of RED-diff over iterations. Descending sampling direction is used. Denoisers from large to small i restore high-level to low-level features, respectively. Adam with 1, 000 steps used and every 100 iterations are visualized.
Figure 10: Evolution of RED-diff over iterations in the frequency domain. Middle row shows the log-magnitude, and the bottom row shows the phase.
20

---

Published as a conference paper at ICLR 2024
Zero-Filled
CSGM-Langevin
RED-diff
Ground Truth
ISNE: 23A2
PONE: EX16
ISNE: 26.5
SOIME
PSAE: 2225 555
PSME 2LS3
Figure 11: Representative reconstructed images for brain with R 4, and knee with R 12.
Table 4: Compressed sensing MRI PSNR (dB) for fastMRI brain and Mridata knee dataset with Rx undersampling. CSGM is SOTA method for complex-valued MRI with public codebase.
Anatomy
Brain
Knee
time
Sampler
R 4
R =
12
R
16
(sec/iter)
CSGM
36.3
31.4
31.8
0.344
RED-diff
37.1
33.2
32.7
0.114
samples are also shown in Fig. 11 At this point, it is also worth noting that RED-diff relates to other variational approaches in the imaging literature such as Knoll et al. (201I); Portilla et al. (2003); Kobler et al. (2017). A through comparison is needed and that is left for future research.
C.2 NOISY INPAINTING
To see the effects of measurement noise RED-diff performance, we add Gaussian noise with
o, = 0.1 to the masked Palette images for inpainting. We compare RED-diff with DPS and PGDM, where for all we use 100 steps. For RED-diff we choose À 0.25 similar all other scenarios with 0.5, and the step size run Ç 0.5/|ly A (x:) adopted from Chung et al. (022aa) where we run a grid search over the range [0, 1] for the coeffficient. Table shows the metrics. It seen that RED-diff outperforms DPS and PiGDM in most metrics.
Ir 0.25. For DPS, we choose
Table 5: Noisy inpainting with oy 0.1 for Palette data with 100 steps.
Sampler
PSNR(dB)
SSIM
KID
LPIPS
top-1
DPS
16.98
0.39
1.9
0.52
8.8
IIGDM
16.15
0.31
3.6
0,45
42.4
RED-diff
18.92
0.41
3.4
0.4
43.3
21

---

Published as a conference paper at ICLR 2024
8000
7000
loss noise (blue), signal (red)
6000
5000
4000
3000
2000
200
400
600
800
1000
step
Figure 12: Evolution of signal residual [ 1112 (red) and noise residual (xx;; t) - (blue) over time for ImageNet inpainting when equally weighting diffusion denoisers different timesteps.
D ABLATIONS
D.I DENOISER WEIGITIING MECHANISM
It was discussed in section 4.3 that the noise residual blows up the last timesteps of the diffusion process. Here, we illustrate that in Fig. 12 when denoisers are equally weighted. It shows the magnitude of both the signal and noise residuals. It is clearly seen that the noise residual goes up drastically at around t 1, 000. Similarly, the signal residual also blows up at carlier steps. This suggests mechanism that guarantees relatively small signal residual all iterations that led the SNR ruler for weighting the denoisers.
D.2 OPTIMIZING RED-DIFF FOR MORE EPOCHS
Since RED-diff is an optimization- sampling, one may ponder that using more iterations by going over the denoisers more than once can improve the sample quality. To test this idea, we choose different epochs for ImageNet inpainting, when we use Adam optimizer with 1, 000 steps per epoch.
few representative examples are shown Fig. 13
It is observed that adding more epochs has
negligible improvement on the sample quality.
D.3 OPTIMIZATION STRATEGY
The proposed variational sampler relies on optimization for sampling. To see the role of optimizer, first ablate SGD versus Adam. We found SGD more sensitive step size which in turn demands more careful tuning. We tune the hyperparameters in each case for the best performance for ImageNet inpainting with 0.25. We use Adam with parameters discussed in the beginning of Section 5 Fig. 14 shows representative inpainting examples, where SGD is tested with momentum (0.9) and without momentum. appears that SGD with momentum can as good as the Adam optimizer, which indicates that RED-diff is not sensitive to the choice of optimizer. Our future direction will explore accelerated optimization for faster convergence.
For the Adam optimizer, also ablated the initial learning rate to see its effect the quality of the generated samples. The results are depicted in Fig. 15 (right). It appears that tuning Adam learning rate is important, where smaller learning rate (e.g.. 0.05) leads better reconstruction quality since the optimizer can better converge to the (optimal) MAP estimate. However, using a larger learning rate (e.g., 0.5) leads to better perceptual quality measured with KID.
22

---

Published as a conference paper at ICLR 2024
21
2
2
Figure 13: RED-diff outputs for a different number of epochs. From left to right: input, epochs=1,2.10,
and the reference, respectively. Adding more epochs does not make any noticeable difference.
2
2
input
Adam
SGD
SGD with momentum
reference
momentum
Ablation of optimization strategy for an inpainting example from ImageNet dataset. Adam and SGD (with momentum 0.9) seem to perform similarly.
Figure 14:
KID for range of À is also depicted in Fig. 15 (left). It is evident that there is an optimal X, which confirms that with proper tuning of the bias Maasaeenance trade-off one can gauge the sampling quality.
D.4 TIMESTEP SAMPLING STRATEGY
We ablate the timestep sampling strategy and the number of sampling steps. For comparison, we consider five different strategies when sampling from during optimization. This includes (1) random sampling: (2) ascending; (3) descending; (4) min-batch random sampling; and (5) mini-batch descending sampling. We adopt Adam optimizer with 1, 000 steps and choose the linear weighting mechanism with À 0.25. Random sampling (1) uniformly selects timestep & 7T, while ascending and descending sampling are ordered over timesteps. As shown in the inpainting example in Fig, [6] it is seen that descending sampling performs significantly better than others. It starts from the denoiser at time T, adding semantic structures initially, and then fine details are gradually added in the process. This appears to generate images with high fidelity and perceptual quality.
23

---

Published as a conference paper at ICLR 2024
20.0
2.8
17.5
2.7
15.0
2.6
12.5
2.5
10.0
7.5
2.3
5.0
2.2
2.5
2.1
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
lambda
Figure 15: KID versus the optimizer learning rate (right), and the weight (left).
The aforementioned sampling strategies choose a single denoiser at time, but one may ponder what if one performs batch sampling regularize based multiple denoisers at the same time. This is also computationally appealing as it can be executed in parallel based on our optimization framework. To test this idea, sort 1, 000 time steps in descending order, and iteration, for 40 iterations. It is again observed from Fig. 16 that batch sampling smears the fine texture details. This needs further investigation find out the proper weighting mechanism that enables parallel sampling. In conclusion, descending sampling with a single (or a few) denoiser at time seems to the best sampling strategy. More details are found in the supplementary material.
batch of 25 denoisers per
Table 6: Performance of RED-diff for inpainting under different step counts, when À 0.25 and Ir = 0.25.
steps
PSNR(dB)
SSIM
KID
LPIPS
top-1
10
18.83
0.75
22.05
0.21
59.4
100
23.13
0.87
1.93
0.12
70.8
1000
23.86
0.88
2.17
0.11
69.8
It is also useful to understand how many steps RED-diff needs generate good samples. To this end, we evaluated ImageNet inpainting for a different number of steps Table 6 One can observe that with 100 steps the best perceptual quality is achieved where KID=1.93. With more steps, the optimizer tends to refine the fidelity so a better reconstruction PSNR achieved. This ablation suggests that not many steps needed for the variational sampler. This is in contrast with DreamFusion Poole et al. (202)) which uses large number of íterations (15k) to generate samples.
D.5 CONNECTION AND DIFFERENCES WITH RED
We want to further elaborate the connections with the RED framework. It is useful to recognize that the classical RED adds no noise the input for denoising, and it is simply fixed-point problem. RED also uses a single deterministic denoiser. RED-diff however fundamentally different. generative and add noise to the input of all denoisers in the diffusion trajectory. As result stochastically navigates towards the prior. This seems crucial to find plausible solution. To highlight the
of the strength of the RED-diff regulrization over RED we performed two simple experiments. first experiment, we removed noise from the input of all denosiers, namely xe = μt, and then ran RED-diff iterations for the image inpainting task. It appears that iterations do not progress and they get stuck around the initial masked image seen in Fig. 17 In the second experiment, we just used a single denoiser at time t O, which has a very small noise, and thus resembles RED the most. We observe again that the solution gets stuck and cannot navigate the real region of prior fill out the masked areas the image. We tested for single denoiser picked from other time steps such as t 500, and observed the same behavior.
24

---

Published as a conference paper at ICLR 2024
m Inpul
M
descending
ascencing
random
mini-butch descending
mini-butch random
reference
Figure 16: Ablation of sampling strategy for ImageNet inpainting. For mini-hatch sampling, the batch size is 25. Sampling in descending direction performs significantly better than the rest. Adam optimizer is used with an unconditional ImageNet guided diffusion score.
Figure 17: Evolution of diffusion when the noise each denoiser is removed and x1. /4.
D.6 COMPARISON WITH PLUG-AND-PLAN (PNP) METHODS
Plug-and-Play (PnP) methods
important class of techniques for solving inverse problems;
e.g., Kamilov et al. (2023); Zhang et al. (2021 ? Wei et al. (2022); Laumont et al. (2022); Hurault al. (2021): Xu et (2020); Pesquet al. (2021) }. PnP and RED methods are indeed related and connected through fixed-point iterations Cohen et al. (2021). It thus deserves to compare RED-din with PnP framework. To demonstrate the benefits RED-dif with respect to PnP methods we compare with two representatives namely DPIR Zhang et al. (202T) and DifiPIR (Zhu et al. (2023). DPIR integrates deep denoiser prior into PnP methods. essence, it first trains a CNN for denosing. It then plugs the deep denoiser prior modular part into half quadratic splitting iterative algorithm for solving inverse problems.
We consider the inpainting task and use the code from public repository DPIR with IRCNN denoiser. To be consistent with the experiments the paper, we use ImageNet data and random Palette masks as described in section 5.1 of the main paper. For 10 random samples, compare PSNR and LPIPS in Table and show representative samples in Fig. [8] is evident that RED-diff achieves a much better quality images thanks to the diffusion prior. Note that DPIR purely deterministic method.
For completeness, we also compare with DiffPIR Zhu et al. (023). It worth noting that the DiffPIR paper Zhhu et al. (2023) has appeared after initial arXiv submission. For DifPPIR adopt the code from the public repository with unconditional guided diffusion score function. The results again shown in Table 7 and representative samples Fig. 18 RED-diff performs slightly better than DifIPIR Zhu et al. 2023). One however should note that RED-diff provides rigorous treament from the maximum lihold perspective (and MAP estimator for no dispersion case) as it directly minimizes the KL divergence, and as result it better lends itself optimization and tuning. In essence, the optimization step in Algorithm can replaced with any off-the-shelf optimizer. With the SNR weighting mechanism described in section 4.3, we observe better quality images compared with DifiPIR; see Fig and Table 7 A more through analysis and comparison left for future research.
"https//gituu..co/cce// bttt:://gthhaa.cc//yyaaaa//bbbuu
25

---

Published as a conference paper at ICLR 2024
Table 7: Comparison with plug-and-play prior methods for image inpainting for samples from Imagenet data.
RED-diff and DiffPIR use 100 diffusion timesteps and both use unconditional guided diffusion score function.
Sampler
PSNR(dB)
SSIM
KID 1
LPIPS
top-1
DPIR
16.50
0.792
42.62
0.237
60.0
DifIPIR
27.29
0.8728
7.96
0.1401
70.0
RED-diff
26.06
0.8792
7.92
0.1060
90.0
input
RED-Diff
DPIR
DiffPIR
Figure 18: Comparison between RED-diff and soug-plaa-methodde namely DPIR and DiffPIR for Palette inpainting.
D.7 EXTENSION TO OBSERVATIONS WITH POISSON NOISE
We used Gaussian noise for the observation model in 2 to simplify the exposition. essence, the variational formulation is flexible with the choice of the observation noise as it eventually reflects in the reconstruction loss in namely - -i w) log p(y| xo)]. Here, we extend the reconstruction loss to Poisson noise distribution, namely y ~ Poisson(A f(xo)), ). with discrete and positive values such photon counts in imaging. Suppose that [f(m]]; for all j. Then, assuming i.i.d. Poisson noise.
" [f(x)]] exp(l-
ply]a0)=
(14)
As result
(ogl(y)) Σ yj log log[s(<r)); [f(a))] log yz!
(15)
Assuming Dirac distribution for q(zolv) = N(H,a²) with o 0, the reconstruction loss admits simple form
E (rolo)) log p(y)2()))) = Σ y log[((]]())] + [f(n]]s + log
(16)
where the last term in constant. With this loss expression, one can apply (sub)gradient based algorithms to solve RED-diff. is important to recognize that contrast with DPS, there no need for intractable likelihood log p( 2z). As result, there is need to for Gaussian approximation of Poisson distribution that limits DPS to work only when the signal intensity is large. This indeed shows another benefit of our variational approximation over DPS.
26

---

Published as a conference paper at ICLR 2024
Figure 19: RED-diff under different timestepping strategies. From left to right: timestepping schedule, masked image input, linear timestepping, logarithmic timestepping, and exponential timestepping output.
D.8 TIMESTEPPING EXPLORATION
Since the ascending sampling works best among the sampling strategies in Fig. xx, we tested ascending sampling with nonuniform timestepping see if can reduce the number of steps. We test logarithmic and exponential timestepping. For the discrete timesteps E [1, 1, 000). we choose 100 timsteps with logarithmic and exponential spacing: see Fig. 19 (left). We test RED-diff for inpainting task. The sample results are shown in Fig. T9 and compered with uniform spacing. It appears that uniform spacing works the best, and then the logarithmic spacing. This indicates that it is important to sufficiently sample low noise denoisers to recover fine details.
D.9 CLARIFICATION ON THE PROS AND CONS OF VARIATIONAL APPROXIMATION
As it was stated in the main paper, we introduced variational approximation to address the caveats of PGDM and DPS that approximates the likelihood with Gaussian distribution around the MMSE estimate. We should note that our variational method is also unimodal approximation, but it approximates the posterior with Gaussian distribution. The benefits of posterior approximation are however threefold: (i) it requires score Jacobian as opposed to DPS and PGDM that need costly and unstable score network inversion; (ii) due to optimization nature,
less sensitive (and more interpretable) to hyperparameter tuning as opposed to DPS; (fi1) it is flexible about the observation noise that appears in the reconstruction loss Eg(z w/ [- log p(y) mg)]] For instance it can casily handle Poisson noise
well as Gaussian noise discussed in section D.8. This in contrast with DPS that needs to approximate the likelihood log p(yl (.)) and thus resorts to approximation which require strict assumptions. For instance, for Poisson noise, it requires the signal intensity to be large enough so Poisson is approximated with Gaussian distribution. Last but not least, our variational framework allows forming more expressive distributions for the posterior p( (e.g., hierarchical, multimodal, or normalizing flow based as done in Vahdat et al. (2021)). ). However, we leave this to future work.
27


Reviews:
[{"review_id": "4IJui4ehai", "reviewer": "Reviewer_xtWM", "summary": "This paper presents a novel stochastic adaptation of regularization by denoising (RED) using the denoising diffusion model as an image prior. The study demonstrates the derivation of the score matching loss under the variational inference framework, which naturally leads to the formulation of stochastic RED. By implementing an appropriate weighting scheme, the proposed variational sampler surpasses existing diffusion-based posterior samplers in numerous inverse imaging problems, exhibiting superior reconstruction faithfulness (measured by PSNR/SSIM) and enhanced visual quality (evaluated via KID/LPIPS).", "questions": "See above\n\n---\nAfter rebuttal: I'm generally satisfied with authors' response, overall I think it's a good paper that can be accepted to ICLR.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly written and comprehensible to readers. Additionally, the formulation of the proposed variational sampler, utilizing a first-order stochastic optimizer (Adam) to address the stochastic RED objective, is both innovative and straightforward, yielding effective results.", "weaknesses": "Despite the paper's clarity, several imprecise arguments and overstatements necessitate revision and clarification:\n\n- The authors claim that a key advantage of their method is the circumvention of unimodal estimation employed in prior works. However, their variational inference approach introduces another level of approximation by using a simple unimodal Gaussian to approximate the complex multi-modal conditional posterior p(x0​∣y). This complicates the justification of the proposed framework's improvement over mitigating posterior score approximation in prior methods (Contribution 1).\n- The claim that the RED framework fundamentally differs from the Plug-and-Play (PnP) framework lacks rigor. The explicit regularization term in RED only exists under certain strict assumptions, which are often not applicable to modern deep learning-based denoisers. Thus, both RED and PnP should be explained from a fixed-point iteration perspective, which provides a unified treatment with theoretical convergence guarantees [A].\n- The proposed weighting mechanism aligns with the parameter setting strategy in the PnP framework [B] (as detailed in Section 4.2). Moreover, the authors overlook discussing several related works in PnP-type literature [C, D, E], which essentially represent the deterministic counterparts of diffusion-based posterior sampling.\n- The experimental comparison with DPS lacks sufficiency to verify the effectiveness of the proposed approach. Benchmarking against the state-of-the-art PnP approach [B] would provide a more comprehensive characterization of the method's advantages. Additionally, the performance in phase retrieval, particularly the PSNR, is too weak to be considered meaningful. To enhance credibility, it is recommended to consider adopting more realistic settings, such as coded diffraction patterns or oversampled Fourier measurements, as in [F].\n\n**References:**\n\n[A] Regev Cohen, Michael Elad, and Peyman Milanfar. \"Regularization by denoising via fixed-point projection (red-pro).\" SIAM Journal on Imaging Sciences, 14(3):1374–1406, 2021.  \n[B] Zhang, Kai, et al. \"Plug-and-play image restoration with deep denoiser prior.\" IEEE Transactions on Pattern Analysis and Machine Intelligence 44.10 (2021): 6360-6376.  \n[C] Kamilov, Ulugbek S., et al. \"Plug-and-play methods for integrating physical and learned models in computational imaging: Theory, algorithms, and applications.\" IEEE Signal Processing Magazine 40.1 (2023): 85-97.  \n[D] Wei, Kaixuan, et al. \"TFPNP: Tuning-free plug-and-play proximal algorithms with applications to inverse imaging problems.\" The Journal of Machine Learning Research 23.1 (2022): 699-746.  \n[E] Laumont, Rémi, et al. \"Bayesian imaging using plug & play priors: when langevin meets tweedie.\" SIAM Journal on Imaging Sciences 15.2 (2022): 701-737.  \n[F] Metzler, Christopher, et al. \"prDeep: Robust phase retrieval with a flexible deep network.\" International Conference on Machine Learning. PMLR, 2018.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cqmfk39hOo", "reviewer": "Reviewer_Qren", "summary": "This paper presents a sampling process based on diffusion models for solving inverse problems. This approach allows to tackle very general inverse problems in images like inpainting, deblurring, super-resolution... The method itself is straightforward when using a pretrained network and the experiments show promising results.", "questions": "I reformulate one of my previous remarks as a question: what happens with wrong or estimated values of the noise variance?\n\nMy next question is also a remark. The authors compare different strategy for the sampling (i.e. the timestep). The conclusion is that the descending sampling leads to the best results. This is normal as there is a time dependency in the sampler since the previous mu is used for the loss. Thus, I would expect the random sampling to fail. So the discussion on this point is weak and I would expect other descending strategy (log, exp...). Would it be a way to reduce the number of timesteps?\n\nLast point, please update the references. For example most of the paper from Chung et al are proceedings of conferences and no just papers on ArXiv.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method takes the form of a sampling algorithm that can solve a large number of inverse problems with additive Gaussian noise. \nThe framework is easy to setup when using pre-trained diffusion models and the theory gives access to formula for the hyperparameters. All in all the experiments show the benefit of this approach compare to similar frameworks.", "weaknesses": "The method is only for Gaussian noise while DPS, for example, can handle Poisson noise (though not in a low regime). Also the hyperparameter formula asks for the variance of the noise. Such quantity may be unknown, it would been interesting to see, experimentally, what happens with wrong or estimated values of the noise variance. Finally, like all methods based on diffusion models, it asks for an appropriate model (pre-trained or not) and solving the inverse problems remains computationally heavy with thousand of iterations to produce a result.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZlrGUT7e6X", "reviewer": "Reviewer_RLDf", "summary": "In this paper, the authors propose a new regularization strategy for solving inverse problems with a denoising prior based on diffusion models. The regularization is reminiscent to RED but uses denoisers at different noise levels. Compared to competitive methods, DiffRED does not assume the posterior $p(x_0|x_t)$ of the diffusion model to be unimodal. Experiments on various inverse problems prove the efficiency of the algorithm.", "questions": "- The authors argue that the advantage of the paper is to avoid the assumption $p(x_0|x_t)$ unimodal. Instead, they take the assumption\n$q(x_0|y)$ unimodal. Can the authors explain why this assumption would be more valid ?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Overall, the paper is well written, the motivation is clear and the method is original. The main strength of the paper is its experimental study. The authors experimented on very diverse inverse problems and perform an exhaustive ablation study. They clearly demonstrate that their algorithm performs state-of-the art performance and gives impressive visual results. Concerning the method, the proposed regularization in (8) is new and original, and the Proposition 2 makes this regularization term very promising.", "weaknesses": "My main concerns are on the theoretical side. The following aspects require further details. \n\n- You introduce KL minimization in (5) which makes sense for sampling from $p(x_0|y)$. However, the Gaussian approximating posterior $q(x_0|y)$ is then used with $\\sigma = 0$. In this case, the minimization of the KL comes back to the MAP problem $argmin_\\mu p(\\mu|y) = argmin_\\mu p(y|\\mu) - \\log p(\\mu) $. Therefore, the objective of this work is then to solve the MAP. This is a classical link between sampling and optimization.  From this observation, assuming Proposition 1 is true, by identification in the case $\\sigma = 0$, we get that $-\\log p(\\mu)$ is equal to the second term in (8). Is this really true ? In general, can the authors explain why they first consider a sampling approach before taking $\\sigma = 0$ ? \n- The proof of Proposition 1 uses a result from (Song et. al , 2021) that seems different (inequality and not equality). As this is the base of the theoretical analysis, I would like to see an exhaustive proof of the result. \n- The proposed regularization is defined with an expectation on $t$, but in practice $t$ is not chosen at random but with a predefined descending scheme. Therefore, the algorithm and the regularization used in practice is different from the ones originally introduced. This is for me the main weakness of the paper. \n- Proposition 2 implies that the right term in the equation is a conservative vector field, and in particular that it has symmetric Jacobian. This looks surprising to me without further assumptions on $\\epsilon_\\theta$. Refer to (Reehorst & Shniter, 2018) for the same observation on RED. Do you have an argument for justifying symmetry of the Jacobian ?\n- The algorithm 1 is proposed without any convergence analysis. \n- Before (8) it should be made clear that you assume exact approximation of the score with $\\epsilon_\\theta$ to get (8).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c1R2AZM1RT", "reviewer": "Reviewer_5Auf", "summary": "In this paper, the authors propose a novel diffusion sampling algorithm incorporating a data-fidelity term, allowing to solve inverse imaging problems with pretrained diffusion models. Interestingly, the authors claim that this novel sampling scheme can be related to a variational formulation, as opposed to traditional diffusion algorithms. This aspect is all the more interesting as it clearly relates the method to traditional, conventional variational methods for solving inverse problems. Eventually, the authors demonstrate its effectiveness on several inverse imaging problems, ranging from image inpainting to MRI imaging.", "questions": "I've enjoyed reading this paper and I thank the authors for their interesting work. I have however some strong feelings regarding how general imaging methods are treated in the paper - and in particular how their methods relates to more general variational methods. Please find below a list of comments.\n\n**Major comments**\n1. Pure variational methods are simply completely absent of this paper. Not only is this problematic with respect to the conveyed message (\"the proposed method links with variational methods\") but it is also detrimental to the experiments - for instance top row of Figure 11, a standard TV method would certainly reach very high PSNR. In the current form, I disagree that the paper links with variational methods; the paper links with RED approaches (which are very specific). Here are some more general references that may be of interest: [1,2,3].\n2. The RED approach plays an important role in this paper, but the authors do not mention an important aspect of RED: the loss that it derives is incorrect. This is explicitely stated in the Reehorst and Schnitter reference cited by the authors. An important point raised there is that without symmetry of the Jacobian, there is no hope that the denoiser approximates a gradient. This is well known in the PnP community which has led to significant works [4,5,6].\n3. Talking about fixed point after (9) is very misleading. In general, a fixed point is derived from the 0 of a subdifferential (or equivalently in the convex case, of the minimum of a convex function). In the case proposed by the authors, the loss varies with each iterate. How can there be a fixed point? (by the way, \"the fixed point\" strongly suggests a relationship to a convex function, switching to \"a fixed point\" would be more appropriate).\n4. While the context on diffusion models is clear and the literature review is appropriate, this cannot be said on more general inverse imaging techniques. The authors do not seem to be aware of state-of-the-art PnP algorithms such as DPIR [7], and more importantly, to methods mixing PnP and diffusion techniques, such as DiffPIR [8]. I think comparing with at least one PnP would be interesting (e.g. DPIR, or since a strong emphasis is put on it, RED).\n5. In my opinion, a key contribution of the paper is equation (8). An interesting point there is that the loss function is on $\\mu$, which relates to Minimum Mean Square Error (MMSE) approaches in imaging. This is an important point as most traditional methods in imaging follow a Maximum a Posteriori (MAP) approach. In fact, it is often believed that MMSE approaches are better than MAP approaches, but unfortuntely too costly and/or difficult to implement. A comment on that would be very necessary and interesting to the imaging community readers.\n6. The presence of $\\epsilon$ in the equation of Proposition 2 is surprising given the remark made at the end of the proof in the appendix (just before A.3). Why keep it in the equation?\n7. Maybe the authors could merge the results from A.2 and A.3 in Proposition 2 with additional notations regrouping the different constants. In fact, A.3 is an interesting result that is a bit hidden in the paper at the moment.\n\n**Minor comments**\n1. Small typos are remaining here and there. For instance: \"stropped\", \"bellows up\", \"approaches to zero; .\" (point after ;)...\n2. \"An inverse problem is often formulated as\" --> \"Inverse problems can be formulated as\"\n\n**References**\n\n[1] Knoll F, Bredies K, Pock T, Stollberger R. Second order total generalized variation (TGV) for MRI. Magn Reson Med. 2011\n\n[2] Portilla J, Strela V, Wainwright MJ, Simoncelli EP. Image denoising using scale mixtures of Gaussians in the wavelet domain. IEEE Transactions on Image processing. 2003\n\n[3] Kobler E, Klatzer T, Hammernik K, Pock T. Variational networks: connecting variational methods and deep learning. InPattern Recognition: 39th German Conference, GCPR 2017, Basel, Switzerland, September 12–15, 2017.\n\n[4] Hurault, Samuel, Arthur Leclaire, and Nicolas Papadakis. \"Gradient Step Denoiser for convergent Plug-and-Play.\" International Conference on Learning Representations. 2021.\n\n[5] Xu X, Sun Y, Liu J, Wohlberg B, Kamilov US. Provable convergence of plug-and-play priors with MMSE denoisers. IEEE Signal Processing Letters. 2020\n\n[6] Pesquet JC, Repetti A, Terris M, Wiaux Y. Learning maximally monotone operators for image recovery. SIAM Journal on Imaging Sciences. 2021\n\n[7] Zhang K, Li Y, Zuo W, Zhang L, Van Gool L, Timofte R. Plug-and-play image restoration with deep denoiser prior. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2021\n\n[8] Zhu Y, Zhang K, Liang J, Cao J, Wen B, Timofte R, Van Gool L. Denoising Diffusion Models for Plug-and-Play Image Restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2023", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper is overall well written and easy to follow, making it comfortable to read. The authors do not engage in unnecessary technicalities and provide the required background in a succint and clear manner.\n- The proposed method is novel (to the best of my knowledge) and simple to implement.\n- Relating the proposed method to the RED approach is very interesting and is a valuable contribution to the community.\n- Experimental results are convincing.", "weaknesses": "Despite the many strong points raised above, this paper suffers from important drawbacks. \n\n- Important references from the variational [1,2,3] and PnP [4,5,6] litterature are missing. Overall, while the context within diffusion models is well set, the link with the general imaging litterature is completely absent, making it difficult for the reader to relate to other imaging techniques.\n- Theoretically, the fact that the Jacobian of the network is not symmetric (a priori) prevents the authors from making a link with a variational loss (see the Reehorst reference cited by the authors), questioning the full approach.\n- Comparisons with DiffPIR [8] or RED are missing.\n\n**References**\n\n[8] Zhu Y, Zhang K, Liang J, Cao J, Wen B, Timofte R, Van Gool L. Denoising Diffusion Models for Plug-and-Play Image Restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)