Denoising Diffusion Restoration Models
Authors: Bahjat Kawar
Michael Elad
Stefano Ermon
Jiaming Song
Venue: NeurIPS 2022 Accept (2022)

Abstract: Many interesting tasks in image restoration can be cast as linear inverse problems. A recent family of approaches for solving these problems uses stochastic algorithms that sample from the posterior distribution of natural images given the measurements. However, efficient solutions often require problem-specific supervised training to model the posterior, whereas unsupervised methods that are not problem-specific typically rely on inefficient iterative methods. This work addresses these issues by introducing Denoising Diffusion Restoration Models (DDRM), an efficient, unsupervised posterior sampling method. Motivated by variational inference, DDRM takes advantage of a pre-trained denoising diffusion generative model for solving any linear inverse problem. We demonstrate DDRM's versatility on several image datasets for super-resolution, deblurring, inpainting, and colorization under various amounts of measurement noise. DDRM outperforms the current leading unsupervised methods on the diverse ImageNet dataset in reconstruction quality, perceptual quality, and runtime, being $5\times$ faster than the nearest competitor. DDRM also generalizes well for natural images out of the distribution of the observed ImageNet training set.

Denoising Diffusion Restoration Models
Babjat Kawar
Michael Elad Department of Computer Science Technion, Haifa, Israel elad@cs. technion ac il
Department of Computer Science Technion, Haifa, Israel bahjat kawarccs. technien ac
Stefano Ermon
Jiaming Song NVIDIA Santa Clara, California, USA j jmmingstnvidda..cmm
Department of Computer Science Stanford, California, USA ermonêcs. stanford. edu
Abstract
Many interesting tasks in image restoration can be cast as linear inverse prob- lems. recent family of approaches for solving these problems uses stochastic algorithms that sample from the posterior distribution of natural images given the measurements. However, efficient solutions often require problem-specific supervised training model the posterior, whereas unsupervised methods that are not sroblee- typically rely on inefficient iterative methods. This work addresses these issues by introducing Denoising Diffusion Restoration Models (DDRM), an efficient, unsupervised posterior sampling method. Motivated by variational inference, DDRM takes advantage of pre-trained denoising diffusion generative model for solving any linear inverse problem. We demonstrate DDRM's versatility several image datasets for super-resolution, deblurring. inpainting. and colorization under various amounts of measurement noise. DDRM outper- forms the current leading unsupervised methods on the diverse ImageNet dataset in reconstruction quality, perceptual quality, and runtime, being 5x faster than the nearest competitor. DDRM also generalizes well for natural images of the distribution of the observed ImageNet training set.'
Introduction
Many problems in image processing, including super-resolution [31, 17], deblurring [28, 48], inpaint- ing [55], colorization [29, 58], and compressive sensing [1], are instances of linear inverse problems, where the goal is to recover an image from potentially noisy measurements given through a known linear degradation model. For specific degradation model, image restoration can be addressed through end-to-end to-end supervised training of neural networks, using pairs of original and degraded im- ages [14, 58, 41]. However, real-world applications such as medical imaging often require flexibility to cope with multiple, possibly infinite, degradation models [46]. Here, unsupervised approaches based on learned priors [36], where the degradation model is only known and used during inference, may be more desirable since they can adapt to the given problem without re-training [51]. By learning sound assumptions over the underlying structure of images (e.g., priors, proximal operators or denoisers), unsupervised approaches can achieve effective restoration without training specific degradation models [51, 40].
Under this unsupervised setting, priors based on deep neural networks have demonstrated impressive empirical results in various image restoration tasks [40, 50, 43, 38, 15]. To recover the signal,
Project website: https: betpp://ddrrmmlll miittmmmmm.iitth.iii..
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

4X
(b) Deblurring (Noisy with dy 0.1)
8X
Largm ipsum
consectefull
eluseand bar
delore maa
16x
mnaim vo voalam
(c) Inpainting (Noisy with oy = 0.1)
Noiseless
Noisy with By 0.1
(a) Super-resolution
(d) Colorization (Noisy with dy 0.1)
Figure 1: Pairs of measurements and recovered images with a 20-step DDRM on super-resolution, deblurring, inpainting, and colorization, with or without noise, and with unconditional generative models. The images are not accessed during training.
most existing methods obtain a prior-related term over the signal from a neural network (e.g.. the distribution of natural images), and likelihood term from the degradation model. They combine the two terms to form a posterior over the signal, and the inverse problem can be posed as solving optimization problem (e.g., maximum posteriori [8, 40]) solving sampling problem (e.g.. posterior sampling [2, 3, 25)). Then, these problems are often solved with iterative methods, such as gradient descent Langevin dynamics, which may be demanding in computation and sensitive hyperparameter tuning. An extreme example is found in [30] where a "fast" version of the algorithm uses 15, 000 neural function evaluations (NFEs).
Inspired by this unsupervised line of work, we introduce an efficient approach named Denoising Diffusion Restoration Models (DDRM), that achieve competitive results in low 20 NFEs. DDRM denoising diffusion generative model [44, 19, 45] that gradually and stochastically denoises sample to the desired output, conditioned the measurements and the inverse problem. This way we introduce a variational inference objective for learning the posterior distribution of the inverse problem hand. We then show its equivalence the objective of unconditional denoising diffusion generative model [19]. which enables us to deploy such models DDRM for various linear inverse problems (see Figure 2). To our best knowledge, DDRM is the first general sampling-based based inverse problem solver that can efficiently produce range of high-quality, diverse, yet valid solutions for general content images.
demonstrate the empirical effectiveness DDRM by comparing with various competitive methods based on learned priors, such Deep Generative Prior (DGP) [38], SNIPS [25], and Regularization by Denoising (RED) [40]. On ImageNet examples, DDRM mostly outperforms the neural network baselines under noiseless super-resolution and deblurring measured in PSNR and KID [5], and is at least 50 more efficient in terms of NFEs when is second-best. Our advantage becomes even larger when measurement noise is involved, noisy artifacts produced by iterative methods do not appear in our case. Over various real-world images, we further show DDRM results on super-resolution, deblurring, inpainting and colorization (see Figure A DDRM trained on ImageNet also works on images that are out of its training set distribution (see Figure 6).
Background
Linear Inverse Problems. A general linear inverse problem is posed as
y Hx
where we aim to recover the signal x E R from measurements E RM where H E R is a known linear degradation matrix, and ~ Ao0 of an i.i.d. additive Gaussian noise with known variance. The underlying structure of x can be represented via generative model, denoted

---

PolXn, X1, X2, X2)
Po Xg. X1, Ng.X2. Xa
X3
Use pre-traines models linear inverse problems
x Xs
g(x X2. X3
X
X3
X2
X
Denoising Diffusion Probabilistic Models (Independent of inverse problem)
Denoising Diffusion Restoration Models (Dependent inverse problem)
Figure Illustration of our DDRM method for specific inverse problem (super-resolution + denoising). We can use unsupervised DDPM models as good solution to the DDRM objective.
as (x).. Given y and H, a posterior over the signal can be posed as: (x y) с (x()) xx), where the "likelihood" term p(y|x is defined via Equation (I); such approach leverages learned prior pe pe(x). and we call it an "unsupervised" approach based on the terminology in [36]. as the prior does not necessarily depend on the inverse problem. Recovering x can be done by sampling from this posterior [2], which may require many iterations to produce a good sample. Alternatively, one can also approximate this posterior by learning model via amortized inference (i.e., supervised learning); the model learns to predict x given y. generated from and specific H. While this can be more efficient than sampling-ben based methods, it may generalize poorly to inverse problems that have not been trained on.
Denoising Diffusion Probabilistic Models. Structures learned by generative models have been applied to various inverse problems and often outperform data-independent structural constraints such sparsity [7]. These generative models learn model distribution pe (x) that approximates data distribution g(x) from samples. In particular, diffusion models have demonstrated impressive unconditional generative modeling performance on images
[13]. Diffusion models are generative X1 + Xo (where x; E IR"), which has
models with Markov chain structure - XT-1 the following joint distribution:
T-1
Po (Xp:r) = P (T)
(xT)
II t=
Pe (x(x++))
After drawing Xp:T, only X6 is kept as the sample of the generative model. To train diffusion model, fixed, factorized variational inference distribution is introduced:
T-1
((x,r|x0)) = q. (x xx)
II
19 (xx(xe+1)x0))
1=
which leads to an evidence lower bound (ELBO) on the maximum likelihood objective [44]. A
(t) and g chosen as conditional Gaussian distributions for all & < T,, and that g(x; Xx) is also Gaussian with known mean and covariance, i.e., x; can be treated X, directly corrupted with Gaussian noise. Thus, the ELBO objective can be reduced into the following denoising autoencoder objective (please refer to [45] for derivations):
special property of some diffusion models is that both Pa
N 1B 7x lx)) [|
(xx))2
where f(() is 0- e-parameterized neural network that aims recover noiseless observation from noisy x,, and "n.r are a set of positive coefficients that depend on ((x1:T|X))
Denoising Diffusion Restoration Models
Inverse problem solvers based on posterior sampling often face a dilemma: unsupervised approaches apply to general problems but are inefficient, whereas supervised ones are efficient but can only address specific problems.

---

To solve this dilemma, we introduce Denoising Diffusion Restoration Models (DDRM), an unsuper- vised solver for general linear inverse problems, capable of handling such tasks with or without noise in the measurements. DDRM is efficient and exhibits competitive performance compared to popular unsupervised solvers [40, 38, 25].
The key idea behind DDRM is to find an unsupervised solution that also suits supervised learning objectives. First, we describe the variational objective for DDRM over a specific inverse problem (Section 3.1). Next, we introduce specific forms of DDRM that are suitable for linear inverse problems and allow pre-trained unconditional and class condiiconiiii diffusion models to be used directly (Sections 3.2, 3.3). Finally, we discuss practical algorithms that are compute and memory efficient (Sections 3.4. 3.5).
3.1
Variational Objective for DDRM
For any linear inverse problem, we define DDRM Markov chain XT -+ XT-1 -x Xo conditioned on y, where
T-1
(x(x::lly) Pg (xTly) ly)
(T)
II 1=0
Pa
(2) (xt (xx(Xx++1,))
and Xg is the final diffiusion output. In order to perform inference, we consider the following factorized variational distribution conditioned on
((xTT Xo. y) ((t) Xx y) II (x
t=
leading to ELBO objective for diffusion models conditioned on (details in Appendix A).
In the remainder of the section, we construct suitable variational problems given H and σy and connect them to unconditional diffusion generative models. To simplify notations, we will construct the variational distribution q such that q(x, (xo) - N(x, of I) for noise levels 0 op < 0i < 02 < < or. In Appendix B, we will show that this is equivalent to the distribution introduced in DDPM [19] and DDIM [45], up to fixed linear transformations over Xt.
3.2 A Diffusson Process for Image Restoration
Similar to SNIPS [25], consider the singular value decomposition (SVD) of H, and perform the diffusion in its spectral space. The idea behind this is to tie the noise present in the measurements with the diffusion noise X1:T, ensuring that the diffusion result xo is faithful to the measurements. By using the SVD, we identify the data from x that is missing in y, and synthesize it using a diffusion process. In conjunction, the noisy data in y undergoes denoising process. For example, inpainting with noise (e.g., H = diag([..... 1, O, 0)), σy 2 0), the spectral space is simply the pixel space, so the model should generate the missing pixels and denoise the observed ones y. For general linear H, its SVD is given as
=
where U E mm V Rax orthogonal matrices, and Σ E Rx is a rectangular diagonal matrix containing the singular values of H, ordered descendingly. As this is the case in most useful degradation models, assume < nt, but our method would work for >> n well. We denote the singular values as 81 2 82 2 2 8m, and define 8; = 0 for E [m + 1,
We use the shorthand notations for values in the spectral space: x(") is the i-th index of the vector X; VT Xx, and (1))) is the i-th index of the vector y EtUT y (where i denotes the Moore-Penross pseudo-inverse). Because V is an orthogonal matrix, we can recover x, from x; exactly by left
This called "Variance Exploding" in [47]. TTis called "Variance Preserving" [47].

---

multiplying V. For each index in Xg, we define the variational distribution as:
[N(5(() .0
if >
(3) [TT
xo. y)
(4)
[N(x)) (0) .
=
N(Xx (1)
v1 11-17am 17
x x
07)
if
N(X + /1 -1 90 01l1
(x) [X++1 Xo.y)
0y/3, )
if o < 2y
(5)
N((1 )x)x00 + nby
if 0 2 dy
where v) E (0, 1] is hyperparameter controlling the variance of the transitions, and r) and t/b may depend on oz, Si, Qy. We further assume that dr 2 0y/s; i for all positive s;."
In the following statement, we show that this construction has the "Gaussian marginals" property similar to the inference distribution used in unconditional diffusion models [19].
Proposition 3.1. The conditional distributions q(t) defined in Equations 4 and 5 satisfy the following:
g(xe|xo) = (x0,,o())
(6)
defined by marginalizing over xy (for all t' > t) and y, where ((y|x)) defined as in Equation (1) with Xg-
We place the proof in Appendix C. Intuitively, our construction considers different cases for each index of the spectral space. (i) If the corresponding singular value is zero, then y does not directly provide any information to that index, and the update is similar to regular unconditional generation. (it) If the singular value non-zero, then the updates consider the information provided by y. which further depends on whether the measurements' noise level in the spectral space (ay/st /8)) is larger than the noise level in the diffusion model (a²)) or not; the measurements in the spectral space (+() are then scaled differently for these two cases in order to ensure Proposition 3.1 holds.
Now that we have defined (*) as series of Gaussian conditionals, we define our model distribution pe as series of Gaussian conditionals as well. Similar to DDPM, we aim to obtain predictions of xo the symbol xa, represent this prediction made by model SS (x++1, / + 1) R x IR- + IR" that takes in the sample X++1 and the conditioned time step (L + 1). We also define x as the i-th index of Xg. V Xa,t-
at every step t; and to simplify notations,
We define DDRM with trainable parameters 0 as follows:
( .o o
Pg (T)
if
yy)
(V(),
=
N(x (1)
/ 17
(1)) 8
s;
0e+1
1t) (x. (x+1,8) Po
N(x (1)
/1 17=
g(t)
-20
if
(8)
0
xx 2 m65 of N((1
(3)
if
(9) Compared to (9) in Equations (4) and (5), our definition of Pg merely replaces x (which we do not know sampling) with x (which depends our predicted x0,:) when f < T. and replaces (1)
X. with when = T. It is possible learn the variances [35] or consider alternative constructions where Proposition 3.1 holds; we leave these options as future work.
3.3 "Learning" Image Restoration Models
(t) and q(t) by choosing di. 7) and Th. we can learn model parameters 0 by Once we have defined Pe maximizing the resulting ELBO objective (in Appendix A). However, this approach is not desirable
"This assumption fair, as we can set a sufficiently large or- Equivalently, the authors of [19] predict the noise values to subtract in order recover X6.3-
5

---

Original
Original
NAAANAZ
Degroee
Degnated
reduce this complexity O(n) for CM RRER?
CROEZBUROERG
DDRM (20)
DDRM
(a) Inpainting results on cat images,
(b) Deblurring results (Øy 0.05) on bedroom images.
Figure 3: DDRM results bedroom and cat images, for inpainting and deblurring.
since we have to learn a different model for each inverse problem (given H and σy), which is not flexible enough for arbitrary inverse problems. Fortunately, this does not have to the case. In the following statement, we show that optimal solution DDPM DDIM can also be an optimal solution to a DDRM problem, under reasonable assumptions used prior work [19, 45].
Theorem 3.2. Assume that the models p(t) and f(()) do not have weight sharing whenever t + tt, 20 then when r = and d7 o. the ELBO objective of DDRM (details in Appendix A) can be rewritten in the form of the DDPM DDIM objective Equation (2).
We place the proof in Appendix C.
Even for different choices of r and 7/6, the proof shows that the DDRM objective is weighted of-squares error in the spectral space, and thus pre-trained DDPM models are good approximations the optimal solution. Therefore, we can apply the same diffusion model (unconditioned on the inverse problem) using the updates in Equation (7) and Equation (8) and only modify H and its SVD (U, Σ, V) for various linear inverse problems.
3.4 Accelerated Algorithms for DDRM
Typical diffusion models are trained with many timesteps (e.g.. 1000) achieve optimal unconditional image synthesis quality, but sampling speed is slow as many NFEs are required. Previous works [45, 13] have accelerated this process by "skipping" steps with appropriate update rules. This is also true for DDRM, since we can obtain the denoising autoencoder objective in Equation (2) for any choice of increasing dT..- For pre-trained diffusion model with T" timesteps, choose ou to be subset of the I" steps used in training.
3.5 Memory Efficient SVD
Our method, similar to SNIPS [25], utilizes the SVD the degradation operator H. This constitutes memory consumption bottleneck in both algorithms well as other methods such Plug and Play (PnP) [51], as storing the matrix V has a space complexity of (n²)) for signals of size n. By leveraging special properties of the matrices H used, denoising, inpainting, super resolution, deblurring, and colorization (details in Appendix D).
4 Related Work
Various deep learning solutions have been suggested for solving inverse problems under different settings (see a detailed survey [37]). We focus on the unsupervised setting, where we have access to dataset of clean images at training time, but the degradation model is known only at inference time. This setup inherently general to all linear inverse problems, a property desired in many real-world applications such as medical imaging [46, 20].

---

Table 1: Noiseless 4x super-resolution and deblurring results on ImageNet 1K (256 x 256).
4x super-resolution SSIM
Deblurring
Method
PSNR1
KID
NFEs
PSNR
SSIM
KID
NFEs
Baseline
25.65
0.71
44.90
19.26
0.48
38.00
DGP
23.06
0.56
21.22
1500
22.70
0.52
27.60
1500
RED
26.08
0.73
53.55
100
26.16
0.76
21.21
500
SNIPS
17.58
0.22
35.17
1000
34.32
0.87
0.49
1000
DDRM
26.55
0.72
7.22
20
35.64
0.95
0.71
20
DDRM-CC
26.55
0.74
6.56
20
35.65
0.96
0.70
20
Almost all unsupervised inverse problem solvers utilize a trained neural network in an iterative scheme. PnP, RED, and their successors [51, 40, 32, 49] apply a denoiser as part of an iterative optimization algorithm such steepest descent, fixed-point, or alternating direction method of multipliers (ADMM). OneNet [39] trained a network to directly learn the proximal operator of ADMM. A similar use of denoisers in different iterative algorithms is proposed in [34, 16, 30j. The authors of [43] leverages robust classifiers learned with additional class labels.
Another approach is to search the latent space of generative model for generated image that, when degraded. as close as possible to the given measurements. Multiple such methods were suggested, mainly focusing on generative adversarial networks (GANs) [7, 11, 33J. While they exhibit impressive results on images of specific class, most notably face images, these methods are not shown to be largely successful under more diverse dataset such as ImageNet [12]. Deep Generative Prior (DGP) mitigates this issue by optimizing the latent input as well as the weights of the GAN's generator [38].
More recently, denoising diffusion models were used solve inverse problems in both supervised (i.e., degradation model is known during training) [42, 41, 13, 10, 54] and unsupervised settings [22, 26, 25, 21, 46, 47, 9]. Unlike previous approaches, most inffusiionbaseed methods can successfully recover images from measurements with significant noise. However, these methods are very slow, often requiring hundreds or thousands of iterations, and are yet be proven on diverse datasets. Our method, motivated by variational inference, obtains problem-specific, non-equilibrium update rules that lead high-quality solutions in much fewer iterations.
ILVR [9] suggests diffusion-basee method that handles noiseless super-resolution. and can run in 250 steps. In Appendix H. we prove that when applied on the same underlying generative diffusion model, ILVR is special case of DDRM. Therefore, ILVR can be further accelerated to run in 20 steps, but unlike DDRM, it provides the authors of [22] suggest a score-based solver for inverse problems that can converge in small number of iterations, but does not handle noise in the measurements.
clear way of handling noise in the measurements. Similarly,
Experiments
5.1 Experimental Setup
We demonstrate our algorithm''s capabilities using the diffusion models from [19], which are trained on CelebA-HQ [23], LSUN bedrooms, and LSUN cats [56] (all 256 x 256 pixels). We test these models on images from FFHQ [24], and pictures from the internet of the considered LSUN category, respectively. In addition, we the models from [13], trained on the training set of ImageNet 256 x 256 and 512 x 512, and tested on the corresponding validation set. Some of the ImageNet models require class information. For these models, denote our algorithm as DDRM class conditional (DDRM-CC). In all experiments, we use 1) = 0.85, 7 1, and uniformly-space timestep schedule based on the 1000-step pre- trained models (more details in Appendix E). The number of NFEs (timesteps) is reported in each experiment.
the ground truth labels as input, and
In each of the inverse problems we show, pixel values are in the range [0, 1). and the degraded measurements are obtained as follows: (i) for super-resolution, we use block averaging filter to downscale the images by factor of 2, 4, or 8 in each axis; (ii) for deblurring, the images are blurred
7

---

Table 4x super resolution and deblurring results on ImageNet 1K (256 256). Input images have an additive noise of By 0.05.
xx super-resolution
Deblurring
Method
PSNR
SSIM
KIDI
NFEs|
PSNR
SSIM
KID
NFEs|
Baseline
22.55
0.46
67.86
18.35
0.20
75.50
0
DGP
20.69
0.43
42.17
1500
21.20
0.45
34.02
1500
RED
22.90
0.49
43.45
100
14.69
0.08
121.82
500
SNIPS
16.30
0.14
67.77
1000
16.37
0.14
77.96
1000
DDRM
25.21
0.66
12.43
20
25.45
0.66
15.24
20
DDRM-CC
25.22
0.67
10.82
20
25.46
0.67
13.49
20
Original
Low-res
DDRM (20)
SNIPS
RED
DGP
Figure 4: 4x noisy super resolution comparison with dy 0.05.
by 9 X 9 uniform kernel, and singular values below certain threshold are zeroed, making the problem more ill-posed. (iii) for colorization, the grayscale image is an average of the red, green, and blue channels of the original image; (iv) and for inpainting, text overlay or randomly drop 50% the pixels. Additive white Gaussian noise can optionally be added to the measurements in all inverse problems. We additionally conduct experiments bicubic super-resolution and deblurring with an anisotropic Gaussian kernel in Appendix
mask parts of the original image with
Our code is available https:// / h//ggtttub.... cwm/ -atrjjattawwwddrn
5.2 Quantitative Experiments
In order to quantify DDRM's performance, we focus the ImageNet dataset (256 x 256) for its diversity. For each experiment, we report the average peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) [52] measure faithfulness the original image, and the kernel Inception distance (KID) [5]. multiplied by 103, to measure the resulting image quality.
We compare DDRM (with 20 and 100 steps) with other unsupervised methods that work in reasonable time (requiring 1500 NFEs or less) and can operate on ImageNet. Namely, we compare with RED
'teapot'
"brown bear"
Original
Grayscale
Samples from DDRM-CC (100)
Figure 5: 512 x 512 ImageNet colorization. DDRM-CC produces various samples for multiple runs on the same input.

---

Original
Blurred
DDRM (20)
Figure 6: Results on 256 x 256 USC-SIPI images using an ImageNet model. Blurred images have noise of By = 0.01.
[40], DGP [38], and SNIPS [25]. The exact setup of each method detailed in Appendix F. We used the same hyperparameters for noisy and noiseless versions of the same problem for DGP, RED, and SNIPS, as tuning them for each version would compromise their unsupervised nature. Nevertheless, the performance of baselines like RED with such tuning does surpass that of DDRM, as show in Appendix F. In addition, we show upscaling by bicubic interpolation as a baseline for super-resolution, and the blurry image itself as a baseline for deblurring. OneNet [39] is not included in the comparisons as it is limited to images of size 64 x 64, and generalization to higher dimensions requires an improved network architecture.
We evaluate all methods on the problems of 4x super-resolution and deblurring, on one validation set image from each of the 1000 ImageNet classes, following [38]. Table shows that DDRM outperforms all baseline methods, in all metrics, and on both problems with only 20 steps. The only exception this is that SNIPS achieves better KID than DDRM in noiseless deblurring, but requires 50x more NFES to do so. Note that the runtime of all the tested methods is perfectly linear with NFEs, with negligible differences in time per iteration. DGP and DDRM-CC use ground-truth class labels for the test images to aid in the restoration process, and thus have an unfair advantage.
DDRM's 's appeal compared to previous methods becomes more substantial when significant noise added to the measurements. Under this setting, DGP, RED, and SNIPS all fail to produce viable results, evident in Table 2 and Figure 4. Since DDRM is fast, also evaluate it the entire ImageNet validation set in Appendix F.
5.3 Qualitative Experiments
DDRM produces high quality reconstructions across all the tested datasets and problems, as can be seen in Figures and 3, and in Appendix I. As it is a posterior sampling algorithm, DDRM can produce multiple outputs for the same input, as demonstrated in Figure Moreover, the unconditional ImageNet diffusion models can be used to solve inverse problems on out-of- distribuutiiiioonnn images with general content. In Figure 6, we show DDRM successfully restoring 256 256 images from USC-SIPI [53] that do not necessarily belong to any ImageNet class (more results in Appendix I).
6 Conclusions
We have introduced DDRM, a general sampling based linear inverse problem solver based on umcondtiional/ias-cooniiii
diffusion generative models as learned priors. Motivated by varia- tional inference, DDRM only requires few number of NFEs (e.g.. 20) compared to other sampling- based baselines (e.g., 1000 for SNIPS) and achieves scalability multiple useful scenarios, including denoising, super-resolution, deblurring, inpainting, and colorization. We demonstrate the empirical

---

successes of DDRM on various problems and datasets, including general natural images outside the distribution of the observed training set. To our best knowledge, DDRM is the first unsupervised method that effectively and efficiently samples from the posterior distribution of inverse problems with significant noise, and can work on natural images with general content.
In terms of future work, apart from further optimizing the timestep and variance schedules, it would be interesting to investigate the following: (7) applying DDRM non-linear inverse problems, (ii) addressing scenarios where the degradation operator is unknown, and (iii) self-supervised training techniques inspired by DDRM as well as ones used in supervised techniques [41] that further improve performance of unsupervised models for image restoration.
Acknowledgements
We thank Kristy Choi, Charlie Marx, and Avital Shafran for insightful discussions and feedback. This research was supported by NSF (#1651565 #1522054, #1 #1733686), ONR 1N00014-19-1-1144 AFOSR (FA9550- 19-1-0024), ARO (W91 WN91INF-111 1-NF-2211--025) Sloan Fellowship. Amazon AWS, Stan- ford Institute for Human-Centered Artificial Intelligence (HAI), Google Cloud, the Israel Science Foundation (ISF) under Grant 335/18, the Isracli Council For Higher Education Planning & Budget- ing Committee, and the Stephen A. Kreynes Fellowship.
References
[1] Richard G Baraniuk. Compressive sensing [lecture notes). IEEE signal processing magazine, 1444):118-221, 2007.
[2] Johnathan M Bardsley. Meme-based image reconstruction with uncertainty quantification.
SIAM Journal on Scientific Computing, 3413):A1316-A133, 2012.
[3] Johnathan M Bardsley, Antti Solonen, Heikki Haario, and Marko Laine. Randomize-then- optimize: A method for sampling from posterior distributions in nonlinear inverse problems. SIAM Journal on Scientific Computing, 36(4):A1895-AID) 2014.
[4] Christopher M Bishop. Pattern recognition. Machine learning, 128(9), 2006.
[5] Mikolaj Bićkowski, Danica J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations, 2018.
[6] Yochai Blau and Tomer Michaeli. The perception -intttionn tradeoff. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6228-6237 2018.
[7] Ashish Bora, Ajil Jalal, Eric Price, and Alexandros G. Dimakis. Compressed sensing using generative models. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 537-546, 2017.
[8] Daniela Calvetti and Erkki Somersalo. Hypermodels in the bayesian imaging framework. Inverse Problems, 24(3):034013, 2008
[9] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. llv:: Conditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2108.0 02938, 2021.
[10] Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Come-oloser- Accelerating conditional diffusion models for inverse problems through stochastic contraction. arXiv preprint arXiv:2/ 02.05/25544.4 2021.
[11] Giannis Daras, Joseph Dean, Ajil Jalal, and Alex Dimakis. Intermediate layer optimization for inverse problems using deep generative models. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 2421-2433 2021.
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large- scale hierarchical image database. In 2009 IEFE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
10

---

[13] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
[14] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. IEEE transactions on pattern analysis and machine intelligence, 38(2):295 307, 2015.
[15] Jinjin Gu, Yujun Shen, and Bolei Zhou. Image processing using multi-code gan prior. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3012-3021, 2020.
[16] Bichuan Guo, Yuxing Han, and Jiangtao Wen. Agem: Solving linear inverse problems via deep priors and sampling. Advances in Neural Information Processing Systems, 32, 2019.
[17] Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita. Deep back paojeprojection networks for super-resolution. In Proceedings of the IEEE conference computer vision and paltern recognition, pages 1664-1673, 2018.
[18] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. In Advances in Neural Information Processing Systems, volume 30, 2017.
[19] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pages 0840-6851, 2020.
[20] Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alex Dimakis, and Jonathan Tamir. Robust compressed sensing mri with deep generative priors. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
[21] Ajil Jalal, Sushrut Karmalkar, Alex Dimakis, and Eric Price. Instance-optimal compressed sensing via posterior sampling. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 4709-4720, 2021.
[22] Zahra Kadkhodaie and Eero Simoncelli. Stochastic solutions for linear inverse problems using the prior implicit in a denoiser. Advances Neural Information Processing Systems, 34, 2021.
[23] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representa- tions, 2018.
[24] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2019.
[25] Bahjat Kawar, Gregory Vaksman, and Michael Elad. SNIPS: Solving noisy inverse problems stochastically, In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
[26] Bahjat Kawar, Gregory Vaksman, and Michael Elad. Stochastic image denoising by sampling from the posterior distribution. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV) Workshops, pages 1866-1875, October 2021.
[27] Diederik P Kingma and Max Welling. Auto-Encoding variational bayes. arXiv preprint 1rXiv:1312.61 14v10, December 2013.
[28] Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. Deblurgan-v2: Deblur- ring (orders-of-magninuc) faster and better. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 0778-887,, 2019.
[29] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic colorization. In European conference on computer vision, pages 577-593. Springer, 2016.
[30] Rémi Laumont, Valentin De Bortoli, Andrés Almansa, Julie Delon, Alain Durmus, and Marcelo Pereyra. Bayesian imaging using plug play priors: When Langevin meets Tweedie. arXiv preprint arXiv:2103.04715, 2021.
11

---

[31] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken. Alykhan Tejani, Johannes Totz, Zehan Wang, al. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 4681-4690, 2017.
[32] Gary Mataev, Peyman Milanfar, and Michael Elad. DeepRED: deep image prior powered by RED. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 2019.
[33] Sachit Menon, Alex Damian, McCourt Hu, Nikhil Ravi, and Cynthia Rudin. Pulse: Self- supervised photo upsampling via latent space exploration of generative models. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[34] Chris Metzler, Ali Mousavi, and Richard Baraniuk. Learned d-amp: based compressive image recovery. In Advances in Neural Information Processing Systems, volume 30, 2017.
Principled neural network
[35] Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. arXiv preprint 2XXi:2202200672 2021.
[36] Gregory Ongie, Ajil Jalal, Christopher A Metzler, Richard G Baraniuk, Alexandros G Dimakis. and Rebecca Willett. Deep learning techniques for inverse problems in imaging. IEEE Journal on Selected Areas Information Theory, 1(1):99-56 56,995666200000 2020.
[37] Gregory Ongie, Ajil Jalal, Christopher A Metzler, Richard G Baraniuk, Alexandros G Dimakis, and Rebecca Willett. Deep learning techniques for inverse problems in imaging. IEEE Journal on Selected Areas in Information Theory, 1(1):39-56, 2020.
[38] Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo. Exploiting deep generative prior for versatile image restoration and manipulation. In European Conference on Computer Vision (ECCV), 2020.
[39] JH Rick Chang, Chun-Liang Li, Barnabas Poczos, BVK Vijaya Kumar, and Aswin C Sankara- narayanan. One network to solve them all-solving linear inverse problems using deep projection models. In Proceedings of the IEEE International Conference on Computer Vision, pages 5888-5897, 2017.
[40] Yaniv Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by denoising (RED). SIAM Journal on Imaging Sciences, 10(4):1804-1844 2017.
[41] Chitwan Saharia, William Chan, Huiwen Chang, Chris A Lee, Jonathan Ho, Tim Salimans, David J Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. arXiv preprint /rXiv:2/ 1.05266, 2021.
[42] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2104.07636. 2021.
[43] Shibani Santurkar, Dimitris Tsipras, Brandon Tran, Andrew Ilyas, Logan Engstrom, and Alek- sander Madry. Image synthesis with single (robust) classifier. arXiv preprint 4rXhv:190.009455 2019.
[44] Jascha Sohl-Dickstein, Eric A Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. arXiv preprint arXiv:1503.0358; March 2015.
[45] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, April 2021.
[46] Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with score-based generative models. arXiv preprint arXiv:2111.08000 2021.
[47] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
12

---

[48] Maitreya Suin, Kuldeep Purohit, and AN Rajagopalan. -atienll-attentivee patch-hierare hierarchical network for adaptive motion deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3606-3615, 2020.
[49] Yu Sun, Brendt Wohlberg, and Ulugbek S Kamilov. An online plug-and-play algorithm for regularized image reconstruction. IEEE Transactions on Computational Imaging, 5(3):395-408. 2019.
[50] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 9446-9454, 2018.
[51] Singanallur V Venkatakrishnan, Charles A Bouman, and Brendt Wohlberg. Plug-and-pla priors for model based reconstruction. In 2013 IEEE Global Conference Signal and Information Processing, pages 445-948. IEEE, 2013.
[52] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing, (3(4))600- 612, 2004.
[53] Allan G Weber. The USC-SIPI image database version 5. USC-SIPI Report, 315(1), 1997.
[54] Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. arXiv preprini 1eXiv:2112.02441 2021.
[55] Raymond Yeh, Chen Chen, Teck Yian Lim, Alexander Schwing, Mark Hasegawa-Johnson, and Minh N Do. Semantic image inpainting with deep generative models. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5485-5493, 2017.
[56] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 1506.03366. 2015.
[57] Kai Zhang, Yawei Li, Wangmeng Zuo, Lei Zhang, Lue Van Gool, and Radu Timofte. Plug-and- play image restoration with deep denoiser prior. IEEE Transactions Pattern Analysis and Machine Intelligence, 2021.
[58] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European Conference on Computer Vision, pages 649-666. Springer, 2016.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In the future work paragraph in Section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We came to the conclusion that our paper does not have potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] In the appendices
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as URL)? [Yes] In the appendices.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] In both the paper and the appendices.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [N/A] We report results averaged over 1, 000 images in the main paper and 50, 000 images in the appendices. Such large numbers eliminate the need for error bars.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] In the appendices.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] The licenses of previous works' code
and datasets will be included in our camera-ready code.
(c) Did you include any new assets either the supplemental material or as URL? [Yes] We include our code in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] Consent was given by the original authors in their work.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes] The datasets we use are anonymized.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "q8p-Tp5JDQ9", "reviewer": "Reviewer_ghAD", "summary": "This paper addresses the efficiency problem of the unsupervised approach in image restoration. The authors proposed DDRM, a denoising diffusion generative model that uses variational inference to learn the the posterior distribution of a linear inverse problem for image restoration. Specifically, the authors define a diffusion process of image restoration using variational distributions, where the trainable parameters of the distributions can be learnt by maximizing an ELBO objective. The experiments on ImageNet show that DDRM has good performance and outperforms previous methods. ", "questions": "(1) From the visual results, it seems that sometimes the results are oversmooth, e.g., see the hand in Figure 4. It would be nice to evaluate the output images with a metric on image structure similarity, e.g., SSIM. Could the authors consider adding this metric to Table 1 or 2? \n\n(2) In Table 1 and 2, how many NFEs are required for DDRM to outperform previous methods? Could the authors also provide the runtime statistics (memory, time) for each NFE? These numbers might support the efficiency claim better.\n\n(3) Section 3.3: from a practical point of view, what happens if we learn a different model for each inverse problem? Would that give even better performance?", "limitations": "\n- There are a few typos in the manuscript, e.g., delubrring, diffsuion. Please use a spell checker to double check. \n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I like the paper in that it presents a solid and sound technical approach with clear theorems and proofs for using diffusion generative models for image restoration. The focus of the paper is on defining the variational objective and the relevant distributions in the diffusion process, which allows efficient inference. The contribution is novel and significant. \n\nThe results of the paper appear to be convincing enough. Both quantitative and qualitative results demonstrate strong performance compared to previous methods. \n\nThe writing is great. The paper reads well and relatively easy to understand. \n\nOn the downside, the experiments shown in the paper can be more extensive, and some analyses on the degradation model could be added. As the proposed method is claimed as an unsupervised method for image restoration with priors learnt on ImageNet, it is a pity that only synthetic degradation models are shown in the results. As ImageNet is a dataset with real images, it would be nice to understand further how well the real degradation can be handled with the proposed model. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "endhsKtULsD", "reviewer": "Reviewer_nQdA", "summary": "The paper proposes a method for the inverse problem based on diffusion model. Attributed to the unsupervised mechanism, it is able to \ngeneralize well in several tasks as shown in the paper.", "questions": "See Weaknesses above.", "limitations": "This work is a non-blind method, and relies on the pre-provide degradation operator, which largely limits its practical value.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. As shown in Theorem C.2, the ELBO objective is equivalent to the form of that in DDPM/DDIM. Thus, the proposed method can readily employ the well-trained DDPM/DDIM model, which avoids the expensive training phase.\n\n2. The proposed method is with fine generalization capability since it does not rely on specific degradation assumpution.\n\nWeaknesses:\nThe main weakness of this work mainly focuses on the experiments. The experimental is too simple compared with related research works. Specifically,\n\n1. In super-resolution, it only consider the block averaging filter. To the best of my knowledge, this is rarely used in super-resolution community.  Bicubic downsampling is the commonly used to evaluate the performance of different methods. I suggest the author should provide more experimental results under such setting, and compared with more recent methods.\n\n2. For deblurring, aside from the more challenged motion blur case, the simple anisotropic Gaussian deblurring should be at least considered.\n\n3. Even though announced to be left in future work, the proposed method relies on the pre-known degradation operator H, which is inaccessible in most of scenarios. Generally, this is a beautiful but unpractical work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1YwnoSb4jf5", "reviewer": "Reviewer_VSry", "summary": "This paper studies image restoration problem and develops an efficient solution called denoising diffusion restoration model (DDRM). The key idea is to borrow principles from variational inference and exploit a pre-trained denoising diffusion generative model. This is conceptually similar to the popular pre-trained models in high-level vision (i.e., one pre-trained model supports different down-stream tasks). The DDRM has been tested on several image restoration tasks including SR, deblurring, inpainting, and colorization. The fast running speed is one of the salient features of the proposed DDRM.", "questions": "1. When DDRM produce a sequence of samples (restored images) like SNIPS, can you estimate their MSE like SURE does? If Yes, it might be possible to obtain a weighted (instead of unweighted) average in Fig. 4 of Supplementary Material.\n2.  What assumption do you make about the degradation model H? Since Eq. (3) does SVD on H, H cannot be arbitrary, right? From Fig. 6, the current deblurring experiments assume optical blur. Can DDRM be applied to motion blur (but still satisfy the separable condition in D.5 of Supplementary)?\n3. I am a bit confused by Fig. 1 in the Supplementary Material. What is the difference between \"DDRM (20)\" and \"Denoised\"? I assume \"Denoised\" means an MMSE denoiser. But the text can be more clear. Meanwhile, does DDRM reach a fixed-point iteration - i.e., x=f(x), where f is the denoiser? The motivation here is that we might assume clean images do satisfy such fixed-point condition. ", "limitations": "n/a", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The paper is well written and easy to follow.\n+ DDRM is appealing because it makes little assumption about the degradation model (i.e., unsupervised approach).\n+ Computational efficiency of DDRM is attractive for real-world applications.\n+ DDRM has achieved improved restoration performance over other competing approaches including DGP (ECCV'2020) and SNIPS (NeurIPS'2021)\n\nWeaknesses:\n- It seems to me that the convergence issue of the DDRM is not discussed in the paper. Will DDRM-based image restoration always converge after 20 or 100 steps?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "0JsjGpz_Upo", "reviewer": "Reviewer_u8iu", "summary": "This paper proposes Denoising Diffusion Restoration Models (DDRM), which is the first general sampling-based inverse problem solver for various unsupervised image restoration tasks. DDRM outperforms existing baselines and is efficient in terms of NFEs. Also, DDRM is more beneficial when there is noise in the input images.", "questions": "- It is preferred to report quantitative results according to the number of iterations for the proposed method. What happens if more than 20 iterations are applied for the proposed method in Table1,2?\n\n- Most of the experiments were performed on ImageNet. Does the proposed method work well for frequently used datasets for each task? For instance, Set5, Set14, B100, Urban100 and Manga109 for super-resolution. What happens to verify the proposed method by applying it to real-world data such as medical imaging mentioned in the introduction?\n", "limitations": "- PSNR and KID were used as evaluation metrics. I am curious about quantitative results when other evaluation metrics are utilized for image reconstruction problems, such as BRISQUE, NIQE, PIQE, RankIQA, MetaIQA. \n\nBRISQUE: Mittal, Anish, Anush Krishna Moorthy, and Alan Conrad Bovik. \"No-reference image quality assessment in the spatial domain.\" IEEE Transactions on Image Processing (TIP) 21.12 (2012): 4695-4708.\n\nNIQE: Mittal, Anish, Rajiv Soundararajan, and Alan C. Bovik. \"Making a “completely blind” image quality analyzer.\" IEEE Signal Processing Letters (SPL) 20.3 (2012): 209-212.\n\nPIQE: Venkatanath, N., et al. \"Blind image quality evaluation using perception based features.\" 2015 Twenty First National Conference on Communications (NCC). IEEE, 2015.\n\nRankIQA: Liu, Xialei, Joost Van De Weijer, and Andrew D. Bagdanov. \"Rankiqa: Learning from rankings for no-reference image quality assessment.\" Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2017.\n\nMetaIQA: Zhu, Hancheng, et al. \"MetaIQA: Deep meta-learning for no-reference image quality assessment.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n- The proposed method requires much fewer iterations compared to existing diffusion-based methods.\n\n- The proposed method was experimented on various image restoration tasks including super-resolution, inpainting and colorization.\n\nWeakness\n\n- As mentioned in the Conclusions, the proposed method is focused on linear inverse problems, so there is likely to be a weakness in the non-linear inverse problem that exists in real data. It would be good to suggest the possible limitations and the analysis when the proposed method is applied to the non-linear inverse problem setup.\n\n- Also, there are many cases where the degradation operator is unknown at the inference stage. Therefore, it would be good to have a mention on how to apply the proposed method in a practical real-world situation (degradation operator is unknown).\n\n- Although the above two limits seem out-of-scope of this paper, it  would be good to analyze the proposed method from various angles by experimenting with various data and setups. For example, it is possible to consider various data sets, image resolution and quality, as well as non-linear inverse problem cases and unknown degradation operators.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept