Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model
Authors: Yinhuai Wang
Jiwen Yu
Jian Zhang
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Zero-Shot
Inverse Problems
Super-Resolution
Diffusion Models
Range-Null Space
Image Restoration
Colorization
Compressed Sensing
Inpainting
Deblur
Old Photo Restoration
Blind Restoration

Abstract: Most existing Image Restoration (IR) models are task-specific, which can not be generalized to different degradation operators. In this work, we propose the Denoising Diffusion Null-Space Model (DDNM), a novel zero-shot framework for arbitrary linear IR problems, including but not limited to image super-resolution, colorization, inpainting, compressed sensing, and deblurring. DDNM only needs a pre-trained off-the-shelf diffusion model as the generative prior, without any extra training or network modifications. By refining only the null-space contents during the reverse diffusion process, we can yield diverse results satisfying both data consistency and realness. We further propose an enhanced and robust version, dubbed DDNM+, to support noisy restoration and improve restoration quality for hard tasks. Our experiments on several IR tasks reveal that DDNM outperforms other state-of-the-art zero-shot IR methods. We also demonstrate that DDNM+ can solve complex real-world applications, e.g., old photo restoration. 

ZERO-SHOT IMAGE RESTORATION USING DENOISING DIFFUSION NULL-SPACE MODEL
Yinhuai W Jiwen Y Zhang!2f
Pekkin University Shenzhen Graduate School, Pengg Cheng Laboratory (yinhuai; yujiwen] aku..du.. cn, zhangjian szêpku. en
ABSTRACT
Most existing Image Restoration (IR) models are task-specific, which can not be generalized to different degradation operators. In this work, we propose the De- noising Diffusion Null-Space Model (DINNM), a novel zero-shot framework for arbitrary linear IR problems, including but not limited to image super-resolution, colorization, inpainting, compressed sensing, and deblurring. DDNM only needs pre-trained off-the-shelf diffusion model the generative prior, without any tra training or network modifications. By refining only the null-space contents during the reverse diffusion process, data consistency and realness. We further propose an enhanced and robust version, dubbed DDNM+ to support noisy restoration and improve restoration quality for hard tasks. Our experiments on several IR tasks reveal that DDNM outperforms other state-of-the- zero-shot IR methods. We also demonstrate that DDNM can solve complex real-world applications, e.g., old photo restoration.
yield diverse results satisfying both
INTRODUCTION
Saper-Resokuiii
Impauting
Colorizaton
19
Robust arbimary scales with realness and striet consistency
Diverse tacks with diverse results
(c) Robust to synthetic/raal-wodl noise
(d) Flexible in solving complex degradations
Figure I: We use + to solve various image restoration tasks in a zero-shot way. Here we show some of the results that best characterize method, where y is the input degraded image and Xo represents the restoration result. Part (a) shows the results of + on image super-resolution (SR) from scale 2x to extreme scale 256x. Note that D assures strict data consistency. Part (b) shows multiple results of + on inpainting and colorization. Part (c) shows the results of + SR with synthetic noise and colorization with real-world noise. Part (d) shows the results of + on old photo restoration. All the results here are yielded in a zero-shot way.
'quual contribution. i Corresponding author. Code is available at https://eitubb.oo///ywaaaa/// work supported in part by Shenzhen Research Project under Grant JCY1202220331092215550 and Grant ISGG71222208200588000.
This

---

Image Restoration (IR) is a long-standing problem due its extensive application value and its ill- posed nature Richardsoon 1972 Andrews & Hunt 1977). IR aims at yielding a high-quality image & from a degraded observation y Ax + where x stands for the original image and represents a non-linear noise. A is a known linear operator, which may be a bicubic downsampler in image super-resolution, a sampling matrix in compressed sensing, even composite type. Traditional IR methods are typically model-based, whose solution can be usually formulated as:
1 20 |Ax
X arg min
v112 + XR(x).
The first data- fita-fidelity term 1/2 A yll optimizes the result toward data consistency while the sec- ond image-pria term (R(x) regularizes the result with formulaic prior knowledge on natural image distribution, e.g., sparsity and Tikhonov regularization. Though the hand-designed prior knowledge may prevent some artifacts, they often fail to bring realistic details.
The prevailing of deep neural networks (DNN) brings new patterns of solving IR tasks (Dong al.) 2015), which typically train an end-to-end DNN De by optimizing network parameters 0 following
N
arg min Σ [Do(y.)) - -x:1
where N pairs of degraded image y; and ground truth image x; are needed to learn the mapping from y to x directly. Although end-to-end learning-based IR methods avoid explicitly modeling the degradation A and the prior term in Eq. [] and are fast during inference, they usually lack in- terpretation. Some efforts have been made in exploring interpretable DNN structures (Zhang & Ghanem 2018 Zhang et al., 2020), however, they still yield poor performance when facing domain shift since Eq. 2 essentially encourage learning the mapping from y; to x;- For the same reason, the end-to-end learning-basee IR methods usually need to train a dedicated DNN for each specific task, lacking generalizability and flexibility in solving diverse IR tasks. The evolution of generative models (Goodfellow et al. 2014; Bahhat & Michaeli 2014] Van Den Oord et al.) 2017 Karras et al., 2019 2020; 2021) further pushes the end-to-end learning-baat based IR methods toward unprecedented performance in yielding realistic results (Yang et al., [021]] Wang et al.) [021] Chan et al. [222]]
Wang et al., 2022). At the same time, some methods (Menon et al., 20220 Pan et al.) 2021] start to leverage the latent space of pretrained generative models to solve IR problems in a zero-shot way.
Typically, they optimize the following objective:
1 w 20
arg min
||AG(w) - y-1 + AR(w),
where G is the pretrained generative model, w is the latent code, G(w) is the corresponding gener- ative result and R(w) constrains w to its original distribution space, e.g., Gaussian distribution. However, this type of method often struggles to balance realness and data consistency.
The Range-Null Null space decomposition (Schwab et al. 2019 Wang al. 2023) offers a new perspec- tive on the relationship between realness and data consistency: the data consistency is only related to the range-space contents, which can be analytically calculated. Hence the data term be strictly guaranteed, and the key problem is to find proper null-space contents that make the result satisfy- ing realness. We notice that the emerging diffusion models (Ho et al., 2020; Dhariwal & Nichol, 2021) are ideal tools to yield ideal null-space contents because they support explicit control over the generation process.
In this paper, we propose a novel zero-shot solution for various IR tasks, which we call the Denoising Diffusion Null-Space Model (DDNM). By refining only the null-space contents during the reverse diffusion sampling, our solution only requires an off-the-shee diffusion model to yield realistic and data-consistent results, without any extra training or optimization nor needing any modifications to network structures. Extensive experiments show that DDNM outperforms state-of-the-arr zero- shot IR methods in diverse IR tasks, including super-resolution, colorization, compressed sensing, inpainting, and deblurring. We further propose enhanced version, DDNM+ which significantly elevates the generative quality and supports solving noisy IR tasks. Our methods are free from domain shifts in degradation modes and thus ftexibly solve complex IR tasks with real-world degradation, such as old photo restoration. Our approaches reveal a promising new path toward solving IR tasks in zero-shots, the data consistency is analytically guaranteed, and the realness
2

---

is determined by the pretrained diffusion models used, which rapidly evolving. Fig. provides some typical applications that fully show the superiority and generality of the proposed methods.
Contributions. (1) In theory, reveal that a pretrained diffusion model can be zero-shot solver for linear IR problems by refining only the null-space during the reverse diffusion process. Corre- spondingly, we propose unified theoretical framework for arbitrary linear IR problems. We further extend our method to support solving noisy IR tasks and propose a rime-travel trick to improve the restoration quality significantly; (2) In practice, our solution the first that can decently solve di- verse linear IR tasks with arbitrary noise levels, in a zero-shot manner. Furthermore, our solution can handle composite degradation and is robust noise types. whereby we can tackle challenging real-world applications. Our proposed DDNMs achieve state-of of-the-art zero-shot IR results.
2 BACKGROUND
2.1 REVIEW THE DIFFUSION MODELS
We follow the diffusion model defined in denoising diffusion probabilistic models (DDPM) (Ho et al., 2020). DDPM defines Tstep forward process and a T-seep reverse process. The forward process slowly adds random noise to data, while the reverse process constructs desired data samples from the noise. The forward process yields the present state x from the previous state X;-1i
(xx xx-)) = (xx; / 1, 3,1) i.c., X = / 3,x;-1 + v BLE, E~N(0,
where X₁ is the noised image at time-step f, Be is the predefined scale factor, and N represents the Gaussian distribution. Using reparameterizztion trick, becomes
g(x: xo) = (xx;; / Va,xo, (1 - with or ===1- 11 Br, O2 Il
04-
(5)
t=0
The reverse process aims yielding the previous state x from x2 using the posterior distribution p p(x,-1|Xe; Xo), which can be derived from the Bayes theorem using Eq. and Eq.
5
q(x)-1(x0) N(x M4,,, Xo), of II
p(x; x,, xo) q(xx Xx,
(x,, Xn)
Vi- and variance -7=1-aa Bt. represents the noise in x; and is the only uncertain variable during the reverse process. DDPM uses neural network Zo predict the noise for each time-step 4, i.e., e, = Zx(x,, l), where e, denotes the estimation of e at time-step t. To train Zo. DDPM randomly picks a clean image Xo from the dataset and samples a noise E NN, I), then picks random time-step and updates the network parameters 0 in Ze with the following gradient descent step (Ho et al. 2020):
with the closed forms of mean pel (x,, (0), x0)= far X, E 1-ar
1-4;
Volle (V + evI āt By iteratively sampling X,-1 from p(x,-1xx, xo), DDPM can yield clean images Xo~ag(x (x) from random noises X7>>N(0, I), where x(x) represents the image distribution in the training dataset.
2.2 RANGE-NULL SPACE DECOMPOSITION
For ease of derivation, we represent linear operators in matrix form and images in vector form. Note that our derivations hold for all linear operators. Given a linear operator A E RdxD its pseudo- inverse A! E IR. Dxd satisfies AA = A. There are many ways to solve the pseudo-inverse A e.g., the Singular Value Decomposition (SVD) is often used to solve At in matrix form, and the Fourier transform is often used to solve the convolutional form At.
A and At have some interesting properties. A A can be seen the operator that projects samples x E Dx1 to the range-space A because AA Ax Ax. In contrast, (I A A) can be seen as the operator that projects samples the null-space of A because A(I A AAAxx O.
Interestingly, any sample x can be decomposed into two parts: one part is in the range-space of A and the other is in the null-space of A, i.e.
x = A + (I-AAA) AtA)x
(8)
This decomposition has profound significance for linear IR problems, which will get to later.
3

---

3 METHOD
3.1 DENOISING DIFFUSION NULL--PACE MODEL.
Null-Space All We Need. We start with noise-free Image Restoration (IR) as below:
Ax,
where x E R A E R and y E Rdd denote the ground-truth (GT) image, the linear degra- dation operator, and the degraded image, respectively. Given an input y, IR problems essentially aim yield an image & E Dx1 that conforms the following two constraints:
Consistency: A& y,
Realness: : X ~~ q(x).
(10)
where (x) denotes the distribution of the GT images.
For the Consistency constraint, we can resort to range-null space decomposition, As discussed in range-space part A "A and null-space part
Sec. 2.2 the GT image
be decomposed (I A A)x. Interestingly, we can find that the range-space part A Ax becomes exactly y after being operated by A. while the null-space part (I A A)x becomes exactly 0 after being operated by A, i.e., Ax = AAtAx + A(I A Ax + 0 y.
More interestingly, for degraded image y, can directly construct general solution & that satisfies the Consistency constraint Ax y, that is x = A -I-AAAAAA A Whatever X is, it does not affect the Consistency all. But x determines whether & (x).. Then our goal is find proper & that makes & ~ (()). We resort to diffusion models to generate the null-space (I A A)A)x which is in harmony with the range-space A ly.
Refine Null-Space Iteratively. We know the reverse diffusion process iteratively samples x;-1 from p(x;-1 [x, Xn) to yield clean images Xy ~ q(x) from random noises XT ~ (0, I). However, this process is completely random, and the intermediate state x, is noisy. To yield clean intermediate states for range-null space decomposition, we reparameterize the mean (x,e Xo) and variance of
of distribution p (xx-11 Xg, X1)
va:-1Bt Xa + vor(1 - Ôt-1) Oz
1 1- 1-ā-11 Br,
(x,,,xx)
Xt
of
(11)
1 Ce.
where Xo is unknown, but we can reverse Eq. 5 to estimate x. from x, and the predicted noise E Zo (xx, t). We denote the estimated xu at time-step as Xojt, which can be formulated as:
XOj=
(x, Zo(x,, t)x, - a.).
(12)
Va
Note that this formulation is equivalent to the original DDPM. We do this because it provides "clean" image Xojz (rather than noisy image x.). To finally yield satisfying Axn = y, we fix the range-space A A and leave the null-space unchanged, yielding rectified estimation Rojs as:
šoz Aly + (I A AtA)xo
(13)
Hence we use Snjt as the estimation of xo in Eq. IT thereby allowing only the null space to partici- pate in the reverse diffusion process. Then we yield x by sampling from p(x;-1|xx x xnp):
10 Roje + Vor(1 - - x, + o.c, e~ C0,
X
(14)
āg
1 1-āc
Roughly speaking, X;-1 is a noised version of Knj= and the added noise erases the disharmony between the range-space contents A and the null-space contents (I A A'A)xo Therefore, iteratively applying Eq. 12 Eq. 13 and Eq. 14 yields final result xx>- Note that all the rectified estimation Roje conforms to Consistency due to the fact that
(15) Considering Xa is equal to Łol1. so the final result X also satisfies Consistency. We call the pro- posed method the Denoising Diffusion Null-Space Model (DDNM) because utilizes the denoising diffusion model to fill up the null-space information.
A = AA'Y + A(I AlA)x = AA + 0 Ax
4

---

Algorithm Sampling DDNM
Algorithm 2 Sampling of DDNM
1: xx NN., T) 2: 1do
1: ~ N/O,
t T....
3:
L = minff 1,
X ~ -((+++ (x.)
for I....., do
3:
Vos (xc Exce f)vI-aa
Xaj++) Vdc+o (x++1 Zo(xe+ii ++ +1)
4: Soj= = A y (I A)Aoooi
7:
Ro X0113 Ee+jA A (AXOILIS -y)
5:
Xt-1 ~ p(xx-11 bcs, Ro1e)
8:
XLii PiX115 118 Rojets)
6:
return
return Xa
A'Axy (I-A'XXxg
A'
Xo
Xy
B
Xp1
Trans
(a)
(b)
Figure Illustration of (a) DDNM and (b) the time-travel trick.
Algo.[and Fig. a] show the whole reverse diffusion process of DDNM. For case of understanding, we visualize the intermediate results of DDNM Appendix G By using a denoising network Ze pre-trained for general generative purposes, DDNM can solve IR tasks with arbitrary forms of linear does not need tpsc-fee training or optimization and forms a zero-shot solution for diverse IR tasks.
degradation operator
is worth noting that our method compatible with most of the recent advances in diffusion models,
e.g.. DDNM can be deployed to score-based models (Song & Ermon 2019 Song et al. 2020) or combined with DDIM (Song al. 2021a) to accelerate the sampling speed.
3.2 EXAMPLES OF CONSTRUCTING A AND A
Typical IR tasks usually have simple forms of A and At, some of which easy to construct by hand without resorting to complex Fourier transform or SVD. Here we introduce three practical examples. Inpainting the simplest case, where A is the mask operator. Due the unique property that A.A = A, we can use A itself as A1. For colorization, A can be a pixel-wise operator la 1 3 b] that converts each RGB channel pixel [r b] into grayscale value [5 + + 3]-
It is easy to construct a pseudo-inverse At = [1
1]" that satisfies AA I. The same idea can used for SR with scale n, where we can set RRlx=" as the average-pooling operator that averages each patch into single value. Similarly, we can construct its pseudo- inverse as A 1=nx1== =
1] We provide pytorch-like codes in Appendix E
Considering as a compound operation that consists of many sub-operations, i.e., A A we may still yield its pseudo-inverse A .|. This provides flexible solution for solving complex IR tasks, such as old photo restoration. Specifically, we can decompose the degradation of old photos as three parts, i.e., A = A A2A3, where A3 is the grayscale operator, Ag the average- prorage-poooiggge operator with scale 4, and A is the mask operator defined by the damaged areas on the photo. Hence the pseudo-inverse is A AJAÍA/- Our experiments show that these hesinde operators work very well (Fig. (,b,d)).
3.3 ENHANCED VERSION: *
DDNM can solve noise-free IR tasks well but fails handle noisy IR tasks and yields poor Realness in the face of some particular forms of A1, To overcome these two limits, as described by Algo. we propose an enhanced version. dubbed D by making the following two major extensions to DDNM to enable it to handle noisy situations and improve its restoration quality.
Scaling Range-Space Correction Support Noisy Image Restoration We consider noisy IR problems in the form of y + n, where I oI noise and Ax represents the clean measurement. Applying DDNM directly yields
represents the additive Gaussian
E Aty (I- AtA)xo Xoje A (Axx) A Ax) + A"n,
(16)
5

---

where A'n RDx is the extra noise introduced into Rof. and will be further introduced into X;-1- A (Axi) Ax) is the correction for the range-space contents, which is the key to Consistency. To solve noisy image restoration, we propose to modify DDNM (on Eq. I3d and Eq. 14) as:
Kol: = X E,A (Axoje y):
(17)
p(x-1 Xx: Ruje) (x;- Me (X,, Rujs), P.I).
(18)
E, E RDDD is utilized to scale the range-space correction A AAAxoo y) and O E DDDD is used to scale the added noise o,E in p( |x,, &ujz). The choice of Σ, and Φ, follows two principles: (i)
D, and & need to assure the total noise variance in x conforms to the definition in q(x,-1|XX) (Eq. (5) so the total noise can be predicted by Zg and gets removed; (ii) E, should be close as possible to I to maximize the preservation of the range-space correction A (AXoje - -y) so as to maximize the Consistency. For SR and colorization defined in 3.2 AT is copy operation. Thus A n can be approximated as a Gaussian noise (0, I then 2, and 4, can be simplified as vor- X0ot + (a.l1-0.-0 x, + o,e, principle (i) is 1-ūt Considering principle (ii), we set:
Σ, A and 9, = I Since x,-1
I-ãee
equivalent to: (an2,ay) )² + it = of with or denotes vā.-1..
I-a.
2 = of (ay) X - -
ot. Z a,
(19)
(o1/aray, 0x < 0,Ty forms of At, where we set Σ, Vdiag A. derived from the SVD of the operator A (= UEV"). The calculation of d and V/h are presented in Appendix Note that the only hyperparameter that need manual setting is oy.
In addition to the simplified version above, we also provide a more accurate version for general
T, V is
b; = ....
We can also approximate non- Gaussia noise like Poisson, speckle, and real-world noise as Gaussian noise, thereby estimating noise level By and resorting the same solution mentioned above.
Hime-Trave For Better Restoration Quality We find that DDNM yields inferior Realness when facing particular cases like SR with large-scale verage-poolinng downsampler, low sampling ratio compressed sensing(CS), and inpainting with a large mask. In these cases, the range-space contents Aty is too local guide the reverse diffusion process toward yielding a global harmony result.
Let us review Eq. 11 p(x,-1 Xn) relies on accurate estimation of xg. DDNM uses Koje as the estimation x1 at time- step f, but if the range-space contents A Aty is too local or uneven, Kojt may have disharmonious null-space contents. How the past. Say we travel back to time-step L + I, we can yield the next state X using the "future" estimation Xofz- which should be more accurate than Xojc-1- By reparameteriizttio. this operation equivalent to sampling X from (x++111 11, Similar to Lugmayr et al. (2022) that use "back and forward" strategy for inpainting tasks, we propose a time-travel trick to improve global harmony for general IR tasks: For a chosen time-step f, we sample X from g ((x+++(x)) -x,).. Then we
We can see that the mean value (x, (o) of the posterior distribution
salvage the disharmony? Well,
time travel back to change
travel back to time-step & ++ | and repeat normal DDNM sampling (Eq. 12 Eq. 13 and Eq. [4] until yielding x is actually the travel length. Fig. b illustrates the basic trav-travel trick.
Intuitively, the time-travel trick produces better "past", which in turn produces a better "future". For ease of use, we assign two extra hyperparameters: s controls the interval of using the time-travel trick; r determines the repeat times. The time-travel trick in Algo. 2 is with 1, r. = 1. Fig. bb) and the right part in Tab. demonstrate the improvements that the time-travel trick brings.
is worth emphasizing that although Algo. and Algo. 2 are derived based on DDPM, they can also be easily extended to other diffusion frameworks, such as DDIM (Song et al. 2021a). Obviously, DDNM* becomes exactly DDNM when setting Σ. I, 4; of1, and O.
4 EXPERIMENTS
Our experiments consist of three parts. Firstly, we evaluate the performance of DDNM on five typical IR tasks and compare it with state-of-the-art zero-shot IR methods. Secondly, experiment D on three typical IR tasks to verify its improvements against DDNM. Thirdly, we show that DDNM and DDNM+ perform well challenging real-world applications.

---

ImageNet
Deblurring
Colorization
25%
Inpointing
ConeL/FIDD
PSNR1/SSIM//FIDD
PSNRT/SSIM
A'y
24.26 0.684 134.4
18.56 0.6616/ 55.42 55.42
0.0/43.37
/0.510/2774
14.52/0.99972.777
DGP
23.18 0.798 64.34
NA
69.55
NIA
N/A
ILVR
27.40 0.870/443....0
N/A
N/A
NIA
N/A
N/A
NIA
31.87/ 12.31
DDRM
27.38 0.869 443.15
43.01 / 0.992 1.48
36,56
19.95 /0.704 147704997.99 97.99
31.73 0.966 4.82
(DNM()ur))
27.46 / 0.870/ 39.26
44.93/ 0,994/ 1.15
42.32/ 36.32
21.66 0.749/ 64.68
32.06 0.968
CelebA-HQ
4x SR
Deblurring
Colorization
CS 25%
Inpainting
Method
PSNRT/SSIMT/FID
/SSM//SSIM//
ConsL/FID
PSNR //S
P/NRT/SSIM//EUL
Aty
27.27/0.785 103.3
0..45 54.31
0.0 68.81
15.09 /0.583/ 377.7
15.57 0.880/ 181.56
PULSE
22.74 0.623 40.33
NIA
N/A
N/A
N/A N/A
HVR
31.59/ 0.945 29.82
N/A
N/A
N/A
RePaint
N/A
NIA
N/A
N/A
35.20/ / 0.981 /14.19
DDRM
31,63/ 0.945 31.04
43.07 0.993 6.24
4 31,26
24.86 0.876 46.77
34.79 / 0.978 /12.53
DDNM
31.63/ 0.945 22.27
46.72/ 0.996 1.41
26.25 26.44
27.56 0.909 28,80
35.64/ 0.982 /4.54
Table I: Quantitative results of zero-shot IR methods on ImageNet(top)) and (obtto-H((bottmm), including five typical IR tasks. We mark N/A for those not applicable and bold the best scores.
2+ 58
4+ SR
$- SR
16+ SR
32+ SX
Ayy
A's
DGP
2
DONMI
(cam))
DONM (OuRS)
Super Resolution CelebA
WAMAEE
DORM
20RM
RePaine
DONM (OEN)
DONMP (csm)
Compressed Sensing on
Iapointing DIV2K
Figure 3: Qualitative results of zero-shot IR methods.
7

---

Coloiiaton n=12
Inpulstin
H6x CHO2
32x
Cdloaanian
CS 19%
AY
DINM
DONM
DOXM
Figure 4: DDNM improves (a) denoising performance and (b) restoration quality.
CdlbA-HQ
a=0.2
Co
CS r5tiow25% ow00.
x2x SR
CS ration
Method
FIDL
PSNRT/SSIM/FIID
PSNRT/SSIMM//III
FID.
PSNR1/5SIM///IDL
DDNM 13, /..2377 /211
216.74
17.89 / 0.4531 / 82.81
17.55/0.437) / 39.37
22.79
15.74/0.27/
DDNM 19,44/ 0.71/0.11//55 58.31
46.11
25.02/ 0.868 51.35
18.44/ 0.500/ 37.50
18.23
26.33 0.741/47.9
Ablation study on denoising improvements (left) and the time-travel trick (right). C repre- sents the colorization task. denotes the noise variance on y.
Table 2:
4.1 EVALUATION ON DDNM
To evaluate the performance of DDNM, we compare DDNM with recent state-of-the-art zero-shot IR methods: DGPnChhen Davies, 2020j, Pulse(Menon et al. 2020), ILVR(Choi al., 202I),
RePaint[Lugmay et al. 2022) and DDRM(Kawar et al. 2022). We experiment on five typical noise-free IR tasks, including 4x SR with bicubic downsampler, deblurring with Gaussian blur ker- nel, colorization with average grayscale operator, compressed sensing (CS) using Walsh-Hadamard sampling matrix with a 0.25 compression ratio, and inpainting with text masks. For each task, we use the same degradation operator for all methods. We choose ImageNet 1K and CelebA-HQ IK datasets with image size x56x256 for validation. For ImageNet IK, we use the 256 denoising network as Ze. which is pretrained on ImageNet by Dhariial & Nichol 2021). For CelebA-HQ IK, we use the 256 denoising network pretrained on CelebA-HQ by Lugmayr et al. (2022). For fair comparisons, we use the same pretrained denoising networks for ILVR, RePaint, DDRM, and DDNM. We use DDIM as the base sampling strategy with r) = 0.85, 100 steps, without classifier guidance, for all iiffusion-based methods. We choose PSNR, SSIM, and FID (Heusel et al. 2017) as the main metrics. Since PSNR and SSIM can not reflect the colorization performance, we use FID and the Consistency metric (calculated by ||Ax, y and denoted as Cons) for colorization.
Tab. shows the quantitative results. For those tasks that are not supported, we mark them as "NA" We can see that DDNM far exceeds previous GAN prior based zero-shot IR methods (DGP, PULSE). Though with the same pretrained denoising models and sampling steps, DDNM achieves significantly better performance both Consistency and Realness than ILVR, RePaint, and DDRM. Appendix shows more quantitative comparisons and qualitative results.
4.2 EVALUATION ON D
We evaluate the performance of DDNM* from two aspects: the denoising performance and the robustness in restoration quality.
Denoising Performance. We experiment D on three noisy IR tasks with 1 = 0, i.e., we disable the time-travel trick only evaluate the denoising performance. Fig. a and the left part in Tab. 2 show the denoising improvements of DDNM+ against DDNM. We can see that DDNM fully inherits the noise contained in y, while DDNM decently removes the noise.
Robustness in Restoration Quality. We experiment DDNM+ on three tasks that DDNM may yield inferior results, they are 32x SR, colorization, and compressed sensing (CS) using orthogonalized sampling matrix with 10% compression ratio. For fair comparison, we set T 250, = 8 20, 1000 for DDNM so that the total sampling steps and computa-
r = 3 for DDNM while set tional consumptions are roughly equal. Fig. Tb) and the right part in Tab. show the improvements of the time-traae travel trick. We can see that the time-travel trick significantly improves the overall per- formance, especially the Realness (measured by FID).
8

---

To the best of our knowledge, DDNM* is the first IR method that robustly handle arbitrary scales of linear IR tasks. As is shown in Fig. 3 We compare DDNM (1 = 10, r = 5) with state-of-the- dataset (Agustsson & Timofte, 2017) as the testset. The results show that DDNM+ owns excellent robustness in dealing with diverse IR tasks, which is remarkable considering DDNM zero-shot method. More experiments of DDNM/DDNM* can be found in Appendix A and B
zeot IR methods
diverse IR tasks. We also crop images from DIV2K
4.3 REAL-WORLD APPLICATIONS
Theoretically, we can use DDNM to solve real-world IR task as long as we can construct an
approximate linear degradation A and its pseudo-inverse A A.. Here we demonstrate two typical
real-world applications using DDNM* with / = s 20, = 3: (1) Real-World Noise. We
experiment DDNM* on real-world colorization with and A defined in Sec. 3.2 We set σy by observing the noise level of y. The results are shown in Fig. 6 Fig. 7 and Fig. C). (2) Old Photo
Restoration. For old photos, we construct A and A as described in Sec 3.2 where we manually
draw mask for damaged areas the photo. The results are shown in Fig. 1 d), and Fig. 15
RELATED WORK
5.1
DIFFUSION MODELS FOR IMAGE RESTORATION
Recent methods using diffusion models to solve image restoration can be roughly divided into two categories: supervised methods and zero-shot methods.
Supervised Methods. SR3 (Saharia et al. 2021) trains conditional diffusion model for image super-resolution with synthetic image pairs as the training data. This pattern is further promoted to other IR tasks (Saharia et al.] 2022). To solve image deblurring, [Whang al. (2022) uses deter- ministic predictor to estimate the initial result and trains a diffusion model to predict the residual. However, these methods all need task-specific training and not generalize to different degrada- tion operators or different IR tasks.
Zero-Shot Methods. (Song& & Ermon (209)) first propose zero-shot image inpainting solution by guiding the reverse diffusion process with the unmasked region. They further propose using gradi- guidance to solve general inverse problems in zero-shot fashion and apply this idea medical imaging problems (Song et al., 2020: 2021b). ILVR (Choi et al., 202I) applies low-frequency guid- ance from a reference image achieve reference-
image generation tasks, RePaint (Lugmayr) et al. 2022) solves the inpainting problem by guiding the diffusion process with the unmasked re- gion. DDRM (Kawar et al., 2022) uses SVD decompose the degradation operators. However, SVD encounters computational bottleneck when dealing with nigh-dimensional matrices. Actu- ally, the core guidance function
ILVR (Choi et al., 202T), RePaint (Lugmayr et al. (022) and DDRM (Kawar et al.] 2022) can be seen special cases of the range-null space decomposition used in DDNM, detailed analysis is in Appendix
5.2 RANGE-NULL SPACE DECOMPOSITION IN IMAGE INVERSE PROBLEMS
Schwab et al. (2019) first proposes using a DNN to learn the missing null null-space contents in image inverse problems and provide detailed theory analysis. Chen & Davies (2020) proposes learning the range and null space respectively. Bahat & Michaeli (2020) achieves editable super-resolution via exploring the null-space contents. Wang et al. (2023 apply range-null space decomposition to existing GAN prior based SR methods to improve their performance and convergence speed.
6 CONCLUSION & DISCUSSION
This paper presents unified framework for solving linear IR tasks in zero-shot manner. We believe that our work demonstrates a promising new path for solving general IR tasks, which may also be instructive for general inverse problems. Theoretically, solve inverse problems of diverse data types, e.g., video, audio, and point cloud, as long as one can collect enough data train a corresponding diffusion model. More discussions are in Appendix C
framework can be easily extended to
9

---

REFERENCES
Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Work- shops (CVPRW), 2017.
Harry C Andrews and Boby Ray Hunt. Digital image restoration. 1977.
Yuval Bahat and Tomer Michaeli. Auto-encoding variational bayes. In The International Confer- ence on Learning Representations (ICLR), 2014.
Yuval Bahat and Tomer Michaeli. Explorable super resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Kelvin CK Chan, Xintao Wang, Xiangyu Xu, Jinwei Gu, and Chen Change Loy. Glean: Generative latent bank for large-factor image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Dongdong Chen and Mike E Davies. Deep decomposition learning for inverse imaging problems. In European Conference on Computer Vision (ECCV). Springer, 2020.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Con- ditionnng method for denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Mare Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint 2tt.:22009146887 2022a.
Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. In Advances in Neural Information Processing Systems (NeurIPS), 2022b.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38, 2015.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Informa- fion Processing Systems (NeurIPS), 2014.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020.
Jonathan Ho, Tim Salimans, Alexey A Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. ICLR Workshop on Deep Generative Models for Highly Struc- tured Data, 2022.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IÉEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Tero Karras, Miia Aittala. Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Process- ing Systems (NeurIPS), 34, 2021.
10

---

Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In ICLR Workshop on Deep Generative Models for Highly Structured Data (ICLRW), 2022.
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. Proceedings of the IEEE/CVF International Confer- ence on Computer Vision Workshops (ICCVW), 2021.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. Pulse: Self- supervised photo upsampling via latent space exploration of generative models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2020.
Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo. Exploiting deep generative prior for versatile image restoration and manipulation. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2021.
William Hadley Richardson. Bayesian-based iterative method of image restoration. JoSA, 62(1), 1972.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2104.07633 2021.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, 2022.
Johannes Schwab, Stephan Antholzer, and Markus Haltmeier. Deep null space learning for inverse problems: convergence analysis and rates. Inverse Problems, 35(2), 2019.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffuusion implicit models. In Interna- tional Conference on Learning Representations (ICLR), 2021a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations (ICLR), 2020.
Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with score-based generative models. International Conference on Learning Representations (ICLR), 2021b.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in Neural Information Processing Systems (NeurIPS). 30, 2017.
Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan. Towards real-world blind face restoration with generative facial prior. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Yinhuai Wang, Yujie Hu, and Jian Zhang. Panini-net: Gan prior based degradation-aware feature interpolation for face restoration. In Proceedings the AAAI Conference on Artificial Intelligence (AAAI), 2022.
Yinhuai Wang, Yujie Hu, Jiwen Yu, and Jian Zhang. Gan prior based null-space learning for con- sistent super-resolution. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2023.
Jay Whang, Mauricio Delbracio, Hossein Talebi. Chitwan Saharia, Alexandros G Dimakis, and Pey- man Milanfar. Deblurring via stochastic refinement. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
11

---

Tao Yang, Peiran Ren, Xuansong Xie, and Lei Zhang. Gan prior embedded network for blind face restoration in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Jian Zhang and Bernard Ghanem. Ista-net: Interpretable optimization-inspired deep network for image compressive sensing, In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2018.
Kai Zhang, Gool Luc-Van and Timofte Radu. Deep unfolding network for image super-resolution. In Proceedingy of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), 2020.
Kai Zhang, Jingyun Liang, Lue Van Gool, and Radu Timofte. Designing a practical degradation model for deep blind image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
12

---

A TIME & MEMORY CONSUMPTION
Our method has obvious advantages in time & memory consumption among recent zero-shot diffusion-based restoration methods (Kawar et al. 2022 Ho et al. 2022, Chung et al. 2022b a). These methods are all based on basic diffusion models, the differences are how to bring the con- straint y Ax + into the reverse diffusion process. We conclude our advantages as below:
DDNM yields almost the same consumption the original diffusion models.
DDNM does not need any optimization toward minimizing lyy Axojell since we di- rectly yield the optimal solution by range-null space decomposition (Section 3.1) and pre- cise range-space denoising (Section 3.3). We notice some recent works (Ho et al. 2022. CChng et al. 2022b a) resort to such optimization, c.g., DPS (Chung et al. (022a) uses
X;-1 X;-1 computation.
V. Xx ly Axojell2 11 to update x;-1i however, this involves costly gradient
Unlike DDRM Kawar al. 2022). our DDNM does not necessarily need SVD. As presented in Section 3.2, we construct A and At for colorization, inpainting, and super- resolution problems by hand, which bring negligible computation and memory consump- tion. In contrast, SVD-based methods suffer heavy cost memory and computation if A has a high dimension (e.g., 128xSR, as shown below).
Experiments in Tab. well support these claims.
ImageNet
4x SR Time(s/image)
64x SR
128x SR
Method
FID.
Mcmory(MB)
Time
Memory
Time
Memory
DDPM
N/A
NIA
11.9
5758
11.9
5758
11.9
5758
DPS
25.51
55.92
36.5
8112
DDRM
27.05
38.05
12.4
5788
36.4
5788
83.3
6792
DDNM
27.04
33.81
11.9
5728
11.9
5728
11.9
$728
Table 3: Comparisons Time & Memory Consumption. We the average-pooling downsampler, 4x SR, 100 DDIM steps with 700.85 and without classifier guidance, on a single 2080Ti GPU with batch size 1. For DPS, we set G=100-/ā--1- *The DDPM here tested on unconditional generation.
B COMPARING DDNM WITH SUPERVISED METHODS
Our method is superior to existing supervised IR methods Zhhag et al. 2021 Liang et al. 2021 in these ways:
DDNM is zero-shot for diverse tasks, but supervised methods need to train separate models for each task.
DDNM is robust to degradation modes, but supervised methods own poor generalized per- formance.
DDNM yields significantly better performance on certain datasets and resolutions (c.g., ImageNet 256x256).
These claims are well supported by experiments in Tab.
ImageNet
Bicubic, =0
Average pooling, cy =0
Average pooling. =0.2
Inference time
Method
PSNR1
SSIMT
FIDI
PSNRT
SSIMI
FID,
PSNR1
SSIMT
FIDJ
sfimage
SwinlR-L
21.21
0.7410
56.77
23.88
0.8010
54.93
18.39
0.5387
134.18
6.1
BSRGAN
21.46
0.7384
68.15
24.14
0.7948
67.70
14.06
0.3663
195.41
0.036
DDNM
27.46
0.8707
39.26
27.04
0.8651
33.81
22.67
0.7400
80.69
Table 4: Comparisons between DDNM and supervised SR methods. DDNM uses 100 DDIM steps with 00.5 and without classifier guidance. We the official SwinIR-L (Liang al. 202 and BSRGAN (Zhang et al. 2021) pretrained for SR tasks.
13

---

C
LIMITATIONS
There remain many limitations that deserve further study.
Though DDNM brings negligible extra cost on computations, it is still limited by the slow inference speed of existing diffusion models.
DDNM needs explicit forms of the degradation operator, which may be challenging acquire for some tasks. Approximations may work well, but not optimal.
In theory, DDNM only supports linear operators. Though nonlinear operators may also have "pseudo-inverse" they may not conform the distributive property, e.g.. sin (anaaall aa-bb b) * sin(a) + sin, so they may not have linearly separable null-space and range-space.
DDNM inherits the randomness of diffusion models. This property benefits diversity but may yield undesirable results sometimes.
The restoration capabilities DDNM are limited by the performance of the pretrained denoiser, which is related to the network capacity and the training dataset. For example, existing diffusion models do not outperform StyleGANs (Karras al. 2019 2020 2021) in synthesizing FFHQ/AFHQ images at 1024x 1024 resolution.
D SOLVING REAL-WORLD DEGRADATION USING DDNM
DDNM+ can well handle real-world degradation, where the degradation operator unknown and non-linear and even contains non-Gaussian noise. We follow these observations:
In theory. DDNM is designed to solve IR tasks of diverse noise levels. As is shown in Fig. 5 DDNM can well handle xx SR even with strong noise =0=9.
For real-world degraded images, the non-linear artifacts can generally divided into global (e.g., the real-world noise in Fig. [c) and local (e.g., the scratches in Fig. dd).
For global non-linear artifacts, we can set proper dy to cover them. As is shown in Fig. 6 the input images y suffer JPEG-like unknown artifacts, but DDNM can still remove them decently by setting proper Ty.
For local non-linear artifacts, need to construct A = Accloormaac and set a proper dy. We have proved Acotor and Amask and their pseudo-inverse be easily constructed by hand. (maybe Asa is needed for resize when too blur)
directly draw mask to cover them. Hence all
GT
A'y
DDNM
Figure 5: DDNM well handle 4x SR even with strong noise =0=9.9.
14

---

DDNM
o--0. 0.1
DDNM G=
Figure 6: Solving JPEG-like artifacts using DDNM Here set A = to exert pure denoising. y denotes the input degraded image. When we set αy 0.1, the artifacts are decently removed. When we set By 0.2, the results become smoother but yield relatively poor identity consistency.
00
5 0.1
6,- 0.15
0.25
Figure 7: Old photo restoration. Zoom in for the best view. By setting Qy 0.1, the noise is removed, and the identity is well preserved. When we set higher σy = 0.25, the results becomes much smoother but yield relatively poor identity consistency.
In Fig. 7 (w demonstrate an example. The input image y is black-and-white photo with unknown noise and scratches. We first manually draw mask Amask cover these scratches. Then use a grayscale operator Acolor to convert the image into grayscale. Definition Aask and Acolor and their pseudo-inverse can be find in Sec. 3.2] Then take A Acolor Amask and At Amask Acolor for DDNM and set proper øy. From the results in Fig. 7 we can that when setting σy = O, the noise is fully inherited by the results. By setting Jy = 0.1, the noise removed, and the identity well preserved. When set higher dy 0.25, the results becomes much smoother but yield relatively poor identity consistency.
The choice of By is critical achieve the best balance between realness and consistency. But for now we can only rely on manual estimates.
15

---

E PYTORCH-LIKE CODE IMPLEMENTATION
Here we provide a basic PyTorch-Like implementation DDNM Readers can quickly implement
basic DDNM* their own projects by referencing Algo. 2 and Sec. 3.3 and the code below.
def color2gray
cocf=1/3
=xx1:.02. :::
xl:.1 de 21:.2.:.:3
return 33..
def graylcolor(x):
x[:.0.:.]]
cocf=1/3
base cocfax2 + cocfs=2 cocfs42
return ((xco(((tee//bbase xecoel/base. xecoef/base),
13 def PatchUpsample(k. scale): h, w shape
15
torch, (n, e, scale w. scale) (i,, 1).
vn, scaleshh, scaleow)
17
Implementation and pseudo-inverse Ap
19
if IR "N.mole=="colorii colorizaiooi::
color2gray
Ap gray2color
elir IR mode==" "ood==iiapainting:: 2: zemask
Ap
27
elif mode==" super resolution";
29
torch nn AdaptiveAvg ((256//caa)))) scale scalc)
Ap lambda PatchUpsample(2, scale)
31
"__mo==m=all photo restoration";
33
Al lambda zemask
Alp Al
A2 color2gray
AZp grayZcolor
(2gpiiie ,256// scale))
A3p lambda 2: PatchUpsamplo scale)
41
lambda AB(A2(AI(2)))
43
Ap lambda 2: A)p(A))(A3p(22))
45
Core Implementation DON+ simplified denoising solution
47
For more accurate denoising. please refer to Appendix 1 and the full souree code.
ddmmp.core(xOt, y. sigma.y, sigma.t, a.t):
#lig
if sigma_1 >= a_tasigma_y:
53
lambda.t tas2 (a talambda tosigma 402
55
else:
lambda.t sigma.t/l a..nattt/(...ssgmaa...
57
gamma.1 =
59
#Eg 17
x0t= x0t lambda.t=Ap(y A(x0t))
return x0t. gamma.t
16

---

F DETAILS OF THE DEGRADATION OPERATORS
Super Resolution (SR). For SR experiments in Tab. [ we use the bicubic downsampler as the degradation operator ensure fair comparisons. For other cases in this paper, we use the average- pooling downsampler as the degradation operator, which is easy to get the pseudo-inverse as de- scribed in Sec. 3.2 Fig. 8 a) and Fig. B(b) show examples of the bicubic operation and the average- pooling operation.
We use text masks, random pixel-wise masks, and hand-drawn masks for inpainting experiments. Fig g d) demonstrates examples of different masks.
Inpainting.
Deblurring. For deblurring experiments, We use three typical kemnlss to implement blurring oper- ations, including Gaussian blur kernel, uniform blur kernel, and anisotropic blur kernel. For Gaus- sian blur, the kernel size is 5 and kernel width is 10; For uniform blur kernel, the kernel size is 9; For anisotropic blur kernel, the kernel size 9 and the kernel widths of each axis 20 and 1. Fig demonstrates the effect of these kernels.
Compressed Sensing (CS). For CS experiments, we choose two types of sampling matrices: one is based on the Hadamard transformation, and the other is orthogonalized random matrix applied to the original image block-wisely. For the Haldamrr sampling matrix, we choose 50% and 25% the sampling ratio. For the orthogonalized sampling matrix, we choose ratios from 40% to 5%. Fee and (f) demonstrate the effects of the Walsh-Hadamard sampling matrix and orthogonalized sampling matrix with different CS ratios.
| for each pixel as we described in Sec. 3.2 Fig demonstrates the example of colorization degradation.
Colorization. For colorization, we choose the degradation matrix A =
Solve the Pseudo-Inverse Using SVD Considering we have linear operator A, need to com- pute its pscudo-inverse A to implement the algorithm of the proposed DDNM. For some sim- ple degradation like inpainting, colorization, and SR based on average pooling. the pseudo-inverse At can be constructed manually, which has been discussed in Sec. [.22 For general cases, we use the singular value decomposition (SVD) of A(= UEV to compute the pseudo-inverse Sings
A V where Σ and Σ have the following relationship:
Σ = diaglsi (1, 82, J.
diag d2.
(20)
1
8 #0
d
(21)
5; 0
where s; means the i-th singular value of and d, means the i-th diagonal element of Σ
G VISUALIZATION OF THE INTERMEDIATE RESULTS
In Fig. we visualize the intermediate results of DDNM on 4x SR, 16x SR, and deblurring. Specifically, we show the noisy result x,. the clean estimation Xojt, and the rectified clean estimation Sot=- The total diffusion step is 1000. From Fig. a), we can see that due to the fixed range-space contents A y, Roft already owns meaningful contents in early stages while X; and Xojz contains limited information. But when = O, we can observe that Xojo contains much more details than A'y. These details are precisely the null-space contents. We may notice potential speed-up trick here. For example, we can replace X0j==100 with A y and start DDNM directly from = 100, which yields a 10 times faster sampling. We leave future work. From Fig. (b), we can see that the reverse diffusion process gradually restores images from low-frequency contours to high-frequene details.
17

---

(0) Hicubic
Average- Pooling
CAN MAZ
in ipsum
contectetur
eluamod
elic eaee mag
CHEE
(d)
(e) CS (Walmn-Il)smmne))
CS (Dlock- Based)
(g) Gresscale
Figure 8: Visualization of different degradation operators. (a) Bicubic downsampler. The scale factors from left to right are x4, x8, x 16, x32; (b) Average-pooling downsampler. The scale factors from left right x4, x8, 16, x32; (c) Blur operators. The type of kernels from left to right are Gaussian, uniform, and anisotropic; (d) Masks; (e) Walsh-Hadamard sampling matrix. The sampling ratios from left right are 0.5 and 0.25; (f) Block-based sampling matrix. The sampling
ratios from left to right are 0.4, 0.3, 0.2, 0.1, 0.05; (g) Grayscale operator.
2
GT
Xay
Visullication DDNM 4MSR, with DDPM
sampling stralegy
1000
GT
1- 1000
500
A'y
(b) Visunlization of DDNM on 16xSR and deblurring, with DDIM the sampling strategy
Figure 9: Visualization of the intermediate results in DDNM. Zoom-in for the best view.
18

---

H COMPARING DDNM WITH RECENT DIFFUSION-BASER IR METHODS
Here we provide detailed comparison between DDNM and recent diffusion-based IR methods, in- cluding RePaint (Lugmayr et al. 2022), ILVR (Choi et 2021). DDRM (Kawar al. (022). SR3 (Saharia et al. 2021) D and SDE (Song et al. 2020). For easier comparison, we rewrite their algorithms based on DDPM (Ho al. 2020) and follow the characters used in DDNM. Algo. 3
Algo. [ show the reverse diffusion process of DDPM and DDNM. We mark in blue those that are most distinct from DDNM. All the IR problems discussed here can be formulated
y Ax n,
(22)
where A, represents the degraded image, the degradation operator, the original image, and the additive noise, respectively.
H.1 REPAINT AND ILVR.
RePaint (Lugmayr al., 2022) solves noise-free image inpainting problems, where 0 and A represents the mask operation. RePaint first create a noised version of the masked image y
Yt-1 A(Vār-y - av-ce, CO.II.
(23)
Then uses y to fill in the unmasked regions in X
X = Yt-1 + (I A)
(24)
Besides, RePaint applies an "back and forward" strategy to refine the results. Algo. 5 shows the algorithm of RePaint.
ILVR (Choi et al. 202I) focuses on reference-based image generation tasks, where n = 0 and A
represents a low-pas wassss filter defined by A A A2 (A) is bicubic upsampler and A₂ is bicubic downsampler). ILVR creates a noised version of the reference image x and uses the low-pass filter A to extract its low-frequency contents:
Yt- A(Vor-ix+ az-c,, c~ IO.I).
(25)
Then combines the high-frequency part of x,-1 with the low-frequency contents in y--1
X;-1 ye- A)x
(26)
Algo. shows the algorithm of ILVR.
Essentially, RePaint and ILVR share the formulations, with different definitions of the degra- dation operator A. DDNM differs from RePaint and ILVR mainly in two parts:
(i) Operating on Different Domains. RePaint and ILVR all operate on the noisy x; domain of dif- fusion models, which is inaccurate in range-space preservation during the reverse diffusion process. Instead, we directly operate on the noise-free Xajr domain, which does not need extra process on y and is strictly derived from the theory and owns strict data consistency.
(ii) As Special Cases. Aside from the difference in operation domain, Eq. 24 of RePaint is essen- tially a special case of the range-null space decomposition. Considering A as mask operator, it satisfies AAA A, so we can use A itself as the pseudo-inverse A Hence the range-mull space decomposition becomes & A'y + (I A A!A)x
Ay + (I AA)x y + (I A)x, which exactly the same as Eq. 24 Similarly, Eq, 26 of ILVR can be seen as a special case of range-null space decomposition, which uses I as the approximation of A Note that the final result x of RePaint satisfies Consistency, i.c., Axi = y, while ILVR does not because the pscudo-inverse A they used is inaccurate.
19

---

Algorithm Reverse Diffusion Process of DDPM
Require: None
1: XT ~ NO.,T) I).
2: for T..... do
3: (O( I) if 1, clse = 0.
4: x Va, (x Zo(X,, t) 3.
VI- v1--a))
+ t0E
5: return Xo
Algorithm 4 Reverse Diffusion Process of DDNM Based On DDPM
Require: The degraded image y, the degradation operator A and its pseudo-inverse A
1: XT ~ No.01 I).
2: for = T, ...., do
3:
~ (O, 1) if 1, clse O.
4: Xol: Jāc (x, (x, t)vi Ote)
5: Rof: = x0 A1(xx0 y)
1a Sulz + yaill- X(-1 I-a:
6:
l-ā: X, + 0,E
7: return X0
Algorithm 5 Reverse Diffusion Process of RePaint
Require: The masked image y. the mask A 1: XT ~ I).
2: for [- = T T..., do
3:
for s = 1,..,,
S₁ do E1, E2 ~ NO.. I) ift 1, else c1, E2
4:
5:
yt-1 = Var-ly + V 04 161
6:
X;-1 = 1
Xt -x t)
Rr
+ 0,
Va.
V1-a.
7:
X = -y--1 + A) 1
8:
if + and + S₁ then
9:
X; v1 3, + BB
10:
return X
Algorithm Reverse Diffusion Process of ILVR
Require: The reference image x, the low-pass filter A
1: XT ().
2: for = T,
do
3:
61, NN I) if 1, else E1, 62 = 0.
4: yt-1 = A(vā + /1 (4-166)
5: Xt-1 = Va.
(x -o 1) 9.
v1-mi 0.62
6:
Xt-1 = Yt-1 + (I A)x
7: return X
20

---

Algorithm 7 Reverse Diffusion Process of DDRM
Require: The degraded image with noise level yy the operator A UEVT, A E R2xD
1: XT ~ NN I).
2: y E y
3: for t =
TT, TT,.., do ~~ (=,N(,, if 1, clse E 0.
4:
5:
Xaii - -V Va. (xt - [x( 1lv1 Okt)
6:
for i i= I..,.., Do
7:
i; 00 then
81 -x 01
8:
x = X 0lt
+ V
1f 0
+ 10(-)
01
9:
else 01-1
then
- (0
10:
x = X 0lt
+ v 1
+ 170 n0:-16
0
11:
else 0 2 then
x = (1) + 0
(1)
_ e0)
13:
Xt-1 = VX
14:
return X0
H.2 DDRM
The forward diffusion process defined by DDRM is
X Xo + o16; 6~ 10,
(27)
The original reverse diffusion process of DDRM is based on DDIM, which is
X Xn + v 11-1pot-
0/ Xt X0
(28)
Oe
For noisy linear inverse problem = Ax + n where n NO,, DDRM first uses SVD to decompose A UΣVT, then use y E'UTy y and Xoj= V x for derivation. Each element in y and Roje corresponds to singular value in E(the noncxistent singular value is defined as O), hence it is possible modify Xajı clement-wisely according to each singular value. Then can yield the final result Xg by Xo = VX.- Algo. describes the whole reverse diffusion process of DDRM.
For noise-free(oy = 0) situation, the final result x, DDRM is essentially yielded through a special range-null space decomposition. Specifically, when 0 and σy 0, can rewrite the formula of the i-th element of X as:
0
8;
(29)
8 + 0
To simplify the representation, we define diagonal matrix Σ₁:
Σ
0,
8; =
(30)
1.
+
Then we can rewrite 8g as and yield the result xo by left multiplying V:
X Eny + (I En |8o11
(31)
Xo VX, VEI + V X
(32)
This result is essentially a special range-null space decomposition:
Xo VΣ, + V(I - X011
-V V - V Xoo1
(33)
=VEUT y (I VΣ, V (oo1
=Aly (I- A
21

---

(I A A) x is the null-space part. However for our DDNM, A can be any linear operator as long as satisfies AAIA = A, where
Now clearly that VEIý -Ay is the sange-ppace part while V(I Σ1 |
VEIUT a special case.
Due to the calculation needs of SVD, DDRM needs to convert the operator A into matrix form. However, common operations in computer vision are in the form of convolution, let alone A as compound or high-dimension one. For example, DDRM is difficult to handle old photo restoration. Rather, our DDNM supports any linear forms of operator A and A', as long as AA A = A is satisfied. It is worth mentioning that there exist diverse ways of yielding the eseudo-inverss A and SVD is just one of them. Besides, DDNM is more concise than DDRM in the formulation and performs better in noise-free IR tasks.
H.3 OTHER DIFFUSION-BASED IR METHOVS
SR3 (Saharia et al. 2021 is task-specific super-resolution method which trains a denoiser with y as an additional input, i.e., ZØ (x,, f,y). Then follow the similar reverse diffusion process in DDPM (Ho al.) 2020) to implement image super-resolution, modify the network structures to support extra input y and needs paired data train the conditional denoiser Zo Xx, t, y). while our DDNM is free from those burdens and is fully zero-shot for diverse IR tasks. Besides, DDNM can be also applied to SR3 to improve its performance. Specifically, we insert the process of DDNM, the range-null space decomposition process, into SR3, yielding Algo Results are demonstrated in Fig TO We can see that the range-null space decomposition can improve the restoration quality by ensuring data consistency.
is shown in Algo. g SR3 needs to
Algorithm 8 Reverse Diffusion Process of SR3
Require: The degraded image y
XT I). 2: for = T..... ..... do
3: E NN ift 1, else c O.
4: Xr-1 Voa X Zo(x,, y. v) V1-aa) + oiE
31
5: return X
Algorithm Reverse Diffusion Process of SR3+DDNM
Require: The degraded image
1: XT NN00111 I).
2: for = T..... do
3:
.N(0, 1) if 1, clse 0.
4: Xol= Vie (xi Zo (x(x t, 1.y)v āt)
5: Roj: = N0 At At(Ax1 yy
var- 6: Xt-1 I-st-
Sule + vai(1-ār-1)
I-ãi X, + 0rE
7: return Xo
Algorithm 10 Reverse Diffusson Process of SDE (conditional)
Require: The condition the operator A and the rate À
1: Xy (O. I).
2: for t = T,..., do
3:
61, (O I) ift 1, else 61, E2 0.
4:
= (
5:
9r x,- ,/, xx Zo t)
0,62
V1-au
6: return
22

---

LR
SR3
23.62 23.52/10.524
27.54 7.696
24.36 10.227
22.11 16.190
22,03 14,216
21.56 14.577
22.80: 13,905
22.20 13.319
SR3 DDNM
15.10 8.889
28.14 6,549
24.91 88700
23,31 13,402
22,50/ 12,016
22,00/ 11,799
23,21 11,429
22.40/ 11,252
Figure 10: DDNM can be applied to SR3 to improve the restoration performance. Here we experi- ment on 8x SR (from image size 16x to 128x 128), the metrics are PSNR/Consissencyy
Song et al. (0200) propose conditional sampling strategy in diffusion models, which we abbreviate as SDE in this paper. Specifically, SDE optimize each latent variable x, toward a specific condition f(Ax, and put the optimized x, back to the original reverse diffusion process, as is shown in Algo. 10 y is the condition and A an operator with f(;,.) measures the distance between Ax, and y.
is worth noting that DDNM is compatible with extra sources of constraints the form of operation 5 in Algo. TO For example, results in Fig. and Fig. are generated using the diffusion model pretrained on ImageNet with classifier guidance.
I SOLVING NOISY IMAGE RESTORATION PRECISELY
For noisy tasks y Ax+nnn NNO. oII), Sec. 3.3 provide a simple solution where A in is
approximated as N(0,0?1) However, the precise distribution A NO a(AAt(At)T) where the covariance matrix usually non-diagonal. To similar principles in Eq. 19 need to orthodiagonalize this matrix. Next, we conduct detailed derivations.
This solution involves the Singular Value Decomposition(SSDDD which decompose the degrada- tion operator A and yield its pseudo-inverse A!:
A = U.VT,
-'=VE'U
(34)
A RdxD At R
Dxd U Baxd V E DxD RddD
Σ E Dxd
(35)
1
# 0,
E diag $1, $2, sa). E Si
E00
(36)
0,
8; 8
To find out how much noise has been introduced into Kofts first rewrite Eq.
17 Laa
o Xof-
E,At FAxoi Ax - n),
(37)
where Ax represents the clean measurements before adding noise. Σ, = VA is the scaling matrix with A₁ diagf d1z, Aez. ADD)- Then
rewrite the additive noise as CyCn
where 6n (0, Now Eq. 37 becomes
XOfo
E,At Ax) + VA. Aten.
(38)
23

---

(Ax)e Ax) denotes the clean part of &oje (written as x It is clear that the noise introduced into Roje is oy VA, V A A'en. The handling of the introduced noise depends on the sampling strategy we used. We will discuss the solution for DDPM and DDIM, respectively.
where Xoje
The Situation in DDPM.
When using DDPM as the sampling strategy, we yield x;-1 by sampling from pxxx-1 Xt, Xn) = (x,- Xt, Xn), T). i.e.,
vā.- 1 Xape Vasl az-1) Xt + O1E, 6~ (0,1),
X
(39)
1
0t
Considering the introduced noise, we change orE to ensure the entire noise level not exceed (N of1). Hence we construct a new noise Enew ~ (0, P,I). Then the Eq. 39 becomes
v var-18 B8
o/e + Varl Ō(-1)
_,-1 1 at
X, + Cintro + Cnew:
(40)
1 - āt
Vūl-1 3
Eintro
By VA.V A en:
(41)
1 ōr
Eintro + 6new ~ (0, oI
(42)
Cintro denotes the introduced noise, which can be further written as
V -- αt
Eintro
By VA, V Aten
(43)
04 31
~ ~NO0 1- ōi
V AVVAVVA
(44)
/ 3r
~NO0,
(45)
04 31
~ NO,
VA V T VE'U T VAVVV)
(46)
1 δ
V 04 3r
~NO0,
Σ
A.V
(47)
1 Ci
The variance matrix of Eintro can be simplified as VD.V with D. diag del dez.
ddn):
- Pog
Eintro ~ NO, VD. du
8 + 0, 8: = 0
(48)
To construct Enew, we define new diagonal matrix Ty(= diagtnu (711, 7e2. NDD)):
r, = -- Djs
TH o0
87
8 + 0,
(49)
00:
8;
Now we can yield Grew by sampling from AO VT, to ensure that Cintro + Cnew (O, V(D, + = If.). An easier implementation method is firstly sampling Etemp from (O r)) and finally get knew Veremp- From Eq. 49 we also observe that A guarantees the noise level of the introduced noise do not exceed the pre-defined noise level dr, so that we can get the formula of d in Ex(= VA A = diag Ae1, A12, >.D)))
1,
1-61
det
Joy
(50)
var-18
5t
Joy
8: 0
The Situation in DDIM. from X, becomes:
When using DDIM as the sampling strategy, the process of getting X;-1
X,-1
Vo, Vou-18011 o. 0. 1 Z0 20 (x,, 1) oone, E~,
(51)
24

---

where de -I- d;-1 is the noise level of the t-th time-step, Zo is the denoiser which estimates the additive noise from x₁ and control the randommess of this sampling process. Considering the noise part is subject to a normal distribution, that is, 0rvi - 172 20 (x+, t) + ofEE ~ (N, so that the equation can be rewritten
x vāe-1Xoje + Corigs Corig (0
(52)
Considering the introduced noise, we change Corig to ensure the entire noise level not exceed (O I). Hence we construct a new noise term Enew ~ (0, P,I):
X,-1 = v0g-18 Gintro + knews
(53)
Cintro - VA.V Atcn,
(54)
Eintro + Enew NO I.).
(55)
Cintro denotes the introduced noise, which can be further written as
Cintro
vāi-19, V A'E
(56)
-NO0, (VA.V (VA,VT A
(57)
NN(00 VA,, V A'(A)) VAVV V0 VVV VE'U" VVVVVV
(58)
(59)
M/ a00 (ET) A
(60)
The variance matrix of Eintro can be simplified as VD,V with Dj diagtdel dez, ,
8; + 0,
Cintro N'O, VD,VT), du
(61)
0,
=
To construct Cuews we define a new diagonal matrix Te(= diag (ee,, Ye2," ... , :DD))
o 0
103
8; + 0,
P r== = - DD, Yu =
(62)
8; =0
Now we can construct Enew by sampling from NO0, VT, to ensure that Eintro Enew ((O, V(D, +
II). An easier implementation is firstly sampling étemp from N r2) and finally get nnww V Weeem- From Eq. 62 also observe that Å guarantees the noise level of the introduced noise do not exceed the pre-defined noise level og so that we can get the formula of A (=(= VA AA diag(de1 / A22. AdD))):
1ar- Tt
A
savyI- 1-
1ay-1
(63)
10y
5;
0,
In the actual implementation, we have adopted the following formula for étemp and it can be proved that its distribution (o)
103 (1)
>
Vi--10y
(4)
of
8: 0t
Ceemp
(1))
Or lny
(64)
8:
oIV'1 - -
(2))
0,
where Cemp denotes the i-th element of the vector etemp and IO.
(1)
Note that the blue y is not necessarily needed, By our theory in Sec. 3.3 y should be 0 to maximize
the preservation of range-space correction. But inspired by DDRM Kawar al. 2022). we find that involving 1 help improves the robustness, though sacrificing some range-space information.
25

---

J ADDITIONAL RESULTS
We present additional quantitative results in Tab. 5] with corresponding visual results of DDNM in Fig. 11 and Fig. 12 Additional visual results of DDNM are shown in Fig, 13 and Fig. 14 Addi-
tional results for real-world photo restoration are presented in Fig. 15 Note that all the additional results presented here do not use the time-travel trick.
CelebA-HIQ
4x bicubic SR
8x bicubic
16x bicubic SR
Method
PSNR
SSIMT
Conzi
FIDL
PSNRT
SSIM
Cons)
FIDE
PSNRT
SSIMT
Cons)
FIDJ
DDRM
31.63
0.9452
33.88
31.04
28.11
0.9039
3.23
38.84
24.80
0.8612
0.36
46.67
DDNM
31663
0,9450
4.80
22.27
28. 18
0.9043
0.68
37.50
24.96
0.8634
0.10
45.5
ImageNet
bieubic SR
bicubic
16x bicubic
Method
PSNR
SSIMT
Conzi
FID
PSNRT
SSIM
Conzi
FIDU
PSNRT
SSIMT
Consl
FID
DDRM
27.38
0.8698
19.79
43.15
23.75
0,7668
2,70
83.67
20.85
0.6842
0.38
130.81
DDNM
27.46
0.8707
4.92
39.26
23.79
0. 0.7684
0.72
80.15
20.90
0.6853
0.11
128.13
CelehA-HQ
inpointing (Mask I)
inpainting (Mask 2)
inpainting (Mask
Method
PSNR
SSIMT
Consl
FID
PSNRT
SSIM
Conz).
FIDE
PSNRT
SSIMT
Consl
FIDJ
DDRM
34.79
0,9783
1325.46
12.53
38.27
0.9879
1357.09
10.34
35.77
0.9767
21.49
DDNM
35.64
0.9823
0.0
4.54
39.38
0.9915
0.0
2,82
36.32
0.9797
12.46
ImageNet
inpointing
I)
inpainting (Mask 2)
inpainting (Mask
Method
PSNR
SSIMI
Consi
FID
PSNRT
SSIMP
ConeL
FID.
PSNRT
SSIMT
Consi
FID1
DDRM
31.73
0.9663
876.86
4.82
34.60 64.60
0.9785
1036.85
3.77
31.34
0.9439
12.84
DDNM
32.06
0.9682
0.0
3.89
34.92
0.9801
0.0
3.19
31.62
0.9461
9.73
CelehA-HQ
(Guussian)
dehlur (anisotropic)
(uniform)
Method
PSNR
SSIM
Cons
FIDE
PSNR1
SSIMP
ConsL
FID
PSNRT
SSIM1
Cons)
FID_
DDRM
43.07
0.9937
297.15
6.24
41.29
0.9909
312.14
7.02
40.95
0.9900
182.27
7.74
DDNM
46.72
0.9966
60.00
1.41
43.19
0.9931
66.14
2,80
42.85
0.9923
41.86
3.79
ImageNet
(Gaussian)
deblur (anisntropic)
(uniform)
Method
PSNR
SSIMT
Cons
FID
PSNRT
SSIM
Cons)
FID
PSNRT
SSIMT
Conz)
FID
DDRM
43.01
0.9921
207.90
1.48
40.01
0.9855
221.23
2.55
39.72
0.9829
134.60
3.73
DDNM
44.93
0.9937
59.09
1.15
40.81
0.9864
63.89
2.14
40.70
0.9844
41.86
3.22
CelebAAHIQ
(ratio=0.5)
CS (ratio=0.25)
Method
SSIM
Conz]
FIDU
PSNRT
SSIM
Cons)
FID
0,9520
2171.76
1869.03
46.77
DDNM
0.9604
1640.67
15.81
27.56
0,9090
1511.51
28.80
ImageNet
CS (ratio=0.5)
CS (ratio=0.25)
Method
SSIMI
Conz]
FIDU
PSNRT
ConeL
FID,
DDRM
26.94
0,8902
6293.69
25.01
19.95
0.7048
3444.50
97.99
DDNM
0.9106
5564.00
18.55
21.66
0.7493
3162.30
64.68
HeQ
Colorization
Method
PSNR
SSIM
Cous]
FTD.]
DDRM
26.38
0.7974
455.90
31.26
DDNM
26.25
0.7947
48.87
26.44
ImageNet
Colorization
Method
PSNR
SSIM
Cous]
FTD.
DDRM
23.34
0.6429
260.43
36.56
DDNM
0.6550
42.32
Table 5: Comprehensive quantitative comparisons between DDNM and DDRM.
26

---

Super Resolution
Debluring
Inpaininng
Colorization
Compreaed Sensing
Figure 11: Image restoration results of DDNM CelebA-HQ.
27

---

GT
Resaluion
Debluring
Lorem ipsum Lorem ipsum Lovem tisen Lo ummc corem insum Loreth lisum Lore osum Lorem iesum
consectetur a DC ctetur
coreectch orr contectetur ac consectetu at gonderter tetu
tamp eiush & tmp elusmod emmp
eiusmo temp diushood temp elusmaod temp elasmod temp
Inpuinting
dobbre Of et moloreymag et dolere
oelere mag et dolore mag et de defone mag er dalare map
Binkki veniam minim vēniam minim vemam
minim venlami minim lami mimimeennaan minim va
Cimpressed
Figure 12: Image restoration results of DDNM ImageNet.
28

---

GT
Super Rosohition
Colorization
Lorem psum em oumrre
m'd
psuu
Loren orem IssUT
ennaes
consectelur ad
sectetu ne
te costtttuua ac
etett
ssectetur oe
CO
sectetu
etetu c
sesccttuu
elssmad Temp sn teme
mu Temp
Mod
STTTO temp
mooe
Usmrod teme
et Secooore negr
Poooremmaa eD olore mase
one
oloze ma
NE
olore
dolore mas
Impainting
mnnimmve am mam
Vee
miseni am
an
Comprnssed Sensing
Figure 13: Noisy image restoration results of + on CelebA-HQ. The results here do not use
the time-travel trick.
29

---

GT
Colorization
Super Resolution
Lorem ipsum storem-ppsnt de em ipsum cLorem ips mre dlai hoan dl (osum coo com inspmm it torem tosum
consectetu tur acco eccontctttur acto ssactetur acconsecte tur expneectt acconse tesur acconse cessecistur acto hsectetur ad
elusmod tempeiusm emp rempeiusmed temp usmmae Tei eiu almd
lusmoo temp eiusmod teme
Inpsioting
etdglare gre pageet dolore mag dolore magiet datere magnet colore
dolere magi Feolora ma
magiet geeet colorra mogr minim veniam
mamii yorkia minim veniam minim veniamminnn veniam minim venia minim veniam
Compreased Sensing
Figure 14: Noisy image restoration results of D on ImageNet. The results here do not use the time-travel trick.
30

---

Y
DDNM
DDNM
Figure 15:
Restoring real-world photos using DDNM. y represents the degraded images collected from the internet. The results here do not the time-travel trick.
31


Reviews:
[{"review_id": "wZXNCQ1-48m", "reviewer": "Reviewer_hK3i", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed denoising diffusion probabilistic model for image restoration. They specially applied null-space projection at every iteration to target variable x and used A^T y as range spaced values. By considering only null-space contents in reverse process, they achieved realistic images with data consistency. They demonstrated several image restoration applications: super-resolution, inpainting, colorization", "summary_of_the_review": "In general, this paper is well written and supported by remarkable results. The idea using null space component for residual components in reverse process in DDPM seems to be interesting. However, their approach is similar with the paper from ICLR 2022(above). Unless I see the performance difference with the ICLR 2022 paper or proper explanation of distinguishable points, I would be a doubt on effectiveness of null space projection. \n\nAlso, I have a concern that reconstructions from severe conditions go beyond image restoration and become closer to arbitrary image synthesizing. To avoid such concerns, the results from supervised learning need to be demonstrated together.", "strength_and_weaknesses": "They introduced null-space decomposition during reverse process in DDPM which helps to pursue robust data consistency and realistic image reconstruction. The detailed descriptions toward it are given in Algorithmic tables and associated illustration. It's easy to understand. Their supported results in Figure 3 and 4 are quite good qualities. The results seem to be beyond the bar of quality in ICLR.\n\nHowever, I'm a bit afraid that the result seems to be far from original contents, which brings fake information such as figure 3 at 32x super-resolution on CelebA. It looks like a posterior sample not like a reconstruction results. In my opinion, 32x SR results can not be called as reconstruction, it's a synthetic image. DDNM provides 32x32 pixels out of 1pixel, and it's hard to avoid referring contents from synthetic textures of training manifold. Therefore, especially for face datasets, I couldn't accept it as recovered result. It is closer to synthesizing images irrelevant with ground-truth. Thus, I recommend the authors to provide residual images and ground-truth images  in super resolution example and other examples, too.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hGhTGK4zgHT", "reviewer": "Reviewer_9BqY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a novel method for zero-shot solution of image restoration problem formulated as linear inverse problems by modifying denoising diffusion models. In particular, they propose to only update the null space of the forward operator during the reverse process of the DDPM. Overall, the idea is novel, clearly explained and a significant amount of good experimental results on various problems is presented to support the work.", "summary_of_the_review": "The paper presents a novel method based on DDPMs for zero-shot image restoration. The technique is novel and shows very competitive performance against other techniques based on generative models. The work is very valuable as an alternative to supervised training of discriminative models but a point of comparison between them should be included.", "strength_and_weaknesses": "Strengths:\n- Interesting and novel idea to specialize the use of pretrained DDPMs for zero-shot solution of linear inverse problems\n- Excellent experimental results on various tasks \n\nWeaknesses:\n- It would be interesting to add a comparison of the proposed method with supervised discriminative models (e.g., SwinIR) trained for a specific restoration task to understand how far behind generative models are\n- The authors acknowledge that computational complexity is a limitation of DDNM. However, they should also provide some experimental metrics such as the method runtime compared to existing methods", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OkWQ5aR8R0", "reviewer": "Reviewer_NtKw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposed a novel zero-shot framework for linear image restoration, and the key module is denoising diffusion null-space model. It only requires a pretrained diffusion model, without further training and optimization. Experimental results show that the proposed method outperforms sota zero-shot restoration methods in many tasks.  ", "summary_of_the_review": "Considering the novelty of this work and the good performance, I recommend accept.  \n\nAfter rebuttal, the authors have addressed my concerns. Therefore, I still recommend accept.  ", "strength_and_weaknesses": "Strength:\n1.\tThis work proposes a unified theoretical framework for linear image restoration problems. Specifically, the sample x is decomposed into the range-space and null-space of A, which is the linear degradation operation. Then, the null-space is refined iteratively. \n2.\tThe proposed method can be utilized in many different tasks and the results are promising. \nWeaknesses:\n\n1.\tAs demonstrated by the authors, DDNM+ can solve any real-world IR task as long as we we can construct an approximate linear degradation A and its pseudo-inverse A^-1. However, for real image degradation, A is most likely non-linear. For example, realistic noise in sRGB domain is nonlinear. If we directly utilize this method to solve real noise removal problem, I guess the performance will be heavily degraded. \n\n2.\tThe computing complexity. The authors should give the inference time to let readers aware of the computing complexity of the proposed method. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "M7iZuh_pLE", "reviewer": "Reviewer_exTL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel zero-shot framework for solving diverse linear image restoration (IR) problems, named Denoising Diffusion Null-Space Model (DDNM). Firstly, it provides detailed theory analysis for using a pre-trained diffusion model to solve linear IR problems. Secondly, the proposed DDNM can well solve those IR tasks with arbitrary degrees of degradation and also support solving noisy IR tasks.", "summary_of_the_review": "In my opinion, this paper is marginally above the acceptance threshold based on the strengths and weaknesses.", "strength_and_weaknesses": "Strength: \n\n(1)This paper has detailed background introduction and theoretical analysis about using diffusion model to solve diverse IR problems. \n\n(2)This paper provides convincing proof and introduction about the usage of the generated null-space contents for image restoration.\n\n(3)Another enhanced version DDNM+ is further proposed to handle noisy IR tasks. A time-travel trick is designed to provide better restoration quality.\n\n(4)The provided quantitative and visual results validate the effectiveness of the proposed DDNM, which are superior over existing state-of-the-art zero-shot IR methods.\n\nWeaknesses:\n\n(1)The author mention the limitation of time efficiency in the appendix part. How about the memory efficiency?\n\n(2)It is a little too absolute to claim that “solve arbitrary IR tasks with arbitrary degrees of degradation” in the introduction of contributions, since the degradation operators are asked to be linear.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%