DreamClean: Restoring Clean Image Using Deep Diffusion Prior
Authors: Jie Xiao
Ruili Feng
Han Zhang
Zhiheng Liu
Zhantao Yang
Yurui Zhu
Xueyang Fu
Kai Zhu
Yu Liu
Zheng-Jun Zha
Venue: ICLR 2024 poster (2024)
Keywords: Image Restoration
Diffusion Models
Image Prior
Blind Restoration
Super-Resolution
Image Denoising
Colorization
JPEG Artifacts Correction
Primary area: generative models
TL;DR: This paper proposes an unsupervised and training-free blind image-restoration method that uses deep diffusion prior.

Abstract: Image restoration poses a garners substantial interest due to the exponential surge in demands for recovering high-quality images from diverse mobile camera devices, adverse lighting conditions, suboptimal shooting environments, and frequent image compression for efficient transmission purposes. Yet this problem gathers significant challenges as people are blind to the type of restoration the images suffer, which, is usually the case in real-day scenarios and is most urgent to solve for this field. Current research, however, heavily relies on prior knowledge of the restoration type, either explicitly through rules or implicitly through the availability of degraded-clean image pairs to define the restoration process, and consumes considerable effort to collect image pairs of vast degradation types. This paper introduces DreamClean, a training-free method that needs no degradation prior knowledge but yields high-fidelity and generality towards various types of image degradation. DreamClean embeds the degraded image back to the latent of pre-trained diffusion models and re-sample it through a carefully designed diffusion process that mimics those generating clean images. Thanks to the rich image prior in diffusion models and our novel Variance Preservation Sampling (VPS) technique, DreamClean manages to handle various different degradation types at one time and reaches far more satisfied final quality than previous competitors. DreamClean relies on elegant theoretical supports to assure its convergence to clean image when VPS has appropriate parameters, and also enjoys superior experimental performance over various challenging tasks that could be overwhelming for previous methods when degradation prior is unavailable.

Published as a conference paper at ICLR 2024
DREAMCLEAN: RESTORING CLEAN IMAGE USING DEEP DIFFUSION PRIOR
Jie Xiao' Ruili Feng"
Han Zhang
Zhiheng Liu' Zhantao Yang*
Yurui Zhu
Xueyang Ful Kai Zhu Yu Liu Zheng-Jun Zha'
UUnivesitty Science and Technology of China 2 Alibaba Group
"Shanghai Jiao Tong University
ust ustchbxjems wsw.s.cc... edu. ruil fengustccggaall 1.om coo xyfu@ustc.edu.
ABSTRACT
Image restoration poses a garners substantial interest due the exponential surge in demands for recovering high-quality images from diverse mobile camera de- vices, adverse lighting conditions, suboptimal shooting environments, and fre- quent image compression for efficient transmission purposes. Yet this problem gathers significant challenges as people are blind to the type of restoration the images suffer, which, is usually the case in real-day scenarios and is most urgent to solve for this field. Current research, however, heavily relies on prior knowl- edge of the restoration type, either explicitly through rules or implicitly through the availability of degraded-clean image pairs to define the restoration process, and consumes considerable effort to collect image pairs of vast degradation types. This paper introduces DreamClean, training-fre frree method that needs no degradation prior knowledge but yields high-fidelity and generality towards various types of image degradation. DreamClean embeds the degraded image back to the latent of pre-trained diffusion models and re-sample it through a carefully designed diffu- sion process that mimics those generating clean images. Thanks the rich image prior in diffusion models and our novel Variance Preservation Sampling (VPS) technique, DreamClean manages to handle various different degradation types at one time and reaches far more satisfied final quality than previous competitors. DreamClean relies elegant theoretical supports to assure its convergence clean image when VPS has appropriate parameters, and also enjoys superior ex- perimental performance over various challenging tasks that could be overwhelm- ing for previous methods when degradation prior is unavailable.
INTRODUCTION
Image Restoration (IR), which is classic ill-posed inverse problem, aims to recover clean ver- sion from degraded observation. Currently, deep learning- based IR techniques have demonstrated promising performance and dominated this field, which could be broadly categorized into supervised and unsupervised paradigms.
snservised-based IR solutions usually rely on large-scale pre-collected paired datasets to train their models. A major challenge is that they implicitly assume training and testing data should be identi- cally distributed. As result, these methods often deteriorate seriously in performance when testing cases deviate the pre-assumed distribution. In addition, once the underlying degradation model is changed, new dataset needs be re-collected and new model has be re-trained, which can be both time-consuming and costly.
Another prevailing research line is insupervised-based IR approaches. They explicitly make of the degradation model to produce a clean image by solving maximum a posterior problem or
t Corresponding author

---

Published as a conference paper at ICLR 2024
FBCNN
Input
JPEG
Figure 1: Results of JPEG artifacts correction. The image is degraded by multiple non-align JPEG compression with QF {5, 10, 20} and shift {0, 6). FBCNN is a supervised method and DDRM- JPEG is unsupervised solution using the worst QF 5 as the degradation model. Our Dream- Clean is blind to the degradation model. DreamClean can still recover a 1024 x 1024 high-quality image given the extremely destroyed image based on the advanced Stable Diffusion XL.
JPEG artifacts
SR
Debiu
Poisson noise
Rain
Snow
Raindrop
SR with Gauseian noiee
SR with Poieson nciee
Calorization
We propose DreamClcan to solve various image restoration problems without task- specific re-training or assuming the known degradation model. DreamClean can resort the inherent prior of diffusion models to tackle with linear degradation, noisy linear degradation, non- non-linear degrada- tion and complex bad weather degradation. y: the degraded image, æ: our result.
Figure 2:
a posterior sampling problem. For example, DDRM (Kawar al., 2022a) hypothesizes the linear degradation model and relies on the desirable property of linear formulation to sample from posterior distribution. In practice, however, the underlying degradation model may be too complex to estimate or computationally prohibitive to apply (Ongie et al., 2020). In addition, these approaches may not be ready be equipped with diffusions trained in VAE-encoded space (Rombach et al., 2022) since VAE projection may complicate the entanglement between degraded and clean information
To release the generative power of diffusion models from the heavy degradation prior, we propose novel training ffee and unsupervised framework, dubbed DreamClean, for general IR problems. DreamClean bypasses the requirement of paired dataset and can generate samples without explicit implicit assumptions about the specific degradation model, resulting in strong robustness to vast degradation types. As shown Figure 2, DreamClean can tackle with various types of degradation, ranging from typical linear degradation (image coloration, super-resolution, deblurring), noisy linear
"egg, the linear form y Ha of the degradation model in pixel space will not hold in the encoded space.
2

---

Published as a conference paper at ICLR 2024
Inverse Process
Sampling Process
9+0+))
Pr
(o
Degraded images
End status Init status
Results with different sampling
Figure 3: Overview of DreamClean, DDIM and inversion can reconstruct the input image, thus providing informative latents. VPS can guide low-probability latents to move towards vicinal high probability region, which produces clean image samples while maintaining similarity with input degraded images. (Best viewed on screen.)
degradation (Poisson noise, SR with Gaussian and Poisson noise), non-linear degradation (multiple non-align JPEG artifacts correction (Jiang et al., 2021) )), to complex bad weather degradation (rain, snow, raindrop). DreamClean works by, like an experienced human, "imagining" the potential clean image purely based on an input degraded observation.
The key idea behind DreamClean is to search in clean image distribution, which is represented by a diffusion prior, find the clean image while being faithful to the input degraded image. Conse- quently, the first core ingredient of our framework is pre-trained diffusion model. We treat such diffusion model as a solution for an extreme IR problem: it can generate clean images even if all in- formation about the clean image is lost 2. Another key issue to be addressed is to ensure faithfulness to the degraded image. We resort to the inversion of ODE sampling algorithm (e.g., DDIM (Song al., 2021a)) of diffusion model accomplish this goal. As illustrated in Figure 3, through constructing the degraded image, DDIM inversion algorithm can produce series of latents which preserve information about the input image. These latent variables locate in low-probability gion since sampling from the diffusion model generally produces clean images rather than degraded ones. Although these latents cannot restore the clean image directly, they can inherit information from the input image, providing good initializations for subsequent sampling. Inspired by this, we propose Variance Preservation Sampling (VPS) to guide these corrupted low-probability latents to- wards nearby high-probabiltyy region from which clean samples can be generated. In this way, VPS functions as general solution to ensure faithfulness even without knowing the specific degradation model. It is also noteworthy that i) DreamClean does not assume specific form for the underlying degradation model. Therefore, it can integrated with diffusion models pre-trained in pixel space well as VAE-encoded space. As shown in Figure 1, DreamClean lenging multiple non-align JPEG artifacts correction when applied the encoded space of Stable Diffusion XIL (Podell al., 2023); ii) DreamClean is orthogonal previous works which exploit the degradation model to sample from posterior distribution. DreamClean can also make use of the degradation model produce more faithful results. Our method enjoys both elegant theoretical guarantees in convergence and superior performance in many challenging scenarios.
still accomplish the chal-
METHOD
2.1 PRELIMINARY
Denoising diffusion probabilistic models (DDPMs) are latent variable models aiming to learn model distribution p(ma) to approximate the data distribution q(mo)) (Ho et al., 2020). DDPMs comprise T T'step forward diffusion process, which disturbs data by slowly adding Gaussian noise and T-step reverse generative process, which samples data by progressively removing noise. The
A diffusion model generate clean image from standard Gaussian noise.

---

Published as a conference paper at ICLR 2024
log mo
degraded images
p((x))
clean images
4
VPS +5
|nitiol Ye
* Update Y/z with VPS
100
200
300
400
500
600
timesteps
Figure 4: Illustration of the proposed Variance Preservation Sampling algorithm.
Figure 5: VPS drives latent variables high prob- ability region.
forward process is Markov chain which is of the form
T
( (0) = II
(( (x:-1); (=1
( ((t == Nasss v1 1,
where { is the variance schedule. { are latent variables, which we refer to as latents below. A property of diffusion process is that the conditional distribution of 2 given zy of form
q(<ı|<0) = NN(x;; Vāgzo. (1 ā,) where = 1 - 3r, ā.
II x=0
The reverse generative process proceeds by sampling from a Markov chain starting Gaussian noise p(ær) = N (wT;0,
(o (xx)) = g( 12,, vn) (( 20) of
with reparamettrizzation, (10 2o) and of have the closed form
1 Vot
B.
1 Cte Br. of
1 (20 mo)
at, VI āe Eo (2)1;
(4)
1 Okty
Song et al. (2021b) demonstrate the aforementioned reverse process a discretization of a continuous-time stochastic process, described by the following reverse-time stochastic differential equation (SDE)
dæ, [f(()=, 9² al) log ((:)) di g())to,
(5)
where W. is a standard Wiener process in the reverse time, f(t) 1 dlog a(t)) d 1-a(1) and @t) a continuous version ā, For the reverse-time SDE, Song et al. (2021b) dr a(t) further prove that there exists a corresponding probability flow ODE that shares the same marginal distribution:
g(t) = (1
dæ; f(t)
1 (t)V (x.) dt. -
With this probability flow ODE, one can generate image from Gaussian noise and vice versa.
2.2 DREAMCLEAN
DreamClean focuses on exploiting image priors captured by diffusion models pre-trained on large- scale diverse-distributed images. DreamClean restores degraded image by finding clean sample z which simultaneously satisfies i) is faithful to the degraded image; ii) it conforms model distribution of pre-trained diffusion models. Below our strategies towards these constraints.
Faithfulness by ODE Inversion. ODE sampling algorithm is approximately invertible, that is, for given image, one find series of latents, any of which reproduce the input image along
4

---

Published as a conference paper at ICLR 2024
the ODE sampling trajectory (as shown in Figure 3). This property implies that these inverse latents should contain desirable information about the input image. On the other hand, high quality images generated when sample from pre-trained diffusion models, which means these latents lie in low probability region. Inspired by this, as illustrated in Figure 3, we propose to utilize the inverse latents as initialization and design correcting algorithm guide these latents towards nearby high probability region. In this work, we choose DDIM (Song et al., 20212) as the default ODE sampling. Assume the degraded image y, we can find latent y/ by the DDIM inversion
V DDIM - (y)
where DDIM (-) is the inversion of DDIM and 0 < S T" denotes the strength of ODE inverse.
Realness by Variance Preservation Sampling. After getting the informative latent yr. We can correct the fow-probability latent and gradually denoise it to get the clean image. We conduct the following two steps at each timestep E [7.
y m- 1 + m V log (y("" + "g5""
(Variance Preservation Sampling)
m = 1, ... M, y. yt: y1-1 = DDIMStep(y," M
(8)
(Denoise Sampling)
where 17 and Ml are required to satisfy the constraint:
n = == āz), fg (2-
0 < < 1 is a scalar determining the step size and a₂ is the noise schedule defined in Equation (2). Such setting of rn and fg is vital to restore a clean image, which we will discuss later in Theorem 2.2.
Intuitively, as shown in Figure 4, given the initial latent which lies in low- oow-probayility region, VPS guides the latent to move towards its vicinal high-probability region. The high-probability region conforms the normal sampling formulation of diffusion models. Therefore, by correcting latents progressively, VPS can produce high quality images. In practice, V Vog p(y, (mm can be computed by pre-trained diffusion models (Hyvärinen & Dayan, 2005; Karras et al., 2022). Specifically, the gradient term has the relation with the predicted noise by a pre-trained diffusion model Eg:
V log Pe (3)" 10 EB (v." VI Ct
(10)
We argue that when iit and ng is subject to Equation (9), VPS converges to a nearby high probability set, which in turn generates potential clean image x. Since latents of diffusion models are typical of high dimensionality, inspired by the concept of typical set in information theory (Shannon, 1948; Cover& Thomas, 2006), we define the set that gathers most density of zgginduced latents as the following High Probability Set, where æn is clean image.
Definition 2.1 (High Probability Set). For 5 > 0, t > 0 and potential clean image Do E N High Probability Set T. (zz; 8) is defined as follows
T:N (xo; 0) = x2
log Pe (az,a0)) - H <
(11)
N
where Pt (x, (20) = Pt (æt, (o,.). Dz.s and zo, denote the i-th elements of xi and xe respec- fively, and H is the Shannon entropy of N 6.)).
According to law of large numbers, the probability of a, E T.N (aro; 8) gets close 1 for sufficiently large N (see Appendix A.2 for details). In other words, T.N (æ0; 5) is a set comprising of latents of a clean image Do whose probability can be sufficiently large. We can prove that VPS drives latents of degraded images to High Probability Set of nearby clean image under appropriate m, Mg.
Theorem 2.2. For â > 0 and the potential image Do. when hu and ng satisfy the constraint in Equa- tion (9), there exists an inverse time T such that y by Variance Preservation Sampling converges to T.N (xo; d) when M + XX.
We delay proof to Appendix A. Satisfying Equation (9) is vital for clean image restoration, which is demonstrated theoretically in Appendix A.1 and empirically in Section 3.6. Theorem 2.2 implies that at timestep T, VPS is capable of correcting latents of a degraded image to a nearby High Proba- bility Set, which then generates the clean image following sampling dynamics of diffusion models.
5

---

Published as a conference paper at ICLR 2024
400
degraded image
(b) VPS boosted latents
Figure 6: Visualization latents of different timesteps. VPS changes original degraded artifacts to Gaussian-like noise, which conforms the formulation of diffusion models.
In implementation, we fix the inverse strength (e.g., 300 for T = 1000), and take 1-step VPS correction before each DDIM step.
Put them together. In conclusion, DDIM inversion finds informative latent as the initialization and VPS corrects it towards the High Probability Set of a nearby clean image. These two mecha- nisms cooperate achieve the goal of realness and faithfulness.
EXPERIMENTS
3.1
EXPERIMENTAL SETUP
Our experiments consist of: i) verifying that DreamClean optimizes latents to higher probability region (Section 3.2); ii) quantitative comparison with previous methods (Sections 3.3 and 3.4); iil) presentation of visual results across multiple degradation types to demonstrate its strong robustness and generality (Section 3.4); iv) exploiting the degradation model (Section 3.5); We demonstrate that DreamClean is orthogonal to prior works, which can exploit the underlying degradation model to achieve challenging inverse problem (e.g., phase retrieval); v) ablation study on the different schedules of and 77..
3.2 MOVING TO HIGHER PROBABILITY
For the latent variable It, we evaluate its probability under a pre-trained diffusion model by log pe (zz)). Given Vo. we can approximate by the alternative score log qi (æx, 2z)). We use the noisy SR as the default IR task and record the average score and the standard deviation on CclebA 1K. Figure 5 shows that latents of degraded images locate in low-probability region when compared with clean images and VPS gradually promotes their probability. We also provide more intuitive visualization of latents with different timesteps in Figure We can find that driven by VPS, latents with unexpected artifacts transformed to the appearance of a clean image with Gaussian-like noise, which conforms the sampling dynamics (Equation (2)) diffusion models.
3.3 QUANTITATIVE EXPERIMENTS
We validate the efficacy of DreamClean using the diffusion models (Ho al., 2020; Dhariwal & Nichol, 2021) trained on CelebA (Karras et al., 2018), LSUN bedroom (Yu et al., 2015) and Im- ageNet (Deng et al., 2009). For quantitative comparison with previous methods, we perform ex- periments on the classic IR tasks, including linear degradation (noisy image super-resolution) and complex non-linear degradation (multiple non-align JPEG compression artifacts correction). We the average peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) to measure faithfulness and Learned Perceptual Image Patch Similarity (LPIPS) as the perceptual metrics. Following (Kawar et al., 2022a), also report number of function evaluations (NFEs) for each experiment to compare efficiency.
We use ImageNet 1K (Deng et al., 2009), CelebA 1K (Karras al., 2018), and validation set of LSUN bedroom (Yu et al., 2015) with image size 256 256 for validation. We perform com- parison with RED (Romano et al., 2017), DGP (Pan al., 2021), SNIPS (Kawar al., 2021),

---

Published as a conference paper at ICLR 2024
Table Quantitative results SR with Gaus- sian noise a = 0.05 on CelebA.
Table 2: Quantitative results of X with Gaus- sian noise = 0.05 on ImageNet.
Method
PSNR
SSIM
LPIPS
NFES +
Method
PSNR
SSIM
LPIPS 4
NFEs
Baseline
23.64
0.51
0.64
Baseline
21.85
0.47
0.58
0
DGP
18,40
0.40
0.70
1500
DGP
9.50
0.12
0.93
1500
SNIPS
26.38
0.74
0.20
1000
RED
22.90
0.49
NA
100
DPS
24.42
0.70
0.17
1000
DPS
21.42
0.70
0.36
1000
DDRM
29.21
0.83
0.09
100
DDRM
25.67
0.73
0.30
100
DDNM
29.17
0.82
DDNM
25.56
0.30
100
GDP
24.38
0.71
1000
GDP
24.33
0.39
1000
27.23
0.77
0.12
Ours
24.31
0.67
0.40
90
Ours*
30.19
0.84
0.08
Ours*
25.84
0.74
0.23
60
Table 3:
Quantitative results of JPEG compres- Table 4: Quantitative results of JPEG compres-
sion artifacts correction on CelebA
sion artifacts correction on LSUN bedroom.
Method
PSNR
SSIM
LPIPS
NFEs
Method
PSNR
SSIM
LPIPS
NFEs 4
Baseline
24.79
0.69
0.41
0
Baseline
23.39
0.68
0.34
QGAC
24.28
0.68
0.32
QGAC
23.41
FBCNN
26.37
0.77
0.24
1
FBCNN
24.10
0.73
0.31
DDNM
24.40
0.66
0.31
100
DDNM
22.73
0.66
0.33
100
DDRM-JPEG
26.41
0.77
0.20
100
DDRM-JPEG
24.06
0.73
0.32
100
Ours
27.58
0.82
0.20
90
Ours
24.35
0.74
0.31
DDRM (Kawar al., 2022a), DDNM (Wang al., 2023), and DPS (Chung al., 2023) for noisy SR, and QGAC (Ehrlich et al., 2020), FBCNN (Jiang et al., 2021), and DDRM-JPEG (Kawar et al., 2022b) for multiple non-align JPEG artifacts correction. DDRM is reported using 20 NFEs in the original paper. For fair comparison, we re-run DDRM for 100 NFEs. We set DDIM inference steps to 100, the inverse strength 300, 7 to 0.05, and M to Hence, our method requires 90 NFEs when the degradation model is unknown (30 for DDIM inverse, 30 for DDIM, and 30 for VPS).
For noisy SR, we use 4x average-pooling downsampler and additive Guassian noise with o 0.05. For JPEG artifacts correction, simulate the real world scenario by multiple non-aligned compression. Specifically, we used cascaded JPEG compression with QF = (10, 20, 40) whose 8 blocks shifted by (0,3, 3, pixels respectively. We show upscaling by the inverse upsampler as baseline for noisy SR and the compressed image itself as a baseline for JPEG artifacts correction. FBCNN is supervised method and we use the pre-trained model for inference. For DDRM-JPEG and DDNM, we choose the worst case (QF 10) as the degradation model for inference. We use "Ours" to mark the case without knowing degradation model and "Ours*" mark the scenario of leveraging the degradation model.
Tables to show quantitative results. We can find i) for noisy SR, even without knowing degra- dation model, DreamClean can be effective in promoting image quality (compared with baseline) and sometimes surpass those exploiting degradation model (e.g., SNIPS); for complex JPEG arti- facts correction, DreamClean outperforms both supervised (QGAC and FBCNN) and unsupervised methods (DDRM-JPEG and DDNM).
3.4 QUALITATIVE EXPERIMENTS
To verify visual results, we conduct qualitative experiments on the various IR tasks using diffusion models (Ho al., 2020; Dhariwal Nichol, 2021), ranging from typical linear degradation (image coloration, super-resolution, deblurring), noisy linear degradation (Poisson noise, SR with Gaus- sian and Poisson noise), non-linear degradation (multiple non-align JPEG artifacts correction (Jiang al., 2021)), complex bad weather degradation (rain, snow, raindrop). Figure present visual results across various tasks. We can find that DreamClean can produce visually pleasing results while maintaining rather similarity with the input degraded image.
We are also interested in integrating DreamClean with the advanced Stable Diffusion XL (Podell al., 2023). As shown in Figure and Appendix A.7, although the input images severely de-

---

Published as a conference paper at ICLR 2024
(a) Fourier phase retrieval
(b) 4x SR with Gaussian Noise d' 0.05
Figure 7: DreamClean can make use of the degradation model to restore clean images.
Table 5: Ablation study on the fg schedule.
Schedule
PSNR
f
LPIPS
26.31
0.75
0.24
inpul
Mg
2 a.)
V7C2 - r)(1 a.)
22 11 (1 (1 ae)
a.)
26.84
0.72
0.16
27.23
0.12
Figure 8: Comparison of different fl schedules.
stroyed, DreamClean can still generate high-quality images while keeping similar with input images. Please refer to Appendix A.7 for more visual results.
3.5 EXPLOITING DEGRADATION MODEL
Like previous works, DreamClean can also make use of the degradation model to initialize the latents, which is faithful to the input observation. To validate that, include experiments the noisy SR task and challenging Fourier phase retrieval, which aims to restore clean image based the magnitude of Fourier transformation of image. Since degradation model is known, do not require DDIM inverse to keep similarity and instead resort to Equation (2) for fast inverse. Thus, we only need 60 NFEs. Tables and 2 shows quantitative results and Figure 7 presents the visual results. Although little perceptual information can be found in observations for phase retrieval, DreamClean can utilize the degradation model recover clean images. Please refer to Appendix A.6 for more quantitative and visual results on other IR tasks, including noisy inpainting, noisy coloration, and noisy deblurring (Uniform and Gaussian kernel).
3.6 ABLATION STUDY
We here conduct ablation study on the constraint of l and f7 in Equation (9). Suppose ny still has the form 711 āt), we investigate two alternate schedules: f) 0 and f/ 22y(1 a.). The former corresponds to plain gradient ascent and the latter corresponds to vanilla lagevien dy- namics. We conduct noisy 4x SR on CelebA. Table 5 and Figure 8 present quantitative and visual comparisons. We can find that the schedule of VPS achieves the best score and visual result. note that plain gradient ascent cannot produce images. This is because it cannot optimize latents to High Probability Set of a clean image.
4 RELATED WORKS
We briefly summarize dominant deep learning approaches for image restoration problems in two categories: supervised and unsupervised methods.
Supervised Methods. A deep neural network, which be CNN (Zhang al., 2017; Dong al., 2015: Xia et al., 2023; Ju et al., 2021; Hu et al., 2022), Transformer (Liang et al., 2021; Zamir al., 2022; Wang et al., 2022; Zhang al., 2023), Diffusion (Saharia et al., 2022b;a; Whang
8

---

Published as a conference paper at ICLR 2024
et al., 2022) models, etc, is trained to learn to map corrupted images to their clean counterparts under the supervision of a matched degraded-clean dataset (Li et al., 2023). Thanks to powerful representation ability of DNN, supervised methods typically achieve remarkable performance for specific degradation. However, the brilliance comes with a high cost of generality: the performance deteriorates seriously if training samples deviate from the underlying degradation model. Besides, it is also difficult to collect a high-quality dataset if one does not know the true degradation model.
Unsupervised Methods. Unsupervised methods bypass the obstacle of matched degraded-clean training pairs by instead exploiting the prior distribution, which can be learned from data or implied in the intrinsic structure of generator network (Ulyanov et al., 2018; Jagatap & Hegde, 2019). They typically weaken the requirements of matched training pairs to unpaired degraded-clean images (En- gin et al., 2018), only ground truth (Venkatakrishnan et al., 2013) or only degraded images (Lehtinen et al., 2018: Bora et al., 2018; Quan et al., 2020; Huang et al., 2021; Mansour & Heckel, 2023). Since the learning decoupled from specific degradation model, unsupervised methods exhibit high gen- erality (Ongie et al., 2020). They usually utilize the image prior in an iterative procedure. One approach (Venkatakrishhann et al., 2013; Romano et al., 2017; Chang et al., 2017; Sun et al., 2019) is to learn denoiser from data and apply the denoiser in place of proximal operators in an optimization algorithm, which needs to know the degradation model at test time. Another approach is to learn a generative prior based on training samples using generative adversarial networks (GANs) (Goodfel- low et al., 2014). They (Bora et al., 2017; Daras et al., 2021; Pan et al., 2021) optimize the latent input or GAN's weight to minimize the distance between the generated image which is corrupted by the degradation model and input degraded image.
Recently, diffusion models have made significant breakthroughs in image generation. Diffusion models are also widely used to solve various inverse problems in unsupervised way (Choi et al., 2021: Kadkhodaie & Simoncelli, 2021; Kawar et al., 2021; Song et al., 2022; 2021b; Murata al., 2023). These methods treat a diffusion model as image prior and exploit desirable property of pre- assumed degradation model. For instance, DDRM (Kawar et al., 2022a;b) tackles with linear inverse problems and perform diffusion in the spectral space, where missing information can be identified and synthesized. Similarly, DDNM (Wang et al., 2023) proposes a zero-shot solver for linear IR problems by refining only the null-space during the reverse diffusion process. DPS (Chung et al., 2023) and GDP (Fei et al., 2023) leverage the degradation model to guide latent variables to ensure consistency with the degraded image for general non-linear degradation. Different from these works, DreamClean significantly weakens the assumption about the degradation model and is capable of producing clean images even without knowing specific form of degradation. Due to its generality, DreamClean can be integrated in the advanced latent diffusion models. In addition, DreamClean in- herits the inherent advantage, which avoids training on the matched data, of unsupervised methods. These together constitute the promising prospect of DreamClean.
5 LIMITATION AND DISCUSSION
There still remain some limitations. First, although DreamClean can promote visual quality signifi- cantly, it can not ensure strict consistency with the input degraded image without knowing degrada- tion model. An effective mechanism to promote consistency deserves further study. Besides, there are some degraded cases that DreamClean struggles to tackle with. For instance, Appendix A.8 pro- vides such a failure case, DreamClean when using diffusion models pre-trained on ImageNet can not remove haze successfully and tends to generate some unexpected content.
6 CONCLUSION
We propose a novel unsupervised method named DreamClean for general IR problems. Dream- Clean figure out novel avenue to tackle with various degradation types even without supervised training on paired images or assuming specific form of degradation model. DreamClean enjoys ele- gant theoretical guarantees and achieves remarkable performance across various degradation types, especially for extremely destroyed scenarios. Thanks to its generality, DreamClean also makes possible to hamesss the advanced generative models such Stable Diffusion XI..

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work was supported by the National Natural Science Foundation of China (NSFC) under Grants 62225207 and 62276243.
REFERENCES
AbdeIrahman Abdelhamed, Stephen Lin, and Michael S. Brown. A high-quality denoising dataset for smartphone cameras. In Computer Vision and Pattern Recognition, 2018.
Ashish Bora, Ajil Jalal, Eric Price, and Alexandros Dimakis. Compressed sensing using genera- tive models. In International Conference Machine Learning, 2017.
Ashish Bora, Eric Price, and Alexandros G Dimakis. Ambientgan: Generative models from lossy measurements. International Conference on Learning Representations, 2018.
J.H. Rick Chang. Chun-Liang Li, Barnabás Póczos, B. B.V.K. Vijaya Kumar, and Aswin C. Sankara- narayanan. One network to solve them all solving linear inverse problems using deep projec- tion models. In International Conference on Computer Vision, 2017.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. ILVR: Conditioning method for denoising diffusion probabilistic models. In International Conference on Computer Vision, 2021.
Hyungjin Chung. Jeongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong Chul Ye. Diffusson posterior sampling for general noisy inverse problems. In International Conference on Learning Representations, 2023.
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory (Wiley Series in Telecom- munications and Signal Processing). Wiley- Ieley-interscience, 2006.
Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Transactions on Image Processing, 2007.
Giannis Daras, Joseph Dean, Ajil Jalal, and Alexandros G Dimakis. Intermediate layer optimization for inverse problems using deep generative models. In International Conference on Machine Learning, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems, 2021.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015.
Max Ehrlich, Larry Davis, Ser-Nam Lim, and Abhinav Shrivastava. Quantization guided JPEG artifact correction. In European Conference Computer Vision, 2020.
Deniz Engin, Anil Genç, and Hazim Kemal Ekenel. Cycle-dehaze: Enhanced cyclegan for single image dehazing. In Computer Vision and Pattern Recognition Workshops, 2018.
Ben Fei, Zhaoyang Lyu, Liang Pan, Junzhe Zhang, Weidong Yang, Tianyue Luo, Bo Zhang, and Bo Dai. Generative diffusion prior for unified image restoration and enhancement. In Computer Vision and Pattern Recognition, 2023.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems, 2014.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
10

---

Published as a conference paper at ICLR 2024
Xiaobin Hu, Wenqi Ren, Jiaolong Yang, Xiaochun Cao, David Wipf, Bjoemn Menze, Xin Tong, and Hongbin Zha. Face restoration via soug-and-llyy 3D facial priors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Tao Huang, Songjiang Li, Xu Jia, Huchuan Lu, and Jianzhuang Liu. Neighbor2Neighbor: Self- supervised denoising from single noisy images. In Computer Vision and Pattern Recognition, 2021.
Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score match- ing. Journal of Machine Learning Research, 2005.
Gauri Jagatap and Chinmay Hegde. Algorithmic guarantees for inverse imaging with untrained network priors. In Advances in Neural Information Processing Systems, 2019.
Jiaxi Jiang, Kai Zhang, and Radu Timofte. Towards flexible blind JPEG artifacts removal. In International Conference on Computer Vision, 2021.
Mingye Ju, Can Ding. Charles A. Guo, Wenqi Ren, and Dacheng Tao. Idrlp: Image dehazing using region line prior. IEEE Transactions on Image Processing, 2021.
Zahra Kadkhodaic and Eero Simoneelli. Stochastic solutions for linear inverse problems using the prior implicit in denoiser. In Advances in Neural Information Processing Systems, 2021.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Advances in Neural Information Processing Systems, 2022.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. SNIPS: Solving noisy inverse problems
stochastically. In Advances in Neural Information Processing Systems, 2021.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In Advances in Neural Information Processing Systems, 2022a.
Bahjat Kawar, Jiaming Song. Stefano Ermon, and Michael Blad. JPEG artifact correction using denoising diffusion restoration models. arXiv preprint arXiv:2209.118888 1888, 2022b.
Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2Noise: Learning image restoration without clean data. In International Con- ference on Machine Learning, 2018.
Chongyi Li, Chun-Le Guo, Man Zhou, Zhexin Liang, Shangchen Zhou, Ruicheng Feng, and Chen Change Loy. Embedding Fourier for ultre-high-definition low-light image enhancement. In International Conference on Learning Representations, 2023.
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. SwinIR: Image restoration using swin transformer. In ICCVW, 2021.
Youssef Mansour and Reinhard Heckel. Zero-shot noise2noise: Efficient image denoising without any data. In Computer Vision and Pattern Recognition, 2023.
Naoki Murata, Koichi Saito, Chich-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Yuki Mitsufuji, and Stefano Ermon. GibbsDDRM: A partially collapsed Gibbs sampler for solving blind inverse problems with denoising diffusion restoration. In International Conference on Machine Learning, 2023.
Gregory Ongie, Ajil Jalal, Christopher A Metzler, Richard G Baraniuk, Alexandros G Dimakis, and Rebecca Willett. Deep learning techniques for inverse problems in imaging. IEEE Journal on Selected Areas in Information Theory, 2020.
Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo. Exploiting deep generative prior for versatile image restoration and manipulation. IEEE Transactions on Patlern Analysis and Machine Intelligence, 2021.

---

Published as a conference paper at ICLR 2024
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXir:2307.01195, 2023.
Yuhui Quan, Mingqin Chen, Tongyao Pang, and Hui Ji. Self2Self with dropout: Learning self-
supervised denoising from single image. In Computer Vision and Pattern Recognition, 2020.
Yaniv Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by denoising (red). SIAM Journal Imaging Sciences, 2017.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Computer Vision and Pattern Recog- nition, 2022.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans. David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In SIGGRAPH, 2022a.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022b.
C. Shannon. mathematical theory of communication. The Bell System Technical Journal, 1948.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 202 la.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021b.
Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with score-based generative models. In International Conference Learning Representations, 2022.
Yu Sun, Brendt Wohlberg, and Ulugbek S Kamilov. An online plug-and-play algorithm for regular- ized image reconstruction. IEEE Transactions on Computational Imaging, 2019.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Computer Vision and Pattern Recognition, 2018.
Singanallur V. Venkatakrishnan, Charles A. Bouman, and Brendt Wohlberg, Plug-and-play priors for model based reconstruction. In IEEE Global Conference on Signal and Information Processing, 2013.
Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In International Conference on Learning Representations, 2023.
Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: A general U-shaped transformer for image restoration. In CVPR, 2022.
Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Pey- man Milanfar. Deblurring via stochastic refinement. In Computer Vision and Pattern Recognition, 2022.
Bin Xia, Yulun Zhang, Yitong Wang, Yapeng Tian, Wenming Yang, Radu Timofte, and Lue Van Gool. Basic binary convolution unit for binarized image restoration network. In International Conference on Learning Representations, 2023.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: / 003660 2015.
12

---

Published as a conference paper at ICLR 2024
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming- Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. Computer Vision and Pattern Recognition, 2022.
Jiale Zhang, Yulun Zhang, Jinjin Gu, Yongbing Zhang, Linghe Kong, and Xin Yuan. Accurate image restoration with attention retractable transformer. In International Conference on Learning Representations, 2023.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond Gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE Transactions on Image Processing, 2017.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 PROOF TO THEOREM 2.2
Proof. In each iteration of VPS defined by Equation (8), yi" yi--1 is updated by gradient term
" V log !) and noise term " c- Note that the gradient V Pl (y:) for any yv RN can be written as
logp (3r)
(A1)
pr(v)) / /V..((bul
1
(A2)
1
/
Van. Vt
1- pe(ysl=z)p
(A3)
pr(y))
1 ā. (v (a, - y:),
(A4)
where Pe (32 (x) the expectation of clean image a conditional on yrs and it can be expressed as
NWn Vāja, ( (o( is the density of clean image distribution, E(x [u]
Pr(Bye (x) (x)dd (V))))
E a: vol
(A5)
Nyyi Vazz (1 - ax)I) JN (yis Votar (1 Or II I)po (xc' )dæ Po (æ)d
(A6)
exp
2(1 - 11ye Va 12
x
Po (2(2 da.
(A7)
exp 2()1)a)) Ve Va 113 po(x"'da'
Suppose yt is a combination of an image zu and a Gaussian noise e in the form of y v Vā +
vI â,e, then 3/: v || in Equation (A7) be approximated
l1 Va 0g ag x: + (1 ã. |||| + 01 āt)c (zo - x)
(A8)
0e a x + (1 a
(A9)
where we reasonably assume that the noise term is approximately orthogonal to xo x. Similarly, have
1/
Vo Rs ãz|| - x' 1|2 + (1 ax)
(A10)
Substitute Equations (A9) and (A10) into Equation (A7).
get approximation of E[<|y,] as
exp
2(1 a.)
(āzll - 21 + (1 - po (xr yda
E[E [c]
po(x))
(A11)
f exp
(āx
a²1 + (1 - ax))
2(1 )
[ - x01
exp
3(1-a:) ci)) 2
Po (xc do
(A12)
exp
200
xx 12
Po co )clar
2(1 ac))
N(xo x.
1-ā:
N(00 x', I- I) (xr )da )o)
de
(A13)
di V my log (arr) + mo.
(A14)
αe
where
( / /N x,
1 --a T)ppo(r)da (
(A15)
5e From Equation (A14), it clear that EE(x)// is approximately only dependent of the image compo- nent Wo in yt-
14

---

Published as a conference paper at ICLR 2024
Thus, get approximation of V (o)b)( by substituting y vārzo + v1 - ōyc and Equation (A14) into Equation (A4)
V mo
log (xo))
V Pog 2
(A16)
Van
vI āt
With Equation (A16) and the relation y!- vā the updating rule Equation (8) as
m
1 ā,e"
we can obtain y from
yi
2
400
m--
1K V log ry (x))
+ (V1 0
111 m "7
(A17)
Oe
v1 Oz
=Vā
40
m-
+
" V log
r (2)
+ +v1 -aac"
(A18)
ae
where the equality holds because the linear combination of two independent Gaussian noises still Gaussian, and their variances satisfies
ar (vI ai
(A19)
ae
which is guaranteed by the relationship of m and My defined by Equation (9).
Compare Equation (A18) with y.m=1 = / mn + VI aLe""! we can find that the variance of the noise component keeps unchanged.
As for the image component æz"' the updating rule x x m--- au Vlogro(>)) exactly the gradient ascent, with the optimization target log rr (3ro) and step size a local maximum of log ( (æo). denoted as zÿj that satisfies
o, m. Thus, 2z"" will converge to
V Voorr 0.
(A20)
Note that (xo) represents the distribution induced by adding a Gaussian noise with variance 1-a. to the clean image distribution Po: as defined by Equation (A15). Thus, for small which cor- responds to small variance, the maxima of log r3(a)) will coineide with clean images, under the mixture of Dirac assumption on image distribution.
&e
Finally, we can conclude that yrM will get into T. (ar;: (6+() for some small بة and sufficient large M by verifying that
Po Pe (3r (xx)
(A21)
N
1
1
1
N
log
(2T(1 ar)) exp
( ār)
Von M + v1 ore - Varalll
(A22)
=
log 2n - -āt) + 2N ax))
VV - x)) + vI 11
(A23)
2
1
-, 2 log 2 σz) + 51.
(A24)
where that last limitation holds because x.M z M increases, 84 11e" 1) is small
because l|e || = 1 for large N, which is guaranteed by the high dimensionality of images and the N law of large numbers.
A.2 THE PROPERTY OF HIGH PROBABILITY SET
We define new variable ve, Ft 04 Since elements of æ; conditional independent given Zp. { are independent and identically distributed and Ut, ~ N (0; 1 - 1ā))) Suppose H is Shannon entropy N (0; ā.), according to weak law of large numbers,
have
log Pt (a))200) N
log Pt (Ir,o, Ft.,,... Ji,N 11 Tt,1,... FN,N)
(A25)
N
1
N log Pt (Vr.o- De.l....
(A26)
>> H in probability
(A27)
15

---

Published as a conference paper at ICLR 2024
for sufficiently large N. For nigh-reeolutton image, N is typically large. Therefore, High Proba- bility Set contains most of probability.
A.3 EXTENSION TO REAL-WORLD APPLICATION
We extend DreamClean to complex real-world applications. First, perform real-world im- age denoising on SIDD dataset (Abdelhamed et al., 2018). For quantitative comparison, we test original unprocessed data scores (Baseline) and evaluate the classic BM3D (Dabov al., 2007) as comparison. Table A reveals that DreamClean can effectively tackle with real-world noise (+-8.26 dB compared with Baseline). Moreover, Figure A1 qualitatively presents results of DreamClean more applications. including restoring real-world bad weather corrupted images, old photo restora- tion, and real-world image denoising.
Table A1: Quantitative results on SIDD.
SIDD
PSNRT
SSIM
LPIPS $
Baseline
23.66
0.35
0.58
BM3D
25.65
0.68
N/A
Ours
31.92
0.76
0.23
(a) Bad weather
(b) Old photo restoration
(c) Real- world image denoising
Figure Al: Extension real-world applications. y: the degraded image, æ; our result.
A.4 VISUALIZATTIO OF DDIM AND VPS
The visualization in Figure A2 intuitively illustrates the respective function of the VPS and DDIM step respectively. We find that after VPS correction, the original degraded artifates translate
16

---

Published as a conference paper at ICLR 2024
to Gaussian-like noise. Therefore, VPS step is responsiable for correcting the corrupted low- probability latents. Moreover, after DDIM step, the amount of noise decreased progressively. Thus, DDIM step is responsible for progressively reducing the amount of Gaussian noise contained in latents.
120
160
200
240
280
320
360
400
(a) Latents of an degraded image
DDIM Step
:aariance Preservation Sampling
(b) Visualization of latents of DreamClean
Figure A2: Visualization of latents DDIM and VPS. VPS translates original degraded artifacts to Gaussian-li like noise and DDIM step is responsible for progressively reducing the amount of Gaussian noise contained latents.
A.5 ALGORITHM
We present DDIM inversion in Algorithm A1, Variance Preservation Sampling algorithm in Algo- rithm A2 and DreamClean algorithm in Algorithm A3.
Algorithm A1 DDIM Inversion
Require: Vr
Require: a pree-trained diffusion model Cp
1: y
2: for t0 t do
3: yt+1 +- Vāc+i 1/ "1-vI- + VI
ast1Ea (Yr, t)
vā,
4: end for
5: return yr
Algorithm A2 Variance Preservation Sampling
Require: M, flg Ut Require: a pre-trained diffusion model Eø
3/ t-g y:
2: for m = 0 to M do
3: yi yim+ <- y.th 1i c
+ 7g
VI-aa
4: end for
5: return y M
A.6 EXPLOITING DEGRADATION MODEL
DreamClean is orthogonal to previous works that make use of the degradation model. To validate that, we perform experiments on classic noisy linear tasks including uniform deblurring, deblur- ring with Gaussian kernel, inpainting and colorization with Gaussian noise 0.05 ImageNet
17

---

Published as a conference paper at ICLR 2024
Algorithm A3 DreamClean
Require: T, M, !it 7g
Require: a pre-trrinee diffusion model Ep
1: < 2/ #DDIM inversion, producing the latent y/r
2: for [
do
y/++ f Vāc+1 N1 (i-v--aico(Mi)) + vI ast1En (Ye.
3:
4: end for
5: for = T to 1 do
6:
yr f- y/. #Variance Preservation Sampling, no change to
7:
for m to M do <oly"",0
8:
u Vi 1II VI-ai
+ 6
9:
end for
10: YE - yid #DDIM Step, from to --1
11-Vi-ās 1- sen (2) + VI ac-150 (yr,t)
11:
- va--1
vā:
12: end for
13:
return yo
1K (Deng et al., 2009) and CelebA 1K (Karras et al., 2018). We conduct Variance Preservation Sampling on null-space (Wang et al., 2023). Tables A2 and A3 present the quantitative results. Figure A10 presents visual results.
Table A2: Quantitative results on ImageNet.
ImageNet
Deblurring(uniform)
Deblurring(gauss) PSNRt/SSIM//LPIPS.
Colorization LPIPS
Inpainting
Method
PSNRY/SSIM/LPIISL
PSNRY/SSIM//LPIPSL
Baseline
18.35/0.26/00.m
17.79/0.31/0.71
0.54
12.32/0.46//0.
DPS
N/A
24.64/0.67/0.3 60.30
N/A
22.14/0.773 73.0.26
DDRM
25.09/0.71, /0.0000 /0.30
27.82/0.80/0.21
0.25
23.09/0.83/0.133
DDNM
24.28/0.65/00400
26.43/0.75/00.29
0.33
23.12/0.82/0.13
Ours
26.78/0.755 /.77///..5//.33
28.92 28.92/0.82/00.
0.11
23.18 10488/0.3//0111
Table A3: Quantitative results on CelebA.
CelebA Method
Dehlurring(uniform) PSNR1/SSIM/ILPIPS
(ehlurringggauss)) PSNR
Colorization LPIPS
Inpainting
PSNRY/SSIMIJALPIIS.
Baseline
19.21/0 19.21/0.31/0.0
18.06/0.34/0.73
0.61
40/8//0.4//044
DPS
N/A
/0.8300.8100.11
N/A
02.72/0.82// 80.
DDRM
28.06/ /0.600.00// /0.00/0.011
40.52/0.85/0..0
0.13
03.24/0.85//0.
DDNM
28.98 00.12/..
30.37 00.1/
0.13
23.23 23.23/0.85
Ours
81.17/0.88/0.13
32.71/0.91/0.09
0.10
24.71/0.8700.0
A.7 MORE VISUAL RESULTS
We present more visual results on various tasks using diffusion models (Ho et al., 2020; Dhariwal & Nichol, 2021) in Figure A9 as well as the Stable Diffusion XI. (Podell et al., 2023) in Figures A7 and A8. DreamClean exhibits strong robustness about degradation types and compatibility with diffusion models.
A.8 FAILURE CASE
We present a failure case of in Figure A3. DreamClean fai Is to remove haze and tends to generate unexpected results using diffusion models (Dhariwal & Nichol, 2021) pre-trained ImageNet.
18

---

Published as a conference paper at ICLR 2024
Figure A3: Failure case of our method.
Input
DPS
DORM
CDP
Durs
Ours
Reforence
Figure A4: Visual comparison of 4x SR with = 0.05.
B
H B
H
H
P
H
Input
FBCNN
DDRM-IPEC
Ours
Reference
Figure A5: Visual comparison JPEG artifacts correction.
A.9
ON THE ACCUMULATION VPS CORRECTING EFFECT
From the previous deduction Appendix A.I, we can find that
y!
Van
Fi +
1i v a2
log re re(
+VT C2
(A28)
The denoising process can be written as
Vār-1
1
a
y
yi
Go (t( + 0,61-1
(A29)
VI
19

---

Published as a conference paper at ICLR 2024
DPS
Input
Figure A6: Visual comparison of 32x SR with 0.1 using Stable Diffusion XL. DDNM and DPS generate unrelated content because they use guided-diffusion pretrained on ImageNet.
The denosing network (0 (), by definition,, will predict the noise term y we have
co co(vi: ; t) = Et,
(A30)
M V log ri (100)
v /0-11/ 1/, vit at
yt-1 Vor-1
80g +
61
(A31)
at
V1 11-0s δ2
at-1
Et + o1E111
(A32)
m v log (ogr((ao)) +
αt a:-1
= -
Vo +
El + ofE:11
(A33)
ā v1
Now if we take noise strength is
1-as- (1
which is the most popular setting, we can find the final
T-o,
2 áq (1 (4-1)² 12
1 1 1--11-1 (1
āi = 1
(A34)
α4-1 ō
0t-1
So we can find after denoising sampling step following the Variance Preservation Sampling, we have
M Voog + + a1-1E:-1
9:-1
V
2p +
(A35)
āz
(+)
04 Po (1)
-1-16-111
(A36)
We can conclude that during the Variance Preservation Sampling and denosing sampling, the yr se- quence is actually updating æb prediction results. In each step, VPS corrects the corrupted oo component by the gradient term !l V log (x()), while the denoising step preserves the correcting & effect. Thus, the correcting effect VPS can be accumulated along the sampling process.
20

---

Published as a conference paper at ICLR 2024
y
Figure A7: Noisy SR results of DreamClean using Stable Diffusion XL. The image resolution is 1024 x 1024. y: the degraded image, æ: our result.
21

---

Published as a conference paper at ICLR 2024
x
y
X
Figure A8: JPEG artifacts correction of DreamClean using Stable Diffusion XL. The image resolu- tion is 1024 x 1024. y: the degraded image, Φ; our result.
22

---

Published as a conference paper at ICLR 2024
(a) Possion noise
(b) Debiur
(c) JPEG artifacts
(d)
Gaussian noise
(g) SR with Gaussian noise
(h) Snow
Figure A9: DreamClean can tackle with linear degradation, noisy linear degradation, non-linear degradation and complex bad weather degradation in blind way. r: the reference image, the degraded image, and æ: our result.
23

---

Published as a conference paper at ICLR 2024
(a) Colorization
(b) Deblur with Gaussian noise
(c) Inpainting
(d) SR
Figure A10:
DreamClean can make use of the degradation model reconstruct clean images. r:: the reference image, y: the degraded image, æ: our result.
24


Reviews:
[{"review_id": "6fSrkdXX3x", "reviewer": "Reviewer_aVXX", "summary": "This paper presents DreamClean, a training-free method for high-fidelity image restoration without prior knowledge of degradation. DreamClean embeds the degraded image into pre-trained diffusion models, uses Variance Preservation Sampling (VPS), and outperforms previous methods, especially in challenging tasks without degradation priors.", "questions": "Refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The innovation is impressive, providing a fresh perspective on the field of image restoration。", "weaknesses": "1. Insufficient experiments. The performance needs to be tested on a broader range of real-world degradation data. The data used in the paper are all synthetic, which lacks convincing power.\n2. The methods for comparison are insufficient. For example, other training-free methods like 'Generative Diffusion Prior for Unified Image Restoration and Enhancement' are also available.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BGUWWIw2dW", "reviewer": "Reviewer_NQCS", "summary": "The paper proposes a novel unsupervised diffusion based approach called DreamClean, which does not rely on specific degradation types, for Image Restoration tasks. The superiority lies in restoring clean images without specific finetuning or assuming known degradation. Specifically, DreamClean exploits the approximately inversible property of DDIM to ensure faithfulness. For realness, the paper introduces Variance Preservation Sampling (VPS) to guide latents to nearby high probability region which conforms statistics of the pretrained diffusion model to produce high-quality images. The paper gives a solid theoretical support for the convergence of VPS. The experiments and analysis are sufficient and the results seems promising.", "questions": "- In section 3.2, the authors should give a more detailed interpretation about  log⁡(q(x_t |x_0)) can be alternative score for log⁡(p_θ (x_t)).", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The paper gives a simple but insightful conclusion for the existing IR methods and points out the advantage of unsupervised method for generalization. Based on this, DreamClean is proposed to mitigate the reliance on the underlying degradation model or data-specific finetuning, which is critical to improve the generalization and the practical. \n- DreamClean can improve image quality using pre-trained diffusion models without finetuning or assuming specific degradation formulation. This method fills this gap in IR field. The results on various degradation types and latent diffusion model seems to support its strong robustness. \n- The analysis of the paper is mathematically complete. The paper constructs a high probability set according to the diffusion’s statistical characteristic and proves that latents will converge to the high probability set under VPS. This perspective is interesting and novel.\n- The experiments are sufficient enough to support the claims and the results are promising. In particular, the result in Figure 1 has much higher visual quality compared with previous methods. The paper validates the efficacy on various IR tasks including single and multiple degradation cases, and well compatibility with latent diffusion models.", "weaknesses": "- The proposed method leverages ODE inversion to ensure faithfulness with motivation that ODE can approximately reconstruct the input image. Given that, why is ODE inversion needed rather than directly conduct VPS on the input image y?\n- The authors should explain the reason that fresh noise is need in VPS. In other words, without the noise, VPS returns latents with locally maximal probability density. Why do higher-density latents perform inferior compared to latents with noise?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "or9zcWoIrr", "reviewer": "Reviewer_HBur", "summary": "This paper works on unsupervised image restoration using the pre-trained diffusion models as deep diffusion prior. It relaxes the linear degradation model and extends to non-linear and bad weather degradation. The faithfulness of the image restoration is achieved via the rich image prior embedded in the pre-trained models and DDIM inversion. The realness is satisfied through the proposed variance preservation sampling, where the latents are driving to the high probability set. Experiments for classical super-resolution, deblurring, colorization, non-align JPEG compression, and bad weather restoration show the effectiveness of the proposed method.", "questions": "- Is the proposed method sensitive to some parameters, e.g., $\\gamma$ and strength?\n- The paper should give more visual illustrations for the sampling algorithm of different timesteps to understand the effectiveness of the two steps in the proposed VPS.\n- For the bad weather, the training of diffusion models often encounters such images in high image quality. It is not evident that the images with bad weather are categorized as degenerated or clean from the perspective of the diffusion prior.\n- The authors should provide more implementation details.\n- It is better to explain $x$ and $y$ in Fig. 2.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper works on an important problem: unsupervised image restoration.\n- The paper proposes a method to drive the latent variables to a high probability set to generate high-quality restoration results and provide theatrical analysis.\n- The proposed method generalizes to non-linear degradation and bad weather as well as applies to diffusion models with latent space, e.g., Stable Diffusion.\n- Well written and easy to follow.", "weaknesses": "- For the proposed variance preservation sampling (VPS), the latent variables fall into the high probability set with a strong assumption that $M$ is sufficiently large (Theorem 2.2). However, $M$ is empirically set to $1$ in this paper, and there is no detailed discussion about this important parameter.\n- The qualitative results are only shown for the proposed method, and there are no visual comparisons to existing methods.\n- For the bad weather degradation, there are no quantitative measurements. The only evidence is a set of visual examples in Fig. 2.\n- The ablation study is only conducted on one type of degradation and is insufficient.\n- The paper should discuss and compare with more recent blind image restoration works, e.g., GDP.\n\nFei, Ben, et al. \"Generative Diffusion Prior for Unified Image Restoration and Enhancement.\" CVPR. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x4O2VrxW7N", "reviewer": "Reviewer_6gxh", "summary": "The paper introduces a novel generic image restoration (IR) approach called DreamClean that recovers degraded images back to high-quality reconstructed images without heavily relying on or knowing degraded models. The DreamClean method consists of two essential steps to restore corrupt images (e.g., noisy images, low-resolution images, or artifacts generated by JPEG compression)to the reconstructed images. First, the method utilizes a pre-trained diffusion model to encode the corrupt image images to latent embeddings in the diffusion models. The paper then presents a novel sampling strategy, Variance Preservation Sampling (VPS), mimicking a process of generating clean images to guide low-probability latent toward nearby high-probability regions for high-quality image synthesis. The contribution of this paper is threefold: 1. the paper introduces a novel aspect of restoring clean images without knowing prior degraded model information. The proposed method could be an IR technique for image restoration applications with variations of camera devices. 2. the proposed method could be generalized to diverse degraded images without training multiple neural network models. 3. the paper evaluates the proposed method on multiple datasets and types of degraded images. The experiment shows DreamClean achieves competitive quantitative results compared with other IR methods.", "questions": "There is a question I would like to ask to clarify all doubts in the paper:\n\n1. In section 3.3, which is quantitative experiments, the paper uses 100 NFEs rather than a configuration in the original paper for DDRM. Does it affect the quantitative comparisons because 100 NFEs may not be the configuration for DDRM?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The strengths of this paper are as follows:\n\n1. This paper proposes a generic IR method that can tackle corrupt images generated by variations of degraded models without training multiple network models. The paper provides theoretical and empirical proof to demonstrate the DreamClean method can synthesize high-quality clean images given the corrupt images.\n\n2. The paper proposed a novel sampling strategy, the VPS method, in diffusion-based models to improve the chance of yielding clean images. The paper demonstrates the capability of VPS through theoretical proof and empirical results. The experimentation shows that DeamClean achieves higher quantitative results under different image quality metrics.\n\n3. This paper contains comprehensive quantitative results on image quality comparisons for distinct IR methods. The DreamClean method outperforms other methods on most degraded models on different datasets.", "weaknesses": "This paper contains two weaknesses in terms of the proposed method:\n\n1. The DreamClean method can synthesize clean images given the corrupt images; however, the proposed method does not consider consistency between the corrupt images and the reconstructed clean images due to a lack of prior degraded models. The proposed method can fail some restored images by synthesizing unexpected clean images.\n\n2. The proposed method utilizes the pre-trained diffusion model on a particular large-scale dataset (i.e., ImageNet) to embed the corrupt images to the latent embeddings. Then, the method applies VPS to sample clean images during the sampling phase in diffusion models. Due to using the pre-trained model, the proposed method can generate unexpected images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)