Title: A Unified Conditional Framework for Diffusion-based Image Restoration
Venue: NeurIPS 2023
Accepted: yes
Keywords: image restoration

diffusion model

denoising

deblurring

JPEG restoration
Abstract: Diffusion Probabilistic Models (DPMs) have recently shown remarkable performance in image generation tasks, which are capable of generating highly realistic images. When adopting DPMs for image restoration tasks, the crucial aspect lies in how to integrate the conditional information to guide the DPMs to generate accurate and natural output, which has been largely overlooked in existing works. In this paper, we present a unified conditional framework based on diffusion models for image restoration. We leverage a lightweight UNet to predict initial guidance and the diffusion model to learn the residual of the guidance. By carefully designing the basic module and integration module for the diffusion model block, we integrate the guidance and other auxiliary conditional information into every block of the diffusion model to achieve spatially-adaptive generation conditioning. To handle high-resolution images, we propose a simple yet effective inter-step patch-splitting strategy to produce arbitrary-resolution images without grid artifacts. We evaluate our conditional framework on three challenging tasks: extreme low-light denoising, deblurring, and JPEG restoration, demonstrating its significant improvements in perceptual quality and the generalization to restoration tasks. The code will be released at https://zhangyi-3.github.io/project/UCDIR/.

A Unified Conditional Framework for Diffusion-based Image Restoration

Abstract

Diffusion Probabilistic Models (DPMs) have recently shown remarkable performance in image generation tasks, which are capable of generating highly realistic images. When adopting DPMs for image restoration tasks, the crucial aspect lies in how to integrate the conditional information to guide the DPMs to generate accurate and natural output, which has been largely overlooked in existing works. In this paper, we present a unified conditional framework based on diffusion models for image restoration. We leverage a lightweight UNet to predict initial guidance and the diffusion model to learn the residual of the guidance. By carefully designing the basic module and integration module for the diffusion model block , we integrate the guidance and other auxiliary conditional information into every block of the diffusion model to achieve spatially-adaptive generation conditioning. To handle high-resolution images, we propose a simple yet effective inter-step patch-splitting strategy to produce arbitrary-resolution images without grid artifacts. We evaluate our conditional framework on three challenging tasks: extreme low-light denoising, deblurring, and JPEG restoration, demonstrating its significant improvements in perceptual quality and the generalization to restoration tasks.

1 Introduction

Image restoration tasks are ill-posed problems whose solutions are not unique. But, most existing methods for image restoration rely on learning regression models that produce deterministic results. Unfortunately, such approaches often learn the 'averaged' results of potential ground truth distribution, leading to blurry outputs that are not consistent with human perception. With the development of Diffusion Probabilistic Models (DPMs), there exists remarkable progress in image generation, which is able to generate realistic images with fine details. This progress has shed light on diffusion-based image restoration tasks, which utilize conditional DPMs to generate images that align well with the corresponding ground truth. With the input degraded input, conditional DPMs can produce realistic images with significant improvements in perceptual quality.

When applying DPMs to image restoration, the crucial challenge lies in how to effectively integrate degraded images and other conditional information ( e . g ., diffusion time step, degradation type, and strength) into the DPMs. A better conditional framework can greatly enhance the generative capacity of DPMs and guide them to generate realistic images that faithfully align with the ground truths. However, it has been mostly overlooked in existing diffusion-based methods [50, 42, 41, 12]. Typically, these methods simply concatenate the degraded image as the input to the diffusion model, which fails to fully exploit the potential of the diffusion model.

In this paper, we present a unified conditional framework for diffusion-based image restoration. It effectively integrates multiple sources of conditional information into blocks of DPMs. This

∗

integration enables spatially adaptive conditioning over blocks of the diffusion models, resulting in perceptual quality improvements of restored images. As shown in Fig. 1 (left), the framework consists of two models: an initial predictor, implemented as a lightweight UNet, and a diffusion model. The initial predictor generates the initial restoration output, capturing the coarse structure and deterministic components of the final output. The diffusion model is then employed to be guided by the initial result to predict the residual of the initial guidance. Since the initial predictor restores the initial results of the final output, we leverage it as the spatial guidance to the block of DPMs. This spatial guidance, along with other scalar conditional information, is integrated into the diffusion model blocks, enabling precise and adaptive conditioning during the restoration process.

To enhance the handling of conditional information within each block, we propose an Adaptive Kernel Guidance Block (AKGB) that effectively fuses the conditional information into DPMs. The key idea is to generate adaptive kernels for each spatial location by fusing a set of learnable kernels based on conditional information. Specifically, we encode both spatial information and auxiliary scalar information into the feature maps. These two types of information are then fused as multi-source fusion weights. The fusion weights are used to linearly combine a set of learnable kernels, resulting in fused kernel weights for each position. This fusion process takes into account both the spatial and scalar conditional information, enabling each position within the diffusion model to have its own customized kernel weights. It efficiently injects the condition information into the diffusion model and enhances its spatial adaptability and flexibility. This approach allows the models to effectively utilize the conditional information, leading to improved performance in handling complex image restoration tasks.

In practical applications, another important problem is to adopt diffusion models to generate highresolution images [41]. However, direct applying the patch-based method often introduces noticeable artifacts due to inconsistent results along the patch boundaries. Inspired by the properties of image denoising models, we propose an inter-step patch-splitting strategy to perform patch-splitting at each diffusion time step. This strategy achieves consistent and artifact-free results in the restoration process for arbitrary-resolution images.

We evaluate this framework on three challenging image restoration tasks: extreme low-light denoising, image deblurring, and JPEG restoration. The experiments demonstrate that our method generalizes well to all three tasks and, achieves significantly higher perceptual quality than strong regression baselines and recent diffusion-based models.

Our contributions can be summarized as follows:

We propose a unified conditional framework for diffusion-based image restoration tasks. It leverages a UNet to predict the initial guidance and enable integrating the multi-sources conditional information to every block to better guide the generative model.

To effectively incorporate conditional information into diffusion models, we design a basic module and an Adaptive Kernel Guidance Block (AKGB). It combines the spatial guidance and auxiliary scalar information to adaptively fuse the dynamic kernels in each diffusion model block.

A simple yet effective inter-step patch-splitting strategy is proposed for handling highresolution images in low-level vision tasks. This practical strategy enables diffusion models to generate consistent high-resolution images without grid artifacts.

Through extensive experiments on extreme low-light denoising, image deblurring, and JPEG restoration tasks, we demonstrate that our method not only achieves a significantly higher perceptual quality than strong regression baselines and recent diffusion-based models but also show good generalization to various restoration tasks.

2 Related Work

Our method draws inspiration from both regression-based and generative methods in image restoration, as well as dynamic networks.

2.1 Generative Image Restoration

Image restoration. Image restoration has been a highly active research topic in computer vision for several years. Due to the ill-posed nature of the problem, each degraded image has multiple potential solutions. Traditional methods have employed various priors and degradation models [18, 56]

to mitigate the challenges in image restoration. Traditional methods often rely on priors such as sparsity [17, 32], non-local self-similarity [13, 5], and total variation [40] to regularize the restoration process. While traditional methods have made significant progress, they struggle with complex or challenging cases. With the rise of deep learning, learning-based methods have become the mainstream approach in image restoration [53, 8, 11, 29, 28, 1, 6, 36, 55, 30]. These methods leverage large-scale training datasets and deep neural networks to learn the mapping between degraded inputs and clean outputs. Learning-based methods have shown remarkable performance improvements by implicitly capturing complex image patterns and structures. While learning image restoration from a large dataset shows promising performance improvements, they can only produce deterministic results for this ill-posed problem.

Generation-based methods. To mitigate the limitations of predicting deterministic solutions, a promising avenue is the adoption of generation-based methods. Unlike traditional approaches, these methods generate a distribution for a given degraded input, offering a more flexible and probabilistic solution. One popular approach in generation-based image restoration is the use of generative adversarial networks (GANs). Although several GAN-based methods have been successfully applied to specific domains such as face restoration [44] or super-resolution [45], generating artifacts-free and realistic outputs for general image restoration remains a considerable challenge for GAN-based methods.

An emergent and promising direction for generation-based methods lies in the utilization of diffusion models. These models have showcased impressive performance in diverse domains such as image generation [39, 14, 20], image editing [19, 4], and super-resolution [42]. In image restoration, there are several methods have been proposed. S3 [42] proposes conditional Diffusion Probabilistic Models (DPMs) to handle image super-resolution. Then, it has been extended to other image restoration tasks [41]. For image deblurring, DvSR [50] proposes a residual learning framework and icDPM [38] improves the out-of-distribution performance through a multi-scale representation. Most existing works simply concatenate the conditional information into the diffusion models, which largely limits the generation ability of diffusion models.

In addition to supervised methods, there exist unsupervised approaches [26, 25, 48] that decompose the image representation from different perspectives and directly integrate the degraded image into a pretrained diffusion model. Although these methods eliminate the need for retraining, they often suffer from poorer performance compared to the task-specific training method.

2.2 Dynamic Networks in Image Restoration

The significance of adaptive processing in image restoration tasks, taking into account factors such as image texture, degradation types, and degradation strength, is widely acknowledged in the field. In line with this concept, several learning-based methods have been proposed to generate dynamic kernels [22] for image restoration.

One track of the work is to employ a network to predict the spatially invariant convolution kernels [2]. This approach aims to capture the varying characteristics of different image regions and adapt the convolution operations accordingly. Building upon this foundation, KPN [34] extends the kernel prediction network into the burst image denoising task. The predicted kernels are utilized to aggregate the input burst images directly, enabling improved denoising performance and better image texture preservation. Instead of predicting the dynamic kernel directly, BPN [51] reduces the complexity of the kernel prediction idea by decomposing the predicted kernel matrix. Malleable convolution [24] improves the efficiency of kernel prediction by producing low-resolution spatial kernels and interpolating them for each position. By incorporating adaptive processing techniques, these methods demonstrate the ability to effectively adapt to different image restoration scenarios, leading to enhanced restoration quality and performance.

Our proposed method shares a similar underlying principle with dynamic kernels, as it leverages the integration of multi-source conditional information. These include the degraded input, diffusion time step, and degradation information. By incorporating these factors, our method dynamically generates adaptive kernels that effectively handle feature maps within each block.

UNet

LayerNorm

Conv

Swish

Diffusion Model Block

time

…

DyConv

Initial Predictor

Degraded image

Swish

LayerNorm

guidance

Interpolation

Conv

SimpleGate

Conv

Linear

Swish

Linear

Guidance Block

Initial

output

Noise

Input

Diffusion Model

Output

Conditional Integration Module

Basic Module

Restored Image

3 Method

3.1 Overview of Conditional Framework

Our objective is to design a unified conditional framework for image restoration tasks. The input conditional information for this framework consists of two components: the degraded image and auxiliary scalar information. The degraded image represents the image to be restored, while the auxiliary scalar information can include degradation types, strength, or other relevant details specific to each restoration task.

To enhance the integration of conditional information, we first employ a lightweight U-Net to predict an initial output as shown in Fig. 1 (left). This initial output captures the low-frequency and deterministic aspects of the final restored image, which are easier to restore and contain essential structural information. We utilize this initial output as the spatial guidance for the diffusion model. Together with the auxiliary scalar information ( e . g ., degradation type, diffusion time step), we inject them into every block of the diffusion model, enabling better control and guidance for the diffusion models. This injection not only provides a comprehensive context but also enhances the flexibility of our framework. Following DvSR [50], we employ the diffusion model to capture the residual distribution of the initial output. Consequently, the training loss is modified as

where the noise vector ϵ ∼ N (0 , I ) , ( x , y ) is the sampled degraded-clean image, f θ denotes the diffusion model, u ϕ represents the initial predictor, and γ ∼ p ( γ ) is a distribution related to the noise schedule. The highlight blue part in Eq. (1) shows the key modification of the 'residual modeling'.

3.2 Diffusion Model Block

Basic Module In our approach, we have designed a basic module for the diffusion model used in image restoration tasks. We aim to keep the module as simple as possible by leveraging existing image restoration backbones. Instead of utilizing complicated operators, we try to make it as simple as possible by adopting existing image restoration backbones. For each block, we leverage two convolution layers. Before each of them, we introduce a LayerNorm to stabilize the training process [15]. Following the previous generative works [3, 42, 50], we employ the Swish as the activation function. A shortcut is applied to perform the residual learning. To enable the injection of the conditional information, the second convolutional kernels are designed to be dynamic based on the conditions.

Conditional Injection Module To better integrate the conditional information into the block, we propose a Conditional Integration Module (CIM). Within CIM, the guidance is first scaled to match the resolution of the feature map within the block. This scaled guidance then passes through two

Multi-source

Fusion Weights

time

…

guidance

Adaptive Kernel Guidance Module

! ∈ #

!×#×$

$ ∈ #

%×%×$

% ∈ #

!×#×$

#

%×%×$

&

&

∈ #

'×'×(×(

'

(&,+)

∈ #

'×'×(×(

( = 3

(", $)

Diffusion Model

Noise

Sample

Patch Splitting

+

Patch Merging

Diffusion Model

Noise

Sample

Patch Splitting

Patch Merging

Inter-step Patch-splitting

·

+

convolution layers with a SimpleGate activation [8], effectively adjusting the channel number and producing the feature map G . Simultaneously, the auxiliary scalar information is processed through a two-linear-layer branch with Swish activation in between, resulting in the feature map S .

The feature maps G and S are subsequently passed to the Adaptive Kernel Guidance Module (AKGM) to generate dynamic kernels for the second convolution layer in the basic module, as depicted in Fig. 1. The key idea behind the AKGM is to adaptively fuse the kernel bases, allowing each spatial location to handle the feature map based on the fused multiple sources of conditional information. As shown in Fig. 2 (left), in each AKGM, we have N learnable kernel bases denoted as W b ∈ R C × C × k × k , where C represents the number of channels, and k denotes the kernel size. These kernel bases are trained to handle different cases and scenarios. The scalar information ( e . g ., time) is encoded by a two-linear-layer branch with Swish activation in between. The feature maps G ∈ R H × W × N and S ∈ R 1 × 1 × N are fused using pointwise production to generate the multi-source fusion weights M ∈ R H × W × N . Here, H and W denote the height and width of the feature map, respectively, and N represents the number of kernel bases. To obtain the fused kernel for a specific position ( i, j ) , denoted as F ( i,j ) , it is obtained by linearly fusing the kernel bases according to the multi-source fusion weights at the same position. Formally, it can be expressed as

Fig. 2 illustrates the dynamic kernel generation process for a specific position ( i, j ) with the number of kernel bases set to N = 3 .

Computational analysis. In the standard convolution operation, given an input size of H × W × C , the computational cost is HW ( k 2 C 2 ) MACs. However, for the Adaptive Kernel Guidance Module (AKGM), calculating the dynamic kernel for each position introduces additional computational complexity. If we use the convolution kernels as the kernel bases, the computational cost is HW ( NC 2 k 2 ) + NC 2 k 2 , which increases linearly with the number of kernel bases. To reduce the computational burden, we adopt grouped convolution kernels with a group number of N as the kernel bases. As a result, the computational cost is reduced to HW ( C 2 k 2 ) + C 2 k 2 . The core operation in AKGMhas a comparable computational cost to a convolution layer. Besides, the additional workload introduced by the module is minimal, as the number of channels is set to a small number.

3.3 High-resolution Image Inference

In theory, constructing the diffusion model using pure convolution layers allows for arbitrary-size image inference. However, in practice, it is impossible to infer a very high-resolution image directly due to the limited GPU memory [41]. As shown in Fig. 2 (right), a common approach in image restoration is to split the input image into overlapping patches and independently perform inference on each patch. The output patches are then merged to reconstruct the entire image. However, when

NAFNet

Ours

GT

Restormer

using generative models, inferring patches independently often results in noticeable grid artifacts along the boundaries of the splitting patches. Even with a large overlap, the neighboring patches are not consistent in the splitting region.

For high-resolution images, directly adopting the diffusion model on the latent space [39] sacrifices the detailed textures when downsampling the image features to the latent space. Another strategy mask-shift [47] is proposed to alleviate the local incoherence but requires significant computational cost. To address this problem, we propose a simple yet effective inter-step patch-splitting strategy for high-resolution image inference. As shown in Fig. 2, we integrate the patch-splitting technique into diffusion iterations to support high-resolution image generation. Specifically, at each diffusion time step, the input image is split to small overlapped patches. Then, the diffusion model performs denoising on the patches, and the overlapping regions of the results are dropped directly when merging the patches. Since the denoising model focuses on the local receptive field, performing the patch-splitting at the denoising step would not introduce obvious boundary artifacts. Some visualizations can be found in Fig. 6.

4 Experiments

4.1 Tasks and Datasets

We evaluate our method on three challenging tasks. For extreme low-light denoising, we use the SID Sony dataset [7], which provides noisy-clean image pairs with an exposure factor of 300. It consists of high-resolution images ( 2120 × 1416 ) captured in various indoor and outdoor scenes. For deblurring, we follow DvSR [50] to train and test on the GoPro dataset. For JPEG restoration, we train on the ImageNet and follow DDRM [26] to test on selected 1K evaluation images [37].

4.2 Experimental Settings and Metrics

To demonstrate our method to be a unified framework, we train the framework under the similar setting without task-specific hyper-parameter tuning. We used the AdamW optimizer with a learning rate of 1 × 10 -4 , and EMA decay rate is 0.9999. In the training, we used the diffusion process with T = 2000 steps with the continuous noise level [10]. During the testing, the inference step is reduced to 50 with uniform interpolation. For each task, we provide a regression baseline that adopts the same architecture and training setting for comparison. We train each task for 500k iterations with batch size 32. The training process takes approximately three days to complete when utilizing 8 A100 GPUs.

In this paper, we focus on the perceptual quality of the results. Four perceptual metrics, LPIPS, NIQE, FID, and KID are adopted for evaluation. LPIPS and NIQE are reference-based and non-reference metrics respectively. For NIQE, different implementations produce different results. In this paper, we adopt it from the BasicSR [46] for all methods. For the inception distance FID and KID, we follow the previous paper [33, 50] to extract patches and compute the FID and KID on patch level to obtain stable evaluation results. We also show the distortion metrics, PSNR and SSIM, for reference.

Ours

Restormer

DvSR

GT

4.3 Comparisons on Extreme Low-light Denoising

To evaluate our method's performance on extreme low-light denoising, we compare it with the official implementations of transformer-based and convolution-based backbones [52, 8, 49] on SID dataset [7]. As the SID dataset consists of high-resolution images, we utilize our inter-step patchsplitting strategy to perform full-resolution image inference without grid artifacts. The quantitative results, presented in Tab. 1, demonstrate that our method achieves the best performance across four perceptual metrics. As shown in Fig. 3, while the regression models generate higher distortion results, their perceptual quality is quite poor. All the previous methods tend to produce very blurry results. In contrast, our method produces clear images with fine details, vivid colors, and well-preserved structures, demonstrating its superior perceptual quality.

Ours

GT

FBCNN

SwinIR

JPEG image

4.4 Comparisons on Image Deblurring

We also evaluate our model on GoPro dataset [35] and compare it with the popular deblurring methods. Since the diffusion-based method DvSR [50] has not released the code, we implemented it based on an open-sourced project. The origin training setting of DvSR requires 32s TPUs, which is hard to reproduce. We train DvSR under the same lightweight training setting as our method. The reproduced perceptual quality has similar conclusions as the original paper. The quantitative results are shown in Tab. 2. Our method achieves the best perceptual quality over all perceptual metrics. Fig. 4 shows the visualization of the deblurring results.

4.5 Comparisons on JPEG restoration

For the JPEG restoration task, we perform a comparison between our method and several baseline models, including regression models (FBCNN [23] and SwinIR [31]) and the diffusion-based method DDR [26].

We follow previous works [26] to conduct the evaluation on restoring 256 × 256 images with a Quality Factor of 10. To ensure a fair comparison, we utilize the official implementations of these baseline methods. The quantitative and qualitative results are consistent with previous tasks. As shown in Tab. 3, our method shows significant improvements in terms of perceptual quality. The visualization results in Fig. 5 further illustrate this conclusion. Unlike previous work [26], which can only work on inputs of a

fixed size. Our method supports arbitrary resolutions by adopting the inter-step patch-splitting strategy. Some high-resolution results are provided in Fig. 5 .

4.6 The Effectiveness of Inter-step Patch-splitting Strategy

In Fig. 6, we provide a visual comparison between the common patch-splitting strategy and our proposed inter-step patch-splitting strategy. As we can see, adopting the common practice always produces severe artifacts around the patch boundaries, and the neighboring patches cannot produce consistent content. Further increasing the amount of overlapped patch padding does not alleviate this issue. In contrast, our inter-step patch-splitting strategy produces more consistent and visually coherent output.

Padding = 256

Padding = 16

Padding = 64

Ours

Noisy

GT

4.7 Ablation Studies

In this section, we conduct ablation studies to analyze the impact of different design choices in our conditional diffusion framework. We use the LPIPS metric to compare the performance of different configurations, and we also report the corresponding computational costs.

The design of the basic module. We first validate the design of the basic module. As shown in the second row block of Tab. 4, removing LayerNorm (denoted as 'w/ → w/o LayerNorm') or replacing the Swish activation functions with ReLU ('Swish → ReLU') leads to decreased performances. We also explored using GroupNorm ('LayerNorm → GroupNorm'), as in S3 [42]. However, we observed that GroupNorm tends to introduce color distortion artifacts in the restored images. This might be caused by the grouping of channels, which might not capture the consistent relationships across channels.

The importance of spatial guidance. We further investigated the impact of removing the spatial guidance branch from our framework ('w/o Spatial guidance') and keeping all other settings the same. The results, shown in the third row block of Tab. 4, indicate that removing the spatial guidance branch decreases the LPIPS score to 0.230, but it does not significantly reduce the MACs. We also explored alternative choices for the spatial guidance, including using the internal feature map ('Internal feature guidance') or the degraded image ('Degraded image guidance') itself as the spatial guidance. However, these alternatives do not yield satisfactory results. Leveraging the output of the initial predictor as the spatial guidance is more effective. The initial predictor captures the low-frequency structural information of the final output, which is more stable and deterministic compared to using the internal feature map or the degraded image. By injecting the initial guidance into all the diffusion model blocks, we ensure consistent spatial guidance throughout the restoration process.

Other integration choices. We also compare the previous condition integration designs with our method. As shown in the fourth row block of Tab. 4, directly adding ('Addition') or concatenating ('Concatenation') the guidance feature map, or using AdaIN ('AdaIN') [21] to generate conditional normalization coefficients as the input to the diffusion block cannot effectively capture and utilize the conditional information for guiding the generation process. The alternative approach fails to fully leverage the potential of the diffusion model and results in suboptimal restoration quality. In contrast, our method, which incorporates the guidance feature map through the Adaptive Kernel Guidance Module, provides a more effective and adaptive integration of the conditional information. By dynamically generating adaptive kernels based on the fused multi-source conditional information, our method achieves spatially adaptive conditioning within each block of the diffusion model.

5 Conclusion

In this paper, we introduce a unified conditional framework for image restoration tasks. Our framework incorporates an initial predictor to generate the initial guidance and facilitates the integration of multiple sources of conditional information into each block of the diffusion model. We propose a basic module and an Adaptive Kernel Guidance Block to effectively combine the spatial guidance and auxiliary scalar information, enhancing the generative capacity of the diffusion model. To handle high-resolution images, we propose an inter-step patch-splitting strategy that ensures consistent and artifact-free high-resolution results. Extensive experiments on extreme low-light denoising, image deblurring, and JPEG restoration demonstrate the effectiveness and generalization capability of our proposed framework. The experimental results highlight the superior perceptual quality achieved by our method when compared to strong regression baselines and recent diffusion-based models.

6 Limitation and Future Direction

While our method achieves superior quantitative and qualitative results on three challenging tasks, there are still several problems that can be further explored. In this work, we simply use a uniform noise schedule with fewer time steps to speed up the sampling process of the diffusion model. This can be further accelerated by adopting recent fast sampling methods. Besides, since the generation does not explicitly consider semantic guidance, it may generate some unnatural textures ( e . g ., incorrect characters), which also has been mentioned in previous works [42]. How to control the generation ability of the diffusion model is also a very interesting direction.

7 Acknowlegement

This project is funded in part by National Key R&D Program of China Project 2022ZD0161100, by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (ITC)'s InnoHK, by General Research Fund of Hong Kong RGC Project 14204021. Hongsheng Li is a PI of CPII under the InnoHK.

References

Abdelrahman Abdelhamed, Radu Timofte, and Michael S Brown. NTIRE 2019 challenge on real image denoising: Methods and results. In CVPR Workshops , 2019.

Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony DeRose, and Fabrice Rousselle. Kernel-predicting convolutional networks for denoising monte carlo renderings. ACM Trans. Graph. , 36(4):97:1-97:14, 2017.

Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations , 2019.

Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09800 , 2022.

Antoni Buades, Bartomeu Coll, and J-M Morel. A non-local algorithm for image denoising. In CVPR , 2005.

Jianrui Cai, Shuhang Gu, Radu Timofte, and Lei Zhang. NTIRE 2019 challenge on real image superresolution: Methods and results. In CVPR Workshops , 2019.

Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun. Learning to see in the dark. In CVPR , 2018.

Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun. Simple baselines for image restoration. arXiv preprint arXiv:2204.04676 , 2022.

Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu, and Chengpeng Chen. Hinet: Half instance normalization network for image restoration. In CVPR Workshops , 2021.

Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In ICLR , 2021.

Sung-Jin Cho, Seo-Won Ji, Jun-Pyo Hong, Seung-Won Jung, and Sung-Jea Ko. Rethinking coarse-to-fine approach in single image deblurring. In ICCV , 2021.

Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. ILVR: conditioning method for denoising diffusion probabilistic models. In ICCV , pages 14347-14356. IEEE, 2021.

Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-D transform-domain collaborative filtering. TIP , 2007.

Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. In NeurIPS , pages 8780-8794, 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021.

Max Ehrlich, Larry Davis, Ser-Nam Lim, and Abhinav Shrivastava. Quantization guided JPEG artifact correction. In ECCV , 2020.

Michael Elad and Michal Aharon. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans. Image Process. , 15(12):3736-3745, 2006.

Alessandro Foi, Mejdi Trimeche, Vladimir Katkovnik, and Karen O. Egiazarian. Practical poissoniangaussian noise modeling and fitting for single-image raw-data. IEEE Trans. Image Process. , 2008.

Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-toprompt image editing with cross attention control. 2022.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. CoRR , abs/2207.12598, 2022.

Xun Huang and Serge J. Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV , 2017.

Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc Van Gool. Dynamic filter networks. In NIPS , pages 667-675, 2016.

Jiaxi Jiang, Kai Zhang, and Radu Timofte. Towards flexible blind JPEG artifacts removal. In ICCV , 2021.

Yifan Jiang, Bartlomiej Wronski, Ben Mildenhall, Jonathan T. Barron, Zhangyang Wang, and Tianfan Xue. Fast and high quality image denoising via malleable convolution. In ECCV , 2022.

Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In NeurIPS , 2022.

Bahjat Kawar, Jiaming Song, Stefano Ermon, and Michael Elad. Jpeg artifact correction using denoising diffusion restoration models. In Neural Information Processing Systems (NeurIPS) Workshop on ScoreBased Methods , 2022.

Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. DeblurGAN-v2: Deblurring (ordersof-magnitude) faster and better. In ICCV , 2019.

Dasong Li, Xiaoyu Shi, Yi Zhang, Xiaogang Wang, Hongwei Qin, and Hongsheng Li. No attention is needed: Grouped spatial-temporal shift for simple and efficient video restorers. CoRR , abs/2206.10810, 2022.

Dasong Li, Yi Zhang, Ka Chun Cheung, Xiaogang Wang, Hongwei Qin, and Hongsheng Li. Learning degradation representations for image deblurring. In ECCV , 2022.

Dasong Li, Yi Zhang, Ka Lung Law, Xiaogang Wang, Hongwei Qin, and Hongsheng Li. Efficient burst raw denoising with variance stabilization and multi-frequency denoising network. IJCV. , 2022.

Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. SwinIR: Image restoration using swin transformer. In ICCV Workshops , 2021.

Julien Mairal, Michael Elad, and Guillermo Sapiro. Sparse representation for color image restoration. TIP , 2007.

Fabian Mentzer, George Toderici, Michael Tschannen, and Eirikur Agustsson. High-fidelity generative image compression. In NeurIPS , 2020.

Ben Mildenhall, Jonathan T Barron, Jiawen Chen, Dillon Sharlet, Ren Ng, and Robert Carroll. Burst denoising with kernel prediction networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018.

Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In CVPR , 2017.

Seungjun Nah, Sanghyun Son, Suyoung Lee, Radu Timofte, and Kyoung Mu Lee. Ntire 2021 challenge on image deblurring. In CVPR Workshops , 2021.

Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo. Exploiting deep generative prior for versatile image restoration and manipulation. In European Conference on Computer Vision (ECCV) , 2020.

Mengwei Ren, Mauricio Delbracio, Hossein Talebi, Guido Gerig, and Peyman Milanfar. Image deblurring with domain generalizable diffusion models. CoRR , abs/2212.01789, 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR , pages 10674-10685. IEEE, 2022.

Leonid I. Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena , 60(1):259-268, 1992.

Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In SIGGRAPH (Conference Paper Track) , 2022.

Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Trans. Pattern Anal. Mach. Intell. , 45(4):4713-4726, 2023.

Maitreya Suin, Kuldeep Purohit, and A. N. Rajagopalan. Spatially-attentive patch-hierarchical network for adaptive motion deblurring. In CVPR , 2020.

Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan. Towards real-world blind face restoration with generative facial prior. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind superresolution with pure synthetic data. In ICCVW , pages 1905-1914. IEEE, 2021.

Xintao Wang, Liangbin Xie, Ke Yu, Kelvin C.K. Chan, Chen Change Loy, and Chao Dong. BasicSR: Open source image and video restoration toolbox. https://github.com/XPixelGroup/BasicSR , 2022.

Yinhuai Wang, Jiwen Yu, Runyi Yu, and Jian Zhang. Unlimited-size diffusion restoration. In CVPR Workshops , pages 1160-1167. IEEE, 2023.

Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In ICLR . OpenReview.net, 2023.

Zhendong Wang, Xiaodong Cun, Jianmin Bao, and Jianzhuang Liu. Uformer: A general u-shaped transformer for image restoration. arXiv:2106.03106 , 2021.

Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G. Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In CVPR , 2022.

Zhihao Xia, Federico Perazzi, Michaël Gharbi, Kalyan Sunkavalli, and Ayan Chakrabarti. Basis prediction networks for effective burst denoising with large kernels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11844-11853, 2020.

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In CVPR , 2022.

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Learning enriched features for real image restoration and enhancement. In ECCV , 2020.

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In CVPR , 2021.

Yi Zhang, Dasong Li, Xiaoyu Shi, Dailan He, Kangning Song, Xiaogang Wang, Hongwei Qin, and Hongsheng Li. Kbnet: Kernel basis network for image restoration. CoRR , abs/2303.02881, 2023.

Yi Zhang, Hongwei Qin, Xiaogang Wang, and Hongsheng Li. Rethinking noise synthesis and modeling in raw denoising. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new framework for diffusion-based image restoration. The underlying architecture is based on that of \"DvSR\" [49], in which there is a deterministic predictor that does an initial restoration, used in conjunction with a probabilistic diffusion denoising network that applies the diffusion reverse process to estimate the additive residual to the initial prediction, that can be combined to get the final restoration.\n\nThe novelty is the change in the residual denoiser network (itself based on SR3 [42]). Instead of using spatially invariant convolutions that are fixed after learning, it employs the BPN [50] idea of having basis convolutional kernels that can be linearly combined at different spatial positions, with combination weights that are themselves estimated from a fusion of the initial deterministic restoration, and other auxiliary guidance (diffusion timestep and degradation type were mentioned).\n\nIn addition, a new inter-step patch-splitting procedure that is integrated with the diffusion denoising process is also used for patch-wise high resolution restoration, to overcome problems with artifacts at patch boundaries.\n\nExperiments were conducted on the SID dataset [7] for low-light denoising, on the GoPro dataset [37] for motion deblurring, and on ImageNet JPEG compression artifact restoration.\n\nSTRENGTHS:\nGenerally there is decent novelty. Although the main ideas are taken from existing methods, the combination of these ideas are relatively new and not immediately obvious. The use of basis kernels from [50] for creating spatially-variant convolutional kernels for the diffusion denoiser is quite interesting.\n\nThe inter-step patch splitting, although simple, appears to be new and effective, with Fig. 6 quite convincing.\n\nExperimental results substantially exceed current SOTA on perceptual metrics (LPIPS, NIQE, FID, KID), though more mediocre on distortion metrics (PSNR, SSIM). It is noted though that the DvSR network was retrained by the authors to match the same training hyperparameters that are more lightweight, due to inaccessible computational requirements of DvSR; otherwise the published results in [49] are better.\n\nWEAKNESSES:\nOn experiments:\n- Since the method is essentially derived from DvSR, and the authors are retraining it based on their hyperparameters, it is important to conduct experiments with DvSR on the other low-light denoising and JPEG restoration tasks as well.\n- Generalization tests from GoPro to the HIDE and RealBlur datasets have been done in various other baseline papers. The authors should have also tested on these.\n- The NIQE numbers don't match the ones in [49], though the other perceptual metrics do. If they are different, the authors should properly justify in the paper why this is the case.\n\nMultiple parts of the paper are unclear, as details have been left out.\n- How is the auxiliary scalar information represented? For example, how are timesteps encoded? Details should at least be in the supplementary material.\n- What exactly is happening in the inter-step patch-splitting procedure? Figure 2 is very vague. Although the results look good, and a reader can perhaps roughly guess what is happening, there are no details given.\n- The descriptions in Section 4.7 Ablation Studies are very vague and need to be explained better.\n    - What happens in \"w/o spatial guidance\"? How does time and other scalar information get injected (or not)?\n    - What are \"internal feature guidance\", and \"degraded image guidance\"? These need clearer explanation.\n\nThe paper is littered with spelling errors. This can simply be resolved by running a spell-checker; that it is not done suggests poor neglect on the part of the authors. Some instances:\n- L120: \"Restoraion\" => \"Restoration\"\n- L210: \"receipt\" => \"receptive\"\n- L257: \"JEPG\" => \"JPEG\"\n\nQUESTIONS:\nPlease answer the questions raised in the Weaknesses section.\n\nThe baseline methods are for the different restoration tasks are barely overlapping. How is the decision for the choice of baselines for each dataset? For example, Uformer is tested on both SID and GoPro, but not Restormer. Why aren't they be tested across the different tasks / datasets?\n\nWhile it is generally accepted that spatially variant kernels can be useful in many situations, I think it is worth a discussion when it comes to denoising residuals. Intuitively we may expect the residuals have a more uniform spatial distribution, so kernels don't have to be spatially varying. This is unlike convolutions on normal images with strong spatially dependent image structures. But it seems the results suggest otherwise. Can the authors provide a better understanding on this?\n\nLIMITATIONS:\nThe technical limitations given in Section 6 are quite reasonable.\n\nHowever, the authors should also include some discussion about potential negative societal impact, since image restoration can also lead to hallucination of details that were not present in the original images.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper proposes a unified framework for diffusion-based framework. In the proposed framework, an initial predictor is used to produce a rough restoration of the input degraded image. The roughly restored image is then used as condition to the diffusion model using a Conditional Integration Module. Experiments show that the proposed approach achieves promising results on low-light denoising, debarring, and JPEG restoration.

STRENGTHS:
1. The proposed method achieves promising performance on three different restoration tasks. The proposed framework is versatile in the sense that it can handle multiple types of degradations. 

2. This paper is easy to follow.

WEAKNESSES:
1. The technical novelty is limited. In particular, the residual prediction formulation was proposed in [49]. The inter-step patch-splitting strategy has been discussed in [*1]. 

2. Many of the proposed components do not lead to significant performance gain. Specifically, in Table 4, the worst model obtains a LPIPS of 0.253, which is still significant better than existing works in Table 1. The performance gain from the proposed components are insignificant. 

3. Given the insignificant performance gain mentioned above, the performance gain could come from the increase of model complexity. Therefore, it is advised to provide additional analysis on the model to demonstrate the superiority of the proposed method. 


[*1] Álvaro Barbero Jiménez, Mixture of Diffusers for scene composition and high resolution image generation, 2023

QUESTIONS:
While the proposed method achieves promising performance on multiple restoration tasks, the novelty is limited and the source of performance gain is unclear. Please refer to the weakness section for the questions.

LIMITATIONS:
Limitations are appropriately discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a unified conditional framework for image restoration tasks based on diffusion models. The framework utilizes a UNet to predict initial guidance and incorporates multi-source conditional information into each block to enhance the generative model's guidance. Adaptive Kernel Guidance Block (AKGB) dynamically fuses kernels in each block. Additionally, a patch-splitting strategy is introduced to handle high-resolution images, ensuring the consistent generation of high-resolution images without grid artifacts. Extensive experiments are conducted on extreme low-light denoising, image deblurring, and JPEG restoration.\n\nSTRENGTHS:\n• The setting for generative image restoration is very interesting and should be a promising direction, as it addresses the limitation of existing methods that primarily focus on improving PSNR without consistent visual quality according to human perception. In recent low-level vision research, although PSNR improvements have been observed, the enhancements in visual quality have been minimal. This paper provides a good starting point for generative image restoration.\n    • The challenge of maintaining consistency in high-resolution images is important for generative models. The proposed ‘inter-step patch-splitting’ idea is cute and effective to solve this problem for diffusion models.\n    • The method demonstrates good generalization across three different image restoration tasks on perceptual quality.\n    • The visualization results are impressive. With the perception-oriented target, the method seems to improve the perception quality a lot.\n\nWEAKNESSES:\n• The guidance map plays an important role in this method, which output the low-frequency information and serves as the spatial guidance for each block. Visualizing the guidance map can help readers to understand this method better.\n    • While the  'Inter-step Patch-splitting Strategy'  is cute, it would be valuable to include quantitative results to demonstrate its effectiveness.\n    • There are some mirror typos. Caption of Fig.1; L286 'LayerNomr -> GroupNorm' ; L158 high light;\n\nQUESTIONS:\n• See the weaknesses.\n    • It is highly recommended to release the code to encourage further research.\n\nLIMITATIONS:\nIn the Limitation and Future Direction section, the authors point out that the method may generate some unnatural textures.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel framework for supervised image restoration. The framework consists of an Initial predictor and a newly designed conditional diffusion model. The initial predictor first produces an initial restoration result, then the conditional diffusion takes the initial result as well as the degradation image as inputs and progressively generates the final result. Besides, the authors also propose a novel solution for restoring high-resolution images. Experiments on low-light denoising, image deblurring, and JPEG restoration show that the proposed framework achieves superior performance in terms of perceptual metrics.\n\nSTRENGTHS:\n1. The solution for high-resolution image restoration seems novel and inspiring.\n2. The proposed method achieves superior performance in three restoration tasks in terms of perceptual metrics.\n3. The proposed denoiser backbone seems reasonable.\n4. The paper is well-written and easy to follow.\n\nWEAKNESSES:\n1. Though seems reasonable, there is no evidence that proves the advantage of the designed backbone. At least, the authors should compare it to the denoiser backbone used in SR3 [1]. This is necessary when designing a new backbone instead of using existing ones.\n2. It seems that the performance on distortion metrics (i.e., PSNR, SSIM) lags far behind state-of-the-art methods.\n3. Some minor issues. (1) The conference information in your citations needs to be updated. For example, SR3 [1], DDNM [2], and LDM [3] should be TPAMI, ICLR, and CVPR, rather than arXiv, CoRR, and None. (2) There exist other methods [3, 4] for high-resolution image restoration, it is appropriate to include them in the discussion. \n\n[1] Saharia et. al., Image super-resolution via iterative refinement. TPAMI 2022.\n\n[2] Wang et. al., Zero-shot image restoration using denoising diffusion null-space model. ICLR 2023.\n\n[3] Rombach et. al., High-resolution image synthesis with latent diffusion models. CVPR 2022.\n\n[4] Wang et. al., Unlimited-size diffusion restoration. CVPRW 2023.\n\nQUESTIONS:\n1. Please see the weaknesses.\n2. I wonder if the proposed framework can be used for general image-to-image translation tasks, or benefit the text-to-image task.\n\nLIMITATIONS:\nGiven the concerns mentioned in the weaknesses, I give a borderline score. However, I may also change my score if there is enough evidence of the profound value of this paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper proposes a new framework for image restoration with diffusion models. They design strategies to better use the condition information and also propose a strategy for high-solution images. The results are competitive over baselines. The presentation is clear. I tend to accept this paper.

STRENGTHS:
- The performance is strong. It outperforms existing baselines on several tasks and datasets quantitatively. The perceptual performance is also good. The data fidelity and details are good.
- This paper proposes detailed experiments and analysis for each design of the framework, which helps the readers to understand their method. However, I would say that it would be better to provide some results on high-resolution images.
- The paper writing is good and clear.
- Most results are obtained on real-world datasets.

WEAKNESSES:
- It would be better to provide some results on high-resolution images.
- It would be better to provide results on more image restoration tasks such as colorization, inpainting.

QUESTIONS:
See weakness.

LIMITATIONS:
See weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07