A Unified Conditional Framework for Diffusion-based Image Restoration
Authors: Yi Zhang
Xiaoyu Shi
Dasong Li
Xiaogang Wang
Jian Wang
Hongsheng Li
Venue: NeurIPS 2023 poster (2023)
Keywords: image restoration
diffusion model
denoising
deblurring
JPEG restoration
TL;DR: This paper propose a unified conditioning framework for diffusion-based image restoration

Abstract: Diffusion Probabilistic Models (DPMs) have recently shown remarkable performance in image generation tasks, which are capable of generating highly realistic images. When adopting DPMs for image restoration tasks, the crucial aspect lies in how to integrate the conditional information to guide the DPMs to generate accurate and natural output, which has been largely overlooked in existing works. In this paper, we present a unified conditional framework based on diffusion models for image restoration. We leverage a lightweight UNet to predict initial guidance and the diffusion model to learn the residual of the guidance. By carefully designing the basic module and integration module for the diffusion model block, we integrate the guidance and other auxiliary conditional information into every block of the diffusion model to achieve spatially-adaptive generation conditioning. To handle high-resolution images, we propose a simple yet effective inter-step patch-splitting strategy to produce arbitrary-resolution images without grid artifacts. We evaluate our conditional framework on three challenging tasks: extreme low-light denoising, deblurring, and JPEG restoration, demonstrating its significant improvements in perceptual quality and the generalization to restoration tasks. The code will be released at https://zhangyi-3.github.io/project/UCDIR/.

A Unified Conditional Framework for Diffusion-based Image Restoration
Yi Zhang'
Xiaoyu Shi
Dasong Li
Xiaogang Wang'
Jian Wang Hongsheng Li 134
CUHK MMLab 2 Snap Research
Centre for Perceptual and Interactive Intelligence
Shanghai AI Laboratory
zhangyi0link cuhk. edu. jvang40anapchat.com com, halißee. cuhk. edu. hk
Abstract
Diffusion Probabilistic Models (DPMs) have recently shown remarkable perfor- mance in image generation tasks, which are capable of generating highly realistic images. When adopting DPMs for image restoration tasks, the crucial aspect lies in how to integrate the conditional information to guide the DPMs to generate accurate and natural output, which has been largely overlooked in existing works. In this paper, present unified conditional framework based on diffusion models for image restoration. We leverage lightweight UNet to predict initial guidance and the diffusion model to learn the residual of the guidance. By carefully designing the basic module and integration module for the diffusion model block, we integrate the guidance and other auxiliary conditional information into every block the diffusion model to achieve patially-adaptivv generation conditioning. To handle high-resolution images, we propose a simple yet effective inter-step patch-splitting strategy to produce arbitrary-resolution images without grid artifacts. We evaluate our conditional framework on three challenging tasks: extreme low-light denoising, deblurring, and JPEG restoration, demonstrating its significant improvements perceptual quality and the generalization to restoration tasks.
1 Introduction
Image restoration tasks are ill-posed problems whose solutions are not unique. But, most existing methods for image restoration rely on learning regression models that produce deterministic results. Unfortunately, such approaches often learn the "averaged" results of potential ground truth distribu- tion, leading to blurry outputs that are not consistent with human perception. With the development of Diffusion Probabilistic Models (DPMs), there exists remarkable progress in image generation, which able to generate realistic images with fine details. This progress has shed light on diffusion-based image restoration tasks, which utilize conditional DPMs to generate images that align well with the corresponding ground truth. With the input degraded input, conditional DPMs can produce realistic images with significant improvements perceptual quality.
When applying DPMs to image restoration, the crucial challenge lies in how to effectively integrate degraded images and other conditional information (e.g., diffusion time step, degradation type, and strength) into the DPMs. A better conditional framework can greatly enhance the generative capacity DPMs and guide them to generate realistic images that faithfully align with ground truths. However, it has been mostly overlooked in existing diffusion- based methods [50] 42 41 [2]. Typically, these methods simply concatenate the degraded image the input to the diffusion model, which fails to fully exploit the potential of the diffusion model.
In this paper, we present unified conditional framework for diffusion-basee image restoration. It effectively integrates multiple sources of conditional information into blocks of DPMs. This
"Corresponding Author
37th Conference Neural Information Processing Systems (NeurIPS 2023).

---

integration enables spatially adaptive conditioning over blocks of the diffusion models, resulting in perceptual quality improvements of restored images. As shown in Fig. [eett, the framework consists of two models: an initial predictor, implemented as a lightweight UNet, and a diffusion model. The initial predictor generates the initial restoration output, capturing the coarse structure and deterministic components of the final output. The diffusion model is then employed be guided by the initial result to predict the residual of the initial guidance. Since the initial predictor restores the initial results of the final output, we leverage it as the spatial guidance to the block of DPMs. This spatial guidance, along with other scalar conditional information, is integrated into the diffusion model blocks, enabling precise and adaptive conditioning during the restoration process.
To enhance the handling of conditional information within each block, we propose an Adaptive Kernel Guidance Block (AKGB) that effectively fuses the conditional information into DPMs. The key idea to generate adaptive kernels for each spatial location by fusing a set of learnable kernels based on conditional information. Specifically, we encode both spatial information and auxiliary scalar information into the feature maps. These two types of information are then fused multi-source fusion weights. The fusion weights are used to linearly combine a set of learnable kernels, resulting in fused kernel weights for each position. This fusion process takes into account both the spatial and scalar conditional information, enabling each position within the diffusion model to have its customized kernel weights. It efficiently injects the condition information into the diffusion model and enhances its spatial adaptability and flexibility. This approach allows the models to effectively utilize the conditional information, leading to improved performance in handling complex image restoration tasks.
In practical applications, another important problem is to adopt diffusion models to generate high- resolution images [41] However, direct applying the basch method often introduces noticeable artifacts due inconsistent results along the patch boundaries. Inspired by the properties of image denoising models, we propose an inter-step patch-splitting strategy to perform patch-splitting at each diffusion time step. This strategy achieves consistent and artifact-free results in the restoration process for arbitrary-resolution images.
We evaluate this framework on three challenging image restoration tasks: extreme low-light denoising, image deblurring, and JPEG restoration. The experiments demonstrate that our method generalizes well to all three tasks and, achieves significantly higher perceptual quality than strong regression baselines and recent diffusion- based models.
Our contributions can be summarized as follows:
We propose unified conditional framework for diffusion-based based image restoration tasks. It leverages a UNet to predict the initial guidance and enable integrating the multi-sources conditional information to every block to better guide the generative model.
To effectively incorporate conditional information into diffusion models, we design a basic module and an Adaptive Kernel Guidance Block (AKGB). It combines the spatial guidance and auxiliary scalar information to adaptively fuse the dynamic kernels in each diffusion model block.
A simple yet effective inter-step saahh-spi strategy is proposed for handling high- resolution images in low-level vision tasks. This practical strategy enables diffusion models generate consistent high-resolution images without grid artifacts.
Through extensive experiments extreme low-light denoising, image deblurring, and JPEG restoration tasks, we demonstrate that our method not only achieves a significantly higher perceptual quality than strong regression baselines and recent diffusion-based models but also show good generalization to various restoration tasks.
2
Related Work
Our method draws inspiration from both regression- based and generative methods in image restoration, as well as dynamic networks.
2.1 Generative Image Restoration
Image restoration. Image restoration has been a highly active research topic in computer vision for several years. Due to the ill-posed nature of the problem, each degraded image has multiple potential solutions. Traditional methods have employed various priors and degradation models [18, 56

---

to mitigate the challenges in image restoration. Traditional methods often rely on priors such as sparsity 32,, non-local self-similarity 5 and total variation 40 to regularize the restoration
process. While traditional methods have made significant progress, they struggle with complex or challenging cases, With the rise of deep learning, learning-ben based methods have become the
mainstream approach in image restoration [53 8 29 28 I 6 36 55 30) These methods leverage large-scale training datasets and deep neural networks to learn the mapping between degraded inputs and clean outputs. Learning-basee methods have shown remarkable performance improvements by implicitly capturing complex image patterns and structures. While learning image restoration from a large dataset shows promising performance improvements, they can only produce deterministic results for this ill ill-posed problem.
Generation-based methods. To mitigate the limitations of predicting deterministic solutions, a promising avenue is the adoption of generation-base based methods. Unlike traditional approaches, these methods generate a distribution for given degraded input, offering a more flexible and probabilistic solution. One popular approach in generation- based image restoration is the use of generative adversarial networks (GANs). Although several GAN- -based methods have been successfully applied to specific domains such as face restoration [44] or super-resolution [45), generating artifacts-fre free and realistic outputs for general image restoration remains a considerable challenge for GAN-based methods.
An emergent and promising direction for imneration-based methods lies in the utilization of diffusion models. These models have showcased impressive performance in diverse domains such as image generation [39 [1]] 22),, image editing [19 4). and super-resolution (42. In image restoration, there are several methods have been proposed. S3 42 proposes conditional Diffusion Probabilistic Models (DPMs) to handle image super-resolution. Then, it has been extended to other image restoration tasks 4T). For image deblurring, DvSR (50) proposes a residual learning framework and icDPM (38] improves the duttr-diistiittionn performance through a multi-scale representation. Most existing works simply concatenate the conditional information into the diffusion models, which largely limits the generation ability of diffusion models.
In addition supervised methods, there exist unsupervised approaches 226 25 48 that decompose the image representation from different perspectives and directly integrate the degraded image into a pretrained diffusion model. Although these methods eliminate the need for retraining, they often suffer from poorer performance compared to the task-specific training method.
2.2 Dynamic Networks in Image Restoration
The significance of adaptive processing in image restoration tasks, taking into account factors such as image texture, degradation types, and degradation strength, is widely acknowledged in the field. In line with this concept, several bearning-based methods have been proposed to generate dynamic kernels (2 for image restoration.
One track of the work is to employ a network predict the spatially invariant convolution kernels 2 This approach aims to capture the varying characteristics of different image regions and adapt the convolution operations accordingly. Building upon this foundation, KPN [34] extends the kernel prediction network into the burst image denoising task. The predicted kernels are utilized to aggregate the input burst images directly, enabling improved denoising performance and better image texture preservation. Instead of predicting the dynamic kernel directly, BPN [II] reduces the complexity of the kernel prediction idea by decomposing the predicted kernel matrix. Malleable convolution (24) improves the efficiency of kernel prediction by producing low-resolution spatial kernels and interpolating them for each position. By incorporating adaptive processing techniques, these methods demonstrate the ability to effectively adapt to different image restoration scenarios, leading to enhanced restoration quality and performance.
Our proposed method shares similar underlying principle with dynamic kernels, as it leverages the integration of multi-source conditional information. These include the degraded input, diffusion time step, and degradation information. By incorporating these factors, our method dynamically generates adaptive kernels that effectively handle feature maps within each block.
3

---

Degraded image
Input
guidance
time
Predictor
LayerNorm
Conv Swish
Interpolation
UNet
Noise
Conv
Linear
LayerNorm
SimpleGate
Swish
Conv
Linear
Initial output
Adaptive Kernel
DyConv
Guidance Block
Diffusion Model
Swish
Conditional Integration Module
Basic Module
Diffusion Model Block
Restored Image
Output
Figure 1: Left: An overview of the conditional framework. Right: The diffusion model block. "..." on the top-right represents the auxiliary scalar information for different tasks (e.g., noise level, blur type.).
3 Method
3.1
Overview of Conditional Framework
Our objective is design a unified conditional framework for image restoration tasks. The input conditional information for this framework consists of two components: the degraded image and auxiliary scalar information. The degraded image represents the image to be restored, while the auxiliary scalar information can include degradation types, strength, or other relevant details specific to each restoration task.
To enhance the integration of conditional information, we first employ lightweight U-Net to predict initial output as shown in Fig. (left). This initial output captures the low-frequency and deterministic aspects of the final restored image, which are easier to restore and contain essential structural information. We utilize this initial output as the spatial guidance for the diffusion model. Together with the auxiliary scalar information (e.g., degradation type, diffusion time step), we inject them into every block of the diffusion model, enabling better control and guidance for the diffusion models. This injection not only provides comprehensive context but also enhances the flexibility of our framework. Following DvSR [50), we employ the diffusion model to capture the residual distribution the initial output. Consequently, the training loss is modified as
B( Berr folvT(y u=(x)) V1- 11-,,,,,1)) </.
(1)
where the noise vector e ~ No I). (x, y) is the sampled degraded-clean image, fa denotes the diffusion model, Us represents the initial predictor, and 7 p(?) is distribution related to the noise schedule. The highlight blue part in Eq. (T) shows the key modification of the "residual modeling".
3.2 Diffusion Model Block
Basic Module In our approach, have designed a basic module for the diffusion model used in image restoration tasks. We aim to keep the module as simple as possible by leveraging existing image restoration backbones. Instead of utilizing complicated operators, we try to make it as simple as possible by adopting existing image restoration backbones. For each block, we leverage two convolution layers. Before each of them, we introduce a LayerNorm to stabilize the training employ the Swish as the activation function. A shortcut applied to perform the residual learning. To enable the injection of the conditional information, the second convolutional kernels are designed to be dynamic based on the conditions.
process [15]. Following the previous generative works B 42 50]
Conditional Injection Module To better integrate the conditional information into the block, we propose a Conditional Integration Module (CIM). Within CIM, the guidance is first scaled to match the resolution of the feature map within the block. This scaled guidance then passes through two

---

guidance
time
Noise
Noise
G Rlxwwx
Patch Splitting
E RIxINN
Patch Splitting
W, E Rexcax
M Rilwwxk
Or
OT
(4)9
=
Diffusion Model
Diffusion Model
Multi-source
Patch Merging
Fusion Weights
Patch Merging
R RXXXN
F(J) RCXXXXX
Sample
Sample
Adaptive Kernel Guidance Module
Common Patch-splitting
Inter-step Patch-splitting
Figure 2: Left: Adaptive Kernel Guidance Module (AKGM). Here, N is the number of kemel bases. We set N 3 for visualization. Right: Inter-step patch-splitting
convolution layers with a SimpleCitte activation [8] effectively adjusting the channel number and producing the feature map G. Simultancously, the auxiliary scalar information is processed through two-linear-layer branch with Swish activation in between, resulting in the feature map S.
The feature maps G and are subsequently passed to the Adaptive Kernel Guidance Module (AKGM) to generate dynamic kernels for the second convolution layer in the basic module, as depicted in Fig. The key idea behind the AKGM is adaptively fuse the kernel bases, allowing each spatial location to handle the feature map based on the fused multiple sources conditional information. As shown in Fig. 2 (left), in each AKGM, we have N learnable kernel bases denoted W,, CxCxkxk where represents the number of channels, and denotes the kernel size. These kernel bases are trained to handle different cases and scenarios. The scalar information (e.g., time) is encoded by two-linear-layer branch with Swish activation in between. The feature maps G E H xxxWxx XN and E IxxxN are fused using pointwise production to generate the multi-source fusion weights M E R Hxxxx Here, H and W denote the height and width of the feature map, respectively, and N represents the number of kernel bases. To obtain the fused kernel for a specific position (i. j). denoted as ((.)), it is obtained by linearly fusing the kernel bases according to the multi-source fusion weights the same position. Formally, it can be expressed
Frs b= Mr, W.-
Fig. 2 illustrates the dynamic kernel generation process for specific position (i,j) with the number of kernel bases set to
Computational analysis. In the standard convolution operation, given an input size of H x W x C, the computational cost HW CK"C)) MACs. However, for the Adaptive Kernel Guidance Mod- (AKGM), calculating the dynamic kernel for each position introduces additional computa- tional complexity. If we use the convolution kernels as the kernel bases, the computational cost is HW(NC2 NC which increases linearly with the number of kernel bases. To reduce the computational burden, we adopt grouped convolution kernels with a group number of bases. As result, the computational cost is reduced to HW (2) The AKGM has comparable computational cost convolution layer. Besides, the additional workload introduced by the module is minimal, as the number of channels is set to a small number.
as the kernel The core operation in
3.3 Hishltreoo Image Inference
In theory, constructing the diffusion model using pure convolution layers allows for arbitrary-size image inference. However, in practice, it is impossible to infer very high-resolution image directly due the limited GPU memory (41). As shown Fig, 2 (right), common approach in image restoration is to split the input image into overlapping patches and independently perform inference on each patch. The output patches then merged to reconstruct the entire image. However, when
5

---

Noisy Input
Restormer
Uformer
Ours
GT
Figure Qualitative comparison on indoor outdoor scenes SID dataset. (Zoom for details)
using generative models, inferring patches independently often results in noticeable grid artifacts along the boundaries of the splitting patches. Even with a large overlap, the neighboring patches are not consistent in the splitting region.
For high-resolution images, directly adopting the diffusion model on the latent space (39 sacrifices the detailed textures when downsampling the image features to the latent space. Another strategy mask-shift [4 is proposed to alleviate the local incoherence but requires significant computational cost. To address this problem. propose simple yet effective inter-step patch-splitinng strategy for high-resolution image inference. As shown in Fig. 2 we integrate the patch-splitting technique into diffusion iterations to support high-resolution image generation. Specifically, at each diffusion time step, the input image is split small overlapped patches. Then, the diffusion model performs denoising on the patches, and the overlapping regions of the results are dropped directly when merging the patches. Since the denoising model focuses on the local receptive field, performing the patch-spliting at the denoising step would not introduce obvious boundary artifacts. Some visualizations can found Fig.
4 Experiments
4.1 Tasks and Datasets
We evaluate method on three challenging tasks. For extreme low-light denoising, SID Sony dataset (7), which provides noisy-clean image pairs with an exposure factor of 300. consists of high-resolution images (2120 x 1416) captured in various indoor and outdoor scenes. For deblurring, we follow DvSR (50) to train and test the GoPro dataset. For JPEG restoration, train on the ImageNet and follow DDRM 26 to test on selected IK evaluation images (37].
the
4.2 Experimental Settings and Metries
To demonstrate method to be unified framework, we train the framework under the similar setting without task specific rate x 10-1, and EMA decay rate is 0.9999. In the training, we used the diffusion process with T 2000 steps with the continuous noise level [10]. During the testing, the inference step is reduced to 50 with uniform interpolation. For each task, we provide regression baseline that adopts the same architecture and training setting for comparison. We train each task for 500k iterations with batch size 32. The training process takes approximately three days complete when utilizing 1100 GPUs.
tuning. We used the AdamW optimizer with learning
In this paper. we focus on the perceptual quality of the results. Four perceptual metrics, LPIPS, NIQE, FID, and KID are adopted for evaluation. LPIPS and NIQE are reference- based and non-reference metrics respectively. For NIQE, different implementations produce different results. In this paper, we adopt from the BasicSR (46) for all methods. For the inception distance FID and KID, follow the previous paper [33 50 to extract patches and compute the FID and KID on patch level to obtain stable evaluation results. We also show the distortion metrics, PSNR and SSIM, for reference.

---

Table 2: Quantitative results GoPro dataset.
Table 1: The quantitative results extreme
low- bog-tight denoising SID dataset.
Model
Perceptual
Distortion
LPIPS NIQEL FID,
KIDI
PSNRT SSIM?
Model
Perceptual
Distortion PSNRT
LPIPS]
NIQEL
FID
KIDI
SSIM
Uformer
49
0.087
5.18
21.60
9.96
33.05
0,962
HINet
0,088
17.91
8.15
32.77
0,960
SID
7
0.361
6.58
83.44
22.47
28.88
0,780
MPRNet
39
0.089
5.16
20.18
9.10
32.66
0,959
Restormer
0.342
6.69
62.05
10.97
29.64
0.796
0.362
78.30
29.14
SAPHNN
43
0.101
19.06
8.48
31.89
0,953
Ufomer
49
0.338
6.45
74.44
17.01
29.27
0.793
DeblurGANv2
27
0.117
DvSR 50
0.084
4.80
12.20
4.33
30.06
0.941
Regression
0.351
6.78
75.79
17.88
29.13
0.778
Ours
0.222
6.43
55.07
6.79
28.78
0.744
Regression
0.097
7.16
31.50
Ours
0.080
2.42
30.27
Blury Input
Restormer
Uformer
DvSR
Ours
Figure 4: Visualization of deblurring results on GoPro dataset.
4.3
Comparisons on Extreme Low-light Denoising
To evaluate our method's performance on extreme low-light denoising, we compare with the official implementations of transformer-basee and convolution-based backbones [52) & 49 on SID dataset 77]. As the SID dataset consists of high-resolution images, utilize inter-step patch- splitting strategy to perform full-resolution image inference without grid artifacts. The quantitative results, presented in Tab. [I] perceptual metries. As shown in Fig. while the regression models generate higher distortion results, their perceptual quality is quite poor. All the previous methods tend to produce very blurry results. In contrast, our method produces clear images with fine details, vivid colors, and well well-preserved structures, demonstrating its superior perceptual quality.
demonstrate that our method achieves the best performance across four
JPEG image
FBCNN
SwinIR
Ours
GT
Figure 5: Qualitative comparison on the JPEG restoration.
7

---

4.4 Comparisons on Image Deblurring
We also evaluate model GoPro dataset 35 and compare it with the popular deblurring
methods. Since the diffusion-based method DvSR [50] has not released the code, we implemented based on an open-sourced project The origin training setting of DvSR requires 32s TPUs, which is hard reproduce. We train DvSR under the same lightweight training setting as our method. The reproduced perceptual quality has similar conclusions as the original paper. The quantitative results are shown in Tab. 2 Our method achieves the best perceptual quality over all perceptual metrics. Fig. 4 shows the visualization of the deblurring results.
4.5
Comparisons on JPEG restoration
For the JPEG restoration task, perform comparison between our method and several baseline models, including regression models (FBCNN 23 and SwinIR (3I and the diffusion-bias based method DDR [26].
We follow previous works [26] to conduct the evaluation restoring 256 x 256 images with Quality Fac- tor of 10. To ensure fair comparison, utilize the official implementations of these baseline methods. The quan- titative and qualitative results are con- sistent with previous tasks. As shown in Tab. 3. our method shows signifi- cant improvements in terms of percep- tual quality. The visualization results in Fig. further illustrate this con- clusion. Unlike previous work [26). which can only work on inputs of fixed size. Our method supports arbitrary resolutions by adopting the inter-step patch-splitting strategy. Some high- ressuuttolttion results are provided in Fig. 5
Table 3: JPEG restoration results ImageNet.
Model
Perceptual
Distortion
APIPS,NIQEL FIDJ
KID, PSNR1 SSIM
SwinIR 31
0.205
8.95
65.81
22.9
28.00
0.883
QGAC [16]
0.260
28.01
0.800
FBCNN 23
0.211
8.70
68.29
24.13
27.81
0.878
DDRM 26
0.260
8.34
74.45
24.87
27.68
0.780
Regression
0.217
9.68
74.38
27.47
27.91
0.807
Ours
0.139
6.92
38.74
6.30
27.75
0.808
4.6 The Effectiveness of Inter-step Patch-splitting Strategy
In Fig. 6. we provide visual comparison between the common patch-spliting strategy and our proposed inter-step patch-splitting strategy. As produces severe artifacts around the patch boundaries, and the neighboring patches cannot produce consistent content. Further increasing the amount of overlapped patch padding does not alleviate this issue. In contrast, our inter-step patch-splitting strategy produces more consistent and visually coherent output.
see, adopting the common practice always
Table 4: The ablation studies on extreme low- light denoising dataset.
Model
LPIPS
MACs|
w/ w/o LayerNorm
0.227
71.7
LayerNorm -> GroupNorm
0.240
71.7
Swish + ReLU
0.235
71.7
Noisy
Padding 16
Padding
w/o Spatial guidance
0.240
70.9
Internal feature guidance
0.247
72.1
Degraded image guidance
0.253
71.7
Addition
0.236
71.5
Concatenation
0.241
112.7
AdaIN 21
0.240
71.5
DvSR 50
0.244
73.7
Pudding 256
Ours
GT
Ours
0.222
71.7
Figure 6: The visual comparison of different patch- based splitting strategy.
8

---

4.7 Ablation Studies
In this section, we conduct ablation studies analyze the impact of different design choices in our conditional diffusion framework. We use the LPIPS metric to compare the performance of different configurations, and we also report the corresponding computational costs.
The design of the basic module. We first validate the design of the basic module. As shown in the second row block of Tab. removing LayerNorm (denoted as "w/-w/o LayerNorm") or replacing the Swish activation functions with ReLU ("Swish->RRelU"" leads to decreased performances. We Sorveroorm-GGruuNooen
also explored using GroupNorm that GroupNorm tends to introduce color distortion artifacts in the restored images. This might be caused by the grouping of channels, which might not capture the consistent relationships across channels.
as in S3 [42]. However, we observed
The importance of spatial guidance. We further investigated the impact of removing the spatial guidance branch from our framework ("w/o Spatial guidance") and keeping all other settings the same. The results, shown in the third row block of Tab. 4 indicate that removing the spatial guidance branch decreases the LPIPS score to 0.230, but it does not significantly reduce the MACs. We also explored alternative choices for the spatial guidance, including using the internal feature map ("Internal feature guidance") or the degraded image ("Degraded image guidance") itself as the spatial guidance. However, these alternatives do not yield satisfactory results. Leveraging the output of the initial predictor as the spatial guidance is more effective. The initial predictor captures the low-frequency structural information of the final output, which is more stable and deterministic compared to using the internal feature map or the degraded image. By injecting the initial guidance into all the diffusion model blocks, we ensure consistent spatial guidance throughout the restoration process.
Other integration choices. We also compare the previous condition integration designs with our method. As shown in the fourth row block of Tab. 4 directly adding ("Addition") or concatenating ("Concatenation") the guidance feature map, using AdaIN ("AdalN"") (21) generate conditional normalization coefficients as the input the diffusion block cannot effectively capture and utilize the conditional information for guiding the generation process. The alternative approach fails to fully leverage the potential of the diffusion model and results in suboptimal restoration quality. In contrast, our method, which incorporates the guidance feature map through the Adaptive Kernel Guidance Module, provides more effective and adaptive integration of the conditional information. By dynamically generating adaptive kernels based the fused multi-source conditional information, our method achieves spatially adaptive conditioning within each block of the diffusion model.
5 Conclusion
In this paper, we introduce unified conditional framework for image restoration tasks. Our frame- work incorporates an initial predictor to generate the initial guidance and facilitates the integration of multiple sources of conditional information into each block of the diffusion model. We propose a basic module and Adaptive Kernel Guidance Block effectively combine the spatial guidance and auxiliary scalar information, enhancing the generative capacity of the diffusion model. To handle high-resolution images, we propose an inter-step patch-splitting strategy that ensures consistent and artifact-free high-resolution results. Extensive experiments on extreme low-light denoising, image deblurring, and JPEG restoration demonstrate the effectiveness and generalization capability of our proposed framework. The experimental results highlight the superior perceptual quality achieved by our method when compared strong regression baselines and recent diffusion-based models.
Limitation and Future Direction
While our method achieves superior quantitative and qualitative results on three challenging tasks, there are still several problems that can be further explored. In this work, we simply use uniform noise schedule with fewer time steps to speed up the sampling process of the diffusion model. This can be further accelerated by adopting recent fast sampling methods. Besides, since the generation does not explicitly consider semantic guidance, may generate some unnatural textures (e.g., incorrect characters), which also has been mentioned in previous works [42]. How to control the generation ability of the diffusion model is also very interesting direction.

---

7 Acknowlegement
This project is funded in part by National Key R&D Program of China Project 2022ZD0161 6000, by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (ITC)'s s InnoHK, by General Research Fund of Hong Kong RGC Project 14204021. Hongsheng Li is PI of CPII under the InnoHK.
References
Abdelrahman Abdelhamed, Radu Timofte, and Michael Brown. NTIRE 2019 challenge on real image denoising: Methods and results. In CVPR Workshops, 2019.
Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony DeRose, and Fabrice Rousselle. Kemel--redicting convolutional networks for denoising monte carlo renderings. ACM Trans. Graph., 36(4)97:11997:1, 2017.
[3] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference Learning Representations, 2019.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint vrXiv:2211.09800 2022.
[5] Antoni Buades, Bartomeu Coll, and J-M Morel. A non-local algorithm for image denoising. In CVPR, 2005.
[6] Jianrui Cai, Shuhang Gu, Radu Timofte, and Lei Zhang. NTIRE 2019 challenge on real image super- resolution: Methods and results. In CVPR Workshops, 2019.
[7] Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun. Learning to see in the dark. In CVPR, 2018.
[8] Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun. Simple baselines for image restoration. arXiv preprint arXiv:2204.04677 2022.
[9] Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu, and Chengpeng Chen, Hinet: Half instance normalization network for image restoration. In CVPR Workshops, 2021.
[10] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In ICLR, 2021.
[11] Sung-Jin Cho, Seo-Wor Won Ji. Jun-Pyo Hong. Seung- Won Jung, and Sung-Jea Ko. Rethinking coarse-to-fine approach in single image deblurring. In ICCV, 2021.
[12] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjunc Gwon, and Sungroh Yoon. ILVR: conditioning method for denoising diffusion probabilistic models. In ICCV, pages 14347-14356 14356. IEEE, 2021.
[13] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse
3-D conesfoom-domminn collaborative filtering. TIP, 2007.
[14] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. In NeurlPS, pages 8780-8794, 2021.
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x16 words:
Transformers for image recognition scale. In ICLR, 2021.
[16] Max Ehrlich, Larry Davis, Scr-Nam Lim, and Abhinav Shrivastava. Quantization guided JPEG artifact correction, ECCV, 2020.
[17] Michael Elad and Michal Aharon. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans. Image Process., 15(12):373-3345, 2006.
[18] Alessandro Foi, Mejdi Trimeche, Vladimir Katkovnik, and Karen O. Egiazarian. Practical poissonian- gaussian noise modeling and fitting for single-image raw-data. IEEE Trans. Image Process., 2008.
[19] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to- prompt image editing with cross attention control. 2022.
[20] Jonathan Ho and Tim Salimans. Rassifier-Frre diffusion guidance. CoRR, abs/2207. 2022.
10

---

[21] Xun Huang and Serge J. Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV, 2017.
[22] Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc Van Gool. Dynamic filter networks, NIPS, pages 667-675, 2016.
[23] Jiaxi Jiang, Kai Zhang, and Radu Timofte. Towards flexible blind JPEG artifacts removal. In ICCV, 2021.
[24] Yifan Jiang, Bartlomiej Wronski, Ben Mildenhall, Jonathan Barron, Zhangyang Wang, and Tianfan Xue. Fast and high quality image denoising via malleable convolution. ECCV, 2022.
[25] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In NeurIPS, 2022.
[26] Bahjat Kawar, Jiaming Song, Stefano Ermon, and Michael Elad. Jpeg artifact correction using denoising diffusion restoration models. In Neural Information Processing Systems (NeurIPS) Workshop on Score- Based Methods, 2022.
[27] Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. DeblurGAN-v2: Deblurring (orders- of-magnitude) faster and better. In ICCV, 2019.
[28] Dasong Li, Xiaoyu Shi, Yi Zhang, Xiaogang Wang, Hongwei Qin, and Hongsheng Li. No atlention is needed: Grouped spatial-temporal shift for simple and efficient video restorers. CoRR, abs/2206. 10800),.0100, 10, 2022.
[29] Dasong Li, Yi Zhang, Ka Chun Cheung, Xiaogang Wang, Hongwei Qin, and Hongsheng Li. Learning degradation representations for image deblurring. ECCV, 2022.
[30] Dasong Li, Yi Zhang, Ka Lung Law, Xiaogang Wang, Hongwei Qin, and Hongsheng Li. Efficient burst raw denoising with variance stabilization and multi mueti-frrquency denoising network. IJCV, 2022.
[31] Jingyun Liang, Jiczhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. SwinIR: Image restoration using swin transformer. ICCV Workshops, 2021.
[32] Julien Mairal, Michael Elad, and Guillermo Sapiro. Sparse representation for color image restoration. 7IP. 2007.
[33] Fabian Mentzer. George Toderici, Michael Tschannen, and Eirikur Agustsson. High-fidelity generative image compression. In NeurIPS, 2020.
[34] Ben Mildenhall, Jonathan T Barron, Jiawen Chen, Dillon Sharlet, Ren Ng, and Robert Carroll. Burst de- noising with kernel prediction networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[35] Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lec. Deep multi-scale convolutional neural network for dynamic scene deblurring. In CVPR, 2017.
[36] Seungjun Nah, Sanghyun Son, Suyoung Lee, Radu Timofte, and Kyoung Mu Lee. Ntie 2021 challenge on image deblurring. In CVPR Workshops, 2021.
[37] Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo. Exploiting deep generative prior for versatile image restoration and manipulation. In European Conference on Computer Vision (ECCV), 2020.
[38] Mengwei Ren, Mauricio Delbracio, Hossein Talehi, Guido Gerig, and Peyman Milanfar. Image deblurring with domain generalizable diffusion models. CoRR, abs/2212.01 01701189,,, 2022.
[39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10674-10685. IRFF, 2022.
[40] Leonid I. Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: Nontinear Phenomena, 60(1):259-266) 1992.
[41] Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Palette: Image-to amage-to-image diffusion models. In SIGGRAPH (Conference Paper Track), 2022.
[42] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Trans. Pottern Anal. Mach. Intell., 45(4):4713-4726, 2023.
[43] Maitreya Suin, Kuldeep Purohit, and A. N. Rajagopalan. Spatially-attentive patch- network for adaptive motion deblurring. In CVPR, 2020.

---

[44] Xintao Wang, Yu Li. Honglun Zhang, and Ying Shan. Towards real-world blind face restoration with generative facial prior. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2021.
[45] Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super- resolution with pure synthetic data. In ICCVW, pages 1905-1914. IEEE, 2021.
[46] Xintao Wang, Liangbin Xie, Yu, Kelvin C.K. Chan, Chen Change Loy, Chao Dong. BasicSR: Open source image and video restoration toolbox. htttp::/
com X wep.xieeeaaee aeriiieerrrrpaaaaee 2022.
[47] Yinhuai Wang, Jiwen Yu. Runyi Yu, and Jian Zhang. Unlimited diffusion restoration. In CVPR Workshops, pages 1160 1167. IEEE, 2023.
[48] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In ICLR. OpenReview.net, 2023.
[49] Zhendong Wang, Xiaodong Cun, Jianmin Bao, Jianzhuang Liu. Uformer: general u-shaped transformer for image restoration. arXiv:2106.03/00 2021.
[50] Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G. Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In CVPR, 2022.
[51] Zhihao Xia, Federico Perazzi, Michaël Gharbi, Kalyan Sunkavalli, and Ayan Chakrabarti, Basis prediction networks for elfective burst denoising with large kemels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 11844-1 1853, 2020.
[52] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In CVPR, 2022.
[53] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Learning enriched features for real image restoration and enhancement. ECCV, 2020.
[54] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Hing--suan Yang, and Ling Shao. Multi-stage progressive image restoration. In CVPR, 2021.
[55] Yi Zhang, Dasong Li, Xiaoyu Shi, Dailan He, Kangning Song, Xiaogang Wang, Hongwei Qin, and Hongsheng Li. Kbnet: Kernel basis network for image restoration. CoRR, abs/2303.0288 2023.
[56] Zhang. Hongwei Qin, Xiaogang Wang, and Hongsheng Li. Rethinking noise synthesis and modeling raw denoising. Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV), October 2021.
12


Reviews:
[{"review_id": "c8elGoQZw0", "reviewer": "Reviewer_4fFj", "summary": "This paper proposes a new framework for image restoration with diffusion models. They design strategies to better use the condition information and also propose a strategy for high-solution images. The results are competitive over baselines. The presentation is clear. I tend to accept this paper.", "questions": "See weakness.", "limitations": "See weakness.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The performance is strong. It outperforms existing baselines on several tasks and datasets quantitatively. The perceptual performance is also good. The data fidelity and details are good.\n- This paper proposes detailed experiments and analysis for each design of the framework, which helps the readers to understand their method. However, I would say that it would be better to provide some results on high-resolution images.\n- The paper writing is good and clear.\n- Most results are obtained on real-world datasets.", "weaknesses": "- It would be better to provide some results on high-resolution images.\n- It would be better to provide results on more image restoration tasks such as colorization, inpainting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Provide more high-resolution image results\n\nIn our paper, the testing images of SID dataset are high-resolution (4256x2848). Due to the limited space, we only show some crops in fig.3 and provide some full-resolution images in the supplementary material. We will revise paper to make this point clear.\n\n> Extension to more image restoration tasks such as colorization and inpainting\n\nIt’s possible to extend the method to other img2img tasks like colorization, deraining, etc. We are actively working on incorporating these tasks and will present them in the final version.\n"}, {"review_id": "KJxVbRsskd", "reviewer": "Reviewer_8ANp", "summary": "This paper proposes a novel framework for supervised image restoration. The framework consists of an Initial predictor and a newly designed conditional diffusion model. The initial predictor first produces an initial restoration result, then the conditional diffusion takes the initial result as well as the degradation image as inputs and progressively generates the final result. Besides, the authors also propose a novel solution for restoring high-resolution images. Experiments on low-light denoising, image deblurring, and JPEG restoration show that the proposed framework achieves superior performance in terms of perceptual metrics.", "questions": "1. Please see the weaknesses.\n2. I wonder if the proposed framework can be used for general image-to-image translation tasks, or benefit the text-to-image task.", "limitations": "Given the concerns mentioned in the weaknesses, I give a borderline score. However, I may also change my score if there is enough evidence of the profound value of this paper.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The solution for high-resolution image restoration seems novel and inspiring.\n2. The proposed method achieves superior performance in three restoration tasks in terms of perceptual metrics.\n3. The proposed denoiser backbone seems reasonable.\n4. The paper is well-written and easy to follow.\n", "weaknesses": "1. Though seems reasonable, there is no evidence that proves the advantage of the designed backbone. At least, the authors should compare it to the denoiser backbone used in SR3 [1]. This is necessary when designing a new backbone instead of using existing ones.\n2. It seems that the performance on distortion metrics (i.e., PSNR, SSIM) lags far behind state-of-the-art methods.\n3. Some minor issues. (1) The conference information in your citations needs to be updated. For example, SR3 [1], DDNM [2], and LDM [3] should be TPAMI, ICLR, and CVPR, rather than arXiv, CoRR, and None. (2) There exist other methods [3, 4] for high-resolution image restoration, it is appropriate to include them in the discussion. \n\n[1] Saharia et. al., Image super-resolution via iterative refinement. TPAMI 2022.\n\n[2] Wang et. al., Zero-shot image restoration using denoising diffusion null-space model. ICLR 2023.\n\n[3] Rombach et. al., High-resolution image synthesis with latent diffusion models. CVPR 2022.\n\n[4] Wang et. al., Unlimited-size diffusion restoration. CVPRW 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> It is suggested to compare the proposed method with the existing backbone.\n\nTo demonstrate the advantage of our designed backbone, we included an additional existing method, DvSR [3], in the ablation studies. As indicated in Table 1 of the newly attached PDF, our approach exhibits comparable computational cost to DvSR while achieving significantly better performance in terms of perceptual metrics. The corresponding visual improvements are presented in Figure 1.\n\n> The concern that the performance of the proposed method on distortion metrics (PSNR, SSIM) lags far behind state-of-the-art methods\n\nNote that PSNR and SSIM are actually not directly related to perceptual quality of the restored images. Most previous methods are deterministic and PSNR-oriented, resulting in predicting averaged results with limited visual improvements and higher PSNR, often appearing blurry [4]. Some visualization can be found in Figure 2 of the newly attached PDF.\n\nIn contrast, our method learns to predict the potential distribution, which may affect PSNR results but significantly enhances visual quality, yielding sharp and visually appealing outcomes.\n\n> Minor issues. \n\nThank for your kind comments. We will update the citations and discuss the existing latent space [1] and Mask-Shift [2] solutions. \nIt’s possible to extend to other image-to-image translation tasks (colorization, deraining, etc). We are actively working on incorporating these tasks and will present them in the final version.\n\n[1] Rombach et. al., High-resolution image synthesis with latent diffusion models. CVPR 2022.\n\n[2] Wang et. al., Unlimited-size diffusion restoration. CVPRW 2023.\n\n[3] Whang et. al., Deblurring via stochastic refinement. CVPR, 2022.\n\n[4] Sajjadi et. al., EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis. ICCV 2017.\n"}, {"review_id": "QGL9zcblD4", "reviewer": "Reviewer_V6gu", "summary": "This paper introduces a unified conditional framework for image restoration tasks based on diffusion models. The framework utilizes a UNet to predict initial guidance and incorporates multi-source conditional information into each block to enhance the generative model's guidance. Adaptive Kernel Guidance Block (AKGB) dynamically fuses kernels in each block. Additionally, a patch-splitting strategy is introduced to handle high-resolution images, ensuring the consistent generation of high-resolution images without grid artifacts. Extensive experiments are conducted on extreme low-light denoising, image deblurring, and JPEG restoration.", "questions": "• See the weaknesses.\n    • It is highly recommended to release the code to encourage further research.", "limitations": "In the Limitation and Future Direction section, the authors point out that the method may generate some unnatural textures.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "• The setting for generative image restoration is very interesting and should be a promising direction, as it addresses the limitation of existing methods that primarily focus on improving PSNR without consistent visual quality according to human perception. In recent low-level vision research, although PSNR improvements have been observed, the enhancements in visual quality have been minimal. This paper provides a good starting point for generative image restoration.\n    • The challenge of maintaining consistency in high-resolution images is important for generative models. The proposed ‘inter-step patch-splitting’ idea is cute and effective to solve this problem for diffusion models.\n    • The method demonstrates good generalization across three different image restoration tasks on perceptual quality.\n    • The visualization results are impressive. With the perception-oriented target, the method seems to improve the perception quality a lot.", "weaknesses": "• The guidance map plays an important role in this method, which output the low-frequency information and serves as the spatial guidance for each block. Visualizing the guidance map can help readers to understand this method better.\n    • While the  'Inter-step Patch-splitting Strategy'  is cute, it would be valuable to include quantitative results to demonstrate its effectiveness.\n    • There are some mirror typos. Caption of Fig.1; L286 'LayerNomr -> GroupNorm' ; L158 high light;", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Visualization of the guidance map\n\nWe appreciate the reviewer's recognition of our efforts to enhance the perceptual quality and qualitative results of our method. In the revised version, we will include visualizations of the guidance map to provide further insights into our approach.  \n\n> It is suggested to add quantitative results for the patch-splitting strategy.\n\nThank you for your valuable advice. We will include quantitative results around the overlapped regions to show the effectiveness of the patch-splitting strategy.\n\n> Minor issues and code: \n\nMinor issues will be fixed, and the code is preparing.\n"}, {"review_id": "hULVn5tD9x", "reviewer": "Reviewer_KQ6K", "summary": "This paper proposes a unified framework for diffusion-based framework. In the proposed framework, an initial predictor is used to produce a rough restoration of the input degraded image. The roughly restored image is then used as condition to the diffusion model using a Conditional Integration Module. Experiments show that the proposed approach achieves promising results on low-light denoising, debarring, and JPEG restoration.", "questions": "While the proposed method achieves promising performance on multiple restoration tasks, the novelty is limited and the source of performance gain is unclear. Please refer to the weakness section for the questions. ", "limitations": "Limitations are appropriately discussed.", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method achieves promising performance on three different restoration tasks. The proposed framework is versatile in the sense that it can handle multiple types of degradations. \n\n2. This paper is easy to follow. ", "weaknesses": "1. The technical novelty is limited. In particular, the residual prediction formulation was proposed in [49]. The inter-step patch-splitting strategy has been discussed in [*1]. \n\n2. Many of the proposed components do not lead to significant performance gain. Specifically, in Table 4, the worst model obtains a LPIPS of 0.253, which is still significant better than existing works in Table 1. The performance gain from the proposed components are insignificant. \n\n3. Given the insignificant performance gain mentioned above, the performance gain could come from the increase of model complexity. Therefore, it is advised to provide additional analysis on the model to demonstrate the superiority of the proposed method. \n\n\n[*1] Álvaro Barbero Jiménez, Mixture of Diffusers for scene composition and high resolution image generation, 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The technical novelty is limited (residual prediction formulation and the inter-step patch-splitting strategy.).\n\nAs described in L67-L81, we actually did not claim the residual prediction as one of our contributions. The residual formulation is a common practice and has been adopted in many previous methods in low-level vision. In this paper, our contributions are a unified conditional framework and an Adaptive Kernel Guidance Block (AKGB) to incorporate the conditional information into diffusion models.\n\nThe inter-step patch-splitting strategy shares the similar spirit with the unpublished paper [1]. The key difference is that [1] needs to fuse the overlapped regions by the Gaussian weights since it adopts several diffusion models on high-level vision tasks, which need to understand the large receptive field sematic information. Our method is designed for low-level vison tasks, which mostly focus on local textures.  Therefore, we can perform the padded patch inference without considering fusing in each diffusion step.\n\n> On the SID dataset, the proposed method outperforms existing regression works significantly. The performance gain from the proposed components is insignificant.\n\nWhen facing severe degradation or highly ill-posed tasks (such as extreme low-light denoising), generative methods tend to outperform regression methods due to the availability of multiple potential candidates for each input. Generative methods can generate more diverse and realistic appearance, , while regression methods tend to predict averaged results, leading to blurry outcomes. Consequently, on the SID dataset, a substantial improvement in terms of LPIPS is observed. Conversely, for the GoPro dataset (Table 2 in our paper), where the problem is less ill-posed, such the performance gain is smaller.\n\nPlease note that, unlike PSNR, LPIPS is not scaled by the log function, resulting in smaller quantitative improvements. However, even small LPIPS improvements correspond to a significant gain. For instance, in Table 2 of our paper, our method improves LPIPS by 0.007 compared to Uformer, yet the corresponding FID improvement exceeds 12.5, highlighting the substantial qualitative gains achieved.\n\n> Additional analysis on the model complexity\n\nWe compare the model complexity with the DvSR [3] in the ablation studies. As indicated in Table 1 of the newly attached PDF, our approach exhibits comparable computational cost to DvSR while achieving significantly better perceptual metrics. The corresponding visual improvements are presented in Figure 1.\n\n[1] Mixture of Diff users for scene composition and high-resolution image generation\n\n[2] EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis\n"}, {"review_id": "ANgpleywDt", "reviewer": "Reviewer_37QZ", "summary": "The paper proposes a new framework for diffusion-based image restoration. The underlying architecture is based on that of \"DvSR\" [49], in which there is a deterministic predictor that does an initial restoration, used in conjunction with a probabilistic diffusion denoising network that applies the diffusion reverse process to estimate the additive residual to the initial prediction, that can be combined to get the final restoration.\n\nThe novelty is the change in the residual denoiser network (itself based on SR3 [42]). Instead of using spatially invariant convolutions that are fixed after learning, it employs the BPN [50] idea of having basis convolutional kernels that can be linearly combined at different spatial positions, with combination weights that are themselves estimated from a fusion of the initial deterministic restoration, and other auxiliary guidance (diffusion timestep and degradation type were mentioned).\n\nIn addition, a new inter-step patch-splitting procedure that is integrated with the diffusion denoising process is also used for patch-wise high resolution restoration, to overcome problems with artifacts at patch boundaries.\n\nExperiments were conducted on the SID dataset [7] for low-light denoising, on the GoPro dataset [37] for motion deblurring, and on ImageNet JPEG compression artifact restoration.", "questions": "Please answer the questions raised in the Weaknesses section.\n\nThe baseline methods are for the different restoration tasks are barely overlapping. How is the decision for the choice of baselines for each dataset? For example, Uformer is tested on both SID and GoPro, but not Restormer. Why aren't they be tested across the different tasks / datasets?\n\nWhile it is generally accepted that spatially variant kernels can be useful in many situations, I think it is worth a discussion when it comes to denoising residuals. Intuitively we may expect the residuals have a more uniform spatial distribution, so kernels don't have to be spatially varying. This is unlike convolutions on normal images with strong spatially dependent image structures. But it seems the results suggest otherwise. Can the authors provide a better understanding on this?\n", "limitations": "The technical limitations given in Section 6 are quite reasonable.\n\nHowever, the authors should also include some discussion about potential negative societal impact, since image restoration can also lead to hallucination of details that were not present in the original images.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Generally there is decent novelty. Although the main ideas are taken from existing methods, the combination of these ideas are relatively new and not immediately obvious. The use of basis kernels from [50] for creating spatially-variant convolutional kernels for the diffusion denoiser is quite interesting.\n\nThe inter-step patch splitting, although simple, appears to be new and effective, with Fig. 6 quite convincing.\n\nExperimental results substantially exceed current SOTA on perceptual metrics (LPIPS, NIQE, FID, KID), though more mediocre on distortion metrics (PSNR, SSIM). It is noted though that the DvSR network was retrained by the authors to match the same training hyperparameters that are more lightweight, due to inaccessible computational requirements of DvSR; otherwise the published results in [49] are better.", "weaknesses": "On experiments:\n- Since the method is essentially derived from DvSR, and the authors are retraining it based on their hyperparameters, it is important to conduct experiments with DvSR on the other low-light denoising and JPEG restoration tasks as well.\n- Generalization tests from GoPro to the HIDE and RealBlur datasets have been done in various other baseline papers. The authors should have also tested on these.\n- The NIQE numbers don't match the ones in [49], though the other perceptual metrics do. If they are different, the authors should properly justify in the paper why this is the case.\n\nMultiple parts of the paper are unclear, as details have been left out.\n- How is the auxiliary scalar information represented? For example, how are timesteps encoded? Details should at least be in the supplementary material.\n- What exactly is happening in the inter-step patch-splitting procedure? Figure 2 is very vague. Although the results look good, and a reader can perhaps roughly guess what is happening, there are no details given.\n- The descriptions in Section 4.7 Ablation Studies are very vague and need to be explained better.\n    - What happens in \"w/o spatial guidance\"? How does time and other scalar information get injected (or not)?\n    - What are \"internal feature guidance\", and \"degraded image guidance\"? These need clearer explanation.\n\nThe paper is littered with spelling errors. This can simply be resolved by running a spell-checker; that it is not done suggests poor neglect on the part of the authors. Some instances:\n- L120: \"Restoraion\" => \"Restoration\"\n- L210: \"receipt\" => \"receptive\"\n- L257: \"JEPG\" => \"JPEG\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> It is suggested to conduct experiments with DvSR on the low-light denoising and JPEG restoration tasks\n\nThanks for the advice. We will try the DvSR [4] on low-light denoising and JPEG restoration tasks. In the Table 1 and Figure 1 of the newly attached PDF, we show the latest LPIPS and the visualization of DvSR respectively on the extreme low-light denoising.\n\n> Generalization tests on the deblurring dataset (HIDE and RealBlur)\n\nWe will conduct the generalization tests. Some preliminary results on HIDE dataset are show in Table 2 of the newly attached PDF. Our method consistently outperforms the Uformer.\n\n> Inconsistency of NIQE numbers with existing papers\n\nWe also noticed this phenomenon in the experiments. Various implementations of NIQE [1-3] report different numerical values but maintain the same rank order. To ensure a fair comparison, we adopted the NIQE implementation from basicSR [2] for all methods in our experiments. We will revise the paper to make it clear.\n\n> Unclear parts and typos of the paper. \n\nWe thank the reviewer for pointing out the unclear parts. We will revise them as follows:\n- We follow the SR3 to employ positional encoding and a stack of two linear layers with a Swish activation function to process all auxiliary scalars.\n- To enhance clarity, we will include a figure and provide a more detailed explanation to present the inter-step patch-splitting procedure.\n- For the 'w/o spatial guidance' variant, we exclude the spatial guidance part of the AKGM module while retaining other scalar information, which is injected by adding it to the feature map.\n- For each AKGM block, the spatial guidance is generated using the output of the initial predictor. The 'internal feature guidance' employs the intermediate feature map within each block to generate spatial guidance. Additionally, the 'degraded image guidance' uses the input degraded image to generate spatial guidance.\n\n> Questions:\n\n- How is the decision for the choice of baselines for each dataset?\n\nWe decided to adopt existing the state-of-the-art methods as the backbones for each task, which might be of different architectures. \n\nIt is a good idea to add Restormer on GoPro dataset, which would be more consistent between different tasks. Since the Uformer shows better performance than Restormer on GoPro dataset, we select Uformer as the representative method at that time. For the SID dataset, they do not show the results on it, so we adopt all the strong backbones for comparisons.\n\n- The understanding of adopting spatially variant kernels on the residual domain.\n\nAlthough the residuals may visually appear more \"uniform\" when visualized, their inherent properties persist. Texture areas remain challenging to learn, making the adoption of spatially adaptive kernels more effective in handling the task. Additionally, considering that the residuals contain high-frequency information while the initial predictor output includes low-frequency components, employing spatially adaptive kernels becomes a natural choice to address the high-frequency details adaptively.\n\n[1] https://github.com/chaofengc/IQA-PyTorch\n\n[2] https://github.com/xinntao/EDVR/blob/master/basicsr/metrics/niqe.py\n\n[3] https://github.com/aizvorski/video-quality/blob/master/niqe.py\n\n[4] Whang et. al., Deblurring via stochastic refinement. CVPR, 2022.\n"}]

Author rebuttal:
We thank reviewers for their valuable and professional comments. Overall, reviewers (4fFj, 8ANp, V6gu, 37QZ) acknowledge the novelty and the performance of our paper. We have uploaded a one-page PDF containing figures and tables to help us addressing the reviewers' concerns, and then, we will respond to each reviewer individually.

Meta-review: Despite one of the reviewers continuing to have strong strong reservations on lack of novelty (in the direction of super-resolution), the preponderance of opinion based on the reviews and rebuttals is that the paper is worthy of publication. I recommend that the paper be accepted for poster presentation at NeurIPS 2023. The paper makes a good contribution to the use of conditional diffusion models for image restoration by proposing a unified conditioning framework. The paper is generally well-written and I find the experiments and rebuttals convincing.
Decision: Accept (poster)