SlowGuess/NeurIPS_2023_OCR huggingface dataset:
FBCNN
Figure 5: Qualitative comparison on the JPEG restoration.
SwinIR
4.4 Comparisons on Image Deblurring
We also evaluate our model on GoPro dataset [35] and compare it with the popular deblurring methods. Since the diffusion-based method DvSR [50] has not released the code, we implemented it based on an open-sourced project. The origin training setting of DvSR requires 32s TPUs, which is hard to reproduce. We train DvSR under the same lightweight training setting as our method. The reproduced perceptual quality has similar conclusions as the original paper. The quantitative results are shown in Tab. 2. Our method achieves the best perceptual quality over all perceptual metrics. Fig. 4 shows the visualization of the deblurring results.
4.5 Comparisons on JPEG restoration
For the JPEG restoration task, we perform a comparison between our method and several baseline models, including regression models (FBCNN [23] and SwinIR [31]) and the diffusion-based method DDR [26].
We follow previous works [26] to conduct the evaluation on restoring  $256 \times 256$  images with a Quality Factor of 10. To ensure a fair comparison, we utilize the official implementations of these baseline methods. The quantitative and qualitative results are consistent with previous tasks. As shown in Tab. 3, our method shows significant improvements in terms of perceptual quality. The visualization results in Fig. 5 further illustrate this conclusion. Unlike previous work [26], which can only work on inputs of a
Table 3: JPEG restoration results on ImageNet.
Model	Perceptual	Distortion
LPIPS↓	NIQE↓	FID↓	KID↓	PSNR↑	SSIM↑
SwinIR [31]	0.205	8.95	65.81	22.9	28.00	0.883
QGAC [16]	0.260	-	-	-	28.01	0.800
FBCNN [23]	0.211	8.70	68.29	24.13	27.81	0.878
DDRM [26]	0.260	8.34	74.45	24.87	27.68	0.780
Regression	0.217	9.68	74.38	27.47	27.91	0.807
Ours	0.139	6.92	38.74	6.30	27.75	0.808
fixed size. Our method supports arbitrary resolutions by adopting the inter-step patch-splitting strategy. Some high-resolution results are provided in Fig. 5.
4.6 The Effectiveness of Inter-step Patch-splitting Strategy
In Fig. 6, we provide a visual comparison between the common patch-splitting strategy and our proposed inter-step patch-splitting strategy. As we can see, adopting the common practice always produces severe artifacts around the patch boundaries, and the neighboring patches cannot produce consistent content. Further increasing the amount of overlapped patch padding does not alleviate this issue. In contrast, our inter-step patch-splitting strategy produces more consistent and visually coherent output.
Table 4: The ablation studies on extreme lowlight denoising dataset.
Model	LPIPS↓	MACs↓
w/ → w/o LayerNorm	0.227	71.7
LayerNorm → GroupNorm	0.240	71.7
Swish → ReLU	0.235	71.7
w/o Spatial guidance	0.240	70.9
Internal feature guidance	0.247	72.1
Degraded image guidance	0.253	71.7
Addition	0.236	71.5
Concatenation	0.241	112.7
AdaIN [21]	0.240	71.5
DvSR [50]	0.244	73.7
Ours	0.222	71.7
Padding  $= 16$
Padding  $= 64$
Padding  $= 256$
GT
Figure 6: The visual comparison of different patch-based splitting strategy.
4.7 Ablation Studies
In this section, we conduct ablation studies to analyze the impact of different design choices in our conditional diffusion framework. We use the LPIPS metric to compare the performance of different configurations, and we also report the corresponding computational costs.
The design of the basic module. We first validate the design of the basic module. As shown in the second row block of Tab. 4, removing LayerNorm (denoted as “w/→w/o LayerNorm”) or replacing the Swish activation functions with ReLU (“Swish→ReLU”) leads to decreased performances. We also explored using GroupNorm (“LayerNorm→GroupNorm”), as in S3 [42]. However, we observed that GroupNorm tends to introduce color distortion artifacts in the restored images. This might be caused by the grouping of channels, which might not capture the consistent relationships across channels.
The importance of spatial guidance. We further investigated the impact of removing the spatial guidance branch from our framework ("w/o Spatial guidance") and keeping all other settings the same. The results, shown in the third row block of Tab. 4, indicate that removing the spatial guidance branch decreases the LPIPS score to 0.230, but it does not significantly reduce the MACs. We also explored alternative choices for the spatial guidance, including using the internal feature map ("Internal feature guidance") or the degraded image ("Degraded image guidance") itself as the spatial guidance. However, these alternatives do not yield satisfactory results. Leveraging the output of the initial predictor as the spatial guidance is more effective. The initial predictor captures the low-frequency structural information of the final output, which is more stable and deterministic compared to using the internal feature map or the degraded image. By injecting the initial guidance into all the diffusion model blocks, we ensure consistent spatial guidance throughout the restoration process.
Other integration choices. We also compare the previous condition integration designs with our method. As shown in the fourth row block of Tab. 4, directly adding ("Addition") or concatenating ("Concatenation") the guidance feature map, or using AdaIN ("AdaIN") [21] to generate conditional normalization coefficients as the input to the diffusion block cannot effectively capture and utilize the conditional information for guiding the generation process. The alternative approach fails to fully leverage the potential of the diffusion model and results in suboptimal restoration quality. In contrast, our method, which incorporates the guidance feature map through the Adaptive Kernel Guidance Module, provides a more effective and adaptive integration of the conditional information. By dynamically generating adaptive kernels based on the fused multi-source conditional information, our method achieves spatially adaptive conditioning within each block of the diffusion model.
In this paper, we introduce a unified conditional framework for image restoration tasks. Our framework incorporates an initial predictor to generate the initial guidance and facilitates the integration of multiple sources of conditional information into each block of the diffusion model. We propose a basic module and an Adaptive Kernel Guidance Block to effectively combine the spatial guidance and auxiliary scalar information, enhancing the generative capacity of the diffusion model. To handle high-resolution images, we propose an inter-step patch-splitting strategy that ensures consistent and artifact-free high-resolution results. Extensive experiments on extreme low-light denoising, image deblurring, and JPEG restoration demonstrate the effectiveness and generalization capability of our proposed framework. The experimental results highlight the superior perceptual quality achieved by our method when compared to strong regression baselines and recent diffusion-based models.
6 Limitation and Future Direction
While our method achieves superior quantitative and qualitative results on three challenging tasks, there are still several problems that can be further explored. In this work, we simply use a uniform noise schedule with fewer time steps to speed up the sampling process of the diffusion model. This can be further accelerated by adopting recent fast sampling methods. Besides, since the generation does not explicitly consider semantic guidance, it may generate some unnatural textures (e.g., incorrect characters), which also has been mentioned in previous works [42]. How to control the generation ability of the diffusion model is also a very interesting direction.
This project is funded in part by National Key R&D Program of China Project 2022ZD0161100, by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (ITC)'s InnoHK, by General Research Fund of Hong Kong RGC Project 14204021. Hongsheng Li is a PI of CPII under the InnoHK.
[44] Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan. Towards real-world blind face restoration with generative facial prior. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[45] Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind superresolution with pure synthetic data. In ICCVW, pages 1905-1914. IEEE, 2021.
[46] Xintao Wang, Liangbin Xie, Ke Yu, Kelvin C.K. Chan, Chen Change Loy, and Chao Dong. BasicSR: Open source image and video restoration toolbox. https://github.com/XPixelGroup/BasicSR, 2022.
[47] Yinhuai Wang, Jiwen Yu, Runyi Yu, and Jian Zhang. Unlimited-size diffusion restoration. In CVPR Workshops, pages 1160-1167. IEEE, 2023.
[48] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In ICLR. OpenReview.net, 2023.
[49] Zhendong Wang, Xiaodong Cun, Jianmin Bao, and Jianzhuang Liu. Uformer: A general u-shaped transformer for image restoration. arXiv:2106.03106, 2021.
[50] Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Sahara, Alexandros G. Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In CVPR, 2022.
[51] Zhihao Xia, Federico Perazzi, Michael Gharbi, Kalyan Sunkavalli, and Ayan Chakrabarti. Basis prediction networks for effective burst denoising with large kernels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11844-11853, 2020.
[52] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In CVPR, 2022.
[53] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Learning enriched features for real image restoration and enhancement. In ECCV, 2020.
[54] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In CVPR, 2021.
[55] Yi Zhang, Dasong Li, Xiaoyu Shi, Dailan He, Kangning Song, Xiaogang Wang, Hongwei Qin, and Hongsheng Li. Kbnet: Kernel basis network for image restoration. CoRR, abs/2303.02881, 2023.
[56] Yi Zhang, Hongwei Qin, Xiaogang Wang, and Hongsheng Li. Rethinking noise synthesis and modeling in raw denoising. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2021.