SlowGuess/NeurIPS_2023_OCR huggingface dataset:
After completing the two-stage training, given a blurry input image  $\mathbf{I}_{Blur} \in \mathbb{R}^{H \times W \times 3}$ , the HI-Diff first compresses  $\mathbf{I}_{Blur}$  into a condition latent  $\mathbf{c} \in \mathbb{R}^{N \times C'}$  via latent encoder  $\mathrm{LE}_{\mathrm{DM}}$ . Then the predicted prior feature  $\hat{\mathbf{z}} \in \mathbb{R}^{N \times C'}$  is generated by the diffusion model. Specifically, the diffusion model iteratively executes the reverse process (defined in Eq. (6))  $T$  times starting from a randomly sampled Gaussian Noise ( $\epsilon \sim \mathcal{N}(0, \mathbf{I})$ ). Finally, the HI-Diff reconstructs the deblurred image  $\mathbf{I}_{DB} \in \mathbb{R}^{H \times W \times 3}$  from the input image  $\mathbf{I}_{Blur}$ , using Transformer, which is enhanced by the prior feature  $\hat{\mathbf{z}}$ .
Method	Prior	Multi-Scale	Joint-training	Params (M)	FLOPs (G)	PSNR (dB)	SSIM
Basline	X	X	X	19.13	117.25	31.96	0.9528
Single-Guide	✓	X	✓	21.98	125.39	32.00	0.9534
Split-Training	✓	✓	X	23.99	125.47	30.73	0.9434
HI-Diff (ours)	✓	✓	✓	23.99	125.47	32.24	0.9558
Table 1: Ablation study. We train and test models on the GoPro [28] dataset. Image size is  $3 \times 256 \times 256$  to calculate FLOPs. Prior: the prior feature generated by the diffusion model; Multi-scale: the multi-scale prior feature for hierarchical integration (as opposed to single-scale); Joint-training: the diffusion model and Transformer are trained jointly in stage two.
Data and Evaluation. Following previous image deblurring methods, we evaluate our method on synthetic datasets (GoPro [28] and HIDE [39]) and the real-world dataset (RealBlur [34] and RWBI [56]). The GoPro dataset contains 2,103 pairs of blurry and sharp images for training and 1,111 image pairs for testing. The HIDE dataset provides testing 2,025 images. The RealBlur dataset has two sub-set: RealBlur-J and RealBlur-R. Each sub-set consists of 3,758 training pairs and 980 testing pairs. The RWBI dataset contains 3,112 blurry images captured with different hand-held devices. For synthetic datasets, we train HI-Diff on the GoPro training set and test it on GoPro and HIDE. Moreover, we further test the GoPro-trained model on RealBlur and RWBI to evaluate the generalization of our method. For real-world datasets, we train and test HI-Diff on RealBlur datasets, following previous works [54, 44]. We adopt two common metrics: PSNR and SSIM [49].
Implementation Details. Our HI-Diff consists of two parts: Transformer and the latent diffusion model. For Transformer, without loss of generality, we apply Restormer [53], a 4-level encoder-decoder Transformer architecture. From level-1 to level-4, we set the number of Transformer blocks as [3,5,5,6], the number of channels as [48,96,192,384], and the attention heads as [1,2,4,8]. Besides, there are 4 blocks in the refinement stage. The channel expansion factor is 2.66. For the latent diffusion model, the token number  $N$  is 16, and the channel dimension  $C'$  is 256. The latent encoder contains  $L = 6$  residual blocks. We set the total time-step  $T$  as 8, and the variance hyperparameters  $\beta_{1:T}$  constants increasing linearly from  $\beta_1 = 0.1$  to  $\beta_T = 0.99$ .
Training Settings. We train our HI-Diff with Adam optimizer [19] with  $\beta_{1} = 0.9$  and  $\beta_{2} = 0.99$ . For stage one, the total training iterations are  $300\mathrm{K}$ . The initial learning rate is set as  $2\times 10^{-4}$  and gradually reduced to  $1\times 10^{-6}$  with the cosine annealing [27]. Following previous work [53], we adopt progressive learning. Specifically, we set the initial patch size as 128 and the patch size as 64. We progressively update the patch size and batch size pairs to  $[(160^{2},40),(192^{2},32),(256^{2},16),(320^{2},16),(384^{2},8)]$  at iterations [20K,40K,60K,80K,100K]. For stage two, we adopt the same training settings as in stage one. Moreover, following previous works [53, 54], we apply random rotation and flips for data augmentation. We use PyTorch [31] to implement our models with 4 A100 GPUs.
In this section, we study the effects of different designs of our proposed method. We conduct all experiments on the dataset GoPro [28]. The iterations for stages one and two are 100K, respectively. The image patch size and batch size are set as  $192 \times 192$  and 32. When we calculate the FLOPs, the image size is  $3 \times 256 \times 256$ . The results are reported in Tab. 1 and Figs. 2 and 3.
Effects of Diffusion Prior. We construct a baseline model without prior generated from diffusion in the first row of Tab. 1, denoted as Baseline, which is actually the vanilla Restormer [53] architecture. For fair comparisons, Baseline adopts the same implementation and training settings as HI-Diff (ours, listed in the fourth row). Comparing Baseline and HI-Diff, we can discover that using the prior feature generated by the latent diffusion model (DM) yields a  $0.28\mathrm{dB}$  improvement. It demonstrates the effectiveness of our proposed method. In addition, the HI-Diff, integrated with diffusion, only adds 4.86M Params and 8.22G FLOPs over Baseline. It reveals that performing the diffusion model in highly compact latent space is very efficient. Furthermore, we show the visual comparisons of the Baseline (without the prior feature) and HI-Diff in Fig. 2 (first row). Our HI-Diff, generates sharper textures and complete structures, compared with Baseline.
HI-Diff
HI-Diff
Figure 2: Deblurred samples for different models in Tab. 1. The first row shows effects of diffusion prior, while the second row exhibits effects of hierarchical integration.
Figure 3: Ablation study of the number of iterations  $T$  in diffusion model,  $T$ :  $\{1,2,4,8,16,32\}$ .
Effects of Hierarchical Integration. We conduct an ablation experiment on the hierarchical integration with the multi-scale prior feature (illustrated in Fig. 1(c). We apply the single-scale prior feature on Transformer, which means we set  $\mathbf{z}_1 = \mathbf{z}_2 = \mathbf{z}_3 = \mathbf{z}$  (or  $\hat{\mathbf{z}}$ ). We denote the new model as Single-Guide, and its result is shown in the second row. We find that the PSNR of Single-Guide drops by  $0.24\mathrm{dB}$  compared with HI-Diff, which adopts the multi-scale prior feature. It indicates that the single-scale prior feature cannot adapt well to complex blurry situations. The visual comparison in Fig. 2 (second row) reveals that applying the hierarchical integration restores better-deblurred images.
Effects of Joint Training Strategy. We explore the impact of the joint training strategy in stage two. We train a model only optimized diffusion model in stage two, denoted as Split-Training. Specifically, for Split-Training, we first generate the prior feature  $\mathbf{z}$  from the ground truth and then apply the training objective defined in Eq. (7) to train the diffusion model alone. Then the diffusion model is directly combined with Transformer for evaluation after training is complete. For fair comparisons, Split-Training applies the same pre-trained (stage one) model as HI-Diff in stage two training, and the iterations are 100K. Comparing Split-Training and HI-Diff, HI-Diff is significantly better than Split-Training by 1.51 dB on PSNR value. These results are consistent with the analysis in Sec. 3.2 and demonstrate the importance of the joint training strategy.
Effects of Iterations Number. We further conduct an ablation study to investigate the influence of iteration numbers in the diffusion model. We set six different iterations  $T$ : {1,2,4,8,16,32} for the diffusion model. Meanwhile, the variance hyperparameters  $\beta_{1:T}$  are always linearly interpolated from  $\beta_{1} = 0.1$  to  $\beta_{T} = 0.99$  for different  $T$ . We plot the PSNR of different iterations  $T$  in Fig. 3. We find that only one iteration inference cannot generate the reasonable prior feature, limiting the deblurring performance. Besides, when the number of iterations reaches 8, the curve basically converges. It indicates that only a small number of iterations is needed to generate the suitable prior feature, since the simple distribution of the highly compact latent space (only contains  $N = 16$  tokens).
4.3 Evaluation on Synthetic Datasets
We compare our HI-Diff with 16 state-of-the-art methods: DeblurGAN [22], DeepDeblur [28], DeblurGAN-v2 [23], SRN [43], DBGAN [56], MT-RNN [30], DMPHN [55], SAPHN [42], SPAIR [32], MIMO-UNet+ [5], TTFA [4], MPRNet [54], HINet [2], Restormer [53], and Stripformer [44]. We show quantitative results in Tab. 2 and visual results in Fig. 4.
Quantitative Results. We train our method on GoPro [28], and test it on GoPro and HIDE [39]. Moreover, we further test the GoPro-trained model on the real-world dataset: RealBlur [34] (RealBlurR and RealBlur-J). The PSNR/SSIM results on four benchmark datasets are reported in Tab. 2. Our method outperforms all compared state-of-the-art methods on all datasets.
When compared on synthetic datasets: GoPro and HIDE, our HI-Diff obtains the 0.25 dB gain on GoPro over Stripformer [44], the second best method. Meanwhile, compared with Restormer [53], the backbone of our method, our HI-Diff yields 0.41 dB and 0.24 dB gains on GoPro and HIDE.
When compared on real-world datasets: RealBlur-R and RealBlur-J, our HI-Diff exhibit a better generalization ability than other state-of-the-art algorithms. Compared with the recent best-performing method on GoPro, Stripformer, our method yields 0.33 dB on the RealBlur-J dataset. Besides, the HI-Diff outperforms the backbone, Restormer [53], by 0.09 dB on RealBlur-R and 0.19 dB on RealBlur-J. All these comparisons demonstrate the effectiveness of our HI-Diff.
Method	GoPro [28]	HIDE [39]	RealBlur-R [34]	RealBlur-J [34]
PSNR ↑	SSIM ↑	PSNR ↑	SSIM ↑	PSNR ↑	SSIM ↑	PSNR ↑	SSIM ↑
DeblurGAN [22]	28.70	0.858	24.51	0.871	33.79	0.903	27.97	0.834
DeepDeblur [28]	29.08	0.914	25.73	0.874	32.51	0.841	27.87	0.827
DeblurGAN-v2 [23]	29.55	0.934	26.61	0.875	35.26	0.944	28.70	0.866
SRN [43]	30.26	0.934	28.36	0.915	35.66	0.947	28.56	0.867
DBGAN [56]	31.10	0.942	28.94	0.915	33.78	0.909	24.93	0.745
MT-RNN [30]	31.15	0.945	29.15	0.918	35.79	0.951	28.44	0.862
DMPHN [55]	31.20	0.940	29.09	0.924	35.70	0.948	28.42	0.860
SAPHN [42]	31.85	0.948	29.98	0.930	N/A	N/A	N/A	N/A
SPAIR [32]	32.06	0.953	30.29	0.931	N/A	N/A	28.81	0.875
MIMO-UNet+ [5]	32.45	0.957	29.99	0.930	35.54	0.947	27.63	0.837
TTFA [4]	32.50	0.958	30.55	0.935	N/A	N/A	N/A	N/A
MPRNet [54]	32.66	0.959	30.96	0.939	35.99	0.952	28.70	0.873
HINet [2]	32.71	0.959	30.32	0.932	35.75	0.949	28.17	0.849
Restormer [53]	32.92	0.961	31.22	0.942	36.19	0.957	28.96	0.879
Stripformer [44]	33.08	0.962	31.03	0.940	36.08	0.954	28.82	0.876
HI-Diff (ours)	33.33	0.964	31.46	0.945	36.28	0.958	29.15	0.890
Table 2: Quantitative comparisons on the four benchmark datasets: GoPro [28], HIDE [39], and RealBlur [34] (RealBlur-R and RealBlur-J). All models are trained only on GoPro dataset. Best and second best results are colored with red and blue. Our HI-Diff outperforms state-of-the-art methods.
GoPro	MPRNet [54]	Restormer [53]	Stripformer [44]	HI-Diff (ours)
GT	Blurry	DBGAN [56]	MIMO-UNet+ [5]
MPRNet [54]	Restormer [53]	Stripformer [44]	HI-Diff (ours)
HIDE	GT	Blurry	DBGAN [56]	MIMO-UNet+ [5]
MPRNet [54]	Restormer [53]	Stripformer [44]	HI-Diff (ours)
GT	Blurry	DBGAN [56]	MIMO-UNet+ [5]
RealBlur-J	MPRNet [54]	Restormer [53]	Stripformer [44]	HI-Diff (ours)
Blurry	DBGAN [56]	DMPHN [55]	MIMO-UNet+ [5]
RWBI	MPRNet [54]	Restormer [53]	Stripformer [44]	HI-Diff (ours)
Figure 4: Visual comparison on GoPro [28], HIDE [39], RealBlur [34], and RWBI [56] datasets. RWBI only contains blurry images are captured with different hand-held devices. Models are trained only on the GoPro dataset. Our HI-Diff generates images with clearer details.
Visual Results. We show visual comparisons on GoPro [28], HIDE [39], RealBlur [34], and RWBI [56] in Fig. 4. We can observe that most compared methods suffer from artifacts or still contain significant blur effects. In contrast, our method can reconstruct more accurate textures and sharper edges. For example, in the HIDE sample, compared methods fail to reconstruct the white lines in the cloth, while our method recovers sharp textures. All these visual comparisons are consistent with quantitative results and further demonstrate the superiority of our method.
Dataset	Method	DeblurGAN-v2 [23]	SRN [43]	MIMO-UNet+ [5]	MPRNet [54]	BANet [45]	Stripformer [44]	HI-Diff (ours)
RealBlur-R	PSNR ↑	36.44	38.65	N/A	39.31	39.55	39.84	41.01
[34]	SSIM ↑	0.935	0.965	N/A	0.972	0.971	0.974	0.978
RealBlur-J	PSNR ↑	29.69	31.38	31.92	31.76	32.00	32.48	33.70
[34]	SSIM ↑	0.870	0.909	0.919	0.922	0.9230	0.929	0.941
Table 3: Quantitative comparisons on RealBlur [34]. All models are trained and tested on the corresponding datasets. Best and second best results are colored with red and blue.
Method	DMPHN [55]	MIMO-UNet+ [5]	MPRNet [54]	HINet [2]	Restormer [53]	Stripformer [44]	HI-Diff (ours)	HI-Diff-2 (ours)
Params (M)	21.70	16.11	20.13	88.67	26.13	19.71	28.49	23.99
FLOPs (G)	195.44	150.68	760.02	67.51	154.88	155.03	142.62	125.47
PSNR (dB)	31.20	32.45	32.66	32.71	32.92	33.08	33.33	33.28
Table 4: Model complexity comparisons. Params, FLOPs, and PSNR on GoPro are reported. When we calculate the FLOPs, the image size is set as  $3 \times 256 \times 256$ .
RealBlur-J
Figure 5: Visual comparison on the RealBlur [34] dataset. Models are trained on the RealBlur dataset.
DeblurGAN-v2 [23]
SRN [43]
4.4 Evaluation on Real-World Datasets
We further compare our HI-Diff with 6 state-of-the-art methods: DeblurGAN-v2 [23], SRN [43], MIMO-UNet+ [5], MPRNet [54], BANet [45], and Stripformer [44]. We show quantitative and visual results in Tab. 4 and Fig. 5. For fair comparisons, all previous method results are directly cited from the original papers or generated from official pre-trained models.
Quantitative Results. Table 3 reports PSNR/SSIM comparisons on real-world datasets: Real-Blur [34] (RealBlur-R and RealBlur-J). We train and test our HI-Diff on the RealBlur datasets, following previous works [54, 44]. Our method significantly outperforms other compared methods on the two datasets. Especially, compared with the recent best method, Stripformer, the HI-Diff obtains 1.17 dB and 1.22 dB gains on RealBlur-R and RealBlur-J, respectively.
Visual Results. We show visual comparisons on RealBlur in Fig. 5. Our method recovers sharper images with more high-frequency textures. However, most compared methods fail to recover clear images. For instance, compared methods have severe artifacts and blurring on green words, while our HI-Diff restores correct textures that are generally faithful to the ground truth. These visual results further demonstrate the strong ability of our HI-Diff for realistic image deblurring.
4.5 Model Size Analyses
We further show the comparison of model size (e.g., Params) and computational complexity (e.g., FLOPs) in Tab. 4. The FLOPs are measured when the image size is set as  $3 \times 256 \times 256$ . It shows that our HI-Diff has less FLOPs than CNN-based methods (e.g., MPRNet [54]). Meanwhile, compared with Transformer-based methods, Restormer [53] and Stripformer [44], our HI-Diff performs better with comparable Params and less FLOPs. It indicates that our method achieves a better trade-off between performance and computational consumption. To further demonstrate the effectiveness of our method, we provide another variant of HI-Diff with less Params and FLOPs and better performance than Restormer. More details about HI-Diff-2 are provided in the supplementary material.
In this paper, we design the Hierarchical Integration Diffusion Model (HI-Diff), for realistic image deblurring. Specifically, HI-Diff performs the diffusion model to generate the prior feature for a regression-based method during deblurring. The regression-based method preserves the general
distribution, while the prior feature generated by the diffusion model enhances the details of the deblurred image. Meanwhile, the diffusion model is performed in a highly compact latent space with computational efficiency. Furthermore, we propose the hierarchical integration module (HIM) to fuse the prior feature and image features of Transformer hierarchically for better generalization ability in complex blurry scenarios. Extensive experiments on synthetic and real-world blur datasets demonstrate that our proposed HI-Diff outperforms state-of-the-art methods.
Acknowledgments. This work is partly supported by NSFC grant 62141220, 61972253, U1908212, 62271414, Science Fund for Distinguished Young Scholars of Zhejiang Province (LR23F010001), Research Center for Industries of the Future at Westlake University.
[1] Abdullah Abuolaim and Michael S Brown. Defocus deblurring using dual-pixel data. In ECCV, 2020. 1, 2
[2] Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu, and Chengpeng Chen. Hinet: Half instance normalization network for image restoration. In CVPR, 2021. 7, 8, 9
[3] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In ICLR, 2020. 2, 3, 5
[4] Zhixiang Chi, Yang Wang, Yuanhao Yu, and Jin Tang. Test-time fast adaptation for dynamic scene deblurring via meta-auxiliary learning. In CVPR, 2021. 7, 8
[5] Sung-Jin Cho, Seo-Won Ji, Jun-Pyo Hong, Seung-Won Jung, and Sung-Jea Ko. Rethinking coarse-to-fine approach in single image deblurring. In ICCV, 2021. 7, 8, 9
[6] Sunghyun Cho and Seungyong Lee. Fast motion deblurring. In ACM SIGGRAPH, 2009. 2
[7] Maurizio Delbracio and Peyman Milanfar. Inversion by direct iteration: An alternative to denoising diffusion for image restoration. TMLR, 2023. 3
[8] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803, 2016. 1
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. 3, 4
[10] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021. 2, 4
[11] Zhenxuan Fang, Fangfang Wu, Weisheng Dong, Xin Li, Jinjian Wu, and Guangming Shi. Self-supervised non-uniform kernel estimation with flow-based motion prior for blind image deblurring. In CVPR, 2023. 1
[12] Rob Fergus, Barun Singh, Aaron Hertzmann, Sam T Roweis, and William T Freeman. Removing camera shake from a single photograph. In ACM SIGGRAPH, 2006. 1, 2
[13] Hongyun Gao, Xin Tao, Xiaoyong Shen, and Jiaya Jia. Dynamic scene deblurring with parameter selective sharing and nested skip connections. In CVPR, 2019. 1
[14] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014. 1
[15] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020. 1, 3, 5
[16] Seo-Won Ji, Jeongmin Lee, Seung-Wook Kim, Jun-Pyo Hong, Seung-Jin Baek, Seung-Won Jung, and Sung-Jea Ko. Xydeblur: divide and conquer for single image deblurring. In CVPR, 2022. 2
[17] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In NeurIPS, 2022. 3
[18] Bahjat Kawar, Jiaming Song, Stefano Ermon, and Michael Elad. Jpeg artifact correction using denoising diffusion restoration models. In NeurIPS Workshop, 2022. 3
[19] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 6
[20] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014. 5
[44] Fu-Jen Tsai, Yan-Tsung Peng, Yen-Yu Lin, Chung-Chi Tsai, and Chia-Wen Lin. Stripformer: Strip transformer for fast image deblurring. In ECCV, 2022. 1, 3, 6, 7, 8, 9
[45] Fu-Jen Tsai, Yan-Tsung Peng, Chung-Chi Tsai, Yen-Yu Lin, and Chia-Wen Lin. Banet: A blur-aware attention network for dynamic scene deblurring. TIP, 2022. 9
[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 3, 4
[47] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In ICLR, 2023. 3
[48] Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: A general u-shaped transformer for image restoration. In CVPR, 2022. 1, 3
[49] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. TIP, 2004. 6
[50] Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In CVPR, 2022. 1, 2, 3, 5
[51] Bin Xia, Yulun Zhang, Shiyin Wang, Yitong Wang, Xinglong Wu, Yapeng Tian, Wenming Yang, and Luc Van Gool. Diffir: Efficient diffusion model for image restoration. arXiv preprint arXiv:2303.09472, 2023. 1, 2, 3, 4, 5
[52] Li Xu, Shicheng Zheng, and Jiaya Jia. Unnatural 10 sparse representation for natural image deblurring. In CVPR, 2013. 2
[53] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In CVPR, 2022. 1, 3, 4, 6, 7, 8, 9
[54] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In CVPR, 2021. 6, 7, 8, 9
[55] Hongguang Zhang, Yuchao Dai, Hongdong Li, and Piotr Koniusz. Deep stacked hierarchical multi-patch network for image deblurring. In CVPR, 2019. 1, 2, 7, 8, 9
[56] Kaihao Zhang, Wenhan Luo, Yiran Zhong, Lin Ma, Bjorn Stenger, Wei Liu, and Hongdong Li. Deblurring by realistic blurring. In CVPR, 2020. 6, 7, 8