Hierarchical Integration Diffusion Model for Realistic Image Deblurring
Authors: Zheng Chen
Yulun Zhang
Ding Liu
Bin Xia
Jinjin Gu
Linghe Kong
Xin Yuan
Venue: NeurIPS 2023 spotlight (2023)
Keywords: image deblurring
diffusion model
TL;DR: Hierarchical integrate generative model (Diffusion Model) and regression-based model (Transformer) for realistic image deblurring

Abstract: Diffusion models (DMs) have recently been introduced in image deblurring and exhibited promising performance, particularly in terms of details reconstruction. However, the diffusion model requires a large number of inference iterations to recover the clean image from pure Gaussian noise, which consumes massive computational resources. Moreover, the distribution synthesized by the diffusion model is often misaligned with the target results, leading to restrictions in distortion-based metrics. To address the above issues, we propose the Hierarchical Integration Diffusion Model (HI-Diff), for realistic image deblurring. Specifically, we perform the DM in a highly compacted latent space to generate the prior feature for the deblurring process. The deblurring process is implemented by a regression-based method to obtain better distortion accuracy. Meanwhile, the highly compact latent space ensures the efficiency of the DM. Furthermore, we design the hierarchical integration module to fuse the prior into the regression-based model from multiple scales, enabling better generalization in complex blurry scenarios. Comprehensive experiments on synthetic and real-world blur datasets demonstrate that our HI-Diff outperforms state-of-the-art methods. Code and trained models are available at https://github.com/zhengchen1999/HI-Diff.

Hierarchical Integration Diffusion Model for Realistic Image Deblurring
Zheng Chen', Yulun Zhang Ding Liu Bin Xia', Jinjin Gu Linghe Kong'; Xin Yuan? 'Shanghai Jiao Tong University, 'ETH Zürich, 'Bytedance Inc, *Tsinghua University, Shanghai AI Laboratory, "The University of Sydney, "Westlake University
Abstract
Diffusion models (DMs) have recently been introduced in image deblurring and exhibited promising performance, particularly terms of details reconstruction. However, the diffusion model requires large number of inference iterations to recover the clean image from pure Gaussian noise, which consumes massive computational resources. Moreover, the distribution synthesized by the diffusion model is often misaligned with the target results, leading to restrictions in distortion- based metrics. To address the above issues, we propose the Hierarchical Integration Diffusion Model (HI-Diff), for realistic image deblurring. Specifically, we perform the DM in highly compacted latent space to generate the prior feature for the deblurring process. The deblurring process is implemented by regression bearessio--bbeedd method obtain better distortion accuracy. Mcanwhile, the highly compact latent space ensures the efficiency of the DM. Furthermore, we design the hierarchical integration module to fuse the prior into the regression-ban baaeee model from multiple scales, enabling better generalization in complex blurry scenarios. Comprehensive experiments on synthetic and Diff outperforms state-of-the-art methods. Code and trained models are available at https: //gi github. com/ wtt. www.gchennnccnennnm 999/HI HDI9/HIIiff..
blur datasets demonstrate that our HI-
Introduction
Image deblurring is long-standing computer vision task, aiming to recover sharp image from blurred observation. Blurred artifacts can arise from various factors, e.g., camera shake and fast- moving objects. address this challenge, traditional methods typically formulate the task as an optimization problem and incorporate natural priors [12, 38, 24, 29] to regularize the solution space. However, since blur real scenarios is complex and non-uniform, which hard be modeled by the specific priors, these algorithms suffer from poor generalization in complex situations.
With the development of deep learning, convolutional neural networks (CNNs) have been applied in image deblurring [37, 43, 13, 55, 1, 11]. Among them, the regression-basse methods have shown remarkable success, especially terms of distortion- bassed metries (e.g.. PSNR). Moreover, Transformer- based approaches [48, 53, 44, 21]. which can capture long-distance dependencies, are introduced as an alternative CNNs. These methods further boost the deblurring performance. However, regression-basee methods are prone to recovering images with fewer details, since the regression losses are conservative with high-frequency details [36].
Apart from regression- based methods, deep generative models, like generative adversarial networks (GANs) [14] and normalizing flows [8], provide other solutions for generating complex details. Recently, Diffusion Models (DMs) [40, 15] have exhibited impressive performance in image syn- thesis [41, 35] and restoration tasks (including image deblurring) [50, 33, 36, 51J. DMs generate high-fidelity images through a stochastic iterative denoising process from pure white Gaussian noise. Compared to other generative models, such as GANs, DMs generate more accurate target distribution without encountering optimization instability or mode collapse.
*Corresponding authors: Yulun Zhang, yulun100@ gmail.com; Linghe Kong, linghe.kong sjtu.edu.en
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Nonetheless, DM-based methods face several challenges. First, DMs are limited by the high computational cost of generating samples. It requires high number of inference steps. Some methods reduce the number of iterations by predicting the residual distribution [50] or applying advanced sampling strategies [41, 3]. However, the overall computational complexity still high, especially for high-resolution images. Furthermore, although some methods perform DMs on latent space [35], the compression ratio is small (e.g., 8 times), due to limitations in generation quality. Second, generative models, including DMs, tend to produce undesired artifacts not present in the original clean image. Besides, the generated details may also be misaligned with real targets. These lead to the poor performance of DM regarding some iittortionbbsel metries (e.g., PSNR).
The aforementioned issues promote us apply DMs from another perspective. The motivation for our work is threefold. First, due to the high computational overhead in the image space, we also consider applying DMs on the low-dimensional latent space. Meanwhile, we increase the compression ratio of latent to effectively reduce the computational complexity. Second, since the advantages of regression-badd based methods in distortion aceuracy, we integrate DMs and regression- -based methods to improve the performance of DMs regarding distortion. Third, considering the non-uniform blur in real scenarios, we apply a hierarchical approach to enhance the generalization of the method.
Specifically, we apply DMs to generate priors in latent space, inspired by the application of priors in traditional deblurring algorithms. The priors are integrated into the regression- based model in a
hierarchical manner to improve the details of restored images. The design contains three benefits: (1) The dimension of latent space can be very low, since the regression- bassed model restores most of the distribution.
(2) The issue of distortion caused by misaligned details generated by DMs can be avoided. (3) The hierarchical integration enables better generalization in complex blurry scenarios.
Based on the above analysis. we propose a novel approach called the Hierarchical Integration Diffusion Model (HI-Diff) for realistic image deblurring. Following previous practice [10, 35, 51J, we perform a two-stage training to realize latent compression and the training of the DM, respectively. Without loss of generality, for the regression-based methods, we choose the encoder- decoder Transformer architecture. In the first stage, we compress the ground- -ruth image into a highly compact latent representation as the prior feature through a latent encoder (LE). We propose the hicrarchical integration module (HIM) that can fuse the prior and intermediate features of Transformer at multiple levels. We jointly train the LE and Transformer to effectively construct the prior. In the second stage, we train a latent diffusion model to generate the prior feature in the latent space from the Gaussian noise and guide Transformer through the HIM. Similar to the first stage, we jointly train the DM and Transformer. In summary, our main contributions are three-fold as follows:
We propose the Hierarchical Integration Diffusion Model (HI-Diff), for realistic image deblurring. The HI-Diff leverages the power of diffusion models to generate informative priors, that are integrated into the deblurring process hierarchically for better results.
We apply the diffusion model in highly compact latent space generate the prior. Mean- while, we propose the hierarchical integration module to fuse the prior into the regression- based model from multiple scales, enabling the generalization in complex blurry scenarios.
Extensive experiments conducted on synthetic and real-world blur datasets demonstrate the superior performance of the HI-Diff in comparison to state-of-the-an deblurring methods.
2
Related Work
2.1 Image Deblurring
Traditional Methods. Traditional deblurring methods generally formulate the problem as opti- mization problem [12, 38, 24, 6, 29]. They utilize various natural image priors for sharp images and blur kernels. Common methods include local smoothness prior [38], sparse image prior [24], Lo-norm gradient prior [52], and dark channel prior [29]. However, these methods rely on manually crafted priors, which results in poor generalization ability and limited performance in complex situations.
Deep CNN- Methods. With the rapid development of deep learning. significant progress has been made in image deblurring using CNN-based methods [37, 28, 43, 55, 1, 16]. For instance, MSCNN [28] designs multi-scale CNN to restore sharp images. SRN [43] proposes a coarse-to-fin scale-recurrent network for more efficient multi-scale image deblurring. DMPHN [55] devises a deep multi-patch hierarchical deblurring network to exploit the deblurring information at different scales. XYDeblur [16], on the other hand, divides the original deblurring problem into two sub- uub-problems.
2

---

Are
LE
Diflusion Process
Linwer
Z
Dermisinng Natwwakk
Ramape
lCT
Zy-1
PCC 2
Z
(b) Diffusion Model
(c) Multi-Scale
ABC
X
Mulplootinn
Upls
Avg Pool
Z1
Z2
Corv 5x33
Linear
Block
Blook
Block
Corn D43
Attur
HIM
Corn 21.3
Block
Block
Norm Preue
NNmm
P Ponkiinsuutty
grar
los
Z1
Zi
Zg
22
X
7g
in!
(a) Transformer
(d) HIM
(e) LE
Figure 1: The overall framework our HI-Diff. (a) Transformer, adopts hierarchical encoder- architecture, equipped with HIM, for the deblurring process. (b) Diffusion Model, performed in a highly compact latent space for computational efficiency. (c) The multi-scale prior feature {Zi, 22, 23) } is obtained stage two, 2₁ =2. (d) The hierarchical integration module (HIM), calculates cross-attention between the intermediate feature of Transformer and the multi-scale prior feature. (c) The latent encoder (LE), where the size of the input feature (in) is H X x6 for LE, and H x W x3 for LEDM.
downsampling the prior feature multiple times. In stage one, % == in
Transformer-basee Methods. Recently, Transformer [46, 9] proposed in natural language processing (NLP), which utilizes the self-attention (SA) mechanism to model long-range dependence, has shown remarkable performance in image deblurring [48, 53, 44]. For example, Restormer [53] utilizes "transposed" attention, which calculates the SA across channel dimension instead of spatial dimension, to model global information and keep computational efficiency to large images. Stripformer [44] constructs horizontal and vertical strip tokens to catch region-specific blurred patterns with different orientations in dynamic scenes. These methods further improved the deblurring performance com- pared with CNN-based Methods. However, they are limited in recovering image details, since the regression-bait methods conservative with nigh-frequency details [36].
2.2 Diffusion Models
Diffusion Models (DMs) [40, 15] are probabilistic generative models that enable the construction of desired data samples from the Gaussian noise via a stochastic iterative denoising process. DMs have demonstrated excellent performance
various image restoration tasks [18, 17, 47, 7], such as super-resolution [36, 25], inpainting [35]. and deblurring [50]. A survey [26] summarizes more diffusion-basse based restoration methods. For example, DiffIR [5] utilizes the diffusion model to generate prior representation for image restoration, and applies two-stage training approach. DvSR [50] introduces conditional DMs into image deblurring with residual model, showing more realistic results than regression-basse methods. InDI [7] directly produces high-qualin results by restoring low-quality images. However, one the restrictions of DMs is the large number of iteration steps required in inference. Some methods attempt to mitigate this limitation by applying well-designed noise schedules [3] and sampling strategies [41] or performing DM on the latent space [35]. Despite these efforts, the overall complexity remains high, especially for high high-resolution images common in image deblurring. Additionally, these methods are susceptible to misaligned details distribution and undesired artifacts, resulting in poor performance in distortion- -aoordd metrics, e.g., PSNR.
Method
Our goal to integrate the diffusion model (DM) and the regression-based model, thus overcoming the shortcomings of the DM and realizing better deblurring. We propose the Hierarchical Integration Diffusion Model (HI-Diff). The HI-Diff performs the DM to generate the prior feature that is integrated hicrarchically into Transformer (the regression-based model).

---

The overall framework of the proposed HI-Diff is depicted in Fig. 1. The HI-Diff consists of two parts: Transformer and the latent diffusion model. We adopt Restormer [53]. hierarchical encoder-decoder Transformer architecture, in our method. Compared with other Transformer networks designed specifically for image deblurring, Restormer a general restoration model. Applying this model in our method can better illustrate the effectiveness of our proposed method. Meanwhile, following previous practice [10, 35, 51J, we train our HI-Diff with a two-stage training strategy, to realize latent compression and the training of the DM. In this section, we first elaborate on the two-stage training framework and then illustrate the whole deblurring inference process.
3.1 Stage One: Latent Compression
In stage one, our purpose is to compress the ground truth image into the highly compact latent space, and utilize guide Transformer the deblurring process. As shown in Fig. I(a, b), we compress the ground truth images through a latent encoder (LE) to obtain compact representation as the prior feature. Then we integrate the prior feature into Transformer through the hierarchical integration module (HIM). The prior feature can provide explicit guidance for Transformer, thus increasing the details of the reconstructed image. Next, we describe these parts in detail.
Latent Encoder. As shown in Fig. 1(b), given the blurry input image E x x3 and its corresponding ground truth counterpart ToTER#*W** x3, we first concatenate them along the channel dimension and feed them into the latent encoder (LE) generate the prior feature ZER~xC* Here H and W represent the image height and width, while N and C" are the token number and channel dimensions of z. Importantly, the token number N is a constant much smaller than H x W. The compression ratio ( Hx ) is much higher than that applied in previous latent diffusion [35] (e.g., 8 times). Therefore, the computational burden of the subsequent latent diffusion model is effectively reduced. The details of the latent encoder are depicted in Fig. I(e), which contains T. residual blocks.
Hierarchical Integration Module. To effectively integrate the prior feature and intermediate feature of Transformer, we propose the hierarchical integration module (HIM). As illustrated in Fig. (a), the HIM is placed in front of each encoder and decoder. For each HIM, cross-attention is computed between the prior and intermediate features for feature fusion. This module allows the information in the prior feature be aggregated into features of Transformer.
Specifically, as shown Fig. (cd), given the intermediate feature &R we reshaped it as tokens X,E An xč; where A x W is spatial resolution, and Ô denotes channel dimension. Then we linearly project X, into QER AW xŌ (guery). Similarly, we project the prior feature Z;ER XCC as XxO (key) and VeR x (value). The cross anssssatttntion is formulated as:
Q Wox,, K WKZ, V Wyzi,
Attention Q2 K, V) = SoftMax (QK" /O) V.
where W WOEROXC WRERC'xC and W ER' C' XO represent learnable parameters of linear pro- jections without bias. As vanilla multi-head self-attention [46, 9], we separate channels into multiple "heads" and calculate the attention operations. Note that Fig. (dd) depicts the situation with sin- gle head and omits some details for simplification. Finally, we reshape and project the output of xX
cross-attention, and add it with X,n to derive the output feature
Moreover, since the non-uniform blur in real scenarios, the single-scale prior feature cannot adapt well to complex blurry situations. Therefore, we generate the multiple-scale prior feature {2,, Z2, Z3 (where z =2), by downsampling the prior feature z, as shown in Fig. 1(c). The multiple-scale prior feature adapts to different scale intermediate features for better fusion. The effectiveness of the hierarchical integration with the multiple-scale prior feature is demonstrated in Sec. 4.2.
Training Strategy. To ensure the effectivvnnss of the latent encoder (LE) in constructing the prior feature, we optimize jointly with Transformer using the L, loss function, defined as:
Lasblur IDB Icr Ila,
where I D the deblurred image, and IcT represents its corresponding ground truth.
3.2 Stage Two: Latent Diffusion Model
In stage two, latent diffusion model (DM) is trained to learn to generate the prior feature, that enhances the deblurring process of Transformer through HIM.

---

Diffusion Model. Specifically, our latent diffusion model is based on conditional denoising diffusion probabillistic models [15, 3, 36, 35]. The diffusion model involves forward diffusion process and reverse denoising process, as illustrated in Fig. 1(b).
In the diffusion process, given ground truth image, we first adopt the latent encoder (LE) trained in stage one to generate the corresponding prior feature ZER-xc", We take the starting point of the forward Markov process, and gradually add Gaussian noise to it over T iterations as follows:
T II g(zz zz-1), q(z; Z NZz V 3,
q Zo)
(3)
where (=1,.. T; be represents the noisy features at the I-th step; zo=z for unification; E1:2-((0) 1) are hyperparameters that control the variance of the noise; W denotes the Gaussian distribution. Through iterative derivation with reparameterization [20]. Eq. (3) can be written as:
2
(Zc (o) = N(zz Vārzo, (I a = 1-188 -e, ai II 04
f=1
In the reverse process, we aim to generate the prior feature from a pure Gaussian distribution. The reverse process is a T-step Markov chain that runs backwards from Zy to Zy. Specifically, for the reverse step from z; to Zt-1, we use the posterior distribution as:
1 1,11,, Me (e zo) q(x:-1 Ze, zo) NN 15 Me (ze, zo):
1 0z E), (5) 1 (z. VI āi
at
Vott
where e represents the noise in 2g, and is the only uncertain variable. Following previous work [15, 36, 35, 51], we adopt neural network (denoted as denoising network, cσ) to estimate the noise for each step. Since DM operates in the latent space, we utilize another latent encoder, denoted as LEDM. with the same structure as LE. LEDM compresses the blurry image Blur into latent space to get the condition latent cER-xx xx", The denoising network predicts the noise conditioned on the z, and c, i.e. (z C, t). With the substitution of εa in Eq. (5) and set the variance to (1-a)), we get:
04
2t-1
yy:
Eo (Z C, t)) + vI -
(6)
Va
V1
where (00, I). By iteratively sampling z, using Eq. (6) T times, we can generate the predicted prior feature NxC² as shown in Fig. 1(b). The predicted prior feature is then used to guide Transformer, that is, 21 in Fig. 1cc). Notably, since the distribution of the latent space (R N X much simpler than that of images (R (R"xxxx), [50, 51], the prior feature can be generated with small number of iterations. We further explore the iteration numbers T in Sec. 4.2.
Training Strategy. Training DM means training denoising network eg. Previous works [15,41] train the model by optimizing the weighted variational bound. The training objective is:
Volle 60 (Vū,z + vI - O4,6, c. 1)1
(7)
where z and e are prior feature and condition latent defined above; TE( is random time-step; e~ I) denotes sampled noise. However, the objective in Eq. (7) only trains DM. Since the slight deviation between the predicted prior feature and the actual prior z, directly combining DM with Transformer could cause a mismatch, which restricts the deblurring performance.
To overcome this issue, we jointly train the diffusion model and Transformer, inspired by previous work [51]. In this approach, for each training iteration, we use the prior feature z to generate the noise sample zr through Eq. (4), As the time-step T is small in latent space, then run the complete T iteration reverse processes (Eq. (6)) to generate the predicted prior feature 2. The à is used to guide Transformer through HIM (z) =)). Finally, the training loss is given by the sum of the deblurring loss Lueblur and the diffusion loss Laillusion where the diffusion loss Ldifiiusinn - z z/|2-
3.3 Inference
After completing the two-stage training, given a blurry input image I II Nxxx3 the HI-Diff first compresses Biur into condition latent NxC" via latent encoder LEpm. Then the predicted prior feature xxR is generated by the diffusion model. Specifically, the diffusion model iteratively executes the reverse process (defined in Eq. (6) T times starting from randomly sampled Gaussian Noise (E~N((, I)). Finally, the HI-Diff reconstructs the deblurred image IDDER from the input image Litur, using Transformer, which is enhanced by the prior feature 2.
5

---

Method
Prior
Multi-Scale
Joint-training
Params (M)
FLOPs (G)
PSNR (dB)
SSIM
Basline
X
X
19.13
117.25
31.96
0.9528
Single-Guide
x
7
21.98
125.39
32.00
0.9534
Split-Training
X
23,99
125.47
30.73
0,9434
HI-Diff (ours)
23.99
125.47
32.24
0.9558
Table 1: Ablation study. We train and test models on the GoPro [28] dataset. Image size is
3x256x256 to calculate FLOPs. Prior: the prior feature generated by the diffusion model; Multi- scale: the multi-scale prior feature for hierarchical integration (as opposed to single-scale); Joint- training: the diffusion model and Transformer trained jointly in stage two.
4 Experiments
4.1 Experimental Settings
Data and Evaluation. Following previous image deblurring methods, we evaluate our method on synthetic datasets (GoPro [28] and HIDE [39]) and the real-world dataset (RealBlur [34] and RWBI [56]). The GoPro dataset contains 2,103 pairs of blurry and sharp images for training and 1,111 image pairs for testing. The HIDE dataset provides testing 2,025 images. The RealBlur dataset has two sub-set: RealBlur-J and RealBlur-E R. Each sub-set consists of 3,758 training pairs and 980 testing pairs. The RWBI dataset contains 3,112 blurry images captured with different hand-hel devices. For synthetic datasets, we train HI-Diff on the GoPro training set and test it on GoPro and HIDE. Moreover, we further test the GoPro-trained model on RealBlur and RWBI to evaluate the generalization of our method. For real-world datasets, we train and test HI-Diff on RealBlur datasets, following previous works [54, 44]. We adopt common metrics: PSNR and SSIM [49].
Implementation Details. Our HI-Diff consists of two parts: Transformer and the latent diffusion model. For Transformer, without loss of generality, we apply Restormer [53]. a 4-level encoder- decoder Transformer architecture. From level-1 to level-4, we set the number of Transformer blocks as [3,5,5,6), the number of channels as 148.96,192,388) 192,384,, and the attention heads [1,2.4.8). Besides, there are 4 blocks in the refinement stage. The channel expansion factor is 2.66. For the latent diffusion model, the token number N is 16, and the channel dimension C' is 256. The latent encoder contains -=6 residual blocks. We set the total time-step T' as 8, and the variance hyperparameters B1.7 constants increasing linearly from 31= =1 =099..
Training Settings. We train our HI-Diff with Adam optimizer [19] with 31=0.9 and $2=0.99. For stage one, the total training iterations are 300K. The initial learning rate is set as 2x 10 and gradually reduced to x 10 with the cosine annealing [27]. Following previous work [53], we adopt progres- sive learning. Specifically, we set the initial patch size as 128 and the patch size as 64. We progres- sively update the patch size and batch size pairs to (1160,40),(199) (4,)),19222 (256), 1320 16) ,8) at iterations [20K, 4IKK44KK660KK.. 60KK8KK,KKKK,,. 100K]. For stage two, we adopt the same training settings in stage one. Moreover, following previous works [53, 54], we apply random rotation and flips for data augmentation. We use PyTorch [31] to implement our models with 4 A100 GPUs.
4.2 Ablation Study
In this section, we study the effects of different designs of our proposed method. We conduct all experiments the dataset GoPro [28]. The iterations for stages one and two are 100K, respectively. The image patch size and batch size are set as 192 x 192 and 32. When we calculate the FLOPs, the image size is 3x256x 256. The results are reported in Tab. and Figs. 2 and 3.
We construct baseline model without prior generated from diffusion in the first row of Tab. 1. denoted as Baseline, which is actually the vanilla Restormer [53] architecture. For fair comparisons, Baseline adopts the same implementation and training settings as HI-Diff (ours, listed in the fourth row). Comparing Baseline and HI-Diff, we can discover that using the prior feature generated by the latent diffusion model (DM) yields a 0.28 dB improvement. It demonstrates the effectiveness of our proposed method. In addition, the HI-Diff, integrated with diffusion, only adds 4. 4.86M Params and 8.22G FLOPs over Baseline. It reveals that performing the diffusion model in highly compact latent space is very efficient. Furthermore, we show the visual comparisons of the Baseline (without the prior feature) and Hi-Diff in Fig. (first row). Our HI-Diff, generates sharper textures and complete structures, compared with Bascline.
Effects of Diffuuson Prior.

---

AEAA د
C
32.00
872-3300
872-2000 872-3300
31.00
P5NR (dB)
10.00
GT
Baseline
IIl Dift
29.00
28.00
20 Number itteaatoons
Single
Deblurred samples for different models in Tab. 1. The first row shows effects of diffusion prior, while the second row exhibits effects of hierarchical integration.
Figure 2:
Figure 3: Ablation study of the num- ber of iterations T in diffusion model, T: {1, 2, 4, 32)..
Effects of Hierarchical Integration. We conduct ablation experiment the hierarchical inte- gration with the multi-scale prior feature (illustrated Fig. (c).. We apply the single-scale prior feature Transformer, which means set Z1 22=2==23= (or 2). We denote the new model as Single-Guide, and its result is shown in the second row. We find that the PSNR of Single-Guide drops by 0.24dB compared with HI-Diff, which adopts the multi-scale prior feature. It indicates that the single-scale prior feature cannot adapt well to complex blurry situations. The visual comparison in Fig. 2 (second row) reveals that applying the hierarchical integration restores better-deblurred images.
Effects of Joint Training Strategy. We explore the impact of the joint training strategy in stage two. We train model only optimized diffusion model in stage two, denoted as Split-Training. Specifically, for Split-Training, we first generate the prior feature z from the ground truth and then apply the training objective defined in Eq. (7) to train the diffusion model alone. Then the diffusion model is directly combined with Transformer for evaluation after training is complete. For fair comparisons, Split-Trainine applies the same pre-trained (stage one) model HI-Diff in stage two training, and the iterations are 100K. Comparing Split-Traie Training and HI-Diff, HI-Diff is significantly better than Split-Training by 1.51 dB on PSNR value. These results are consistent with the analysis in Sec. 3.2 and demonstrate the importance of the joint training strategy.
Effects of Iterations Number. We further conduct an ablation study investigate the influence of iteration numbers in the diffusion model. We set six different iterations T: {1, 2, 4, 8, 16, 32} for the diffusion model. Meanwhile, the variance hyperparameters 31.T are always linearly interpolated from 31=0.1 to 0.99 for different T. We plot the PSNR of different iterations T Fig. 3. We find that only one iteration inference cannot generate the reasonable prior feature, limiting the deblurring performance. Besides, when the number of iterations reaches 8, the curve basically converges. indicates that only small number of iterations is needed to generate the suitable prior feature, since the simple distribution of the highly compact latent space (only contains 1= tokens).
4.3 Evaluation on Synthetic Datasets
We compare our HI-Diff with 16 tate-of-thheart methods: DeblurGAN [22], DeepDeblur [28].
DeblurGAN-vi [23], SRN [43], DBGAN [56], MT-RNN [30], DMPHN [55]. SAPHN [42].
SPAIR [32], MIMO-UNet+ [5], TTFA [4], MPRNet [54]. HINet [2], Restormer [53]. and Strip-
former [44]. We show quantitative results in Tab. 2 and visual results in Fig. 4.
Quantitative Results. We train our method on GoPro [28]. and test it on GoPro and HIDE [39].
Moreover, we further test the GoPro-trained model on the real-world dataset: RealBlur [34] (RealBlur- and RealBlur-J). The PSNR/SSIM results on four benchmark datasets are reported in Tab. 2. Our method outperforms all compared state-of-the- methods on all datasets.
When compared on synthetic datasets: GoPro and HIDE, our HI-Diff obtains the 0.25 dB gain on GoPro over Stripformer [44], the second best method. Meanwhile, compared with Restormer [53], the backbone of our method, our HI-Diff yields 0.41 dB and 0.24 dB gains on GoPro and HIDE.
When compared on real-world datasets: RealBlur-R R and RealBlur-J, our HI-Diff exhibit better gen- eralization ability than other state-of-the-art algorithms. Compared with the recent best-performing method on GoPro, Stripformer, our method yields 0.33 dB on the RealBlur-J dataset. Besides, the HI-Diff outperforms the backbone, Restormer [53], by 0.09 dB on RealBlur-R and 0. dB on RealBlur-J. J. All these comparisons demonstrate the effectiveness of our HI-Diff.
7

---

GoPro [28]
HIDE
RealBlur-R [34]
RealBlur-J [34]
Method
PSNR 1
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM f
DeblurGAN [22]
0.858
0.871
33.79
0,903
27.97
DeepDehlur [28]
29.08
0.914
25.73
0.874
32.51
0.841
27.87
0.827
DeblurGAN-v2 [23]
29.55
0.934
26.61
0.875
35.26
0.944
28.70
0.866
SRN [43]
30.26
0.934
28.36
0.915
35,66
0.947
28.56
0.867
DBGAN [56]
31.10
0.942
28.94
0.915
33.78
0.909
24.93
0.745
MT-RNN [30]
31.15
0.945
29.15
0.918
35.79
0,951
28.44
0.862
DMPHIN [55]
0.940
29.09
35.70
28.42
SAPHN [42]
31.85
0.948
29.98
0.930
NIA
N/A
N/A
N/A
SPAIR [32]
32.06
0.953
30.29
0,931
N/A
NIA
28.81
0.875
MIMO-UNet+ [5]
32,45
0.957
29.99
0.930
35.54
0.947
27.63
0.837
TTFA [4]
32.50
0.958
30.55
0.935
N/A
N/A
N/A
N/A
MPRNet [54]
32,66
0,959
30.96
0,939
35,99
0,952
28.70
IIINet
0.959
30.32
35.75
28.17
Restormer [53]
32,92
0.961
31.22
0.942
36.19
0.957
28.96
0.879
Stripformer [44]
33.08
0.962
31.03
0,940
36.08
0,954
28.82
0.876
HI-Diff (ours)
33.33
0.964
31.46
0.945
36.28
0.958
29.15
0.890
Table 2: Quantitative comparisons on the four benchmark datsets: GoPro [28], HIDE [39], and
RealBlur [34] (RealBlur-R and RealBlur-J,. All models are trained only on GoPro dataset. Best and second best results are colored with red and blue. Our HI-Diff outperforms state-of-it of-the-art methods.
5511, 5513, 5515
5515, 501
551
5511, 5511, 501, 750A, 750B
501 750A, 750B
750A, 7508
GT
Blurry
DBGAN [56]
MIMO-UNet-
5511 5513,
5511 1513,513,555
5511 1111,551..557 5515
5511, 5511, 5513. 5515 501, 750A, 7508
501 750A, 7508
501, 750A, 7508 501, 750A 750
GoPro
MPRNet |S4]
Restormer [53]
Stripformer [44]
HI-Diil (ours)
GT
Blurry
DBGAN [56]
MIMO UNet-
IIDE
MPRNet [S4]
Restormer [33
Stripformer [44]
III DifT (ours)
23 1720
1720
GT
Blurry
DBGAN [56]
MIMO UNeO-
235 1720
23 1720
RealBlur.J
MPRNet [54]
Restormer [53]
Stripformer [44]
HI-Diff (ours)
Blurry
DBGAN [6]]
DMPHN [S5]
MIMO-UNet-
RWBI
MPRNet [54]
Restormer [53]
Stripformer [44]
HI-Diff (ours)
Figure 4: Visual comparison on GoPro [28], HIDE [39], RealBlur [34], and RWBI [56] datasets.
RWBI only contains blurry images captured with different hand-held devices. Models are trained only on the GoPro dataset. Our HI-Diff generates images with clearer details.
Visual Results. We show visual comparisons on GoPro [28], HIDE [39], RealBlur [34], and RWBI [56] in Fig. 4. We can observe that most compared methods suffer from artifacts or still contain significant blur effects.
contrast, our method can reconstruct more accurate textures and sharper edges. For example, in the HIDE sample, compared methods fail to reconstruct the white lines in the cloth, while our method recover sharp textures. All these visual comparisons are consistent with quantitative results and further demonstrate the superiority of our method.

---

DeblurGAN-v2
SRN
MIMO-UNet+
MPRNet
BANet [45]
Stripformer [44]
HI-Diff
Dataset
Method
[23]
[43]
[S]
[54]
(ours)
RealBlur-R
PSNR
36.44
38.65
N/A
39.31
39.55
39.84
41.01
[34]
SSIM
0.935
0.965
N/A
0.972
0.971
0.974
0.978
RealBlur-J
PSNR
29.69
31.38
31,92
31.76
32.00
32.48
33.70
SSIM
0.870
0.909
0.919
0.922
0.9230
0.929
0.941
Table 3: Quantitative comparisons RealBlur [34]. All models trained and tested the corresponding datasets. Best and second best results are colored with red and blue.
DMPHN [55]
MIMO-UNer+
MPRNet
HINet
Restormer
Stripformer [44]
HI-Diff (ours)
HI-Diff-2
Method
[5]
[54]
[2]
[53]
(ours)
Params (M)
21.70
16.11
20.13
88.67
26.13
19.71
28.49
23.99
FLOPs (G)
195,44
150.68
760.02
67.51
154,88
155.03
142.62
125,47
PSNR (dB) 31.20
32.45
32.66
32.71
32,92
33.08
33.33
33.28
Table 4:
Model complexity comparisons. Params, FLOPs, and PSNR on GoPro are reported. When we calculate the FLOPs, the image size is set 2565
Shat
GT
Blurry
DoblurGAN-v2 [23]
SRN [43]
2hal
RealBlur-J
MIMO-UNct+ [5]
MPRNot [54]
Stripfoemer [44]
HI-Diff (ours)
Figure 5: Visual comparison on the RealBlur [34] dataset. Models are trained on the RealBlur dataset.
4.4 Evaluation on Real- World Datasets
We further compare our HI-Diff with state-of-the-arr methods: DeblurGAN-vv [23], SRN [43], MIMO-UNet+ [5]. MPRNet [54], BANet [45], and Stripformer [44]. We show quantitative and visual results in Tab. 4 and Fig. 5. For fair comparisons, all previous method results are directly cited from the original papers generated from official pre-trained models.
Quantitative Results. Table reports PSNR/SSIM comparisons on real-world datasets: Real- Blur [34] (RealBlur-R and RealBlur-J). We train and test our HI-Diff on the RealBlur datasets, following previous works [54,44] Our method significantly outperforms other compared methods on the two datasets. Especially, compared with the recent best method, Stripformer, the HI-Diff obtains 1.17 dB and 1.22 dl gains on RealBlur-R and RealBlur-J, respectively.
Visual Results. We show visual comparisons on RealBlur in Fig. 5. Our method recovers sharper images with more high-frequency textures. However, most compared methods fail recover clear images. For instance, compared methods have severe artifacts and blurring on green words, while our HI-Diff restores correct textures that are generally faithful to the ground truth. These visual results further demonstrate the strong ability of our HI-Diff for realistic image deblurring.
4.5 Model Size Analyses
We further show the comparison of model size (e.g., Params) and computational complexity (e.g.. FLOPs) Tab. 4. The FLOPPs are measured when the image size is set as xx56xx25 shows that our HI-Diff has less FLOPs than CNN-based methods (e.g., MPRNet [54]). Meanwhile, compared with Transformer-bant based methods, Restormer [53] and Stripformer [44], our HI-Diff performs better with comparable Params and less FLOPs. It indicates that our method achieves a better trade-off between performance and computational consumption. To further demonstrate the effectiveness of our method, we provide another variant of HI-Diff with less Params and FLOPs and better performance than Restormer. More details about HI-Diff-2 are provided in the supplementary material.
5 Conclusion
In this paper, we design the Hierarchical Integration Diffusion Model (HI-Diff), for realistic image deblurring. Specifically, HI-Diff performs the diffusion model to generate the prior feature for regression-bart based method during deblurring. The regression-basts method preserves the general

---

distribution, while the prior feature generated by the diffusion model enhances the details of the deblurred image. Meanwhile, the diffusion model is performed in a highly compact latent space with computational efficiency. Furthermore, we propose the hierarchical integration module (HIM) to fuse the prior feature and image features of Transformer hierarchically for better generalization ability in complex blurry scenarios. Extensive experiments on synthetic and real-world blur datasets demonstrate that our proposed HI-Diff outperforms state-of-the-ar methods.
Acknowledgments. This work is partly supported by NSFC grant 62141220, 61972253, U1908212, 62271414, Science Fund for Distinguished Young Scholars of Zhejiang Province (1.R230010001), Research Center for Industries of the Future at Westlake University.
References
[1] Abdullah Abuolaim and Michael S Brown. Defocus deblurring using dual-pixel data. In ECCV, 2020. 1,2
[2] Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu, and Chengpeng Chen. Hinet: Half instance normalization network for image restoration. In CVPR, 2021. 7, 8,9
[3] Nanxin Chen, Yu Zhang. Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In ICLR, 2020. 3,5
[4] Zhixiang Chi, Yang Wang, Yuanhao Yu, and Jin Tang. Test-time fast adaptation for dynamic seene deblurring via mela-auxiliary learning. In CVPR, 2021. 7,8
[5] Sung-Jin Cho, Seo-Won Ji. Jun-Pyo Hong, Seung-Wor Won Jung, and Sung-Jea Ko. Rethinking coarse-to-fnee approach in single image deblurring. In ICCV, 2021. 7, 8.9
[6] Sunghyun Cho and Seungyong Lee. Fast motion deblurring. In ACM SIGGRAPH, 2009.
[7] Mauricio Delbracio and Peyman Milanfar. Inversion by direct iteration: An altemnatie to denoising diffusion for image restoration. TMLR, 2023.
[8] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.0880 2016.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. In ICLR, 2021. 3,4 4
[10] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high- resolutiooonn image synthesis. In CVPR, 2021. 4
[11] Zhenxuan Fang, Fangfang Wu, Weisheng Dong, Xin Li, Jinjian Wu, and Guangming Shi. Self-supervised non-uniform kernel estimation with flow-based motion prior for blind image deblurring. In CVPR, 2023. 1
[12] Rob Fergus, Barun Singh. Aaron Hertzmann, Sam T Roweis, and William T Freeman. Removing camera shake from single photograph. In ACM SIGGRAPH, 2006. 1.2
[13] Hongyun Gao, Xin Tao, Xiaoyong Shen, and Jiaya Jia. Dynamic scene deblurring with parameter selective sharing and nested skip connections. In CVPR, 2019.
[14] Ian Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
[15] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020. 1, 5.
[16] Sco-Won Ji, Jeongmin Lee, Scung- Wook Kim, Jun-Pyo Hong, Seung-Jin Back, Scung-Wor Won Jung, and Sung-Jca Ko. Xydeblur: divide and conquer for single image deblurring. In CVPR, 2022. 2
[17] Bahjat Kawar, Michael Flad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In NeurlPS, 2022. 3
[18] Bahjat Kawar, Jiaming Song, Stefano Ermon, Michael Elad. Jpeg artifact correction using denoising diffusion restoration models. In NeurlPS Workshop. 2022. 3
[19] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[20] Diederik Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
10

---

[21] Lingshun Kong. Jiangxin Dong, Mingqiang Li, Jianjun Ge, and Jinshan Pan. Efficient frequency domain- based transformers for high-quality image deblurring. In CVPR, 2023.
[22] Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiff Matas. Deblurgan: Blind motion deblurring using conditional adversarial networks. In CVPR, 2018. 7,8
[23] Orest Kupyn. Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. Deblurgan-v2: Deblurring (orders-of- magnitude) faster and better. In CVPR, 2019. 7, 8,9
[24] Anat Levin, Yair Weiss, Fredo Durand, and William T Freeman. Understanding and evaluating blind deconvolution algorithms. In CVPR, 2009. 1,2
[25] Haoying Li, Yifan Yang. Meng Chang. Shiqi Chen, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen. Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing, 2022. 3
[26] Xin Li, Yulin Ren, Xin Jin, Cuiling Lan, Xingrui Wang, Wenjun Zeng, Xinchao Wang, and Zhibo Chen. Diffusion models for image restoration and enhancement a comprehensive survey, arXiv preprint arXiv:2.388.09938. 2023. 3
[27] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2017. 6
[28] Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring, In CVPR, 2017. 2, 6, 7, 8
[29] Jinshan Pan, Deqing Sun, Hanspeter Pfister, and Ming- Hsuan Yang. Deblurring images via dark channel prior. TPAMI, 2017. 1,2
[30] Dongwon Park, Dong Un Kang, Jisoo Kim, and Se Young Chun. Multi-temppral recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training. ECCV, 2020.
7, 8
[31] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, perfforrree deep learning library. In NewrIPS, 2019. 6
[32] Kuldeep Purohit, Maitreya Suin, AN Rajagopalan, and Vishnu Naresh Boddeti. Peatially-adaptiin image restoration using distortion guided networks. In ICCV, 2021. 7,
[33] Mengwei Ren, Mauricio Delbracio, Hossein Talebi, Guido Gerig, and Peyman Milanfar. Image deblurring with domain generalizable diffusion models. arXiv preprint arXiv:2212.01788 2022.
[34] Jaesung Rim, Haeyun Lee, Jucheol Won, and Sunghyun Cho. Real-world blur dataset for learning and benchmarking deblurring algorithms. In ECCV, 2020. 6, 7, 8,
[35] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusson models. In CVPR, 2022. 1, 4,
[36] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Flect, and Mohammad Norouzi. Image super-resolution via iterative refinement. TPAMI, 2022. 1, 3,5
[37] Christian J Schuler, Michael Hirsch, Stefan Harmeling, and Bernhard Schölkopf. Learning deblur. TPAMI, 2015. 1,2
[38] Qi Shan, Jiaya Jia, and Aseem Agarwala. High-quality motion deblurring from a single image. TOG, 2008
1,
[39] Ziyi Shen, Wenguan Wang, Xiankai Lu, Jianbing Shen, Haibin Ling, Tingfa Xu, and Ling Shao. Human- aware motion deblurring. ICCV, 2019, 7,8
[40] Jascha Sohl-Dickstein, Fric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In ICMI, 2015. 3,3
[41] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2020. 1, 5.
[42] Maitreya Suin. Kuldeep Purohit, and AN Rajagopalan. Spatially-attentive patch-hierarchhcal network for adaptive motion deblurring. In CVPR, 2020. 7,8
[43] Xin Tao, Hongyun Gao, Xiaoyong Shen, Jue Wang, and Jiaya Jia. Scale-recurrent network for deep image deblurring. In CVPR, 2018. 2. 8,8,9
11

---

[44] Fu-Jen Tsai, Yan-Tsung Peng. Yen-Yu Lin, Chung-Chi Tsai, and Chia-Wen Lin. Stripformer: Strip transformer for fast image deblurring. In ECCV, 2022. 1, 8,
[45] Fu-Jen Tsai, Yan-Tsung Peng, Chung-Chi Tsai, Yen-Yu Lin, and Chia-Wen Lin. Banet: A blur-aware attention network for dynamic scene deblurring. TIP, 2022. 9
[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, 2017. 3,4
[47] Yinhuai Wang. Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In ICLR. 2023. 3
[48] Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer:
A general u-shaped transformer for image restoration. In CVPR, 2022. 1,3
[49] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro P Simoncelli. Image quality assessment: from error visibility to structural similarity. TIP, 2004.
[50] Jay Whang, Mauricio Delbracio, Hossein Talehi, Chitwan Saharia, Alexandros G Dimakis, and Peyman Milanfar. Dehlurring via stochastic refinement. In CVPR, 2022. 1, 3,5
[51] Bin Xia, Yulun Zhang, Shiyin Wang, Yitong Wang, Xinglong Wu, Yapeng Tian, Wenming Yang. and Luc Van Gool. Diffir: Efficient diffusion model for image restoration. arXiv preprint 27X4:2333..0947,, 2023.
1, 4,5
[52] Li Xu, Shicheng Zheng, and Jiaya Jia. Unnatural 10 sparse representation for natural image dehlurring. In CVPR, 2013.
[53] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. CVPR, 2022. 1,3, 4, 6, 7, 8,
[54] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In CVPR, 2021. 6, 8,
[55] Hongguang Zhang, Yuchao Dai, Hongdong Li, and Piotr Koniusz. Deep stacked hicrarchical multi-patch network for image deblurring. In CVPR, 2019. 2, 7, 8, 9
[56] Kaihao Zhang, Wenhan Luo, Yiran Zhong, Lin Ma, Bjorn Stenger, Wei Liu, and Hongdong Li. Deblurring by realistic blurring. In CVPR, 2020. 6, 8
12


Reviews:
[{"review_id": "Gx3h0LWoE1", "reviewer": "Reviewer_mQcX", "summary": "The paper introduces the Hierarchical Integration Diffusion Model (HI-Diff), a novel approach for realistic image deblurring. It combines a diffusion model and a regression-based model, performing the diffusion process in a compact latent space to generate informative priors for deblurring. These priors are integrated into the regression-based model using a hierarchical module that adapts to complex blurry scenarios. The paper also presents a two-stage training strategy to optimize the latent encoder and diffusion model together. Experimental results on synthetic and real-world datasets demonstrate the effectiveness, efficiency, and superiority of the proposed method over state-of-the-art techniques in terms of PSNR, SSIM, and visual quality.", "questions": "-  Can you please elaborate on the structure and parameters of the diffusion model in more detail?\n- It would be beneficial to include some visual results showcasing the effects of the diffusion process, if available.\n- In the event of replacing the diffusion model with alternative restoration models like transformer-based approaches, would the proposed method still maintain its effectiveness?", "limitations": "Based on the available information on the web page, the paper does not explicitly address the limitations of their proposed method.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper presents a novel and effective approach for leveraging the diffusion model in image deblurring, yielding realistic details and mitigating unwanted artifacts.\n- By executing the diffusion model within a compact latent space, the paper successfully reduces computational complexity, enabling faster inference with fewer iterations.\n- Through comprehensive experiments on synthetic and real-world datasets, the paper demonstrates the superior performance of the proposed method compared to state-of-the-art approaches.", "weaknesses": "- The diffusion models have an advantage in generating high-quality image details. However, in this paper the diffusion model reconstruct feature in the compacted latent space. The paper does not show and explain the superiority of diffusion models over other models in this task.\n- The paper does not conduct a comprehensive comparison with other diffusion models for image deblurring. These methods also apply diffusion models in different ways to address the challenges of image deblurring, such as computational efficiency, distortion accuracy, and generalization ability. It would be interesting to see how HI-Diff compares with these methods in terms of performance and complexity.\n- The paper does not analyze the effectiveness of the latent compression and evaluate the hyperparameters. It would be helpful to provide some ablation studies or analysis on the latent compression encoder.\n- Lack of one-stage ablation study or a clearer explanation is needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer mQcX (denoted as R1)\n\n`Q1-1:` The paper does not show and explain the superiority of diffusion models over other models in this task.\n\n`A1-1:` Thanks for pointing it out. We explain and conduct experiments to show the superiority of diffusion models (DMs).\n\n1. **Explanation:** Compared with other methods, DMs have a stronger distribution (image / latent) modeling ability. Thus, DM can generate high-quality image details. Meanwhile, not limited to image distributions, it also applies to latent spaces. Therefore, we adopt DM to generate the prior in latent space.\n\n2. **Experiments:** We replace DM with a Transformer-based model in HI-Diff to generate priors. We use the same training settings as HI-Diff for the new model. Models are tested on GoPro, and the input size is 3×256×256 to calculate FLOPs. Applying DM outperforms using Transformer.\n\n   | Method      | Params (M) | FLOPs (G) | PSNR (dB) |  SSIM  |\n   | ----------- | :--------: | :-------: | :-------: | :----: |\n   | Transformer |   24.84    |  125.48   |   32.09   | 0.9545 |\n   | DM          |   23.99    |  125.47   |   32.24   | 0.9558 |\n\n\n\n\n`Q1-2:` It would be interesting to see how HI-Diff compares with other diffusion models for image deblurring.\n\n`A1-2:` Thanks for your suggestions. We compare HI-Diff with other diffusion models: **DvSR [48]** and **DiffIR [49]**. Models are tested on GoPro, and the input size is 3×256×256 to calculate FLOPs.\n\n| Method           | Params |    FLOPs    | PSNR  | SSIM  |\n| ---------------- | :----: | :---------: | :---: | :---: |\n| DvSR (CVPR'22)   | 26.07M | 170.31**T** | 31.66 | 0.948 |\n| DiffIR (ICCV'23) | 26.94M |   120.99G   | 33.20 | 0.963 |\n| HI-Diff (ours)   | 28.49M |   142.62G   | 33.33 | 0.964 |\n| HI-Diff-2 (ours) | 23.99M |   125.47G   | 33.28 | 0.964 |\n\n1. Compared with the DvSR, our methods have much smaller FLOPs because DM is performed in latent space. Meanwhile, our methods achieve better performance.\n2. Compared with DiffIR, our methods perform better with comparable Params and FLOPs. This is because HI-Diff applies hierarchical integration, which is more suitable for processing non-uniform blur.\n\n**We provide more comprehensive comparisons in Tab. 1 of the PDF.**\n\n\n\n`Q1-3:` The paper does not analyze the effectiveness of the latent compression and evaluate the hyperparameters. It would be helpful to provide some ablation studies or analysis on the latent compression encoder.\n\n`A1-3:` Thanks for the suggestion. We ablation on the token number $N$ of latent space to analyze the effectiveness of latent compression. We set $N$ as 4, 16, and 64.\n\nAll experiment settings are consistent with the ablation study (**Sec. 4.2**). Models are tested on GoPro, and the input size is 3×256×256 to calculate FLOPs.\n\n| $N$  | Params (M) | FLOPs (G) | PSNR (dB) |  SSIM  |\n| :--: | :--------: | :-------: | :-------: | :----: |\n|  4   |   23.985   |  125.05   |   32.12   | 0.9542 |\n|  16  |   23.986   |  125.47   |   32.24   | 0.9558 |\n|  64  |   23.990   |  127.33   |   32.28   | 0.9560 |\n\nWith the increase of $N$, the performance increases, but the gain magnitude decreases. It may be because as $N$ increases, tokens become more redundant, thus the performance increase is limited. And the increase of redundant tokens also increases complexity and resource consumption. To balance performance, redundancy, and consumption, we choose $N$=16.\n\n\n\n`Q1-4:` Lack of one-stage ablation study or a clearer explanation is needed.\n\n`A1-4:` Thanks for pointing it out. We explain it below.\n\n1. In our two-stage training architecture, the first stage uses the **ground truth image** as input to participate in model training. However, the second stage only adopts the **blurry image** as input. Therefore, we cannot combine the two stages into one stage. And we do not conduct the one-stage ablation study.\n2. As we mentioned in **Sec 3.2** (Limitations) of the **supplementary material**, two-stage training is more tedious than one-stage training, which is a shortcoming of our method. One of our follow-up research directions is to explore how to conduct efficient one-stage training.\n\n\n\n`Q1-5:` Can you please elaborate on the structure and parameters of the diffusion model in more detail?\n\n`A1-5:` Thanks for asking for those details. We clarify them as follows.\n\n**For the structure**: The main component of the diffusion model is the Denoising Network (DN), which consists of some Linear Layers and MLP-Mixer Layers. The structure of DN is as follows.\n\n```shell\ninput -> (Linear+LRelU) -> (MLP-Mixer)×4 -> (Linear+LRelU) -> output\n```\n\nMeanwhile, we apply the same diffusion and reverse process as DDPM (scheduler) [14].\n\n**For parameters**: We provide the Params and FLOPs of the diffusion model. We separately calculate FLOPs (input size: 16×256) for one diffusion step and total steps.\n\n| Params | FLOPs (one step) | FLOPs (total steps) | Step |\n| :----: | :--------------: | :-----------------: | :--: |\n| 2.63M  |      42.37M      |       338.95M       |  8   |\n\n\n\n`Q1-6:` It would be beneficial to include some visual results showcasing the effects of the diffusion process, if available.\n\n`A1-6:` Thanks for your valuable suggestion. We provide some visual results in **Fig. 1 of the PDF**.\n\nThe blurred image gradually becomes sharp as the diffusion (reverse) process proceeds.\n\nMeanwhile, when the prior is Gaussian noise (i.e., $\\mathbf{z}_{8}$), the output of Transformer is **not noise**. This may be because Transformer features actively ignore the invalid priors through cross-attention when fused with priors.\n\n\n\n`Q1-7:` In the event of replacing the diffusion model with alternative restoration models like transformer-based approaches, would the proposed method still maintain its effectiveness?\n\n`A1-7:` Thanks for your suggestions. We find that replacing DM with Transformer degrades model performance.\n\n**We have responded to another similar question, `Q1-1`. Please refer to `A1-1` for more details.**\n\n"}, {"review_id": "agu70DNgzA", "reviewer": "Reviewer_ymRm", "summary": "The authors propose a new image deblurring model, Hierarchical Integration Diffusion Model (HI-Diff). The HI-Diff uses the diffusion models to produce priors in a highly compacted latent space, and is integrated into the deblurring process hierarchically with the proposed hierarchical integration module (HIM). The effectiveness of each component is demonstrated by the ablation study. The main results on synthetic and real-world deblurring show that HI-Diff outperforms recent state-of-the-art methods.", "questions": "1. Please clarify the differences between the HI-Diff and DiffIR [49].\n\n2. Compare HI-Diff with more generative models (like GAN and Diffusion), and evaluate on perceptual metrics.\n\n3. Provide the latency comparisons to further show the effectiveness of the method.\n\n4. The authors conduct train models on synthetic and real-world datasets. Are the settings of two model the same? if different, clarify the details.", "limitations": "The limitations and potential negative societal impact of the work have been discussed in the supplementary material.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The proposed HI-Diff is simple and effective. The idea that leveraging the power of the diffusion model and integrating it into the deblurring process hierarchically is reasonable and novel for image deblurring.\n\n2. The design of the HI-Diff is stated clearly and logically. The authors expound the motivation and realization of the design in detail. And the effectiveness of each part is proved by the ablation study.\n\n3. The main comparisons with recent state-of-the-art methods are extensive. The evaluation on both synthetic and real-world datasets demonstrates the superiority of the proposed methods.\n\n4. In the supplementary material, the authors provide more variant models and more quantitative and qualitative comparisons, further revealing the promising performance of HI-DIff.\n\n5. The paper is well-organized, and the writing is good and easy to read. \n\n6. The authors also provide the code and pre-trained models for results reproduction. This reveals the solidness of the work and helps other researchers to follow.", "weaknesses": "1. The feature prior is the key of this work. However, the paper lacks a specific analysis of the prior. For example, the difference and impact of prior generated on different inputs.\n\n2. As mentioned in the paper, the proposed method generates more realistic deblurred images. But the comparisons in Tabs. 2 and 3 are about distortion-based metrics (e.g., PSNR). More comparisons on perceptual metrics should be provided.\n\n3. Although the authors provide the FLOPs and Params comparisons in Tab. 4, the latency (running time), another important indicator in low-level tasks, needs to be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer ymRm (denoted as R2)\n\n`Q2-1:` The feature prior is the key of this work. However, the paper lacks a specific analysis of the prior.\n\n`A2-1:` Thanks for pointing it out. We provide an analysis of the prior.\n\n1. We compare the similarity of priors generated on different inputs. We find that the similarity between priors (e.g., MSE) positively correlates with the similarity between input images (e.g., SSIM).\n2. We further provide some visual results in **Fig. 1 of the PDF**, which intuitively show the impact of priors.\n\n\n\n`Q2-2:` More comparisons on perceptual metrics should be provided.\n\n`A2-2:` Thanks for your suggestion. We have provided a comparison of perceptual metrics in **Tab. 2** of the **supplementary material**. We also show part of them here (with more compared methods). Models are tested on GoPro.\n\n| Method              | Model | LPIPS $\\downarrow$ | DISTS $\\downarrow$ | NIQE $\\downarrow$ | PSNR $\\uparrow$ | SSIM $\\uparrow$ |\n| ------------------- | :---: | :----------------: | :----------------: | :---------------: | :-------------: | :-------------: |\n| DBGAN (CVPR'20)     |  GAN  |       0.110        |       0.078        |       4.06        |      31.10      |      0.942      |\n| DvSR (CVPR'22)      |  DM   |       0.059        |        N/A         |       3.39        |      31.66      |      0.948      |\n| DiffIR (ICCV'23)    |  DM   |       0.081        |       0.071        |       4.13        |      33.20      |      0.963      |\n| HI-Diff (ours)      |  DM   |       0.080        |       0.071        |       4.12        |      33.33      |      0.964      |\n| HI-Diff-PE-1 (ours) |  DM   |       0.051        |       0.031        |       3.53        |      33.27      |      0.963      |\n| HI-Diff-PE-2 (ours) |  DM   |       0.044        |       0.029        |       3.30        |      32.84      |      0.959      |\n\nOur method achieves the best performance on both distortion-based and perceptual metrics.\n\n**We provide more comprehensive comparisons in Tab. 1 of the PDF.**\n\n\n\n`Q2-3:` Although the authors provide the FLOPs and Params comparisons in Tab. 4, the latency (running time), another important indicator in low-level tasks, needs to be provided.\n\n`A2-3:` Thanks for pointing it out. We provide the latency (i.e., running time) comparison. The running time is tested on one 3090 GPU with the input size of 3×256×256. We calculate the average time over 100 images. Our method achieves comparable running time with other methods.\n\n| Method       | MPRNet | Restormer | HI-Diff (ours) | HI-Diff-2 (ours) |\n| ------------ | :----: | :-------: | :------------: | :--------------: |\n| Latency (ms) | 77.46  |   82.05   |     75.89      |      65.98       |\n\n\n\n`Q2-4:` Please clarify the differences between the HI-Diff and DiffIR [49].\n\n`A2-4:` Thanks for asking for those details. We clarify them as follows.\n\n*Note: All models are tested on GoPro, and the input size is 3×256×256 to calculate FLOPs.*\n\n1. In general, the difference between HI-Diff and DiffIR is the **hierarchical integration**: **multi-scale prior** and **cross-attention interaction**, which is more suitable for non-uniform deblurring.\n\n2. **The prior is different.** HI-Diff applies the multi-scale prior, while DiffIR uses the single-scale prior. The multi-scale prior adapts to different scale features in encoder-decoder Transformer architecture for better integration.\n\n   | Method       | Params (M) | FLOPs (G) | PSNR (dB) |  SSIM  |\n   | ------------ | :--------: | :-------: | :-------: | :----: |\n   | Single-scale |   21.98    |  125.39   |   32.00   | 0.9534 |\n   | Multi-scale  |   23.99    |  125.47   |   32.24   | 0.9558 |\n\n3. **The interaction is different.** HI-Diff adopts cross-attention, while DiffIR uses prior as dynamic modulation parameters. Features pay different attention to the prior with cross-attention, which is more suitable for non-uniform deblurring. Experiments demonstrate that applying cross-attention outperforms using dynamic modulation.\n\n   | Method             | Params (M) | FLOPs (G) | PSNR (dB) |  SSIM  |\n   | ------------------ | :--------: | :-------: | :-------: | :----: |\n   | dynamic modulation |   29.63    |  121.01   |   32.10   | 0.9544 |\n   | cross-attention    |   23.99    |  125.47   |   32.24   | 0.9558 |\n\n4. **Experimental comparison.** Our method performs better than DiffIR with comparable Params and FLOPs. It shows that our method, applying hierarchical integration, is more suitable for realistic non-uniform deblurring.\n\n   | Method           | Params (M) | FLOPs (G) | PSNR (dB) | SSIM  |\n   | ---------------- | :--------: | :-------: | :-------: | :---: |\n   | DiffIR           |   26.94    |  120.99   |   33.20   | 0.963 |\n   | HI-Diff (ours)   |   28.49    |  142.62   |   33.33   | 0.964 |\n   | HI-Diff-2 (ours) |   23.99    |  125.47   |   33.28   | 0.964 |\n\n\n\n`Q2-5:` Compare HI-Diff with more generative models (like GAN and Diffusion), and evaluate on perceptual metrics.\n\n`A2-5:` Thanks for your suggestion. We compare HI-Diff with GAN: **DBGAN [54]**, Diffusion: **DvSR [48]** and **DiffIR [49]** on distortion-based and perceptual metrics. Our HI-Diff achieves the best results. \n\n**We have responded to another similar question, `Q2-2`. Please refer to `A2-2` for more details.**\n\n\n\n`Q2-6:` Provide the latency comparisons to further show the effectiveness of the method.\n\n`A2-6:` Thanks for your suggestion. \n\nWe further compare the latency of HI-Diff with other methods. Our HI-Diff achieves a comparable running time with other methods. **We have responded to another similar question, `Q2-3`. Please refer to `A2-3` for more details.**\n\n\n\n`Q2-7:` The authors conduct train models on synthetic and real-world datasets. Are the settings of two model the same? if different, clarify the details.\n\n`A2-7:` Thanks for your question. The settings are the same for the two models on synthetic and real-world datasets.\n\n"}, {"review_id": "auexfnFZHC", "reviewer": "Reviewer_5Ars", "summary": "This paper presents the Hierarchical Integration Diffusion Model (HI-Diff) for realistic image deblurring. The HI-Diff utilizes diffusion models to generate multiscale priors in the latent space, which are integrated hierarchically into the deblurring process to improve the results.   Experiments are conducted on both synthetic and real-world blur datasets.", "questions": "See the Weaknesses.", "limitations": "See the Weaknesses.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A hierarchical integration module is proposed to fuse the prior into the model from multiple scales\n- Experiments conducted on synthetic and real-world blur datasets demonstrate the state-of-the-art results, and the code would be released.", "weaknesses": "- It's unclear why authors adopt Diffusion Models (DMs) to model the prior. For one blurry image, the corresponding blurry prior should be deterministic instead of a distribution generated by DMs.\n- The loss $L_{difussion}$ for the DMs is also weird (L202). If the DMs learn to produce the z, why not integrate z directly into the model?\n- Tab. 1 and Fig.3 indicate that the DMs do not really matter. In Tab.1, compared to the baseline, it only improves by 0.04 dB. The gain of this method mainly comes from the multi-scale representation of the prior (> 0.2dB). Fig. 3, to some degree, also supports this. More iteration numbers (>5) in the diffusion model cannot improve the results.\n- L222 L 243 seems to indicate the authors use the original Restormer [1]. But, with the additional DMs, the flops of this method are less than Restormer, as shown in Tab. 3. What is the difference compared to the original Restormer? Why not use the original setting?\n\n[1] Restormer: Efficient transformer for high-resolution image restoration", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer 5Ars (denoted as R3)\n\n`Q3-1:` It's unclear why authors adopt Diffusion Models (DMs) to model the prior. For one blurry image, the corresponding blurry prior should be deterministic instead of a distribution generated by DMs.\n\n`A3-1:` Thanks for your question. We explain it below.\n\n1. Compared with other methods, DMs have a stronger modeling ability to generate target priors. Therefore we apply DM to model the distribution of prior. Meanwhile, DM generates the prior **conditioned on** the blurry image, **not entirely randomly**.\n\n2. Furthermore, the prior is not deterministic, since a blurry image may correspond to multiple sharp images (**ill-posed**). Therefore, some level of randomness in DM sampling is suitable.\n\n3. We replace DM with Transformer in HI-Diff to generate priors. Models are tested on GoPro, and the input size is 3×256×256 to calculate FLOPs. Applying DM outperforms using Transformer.\n\n   | Method      | Params (M) | FLOPs (G) | PSNR (dB) |  SSIM  |\n   | ----------- | :--------: | :-------: | :-------: | :----: |\n   | Transformer |   24.84    |  125.48   |   32.09   | 0.9545 |\n   | DM          |   23.99    |  125.47   |   32.24   | 0.9558 |\n\n\n\n`Q3-2:` The loss $L_{diffusion}$ for the DMs is also weird (L202). If the DMs learn to produce the z, why not integrate z directly into the model?\n\n`A3-2:` Thanks for your question. We clarify them as follows.\n\n**For $L_{diffusion}$.**\n\n1. The loss $L_{diffusion}$=$\\Vert \\hat{\\mathbf{z}} - \\mathbf{z} \\Vert_1$, where $\\mathbf{z}$ is prior from ground truth image and $\\hat{\\mathbf{z}}$ is the predicted prior after complete $T$ step reverse processes of DM. The loss is to allow DM to **learn to generate prior** directly.\n2. For general DM (e.g., DDPM), it optimizes **one denoising step** at each training step. Therefore, it cannot generate the prior directly (needs total denoising steps). Thus, its training objective (Eq. (7) in our paper) is the noise ($\\epsilon$). However, the final output of general DM is still the prior.\n3. Different from general DM, we execute the **complete process** of DM at each training step, and let the DM directly learn to generate z. This is consistent with the final goal of general DM. Therefore, **the loss $L_{diffusion}$ is reasonable**.\n\n**For \"why not integrate z directly into the model\".**\n\n1. It is feasible to integrate z directly into the (Transformer) model. However, this will cause Transformer to be performed at each denoising step. The overall **complexity is too high**, since Transformer is performed $T$ times.\n\n2. Instead, we execute DM separately on the latent space, and only use the final result of DM for Transformer. Therefore, **the complexity is effectively reduced**, since Transformer is conducted once.\n\n3. Furthermore, compared with methods (e.g., DvSR [48]) that execute the complete model at each step, our method (generate z separately) achieves better performance with much smaller FLOPs (evaluated on GoPro, the input size: 3×256×256 to calculate FLOPs).\n\n   | Method         | Params |    FLOPs    | PSNR  | SSIM  |\n   | -------------- | :----: | :---------: | :---: | :---: |\n   | DvSR (CVPR'22) | 26.07M | 170.31**T** | 31.66 | 0.948 |\n   | HI-Diff (ours) | 28.49M |   142.62G   | 33.33 | 0.964 |\n\n\n\n`Q3-3:` Tab. 1 and Fig.3 indicate that the DMs do not really matter. In Tab.1, compared to the baseline, it only improves by 0.04 dB. The gain of this method mainly comes from the multi-scale representation of the prior (> 0.2dB). Fig. 3, to some degree, also supports this. More iteration numbers (>5) in the diffusion model cannot improve the results.\n\n`A3-3:` Thanks for your question. We explain it as follows.\n\n**For \"the DMs do not really matter\".** We replace DM with Transformer in HI-Diff to generate priors, and the model performance decreases. It demonstrates that the DM is important.\n\n| Method      | Params (M) | FLOPs (G) | PSNR (dB) |  SSIM  |\n| ----------- | :--------: | :-------: | :-------: | :----: |\n| Transformer |   24.84    |  125.48   |   32.09   | 0.9545 |\n| DM          |   23.99    |  125.47   |   32.24   | 0.9558 |\n\n**For \"the gain mainly comes from the multi-scale representation\".**\n\n1. Without multi-scale representation, the prior cannot be effectively fused in Transformer features, restricting performance. Thus, the gain is little.\n2. The result of (Multi-Scale + Transformer) is lower than that of applying DM. This indicates that both DM and Multi-Scale are important. **Both two components should be considered together.**\n\n**For \"more iteration numbers cannot improve the results\".**\n\n1. Since the prior is in the latent space, DM does not need too many iterations to model it. However, it doesn't mean DM isn't important.\n2. Compared with 1000+ iterations in general DM (DDPM), fewer iterations (i.e., 8) further indicate the effectiveness of our proposed method.\n\n\n\n`Q3-4:` The authors use the original Restormer [1]. With the additional DMs, the flops of this method are less than Restormer (Tab. 3). What is the difference compared to the original Restormer? Why not use the original setting?\n\n`A3-4:` Thanks for your question. We explain it below.\n\n**For \"the difference compared to Restormer\".**\n\n1. Our HI-Diff (model in Tab. 3 (4)) applies the original structure of Restormer with **fewer** block numbers in each stage. **Other settings remain the same.**\n\n2. We provide a comparison between HI-Diff and Restormer. Models are tested on GoPro, and the input size is 3×256×256 to calculate FLOPs.\n\n   | Method    | Block in each stage | Params (M) | FLOPs (G) | PSNR (dB) |\n   | --------- | :-----------------: | :--------: | :-------: | :-------: |\n   | Restormer |    [4, 6, 6, 8]     |   26.13    |  154.88   |   32.92   |\n   | HI-Diff   |    [3, 5, 5, 6]     |   28.49    |  142.62   |   33.33   |\n\n**For \"why not the original setting\".** We apply fewer blocks to make the overall Params and FLOPs of HI-Diff close to Restormer to realize **a fair comparison**."}, {"review_id": "2LF44D5B07", "reviewer": "Reviewer_w6Fg", "summary": "This paper proposed a novelty Hierarchical Integration Diffusion Model for deblurring task. By incorporation multi-scale latent priors, the proposed method achieved SOTA performance on both synthetic and real-world datasets.", "questions": "1. What are the difference between the proposed method and DiffIR? I think only using multi-scale latent space z seems not very novelty enough.\n2. I suggest authors add more DM-based compare method, e.g., retrained DiffIR and DvSR.\n  ", "limitations": "As state in the introduction part, the diffusion model may gengerate unpleasing artifacts in the restoration results. Does this method also have some fail cases?", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method achieved SOTA performance.\n2. The ablation study is very complete and the paper is easy to understand.", "weaknesses": "1. I think this paper is meaningful. But I argue that the authors should add more discussion about the difference between DiffIR. In my understanding, the difference is mainly using multi-scale latent prior, which has some novelty but seems small. Also in the introduction section, the authors state the motivation as \"since the advantages of regression-based methods in distortion accuracy, we integrate DMs and regression-based methods\", which seems similar with DiffIR. \n2. Experiment comparison. This paper seems highly inspried by DiffIR. I suggest the authors also compare the results with the re-trained DiffIR model. The compare methods do not contain any diffusion models. In section 2.2, DvSR seems handle deblurring task using DM. I think the authors should add more comparisions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer w6Fg (denoted as R4)\n\n`Q4-1:` The authors should add more discussion about the difference between DiffIR. The difference is mainly using multi-scale latent prior. Also, the motivation: \"since the advantages of regression-based methods in distortion accuracy, we integrate DMs and regression-based methods\", seems similar with DiffIR.\n\n`A4-1:` Thanks for your valuable suggestions. We clarify the differences between DiffIR, and the novelty of our work below.\n\n*Note: All models are tested on GoPro, and the input size is 3×256×256 to calculate FLOPs.*\n\n**For \"the difference is mainly using multi-scale latent prior\".** The multi-scale latent prior is part of our novelty.\n\n1. In general, the difference between HI-Diff and DiffIR is the integration approach between DM and Transformer. We design the **hierarchical integration**: **multi-scale prior** and **cross-attention interaction**, which is more suitable for non-uniform deblurring in real scenarios.\n\n2. **The prior is different.** HI-Diff applies the multi-scale prior, while DiffIR uses the single-scale prior. The multi-scale prior adapts to encoder-decoder Transformer architecture for better integration. The ablation in Tab. 1 demonstrates this point.\n\n   | Method       | Params (M) | FLOPs (G) | PSNR (dB) |  SSIM  |\n   | ------------ | :--------: | :-------: | :-------: | :----: |\n   | Single-scale |   21.98    |  125.39   |   32.00   | 0.9534 |\n   | Multi-scale  |   23.99    |  125.47   |   32.24   | 0.9558 |\n\n3. **The interaction is different.** HI-Diff adopts cross-attention in the hierarchical integration module (HIM), while DiffIR uses prior as dynamic modulation parameters. With cross-attention, the regions with varying degrees of blur in features can pay different attention to the prior, resulting in better (non-uniform) deblurring performance.\n\n   We replace cross-attention in HI-Diff with dynamic modulation and find that cross-attention yields better performance.\n\n   | Method             | Params (M) | FLOPs (G) | PSNR (dB) |  SSIM  |\n   | ------------------ | :--------: | :-------: | :-------: | :----: |\n   | dynamic modulation |   29.63    |  121.01   |   32.10   | 0.9544 |\n   | cross-attention    |   23.99    |  125.47   |   32.24   | 0.9558 |\n\n   Moreover, the **HIM** we proposed is **plug-and-play**, which is more convenient without modifying the components of the original model.\n\n\n\n**For \"the motivation is similar with DiffIR\".** The integration of DM and Transformer in our method is inspired by DiffIR. But our motivation is different from DiffIR.\n\n1. **The motivation is different.** As described in the introduction, our motivation is threefold: reduce DM complexity, take advantage of Transformer, and better address the non-uniform blur. However, DiffIR focuses on improving the efﬁciency and stability of DM without **considering properties of realistic deblurring**.\n\n2. **HI-Diff outperforms DiffIR.** We compare our HI-DIff with DiffIR. Our method performs better than DiffIR with comparable Params and FLOPs. It shows that our method, applying hierarchical integration, is more suitable for realistic non-uniform deblurring.\n\n   | Method           | Params (M) | FLOPs (G) | PSNR (dB) | SSIM  |\n   | ---------------- | :--------: | :-------: | :-------: | :---: |\n   | DiffIR (paper)   |   26.94    |  120.99   |   33.20   | 0.963 |\n   | HI-Diff (ours)   |   28.49    |  142.62   |   33.33   | 0.964 |\n   | HI-Diff-2 (ours) |   23.99    |  125.47   |   33.28   | 0.964 |\n\n\n\n`Q4-2:` Compare the results with the re-trained DiffIR model and DvSR.\n\n`A4-2:` Thanks for your suggestion. We provide more comparisons with diffusion models: **DvSR [48]** and **DiffIR [49]**.\n\nFor DiffIR, apart from the results provided in the paper, we also **retrain the model** with the official code. Models are tested on GoPro, and the input size is 3×256×256 to calculate FLOPs.\n\n| Method             | Params |    FLOPs    | PSNR  | SSIM  |\n| ------------------ | :----: | :---------: | :---: | :---: |\n| DvSR               | 26.07M | 170.31**T** | 31.66 | 0.948 |\n| DiffIR (paper)     | 26.94M |   120.99G   | 33.20 | 0.963 |\n| DiffIR (retrained) | 26.94M |   120.99G   | 33.18 | 0.963 |\n| HI-Diff (ours)     | 28.49M |   142.62G   | 33.33 | 0.964 |\n| HI-Diff-2 (ours)   | 23.99M |   125.47G   | 33.28 | 0.964 |\n\n1. Compared with the DvSR, our methods have much smaller FLOPs and better performance.\n2. Compared with DiffIR, our methods perform better with comparable Params and FLOPs. This is because HI-Diff applies hierarchical integration, which is more suitable for processing non-uniform blur in real scenarios.\n\n**We provide more comprehensive comparisons in Tab. 1 of the PDF.**\n\n\n\n`Q4-3:` What are the difference between the proposed method and DiffIR? I think only using multi-scale latent space z seems not very novelty enough.\n\n`A4-3:` Thanks for asking for those details. We clarify them as follows.\n\n1. The difference between our HI-Diff and DiffIR is the **hierarchical integration**: **multi-scale prior** and **cross-attention interaction**, which is more effective and suitable for non-uniform deblurring. \n2. The multi-scale latent prior is part of our novelty.\n\n**We have responded to another similar question, `Q4-1`. Please refer to `A4-1` for more details.**\n\n\n\n`Q4-4:` I suggest authors add more DM-based compare method, e.g., retrained DiffIR and DvSR.\n\n`A4-4:` Thanks for the valuable suggestions. We compare more DM-based methods, e.g., DvSR [48] and DiffIR [49]. \n\nOur HI-Diff outperforms other DM-based methods. **We have responded to another similar question, `Q4-2`. Please refer to `A4-2` for more details.**\n\n\n\n`Q4-5:` As state in the introduction part, the diffusion model may gengerate unpleasing artifacts in the restoration results. Does this method also have some fail cases?\n\n`A4-5:` Thanks for asking for those details. In our method, there are also some failure cases. \n\nWe provide some cases in **Fig. 2 of the PDF**. \n\n"}]

Author rebuttal:
## Response to all reviewers and area chairs for a brief summary 

Dear reviewers and area chairs,

We thank all reviewers and area chairs for their valuable time and comments.

We are encouraged that:

1. Reviewer mQcX and Reviewer ymRm agree that our method is novel.
2. Reviewer ymRm thinks our experiments are extensive, and Reviewer w6Fg thinks the ablation study is complete.
3. All reviewers recognize that our method achieves state-of-the-art performance.



We have responded to each reviewer individually to address any comments. We would like to give a brief summary.

1. We explain the **reason for applying the diffusion model** (DM), and conduct additional experiments (comparing it with Transformer) to demonstrate **its superiority and importance**. Meanwhile, we also analyze the rationality of the loss for DM.
2. We provide comprehensive **comparisons with other DMs** for image deblurring in terms of performance and complexity.
3. We provide analyses and experiments to clarify the **differences** between our method and DiffIR, and show the **novelty** of our work.
4. We clarify the **details of our method**, including the structure and parameters of DM, differences from Restormer, and the settings of models on different datasets.
5. We analyze and experiment on **latent compression**. Meanwhile, we analyze our shortcomings and follow-up directions regarding **one-stage** training.
6. We provide the **latency comparisons** and analyze more on priors.
7. Finally, we provide more **visualizations** to enhance the solidity of the work, including the diffusion process and some failure cases.

We thank all reviewers and area chairs again!

Best,

Authors



Meta-review: This is a nice paper and the reviewers, after having seen considered the rebuttal, seem to largely agree that it is a strong enough contribution to be included in the conference. I congratulate the authors. 

I have a couple of suggestions for the References. The inclusion of the two works listed below will serve only a pedagogic purpose - neither work contains anything that takes away from the novelty of your contribution. Please include a brief mention of them in the related works section.

https://arxiv.org/abs/2308.09388
https://openreview.net/forum?id=VmyFF5lL3F
Decision: Accept (spotlight)