Gaussian Kernel Mixture Network for Single Image Defocus Deblurring
Authors: Yuhui Quan
Zicong Wu
Hui Ji
Venue: NeurIPS 2021 Poster (2021)
Keywords: Defocus Deblurring
Deep Learning
Image Recovery
Unrolling
Attention

Abstract: Defocus blur is one kind of blur effects often seen in images, which is challenging to remove due to its spatially variant amount. This paper presents  an end-to-end deep learning approach for removing defocus blur from a single image, so as to have an all-in-focus image for consequent vision tasks. First, a  pixel-wise Gaussian kernel mixture (GKM) model is  proposed for representing spatially variant defocus blur kernels in an efficient linear parametric form, with higher accuracy than existing models. Then, a deep neural network called GKMNet is developed by unrolling a fixed-point iteration of the GKM-based deblurring.  The GKMNet is built on a lightweight scale-recurrent architecture, with a scale-recurrent attention module for estimating the mixing coefficients in GKM for defocus deblurring. Extensive experiments show that the GKMNet not only noticeably outperforms existing defocus deblurring methods, but also has its advantages in terms of model complexity and computational efficiency.

Gaussian Kernel Mixture Network for Single Image Defocus Deblurring
Yuhui Quan', Zicong Wu" School of Computer Science and Engineering South China University of Technology
Hui Ji Department of Mathematics National University of Singapore
Abstract
Defocus blur is one kind of blur effects often seen in images, which is challenging to remove due to its spatially variant amount. This paper presents an end-to-end deep learning approach for removing defocus blur from single image, as have an all-in-focus image for consequent vision tasks. First, a pixel-wise Gaussian kernel mixture (GKM) model is proposed for representing spatially variant defocus blur kernels an efficient linear parametric form, with higher accuracy than existing models. Then, deep neural network called GKMNet is developed by unrolling fixed-point iteration of the GKM-based deblurring. The GKMNet is built lightweight scale-recurrent architecture, with scale-recurrent attention module for estimating the mixing coefficients Extensive experiments show that the GKMNe not only noticeably outperforms existing defocus deblurring methods, but also has its advantages in terms of model complexity and computational efficiency.
GKM for defocus deblurring.
Introduction
The appearance sharpness of an object in an image taken by camera is determined by the scene distance of the object to the focal plane the camera. object will have the sharpest appearance when it is on the focal plane, i.e., the object is in focus. The area around the focal plane where objects appear to be in focus is called the depth field (DoF). When an object is away from the DoF, will appear blurry. The further is object away from the DoF, the more blurry it appears. Such phenomenon is called defocus blur or out-of-focus blur. Defocus blur effects will be prominent in image with shallow DoF, e.g. images captured with large aperture. This paper concerns the problem of single image defocus deblurring (SIDD) which is about reconstructing an all-in-focus image from defocused image (i.e. image with defocused regions). SIDD is of practical values to many applications in machine vision, e.g. photo refocusing, object recognition, and many others I
Consider defocused image y, which relates to its all-in-focus counterpart by
y B +
where denotes the measurement noise, and B is linear operator defined by
B [Box)Jm,n]:
ΣΣ b,,n j i, 5].
Each pixel at location (m, r2] is associated with a defocus kernel dm.ns also referred point spread function (PSF), which determined by the distance to the focal plane. Often, these pixel-wise PSFs approximated by Gaussian kernels 2 information on the scene depth, these pixel-wise wise PSFs are unknown. Therefore, SIDD is a challenging nonlinear inverse problem which needs to estimate both B and from (1)
5 6 disk kernels 8). Without supplementary
"mmail: Sechhuua@@scut.ddli (Y. Quan): eszicongwu mai..scut.ed..coo (Z. Wu); matj Siiihh@uu. nus.edu..g (H. Ji).
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

1.1 Discussion on Existing Work
5 take a two-stage approach which (i) estimates a dense defocus map to derive the operator B and then (ii) recovers the image z by using nonblind image deconvolution solve (I) with the estimated B. Generally, such two-stage approach has a long pipeline with many modules, and the estimation error in one module will be magnified in the consequent modules. For instance, the defocus amount only can be estimated on subset of image pixels such as edge points. Then a dense defocus map needs to be constructed by propagating these few estimations to all pixels. can be seen that any error in the sparse defocus map will result in erroneous PSFs, and unfortunately deconvolution is very sensitive to the errors in PSFs [TO] [T] As result, the two-stage approach does not perform well in practice. Also, the computational cost the second stage is high for a non-uniform blurring operator B, as the inversion process regrading B, which is often called many times in nonblind image deconvolution, cannot be efficiently computed via fast Fourier transform (FFT).
Most existing methods (e.g. 2
Deep learning has become one prominent tool for solving a wide range of image restoration problems. In comparison to the rapid progress of DNNs for spatially-varying motion deblurring (e.g- [12 13 14
15 16 17), there have been few works on studying DNN-based approaches to defocus deblurring.
One might directly adapt an existing motion deblurring DNN for SIDD. However, the kernels of defocus blur are very different from those of motion blur, egg. roughly isotropic support vs highly curvy support. Also, the spatial variation in defocus blur differs much from that in motion blur, e.g. transparency effects of moving objects in motion blur do not exist for out-of-focus objects in defocus blur. As result, it is sub-optimal to directly call motion deblurring method for SIDD.
Another straightforward implementation of introducing deep learning to SIDD is to replace the defocus map estimator in traditional two-stage approach by DNN-based method (e.g. [6].. Such an implementation still suffers from the issues existing in traditional methods, Lee. inaccurate estimation of B from a non-perfect defocus map and high computational cost for deblurring with a spatially- variant blurring operator. To fully exploit the potential of deep learning for SIDD, one needs to specifically design an end-to-end DNN that directly predicts the all-in-foctt image from the defocused one. Recently Abuolaim and Brown [1]] developed an end-to-end DNN for constructing an all-in- focus image from pair of images containing two sub-aperture views of the same scene. They also adapted their DNN to SIDD, but saw a significant performance decrease.
1.2 Main Idea
This paper aims at developing an end-to-end DNN for SIDD with better performance than existing methods, which is based on the following two derivations.
GKM-based model for defocus blurring Since defocus PSFs show strong isotropy and smooth- ness, we propose to model the kernels (bm,,)m,, by Gaussian kernel mixture (GKM):
bm,
Balm, nlg((k).
k=1
where g(r) denotes the 2D Gaussian kernel of variance o², and Bg denotes the matrix of mixing coefficients for the k-th Gaussian kernel in the GKM. As the GKM can fit well most isotropic kernels, Eq. (3) is a more accurate model for real defocus PSFs than the often-used single Gaussian/disk form; see supplementary materials for a demonstration.
Remark 1. The GKM degenerates to the single Gaussian form when only one mixing coefficient is 1 and the others are 0 for every location [m,n). In general cases, the weighted summation of Gaussian kernels can represent non- -an-Gassiian kernels, and thus the GKM can express a wider family of defocus PSFs than the single Gaussian form. There is also another work [9] that models defocus PSFs beyond single Gaussian kernels. uses the generalized Gaussian function [9] where the parameters to be estimated are wrapped in a complex nonlinear function. In comparison, our GKM model is linear with pre-defined (ak has which facilitates the estimation on its parameter (Ba]
nuntinearfunction.
We can rewrite 1 as
K
K Σ
y Σ n, nn 2 (bm, 2 x)
-EX
Σ
5mn c (100mm)ngg(())))
Br o (g(ox)
k=

---

where m. denotes the Dirac delta centered at location [m, ni,, and @, O denote the operations of 2D convolution and entry-wise multiplication. respectively. Then we have the GKM-based model for defocus blurring:
K Bx @ (g(ok) @ @x).
(4)
Fixed-point iteration unrolling Recall that the blurring operator B is about keeping the low- frequency components and attenuating high-frequency ones of an image. Let I denote the identity mapping. The mapping I - B is then about attenuating the low-frequency components and keeping the high-frequency ones. Neglecting the noise and rewriting (I) by
a - y + (I B) o x;
(5)
we have then a fixed-point iteration for solving defocus deblurring, which is given by
(++++) = ((x y a+ (t) - (t y+ (t) (g(a) for t = 1....... (6)
Note that the fixed-poine point iteration above will be convergent if I is contractive mapping, or equivalently the eigenvalues of B fall in (0, 1). which holds true when the defocus blurring is uniform with normalized Gaussian kernel, ie., the scene depths are constant the view.
Define 0, = 0 and g(r)) = 6, so that clear regions can be modeled by setting 31 = 1 and zeroing BR for 1. Let 71 ,3 and Y Bg for k 1. The iteration (6) can be expressed
K
x (1+1) (0) + Σ o (g(ox) @ x (1),, for t = -1,2....
In short, based on the GKM model of defocus PSFs, we can unroll fixed- iteration to solve ( with learnable coefficient matrices "1,-. Yk- The motivation of unrolling a fixed fxee-pooint iteration, instead of other iterative schemes such as gradient descent [19] and half quadratic splitting (HQS) 20)
to involve the forward operator B only, without introducing the transpose B and the pseudo- inverse B!.
Remark 2. Our approach is sort of in the category of optimization unrolling, a widely-used method- ology of designing DNNs for solving inverse problems. The key is to choose appropriate iteration scheme that fits the problem well. Most existing optimization unrolling based image deblurring 27 (71) consider uniform blurring, where the matrix B can be represented by a convolution. The iterative schemes they adopt such as HQS, usually involve an inversion process for B, which can be efficiently computed using FFT when B is convolution oper- ator. In our case, B is a spatially-varying blurring operator which does not have a computationally efficient inversion process. The proposed fixed-point iteration unrolling enables us avoid such inversion process in the DNN and use the forward operator only.
methods (e.g. [22] 23 20 24
25
26
The matrices Y1. 7KK of mixing coefficients can be intezpreted the attention maps associated to the feature maps generated by diffcrent Gaussian kernels. Thus, we construct a DNN with attention modules and long skip connections to utilize 77) for SIDD. In addition, we take multi-scale scheme to implement the unrolling: at each iteration the DNN predicts the all-in-focus image at current seale and up-samples it for the calculation of the next iteration, with weight sharing used across scales. This leads to a scale-recurrent attentive DNN with a lightweight implementation.
1.3 Main Contributions
In comparison to existing two-stage or dual-view-based methods, this paper is among the first ones to present an end-to-end DNN for SIDD. See below for the summary of our technical contributions:
A new and efficient parametric model based on GKM for defocus blur kernels, which fits real-world data better than existing models and thus leads to better performance in SIDD;
A new formulation of the deblurring process derived from fixed-point iteration so as to have a simple and efficient parameterization of defocus deblurring, which inspires an effective DNN for SIDD with low model complexity and high computational efficiency;
3

---

A scale-recurrent attention mechanism which combines the coarse-to-fine progressive esti- mation and the unrolled deblurring process for better performance.
The experiments show that the proposed DNN brings noticeable improvement over existing ap- proaches to SIDD, in terms of recovery quality, model complexity and computational efficiency.
2 Related Work
Two-stage SIDD Most studies of two-stage methods for SIDD are concentrated on the first stage, Lee. defocus map estimation, while the second stage is often done by calling existing non-blind deconvolution methods (e.g. [28 [I]. Defocus map estimation itself a challenging task. There available for sparse defocus map estimation edge points or regions. The dense defocus map is then constructed via some propagation method (e.g. Poisson matting (30). Recently, deep learning has been extensively studied for defocus map estimation; see e.g. 7 4 6 3T As discussed in Section L.I the two-stage methods suffer from the sensitivity to estimation errors and high computational costs. In comparison, the proposed end-to-end DNN does not involve the defocus map estimation and thus does not suffer from these issues.
several non-learning-base based methods 2
3
5 8
End-to-end learning for defocus deblurring There are few works learning end-to-end DNN for defocus deblurring. Abuolaim and Brown [8] proposed to train an end-to-end U-Net for predicting an all-in-focus image from two view images captured by dual-pixel sensor, and contributed a dataset of quadruples: a defocus blurred image, its all-in-focus counterpart, and two dual-pixel view images. Such dual-pixel-based DNN showed impressive performance. However, its performance significantly decreases when being used for SIDD where only a single image is available for input. The prerequisite on dual-view inputs also limits the wider applicability of this method. In contrast, the proposed DNN is grounded by the defocus blurring model and only assumes a single image as input, thus being applicable to commodity cameras, Very recently, one parallel work to ours on end-to-end learning of SIDD was done by Lee et al. (32. They proposed a DNN that predicts the pixel-wiss wise filters for deblurring the deep defocused features of an image. In comparison, our DNN predicts the blurring filters for deblurring the image in an unrolled fixed-point iteration framework.
DNNs for spatially-varying motion deblurring There have been many studies on deep learn- ing for spatially-variant variant motion deblurring, especially on dynamic scenes with moving objects;
see e.g. [12 14
17). The methods using optical flow (e.g. (36)) or temporal cues of moving objects (e.g. [7 for training are not applicable to SIDD. Many of these
13
16
15
33
35
36
37
methods also adopt multi-scale structures (e.g-
[3] or attention mechanisms (e.g- [17), but with
generic designs which cannot effecttiely exploit the inherent characteristics of defocus blurring, e.g. strong isotropy and high correlation of the shapes of pixel-wise defocus PSFs. In comparison, our DNN is specifically designed for defocus deblurring and thus enjoys better performance in SIDD.
Unrolling-base deep learning for image deblurring Unrolling- based DNNs have been exten- sively studied for non-blind image deblurring where the PSF is given an input; see e.g. 20 24 21) There are also some works 22 23 25 27] on unrolling-
blind image deblurring where the PSF restricted the case of uniform blurring. The proposed unrolling- based DNN is the first one that can handle vatrally-vvrying defocus blurring without given the PSFs, thanks to the proposed GKM- -esd-- defocus blurring model.
is unknown. However, these methods
3 Network Architecture
The DNN we construct for SIDD, named as GKMNet (Gaussian Kernel Mixture Network), is based on the fix- point iteration 7 with multi-scale recurrent fashion. See Figure for the outline.
Given an input image y. we generate its multi-scale versions y1 yr via bi-linear downsampling with factors 27-1 2 respectively. Let Φ; yı and let 12 denote the upsampling operation by factor 2. We consider the following multi-scale extension of 7
K
L++1
yt +
7 @ (g(ok) @ 2x12), fort 1,... T.
(8)

---

The GKMNet employs T recurrent blocks to implement 8). The t-th block takes and 2;12 as input and outputs 1-+1- There are two modules in each block: a Gaussian convolution module (GCM) and scale-recurrent attention module (SRAM). The GCM performs the Gaussian filtering in (8) to provide K feature maps, denoted by (21,1..... 2t.K) ). for the i-th block. The SRAM generates the corresponding coefficient maps (Nt,1s Yt,K) from Vr- The output of the 1-th block is given by
K
yt (r, O st,4),
k=1
where the weights {wi,k)e. are for scaling the mixing coefficients {7E predicted by the SRAM within a certain range. The weighted summation with (wr,k) implemented by x convolution. The output of the T'-th block, i.e. æT, is used as the final deblurring result. Note that the GKMNet relates the scales not only by passing the output from one scale to the next, but also by the recurrence mechanism built in the SRAM. Let æt denotes the downsampled ground truth of the same size æt. The training loss is defined by the supervision at all scales:
T
=
Σ C(x,, af"), for some cost function C(:s
(10)
where assign equal weights to the losses at different scales for simplicity.
Input
2
y;
y
Scale Recurrett
SRAM
12
GCM
Gazisin Comsoiution Modsle
Muliiiltation
Drate te
Kauli
Dewncampinng
Upcompinng
2
+
Concaterate
x conN
1 ConV
1 conv
Ground Tuuh
OuTQuE Immgg
Loss
12
x
Loss
Figure 1: Diagram proposed GKMNet for SIDD with 3.
Gaussian Convolution Module The GCM is simple group convolutional layer defined by series of pre-designed 2D Gaussian kernels applied to the R, G, channels, respectively. The kernel sizes set to x 1, 3 x 3,5 5, ... .M x M. For kernel size X with 1. we generate two Gaussian kernels with a set to {mm 2) and (m 1) respectively. The Gaussian kernel size 1 x is the Dirac delta kernel. As result, there totally M different kernels. Their parameters are fixed across all scales. The convolution kernels GCM are not learned for two reasons: (i) reduces the model complexity and leads to faster training; and (ii) the learned convolution kernels do not benefit the performance as empirically observed. Note that in addition to H,, we also input yi in parallel for feeding more information the next stage.
Scale-Re Geale-Recurrent Attention Module The SRAM maps an input image ye to the coefficient maps {+r, 1K jkm The coefficient maps can be viewed as the spatial-channe attention maps associated to the feature maps generated by different Gaussian kernels. We thus draw inspirations from existing attention modules (e.g. [13 [ [7) to have lightweight design SRAM. As illustrated in Figure 2 the SRAM consists of (i) an attentive encoder-decode backbone 38 for feature extraction; and (ii) an attention prediction unit (APU) based on Conv-L.STM (convolutional long short-term memory) 33)
The attentive encoder-decoder backbone sequentially connects a convolutional layer, two encoder blocks, and two decoder blocks. Each encoder/decoder block contains one convolutional layer with
5

---

downsampling//psamppi two convolutional layers with residual connections, and triplet attention block [40]. The triplet attention used improve the NN's spatial and channel adaptivity for better prediction, which can be viewed as attention-in intioonnii-atttttin mechanism in the SRAM. Given a feature tensor X' E RCCHXx defined in the channel-height-wee width space, the triplet attention block generates three parallel attention maps aw E TR dg RC ac R& from the thannel-height channel-width and height-width slices of X respectively, and then applies them for re-calibrating X so to encode spatial-channel dependencies into the features.
The APU contains two paths. The first path mainly contains Conv-LSTM and applies downsam- sling/ppsampling before/after it for inducing local smoothness on the predicted coefficient maps. The second path contains two convolutional layers, but without nownsamping/upsaapiigg for preserving detailed information for the prediction. Let 71 72 denote the predictions from these two paths. Then the final coefficient map is predicted by y = tanh( ( ④ 22 + ). Such a design motivated from the saneezee-nn-Attentie
[1]] The Conv-L.STM, at the first path in the APU is used for capturing the dependencies among the blur amount at different scales. The hidden states in the Conv-LSTM can capture usefuu information from different scales and benefit the restoration across scales. This enables the SRAM to progressively improve the estimation on the coefficient maps.
(- 33th Masimm
Barkhoss
[R
[Rx
ON ReLU
Paofing
ConsTrarst
wise Multiplicntion
Comsus BN + ReLU
Unsampptnn
Triplet Attention
APU
Stement - wise Sunmnation
Figure 2: Diagram of SRAM. All convolutional kernels are of size 3.
21 27)) for image deblurring. the GKMNet does not have any explicit artifact removal block. However, it still performs well empirically, shown in the experiments. The reasons may be as follows. First, the inversion process implemented by the GCM and SRAM is not the classic non-learnable estimator in existing unrolling based DNNs, but the one containing learnable blocks. Thus, the GKMNet can be viewed as some forms of the inversion process with implicit built-in artifact removal. For instance, in the GKM model, the learned combination of Gaussian kernels not only can express the defocus kernels, but also may encode a component to suppress the artifacts. Second, there is probably a regularization effect due the fact that the fixed point iterations do not necessarily converge to a fixed point (similar to early stopping), as well as a regularization effect due to the relaxation of the fixed-point iteration the multi-scale iteration. While the exact nature of the regularizations and built-in artifact removal is unknown, it does not necessarily prevent the GKMNet from working, since the parameters are and-o-endd learned in a supervised manner.
Remark 3. Unlike many existing unrolling-based DNNs (e.g. [23 20 24
25
26
4 Experiments
There are few datasets available for benchmarking defocus deblurring. The most well-known one the DPD dataset 18. provides 500 pairs of images with defocus blur and their corresponding all-in-focus images, as well as the two associated sub-aperture views called dual-pixel images, all in 16-bit color. The training/validaiinn/ee
splits in the dataset consist of 350/74/76 samples. Our GKMNet is trained and tested on the DPD dataset without using the dual-pixel images. In addition, we test GKMNet on the RTF test set 7 as well as the images from CUHK-BD dataset 42
Three image quality metrics are used for quantitative evaluation, including two standard metrics: PSNR (Peak Signal to Noise Ratio) and SSIM (Structural Similarity Index Measure) [43), and the LPIPS (Learned Perceptual Image Patch Similarity) 44 for perceptual quality (also used in (18). The GKMNet is also compared to other DNNs in terms of model complexity and computational efficiency. The model complexity is measured by three metrics: number of parameters, number of FLOOPs (floating-pon) operations per second), and model size. The computational efficiency is measured by the average inference time an image of 1680 1120 pixels, tested on an Intel

---

15-9600KF CPU and on an NVIDIA GTX 1080Ti GPU, respectively. For non- non-DNN-based methods, only the inference time on CPU is reported.
Through all experiments, the maximum size of Gaussian kernels in GCM is set to M = 21. The number of scales is set to T 3. The learnable parameters in SRAM are initialized by Xavier [45] The Adam optimizer [46] is used for training with 3000 epochs and batch size 4. The learning rate is 4 in the first 2000 epochs and decayed to 10 in the last 1000 epochs. The cost function in (TO) is set to the squared f2 loss in the first 1000 iterations, and alternatively set to the SSIM loss and squared Eg loss every 500 epochs afterwards. Data augmentation is done by random cropping to 256 x 256 pixels. The code of GKMNet is available at httpt::///// //githu com cwZccu/ Cscccc//GWMK/N GKMNet
fixed 10
4.1 Evaluation on DPD Dataset
The methods for comparison include JNB 21. EBDB (5), DMENet 6 and DPDNet [8]. The DPDNet [18] has two versions: DPDNet-D taking two dual-pixel images as input and DPDNet-S only taking a single image input. Both versions included for comparison whose results reproduced by the pre-trained models from their authors. The JNB, EDBD) and DMENet are all two-stage methods focusing on defocus map estimation, with results quoted from [8] In addition, we also include two DNNs of dynamic scene deblurring: SRN 13 and AttNet 17 These two DNNs also adopt a multi-scale architecture. Both DNNs are retrained using the same data as ours.
See Table for the quantitative comparison on deblurring performance. Our GKMNet outperforms all other compared methods. The performance of EBDB, DMENet and JNB is much worse than that of GKMNet. This is probably because the errors in their defocus maps are magnified in the consequent deconvolution process. It indicates the advantage of the end-to-end learning of GKMNet which does not require defocus map estimation. The GKMNNet also outperforms DPDNet-S by a large margin. This is not surprising as DPDNet-S is originally designed for defocus deblurring on two view images, not a single one. Also unsurprisingly. GKMNet outperforms SRN and AttNet, two DNNs designed for dynamic scene deblurring rather than SIDD. Surprisingly, our GKMNet with single-image input even outperforms DPDNet-D which takes dual-pixel images as input. All these results have clearly demonstrated the effectiveness of the proposed GKMNet on SIDD. See supplementary materials for the visualization of coefficient maps generated by the SRAM.
Table also lists the results on model complexity and computational efficiency. In terms of all three metries, GKMNet's s complexity is much lower than that other deep models. For instance, around 1/23 of DPDNet and 1/7 of SRN for number of parameters, and around 1/62 of DPDNet for model size. Such low complexity comes from the compact architecture of GKMNet. Regarding the running time, GKMNet is also much faster than the JNB, EBDB and DEMNet, these methods need to call an iterative deconvolution post-process which is slow. GKMNet is about ten time faster than DPDNet, and its speed is comparable SRN and AttNet. It is noted that GKMNet has a much less training time due to its low complexity. Its training on the DPD dataset takes less than 32 hours, while SRN and AttNet take nearly three days, on an NVDIA GTX 1080Ti GPU.
Table I: Quantitative comparison of different methods on DPD test set.
PSNR
SSIM
LPIPS
#Parameters
#FLOPs
Model Size
Time (Seconds)
Model
(dB)
(Million)
(Billion)
(MegaBytes)
CPU
GPU
JNB 2
23.84
0.715
0.315
843.1
EBDB
5
23.45
0.683
0.336
929.7
DMENet
6
23.41
0.714
0.349
26.94
613.7
DPDNet-S
18
24.34
0.747
0.277
32.25
4042.5
355.3
56.6
0.41
DPDNet-D
18
25.12
0.786
0,223
32.25
4048.6
355.3
56.6
0.41
SRN
13
24.61
0.612
0.265
10.25
3117.6
32.0
58.3
0.032
AttNet 11
25.22
0.781
0.219
6.91
3450.4
39.5
62.8
0.041
GKMNet [Ours]
25.47
0.789
0.219
1.41
603.5
5.7
43.9
0.040
As mentioned in Remark 3. without any explicit artifact/noise removal block in the GKMNet, the robustness of GKMNet is one concern. Thus, we evaluate the robustness to image noise using the DPD dataset with additional corruption by additive Gaussian white noise with standard deviation G 1,3,5 5 respectively. The DPDNet-D, SRN and AttNet are selected for comparison. Both the
7

---

models trained on the original DPD training set and those retrained on the noisier DPD training set are included. The later ones are marked with a "+". See Table 2 for the results. In the presence of additional noise, our GKMNet still outperforms other models for both versions of training data.
Table 2: Quantitative comparison of different methods on noisier DPD test set.
Noise
DPDNet-D
SRN SSIM
AttNet SSIM
GKMNet SSIM
PSNR
SSIM LPIPS
PSNR
LPIPS
PSNR
LPIPS
PSNR
LPIPS
25.08
0.788
0.220
24.58
0.692
0.315
25.18
0.783
0.221
25.43
0.784
0.225
3
24.73
0.746
0.248
24.48
0.668
0.366
24.77
0.749
0.261
25.02
0.758
0.246
5
24.07
0.665
0.441
24.30
0.628
0.411
24.36
0.660
0.347
24.61
0.712
0.299
Noise ô
DPDNet-D+
SRN+
AuNet+
GKMNet+
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
25.09
0.788
0.220
24.61
0.697
0.316
25.18
0.783
0.221
25.48
0.789
0.221
3
24.76
0.748
0.248
24.53
0.672
0.360
24.85
0.753
0.255
25.12
0.760
0.247
5
24.37
0.710
0.379
24.34
0.632
0.399
24.48
0.681
0.332
24.92
0.742
0.273
4.2
Evaluation on RTF Dataset and CUHK-BD Sample Images
The RTF dataset 7 contains 22 pairs of defocused and all-in-focus images. Following 77 in addition to the original images, two noisier versions are generated by adding Gaussian white noise with standard deviation ā and 2.55 respectively. The DPDNet-S, SRN and AttNet are selected for comparison. Note that the training set of RTF is not available. Thus, we directly apply the models that are respectively trained on the original DPD dataset and the noisier DPD dataset. See Table for for the results. The GKMNet again outperforms other DNNs. Since the models art trained and tested with different datasets, such results also demonstrate the generalizability of our GKMNet.
Table 3: Quantitative comparison of different methods on RTF test set.
Noise Ô
DPDNet-S SSIM LPIPS
SRN SSIM
AttNet SSIM
GKMNet SSIM
PSNR
PSNR
LPIPS
PSNR
LPIPS
PSNR
LPIPS
0
23.61
0.597
0.296
23.71
0.617
0.324
25.45
0.802
0.219
25.72
0.811
0.211
1
23,58
0.592
0.332
23,68
0.611
0.311
24.99
0.797
0.243
25.56
0.798
0.207
2.55
23.44
0.576
0.344
23.50
0.591
0.276
23.83
0.727
0.257
25.01
0.761
0.205
Noise ô
DPDNet-S+
SRN+
AutNet+
GKMNet+
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
0
23.65
0.605
0.290
23.70
0.617
0.325
25.40
0.792
0.225
25.69
0.808
0.212
1
23.59
0.594
0.329
23.65
0.605
0.313
25.06
0.800
0.236
25.60
0.802
0.206
2.55
23.49
0.581
0.341
23.54
0.597
0.281
24.33
0.749
0.250
25.21
0.787
0.210
The CUHK-BD dataset 42 contains 704 defocused images without ground truths. We select some of its images for test. See For the visual inspection on the results. Our GKMNet can successfully restore fine details with less visual artifacts, in comparison to DPDNet-S, SRN and AutNet. More visual comparison can be found in supplementary materials. It is worth mentioning that the defocus PSFs the image synthesized from dual-pixel images are slightly different from the ones in real images 47 48 49] However, our GKMNet still generalizes well.
4.3 Ablation Study
To evaluate the performance contribution of each component in GKMNet, we implement the following baselines from GKMNet for comparison, which are trained in the same way as the original GKMNet. (a) GKMNet(1)): Use only the original image scale in GKMNet with T = (b) w/o ConV-LSTM: Replace the Conv-LSTM block at the APU by a convolutional layer with the same kernel size, and share its weights across scales for relating different scales; (c) SRAM* Use the SRAM as a whole DNN for SIDD, with an 1 x convolutional layer with Sigmoid activation attached for outputting an image; (d) Learned GCM (GKM): The kernels in GCM are initialized by the predefined Gaussian kernels and learned with the DNN. (c) Learned GCM (Rand): The kernels in GCM are randomly
8

---

ARRET
ARRET
ARRET
ARRET
ARRET
Input
DPDNet-S
SRN
AttNet
Ours
Figure Visual comparison of SIDD results of different methods on CUHK-BD sample images.
initialized by Xavier 45 and learned with the DNN. Note that weight sharing is adopted for GCM across scales in (d) and (c) because it leads to performance improvement.
See Tor the results of the baselines, from which we have the following observations. (a) The multi-scale estimation scheme in SRAM leads to significant improvement over the single-scale one, with only few additional model parameters introduced. (b) The improvement from the Conv-LSTM over simple cross-scale veight-sharing convolutions significant, which implies that the Conv-LSTM can effectively exploit the features learned from different scales to guide the deblurring process. (c) Directly using the SRAM as an end-to-end DNN for SIDD leads to much worse results, which justifies the effectiveness of our specific DNN design. (d) Learning the GCM with Gaussian kernel initialization leads to a very minor improvement over using fixed Gaussian kernels in GCM. (e) Learning the GCM with random initialization even yields slightly worse results. Both (d) and (c)

---

indicate the effectiveness of our GKM model for defocus PSFs. See also Figure 4 for the kernels w/o and w/ learning GCM. The kernels learned with (d) are very close original predefined ones.
Table 4: Quantitative comparison of GKMNet and its baselines.
PSNR (dB)
SSIM
LPIPS
#Parameters (Million)
#FLOPs (Billion)
Model Size (MegaBytes)
Time (Seconds)
Model
SRAM
21.98
0.724
0.251
1.41
606.1
5.30
0.040
ASGMNe((1)
23.63
0.723
0.248
1,41
459.7
5.70
0.036
w/o Conv-LSTM
25.00
0.774
0.231
1.11
418.7
4.33
0.036
Learned GCM (Rand)
25.35
0.787
0,223
1.41
603.5
5.70
0.040
Learned GCM (GKM)
25.49
0.790
0.217
1,41
603.5
5.70
0.040
GKMNet
25.46
0.789
0.219
1,41
603.5
5.70
0.040
Predefined Gaussian kernels (fixed in learning)
Learned kemnlls (initialized Gaussian kernels)
2nd
21th
2nd
9th
21th
Figure 4: Visualization of predefined kernels and learned kernels in GCM.
Recall that the fixed point iteration 7 is applied multiple iterations the same scale, while our actual architecture also simultaneously increases the scale. We could further write our architecture as scales and R iterations per scale, where effectiveness of the multi-scale estimation in GKMNet, Table 5 lists the results using different values of S and R. shows that using single or fewer scales with multiple iterations noticeably decreases the performance and increases the inference time. Using an additional iteration with $ = 3 only brings a minor improvement but nearly doubles the complexity and time. Thus, our coarse-to- fine estimation scheme is better implementation for high performance and low computational cost.
S R 1 in the GKMNet. To further verify the
Table Quantitative comparison extended GKMNet models using different values of S and R.
PSNR (dB)
SSIM
LPIPS
Parameters (Million)
FLOPs (Billion)
Model Size (MegaBytes)
Time (Seconds)
Model
S=1, R=3
25.11
0.773
0.237
1.41
1384.2
5.7
0.069
S=2, R=3
25.13
0.789
0.205
4.22
1730.3
17.1
0.122
S=3, R=2
25,53
0.773
0.201
2.81
1211.2
11.3
0.081
S=3, R=1
25.47
0.789
0.219
1.41
603.5
5.7
0.04
Conclusion
Defocus blur often occurs in images and has its characteristies from motion blur. This paper proposed DNN for SIDD with strong motivations from unrolling a fixed-point iteration derived from GKKM-base model of defocus blurring process. Together with scale-recurrent implementation, we developed a lightweight DNN with state-of fatte-of-the of-the--rtt performance. future, we will study the extension other image deblurring problems such as dynamic scene deblurring.
While our approach achieved stateoof-the-att results on existing datasets, its implicit regularizations and built-in artifact removal mechanisms not very clear. In addition, its performance severcly blurred regions is not that satisfactory, with much room for improvement. These issues will also be studied in our further work.
Acknowledgments and Disclosure of Funding
Yuhui Quan would like to acknowledge the support from National Natural Science Foundation of China (Grant No. 61872151) and CCF-Tencent Open Fund 2020. Hui Ji would like to acknowledge
10

---

the support from Singapore MOE Academic Research Fund (AcRF) Tier 1 Research Grant (R- 14-1 000-315-114).
References
[1] Anat Levin, Rob Fergus, Frédo Durand, and William Freeman. Image and depth from a conventional camera with a coded aperture. In ACM Special Interest Group on Computer Graphics, 2007.
[2] Jianping Shi, Li Xu, and Jiaya Jia. Just noticeable defocus blur detection and estimation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 657-665, 2015.
[3] Guodong Xu, Yuhui Quan, and Hui Ji. Estimating defocus blur via rank of local patches. In IEEE International Conference on Computer Vision, pages 5371-5379, 2017.
[4] Jinsun Park, Yu-Wing Tai, Donghyeon Cho, and In So Kweon. A unified approach of multi-scale deep and hand-crafted features for defocus estimation. In IEFE Conference on Computer Vision and Pattern Recognition, pages 2760-2769, 2017.
[5] Ali Karaali and Claudio Jung. Edge-based based defocus blur estimation with adaptive scale selection. IEEE Transactions on Image Processing. 27:1126 1137, 2018.
[6] Junyong Lee, Sungkil Lee. Sunghyun Cho, and Seungyong Lee. Deep defocus map estimation using domain adaptation. In IEEE Conference Computer Vision and Pattern Recognition, pages 12214-12222, 2019.
[7] Laurent D' Andrès, Jordi Salvador, Axel Kochale, and Sabine Süsstrunk. Non-parametric blur map regression for depth of field extension. IEEE Transactions Image Processing, 25:1660-1673, 2016.
[8] Sunghyun Cho and Seungyong Lee. Convergence analysis of map based blur kernel estimation. In IEEE International Conference Computer Vision, pages 4818 4826, 2017.
[9] Yu-Qi Qi Liu, Xin Du, Hui-Liang Shen, and Shu-Jic Chen. Estimating generalized gaussian blur kernels for out-of-focur image deblurring. IEEE Transactions Circuits and Systems for Video Technology, 31:829-843, 2020.
[10] Hui Ji and Kang Wang. Robust image deblurring with an inaccurate blur kemnl. IEEE Trunsactions on Image Processing. 1124):16241164, 2011.
[11] Yuesong Nan and Hui Ji. Deep learning for handling kemel/mmoeel uncertainty in image deconvolution. In IEEE Conference on Computer Vision and Pattern Recognàtion, pages 2388-2397, 2020.
[12] Seungjun Nah, Tae Kim, and Kyoung Mu Lee. Deep multi- scale convolutional neural network for dynamic scene deblurring. In JEEE Conference on Computer Vision and Pattenn Recognition, pages 255-265, 2017.
[13] Xin Tao, Hongyun Gao, Yi Wang, Xiaoyong Shen, Jue Wang, and Jiaya Jia. Scale-recurrent network for deep image deblurring. In IEEE Conference on Computer Vision and Pattern Recognition, pages 8174-8182, 2018.
[14] Jiawei Zhang, Jinshan Pan, Jimmy Ren, Yibing Song, Linchao Bao, Rynson WH Lau. and Ming-Hsuan Yang. Dynamic scene deblurring using spatially variant recurrent neural networks. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2521-2522 2018.
[15] Hongyun Gao, Xin Tao, Xiaoyong Shen, and Jiaya Jia. Dynamic scene deblurring with parameter selective sharing and nested skip connections. In IEEE Conference on Computer Vision and Puttern Recognition, pages 3356, 2019.
[16] Hongguang Zhang, Yuchao Dai, Hongdong Li, and Piotr Koniusz. Deep stacked hicrarchical multi-patc) network for image deblurring. In IEEE Conference on Computer Vision and Pattern Recognition, pages 5978-5986, 2019.
[17] Yong Xu. Ye Zhu, Yuhui Quan, and Hui Ji. Attentive deep network for blind motion deblurring dynamic scenes. Computer Vision and Image Understanding, 205:103169, 2021.
[18] Ahdullah Abuolaim and Michael S. Brown. Defocus deblurring using dual-pixel data. In European Conference on Computer Vision, 2020.
[19] Dong Gong. Zhen Zhang, Qinfeng Shi, Anton van den Hengel, Chunhua Shen, and Yanning Zhang Learning deep gradient descent optimization for image deconvolution. IEEE Transactions on Neural Networks and Learning Systems, 2020.

---

[20] Jiawei Zhang, Jinshan Pan, Wei-Sheng Lai, Ryason WH Lau, and Ming-Hisue Yang. Learning fully convolutional networks for iterative non-blind deconvolution. In IEEE Conference on Computer Vision and Pattern Recognition. pages 3817-3825, 2017.
[21] Yuesong Nan, Yuhui Quan, and Hui Ji. Variational-em: based deep learning for noise-blind image deblurring. In IEEE/CVF Conference on Computer Vision and Patiern Recognition, pages
2020.
[22] Christian Schuler, Michael Hirsch, Stefan Harmeling, and Bernhard Schölkopf. Learning to deblur. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(():1439-15)) 1451, 2015.
[23] Wangmeng Zuo, Dongwei Ren, David Zhang, Shuhang Gu, and Lei Zhang. Learning iteration- wise generalized shrinkage-thresholding operators for blind deconvolution. IEEE Transactions on Image Processing, 25(4):1751-1766 2016.
[24] Weisheng Dong, Peiyao Wang. Wotao Yin, and Guangming Shi. Denoising prior driven deep neural ketwork for image restoration, IEEE Trunsactions on Puttern Analysis and Machine Intelligence, 41(10):2305-2318 2019.
[25] Yuelong Li, Mohammad Tofighi, Vishal Monga, and Yonina C Eldar. An algorithm unrolling approach to deep image deblurring. In JEEE International Conference on Acousticx, Speech and Signal Processing, pages 7675-7679, IEEE, 2019.
[26] Dipan Pal Raied Aljadaany and Marios Savvides. Douglas-rachford networks: Learning both the image prior and data fidelity terms for blind image deconvolution. In JEEE Conferencee on Computer Vision and Puttern Recognition, pages 01245-10244, 2019.
[27] Yuelong Li, Mohammad Tofighi, Junyi Geng, Vishal Monga, and Yonina C Eldar. Efficient and interpretable deep blind image deblurring via algorithm unrolling. IEEE Transactions on Computational Imaging. 6:666-681 (166-681,25 2020.
[28] D. Fish, A. M. Brinicombe, E.R. Pike, and Walker. Blind deconvolution by means of the richardson- lucy algorithm. Journal of The Optical Society of America A A-optics Image Science and Vision, 12:58- 65, 1995.
[29] Dilip Krishnan and Rob Fergus. Fast image deconvolution using hyper-laplacian priors. In Advances in neural information processing systems, 2009.
[30] Jian Sun, Jiaya Jia, Chi-Keung Tang, and Heung-Yeuur Shum. Poisson matting. ACM Special Interest Group on Computer Gruphics, pages 315-321. 2004.
[31] Saeed Anwar, Zeeshan Hayder, and Fatih Porikli. Deblur and deep depth from single defocus image. Machine Vision and Applications, 32: 12:1-13, 2021.
[32] Jaesung Rin Sunghyun Cho Junyong Lee, Hycongseok Son and Seungyong Lee. Iterative filter adaptive network for single image defocus deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 2034-2042, 2021.
[33] Kuldeep Purohit and AN Rajagopalan. Region-adaptive dense network for efficient motion deblurring. In National Conference of the American Association for Artificial Intelligence, volume 34, pages 11882- 11889, 2020.
[34] Maitreya Suin, Kuldeep Purohit, and AN Rajagopalan. Spatially-attnntive patch- patch-tierarchical network for adaptive motion deblurring. In IEEE Conference Computer Vision and Pattern Recognition, pages 3606 3615, 2020.
[35] Kaihao Zhang, Wenhan Luo. Yiran Zhong, Lin Ma. Bjorn Steager. Wei Liu, and Hongdong Li. Deblurring by realistic blurring. In IEEE Conference Computer Vision and Pattern Recognition, pages 2737-2746. 2020.
[36] Yuan Yuan, Wei Su, and Dandan Ma. Efficient dynamic scene deblurring using spatially variant deconvo- lution network with optical flow guided training. In IEEE Conference Computer Vision and Pattern Recognition, pages 3555-3564, 2020.
[37] Dongwon Park, Dong Un Kang. Jisoo Kim. and Se Chun. Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training. In European Conference on Computer Vision, 2020.
[38] Xiao- Mao, Chunhua Shen, and Yubin Yang. Image restoration using very deep convolutional encoder- decoder networks with symmetric skip connections. In Advances in neural information processing systems. 2016.
12

---

[39] Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang chun Woo. Convolu- tional Istm network: A machine learning approach for precipitation nowcasting. In Advances in neural information processing systems, 2015.
[40] Diganta Misra, Trikay Nalamada, Ajay Uppili Arasanipalai. and Qibin Hou. Rotate to attend: Convolutional triplet attention module. In IEEE Winter Conference on Applications of Computer Vision, pages 3139-3148, 2021.
[41] Zilong Zhong, Zhong Qiu Lin, Rene Bidart, Xiaodan Hu, Ibrahim Ben Daya, Jonathan Li, and Alexander Wong, Squeeze-ang and-attention networks for semantic segmentation. In JEEE Conference on Computer Vision and Pattern Recognition, pages 13062 13071, 2020.
[42] Jianping Shi, Li Xu, and Jiaya Jia. Discriminative blur detection features. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2965-2972, 2014.
[43] Zhou Wang, Alan Bovik, Hamid Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing. 13:600 612. 2004.
[44] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Conference on Computer Vision and Pattern Recognition, pages 586 595, 2018.
[45] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In International Conference Artificial Intelligence and Statistics, pages 249-256 JMLR Workshop and Conference Proceedings, 2010.
[46] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zach Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation pytorch. 2017.
[47] Sameer Ansari Rahul Garg, Neal Wadhwa and Jonathan T Barron. Learning single camera depth estimation using dual- duxl-pixels. In IEEE Conference on Computer Vision and Pattern Recognition, pages 7628-768777 7637, 2019.
[48] Mahmoud Afifi Abhijith Punnappurath, Abdullah Abuolaim and Michael S Brown. Modeling defocus- disparity in dual-pixel sensors. IEEE International Conference on Computational Photograpky, pages 1-12, 2020.
[49] Richard Hartley Miaomiao Liu Hongguang Zhang Liyuan Pan, Shah Chowdhury and Hongdong Li. Dual pixel exploration: Simultaneous depth cstimation and image restoration. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4340-4349, 2021.
13


Reviews:
[{"review_id": "ynD2qZcXkrx", "reviewer": "Reviewer_SpFs", "summary": "This paper proposed a deep attentive network for defocus blur removal. The basic idea is the unfolding of a fixed point algorithm for defocus blur removal using a Gaussian Scale Mixture model for modeling the spatially varying blur kernels. The proposed network is a multi-scale architecture with GCM block and SRAM block. The proposed network is evaluated on several defocus blur datasets, and showed improved performance compared with some other methods, e.g., AttNet, ASG, etc.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The major interesting contributions of this work are the proposed GSM modeling of  defocus blur and the attentive module to estimate the component coefficients. The proposed network is inspired by unfolding a fixed point algorithm for deblur in Eqn. 7.\n\n1. On the fixed point iterations.\n\nThe fixed point iterations in Eqn. (5) is not fully analyzed, for example, its convergence analysis. Because it seems that the proposed network is faithfully implementing this iterations in GCM block, these analysis are important to understand the network performance. \n\n2. Whether the network learn to remove artifacts.\n\nIt is not new to unfold algorithms, such as HQS, for image deconvolution. What is the advantage of this proposed fixed point algorithm (and its derived network) compared with these previous HQS-based unfolded networks for image deblur? Moreover, it seems that the proposed network does not explicitly have the artifact removal block (commonly corresponds to proximal operator of regularizer). How to guarantee that the restored sharp image is artifact-free?\n\n3. Comparisons.\n\nIt is true that there are very few papers focusing on the blind defocus blur removal. However, the proposed approach can be further compared with other possible variants, such as different unfolded network structures (e.g., HQS) .\n\nAre the proposed network trained once, and tested on different datasets? Is the trained network generalizable to defocus image captured by different cameras?  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Removing defocus blur is a fundamental task in image processing, benefiting to applications such as auto-focus, depth from defocus, downstream recognition tasks, etc. The paper did not fully discuss on the limitations of the proposed approach. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5 hours"}, "extra_text": {}}, {"review_id": "sMvSQYX1AdK", "reviewer": "Reviewer_mowT", "summary": "This paper focuses on addressing the challenge of single-image defocus deblurring.\nThis work is based on the GSM (Gaussian scale mixture) theory to construct a deblurring pipeline with the DNN (deep neural network).\nThis paper uses extensive experiments to verify the effectiveness and robustness of the model.\n", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths] \n\n1)\tGaussian mixture methods are usually used to fit unfriendly data distributions, and this is the first case I have visited where the use of both the Gaussian kernels and the coefficient  matrices to accomplish fitting arbitrary data distributions, may be a very bold paradigm of innovation.\n\n2)\tA large number of experimental comparisons have provided the evidences to support the validity of this work.\n\n\n[Weaknesses]\n\nAlthough the proposed model is observed to be valid in terms of quantitative and qualitative results, I still have doubts about the contributions of this work.\n\n1) In L144-145, this work assigns equal weights to the loss at each scale of AGSMNet. In this regard, as far as I know we are more concerned about the clarity of the output of the last scale, and it is natural to think that the last scale los should be assigned the largest weight.\n\n2) In L160-161, this work uses the APU to generate coefficient maps, APU is based Conv-LSTM. This method still seems to have a high computational, so why not consider using the encoder of the transformer to replace it? I'm looking forward to seeing the new ablation experiments.\n\n\n3) In Table1, I am still troubled by why the proposed model has the least FLOPs, but  process of the GPU is not fast.\n\n4) The Gaussian kernel itself has been applied to the denoising task, so the robustness of the model is necessarily excellent and the robustness consideration of the experimental part seems unnecessary.\n\n5) I think the biggest problem of this paper is still the limited innovation, which seems to box up multi-scale, long range dependence, dual path and other methods to build the model.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "none", "review_text": "", "extra_scores": {"ethical_concerns": "none", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "bgW7f4RiMYl", "reviewer": "Reviewer_1KQg", "summary": "The paper proposes a neural network architecture for deblurring images affected my single image defocus blur. The architecture is inspired by the notion that (a) the blur kernel at any pixel for defocus blur can be well approximated as a combination of Gaussian kernels with different widths, and (b) given knowledge of the blur kernel, the deblurred image is given by the fixed point of a repeated filtering operation. These notions are combined with multi-scale reasoning, and leads to a network that performs well in practice while being very light-weight and with fewer number of parameters. This is demonstrated by experiments and ablations.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the proposed architecture is novel, well-motivated, and the experiments are fairly convincing. I'm leaning towards acceptance, but hope the authors will respond to/address the following issues:\n\n1. Calling the kernel model a Gaussian Scale Mixture (GSM) is very confusing. This is because Gaussian Scale Mixture *distributions*, as probabilistic models of image gradients/wavelet coefficients, were a very popular, successful, and widely used approach to traditional image restoration. In this context, the \"GSM\" refers to the approximation of isotropic kernels as sums of Gaussian of different widths (scales). I strongly recommend the authors consider calling this \"Gaussian Kernel Mixture\" or something else.\n\nOn a similar note, it may be worth calling \\gamma's mixing weights or some such instead of attention masks (because again, attention has different connotations in neural network architectures). \n\n2. The description of the fixed point iteration as motivation is great, but while in Sec 1.2, this is applied to multiple iterations at the same scale, the actual architecture also simutaneously keeps increasing the scale at which the method is applied. This makes things a little bit confusing. Perhaps one could write the architecture as K scales and R iterations per scale, with total number of \"iterations\" = K x R, and that in this case, the authors chose to go with R=1. And that brings up another interesting ablation: is multi-scale really helpful, or would the network have achieved equivalent accuracy with fewer (or even a single) scale, and multiple blocks of GCM/SRAM per scale.\n\n3. Some of the specific details of the architecutre are hard to decipher from the figures. For one, I think there's a typo in equation (8)---I imagine it should be y_t + not y +. Also, it isn't clear what if any weights are shared across different scales. The GCM section talks about a fixed set of kernels, but it isn't clear whether the \\sigma values are at the downsampled or original resolution. And in general, for both the GCM and SRAM modules, it would be good to have text describing the precise input/output relationships in the supplementary (it's fine to use existing architecture blocks such as Conv-LSTM as \"functions\", but deciphering the architecture is a bit hard from just the figures).\n\n4. What's the difference between #parameters and model-size in Table 1?\n\n5. It isn't immediately clear how the attention maps are closely related to the defocus map in Figure 3.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "MjATGo0wKeN", "reviewer": "Reviewer_SBT5", "summary": "The authors propose an attentive GSM network for defocus deblurring, motivated from the unrolling of a fixed-point iteration. \nExperiments show that the proposed method in not only lightweight, but also outperforms existing defocus deblurring methods. \n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and organized. \nBased on a fixed-point iteration, the authors propose a new formulation of deblurring process.\nThe authors combine coarse-to-fine progressive estimation and unrolled deblurring process into a scale-recurrent attention mechanism.\nThe experiments show the superior of the proposed method in terms of quality, model size, Flops, and running time.\nIt would be better if the authors could provide demo testing code for the method.\n\nAfter rebuttal\n\nThanks for providing the feedback. I also read comments from other reviewers and like to keep my original rating.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "11"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received mixed reviews (3,5,6,7). One reviewer is particularly enthusiastic about the paper.  One reviewer was negative but did not participate in the discussion. His/her comments have been well addressed in the rebuttal and his/her final score of 3 has been ignored from the decision process. Two reviewers have borderline ratings (5,6), and the reviewer with score 5 agreed in the private discussion that it would be fine if the paper is accepted.

Overall, the AC believes that the paper introduces interesting ideas and recommends acceptance.  Nevertheless, Reviewer SpFs raises relevant questions about why the method works and why it makes sense. These should be taken into account in the final version of the paper. You will find below part of the private discussion, which will clarify these concerns and give some hints about how they can be addressed.

***************************
**Reviewer SpFs:**
In the second round of responses, the authors conduct experiments to compare with HQS-unrolling based method, and the conclusions are that the HQS is with higher computation cost and the additional artifacts removal block did not improve much. I appreciate these additional comparisons and it is still a mystery to me on the reason why this propose approach works well even without using artifacts removal block, because the artifacts are commonly serious in deblur / non-uniform deblur tasks. If not explicitly penalize the artifacts, purely relying on deconvolution (implemented by fixed point iterations with learned parameters for blur kernel modeling) can hardly work well. Moreover, the paper did not provide accuracy results of HQS-based methods. However, maybe I miss something / mis-understand on something. I stand on the borderline and is OK with both rejection / acceptance.

**Reviewer 1KQg:**
So, my reading of the paper is a bit different. I think this paper's a little different from HQS methods where the iterations are explicitly minimizing a deblurring objective (i.e., estimate * kernel = observation) with a denoising network as the "prior".

The proposed method is just using fixed point iterations as the 'motivation' for it's architecture design: what they're proposing is essentially just a new kind of network architecture that's trained end-to-end (with the benefit that it's very light-weight and efficient). In the literature, we've seen that direct CNNs trained end-to-end with a reconstruction (and other) losses can achieve state-of-the-art performance.

This paper's method is sort of in that "family" of methods, and so perhaps it's not surprising it can do well without an explicit artifact removal step. In some sense, the fact that the underlying fixed point iterations didn't have an artifact penalty doesn't matter because the kernels are all being learned and with the objective of maximizing the final output's quality. What's novel about the paper is that compared to prior end-to-end networks, this one's architecture is efficient and has this nice motivation from classical deblurring methods.

**Reviewer SpFs**: Thanks, I agree that the proposed network is lightweight and efficient. Regarding the comment "fixed point iterations didn't have an artifact penalty doesn't matter because the kernels are all being learned and with the objective of maximizing the final output's quality", my understanding is as follows. Even the blur kernel is accurately estimated, the deconvolution step will still introduce artifacts (please consider the non-blind image deblurring task). Since this proposed network may not have enough capacity to remove artifacts, to minimize the reconstruction loss, the network may be trained to be conservative and insufficiently remove blurs (because sufficiently removing blur may introduce artifacts and increases the reconstruction error). I maybe wrong, but these are my personal concern.

**AC**: I agree that we do not understand perfectly why the approach works and that there is probably a regularization effect due to the fact that the fixed point iterations do not necessarily converge to a fixed point (in the same way as early stopping can induce regularization in optimization methods). The exact nature of the regularization is unknown, but it does not necessarily prevent it from working since all parameters are learned end to end. I will encourage the authors to think about it and include a discussion in the paper.

Decision: Accept (Poster)