Gaussian Kernel Mixture Network for Single Image Defocus Deblurring
Authors: Yuhui Quan
Zicong Wu
Hui Ji
Venue: NeurIPS 2021 Poster (2021)
Keywords: Defocus Deblurring
Deep Learning
Image Recovery
Unrolling
Attention

Abstract: Defocus blur is one kind of blur effects often seen in images, which is challenging to remove due to its spatially variant amount. This paper presents  an end-to-end deep learning approach for removing defocus blur from a single image, so as to have an all-in-focus image for consequent vision tasks. First, a  pixel-wise Gaussian kernel mixture (GKM) model is  proposed for representing spatially variant defocus blur kernels in an efficient linear parametric form, with higher accuracy than existing models. Then, a deep neural network called GKMNet is developed by unrolling a fixed-point iteration of the GKM-based deblurring.  The GKMNet is built on a lightweight scale-recurrent architecture, with a scale-recurrent attention module for estimating the mixing coefficients in GKM for defocus deblurring. Extensive experiments show that the GKMNet not only noticeably outperforms existing defocus deblurring methods, but also has its advantages in terms of model complexity and computational efficiency.

Gaussian Kernel Mixture Network for Single Image Defocus Deblurring
Yuhui Quan', Zicong Wu" School of Computer Science and Engineering South China University of Technology
Hui Ji Department of Mathematics National University of Singapore
Abstract
Defocus blur is one kind of blur effects often seen in images, which is challenging to remove due to its spatially variant amount. This paper presents an end-to-end deep learning approach for removing defocus blur from single image, as have an all-in-focus image for consequent vision tasks. First, a pixel-wise Gaussian kernel mixture (GKM) model is proposed for representing spatially variant defocus blur kernels an efficient linear parametric form, with higher accuracy than existing models. Then, deep neural network called GKMNet is developed by unrolling fixed-point iteration of the GKM-based deblurring. The GKMNet is built lightweight scale-recurrent architecture, with scale-recurrent attention module for estimating the mixing coefficients Extensive experiments show that the GKMNe not only noticeably outperforms existing defocus deblurring methods, but also has its advantages in terms of model complexity and computational efficiency.
GKM for defocus deblurring.
Introduction
The appearance sharpness of an object in an image taken by camera is determined by the scene distance of the object to the focal plane the camera. object will have the sharpest appearance when it is on the focal plane, i.e., the object is in focus. The area around the focal plane where objects appear to be in focus is called the depth field (DoF). When an object is away from the DoF, will appear blurry. The further is object away from the DoF, the more blurry it appears. Such phenomenon is called defocus blur or out-of-focus blur. Defocus blur effects will be prominent in image with shallow DoF, e.g. images captured with large aperture. This paper concerns the problem of single image defocus deblurring (SIDD) which is about reconstructing an all-in-focus image from defocused image (i.e. image with defocused regions). SIDD is of practical values to many applications in machine vision, e.g. photo refocusing, object recognition, and many others I
Consider defocused image y, which relates to its all-in-focus counterpart by
y B +
where denotes the measurement noise, and B is linear operator defined by
B [Box)Jm,n]:
ΣΣ b,,n j i, 5].
Each pixel at location (m, r2] is associated with a defocus kernel dm.ns also referred point spread function (PSF), which determined by the distance to the focal plane. Often, these pixel-wise PSFs approximated by Gaussian kernels 2 information on the scene depth, these pixel-wise wise PSFs are unknown. Therefore, SIDD is a challenging nonlinear inverse problem which needs to estimate both B and from (1)
5 6 disk kernels 8). Without supplementary
"mmail: Sechhuua@@scut.ddii (Y. Quan): eszicongwu mai..scut.ed..coo (Z. Wu); matj Siiihh@uus nus.edu..g (H. Ji).
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

1.1 Discussion on Existing Work
5 take a two-stage approach which (i) estimates a dense defocus map to derive the operator B and then (ii) recovers the image z by using nonblind image deconvolution solve (I) with the estimated B. Generally, such two-stage approach has a long pipeline with many modules, and the estimation error in one module will be magnified in the consequent modules. For instance, the defocus amount only can be estimated on subset of image pixels such as edge points. Then a dense defocus map needs to be constructed by propagating these few estimations to all pixels. can be seen that any error in the sparse defocus map will result in erroneous PSFs, and unfortunately deconvolution is very sensitive to the errors in PSFs [TO] [T] As result, the two-stage approach does not perform well in practice. Also, the computational cost the second stage is high for a non-uniform blurring operator B, as the inversion process regrading B, which is often called many times in nonblind image deconvolution, cannot be efficiently computed via fast Fourier transform (FFT).
Most existing methods (e.g. 2
Deep learning has become one prominent tool for solving a wide range of image restoration problems. In comparison to the rapid progress of DNNs for spatially-varying motion deblurring (e.g- [12 13 14
15 16 17), there have been few works on studying DNN-based approaches to defocus deblurring. One might directly adapt an existing motion deblurring DNN for SIDD. However, the kernels of defocus blur are very different from those of motion blur, egg. roughly isotropic support vs highly curvy support. Also, the spatial variation in defocus blur differs much from that in motion blur, e.g. transparency effects of moving objects in motion blur do not exist for out-of-focus objects in defocus blur. As result, it is sub-optimal to directly call motion deblurring method for SIDD.
Another straightforward implementation of introducing deep learning to SIDD is to replace the defocus map estimator in traditional two-stage approach by DNN-based method (e.g. [6].. Such an implementation still suffers from the issues existing in traditional methods, Lee. inaccurate estimation of B from a non-perfect defocus map and high computational cost for deblurring with a spatially- variant blurring operator. To fully exploit the potential of deep learning for SIDD, one needs to specifically design an end-to-end DNN that directly predicts the all-in-foctt image from the defocused one. Recently Abuolaim and Brown [1]] developed an end-to-end DNN for constructing an all-in- focus image from pair of images containing two sub-aperture views of the same scene. They also adapted their DNN to SIDD, but saw a significant performance decrease.
1.2 Main Idea
This paper aims at developing an end-to-end DNN for SIDD with better performance than existing methods, which is based on the following two derivations.
GKM-based model for defocus blurring Since defocus PSFs show strong isotropy and smooth- ness, we propose to model the kernels (bm,,)m,, by Gaussian kernel mixture (GKM):
bm,
Balm, nlg((k).
k=1
where g(r) denotes the 2D Gaussian kernel of variance o², and Bg denotes the matrix of mixing coefficients for the k-th Gaussian kernel in the GKM. As the GKM can fit well most isotropic kernels, Eq. (3) is a more accurate model for real defocus PSFs than the often-used single Gaussian/disk form; see supplementary materials for a demonstration.
Remark 1. The GKM degenerates to the single Gaussian form when only one mixing coefficient is 1 and the others are 0 for every location [m,n). In general cases, the weighted summation of Gaussian kernels can represent non- -an-Gassiian kernels, and thus the GKM can express a wider family of defocus PSFs than the single Gaussian form. There is also another work [9] that models defocus PSFs beyond single Gaussian kernels. uses the generalized Gaussian function [9] where the parameters to be estimated are wrapped in a complex nonlinear function. In comparison, our GKM model is linear with pre-defined (ak has which facilitates the estimation on its parameter (Ba]
nuncinnarfunction.
We can rewrite 1 as
K
K Σ
y Σ n, n 2 (bm, 2 x)
-EX
Σ
5mn c (100mm)ngg(())))
Br o (g(ox)
k=

---

where m. denotes the Dirac delta centered at location [m, ni,, and @, O denote the operations of 2D convolution and entry-wise multiplication. respectively. Then we have the GKM-based model for defocus blurring:
K Bx @ (g(ok) @ @x).
(4)
Fixed-point iteration unrolling Recall that the blurring operator B is about keeping the low- frequency components and attenuating high-frequency ones of an image. Let I denote the identity mapping. The mapping I - B is then about attenuating the low-frequency components and keeping the high-frequency ones. Neglecting the noise and rewriting (I) by
a - y + (I B) o x;
(5)
we have then a fixed-point iteration for solving defocus deblurring, which is given by
(++++) = ((x y a+ (t) - (t y+ (t) (g(a) for t = 1....... (6)
Note that the fixed- fixed-point iteration above will be convergent if I is contractive mapping, or equivalently the eigenvalues of B fall in (0, 1). which holds true when the defocus blurring is uniform with normalized Gaussian kernel, ie., the scene depths are constant the view.
Define 0, = 0 and g(r)) = 6, so that clear regions can be modeled by setting 31 = 1 and zeroing BR for 1. Let 71 ,3 and Y Bg for k 1. The iteration (6) can be expressed
K
x (1+1) (0) + Σ o (g(ox) @ x (1),, for t = -1,2....
In short, based on the GKM model of defocus PSFs, we can unroll fixed- iteration to solve ( with learnable coefficient matrices "1,-. Yk- The motivation of unrolling a fixed fxee-pooint iteration, instead of other iterative schemes such as gradient descent [19] and half quadratic splitting (HQS) 20)
to involve the forward operator B only, without introducing the transpose B and the pseudo- inverse B!.
Remark 2. Our approach is sort of in the category of optimization unrolling, a widely-used method- ology of designing DNNs for solving inverse problems. The key is to choose appropriate iteration scheme that fits the problem well. Most existing optimization unrolling based image deblurring 27 (71) consider uniform blurring, where the matrix B can be represented by a convolution. The iterative schemes they adopt such as HQS, usually involve an inversion process for B, which can be efficiently computed using FFT when B is convolution oper- ator. In our case, B is a spatially-varying blurring operator which does not have a computationally efficient inversion process. The proposed fixed-point iteration unrolling enables us avoid such inversion process in the DNN and use the forward operator only.
methods (e.g. [22] 23 20 24
25
26
The matrices Y1. 7KK of mixing coefficients can be intezpreted the attention maps associated to the feature maps generated by diffcrent Gaussian kernels. Thus, we construct a DNN with attention modules and long skip connections to utilize 77) for SIDD. In addition, we take multi-scale scheme to implement the unrolling: at each iteration the DNN predicts the all-in-focus image at current seale and up-samples it for the calculation of the next iteration, with weight sharing used across scales. This leads to a scale-recurrent attentive DNN with a lightweight implementation.
1.3 Main Contributions
In comparison to existing two-stage or dual-view-based methods, this paper is among the first ones to present an end-to-end DNN for SIDD. See below for the summary of our technical contributions:
A new and efficient parametric model based on GKM for defocus blur kernels, which fits real-world data better than existing models and thus leads to better performance in SIDD;
A new formulation of the deblurring process derived from fixed-point iteration so as to have a simple and efficient parameterization of defocus deblurring, which inspires an effective DNN for SIDD with low model complexity and high computational efficiency;
3

---

A scale-recurrent attention mechanism which combines the coarse-to-fine progressive esti- mation and the unrolled deblurring process for better performance.
The experiments show that the proposed DNN brings noticeable improvement over existing ap- proaches to SIDD, in terms of recovery quality, model complexity and computational efficiency.
2 Related Work
Two-stage SIDD Most studies of two-stage methods for SIDD are concentrated on the first stage, Lee. defocus map estimation, while the second stage is often done by calling existing non-blind deconvolution methods (e.g. [28 [I]. Defocus map estimation itself a challenging task. There available for sparse defocus map estimation on edge points or regions. The dense defocus map is then constructed via some propagation method (e.g. Poisson matting (30). Recently, deep learning has been extensively studied for defocus map estimation; see e.g. 7 4 6 3T As discussed in Section L.I the two-stage methods suffer from the sensitivity to estimation errors and high computational costs. In comparison, the proposed end-to-end DNN does not involve the defocus map estimation and thus does not suffer from these issues.
several non-learning-base based methods 2
3
5 8
End-to-end learning for defocus deblurring There are few works learning end-to-end DNN for defocus deblurring. Abuolaim and Brown [8] proposed to train an end-to-end U-Net for predicting all-in-focus image from two view images captured by dual-pixel sensor, and contributed a dataset of quadruples: a defocus blurred image, its all-in-focus counterpart, and two dual-pixel view images. Such dual-pixel-based DNN showed impressive performance. However, its performance significantly decreases when being used for SIDD where only a single image is available for input. The prerequisite on dual-view inputs also limits the wider applicability of this method. In contrast, the proposed DNN is grounded by the defocus blurring model and only assumes a single image as input, thus being applicable to commodity cameras, Very recently, one parallel work to ours on end-to-end learning of SIDD was done by Lee et al. (32. They proposed a DNN that predicts the pixel-wiss wise filters for deblurring the deep defocused features of an image. In comparison, our DNN predicts the blurring filters for deblurring the image in an unrolled fixed-point iteration framework.
DNNs for spatially-varying motion deblurring There have been many studies on deep learn- ing for spatially-variant variant motion deblurring, especially on dynamic scenes with moving objects;
see e.g. [12 14
17). The methods using optical flow (e.g. (36)) or temporal cues of moving objects (e.g. [7 for training are not applicable to SIDD. Many of these
13
16
15
33
35
36
37
methods also adopt multi-scale structures (e.g-
[3] or attention mechanisms (e.g- [17), but with
generic designs which cannot effecttiely exploit the inherent characteristics of defocus blurring, e.g. strong isotropy and high correlation of the shapes of pixel-wise defocus PSFs. In comparison, our DNN is specifically designed for defocus deblurring and thus enjoys better performance in SIDD.
Unrolling-base deep learning for image deblurring Unrolling- based DNNs have been exten- sively studied for non-blind image deblurring where the PSF is given an input; see e.g. 20 24 21) There are also some works 22 23 25 27] on unrolling-
blind image deblurring where the PSF restricted the case of uniform blurring. The proposed unrolling- based DNN is the first one that can handle vatrally-vvrying defocus blurring without given the PSFs, thanks to the proposed GKM- -esdd- defocus blurring model.
is unknown. However, these methods
3 Network Architecture
The DNN we construct for SIDD, named as GKMNet (Gaussian Kernel Mixture Network), is based on the fix- point iteration 7 with multi-scale recurrent fashion. See Figure for the outline.
Given an input image y. we generate its multi-scale versions y1 yr via bi-linear downsampling with factors 27-1 2 respectively. Let Φ; yı and let 12 denote the upsampling operation by factor 2. We consider the following multi-scale extension of 7
K
L++1
yt +
7 @ (g(ok) @ 2x12), fort 1,... T.
(8)

---

The GKMNet employs T recurrent blocks to implement 8). The t-th block takes and 2;12 as input and outputs 1-+1- There are two modules in each block: a Gaussian convolution module (GCM) and scale-recurrent attention module (SRAM). The GCM performs the Gaussian filtering in (8) to provide K feature maps, denoted by (21,1..... 2t.K) ). for the i-th block. The SRAM generates the corresponding coefficient maps (Nt,1s Yt,K) from Vr- The output of the 1-th block is given by
K
yt (r, O 2r,a),
k=1
where the weights {wi,k)e. are for scaling the mixing coefficients {7E predicted by the SRAM within a certain range. The weighted summation with (wr,k) implemented by x convolution. The output of the T'-th block, i.e. æT, is used as the final deblurring result. Note that the GKMNet relates the scales not only by passing the output from one scale to the next, but also by the recurrence mechanism built in the SRAM. Let æt denotes the downsampled ground truth of the same size æt. The training loss is defined by the supervision at all scales:
T
=
Σ C(x,, af"), for some cost function C(:s
(10)
where assign equal weights to the losses at different scales for simplicity.
Input
2
y;
y
Scale Recurrett
SRAM
12
GCM
Gazisin Comsoiution Modsle
Muliiiltation
Drate te
Kauli
Dewncampinng
Upcompinng
2
+
Concaterate
x conN
1 ConV
1 conv
Ground Tuth
OuTQuE Immgg
Loss
12
x
Loss
Figure 1: Diagram proposed GKMNet for SIDD with 3.
Gaussian Convolution Module The GCM is simple group convolutional layer defined by series of pre-designed 2D Gaussian kernels applied to the R, G, channels, respectively. The kernel sizes set to x 1, 3 x 3,5 5, ... .M x M. For kernel size X with 1. we generate two Gaussian kernels with a set to {mm 2) and (m 1) respectively. The Gaussian kernel size 1 x is the Dirac delta kernel. As result, there totally M different kernels. Their parameters are fixed across all scales. The convolution kernels GCM are not learned for two reasons: (i) reduces the model complexity and leads to faster training; and (ii) the learned convolution kernels do not benefit the performance as empirically observed. Note that in addition to H,, we also input yi in parallel for feeding more information the next stage.
Scale-Re Geale-Recurrent Attention Module The SRAM maps an input image ye to the coefficient maps {+r, 1KK jk= The coefficient maps can be viewed as the spatial-channe attention maps associated to the feature maps generated by different Gaussian kernels. We thus draw inspirations from existing attention modules (e.g. [13 [ [7) to have lightweight design SRAM. As illustrated in Figure 2 the SRAM consists of (i) an attentive encoder-decode backbone 38 for feature extraction; and (ii) an attention prediction unit (APU) based on Conv-L.STM (convolutional long short-term memory) 33)
The attentive encoder-decoder backbone sequentially connects a convolutional layer, two encoder blocks, and two decoder blocks. Each encoder/decoder block contains one convolutional layer with
5

---

downsampling//psamppi two convolutional layers with residual connections, and triplet attention block [40]. The triplet attention used improve the NN's spatial and channel adaptivity for better prediction, which can be viewed as tensor X' E RCCHXx defined in the channel-height-wee width space, the triplet attention block generates three parallel attention maps aw E TR dg RC ac R& from the thannel-height channel-width and height-width slices of X respectively, and then applies them for re-calibrating X so to encode spatial-channel dependencies into the features.
ceeentionnin-attntie mechanism in the SRAM. Given a feature
The APU contains two paths. The first path mainly contains Conv-LSTM and applies downsam- sling/ppsampling before/after it for inducing local smoothness on the predicted coefficient maps. The second path contains two convolutional layers, but without nownsamping/upsaapiigg for preserving detailed information for the prediction. Let 71 72 denote the predictions from these two paths. Then the final coefficient map is predicted by y = tanh( ( ④ 22 + ). Such a design motivated from the saneezee-nn-Attentie
[1]] The Conv-L.STM, at the first path in the APU is used for capturing the dependencies among the blur amount at different scales. The hidden states in the Conv-LSTM can capture usefuu information from different scales and benefit the restoration across scales. This enables the SRAM to progressively improve the estimation on the coefficient maps.
(- 33th Masimm
Barkhoss
[R
[Rx
ON ReLU
Paofing
ConsTrarst
wise Multiplicntion
Comsus BN ReLU
Unsampptnn
Triplet Attention
APU
Stement - wise Sunmnation
Figure 2: Diagram of SRAM. All convolutional kernels are of size 3.
21 27)) for image deblurring. the GKMNet does not have any explicit artifact removal block. However, it still performs well empirically, shown in the experiments. The reasons may be as follows. First, the inversion process implemented by the GCM and SRAM is not the classic non-learnable estimator in existing unrolling based DNNs, but the one containing learnable blocks. Thus, the GKMNet can be viewed as some forms of the inversion process with implicit built-in artifact removal. For instance, in the GKM model, the learned combination of Gaussian kernels not only can express the defocus kernels, but also may encode a component to suppress the artifacts. Second, there is probably a regularization effect due the fact that the fixed point iterations do not necessarily converge to a fixed point (similar to early stopping), as well as a regularization effect due to the relaxation of the fixed-point iteration the multi-scale iteration. While the exact nature of the regularizations and built-in artifact removal is unknown, it does not necessarily prevent the GKMNet from working, since the parameters are and-o-endd learned in a supervised manner.
Remark 3. Unlike many existing unrolling-based DNNs (e.g. [23 20 24
25
26
4 Experiments
There are few datasets available for benchmarking defocus deblurring. The most well-known one the DPD dataset 18. provides 500 pairs of images with defocus blur and their corresponding all-in-focus images, as well as the two associated sub-aperture views called dual-pixel images, all in 16-bit color. The training/validaiion/ee
splits in the dataset consist of 350/74/76 samples. Our GKMNet is trained and tested on the DPD dataset without using the dual-pixel images. In addition, we test GKMNet on the RTF test set 7 as well as the images from CUHK-BD dataset 42
Three image quality metrics are used for quantitative evaluation, including two standard metrics: PSNR (Peak Signal to Noise Ratio) and SSIM (Structural Similarity Index Measure) [43), and the LPIPS (Learned Perceptual Image Patch Similarity) 44 for perceptual quality (also used in (18). The GKMNet is also compared to other DNNs in terms of model complexity and computational efficiency. The model complexity is measured by three metrics: number of parameters, number of FLOOPs (floating-pon) operations per second), and model size. The computational efficiency is measured by the average inference time an image of 1680 1120 pixels, tested on an Intel

---

15-9600KF CPU and on an NVIDIA GTX 1080Ti GPU, respectively. For non- non-DNN-based methods, only the inference time on CPU is reported.
Through all experiments, the maximum size of Gaussian kernels in GCM is set to M = 21. The number of scales is set to T 3. The learnable parameters in SRAM are initialized by Xavier [45] The Adam optimizer [46] is used for training with 3000 epochs and batch size 4. The learning rate is fixed 10
4 in the first 2000 epochs and decayed to 10 in the last 1000 epochs. The cost function in (TO) is set to the squared f2 loss in the first 1000 iterations, and alternatively set to the SSIM loss and squared Eg loss every 500 epochs afterwards. Data augmentation is done by random cropping to 256 x 256 pixels. The code of GKMNet is available at httpt::///// //githu com cwZccu/ Cscccc//GWMK/N GKMNet
4.1 Evaluation on DPD Dataset
The methods for comparison include JNB 21. EBDB (5), DMENet 6 and DPDNet [8]. The DPDNet [18] has two versions: DPDNet-D taking two dual-pixel images as input and DPDNet-S only taking a single image input. Both versions included for comparison whose results reproduced by the pre-trained models from their authors. The JNB, EDBD) and DMENet are all two-stage methods focusing on defocus map estimation, with results quoted from [8] In addition, we also include two DNNs of dynamic scene deblurring: SRN 13 and AttNet 17 These two DNNs also adopt a multi-scale architecture. Both DNNs are retrained using the same data as ours.
See Table for the quantitative comparison on deblurring performance. Our GKMNet outperforms all other compared methods. The performance of EBDB, DMENet and JNB is much worse than that of GKMNet. This is probably because the errors in their defocus maps are magnified in the consequent deconvolution process. It indicates the advantage of the end-to-end learning of GKMNet which does not require defocus map estimation. The GKMNNet also outperforms DPDNet-S by a large margin. This is not surprising as DPDNet-S is originally designed for defocus deblurring on two view images, not a single one. Also unsurprisingly. GKMNet outperforms SRN and AttNet, two DNNs designed for dynamic scene deblurring rather than SIDD. Surprisingly, our GKMNet with single-image input even outperforms DPDNet-D which takes dual-pixel images as input. All these results have clearly demonstrated the effectiveness of the proposed GKMNet on SIDD. See supplementary materials for the visualization of coefficient maps generated by the SRAM.
Table also lists the results on model complexity and computational efficiency. In terms of all three metries, GKMNet's s complexity is much lower than that other deep models. For instance, around 1/23 of DPDNet and 1/7 of SRN for number of parameters, and around 1/62 of DPDNet for model size. Such low complexity comes from the compact architecture of GKMNet. Regarding the running time, GKMNet is also much faster than the JNB, EBDB and DEMNet, these methods need to call an iterative deconvolution post-process which is slow. GKMNet is about ten time faster than DPDNet, and its speed is comparable SRN and AttNet. It is noted that GKMNet has a much less training time due to its low complexity. Its training on the DPD dataset takes less than 32 hours, while SRN and AttNet take nearly three days, on an NVDIA GTX 1080Ti GPU.
Table I: Quantitative comparison of different methods on DPD test set.
PSNR
SSIM
LPIPS
#Parameters
#FLOPs
Model Size
Time (Seconds)
Model
(dB)
(Million)
(Billion)
(MegaBytes)
CPU
GPU
JNB 2
23.84
0.715
0.315
843.1
EBDB
5
23.45
0.683
0.336
929.7
DMENet
6
23.41
0.714
0.349
26.94
613.7
DPDNet-S
18
24.34
0.747
0.277
32.25
4042.5
355.3
56.6
0.41
DPDNet-D
18
25.12
0.786
0,223
32.25
4048.6
355.3
56.6
0.41
SRN
13
24.61
0.612
0.265
10.25
3117.6
32.0
58.3
0.032
AttNet 11
25.22
0.781
0.219
6.91
3450.4
39.5
62.8
0.041
GKMNet [Ours]
25.47
0.789
0.219
1.41
603.5
5.7
43.9
0.040
As mentioned in Remark 3. without any explicit artifact/noise removal block in the GKMNet, the robustness of GKMNet is one concern. Thus, we evaluate the robustness to image noise using the DPD dataset with additional corruption by additive Gaussian white noise with standard deviation & 1,3,5 5 respectively. The DPDNet-D, SRN and AttNet are selected for comparison. Both the
7

---

models trained on the original DPD training set and those retrained on the noisier DPD training set are included. The later ones are marked with a "+". See Table 2 for the results. In the presence of additional noise, our GKMNet still outperforms other models for both versions of training data.
Table 2: Quantitative comparison of different methods on noisier DPD test set.
Noise
DPDNet-D
SRN SSIM
AttNet SSIM
GKMNet SSIM
PSNR
SSIM LPIPS
PSNR
LPIPS
PSNR
LPIPS
PSNR
LPIPS
25.08
0.788
0.220
24.58
0.692
0.315
25.18
0.783
0.221
25.43
0.784
0.225
3
24.73
0.746
0.248
24.48
0.668
0.366
24.77
0.749
0.261
25.02
0.758
0.246
5
24.07
0.665
0.441
24.30
0.628
0.411
24.36
0.660
0.347
24.61
0.712
0.299
Noise ô
DPDNet-D+
SRN+
AuNet+
GKMNet+
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
25.09
0.788
0.220
24.61
0.697
0.316
25.18
0.783
0.221
25.48
0.789
0.221
3
24.76
0.748
0.248
24.53
0.672
0.360
24.85
0.753
0.255
25.12
0.760
0.247
5
24.37
0.710
0.379
24.34
0.632
0.399
24.48
0.681
0.332
24.92
0.742
0.273
4.2
Evaluation on RTF Dataset and CUHK-BD Sample Images
The RTF dataset 7 contains 22 pairs of defocused and all-in-focus images. Following 77 in addition to the original images, two noisier versions are generated by adding Gaussian white noise with standard deviation ā and 2.55 respectively. The DPDNet-S, SRN and AttNet are selected for comparison. Note that the training set of RTF is not available. Thus, we directly apply the models that are respectively trained on the original DPD dataset and the noisier DPD dataset. See Table for for the results. The GKMNet again outperforms other DNNs. Since the models art trained and tested with different datasets, such results also demonstrate the generalizability of our GKMNet.
Table 3: Quantitative comparison of different methods on RTF test set.
Noise Ô
DPDNet-S SSIM LPIPS
SRN SSIM
AttNet SSIM
GKMNet SSIM
PSNR
PSNR
LPIPS
PSNR
LPIPS
PSNR
LPIPS
0
23.61
0.597
0.296
23.71
0.617
0.324
25.45
0.802
0.219
25.72
0.811
0.211
1
23,58
0.592
0.332
23,68
0.611
0.311
24.99
0.797
0.243
25.56
0.798
0.207
2.55
23.44
0.576
0.344
23.50
0.591
0.276
23.83
0.727
0.257
25.01
0.761
0.205
Noise ô
DPDNet-S+
SRN+
AutNtt+
GKMNet+
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
0
23.65
0.605
0.290
23.70
0.617
0.325
25.40
0.792
0.225
25.69
0.808
0.212
1
23.59
0.594
0.329
23.65
0.605
0.313
25.06
0.800
0.236
25.60
0.802
0.206
2.55
23.49
0.581
0.341
23.54
0.597
0.281
24.33
0.749
0.250
25.21
0.787
0.210
The CUHK-BD dataset 42 contains 704 defocused images without ground truths. We select some of its images for test. See For the visual inspection on the results. Our GKMNet can successfully restore fine details with less visual artifacts, in comparison to DPDNet-S, SRN and AutNet. More visual comparison can be found in supplementary materials. It is worth mentioning that the defocus PSFs the image synthesized from dual-pixel images are slightly different from the ones in real images 47 48 49] However, our GKMNet still generalizes well.
4.3 Ablation Study
To evaluate the performance contribution of each component in GKMNet, we implement the following baselines from GKMNet for comparison, which are trained in the same way as the original GKMNet. (a) GKMNet(1)): Use only the original image scale in GKMNet with T = (b) w/o ConV-LSTM: Replace the Conv-LSTM block at the APU by a convolutional layer with the same kernel size, and share its weights across scales for relating different scales; (c) SRAM* Use the SRAM as a whole DNN for SIDD, with an 1 x convolutional layer with Sigmoid activation attached for outputting an image; (d) Learned GCM (GKM): The kernels in GCM are initialized by the predefined Gaussian kernels and learned with the DNN. (c) Learned GCM (Rand): The kernels in GCM are randomly
8

---

ARRET
ARRET
ARRET
ARRET
ARRET
Input
DPDNet-S
SRN
AttNet
Ours
Figure Visual comparison of SIDD results of different methods on CUHK-BD sample images.
initialized by Xavier 45 and learned with the DNN. Note that weight sharing is adopted for GCM across scales in (d) and (c) because it leads to performance improvement.
See Tor the results of the baselines, from which we have the following observations. (a) The multi-scale estimation scheme in SRAM leads to significant improvement over the single-scale one, with only few additional model parameters introduced. (b) The improvement from the Conv-LSTM over simple cross-scale veight-sharing convolutions significant, which implies that the Conv-LSTM can effectively exploit the features learned from different scales to guide the deblurring process. (c) Directly using the SRAM as an end-to-end DNN for SIDD leads to much worse results, which justifies the effectiveness of our specific DNN design. (d) Learning the GCM with Gaussian kernel initialization leads to a very minor improvement over using fixed Gaussian kernels in GCM. (e) Learning the GCM with random initialization even yields slightly worse results. Both (d) and (c)

---

indicate the effectiveness of our GKM model for defocus PSFs. See also Figure 4 for the kernels w/o and w/ learning GCM. The kernels learned with (d) are very close original predefined ones.
Table 4: Quantitative comparison of GKMNet and its baselines.
PSNR (dB)
SSIM
LPIPS
#Parameters (Million)
#FLOPs (Billion)
Model Size (MegaBytes)
Time (Seconds)
Model
SRAM
21.98
0.724
0.251
1.41
606.1
5.30
0.040
ASGMNe((1)
23.63
0.723
0.248
1,41
459.7
5.70
0.036
w/o Conv-LSTM
25.00
0.774
0.231
1.11
418.7
4.33
0.036
Learned GCM (Rand)
25.35
0.787
0,223
1.41
603.5
5.70
0.040
Learned GCM (GKM)
25.49
0.790
0.217
1,41
603.5
5.70
0.040
GKMNet
25.46
0.789
0.219
1,41
603.5
5.70
0.040
Predefined Gaussian kernels (fixed in learning)
Learned kemnlls (initialized Gaussian kernels)
2nd
21th
2nd
9th
21th
Figure 4: Visualization of predefined kernels and learned kernels in GCM.
Recall that the fixed point iteration 7 is applied multiple iterations the same scale, while our actual architecture also simultaneously increases the scale. We could further write our architecture as scales and R iterations per scale, where effectiveness of the multi-scale estimation in GKMNet, Table 5 lists the results using different values of S and R. shows that using single or fewer scales with multiple iterations noticeably decreases the performance and increases the inference time. Using an additional iteration with $ = 3 only brings a minor improvement but nearly doubles the complexity and time. Thus, our coarse-to- fine estimation scheme is better implementation for high performance and low computational cost.
S R 1 in the GKMNet. To further verify the
Table Quantitative comparison extended GKMNet models using different values of S and R.
PSNR (dB)
SSIM
LPIPS
Parameters (Million)
FLOPs (Billion)
Model Size (MegaBytes)
Time (Seconds)
Model
S=1, R=3
25.11
0.773
0.237
1.41
1384.2
5.7
0.069
S=2, R=3
25.13
0.789
0.205
4.22
1730.3
17.1
0.122
S=3, R=2
25,53
0.773
0.201
2.81
1211.2
11.3
0.081
S=3, R=1
25.47
0.789
0.219
1.41
603.5
5.7
0.04
Conclusion
Defocus blur often occurs in images and has its characteristies from motion blur. This paper proposed DNN for SIDD with strong motivations from unrolling a fixed-point iteration derived from GKKM-base model of defocus blurring process. Together with scale-recurrent implementation, we developed a lightweight DNN with state-of fatte-of-the of-the--rtt performance. future, we will study the extension other image deblurring problems such as dynamic scene deblurring.
While our approach achieved stateoof-the-att results on existing datasets, its implicit regularizations and built-in artifact removal mechanisms not very clear. In addition, its performance severcly blurred regions is not that satisfactory, with much room for improvement. These issues will also be studied in our further work.
Acknowledgments and Disclosure of Funding
Yuhui Quan would like to acknowledge the support from National Natural Science Foundation of China (Grant No. 61872151) and CCF-Tencent Open Fund 2020. Hui Ji would like to acknowledge
10

---

the support from Singapore MOE Academic Research Fund (AcRF) Tier 1 Research Grant (R- 14-1 000-315-114).
References
[1] Anat Levin, Rob Fergus, Frédo Durand, and William Freeman. Image and depth from a conventional camera with a coded aperture. In ACM Special Interest Group on Computer Graphics, 2007.
[2] Jianping Shi, Li Xu, and Jiaya Jia. Just noticeable defocus blur detection and estimation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 657-665, 2015.
[3] Guodong Xu, Yuhui Quan, and Hui Ji. Estimating defocus blur via rank of local patches. In IEEE International Conference on Computer Vision, pages 5371-5379, 2017.
[4] Jinsun Park, Yu-Wing Tai, Donghyeon Cho, and In So Kweon. A unified approach of multi-scale deep and hand-crafted features for defocus estimation. In IEFE Conference on Computer Vision and Pattern Recognition, pages 2760-2769, 2017.
[5] Ali Karaali and Claudio Jung. Edge-based based defocus blur estimation with adaptive scale selection. IEEE Transactions on Image Processing. 27:1126 1137, 2018.
[6] Junyong Lee, Sungkil Lee. Sunghyun Cho, and Seungyong Lee. Deep defocus map estimation using domain adaptation. In IEEE Conference Computer Vision and Pattern Recognition, pages 12214-12222, 2019.
[7] Laurent D' Andrès, Jordi Salvador, Axel Kochale, and Sabine Süsstrunk. Non-parametric blur map regression for depth of field extension. IEEE Transactions Image Processing, 25:1660-1673, 2016.
[8] Sunghyun Cho and Seungyong Lee. Convergence analysis of map based blur kernel estimation. In IEEE International Conference Computer Vision, pages 4818 4826, 2017.
[9] Yu-Qi Qi Liu, Xin Du, Hui-Liang Shen, and Shu-Jic Chen. Estimating generalized gaussian blur kernels for out-of-focur image deblurring. IEEE Transactions Circuits and Systems for Video Technology, 31:829-843, 2020.
[10] Hui Ji and Kang Wang. Robust image deblurring with an inaccurate blur kemnl. IEEE Trunsactions on Image Processing. 1124):16241164, 2011.
[11] Yuesong Nan and Hui Ji. Deep learning for handling kemel/mmoee uncertainty in image deconvolution. In IEEE Conference on Computer Vision and Pattern Recognàtion, pages 2388-2397, 2020.
[12] Seungjun Nah, Tae Kim, and Kyoung Mu Lee. Deep multi- scale convolutional neural network for dynamic scene deblurring. In JEEE Conference on Computer Vision and Pattenn Recognition, pages 257-265, 2017.
[13] Xin Tao, Hongyun Gao, Yi Wang, Xiaoyong Shen, Jue Wang, and Jiaya Jia. Scale-recurrent network for deep image deblurring. In IEEE Conference on Computer Vision and Pattern Recognition, pages 8174-8182, 2018.
[14] Jiawei Zhang, Jinshan Pan, Jimmy Ren, Yibing Song, Linchao Bao, Rynson WH Lau. and Ming-Hsuan Yang. Dynamic scene deblurring using spatially variant recurrent neural networks. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2521-2522 2018.
[15] Hongyun Gao, Xin Tao, Xiaoyong Shen, and Jiaya Jia. Dynamic scene deblurring with parameter selective sharing and nested skip connections. In IEEE Conference on Computer Vision and Puttern Recognition, pages 3356, 2019.
[16] Hongguang Zhang, Yuchao Dai, Hongdong Li, and Piotr Koniusz. Deep stacked hicrarchical multi-patc) network for image deblurring. In IEEE Conference on Computer Vision and Pattern Recognition, pages 5978-5986, 2019.
[17] Yong Xu. Ye Zhu, Yuhui Quan, and Hui Ji. Attentive deep network for blind motion deblurring dynamic scenes. Computer Vision and Image Understanding, 205:103169, 2021.
[18] Ahdullah Abuolaim and Michael S. Brown. Defocus deblurring using dual-pixel data. In European Conference on Computer Vision, 2020.
[19] Dong Gong. Zhen Zhang, Qinfeng Shi, Anton van den Hengel, Chunhua Shen, and Yanning Zhang Learning deep gradient descent optimization for image deconvolution. IEEE Transactions on Neural Networks and Learning Systems, 2020.

---

[20] Jiawei Zhang, Jinshan Pan, Wei-Sheng Lai, Ryason WH Lau, and Ming-Hisue Yang. Learning fully convolutional networks for iterative non-blind deconvolution. In IEEE Conference on Computer Vision and Pattern Recognition. pages 3817-3825, 2017.
[21] Yuesong Nan, Yuhui Quan, and Hui Ji. Variational-em: based deep learning for noise-blind image deblurring. In IEEE/CVF Conference on Computer Vision and Patiern Recognition, pages
2020.
[22] Christian Schuler, Michael Hirsch, Stefan Harmeling, and Bernhard Schölkopf. Learning to deblur. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(():1439-15)) 1451, 2015.
[23] Wangmeng Zuo, Dongwei Ren, David Zhang, Shuhang Gu, and Lei Zhang. Learning iteration- wise generalized shrinkage-thresholding operators for blind deconvolution. IEEE Transactions on Image Processing, 25(4):1751-1766 2016.
[24] Weisheng Dong, Peiyao Wang. Wotao Yin, and Guangming Shi. Denoising prior driven deep neural ketwork for image restoration, IEEE Trunsactions on Puttern Analysis and Machine Intelligence, 41(10):2305-2318 2019.
[25] Yuelong Li, Mohammad Tofighi, Vishal Monga, and Yonina C Eldar. An algorithm unrolling approach to deep image deblurring. In JEEE International Conference on Acousticx, Speech and Signal Processing, pages 7675-7679, IEEE, 2019.
[26] Dipan Pal Raied Aljadaany and Marios Savvides. Douglas-rachford networks: Learning both the image prior and data fidelity terms for blind image deconvolution. In JEEE Conferencee on Computer Vision and Puttern Recognition, pages 01245-10244, 2019.
[27] Yuelong Li, Mohammad Tofighi, Junyi Geng, Vishal Monga, and Yonina C Eldar. Efficient and interpretable deep blind image deblurring via algorithm unrolling. IEEE Transactions on Computational Imaging. 6:666-681 (166-681,25 2020.
[28] D. Fish, A. M. Brinicombe, E.R. Pike, and Walker. Blind deconvolution by means of the richardson- lucy algorithm. Journal of The Optical Society of America A A-optics Image Science and Vision, 12:58- 65, 1995.
[29] Dilip Krishnan and Rob Fergus. Fast image deconvolution using hyper-laplacian priors. In Advances in neural information processing systems, 2009.
[30] Jian Sun, Jiaya Jia, Chi-Keung Tang, and Heung-Yeuur Shum. Poisson matting. ACM Special Interest Group on Computer Gruphics, pages 315-321. 2004.
[31] Saeed Anwar, Zeeshan Hayder, and Fatih Porikli. Deblur and deep depth from single defocus image. Machine Vision and Applications, 32: 12:1-13, 2021.
[32] Jaesung Rin Sunghyun Cho Junyong Lee, Hycongseok Son and Seungyong Lee. Iterative filter adaptive network for single image defocus deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 2034-2042, 2021.
[33] Kuldeep Purohit and AN Rajagopalan. Region-adaptive dense network for efficient motion deblurring. In National Conference of the American Association for Artificial Intelligence, volume 34, pages 11882- 11889, 2020.
[34] Maitreya Suin, Kuldeep Purohit, and AN Rajagopalan. Spatially-attnntive patch- patch-tierarchical network for adaptive motion deblurring. In IEEE Conference Computer Vision and Pattern Recognition, pages 3606 3615, 2020.
[35] Kaihao Zhang, Wenhan Luo. Yiran Zhong, Lin Ma. Bjorn Steager. Wei Liu, and Hongdong Li. Deblurring by realistic blurring. In IEEE Conference Computer Vision and Pattern Recognition, pages 2737-2746. 2020.
[36] Yuan Yuan, Wei Su, and Dandan Ma. Efficient dynamic scene deblurring using spatially variant deconvo- lution network with optical flow guided training. In IEEE Conference Computer Vision and Pattern Recognition, pages 3555-3564, 2020.
[37] Dongwon Park, Dong Un Kang. Jisoo Kim. and Se Chun. Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training. In European Conference on Computer Vision, 2020.
[38] Xiao- Mao, Chunhua Shen, and Yubin Yang. Image restoration using very deep convolutional encoder- decoder networks with symmetric skip connections. In Advances in neural information processing systems. 2016.
12

---

[39] Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang chun Woo. Convolu- tional Istm network: A machine learning approach for precipitation nowcasting. In Advances in neural information processing systems, 2015.
[40] Diganta Misra, Trikay Nalamada, Ajay Uppili Arasanipalai. and Qibin Hou. Rotate to attend: Convolutional triplet attention module. In IEEE Winter Conference on Applications of Computer Vision, pages 3139-3148, 2021.
[41] Zilong Zhong, Zhong Qiu Lin, Rene Bidart, Xiaodan Hu, Ibrahim Ben Daya, Jonathan Li, and Alexander Wong, Squeeze-ang and-attention networks for semantic segmentation. In JEEE Conference on Computer Vision and Pattern Recognition, pages 13062 13071, 2020.
[42] Jianping Shi, Li Xu, and Jiaya Jia. Discriminative blur detection features. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2965-2972, 2014.
[43] Zhou Wang, Alan Bovik, Hamid Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing. 13:600 612. 2004.
[44] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Conference on Computer Vision and Pattern Recognition, pages 586 595, 2018.
[45] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In International Conference Artificial Intelligence and Statistics, pages 249-256 JMLR Workshop and Conference Proceedings, 2010.
[46] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zach Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation pytorch. 2017.
[47] Sameer Ansari Rahul Garg, Neal Wadhwa and Jonathan T Barron. Learning single camera depth estimation using dual- duxl-pixels. In IEEE Conference on Computer Vision and Pattern Recognition, pages 7628-768777 7637, 2019.
[48] Mahmoud Afifi Abhijith Punnappurath, Abdullah Abuolaim and Michael S Brown. Modeling defocus- disparity in dual-pixel sensors. IEEE International Conference on Computational Photograpky, pages 1-12, 2020.
[49] Richard Hartley Miaomiao Liu Hongguang Zhang Liyuan Pan, Shah Chowdhury and Hongdong Li. Dual pixel exploration: Simultaneous depth cstimation and image restoration. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4340-4349, 2021.
13


Reviews:
[{"review_id": "tZ2Un78tS5P", "reviewer": "Reviewer_uKyg", "summary": "In this manuscript, the authors address a single image-based defocus deblurring problem. Based on a pixel-wise Gaussian scale mixture (GSM) model for modelling defocus blur effects, this manuscript proposes an end-to-end deep learning-based approach with a strong motivation from unrolling a fixed-point iteration for deblurring.  \n\nExperiments demonstrate that the proposed method achieves competitive results compared to the state-of-the-art methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The manuscript is well written, and the exposition is clear. The results from the paper are impressive, and the detailed illustrations in the supplementary material are helpful. \n\nThe theoretical description of the Gaussian scale mixture model and the formulation of the Unrolling fixed-point iteration are easy to follow and well explained in the paper. The GSM-based parametric model for defocus PSF catered specifically to non-uniform blurred images is a good contribution.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1**. The model is designed for defocus blur with depth-based blur kernels that Gaussian kernels can well approximate. However, the method is tested and trained on the dual-pixel dataset mainly. The PSF of a dual pixel blurred image, usually not approximated by Gaussian kernels, is slightly different from the common defocus blurred image [1,2,3]. \n\nIt would be nice if the authors analysed the connection between the two kinds of blur, which are defocusing blur and the `dual-pixel’ blur. \n\nThe proposed method still achieves competitive performance compared to the STOA, even using the `dual-pixel’ blurred images (the single image (Ic) rather than the dual-pixel pair) in the DPD dataset. Why the model has a good generalisation ability? It would be nice if several sentences were analysing the reasons.\n\nIt is good to show that the authors are aware of what model is used in dual-pixel pairs and what is for defocus blur.\n\n[1] Garg, Rahul, et al. \"Learning single camera depth estimation using dual-pixels.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.\n[2] Pan, Liyuan, et al. \"Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n[3] Punnappurath, Abhijith, et al. \"Modeling defocus-disparity in dual-pixel sensors.\" 2020 IEEE International Conference on Computational Photography (ICCP). IEEE, 2020.\n\n2. The reason for using SRN and AttNet as the baseline.\n\nThese two methods targeting at motion deblurring while the manuscript is for defocus blur. If the author aims to tell model designed for motion blur is not suit for defocus blur, we suggest the authors add a new baseline here, any of the STOA motion deblurring methods.\n\nAlso, SRN has very similar network architectures to the proposed method. Besides SRN, inserting the GCM module to SRN or attention module to SRN would be a fairer baseline. \n\n3. Enhance figure 3. It is hard to figure out any information via this figure.\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3h"}, "extra_text": {}}, {"review_id": "lC4PIAMJ49l", "reviewer": "Reviewer_e2uj", "summary": "The authors propose a supervised end-to-end model architecture for single-image defocus deblurring (SIDD). Therein, unknown point spread functions are modeled by means of Gaussian scale mixtures, and the deblurring operation itself is reformulated in terms of an unrolled fixed point equation. The unrolled scheme is finally combined with a progressive coarse-to-fine estimation.\n\nIn the introduction, the authors address a typical challenge in SIDD. Namely, both the defocus operator $B$ and the sought all-in-focus image $x$ need to be reconstructed. To that end, existing methods take a two-step approach where the defocus operator is estimated and fixed first, after which the image reconstruction takes place. The two-step approach comes with a few drawbacks: On the one hand, common estimation procedures for the defocus operator are error-prone and errors from the first stage tend to be propagated or even amplified in the second stage. Moreover, evaluations of the defocus operator in iterative reconstruction schemes can involve high numerical expense. The authors claim that the proposed method overcomes the above-mentioned challenges.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "From my point of view, this is a very interesting paper. To the best of my knowledge, it is technically correct, the proposed architecture is novel and the reported numerical experiments show state-of-the-art results.\n\nI only have a few points on which the authors could maybe comment:\n\n- I do not understand why, without further assumptions on $B$, the fixed point iteration **(5)** should converge to an optimal solution. Therefore, although subsequent empirical results show that the derived method works very well, the connection between the motivation and the final method is from my point of view not too strong.\n- What is the difference between $\\varepsilon$ in **(1)** and $n$ in **(5)**? Why is the measurement noise no further mentioned?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "TO91hFsgBh4", "reviewer": "Reviewer_egt5", "summary": "This paper proposes a defocus deblurring method based on an end-to-end trainable network. The proposed method is motivated by the fixed-point iteration and develop a lightweight scale-recurrent architecture with an attention module to achieve pixel-wise GSM-based deblurring under the unrolling framework. Experimental results demonstrate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a defocus deblurring method based on an end-to-end trainable network which is motivated by the fixed-point iteration. It embeds the pixel-wise GSM into an end-to-end trainable network with a lightweight scale-recurrent architecture and an attention module to solve defocus blur. The performance of the proposed method is better than the evaluated method. \n\nAs the defocus blur is spatially variant and some focused regions are clear, it is not clear whether using pixel-wise GSM can model the property of the clear regions. That is, it is not clear whether the proposed method affects the regions that do not contain blur.\n\nThere are some defocus deblurring methods by modifying the Gaussian blur kernels to approximate the defocus blur.  For example, the work by [Liu et al. Estimating Generalized Gaussian Blur Kernels for Out-of-Focus Image Deblurring, TCSVT 2021] shows that commonly used Gaussian kernel does not work well for the defocus blur estimation. It would be better to discuss and compare with the method.\n\nThe proposed method is motivated by the fixed-point iteration. However, there are lots of deblurring methods based on the conventional optimization methods, e.g., Aljadaany et al., Douglas-Rachford Networks: Learning Both the Image Prior and Data Fidelity Terms for Blind Image Deconvolution, CVPR 2019. The authors should clarify the major differences and compare the proposed method with them.\n\nSome important discussions of the effect of the proposed method are missing. Since the defocus blur is spatially variant and some focused regions are clear, one commonly used baseline is to first detect blur regions and then use the blur regions (e.g., concatenation of detected blur maps and the input images) to guide the estimations of the clear images in a deep neural network. It would be better to discuss and compare with this baseline method.\n\nWhat are the limitations of the proposed method? ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does not provide an analysis of the limitations ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses a deep learning model which is trained  end-to-end for single image defocus deblurring. In general, the paper is well written and contains interesting contribution. The authors made their efforts to respond to the concerns raised by reviewers. The paper has both positive and negative sides. Reviews on the negative side remained during the discussion period. All the reviewers stood by their decision. Unfortunately one reviewer did not responded even to a few reminders, so the paper has only three reviews. 
Decision: Accept (Poster)