Gaussian Kernel Mixture Network for Single Image Defocus Deblurring

Abstract: Defocus blur is one kind of blur effects often seen in images, which is challenging to remove due to its spatially variant amount. This paper presents  an end-to-end deep learning approach for removing defocus blur from a single image, so as to have an all-in-focus image for consequent vision tasks. First, a  pixel-wise Gaussian kernel mixture (GKM) model is  proposed for representing spatially variant defocus blur kernels in an efficient linear parametric form, with higher accuracy than existing models. Then, a deep neural network called GKMNet is developed by unrolling a fixed-point iteration of the GKM-based deblurring.  The GKMNet is built on a lightweight scale-recurrent architecture, with a scale-recurrent attention module for estimating the mixing coefficients in GKM for defocus deblurring. Extensive experiments show that the GKMNet not only noticeably outperforms existing defocus deblurring methods, but also has its advantages in terms of model complexity and computational efficiency.


Reviews and author responses:

 We sincerely thank the reviewer for his/her constructive comments.

------------------------------------------------------------------------------
**Regarding whether the GSM can model the property of the clear regions**

**Yes. Indeed our method can model the property of clear regions.** Recall that the proposed GSM model essentially models a defocus blur kernel by a linear combination of a set of Gaussian kernels with different variances. Such a set includes a Dirac Delta (1 at origin and 0 otherwise, i.e., a Gaussian kernel with zero s.t.d.) in our implementation. As convolving an image with Dirac Delta keeps the image unchanged, the Dirac Delta indeed corresponds to focused regions. Then, our method can model clear regions by zeroing all linear combination coefficients in the GSM model except the one associated to the Dirac delta kernel. Then the corresponding defocus kernels of the clear regions become a Dirac delta. As a result, our method can preserve clear regions.

------------------------------------------------------------------------------
**Regarding the reference [TCSVT 2021 March]**

Thanks for such a new reference. We will include it in revision. As the reviewer pointed out, this reference shows that the Gaussian kernel does not model defocus blur well, which is exactly why we propose a GSM-based model for modeling defocus blur kernels.

**(i) Differences on models** This reference used a generalized Gaussian function to model the defocus PSF, while ours uses the linear combination of Gaussian kernels with different widths to model the defocus kernel. The generalized Gaussian kernel has a nonlinear parametric form, which is estimated by a non-learnable scheme. In contrast, our model has a linear parametric form, which is easy to learn via deep learning.

**(ii) Comparison** The reference is about how to recover the image whose defocus blur is uniform (only one single PSF for the whole image). Our work considers the case whose defocus is non-uniform (i.e., every pixel has its own PSF). Extending a method designed for uniform blurring to handling non-uniform blurring is highly non-trivial. Thus, we did not use the reference for experimental comparison.

------------------------------------------------------------------------------
**Regarding the differences from conventional optimization-based methods and the comparison to the suggested reference [CVPR-19]**

**The difference between our method and existing optimization unrolling based methods for image deblurring is quite big.** Most existing optimization unrolling based methods for image deblurring focus on uniform blurring, where the matrix $B$ can be represented by a convolution. The iterative schemes they adopted such as half-quadratic splitting, proximal method or ADMM, usually need to have an inversion process for the matrix $B$, which can be efficiently computed using FFT when the matrix $B$ is a convolution operator. In our case, the matrix $B$ is a spatially-varying blurring operator which does not have a computationally efficient inversion process. Thus, we proposed a fixed-point iteration to avoid the direct inversion computation of $B$. In addition, most existing optimization unrolling based methods assume the blur kernel is given (i,e. nonblind deblurring), while the defocus deblurring in our method assumes the blurring operator is unknown (i.e. blind deblurring). 

**About the comparison to the suggested reference.** We will add it for discussion. The reference is for blind motion deblurring. It unrolls an iteration scheme for uniform deblurring, which is then adapted to non-uniform deblurring by learning a deep network for the fidelity term. In contrast to our approach, such a network learns general deblurring processes without exploiting any special characteristic of defocus blurring. A direct comparison to the reference is challenging, as its code/model is not available online.

Indeed, the importance of exploiting the characteristics of defocus blurring has been demonstrated by the superior performance of our method over AttNet and SRN, two deep networks for motion deblurring, in the experiments of our main paper. To further demonstrate such an importance, we run a comparison to MPRNet [a], a recent SOTA motion deblurring method (rank 2 on GoPro motion deblurring dataset). Its model is retrained using the same data used by our model. It can be seen that MPRNet performs noticeably worse than our method. This showed that motion blurring has quite different characteristics from defocus blurring. One needs to design specific approaches to effectively handle defocus blurring by utilizing its special blurring characteristics.

| Method | PSNR(dB)$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | FLOPs(Billion)$\downarrow$ |
| :------: | :-----:| :-----:| :-----:| :---------:|
| MPRNet | 25.02 | 0.772 | 0.225 | 21835.5|
| Ours   | **25.47** | **0.789** | **0.219** | **603.5**|

[a] Multi-stage progressive image restoration. CVPR 2021.


------------------------------------------------------------------------------
**Regarding “first detect blur regions and then use the blur regions to guide the estimations of the clear images in a deep neural network”**

We briefly discussed the issues of such a two-stage method in Related Work. One main issue is that the accurate detection of clear regions and blurred regions with different blurring degrees itself is a challenging task, which is under active research in the category of defocus map estimation. The unavoidable errors in region detection will be propagated to the stage of deblurring. Our approach avoids such a challenging process. 

There is no existing study using the suggested baseline. Thus, we conducted an additional experiment as suggested, where the defocus map output by [b] is concatenated with the blurry image as the input of the DPDNet. The DPDNet is retrained based on the pairs of truth images and images outputs by [b]. The results on the DPD dataset are listed as follows. Not surprisingly, such results are noticeably inferior to ours.

Indeed, using blur detection result as additional input might bring possible further improvement to a deep network, but the key still remains how to have an effective network design that can exploit the characteristics of defocus blur to recover defocus blurred regions with different PSFs. This is what we addressed in our paper, and missing such a baseline does not weaken the value of our work.

| Method | PSNR(dB)$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |FLOPs(Billion)$\downarrow$ |
| :------: | :-----: | :-----: | :-----: | :-----: |
| 2-Stage | 24.54 | 0.753 | 0.268 | 6782.9 |
| Ours   | **25.47** | **0.789** | **0.219** | **603.5** |

[b] Deep Defocus Map Estimation using Domain Adaptation. CVPR 2019.

------------------------------------------------------------------------------
**Regarding analysis of the limitations**

While our approach has achieved SOTA results on existing datasets, the performance on some severely blurred images remains unsatisfactory, as most information is lost by severe blurring. Building a dataset that covers more severely blurred images and developing effective methods to recover such images remain to be further studied. We will add this discussion in Conclusion.

 We sincerely thank the reviewer for his/her constructive comments.

-------------------------------------------------------------------------------------------------------------------
**Regarding connection between the motivation of final method**

**(i) Motivation of using fixed-point iteration** Our approach belongs to the category of optimization unrolling, a widely-used methodology of designing deep networks for solving inverse problems. The key is to choose an appropriate iteration scheme that fits the problem well. In our case, different from the uniform blurring operator whose inversion can be efficiently done using FFT, the spatially-varying defocus blurring operator $B$ in our problem does not have a computationally-efficient inversion. The proposed fixed-point iteration enables us to avoid such an inversion process in the network and use the forward operator $B$ only. This is one main motivation of using fixed-point iteration for optimization unrolling. We will elaborate it more in the revision.

 **(ii) Convergence** For a fixed-point iteration, it will be convergent if the matrix $I-B$ is a contractive mapping, or equivalently the eigen-values of $B$ fall in (0,1), which holds true if the blurring is a uniform one with a normalized Gaussian kernel. We will elaborate this more in revision. For spatially-varying blurring, while we empirically observed it converges, the theoretical analysis of the eigenvalues of a spatially-varying blurring operator is very challenging. 

------------------------------------------------------------------------------
**Regarding the difference between (1) and (5)**

Sorry for typo. The $n$ in (5) should be $\epsilon$. When deriving the fixed-point iteration, for simplicity, we assume the noise is neglectable. Thus, the noise is no mentioned after (5). We will clarify this in revision. Note that even the noise is omitted in the fixed-point iteration, our network can still learn to handle noisy cases, as the experimental results demonstrated.

 We sincerely thank the reviewer for his/her constructive comments.

------------------------------------------------------------------------------
**Regarding defocus blur versus dual-pixel blur**

First, we would like to mention that while the PSF of “dual-pixel blurring” cannot not be approximated well by a single Gaussian function, the PSF of real-world defocus blur using a regular camera cannot be accurately approximated by a single Gaussian function either. See e.g. [a,b]. 

Thanks for the references which will be added for discussion. As dual pixels use left/right sub-apertures to capture two images whose PSFs will approximately encode left/right half of the defocus PSF of regular image, the merged image from a dual-pixel image pair can simulate the regular defocused image quite well. The difference between the PSF of the merged defocus image and that of the regular defocus image is not significant either, as shown in the reference of DPD dataset. The reason we use the defocus image from the DPD dataset for training is because there is no available dataset of pairs of good-quality regular defocus images and truth images. It is observed in the experiments, our model trained on the DPD dataset still performed better than other methods on real defocus images.

[a] Blur calibration for depth from defocus. IEEE Conference on Computer and Robot Vision, 2016 June.

[b] Estimating Generalized Gaussian Blur Kernels for Out-of-Focus Image Deblurring. TCSVT 2021 March.
 
------------------------------------------------------------------------------
**Regarding why the model has a good generalization ability**

The good generalization ability could be coming from the adaptivity and accuracy of our GSM-based model for modeling defocus blurring. 

(i) Our Gaussian kernel mixture model can be adaptive to different types of non-gaussian defocus kernels, including the ones from the images synthesized from dual-pixel images or the real regular defocus images. In addition, the difference of the characteristics of defocus PSFs between the defocus images on DPD dataset and the real regular defocus images is not significant. Thus, our trained model showed good generalizability in the experiments.

(ii) The parameter space of our GSM-based model is limited to the coefficients associated with the Gaussian kernel set. Thus, there is an implicit constraint on all PSFs, i.e., they are in the linear space spanned by such a set. This can help reducing possible overfitting. 

------------------------------------------------------------------------------
**Regarding the comparison to SOTA motion deblurring**

Please see below for the comparison to MPRNet [a], a recent SOTA motion deblurring method (rank 2 on GoPro motion deblurring dataset). We retrained it on the same data used by our model. It can be seen that MPRNet performs noticeably worse than our method. This showed that motion blurring has quite different characteristics from defocus blurring. One needs to design specific approaches to effectively handle defocus blurring by utilizing its special blurring characteristics.

| Method | PSNR(dB)$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |FLOPs(Billion)$\downarrow$ |
| :------: | :-----:| :-----:| :-----:| :-------:|
| MPRNet | 25.02 | 0.772 | 0.225 |21835.5|
| Ours   | **25.47** | **0.789** | **0.219** |**603.5**|

[a] Multi-stage progressive image restoration. CVPR 2021.

------------------------------------------------------------------------------
**Regarding baselines constructed by inserting the GCM module to SRN or attention module to SRN**

**We did have one such similar baseline in the paper.** The AttNet compared in the experiments can be viewed as inserting attention modules to SRN with some modifications for improvement. The experiments showed that our approach performs better. We also additionally conducted the other suggested experiment of inserting GCM to SRN. It is not straightforward to construct such a modified version. We tried several ways and used the one with the best results. In our implementation, the GCM is inserted to the front, middle and last convolutional layers respectively, that is, there are two parts on each modified convolutional layer: one is the original learnable layers, and the other part is the convolutions with the fixed Gaussian kernels in GCM. Then the model is trained on the DPD dataset as ours. See below for the results, where GCM+SRN performs noticeably worse than our method. 

| Method | PSNR(dB)$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |FLOPs(Billion)$\downarrow$ |
| :------: | :-----:| :-----:| :-----:| :-------:|
| GCM+SRN | 24.18 | 0.602 | 0.228 |3236.2 |
| Ours   | **25.47** | **0.789** | **0.219** |**603.5** |


------------------------------------------------------------------------------
**Regarding enhancing Figure 3**

Thanks for the suggestion. We will show more intuitive examples in it.


The authors propose a supervised end-to-end model architecture for single-image defocus deblurring (SIDD). Therein, unknown point spread functions are modeled by means of Gaussian scale mixtures, and the deblurring operation itself is reformulated in terms of an unrolled fixed point equation. The unrolled scheme is finally combined with a progressive coarse-to-fine estimation.

In the introduction, the authors address a typical challenge in SIDD. Namely, both the defocus operator $B$ and the sought all-in-focus image $x$ need to be reconstructed. To that end, existing methods take a two-step approach where the defocus operator is estimated and fixed first, after which the image reconstruction takes place. The two-step approach comes with a few drawbacks: On the one hand, common estimation procedures for the defocus operator are error-prone and errors from the first stage tend to be propagated or even amplified in the second stage. Moreover, evaluations of the defocus operator in iterative reconstruction schemes can involve high numerical expense. The authors claim that the proposed method overcomes the above-mentioned challenges.  From my point of view, this is a very interesting paper. To the best of my knowledge, it is technically correct, the proposed architecture is novel and the reported numerical experiments show state-of-the-art results.

I only have a few points on which the authors could maybe comment:

- I do not understand why, without further assumptions on $B$, the fixed point iteration **(5)** should converge to an optimal solution. Therefore, although subsequent empirical results show that the derived method works very well, the connection between the motivation and the final method is from my point of view not too strong.
- What is the difference between $\varepsilon$ in **(1)** and $n$ in **(5)**? Why is the measurement noise no further mentioned? Yes

This paper proposes a defocus deblurring method based on an end-to-end trainable network. The proposed method is motivated by the fixed-point iteration and develop a lightweight scale-recurrent architecture with an attention module to achieve pixel-wise GSM-based deblurring under the unrolling framework. Experimental results demonstrate the effectiveness of the proposed method.  This paper proposes a defocus deblurring method based on an end-to-end trainable network which is motivated by the fixed-point iteration. It embeds the pixel-wise GSM into an end-to-end trainable network with a lightweight scale-recurrent architecture and an attention module to solve defocus blur. The performance of the proposed method is better than the evaluated method. 

As the defocus blur is spatially variant and some focused regions are clear, it is not clear whether using pixel-wise GSM can model the property of the clear regions. That is, it is not clear whether the proposed method affects the regions that do not contain blur.

There are some defocus deblurring methods by modifying the Gaussian blur kernels to approximate the defocus blur.  For example, the work by [Liu et al. Estimating Generalized Gaussian Blur Kernels for Out-of-Focus Image Deblurring, TCSVT 2021] shows that commonly used Gaussian kernel does not work well for the defocus blur estimation. It would be better to discuss and compare with the method.

The proposed method is motivated by the fixed-point iteration. However, there are lots of deblurring methods based on the conventional optimization methods, e.g., Aljadaany et al., Douglas-Rachford Networks: Learning Both the Image Prior and Data Fidelity Terms for Blind Image Deconvolution, CVPR 2019. The authors should clarify the major differences and compare the proposed method with them.

Some important discussions of the effect of the proposed method are missing. Since the defocus blur is spatially variant and some focused regions are clear, one commonly used baseline is to first detect blur regions and then use the blur regions (e.g., concatenation of detected blur maps and the input images) to guide the estimations of the clear images in a deep neural network. It would be better to discuss and compare with this baseline method.

What are the limitations of the proposed method?  The paper does not provide an analysis of the limitations 

In this manuscript, the authors address a single image-based defocus deblurring problem. Based on a pixel-wise Gaussian scale mixture (GSM) model for modelling defocus blur effects, this manuscript proposes an end-to-end deep learning-based approach with a strong motivation from unrolling a fixed-point iteration for deblurring.  

Experiments demonstrate that the proposed method achieves competitive results compared to the state-of-the-art methods.  The manuscript is well written, and the exposition is clear. The results from the paper are impressive, and the detailed illustrations in the supplementary material are helpful. 

The theoretical description of the Gaussian scale mixture model and the formulation of the Unrolling fixed-point iteration are easy to follow and well explained in the paper. The GSM-based parametric model for defocus PSF catered specifically to non-uniform blurred images is a good contribution.
 1**. The model is designed for defocus blur with depth-based blur kernels that Gaussian kernels can well approximate. However, the method is tested and trained on the dual-pixel dataset mainly. The PSF of a dual pixel blurred image, usually not approximated by Gaussian kernels, is slightly different from the common defocus blurred image [1,2,3]. 

It would be nice if the authors analysed the connection between the two kinds of blur, which are defocusing blur and the `dual-pixel’ blur. 

The proposed method still achieves competitive performance compared to the STOA, even using the `dual-pixel’ blurred images (the single image (Ic) rather than the dual-pixel pair) in the DPD dataset. Why the model has a good generalisation ability? It would be nice if several sentences were analysing the reasons.

It is good to show that the authors are aware of what model is used in dual-pixel pairs and what is for defocus blur.

[1] Garg, Rahul, et al. "Learning single camera depth estimation using dual-pixels." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.
[2] Pan, Liyuan, et al. "Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
[3] Punnappurath, Abhijith, et al. "Modeling defocus-disparity in dual-pixel sensors." 2020 IEEE International Conference on Computational Photography (ICCP). IEEE, 2020.

2. The reason for using SRN and AttNet as the baseline.

These two methods targeting at motion deblurring while the manuscript is for defocus blur. If the author aims to tell model designed for motion blur is not suit for defocus blur, we suggest the authors add a new baseline here, any of the STOA motion deblurring methods.

Also, SRN has very similar network architectures to the proposed method. Besides SRN, inserting the GCM module to SRN or attention module to SRN would be a fairer baseline. 

3. Enhance figure 3. It is hard to figure out any information via this figure.



Meta-review: This paper addresses a deep learning model which is trained  end-to-end for single image defocus deblurring. In general, the paper is well written and contains interesting contribution. The authors made their efforts to respond to the concerns raised by reviewers. The paper has both positive and negative sides. Reviews on the negative side remained during the discussion period. All the reviewers stood by their decision. Unfortunately one reviewer did not responded even to a few reminders, so the paper has only three reviews. 

Decision: accept