Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: summary: This paper focuses on addressing the challenge of single-image defocus deblurring.
This work is based on the GSM (Gaussian scale mixture) theory to construct a deblurring pipeline with the DNN (deep neural network).
This paper uses extensive experiments to verify the effectiveness and robustness of the model.

main_review: [Strengths] 

1)	Gaussian mixture methods are usually used to fit unfriendly data distributions, and this is the first case I have visited where the use of both the Gaussian kernels and the coefficient  matrices to accomplish fitting arbitrary data distributions, may be a very bold paradigm of innovation.

2)	A large number of experimental comparisons have provided the evidences to support the validity of this work.


[Weaknesses]

Although the proposed model is observed to be valid in terms of quantitative and qualitative results, I still have doubts about the contributions of this work.

1) In L144-145, this work assigns equal weights to the loss at each scale of AGSMNet. In this regard, as far as I know we are more concerned about the clarity of the output of the last scale, and it is natural to think that the last scale los should be assigned the largest weight.

2) In L160-161, this work uses the APU to generate coefficient maps, APU is based Conv-LSTM. This method still seems to have a high computational, so why not consider using the encoder of the transformer to replace it? I'm looking forward to seeing the new ablation experiments.


3) In Table1, I am still troubled by why the proposed model has the least FLOPs, but  process of the GPU is not fast.

4) The Gaussian kernel itself has been applied to the denoising task, so the robustness of the model is necessarily excellent and the robustness consideration of the experimental part seems unnecessary.

5) I think the biggest problem of this paper is still the limited innovation, which seems to box up multi-scale, long range dependence, dual path and other methods to build the model.

limitations_and_societal_impact: none
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose an attentive GSM network for defocus deblurring, motivated from the unrolling of a fixed-point iteration. 
Experiments show that the proposed method in not only lightweight, but also outperforms existing defocus deblurring methods. 


main_review: The paper is well-written and organized. 
Based on a fixed-point iteration, the authors propose a new formulation of deblurring process.
The authors combine coarse-to-fine progressive estimation and unrolled deblurring process into a scale-recurrent attention mechanism.
The experiments show the superior of the proposed method in terms of quality, model size, Flops, and running time.
It would be better if the authors could provide demo testing code for the method.

After rebuttal

Thanks for providing the feedback. I also read comments from other reviewers and like to keep my original rating.

limitations_and_societal_impact: yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a neural network architecture for deblurring images affected my single image defocus blur. The architecture is inspired by the notion that (a) the blur kernel at any pixel for defocus blur can be well approximated as a combination of Gaussian kernels with different widths, and (b) given knowledge of the blur kernel, the deblurred image is given by the fixed point of a repeated filtering operation. These notions are combined with multi-scale reasoning, and leads to a network that performs well in practice while being very light-weight and with fewer number of parameters. This is demonstrated by experiments and ablations.

main_review: Overall, the proposed architecture is novel, well-motivated, and the experiments are fairly convincing. I'm leaning towards acceptance, but hope the authors will respond to/address the following issues:

1. Calling the kernel model a Gaussian Scale Mixture (GSM) is very confusing. This is because Gaussian Scale Mixture *distributions*, as probabilistic models of image gradients/wavelet coefficients, were a very popular, successful, and widely used approach to traditional image restoration. In this context, the "GSM" refers to the approximation of isotropic kernels as sums of Gaussian of different widths (scales). I strongly recommend the authors consider calling this "Gaussian Kernel Mixture" or something else.

On a similar note, it may be worth calling \gamma's mixing weights or some such instead of attention masks (because again, attention has different connotations in neural network architectures). 

2. The description of the fixed point iteration as motivation is great, but while in Sec 1.2, this is applied to multiple iterations at the same scale, the actual architecture also simutaneously keeps increasing the scale at which the method is applied. This makes things a little bit confusing. Perhaps one could write the architecture as K scales and R iterations per scale, with total number of "iterations" = K x R, and that in this case, the authors chose to go with R=1. And that brings up another interesting ablation: is multi-scale really helpful, or would the network have achieved equivalent accuracy with fewer (or even a single) scale, and multiple blocks of GCM/SRAM per scale.

3. Some of the specific details of the architecutre are hard to decipher from the figures. For one, I think there's a typo in equation (8)---I imagine it should be y_t + not y +. Also, it isn't clear what if any weights are shared across different scales. The GCM section talks about a fixed set of kernels, but it isn't clear whether the \sigma values are at the downsampled or original resolution. And in general, for both the GCM and SRAM modules, it would be good to have text describing the precise input/output relationships in the supplementary (it's fine to use existing architecture blocks such as Conv-LSTM as "functions", but deciphering the architecture is a bit hard from just the figures).

4. What's the difference between #parameters and model-size in Table 1?

5. It isn't immediately clear how the attention maps are closely related to the defocus map in Figure 3.

limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposed a deep attentive network for defocus blur removal. The basic idea is the unfolding of a fixed point algorithm for defocus blur removal using a Gaussian Scale Mixture model for modeling the spatially varying blur kernels. The proposed network is a multi-scale architecture with GCM block and SRAM block. The proposed network is evaluated on several defocus blur datasets, and showed improved performance compared with some other methods, e.g., AttNet, ASG, etc.
main_review: The major interesting contributions of this work are the proposed GSM modeling of  defocus blur and the attentive module to estimate the component coefficients. The proposed network is inspired by unfolding a fixed point algorithm for deblur in Eqn. 7.

1. On the fixed point iterations.

The fixed point iterations in Eqn. (5) is not fully analyzed, for example, its convergence analysis. Because it seems that the proposed network is faithfully implementing this iterations in GCM block, these analysis are important to understand the network performance. 

2. Whether the network learn to remove artifacts.

It is not new to unfold algorithms, such as HQS, for image deconvolution. What is the advantage of this proposed fixed point algorithm (and its derived network) compared with these previous HQS-based unfolded networks for image deblur? Moreover, it seems that the proposed network does not explicitly have the artifact removal block (commonly corresponds to proximal operator of regularizer). How to guarantee that the restored sharp image is artifact-free?

3. Comparisons.

It is true that there are very few papers focusing on the blind defocus blur removal. However, the proposed approach can be further compared with other possible variants, such as different unfolded network structures (e.g., HQS) .

Are the proposed network trained once, and tested on different datasets? Is the trained network generalizable to defocus image captured by different cameras?  
limitations_and_societal_impact: Removing defocus blur is a fundamental task in image processing, benefiting to applications such as auto-focus, depth from defocus, downstream recognition tasks, etc. The paper did not fully discuss on the limitations of the proposed approach.
review_title: null

Meta-review: The paper received mixed reviews (3,5,6,7). One reviewer is particularly enthusiastic about the paper.  One reviewer was negative but did not participate in the discussion. His/her comments have been well addressed in the rebuttal and his/her final score of 3 has been ignored from the decision process. Two reviewers have borderline ratings (5,6), and the reviewer with score 5 agreed in the private discussion that it would be fine if the paper is accepted.

Overall, the AC believes that the paper introduces interesting ideas and recommends acceptance.  Nevertheless, Reviewer SpFs raises relevant questions about why the method works and why it makes sense. These should be taken into account in the final version of the paper. You will find below part of the private discussion, which will clarify these concerns and give some hints about how they can be addressed.

***************************
**Reviewer SpFs:**
In the second round of responses, the authors conduct experiments to compare with HQS-unrolling based method, and the conclusions are that the HQS is with higher computation cost and the additional artifacts removal block did not improve much. I appreciate these additional comparisons and it is still a mystery to me on the reason why this propose approach works well even without using artifacts removal block, because the artifacts are commonly serious in deblur / non-uniform deblur tasks. If not explicitly penalize the artifacts, purely relying on deconvolution (implemented by fixed point iterations with learned parameters for blur kernel modeling) can hardly work well. Moreover, the paper did not provide accuracy results of HQS-based methods. However, maybe I miss something / mis-understand on something. I stand on the borderline and is OK with both rejection / acceptance.

**Reviewer 1KQg:**
So, my reading of the paper is a bit different. I think this paper's a little different from HQS methods where the iterations are explicitly minimizing a deblurring objective (i.e., estimate * kernel = observation) with a denoising network as the "prior".

The proposed method is just using fixed point iterations as the 'motivation' for it's architecture design: what they're proposing is essentially just a new kind of network architecture that's trained end-to-end (with the benefit that it's very light-weight and efficient). In the literature, we've seen that direct CNNs trained end-to-end with a reconstruction (and other) losses can achieve state-of-the-art performance.

This paper's method is sort of in that "family" of methods, and so perhaps it's not surprising it can do well without an explicit artifact removal step. In some sense, the fact that the underlying fixed point iterations didn't have an artifact penalty doesn't matter because the kernels are all being learned and with the objective of maximizing the final output's quality. What's novel about the paper is that compared to prior end-to-end networks, this one's architecture is efficient and has this nice motivation from classical deblurring methods.

**Reviewer SpFs**: Thanks, I agree that the proposed network is lightweight and efficient. Regarding the comment "fixed point iterations didn't have an artifact penalty doesn't matter because the kernels are all being learned and with the objective of maximizing the final output's quality", my understanding is as follows. Even the blur kernel is accurately estimated, the deconvolution step will still introduce artifacts (please consider the non-blind image deblurring task). Since this proposed network may not have enough capacity to remove artifacts, to minimize the reconstruction loss, the network may be trained to be conservative and insufficiently remove blurs (because sufficiently removing blur may introduce artifacts and increases the reconstruction error). I maybe wrong, but these are my personal concern.

**AC**: I agree that we do not understand perfectly why the approach works and that there is probably a regularization effect due to the fact that the fixed point iterations do not necessarily converge to a fixed point (in the same way as early stopping can induce regularization in optimization methods). The exact nature of the regularization is unknown, but it does not necessarily prevent it from working since all parameters are learned end to end. I will encourage the authors to think about it and include a discussion in the paper.

Decision: Accept (Poster)