Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes a method for blind super-resolution (SR) of images degraded by arbitrary blur kernels. The method makes use of a novel blur kernel estimation technique, that proposes to estimate such kernels in the frequency domain following an end-to-end deep learning approach. The authors claim that the Fourier amplitude spectrum kernel representation is better posed for the learning task of kernel estimation than the spatial representation, under the assumption that the kernels are sparse both in frequency and space domains (this assumption holds form most common kernels in real settings). They support this claim with theoretical and experimental evidence. 

The image formation model considered here, that produces the degraded LR image from the HR image, is the following. The clean high-resolution (HR) image first downsampled, then convolved with the blur kernel, and finally contaminated with additive noise.  

Based on the proposed frequency domain representation, the authors propose a kernel generator network that allows to translate this representation to the kernel spatial representation. The generator $G$ takes as input the single-channel amplitude spectrum (the FFT) of the degraded LR image bi-cubicly downsampled by two, and outputs the single-channel blur kernel estimate. This generator consists of an encoder-decoder network built on a U-net with mirror connection between each of the seven down-sampling convolutional layers and up-sampling transposed convolutional layers with the same feature size. 

To ensure that the kernel generated by $G$ is at the same time close to the ground truth kernel and consistent with the SR results, they propose a three-terms loss for $G$. The first term imposes similarity between the ground truth kernel and the kernel generated by G from the degraded image (in the spatial domain) by means of the $\ell_1$; the second term also imposes similarity between these two kernels but in seen as probability density functions, using an adversarial loss that involves a trainable critic $D$ that has to  discriminate between the ground truth kernel and the output of G (actually, de concatenation of each of them with the input of G are considered instead). This patch discriminator is trained using the loss proposed in LSGAN. Finally, the third term is a regularization loss on the output of G; unless I missed it, the regularizer is not specified. This kernel estimation model is called S2K. 

Then, the authors propose a pipeline for blind SR that combines the S2K module with a non-blind SR module. For the non-blind SR module, they consider two methods:  the multiple degradation-based method SFTMD (Gu et al., CVPR 2019; ref 6) and the kernel modeling-based method RCAN (Zhang et al., ECCV 2018; ref. 35). Given a ground truth kernel of size 15x15 and a HR image, a LR degraded image is generated and cropped to 512x512. The input to the S2K module is the ground truth kernel upsampled to 256x256 and the FFT of the LR image, downsampled to 256x256. The estimated kernel of size 256x256 returned by G is then downsampled at size 15x15. Finally, the non-blind SR module is fed with the LR degraded image and the 15x15 kernel estimated with the S2K module.

The experiments are carried out considering Gaussian kernels of size 15x15 of random covariance matrix, and random motion kernels of size 23x23 (instead of 15x15) with Boracchi and Foi’s method (IEEE TIP, 2012; ref 3). The proposed approach is compared to an extensive list of state-of-the-art methods, both for the task of blur kernel estimation and blind SR for 2x, 3x and 4x SR. All methods were trained on the the same degraded dataset (DIV2K), including S2K and the non-blind methods used in the proposed blind SR pipeline. Blind SR methods are compared on synthetic images using the PSNR and SSIM metrics. Kernel estimation performance is evaluated using the $\ell_1$ norm and KL divergence. The results reported by the authors show that the proposed S2K and the blind SR pipeline significantly outperform the others. Finally, an ablation study on the loss components and frequency vs. spatial inputs is conducted, showing the pertinence of the proposed approach. 

Finally, blind SR experiments on real images using the dataset proposed by Cai et al. (ICCV 2019; ref. 4), evaluated using two reference-free metrics (PIQE and BRISQUE) show that the proposed pipeline also outperforms its competitors, including the NITRE20 winner (Ji et al., CVPR 2020; ref. 10). 

main_review: (+) The method is sound, elegant and simple

(+) References and review of previous work are adequate.

(+) Extensive evaluation, both on blur kernel estimation and blind SR. 

(+) According to the results reported by the authors, the proposed approaches for blur kernel estimation and the blind SR tasks both outperform state-of-the-art methods by significant margins. 

(+) Code is provided as supplementary material (I did not check it).

(-) The image formation model is not realistic. The way LR images are produced from HR images is by first convolving with the HR blur kernel, and then subsampling. Not the other way round. 

(-) The fact that comparison of the spectrums of the degraded image with the blur kernel is more conducive than comparison of the degraded image and the blur kernel in space is evident. As the blur kernel low-pass filter the image, it is obvious that the coarse features of the spectrum of the kernel and of the degraded image are similar. 

(-) Frequency consistency was already proposed and demonstrated in ref. 11 (not published but reported in arXiv). Moreover, the kernel estimation method proposed in ref. 11 bears strong similarities to the one proposed here. This works even proposes an adversarial loss for the high-frequency components of the blur kernels. The authors should clearly state the differences and contributions of their work compared with ref. 11. 

(-) Last but not least, the paper is poorly written. While the idea is simple, the paper does not read well. For instance:

* When the authors show that comparison of the degraded image and the kernel in the frequency domain is more conducive than in the spatial domain, one wonders why this comparison is relevant. They only show its relevance with an example using a Gaussian kernel afterwards. 

* Notation is sometimes misleading; for instance, in Section 3.2 $\mathbf{S}$ is used to represent the Fourier transform of the clean image, and then from Section 3.3 it is used to represent the spectrum of the degraded image. 

* The methods presented in the tables should also be referenced in the tables. 

* English is poor, both grammatically and syntactically. The text is plagued with conjugation errors, unstructured sentences and typos. The article should be almost completely rewritten and checked by someone fluent in English.  

limitations_and_societal_impact: I do not see any limitation.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new blind super-resolution method, which can due with arbitrary blur kernels instead of Gaussian kernels. The whole method estimates kernels in the frequency domain. The method involves a kernel generator that takes the frequency amplitude as input and outputs a kernel. A patch discriminator is designed to perform adversarial loss to the predicted kernel. This design is novel and the experiments show good results.
main_review: I have several questions:

1. Can we know how this kernel was predicted? Is there any visualization method that can tell us which part of the frequency amplitude contributes to the kernel prediction? This is important when it comes to the generalization performance of the proposed method. Although the experiment shows good generalization performance in Figure 6, an interpretational explanation is also wanted to address our concerns about how this system works.

2. As a common sense that the kernel prediction is an ill-posed problem. How does the proposed method address this issue?

3. This paper actually proposes a kernel estimation method. Can this method be used for deburring motion blur? How is the performance? If it is specially designed for blind super-resolution, what makes it especially suitable for SR task? 

4. In Table 2, the motion blur part, The IKC, Pan et al. [22] w. SFTMD, Pan et al. [23] w. SFTMD and other methods are not trained for motion blur, the author should compare their method with motion-deblur + regular SR for better results. The claim of nearly 6db improvement is not objective as the STOA performance on these images is not obtained by the abovementioned methods.

As the good performance and novel idea (predict in frequency domain), I give a positive initial rating. I hope the authors can answer my questions so that I can make the final suggestion.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a novel framework for solving the problem of blind super resolution in which the low resolution input can be degraded by arbitrary blur kernels. In their framework, the degradation kernel estimation is performed in the frequency domain. The authors show that the frequency domain is better suited for kernel representation and reconstruction than the spatial domain and propose a Spectrum-to-Kernel mapping network to estimate diverse blur kernels. They use a conditional GAN to learn how to 'translate' the spectra of degraded images to the unknown kernels.
main_review: Estimating degradation kernels for the case of blind upscaling is typically an ill posed and difficult task and estimation errors can carry over into conditional upscaling methods resulting in artefacts in the upscaled image. Therefore, efforts to better represent and estimate degradations automatically is relevant.

The idea of estimating kernels in a different representation seems nice and the authors demonstrate that for commonly existing sparse kernels, reconstruction in the frequency domain is preferable compared to the spatial domain. The observation that there is a closer resemblance between the frequency spectrum of the kernel and the frequency spectrum of the image degraded with that kernel as compared to the spatial kernel and the degraded image in the spatial domain is interesting. Also as the authors state, I am not aware that this has been leveraged for the task of blind super resolution so far. Therefore it seems intriguing to design a network to estimate a kernel from the Fourier amplitude spectrum. Also the results shown look competitive.

However, in my opinion the work [CORNILLÈRE et al., Blind Image Super-Resolution with Spatially Variant Degradations, Siggraph Asia 2019] is quite related and should be cited and compared to. There, the estimation of the degradation is based on an optimisation that discriminates artefacts in the resulting high res while being parameterised through a neural network embedding. Fundamentally, it seems like a very different approach to estimate the degradation in the low res. A lot of existing works follow the path of estimating the degradation from the low res and here I can see that it could be better solved int the frequency domain. However, in the resulting high res where kernel estimation errors quickly become very visible in the form of ringing/oversharpening or excessive blurring, it seems that also methods operating in the spatial domain might find accurate solutions. Therefore I would like to see comparisons as mentioned before and ideally a small paragraph to interpret the results and differences in the approaches.

Overall Equation 8 seems to indicate that the frequency domain might be better for estimating degradation kernels but I'm not sure it can directly be derived from the equation since it should only show that kernel and degraded image are closer together w.r.t. a given metric in frequency space as opposed to image space. While this gives a good motivation to follow that strategy, it seems that further assumptions/ restrictions would be needed to conclude that this will also allow more accurate kernel estimation / reconstruction. It would be great if the authors could clarify that a bit more. As a more minor point, it would also be great to detail a bit more about how reconstructing the kernel in the spatial domain would compare to reconstructing in the frequency domain.

Overall, I feel this paper offers an interesting view on blind image super resolution and tries a novel approach that has not been analysed so far in this context. However at the same time some important comparisons are missing and the paper seems slightly unpolished with a larger number of typos.
limitations_and_societal_impact: I would not expect negative societal impact from this work. The limitations are listed in the supplemental material but might actually deserve a place in the paper. At least the point regarding spatially varying degradations.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors make a key observation that degradation kernels are highly correlated to degraded images in the frequency domain. From this, they design a kernel generator that directly generates the degradation kernel from the frequency-domain LR image. Then, a non-blind SR model is employed to utilize the estimated kernels for SR. The proposed method outperforms existing methods on two types of kernels (Gaussian and motion) with two backbone networks (SFTMD and RCAN).
main_review: Pros:
- This paper provides insightful observations that degradation kernels are highly correlated to the degraded image in the frequency domain (Fig. 1 and Fig. 2).
- From the above observation, a novel approach is taken, where the blur kernel is directly estimated from the frequency-domain LR image.
- A sound theoretical analysis is provided in Section 3.1 .
- Supplemetary material provides additional details that helps to further understand the main paper.
- Ablation studies verify the benefits of each of the proposed components including the spectrum-to-kernel scenario.
- The proposed method outperforms existing methods on real images.

Cons:
- Clarifications are needed in some parts of the paper:
  - The phrase in lines 37-39, “the iterative form can not utilize the maximum performance of both estimate and SR module during the testing process, and the barrel effect is inevitable", is unclear. Why can't IKC reach maximum performance for both the estimator and the restorer? What do the authors mean by barrel effect?
  - What do the authors mean by "inconsistency of training and test domain" in lines 216-217? Aren’t kernels generated randomly for both training and testing?
  - In line 208, what do the authors mean by 'random blurred'? Is this random Gaussian blur or randomly selected motion blur kernels?
- Some missing analyses in experiments:
  - Is there a reason why the performance gain reduces as scale factor increases?
  - Why do existing methods perform poorly for motion kernels? Aren’t they trained on the same random motion kernels used for training the proposed model?
- In Table 2, comparison to more recent blind SR methods such as DAN (NeurIPS 2020) is missing.
- It is common to include SR results on historic images (http://vllab.ucmerced.edu/wlai24/LapSRN/) as in ZSSR, IKC, etc in blind SR literature.
- The motivation behind bicubic downsampling the FFT image before the kernel generator is unclear. Wouldn't this lose information? If it is to reduce computations, why not downsample by 4 or 8?
- Statements like “existing blind SR methods are far from meeting the needs of various degradation processes in general scenes” (lines 30-31), need references or evidence. How are they ‘far from meeting the needs of various degradation’?

Minor:
- In Table 4, what is the meaning of para.?
- It would be useful to show LR input images in Fig. 5.
- Typos: (i) Table 1 caption, estimaion -> estimation, (ii) line 234, we -> We.

========================  
**Post Rebuttal:**
Raising my score from 6 to 7 after the authors' reply. More details are in a separate comment. Well done on the new experiments and the clarifications!
limitations_and_societal_impact: Limitations are discussed in the supplementary material but potential negative societal impact has not been addressed. Some potential negative societal impacts could include privacy issues due to upscaling degraded data.
review_title: null

Meta-review: This paper tackles the problem of blind super-resolution when the images degraded by arbitrary blur kernels (assumed to be sparse both in frequency and space domains). The method proposes a new blur kernel estimation technique in the frequency domain as an end-to-end deep learning approach. The paper provides theoretical and experimental evidence to back the idea.

The core idea of the work is predicting degradation kernels in the frequency domain using deep neural networks. The proposed method is simple and effective, this was highlighted by all reviewers. The empirical evaluation seems convincing (and were further improved during the response period). The manuscript includes thorough ablation studies which clarify the benefits of each of the proposed components. This was highlighted by Reviewers wMvH and bKSR.

The authors provided a high quality rebuttal including new experiments and clarifying several points raised by the reviewers. In particular, the authors incorporated experiments on (i) mixed blur kernels, (ii) comparison to a more recent blind SR baselines, and (iii) quantitative and qualitative results on real historic images. All reviewers and the AC appreciate the authors response.

After the authors’ response, Reviewer wMvH increased the score to 6. The clarifications provided by the authors were considered sufficient. However, she/he expressed concerns regarding the papers presentation. Similarly Reviewer bKSR increased the score to 7. In the original review, the main concern was clarity. 

he AC appreciates the authors response and counts that the authors will incorporate the the clarifications provided in their response. All reviewers provide an accepting score. The AC agrees and recommends accepting the paper.

Decision: Accept (Poster)