Spectrum-to-Kernel Translation for Accurate Blind Image Super-Resolution
Year: 2022

Abstract:
Deep-learning based Super-Resolution (SR) methods have exhibited promising performance under non-blind setting where blur kernel is known. However, blur kernels of Low-Resolution (LR) images in different practical applications are usually unknown. It may lead to significant performance drop when degradation process of training images deviates from that of real images. In this paper, we propose a novel blind SR framework to super-resolve LR images degraded by arbitrary blur kernel with accurate kernel estimation in frequency domain. To our best knowledge, this is the first deep learning method which conducts blur kernel estimation in frequency domain. Specifically, we first demonstrate that feature representation in frequency domain is more conducive for blur kernel reconstruction than in spatial domain. Next, we present a Spectrum-to-Kernel (S2K) network to estimate general blur kernels in diverse forms. We use a Conditional GAN (CGAN) combined with SR-oriented optimization target to learn the end-to-end translation from degraded images’ spectra to unknown kernels. Extensive experiments on both synthetic and real-world images demonstrate that our proposed method sufficiently reduces blur kernel estimation error, thus enables the off-the-shelf non-blind SR methods to work under blind setting effectively, and achieves superior performance over state-of-the-art blind SR methods, averagely by 1.39dB, 0.48dB on commom blind SR setting (with Gaussian kernels) for scales 2× and 4×, respectively.

heading: 1 Introduction
text: Single Image Super-Resolution (SISR) is a low-level visual task to recover High-Resolution (HR) images from its degraded Low-Resolution (LR) counterparts. Deep learning methods have significantly promoted SR research and achieved remarkable results on benchmarks [16, 26, 27, 13, 29, 35, 17]. Most of these works are based on the degradation model assuming that the LR image is down-sampled with blur kernel (e.g., bicubic kernel) and additional noise from its HR source. Recent studies [6, 11] show that pre-trained SR model is sensitive to the degradation of LR image. The mismatch for Gaussian kernels may cause SR result either over-sharpening or over-smoothing, and for motion kernels, it may bring unpleasant jitters and artifacts. Blind SR is to super-resolve a single LR image with its kernel unknown. It has attracted more and more attention due to its close relationship with the real scene. However, compared with the advanced non-blind SR researches, existing blind SR methods are insufficient to meet the needs of various degradation processes in real general scenes. ∗Work done during an intership at Tencent Youtu Lab. † Corresponding author (lutong@nju.edu.cn).
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Some representative works have been proposed on this challenging problem. Bell-Kligler et al. estimate the kernel with internal-Generative Adversarial Network (GAN) on the basis of maximum similarity of patches across scales, whose performance is limited even with handcraft constraints for the weak prior fails to capture the kernel-related features directly [2]. Gu et al. propose an Iterative Kernel Correction (IKC) framework to correct the kernel estimation from the mismatch between kernel and SR result in an iterative way [6]. IKC is based on Maximum A Posteriori (MAP), in which the estimator depends on the posterior of the pre-trained restorer. As demonstrated in [14], due to the accuracy limitation of the restorer, the maximum performance of the estimator cannot be guaranteed. Ji et al. introduce the frequency consistency constraint between the generated down-sampled image and the real input to estimate the blur kernel in spatial domain [11]. However, its training requires a certain amount of data with the same degradation and fails to capture blur diversities in various directions, limiting its application for a specific input LR image degraded with an arbitrary kernel.
In this paper, we capture the convolution degradation feature of the local distribution in LR images from a novel perspective. Based on convolution theorem and sparsity analysis, we demonstrate the Fourier frequency spectrum of degraded LR image provides a robust shape structure of the degradation kernel in the frequency domain, which can be utilized to reconstruct the unknown kernel more accurately and robustly. Based on the above analysis, we design a Spectrum-to-Kernel (S2K) translation network to directly obtain the kernel interpolation result from degraded LR’s frequency spectrum. As far as we know, this is the first work to estimate kernel thoroughly in the frequency domain instead of in the spatial domain previously. The experimental results demonstrate that our method accurately estimates the target kernel and has a high universality capacity for diverse kernels. Moreover, when combined with the existing non-blind methods, we achieve the best results in quality evaluation and visual perception on synthetic and real images against existing blind SR methods. Figure 1 compares 4× blind SR results with recently proposed methods on real historical images. In general, our main contributions are summarized as follows:
• We theoretically demonstrate that for common sparse kernels, it is more conducive to reconstruct it in the frequency domain instead of the commonly used spatial domain. • We analyze the impacts of kernel estimation optimization objectives for blind SR performance and design a novel network S2K to estimate kernel from Fourier amplitude spectrum. • When plugged into existing non-blind SR models, our proposed pipeline achieves the best performance on both synthetic and real-world images against existing blind SR methods.

heading: 2 Related Work
text: Non-blind SR Under the non-blind setting, HR image IHR is recovered from paired given paired LR image ILR and kernel k. Many successful pieces of research have been proposed where the bicubic kernel is used to construct paired training data, so they belong to the non-blind setting in essence. We divide those kernel-aware SR methods into two categories. (1) Multiple Degradation SR (MDSR) based methods concatenate degraded image ILR with stretched blur kernel k as SR model’s input [33, 31] or conduct intermediate spatial feature transform according to the corresponding kernel
to get target SR result [6, 31]. (2) Kernel Modeling SR (KMSR) based methods incorporate the blur kernel in the model’s training parameters by constructing the paired kernel modeling dataset for the known test kernels to get a specific SR model for test data’s degradation restoration [36, 10]. Inconsistency between ILR and blur kernel k leads to significant performance deterioration in test phase [33]. We propose a spectrum-to-kernel framework (S2K) that provides accurate kernel estimation to help both MDSR and KMSR based non-blind methods work well under blind settings.
Blind SR Under blind setting, IHR is recovered from ILR degraded with unknown kernel. Since it is more closely related to real application scenarios, blind SR has attracted research attention for a long time [21], mainly involving transforming unknown degraded image into target IHR’s bicubic down-sampled one, and non-blind SR methods with kernel estimation from spatial input ILR.
In the former category, Hussein et al. modify the ILR with unknown k into that down-sampled from IHR by bicubic kernel [9]. Maeda et al. train a correction network to get the corresponding clean LR and construct pseudo pair to train the SR model [19]. In the latter category, Bell-Kligler et al. utilize the recurrence of patches across scales to generate the same-degraded down-sampled image as the test input [2]. However, the estimation accuracy is limited even with many handcrafted constraints due to the weak prior. Gu et al. use a kernel prediction network with a correction network taking the current kernel prediction and SR result of the pre-trained SR model as input to correct the kernel estimation iteratively [6]. Huang et al. adopt an alternating optimization algorithm with two modules restorer and estimator [8]. Ji et al. obtain the approximation of the blur kernel by utilizing the frequency domain consistency to estimate kernel in an unsupervised way [11]. Liang et al. propose Flow-based Kernel Prior (FKP) learning an invertible mapping between the kernel distribution and a tractable distribution to generate reasonable kernel initialization, improving the optimization stability [15].
Different from previous kernel estimation methods in the spatial domain, we demonstrate Fourier spectrum in frequency domain can provide a more effective kernel shape structure, which contributes to accurate and robust estimation for arbitrary kernels. Further, utilizing the shape structure relationship of the sparse kernel between the frequency domain and the spatial domain, our method can estimate diverse blur kernels end-to-end with implicit cross-domain translation. Besides, we propose SR-oriented kernel estimation optimization objection and obtain a more accurate estimation of arbitrary unknown blur kernels directly and independently. When combining S2K with existing advanced non-blind SR methods, we achieve the best quantitative and qualitative performance on synthetic and real-world images against existing blind SR methods.

heading: 3 Method
text: In this section, we first formulate the problem, then theoretically analyze the advantage of blur kernel estimation from the frequency domain and the feasibility of end-to-end translation learning. Then we introduce the pipeline and component of the proposed S2K network in detail. Finally, we present the blind SR pipeline combined with the proposed kernel estimation module.

heading: 3.1 Problem Formulation
text: Same as [33, 34, 11], we assume the image degradation process from IHR to ILR is as follows:
ILR = (IHR↓s)⊗ k+ n, (1) where k denotes degradation kernel, and n is additional noise3. ↓s and ⊗ are down-sampling and convolution operations. Under non-blind setting, both k and n are known, denoting the pre-trained SR model as R, the upper bound of SR result I∗SR can be expressed as: I∗SR = R(ILR | k,n). (2) Under blind setting, the degradation prior needs to be obtained from ID. In this paper, we mainly focus on the estimation of degradation kernel from the degraded input ILR, because the extra denoising method can be easily plugged in from the previous denoising methods [7, 10]. Theoretically, the optimal kernel estimation target k∗ for blind SR performance is defined as follows:
k∗ = argmin kest ‖R(ILR | kest)− I∗SR‖. (3)
3Above variables are all matrices, and blur kernel size is far less than LR image size.
Accurate kernel estimation is the key to make the most of the powerful performance of existing nonblind methods under the blind setting. However, it is difficult for its ill-posed property. Meanwhile, the non-blind SR models are very sensitive to the consistency of input {ILR,kest} pairs, and subtle degradation inconsistencies may cause obvious artifacts [6]. We explored the influence of the kernel estimation optimization target on the performance of subsequent non-blind SR to ensure the consistency of kest and ID, and get the result close to the upper bound I∗SR in the following parts.

heading: 3.2 Spectrum-to-Kernel
text: Different from previous kernel estimation methods in spatial domain, S2K is the first deep-learningbased attempt to conduct kernel estimation entirely in the frequency domain. Shape structure is conducive to the reconstruction task learning: Chen et al. obtain better face SR results with shape prior [5], Ma et al. maintain the gradients structure, which proves to be beneficial to the SR result [18]. Some researchers utilize structure information to improve the performance of image inpainting [25, 22]. We find the kernel representation obtained from the Fourier spectrum can provide more robust and favorable feature from shape structure than spatial patches. We below give a formal proof that kernel estimation in the frequency domain is more conducive than in the spatial domain under the reasonable assumption that blur kernels of different forms satisfy sparsity.
Advantages of the Frequency Domain In frequency domain (Fourier frequency domain unless specified), denoting clean images, degraded images, and degradation kernels as S, G and F respectively. According to the convolution theorem, denoting hadamard product as , we have:
G = F S. (4)
We adopt φ(X,Y) to measure shape similarity of two matrix variables X,Y [12, 32] as follows:
φ(X,Y) = ‖δτ (X−Y)‖0, (5)
where δτ (·) is a truncated function to ignore the extremely small divergence in each elements and ‖ · ‖0 denotes `0-norm operator. When the absolute value of element x in (X −Y) is less than τ
(positive but very small), it returns 0, otherwise returns the absolute value of x. Under the assumption that common kernels are sparse in both the frequency and spatial domains, we compare the shape structure similarity of the input and output for kernel estimation in the frequency and spatial domains, respectively. In the frequency domain, we have:
φ(G,F) = ∑
Fij<τ
sign(δτ (FijSij − Fij)) + ∑
Fij≥τ
sign(δτ (FijSij − Fij))
≤ ∑
Fij<τ
sign(δτ (Fij(Sij − 1))) + ∑
Fij≥τ
sign(δτ (Fij))
= 0 + ∑
Fij≥τ
sign(δτ (Fij)) = ‖δτ (F)‖0.
(6)
Here, if x is positive, sign(x) > 0, else if x is negative, sign(x) < 0, else sign(x) = 0, and i, j denote position coordinates in spectrum. Eq. 6 gives the upper bound of the distance of the shape structure from the degraded input to the output kernel in the frequency domain. While in the spatial domain, denoting the zero-padded degradation kernel with kp which has the same size of the degraded image ILR, we have:
φ(kp, ILR) = ∑
[kp]ij<τ
sign(δτ ([kp]ij − [ILR]ij)) + ∑
[ksp]ij>τ
sign(δτ ([kp]ij − [ILR]ij))
≥ ∑
[kp]ij<τ
sign(δτ ([kp]ij − [ILR]ij)) ≥ ∑
[kp]ij<τ, [ILR]ij>2τ
sign(δτ ([kp]ij − [ILR]ij))
≥ ‖δ2τ (ILR)‖0 − ‖δτ (kp)‖0,
(7)
which gives the lower bound of the distance of the shape structure on the spatial domain. Further, since both F and kp are sparse, ‖δτ (F)‖0 and ‖δτ (kp)‖0 are significantly smaller than ‖δ2τ (ILR)‖0 due to the non-sparsity of ILR. In this case, ‖δτ (F)‖0 ‖δ2τ (ILR)‖0 − ‖δτ (kp)‖0. Thus we have:
φ(G,F) φ(kp, ILR). (8)
Eq. 8 reveals that the divergence between the blur kernel and its corresponding degraded image in the frequency domain is significantly smaller than that in the spatial domain. As illustrated in Figure 2, compared with the spatial domain, G and F have a more significant and robust kernel shape correlation, which is conducive for the kernel reconstruction in the frequency domain.
After demonstrating the shape advantage for general kernel estimation from the frequency domain over the spatial domain, we further analyze the cross-domain relationship of the kernel’s shape structure. Taking the Gaussian kernel as the example, according to Fourier transform property of 2D Gaussian function, kernel in the frequency domain is also in Gaussian function form, and its variances are inversely proportional with that in spatial. Using the notation in Figure 3 left, we have:
σu = C1/σx, (9) σv = C2/σy. (10)
Here, C1,C2 refer to two constants4. It reveals that it is feasible to learn the cross-domain translation implicitly and estimate spatial kernel from the degraded image in the frequency domain with spectrum in an end-to-end manner. Experimentally, we find it still works for motion and disk kernels.

heading: 3.3 Spectrum based Kernel Estimation
text: Based on the analysis above, we propose a new kernel estimation method from the frequency domain, which makes full use of the shape structure from ILR’s spectrum to reconstruct the kernel . We further utilize kernel’s shape relationship to learn the cross-domain translation, thus estimating the unknown spatial kernel from the frequency spectrum end-to-end.
4More details provided in supplementary materials.
Kernel Generator Unlike the previous methods estimating kernel’s parameter from spatial input, we obtain the target kernel in spatial from the frequency amplitude. The Generator is an encoderdecoder structure (Figure 4) and takes the 256×256 single-channel amplitude spectrum of the LR image as the input. After through an input layer with stride 2, the feature is sent into a U-net where a mirror connection is between each of the seven down-sampling Conv layers and up-sampling transposed Conv layers of the same feature size. Finally, we obtain the single-channel estimation kernel map through the output layer. As discussed in Figure 4 right, we use relative shape loss `s and smoothing loss `reg in addition to the `1 loss to ensure that the estimated kernel close to ground truth and favorable for SR results. For input pair {si,ki} (si denotes spectrum of ILRi), to make the generator output G(si) closer to target ki, the generator’s loss `G consists of three parts:
`G = Esi,ki [λ1‖ki −G(si)‖1︸ ︷︷ ︸ `1 +λ2[D(si;G(si))− 1]2︸ ︷︷ ︸ `s +λ3R(G(si))]︸ ︷︷ ︸ `reg , (11)
where ( ; ) denotes feature concatenating and D denotes the discriminator,R denotes regularization item, λi denotes weight of respective loss component. The first loss `1 pushes G(si) pixel distance closer to target, the second loss `s makes G(si) satisfy the target kernel’s spatial surface shape. We use adversarial loss to make the generated results conform to target kernel’s distribution, serving as the shape loss. The third total variation loss `reg regularizes the output with smoothing regularization.
Patch Discriminator We use a fully-convolutional patch discriminator to learn the distribution of different kernel forms. To make the training more stable, we adopt variant of LSGAN [20] where the least square loss is calculated between the patch embeddings of the predicted kernel and the
corresponding ground truth (both concat. with si). Discriminator’s training loss can be written as:
`D = Esi,ki [(D(si;G(si))) 2 + (D(si;ki)− 1)2]. (12)

heading: 3.4 Pipeline of Blind SR
text: Based on the proposed S2K kernel estimation module as shown in Figure 4. We further introduce our basic pipeline for blind SR. For kernels defined by continuous parameters i.e., Gaussian kernels and disk kernels (3 and 1 parameters), both MDSR and KMSR [36] (without kernel expansion) based non-blind models are applicable. For motion kernels with more discrete and irregular form, we use MDSR based SR model [6] to get blind SR result taking ILR and estimated kernel k as input.

heading: 4 Experiments
text: Experiment Setup In theory, our method can handle general sparse kernels. We select the two most commonly used blur kernels in image processing, i.e., Gaussian and motion kernels to conduct blind SR experiments. We set Gaussian kernel size as 15 and variance between 1.0 and 3.0, rotation angle between 0 and 2π. We randomly select from 500 motion kernels using code from [3], and the kernel size is 23, exposure time is set in [0.15, 0.3], anxiety is 0.005. In addition, we add estimation of disk kernels and use Matlab function fspecial to randomly generate disk kernel according to the input radius between 1.0 and 3.0. All kernel-aware methods compared are trained on the same dataset degraded with random kernels (three types of random kernels are estimated separately). We use DIV2K [28] training set to train S2K and the non-blind SR model. Then, we use random blurred DIV2K validation set, 100 images in Flicker2K [1] as test datasets in the synthetic experiment, and conduct 2×, 3×, 4× SR comparison and kernel estimation accuracy comparison. Finally, we use our pre-trained estimation model of Gaussian kernels to perform 4× SR test on the real image from RealSR [4] and historical photos. We provide qualitative and quantitative comparisons with state-of-the-art methods. We set the weights of loss item in generator as λ1 = 100, λ2 = 1, λ3 = 1 respectively. For optimization, we use Adam with β1 = 0.5, β2 = 0.999, and learning rate is 0.001.

heading: 4.1 Blind SR on Synthetic Images
text: We combine proposed S2K with non-blind methods and provide comparisons with existing methods in Table 1. Due to the inconsistency of training (clean dataset) and test (unknown degradation) domain, the performance of ordinary ideal kernel-awareless SR models (e.g., RCAN, ZSSR) suffers a significant deterioration on the test data. The performance of blind methods trying to correct test LR image to get clear LR image (e.g., DeblurGAN w. RCAN) is limited because down-sampling in degradation process makes the recovery of high-frequency components more difficult in LR image. KernelGAN and FKP obtain more accurate predictions than bicubic interpolation kernel, but the
inadequate estimation accuracy degrades the subsequent SR model’s performance. Using the same pre-trained SFTMD model, we can obtain better results against IKC, for our estimation kernel model is not based on the pre-trained MDSR model but in the frequency domain. FCA is unable to deal with the kernel estimation of the specific degraded image while our method can estimate more general kernels more accurately without these restrictions. Visual comparison results are provided in Figure 5.
Kernel Estimation We provide the `1 distance Dv and relative shape distance Ds as in Eq. 13 between predict kernel and the ground truth on 100 test ILR images. Here h, w denote the height and width of the kernel, i, j denote position indices, respectively.
Dv(X , Y ) = 1 h ∗ w ∑ i,j |Xi,j − Yi,j |, Ds(X , Y ) = ∑ i,j Xi,j log( Xi,j Yi,j ). (13)
Δ =9.45 dB
Δ =1.55 dB
4.16M 54.40M
3.48M
Flops (*1011)
Δ =0.63 dB
Δ =10.08 dB
Frequency U-Net
Spatial U-Net
Frequency E-D
Spatial E-D
0 1 2 3 4 5
12.5
15.0
17.5
20.0
22.5
25.0
27.5 39.07M ParametersPSNR (dB) Figure 7: SR performance with different gen-
erator models. Here, E-D denotes U-net removing the mirror skip connection. We use Unet-864, Unet-5-32 and ED-8-64, ED-5-32 (i.e., arch.downsample/residual blocks-channels). Estimation in frequency domain is less affected by model capability, and the PSNR drop from model reduction is far less than that in spatial domain, showing the advantages of frequency domain estimation.
The results are shown in Figure 6. Compared with existing single image kernel estimation methods, our proposed method can more accurately estimate Gaussian, motion, and disk kernels qualitatively and quantitatively. Further, to verify the advantage and generality of estimation in the frequency domain, we use different reconstruction model architectures and sizes, respectively, and compare the
Table 2: Ablation study on loss components.
Table 3: Ablation study on input/output forms.
blind SR performance as shown in Figure 7. With the same model architecture, when the number of parameters decreases significantly, subsequent SR performance with kernel estimated in the frequency domain estimation is reduced by 1.55dB and 0.63db respectively, and for spatial domain is reduced by 9.45dB and 10.08dB respectively. The effective shape information provided in the frequency domain can still ensure relatively good SR performance for the model with fewer parameters. However, spatial estimation is more sensitive to the model size and capability.
Ablation Study We conduct ablation study on loss components of the proposed generator. The results are shown in Table 2. We conduct extra comparisons with the same encoder network and training pipeline to verify the superiority of obtaining kernel directly from the frequency spectrum. Results can be found in Table 3. The ablation experiments are conducted on 2 additional test sets using the setting of 3× blind SR on DIV2K valid set degraded with random Gaussian kernels (Para. and Kernel↑ briefly denote kernel parameters and up-sampling interpolation of kernel matrix). Kernel estimation errors in different positions have unequal influences on subsequent SR results, `1 without position information is inconsistent with SR results when the gap is small. `s allows the generator to learn the shape of the target kernel (i.e., 2D-Gaussian curve). The smooth prior `reg is conducive to the initial generator’s learning. Our method uses the translation from the spectrum to the interpolated kernel, which avoids the learning of local spatial pixel distribution, dramatically improving the learning stability. We achieve more superior kernel estimation accuracy and better SR results with the same network structure and pipeline compared with other settings.

heading: 4.2 Blind SR on Real Images
text: For real images, we use the paired image taken with a closer view in the dataset [4] as the contentfidelity reference and take into account the visual perception of the SR results. The comparison methods include KernelGAN [2], LP-KPN [4], IKC [6] and NTIRE20 winner [10]. We adopt the Gaussian kernel estimation setting with smaller variances and use the same SR pipeline as [10] but replace the kernel estimation part with our proposed S2K. The result of 4× SR is shown in Figure 8. Compared with NTIRE20 winner [10], our results achieve higher fidelity (red indicators) and fewer artifacts (green indicators). There is no ground truth in the strict sense, and we provide the non-reference evaluation metrics for the real image SR results as Figure 8 right.

heading: 5 Conclusion
text: This paper theoretically demonstrates that feature representation in the frequency domain is more conducive for kernel estimation than in the spatial domain. We further propose S2K network, which makes the full use of the shape structure from the degraded LR image’s frequency spectrum and, with
a feasible implicit cross-domain translation, outputs the corresponding up-sampled spatial kernel directly. In addition, we apply the kernel estimation optimization objective consistent with subsequent SR performance. Comprehensive experiments on synthetic and real-world datasets show that, when plugging S2K into the off-the-shelf non-blind SR models, we achieve better results both visually and quantitatively against state-of-the-art blind SR methods, by a large margin.
Acknowledgments and Disclosure of Funding
This work is supported by the Natural Science Foundation of China under Grant 61672273 and Grant 61832008.

Review summaries:
This work proposes a method for blind super-resolution (SR) of images degraded by arbitrary blur kernels. The method makes use of a novel blur kernel estimation technique, that proposes to estimate such kernels in the frequency domain following an end-to-end deep learning approach. The authors claim that the Fourier amplitude spectrum kernel representation is better posed for the learning task of kernel estimation than the spatial representation, under the assumption that the kernels are sparse both in frequency and space domains (this assumption holds form most common kernels in real settings). They support this claim with theoretical and experimental evidence.   The image formation model considered here, that produces the degraded LR image from the HR image, is the following. The clean high-resolution (HR) image first downsampled, then convolved with the blur kernel, and finally contaminated with additive noise.    Based on the proposed frequency domain representation, the authors propose a kernel generator network that allows to translate this representation to the kernel spatial representation. The generator $G$ takes as input the single-channel amplitude spectrum (the FFT) of the degraded LR image bi-cubicly downsampled by two, and outputs the single-channel blur kernel estimate. This generator consists of an encoder-decoder network built on a U-net with mirror connection between each of the seven down-sampling convolutional layers and up-sampling transposed convolutional layers with the same feature size.   To ensure that the kernel generated by $G$ is at the same time close to the ground truth kernel and consistent with the SR results, they propose a three-terms loss for $G$. The first term imposes similarity between the ground truth kernel and the kernel generated by G from the degraded image (in the spatial domain) by means of the $\ell_1$; the second term also imposes similarity between these two kernels but in seen as probability density functions, using an adversarial loss that involves a trainable critic $D$ that has to  discriminate between the ground truth kernel and the output of G (actually, de concatenation of each of them with the input of G are considered instead). This patch discriminator is trained using the loss proposed in LSGAN. Finally, the third term is a regularization loss on the output of G; unless I missed it, the regularizer is not specified. This kernel estimation model is called S2K.   Then, the authors propose a pipeline for blind SR that combines the S2K module with a non-blind SR module. For the non-blind SR module, they consider two methods:  the multiple degradation-based method SFTMD (Gu et al., CVPR 2019; ref 6) and the kernel modeling-based method RCAN (Zhang et al., ECCV 2018; ref. 35). Given a ground truth kernel of size 15x15 and a HR image, a LR degraded image is generated and cropped to 512x512. The input to the S2K module is the ground truth kernel upsampled to 256x256 and the FFT of the LR image, downsampled to 256x256. The estimated kernel of size 256x256 returned by G is then downsampled at size 15x15. Finally, the non-blind SR module is fed with the LR degraded image and the 15x15 kernel estimated with the S2K module.  The experiments are carried out considering Gaussian kernels of size 15x15 of random covariance matrix, and random motion kernels of size 23x23 (instead of 15x15) with Boracchi and Foi’s method (IEEE TIP, 2012; ref 3). The proposed approach is compared to an extensive list of state-of-the-art methods, both for the task of blur kernel estimation and blind SR for 2x, 3x and 4x SR. All methods were trained on the the same degraded dataset (DIV2K), including S2K and the non-blind methods used in the proposed blind SR pipeline. Blind SR methods are compared on synthetic images using the PSNR and SSIM metrics. Kernel estimation performance is evaluated using the $\ell_1$ norm and KL divergence. The results reported by the authors show that the proposed S2K and the blind SR pipeline significantly outperform the others. Finally, an ablation study on the loss components and frequency vs. spatial inputs is conducted, showing the pertinence of the proposed approach.   Finally, blind SR experiments on real images using the dataset proposed by Cai et al. (ICCV 2019; ref. 4), evaluated using two reference-free metrics (PIQE and BRISQUE) show that the proposed pipeline also outperforms its competitors, including the NITRE20 winner (Ji et al., CVPR 2020; ref. 10).  

The authors make a key observation that degradation kernels are highly correlated to degraded images in the frequency domain. From this, they design a kernel generator that directly generates the degradation kernel from the frequency-domain LR image. Then, a non-blind SR model is employed to utilize the estimated kernels for SR. The proposed method outperforms existing methods on two types of kernels (Gaussian and motion) with two backbone networks (SFTMD and RCAN).

This paper proposes a new blind super-resolution method, which can due with arbitrary blur kernels instead of Gaussian kernels. The whole method estimates kernels in the frequency domain. The method involves a kernel generator that takes the frequency amplitude as input and outputs a kernel. A patch discriminator is designed to perform adversarial loss to the predicted kernel. This design is novel and the experiments show good results.

The authors propose a novel framework for solving the problem of blind super resolution in which the low resolution input can be degraded by arbitrary blur kernels. In their framework, the degradation kernel estimation is performed in the frequency domain. The authors show that the frequency domain is better suited for kernel representation and reconstruction than the spatial domain and propose a Spectrum-to-Kernel mapping network to estimate diverse blur kernels. They use a conditional GAN to learn how to 'translate' the spectra of degraded images to the unknown kernels.