Spectrum-to-Kernel Translation for Accurate Blind Image Super-Resolution

1 Introduction . Single Image Super-Resolution ( SISR ) is a low-level visual task to recover High-Resolution ( HR ) images from its degraded Low-Resolution ( LR ) counterparts . Deep learning methods have significantly promoted SR research and achieved remarkable results on benchmarks [ 16 , 26 , 27 , 13 , 29 , 35 , 17 ] . Most of these works are based on the degradation model assuming that the LR image is down-sampled with blur kernel ( e.g. , bicubic kernel ) and additional noise from its HR source . Recent studies [ 6 , 11 ] show that pre-trained SR model is sensitive to the degradation of LR image . The mismatch for Gaussian kernels may cause SR result either over-sharpening or over-smoothing , and for motion kernels , it may bring unpleasant jitters and artifacts . Blind SR is to super-resolve a single LR image with its kernel unknown . It has attracted more and more attention due to its close relationship with the real scene . However , compared with the advanced non-blind SR researches , existing blind SR methods are insufficient to meet the needs of various degradation processes in real general scenes . ∗Work done during an intership at Tencent Youtu Lab . † Corresponding author ( lutong @ nju.edu.cn ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Some representative works have been proposed on this challenging problem . Bell-Kligler et al . estimate the kernel with internal-Generative Adversarial Network ( GAN ) on the basis of maximum similarity of patches across scales , whose performance is limited even with handcraft constraints for the weak prior fails to capture the kernel-related features directly [ 2 ] . Gu et al . propose an Iterative Kernel Correction ( IKC ) framework to correct the kernel estimation from the mismatch between kernel and SR result in an iterative way [ 6 ] . IKC is based on Maximum A Posteriori ( MAP ) , in which the estimator depends on the posterior of the pre-trained restorer . As demonstrated in [ 14 ] , due to the accuracy limitation of the restorer , the maximum performance of the estimator can not be guaranteed . Ji et al . introduce the frequency consistency constraint between the generated down-sampled image and the real input to estimate the blur kernel in spatial domain [ 11 ] . However , its training requires a certain amount of data with the same degradation and fails to capture blur diversities in various directions , limiting its application for a specific input LR image degraded with an arbitrary kernel . In this paper , we capture the convolution degradation feature of the local distribution in LR images from a novel perspective . Based on convolution theorem and sparsity analysis , we demonstrate the Fourier frequency spectrum of degraded LR image provides a robust shape structure of the degradation kernel in the frequency domain , which can be utilized to reconstruct the unknown kernel more accurately and robustly . Based on the above analysis , we design a Spectrum-to-Kernel ( S2K ) translation network to directly obtain the kernel interpolation result from degraded LR ’ s frequency spectrum . As far as we know , this is the first work to estimate kernel thoroughly in the frequency domain instead of in the spatial domain previously . The experimental results demonstrate that our method accurately estimates the target kernel and has a high universality capacity for diverse kernels . Moreover , when combined with the existing non-blind methods , we achieve the best results in quality evaluation and visual perception on synthetic and real images against existing blind SR methods . Figure 1 compares 4× blind SR results with recently proposed methods on real historical images . In general , our main contributions are summarized as follows : • We theoretically demonstrate that for common sparse kernels , it is more conducive to reconstruct it in the frequency domain instead of the commonly used spatial domain . • We analyze the impacts of kernel estimation optimization objectives for blind SR performance and design a novel network S2K to estimate kernel from Fourier amplitude spectrum . • When plugged into existing non-blind SR models , our proposed pipeline achieves the best performance on both synthetic and real-world images against existing blind SR methods . 2 Related Work . Non-blind SR Under the non-blind setting , HR image IHR is recovered from paired given paired LR image ILR and kernel k. Many successful pieces of research have been proposed where the bicubic kernel is used to construct paired training data , so they belong to the non-blind setting in essence . We divide those kernel-aware SR methods into two categories . ( 1 ) Multiple Degradation SR ( MDSR ) based methods concatenate degraded image ILR with stretched blur kernel k as SR model ’ s input [ 33 , 31 ] or conduct intermediate spatial feature transform according to the corresponding kernel to get target SR result [ 6 , 31 ] . ( 2 ) Kernel Modeling SR ( KMSR ) based methods incorporate the blur kernel in the model ’ s training parameters by constructing the paired kernel modeling dataset for the known test kernels to get a specific SR model for test data ’ s degradation restoration [ 36 , 10 ] . Inconsistency between ILR and blur kernel k leads to significant performance deterioration in test phase [ 33 ] . We propose a spectrum-to-kernel framework ( S2K ) that provides accurate kernel estimation to help both MDSR and KMSR based non-blind methods work well under blind settings . Blind SR Under blind setting , IHR is recovered from ILR degraded with unknown kernel . Since it is more closely related to real application scenarios , blind SR has attracted research attention for a long time [ 21 ] , mainly involving transforming unknown degraded image into target IHR ’ s bicubic down-sampled one , and non-blind SR methods with kernel estimation from spatial input ILR . In the former category , Hussein et al . modify the ILR with unknown k into that down-sampled from IHR by bicubic kernel [ 9 ] . Maeda et al . train a correction network to get the corresponding clean LR and construct pseudo pair to train the SR model [ 19 ] . In the latter category , Bell-Kligler et al . utilize the recurrence of patches across scales to generate the same-degraded down-sampled image as the test input [ 2 ] . However , the estimation accuracy is limited even with many handcrafted constraints due to the weak prior . Gu et al . use a kernel prediction network with a correction network taking the current kernel prediction and SR result of the pre-trained SR model as input to correct the kernel estimation iteratively [ 6 ] . Huang et al . adopt an alternating optimization algorithm with two modules restorer and estimator [ 8 ] . Ji et al . obtain the approximation of the blur kernel by utilizing the frequency domain consistency to estimate kernel in an unsupervised way [ 11 ] . Liang et al . propose Flow-based Kernel Prior ( FKP ) learning an invertible mapping between the kernel distribution and a tractable distribution to generate reasonable kernel initialization , improving the optimization stability [ 15 ] . Different from previous kernel estimation methods in the spatial domain , we demonstrate Fourier spectrum in frequency domain can provide a more effective kernel shape structure , which contributes to accurate and robust estimation for arbitrary kernels . Further , utilizing the shape structure relationship of the sparse kernel between the frequency domain and the spatial domain , our method can estimate diverse blur kernels end-to-end with implicit cross-domain translation . Besides , we propose SR-oriented kernel estimation optimization objection and obtain a more accurate estimation of arbitrary unknown blur kernels directly and independently . When combining S2K with existing advanced non-blind SR methods , we achieve the best quantitative and qualitative performance on synthetic and real-world images against existing blind SR methods . 3 Method . In this section , we first formulate the problem , then theoretically analyze the advantage of blur kernel estimation from the frequency domain and the feasibility of end-to-end translation learning . Then we introduce the pipeline and component of the proposed S2K network in detail . Finally , we present the blind SR pipeline combined with the proposed kernel estimation module . 3.1 Problem Formulation . Same as [ 33 , 34 , 11 ] , we assume the image degradation process from IHR to ILR is as follows : ILR = ( IHR↓s ) ⊗ k+ n , ( 1 ) where k denotes degradation kernel , and n is additional noise3 . ↓s and ⊗ are down-sampling and convolution operations . Under non-blind setting , both k and n are known , denoting the pre-trained SR model as R , the upper bound of SR result I∗SR can be expressed as : I∗SR = R ( ILR | k , n ) . ( 2 ) Under blind setting , the degradation prior needs to be obtained from ID . In this paper , we mainly focus on the estimation of degradation kernel from the degraded input ILR , because the extra denoising method can be easily plugged in from the previous denoising methods [ 7 , 10 ] . Theoretically , the optimal kernel estimation target k∗ for blind SR performance is defined as follows : k∗ = argmin kest ‖R ( ILR | kest ) − I∗SR‖ . ( 3 ) 3Above variables are all matrices , and blur kernel size is far less than LR image size . Accurate kernel estimation is the key to make the most of the powerful performance of existing nonblind methods under the blind setting . However , it is difficult for its ill-posed property . Meanwhile , the non-blind SR models are very sensitive to the consistency of input { ILR , kest } pairs , and subtle degradation inconsistencies may cause obvious artifacts [ 6 ] . We explored the influence of the kernel estimation optimization target on the performance of subsequent non-blind SR to ensure the consistency of kest and ID , and get the result close to the upper bound I∗SR in the following parts . 3.2 Spectrum-to-Kernel . Different from previous kernel estimation methods in spatial domain , S2K is the first deep-learningbased attempt to conduct kernel estimation entirely in the frequency domain . Shape structure is conducive to the reconstruction task learning : Chen et al . obtain better face SR results with shape prior [ 5 ] , Ma et al . maintain the gradients structure , which proves to be beneficial to the SR result [ 18 ] . Some researchers utilize structure information to improve the performance of image inpainting [ 25 , 22 ] . We find the kernel representation obtained from the Fourier spectrum can provide more robust and favorable feature from shape structure than spatial patches . We below give a formal proof that kernel estimation in the frequency domain is more conducive than in the spatial domain under the reasonable assumption that blur kernels of different forms satisfy sparsity . Advantages of the Frequency Domain In frequency domain ( Fourier frequency domain unless specified ) , denoting clean images , degraded images , and degradation kernels as S , G and F respectively . According to the convolution theorem , denoting hadamard product as , we have : G = F S. ( 4 ) We adopt φ ( X , Y ) to measure shape similarity of two matrix variables X , Y [ 12 , 32 ] as follows : φ ( X , Y ) = ‖δτ ( X−Y ) ‖0 , ( 5 ) where δτ ( · ) is a truncated function to ignore the extremely small divergence in each elements and ‖ · ‖0 denotes ` 0-norm operator . When the absolute value of element x in ( X −Y ) is less than τ ( positive but very small ) , it returns 0 , otherwise returns the absolute value of x . Under the assumption that common kernels are sparse in both the frequency and spatial domains , we compare the shape structure similarity of the input and output for kernel estimation in the frequency and spatial domains , respectively . In the frequency domain , we have : φ ( G , F ) = ∑ Fij < τ sign ( δτ ( FijSij − Fij ) ) + ∑ Fij≥τ sign ( δτ ( FijSij − Fij ) ) ≤ ∑ Fij < τ sign ( δτ ( Fij ( Sij − 1 ) ) ) + ∑ Fij≥τ sign ( δτ ( Fij ) ) = 0 + ∑ Fij≥τ sign ( δτ ( Fij ) ) = ‖δτ ( F ) ‖0 . ( 6 ) Here , if x is positive , sign ( x ) > 0 , else if x is negative , sign ( x ) < 0 , else sign ( x ) = 0 , and i , j denote position coordinates in spectrum . Eq . 6 gives the upper bound of the distance of the shape structure from the degraded input to the output kernel in the frequency domain . While in the spatial domain , denoting the zero-padded degradation kernel with kp which has the same size of the degraded image ILR , we have : φ ( kp , ILR ) = ∑ [ kp ] ij < τ sign ( δτ ( [ kp ] ij − [ ILR ] ij ) ) + ∑ [ ksp ] ij > τ sign ( δτ ( [ kp ] ij − [ ILR ] ij ) ) ≥ ∑ [ kp ] ij < τ sign ( δτ ( [ kp ] ij − [ ILR ] ij ) ) ≥ ∑ [ kp ] ij < τ , [ ILR ] ij > 2τ sign ( δτ ( [ kp ] ij − [ ILR ] ij ) ) ≥ ‖δ2τ ( ILR ) ‖0 − ‖δτ ( kp ) ‖0 , ( 7 ) which gives the lower bound of the distance of the shape structure on the spatial domain . Further , since both F and kp are sparse , ‖δτ ( F ) ‖0 and ‖δτ ( kp ) ‖0 are significantly smaller than ‖δ2τ ( ILR ) ‖0 due to the non-sparsity of ILR . In this case , ‖δτ ( F ) ‖0 ‖δ2τ ( ILR ) ‖0 − ‖δτ ( kp ) ‖0 . Thus we have : φ ( G , F ) φ ( kp , ILR ) . ( 8 ) Eq . 8 reveals that the divergence between the blur kernel and its corresponding degraded image in the frequency domain is significantly smaller than that in the spatial domain . As illustrated in Figure 2 , compared with the spatial domain , G and F have a more significant and robust kernel shape correlation , which is conducive for the kernel reconstruction in the frequency domain . After demonstrating the shape advantage for general kernel estimation from the frequency domain over the spatial domain , we further analyze the cross-domain relationship of the kernel ’ s shape structure . Taking the Gaussian kernel as the example , according to Fourier transform property of 2D Gaussian function , kernel in the frequency domain is also in Gaussian function form , and its variances are inversely proportional with that in spatial . Using the notation in Figure 3 left , we have : σu = C1/σx , ( 9 ) σv = C2/σy . ( 10 ) Here , C1 , C2 refer to two constants4 . It reveals that it is feasible to learn the cross-domain translation implicitly and estimate spatial kernel from the degraded image in the frequency domain with spectrum in an end-to-end manner . Experimentally , we find it still works for motion and disk kernels .

Summary: This work proposes a method for blind super-resolution (SR) of images degraded by arbitrary blur kernels. The method makes use of a novel blur kernel estimation technique, that proposes to estimate such kernels in the frequency domain following an end-to-end deep learning approach. The authors claim that the Fourier amplitude spectrum kernel representation is better posed for the learning task of kernel estimation than the spatial representation, under the assumption that the kernels are sparse both in frequency and space domains (this assumption holds form most common kernels in real settings). They support this claim with theoretical and experimental evidence.   The image formation model considered here, that produces the degraded LR image from the HR image, is the following. The clean high-resolution (HR) image first downsampled, then convolved with the blur kernel, and finally contaminated with additive noise.    Based on the proposed frequency domain representation, the authors propose a kernel generator network that allows to translate this representation to the kernel spatial representation. The generator $G$ takes as input the single-channel amplitude spectrum (the FFT) of the degraded LR image bi-cubicly downsampled by two, and outputs the single-channel blur kernel estimate. This generator consists of an encoder-decoder network built on a U-net with mirror connection between each of the seven down-sampling convolutional layers and up-sampling transposed convolutional layers with the same feature size.   To ensure that the kernel generated by $G$ is at the same time close to the ground truth kernel and consistent with the SR results, they propose a three-terms loss for $G$. The first term imposes similarity between the ground truth kernel and the kernel generated by G from the degraded image (in the spatial domain) by means of the $\ell_1$; the second term also imposes similarity between these two kernels but in seen as probability density functions, using an adversarial loss that involves a trainable critic $D$ that has to  discriminate between the ground truth kernel and the output of G (actually, de concatenation of each of them with the input of G are considered instead). This patch discriminator is trained using the loss proposed in LSGAN. Finally, the third term is a regularization loss on the output of G; unless I missed it, the regularizer is not specified. This kernel estimation model is called S2K.   Then, the authors propose a pipeline for blind SR that combines the S2K module with a non-blind SR module. For the non-blind SR module, they consider two methods:  the multiple degradation-based method SFTMD (Gu et al., CVPR 2019; ref 6) and the kernel modeling-based method RCAN (Zhang et al., ECCV 2018; ref. 35). Given a ground truth kernel of size 15x15 and a HR image, a LR degraded image is generated and cropped to 512x512. The input to the S2K module is the ground truth kernel upsampled to 256x256 and the FFT of the LR image, downsampled to 256x256. The estimated kernel of size 256x256 returned by G is then downsampled at size 15x15. Finally, the non-blind SR module is fed with the LR degraded image and the 15x15 kernel estimated with the S2K module.  The experiments are carried out considering Gaussian kernels of size 15x15 of random covariance matrix, and random motion kernels of size 23x23 (instead of 15x15) with Boracchi and Foi’s method (IEEE TIP, 2012; ref 3). The proposed approach is compared to an extensive list of state-of-the-art methods, both for the task of blur kernel estimation and blind SR for 2x, 3x and 4x SR. All methods were trained on the the same degraded dataset (DIV2K), including S2K and the non-blind methods used in the proposed blind SR pipeline. Blind SR methods are compared on synthetic images using the PSNR and SSIM metrics. Kernel estimation performance is evaluated using the $\ell_1$ norm and KL divergence. The results reported by the authors show that the proposed S2K and the blind SR pipeline significantly outperform the others. Finally, an ablation study on the loss components and frequency vs. spatial inputs is conducted, showing the pertinence of the proposed approach.   Finally, blind SR experiments on real images using the dataset proposed by Cai et al. (ICCV 2019; ref. 4), evaluated using two reference-free metrics (PIQE and BRISQUE) show that the proposed pipeline also outperforms its competitors, including the NITRE20 winner (Ji et al., CVPR 2020; ref. 10).