Spectrum-to-Kernel Translation for Accurate Blind Image Super-Resolution
Authors: Guangpin Tao
Xiaozhong Ji
Wenzhuo Wang
Shuo Chen
Chuming Lin
Yun Cao
Tong Lu
Donghao Luo
Ying Tai
Venue: NeurIPS 2021 Poster (2021)
Keywords: deep learning
computer vision
blind kernel estimation
blind super-resolution

Abstract: Deep-learning based Super-Resolution (SR) methods have exhibited promising performance under non-blind setting where blur kernel is known; however, blur kernels of Low-Resolution (LR) images in different practical applications are usually unknown. It may lead to a significant performance drop  when degradation process of training images  deviates from that of real images. In this paper, we propose a novel blind SR framework to super-resolve LR images degraded by arbitrary blur kernel with accurate kernel estimation in frequency domain. To our best knowledge, this is the first deep learning method which conducts blur kernel estimation in frequency domain. Specifically, we first demonstrate that feature representation in frequency domain is more conducive for blur kernel reconstruction than in spatial domain. Next, we present a Spectrum-to-Kernel (S$2$K) network to estimate general blur kernels in diverse forms. We use a conditional GAN (CGAN) combined with SR-oriented optimization target to learn the end-to-end translation from degraded images' spectra to unknown kernels. Extensive experiments on both synthetic and real-world images demonstrate that our proposed method sufficiently reduces blur kernel estimation error, thus enables the off-the-shelf non-blind SR methods to work under blind setting effectively, and achieves superior performance over state-of-the-art blind SR methods, averagely by 1.39dB, 0.48dB (Gaussian kernels) and 6.15dB, 4.57dB (motion kernels) for scales $2\times$ and $4\times$ respectively.

Spectrum-to-Kernel Translation for Accurate Blind Image Super-Resolution
Guangpin Tao
Xiaozhong Ji
Wenzhuo Wang'
Shuo Chen
Chuming Lin²
Yun Cao²
Tong Lu
Donghao Luo²
Ying Tai²
National Key Lab for Novel Software Technology, Nanjing University Tencent Youtu Lab 'RIKEN Center for Advanced Intelligence Project
Abstract
Deep-learning based Super-Resolution (SR) methods have exhibited promising performance under non-blind setting where blur kernel is known. However, blur kernels of Low-Resolution (LR) images in different practical applications are usually unknown. It may lead to significant performance drop when degradation process of training images deviates from that of real images. In this paper, propose a novel blind SR framework to super-resolve LR images degraded by arbitrary blur kernel with accurate kernel estimation in frequency domain. To best knowledge, this is the first deep learning method which conducts blur kernel estimation in frequency domain. Specifically, first demonstrate that feature representation in frequency domain is more conducive for blur kernel reconstruction than in spatial domain. Next, present Spectrum-to-Kermee estimate general blur kernels in diverse forms. We use a Conditional GAN (CGAN) combined with SR-oriented optimization target learn the end-to-end translation from degraded images' spectra to unknown kernels. Extensive experiments on both synthetic and real-world images demonstrate that our proposed method sufficiently reduces blur kernel estimation eтror, thus enables the off-the-shelf non-blind SR methods work under blind setting effectively, and achieves superior performance over state-of- srate-of-the-att the-art blind SR methods, averagely by 1.39dB, 0.48dB commom blind SR setting (with Gaussian kernels) for scales 2x and 4x, respectively.
(S2K) network
Introduction
Single Image Super-Resolution (SISR) is low-level visual task recover High-Resolution (HR) im-
ages from its degraded Low-Resolution (LR) counterparts. Deep learning methods have significantly
promoted SR research and achieved remarkable results benchmarks [16] 26 27
3 29
35 I7].
Most of these works are based on the degradation model assuming that the LR image is down-sampled with blur kernel (e.g., bicubic kernel) and additional noise from its HR source. Recent studies [ show that pre- traitre SR model is sensitive to the degradation of LR image. The mismatch for Gaussian kernels may cause SR result either over-sharp or wverssmoohinng and for motion kernels, it may bring unpleasant jitters and artifacts. Blind SR is to super-rees resolve single LR image with its kernel unknown. has attracted more and more attention due to close relationship with the real scene. However, compared with the advanced non-blind SR researches, existing blind SR methods are insufficient to meet the needs of various degradation processes in real general scenes.
Work done during intership Tencent Youtu Lab. Corresponding author (lutong @nw..cdu.cn)..
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

HIM
CRUSEL HUT
HOME
HOME
ANTICH DEW
AAITIEE OTWY
ANTICCH DOWY
ANTIOCH DEWY
ANTICH DEWy
ANTIOW DEW
THE
THE
THE
NT
THE
NT
THE
NT
Binabc
ESRGAN (BOCVW 2012
DAN (NainPP 2033
Comation Filter (CVPR 20000)
DUFFX (CVPR
Ours
Figure I: 4x blind SR results comparison on real historic images. Our method achieves the best visual perception in recovery of real-scene objects such clearer text shape and building structures. LR patches are cropped from images in http://vllabbucmerree wwwp:..wwwweeeeeeweeee.eew wttp///aai////SSNNN.N
Some representative works have been proposed on this challenging problem. Bell-Kligler et al. estimate the kernel with similarity of patches across scales, whose performance is limited even with handcraft constraints for the weak prior fails to capture the kernel-related features directly (2). Gu et al. propose an Iterative Kernel Correction (IKC) framework to correct the kernel estimation from the mismatch between kernel and SR result in an iterative way 6). IKC is based on Maximum A Posteriori (MAP), in which the estimator depends the posterior of the pre-trained restorer. As demonstrated in [4], due the accuracy limitation of the restorer, the maximum performance of the estimator cannot be guaranteed. Ji al. introduce the frequency consistency constraint between the generated down-sampled image and the real input to estimate the blur kernel in spatial domain [I]. However, its training requires certain amount of data with the same degradation and fails capture blur diversities in various directions, limiting its application for a specific input LR image degraded with an arbitrary kemel..
Adversarial Network (GAN) on the basis of maximum
In this paper, we capture the convolution degradation feature of the local distribution in LR images from a novel perspective. Based on convolution theorem and sparsity analysis, we demonstrate the Fourier frequency spectrum of degraded LR image provides robust shape structure of the degradation kernel in the frequency domain, which can be utilized to reconstruct the unknown kernel more accurately and robustly. Based on the above analysis, we design Spectrum- to-nnenee (S2K) translation network to directly obtain the kernel interpolation result from degraded LR's 's frequency spectrum. As far as we know, this is the first work to estimate kernel thoroughly in the frequency domain instead of in the spatial domain previously. The experimental results demonstrate that our method accurately estimates the target kernel and has a high universality capacity for diverse kernels. Morcover, when combined with the existing non-blind methods, we achieve the best results in quality evaluation and visual perception on synthetic and real images against existing blind SR methods. Figure I compares 4x blind SR results with recently proposed methods on real historical images.
In general, our main contributions are summarized follows:
We theoretically demonstrate that for common sparse kernels, it is more conducive to reconstruct in the frequency domain instead of the commonly used spatial domain.
We analyze the impacts of kernel estimation optimization objectives for blind SR performance and design a novel network S2K to estimate kernel from Fourier amplitude spectrum.
When plugged into existing non-blind SR models, our proposed pipeline achieves the best perfor- mance on both synthetic and real-world images against existing blind SR methods.
2 Related Work
Non-blind SR
Under the non-blind setting, HR image L is recovered from paired given paired LR image IL and kernel k. Many successful pieces of research have been proposed where the bicubic kernel is used to construct paired training data, so they belong to the non-blind setting in essence. We divide those kernel-aware SR methods into two categories. (1) Multiple Degradation SR (MDSR) based methods concatenate degraded image I with stretched blur kernel as SR model's input 33 31 or conduct intermediate spatial feature transform according to the corresponding kernel

---

to get target SR result 6 B1 (2) Kernel Modeling SR (KMSR) based methods incorporate the blur kernel in the model's training parameters by constructing the paired kernel modeling dataset for the known test kernels to get specific SR model for test data's is degradation restoration [36 10] Inconsistency between In and blur kernel k leads to significant performance deterioration in test phase [33]. We propose a spectrun-to-kemel framework (S2K) that provides accurate kernel estimation to help both MDSR and KMSR based non-blind methods work well under blind settings.
Blind SR Under blind setting, LA is recovered from LLR degraded with unknown kernel. Since it more closely related to real application scenarios, blind SR has attracted research attention for long time [21], mainly involving transforming unknown degraded image into target n bicubic down-sampled one, and non-blind SR methods with kernel estimation from spatial input Ir.z-
In the former category, Hussein et al. modify the LR with unknown k into that down-sampled from IhR by bicubic kernel [9] Maeda et al. train correction network get the corresponding clean LR and construct pseudo pair to train the SR model [1]]. In the latter category, Gell-Kligler et al. utilize the recurrence of patches across scales to generate the same-degraded down-sampled image as the test input (2). However, the estimation accuracy is limited even with many handcrafted constraints due to the weak prior. Gu et al. use a kernel prediction network with a correction network taking the current kernel prediction and SR result of the pre-trained SR model as input to correct the kernel estimation iteratively []. Huang et al. adopt an alternating optimization algorithm with two modules restorer and estimator g Ji et al. obtain the approximation of the blur kernel by utilizing the frequency domain consistency to estimate kernel in an unsupervised way [1] Liang el al. propose Flow-based Kernel Prior (FKP) learning an invertible mapping between the kernel distribution and tractable distribution to generate reasonable kernel initialization, improving the optimization stability [5].
Different from previous kernel estimation methods in the spatial domain, we demonstrate Fourier spectrum in frequency domain can provide more effective kernel shape structure, which contributes to accurate and robust estimation for arbitrary kernels. Further, utilizing the shape structure rela- tionship of the sparse kernel between the frequency domain and the spatial domain, our method can estimate diverse blur kernels end-to-end with implicit cross-domain translation. Besides, we propose SR-oriented kernel estimation optimization objection and obtain a more accurate estimation of arbitrary unknown blur kernels directly and independently. When combining S2K with existing advanced non-blind SR methods, we achieve the best quantitative and qualitative performance on synthetic and real-world images against existing blind SR methods.
3 Method
In this section, we first formulate the problem, then theoretically analyze the advantage of blur kernel estimation from the frequency domain and the feasibility of end-to-end translation learning. Then we introduce the pipeline and component of the proposed S2K network in detail. Finally, we present the blind SR pipeline combined with the proposed kernel estimation module.
3.1
Problem Formulation
Same as
33
34
we assume the image degradation process from IHR R to IL is as follows:
I = (LHRl,) 8 + n,
where k denotes degradation kernel, and n is additional noise te and @ are down-sampling and convolution operations. Under non-blind setting, both k and are known, denoting the pre-trained SR model as R. the upper bound of SR result sn can be expressed as:
F
IS R(I.R k,
Under blind setting, the degradation prior needs to be obtained from I In this paper, we mainly focus on the estimation of degradation kernel from the degraded input I because the extra denoising
method can be casily plugged in from the previous denoising methods 77 10). Theoretically, the optimal kernel estimation target k* for blind SR performance is defined as follows:
k" arg min |
kest) - Is
(3)
k.at
Above variables are all matrices, and blur kernel size is far less than LR image size.
3

---

a.
Spatial
image
Frequency spectrum
Clesn
Gaussion kermel
Gaussian blured
Disk kereel
Disk blurned
Motion kemel
b.
Spottal
C.
Fooquency
nGB
Sum mostrieE...
Footure Pnauredistton distonce
AA
100
100
2003
300
400
3001
Samples
Coordinate
Figure 2: (a). Relationship between LHR, k and IL in frequency and spatial domain. Fourier spectrum of IR indicates shape kernel in frequency domain, which provides significant and robust kernel representation. However, spatial distribution show minor connection between LLæ and k, and the blurring caused by the same kernel k in different positions the IL may be different (as indicated in red and yellow boxes), leading to the instability of blur kemnll estimation training. (b). Pre-trained VGG19 feature distance between kernel estimation's 's input and output in spatial domain and frequency domain respectively. (c). For single image, the pixel intensity sum (along one dimension) distance between estimation input and the target kernel in different domains.
Accurate kernel estimation the key to make the most of the powerful performance of existing non- blind methods under the blind setting. However, it is difficult for its ill-posed property. Meanwhile, the non-blind SR models are very sensitive to the consistency input [I,R, kest] pairs, and subtle degradation inconsistencies may cause obvious artifacts (6] We explored the influence of the kernel estimation optimization target on the performance of subsequent non-blind SR to ensure consistency of kest and ID. and get the result close to the upper bound Isn in the following parts.
3.2
Spectrum-to-Kernell
Different from previous kernel estimation methods spatial domain, S2K is the first deep-learning- based attempt conduct kernel estimation entirely in the frequency domain. Shape structure conducive to the reconstruction task learning: Chen el obtain better face SR results with shape prior [5], Ma et al. maintain the gradients structure. which proves to be beneficial the SR result [8]. Some researchers utilize structure information to improve the performance of image inpainting 25 22 We find the kernel representation obtained from the Fourier spectrum can provide more robust and favorable feature from shape structure than spatial patches. We below give formal proof that kernel estimation in the frequency domain is more conducive than in the spatial domain under the reasonable assumption that blur kernels of different forms satisfy sparsity.
Advantages of the Frequency Domain In frequency domain (Fourier frequency domain unless specified), denoting clean images, degraded images, and degradation kernels as S, G and respec- tively. According to the convolution theorem, denoting hadamard product O, have:
G = FOO S.
(4)
We adopt o(X, Y) to measure shape similarity of two matrix variables X, Y
12 32
as follows:
oXX Y) = || Y)llu,
(5)
where 8-(-) a truncated function to ignore the extremely small divergence in each elements and ||- llo denotes to fo-mrrm operator. When the absolute value of element x in (X Y) is less than

---

(positive but very small), it returns O, otherwise returns the absolute value of Under the assumption that common kernels are sparse in both the frequency and spatial domains, we compare the shape structure similarity of the input and output for kernel estimation in the frequency and spatial domains, respectively. In the frequency domain, we have:
G,
F)
Σ
(ign(o((FF))) - Fiy)) +
Σ
si (F,, - Fiy))
F Py
F Fu27
Σ
sign 6, (F (Fi 1))) Σ
sign(6; (6, (Ft))
(6)
25
= 0 +
sign(6, (gn(6,(F;))) (Fiy)) = 1|,, (F)
Fg2T
Here, if is positive, sign(x) i.j denote position coordinates in spectrum. Eq. 6 gives the upper bound of the distance of the shape structure from the degraded input to the output kernel in the frequency domain. While in the spatial domain, denoting the zero- paro-padded degradation kemnel with kp which has the same size of the degraded image LRR, have:
else if æ is negative, sign(x) else sign(x) 0, and
Φ(kp. p: IAR = Σ
sign(6, ((kpk - (TRR())) +
Σ [keples>7
sign(0, (kpls - (InRl)))
Σ
[IRR)))
Σ
sign (8, (kkpl [RRl)))
(7)
<p
Kplust, [LR)y>2r
2 1|22r (IoR)l
| (kp)llo
which gives the lower bound of the distance of the shape structure on the spatial domain. Further, since both F and kp are sparse, ||,, (F) and [|,, (k,) are significantly smaller than ||02, (LLR)llo llo due to the non-spornee non-sparsity of LLR. this case, ||6, (F) < |162r (ILx)llo ||,, (kp) llo- Thus we have:
(G, < @(kp, IRR)
Eq. 8 reveals that the divergence between the blur kernel and its corresponding degraded image in the frequency domain is significantly smaller than that in the spatial domain. As illustrated in Figure 2 compared with the spatial domain, G and F have a more significant and robust kernel shape correlation, which is conducive for the kernel reconstruction in the frequency domain.
After demonstrating the shape advantage for general kernel estimation from the frequency domain over the spatial domain, we further analyze the cross-domain relationship of the kernel's 's shape structure. Taking the Gaussian kernel as the example, according Fourier transform property of 2D Gaussian function, kernel in the frequency domain is also in Gaussian function form, and its variances are inversely proportional with that in spatial. Using the notation in Figure left, we have:
T₂ = C1/02.
(9)
o, C2/ay
(10)
Here, C1, C2 refer two constants" reveals that is feasible learn the cross-doman translation implicitly and estimate spatial kernel from the degraded image in the frequency domain with spectrum an end-to-end manner. Experimentally, we find it still works for motion and disk kernels.
3.3 Spectrum based Kernel Estimation
Based on the analysis above, we propose new kernel estimation method from the frequency domain, which makes full use of the shape structure from In spectrum to reconstruct the kernel. We further utilize kernel's 's shape relationship to learn the cross-domain translation, thus estimating the unknown spatial kernel from the frequency spectrum end-to-end.
More details provided in supplementary materials.
5

---

276
16
Spadill denain Lan
Trqgeny Lik
Inpus
Eraseber
Deaeler
Osiggtt
Figure 3: Lelf: A. Shape structure from G conducive to the reconstruction of kernel spectrum F in the frequency domain. B. Shape correlation of the Gaussian kernel between Fourier domain and spatial domain: variances are inversely proportional, respectively. C. Based on A and B, it is feasible to estimate the spatial kernel k from the degraded image's spectrum G. Right: Visualization of intermediate feature maps of different sizes of the pre-trained S2K. We show the shape structure from the degraded LR image (as marked in red) conducive kernel estimation in frequency domain.
12383581
Rest
Ricasc nddstiodee
dactitiinttt
Paasee
Korad penevater
(292.29)
Kess
Degald
Concatonali
Figure 4: Left: Pipeline of proposed S2K with conditional kernel generator and a patch discriminator. Right: Illustration two toy examples to show the limitations of using pixel loss & (a.) and relative shape loss l, (b.) alone respectively, where optimization objective of kernel estimation inconsistent with subsequent SR performance. In case (a.), k!.r and k2.r share the same & distance to kgt. but obtain more blurry SR result with k. than kLst- In case (b.). kgst and ke have the same shape distance kgt, but inconsistent with SR results. We find the combination of the pixel-level value loss and relative shape loss is more beneficial to subsequent SR performance with estimated kernel.
Kernel Generator Unlike the previous methods estimating kernel's parameter from spatial input, we obtain the target kernel in spatial from the frequency amplitude. The Generator an encoder- decoder structure (Figure 4) and takes the 256x256 single-channel amplitude spectrum of the LR image the input. After through an input layer with stride 2, the feature is sent into U-net where mirror connection is between each of the seven down- sampling Conv layers and up-sampling transposed Conv layers of the same feature size. Finally, we obtain the single-channel estimation kernel map through the output layer. As discussed in Figure 4 right, we use relative shape loss E, and smoothing loss Greg in addition to the é loss ensure that the estimated kernel close to ground truth and favorable for SR results. For imput pair {s,, k,) (s; denotes spectrum IRR)). to make the generator output G(s;)) closer to target k,, the generator's 's loss l consists of three parts:
tG Eg,k, ki (A k G(s:)) (s( G(s)) 112 \cR((G(s))),
(11)
Creg
where (: denotes feature concatenating and denotes the discriminator, R denotes regularization item, X; denotes weight of respective loss component. The first loss l pushes G(s.) pixel distance closer to target, the second loss f, makes G(s,) satisfy the target kernel's 's spatial surface shape. We use adversarial loss to make the generated results conform to target kemne''s I's distribution, serving as the shape loss. The third total variation loss lreg regularizes the output with smoothing regularization.
Patch Discriminator We use cully-convolutional patch discriminator to learn the distribution of different kernel forms. To make the training more stable, we adopt variant of LSGAN [20) where the least square loss is calculated between the patch embeddings of the predicted kernel and the

---

DIV2K 28
Flicke22]
Method
Kerel
4x
3x
RCAN linetuned
24.92 0,6903
70.3970.630
22.37/0.5977
24.88 / 70.6832
23,46/0.6333
22.47/0.6011
7SSR DeblurGAN RCAN KemelGAN FCA RCAN
24.90
23.36/
22.36/0.5974
24.91 /0.6907
23,43/0.6332
22.46/ 0.6013
25.38 0.2000
22.64/0.0039
25.39 /0,7091
23.85 /0.6465
22.72/0.6008
25.53/
23.57/ 0.6370
22.97 /..613
25.48 25.48 /0.7126
23.66/0.6399
23.09 0.6117
Gaussion
26.19 0..160
0.7332
24.93/0.6873
28.45 0.81660
26.30/0.7414
24.81 / 0..690
FKP
/0.13/0
26.05/ 0.7219
24.82 0.6678
28.40 /08188
26.22 00.442
24.68 0.6912
25.56 0.7595
26.72/
/.7114
S2K SETMD
29.31 0..359
27.19/ 0.7550
25.84
7.70
29.38 /0.3411
26. 26.96/ 00,7582
25.79 0.7143
w. RCAN
29.77/0.8444
27.47 0.7674
26.05
79.7
29,91 /0,8469
27.39 0.7639
26.10/0.714
RCAN linetuned
23.34/0.6522 22.06 / 0.6080
705572 22.04
23.43/ 0.6535
22.11 / 00.599
21.49 0,5803
DeblurGAN w. RCAN
23.30 0.6555 22.24 0.6096
21.24 0.5778
23.22 0.6550
22.10
0.6018
21 21.40/ 0.5801
ZSSR
23.33
22.25 0.6074
21.32 0.5785
00.6907
/.66332
0.6013
KemelGAN
22.40 0.6080 22.17 0.6015
20.45 0.5414
22.40 / 0.661
22.00 0.5935
20 56 0.5363
IKC
Motion
24.96 0.7543 22.27
0.6357
22.11
633630
70 /0, 7352
21.95 0.6244
21 21.89 0.6244
Pan of.
23
W. SFTMD
24.10 0.6729 20.04/ / 0.5539
19.21
0.5318
21,04 / 0.599
19,90/ 0,55229
18.80/ 0.5168
Pan et al.
24
w. SFTMD
21.50 0,5673 20.37 / 0.5113
0.4765 19.63
21.28 / 0.5538
20.22 00.5182
19.83 0.4932
FKP
24.38 0.6897
22.39 0.6273
21.02/
0 085887
24.43 0.6902
22.41 /0.6268
21.10/0.701
S2K SFTMD
30.95 0.8870 28.28 0.8118 26.95 0.7543
31.21 /0.991
28.32 008212 26.70 0.7557
Table I: Quantitative [PSNRt SSIM[] comparison results of fidelity-orietted SR model for 2x,
3x, 4x up-sampling. The best performance shown red and the second best in blue.
corresponding ground truth (both concat. with s,). Discriminator's 's training loss can be written as:
tD Es, &. ((((s: + (D(s.:k) 1)²].
(12)
3.4 Pipeline of Blind SR
Based the proposed S2K kernel estimation module shown in Figure 4 We further introduce our basic pipeline for blind SR. For kernels defined by continuous parameters iee, Gaussian kernels and disk kernels (3 and I parameters). both MDSR and KMSR (36) (without kernel expansion) based non-blind models are applicable. For motion kernels with more discrete and irregular form, we use MDSR based SR model 66 to get blind SR result taking IL and estimated kernel k as input.
Experiments
Experiment Setup In theory, method can handle general sparse kernels. We select the two most commonly used blur kernels in image processing, i.e., Gaussian and motion kernels to conduct blind SR experiments. We set Gaussian kernel size 15 and variance between 1.0 and 3.0, rotation angle between and 2T. We randomly select from 500 motion kernels using code from []. and the kernel size is 23, exposure time is set in [0.15, 0.3), anxiety is 0.005. In addition, we add estimation of disk kernels and use Matlab function /special to randomly generate disk kernel according to the input radius between 1.0 and 3.0. All kernel-aware methods compared are trained the same dataset degraded with random kernels (three types of random kernels are estimated separately). We use DIV2K (28) training set train S2K and the non-blind SR model. Then, we use random blurred DIV2K validation set, 100 images in Flicker2K (1] as test datasets in the synthetic experiment, and conduct 2x, 3x, 4x SR comparison and kernel estimation accuracy comparison. Finally, we use our pre-trained estimation model of Gaussian kernels to perform 4x SR test on the real image from RealSR and historical photos. We provide qualitative and quantitative comparisons with state-of thr- methods. We set the weights of loss item in generator as a = 100, x = 1, Ag = respectively. For optimization, we use Adam with 3 = 0.5, B2 0.999, and learning rate 0.001.
4.1 Blind SR on Synthetic Images
We combine proposed S2K with non-blind methods and provide comparisons with existing methods in [ Due the inconsistency of training (clean dataset) and test (unknown degradation) domain, the performance of ordinary ideal kernel-awareless SR models (e.g., RCAN, ZSSR) suffers a significant deterioration on the test data. The performance of blind methods trying to correct test LR image to get clear LR image (e.g., DeblurGAN w. RCAN) is limited because down-sampling in degradation process makes the recovery of high-frequency components more difficult in LR image. KemelGAN and FKP obtain more accurate predictions than bicubic interpolation kernel, but the
7

---

0946 DIV2X (Cazzeim lasmel)
7SSR
KemdGAN
FKP
IKC
RCA
Ourr
MATT
BHD
BONE
000015. Rickx22K (Miico
2SSR
KemelGAN
Pan et.
IKC
Ours
Figure 5: 2x blind SR results on synthetic images with oftte-of-thearr methods. For Gaussian and motion blurred images, our method obtains results with more clarity and less jitter effects.
Methods
Gaussian kernll
Motion kermll
Disk keamel IK
D, (Ka,. Kg) GazssinnMMttoooolkk
E K
KE Ke.l
IKEu Kg.l
Kal
Cansiaa/Motn/DDae
Ko:
Xu al.
0:2813044555001.99
144.24228(9933353
9
0.12090.3705001188
1682/466TYV1554
KemelGAN
6
6
021460.30220.3310
27:27/51//7927722
FKP
0.98390-4451/055229
129.0088847/777888
0,17740.0672000009999
06789/227006337
Figure 6: Kernel estimation comparison on 3x degraded LR of DIV2K dataset with random kernels. Compared with [30 23 [5 Ours can get more accurate blur kernel estimation results close GT.
inadequate estimation accuracy degrades the subsequent SR model's performance. Using the same pre- pre-trained SFTMD model, we obtain better results against IKC, for our estimation kernel model is not based on the pre-trained MDSR model but the frequency domain. FCA unable to deal with general the kernel estimation of the specific degraded image while method can estimate kernels more accurately without these restrictions. Visual comparison results are provided Figure
Kernel Estimation We provide the & distance D, and relative shape distance D, in Eq. between predict kernel and the ground truth on 100 test ILR images. Here h, w denote the height and width of the kernel, i, j denote position indices, respectively.
D.(X Y)
Σ Xi,, Ye,l, DA(X, r) XI,
Xi, log(
X Y
(13)
PSNR (dBb)
Puramelers 10MM
Figure 7: SR performance with different gen- erator models. Here, E-D denotes U-net remov- ing the mirror skip connection. We use Unet-8- 64. Unet-5-32 and ED-8-64 8-64, ED-5-32 (i.e.. arch.- downsampic/rsidual blocks-channcle tion in frequency domain is less affected by model capability, and the PSNR drop from model reduc- tion is far less than that in spatial domain, showing the advantages of frequency domain estimation.
27.3
4.50a
A -0.63 dB
25.0
1.35 dB
Frequency E-D
* Spatial K-D
100.00 dB
Estima-
A 45 gR_
Frequency U-Net Sputial U-Net
12.5
Flaps (100))
The results are shown Figure 6 Compared with existing single image kernel estimation methods, our proposed method can more accurately estimate Gaussian, motion, and disk kernels qualitatively and quantitatively. Further, to verify the advantage and generality of estimation in the frequency domain, use different reconstruction model architectures and sizes, respectively, and compare the

---

Table Ablation study on loss components.
Table Ablation study on input/output forms.
Settings
Metrics
Settings
L1
La
Lreg
PSNR
SSIM
Input
Output
FSNK
SSIM
x
8.462
0.1533
Spatial
Para.
20.01
0.5169
19.46
0,4952
Spatial
Kemelf
22.50
0.6585
X
26.68
0.7562
DFT
Para.
20.53
0.5417
Kernelf
0.7576
PIQU |
BRISOU
Nethod
KemaGAAN
IKC
77.85
40.89
NTTIEI0 Wirree
10.30
05/03
LR image taken by Canoa
Clverrie
CP-KPX
KemeCAA
IKC
FKP
WIEE
OCE'
Figure 8: Visual comparison of 4x blind SR results on real images taken by Canon from RealSR
blind SR performance as shown in Figure With the same model architecture, when the number of parameters decreases significantly, subsequent SR performance with kernel estimated in the frequency domain estimation is reduced by 1.55dB and 0.63db respectively, and for spatial domain is reduced by 9.45dB and 10.08dB respectively. The effective shape information provided in the frequency domain can still ensure relatively good SR performance for the model with fewer parameters. However, spatial estimation is more sensitive the model size and capability.
Ablation Study We conduct ablation study on loss components of the proposed generator. The results are shown in Table 2 We conduct extra comparisons with the same encoder network and training pipeline to verify the superiority of obtaining kernel directly from the frequency spectrum. Results can be found in Table 3 The ablation experiments are conducted on additional test sets using the setting of 3x blind SR DIV2K valid degraded with random Gaussian kernels (Para. and Kernelf briefly denote kernel parameters and up-sampling interpolation of kernel matrix).
Kernel estimation errors in different positions have unequal influences subsequent SR results, l without position information is inconsistent with SR results when the gap is small. lg allows the generator to learn the shape of the target kernel (i.e., 2D-Gaussian curve). The smooth prior Eyeg is conducive to the initial generator's 's learning. Our method uses the translation from the spectrum the interpolated kernel, which avoids the learning of local spatial pixel distribution, dramatically improving the learning stability. We achieve more superior kernel estimation accuracy and better SR results with the same network structure and pipeline compared with other settings.
4.2 Blind SR on Real Images
For real images,
the paired image taken with closer view in the dataset 44) the content- fidelity reference and take into account the visual perception of the SR results. The comparison methods include KernelGAN (2), LP-KPN 4 IKC (6) and NTIRE20 winner (10). We adopt the
Gaussian kernel estimation setting with smaller variances and use the same SR pipeline as [10] but replace the kernel estimation part with our proposed S2K. The result of 4x SR is shown in Figure Compared with NTIRE20 winner [10), our results achieve higher fidelity (red indicators) and fewer artifacts (green indicators). There is no ground truth in the strict sense, and we provide the non-reference evaluation metrics for the real image SR results as Figure right.
g
5 Conclusion
This paper theoretically demonstrates that feature representation in the frequency domain is more conducive for kernel estimation than in the spatial domain. We further propose S2K network, which makes the full use of the shape structure from the degraded LR image's ' frequency spectrum and, with

---

feasible implicit cross pross-domain translation, outputs the corresponding up up-sampled spatial kernel directly. In addition, we apply the kernel estimation optimization objective consistent with subsequent SR performance. Comprehensive experiments on synthetic and real-world datasets show that, when plugging S2K into the off-the-shelf non-blind SR models, we achieve hetter results both visually and quantitatively against ttateofftthe-a blind SR methods, by a large margin.
Acknowledgments and Disclosure of Funding
This work is supported by the Natural Science Foundation of China under Grant 61672273 and Grant 61832008.
References
[1] Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 126-135, 135, 2017.
[2] Sefi Bell- Kligler, Assaf Shocher, and Michal Irani. Blind super-resolution kernel estimation using internal-gan. In Advances in Neural Information Processing Systems (NeurIPS), pages 284-293, 2019.
[3] Giacomo Boracchi and Alessandro Foi. Modeling the performance of image restoration from motion blur. IEEE Transactions on Image Processing (TIP), pages 3502-3517, 2012.
[4] Jianrui Cai, Hui Zeng, Hongwei Yong, Zisheng Cao, and Lei Zhang. Toward soald single image super-resolution: new benchmark and a new model. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 3086-3095, 2019.
[5] Yu Chen, Ying Tai, Xiaoming Liu, Chunhua Shen, and Jian Yang. Fsrnet: End-to-end learning face super-resolution with facial priors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018
[6] Jinjin Gu, Hannan Lu, Wangmeng Zuo, and Chao Dong. Blind super-resolution with iterative kernel correction. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1604-1613,2011
[7] Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang. Toward convolutional blind denoising of real photographs. 2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[8] Yan Huang, Shang Li, Liang Wang. Tieniu Tan, al. Unfolding the alternating optimization for blind super resolution. Advances in Neural Information Processing Systems (NeurIPS), 2020.
[9] Shady Abu Hussein, Tom Tirer, and Raja Giryes. Correction filter for single image super- resolution: Robustifying off-the-shelf deep super-resolvers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 1428 1488-143,,00 2020.
[10] Xiaozhong Ji, Yun Cao, Ying Tai, Chengjie Wang, Jilin Li, and Feiyue Huang. Real-world super-resolution via kernel estimation and noise injection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
[11] Xiaozhong Ji, Guangpin Tao, Yun Cao, Ying Tai. Tong Lu, Chengjic Wang, Jilin Li, and Feiyue Huang. Frequency consistent adaptation for real world super resolution. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1664-1672, 2021.
[12] Scyoung Kim and Eric P Xing. Tree- group lasso for multi-task regression with structured sparsity. In In Proceedings of the International Conference on Machine Learning (ICML), 2010.
[13] Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 224-632,44 2017.
10

---

[14] Anat Levin. Yair Weiss, Fredo Durand, and William T Freeman. Understanding and evaluating blind deconvolution algorithms. In 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1964-1971. IEEE, 2009.
[15] Jingyun Liang, Kai Zhang, Shuhang Gu, Luc Van Gool, and Radu Timofte. Flow-based kemel prior with application to blind super-resolution. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 10601-10610, 2021.
[16] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 136-144, 2017.
[17] Jie Liu, Wenjie Zhang, Yuting Tang, Jie Tang, and Gangshan Wu. Residual feature aggregation network for image super-resolution In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition (CVPR), pages 2359-2368, 2020.
[18] Cheng Ma. Yongming Rao, Yean Cheng, Ce Chen, Jiwen Lu, and Jie Zhou, Structure-preservine super resolution with gradient guidance. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
[19] Shunta Maeda. Unpaired image super-resolution using pseudo-supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 291-300, 2020.
[20] Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley, Least squares generative adversarial networks. In Proceedings of the IEEE international Conference on Computer Vision (ICCV), pages 2794-2802, 2017.
[21] Tomer Michaeli and Michal Irani. Nonparametric blind super-resolution. In Proceedings of the 2013 IEEE International Conference on Computer Vision (ICCV), 2013.
[22] Kamyar Nazeri, Eric Ng. Tony Joseph, Faisal Qureshi, and Mchran Ebrahimi. Edgeconnect: Structure guided image inpainting using edge prediction. In The IEEE International Conference on Computer Vision (ICCV) Workshops, Oct 2019.
[23] Jinshan Pan, Deqing Sun, Hanspeter Pfister, and Ming-Hsuan Yang. Blind image deblurring using dark channel prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1628-1636, 2016.
[24] Liyuan Pan, Richard Hartley, Miaomiao Liu, and Yuchao Dai. Phase-only image based kernel estimation for single image blind deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6034 6043, 2019.
[25] Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H Li. Shan Liu, and Ge Li. Structureflow: Image inpainting via structure-aware appearance flow. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 181-190, 2019.
[26] Ying Tai, Jian Yang, and Xiaoming Liu. Image super-resolution via deep recursive residual network. In Proceedings of the IEFE Conference Computer Vision and Pattern Recognition (CVPR), 2017.
[27] Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu. Memnet: A persistent memory network for image restoration. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 4547, 2017.
[28] Radu Timofte, Eirikur Agustsson, Lue Van Gool, Ming-Hsuan Hsuan Yang, and Lei Zhang. Ntire 2017 challenge on single image super-resolution: Methods and results. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR) Workshops, pages 115- 2017.
[29] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceed- ings of the European Conference on Computer Vision (ECCV), 2018.

---

[30] Li Xu, Shicheng Zheng, and Jiaya Jia. Unnatural 10 sparse representation for natural image deblurring. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1107-1114 14, 2013.
[31] Yu-Syuan Xu, Shou-Ya Yao Roy Tseng, Yu Tseng, Hsien-Kai Kuo, and Yi-Min Tsai. Unified dynamic convolutional network for super-resolution with variational degradations. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12505, 2020.
[32] Jian Yang, Lei Luo, Jianjun Qian, Ying Tai, Fanlong Zhang, and Yong Xu. Nuclear norm based matrix regression with applications to face recognition with occlusion and illumination changes. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 39(1):156-17,, 2016.
[33] Kai Zhang, Wangmeng Zuo, and Lei Zhang. Learning a single convolutional super-resolution network for multiple degradations. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 3262-3271, 2018.
[34] Kai Zhang, Wangmeng Zuo, and Lei Zhang. Deep plug-and-play super-resolution for arbi- trary blur kernels. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition (CVPR), pages 1671-1681,2 2019.
[35] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention networks. In Proceedings the European Conference on Computer Vision (ECCV), pages 286-301, 2018.
[36] Ruofan Zhou and Sabine Susstrunk. Kernel modeling super-resolution on real low-resolution images. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2433 2443, 2019.
12


Reviews:
[{"review_id": "wlAsgrNjQkE", "reviewer": "Reviewer_wMvH", "summary": "This work proposes a method for blind super-resolution (SR) of images degraded by arbitrary blur kernels. The method makes use of a novel blur kernel estimation technique, that proposes to estimate such kernels in the frequency domain following an end-to-end deep learning approach. The authors claim that the Fourier amplitude spectrum kernel representation is better posed for the learning task of kernel estimation than the spatial representation, under the assumption that the kernels are sparse both in frequency and space domains (this assumption holds form most common kernels in real settings). They support this claim with theoretical and experimental evidence. \n\nThe image formation model considered here, that produces the degraded LR image from the HR image, is the following. The clean high-resolution (HR) image first downsampled, then convolved with the blur kernel, and finally contaminated with additive noise.  \n\nBased on the proposed frequency domain representation, the authors propose a kernel generator network that allows to translate this representation to the kernel spatial representation. The generator $G$ takes as input the single-channel amplitude spectrum (the FFT) of the degraded LR image bi-cubicly downsampled by two, and outputs the single-channel blur kernel estimate. This generator consists of an encoder-decoder network built on a U-net with mirror connection between each of the seven down-sampling convolutional layers and up-sampling transposed convolutional layers with the same feature size. \n\nTo ensure that the kernel generated by $G$ is at the same time close to the ground truth kernel and consistent with the SR results, they propose a three-terms loss for $G$. The first term imposes similarity between the ground truth kernel and the kernel generated by G from the degraded image (in the spatial domain) by means of the $\\ell_1$; the second term also imposes similarity between these two kernels but in seen as probability density functions, using an adversarial loss that involves a trainable critic $D$ that has to  discriminate between the ground truth kernel and the output of G (actually, de concatenation of each of them with the input of G are considered instead). This patch discriminator is trained using the loss proposed in LSGAN. Finally, the third term is a regularization loss on the output of G; unless I missed it, the regularizer is not specified. This kernel estimation model is called S2K. \n\nThen, the authors propose a pipeline for blind SR that combines the S2K module with a non-blind SR module. For the non-blind SR module, they consider two methods:  the multiple degradation-based method SFTMD (Gu et al., CVPR 2019; ref 6) and the kernel modeling-based method RCAN (Zhang et al., ECCV 2018; ref. 35). Given a ground truth kernel of size 15x15 and a HR image, a LR degraded image is generated and cropped to 512x512. The input to the S2K module is the ground truth kernel upsampled to 256x256 and the FFT of the LR image, downsampled to 256x256. The estimated kernel of size 256x256 returned by G is then downsampled at size 15x15. Finally, the non-blind SR module is fed with the LR degraded image and the 15x15 kernel estimated with the S2K module.\n\nThe experiments are carried out considering Gaussian kernels of size 15x15 of random covariance matrix, and random motion kernels of size 23x23 (instead of 15x15) with Boracchi and Foi’s method (IEEE TIP, 2012; ref 3). The proposed approach is compared to an extensive list of state-of-the-art methods, both for the task of blur kernel estimation and blind SR for 2x, 3x and 4x SR. All methods were trained on the the same degraded dataset (DIV2K), including S2K and the non-blind methods used in the proposed blind SR pipeline. Blind SR methods are compared on synthetic images using the PSNR and SSIM metrics. Kernel estimation performance is evaluated using the $\\ell_1$ norm and KL divergence. The results reported by the authors show that the proposed S2K and the blind SR pipeline significantly outperform the others. Finally, an ablation study on the loss components and frequency vs. spatial inputs is conducted, showing the pertinence of the proposed approach. \n\nFinally, blind SR experiments on real images using the dataset proposed by Cai et al. (ICCV 2019; ref. 4), evaluated using two reference-free metrics (PIQE and BRISQUE) show that the proposed pipeline also outperforms its competitors, including the NITRE20 winner (Ji et al., CVPR 2020; ref. 10). \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(+) The method is sound, elegant and simple\n\n(+) References and review of previous work are adequate.\n\n(+) Extensive evaluation, both on blur kernel estimation and blind SR. \n\n(+) According to the results reported by the authors, the proposed approaches for blur kernel estimation and the blind SR tasks both outperform state-of-the-art methods by significant margins. \n\n(+) Code is provided as supplementary material (I did not check it).\n\n(-) The image formation model is not realistic. The way LR images are produced from HR images is by first convolving with the HR blur kernel, and then subsampling. Not the other way round. \n\n(-) The fact that comparison of the spectrums of the degraded image with the blur kernel is more conducive than comparison of the degraded image and the blur kernel in space is evident. As the blur kernel low-pass filter the image, it is obvious that the coarse features of the spectrum of the kernel and of the degraded image are similar. \n\n(-) Frequency consistency was already proposed and demonstrated in ref. 11 (not published but reported in arXiv). Moreover, the kernel estimation method proposed in ref. 11 bears strong similarities to the one proposed here. This works even proposes an adversarial loss for the high-frequency components of the blur kernels. The authors should clearly state the differences and contributions of their work compared with ref. 11. \n\n(-) Last but not least, the paper is poorly written. While the idea is simple, the paper does not read well. For instance:\n\n* When the authors show that comparison of the degraded image and the kernel in the frequency domain is more conducive than in the spatial domain, one wonders why this comparison is relevant. They only show its relevance with an example using a Gaussian kernel afterwards. \n\n* Notation is sometimes misleading; for instance, in Section 3.2 $\\mathbf{S}$ is used to represent the Fourier transform of the clean image, and then from Section 3.3 it is used to represent the spectrum of the degraded image. \n\n* The methods presented in the tables should also be referenced in the tables. \n\n* English is poor, both grammatically and syntactically. The text is plagued with conjugation errors, unstructured sentences and typos. The article should be almost completely rewritten and checked by someone fluent in English.  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not see any limitation. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "Sx8qWw1HwhT", "reviewer": "Reviewer_idYk", "summary": "The authors propose a novel framework for solving the problem of blind super resolution in which the low resolution input can be degraded by arbitrary blur kernels. In their framework, the degradation kernel estimation is performed in the frequency domain. The authors show that the frequency domain is better suited for kernel representation and reconstruction than the spatial domain and propose a Spectrum-to-Kernel mapping network to estimate diverse blur kernels. They use a conditional GAN to learn how to 'translate' the spectra of degraded images to the unknown kernels.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Estimating degradation kernels for the case of blind upscaling is typically an ill posed and difficult task and estimation errors can carry over into conditional upscaling methods resulting in artefacts in the upscaled image. Therefore, efforts to better represent and estimate degradations automatically is relevant.\n\nThe idea of estimating kernels in a different representation seems nice and the authors demonstrate that for commonly existing sparse kernels, reconstruction in the frequency domain is preferable compared to the spatial domain. The observation that there is a closer resemblance between the frequency spectrum of the kernel and the frequency spectrum of the image degraded with that kernel as compared to the spatial kernel and the degraded image in the spatial domain is interesting. Also as the authors state, I am not aware that this has been leveraged for the task of blind super resolution so far. Therefore it seems intriguing to design a network to estimate a kernel from the Fourier amplitude spectrum. Also the results shown look competitive.\n\nHowever, in my opinion the work [CORNILLÈRE et al., Blind Image Super-Resolution with Spatially Variant Degradations, Siggraph Asia 2019] is quite related and should be cited and compared to. There, the estimation of the degradation is based on an optimisation that discriminates artefacts in the resulting high res while being parameterised through a neural network embedding. Fundamentally, it seems like a very different approach to estimate the degradation in the low res. A lot of existing works follow the path of estimating the degradation from the low res and here I can see that it could be better solved int the frequency domain. However, in the resulting high res where kernel estimation errors quickly become very visible in the form of ringing/oversharpening or excessive blurring, it seems that also methods operating in the spatial domain might find accurate solutions. Therefore I would like to see comparisons as mentioned before and ideally a small paragraph to interpret the results and differences in the approaches.\n\nOverall Equation 8 seems to indicate that the frequency domain might be better for estimating degradation kernels but I'm not sure it can directly be derived from the equation since it should only show that kernel and degraded image are closer together w.r.t. a given metric in frequency space as opposed to image space. While this gives a good motivation to follow that strategy, it seems that further assumptions/ restrictions would be needed to conclude that this will also allow more accurate kernel estimation / reconstruction. It would be great if the authors could clarify that a bit more. As a more minor point, it would also be great to detail a bit more about how reconstructing the kernel in the spatial domain would compare to reconstructing in the frequency domain.\n\nOverall, I feel this paper offers an interesting view on blind image super resolution and tries a novel approach that has not been analysed so far in this context. However at the same time some important comparisons are missing and the paper seems slightly unpolished with a larger number of typos.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I would not expect negative societal impact from this work. The limitations are listed in the supplemental material but might actually deserve a place in the paper. At least the point regarding spatially varying degradations.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2-3 hours"}, "extra_text": {}}, {"review_id": "RpvptgO0G6", "reviewer": "Reviewer_ZgQH", "summary": "This paper proposes a new blind super-resolution method, which can due with arbitrary blur kernels instead of Gaussian kernels. The whole method estimates kernels in the frequency domain. The method involves a kernel generator that takes the frequency amplitude as input and outputs a kernel. A patch discriminator is designed to perform adversarial loss to the predicted kernel. This design is novel and the experiments show good results.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have several questions:\n\n1. Can we know how this kernel was predicted? Is there any visualization method that can tell us which part of the frequency amplitude contributes to the kernel prediction? This is important when it comes to the generalization performance of the proposed method. Although the experiment shows good generalization performance in Figure 6, an interpretational explanation is also wanted to address our concerns about how this system works.\n\n2. As a common sense that the kernel prediction is an ill-posed problem. How does the proposed method address this issue?\n\n3. This paper actually proposes a kernel estimation method. Can this method be used for deburring motion blur? How is the performance? If it is specially designed for blind super-resolution, what makes it especially suitable for SR task? \n\n4. In Table 2, the motion blur part, The IKC, Pan et al. [22] w. SFTMD, Pan et al. [23] w. SFTMD and other methods are not trained for motion blur, the author should compare their method with motion-deblur + regular SR for better results. The claim of nearly 6db improvement is not objective as the STOA performance on these images is not obtained by the abovementioned methods.\n\nAs the good performance and novel idea (predict in frequency domain), I give a positive initial rating. I hope the authors can answer my questions so that I can make the final suggestion.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "NKIh63XUHu", "reviewer": "Reviewer_bKSR", "summary": "The authors make a key observation that degradation kernels are highly correlated to degraded images in the frequency domain. From this, they design a kernel generator that directly generates the degradation kernel from the frequency-domain LR image. Then, a non-blind SR model is employed to utilize the estimated kernels for SR. The proposed method outperforms existing methods on two types of kernels (Gaussian and motion) with two backbone networks (SFTMD and RCAN).", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- This paper provides insightful observations that degradation kernels are highly correlated to the degraded image in the frequency domain (Fig. 1 and Fig. 2).\n- From the above observation, a novel approach is taken, where the blur kernel is directly estimated from the frequency-domain LR image.\n- A sound theoretical analysis is provided in Section 3.1 .\n- Supplemetary material provides additional details that helps to further understand the main paper.\n- Ablation studies verify the benefits of each of the proposed components including the spectrum-to-kernel scenario.\n- The proposed method outperforms existing methods on real images.\n\nCons:\n- Clarifications are needed in some parts of the paper:\n  - The phrase in lines 37-39, “the iterative form can not utilize the maximum performance of both estimate and SR module during the testing process, and the barrel effect is inevitable\", is unclear. Why can't IKC reach maximum performance for both the estimator and the restorer? What do the authors mean by barrel effect?\n  - What do the authors mean by \"inconsistency of training and test domain\" in lines 216-217? Aren’t kernels generated randomly for both training and testing?\n  - In line 208, what do the authors mean by 'random blurred'? Is this random Gaussian blur or randomly selected motion blur kernels?\n- Some missing analyses in experiments:\n  - Is there a reason why the performance gain reduces as scale factor increases?\n  - Why do existing methods perform poorly for motion kernels? Aren’t they trained on the same random motion kernels used for training the proposed model?\n- In Table 2, comparison to more recent blind SR methods such as DAN (NeurIPS 2020) is missing.\n- It is common to include SR results on historic images (http://vllab.ucmerced.edu/wlai24/LapSRN/) as in ZSSR, IKC, etc in blind SR literature.\n- The motivation behind bicubic downsampling the FFT image before the kernel generator is unclear. Wouldn't this lose information? If it is to reduce computations, why not downsample by 4 or 8?\n- Statements like “existing blind SR methods are far from meeting the needs of various degradation processes in general scenes” (lines 30-31), need references or evidence. How are they ‘far from meeting the needs of various degradation’?\n\nMinor:\n- In Table 4, what is the meaning of para.?\n- It would be useful to show LR input images in Fig. 5.\n- Typos: (i) Table 1 caption, estimaion -> estimation, (ii) line 234, we -> We.\n\n========================  \n**Post Rebuttal:**\nRaising my score from 6 to 7 after the authors' reply. More details are in a separate comment. Well done on the new experiments and the clarifications!", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are discussed in the supplementary material but potential negative societal impact has not been addressed. Some potential negative societal impacts could include privacy issues due to upscaling degraded data.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles the problem of blind super-resolution when the images degraded by arbitrary blur kernels (assumed to be sparse both in frequency and space domains). The method proposes a new blur kernel estimation technique in the frequency domain as an end-to-end deep learning approach. The paper provides theoretical and experimental evidence to back the idea.

The core idea of the work is predicting degradation kernels in the frequency domain using deep neural networks. The proposed method is simple and effective, this was highlighted by all reviewers. The empirical evaluation seems convincing (and were further improved during the response period). The manuscript includes thorough ablation studies which clarify the benefits of each of the proposed components. This was highlighted by Reviewers wMvH and bKSR.

The authors provided a high quality rebuttal including new experiments and clarifying several points raised by the reviewers. In particular, the authors incorporated experiments on (i) mixed blur kernels, (ii) comparison to a more recent blind SR baselines, and (iii) quantitative and qualitative results on real historic images. All reviewers and the AC appreciate the authors response.

After the authors’ response, Reviewer wMvH increased the score to 6. The clarifications provided by the authors were considered sufficient. However, she/he expressed concerns regarding the papers presentation. Similarly Reviewer bKSR increased the score to 7. In the original review, the main concern was clarity. 

he AC appreciates the authors response and counts that the authors will incorporate the the clarifications provided in their response. All reviewers provide an accepting score. The AC agrees and recommends accepting the paper.

Decision: Accept (Poster)