Knowledge Distillation based Degradation Estimation for Blind Super-Resolution
Authors: Bin Xia
Yulun Zhang
Yitong Wang
Yapeng Tian
Wenming Yang
Radu Timofte
Luc Van Gool
Venue: ICLR 2023 poster (2023)
Keywords: Image Super-Resolution

Abstract: Blind image super-resolution (Blind-SR) aims to recover a high-resolution (HR) image from its corresponding low-resolution (LR) input image with unknown degradations. Most of the existing works design an explicit degradation estimator for each degradation to guide SR. However, it is infeasible to provide concrete labels of multiple degradation combinations (\eg, blur, noise, jpeg compression) to supervise the degradation estimator training. In addition, these special designs for certain degradation, such as blur, impedes the models from being generalized to handle different degradations. To this end, it is necessary to design an implicit degradation estimator that can extract discriminative degradation representation for all degradations without relying on the supervision of degradation ground-truth. In this paper, we propose a Knowledge Distillation based Blind-SR network (KDSR). It consists of a knowledge distillation based implicit degradation estimator network (KD-IDE) and an efficient SR network. To learn the KDSR model, we first train a teacher network: KD-IDE$_{T}$. It takes paired HR and LR patches as inputs and is optimized with the SR network jointly. Then, we further train a student network KD-IDE$_{S}$, which only takes LR images as input and learns to extract the same implicit degradation representation (IDR) as KD-IDE$_{T}$. In addition, to fully use extracted IDR, we design a simple, strong, and efficient IDR based dynamic convolution residual block (IDR-DCRB) to build an SR network. We conduct extensive experiments under classic and real-world degradation settings. The results show that KDSR achieves SOTA performance and can generalize to various degradation processes. The source codes and pre-trained models will be released.

Published as a conference paper at ICLR 2023
KNOWLEDGE DISTILLATION BASED DEGRADATION ESTIMATION FOR BLIND
Bin Xia!, Yulun Zhang", Yitong Wang', Yapeng Tian', Wenming Yang', Radu Timofte', and Luc Van Gool 'Tsinghua University 'ETH Zürich 'ByteDance Inc *University of Texas at Dallas University of Würzburg
ABSTRACT
Blind image super-resolution (Blind-SR) aims to recover a high-resolution (HR) image from its corresponding low-resolution (LR) input image with unknown degradations. Most of the existing works design an explicit degradation estimator for each degradation to guide SR. However, it is infeasible to provide concrete labels of multiple degradation combinations (e.g., blur, noise, jpeg compression) supervise the degradation estimator training. In addition, these special designs for certain degradation, such as blur, impedes the models from being generalized handle different degradations. To this end, is necessary design implicit degradation estimator that can extract discriminative degradation representation for all degradations without relying on the supervision of degradation ground- truth. In this paper, we propose Knowledge Distillation based Blind-SR network (KDSR). It consists of a knowledge distillation based implicit degradation estima- tor network (KD-IDE) and an efficient SR network. To learn the KDSR model, we first train teacher network: KD-IDEy- puts and is optimized with the SR network jointly. Then, we further train a student network KD-IDEs, which only takes LR images as input and learns to extract the same implicit degradation representation (IDR) as KD-IDEr. In addition, to fully use extracted IDR, we design a simple, strong, and efficient IDR based dynamic convolution residual block (IDR-DCRB) to build an SR network. We conduct extensive experiments under classic and real-world degradation settings. The re- sults show that KDSR achieves SOTA performance and can generalize various degradation processes. The code is available at Github.
takes paired HR and LR patches as in-
INTRODUCTION
Single image super-resolution (SISR) aims to recover details of a high-resolution (HR) image from its low-resolution (LR) counterpart, which has a variety of downstream applications (Dong et al., 2014; Zhang et al., 2019; Xia et al., 2022d; Fritsche et al., 2019; Xia et al., 2022c;b). These state- of-the-art methods (Kim et al., 2016; Lim et al., 2017: Lai et al., 2017; Xia et al., 2022a; Wang et al., 2018b) usually assume that there is an ideal bicubic downsampling kernel to generate LR images. However, this simple degradation is different from more complex degradations existing in real-world LR images. This degradation mismatch will lead to severe performance drops.
To address the issue, blind super-resolution (Blind-SR) methods are developed. Some Blind-SR works (Wang et al., 2021a; Luo et al., 2022) use the classical image degradation process, given by Eq. 1. Recently, some works (Cai et al., 2019; Bulat al., 2018) attempted to develop new and complex degradation process to better cover real-world degradation space, which forms a variant of Blind-SR called real-world super-resolution (Real-SR). The representative works include BSR- GAN (Zhang et al., 2021) and Real-ESRGAN (Wang et al., 2021b), which introduce comprehensive degradation operations such as blur, noise, samplian, and JPEG compression, and control the severity of each operation by randomly sampling the respective hyper- hyper-paramess parameters. To better simulate the complex degradations in real-world, they also apply random shuffle of degradation orders (Zhang et al., 2021) and second-order degradation (Wang et al., 2021b) respectively.
Since Blind-SR faces almost infinite degradations, introducing prior degradation information to SR networks can help to constrain the solution space and boost SR performance. As shown in Fig. 1, the way to obtain degradation information can be divided into three categories: (1) Several Non- Blind SR methods (Zhang et al., 2018a; Shocher et al., 2018; Zhang et al., 2020; Soh et al., 2020; Xu et al., 2020) directly take the known degradation information as prior (Fig. 1 (a)). (2) Most of
"Correponningg Author: Wenming Yang, yang

---

Published as a conference paper at ICLR 2023
Nose
(a) Non-blind SR with multiple degradation
SR Network
Blur Kamei
Leariing disse
Degradation Estimator
'S
Blr Komel
(b) blind SR with explicit degradation estimation
Komel
Mebic Looning Push away fectures
SR
UR
Mezia Learning
Implicit Degradation Estimator
(c) blind SR with Metric Learning based implicit
Knowwetge Cladilation
degradation estimation
HR
Weight
Teacher Implici
Teacher
(d) blind SR with KD based implicit
Degradation Estimator
SR Network
degradation estimation (Durs)
LR
Student Implicit
Student
Degrudation
Degradation Estimator
SR Network
snudent remwass Woleiig
Figure I: The illustration of different degradation estimators. (a) Non-blind SR methods directly use known degradation information to guide SR networks, such as SRMD (Zhang et al., 20)
(b) Many Blind-SR methods estimate the explicit degradation with the supervision of ground-truth degradation. (c) Several methods use metric learning to distinguish degradation roughly. (d) Our knowledge distillation (KD) based implicit degradation estimator can estimate accurate implicit degradation representation to guide SR without ground-truth degradation supervision.
Blind-SR methods (Gu et al., 2019; Luo et al., 2020; Wang et al., 2021a; Liang al., 2022; Luo et al., 2022) adopt explicit degradation estimators, which are trained with ground-truth degradation (Fig. (b)). However, these explicit degradation estimators are elaborately designed for specific degradation processes. The specialization makes them hard to transfer to handle other degradation processes. In addition, is challenging to annotate precise ground-truth labels to represent the mul- tiple degradation combination (Zhang et al., 2021; Wang et al., 2021b) for supervised degradation learning. Therefore, developing implicit degradation representation (IDR) based methods is impor- tant. (3) Recently, as shown in Fig. 1 (c), DASR (Wang et al., 2021a) and MM-RealSR (Mou et al., 2022) use metric learning to estimate IDR and quantize degradation severity respectively. However, metric learning methods roughly distinguish degradations by pushing away pulling close features, which is unstable and cannot fully capture discriminative degradation characteristics for Blind-SR.
In this paper, aim design efficient implicit degradation representation (IDR) learning SR framework that can easily adapt to any degradation process. To this end, we develop a novel knowl- edge distillation based Blind-SR Network (KDSR). Specifically, as shown in Fig. 1 (d), KDSR knowledge distillation based implicit degradation estimator (KD-IDE) to predict accurate IDR. Fur- thermore, we propose strong yet efficient SR network based on our newly developed IDR based Dynamic Convolution Residual Blocks (IDR-DCRB) to reconstruct the HR image with the guidance of IDR. For the training process, first input HR and LR images the teacher KD-IDE,, which is optimized with the SR network together. Given the paired HR and LR images, teacher KD-IDE can easily extract the latent degradation information in LR images. Then, we use student KD-IDEş to learn to extract the same IDR that of KD-IDET from LR images directly. Extensive experiments can demonstrate the effectiveness of the proposed KDSR. Our main contributions are threefold:
We propose KDSR, a strong, simple, and efficient baseline for Blind-SR, can generalize to any degradation process, which addresses the weakness of explicit degradation estimation.
We propose a novel KD based implicit degradation representation (IDR) estimator. To the best of our knowledge, the design of IDR estimation has received little attention far. Besides, we propose an efficient based SR network to fully utilize IDR to guide SR.
Extensive experiments show that the proposed KDSR can achieve excellent Blind-SR per- formance in different degradation settings from simple to complex.
RELATED WORK
2.1 BLIND
In the past few years, numerous Non-Blind SISR methods (Dong et al., 2014; Lim et al., 2017; Zhang et al., 2018a; Ledig et al., 2017; Johnson al., 2016; Ma al., 2020; Xia et al., 2023) have
2

---

Published as a conference paper at ICLR 2023
achieved promising performance and have been widely studied. However, the performance of these methods will dramatically decline as there is a degradation gap between training and test data, As a remedy, SRMD (Zhang al., 2018a), USRNet (Zhang et al., 2020) and some other methods (Zhang et al., 2019; Xu et al., 2020; Shocher et al., 2018; Soh et al., 2020) utilize the blur kernel and noise level as additional input. Although these methods can deal with multiple degradations with single model, they require accurate degradation estimation, which is also a challenging task.
To handle unknown degradation. few Blind-SR methods have been proposed. Some methods, such as IKC (Gu et al., 2019) and DAN (Luo et al., 2020), use the classical Blind-SR degradation process and combine a blur kernel estimator with SR networks, which can be adaptive images degraded from various blur kernels (Kim et al., 2021; Cormillere et al., 2019). Besides, KMSR (Zhou & Susstrunk, 2019) constructs kernel pool by utilizing generative adversarial network (Goodfellow et al., 2014). Recently, some works like BSRGAN (Zhang et al., 2021) and Real-ESRGAN (Wang et al., 2021b) design more complex and comprehensive degradation processes to better cover the real-world degradation space, which becomes a variant of Blind-SR called Real-SR.
For each degradation type and process, previous Blind-SR methods (Gu et al., 2019; Liang et al., 2022) tend to specially design an explicit degradation estimator. That is quite complex and hard to provide ground-trutte labels for multiple degradation combinations. Recently, DASR (Wang et al., 2021a) and MM-RealSR (Mou et al., 2022) metric learning to roughly distinguish various degra- dations, which is not accurate enough to provide degradation representation to guide SR. In this paper, we propose to estimate IDR accurately and fully use it for SR in an efficient way.
2.2 KNOWLEDGE DISTILLATION
The purpose of knowledge distillation (KD) (Hinton al., 2015) is to transfer the representation ability of teacher network to a student network. KD has been widely used to compress models, typ- ically for classification tasks. Specifically, they (Ahn al., 2019) compress the classification models by enforcing the output distribution between the teacher and student networks to be close. Recently some works extend KD to feature distillation, such as intermediate feature learning (Romero al., 2014) and pairwise relations in intermediate feature learning (Liu et al., 2019). For the SISR task, SRKD (Gao et al., 2018) and FAKI (He et al., 2020) apply the KD between intermediate feature maps to compress models. To obtain more efficient SR networks, PISR (Leee et al., 2020) further in- troduces variational information distillation (Ahn et al., 2019) to maximize the mutual information between intermediate feature maps of teacher and student SR networks. Different from previous works adopting KD for model compression, we develop KDSR to obtain accurate IDR.
METHODOLOGY
3.1 OVERVIEW
Blind SR methods can be summarized two categories: classic blind-SR and real-world blind-SR.
For classic blind-SR, some Blind-SR works (Wang et al., 2021a; la; Luo et al., 2022) use the classical image degradation process, given by
y = (x @x k) 4e- +n.
(1)
where @ denotes convolution operation. and y are HR and corresponding LR images respectively. is blur kernel and n is additional white Gaussian noise. ts refers to downsampling operation with scale factor s. The severity of blur and noise unknown, which randomly sampling the respective parem--rrseeters to adjust severity and form almost infinite degradation space.
Given input LR image y and applied blur kernel k, the classic blind-SR methods (Gu al., 2019; Luo et al., 2020; 2022) pretrain an explicit degradation estimator to estimate the blur kernel applied on y with the supervision of groundtruth k. Then, their SR network can use the estimated blur kernel to perform SR on LR image y. The SR network is trained with loss function Cree-
Crec [ Isallı
where IHR and ISR are real and SR images separately.
Real-world blind-SR is a variant of classic blind-SR, in which more complicated degradation is adopted. The real-world blind-SR approaches (Wang et al., 2021b; Zhang et al., 2021) introduce comprehensive degradation operations such as blur, noise, down-sampling, and JPEG compression, and control the severity of each operation by randomly sampling the respective hyper-parameters. parameters. Moreover, they apply random shuffle of degradation orders and second-order degradation to increase
3

---

Published as a conference paper at ICLR 2023
CXtExit
Dipiisiie
Fallxw
Convclution
Pest
Surmation
(e) IDRt brewe Deptitoten Dynamis: Cormatittion (DR-DOC)
1bj IDR based Dynamic Convelution Residual Block
(DR-0CBB)
Eorry
Imput
Linser
Liteio
Laskyftel
Paol Uhaiuuffe
4C
First Slager KDGR
3d Coneelaionll Layw
LN
Krowlddge latin
(c) Kinwweeige Dirillationn based Implitt Degriatio Esomater (KD ICE)
16 Becond Stage: Traniin KDSR, Inforenee
The overview of our proposed knowledge distillation based Blind-SR network (KDSR), which consists of KD based implicit degradation estimator network (KD-IDE) and SR network mainly formed by the IDR based Depthwise Dynamic convolution (IDR-DDC).
Figure 2:
degradation complexity. Since degradation is complex and cannot provide specific degradation la- bels, they directly use SR networks without degradation estimators. Their SR networks emphasize visual quality trained with Lvis-
Lois AreeCrec + Aper + Aady
(3)
where Cper and Ladv perceptual (Johnson et al., 2016) and adversarial loss (Wang et al., 2021b).
As shown Fig. 2, we propose KDSR, consisting KD-IDE and an efficient network. Dif- ferent previous explicit degradation estimation based blind-SR methods Gu et al. (2019); Luo et al. (2020; 2022); Liang et al. (2022), our KD-IDE does not requires degradation labels for training, which generalize to any degradation process. Morcover, the design of our SR network is neat and efficient, which is practical and can fully use degradation information for SR. There are two stage training processes for KDSR, including Teacher KDSR, and Student KDSR, training. We first train the KDSR7: we input the paired HR and LR images to the KD-IDE, and obtain the implicit degradation representation (IDR) Dr and DT,, where Dr is used guide the SR. After that, we move on to KDSRs training. We initialize the KDSR, y with the KDSRT's 's parameters and make the KDSR learn directly extract D's same as D', from LR images.
3.2 KNOWLEDGE DISTILLATION BASED IMPLICIT DEGRADATION ESTIMATOR
Most Blind-SR methods elaborately design an explicit degradation estimator for each degradation type and process. There several limitations for explicit degradation estimators: (1) These special designs for specific degradation processes make the explicit estimator hard to be transferred to other degradation settings. (2) It is complex provide various degradation labels for explicit degradation estimator training, especially the random combination of multiple degradations (Wang et al., 2021 Therefore, we develop KD based implicit degradation estimator (KD-IDE), which can distinguish various degradations accurately without the supervision of degradation ground-truth.
As shown in Fig. 2 (c). can divide KD-IDE into several parts: (1) We take the LR images and the concatenation of LR and HR images as input for KD-IDEg and KD-IDEy. respectively. Specially, for the KD-IDEr (Fig. (d)), it can easily extract the degradation which makes HR degrade to LR images by providing paired HR and LR images and jointly being optimized with the SR network. Since there is spatial size difference between HR and LR images, we perform the Pixel-Unshuffle operation on HR images JH E x4W to be TH E IR T848x HxW and then concatenate it with LR images to obtain an input I E R (2) The input passes through the first convolution to become feature maps. It is noticeable that the input channels in the first convolution are 3 and 51 for KD-IDEs and KD-IDE,, respectively. (3) After that, we use numerous residual blocks to further extract features and obtain rough degradation vector by Average Pooling operation. (4) We use the two linear layers to refine the degradation vector and obtain IDR D' E R24C, which used for KD. (5) Although D' has 4C channels to accurately present degradation and give

---

Published as a conference paper at ICLR 2023
more degradation information for the student network to learn it will consume large number of computational resources used in IDR-DDC Hence, we need to further compress it with linear layer and obtain another IDR D R to guide SR. More details of KD training are given in Sec. 3.4.
3.3 IMAGE SUPER- RESOLUTION NETWORK
As for the design of SR network, we should consider three points: (1) After we obtain the IDR, is important to design a SR network that can fully use the estimated degradation prior for SR. (2) An ideal Blind-SR network is likely to used in practice, the structure of which should be simple. Thus, we also try to make the network formed by one type of simple and strong enough module. (3) The huge computation consumption usually limits the application of models, especially on edge devices. Thus, it necessary to design efficient model.
As shown in Fig. (a), (b), and (d), our SR network can be divided into three hierarchies. (1) We first propose convolution unit called IDR based Depthwise Dynamic Convolution (IDR-DDC). Motivated by UDVD (Xu et al., 2020), we adopt the dynamic convolution to use IDR to guide SR. Specifically, fully use the estimated IDR, displayed in Fig. (a), generate specific convo- lution weights according to the IDR D. However, if we generate ordinary convolution weights, the computational cost will be quite large and affect the efficiency of the network. Thus, we further in- troduce depthwise convolution (Howard et al., 2017), which merely consumes about & computation and parameters of ordinary convolution. The IDR-DDC can be mathematically expressed as:
W Reshape (db (D)),
F Fi :; = ;. & i, [0,C),
(5)
where ((.) and @ are two linear layers and convolution operation separately; D E RC indicates IDR, Φ (D) E RCK.KK is the output of c(.). W E RCxIXXx xx is weights of dynamic convo- lution: Fi.. and Foot E Cx are input and output feature maps respectively. (2) As shown in Fig. 2 (b), motivated by EDSR (Lim al., 2017), we develop IDR based Dynamic Convolu- tion Residual Blocks (DR-DCRB) to realize deep model. For the first convolution of IDR-DCRB, use the IDR-DDC utilize degradation information. However, IDR-DDC lacks interaction be- tween different channels. Thus, we adopt ordinary convolution as the second convolution. (3) For simplicity, as shown in Fig. 2 (d) or (e), we mainly stack the IDR-DCRB to form the SR network.
3.4 TRAINING PROCESS
KDSR has a two-stage training process. (1) As shown in the Fig. 2 (d), we first train the teacher KDSRT. we input the paired LR and HR images to the KD-IDEr obtain the IDR DT and D Then, DT is used to generate specific degradation weights for dynamic convolution. After that, the specific degradation SR network will restore the LR images. By jointly optimizing the teacher SR network and KD-IDE, with the L1 Loss (Eq. 2), the KD-IDE can effectively extract accurate IDR to guide SR network. (2) After finishing KDSRT training, we move on train KDSRs, As shown in the Fig. 2 (e), different from KD-IDET, we only input the LR images to the KD-IDEs, obtaining IDR Ds and D's. The other steps are the same as KDSRG training except for the adopted loss functions. Specifically, we introduce a knowledge distillation (KD) function (Eq. 6) to enforce the KD-IDE3 directly extracting the same accurate IDR as KD-IDE, from LR images. In addition, for the classic degradation model (Eq. 1), following previous Blind-SR works (Gu et al., 2019; Wang et al., 2021a), we adopt Lree (Eq. 2) and can set the total loss function as Lalassic (Eq. 7). For more complex degradation processes (Real-SR), following Lvis (Eq. 3) of Real-ESRGAN (Wang et al., 2021b), we propose Creat (Eq. 8). More details are given in appendix.
D D'nnom (i)
LH =
Σ D'roor (j) log )=(0,C))
D D'snorm (i)
(6)
Letassic = Arechrec + daL
(7)
Creat = Xree + NEICRI + Aper + dady
(8)
where D' and D's Snorm are normalized with softmax operation of D'r and D's separately. Eper and Ladv are perceptual and adversarial loss. Auta Aper and Aadv denote the balancing parameters.
EXPERIMENTS
4.1 SETTINGS
We train and test our method on classic and real-world degradation settings. For the classic degradation, following previous works (Gu al., 2019; Luo al., 2022), combine
5

---

Published as a conference paper at ICLR 2023
Table 1: 4x SR quantitative comparison datasets with Gaussian8 kernels. The bottom three methods marked in rouse use IDR to guide blind SR. The FLOPs and runtime are computed based on an LR size of 180 320. Best and second best performance are in red and blue colors, respectively.
Se15
Set14
BSD100
Urhan100
Manga109
Methods
Param (M)
FLOPs (G)
Time (ms))
SSIM
PSNR
SSIM
PSNR
Bicubic
24.57
0.7108
22.79
0.6032
23.29
0.5786
20.35
0.5532
21.50
0.6933
RCAN
1082.41
$56.21
26.60
0.7598
0.6113
0.608
0.7428
BkuNce77SR
30946
26.45
0.7279
24.78
0.6368
24.97
0.5989
21.11
0.5505
23.53
IKC
3.32
2528.03
1053.79
31.67
0,8829
28.31
0.7643
27.37
0.7192
25.33
0.7504
28.91
0,8782
DAN+1
4.33
1008. 33
201.04
31,89
0.8864
28.42
0,7687
27.51
0.7248
25,86
0.7721
30.50
0.9037
DANY2
1088.
201.51
32.00
0.8885
28.50
0.7715
27.56
0.7277
25.94
0.7748
30.45
0.9037
Ada Target
16.70
1032.59
31.58
27.43
0.7683
DCLS
13.63
175.84
32.12
0.8890
28.54
0.7728
27.60
0.7285
26.15
0.7809
30.86
0.9086
DASR
5.81
185.66
44.14
31.46
0.8789
28.11
0,7603
27.41
0.7214
25.36
0.7306
29.39
0,8861
K (Ours)
5.80
191.12
38.74
32.02
0.8892
28.46
0.7761
27.52
0.7281
25.96
0.7760
30.58
0.9026
XDSRs-L (Ours)
149.14
32.11
27.64
0,7830
HBR
RCAN
DCLS
KDSR.-M
HR
RCAN
IXTS
KDSR,-M
HBR
RCAN
DCLS
KDSR.-M
HR
RCAN
DCLS
KISR,-M
Figure 3: Visual comparison (4x) of Blind-SR methods on isotropic Gaussian kernels.
800 images in DIV2K (Agustsson & Timofte, 2017) and 2,650 images in Flickr2K (Timofte et al., 64, and the L.R patch sizes 64x64. We use Adam optimizer with 81 = 0.9, 32 0.99. We train both teacher and student networks with 600 epochs and set their initial learning rate 10-4 and decrease half after every 150 epochs. The loss coefficient Arec and Apd are set to I and 0.15 separately. The SR results are evaluated with PSNR and SSIM the Y channel in the YCbCr space. (1) In Sec. 4.2, train and test on isotropic Gaussian kernels following the setting in Gu et al. (2019). Specifically, the kernel sizes are fixed to 21x21. training, the kernel width o ranges are set to [0.2. 4.0] for scale factors 4. We uniformly sample the kernel width in the above ranges. For testing, we adopt the Gaussian8 (Gu et al., 2019) kernel setting to generate evaluation datasets. Gaussian8 uniformly chooses 8 kernels from range [1.80, 3.20] for scale 4. The LR images are obtained by blurring and downsampling the HR images with selected kernels. (2) In Sec. 4.3, we also validate our methods on anisotropic Gaussian kernels and noises following the setting in (Wang et al., 2021a). Specifically, We set the kernel size 21 x21 for scale factor 4. In training, we use the additive Gaussian noise with covariance a = 25 and adopt anisotropic Gaussian kernels characterized by Gaussian probability density function (00, Σ) with zero mean and varying covariance matrix Σ. The covariance matrix Σ is determined by two random eigenvalues λ1, A2 (0.2, 4) and random rotation angle UO0, T).
2017) as the DF2K training set. The batch sizes are
For the real-world degradation, in Sec. 4.4, similar to RSal-N (Wang et al., 2021b), we adopt DF2K and OutdoorSceneTrainne Training (Wang et al., 2018a) datasets for training. We set the learn- ing rate of the KDSR, to 2 x 10-4 and pre-train it with only Eq. 2 by 1000K iterations. Then, we optimize KDSRg with Eq. by 1000K iterations and continue train with Eq. 8 by 400K itera- tions. The learning rate is fixed as 10-4, For optimization, we use Adam with B1 0.9, 32 = 0.99. In both two stages of training, we set the batch size 48, with the input patch size being 64.
4.2 EVALUATION WITH ISOTROPIC GAUSSIAN KERNELS
We first evaluate our KDSR on degradation with isotropic Gaussian kernels. We compare the KDSR with several SR methods, including RCAN (Zhang et al., 2018c), 7SSR (Shocher al., 2018),

---

Published as a conference paper at ICLR 2023
Table 2: PSNR results achieved Set14 (Zeyde al., 2010) under anisotropic Gaussian blur and noises. The bottom two methods marked in rouse use IDR to guide blind SR. The best results are marked in bold. The runtime measured on an LR size of 180 320.
Blur Kemee
Method
Params
Time
Noise
0
24.28
24.47
24.6
24.64
24.58
24.47
24.31
23.97
23.01
DrCNN RCAN
650K+15.59M
1m6.2
10
24.03
24.16
24.21
24.13
24.03
23.88
23.62
22.76
20
23.45
23.58
23,70
23.73
23.09
23,57
23.42
23.23
22.46
0
24.76
25.55
26.54
27.33
26.55
25.55
24.64
25.99
25.49
DoCNN +IKC
650K+5.32M
79mms
10
24.20
24.54
24.86
24.96
24.78
24.52
24.23
24.19
23.14
20
23.62
23.87
24.07
24.15
24.06
23.86
23.65
23.59
22.71
80.80
26.20
26.45
26.46
26.30
26. 20
26. 39
25.57
23.96
DRUNN
650K+19.05M
192.83ms
10
24.05
24.28
24.44
24.50
24.40
24.27
24.09
23.85
22.90
20
23.58
23.75
23.88
23,93
23.88
23.72
23.56
23.40
22.58
27 20
27.62
27.74
27.85
27.82
27.
27.38
27.
26.27
44.14ms
10
25.26
25.57
25.64
25.69
25.62
25.54
25.42
25.20
24.37
23.68
23.87
24.20
24.32
24.09
23.91
23.76
23.81
22.87
0
27.67
27.99
28.14
28.20
28.12
27.99
27.80
27.87
26.52
3DSR (Ours)
5.80M
38.74ms
25.74
25.97
20
24.72
24.89
24.92
24.89
24.92
24.82
24.70
24.59
23.84
HR
DCLS
DASR
DCLS
DASR
Bicubic
ICAN
KDSK;
Bicubic
RCAN
KDSR:
Figure 4: 4x visual comparison. Noise levels are set to 10 and 20 for these two images separately. IKC (Gu et al., 2019), DAN (Luo et al., 2020), AdaTarget (Jo et al., 2021), and DASR (Wang al., 2021a). Note that RCAN is state-of-the-art SR method for bicubic degradation. For fair comparison on different model sizes, we develop KDSRg-M and KDSR by adjusting the depth and channels of the network. We apply Gaussian8 (Gu et al., 2019) kernel setting on five datasets, including Set5 (Bevilacqua et al., 2012), Set14 (Zeyde et al., 2010), B100 (Martin et al., 2001), Urban 100 (Huang al., 2015), and Manga109 (Matsui et al., 2017), to generate evaluation datasets.
The quantitative results are shown in Tab. 1. We can see that our KDSR surpasses DASR by 0.6dB, 0.39dB, 0.67dB and 1.24dB on Set5, Set14, Urban 100 and Manga 109 datasets separately. In addition, compared with the Blind-SR method DANv2, our KDSR achieves better performance consuming only 21% FLOPs of DANv2. It is because that DANy2 uses an iterative strategy to esti- mate accurate explicit blur kernels, which requires many computations. Besides, compared with the SOTA Blind-SR method DCLS, our x-LR5-I. achieves better performance on almost all datasets consuming less time. It is notable that DCLS specially designed explicit degradation estimator for blur kernel, while the KD-IDE in our KDSR is simple and can adapt to any degradation process. The qualitative results are shown in Fig. 3. We can see that our KDSR has more clear textures compared with other methods. Our KDSR also achieves better visual results than DANv2.
4.3 EVALUATION WITH ANISOTROPIC GAUSSIAN KERNELS AND NOISES
We evaluate KDSR on degradation with anisotropic Gaussian kernels and noises by adopting 9 typical blur kernels and different noise levels. We compare our KDSR with SOTA blind-SR methods, including RCAN (Zhang et al., 2018c), IKC (Gu al., 2019). DCLS (Luo et al., 2022) and DASR (Wang al., 2021a). Since RCAN, IKC, and DCLS cannot deal with noise degradation, we use DrCNN (Zhang et al., 2017), SOTA denoising method, denoise images for them.
The quantitative results are shown in Tab. 2. Compared with the SOTA explicit degradation esti- mation based on Blind-SR methods DCLS, our KDSRg surpasses it by over dB under almost all 20, our
degradation settings consuming 29.4% parameters and 5.1% runtime. Furthermore, KDSR surpasses DASR about 1dB with less parameters and runtime. This shows the superiority of knowledge distillation based IDR estimation and efficient SR network structure. In addition, provide visual comparison Fig. We can see that KDSR, has sharper edges, more realistic details, and fewer artifacts compared with other methods. More visual results are given in appendix.

---

Published as a conference paper at ICLR 2023
Table 3: 4x SR quantitative comparison real-world SR competition benchmarks. The FLOPS
and runtime are computed based on an LR size of 180 x 320. The best results are marked in bold.
AIM2019
NTTRE2020 PSNR1
Methods
Parms (M)
FLOPs(G)
Runtime (ms)
LPIPS
PSNR
SSIM
LPIPS]
SSIM1
ESRGAN
16.69
871.25
236.04
0.5558
23.17
0.6192
0.5938
21.14
0.3119
BSRGAN
16.69
871.25
236.04
0.4048
24.20
0.6904
0.3691
26.75
0.7386
Real-ESRGAN
16.69
871.25
236.04
0.3956
23.89
0,6892
0,3471
26.40
0.7431
MM ReallR
26.13
930.54
290.64
0.3948
23.45
0.6889
0.3446
25.19
0.7404
KDSR,-GAN (Ours)
18.85
640.84
154.62
0.3758
24.22
0.7038
0.3198
27.12
0.7614
HR
BSRGAN
MM- RcaiSR
11
BSRGAN
MM- MM-RoalSR
Bicubic
Real-ESRGAN KDSR,-GAM
Bicubic
Real-ESRGAN
KDSR,-GAN
Figure 5: 4x visual comparison on real-world SR competition benchmarks.
4.4 EVALUATION ON REAL-WORLD SR
We further validate the effectiveness of KDSR on Real-World datasets. As described in Sec. 4.1, we introduce GAN (Goodfellow al., 2014) and perceptual Johnson al. (2016) loss to train our network with the same high-order complex degradation process Real-ESRGAN (Wang et al., 2021b), obtaining 5 GAN. We compare our methods with the state-of-the-art GAN-based SR methods, including Real-ESRGAN, BSRGAN (Zhang al., 2021), MM-RealSR (Mou al., 2022), ESRGAN (Wang et al., 2018b). We evaluate all methods on the dataset provided in the challenge of Real-Worke Super-Resolution: AIM19 Track2 (Lugmayr et al., 2019) and NTIRE2020 Trackl (Lugmayr et al., 2020). Since AIM19 and NTIRE2020 datasets provide a paired validation set, the LPIPS (Zhang al., 201 2018b), PSNR, and SSIM for the evaluation.
The quantitative results are shown in Tab. 3. Compared with the recent real-world SR method MM-RealSR, our KDSR5-GAN performs better, only consuming about 50% runtime. In addi- tion, KDSR5-GAN outperforms SOTA real-world SR method Real-ESRGAN on LPIPS, PSNR, and SSIM, only consuming 75% FLOPs. Furthermore, we provide qualitative results in Fig. 5. We can see that our KDSR -GAN produces more visually promising results with clearer details and textures. More qualitative results are provided in appendix.
5 ABLATION STUDY
Knowledge Distillation Based Blind-SR Network. In this part, we validate the effectiveness of the components in KDSR, such as KD and IDR-DDC (Tab. 4). KDSR 4 is actually the KDSR adopted in Tab. 1, and KDSR KDSR corresponding teacher network.
(1) We directly in- put the degradation blur kernels into the KDSR obtaining KDSR Compared with KDSRs3, KDSR has a similar performance by estimating IDR. That demonstrates that our KDSR can es- timate quite accurate IDR guide Blind-SR. (2) We cancel the KD in KDSR 4 to obtain KDSR which means that 2 cannot learn the IDR extraction from KDSR,- Comparing KDSR and KDSR we can see that the KD scheme can bring 0.42dB improvement for KDSR which demonstrates that KD can effectively help KDSR 4 to learn the IDR extraction ability from KDSRT. (3) Based KDSR 2, we replace the IDR-DDC in IDR-DCRB with ordinary convolution to ob- tain 1. 2 is 0.17dB higher than KDSR 1, which demonstrates the effectiveness of IDR-CDC. (4) Besides, 4 0.2dB lower than its teacher KDSR,. That means KDSR cannot completely learn the IDR extraction ability from KDSRT-
The Loss Functions for Knowledge Distillation. We explore which KD function is best to guide the KDSRs learn to directly extract the same IDR as KDSR from LR images. Although there some works (Gao et al., 2018; He et al., 2020) have explored the KD function for SR, they take intermediate feature maps F E R H W learning objects to compress SR models. However,
Table 5: Comparison between different KD loss functions.
Loss
PNSR (dB)
L1 (Bq. 9)
25.92
La (Eq. 10)
25,88
Les (Eq. 6)
25.96
we take IDR D/r E R4C as learning objects to learn the ability to extract IDR from LR images. Therefore, cannot directly apply these experiences from previous
8

---

Published as a conference paper at ICLR 2023
Table 4: PSNR results evaluated Urban with Gaussian8 (Gu et al., 2019) kernels for 4x SR.
The FLOPs and runtime are both measured on an LR size of 180 x 320.
Oracle
Method
KD
IDR-DDC
Param (M)
FLOPs (G)
Time (ms)
PSNR (dB)
Degradation
KDSRy (Ours)
X
x
5.82
192.77
39.05
16.16
1
X
x
X
5.58
293.46
47.80
25.37
KDSR₂2
X
X
V
5.80
191.42
38.74
25.54
K
V
X
V
6.13
166.26
41.06
26.08
g (Ours)
X
V
5.80
191.42
38.74
25.96
27
+10
-20
-30 -15 30
-20 -3
-20
16 20 26 +23 -20
(a) IDK extracted by KUSKY
(b) IDR extracted by KUSRs.
(c) IDK extracted by KUSK; without K.D.
(d) IDR extracted by DASR.
Figure 6: Visualization of IDR with different isotropic Gaussian blur kernels on various methods. works our model. Here, we define three classic KD functions: (1) We use the Kullback Leibler divergence to measure distribution similarity (LH, Eq. 6). (2) We define L1 for optimization (Eq. 9). (3) Motivated by KD loss SR model compression (Gao et al., 2018), we define L2 (Eq. 10).
40
L1 =
(1s(1) D((()),,
(9)
4C
L 4C
(D'(()) -D)(()²
(10)
where D'r and D's E RAC IDRs extracted by KDSR7-M and KDSR respectively. We apply these three loss functions on KDSRs-M separately to learn the IDR from KDSR,-M. Then, evaluate them on 4x Urban100 with Gaussian8 kernels. The results are shown Tab. 5. We can see that the performance of L is better than &1 and L2. That means that the degradation information mainly contained in the distribution IDR rather than its absolute values.
The Visualization of KD-IDE. To further validate the effectiveness of our KD-IDE, we use t- SNE (Van der Maaten Hinton, 2008) to visualize the distribution of extracted IDR. Specifically, we generate LR images from BSD100 (Martin et al., 2001) with different isotropic Gaussian kernels and feed them to KDSRT, KDSR s, KDSRg without KD, and DASR (Wang et al., 2021a) to generate IDR D for Fig. 6 (a), (b), (c), and (d) respectively. We can from Fig. 6 (a) and (b) that KDSR, can distinguish different degradations, and KDSRg also learn this ability from KDSRG well. In addition, comparing Fig. 6 (b) and (c). can see that KDSR obtaining IDR extraction knowledge from KDSR, can distinguish various degradations better than KDSR, without adopting KD. That further demonstrates the effectiveness of our KD-IDE. Furthermore,
compare KDSR and DASR (Fig. 6 (b) and (d)), and the results show that KDSR can distinguish various degradations more clear than DASR, which shows the superiority of KD based IDE metric learning based IDE.
CONCLUSION
Most Blind-SR methods tend elaborately design an explicit degradation estimator for a specific type of degradation to guide SR. Nevertheless, it is difficult to provide the labels of multiple degra- dation combinations to train explicit degradation estimators, and these specific designs for certain degradation make them hard to transfer to other degradation processes. To address these issues, we develop a knowledge distillation based Blind-SR (KDSR) network, consisting efficient SR network that is stacked by IDR-DCRB. We use KD to make KD-IDEg directly extract the same accurate IDR as KD-IDE; from LR images. IDR-DCRBs of SR network use IDR based depthwise dynamic convolution to fully and efficiently utilize the extracted IDR guide SR. Ex- tensive experiments on classic and complex real-world degradation processes demonstrate that the proposed KDSR can achieve general state-of-the-art Blind SR performance.
KD-IDE and an

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was partly supported by the Alexander von Humboldt Foundation, the National Nat- ural Science Foundation of China(No. 62171251), the Natural Science Foundation of Guangdong the Special Foundations for the Development of Strategic Emerg- JCISS0000010000111000 Oversea Cooperation Foundation of Tsinghua.
ing Industries
and CGGJ2D220020440022344410 1) and
REFERENCES
Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge single image super-resolution: Dataset and study. In CVPRW, 2017.
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In CVPR, 2019.
Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie line Alberi Morel. Low- complexity single-image super-resolution based on nonnegative neighbor embedding. BMVC, 2012. 7
Adrian Bulat, Jing Yang, and Georgios Tzimiropoulos. To learn image super-resolution, use a gan to learn how to do image degradation first. In
Jianrui Cai, Hui Zeng, Hongwei Yong, Zisheng Cao, and Lei Zhang. Toward real-worlh single image repeetresolttion: A new benchmark and a new model. In ICCV, 2019. 1
Victor Comnillere Abdelaziz Djelouah, Wang Yifan, Olga Sorkine-Hornung, and Christopher Schroers. Blind image super-resolution with spatially variant degradations. TOG, 2019.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning deep convolutional network for image super-resolution In ECCV, 2014. 1,2
Manuel Fritsche, Shuhang Gu, and Radu Timofte. Frequency separation for real-world super- resolution. In ICCVW, 2019.
Qinquan Gao, Yan Zhao, Gen Li, and Tong Tong. Image super-resolution using knowledge distilla- tion. In ACCV, 2018. 8,
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurlPS, 2014. 8
Jinjin Gu, Hannan Lu, Wangmeng Zuo, and Chao Dong. Blind super-resolution with iterative kernel correction. In CVPR, 2019. 3, 4, 5, 6, 7,
Zibin He, Tao Dai, Jian Lu, Yong Jiang, and Shu-Tao Xia. Fakd: Feature-affinity based knowledge distillation for efficient image super-resolution. In ICIP, 2020. 3,
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.0253/ 2015. 3
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 1004:00704.0461,, 2017. 5
Jia-Bin Huang, Abhishek Singh. and Narendra Ahuja. Single image super-resolution from trans- formed self-exemplars. In CVPR, 2015. 7
Younghyun Jo, Seoung Wug Oh, Peter Vajda, and Seon Joo Kim. Tackling the ill-posedness of super-resolution through adaptive target generation. In CVPR, 2021.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016. 2, 8
10

---

Published as a conference paper at ICLR 2023
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In CVPR, 2016. 1
Soo Ye Kim, Hyeonjun Sim, and Munchurl Kim. Koalanet: Blind super-resolution using kernel- oriented adaptive local adjustment. In CVPR, 2021. 3
Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. In CVPR, 2017.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et Photo-realistic single image super-resolution using generative adversarial network. In CVPR, 2017. 2
Wonkyung Lee, Junghyup Lee, Dohyung Kim, and Bumsub Ham. Learning with privileged infor- mation for efficient image super resolutoso. In ECCV, 2020. 3
Jie Liang, Hui Zeng, and Lei Zhang. Efficient and degradation-adaptivy network for real-world image super-resolution. arXiv preprint arXiv:2203. 14216, 2022. 2, 3,4
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep resid- ual networks for single image super-resolution. In CVPRW, 2017. 2,
Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. Structured knowledge distillation for semantic segmentation. In CVPR, 2019.
Andreas Lugmayr, Martin Danelljan, Radu Timofte, Manuel Fritsche, Shuhang Gu, Kuldeep Puro- hit, Praveen Kandula, Maitreya Suin, AN Rajagoapalan, Nam Hyung Joon, et al. Aim 2019 challenge real-world image super-resolution: Methods and results. In ICCVW, 2019. 8
Andreas Lugmayr, Martin Danelljan, and Radu Timofte. Ntire 2020 challenge on real-world image super-resolution: Methods and results. In CVPRW, 2020. 8
Zhengxiong Luo, Yan Huang, Shang Li, Liang Wang, and Tieniu Tan. Unfolding the alternating optimization for blind super resolution. arXiv preprint arXiv:2010.02631, 2020. 2, 4,7
Ziwei Luo, Haibin Huang, Lei Yu, Youwei Li, Haoqiang Fan, and Shuaicheng Liu. Deep constrained least squares for blind image super-resolution. In CVPR, 2022. 1, 3, 5,
Cheng Ma. Yongming Rao, Yean Cheng, Ce Chen, Jiwen Lu, and Jie Zhou. Structure-preserving super resolution with gradient guidance. In CVPR, 2020. 2
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV, 2001. 7,
Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-based manga retrieval using mangal09 dataset. Multimedia Tools and Applications, 2017.
Chong Mou, Yanze Wu, Xintao Wang, Chao Dong, Jian Zhang, and Ying Shan. Metric learning based interactive modulation for real-world super-resolution. ECCV, 2022. 2, 3,
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014. 3
Assaf Shocher, Nadav Cohen, and Michal Irani. "zero-shot" super-resolution using deep internal learning. CVPR, 2018. 3,
Jae Woong Soh, Sunwoo Cho, and Nam Ik Cho. Meta-transfer learning for zero-shot super- resolution. In CVPR, 2020. 1,
Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming- Yang, and Lei Zhang. Ntire 2017 challenge single image super-resolution: Methods and results. In CVPRW, 2017. 6
11

---

Published as a conference paper at ICLR 2023
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 2008. 9
Longguang Wang, Yingqian Wang, Xiaoyu Dong. Qingyu Xu, Jungang Yang, Wei An, and Yulan Guo. Unsupervised degradation representation learning for blind super-resolution. In CVPR, 2021a. 1, 2, 3, 6, 7,
Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Recovering realistic texture in image super-resolution by deep spatial feature transform. In CVPR, 2018a.
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In ECCVW, 2018b. 1,8
Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In ICCV, 2021b. 1, 2, 3, 4, 5, 6, 8
Bin Xia, Yucheng Hang, Yapeng Tian, Wenming Yang, Qingmin Liao, and Jie Zhou. Efficient non-local contrastive attention for image super-resolution. AAAI, 2022a.
Bin Xia, Jingwen He, Yulun Zhang, Yucheng Hang. Wenming Yang, and Lue Van Gool. Structured sparsity learning for efficient video super-resolution. arXiv preprint arXiv:2206.07683 2022b.
Bin Xia, Yapeng Tian, Yucheng Hang, Wenming Yang. Qingmin Liao, and Jie Zhou. Coarse-to-fine embedded patchmatch and multi-scale dynamic aggregation for reference-based super-resolution. In AAAI, 2022c.
Bin Xia, Yapeng Tian, Yulun Zhang, Yucheng Hang. Wenming Yang. and Qingmin Liao. Meta-learning based degradation representation for blind super-resolution. arXiv preprint a7Xiv:2207.13963, 2022d.
Bin Xia, Yulun Zhang, Yitong Wang, Yapeng Tian, Wenming Yang, Radu Timofte, and Luc Van Gool. Basic binary convolution unit for binarized image restoration network. ICLR, 2023.
Yu-Syuan Xu, Shou- Yao Roy Tseng, Yu Tseng, Hsien-Kai Kuo, and Yi-Min Tsai. Unified dynamic convolutional network for super-resolution with variational degradations. In CVPR, 2020. 1,
Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse-
representations. International conference on curves and surfaces, 2010.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond gaussian denoiser: Residual learning deep enn for image denoising. TIP. 2017.
Kai Zhang, Wangmeng Zuo, and Lei Zhang. Learning a single convolutional super-resolution net- work for multiple degradations. In CVPR, 2018a. 2,
Kai Zhang, Wangmeng Zuo, and Lei Zhang, Deep plug-and-play super-resolution for arbitrary blur kernels. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 1671-1681, 2019. 3,3
Kai Zhang, Lue Van Gool, and Radu Timofte. Deep unfolding network for image super-resolution. In CVPR, 2020. 3,3
Kai Zhang, Jingyun Liang, Luc Van Gool, and Radu Timofte. Designing a practical degradation model for deep blind image super-resolution. arXiv preprini arXiv:2 10111113.00,,0 2021. 1, 2, 3, 8
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as perceptual metric. In CVPR, 2018b.
Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention networks. In Proceedings of the European conference on computer vision (ECCV), pp. 286-301, 2018c. 6,7
Ruofan Zhou and Sabine Susstrunk. Kernel modeling super-resolution on real low-resolution im- ages. In ICCV, 2019. 3
12


Reviews:
[{"review_id": "xey6O-Z4wm", "reviewer": "Reviewer_y283", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Most of the prior works for blind image super-resolution design explicit degradation estimation, which is often infeasible due to lack of proper degradation labels. Also, there might not exist a universal degradation model to capture various practical degradation types, such as blur, noise, compression, and platform instability which makes this problem particularly interesting. This paper proposes a method to learn an implicit degradation model leveraging learned representation in a teacher-student model. One of the important highlights of this paper is the generalization to various (real-world) degradation settings. Experiments show comparable or better performance than some existing models on commonly used datasets. The experiments on AIM19 and NTIRE2020 do not offer a significant gain over prior works which have not been included due to some reasons. The proposed approach is an amalgamation of previously known methods in related disciplines. \n", "summary_of_the_review": "This paper proposes a method to learn an implicit degradation model leveraging learned representation in a teacher-student model. One of the important highlights of this paper is the generalization to various (real-world) degradation settings. Experiments show comparable or better performance than some existing models on commonly used datasets. The experiments on AIM19 and NTIRE2020 do not offer a significant gain over prior works which have not been included due to some reasons. The proposed approach is an amalgamation of previously known methods in related disciplines. \n\n===========================================\nI thank the authors for their detailed response. I think the paper is in a good shape now, and it'd be interesting to many at the conference. Based upon the discussion, I plan to rate this as marginally above the acceptance level.  \n", "strength_and_weaknesses": "**Main Review**\n\n**Major Strengths**\n\nReal-world blind super-resolution with a combination of multiple unknown degradation is a challenging problem to deal with using explicit degradation models. This paper proposes to learn an implicit degradation model by leveraging a previously known method called knowledge distillation in a teacher-student setup. \n\n***Experiments***\n\nThe experiments are comprehensive and provide comparable or better performance than compared methods. The proposed method is computationally efficient in general. \n\nExperiments on various datasets with different degradation models show promising results. \n\nThe authors demonstrate generalization to several degradations which is an important highlight of this work. \n\n**Weaknesses**\n\nThe main idea is an amalgamation of previously known methods, such as knowledge distillation and efficient super-resolution using GANs.\n\nThe loss function consists of standard objective functions with several regularization parameters. The authors should address how to tune all these parameters in large-scale real-world blind super-resolution tasks.\n\nTable 3: Are the compared methods really the state-of-the-art for real-world SR on AIM19 and NTIRE2020?\n\nIn the literature, the results of DASR on AIM19 are reported to be better than the results obtained by the proposed method KDSR. In my opinion, the authors should clarify how the compared methods are SOTA, maybe in terms of the training procedure, e.g., paired or unpaired learning or appropriate metrics to make a proper assessment of the baselines. \n\nThere is a disparity between the perceptual metrics, such as FID and LPIPS of popular SR methods including DASR (also the winner of AIM19 challenge FSSR) and the distortion metrics, such as PSNR and SSIM. As a result, there is a trade-off between perceptual and distortion quality. I wonder whether the disparity exists in the proposed method KDSR. If yes, how to overcome such issues?\n\nIt is well-known that quantitative improvements in terms of PSNR and SSIM do not necessarily indicate qualitative improvements, especially for the tasks under consideration. This is evident from Table 1, 2, 3, 4, and 5 of the main paper because most methods have similar scores. To make a fair comparison with baselines, I suggest the authors include LPIPS, FID or other image quality metrics on all the datasets.\n\nFigure 8: Are the results generated for isotropic or non-isotropic Gaussian kernels?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "crNGz7Hg_3", "reviewer": "Reviewer_GVEj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for blind SR in which a teacher network is used to estimate degradation from LR and HR to guide the student network to get the degradation from just LR. The experiment results show the proposed method performs better than existing methods (especially for the synthetic experiments).", "summary_of_the_review": "The novelty is somehow marginal and the experiments cannot fully demonstrate the necessity of the proposed method.\n\nAfter rebuttal, the authors address most of my concerns. I lean to accept this paper as long as the authors add more experiments and explain the reason that the proposed method has good generalization ability.", "strength_and_weaknesses": "Strength:\nThe idea that implicitly estimating degradation and then using it for blind SR is reasonable.\n\nWeaknesses:\n1. Estimating a degradation and then using it for blind SR is not new, the novelty of the proposed method resides in estimating it implicitly. However, most of the experiments only consider blur kernel, SR and noises. They are not sufficient enough to validate the advantages of the proposed method over existing arts. Since these degradations can be easily simulated and the settings are easily treated as GT to be learned from a network and used in the following SR network (as explicit degradation). More complicated degradations should be considered to validate the effectiveness of the proposed method.\n2. According to the results of Table 3, the proposed method does not have obvious advantages relative to other methods (especially BSRGAN).\n3. For Table1 and Table2, are the compared methods trained on the same training set of the proposed method? If not, the comparison is unfair.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LwROlJsg-z", "reviewer": "Reviewer_cRmZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a method for blind single image super-resolution. Contrary to the mainstream the idea of this method is to design an implicit degradation estimator that can extract information regarding the degradation without having an explicit model. This is done using ideas from Knowledge distillation where first a teacher model is trained using paired LR/HR images and then a student model is trained only using LR images to predict the same degradation representation. ", "summary_of_the_review": "This paper presents an interesting idea of using knowledge distillation to learn the degradation operation on the blind single image super-resolution task. The method produces very good results in the shown benchmarks. The main issue is the presentation. It is hard to disentangle the mathematical formulation from the implementation as everything is mixed together. Nonetheless I think this paper introduced a novel idea so I'm leaning towards accepting the paper. But, I would like to see a better presentation where the mathematical formulation (e.g., the abstract description of the modules, the goals of each module, the loss functions) is given in a separate section as the details about the architecture (and the exact implementation of the modules). I would like to see also results on real images (not from a benchmark). For example results on any image of the Open Image Dataset and a visual comparison in terms of artifacts (since no metric will be available).\n\nAfter Rebuttal.\nThe authors addressed the concerns raised during the first round of reviews, and updated the manuscript accordingly. I'm leaning towards acceptance and thus I keep my original score.", "strength_and_weaknesses": "Strengths:\n1. The paper introduces an interesting idea to address a highly relevant practical problem (blind real image super-resolution)\n2. The paper presents many experiments and ablation studies showing that the method is robust and produces state-of-the-art results in terms of distortion metrics (PSNR).\n3. The method is well detailed, and well motivated.\n\nWeakness:\n1. Presentation is cumbersome. It is hard to disentangle the contributions of the paper since everything is mixed together. Details about the architecture are given as long as the mathematical formulation. It would much better if the paper first present a mathematical formulation of the problem (the training, the loss functions, the goals), and later present the architecture and the details about the modules. In the current formulation is extremely hard to distill the ideas and understand what is new and what is not.\n2. All the results are with benchmarks that use simulated degradations. It would be nice to see results (visual) on real images captured by any camera since the paper is claiming that addresses the real image super-resolution problem.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AQek_fJdQU", "reviewer": "Reviewer_HYPQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use a knowledge distillation based to learn an implicit representation of degradations. The proposed method can handle complex degradation and does not need explicit degradation supervision. The proposed method finally achieve better results on several synthesized datasets. ", "summary_of_the_review": "This paper proposes a novel idea to implicitly estimate the degradation. While the experiments do not fully explore the estimated degradations and verify the method on real-image datasets. I will consider changing my opinion if the authors can provide more experiment details in the rebuttal phase.\n\n--Post Rebuttal--:\nThe authors provide more persuasive experimental results and the response addresses most of my concerns. I would like to improve my score.", "strength_and_weaknesses": "1.  The proposed method is novel and performs well on synthetic datasets.\n\n2. The authors claim that the proposed KDSR implicitly estimates a degradation representation. While there are no experiments analyzing the estimated representations. The authors are suggested to compare the estimated degradations of images with the same/similar/different degradations. If the authors' claim stands, images that have different content but the same degradations will also have a similar estimated representation.\n\n3. The experiments are performed on synthetic datasets. The authors are suggested to verify their methods on real-image datasets, such as the track2 of NTIRE2020. \n\n4. It will be better if the visual results provide comparisons of the whole images, since the degradation of the LR image may be spatially variant, while the estimated degradation representation is spatially invariant.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster