Discovering Distinctive ``Semantics'' in Super-Resolution Networks
Authors: Yihao Liu
Anran Liu
Jinjin Gu
Zhipeng Zhang
Wenhao Wu
Yu Qiao
Chao Dong
Venue: Submitted to ICLR 2023 (2023)

Abstract: Image super-resolution (SR) is a representative low-level vision problem. Although deep SR networks have achieved extraordinary success, we are still unaware of their working mechanisms. Specifically, whether SR networks can learn semantic information, or just perform complex mapping function? What hinders SR networks from generalizing to real-world data? These questions not only raise our curiosity, but also influence SR network development. In this paper, we make the primary attempt to answer the above fundamental questions. After comprehensively analyzing the feature representations (via dimensionality reduction and visualization), we successfully discover the distinctive ``semantics'' in SR networks, i.e., deep degradation representations (DDR), which relate to image degradation instead of image content. We show that a well-trained deep SR network is naturally a good descriptor of degradation information. Our experiments also reveal two key factors (adversarial learning and global residual) that influence the extraction of such semantics. We further apply DDR in several interesting applications (such as distortion identification, blind SR and generalization evaluation) and achieve promising results, demonstrating the correctness and effectiveness of our findings.

Under review as a conference paper at ICLR 2023
DISCOVERING DISTINCTIVE "SEMANTICS" IN SUPER- RESOLUTION NETWORKS
Anonymous authors
Paper under double-blind review
Clevification
Supar-rsooittonn
Test
Projection
Classification Networks
Only trained clean images without exposure to other degraded data
deep represeitaiions
eat
noist
blur
Test
Projection
plane
Super-resoution Networlss
Content: related semantics
Degradation semantics
Figure I: Distributions of the deep representations of classification and super-resolution networks. For classi- fication networks, the semantics of the deep feature representations are artificially predefined according to the training data (category labels). However, for SR networks, the learned deep representations have different kind of "semantics" from classification. During training, the SR networks only provided with downsampled clean LR images. There not any supervision signal related image degradation information. Surprisingly, we find that SR networks' deep representations are spontancously discriminative to different degradations. No- tably, NOT an arbitrary SR network has such a property. In Sec. 4.3 reveal two factors that facilitate SR networks to extract such regradation-related representations, i.e., adversarial learning and glohal residual.
ABSTRACT
Image super-resolution (SR) is a representative low-level vision problem. Al- though deep SR networks have achieved extraordinary success, aware of their working mechanisms. Specifically, whether SR networks can learn semantic information, just perform complex mapping functions? What hinder- SR networks from generalizing to real-world data? These questions not only raise our curiosity, but also influence SR network development. this paper, we make the primary attempt to answer the above fundamental questions. After com- prehensively analyzing the feature representations (via dimensionality reduction and visualization), we successfully discover the distinctive "semantics" in SR net- works, i.e., deep degradation representations (DDR), which relate to image degra- dation instead of image content. We show that well-trained deep SR network is naturally a good descriptor of degradation information. Our experiments also reveal two key factors (adversarial learning and global residual) that influence the extraction of such semantics. We further apply DDR in several interesting appli- cations (such distortion identification, blind SR and generalization evaluation) and achieve promising results, demonstrating the correctness and effectiveness of our findings.
still
INTRODUCTION
The emergence of deep convolutional neural network (CNN) has given birth to large number of
new solutions to low-level vision tasks (Dong et al.] 20044 Zhang et al. 2017). Among these signs of progress, image super-resolution (SR) has enjoyed great performance leap. Compared with traditional methods (c.g., interpolation (Keys) 1981) h and sparse coding (Yang et al. 2008)), SR networks achieve better performance with improved efficieccc.

---

Under review as a conference paper at ICLR 2023
However, even if we have benefited a lot from the powerful CNNs, we have little knowledge about what happens in SR networks and what distinguishes them from traditional approaches on earth. Does the performance gain merely come from more complex mapping functions? Or is there any- thing different inside SR networks, like classification networks with discriminative capability? On the other hand, as classic regression task, SR is expected to perform continuous mapping from low-resolution (LR) to high-resolution (HR) images. It is generally a local operation without the consideration of the global context. But with the introduction of GAN-based models [edig et al. (017)); Wang et al. (2018), more delicate SR textures can be generated. It seems that the network has learned some kind of semantic, which is beyond our common perception for regression tasks.
Then, we may raise the question: are there any "semantics" in SR networks? If yes, do these se- mantics have different definitions from those in classification networks? Existing literature cannot answer these questions, as there is little research on interpreting low-level vision deep models. Nev- ertheless, discovering the semantics in SR networks is of great importance. It can not only help us further understand the underlying working mechanisms, but also guide us to design better networks and evaluation algorithms.
In this study, we give affirmative answers to the above questions by unfolding the semantics hid- den in super-resolution networks. Specifically, different from the artificially predefined semantics associated with object classes in high-level vision, semantics in SR networks are distinct in terms of image degradation instead of image content. Accordingly, we name such semantics deep degra- dation representations (DDR). More interestingly, such degradation-related semantics are sponta- neously existing without any predefined labels. We reveal that a well-trained deep SR network is naturally a good descriptor of degradation information.
Notably, the semantics in this paper have different implications from those in high-level vision. Previously, researchers have disclosed the hierarchical nature of classification networks (Zeiler & Fergus 2014 Gu et al. 2018). As the layer deepens, the learned features respond more to abstract high-level patterns (e.g., faces and legs), showing a stronger discriminability to object categories (see Fig. 4. However, similar research in low-level vision is absent, since there are no predefined semantic labels. In this paper, we reveal the differences in deep "semantics" between classification and SR networks, as illustrated in Fig.
Our observation stems from a representative blind SR method CinCGAN Yuan et al. (2018). and we further extend it to more common SR networks - SRReseRReeeeeeeee and SRGAN Ledig et al. (2017) We have also revealed more interesting phenomena to help interpret the semantics, including the analogy to classification networks and the influential factors for extracting DDR. Moreover, improve the results of several tasks by exploiting DDR. We believe our findings could lay the groundwork for the interpretability of SR networks, and inspire more exploration of the mechanism of low-level vision deep models.
Contributions. 1) We have successfully discovered the "semanties" in SR networks, denoted as deep degradation representations (DDR). Through in-depth analysis, we also find that global residual learning and adversarial learning can facilitate the SR network extract such degradation-relates representations. 2) We reveal the differences in deep representations between classification and SR networks. for the first time. This further expands our knowledge of the deep representations of high- and low-level vision models. 3) We exploit our findings to several fundamental tasks and achieve very appealing results, including distortion identification, blind SR and generalization evaluation.
2 RELATED WORK
Super-resolution (SR) is a fundamental task in low-level vision, which aims to reconstruct the high-resolution (HR) image from the corresponding low-resolution (LR) counter- part. SRCNN (Dong et al., 2014) is the first proposed CNN-based method for SR. Since then, a large number of Seeep-learning-base based methods have been developed (Dong et al. 2016 Lim et al. 2017 2017 et al., 2018b) Ledig et al. 20 et al., 2019). Generally, current CNN-based SR methods can be categorized into two groups One is MSE-based method, which targets at minimiz- ing the distortion (e.g.. Mean Square Error) between the ground-truth HR image and super-resolved image to yield high PSNR values, such as SRCNN (Dong et al.] 2014). VDSR (Kim et al.) 2016), EDSR (Lim et al. 2017). RCAN (Zhang et al. 2018b), SAN (Dai et al., 2019). etc. The other is GAN-based method, which incorporates generative adversarial network (GAN) and perceptual loss (Johnson et al. 2016) to obtain perceptually pleasing results, such SRGAN (Ledig et al. 2017).
Super-resolution.
2

---

Under review as a conference paper at ICLR 2023
CinCGAN (eiV (DNV2C-mii))
Ingut
BM3D
Different degraded in- put images and their correspond-
Figure 2:
DIVaK-mil
ing outputs produced by CinC-
GAN (Yuan et al. 2018). BM3D
(a)
(Dabov et al., 2007). and SRC-
NN (Dong et al. 2014). CinC-
GAN (Yuan et al. 2018 is trained
DIV2K dataset in
DIVIE
paired manner. If the input im-
age conforms to the training data distribution, CinCGAN will gener- ate better restoration results than BM3D (a). Otherwise, it tend- ignore the unseen degradation types (b)&(c). On the other hand, the traditional method BM3D (D- abov et al., (007) has stable per- formance and similar denoising ef- fects on all input images, regard- less of the input degradation types. Zoom in for the best view.
(b)
Hollywood
(c)
ESRGAN (Wang et al. 2018). RankSRGAN (Zhag et al. 2019).
SROBB RRad et al. 2019 Re-
cently, blind SR has attracted more and more attention (Gu al. 2019 Bell-Kligler al. 2019 Luo et al. 2020 Wang et al. 2021). which aims to solve SR with unknown real-world degrada- tion. comprehensive survey for blind SR is newly proposed (Liu al., (022), which summarizes existing methods. We regard SR representative research object and study its deep semantic representations. can also draw inspirations on other low-level vision tasks.
Network interpretability. At present, most existing works on neural network interpretability fo- cus on high-level vision tasks, especially for image classification. Zhang al. (Zhang et al.) (020) systematically reviewed existing literature network interpretability and proposed novel taxon- omy to categorize them. Here we only discuss several classic works. By adopting deconvolutional
networks (Zeiler et al. 20100), Zeiler al. (Zeiler & Fergus) 2014) projected the downsampled low- resolution feature activations back to the input pixel space, and then performed a sensitivity analysis reveal which parts of the image important for classification. Simonyan al. (Simonyan al.) 2013) generated a saliency map from the gradients through a single backpropagation pass. Based on class activation maps (CAM) (Zhou et al. 2016), Selvaraju al. (Selvaraju et al. 2017) proposed CAM) to produce a coarse-grained attribution map of the important
Grad-CAM (Gradient-weighee regions in the image, which was broadly applicable any CNN-based architecture. For more infor- mation about the network interpretability literature, please refer to the survey paper (Zhang et al.) 2020). However, for low-level vision tasks, similar researches are rare. Recently, the local attri- bution map (LAM) (Gu & Dong 2021) has been proposed to interpret super-resolution networks, which can be used to localize the input features that influenced the network outputs. Besides, Wang et al. (Wang et al. 2020b) presented a pioneer work that bridges the representation relationship be- tween high- and low-level vision. They learned the mapping between deep representations of low- and high-quality images, and leveraged it deep degradation prior (DDP) for low-quality im- age classification. Inspired by these previous works, we interpret SR networks from another new perspective. We dive into their deep feature representations, and discover the "semantics" of SR networks. More background knowledge is described in the supplementary file.
MOTIVATION
To begin with, we present an interesting phenomenon, which drives us to start exploring the deep representations of SR networks. It is well known that SR networks are superior to traditional meth- ods in specific scenarios, but are inferior in generalization ability. In blind SR, the degradation types of the input test images are unknown. For traditional methods, they treat different images equally without distinction of degradation types, thus their performance is generally stable and predictable. How about the SR networks, especially those designed for blind SR?
3

---

Under review as a conference paper at ICLR 2023
(a) CisCGAN (trainDDVVIK milld)
(b) SRCNN Itvan:DDV2KKmilld
(c) SRCNN (train: 2l-cl)
(d) SRGAN (trainDDI 2K 2leeeuuu
(c) Network Architoctures
Figure 3: (a)-(d): The projected deep feature representations. The deep features of CinCGAN and SRGAN are separated by degradation types, even if the image contents are aligned. (c)-a: 88 (He et al. 2016) for classification. "Conv2.x" represents represetssssssssss the 2nd group of residual blocks. (e)-b: NeR (without global residual). (e)-c: SRResNet (with global residual). "RBI" represents the 1st residual block.
CinCGAN (Yuan et al., 2018) is representative solution for real-world SR without paired training data. It maps a degraded LR to its clean version using data distribution learning before conducting SR operation. However, we find that it still has limited application scope even if CinCGAN is developed for blind settings. If the degradation of the input image is not included in the training data, CinCGAN will fail to transfer the degraded input to clean one. More interestingly. instead of producing extra artifacts in the image, it seems that CinCGAN does not process the input image and retains all the original defects. Readers can refer to Fig. 2 for an illustration, where CinCGAN performs well on the testing image of the DIV2K-mild dataset (same distribution as its training da- ta). but produces unsatisfactory results for other different degradation types. In other words, the network seems to figure oul the specific degradation types within its training data distribution, and distribution mismatch may make the network "turn off" its ability. This makes the performance of CinCGAN unstable and unpredictable. For comparison, we process the above three types of degrad- ed images by a traditional denoising method BM3D (Dabov et al. 2007 The visual results show that BM3D has an obvious and stable denoising performance for all different degradation types. Although the results of BM3D may be medioere (the image textures are largely over-smoothed), does take effect every input image. This observation reveals significant discrepancy between traditional methods and SR networks.
The above interesting phenomenon indicates that the deep network has learned more than a regres- sion function, since it demonstrates the ability to distinguish among different degradation types. Inspired by this observation, we try to find any semantics hidden in SR networks
DIVING INTO THE DEEP DEGRADATION REPRESENTATIONS
4.1 DISCRIMINABILITY OF DEEP REPRESENTATIONS IN DEEP SR NETWORKS
Feature projection and visualization. Since the final outputs are always derived from features in CNN layers, we start the exploration with feature maps, especially the deep ones potentially with more global and abstract information. To interpret the deep features of CNN, one common and ra- tional way is to convert the high-dimensional CNN feature maps into lower-dimensional datapoints that can be visualized in a scatterplot. Afterwards, one can intuitively understand the data struc- tures and manifolds. Specifically, we adopt t-Distributed Stochastic Neighbor Embedding (t-SNE) (Van der Maaten & Hinton| 2008) for dimensionality reduction. This algorithm is commonly used in manifold learning, and it has been successfully applied in previous works (Donahue al. 2014 Mnih et al. 2013: /Wen et af.] 2016,] Zahavy et al.) 2016)] Veličković et al. [017] Wang et al. 20206
Huang et (0200) for feature projection and visualization. In our experiments, we first reduce the dimensionality of feature maps to a reasonable amount (50 in this paper) using PCA (Hotelling 1933), then apply t-SNE to project the 50-dimensional representation
two-dimensional space, af- ter which the results are visualized in scatterplot. Furthermore, we also introduce CHI (Calinski & Harabasz 1974) score to quantitatively evaluate the distributions of visualized datapoints. The CHI score is higher when clusters are well separated, which indicates stronger semantic discrininability.
What do the deep features of SR networks represent? As discussed in See 3 since CinCGAN performs differently various degradations, we compare the features generated from three test- ing datasets: 1) miil:- training and testing data used in CinCGAN, which are synthesized
Note that BM3D is a denoising method while CinCGAN is able to upsample the resolution of the input image. Thus, after applying BM3D, apply hicubic interpolation to unify the resolution of the output image. This is reasonable we only evaluate their denoising effects.
4

---

Under review as a conference paper at ICLR 2023
from DIV2K (Agustsson & Timoftel 2017) dataset, containing noise, blur, pixel shifting and other
degradations. 2) Div2K-noise20; add Gaussian noise (a = 20) to DIV2K set. 3) Hollywood 100 images selected from Hollywood dataset (Laptev al. 2008 containing real-world old film degradations. Each test dataset includes 100 images.
As shown in Fig. B)) there is a strong feature discriminability for various degradations. Images with aligned contents but different degradation types are still separated into different clusters. P This phenomenon conforms our observation that CinCGAN does treat various input degradations in different ways. It naturally reveals the "semantics" of deep representations in CinCGAN, which are closely related to the degradation types rather than the image content. For comparison. we may wonder whether traditional methods have similar behaviors (or "semantics"), "). However, our feature analysis method can only work for deep models, which contain hierarchical feature maps. It is acknowledged that the simplest network- - SRCNN can be analogous to sparse-coding- based method, thus we can use SRCNN to shed light on the behaviors of traditional methods. We train an SRCNN with the same data as CinCGAN, and visualize the feature representations of the last layer in Fig. Bb.). It is obvious that different degradations cannot be clearly separated. This phenomenon is completely different from CinCGAN. We can conjecture that the degradation-relatee semantics only exist in deep models, not traditional methods or shallow networks. More analysis on shallow networks can be found in the supplementary file.
From CinCGAN to Generic SRGAN. Notably, the training of CinCGAN involves degraded im- ages (DIV2K-mild). It actually performs simultaneous restoration and SR. We also wonder how this kind of degradation-relatee semantics manifests in classical SR networks (without exposure to other degradation types except for downsampling) Therefore, we adopt a generic GAN-based SR network SRGAN (Ledig et al. 2017: |Wann et al., 2018)) to conduct the visualization experiment. SRGAN trained with DIV2K dataset (Agustsson & Timofte, (017) with only secubic-downsampled LR images. According to the common degradation modelling in low-level vision, we use three datasets with different degradation types for testing: 1) DCV2k-clean: the original DIV2K validation set containing only bicubic downsampling degradation, which conforms to the training data distribu- tion. 2) DIV2K-blur introduce blurring degradation with Gaussian blur kernel on the DIV2K -clean set. The kernel width is randomly sampled from [2, 4 for each image and the kernel size is fixed to 15 x 15. 3) Dnoseen add Gaussian noises to the DIV2K-cleat set. The noise level is randomly sampled from [5, 30] for each image. These three testing datasets are aligned in image content but different in degradation types.
As shown in Fig a clustering trend similar to CinCGAN is clearly demonstrated. This provides stronger evidence for the existence of degradation-related semantics. Even if the three testing sets share the same content, they are still separated into distinet clusters according to the degradation types. In the supplementary file, similar phenomena are observed with other network structures. Note again, shallow SRCNN does not have such feature discriminability (see Fig c)))
There, we successfully find the semantics hidden in deep SR networks. They are perceivable to humans when visualized in fow-dimensional space. Specifically, semantics in deep SR networks are in terms of degradation types regardless of the image contents. Simply but vividly, we name this kind of semantics as deep degradation representations (DDR).
Is DDR a natural and trivial observation? No, there are three reasons. First, DDR has never been discussed before. The function of deep SR networks is beyond simple regression. The learned deep features can spontaneously characterize the image degradations, indicating that a well-trained deep SR network is naturally a good descriptor of degradation information. Note again that the deep SR networks have not observed any blurry noisy data during training, but still have the discrimina- tive ability on different degradations. Second, DDR in SR is not simply caused by different input patterns. We find that different networks will learn different semantic representations. For example, in Sec. 4.2 we reveal the differences in the learned representations between classification and SR Networks. In Sec. 4.3 we show that not all SR network structures can easily obtain DDR. DDR does not exist in specific cases and shallow networks. Third, DDR has important applications and inspirations. It can not only expand our understanding of the underlying mechanisms of low-level
Note that the class labels in the seatterplots are only used to assign color/symbol the datapoints for better visualization.
We use the same architecture the original paper Dong et al. 2014)) and add global residual for hetter visualization.
5

---

Under review as a conference paper at ICLR 2023
Gar
60g
horce
"Conv1"
"Conv2.4"
"Conv3.4""
"Conv4_4"
"oov5_
Figure 4: Projected feature representations extracted from different layers of ResNet18 using t-SNE. With the network deepens, the representations become more discriminative to object categories, which clearly shows the semantics of the representations in classification.
Confe
birdidiee
cortaour
praineino BAI
bindinobre)
ResNet18 (classification
(b) SRReaNet-WGR
(c) SRGAN-WGR
Figure 5: Feature representation differences between classification and SR networks, The same object catego- represented the same color, and the same image degradation type is depicted by the same marker shape. For the classification network, feature representations are clustered by the same color, while representations of SR networks are clustered by the same marker shape, suggesting that there a significant difference feature representations between classification and SR networks.
vision models, but also help promote the development of other tasks. In Sec. 5 several fundamental tasks and achieve appealing results, implying the great potential of DDR.
we apply DDR to
4.2
DIFFERENCES SEMANTICS BETWEEN CLASSIFICATION AND SR NETWORKS
In the high-level vision, classification is one of the most representative tasks, where artificially pre- defined semantic labels on object classes are given as supervision. We choose ResNet18 (He et al.] 2016 the classification backbone and conduct experiments CIFAR10 dataset (Krizhevsky et al. 2009). We extract the forward features of each input testing image at different network layers, as described in Fig, e)-aa
Fig. shows that as the network deepens, the extracted feature representations produce obvious discriminative clusters, i.e., the learned features are increasingly becoming semantically discrimi- native. Such discriminative semantics in classification networks are coherent with the artificially predefined labels. This is an intuitive and natural observation, on which lots of representation and
discriminative learning methods are based (Wen et al., 2016 Oord et al. 2018 Lee et al. 2019
Wang et al. 2020b).
Further, we add blur and noise degradation to the CIFARIO test images, and then investigate the feature representations of classification and SR networks. Note that no degradation added the training data. As shown in Fig. [5] after adding degradations to the test data, the deep representations obtained by the classification network (ResNet18) are still clustered by object categories, indicating that the features focus more high-level object class information. On the contrary, the deep rep- resentations obtained by SR networks (SRResNet and SRGAN) are clustered with regard to degra- dation types. The features of the same object category are not clustered together, while those of the same degradation type are clustered together, showing different "semantic" discriminability. This phenomenon intuitively illustrates the differences in the deep semantic representations between SR and classification networks, i.e., degradation-related semantics and content-related semantics. More interestingly, the "semantics" in SR networks exists naturally, because the SR networks only clean data without any input or labelled degradation information.
4.3 How Do GLOBAL RESIDUAL AND ADVERSARIAL LEARNING AFFECT THE DEEP REPRESENTATIONS?
Previously, have elaborated on the deep degradation representations CinCGAN, SRGAN and SRResNet. Nevertheless, we further discover that no arbitrary SR network structure has such property. To be specific, we find two crucial factors that can influence the learned representations: i) image global residual (GR), and ii) generative adversarial learning (GAN).
Forr efficiency, selected 100 testing images of cach category (1000 images in total).

---

Under review as a conference paper at ICLR 2023
"Cuny
"ResBlockd"
"ResBlockk"
"ResBlock6""
CIll: 0.00 di 0.00
(c) CHIl: 1 0.03
DiG cean
(c) CHI: 0.00 0.00
(f) CHI: 0.11 0.00
(g) CHI: 38 21 + 9.25
CHI: 613.77 + 33.40
Figure 6: Projected feature representations extracted from different layers of SeResNet-woGR (1st row) and SRResNet (2nd row) using t-NE With image global residual (GR), the representations of MSE-base SR networks show discriminability to degradation types.
"Cony
"ResBlack4"
"ResBlock8"
"ResBlock
(a) CHI: 0.00 0.00
(b) CHT: 0.01 + 0.00
(c) CHT: 34.00 + 22.00
(d) CHI: 234.43 + 30.34
(e) CHI: 0.00 0.00
(I) CHI: 0.03 + 0.02
CHI: 35.68 + 2.52
(h) CHI: 626 46 + 31.56
Figure 7: Projected feature representations extracted from different layers of SRGAN-woGR (Ist row) and
SRGAN (2nd row) using t-SNE. Even without GR, GAN-based SR networks can still obtain DDR.
Global Residual. We train two SRResNet networks - SRResNet (with global residual) and SoRG (without global residual), as shown in Fig. 3 The two architectures are both common in practice (Kim et al.) 2016) [hh et al. (016). DIV2K (Agustsson & Timofte, (017)) dataset is used for training, where the LR images are bicubi--donnsampled and clean. Readers can refer to the supplementary file for more details. After testing, the feature visualization analysis is
shown in Fig. 6
The results show that for MSE-based SR method, GR is essential for producing discriminative rep- resentations on degradation types. The features in "ResBlock16" of SRResNet have shown distinct discriminability, where the clean, blur, and noise data are clustered separately. On the contrary, SRResNet- shows no discriminability even in deep layers. This phenomenon reveals that GR significantly impacts the learned feature representations. It is inferred that learning the global residual could remove most of the content information and make the network concentrate more on the contained degradation. This claim is also corroborated by visualizing the feature maps in the supplementary file.
Ad Adversarial Learning. MSE-based and GAN-based methods are currently two prevailing trend- s in CNN-based SR methods. Previous studies only reveal that the output images of MSE-based and GAN-based methods are different, but the differences between their feature representations are rarely discussed. Since their learning mechanisms are quite different, will there be a discrepan- cy in their deep feature representations? We directly adopt SRResNet and SRResNet-woGG as generators. Consequently, we build two corresponding GAN-based models, namely SRGAN and SRGAN-woGR. After training, we perform the same test and analysis process mentioned earlier.
The results show that the deep features are bound to be discriminative to degradation types for the GAN-based method, whether there is GR or not. As shown in Fig. d)),,, the deep representations in "ResBlock16" of SRGAN-woGR have already been clustered according to different degradation types. This suggests that the learned deep representations method are dissimilar. Adversarial learning can help the network learn more informative features for distinguishing image degradation rather than image content.
MSE-basee method and GAN-based
4.4 How DOES DDR EVOLVE THROUGH THE TRAINING PROCESS?
We also reveal the relationship between the model performance and DDR discriminability. We select Net models with different training iterations for testing. We report the model performance
7

---

Under review as a conference paper at ICLR 2023
Fave poin oede
(a) Random initialized PSNR; 26.10
(b) Iteeation: 10K PSNR: 28.48
(c) Iteration: 100K PSNK: 28.81
(d) Iteration: 600K
PSNR: 28.97
CHI 0.00
CIII: 26.95
CII: 357.94
CIII: 591 68
Figure 8: As the training process goes, the performance and discriminabilityy improve simultancously.
Table 1: Distortion identification accuracy.
Table 2: The PSNRT/NIQEL results on datasets with
(a) BRISQUE (b) MLLNet-2 PA.
different degradations.
GB
WN
JPEG
JP2K
FF
ALL
Blar2
Nois:
B2+N20
(8)
0.97
1.00
0.89
0.83
0.84
0.89
DASR (hen)
23.28/6.7 74
22.23/7.05
21.32/7.55
(b)
0.91
IKC (b)
23.74/6.56
16,6077.22
16.19/6.87
DDR
0.97
1,00
0.98
MANet (b-n)
13,97/6.37
16.3276.61
16.83/7.29
DAN (b+n)
23,94/6.44
18.4678.20 20
17.76/8 04
RRDB (Clenn)
21.40/8.01
17,8088.2 29
17.2378.72 73
RRDB (b+n)
23.79/6.36
22.54/6.66
21.36/7.36
RRDB-DDR (bea)
24.01/6.34
22.52/6.60
21.4177277
on DIV2K-cleat validation dataset and calculate the CHI scores evaluate its discriminability with clean, blur and noise data. As shown in Fig. 8 as the training process goes, the performance of the model is improved, while the feature discriminability for degradation also enhanced. From random initialization to 700k iterations, the CHI score increases significantly from 0.00 to 591.68, while the PSNR value improves by 2.87dB (Due to GR, the initial PSNR value is relatively high). The training data only include clean LR images, but the trained model has the ability to discriminate unseen degradation types. This clearly implies that well-trained deep SR network naturally a good descriptor of degradation information.
4.5 FURTHER DISCUSSION ON THE CAUSES OF DDR PHENOMENON
In the previous sections, we reveal several important factors that promote the manifestation of DDR phenomenon, including global residual, adversarial learning (Sec. 4.3) and training iterations (Sec. 4.4). Based on the above findings and more visualization results, we can analyze the causes of DDR more deeply. We visualize the feature maps of SRResNet-wGR. SRResNettwoGR, SRGAN-wGR and SRGAN-woGR on test images with different degradations in the Appendix.
The DDR phenomenon is mainly introduced by overfitting the degradation in the training data. Specifically, since the training data (DleaK) do not contain extra degradations, the trained SR network lacks the ability to deal with the unseen degradations. When feeding images with degra- dations (e.g., noise and blur), it will produce features with unprocessed noises or blurring. These patterned features naturally show strong discriminability between different degradations. As for GR, models with GR produce features that contain less components of original content information. GR can help remove the redundant image content information and make the network concentrate more on deglad information. GAN training also enhances the high- tigh-frequency degrada- tion information. Besides, prolonging the training iterations and deepening the network depth will make the network further overfit to the training data.
4.6 WHY SR NETWORKS CAN HARDLY GENERALIZE TO UNSEEN DEGRADATIONS?
Classical SR models (Dong al. 2014 Lim et al. 2017) assume that the input LR images are generated by fixed downsampling kernel (e.g., bicubic). However, it is difficult to apply such simple SR models real scenarios with unknown degradations. We claim that SR and restoration networks learn to overfit the distribution of degradations, rather than the distribution of natural clean images.
To verify our statements, we compare the representations between SRGAN-wGR models trained on clean data and clean+noise data, respectively. As presented in Fig. 9 if the model is trained only on clean LR data, the deep representations show strong discriminability to clean and noise data. In contrast, if the model sees noise data during training, such discriminability diminishes. The model will become more robust to more degradation types, as the distributions of the deep representations become unanimous. In summary, to improve the model generalization for various degradations, we need to diminish the feature discriminability to degradations. Adding more degraded data into training is a plausible way to enhance the generalization.
8

---

Under review as a conference paper at ICLR 2023
EROAN
Degrided Image
Ostpet
Trained input (b) TTriined input
Figure 10: RRDBNet with DDR guidance. The degradation embedding is injected into the backbone network.
Figure 9: By training with more degraded data, the
deep representations become unanimous.
5 APPLICATIONS AND INSPIRATIONS
Image Distortion Identification Using DDR Features. Image distortion identification (Liang al., 2020) is an important subsidiary pretreatment for many image processing systems, especially for im- age quality assessment (IQA). It aims to recognize the distortion type from the distorted images, so as to facilitate the downstream tasks (Mittal et al.) [202aa] Gu et al. 2 Liang et al. (2020). Previ- ous methods usually resort to design handcrafted features that can distinguish different degradation types (Mittal et al., 2012aBa b) or train classification model via supervised learning (Kang et al.) 2014 Bosse et al.j 2017 Liang et al. 2020). Since DDR is related to image degradation, it can naturally be used as an excellent prior feature for image distortion identification. To obtain DDR, we do not need any degradation information but only a well-trained SR model (train on clean da- ta). Following BRISQUE (Mittal al., 2012a),
adopt the deep representations of SRGAN as input features (using PCA to reduce the original features to a 120-dimensional vector), and then use linear SVM to classify the degradation types of LIVE dataset (Sheikh al.) 2006). As shown in compared with BRISQUE and MLLNet (Liang et al. (020), DDR features achieve excel-
Tab. D lent results on recognizing different distortion types. More inspiringly, DDR is not obtained by any distortion-relatee supervision.
Blind SR with DDR Guidance. To super-resolve real images with unknown degradations, many blind SR methods resort to estimating and utilising the degradation information. For instance, IKC (Gu et 2019) iteratively corrects the estimated blur kernel, and DASR (Wang et al. 202T) implic- itly learns the degradation representations by contrastive learning. Based on the findings of DDR, we adopt a trained SRGAN model to extract degradation embedding to promote blind SR models. RRDBNet (Wang et al. 2018) is adopted as the backbone. The DDR embedding is injected into each RRDB module by the StyleMod Karras et al. (2020) (see Fig. [O]. The training data are described in Tab. 2 e.g., "b+n" means that the training data include blur and noise images. DDR guidance can help improve the model performance. Fig. [ reveals that DDR guidance can make the deep features become more homogeneous (CHI scores drop from 14.04 to 4.95).
2019)
RRDB CHI: 322.16
RRDB (blur+noise) CHI: 14.04
RRDB-DDR (blur+noise) CHI: 4.95
Figure 11: DDR clustering of different models. A lower CHI score connotes better generalization.
CONCLUSIONS
In this paper, we discover the deep degradation representations in deep SR networks, which are d- ifferent from high-level vision networks. We demonstrate that a well-trained deep SR network is naturally good descriptor of degradation information. We reveal the differences in deep represen- tations between classification and SR networks. We draw a series of interesting observations on the intrinsic features of deep SR networks, such the effects of global residual and adversarial learning. Further, we apply DDR to several fundamental tasks and achieve appealing results. The exploration on DDR is of great significance and inspiration for relevant work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, lan Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. arXiv preprint arXiv:1810.03292 2018.
Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR) Workshops, July 2017.
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- decoder architecture for image segmentation. IEEE transactions pattern analysis and machine intelligence, 39(12):2481-2499 2017.
Sefi Bell-Kligler, Assaf Shocher, and Michal Irani. Blind super-resolution kernel estimation using an nan. arXiv preprint arXiv: 06581, 2019.
Sebastian Bosse, Dominique Maniry, Klaus-Rober Müller, Thomas Wiegand, and Wojeiech Samek. Deep neural networks for no-reference and full-reference image quality assessment, IEEE Trans- actions on image processing, 27(1): 27(6)220--,19 2017.
Bolun Cai, Xiangmin Xu, Kui Jia, Chunmei Qing, and Dacheng Tao. Dehazenet: An end-to-end system for single image haze removal. IEEE Transactions Image Processing, 25(11):5187- 5198, 2016.
Tadeusz Calinski and Jerzy Harabasz. A dendrite method for cluster analysis. Communications in Statistics-theory and Methods, 3(1):1-27, 1974.
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 2017.
Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-d transform-doominn collaborative filtering. IEEE Transactions on image processing, 16 (8):2080-2095, 2007.
Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang. Second-order attention network for single image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11065-11074, 2019.
Pierre Demartines and Jeanny Hérault. Curvilinear component analysis: A self-organizing neural network for nonlinear mapping of data sets. IEEE Transactions neural networks, 8(1 8(1):148- 154, 1997.
Qili Deng, Ziling Huang, Chung-Ch Tsai, and Chia-Wen Lin. Hardgan: haze-aware representa- tion distillation gan for single image dehazing. In European Conference on Computer Vision, pp. 722-738. Springer, 2020a.
Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, and Xin Tong. Disentangled and controllable face image generation via 3d imitative-contrastive learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020b.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In Inter- national conference machine learning, pp. 647-655. PMLR, 2014.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning deep convolutional network for image super-resolution. In European conference on computer vision, pp. 184-199. Springer, 2014.
Chao Dong, Chen Change Loy, and Xiaoou Tang. Accelerating the super-resolution convolutional neural network. In European conference on computer vision, pp. 391-407. Springer, 2016.
Yu Dong, Yihao Liu, He Zhang, Shifeng Chen, and Yu Qiao. Fd-gan: Generative adversarial net- works with disiionddscriminator for single image dehazing. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 34, pp. 10729-107366 2020.
10

---

Under review as a conference paper at ICLR 2023
Jinjin Gu and Chao Dong. Interpreting super-resolution networks with local attribution maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Jinjin Gu, Hannan Lu, Wangmeng Zuo, and Chao Dong. Blind super-resolution with iterative k- emnel correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1604-1613, 2019.
Jiuxiang Gu, Zhenhua Wang, Jason Kuen, Lianyang Ma, Amir Shahroudy, Bing Shuai, Ting Liu, Xingxing Wang, Gang Wang. Jianfei Cai, et al. Recent advances in convolutional neural networks. Pattern Recognition, 77:354-377, 2018.
Jingwen He, Yihao Liu, Yu Qiao, and Chao Dong. Conditional sequential modulation for efficient global image retouching. In European Conference on Computer Vision, 679-695. Springer, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Geoffrey Hinton and Sam T Roweis. Stochastic neighbor embedding. In NIPS, volume 15, pp- 833-840. Citescer, 2002.
Harold Hotelling. Analysis of a complex statistical variables into principal components. Journal of educational psychology, 24(6))417, 1933.
Jie Hu, Shen, and Gang Sun. Squeezo-and-exciie -ncittcttniiion networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7132-7141, 2018.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4700-4708, 2017.
W Ronny Huang, Zeyad Emam, Micah Goldblum, Liam Fowl, Justin K Terry, Furong Huang, and Tom Goldstein. Understanding generalization through visualizations. 2020.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pp. 194-711. Springer, 2016.
Le Kang, Peng Ye, Yi Li, and David Doermann. Convolutional neural networks for no-reference image quality assessment. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1733-1740, 2014.
Andrej Karpathy, Justin Johnson, and Fei-Fei. Visualizing and understanding recurrent networks. arXiv preprini arXiv: 150.0000100000778 2015.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401-4411 2019.
Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 81 10-8119, 2020.
Robert Keys. Cubic convolution interpolation for digital image processing. JEEE transactions on acoustics, speech, and signal processing, 1966) 1981.
11

---

Under review as a conference paper at ICLR 2023
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1646-1654, 2016.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Ivan Laptev, Marcin Marszalek, Cordelia Schmid, and Benjamin Rozenfeld. Learning realistic human actions from movies. In 2008 IEEE Computer Society Conference Computer Vision and Pattern Recognition CVPR 2008, 2008.
Christian Ledig. Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro A- costa, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, al. Photo-realistic single image super-resolution using generative adversarial network. In Proceedings of the IEEE con- ference on computer vision and pultern recognition, pp. 4681 1681-4 2017.
Hsin- Ying Lee, Hung- Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Diverse image-to-image translation via disentangled representations. In Proceedings of the European conference on computer vision (ECCV), pp. 35-51, 2018.
Seungmin Lee, Dongwan Kim, Namil Kim, and Seong- Gyun Jeong. Drop adapt: Learning discriminative features for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 91-100, 2019.
Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. Visualizing and understanding neural models in nlp. arXiv preprint arXiv:1506.0100 2015.
Dong Liang, Xinbo Gao, Wen Lu, and Lihuo He. Deep multi-label learning for image distortion identification. Signal Processing, 172:107536, 2020.
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residu- al networks for single image super-resolution. In Proceedings of IEEE conference on computer vision and pattern recognition workshops, pp. 136-144, 2017.
Anran Liu, Yihao Liu, Jinjin Gu, Yu Qiao, and Chao Dong. Blind image super-resolution: A survey and beyond. arXiv preprint arXiv:2107.03005, 2021.
Yihao Liu, Hengyuan Zhao, Jinjin Gu, Yu Qiao, and Chao Dong. Evaluating the generalization ability of super-resolution networks. arXiv preprint arXiv:2205.07011 2022.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
Zhengxiong Luo, Y. Huang, Shang Li, Liang Wang, and T. Tan. Unfolding the alternating optimiza- tion for blind super resolution. ArXiv, abs/201 0.02631, 2020.
Liqian Ma, Qiannu Sun, Stamatios Georgoulis, Lue Van Gool, Bernt Schiele, and Mario Fritz. Dis- entangled person image generation. In P Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 99-108, 2018
Aravindh Mahendran and Andrea Vedaldi. Visualizing deep convolutional neural networks using natural pre-images. International Journal of Computer Vision, 120(3):233-255 2016.
Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. No-reference image quality assess- ment in the spatial domain. IEFE Transactions image processing, (1(12)4669-4700 4708, 201 2a.
12

---

Under review as a conference paper at ICLR 2023
Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. Making completely blind image quality analyzer. IEEE Signal processing letters, 20(3):00--11,, 2012b.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 ((77540):529-5555 2015.
Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Methods for interpreting and un- derstanding deep neural networks. Digital Signal Processing, 73:1-15, 2018.
Yotam Nitzan, Amit Bermano, Yangyan Li, and Daniel Cohen-Or. Face identity disentanglement via latent space mapping. ACM Transactions on Graphics (TOG), -9(6):1-14, 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03744. 2018.
Yuhui Quan, Mingqin Chen, Tongyao Pang. and Hui Ji. Self2self with dropout: Learning self- supervised denoising from single image. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pp. 1890-1898, 1898, 2020.
Mohammad Saeed Rad, Behzad Bozorgtabar, Urs- Viktor Marti, Max Basler, Hazim Kemal Ekenel, and Jean-Philippe Thiran. Srobb: Targeted perceptual loss for single image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2710-2719, 2019.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv preprint 106.01956 2015.
Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embed- ding. science, 290(5500):2323-2222) 2000.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 15(3):211-252) 2015. doi: 10. 1007/s 1263-015-0816-y.
Wojeiech Samek, Thomas Wiegand, and Klaus-Rohert Müller. Explainable artificial intelli- gence: Understanding, visualizing and interpreting deep learning models. arXiv preprint arX- 0v:1708.0829,, 2017.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Hamid R Sheikh, Muhammad F Sabir, and Alan C Bovik. A statistical evaluation recent full reference image quality assessment algorithms. IEEE Transactions image processing, 15(11): 3440-3451, 2006.
Yujun Shen and Bolei Zhou. Closed-form factorization of latent semantics in gans. arXiv preprint arXiv:2007.06600, 2020.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9243-9252, 2020a.
Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou. Interfacegan: Interpreting the disentan- gled face representation learned by gans. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020b.
13

---

Under review as a conference paper at ICLR 2023
Wenzhe Shi, Jose Caballero, Ferene Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1874-1883, 2016.
K. Simonyan and Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In International Conference on Learning Representations, May 2015.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional network- Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander Alemi. Inception-v4, inception- resnet and the impact of residual connections on learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Joshua Tenenbaum, Vin De Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. science, 290(55000:2319-2233,, 2000.
Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, Lei Zhang, Bee Lim, et al. Ntire 2017 challenge on single image super-resolution: Methods and results. In The IEEE Con- ference Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
Radu Timofte, Shuhang Gu, Jiqing Wu, Luc Van Gool, Lei Zhang, Ming-Hsuan Yang. Muhammad Haris, al. Ntire 2018 challenge on single image super-resolution: Methods and results. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2018.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: 10903, 2017.
Li Wang, Dong Li, Yousong Zhu, Lu Tian, and Yi Shan. Dual super-resolution learning for semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3774-3783 2020a.
Longguang Wang, Yingqian Wang, Xiaoyu Dong, Qingyu Xu, Jungang Yang, Wei An, and Yulan Guo. Unsupervised degradation representation learning for blind super-resolution. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10581 10590,, 2021.
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. European Confer- ence on Computer Vision Workshops, pp. 63-79. Springer, 2018.
Yang Wang, Yang Cao, Zheng-Jun Zha, Jing Zhang, and Zhiwei Xiong. Deep degradation prior for low-quality image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10049-11058, 2020b.
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In European conference on computer vision, pp. 499-515. Springer, 2016.
Jianchao Yang. John Wright, Thomas Huang, and Yi Ma. Image super-resolution sparse rep- resentation of raw image patches. 2008 IEEE conference on computer vision and pattern recognition, pp. 1-8. IEFF, 2008.
14

---

Under review as a conference paper at ICLR 2023
Yuan Yuan, Siyuan Liu, Jiawei Zhang, Yongbing Zhang, Chao Dong, and Liang Lin. Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks. In CVPRW, pp- 701-710, 2018.
Tom Zahavy, Nir Ben- Zei-Zri and Shie Mannor. Graying the black box: Understanding dqns. In International Conference on Machine Learning, pp. 1899-1908. PMLR, 2016.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 318-833. Springer, 2014.
Matthew D Zeiler, Dilip Krishnan, Graham W Taylor, and Rob Fergus. Deconvolutional networks. In 2010 IEEE Computer Society Conference on computer vision and pattern recognition, pp- 2528 2535. IEEE, 2010.
He Zhang and Vishal M Patel. Densely connected pyramid dehazing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2018.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond gaussian denoiser: Residual learning of deep enn for image denoising. IEEE transactions on image processing, 26 (7):3142-3155, 2017.
Kai Zhang, Wangmeng Zuo, and Lei Zhang. Ffdnet: Toward fast and flexible solution for cnn- based image denoising. IEEE Transactions on Image Processing, 27(9):4008 4622, 201 8a.
Wenlong Zhang, Yihao Liu, Chao Dong, and Yu Qiao. Ranksrgan: Generative adversarial networks with ranker for image super-resolution. In Proceedings of the IEEF/CVF International Confer- ence on Computer Vision, pp. 3096-3105, 2019.
Yu Zhang, Peter Tiño, Aleš Leonardis, and Ke Tang. A survey neural network interpretability. arXiv preprint arXiv:2012.14000 2020.
Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention networks. Proceedings of the European conference on computer vision (ECCV), pp. 286-301, 2018b.
Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2472-2481, 2018c.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2921-2929, 2016.
Bo Zhu, Jeremiah Z Liu, Stephen F Cauley, Bruce R Rosen, and Matthew S Rosen. Image recon- struction by domain twwaain-ntrnnormmmmm manifold learning. Nature, 555(7697):487 2018.
A APPENDIX
A.1 BACKGROUND
Since the emergence of deep convolutional neural network (CNN), a large number of computer
vision tasks have been drastically promoted, including high-level vision tasks such as image clas-
sification Russakovsky et al. (015)); Simonyan & Zisserman (2015); He et al. (2016);
et al.
(017): Hu al. (2018). object localization Ren et al. (2015); He al. (201 p): Redmon et al. (2016) and semantic segmentation Long et al.] (2015); |Badrinarayanan et al. (2017); Chen et al. (201 7)
Wang et al. (2020a), as well as low-level vision tasks such as image super-resolution Dong et al. 2014)); Ledig et al. (2017); Wang al. (2018): Zhang et al. (2019); Dai et al. (2019). denoising Zhang et al. (2017: 2018a); Gu et al. 20 Quan et al. (2020), dehazing Cai et al. (2016); Zhang & Patel (2018); Dong et al. (2020); Deng al. (020a)) etc. However, interesting phenomenon is that even if we have successfully applied CNNs to many tasks, yet we still do not have a thorough understanding of its intrinsic working mechanism.
15

---

Under review as a conference paper at ICLR 2023
To better understand the behaviors of CNN, many efforts have been put in the neural network inter-
pretability for high-level vision Simonyan&e et al. SSamek et al. (0)1);; Zeiler & Fergus (2014))
Selvaraju et al. (2017), Montavon et al. (2018)) Karpathy et at. (2015) Mahendran & Vedaldi 2016);
Zhang et al. 2200)): Adebayo et al. (2018). Most of them attempt to interpret the CNN decisions
by visualization techniques, such as visualizing the intermediate feature maps (or saliency maps and class activation maps) Simonyan et al. (2013); Zeiler & Fergus (2014); Adebayo et al. (2018) 8); Zhou ct al. (2016); /elvaraju al. (2017), computing the class notion images which maximize the class score Simonyan et al. (2013), or projecting feature representations Wen et al. (2016); Wang et al. (2020b): Zhu al. (2018) 8); Huang et al. (2020). For high-level vision tasks, especially image classi- fication, researchers have established set of techniques for interpreting deep models and have built up a preliminary understanding of CNN behaviors Gu et al. (2018). One representative work is done by Zeiler et al. [Zeiler & Fergus (2014), who reveal the hierarchical nature of CNN by visualizing and interpreting the feature maps: the shallow layers respond to low-level features such as corners, curves and other edge/color conjunctions; the middle layers capture more complex texture combi- nations; the deeper layers are learned to encode more abstract and class-specific patterns, e.g., faces and legs. These patterns can be well interpreted by human perception and help partially explain the CNN decisions for high-level vision tasks.
As for low-level vision tasks, however, similar research work is absent. The possible reasons are as follows. In high-level vision tasks, there are usually artificially predefined semantic label- s/categories. Thus, we can intuitively associate feature representations with these labels. Neverthe- less, in low-level vision tasks, there is no explicit predefined semantics, making it hard to map the representations into domain that the human can make sense of. Further, high-level vision usually performs classification in discrete target domain with distinct categories, while low-level vision aims to solve a regression problem with continuous output values. Hence, without the guidance of predefined category semantics, it seems not so straightforward to interpret low-level vision networks.
In this paper, we take super-resolution (SR), one of the most representative tasks in low-level vision, as research object. Previously, it is generally thought that the features extracted from the SR network have no specific "semantic" information, and the network simply learns some complex non-linear functions to model the relations between network input and output. Are CNN features SR networks really in lack of any semantics? Can we find any kind of "semantics" in SR networks? In this paper, aim to give an answer to these questions. We reveal that there are semantics existing in SR networks. We first discover and interpret the "semantics" of deep representations in SR networks. But different from high-level vision networks, such semantics relate to the image degradation types and degrees. Accordingly, we designate the deep semantic representations in SR networks as deep degradation representations (DDR).
A.2 LIMITATIONS
In this paper, we only explore the deep representations of SR networks. Other low-level vision networks are also worth exploring. We apply DDR to three tasks without too elaborate design in the application parts. For blind SR, we make a simple attempt to improve the model performance. The design is not optimal. We believe that there should be more efficient and effective way to utilize DDR. For generalization evaluation, DDR can only evaluate the model generalization under constrained conditions. It shows the possibility of designing a generalization evaluation metric, but there is still a long way to realize this goal.
A.3 DEEP REPRESENTATIONS OF REAL- IMAGES
In the main paper, we mainly conduct experiments on synthetic degradations. The difficulty of real- world dataset that is hard to keep the content the same but change the degradations. we simply use two real-world datasets which contains different contents and different degradations, it is hard to say whether the feature discriminability is targeted at image content or at image degradation. Hence, synthetic data at least can control the variables.
In addition, we find a plausible real- worldworld dataset Real-City City,,,yy which is proposed in paper Cameral SR. The authors use iPhoneX and NikonD5500 devices to capture controllable images. By adjusting the cameral focal length. each camera captures paired images with the same content but different resolutions. The low-resolution images contain real-world degradations such real noise and real
16

---

Under review as a conference paper at ICLR 2023
iphoneX-HQ
iphoneX-LQ
NikonD5500-HO
NikonD5500-LQ
Figure 12: Projected feature representations of SRGAN-wGR on Real-City 100 dataset.
blur. We test SRGAN on this dataset and obtain corresponding visualization results, as shown in 12 It can be seen that the deep representations of SRGAN can still distinguish among different degradations across different devices.
A.4 CLASSIFICATION vs. SUPER- -ONRRRR
A.4.1 FORMULATION
Classification. Classification aims to categorize input image X into discrete object class:
GCi((,),
where GCL represents the classification network, and y R is the predicted probability vector indicating which of the C' categories X belongs to. In practice, cross-entropy loss is usually adopted train the classification network:
CE(YY)) = E vlogg,
where Y E R is a one-hot vector representing the ground-truth class label. ÿi is the i-th row element of Y, indicating the predicted probability that X belongs to the i-th class.
Super-resolution. A general image degradation process can be model as follows:
(3) where Y is the high-resolution (HR) image and & denotes the convolution operation. X is the degraded high-resolution (LR) image. There are three types of degradation in this model: blur kernel k, downsampling operation is and additive noise n. Hence, suser-resolution can be regarded as superset of other restoration tasks like denoising and deblurring.
X= = (Y & k) 4. +n,
Super-resolution (SR) is the inverse problem of Equ. 3.. Given the input LR image X E RMM, the super-resolution network attempts to produce its HR version:
GsR(X),
(4)
where GSR represents the super-resolution network, Ý e IRM the predicted HR image and is the upscaling factor. This procedure can be regarded as typical regression task. At present, there are two groups of method: MSE-based and GAN-based methods. The former one treats SR as reconstruction problem, which utilizes pixel-wise loss such as Log loss to achieve high PSNR values.
N rM
L2
1
ΣΣ
Yr Y.o123
(5)
17

---

Under review as a conference paper at ICLR 2023
This is the most widely used loss function in many image restoration tasks Dong et al. (2014) Lim et al. (2017); Zhang et al. (2018b a); Cai et al. (2016); He et al. (2020). However, such loss tends to produce over-smoothed images. To generate photo-realiste SR results, the latter method incorpo-
rates adversarial learning and perceptual loss to benefit better visual perception. The optimization is expressed as following min-max problem:
min maxBy upan llog Ds DesR PDER
(6)
Ex~peR [og(1 Dsn(Gsn(X)))
In such adversarial learning, discriminator DSR is introduced to distinguish super-resolved images from real HR images. Then, the generator loss is defined as:
Lg = log DsR(GGR(X)))
From the formulation, we can clearly see that image classification and image super-resolution rep- resent two typical tasks in machine learning: classification and regression. The output of the classi- fication task is discrete, while the output of the regression task continuous.
A.4.2 ARCHITECTURES
Due to the different output types, the CNN architectures of classification and super-resolution net- works also differ. Generally, classification networks often contain multiple downsampling layers (e.g., pooling and strided convolution) to gradually reduce the spatial resolution of feature maps. After several convolutional and downsampling layers, there may be one or more fully-connected layers aggregate global semantic information and generate vector containing C' elements. For the output layer, the SoftMax operator is frequently used to normalize the previously obtained vec- tor into probabilistic representation. Some renowned classification network structures include AlexNet Krizhevsky et al. (2012). VGG SSimonyan & Zisserman (2015). ResNet He et al. 2016)) De
al. (2013); loffe Szegedy (2015): Szegedy et al. (2017). DenseNet Huang
et al. 2017), SENe Badrinarayanan et al. (2017), etc.
Unlike classification networks, super-resolution networks usually do not rely on downsampling lay- ers, but upsampling layers (e.g., bilinear upsampling, transposed convolution /Ziler et al. (2010) or subpixel convolution (hi et al. (2016)) Thus, the spatial resolution feature maps would in- crease. Another difference is that the output of the SR network is three-channel image, rather than an abstract probability vector. The well-known SR network structures include SRCNN Dong et al. 2014), FSRCNN Dong et al.] (2016). SRResNet Ledig al. (2017), RDN Zhang et al. (2018c), RCAN Zhang et al. (2018b). etc. An intuitive comparison of classification and SR networks in CNN architecture shown in Fig. 18 We notice that is gradually downsampling, and the other is gradually upsampling, which displays the discrepancy between high-level vision and low-level vision tasks in structure designing.
Although there are several important architectural differences, classification networks and SR net- works can share and adopt some proven effective building modules, like skip connection He et al. 2016
Lim et al. (2017) and attention mechanism et al. (2018): Zhang et al. (2018b).
A.5
IMPLEMENTATION DETAILS
In the main paper, we conduct experiments on ResNet18 He et al. (2016) and SRResNet/SRGAN Ledig et al. (2017). We elaborate more details on the network structures and training settings here.
For ResNet18. 8. we directly adopt the network structure depicted in He et al. (2016). Cross-entropy
loss (Eq.) 2 is used as the loss function. The learning rate is initialized to 0.1 and decreased with
cosine annealing strategy. We apply SGD optimizer with weight decay x 10-4 1, The trained model
yields accuracy of 92.86% on CIFARIO testing set which consists of 10, 000 images.
For SRRE/NEEWWGR/RRRENNEEWG..RG we stack residual blocks (RB) as shown in Fig. of the main paper. The residual block is the same as depicted in Wang et al. (2018), in which all the BN layers are removed. Two Pixel-shuffle layers Shi et al. (2016) are utilized to conduct upsam- pling in the network, while the global residual branch upsampled by bilinear interpolation. Lı loss is adopted as the loss function. The learning rate is initialized to x 10 10-4 and is halved at [100k; 300k, 500k, 600k iterations. A total of 600, 000 iterations executed.
18

---

Under review as a conference paper at ICLR 2023
For BEGGAN-WGR/RGAAN The discriminator is designed as in Ledig et al. (2017). Adversarial loss (Eq. 7 and perceptual loss Johnson et al. (2016) combined as the loss functions, which are kept the same as in Ledig et al. (2177) The learning rate of both generator and discriminator is initialized to 1 x 10-4 and is halved [50k; 100k, 200k, 300k] iterations. A total of 600, 000 iterations are executed. For all the super- resolution networks, we apply Adam optimizer Kingma & Ba (2014) with 31 = 0.9 and 32 = 0.99. All the training LR patches are of size 128 128. When testing, 32 32 patches are fed into the networks to obtain deep features. In practice, we find that the patch size has little effect on revealing the deep degradation representations. All above models are trained on PyTorch platform with Geforce RTX 2080 Ti GPUs.
the generator is the same SRRReNNer
For the experiment of distortion identification, we use the aforementioned trained models to conduct inferencing on the LIVE dataset (Sheikh et al. (2006). We crop the central 96 x 96 patch of each image feed into the SR networks and obtain the corresponding deep representations. Then, the deep representations of each image are reduced to 120-dimensional vector using PCA. Afterwards, the linear SVM is adopted as the classification tail. In practice, we find that the vector dimension can be even larger for better performance. Notably, unlike previous methods, the features here are not trained on any degradation related labels or signals. The SR networks are only trained using clean data. However, the deep representations can be excellent prior features for recognizing various distortion types. This is of great importance and very encouraging.
A.6 DEFINITIONS OF WD, BD AND CHI
In Sec. 3.1 of the main paper. we describe the adopted analysis method on deep feature representa- tions. Many other literatures also have adopted similar approaches to interpret and visualize the deep models, such as Graph Attention Network VVllčković al. (2017), Recurrent Networks Karpathy et al. (2015), Deep Q-Network Zahavy et al. (2016) and Neural Models in NLP Li et al. (2015)- Most aforementioned researches adopt E-SN as a qualitative analysis technique. To better illustrate and quantitatively measure the semantic discriminability of deep feature representations, we take a step further and introduce several indicators, which are originally used to evaluate the clustering performance, according the data structure after dimensionality reduction by t-SNE. Specifically, we propose to adopt within-cluster dispersion (WD), aetween-clustedd dispersion (BD) and Calinski- Harabaz Index (CHI) Caliñski & Harabasz (1974) to provide some rough yet practicable quantitative measures for reference. For K clusters, WD, BD and CHI are defined as:
K ((k
WD(K) = ΣΣ
3g
(8)
where σ represents the i-th datapoint belonging to class & and T is the average mean of all rı(k) datapoints that belong to class Datapoints belonging to the same class should be close enough to each other and WD measures the compactness within cluster.
K
BD(K) Σ [<) - =1
(9)
1=1
where E represents the average mean of all datapoints. BD measures the distance between clusters. Intuitively, larger BD value indicates stronger discriminability between different feature clusters. Given K clusters and N datapoints in total (N = Ex n(k)), by combining WD and BD, the CHI is formulated as:
CHI(K) BD(K) (N-K)
(10)
WD(K) (K 1))
It is represented the ratio of the between-clusters dispersion mean and the within-cluster disper- sion. The CHI score is higher when clusters are dense and well separated, which relates to a standard concept of cluster.
Rationality of Using Quantitative Measures with t-SNE. Notably, t-SNE is not a numerical tech- nique but probabilistic one. It minimizes the Kullback-Leihhe (KI.) divergence between the dis-
19

---

Under review as a conference paper at ICLR 2023
6.0
d5g
Rrgg
Brack
"Convl"
"Conv2 4"
"Conv3 4"
"Conv4 4"
"Cnvv 4"
Figure 13: Projected feature representations extracted from different layers of ResNet18 using t- SNE. With the network deepens, the representations become more discriminative to object cate- gories, which clearly shows the semantics of the representations in classification.
#Layer
Convl
Conv2,4
Conv3.4
Conv4.4
Conv5,4
Dim
64x32xx3
64x32xx3
16xx16x
x56xxxx8
512x4x4
WD | (x10")
4.07 +: 0.43
3.41 4: 0.31
3.32 1: 0.31
2.06 d: 0.13
0.71 0.06
BD 7 (x10")
1.04 0.13
1.22 1: 0.11
1.84 : 0.40
5.77 3: 0.23
10.74 = 0.20
1.
CHI
28.18 + 1.69
39.22 + 1.44
61.12 1 13.62
309.31 = 31.10
145.15
Table 3: Quantitative measures for the discrininability of the projected deep feature representation- s. We statistically report the mean value and the standard deviation of each metric. The adopted indicators well reflect the effect of feature clustering quantitatively.
Set5
Set14
Urban 100
DIV2K
SRCNN-3L
28.51
25.72
22.86
27.80
SRCNN-3L
28.89
23.99
23.22
28.05
SRCNN-7L
28.97
26.02
SRCNN-9L
29.17
26.17
23.48
28.24
SRCNN IIL.
29.27
26.21
23.56
28.29
SRCNN-I3L
29.39
Table 4: The PSNR values of SRCNN with different depth on classical SR benchmark datasets.
TEOCK
(a) layers 87622.61
(b) 5 layers WD: 49458.40
(c) 9 layers WD: 41893.05
(d) 13 layers 20221.11119
BD: 0.73
BD: 13964.10
BD: 28574.68
BD: 33070.92
CHI:
CHI: 41.93
CHI: 101.29
CHI: 168.12
Figure 14: With more layers, the model deep representations gradually manifest the discriminability on degradation types.
20

---

Under review as a conference paper at ICLR 2023
"Cuny
"ResBlockd"
"ResBlock8"
"ResBlock6""
CIll:
(c) CHIl: 1 0.03
DMG cean cear Geaseesse
(c) CHI: 0.00 0.00
(1) CHI: 0.11 0.00
(g) CHI: 38.21 + 9.25
CHI: 613.77 + 33. 40
Figure 15: Projected feature representations extracted from different layers of SRResNet- (Ist row) and SRResNet-wGH (2nd row) using t-SNE. With image global residual (GR), the representa- tions of MSE-based SR networks show discriminability to degradation types.
tributions that measure pairwise similarities the input high-dimensional data and that of the cor- responding low-dimensional points in the embedding. Further, t-SNE is non-convex optimization process which is performed using a gradient descent method, as a result of which several optimiza- tion parameters need to be chosen, like perplexity, iterations and learning rate. Hence, the recon- struction solutions may differ due to the choice of different optimization parameters and the initial random states. In this paper, Moreover, we conduct extensive experiments using different parameters and demonstrate that the quality of the optima does not vary much from run run, which also emphasized in the t-SNE paper. To make the quantitative analysis more statistically solid, for each projection process, we run t-SNE five times and report the average and standard deviations of every metric.
used exactly the same optimization procedure for all experiments.
A.7 FROM SHALLOW DEEP SR NETWORKS
In the main paper, reveal that shallow 3-layer SRCNN Dong al. (2014) does not manifest representational discriminability on degradation types. Thus, we hypothesize that only deep SR net- works possess such Degradation-relate semanties. To verify the statement, we gradually deepen the depth of SRCNN and observe how its deep representations change. We construct SRČNN models with different layer depths from shallow 3 layers to 13 layers. We train these models on DIV2K- clean data (inputs are only downsampled without other degradations) and test them on classical SR benchmarks. As shown in Tab. 4 the model achieves better SR performance with the increase of network depth, suggesting that deeper networks and more parameters can lead to greater learning capacity. On the other hand, the deep representations also gradually manifest discriminability on degradation types, as depicted in Fig. 14 When the model only has 3 layers, its representations cannot distinguish different degradation types. However, when we increase the depth 13 layers, the deep representations begin to show discriminability on degradation types, with the CHI score increasing to 128
Maayee
Conv
ResBlock4
ResBlock8
ResBlack16
WDJ(X10 (1*)
8.35 0.14
8,90
0.22
9.28
0.31
4.98 0.48
BDT
0.29 1 0.14
1.98 1 1.47
25.60 1. 17.73
1149.20 29.20 1 765. 12
CHI
SRResNet
#Layer
Convl
ResBlock4
ResBlock8
ResBlack16
WDJ(X 1
8.20 1. 0.18
8.40 1. 0.09
4.40 1. 0.50
0.86 0.11
BDT
0.48
62.74
33.99
11096. 2051.02
35470.66
CHT
0.00 1 0.00
0.11 1 0.06
38.21 1 9.25
613.77 1 33.40
Table 5: Quantitative measures for the projected deep feature representations obtained by SRResNet- woGR and NeRe
A.8 MORE APPLICATIONS
Evaluating the Generalization Ability. According the discussions in Sec. 4.6 DDR can be used as an approximate evaluation metric for generalization ability. Specifically, given a trained model and several test datasets with different degradations, obtain their DDR features. By
21

---

Under review as a conference paper at ICLR 2023
"Cuny
"ResBlockd"
"ResBlockk"
"ResBlock
CIIl: 0.01 d: 0.00
(c) CHII 22.00
234.43
DMG cean cear Gease
(c) CHI: 0.00 0.00
(f) CHI: 0.03 0.02
(g) CHI: 35.68 + 2.52
(h) CHI: 626.46 + 31.56
Figure 16: Projected feature representations extracted from different layers of SRGAN- (Ist row) and SRGAN-wGR (2nd row) using t-SNE. Even without GR, GAN-based SR networks can still obtain deep degradation representations.
SRGAN-WOUR
PLayer
Conyl
ResBlock4
ResBlockS
ResBlock16
WDL(x10")
7.91 = 0.20
7.83 + 0.33
4.65 + 0.58
1.44 + 0.28
BDT
0.58 I 0.41
4.79 1 2.43
9809,00 1. 4501.
22459 35 3360.33
CHIT
0.00 = 0.00
0.01 0.00
34.00 + 22.00
231, 13 + 30.34
SRGAN WGR
PLayer
ConvI
ResBlock4
ResBlock8
ResBlockT6
WDL1X10')
7.47 0.20
7.97 0.19
4.83 +: 0.52
0.72 1: 0.10
BDT
0.41 = 0:30
11.89 : 8.85
11600.91 1021.10
30180 288-1.65
CmT
0.00 1 0.00
0.03 1. 0.02
35.68 2.52
626.46 1. 31.56
Table 6: Quantitative measures for the projected deep feature representations obtained by SRGAN- woGR and SRGAN-wGR.
evaluating the discriminability of the projection results (clustering effect), we can roughly measure the generalization performance over different degradation types. The worse the clustering effect, the better the generalizability. Fig [T shows the DDR clustering of different models. RRDB (clean) is unable to deal with degraded data and obtains lower PSNR values on blur and noise inputs. Its CHI score is 322. 16. By introducing degraded data into training, the model gains better generalization and the CHI score is 14.04. With DDR guidance, the generalization ability is further enhanced. The CHI score decreases to 4.95. The results are consistent with the results in the previous section. Interestingly, we do not need ground-trui images to evaluate the model generalization. A similar attempt has been made in recent work Liu et al. (2022). Note that CHI is only rough index, which cannot accurately measure the minor differences. DDR shows the possibility of designing a generalization evaluation metric, but there is still long way to realize this goal.
A.9 EXPLORRATION ON DIFFERENT DEGRADATION DEGREES
Even for the same type of degradation, different degradation degrees will also cause differences in features. The greater the difference between degradation degrees, the stronger the discriminability. First row: SRResNet-wGiR, Second row: SRGAN-wGR.
Figure 17:
Previously, introduce deep degradation representations by showing that the deep representations of SR networks are discriminative to different degradation types (c.g., clean, blur and noise). How about the same degradation type but with different degraded degrees? Will the deep representa-
22

---

Under review as a conference paper at ICLR 2023
Closs-dogrndaton
Inraadepradiatoon (degrndation degrees)
structure
Clean-Blur-Nois
(3,0,33
Noise[T0,330
Noise[10,20) 20)
Blur 1.4]
SRResNet
(3.55)
(6.29)
(7.84)
(0.23)
(0.02)
GR
+4+ (613.77)
-(36,53)
+ +41.500
-0.59)
(53.37)
MSRGAN
WOGR
++ (234.43)
++ (551.26)
+++ (525.55)
+ (52.67)
(1.40)
(815.11)
-(99.40)
(35.04)
0 20.
20 100. 100 500. +++ 500.
Table 7: Quantitative evaluations (CHI). There appears to be spectrum (continuous transition) for the discriminability of DDR.
tions still be discriminative to them? To explore this question, more experiments and analysis are performed.
We test super-resolution networks on degraded images with different noise degrees and blur de- grees. The results are depicted in Table. 7 and Fig. 17 It can be seen that the deep degradation representations are discriminative not only to cross-degradation (different degradation types) but also to intra-degradation (same degradation type but with different degrees). This suggests that even for the same type of degradation, different degradation degrees will also cause significant d- ifferences features. The greater the difference between degradation degrees, stronger discriminability of feature representations. This also reflects another difference between the repre- sentation semantics of super-resolution network and classification network. For classification, the semantic discriminability of feature representations is generally discrete, because the semantics associated with discrete object categories. Nevertheless, there appears to be a spectrum (continuous transition) for the discriminability of the deep degradation representations, i.e., the discriminability has monotonic relationship with the divergence between degradation types and degrees. For ex- ample, the degradation difference between noise levels 10 and 20 is not that much distinct, and the discriminability of feature representations is relatively smaller, comparing with noise levels 10 and
30.
From T there are notable observations. 1) Comparing with blur degradation, noise degradation easier to be discriminated. Yet, it is difficult to obtain deep representations that have strong discriminability for different blur levels. Even for GAN-based method, global residual (GR) is indispensable to obtain representations that can discriminative to different blur levels. 2) The representations obtained by GAN-based method have more discriminative semantics to degradation types and degrees than those MSE- based method. 3) Again, global residual can strengthen the representation discriminability for degradations.
Closstradee
Figure 18:
Unified backbone framework for classification and super-resolution. The two networks share the same backbone structure and different tails.
A. EXPLORATION OF NETWORK STRUCTURE
In the main paper, choose R8 He et al. (016)) and SRResNer/SRGAN Ledig et al. (2017) as the backbones of classification and SR networks, respectively. In order to eliminate the influence of different network structures, design unified backbone framework, which is composed of the
23

---

Under review as a conference paper at ICLR 2023
3
Bruck
"Conel"
"Down
"Down2"
"Down3"
"Downd"
Figure 19: Projected feature representations extracted from different layers of unified backbone framework (classification) using t-SNE. The results are similar to ResNet18, which validates that the deep semantic representations are uncorrelated with network structures but associated with the task itself.
#Layer
Conv1
Down
DownZ
Down3
Down4
x2x32x33
x2x32x32
16
512x4x4
WD I (x10")
3.64 + 0.33
2.76 0.27
2.52 0.19
+ 0.05
0.59
BD (x10")
1.10 + 0,13
0.97 # 0.18
1.60 + 0.19
3.81
0.40
7.48 0.32
CIT
33.11 = 1.38
39.53 I 9.98
70.11 9.34
230.95
22.63
1403.96 1 27.17
Table 8: Quantitative measures for the discriminability of the projected deep feature representations obtained by unified backbone framework (classification).
DIV2K-clean
DIV2K-blur
DIViK-noise
Figure 20: Projected feature representations extracted from unified backbone framework (super- resolution) using t-SNE.
24

---

Under review as a conference paper at ICLR 2023
same basic building modules but connected with different tails for downsampling and upsampling to conduct classification and super-resolution respectively.
The unified architecture is shown in Fig. 18 To differ from the residual block in the main paper, we adopt residual channel attention layer as basic building block, which is inspired by SENet Hu et al. (2018) and RCAN Zhang et al. (2018b). For classification, the network tail consists of three maxpooling layers and a fully connected layer; for super-resolution, the network tail consists two pixel-shuffle layers upsample the feature maps. According to the conclusions in the main paper, we adopt global residual (GR) in the network design to obtain deep degradation representations (DDR). Except the network structure, all the training protocols are kept the same as in the main paper. The training details are the same as depicted in Sec. AA.5 After training, the unified backbone framework for classification yields an accuracy of 92.08% on CIFARIO testing set.
The experimental results are shown in Fig. 19 Fig. 20 and Tab. 8 From the results, we can see that the observations are consistent with the findings the main paper. It suggests that the semantic representations do not stem from network structures, but from the task itself. Hence, our findings are not only limited to specific structures but are universal.
A.11 MORE INSPIRATIONS AND FUTURE WORK
Disentanglement of Image Content and Degradation In plenty of image editing and synthesizing tasks, researchers seek to disentangle image through different attributes, so that the image can be finely edited Karras et al. (2019;; Ma et al. (0188; Deng et al. (020bb; Lee et al. 2018); Nitzan
et al. (2020). For example, semantic face editing Shen et al. [2020a b Shen & Zhou (2020) aims at manipulating facial attributes a given image, e.g., pose, gender, age, smile, etc. Most methods attempt learn disentangled representations and to control the facial attributes by manipulating the latent space. In low-level vision, the deep degradation representations can make it possible to decompose an image into content and degradation information, which can promote a number of new areas, such as degradation transferring and degradation editing. Further, more in-depth research on deep degradation representations will also greatly improve our understanding of the nature of images.
A.12 DISCUSSIONS ON DIMENSIONALITY REDUCTION
Among the numerous dimensionality reduction techniques (e.g.. PCA Hotelling (933 CCA Demartines & Hérault (1997). LLE Roweis & Saul (2000). Isomag]Tenenhar
al. (2000) SNFHinton & Roweis 2.02)) t-Distributed Stochastic Neighbor Embedding (ISNN)) der Maat- en & Hinton| (2008) is a widely-used and effective algorithm. It can greatly capture the local struc- ture of the high-dimensional data and simultaneously reveal global structure such as the presence of clusters at several scales. Following Donahue et al.; (2014); Mnih et al. (2015): Wen et al. (2016); Zahavy et al. (2016); Veličković al. (2017): Wang et al. (2020b); Huang et al. (2020), we also take advantage of the superior manifold learning capability of S-NN for feature projection.
In this section we further explain the effectiveness of adopting t-SNE and why we choose project hign-dimensional features into two-dimensional datapoints. We first compare the projection results of PCA and t-SNE. From the results shown in Fig. 21 it can be observed that the projected features by t-SNE are successfully clustered together according the semantic labels, while the projected fea- tures by PCA are not well separated. It because that PCA is a linear dimension reduction method which cannot deal with complex non-linear data obtained by the neural networks. Thus, t-SNE is a better choice to conduct dimension reduction on CNN features. This suggests the effectiveness of t-SNE for the purpose of feature projection. Note that we do not claim t-SNE is the optimal or the best choice for dimensionality reduction. We just utilize t-SNE as a rational tool to show the trend behind deep representations, since t-SNE has been proven effective and practical in our experiments and other literatures.
Then, we discuss the dimensions to reduce. We conduct dimensionality reduction to different di- mensions. Since the highest dimension supported by t-SNE is 3, we first compare the effect between the two-dimensional projected features and the three-dimennt dimeessonnii projected features by t-SNE. The qualitative and quantitative results are shown in Fig. 21 and Tab. 9 When we reduce the features to three dimensions, the reduced representations also show discriminability to semantic labels. How-
25

---

Under review as a conference paper at ICLR 2023
(0) SSNE (2d)
(b) PCA (2d)
(c) t-SNE (30)
(d) PCA (3d)
Figure 21: Comparison between PCA and t-SNE for projecting feature representations ("Conv5_47 _4" layer of ResNet18).
ever, quantitative results show that two dimensions can better portray the discriminability than three or higher dimensions. For PCA, the results are similar. With higher dimensions, the discriminabilicy decrease. Hence, it is reasonable reduce high-dimensional features into two-dimensional data- points. Such settings are also adopted in Donahue et al. (2014); Wang et al. (2020b); Veličkovid
al.
2017); Huang al. 2020 which are proven effective.
PLayer
Conv5
Input #Dim
512 X TX
Method
FCACDHAESNEC)
FCA(50)E-SNEC)))
PCA
PCA
PCA
PCA
Redured #Dim
3
WD L (x10")
0.71 0.06
0.24 1 0.06
0.19
0.32
0.39
0.47
(x10")
10.74 + 0.20
2.09
0.04
1,27
1.61
2.24
CHIT
1688.62 4 145.15
978.58 = 224.77
729.64
36283
55492
326.64
Table 9: Quantitative comparison with dimensionality reduction methods and reduced dimensions. To utilize t-SNE, we first use PCA to pre-reduce the features to 50 dimensions. Since PCA is numerical method, the result is fixed. For t-SNE, we report the mean and standard deviation for runs. The quantitative results show that t-SNE surpasses PCA and reducing to two dimensions better. The features are obtained by "Conv5_4" layer of ResNet18.
A. VISUALIZATION OF FEATURE MAPS
So far, have successfully revealed the degradation-related semantics in SR networks with di- mensionality reduction. In this section, we directly visualize the deep feature maps extracted from SR networks provide some intuitive and qualitative interpretations. Specifically, we extract the feature maps obtained from four models (SRResNet-wGR, SRResNet-woGR, SRGAN-wGR and SRGAN-woGR) on images with different degradations (clean, blur4, noise20), respectively. Then we treat each feature map as a one channel image and plot it. The visualized feature maps are shown Fig. 22 We select feature maps with the largest eigenvalues for display. The complete results are shown in the supplementary file.
Influence of degradations on feature maps. From Fig. (). we can observe that the deep features obtained by SRResNet-woGR portray various characteristics of the input image, including edges, textures and contents. In particular, we highlight in "red rectangles" the features that retain most of the image content. As shown in Fig. 22 b). after applying blur and noise degradations to the input image, the extracted features appear similar degradations as well. For blurred/noisy input images, the extracted feature maps also contain homologous blur/noise degradations.
Effect of global residual. Sec. 4.3 we have revealed the importance and effectiveness of global residual (GR) for obtaining deep degradation representations for SR networks. But why GR is so
26

---

Under review as a conference paper at ICLR 2023
a SRRCshNt-woGlt
cleam
Hur
noise
(b) WeR
har
O SRGAN
clean
hir
DOUse
SRGAN-GR
clesm
hur
noise
Figure 22: Visualization of feature maps. GR and GAN can facilitate the network obtain more features on degradation information.
27

---

Under review as a conference paper at ICLR 2023
important? What is the role of GR? Through visualization, we provide qualitative and intuitive explanation here. Comparing Fig. 22 a) and Fig. b2)b), can be observed that by adopting GR, the extracted features seem to contain less components of original shape and content information. Thus, GR can help remove the redundant image content information and make the network concentrate more obtaining features that related to low-level degradation information.
Effect of GAN. Previously, we have discussed the difference between MSE-based and GAN-based SR methods in their deep representations. We find that GAN-based method can better obtain feature representations that are discriminative to different degradation types. As shown in Fig. 2 a) and Fig. c2) the feature maps extracted by GAN-based method contain less object shape and content information compared with MSE-base method. This partially explains why the deep representa- tions of GAN-based method are more discriminative, even without global residual. Comparing Fig. 22(c) and Fig. 22 d), when there is global residual, the feature maps containing the image original content information are further reduced, leading to stronger discriminability to degradation types.
A.14 SAMPLES OF DIFFERENT DATASETS
In the main paper, we adopt several different datasets to conduct experiments. Fig. 23 displays some example images from these datasets.
(a) DIV2K-elean: the original DIV2K Agustsson & Timofte (207)) dataset. The high-resolution (HR) ground-truth (GT) images have 2K resolution and are of high visual quality. The low- resolution (LR) input images are downsampled from HR by bicubic interpolation, without any fur- ther degradations.
(b) DIVeK-noise: adding Gaussian noises DIV2K-clean LR input, thus making it contain extra noise degradation. DIV2K-noise20 means the additive Gaussian noise level o is 20, where the number denotes the noise level.
(c) DDV2K- blur: applying Gaussian blur to DIV2K-clean LR input, thus making it contain extra blur degradation. DIV2K- means the Gaussian blur width is 4.
(d) DIV2K- mild: officially synthesized from DIV2K Agustsson & Timofte (2017) dataset as chal-
lenge datasst Timofte et af. (2017, 2018), which contains noise, blur, pixel shifting and other degra- dations. The degradation modelling is unknown to challenge participants.
(e) Hollywood100: 100 images selected from Hollywood dataset Laptev et al. (2008), containing real-world old film frames with unknown degradations, which may have compression, noise, blur and other real-world degradations.
Dataset (a), (b), (c) and (d) have the same image contents but different degradations. However, we find that the deep degradation representations (DDR) obtained by SR networks have discriminability to these degradation types, even if the network has not seen these degradations all during training. Further, for real-world degradation like in (e), the DDR are still able to discern it.
28

---

Under review as a conference paper at ICLR 2023
(a) DIV 2K -clenn
(b) DIV 2k-nosse20
(c) DIV 2K blurd
(c) DIV2K -mild
(e) Holly ood 100
Figure 23: Example images from different datasets. (a) DIV2k-clean (b) DIV2k-noisel2 (c) blu2k-blurd. (d) miv2k-mild. (e) Mollywood100. Different datasets contain different degradation types. (a), (b), (c) and (d) are aligned with image content, but contains degradations. The deep degradation representations (DDR) are discriminative to various degradations.
29


Reviews:
[{"review_id": "cE_qyh0NyQ", "reviewer": "Reviewer_EvNB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes that SR network can learn the distinctive “semantics” named deep degradation representations (DDR). Considering through GAN-based models the SR network can generate more delicate SR textures, the authors believe that there are some kinds of semantic the SR network has learned. Through comprehensively analyzing the feature representations, the authors discover that the semantics in SR network is related to different degradations and two factors are adversarial learning and global residual. Experiments show that adversarial learning and global residual are important and DDR can be applied in many applications and achieves promising results.", "summary_of_the_review": "Overall, I would like to have this work being accepted because of inspiring observation and insightful analysis.", "strength_and_weaknesses": "Pros:\nStrength: \n1 The paper is well-written and easy to read. \n2 THis work is well motivated by the observation of applying pretrained SR models to images of different types of degradations. The idea about DDR is very interesting and seems novel to me.\n3. Authors conduct extensive experiments to analyze the observation and give reasonable explanations.\n\nCons:\n1 In this work CNN-based models such as SRResNet and SRGAN are adopted to illustrate the DDR generated by SR network. Currently Transformer-based SR network like SwinIR also achieves promising results. Does the observation about DDR also apply to the Transformer-based methods?\n2. Do SR models with stronger SR performance perform better in indicating degradation? Can DDR also has the ability to indicate different levels of degradations?\n3. In Fig. 3, why are training and test set for CinCGAN and SRGAN chosen to be different? It seems to me that the same setting is more meaningful.\n4. In section5, injecting DDR embedding shows better performance, it is better to show some visual results for comparison to illusrate the advantage of DDR guidance. \n5. DDR can indeed indicate the feature difference among the input with different degradations, however, is it reasonable to call DDR a kind of sematics?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "NhtHcXydtgr", "reviewer": "Reviewer_xxDc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper analyzes semantics learned in the SR network, and finds that a well trained SR network can naturally extract degradation descriptors. Authors find that conventional or shallow CNN models have difficulty in separating degradation-related semantics. Especially, global residual and adversarial learnings make the SR network extract degradation-related representation. Then, authors exploit their findings for other vision tasks.", "summary_of_the_review": "This paper provides interesting views in the SR network, and analyzes/verifies their assumptions via various experiments. However, experimental validation of applications utilizing degradation-related representations is somewhat lacking. In particular, degradation-related representations should be combined to the recent SOTA  blind SR methods.", "strength_and_weaknesses": "Strength\n- In this paper, interesting findings are introduced such as degradation-related semantics.\n- The proposed findings can be adopted for various low-level vision tasks.\n- It is interesting that we need to diminish the feature discriminability to improve the model generalization for various degradations. \n\nWeaknesses\n- Although the observation and finding of this paper are interesting, the finding of this paper should be verified by real applications. Although applied to distortion identification and blind SR, it is somewhat insufficient. Comparative analysis with existing blind SR techniques such as IKC is required, and experiments on other degradations such as compressing rather than noise and blur are also required.\n\n- I think that this paper mainly focuses on the SR. However, in the motivation section, experiments for observations are biased to image denoising (or restoration). Also, compared to the traditional method is BM3D which is a representative denoising method. When the reader first sees the paper, it seems strange. Figure 2 should be an experiment on SR. \n\n- Figure 2 does not show observations for various degradations. Not only noise, but also degradation such as compression, irregular holes, and color tones should be dealt with as well.\n\n- Figure 5~8 seems to be convincing only if more diverse degradations. In addition, it is necessary to verify whether there is a different phenomenon that varies depending on the upsampling factor of the SR network.\n\n- Minor points\n    - [1pp] sparse coding(Yang ) → sparse coding (Yang )\n    - [3pp] Wang et al. Wang et al. (2020b) → Wang et al. (Wang et al. (2020b))\n    - In Figure 2, the results in the last column are from SRCNN? Text above the figure denotes it as SRCNN but texts in the caption denotes as CinCGAN (Yuan et al., 2018) is trained on DIV2K-mild dataset in an unpaired manner.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "F3v8QreN3jI", "reviewer": "Reviewer_DDMx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper discovers an interesting phenomenon that the SR network is powerful in discriminating the image degradations instead of image contents, especially for well-trained deep networks with global residual and generative adversarial training. To validate the claim, the authors give some empirical evidence by visualizing the distribution of features via t-SNE. In addition, some inspirable analysis and applications are provided and reported.", "summary_of_the_review": "This paper reports an interesting phenomenon that the SR network shows more discriminative capacity in degradation types rather than semantics. While some analysis are inspirable, the authors fails to validate its effectiveness in facilitating the performance of the SR task.", "strength_and_weaknesses": "Strength:\n\n1 Exploring the intrinsic representation of low-level vision models is of great importance to help the design of new methods. This paper makes a step forward towards this goal. \n2 This paper is easy to read, some analysis are inspirable. \n\nWeakness:\n\n1 The key issue of this paper is how this empirical finding can facilitate the SR task. In common sense, the restoration of textures, especially those regular patterns with strong perceptual prior, relies on the semantic information, or at least the similar patterns that are included in the training set. However, according to the findings in this paper, the model mainly learns the degradation-related information. Is this good or bad for the performance of SR? or in other words if the authors recommend the following researches to keep the global residual and the adversarial loss in the designing of SR network?\n2 There lacks sufficient experiments to verify the performance of blind SR with the proposed DDR guidance. Detailed comparison against the state-of-the-art methods should be conducted.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EjaIJOI3Zi", "reviewer": "Reviewer_CXK4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to analyze the feature representations and explore the \"semantics\" in SR networks, namely, deep degradation representations. And authors reveal two factors, i.e., adversarial learning and global residual, which influence the extraction of \"semantics\".", "summary_of_the_review": "Most of demonstrations are mainly based on synthetic data. It is not optimistic for its insights to the future work. Even it may bring the misleading to understand the SR research.", "strength_and_weaknesses": "[Strength]\n1. The paper is well-written and easy to understand.\n2. The paper provides comprehensive analyses, demonstrations, and discussions on DDR. The applications to blind SR, distortion identification and generalization evaluation are also interesting.\n\n[Weakness]\n1. The main concern is that the issue this paper explores would be trivial for further low-level research and realistic SR applications. There are three aspects that I doubt the significance of this work:\n- Empirical settings: All the empirical demonstrations and the important observations and conclusions are mainly based on synthetic data, including artificial blur and noise. Particularly, it is strange that for noise, two version datasets are used for demonstrations, namely, DIV2K-noise20 and DIV2K-noise. For example, in Fig.3, subfigure (a)~(d) has different datasets.\n- Although it claims Hollywood100 with real-world old film degradations, it has a stable intrinsic property, indicating film style, image degradation, etc. Even though it presents a different feature distribution from DIV2K-mild/-noise in Fig.3, it is not direct to draw the conclusion that this difference results from the image degradation. I do not think the analyses for DDR or \"semantic\" are convincing. Besides, why not use a more complex real-world SR dataset, e.g., RealSR, rather than Hollywood100? \n- If ignoring the problems above, it assumes that the observed \"semantics\" hold. In this case, are those observations and conclusions important for the SR task, especially for real-world SR? The paper has few insights on how to indeed address SR problems. Although there are discussions on blind SR, the paper just conducts the evaluation on synthetic data and shows the learned features. What is the importance of the observation from Fig.11 (in Page 9, \"Fig.11 reveals that DDR guidance can make the deep features become more homogenous\")? Besides,  the demonstrations in Fig.11 are conducted on Urban100? it is also confusing. Urban100 is not mentioned in main paper.\n2. On one hand, the demonstrations and analyses are based on very few SR methods with simple network architectures, for example, not including RCAN with attention mechanism or Transformer based networks. It would be misleading, confusing, and not convincing to understand the contributions of this paper, let alone do further SR research based on this paper. On the other hand, most of the experiments are conducted on synthetic data with artificial image degradation. It is ideal and hand-craft, showing few promising insights on further research. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject