SEAL: A Framework for Systematic Evaluation of Real-World Super-Resolution
Authors: Wenlong Zhang
Xiaohui Li
Xiangyu Chen
Xiaoyun Zhang
Yu Qiao
Xiao-Ming Wu
Chao Dong
Venue: ICLR 2024 spotlight (2024)
Keywords: Image super-resolution; Real-World super-resolution
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Real-world Super-Resolution (Real-SR) methods focus on dealing with diverse real-world images and have attracted increasing attention in recent years. The key idea is to use a complex and high-order degradation model to mimic real-world degradations. 
Although they have achieved impressive results in various scenarios, they are faced with the obstacle of evaluation. Currently, these methods are only assessed by their average performance on a small set of degradation cases randomly selected from a large space, which fails to provide a comprehensive understanding of their overall performance and often yields inconsistent and potentially misleading results.
To overcome the limitation in evaluation, we propose SEAL, a framework for systematic evaluation of real-SR. In particular, we cluster the extensive degradation space to create a set of representative degradation cases, which serves as a comprehensive test set. Next, we propose a coarse-to-fine evaluation protocol to measure the distributed and relative performance of real-SR methods on the test set. The protocol incorporates two new metrics: acceptance rate (AR) and relative performance ratio (RPR), derived from acceptance and excellence lines. Under SEAL, we benchmark existing real-SR methods, obtain new observations and insights into their performance, and develop a new strong baseline. We consider SEAL as the first step towards creating an unbiased and comprehensive real-SR evaluation platform, which can promote the development of real-SR.

Published as a conference paper at ICLR 2024
SEAL: A FRAMEWORK FOR SYSTEMATIC EVALUA- TION OF REAL-WORLD SUPER-RESOLUTION
Wenlong Zhang!? Xiaohui , Xiangyu Chen Yu Qiao" Xiao-Ming Wu Chao Dong
Xiaoyun Zhang*
'he HongKong Polytechnic University, SShanghii Al Laboratory 'Shanghai Jiao Tong University
' Uniierssity of Macau 'Shenzhen Institute of Advanced Technology, CAS
wenleng ong. zhange connect woa.eecttppoll..hh... polyu.hk, xiao-ming wu@pol edk.kkk chao.
ABSTRACT
Real-world Super-Resolution (Real-SR) methods focus on dealing with diverse real-world images and have attracted increasing attention in recent years. The key idea to use a complex and high-order degradation model to mimic real-world degradations. Although they have achieved impressive results in various scenar- ios, they are faced with the obstacle of evaluation. Currently, these methods are only assessed by their average performance on small set of degradation cases randomly selected from large space, which fails to provide comprehensive understanding of their overall performance and often yields inconsistent and po- tentially misleading results. To overcome the limitation in evaluation, we propose SEAL, framework for systematic evaluation of real-SR. In particular, clus- ter the extensive degradation space to create a set of representative degradation cases, which serves as comprehensive evaluation protocol to measure the distributed and relative performance of real-SR methods the test set. The protocol incorporates rate (AR) and relative performance ratio (RPR), derived from acceptance and excellence lines. Under SEAL, we benchmark existing real-SR methods, obtain new observations and insights into their performance, and develop a new strong baseline. We consider SEAL as the first step towards creating a comprehensive real-SR evaluation platform, which can promote the development of real-SR. The source code is available htttp: /t//tggtuuttt thub. com /XP Group//EAA
set. Next, we propose coarse-to-fine
metries: acceptance
INTRODUCTION
Image super-resolution (SR) aims to reconstruct high-resolution (HR) images from their low- resolution (LR) counterparts. Recent years have witnessed great success in classical SR settings (i.e., bicubic downsampling) with deep learning techniques (Dong al. 2014 Zhang et al. 2018b c Ledig et al. 2017 Wang at.,][2013 To further approach real-world applications, series of "blid" SR methods have been proposed to deal with complex and unknown degradation kernels (Zhang al. 2018a; Gu al.
2019 Wang al. 2021a Luo et al. 2020). Among them, real-SR meth- ods, such as BSRGAN (Zhang et al. 2021) and RealESRGAN (Wang et al., (2021b), have attracted increasing attention due to their impressive results in various real-world scenarios.
Different from classical SR that only adopts simple downsampling kernel, real-SR methods pro- pose complex degradation models (e.g., a sequence of blurring, resizing and compression opera- tions) that can represent much larger degradation space, covering a wide range of real-world cases. However, they also face a dilemma evaluation: As the degradation space is vast, how evaluate their overull performance? Directly testing on all degradations is obviously infeasible, as there are numerous degradation combinations in the vast degradation space.
To evaluate the performance of real-SR methods, previous works directly calculate the average per- formance on a mandomly-sampla small-sized test set based on an IQA metric (c.g., PSNR). How- ever, find that this evaluation protocol is fatally flawed. Due to the vastness of the degradation
"Corresponding author

---

Published as a conference paper at ICLR 2024
BSANet
Accestance Line
26.0
Biithee 0.54]
RR (AR: 0.15)
PSNRUBI
26.0
24.6
Racdom test set
Ran-Som test set
20
80
100
Rapresentiatve
(a) Results with the conventional evaluation ap- proach.
(b) Results with our framework SEAL.
Figure I: (a) We compare the average performance BSRNet and RealESRNet on two real test sets generated by common practice. There is a significant variance in their performance: the differences between their average PSNR the two
-0.23dB and 0. respectively, leading to contradictory conclusions. (b) BSRNet and RealESRNet assessed under our SEAL framework in distributed manner with 100 representative sets. It shows the former outperforms the latter in 60% cases, providing comprehensive overview of their performance.
space, small test set that is selected randomly cannot reliably represent the degradation space and may cause significant bias and randomness in the evaluation results, illustrated in Fig. [ In addition, the current evaluation strategy is not enough for assessing real-SR methods, as they typically average quantitative results across all testing samples, which may also lead to misleading comparison results. For example, one method may outperform another 60% of the degradation types, but it may not achieve a higher mean PSNR value for the entire test set (Sec. (Sec. The av- erage score cannot adequately represent the overall performance and distribution. Furthermore, if the goal is to improve the average score, we could focus solely on enhancing the performance of simple cases (e.g., small noise blur), which, however, would adversely affect difficult ones (e.g., complex degradation combinations). This would contradict our main objective. Instead, once we have achieved satisfactory outcomes in easy cases, we should divert our focus towards challenging ones to enhance the overall performance. The aforementioned points indicate the need for a new framework that can comprehensively evaluate the performance real-SR methods.
In this work, we establish a systematic evaluation framework for real-SR, namely SEAL, which assesses relative, distributed, and overall performance rather than relying solely on absolute, average, and misleading evaluation strategy commonly used in current evaluation methods. Our first step is use a clustering approach to partition the expansive degradation space to identify representative degradation cases, which form comprehensive test set. In the second step, we propose an evaluation protocol that incorporates two new relative evaluation metrics, namely Acceptance Ratio (All) and Relative Performance Ratio (RPR), with the introduction of acceptance and excellence lines. The AR metric indicates the percentage by which the real-SR method surpasses the acceptance line, which is minimum quality benchmark required for the method to be considered satisfactory. The RPR metric measures the improvement of the real-SR method relative to the distance between the acceptance and excellence lines. The integration of these metrics intends to provide more thorough and detailed evaluation of real-SR methods.
With SEAL, it becomes possible conduct comprehensive evaluation of the overall performance of real-SR methods, as illustrated in Fig. [ The significance of our work can be summarized as:
Our relative, distributed evaluation approach serves as a complement to existing evaluation methods that solely rely absolute, average performance, addressing their limitations and providing a valuable alternative perspective for evaluation.
By employing SEAL, we benchmark existing real-SR models, leading the discovery of new observations and valuable insights, which further enables us to develop new strong real-SR model.
The components of our SEAL framework flexibly customizable, including the cluster- ing algorithm, acceptance/cxcelllnee lines, and evaluation protocol. can facilitate the development of appropriate test sets and comparative evaluation metrics for real-SR.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
Image super-resolution. Since Dong et al. (Dong et. 2014) first introduced Convolutional Neural Networks (CNNs) to the Super-Resolution (SR) task, there have been significant advancements in the field. A variety of techniques have been developed, including residual networks (Kim et al,
2016), dense connections (Zhang et al. 2018c). channel attention (Zhang et al., 2018b), residual- in-residual dense blocks (Wang et al. 20188), and transformer structure (Liang et 20.Iaa Chen al. 2023b). To reconstruct realistic textures. Generative Adversarial Networks (GANs) (Ledig et al. 2017; Wang et al., 2018 Zhang et al., 2019) Wenlong et al., 2021) are introduced to SR approaches for generating visually pleasing results. Although these methods have made significant progress, they often rely on simple degradation model (i.e., bicubic downsampling), which may not adequately recover the low-quality images in real-world scenarios.
Blind super-resolution. Several works have been made to improve the generalization of SR net- works in real-world scenarios. These works employ multiple degradation factors (e.g., Gaussian blur, noise, and JPEG compression) to formulate blind degradation model. SRMD Zhang al., 2018a) employs a single SR network to learn multiple degradations. Kernel estimation based meth- ods (Gu al. 2019 Luo al. 2020 Bell-Kliglerr et al.] 2019 Wang al.) (201a) introduce kernel estimation network to guide the SR network for the application of the low-quality image with different kernels. To cover the diverse degradations of real images, BSRGAN (Zhang et al. 2021) proposes practical degradation model that includes multiple degradations with a shuffled strategy. RealESRGAN (Wang al., 2021b) introduces high-order strategy to construct a large degradation model. These works demonstrate the potential of blind SR in real-world applications.
Model evaluation for super-resolution. For non-blind SR model evaluation, a relatively standard process employs the fixed bicubic down-sampling on the benchmark test datasets to generate low- quality images. However, is typically implemented using a predefined approach (e.g., uniform sampling) for blind SR, such as the general Gaussian blur kernels (Zhang et al. [018aa] Liang et al.) 2021c), Gaussian8 kernels (Gu et al., 2019), and five spatially variant kernel types (Liang et al, 2021b). For real-SR, existing methods often add random degradations to DIV2K val (includes 100 Ground -ruth images) to construct the real test set, such as DIV2K4D in BSRGAN (Zhang et al.) 2021) and DIV2K val with three Levels in DASR (Liang et al. 2022). However, these methods use small test sets with average performance, making it difficult to evaluate overall performance across different degradation combinations in real-world scenarios.
DEGRADATION SPACE MODELING
3.1 GENERATING THE DEGRADATION SPACE
In real-SR, the degradation process Wang et al. 20216 can be simulated by
FLR (d, o d2 o (I!R),
(1)
where s is the number of degradations applied on high-resolution image HHR and d, (1 S i 8) represents a randomly selected degradation. Assume there are only s degradation types (c.g.. blur, resize, noise, and compression), and each type contains only is discrete degradation levels. The total degradation should be A x A.. With & = 10 and & = 10, will generate a degradation space astronomical figure. Clearly, randomly sampling
of magnitude (418) 10 which is already limited number of degradations (e.g., 100 in existing works (Zhang et al. (202])) from such huge space cannot adequately represent the entire space, which will inevitably result in inconsistent and potentially misleading outcomes, as illustrated in Fig.
3.2 REPRESENTING THE DEGRADATION SPACE
To represent the degradation space D. straightforward way is to divide the space by degradation parameters, which may seem reasonable at first glance. However, we observe that different com- binations of degradation types may have similar visual quality and restoration difficulty. As shown in Fig. Of of the Appendix, the images undergone different degradations have similar appearances. This suggests that it might be more reasonable to distinguish the degraded images with their low- level features instead of degradation parameters.
3

---

Published as a conference paper at ICLR 2024
Clase center
Truia
Acceptance Line Exceilence Line
Accepeable
RF RA
Calealare
RP &
GT
Text
Distributed
RPR
Performancce
Cnacceptable
Degradation Space
Degradation Param.
Real-SR Model
Evaluation Metrics
Figure 2: Our proposed evaluation framework consists of clustering- approach for degra- dation space modeling (Sec. B) and set of metrics based on representative degradation cases (Sec. 4). We divide the degradation space into K clusters and use the degradation parameters of the class centers to create K training datasets to train K' tinyllarge SR models as the accep- tance/excellence line. The distributed performance (Eq. 4 of the real-SR model across the test datasets will be compared with the acceptance and excellence lines and evaluated by a set of metrics.
Therefore, we propose to find prototypical degradation cases to represent the vast degradation space. As shown in Fig. 2 a plausible solution is to cluster the degradation space by grouping the degraded images into K groups and choose the K group centers the representative cases:
D = {c1, C2 CK),
where c11 si K) is the center of the i-th group. Note that the images can be represented by their features (e.g.. image histograms) and clustered by conventional clustering algorithm such as spectral clustering.
3.3 EVALUATING REAL-SR MODELS USING THE REPRESENTATIVE DEGRADATION CASES
We then use the degradation parameters of the cluster centers c; to construct test set for systematic evaluation, denoted as the SE test set:
D (e., De.. Deek
(3)
where De, Sis K)) a of low-quality images obtained by using the degradation parameters of 6; on set of clean images (e.g., DIV2K dataset). DD, can be used to evaluate the distributional performance of real-SR model on a representative degradation case. Diest can be used to provide full picture of the performance all representative degradation cases.
4 EVALUATION METRICS
To provide comprehensive and systematic overview of the performance of real-SR model on Diesi, we develop a set of evaluation metrics to assess its effectiveness in a quantitative manner.
Distributed Absolute Performance. The most straightforward way to evaluate real-SR model is to compute its distributed performance on Diest:
ea (Q1,
Que
1 Σ K
(4)
where Q represents the average absolute performance of real-SR model on the i-th representative test D,, and Qd denotes the average absolute performance on Diest
Distributed Relative Performance. To comparatively evaluate real-SR model and pinpoint the representative cases where its performance is deemed inadequate, we specify an acceptance line and an excellence line. The acceptance line is designated by small network (e.g., FSRCNN (Dong et al. 2016)), while the excellence line is provided by a large network (e.g., SRResNet (Ledig et al. 2017 If the real-SR model is unable to surpass the small network is considered failed on that case. We use Q and Qex to represent the acceptance line and the excellence line:
representative case,

---

Published as a conference paper at ICLR 2024
@ -
.Kk), -- --- 02 . .Kk)
(5)
where Q and Q the performance of the small and large networks trained with the degradation parameters of e; in a non-blind manner.
Acceptance Rate (AR) measures the percentage of acceptable cases among all K representative degradation cases for a real-SR model. An acceptable case is one in which the performance of real-SR model surpasses the acceptance line. ARt is defined
AR
K Σ
Q.9),
where II represents the indicator function. real-SR model.
AR can reflect the overall generalization ability of
Relative Performance Ratio (RPR) is devised to compare the performance of real-SR models the same scale w.r.t. the acceptance and excellence lines. is defined as
Q - Qte 000h Quh )xr) and R (RPR1, RPR2, RRPKK)
RPR: =
where a denotes the sigmoid function, which is used map the value to (0, 1). Note that RPR, > (0) = 0.5 indicates that the real-SR model is better than the acceptance line on the i-th degradation case, and RPR; (1) 0.73 means it is better than the excellence line.
a) Interquartile range of RPR (RPR)) is used to access the level of variance in the performances of real-SR model on Dies It is defined as:
RPR, = Rws Rw:,
where Rw, and Rw, denote the 75th and 25th percentiles Wan al. 2014 of the RPR scores, respectively. Low RPR, means the real-SR model demonstrates a similar relative improvement in most degradation cases.
b) Average RPR on acceptable cases (RPRA) computes the mean of RPR scores on acceptable cases:
RPRA
Σ R4 RA = (R. R|R, 2 0.5].
[RAl
R,ERA
Note that RPRA (0.5, 1), and RPRA 0.73 means the average performance of real-SR model on acceptable cases exceeds the excellence line.
Average RPR unacceptable cases (RPRy) computes the mean of RPR scores unac- ceptable cases:
RPR
Σ R.
Ry [R, R/R, < 0.5).
(10)
Ru
R.ERU
Note that RPR E (0, 0.5), and RPRA near 0.5 means the average performance real-SR model on unacceptable cases is close to the acceptance line.
Coarse-to-fine Evaluation Protocol. Based on the proposed metrics, we develop coarse-to- fine evaluation protocol to rank different real-SR models. As illustrated in Fig. 3 the models are compared by the proposed metrics sequentially by order priority. AR represents a coarse-grain garane- comparison, while RPR provides a fine-grained comparison. If their performances are too close to the current metric, the next metric is used to rank them.
Real-SR Models
AR
RPR,
RPR
RPRR,
Coarse-grained comparison
Fine-grained comparison
Figure 3: A coarse-to-fi evaluation protocol rank real-SR models with the proposed metrics.
5

---

Published as a conference paper at ICLR 2024
Table 1: Benchmark results and ranking of MSE-based real-SR methods in PSNR by our proposed SEAL. The subscript denotes the rank order. x means the model fails in majority of degradation cases.
PARCAM
Models
PSNR
ART
RPR:
RPRA
RPRr t Rank
SRResNet
20.95
0.00
0.02
0.00
X
DASR
21.08
0.00
0.01
0.00
0.02
22.77
()
0.42g)
0.72(z)
(.27(
Figure 4: Visualization of distributed performance in PSNR for MSE-based real-SR methods on Set14-SE.
RealESRNet
22.67
0.27()
0.28(a)
0.63
0.28
RDSR
22.44())
0.08px)
0.23
0.63
0.21
RealESRNat-GiD
22.82
0.44(z)
(.37(s)
0.74())
0.33())
SwinIR
22.61(4)
0.41g)
0.24())
0.58(4)
0.29
Inger PSNR (dB)
ROSH 22.55
RailEsNN 22,73
Swenlk 23.04
ESRCAN
SRARee 23,52
BSRNO 23
23,85
23.31
STATE BANK O
STATE BANK ON
STATE BANK O
STATE BANK O
STATE BANK O
STATE BANK MARKET
STATE BANK O
STATE BANK O
MARKET BE
MARKET BF
MARKET DF
MARKET BF
MARKET BR
MARKET DA
MARKET DF
Insu
RenlESRNa
SoiniR 23.52
ESRCAN
ADSR
BSRNE
Ren EERESNNGAAD
PSNR (JB)
23 06
23.65
23,68
23.70
23.96
24.20
Figure 5: Visual results of MSE-based real-SR methods with the acceptance line FSRCNN and
excellence line SRResNet. best viewed color.
5 EXPERIMENTS
5.1 IMPLEMENTATION
Constructing the test set for systematic evaluation. We utilize two widely-used degradation mod- els. BSRGAN (Zhang et al. 202T) and RealESRGAN (Wang et al. 2021b), which are designed to simulate the real-world image space. By combining the two degradation models with equal prob- ability [0.5, 0.5], we generate dataset of 10 low-quality images from the ground-truth (GT) image, lenna, from the Set14 dataset (Zeyde et al. 2010). To categorize the degraded images, employ spectral clustering due to its effectiveness in identifying clusters of arbitrary shape, making flexible choice for our purposes. Specifically, we first use the histogram feature (Tang et al. [
Ye & Docrmann 2012) with 768 values (bins) to represent the degraded image. Then, we compute the pairwise similarities of all degraded images. Next, implement spectral clustering based on the computed similarity matrix to generate 100 cluster centers. The degradation parameters of the cluster centers are then utilized to generate the distributional test set Diest, We take Set14 (Zeyde et al., 2010) and DIV2K val (Lim et al. 2017) to construct the test sets for systematic evaluation, denoted Set14-SE and DIV2K_ val-SE, respectively.
Establishing the acceptance and excellence lines. We use the 100 representative degradation parameters to synthesize 100 training datasets based on DIV2K. In the case of MSE-based real- SR methods, we utilize a variant of FSRCNN (Dong et al. 2016), referred to as FSRCNN-mz, to train a collection of 100 non-blind SR models, which serve as acceptance line. Concurrently, we employ SRResNet (Ledig et al., 2017). following an identical procedure, the excellence line D The models within the model zoo are initially pre-trained under the real-SR setting. Subsequently, they undergo a fine-tuning process consisting of total of x 10a iterations. The Adam (Kingma Ba 2014) optimizer with 3, 0.9 and 32 = 0.99 is used for training. The initial learning rate is 2 10 We adopt L1 loss optimize the networks. Regarding GAN- based SR methods, adopt the widely recognized RealESRGAN (Wang et al. (2021b) as our acceptance line. Concurrently, we consider the state-of-the-ard RealH (Chen al. 2023b as excellence line. We utilize the officially released models for our experiments.
Both the lines and the distributed test will be released.

---

Published as a conference paper at ICLR 2024
0.7
Table 2: Benchmark results and ranking of GAN-based real-SR methods in LPIPS by our proposed SEAL. The subscript denotes the rank order. x means the model fails a majority of degradation cases.
Rea
Roa SREAN
0.5
Models
LPIPS
AR 1
RPR; L
RPRA 1
RPRg 1
Rank
detncet
ESRGAN
0.6224,))
(x00
ReaISRGAN
0,5172
0.01
0.10
0.53
0.14
Figure 6: Visualization of distributed performance in LPIPS for GAN-based real-SR methods on Set14-SE.
DASR
0.5230
(x02,x)
BSRGAN
0,4810
0.44,s:
0.40:a)
0.72(1)
0.28,g)
MMReaSSR
0.4770(2)
0.80gg:
0.08
(.57
0.41g)
SwinIR
0.4656(1)
0.81
0.24.g)
0.71
0.31ga
ROISRGAN 0,424
MARROAS 0,384
RESHATGAN 0,530
LPISS
0.649
0.440
0,428
0,416
Ippil LPIPS
RealSRCAAN 0,507
DASK 0.495
BSSKIANN 0.471
Reall SEGAN
MMlleaiS 0,461
Swelit 0,456
Reall ATAN 0,439
0.470
Figure 7: Visual results of GAN-based real-SR methods with the acceptance line RealESRGAN and excellence line RealHATGAN. is best viewed in color.
5.2 BENCHMARKING EXISTING MSE-BASED AND GAN-BASED REAL-SR METHODS
We utilize the proposed SEAL to evaluate the performance of existing MSE-based real-SR methods, including DASR (Wang al. 2021a). BSRNet (Zhang et al. 2021), SwinIR (Liang al. 2021a
RealESRNet (Wang et al. 20211), RDSR (Kong et al. 2022 and RealESRNet-GG (Zhang et al.
2022). Furthermore, we benchmark GAN-based real-SR methods such ESRGAN (Wang al.
2018 DASR (Liang et al. 2002), BSRGAN (Zhang et al. 2021). MMRealSR (Mou et al. 2022).
SwinIR (Liang et al. 2021a). We also modify SRGAN (Ledig et al. 2017) to achieve ReaiSRGAN under the RealESRGAN training setting.
The visualization of the distributed performance offers a comprehensive insight into real-SR performance. Fig. 4 illlustrrats the distribution performance for MSE-based real-SR methods, using PSNR as the IQA metric. On the other hand, Fig. 6 depicts the distribution performance for GAN- based real-SR methods, with LPIPS as the metrie. Both visualizations are generated using proposed SE test set. The SE test sets are arranged in ascending order based on the PSNR values of the acceptance line output. Test sets with lower numbers represent more challenging cases. It's noticeable that there are a few degradation cases that fall significantly below the acceptance line in Fig. 4 Interestingly, real-SR methods seem perform better on more challenging degradation cases. This is evident in test datasets 0-20 in Fig. and 80-100 in Fig. 6
The coarse-to-fine evaluation protocol offers systematic ranking. In Tab. and Tab. 2 the real- SR models with ARt below 0.25 are excluded from the ranking due to their low acceptance rates. For the real-SR models with AR 0.25, by-p-b ranking is performed based on {AR. RRP,, RPRA. RPRc), ). with thresholds {0.02, 0.02, 0.05, 0.05] respectively. If the difference in the cur- rent metric exceeds the threshold, the metric used to represent the overall ranking. Otherwise, the next metric is considered. From our proposed SEAL evaluation, we can make several observa- tions: (I) Some existing methods fail on the majority of degradation cases. The AR values of some existing methods are below 0.5, as shown in Tab. and Tab, 2 For instance, most MSE-based real-SR models can not even outperform the small network (i.e., FSRCNN-m2 in most degrada- tion cases. (2) Our SEAL is capable of ranking existing methods across various dimensions, such as robustness, denoted by RPR, and performance bound indicated by RPRA In Tab. [ the metric learning based MMRealSR achieves significant robustness (RPR, 0.08) compared with

---

Published as a conference paper at ICLR 2024
the transformer-basee SwinlR (RPR, 0.24). Therefore, under our current coarse-to-fin evaluation protocol, MMRealSR ranked in the first place. Interestingly, we observed that SwinIR achieves higher RPRA the same AR level. If the user prioritizes the performance of acceptance cases, SwinlR would be a better choice. Consequently, we can also flexibly set R.P R as the first finer metric. In this way, SwinIR would take the first place. (3) The acceptance line serves
useful
reference line for visual comparison. Visual results are presented in Fig. S and Fig. 7 It's evi- dent that the visual results of the acceptance line can serve basic need for image quality, while the visual results of the excellence line represent the upper bound of image quality under the cur- rent evaluation protocol. The visuals below the acceptance line clearly exhibit unacceptable visual effects, including blurring (as seen in the crocodile results of ReaISRGAN and DASR in Fig. 7
over-sharpening (as seen the text results of RealESRNet in Fig. 5p. and other artifacts. Notably, our SEAL can flexibly use new reference lines for future needs.
5.3 COMPARISON WITH THE CONVENTIONAL EVALUATION
Here, we compare our SEAL with the conventional strategy (Zhang et al. 2021 Liang et al. 2022) used for evaluating real-SR models.
Randomly generated multiple synthetic test sets fail to establish a clear ranking with distinct differences. We randomly sample 100 degradation cases and add them to Set14 to obtain 100 test sets (Set14-Random). Tab. shows that 1) the mean and standard deviations (std) of PSNR obtained on the two 1ee14-Randomllee 100 datasets show significant inconsistency, demonstrating the presence of high randomness and variability in the sampled degradation cases. 2) On our Set14-SE (formed with the 100 representative cases), the means and stds of the compared methods are very close, making it hard to establish a clear ranking with distinct differences among the methods. In contrast, our SEAL offers a definitive ranking of these methods based on their AR scores, offering new systematic evaluation view.
Table 3: Comparison with multiple synthetic test sets on mean and standard deviations.
Se114-Radomm100 (H1)
Bel1--Ramdo1100 (#2)
Se114SE
PSNR
1
sid
1
sid
std
AR1
RPR, 4. RPRA
RPRp
rank
BSRNet
23.39
1,56
22.98:1)
1.64g)
22.77:g)
1.65 0.59g)
0.42,)
0.72
0.27
RealESRNet-GD
23.72(1)
1.64(;
22.98
(,95())
22,82/1)
1.83())
0.43/2)
0.37())
0.74g))
0.33
SwinIR
23.25
1.62
22.79:
1.09
22.61.
1.69
0.41
0.24g)
0,58
0.29
RealESRNet
23.54:2)
1.55g:
22.80
(383
22.67
1.73 0.27())
0.28
0.63
0.28
The utilization of a randomly generated synthetic test set may lead to misleading outcomes. Following /Zhang al. (2021); Liang et al. (2022), randomly add degradations to images in the DIV2K (Agustsson & Timofte 2017) validation set to construct a single real-DIV2K val set. Tab. J shows that RealESRNet achieves a higher average PSNR than BSRNet (24.93dB vs. 24.77dB) on real-DIV2K_ve _val, leading to the misleading impression that RealESRNet is superior than BSRNet. However, our SEAL framework leads contrary conclusion. BSRNet obtains much higher PSNR (24.74dB vs. 24.43dB) and AR value (0.55 vs. 0.15) than RealESRNet on our DIV2K vaE-SE, illustrating that the former outperforms the latter in most representative degradation cases.
Table 4: Comparison with the single synthetic test set. Under our SEAL framework, BSRNet is ranked first, contrary the results obtained by the conventional method. PSNR-SE (Eq. 4 denotes the average PSNR on our val-S..
PSNR
Rank
PSNR-SE
ARt
RPR, $
RPRA* f
RPRp
Rank (ours)
BSRNet
24.77
2
24.74
0.55
0.36
0.65
0.26
RealESRNee
24.93
1
24.43
0.15
0.33
0.59
0.18
5.4 ABLATION STUDIES AND ANALYSIS
In this section, we first conduct ablation studies several factors that affect spectral clustering, including the number of sampled degradations, similarity metrics, and the number of clusters. Then, study the stability of degradation clustering for real-SR evaluation.
8

---

Published as a conference paper at ICLR 2024
235
ESRveet Roa ESRNiCt
86
Manseref
Dataset of clusteen
Figure 8: Effect of the number of clusters on RPR value of Set14-SE.
Figure 9;
Effect of the dataset used for clus- tering average PSNR of Set14-SE.
Number of sampled degradations. To assess the effect of the number of sampled degradations, we randomly generate four datasets, each containing 500, 1000, 5000, and 10000 degradation samples, respectively. We compute the variance of the similarity matrices for each of these datasets, which are 8.32, 8,45, 8.68, and 8.71, respectively. The observation indicates that the change in variance not significant when the number of samples increases from 5000 to 10000. This observation suggests that a sample size of 5000 random degradations sufficiently represents the degradation space. However, ensure the highest possible accuracy in results, opted to use 10000 degradation samples for the clustering process.
Choice of similarity metric. We compare differ- ent metrics that are used to compute the similarity matrix for degradation clustering, including MSE, SSIM, and histogram similarity. In Tab, [ the pu- rity accuracy of the clustering results using MSE or SSIM is significantly lower than that with his- togram similarity, especially when noise is consid- ered. Thus, we adopt histogram similarity as the similarity metric.
Table 5: The purity accuracy of the cluster- ing results with different similarity metrics on Blur100, Noise100, and BN100 datasets.
Range
Histogram
Blart00 Noise 100
4
78.2%
78.2%
80.2%
1 1.40
4
99.698
34.6%
80.25
Blart00 Noise100
51.7%
58.2%
80.5%
Choice of the number of clusters. Our goal is to generate as many representative classes as possible while maintaining the clustering quality so that
the class centers can serve representative cases. The results in Fig. S show the performance of RealESRNet and BSRNet becomes stable as & approaches 100, with minimal variations observed for k 60, 80, 100. Therefore, to achieve more comprehensive assessment and strike balance between clustering quality and time cost, we set k = 100 without further increasing its value. In the appendix, have included the quantitative results of the silhouette score (Rousseeuw) 1987). metric commonly employed to evaluate the quality of clusters.
Stability of degradation clustering for real-SR evaluation. In Fig. we study the stability of degradation clustering by using different images as a reference for evaluation. Beyond the Lenna image, our study incorporated four additional images--peecfffaally, Baboon, Barbara, Flowers, and Zebra-free frrm-ffrommoo the Set14 dataset. These images were employed as Ground Truth images in the con- struction of the clustering dataset, adhering the same degradation clustering process. Despite using different reference images, our results show that the average PSNR of BSRNet is consistently higher than that of RealESRNer by more than 0. indicating that our degradation clustering method exhibits excellent stability for real-SR evaluation.
CONCLUSIONS
In this work, we have developed new evaluation framework for fair and comprehensive evaluation of real-SR models. We first use a clustering-based approach to model large degradation space and design two new evaluation metrics, AR and RPR, to comparatively assess real-SR models on representative degradation cases. Then, we benchmark existing real-SR methods with the proposed evaluation protocol and present new observations and insights. Finally, extensive ablation studies are conducted on the degradation clustering. We have demonstrated the effectiveness and generality of SEAL via extensive experiments and analysis.

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China under Grant (6227625,,6272255)),
REFERENCES
Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, volume 3, pp. 2017.
Sefi BelleKligler, Assaf Shocher, and Michal Irani. Blind super-resolution kernel estimation using an internal-gan Advances Neural Information Processing Systems, 32, 2019.
Xiangyu Chen, Xintao Wang, Wenlong Zhang, Xiangtao Kong, Yu Qiao, Jiantao Zhou, and Chao Dong. Hat: Hybrid attention transformer for image restoration. arXiv preprint arXiv:2309.052333 052399 2023a.
Xiangyu Chen, Xintao Wang, Jiantao Zhou, Yu Qiao, and Chao Dong. Activating more pixels in image super-resolution transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22377,22377, 2023b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning deep convolutional network for image super-resolution. In European conference on computer vision, pp. 184-199. Springer, 2014.
Chao Dong, Chen Change Loy, and Xiaoou Tang. Accelerating the super-resolution convolutional neural network. In European Conference on Computer Vision, pp. 391-407. Springer, 2016.
Jinjin Gu, Hannan Lu, Wangmeng Zuo, and Chao Dong. Blind super-resolution with iterative ker- nel correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1604-1613, 2019.
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1646-1654, 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1rXiv:1412.69880 2014.
Xiangtao Kong, Xina Liu, Jinjin Gu, Yu Qiao, and Chao Dong. Reflash dropout in image super- resolution. In Proceedings of the JEEF/CV Conference on Computer Vision and Pattern Recog- nition, pp. 6002-6012, 2022.
Christian Ledig, Lucas Theis, Ferene Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using generative adversarial network. In CVPR, volume 2, pp. 4, 2017.
Jie Liang Hui Zeng, and Lei Zhang. Efficient and degradation-adaplie network for real-world image ruper-resolution. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings. Part XVIIII pp. 574-591. Springer, 2022.
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Lue Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. IEEE International Conference on Computer Vision Workshops, 2021a.
10

---

Published as a conference paper at ICLR 2024
Jingyun Liang, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Mutual affine network for spatially variant kemel estimation in blind image super-resolution. In Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 4096-4105, 2021b.
Jingyun Liang, Kai Zhang, Shuhang Gu, Luc Van Gool, and Radu Timofte. Flow-based kernel prior with application to blind super-resolution. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10601-10610, June 2021c
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep resid- ual networks for single image super-resolution. In Proceedings of the IEEE conference on com- puter vision and pattern recognition workshops, pp. 136-144, 2017.
Zhengxiong Luo, Yan Huang, Shang Li, Liang Wang, and Tieniu Tan. Unfolding the alternating optimization for blind super resolution. arXiv preprint a7Xiv:2010.02631, 2020.
Chong Mou, Yanze Wu, Xintao Wang, Chao Dong, Jian Zhang, and Ying Shan. Metric learning based interactive modulation for real-world super-resolution. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Purt XVII, pp- 723-740. Springer, 2022.
Peter Rousseeuw. Silhouettes:
graphical aid to the interpretation and validation of cluster analy- sis. Journal of computational and applied mathematics, 20:53-66 65, 1987.
Hinrich Schütze, Christopher D Manning, and Prabhakar Raghavan. Introduction to information retrieval, volume 39, Cambridge University Press Cambridge, 2008
Huixuan Tang, Neel Joshi, and Ashish Kapoor. Learning a blind measure of perceptual image quality. In CVPR 201/, pp. 305-312. IEEE, 2011.
Xiang Wan, Wenqian Wang, Jiming Liu, and Tiejun Tong. Estimating the sample mean and standard deviation from the sample size, median, range and/or interquartile range. BMC medical research methodology, 14(1):1-13, 2014.
Longguang Wang. Yingqian Wang, Xiaoyu Dong, Qingyu Xu, Jungang Yang. Wei An, and Yulan Guo. Unsupervised degradation representation learning for blind super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10581-10590, 2021a.
Xintao Wang. Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Chen Change Loy, Yu Qiao, and Xiaoou Tang. Esrgan: Enhanced super-resolution generative adversarial networks. 2018.
Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 1905-1914, 2021b.
Zhang Wenlong, Liu Yihao, Chao Dong, and Yu Qiao. Ranksrgan: Generative adversarial networks with ranker for image super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Peng Ye and David Doermann. No-reference image quality assessment using visual codebooks.
IEEE Transactions on Image Processing, 21(7):3129-3) 2933138, 2012.
Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse- representations. In International conference on curves and surfaces, pp. 711-730. Springer, 2010.
Kai Zhang, Wangmeng Zuo, and Lei Zhang. Learning a single convolutional super-resolution net- work for multiple degradations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3262-3271, 20.8a.
Kai Zhang, Jingyun Liang, Lue Van Gool, and Radu Timofte. Designing a practical degradation model for deep blind image super-resolution. arXiv preprint 103.v:2103./4000 2021.
Wenlong Zhang, Yihao Liu, Chao Dong, and Yu Qiao. Ranksrgan: Generative adversarial networks with ranker for image super-resolution. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision pp. 3096-3105, 2019.
11

---

Published as a conference paper at ICLR 2024
Wenlong Zhang, Guangyuan Shi, Yihao Liu, Chao Dong, and Xiao Wu. A closer look at blind super-resolution Degradation models, baselines, and performance upper bounds. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 527-536, 2022.
Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention networks. In ECCV, 2018b.
Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018c. 8c.
12

---

Published as a conference paper at ICLR 2024
A ADAPTABILITY OF OUR SEAL FRAMEWORK
We contend that the components of our SEAL framework are highly adaptable to user preferences. For instance, users have the option choose different reference line to visualize distributed per- formance, reorganize the SE test sets into new groups, and utilize any IQA metrics for evaluation.
A.I INCORPORATING NEW IQA METRICS
To illustrate the adaptability of our SEAL framework, we have opted for SSIM as the IQA metric to perform comprehensive evaluation of real-SR methods. As depicted in Tab. 6 RealESRNet surpasses other methods in terms of AR, an outcome that can be credited to use of sharpened Ground-Truth images. It is significant that RealESRNet and SwinIR exhibit remarkable stability, as evidenced by their RPR, values. Furthermore, our findings indicate that SwinIR attains the highest RPRA value, implying that transformer- based networks favor acceptance degradation cases. As evidenced by these observations, our proposed evaluation framework displays considerable adapt- ability. It accommodates various IQA metrics to systematically evaluate real-SR methods from diverse angles, such as reconstruction capability (PSNR) and structural similarity (SSIM).
Table 6: Results and ranking of different methods in SSIM by our SEAL framework. The subscript denotes the rank order. represents failed SR model in large degradation space.
Set14-SE
ART f
RPR
RPRA
RPR
Rank
SRResNet
0.00
0.04
0.00
0.04
DASR
0.00
0.03
0.00
0.04
X
BSRNet
0.76(3)
0.27
0,70(2)
0.36
3
RealESRNet
0.91(1)
0.16(1)
0.67
0.43
RDSR
0.32(z)
0.22
0.59
0.33:5)
ReaIESRNeI-GD
0.69
0.26
0.67
0.39
SwinIR
0.84(2)
0.17
0.72(1)
0.38
2
A.2 USING MULTIPLE REFERENCE IMAGES FOR DEGRADATION CLUSTERING
To illustrate the adaptability of the component in SEAL framework, have incorporated new experiment by utilizing the reference images in Figure for degradation clustering. To accomplish this, we computed the average of the similarity matrices induced by these images. We used the newly identified representative degradation cases to rank GAN-based methods and provided the results in Tab. 7 Notably, we have observed that this adjustment has led to a more conclusive ranking compared to the results obtained when using single image (Tab. D In Tab. [ we observed that SwinIR and MMRealSR had very close difference only 0.01 in the AR metric. Consequently, their ranks needed to be determined using finer metrics such RPR,, However, in Table 7, we found that SwinlR (AR: 0.86) and MMRealSR (AR: 0.75) could be easily ranked based on the AR metric alone. This suggests that the degradation cases identified by combining the 5 images may indeed be more representative. It also emphasizes the potential benefits of applying enhanced clustering algorithms, which can further enhance the stability and representativeness of our SEAL. framework.
Table 7: Benchmark results and ranking of GAN-based real-SR methods in LPIPS by proposed SEAL with five reference images. The subscript denotes the rank order. x means the model fails in majority of degradation cases.
Models
LPIPS-SE SE
ART
RPR; 1
RPRA 1
RPR t
Rank
ESRGAN
0.6152
0.01
0.01
0.73
0.03
X
RealSRGAN
0.5180
0.02
0.16
0.55
0.15
DASR
0.5228
0.04
0.13
0.59
0.13
BSRGAN
0,4809
0.52(s)
0.33(3)
0.69
0.29
MMRealSR
0.4777 (2)
0.75(2)
0.10
0.59
0.41
SwinIR
0.4672
0.86(1)
0.19(2)
0.71
0.28
13

---

Published as a conference paper at ICLR 2024
A.3 USER- CUSTOMIZED SE TEST SETS
In order to accommodate varying user preferences, such as the analysis of the quantitative perfor- mance of IQA metrics, the SE test sets organized in ascending order based on the PSNR values of the FSRCNN-mz output. These sets are then partitioned into five groups of equal size. Group I encompasses the most challenging cases, while Group includes the least challenging ones. As shown in Table g the average RPR value of BSRNet closely matches that of RealESRNet-GD. How- ever, there a variation in their performance across different groups. RealESRNet-GG outperforms in groups {3, 4, 5), whereas BSRNet takes the lead in groups {1. 2).
Table 8: RPR value of different methods on Set14-SE in PSNR. Blue: better than FSRCNN-
Model
SRResNet
DASR
BSRNet
RealESRNet
RDSR
RealESRNet-GD
SwinIR
Group
0.03
0.03
0.64
0.37
0.26
0.34
0.48
Group 2
0.02
0.02
0.60
0.37
0.21
0.45
0.43
Group 3
0.07
0.07
0.51
0.42
0.31
0.57
0.40
Group 4
0.02
0.01
0.37
0.40
0,29
0.68
0.27
Group 5
0.03
0.44
0.36
0.17
0.55
0.36
Average
0.03
0.51
0.38
0.25
0.52
FSRCNN-ma - 0.4M
(train/est time: 167b/10.7s)
SRReaNdt-mz
(trainkest time: 183b/15.66)
RRDBNNt-mzz 16.7M
PSNR (dB)
(traimkest time: 10088/81194
25
75
100
Test dataset
Figure 10: Comparison of network structures for the acceptance and excellence lines.
A.4 EXTENSION ON ACCEPTANCE LINE AND EXCELLENCE LINE.
For the acceptance line, we hope can represent an acceptable lower bound of performance with good discrimination for different models. Concretely, the acceptance line cannot be high that AR of most methods cannot exceed O, nor can it be so low that AR can easily reach 1.0. FSRCNN (Dong et al. 2016) is small network (0.4M Params.) while can distinguish the performance difference well, as shown in Tab. 1 Therefore, we choose FSRCNN-ma as the acceptance line.
For the excellence line, compare the networks of SetesNet (Leiig al. 2017) (1.5M Params.) and RRDBNet (Wang et al. (018) (16.7M Params.). Fig. T0 we observe that SRResNet-mz and FSRCNN-mz can already distinguish the performance difference. Although RRDBNet-mz exhibits a slight performance improvement, it comes at the expense of increased training and testing time, far surpassing those of other models. Considering the trade-off between performance and time costs, we choose SRResNet-mi mz as the excellence line. Nonetheless, we emphasize that our rationale for choosing these two lines that they can well differentiate the methods for comparison. Note that the two lines can be changed flexibly to meet specific requirements of other scenarios.
A.5 TIME COST OF OUR EVALUATION FRAMEWORK
We provide information on time cost in Tab. As expected, our approach does result in an increase in inference time, which scales linearly with the number of identified representative degradation cases (e.g.. 100 in our experiments). However, since the inference time remains within acceptable limits, believe this is worthwhile tradeoff between evaluation efficiency and quality.
14

---

Published as a conference paper at ICLR 2024
Table Comparison of the time cost between the conventional evaluation method and our approach
using RRDBNet (Zhagg et al. 2021 Wang et al. 2021b).
inference time
PSNR run-time
AR, RPR run-time
Sct14
4.22
0.52
0.013
Setl4-SE (ours)
382.74
49.47
0.013
Table 10: Basic strategies for model comparison. We use three basic strategies to compare the overall performance of real-SR models. The real-SR models trained (1) different network structures, (2) different training datasets, and (3) the RealESRGAN degradation model with different gate probability as proposed in Zhang al. 2022).
RPR; 4
RPRA f
RPRy t
Rank
SRResNet (1.5)
0.12
0.20
RCAN (15.6)
0.37,2)
0.15;;
0.62:2)
0.39ga)
2
Network (Parameter [M)
RRDBNet (16.7)
0.37,2)
0.33:s;
0.68.1)
(.32
3
SwinIR (11.9)
0.67(1)
0.15
0.62
0.41())
DIV2K
0.32ga)
0.25
0.64r2)
0,33
Training dataset
DF2K
0.43
0.24:2)
(.67(1)
0.39
ImageNet
0.63
0.22
0.67(1)
0.41
1.00
0,37,a)
0.33m)
0.6 0.68,3)
0.32
3
0.75
0,44.1)
0.35
0.69
0,34
Gate probability
0.43
0.35
0.70
0,3
0.25
0,40,3)
0,43;:
0.66.4)
0.21
BSR (SOTA)
0.59
0.42,a:
0.72(1)
0.27
2
SwinIR-GD-I (Ours)
0.85
0.25
0.72
0.40.1)
B DEVELOPING NEW STRONG REAL-SR MODELS
According to the evaluation results by our framework, as shown in Tab. TO we can improve the real-SR performance in three aspects to develop stronger real-SR model: 1) A powerful back- bone is vital for overall performance. We can observe that SwinIR obtains the highest AR and the lowest RPR,, 2) Using large-scale dataset (i.e., ImageNet (Deng et al.; 2009))
also greatly improve the real-SR performance. 3) A degradation model with the appropriate distribution (i.e., gate probability: 0.75 (Zhang al.) 2022.) also has non-negligible impact on the real-SR performance. Based on these observations, we use SwinIR as the backbone to train a new strong real-SR model ImageNet with high-order gate degradation (GD) model (gate probability: 0.75), denoted as SwinlR-Gii -D-I. The evaluation results in Tab, I0 show that SwinIR-GD GD-I obtain signif- icant improvement over the SOTA performance of BSRNet. Fig. IT shows that the visual results of SwinlR-Gin I are obviously better than BSRNet and SwinIR. We believe our framework would inspire more powerful real-SR methods in the future.
BSRNee
SwinR
SwalR GD
BSRNNe
Swinli
Swinli GDA1 G
ASRNNe
SwinlR
SwiiR-GGD-I
RSRNet
SwinIR
SwinlR-GPMI
DSRNet
SwinlR
SwinlR GD
DSRNet
SwintR
SwinlR GD
Figure 11: Visual results of the proposed baseline SwinIR-GD-I and other real-SR methods.
15

---

Published as a conference paper at ICLR 2024
C LIMITATIONS OF THE CONVENTIONAL EVALUATION METHOD
To further demonstrate the limitations of the traditional evaluation method, we generated 20 random test sets for comprehensive analysis. Using a large degradation model, we randomly apply degra- dations to the images from the DIV2K_ __al dataset. As shown in Fig. [2] each bar represents the PSNR difference between BSRNet and RealESRNet on single test set. Across these 20 test sets, BSRNet and RealESRNet each outperform the other in approximately half of the cases, with their performance gap ranging from -0.22 to 0.18. It is difficult to definitively determine which method is superior. This observation further highlights the inadequacy of using randomized test sets for evaluation.
0.2
0.1
0.0
-0.
20
Blu33.0
Nosc20=Bluu30
JPEGTO+Bluu33
Figure 12: Results of PSNR distance between BSR- Net and RealESRNet on 20 randomly generated test sets, sorted in ascending order.
Figure 13: Similar visual effects of different degradation combinations.
D DETAILS OF DEGRADATION CLUSTERING
D.1 SPECTRAL CLUSTERING
We use the shuffled degradation model of BSRGAN Zhang al. 2021 and the high-order degra- dation model of RealESRGAN (Wang et al., 2021b) to construct a large degradation space. The degraded images
generated by the shuffled (Zhang al. 2021 and high-order (Wang et al.) 2021b) degradation models with probabilities of (0.5, 0.5) The degradation types mainly consist of Gaussian, Poisson, and Speckle noise with gray and color scale; 3) multiple resize strategy: area, bilinear and bicubic; 4) JPEG noise.
1) various types of Gaussian blur; 2) commonly- noise:
Algorithm Image degradation clustering
Input: Similarity matrix S E R= number of clusters to construct.
1: Compute Adjacency Matrix W and Degree Matrix D.
2: Compute Laplacian Matrix L -= D W.
3: Compute the first K eigenvectors u1, Ug of L.
4: Let U E ng be the matrix containing the vectors ui,... Uy as columns.
5: For i 1, ..., n., let yi E & be the vector corresponding the i-th row of U.
6: Cluster the points (yi)i=1.....n in R* with the k-means algorithm into clusters C₁ Ck
Output: Cluster centers C1, CK with C₁ E C.
We use spectral clustering to group the degraded images (x) due to its effectiveness and flexibility in finding arbitrarily shaped clusters. First, we use the RGB histogram (h) (Tang al. 2011 Ye & Doermann 2012) with 768 values (bins) as the image feature to calculate the similarity &ij [h(xz)) (x,;)) The histograms of R, G. and B are generated separately and then concatenated into a single vector to represent the degraded image. The similarity matrix is defined matrix S. where Siy represents a measure the similarity between data points x with indices i and j for data points. We execute Algo. D step by step to obtain the degradation parameter of cluster centers D {c1, C2.,, Then, we use the degradation parameter of cluster centers as the representative degradations construct the systematic set.
symmetric
16

---

Published as a conference paper at ICLR 2024
D.2 SIMILARITY METRICS
In this section, we provide more experimental details for the Sec. similarity metrics in the main paper. To select an appropriate similarity metric, we create two datasets with simple degradation types Gaussian blur with a range of [0.1, 4.0] and Gaussian noise [1. 40]. We use the image lenna Sct14 Zeydee al. 2010) as our Ground-Trute image. Firstly, we generate 100 low-quality images named Blur100 by applying Gaussian blur within a range of [0.1, 4.0). Each cluster is assigned label based on the degradation intensity. We label the low-quality images with {[0.1, 1.0]. [1.0, 2.0], [2.0, 3.0]. [3.0. 4.0]} as {1. 2, 3, 4) respectively. Similarly, we generate 100 low- quality images named Noise 100 by applying Gaussian noise within the range [1, 40J, labeled as {1, 2, 3, 4) based noise intensity.
To evaluate the effectiveness of the similarity metric, we combine Blur100 and Noise 100 to produce BN100, which comprises 100 blurred images and 100 noised images. BN100 is labeled {1, 2, 3, 4, 5, 6, 7, 8) using the same criteria as the previous datasets. Purity (Schütze et al., 2008) is an external criterion (similar NMI, F measure (Schütze al. (0088) that assesses the alignment of the clustering outcome with the actual, known classes. It is defined as:
1 Σ N
Purity
max lcs ntyl:
(11)
where N is the total number of data points, fci, ....... cJ} is the set of classes. To compute purity. each cluster is assigned to the class that appears most frequently within that cluster. The accuracy of this assignment then evaluated by the number of correctly assigned points divided by the total number of points.
{wi {wi,w2 ............
... Wik } is the of clusters and C
(a) 1:5 s0.12
(b) 140 s0.04
(c) 100 s0.03
(d) A:00 s0.01
Figure 14: Ablation of the number of clusters k. s denotes silhouette score.
D.3 THE NUMBER OF CLUSTERS
To determine the number of clusters, we use silhouette scores (Rousseeww 1987) to measure the quality of the clusters. higher silhouette score represents better cluster, while the clustering result is acceptable if the silhouette score is greater than O. As demonstrated in Fig. [4 the silhouette scores of 4=40 and 100 are very close, thus we utilize 100 clusters to find the representative cases.
E MORE EXPERIMENTAL RESULTS
E. MORE VISUAL RESULTS ON REAL-SR METHODS
In this section, we further explore the effectiveness of our evaluation framework by providing addi- tional qualitative results. We compare proposed lines of acceptance and excellence with existing real-SR methods. The MSE-based methods that we consider include DASR (Wang et al. 2021a), BSRNet (Zhang et al. (021). SwinIR (Liang et al. 202 RealESRNet (Wang et al. 202 1b RDSR (Kong et al.] (022), and RealESRNet- (Zhang et al. 2022). In Fig. [5] we use FSRCNN (green) denote the acceptance line, and SeResNet (red) represent the excellence line. Moving to
17

---

Published as a conference paper at ICLR 2024
020
PAMAN 25.69
noM
25ANNI
25-31
BGAN 25.37
Raw EdbMee
BralSSpNet-CD
D1M
ROSA 19
Pooa EStter 50
RealEsabor 19.76
FAROAN 20110
SwhR 20.31
RSAbor 20.32
StiteWett
nzM
NOSA 11
Toa Eihre 22.33
PaalEsMNett 6O 32.29
25Ahet 33.01
SahRt
FSROW 33.17
StRusNe 3355
nout
RealEShhet 21.00
RO5B 21,00
BealtShhes.co Co 22.17
Swirth 22.50
PSACHN 22.54
25tAiG 22.35
StRapNet 22.97
Figure 15: Visual results of MSF-based real-SR methods and the acceptance line FSRCNN and excellence line SRResNet with PSNR metric. It is best viewed in color.
the GAN-based methods, we include DASR (Liang al. 2022). BSRGAN (Zhang et al. 202
MMRealSR (Mou et al. 2022). SwinIR (Liang et al. 2021a) and RealSRGAN (Ledig et al. In Fig. 16 RealESRGAN (green) is used to denote the acceptance line, and RealHATGAN (red)
2017
used to represent the excellence line. This comprehensive comparison provides clear understand- ing of the performance of our proposed lines against the existing methods. thereby demonstrating the effectiveness of our evaluation framework.
18

---

Published as a conference paper at ICLR 2024
Inout
DASS 295
0.377
AralFSSGAN AralH
Input
DA55, 0 O97
DAIIFRSSAN O 092
Des FORSAAN 0.774
104Ba1/5R..075 750
Ro BATTAN 755
SATIRR 5457
BERGAN 527
DASA 526
ReaIERGAN
BSRGAN 460
ESRIGN
R
NiRRAa
ReSPHATGAN
Inout
DASS
RealFRGAN O
BARGAN
ESRGAN 0.514
AralHoTiaN
Incut
ReaISRGEN 0.52
DASR.0.115
MIRRA SR.0 55.0-55
BSRGAN 3,792
BeNEESGGN 2.20
Swin
ReajHATOWN 200
SeafERICEN
WNilaa 0.614
BSSIGNN
BANEECRGAN
Visual results of GAN-based real-SR methods and the acceptance line RealESRGAN and excellence line RealHATGAN with LPIPS metric. best viewed color.
Figure 16:
E.2 VISUAL RESULTS OF DEGRADATION CLUSTERING
We provide the visual comparison of samples both between and within clusters. Fig. 17 we randomly selected five clusters out of total of 100 clusters and chose six samples from each cluster. We observed that samples within each cluster exhibit similar degradation patterns, indicating they share comparable level of restoration difficulty. In contrast, samples belonging to different clusters display remarkably distinct degradation patterns. In addition, we present the visualization of the degradation cluster centers in Fig. 18 Fig. 19
Fig. 20 Fig. 21 and Fig. 22 The cluster centers
sorted based on the PSNR value the output of FSRCNN-mz. The results demonstrate that the degradation clustering algorithm performed expected.
19

---

Published as a conference paper at ICLR 2024
Cluster
Cluster 2
Cluster
Claster 4
Clister
Figure 17: Visualization of samples from five randomly selected clusters. Best viewed in color
2
4
12
17
19
Figure 18: Visualization of the degradation cluster centers [1, 20), sorted by the PSNR of the output of FSRCNN-mZ. Best viewed in color.
20

---

Published as a conference paper at ICLR 2024
21
22
23
24
25
29
30
31
32
33
34
35
Figure 19: Visualization of the degradation cluster centers [21, 40J. sorted by the PSNR of the output of FSRCNN-mz. Best viewed in color.
41
42
43
44
45
46
49
51
54
56
57
58
59
60
Figure 20: Visualization of the degradation cluster centers [41, 60), sorted by the PSNR of the output of FSRCNN-mz Best viewed in color.
21

---

Published as a conference paper at ICLR 2024
66
67
68
69
70
71
73
74
75
78
79
80
Figure Visualization the degradation cluster centers [61, 80), sorted by the PSNR of the output of FSRCNN-mz. Best viewed in color.
86
87
88
89
90
91
93
94
95
99
100
Figure 22: Visualization of the degradation cluster centers [81, 100], sorted by the PSNR of the output of FSRCNN-mZ. Best viewed in color.
22


Reviews:
[{"review_id": "k0W0oPJcDS", "reviewer": "Reviewer_xKgq", "summary": "The paper presents a framework for evaluating blind single-image super-resolution techniques against multiple different degradation types.  The process generates a large random sampling of degradations and then clusters the resulting images based on histograms. Each cluster represents functionally similar degradations, regardless of the process of obtaining them. The authors can then generate a diverse and comprehensive test set by sampling from the degradations represented by each cluster. \n\nThe authors also use a fully automated method of evaluating new networks relative to two existing networks that play the role of baseline (acceptable) and best-in-class (excellent). By evaluating each test image relative to these two networks, they can provide more meaningful statistics about the robustness of each network to the range of potential degradations.", "questions": "Why use PSNR as the basis for whether an image is Acceptable?  Why not SSIM, perceptual distance, or some combination of the three?  Do you get different rankings with different metrics under this evaluation system?\n\nThe histograms used in the clustering are histograms of what?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses an issue in blind single-image super-resolution that has not yet been fully solved: how to effectively evaluate performance. It is clear from prior work that PSNR is insufficient, since it does not always agree with human perception.  SSIM and perceptual distance provide potentially more informative metrics, but using aggregate averages does not convey strength across different types of degradation.\n\nThe use of other networks to provide baseline (acceptable) and state-of-the-art (excellent) performance is an interesting approach. One downside is the continued reliance on PSNR, which has a hard time capturing fine textures or visual structures.\n\nThe clustering-based approach to identifying diverse degradations is an interesting idea.", "weaknesses": "The authors use histograms as the basis for clustering degradations. However, the form of the histograms does not appear to be given in the paper (sections 3.2, 3.3).  Are these histograms of r, g, and b separately?  Is this a 2-D chromaticity histogram, a 3-D RGB histogram, or a set of histograms of filter bank outputs?  It's not clear that all of these would be equally effective at capturing diverse degradations.\n\nThe sole use of PSNR as a way to rank-order results may not be the best choice.  PSNR rankings don't always reflect human rankings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h7dkbYRw4E", "reviewer": "Reviewer_eBzL", "summary": "This paper revisits real-world super-resolution evaluation from a distributional viewpoint, integrating various representative degradation types into the test sets. Previous works only conduct evaluation using average performance on a small set of degradation cases randomly selected from a large space, which often yields biased results. To address this issue, the authors adopt a simple yet effective degradation clustering strategy to select representative degradation. Then, they present an evaluation protocol with two new model-based metrics suitable for real-SR tasks. The experimental results offer a new perspective on real-SR evaluation through the proposed SEAL systematic evaluation framework.", "questions": "1.  Figure 3 presents a coarse-to-fine evaluation protocol. Initially, the authors utilize AR to validate the effectiveness of the real-SR model in comparison to the acceptance line. Subsequently, they employ coarse metrics to rank the real-SR model. However, actual evaluations may be contingent on user requirements. Could the sequence of fine-grained indicators possibly be adjusted to meet specific needs?\n\n2.  Why is it that the evaluation based on MSE-based real-SR employs multiple smaller models as reference lines, whereas the GAN-based real-SR utilizes a single model as reference lines?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1.It is reasonable and interesting to use representative degradation to construct test set for real-SR evaluation.\n\n2.  The authors employ a straightforward yet effective strategy (spectral clustering) for degradation clustering. This approach is not only easy to comprehend, but it also offers potential for further extensions.\n\n3.  The systematic test set offers flexible customization options. \n\n4.  The introduction of new evaluation metrics provides a fresh perspective for real-SR evaluations. These metrics can also complement existing ones, thereby enriching the evaluation strategy.\n\n5.The paper is well-written, and its main idea is easy to follow.", "weaknesses": "1.  In Figure 1, the author compares the average performance of two randomly selected test sets for conventional evaluation. If evaluations were conducted on multiple randomized test sets, it could potentially provide a more comprehensive understanding of the model’s performance. It would allow for the observation of performance trends across different test sets and offer insights into the model’s consistency and reliability.\n\n2.  Utilizing a hundred test sets can indeed offer substantial reference points for real-SR evaluation. However, this approach may lead to an increase in the inference time required for evaluation. It would be beneficial for the author to compare the time cost of the existing evaluation method with that of the proposed SEAL evaluation to provide a more comprehensive understanding of their efficiency.\n\n3.  A more detailed explanation of the spectral clustering process, as outlined in Section 3.2, would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k7t2mmvu1u", "reviewer": "Reviewer_JzYv", "summary": "In this paper, the authors proposed a new evaluation framework for the evaluation of real-world super-resolution (SR) methods. They first used a clustering-based approach to model a large degradation space, and then designed two new evaluation metrics to assess real-SR models on representative degradation cases. The authors benchmarked existing real-SR methods with the proposed evaluation protocol and presented new observations and insights.", "questions": "In Section 5.4, what is the purity accuracy defined as? More details should be explained.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors handle the issue of fair and comprehensive evaluation of real-SR methods, which can facilitate the development of real-world SR.", "weaknesses": "1. In Introduction, the authors argued that random selection may cause significant bias and randomness to the evaluation results. From my point of view, random selection will not cause bias because the posibility of choosing each degradation is identical. The authors are suggested to check this claim to avoid misunderstanding. Same issues exist in Section 5.3.\n\n2. In Section 3.1, there may be a mistake in the calculation of the total combination of degradations. Since the order of degradation types can be switched, the total degradation combination for $s$ degradation type and $k$ degradation levels should be $A^{s}_{s} \\times k^{s}$.\n\n3. In Section 3.2, the authors claimed that different combinations of degradation types may have similar visual quality and restoration defficulty. Consequently, the authors designed a clustering-based approach to handle this problem. However, this approach seems intuitive, and experiments should be conducted to validate whether the clustering works as expected, i.e., whether the examples within a cluster have similar visual quality and restoration defficulty whereas the examples in different clusters differ significantly. Only using Fig. 13 for demonstration is not enough.\n\n4. The authors only used one image (i.e., lenna) to generate 100 degradation parameters. It can be observed in Fig. 9 that the image content can affect the final results (0.1-0.3 dB in PSNR). Consequently, have the authors considered using more images to generate more stable and representative clustering centers?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4D25BaM1vM", "reviewer": "Reviewer_p1DW", "summary": "This work aims to come up with a benchmarking approach to evaluate real SR methods in literature. To this end authors propose to use Acceptance Rate (AR) for coarse grained comparisons and Relative Performance Ratio (RPR) based measures for fine-grained analysis. The motivation and the impact of the proposed metrics are clearly demonstrated with interesting findings on why the previous approaches are misguiding and how the proposed method can alleviate the associated problems.", "questions": "Please address the comments under weaknesses", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem tackled in this work has significant practical impact.\n2. The proposals are intuitive and supported by experimental evidences.\n3. The proposed method comes up with a new and more reliable benchmarking approach for evaluating real-SR methods.", "weaknesses": "The reliability of clustering algorithms is measured using purity accuracy.\n1. The definition of purity accuracy is fuzzy and will need better explanation.\n2. I would advise to demonstrate via visual comparisons that the clusters formed with the proposed method indeed are meaningful to answer the following questions - does a bunch of randomly sampled images from a given cluster look degraded by a comparable amount? Does, random samples from different clusters exhibit contrasting degradation levels?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)