Efficient Test-Time Adaptation for Super-Resolution with Second-Order Degradation and Reconstruction
Authors: Zeshuai Deng
Zhuokun Chen
Shuaicheng Niu
Thomas H. Li
Bohan Zhuang
Mingkui Tan
Venue: NeurIPS 2023 poster (2023)
Keywords: Image Super-resolution
Test-time Adaptation
Self-supervised Learning
Second-Order Degradation
TL;DR: we propose an efficient test-time adaptation method to alleviate the degradation shift issue for image super-resolution.

Abstract: Image super-resolution (SR) aims to learn a mapping from low-resolution (LR) to high-resolution (HR) using paired HR-LR training images. Conventional SR methods typically gather the paired training data by synthesizing LR images from HR images using a predetermined degradation model, e.g., Bicubic down-sampling. However, the realistic degradation type of test images may mismatch with the training-time degradation type due to the dynamic changes of the real-world scenarios, resulting in inferior-quality SR images. To address this, existing methods attempt to estimate the degradation model and train an image-specific model, which, however, is quite time-consuming and impracticable to handle rapidly changing domain shifts. Moreover, these methods largely concentrate on the estimation of one degradation type (e.g., blur degradation), overlooking other degradation types like noise and JPEG in real-world test-time scenarios, thus limiting their practicality. To tackle these problems, we present an efficient test-time adaptation framework for SR, named SRTTA, which is able to quickly adapt SR models to test domains with different/unknown degradation types. Specifically, we design a second-order degradation scheme to construct paired data based on the degradation type of the test image, which is predicted by a pre-trained degradation classifier. Then, we adapt the SR model by implementing feature-level reconstruction learning from the initial test image to its second-order degraded counterparts, which helps the SR model generate plausible HR images. Extensive experiments are conducted on newly synthesized corrupted DIV2K datasets with 8 different degradations and several real-world datasets, demonstrating that our SRTTA framework achieves an impressive improvement over existing methods with satisfying speed. The source code is available at https://github.com/DengZeshuai/SRTTA.

Efficient Test-Time Adaptation for Super-Resolution with Second-Order Degradation and Reconstruction
Zeshuai Deng
Zhuokun Chen 12+ Shuaicheng Niu Thomas Li*
Bohan Zhuang2t Mingkui Tan 244
South China University of Technology. "Pazhou Lab, 'ZIP Lab, Monash University, "Key Laboratory of Big Data and Intelligent Robot, Ministry of Education, Pekknng University Shenzhen Graduate School
Abstract
Image super-resolution (SR) aims to learn a mapping from low-resolution (LR) high-resolution (HR) using paired HR-LR training images. Conventional SR methods typically gather the paired training data by synthesizing LR images from HR images using a predetermined degradation model, e.g., Bicubic down-sampling, However, the realistic degradation type of test images may mismatch with the training-tine -imeingg-tteeeeee degradation type due to the dynamic changes of the real-world sce- narios, resulting in inferior-quality SR images. To address this, existing methods attempt estimate the degradation model and train an apage-specific model, which, however, is quite time-consuming and impracticable to handle rapidly changing do- main shifts. Moreover, these methods largely concentrate on the estimation of one degradation type (e.g., blur degradation). overlooking other degradation types like noise and JPEG in real-world test-time scenarios, thus limiting their practicality. To tackle these problems, we present efficient test-time adaptation framework for SR, named SRTTA, which is able to quickly adapt SR models to test domains with different/unknown degradation types. Specifically, we design second-order degradation scheme to construct paired data based on the degradation type of the test image, which is predicted by pre-trained degradation classifier. Then, adapt the SR model by implementing feature-leve reconstruction learning from the initial test image to its second-order degraded counterparts, which helps the SR model generate plausible HR images. Extensive experiments are conducted newly synthesized corrupted DIV2K datasets with 8 different degradations and several real-world datasets, demonstrating that our SRTTA framework achieves an impressive improvement over existing methods with satisfying speed. The source code is available at https://github.com www..ooogeneessauaSSTTTTTTT
Introduction
Image super-resolution (SR) aims reconstruct plausible high-resolution (HR) images from the given low-resolution (LR) images, which is widely applied in microscopy [42, 44], remote sensing [17, 35] and surveillance [40, 56]. Most previous SR methods [11, 31, 60, 8] hypothesize that the LR images are downsampled from HR images using a predefined degradation model, e.g., Bicubic swmmlsam However, due to the diverse imaging sensors and multiple propagations on the Internet, real-world images may contain different degradation types (e.g., Gaussian blur, Poisson noise, and JPEG artifact) [32, 52, 53, 25]. Besides, the realistic degradations of real-world images
"qqual contribution. Email: sedengzeshuai @ mail.scuteddu (caesard216, niushuaicheng gmmail.com Corresponding author. Email: mingkuitan @ scut.edu.cn. bohan, @ gmail.com
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

may dynamically change, which are often different from the training one, limiting the performance of pre-trained SR models in dynamically changing test-time environments.
Recently, zero-shot SR methods [43, 9, 12] are proposed to train an image-specific SR model for each test image to alleviate the degradation shift issue. For example, 7SSR [43] uses a predefined/estimated degradation model to generate an image with a lower resolution for each test image. With this downsampled image and the test image, they can train an image-specific SR model to super-resolve the test image. Moreover, DualSR [12] estimates the degradation model and trains the SR model simultaneously to achieve better performance. However, these methods usually require thousands of iterations to estimate the degradation model or train the SR model, which is very time-consuu consuming. Thus, these methods cannot handle real-world test images with rapidly changing domain shifts.
To reduce the inference time of zero-shot methods, some recent works [45, 41] introduce meta- feeefiiee/ett
learning [13] to accelerate the adaptation of the SR model, which still requires degradation model to construct paired data to update the model. However, most degradation estimation methods [2, 30] focus on the estimation of one degradation type, which limits the adaptation of SR models to test images with other degradations. Recently, test-time adaptation (TTA) methods [46, 47, 57, 38, 48, 39] are proposed to quickly adapt the pre-trained model to the test data in target domain without accessing any source training data. These methods often use simple augmentation operations (e.g., rotation or horizontal flip) on the test image, and construct the pseudo label as the average of the predicted results [57, 48]. For image SR, the pseudo- -R image constructed using this scheme [57, 48] may still contain the degradation close to the test image (e.g.. Gaussian blur). With such a pseudo-HR image, the adapted SR model may not be able to learn how to remove the degradation from the test image (see results in Table 1). Therefore, how to quickly and effectively construct the paired data to encourage the SR model to remove the degradation is still an open question.
In this paper, we propose a super-resolution test-time adaptation framework (SRTTA) adapt trained super-resolution model to target domains with unknown degradations, as shown in Figure 1. When the degradation shift issue occurs, the key challenge is how to quickly and effectively construct (pseudo) paired data to adapt SR models to the target domain without accessing any clean HR images. To this end, we propose a second-order degradation scheme to construct (pseudo) paired data. Specifically, with pre-trained degradation classifier, we quickly identify the degradation type from the test images and randomly generate a set of degradations to obtain the second-order degraded images. The paired data, which consists of the second-order degraded images and the test image, enables rapid adaptation of SR models to the target domain with different degradations. To facilitate the learning of reconstruction, we design second-order reconstruction loss to adapt the pre- trainee model using the paired data in self-supervised manner. After fast adaptation using our method, the SR model is able to learn how to remove this kind of degradations and generate plausible HR images. Moreover, we also design an adaptive parameter preservation strategy to preserve the knowledge of the pre-trained model to avoid the catastrophic forgetting issue in long-term adaptation. Last but not least, we use eight different degradations to construct two new benchmarks, named DIV2K-C and DIV2K-MC. to comprehensively evaluate the practicality of our method. Experimental results on both our synthesized datasets and several real-world datasets demonstrate that our SRTTA is able to quickly adapt the SR model to the test-time images and achieve impressive improvement.
Our main contributions are summarized as follows:
A novel test-time adaptation framework for image super-resolution: We propose a super- resolution test-time adaptation (SRTTA) framework to adapt any pre-trained SR models to different target domains during the test time. Without accessing any ground-truth HR images, our SRTTA is applicable to practical scenarios with unknown degradation in self-supervised manner.
A fast data construction scheme with second-order degradation: We use a pre-trained classifier to identify the degradation type for a test image and construct the paired data using our second-order degradation scheme. Since we do not estimate the parameters the degradation model, our scheme enables rapid model adaptation to wide range of degradation shifts.
New test datasets with eight different domains: We construct new test datasets named DIV2K-C C and DMC which contain eight common degradations, to evaluate the practicality of different SR methods. Experimental results on both synthesized datasets and real-world datasets demonstrate the superiority of our SRTTA, e.g., 0.84 dB PSNR improvement on DIV2K-C over ZSSR [43].

---

2 Related Work
Real-world super-resolution To alleviate the domain shift issues, GAN-based methods [54, 5, 24, 36] tend to learn the degradation model of the real-world images in the training stage. These methods often train generator that explicitly learns the degradation model of real-world images. Besides, some methods [55, 60, 49] try to enumerate most of the degradation models that can be encountered in real-world applications. Based on the estimated/predefinel degradation models, these methods can generate LR images whose distribution is similar to weal-world images. However, due to the complex and unknown processing of real-world images, it is hard to mimic all types of degradation during the training phase. Instead, some existing methods [15, 2, 22, 21, 34] try to estimate the image-specific degradation model during the test time, which helps to reconstruct more plausible HR images. For instance, optimization-ball based methods [15, 22, 21] estimate the blur kernel and SR image together in an iterative manner. However, these methods cannot generate satisfactory results when the test images contain different types of degradation (e.g., Poisson noise and JPEG artifact) [32]. Thus, these methods still suffer from the domain shift on test images with unknown degradation.
Zero-shot super-resolution. Zero-shot methods [43, 9, 45, 41, 12] aim to train an image-specific SR model for each test image to alleviate the domain shift issue. These methods [43, 12] use predefined/estimated degradation model to generate an image with a lower resolution from each test image. To estimate the image-specific degradation in zero-shot manner, KernelGAN [2] utilizes the internal statistics of each test image to learn the degradation model specifically and then uses ZSSR [43] to train an SR model with the estimated degradation. However, these methods usually require lot of time to estimate the degradation model or train the SR model. MZSR [45] and MLSR [41] are proposed reduce the number of iteration steps for each test image during test time. Recently, DDNM [51] was proposed to use pre-trained diffusion model to ensure the generated images obey the distribution of natural images. However, these methods still require a predefined (Bicubic downsampling) or an estimated degradation model. The predefined Bicubic degradation suffers from the domain shift due to its difference from the underlying degradation of real-world images. The estimation methods [2, 30] may focus on the estimation of a single degradation type (e.g., blur) while ignoring other degradation. Thus, these methods often result in unsatisfactory HR images for the test images with different degradation types [32]. In this paper, we use a degradation classifier to quickly recognize the degradation type and randomly generate the degradations with this type. Therefore, we do not need to estimate the degradation model, which is time-saving.
Test-time adaptation. Recently, test-time adaptation (TTA) methods [46, 47, 57, 38, 48, 6, 39, 59] have been proposed to alleviate the domain shift issue by online updating the pre-trained model on the test data. TTT [46] uses an auxiliary head to learn the test image information from the self superviee task. Tent [47] proposed to adapt the pre-trained model with entropy-based loss in an unsupervised manner. COTTA [48] uses a weight-averaged pseudo-label over training steps to guide the pre-trained model adaptation. However, these methods are mainly developed for image classification and may ignore the characteristics of image super-resolution. Thus, these methods may not be effective in adapting the SR model to remove the degradation from the test image. In this paper, we focus on the image SR task and address the degradation shift issue with our SRTTA framework.
3 Preliminary and Problem Definition
Notation. Without loss of generality, y y be a clean high-resolution (HR) image and x. be the clean low-resolution (LR) image downsampled from y using Bicubic interpolation, i.e., x. = y 4s. where 8 is the scale factor of Bicubic downsampling. Let x denote a real-world test image degraded from y, i.e., x D(y), where D--) is the degradation process. We use x,d to denote the LR image that is further degraded from the real-world image x. In this paper, we call the test image x as a first-order degraded image and the image X degraded from super-resolution (SR) model with parameters 0.
second-order degraded image. fo(-) is
Image degradation. The degradation process of real-world test images can be modeled by classical degradation model D(-) [33, 49], Formally, let be a blur kernel, n be an additive noise map and be the quality factor of PEG compression, the degraded image x is defined by
x = D(y) [[y @ k) 4. +n]JPEG, where @ denotes the convolution operation, 4s denotes the downsampling with scale factor of 8, and J JEE,, denotes the JPEG compression with the quality factor q. Similarly, the second-order
(1)
3

---

Pretrained SR Model (
Second-order Degradation
C(-)
C(X)
Corw
ResHBock
Can
Upearple
Com
(x)
clean
Test Sample
Degradation Classifier
blur
x 2
nolse
+
Forward
JPEG
JPEG,
Adaptive Parameter Preservation
Backward
no (Kea)
Frozen W Weights
A daptation Consistency L oss
Xsd
Cur
ResHblokk
ResTBlock
Con
Upsmpple
Corr
Self-supervised Adaptation Loss
1x())
X
SR Model fo(-) for TTA
Figure An overall illustration of the proposed super-resolution test-time adaptation (SRTTA) framework. Given test image x, we pre-trained degradation classifier predict the degradation type C(x), e.g., blur, noise, and JPEG degradation. Based on the predicted degradation type ((x), construct set of paired data {x } and adapt the SR model with our adaptation loss La and L. When test samples are clean images, we directly use the frozen pre-trained SR model to super-resolve these clean images without adaptation.
degraded image X can be formulated as
X D(D(y)) D(x) = [(x k) 4a +n].PEG,
Degradation shift between training and testing. Existing SR methods [31, 60, 16] often construct paired HR-LR training images by either collecting from the real world or synthesizing LR images from HR images via pre-defined degradation model, i.e., Bicubic down-sampling, However, due to diverse camera sensors and the unknown processing the Internet, the degradation process of real-world test images may differ from that training images, called domain shifts [27, 48, 32]. In these cases, the SR model often fails generate satisfactory HR images.
Motivation and challenges. Though recently some blind SR methods [12, 43] have been proposed to address the degradation shift issue, they still suffer from two key limitations: low efficiency and narrow focus on single degradation type, e.g., blur degradation. In this work, we seek to resolve these issues by directly learning from the shifted testing LR image at test time, which poses two major challenges: 1) How to quickly and effectively construct the (pseudo) paired data adapt SR models to test domains with unknown degradations? and 2) How to design generalized test-time learning framework that facilitates the removal of various types of degradation, considering that we have only low-resolution test image our disposal?
4 Efficient Test-Time Adaptation for Image Super-Resolution
In this section, we illustrate our proposed super-resolution test-time adaptation (SRTTA) framework that is able to quickly adapt the pre-trained SR model to real-world images with different degradations. The overall framework and pipeline are shown in Figure and Algorithm 1.
Given image x, we first construct the paired data using our second-order degradation scheme. Specifically, we use pre-trained degradation classifier to recognize the degradation type for the test image. Based on the predicted degradation type, we randomly generate a of degradation (e.g., set of blur kernels k) and use them to construct set of paired data (x) x) X With the paired data, we adapt the pre-trained SR model to remove the degradation from the test image. Notably, before performing the test-time adaptation, we freeze the important parameters to preserve the knowledge of pre-trained model alleviate the forgetting problem long-term adaptation. After adaptation, use the adapted model to generate the corresponding HR image for the test image.

---

Algorithm 1: The pipeline of the proposed Super- Resolution Test-Times Adaptation.
Input:
Real-world test images {x+]== adaptation iteration steps S for each image, learning rate 7), batch size N, preservation ratio f.
1 Load the pretrained SR models foo and the degradation classifier C(-).
2 Select and freeze the important parameters using Eqn. (9) with p.
3 for X; in {x,}{= do for 8 in S). do
5
Construct paired data {x'd X )im based on C(x,) using Eqn. (3);
Adapt the SR model using Eqn. (6) with r);
end 8 end
Output: The adapted SR model fe, the predictions {&r fo(x))) for all x, in {x 1x
4.1
Adaptive Data Construction with Second-Order Degradation
In this part, we propose novel second-order degradation scheme to effectively construct paired data, enabling the fast adaptation of SR models to target domain with different degradations.
Unlike existing methods [43, 2, 30], we consider more degradation types and avoid estimating the degradation model. Existing methods [43, 2, 30] mainly focus on precisely estimating the blur kernels images (second-orgy -oooer degraded images), which is time- consuming. Instead, we use a pre-trained degradation classifier to quickly identify the degradation types (blur, noise, and JPEG) of test images, and then we construct the second-order degraded images based on the predicted degradation types. Without the time-consuming degradation estimation process, our scheme enables fast model adaptation to wide range of degradation shifts.
when constructing the lower-resolution
Adaptive data construction. In this part, design an adaptive data construction method obtain the second-orde degraded images x Specifically, based on the classical degradation model in Eqn. (I), we train multi-label degradation classifier C(-) predict the degradation types for each test image, including blur, noise and JPEG degradation types, denoted by Cn. and E {0, J. respectively. With the predicted degradation types, we randomly generate N degradations and construct a set of second-order degraded images (x 1imi which can be formulated as
X D(x, C(x)) D,(D, ca) + Dn(cn),
Do(x, cb) = oo +)
(x(G,) = n. (1-
((xx,c))=,,JPPCC)
where the blur kernel k, noise map and quality factor are randomly generated using similar recipe of Real-ESRGAN [49]. Unlike previous methods [43, 2], we do not further downsample the test image when constructing X.d. since the pretrained SR model has learned the upsampling function (the inverse function of downsampling) during the training phase. Due to the page limit, we more details in the supplementary materials.
Since the pre-trained SR model has been well-trained on the clean domain (Bicubic downsampling). we simply ignore adapting the clean images in test-time. For these images, we use the pre-trained SR model super-resolve them, i.e., y for (x) when C Gn Cj
4.2 Adaptation with Second-Ordas Reconstruction
In our SRTTA framework, design a self-supervised adaptation loss and adaptation consistency loss to update the pre-trained SR models to test images with degradation.
Self-supervised adaptation. To adapt the pre-trained model to remove the degradation, we design self-supervised adaptation loss based on the Charbonnier penalty function [4, 28]. Specifically, we
encourage the SR model to reconstruct the test images x from the second-order degraded images X the feature level, which can be formulated as
(x,, Xad) = (f((x) - 12 +
(4)
5

---

where fo(-) denotes the output features of the 1-th layer. We simply set f.(() ()) to be the output features of the second-to-last convolution layer. 6 is a small positive value that is set to 10-3 empirically.
Consistency maximization. To keep the model consistent across adaptation, we design an adaptation consistency loss to encourage the output of the adapted model to be close to that of the pre-trained model, which is formulated as
La X (ggo (x) (Xaa)) +
(5)
where (-) denotes the output features of the (-th layer of the pre-trained SR model.
Second-order reconstruction loss. Our final second-order reconstruction loss consists of self- supervised adaptation loss and an adaptation consistency loss, which is formulated as
L = (x, xsa) + a.ha x,a),
(6)
where a is balance hyperparameter (the ablation study can be found in Table 5).
4.3 Adaptive Parameter Preservation for Anti-Forgetting
To avoid catastrophic forgetting in long-term adaptation, we propose an adaptive parameter preserva- tion (APP) strategy to freeze the important parameters during adaptation. To select the important parameters, we evaluate the importance of each parameter using the Fisher information matrix [26, 38). Given set of collected clean images De, we design an augmentation consistency loss C. to compute the gradient of each parameter. Based on the gradient, we compute the diagonal Fisher information matrix to evaluate the importance of each parameter 0P, which formulated as
1
(x,( 30
w(09)
)
De.
X.EDe
(x(x,) = - (x) te, s.t. y
Σ ==1
(8)
where A, E {Aj}ÿ= is an augmentation operation, which is the random combination of a 90-degree rotation, horizontal and a vertical flip on the input image x.. R; is the inverse operation of A; that rolls back the image (xx)) to its original version x.. With ((f)), we select the most important parameters using a ratio of p and freeze these parameters during the adaptation. The set of selected parameters S can be formulated as
S (0) (w?)) > Tp: 00 80).
where t, denotes the first p-ratio largest value obtained by ranking the value ((?)), rameter to control the ratio of parameters to be frozen. Note that we only need to select the set of significant parameters S once before performing test-time adaptation.
> p is hyperpa-
5
Experiments
5.1 Experimental Details
Testing data. Following ImageNet-C [18], degraded 100 validation images from the DIV2K [1] dataset into eight domains. We select the eight degradation types that do not extremely change the image content, including Gaussian Blur, Defocus Blur, Glass Blur, Gaussian Noise, Poisson Noise (Shot Noise), Impulse Noise, Speckle Noise, and JPEG compression. In total, we create new benchmark dataset DIV2K-C, which contains 800 images with different single degradation, to evaluate the performance of different SR methods. Besides, we further construct a dataset named DIV2K-MC, which consists of four domains with mixed multiple degradations, including BlurNoise. BlurJPEG, NoiseJPEG, and BlurNoiseJPEGG Specifically. test images from the BlurNoise.PEG domain contain the combined degradation of Gaussian Blur, Gaussian Noise and JPEG simultaneously. Moreover, we also evaluate our SRTTA on real-world images from DPED [23]. ADE20K [61] and OST300 [50], whose corresponding ground-truth HR images can not be found. To evaluate the anti-forgetting performance,
a benchmark dataset Set5 [3].

---

Table 1: Comparison with existing state- sfate-of-the-art SR methods on DIV2K-C for 2x SR regarding PSNR ( and inference time (second/image), which is measured on a single TITAN XP GPU. The bold number indicates the best result and the underlined number indicates the second best result.
Method
Gaussian Blur
Defocus Blur
Glass Blur
Gaussion Noise
Poisson Noise
Impulsc Noise
Speekle Noise
Mean
GPU Time (scconds/inag)
SwinlR [29]
30.40
25. 52
27.82
25.35
22.
15,34
30.45
30.74
26.00
13.08
[7]
28.93
24.08
26.39
22.96
20.08
13.06
28.27
28.36
24.02
55.36
HAT [8]
29.00
24.08
26.40
22.31
19.33
11.91
28.02
28.25
23.66
25.01
DAN [21]
34.32
31.77
26.36
23.28
11.46
30.64
31,08
26.81
3.10
DCLS-SR [34]
33.93
25. 55
81.98
25.45
21.59
8.12
30.66
30.86
26.02
1.45
ZSSR [43]
27.79
24.24
KernaGGAN [15)+2SSR
30.18
26.87
29:01
21.45
19:32
17.93
25.07
26.1
24.37
231
MZSR [45]
30.14
25.54
28.03
25.94
23.48
17.05
30.00
30.49
26.33
3.34
DualSR [12]
29.00
24.40
28.18
22.30
20.11
17.22
24.99
24.74
23.87
210.85
DDNM [5]]
28.46
24.09
26.39
24.37
21.92
13.98
28.60
28.26
24.51
2,288.55
EDSR [31]
30.28
25. 52.52
27.82
25.87
22.96
15,87
30.52
30.83
26.21
TIA-C
30.21
25.50
27.79
26.37
23.57
16.41
30.25
30,91
26.38
13.59
SRTTA (ours)
31.07
29:01
26.69
16.15
32.33
27.76
5.38
SRTTA-lifekng lifeking (ours)
31.07
25.83
29.18
29.48
27.10
16.27
31.71
31.22
27.73
5.38
Implementation details and evaluation metric. We evaluate our approach using the baseline model of EDSR [31] with only 1.37 M parameters for 2 x SR. To demonstrate the effectiveness of our SRTTA, we conduct experiments in two settings, including parameter-reset setting and a lifelong setting. In the parameter-reset setting, the model parameters will be reset after the adaptation of each domain, which is the default setting of our SRTTA. In the lifelong setting, the model parameters will never reset in the long-term adaptation,
this case, we call our methods as SRTTA-lifelong. For the balance weight in Eqn. (6), we set to 1. For the ratio of parameters to be frozen, we set the P to 0.50. Please refer to more details in the supplementary materials.
To compare the inference times of different SR methods, we measure all methods on a TITAN XP with 12G graphics memory for a fair comparison. Due the memory overload of HAT [8] and DDNM [,1], we chop the whole image into smaller patches and process them individually for these two methods. To evaluate the performance of different methods, we use the common metrics PSNR [20] and SSIM [20] and report the results of all methods on the DIV2K-C dataset. Due to the page limit, we mainly report PSNR results and put more results in the supplementary materials.
Compared methods, We compare our SRTTA with several state-of-the-art methods including supervised pre-trained SR methods, blind SR methods, zero-shot SR methods, and TTA baseline method. 1) The supervised pre-trained SR models learn super-resolution knowledge with predefined degradation process, i.e., the Bicubic downsampling. These methods include SwinIR [29], IPT [7] and HAT [8], and the EDSR baseline [31]. 2) Blind SR models predict the blur kernel of test images and generate HR images simultaneously. e.g., DAN [21] and DCLS-SR [34]. 3) Zero-shot SR models often use a predefined/estimated degradation kernel to construct the LR-HR paired images based on the assumption of the ccoss-scale patch recurrence [14, 62, 37], and they use the LR-HR paired images to train/update the SR models for each test image. These methods include ZSSR [43]. KernelGAN [2]+ZSSR [43], MZSR [9], DualSR [12], DDNM [51]. 4) Moreover, we implement a baseline TTA method (dubbed TTA-C) that utilizes the augmentation consistency loss L. in Eqn. (8) to adapt the pre-traines model, similar to MEMO [57] and CoTTA [48].
5.2 Comparison with State-of-the-art Methods
We evaluate the effectiveness of our methods in terms of quantitative results and visual results. We report the PSNR results of our SRTTA and existing methods on the DIV2K-C dataset for 2 x SR in Table I (more results are put in the supplementary materials). Since DAN [21] and DCLS-SR [34] are trained on paired images with Gaussian Blur degradation, they achieve the best results in Gaussian Blur and Glass Blur degradation. However, they merely achieve a limited performance on average due to the ignoring of the noise and JPEG degradations in their degradation model. Moreover, ZSSR [43] achieves tftt--of-hh-arr art performance on average due to the thousands of iterations of training steps for each image. Though KernelGAN [15] estimates a more accurate blur kernel and helps to generate more plausible HR images for Gaussian Blur and Glass Blur degradation images, performance of 7SSR [43] on the noise images, since the degradation model of KernaGGAN [15] does not cover the noise degradation. Moreover, the baseline TTA-C may be harmful to adapting the pre-trained model to blur degradation due to the simple augmentation, resulting in limited
harmful the
7

---

Blabbic
DAN CANHSSSR
DDNM
SRTTA
Ground-truth HR
ZSSR
DualSR
HAT
EDSR SRI AKfTTA-iifeoongoaauss
(a) Visualizations under Gaussian Noise for 2x
Blabbic
DAN KEmEGANIISSR DDNM
OOTTA
Ground-trutthHR
ZSSR
DualSR
HAT
EDSR SRITA- (fllooooo)
(b) Visualizations under JEPG Compression for 2x SR
Figure 2: Visualization comparison DIV2K-C test images with degradation for 2x SR.
adaptation performance. Instead, our methods achieve the best performance in terms of PSNR average. For quality comparison, we provide the visual results of our SRTTA and the compared methods in Figure 2. As shown in Figure 2, SRTTA is able to remove the degradation and reconstruct clearer SR images with less noise or fewer JPEG artifacts.
Comparison of inference time. Moreover, we also compare the inference time of different methods Table 1. Due to the additional time for test-time adaptation, our SRTTA cannot achieve the best inference speed. However, those methods with less inference time are mostly trained on domains with Gaussian blur degradation only, such as DAN [21], DCLS-SR [34], and MZSR [45], which still suffer from the domain shift issue under noise or JPEG degradation. Instead, with comparable efficiency, our SRTTA achieves an impressive improvement on average for all domains (see results in Table 1). In conclusion, our SRTTA achieves better tradeoff between performance and efficiency.
More results on test domains with mixed multiple degradations. In this part, we evaluate our SRTTA on DIV2K-MC to further investigate the effectiveness of SRTTA under mixed multiple degradations. In Table our SRTTA achieves the best performance on four domains with different mixed degradations, e.g., 26.47 (ZSSR) - 28.48 (our SRTTA- regarding the average PSNR metric. These results further validate the effectiveness of our proposed methods.
Evaluation in terms of human eye-related metrics. In this part, we further evaluate different methods in terms the Fréchet Inception Distance (FID) [19] and the Learned Perceptual Image Patch Similarity (LPIPS) distance [58], which correlate well with perceived image quality and are commonly used evaluate the quality of generated images [10, 55J. We evaluate different methods on two synthesized datasets, including DIV2K- - and DIV2K-MC. As shown in Table 3, our SRTTA achieves the lowest values of both FID and LPIPS scores, demonstrating our SRTTA is able to generate images with higher visual quality.
5.3 Further Experiments
In this part, we further conduct several ablation studies to demonstrate the effectiveness of each component our SRTTA. Last, we evaluate our SRTTA on several datasets with real-world test images and provide the visual comparison results of different methods.

---

Table 2: Comparison with different SR methods on the synthesized DIV2K-MC dataset. We report the PSNR(1) values of different methods.
Table 3: Comparison with different methods in terms of FID(L) and LPIPS()) on both DIV2K-C and DIV2K-MC datasets.
Methods
Blur Noisc
Blur JPBG
Noise JPEG
BturNoisc JPBG
Mean
Methods
DIV2K-C LDD/
DIV2K-MC
LPIPS
SwinlR [29]
20.91
26.83
23.86
22.77
SwinIR [29]
72.90/0.2441
60.62/ 0.2781
IPT [7]
21.28
26.83
24.15
22.96
23.81
[7]
68.22 /0.2345
38.2470.244
HAT [8]
23.41
28.86
25.69
24.42
25.59
HAT [8]
64,92/ 0.2352
60.73/ 0,264
[21]
DAN [21]
73.59/ /0,2260
56.96 0,2263
DCLS-SBR (3-SS(344 [34]
23,84
28.93
26.37
24.92
26.02
DCLS-SR [34]
$7.93 0.2299
ZSSR [43]
24.9
29.02
26.68
25.24
26.47
ZSSR [43]
56.66 /0.1931
52.78 0,2155
KemelGAN [2]+2SSR
KemelGAN [2]+2SSR
88.28 0.2160
80.19 0.2371
DualRR [13]
MZSR [9]
25.59
18.73
24.90
28.24
23.78
20.5
24.62
20.62
25.56
21.16
MZSR [9]
68.27 00,2085
162.72 00,4463
DDNM
26.82
25.11
24.59
DDNM [S1]
70.80/0.2100 0.2110
59.64 0,2083
EDSR [31]
24.02
28.93
26.08
24.73
25. .4.94
EDSR [31]
69.20/
57.95 10,2338
TTA-C
24.29
28.93
26.35
24.91
26.12
TIA-C
66.95 0.2118
06,22/0.2293 0.2293
SRTTA (ouns)
29.13
27.12
SRTTA (ours)
54.37 0..1877
36.88 88 0.1915
SKIIA--ifeloog (ours)
27.67
29.02
29.70
27.52
28.48
SKI'IA-Lifeoon (ours)
53,30 0.1828
35.72/ 0.1832
Table Effectiveness of compo- nents in SRTTA on DIV2K-C,
Table 5: Effects of different a (in Eqn. (6) under parameter- reset setting. We report average PSNR (f) on DIV2K-C (with degradation shift) and Set5 (w/o degradation shift).
Method
Avg. PSNR
SRTTA (ours)
27.76
0.1
0.5
2
5
w/o classifier C(-)
26.06
w/o C.
27.15
DIV2K-C
10,24
13.96
22.63
27.76
27.52
27.31
w/o C..
10.24
Sct5
11.42
37,66
37.75
34.59
35,41
35.89
Effect of the degradation classifier C(-). To investigate the effect of the degradation classifier, we compare our SRTTA with a baseline that does not use the degradation classifier. Specifically, this baseline generates the second-order degraded images with the random degradation type. In this case, a test image with blur degradation can be randomly degraded with blur, noise, or JPEG degradation. We report the PSRN results of our SRTTA and this baseline in Table 4. Experimental results demonstrate the necessity of the degradation classifier in second-order degradation scheme.
Effect of C. and C. in Eqn. (6). To investigate the effect of the self-supervised adaptation loss f. and adaptation consistency loss La in Eqn. (6), we report the mean PSNR results of our SRTTA with Lnly and -nlon As shown in Table 4, without the adaptation consistency loss La, the SR models with only the L. will inevitably result in a model collapse. This is because the SR model prone to output the same output for any input images without meaning. When we remove the L. loss, SRTTA can only achieve limited performance, which demonstrates that L, truly helps to encourage the SR models to learn how to remove the degradation during the adaptation process. These experimental results demonstrate the efffcctiveness of the L, and La in framework.
Effect of the syper-parameters in Eqn. (6). To investigate the effect of the weight of adaptation consistency loss a in Eqn. (6), we report the mean PSNR results of our SRTTA with different a in Table 5. When the a is too small (a < 1), the culf-super degradation loss L. dominates the adaptation process and often results in the collapse of SR models. When the a is too large (a > 1), the adaptation consistency loss C. may have great constraint on the adapted SR model be the same as the pre-trained model, which may be harmful to the adaptation of the SR performance. With = 1, our SRTTA achieves the best results on the DIV2K-C dataset on average. Therefore, we simply set the a to be 1 for all other experiments.
Effect of adaptive parameter preservation in Eqn. (9). In this part, we investigate the effect of our adaptive parameter preservation (APP) strategy on test-time adaptation. We compare our APP with the existing anti-forgetting method Stochastic Restoration (STO) [48], which randomly restores different set of parameters (1% parameters) after each adaptation step. Moreover, we also compare our APP with the random selection (RS) baseline, which randomly selects fixed set of parameters to freeze before adaptation. As shown in Table 6, though STO achieves the best anti- forgetingg performance on the clean Set5 dataset, the STO severely hinders the TTA performance. The random selection baseline only achieves a limited performance of both TTA and anti-forgetting. Instead, our APP consistently outperforms the random selection baseline with the same ratio of frozen parameters. Moreover, our APP with p 0.5 achieves the best adaptation performance DIV2K-C (see more

---

Table 6: Ablation studies of adaptive parameter preservation (APP) strategy on DIV2K-C and Set5 under the lifelong setting. We report PSNR (?) and results on DIV2K-C are averaged over 8 different degradation types. The compared stochastic restoration (STO) [48] select 1% parameters for each adaptation iteration and Random Selection (RS) is evaluated by selecting different p parameters.
RS with different
APP with different p (ours)
Dataset
STO [48]
0.3
0.5
0.7
0.3
0.5
0.7
DIV2K-C (with degradation shift)
27.17
27.52
27.62
27.68
27.72
27.73
27.73
Set5 (w/o degradation shift)
35.57
33,95
34.02
34.24
34.11
34,23
34.38
198 Blabicc
198
198
198
198
Josefstr. 161-198
DAN
keTHGGAN DCLS-SR
HAT 198 EDSR SKTTA-IIfeeanjjo -ifeedooooooorsss
ner AG 198
198
198 SwinIR
198
198
nerei
LR
ZSSR
DINM
(a) Visualization results of the image from DPRD [23].
SX
SX-
SX-
SX-
SX HAT
Bicubic
DAN
keTree GAN4ZSSR
DCLS
SX
SX
SX
SX EDSR
SX
SX
LR
ZSSR
SwinIR
DDNM
SRT (alillllloon))))ss))))
(b) Visualization results of the image from ADE20K [61].
Figure 3: Visualization comparison on real-world test images from DPED [23] and ADE20K [61].
results in the supplementary materials), thus set the ratio of preservation parameters to be 0.5 in our SRTTA by default. These results demonstrate the effectiveness our APP strategy.
Visualization results on the real-world images. We also provide the visual results of different methods on the real-worke images from DPED [23] and ADE20K [61]. We use our SRTTA-lifelone model that has been adapted on DIV2K-C to perform test-time adaptation on the real-world images from DPED [23] and ADE20K [61], respectively.
shown Figure 3, SRTTA- -lifellg is able to generate HR images with fewer artifacts. These results demonstrate that our method is able to be applied to real-worke applications. Please refer to more results the supplementary materials.
Conclusion
In this paper, we propose super-resolution test-time adaptation (SRTTA) framework to quickly alleviate the degradation shift issue for image super-resolution (SR). Specifically, we propose a second-order degradation scheme construct paired data for each test image. Then, our second-order reconstruction loss is able to quickly adapt the pre-trained SR model to the test domains with unknown degradation. To evaluate the effectiveness of our SRTTA, we use eight different types of degradations to synthesize two new datasets named DIV2K-C and DIV2K-MC. Experiments on the synthesized datasets and several real-world datasets demonstrate that our SRTTA is able quickly adapt the SR model for each image and generate plausible nigh-resolution images.
10

---

Acknowledgements
This work partially supported by the National Natural Science Foundation of China (NSFC) (62072190), National Natural Science Foundation of China (NSFC) 61836003 (key project), Program for Guangdong Introducing Innovative and Enterprencurial Teams 20172T07XI83, and CCF-Tencent Open Research Fund (CCF-Tencent RAGR20220108).
References
[1] E. Agustsson and R. Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In The IEEE Conference on Computer Vision and Pattern Recognition Workshops, July 2017.
[2] Bell- Bligler A. Shocher, and M. Irani. Blind super-resolution kernel estimation using an internal-gan. In Advances in Neural Information Processing Systems, volume 32, 2019.
[3] M. Bevilacqua, A. Roumy, C. Guillemot, and M. L. Alberi-Morel. Low-complexity single- image super-resolution based nonnegative neighbor embedding. In British Machine Vision Conference, pages 1-10. BMVA press, 2012.
[4] A. Bruhn, J. Weickert, and C. Schnörr. Lucas/kanade meets horn/schunck: Combining local and global optic flow methods. International Journal of Computer Vision, 61:211-231, 2005.
[5] A. Bulat, J. Yang, and G. Trimiropoulos. To learn image super-resolution, use gan to learn how to do image degradation first. In European Conference on Computer Vision, pages 185-200, 2018.
[6] D. Chen, D. Wang. T. Darrell, and S. Ebrahimi. Contrastive test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 295-305, 2022.
[7] H. Chen, Y. Wang, T. Guo, C. Xu, Y. Deng, Z. Liu, S. Ma. Xu, C. Xu, and W. Gao. Pre-trained image processing transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12299-1230 10, 2021.
[8] X. Chen, X. Wang, J. Zhou, Y. Qiao, and C. Dong. Activating more pixels image super- resolution transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22367-22377, 2023.
[9] X. Cheng, Z. Fu, and Yang. Zero-shot image super-resolution with depth guided internal degradation learning. In European Conference on Computer Vision, pages 265-280. Springer, 2020.
[10] H. Chung, B. Sim, D. Ryu, and C. Ye. Improving diffusion models for inverse problems using manifold constraints. In Advances in Neural Information Processing Systems, volume 35, pages 25683-22 25696, 2022.
[11] C. Dong, C. C. Loy, K. He, and Tang. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2):295-307, 2016.
[12] M. Emad, M. Peemen, and H. Corporaal. Dualsr: resolution. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1630-1639, 2021.
Zero-shot dual learning for real-world super-
[13] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pages 1 111261132 135. PMLR, 2017.
[14] D. Glasner, S. Bagon, and M. Irani. Super-resolution from a single image. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 356-356. IEEE, 2009.
[15] J. Gu, H. Lu, W. Zuo, and C. Dong. Blind super-resolution with iterative kernel correction. In Proceedings of the JEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 1604-1613, 2019.
11

---

[16] Y. Guo, J. Chen, J. Wang, Q. Chen, J. Cao. Z. Deng, Y. Xu, and M. Tan. Closed-loop matters: Dual regression networks for single image super-resolution. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 5407-5416, 2020.
[17] J. M. Haut, R. Fernandez-Beltran, M. E. Paoletti, J. Plaza, A. Plaza, and F. Pla. A new deep generative network for unsupervised remote sensing single-image super-resolution. IEEE Transuctions on Geoscience and Remote sensing, 56(1 (:611))672-6810) 2018.
[18] D. Hendryeks and T. Dietterich. Benchmarking neural network robustness common corrup- tions and perturbations. In International Conference on Learning Representations, 2019.
[19] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in neural information processing systems, volume 30, 2017.
[20] A. Hore and D. Ziou. Image quality metries: PSNR vs. SSIM. In International Conference on Pattern Recognition, pages 2366-2369, 2010.
[21] Y. Huang, S. Li, L. Wang. T. Tan, et al. Unfolding the alternating optimization for blind super resolution. In Advances in Neural Information Processing Systems, volume 33, pages 5632-5643, 2020.
[22] S. A. Hussein, T. Tirer, and R. Giryes. Correction filter for single image super-resolution: Robustifying off-the-shelf deep super-resolvers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1428-1437, 2020.
[23] A. Ignatov, N. Kobyshev, R. Timofte, K. Vanhoey, and L. Van Gool. Dslr-quality photos on mobile devices with deep convolutional networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3277-3285, 2017.
[24] X. Ji, Y. Cao, Y. Tai, C. Wang, J. Li, and F. Huang. Real-world super-resolution via kernel estimation and noise injection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pages 466-467, 2020.
[25] X. Kang, J. Li, P. Duan, F, Ma, and S. Li. Multilayer degradation representation-guide guided blind super-resolution for remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 1-21-1 2022.
[26] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardinss A. A. Rusu, K. Milan, Quan, T. Ramalho, neural networks. Proceedings of the National Academy of Sciences, 114(13):35213352) 2017.
Trabska-Barwinskke et al. Overcoming catastrophic forgetting in
[27] P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Ya- sunaga, R. L. Phillips, I. Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pages 5637-5664. PMLR, 2021.
[28] W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 624 632, 2017.
[29] J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1833-1844, 2021.
[30] J. Liang, K. Zhang, S. Gu, L. Van Gool, and R. Timofte. Flow-based kernel prior with application to blind super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10601-10610, 2021.
[31] B. Lim, Son, H. Kim, S. Nah, and K. Mu Lee. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 136-144, 2017.
[32] A. Liu, Liu, Gu, Y. Qiao, and C. Dong. Blind image super-resolution: survey and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
12

---

[33] P. Liu, H. Zhang, Y. Cao, Liu, D. Ren, and W. Zuo. Learning cascaded convolutional networks for blind single image super-resolution. Neurocomputingg 417:371-383, 2020.
[34] Z. Luo, H. Huang, L. Yu, Y. Li, H. Fan, and S. Liu. Deep constrained least squares for blind image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17642-17652, 2022.
[35] W. Ma, Z. Pan, J. Guo, and B. Lei. Achieving super-resolution remote sensing images via the wavelet transform combined with the recursive res-net. IEEE Transactions on Geoscience and Remote Sensing, 57(6):3512-3521 2019.
[36] S. Maeda. Unpaired image super-resolution using pseudo-supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 291-300, 2020.
[37] Michaeli and M. Irani. Nonparametric blind super-resolution. In Proceedings of the IEEE International Conference Computer Vision, pages 945-952, 2013.
[38] S. Niu, J. Wu, Y. Zhang, Y. Chen, S. Zheng, P. Zhao, and M. Tan. Efficient test-time model adaptation without forgetting. In International Conference on Machine Learning, pages 16888- 16905. PMLR, 2022.
[39] S. Niu, J. Wu, Y. Zhang, Z. Wen, Y. Chen, P. Zhao, and M. Tan. Towards stable test-time adaptation in dynamic wild world. In The International Conference on Learning Representations, 2023.
[40] Y. Pang, J. Cao, Wang, and Han. Jcs-net: Joint classification and super-resolution network for small-scale pedestrian detection in surveillance images. IEEE Transactions Information Forensics and Security, 14(12):332233311 2019.
[41] S. Park, J. Yoo, D. Cho, J. Kim, and T. H. Kim. Fast adaptation to super-resolution networks via meta-learning. In European Conference on Computer Vision, pages 754-769. Springer, 2020.
[42] L.. Schermelleh, A. Ferrand, T. Huser, C. Eggeling, M. Sauer, O. Biehlmaier, and G. P. Drummen. Super-resolution microscopy demystified. Nature Cell Biology, 21(1):72-84, 2019,
[43] A. Shocher. N. Cohen, and M. Irani. "zero-shot" super-resolution using deep internal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 311 18 00-666,22220 2018.
[44] Y. M. Sigal, R. Zhou, and X. Zhuang. Visualizing and discovering cellular structures with super-resolution microscopy. Science, 361(6405):830-887, 2018.
[45] J. W. Soh, S. Cho, and N. I. Cho. Meta-transfer learning for zero-shot super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 3516-3525, 2020.
[46] Y. Sun, X. Wang, Z. Liu, J. Miller, A. Efros, and M. Hardt. Test-time training with self- supervision for generalization under distribution shifts, In International Conference on Machine Learning, pages
PMLR, 2020.
[47] D. Wang, E. Shelhamer, S. Liu, B. Olshausen, and T. Darrell. Tent: Fully test-time adaptation by entropy minimization. International Conference on Learning Representations, 2021.
[48] Q Wang, O. Fink, L. Van Gool, and D. Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7201-7211, 2022.
[49] X. Wang. L. Xie, C. Dong, and Y. Shan. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In Proceedings of the IEEE/CVF International Conference on Com- puter Vision, pages 1914, 2021.
[50] X. Wang, K. Yu, C. Dong, and C. C. Loy. Recovering realistic texture in image super-resolution by deep spatial feature transform. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 606-615, 2018.
13

---

[51] Y. Wang. J. Yu, and J. Zhang. Zero-shot image restoration using denoising diffusion null-space model. In The International Conference on Learning Representations, 2023.
[52] Y. Xiao, Q. Yuan, Q. Zhang, and L. Zhang. Deep blind super-resolution for satellite video. IEEE Transactions on Geoscience and Remole Sensing, 2023.
[53] J. Yang, K. Fu, Y. Wu, W. Diao, W. Dai, and X. Sun. Mutual-feed learning for super-resolution and object detection in degraded aerial imagery. IEEE Transactions on Geoscience and Remote Sensing, 10:1-16, 2022.
[54] Y. Yuan, S. Liu, J. Zhang, Y. Zhang, C. Dong, and L. Lin. Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 701-710, 2018
[55] K. Zhang. J. Liang, Van Gool, and Timofte. Designing practical degradation model for deep blind image super-resolution. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 4791-4800, 2021.
[56] L. Zhang, H. Zhang, H. Shen, and Li. A super-resolution reconstruction algorithm for surveillance images. Signal Processing, 90(3):848 859, 2010.
[57] M. Zhang, S. Levine, and C. Finn. Memo: Test time robustness via adaptation and augmentation.
In Advances in Neural Information Processing Systems, volume 35, pages 38629-38642, 2022.
[58] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 586-58 595, 2018.
[59] Y. Zhang, B. Hooi, L. Hong, and J. Feng. Self-supervised aggregation of diverse experts for test-agnostic long-tailed recognition. In Advances in Neural Information Processing Systems, pages 34077-34090, 2022.
[60] Y. Zhang, K. Li. K. Li, L. Wang, B. Zhong, and Y. Fu. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European conference on computer vision, pages 286-301, 2018.
[61] B. Zhou, H. Zhao, X. Puig, T. Xiao, S. Fidler, A. Barriuso, and A. Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-321, 2019.
[62] M. Zontak and M. Irani. Internal statisties of a single natural image. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 977-984. IEEE, 2011.
14

---

Supplementary Materials for " "Efficient Test-Time Adaptation for Super-Resolution with Second-Order Degradation and Reconstruction
Contents
More Details of the Second-Order Degradation
A.1 Random Blur Degradation
A.2 Random Noise Degradation
A.3 Random JPEG Degradation
The Difference from Real-ESRGAN
Solving Different Problems
B.2 Different Construction Schemes
Experimental Datasets
C.1 Construction Details the DIV2K-C Dataset
C.2 Construction Details the DIV2K-MC Dataset
C.3 More Test Datasets for Test-Time Image Super Super-Resolution
Implementation Details
D.1 Implementation Details of the Degradation Classifier
D.2 More Details of
Test-Time Adaptation
E More Experimental Results
6
E.1 More Results of Test-Time Image Super-Resolution
6
E.2 More Results on Test domains with Mixed Multiple Degradations.
E.3 More Results of Ablation Studies
E.4 Effectiveness on New Unknown Domain
8
E.5 Comparison with Patch-Recurrence Reconstruction Loss
8
E.6 Effect of the Feature-level Reconstruction
8
E.7 Effect of Adaptation Iterations for Each Image
9
E.8 The Statistics of the Degradation Types of Real-world Images
E.9
Further Analysis of the Domain Shift Issue
9
Visualization Results
10
E.1
Visualization Results on the DIV2K-C Images
10
F22 Visualization Results on the Real-World Images
10
Limitation Analysis and Border Impacts
11
G.I Limitation Analysis
11
G.2 Broader Impacts
12

---

A More Details of the Second-Order Degradation
In our second-order degradation scheme, we randomly generate different degradations degrade the test image into its second-order degraded counterparts. In this section, we illustrate how to randomly generate different types of degradations. Notably, we degrade the test images on the GPU device to accelerate the degradation process.
A.1 Random Blur Degradation
Following [26, 22]. we model blur degradation as a convolution with linear Gaussian blur fil- ter/kemnel. Given a test image, we randomly generate a set of isotropic or anisotropie Gaussian kemels k and use them to perform blur degradation on the test image. The probabilities of generating an isotropic kernel and an anisotropic kernel are set to 0.5 and 0.5, respectively. The size of each generated kernel is uniformly sampled from {7 x 7, 9 9, ..... 21 x 21). We sample the standard deviation of the blur kernel along the two principal axes σ and d2 uniformly from [0.2, 3]. is an isotropic Gaussian blur kernel, we d2 equal to 01. If k is anisotropic Gaussian kernel, further sample a rotation angle uniformly from [- T.. and use a rotation matrix to transform the generated kernel based on the angle More details about how generate Gaussian blur kernel can refer to Real-ESRGAN [22].
A.2 Random Noise Degradation
In our second-order degradation, we randomly generate a set of Gaussian noise maps and add them the test image to obtain set of second-order images with different additive Gaussian noise. With probability of 60%, we generate noise for each channel of RGB images independently, otherwise, we generate the same noise map for all three channels. We first generate a noise map whose values are randomly generated from a normal Gaussian distribution. Then sample a scale value to enlarge the noise uniformly from [1.30]. More details can be referred to Real-ESRGAN [22].
A.3 Random JPEG Degradation
For JPEG compression JPEG,, we sample a quality factor q uniformly from [30, 95], and use the JPEG compression with the degradation q to degrade test images into of second-order degraded images with compression artifacts. Note that JPEG compression with lower & compress the test image with higher compression ratio and the compressed images are generally of lower quality. To accelerate the degradation process, we use DiftJPEG!, which is the Py PyTorch implementation of JPEG compression, process the test image on the GPU device.
B The Difference from Real-ESRGAN
In this part, we would like to discuss the difference between our SRTTA with Real-ESRGAN[22), which proposes the concept of second-order degradations, highlight the contribution of our SRTTA.
B.1 Solving Different Problems
Real-ESRGAN tries enumerate all the degradations in real-world scenes and train an SR model to solve the image restoration on any degradation. However, it is non-trivial to obtain all real-world degradations, leading to domain shift issues when encountering unknown degradations during testing. as shown in Figure 11 of real-ESRGAN [22]. Unlike real-ESRGAN [22], our SRTTA aims to adapt the SR models to the test domains when test images contain unknown degradations. Our second-order degradation scheme aims to quickly construct the pseudo-paired data (instead of the paired training data) to adapt the model the test domains.
B.2 Different Construction Schemes
Real-ESRGAN [22] proposes the second-order degradation to construct the paired training data, whose low-resolution (LR) images are obtained from the ground-truth high-resolution (HR) images.

---

Then, the paired data is used to train SR model during the training phase in supervised learning manner. Notably, the trained Real-ESRGAN [22] model is fixed during the test time. Instead, our second-order degradation scheme constructs the pseudo-paired data using the test images with unknown degradation (first-order degraded images). Our SRTTA model continuously adapted to different domains during testing in self-supervised learning manner.
C Experimental Datasets
C.1 Construction Details of the DIV2K-C Dataset
To evaluate the practicality, we construct new benchmark dataset named DIV2K-C, which contains eight different degradations. We select the eight degradation types from the 15 corruptions of ImageNet-C [11] that do not extremely change the image content, including Gaussian Blur, Defocus Blur, Glass Blur, Gaussian Noise, Poisson Noise (Shot Noise), Impulse Noise, Speckle Noise, and JPEG compression. Unlike the ImageNet-C [11], we do not use the same degradation level to degrade all test images. Instead, we randomly generate the degradation level and further generate degradation for each image based on the degradation level. Unlike prior SR methods that investigate a limited number of degradation types, the degradation scenarios we considered are more complex (eight degradation types in total), which is more practical for real-world applications.
Given a degradation, we use the classical image degradation model [17, 22] generate the low- resolution (LR) test images from the high-resolution clean images. For blur degradation, we perform the blur convolution on the HR images and then use the Bicubic downsampling obtain the test LR images. For noise and JPEG degradation, we first use the Bicubic downsampling to obtain clean LR images and then perform noise or JPEG degradation on the clean LR images obtain the final test images. We show the visualization of some examples regarding each degradation type in Figure A.
HR
Gaussian Blur
Defocus Blur
Gaas Blur
JPEG
0826 from DIV 2K
Clean LR
Gaussian Nodse
Polsson Noise
Impalse Noise
Speckle Nolse
HR
Gaussian Blur
Defocus Blur
G Glas Blur
JPEG
0816 from
Clean
Gaussian Noise
Polsson Noise
Impulse Noise
Specide
Figure A: The visualization of examples regarding each degradation type on the DIV2K-C dataset.
Gaussian blur. Following BSRGAN [26], we generate low-resolution (LR) images with Gaussian blur degradation. We randomly generate an isotropic Gaussian kernel or an anisotropic Gaussian kernel for each high-resolution (HR) image. Then, we use the blur kernel to perform blur convolution on the HR image and use Bicubic downsampling to obtain the final LR test images. For simplicity, we follow the recipe of BSRGAN [26] to generate test images with blur degradation.

---

Defocus blur. To better compare the performance of different SR methods, we also use the common Defocus blur degradation to degrade the HR images and obtain the LR images using the Bicubic downsampling. As illustrated in ImageNet-C: [1 JJ. Defocus blur often occurs when an image is out of focus when we take pictures. We generate the blur kernel as ImageNet-C [11] and perform a blur convolution on the HR images. But unlike ImageNet-C [11], the degradation level of Defocus blur is randomly sampled from a given range.
Glass blur. We also choose another common degradation type, Glass Blur, which appears with "frosted glass" windows or panels [11]. This blur degradation requires two Gaussian blur operations and an operation that locally shuffles pixels between two blur operations. As mentioned above, the degradation level is randomly sampled from a given range, such as the standard deviation of the Gaussian blur kernel or the window size of the shuffling operation.
Gaussian noise. To generate test images with Gaussian noise, we sample the noise for each pixel from a normal Gaussian distribution. The mean of the Gaussian distribution is zero, and the standard deviation is uniformly sampled from the range of {2/255, 3/255, ..., 25/ 25/255.. More details can refer to the implementation of BSRGAN [26].
Poisson (Shot) noise. Poisson noise, also called Shot noise, can model the sensor noise caused by statistical quantum fluctuations. We randomly generate the Poisson noise map from a Poisson distribution, which has an intensity proportional to the image intensity. Then, we add the generated Poisson noise into the clean LR images to obtain the test images with Poisson noise. More details can refer to the implementation of Real-ESRGAN [22].
Impulse noise. Impulse noise caused by errors in the data transmission generated in noisy sensors or communication channels, or by errors during the data capture from digital cameras [19]. The most common form of Impulse noise is called salt-and- pepper noise. To generate test images with Impulse noise, we uniformly select set of pixels and replace them with zero or the maximum value (255). More details can be referred to ImageNet-C [I]].
Speckle noise, Speckle noise is an additive noise where the noise added to pixel tends to be larger if the original pixel intensity is larger. We first sampled the noise for each pixel from a Gaussian distribution and multiple the noise value by the original pixel. Last, we add the generated noise map into the LR clean images to obtain the final test images with Speckle noise.
JPEG compression. For JPEG compression, we use the OpenCV implementation of JPEG com- pression2 to degrade the clean LR image into final test images. The compression quality factor q is randomly sampled from [30, 90]. We first encode the clean LR images into the bit stream using JPEG compression with the quality factor q and decode the bit stream to obtain the final test images. Note that JPEG compression is a lossy compression technique, so the final test images are inevitably corrupted with JPEG compression artifacts.
C.2 Construction Details of the DIV2K-MC Dataset
Since the real-world test images may contain multiple degradation types simultaneously. we further develop a new benchmark dataset named DIV2K-MC, which includes four test domains with mixed multiple degradations. The four domains are BlurNoise, BlurJPEG, NoiseJPEGG and BlurNoiseJPPGG. The test images in the BlurNoiseJPEGG domain contain the combined degradation of Gaussian blur, Gaussian noise and JPEG degradations simultaneously.
BlurNoise. We generate LR images from HR images using Gaussian blur and Gaussian noise degradation. We first randomly generate a Gaussian blur kernel to perform blur convolution on the HR image. Then, we downsample the resulting image using Bicubic interpolation. Last, we randomly sample a Gaussian noise map and add it to the downsampled image to obtain the final LR image.
We generate LR images from HR images using Gaussian blur and JPEG degradation. We first randomly generate a Gaussian blur kernel to perform blur convolution on the HR image. Then, the resulting image is downsampled by using Bicubic interpolation. Last, we use JPEG compression with a random quality factor to compress the downsampled image to obtain the final LR image.
Blur
JoEG We generate LR images from HR images using Gaussian noise and JPEG degradation. We first downsample HR image using Bicubic interpolation. Then, we randomly sample a Gaussian
haups://ithub.ccmm/ppaav//pgppp

---

noise map and add it to the downsampled image. Last, we use JPEG compression with a random quality factor q to compress the downsampled image obtain the final LR image.
Blur.Nise.JPEG. We generate LR images from HR images using Gaussian blur, Gaussian noise and JPEG degradation. We first randomly generate Gaussian blur kernel perform blur convolution the HR image. Second, the resulting image is downsampled by using Bicubic interpolation. Then, we randomly sample Gaussian noise map and add it to the downsampled image. Last, we use JPEG compression with a random quality factor to compress the image to obtain the final LR image.
C.3 More Test Datasets for Test-Time Image Super-Resolution
Moreover, we also evaluate the performance of SR methods on real-world test images from DPED [13], ADE20K [28] and OST300 [23], whose corresponding ground-truth HR images can not be found. To evaluate the anti-forgetting performance,
report the adapted model performance clean benchmark dataset Sct5 [2] whose images are clean images that are downsampled from HR images with Bicubic interpolation. Thus, these LR images do not contain any degradation.
D Implementation Details
D.1 Implementation Details of the Degradation Classifier
In our second-order degradation scheme, we use a pre-trained degradation classifier to predict the degradation type for each test image. To obtain the pre-trained degradation classifier, we use ResNet-50 [10] as the classifier and train it to recognize the degradation from test images.
In real-world scenes, test images may contain degradations other than these eight degradation types, such as ringing or overshoot artifacts [22], which may be viewed as variations of blur, noise or JPEG. Since it is infeasible to cover all the degradation types in real-world scenes, make the degradation classifier to predict the coarse-level four classes, including clean, blur, noise and JPEG.
Training details. Specifically, we use the 800 training HR images of DIV2K and randomly crop them into patches with the size of 224 x 224 (instead of resizing them into 224 224). Similar to the construction of DIV2K-C, we degrade each patch using a random selection of one of eight degradation types. As for clean data, we do not perform any degradation on the patches. For training, we apply Adam with 31 = 0.9, B2 = 0.999 and set the batch size as 256. The learning rate is initialized to 10-3 and decreased to 10 6 with a cosine annealing out of 400 epochs in total.
Testing details. During testing, we directly input the whole test image with original resolution into the classifier and output the predicted results to recognize the degradation type. The predicted results of the multi-label degradation classifier C(-) are the probabilities of the three degradations, including blur, noise and JPEG degradation. If the predicted probability of one degradation type is larger than the threshold of 0.5. the test image is considered to contain the degradation of this type. The clean image means that this image does not contain any degradation such as blur, noise, or JPEG, and we directly use the pre-trained SR model to super-resolve these clean test images.
D.2 More Details of Super-Resolution Rist-Time Adaptation
We use the baseline model of EDSR [16] with less than 2M parameters our pre- mreitranne SR model for 2x and 4x SR. During adaptation, we only update the parameters in the Resblock of the EDSR model. To avoid anti-forg,,,ing, we five clean test LR images from Set5 [2] select important parameters to be frozen in Eqn. (9). Moreover, when evaluating the anti-performance Set5 [2], we directly use the adapted model to super-resolve the test images without using the classifier.
In our experiment, we conduct experiments in parameter- reset and lifelong settings. In the parameter- reset setting, the model parameters will be reset after the adaptation each domain, which is the default setting of our SRTTA. In the lifelong setting, the model parameters will never be reset in the long-term adaptation, in this case, call our methods as SRTTA-lifelong.
For test-time adaptation, we use the Adam optimizer with the learning rate of 5 x 10-5 for the pre-trained SR models. We set the batch size N to 32, and we randomly crop the test image into N patches of size 96 x 96 and 64 x 64 for 2x and 4x SR, and degrade them into second-order degraded patches. We perform S 10 iterations of adaptation for each test image. For the balance weight in
5

---

Eqn. (6), we set a to 1. For the ratio of parameters to be frozen, we set the to 0.50. To compare the inference times of different SR methods, we measure all methods on a TITAN XP with 12G graphies memory for fair comparison. Due to the memory overload of HAT [6] and DDNM [24], we chop the whole image into smaller patches and process them individually for these two methods, In our experiments, we use the bold number to indicate the best result and the underlined number to indicate the second-best result.
E More Experimental Results
E.1
More Results of Test-Time Image Super-Resolution
In this part, we compare our SRTTA with existing SR methods, including supervised pre-trained SR methods, blind SR methods, zero-shot SR methods, and TTA baseline with consistency loss. 1) The supervised pre-trained SR methods learn to process test images with predefined degradation process, i.e., the Bicubic downsampling. These methods include the EDSR baseline [16]. SwinlR [15]. IPT [5] and HAT [6], SRDiff [14], and Real-ESRNet [22]. Note that Real-ESRNet [22] is the PSNR-oriented model of Real-ESRGAN [22]. which use complex combination of different degradation to construct pair training data, including Gaussian blur, Gaussian Noise, Poisson Noise and JPEG compression, and so on. 2) Blind SR models predict the blur kernel of test images and generate HR images simultaneously, we compare with the state-of-the-arr DAN [12] and DCLS [18] methods. 3) Zero- shot SR models construct the LR-HR paired images based on the assumption of the cross-scale patch recurrence and train/update their SR model for each test image. These methods include ZSSR [20], KernelGAN [1 [1]+ZSSR [20], MZSR [7], DualSR [8], DDNM [24]. 4) Moreover, we implement a baseline TTA method (dubbed TTA-C) that utilizes the augmentation consistency loss in Eqn. (8) to adapt the pre-trained model, similar to MEMO [27] and CoTTA [21].
We provide more results of our SRTTA for 2 and 4x SR in terms of PSNR and SSIM metrics. As shown in Table A and Table B, our SRTTA consistently outperforms existing SR methods on the DIV2K-C dataset on average. We further provide more visualization results in Figure D, E, and G, which demonstrate our SRTTA is able to remove the degradation from test images and generate plausible HR images. These results demonstrate that our SRTTA is able to quickly adapt the pre-trained SR model to the test images with different degradation.
Table A: We report the PSNR/SSIM results of all corruption fields in DIV2K-C for 2x SR.
Modhods
GansiianBia
DelocuiBla
Glosilu
PosietNeise Gus-isaNiis
EmpaleNNii
SpocileNose
JPEG
Mea
GPU Time
PSNKSSIN
Bicllie
25.04/0.800
14.10/0784
26.31/0.745
25 23.550.554
23 33333
15. 05.55113 324
23 214550.744
28.2540.00
24.988.611
SwHiR [15]
30.4200885 58
535536655
27.520.755
33350.110
22 43300-45
15 34440 242
22450.774
30.440 $46
260000 (36
13:08
25,0000821
22.31-0.477
19.330 349
11.91811 192
732
25.2580
23.6611597
2501
DAN [12)
34.22/11.550
25. 33.883575
1177/28772
26.36/0355
23 23.2604451
11 1.4601200
30,45.275
31.00 11.0618857
3.10
DCLL--K [18]
31.9009914
25 23.558671
31 9873887777
25.45/0521
21 9900341
5 11200112
30 31.501.754
26.001642
1.45
235K E1342551
2639/68811
25.3605544
21-766-771
25.760.500
2423005500
19.1601335
30.930.311
31.0108555
26.2211977
117.65
MESK [T]
301040L835
25.545151
28,2467777
25.9493343
23 4374804712
30.000.711
30.490.945 945
26.5511654
3.34
TaaiSR (8)
70 25:00R1854
14400-40
381162.11 K05
21 320300300 529
30 111194466
17 17.2281 376
24 94.9 736
34.740 751
210.55
DONM [S]]
78 4505tE
31/90466
31 17-
31 9300
961 310
78 16.76/1
7,745 55
(16)
TTA-C"
30.710LE34
75 5023435
13.756.772
76 13-07.355 159
23 37.5739477
16 498
90 00.2500222
30.9101
1601665
13.99
SILTTA TA Touns
31,010L599
25 233862644
25.0/0.815
29.8501 182
25.680637
J8. 1581184
32-3330L775
31. 837
27. 27.760-221
5.38
SIITX-Hickeng (DMB))
310000099
2533305544
29.156.819
20.500.79T
27.3090073
15.2011223
31.100854
31.3180855
252340.72
5.38
Table B: We report the PSNR/SSIM results of all corruption fields in DIV2K-C for 4x SR.
GasssiaaBla
DetocviBlu
ClasiBia
GasssiaNoos
PoscouNos
LugileNNis
SpocileNase
JFEG
Meas
GPL Time
PSNRS5IM
Bicabic
2538162118 16
24. H40.51
28 5555 400
25.1200000 300
22. 5554475
15.10.355
25.2806555
25.5901961
25 25-2201 503
Meal-ESttNNe [23]
263316/88
15 1000
701
25.5//1.002
19.2601.515
25.110:549
25. (001L7700
25
369560.700
243561
25000.726
20
013001255 357
12 238601220
342800625
25 23258.65
57050 559
35.35
HAT 10)
37 21.096.764 766
24.571160 660
36 01000.776 736
30 5851 365
15.76 16766112 340
11 11.051111 177
24 34200413
25.11 600
23 5616 543
5.05
DAN (1:)
26.210 K79
25.003667
31 11796)444
17000.100
25 1100631
25 066
22 9616 505
1.18
235R
273846.763
25
26 0622611255
23.0601500
17 757559
26
17.00 727
117.34
KANN E11425SR
26.2467544
25 15.441596
26 06223..255 255
20 0000 427
19.5314477
16.5800200
22 22.500.576
23.3641563
22 22.60 500
411.30
SEEIR [T]
25.750//22
25.0616576
25.770.713
22 3850
21.37/0429
16 40400 341
24.230.621
25.2601505
23 23.2001
2.14
SRUNH
3552263366
24.
194400335
34.1541.400
258UN1992
23
22.1011445
15
24.570003
TTA-C
28.1960.7%
25 25
27 27.244
24 24.000.300 504
32.5100466
16.380.357
21 6246556555
27.4540.736
24.500
20.11
SRTTA UO0KS)
25.310.792
36 200
25 1500 775
26- 54.0500000 /84
62
35
25. 268-000003 768
27. 07..66---33 7.9
35. $5.5000 674
4.47
38,610.792
7770
25.0200000
25.500
35
27.0100770
25.200.63

---

E.2 More Results on Test domains with Mixed Multiple Degradations.
In this part, we evaluate our SRTTA on DIV2K-MC, which consists of four test domains with mixed multiple degradations. In Table C, our SRTTA achieves the best performance on 4 domains with different mixed degradations, e.g., 0.619 (DualSR) - 0.775 (our SRTTA-lifelong) regarding the average SSIM metric. These results further validate the effectiveness of our proposed methods.
Table C: Comparison results with prior methods DIV2K-MC. We report the PSNR(()SSSIM() values of different methods.
Methods
BlurNoise
Blur/PEG
NoisePPBG
BlurNoiscJPEG
Mean
SwinIR [15]
20.91/0.311
26.83/0.748
23.86/0.523
22.77/0.450
23.59/0.508
IPT [5]
21.28/0.327
26.83/0.748
24.15/0,535
22.96/0.459
23.81/0.517
HAT [6]
23.41/0.399
28.86/0.788
25.69/0.572
24.42/0.502
25.59/0.565
DAN [12]
24.14/0.438
28.95/0.791
26.20/0.593
24.82/0.519
26.03/0.585
DCLS-SR [18]
23.84/0.420
28,93/0.790
26.37/0.599
24.92/0.523
26.02/0.583
7SSR [20]
24.95/0.493
29.02/0.793
26.68/0.617
25.24/0.542
26.47/0.611
KemelGAN [1]+ZSSR
23.08/0.424
28.32/0.786
21.90/0.474
22.76/0.443
24.02/0.532
MZSR [7]
18.73/0.213
24.90/06667
20.37/0.398
20.62/0.354
26.16/0
DualSR [8]
25.59/0.561
78.24/0.787
23.78/0.586
24.62/0.541
25.56/0.619
DDNM [24]
22.62/0.389
26.82/0.746
25.11/0.588
23.81/0.504
24.59/0.555
EDSR [16]
24.02/0.430
78.93/0.790
26.08/0.587
24.73/0.514
25. 94/
TTA-C
24.29/0.446
28.93/0.790
26.35/0.5598
24.91/0.522
26. 12/0.599
SRTTA (ours)
26.93/0.709
28.93/0.798
29.13/0.784
12/12.87
28.02/0.755
(ours)
27.67/0.749
29.02/0.793
29.70/0.810
27.52/0.747
28.48/0.775
E.3 More Results of Ablation Studies
Effect of each component. In this part, we investigate the effect of each component and provide more ablation studies. As shown in Table D. the baseline without the degradation classifier, generates the second-order degraded images with random degradation types, achieving limited performance in terms of both PSNR and SSIM. The baseline without the adaptation consistency loss La results in the model collapse due the lack of the consistency constraint. Without the sulf-supervvsed adaptation loss L., the TTA performance of the adapted model drops significantly. These experimental results demonstrate the effectiveness of each component of our framework.
Table D: We report the PSNR/SSIM results of ablation studies of different components for 2x SR.
Methods
CanssiaanBlu
DefocusBur
GlassBur
GauviannNise
PosiooNosse
ImpulseNoise
SpecileNoise
JPEG
Moua
PSNR/SSIM
PSNRSSIM
PSNR/SSIM
PSNR/SSM
PSNR/SSIM
PSNR/SSI
PSNR/SSIM
PSNR/SSIM
PSNRISSIM
SKTTA (ours)
$1.07/0.800
258660.744
29.01/0.815
20.66/0.762
26.60/0.653
15.15/02284
32.55/0.873
$1.50/0.857
27.760.731
-if clasodiee (X))
29.43/1812
25.5130.775
27.51.1.756
22.0500.46
25.6400577
15 66600.261
$1. 50.0 836
31 11.10/033 855
26.0686664
-woo E,
30.65/1844
26.87:0.680
28.4381.705
25.040.644
24.330.554
279900.774
$1. 17.9691547 547
31,47/1862
27.1580686
w/o C.
12.29/1254
5.6700.997
5.650.403
12.87004777
10.29/0072
11.4800.13
11.67A1.188
11.9900477
10.2400314
Effect of the hyperparameter a in Eqn. (6). In this part, we investigate the effect of the weight of adaptation consistency loss a. As shown in Table E, the adapted model with 1 achieves the best TTA performance. Thus, we set the a 1 by default for our SRTTA during adaptation.
Table E: We report the PSNR/SSIM results of ablation studies of a for 2x SR.
GaureianBlur
DefocueBla
Glassaluu
CaussiatNeisc
PossianNoisc
IngulscNoisc
SpeckleNoise
JPBG
Mian
Sc15
PSNR/SSIM
PSNR/SSIM
PSNR:SSIM
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
PSNR/SSI
PENR/SSIM
PSNR/SSIM
12. 2900.222 254
5.670 397
5,65.0 403
12.8510.477
10.2900.072
11,4892233
11,67002116
11.9900.777
10.2410.314
11.42/0.421
0.3
13.1180.318
8.02/1.043
3,6500,003
22.76.1.685
5,6500400
12.509.440
15.0300.52
30.9500850
13.96/1.40
37.56/0.959
0.5
10,4700 134
10.46-0.313
23.0900.708
29.21/0.70 797
80000717
16.30/0.20
32.10/0.571
30.98008550
22.63/0.586
37.75/0.059
1..77/00000 809
29.01/0.55 915
03.6660.77 762
26 637
15555 264
32.33/0.55 873
31.36/0. 8.57
27.76/0.72
34 34.59-00-224
50 888600882
25 65.91106676
28.71/0.300 504
29.04/0..10
25 8910.991
16.069..79
32.2300 965
$1.4550.651
27.52/0.000
35.41/09933
30,7400.557
26.91/0.650
28.53A0.798
26.47A1670
25.2900.558
16.009.266
32,0541.355
51.48/0.622
27.3100.645
35.8990-959
Effect of the hyperparameter p in Eqn. (9). In this part, we analyze the effect of the hyperparameter A. which decides the ratio of parameters to freeze, for the test-time adaptation. In Table F, when p 0.50, our SRTTA achieves the best TTA performance on the DIV2K-C dataset on average in the lifelong setting. Meanwhile, we also investigate the effect of the adaptive parameter preservation (APP) strategy in the parameter-reset setting. As shown in Table F, our APP strategy (with p 0.50)
7

---

merely has little impact on the TTA in the parameter-ress reseet setting. These results demonstrate the effectiveness of the APP strategy in test-time adaptation for image super-resolution.
Table F: We report the PSNR/SSIM results of ablation studies of for 2x SR in the parameter-reset and lifelong setting, our model is SRTTA and SRTTA-lifelong.
Setting
0.2
SRTTA
27.74/0.729 77.79/0.730 27.60N0.727
27.82/0.729
27.82/0,727
77.76/0.721 27.73/0.728
27.55/0.709
27. 27.08/0.682
26.21/0,645
SKTTA-Lifelong
27.46/0.726
27.66/0.728
27.72/0.728
27.73/0.725
27.5010.706
26.21/0.645
Comparison with other anti-forgetting methods. In this part, we compare our adaptive parameter preservation (APP) strategy with two baseline methods to demonstrate the effectiveness of our strategy in preserving the learned knowledge of pre-trained SR models. Stochastic Restoration (STO) [21] randomly selects different set of parameters (with a ratio of 1%) and restores them back to the parameters of the pre-trained models. Random Selection (RS) selects fixed set of parameters before adaption and freezes them not to update. As shown in Table G, our APP strategy achieves the best TTA results the DIV2K-C dataset. Mcanwhile, with the same ratio of selected parameters, our APP strategy consistently outperforms the Random Selection baseline for the anti-forgetting. These results demonstrate that our adaptive selection is able to select the important parameters and preserve the knowledge of the pre-trained model.
Table G: We report the PSNR/SSIM results of ablation studies of adaptive parameter preservation (APP) strategy for 2x SR in the lifelong setting.
Duliset
STO [21]
different
APP with different e (ours)
0.3
0.5
0.7
0.3
0.5
0,7
DIV2K-C (with degradation shift)
27. 17.16008877
27.52/0.227
27. 27.62/0.728
27.6800.22
27.7200728
27.73/0728
27.73/0.725
Setō (wo degradation shift)
35.57/0.938
33.95/0.113
34,02/0.144
34.24/0.18
14.1119
34.23/0.117
34.38/0.920
E.4 Effectiveness on a New Unknown Domain
In this part, we further evaluate our SRTTA a unknown domain with the degradation of processed camera sensor noise [3, 26], which is not used in the training phase of the SR model or that of the degradation classifier. We report the PSNR(1) and SSIM(1) values of different methods 100 images with random processed camera sensor noise. In Table H. our SRTTA method is also able improve the model performance on this unknown degradation. These experimental results further demonstrate the generalization capabilitt of our SRTTA model to unknown degradation types.
E.5 Comparison with Patch-Recurrenee Reconstruction Loss
In this part, we investigate the effect of our second-order reconstruction loss. We compare our loss with the loss of existing zero-shot methods [20, 7, 1]. Based on the assumption of patch recurrence across scales [9, 29], these methods downsample the test image to obtain image with a lower resolution and reconstruct the test image from the downsampled image. For simplicity, call this patch- nettc-ree loss. For a fair comparison, we further downsample the second-order degraded images that are obtained using our second-order degradation scheme and reconstruct the test image with the patch-recurrence loss. As shown in Table I, our SRTTA with our second-order reconstruction loss consistently outperforms the baseline with the patch-recurrence loss. These results demonstrate the effectiveness of our second-order reconstruction loss.
Table H: Results of different methods on the unknown domain with the degrada- tion of processed camera sensor noise.
Methods
PSNR
SSIM
SwinIR [15]
19.45
0.496
IPT [5]
19.51
0.500
HAT [6]
21.52
0.596
DDNM [24]
19.63
0.518
DAN [12]
21.53
0.598
DCLS-SR [18]
21.57
0.605
Dual [8]
21.14
0.586
MZSR [7]
20.40
0.438
7SSR [20]
21.57
0.621
KemeIGAN [1]+ZSSR
20.60
0.543
EDSR [16]
21.56
0.601
SKTTA (ours)
21.81
0.647
E.6 Effeec of the Feature-level Reconstruction
In this part, we investigate the effect of different reconstruction levels. In our second-order recon- struction, use the feature-level reconstruction to adapt the pre-trained model as in Eqn. (6). We
8

---

Table I: We report the PSNR/SSIM results of ablation studies of the patch-recurrence reconstruction loss for 2XSR in parameter-reset and lifelong settings.
Detocusllr
Glasslur
GuussiaaNoree
PossouNNsee
InpulseNiiie
SpeckieNeise
JPBG
Meat
nethods
Setting
PSNRASIM
PENRISSI
PSNRISSIM
PSNRISSIM
PSNRASSIM
PENRASSM
PSNRASIM
PENRASSI
PSNRANSMM
29,8000825
25.5108624
27.670033 767
28
284100605
19.720040
31.21,8.42
$1002/9633
27.960 207
Nuchessumees loss
Hilekon
29,5001525
25.5180624
27.6700 267
28. 528.5788
26.4000.004
20.45-0.430
31.960.544
31 31.00065S.
23 07,550 710
SRTTAouus)
panmettr-reee
31 (770000000 805
25.8631674
19
29. 762
76 66.6666677
16. 16.1580254
32.4300 578
51 22.3000.85 857
27-760:73
SRTTStons)
119kking
$1.07/0.869
25.410.004
20. 180.888
29, 1800 787
27.1000.678
16.2700273
31.7189.66
31,2300855
27.71/0.725
Table J: We report the PSNR/SSIM results of ablation studies of feature-level and image-level reconstruction for 2x SR in the lifelong setting.
GausianBlur
DetocudBlur
Glass
GaussianNosse
PossioaNotse
ImpulseNoise
SpeckleNoise
JPEG
Recunsimet-loed
PSNR/SSIM
PENI/SSIM
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
PSNI/SSIM
PSNR/SSI
PSNR/SSIM
PENR/SSIM
BBSSK
300 20/12/11335
25 675
27 8270
25 57405566
22 1.96600440
15,8781 269
30.52A0771
30.8300:47
645
Image-lovel
221 /0.871
25.7366671
29 160.817
28.56.0 796
0400400554
16,2400
32.47H0.8 88-
51.4400.800
27.82/0
Fooemn-loval
31 0..00.8889
25.83/1877
29.1801881
79 79-4800.717
27.10/0.675
16.27/0.273
34.110080
31,2300,553
32. 27.7500.726
DDSSK (10)
28, (/1/790
25 8181692
27.8000:551
23.AW0L 679
22. 1000.83
15.200 15,2800260 283
26.80000000 076
21.3600
24.57AL60
Image-lovel
28.6 6 (01.700
26 230230 697
78 150-775
26.460L685
25. 6211622
15.72012997
28.02/0 751
27.7400-740
25.77A1 670
Frominellovel
28. 39.78800795
26.31/0.7033
28. 1601660777
26.28/0.691
25.06/0..45
15,6500204
22 7700775
27.61.0:549
28.780.675
compare a baseline with image-level reconstruction, which means we reconstruct the output of the SR model instead of the feature in the middle layer. As shown in Table J, when reconstructing at the image level, the adapted model achieves comparable performance for both 2 SR and 4x SR. Thus, both
reconstrucc reconstruction levels are optional, we use the feature- levell reconstruction by default.
Table K: We report the PSNR/SSIM results of ablation studies of adapted iterations for 2x SR in the parameter-reset setting.
CaussiuBBer
DefacusBuur
GlussBar
GussciatNoii
PosiorNoise
ImpulseNois
SpeckleNoise
JPEG
Inmations
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
soxurddinsag
30,690,844
59.5995775
28.06007922
27.050 592
20.340.507
15.98012299
31.32A0,100
51,0998855
26. 26.7400568
0.60
30:7200
K2406679
28.220
27.760
28.990.536
16,020
31,260
31.2348856
1.41
3
30.6400.861
25.8041.677
28.8530.511
28 9000.002
36.1000.600
15.98/1210 279
32.1900 561
$1.57/0.859
27 46/0.7106
2.73
10
31,0710.800
25. 960.674
29.6100.515
29. 09.6660.762
26.69/0.600 687
16.15/0.220 264
82.33/0.977
51. 51.36/0.57
27.78/0.721
5.38
30
31.1550880
35 575
201.3200.824
70 26-2447772
36.559/001
16.06/1-276
12. 440 965
51. 31/0188866
27 640-717
1082
50
30.7100.856
6411.6677
29 00.2659-221
39. 1230.743
25.40005660
16.02/0265
31 5480658.
25.46/08846
32 02200701
25.64
E.7
Effect of Adaptation Iterations for Each Image
In this part, we investigate the effect of different adaptation iterations for each image. As shown in Table K, we compare the TTA performance of SRTTA with a different number of iterations S for each image. When the number of iterations S is small, the adapted SR model is not able to learn how to remove the degradation for these images well. When the number of iterations is too large, the performance improvement of SRTTA diminishes and the adaptation cost will be greatly increased. Thus, we set S to 10 for a better efficiency-accuracy trade-off.
E.8 The Statistics of the Degradation Types of Real-world Images
In this part, count the statistics of the degradation types of real-world images and report the results in Table L.. We use our degradation classifier to identify the degradation types of real-world images from five datasets, including RealSR [4], DRealSR [25]. DPED [13], OST300 [23] and ADE20K [28]. Experimental results in Table L show that the degradation type of blur happens the most among these real-world datasets.
E.9 Further Analysis of the Domain Shift Issue
In this part, we investigate the effect of the domain shift issues for pre-trained SR models, which are trained specific domains with different degradation types. We use the EDSR baseline model the model for analysis. In total, we separately train four EDSR baseline models on clean, blur, noise and JPEG domains, respectively. The corresponding four models named EDSR, EDSR-B, EDSR-N and EDSR-J, respectively. We evaluate these four models on clean images and the test images with Gaussian Blur, Gaussian Noise JPEG degradations in Figure B and Figure C.
As shown in Figure B and Figure C, when domain shift occurs, the pre-trained EDSR models, which trained on domains different from the test domains, cannot remove the degradation from test

---

Table L: The count of the predicted degradation types of the real-world images from the five datasets. Note that some images can contain more than one degradation type simultaneously.
Dataset (# Images)
Clean
Blur
Noise
JPEG
RealSR (912)
48
860
149
DRealSR (35148)
1378
33770
0
1
DPED (187)
103
33
21
64
OST300 (300)
52
23
14
221
ADE20K (27574)
0
3452
2424
27573
Total (64121)
1581
38138
2460
28008
EDRR
Ogperr beaand)
6056-1
(a) Visualization results under clean domain.
(b) Visualization results under Gaussian Blur.
EOOR
SATTAJ
SKTTAUuMI
EDGR-8
EDSR-N hupper bound)
EDSR-J
EDSR.B
EDSR-N N
EDSR- lupper boand)
(c) Visualization results under Gaussian Noise.
(d) Visualization results under JPEG.
Figure Visualization of the domain shift issue under different domains for 2x SR.
images and generate unsatisfactory HR images with artifacts. For example, EDSR-B models cannot remove the noise and JPEG degradation, the EDSR-N and EDSR-J also unable remove the blur degradation. Instead, after test-time adaptation, our SRTTA is capable handling the test images with unknown degradations and generating HR images with fewer artifacts. For example, our SRTTA is able to generate sharper HR images than EDSR-N and EDSR-J under Gaussian Blur domains. Indeed, our SRTTA models may be unable to completely remove the degradation compared with the upper-bound models, such as EDSR-B under Gaussian Blur. Thus, these drawbacks are required to be further addressed in future works.
Visualization Results
F1 Visualization Results on the DIV2K-C Images
In this part, we show more visualization comparison results different SR methods on test images of the DIV2K-C dataset for both 2x and 4x SR. As shown in Figures D and E, our SRTTA is able to reduce the degradation from the test images and generate more plausible HR images.
F.2 Visualization Results on the Real-World Images
In this part, we conduct a comprehensive comparison of our SRTTA with existing approaches on two real-world datasets, including DPED [13], ADE20K [28] and OST300 [23]. As shown in Figures F and G, our SRTTA methods consistently generate more satisfactory HR images with less degradation of unknown noise artifacts.
10

---

EDSK
SRTTALGur)
SRITAINN
LR
ED5R-
EDSKIN
IDMRJ
LA
LDSR e Jupper board)
EDMR-
IDSKJ
(a) Visualization results under clean domain.
hb) Visualization results under Gaussian Blur.
HR
1054
SKTTA(e4r0)
HR
105R
SATTA/WA
105K
1054
Sapper boand)
[0SBJ
ED58-8
EDSENN
E05R. toppo hound
(c) Visualization results under Gaussian Noise.
(d) Visualization results under JPEG.
Figure Visualization of the domain shift issue under different domains for 2x SR.
Biabbic
DAN
KATeIGANHZSSR
DDNM
(OUTS
Ground-truth HR
ZSSR
DR
HAT
EDSR SRTTA KALTA-IliA lifelong(oours)
Visualizations under Gaussian Noise for 2 SR
24 EA
Bicubic
DAN K DDNM
SRI SRUTA
Ground- HR
ZSSR
DUalSR
HAT
EDSR SKITTA-ifeeonglluuuss
(b) Visualizations under JEPG Compression for 2x SR
Figure D: Visualization comparison on DIV2K-C test images with degradation for 2x SR.
G Limitation Analysis and Border Impacts
G.1 Limitation Analysis
In this part, we analyze limitations of our SRTTA and existing SR methods. When test images are comupted at a high level, our SRTTA may not be able to completely remove the degradation and result in generated HR images with the existing degradation. For example, we show more visualization results of different methods on test images with Impulse Noise degradation in Figure H.

---

Blabbic
DAN
KemelGAN4ZSSRR
SRDIfT
(OUTA(
Ground-truth
ZSSR
IPT
HAT
EDSR SKITTA-lifelongaur
(a) Visualizations under PossionNoise for 4x SR
STATE
STATE
STATE
STATE
STATE
BANK HYSORE PRCT VAREORANOM
M
M
M
M
Bicubic
DAN KemelGAN
SRD3f
SRTTA
STATE
STATE
STATE
STATE
STATE
STATE
M HrR
M
M
M
M
M
ZSSR
HAT
EDSR SRITH (olT
Visualizations under JEPG Compression for 4x SR
Figure E: Visualization comparison DIV2K-C test images with degradation for 4x SR.
Since Real-ESRGAN [22] uses several different degradation types to construct training data, this model is able to remove the degradation in many cases. However, this model still suffers from the degradation shift issue, such as it cannot remove the gray Impulse noise from the test images as shown in Figure H(a). Moreover, Real- -aa--ESRAAN may generate HR images with over-smooth regions when removing the noise degradation and introduce some unplcasant artifacts due to the GAN training [22], which are shown in Figure H(b) and Figure H(c), respectively. Although completely remove the degradation from the test images in these cases, our SRTTA often preserves the original information of the test images. These results show the limitations of our SRTTA and existing methods and have a great impact on the practical application. Thus, these drawbacks are in urgent need address in future works.
SRTTA cannot also
G.2 Broader Impacts
Our proposed SRTTA method is capable of improving the resolution of low-resolution test images in real-world applications, resulting in enhanced image clarity and enabling precise understanding of image content. However, it important to exercise caution during aggressive TTA adaptation, as this may result in the introduction of artifacts distortions that have the potential negatively impact downstream analyses such as microscopy, remote sensing, and surveillance.
12

---

198
198
198
198
198
Josefstr. 161-198
Blaubic
DAN
KeTTIGANNZZSRR DCLS-SR
HAT
198
ner AG
198
198
198
198
198
nerei
ZSSR
SwinIR
DDNM
EDSR SRTTA SKTTA-Ifellrylan lifellngggouss
LR
Pestalozzistrasse
sse
ssse
nsse
ssee
sse
Biabicc
DAN
KHNNSSN DCLS-SR
HAT
sse
asse
sse
sse
sse
isse
LR
ZSSR
SwinIR
DDNM
EDSR SRITA- (ifeT--illoogg((urr
Bioubic
DAN
kemiIGANIZSSR DCLS
HAT
ZSSR
SwinIR
DDNM
EDSR SRTTA- (ife--iifelogg(ouss
LR
Bicubic
DAN
GETHIGAN DCLS-SR
HAT
ZSSR
SwinIR
DINM
EDSR SRITA-Ifellayonss
LR
Figure F: Visualization comparison of different methods on real-world test images from DPED [13].
13

---

SX
SX
SX-
SX-
SX
60
Bicubic
DAN
DCLS.SR
HAT
SX-
SX
SX-
SX-
SX-
SX-
LR
ZSSR
SwinIR
DDNM
EDSR
SRITA (un)
SL
SL
SL
Biabbic
DAN
FeTE GANHZSSR DCLS
HAT
SL
Su
SL
SL
SL
SL
SL
ZSSR
SwinIR
DDNM
EDSR SRI AAliieeoogglonr
LR
(a) Visualization comparisons on ADE20K [28] test images.
おラ
1
Blaubic
DAN
kImIGANNZSSRI DCLS-SR
HAT
LR
ZSSR
SwinIR
DDNM
EDSR
SRITTA-lffeaangoouss)
5
5
5
6
6
11
10
9
8
7
Bicubic
DAN
kemeiGANIZSSR DCLSSR
HAT
5
5.
5
6
6
ZSSR
SwinIR
DDNM
EDSR SRI ous
LR
(b) Visualization comparisons on OST300 [23] test images.
Figure G: Visualization comparison on real-world test images from for 2x SR.
14

---

Blabic
IPT
KAN SRDfft
SOUTA
Ground-truth HR
ZSSR
REaleStGAN
HAT
EDSR SRITA IITTA-Ilielagoouss
Limitation visualization under Impulse Noise.
Biaubic
IPT KeNHGANNZZSR SRDiff
SRITA (
Ground-truth HR
ZSSR
RealestGAN
HAT
EDSR SRITAAllfooooggouus))
(b) Limitation Visualization under Impulse Noise.
Blaubic
IPT
KemeIGANHZSSR
SRDHf
SRTTA
Ground-truth HR
ZSSR
RealestGAN
HAT
EDSR ERITTA-Ifeloog(amm
(c) Limitation visualization under Impulse Noise.
Figure H: Limitations visualization on DIV2K-C test images with degradation for 4x SR.
15

---

References
[1] S. Bell-Kligler, A. Shocher, and M. Irani. Blind super-resolution kernel estimation using an internal-gan In Advances in Neural Information Processing Systems, volume 32, 2019.
[2] Bevilacqua, A. Roumy, Guillemot, and M. L. Alberi-Morel. Low-complexity single-image super- resolution based on nonnegative neighbor embedding. In British Machine Vision Conference, pages 1-10. BMVA press, 2012.
[3] T. Brooks, B. Mildenhall, T. Xue, J. Chen, D. Sharlet, and J. T. Barron. Unprocessing images for Icarned raw denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11036-11045, 2019.
[4] J. Cai, H. Zeng, H. Yong, Z. Cao,, and L. Zhang. Toward real-world single image super-resolution: A new benchmark and new model. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 3086-3099 3095,099,200 2019.
[5] H. Chen, Y. Wang. T. Guo, C. Xu, Y. Deng, Z. Liu, S. Ma. C. Xu, C. Xu, and W. Gao. Pre-trained image processing transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12299-12310, 2021.
[6] X. Chen, X. Wang, J. Zhou, Y. Qiao, and C. Dong. Activating more pixels in image super-resolution transformer. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 22367-22377 2023.
[7] X. Cheng, Z. Fu, and Yang. Zero-shot image super-resolution with depth guided internal degradation learning. In European Conference on Computer Vision, pages 265 280. Springer, 2020.
[8] M. Emad, M. Peemen, and H. Corporaal. Dualsr: Zero-shot dual learning for real-world super-resolution. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1639, 2021.
[9] D. Glasner, S. Bagon, and M. Irani. Super-resolution from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 349-356. IEEE, 2009.
[10] K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pultern recognition, pages 770-778, 2016.
[11] D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
[12] Y. Huang, S. Li, L. Wang. T. Tan, et al. Unfolding the alternating optimization for blind super resolution. In Advances in Neural Information Processing Systems, volume 33, pages 5632-5643, 2020.
[13] A. Ignatov, N. Kobyshev, R. Timofte, K. Vanhocy, and L. Van Gool. Dslr-quality photos on mobile devices with deep convolutional networks. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 3277-3285, 2017.
[14] H. Li, Y. Yang, M. Chang, S. Chen, H. Feng, Z. Xu, Q. Li, and Y. Chen. Srdiff: Single image super- resolution with difffusion probabilistic models. Newrocomputing. 479:47-59, 2022.
[15] Liang, Cao, G. Sun, K. Zhang, L. Gool, and R. Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1833-1844 2021.
[16] B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pages 144, 2017.
[17] P. Liu, H. Zhang, Y. Cao, S. Liu, D. Ren, and W. Zuo. Learning cascaded convolutional networks for blind single image super-resolution. Neuroconputing. 417:371-38,, 2020.
[18] Z. Luo, Huang, L. Yu. Y. H. Fan, and S. Liu. Deep constrained least squares for blind image super- resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17642217655 2022.
[19] S. Schulte, M. Nachtegael, V. De Witte, D. Van der Weken, and E. E. Kerre. A fuzzy impulse noise detection and reduction method. IEEE Transactions Image Processing, 15(5):1153-1166 2006.
16

---

[20] A. Shocher, N. Cohen, and M. Irani. "zero-shot" super-resolution using deep internal learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3118-3126. 2018.
[21] Q. Wang, O. Fink, L. Van Gool, and D. Dai. Continual test-time domain adaptation, In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 7201-7211, 2022.
[22] X. Wang. L. Xie, Dong, and Y. Shan, Real-esrgan: Training real-world blind super-resolution with pure synthetic data. Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 1905 1914, 2021.
[23] Wang, K. Yu, C. Dong, and Loy. Recovering realistic texture in image super-resolution by deep spatial feature transform. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 606-615, 2018
[24] Y. Wang, J. Yu, and J. Zhang. Zero-shot image restoration using denoising diffusion null-space model. In The International Conference on Learning Representations. 2023.
[25] P. Wei, Z. Xie, H. Lu, Z. Zhan, Q. Ye, W. Zuo, and L. Lin. Component divide-and-conquer for real-world image super-resolution. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part VIII 16, pages 101-117. Springer, 2020.
[26] K. Zhang, J. Liang, L. Van Gool, and R. Timofte. Designing a practical degradation model for deep blind image super-resolution. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4791-4800, 2021.
[27] M. Zhang, S. Levine, and C. Finn. Memo: Test time robustness via adaptation and augmentation. In Advances Neural Information Processing Systems, volume 35, pages 38629-38644 2022.
[28] B. Zhou, H. Zhao, X. Puig. T. Xiao, S. Fidler, A. Barriuso, and A. Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302 321, 2019.
[29] M. Zontak and M. Irani. Internal statistics of single natural image. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 977-984, IEEE, 2011.
17


Reviews:
[{"review_id": "tdRzR840Nr", "reviewer": "Reviewer_pkZJ", "summary": "The paper proposes to solve the domain shift problem for a pretrained super-resolution model. A pretrained classifier is utilized to predict the type of degradation of the test image. After the degradation type is detected, the degradation is used as an augmentation scheme for adaptation training for the new SR model.", "questions": "see the weaknesses\n\n(Weaknesses are addressed after rebuttal and discussions )", "limitations": "no limitations", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is easy to follow. \n\nThe motivation for the work is reasonable since super-resolution is an important task in Computer Vision, while the existing methods only focus on one degradation type. This results in the poor performance of the SR model in some cases.", "weaknesses": "The analysis is the weakest part of the work:\n\n1. What are the consequences of domain shift when applying pretrained SR models? The work only mentions blur degradation is the most popular degradation type that other SR models concentrate on. However, no qualitative or quantitative observation is provided if another type of degradation happens in the dataset. What happens if the provided test datasets are degraded by blur and JPEG degradation? or both blur and noise degrade the provided test datasets? or only JPEG, or only noise?\n\n2. In practice, which type of degradation happens the most? Whether the cover for the domain shift is necessary? Which scenarios will we need to do the domain adaptation training for the SR model?\n\n3. The training of classifier in C.1 section mentions 8 types of degradation. What are the 8 types? Why, in the main paper, the classifier only has 4 classes?\n\n4. The qualitative results in Figure 3 shows that KernelGAN+ZSSR is better than the proposed method.\n\n5. In line 185, the author mentions, \"encourage the pre-trained model to learn how to reconstruct ...\", so the pretrained model will also be trained?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. What are the consequences of domain shift when applying pretrained SR models?\n\nA1: When the domain shift issue occurs, pre-trained SR models often generate HR images with unplausible artifacts, such as over-sharp edges or amplified noise [1, 2].\n\n[1] Blind super-resolution with iterative kernel correction, CVPR 2019.  \n[2] Real-ESRGAN: Training real-world blind super-resolution with pure synthetic data, ICCV 2021.\n\n> Q2. Visualization of test images with different degradations.\n\nA2. Actually, we have provided visualizations of the test images from the DPED dataset, which contains a combined degradation of noise, blur and some unknown artifacts, as shown in Figure 3 of the paper and Figure 5 of the supplementary materials. We will provide more qualitative observations of test images with different degradations in the revised paper.\n\n> Q3. Evaluation on the test images with mixed multiple degradations.\n\nA3. We have investigated our SRTTA performance on 8 domains with single degradations, including blur, noise and JPEG degradations. In the rebuttal, we further evaluate our SRTTA on four domains with a combination of different degradation, including BlurNoise, BlurJPEG, NoiseJPEG and BlurNoiseJPEG. Experimental results show that our SRTTA significantly outperforms previous methods in these domains. Please refer to General Response 2 for more details and results.\n\n> Q4. Which type of degradation happens the most?\n\nA4: It is hard to know which type of degradation happens the most due to the innumerable images on the Internet. But most previous works [1,2] mainly focus on the common degradations of Gaussian Blur, Gaussian Noise, and JPEG compression.\n\n[1] Learning a single convolutional super-resolution network for multiple degradations, CVPR 2018.  \n[2] Designing a Practical Degradation Model for Deep Blind Image Super-Resolution, ICCV 2021.\n\n> Q5. Whether the cover for the domain shift is necessary? \n\nA5: The domain shift poses a critical challenge in image super-resolution due to the diverse degradation types in real-world images [1]. Existing SR models trained on one domain (such as Gaussian Blur) tend to perform poorly on other common domains (such as Gaussian noise or JPEG). Consequently, the cover of the domain shift is necessary, allowing the adapted SR models to be applicable to diverse real-world scenes without requiring additional data collection or model retraining.\n\n[1] Blind image super-resolution: A survey and beyond, TPAMI 2022.\n\n> Q6. Which scenarios need to do the domain adaptation training for the SR model?\n\nA6: In real-world scenarios, the need for domain adaptation training (test-time adaptation) is common due to the diverse degradation types in real-world images. When the degradation of test images differs from that of training data, i.e., when the domain shift issue occurs, the SR models often require to perform test-time adaptation to adapt to these new domains. And the diverse degradation types are driven by several reasons.\n\n**First**, imaging devices differ greatly in the characteristics of the photos taken. Compared with a DSLR camera, a smartphone camera tends to produce \"flattened\" and noisy images [1].  \n**Second**, due to the diversity of the imaging environment, real-world images, such as remote sensing images [2], may contain different types of degradation.  \n**Moreover**, image processing algorithms, such as image compression, may also introduce complex degradation on real-world images [1].\n\n[1] Blind image super-resolution: A survey and beyond, TPAMI 2022.  \n[2] Multilayer degradation representation-guided blind super-resolution for remote sensing images, TGRS, 2022.\n\n> Q7. The training of classifier in C.1 section mentions 8 types of degradation. What are the 8 types? Why, in the main paper, the classifier only has 4 classes?\n\nA7: The **8 degradation types** are Gaussian Blur, Defocus Blur, Glass Blur, Gaussian Noise, Poisson Noise, Impulse Noise, Speckle Noise and JPEG. Following the classical degradation model [1], we categorize these types into 3 distinct classes, namely **blur**, **noise** and **JPEG**. Notably, many images in real-world scenes are **clean** images, which require no need to perform test-time adaptation. Consequently, we filter out these images during the test-time adaptation. Finally, we set the classifier to predict the **4 classes**, including **clean**, **blur**, **noise**, and **JPEG**.\n\n**Motivation for predicting 4 classes:** In real-world scenes, test images may contain degradations other than these 8 types, such as ringing or overshoot artifacts [1], which can be viewed as specific variations of blur, noise or JPEG. Since it is infeasible to cover all the degradation types in real-world scenes, we make the degradation classifier to predict the **coarse-level** classes.\n\n[1] Real-ESRGAN: Training real-world blind super-resolution with pure synthetic data, ICCV 2021.\n\n> Q8. The results in Figure 3 show that KernelGAN+ZSSR is better than the proposed method.\n\nA8: The KernelGAN+ZSSR focuses on removing blur degradation from test images, leading to visually sharp results. But KernelGAN+ZSSR may amplify the noise in the test images, leading to generated images with increased artifacts in Figure 2 and Figure 3. Instead, our SRTTA identifies the noise degradation in the test image of Figure 3, resulting in images with significantly reduced noise levels. Moreover, the KernelGAN+ZSSR model is extremely slower than our SRTTA (231.41 vs. 5.38 seconds/image).\n\n> Q9. Will the pre-trained model also be trained?\n\nA9: During the test-time adaptation (TTA), we freeze all parameters of the pre-trained SR model, and we only update the TTA model (SRTTA). To avoid ambiguity, we will revise the paper accordingly."}, {"review_id": "ZrvrzG1y5x", "reviewer": "Reviewer_aSmb", "summary": "The submission presents Test-time adaptation Super Resolution (SRTTA) via learning from the second-order degradation and reconstruction. This method super-resolve images with various degradation types while maintaining high efficiency. In addition, the paper also introduce a new dataset DIV2K-C, C stands for corruption, to better depict various image corruptions in the real-world SR use cases. The method is compared to other SOTA based this new dataset and achieves better metrics.", "questions": "Since your degradation classifier is to determine the types of the degradation, which is usually a local effect, what's the reason of putting the whole image into the classifier (I assume you also have to resize your input image to 224x224) instead of putting patches of your input image into the classifier?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "There are 2 major contributions in this submission:\n1) Adaptive Parameter Preservation module for the test time adaptation;\n2) DIV2k-C dataset.", "weaknesses": "I'm really doubt about the problem setting. First, modern ISP always removes noises like Poisson noise, most images are clean besides some compression artifacts. Second, as the authors admit, if an image has some corruptions, it usually has more than 1 type of corruptions. The submitted method based on a classifier to choose at most 1 types of corruption does not follow the real-world scenarios.  Both of these two weakness challenges the fundamental design of the method.\nIn addition, the method is only evaluated on the new dataset created by the authors, which is far from enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. **Significance of the problem setting**: I'm really doubt about the problem setting. Modern ISP always removes noises like Poisson noise, most images are clean besides some compression artifacts.\n\nA1: In this paper, we seek to adapt a trained super-resolution (SR) model to new domains with unseen degradations. This problem is particularly practical and has gained great attention in the SR field [3], as the real-world environment may dynamically change and the collected images may continuously suffer from unseen degradations. We further clarify the degradation types considered in our SRTTA as follows:\n\n- **Noise degradations indeed commonly exist in the real-world collected images**. For example, traditional cameras may capture a noisy image with a high ISO value [1] or a short exposure time [2]. Moreover, ISP may introduce more complex degradation on the coloured images, which are unpredictable and varying among different devices [3].\n\n- **All degradation types considered in our SRTTA have been widely explored by prior SR methods [4,5]** or image recognition methods [6]. Here, our difference from prior SR methods is that the degradation scenarios we considered are more complex (total 8 degradations). In contrast, prior SR methods investigate a limited number of degradations (e.g., 1 type in [7] and 2 types in [8]). In this sense, our SRTTA is more practical for real-world applications.\n\n[1] Benchmarking denoising algorithms with real photographs, CVPR 2017.\n[2] Learning to see in the dark, CVPR 2018.  \n[3] Blind image super-resolution: A survey and beyond. TPAMI 2022.  \n[4] Designing a practical degradation model for deep blind image super-resolution, ICCV 2021.  \n[5] Real-ESRGAN: Training real-world blind super-resolution with pure synthetic data, ICCV 2021.  \n[6] Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. ICLR 2019.  \n[7] Unfolding the Alternating Optimization for Blind Super Resolution, NeurIPS 2020.  \n[8] Learning a single convolutional super-resolution network for multiple degradations, CVPR 2018.\n\n> Q2. The SRTTA is inapplicable to domains with mixed multiple degradations.\n\nA2: Actually, our SRTTA framework can be applied to more complex real-world scenarios with mixed multiple degradations. In our experiments, we have investigated our SRTTA performance on 8 domains with single degradation. In the rebuttal, we further evaluate our SRTTA on four domains with mixed multiple degradations, including BlurNoise, BlurJPEG, NoiseJPEG and BlurNoiseJPEG. Please refer to General Response 2 for more details and results. Experimental results show that our SRTTA achieves an impressive improvement over existing methods in these domains, further demonstrating the effectiveness of our SRTTA.\n\n\n> Q3. The method is only evaluated on the new dataset created by the authors, which is far from enough.\n\nA3: We indeed have shown the experimental results on a real-world test dataset DPED[1] in Figure 3 of the paper and Figure 5 of the supplementary materials, which shows the effectiveness of our SRTTA in real-world scenarios. To further demonstrate the effectiveness of our method, we conduct experiments on more real-world datasets, including OST300[2] and ADE20K[3]. As shown in **Figure A of the supplementary PDF file accompanying this rebuttal**, experimental results demonstrate that our SRTTA is able to generate images with fewer artifacts for real-world test images.\n\n[1] DSLR-quality photos on mobile devices with deep convolutional networks, ICCV 2017.  \n[2] Recovering realistic texture in image super-resolution by deep spatial feature transform, CVPR 2018.  \n[3] Semantic understanding of scenes through the ADE20K dataset, IJCV 2019.\n\n> Q4. Since your degradation classifier is to determine the types of **the degradation, which is usually a local effect**, what's the reason of putting the whole image into the classifier (I assume you also have to **resize your input image** to 224x224) instead of putting patches of your input image into the classifier?\n\nA4: In our SRTTA, we input the whole image **without resize operation** (named **Whole Image**) into the classifier for degradation determination. However, as you suggested, we can also input the patches of the image into the degradation classifier (named **Patches**) to recognize the local effect. Here, we further compare these two types of inputs (**Whole Image** and **Patches**) in Table A. Results show that taking Patches as inputs achieves lower degradation classification accuracy, resulting inferior adaptation performance on the DIV2K-C dataset. \n\n\nTable A. Results of our SRTTA with different inputs of the degradation classifier. For Patches, we average the predicted results of all patches as the final predicted degradation type.\n\n| Input of the classifier | Accuracy | DIV2K-C  PSNR/SSIM | Total Inference Time  seconds/image |\n| :---: | :---: | :---: | :---: |\n| Whole Image | 0.9400 | 27.78/0.7290 | 5.38 |\n| Patches | 0.9122 | 27.64/0.7233 | 5.41 |"}, {"review_id": "uDBytU1piy", "reviewer": "Reviewer_tgyp", "summary": "This paper proposed a fast test time adaption method for image SR. The authors point out that the existing image SR method use synthesized paired LR-HR data for training, which degradation type has a gap with the real-world sceneries. The authors also proposed a second-order based data synthesis circuit with a degradation classifier. The proposed method achieves impressive results on both real-world and synthesis data.\n\nAfter reading the author's response, most of my problems have been addressed. Therefore, I would like to increase my initial score.", "questions": "Here are some detailed comments and suggestions for improving the quality of this paper：\n1.\tIn the abstract, I think it’s better to say ‘conventional SR’ or ‘PSNR/fidelity oriented SR’use paired data for training.\n2.\tNovelty problems, second order degradation is already proposed by real-ESRGAN, the authors need to enhance the explanation of the contribution of this paper.\n3.\tIt is better to give a definition of the second-order degraded image like Equation.1\n4.\tThe degradation category definition is a discrete single label, second order requires repetition and mixing, so is the design of the classifier reasonable? Did the authors consider the sequence of the degradations?\n5.\tThe diagonal Fisher Information matrix in Section 4.3 requires independent identical distribution, is the image identically distributed under different degenerations? Equation 7 only reflects the stability of the same degradation under different augmentations, which does not seem to be able to target different degradations. In Equation 8, the parameter selection part I think is also not reasonable enough, through the artificial design of hyperparameters is usually difficult to meet the different datasets, so it is recommended to let the parameter be learnable.\n6.\tThe visualization results look a bit over-smoothed, it is suggested to add some analysis to it.\n", "limitations": "See the above comments.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Strengths:\nExperimental results show that the proposed method is effective for real-world image super-resolution.\nVisual results are clear to show the improvement.", "weaknesses": "Weaknesses:\n1. The concept of second-order degradations is already proposed by real-ESRGAN in 2021.\n2. Please see the following detailed comments and suggestions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. In the abstract, it's better to say 'conventional SR' or 'PSNR/fidelity oriented SR' use paired data for training.\n\nA1: We will revise the sentence of the abstract in the revised paper to avoid misleading.\n\n> Q2. Differences from Real-ESRGAN[1].\n\nA2: We would like to highlight the **difference** between our SRTTA with Real-ESRGAN in the following two folds:\n\n- **Solving different problems:** \nReal-ESRGAN tries to enumerate all the degradations in real-world scenes and train an SR model to solve the image restoration on any degradation. However, it is non-trivial to obtain all real-world degradations, **leading to domain shift issues when encountering unseen degradations** during testing, as shown in Figure 11 of real-ESRGAN [1].  \nUnlike real-ESRGAN, our SRTTA aims to **adapt** the SR models to **different domains** when test images contain unknown/unseen degradations. Our second-order degradation scheme aims to construct the **pseudo paired data** to quickly update the SR model to the test domains.\n\n- **Different construction schemes:** \nReal-ESRGAN proposes the second-order degradation to construct the paired training data, whose low-resolution images are obtained from the ground-truth HR images. Then, the paired data is used to train an SR model in the **training phase** in a **supervised learning** manner, and the Real-ESRGAN model is fixed during the test time. \nInstead, our second-order degradation constructs the paired data using the test images with degradation (first-order degraded images). And our SRTTA model updates the SR model in a **self-supervised** manner to different domains during **testing**.\n\nPlease refer to General Response 1 for a comprehensive explanation of the novelty and the contributions of our paper.\n\n[1] Real-ESRGAN: Training real-world blind super-resolution with pure synthetic data, ICCV 2021.\n\n\n> Q3. It is better to give a definition of the second-order degraded image like Equation.1\n\nA3: We will add a definition of the second-order degraded image in the revised paper.\n\n> Q4. The degradation category definition is a discrete single label, second order requires repetition and mixing, so is the design of the classifier reasonable? Did the authors consider the sequence of the degradations?\n\nA4: Our SRTTA is able to adapt to a wide range of degradation shifts. Indeed, the single-label classifier can limit the adaptation of SR models on the domains with mixed multiple degradations. To address this issue, we change the classifier to a multi-label classifier, which predicts the probabilities of multiple degradation types simultaneously. The results in Table A of General Response 2 demonstrate the effectiveness of our SRTTA with the multi-label classifier on the domains with mixed multiple degradations.\n\nWhen considering the sequence of the degradation in the second-order degradation process, it may contribute to more complex test domains, and we leave this investigation to our future work.\n\n> Q5. The diagonal Fisher Information matrix in Section 4.3 requires independent identical distribution, is the image identically distributed under different degenerations? Equation 7 only reflects the stability of the same degradation under different augmentations, which does not seem to be able to target different degradations.\n\nA5: Our SRTTA is able to preserve important parameters regardless of different degradation. The distribution of test images under different degradations may be different from each other. However, as explained in lines 211 to 212, we only use Equations (6) and (7) to compute the diagonal Fisher Information matrix once on the clean domain. **Regardless of different degradations**, we use the computed matrix to select important parameters to freeze during the whole test-time adaptation procedure.\n\n> Q6. In Equation 8, through the artificial design of hyperparameters is usually difficult to meet the different datasets, so it is recommended to let the parameter be learnable.\n\nA6: Our adaptive parameter preservation strategy only involves a single hyperparameter $\\rho$ in Equation 8, which is not sensitive during the test-time adaptation. As shown in Table 4 of the paper, when $\\rho$ is set within the range [0.2, 0.7], the results of our SRTTA-lifelong are slightly different, with fluctuations within 0.08 dB on the DIV2K-C test dataset. Consequently, we set $\\rho$ to 0.5 for all datasets and domain data with different degradations.\n\nWe agree that learning the parameter for each specific dataset can further improve the TTA performance, and we leave this to our future work.\n\n> Q7. The visualization results look a bit over-smoothed, it is suggested to add some analysis to it.\n\nA7: Our SRTTA is able to remove the degradation of real-world images and generate images with fewer artifacts, as shown in Figure 2 and Figure 3 of the paper. The results in Figure 2(a) look a bit over-smoothed, which may be caused by the improper removal of the original high-frequency information from the test image. Notably, denoising aims to eliminate the useless/noisy high-frequency information, which may inadvertently remove the original high-frequency information from sharp regions and lead to over-smoothed textures [1].  \nMoreover, our second-order reconstruction loss is based on the Charbonnier penalty loss, which may result in generated images with over-smooth edges [2]. To compensate for the loss of high-frequency information, one can combine GAN loss [3] and perceptual loss [4] with our second-order reconstruction loss, following existing methods [3]. We leave this to our future work.\n\n[1] Accurate and fast image denoising via attention guided scaling, TIP, 2021.  \n[2] Recovering realistic texture in image super-resolution by deep spatial feature transform. CVPR 2018.  \n[3] Photo-realistic single image super-resolution using a generative adversarial network. CVPR 2017.  \n[4] Perceptual losses for real-time style transfer and super-resolution, ECCV 2016."}, {"review_id": "gkas4aGtpz", "reviewer": "Reviewer_G7Pf", "summary": "The paper proposes a test-time adaptation framework called SRTTA for image super-resolution (SR) to address the degradation shift issue between training and test images. The SRTTA framework adapts a pre-trained SR model to different degradation types observed in real-world test images. It introduces a second-order degradation scheme that generates paired data based on the degradation type predicted by a pre-trained degradation classifier. The SR model is then adapted through feature-level reconstruction learning from the initial test image to its second-order degraded counterparts. Experimental results on a newly synthesized DIV2K-C dataset and a real-world DPED dataset demonstrate the effectiveness and practicality of the SRTTA framework.", "questions": "refer to the weakness", "limitations": "yes, the authors have addressed some limitations of the paper", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper introduces a new framework, SRTTA, that allows rapid adaptation of a pre-trained SR model to various degradation types observed in test images. The second-order degradation scheme and feature-level reconstruction learning provide a practical and effective solution for adapting the model to real-world scenarios.\n\nUnlike some existing methods that focus on a single degradation type, SRTTA considers multiple degradation types, such as blur, noise, and JPEG artifacts, making it more applicable in real-world scenarios where images can have diverse degradation types.\n\nSRTTA achieves fast adaptation by using only a single degraded low-resolution image during testing. This makes it more practical for real-time applications compared to methods that require extensive iterations or GAN-based approaches for degradation estimation.", "weaknesses": "1. Although the paper claims to evaluate the framework on a real-world DPED dataset, the evaluation results and analysis are not sufficient, in the per. The lack of comprehensive evaluation on diverse real-world datasets limits the generalizability and applicability of the proposed framework.\n\n2. The paper focuses on adapting the pre-trained SR model to different degradation types using second-order degradation. However, there is limited discussion or analysis on the generalization capability of the adapted model to unseen degradation types. It would be valuable to investigate the model's performance when faced with degradation types not encountered during training, or different types combine the degradations.\n\n3. The paper primarily focuses on evaluating the performance of the proposed method in terms of PSNR (Peak Signal-to-Noise Ratio). While PSNR is a commonly used metric, it does not always correlate well with perceived image quality. The evaluation could have included other quality metrics, such as Mean Opinion Score (MOS) or other human eye-related metrics, to provide a more comprehensive assessment of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. Evaluation merely on a real-world dataset limits the generalizability and applicability of the proposed framework.\n\nA1: Our SRTTA is able to remove the degradation of real-world images and generate images with fewer artifacts, as shown in Figure 3 of the paper and Figure 5 of the supplementary materials. To comprehensively demonstrate the effectiveness of our SRTTA, we further evaluate our SRTTA on two real-world datasets, including OST300[1] and ADE20K[2]. As shown in **Figure A of the supplementary PDF file accompanying this rebuttal**, results demonstrate the superiority of our SRTTA, resulting in the generated images with fewer artifacts.\n\n[1] Recovering realistic texture in image super-resolution by deep spatial feature transform, CVPR 2018.  \n[2] Semantic understanding of scenes through the ADE20K dataset. IJCV 2019.\n\n>  Q2. There is limited discussion or analysis on the generalization capability of the adapted model to unseen degradation types.\n\nA2: Our SRTTA framework aims to adapt the pre-trained model to different domains with unseen degradations. For example, the pre-trained EDSR is trained on the domain of clean images, and our SRTTA is able to adapt the pre-trained EDSR model to the unseen domain such as Gaussian blur or JPEG during the test time, as shown in Table 1 of the paper. Furthermore, we have shown the visualization results on the real-world test images with unseen/unknown degradations, as shown in Figure 3 of the paper and Figure 5 of the supplementary materials.\n\n**Effectiveness on the unseen domain:** To further evaluate our SRTTA on unseen degradation, we evaluate our SRTTA on a new unseen domain with the degradation of processed camera sensor noise[2, 3], which is not used in the training phase of the SR model or that of the degradation classifier. In Table A, results show that our SRTTA method is also able to improve the model performance on this unseen degradation.\n\n**Effectiveness on the domains with mixed multiple degradations:** We also evaluate our SRTTA on the domain with mixed multiple degradations, where we change the degradation classifier to predict the probabilities of blur, noise, and JPEG degradation simultaneously. Experimental results in Table A of General Response 2 demonstrate the effectiveness of our SRTTA on these domains. \n\nTable A. Results of different methods on **unseen** domain with the degradation of processed camera sensor noise [2,3]. We report the PSNR and SSIM values of different methods on 100 images with random processed camera sensor noise.\n\n| Methods | PSNR | SSIM |\n|---|---|---|\n| SwinIR | 19.45 | 0.4960 |\n| IPT | 19.51 | 0.5005 |\n| HAT | 21.52 | 0.5957 |\n| DDNM | 19.63 | 0.5179 |\n| DAN | 21.53 | 0.5979 |\n| DCLS-SR | 21.57 | 0.6049 |\n| DualSR | 21.14 | 0.5861 |\n| MZSR | 20.40 | 0.4379 |\n| ZSSR | 21.57 | 0.6214 |\n| ZSSR+KernelGAN | 20.60 | 0.5430 |\n| EDSR | 21.56 | 0.6008 |\n| SRTTA (Ours) | **21.76** | **0.6503** |\n\n[1] DSLR-quality photos on mobile devices with deep convolutional networks, ICCV 2017.  \n[2] Unprocessing images for learned raw denoising, CVPR 2019.  \n[3] Designing a Practical Degradation Model for Deep Blind Image Super-Resolution, ICCV 2021.\n\n\n> Q3. Lack of evaluation results in terms of human eye-related metrics.\n\nA3: Our SRTTA is able to remove the degradation and generate images with better visual quality. To provide a more comprehensive assessment, we further evaluate different methods in terms of the **Fréchet Inception Distance (FID)** [1] and the **Learned Perceptual Image Patch Similarity (LPIPS)** distance [2], which correlate well with perceived image quality and are commonly used to evaluate the quality of generated images [3, 4]. Note that a lower FID means that the visual quality and distribution distance of the generated image is more similar to those of the ground-truth image. And a lower LPIPS value means the generated image is more perceptually similar to the ground-truth image.\n\nWe evaluate different methods in terms of FID[1] and LPIPS[2] on two datasets, including **DIV2K-C** and **DIV2K-MC**. Note that **DIV2K-MC** contains four domains with a combination of multiple degradations, including BlurNoise, BlurJPEG, NoiseJPEG, and BlurNoiseJPEG. The domain \"BlurNoiseJPEG\" indicates that the test images in this domain contain the combined degradation of blur, noise and JPEG simultaneously. As shown in Table B, our SRTTA with a multi-label classifier (SRTTA-MC) achieves the lowest values of both FID and LPIPS scores, demonstrating our SRTTA is able to generate images with higher visual quality.\n\nTable B. Comparison results of our SRTTA with different methods in terms of FID[1] and LPIPS[2]. The reported FID/LPIPS values are the averages of all test images from the two datasets DIV2K-C and DIV2K-MC, respectively. Lower values indicate better performance.\n\n| Methods | DIV2K-C | DIV2K-MC |\n|:---:|:---:|:---:|\n| HAT | 64.92 / 0.2352 | 60.73 / 0.2640 |\n| SwinIR | 72.90 / 0.2441 | 60.62 / 0.2781 |\n| IPT | 68.22 / 0.2345 | 58.24 / 0.2453 |\n| KernalGAN+ZSSR | 88.28 / 0.2160 | 80.19 / 0.2371 |\n| MZSR | 68.27 / 0.2085 | 162.72 / 0.4463 |\n| DDNM | 70.80 / 0.2101 | 59.64 / 0.2083 |\n| DAN | 73.59 / 0.2260 | 56.96 / 0.2263 |\n| DCLS-SR | 83.44 /0.2472 | 57.93 / 0.2299 |\n| ZSSR | 56.66 / 0.1931 | 52.78 / 0.2152 |\n| EDSR | 69.70 / 0.2242 | 57.95 / 0.2338 |\n| SRTTA-MC(Ours) | 54.13 / **0.1927** | **37.71 / 0.1787** |\n| SRTTA-lifelong-MC(Ours) | **51.25** / 0.1964 | 45.68 / 0.1988 |\n\n[1] GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium, NeurIPS 2017.  \n[2] The unreasonable effectiveness of deep features as a perceptual metric, CVPR 2018.  \n[3] Improving diffusion models for inverse problems using manifold constraints, NeurIPS 2022.  \n[4] Designing a practical degradation model for deep blind image super-resolution, ICCV 2021."}, {"review_id": "l84Q8RMGqB", "reviewer": "Reviewer_QTZE", "summary": "This paper proposes a fast test-time adaptation SR framework named SRTTA, which is capable of super-resolving images with various degradation types while maintaining efficiency. Specifically, the authors first exploit a pre-trained degradation classifier to predict the degradation type of the test image and construct the paired dataset based on the degradation. Then, the authors adapt the SR model by conducting the feature-level reconstruction learning from the test image and its second-order degraded image generated by the designed second-order degradation scheme. Finally, the authors design the second-order reconstruction loss to adapt the pre-trained model to test images with different degradation types.\n\nAfter reviewing the rebuttal, I've noticed that most of my previous concerns have been addressed. Therefore, I would like to increase my initial score.", "questions": "1. The authors should clarify the 1st and 2nd concerns in section weaknesses. This seems to be an important reason why the proposed method is superior to other SOTA methods.\n2.  A detailed explanation or citation for the domain shifts should be given in line 136 on page 3.\n3. The authors should clarify the 3rd concern in section weaknesses and provide more convincing visualization.\n4. The authors mention that some previous methods are quite time-consuming in the motivation, so the authors should clarify the 4th concern in section weaknesses.\n5. How does the proposed network work when the test images are degraded by multiple degradation types at the same time? \n6. The motivation mentioned from line 158 to line 160 in section 4.1 should be further demonstrated or cited.", "limitations": "The application scenario of the proposed method is limited for the single degradation.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper provides an interesting image super-resolution idea for different degradations in the real-world test-time scenario. Compared with the classical deep network, it is applicable to any pretrained SR model. The proposed SRTTA can adapt a given SR model to estimate the degradation shifts in the test images. In addition, the authors construct a dataset named DIV2K-C with 8 different degradation types.", "weaknesses": "1. The proposed framework seems to be oriented towards a single degradation from different degradation types in the real-world test-time scenario. However, for the real-world scenario, the practical degradation is much more complex and could be the combinations of different degradations. This paper may not be applicable to this situation.\n2. The proposed dataset also may not be suitable for the aforementioned situation.\n3. The methods for visualization comparison are older compared to the methods for quantitative comparison.\n4. The GPU Time of the proposed method in Table 1 is not the optimal.\n5. The notation in section Super-Resolution Test-Time Adaptation should be consistent and continuous, such as x_t in line 149.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. Lack of evaluation on domains with mixed multiple degradations.\n\nA1: In our paper, we seek to adapt pre-trained SR models to target domains when real-world images contain different degradations, which can severely limit the application of most SR models. To this end, we propose a novel SRTTA framework to adapt SR models to domains with unseen degradations during testing.\n\nOur SRTTA framework is applicable to real-world test images with mixed multiple degradations by substituting the degradation classifier with a multi-label classifier. More details and experimental results can be referred to General Response 2. From the results, our SRTTA achieves the best performance on mixed multiple degradations, which demonstrates the effectiveness of our SRTTA framework in these complex scenarios.\n\n> Q2. The proposed dataset does not contain test images with mixed multiple degradations.\n\nA2. In the rebuttal, we further construct a dataset named **DIV2K-MC**, which consists of four domains with mixed multiple degradations, including **BlurNoise**, **BlurJPEG**, **NoiseJPEG**, and **BlurNoiseJPEG**. The test images in the **BlurNoiseJPEG** domain contain the combined degradation of blur, noise and JPEG simultaneously. Experimental results on this DIV2K-MC dataset further demonstrate the effectiveness of our SRTTA on domains with mixed multiple degradations. Please refer to General Response 2 for more experimental results.\n\n> Q3. The methods for visualization comparison are older compared to the methods for quantitative comparison.\n\nA3: We indeed have included comprehensive visualization comparisons of different methods in Figures 3, 4, and 5 of the supplementary materials. As shown in these figures, our SRTTA is able to reduce the degradation from the test images and generate more plausible HR images.\n\n> Q4. The GPU Time of the proposed method in Table 1 is not the optimal.\n\nA4: In our paper, we seek to adapt a pre-trained SR model to domains with unseen degradations during testing. Due to the additional time for test-time adaptation, our SRTTA cannot achieve the best inference speed. However, those methods that require fewer inference times are trained on domains with Gaussian blur degradation only, such as DAN, DCLS-SR, and MZSR. Consequently, they often suffer from the domain shift issue when applied to other domains, such as domains with Gaussian noise or JPEG degradations. Instead, with comparable efficiency, our SRTTA achieves an impressive improvement on average for all domains (Refer to Table 1 of the paper). In conclusion, our SRTTA achieves a better tradeoff between performance and efficiency.\n\n> Q5. Typos in line 149.\n\nA5: We will carefully check the notations in the revision.\n\n> Q6. Lack of relevant citations for the domain shifts.\n\nA6: We will discuss and cite the related reference [1,2,3] in the revised paper.\n\n[1] WILDS: A benchmark of in-the-wild distribution shifts, ICML 2021.  \n[2] Continual test-time domain adaptation, CVPR 2022.  \n[3] Blind image super-resolution: A survey and beyond, TPAMI 2022.\n\n> Q7. How does the proposed network work when the test images are degraded by multiple degradation types at the same time?\n\nA7: Our SRTTA is applicable to the domain with single degradation or mixed multiple degradations. To adapt SR models to the domains with mixed multiple degradations, we change the degradation classifier to a multiple-label classifier. The multi-label classifier predicts the probabilities of blur, noise and JPEG degradation simultaneously for each test image. With the predicted multiple degradation types, we degrade the test images $\\mathbf{x}$ to generate corresponding second-order degraded images $\\mathbf{x}_{sd}$ based on the classical degradation model [1]. Experimental results in Table A of General Response 2 demonstrate the effectiveness of our SRTTA on the domains with mixed multiple degradations.\n\n[1] Real-ESRGAN: Training real-world blind super-resolution with pure synthetic data, ICCV, 2021.\n\n> Q8. The motivation of the second-order degradation scheme should be further demonstrated or cited.\n\nA8: In real-world scenarios, when the domain shift issue occurs, the clean images corresponding to test images are often unavailable, making the SR models hard to learn how to remove the degradation from the test images. In this context, the key challenge is how to effectively construct (pseudo) paired data to adapt SR models to the target domain. To this end, the commonly used technique is to construct paired data with the second-order degradation scheme, which has been proven to be highly effective by previous works [1, 2].\n\nHowever, these methods [1, 2] primarily concentrate on estimating blur degradation while neglecting other degradations, such as noise and JPEG degradation. Moreover, these methods often require numerous iterations to accurately estimate the blur degradation model, leading to significant time consumption. In this work, we address these two limitations by proposing our SRTTA framework. We will provide a clearer explanation and the citation in the revised paper.\n\n[1] Blind super-resolution kernel estimation using an internal-GAN, NeurIPS 2019.  \n[2] Flow-based kernel prior with application to blind super-resolution, CVPR 2021."}]

Author rebuttal:
# **General Response**

> G1. Significance and novelty of our SRTTA.

In this paper, we seek to adapt a trained super-resolution (SR) model to domains with unseen degradations. This problem is practical and challenging, as the real-world environment may dynamically change and the collected images may continuously suffer from unseen degradations. To address this, we propose a novel test-time adaptation framework for SR models, which achieves an impressive improvement over existing methods with satisfying speed. We would like to highlight our **novelty** and **contributions** in the following three folds:

- **A novel test-time learning framework for SR model adaptation.** We propose a super-resolution test-time adaptation (SRTTA) framework to adapt any pre-trained SR models to different target domains during the test time. Without accessing any ground-truth high-resolution image, our SRTTA is applicable to practical scenarios with unknown degradation in a self-supervised manner.

- **A fast data construction scheme with second-order degradation.** We use a pre-trained classifier to identify the degradation type from test images and construct the paired data using the second-order degradation scheme. Since it is unnecessary to estimate the parameters of the degradation model, our second-order degradation scheme enables a rapid model adaptation to a wide range of degradation shifts.

- **A new dataset with eight different domains.** We construct a new test dataset named DIV2K-C, which contains eight common domains, to comprehensively evaluate the practicality of different SR methods. Previous methods mainly investigate the SR performance on a limited number of domains, while we generalize to more domains, which is more practical for real-world applications.

> G2. Good performance of our SRTTA on test images with mixed multiple degradations.

Our SRTTA can be definitely applied to super-resolve images with mixed multiple degradations and achieve excellent performance. To verify this, we extend the degradation classifier in SRTTA to a multi-label classifier, which simultaneously predicts the probabilities of blur, noise and JPEG degradation. Then, based on the predicted multiple degradations, we construct the second-order degraded image that contains multiple degradations for test-time model learning. We name this variant SRTTA-MC. 

From the results in Table A, our SRTTA-MC achieves the best performance on 4 domains with different mixed degradations, e.g., 0.6187 (DualSR) $\rightarrow$ 0.7661 (our SRTTA-MC) regarding the average SSIM metric. These results further validate the effectiveness of our proposed methods.

Table A. Comparison with prior methods on **DIV2K-MC** dataset, which consists of four domains with the combination of multiple degradations. The term "BlurNoiseJPEG" indicates that the test images in this domain contain the combined degradation of blur, noise and JPEG simultaneously. We report the **PSNR($\uparrow$)/SSIM($\uparrow$)** values of different methods.

|     Methods |    BlurNoise | BlurJPEG | NoiseJPEG | BlurNoiseJPEG | Mean |
|:---:|:---:|:---:|:---:|:---:|:---:|
| HAT | 23.41/0.3992 | 28.86/0.7884 | 25.69/0.5721 | 24.42/0.5017 | 25.59/0.5654 |
| SwinIR | 20.91/0.3111 | 26.83/0.7481 | 23.86/0.5228 | 22.77/0.4502 | 23.59/0.5081 |
| IPT | 21.28/0.3273 | 26.83/0.7477 | 24.15/0.5346 | 22.96/0.4586 | 23.81/0.5171 |
| MZSR | 18.73/0.2129 | 24.90/0.6670 | 20.37/0.3977 | 20.62/0.3537 | 21.16/0.4078 |
| DualSR | 25.59/0.5613 | 28.24/0.7867 | 23.78/0.5855 | 24.62/0.5412 | 25.56/0.6187 |
| DAN | 24.14/0.4383 | 28.95/0.7907 | 26.20/0.5931 | 24.82/0.519 | 26.03/0.5853 |
| DCLS-SR | 23.84/0.4203 | 28.93/0.7905 | 26.37/0.5991 | 24.92/0.5233 | 26.02/0.5833 |
| DDNM | 22.62/0.3887 | 26.82/0.7462 | 25.11/0.5823 | 23.81/0.5041 | 24.59/0.5553 |
| ZSSR | 24.95/0.4933 | **29.02**/0.7934 | 26.68/0.6169 | 25.24/0.5421 | 26.47/0.6114 |
| ZSSR+KernelGAN | 23.08/0.4236 | 28.32/0.7864 | 21.90/0.4741 | 22.76/0.4427 | 24.02/0.5317 |
| EDSR | 24.02/0.4299 | 28.93/0.7903 | 26.08/0.5866 | 24.73/0.5136 | 25.94/0.5801 |
| TTA-C | 24.29/0.4462 | 28.93/0.7904 | 26.35/0.5975 | 24.91/0.5223 | 26.12/0.5891 |
| SRTTA-MC(Ours) | 26.82/0.6932 | 28.95/**0.7940** | 29.10/0.7878 | 26.97/0.7227 | 27.96/0.7494 |
| SRTTA-MC-lifelong(Ours) | **26.90**/**0.7346** | 29.00/0.7915 | **29.37**/**0.8014** | **27.36**/**0.7367** | **28.16**/**0.7661** |


> G3. Good performance of our SRTTA on practical scenarios.

Our SRTTA seeks to adapt a trained image SR model to domains with unknown degradations. In this sense, our SRTTA is particularly practical as the real world often dynamically changes and the collected image may suffer from various degradations. In our experiments, we have demonstrated the effectiveness of our SRTTA on a real-world dataset DPED[1] (see Figure 3 in the main paper and Figure 5 in the supplementary materials). 

Here, we further conduct experiments on two more real-world datasets, i.e., OST300[2] and ADE20K[3], and put the visualization comparisons in **Figure A of the supplementary PDF file accompanying this rebuttal**. From the results, the generated images of our SRTTA contain less degradation, such as less noise on the flat area and textures with fewer ring artifacts. These results further demonstrate the effectiveness of our SRTTA on real-world collected images.

[1] DSLR-quality photos on mobile devices with deep convolutional networks, ICCV 2017.  
[2] Recovering realistic texture in image super-resolution by deep spatial feature transform, CVPR 2018.  
[3] Semantic understanding of scenes through the ADE20K dataset. IJCV 2019.


Meta-review: The reviewers recognize the authors' contribution to the field of super-resolution. They acknowledge the authors' explanation of how their approach can handle images with arbitrary resolutions and the supportive experimental evidence provided. The paper's proposal of a fast test-time adaptation method for image super-resolution is noted, especially in the context of addressing the degradation shift issue between training and test images. The inclusion of a second-order degradation scheme and its integration with a degradation classifier is recognized as a significant advancement. The reviewers indicate that most of their concerns have been satisfactorily addressed and express intent to increase their initial score, recommending the paper's acceptance.
Decision: Accept (poster)