Title: Efficient Test-Time Adaptation for Super-Resolution with Second-Order Degradation and Reconstruction
Venue: NeurIPS 2023
Accepted: yes
Keywords: Image Super-resolution

Test-time Adaptation

Self-supervised Learning

Second-Order Degradation
Abstract: Image super-resolution (SR) aims to learn a mapping from low-resolution (LR) to high-resolution (HR) using paired HR-LR training images. Conventional SR methods typically gather the paired training data by synthesizing LR images from HR images using a predetermined degradation model, e.g., Bicubic down-sampling. However, the realistic degradation type of test images may mismatch with the training-time degradation type due to the dynamic changes of the real-world scenarios, resulting in inferior-quality SR images. To address this, existing methods attempt to estimate the degradation model and train an image-specific model, which, however, is quite time-consuming and impracticable to handle rapidly changing domain shifts. Moreover, these methods largely concentrate on the estimation of one degradation type (e.g., blur degradation), overlooking other degradation types like noise and JPEG in real-world test-time scenarios, thus limiting their practicality. To tackle these problems, we present an efficient test-time adaptation framework for SR, named SRTTA, which is able to quickly adapt SR models to test domains with different/unknown degradation types. Specifically, we design a second-order degradation scheme to construct paired data based on the degradation type of the test image, which is predicted by a pre-trained degradation classifier. Then, we adapt the SR model by implementing feature-level reconstruction learning from the initial test image to its second-order degraded counterparts, which helps the SR model generate plausible HR images. Extensive experiments are conducted on newly synthesized corrupted DIV2K datasets with 8 different degradations and several real-world datasets, demonstrating that our SRTTA framework achieves an impressive improvement over existing methods with satisfying speed. The source code is available at https://github.com/DengZeshuai/SRTTA.

Efficient Test-Time Adaptation for Super-Resolution with Second-Order Degradation and Reconstruction

Abstract

Image super-resolution (SR) aims to learn a mapping from low-resolution (LR) to high-resolution (HR) using paired HR-LR training images. Conventional SR methods typically gather the paired training data by synthesizing LR images from HR images using a predetermined degradation model, e.g. , Bicubic down-sampling. However, the realistic degradation type of test images may mismatch with the training-time degradation type due to the dynamic changes of the real-world scenarios, resulting in inferior-quality SR images. To address this, existing methods attempt to estimate the degradation model and train an image-specific model, which, however, is quite time-consuming and impracticable to handle rapidly changing domain shifts. Moreover, these methods largely concentrate on the estimation of one degradation type ( e.g. , blur degradation), overlooking other degradation types like noise and JPEG in real-world test-time scenarios, thus limiting their practicality. To tackle these problems, we present an efficient test-time adaptation framework for SR, named SRTTA, which is able to quickly adapt SR models to test domains with different/unknown degradation types. Specifically, we design a second-order degradation scheme to construct paired data based on the degradation type of the test image, which is predicted by a pre-trained degradation classifier. Then, we adapt the SR model by implementing feature-level reconstruction learning from the initial test image to its second-order degraded counterparts, which helps the SR model generate plausible HR images. Extensive experiments are conducted on newly synthesized corrupted DIV2K datasets with 8 different degradations and several real-world datasets, demonstrating that our SRTTA framework achieves an impressive improvement over existing methods with satisfying speed. The source code is available at https://github.com/DengZeshuai/SRTTA.

1 Introduction

Image super-resolution (SR) aims to reconstruct plausible high-resolution (HR) images from the given low-resolution (LR) images, which is widely applied in microscopy [42, 44], remote sensing [17, 35] and surveillance [40, 56]. Most previous SR methods [11, 31, 60, 8] hypothesize that the LR images are downsampled from HR images using a predefined degradation model, e.g. , Bicubic down-sampling. However, due to the diverse imaging sensors and multiple propagations on the Internet, real-world images may contain different degradation types ( e.g. , Gaussian blur, Poisson noise, and JPEG artifact) [32, 52, 53, 25]. Besides, the realistic degradations of real-world images

may dynamically change, which are often different from the training one, limiting the performance of pre-trained SR models in dynamically changing test-time environments.

Recently, zero-shot SR methods [43, 9, 12] are proposed to train an image-specific SR model for each test image to alleviate the degradation shift issue. For example, ZSSR [43] uses a predefined/estimated degradation model to generate an image with a lower resolution for each test image. With this downsampled image and the test image, they can train an image-specific SR model to super-resolve the test image. Moreover, DualSR [12] estimates the degradation model and trains the SR model simultaneously to achieve better performance. However, these methods usually require thousands of iterations to estimate the degradation model or train the SR model, which is very time-consuming. Thus, these methods cannot handle real-world test images with rapidly changing domain shifts.

To reduce the inference time of zero-shot methods, some recent works [45, 41] introduce metalearning [13] to accelerate the adaptation of the SR model, which still requires a predefined/estimated degradation model to construct paired data to update the model. However, most degradation estimation methods [2, 30] focus on the estimation of one degradation type, which limits the adaptation of SR models to test images with other degradations. Recently, test-time adaptation (TTA) methods [46, 47, 57, 38, 48, 39] are proposed to quickly adapt the pre-trained model to the test data in target domain without accessing any source training data. These methods often use simple augmentation operations ( e.g. , rotation or horizontal flip) on the test image, and construct the pseudo label as the average of the predicted results [57, 48]. For image SR, the pseudo-HR image constructed using this scheme [57, 48] may still contain the degradation close to the test image ( e.g. , Gaussian blur). With such a pseudo-HR image, the adapted SR model may not be able to learn how to remove the degradation from the test image (see results in Table 1). Therefore, how to quickly and effectively construct the paired data to encourage the SR model to remove the degradation is still an open question.

In this paper, we propose a super-resolution test-time adaptation framework (SRTTA) to adapt a trained super-resolution model to target domains with unknown degradations, as shown in Figure 1. When the degradation shift issue occurs, the key challenge is how to quickly and effectively construct (pseudo) paired data to adapt SR models to the target domain without accessing any clean HR images. To this end, we propose a second-order degradation scheme to construct (pseudo) paired data. Specifically, with a pre-trained degradation classifier, we quickly identify the degradation type from the test images and randomly generate a set of degradations to obtain the second-order degraded images. The paired data, which consists of the second-order degraded images and the test image, enables a rapid adaptation of SR models to the target domain with different degradations. To facilitate the learning of reconstruction, we design a second-order reconstruction loss to adapt the pre-trained model using the paired data in a self-supervised manner. After fast adaptation using our method, the SR model is able to learn how to remove this kind of degradations and generate plausible HR images. Moreover, we also design an adaptive parameter preservation strategy to preserve the knowledge of the pre-trained model to avoid the catastrophic forgetting issue in long-term adaptation. Last but not least, we use eight different degradations to construct two new benchmarks, named DIV2K-C and DIV2K-MC, to comprehensively evaluate the practicality of our method. Experimental results on both our synthesized datasets and several real-world datasets demonstrate that our SRTTA is able to quickly adapt the SR model to the test-time images and achieve an impressive improvement.

Our main contributions are summarized as follows:

A novel test-time adaptation framework for image super-resolution : We propose a superresolution test-time adaptation (SRTTA) framework to adapt any pre-trained SR models to different target domains during the test time. Without accessing any ground-truth HR images, our SRTTA is applicable to practical scenarios with unknown degradation in a self-supervised manner.

A fast data construction scheme with second-order degradation : We use a pre-trained classifier to identify the degradation type for a test image and construct the paired data using our second-order degradation scheme. Since we do not estimate the parameters of the degradation model, our scheme enables a rapid model adaptation to a wide range of degradation shifts.

New test datasets with eight different domains : We construct new test datasets named DIV2K-C and DIV2K-MC, which contain eight common degradations, to evaluate the practicality of different SR methods. Experimental results on both synthesized datasets and real-world datasets demonstrate the superiority of our SRTTA, e.g. , 0.84 dB PSNR improvement on DIV2K-C over ZSSR [43].

2 Related Work

Real-world super-resolution. To alleviate the domain shift issues, GAN-based methods [54, 5, 24, 36] tend to learn the degradation model of the real-world images in the training stage. These methods often train a generator that explicitly learns the degradation model of real-world images. Besides, some methods [55, 60, 49] try to enumerate most of the degradation models that can be encountered in real-world applications. Based on the estimated/predefined degradation models, these methods can generate LR images whose distribution is similar to real-world images. However, due to the complex and unknown processing of real-world images, it is hard to mimic all types of degradation during the training phase. Instead, some existing methods [15, 2, 22, 21, 34] try to estimate the image-specific degradation model during the test time, which helps to reconstruct more plausible HR images. For instance, optimization-based methods [15, 22, 21] estimate the blur kernel and SR image together in an iterative manner. However, these methods cannot generate satisfactory results when the test images contain different types of degradation ( e.g. , Poisson noise and JPEG artifact) [32]. Thus, these methods still suffer from the domain shift on test images with unknown degradation.

Zero-shot super-resolution. Zero-shot methods [43, 9, 45, 41, 12] aim to train an image-specific SR model for each test image to alleviate the domain shift issue. These methods [43, 12] use a predefined/estimated degradation model to generate an image with a lower resolution from each test image. To estimate the image-specific degradation in a zero-shot manner, KernelGAN [2] utilizes the internal statistics of each test image to learn the degradation model specifically and then uses ZSSR [43] to train an SR model with the estimated degradation. However, these methods usually require a lot of time to estimate the degradation model or train the SR model. MZSR [45] and MLSR [41] are proposed to reduce the number of iteration steps for each test image during test time. Recently, DDNM [51] was proposed to use a pre-trained diffusion model to ensure the generated images obey the distribution of natural images. However, these methods still require a predefined (Bicubic downsampling) or an estimated degradation model. The predefined Bicubic degradation suffers from the domain shift due to its difference from the underlying degradation of real-world images. The estimation methods [2, 30] may focus on the estimation of a single degradation type ( e.g. , blur) while ignoring other degradation. Thus, these methods often result in unsatisfactory HR images for the test images with different degradation types [32]. In this paper, we use a degradation classifier to quickly recognize the degradation type and randomly generate the degradations with this type. Therefore, we do not need to estimate the degradation model, which is time-saving.

Test-time adaptation. Recently, test-time adaptation (TTA) methods [46, 47, 57, 38, 48, 6, 39, 59] have been proposed to alleviate the domain shift issue by online updating the pre-trained model on the test data. TTT [46] uses an auxiliary head to learn the test image information from the self-supervised task. Tent [47] proposed to adapt the pre-trained model with entropy-based loss in an unsupervised manner. CoTTA [48] uses a weight-averaged pseudo-label over training steps to guide the pre-trained model adaptation. However, these methods are mainly developed for image classification and may ignore the characteristics of image super-resolution. Thus, these methods may not be effective in adapting the SR model to remove the degradation from the test image. In this paper, we focus on the image SR task and address the degradation shift issue with our SRTTA framework.

3 Preliminary and Problem Definition

Notation. Without loss of generality, let y be a clean high-resolution (HR) image and x c be the clean low-resolution (LR) image downsampled from y using Bicubic interpolation, i.e. , x c = y ↓ s , where s is the scale factor of Bicubic downsampling. Let x denote a real-world test image degraded from y , i.e. , x = D ( y ) , where D ( · ) is the degradation process. We use x sd to denote the LR image that is further degraded from the real-world image x . In this paper, we call the test image x as a first-order degraded image and the image x sd degraded from x as a second-order degraded image. f θ ( · ) is a super-resolution (SR) model with parameters θ .

Image degradation. The degradation process of real-world test images can be modeled by a classical degradation model D ( · ) [33, 49]. Formally, let k be a blur kernel, n be an additive noise map and q be the quality factor of JPEG compression, the degraded image x is defined by

where ⊗ denotes the convolution operation, ↓ s denotes the downsampling with a scale factor of s , and JPEG q denotes the JPEG compression with the quality factor q . Similarly, the second-order

Pretrained SR Model

𝒇 𝜽 𝟎

∙

Conv

ResBlock

ResBlock

Conv

Upsample

SR Model

𝒇𝜽

∙

for TTA

Conv

ResBlock

ResBlock

Conv

Upsample

Adaptive Parameter Preservation

𝒇

𝜽 𝟎

𝒍

(𝐱)

𝒇 𝜽

𝒍

(𝐱)

Test Sample

𝐱

𝑪(∙)

𝐱𝒔𝒅

𝐱

Degradation

Classifier

Conv

Conv

Forward

Backward

ℒ𝑎

ℒ𝑎

Self-supervised

Adaptation Loss

Adaptation

Consistency Loss

𝒇 𝜽

𝒍

𝐱𝒔𝒅

𝑪(𝐱)

𝐱𝒔𝒅

Second-order

Degradation

blur

𝐱 ⊗ 𝐤

noise

𝐱 + 𝐧

JPEG

𝑱𝑷𝑬𝑮𝒒(𝐱)

clean

-

degraded image x sd can be formulated as

Degradation shift between training and testing. Existing SR methods [31, 60, 16] often construct paired HR-LR training images by either collecting from the real world or synthesizing LR images from HR images via a pre-defined degradation model, i.e. , Bicubic down-sampling. However, due to diverse camera sensors and the unknown processing on the Internet, the degradation process of real-world test images may differ from that of training images, called domain shifts [27, 48, 32]. In these cases, the SR model often fails to generate satisfactory HR images.

Motivation and challenges. Though recently some blind SR methods [12, 43] have been proposed to address the degradation shift issue, they still suffer from two key limitations: low efficiency and narrow focus on a single degradation type, e.g. , blur degradation. In this work, we seek to resolve these issues by directly learning from the shifted testing LR image at test time, which poses two major challenges: 1) How to quickly and effectively construct the (pseudo) paired data to adapt SR models to test domains with unknown degradations? and 2) How to design a generalized test-time learning framework that facilitates the removal of various types of degradation, considering that we have only a low-resolution test image at our disposal?

4 Efficient Test-Time Adaptation for Image Super-Resolution

In this section, we illustrate our proposed super-resolution test-time adaptation (SRTTA) framework that is able to quickly adapt the pre-trained SR model to real-world images with different degradations. The overall framework and pipeline are shown in Figure 1 and Algorithm 1.

Given a test image x , we first construct the paired data using our second-order degradation scheme. Specifically, we use a pre-trained degradation classifier to recognize the degradation type for the test image. Based on the predicted degradation type, we randomly generate a set of degradation ( e.g. , a set of blur kernels k ) and use them to construct a set of paired data { x i sd , x } N i =1 . With the paired data, we adapt the pre-trained SR model to remove the degradation from the test image. Notably, before performing the test-time adaptation, we freeze the important parameters to preserve the knowledge of the pre-trained model to alleviate the forgetting problem in long-term adaptation. After adaptation, we use the adapted model to generate the corresponding HR image for the test image.

4.1 Adaptive Data Construction with Second-Order Degradation

In this part, we propose a novel second-order degradation scheme to effectively construct paired data, enabling the fast adaptation of SR models to the target domain with different degradations.

Unlike existing methods [43, 2, 30], we consider more degradation types and avoid estimating the degradation model. Existing methods [43, 2, 30] mainly focus on precisely estimating the blur kernels when constructing the lower-resolution images (second-order degraded images), which is timeconsuming. Instead, we use a pre-trained degradation classifier to quickly identify the degradation types (blur, noise, and JPEG) of test images, and then we construct the second-order degraded images based on the predicted degradation types. Without the time-consuming degradation estimation process, our scheme enables a fast model adaptation to a wide range of degradation shifts.

Adaptive data construction. In this part, we design an adaptive data construction method to obtain the second-order degraded images x i sd . Specifically, based on the classical degradation model in Eqn. (1), we train a multi-label degradation classifier C ( · ) to predict the degradation types for each test image, including blur, noise and JPEG degradation types, denoted by c b , c n and c j ∈ { 0 , 1 } , respectively. With the predicted degradation types, we randomly generate N degradations and construct a set of second-order degraded images { x i sd } N i =1 , which can be formulated as

where the blur kernel k , noise map n and quality factor q are randomly generated using a similar recipe of Real-ESRGAN [49]. Unlike previous methods [43, 2], we do not further downsample the test image x when constructing x sd , since the pretrained SR model has learned the upsampling function (the inverse function of downsampling) during the training phase. Due to the page limit, we put more details in the supplementary materials.

Since the pre-trained SR model has been well-trained on the clean domain (Bicubic downsampling), we simply ignore adapting the clean images in test-time. For these images, we use the pre-trained SR model to super-resolve them, i.e. , ˆ y = f θ 0 ( x ) when c b = c n = c j = 0 .

4.2 Adaptation with Second-Order Reconstruction

In our SRTTA framework, we design a self-supervised adaptation loss and an adaptation consistency loss to update the pre-trained SR models to test images with degradation.

Self-supervised adaptation. To adapt the pre-trained model to remove the degradation, we design a self-supervised adaptation loss based on the Charbonnier penalty function [4, 28]. Specifically, we encourage the SR model to reconstruct the test images x from the second-order degraded images x sd at the feature level, which can be formulated as

where f l θ ( · ) denotes the output features of the l -th layer. We simply set f l θ ( · ) to be the output features of the second-to-last convolution layer. ϵ is a small positive value that is set to 10 -3 empirically.

Consistency maximization. To keep the model consistent across adaptation, we design an adaptation consistency loss to encourage the output of the adapted model to be close to that of the pre-trained model, which is formulated as

where f l θ 0 ( · ) denotes the output features of the l -th layer of the pre-trained SR model.

Second-order reconstruction loss. Our final second-order reconstruction loss consists of a selfsupervised adaptation loss and an adaptation consistency loss, which is formulated as

where α is a balance hyperparameter (the ablation study can be found in Table 5).

4.3 Adaptive Parameter Preservation for Anti-Forgetting

To avoid catastrophic forgetting in long-term adaptation, we propose an adaptive parameter preservation (APP) strategy to freeze the important parameters during adaptation. To select the important parameters, we evaluate the importance of each parameter using the Fisher information matrix [26, 38]. Given a set of collected clean images D c , we design an augmentation consistency loss L c to compute the gradient of each parameter. Based on the gradient, we compute the diagonal Fisher information matrix to evaluate the importance of each parameter θ 0 i , which is formulated as

where A i ∈ { A j } 8 j =1 is an augmentation operation, which is the random combination of a 90-degree rotation, a horizontal and a vertical flip on the input image x c . R i is the inverse operation of A i that rolls back the image A i ( x c ) to its original version x c . With ω ( θ 0 i ) , we select the most important parameters using a ratio of ρ and freeze these parameters during the adaptation. The set of selected parameters S can be formulated as

where τ ρ denotes the first ρ -ratio largest value obtained by ranking the value ω ( θ 0 i ) , ρ is a hyperparameter to control the ratio of parameters to be frozen. Note that we only need to select the set of significant parameters S once before performing test-time adaptation.

5 Experiments

5.1 Experimental Details

Testing data. Following ImageNet-C [18], we degraded 100 validation images from the DIV2K [1] dataset into eight domains. We select the eight degradation types that do not extremely change the image content, including Gaussian Blur, Defocus Blur, Glass Blur, Gaussian Noise, Poisson Noise (Shot Noise), Impulse Noise, Speckle Noise, and JPEG compression. In total, we create a new benchmark dataset DIV2K-C , which contains 800 images with different single degradation, to evaluate the performance of different SR methods. Besides, we further construct a dataset named DIV2K-MC , which consists of four domains with mixed multiple degradations, including BlurNoise, BlurJPEG, NoiseJPEG, and BlurNoiseJPEG. Specifically, test images from the BlurNoiseJPEG domain contain the combined degradation of Gaussian Blur, Gaussian Noise and JPEG simultaneously. Moreover, we also evaluate our SRTTA on real-world images from DPED [23], ADE20K [61] and OST300 [50], whose corresponding ground-truth HR images can not be found. To evaluate the anti-forgetting performance, we use a benchmark dataset Set5 [3].

Implementation details and evaluation metric. We evaluate our approach using the baseline model of EDSR [31] with only 1.37 M parameters for 2 × SR. To demonstrate the effectiveness of our SRTTA, we conduct experiments in two settings, including a parameter-reset setting and a lifelong setting. In the parameter-reset setting, the model parameters will be reset after the adaptation of each domain, which is the default setting of our SRTTA. In the lifelong setting, the model parameters will never be reset in the long-term adaptation, in this case, we call our methods as SRTTA-lifelong. For the balance weight in Eqn. (6), we set α to 1. For the ratio of parameters to be frozen, we set the ρ to 0.50. Please refer to more details in the supplementary materials.

To compare the inference times of different SR methods, we measure all methods on a TITAN XP with 12G graphics memory for a fair comparison. Due to the memory overload of HAT [8] and DDNM [51], we chop the whole image into smaller patches and process them individually for these two methods. To evaluate the performance of different methods, we use the common metrics PSNR [20] and SSIM [20] and report the results of all methods on the DIV2K-C dataset. Due to the page limit, we mainly report PSNR results and put more results in the supplementary materials.

Compared methods. We compare our SRTTA with several state-of-the-art methods including supervised pre-trained SR methods, blind SR methods, zero-shot SR methods, and a TTA baseline method. 1) The supervised pre-trained SR models learn super-resolution knowledge with a predefined degradation process, i.e. , the Bicubic downsampling. These methods include SwinIR [29], IPT [7] and HAT [8], and the EDSR baseline [31]. 2) Blind SR models predict the blur kernel of test images and generate HR images simultaneously, e.g. , DAN [21] and DCLS-SR [34]. 3) Zero-shot SR models often use a predefined/estimated degradation kernel to construct the LR-HR paired images based on the assumption of the cross-scale patch recurrence [14, 62, 37], and they use the LR-HR paired images to train/update the SR models for each test image. These methods include ZSSR [43], KernelGAN [2]+ZSSR [43], MZSR [9], DualSR [12], DDNM [51]. 4) Moreover, we implement a baseline TTA method (dubbed TTA-C) that utilizes the augmentation consistency loss L c in Eqn. (8) to adapt the pre-trained model, similar to MEMO [57] and CoTTA [48].

5.2 Comparison with State-of-the-art Methods

We evaluate the effectiveness of our methods in terms of quantitative results and visual results. We report the PSNR results of our SRTTA and existing methods on the DIV2K-C dataset for 2 × SR in Table 1 (more results are put in the supplementary materials). Since DAN [21] and DCLS-SR [34] are trained on paired images with Gaussian Blur degradation, they achieve the best results in Gaussian Blur and Glass Blur degradation. However, they merely achieve a limited performance on average due to the ignoring of the noise and JPEG degradations in their degradation model. Moreover, ZSSR [43] achieves state-of-the-art performance on average due to the thousands of iterations of training steps for each image. Though KernelGAN [15] estimates a more accurate blur kernel and helps to generate more plausible HR images for Gaussian Blur and Glass Blur degradation images, it is harmful to the performance of ZSSR [43] on the noise images, since the degradation model of KernalGAN [15] does not cover the noise degradation. Moreover, the baseline TTA-C may be harmful to adapting the pre-trained model to blur degradation due to the simple augmentation, resulting in a limited

Ground-truth HR

KernelGAN+ZSSR

ZSSR

EDSR

SRTTA(ours)

Bicubic

DAN

DDNM

HAT

DualSR

SRTTA-lifelong(ours)

×

SR

Ground-truth HR

Bicubic

DAN

KernelGAN+ZSSR

ZSSR

EDSR

SRTTA(ours)

DDNM

HAT

DualSR

SRTTA-lifelong(ours)

×

品h

品尼扣

(b) Visualizations under JEPG Compression for 2

SR

adaptation performance. Instead, our methods achieve the best performance in terms of PSNR on average. For quality comparison, we provide the visual results of our SRTTA and the compared methods in Figure 2. As shown in Figure 2, our SRTTA is able to remove the degradation and reconstruct clearer SR images with less noise or fewer JPEG artifacts.

Comparison of inference time. Moreover, we also compare the inference time of different methods in Table 1. Due to the additional time for test-time adaptation, our SRTTA cannot achieve the best inference speed. However, those methods with less inference time are mostly trained on domains with Gaussian blur degradation only, such as DAN [21], DCLS-SR [34], and MZSR [45], which still suffer from the domain shift issue under noise or JPEG degradation. Instead, with comparable efficiency, our SRTTA achieves an impressive improvement on average for all domains (see results in Table 1). In conclusion, our SRTTA achieves a better tradeoff between performance and efficiency.

More results on test domains with mixed multiple degradations. In this part, we evaluate our SRTTA on DIV2K-MC to further investigate the effectiveness of SRTTA under mixed multiple degradations. In Table 2, our SRTTA achieves the best performance on four domains with different mixed degradations, e.g., 26.47 (ZSSR) → 28.48 (our SRTTA-lifelong) regarding the average PSNR metric. These results further validate the effectiveness of our proposed methods.

Evaluation in terms of human eye-related metrics. In this part, we further evaluate different methods in terms of the Fréchet Inception Distance (FID) [19] and the Learned Perceptual Image Patch Similarity (LPIPS) distance [58], which correlate well with perceived image quality and are commonly used to evaluate the quality of generated images [10, 55]. We evaluate different methods on two synthesized datasets, including DIV2K-C and DIV2K-MC. As shown in Table 3, our SRTTA achieves the lowest values of both FID and LPIPS scores, demonstrating our SRTTA is able to generate images with higher visual quality.

5.3 Further Experiments

In this part, we further conduct several ablation studies to demonstrate the effectiveness of each component of our SRTTA. Last, we evaluate our SRTTA on several datasets with real-world test images and provide the visual comparison results of different methods.

Effect of the degradation classifier C ( · ) . To investigate the effect of the degradation classifier, we compare our SRTTA with a baseline that does not use the degradation classifier. Specifically, this baseline generates the second-order degraded images with the random degradation type. In this case, a test image with blur degradation can be randomly degraded with blur, noise, or JPEG degradation. We report the PSRN results of our SRTTA and this baseline in Table 4. Experimental results demonstrate the necessity of the degradation classifier in our second-order degradation scheme.

Effect of L s and L a in Eqn. (6). To investigate the effect of the self-supervised adaptation loss L s and adaptation consistency loss L a in Eqn. (6), we report the mean PSNR results of our SRTTA with L s -only and L a -only. As shown in Table 4, without the adaptation consistency loss L a , the SR models with only the L s will inevitably result in a model collapse. This is because the SR model is prone to output the same output for any input images without meaning. When we remove the L s loss, SRTTA can only achieve a limited performance, which demonstrates that L s truly helps to encourage the SR models to learn how to remove the degradation during the adaptation process. These experimental results demonstrate the effectiveness of the L s and L a in our framework.

Effect of the hyper-parameters α in Eqn. (6). To investigate the effect of the weight of adaptation consistency loss α in Eqn. (6), we report the mean PSNR results of our SRTTA with different α in Table 5. When the α is too small ( α < 1 ), the self-supervised degradation loss L s dominates the adaptation process and often results in the collapse of SR models. When the α is too large ( α > 1 ), the adaptation consistency loss L a may have a great constraint on the adapted SR model to be the same as the pre-trained model, which may be harmful to the adaptation of the SR performance. With α = 1 , our SRTTA achieves the best results on the DIV2K-C dataset on average. Therefore, we simply set the α to be 1 for all other experiments.

Effect of adaptive parameter preservation in Eqn. (9). In this part, we investigate the effect of our adaptive parameter preservation (APP) strategy on test-time adaptation. We compare our APP with the existing anti-forgetting method Stochastic Restoration (STO) [48], which randomly restores a different set of parameters ( 1% parameters) after each adaptation step. Moreover, we also compare our APP with the random selection (RS) baseline, which randomly selects a fixed set of parameters to freeze before adaptation. As shown in Table 6, though STO achieves the best anti-forgetting performance on the clean Set5 dataset, the STO severely hinders the TTA performance. The random selection baseline only achieves a limited performance of both TTA and anti-forgetting. Instead, our APP consistently outperforms the random selection baseline with the same ratio of frozen parameters. Moreover, our APP with ρ = 0 . 5 achieves the best adaptation performance on DIV2K-C (see more

Dataset

STO [48]

RS with different

ρ

APP with different

ρ

(ours)

0.3

0.5

0.7

0.3

0.5

0.7

DIV2K-C (with degradation shift)

27.17

27.52

27.62

27.68

27.72

27.73

27.73

Set5 (w/o degradation shift)

35.57

33.95

34.02

34.24

34.11

34.23

34.38

LR

Bicubic

DAN

kernelGAN+ZSSR

ZSSR

SwinIR

EDSR SRTTA-lifelong(ours)

DCLS-SR

HAT

DDNM

(a) Visualization results of the image from DPED [23].

LR

Bicubic

DAN

kernelGAN+ZSSR

ZSSR

EDSR

SRTTA-lifelong(ours)

DCLS-SR

HAT

DDNM

SwinIR

(b) Visualization results of the image from ADE20K [61].

Josefstr.161

198

198

nerA

AG

198

nerei

SX

SX

SXE

SX

SX

SX-

ED

SXE

SX

SX

SX

SX

SX

results in the supplementary materials), thus we set the ratio of preservation parameters to be 0.5 in our SRTTA by default. These results demonstrate the effectiveness of our APP strategy.

Visualization results on the real-world images. We also provide the visual results of different methods on the real-world images from DPED [23] and ADE20K [61]. We use our SRTTA-lifelong model that has been adapted on DIV2K-C to perform test-time adaptation on the real-world images from DPED [23] and ADE20K [61], respectively. As shown in Figure 3, SRTTA-lifelong is able to generate HR images with fewer artifacts. These results demonstrate that our method is able to be applied to real-world applications. Please refer to more results in the supplementary materials.

6 Conclusion

In this paper, we propose a super-resolution test-time adaptation (SRTTA) framework to quickly alleviate the degradation shift issue for image super-resolution (SR). Specifically, we propose a second-order degradation scheme to construct paired data for each test image. Then, our second-order reconstruction loss is able to quickly adapt the pre-trained SR model to the test domains with unknown degradation. To evaluate the effectiveness of our SRTTA, we use eight different types of degradations to synthesize two new datasets named DIV2K-C and DIV2K-MC. Experiments on the synthesized datasets and several real-world datasets demonstrate that our SRTTA is able to quickly adapt the SR model for each image and generate plausible high-resolution images.

References

E. Agustsson and R. Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In The IEEE Conference on Computer Vision and Pattern Recognition Workshops , July 2017.

S. Bell-Kligler, A. Shocher, and M. Irani. Blind super-resolution kernel estimation using an internal-gan. In Advances in Neural Information Processing Systems , volume 32, 2019.

M. Bevilacqua, A. Roumy, C. Guillemot, and M. L. Alberi-Morel. Low-complexity singleimage super-resolution based on nonnegative neighbor embedding. In British Machine Vision Conference , pages 1-10. BMVA press, 2012.

A. Bruhn, J. Weickert, and C. Schnörr. Lucas/kanade meets horn/schunck: Combining local and global optic flow methods. International Journal of Computer Vision , 61:211-231, 2005.

A. Bulat, J. Yang, and G. Tzimiropoulos. To learn image super-resolution, use a gan to learn how to do image degradation first. In European Conference on Computer Vision , pages 185-200, 2018.

D. Chen, D. Wang, T. Darrell, and S. Ebrahimi. Contrastive test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 295-305, 2022.

H. Chen, Y. Wang, T. Guo, C. Xu, Y. Deng, Z. Liu, S. Ma, C. Xu, C. Xu, and W. Gao. Pre-trained image processing transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12299-12310, 2021.

X. Chen, X. Wang, J. Zhou, Y. Qiao, and C. Dong. Activating more pixels in image superresolution transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22367-22377, 2023.

X. Cheng, Z. Fu, and J. Yang. Zero-shot image super-resolution with depth guided internal degradation learning. In European Conference on Computer Vision , pages 265-280. Springer, 2020.

H. Chung, B. Sim, D. Ryu, and J. C. Ye. Improving diffusion models for inverse problems using manifold constraints. In Advances in Neural Information Processing Systems , volume 35, pages 25683-25696, 2022.

C. Dong, C. C. Loy, K. He, and X. Tang. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence , 38(2):295-307, 2016.

M. Emad, M. Peemen, and H. Corporaal. Dualsr: Zero-shot dual learning for real-world superresolution. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1630-1639, 2021.

C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning , pages 1126-1135. PMLR, 2017.

D. Glasner, S. Bagon, and M. Irani. Super-resolution from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 349-356. IEEE, 2009.

J. Gu, H. Lu, W. Zuo, and C. Dong. Blind super-resolution with iterative kernel correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1604-1613, 2019.

Y. Guo, J. Chen, J. Wang, Q. Chen, J. Cao, Z. Deng, Y. Xu, and M. Tan. Closed-loop matters: Dual regression networks for single image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5407-5416, 2020.

J. M. Haut, R. Fernandez-Beltran, M. E. Paoletti, J. Plaza, A. Plaza, and F. Pla. A new deep generative network for unsupervised remote sensing single-image super-resolution. IEEE Transactions on Geoscience and Remote sensing , 56(11):6792-6810, 2018.

D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations , 2019.

M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in neural information processing systems , volume 30, 2017.

A. Hore and D. Ziou. Image quality metrics: PSNR vs. SSIM. In International Conference on Pattern Recognition , pages 2366-2369, 2010.

Y. Huang, S. Li, L. Wang, T. Tan, et al. Unfolding the alternating optimization for blind super resolution. In Advances in Neural Information Processing Systems , volume 33, pages 5632-5643, 2020.

S. A. Hussein, T. Tirer, and R. Giryes. Correction filter for single image super-resolution: Robustifying off-the-shelf deep super-resolvers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1428-1437, 2020.

A. Ignatov, N. Kobyshev, R. Timofte, K. Vanhoey, and L. Van Gool. Dslr-quality photos on mobile devices with deep convolutional networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3277-3285, 2017.

X. Ji, Y. Cao, Y. Tai, C. Wang, J. Li, and F. Huang. Real-world super-resolution via kernel estimation and noise injection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , pages 466-467, 2020.

X. Kang, J. Li, P. Duan, F. Ma, and S. Li. Multilayer degradation representation-guided blind super-resolution for remote sensing images. IEEE Transactions on Geoscience and Remote Sensing , 60:1-12, 2022.

J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences , 114(13):3521-3526, 2017.

P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Yasunaga, R. L. Phillips, I. Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning , pages 5637-5664. PMLR, 2021.

W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 624-632, 2017.

J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1833-1844, 2021.

J. Liang, K. Zhang, S. Gu, L. Van Gool, and R. Timofte. Flow-based kernel prior with application to blind super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10601-10610, 2021.

B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , pages 136-144, 2017.

A. Liu, Y. Liu, J. Gu, Y . Qiao, and C. Dong. Blind image super-resolution: A survey and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

P. Liu, H. Zhang, Y. Cao, S. Liu, D. Ren, and W. Zuo. Learning cascaded convolutional networks for blind single image super-resolution. Neurocomputing , 417:371-383, 2020.

Z. Luo, H. Huang, L. Yu, Y. Li, H. Fan, and S. Liu. Deep constrained least squares for blind image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17642-17652, 2022.

W. Ma, Z. Pan, J. Guo, and B. Lei. Achieving super-resolution remote sensing images via the wavelet transform combined with the recursive res-net. IEEE Transactions on Geoscience and Remote Sensing , 57(6):3512-3527, 2019.

S. Maeda. Unpaired image super-resolution using pseudo-supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 291-300, 2020.

T. Michaeli and M. Irani. Nonparametric blind super-resolution. In Proceedings of the IEEE International Conference on Computer Vision , pages 945-952, 2013.

S. Niu, J. Wu, Y. Zhang, Y. Chen, S. Zheng, P. Zhao, and M. Tan. Efficient test-time model adaptation without forgetting. In International Conference on Machine Learning , pages 1688816905. PMLR, 2022.

S. Niu, J. Wu, Y. Zhang, Z. Wen, Y. Chen, P. Zhao, and M. Tan. Towards stable test-time adaptation in dynamic wild world. In The International Conference on Learning Representations , 2023.

Y. Pang, J. Cao, J. Wang, and J. Han. Jcs-net: Joint classification and super-resolution network for small-scale pedestrian detection in surveillance images. IEEE Transactions on Information Forensics and Security , 14(12):3322-3331, 2019.

S. Park, J. Yoo, D. Cho, J. Kim, and T. H. Kim. Fast adaptation to super-resolution networks via meta-learning. In European Conference on Computer Vision , pages 754-769. Springer, 2020.

L. Schermelleh, A. Ferrand, T. Huser, C. Eggeling, M. Sauer, O. Biehlmaier, and G. P. Drummen. Super-resolution microscopy demystified. Nature Cell Biology , 21(1):72-84, 2019.

A. Shocher, N. Cohen, and M. Irani. 'zero-shot' super-resolution using deep internal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3118-3126, 2018.

Y. M. Sigal, R. Zhou, and X. Zhuang. Visualizing and discovering cellular structures with super-resolution microscopy. Science , 361(6405):880-887, 2018.

J. W. Soh, S. Cho, and N. I. Cho. Meta-transfer learning for zero-shot super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3516-3525, 2020.

Y. Sun, X. Wang, Z. Liu, J. Miller, A. Efros, and M. Hardt. Test-time training with selfsupervision for generalization under distribution shifts. In International Conference on Machine Learning , pages 9229-9248. PMLR, 2020.

D. Wang, E. Shelhamer, S. Liu, B. Olshausen, and T. Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations , 2021.

Q. Wang, O. Fink, L. Van Gool, and D. Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7201-7211, 2022.

X. Wang, L. Xie, C. Dong, and Y. Shan. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1905-1914, 2021.

X. Wang, K. Yu, C. Dong, and C. C. Loy. Recovering realistic texture in image super-resolution by deep spatial feature transform. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 606-615, 2018.

Y. Wang, J. Yu, and J. Zhang. Zero-shot image restoration using denoising diffusion null-space model. In The International Conference on Learning Representations , 2023.

Y. Xiao, Q. Yuan, Q. Zhang, and L. Zhang. Deep blind super-resolution for satellite video. IEEE Transactions on Geoscience and Remote Sensing , 2023.

J. Yang, K. Fu, Y. Wu, W. Diao, W. Dai, and X. Sun. Mutual-feed learning for super-resolution and object detection in degraded aerial imagery. IEEE Transactions on Geoscience and Remote Sensing , 60:1-16, 2022.

Y. Yuan, S. Liu, J. Zhang, Y. Zhang, C. Dong, and L. Lin. Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , pages 701-710, 2018.

K. Zhang, J. Liang, L. Van Gool, and R. Timofte. Designing a practical degradation model for deep blind image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4791-4800, 2021.

L. Zhang, H. Zhang, H. Shen, and P. Li. A super-resolution reconstruction algorithm for surveillance images. Signal Processing , 90(3):848-859, 2010.

M. Zhang, S. Levine, and C. Finn. Memo: Test time robustness via adaptation and augmentation. In Advances in Neural Information Processing Systems , volume 35, pages 38629-38642, 2022.

R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 586-595, 2018.

Y. Zhang, B. Hooi, L. Hong, and J. Feng. Self-supervised aggregation of diverse experts for test-agnostic long-tailed recognition. In Advances in Neural Information Processing Systems , pages 34077-34090, 2022.

Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European conference on computer vision , pages 286-301, 2018.

B. Zhou, H. Zhao, X. Puig, T. Xiao, S. Fidler, A. Barriuso, and A. Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision , 127:302-321, 2019.

M. Zontak and M. Irani. Internal statistics of a single natural image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 977-984. IEEE, 2011.

Supplementary Materials for

' Efficient Test-Time Adaptation for Super-Resolution with Second-Order Degradation and Reconstruction '

Contents

A More Details of the Second-Order Degradation

In our second-order degradation scheme, we randomly generate different degradations to degrade the test image into its second-order degraded counterparts. In this section, we illustrate how to randomly generate different types of degradations. Notably, we degrade the test images on the GPU device to accelerate the degradation process.

A.1 Random Blur Degradation

Following [26, 22], we model blur degradation as a convolution with a linear Gaussian blur filter/kernel. Given a test image, we randomly generate a set of isotropic or anisotropic Gaussian kernels k and use them to perform blur degradation on the test image. The probabilities of generating an isotropic kernel and an anisotropic kernel are set to 0.5 and 0.5, respectively. The size of each generated kernel is uniformly sampled from { 7 × 7 , 9 × 9 , ..., 21 × 21 } . We sample the standard deviation of the blur kernel along the two principal axes σ 1 and σ 2 uniformly from [0 . 2 , 3] . If k is an isotropic Gaussian blur kernel, we set σ 2 equal to σ 1 . If k is an anisotropic Gaussian kernel, we further sample a rotation angle a uniformly from [ -π, π ] , and use a rotation matrix to transform the generated kernel based on the angle a . More details about how to generate a Gaussian blur kernel can refer to Real-ESRGAN [22].

A.2 Random Noise Degradation

In our second-order degradation, we randomly generate a set of Gaussian noise maps n , and add them to the test image to obtain a set of second-order images with different additive Gaussian noise. With a probability of 60% , we generate noise for each channel of RGB images independently, otherwise, we generate the same noise map for all three channels. We first generate a noise map whose values are randomly generated from a normal Gaussian distribution. Then we sample a scale value to enlarge the noise uniformly from [1 , 30] . More details can be referred to Real-ESRGAN [22].

A.3 Random JPEG Degradation

For JPEG compression JPEG q , we sample a quality factor q uniformly from [30 , 95] , and use the JPEG compression with the degradation q to degrade test images into a set of second-order degraded images with compression artifacts. Note that JPEG compression with a lower q compress the test image with a higher compression ratio and the compressed images are generally of a lower quality. To accelerate the degradation process, we use DiffJPEG 1 , which is the PyTorch implementation of JPEG compression, to process the test image on the GPU device.

B The Difference from Real-ESRGAN

In this part, we would like to discuss the difference between our SRTTA with Real-ESRGAN[22], which proposes the concept of second-order degradations, to highlight the contribution of our SRTTA.

B.1 Solving Different Problems

Real-ESRGAN tries to enumerate all the degradations in real-world scenes and train an SR model to solve the image restoration on any degradation. However, it is non-trivial to obtain all real-world degradations, leading to domain shift issues when encountering unknown degradations during testing, as shown in Figure 11 of real-ESRGAN [22]. Unlike real-ESRGAN [22], our SRTTA aims to adapt the SR models to the test domains when test images contain unknown degradations. Our second-order degradation scheme aims to quickly construct the pseudo-paired data (instead of the paired training data) to adapt the SR model to the test domains.

B.2 Different Construction Schemes

Real-ESRGAN [22] proposes the second-order degradation to construct the paired training data, whose low-resolution (LR) images are obtained from the ground-truth high-resolution (HR) images.

Then, the paired data is used to train an SR model during the training phase in a supervised learning manner. Notably, the trained Real-ESRGAN [22] model is fixed during the test time. Instead, our second-order degradation scheme constructs the pseudo-paired data using the test images with unknown degradation (first-order degraded images). Our SRTTA model is continuously adapted to different domains during testing in a self-supervised learning manner.

C Experimental Datasets

C.1 Construction Details of the DIV2K-C Dataset

To evaluate the practicality, we construct a new benchmark dataset named DIV2K-C, which contains eight different degradations. We select the eight degradation types from the 15 corruptions of ImageNet-C [11] that do not extremely change the image content, including Gaussian Blur, Defocus Blur, Glass Blur, Gaussian Noise, Poisson Noise (Shot Noise), Impulse Noise, Speckle Noise, and JPEG compression. Unlike the ImageNet-C [11], we do not use the same degradation level to degrade all test images. Instead, we randomly generate the degradation level and further generate a degradation for each image based on the degradation level. Unlike prior SR methods that investigate a limited number of degradation types, the degradation scenarios we considered are more complex (eight degradation types in total), which is more practical for real-world applications.

Given a degradation, we use the classical image degradation model [17, 22] to generate the lowresolution (LR) test images from the high-resolution clean images. For blur degradation, we perform the blur convolution on the HR images and then use the Bicubic downsampling to obtain the test LR images. For noise and JPEG degradation, we first use the Bicubic downsampling to obtain clean LR images and then perform noise or JPEG degradation on the clean LR images to obtain the final test images. We show the visualization of some examples regarding each degradation type in Figure A.

HR

Defocus Blur

JPEG

Clean LR

Impulse Noise

Speckle Noise

0826 from DIV2K

HR

Defocus Blur

JPEG

Clean LR

Impulse Noise

Speckle Noise

0846 from DIV2K

信武華中

K

Gaussian blur. Following BSRGAN [26], we generate low-resolution (LR) images with Gaussian blur degradation. We randomly generate an isotropic Gaussian kernel or an anisotropic Gaussian kernel for each high-resolution (HR) image. Then, we use the blur kernel to perform blur convolution on the HR image and use Bicubic downsampling to obtain the final LR test images. For simplicity, we follow the recipe of BSRGAN [26] to generate test images with blur degradation.

Defocus blur. To better compare the performance of different SR methods, we also use the common Defocus blur degradation to degrade the HR images and obtain the LR images using the Bicubic downsampling. As illustrated in ImageNet-C [11], Defocus blur often occurs when an image is out of focus when we take pictures. We generate the blur kernel as ImageNet-C [11] and perform a blur convolution on the HR images. But unlike ImageNet-C [11], the degradation level of Defocus blur is randomly sampled from a given range.

Glass blur. We also choose another common degradation type, Glass Blur, which appears with 'frosted glass' windows or panels [11]. This blur degradation requires two Gaussian blur operations and an operation that locally shuffles pixels between two blur operations. As mentioned above, the degradation level is randomly sampled from a given range, such as the standard deviation of the Gaussian blur kernel or the window size of the shuffling operation.

Gaussian noise. To generate test images with Gaussian noise, we sample the noise for each pixel from a normal Gaussian distribution. The mean of the Gaussian distribution is zero, and the standard deviation is uniformly sampled from the range of { 2 / 255 , 3 / 255 , ..., 25 / 255 } . More details can refer to the implementation of BSRGAN [26].

Poisson (Shot) noise. Poisson noise, also called Shot noise, can model the sensor noise caused by statistical quantum fluctuations. We randomly generate the Poisson noise map from a Poisson distribution, which has an intensity proportional to the image intensity. Then, we add the generated Poisson noise into the clean LR images to obtain the test images with Poisson noise. More details can refer to the implementation of Real-ESRGAN [22].

Impulse noise. Impulse noise is caused by errors in the data transmission generated in noisy sensors or communication channels, or by errors during the data capture from digital cameras [19]. The most common form of Impulse noise is called salt-and-pepper noise. To generate test images with Impulse noise, we uniformly select a set of pixels and replace them with zero or the maximum value (255). More details can be referred to ImageNet-C [11].

Speckle noise. Speckle noise is an additive noise where the noise added to a pixel tends to be larger if the original pixel intensity is larger. We first sampled the noise for each pixel from a Gaussian distribution and multiple the noise value by the original pixel. Last, we add the generated noise map into the LR clean images to obtain the final test images with Speckle noise.

JPEG compression. For JPEG compression, we use the OpenCV implementation of JPEG compression 2 to degrade the clean LR image into final test images. The compression quality factor q is randomly sampled from [30 , 90] . We first encode the clean LR images into the bit stream using JPEG compression with the quality factor q and decode the bit stream to obtain the final test images. Note that JPEG compression is a lossy compression technique, so the final test images are inevitably corrupted with JPEG compression artifacts.

C.2 Construction Details of the DIV2K-MC Dataset

Since the real-world test images may contain multiple degradation types simultaneously, we further develop a new benchmark dataset named DIV2K-MC, which includes four test domains with mixed multiple degradations. The four domains are BlurNoise, BlurJPEG, NoiseJPEG and BlurNoiseJPEG. The test images in the BlurNoiseJPEG domain contain the combined degradation of Gaussian blur, Gaussian noise and JPEG degradations simultaneously.

BlurNoise. We generate LR images from HR images using Gaussian blur and Gaussian noise degradation. We first randomly generate a Gaussian blur kernel to perform blur convolution on the HR image. Then, we downsample the resulting image using Bicubic interpolation. Last, we randomly sample a Gaussian noise map and add it to the downsampled image to obtain the final LR image.

BlurJPEG. We generate LR images from HR images using Gaussian blur and JPEG degradation. We first randomly generate a Gaussian blur kernel to perform blur convolution on the HR image. Then, the resulting image is downsampled by using Bicubic interpolation. Last, we use JPEG compression with a random quality factor q to compress the downsampled image to obtain the final LR image.

NoiseJPEG. We generate LR images from HR images using Gaussian noise and JPEG degradation. We first downsample HR image using Bicubic interpolation. Then, we randomly sample a Gaussian

noise map and add it to the downsampled image. Last, we use JPEG compression with a random quality factor q to compress the downsampled image to obtain the final LR image.

BlurNoiseJPEG. We generate LR images from HR images using Gaussian blur, Gaussian noise and JPEG degradation. We first randomly generate a Gaussian blur kernel to perform blur convolution on the HR image. Second, the resulting image is downsampled by using Bicubic interpolation. Then, we randomly sample a Gaussian noise map and add it to the downsampled image. Last, we use JPEG compression with a random quality factor q to compress the image to obtain the final LR image.

C.3 More Test Datasets for Test-Time Image Super-Resolution

Moreover, we also evaluate the performance of SR methods on real-world test images from DPED [13], ADE20K [28] and OST300 [23], whose corresponding ground-truth HR images can not be found. To evaluate the anti-forgetting performance, we report the adapted model performance on a clean benchmark dataset Set5 [2] whose images are clean images that are downsampled from HR images with Bicubic interpolation. Thus, these LR images do not contain any degradation.

D Implementation Details

D.1 Implementation Details of the Degradation Classifier

In our second-order degradation scheme, we use a pre-trained degradation classifier to predict the degradation type for each test image. To obtain the pre-trained degradation classifier, we use ResNet-50 [10] as the classifier and train it to recognize the degradation from test images.

In real-world scenes, test images may contain degradations other than these eight degradation types, such as ringing or overshoot artifacts [22], which may be viewed as variations of blur, noise or JPEG. Since it is infeasible to cover all the degradation types in real-world scenes, we make the degradation classifier to predict the coarse-level four classes, including clean, blur, noise and JPEG.

Training details. Specifically, we use the 800 training HR images of DIV2K and randomly crop them into patches with the size of 224 × 224 (instead of resizing them into 224 × 224). Similar to the construction of DIV2K-C, we degrade each patch using a random selection of one of eight degradation types. As for clean data, we do not perform any degradation on the patches. For training, we apply Adam with β 1 = 0.9, β 2 = 0 . 999 and set the batch size as 256. The learning rate is initialized to 10 -3 and decreased to 10 -6 with a cosine annealing out of 400 epochs in total.

Testing details. During testing, we directly input the whole test image with original resolution into the classifier and output the predicted results to recognize the degradation type. The predicted results of the multi-label degradation classifier C ( · ) are the probabilities of the three degradations, including blur, noise and JPEG degradation. If the predicted probability of one degradation type is larger than the threshold of 0.5, the test image is considered to contain the degradation of this type. The clean image means that this image does not contain any degradation such as blur, noise, or JPEG, and we directly use the pre-trained SR model to super-resolve these clean test images.

D.2 More Details of Super-Resolution Test-Time Adaptation

We use the baseline model of EDSR [16] with less than 2M parameters as our pre-trained SR model for 2 × and 4 × SR. During adaptation, we only update the parameters in the Resblock of the EDSR model. To avoid anti-forgetting, we use five clean test LR images from Set5 [2] to select important parameters to be frozen in Eqn. (9). Moreover, when evaluating the anti-performance Set5 [2], we directly use the adapted model to super-resolve the test images without using the classifier.

In our experiment, we conduct experiments in parameter-reset and lifelong settings. In the parameterreset setting, the model parameters will be reset after the adaptation on each domain, which is the default setting of our SRTTA. In the lifelong setting, the model parameters will never be reset in the long-term adaptation, in this case, we call our methods as SRTTA-lifelong.

For test-time adaptation, we use the Adam optimizer with the learning rate of 5 × 10 -5 for the pre-trained SR models. We set the batch size N to 32, and we randomly crop the test image into N patches of size 96 × 96 and 64 × 64 for 2 × and 4 × SR, and degrade them into second-order degraded patches. We perform S = 10 iterations of adaptation for each test image. For the balance weight in

Eqn. (6), we set α to 1. For the ratio of parameters to be frozen, we set the ρ to 0.50. To compare the inference times of different SR methods, we measure all methods on a TITAN XP with 12G graphics memory for a fair comparison. Due to the memory overload of HAT [6] and DDNM [24], we chop the whole image into smaller patches and process them individually for these two methods. In our experiments, we use the bold number to indicate the best result and the underlined number to indicate the second-best result.

E More Experimental Results

E.1 More Results of Test-Time Image Super-Resolution

In this part, we compare our SRTTA with existing SR methods, including supervised pre-trained SR methods, blind SR methods, zero-shot SR methods, and a TTA baseline with consistency loss. 1) The supervised pre-trained SR methods learn to process test images with a predefined degradation process, i.e., the Bicubic downsampling. These methods include the EDSR baseline [16], SwinIR [15], IPT [5] and HAT [6], SRDiff [14], and Real-ESRNet [22]. Note that Real-ESRNet [22] is the PSNR-oriented model of Real-ESRGAN [22], which use a complex combination of different degradation to construct pair training data, including Gaussian blur, Gaussian Noise, Poisson Noise and JPEG compression, and so on. 2) Blind SR models predict the blur kernel of test images and generate HR images simultaneously, we compare with the state-of-the-art DAN [12] and DCLS-SR [18] methods. 3) Zeroshot SR models construct the LR-HR paired images based on the assumption of the cross-scale patch recurrence and train/update their SR model for each test image. These methods include ZSSR [20], KernelGAN [1]+ZSSR [20], MZSR [7], DualSR [8], DDNM [24]. 4) Moreover, we implement a baseline TTA method (dubbed TTA-C) that utilizes the augmentation consistency loss in Eqn. (8) to adapt the pre-trained model, similar to MEMO [27] and CoTTA [21].

We provide more results of our SRTTA for 2 × and 4 × SR in terms of PSNR and SSIM metrics. As shown in Table A and Table B, our SRTTA consistently outperforms existing SR methods on the DIV2K-C dataset on average. We further provide more visualization results in Figure D, E, F and G, which demonstrate our SRTTA is able to remove the degradation from test images and generate plausible HR images. These results demonstrate that our SRTTA is able to quickly adapt the pre-trained SR model to the test images with different degradation.

E.2 More Results on Test domains with Mixed Multiple Degradations.

In this part, we evaluate our SRTTA on DIV2K-MC, which consists of four test domains with mixed multiple degradations. In Table C, our SRTTA achieves the best performance on 4 domains with different mixed degradations, e.g., 0.619 (DualSR) → 0.775 (our SRTTA-lifelong) regarding the average SSIM metric. These results further validate the effectiveness of our proposed methods.

E.3 More Results of Ablation Studies

Effect of each component. In this part, we investigate the effect of each component and provide more ablation studies. As shown in Table D, the baseline without the degradation classifier, generates the second-order degraded images with random degradation types, achieving a limited performance in terms of both PSNR and SSIM. The baseline without the adaptation consistency loss L a results in the model collapse due to the lack of the consistency constraint. Without the self-supervised adaptation loss L s , the TTA performance of the adapted model drops significantly. These experimental results demonstrate the effectiveness of each component of our framework.

Effect of the hyperparameter α in Eqn. (6). In this part, we investigate the effect of the weight of adaptation consistency loss α . As shown in Table E, the adapted model with α = 1 achieves the best TTA performance. Thus, we set the α = 1 by default for our SRTTA during adaptation.

Effect of the hyperparameter ρ in Eqn. (9). In this part, we analyze the effect of the hyperparameter ρ , which decides the ratio of parameters to freeze, for the test-time adaptation. In Table F, when ρ = 0 . 50 , our SRTTA achieves the best TTA performance on the DIV2K-C dataset on average in the lifelong setting. Meanwhile, we also investigate the effect of the adaptive parameter preservation (APP) strategy in the parameter-reset setting. As shown in Table F, our APP strategy (with ρ = 0 . 50 )

merely has little impact on the TTA in the parameter-reset setting. These results demonstrate the effectiveness of the APP strategy in test-time adaptation for image super-resolution.

Comparison with other anti-forgetting methods. In this part, we compare our adaptive parameter preservation (APP) strategy with two baseline methods to demonstrate the effectiveness of our strategy in preserving the learned knowledge of pre-trained SR models. Stochastic Restoration (STO) [21] randomly selects a different set of parameters (with a ratio of 1% ) and restores them back to the parameters of the pre-trained models. Random Selection (RS) selects a fixed set of parameters before adaption and freezes them not to update. As shown in Table G, our APP strategy achieves the best TTA results on the DIV2K-C dataset. Meanwhile, with the same ratio of selected parameters, our APP strategy consistently outperforms the Random Selection baseline for the anti-forgetting. These results demonstrate that our adaptive selection is able to select the important parameters and preserve the knowledge of the pre-trained model.

E.4 Effectiveness on a New Unknown Domain

In this part, we further evaluate our SRTTA on a new unknown domain with the degradation of processed camera sensor noise [3, 26], which is not used in the training phase of the SR model or that of the degradation classifier. We report the PSNR( ↑ ) and SSIM( ↑ ) values of different methods on 100 images with random processed camera sensor noise. In Table H, our SRTTA method is also able to improve the model performance on this unknown degradation. These experimental results further demonstrate the generalization capability of our SRTTA model to unknown degradation types.

E.5 Comparison with Patch-Recurrence Reconstruction Loss

In this part, we investigate the effect of our second-order reconstruction loss. We compare our loss with the loss of existing zero-shot methods [20, 7, 1]. Based on the assumption of patch recurrence across scales [9, 29], these methods downsample the test image to obtain an image with a lower resolution and reconstruct the test image from the downsampled image. For simplicity, we call this patch-recurrence loss. For a fair comparison, we further downsample the second-order degraded images that are obtained using our second-order degradation scheme and reconstruct the test image with the patch-recurrence loss. As shown in Table I, our SRTTA with our second-order reconstruction loss consistently outperforms the baseline with the patch-recurrence loss. These results demonstrate the effectiveness of our second-order reconstruction loss.

E.6 Effect of the Feature-level Reconstruction

In this part, we investigate the effect of different reconstruction levels. In our second-order reconstruction, we use the feature-level reconstruction to adapt the pre-trained model as in Eqn. (6). We

compare a baseline with an image-level reconstruction, which means we reconstruct the output of the SR model instead of the feature in the middle layer. As shown in Table J, when reconstructing at the image level, the adapted model achieves a comparable performance for both 2 × SR and 4 × SR. Thus, both reconstruction levels are optional, we use the feature-level reconstruction by default.

E.7 Effect of Adaptation Iterations for Each Image

In this part, we investigate the effect of different adaptation iterations for each image. As shown in Table K, we compare the TTA performance of SRTTA with a different number of iterations S for each image. When the number of iterations S is small, the adapted SR model is not able to learn how to remove the degradation for these images well. When the number of iterations S is too large, the performance improvement of SRTTA diminishes and the adaptation cost will be greatly increased. Thus, we set S to 10 for a better efficiency-accuracy trade-off.

E.8 The Statistics of the Degradation Types of Real-world Images

In this part, we count the statistics of the degradation types of real-world images and report the results in Table L. We use our degradation classifier to identify the degradation types of real-world images from five datasets, including RealSR [4], DRealSR [25], DPED [13], OST300 [23] and ADE20K [28]. Experimental results in Table L show that the degradation type of blur happens the most among these real-world datasets.

E.9 Further Analysis of the Domain Shift Issue

In this part, we investigate the effect of the domain shift issues for pre-trained SR models, which are trained on specific domains with different degradation types. We use the EDSR baseline model as the model for analysis. In total, we separately train four EDSR baseline models on clean, blur, noise and JPEG domains, respectively. The corresponding four models are named EDSR, EDSR-B, EDSR-N and EDSR-J, respectively. We evaluate these four models on clean images and the test images with Gaussian Blur, Gaussian Noise or JPEG degradations in Figure B and Figure C.

As shown in Figure B and Figure C, when domain shift occurs, the pre-trained EDSR models, which are trained on domains different from the test domains, cannot remove the degradation from test

EDSR

EDSR-B

EDSR-N

EDSR-J

HR

LR

SRTTA(ours)

(a) Visualization results under clean domain.

EDSR

EDSR-B

EDSR-N (upper-bound)

EDSR-J

HR

LR

SRTTA(ours)

(c) Visualization results under Gaussian Noise.

EDSR

EDSR-B (upper-bound)

EDSR-N

EDSR-J

HR

LR

SRTTA(ours)

(b) Visualization results under Gaussian Blur.

EDSR

SRTTA(ours)

HR

EDSR-B

EDSR-N

EDSR-J (upper-bound)

LR

(d) Visualization results under JPEG.

images and generate unsatisfactory HR images with artifacts. For example, EDSR-B models cannot remove the noise and JPEG degradation, the EDSR-N and EDSR-J are also unable to remove the blur degradation. Instead, after test-time adaptation, our SRTTA is capable of handling the test images with unknown degradations and generating HR images with fewer artifacts. For example, our SRTTA is able to generate sharper HR images than EDSR-N and EDSR-J under Gaussian Blur domains. Indeed, our SRTTA models may be unable to completely remove the degradation compared with the upper-bound models, such as EDSR-B under Gaussian Blur. Thus, these drawbacks are required to be further addressed in future works.

F Visualization Results

F.1 Visualization Results on the DIV2K-C Images

In this part, we show more visualization comparison results of different SR methods on test images of the DIV2K-C dataset for both 2 × and 4 × SR. As shown in Figures D and E, our SRTTA is able to reduce the degradation from the test images and generate more plausible HR images.

F.2 Visualization Results on the Real-World Images

In this part, we conduct a comprehensive comparison of our SRTTA with existing approaches on two real-world datasets, including DPED [13], ADE20K [28] and OST300 [23]. As shown in Figures F and G, our SRTTA methods consistently generate more satisfactory HR images with less degradation of unknown noise or artifacts.

HR

EDSR

EDSR-B

EDSR-N

EDSR-J

LR

SRTTA(ours)

(a) Visualization results under clean domain.

EDSR

EDSR-B (upper-bound)

EDSR-N

EDSR-J

HR

LR

SRTTA(ours)

EDSR

EDSR-B

EDSR-N (upper-bound)

EDSR-J

HR

LR

SRTTA(ours)

EDSR

EDSR-B

EDSR-N

EDSR-J (upper-bound)

HR

LR

SRTTA(ours)

(c) Visualization results under Gaussian Noise.

(d) Visualization results under JPEG.

ZSSR

HAT

DualSR

Bicubic

DAN

KernelGAN+ZSSR

ZSSR

DDNM

HAT

DualSR

品尼扣扣

(b) Visualizations under JEPG Compression for 2 × SR

G Limitation Analysis and Border Impacts

G.1 Limitation Analysis

In this part, we analyze the limitations of our SRTTA and existing SR methods. When test images are corrupted at a high level, our SRTTA may not be able to completely remove the degradation and result in generated HR images with the existing degradation. For example, we show more visualization results of different methods on test images with Impulse Noise degradation in Figure H.

Ground-truth HR

Bicubic

DAN

KernelGAN+ZSSR

ZSSR

EDSR

SRTTA(ours)

SRDiff

HAT

IPT

SRTTA-lifelong(ours)

Ground-truth HR

Bicubic

DAN

KernelGAN+ZSSR

ZSSR

EDSR

SRTTA(ours)

SRDiff

HAT

IPT

SRTTA-lifelong(ours)

STATE

STATE

STATE

STATE

STATE

STATE

BANKOFMYSORE

ARKETBRANCH

S

STATE

STATE

STATE

STATE

STATE

STATE

(b) Visualizations under JEPG Compression for 4

×

SR

Since Real-ESRGAN [22] uses several different degradation types to construct training data, this model is able to remove the degradation in many cases. However, this model still suffers from the degradation shift issue, such as it cannot remove the gray Impulse noise from the test images as shown in Figure H(a). Moreover, Real-ESRGAN may generate HR images with over-smooth regions when removing the noise degradation and introduce some unpleasant artifacts due to the GAN training [22], which are shown in Figure H(b) and Figure H(c), respectively. Although our SRTTA cannot also completely remove the degradation from the test images in these cases, our SRTTA often preserves the original information of the test images. These results show the limitations of our SRTTA and existing methods and have a great impact on the practical application. Thus, these drawbacks are in urgent need to address in future works.

G.2 Broader Impacts

Our proposed SRTTA method is capable of improving the resolution of low-resolution test images in real-world applications, resulting in enhanced image clarity and enabling a precise understanding of image content. However, it is important to exercise caution during aggressive TTA adaptation, as this may result in the introduction of artifacts or distortions that have the potential to negatively impact downstream analyses such as microscopy, remote sensing, and surveillance.

LR

Bicubic

DAN

kernelGAN+ZSSR

ZSSR

SwinIR

EDSR SRTTA-lifelong(ours)

DCLS-SR

HAT

DDNM

LR

Bicubic

DAN

kernelGAN+ZSSR

ZSSR

EDSR

SwinIR

SRTTA-lifelong(ours)

DCLS-SR

HAT

DDNM

LR

Bicubic

DAN

kernelGAN+ZSSR

ZSSR

EDSR

SwinIR

SRTTA-lifelong(ours)

DCLS-SR

HAT

DDNM

Bicubic

DAN

kernelGAN+ZSSR

ZSSR

EDSR

SwinIR

SRTTA-lifelong(ours)

DCLS-SR

HAT

DDNM

198

198

198

198

198

Josefstr.161

198

198

ner AG

198

198

198

198

198

nerei

Pestalozzistrasse

sse

sse

asse

sse

sse

sse

sse

sse

sse

sse

sse

LR

ED

SL

SL

SL

SL

LSX-

SL

Su

ISL

SL

SL

Su

SL

SL

藤家

藤

10

33

Ground-truth HR

Bicubic

IPT

KernelGAN+ZSSR

ZSSR

EDSR

SRTTA(ours)

SRDiff

HAT

SRTTA-lifelong(ours)

(a) Limitation visualization under Impulse Noise.

Ground-truth HR

Bicubic

IPT

KernelGAN+ZSSR

ZSSR

EDSR

SRTTA(ours)

SRDiff

HAT

SRTTA-lifelong(ours)

(b) Limitation Visualization under Impulse Noise.

Ground-truth HR

Bicubic

IPT

KernelGAN+ZSSR

ZSSR

EDSR

SRTTA(ours)

SRDiff

HAT

SRTTA-lifelong(ours)

口

(c) Limitation visualization under Impulse Noise.

References

S. Bell-Kligler, A. Shocher, and M. Irani. Blind super-resolution kernel estimation using an internal-gan. In Advances in Neural Information Processing Systems , volume 32, 2019.

M. Bevilacqua, A. Roumy, C. Guillemot, and M. L. Alberi-Morel. Low-complexity single-image superresolution based on nonnegative neighbor embedding. In British Machine Vision Conference , pages 1-10. BMVA press, 2012.

T. Brooks, B. Mildenhall, T. Xue, J. Chen, D. Sharlet, and J. T. Barron. Unprocessing images for learned raw denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11036-11045, 2019.

J. Cai, H. Zeng, H. Yong, Z. Cao, and L. Zhang. Toward real-world single image super-resolution: A new benchmark and a new model. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3086-3095, 2019.

H. Chen, Y. Wang, T. Guo, C. Xu, Y. Deng, Z. Liu, S. Ma, C. Xu, C. Xu, and W. Gao. Pre-trained image processing transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12299-12310, 2021.

X. Chen, X. Wang, J. Zhou, Y. Qiao, and C. Dong. Activating more pixels in image super-resolution transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22367-22377, 2023.

X. Cheng, Z. Fu, and J. Yang. Zero-shot image super-resolution with depth guided internal degradation learning. In European Conference on Computer Vision , pages 265-280. Springer, 2020.

M. Emad, M. Peemen, and H. Corporaal. Dualsr: Zero-shot dual learning for real-world super-resolution. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1630-1639, 2021.

D. Glasner, S. Bagon, and M. Irani. Super-resolution from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 349-356. IEEE, 2009.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations , 2019.

Y. Huang, S. Li, L. Wang, T. Tan, et al. Unfolding the alternating optimization for blind super resolution. In Advances in Neural Information Processing Systems , volume 33, pages 5632-5643, 2020.

A. Ignatov, N. Kobyshev, R. Timofte, K. Vanhoey, and L. Van Gool. Dslr-quality photos on mobile devices with deep convolutional networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3277-3285, 2017.

H. Li, Y. Yang, M. Chang, S. Chen, H. Feng, Z. Xu, Q. Li, and Y. Chen. Srdiff: Single image superresolution with diffusion probabilistic models. Neurocomputing , 479:47-59, 2022.

J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1833-1844, 2021.

B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , pages 136-144, 2017.

P. Liu, H. Zhang, Y. Cao, S. Liu, D. Ren, and W. Zuo. Learning cascaded convolutional networks for blind single image super-resolution. Neurocomputing , 417:371-383, 2020.

Z. Luo, H. Huang, L. Yu, Y. Li, H. Fan, and S. Liu. Deep constrained least squares for blind image superresolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17642-17652, 2022.

S. Schulte, M. Nachtegael, V. De Witte, D. Van der Weken, and E. E. Kerre. A fuzzy impulse noise detection and reduction method. IEEE Transactions on Image Processing , 15(5):1153-1162, 2006.

A. Shocher, N. Cohen, and M. Irani. 'zero-shot' super-resolution using deep internal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3118-3126, 2018.

Q. Wang, O. Fink, L. Van Gool, and D. Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7201-7211, 2022.

X. Wang, L. Xie, C. Dong, and Y. Shan. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1905-1914, 2021.

X. Wang, K. Yu, C. Dong, and C. C. Loy. Recovering realistic texture in image super-resolution by deep spatial feature transform. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 606-615, 2018.

Y. Wang, J. Yu, and J. Zhang. Zero-shot image restoration using denoising diffusion null-space model. In The International Conference on Learning Representations , 2023.

P. Wei, Z. Xie, H. Lu, Z. Zhan, Q. Ye, W. Zuo, and L. Lin. Component divide-and-conquer for real-world image super-resolution. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VIII 16 , pages 101-117. Springer, 2020.

K. Zhang, J. Liang, L. Van Gool, and R. Timofte. Designing a practical degradation model for deep blind image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4791-4800, 2021.

M. Zhang, S. Levine, and C. Finn. Memo: Test time robustness via adaptation and augmentation. In Advances in Neural Information Processing Systems , volume 35, pages 38629-38642, 2022.

B. Zhou, H. Zhao, X. Puig, T. Xiao, S. Fidler, A. Barriuso, and A. Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision , 127:302-321, 2019.

M. Zontak and M. Irani. Internal statistics of a single natural image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 977-984. IEEE, 2011.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a fast test-time adaptation SR framework named SRTTA, which is capable of super-resolving images with various degradation types while maintaining efficiency. Specifically, the authors first exploit a pre-trained degradation classifier to predict the degradation type of the test image and construct the paired dataset based on the degradation. Then, the authors adapt the SR model by conducting the feature-level reconstruction learning from the test image and its second-order degraded image generated by the designed second-order degradation scheme. Finally, the authors design the second-order reconstruction loss to adapt the pre-trained model to test images with different degradation types.\n\nAfter reviewing the rebuttal, I've noticed that most of my previous concerns have been addressed. Therefore, I would like to increase my initial score.\n\nSTRENGTHS:\nThis paper provides an interesting image super-resolution idea for different degradations in the real-world test-time scenario. Compared with the classical deep network, it is applicable to any pretrained SR model. The proposed SRTTA can adapt a given SR model to estimate the degradation shifts in the test images. In addition, the authors construct a dataset named DIV2K-C with 8 different degradation types.\n\nWEAKNESSES:\n1. The proposed framework seems to be oriented towards a single degradation from different degradation types in the real-world test-time scenario. However, for the real-world scenario, the practical degradation is much more complex and could be the combinations of different degradations. This paper may not be applicable to this situation.\n2. The proposed dataset also may not be suitable for the aforementioned situation.\n3. The methods for visualization comparison are older compared to the methods for quantitative comparison.\n4. The GPU Time of the proposed method in Table 1 is not the optimal.\n5. The notation in section Super-Resolution Test-Time Adaptation should be consistent and continuous, such as x_t in line 149.\n\nQUESTIONS:\n1. The authors should clarify the 1st and 2nd concerns in section weaknesses. This seems to be an important reason why the proposed method is superior to other SOTA methods.\n2.  A detailed explanation or citation for the domain shifts should be given in line 136 on page 3.\n3. The authors should clarify the 3rd concern in section weaknesses and provide more convincing visualization.\n4. The authors mention that some previous methods are quite time-consuming in the motivation, so the authors should clarify the 4th concern in section weaknesses.\n5. How does the proposed network work when the test images are degraded by multiple degradation types at the same time? \n6. The motivation mentioned from line 158 to line 160 in section 4.1 should be further demonstrated or cited.\n\nLIMITATIONS:\nThe application scenario of the proposed method is limited for the single degradation.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a test-time adaptation framework called SRTTA for image super-resolution (SR) to address the degradation shift issue between training and test images. The SRTTA framework adapts a pre-trained SR model to different degradation types observed in real-world test images. It introduces a second-order degradation scheme that generates paired data based on the degradation type predicted by a pre-trained degradation classifier. The SR model is then adapted through feature-level reconstruction learning from the initial test image to its second-order degraded counterparts. Experimental results on a newly synthesized DIV2K-C dataset and a real-world DPED dataset demonstrate the effectiveness and practicality of the SRTTA framework.\n\nSTRENGTHS:\nThe paper introduces a new framework, SRTTA, that allows rapid adaptation of a pre-trained SR model to various degradation types observed in test images. The second-order degradation scheme and feature-level reconstruction learning provide a practical and effective solution for adapting the model to real-world scenarios.\n\nUnlike some existing methods that focus on a single degradation type, SRTTA considers multiple degradation types, such as blur, noise, and JPEG artifacts, making it more applicable in real-world scenarios where images can have diverse degradation types.\n\nSRTTA achieves fast adaptation by using only a single degraded low-resolution image during testing. This makes it more practical for real-time applications compared to methods that require extensive iterations or GAN-based approaches for degradation estimation.\n\nWEAKNESSES:\n1. Although the paper claims to evaluate the framework on a real-world DPED dataset, the evaluation results and analysis are not sufficient, in the per. The lack of comprehensive evaluation on diverse real-world datasets limits the generalizability and applicability of the proposed framework.\n\n2. The paper focuses on adapting the pre-trained SR model to different degradation types using second-order degradation. However, there is limited discussion or analysis on the generalization capability of the adapted model to unseen degradation types. It would be valuable to investigate the model's performance when faced with degradation types not encountered during training, or different types combine the degradations.\n\n3. The paper primarily focuses on evaluating the performance of the proposed method in terms of PSNR (Peak Signal-to-Noise Ratio). While PSNR is a commonly used metric, it does not always correlate well with perceived image quality. The evaluation could have included other quality metrics, such as Mean Opinion Score (MOS) or other human eye-related metrics, to provide a more comprehensive assessment of the proposed method.\n\nQUESTIONS:\nrefer to the weakness\n\nLIMITATIONS:\nyes, the authors have addressed some limitations of the paper", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a fast test time adaption method for image SR. The authors point out that the existing image SR method use synthesized paired LR-HR data for training, which degradation type has a gap with the real-world sceneries. The authors also proposed a second-order based data synthesis circuit with a degradation classifier. The proposed method achieves impressive results on both real-world and synthesis data.\n\nAfter reading the author's response, most of my problems have been addressed. Therefore, I would like to increase my initial score.\n\nSTRENGTHS:\nStrengths:\nExperimental results show that the proposed method is effective for real-world image super-resolution.\nVisual results are clear to show the improvement.\n\nWEAKNESSES:\nWeaknesses:\n1. The concept of second-order degradations is already proposed by real-ESRGAN in 2021.\n2. Please see the following detailed comments and suggestions.\n\nQUESTIONS:\nHere are some detailed comments and suggestions for improving the quality of this paper：\n1.\tIn the abstract, I think it’s better to say ‘conventional SR’ or ‘PSNR/fidelity oriented SR’use paired data for training.\n2.\tNovelty problems, second order degradation is already proposed by real-ESRGAN, the authors need to enhance the explanation of the contribution of this paper.\n3.\tIt is better to give a definition of the second-order degraded image like Equation.1\n4.\tThe degradation category definition is a discrete single label, second order requires repetition and mixing, so is the design of the classifier reasonable? Did the authors consider the sequence of the degradations?\n5.\tThe diagonal Fisher Information matrix in Section 4.3 requires independent identical distribution, is the image identically distributed under different degenerations? Equation 7 only reflects the stability of the same degradation under different augmentations, which does not seem to be able to target different degradations. In Equation 8, the parameter selection part I think is also not reasonable enough, through the artificial design of hyperparameters is usually difficult to meet the different datasets, so it is recommended to let the parameter be learnable.\n6.\tThe visualization results look a bit over-smoothed, it is suggested to add some analysis to it.\n\nLIMITATIONS:\nSee the above comments.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
The submission presents Test-time adaptation Super Resolution (SRTTA) via learning from the second-order degradation and reconstruction. This method super-resolve images with various degradation types while maintaining high efficiency. In addition, the paper also introduce a new dataset DIV2K-C, C stands for corruption, to better depict various image corruptions in the real-world SR use cases. The method is compared to other SOTA based this new dataset and achieves better metrics.

STRENGTHS:
There are 2 major contributions in this submission:
1) Adaptive Parameter Preservation module for the test time adaptation;
2) DIV2k-C dataset.

WEAKNESSES:
I'm really doubt about the problem setting. First, modern ISP always removes noises like Poisson noise, most images are clean besides some compression artifacts. Second, as the authors admit, if an image has some corruptions, it usually has more than 1 type of corruptions. The submitted method based on a classifier to choose at most 1 types of corruption does not follow the real-world scenarios.  Both of these two weakness challenges the fundamental design of the method.
In addition, the method is only evaluated on the new dataset created by the authors, which is far from enough.

QUESTIONS:
Since your degradation classifier is to determine the types of the degradation, which is usually a local effect, what's the reason of putting the whole image into the classifier (I assume you also have to resize your input image to 224x224) instead of putting patches of your input image into the classifier?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to solve the domain shift problem for a pretrained super-resolution model. A pretrained classifier is utilized to predict the type of degradation of the test image. After the degradation type is detected, the degradation is used as an augmentation scheme for adaptation training for the new SR model.\n\nSTRENGTHS:\nThe paper is easy to follow. \n\nThe motivation for the work is reasonable since super-resolution is an important task in Computer Vision, while the existing methods only focus on one degradation type. This results in the poor performance of the SR model in some cases.\n\nWEAKNESSES:\nThe analysis is the weakest part of the work:\n\n1. What are the consequences of domain shift when applying pretrained SR models? The work only mentions blur degradation is the most popular degradation type that other SR models concentrate on. However, no qualitative or quantitative observation is provided if another type of degradation happens in the dataset. What happens if the provided test datasets are degraded by blur and JPEG degradation? or both blur and noise degrade the provided test datasets? or only JPEG, or only noise?\n\n2. In practice, which type of degradation happens the most? Whether the cover for the domain shift is necessary? Which scenarios will we need to do the domain adaptation training for the SR model?\n\n3. The training of classifier in C.1 section mentions 8 types of degradation. What are the 8 types? Why, in the main paper, the classifier only has 4 classes?\n\n4. The qualitative results in Figure 3 shows that KernelGAN+ZSSR is better than the proposed method.\n\n5. In line 185, the author mentions, \"encourage the pre-trained model to learn how to reconstruct ...\", so the pretrained model will also be trained?\n\nQUESTIONS:\nsee the weaknesses\n\n(Weaknesses are addressed after rebuttal and discussions )\n\nLIMITATIONS:\nno limitations", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}