SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zeshuai Deng $^{1*}$  Zhuokun Chen $^{12*}$  Shuaicheng Niu $^{1*}$  Thomas H. Li $^{5}$  Bohan Zhuang $^{3\dagger}$  Mingkui Tan $^{124\dagger}$
$^{1}$ South China University of Technology,  $^{2}$ Pazhou Lab,  $^{3}$ ZIP Lab, Monash University,
<sup>4</sup>Key Laboratory of Big Data and Intelligent Robot, Ministry of Education,
$^{5}$ Peking University Shenzhen Graduate School
Image super-resolution (SR) aims to learn a mapping from low-resolution (LR) to high-resolution (HR) using paired HR-LR training images. Conventional SR methods typically gather the paired training data by synthesizing LR images from HR images using a predetermined degradation model, e.g., Bicubic down-sampling. However, the realistic degradation type of test images may mismatch with the training-time degradation type due to the dynamic changes of the real-world scenarios, resulting in inferior-quality SR images. To address this, existing methods attempt to estimate the degradation model and train an image-specific model, which, however, is quite time-consuming and impracticable to handle rapidly changing domain shifts. Moreover, these methods largely concentrate on the estimation of one degradation type (e.g., blur degradation), overlooking other degradation types like noise and JPEG in real-world test-time scenarios, thus limiting their practicality. To tackle these problems, we present an efficient test-time adaptation framework for SR, named SRTTA, which is able to quickly adapt SR models to test domains with different/unknown degradation types. Specifically, we design a second-order degradation scheme to construct paired data based on the degradation type of the test image, which is predicted by a pre-trained degradation classifier. Then, we adapt the SR model by implementing feature-level reconstruction learning from the initial test image to its second-order degraded counterparts, which helps the SR model generate plausible HR images. Extensive experiments are conducted on newly synthesized corrupted DIV2K datasets with 8 different degradations and several real-world datasets, demonstrating that our SRTTA framework achieves an impressive improvement over existing methods with satisfying speed. The source code is available at https://github.com/DengZeshuai/SRTTA.
Image super-resolution (SR) aims to reconstruct plausible high-resolution (HR) images from the given low-resolution (LR) images, which is widely applied in microscopy [42, 44], remote sensing [17, 35] and surveillance [40, 56]. Most previous SR methods [11, 31, 60, 8] hypothesize that the LR images are downsampled from HR images using a predefined degradation model, e.g., Bicubic down-sampling. However, due to the diverse imaging sensors and multiple propagations on the Internet, real-world images may contain different degradation types (e.g., Gaussian blur, Poisson noise, and JPEG artifact) [32, 52, 53, 25]. Besides, the realistic degradations of real-world images
*Equal contribution. Email: sedengzeshuai@mail.scut.edu.cn, {caesard216, niushuaicheng}@gmail.com
†Corresponding author. Email: mingkuitan@scut.edu.cn, bohan.zhuang@gmail.com
may dynamically change, which are often different from the training one, limiting the performance of pre-trained SR models in dynamically changing test-time environments.
Recently, zero-shot SR methods [43, 9, 12] are proposed to train an image-specific SR model for each test image to alleviate the degradation shift issue. For example, ZSSR [43] uses a predefined/estimated degradation model to generate an image with a lower resolution for each test image. With this downsampled image and the test image, they can train an image-specific SR model to super-resolve the test image. Moreover, DualSR [12] estimates the degradation model and trains the SR model simultaneously to achieve better performance. However, these methods usually require thousands of iterations to estimate the degradation model or train the SR model, which is very time-consuming. Thus, these methods cannot handle real-world test images with rapidly changing domain shifts.
To reduce the inference time of zero-shot methods, some recent works [45, 41] introduce meta-learning [13] to accelerate the adaptation of the SR model, which still requires a predefined/estimated degradation model to construct paired data to update the model. However, most degradation estimation methods [2, 30] focus on the estimation of one degradation type, which limits the adaptation of SR models to test images with other degradations. Recently, test-time adaptation (TTA) methods [46, 47, 57, 38, 48, 39] are proposed to quickly adapt the pre-trained model to the test data in target domain without accessing any source training data. These methods often use simple augmentation operations (e.g., rotation or horizontal flip) on the test image, and construct the pseudo label as the average of the predicted results [57, 48]. For image SR, the pseudo-HR image constructed using this scheme [57, 48] may still contain the degradation close to the test image (e.g., Gaussian blur). With such a pseudo-HR image, the adapted SR model may not be able to learn how to remove the degradation from the test image (see results in Table 1). Therefore, how to quickly and effectively construct the paired data to encourage the SR model to remove the degradation is still an open question.
In this paper, we propose a super-resolution test-time adaptation framework (SRTTA) to adapt a trained super-resolution model to target domains with unknown degradations, as shown in Figure 1. When the degradation shift issue occurs, the key challenge is how to quickly and effectively construct (pseudo) paired data to adapt SR models to the target domain without accessing any clean HR images. To this end, we propose a second-order degradation scheme to construct (pseudo) paired data. Specifically, with a pre-trained degradation classifier, we quickly identify the degradation type from the test images and randomly generate a set of degradations to obtain the second-order degraded images. The paired data, which consists of the second-order degraded images and the test image, enables a rapid adaptation of SR models to the target domain with different degradations. To facilitate the learning of reconstruction, we design a second-order reconstruction loss to adapt the pre-trained model using the paired data in a self-supervised manner. After fast adaptation using our method, the SR model is able to learn how to remove this kind of degradations and generate plausible HR images. Moreover, we also design an adaptive parameter preservation strategy to preserve the knowledge of the pre-trained model to avoid the catastrophic forgetting issue in long-term adaptation. Last but not least, we use eight different degradations to construct two new benchmarks, named DIV2K-C and DIV2K-MC, to comprehensively evaluate the practicality of our method. Experimental results on both our synthesized datasets and several real-world datasets demonstrate that our SRTTA is able to quickly adapt the SR model to the test-time images and achieve an impressive improvement.
- A novel test-time adaptation framework for image super-resolution: We propose a super-resolution test-time adaptation (SRTTA) framework to adapt any pre-trained SR models to different target domains during the test time. Without accessing any ground-truth HR images, our SRTTA is applicable to practical scenarios with unknown degradation in a self-supervised manner.
- A fast data construction scheme with second-order degradation: We use a pre-trained classifier to identify the degradation type for a test image and construct the paired data using our second-order degradation scheme. Since we do not estimate the parameters of the degradation model, our scheme enables a rapid model adaptation to a wide range of degradation shifts.
- New test datasets with eight different domains: We construct new test datasets named DIV2K-C and DIV2K-MC, which contain eight common degradations, to evaluate the practicality of different SR methods. Experimental results on both synthesized datasets and real-world datasets demonstrate the superiority of our SRTTA, e.g., 0.84 dB PSNR improvement on DIV2K-C over ZSSR [43].
Real-world super-resolution. To alleviate the domain shift issues, GAN-based methods [54, 5, 24, 36] tend to learn the degradation model of the real-world images in the training stage. These methods often train a generator that explicitly learns the degradation model of real-world images. Besides, some methods [55, 60, 49] try to enumerate most of the degradation models that can be encountered in real-world applications. Based on the estimated/predefined degradation models, these methods can generate LR images whose distribution is similar to real-world images. However, due to the complex and unknown processing of real-world images, it is hard to mimic all types of degradation during the training phase. Instead, some existing methods [15, 2, 22, 21, 34] try to estimate the image-specific degradation model during the test time, which helps to reconstruct more plausible HR images. For instance, optimization-based methods [15, 22, 21] estimate the blur kernel and SR image together in an iterative manner. However, these methods cannot generate satisfactory results when the test images contain different types of degradation (e.g., Poisson noise and JPEG artifact) [32]. Thus, these methods still suffer from the domain shift on test images with unknown degradation.
Zero-shot super-resolution. Zero-shot methods [43, 9, 45, 41, 12] aim to train an image-specific SR model for each test image to alleviate the domain shift issue. These methods [43, 12] use a predefined/estimated degradation model to generate an image with a lower resolution from each test image. To estimate the image-specific degradation in a zero-shot manner, KernelGAN [2] utilizes the internal statistics of each test image to learn the degradation model specifically and then uses ZSSR [43] to train an SR model with the estimated degradation. However, these methods usually require a lot of time to estimate the degradation model or train the SR model. MZSR [45] and MLSR [41] are proposed to reduce the number of iteration steps for each test image during test time. Recently, DDNM [51] was proposed to use a pre-trained diffusion model to ensure the generated images obey the distribution of natural images. However, these methods still require a predefined (Bicubic downsampling) or an estimated degradation model. The predefined Bicubic degradation suffers from the domain shift due to its difference from the underlying degradation of real-world images. The estimation methods [2, 30] may focus on the estimation of a single degradation type (e.g., blur) while ignoring other degradation. Thus, these methods often result in unsatisfactory HR images for the test images with different degradation types [32]. In this paper, we use a degradation classifier to quickly recognize the degradation type and randomly generate the degradations with this type. Therefore, we do not need to estimate the degradation model, which is time-saving.
Test-time adaptation. Recently, test-time adaptation (TTA) methods [46, 47, 57, 38, 48, 6, 39, 59] have been proposed to alleviate the domain shift issue by online updating the pre-trained model on the test data. TTT [46] uses an auxiliary head to learn the test image information from the self-supervised task. Tent [47] proposed to adapt the pre-trained model with entropy-based loss in an unsupervised manner. CoTTA [48] uses a weight-averaged pseudo-label over training steps to guide the pre-trained model adaptation. However, these methods are mainly developed for image classification and may ignore the characteristics of image super-resolution. Thus, these methods may not be effective in adapting the SR model to remove the degradation from the test image. In this paper, we focus on the image SR task and address the degradation shift issue with our SRTTA framework.
3 Preliminary and Problem Definition
Notation. Without loss of generality, let  $\mathbf{y}$  be a clean high-resolution (HR) image and  $\mathbf{x}_c$  be the clean low-resolution (LR) image downsampled from  $\mathbf{y}$  using Bicubic interpolation, i.e.,  $\mathbf{x}_c = \mathbf{y} \downarrow s$ , where  $s$  is the scale factor of Bicubic downsampling. Let  $\mathbf{x}$  denote a real-world test image degraded from  $\mathbf{y}$ , i.e.,  $\mathbf{x} = \mathbf{D}(\mathbf{y})$ , where  $\mathbf{D}(\cdot)$  is the degradation process. We use  $\mathbf{x}_{sd}$  to denote the LR image that is further degraded from the real-world image  $\mathbf{x}$ . In this paper, we call the test image  $\mathbf{x}$  as a first-order degraded image and the image  $\mathbf{x}_{sd}$  degraded from  $\mathbf{x}$  as a second-order degraded image.  $f_{\theta}(\cdot)$  is a super-resolution (SR) model with parameters  $\theta$ .
Image degradation. The degradation process of real-world test images can be modeled by a classical degradation model  $\mathbf{D}(\cdot)$  [33, 49]. Formally, let  $\mathbf{k}$  be a blur kernel,  $\mathbf{n}$  be an additive noise map and  $q$  be the quality factor of JPEG compression, the degraded image  $\mathbf{x}$  is defined by
$$
\mathbf {x} = \mathbf {D} (\mathbf {y}) = [ (\mathbf {y} \otimes \mathbf {k}) \downarrow_ {s} + \mathbf {n} ] _ {J P E G _ {q}}, \tag {1}
$$
where  $\otimes$  denotes the convolution operation,  $\downarrow_{s}$  denotes the downsampling with a scale factor of  $s$ , and  $JPEG_{q}$  denotes the JPEG compression with the quality factor  $q$ . Similarly, the second-order
Figure 1: An overall illustration of the proposed super-resolution test-time adaptation (SRTTA) framework. Given a test image  $\mathbf{x}$ , we use a pre-trained degradation classifier to predict the degradation type  $C(\mathbf{x})$ , e.g., blur, noise, and JPEG degradation. Based on the predicted degradation type  $C(\mathbf{x})$ , we construct a set of paired data  $\{\mathbf{x}_{sd}^i,\mathbf{x}\}_{i = 1}^N$  and adapt the SR model with our adaptation loss  $\mathcal{L}_a$  and  $\mathcal{L}_s$ . When test samples are clean images, we directly use the frozen pre-trained SR model to super-resolve these clean images without adaptation.
degraded image  $\mathbf{x}_{sd}$  can be formulated as
$$
\mathbf {x} _ {s d} = \mathbf {D} (\mathbf {D} (\mathbf {y})) = \mathbf {D} (\mathbf {x}) = [ (\mathbf {x} \otimes \mathbf {k}) \downarrow_ {s} + \mathbf {n} ] _ {J P E G _ {q}}. \tag {2}
$$
Degradation shift between training and testing. Existing SR methods [31, 60, 16] often construct paired HR-LR training images by either collecting from the real world or synthesizing LR images from HR images via a pre-defined degradation model, i.e., Bicubic down-sampling. However, due to diverse camera sensors and the unknown processing on the Internet, the degradation process of real-world test images may differ from that of training images, called domain shifts [27, 48, 32]. In these cases, the SR model often fails to generate satisfactory HR images.
Motivation and challenges. Though recently some blind SR methods [12, 43] have been proposed to address the degradation shift issue, they still suffer from two key limitations: low efficiency and narrow focus on a single degradation type, e.g., blur degradation. In this work, we seek to resolve these issues by directly learning from the shifted testing LR image at test time, which poses two major challenges: 1) How to quickly and effectively construct the (pseudo) paired data to adapt SR models to test domains with unknown degradations? and 2) How to design a generalized test-time learning framework that facilitates the removal of various types of degradation, considering that we have only a low-resolution test image at our disposal?
4 Efficient Test-Time Adaptation for Image Super-Resolution
In this section, we illustrate our proposed super-resolution test-time adaptation (SRTTA) framework that is able to quickly adapt the pre-trained SR model to real-world images with different degradations. The overall framework and pipeline are shown in Figure 1 and Algorithm 1.
Given a test image  $\mathbf{x}$ , we first construct the paired data using our second-order degradation scheme. Specifically, we use a pre-trained degradation classifier to recognize the degradation type for the test image. Based on the predicted degradation type, we randomly generate a set of degradation (e.g., a set of blur kernels  $\mathbf{k}$ ) and use them to construct a set of paired data  $\{\mathbf{x}_{sd}^i,\mathbf{x}\}_{i = 1}^N$ . With the paired data, we adapt the pre-trained SR model to remove the degradation from the test image. Notably, before performing the test-time adaptation, we freeze the important parameters to preserve the knowledge of the pre-trained model to alleviate the forgetting problem in long-term adaptation. After adaptation, we use the adapted model to generate the corresponding HR image for the test image.
Algorithm 1: The pipeline of the proposed Super-Resolution Test-Time Adaptation.
Input: Real-world test images  $\{\mathbf{x}_t\}_{t = 1}^T$  , adaptation iteration steps  $S$  for each image, learning rate  $\eta$  , batch size  $N$  , preservation ratio  $\rho$    
1 Load the pretrained SR models  $f_{\theta^0}(\cdot)$  and the degradation classifier  $C(\cdot)$    
2 Select and freeze the important parameters using Eqn. (9) with  $\rho$    
3 for  $\mathbf{x}_t$  in  $\{\mathbf{x}_t\}_{t = 1}^T$  do for s in  $\{1,2,\dots,S\}$  do Construct paired data  $\{\mathbf{x}_{sd}^{i},\mathbf{x}_{t}\}_{i = 1}^{N}$  based on  $C(\mathbf{x}_t)$  using Eqn. (3); Adapt the SR model using Eqn. (6) with  $\eta$  .   
7 end   
8 end Output: The adapted SR model  $f_{\theta}$  , the predictions  $\{\hat{\mathbf{y}}_t = f_\theta (\mathbf{x}_t)\}_{t = 1}^T$  for all  $\mathbf{x}_t$  in  $\{\mathbf{x}_t\}_{t = 1}^T$
4.1 Adaptive Data Construction with Second-Order Degradation
In this part, we propose a novel second-order degradation scheme to effectively construct paired data, enabling the fast adaptation of SR models to the target domain with different degradations.
Unlike existing methods [43, 2, 30], we consider more degradation types and avoid estimating the degradation model. Existing methods [43, 2, 30] mainly focus on precisely estimating the blur kernels when constructing the lower-resolution images (second-order degraded images), which is time-consuming. Instead, we use a pre-trained degradation classifier to quickly identify the degradation types (blur, noise, and JPEG) of test images, and then we construct the second-order degraded images based on the predicted degradation types. Without the time-consuming degradation estimation process, our scheme enables a fast model adaptation to a wide range of degradation shifts.
Adaptive data construction. In this part, we design an adaptive data construction method to obtain the second-order degraded images  $\mathbf{x}_{sd}^{i}$ . Specifically, based on the classical degradation model in Eqn. (1), we train a multi-label degradation classifier  $C(\cdot)$  to predict the degradation types for each test image, including blur, noise and JPEG degradation types, denoted by  $c_{b}$ ,  $c_{n}$  and  $c_{j} \in \{0,1\}$ , respectively. With the predicted degradation types, we randomly generate  $N$  degradations and construct a set of second-order degraded images  $\{\mathbf{x}_{sd}^{i}\}_{i=1}^{N}$ , which can be formulated as
$$
\mathbf {x} _ {s d} = D (\mathbf {x}, C (\mathbf {x})) = D _ {j} (D _ {b} (\mathbf {x}, c _ {b}) + D _ {n} (c _ {n}), c _ {j}),
$$
$$
D _ {b} (\mathbf {x}, c _ {b}) = c _ {b} (\mathbf {x} \otimes \mathbf {k}) + (1 - c _ {b}) \mathbf {x}, D _ {n} \left(c _ {n}\right) = c _ {n} \mathbf {n}, \tag {3}
$$
$$
D _ {j} (\mathbf {x}, c _ {j}) = c _ {j} J P E G _ {q} (\mathbf {x}) + (1 - c _ {j}) \mathbf {x},
$$
where the blur kernel  $\mathbf{k}$ , noise map  $\mathbf{n}$  and quality factor  $q$  are randomly generated using a similar recipe of Real-ESRGAN [49]. Unlike previous methods [43, 2], we do not further downsample the test image  $\mathbf{x}$  when constructing  $\mathbf{x}_{sd}$ , since the pretrained SR model has learned the upsampling function (the inverse function of downsampling) during the training phase. Due to the page limit, we put more details in the supplementary materials.
Since the pre-trained SR model has been well-trained on the clean domain (Bicubic downsampling), we simply ignore adapting the clean images in test-time. For these images, we use the pre-trained SR model to super-resolve them, i.e.,  $\hat{\mathbf{y}} = f_{\theta^0}(\mathbf{x})$  when  $c_{b} = c_{n} = c_{j} = 0$ .
4.2 Adaptation with Second-Order Reconstruction
In our SRTTA framework, we design a self-supervised adaptation loss and an adaptation consistency loss to update the pre-trained SR models to test images with degradation.
Self-supervised adaptation. To adapt the pre-trained model to remove the degradation, we design a self-supervised adaptation loss based on the Charbonnier penalty function [4, 28]. Specifically, we encourage the SR model to reconstruct the test images  $\mathbf{x}$  from the second-order degraded images  $\mathbf{x}_{sd}$  at the feature level, which can be formulated as
$$
\mathcal {L} _ {s} (\mathbf {x}, \mathbf {x} _ {s d}) = \sqrt {\left(f _ {\theta} ^ {l} (\mathbf {x}) - f _ {\theta} ^ {l} (\mathbf {x} _ {s d})\right) ^ {2} + \epsilon}, \tag {4}
$$
where  $f_{\theta}^{l}(\cdot)$  denotes the output features of the  $l$ -th layer. We simply set  $f_{\theta}^{l}(\cdot)$  to be the output features of the second-to-last convolution layer.  $\epsilon$  is a small positive value that is set to  $10^{-3}$  empirically.
Consistency maximization. To keep the model consistent across adaptation, we design an adaptation consistency loss to encourage the output of the adapted model to be close to that of the pre-trained model, which is formulated as
$$
\mathcal {L} _ {a} (\mathbf {x}, \mathbf {x} _ {s d}) = \sqrt {\left(f _ {\theta 0} ^ {l} (\mathbf {x}) - f _ {\theta} ^ {l} (\mathbf {x} _ {s d})\right) ^ {2} + \epsilon}, \tag {5}
$$
where  $f_{\rho^0}^l (\cdot)$  denotes the output features of the  $l$ -th layer of the pre-trained SR model.
Second-order reconstruction loss. Our final second-order reconstruction loss consists of a self-supervised adaptation loss and an adaptation consistency loss, which is formulated as
$$
\mathcal {L} = \mathcal {L} _ {s} (\mathbf {x}, \mathbf {x} _ {s d}) + \alpha \mathcal {L} _ {a} (\mathbf {x}, \mathbf {x} _ {s d}), \tag {6}
$$
where  $\alpha$  is a balance hyperparameter (the ablation study can be found in Table 5).
4.3 Adaptive Parameter Preservation for Anti-Forgetting
To avoid catastrophic forgetting in long-term adaptation, we propose an adaptive parameter preservation (APP) strategy to freeze the important parameters during adaptation. To select the important parameters, we evaluate the importance of each parameter using the Fisher information matrix [26, 38]. Given a set of collected clean images  $\mathcal{D}_c$ , we design an augmentation consistency loss  $\mathcal{L}_c$  to compute the gradient of each parameter. Based on the gradient, we compute the diagonal Fisher information matrix to evaluate the importance of each parameter  $\theta_i^0$ , which is formulated as
$$
\omega \left(\theta_ {i} ^ {0}\right) = \frac {1}{\left| \mathcal {D} _ {c} \right|} \sum_ {\mathbf {x} _ {c} \in \mathcal {D} _ {c}} \left(\frac {\partial \mathcal {L} _ {c} \left(\mathbf {x} _ {c}\right)}{\partial \theta_ {i} ^ {0}}\right) ^ {2}, \tag {7}
$$
$$
\mathcal {L} _ {c} \left(\mathbf {x} _ {c}\right) = \sqrt {\left(\bar {\mathbf {y}} - f _ {\theta^ {0}} \left(\mathbf {x} _ {c}\right)\right) ^ {2} + \epsilon}, \quad s. t. \quad \bar {\mathbf {y}} = \frac {1}{8} \sum_ {i = 1} ^ {8} \mathbf {R} _ {i} \left(f _ {\theta^ {0}} \left(\mathbf {A} _ {i} \left(\mathbf {x} _ {c}\right)\right)\right), \tag {8}
$$