SlowGuess/NeurIPS_2023_OCR huggingface dataset:
E.1 More Results of Test-Time Image Super-Resolution 6
E.2 More Results on Test domains with Mixed Multiple Degradations. 7
E.3 More Results of Ablation Studies 7
E.4 Effectiveness on a New Unknown Domain 8
E.5 Comparison with Patch-Recurrence Reconstruction Loss 8
E.6 Effect of the Feature-level Reconstruction 8
E.7 Effect of Adaptation Iterations for Each Image 9
E.8 The Statistics of the Degradation Types of Real-world Images 9
E.9 Further Analysis of the Domain Shift Issue 9
F Visualization Results 10
F.1 Visualization Results on the DIV2K-C Images 10
F.2 Visualization Results on the Real-World Images 10
G Limitation Analysis and Border Impacts 11
G.1 Limitation Analysis 11
G.2 Broader Impacts 12
A More Details of the Second-Order Degradation
In our second-order degradation scheme, we randomly generate different degradations to degrade the test image into its second-order degraded counterparts. In this section, we illustrate how to randomly generate different types of degradations. Notably, we degrade the test images on the GPU device to accelerate the degradation process.
A.1 Random Blur Degradation
Following [26, 22], we model blur degradation as a convolution with a linear Gaussian blur filter/kernel. Given a test image, we randomly generate a set of isotropic or anisotropic Gaussian kernels  $\mathbf{k}$  and use them to perform blur degradation on the test image. The probabilities of generating an isotropic kernel and an anisotropic kernel are set to 0.5 and 0.5, respectively. The size of each generated kernel is uniformly sampled from  $\{7\times 7,9\times 9,\dots,21\times 21\}$ . We sample the standard deviation of the blur kernel along the two principal axes  $\sigma_{1}$  and  $\sigma_{2}$  uniformly from [0.2, 3]. If  $\mathbf{k}$  is an isotropic Gaussian blur kernel, we set  $\sigma_{2}$  equal to  $\sigma_{1}$ . If  $\mathbf{k}$  is an anisotropic Gaussian kernel, we further sample a rotation angle  $a$  uniformly from  $[- \pi, \pi]$ , and use a rotation matrix to transform the generated kernel based on the angle  $a$ . More details about how to generate a Gaussian blur kernel can refer to Real-ESRGAN [22].
A.2 Random Noise Degradation
In our second-order degradation, we randomly generate a set of Gaussian noise maps  $\mathbf{n}$ , and add them to the test image to obtain a set of second-order images with different additive Gaussian noise. With a probability of  $60\%$ , we generate noise for each channel of RGB images independently, otherwise, we generate the same noise map for all three channels. We first generate a noise map whose values are randomly generated from a normal Gaussian distribution. Then we sample a scale value to enlarge the noise uniformly from [1, 30]. More details can be referred to Real-ESRGAN [22].
A.3 Random JPEG Degradation
For JPEG compression  $JPEG_{q}$ , we sample a quality factor  $q$  uniformly from [30, 95], and use the JPEG compression with the degradation  $q$  to degrade test images into a set of second-order degraded images with compression artifacts. Note that JPEG compression with a lower  $q$  compress the test image with a higher compression ratio and the compressed images are generally of a lower quality. To accelerate the degradation process, we use  $\mathrm{DiffJPEG}^{1}$ , which is the PyTorch implementation of JPEG compression, to process the test image on the GPU device.
B The Difference from Real-ESRGAN
In this part, we would like to discuss the difference between our SRTTA with Real-ESRGAN[22], which proposes the concept of second-order degradations, to highlight the contribution of our SRTTA.
B.1 Solving Different Problems
Real-ESRGAN tries to enumerate all the degradations in real-world scenes and train an SR model to solve the image restoration on any degradation. However, it is non-trivial to obtain all real-world degradations, leading to domain shift issues when encountering unknown degradations during testing, as shown in Figure 11 of real-ESRGAN [22]. Unlike real-ESRGAN [22], our SRTTA aims to adapt the SR models to the test domains when test images contain unknown degradations. Our second-order degradation scheme aims to quickly construct the pseudo-paired data (instead of the paired training data) to adapt the SR model to the test domains.
B.2 Different Construction Schemes
Real-ESRGAN [22] proposes the second-order degradation to construct the paired training data, whose low-resolution (LR) images are obtained from the ground-truth high-resolution (HR) images.
$^{1}$ https://github.com/mlomnitz/DiffJPEG
Then, the paired data is used to train an SR model during the training phase in a supervised learning manner. Notably, the trained Real-ESRGAN [22] model is fixed during the test time. Instead, our second-order degradation scheme constructs the pseudo-paired data using the test images with unknown degradation (first-order degraded images). Our SRTTA model is continuously adapted to different domains during testing in a self-supervised learning manner.
C Experimental Datasets
C.1 Construction Details of the DIV2K-C Dataset
To evaluate the practicality, we construct a new benchmark dataset named DIV2K-C, which contains eight different degradations. We select the eight degradation types from the 15 corruptions of ImageNet-C [11] that do not extremely change the image content, including Gaussian Blur, Defocus Blur, Glass Blur, Gaussian Noise, Poisson Noise (Shot Noise), Impulse Noise, Speckle Noise, and JPEG compression. Unlike the ImageNet-C [11], we do not use the same degradation level to degrade all test images. Instead, we randomly generate the degradation level and further generate a degradation for each image based on the degradation level. Unlike prior SR methods that investigate a limited number of degradation types, the degradation scenarios we considered are more complex (eight degradation types in total), which is more practical for real-world applications.
Given a degradation, we use the classical image degradation model [17, 22] to generate the low-resolution (LR) test images from the high-resolution clean images. For blur degradation, we perform the blur convolution on the HR images and then use the Bicubic downsampling to obtain the test LR images. For noise and JPEG degradation, we first use the Bicubic downsampling to obtain clean LR images and then perform noise or JPEG degradation on the clean LR images to obtain the final test images. We show the visualization of some examples regarding each degradation type in Figure A.
0826 from DIV2K
Glass Blur
0846 from DIV2K
Figure A: The visualization of examples regarding each degradation type on the DIV2K-C dataset.
Clean LR
Poisson Noise
Impulse Noise
Speckle Noise
Gaussian blur. Following BSRGAN [26], we generate low-resolution (LR) images with Gaussian blur degradation. We randomly generate an isotropic Gaussian kernel or an anisotropic Gaussian kernel for each high-resolution (HR) image. Then, we use the blur kernel to perform blur convolution on the HR image and use Bicubic downsampling to obtain the final LR test images. For simplicity, we follow the recipe of BSRGAN [26] to generate test images with blur degradation.
Defocus blur. To better compare the performance of different SR methods, we also use the common Defocus blur degradation to degrade the HR images and obtain the LR images using the Bicubic downsampling. As illustrated in ImageNet-C [11], Defocus blur often occurs when an image is out of focus when we take pictures. We generate the blur kernel as ImageNet-C [11] and perform a blur convolution on the HR images. But unlike ImageNet-C [11], the degradation level of Defocus blur is randomly sampled from a given range.
Glass blur. We also choose another common degradation type, Glass Blur, which appears with "frosted glass" windows or panels [11]. This blur degradation requires two Gaussian blur operations and an operation that locally shuffles pixels between two blur operations. As mentioned above, the degradation level is randomly sampled from a given range, such as the standard deviation of the Gaussian blur kernel or the window size of the shuffling operation.
Gaussian noise. To generate test images with Gaussian noise, we sample the noise for each pixel from a normal Gaussian distribution. The mean of the Gaussian distribution is zero, and the standard deviation is uniformly sampled from the range of  $\{2 / 255,3 / 255,\dots,25 / 255\}$ . More details can refer to the implementation of BSRGAN [26].
Poisson (Shot) noise. Poisson noise, also called Shot noise, can model the sensor noise caused by statistical quantum fluctuations. We randomly generate the Poisson noise map from a Poisson distribution, which has an intensity proportional to the image intensity. Then, we add the generated Poisson noise into the clean LR images to obtain the test images with Poisson noise. More details can refer to the implementation of Real-ESRGAN [22].
Impulse noise. Impulse noise is caused by errors in the data transmission generated in noisy sensors or communication channels, or by errors during the data capture from digital cameras [19]. The most common form of Impulse noise is called salt-and-pepper noise. To generate test images with Impulse noise, we uniformly select a set of pixels and replace them with zero or the maximum value (255). More details can be referred to ImageNet-C [11].
Speckle noise. Speckle noise is an additive noise where the noise added to a pixel tends to be larger if the original pixel intensity is larger. We first sampled the noise for each pixel from a Gaussian distribution and multiple the noise value by the original pixel. Last, we add the generated noise map into the LR clean images to obtain the final test images with Speckle noise.
JPEG compression. For JPEG compression, we use the OpenCV implementation of JPEG compression<sup>2</sup> to degrade the clean LR image into final test images. The compression quality factor  $q$  is randomly sampled from [30, 90]. We first encode the clean LR images into the bit stream using JPEG compression with the quality factor  $q$  and decode the bit stream to obtain the final test images. Note that JPEG compression is a lossy compression technique, so the final test images are inevitably corrupted with JPEG compression artifacts.
C.2 Construction Details of the DIV2K-MC Dataset
Since the real-world test images may contain multiple degradation types simultaneously, we further develop a new benchmark dataset named DIV2K-MC, which includes four test domains with mixed multiple degradations. The four domains are BlurNoise, BlurJPEG, NoiseJPEG and BlurNoiseJPEG. The test images in the BlurNoiseJPEG domain contain the combined degradation of Gaussian blur, Gaussian noise and JPEG degradations simultaneously.
BlurNoise. We generate LR images from HR images using Gaussian blur and Gaussian noise degradation. We first randomly generate a Gaussian blur kernel to perform blur convolution on the HR image. Then, we downsample the resulting image using Bicubic interpolation. Last, we randomly sample a Gaussian noise map and add it to the downsampled image to obtain the final LR image.
BlurJPEG. We generate LR images from HR images using Gaussian blur and JPEG degradation. We first randomly generate a Gaussian blur kernel to perform blur convolution on the HR image. Then, the resulting image is downsampled by using Bicubic interpolation. Last, we use JPEG compression with a random quality factor  $q$  to compress the downsampled image to obtain the final LR image.
NoiseJPEG. We generate LR images from HR images using Gaussian noise and JPEG degradation. We first downsample HR image using Bicubic interpolation. Then, we randomly sample a Gaussian
$^{2}$ https://github.com/opencv/opencv
noise map and add it to the downsampled image. Last, we use JPEG compression with a random quality factor  $q$  to compress the downsampled image to obtain the final LR image.
BlurNoiseJPEG. We generate LR images from HR images using Gaussian blur, Gaussian noise and JPEG degradation. We first randomly generate a Gaussian blur kernel to perform blur convolution on the HR image. Second, the resulting image is downsampled by using Bicubic interpolation. Then, we randomly sample a Gaussian noise map and add it to the downsampled image. Last, we use JPEG compression with a random quality factor  $q$  to compress the image to obtain the final LR image.
C.3 More Test Datasets for Test-Time Image Super-Resolution
Moreover, we also evaluate the performance of SR methods on real-world test images from DPED [13], ADE20K [28] and OST300 [23], whose corresponding ground-truth HR images can not be found. To evaluate the anti-forgetting performance, we report the adapted model performance on a clean benchmark dataset Set5 [2] whose images are clean images that are downsampled from HR images with Bicubic interpolation. Thus, these LR images do not contain any degradation.
D.1 Implementation Details of the Degradation Classifier
In our second-order degradation scheme, we use a pre-trained degradation classifier to predict the degradation type for each test image. To obtain the pre-trained degradation classifier, we use ResNet-50 [10] as the classifier and train it to recognize the degradation from test images.
In real-world scenes, test images may contain degradations other than these eight degradation types, such as ringing or overshoot artifacts [22], which may be viewed as variations of blur, noise or JPEG. Since it is infeasible to cover all the degradation types in real-world scenes, we make the degradation classifier to predict the coarse-level four classes, including clean, blur, noise and JPEG.
Training details. Specifically, we use the 800 training HR images of DIV2K and randomly crop them into patches with the size of  $224 \times 224$  (instead of resizing them into  $224 \times 224$ ). Similar to the construction of DIV2K-C, we degrade each patch using a random selection of one of eight degradation types. As for clean data, we do not perform any degradation on the patches. For training, we apply Adam with  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$  and set the batch size as 256. The learning rate is initialized to  $10^{-3}$  and decreased to  $10^{-6}$  with a cosine annealing out of 400 epochs in total.
Testing details. During testing, we directly input the whole test image with original resolution into the classifier and output the predicted results to recognize the degradation type. The predicted results of the multi-label degradation classifier  $C(\cdot)$  are the probabilities of the three degradations, including blur, noise and JPEG degradation. If the predicted probability of one degradation type is larger than the threshold of 0.5, the test image is considered to contain the degradation of this type. The clean image means that this image does not contain any degradation such as blur, noise, or JPEG, and we directly use the pre-trained SR model to super-resolve these clean test images.
D.2 More Details of Super-Resolution Test-Time Adaptation
We use the baseline model of EDSR [16] with less than 2M parameters as our pre-trained SR model for  $2 \times$  and  $4 \times$  SR. During adaptation, we only update the parameters in the Resblock of the EDSR model. To avoid anti-forgetting, we use five clean test LR images from Set5 [2] to select important parameters to be frozen in Eqn. (9). Moreover, when evaluating the anti-performance Set5 [2], we directly use the adapted model to super-resolve the test images without using the classifier.
In our experiment, we conduct experiments in parameter-reset and lifelong settings. In the parameter-reset setting, the model parameters will be reset after the adaptation on each domain, which is the default setting of our SRTTA. In the lifelong setting, the model parameters will never be reset in the long-term adaptation, in this case, we call our methods as SRTTA-lifelong.
For test-time adaptation, we use the Adam optimizer with the learning rate of  $5 \times 10^{-5}$  for the pre-trained SR models. We set the batch size  $N$  to 32, and we randomly crop the test image into  $N$  patches of size  $96 \times 96$  and  $64 \times 64$  for  $2 \times$  and  $4 \times$  SR, and degrade them into second-order degraded patches. We perform  $S = 10$  iterations of adaptation for each test image. For the balance weight in
Eqn. (6), we set  $\alpha$  to 1. For the ratio of parameters to be frozen, we set the  $\rho$  to 0.50. To compare the inference times of different SR methods, we measure all methods on a TITAN XP with 12G graphics memory for a fair comparison. Due to the memory overload of HAT [6] and DDNM [24], we chop the whole image into smaller patches and process them individually for these two methods. In our experiments, we use the bold number to indicate the best result and the underlined number to indicate the second-best result.
E.1 More Results of Test-Time Image Super-Resolution
In this part, we compare our SRTTA with existing SR methods, including supervised pre-trained SR methods, blind SR methods, zero-shot SR methods, and a TTA baseline with consistency loss. 1) The supervised pre-trained SR methods learn to process test images with a predefined degradation process, i.e., the Bicubic downsampling. These methods include the EDSR baseline [16], SwinIR [15], IPT [5] and HAT [6], SRDiff [14], and Real-ESRNet [22]. Note that Real-ESRNet [22] is the PSNR-oriented model of Real-ESRGAN [22], which use a complex combination of different degradation to construct pair training data, including Gaussian blur, Gaussian Noise, Poisson Noise and JPEG compression, and so on. 2) Blind SR models predict the blur kernel of test images and generate HR images simultaneously, we compare with the state-of-the-art DAN [12] and DCLS-SR [18] methods. 3) Zero-shot SR models construct the LR-HR paired images based on the assumption of the cross-scale patch recurrence and train/update their SR model for each test image. These methods include ZSSR [20], KernelGAN [1]+ZSSR [20], MZSR [7], DualSR [8], DDNM [24]. 4) Moreover, we implement a baseline TTA method (dubbed TTA-C) that utilizes the augmentation consistency loss in Eqn. (8) to adapt the pre-trained model, similar to MEMO [27] and CoTTA [21].
We provide more results of our SRTTA for  $2 \times$  and  $4 \times$  SR in terms of PSNR and SSIM metrics. As shown in Table A and Table B, our SRTTA consistently outperforms existing SR methods on the DIV2K-C dataset on average. We further provide more visualization results in Figure D, E, F and G, which demonstrate our SRTTA is able to remove the degradation from test images and generate plausible HR images. These results demonstrate that our SRTTA is able to quickly adapt the pre-trained SR model to the test images with different degradation.
Table A: We report the PSNR/SSIM results of all corruption fields in DIV2K-C for  $2 \times$  SR.
Methods	GaussianBlur	DefocusBlur	GlassBlur	GaussianNoise	PossionNoise	ImpulseNoise	SpeckleNoise	JPEG	Mean	GPU Time
PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	(seconds/image)
Bicubic	28.04/0.803	24.10/0.784	26.31/0.745	25.35/0.554	23.33/0.496	15.28/0.324	28.65/0.774	28.28/0.806	24.92/0.661	-
SwinIR [15]	30.40/0.838	25.52/0.673	27.82/0.773	25.35/0.510	22.36/0.428	15.34/0.242	30.45/0.774	30.74/0.846	26.00/0.636	13.08
IPT [5]	28.93/0.820	24.08/0.640	26.39/0.749	22.96/0.439	20.08/0.369	13.06/0.241	28.27/0.728	28.36/0.804	24.02/0.599	55.36
HAT [6]	29.00/0.821	24.08/0.640	26.40/0.749	22.31/0.417	19.33/0.349	11.91/0.192	28.02/0.722	28.25/0.802	23.66/0.587	25.01
DAN [12]	34.32/0.916	25.58/0.673	31.77/0.872	26.36/0.558	23.28/0.461	11.46/0.203	30.64/0.777	31.08/0.857	26.81/0.665	3.10
DCLS-SR [18]	33.93/0.914	25.55/0.671	31.98/0.872	25.45/0.521	21.59/0.415	8.12/0.112	30.66/0.784	30.86/0.848	26.02/0.642	1.45
ZSSR [20]	29.91/0.831	25.54/0.674	27.79/0.771	26.79/0.590	24.24/0.509	19.14/0.375	30.95/0.813	31.01/0.853	26.92/0.677	117.65
KernalGAN [1]+ZSSR	30.18/0.859	25.87/0.679	29.01/0.808	21.45/0.436	19.32/0.366	17.93/0.354	25.07/0.686	26.11/0.774	24.37/0.620	231.41
MZSR [7]	30.14/0.838	25.54/0.670	28.03/0.777	25.94/0.543	23.48/0.472	17.05/0.314	30.00/0.771	30.49/0.845	26.33/0.654	3.34
DualSR [8]	29.00/0.854	24.40/0.640	28.18/0.805	22.30/0.509	20.11/0.436	17.22/0.376	24.99/0.738	24.74/0.751	23.87/0.639	210.85
DDNM [24]	28.46/0.808	24.09/0.636	26.39/0.744	24.37/0.497	21.92/0.432	13.98/0.310	28.60/0.753	28.26/0.802	24.51/0.623	2,288.55
EDSR [16]	30.28/0.837	25.52/0.673	27.82/0.773	25.87/0.536	22.96/0.449	15.87/0.269	30.52/0.778	30.83/0.847	26.21/0.645	-
TIA-C	30.21/0.835	25.50/0.673	27.79/0.772	26.37/0.559	23.57/0.473	16.40/0.298	30.25/0.783	30.91/0.849	26.38/0.655	13.59
SRTTA (ours)	31.07/0.869	25.86/0.674	29.01/0.815	29.65/0.762	26.69/0.637	16.15/0.284	32.33/0.873	31.30/0.857	27.76/0.721	5.38
SRTTA-lifelong (ours)	31.07/0.869	25.83/0.674	29.18/0.819	29.48/0.797	27.10/0.673	16.27/0.273	31.71/0.864	31.22/0.853	27.73/0.728	5.38
Table B: We report the PSNR/SSIM results of all corruption fields in DIV2K-C for  $4 \times$  SR.
Methods	GaussianBlur	DefocusBlur	GlassBlur	GaussianNoise	PoissonNoise	ImpulseNoise	SpecgleNoise	JPEG	Mean	GPU Time
PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	PSNR/SSIM	(seconds/image)
Bicubic	25.83/0.718	24.10/0.641	25.35/0.699	25.35/0.699	23.17/0.500	22.15/0.475	22.15/0.475	22.15/0.475	15.1/0.384	25.29/0.658	25.07/0.681
Real-ESRNet [22]	26.82/0.765	25.17/0.704	26.75/0.762	25.49/0.701	25.06/0.692	19.24/0.509	19.24/0.509	19.24/0.509	26.47/0.749	25.70/0.720
SwinIR [15]	28.48/0.785	25.81/0.692	27.44/0.753	27.96/0.454	21.40/0.420	14.59/0.225	14.59/0.225	14.59/0.225	26.66/0.670	27.25/0.731
IPT [5]	26.98/0.760	24.36/0.660	25.98/0.726	20.94/0.385	19.30/0.357	12.86/0.270	12.86/0.270	12.86/0.270	24.89/0.628	25.22/0.685
HAT [6]	27.09/0.764	24.37/0.660	26.01/0.728	20.38/0.368	18.76/0.344	11.65/0.177	11.65/0.177	11.65/0.177	24.70/0.623	25.11/0.680
DAN [12]	28.71/0.809	25.02/0.679	28.88/0.812	21.79/0.414	20.26/0.387	8.70/0.110	8.70/0.110	8.70/0.110	25.10/0.631	25.21/0.686
DCLS-SR [18]	30.38/0.834	26.48/0.709	30.58/0.838	24.43/0.525	22.90/0.479	6.94/0.038	6.94/0.038	6.94/0.038	27.27/0.696	27.46/0.736
ZSSR [20]	27.84/0.763	25.83/0.691	27.34/0.745	24.26/0.543	23.04/0.500	17.75/0.402	17.75/0.402	17.75/0.402	26.72/0.700	27.03/0.727
KernalGAN [1]+ZSSR	26.04/0.754	25.84/0.696	26.75/0.755	20.64/0.427	19.63/0.407	16.58/0.361	16.58/0.361	16.58/0.361	22.50/0.578	23.36/0.663
MZSR [7]	25.76/0.722	25.05/0.676	25.77/0.712	22.38/0.471	21.37/0.429	16.46/0.342	16.46/0.342	16.46/0.342	24.20/0.621	25.09/0.695
SRDiff [14]	26.52/0.746	24.18/0.649	25.92/0.723	16.25/0.180	15.50/0.172	12.23/0.172	12.23/0.172	12.23/0.172	19.41/0.356	24.18/0.649
EDSR [16]	28.31/0.780	25.81/0.692	27.40/0.751	23.49/0.479	22.10/0.443	15.28/0.283	15.28/0.283	15.28/0.283	26.80/0.676	27.34/0.734
TTA-C	28.19/0.776	25.76/0.691	27.29/0.747	24.03/0.504	22.71/0.468	16.38/0.357	17.03/0.685	17.03/0.685	27.45/0.736	24.85/0.621
SRTTA (ours)	28.61/0.792	26.24/0.702	28.09/0.775	26.58/0.684	25.27/0.617	15.73/0.318	28.24/0.763	27.66/0.742	25.80/0.674
SRTTA-lifelong (ours)	28.61/0.792	26.25/0.701	28.18/0.776	26.43/0.699	25.56/0.658	15.92/0.312	27.74/0.757	27.61/0.740	25.79/0.679
E.2 More Results on Test domains with Mixed Multiple Degradations.
In this part, we evaluate our SRTTA on DIV2K-MC, which consists of four test domains with mixed multiple degradations. In Table C, our SRTTA achieves the best performance on 4 domains with different mixed degradations, e.g., 0.619 (DualSR)  $\rightarrow$  0.775 (our SRTTA-lifelong) regarding the average SSIM metric. These results further validate the effectiveness of our proposed methods.