SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Datasets. We use DIV2K [1] to train the SR models, which is a high-quality dataset widely used for image SR. The whole dataset includes 800 training images and 100 validation images totally with diverse contents and texture details. The LR images are obtained in the same way as [52, 15]. To demonstrate the effectiveness of our method, the SR models are also evaluated on five public SR benchmark datasets: Set5 [3], Set14 [48], B100 [2], Urban100 [14] and Manga109 [29].
(a) DIV2K training dataset
(b) DIV2K validation dataset
Figure 3: The PSNR of each class on the classified DIV2K training and validation datasets for uniform sampling and inverse sampling.
Uniform SR
Figure 4: Visual comparisons of the SR results by the uniform sampling (uniform SR) and the inverse sampling (inverse SR), and the corresponding error maps with the GT.
Error map
Inverse SR
Implementation Details. During training, we fix the patch size of the HR image as  $128 \times 128$  for  $2 \times$ ,  $4 \times$  SR, and  $129 \times 129$  for  $3 \times$  SR. We follow ClassSR [19] for data preprocessing in inverse sampling, which crops the whole image into multiple sub-images with sliding and overlap. For  $2 \times$ ,  $3 \times$  SR,  $4 \times$  SR, it produces 499875, 502200, 499875 sub-images, respectively. Following [39], the number of the data repetition is set to 1. The training data is augmented by random flipping and rotation. We use Adam optimizer with  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$  to train the SR models. The mini-batch size is set to 16. The learning rate is initialized as  $2e - 4$  and reduced by half per 200 epochs for 400 epochs totally. The unbalanced factor for the inverse sampling data is set to 10 and  $\beta$  is set to 0.1. The interval of alternate training is 50 epochs and the number of classes of inverse sampling for DIV2K training dataset is 10. In our experiments,  $\mu$  is derived to 0.7743 according to the number of classes and the unbalanced factor. All the experiments are conducted with PyTorch framework on NVIDIA 2080Ti GPUs. Besides, objective criteria, i.e., peak signal-to-noise ratio (PSNR), structural similarity (SSIM) are utilized to evaluate the model performance. Note that the two metrics are both calculated on the Y channel of the YCbCr space as adopted in the previous works [6, 52].
In this subsection, we mainly give a detailed analysis about the proposed bi-sampling parameter attribution (BSPA) for accurate image SR. Following [31], EDSR基礎 (16 residual blocks with 64 feature channels) is exploited as the backbone network.
Bi-sampling Strategy. The bi-sampling strategy is adopted to capture the training samples for
the two-step iterative optimization procedure. Uniform sampling ensures the original data distribution, while inverse sampling mainly enhances the feature extraction for the edge and texture regions. Note that all the cropped sub-images are grouped as 10 classes for inverse sampling in our experiments. In Tab. 1, it is observed that the baseline model obtains  $28.22\mathrm{dB}$  in PSNR on Set14, while the model trained with inverse sampling for unbalanced factor 10 achieves 28.25 dB. Besides, we present the average PSNR value of each class on the cropped DIV2K training and validation sub-image dataset for the SR model separately trained by uniform sampling and inverse sampling in Fig. 3. It illustrates that the
Figure 5: The histogram distribution of all training sub-images in DIV2K for EDSR_base and BSPA  $(4\times SR)$ . It is observed that the distribution of BSPA becomes short-tailed.
Table 1: Ablation studies about the bi-sampling parameter attribution on benchmarks for  $4 \times$  SR.
Model	Baseline (uniform sampling)	inverse sampling	w/o IG	FIG	WIG
Set14	28.22	28.25	28.28	28.32	28.37
B100	27.31	27.33	27.35	27.38	27.40
Urban100	25.27	25.34	25.38	25.47	25.53
DIV2K valid	29.97	30.02	30.06	30.09	30.13
Table 2: The quantitative comparisons (PSNR) of different selective ratios on benchmarks for  $4 \times$  SR.
β	0.1	0.5	0.8	1.0
Set14	28.37	28.35	28.34	28.32
B100	27.40	27.37	27.35	27.31
Urban100	25.53	25.51	25.46	25.43
DIV2K valid	30.13	30.10	30.07	30.05
Table 3: The quantitative comparisons (PSNR) of different unbalanced factors on benchmarks for  $4 \times$  SR.
Unbalanced factor	10	50	100	200
Set14	28.37	28.33	28.28	28.21
B100	27.40	27.35	27.31	27.23
Urban100	25.53	25.49	25.42	25.30
DIV2K valid	30.13	30.06	30.00	29.91
SR model trained with inverse sampling performs
much better than the SR model with the uniform sampling on the tail hard classes, while performing worse than the SR model with the uniform sampling on the simple classes. Therefore, inverse sampling helps the feature extraction on hard texture regions. We also visualize the SR results and their error maps with the GT of uniform sampling (uniform SR) and inverse sampling (inverse SR) in Fig. 4. It shows that inverse SR has a smaller error than the uniform SR on the texture region.
Integrated Gradient. To demonstrate the effect of the integrated gradient (IG), we first use uniformly sampled and inversely sampled data to alternately train the SR model. As the model "w/o IG" shows in Tab. 1, the hybrid training obtains  $0.06\mathrm{dB}$  improvement in PSNR against the baseline model on Set14. Besides, we compare the filter-level IG (FIG) and weight-level IG (WIG), which denote the parameter granularity in Eq.10 for each filter and each parameter. It is observed that WIG achieves the best PSNR, which illustrates that the weight is more discriminative than the filter for the model performance. Note that we adopt the weight-level IG in the experiments. Besides, we visualize the distribution curve if the MSE values are evaluated by the model trained with BSPA. As Fig. 5 shows, it is observed that the histogram distribution of BSPA becomes short-tailed. The reason is that the SR performance on the tail data is improved, while the performance on the head data is kept. Therefore, it demonstrates that our BSPA is effective in obtaining a more compact representation.
Effect of Selective Ratio. The selective factor in Eq. (11) aims to control the proportion of the significant parameters and update the remaining trivial parameters. As shown in Tab. 2, we give the quantitative comparisons of different  $\beta$  values for the selective ratio. It can be observed that the best PSNR is obtained when  $\beta$  is 0.1. Therefore, we fix  $\beta$  as 0.1 in our experiments.
Effect of Unbalanced Factor. To demonstrate the effect of the unbalanced factor used for inverse sampling data generation, we give the results of our BSPA with different unbalanced factors as shown in Table 3. It shows that the best PSNR is obtained when the unbalanced factor is equal to 10. Since the unbalanced factor determines the sampling probability of each class in inverse sampling, the large factor causes multiple repetitive sampling to the tail hard samples. It may bring about the overfitting on these areas so as to hinder the model representation.
Effect of the Number of Classes. For the classes, we compare different number of classes in Table 5. It shows that the SR performance changes little to the number of classes. Because the data in different classes can be sampled according to their proportions by means of inverse sampling so as to remedy the unbalanced data bias. In our experiments, we set the number of classes to 10.
4.3 Comparisons with the State-of-the-arts
To demonstrate the effectiveness of our BSPA, we integrate it with three representative SR models with different model capacities, i.e., FSRCNN [10] (small), EDSR基礎 [23] (medium), and RCAN [52] (large). Besides, we compare BSPA with SamplingAug [39] (a sampling-based data augmentation method) and UDN [31] (a loss reweighting method). All these SR models are retrained under the same experimental settings with the released codes.
Quantitative Results. In Tab. 4, it shows that SamplingAug and our BSPA outperform the baseline methods with different model capacities for all scaling factors, which is even higher 1.32dB than the original FSRCNN on DIV2K validation dataset for  $2 \times \mathrm{SR}$ . The results of UDN on some datasets are
even inferior to the baseline model, which may originate from the inaccurate estimation of uncertainty. Note that the model's performance on FSRCNN for  $3 \times$  and  $4 \times$  scale is worse than UDN. FSRCNN is an extremely lightweight model, which has only 24.5K parameters and several convolutional layers. The model representation ability is restricted. Therefore, it may not fit the training well no matter the easy data or the hard data by the limited model capacity, especially for large scaling factors. Although the bi-sampling strategy increases the data diversity, there lacks sufficient model capacity to it. UDN is a loss re-weighting method for solving the data imbalance problem. It provides a regularization constraint for shrinking the solution space. Besides, our method has advantages over other compared methods on large models. For EDSR and RCAN, our BSPA almost performs the best for all scaling factors against the compared methods.
Qualitative Results. The visual comparisons on benchmark datasets for  $4 \times$  SR are shown in Fig. 6. For images "img042" and "img092" in Urban100 dataset, we can observe that BSPA is more favorable and can recover more texture details than other compared methods. Therefore, our proposal can not only achieve excellent performance but also be superior in the visual effect. Besides, we also provide the perceptual metrics (NIQE/LPIPS) comparisons for RCAN in Table 6. It shows that our method obtains superior NIQE and LPIPS values against the compared methods.
Table 5: The quantitative comparisons (PSNR) of different numbers of classes on benchmarks for  $4 \times$  SR.
Classes (K)	5	10	20
Set14	28.36	28.37	28.38
B100	27.41	27.40	27.42
Urban100	25.53	25.53	25.54
DIV2K valid	30.13	30.13	30.14
Table 6: The quantitative comparison results (NIQE/LPIPS) on benchmarks for  $4\times$  SR.
Model	Set14	B100	Urban100	DIV2K valid
RCAN	6.59/0.2932	6.73/0.3868	5.83/0.2508	6.03/0.2836
SamplingAug	6.33/0.2876	6.53/0.3809	5.60/0.2416	5.83/0.2784
UDN	6.61/0.2926	6.69/0.3850	5.76/0.2475	5.85/0.2813
BSPA (ours)	6.26/0.2844	6.46/0.3750	5.61/0.2314	5.79/0.2748
zebra in Set14
HR
Bicubic
EDSR基礎
UDN
SamplingAug
BSPA
baboon in Set14
RCAN
UDN
img042 in Urban100
Figure 6: Visual comparisons of our BSPA with other methods on Set14 and Urban100 for  $4 \times$  SR.
UDN
BSPA
img092 in Urban100
We adopt a re-sampling strategy to address the unbalanced data distribution problem for image SR. It has several limitations. 1) Unlike the end-to-end loss re-weighting methods [31], our method needs to implement extra data pre-processing for classifying sub-images at first. 2) Inverse sampling allocates larger weights for the tail training samples, which may lead to over-fit upon these data. Therefore, it is important to define an appropriate unbalanced factor. Most of the existing SR models only consider that different image regions have different reconstruction difficulties, and then mainly focus on the delicate structure design (like attention mechanism) or complex regularization constraints. Few discuss the sampling strategies of training data and model this as a long-tail problem. The key idea behind the proposed Bi-Sampling Parameter Attribution method is to reconcile the unbalanced inherent data bias, namely the heavy-tailed distribution are visually more important than smooth areas. The newly developed technique with integrate gradient involves an inverse sampling strategy for enhancing the feature extraction ability of the model on the hard samples. We believe that our method has such potential for fully mining the dataset information and may inspire the research of this direction in the future.
In this paper, we rethink the current paradigm of merely using uniform sampling to capture the minibatch data for training the SR model, which would lead to the notorious data unbalanced problem. To address this issue, we propose a bi-sampling parameter attribution (BSPA) method for accurate image SR. The bi-sampling is introduced to remedy the unbalanced data distribution, including the usually adopted uniform sampling and inverse sampling. The former aims to keep the intrinsic data distribution, the latter is designed to promote the model performance on the few hard samples, which is inversely proportional to the number of each class. To achieve this, we divide the whole training dataset into multiple sub-images and group these sub-images into different classes according to the reconstruction error. Moreover, the integrated gradient method is introduced to calculate the parameter importance, with which the significant parameters are fixed and the remaining trivial parameters are further refined. Extensive experiments on SR benchmark datasets have demonstrated that our proposal achieves superior performance against the baseline methods. In our future work, we will apply the proposal to other low-level tasks based on patch training.
Acknowledgments. This work is supported by the National Key Research and Development Program of China No.2020AAA0108301; National Natural Science Foundation of China under Grants No.62176224 and No.62222602; Natural Science Foundation of Chongqing under No.CSTB2023NSCO-JOX0007; CCF-Lenovo Blue Ocean Research Fund.
[1] Eirikur Agustsson and Radu Timofte. NTIRE 2017 challenge on single image super-resolution: Dataset and study. In CVPRW, 2017.
[2] Pablo Arbelaez, Michael Maire, Charless C. Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. TPAMI, 2011.
[3] Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie-Line Alberi-Morel. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In BMVC, 2012.
[4] Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojciech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In ICANN, 2016.
[5] Zheng Chen, Yulun Zhang, Jinjin Gu, Linghe Kong, Xiaokang Yang, and Fisher Yu. Dual aggregation transformer for image super-resolution. In ICCV, 2023.
[6] Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang. Second-order attention network for single image super-resolution. In CVPR, 2019.
[7] André Pierro de Camargo. A new proof of the equivalence of the cauchy mean value theorem and the mean value theorem. Am. Math. Mon., 2020.
[8] Kedar Dhamdhere, Mukund Sundararajan, and Qiqi Yan. How important is a neuron. In ICLR, 2019.
[9] Chao Dong, Chen Change Loy, Kaiming He, and Xiaou Tang. Image super-resolution using deep convolutional networks. TPAMI, 2016.
[10] Chao Dong, Chen Change Loy, and Xiaou Tang. Accelerating the super-resolution convolutional neural network. In ECCV, 2016.
[11] Agrim Gupta, Piotr Dólár, and Ross B. Girshick. LVIS: A dataset for large vocabulary instance segmentation. In CVPR, 2019.
[12] Hui Han, Wenyuan Wang, and Binghuan Mao. Borderline-smote: A new over-sampling method in imbalanced data sets learning. In ICIC, 2005.
[13] Chen Huang, Yining Li, Chen Change Loy, and Xiaou Tang. Deep imbalanced learning for face recognition and attribute prediction. TPAMI, 2020.
[14] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In CVPR, 2015.
[15] Zheng Hui, Xinbo Gao, Yunchu Yang, and Xiumei Wang. Lightweight image super-resolution with information multi-distillation network. In ACM MM, 2019.
[16] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.
[17] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In CVPR, 2016.
[18] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Deeply-recursive convolutional network for image super-resolution. In CVPR, 2016.
[19] Xiangtao Kong, Hengyuan Zhao, Yu Qiao, and Chao Dong. Classssr: A general framework to accelerate super-resolution networks by data characteristic. In CVPR, 2021.
[20] Klas Leino, Shayak Sen, Anupam Datta, Matt Fredrikson, and Linyi Li. Influence-directed explanations for deep convolutional networks. In ITC, 2018.
[21] Feng Li, Running Cong, Huihui Bai, and Yifan He. Deep interleaved network for single image super-resolution with asymmetric co-attention. In IJCAI, 2020.
[42] Liangbin Xie, Xintao Wang, Chao Dong, Zhongang Qi, and Ying Shan. Finding discriminative filters for specific degradations in blind super-resolution. In NeurIPS, 2021.
[43] Wenbin Xie, Dehua Song, Chang Xu, Chunjing Xu, Hui Zhang, and Yunhe Wang. Learning frequency-aware dynamic network for efficient super-resolution. In ICCV, 2021.
[44] Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker. Feature transfer learning for face recognition with under-represented data. In CVPR, 2019.
[45] Ke Yu, Chao Dong, Liang Lin, and Chen Change Loy. Crafting a toolchain for image restoration by deep reinforcement learning. In CVPR, 2018.
[46] Ke Yu, Xintao Wang, Chao Dong, Xiaou Tang, and Chen Change Loy. Path-restore: Learning network path selection for image restoration. TPAMI, 2022.
[47] Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.
[48] Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse-representations. In ICCS, 2010.
[49] Xinyi Zhang, Tao Dai, Bin Chen, and Shu-Tao Xia. DDA: A dynamic difficulty-aware data augmenter for image super-resolution. In IJCNN, 2023.
[50] Yachao Zhang, Yuan Xie, Cuihua Li, Zongze Wu, and Yanyun Qu. Learning all-in collaborative multiview binary representation for clustering. TNNLS, 2022.
[51] Yongshun Zhang, Xiu-Shen Wei, Boyan Zhou, and Jianxin Wu. Bag of tricks for long-tailed visual recognition with deep convolutional neural networks. In AAAI, 2021.
[52] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image superresolution using very deep residual channel attention networks. In ECCV, 2018.
[53] Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. BBN: bilateral-branch network with cumulative learning for long-tailed visual recognition. In CVPR, 2020.