SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiaotong Luo $^{1}$ , Yuan Xie $^{2}$ , Yanyun  $\mathbf{Q}\mathbf{u}^{1*}$
$^{1}$ School of Informatics, Xiamen University, Fujian, China
$^{2}$ School of Computer Science and Technology, East China Normal University, Shanghai, China
$^{2}$ Chongqing Institute of East China Normal University, Chongqing, China
xiaotluo@stu.xmu.edu.cn, yxie@cs.ecnu.edu.cn, yyqu@xmu.edu.cn
Single image super-resolution (SISR) has made a significant breakthrough benefiting from the prevalent rise of deep neural networks and large-scale training samples. The mainstream deep SR models primarily focus on network architecture design as well as optimization schemes, while few pay attention to the training data. In fact, most of the existing SR methods train the model on uniformly sampled patch pairs from the whole image. However, the uneven image content makes the training data present an unbalanced distribution, i.e., the easily reconstructed region (smooth) occupies the majority of the data, while the hard reconstructed region (edge or texture) has rarely few samples. Based on this phenomenon, we consider rethinking the current paradigm of merely using uniform data sampling way for training SR models. In this paper, we propose a simple yet effective Bi-Sampling Parameter Attribution (BSPA) method for accurate image SR. Specifically, the bi-sampling consists of uniform sampling and inverse sampling, which is introduced to reconcile the unbalanced inherent data bias. The former aims to keep the intrinsic data distribution, and the latter is designed to enhance the feature extraction ability of the model on the hard samples. Moreover, integrated gradient is introduced to attribute the contribution of each parameter in the alternate models trained by both sampling data so as to filter the trivial parameters for further dynamic refinement. By progressively decoupling the allocation of parameters, the SR model can learn a more compact representation. Extensive experiments on publicly available datasets demonstrate that our proposal can effectively boost the performance of baseline methods from the data re-sampling view.
Single image super-resolution (SISR) aims to reconstruct a high-resolution (HR) image from its downsampled low-resolution (LR) counterpart. As a classical ill-posed problem, multiple HR images can be recovered from the same LR image. With the boom of neural networks, numerous excellent deep SR methods [9, 23, 25, 5] have been proposed to chase for a more accurate solution. It is well-known that the success of deep learning primarily attributes to three significant factors, including large-scale data [1], network structures [10, 52, 23], and optimization strategies [16, 31, 26]. Most of the existing SR models mainly focus on the delicate structure design or complex regularization constraints, while neglecting the in-depth analysis of the intrinsic training data.
Due to resource constraints, the mainstream deep SR methods mostly train the models with uniformly sampled LR-HR patch pairs rather than the whole images. However, they ignore the
underlying fact that the content of patches is usually unevenly distributed in an image. As presented in Fig. 1, we measure the patch content by mean square error (MSE) and count it for all the cropped sub-images on the commonly used DIV2K training dataset. It reveals a phenomenon of long-tailed distribution, i.e., the flat region occupies the majority of the training samples, while the sharp region with abundant texture details only holds a very small percentage.
The imbalanced training data would undoubtedly impair the reconstruction accuracy, especially for the hard areas in the tail. Therefore, we consider enhancing the model representation from the data sampling perspective.
In this paper, we propose a simple yet effective Bi-Sampling Parameter Attribution (BSPA) method for image SR, which aims to obtain the compact feature representation by explicitly increasing the sampling proportion of the tail hard patch pairs.
Specifically, the bi-sampling consists of uniform sampling and inverse sampling to make up for the unbalanced training data bias. As the most common way of data sampling, uniform sampling helps to keep the original data distributions, while inverse sampling aims to allocate a higher probability to the fewer patches sampled from the tail data. The SR model is alternately trained with the two kinds of sampling data. However, the SR models treat all the parameters equally and perform gradient updating to each parameter, which ignores the parameter redundancy. Therefore, it is necessary to obtain a compact model according to the parameter contribution to the output. Con
sidering that the weight values would not change drastically with the small learning rate, it is difficult to determine the impact of different weights. Inspired by [35], we introduce the integrated gradient (IG) attribution method to measure the parameter importance. The IG is calculated between the previous model trained with uniform sampling data and the next model trained with inverse sampling data. Based on the importance ranking, the significant parameters do not perform gradient updating, and the remaining trivial parameters are encouraged to contribute more by further optimization. Extensive experiments have demonstrated that our proposal can improve the SR performance of baseline methods. To the best of our knowledge, this is the first attempt to investigate image SR as a long-tail data distribution problem.
In summary, the main contributions of this paper are: 1) We definitely find the data unbalanced problem in the training samples caused by uniform sampling for image SR. 2) A bi-sampling paradigm is designed to remedy the unbalanced data bias, including uniform sampling and inverse sampling, respectively. 3) To obtain a compact representation, an integrated gradient based attribution method is introduced for measuring the parameter importance so that the significant parameters are progressively selected to stop gradient updating for better distilling the remaining trivial parameters.
Figure 1: The illustration of long-tailed data distribution with uniform sampling in image SR. Following [19], we first crop the commonly used training dataset, e.g., DIV2K [1], into multiple sub-images, and adopt the mean square error (MSE) between the SR output by MSRResNet [40] and the HR image to measure the reconstruction difficulty of an LR sub-image. It shows that the data distribution of the training samples is uneven, which presents a long-tailed distribution.
2.1 Deep Image SR Models
Nowadays, numerous SR models have been proposed with remarkable performance. The pioneering work, SRCNN [9], proposed a three-layer network for image SR. Then, FSRCNN [10] and ESPCN [32] introduced deconvolution and sub-pixel upsampling for mapping LR features to HR images, which could largely reduce computation complexity. VDSR [17] introduced residual learning to extract missing high-frequency information. DRRN [18] and DRCN [18] adopted recursive neural networks to reduce the model parameters while achieving excellent performance. RCAN [52] adopted a residual-in-residual network design and embedded the channel attention in each residual block to build an extremely deep network for image SR. DIN [21] designed a deep interleaved network to
combine the information at different states. Wang et. al. [37] learned a scale-arbitrary image SR network from scale-specific networks. LatticeNet [27] proposed a lattice block to adaptively combine pair-wise residual blocks. All these SR models have been trained on uniformly sampled LR-HR patch pairs, which ignore the unbalanced data distribution in the training samples.
Recently, several region-aware SR methods have been proposed to distinguish different image regions in an image and aim to assign more computation resources on more challenging pixels adaptively. RL-Restore [45] and Path-Restore [46] decomposed the image into different sub-images and then estimated an appropriate processing path for each sub-image by reinforcement learning. ClassSR [19] proposed a new classification method to determine the processing of different image regions and then reconstructed the sub-image by models of different sizes. Similar to ClassSR, FADN [43] proposed a framework for processing the different image regions divided by the corresponding information in the frequency domain with different computation burdens. APE [38] introduced an adaptive patch exiting strategy to super-resolve patches with different difficulties for achieving a flexible tradeoff between performance and efficiency. The above content-related SR methods mainly focus on the architecture design or optimization schemes. Except for the sub-image decomposition methods, UDN [31] proposed a new adaptive weighted loss for image SR to train deep networks focusing on challenging situations such as textured and edge pixels with high uncertainty.
In this paper, we mainly rethink the unbalanced distribution for image SR from the view of data sampling. SamplingAug [39] is a sampling-based data augmentation method for image SR. It only sampled the  $p\%$  most informative patches for model training, while the rest is ignored. DDA [49] designed a difficulty-aware data sampling strategy, which controls the sampling probability of each class by the relative loss. Different from these works, we propose a bi-sampling strategy and the model is alternatively updated with uniformly and inversely sampled image data. All patches are effectively utilized for SR training to help the model fully mine the image information.
2.2 Long-tail Image Classification
Due to the imbalanced data distribution caused by the difficult data acquisition in actual applications, long-tailed image recognition has attracted widespread attention [41, 11, 50]. The existing long-tail studies can be roughly divided into three types: data re-sampling, loss re-weighting, and transfer learning. The data re-sampling based methods [28, 12] aim to re-sample the training dataset for achieving a more balanced distribution. Besides, the loss re-weighting based methods [13, 33] are proposed to allocate different weights to each training samples for loss optimization. Recently, the transfer learning based methods [44, 24] have sprung up to transfer the informative features learned from the head classes with abundant training samples to the under-represented tail classes. Here, we mainly explore the data re-sampling method for image SR.
As the widely adopted data re-balancing strategy for solving the long-tailed problem, re-sampling methods aim to sample the data to generate an evenly distributed dataset, which can be classified as: over-sampling for minority classes and under-sampling for dominant classes. However, repeated sampling for tailed samples may lead to the over-fitting problem on minority classes, while discarding the head samples will undoubtedly damage the generalization ability of deep models. Therefore, we integrate the two data sampling ways together for remedying the dataset bias in image SR task.
2.3 Attribution Analysis
Attribution analysis is usually utilized for model interpretation on the high-level image classification task, which aims to attribute the influence of each pixel in the input on prediction accuracy. The existing attribution analysis methods can be classified as: the gradient-based methods [34, 35], propagation-based methods [30, 4] and occlusion-based methods [47, 22]. Here, we primarily focus on the gradient-based methods, which utilize the gradient of the output against the input as a metric and accumulate the integrated gradients (IG) at all points along a path from the baseline input to the target input. IntInf [20] and neuron conductance [8] further extend feature-important IG to neurons. Unlike the classical attribution methods [35, 36], which are used to attribute the influence of the input on the classification model by attributing the contribution of each pixel of the input image to the final output label, FAIG [42] first introduced IG to attribute network functions to the filters in the blind SR model and successfully find discriminative filters that closely relate to the image degradation removal. In this paper, we mainly adopt the integrated gradient method to help locate the position of important weights in SR models trained with uniform sampling data and inverse sampling data.
Figure 2: The framework of our proposed bi-sampling parameter attribution method for image SR.
3.1 Method Overview
Considering the intrinsic dataset-specific bias shown in Fig. 1, we adopt two kinds of sampling ways (bi-sampling) for capturing the training data: uniform sampling and inverse sampling, which are combined to re-balance the distribution of training data. As shown in Fig 2, the proposed bi-sampling parameter attribution (BSPA) framework includes two steps, which are alternately performed. In Step 1, the SR model is trained with the data from uniform sampling to obtain a trivial solution toward normal data distribution (baseline model). In Step 2, the SR model is fine-tuned with the data from inverse sampling for generating a biased solution towards processing well on the tail edge and texture regions (target model). After training with inverse sampling data, the integrated gradient is calculated between the baseline model and the target model, which is utilized to obtain the parameter importance so as to control the relevant weight updating.
3.2 Bi-sampling Strategy
Uniform Sampling. Uniform sampling refers to cropping the patch samples from the whole image in the training dataset randomly, which is the most common way of data sampling and widely used in image SR task [21]. To be specific, the input data comes from a uniform sampler, where each sample in the training dataset is sampled with equal probability in a training epoch. The probability  $P_{i}^{us}$  that a sample  $i$  is chosen from the whole training dataset can be formulated as:
$$
P _ {i} ^ {u s} = \frac {1}{N _ {T}}, \tag {1}
$$
where  $N_{T}$  is the total number of training samples.
Inverse Sampling. Inverse sampling aims to allocate a higher probability to capture the tail hard data with few samples. Unlike long-tailed image classification, which has a definite number of classes, image SR is a regression task. We adopt MSE to measure the reconstruction difficulties, which is a continuous value and each value corresponds to a sample as shown in Fig. 1. Considering the MSE value is continuous and divergently distributed, it needs to collect more samples within a certain range to calculate the proportion. Therefore, we first split data patches into multiple groups to better calculate the sampling probability.
Data classification. Specifically, the number of classes is predefined as  $K$ . Analogy to the long-tailed CIFAR, which is created by reducing the number of training samples per class via an exponential function [51], then the number of samples for each class can be calculated as:
$$
N _ {k} = N \times \mu^ {k}, \tag {2}
$$
where  $k \in (1, K)$  is the class index,  $N$  is the total number of training sub-images, and  $\mu \in (0, 1)$  is an attenuation factor. Therefore, a long-tailed version of DIV2K training dataset can be created by setting different MSE threshold values.
Moreover, the imbalance factor of a long-tailed dataset is defined as the number of training samples in the largest class  $(N_{1})$  divided by that of the smallest  $(N_K)$ , which usually ranges from 10 to 200 in long-tail research and is set as 10 in our experimental setting. Finally, the MSE threshold values used for classifying the sub-images can be obtained by:
$$
t _ {k} = t \left[ N _ {1: k} \right], \tag {3}
$$
where  $t_k$  is the MSE threshold sorted by ascending.  $N_{1:k}$  refers to the total number of sample from  $N_1$  to  $N_k$ . Therefore, we can obtain the long-tailed version of DIV2K training dataset. It is noted that the test dataset remains unchanged and the SR model is still tested on the whole image.
Sampling procedure. Following [53], the sampling possibility of each class for inverse sampling  $P_{k}^{is}$  is inversely proportional to the class sample capacity, i.e., the more samples in a class, the small sampling possibility that class has. Therefore, the procedures of inverse sampling are as follows: 1)  $P_{k}^{is}$  for the class  $k$  can be calculated according to the number of samples as:
$$
P _ {k} ^ {i s} = \frac {\frac {1}{N _ {k}}}{\sum_ {j = 1} ^ {K} \frac {1}{N _ {j}}}. \tag {4}
$$
2) sample a class  $k$  according to  $P_{k}^{dis}$  randomly; 3) select a sample from class  $k$  randomly. The mini-batch inverse sampling data can be obtained by repeating the above operations.
3.3 Integrated Gradient for Parameter Attribution
Parameter Importance Formulation. The uniform sampling data is adopted to maintain the inherent data distribution. The inverse sampling data is used to train the SR model for providing an elegant solution on edge and texture areas. In this paper, we adopt a hybrid training mechanism with uniform and inverse sampling data, which are iteratively input into the SR model.
Since the SR model is trained on uniform sampling and inverse sampling iteratively, there exist different preferences in the SR reconstruction regions. Then, it is meaningful to calculate the parameter importance for obtaining a compact model. Assume the parameter sets of the SR model  $F(\cdot)$  trained with uniform sampling data and inverse sampling data are denoted as  $\{\theta_i^{us} : 1 \leq i \leq L\}$  and  $\{\theta_i^{is} : 1 \leq i \leq L\}$ , where  $\theta_i^{us}, \theta_i^{is} \in \mathbb{R}^{c_o \times c_i \times w \times h}$  are the weight matrices at the  $i$ -th layer,  $L$  is the number of convolutional layers,  $c_o$  and  $c_i$  are the number of output and input channels,  $w$  and  $h$  are the kernel sizes. The two sampling LR images are denoted as  $X^{us}$ ,  $X^{is}$ , and the corresponding HR images are  $Y^{us}$ ,  $Y^{is}$ , respectively. Then, for a uniformly sampled patch pair  $(x^{us}, y^{us})$ , the variation of network performance on the baseline model and the target model can be measured by:
$$
\Delta F \left(\theta^ {i s}, \theta^ {u s}; x ^ {u s}, y ^ {u s}\right) = \left| l \left(F \left(x ^ {u s}; \theta^ {i s}\right), y ^ {u s}\right) - l \left(F \left(x ^ {u s}; \theta^ {u s}\right), y ^ {u s}\right) \right|, \tag {5}
$$
where  $l$  means the loss function used for training the SR network. For simplicity, we denote  $l(F(x;\theta),y)$  as  $l(\theta ;x,y)$ .
Take the element of parameter  $\theta_{ik}^{us}$  (i-th layer) for example, the change of parameter  $\theta_{ik}^{us}$  when the model is trained on the inverse sampling data is represented as  $\delta \theta_{ik}^{us}$ , i.e.,
$$
\delta \theta_ {i k} ^ {u s} = \theta_ {i k} ^ {i s} - \theta_ {i k} ^ {u s}, \tag {6}
$$
To further measure  $\Delta F(\theta^{is},\theta^{us};x^{us},y^{us})$ , Cauchy mean value theorem [7] demonstrates that when a function  $f$  is differentiable at point  $x_0$ , it always stands up in the neighborhood  $U(\cdot)$  of the point  $x_0$ :
$$
f (x) = f \left(x _ {0}\right) + f ^ {\prime} \left(x _ {0}\right) \left(x - x _ {0}\right) + o \left(x - x _ {0}\right), \tag {7}
$$
Since  $o(x - x_0)$  is an infinitesimal, it can be derived that  $f(x) \approx f(x_0) + f'(x_0)(x - x_0)$ . This is one of the formulas that is often used in local linearization of a function. Geometrically, it adopts the tangent line to approximate the curve in a small neighborhood.
Then, Eq. (5) can be further formulated as:
$$
l \left(\theta_ {i k} ^ {u s} + \delta \theta_ {i k} ^ {u s}; x ^ {u s}, y ^ {u s}\right) - l \left(\theta_ {i k} ^ {u s}; x ^ {u s}, y ^ {u s}\right) \approx \left(\nabla_ {\theta_ {i k} ^ {u s}} l\right) ^ {T} \cdot \delta \theta_ {i k} ^ {u s}. \tag {8}
$$
Integrated Gradient. The above formula means that we can calculate the parameter importance by the value change and its gradient, which both consider the influence of network alteration on the model outputs. Here, we apply the integrated gradients (IG) to calculate the parameter importance, which was originally an attribution method to attribute the prediction of a deep network to its input features by accumulating the gradient of input changes. Inspired by FAIG [42], which utilizes paths in the parameter space for attributing network functional alterations to filter changes, we adopt IG to
Algorithm 1 The bi-sampling parameter attribution for compact image SR.
Input: the SR model  $F(\cdot)$  with the randomly initialized weights:  $\theta_0$ ; the uniform sampling and inverse sampling datasets:  $D^{us}$ ,  $D^{is}$ ; the total epochs:  $T$ ; the interval for training  $F(\cdot)$  with inverse sampling:  $E$ ; the remaining ratio for parameters without gradient updating:  $p$ .
Output: the compact SR model  $F(\cdot)$  with parameter  $\theta$ .
1: while epoch  $\leq T$  do
2: Sample the LR-HR patch pairs  $(x_{lr}^{us}, y_{hr}^{us})$  from  $D^{us}$ ;
3: Update  $F(\cdot)$  with  $\theta^{us}$  on the uniformly sampled data;
4: if epoch  $\% E = 0$  then
5: Sample the LR-HR patch pairs  $(x_{lr}^{is},y_{hr}^{is})$  from  $D^{is}$
6: Update  $F(\cdot)$  with  $\theta^{is}$  on the inversely sampled data;
7: Calculate the integrated gradient between the baseline model  $(\theta^{us})$  and the target model  $(\theta^{is})$  as Eq. (10) to obtain the parameter importance;
8: Keep  $p$  significant parameters stop gradient updating as Eq.(11) after sorting the importance;
9: end if
0: end while
obtain the parameter importance so as to help us discover the location of significant parameters in the sampling iteration.
The integrated path is calculated from the model trained with uniform sampling data to the model trained with inverse sampling data. Let  $\gamma (\alpha),\alpha \in [0,1]$  denote a continuous path between the baseline model  $(\theta^{\prime})$  and the target model  $(\theta)$  with  $\gamma (0) = \theta^{\prime},\gamma (1) = \theta$ . Then, the integrated path can be represented as:
$$
\gamma (\alpha) = \theta^ {\prime} + \alpha \left(\theta - \theta^ {\prime}\right). \tag {9}
$$
In addition, the  $L_{1}$  loss is utilized to train the SR model, while the gradient of each parameter can be obtained according to the back-propagation algorithm. After defining the baseline and target model, integrated path and the loss function, the changes of network functions to each parameter with integrated gradient for an input image  $x$  can be attributed as follows:
$$
\begin{array}{l} I G (\theta , x) = l (\gamma (1), x) - l (\gamma (0), x) \\ = \sum_ {i} \int_ {\alpha = 0} ^ {1} \frac {\partial l (\gamma (\alpha) , x)}{\partial \gamma (\alpha) _ {i}} \times \frac {\partial \gamma (\alpha) _ {i}}{\partial \alpha} d \alpha \tag {10} \\ \approx \frac {1}{M} [ \theta - \theta^ {\prime} ] \sum_ {s = 0} ^ {M - 1} \left[ \frac {\partial l (\gamma (\alpha) , x)}{\partial \gamma (\alpha)} | _ {\gamma (\alpha) = \theta^ {\prime} + \alpha (\theta - \theta^ {\prime}), \alpha = \frac {s}{M}} \right]. \\ \end{array}
$$
Parameter Refinement. After calculating the integrated gradient, the position index of parameter importance can be located. The alternate training scheme is adopted for the bi-sampling data. To obtain a more compact representation, we expect the trivial parameters can also learn more effective information so we strive to distill them with other parameters fixed. Therefore, a specific proportion of significant parameters can be selected to keep unchanged and only perform the gradient updating on the remaining trivial parameters for further refinement. Especially, the selective ratio is set as:
$$
p = \beta * (e / T), \tag {11}
$$
where  $e$  is the  $e$ -th training epoch, and  $T$  is the total training epochs.  $\beta$  is a scaling factor. The detailed training procedure of our proposal is given in Algorithm 1.