Title: Learning Re-sampling Methods with Parameter Attribution for Image Super-resolution
Venue: NeurIPS 2023
Accepted: yes
Keywords: image super-resolution

long-tail distribution

re-sampling

integrated gradient
Abstract: Single image super-resolution (SISR) has made a significant breakthrough benefiting from the prevalent rise of deep neural networks and large-scale training samples. The mainstream deep SR models primarily focus on network architecture design as well as optimization schemes, while few pay attention to the training data. In fact, most of the existing SR methods train the model on uniformly sampled patch pairs from the whole image. However, the uneven image content makes the training data present an unbalanced distribution, i.e., the easily reconstructed region (smooth) occupies the majority of the data, while the hard reconstructed region (edge or texture) has rarely few samples. Based on this phenomenon, we consider rethinking the current paradigm of merely using uniform data sampling way for training SR models. In this paper, we propose a simple yet effective Bi-Sampling Parameter Attribution (BSPA) method for accurate image SR. Specifically, the bi-sampling consists of uniform sampling and inverse sampling, which is introduced to reconcile the unbalanced inherent data bias. The former aims to keep the intrinsic data distribution, and the latter is designed to enhance the feature extraction ability of the model on the hard samples. Moreover, integrated gradient is introduced to attribute the contribution of each parameter in the alternate models trained by both sampling data so as to filter the trivial parameters for further dynamic refinement. By progressively decoupling the allocation of parameters, the SR model can learn a more compact representation. Extensive experiments on publicly available datasets demonstrate that our proposal can effectively boost the performance of baseline methods from the data re-sampling view.

Learning Re-sampling Methods with Parameter Attribution for Image Super-resolution

Abstract

Single image super-resolution (SISR) has made a significant breakthrough benefiting from the prevalent rise of deep neural networks and large-scale training samples. The mainstream deep SR models primarily focus on network architecture design as well as optimization schemes, while few pay attention to the training data. In fact, most of the existing SR methods train the model on uniformly sampled patch pairs from the whole image. However, the uneven image content makes the training data present an unbalanced distribution, i.e., the easily reconstructed region (smooth) occupies the majority of the data, while the hard reconstructed region (edge or texture) has rarely few samples. Based on this phenomenon, we consider rethinking the current paradigm of merely using uniform data sampling way for training SR models. In this paper, we propose a simple yet effective Bi-Sampling Parameter Attribution (BSPA) method for accurate image SR. Specifically, the bi-sampling consists of uniform sampling and inverse sampling, which is introduced to reconcile the unbalanced inherent data bias. The former aims to keep the intrinsic data distribution, and the latter is designed to enhance the feature extraction ability of the model on the hard samples. Moreover, integrated gradient is introduced to attribute the contribution of each parameter in the alternate models trained by both sampling data so as to filter the trivial parameters for further dynamic refinement. By progressively decoupling the allocation of parameters, the SR model can learn a more compact representation. Extensive experiments on publicly available datasets demonstrate that our proposal can effectively boost the performance of baseline methods from the data re-sampling view.

1 Introduction

Single image super-resolution (SISR) aims to reconstruct a high-resolution (HR) image from its downsampled low-resolution (LR) counterpart. As a classical ill-posed problem, multiple HR images can be recovered from the same LR image. With the boom of neural networks, numerous excellent deep SR methods [9, 23, 25, 5] have been proposed to chase for a more accurate solution. It is well-known that the success of deep learning primarily attributes to three significant factors, including large-scale data [1], network structures [10, 52, 23], and optimization strategies [16, 31, 26]. Most of the existing SR models mainly focus on the delicate structure design or complex regularization constraints, while neglecting the in-depth analysis of the intrinsic training data.

Due to resource constraints, the mainstream deep SR methods mostly train the models with uniformly sampled LR-HR patch pairs rather than the whole images. However, they ignore the

1

underlying fact that the content of patches is usually unevenly distributed in an image. As presented in Fig. 1, we measure the patch content by mean square error (MSE) and count it for all the cropped sub-images on the commonly used DIV2K training dataset. It reveals a phenomenon of long-tailed distribution, i.e., the flat region occupies the majority of the training samples, while the sharp region with abundant texture details only holds a very small percentage.

The imbalanced training data would undoubtedly impair the reconstruction accuracy, especially for the hard areas in the tail. Therefore, we consider enhancing the model representation from the data sampling perspective.

In this paper, we propose a simple yet effective BiSampling Parameter Attribution (BSPA) method for image SR, which aims to obtain the compact feature representation by explicitly increasing the sampling proportion of the tail hard patch pairs.

Specifically, the bi-sampling consists of uniform sampling and inverse sampling to make up for the unbalanced training data bias. As the most common way of data sampling, uniform sampling helps to keep the original data distributions, while inverse sampling aims to allocate a higher probability to the fewer patches sampled from the tail data. The SR model is alternately trained with the two kinds of sampling data. However, the SR models treat all the parameters equally and perform gradient updating to each parameter, which ignores the parameter redundancy. Therefore, it is necessary to obtain a compact model according to the parameter contribution to the output. Con-

Number of patches

Head

Tail

Mse value

sidering that the weight values would not change drastically with the small learning rate, it is difficult to determine the impact of different weights. Inspired by [35], we introduce the integrated gradient (IG) attribution method to measure the parameter importance. The IG is calculated between the previous model trained with uniform sampling data and the next model trained with inverse sampling data. Based on the importance ranking, the significant parameters do not perform gradient updating, and the remaining trivial parameters are encouraged to contribute more by further optimization. Extensive experiments have demonstrated that our proposal can improve the SR performance of baseline methods. To the best of our knowledge, this is the first attempt to investigate image SR as a long-tail data distribution problem.

In summary, the main contributions of this paper are: 1) We definitely find the data unbalanced problem in the training samples caused by uniform sampling for image SR. 2) Abi-sampling paradigm is designed to remedy the unbalanced data bias, including uniform sampling and inverse sampling, respectively. 3) To obtain a compact representation, an integrated gradient based attribution method is introduced for measuring the parameter importance so that the significant parameters are progressively selected to stop gradient updating for better distilling the remaining trivial parameters.

2 Related Work

2.1 Deep Image SR Models

Nowadays, numerous SR models have been proposed with remarkable performance. The pioneering work, SRCNN [9], proposed a three-layer network for image SR. Then, FSRCNN [10] and ESPCN [32] introduced deconvolution and sub-pixel upsampling for mapping LR features to HR images, which could largely reduce computation complexity. VDSR [17] introduced residual learning to extract missing high-frequency information. DRRN [18] and DRCN [18] adopted recursive neural networks to reduce the model parameters while achieving excellent performance. RCAN [52] adopted a residual-in-residual network design and embedded the channel attention in each residual block to build an extremely deep network for image SR. DIN [21] designed a deep interleaved network to

combine the information at different states. Wang et. al. [37] learned a scale-arbitrary image SR network from scale-specific networks. LatticeNet [27] proposed a lattice block to adaptively combine pair-wise residual blocks. All these SR models have been trained on uniformly sampled LR-HR patch pairs, which ignore the unbalanced data distribution in the training samples.

Recently, several region-aware SR methods have been proposed to distinguish different image regions in an image and aim to assign more computation resources on more challenging pixels adaptively. RL-Restore [45] and Path-Restore [46] decomposed the image into different sub-images and then estimated an appropriate processing path for each sub-image by reinforcement learning. ClassSR [19] proposed a new classification method to determine the processing of different image regions and then reconstructed the sub-image by models of different sizes. Similar to ClassSR, FADN [43] proposed a framework for processing the different image regions divided by the corresponding information in the frequency domain with different computation burdens. APE [38] introduced an adaptive patch exiting strategy to super-resolve patches with different difficulties for achieving a flexible tradeoff between performance and efficiency. The above content-related SR methods mainly focus on the architecture design or optimization schemes. Except for the sub-image decomposition methods, UDN [31] proposed a new adaptive weighted loss for image SR to train deep networks focusing on challenging situations such as textured and edge pixels with high uncertainty.

In this paper, we mainly rethink the unbalanced distribution for image SR from the view of data sampling. SamplingAug [39] is a sampling-based data augmentation method for image SR. It only sampled the p % most informative patches for model training, while the rest is ignored. DDA [49] designed a difficulty-aware data sampling strategy, which controls the sampling probability of each class by the relative loss. Different from these works, we propose a bi-sampling strategy and the model is alternatively updated with uniformly and inversely sampled image data. All patches are effectively utilized for SR training to help the model fully mine the image information.

2.2 Long-tail Image Classification

Due to the imbalanced data distribution caused by the difficult data acquisition in actual applications, long-tailed image recognition has attracted widespread attention [41, 11, 50]. The existing long-tail studies can be roughly divided into three types: data re-sampling, loss re-weighting, and transfer learning. The data re-sampling based methods [28, 12] aim to re-sample the training dataset for achieving a more balanced distribution. Besides, the loss re-weighting based methods [13, 33] are proposed to allocate different weights to each training samples for loss optimization. Recently, the transfer learning based methods [44, 24] have sprung up to transfer the informative features learned from the head classes with abundant training samples to the under-represented tail classes. Here, we mainly explore the data re-sampling method for image SR.

As the widely adopted data re-balancing strategy for solving the long-tailed problem, re-sampling methods aim to sample the data to generate an evenly distributed dataset, which can be classified as: over-sampling for minority classes and under-sampling for dominant classes. However, repeated sampling for tailed samples may lead to the over-fitting problem on minority classes, while discarding the head samples will undoubtedly damage the generalization ability of deep models. Therefore, we integrate the two data sampling ways together for remedying the dataset bias in image SR task.

2.3 Attribution Analysis

Attribution analysis is usually utilized for model interpretation on the high-level image classification task, which aims to attribute the influence of each pixel in the input on prediction accuracy. The existing attribution analysis methods can be classified as: the gradient-based methods [34, 35], propagation-based methods [30, 4] and occlusion-based methods [47, 22]. Here, we primarily focus on the gradient-based methods, which utilize the gradient of the output against the input as a metric and accumulate the integrated gradients (IG) at all points along a path from the baseline input to the target input. IntInf [20] and neuron conductance [8] further extend feature-important IG to neurons. Unlike the classical attribution methods [35, 36], which are used to attribute the influence of the input on the classification model by attributing the contribution of each pixel of the input image to the final output label, FAIG [42] first introduced IG to attribute network functions to the filters in the blind SR model and successfully find discriminative filters that closely relate to the image degradation removal. In this paper, we mainly adopt the integrated gradient method to help locate the position of important weights in SR models trained with uniform sampling data and inverse sampling data.

Uniform sampling

Inverse sampling

Target model

Baseline model

Importance

Sort

Integrated

path

Integrated gradient

0

1

Update

Parameter

Significant

parameter

Trivial

parameter

Input

Input

3 Proposed Method

3.1 Method Overview

Considering the intrinsic dataset-specific bias shown in Fig. 1, we adopt two kinds of sampling ways (bi-sampling) for capturing the training data: uniform sampling and inverse sampling, which are combined to re-balance the distribution of training data. As shown in Fig 2, the proposed bi-sampling parameter attribution (BSPA) framework includes two steps, which are alternately performed. In Step 1, the SR model is trained with the data from uniform sampling to obtain a trivial solution toward normal data distribution (baseline model). In Step 2, the SR model is fine-tuned with the data from inverse sampling for generating a biased solution towards processing well on the tail edge and texture regions (target model). After training with inverse sampling data, the integrated gradient is calculated between the baseline model and the target model, which is utilized to obtain the parameter importance so as to control the relevant weight updating.

3.2 Bi-sampling Strategy

Uniform Sampling. Uniform sampling refers to cropping the patch samples from the whole image in the training dataset randomly, which is the most common way of data sampling and widely used in image SR task [21]. To be specific, the input data comes from a uniform sampler, where each sample in the training dataset is sampled with equal probability in a training epoch. The probability P us i that a sample i is chosen from the whole training dataset can be formulated as:

where N T is the total number of training samples.

Inverse Sampling. Inverse sampling aims to allocate a higher probability to capture the tail hard data with few samples. Unlike long-tailed image classification, which has a definite number of classes, image SR is a regression task. We adopt MSE to measure the reconstruction difficulties, which is a continuous value and each value corresponds to a sample as shown in Fig. 1. Considering the MSE value is continuous and divergently distributed, it needs to collect more samples within a certain range to calculate the proportion. Therefore, we first split data patches into multiple groups to better calculate the sampling probability.

Data classification. Specifically, the number of classes is predefined as K . Analogy to the long-tailed CIFAR, which is created by reducing the number of training samples per class via an exponential function [51], then the number of samples for each class can be calculated as:

where k ∈ (1 , K ) is the class index, N is the total number of training sub-images, and µ ∈ (0 , 1) is an attenuation factor. Therefore, a long-tailed version of DIV2K training dataset can be created by setting different MSE threshold values.

Moreover, the imbalance factor of a long-tailed dataset is defined as the number of training samples in the largest class ( N 1 ) divided by that of the smallest ( N K ), which usually ranges from 10 to 200 in long-tail research and is set as 10 in our experimental setting. Finally, the MSE threshold values used for classifying the sub-images can be obtained by:

where t k is the MSE threshold sorted by ascending. N 1: k refers to the total number of sample from N 1 to N k . Therefore, we can obtain the long-tailed version of DIV2K training dataset. It is noted that the test dataset remains unchanged and the SR model is still tested on the whole image.

Sampling procedure. Following [53], the sampling possibility of each class for inverse sampling P is k is inversely proportional to the class sample capacity, i.e., the more samples in a class, the small sampling possibility that class has. Therefore, the procedures of inverse sampling are as follows: 1) P is k for the class k can be calculated according to the number of samples as:

2) sample a class k according to P is k randomly; 3) select a sample from class k randomly. The mini-batch inverse sampling data can be obtained by repeating the above operations.

3.3 Integrated Gradient for Parameter Attribution

Parameter Importance Formulation. The uniform sampling data is adopted to maintain the inherent data distribution. The inverse sampling data is used to train the SR model for providing an elegant solution on edge and texture areas. In this paper, we adopt a hybrid training mechanism with uniform and inverse sampling data, which are iteratively input into the SR model.

Since the SR model is trained on uniform sampling and inverse sampling iteratively, there exist different preferences in the SR reconstruction regions. Then, it is meaningful to calculate the parameter importance for obtaining a compact model. Assume the parameter sets of the SR model F ( · ) trained with uniform sampling data and inverse sampling data are denoted as { θ us i : 1 ≤ i ≤ L } and { θ is i : 1 ≤ i ≤ L } , where θ us i , θ is i ∈ R c o × c i × w × h are the weight matrixs at the i -th layer, L is the number of convolutional layers, c o and c i are the number of output and input channels, w and h are the kernel sizes. The two sampling LR images are denoted as X us , X is , and the corresponding HR images are Y us , Y is , respectively. Then, for a uniformly sampled patch pair ( x us , y us ) , the variation of network performance on the baseline model and the target model can be measured by:

where l means the loss function used for training the SR network. For simplicity, we denote l ( F ( x ; θ ) , y ) as l ( θ ; x, y ) .

Take the element of parameter θ us ik ( i -th layer) for example, the change of parameter θ us ik when the model is trained on the inverse sampling data is represented as δθ us ik , i.e.,

To further measure ∆ F ( θ is , θ us ; x us , y us ) , Cauchy mean value theorem [7] demonstates that when a function f is differentiable at point x 0 , it always stands up in the neighborhood U ( · ) of the point x 0 :

Since o ( x -x 0 ) is an infinitesimal, it can be derived that f ( x ) ≈ f ( x 0 ) + f ′ ( x 0 )( x -x 0 ) . This is one of the formulas that is often used in local linearization of a function. Geometrically, it adopts the tangent line to approximate the curve in a small neighborhood.

Then, Eq. (5) can be further formulated as:

Integrated Gradient. The above formula means that we can calculate the parameter importance by the value change and its gradient, which both consider the influence of network alteration on the model outputs. Here, we apply the integrated gradients (IG) to calculate the parameter importance, which was originally an attribution method to attribute the prediction of a deep network to its input features by accumulating the gradient of input changes. Inspired by FAIG [42], which utilizes paths in the parameter space for attributing network functional alterations to filter changes, we adopt IG to

Algorithm 1 The bi-sampling parameter attribution for compact image SR.

Input : the SR model F ( · ) with the randomly initialized weights: θ 0 ; the uniform sampling and inverse sampling datasets: D us , D is ; the total epochs: T ; the interval for training F ( · ) with inverse sampling: E ; the remaining ratio for parameters without gradient updating: p .

Output : the compact SR model F ( · ) with parameter θ .

1: while epoch ≤ T do

2: Sample the LR-HR patch pairs ( x us lr , y us hr ) from D us ;

3: Update F ( · ) with θ us on the uniformly sampled data;

4: if epoch % E = 0 then

5: Sample the LR-HR patch pairs ( x is lr , y is hr ) from D is ;

6: Update F ( · ) with θ is on the inversely sampled data;

7: Calculate the integrated gradient between the baseline model ( θ us ) and the target model ( θ is ) as Eq. (10) to obtain the parameter importance;

8: Keep p significant parameters stop gradient updating as Eq.(11) after sorting the importance;

9: end if

10: end while

obtain the parameter importance so as to help us discover the location of significant parameters in the sampling iteration.

The integrated path is calculated from the model trained with uniform sampling data to the model trained with inverse sampling data. Let γ ( α ) , α ∈ [0 , 1] denote a continuous path between the baseline model ( θ ′ ) and the target model ( θ ) with γ (0) = θ ′ , γ (1) = θ . Then, the integrated path can be represented as:

In addition, the L 1 loss is utilized to train the SR model, while the gradient of each parameter can be obtained according to the back-propagation algorithm. After defining the baseline and target model, integarted path and the loss function, the changes of network functions to each parameter with integrated gradient for an input image x can be attributed as follows:

Parameter Refinement. After calculating the integrated gradient, the position index of parameter importance can be located. The alternate training scheme is adopted for the bi-sampling data. To obtain a more compact representation, we expect the trivial parameters can also learn more effective information so we strive to distill them with other parameters fixed. Therefore, a specific proportion of significant parameters can be selected to keep unchanged and only perform the gradient updating on the remaining trivial parameters for further refinement. Especially, the selective ratio is set as:

where e is the e -th training epoch, and T is the total training epochs. β is a scaling factor. The detailed training procedure of our proposal is given in Algorithm 1.

4 Experiments

4.1 Datasets and Implementation Details

Datasets . We use DIV2K [1] to train the SR models, which is a high-quality dataset widely used for image SR. The whole dataset includes 800 training images and 100 validation images totally with diverse contents and texture details. The LR images are obtained in the same way as [52, 15]. To demonstrate the effectiveness of our method, the SR models are also evaluated on five public SR benchmark datasets: Set5 [3], Set14 [48], B100 [2], Urban100 [14] and Manga109 [29].

Uniform sampling

Inverse sampling

50

40

30

Average PSNR (dB)

20

10

18

17

16

15

c1

c2

C4

c5

c6

c7

c8

c9

c10

Class

(a) DIV2K training dataset

Uniform sampling

Inverse sampling

50

40

30

Average PSNR (dB)

20

10

18

17

16

15

C1

c2

c3

C4

c5

c6

c7

c8

c9

c10

Class

(b) DIV2K validation dataset

Uniform SR

Error map

Inverse SR

Error map

1.0

1.0

- 0.8

80

- 0.6

0.6

0.4

0.4

- 0.2

0.2

0.0

0.0

Implementation Details . During training, we fix the patch size of the HR image as 128 × 128 for 2 × , 4 × SR, and 129 × 129 for 3 × SR. We follow ClassSR [19] for data preprocessing in inverse sampling, which crops the whole image into multiple sub-images with sliding and overlap. For 2 × , 3 × SR, 4 × SR, it produces 499875 , 502200 , 499875 sub-images, respectively. Following [39], the number of the data repetition is set to 1 . The training data is augmented by random flipping and rotation. We use Adam optimizer with β 1 = 0 . 9 , β 2 = 0 . 999 to train the SR models. The mini-batch size is set to 16 . The learning rate is initialized as 2 e -4 and reduced by half per 200 epochs for 400 epochs totally. The unbalanced factor for the inverse sampling data is set to 10 and β is set to 0 . 1 . The interval of alternate training is 50 epochs and the number of classes of inverse sampling for DIV2K training dataset is 10 . In our experiments, µ is derived to 0 . 7743 according to the number of classes and the unbalanced factor. All the experiments are conducted with PyTorch framework on NVIDIA 2080Ti GPUs. Besides, objective criteria, i.e., peak signal-to-noise ratio (PSNR), structural similarity (SSIM) are utilized to evaluate the model performance. Note that the two metrics are both calculated on the Y channel of the YCbCr space as adopted in the previous works [6, 52].

4.2 Ablation Study

In this subsection, we mainly give a detailed analysis about the proposed bi-sampling parameter attribution (BSPA) for accurate image SR. Following [31], EDSR_baseline ( 16 residual blocks with 64 feature channels) is exploited as the backbone network.

Bi-sampling Strategy. The bi-sampling strategy is adopted to capture the training samples for the two-step iterative optimization procedure.

Uniform sampling ensures the original data distribution, while inverse sampling mainly enhances the feature extraction for the edge and texture regions. Note that all the cropped sub-images are grouped as 10 classes for inverse sampling in our experiments. In Tab. 1, it is observed that the baseline model obtains 28 . 22 dB in PSNR on Set14, while the model trained with inverse sampling for unbalanced factor 10 achieves 28 . 25 dB. Besides, we present the average PSNR value of each class on the cropped DIV2K training and validation sub-image dataset for the SR model separately trained by uniform sampling and inverse sampling in Fig. 3. It illustrates that the

0.00

0.01

0.02

0.03

0.04

0.05

0.06

Mse value

0

50000

100000

150000

200000

250000

300000

Number of patches

EDSR_baseline

SR model trained with inverse sampling performs much better than the SR model with the uniform sampling on the tail hard classes, while performing worse than the SR model with the uniform sampling on the simple classes. Therefore, inverse sampling helps the feature extraction on hard texture regions. We also visualize the SR results and their error maps with the GT of uniform sampling (uniform SR) and inverse sampling (inverse SR) in Fig. 4. It shows that inverse SR has a smaller error than the uniform SR on the texture region.

Integrated Gradient. To demonstrate the effect of the integrated gradient (IG), we first use uniformly sampled and inversely sampled data to alternately train the SR model. As the model 'w/o IG' shows in Tab. 1, the hybrid training obtains 0 . 06 dB improvement in PSNR against the baseline model on Set14. Besides, we compare the filter-level IG (FIG) and weight-level IG (WIG), which denote the parameter granularity in Eq.10 for each filter and each parameter. It is observed that WIG achieves the best PSNR, which illustrates that the weight is more discriminative than the filter for the model performance. Note that we adopt the weight-level IG in the experiments. Besides, we visualize the distribution curve if the MSE values are evaluated by the model trained with BSPA. As Fig. 5 shows, it is observed that the histogram distribution of BSPA becomes short-tailed. The reason is that the SR performance on the tail data is improved, while the performance on the head data is kept. Therefore, it demonstrates that our BSPA is effective in obtaining a more compact representation.

Effect of Selective Ratio. The selective factor in Eq. (11) aims to control the proportion of the significant parameters and update the remaining trivial parameters. As shown in Tab. 2, we give the quantitative comparisons of different β values for the selective ratio. It can be observed that the best PSNR is obtained when β is 0 . 1 . Therefore, we fix β as 0 . 1 in our experiments.

Effect of Unbalanced Factor. To demonstrate the effect of the unbalanced factor used for inverse sampling data generation, we give the results of our BSPA with different unbalanced factors as shown in Table 3. It shows that the best PSNR is obtained when the unbalanced factor is equal to 10 . Since the unbalanced factor determines the sampling probability of each class in inverse sampling, the large factor causes multiple repetitive sampling to the tail hard samples. It may bring about the overfitting on these areas so as to hinder the model representation.

Effect of the Number of Classes. For the classes, we compare different number of classes in Table 5. It shows that the SR performance changes little to the number of classes. Because the data in different classes can be sampled according to their proportions by means of inverse sampling so as to remedy the unbalanced data bias. In our experiments, we set the number of classes to 10 .

4.3 Comparisons with the State-of-the-arts

To demonstrate the effectiveness of our BSPA, we integrate it with three representative SR models with different model capacities, i.e., FSRCNN [10] (small), EDSR_baseline [23] (medium), and RCAN [52] (large). Besides, we compare BSPA with SamplingAug [39] (a sampling-based data augmentation method) and UDN [31] (a loss reweighting method). All these SR models are retrained under the same experimental settings with the released codes.

Quantitative Results. In Tab. 4, it shows that SamplingAug and our BSPA outperform the baseline methods with different model capacities for all scaling factors, which is even higher 1 . 32 dB than the original FSRCNN on DIV2K validation dataset for 2 × SR. The results of UDN on some datasets are

even inferior to the baseline model, which may originate from the inaccurate estimation of uncertainty. Note that the model's performance on FSRCNN for 3 × and 4 × scale is worse than UDN. FSRCNN is an extremely lightweight model, which has only 24 . 5 Kparameters and several convolutional layers. The model representation ability is restricted. Therefore, it may not fit the training well no matter the easy data or the hard data by the limited model capacity, especially for large scaling factors. Although the bi-sampling strategy increases the data diversity, there lacks sufficient model capacity to it. UDN is a loss re-weighting method for solving the data imbalance problem. It provides a regularization constraint for shrinking the solution space. Besides, our method has advantages over other compared methods on large models. For EDSR and RCAN, our BSPA almost performs the best for all scaling factors against the compared methods.

Qualitative Results. The visual comparisons on benchmark datasets for 4 × SR are shown in Fig. 6. For images 'img042' and 'img092' in Urban100 dataset, we can observe that BSPA is more favorable and can recover more texture details than other compared methods. Therefore, our proposal can not only achieve excellent performance but also be superior in the visual effect. Besides, we also provide the perceptual metrics (NIQE/LPIPS) comparisons for RCAN in Table 6. It shows that our method obtains superior NIQE and LPIPS values against the compared methods.

zebra in Set14

HR

EDSR_baseline SamplingAug

Bicubic

UDN

baboon in Set14

HR

RCAN

SamplingAug

Bicubic

UDN

img042 in Urban100

HR

EDSR_baseline SamplingAug

Bicubic

UDN

img092 in Urban100

HR

RCAN

SamplingAug

Bicubic

UDN

4.4 Limitations

We adopt a re-sampling strategy to address the unbalanced data distribution problem for image SR. It has several limitations. 1) Unlike the end-to-end loss re-weighting methods [31], our method needs to implement extra data pre-processing for classifying sub-images at first. 2) Inverse sampling allocates larger weights for the tail training samples, which may lead to over-fit upon these data. Therefore, it is important to define an appropriate unbalanced factor. Most of the existing SR models only consider that different image regions have different reconstruction diîculties, and then mainly focus on the delicate structure design (like attention mechanism) or complex regularization constraints. Few discuss the sampling strategies of training data and model this as a long-tail problem. The key idea behind the proposed Bi-Sampling Parameter Attribution method is to reconcile the unbalanced inherent data bias, namely the heavy-tailed distribution are visually more important than smooth areas. The newly developed technique with integrate gradient involves an inverse sampling strategy for enhancing the feature extraction ability of the model on the hard samples. We believe that our method has such potential for fully mining the dataset information and may inspire the research of this direction in the future.

5 Conclusion

In this paper, we rethink the current paradigm of merely using uniform sampling to capture the minibatch data for training the SR model, which would lead to the notorious data unbalanced problem. To address this issue, we propose a bi-sampling parameter attribution (BSPA) method for accurate image SR. The bi-sampling is introduced to remedy the unbalanced data distribution, including the usually adopted uniform sampling and inverse sampling. The former aims to keep the intrinsic data distribution, the latter is designed to promote the model performance on the few hard samples, which is inversely proportional to the number of each class. To achieve this, we divide the whole training dataset into multiple sub-images and group these sub-images into different classes according to the reconstruction error. Moreover, the integrated gradient method is introduced to calculate the parameter importance, with which the significant parameters are fixed and the remaining trivial parameters are further refined. Extensive experiments on SR benchmark datasets have demonstrated that our proposal achieves superior performance against the baseline methods. In our future work, we will apply the proposal to other low-level tasks based on patch training.

Acknowledgments. This work is supported by the National Key Research and Development Program of China No.2020AAA0108301; National Natural Science Foundation of China under Grants No.62176224 and No.62222602; Natural Science Foundation of Chongqing under No.CSTB2023NSCO-JOX0007; CCF-Lenovo Blue Ocean Research Fund.

References

Eirikur Agustsson and Radu Timofte. NTIRE 2017 challenge on single image super-resolution: Dataset and study. In CVPRW , 2017.

Pablo Arbelaez, Michael Maire, Charless C. Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. TPAMI , 2011.

Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie-Line Alberi-Morel. Lowcomplexity single-image super-resolution based on nonnegative neighbor embedding. In BMVC , 2012.

Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojciech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In ICANN , 2016.

Zheng Chen, Yulun Zhang, Jinjin Gu, Linghe Kong, Xiaokang Yang, and Fisher Yu. Dual aggregation transformer for image super-resolution. In ICCV , 2023.

Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang. Second-order attention network for single image super-resolution. In CVPR , 2019.

André Pierro de Camargo. A new proof of the equivalence of the cauchy mean value theorem and the mean value theorem. Am. Math. Mon. , 2020.

Kedar Dhamdhere, Mukund Sundararajan, and Qiqi Yan. How important is a neuron. In ICLR , 2019.

Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. TPAMI , 2016.

Chao Dong, Chen Change Loy, and Xiaoou Tang. Accelerating the super-resolution convolutional neural network. In ECCV , 2016.

Agrim Gupta, Piotr Dollár, and Ross B. Girshick. LVIS: A dataset for large vocabulary instance segmentation. In CVPR , 2019.

Hui Han, Wenyuan Wang, and Binghuan Mao. Borderline-smote: A new over-sampling method in imbalanced data sets learning. In ICIC , 2005.

Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Deep imbalanced learning for face recognition and attribute prediction. TPAMI , 2020.

Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In CVPR , 2015.

Zheng Hui, Xinbo Gao, Yunchu Yang, and Xiumei Wang. Lightweight image super-resolution with information multi-distillation network. In ACM MM , 2019.

Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV , 2016.

Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In CVPR , 2016.

Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Deeply-recursive convolutional network for image super-resolution. In CVPR , 2016.

Xiangtao Kong, Hengyuan Zhao, Yu Qiao, and Chao Dong. Classsr: A general framework to accelerate super-resolution networks by data characteristic. In CVPR , 2021.

Klas Leino, Shayak Sen, Anupam Datta, Matt Fredrikson, and Linyi Li. Influence-directed explanations for deep convolutional networks. In ITC , 2018.

Feng Li, Runming Cong, Huihui Bai, and Yifan He. Deep interleaved network for single image super-resolution with asymmetric co-attention. In IJCAI , 2020.

Jiwei Li, Will Monroe, and Dan Jurafsky. Understanding neural networks through representation erasure. CoRR , abs/1612.08220, 2016.

Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In CVPRW , 2017.

Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu. Large-scale long-tailed recognition in an open world. In CVPR , 2019.

Xiaotong Luo, Mingliang Dai, Yulun Zhang, Yuan Xie, Ding Liu, Yanyun Qu, Yun Fu, and Junping Zhang. Adjustable memory-efficient image super-resolution via individual kernel sparsity. In ACM MM , 2022.

Xiaotong Luo, Qiuyuan Liang, Ding Liu, and Yanyun Qu. Boosting lightweight single image super-resolution via joint-distillation. In ACM MM , 2021.

Xiaotong Luo, Yanyun Qu, Yuan Xie, Yulun Zhang, Cuihua Li, and Yun Fu. Lattice network for lightweight image restoration. TPAMI , 2022.

Dhruv Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV , 2018.

Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-based manga retrieval using manga109 dataset. Multim. Tools Appl. , 2017.

Grégoire Montavon, Alexander Binder, Sebastian Lapuschkin, Wojciech Samek, and KlausRobert Müller. Layer-wise relevance propagation: An overview. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning . 2019.

Qian Ning, Weisheng Dong, Xin Li, Jinjian Wu, and Guangming Shi. Uncertainty-driven loss for single image super-resolution. In NeurIPS , 2021.

Wenzhe Shi, Jose Caballero, Ferenc Huszar, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR , 2016.

Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Metaweight-net: Learning an explicit mapping for sample weighting. In NeurIPS , 2019.

Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In ICLRW , 2014.

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In ICML , 2017.

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Gradients of counterfactuals. In ICLR , 2017.

Longguang Wang, Yingqian Wang, Zaiping Lin, Jun-Gang Yang, Wei An, and Yulan Guo. Learning for scale-arbitrary super-resolution from scale-specific networks. In ICCV , 2021.

Shizun Wang, Jiaming Liu, Kaixin Chen, Xiaoqi Li, Ming Lu, and Yandong Guo. Adaptive patch exiting for scalable single image super-resolution. In ECCV , 2022.

Shizun Wang, Ming Lu, Kaixin Chen, Jiaming Liu, Xiaoqi Li, Chuang Zhang, and Ming Wu. Samplingaug: On the importance of patch sampling augmentation for single image super-resolution. In BMVC , 2021.

Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. ESRGAN: enhanced super-resolution generative adversarial networks. In ECCVW , 2018.

Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. In NeurIPS , 2017.

Liangbin Xie, Xintao Wang, Chao Dong, Zhongang Qi, and Ying Shan. Finding discriminative filters for specific degradations in blind super-resolution. In NeurIPS , 2021.

Wenbin Xie, Dehua Song, Chang Xu, Chunjing Xu, Hui Zhang, and Yunhe Wang. Learning frequency-aware dynamic network for efficient super-resolution. In ICCV , 2021.

Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker. Feature transfer learning for face recognition with under-represented data. In CVPR , 2019.

Ke Yu, Chao Dong, Liang Lin, and Chen Change Loy. Crafting a toolchain for image restoration by deep reinforcement learning. In CVPR , 2018.

Ke Yu, Xintao Wang, Chao Dong, Xiaoou Tang, and Chen Change Loy. Path-restore: Learning network path selection for image restoration. TPAMI , 2022.

Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV , 2014.

Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparserepresentations. In ICCS , 2010.

Xinyi Zhang, Tao Dai, Bin Chen, and Shu-Tao Xia. DDA: A dynamic difficulty-aware data augmenter for image super-resolution. In IJCNN , 2023.

Yachao Zhang, Yuan Xie, Cuihua Li, Zongze Wu, and Yanyun Qu. Learning all-in collaborative multiview binary representation for clustering. TNNLS , 2022.

Yongshun Zhang, Xiu-Shen Wei, Boyan Zhou, and Jianxin Wu. Bag of tricks for long-tailed visual recognition with deep convolutional neural networks. In AAAI , 2021.

Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image superresolution using very deep residual channel attention networks. In ECCV , 2018.

Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. BBN: bilateral-branch network with cumulative learning for long-tailed visual recognition. In CVPR , 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe key idea behind the proposed Bi-Sampling 12 Parameter Attribution (BSPA) method is to reconcile the unbalanced inherent data bias, namely the heavy-tailed distribution are visually more important than smooth areas. Similar observation has been extensively observed in the literature of image restoration and super-resolution. The newly developed technique involves an inverse sampling strategy for enhancing the feature extraction ability of the model on the hard samples. Another technical contribution is to introduce integrated gradient for further improvement. Limited experimental results are reported to show marginal improvements over UDR (NeurIPS'2021).\n\nSTRENGTHS:\n1. The motivation behind BSPA is clear and well explained. Although similar observation exists in the literature of SR, sampling-based approach for unbalanced data has not been considered for SR before (previous studies on long-tailed data distribution are mostly for high-level vision tasks such as image recognition).\n2. The proposed integrate gradient (IG) method is based on importance ranking and only updates the class of trivial parameters. This strategy is conceptually similar to the reweighting idea in the literature but easier to implement. \n3. The reported experimental results have shown descent improvement over previous benchmark method such as UDN [27].\n\nWEAKNESSES:\n1. The proposed bi-sampling framework in Fig. 2 seems to be based on heuristics. When only LR observation is available, it is unclear from the figure how to adapt the second step toward the prioritization of edge and texture regions. \n2. For inverse sampling, the procedure described in Eq. (4) lacks substantial justification. P(x)~1/x in Eq. (1) is difficult to follow (ref. [20] does not seem to directly suggest this formula).\n3. The idea of integrated gradient is inherited from previous work FAIG [38]. What is the novel contribution here? It seems that the derivation of Eqs. (9)-(10) is a direct consequence of Eq. (4) in the original FAIG paper. \n4. The reported experimental results do not support the claim of \"significantly boost the performance\" in the abstract. Both subjective and objective evaluation suggest that the performance of BSPA is comparable to that of other competing methods. With only one figure with small size images included, the superiority of BSPA is unconvincing.\n\nQUESTIONS:\n1. What is the point for Fig. 4 to make? It is not easy to see the difference between uniform and inverse SR in the current comparison (including both images and error maps).\n2. Fig. 3 (a) looks confusing. Not much difference between random sampling and inverse sampling, what does this figure attempt to show?\n3. The idea of \"better distilling the remaining trivial parameters\" is counter-intuitive. This could be an issue of literary presentation. If some parameters are not important (i.e., \"trivial\"), why do we strive to distill them better? \n4. Why do you coin the term \"bi-sampling\"? Does uniform sampling and inverse sampling carry equal importance? If not, you might want to consider a more appropriate title for this work. Bi-sampling is different from bilateral filtering where the domain and range can be viewed as a dual representation. I am not sure if a similar duality holds for sampling procedure.\n\nLIMITATIONS:\nHave the authors considered other recent competing approaches to SR such as diffusion model-based?  The baseline methods used in this paper do not seem to represent the current SOTA in SR.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-24"}

comments: SUMMARY:
The uniform sampling of the data, with flat regions occupying most of the training samples, can impair the accuracy of the reconstruction. Therefore, the authors enhance the model representation from the perspective of data sampling and propose a simple and effective Bi-Sampling Parameter Attribution (BSPA) method. Extensive experiments demonstrate that our method effectively promotes the performance of baseline models.

STRENGTHS:
1. This paper is interesting and improves the performance of existing methods from the perspective of data sampling.
2. This paper clearly written and easy to understand.

WEAKNESSES:
1. The significance of the experimental results is not sufficient.

QUESTIONS:
1. The authors do not compare their method with the latest papers, nor do they integrate the proposed method on the latest models. What is the motivation of the authors for choosing several comparative methods in the paper? For example, what is the effectiveness of this method on a transformer-based framework?

2. The authors mention that the SR data were divided into multiple groups, how was the parameter K determined here? How does this parameter affect the results?

LIMITATIONS:
I suggest that the authors could quantify the additional time that the proposed method brings to the training phase, so that the performance of the method can be more intuitively represented.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-30

comments: SUMMARY:
This work focuses on studying the unbalanced distribution of the SR training data. The authors propose a bi-sampling strategy with parameter attribution. The bi-sampling consists of uniform sampling and inverse sampling, which pay more attention to hard samples. Moreover, integrated gradient is introduced to measuring the parameter importance and stopping gradient updating of significant filters.

STRENGTHS:
+ Introducing parameter attribution into SR is a novel idea. Calculating the parameter importance for obtaining a compact model sounds reasonable. 


+ Extensive experiments are conducted to shows the advantages of the proposed method. The ablation study also helps to verify the effectiveness of the proposed techniques.

WEAKNESSES:
- The motivation is not so novel. The data unbalance of SR training data has been widely mentioned in previous works. Many sampling strategies have also been proposed to solve this problem.

-According to the descriptions in ‘parameter refinement’, the most important parameters keep unchanged after a few epochs, which may cause underfitting. Authors should provide more theoretical support or conduct more experiments to verify its effectiveness.

-Some descriptions are not very clear.

QUESTIONS:
-What is wight-level IG and filter-level IG? Clear definitions are missing. 

-In ablation study, the best PSNR is obtained when the unbalanced factor is equal to 10 and 200 gets the worst results. Why is the imbalance factor set to 200 in the experimental setting in line 167?
 
-What is the effect of different numbers of classes and intervals of alternate training? 

-What is the extra computation cost?

LIMITATIONS:
See the above comments
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nObserving the issue of uneven distribution of image contents, the author proposed to utilize inverse data sampling to resolve the inherent unbalanced data bias. In the proposed BSPA method, SR model is alternatively updated with uniformly and inversely sampled image data. For the latter, only part of the trivial parameters identified by parameter attribution are updated and the selective probability progressively increases during training.\n\nSTRENGTHS:\nThe data unbalanced problem this paper aiming to solve has been investigated in several prior literatures though from different perspectives. This work innovatively adopts the idea of resampling to address the issue, and conducts parameter attribution to balance the uniformly and inversely sampled data. The paper is organized well and reasonable experiments are provided to demonstrate the effectiveness of the proposed BSPA method.\n\nWEAKNESSES:\n1.\tA few notations in the paper are inconsistent and confusing. In Section 3.2, the superscript for “uniform sampling” is “rs”, for example, the uniformly sampled LR-HR patch pair is denoted as $(x_{lr}^{rs}, y_{hr}^{rs})$ in Algorithm 1. However, in section 3.3, the uniformly sampled patch pair is denoted as $(x_{lr}^{us}, y_{hr}^{us})$.\n2.\tThe experimental results of ablation studies presented in Tab. 1, 2, 3 only report the PSNR on the Set14 dataset which has a relatively small scale and the data distribution is not representative enough for assessing generalizability. More comprehensive and solid ablation analysis could be conducted by evaluating the models on larger testing dataset like the DIV2K validation set, BSD100, or Urban100.\n3.\tThere is a lack of ablation analysis on the number of classes $K$ which is simply set to be 10.\n4.\tThe model’s performance on FSRCNN model for x3 and x4 scale is worse than UDN method. It should be mentioned and discussed in the experiment section.\n\nQUESTIONS:\n1.\tWhat is the additional computational time cost of adding inverse sampling and parameter attribution?\n2.\tHow is the scalability of the integrated gradient for parameter attribution for larger models with more parameters?\n3.\tWhat is the shape of the distribution curve in Figure 1 if the MSE values are evaluated by the model trained with the proposed BSPA method? Will the distribution curve be more uniform or short-tailed?\n4.\tIt would be helpful to present more qualitative comparisons to demonstrate the visual effect.\n5.\tHow the data preprocessing is conducted (cropping with/without overlap?) and how many sub-images are produced?\n\nLIMITATIONS:\nLimitations of the proposed method are discussed in this paper. It would be helpful if over-fitting issue is evidenced by any experiment result and the cost of extra data preprocessing is quantitatively measured.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of data imbalance in single image super-resolution (SISR) training, where the majority of training samples contain flat regions while only a small percentage represents sharp regions with rich texture details. The authors propose a Bi-Sampling Parameter Attribution (BSPA) method to enhance model representation by explicitly increasing the sampling proportion of difficult patch pairs. This is achieved through a combination of uniform sampling and inverse sampling, which preserves the original data distribution while allocating a higher probability to patches from the tail data. Additionally, the authors propose a non-trivial solution, integrated gradient (IG), to identify important parameters and encourage their contribution by preventing gradient updating. Experimental results show that the proposed method improves the performance over baseline models. The main contributions of the paper are the identification of data imbalance in SISR training, the introduction of a bi-sampling paradigm to address the imbalance, the use of IG attribution to select important parameters, and the demonstrated effectiveness of the proposed method in enhancing SISR performance.\n\nSTRENGTHS:\n- Data imbalance in single image super-resolution (SISR) has long drawn attention and has been discussed in the community. This paper, for the first time, investigates and introduces a non-trivial technique for this problem, contributing to the advancement of the field.\n- The proposed Bi-Sampling Parameter Attribution (BSPA) method is an original approach to enhance model representation and tackle the limitations of uniform sampling commonly used in SISR methods.\n- The paper provides a thorough analysis of the data distribution problem in SISR training and supports its claims with empirical evidence.\n- Extensive experiments are conducted to validate the effectiveness of the proposed method, and the results are presented in a comprehensive and organized manner.\n\nWEAKNESSES:\n**Weaknesses**\n- In Table 1-3, the authors perform evaluations on Set14, which contains only 14 images. This may not help draw robust conclusions to more general scenarios under limited data size. The authors are suggested to conduct ablation study on larger dataset (*e.g.*, BSD100, Urban100, or DIV2k) to make more convincing conclusions.\n- What is the benefit of splitting data patches into multiple groups, compared to directly measuring balanced weight within the continuous range?\n- In inverse sampling preprocessing, what model is used to measure MSE of each patch? As $D^{rs}$ and $D^{is}$ are predefined and fixed in Algorithm 1, would it be beneficial to re-calculate and re-balance $D^{is}$ dynamically after updating $F$ with $\\theta^{us}$ in each epoch?\n- The qualitative results are limited. The authors should prepare a supplementary material with more visual results.\n\n\n**Additional comments**\n- Make consistent use of symbols. For example, $rs$ and $us$ in algorithm 1.\n- Do not use the same notations for different quantities. For example, $N$, $N_k$ and $N_i$ in Eqn 2 and algorithm 1.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nLimitations are well discussed in this paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}