Learning Re-sampling Methods with Parameter Attribution for Image Super-resolution
Authors: Xiaotong Luo
Yuan Xie
Yanyun Qu
Venue: NeurIPS 2023 poster (2023)
Keywords: image super-resolution
long-tail distribution
re-sampling
integrated gradient

Abstract: Single image super-resolution (SISR) has made a significant breakthrough benefiting from the prevalent rise of deep neural networks and large-scale training samples. The mainstream deep SR models primarily focus on network architecture design as well as optimization schemes, while few pay attention to the training data. In fact, most of the existing SR methods train the model on uniformly sampled patch pairs from the whole image. However, the uneven image content makes the training data present an unbalanced distribution, i.e., the easily reconstructed region (smooth) occupies the majority of the data, while the hard reconstructed region (edge or texture) has rarely few samples. Based on this phenomenon, we consider rethinking the current paradigm of merely using uniform data sampling way for training SR models. In this paper, we propose a simple yet effective Bi-Sampling Parameter Attribution (BSPA) method for accurate image SR. Specifically, the bi-sampling consists of uniform sampling and inverse sampling, which is introduced to reconcile the unbalanced inherent data bias. The former aims to keep the intrinsic data distribution, and the latter is designed to enhance the feature extraction ability of the model on the hard samples. Moreover, integrated gradient is introduced to attribute the contribution of each parameter in the alternate models trained by both sampling data so as to filter the trivial parameters for further dynamic refinement. By progressively decoupling the allocation of parameters, the SR model can learn a more compact representation. Extensive experiments on publicly available datasets demonstrate that our proposal can effectively boost the performance of baseline methods from the data re-sampling view.

Learning Re-sampling Methods with Parameter Attribution for Image Super-resolution
Xiaotong Luo', Yuan Xie*, Yanyun Qu
'Scooo of Informatics, Xiamen University. Fujian, China
School of Computer Science and Technology. East China Normal University, Shanghai, China CChongiing Institute of East China Normal University, Chongqing, China xiaotluodstu. xmu. edu. en, yxielca ecnu. edu. en, yyquεxmu. edw..
Abstract
Single image super-resolution (SISR) has made a significant breakthrough benefit- ing from the prevalent rise of deep neural networks and large-scale training samples. The mainstream deep SR models primarily focus on network architecture design well as optimization schemes, while few pay attention to the training data. In fact, most of the existing SR methods train the model on uniformly sampled patch pairs from the whole image. However, the uneven image content makes the training data present an unbalanced distribution, i.e., the easily reconstructed region (smooth) occupies the majority of the data, while the hard reconstructed region (edge texture) has rarely few samples. Based on this phenomenon, we consider rethinking the current paradigm of merely using uniform data sampling way for training SR models. In this paper, we propose a simple yet effective Bi-Sampling Parameter Attribution (BSPA) method for accurate image SR. Specifically, the bi-sampling consists of uniform sampling and inverse sampling, which introduced to rec- oncile the unbalanced inherent data bias. The former aims to keep the intrinsic data distribution, and the latter is designed to enhance the feature extraction ability of the model on the hard samples. Moreover, integrated gradient is introduced attribute the contribution of each parameter in the alternate models trained by both sampling data By progressively decoupling the allocation of parameters, the SR model can learn a more compact representation. Extensive experiments on publicly available datasets demonstrate that our proposal can effectively boost the performance of baseline methods from the data re-sampling view.
filter the trivial parameters for further dynamic refinement.
Introduction
Single image super-resolution (SISR) aims to reconstruct high-resolution (HR) image from its downsampled low-resolution (LR) counterpart. As a classical ill- ill-posed problem, multiple HR images can be recovered from the same LR image. With the boom of neural networks, numerous excellent
deep SR methods 9 23 25 5 have been proposed to chase for more accurate solution. It is
well-known that the success of deep learning primarily attributes three significant factors, including large-scale data I. network structures [TO] 52 23), and optimization strategies [16,31(22 [26). Most
of the existing SR models mainly focus on the delicate structure design or complex regularization constraints, while neglecting the in-depth analysis of the intrinsic training data.
Due to resource constraints, the mainstream deep SR methods mostly train the models with uni- formly sampled LR-HR patch pairs rather than the whole images. However, they ignore the
"Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

underlying fact that the content of patches is usually unevenly distributed in an image. As pre- sented in Fig. [I] we measure the patch content by mean square error (MSE) and count for all the cropped sub-images on the commonly used DIV2K training dataset. It reveals a phe- nomenon of long-tailed distribution, i.e., the flat region occupies the majority of the training samples, while the sharp region with abundant texture details only holds very small percentage. The imbalanced training data would undoubtedly impair the reconstruction accuracy, especially for the hard areas in the tail. Therefore, we consider enhancing the model representation from the data sampling perspective.
Numbe of patches
In this paper, we propose a simple yet effective Bi- Sampling Parameter Autribution (BSPA) method for image SR, which aims to obtain the compact feature representation by explicitly increasing the sampling proportion of the tail hard patch pairs.
Head
Specifically, the bi-sampling consists of uniform sampling and inverse sampling to make up for the unbalanced training data bias. As the most common way of data sampling, uniform sampling helps to keep the original data distributions, while inverse sampling aims to allocate higher prob- ability to the fewer patches sampled from the tail data. The SR model is alternately trained with the two kinds of sampling data. However, the SR models treat all the parameters equally and per- form gradient updating to each parameter, which ignores the parameter redundancy. Therefore, it is necessary to obtain a compact model according to the parameter contribution to the output. Con- sidering that the weight values would not change drastically with the small learning rate, it is difficult to determine the impact of different weights. Inspired by [55), we introduce the integrated gradient (IG) attribution method to measure the parameter importance. The IG is calculated between the previous model trained with uniform sampling data and the next model trained with inverse sampling data. Based on the importance ranking, the significant parameters do not perform gradient updating, and the remaining trivial parameters are encouraged contribute more by further optimization. Extensive experiments have demonstrated that our proposal can improve the SR performance of baseline methods. To the best of our knowledge, this the first attempt investigate image SR as a long-tail data distribution problem.
T Tail
Mse value
Figure I: The illustration of long-tailed data dis- tribution with uniform sampling in image SR. Fol- lowing [19], we first crop the commonly used training dataset, e.g. DIV2K [. into multiple sub-images, and adopt the mean square error (MSE) between the SR output by MSRResNet 40) and the HR image to measure the reconstruc- tion difficulty of an LR sub-image. It shows that the data distribution of the training samples is uneven, which presents a long-tailed distribution.
In summary, the main contributions of this paper are: 1) problem in the training samples caused by uniform sampling for image SR. 2) A bi-sampling paradigm is designed to remedy the unbalanced data bias, including uniform sampling and inverse sampling, respectively. 3) To obtain a compact representation, an integrated gradient based attribution method is introduced for measuring the parameter importance so that the significant parameters are progressively selected to stop gradient updating for better distilling the remaining trivial parameters.
We definitely find the data unbalanced
2 Related Work
2.1 Deep Image SR Models
Nowadays, numerous SR models have been proposed with remarkable performance. The pioncering work, SRCNN [9] proposed a three-layer network for image SR. Then, FSRCNN [IO] and ESPCN [32 introduced deconvolution and sub-pixel upsampling for mapping LR features to HR images, which could largely reduce computation complexity. VDSR [I7] introduced residual learning to
extract missing high-frequency information. DRRN 18 and DRCN [18] adopted recursive neural networks to reduce the model parameters while achieving excellent performance. RCAN (52) adopted a residual-in-residiee network design and embedded the channel attention in each residual block to build extremely deep network for image SR. DIN 21 designed deep interleaved network to

---

combine the information at different states. Wang et. al. [7 learned a scale-arbitrary image SR network from scale- veale-specifie networks. LatticeNet (77 proposed a lattice block to adaptively combine pair-wise residual blocks. All these SR models have been trained on uniformly sampled LR-HR patch pairs, which ignore the unbalanced data distribution in the training samples.
Recently, several region-aware SR methods have been proposed to distinguish different image regions in an image and aim to assign more computation resources on more challenging pixels adaptively. RL-Restore 4 and Path-Restore [46] decomposed the image into different sub-images and then estimated an appropriate processing path for each sub-image by reinforcement learning. ClassSR [19] proposed new classification method to determine the processing of different image regions and then reconstructed the sub-image by models of different sizes. Similar to ClassSR, FADN [43] proposed a framework for processing the different image regions divided by the corresponding information in the frequency domain with different computation burdens. APE 3X introduced an adaptive patch exiting strategy to super-resolve patches with different difficulties for achieving a flexible tradeoff between performance and efficiency. The above content-relate SR methods mainly focus on the architecture design or optimization schemes. Except for the sub-image decomposition methods, UDN (31 proposed a new adaptive weighted loss for image SR to train deep networks focusing on challenging situations such textured and edge pixels with high uncertainty.
In this paper, we mainly rethink the unbalanced distribution for image SR from the view of data sampling. SamplingAug [39] is a sampling- based data augmentation method for image SR. It only sampled the 5% most informative patches for model training. while the rest is ignored. DDA [4] designed difficulty-awan class by the relative loss. Different from these works, we propose a bi-sampling strategy and the model is alternatively updated with uniformly and inversely sampled image data. All patches are effectively utilized for SR training to help the model fully mine the image information.
data sampling strategy, which controls the sampling probability of each
2.2 Long-tail Image Classification
Due to the imbalanced data distribution caused by the difficult data acquisition in actual applications, long-tailed image recognition has attracted widespread attention [41 II (5). The existing long-tail studies can be roughly divided into three types: data sampling, loss reiweightnng, and transfer learning. The data re-sampling based methods (28) [2] aim to re-sample the training dataset for achieving a more balanced distribution. Besides, the loss re- re-weingting based methods [13 33] are proposed to allocate different weights each training samples for loss optimization. Recently, the transfer learning based methods [4 24 have sprung up to transfer the informative features learned from the head classes with abundant training samples to the under-represented tail classes. Here, we mainly explore the data re-sampling method for image SR.
As the widely adopted data re- re-balancing strategy for solving the long-tailed problem, re-sampling methods aim to sample the data to generate an evenly distributed dataset, which can be classified as: verrsampling for minority classes and under-sampling for dominant classes. However, repeated sampling for tailed samples may lead to the over-fitting problem on minority classes, while discarding the head samples will undoubtedly damage the generalization ability of deep models. Therefore, we integrate the two data sampling ways together for remedying the dataset bias in image SR task.
2.3 Attribution Analysis
Attribution analysis is usually utilized for model interpretation on the high-level image classification task, which aims to attribute the influence of each pixel in the input on prediction accuracy. The existing attribution analysis methods can be classified as: the gradient-based methods [34] 35), propagation-bai based methods [30] 4 and occlusion-basee
methods [47 (22. Here, we primarily focus on the gradient-basse based methods, which utilize the gradient of the output against the input as metric and accumulate the integrated gradients (IG) at all points along a path from the baseline input the target input. IntInf (20) and neuron conductance [8] further extend
caature-importat IG to neurons. Unlike the classical attribution methods [35] which are used to attribute the influence of the input on the classification model by attributing the contribution of each pixel of the input image to the final output label, FAIG [42 first introduced IG to attribute network functions to the filters the blind SR model and successfully find discrininative filters that closely relate to the image degradation removal. In this paper, we mainly adopt the integrated gradient method to help locate the position of important weights in SR models trained with uniform sampling data and inverse sampling data.
3

---

Integrated
Input
path
Ahnograa gradient
Baseline model
Uniform sampling
f Update
Input
Sort
Targe
Iiportannce
Significant parameter
Inverse sampling
Parameter
parameter
Figure 2: The framework of our proposed bi-sampling parameter attribution method for image SR.
3 Proposed Method
3.1 Method Overview
Considering the intrinsic dataset-specifet bias shown in Fig. [ adopt two kinds of sampling ways (bi-sampling) for capturing the training data: uniform sampling and inverse sampling, which are combined to re-balance the distribution of training data. As shown in Fig 2 the proposed bi-sampling parameter attribution (BSPA) framework includes two steps, which are alternately performed. In Step the SR model is trained with the data from uniform sampling to obtain trivial solution toward normal data distribution (baseline model). In Step 2, the SR model is fine-tuned with the data from inverse sampling for generating biased solution towards processing well on the tail edge and texture regions (target model). After training with inverse sampling data, the integrated gradient is calculated between the baseline model and the target model, which utilized obtain the parameter importance so to control the relevant weight updating.
3.2 Bi-sampling Strategy
Uniform Sampling. Uniform sampling refers to cropping the patch samples from the whole image in the training dataset randomly, which is the most common way of data sampling and widely used in image SR task [21] To be specific, the input data comes from a uniform sampler, where each sample in the training dataset is sampled with equal probability in training epoch. The probability Pus that a sample chosen from the whole training dataset can be formulated as:
pus
(1)
NT
where Ny the total number of training samples.
Inverse Sampling. Inverse sampling aims to allocate a higher probability to capture the tail hard data with few samples. Unlike long-tailed image classification, which has definite number of classes, image SR is regression task. We adopt MSE to measure the reconstruction difficulties, which is continuous value and each value corresponds value is continuous and divergently distributed, it needs to collect more samples within a certain range calculate the proportion. Therefore, we first split data patches into multiple groups to better calculate the sampling probability.
sample as shown in Fig. [ Considering the MSE
Data classification. Specifically, the number of classes is predefined K. Analogy to the long-tailed CIFAR, which is created by reducing the number of training samples per class via an exponential function (5I), then the number of samples for each class can be calculated
N₂ = N x # where & E (1, K) is the class index, N is the total number of training sub-images, and (0, 1) is an attenuation factor. Therefore, a long-tailed version of DIV2K training dataset can created by setting different MSE threshold values.
(2)

---

Moreover, the imbalance factor of a long-tailed dataset is defined as the number of training samples in the largest class (N1) divided by that of the smallest (Nx)., which usually ranges from 10 200 in long-tail research and is set as 10 in our experimental setting. Finally, the MSE threshold values used for classifying the sub-images can be obtained by:
(3) where t5 is the MSE threshold sorted by ascending. Ni. refers to the total number of sample from N1 to NR. Therefore, we can obtain the long-tailed version of DIV2K training dataset. It is noted that the test dataset remains unchanged and the SR model is still tested on the whole image.
f = t[N1:))
Sampling procedure. Following [53], the sampling possibilityy of each class for inverse sampling Pk is inversely proportional to the class sample capacity, i.e., the more samples in a class, the small sampling possibility that class has. Therefore, the procedures of inverse sampling are as follows: 1) Pi for the class k can be calculated according to the number of samples as:
P
N SJ=1
(4)
N
2) sample class i according Pke randomly; 3) select sample from class k randomly. The mini-batch inverse sampling data can be obtained by repeating the above operations.
3.3 Integrated Gradient for Parameter Attribution
Parameter Importance Formulation. The uniform sampling data is adopted to maintain the inherent data distribution. The inverse sampling data is used to train the SR model for providing an elegant solution on edge and texture areas. In this paper, adopt hybrid training mechanism with uniform and inverse sampling data, which are iteratively input into the SR model.
Since the SR model is trained on uniform sampling and inverse sampling iteratively, there exist different preferences the SR reconstruction regions. Then, it is meaningful to calculate the parameter importance for obtaining a compact model. Assume the parameter sets of the SR model F()) trained with uniform sampling data and inverse sampling data are denoted as (0 : S i L} and { 1 < i S I.). where 0,1e, 0/* E Rexxxx are the weight matrixs at the i-th layer, is the number of convolutional layers, and e; are the number of output and input channels, w and h are the kernel sizes. The two sampling LR images are denoted as Xuu X and the corresponding HR images are Yus, y's, respectively. Then, for uniformly sampled patch pair (x." y²*), the variation of network performance on the baseline model and the target model can be measured by: (5)
AF(0' ,0"s; x y²*))= = [[( (1:" 00*), =*) [(F(a::", " ;0*), y,y"))
where 1 means the loss function used for training the SR network. For simplicity, we denote ((F f), y) as I(0; x. y).
Take the element of parameter 0% (i-th layer) for example, the change of parameter 6% when the model is trained on the inverse sampling data represented d0us i.e.,
80 = 6
6ur
(6)
To further measure AF qus;,""s yy" Cauchy mean value theorem 77 demonstates that when a function f is differentiable at point xp. it always stands up in the neighborhood U--) of the point Zo:
f(xx) = f(xo) + (x(x) - xo) + o(x - xo); Since o(z - xo) is an infinitesimal, it can be derived that f(x) z fzo) + f' - mo). This is one of the formulas that is often used in local linearization of function. Geometrically, it adopts the tangent line to approximate the curve in a small neighborhood.
(7)
Then, Eq. 5) can be further formulated as:
(O 504e; 1y "d) (0 20 ","""""""""" 1y
(Vogz TT 504s.
(8)
Integrated Gradient. The above formula means that we can calculate the parameter importance by the value change and its gradient, which both consider the influence of network alteration on the model outputs. Here, we apply the integrated gradients (IG) to calculate the parameter importance, which was originally an attribution method to attribute the prediction of a deep network to its input features by accumulating the gradient of input changes. Inspired by FAIG 42), which utilizes paths in the parameter space for attributing network functional alterations to filter changes, we adopt IG to
5

---

Algorithm The bi-sampling parameter attribution for compact image SR.
the SR model F(+) with the randomly initialized weights: 0o: the uniform sampling and inverse sampling datasets: D que D**; the total epochs: T; the interval for training F-) with inverse sampling: E; the remaining ratio for parameters without gradient updating:
Input:
Output: the compact SR model F-) with parameter 0.
while epoch Σ 7 do
2: Sample the LR- AR-HR patch pairs (2/ : Vhr) from Dus;
3: Update F(-) with 8 on the uniformly sampled data;
4: if epoch % E 0 then
5:
Sample the LR-HR patch pairs (#ff, phr) from Díe;
6: Update F(-) with Hü on the inversely sampled data;
7:
Calculate the integrated gradient between the baseline model (8"2) and the target model (0'*) as Eq. (10) to obtain the parameter importance;
8: Keep significant parameters stop gradient updating as Eq. [I) after sorting the importance;
9: end if
10:
end while
obtain the parameter importance as help us discover the location of significant parameters in the sampling iteration.
The integrated path is calculated from the model trained with uniform sampling data to the model trained with inverse sampling data. Let ((x), a E [0, 1] denote a continuous path between the baseline model (0') and the target model (0) with 7()) 0¹ >(1) = 0. Then, the integrated path can be represented as:
7(a) = 0' - a 0').
In addition, the L, loss is utilized train the SR model, while the gradient of each parameter can be obtained according to the back-pro propagation algorithm. After defining the baseline and target model, integarted path and the loss function, the changes of network functions to each parameter with integrated gradient for input image x can be attributed as follows:
IG x) (()(11)) x)) 1(7(0),
= =E ((((a),
x ar(a); do
a=0 Or(a),
da
(10)
00 0'1 M-1 M
[a(r(()).)) Sy(a)
Σ
((() +o(0- d').0=)))
Parameter Refinement. After calculating the integrated gradient, the position index of parameter importance can be located. The alternate training scheme is adopted for the bi-sampling data. To obtain a more compact representation, we expect the trivial parameters can also learn more effective information so we strive distill them with other parameters fixed. Therefore, a specific proportion of significant parameters can be selected to keep unchanged and only perform the gradient updating on the remaining trivial parameters for further refinement. Especially, the selective ratio is set as:
= * (e/T).
(11)
where e is the e-th training epoch, and T is the total training epochs. B is a scaling factor. The detailed training procedure of our proposal given Algorithm
Experiments
4.1 Datasets and Implementation Details
Datasets. We use DIV2K [ to train the SR models, which is a high-quality dataset widely used for image SR. The whole dataset includes 800 training images and 100 validation images totally with
diverse contents and texture details. The LR images are obtained in the same way [52 15). To
demonstrate the effectiveness of our method, the models are also evaluated on five public SR
benchmark datasets: Set5 (3) Set14 [48), B100 22. Urban100 [4] and Mangal09 29

---

18
training dataset
DIV2K validation dataset
Figure 3: The PSNR of each class the classified DIV2K training and validation datasets for uniform sampling and inverse sampling.
map Figure 4: Visual comparisons of the SR results by the uniform sampling (uniform SR) and the inverse sampling (inverse SR), and the corresponding error maps with the GT.
Uniform
Error mup
Implementation Details. During training, we fix the patch size of the HR image as 128 x 128 for 2x, 4x SR, and 129 x 129 for 3x SR. We follow ClassSR [19] for data preprocessing in inverse sampling, which crops the whole image into multiple sub-images with sliding and overlap. For 2x, 3x SR, 4x SR, it produces 499875, 502200, 499875 sub-images, respectively. Following [39], the number of the data repetition is set to The training data is augmented by random flipping and rotation. We use Adam optimizer with 81 0.9. 82 0.999 to train the SR models. The mini-bated size set to 16. The learning rate initialized as 2e - 4 and reduced by half per 200 epochs for 400 epochs totally. The unbalanced factor for the inverse sampling data is set to 10 and is set to 0.1. The interval of alternate training 50 epochs and the number of classes of inverse sampling for DIV2K training dataset is 10. our experiments, μ derived to 0.7743 according the number of classes and the unbalanced factor. All the experiments are conducted with Py ToTcch framework on NVIDIA 2080Ti GPUs. Besides, objective criteria, i.e., peak signal-to-noise ratio (PSNR), structural similarity (SSIM) are utilized to evaluate the model performance. Note that the two metrics are both calculated on the Y channel the YCbCr space adopted in the previous works 6 52).
4.2 Ablation Study
In this subsection, we mainly give detailed analysis about the proposed bi-sampling parameter attribution (BSPA) for accurate image SR. Following [31], EDSR bascline (16 residual blocks with 64 feature channels) is exploited as the backbone network.
Bi-sampling Strategy. The bi-sampling strategy is adopted capture the training samples for the two-step iterative optimization procedure. Uniform sampling ensures the original data distri- bution, while inverse sampling mainly enhances the feature extraction for the edge and texture regions. Note that all the cropped sub-images are grouped as 10 classes for inverse sampling our experiments. In Tab. [ it is observed that the baseline model obtains 28.22 dB in PSNR on Set14, while the model trained with inverse sampling for unbalanced factor 10 achieves 28.25 dB. Besides, we present the average PSNR value of each class on cropped DIV2K training and validation sub-image dataset for the SR model separately trained by uniform sampling and verse sampling in Fig. It illustrates that the
CESR
22002
200003
3.00
6.02
6.32
0.03
0.00
3.00
0.00
Figure 5: The histogram distribution of train- ing sub-images in DIV2K for EDSR baseline and BSPA (4 x SR). It is observed that the dis- tribution of BSPA becomes short-tailed.
7

---

Table 1: Ablation studies about the bi- bi-sampling parameter attribution on benchmarks for 4x SR.
Model
Baveline (uniform sampling
inverse sampling
w/a IG
FIG
WIG
Se114
22
28.25
28.28
28.52
28.37
B100
27.31
27.33
27.35
27.38
27.40
Urban
25.27
25.34
25.38
25.47
DIV2K valid
29.97
30.02
30.06
30.09
30.13
Table 2: The quantitative comparisons (PSNR) of different selective ratios on bench-
Table 3:
The quantitative comparisons (PSNR) of different unbalanced factors on
marks for 4x SR.
benchmarks for 4x SR.
Unbalanced Factor
10
50
100
Set14
28.37
28.35
28.34
28.32
Seil4
28.37
28.33
28.28
28.21
B100
27.40
27.37
27.35
27.31
B100
27.40
27.35
27.31
27.23
Urban110
25.53
25.51
Urban100
25.53
25.42
25.30
DIV2K valid
30.13
30.10
30.07
30.05
DIV2K valid
30.13
30.06
30.00
29.91
SR model trained with inverse sampling performs
much better than the SR model with the uniform sampling the tail hard classes, while performing worse than the SR model with the uniform sampling on the simple classes. Therefore, inverse sampling helps the feature extraction on hard texture regions. We also visualize the SR results and their error maps with the GT of uniform sampling (uniform SR) and inverse sampling (inverse SR) in It shows that inverse SR has smaller error than the uniform SR on the texture region.
Fig. 4
Integrated Gradient. To demonstrate the effect of the integrated gradient (IG), we first use uniformly sampled and inversely sampled data to alternately train the SR model. As the model "w/o IG shows in Tab. 1. the hybrid training obtains 0.06 dB improvement in PSNR against the baseline model on Set14. Besides, we compare the filter-level IG (FIG) and weight-level IG (WIG), which denote the parameter granularity in Eq for each filter and each parameter. It is observed that WIG achieves the best PSNR, which illustrates that the weight is more discriminative than the filter for the model performance. Note that we adopt the weight- IG in the experiments. Besides, we visualize the distribution curve if the MSE values are evaluated by the model trained with BSPA. As Fig. 5 shows, it is observed that the histogram distribution of BSPA becomes short-tailed. The reason that the SR performance on the tail data is improved, while the performance on the head data is kept. Therefore, it demonstrates that our BSPA is effective in obtaining a more compact representation.
aims to control the proportion of the significant parameters and update the remaining trivial parameters, As shown in Tab. 2 we give the quantitative comparisons of different 3 values for the selective ratio. It can be observed that the best PSNR is obtained when 33 is 0.1. Therefore, we fix 3 as 0.1 in our experiments.
Effect of Selective Ratio. The selective factor in Eq.
Effect of Unbalanced Factor. To demonstrate the effect of the unbalanced factor used for inverse sampling data generation, we give the results of our BSPA with different unbalanced factors as shown in Table 3 It shows that the best PSNR is obtained when the unbalanced factor is equal to 10. Since the unbalanced factor determines the sampling probability of each class in inverse sampling, the large factor causes multiple repetitive sampling to the tail hard samples. It may bring about the overfitting on these areas so as to hinder the model representation.
Effect of the Number of Classes. For the classes, we compare different number of classes in Table 5] It shows that the SR performance changes little the number of classes. Because the data in different classes can be sampled according to their proportions by means of inverse sampling so as to remedy the unbalanced data bias. In our experiments, we the number of classes to 10.
4.3 Comparisons with the State-of-the-ars
To demonstrate the effectiveness of our BSPA, we integrate with three representative SR models with different model capacities, i.c., FSRCNN [10] (small), EDSR_ baselin 23 (medium), and
RCAN [52] (large). Besides, we compare BSPA with SamplingAug [39] (a sampling-bar based data augmentation method) and UDN Bi] (a loss reweighting method). All these SR models retrained under the same experimental settings with the released codes.
Quantitative Results. In Tab. 4 it shows that Sampling and our BSPA outperform the baseline methods with different model capacities for all scaling factors, which even higher 1. 1.32dB than the original FSRCNN on DIV2K validation dataset for 2x SR. The results of UDN on some datasets are
8

---

Table 4: The quantitative comparison results of average PSNR/SSIM for 2x, 3x, 4x image SR. The best results are highlighted in black bold and the second best is in underline.
Scale
Method
Params
PSNR/SSIM
Se15
PSNR/SSIM
S
PSNR/SSIM
B100
PSNR/SSIM
Urban 1O0
Mangal09 PSNR/SSIM
DIV2K valid
PSNR/SSIM
FSRCNN 10
24.5K
$5.0551.9180
30.02/0.8625 8625
29.44/0,8366
26.77/0.8320
30.29/0.9172
32.1890.989
SamplingAug
24.5K
34.71/0.9414
31.24/0 8942
30.51/0.8766
27.83/0 8667
32.28/0.9452
33,46/0.9225
UDN [1]
24.5K
34.79/0.9412
31,29/0 8940
30.5000.8758
27.81/0.8660
32.34/0.9448
33.4770.9223
BSPA (ours)
24.5K
34.80/0.9420
31.27/0.8943
30.54/0.8772
27.86/0.8622
32.27/0.9454
33,50/0,9232
25
14M
37.5800.5590
33.15/0.9140
31 8959
30,8900:9156
37.57/0.9745
35.5400.9411
SamplingAug 89
14M
37.6540. 9591
33.25/00.9 9146
31 96/96
31.15/0.9184
37.76/099747
35.66/0. 9420
2
UDN [1
1.4M
37. 77.6000.9590
13.12/0.9115
31 89/8.8960
30,93/0.9160
37.67/0.9747
35.56/0.9411
BSPA (ours)
1.4M
37.7010.5594
33.27/0.9149
31.96/0.8968
31.1840.9185
37.78/0.7752
35.69/0.9420
RCAN 522
37.7 1A1.9596
35.31/0.9151
99/18876
31,41009223
37,9309755
35.744199428
SamplingAng 300
15.4M
37 89/0 9599
33.47/0.9161
32. 8978
31 69/00.92240
38. 38.39/0.9762
35.91/0,9436
UDN []
15.4M
37. 86/0 9601
33.43/0.9162
32. 32.05/0.9883
31.65/099239
38.15/0.9760
944
BSPA (ours)
15.4M
37.94/0.9603
33.580.9181
82.1440.8994
91.922
38.49/0.9767
36.06/0.9449
FSRCNN 10
24.3K
29.9100.8520
27.26/0. 7625
27 27.0770.733 7309
24. 24.347077277
26.46/083188
29. 29.3870.8221
SamplingAug
24.5K 02.02/0..0 8794
28. 13/0 7978
24.97/0.7574 7574
27.58/0.8613 8632
1800 8476
UDN [3]
24.5K
28.27:0.8018 31.21/0.8855
27.75
25.0330.7613
27.78/0.662
30,2
BSPA (ours)
24.5K
31.1300 8823
28 20.0 8008
27. 24.74/0. 7693
25.01/0.7611
27.65/0.8680
84,2400.8498
EDSR buseline
23
16M
9291
29.91A0 8340
28. 7379
27.21N18305
32. 900.9455
3187718777
SamplingAug
39
1.6M
33,8500 9223
30,00/0 8353
28 86/0 7987
27.35/0.8344
32. 55/09361131
310950.8800
3x
UDN [1]
1.6
33, 750.110
29, 89/89/0.0 8335
28 82/0
27.15/0.8298
32.33/0.9344
31 8540.8766
BSPA (ours)
1.6M
33.97/0.9237
30.03/0.8361
79.999
27.42/0.8359
32.60/0.9372
31.99/0.8807
KCAN [52]
15.0
9006
29 27,8770 8328
7800
27.
32. 9320
31. 810.867776
Sampling.Ag
30
15.6M
34.15/0.2277
30.20/0 8390
28. 96196000011
27.78/0.8439
33.13/0,9403
32.1400.8800
UDN [X]
15.6M
34.26/0. 92.5
30.22/0.8:30 8397
29 8026
27.80N0.8448 8448
33.05/0.94407
32.2300.8846
BSPA (ours)
15.6M
34,35/0. 9268
30.31.00.441
29.07/0.80044
28. 15
33,35/0.9433 94.32
32.36/0 8867
FSRCNN 10
24.5K
28.14/0. 7966
25. 25.9170.69779 6979
25 6666
23. 1410.6577
24.71/0.7705
28 7710
Sampling/Aug
29. (2/0222..0
26.52.00.22 7266
26.37/0.69933
23.599.6813
28.57A0.791 25.40007950
LJDN [S]
24.5K
29. 19.1100
26.53/0.7272
26.37700.9366
23.58/0.6811
25.45/0.7965
28.60/0.7927
BSPA (ours)
24.5K
29.040088411
26.51/0.7261
26.35/0.692 69211
23.55/0.6790
25 79/77
28. 78.56
FTTSR baseline 23
1.5M
31.5770.861
28.22/0.7722
27.31ML 1...1...266 7268
25,2720.5574
29.27/0 8902
SamplingAng 30
1.5M
31.72/0. 8876
28. 7.33
27. 34/04/007 728
25.39/0. 7624
29. 12.57700 8936
00.04/0.8279
UDN [1
1.5M
31.44/0. 8833
28. 15/0.707777
27.277/77257
25.17/0.7539
29.05/0 8866
29.90038223
BSPA (ours)
1.5M
81.00..8897
28.37/0.7765
27.40/0.7312
25.53M0.7685
29.72/0.9922
30.13/0.8312
RCAN 152
91192189910
28, 58/0.7766
25.5410. 706
29.77/0.982
30.16AL8111
SamplingAAg
30
15.6M
32.00/0.8911
28, 48/0.7788
27,47/077325 7323
25.8000.7771
30. 17/0-9027
30.23/0.3311
UDN X1
15.6M
31.91/0.8900 8907
28, 08.440007773
27, 7311
25.70/0.7721
86/8.68/999
30.21/0.8321
BSPA (ours)
15.6M
32. 18/0.8933
28.5690.7807
27.54/0.7348
26.02N0.7839
30.34/0.9000
30.39/0.8363
even inferior to the baseline model, which may originate from the inaccurate estimation of uncertainty. Note that the model's performance on FSRCNN for 3 and 4x scale is worse than UDN. FSRCNN an extremely lightweight model, which has only 24.5K parameters and several convolutional layers. The model representation ability is restricted. Therefore, it may not fit the training well no matter the easy data or the hard data by the limited model capacity, especially for large scaling factors. Although the bi-sampling strategy increases the data diversity, there lacks sufficient model capacity to it. UDN is a loss re-weighting method for solving the data imbalance problem. It provides a regularization constraint for shrinking the solution space. Besides, our method has advantages over other compared methods on large models. For EDSR and RCAN, our BSPA almost performs the best for all scaling factors against the compared methods.
Qualitative Results. The visual comparisons on benchmark datasets for 4x SR are shown in Fig. 6. For images "img042" and "img092" in Urban dataset, we can observe that BSPA is more favorable and can recover more texture details than other compared methods. Therefore, our proposal can not only achieve excellent performance but also be superior in the visual effect. Besides, we also provide the perceptual metrics (NIQE/LPIPS) comparisons for RCAN in Table It shows that our method obtains superior NIQE and LPIPS values against the compared methods.
Table 5: The quantitative comparisons (PSNR) of different numbers of classes benchmarks for 4x SR.
Table 6: The quantitative comparison results (NIQE/LPIPS) on benchmarks for 4x SR.
Model
Set14
B100
Urhan100
DIV2K valid
Classes (K)
5
10
20
RCAN
6.39/0.2932
6.73/0.386
5.8380.2506
6.0310.28. 28.36
Se114
28.36
28.37
28,38
SamplingAug
6.33/0.2876
6.53/038809
5.6000.2416
5.8310.7784
B100
27.41
27.40
27.42
UDN
6.61/0.2926 2.69/0.3800
5. 5.76/0.2475
5.85/0.2813
Urban 100
25.53
25.53
25.54
BSPA (ours)
6.26/0.2844
6.46/0.3750
5.61/0.2314
5.79/0.2748
DIV2K valid
30.13
30.13
30.14
9

---

IIR
EDSR, buseline SamplingAAg
II
RCAN
SamplingAuu Aug
zebra in Set14
Bicubic
UDN
BSPA
bahoon Sct
Bicubic
UDN
BSPA
HR
EDSR baseline SamplingAug
HR
RCAN
Sampling Aug
img042 in Urban100
Bicubic
UDN
BSPA
img092 Urban100
Bicubic
UDN
BSPA
Figure 6: Visual comparisons of our BSPA with other methods on Set14 and Urban 100 for 4x SR.
4.4 Limitations
We adopt a re-sampling strategy to address the unbalanced data distribution problem for image SR. has several limitations. 1) Unlike the end-to-end loss re-weighting methods [31. our method needs to implement extra data pre-processing for classifying sub-images at first. 2) Inverse sampling allocates larger weights for the tail training samples, which may lead to over-fit upon these data. Therefore, is important to define an appropriate unbalanced factor. Most of the existing SR models only consider that diffferent image regions have different reconstruction diiculties, and then mainly focus on the delicate structure design (like attention mechanism) or complex regularization constraints. Few discuss the sampling strategies of training data and model this as a long-tail problem. The key idea behind the proposed Bi-Sampling Parameter Attribution method is to reconcile the unbalanced inherent data bias, namely the heavy-tailed distribution visually more important than smooth areas. The newly developed technique with integrate gradient involves an inverse sampling strategy for enhancing the feature extraction ability of the model the hard samples. We believe that method has such potential for fully mining the dataset information and may inspire the research of this direction in the future.
Conclusion
In this paper, we rethink the current paradigm of merely using uniform sampling to capture the mini- batch data for training the SR model, which would lead to the notorious data unbalanced problem. To address this issue, we propose bi-sampling parameter attribution (BSPA) method for accurate image SR. The bi-sampling introduced to remedy the unbalanced data distribution, including the usually adopted uniform sampling and inverse sampling. The former aims to keep the intrinsic data distribution, the latter designed to promote the model performance on the few hard samples, which is inversely proportional to the number of each class. To achieve this, we divide the whole training dataset into multiple sub-images and group these sub-images into different classes according to the reconstruction error. Moreover, the integrated gradient method is introduced to calculate the parameter importance, with which the significant parameters are fixed and the remaining trivial parameters are further refined. Extensive experiments SR benchmark datasets have demonstrated that our proposal achieves superior performance against the baseline methods. In our future work, we will apply the proposal to other level tasks based patch training.
Acknowledgments. This work supported by the National Key Research and Development Program of China A0.2020AAA0108300 der Grants No.62176224 and No. No.62222602; Natural Science Foundation of Chongqing under No. No.CSTB2U2ANSCO-JOX000 CCF-Lenovo Blue Ocean Research Fund.
National Natural Science Foundation of China un-
10

---

References
[1] Eirikur Agustsson and Radu Timofte. NTIRE 2017 challenge on single image super-resolution: Dataset and study. In CVPRW, 2017.
[2] Pablo Arbelaez, Michael Maire, Charless C. Fowlkes, and Jitendra Malik. Contour detection and hicrarchical image segmentation. TPAMI, 2011.
[3] Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie-Line Alberi-Morel. Low- complexity single-image super-resolution based on nonnegative neighbor embedding. In BMVC, 2012.
[4] Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Rabs-Robert Müller, and Woj- ciech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In ICANN, 2016.
[5] Zheng Chen, Yulun Zhang, Jinjin Gu, Linghe Kong, Xiaokang Yang, and Fisher Yu. Dual aggregation transformer for image super-resolution. In ICCV, 2023.
[6] Tao Dai, Jianrui Cai, Yongbing Zhang. Shu-Tao Xia, and Lei Zhang. Second-order attention network for single image super- rupertresolution. In CVPR, 2019.
[7] André Pierro de Camargo. A new proof of the equivalence the cauchy mean value theorem and the mean value theorem. Am. Math. Mon., 2020.
[8] Kedar Dhamdhere, Mukund Sundararajan, and Qiqi Yan. How important is neuron. In ICLR, 2019.
[9] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. TPAMI, 2016.
[10] Chao Dong, Chen Change Loy, and Xiaoou Tang. Accelerating the super-resolution convolu- tional neural network. In ECCV, 2016.
[11] Agrim Gupta, Piotr Dollár, and Ross B. Girshick. LVIS: A dataset for large vocabulary instance segmentation. In CVPR, 2019.
[12] Hui Han, Wenyuan Wang, and Binghuan Mao. Borderline-smote: A new over-sampling method in imbalanced data sets learning. In ICIC, 2005.
[13] Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Deep imbalanced learning for face recognition and attribute prediction. TPAMI, 2020.
[14] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In CVPR, 2015.
[15] Zheng Hui, Xinbo Gao, Yunchu Yang, and Xiumei Wang. Lightweight image super-resolution with information multi-distillation network. In ACM MM, 2019.
[16] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.
[17] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In CVPR, 2016.
[18] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Deeply-recursive convolutional network for image super-resolution. In CVPR, 2016.
[19] Xiangtao Kong, Hengyuan Zhao, Yu Qiao, and Chao Dong. Classsr: general framework to accelerate super-resolution networks by data characteristic. In CVPR, 2021.
[20] Klas Leino, Shayak Sen, Anupam Datta, Matt Fredrikson, and Linyi Li. Influence-directed explanations for deep convolutional networks. In ITC, 2018.
[21] Feng Li, Runming Cong. Huihui Bai, and Yifan He. Deep interleaved network for single image super-resolution with asymmetric co-attention. In IJCAI, 2020.
11

---

[22] Jiwei Li, Will Monroe, and Dan Jurafsky. Understanding neural networks through representation erasure. CoRR, abs/1612.08220,, 2016.
[23] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In CVPRW, 2017.
[24] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang. Boging Gong, and Stella X. Yu. Large-scale long-tailed recognition in an open world. In CVPR, 2019.
[25] Xiaotong Luo, Mingliang Dai, Yulun Zhang, Yuan Xie, Ding Liu, Yanyun Qu. Yun Fu, and Junping Zhang. Adjustable memory-efficien image super-resolution via individual kernel sparsity. In ACM MM, 2022.
[26] Xiaotong Luo, Qiuyuan Liang, Ding Liu, and Yanyun Qu. Boosting lightweight single image super-resolution via distilistilllttion. In ACM MM, 2021.
[27] Xiaotong Luo, Yanyun Qu, Yuan Xie, Yulun Zhang, Cuihua Li, and Yun Fu. Lattice network for lightweight image restoration. TPAMI, 2022.
[28] Dhruv Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 2018.
[29] Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto. Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-based manga retrieval using manga109 dataset. Multim, Tools Appl., 2017.
[30] Grégoire Montavon, Alexander Binder, Sebastian Lapuschkin, Wojeiech Samek, and Klaus- Robert Müller. Layer-wise relevance propagation: An overview. In Explainable AI: Interpreting. Explaining and Visualizing Deep Learning. 2019.
[31] Qian Ning, Weisheng Dong, Xin Li, Jinjian Wu, and Guangming Shi. Uncertainty-driver loss for single image super-resolution. In NeurIPS, 2021.
[32] Wenzhe Shi, Jose Caballero, Ferenc Huszar, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR, 2016.
[33] Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta- weight-net: Learning an explicit mapping for sample weighting. In NeurIPS, 2019.
[34] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In ICLRW, 2014.
[35] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In ICML, 2017.
[36] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Gradients of counterfactuals. In ICLR, 2017.
[37] Longguang Wang, Yingqian Wang, Zaiping Lin, Jun-Gang Yang. Wei An, and Yulan Guo. Learning for scale-arbitrary super-resolution from scale-specific networks. In ICCV, 2021.
[38] Shizun Wang, Jiaming Liu, Kaixin Chen, Xiaoqi Li, Ming Lu, and Yandong Guo. Adaptive patch exiting for scalable single image super-resolution. In ECCV, 2022.
[39] Shizun Wang, Ming Lu, Kaixin Chen, Jiaming Liu, Xiaoqi Li, Chuang Zhang, and Ming Wu. Samplingaug: On the importance of patch sampling augmentation for single image super-resolution. BMVC, 2021.
[40] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. ESRGAN: enhanced super-resolution generative adversarial networks. In ECCVW, 2018.
[41] Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. In NeurlPS, 2017.
12

---

[42] Liangbin Xie, Xintao Wang, Chao Dong, Zhongang Qi, and Ying Shan. Finding discriminative filters for specific degradations in blind super-resolution In NeurIPS, 2021.
[43] Wenbin Xie, Dehua Song, Chang Xu, Chunjing Xu, Hui Zhang, and Yunhe Wang. Learning frequency-aware dynamic network for efficient super-resolution. In ICCV, 2021.
[44] Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker. Feature transfer learning for face recognition with under-represented data. In CVPR, 2019.
[45] Ke Yu, Chao Dong, Liang Lin, and Chen Change Loy. Crafting a toolchain for image restoration by deep reinforcement learning. In CVPR, 2018.
[46] Ke Yu, Xintao Wang. Chao Dong, Xiaoou Tang, and Chen Change Loy. Path-restore: Learning network path selection for image restoration. TPAMI. 2022.
[47] Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks, In ECCV, 2014.
[48] Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse- representations. In ICCS, 2010.
[49] Xinyi Zhang, Tao Dai, Bin Chen, and Shu-Tao Xia. DDA: A dynamic difficulty-wware data augmenter for image super-resolution. In IJCNN, 2023.
[50] Yachao Zhang, Yuan Xie, Cuihua Li, Zongze Wu, and Yanyun Qu. Learning all- collaborative multiview binary representation for clustering. TNNLS, 2022.
[51] Yongshun Zhang, Xiu-Shen Wei, Boyan Zhou, and Jianxin Wu. Bag of tricks for long-tailed visual recognition with deep convolutional neural networks. In AAAI, 2021.
[52] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang. Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention networks. In ECCV, 2018.
[53] Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zha Chen. BBN: bilateral-branch network with cumulative learning for long-tailed visual recognition. In CVPR, 2020.
13


Reviews:
[{"review_id": "FCoJRFH3M4", "reviewer": "Reviewer_zt8x", "summary": "This paper addresses the problem of data imbalance in single image super-resolution (SISR) training, where the majority of training samples contain flat regions while only a small percentage represents sharp regions with rich texture details. The authors propose a Bi-Sampling Parameter Attribution (BSPA) method to enhance model representation by explicitly increasing the sampling proportion of difficult patch pairs. This is achieved through a combination of uniform sampling and inverse sampling, which preserves the original data distribution while allocating a higher probability to patches from the tail data. Additionally, the authors propose a non-trivial solution, integrated gradient (IG), to identify important parameters and encourage their contribution by preventing gradient updating. Experimental results show that the proposed method improves the performance over baseline models. The main contributions of the paper are the identification of data imbalance in SISR training, the introduction of a bi-sampling paradigm to address the imbalance, the use of IG attribution to select important parameters, and the demonstrated effectiveness of the proposed method in enhancing SISR performance.", "questions": "See weaknesses.", "limitations": "Limitations are well discussed in this paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Data imbalance in single image super-resolution (SISR) has long drawn attention and has been discussed in the community. This paper, for the first time, investigates and introduces a non-trivial technique for this problem, contributing to the advancement of the field.\n- The proposed Bi-Sampling Parameter Attribution (BSPA) method is an original approach to enhance model representation and tackle the limitations of uniform sampling commonly used in SISR methods.\n- The paper provides a thorough analysis of the data distribution problem in SISR training and supports its claims with empirical evidence.\n- Extensive experiments are conducted to validate the effectiveness of the proposed method, and the results are presented in a comprehensive and organized manner.", "weaknesses": "**Weaknesses**\n- In Table 1-3, the authors perform evaluations on Set14, which contains only 14 images. This may not help draw robust conclusions to more general scenarios under limited data size. The authors are suggested to conduct ablation study on larger dataset (*e.g.*, BSD100, Urban100, or DIV2k) to make more convincing conclusions.\n- What is the benefit of splitting data patches into multiple groups, compared to directly measuring balanced weight within the continuous range?\n- In inverse sampling preprocessing, what model is used to measure MSE of each patch? As $D^{rs}$ and $D^{is}$ are predefined and fixed in Algorithm 1, would it be beneficial to re-calculate and re-balance $D^{is}$ dynamically after updating $F$ with $\\theta^{us}$ in each epoch?\n- The qualitative results are limited. The authors should prepare a supplementary material with more visual results.\n\n\n**Additional comments**\n- Make consistent use of symbols. For example, $rs$ and $us$ in algorithm 1.\n- Do not use the same notations for different quantities. For example, $N$, $N_k$ and $N_i$ in Eqn 2 and algorithm 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response to Reviewer zt8x (denoted as R1)\n\n*Q1-1: In Table 1-3, the authors perform evaluations on Set14, which contains only 14 images. This may not help draw robust conclusions to more general scenarios under limited data size. The authors are suggested to conduct ablation study on larger dataset (e.g., BSD100, Urban100, or DIV2k) to make more convincing conclusions.*\n\nA1-1: Thanks for your suggestions. In this work, we mainly perform evaluations on Set14 as the previous works [ref1-1][ref1-2] do. Besides, we supplement more results on other datasets in the following tables. It is observed that our method is robust to more general scenarios.\n\nTable 1-1: Ablation studies about the bi-sampling parameter attribution on benchmark datasets for $4\\times$ SR.\n| Model  |\tBaseline (uniform sampling)\t| inverse sampling\t| w/o IG\t| WIG |\tFIG | \n|------ |:-----:|:-----:|:-----:|:-----:|---------:|\n| Set14 |\t28.22 |\t28.25 |\t28.28 |\t28.32 |\t28.37 |\n| B100\t| 27.31 |\t27.33 |\t27.35 |\t27.38 |\t27.40 |\n| Urban100 |\t25.27 |\t 25.34 |\t25.38 |\t25.47 |\t25.53 |\n| DIV2K |\t29.97 |\t30.02 |\t30.06 |\t30.09 |\t30.13 |\n\nTable 1-2: The quantitative comparisons of different scaling factor in Eq.(11) on benchmark datasets for $4\\times$ SR.\n| $\\beta$ | 0.1 |\t0.5 |\t0.8 |\t1.0 |\n|------ |:-----:|:-----:|:-----:|---------:|\n|Set14|\t28.37 |\t28.35 |\t28.34 |\t28.32|\n|B100|\t27.40 |\t27.37 |\t27.35 |\t27.31|\n|Urban100| \t25.53|\t25.51 |\t25.46 |\t25.43|\n|DIV2K |\t30.13 | 30.10 |\t30.07 |\t30.05|\n\nTable 1-3: The quantitative comparisons (PSNR) of different unbalanced factors on benchmark datasets for $4\\times$ SR.\n|Unbalanced factor |\t 10 |\t50 |\t100 |\t200 |\n|------ |:-----:|:-----:|:-----:|---------:|\n|Set14 |\t28.37 |\t28.33 |\t28.28 |\t28.21 |\n|B100 |\t27.40 |\t27.35 |\t27.31 |\t27.23 |\n|Urban100 |\t25.53 |\t25.49 |\t25.42 |\t25.30 |\n|DIV2K |\t30.13 | 30.06 |\t30.00 |\t29.91 |\n\n[ref1-1] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, Yun Fu: Image Super-Resolution Using Very Deep Residual Channel Attention Networks. ECCV 2018.\n\n[ref1-2] Yiqun Mei, Yuchen Fan, Yuqian Zhou: Image Super-Resolution With Non-Local Sparse Attention. CVPR 2021\n\n*Q1-2: What is the benefit of splitting data patches into multiple groups, compared to directly measuring balanced weight within the continuous range?*\n\nA1-2: It **cannot** measure balanced weight within the continuous range. Inverse sampling aims to allocate a higher probability to capture the tail hard data with few samples. We adopt MSE to measure the reconstruction difficulties, which is a continuous value and each value corresponds to a sample. Considering the MSE value is continuous and divergently distributed, it needs to collect more samples within a certain range to calculate the proportion. Therefore, we split data patches into multiple groups for better calculating the sampling probability. \n\n*Q1-3: In inverse sampling preprocessing, what model is used to measure MSE of each patch? As Drs and Dis are predefined and fixed in Algorithm 1, would it be beneficial to re-calculate and re-balance Dis dynamically after updating F withθus in each epoch?*\n\nA1-3: (1) We adopted the **pretrained MSRResNet** to measure MSE of each patch and **mentioned** this in Line 50-57 of the manuscript.\n\n(2) It would be **not** beneficial to re-calculate and re-balance $D^{is}$ dynamically after updating F with $\\theta^{us}$ in each epoch. The inverse sampling preprocessing consists of data classification and sampling procedures.\n\n- We adopt the pretrained SR model to measure MSE of each patch at the begining, which aims to obtain the recovery difficulty of each patch for better classification. However, the dynamic adjustment of $D^{is}$ would introduce extra noise for the poor model performance in the early training process.\n- It is time-consuming for adjustment in each epoch. All the cropped patches should be evaluated to obtain their reconstruction difficulty and then classified into different groups according to MSE, which is measured in a global view. Besides, the number of sub-images is 499875, 502200, 499875 for $2\\times$, $3\\times$, and $4\\times$ SR. It would bring in more time costs for dynamic re-calculation and re-balance in each epoch.\n\n*Q1-4: The qualitative results are limited. The authors should prepare a supplementary material with more visual results.*\n\nA1-4: We have supplemented more visual results in **Figure 1 of the uploaded PDF**. It is observed that our proposal is more favorable and recovers more texture details than other compared methods.\n\n*Q1-5: Make consistent use of symbols. For example, rs and us in algorithm 1.*\n\nA1-5: Thanks for your careful comments. We will revise it in the new version.\n\n*Q1-6: Do not use the same notations for different quantities. For example, N, Nk and Ni in Eqn 2 and algorithm 1.*\n\nA1-6: Thanks for your careful comments. We have revised $N$ and $N_i$ as $T$ and $T_i$ in the new version."}, {"review_id": "LQcw3nOiDC", "reviewer": "Reviewer_tDMi", "summary": "Observing the issue of uneven distribution of image contents, the author proposed to utilize inverse data sampling to resolve the inherent unbalanced data bias. In the proposed BSPA method, SR model is alternatively updated with uniformly and inversely sampled image data. For the latter, only part of the trivial parameters identified by parameter attribution are updated and the selective probability progressively increases during training.", "questions": "1.\tWhat is the additional computational time cost of adding inverse sampling and parameter attribution?\n2.\tHow is the scalability of the integrated gradient for parameter attribution for larger models with more parameters?\n3.\tWhat is the shape of the distribution curve in Figure 1 if the MSE values are evaluated by the model trained with the proposed BSPA method? Will the distribution curve be more uniform or short-tailed?\n4.\tIt would be helpful to present more qualitative comparisons to demonstrate the visual effect.\n5.\tHow the data preprocessing is conducted (cropping with/without overlap?) and how many sub-images are produced?", "limitations": "Limitations of the proposed method are discussed in this paper. It would be helpful if over-fitting issue is evidenced by any experiment result and the cost of extra data preprocessing is quantitatively measured.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The data unbalanced problem this paper aiming to solve has been investigated in several prior literatures though from different perspectives. This work innovatively adopts the idea of resampling to address the issue, and conducts parameter attribution to balance the uniformly and inversely sampled data. The paper is organized well and reasonable experiments are provided to demonstrate the effectiveness of the proposed BSPA method.", "weaknesses": "1.\tA few notations in the paper are inconsistent and confusing. In Section 3.2, the superscript for “uniform sampling” is “rs”, for example, the uniformly sampled LR-HR patch pair is denoted as $(x_{lr}^{rs}, y_{hr}^{rs})$ in Algorithm 1. However, in section 3.3, the uniformly sampled patch pair is denoted as $(x_{lr}^{us}, y_{hr}^{us})$.\n2.\tThe experimental results of ablation studies presented in Tab. 1, 2, 3 only report the PSNR on the Set14 dataset which has a relatively small scale and the data distribution is not representative enough for assessing generalizability. More comprehensive and solid ablation analysis could be conducted by evaluating the models on larger testing dataset like the DIV2K validation set, BSD100, or Urban100.\n3.\tThere is a lack of ablation analysis on the number of classes $K$ which is simply set to be 10.\n4.\tThe model’s performance on FSRCNN model for x3 and x4 scale is worse than UDN method. It should be mentioned and discussed in the experiment section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response to Reviewer tDMi (denoted as R2)\n\n*Q2-1: A few notations in the paper are inconsistent and confusing.*\n\nA2-1: We are sorry about this. It should be $(x^{us},y^{us})$. We will revise it in the new version.\n\n*Q2-2: The experimental results of ablation studies presented in Tab. 1, 2, 3 only report the PSNR on the Set14 dataset.*\n\nA2-2: Please refer to **A1-1 of Reviewer zt8x (R1)**.\n\n*Q2-3: There is a lack of ablation analysis on the number of classes which is simply set to be 10.*\n\nA2-3: We provide the ablation analysis on the number of classes in the following table. Here, we compare 5, 10, and 20. It shows that the SR performance is not sensitive to the number of classes. Because the data in different classes can be sampled according to their proportion by means of inverse sampling so as to remedy the unbalanced data bias. In our experiments, we fix the number of classes to 10.\n\n|K |\t 5 |\t10 |\t20 |\t\n|------ |:-----:|:-----:|---------:|\n|Set14|\t28.358 |\t28.37 |\t28.380 |\n|B100|\t27.408 |\t27.40 |\t27.417 |\n|Urban100|\t25.531 |\t25.53 |\t25.544 |\n|DIV2K|\t30.128 |\t30.13 |\t30.143 |\n\n*Q2-4: The model’s performance on FSRCNN model for x3 and x4 scale is worse than UDN method.*\n\nA2-4: Thanks for your valuable suggestion. We would like to clarify it as follows:\n\n(1) FSRCNN is an extreme lightweight model, which has only 24.5K parameters and several convolutional layers. The model representation ability is restricted. Therefore, it cannot fit the training well no matter the easy data or the hard data by the limited model capacity, especially for large scaling factors. Although the bi-sampling strategy increases the data diversity, there lacks sufficient model capacity to fit. \n\n(2) UDN is a loss re-weighting method for solving the data imbalance problem. It provides a regularization constraint for shrinking the solution space. \n\nTherefore, the model’s performance on FSRCNN model for x3 and x4 scale is worse than UDN method.\n\n*Q2-5: What is the additional computational time cost of adding inverse sampling and parameter attribution?*\n\nA2-5: The additional computation cost includes adding the inverse sampling and parameter attribution. 1) For the inverse sampling, our proposal needs to implement extra data processing for classifying all sub-images into different groups. It spends nearly 2 hours for the inverse sampling preprocessing, which can be optimized through multi-threaded implementation in our future work.  2) For the parameter attribution, it spends an extra half hour compared to the baseline model. Although our proposal brings extra training costs, it would not introduce any expense during the inference phase.\n\n*Q2-6: How is the scalability of the integrated gradient for parameter attribution for larger models with more parameters?*\n\nA2-6: To demonstrate the  **scalability** and **generalization** of our method, we integrate it with several representative SR models with **different capacities and topologies**, i.e., FSRCNN (small), EDSR_baseline (medium), and RCAN (large), which are often used as the backbone for most SR models. It is observed from Table 4 of the manuscript that our BSPA has strong versatility and can be used in various SR models.\n\nHere, we integrate our BSPA with the NLSN model [ref2-1], which has nearly 44M parameters and powerful feature extraction ability. It shows that our method still improves the performance by the bi-sampling strategy.\n\n|model |\tSet14 |\tB100 |\tUrban100 | DIV2K|\n|------ |:-----:|:-----:|:-----:|---------:|\n|NLSN |\t\t28.514 |\t27.516 |\t25.811 |\t        30.268|\n|BSPA |\t\t28.551 |\t27.536 |\t25.962 |\t\t30.297|\n\n[ref2-1] Yiqun Mei, Yuchen Fan, Yuqian Zhou: Image Super-Resolution With Non-Local Sparse Attention. CVPR 2021\n\n*Q2-7: What is the shape of the distribution curve in Figure 1 if the MSE values are evaluated by the model trained with the proposed BSPA method?* \n\nA2-7: We present the distribution curve of Figure 1 in the **uploaded PDF**. Note that we adopt EDSR_baseline as the backbone and integrate the proposed BSPA on it. As Figure 2 in the PDF shows, it is observed that the histogram distribution of BSPA becomes **short-tailed**. The reason is that the SR performance on the tail data is improved, while the performance on the head data is kept. Therefore, it demonstrates that our BSPA is effective in obtaining a more compact representation.\n\n*Q2-8: It would be helpful to present more qualitative comparisons to demonstrate the visual effect.*\n\nA2-8: We have supplemented more visual results in **Figure 1 of the uploaded PDF**. It is observed that our proposal is more favorable and recovers more texture details than other compared methods.\n\n*Q2-9: How the data preprocessing is conducted?*\n\nA2-9: We follow ClassSR [ref2-2] for data preprocessing, which crops the whole image into multiple sub-images with sliding and overlap. For 2x, 3x, 4x SR, it produces 499875, 502200, 499875 sub-images, respectively. More details can refer to ClassSR.\n\n[ref2-2] Xiangtao Kong, Hengyuan Zhao, Yu Qiao, Chao Dong. ClassSR: A General Framework to Accelerate Super-Resolution Networks by Data Characteristic. CVPR 2021."}, {"review_id": "KdCBYXyN95", "reviewer": "Reviewer_4W2B", "summary": "This work focuses on studying the unbalanced distribution of the SR training data. The authors propose a bi-sampling strategy with parameter attribution. The bi-sampling consists of uniform sampling and inverse sampling, which pay more attention to hard samples. Moreover, integrated gradient is introduced to measuring the parameter importance and stopping gradient updating of significant filters.", "questions": "-What is wight-level IG and filter-level IG? Clear definitions are missing. \n\n-In ablation study, the best PSNR is obtained when the unbalanced factor is equal to 10 and 200 gets the worst results. Why is the imbalance factor set to 200 in the experimental setting in line 167?\n \n-What is the effect of different numbers of classes and intervals of alternate training? \n\n-What is the extra computation cost?", "limitations": "See the above comments", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Introducing parameter attribution into SR is a novel idea. Calculating the parameter importance for obtaining a compact model sounds reasonable. \n\n\n+ Extensive experiments are conducted to shows the advantages of the proposed method. The ablation study also helps to verify the effectiveness of the proposed techniques.", "weaknesses": "- The motivation is not so novel. The data unbalance of SR training data has been widely mentioned in previous works. Many sampling strategies have also been proposed to solve this problem.\n\n-According to the descriptions in ‘parameter refinement’, the most important parameters keep unchanged after a few epochs, which may cause underfitting. Authors should provide more theoretical support or conduct more experiments to verify its effectiveness.\n\n-Some descriptions are not very clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response to Reviewer 4W2B (denoted as R3)\n\n*Q3-1: The motivation is not so novel. The data unbalance of SR training data has been widely mentioned in previous works. Many sampling strategies have also been proposed to solve this problem.*\n\nA3-1: **Motivation**. Most of the existing SR models only consider that different image regions have different reconstruction difficulties, and then mainly focus on the delicate structure design (like attention mechanism) or complex regularization constraints. Few discusses the sampling strategies of training data and models this as a long-tail problem. \n\n**Idea**. The key idea behind the proposed Bi-Sampling Parameter Attribution method is to reconcile the unbalanced inherent data bias, namely the heavy-tailed distribution are visually more important than smooth areas. The newly developed technique with integrate gradient involves an inverse sampling strategy for enhancing the feature extraction ability of the model on the hard samples. \n\n**Experiment**. For fair comparison, we compare with the imbalanced data based SR methods on the same backbone model, SamplingAug (data re-sampling) and UDN (loss re-weighting). If there are other works related about this, please tell us. We are more than happy to include them for comparison.\n\n*Q3-2: According to the descriptions in ‘parameter refinement’, the most important parameters keep unchanged after a few epochs, which may cause underfitting. Authors should provide more theoretical support or conduct more experiments to verify its effectiveness.*\n\nA3-2: The reviewer might misunderstand this. In ‘parameter refinement’, we select a specific proportion of significant parameters according to the index sorting to keep unchanged, and only perform the gradient updating on the remaining trivial parameters for further refinement. It would **not** cause underfitting since we adopt the alternate training scheme, which is designed for distilling all parameters to obtain a more compact representation space. We expect the trivial parameters can also learn more effective information so we strive to distill them with other parameters fixed.\n\n*Q3-3: What is wight-level IG and filter-level IG? Clear definitions are missing.*\n\nA3-3: The weight-level IG and filter-level IG denote the parameter granularity in Eq.(10). Weight-level refers to calculate the integrated gradient for each weight. Filter-level refers to calculate the integrated gradient for each filter. Thanks for asking for the details that could be important to further improve our method. We will include it in the new version.\n\n*Q3-4: In ablation study, the best PSNR is obtained when the unbalanced factor is equal to 10 and 200 gets the worst results.\nWhy is the imbalance factor set to 200 in the experimental setting in line 167?*\n\nA3-4: We are sorry for the writing mistake. It should be 10 in line 167.\n\n*Q3-5: What is the effect of different numbers of classes and intervals of alternate training?*\n\nA3-5: We perform the ablation study on the number of classes and intervals of alternate training for EDSR_baseline ($4\\times$ SR) in the following tables. \n\n**For the classes**, we compare 5, 10, and 20. It shows that the SR performance is not sensitive to the number of classes.  Because the data in different classes can be sampled according to their proportion by means of inverse sampling so as to remedy the unbalanced data bias. In our experiments, we fix the number of classes to 10.\n\n|Classes |\t5 |\t10 |\t20 |\n|------ |:-----:|:-----:|--------:|\n|Set14 |\t28.358 |\t28.37 |\t28.380 |\n|B100 |\t27.408 |\t27.40 |\t27.417 |\n|Urban100 |\t25.531 |\t25.53 |\t25.544 |\n|DIV2K |\t30.128 |\t30.13 |\t30.143 |\n\n**For the intervals**, we compare 10, 50 and 100. It shows that the SR performance is better with smaller intervals. Because it is beneficial to obtain a compact representation during the integrated gradient process and more trivial parameters can be fully exploited. For a better tradeoff between performance and efficiency, we set the intervals to 50 in our experiments.\n\n|Interval |\t10 |\t50 |\t100 |\n|------ |:-----:|:-----:|--------:|\n|Set14 |\t28.417 |\t28.37 |\t28.303 |\n|B100 |\t27.449 |\t27.40 |\t27.373 |\n|Urban100 |\t25.670 |\t25.53 |\t25.420 |\n|DIV2K |\t30.225 |\t30.13 |\t30.062 |\n\n*Q3-6: What is the extra computation cost?*\n\nA3-6: The additional computation cost includes adding the inverse sampling and parameter attribution. 1) For the inverse sampling, our proposal needs to implement extra data processing for classifying all sub-images into different groups. It spends nearly 2 hours for the inverse sampling preprocessing, which can be optimized through multi-threaded implementation in our future work.  2) For the parameter attribution, it spends an extra half hour compared to the baseline model. Although our proposal brings extra training costs, it would not introduce any expense during the inference phase.\n"}, {"review_id": "TQMsnJPGX0", "reviewer": "Reviewer_xmJz", "summary": "The uniform sampling of the data, with flat regions occupying most of the training samples, can impair the accuracy of the reconstruction. Therefore, the authors enhance the model representation from the perspective of data sampling and propose a simple and effective Bi-Sampling Parameter Attribution (BSPA) method. Extensive experiments demonstrate that our method effectively promotes the performance of baseline models.", "questions": "1. The authors do not compare their method with the latest papers, nor do they integrate the proposed method on the latest models. What is the motivation of the authors for choosing several comparative methods in the paper? For example, what is the effectiveness of this method on a transformer-based framework?\n\n2. The authors mention that the SR data were divided into multiple groups, how was the parameter K determined here? How does this parameter affect the results?", "limitations": "I suggest that the authors could quantify the additional time that the proposed method brings to the training phase, so that the performance of the method can be more intuitively represented.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is interesting and improves the performance of existing methods from the perspective of data sampling.\n2. This paper clearly written and easy to understand.", "weaknesses": "1. The significance of the experimental results is not sufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response to Reviewer xmJz (denoted as R4)\n\n*Q4-1: The significance of the experimental results is not sufficient.*\n\nA4-1: We have already supplemented more experiments about integrating the proposed method on the latest model, the effect of parameter K, and the additional time costs. Thanks for asking for the details that could be important to further improve our method. We will include this in the final version.\n\n*Q4-2: The authors do not compare their method with the latest papers, nor do they integrate the proposed method on the latest models. What is the motivation of the authors for choosing several comparative methods in the paper? For example, what is the effectiveness of this method on a transformer-based framework?*\n\nA4-2: **The motivation of choosing several comparative methods**. To demonstrate the  **scalability** and **generalization** of our method, we integrate it with several representative SR models with **different capacities and topologies**, i.e., FSRCNN (small), EDSR_baseline (medium), and RCAN (large), which are often used as the backbone for most SR models. It is observed from Table 4 of the manuscript that our BSPA has strong versatility and can be used in various SR models.\n\n**The comparison with latest models**. 1) Most of the existing SR models mainly focus on the **delicate structure design** or **complex regularization constraints**, while few discuss the **distribution property of training data**. For fair comparison, we compare with the imbalanced data based SR methods on the same backbone model, SamplingAug (data re-sampling) and UDN (loss re-weighting). If there are other works related about this, please tell us. We are more than happy to include them for comparison. \n\n**The effectiveness on a transformer-based framework**. Thanks for the valuable suggestion. We continue to integrate our method on SwinIR [ref4-1] for $4\\times$ SR and report the PSNR (dB) in the following table. It is observed that our BSPA performs better overall. Therefore, our proposal can also be generalized to transformer-based models.\n\n|model |\tSet5 | \tSet14 | B100 | Urban100 | Manga109 |\tDIV2K |\n|------ |:-----:|:-----:|:-----:|:-----:|:-----:|--------:|\n|SwinIR |\t31.955 |\t28.445 |\t27.482 |\t25.692 |\t29.909 |\t30.265 |\n|SamplingAug |\t32.054 |\t28.544 |\t27.515 |\t25.900 |\t30.177 |\t30.314 |\n|UDN |\t32.025 |\t28.544 |\t27.514 |\t25.873 |\t30.079 |\t30.332 |\n|BSPA|\t32.212 |\t28.668 |\t27.599 |\t26.118 |\t30.497 |\t30.487 |\n\n[ref4-1] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, Radu Timofte: SwinIR: Image Restoration Using Swin Transformer. ICCVW 2021.\n\n*Q4-3: The authors mention that the SR data were divided into multiple groups, how was the parameter K determined here? How does this parameter affect the results?*\n\nA4-3:  We perform the ablation study on the number of groups in the following table. Here, we compare 5, 10, and 20. It shows that the SR performance is not sensitive to the number of classes.  Because the data in different classes can be sampled according to their proportion by means of inverse sampling so as to remedy the unbalanced data bias. In our experiments, we fix the number of classes to 10.\n\n|K |\t5 |\t10 |\t20 |\n|------ |:-----:|:-----:|--------:|\n|Set14 |\t28.358 |\t28.37 |\t28.380 |\n|B100 |\t27.408 |\t27.40 |\t27.417 |\n|Urban100 |\t25.531 |\t25.53 |\t25.544 |\n|DIV2K |\t30.128 |\t30.13 |\t30.143 |\n\n*Q4-4: suggest that the authors could quantify the additional time that the proposed method brings to the training phase,\nso that the performance of the method can be more intuitively represented.*\n\nA4-4: The additional computation cost includes adding the inverse sampling and parameter attribution. 1) For the inverse sampling, our proposal needs to implement extra data processing for classifying all sub-images into different groups. It spends nearly 2 hours for the inverse sampling preprocessing, which can be optimized through multi-threaded implementation in our future work.  2) For the parameter attribution, it spends an extra half hour compared to the baseline model. Although our proposal brings extra training costs, it would not introduce any expense during the inference phase.\n"}, {"review_id": "9JmzDdsra0", "reviewer": "Reviewer_ehLM", "summary": "The key idea behind the proposed Bi-Sampling 12 Parameter Attribution (BSPA) method is to reconcile the unbalanced inherent data bias, namely the heavy-tailed distribution are visually more important than smooth areas. Similar observation has been extensively observed in the literature of image restoration and super-resolution. The newly developed technique involves an inverse sampling strategy for enhancing the feature extraction ability of the model on the hard samples. Another technical contribution is to introduce integrated gradient for further improvement. Limited experimental results are reported to show marginal improvements over UDR (NeurIPS'2021).", "questions": "1. What is the point for Fig. 4 to make? It is not easy to see the difference between uniform and inverse SR in the current comparison (including both images and error maps).\n2. Fig. 3 (a) looks confusing. Not much difference between random sampling and inverse sampling, what does this figure attempt to show?\n3. The idea of \"better distilling the remaining trivial parameters\" is counter-intuitive. This could be an issue of literary presentation. If some parameters are not important (i.e., \"trivial\"), why do we strive to distill them better? \n4. Why do you coin the term \"bi-sampling\"? Does uniform sampling and inverse sampling carry equal importance? If not, you might want to consider a more appropriate title for this work. Bi-sampling is different from bilateral filtering where the domain and range can be viewed as a dual representation. I am not sure if a similar duality holds for sampling procedure.", "limitations": "Have the authors considered other recent competing approaches to SR such as diffusion model-based?  The baseline methods used in this paper do not seem to represent the current SOTA in SR.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation behind BSPA is clear and well explained. Although similar observation exists in the literature of SR, sampling-based approach for unbalanced data has not been considered for SR before (previous studies on long-tailed data distribution are mostly for high-level vision tasks such as image recognition).\n2. The proposed integrate gradient (IG) method is based on importance ranking and only updates the class of trivial parameters. This strategy is conceptually similar to the reweighting idea in the literature but easier to implement. \n3. The reported experimental results have shown descent improvement over previous benchmark method such as UDN [27].", "weaknesses": "1. The proposed bi-sampling framework in Fig. 2 seems to be based on heuristics. When only LR observation is available, it is unclear from the figure how to adapt the second step toward the prioritization of edge and texture regions. \n2. For inverse sampling, the procedure described in Eq. (4) lacks substantial justification. P(x)~1/x in Eq. (1) is difficult to follow (ref. [20] does not seem to directly suggest this formula).\n3. The idea of integrated gradient is inherited from previous work FAIG [38]. What is the novel contribution here? It seems that the derivation of Eqs. (9)-(10) is a direct consequence of Eq. (4) in the original FAIG paper. \n4. The reported experimental results do not support the claim of \"significantly boost the performance\" in the abstract. Both subjective and objective evaluation suggest that the performance of BSPA is comparable to that of other competing methods. With only one figure with small size images included, the superiority of BSPA is unconvincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response to Reviewer xmJz (denoted as R5)\n\n*Q5-1: The proposed bi-sampling framework in Fig. 2 seems to be based on heuristics.*\n\nA5-1:  It is **unreasonable** to say the proposed framework is heuristics. We aim to propose a simple yet effective bi-sampling parameter attribution method for accurate image SR from the data re-sampling view. Similar to most of the existing SR works, our proposal is a **supervised** method and the LR-HR pairs are available. The bi-sampling strategy takes full advantage of the paired samples, where the inverse sampling captures more hard samples to train the model for performing well on edge and texture regions. The unsupervised case is out of our scope.\n\n*Q5-2: For inverse sampling, the procedure described in Eq. (4) lacks substantial justification. P(x)~1/x in Eq. (1) is difficult to follow (ref. [20] does not seem to directly suggest this formula).*\n\nA5-2: The reviewer might misunderstand this. 1) Eq.(4) presents the sampling possibility of each class for inverse sampling, which is inversely proportional to the number of samples in each class. 2) Eq.(1) denotes the sampling probability is the same for each sample in uniform sampling. More generally, it refers to random sampling, which is widely used in the existing SR model, e.g., ref[20]. However, due to the uneven content distribution within the image, the uniform sampling would cause the data unbalanced problem, which motivates us to design such a bi-sampling parameter attribution method.\n\n*Q5-3: The idea of integrated gradient is inherited from previous work FAIG [38]. What is the novel contribution here? It seems that the derivation of Eqs. (9)-(10) is a direct consequence of Eq. (4) in the original FAIG paper.*\n\nA5-3: The reviewer might misunderstand this. We do not claim integrated gradient is our novel contribution.\n\n（1) The key idea behind the proposed BSPA is to reconcile the unbalanced inherent data bias, for enhancing the feature extraction ability of the model on the hard samples. Therefore, we need to find the significant parameters for different sampling ways to obtain a compact representation space. \n\n（2) To achieve this, we first formulate the parameter importance with Cauchy mean value theorem as Eq. (8). The integrated gradient merely serves as a tool for the attribution analysis by splitting the weight changes into a continuous path, which is usually used in numerical analysis and other level tasks. \n\n*Q5-4: The reported experimental results do not support the claim of \"significantly boost the performance\" in the abstract. Both subjective and objective evaluation suggest that the performance of BSPA is comparable to that of other competing methods. With only one figure with small size images included, the superiority of BSPA is unconvincing.*\n\nA5-4: The reviewer might misunderstand this. 1) In the abstract, we claim that our proposal can significantly boost the performance of **baseline methods** from the data re-sampling view. Besides, the quantitative and qualitative results also illustrate our proposal achieves superior or comparable performance against the compared methods. 2) For the subjective evaluation, we supplemented more visual results in **Figure 1 of the uploaded PDF**. It is observed that our proposal is more favorable and recovers more texture details than other compared methods. 3) For the objective evaluation, we also supplement more results on other datasets (please refer to **A1-1 of Reviewer zt8x**), which shows the effectiveness of our BSPA. We hope R5 can notice these results and analyses.\n\n*Q5-5: What is the point for Fig. 4 to make? It is not easy to see the difference between uniform and inverse SR in the current comparison (including both images and error maps).*\n\nA5-5: Fig.4 visualizes the SR results and their error maps with the GT of uniform sampling (uniform SR) and inverse sampling (inverse SR). It shows that inverse SR has a smaller error than the uniform SR on the texture region. Please zoom in for a better view, especially for the framed regions. Due to the limited space, we will provide more results in the new version.\n\n*Q5-6: Fig. 3 (a) looks confusing. Not much difference between random sampling and inverse sampling, what does this figure attempt to show?*\n\nA5-6: Fig.3 (a) illustrates that inverse sampling helps the feature extraction on hard texture regions, and performs better on tail hard classes against uniform sampling on the training dataset. The heavy-tailed distribution are visually more important than smooth areas, which is difficult for image SR. Since the scale of the ordinate is larger, you can zoom in for a better view. \n\n*Q5-7: The idea of \"better distilling the remaining trivial parameters\" is counter-intuitive. This could be an issue of literary\npresentation. If some parameters are not important (i.e., \"trivial\"), why do we strive to distill them better?*\n\nA5-7: The reviewer might misunderstand this. In 'parameter refinement', we select a specific proportion of significant parameters according to the index sorting to keep unchanged, and only perform the gradient updating on the remaining trivial parameters for further refinement. The alternate training scheme is designed for distilling all parameters to obtain a more compact representation space. We expect the trivial parameters can also learn more effective information so we strive to distill them with other parameters fixed.\n\n*Q5-8: Why do you coin the term \"bi-sampling\"? Does uniform sampling and inverse sampling carry equal importance? If not, you might want to consider a more appropriate title for this work. Bi-sampling is different from bilateral filtering where the domain and range can be viewed as a dual representation.*\n\nA5-8: The bi-sampling refers to the two kinds of sampling ways, i.e., uniform sampling and inverse sampling. In our proposal, uniform sampling and inverse sampling carry equal importance."}]

Author rebuttal:
In this uploaded PDF, we mainly provide more visual results on benchmark datasets and the histogram distribution.

Meta-review: Before the discussion and rebuttal, reviewers highlighted the following strengths and weaknesses

*Strengths:*
- The Bi-Sampling Parameter Attribution (BSPA) method is original, addressing the limitations of uniform sampling in SISR.
- Thorough analysis of data distribution in SISR training backed by empirical evidence.
- Extensive and organized experiments validate the proposed method's effectiveness.
- Novel introduction of parameter attribution into SR; calculating parameter importance for a compact model is reasonable.
- BSPA is clear and well-explained with descent improvement over benchmarks.

*Weaknesses:*
- Evaluations on Set14 with only 14 images limit robust conclusions; a call for studies on larger datasets like BSD100, Urban100, or DIV2k.
- Unclear benefits of splitting data patches into multiple groups.
- Limited qualitative results with a need for supplementary visual results.
- Inconsistent notations causing confusion.
- Lack of analysis on parameter 'K' and its effect.
- Some descriptions are unclear; parts of the model might cause underfitting.
- Questions arise on the novel contribution of the integrated gradient; seems derived from previous work [38].
- The reported experimental results don't "significantly boost performance", questioning the superiority of BSPA.

This is a borderline paper, with most reviewers leaning towards acceptance. Reviewer zt8x appreciates the addressed concerns and suggests further inclusion of discussions in the revised paper. Reviewer tDMi's concerns were resolved after reading the rebuttal and other reviews. Reviewer 4W2B, after pointing out related work, has increased the score to "borderline accept" post-rebuttal. Reviewer ehLM highlights the importance of visual quality in image SR studies, suggesting more subjective quality comparisons. The authors have responded by providing additional NIQE/LPIPS results.
Decision: Accept (poster)