Rethinking Imbalance in Image Super-Resolution for Efficient Inference
Authors: Wei Yu
Bowen Yang
Qinglin Liu
Jianing Li
Shengping Zhang
Xiangyang Ji
Venue: NeurIPS 2024 poster (2024)
Keywords: Efficient Image Super-Resolution; Dynamic Network;  Imbalanced Data Learning
Primary area: deep_learning_architectures
TL;DR: This paper explores the imbalance in the image SR task and proposes a plug-and-play Weight-Balancing framework (WBSR) based on a HES strategy and a BDLoss to achieve accurate and efficient inference.

Abstract: Existing super-resolution (SR) methods optimize all model weights equally using $\mathcal{L}_1$ or $\mathcal{L}_2$ losses by uniformly sampling image patches without considering dataset imbalances or parameter redundancy, which limits their performance. To address this, we formulate the image SR task as an imbalanced distribution transfer learning problem from a statistical probability perspective, proposing a plug-and-play Weight-Balancing framework (WBSR) to achieve balanced model learning without changing the original model structure and training data. Specifically, we develop a Hierarchical Equalization Sampling (HES) strategy to address data distribution imbalances, enabling better feature representation from texture-rich samples. To tackle model optimization imbalances, we propose a Balanced Diversity Loss (BDLoss) function, focusing on learning texture regions while disregarding redundant computations in smooth regions. After joint training of HES and BDLoss to rectify these imbalances, we present a gradient projection dynamic inference strategy to facilitate accurate and efficient inference. Extensive experiments across various models, datasets, and scale factors demonstrate that our method achieves comparable or superior performance to existing approaches with about 34\% reduction in computational cost.

Rethinking Imbalance in Image Super-Resolution for Efficient Inference
Wei Yu!, Bowen Yang 1 Qinglin Liu 1, Jianing Li Shengping Zhang 1,x Xiangyang Ji 2.*
School of Computer Science and Technology. Harbin Institute of Technology School of Information Science and Technology. Tsinghua University 2009030140stu, hit.edu. edu. cn, 2022211119@stu wew.tdt..tt....ee... cn, qlliu@hit edu. en, lijianing@pku. edu. 5.zhang@hit edu, en, xyji@tsinghua edu. on
Abstract
Existing super-resolution (SR) methods optimize all model weights equally using C1 or C2 losses by uniformly sampling image patches without considering dataset imbalances or parameter redundancy, which limits their performance. To address this issue, we formulate the image SR task as an imbalanced distribution transfer learning problem from a statistical probability perspective and propose plug- and-play Weight-Balancing framework (WBSR) for image SR to achieve balanced model learning without changing the original model structure or training data. Specifically, we develop a Hicrarchical Equalization Sampling (HES) strategy to address data distribution imbalances, enabling better feature representation from texture-rich samples. To tackle model optimization imbalances, we propose a Balanced Diversity Loss (BDLoss) function, focusing on learning texture regions while disregarding redundant computations in smooth regions. After joint training of HES and BDLoss rectify these imbalances, present gradient projection dynamic inference strategy to facilitate accurate and efficient reconstruction during inference. Extensive experiments across various models, datasets, and scale factors demonstrate that our method achieves comparable or superior performance to existing approaches with approximately 34% reduction in computational cost. The code is available at https:/ /etp://gitttt/tt com aipixel/WBSR.
Introduction
Image super-resolution (SR) aims to reconstruct high-resolution (HR) images with more details from low-resolution (LR) images. Recently, deep learning basedd image SR methods have made significant progress in reconstruction performance through deeper network models and large-scale training datasets, but these improvements place higher demands on both computing power and memory resources, thus requiring more efficient solutions. Therefore, various techniques such as pruning [36, 53, 55], quantization [34, 39, 5], knowledge distillation [18, 45), and designing lightweight architectures [17, 25, 40] have been widely researched to accelerate inference and meet the requirements of deployment inference resource-constrained platforms. However, these methods rely on static networks to process all input samples fairly, ignoring the different requirements of diverse samples for network computational cost, which limits the representation ability of the model.
In contrast, dynamic neural network based methods [4, 42, 54, 48] can dynamically adjust the network structure or parameters and reduce the average computational cost, becoming mainstream research focus recent years. These methods can adaptively allocate networks with suitable computational
Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Resample
Third
Vodiun
Hard
$1.8%
7553 Vake
(a) Data Distribution Imbalance
(b) Model Optimization Imbalance
(c) Weight Balancing
Figure 1: Illustration of (a) the data distribution from the widely used DIV2K [1] training set, (b) the framework.
reconstruction results of RCAN [51] model, and (c) the proposed
costs according to the content of the input samples during inference. Despite the advancements in these dynamic network solutions, practical applications are still hindered by two prevalent limitations:
Data Distribution Imbalance. Existing SR methods [51, 52, 4, 26] mostly use uniformly sampled LR-HR patch pairs instead of the entire image train models due the limitation of memory resources. However, they ignore the underlying fact that patch contents in images exhibit imbalanced distributions (i.e., the abundant easily reconstructed smooth flat patches and rare hardly reconstructed edge texture patches), resulting in inherent data bias. Figure 1 (a) shows that the number proportion of easy flat patches (48.8%) is much larger than that of hard textured patches (16.6%).
Model Optimization Imbalance. Current SR methods [27, 6, 12, 33] typically employ L1 or Le losses treat all patch areas and optimize each weight equally, which lacks reasonable optimization for their model training. Since the details lost in low-resolution images mainly exist in edges and texture locations, fewer computational resources of the model are required for those flat patches. Therefore, existing SR methods involve redundant calculations in flat areas, which leads to imbalanced inference performance where the model overfits in simple areas and underfits in complex ones and results in uneven distribution of model computational resources as shown in Figure (b). For the same image, the optimized RCAN [51] model exhibits overfitting in the smooth background area (green box, with error pixels accounting for only 0.08%), while it shows obvious underfitting in the textured foreground area (red box, with error pixels accounting for up to 52%).
Overall, these prevalent imbalance problems of data distribution and model optimization in the real world limit the performance current image SR algorithms. As motivated, although this imbalance is a well-known observation in the classification task [3. 46, 21J. we formulate the image SR task as an imbalanced distribution transfer learning problem from statistical probability perspective. To mitigate the gap, we propose a plug-and-play weight-balancing framework, dubbed WBSR, to achieve balanced model learning without additional computation costs, which improves the restoration effect and inference efficiency of models without changing the original model structure and training data, as shown in Figure (c). Specifically, address the imbalance problem of data distribution, we develop a Hierarchical Equalization Sampling (HES) strategy, enabling better feature representation from texture-rich samples to mitigate data biases. Then, to solve the imbalance problem of model optimization, we propose a Balanced Diversity Loss (BDLoss) function, focusing on learning texture areas while disregarding redundant computations in smooth areas. After joint training of HES and BDLoss within WBSR to rectify these imbalances, we present gradient projection dynamic inference strategy to facilitate accurate and efficient inference.
In summary, we make the following three key contributions: (1) This paper is the first attempt to explore the imbalance in the image super-resolution field and gives reasonable analysis from perspective of probability statistics, i.e., the imbalance of data distribution and model optimization limits the algorithm performance. (2) We propose a plug-and
weight- brightt-allncing framework dubbed WBSR upon HES and BDLoss to achieve balance training without additional computation costs, which improves the restoration effect and inference efficiency of models without changing the original model structure and training data. (3) Extensive experiments across various models,

---

datasets, and scale factors demonstrate that our achieves comparable or superior performance to existing methods with less computational cost.
2 Related Work
2.1 Deep Imbalanced Learning
Deep imbalanced learning has attracted widespread attention due the imbalanced data distribution caused by the difficulty of data acquisition in practical applications [20, 47]. The data imbalance problem presents significant challenge in deep learning, when some classes have fewer samples than others, resulting in poorer model prediction performance for the minority class. Previous imbalanced learning methods [43, 50] have mainly studied data resampling techniques solve this problem. For example, over-sampling minority classes [30, 3] and under-sampling common classes [2, 44]. However, oversampling increases memory storage and training time, while under-sampling causes overfitting problems [10, 5, 29]. Recently, several works [41, 49] attempt to develop data resampling methods as data augmentation strategies for the image super-resolution task to compensate for the imbalance of training patches between different classes.
Another category of class- imbalanced learning methods is reweighting techniques. Recent reweight- ing methods assign weights to different classes [14, 7, 28] and training examples [15, 13, 38), which aim to modify their gradients make models balance. [19] process from a domain adaptation perspective and enhance classic between different class distributions using veta-learning methods. In contrast, these methods balance the data loss by reweighting each class instead of sampling to achieve a balanced data distribution.
learning by explicitly estimating the differences
2.2 Dynamic Network for Efficient Image Super-Resolutee
Recent researches address this problem with efficient dynamic network frameworks, which mostly adopt content-aware modules to dynamically send image patches to sub-networks with different complexities to accelerate model inference. ClassSR [23] combines classification and SR in a unified framework, which uses an additional class module to classify image patches into different classes, and then applics the subnets to perform SR on diffcrent classes. ARM [4] further adopts the validation set to build Edge-to-PSNRR lookup table by mapping edge scores of image patches to the performance of each sub-network to select appropriate subnets to further improve efficiency. PathRestore [48] introduces pathfinder to implement a multi-path CNN, which dynamically select appropriate routes for different image areas according to the difficulty of restoration. However, these techniques still have two key problems. One is the additional amount of parameters and calculations brought by the introduction of classifiers or selectors, and the other is the neglect of data and network imbalance that affects the performance of the model.
3 Theoretical Analysis
Let and denote LR and HR patches and L1 loss as example (Note that the theoretical applies to the L22,, the optimization object of SR task can be written as
min E (nss)aaa lly Ø111
(1)
where 9 = fo() represents the SR result estimated from LR x with SR model fe. 0 denotes the model parameters. Pdeta indicates the data distribution space. It aims to minimize all absolute errors between predicted images and ground-truth images from the whole data. From the natural assumption that the distribution of the training set imbalanced, whereas the independent testing set is balanced [11, 14, 10], so we set the training data and testing data are drawn from different joint data distributions, Ptrain (2, y) and Phat v). respectively. The conditional probability p( (yl)) the same in both training and testing sets due to the fixed downsampling degradation in the SR task.
From the probabilistic view, the prediction of the SR network is considered as the mean of noisy prediction distribution, which can be modeled as Gaussian distribution
ply|:; 2; 0) = (y; ÿ. of ooisee
where Po indicates the variance of the independently distributed error term. The prediction ý can be treated as the mean of noisy prediction distribution. Eq. 2 can be interpreted as the distribution
3

---

form of Eq. 1, corresponding to the maximized negative log- log-likelhhoo (NLL) loss in the regression of the prediction distribution. Consequently, the prediction model trained by L, actually captures the mean value of the entire solution space, i.e., the distribution of the training set.
Theorem (Distribution Transformation). Considering the discordance between Perain (y) (x) and Poally (r) attributable to the distribution shift. Given the identical conditional probability p(rly) across both the training and testing sets, we leverage the Bayes rule p(y(z) x p(xly) p(y) to establish the relationship through variable substitution as follows
Pbai(y)z) Phat (x)
Ptrain (y)z) Pirain (y)
(3)
Pbally) Ptrain
This theorem reveals that the existence of imbalance issues stems from the direct proportionality between Purain (y/x:) and Perain (y) with ratio of Pbat(a)
(a)ain(2) When a specific type of patch sample is infrequently present the training set, i.e., when Pirsin (y) is low, the value of Perain (y|æz) decrease as well, which results in a decrease in the accuracy of predictions. As a consequence, the trained SR model tends to underestimate the occurrence of rare patches during prediction. Meanwhile, considering that the integral of Perain (yrain(ylz)) equals 1, we can obtain
Ptrain (train(yl(r)
Perain 2)) = Jy Ptrain (1 (x)dy
(4)
where Y denotes the entire training sample space. Then, substitute Eq. 3 into Eq. 4 to model the relationship between the two distributions through explicit distribution transformation
Pbal (3) (r) Purain (v)
Purain (+) Jy Poal (3/ (z) Ptrain (y' 'dy'
(5)
where y' denotes the integral variable. Diverging from previous works that focus on modeling Pusin (ylar) our objective estimate (bal(y)z)) (z) for achieving balanced prediction on the testing set. The detailed proof is available in the supplementary materials. The aforementioned theory proves that the imbalanced model optimization caused by imbalanced data distribution and loss function is reasonable. Therefore, our approach aims to correct this imbalance without introducing additional datasets or computational costs.
4 Methodology
4.1
Weight-Balancing Training Framework
Based on the observed phenomenon and analysis, the imbalanced model optimization of image SR undoubtedly limits the reconstruction performance of the model, especially for rare hard texture patches. We consider attaining a robust model representation with balanced weights from the perspective of two aspects: data sampling and optimization function. Figure 2 (a) illustrates the training process of the proposed framework, dubbed WBSR, which consists of two main components: Hierarchical Equalization Sampling (HES) and Balanced Diversity Loss (BDLoss). Given input LR patches from the training set, we employ HES to sample batch of approximately balanced patches to optimize each subnet model with our BDLoss Lod. The overall optimization objective is
min E. (n.v)) Loaly Sing (z:)
(6)
where Smn represents the m-th subnet the supernet with parameters 9m We employ a divide-and- conquer optimization strategy to learn nearly balanced weights, minimizing the overall objective by ensuring that each individual subnet within the supernet is well-optimized. Each subnet with varying computational cost shares the weights of the supernet and is intended to handle image patches of different complexities, which does not introduce additional complexity that impedes the inference speed. In the following, we describe the details of our HES and BDL loss, respectively.
4.1.1 Hierarchical Equalization Sampling
Without prior data classification, we propose a simple yet effective Hierarchical Equalization Sampling (HES) strategy, which utilizes inherent gradient information of patches to perform sample-level

---

Hierarchical Equalization Sampling
Update
Balanced
Diversity Loss
Imbalanced
Balanced
Weight
Weight
Imbalanced LR Patches from training set
(a) Training Stage
HR Patches
Chop
Combine
Proj Map
Grad
LR Image
LR Patches
Dynamic Network
Gradient Projection Dynamic Inference
SR Image
SR Patches
(b) Testing Stage
Figure 2: Illustration of the proposed weight-balanciee framework. (a) The training stage combines hierarchical equalization sampling and balanced diversity loss to jointly train a supernet model with balanced weights. (b) The testing stage adopts the gradient projection dynamic inference with gradient projection map and multiple dynamic subnets for efficient inference.
sampling and class-level sampling of difficult and easy classes to achieve equalization between the abundant simple samples and rare difficult samples.
Sample-Level Sampling refers to uniformly sampling patches from the training dataset, Each sample sampled with equal probability during the training stage, whose probability is P: = N indicates the i-th samples. N denotes the total number of training patch samples. It ensures that the model learns stable initial weights early in training, capturing general features across different sample types.
Ceves-Level Sampling aims to assign a higher sampling probability to rare difficult samples. Unlike the image classification task where the number of categories is determined, samples in image SR are unclassified and the number is unknown. To address it. we calculate the gradient vectors online consisting of the mean and standard deviation of the gradient magnitude of the input samples in the horizontal and vertical directions, which assess the reconstruction difficulty of samples and then classify them using vector thresholds to obtain the sampling probability. The threshold for the k-th class is defined as follows
ks K 4 = 11 K R [1,
where K is the number of classes. t and ls represent the gradient threshold of the simplest and most difficult classes, respectively. The number of samples for the k-th class corresponds to the N samples whose gradient vectors fall within the range from 4 to fg. The sampling possibility Pk can be calculated by
E N Pk
8
N₂- 8k
where 8 E (0, 1) indicates the exponential factor to avoid overfitting simple data by reducing the number of samples. It enables the sampled batch training data containing samples from difficult classes, thereby achieving equalized data sampling.
The core concept the proposed hierarchical equalization sampling strategy to reconcile the data bias caused by the inherent imbalance, i.e., difficult samples are visually more important than smooth samples. During training and testing, the gradient vectors image patches can quickly exported using existing gradient operator [32]. Therefore, our HES method does not impose any additional computational burden and effectively leverages dataset information to enhance the model's feature representation capabilities for hard samples.
5

---

4.1.2 Balanced Diversity Loss
The commonly used L1 and L2 losses of previous methods treat all patches equally and perform gradient updates on each weight parameter, which ignores parameter redundancy and leads to overfitting simple patches and underfitting on rare hard patches. To achieve reasonable optimization of models for diversity patches, we propose a novel Balanced Diversity Loss (BDLoss) to learn approximate balanced model weights, which performs distribution transformation by exploiting the training distribution without additional data achieve balanced predictions. accordance with Theorem I, we first estimate the desired Pbal (3)zz) by minimizing the NLL loss
Phat <x; 0) Nyy D, nnoisal)
Definition 1.
To balance the uncertainty of model diversity predictions and avoid excessive optimiza- lion, our BDLoss is defined as the likelihood function
Lod = log Ptrain (3/ x 0) + 2|10|12
(10)
where log Perain (y; f) denotes the converted conditional probability aimed to obtain balanced model weights 0. || 1l2 indicates the L2 regularization function prevent model overfitting. represents regularization coefficient. Next, we derive the implementation of Lod based on Eq. 9
Pbal ( xx; 0) Ptrsin (y)
log Prrain (1) 0) = log Jy Phal(y/ (z, 0) Ptrain (y Jdy
(11)
= llg U, uoise D) + log Purain (y) log f
N (w''::. U, moisee Pirain ddy
where log Perain (y) is constant term that can be omitted. The first remaining term is probability form of L1 loss as Eq. 2. The last term of log Jy (yy: D, moiee Prrain (3/ )dy' indicates the key diversity balancing term that obeys Gaussian distribution, which involves the integral operation and necessitates finding a closed-form expression.
Building upon the design of previous classification tasks [15, 19, 35], we utilize the Gaussian Mixture Model (GMM) technique to represent the constant term
Purain (v) = Σ (N
(12)
where L denotes the number of Gaussian components. Ø, μl, indicate the weights, means and covariances of mullt-dimensionall GMM, respectively. As the multiplication of two Gaussian functions results in another unnormalized Gaussian, the diversity balancing term can be expressed as
J N(y: ú. noisee 1)
Σ p,
Σ Φish J NWv Pu. E,jdy
(13)
where 8s, n. and E. are the norms, means, and covariances of the resulting unnormalized Gaussian, respectively. Now, the integral of the balanced diversity term adheres to a Gaussian distribution and solved straightforwardly. so the BDLoss of Eq. 10 can be derivable as follows
Lsd log.N(y 0, o2 noise I) + log Σ de Ni;; P4: Eg + o2 oois I) \||0||2
(14)
4.2 Gradient Projection Dynamic Inference
Figure 2 (b) illustrates the testing process of our WBSR framework, we propose a gradient projection dynamic inference strategy to achieve dynamic balance efficiency and performance. adaptively allocates the subnet model without any increase in additional parameters by calculating the gradient projection map based on the input content.
Gradient Projection. We observe that patches with complex (simple) structures exhibit high (low) image gradient magnitude and do not suffer more (less) score degradation with SR scale changes. Following the approach described in Section 4.1.1, we calculate gradient vectors to measure the

---

complexity of the patch contents and construct a gradient projection map online to project the gradient vector of an image patch to the selection of each subnet model. At inference time, each patch can select suitable subnet upon its gradient vector. When low-resolution noise exists in image patches, the edge detection methods [37, 8] ignore the local complexity of the patch and result in missed detections, thereby erroneously categorizing the patch as a simple sample. We count the changes in gradient strength by calculating the standard deviation directly, when there is a large amount noise or texture changes of varying intensity in the local area of the patch, it can still be correctly assigned as a difficult sample. As shown in Figure 3, yellow boxes represent areas of local texture change, such the clouds in the previous row and the railings in the next row. It can be intuitively seen that our gradient projection method can accurately distinguish local smooth regions or textured regions and assign them to the corresponding small large subnets.
Dynamic Inference. To facilitate the deployment of the model across any hardware resources, our dynamic supernet contains multiple subnets by gradually shrinking the model calculation with structured iteration to dynamically adapt various computational and performance require- ments. During inference, we adopt the dynamic supernet to individually distribute image patches of K classes to M subnets to obtain better computational performance trade-off's. Given first calculate its gradient vector and derive its class i according the
new LR patch, threshold t. Then, the selected subnet for inference can be easily obtained by equally split- ting the gradient vector interval into total of M subintervals, which can be expressed as
R M K
77
(15)
C
where m E [1, .]] denotes the index of the selected subnet to reconstruct this LR patch. [.] indicates the ceiling function that tends to select the larger subnet. However, the larger subnet selection leads to better performance with heavier computation, we further consider selecting the inference sub- net under the limited computational resources C.
(K Elp
Figure 3: Visualizations of (a) the edge detection results, (b) the gradient magnitude results, and (c) the projected subnet selection. For ease of observation, signed subnets with small, medium, and large computational costs green, yellow, and red, respectively.
i arg min a k C. m
Crl -Cl (16)
visualize three as-
where mn indicates the selected opti- mal subnet under resource constraints. Cm denotes the computational cost of the m-th subnet. a is a hyperparameter that is utilized to strike balance between the computational cost and performance, where higher values prioritize improved performance, while lower values favor reduced computational overhead. Consequently, our WBSR framework can be flexibly adjusted to accommodate diverse application scenarios based on actual performance and hardware resource requirements.
5 Experiments
5.1 Experimental Details
Following the previous works [23, 4], we apply DIV2K [1] as the training dataset widely used for image SR, which includes 800 high-quality images with diverse contents and texture details. To verify the model performance under different image content distributions, four datasets are employed for model testing, including B 1100 [31], Urban100 [16], Test2K, and Test4K. Test2K and Test4K are downsampled from DIVSK [9]. For metrics, we adopt peak signal-to-nois. ratio (PSNR) and structural similarity (SSIM) to quantitatively evaluate all methods. Additionally, the FLOPs are calculated as the average results of all patches in the entire test dataset images.
Datasets and Metries.
Implementation Details. Our proposed WBSR can be easily incorporated into existing CNN-based SR networks to achieve efficient inference. SRResNet [24] and RCAN [51] are selected as two baselines in our experiments for a fair comparison, we conduct extensive experiments on four datasets of different SR scales to verify the effectiveness of our framework. During training, we set nine
7

---

Method
#Pramas (M)
PSNR?
8100 [31]
4FLOPs (G)
PSNR1
Uirha1000 [16]
#FLOPs (G)
PSNR1
Tes12K
#FLOPs (G)
PSNR1
TessKK
#FLOPs (G)
SRRosNet [24]
1.52
32.19
20.78 (100%)
32.11
20.78 (100%)
30.39
20.78 (100%)
31.90
20.78 (300%)
+ClasSSR [23]
3.12
31.68
(7175
$1.15
16.21 (78%))
50.24
14.13 (68%)
31.89
13:51 (655)
+ARM
1.52
31.49
16.21
31.16
90.26
15.59 (75%)
31:90
13.71 (00%)
+WBSR (Ours)
1.52
(39988
31,98
(649)
30.41
12:05 (5896)
32.02
12.68 (61%)
RCAN [50]
15.59
32.40
130.40 (1005))
32.33
(100%)
30.86
130.411008 (100%)
32:26
130.40 (100%)
+ClassSR [23]
30.10
31.85
91 (70%5)
51.72
103.00 (799))
30.79
83:46 (6496)
12:24
83.46 (6498)
+ARM [4]
15.59
31.89
(76%)
31.74
109.5 (84%)
30.80
105.62 (81%)
32:24
30 (75%)
(Ouns)
(743))
(58%)
32.37
(60%)
SRResNet [24]
1.52
27.34
5.19 (100%)
25.30
5.19 (100%)
26.19
5,19 (100%)
27.65
5.19 (100()))))
+ClassSR [23]
(749)
(8158)
(701))
3.30
+ARM [4]
1.52
26.53
4.34 (839%)
21.54
4.18 (86%))
26.21
3.76 (728)
27. 37.06
3.33 (61%)
x4
+WBSR (Ouns)
1.52
27.36
(77%))
25.32
4,36 (8496)
26.26
5.37 (655%)
27.75
3:22 (62%)
RCAN [51]
15.59
27.76
32,60
25.82
32.60 (10095)
26.39
32,60 (100%)
27.80
32,60 (300%)
+ClasSRR [23]
(80%)
39
21.22
(605)
+ARM [4]
(798)
26.70 (825))
25. (77%)
+WBSR (Ours)
15.59
27:75
(7790
25.81
27.01 (8396)
26.45
18:52 (5756)
27.94
19,40| 59%)
Table 1: Quantitative comparison results of our method and other SOTA methods on the GoPro and H2D datasets. The optimal and suboptimal results are highlighted.
subnets (i=9) with different parameters 0; in each supernet. For SRResNet, the widths and depths of the subnets are set ([36, 52, 64)) and ([4, 8, 16)), respectively. As for RCAN, the widths and depths of the subnets are configured as ([36, 52. 64]) and ([5, 10, 20J), respectively. The compared width adaptation algorithms [23, 4] also follow such model width configuration to ensure a fair comparison. All methods are implemented using PyTorch and trained on an NVIDIA GeForce RTX 3090 for 100 epochs with 16 batch sizes, where the first 70 epochs are sample-level sampling and the rest are class-level sampling. The former aims to maintain the original data distribution of the entire dataset, ensuring stable and comprehensive feature representation. The latter focuses on correcting the imbalance of dataset and enhancing the model's 's ability to represent difficult texture samples. The training times of SRResNet and RCAN are 25 and 28 GPU hours using a single GPU, respectively The training patch size is set to 128 x 128 and augmented by horizontal and vertical flipping to enhance its robustness. We utilize our Lad loss along with the Adam optimizer [22], setting 31 0.9 and B2 0.999. To adjust the learning rate, we apply cosine annealing learning strategy, starting with an initial learning rate of x 10 and decaying to 10-7
5.2 Comparison results
Table shows the quantitative performance of our approach coupled with various SR baselines in terms of the metrics, parameter number, and computational cost. ClassSR [23] with an additional classifier module has more parameters, resulting in additional computational and parameter costs. lookup table, which generates additional
ARM [4] adopts the validation set to build an Edge-to- inference time and parameter storage overhead. In comparison, our framework achieves superior performance with less computation (average 62%) than baselines, without incurring additional parameters or computational costs. When tested on unseen datasets such as B100 and Urban 100, which lie outside the trained distribution, the compared methods exhibit performance degradation due to overfitting to specific features of the original training dataset and lack of generalization ability to diverse data. In contrast, our method maintains comparable performance to the original model with lower computational costs (averaging 70%). benefiting from our balanced sampling and optimization strategies during training.
Figure 4 shows visual comparisons across four testing datasets. The SR images produced by ClassSR and ARM cxhibit structural blur and noise. In contrast, our method recovers more detailed information, resulting in better outcomes that are more faithful the HR ground truth.
5.3 Ablation Studies
To verify the effectiveness of our WBSR upon HES and BDLoss, we conduct ablation studies on Tesk2K and Test4K with sclae factor of x4, as shown in Table 2.
Effectiveness HES. During model training, we replace the original uniform sampling in baseline with our HES strategy. As the method "+HES" shown in Table 2, HES achieves a 0.05 dB average improvement in terms of PSNR compared with the baseline, which benefits from enhanced feature representations in hard texture patches.
8

---

ClassSR
ClasSR
ARM
B100
U1ben100 LR
Ouus
HR
Postlle
ClasSRR
ARM
ClassSR
ARM
A
Test2K LR
OuIs
HR
TestAK LR
OuBs
HR
Figure 4:
Qualitative comparison results of our method with other methods for x 4 SR on the four testing datasets. Please zoom in for details.
Table 2:
Ablation studies of our WBSR network with 100% FLOPs for inference. The optimal and suboptimal results are highlighted.
benchmarks of SR. indicates using the whole
TestK [9]
Test4K [9]
Method (x4)
PSNR
SSIM1
#FLOPs (G)
PSNR
SSIM
#FLOPs (G)
SRResNet [24]
26.19
0.7624
5.19 (100%)
27.65
0.7966
5.19 (100%)
+HES
26.24
0.7665
3.58 (69%)
27.71
0.7986
3.43 (66%)
+Lua
26.21
0.7658
3.58 (69%)
27.70
0.7984
3,43 (66%)
+WBSR
26.26
0.7673
3.37 (65%)
27.73
0.7993
3.22 (62%)
+WBSR1
26.38
0.7684
5.19 (100%)
27.80
0.8026
5.19 (100%)
RCAN [51]
26.39
0.7706
32.60 (100%)
27.89
0.8058
32.60 (100%)
+HES
26.43
0.7748
20.86 (64%)
27.92
0.8086
19.89 (61 %)
+Lad
26.42
0.7746
20.86 (64%)
27.91
0.8077
19.89 (61 %)
+WBSR
26.45
0.7755
18.52 (57%)
27.94
0.8 0.8106
19.40 (59%)
+WBSR
26.51
0.7756
32.60 (100%)
28.10
0.8138
32.60 (100%)
Furthermore, we conduct additional experiments to compare our HES with existing sampling works [41, 49, 29] in Table 3. It shows that our HES outperforms the previous best sampling method BSPA of an average of 0. in terms of PSNR and demonstrates the superiority and gener- alization capabilities of our HES. In "+WBSR*" 0.18 dB by integrating our HES with our BDLoss. HES first performs more sample-level sampling to learn generalized feature representations followed by fewer selective class-level sampling focus on texture-rich regions to correct sample bias with stable learning and prevent the model's ''s overfitting, which mitigates the model oscillation and addresses the overfitting problem. Furthermore, HES achieves balanced stable training with our BDLoss for diverse samples in each training step, which solves the instability and training bias issues.
can achieve greater performance gains of
Effectiveness of BDLoss. To demonstrate the effect of BDLoss, we train the SR model using uniform sampling and replace only the C1 loss with Lta. As shown for method "+Coa" in Table 2, the PSNR

---

Table 3: Quantitative comparison results of our method with other sampling strategies.
B100 [31]
Urban 100 [16]
Method
PSNRT
SSIM
#FLOPs (G)
PSNR
SSIM
#FL.Ps (G)
RCAN
27.40
0.7306
32.60 (100%)
25.54
0.7684
32.60 (100%)
+BSPA [29]
27.54
0.7348
32.60 (100%)
26.02
0.7839
32.60 (100%)
+SamplingAug [41]
27.47
0.7323
32.60 (100%)
25.80
0.7771
32.60 (100%)
+DDA [49]
27.51
32.60 (100%)
25.89
32.60 (100%)
+HES
27.73
0.7388
32.60 (100%)
26.04
0.7863
32.60 ( (100%)
+WBSR
27.81
0.7402
32.60 (100%)
26.10
0.7889
32.60 (100%)
+WBSR
27.77
0.7391
26.41 (81%)
26.03
0.7850
29.67 (91%)
improved by balanced training is 0.06 dB with an average 65% computing cost compared to the baseline model, which demonstrates the superiority of our BDLoss.
Effectiveness of Joint Training. When applying joint training of HES and BDLoss within WBSR to the baseline network, we can further improve the PSNR and SSIM results by 0.13 dB and 0.0043, respectively, which achieve an overall performance improvement with average 66% calculation. Furthermore, to fully demonstrate the effectiveness of our WBSR, we adopt the weight-balancing framework to retrain the full baseline model instead of the dynamic supernet model. can be seen from the *WBSRR" "method of Table 2, our WBSR with computational cost comparable to baseline obtains average PSNR and SSIM gains of 0.33 dB and 0. 0.0078, respectively. Models trained using our WBSR show consistent performance improvements that are not affected by the skewness of the training sample distribution. In Figure 5, the SR performance on rare samples obtains gains, while the performance abundant samples remains the same slightly decreases, which proves that our bright-balancing strategy not only enhances the learning of texture areas and reduces redundant computation of flat areas. Additional experimental results placed in supplementary materials.
6 Conclusion
In this paper, we rethink the imbalance prob- lem in image SR from statistical probability perspective and propose a plug-and-play Weight- Balancing framework (WBSR) achieve bal- anced model learning without changing the orig- inal model structure and training data. Specif- ically, to tackle the imbalance problem of data distribution, propose a Hierarchical Equal- ization Sampling strategy (HES) to enhance the model's capability to extract features from dif- ficult samples to mitigate inherent data biases. Then, to solve the imbalance problem of model optimization, we propose a Balanced Diversity Loss (BDLoss) function focus on learning tex- ture regions and ignore redundant computations in those smooth regions. After joint training of HES and BDLoss, our WBSR rectifies the imbalance to achieve accurate and efficient inference via gradient projection dynamic inference strategy. Extensive qualitative and quantitative experiments across various models, datasets, and scaling factors demonstrate that our method achieves comparable or superior performance to existing approaches with less computational cost.
Cangraee
Figure 5: Illustaaioon of the gain of our weight- balancing framework relative to the baseline model and its weight rectification diagram.
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grants 62272134 and 62072141, in part by the National Science and Technology Major Project under Grant 202IZD01 10901.
10

---

References
[1] Eirikur Agustsson and Radu Timofte. NITRE 2017 challenge on single image super-resolution:
Dataset and study. In CVPRW, pages 1122-1131, 2017. 2,
[2] Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning?
In International conference on machine learning, pages 872-881 PMLR, 2019. 3
[3] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-avaaa margin loss. Advances in neural information processing systems, 32, 2019. 2,
[4] Bohong Chen, Mingbao Lin, Kekai Sheng, Mengdan Zhang, Peixian Chen, Ke Li, Liujuan Cao, and Rongrong Ji. Arm: Any-time super-resolution method. In ECCV, pages 254-270, 2022.
2, 3, 7, 8, 17
[5] Ting-An Chen, De-Nian Yang, and Ming-Syan Chen. Climbqq Class imbalanced quantization enabling robustness on efficient inferences, Advances in Neural Information Processing Systems, 35:37134-37144 2022. 1, 3
[6] Ki Scok Chung and Changwoo Lee. Gram: Gradient rescaling attention model for data uncertainty estimation in single image super resolution. In 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA), 2019. 2
[7] Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongic. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9268-9277 2019. 3
[8] Faming Fang, Juncheng Li, and Tieyong Zeng. Soft-edge assisted network for single image super-resolution. IEEE Transactions on Image Processing, 29:4656-4668 2020. 7
[9] Shuhang Gu, Andreas Lugmayr, Martin Danelljan, Manuel Fritsche, Julien Lamour, and Radu Timofte. Div8k: Diverse 8k resolution image dataset. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCWW), pages 3512-3311 --16. IEEE, 2019. 7, 8, 9, 18
[10] Guo Haixiang, Li Yijing, Jennifer Shang, Gu Mingyun, Huang Yuanyue, and Gong Bing. Learning from class-inbalanced data: Review of methods and applications. Expert systems with applications, 73:220-239, 2017.
[11] Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions knowledge and data engineering, 1263))222612844 2009. 3
[12] Xiangyu He and Jian Cheng. Revisiting 11 loss in super-resolution: probabilistic view and beyond. arXiv preprint WrXii:2201.1008, 2022. 2
[13] Youngkyu Hong. Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang. Disentangling label distribution for long-tailed visual recognition. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 6626-6636, 2021. 3
[14] Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalanced classification. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5375-5384, 2016. 3
(15) Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Deep imbalanced learning for face recognition and attribute prediction. IEEE transactions on pattern analysis and machine intelligence, 42(1 (1 1 1):2781-2794 2019. 3,6
[16] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In CVPR, pages 5197-5206, 2015. 8, 10.
[17] Zheng Hui, Xinbo Gao, Yunchu Yang, and Xiumei Wang. Lightweight image super-resolution with information multi-distillation network. In Proceedings of the 27th acm international conference multimedia, pages 2024-2032, 2019. 1

---

[18] Zheng Hui, Xiumei Wang, and Xinbo Gao. Fast and accurate single image super- reper-resolution via information distillation network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 723-731, 2018.
[19] Muhammad Abdullah Jamal, Matthew Brown, Ming- Hing-Hsuan Yang, Liqiang Wang, and Boqing Gong. Rethinking class-balanced methods for long-tailed visual recognition from a domain adaptation perspective. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 7610-7619, 2020. 3,6
[20] Bingyi Kang. Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for representation learning. International Conference on Learning Representations, 2020.
[21] Jaehyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via major- to-minor translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13896-13905, 2020.
[22] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6960 69800, 2014. 8
[23] Xiangtao Kong, Hengyuan Zhao, Yu Qiao, and Chao Dong. Classsr: A general framework accelerate super-resolution networks by data characteristic. In CVPR, pages 12016-12025, 2021. 3, 7, 8, 17
[24] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero. Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using generative adversarial network. In CVPR, pages 4681- 1- 4690, 2017. 9,9
[25] Jinmin Li, Tao Dai, Mingyan Zhu, Bin Chen, Zhi Wang, and Shu-Tao Xia. Fsr: A general frequency-oriented framework to accelerate image super-resolution networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1343-1350, 2023.
[26] Jingyun Liang. Jiezhang Cao, Guolei Sun, Kai Zhang, Lue Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Int. Conf. Comput. Vis., pages 1833-1844, 2021.
[27] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lec, Enhanced deep residual networks for single image super-resolution. In CVPRW, pages 133-1140, 2017. 2
[28] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 9980-2988, 2017.
[29] Xiaotong Luo, Yuan Xie, and Yanyun Qu Learning re-sampling methods with parameter attribution for image super-resolution. Advances in Neural Information Processing Systems, 36, 2024. 3, 9, 10
[30] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV), pages 181-196, 2018. 3
[31] David R. Martin, Charless C. Fowlkes, Doron Tal, and Jitendra Malik. A database of human seg- mented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV, pages 416-425, 2001. 7, 8, 10, 17
[32] Terrie E Moffitt, Louise Arsencault, Daniel Belsky, Nigel Dickson, Robert J Hancox, HonaLee Harrington, Renate Houts, Richie Poulton, Brent W Roberts, Stephen Ross, et al. A gradient of childhood self-control predicts health, wealth, and public safety. Proceedings of the national Academy of Sciences, (08(7):2693-261)) 20
[33] Qian Ning, Weisheng Dong, Xin Li, Jinjian Wu, and Guangming Shi. Uncertainty-drive loss for single image super-resolution. Advances in Neural Information Processing Systems, 34:16398- 16409, 2021. 2
12

---

[34] Haotong Qin, Yulun Zhang. Yifu Ding, Xianglong Liu, Martin Danelljan, Fisher Yu, et al. Quantsr: Accurate low-bit quantization for efficient image super-resolution Advances in Neural Information Processing Systems, 36, 2024.
[35] Jiawei Ren, Mingyuan Zhang, Cunjun Yu, and Ziwei Liu. Balanced mse for imbalanced visual regression. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 7926-7935, 2022. 6
[36] Xiangsheng Shi, Xuefei Ning, Lidong Guo, Tianchen Zhao, Enshu Liu, Yi Cai, Yuhan Dong, Huazhong Yang, and Yu Wang. Memory-oriented structural pruning for efficient image restora- tion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 2245 2253, 2023.
[37] Yu- Yu-Wing Tai, Shuaicheng Liu, Michael S Brown, and Stephen Lin. Super resolution using edge prior and single image detail synthesis. In CVPR, pages 2400-2407, San Francisco, USA, 2010.
7
[38] Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-tailed classification by keeping the good and removing the bad momentum causal effect. Advances in neural information processing systems, 33:1513-1524, 2020. 3
[39] Zhijun Tu, Jie Hu, Hanting Chen, and Yunhe Wang. Toward accurate post-training quantization for image super resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3856-5865 2023. 1
[40] Huan Wang, Yulun Zhang, Can Qin, Luc Van Gool, and Yun Fu. Global aligned structured sparsity learning for efficient image super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[41] Shizun Wang, Ming Lu, Kaixin Chen, Jiaming Liu, Xiaoqi Li, Ming Wu, al. Samplingaug: On the importance of patch sampling augmentation for single image super-resolution. arXiv preprint arXiv:2111.15185, 2021. 3, 9, 10
[42] Yan Wang, Shijie Zhao, Yi Liu, Junlin Li, and Li Zhang. Camixersr: Only details need more"" attention". arXiv preprint arXiv: 19289, 2024.
[43] Yu-Xiong Wang. Deva Ramanan, and Martial Hebert. Learning to model the tail. Advances in neural information processing systems, 30, 2017. 3
[44] Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, and Fan Yang. Crest: A class- rebalancing self-training framework for imbalanced coni-supervised learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10857-10866. 2021. 3
[45] Chengxing Xie, Xiaoming Zhang, Linze Li, Haiteng Meng, Tianlin Zhang, Tianrui Li, and Xiaole Zhao. Large kernel distillation network for efficient single image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1283-1299 1292, 2023.
[46] Yibo Yang, Shixiang Chen, Xiangtai Li, Liang Xie, Zhouchen Lin, and Dacheng Tao. Inducing neural collapse in imbalanced learning: Do really need learnable classifier the end of deep neural network? Advances in neural information processing systems, 35:37991-38002, 2022.
[47] Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class- -aas--imallnneed learning. Advances in neural information processing systems, 33:19290-19301 2020.
[48] Ke Yu, Xintao Wang, Chao Dong, Xiaoou Tang, and Chen Change Loy. Path-restore: Learning network path selection for image restoration. IEEE TPAMI, 44(10):008-77992, 2021. 1,
[49] Xinyi Zhang, Tao Dai, Bin Chen, and Shu-Tao Xia. Dda: A dynamic difficulty-wware data augmenter for image super-resolution. In 2023 International Joint Conference on Neural Networks (IJCNN,, pages 1-8. IEEE, 2023. 3, 10
13

---

[50] Yachao Zhang, Yuan Xie, Cuihua Li, Zongze Wu, and Yanyun Qu. Learning all-in collaborative multiview binary representation for clustering. IEEE Transactions on Neural Networks and Learning Systems, 2022. 3
[51] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention networks. In ECCV, volume 11211, pages 294-310, 2018. 2, 7, 8,
[52] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In CVPR, pages 2472-2481, 2018.
[53] Yulun Zhang, Huan Wang, Can Qin, and Yun Fu. Learning efficient image super-resolution net- works via structure-regullarize pruning. In International conference on learning representations, 2021.
[54] Yuxin Zhang, Minghao Lin, Xunchao Li, Han Liu, Guozhi Wang, Fei Chao, Shuai Ren, Yafei Wen, Xiaoxin Chen, and Rongrong Ji. Real-time image demoireing on mobile devices. arXiv preprint arXiv:3302.02184 2023. 1
[55] Yang Zhou, Yuda Song. Hui Qian, and Xin Du Classpruning: Speed up image restoration networks by dynamic n: M pruning. arXiv preprint arXiv:2211.05488 2022.
14

---

A Derivations and Proofs
In this supplementary material, we give a detailed derivation of Theorem from Eq. to Eq.
Piraiu (3)22)
Ptrain (1) (3)
Jv Perain (1)
Phat
Phaio(U)
Phasl (ax)
Ponl(3)
Punin (28)
Jy Poal Pbal(y'((3)
Puain
Phoio dyy
Piual (2z)
Poai(y)
(17)
Phatly (z)
Phain(y)
Poul(
Jy Poal (3/ (x:
Pbain (1²)) dy'
Pbal (y) xx)
Pirain (y)
Jy Poal ()) Ptrain (y )dy
Then, we give derivation from Eq. 13 to Eq. 14. Since in Eq. 13 indicates the norm of the product of multi Gaussians, its inherent characteristic also be delineated as Gaussian distribution
N p4, 2; 1 o
(18)
where N denotes the probability density function of a Gaussian distribution parameterized by mean vector 14, and covariance matrix Σ, + noise denotes the identity matrix, ensuring that the covariance matrix remains positive definite. Building upon this foundation, the weighted Gaussian terms can be expressed as
Σ 0.s; - Σ 0z N(D:p.. + o Tjdy
(19)
where d denotes the weight of individual Gaussian components. This weighting mechanism al- lows the model to flexibly learn balanced optimization of parameters based on the different data characteristics.
Subsequently, we substitute Eq. 19 and Eq. 12 into Eq. 13 to derive the pivotal diversity balancing term, which integrates the principles of predictive uncertainty with the underlying training distribution
log [ N/ ;0, o Perain (y')dy' 1 log Σ Φi
N u4, Ei + o noiso!!)
(20)
i=1
Finally, we integrate Eq. 20 into the our Lad formula from Eq. 11 to obtain the final loss function expression
End = Log.N(y; 02 moise I) + log oi No p4: E, + o² o + A|101|2
(21)
where balance weights are rectified through the joint optimization of two key terms with the loss: the diversity balancing term ensure balanced optimization, and the regularization term prevent overfitting.
B Ablation Studies
Visualization comparison of ablation. To further verify the feature representation ability of our BDLoss on hard texture patches, we also visualize the eтror map of SR results with GT images in Figure 6. As can be seen from the yellow box area, the error of the model trained with Lss on the texture area is significantly smaller than that of L1. indicating that our Lnd function improves the fitting accuracy challenging texture regions. In addition, the effectiveness of our gradient projection strategy can also be proved according to the subnet allocation map, which reasonably allocates large subnets (i.e., green masks) to complex texture areas. This demonstrates that our gradient projection strategy effectively enhance the model's ability to handle complex textures.
15

---

M
3
9
12
16
W&D
3&1
3&2
3&3
4&
4&4
PSNR
26.28
26.26
26.26
26.13
26.12
SSIM
0.7678
0.7675
0.7673
0.7657
0.7654
Table Influence of different numbers M of subnets. W represents the number of subnet widths, and D represents the number of subnet depths, constituting subnets with varying parameters and computational costs.
Eror map w/o BDLoss
Eroor map w/ BDLos
SubnetAllocation Map
Figure 6: Visual comparison of error map from SR model trained by Lı and our Lhd-
Analysis of the number M of subnets. We verify the impact of the number of subnets network inference performance Table 4. As the number of subnets increases, the inference effect will decrease certain extent because the fixed training epochs cause the extra subnets not be fully trained. Increasing the number of training epochs will solve this problem to certain extent. However, achieve trade-off between performance and efficiency, adopt total of nine subnet branches of three widths and three depths to process different image patches.
Analysis of the class l: of samples. To analyze the impact of the number of sample categories, we visualize the variation curves computational cost GFLOPs and performance PSNR under different K classes in Figure 7. When the K 5 value is small, the results of gradient vector projection of
K-5
K-10
26.20
K-15
26.10
PSNRR (dB)
26.00
25.90
0.5
1.3
1.5
2.0
2.5
3.0
3.5
GFLOPs
Figure 7: Effectiveness of different class values of samples.
different patches are too concentrated, which will lead inaccurate network division. Increasing the value of K results in more suitable subnet selection. Considering the memory limitation, set the
16

---

number of batches to 16 in per training step. and the number K of class at each training step in all experiments is 10 to achieve a better trade-off of performance and efficiency.
C More Qualitative Results
In Figures 11, present more visual comparison results of our WBSR and other SOTA meth- ods [23, 4] on different testing datasets at different scales. It can be seen that our algorithm can accurately reconstruct spatial structures and more texture details and other algorithms suffer from loss detail on difficult texture patches, which demonstrates the superiority of our algorithm.
B100 LR
ClassSR
ARM
Ous
HR
Figure 8:
Qualitative comparison results of on B100 [31] dataset between our method and other methods. Please zoom in for details.
Urtan100 LR
ClassRR
ARM
Ours
HR
Figure 9; Qualitative comparison results of x2 Urban100 [16] dataset between our method and other methods, Please zoom in for details.
17

---

Tesk2K LR
ClasSR
ARM
Ours
Figure 10:
Qualitative comparison results of x4 on Test2K [9] dataset between our method and other methods. Please zoom in for details.
Tesk4K LR
ClassSR
ARM
Ours
HR
Figure 11: Qualitative comparison results of x2 on Test4K [9] dataset between our method and other methods. Please zoom in for details.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discuss the limitations of the work performed by the authors.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

Justification:
This work provide the full set of assumptions and complete proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: This paper fully disclose all the information needed to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
Data and code of this paper will be published upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.cm cc/ mublic/guides Stt.ddde/CCoeeeumisssiiiiii wwwssssioooooooooo oniiooiopyyy ooiicylyyyy)oo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /ttpps.ccppublll//// ctcccc/pp.o wwtpccccpppccc.eec.c........ceet gwtp////d////eee// gttdge///o/oouuuiiii
onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: This paper specifies all the training and test details (e.g., data splits, hyperpa- rameters, how they were chosen, type of optimizer, etc.)
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: This paper report error bars suitably and correctly defined about the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: This paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ttp///eer.///////e wwwpurips.ccpublllleee cctppclicc/c/ccccccccccccccccccc c/Ethi wtw.iiiieeleeeeeeee
Answer: [Yes]
Justification:
The research conducted in the paper conform in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification; This paper discuss both potential positive societal impacts and negative societal impacts of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper not adopts pretrained language models, image generators, or scraped datasets.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "6DmgpQEcAY", "reviewer": "Reviewer_D78d", "summary": "To address imbalances and parameter redundancy problems, author proposed the Weight-Balancing framework (WBSR), which balances model learning without altering the original model structure or training data. The approach includes a Hierarchical Equalization Sampling (HES) strategy to handle data distribution imbalances and a Balanced Diversity Loss (BDLoss) function to optimize learning for texture-rich regions while reducing redundant computations in smooth areas. They introduce a gradient projection dynamic inference strategy for accurate and efficient inference.", "questions": "To enhance generalization, consider testing the proposed methods on a broader range of datasets, particularly those with varying characteristics (e.g., different textures, lighting conditions). This will help validate the robustness of the framework across diverse scenarios.\n\nAddress the potential complexity of the Weight-Balancing framework (WBSR) by providing a more user-friendly implementation or detailed guidelines. Including example codes or a simplified version could facilitate adoption by practitioners with varying levels of expertise.\n\n\nProvide well-defined interfaces for each module with clear input and output specifications. This will help users know what data to provide and what to expect in return. \nInclude a section that addresses common issues users might encounter, along with solutions and tips for resolving them.\n\nInclude additional examples showcasing how WBSR can be applied in various real-world scenarios, such as different types of images or specific applications (e.g., medical imaging, satellite imagery)", "limitations": "While the approach emphasizes learning from texture-rich regions, this focus may result in suboptimal performance on images with predominantly smooth areas, leading to a risk of overfitting to specific features while neglecting others.\n\nAlthough the framework is validated on various models and datasets, it may not have been tested extensively across all possible scenarios, raising concerns about its generalization capabilities in diverse applications.\n\nThe formulation of the super-resolution task as an imbalanced distribution transfer learning problem relies on certain statistical assumptions that may not hold in all real-world scenarios, potentially limiting its effectiveness", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The Weight-Balancing framework (WBSR) achieves balanced model learning without altering the original model structure or training data, addressing dataset imbalances and parameter redundancy effectively\n\n The Balanced Diversity Loss (BDLoss) function optimizes model learning by focusing on texture regions and minimizing redundant computations in smooth areas, leading to more efficient training\n\nThe method achieves comparable or superior performance to existing approaches with a 34% reduction in computational cost, demonstrating significant efficiency improvements.", "weaknesses": "The approach may still face scalability challenges when applied to extremely large datasets or high-resolution images, limiting its applicability in some real-world scenarios.\n\nWhile focusing on texture-rich regions can improve feature representation, it may also lead to overfitting, reducing generalization performance on smooth or less textured areas", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thanks for your positive evaluation and valuable suggestions.**\n\n**Questions:**\n\n***[Q1:]: To enhance generalization, consider testing the proposed methods on a broader range of datasets (e.g., different textures, lighting conditions).***\n\n***[A1:]***: Following this suggestion, to validate the robustness of our method across diverse scenarios, we conduct additional experiments on four datasets including an autonomous driving scene dataset (KITTI2015), two satellite remote sensing datasets (CAVE, NTIRE2020), and a low-light super-resolution dataset (RELLISUR). The experiential results are shown in Tables 1-3 of the attached PDF in the Author Rebuttal. As we can see, our method achieves good generalization and robustness across diverse scenarios.  Specifically, our method obtains an average PSNR improvement of 0.11 dB on the autonomous driving scene dataset, 0.27 dB and 0.25 dB on two satellite remote sensing datasets, and 0.1 dB on the low-light condition SR dataset, respectively. \n\n***[Q2:]: Address the potential complexity of the Weight-Balancing framework (WBSR) by providing a more user-friendly implementation or detailed guidelines..***\n\n***[A2:]***: Thanks for this suggestion. To facilitate the implementation of the WBSR framework, we will provide implementation details in the supplementary materials of the revised manuscript and we will also release the source codes of our method.\n\n***[Q3:]: Provide well-defined interfaces for each module with clear input and output specifications.***\n\n***[A3:]***:  In the revised manuscript, we will provide the well-defined interfaces for each module with clear input and output specifications. Additionally, we will include a section in the supplementary materials to discuss common issues that users might encounter. \n\n***[Q4:]: Include additional examples (e.g., medical imaging, satellite imagery).***\n   \n***[A4:]***: In Figure 1 of the attached PDF in the Author Rebuttal, we provide additional examples that show how WBSR  performs in medical MRI images and satellite hyperspectral images, which also have imbalanced sample classes.\nIn addition, we also provide additional quantitative results of our WBSR on a broader range of datasets in Tables 1-3 of the attached PDF, our method performs well in various real-world applications.\n\n**Limitations:**\n\n***[Q5:]: While the approach emphasizes learning from texture-rich regions, this focus may result in suboptimal performance on images with predominantly smooth areas, leading to a risk of overfitting to specific features while neglecting others.***\n\n***[A5:]***  Existing SR networks are apt to overfit smooth areas and underfit texture-rich areas due to their inherent lack of diverse and intricate features. Although our approach emphasizes learning from texture-rich regions, it will not overfit to texture-rich regions. \nBecause we only employ fewer selective class-level sampling to focus on texture-rich regions while more sample-level sampling to learn generalized feature representations in HES.  In addition, our BDLoss also encourages balanced learning of more diverse features by distribution transformation to avoid overfitting certain specific features, which also includes an L2 regularization function by reducing the complexity of model weights to prevent this. \n\n***[Q6:]: Raising concerns about its generalization capabilities in diverse applications.***\n\n***[A6:]***  Regarding the generalization capabilities of our method, please see our response to Q1.\n\n***[Q7:]: The formulation of the super-resolution task as an imbalanced distribution transfer learning problem relies on certain statistical assumptions that may not hold in all real-world scenarios, potentially limiting its effectiveness.***\n\n***[A7:]*** We assume the training sets are imbalanced and the testing sets are balanced due to the inherent characteristics of real-world datasets and the objectives of model evaluation. This imbalance problem assumption is common in machine learning and computer vision tasks, such as Imbalanced Learning[1,2], long-tail classification[3,4], and Anomaly Detection[5,6] tasks.\n\nSpecifically, training sets with a large number of samples follow a skewed Gaussian distribution, typically reflecting the natural distribution of real-world data, where some classes are overrepresented (major classes) and others are underrepresented (minor classes). \n\nTo achieve fair model evaluation, testing sets are usually considered to have a balanced uniform distribution with an equal number of samples in each class, as each sample is independently tested by the model.\nWhen the testing dataset contains a large number of simple smooth images and a small number of complex texture images, even if the texture images are poorly restored, the overall PSNR will still be high, which will affect the accuracy of objective measurement.\nThus, balanced testing sets ensure that evaluation metrics accurately reflect the model's ability to handle different types of features, thereby providing more reliable evaluation results.\n\nLastly, extensive experiments on diverse real-world datasets (shown in Tables 1-3) demonstrate that our methods are robust and effective across various real-world applications, supporting the validity of our assumptions and method.\n\n[1] He H, Garcia E A. Learning from imbalanced data[J]. IEEE TKDE 2009.\n\n[2] Haixiang G, Yijing L, et al. Learning from class-imbalanced data: Review of methods and applications[J]. ESWA, 2017.\n\n[3] Park S, Lim J, et al. Influence-balanced loss for imbalanced visual classification[C]. In CVPR 2021.\n﻿\n[4] Wang P, Han K, et al. Contrastive learning based hybrid networks for long-tailed image classification[C]. In CVPR 2021.\n\n[5] Zhang G, Yang Z, Wu J, et al. Dual-discriminative graph neural network for imbalanced graph-level anomaly detection[J]. In NeurIPS 2022.\n\n[6] Dragoi M, Burceanu E, Haller E, et al. AnoShift: A distribution shift benchmark for unsupervised anomaly detection[J]. In NeurIPS 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ONfsTMozsm", "reviewer": "Reviewer_FS1Z", "summary": "This paper proposes a Weight-Balancing framework to address the imbalanced learning issues in image super-resolution. Two categories of imbalance are involved, including data distribution imbalance and model optimization imbalance. Experiments demonstrate the effectiveness of the proposed method.", "questions": "Please see the weaknesses part.", "limitations": "Please see the weaknesses part.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The idea of addressing the imbalanced issues in image SR is reasonable, straightforward, and effective.\n2. The experiments demonstrate that the propsoed method enables computatation cost reduction but maintaining the pixel-domain accuracy.", "weaknesses": "1. Some grammatical errors should be corrected, e.g., \"are used to accelerate inference have been widely xxx\" in Line25-26. Please check the whole paper.\n2. I doubt the assumption that the distribution of the training set is imbalanced, whereas the independent testing set is balanced. Why is there this difference between the training and test sets？\n3. The description of the proposed HES sampling method is confusing. For example, what dose the \"classes\" in the images SR mean? How to determine the number of classes K？Please improve the clarity and add more technical details.\n4. From the probabilistic view, in my opinion, the prediction of SR network trained with L1 loss should correspond to the medium of a latent noisy prediction distribution. Therefore, I doubt the correctness of Eq. (2).\n5. In the ablation study, the impact of the dynamic supernet model should be isolated. For instance, I wonder the performance gain of RCAN+HES over the vanilla RCAN.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thanks for your positive evaluation and valuable suggestions.**\n\n**Weaknesses:**\n\n***[Q1:]: Some grammatical errors should be corrected, e.g., \"are used to accelerate inference have been widely xxx\" in Line25-26. Please check the whole paper.***\n\n***[A1:]***: Thanks for this suggestion. We will proofread the whole paper and correct the grammatical errors in the revised manuscript. \n\n***[Q2:]: I doubt the assumption that the distribution of the training set is imbalanced, whereas the independent testing set is balanced. Why is there this difference between the training and test sets?***\n\n***[A2:]***: We assume the training sets are imbalanced and the testing sets are balanced due to the inherent characteristics of real-world datasets and the objectives of model evaluation. This imbalance problem assumption is common in machine learning and computer vision tasks, such as Imbalanced Learning[1,2], long-tail classification[3,4], and Anomaly Detection[5,6] tasks.\n\nSpecifically, training sets with a large number of samples follow a skewed Gaussian distribution, typically reflecting the natural distribution of real-world data, where some classes are overrepresented (major classes) and others are underrepresented (minor classes). \n\nTo achieve fair model evaluation, testing sets are usually considered to have a balanced uniform distribution with an equal number of samples in each class, as each sample is independently tested by the model. \nWhen the testing dataset contains a large number of simple smooth images and a small number of complex texture images, even if the texture images are poorly restored, the overall PSNR will still be high, which will affect the accuracy of objective measurement.\nThus, balanced testing sets ensure that evaluation metrics accurately reflect the model's ability to handle different types of features, thereby providing more reliable evaluation results.\n\nLastly, extensive experiments on diverse real-world datasets in Tables 1-3 of the attached PDF in the Author Rebuttal demonstrate that our methods are robust and effective across various real-world applications, supporting the validity of our assumptions and method.\n\n[1] He H, Garcia E A. Learning from imbalanced data[J]. IEEE TKDE 2009.\n\n[2] Haixiang G, Yijing L, et al. Learning from class-imbalanced data: Review of methods and applications[J]. ESWA, 2017.\n\n[3] Park S, Lim J, Jeon Y, et al. Influence-balanced loss for imbalanced visual classification[C]. In CVPR 2021.\n﻿\n[4] Wang P, Han K, Wei X S, et al. Contrastive learning based hybrid networks for long-tailed image classification[C]. In CVPR 2021.\n\n[5] Zhang G, Yang Z, Wu J, et al. Dual-discriminative graph neural network for imbalanced graph-level anomaly detection[J]. In NeurIPS 2022.\n\n[6] Dragoi M, Burceanu E, Haller E, et al. AnoShift: A distribution shift benchmark for unsupervised anomaly detection[J].  In NeurIPS 2022.\n\n***[Q3:]: The description of the proposed HES sampling method is confusing. For example, what does the \"classes\" in the image SR mean? How to determine the number of classes K？ Please improve the clarity and add more technical details.***\n\n***[A3:]***:  To solve the imbalance problems of image SR, we classify the imbalanced training dataset into multiple classes.  Here, \"classes\" refer to the levels of restoration difficulties of image patches. For example, texture-rich patches are usually considered as samples from difficult classes, while smooth patches are considered as samples from easy classes.\n\nThe number of classes K is set manually to 10 in our method to ensure a balance between performance and computational cost. Different values of K can lead to varying restoration performance due to the allocation of different subnetworks for inference under computational cost limitations.  A higher K allows for more suitable subnetwork selection and potentially better inference performance, but it also requires more computational. \n\nFor a detailed analysis and experimental results, please refer to the supplementary material, \"Analysis of the class K of samples\", in the original manuscript.\n\n***[Q4:]: From the probabilistic view, in my opinion, the prediction of the SR network trained with L1 loss should correspond to the medium of a latent noisy prediction distribution. Therefore, I doubt the correctness of Eq. (2).***\n\n***[A4:]***:  Thanks for pointing out this error. Your observation is correct. From a probabilistic perspective, when a super-resolution (SR) network is trained using L1 loss, the prediction aligns with the median of the latent noisy prediction distribution. This is because L1 loss minimizes the sum of absolute differences between the predictions and the ground truth, which corresponds to estimating the median of the error distribution. In contrast, L2 loss minimizes the sum of squared differences, which is more closely associated with estimating the mean of a Gaussian distribution.\n\nIt is important to note that our Distribution Transformation theory remains applicable and valid whether L1 or L2 loss is employed. For a balanced uniform distribution, the median and the mean are equivalent. Therefore, both L1 and L2 losses provide different perspectives on the training set distribution without affecting the fundamental principles of our theory.\n\n***[Q5:]: In the ablation study, the impact of the dynamic supernet model should be isolated. For instance, I wonder the performance gain of RCAN+HES over the vanilla RCAN.***\n\n***[A5:]***: As shown in Table 4 of the attached PDF in the Author Rebuttal, we provide the results for this comparison by using the whole supernet with 100% FLOPs for inference, i.e., RCAN+HES. \nIt can be seen from the results that RCAN+HES improves performance by 0.11 dB compared to the vanilla RCAN, which demonstrates the effectiveness of our HES. \nFor a more detailed comparison of our method with other methods and ablation studies, please refer to Table 5 of the attached PDF in the Author Rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L4V7A9vTGH", "reviewer": "Reviewer_SYab", "summary": "This paper rethinks the imbalance problem in image SR and proposes a plug-and-play weight-balancing framework. It combines a Hierarchical Equalization Sampling strategy and a Balanced Diversity Loss to reduce computational cost while keeping or improving SR performance. Extensive experimental results demonstrate the effectiveness and superiority of the proposed method.", "questions": "1. Is other classification criteria tried, such as PSNR measurement like ClassSR? Actually, the mean and standard deviation of the gradient magnitude of the input samples may not well reflect the sample reconstruction difficulty.\n\n2. How about the generalization of the proposed methods on other restoration tasks, like image denoising, JPEG compression. It seems that the mean and standard deviation of the gradient magnitude would be affected by some noises or more complex degradation factors.\n\n3. There are some writing problems, such as\n- Line 24-25\n- Line 68", "limitations": "See the above comments", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Exploring Data imbalance for low-level tasks is meaningful.\n\n2. This paper is well-written and easy to understand.\n\n3. Extensive quantitative experiments are provided that helps to understand the significance of the whole proposed method.", "weaknesses": "1. The proposed method is of incremental contributions. This work is not the first to explore imbalance in image SR, which has been explored by the previous work, such as \n\n[1] Xiaotong Luo, Yuan Xie, Yanyun Qu: Learning Re-sampling Methods with Parameter Attribution for Image Super-resolution. In NeurIPS, 2023.\n \n2. The data sampling is also explored in image SR. The experiments lack the comparison and discussion with the latest related works, such as [1] and:\n\n[2] Shizun Wang, Ming Lu, Kaixin Chen, Jiaming Liu, Xiaoqi Li, Chuang Zhang, and Ming Wu. Samplingaug: On the importance of patch sampling augmentation for single image super-resolution. In BMVC, 2021.\n\n[3] Xinyi Zhang, Tao Dai, Bin Chen, and Shu-Tao Xia. DDA: A dynamic difficulty-aware data augmenter for image super-resolution. In IJCNN, 2023.\n\n3. The transformer-based SR backbones should be considered to compare in the experiment parts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thanks for your positive evaluation and valuable suggestions.**\n\n**Weaknesses:**\n\n***[Q1:]: The proposed method is of incremental contributions. This work is not the first to explore imbalance in image SR.***\n\n**[A1:]**: We want to clarify the contributions and novelties of our work from both the motivation and methodology aspects as follows.\n\n**Motivation**. Although previous works[1-3] have explored imbalance in image SR, they primarily focus on the imbalance in data distribution. In contrast, our method explores both data distribution imbalance and model optimization imbalance inspired by our theoretical analysis and experimental verification that the imbalance in model optimization is more critical for SR performance.\n\n**Methodology**. To solve both data distribution imbalance and model optimization imbalance, we propose a novel data sampling strategy, network optimization method, and efficient inference framework.\nRegarding **data sampling**,  we design the Hierarchical Equalization Sampling (HES) strategy to address data distribution imbalance with sample- and class-level sampling, enabling balanced generalization of feature representation from diverse samples.\nRegarding **network optimization**, we propose the Balanced Diversity Loss (BDLoss) to correct model optimization imbalances based on the distribution transformation theorem, which focuses on learning texture regions while disregarding redundant computations in smooth regions.\nRegarding the **efficient inference framework**,  we present a gradient projection dynamic Inference strategy to adaptively allocate the subnet model without any increase in additional parameters by calculating the gradient projection map.\n\n***[Q2:]: The data sampling is also explored in image SR. The experiments lack the comparison and discussion with the latest related works.***\n\n***[A2:]***:  The data sampling is explored in image SR [1,2,3] to address the data distribution imbalance.\nIn particular, the Dual-sampling technique[1] alternates sampling between uniform samples and hard samples, which causes unreliable learning due to the model to oscillate between two types of samples.\nThe greedy sampling approach[2] focuses on collecting more hard samples, which leads to overfitting specific samples and failing to generalize well across diverse data.\nThe dynamic sampling method[3] controls the sampling probability of each class by the relative loss, which can be influenced by various factors (e.g., noise) and lead to instability as the loss values fluctuate during training.\n\nDifferent the above approaches, our method designs a novel two-tier data sampling strategy (Hierarchical Equalization Sampling, HES) to address both the data distribution imbalance.\nHES first performs more sample-level sampling to learn generalized feature representations followed by fewer selective class-level sampling to focus on texture-rich regions to correct sample bias with stable learning and prevent the model's overfitting, which mitigates the model oscillation in [1] and addresses the overfitting problem in [2]. Furthermore, our HES achieves balanced stable training with our BDLoss for diverse samples in each training step, which solves the instability and training bias issues in [3].\n\nIn Table 5, we conduct additional experiments to compare our  HES with these works[1-3], which show that our HES outperforms the previous best sampling method BSPA of an average of 0.1dB in terms of PSNR and demonstrates the superiority and generalization capabilities of our HES.\nIn \"WBSR†\", we can achieve even greater performance gains of 0.18 dB by integrating our HES with our BDLoss. \n\n***[Q3:]: The transformer-based SR backbones should be considered.***\n\n***[A3:]***: Thanks for this suggestion. We conduct additional comparisons with transformer-based SR backbones. As shown in Table 6 of the attached PDF in the Author Rebuttal,  our approach achieves performance improvements on both natural and remote sensing datasets with average gains of 0.16 dB and 0.26 dB, respectively.\n\n**Questions:**\n\n***[Q4:]: Is other classification criteria tried, such as PSNR measurement like ClassSR?***\n\n***[A4:]***:  Actually, we tried the PSNR measurement of ClassSR to assess the sample reconstruction difficulty before the paper submission.  Since this approach necessitates the introduction of additional classification networks to categorize images, leading to increased training costs and added computational complexity. This contradicts our goal of achieving efficient inference. Therefore, we finally give up the PSNR measurement and adopt the gradient-based method.\n\nFollowing this suggestion, we present the comparison between  our gradient-based method and classifier-guided method (PSNR measurement) in Table 7 of the PDF attached in the author rebuttal. As we can see our gradient-based method achieves comparable performance to more complex classifier-guided methods while maintaining computational cost, which proves its effectiveness.\n\n***[Q5:]: How about the generalization of the proposed methods on other restoration tasks.***\n\n***[A5:]***  Thanks for your valuable suggestion. We conduct additional comparative experiments on various restoration tasks, including image denoising and JPEG Compression Artifact Removal (CAR) in Table 8.\nOur method achieve performance improvement of 0.07 dB compared to the denoising baseline FFDNet, and 0.11 dB compared to the CAR baseline RNAN. \nAlthough the interference of noise and blocking artifacts, these improvements are not as substantial as the improvement of 0.18 dB achieved in super-resolution baselines, our method remains effective even in complex degradations factors. Because our weight-balancing approach effectively mitigates the imbalance issues prevalent in these restoration fields.\n\n***[Q6:]: There are some writing problems.***\n\n***[A6:]***: Thanks for this suggestion. We will proofread the whole paper and correct the writing problems in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hna0S6hjXs", "reviewer": "Reviewer_Cw8X", "summary": "In this paper, the authors propose a novel framework called Weight-Balancing Super-Resolution (WBSR) that reformulates the SR task as an imbalanced distribution transfer learning problem.", "questions": "1. line 203 to 205 line of the article, the author introduces the reasoning method of gradient dynamic projection, but does not specifically illustrate the specific network used in this method within the article. If the structure of the network could be described more carefully, the article would be better.", "limitations": "2. The article has designed a data sampling technique, network training technique, and network inference technique to enhance super-resolution technology. However, in the experimental section, it seems that only a limited number of experiments were conducted to support the argument. If more comparative experiments could be added to substantiate the claims, the article would be significantly improved.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The key contributions of the paper are: 1. Introduction of a Hierarchical Equalization Sampling (HES) strategy to tackle data distribution imbalances by enhancing feature representation from texture-rich samples. 2. Development of a Balanced Diversity Loss (BDLoss) function that focuses on learning texture regions while ignoring redundant computations in smooth areas, aiming to correct model optimization imbalances. 3. Presentation of a gradient projection dynamic inference strategy for accurate and efficient inference without changing the original model structure or training data. 4. Extensive experimental results demonstrating that the proposed method achieves comparable or superior performance to existing approaches with a significant reduction in computational cost (approximately 34%).", "weaknesses": "1. line 203 to 205 line of the article, the author introduces the reasoning method of gradient dynamic projection, but does not specifically illustrate the specific network used in this method within the article. If the structure of the network could be described more carefully, the article would be better.\n\n\t2. The article has designed a data sampling technique, network training technique, and network inference technique to enhance super-resolution technology. However, in the experimental section, it seems that only a limited number of experiments were conducted to support the argument. If more comparative experiments could be added to substantiate the claims, the article would be significantly improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thanks for your positive evaluation and valuable suggestions.**\n\n**Weaknesses:**\n\n***[Q1:]: line 203 to 205 line of the article, the author introduces the reasoning method of gradient dynamic projection, but does not specifically illustrate the specific network used in this method within the article. If the structure of the network could be described more carefully, the article would be better.***\n\n***[A1:]***:  Thank you for this suggestion. Our gradient dynamic projection method directly computes the mean and standard deviation of the gradient magnitude from testing images for classification, allowing for efficient inference without the need for an additional specific network structure. Our framework is designed to be plug-and-play, enabling the use of different restoration backbone networks depending on the specific task. For a detailed explanation of the inference process, please refer to Figure 2(b) in the original manuscript.\n\n ***[Q2:]: The article has designed a data sampling technique, network training technique, and network inference technique to enhance super-resolution technology. However, in the experimental section, it seems that only a limited number of experiments were conducted to support the argument. If more comparative experiments could be added to substantiate the claims, the article would be significantly improved.***\n\n***[A2:]***:  Following your suggestion, we conduct additional comparative experiments to provide more robust support for our proposed data sampling, network training, and inference techniques. The results of these experiments are presented in Tables 4, 5, and 7 of the attached PDF in the Author Rebuttal. These experiments demonstrate the robustness and effectiveness of our approach, providing stronger evidence to support our claims. We will incorporate these additional experiments into the revised manuscript to enhance the overall evaluation of our methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all reviewers with their positive comments and valuable suggestions.  

**Reviewer D78d (Rating: 7 - Accept)**  gives positive comments on both our method and experimental results. The reviewer's concerns are on the applicability and generalization in some real-world scenarios. To response to reviewer's concerns, we have presented additional experiments on four datasets to validate the generalization of the proposed method . 

**Reviewer FS1Z (Rating: 5 - Borderline accept)**  has the main concerns on the presentation of the paper and ablation study. To response to reviewer's concerns, we have present additional ablation study  and will improve the presentation  in the revision. 

**Reviewer SYab (Rating: 3 - Reject)** has the main concerns on the contributions and additional experiments. To response to reviewer's concerns, we have clarified our contributions from both the motivation and methodology aspects. In addition, we also presented additional experiments to evaluation the generalization of the proposed method.

**Reviewer Cw8X (Rating: 7 - Accept)** acknowledges our contributions and performance improvement. The reviewer's concerns are on the details of the network structure and expectation of more experimental evaluation. To response to reviewer's concerns, we have explained the network structure and also present additional comparative experiments.

Our responses to individual comments  of each reviewer are posted in the rebuttal under each reviewer's report.  All the required experimental results are presented in the PDF attached in  this rebuttal. 

Specifically:
﻿
- **Table 1** , **Table 2**, and  **Table 3** show quantitative experiments of our method on four datasets with varying scenes.

- **Table 4** , **Table 5**, and  **Table 7** provide ablation studies of our method and quantitative comparison results with other related methods.
﻿
- **Table 6**  and  **Table 8** present quantitative experiments of our method on transformer-based SR backbones and other restoration backbones.

- **Figure 1** showcases representative visualization examples for our method on medical and remote sensing datasets.

For convenience, we highlight the figure and tables relevant to each reviewer's comments as follows:
﻿
- **Reviewer D78d**: Table 1, Table 2, Table 3, and Figure 1.
﻿
- **Reviewer FS1Z**: Table 4 and Table 5.
﻿
- **Reviewer SYab**: Table 5, Table 6. Table 7, and Table 8.

- **Reviewer Cw8X**: Table 4, Table 5, and Table 7.

We sincerely hope the rebuttal will address the reviewers' concerns and convince the reviewers to give more convincing decisions.

Meta-review: This paper receives 3 positive and 1 negative ratings. Most reviewers think that the idea about the imbalanced issue in image SR is reasonable and deserves to be explored. The proposed weight-balancing framework alleviates the imbalanced issue well and achieves good results. The main comparisons with other methods further show the improvements of the proposed one. One reviewer with a negative rating has a main concern about the contribution. After reading the author's rebuttal and other reviewers' comments, I believe this work has good motivation and contributions by exploring both imbalanced issue and model optimization imbalance. I would like to accept this paper. Of course, the authors are encouraged to further refine the paper based on the reviewers' comments.
Decision: Accept (poster)