Rethinking Imbalance in Image Super-Resolution for Efficient Inference

Authors: wei  yu
bowen  yang
qinglin  liu
jianing  li
shengping  zhang
ji  xiangyang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-08 10:32:49

Abstract: Existing super-resolution (SR) methods optimize all model weights equally using $\mathcal{L}_1$ or $\mathcal{L}_2$ losses by uniformly sampling image patches without considering dataset imbalances or parameter redundancy, which limits their performance. To address this, we formulate the image SR task as an imbalanced distribution transfer learning problem from a statistical probability perspective, proposing a plug-and-play Weight-Balancing framework (WBSR) to achieve balanced model learning without changing the original model structure and training data. Specifically, we develop a Hierarchical Equalization Sampling (HES) strategy to address data distribution imbalances, enabling better feature representation from texture-rich samples. To tackle model optimization imbalances, we propose a Balanced Diversity Loss (BDLoss) function, focusing on learning texture regions while disregarding redundant computations in smooth regions. After joint training of HES and BDLoss to rectify these imbalances, we present a gradient projection dynamic inference strategy to facilitate accurate and efficient inference. Extensive experiments across various models, datasets, and scale factors demonstrate that our method achieves comparable or superior performance to existing approaches with about 34\% reduction in computational cost.

Main paper: 
 introduction: 
 Image super-resolution (SR) aims to reconstruct high-resolution (HR) images with more details from low-resolution (LR) images. Recently, deep learning-based image SR methods have made significant progress in reconstruction performance through deeper network models and large-scale training datasets, but these improvements place higher demands on both computing power and memory resources, thus requiring more efficient solutions. Therefore, various techniques such as pruning [36, 53, 55] , quantization [34, 39, 5] , knowledge distillation [18, 45] , and designing lightweight architectures [17, 25, 40] have been widely researched to accelerate inference and meet the requirements of deployment inference on resource-constrained platforms. However, these methods rely on static networks to process all input samples fairly, ignoring the different requirements of diverse samples for network computational cost, which limits the representation ability of the model.In contrast, dynamic neural network based methods [4, 42, 54, 48] can dynamically adjust the network structure or parameters and reduce the average computational cost, becoming a mainstream research focus in recent years. These methods can adaptively allocate networks with suitable computational costs according to the content of the input samples during inference. Despite the advancements in these dynamic network solutions, practical applications are still hindered by two prevalent limitations:Data Distribution Imbalance. Existing SR methods [51, 52, 4, 26] mostly use uniformly sampled LR-HR patch pairs instead of the entire image to train models due to the limitation of memory resources. However, they ignore the underlying fact that patch contents in images exhibit imbalanced distributions (i.e., the abundant easily reconstructed smooth flat patches and rare hardly reconstructed edge texture patches), resulting in inherent data bias. Figure 1 (a) shows that the number proportion of easy flat patches (48.8%) is much larger than that of hard textured patches (16.6%).Model Optimization Imbalance. Current SR methods [27, 6, 12, 33] typically employ L 1 or L 2 losses to treat all patch areas and optimize each weight equally, which lacks reasonable optimization for their model training. Since the details lost in low-resolution images mainly exist in edges and texture locations, fewer computational resources of the model are required for those flat patches. Therefore, existing SR methods involve redundant calculations in flat areas, which leads to imbalanced inference performance where the model overfits in simple areas and underfits in complex ones and results in uneven distribution of model computational resources as shown in Figure 1 (b). For the same image, the optimized RCAN [51] model exhibits overfitting in the smooth background area (green box, with error pixels accounting for only 0.08%), while it shows obvious underfitting in the textured foreground area (red box, with error pixels accounting for up to 52%).Overall, these prevalent imbalance problems of data distribution and model optimization in the real world limit the performance of current image SR algorithms. As motivated, although this imbalance is a well-known observation in the classification task [3, 46, 21] , we formulate the image SR task as an imbalanced distribution transfer learning problem from a statistical probability perspective. To mitigate the gap, we propose a plug-and-play weight-balancing framework, dubbed WBSR, to achieve balanced model learning without additional computation costs, which improves the restoration effect and inference efficiency of models without changing the original model structure and training data, as shown in Figure 1 (c). Specifically, to address the imbalance problem of data distribution, we develop a Hierarchical Equalization Sampling (HES) strategy, enabling better feature representation from texture-rich samples to mitigate data biases. Then, to solve the imbalance problem of model optimization, we propose a Balanced Diversity Loss (BDLoss) function, focusing on learning texture areas while disregarding redundant computations in smooth areas. After joint training of HES and BDLoss within WBSR to rectify these imbalances, we present a gradient projection dynamic inference strategy to facilitate accurate and efficient inference.In summary, we make the following three key contributions: (1) This paper is the first attempt to explore the imbalance in the image super-resolution field and gives a reasonable analysis from a perspective of probability statistics, i.e., the imbalance of data distribution and model optimization limits the algorithm performance. 2 Related Work 
deep imbalanced learning: 
 Deep imbalanced learning has attracted widespread attention due to the imbalanced data distribution caused by the difficulty of data acquisition in practical applications [20, 47] . The data imbalance problem presents a significant challenge in deep learning, when some classes have fewer samples than others, resulting in poorer model prediction performance for the minority class. Previous imbalanced learning methods [43, 50] have mainly studied data resampling techniques to solve this problem. For example, over-sampling minority classes [30, 3] and under-sampling common classes [2, 44] . However, oversampling increases memory storage and training time, while under-sampling causes overfitting problems [10, 5, 29] . Recently, several works [41, 49] attempt to develop data resampling methods as data augmentation strategies for the image super-resolution task to compensate for the imbalance of training patches between different classes.Another category of class-imbalanced learning methods is reweighting techniques. Recent reweighting methods assign weights to different classes [14, 7, 28] and training examples [15, 13, 38] , which aim to modify their gradients to make models balance. [19] process from a domain adaptation perspective and enhance classic class-balanced learning by explicitly estimating the differences between different class distributions using meta-learning methods. In contrast, these methods balance the data loss by reweighting each class instead of sampling to achieve a balanced data distribution.Recent researches address this problem with efficient dynamic network frameworks, which mostly adopt content-aware modules to dynamically send image patches to sub-networks with different complexities to accelerate model inference. ClassSR [23] combines classification and SR in a unified framework, which uses an additional class module to classify image patches into different classes, and then applies the subnets to perform SR on different classes. ARM [4] further adopts the validation set to build an Edge-to-PSNR lookup table by mapping edge scores of image patches to the performance of each sub-network to select appropriate subnets to further improve efficiency. PathRestore [48] introduces a pathfinder to implement a multi-path CNN, which can dynamically select appropriate routes for different image areas according to the difficulty of restoration. However, these techniques still have two key problems. One is the additional amount of parameters and calculations brought by the introduction of classifiers or selectors, and the other is the neglect of data and network imbalance that affects the performance of the model.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The paper discuss the limitations of the work performed by the authors. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
theoretical analysis: 
 Let x and y denote LR and HR patches and L 1 loss as an example (Note that the theoretical applies to the L 2 ), the optimization object of SR task can be written asEQUATIONwhere ŷ = f θ (x) represents the SR result estimated from LR x with SR model f θ . θ denotes the model parameters. p data indicates the data distribution space. It aims to minimize all absolute errors between predicted images and ground-truth images from the whole data. From the natural assumption that the distribution of the training set is imbalanced, whereas the independent testing set is balanced [11, 14, 10] , so we set the training data and testing data are drawn from different joint data distributions, p train (x, y) and p bal (x, y), respectively. The conditional probability p(x|y) is the same in both training and testing sets due to the fixed downsampling degradation in the SR task.From the probabilistic view, the prediction ŷ of the SR network is considered as the mean of a noisy prediction distribution, which can be modeled as a Gaussian distributionEQUATIONwhere σ 2 noise indicates the variance of the independently distributed error term. The prediction ŷ can be treated as the mean of a noisy prediction distribution. Eq. 2 can be interpreted as the distribution form of Eq. 1, corresponding to the maximized negative log-likelihood (NLL) loss in the regression of the prediction distribution. Consequently, the prediction model trained by L 1 actually captures the mean value of the entire solution space, i.e., the distribution of the training set. Theorem 1 (Distribution Transformation). Considering the discordance between p train (y|x) and p bal (y|x) attributable to the distribution shift. Given the identical conditional probability p(x|y) across both the training and testing sets, we leverage the Bayes rule p(y|x) ∝ p(x|y) • p(y) to establish the relationship through variable substitution as followsEQUATIONThis theorem reveals that the existence of imbalance issues stems from the direct proportionality between p train (y|x) and p train (y) with a ratio of p bal (x) ptrain(x) . When a specific type of patch sample is infrequently present in the training set, i.e., when p train (y) is low, the value of p train (y|x) decrease as well, which results in a decrease in the accuracy of predictions. As a consequence, the trained SR model tends to underestimate the occurrence of rare patches during prediction. Meanwhile, considering that the integral of p train (y|x) equals 1, we can obtainEQUATIONwhere Y denotes the entire training sample space. Then, we substitute Eq. 3 into Eq. 4 to model the relationship between the two distributions through explicit distribution transformationEQUATIONwhere y ′ denotes the integral variable. Diverging from previous works that focus on modeling p train (y|x), our objective is to estimate p bal (y|x) for achieving balanced prediction on the testing set. The detailed proof is available in the supplementary materials. The aforementioned theory proves that the imbalanced model optimization caused by imbalanced data distribution and loss function is reasonable. Therefore, our approach aims to correct this imbalance without introducing additional datasets or computational costs.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes]• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
weight-balancing training framework: 
 Based on the observed phenomenon and analysis, the imbalanced model optimization of image SR undoubtedly limits the reconstruction performance of the model, especially for rare hard texture patches. We consider attaining a robust model representation with balanced weights from the perspective of two aspects: data sampling and optimization function. Figure 2 (a) illustrates the training process of the proposed framework, dubbed WBSR, which consists of two main components: Hierarchical Equalization Sampling (HES) and Balanced Diversity Loss (BDLoss). Given input LR patches from the training set, we employ HES to sample a batch of approximately balanced patches to optimize each subnet model with our BDLoss L bd . The overall optimization objective isEQUATIONwhere S m θ represents the m-th subnet in the supernet with parameters θ m . We employ a divide-andconquer optimization strategy to learn nearly balanced weights, minimizing the overall objective by ensuring that each individual subnet within the supernet is well-optimized. Each subnet with varying computational cost shares the weights of the supernet and is intended to handle image patches of different complexities, which does not introduce additional complexity that impedes the inference speed. In the following, we describe the details of our HES and BDLoss, respectively.Without prior data classification, we propose a simple yet effective Hierarchical Equalization Sampling (HES) strategy, which utilizes inherent gradient information of patches to perform sample-level Class-Level Sampling aims to assign a higher sampling probability to rare difficult samples. Unlike the image classification task where the number of categories is determined, samples in image SR are unclassified and the number is unknown. To address it, we calculate the gradient vectors online consisting of the mean and standard deviation of the gradient magnitude of the input samples in the horizontal and vertical directions, which assess the reconstruction difficulty of samples and then classify them using vector thresholds t to obtain the sampling probability. The threshold for the k-th class is defined as followsEQUATIONwhere K is the number of classes. t 1 and t K represent the gradient threshold of the simplest and most difficult classes, respectively. The number of samples for the k-th class corresponds to the N k samples whose gradient vectors fall within the range from t k-1 to t k . The sampling possibility P k can be calculated byP k = K j=1 1 Nj N k • δ k (8)where δ ∈ (0, 1) indicates the exponential factor to avoid overfitting simple data by reducing the number of samples. It enables the sampled batch training data containing samples from difficult classes, thereby achieving equalized data sampling.The core concept of the proposed hierarchical equalization sampling strategy is to reconcile the data bias caused by the inherent imbalance, i.e., difficult samples are visually more important than smooth samples. During training and testing, the gradient vectors of image patches can be quickly exported using existing gradient operator [32] . Therefore, our HES method does not impose any additional computational burden and effectively leverages dataset information to enhance the model's feature representation capabilities for hard samples.The commonly used L 1 and L 2 losses of previous methods treat all patches equally and perform gradient updates on each weight parameter, which ignores parameter redundancy and leads to overfitting on simple patches and underfitting on rare hard patches. where log p train (y) is constant term that can be omitted. The first remaining term is a probability form of L 1 loss as Eq. 2. The last term of log Y N (y ′ ; ŷ, σ 2 noise I) • p train (y ′ )dy ′ indicates the key diversity balancing term that obeys Gaussian distribution, which involves the integral operation and necessitates finding a closed-form expression.Building upon the design of previous classification tasks [15, 19, 35] , we utilize the Gaussian Mixture Model (GMM) technique to represent the constant termEQUATION)where L denotes the number of Gaussian components. ϕ, µ, σ indicate the weights, means and covariances of multi-dimensional GMM, respectively. As the multiplication of two Gaussian functions results in another unnormalized Gaussian, the diversity balancing term can be expressed asEQUATIONwhere s i , μ, and Σ are the norms, means, and covariances of the resulting unnormalized Gaussian, respectively. Now, the integral of the balanced diversity term adheres to a Gaussian distribution and is solved straightforwardly, so the BDLoss of Eq. 10 can be derivable as follows Gradient Projection. We observe that patches with complex (simple) structures exhibit high (low) image gradient magnitude and do not suffer more (less) score degradation with SR scale changes. Following the approach described in Section 4.1.1, we calculate gradient vectors to measure the complexity of the patch contents and construct a gradient projection map online to project the gradient vector of an image patch to the selection of each subnet model. At inference time, each patch can select a suitable subnet upon its gradient vector. When low-resolution noise exists in image patches, the edge detection methods [37, 8] ignore the local complexity of the patch and result in missed detections, thereby erroneously categorizing the patch as a simple sample. We count the changes in gradient strength by calculating the standard deviation directly, when there is a large amount of noise or texture changes of varying intensity in the local area of the patch, it can still be correctly assigned as a difficult sample. As shown in Figure 3 , yellow boxes represent areas of local texture change, such as the clouds in the previous row and the railings in the next row. It can be intuitively seen that our gradient projection method can accurately distinguish local smooth regions or textured regions and assign them to the corresponding small or large subnets.EQUATIONDynamic Inference. To facilitate the deployment of the model across any hardware resources, our dynamic supernet contains multiple subnets by gradually shrinking the model calculation with structured iteration to dynamically adapt various computational and performance requirements. During inference, we adopt the dynamic supernet to individually distribute image patches of K classes to M subnets to obtain better computational performance trade-offs. Given a new LR patch, we first calculate its gradient vector and derive its class k according to the threshold t. Then, the selected subnet for inference can be easily obtained by equally splitting the gradient vector interval into a total of M subintervals, which can be expressed as For ease of observation, we visualize three assigned subnets with small, medium, and large computational costs as green, yellow, and red, respectively.EQUATION)where m ∈ [1, M ] denotes the index of the selected subnet to reconstruct this LR patch. ⌈•⌉ indicates the ceiling function that tends to select the larger subnet. However, the larger subnet selection leads to better performance with heavier computation, we further consider selecting the inference subnet under the limited computational resources C t m = arg minEQUATION)where m indicates the selected optimal subnet under resource constraints. C m denotes the computational cost of the m-th subnet. α is a hyperparameter that is utilized to strike a balance between the computational cost and performance, where higher values prioritize improved performance, while lower values favor reduced computational overhead. Consequently, our WBSR framework can be flexibly adjusted to accommodate diverse application scenarios based on actual performance and hardware resource requirements.5 Experiments 
experimental details: 
 Datasets and Metrics. Following the previous works [23, 4] , we apply DIV2K [1] as the training dataset widely used for image SR, which includes 800 high-quality images with diverse contents and texture details. To verify the model performance under different image content distributions, four datasets are employed for model testing, including B100 [31] , Urban100 [16] , Test2K, and Test4K. Test2K and Test4K are downsampled from DIV8K [9] . For metrics, we adopt peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) to quantitatively evaluate all methods. Additionally, the FLOPs are calculated as the average results of all patches in the entire test dataset images.Implementation Details. Our proposed WBSR can be easily incorporated into existing CNN-based SR networks to achieve efficient inference. SRResNet [24] and RCAN [51] The training patch size is set to 128 × 128 and augmented by horizontal and vertical flipping to enhance its robustness. We utilize our L bd loss along with the Adam optimizer [22] , setting β 1 = 0.9 and β 2 = 0.999. To adjust the learning rate, we apply a cosine annealing learning strategy, starting with an initial learning rate of 2 × 10 -4 and decaying to 10 -7 .Table 1 shows the quantitative performance of our approach coupled with various SR baselines in terms of the metrics, parameter number, and computational cost. ClassSR [23] with an additional classifier module has more parameters, resulting in additional computational and parameter costs. ARM [4] adopts the validation set to build an Edge-to-PSNR lookup table, which generates additional inference time and parameter storage overhead. In comparison, our framework achieves superior performance with less computation (average 62%) than baselines, without incurring additional parameters or computational costs. When tested on unseen datasets such as B100 and Urban100, which lie outside the trained distribution, the compared methods exhibit performance degradation due to overfitting to specific features of the original training dataset and a lack of generalization ability to diverse data. In contrast, our method maintains comparable performance to the original model with lower computational costs (averaging 70%), benefiting from our balanced sampling and optimization strategies during training.Figure 4 shows visual comparisons across four testing datasets. The SR images produced by ClassSR and ARM exhibit structural blur and noise. In contrast, our method recovers more detailed information, resulting in better outcomes that are more faithful to the HR ground truth.To verify the effectiveness of our WBSR upon HES and BDLoss, we conduct ablation studies on Tesk2K and Test4K with sclae factor of ×4, as shown in Table 2 .During model training, we replace the original uniform sampling in baseline with our HES strategy. As the method "+HES" shown in Table 2 , HES achieves a 0.05 dB average improvement in terms of PSNR compared with the baseline, which benefits from enhanced feature representations in hard texture patches. Furthermore, we conduct additional experiments to compare our HES with existing sampling works [41, 49, 29] in Table 3 . It shows that our HES outperforms the previous best sampling method BSPA of an average of 0.1dB in terms of PSNR and demonstrates the superiority and generalization capabilities of our HES. In "+WBSR †", we can achieve even greater performance gains of 0.18 dB by integrating our HES with our BDLoss. HES first performs more sample-level sampling to learn generalized feature representations followed by fewer selective class-level sampling to focus on texture-rich regions to correct sample bias with stable learning and prevent the model's overfitting, which mitigates the model oscillation and addresses the overfitting problem. Furthermore, our HES achieves balanced stable training with our BDLoss for diverse samples in each training step, which solves the instability and training bias issues.Effectiveness of BDLoss. To demonstrate the effect of BDLoss, we train the SR model using uniform sampling and replace only the L 1 loss with L bd . As shown for method "+L bd " in Table 2 , the PSNR improved by balanced training is 0.06 dB with an average 65% computing cost compared to the baseline model, which demonstrates the superiority of our BDLoss.Effectiveness of Joint Training. When applying joint training of HES and BDLoss within WBSR to the baseline network, we can further improve the PSNR and SSIM results by 0.13 dB and 0.0043, respectively, which achieve an overall performance improvement with average 66% calculation. Furthermore, to fully demonstrate the effectiveness of our WBSR, we adopt the weight-balancing framework to retrain the full baseline model instead of the dynamic supernet model. It can be seen from the "+WBSR † "method of Table 2 , our WBSR with a computational cost comparable to baseline obtains average PSNR and SSIM gains of 0.33 dB and 0.0078, respectively. Models trained using our WBSR show consistent performance improvements that are not affected by the skewness of the training sample distribution. In Figure 5 , the SR performance on rare samples obtains gains, while the performance on abundant samples remains the same or slightly decreases, which proves that our weight-balancing strategy not only enhances the learning of texture areas and reduces redundant computation of flat areas. Additional experimental results are placed in supplementary materials.6 Conclusion In this supplementary material, we give a detailed derivation of Theorem 1 from Eq. 4 to Eq. 5EQUATIONThen, we give a derivation from Eq. 13 to Eq. 14. Since s i in Eq. 13 indicates the norm of the product of multi Gaussians, its inherent characteristic can also be delineated as a Gaussian distributionEQUATIONwhere N denotes the probability density function of a Gaussian distribution parameterized by a mean vector µ i and a covariance matrix Σ i + σ 2 noisc I. I denotes the identity matrix, ensuring that the covariance matrix remains positive definite. Building upon this foundation, the weighted Gaussian terms can be expressed asEQUATIONwhere ϕ i denotes the weight of individual Gaussian components. This weighting mechanism allows the model to flexibly learn balanced optimization of parameters based on the different data characteristics.Subsequently, we substitute Eq. 19 and Eq. 12 into Eq. 13 to derive the pivotal diversity balancing term, which integrates the principles of predictive uncertainty with the underlying training distributionEQUATIONFinally, we integrate Eq. 20 into the our L bd formula from Eq. 11 to obtain the final loss function expressionEQUATIONwhere balance weights θ are rectified through the joint optimization of two key terms with the loss: the diversity balancing term to ensure balanced optimization, and the regularization term to prevent overfitting.Visualization comparison of ablation. To further verify the feature representation ability of our BDLoss on hard texture patches, we also visualize the error map of SR results with GT images in Figure 6 . As can be seen from the yellow box area, the error of the model trained with L bd on the texture area is significantly smaller than that of L 1 , indicating that our L bd function improves the fitting accuracy in challenging texture regions. In addition, the effectiveness of our gradient projection strategy can also be proved according to the subnet allocation map, which reasonably allocates large subnets (i.e., green masks) to complex texture areas. This demonstrates that our gradient projection strategy can effectively enhance the model's ability to handle complex textures. Analysis of the number M of subnets. We verify the impact of the number of subnets on network inference performance in Table 4 . As the number of subnets increases, the inference effect will decrease to a certain extent because the fixed training epochs cause the extra subnets to not be fully trained. Increasing the number of training epochs will solve this problem to a certain extent. However, to achieve a trade-off between performance and efficiency, we adopt a total of nine subnet branches of three widths and three depths to process different image patches.Analysis of the class k of samples. To analyze the impact of the number of sample categories, we visualize the variation curves of computational cost GFLOPs and performance PSNR under different K classes in Figure 7 . When the K = 5 value is small, the results of gradient vector projection of In Figures 8 9 10 11 , we present more visual comparison results of our WBSR and other SOTA methods [23, 4] on different testing datasets at different scales. It can be seen that our algorithm can accurately reconstruct more spatial structures and more texture details and other algorithms suffer from loss of detail on difficult texture patches, which demonstrates the superiority of our algorithm.ClassSR ARM Ours HR B100 LR Figure 8 : Qualitative comparison results of ×4 on B100 [31] dataset between our method and other methods. Please zoom in for details. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper not adopts pretrained language models, image generators, or scraped datasets.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: The paper does not use existing assets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix:  


Decision: This paper receives 3 positive and 1 negative ratings. Most reviewers think that the idea about the imbalanced issue in image SR is reasonable and deserves to be explored. The proposed weight-balancing framework alleviates the imbalanced issue well and achieves good results. The main comparisons with other methods further show the improvements of the proposed one. One reviewer with a negative rating has a main concern about the contribution. After reading the author's rebuttal and other reviewers' comments, I believe this work has good motivation and contributions by exploring both imbalanced issue and model optimization imbalance. I would like to accept this paper. Of course, the authors are encouraged to further refine the paper based on the reviewers' comments.

Reviews: review: paper_summary: paper_summary: To address imbalances and parameter redundancy problems, author proposed the Weight-Balancing framework (WBSR), which balances model learning without altering the original model structure or training data. The approach includes a Hierarchical Equalization Sampling (HES) strategy to handle data distribution imbalances and a Balanced Diversity Loss (BDLoss) function to optimize learning for texture-rich regions while reducing redundant computations in smooth areas. They introduce a gradient projection dynamic inference strategy for accurate and efficient inference.
strength_weakness: strength_weakness: The approach may still face scalability challenges when applied to extremely large datasets or high-resolution images, limiting its applicability in some real-world scenarios.

While focusing on texture-rich regions can improve feature representation, it may also lead to overfitting, reducing generalization performance on smooth or less textured areas
questions: questions: To enhance generalization, consider testing the proposed methods on a broader range of datasets, particularly those with varying characteristics (e.g., different textures, lighting conditions). This will help validate the robustness of the framework across diverse scenarios.

Address the potential complexity of the Weight-Balancing framework (WBSR) by providing a more user-friendly implementation or detailed guidelines. Including example codes or a simplified version could facilitate adoption by practitioners with varying levels of expertise.


Provide well-defined interfaces for each module with clear input and output specifications. This will help users know what data to provide and what to expect in return. 
Include a section that addresses common issues users might encounter, along with solutions and tips for resolving them.

Include additional examples showcasing how WBSR can be applied in various real-world scenarios, such as different types of images or specific applications (e.g., medical imaging, satellite imagery)
limitations: limitations: While the approach emphasizes learning from texture-rich regions, this focus may result in suboptimal performance on images with predominantly smooth areas, leading to a risk of overfitting to specific features while neglecting others.

Although the framework is validated on various models and datasets, it may not have been tested extensively across all possible scenarios, raising concerns about its generalization capabilities in diverse applications.

The formulation of the super-resolution task as an imbalanced distribution transfer learning problem relies on certain statistical assumptions that may not hold in all real-world scenarios, potentially limiting its effectiveness
paper_summary: paper_summary: This paper proposes a Weight-Balancing framework to address the imbalanced learning issues in image super-resolution. Two categories of imbalance are involved, including data distribution imbalance and model optimization imbalance. Experiments demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. Some grammatical errors should be corrected, e.g., "are used to accelerate inference have been widely xxx" in Line25-26. Please check the whole paper.
2. I doubt the assumption that the distribution of the training set is imbalanced, whereas the independent testing set is balanced. Why is there this difference between the training and test sets？
3. The description of the proposed HES sampling method is confusing. For example, what dose the "classes" in the images SR mean? How to determine the number of classes K？Please improve the clarity and add more technical details.
4. From the probabilistic view, in my opinion, the prediction of SR network trained with L1 loss should correspond to the medium of a latent noisy prediction distribution. Therefore, I doubt the correctness of Eq. (2).
5. In the ablation study, the impact of the dynamic supernet model should be isolated. For instance, I wonder the performance gain of RCAN+HES over the vanilla RCAN.
questions: questions: Please see the weaknesses part.
limitations: limitations: Please see the weaknesses part.
paper_summary: paper_summary: This paper rethinks the imbalance problem in image SR and proposes a plug-and-play weight-balancing framework. It combines a Hierarchical Equalization Sampling strategy and a Balanced Diversity Loss to reduce computational cost while keeping or improving SR performance. Extensive experimental results demonstrate the effectiveness and superiority of the proposed method.
strength_weakness: strength_weakness: 1. The proposed method is of incremental contributions. This work is not the first to explore imbalance in image SR, which has been explored by the previous work, such as 

[1] Xiaotong Luo, Yuan Xie, Yanyun Qu: Learning Re-sampling Methods with Parameter Attribution for Image Super-resolution. In NeurIPS, 2023.
 
2. The data sampling is also explored in image SR. The experiments lack the comparison and discussion with the latest related works, such as [1] and:

[2] Shizun Wang, Ming Lu, Kaixin Chen, Jiaming Liu, Xiaoqi Li, Chuang Zhang, and Ming Wu. Samplingaug: On the importance of patch sampling augmentation for single image super-resolution. In BMVC, 2021.

[3] Xinyi Zhang, Tao Dai, Bin Chen, and Shu-Tao Xia. DDA: A dynamic difficulty-aware data augmenter for image super-resolution. In IJCNN, 2023.

3. The transformer-based SR backbones should be considered to compare in the experiment parts.
questions: questions: 1. Is other classification criteria tried, such as PSNR measurement like ClassSR? Actually, the mean and standard deviation of the gradient magnitude of the input samples may not well reflect the sample reconstruction difficulty.

2. How about the generalization of the proposed methods on other restoration tasks, like image denoising, JPEG compression. It seems that the mean and standard deviation of the gradient magnitude would be affected by some noises or more complex degradation factors.

3. There are some writing problems, such as
- Line 24-25
- Line 68
limitations: limitations: See the above comments
paper_summary: paper_summary: In this paper, the authors propose a novel framework called Weight-Balancing Super-Resolution (WBSR) that reformulates the SR task as an imbalanced distribution transfer learning problem.
strength_weakness: strength_weakness: 1. line 203 to 205 line of the article, the author introduces the reasoning method of gradient dynamic projection, but does not specifically illustrate the specific network used in this method within the article. If the structure of the network could be described more carefully, the article would be better.

	2. The article has designed a data sampling technique, network training technique, and network inference technique to enhance super-resolution technology. However, in the experimental section, it seems that only a limited number of experiments were conducted to support the argument. If more comparative experiments could be added to substantiate the claims, the article would be significantly improved.
questions: questions: 1. line 203 to 205 line of the article, the author introduces the reasoning method of gradient dynamic projection, but does not specifically illustrate the specific network used in this method within the article. If the structure of the network could be described more carefully, the article would be better.
limitations: limitations: 2. The article has designed a data sampling technique, network training technique, and network inference technique to enhance super-resolution technology. However, in the experimental section, it seems that only a limited number of experiments were conducted to support the argument. If more comparative experiments could be added to substantiate the claims, the article would be significantly improved.

Comments: title: The rebuttal is not very satisfying.
comment: Thank the reviewer very much for the great efforts in reviewing our paper. We kindly wish to remind the reviewer to consider our response and additional experiments. We are more than willing to provide further clarifications if there are any lingering questions or concerns.
Thank the reviewer very much for the great efforts in reviewing our paper. We kindly wish to remind the reviewer to consider our response and additional experiments. We are more than willing to provide further clarifications if there are any lingering questions or concerns.
Thank you very much for your further feedbacks. We sincerely apologize for the lack of organization and explanation for the experiments and notations in the PDF attached in the author rebuttal. 

Table 1, Table 2, and Table 3 present the comparison results between three SR networks (RCAN, Fusformer, and SRResNet) and our methods integrated with these backbone networks. The compared results are evaluated on four datasets including an autonomous driving scene dataset (KITTI2015), two satellite remote sensing datasets (CAVE and NTIRE2020), and a low-light super-resolution dataset (RELLISUR), which validate the generalization of our method across diverse scenarios. 

Table 4 provides the comparison between the vanilla RCAN method and the combined method with our sampling method (RCAN+HES).

Table 5 shows the comparison results between the backbone RCAN and the combined methods when it is integrated with three existing sampling methods (+BSPA, +SamplingAug, and +DDA), our sampled method (+HES), our loss function (+BDLoss) and both our sampled method and loss function (+ WBSR† and + WBSR). 
Here, "WBSR" indicates the model dynamic inference using multiple smaller subnetworks with lower computational costs.  "WBSR†" indicates the model inference using the whole supernet with maximum computational cost. (i.e., 100% FLOPs).

Table 6 presents the comparison results between two transformer-based SR backbones  (SwinIR and Fusformer) and  the combined methods when they are integrated with our WBSR.

Table 7 provides the comparison results between the backbone SRResNet and the combined methods when it is integrated with the classifier-guided method (Classifier) and our gradient-based method, respectively.

Table 8 presents the comparison results between two other restoration backbones (Image denoising backbone FFDNet and JPEG compression artifact removal backbone RNAN) and the combined methods  when they are integrated with our WBSR, which demonstrates the generalization of our method to other tasks.

Figure 1 showcases visualization examples of our method on medical and remote sensing datasets. 
"SR" in the first and third columns represents the SR results of our method. 
"Patch Classification" in the second and fourth columns represent the visualization results of image patches with different restoration difficulty after classification using our gradient-based method. 
Different color patches represent different restoration difficulties, e.g., dark blue represents texture areas that are difficult to restore, and green represents smooth areas that are easy to restore.

**[Q1:]**: the author did not answer my question, what is the backbone network of your plug and play part worked on? Please explain at least in the experiment. Also, this content should be reflected in camera ready of this paper.

**[A1:]**:  In the above explanations, we clearly indicated what are the backbone networks in the experiments (i.e, the first method in each tables). We will reflect them in camera ready of this paper. 

**[Q2:]**: The authors gave a page of further experiments, but there was a lack of organization and explanation for the experiments, as well as no explanation for the notation in the experiments. For example, in Table 5, the difference between WBSR and WBSR+ was not given, so listing experiments actually brings more problems. I suggest the authors add appropriate explanations and organization of experiments to enhance persuasiveness.

**[A2:]**: Thanks for this suggestion. As explained above, in table 5, "WBSR" indicates the model dynamic inference using multiple smaller subnetworks with lower computational costs.  "WBSR†" indicates the model inference using the whole supernet with maximum computational cost. (i.e., 100% FLOPs). 

We will follow the reviewer's suggestion and add appropriate explanations and organization of experiments to enhance persuasiveness.
The concerns raised in the review is addressed in this rebuttal, but not very satisfying.   
1. the author did not answer my question, what is the backbone network of your plug and play part worked on? Please explain at least in the experiment. Also, this content should be reflected in camera ready of this paper. 
2. The authors gave a page of further experiments, but there was a lack of organization and explanation for the experiments, as well as no explanation for the notation in the experiments. For example, in Table 5, the difference between WBSR and WBSR+ was not given, so listing experiments actually brings more problems. I suggest the authors add appropriate explanations and organization of experiments to enhance persuasiveness.