UPS: Unified Projection Sharing for Lightweight Single-Image Super-resolution and Beyond

Authors: kun  zhou
xinyu  lin
zhonghang  liu
xiaoguang  han
jiangbo  lu
  sse
  cuhk-shenzhen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 07:08:29

Abstract: To date, transformer-based frameworks have demonstrated impressive results in single-image super-resolution (SISR). However, under practical lightweight scenarios, the complex interaction of deep image feature extraction and similarity modeling limits the performance of these methods, since they require simultaneous layer-specific optimization of both two tasks. In this work, we introduce a novel Unified Projection Sharing algorithm(UPS) to decouple the feature extraction and similarity modeling, achieving notable performance. To do this, we establish a unified projection space defined by a learnable projection matrix, for similarity calculation across all self-attention layers. As a result, deep image feature extraction remains a per-layer optimization manner, while similarity modeling is carried out by projecting these image features onto the shared projection space. Extensive experiments demonstrate that our proposed UPS achieves state-of-the-art performance relative to leading lightweight SISR methods, as verified by various popular benchmarks. Moreover, our unified optimized projection space exhibits encouraging robustness performance for unseen data (degraded and depth images). Finally, UPS also demonstrates promising results across various image restoration tasks, including real-world and classic SISR, image denoising, and image deblocking.

Main paper: 
 introduction: 
 Single-image super-resolution is a fundamental task in computer vision, aiming to enhance the resolution and quality of a low-resolution image. Recently, Transformer-based methods [1] [2] [3] [4] [5] [6] , especially, SwinIR [7] , combines the benefits of window-based self-attention and convolutional feature extraction, thus achieving effective similarity modeling and feature extraction. It yields promising outcomes, reducing computational demand compared to global/non-local attention mechanisms.However, the coupled optimization in existing Transformer-based methods may face two challenges. First, in a lightweight configuration characterized by a very limited number of learnable parameters, performing layer-specific optimization for both image feature extraction and similarity modeling remains challenging. Second, such a tightly coupled optimization scheme (image feature extraction and projection similarity are synchronously updating in each layer during the training phase) may suffer from co-adaptation issue [8, 9] , potentially leading to inferior results.Interestingly, we observe that projection spaces (layer) in trained SwinIR-light exhibit substantial layer-to-layer (CKA [10] ) similarities 4 . Layer Index Layer Index Ratio s > 0.9 (×2) s > 0.9 (×3) s > 0.9 (×4)Layer Index Layer IndexFigure 1 : (a) We observe that the SwinIR-light (termed as Base) models exhibit significant similarities (CKA [10] ) in projection layers. (b) Comparison between our proposed UPS and SOTA lightweight SISR models on BSD100 [11] for ×2 setting. A bigger circle size means a larger number of parameters. While being the most computationally and parameter-efficient, UPS-S (a more lightweight version of our method) demonstrates highly competitive results compared to SOTA methods.×{2, 3, 4}) (projection layer) pairs get over 0.9 scores (ranging from 0 to 1) 5 . This experiment suggests that all the projection layers are highly similar.To mitigate the two problems, we are motivated by the observation and explore a novel Unified Projection Sharing (UPS) technique for lightweight SISR. In particular, UPS decouples the deep image feature representation and similarity learning: it performs the layer-specific image feature extraction while calculating the self-similarity in a unified projection space. In other words, the similarity modeling is optimized in a layer-invariant manner, effectively separating the learning of both two tasks. More specifically, UPS accomplishes self-similarity modeling with the following three steps: (i) UPS defines a unified projection space by a learnable matrix; (ii) for each self-attention layer, it projects deep image features onto the unified projection space; (iii) it calculates the attention map using the Cosine similarity metric in the projection space and performs attention-based aggregation.Our proposed UPS consistently demonstrates superior performance compared to existing approaches across all testing benchmarks. Notably, our method outperforms the second-best model by more than 0.33dB on the Manga109 dataset for the ×2 settings. Furthermore, our model exhibits significant improvement over our baseline model, SwinIR-light [7] , achieving enhancements of up to 0.50dB, 0.55dB, 0.47dB on the Manga109 dataset for the ×2, ×3, ×4 settings, while utilizing fewer parameters. Our contributions are summarized as follows:-We propose UPS, an effective decoupled SISR optimization framework, to address the challenge of simultaneous layer-specific feature extraction and similarity modeling for lightweight SISR.-UPS simplifies the similarity optimization process by learning a layer-invariant projection space, leading to effective aggregation (activating more local/non-local pixels as shown in Fig. 3 ) and improved performance, even with reduced model capacity (see Fig. 1 ) and less training samples (see the data efficiency analysis in Sec. A.2).-Extensive robustness analysis in Sec. 5.4, 5.5, A.3, A.4, have confirmed the good generalization ability of our proposed UPS for unseen data, such as noisy image and depth map SR. 
related works: 
 CNN-based SISR. Due to their low complexity and helpful feature extraction abilities, CNNs have been widely used for SISR task. SRCNN [12] pioneered the use of deep convolutional neural network (CNN) architectures specifically designed for single image super-resolution (SISR). SRCNN consists of only three layers: patch extraction, non-linear mapping, and reconstruction. It has demonstrated competitive performance compared to traditional non-deep methods, inspiring the development of numerous lightweight CNN approaches in the SISR field. ESPCN [13] introduced a compact network architecture that employs sub-pixel convolutional layers to upscale low-resolution image features. In contrast, LapSRN [14] utilizes image structure priors across different pyramid representations, resulting in improved performance while minimizing computational overhead. Taking inspiration from dictionary-learning models [15] [16] [17] , LAPAR [18] learns linear coefficients associated with pre-defined basic up-sampling kernels to produce an optimal pixel-specific kernel, achieving superior super-resolution results. LatticeNet [19] designs a parameter-efficient convolutional lattice block to extract hierarchical contextual features. Despite their computational efficiency, CNN-based models are limited in terms of long-term aggregation due to content-invariant similarity optimization.Transformer-based SISR. Recently, Transformer-based techniques [20] [21] [22] [23] [24] [25] have achieved remarkable outcomes in SISR but still suffer from high complexity. The computational cost of non-local self-similarity modeling increases quadratically with the size of the image. Inspired by the success of Swin Transformer [26] , numerous window-based Transformer frameworks emerged to address the efficiency of SISR. For example, SwinIR [7] introduces a residual window-based transformer block (RSTB) for image feature extraction and similarity-based aggregation, outperforming previous CNN-based and Transformer-based approaches. DLGSA-l [27] proposes a global sparse attention technique to enhance the aggregation of relevant tokens. NGswin [28] incorporates the N-Gram context to attain a larger receptive field, activating more neighboring pixels for effective aggregation. However, when it comes to lightweight setups, the optimization of coupled feature extraction and similarity calculation is limited, resulting in inferior performance.Efficient Transformers. On the other hand, some advanced transformers have been proposed to reduce the computational complexity, enhancing inference or training efficiency. ShareFormer [29] presents a local similarity map-sharing scheme between neighboring attention layers for lower latency. Thus, ShareFormer shares a static similarity map for neighboring attention layers while UPS calculates dynamic similarity maps with layer-refined features in a shared projection space.Skip-Attention [30] cuts off some intermediate attention layers to improve efficiency and performance for high-level tasks. LaViT [31] proposes a residual-based attention downsampling that fuses the initial calculated attention scores to guide the aggregation of the following layers, resulting in faster efficiency and improved classification accuracy.Therefore, Skip-Attention and LaViT follow the existing coupled optimization scheme (reduce some attention calculations), and UPS proposes a decoupled learning strategy to enhance performance. We will cite the insightful studies and add this discussion to our revised paper.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We have discussed the limitations of our method in Sec.A.7.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
understanding swin transformer: 
 Preliminaries. Swin Transformer [26] proposes an effective self-attention mechanism, achieving long-range information capture at a lower computation complexity. Inspired by Swin Transformer, several subsequent methods [7, 32, 28] dedicated to solving lightweight SISR have emerged, consistently enhancing the quality of super-resolved images. Fig. 2 (a) illustrates the general framework architecture of the Swin Transformer-based SISR method. It consists of three primary components: a shallow head module, a deep image feature extraction and aggregation (FEA) module, and a tail reconstruction module. The head module is tasked with converting the input low-resolution RGB image into a high-dimensional feature space. The FEA module, the key role in the whole architecture, is composed of multiple (N ) Swin Transformer layers (STLs). Each STL has two main objectives: (i) extracting image features and (ii) modeling similarities using a learnable projection space. The former focuses on capturing essential image features, while the latter employs window-based self-attention to facilitate spatially adaptive aggregation. Notably, similarity modeling optimizes a projection space to obtain pixel-wise correlations, which is achieved by projecting image features into the learnable projection space and calculating similarity scores. Finally, the tail module generates the final high-resolution output image, completing the SISR process. In the subsequent section, we will delve into the details of the STL, with a particular emphasis on deep feature extraction and similarity modeling aspects.Efficient deep feature extraction and similarity modeling are accomplished by the Swin Transformer Layer (STL), the fundamental unit within the FEA module of the Swin Transformer. Illustrated in Fig. 2 (b), in the i-th STL, the process begins by employing a convolutional layer to extract deep image feature Fi from an input feature F i-1 , which is the output of the preceding (i -1)-th STL: Fi = Conv(F i-1 ).(1) Subsequently, the STL executes a conventional window-based self-attention mechanism, comprising four basic steps: (i) window-partitioning, (ii) deep feature projection for similarity calculation, (iii) aggregation based on similarity to merge neighboring pixels, and (iv) patch merging.(i) Window-partitioning. Initially, the updated image feature Fi is reshaped into HW M 2 nonoverlapping patches, each with a shape of M 2 × C, where M 2 represents the spatial size of each patch and C is the channel dimension.(ii) Layer-specific projection. Following window-partitioning, each divided image patch X i from Fi is projected to generate the corresponding query, key, and value matrices Q i , K i , V i :EQUATIONwhere P Q i , P K i , P V i ∈ R {d×C} denote the learnable projection parameters specific to the i-th STL, while Q i , K i , V i ∈ R {M 2 ×d} represent the projected features of patch X i and d is the projection dimension. The similarity matrix is then computed:EQUATIONwhere B i represents a relative position encoding, and S i is the predicted similarity map for the X i .(iii) Similarity-based Aggregation. Later on, neighboring information within the patch X i is aggregated based on the computed similarity map S i :EQUATION(iv) Patch Merging. Finally, all the aggregated image patches are reshaped into a 2D image feature which is fed into the next STL for further processing.Discussion. With sufficient model capability, i.e., millions of parameters, SwinIR [7] , a SOTA Swin Transformer SISR model, exhibits strong abilities for the SISR task. However, in resource-constrained, lightweight settings as previously mentioned, it potentially poses challenges to simultaneously optimize deep image feature extraction and projection space. We will compare the per-layer projection space optimization with our proposed UPS scheme later. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [No] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: We will release our code and all the command and environment needed to reproduce the results.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
unified projection sharing: 
 We follow the general framework structure of Swin Transformer but use decoupled projection space optimization. As shown in Fig. 2 (c), UPS consists of three basic modules, namely the convolutional head module, FEA module, and reconstruction tail module. In the FEA, we develop a decoupled Swin Transformer layer (D-STL) for deep image feature extraction, while optimizing a unified projection space for similarity modeling. Next, we will provide a detailed description of our D-STL.We take the i-th D-STL for illustration. Given an input image feature F i-1 produced by the last (i -1)-th D-STL, we aim to perform feature updating as well as self-similarity-based aggregation.Similarly, we adopt the Eq. 1 to conduct deep image feature extraction and obtain the transformed image feature Fi . Then we employ the window-partitioning process to reshape the F into HW M 2 non-lapped image patches.Unified Projection. Unlike the layer-specific projection scheme in Swin Transformers, we introduce a layer-invariant (unified) projection space defined by a learnable matrix U Q ∈ R {D×C} (D refers to the unified projection dimension) and project the deep feature X i on this unified projection space:EQUATIONAfter that, we consider the calculation of the self-similarity in the unified projection space. Motivated by ReLUFormer [34] that addresses the over-centralized distribution in Softmax by incorporating ReLU activation for self-similarity calculation, we get the similarity scores as:EQUATIONNote that we conduct normalization operation for the projected image features Q i6 . Subsequently, we utilize the Cosine similarity metric, followed by a ReLU activation function, to obtain the final similarity map S i . We also assess our design in Sec. 5.3. Finally, leveraging the calculated similarity map S i , we perform image feature aggregation using Eq. 4.Discussion. In Algorithm. 1, 2, we provide side-by-side illustrations of standard STL and our D-STL and highlight the differences between the two methods. Projection: Qi, Ki, Vi = XiP Q i , XiP K i , XiP V i 7: Similarity cal.: Si = SoftMax Q i K T i √ d+ BiAggregation: Yi = SiVi 9: Benchmark Datasets. Following previous studies [7, 28, 18] , we utilize the DIV2K [37] image dataset for training. Subsequently, we conduct comprehensive evaluations on several widely-used SISR benchmarks, including Set5 [38] , Set14 [39] , BSD100 [11] , Urban100 [40] , and Manga109 [41] .EQUATIONOur quantitative comparison is based on PSNR and SSIM. Consistent with established research, we report the results specifically for the Y channel derived from the YCbCr color space. 
comparison with sota methods: 
 We perform extensive comparisons with a wide range of lightweight SISR models: MAFFSRN (ECCV20) [42] , LAPAR-A (NeurIPS20) [18] , LatticeNet (ECCV20) [19] , RLFN (CVPRW22) [43] , SwinIR-light [7] , NGswin [28] , SwinIR-NG [28] , and DLGSA-l (ICCV23) [27] . More comprehensive comparisons with early SOTA lightweight models can be accessed in our supplementary material. Quantitative Comparison. Tab.1 illustrates the quantitative evaluation. Our proposed UPS consistently outperforms existing methods across all benchmarks. Notably, UPS exceeds the second-best model by over 0.33dB on the Manga109 dataset [41] for the ×2 setting. Additionally, our model shows significant improvements over SwinIR-light [7] , achieving improvements of up to 0.5dB, 0.55dB, and 0.47dB on the Manga109 dataset [41] for the ×2, ×3, and ×4 settings, respectively, while using fewer parameters. These results confirm the effectiveness of our decoupled optimization strategy. Significantly, when constrained by model complexities, our UPS demonstrates superior performance over SwinIR-light [7] . As shown in Tab.1, our approach achieves a 0.55dB increase in PSNR for ×2 super-resolution on the Urban100 dataset [40] , highlighting the advantages of our decoupled optimization in feature extraction and similarity modeling under compact parameter conditions.Additionally, we evaluate the inference efficiency of various state-of-the-art (SOTA) lightweight single image super-resolution (SISR) models. As shown in Table 2 , UPS reduces the overall inference cost by 33% in terms of FLOPs compared to our baseline model, SwinIR-light Last, we have extensively explored the benefits of UPS for real-world SR and other frameworks, including HAT and DRCT, under both lightweight and parameter-intensive scenarios. Our experiments show that the proposed UPS consistently enhances efficiency and performance across all these settings (real-world SR, lightweight, and SISR tasks).For real-world super-resolution (Real-world SR), as shown in Tab. 3 of the PDF file (also the table below), our proposed UPS-GAN outperforms other state-of-the-art GAN-based [44, 45] and even Diffusion-based methods (Reshift [46] and StableSR [47] ) in terms of NIQE, NRQM, and PI metrics, achieving the best quantitative results ( [18] , SwinIR-light [7] , NGSwin [28] and UPS (ours) on four popular benchmarks (BSD100 [11] , Urban100 [40] , Manga109 [41] and DIV2K [37] .) under ×4 setting. Our predictions present more detailed textures and fewer artifacts.To verify the effectiveness of our design, we conduct a series of comprehensive experiments. Note that we primarily adopt the UPS-S architecture for fast analysis. We generally report the quantitative results on the Set14 benchmark under the ×2 setting.Different Groups of Projection Space. Unlike conventional attention-based frameworks, our approach introduces a unified projection space for similarity calculation. In this evaluation, we investigate the impact of incorporating additional groups of projection space. The quantitative results in Fig. 5 (Left) shows that a higher number of block-wise projection groups reduce performance compared to our unified projection method. Additionally, layer-specific projection optimization exhibits inferior performance. Fig. 5 (Right) confirms that our unified projection-sharing scheme outperforms models with multiple projection spaces. This experiment highlights the challenges of coupled optimization in image feature extraction and similarity modeling.Similarity Calculation. Differing from the conventional similarity calculation paradigm (Matrix dot product + SoftMax), we incorporate the Cosine distance followed by a ReLU activation for similarity computation. Here, to assess the effectiveness of our choice, we train different models that utilize various combinations of distance metrics (Matrix dot and Cosine) and activation functions (SoftMax and ReLU). The results, as displayed in the left Fig. 6 , indicate that our design (Cosine + ReLU) achieves the best performance among all the competing strategies. Additionally, we provide a corresponding visual comparison in the right Fig. 6 . It shows that our design activates more non-local pixels, resulting in a larger valid receptive field and more precise reconstructed image details. In this section, we explore the benefits of applying UPS for other image restoration tasks. While SwinIR (baseline model) requires millions of parameters for image denoising and deblocking, our lightweight UPS frameworks handle these common low-level restoration tasks more efficiently. As shown in Fig. 8 , the results, indicate that UPS achieves performance comparable to the large baseline model (SwinIR) while requiring only 1 13 of the model complexity on Denoising. Additionally, the compact baseline models exhibit inferior performance relative to our UPS. We describe the framework of these models in Sec. A.6 of our appendix. We also compare it with our baseline model (SwinIR-light) on the depth SR task. To do this, without training on any depth images, we directly test the two models on the NYU V2 [48] depth benchmark8 under ×4 and ×16 settings. We use the PSNR, SSIM, and RMSE (the root-mean-square error) metrics for quantitative evaluation. The quantitative results are shown in the left Fig. 9 . We can see UPS consistently outperforms its baseline model on all the objective metrics. For instance, UPS exhibits superior performance compared to SwinIR-light with a PSNR improvement of 0.54dB (0.73dB) for ×4 (×16) configurations. The visual examples in the right Fig. 9 illustrate that UPS generates clearer structures, leading to higher accuracy when compared to our baseline model.In addition to these two extensions, we also explore the improvement over SwinIR, DRCT, HAT for both lightweight and classic SISR to comprehensive analyze the potential capabilities of the proposed UPS. Please refer to the appendix sections. 
conclusion: 
 In A Appendix / supplemental material A.1 Improvements over SOTA Swin-transformersIn our main paper, we have explore the improvements over SwinIR-light for lightweight SISR task.Here, we discuss the potential improvement and generalization capability of UPS for more Swintransformers (such as DRCT [49] and HAT [21] ) on both lightweight and classic SISR. Table . 5 shows our UPS is capable of enhancing several SOTA Swin-transformers with fewer parameters, FLOPs and inference latency. In this part, We assess the data efficiency of several models, including LAPAR-A, LatticeNet, our baseline model SwinIR-S [7] (a more lightweight model with identical framework configuration as our UPS-S), and our proposed UPS-S. In addition to training all models on the complete training set, we gradually adjust the percentage of used training data. As shown in Fig. 10(a) , UPS consistently outperforms other models regardless of training data size, demonstrating superior data efficiency. Fig. 10(b) shows PSNR values during training iterations, with UPS-S converging faster and providing better early predictions. These results highlight the effectiveness of our UPS scheme.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We have stated all the experimental details for the reproduction of our method, including data, hyperparameters, optimizer, learning rate, and so on.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [NA] Justification: Following previous works, we conduct extensive experiments to demonstrate the effectiveness of our proposed UPS. Moreover, our method also exhibits superior generalization(robust) ability for unseen data.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We have provided the experimental details on the computer resources, including GPU, memory and execution time.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: As stated in our introduction, super resolution is a fundamental task in computer vision, and is pivotal in various fields such as medical imaging, satellite imagery, and consumer electronics, where clear and detailed visual information is crucial.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We cited related papers, codes and data in our paper, and we also stated the version we used.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: our paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Intuitively, given noisy input features, the error will be invertibly accumulated and affect the following projection space optimization. To better understand the effectiveness and robustness of our unified projection-sharing scheme, we compare SwinIR [7] and UPS under noisy input features to simulate the perturbation training. Firstly, we train both two models with perfect training images. Then, we add different levels of Gaussian white noise (zero mean and std values ranged from 0.01 to 0.3) on the input image feature F 1 (the input of the FEA module) for both training and evaluation:EQUATIONwhere n is the Gaussian white noise. The results are shown in Fig. 11 . We observe that, with severe noise, the SwinIR-light fails to restore high-frequency details in the output feature F N produced by the last STL layer, thus it produces incorrect image structures. In contrast, UPS is able to recover high-frequency signals from the noisy input feature and accordingly reconstruct accurate image details. This experiment suggests our proposed UPS is more optimization robust and effective with noisy input features.Table 7 : Robustness comparison of SwinIR-light [7] , NGSwin [28] and our proposed UPS. While being trained on clean DIV2K [37] training samples, we directly evaluate their generalization ability on the degraded Set14 benchmark under the x4 setting. We report the PSNR (dB)/SSIM values for quantitative evaluation. The co-adaptation issue [8, 9] reveals that deep-learning models may fit the training samples well but exhibit poor generalization for unseen data, especially for out-of-domain samples. Here, we aim to evaluate the robustness of SOTA SISR models (including SwinIR-light and NGSwin) and our proposed UPS. To do this, we apply different data degradations (i.e., JPEG compression, Gaussian blur, and Gaussian White noise) and obtain degraded samples. Without training on these degradations, we directly evaluate the performance of these lightweight SISR models. The results presented in Tab. 7 illustrate our proposed UPS is more robust on unseen data. Notably, UPS outperforms the competing models up to 0.36dB on the noise testing data. Moreover, Fig. 12 shows both SwinIR-light and NGSwin restore incorrect and blur image structures due to their poor generalization (not robust) for unseen degradations. In contrast, UPS produces more accurate results with clearer image contents.1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The abstract and/or introduction clearly state the claims made, also match theoretical and experimental results. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: Our research conducted in the paper conform with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).The numerical values on the axes indicate the layer indices.To decrease the model complexity, we set the Ki to be identical to the Qi, as the SISR task typically involves only one data modality.Xi, Yi denote input and output, and Si is the similarity matrix.NYU V2 consists of 449 testing depth images at a size of 480 × 640. We use bicubic interpolation to generate the low-resolution depth images. 


Decision: The paper presents a lightweight transformer-based single-image super-resolution model. The key idea is decoupling feature extraction and similarity modeling with Unified Projection Sharing. The reviews are mixed, with one Accept, one Borderline accept, and two borderline reject. The general weakness raised by the reviewers are the lack of latency comparison, real-world SR results, and comparison with SOTA super-resolution models. 

The authors provided a detailed rebuttal responding to these concerns with additional experimental results. More specifically, the authors provided results on latency comparisons, results on the real-world SR evaluation, and a comparison with Omni-SR using the same training dataset. The reviewers who were leaning positive are satisfied with the clarification and additional results. Unfortunately, the two reviewers who initially leaned negatively toward the paper did not fully engage with the newly presented results. 

The AC reviewed the new experimental results and believes that the authors have sufficiently addressed the concerns that were initially raised. Thus, the AC recommends to accept.

Reviews: review: paper_summary: paper_summary: This paper presents a novel algorithm named UPS designed to enhance the performance of Transformer-based frameworks in single-image super-resolution (SISR), particularly under lightweight scenarios. The authors identify the challenge posed by the simultaneous layer-specific optimization required for deep image feature extraction and similarity modeling in existing methods. To address this, UPS decouples these tasks by establishing a unified projection space via a learnable projection matrix. The proposed UPS method demonstrates state-of-the-art performance. Additionally, UPS shows good performance on broader image restoration applications.
strength_weakness: strength_weakness: 1. “Fig. a.(1-3) below shows over 0.95% (0.99%, 0.95%, 0.96%) for ×{2, 3, 4}) (projection layer) pairs get over 0.9 scores (ranging from 0 to 1) ". If there are no errors in this paragraph, the similarity between layers is very low.
2. The contribution point in this paper is the Unified Projection Sharing. However, in the method, the activation function is also modified. I wonder how much of a performance boost I would get if I left the activation function unchanged and just used consistent space sharing. Please give the analytical or experimental results.
3. As a lightweight method, it would be better to compare inference time.
questions: questions: In this paper, it is mentioned that each layer of the model carries out image feature extraction and similarity modeling at the same time, which is difficult to train and will affect the performance of the model. Then, can the model performance be improved by improving the training strategy and modifying the loss function? In theory, if you can train the model properly, will you get better performance than UPS?
limitations: limitations: The paper has discussed methodological limitations.
paper_summary: paper_summary: The paper proposes an effective lightweight decoupled SISR algorithm that simultaneously performs layer-specific optimization for deep feature extraction and similarity modeling. Specifically, the proposed method casts the deep feature extraction as per-layer optimization, while the similarity modeling is achieved by shared projection space. The proposed method is able to be extended to many restoration tasks, such as denoising, and JPEG image deb locking.
strength_weakness: strength_weakness: 1. Line 49: "simultaneously" -> "simultaneous".
2. Figure 2: the placement of S1 and Si in Fig. 2(c) is incorrect.
3. Eq. (5): why do you set Vi directly equal to Xi instead of projecting Xi to Vi as done in Swin Transformer?
4. Eq. (6): what does $Q_{i}^D$ denote in this equation? What does D represent?
5. It's recommended to report the quantitative results on the DIV2K dataset.
questions: questions: See the Weaknesses.
limitations: limitations: If the authors can address my concerns, I am ready to change my recommendation based on the comments.
paper_summary: paper_summary: This work introduces a novel unified projection sharing algorithm that decouples feature extraction and similarity modeling. A unified projection space defined by a learnable projection matrix is created for similarity computation across all self-concerned layers. Extensive experiments demonstrate that the proposed UPS achieves state-of-the-art performance compared to leading lightweight SISR methods.
strength_weakness: strength_weakness: 1. The idea of shared projection is somewhat similar to attention sharing [1,2,3]. Discussion and analysis with these highly relevant studies is needed.

2. The needs in Table 1 include computational quantities for the different models. It is unfair to compare with CNN-based methods only the parameters, which are much less computationally intensive.

3. Inference efficiency must be included in the comparison, including CNN and Transformer based methods. Paramas and FLOPs do not fully reflect the on-device running speed of the model.

4. Lack of experimental evaluation in more complex real-world scenarios.

5. Similarity calculation methods have been discussed in past studies [4, 5, 6].

> 1.  ShareFormer: Share Attention for Efficient Image Restoration. arxiv 2023.

> 2. Skip-Attention: Improving Vision Transformers by Paying Less Attention. arxiv 2023.

> 3. You Only Need Less Attention at Each Stage in Vision Transformers. CVPR 2024.

> 4. EfficientViT: Multi-Scale Linear Attention for High-Resolution Dense Prediction. ICCV 2023.

> 5. Swin transformer v2: scaling up capacity and resolution. CVPR 2022.

> 6. Swin2SR: SwinV2 Transformer for Compressed Image Super-Resolution and Restoration. ECCVW 2022.
questions: questions: Please address the issues raised in the Weaknesses.
limitations: limitations: Limitations were discussed.
paper_summary: paper_summary: This paper introduces Unified Projection Sharing (UPS), a novel algorithm for lightweight single-image super-resolution (SISR) that decouples feature extraction from similarity modeling by employing a unified projection space. This approach achieves state-of-the-art performance across various benchmarks while demonstrating robustness for unseen data and promising results for additional image restoration tasks, all with a computationally efficient model.
strength_weakness: strength_weakness: 1. The author did not conduct experiments outside of the lightweight model [1][2], and from an optimization perspective, the proposed method should be possible to improve performance in various situations. The authors need to be able to justify themselves.

2. Lack of actual latency comparison. Due to some differences in network structure, the params and FLOPs may not accurately reflect the efficiency of the model.

3. Lack of comparison with the SOTA model, e.g., [3]. outperforms the UPS almost on every benchmark.

Ref:

1. Chen, Xiangyu et al. “Activating More Pixels in Image Super-Resolution Transformer.” 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022): 22367-22377.

2. Hsu, Chih-Chung et al. “DRCT: Saving Image Super-resolution away from Information Bottleneck.” ArXiv abs/2404.00722 (2024): n. pag.

3. Wang, Hang et al. “Omni Aggregation Networks for Lightweight Image Super-Resolution.” 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023): 22378-22387.
questions: questions: See above.
limitations: limitations: Yes.

Comments: title: Dear Reviewer RnSV
Thanks for your comments
Thanks
Response for the further problems
Response to author's rebuttal
Thanks for your comment
Rating
comment: Dear Reviewer HE6k:
We have made every effort to address all concerns and provide comprehensive evidence.

For Q1, we explored real-world SR and large models for SISR, and also integrated UPS into DRCT/DRCT-light and HAT/HAT-light to boost their performance.

For Q2, as acknowledged by all other reviewers, we provided inference efficiency for further comparison.

For Q3, we conducted a detailed comparison with Omni-SR and Omni-SR+, showing that UPS/UPS+ outperforms them under a fair setup (using DIV2K/DF2K for training).


Could you please let us know if our rebuttals and further responses have answered all your questions? We greatly appreciate it.
We have made every effort to address all concerns and provide as much evidence as possible. Could you please let us know if our rebuttals and further responses have answered all your questions? We greatly appreciate it.
Thank you for your kind comments and for appreciating our work. We will include these valuable discussions in the final version of our paper.
Thanks for the response. Most of the concerns are addressed. I have increased my rating
Thank you for your additional comments. We're pleased that our previous response addressed some of your concerns. Here are our further responses to the issues you've raised:

1. As acknowledged by you and the other reviewers, UPS is a novel and effective lightweight SISR algorithm. Nonetheless, we explored the potential benefits of applying UPS to this new task. We ensured a fair evaluation by strictly following the same baseline (SwinIR, BSRGAN) and using consistent evaluation metrics, such as NIQE, NRQM, and PI, for all competing methods on the RealSRSet [1] benchmark.

2. In response to Q1, we have thoroughly examined the contributions of the suggested works and outlined the key differences between them and UPS. To summarize, UPS operates independently from these transformers. For instance, while ShareFormer generates a static similarity map for neighboring layers, UPS conducts dynamic similarity calculations within a shared projection space. Furthermore, unlike these three works, which adhere to coupled layer-specific similarity and feature extraction optimization, UPS introduces a decoupled approach for these aspects, specifically designed for lightweight SISR.

3. Due to the absence of ground truth data in the RealSRSet [1] benchmark, we cannot calculate reference-based metrics like PSNR, SSIM, and LPIPS. Instead, we present results using CLIPIQA and MUSIQ, which are widely accepted in diffusion-based models. Although UPS-GAN follows the GAN-based framework of our baseline models (SwinIR-GAN), it consistently achieves top-1 and top-2 performance when compared to other leading methods, as shown in the Table below.

| Metrics | BSRGAN | RealSR |  ResShift$\dagger$ |  StableSR$\dagger$ | SwinIR-GAN | UPS-GAN |
|---------|--------|--------|-----------|-----------|------------|---------|
| CLIPIAQ↑  | 0.6321   | 0.6051   | 0.5834      | 0.5025  | 0.5996       | **0.6577** |
| MUSIQ ↑  | **65.85**  | 62.59   | 54.29      | 60.32      | 63.45   | 64.79 |

1. Zhang et al. "Designing a Practical Degradation Model for Deep Blind
Image Super-Resolution." ICCV, 2021.
Thank you for your response. Concerns about FLOPs and running efficiency were addressed. However, I still have some issues regarding the evaluation of real-world SR. NIQE, NRQM, and PI are not commonly used metrics for evaluating real-world SR, which is typically measured using PSNR, SSIM, LPIPS, CLIPIQA, and MUSIQ. Additionally, the response does not discuss the differences and uniqueness of the similarity calculation strategy from previous work. Therefore, I am inclined to keep the original rating.
Thank you for your kind comments and appreciation of our work. We will do our best to improve the final version of our paper based on your suggestions.
All concerns have been addressed here, so I‘d like to accept this paper (rating: 7).