UPS: Unified Projection Sharing for Lightweight Single-Image Super-resolution and Beyond
Authors: Kun Zhou
Xinyu Lin
Zhonghang Liu
Xiaoguang Han
Jiangbo Lu
Venue: NeurIPS 2024 poster (2024)
Keywords: Lightweight SISR
Projection Space
Primary area: machine_vision

Abstract: To date, transformer-based frameworks have demonstrated impressive results in single-image super-resolution (SISR). However, under practical lightweight scenarios, the complex interaction of deep image feature extraction and similarity modeling limits the performance of these methods, since they require simultaneous layer-specific optimization of both two tasks. In this work, we introduce a novel Unified Projection Sharing algorithm(UPS) to decouple the feature extraction and similarity modeling, achieving notable performance. To do this, we establish a unified projection space defined by a learnable projection matrix, for similarity calculation across all self-attention layers. As a result, deep image feature extraction remains a per-layer optimization manner, while similarity modeling is carried out by projecting these image features onto the shared projection space. Extensive experiments demonstrate that our proposed UPS achieves state-of-the-art performance relative to leading lightweight SISR methods, as verified by various popular benchmarks. Moreover, our unified optimized projection space exhibits encouraging robustness performance for unseen data (degraded and depth images). Finally, UPS also demonstrates promising results across various image restoration tasks, including real-world and classic SISR, image denoising, and image deblocking.

UPS: Unified Projection Sharing for Lightweight Single-Image Super-resolution and Beyond
Kun 42 Xinyu Lin
Zhonghang Liu?,
Xiaoguang Han
Jiangbo Lu
SSE, CUHK-Shenzhen,
SmmrrMMrr Corporation *SMU, Singapore
hanziaoguangēcuhk. edu. en, jiangbo. www.malll.m..ooo
Abstract
To date, Transformer-bart frameworks have demonstrated impressive results in single-image super-resolution (SISR). However, under practical lightweight scenarios, the complex interaction of deep image feature extraction and similarity modeling limits the performance of these methods, since they require simultaneous layer-specific optimization of both two tasks. this work, we introduce a novel Unified Projection Sharing (UPS) algorithm decouple the feature extraction and similarity modeling. To achieve this, we establish unified projection space defined by learnable projection matrix, for similarity calculation across all self-attention layers. As result, deep image feature extraction remains a per-layer optimization manner, while similarity modeling carried out by projecting these image features onto the shared projection space. Extensive experiments demonstrate that our proposed UPS achieves state-of-the-me performance relative leading lightweight SISR methods, verified by various popular benchmarks. Moreover, our unified optimized projection space exhibits encouraging robustness performance for unseen data (degraded and depth images). Finally, UPS also demonstrates promising results across various image restoration tasks, including real-world and classic SISR, image denoising, and image deblocking.
Introduction
Single-image super-resolution fundamental task in computer vision, aiming to enhance the resolution and quality of low-resolution image. Recently, Transformer-basee methods (1+6, espe- cially, SwinIR [7], combines the benefits window-based self-attention and convolutional feature extraction, thus achieving effective similarity modeling and feature extraction. It yields promising outcomes, reducing computational demand compared to global/non-local attention mechanisms.
However, the coupled optimization in existing Transformer based methods may face two challenges. First, in a lightweight configuration characterized by very limited number of learnable parameters, performing layer-specific optimization for both image feature extraction and similarity modeling remains challenging. Second, such a tightly coupled optimization scheme (image feature extraction and projection similarity are synchronously updating in each layer during the training phase) may suffer from co-adaptation issue g [9. potentially leading to inferior results.
Interestingly, we observe that projection spaces (layer) in trained SwinlR-light exhibit substantial layer-to-layer (CKA (10) similarities"
Fig. a.(1-3) below shows over 0.95 (0.99, 0.95, 0.96) for
PProject leader
I CCo-first author
I 'Corresponding author
'The dimensions remain consistent across all projection layers in SwinlR-light Thus we can directly evaluate the pair-wise similarity scores.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(b)
Basee (x2)
Buse (x3)
Base (X4)
32:47
UPS
Layer Index
32.42
824K
17
UPS-S
32.37
453K
DLGSA-light
NGSwin
0
PSNR (dB)
32:32
DLGSA-tiny 745K
910K
118K
17
11
25
92.27
366K
LatticeNet
SwinIR-light
> 0.9 (X2)
S 0.9 (X3)
S 0.9 (X4)
32:22
#92K
EDSR
L.
SwinTR-8
Rañio
FALSR-A
1370K
32.12
1021K
11
23
0.7
17
11
23
160
210
Layer Index
Layer Index
Layer Index
FLOPSs (G)
Figure I: (a) We observe that the SwinIR-light (termed as Base) models exhibit significant similarities
(CKA [0]) in projection layers. (b) Comparison between proposed UPS and SOTA lightweight SISR models on BSD100 [11] for x2 setting. A bigger circle size means a larger number of param- eters. While being the most computationally and parameter-efficient UPS-S (a more lightweight version of our method) demonstrates highly competitive results compared to SOTA methods.
x (2)3,4)) (projection layer) pairs get over 0.9 scores (ranging from to This experiment suggests that all the projection layers are highly similar.
To mitigate the two problems, we are motivated by the observation and explore novel Unified Projection Sharing (UPS) technique for lightweight SISR. In particular, UPS decouples the deep image feature representation and similarity learning: it performs the layer-specific image feature extraction while calculating the self-similarity in a unified projection space. In other words, the similarity modeling is optimized in a layer-invariant manner, effectively separating the learning of both two tasks. More specifically, UPS accomplishes self- self--iiillartt modeling with the following three steps: (i) UPS defines a unified projection space by a learnable matrix; (ii) for each self-attention layer, it projects deep image features onto the unified projection space; (iii) it calculates the attention map using the Cosine similarity metric in the projection space and performs attention-basse aggregation.
Our proposed UPS consistently demonstrates superior performance compared to existing approaches across all testing benchmarks. Notably, our method outperforms the second-best model by more than 0.33dB the Mangal09 dataset for the x2 settings. Furthermore, our model exhibits sig- nificant improvement over our baseline model, SwinlR-light [7], achieving enhancements of up to 0.50dB, 0.55dB, 0.47dB the Mangal09 dataset for the x2, x3, x4 settings, while utilizing fewer parameters. Our contributions are summarized as follows:
We propose UPS, an effective decoupled SISR optimization framework, to address the challenge of simultaneous layer-specific feature extraction and similarity modeling for lightweight SISR.
UPS simplifies the similarity optimization process by learning a layer-invariant projection space, leading effective aggregation (activating more local/non-local pixels shown in Fig. 3 and improved performance, even with reduced model capacity (see Fig. T and less training samples (see the data efficiency analysis in Sec. A.2)
A.4, have confirmed the good generalization ability of our proposed UPS for unseen data, such noisy image and depth map SR.
Extensive robustness analysis in Sec. 5.4 5.5
2 Related Works
CNN-based SISR. Due their low complexity and helpful feature extraction abilities, CNNs have been widely used for SISR task. SRCNN [12] pioneered the use of deep convolutional neural network (CNN) architectures specifically designed for single image super-resolution (SISR). SRCNN consists of only three layers: patch extraction, non-linear mapping. and reconstruction. It has demonstrated competitive performance compared to traditional non-deep methods, inspiring the development of numerous lightweight CNN approaches in the SISR field. ESPCN [13] introduced compact network
'The numerical values the axes indicate the layer indices.

---

architecture that employs sub-pixel convolutional layers to upscale low-resolution image features. In contrast, LapSRN [14] utilizes image structure priors across different pyramid representations. resulting in improved performance while minimizing computational overhead. Taking inspiration from dictionary-learning models [7 LAPAR [8 learns linear coefficients associated with pre-defined basic up-sampling kernels to produce an optimal pixel-specific kernel, achieving superior super-resolution results. LatticeNet [19] designs a parameter-efficiet convolutional lattice block to extract hierarchical contextual features. Despite their computational efficiency, CNN-based models are limited in terms of long-term aggregation due to content-invarian similarity optimization.
Transformer-basee SISR. Recently, Transformer- techniques 25 have achieved remark- able outcomes in SISR but still suffer from high complexity. The computational cost of non-local self-similarity modeling increases quadratically with the size of the image. Inspired by the success of Swin Transformer [26). numerous Hindow-based Transformer frameworks emerged address the efficiency of SISR. For example, SwinIR 77] introduces a residual window- based transformer block (RSTB) for image feature extraction and similarity -baed aggregation, outperforming previous CNN-based and Transformer-based approaches. DLGSA-1 (27) proposes a global sparse attention technique enhance the aggregation of relevant tokens. NGswin 28 incorporates the N-Gram context to attain a larger receptive field, activating more neighhoring pixels for effective aggregation. However, when it comes to lightweight setups, the optimization of coupled feature extraction and similarity calculation is limited, resulting in inferior performance.
Efficient Transformers. On the other hand, some advanced transformers have been proposed to reduce the computational complexity, enhancing inference or training efficiency. ShareFormer [29] presents a local similarity map-sharing scheme between neighboring attention layers for lower latency. Thus. ShareFormer shares a static similarity map for neighboring attention layers while UPS calculates dynamic similarity maps with layer-refined features in shared projection space.
Skip-Attention 30 cuts off some intermediate attention layers to improve efficiency and performance for high-level tasks. LaViT [I proposes a residual-based attention downsampling that fuses the initial calculated attention scores guide the aggregation of the following layers, resulting in faster efficiency and improved classification accuracy.
Therefore, Skip-Attention and LaViT follow the existing coupled optimization scheme (reduce some attention calculations). and UPS proposes a decoupled learning strategy to enhance performance. We will cite the insightful studies and add this discussion to revised paper.
Understanding Swin Transformer
proposes an effective self-attention mechanism, achieving long-range information capture at a lower computation complexity. Inspired by Swin Transformer,
Preliminaries. Swin Transformer 266
several subsequent methods 32
28 dedicated solving lightweight SISR have emerged, consis- tently enhancing the quality of resor-re images. Fig. (1a) illustrates the general framework architecture of the Swin Transformer-based SISR method. consists of three primary components: shallow head module. a deep image feature extraction and aggregation (FEA) module, and tail reconstruction module. The head module tasked with converting the input low-resolution RGB image into high-dimensional feature space. The FEA module, the key role in the whole architecture, composed of multiple (N) Swin Transformer layers (STLs). Each STL has two main objectives: (i) extracting image features and (ii) modeling similarities using learnable projection space. The former focuses on capturing essential image features, while the latter employs window-based self-attention to facilitate spatially adaptive aggregation. Notably, similarity modeling optimizes projection space to obtain pixel-wise correlations, which is achieved by projecting image features into the learnable projection space and calculating similarity scores. Finally, the tail module generates the final high-resolution output image, completing the SiSR process. In the subsequent section, we will delve into the details of the STL, with a particular emphasis deep feature extraction and similarity modeling aspects.
3.1 Decomposing Swin Transformer Layer
Efficient deep feature extraction and similarity modeling are accomplished by the Swin Transformer Layer (STL), the fundamental unit within the FEA module of the Swin Transformer. Illustrated in
3

---

General transfomee-heeet architecture for SISR
Feature ex traction (FE)
M Matrix dot Softmax
Element wise multiplication
LR
Projection space (PS)
Cosine ReL.U
FEA
HR
(b) Previous Swin Transfomeess
Layer specitic optmizzaiin
S
Fi-t
V
Xiist
STL
Visc
Ajist
STL
Visst
(c) (Ours)
Uq
United PS Opt
Layer-speclfic FE optimization
O
Q
Fi-1
P
x
Xliet
D-STL,
Figure 2: Overview of Transformer- based architecture for lightweight SISR. There are three main components: (i) head shallow feature extraction module, (ii) deep feature extraction and aggre- gation (FEA) module consisting of Swin Transformer layers (STL)..... STLx), and (iii) tail reconstruction module. Previous transformers (i.e., SwinIR 77). NGSwin 28 synchronously perform multiple layer-specific deep image feature extraction (FE) and projection space (PS) optimization within Swin Transformer Layer (STL). In contrast, develop decoupled Swin Transformer Layer (D-STL) in UPS to optimize per-layer feature extraction and a unified projection space ("PS,," defined by learnable projection matrix UQ).
Fig. b)), in the i-th STL, the process begins by employing a convolutional layer to extract deep image feature F. from an input feature F,-1, which the output of the preceding (i 1)-th STL.:
F = Conv(F;-1).
(I)
Subsequently, the STL executes conventional window-based self- self-attention mechanism. comprising four basic steps: (i) sindow-partitioning (ii) deep feature projection for similarity calculation, (iii) aggregation based on similarity to merge neighboring pixels, and (iv) patch merging.
(i) Window-partitioning. Initially, the updated image feature F. is reshaped into HW non- MT
overlapping patches, each with shape of M2 C, where M represents the spatial size of each patch and C' is the channel dimension.
(ii) Layer-specific projection. Following window- mandoo-partitioning,, each divided image patch X; from F is projected to generate the corresponding query, key, and value matrices Q, K,, Vi:
Qi = X.PP. K, X,P,K == X
(2)
where p.o, PK PV R(dxC} denote the learnable projection parameters specifie the i-th STL, while Q. K,, V/2 E R(M[xd) represent the projected features of patch X, and is the projection dimension. The similarity matrix is then computed:
Q.K?
+B)).
S Soft
Vd
where B, represents a relative position encoding, and S; the predicted similarity map for the X,,
(iii) Similarity Aggregation. Later on, neighboring information within the patch X, is aggregated based on the computed similarity map S,:
Y S.V.
(4)
(iv) Patch Merging. Finally, all the aggregated image patches are reshaped into 2D image feature which is fed into the next STL for further processing.
With sufficient model capability, i.e., millions of parameters, SwinIR 77, SOTA Swin Transformer SISR model, exhibits strong abilities for the SISR task. However, in resource-constrained. lightweight settings as previously mentioned, it potentially poses challenges to simultaneously optimize deep image feature extraction and projection space. We will compare the per-layer projection space optimization with our proposed UPS scheme later.
Discussion.

---

Ground Truth
lligt
NGSSii
UPS
Ground Truth
SwinlR-ligh
NGSwin
UPS
DI PSNR (dRt) 2.09/17.29 2.96 17.61
4,95/ 19.57
D1 PSNR (d13) 9.29/24.39
6.15/24.37
22.27/28.43
Comparison between SOTA SISR models and ours. We show the SR results overlaid with the local attribution map (LAMM
Figure 3:
33 of each model. The LAM visually illustrates the activation of local and non-local pixels involved in super-resolving the highlighted patch within the red box. The numbers beneath are the DI (t) 33 and PSNR (1) values. Zoom in for better visual comparison.
Unified Projection Sharing for Lightweight SISR
To address the entanglement optimization of image feature extraction and similarity modeling, we introduce a Unified Projection Sharing (UPS) technique for lightweight SISR. Fig, 2b)) (c) summarizes the optimization schemes of existing Transformer-basee proposed UPS. As can be seen, previous methods typically focus on jointly optimizing deep image features and similarity modeling within each layer. In contrast, UPS adopts shared projection space for similarity modeling, allowing layer-specific feature extraction while separating the optimization of similarity calculation.
Overview.
SISR frameworks and our
4.1 Unified Projection Sharing
We follow the general framework structure of Swin Transformer but use decoupled projection space optimization. As shown in Fig. cc). UPS consists of three basic modules, namely the convolutional head module, FEA module, and reconstruction tail module. In the FEA, we develop a decoupled Swin Transformer layer (D-STL) for deep image feature extraction, while optimizing a unified projection space for similarity modeling. Next, we will provide detailed description of our D-STL.
4.2 Decoupled STL (D-STL)
We take the i-th D-STL for illustration. Given an input image feature F;-1 produced by the last (i (i- 1)-th D-STL, we aim to perform feature updating as well as self-similarity-baseee aggregation. Similarly, we adopt the Eq, too conduct deep image feature extraction and obtain the transformed image feature F. Then we employ the window-pare sarddooptttiiiinnngg process to reshape the F into HW non-lapped image patches.
M
Unified Projection. Unlike the layer-specific projection scheme in Swin Transformers, we introduce a layer-invariant (unified) projection space defined by learnable matrix Uº R(DxC) (D refers to the unified projection dimension) and project the deep feature X; on this unified projection space:
Q: X,U9, V, ==x; X,
After that, we consider the calculation of the self-similarity in the unified projection space. Motivated by ReLUFormer (34 that addresses the over-centralized distribution Softmax by incorporating ReLU activation for self-similarity calculation, we get the similarity scores as:
S (RoLi(Cssiee() Qi, Q?) B.).
Note that we conduct normalization operation for the projected image features Q 6 Subsequently. we utilize the Cosine similarity metric, followed by a ReLU activation function, to obtain the final similarity map S. We also assess our design in Sec. 5.3 Finally, leveraging the calculated similarity map S, we perform image feature aggregation using Eq.
Discussion. In Algorithm. 1, 2, we provide bide-by-side illustrations of standard STL and our D-STL and highlight the differences between the two methods. The STL in previous Swin Transformers learns the coupled projection spaces and deep image feature extraction. In contrast, by a unified projection optimization scheme, each of our D-STLs only focuses on the deep image feature extraction. It largely
"To decrease the model complexity, we set the K; to be identical to the Q. the SISR task typically involves only one data modality.
5

---

Algorithm 1 Pseuda Code of the é-th STI.
Algorithm 2 Pseudo Code i-th Decoupled STL.
1: Require: Input Pt-1. window size
1: Require: Fi-1. M. unified projection matrix UR
2: Feature extraction: F.- Conv(F,--
3: Parttioning: X!les Partitioning fi, Ad)
2: Feature extraction: F; Conv( 1)
4: Detine aggregated patch list: y.list
3: Partitioning: X = Partitioning( (Frs M)
5: for X, in X! do
4: Define aggregated patch list: Y!
5: for X, in X! do
6:
Projection: Qi K. X.P X.PR, X.PY
6:
Projection: Q. V. =x,
7:
Similarity cal: 8; SoftMax
B3
7: Similarity cal.: S. ReLU ReLU(CCssneeQ2 Q,") + B,)
8:
Aggregation: S.V.
8: Aggregation: Y; = S.V.
9:
yillas -Xppend(Y))
9: V: (Yppend(()))
10:
end for return Reshape(XX.!nn
10: end for
11:
11: return Reshapo((Y/ing
reduces the overall optimization complexity by learning the similarity modeling in a unified projection space throughout all D-STL layers. As shown in Fig. [3] compared with SOTA lightweight Swin Transformers, UPS activates more non-local pixels and restores correct fine-grain image structures.
Table 1: Quantitative comparison with SOTA lightweight SISR methods on multiple benchmark datasets. The best and second-best results on the default training setting (DIV2K) are highlighted in red and blue, respectively. The "+" indicates that the two methods are trained on the DF2K dataset. We use bold to highlight the lowest FLOPs of Transformer- based methods. All FLOPs (also in Tab.
2b, 3,4) are calculated with an output size of 1280 x 720.
Scale
Parameters (K)
FLOPs (G)
Set5
Ser14
BSD100
Urban100
Manga109
Method
PSNR SSIM
PSNR SSIM
PSIR
PSNR/SSIM
PSNR SSIM
IMDN
694
158.8
15.00/0.99003
33.63/0.911
32.19/0.899
32.17 0:9283
0.9774
RFDN-L.
626
145.8
38.08 0.9606
/0.919
32.18/02 0.8996
32.24/ / 0.9290
/0.95/0.9773 0.9773
SwinlR-light light
910
244.4
38.14 0.9611
33.86 0.9206
32.31 0.9001
32.76 0.9340
39.12/ 0.9783
DLGSA light
X2
745
170.0
38,20 0.9612
33.89/ 0.9220
32.30/0.9011.
32,94/ / 0.9355
39.29 / 0.9780
Omni-SR
772
194.5
38.22 0 0.9613
33.98 0..9210
32.36/ / 0,9020
33,05 / 0.9363
29 00.9784
UPS
824
162.5
38.26 0.9642
34.16 0,9232
32.42/ 7 0.9031
33,08/ 7 0.9373
39.62 7 0,9900
SwintR-S S
x2
497
107.3
38.06 0 0.9603
33.80/ 0.9186
32.23/ 0.9006
32.24/ 0.9301
38.76/ 0.9778
UPS-S
x2
453
90.6
38.16 0,91638
34.00/ 0,9220
0.9023 32.36
32.79/ 0.9346
39.26 0,9790
Omni-SR-
772
194.5
38,29 7.99617
34,27 0.9238
32.41 0.9026
33.30/ 0.9386
39,53/ 0.9779
UPS+
X2
824
162.5
38.31 0.9643
34.37 0.9247
32.43 /0.9032
33.34 / 0.9388
39.80/ 0.9802
IMDN
703
71.5
34.36 / 0.9270
0.8417 30.32/
29.09 09.09/ 0.88046
28.17 0 0.8519
33.61 0.9445
RFDN-L
633
65.6
34.47 0.9280
0.8421 30.35/
29.11 0..003
28.32/ 0,8547
33.78 0.9458
SwinIR light
918
110.8
34.62 0.9289
30.54/
29: 0.8082
0.8664
D.94488
lighh
x3
752
75.4
34,70 0. 0.9295
30.58/ 0.8465
29.24/ 0.8089
28. 83/ 0.8663
34.16 / 0.94483
Omni-SR
780
88.4
34.70 0,9294
30.57 0. 0,8469
29.28 0.8094
28.84/ 0.8656
34.22/ 0,9487
UPS
832
72.4
34.66 0,9322
30.72/ 0,8489
29.31/ 0,8114
28.98 / 0.8685
34.53 0,9505
SwinlR-S S
x3
503
47.9
34.38 0.9281
0.8448 30.46
29.15 0.8073
28.37 / 0.8572
33.77 0.9464
UPS-S
x3
459
40.4
34.53 0,9312
30.55
0.8463
29.24 0.8093
28.60/ 0.8614
34.12 0.9484
Omni-SR
780
88.4
34.77
0.9304
30.70/
8489
29. 33 0.8111
0.8712
34.64 /.9550
UPS+
x3
832
72.4
34.78 0.9325
30.78/ / 0.899
29.36 0.8122
29.28 / 0.8728
34.84 0.9517
IMDN
715
40.9
32.21 0.8948
28.58 0.7811
0.7353
26.04 0.7338
30,45 0.9075
RFDN-L
643
37.4
32,28 / 8957
28. 61 00/00.7 0.7818
27.58 0.7333
26.20/ 0,7883
30.61 / 9096
SwinlR-light
930
63.6
32.44 0,8976
28.77 0.7858
27.69) 0.7406
26.47 0.7980
30.92 0.9151
DLGSA-ligt
X4
761
42.5
32,54 O. 098933
28.84/ 0,7871
27.73 0.7415
26.66/ 0.8033
31.13 0.9161
Orni SR
792
50.9
32.49 / 0.8988
28. 0.78559
27.71 0.7415
26.64 / 0.8018
31.02/ D.9151
UPS
843
41.3
32.50 0.9024
28,90 0.7892
27.79 / 0.7435
26.83/ / 0.5073
31.39/ 0.99194
SwinlR-S
512
27.3
32.14/ 8955
28.67 0.7832
27.63 0.7382
26.22 0. 0.790
30,68 0.9111
UPS-S
x4
468
23.0
32.41 0,9008
28, 80 0,7863
27.73 7.7411
26.58 7 0,7905
0.9163
Ornii- SR+
X4
792
50.9
32.57 0.8993
28.95 0.78998
27.81 0.7439
26.95 / 0.8105
31.50/0.9192
UPS+
843
41.3
32.60 0:9029
28.97 0. 0.7896
0..837
0.8136
30.79/
5 Experiments
5.1
Settings
Implementation Details. Our UPS model is developed by PyTorch and incorporates several com- monly used data augmentation techniques, including random cropping, vertical/horizonnal flipping, and rotation. During training, we employ the Adam (35] optimization with cosine annealing [36]. starting with an initial learning rate of 4e We set the batch size as 32 and the input image size as 64 x 64. Training is conducted for 600K iterations, utilizing four NVIDIA RTX 3090 GPUs.

---

Scalable Model Size. Generally, we train our UPS and UPS-S with different configurations. Our UPS model follows the setting of SwinIR-light 77. consisting of 4 D-RSTB blocks with 6 decoupled Swin Transformer layers (channel size: 60). Additionally, our UPS-S model is more lightweight with 4 compact D-RSTB blocks with varying numbers of decoupled Swin Transformer layers (6. 4, 4) and channel size of 48. Training various UPS models requires approximately 2-3 days.
Benchmark Datasets. Following previous studies dataset for training. Subsequently, we conduct comprehensive evaluations on several widely-used SISR benchmarks, including Set5 [38]. Set14 39,, BSD100 [.. Urban100 [40], and Manga [T
28
18] we utilize the DIV2K (37 image
Our quantitative comparison is based on PSNR and SSIM. Consistent with established research, we report the results specifically for the Y channel derived from the YCbCr color space.
5.2 Comparison with SOTA Methods
We perform extensive comparisons with a wide range of lightweight SISR models: MAFFSRN (ECCV20) 42), LAPAR-A (NeurIPS20) [8]. LatticeNet (ECCV20) [19], RLFN (CVPRW22) 43),
SwinIR-light 77. NGswin [28], SwinIR-NG [28], and DLGSA-I (ICCV23) (ICCA))))CV233 277. More comprehensive comparisons with early SOTA lightweight models can be accessed in supplementary material.
Table 2: Results of inference time (ms), FLOPPs (G) and GPU memory usage (MB). The speed is tested on an NVIDIA GeForce RTX 2080Ti GPU with an input size of 256 x 256 under x2 lightweight SISR. FLOPs is calculated output resolution of 1280 720.
Metries
RFDN-L LatticeeNe ligg- Omni-SR SwinIR-light UPS
Time (ms) |
13
18
225
112
175
119
FLOPPs (G)
146
170
170
195
244
163
Memory (GB) L
1577
1639
1800
1842
2051
1785
Quantitative Comparison. Tab illustrates the quantitative evaluation. Our proposed UPS consis- tently outperforms existing methods across all benchmarks. Notably, UPS exceeds the second-best model by over 0.33dB on the Manga 109 dataset 41 for the x2 setting. Additionally, our model shows significant improvements over SwinIR- -liht 77 achieving improvements of up 0.5dB, 0.55dB, and 0.47dB on the Manga109 dataset [1 for the x2, x3, and x4 settings, respectively. while using fewer parameters. These results confirm the effectiveness of our decoupled optimization strategy. Significantly, when constrained by model complexities, our UPS demonstrates superior performance over SwinlR-light 7 As shown in Tab our approach achieves a 0.55dB increase PSNR for x2 super-resolution on the Urban 1 100 dataset [40) highlighting the advantages of our decoupled optimization in feature extraction and similarity modeling under compact parameter conditions.
Additionally, we evaluate the inference efficiency of various state-of-the-art (SOTA) lightweight single image super-resolution (SISR) models. As shown in Table UPS reduces the overall inference cost by 33% in terms of FLOPs compared to our baseline model, SwinlR- -ight
Last, we have extensively explored the benefits of UPS for real-world SR and other frameworks, including HAT and DRCT, under both lightweight and parameter-intensive scenarios. Our experi- ments show that the proposed UPS consistently enhances efficiency and performance across all these settings (real-world SR, lightweight, and SISR tasks).
For real-world super-resolution (Real-world SR), shown in Tab. of the PDF file (also the table below), our proposed UPS-GAP outperforms other state-of-the-arr GAN-based 44 451 and even Diffusion-basse methods (Reshift [46] and StableSR (47)) in terms of NIQE, NRQM, and PI metrics, achieving the best quantitative results 15.09/6..4/44.99 This confirms the effectiveness of UPS for real-world SR tasks.
Qualitative Comparison. Fig. presents some visual examples. is evident that UPS is capable of producing correct image textures with fewer super-resolved artifacts. Conversely, previous CNN- based and Transformer-based frameworks either fail to reconstruct clear image patterns or suffer from unplcasing artifacts. We provide more visual comparison in the supplementary material.
7

---

Table Non-reference results of real-world SISR on RealSRSet 44
Metrics
BSRGAN
RealSR ResShift
StableSR
SwinIR-GAN
UPS-GAN
NIQE +
5.66
5.83
8.37
5.24
5.49
5.09
NRQM T
6.27
6.32
4.56
6.12
6,48
6.84
PI
4.75
4,40
7.03
4.66
4.72
4.19
BSD100 091
LR
LAPAR-A
NGSSwn
Manga 109 003
LR
LAPAR-A
NGSwin
NUU-HAM
AKU-HAM
AKU-HAM
Ground
SwinIR-light
Ground Truth
SwinlR-
AKU-HAM
AKU-HAM
AKU-HAM
U1bban000
LR
LAPAR-A
NGSwin
DIV2K 0820
LR
LAPAR-A
NGSwin
Ground Truth
SwintR-light
UPS
Ground Truth
SwinlR light
UPS
Figure 4:
Qualitative comparison between LAPAR-A 18. SwinIR-light 7 NGSwin 28 and UPS
(ours) on four popular benchmarks (BSD100 (T]. Urban 100 [40), Mangal 41 and DIV2K (37.)
under x 4 setting. Our predictions present more detailed textures and fewer artifacts.
5.3 Ablation Studies
To verify the effectiveness of our design, we conduct series of comprehensive experiments. Note that we primarily adopt the UPS-S architecture for fast analysis. We generally report the quantitative results on the S benchmark under the x setting.
Different Groups of Projection Space. Unlike conventional attention-based frameworks, our approach introduces a unified projection space for similarity calculation. In this evaluation, we investigate the impact of incorporating additional groups of projection space. The quantitative results in 5 (Left) shows that higher number of block-wise projection groups reduce performance compared to our unified projection method. Additionally, layer-specific projection optimization exhibits inferior performance. Fig 5 (Right) confirms that our unified projection-sharing scheme outperforms models with multiple projection spaces. This experiment highlights the challenges of coupled optimization in image feature extraction and similarity modeling.
Similarity Calculation. Differing from the conventional similarity calculation paradigm (Matrix dot product + SoftMax), we incorporate the Cosine distance followed by a ReLU activation for similarity computation. Here, to assess the effectiveness of our choice, we train different models that utilize various combinations of distance metrics (Matrix dot and Cosine) and activation functions (SoftMax and ReLU). The results, as displayed in the left Fig. 6 indicate that our design (Cosine + ReLU) achieves the best performance among all the competing strategies. Additionally, we provide corresponding visual comparison the right Fig. & It shows that our design activates more non-local pixels, resulting in larger valid receptive field and more precise reconstructed image details.
Projection Matrix Dimension. We investi- gate the impact of various projection dimen- sions for similarity calculation, ranging from 8 to 256 (D = 8, 32, 64, 128, 256). Tab. 4 in- dicates that performance initially improves as the projection dimension increases, but slightly drops after reaching extremely high dimensions (e.g., 256). We also observe that higher projec- tion dimensions lead increased computational costs both learnable parameters and FLOPS.
Table 4: The effect of varying projection dimen- sions on similarity calculation.
Dimension
256
PSNR (dB)
33.83
33.86
33.90
34.00
33.96
SSIM
0.9206
0.9207
0.9207
0.9220
0.9221
#Params
452K
452K
452K
453K
454K
FLOPS
107G
113G
119G
124G
159G

---

Ground Truth
G19
G1 (Qun))
Proj. Group G19
G4
GI (Ours)
PSNR (dB) 33.63
33.91
33.97
34.00
SSIM
0.91860.9205
0.9216
0.9220
DUPSNR
5.5116250
6. 46220.4d
6.14/5
Figure 5: Analysis of several UPS-S models with different projection groups, including the layer- specific projection model (consists of 19 attention layers). (Left); PSNR/SSIMs are examined for quantitative comparison. (Right): Visual results on the Urban 100
39 benchmark under x2 setting.
Matrix Cosins SoftMax ReLU PSNR/SSIM
33.60/0.9192
33.84/0.9202
DEPSNR
0.57/24 23dB
87dB
0.3624 SSIB
33.41/0.9169
G1
33.61/0.9208
33.73/0.9194
34.00/0.9220
DEPSNR
7498 88:RR
2. 19.944 4MAR
5.M27.3208
Figure 6: Impact of different similarity calculation methods. The left Table shows the quantitative
results of employing different similarity calculation methods on the Urban 100 (39 (x2). The right
figure gives visual example illustrate the SR results overlaid the LAM 33 maps of each model. The numbers beneath are the DI (t) 33 and PSNR (t) values.
SwinIR- light
UPS
Layer 12
Layer 23
Layer 12
Layer 23
+>
LR
GT
X12
S
12 X
S23
Y
X
S
122
X
23
S23
23
Figure 7: Visual comparison of layer-specific projection optimization and our proposed UPS scheme. UPS achieves better similarity calculation and yields better image structural restoration.
Similarity Map Visualization. To better un-
derstand the effect of our pnpjecttoo-sharinn scheme, we visualize the deep image features of our baseline model (SwinIR-Jh) -ligtt and UPS. For an LR image, Fig. 7 illustrates the updating of deep image features layers 12, in both the base and UPS models. The right arrows indicate the input-output data flow for each layer. Following that, we present detailed visualization for Alg. 1,2 in our paper Notably, the similarity maps S12, S23 produced by UPS (highlighted in the red box) are more effective in aggregating neighboring pixels, resulting in sharper final SR image.
5.4 Extension Image Denoising and JPEG Image Deblocking
In this section, we explore the benefits of applying UPS for other image restoration tasks. While SwinIR (baseline model) requires millions of parameters for image denoising and deblocking, our lightweight UPS frameworks handle these common low-level restoration tasks more efficiently. As shown in Fig. 8, the results, indicate that UPS achieves performance comparable to the large baseline model (SwinIR) while requiring only J of the model complexity on Denoising. Additionally, the compact baseline models exhibit inferior performance relative to our UPS. We describe the framework of these models Sec. of our appendix.
X;, Y; denote input and output, and S; is the similarity matrix.

---

Ground Truth
SwinlR
SwinIR-C C
UPS
Tasks
Metrics
SwinIR
SwinTR-C
UPS
Deblocking
PSNR
29.86
29.63
29.98
9
3.49M
Denoising
PSNR
28.56
28.20
28.37
50
Param.
11,50M
0.959M
0.873M
Figure 8: Extension on other image restoration problems. (Left) UPS attains comparable results compared with its larger baseline SwinIR and outperforms SwinIR-C with similar model sizes. (Right) visual example of image deblocking.
LR/HI Depth mup
SwinIR Rigght
UPS
Settings
Metrics
SwinlR-light
UPS
PSNR
47.25
47.79
x4
SSIM
0.994
0.995
RMSE
2.339
2.198
PSNR/SSIM/RMSE
33.43/0.951/5.156
34.1510.9564.725
PSNR
37.25
37.98
x16
SSIM
0.969
0.972
RMSE
7,832
7.236
PSNR/SSIMIR
54.04/0.963399966
36,01.0 36.01..0.7703.000 970/5, 966
Figure 9: Generalization comparison between our baseline model and UPS. The quantitative results on NYU V2 [48] (x4, 4, x 16) are displayed in the left table, while the right figure illustrates two visual examples. Additionally, normalized error maps are included in the left corner facilitate comparison.
5.5 Extension 2: Depth Map Super-resolution
We also compare it with our baseline model (SwinIR-light light) the depth SR task. To do this, without training on any depth images, we directly test the two models on the NYU V2 (48) depth benchmark under x4 and x 16 settings. We use the PSNR, SSIM, and RMSE (the root-mean-squae error) metrics for quantitative evaluation. The quantitative results are shown the left Fig. consistently outperforms its baseline model on all the objective metrics. For instance, UPS exhibits superior performance compared to SwinlR-light with a PSNR improvement of 0.54dB (0.73dB) for x4 (x 16) configurations. The visual examples in the right Fig. illlsstaaee that UPS generates clearer structures, leading to higher accuracy when compared to our baseline model.
g We can see UPS
In addition to these two extensions, we also explore the improvement over SwinIR, DRCT, HAT for both lightweight and classic SISR to comprehensive analyze the potential capabilities of the proposed UPS. Please refer to the appendix sections.
6 Conclusion
In this paper, we propose a unified projection sharing (UPS) technique for lightweight SISR. A layer-invariant projection space is optimized for similarity modeling. Comprehensive experiments have demonstrated the effectiveness of the proposed decoupled learning algorithm. Notably. UPS achieves Kate-of-the-ar performance on multiple SISR benchmarks. Moreover, UPS-S exhibits competitive results compared with leading approaches, while requiring fewer learnable parameters. Additionally, experiments indicate that our proposed UPS demonstrates superior data efficiency. Code will be made publicly available at httpps wtt.hh/.coo.eerroooo3////SSS NPS200
Acknowledgments. This work is partially supported by Shenzhen Science and Technology Program CQTD2021081 1090149099 and also the Pearl River Talent Recruitment Program 2019QN01X226. The work was supported in part by the Basic Research Project No. HZQB-KCZYZ-200 Hetao Shenzhen-HK S&T Cooperation Zone, Guangdong Provincial Outstanding Youth Fund (No. 202381515020055 the National Key R&D Program of China with grant No. 2018YFB1 1800800 by Shenzhen Outstanding Talents Training Fund 202002, by Guangdong Research Projects No. 20172T07X152 and No. 2019CX01X104, by Key Area R&D Program of Guangdong Province (Grant No. 20188030338001 (2000)bbu by the Guangdong Provincial Key Laboratory of Future Networks of In- telligence (Grant No. 2022B1 121112220111 212010001), and by Shenzhen Key Laboratory of Big Data and Artificial
of
'NYU V2 consists of 449 testing depth images at size of 480 640. We use bicubic interpolation to generate the now-resolution depth images.
10

---

Intelligence (Grant No. ZDSYS2201002514099000 It is also partly supported by NSFC-61931024. NSFC-62172348, and Shenzhen Science and Technology Program No. JCYII000005331445000000
References
[1] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1728-5739, 2022.
[2] Jiezhang Cao, Qin Wang, Yongqin Xian, Yawei Li, Bingbing Ni, Zhiming Pi, Kai Zhang, Yulun Zhang, Radu Timofte, and Luc Van Gool. Ciaosr: Continuous implicit ceentioonii--ttentiin network for arbitrary-scaat image super-resolution. on Computer Vision and Pattern Recognition, pages 1796-1 1807, 2023.
Proceedings of the IEEE/CVF Conference
[3] Hao- Wei Chen, Yu-Syuan Xu, Min-Fong Hong, Yi-Min Tsai, Hsien-Kai Kuo, and Chun-Y Lee. Cascaded local implicit transformer for arbitrary-scale super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18257-18267, 2023.
[4] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxim: Multi-axis mlp for image processing. CVPR, 2022.
[5] Xiaoqiang Zhou, Huaibo Huang, Ran He, Zilei Wang, Jie Hu, and Tieniu Tan. Msra-sr: Image super-resolution transformer with multi-scale shared representation acquisition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12665-12676, 2023.
[6] Qiang Zhu, Pengfei Li, and Qianhui Li. Attention retractable frequency fusion transformer for image super resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1756-1763, 2023.
[7] Jingyun Liang. Jiezhang Cao, Guolei Sun, Kai Zhang. Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF international conference computer vision, pages 1833-1844, 2021.
[8] Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhut- dinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprini arXiv: 1207.0000. 2012.
[9] Xiangtao Kong, Xina Liu, Jinjin Gu, Yu Qiao, and Chao Dong. Reflash dropout in image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6002-6012, 2022.
[10] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference machine learning, pages 3519-3529. PMLR, 2019.
[11] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human seg- mented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2, pages 416-423. IEFF, 2001.
[12] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. IEEE transactions on pattern analysis and machine intelligence, 38(2):295-307, 2015.
[13] Wenzhe Shi. Jose Caballero, Ferene Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop. Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1874-1883. 2016.
[14] Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
11

---

[15] Yaniv Romano, John Isidoro, and Peyman Milanfar. Raisr: Rapid and accurate image super resolution. IEEE Transactions on Computational Imaging. 3(1):110-125, 2016.
[16] Younghyun Jo. Scoung Wug Oh, Jaeyeon Kang, and Seon Joo Kim. Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation. In Proceedings of the IEEE conference computer vision and pattern recognition. pages 3224-3233 2018.
[17] Xuecai Hu, Haoyuan Mu, Xiangyu Zhang, Zilei Wang, Tieniu Tan, and Jian Sun. Meta-sr: magnification - network for super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1575-1584, 2019.
[18] Wenbo Li, Kun Zhou, Lu Qi, Nianjuan Jiang, Jiangbo Lu, and Jiaya Jia. Lapar: Linearly- assembled pixel-adaptive regression network for single image super-resolution and beyond. Advances in Neural Information Processing Systems, 33:20343-20355 2020.
[19] Xiaotong Luo, Yuan Xie, Yulun Zhang, Yanyun Qu, Cuihua Li, and Yun Fu. Latticenet: Towards lightweight image super-resolution with lattice block. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXII 16, pages 272-289. Springer, 2020.
[20] Zheng Chen, Yulun Zhang, Jinjin Gu, Linghe Kong, Xiaokang Yang, and Fisher Yu. Dual aggre- gation transformer for image super-resolution. In Proceedings of the IEEF/CVF international conference on computer vision, pages 12312-12321, 2023.
[21] Xiangyu Chen, Xintao Wang, Jiantao Zhou, Yu Qiao, and Chao Dong. Activating more pixels in image super-resolution transformer. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 02367-22377 2023.
[22] Wenbo Li, Xin Lu, Shengju Qian, Jiangbo Lu, Xiangyu Zhang, and Jiaya Jia. On efficient transformer-basee based image pre-training for low-level vision. arXiv preprint 11X1v:21/2.00175 2021.
[23] Jinsu Yoo, Taehoon Kim, Sihaeng Lee, Seung Hwan Kim, Honglak Lee, and Tae Hyun Kim. Enriched -nntrranfforer feature aggregation networks for super-resolution. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4956 4965, 2023.
[24] Yupeng Zhou, Zhen Li, Ch Guo, Song Bai, Ming-Ming Cheng, and Qibin Hou. Srformer: Permuted self-ateention for single image super-resolution. arXiv preprint arXiv:2303.09735 2023.
[25] Ke Chen, Liangyan Li, Huan Liu, Yunzhe Li, Congling Tang, and Jun Chen. Swinfsr: Stereo image reser-renolution using swinir and frequency domain knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1764-1774, 2023.
[26] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[27] Xiang Li, Jiangxin Dong, Jinhui Tang, and Jinshan Pan. Digsanet: Lightweight dynamic local and global self-attention networks for image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision pages 12792-12801, 2023.
[28] Haram Choi, Jeongmin Lee, and Jihoon Yang. N-gram in swin transformers for efficient lightweight image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2071-2081, 2023.
[29] Yuning Cui, Yi Tao, Luoxi Jing, and Alois Knoll. Strip attention for image restoration. In Edith Elkind, editor, Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pages 645-653 International Joint Conferences on Artificial Intelligence Organization, 2023. doi: 10.24963/ijcai.202377 2023/777. URI, hattps://dant hww.dd...ggg.g/o org/10. 24963/ ijcai 2023/72 Main Track.
12

---

[30] Shashanka Venkataramanan, Amir Ghodrati, Yuki M Asano, Fatih Porikli, and Amirhossein Habibian. Skip-attention: Improving vision transformers by paying less attention. arXiv preprint M.Xiv:3301.02240, 2023.
[31] Shuoxi Zhang, Hanpeng Liu, Stephen Lin, and Kun He. You only need less attention at each stage in vision transformers. In Pruceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6057-6066, 2024.
[32] Marcos V Conde, Ui-Jin Choi, Maxime Burchi, and Radu Timofte. Swin2sr: Swinv2 transformer for compressed image super-resolution and restoration. In European Conference on Computer Vision, pages 669-687, Springer, 2022.
[33] Jinjin Gu and Chao Dong. Interpreting super-resolution networks with local attribution maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9199-9208, 2021.
[34] Mitchell Wortsman, Jachoon Lee, Justin Gilmer, and Simon Kornblith. Replacing softmax with relu in vision transformers. arXivv preprint arXiv:2309.08588 2023,
[35] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[36] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 1608v:160..0380 2016.
[37] Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 126-135, 2017.
[38] Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alheri-Morel. Low- complexity single-image super-resolution based on nonnegative neighbor embedding. 2012.
[39] Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse- representations. In Curves and Surfaces: 7th International Conference, Avignon, France, June 24-30, 2010, Revised Selected Papers 7, pages 71 1-730.. Springer, 2012.
[40] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super- resolutoso from transformed self-exemplars. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5197-5206, 2015.
[41] Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto. Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-based manga retrieval using manga109 dataset. Multimedia Tools and Applications, 76:21811-2188
2017.
[42] Abdul Muqeet, Jiwon Hwang, Subin Yang, JungHeum Kang, Yongwoo Kim, and Sung-Ho Bae. Multi-attention based ultra lightweight image super-resolution. In Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings. Part III 16, pages 103-1 118. Springer, 2020.
[43] Fangyuan Kong, Mingxi Li, Songwei Liu, Ding Liu, Jingwen He, Yang Bai, Fangmin Chen, and Lean Fu. Residual local feature network for efficient super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 766-776, 2022.
[44] Kai Zhang, Jingyun Liang. Luc Van Gool, and Radu Timofte. Designing practical degradation model for deep blind image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4791-4800, 2021.
[45] Xiaozhong Ji. Yun Cao, Ying Tai, Chengjie Wang. Jilin Li, and Feiyue Huang, Real-world super-resolution via kernel estimation and noise injection. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
[46] Zongsheng Yue, Jianyi Wang, and Chen Change Loy. Resshift: Efficient diffusion model for image super-resolution by residual shifting. Advances Neural Information Processing Systems, 36, 2024.
13

---

[47] Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin C.K. Chan, and Chen Change Loy. Exploiting diffusion prior for real-world image super-resolution. 2024.
[48] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Computer Vision-ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings. Part V 12, pages 746-760. Springer, 2012.
[49] Chih-Chung Hsu, Chia-Ming Lee, and Yi-Shiuan Chou. Dret: Saving image super-resolution away from information bottleneck. arXiv preprini arXiv:2404.00722, 2024.
14

---

A Appendix supplemental material
A.1 Improvements over SOTA Swin-transformers
In our main paper, we have explore the improvements over SwinlR-light for lightweight SISR task. Here, we discuss the potential improvement and generalization capability of UPS for more Swin- transformers (such as DRCT [49 and HAT (21) on both lightweight and classic SISR. Table. 5 shows our UPS is capable of enhancing several SOTA trin-traasformers with fewer parameters, FLOPs and inference latency.
Table 5: Quantitative comparison with SOTA lightweight models for x2 SISR. All the models trained on the DIV2K dataset for fair comparison. Inference time is tested at input size of 256 x 256 on an NVIDIA GeForce RTX 2080Ti GPU.
Parans/FLOPP/Time (K/G ms)
Sct5
Sct14
BSD100
Urban 100
Manga 109
Lightwight
PSNR / SSIM PSNR SSIM PSNR SSIM PSNR SSIM PSNR SSIM
SwinIR- light
910/244/ 175
38,1470.9611 33.86 7.9206 32.31 7 0,9012 32.76 0.9340 7,9978
SwinlR-light-UPS
843/ 163 119
38.26 0.9642 34.16 0.9232 32.42 0.9031 33.08 0.9373 39.62 0.9800
DRCT-light
1137 137 992
38.05 / 0.9632 33.76 0,9201 32.28 0.9012 32.48 18 38.87 0,9783
DRCT-light-UPS
996 125 85
38.06 0.9634 33.89 0.9213 32.30 0.9013 32.59 0,9325 39.27 / 0,9786
HAT-light
813/ 102 153
38.02 / 0,9612 33.88 0.9200 32.28 / 0.9016 32.64 / 0.9330 38.82 0.9783
HAT-light-UPS
797/91 136
38.16 0.9636 34.16/
/ 0.9223 32.36/ 0.9022 32.92
0.9351 39.35 / 0.9791
Table 6: Quantitative comparison with SOTA Classic models for x4 SISR. All the models are trained on the DF2K dataset for fair comparison. Inference time is tested at an input size of 256 x 256 on an NVIDIA GeForce RTX 2080Ti GPU.
Classic
Param Param/FLLP- (GVTime
Sct5
Sct14
BSD100
Urban100
Manga109
(Millions
SSIM
PSNR/
PSNR SSIM
PSNR SSIM
51.9/84//
32.92/0.9044
29.09
70.7900
27.92 0.7489
27.45/0.8254
SwinIR-UPS
10.73 (-1,))111)) (-1.17/ 17)
(421))542((1141)
33,29/ 0,9166
29.51 0.8027
28.24 0.7595
28,03 0.8538
32.96 0,9332
HAT
20.77
728 1419
33.04/ 0.9056
59.23/0.7977
28.00 0.7517
27.97 / 00.8368
32.48 / 0.9292
HAT-UPS
17.25 (-3.52)
(:95) 1224 (-19)
33.11/ 0,9998
29.29 0.7991
28.08 0.7548
2 0.88479
32.87 / 0,9319
DRCT
14.14 520
33.11 70,9064
29.35 / 0077984
28. 0.7532
28.06 0088378
32.59/ 0.9304
DRCT-UPS
12.31(-1 83y/ 482(-88)) 482(-39)/6691-422) (69(-142)
33.1770.9088
29.38 / 0.7989
28.20 0.7536
28.32 / 0.8416
32.68 / 0.9331
A.2 Data Efficiency
(a)
(b)
PSNR (dB) with 5% Training Images
34.0
33.5
33.9
33.0
33.8
(dB)
33.7
32.5
33.6
SwinIR-S
33.5
32.0
LAPAR-A
33.4
LatticeNet
33.3
31.5
UPS-S
0
10 20 30 40
70
100
0
10
20
30
40
Percentage Used Images
Iterations (5e3)
Figure 10: Comparison between several lightweight SISR models on Set14 [39] (x4). (a) Evaluation of using different numbers of training samples.
(b) Validation performance of different models.
In this part, We assess the data efficiency of several models, including LAPAR-A, LatticeNet, our baseline model SwinlR-S S [ (a more lightweight model with identical framework configuration as our UPS-S), and our proposed UPS-S. In addition to training all models on the complete training set, we gradually adjust the percentage of used training data. As shown in Fig a), UPS consistently outperforms other models regardless of training data size, demonstrating superior data efficiency, Fig bO) shows PSNR values during training iterations, with UPS-S converging faster and providing better early predictions. These results highlight the effectiveness of our UPS scheme.
15

---

A.3
Robustness Optimization of UPS
Intuitively, given noisy imput features, the error will be invertibly accumulated and affect the following
projection space optimization.
34.1
Fj
P1
FN
SR
LR
UPS(Ours) !SwinlR-ligg light
34.0
33.9
33.8
33.7
33.6
- SwinIR-S
33.5
UPS-S
33.4
1
10 20 30
F F++ (Bq.
GT
Percent(%))
Figure 11: Comparison between SwinIR and our proposed UPS under noise optimization setup. We also report visual examples of the 'input featureFi', "input feature with noise (the noise std is set to 0.3) PT" and 'output feature Fv' enhanced by methods and corresponding final predictions.
To better understand the effectiveness and robustness of our unified mooeetion-shariie scheme, we compare SwinlR 7 and UPS under noisy input features to simulate the perturbation training. Firstly, we train both two models with perfect training images. Then, we add different levels of Gaussian white noise (zero mean and std values ranged from 0.01 to 0.3) on the input image feature Fi (the input of FEA module) for both training and evaluation:
FT +
where n is the Gaussian white noise. The results are shown Fig. We observe that, with severe noise, the SwinlR-light fails to restore high-frequency details in the output feature FN produced by the last STL layer, thus produces incorrect image structures. In contrast, UPS is able recover high-frequency signals from the noisy input feature and accordingly reconstruct accurate image details. This experiment suggests our proposed UPS is more optimization robust and effective with noisy input features.
Table 7: Robustness comparison of SwinIR-light 77) NGSwin (28 and our proposed UPS. While being trained clean DIV2K (37 training samples, directly evaluate their generalization ability on the degraded Set14 benchmark under the x4 setting. We report the PSNR (dB)SSIM values for quantitative evaluation.
Degradation
None
Compression
Blur
Noise
SwinIR-light
28.77/0.7858 28.45 08.45/0.7783
28. 69/0078299
27.93/0.7562
NGSwin 28.83/0.7870 28.46/0.778 7783
28.65/0.7815
27 27.93/0.7532
UPS 28.90/0.7892
28.73/0.788 7833
28.87/0.7864
28.29/0.7616
A.4 Robustness on Degraded Data
The co-adaptation issue 8 reveals that deep-learning models may fit the training samples well but exhibit poor generalization for unseen data, especially for out-of-domain samples. Here, we aim evaluate the robustness of SOTA SISR models (including SwinIR- light and NGSwin) and our proposed UPS. To do this, we apply different data degradations (i.e., JPEG compression, Gaussian blur, and Gaussian White noise) and obtain degraded samples. Without training on these degradations, we directly evaluate the performance of these lightweight SISR models. The results presented in Tab. Fillustrate our proposed UPS is more robust on unseen data. Notably, UPS outperforms competing models up to 0.36dB on the noise testing data. Moreover, Fig. 12 shows both SwinIR-light and NGSwin restore incorrect and blur image structures due their poor generalization (not robust) for unseen degradations. In contrast, UPS produces more accurate results with clearer image contents.
16

---

Ground Truth
SwinlR-light
NGSwin
UPS (Ours)
Compression
How Do Everything with
How to Do Everything with
How to Do Everything with
to Do Everything with
PSNR,CBYSSSM
24.67/0.9264
24.77/0.9264
25.29/0.9330
Blur
PSNR(dRySSIM
26.46/0
26.29/0.7473
26.96/0.7665
Noise
PSNR(dBSSSM
26.08/0.721 7211
26.09/0.7214
26.75/0.7392
Figure 12: Qualitative robustness comparison of different lightweight SISR models on out-of-domain degraded inputs under x4 setting. The first sample from S and the last two samples are from the Urban100 100 benchmark.
Table 8: More ablation studies (PSNR/SSIM). A: The impacts of ReLU and Softmax activations in 6q. B. Quantitative comparison between two advanced optimization schemes (Dropout in RDSR CVPR 2022 and progressive training in DRCT ARXIV 2024.) and UPS.
Analysis
SwinIR light JPSS (Softmax)
Activation
UPS (ReLU)
SwinIR light |wiinRR + Dropout SwinIR Pro. Train]
B. Optimization schemes UPS
UPS
Urban100 (x
26.47 0.7990
26.79/ 0.8069
26.83/ 0.8073
26.47 / 0.7990
26.52 / 0.79888
26.56 0.7986
26.83/ 0,8073
Improve.
+0.32/ +0.0099
+0.09 +0.0006
+0.36
Param. (K)
930
843 (-87)
843 (-87)
930
930
930
843 (-87)
A.5
More Ablated Studies
Impact of ReLU and Softmax. We conduct the experiment to demonstrate the impact of different activations. The results in TabBAA suggests the used ReLU performs better than the softmax activation. As we can see, the main improvement comes from our UPS design instead of the ReLU activation. The performance gap between the two different activation choices is only 0.04dB, which represents 11% of the total improvement of 0.36dB. In other words, the 89% improvements come from the UPS design. We will include this ablation analysis in revised paper.
UPS vs. Other Optimization Schemes. We investigate two existing training strategies for the SISR task. RDSR 9 incorporates dropout techniques to achieve better testing results, while DRCT [49 employs progressive training scheme that involves multi-stage training to enhance final performance. Here, we compare UPS with RDSR and progressive training schemes in DRCT. To do this, we re-train SwinIR-light using the above two training strategies. As shown in Tab B), UPS delivers superior results compared to both of these optimization methods. Nevertheless, inspire future research to develop more effective algorithms to better address this challenge.
hope our exploration will
The Identity Mapping of X; and V;. For the lightweight scenario, we aim to further reduce the computational cost and model size. Thus, we explore cutting off the linear mapping between X, and V,, and our early experimental analysis (presented in Tab. 99) suggests such design will not lead to performance drop. We will add this discussion to our revised paper.
A.6 Framework Details for Image Denoising and Deblocking
we delve into advantages of UPS in tasks like image denoising and JPEG removal, conducting a comparative analysis among various baseline models and UPS. For both tasks, the SwinIR model adheres to the default framework settings outlined the original paper, featuring RSTB blocks with a channel size of 180. SwinIR-C and UPS follow similar framework configurations:
In Section 5.4
17

---

Table C. V projection indicates the linear projection for transforming the input X; into V,.
Pesults&Param,
w/
V proj.
w/o V proj. (Default)
Urban 100 (x4) 26.80 0.8071
26.83 0.8073
Sei14 (x-)
28.91
0.7892
28.90 / 0.7892
Param, (K)
895
843
8 RRST/D-RSTB blocks with channel dimension of 90 for image deblocking, and 6 RSTB/D-RSTB blocks with a channel size of 60 for image denoising.
A.7 Limitation
While UPS exhibits SOTA results in lightweight SISR, have not investigated its potential benefits for large (UPS-based) models. Exploring larger UPS-based models is an interesting future work. On the other hand, will explore more applications for wide range of low-level tasks, such as real-world image restoration.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and/or introduction clearly state the claims made, also match theoretical and experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our method in Sec A.7
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [No]
19

---

Justification:
No theoretical results in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Ys]]
Justification: We have disclosed all the information needed to reproduce the main experi- mental results. We have carefully explained the structure and experimental settings of our model.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
20

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: We will release our code and all the command and environment needed to reproduce the results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
mnips.cc/publ pubbic//uidees gwtgguuiess/ee/ hat////oooubbiseeeeeeeeeee swibbbbmissoooooooocy. for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have stated the experimental details for the reproduction our method, including data, hyperparameters, optimizer, learning rate, and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: Following previous works, we conduct extensive experiments to demonstrate the effecctivnness of our proposed UPS. Moreover, our method also exhibits superior general- ization(robust) ability for unseen data.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for caleulating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Ye]]
Justification: We have provided the experimental details on the computer resources, includ- ing GPU, memory and execution time.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttpp///herrips.ccppubbiiiii ctw.e
Answer:
[Yes]
Justification:
Our research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: As stated in introduction, super resolution is fundamental task in computer vision, and is pivotal in various fields such as medical imaging, satellite imagery, and consumer electronics, where clear and detailed visual information is crucial.
Guidelines:
22

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cited related papers, codes and data in our paper, and we also stated the version we used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
23

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "pmHoWJf9Wg", "reviewer": "Reviewer_fbd3", "summary": "This paper presents a novel algorithm named UPS designed to enhance the performance of Transformer-based frameworks in single-image super-resolution (SISR), particularly under lightweight scenarios. The authors identify the challenge posed by the simultaneous layer-specific optimization required for deep image feature extraction and similarity modeling in existing methods. To address this, UPS decouples these tasks by establishing a unified projection space via a learnable projection matrix. The proposed UPS method demonstrates state-of-the-art performance. Additionally, UPS shows good performance on broader image restoration applications.", "questions": "In this paper, it is mentioned that each layer of the model carries out image feature extraction and similarity modeling at the same time, which is difficult to train and will affect the performance of the model. Then, can the model performance be improved by improving the training strategy and modifying the loss function? In theory, if you can train the model properly, will you get better performance than UPS?", "limitations": "The paper has discussed methodological limitations.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper presents a simple yet effective lightweight method called UPS. Using this method, the training difficulty of the model is reduced, and the model effect is improved on the premise of reducing the number of parameters and FLOPS.\n2. Sufficient quantitative experimental results are given in this paper.", "weaknesses": "1. “Fig. a.(1-3) below shows over 0.95% (0.99%, 0.95%, 0.96%) for ×{2, 3, 4}) (projection layer) pairs get over 0.9 scores (ranging from 0 to 1) \". If there are no errors in this paragraph, the similarity between layers is very low.\n2. The contribution point in this paper is the Unified Projection Sharing. However, in the method, the activation function is also modified. I wonder how much of a performance boost I would get if I left the activation function unchanged and just used consistent space sharing. Please give the analytical or experimental results.\n3. As a lightweight method, it would be better to compare inference time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. In line 30, \"Fig. 1a.(1-3) below shows over 0.95% (0.99%, 0.95%, 0.96%) ...\"**\n\nThanks a lot for pointing out this typo, actually it's a writing error. It should be \"Fig. 1a.(1-3) below shows over **0.95 (0.99, 0.95, 0.96)** ...\". We will revise our manuscript to avoid misunderstanding. \n\n**Q2. Experimental results when the activation function is consistent with SwinIR-light**\n\nThat is quite a profound question. We have provided the ablated analysis (Tab. 5-A) in the global response PDF file. Also, we report the specific quantitative results in the table below. As we can see, the main improvement comes from our UPS design instead of the ReLU activation. The performance gap between the two different activation choices is only 0.04dB, which represents 11% of the total improvement of 0.36dB. In other words, the 89% improvements come from the UPS design. We will include this ablation analysis in our revised paper.\n\n| Activation  Function    | SwinIR-light (base) | UPS (Softmax) |  UPS (ReLU, Default) |   \n| :----        |    :----    |    :---- |   :----  |   \n| Urban100 (×4)     |         26.47/0.7980      |         26.79 (+0.32)/ 0.8069  (+0.0089)           |       26.83 (+0.36)/0.8073 (+0.0093)   | \n| Parameters (K)           |             930                 |           843          |        843                                       |   \n\n**Q3. To compare the inference time**\n\nPlease see the first answer of the global response. We will provide all these discussions in our revised paper.\n\n\n**Q4. More effective optimization strategy may also enhance the performance**\n\nWe agree with you on the point that an advanced training strategy may also boost the performance of entangled SISR models. As praised by you and other Reviewers, UPS provides a simple yet effective optimization scheme:  without any special training strategies, e.g., taking additional training costs or carefully model finetuning, while yielding improved results. \n   \n   We investigate two existing training strategies for the SISR task. RDSR [1] incorporates dropout techniques to achieve better testing results, while DRCT [2] employs a progressive training scheme that involves multi-stage training to enhance final performance. Here, we compare UPS with RDSR [1] and progressive training schemes in DRCT [2]. To do this, we re-train SwinIR-light using the above two training strategies. As shown in Table 5-B of our PDF file (also presented in the Tab. below), UPS delivers superior results compared to both of these optimization methods. Nevertheless, we hope our exploration will inspire future research to develop more effective algorithms to better address this challenge.\n\n\n\n|                             | SwinIR (base)   | SwinIR + Dropout | SwinIR + Pro. Train | UPS\n|-----------------------------|----------------|-------------------|---------------------|---------------------|\n| **PSNR / SSIM**             | 26.47 / 0.7980 | 26.52 / 0.7988    | 26.56 / 0.7986      | 26.83 / 0.8073\n| **Improvement**             |       -        | +0.05 / +0.0008   | +0.09 / +0.0006     | +0.36 / +0.0093\n| **Parameters (K)**     | 930            | 930               | 930                 | 843 (-87)\n\n## References\n\n[1] Kong, et al. \"Reflash dropout in image super-resolution.\" CVPR, 2022.\n\n[2] Hsu, et al. \"DRCT: Saving Image Super-resolution away from Information Bottleneck.\", arxiv, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bh41P3J9sW", "reviewer": "Reviewer_Pasd", "summary": "The paper proposes an effective lightweight decoupled SISR algorithm that simultaneously performs layer-specific optimization for deep feature extraction and similarity modeling. Specifically, the proposed method casts the deep feature extraction as per-layer optimization, while the similarity modeling is achieved by shared projection space. The proposed method is able to be extended to many restoration tasks, such as denoising, and JPEG image deb locking.", "questions": "See the Weaknesses.", "limitations": "If the authors can address my concerns, I am ready to change my recommendation based on the comments.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written.\n2. The authors bring a novel perspective for lightweight SISR that the deep feature extraction and similarity modeling can be decoupled.", "weaknesses": "1. Line 49: \"simultaneously\" -> \"simultaneous\".\n2. Figure 2: the placement of S1 and Si in Fig. 2(c) is incorrect.\n3. Eq. (5): why do you set Vi directly equal to Xi instead of projecting Xi to Vi as done in Swin Transformer?\n4. Eq. (6): what does $Q_{i}^D$ denote in this equation? What does D represent?\n5. It's recommended to report the quantitative results on the DIV2K dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your appreciation of the novelty and effectiveness of our UPS for lightweight SISR.\n\n**Q1. A typo: Line 49...**\n\nWe appreciate your kind comment. We will carefully revise our paper to fix the typo. \n\n**Q2. Figure 2: the misplacement of $S_1$ and $S_i$**\n  \nThanks, we will correct the placement of the $S_1$ and $S_i$ accordingly. \n\n**Q3. The indetity mapping of $X_i$ and $V_i$**\n\nFor the lightweight scenario, we aim to further reduce the computational cost and model size. Thus, we explore cutting off the linear mapping between $X_i$ and $V_i$, and our early experimental analysis (presented in Tab. 5-C of our PDF file) suggests such a design will not lead to a performance drop. We will add this discussion to our revised paper. \n\n| Settings           |  w/ V proj.    |     w/o V proj (Default)        |   \n| :----              |    :----                  |    :----             |\n| Urban100 (×4)      |         26.80/0.8071      |        26.83/0.8073 |   \n| Set14 (×4)         |         28.91/0.7892      |        28.90/0.7892 |   \n| Parameters (K)/FLOPs (G)         |             895/179           |           843/163       |\n\n**Q4. The denotation of $D$ in $Q_i^D$ in Eq. 6**\n\n$D$ means the projection dimension in our UPS. We are sorry for this typo. The correct Eq.6 should be: \n   $$\n   ReLU(Consine(Q_i,Q_i^T) + B_i),\n   $$\nwhere $T$ means the transpose operation for matrix multiplication. We will double-check all these equations to avoid any difficulties in understanding our work.\n\n**Q5. Quantitative results on the DIV2K dataset...** \n\nIn this work, we adopt the DIV2K for training our models as indicated in Line 175 \"we utilize the DIV2K image dataset for training\". We will highlight this important training setting in our revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T3ug8Ac0Qu", "reviewer": "Reviewer_RnSV", "summary": "This work introduces a novel unified projection sharing algorithm that decouples feature extraction and similarity modeling. A unified projection space defined by a learnable projection matrix is created for similarity computation across all self-concerned layers. Extensive experiments demonstrate that the proposed UPS achieves state-of-the-art performance compared to leading lightweight SISR methods.", "questions": "Please address the issues raised in the Weaknesses.", "limitations": "Limitations were discussed.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is clearly written and well organized.\n\n2. The sharing of unified projections can effectively reduce computation and enable performance improvements.\n\n3. Experiments show the promising performance of the proposed method.", "weaknesses": "1. The idea of shared projection is somewhat similar to attention sharing [1,2,3]. Discussion and analysis with these highly relevant studies is needed.\n\n2. The needs in Table 1 include computational quantities for the different models. It is unfair to compare with CNN-based methods only the parameters, which are much less computationally intensive.\n\n3. Inference efficiency must be included in the comparison, including CNN and Transformer based methods. Paramas and FLOPs do not fully reflect the on-device running speed of the model.\n\n4. Lack of experimental evaluation in more complex real-world scenarios.\n\n5. Similarity calculation methods have been discussed in past studies [4, 5, 6].\n\n> 1.  ShareFormer: Share Attention for Efficient Image Restoration. arxiv 2023.\n\n> 2. Skip-Attention: Improving Vision Transformers by Paying Less Attention. arxiv 2023.\n\n> 3. You Only Need Less Attention at Each Stage in Vision Transformers. CVPR 2024.\n\n> 4. EfficientViT: Multi-Scale Linear Attention for High-Resolution Dense Prediction. ICCV 2023.\n\n> 5. Swin transformer v2: scaling up capacity and resolution. CVPR 2022.\n\n> 6. Swin2SR: SwinV2 Transformer for Compressed Image Super-Resolution and Restoration. ECCVW 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Discussion and analysis with attention sharing...**\n\nThanks for your suggestion. ShareFormer [1] presents a local similarity map-sharing scheme between neighboring attention layers for lower latency. Thus, ShareFormer shares a static similarity map for neighboring attention layers while UPS calculates dynamic similarity maps with layer-refined features in a shared projection space.\n\nSkip-Attention cuts off some intermediate attention layers to improve efficiency and performance for high-level tasks. LaViT [2] proposes a residual-based attention downsampling that fuses the initial calculated attention scores to guide the aggregation of the following layers, resulting in faster efficiency and improved classification accuracy. \n\nTherefore, Skip-Attention [3] and LaViT follow the existing coupled optimization scheme (reduce some attention calculations), and UPS proposes a decoupled learning strategy to enhance performance. We will cite the insightful studies and add this discussion to our revised paper.\n\n**Q2. Report the computational quantities of different models**\n\nThanks. We agree with you on this point and we will include the FLOPs (like Fig. 1(c) of our paper) of different CNNs and Transformers for a comprehensive comparison. Please also refer to the FLOPs results in Tab. 1, 2 (b), 3 and Table 4 of our response PDF file.\n\n**Q3. Inference efficiency of different models**\n\nPlease see the first answer of the global response and more inference comparison in the Tab. 1. 2(b), 3, 4 in our response PDF file. We will provide all these discussions in our revised paper.\n\n**Q4. Experimental evaluation real-world SR.**\n\n   Thanks for this good comments. Following SwinIR-GAN, our baseline model, we train UPS-GAN with the same configuration using the widely used BSRGAN degradation. We compare UPS-GAN with SwinIR-GAN, BSRGAN, RealSR, Reshift, and StableSR for real-SR task. We evaluate all real-world SR models using the same benchmarks (RealSRSet) and assessment metrics as SwinIR-GAN. As shown in Table 2a of the PDF file (also the Tab below), our proposed UPS-GAN outperforms other state-of-the-art GAN-based and even Diffusion-based methods (Reshift NeurIPS 2023[7] and StableSR IJCV 2024[8]) in terms of NIQE, NRQM, and PI metrics, achieving the best quantitative results (5.09/6.84/4.19). This confirms the effectiveness of UPS for real-world SR tasks.\n\n| Metrics | BSRGAN | RealSR |  ResShift  |  StableSR  | SwinIR-GAN | UPS-GAN |\n|---------|--------|--------|-----------|-----------|------------|---------|\n| NIEQ ↓  | 5.66   | 5.83   | 8.37      | 5.24  | 5.49       | **5.09** |\n| NRQM ↑  | 6.27   | 6.32   | 4.56      | 6.12      | 6.48   | **6.84** |\n| PI ↓    | 4.75   | 4.40| 7.03     | 4.66      | 4.72       | **4.19** |\n\n**Q5. The adopted similarity calculation method in UPS**\n\nAs you pointed out, apart from the ReLUFormer discussed in our method section, we will cite and discuss the related works in our revised paper.\n\n### References \n\n[1] ShareFormer: Share Attention for Efficient Image Restoration. arxiv 2023. \n\n[2] You Only Need Less Attention at Each Stage in Vision Transformers. CVPR 2024.\n\n[3] Skip-Attention: Improving Vision Transformers by Paying Less Attention. arxiv 2023.\n\n[4] EfficientViT: Multi-Scale Linear Attention for High-Resolution Dense Prediction. ICCV 2023.\n\n[5] Swin transformer v2: scaling up capacity and resolution. CVPR 2022.\n\n[6] Swin2SR: SwinV2 Transformer for Compressed Image Super-Resolution and Restoration. ECCVW 2022.\n\n[7] Yue, et al. \"\"Resshift: Efficient diffusion model for image super-resolution by residual shifting.\" NeurIPS, 2023\"\n\n[8] Wang et al. \"Exploiting Diffusion Prior for Real-World Image Super-Resolution.\" IJCV, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6w9BYolRTf", "reviewer": "Reviewer_HE6k", "summary": "This paper introduces Unified Projection Sharing (UPS), a novel algorithm for lightweight single-image super-resolution (SISR) that decouples feature extraction from similarity modeling by employing a unified projection space. This approach achieves state-of-the-art performance across various benchmarks while demonstrating robustness for unseen data and promising results for additional image restoration tasks, all with a computationally efficient model.", "questions": "See above.", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper has a clear writing approach and brings new insights.\n2. The UPS algorithm demonstrates superior performance compared to existing lightweight SISR methods across multiple benchmarks, showcasing its effectiveness.\n3. The method shows promise for extension to other image restoration tasks beyond SISR, suggesting its potential for broader application in image processing.", "weaknesses": "1. The author did not conduct experiments outside of the lightweight model [1][2], and from an optimization perspective, the proposed method should be possible to improve performance in various situations. The authors need to be able to justify themselves.\n\n2. Lack of actual latency comparison. Due to some differences in network structure, the params and FLOPs may not accurately reflect the efficiency of the model.\n\n3. Lack of comparison with the SOTA model, e.g., [3]. outperforms the UPS almost on every benchmark.\n\nRef:\n\n1. Chen, Xiangyu et al. “Activating More Pixels in Image Super-Resolution Transformer.” 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022): 22367-22377.\n\n2. Hsu, Chih-Chung et al. “DRCT: Saving Image Super-resolution away from Information Bottleneck.” ArXiv abs/2404.00722 (2024): n. pag.\n\n3. Wang, Hang et al. “Omni Aggregation Networks for Lightweight Image Super-Resolution.” 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023): 22378-22387.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Conduct experiments outside of the lightweight model...**\n\nThanks for your thoughtful comments. As you appreciated the potential generalization of UPS on more image restoration tasks, e.g., JPEG compression removal and image de-noising tasks, we explore the potential application of UPS for common SISR and real-world SR tasks and more baseline frameworks (e.g., HAT/HAT-light, DRCT/DRCT-light) and report the results in Tab. 2 (a) and Tab. 3/4 of our global PDF response. \n\n\nFor both common and lightweight single image super-resolution (SISR), we incorporate UPS into other state-of-the-art SISR models, including SwinIR, HAT/HAT-light [4], and DRCT/DRCT-light [5], based on your suggestion. As shown in Tables 3 and 4 of our response file, we observe that UPS consistently improves the performance and inference efficiency of these models (HAT-UPS, DRCT-UPS, SwinIR-UPS, and their lightweight versions HAT-light-UPS, DRCT-light-UPS, and SwinIR-light-UPS). We will include these insightful experiments in our revised paper. We believe your insightful suggestion and our exploration enhance the wide-ranging value of UPS.\n\nAdditionally, for real-world super-resolution (SR), we follow our baseline model, SwinIR-GAN (trained using BSRGAN degradation), and train UPS-GAN. We evaluate all real-world SR models using the same benchmarks (RealSRSet) and evaluation metrics as SwinIR-GAN. As shown in Table 2a of the PDF file (also the Tab below), our proposed UPS-GAN outperforms other state-of-the-art GAN-based and even Diffusion-based methods (Reshift NeurIPS 2023[1] and StableSR IJCV 2024[2]) in terms of NIQE, NRQM, and PI metrics, achieving the best quantitative results (5.09/6.84/4.19). This confirms the effectiveness of UPS for real-world SR tasks.\n\n| Metrics | BSRGAN | RealSR |  ResShift |  StableSR  | SwinIR-GAN | UPS-GAN |\n|---------|--------|--------|-----------|-----------|------------|---------|\n| NIEQ ↓  | 5.66   | 5.83   | 8.37      | 5.24  | 5.49       | **5.09** |\n| NRQM ↑  | 6.27   | 6.32   | 4.56      | 6.12      | 6.48   | **6.84** |\n| PI ↓    | 4.75   | 4.40| 7.03     | 4.66      | 4.72       | **4.19** |\n\n\n\n**Q2. Latency comparison**\n\nPlease see the first answer of the global response. We will provide all these discussions in our revised paper.\n\n\n**Q3. Comparison with the SOTA model, e.g., Omni-SR**\n\nAs discussed in our paper (Line 175), we use the widely used DIV2K for training. When compared with Omni-SR [6] (trained on the same DIV2K), UPS surpasses it in most cases. \n\nTo make a fair comparison with Omni-SR+ (trained on DF2K), we re-trained UPS+ from scratch on DF2K. We can see that UPS+ generally attains superior performance than Omni-SR+ in the below table. For instance, UPS+ achieves 0.27dB/0.20dB/0.29dB improvements than Omni-SR+ on Manga109 x{2,3,4}. \n\n| Method    | Scale|  Set5          | Set14         | BSD100        | Urban100       | Manga109       |\n|-----------|---------------|---------------|---------------|---------------|----------------|----------------|\n| Omni-SR  | ×2 | 38.22 / 0.9613 | 33.98 / 0.9210 | 32.36 / 0.9020 | 33.05 / 0.9363 | 39.28 / 0.9784 |\n| UPS      | ×2 | **38.26 / 0.9642** | **34.16 / 0.9232** | **32.42 / 0.9031** | **33.08 / 0.9373** | **39.62 / 0.9800** |\n| | | | | | |\n| Omni-SR+ | ×2 | 38.29 / 0.9617 | 34.27 / 0.9238 | 32.41 / 0.9026 | 33.30 / 0.9386 | 39.53 / 0.9792 |\n| UPS+    | ×2  | **38.31 / 0.9643** | **34.37 / 0.9247** | **32.43 / 0.9032** | **33.34 / 0.9388** | **39.80 / 0.9802** |\n| | | | | | |\n| Omni-SR    | ×3     | **34.70** / 0.9294   | 30.57 / 0.8469    | 29.28 / 0.8094 | 28.84 / 0.8656   | 34.22 / 0.9487       |\n| UPS        | ×3    | 34.66 / **0.9322** | **30.72** / **0.8489**    | **29.31** / **0.8114**| **28.98** / **0.8685**| **34.53** / **0.9505**|\n| | | | | | |\n| Omni-SR+ | ×3 |34.77 / 0.9304 | 30.70 / 0.8489 | 29.33 / 0.8111 |  29.12 / 0.8712 | 34.64 / 0.9507 |\n| UPS+       | ×3     | **34.78 / 0.9325**   | **30.78 / 0.8492**    | **29.36 / 0.8122**     | **29.28 / 0.8728**       | **34.84** / **0.9517**       |\n|            |       |                 |           |                  |                   |                    |                      |                      |\n| Omni-SR    | ×4    | 32.49 / 0.8988 | 28.78 / 0.7859 | 27.71 / 0.7415 | 26.64 / 0.8018 | 31.02 / 0.9151      |\n| UPS        | ×4    | **32.50** / **0.9024**| **28.90 / 0.7892**    | **27.79** / **0.7435**| **26.83** / **0.8073**| **31.39** / **0.9194**|\n| | | | | | |\n| Omni-SR+ | ×4 |32.57 / 0.8993 |  28.95 / 0.7898 |  27.81 / 0.7439 | 26.95 / 0.8105 | 31.50 / 0.9192|\n| UPS+       | ×4    | **32.60 / 0.9029**   | **28.97 / 0.7896**    | **27.83 / 0.7446**     | **27.10 / 0.8136**       | **31.79 / 0.9223**       |\n\nMore results can be found in Tab. 1 of our response PDF. We will include this comparison in Tab. 1 of our revised paper and highlight the two different training setups to ensure a clear and fair comparison.\n   \n## References \n[1] Yue, et al. \"\"Resshift: Efficient diffusion model for image super-resolution by residual shifting.\" NeurIPS, 2023\"\n\n[2] Wang et al. \"Exploiting Diffusion Prior for Real-World Image Super-Resolution.\" IJCV, 2024.\n\n[3] Liang, et al. \"Swinir: Image restoration using swin transformer.\" CVPR, 2021.\n\n[4] Chen, et al. “Activating More Pixels in Image Super-Resolution Transformer.” CVPR, 2022.\n\n[5] Hsu, et al. “DRCT: Saving Image Super-resolution away from Information Bottleneck.” arxiv, 2024.\n\n[6] Wang, et al. “Omni Aggregation Networks for Lightweight Image Super-Resolution.” CVPR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear AC and Reviewers,

We sincerely thank all the reviewers for their constructive comments and consistent appreciation of the novelty and effectiveness of our UPS for lightweight SISR. Reviewer fbd3, Reviewer RnSV, and Reviewer HE6k have raised concerns about inference efficiency, and both Reviewer RnSV and Reviewer HE6k have shown interest in extending UPS to other SISR scenarios, such as real-world SR or common SISR tasks. Therefore, we address these two issues here.

**Q1. The inference efficiency (latency)**

Thank you for this valuable comment. Results of inference time (ms), FLOPs (G) and GPU memory usage (MB). The speed is tested on an NVIDIA GeForce RTX 2080Ti GPU with an input size of 256 × 256 under x2 lightweight SISR. And we follow other works to calculate the FLOPs at an output resolution of 1280 x 720. Moreover, UPS, HAT-light-UPS, DRCT-light-UPS enhance the inference efficiencies compared with their counterparts.

| Method         | Time (ms)  | FLOPs(G) | Memory  |
| ----------------| ----------- | ---------------------- | ---------------------- |
| RFND-L       | 13      | 146  | 1577      |
| LatticeNet | 18      | 170  | 1639      |
| DLGSA-l    | 225      | 170  | 1800      |
| Omni-SR   | 112      | 195  | 1842      |
| SwinIR-light   | 175   | 244  | 2051      |
| **UPS**            | 119      | 163  | 1785      |
| SwinIR-S    | 117      | 107  | 1365      |
| **UPS-S**          | 71      | 91  | 1039      |
| HAT-light | 153      | 102  | 2039      |
| **HAT-light-UPS** | 136      | 91  | 1763      |
| DRCT-light | 92      | 137  | 2330      |
| **DRCT-light-UPS** | 85      | 125  | 1991      |

 More inference cost comparison can be found in Tab. 2(b)/3/4 of the associated rebuttal PDF for both lightweight and large SISR models. When compared to our baseline model SwinIR-light, UPS reduces the overall inference cost by 33% in terms of FLOPs. 

Additionally, as shown in Tab. 3/4, UPS can be adapted to other transformers, consistently enhancing their performance while also reducing inference costs. For example, we integrated UPS into HAT, a state-of-the-art SISR model, resulting in HAT-UPS. We can observe that HAT-UPS significantly reduces computational complexity by 3.52M parameters, 95G FLOPs, and 195ms inference time while yielding improved results. Lastly, we also adopt integrated UPS into lightweight DRCT-light, the DRCT-light-UPS reduces the inference costs by 141K/12G/7ms (parameters/FLOPs/speed) compared to its original version DRCT-light.
  
**Q2.1. UPS for Real-world SR**

Thank you for appreciating the generalization capability of UPS. We agree with you, as UPS is designed to decouple the optimization of feature extraction and similarity modeling, which is orthogonal to specific methods such as SwinIR. Accordingly, we have extensively explored the benefits of UPS for real-world SR and other frameworks, including HAT and DRCT, under both lightweight and parameter-intensive scenarios. Our experiments show that the proposed UPS consistently enhances efficiency and performance across all these settings (real-world SR, lightweight, and SISR tasks).

For real-world super-resolution (SR), as shown in Tab. 2(a) of the PDF file (also the table below), our proposed UPS-GAN outperforms other state-of-the-art GAN-based and even Diffusion-based methods (Reshift NeurIPS 2023[1] and StableSR IJCV 2024[2]) in terms of NIQE, NRQM, and PI metrics, achieving the best quantitative results (5.09/6.84/4.19). This confirms the effectiveness of UPS for real-world SR tasks.

| Metrics | BSRGAN | RealSR |  ResShift |  StableSR  | SwinIR-GAN | UPS-GAN |
|---------|--------|--------|-----------|-----------|------------|---------|
| NIEQ ↓  | 5.66   | 5.83   | 8.37      | 5.24  | 5.49       | **5.09** |
| NRQM ↑  | 6.27   | 6.32   | 4.56      | 6.12      | 6.48   | **6.84** |
| PI ↓    | 4.75   | 4.40| 7.03     | 4.66      | 4.72       | **4.19** |

**Q2.2. UPS for large model**

Additionally, we explored two more transformers (HAT-light [3] and DRCT-light [4]) for lightweight SISR. As shown in Tab. 4, HAT-light-UPS and DRCT-light-UPS consistently improved their results while achieving better inference latency and lower FLOPs. For instance, DRCT-light-UPS enhances DRCT-light by 0.4dB on the Manga109 dataset and HAT-light-UPS improves HAT-light by 0.28dB on the Urban100 benchmark.

Lastly, we incorporated UPS into large SISR models including SwinIR, HAT [3], and DRCT [4]. The results in Tab. 3 suggest that their enhanced versions by UPS can achieve promising performance and significantly lower inference costs. We can observe that DRCT-UPS surpasses its baseline counterpart (DRCT) by 0.26dB on Urban100.

Thank you again for this constructive suggestion. We will include these analyses in our revised paper. Besides extending our work to image JPEG compression removal and image de-noising tasks, we believe all these comprehensive additional analyses (wide-range SR tasks and easy adaption for other scalable SOTA transformer frameworks) will enhance the value of UPS and inspire future research in developing more effective decoupling optimization algorithms.

[1] Yue, et al. ""Resshift: Efficient diffusion model for image super-resolution by residual shifting." NeurIPS, 2023"

[2] Wang et al. "Exploiting Diffusion Prior for Real-World Image Super-Resolution." IJCV, 2024.

[3] Chen, et al. “Activating More Pixels in Image Super-Resolution Transformer.” CVPR, 2022.

[4] Hsu, et al. “DRCT: Saving Image Super-resolution away from Information Bottleneck.” arxiv, 2024.

Meta-review: The paper presents a lightweight transformer-based single-image super-resolution model. The key idea is decoupling feature extraction and similarity modeling with Unified Projection Sharing. The reviews are mixed, with one Accept, one Borderline accept, and two borderline reject. The general weakness raised by the reviewers are the lack of latency comparison, real-world SR results, and comparison with SOTA super-resolution models. 

The authors provided a detailed rebuttal responding to these concerns with additional experimental results. More specifically, the authors provided results on latency comparisons, results on the real-world SR evaluation, and a comparison with Omni-SR using the same training dataset. The reviewers who were leaning positive are satisfied with the clarification and additional results. Unfortunately, the two reviewers who initially leaned negatively toward the paper did not fully engage with the newly presented results. 

The AC reviewed the new experimental results and believes that the authors have sufficiently addressed the concerns that were initially raised. Thus, the AC recommends to accept.
Decision: Accept (poster)