GRFormer: Grouped Residual Self-Attention for Lightweight Single Image Super-Resolution

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 14:56:58

Abstract: Previous works have shown that reducing parameter overhead and computations for transformer-based single image super-resolution (SISR) models (e.g., SwinIR) usually leads to a reduction of performance. In this paper, we present GRFormer, an efficient and lightweight method, which not only reduces the parameter overhead and computations, but also greatly improves performance. The core of GRFormer is Grouped Residual Self-Attention (GRSA), which is specifically oriented towards two fundamental components. Firstly, it introduces a novel grouped residual layer (GRL) to replace the QKV linear layer in self-attention, aimed at efficiently reducing parameter overhead, computations, and performance loss at the same time. Secondly, it integrates a compact Exponential-Space Relative Position Bias (ES-RPB) as a substitute for the original relative position bias to improve the ability to represent position information while further minimizing the parameter count.
Extensive experimental results demonstrate that GRFormer outperforms state-of-the-art transformer-based methods for x2, x3 and x4 SISR tasks, notably outperforming SOTA by a maximum PSNR of 0.23dB when trained on the DIV2K dataset, while reducing the number of parameter and MACs by about 60% and 49% in only self-attention module respectively. We hope that our simple and effective method that can easily applied to SR models based on window-division self-attention can serve as a useful tool for further research in image super-resolution. The code is available at https://github.com/sisrformer/GRFormer.

Main paper: 
 introduction: 
 Single Image Super-Resolution (SISR) aims to enhance image resolution by reconstructing a high-resolution image from a lowresolution counterpart. With the development of CNN-based and transformer-based SR models, one achievement after another has been achieved for single image super-resolution tasks. For whether CNN-based models or transformer-based ones, it is an easy way to improve performance by increasing the number of the network layers and feature dimension, accompanied by the increase of parameters and calculations. A straight question should be: Is it possible to improve the performance while reducing number of the parameters and calculations for transformer-based SR models? Motivated by this question, we conduct in-depth research into self-attention and present three sub-questions concerning self-attention: • RQ1. Is there any redundancy within self-attention? • RQ2. Can the expressive power of self-attention be further improved? • RQ3. Is there a better alternative to relative position bias for representing the position information? The research into RQ1. Self-attention [20] has achieved great performance in the fields of text, image, and video since it was published in 2017. However, despite the effectiveness of self-attention mechanisms, they are often criticized for their extensive parameter count and computational demands. Existing work [3] has observed redundancy in self-attention layers, but its solutions focus mainly on how to reduce the size of the attention window [21] . Through empirical analysis, we explore the interaction among the varying parameter counts, MACs, and the performance of the self-attention module, as depicted in Fig. 1 . This analysis reveals significant potential for optimization in both parameters and computational efficiency within the SwinIR's self-attention mechanism. Inspired by these findings, we propose a novel grouping scheme of Q, K, V linear layer, aimed at reducing both the parameter overhead and computational complexity. The research into RQ2. In the field of single image super-resolution, previous transformer-based approaches [12, 22] have primarily utilized residual learning [8] at the outer layers of self-attention mechanisms. This methodology has been effective in mitigating network degradation and improving the expressive capabilities of deep networks. Given the success of residual learning in enhancing network performance, an intriguing question arises: could the integration of residual connections within the Query, Key, and Value (QKV) linear layers of self-attention mechanisms further augment their expressive power? Motivated by this consideration, our work explores the incorporation of residual connections directly into the QKV linear layers, aiming to enhance their representational efficacy in deep neural architectures. The research into RQ3. When self-attention [20] is proposed, it uses position embedding to provide position information of words in text. After self-attention is applied to computer vision, relative position bias (RPB) is found to be more suitable for representing relative position information, which is important for models in computer vision. However, RPB has four fatal flaws: First, if we assume the shape of the window in self-attention is H × W, the number of parameters occupied by RPB is (2H-1)×(2W-1), which is a huge parameter overhead. Second, the original RPB in SwinIR [12] sets a position parameter for each position within the window, which is not only the redundancy of the parameters, but also it is easy to be interfered by noises during training. Specifically, many subtle fluctuations can be found in the figures of Fig. 2 (a). Third, during training, each parameter of RPB is trained independently, ignoring the relative relationship between the weights of different positions. Fourth, RPB fails to clearly reflect intuition: For reconstructing an image, near pixels tend to be more important than far pixels. Motivated by this, we designed an Exponential-Space Relative Position Bias (ES-RPB) to replace RPB.First and foremost, to solve the RQ1, we propose a grouping scheme for the QKV linear layer. Furthermore, to solve the RQ2 and compensate for performance loss from the grouping scheme, we add residuals for the QKV linear layer. Lastly, to solve RQ3, we propose Exponential-Space Relative Position Bias (ES-RPB). The three methods above constitute the two fundamental components of Grouped Residual Self-Attention (GRSA). Given the proposed GRSA, we design a lightweight network for SR, termed GRFormer. We evaluate our GRFormer on five widely-used datasets. Benefiting from the proposed GRSA, our GRFormer achieves significant performance improvements on almost five benchmark datasets. Notably, trained on DIV2K dataset [19] for x2 SR task, our GRFormer achieves a 33.17 PSNR score on the challenging Urban100 dataset [9] . The result is much higher than recent SwinIR-light [12] (32.76) and the SOTA lightweight SR model (32.94). This improvement is consistently observed across x3 and x4 tasks. Comprehensive experiments show that GRFormer not only outperforms previous lightweight SISR models [12, 14, 22] , but also reduces the parameter count by about 20% in total model architecture, compared with SwinIR [12] (1000k parameters) with the same hyperparameter. To sum up,our contributions can be summarized as follows:• We propose a novel Grouped Residual Self-Attention (GRSA) for lightweight image super-resolution, which can not only reduces the parameter count but also enhances the performance in an easy-to-understand way for SR tasks. In addition, our proposed GRSA module can seamlessly replace the self-attention module and its variants in other transformer-based SR models, simultaneously reducing the number of parameter by about 60% and the number of MACs by about 49% in only self-attention module.  
related works 2.1 cnn-based image super-resolution: 
 A lot of CNN-based SR models [1, 13, 17] have emerged since SR-CNN [6] introduces CNN-based deep learning method for image SR. With very deep convolutional network and residual learning, VDSR [10] achieves a high accuracy for image super-resolution, which deeply influences the subsequent SR models. In order to accelerate SR inference process, FSRCNN [7] learns the mapping from the original low-resolution image to high-resolution and perform an upsampling operation at the end of the network. The pipeline with pixel shuffle upsampling is widely used by subsequent models. While most of other CNN-based SISR methods mainly focus on wider or deeper architecture design, SAN [5] explores more powerful feature expression and feature correlation learning.Since Swin Transformer [16] introduces hierarchical architecture and shifted windowing scheme, the feasibility of transformer application in the field of computer vision has been greatly improved.In order to introduce transformer into the field of image superresolution, SwinIR [12] is proposed, which has been baseline model for transformer-based SR models. However, for some lightweight scenarios, the amount of parameters and calculations in SwinIR is still too large and its relative position bias lacks a certain organization.Deeper neural networks are more difficult to train, which prevents implementation of deeper networks to improve performance. To ease the difficulty in training of deeper networks, deep residual learning framework is proposed, which helps a large number of subsequent deep models to improve performance. There are a lot of explanations on the reason why residual learning works, among which two are popular: First, residual learning enhance the fitting ability of deep model. Second, residual learning reduces the difficulty of training the model, making it easier to train a better model. Those inspires us to use residual learning framework to ease the difficulty in training of self-attention.Since self-attention [20] is proposed, a large amount of research has been devoted to solving the redundancy of self-attention. After analysis of the empirical similarity of pairwise attention scores across heads and layers, those pairwise attention scores in multiple heads in multiple layers are found to be considerably redundant and Reuse Transformer [2] is proposed to solve the redundancy. Although Reuse Transformer solves the redundancy of self-attention across heads and layers, it fails to pay attention to the redundancy within self-attention. For image super-resolution, SRFormer [22] shrinks the channel dimensions of K and V matrices and then permutes to convey the part of spatial information into the channel dimension, which reduces the redundancy within self-attention while not degenerating too much performance. However, The reduction of the channel dimension of Q and K matrices will make it difficult to add mechanisms such as residuals on this basis. It can be seen from the above discussion that, there is a lot of redundancy within self-attention.When self-attention is proposed to solve problems in the field of natural language processing (NLP), absolute position embedding is designed to supplement position information of words in text.After self-attention is applied into image super-resolution, relative position bias enjoys more popularity than the absolute position embedding, because relative position bias can provide the relative position information, which is intrinsically more suitable for image super-resolution. However, there are some problems concerning relative position bias that cannot be ignored, such as parameter redundancy, weak ability to resist interference during training and so on. Afterwards, in order to tackle resolution gaps between pre-training of large vision models, SwinIR-v2 [15] proposes a log-spaced continuous position bias method to effectively transfer large-scale models pre-trained using low-resolution images to downstream tasks with high-resolution inputs. Although the log-spaced continuous position bias is designed to solve the problem of resolution difference of pre-trained large-scale models, it is very inspiring for the design of relative position bias of transformer-based SR models. 
method 3.1 overall architecture: 
 The overall architecture of our GRFormer is shown in Fig. EQUATIONwhere 𝐶 𝑖𝑛 and C is the channel number of LR input and shallow feature. This convolution layer simply converts the input from image space into high-dimensional feature space. Then, we use N grouped residual self-attention block groups 𝐿 𝐺𝑅𝑆𝐴𝐵𝐺 and a 3×3 convolution layer 𝐿 𝑐𝑜𝑛𝑣 at the end to extract the deep feature 𝐼 𝐷𝐹 ∈ 𝑅 𝐻 ×𝑊 ×𝐶 . The process can be expressed asEQUATIONIn GRSAB Group, given 𝑋 𝑖 as input, we use M grouped residual selfattention block 𝐿 𝐺𝑅𝑆𝐴𝐵 to get 𝑋 𝑖,𝑀 . Then we use a 3×3 convolution layer 𝐿 𝑐𝑜𝑛𝑣 to get 𝑋 𝑖+1 . The process can be expressed asEQUATIONFinally, we use a 3×3 convolution layer 𝐿 𝑐𝑜𝑛𝑣 to get better feature aggregation, and aggregate shallow and deep features to reconstruct HR image 𝐼 𝐻𝑅 ∈ 𝑅 𝐻 ×𝑊 ×𝐶 𝑜𝑢𝑡 asEQUATIONwhere 𝐶 𝑜𝑢𝑡 is the channel number of the high-resolution image and 𝐿 𝑠ℎ𝑢 𝑓 𝑓 𝑙𝑒 is a PixelShuffle [18] module.The Grouped Residual Self-Attention Block (GRSAB) mainly consists of two core components: Grouped Residual Self-Attention (GRSA), Feed Forward Network module (FFN). Given the input of GRSAB as 𝐼 𝑖𝑛 ∈ 𝑅 𝐻 ×𝑊 ×𝐶 , we first use a grouped residual selfattention module 𝐿 𝐺𝑅𝑆𝐴 to learn the global relationships of pixels in a window. After 𝐿 𝐺𝑅𝑆𝐴 , we use a LayerNorm module to normalize the feature, because the normalized features can eliminate gradient vanishing and make the training stable. The process can be expressed asEQUATIONwhere 𝐼 𝐺𝑅𝑆𝐴 is the ouput of GRSA module. Then we use a feed forward network to transform the 𝐼 𝐺𝑅𝑆𝐴 to another feature space and a LayerNorm module to normalize the feature as EQUATIONTo reduce the number of parameters as well as calculations and enhance the performance, we introduce the Grouped Residual Self-Attention (GRSA), which incorporates two novel and compact components: grouped residual layer (GRL) and exponential-space relative position (ES-RPB). GRL consists of twp parts: residuals for QKV linear layer to transform the feature space of self-attention from the linear space to residual space, grouping scheme for QKV linear layer to reduce the parameters and calculations. By reducing the noise interference during training and making self-attention sensitive to pixel distance, ES-RPB improves the expression ability of position information. The proposed GRSA aggregates features of pixels globally in the window. Specifically, given input of GRSA as X ∈ 𝑅 𝐻 ×𝑊 ×𝐶 , we uses 𝐿 𝐺𝑅𝐿 𝑄 , 𝐿 𝐺𝑅𝐿 𝐾 , 𝐿 𝐺𝑅𝐿 𝑉 to get Q, K, V.EQUATIONwhere 𝐿 𝐺𝑅𝐿 𝑄 , 𝐿 𝐺𝑅𝐿 𝐾 , 𝐿 𝐺𝑅𝐿 𝑉 are the GRL module corresponding to Q, K, V. Then we normalize Q and K and calculate the matrix product of Q and K to get the similarity of Q and K. We multiply 𝑄𝐾 𝑇 by a trainable self-attention scaling factor 𝜆, add 𝐵 𝐸𝑆 -𝑅𝑃𝐵 and perform a Softmax operation. Next, we calculate the matrix product of the 𝑄𝐾 𝑇 after Softmax and V. If multi-head self-attention is applied, we will use a grouped linear layer 𝐿 𝑝𝑟𝑜 𝑗 as a projection at the last to map the multi-head to one head. The formulation can be written as:EQUATION3.4 Grouped Residual Linear.As a substitute for the QKV linear layer, the Grouped Residual Linear (GRL) is one of the core modules of GRSA, with the objective of reducing the amount of parameters and calculations and basically maintaining the feature learning ability. By integrating concepts of grouping and residuals into the self-attention mechanism, we enhance its structural efficiency and functional effectiveness.• Grouping scheme of QKV linear layer in self-attention.A lot of works, such as [22] , show that there is redundancy in generation of Q, K, V and matrix product of Q, K. To reduce the redundancy, we apply the idea of grouping. Given the input as X, we divide X into two equal parts in the channel dimension and then use two independent linear layers to get Q, K, V respectively. Grouping scheme of Q, K, V will not significantly reduce the interaction of pixel features, because the matrix multiplication of Q and K in self-attention will offset these shortcomings to some extent, which is discussed in section 3.6. • Residuals of the Q, K, V linear layers. Residuals allows training to be performed in the residual space, which makes the network find the optimal solution in the residual space. So we add it into the QKV linear layer to transform the training space of QKV linear layer from linear space to residual space in order to enhance the feature learning ability of QKV linear layer.Specifically, we assume the input of GRL module is X ∈ 𝑅 𝐻 ×𝑊 ×𝐶 . We first divide X in channel dimension into two parts 𝑋 𝑖𝑛 1 , 𝑋 𝑖𝑛 2 . Then, by using residuals, each uses a linear layer to get 𝑋 𝑜𝑢𝑡 1 , 𝑋 𝑜𝑢𝑡 2 . Finally, we merge 𝑋 𝑜𝑢𝑡 1 , 𝑋 𝑜𝑢𝑡 2 in channel dimension to get the output 𝑋 𝑜𝑢𝑡 at the last asEQUATIONwhere 𝐿 𝐿𝑖𝑛𝑒𝑎𝑟 1 and 𝐿 𝐿𝑖𝑛𝑒𝑎𝑟 2 are linear layers, 𝑋 𝑖𝑛 1 , 𝑋 𝑖𝑛 2 , 𝑋 𝑜𝑢𝑡 1 ,𝑋 𝑜𝑢𝑡 2 ∈ 𝑅 𝐻 ×𝑊 × 𝐶 2 .To solve the four disadvantages of original RPB mentioned in RQ3, we propose the Exponential-Space Relative Position Bias (ES-RPB). We design a exponential mapping for original absolute position coordinates to forcibly add pixel distance sensitive rules to original RPB, which makes it give more weight to nearby pixels. What's more, we use a tiny multilayer perceptron (MLP) to obtain the mapping of all absolute position coordinates, which reduces the impact of noise during training and makes it easier to be trained. Specifically, we transform the abscissa ΔX and the ordinate ΔY in absolute position coordinate matrix from linear space to exponential space, and then we use a tiny MLP to get 𝐵 𝐸𝑆 -𝑅𝑃𝐵EQUATIONwhere 𝛼 and 𝛽 is trainable distance-sensitive factors to control the sensitivity to distances between reference pixel and the others in the same window and the symbol * represents the multiplication of each position in the matrix. MLP consists of two linear layers and an activation layer sandwiched between them.GRL consists of two parts: a grouping scheme and a residual structure. Grouping scheme is proposed to efficiently reduce the parameters and calculations without severe performance degradation, and residual structure makes the network find the optimal solution in the residual space, thereby reducing the difficulty of feature learning. The explanation is as follows:• Explanation of the effectiveness of grouping scheme. First, we analyse the effectiveness in reduction of parameters and MACs. We assume that the number of input features is N. Then, both the parameters and the MACs occupied by the linear layer are 𝑁 2 . But, if we group the input features into halves and use two linear layers with the number of input features of 𝑁 2 to process the two parts of input features, respectively, both parameters and MACs occupied by the two linear layers are only 𝑁 2 2 . Obviously, the grouping scheme can halve the number of parameters and MACs. Second, we analyse the effectiveness analysis in preventing severe performance degradation. A possible concern about the grouping scheme is that grouping features will result in a lack of aggregation of the two groups of features, leading to performance degradation. We will analyze below that, at least for Q, K matrices, this worry is unnecessary. As shown in the self-attention formula: Attention(𝑄, 𝐾, 𝑉 ) = Softmax 𝑄𝐾 𝑇 √ 𝑑 𝑘 𝑉 , Q and K matrices are only used to perform matrix product and get 𝑄𝐾 𝑇 matrix. During the matrix product of Q and K, even if we employ the grouping scheme to divide the input features into two groups and perform feature aggregation individually, the subsequent Table 1 : Quantitative evaluations of the lightweight GRFormer against state-of-the-art methods on commonly used benchmark datasets. Best and second best results are marked in red and blue colors. #Params means the number of the network parameters. #MACs denotes the number of the MACs which are calculated on images with an upscaled spatial resolution of 1280 × 720 pixels. #Weighted-Avg means the weighted average PSNR and SSIM on five benchmark datasets. #Weighted-Avg = 5 𝑖=1 𝑀 𝑖 × 𝑆𝑐𝑜𝑟𝑒 𝑖 , where M is the number of images in the dataset and Score is the corresponding PSNR or SSIM score.Year #Params(/K) #MACs(/G) Set5 Set14 B100 Urban100 Manga109 #Weighted-Avg matrix product of Q and K will deeply aggregate the two groups of features. The process can be shown as follows. Specifically, given input of self-attention as X ∈ 𝑅 𝑁 ×𝐶 , where N is the number of pixels in a window and C is number of input features, we cut up X into 2×2 blocks as 𝑋𝑀 𝐾 1 , 𝑀 𝐾 2 ∈ 𝑅 𝐶 2 × 𝐶 2 .Then we can perform matrix product of X and the parameter matrices of Q, K projections respectively asEQUATION)where ∨ represents the logical symbol "or". Then, we perform the matrix product of Q and K as follows:EQUATIONthen, each element in matrix of 𝑄𝐾 𝑇 is calculated: 𝑂 11 =It can be seen from Equation 14, in the matrix product of Q and K of the self-attention mechanism, both 𝑋 11 and 𝑋 12 are used to perform a series of matrix products to get 𝑂 11 , 𝑂 12 , and so do 𝑋 21 and 𝑋 22 to get 𝑂 21 , 𝑂 22 , which aggregates two groups of features. Therefore, the matrix product of Q and K will offset the weak aggregation of input features brought by the use of grouping scheme. In other words, our grouping scheme won't lead to lack of aggregation of the two groups of features and thereby won't lead to severe performance degradation. • Explanation of the effectiveness of residual structure for the QKV linear layer. As shown in Formula 8, self-attention involves multiple matrix products, which makes it difficult for the network to learn the optimal parameters. Given the input of self-attention as X and the optimal Q, K, V as 𝑄, 𝐾, 𝑉 , if we add the residual structure for QKV linear layer, the network only needs to optimize 𝑄-X, 𝐾 -X, 𝑉 -X instead of 𝑄, 𝐾, 𝑉 . The residual structure transforms the learning space of QKV linear layer from linear space to residual space, enhancing the feature learning ability of QKV linear layer. We mainly analyze the effectiveness of ES-RPB from the perspective of parameter reduction and performance improvement for ES-RPB.• Explanation of parameter reduction for ES-RPB. We assume that the size of the window of self-attention is W×H and the selfattention head is 1. The number of parameters occupied by the original RPB is (2×W-1)×(2×H-1). Specifically, the window size is usually set to 16×16, so the parameter count is 961 in this case. In contrast, if we suppose that the number of features of the hidden layer in MLP is 𝐶 ℎ𝑖𝑑𝑑𝑒𝑛 , the amount of parameters occupied by ES-RPB is 3𝐶 ℎ𝑖𝑑𝑑𝑒𝑛 . Specifically, 𝐶 ℎ𝑖𝑑𝑑𝑒𝑛 is set to 128 in our GRFormer, so the number of parameters occupied by ES-RPB in GRFormer is 384, which is much less than 961. What has to be noted is that, when the height and width of window grow simultaneously at a linear rate, the parameters occupied by original RPB will grow squarely, while the parameters occupied by ES-RPB will not grow. • Explanation of performance improvement for ES-RPB. The ES-RPB mechanism within GRFormer improves performance through three key strategies. Firstly, instead of training positional parameters directly, which can be noise-sensitive, we utilize a tiny MLP to get the 𝐵 𝐸𝑆 -𝑅𝑃𝐵 . This approach minimizes the noise impact on positional parameters during training. Secondly, the mechanism enhances interaction of parameters representing relative position information by training them through this tiny MLP rather than in isolation. Thirdly, ES-RPB introduces a distancesensitive design. It employs an exponential function to map the absolute positional coordinates (Δ𝑋, Δ𝑌 ) from linear space to exponential space, resulting in Δ 𝑋 and Δ 𝑌 . This transformation ensures that positions closer to the reference pixel exhibit more significant changes, aligning with the principle that nearer pixels should attract more attention. 
experiments: 
 In this section, we conduct experiments on the lightweight image SR tasks, compare our GRFormer with existing state-of-the-art methods, and do ablation analysis of the proposed method. , 270 • and randomly flip images horizontally for data augmentation. We adopt the Adam optimizer with 𝛽 1 = 0.9 and 𝛽 2 = 0.99 to train the model for 600k iterations. The learning rate is initialized as 2 × 10 -4 and halves on {250000, 400000, 510000, 540000}-th iterations. We use L1 loss to train the model. The whole process is implemented by Pytorch on NVIDIA GeForce RTX 3080 GPUs.We compare our GRFormer with commonly used lightweight SR models for upscaling factor ×2, ×3, ×4, including EDSR [13] , CARN[1], LatticeNet [17] , SwinIR [12] , SwinIR-NG [4] , HPINet [14] , DLGSANet [11] , SRFormer [22] . We compare the parameters, calculations as well as performance on five commonly used SR benchmark datasets (Set5, Set14, B100, Urban100, Manga109). The comparison results are grouped for ×2, ×3, ×4 upscaling factor.Quantitative Comparison Table 1 shows quantitative comparisons in terms of PSNR and SSIM on five benchmark datasets. As shown in Table 1 , our GRFormer achieves the best PSNR score and SSIM score for ×2, ×3, ×4 task on Set14, Urban100, Manga109 and weighted average of the five benchmark datasets, and the PSNR score and SSIM score achieved by our GRFormer is either quite close or superior to that of SOTA model on Set5 and B100. What's more, GRFormer outperforms SOTA model on Urban100 and Manga109 by a large margin. It is worth noting that, our GRFormer outperforms SwinIR-light by a maximum PSNR of 0.42dB and SOTA by a maximum PSNR of 0.23dB, which is a significant improvement for image SR. Furthermore, our GRFormer outperforms SOTA model by about 0.1dB on the weighted average of the five benchmark datasets for ×2, ×3, ×4 task. Although our GRFormer achieves great performance, the parameters and MACs of GRFormer is relatively low. As shown in Fig. 1 (a) , compared with the self-attention of other transformer-based SR models, our GRSA has the smallest number of parameters and calculations.Qualitative Comparison We further show visual examples of common used methods under scaling factor ×4. As shown in Fig. 4 , we use three images reconstructed by EDSR, CARN, LatticeNet, SwinIR, SwinIR-NG, SRFormer, HPINet and GRFormer to make qualitative comparisons.First, we make qualitative comparisons on Urban100-img062. We can see that, the texture and color of the image reconstructed by EDSR [13] , CARN[1] and LatticeNet [17] are distorted, and the lower right corner of the restored image is severely distorted. SwinIR, SwinIR-NG and SRFormer reconstruct part of the texture well, but there are still large areas with severe distortion. The relatively large area of image texture reconstructed by HPINet is distorted. The image reconstructed by our GRFormer has the smallest distortion area, and the restored color is also closest to HR.Second, we make qualitative comparisons on Urban100-img028. Urban100-img028 is an image of the ground whose texture regularly changes from large to small. The distorted and blurred areas of the image reconstructed by EDSR, CARN, LatticeNet and SRFormer are visibly large. For the middle area of the image, the image reconstructed by SwinIR is relatively blurry and the image reconstructed by HPINet is slightly blurry. There is relatively large deformation in the image reconstructed by SwinIR-NG. Apparently, compared with other models, the picture quality recovered by GRFormer is the best and GRFormer has good reconstruction effect on a relatively small texture scale. Third, we make qualitative comparison on Urban100-img074. The direction of the window frame in Urban100-img074 (a) is clear, which can help us easily distinguish whether it is distorted. It can be easily seen that except GRFormer, the image reconstructed by all other methods are more or less distorted. Obviously, our GRFormer reconstructs the image very well in various details, which shows the superiority of our method.We conduct ablation experiments to study the effect of Grouped Residual Layer (GRL) and Exponential-Space Relative Position Bias (ES-RPB). Ablation experiments are trained on DF2K and evaluated on the Set5, Set14, B100, Urban100, Manga109 datasets. PSNR and SSIM are adopted to evaluate the perceptual quality of recovered images. We also adopt parameters and MACs on images with an upscaled spatial resolution of 1280 × 720 pixels to evaluate the complexity.Specifically, ablation experiments are conducted as follows. First, we start with a complete model with GRL and ES-RPB (model 1 ○). Second, we replace the GRL of model 1 ○ with a linear layer to get model 3 ○. Third, we replace the ES-RPB of model 1 ○ with RPB in SwinIR [12] to obtain model 2 ○. Finally, to prove the effectiveness of the grouping scheme and residual structure in GRL, we retain the ES-RPB and remove the grouping scheme and residual structure separately to obtain model 4 ○ and model 5 ○ respectively. The results are shown in Table 2 .Effectiveness of GRL As shown in Table 2 , compared with model 3 ○, #SA-Params, #Params, #SA-MACs and #MACs of model 1 ○ reduce by 47%, 17%, 43%, 17% respectively. A significant reduction in #SA-Params and #SA-MACs can be seen, because our methods mainly act on self-attention. Although the number of parameters and MACs are significantly reduced, application of GRL barely degrades performance, which shows the superiority of GRL. To further show the effectiveness of grouping scheme and residual structure in GRL, we conduct further experiments to get the model 4 ○ and model 5 ○ respectively.• Effectiveness of grouping scheme. Compared with model 4 ○, #SA-Params, #Params, #SA-MACs and #MACs of model 1 ○ reduce by 47%, 17%, 43%, 17% respectively, but model 1 ○ suffers some performance degradation, especially on Urban100 as well as Manga109. However, our residual structure for QKV linear layer can greatly reduce performance degradation. ○. As shown in Fig. 2 , we can see that, the curve of ES-RPB in model 1 ○ is roughly similar to that of RPB in model 2 ○, because both ES-RPB and RPB represent the relative position information. As shown in Fig. 2 , after comparing the curves of RPB and ES-RPB, we can find two differences: First, minor fluctuations on the curve of ES-RPB are much less than that of RPB, which means that most of noise interference is removed. Secondly, we can see from the curve of RPB that RPB is overfitted, which will lead to poor generalization ability. In contrast, our ES-RPB use simpler structure to improve generalization ability. It means that RPB learns a lot of content that is not universally applicable, which affects its generalization ability to represent relative position information. 
conclusion: 
 In this paper, we propose GRSA, an efficient self-attention mechanism which consists of two components: GRL to significantly reduce the amount of parameters and calculations as well as ES-RPB to efficiently and effectively represent the relative position information and make it distance-sensitive. Within GRL module, we use grouping scheme to reduce redundancy in terms of parameters as well as calculations with as little performance degradation as possible and residual structure to enhance feature learning ability for QKV linear layer. Based on GRSA, we design a simple yet effective model for lightweight single image super-resolution, called GRFormer. Benefiting from GRL and ES-RPB, GRFormer not only significantly reduces the number of parameters and MACs, but also enhances the performance in terms of PSNR and SSIM. Experimental results show the superior performance of GRFormer over previous stateof-the-art lightweight SR models on benchmark datasets, especially on Urban100 and Manga109. We hope our GRSA can serve as a useful tool for future research on the design of SR models.  
 
 Appendix:  


Decision: Given the significant performance improvements demonstrated by GRFormer but also considering the areas requiring further clarification and expansion, the paper currently stands at a borderline accept. Addressing the aforementioned recommendations will significantly enhance the paper's quality and may lead to a more favorable evaluation in future submissions.

Reviews: review: paper_summary: paper_summary: This paper designs a lightweight single-image super-resolution model, called GRFormer, and proposes a novel self-attention mechanism, GRSA, which consists of two core components: the GRL and the ES-RPB. The GRL reduces parameter and computational redundancy through a grouping scheme, and enhances the feature learning capability of the linear layer of the QKV through the residual structure. ES-RPB is used to represent relative position information effectively and efficiently.
strength_weakness: strength_weakness: 1. The proposed ES-RPB achieves higher performance with a smaller number of parameters in a simple and effective manner.
2. This paper is well-organized, clearly explains the motivations, and analyzes the reasonableness of the methodology.
limitations: limitations: 1. The experiment is inadequate. In the introduction, it is claimed that “the GRSA module can seamlessly replace the self-attention module and its variants in other transformer-based SR models”. Relevant experiments are necessary to demonstrate the potential of the GRSA module.

2. Grouped convolution has been proven to reduce the number of operations and parameters and has been widely used in various lightweight models. The authors should further compare it with existing methods and clarify the unique innovations of the proposed method.

3.Some small mistakes:
a) “To sum up,our contributions can be summarized as follows:” should be “To sum up, our”
b) “GRL consists of twp parts” should be “GRL consists of two parts”
c) Figure 1 should indicate what each axis represents.
paper_summary: paper_summary: This paper introduces GRFormer, an efficient and lightweight single-image super-resolution (SISR) method. GRFormer significantly reduces parameter overhead and computation while enhancing performance by incorporating a Grouped Residual Self-Attention mechanism (GRSA). Extensive experiments are conducted to show the effectiveness of the proposed approach.
strength_weakness: strength_weakness: 1. This paper proposes an effective  lightweight model GRFormer for image super-resolution.
2. The proposed method achieves notable performance improvements compared to previous work.
limitations: limitations: 1. The writing quality of this paper is insufficient, with an excessive amount of textual description and a lack of relevant visual experiments to demonstrate the effectiveness of the proposed method. For instance, it is unclear whether the proposed modifications actually reduce redundancy in self-attention. Additionally, there are many issues with the citation of references.
2. The content and innovativeness of this paper are insufficient, as it essentially makes only minor modifications to the extraction of QKV and positional encoding, and it focuses solely on the task of lightweight super-resolution.
3. The ablation experiments are incomplete and somewhat disorganized. Table 2 lacks a baseline, and it is unclear why the ablation experiments are conducted using the DF2K dataset.
4. The model uses a larger window size (i.e., 8×32), and the performance improvement might be primarily due to the increase in window size rather than the proposed methods [1]. This has not been clarified through ablation studies.
5. For the task of lightweight super-resolution, comparing the increase in model inference speed, rather than just the number of parameters and MACs, might better reflect the lightweight nature of the model.

[1] SRFormer: Permuted Self-Attention for Single Image Super-Resolution, ICCV2023.
paper_summary: paper_summary: This paper proposed Grouped Residual Self-Attention Transformer (GRFormer) for lightweight image SR. The authors primarily focus on enhancing the self-attention mechanism by introducing two key components: the Grouped Residual Layer (GRL) and Exponential-Space Relative Position (ES-RPB).  The proposed methods show comparable performance with previous advanced lightweight SR methods.
strength_weakness: strength_weakness: 1. The proposed ES-RPB effectively reduced the noise and parameters between positional embedding bias, and the authors provided good visualized examples to further illustrate its effectiveness. This insight is interesting and novel.
2. The overall writing is clear and easy to follow, facilitating comprehension for readers.
3. The visual comparison is good.
limitations: limitations: 1. The overall improvements are relatively minor compared with the second best, especially in SR $\times2$ and $\times3$.
2. The proposed methods are only conducted on lightweight SR methods, however more solid methods e.g., ELAN (ECCV 2022) demonstrate its effectiveness on both lightweight and large-scale tasks. Additionally, the ablation study of the proposed methods are not adequate, so the overall workflow is not solid for this paper.
3. The window size is $8\times32$, and this will contribute a lot to the overall performance but not equal with previous methods.


Minor points:
1. The motivation behind the proposed GRL remains ambiguous. It is unclear why the feature is partitioned into two segments, undergoes linear transformation, and is subsequently concatenated together.
2. $\times$ and $x$ are not consistent in the manuscript. 
3. QKV linear layer is better to declare as Query, Key, and Value
(QKV) linear layers in the beginning.
paper_summary: paper_summary: The paper introduces GRFormer, a lightweight image super-resolution (SR) model. The authors designed the Grouped Residual Self-Attention (GRSA), an efficient self-attention mechanism. GRSA leverages a Grouped Residual Layer (GRL) to reduce the computational complexity of linear operations and an Exponential Space Relative Position Bias (ES-RPB) as a replacement for RPB. Experiments demonstrate that the proposed method achieves better SR performance.
strength_weakness: strength_weakness: 1. The introduction of Grouped Residual Self-Attention (GRSA), utilizing a Grouped Residual Layer (GRL), reduces complexity and parameter count while maintaining performance.
2. The design of the ES-RPB is well-suited for IR tasks.
3. The method show better performance when compared with existing lightweight approaches.
limitations: limitations: 1. The paper does not provide concrete comparisons of running times. Although the proposed GRL and ES-RPB reduce the number of parameters and computational complexity, operations such as split operation, residual connections, and MLP significantly increase runtime, and the first two operations do not add to the computational complexity. Thus, merely reducing parameters and FLOPs is of limited significance.
2. In the ablation studies, there is a lack of a baseline without both GRL and ES-RPB. Additionally, the baseline used in the paper is not clearly defined. The method section indicates the use of global SA in GRSA, which contradicts the low computational complexity. Therefore, clarification is needed on the specific self-attention variant used in GRFormer.
3. Typographical error: '285 Swin Transformer v2' (L285, SwinIR-v2).
4. Increased citations are needed; the paper compares various Transformer-based methods in experimental sections but should also expand the related work section to include these comparisons.