Recursive Generalization Transformer for Image Super-Resolution
Authors: Zheng Chen
Yulun Zhang
Jinjin Gu
Linghe Kong
Xiaokang Yang
Venue: ICLR 2024 poster (2024)
Keywords: Transformer
image super-resolution
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: Recursive Generalization Transformer for Image Super-Resolution

Abstract: Transformer architectures have exhibited remarkable performance in image super-resolution (SR). Since the quadratic computational complexity of the self-attention (SA) in Transformer, existing methods tend to adopt SA in a local region to reduce overheads. However, the local design restricts the global context exploitation, which is crucial for accurate image reconstruction. In this work, we propose the Recursive Generalization Transformer (RGT) for image SR, which can capture global spatial information and is suitable for high-resolution images. Specifically, we propose the recursive-generalization self-attention (RG-SA). It recursively aggregates input features into representative feature maps, and then utilizes cross-attention to extract global information. Meanwhile, the channel dimensions of attention matrices ($query$, $key$, and $value$) are further scaled to mitigate the redundancy in the channel domain. Furthermore, we combine the RG-SA with local self-attention to enhance the exploitation of the global context, and propose the hybrid adaptive integration (HAI) for module integration. The HAI allows the direct and effective fusion between features at different levels (local or global). Extensive experiments demonstrate that our RGT outperforms recent state-of-the-art methods quantitatively and qualitatively. Code and pre-trained models are available at https://github.com/zhengchen1999/RGT.

Published as a conference paper at ICLR 2024
RECURSIVE GENERALIZATION TRANSFORMER FOR IMAGE SUPER-RESOLUTION
Zheng Chen', Yulun Zhang' .t Jinjin Gu Linghe Kong',, Xiaokang Yang' Shanghai Jiao Tong University, Shanghaii AI Laborator, 'The University of Sydney
ABSTRACT
Transformer architectures have exhibited remarkable performance in image super- resolution (SR). Since the quadratic computational complexity of the self- attention (SA) in Transformer, existing methods tend to adopt SA in a local re- gion to reduce overheads. However, the local design restricts the global context exploitation, which is crucial for accurate image reconstruction. In this work, we propose the Recursive Generalization Transformer (RGT) for image SR, which can capture global spatial information and is suitable for high-resolution images. Specifically, we propose the recursiv-genneraliaationn self-attention (RG-SA), It recursively aggregates input features into representative feature maps, and then utilizes coss-attentionn to extract global information. Meanwhile, the channel di- mensions of attention matrices (guery, key, and value) are further scaled to miti- gate the redundancy the channel domain. Furthermore, we combine the RG-SA with local self-attention to enhance the exploitation of the global context, and propose the hybrid adaptive integration (HAI) for module integration. The HAI allows the direct and effective fusion between features different levels (local or global). Extensive experiments demonstrate that our RGT outperforms recent state-of-the of-the-arr methods quantitatively and qualitatively. Code and pre-trained models are available at //ttps:://gii com/ zhengchen 999 RCT
INTRODUCTION
Image super-resolution (SR) aims to recover high-resolution (HR) images from its low-resolution (LR) counterpart. Image SR is an ill-posed problem, there are multiple solutions that can map to any given LR input. To tackle this challenging inverse problem, researchers have proposed numerous deep convolutional neural networks (CNNs) (Dong et al. 2014 [Lm et al. 2017 Zhang et al. 2018b) Mei et al.] 2020) in the past few years. Thanks to their impressive performance against conventional approaches, CNNs have almost dominated the field of SR.
However, deep CNNs still suffer from a limitation in global context awareness, due to the local processing principle of the convolution operator. Recently, alternative method, Transformer, has exhibited considerable performance compared with CNN-based methods on multiple high-level computer vision tasks (Dosovitskiy al. 2021; Liu al., 202 I
Wang al. 2021 Chu al. 1021 Yang et al. 2022). Transformer is first developed in the natural language processing (NLP) field, and its core component is the
(SA) mechanism. This mechanism can directly model long- range dependencies by capturing the interaction between all input data. However, the computational complexity of the vanilla self-attention grows quadratically with image size, limiting its application in high-resolution scenarios, especially low-level vision tasks (e.g., image SR).
To apply Transformer in image SR, several methods have been proposed to reduce the computational cost of self-attention. Some researchers apply local (window) self-attention, which divides the fea- ture maps into sub-regions to limit the scope of self-attention. Meanwhile, they utilize the shift mechanism (Liang al. 2021). overlapping windows (Chen et al. (022b), or the cross rross-aggregation operation (Chen et al. 2022c), to enhance the interaction between windows. These methods achieve linear complexity with respect to image size and outperform previous CNN-based methods. How- ever, compared with global attention, the local design needs to stack many blocks to establish global dependencies. Furthermore, some methods propose "transposed" self-attention (Zamir et al.) 2022) that operates across the channel dimension instead of the spatial dimension. Although this method
Cooresponning authors: Yulun Zhang, yulun 100@ gmail.com Linghe Kong, kinghe.kong@e @ sjtu.edu.cn. TTe work was mainly done when Yulun Zhang was at ETH Zürich.

---

Published as a conference paper at ICLR 2024
can implicitly capture global information, it hinders modeling spatial dependencies, which is cru- cial to image SR. Therefore, there is need to develop a method for image SR that can effectively capture global spatial information with low computational cost on high-resolution images.
In this paper, we propose the Recursive Generalization Transformer (RGT) for image SR, which model global spatial information and is suitable for high-resolution images. Specifically, we self-attention (RG-SA) to explore global information directly
propose the eecursive-generaizan in linear computational complexity. The RG-SA first generalizes the input features of arbitrary resolution into representative feature maps with a small, constant size, via the recursive general- ization module (RGM). Intuitively, the global information aggregated into representative maps. Then cross-attention is utilized between input features and representative maps to exchange global information. Since the size of representative maps is much smaller than input features, the whole process is at low computational cost. Moreover, the RG-SA further adjusts the channel dimension of query, key, and value matrices in SA mitigate the redundancy in the channel domain.
Furthermore, considering that the RG-SA aggregates the image features via the RGM, it is inevitable to lose some local details. Thus, we combine the RG-SA with the local self-attention (L-SA) in an alternate arrangement to better utilize the global context. To enhance the integration of two different SA modules, we propose the hybrid adaptive integration (HAI), which acts on the outside of each Transformer block. The HAI directly fuses features at different levels (local global) before and after the block. Besides, HAI adaptively adjusts the input features through a learnable adaptor for feature alignment. Overall, equipped with the above designs, our RGT can capture global information for accurate image SR while the complexity is manageable.
Our contributions can be summarized as follows:
We propose the Recursive Generalization Transformer (RGT) for image SR. The RGT is capable of capturing global spatial information and is suitable for high-resolution images. Our RGT obtains notable SR performance quantitatively and visually.
We propose the recursive-generallin gregation module and noss-attention to model global dependency with linear complexity.
self-attention (RG-SA), utilizing the recursive ag-
We further combine RG-SA with local self-attention to better exploit the global context, and propose the hybrid adaptive integration (HAI) for module integration.
RELATED WORK
Image Super-Resolution. In recent years, CNN-based methods have shown superior performance over conventional SR approaches. SRCNN (Dong et al.) (014) is the pioneering work, that in- troduces three convolutional layers for image SR. Following this attempt, many works deepen the architecture to improve performance (Zhang et al.) 2019: Magid et al.) 2021 Dai et al. 2019). VDSR (Kim et al. 2016) introduces residual learning to build network with 20 layers. EDSR (Lim) et al. 2017) further simplifies residual block, which allows deeper networks. RCAN (Zhang al. 2018a) proposes a residual-in iittaal-i--ree re-risiuualll structure to train a model over 400 layers. Moreover, numer- ous spatial and channel attention mechanisms (Zhang al. 2019 Liu al.) 2020 Zhou et al. 2020)
are proposed to improve the reconstruction quality. For instance, HAN (Niu et al. (0200) proposes a layer attention module and channel-spatial attention. Although these CNN-based methods produce remarkable results, they still suffer from a limitation in global modeling capability.
Vision Transformer. Transformer is proposed in natural language processing (NLP) and has been adapted to multiple high-level vision tasks, such as image classification (Dosovitskly et al.) 202T Liu al. 2021 Dong et al., (002)) semantic segmentation (Xie al., (021), and object detec- tion (Yang et al. 2022 Tu et al., 2022). Due to the impressive performance in high-level tasks, Transformer has also been introduced to low-level vision tasks (Liang et al., 2021 Zamir et al.]
2022 Wang et al., 2022 Tsai et al. Chen et al., 2022c 2023 Li et al. (023). including im- age SR. SwinIR (Liang et al. 2021), following the design Swin Transformer (Liu et al., 2021), utilizes local window self-attention and shift mechanism. ELAN (Zhang et al. (022) proposes multi-scale self-attention to reduce the computational cost. CAT (Chen et al.) (022c) designs the rectangle-windd window self-attention aggregate the features across different windows. These methods all reduce computational complexity by applying self- atf-attention within local regions. However, the local designs restrict the exploitation of global information that is crucial to image SR.
2

---

Published as a conference paper at ICLR 2024
Residual Group (RG)
Pixel shuuffe
Cov Sov
Cow Cow-sk3
Cov 3x3
RG
L-SA
RG-SA
L-BA
RG
Cov
LN
L-SA
LN
MLP
LN
RG-SA
LN
MLP
Layer iration
Mu Multayyer Perceptron
L-SA Block
RG-SA Block
HAI
HAI
+
Element wise Addition
X
The architecture of the Recursive Generalization Transformer (RGT), The local self- attention (L-SA) blocks, and recursive- suuusive-generalzzatoon self-attention (RG-SA) blocks are alternately arranged. og is a learnable adaptor in the hybrid adaptive integration (HAI) of the 7ch block.
Figure 1:
Global Attention. To reduce the computational complexity of vanilla self-attention, in addition to local design, several works attempt to propose global attention with low overheads (Wang et al.
2021 Tu al. 2022 Yang al. 2022 Chen al. 2022aa] Ali al. 2021). PVT (Wang
2021
designs a spatial-reduction module to merge tokens of key and value. MaxViT (Tu et al. 2022
proposes the grid attention gain sparse global attention. Scalable ViT (Yang et al., (022) scales attention matrices from both spatial and channel dimensions. Although these methods reduce the computational complexity to certain extent, the theoretical complexity remains quadratic, which hinders their effective application on high-resolution images. Region VìT Chen et al. (2022a) cap- tures global information among regional tokens to alleviate the overhead of global attention. More- over, XCIT (Ali et al.) 2021) proposes "transposedd version of self-attention that operates across channels dimension rather than the spatial dimension to achieve linear complexity, However, it can- not explicitly model the spatial relationship. In contrast, propose the eeursive-generalizziee self-attention (RG-SA), which can explore global spatial information in linear complexity.
METHOD
We propose the Recursive Generalization Transformer (RGT) for image SR, which capable of capturing global context and handling high-resolution images effectively. In this section, we first introduce the architecture of RGT. Then, we focus on our proposed recursive-ennrralizatonn self- attention (RG-SA) and hybrid adaptive integration (HAI).
3.1 OVERALL ARCHITECTURE
The overall architecture of our proposed RGT illustrated in Fig. [ consisting of three modules: shallow feature extraction, deep feature extraction, and image reconstruction. Given low-resolution (LR) image I #x x3 RGT first leverages convolutional layer as the shallow feature extrac- where H, W, and C' represent the image height,
tion to get the low-level feature F FoER"xwXX width, and channel number. Fo used for the deep feature extraction module, which composed of N₁ residual groups (RGs) and a convolutional layer. Through this module, the Fo is trans- formed into the deep feature F " connection, and processed by the reconstruction module to generate the high-resolution (HR) im- age R ft x x3 where И and W. are the output height and width. The reconstruction module consists of pixel-shuffle (Shi et al.) 2016) and convolutional layers.
Finally, the Fo and F a are fused through residual
Each RG contains N₂ Transformer blocks and convolutional layer. And the residual connection is employed to ensure training stability. As shown in Fig. [ there are two types of Transformer blocks: Local Self-Attention (L-SA) blocks and RG-SA blocks. The two types of blocks are arranged al- ternately to organize the topological structure. Each Transformer block is composed of two layer normalization (LN), self-attention, and multilayer perceptron (MLP) (Vaswani et al., 2017). Mean- while, the HAI acts outside each Transformer block with a learnable adaptor apply the recently proposed rectangle- windoww self-attention (Rwin-SA) (Chen et al. 2022c) as local self-attention by default. Next, we pay more attention our proposed RG-SA and HAI.
In this work,
3

---

Published as a conference paper at ICLR 2024
Reshape
Wo Q
RGM
DV/-oow strabesh
DW Cony
Come 1x
Restape
Wy
bunkly
Tra
AXCWKG
Recursion?
X
Wi
HNWF X
HWW NC
Xout
Figure 2: The illustration of the recursive- recursive-enneralzzne generalization self-attention (RG-SA). The RG-SA first generates the representative feature maps with constant size (hxw), through the recursive general- ization module (RGM). Then, the cross rross-attention between input features and representative maps is performed to capture global information. ) denotes matrix multiplication.
3.2
SEAASIIVE--EERAIZZATIIIO SELF-ATTER SETENTITNN
The vanilla self-attention (SA) mechanism establishes connections between all input tokens. Al- though can capture global context, the SA suffers the quadratic computational complexity with image size, limiting its the application of SA in high-resolution scenarios, including image SR. To tackle this problem, we propose the recursive- recursie--enneraiizatio generalization self-attention (RG-SA), shown in Fig. 2 that maintain linear computational complexity while capturing global information.
The RG-SA first aggregates input image features of arbitrary resolution into compressed maps (de- noted as representative feature maps), through the recursive generalization module (RGM), Intu- itively, the representative maps aggregate the information of the whole image features, providing a global view of the image. Then, the cross-attention is calculated between input features and repre- sentative maps. Therefore, each token in the input image features can obtain a global receptive field. Meanwhile, further scale the channel dimension of query, key, and value matrices in attention to mitigate the channel redundancy. It improves the performance and reduces consumption.
For simplicity, our RGM only consists of depth-wis and pixel- wise convolutions, shown in Fig. 2 Given an input X... # x xC we first compress spatial size of the features by recursively reusing single depth-wise convolution T T'-loo,, H. times obtain the rough aggregation maps xx where is the convolution stride size, f is constant, and w==W<< Without loss of generality, we assume W IH, then w Sh. Next, we refine the rough aggregation maps to generate the representative maps ERhxxxX, through 3x3 depth-wise convolution and 1 xx point-wise convolution. The RGM is formulated as:
Recursive Generalization Module.
X W (Xm) = W,WW, (X,, (W.(X.)))),,
X, W M.Wll (X()).
where W, is the depth-wise convolution with s, stride, W is the 3x3 depth-wise convolution, and Wp is the x point-wise convolution. Also, the 1 xx1 point-wise convolution scales the channels from C' to C,=C'e,, where c, is the adjustment factor. Through RGM, we can aggregate the global information of the input image features. Meanwhile, the recursive design is flexible for processing inputs with varying sizes (common image SR) by dynamically choosing the recursion times T.
Cross-Attention Subsequently, we reshape and project the input features X as the HWXX (query) and the representative maps the Ke CCC (key), and VeR (value) to compute the cross-attention. The attention matrix ATT HW xhw is calculated from the dot-product interaction of query and key. Note that further scale the channel dimensions of query, key, and value. Overall, the whole cross-attention process is defined as:
Q WQX in, K WKX, V = WWW
SoftMax (QKT) /VC,,
X,) W. V),
where W WOERCXC, WxERC.XC, and WvERG.XC C.XC are learnable parameters and biases are omitted for simplification; W W.ERCCC is the projection matrix for feature fusion. Similar to vanilla self-attention (Vaswani al. 2017 Dosovitskiy al. 2021), divide the channels into
4

---

Published as a conference paper at ICLR 2024
multiple "heads" and execute the attention operation in parallel. Finally, we reshape the result of cross-attention to obtain the output features Xout ER "XR"xxXX Through RGM, tross-attentinn and channel adjustment, our RG-SA can capture global spatial information while maintaining low com- putational overheads. Next, we analyze the complexity of the RG-SA in detail.
Complexity Analysis. Our RG-SA can be divided into two components: RGM and cross-attention. For RGM, the computational complexity (HWC). For cross-attention, the computational com- Here, we analyze the single-head
plexity is self-attention for simplicity. Since and w are constants, and C,=Cxee, the complexity of cross- attention is O CHWC"). In general, The total computational complexity of our RG-SA is linear with the input features size (H x W). Additionally, by applying the small (<1) adjustment factor Cr, we can mitigate the channel redundancy, thus further reducing complexity.
3.3 HYBRID ADAPTIVE INTEGRATION
Alternate Arrangement. In RG-SA, the global information is captured through the cross-attention between input features and representative maps, ensuring low computational overheads. However, the RGM in RG-SA coarse goanse-raiied design, which leads to losing some local details and ul- timately limits modeling global information. To improve the exploitation of the global context, we introduce local self-attention (L-SA) and combine with our proposed RG-SA. Two attention modules are alternately arranged in each residual group (RG), illustrated in Fig.
Analysis and Motivation. Although the two blocks are integrated, the linear arrangement lacks direct interaction between features at different levels (global or local), thus still cannot exploit global information effectively. For further analysis, under the alternating topology, the input and output of each Transformer block are different level features. Specifically, the input of the RG-SA block is the local features generated from L-SA, while the output is the global features. Correspondingly, the input and output of L-SA are global and local features, respectively. This observation inspires us to enhance information fusion by combining the input and output features of each block.
Specific Design. The intuitive idea is directly integrating the input and output features via the vanilla skip connection (He et al. 2016). Nevertheless, since the misalignment between global and local features, simple addition cannot fuse features effectively. To overcome the above issues, we propose hybrid adaptive integration (HAI). As shown in Fig. [ our HAI acts the outside of each Transformer block. The input features are adaptively adjusted by a learnable adaptor a, and added to the output features. The process of the 1th Transformer block B equipped with HAI is:
z' z'=B'' Z'=B'(z =z='(z"- + aq -²²
where 71-1 and 7' represent the input and the output of the 1th Transformer block, R/ is the learnable parameter in the 1 block. Overall, the HAI is able to enhance the integration of different SA modules on the basis of the alternate arrangement, which advances the modeling of global information. Moreover, similar to the regular residual connection, our HIA encourages more information flows to the deep network layers, resulting in better performance.
Visual Results. To intuitively show the ef- fectiveness of HAL, visualize the rel- evant features of one RG-SA block (first row, 2nd block in a RG) and one L-SA block (second row, 5'd block in 1st RG) of RGT in Fig. 3 The deeper color in- dicates larger weights. First, by observ- ing columns (a) and (b), we can find great differences in the input and output of SA modules in some cases. It reveals the mis- alignment between global and local fea- tures. Secondly, directly fused by the vanilla skip connection, the features are over- ine-iinteeeted (top of (c) column), or not changed obviously (bottom of (c) column). In contrast, through HAI, the features shown in column (d) are adaptively changed according different blocks. indicates that the input and output features are fused effectively. More discussions on the role of HAT and the value distribution of learnable adaptors a are given in Sec. 4.2
L-SA RG-SA
(e) Black input
(c) Block oulput
(c) Varilla Skip
(6) HAI
Figure 3: Visualization the features in RGT. a) Block
input. (b) Block output before connection. (c) Vanilla skip connection result: directly add input to output. (d) HAI result: adjust input with before adding output. Please zoom in for a better visualization.
5

---

Published as a conference paper at ICLR 2024
L-SA RG-SA HAI Params(M) FLOPs(G) PSNR(dB) SSIM
Methed
Recursion
[araas(MM FLOPs(G) PSNR(dB)
SSIM
10.69
229.42
0.9396
Recur
10.05
274.54
33,57
0,9412
10.04
183.08
33.52
0.9405
w/o Scale
11.37
189.62
33.54
0.9404
10.05
33.68
0.9414
RGT-S
10.05
183.08
33.68
0.9414
(a) Break down ablation on each component.
(b) Ablation study RG-SA.
Method Vanilla Skip a |Params (M) FLOPS (G) PSNR (dB) SSIM
Method
(l PSN SSIM
w/o HAT
10.04
183.08
33.52
0.9405
L-SA only
10.69
229.42
33.43
0.9396
w/ Skip
10.04
183.08
32.71
0.9339
w/ HAI
10.05
183,08
33.68
0,9414
L-SA w/ HAI
10.69
229.42
33.44
0,9400
(c) Ablation study on HAL
(d) Further ablation study on HAI.
Table 1: Ablation studies. We train models on DIV2K and Flickr2K, and test on Urban100 (x2).
EXPERIMENTS
4.1
EXPERIMENTAL SETTINGS
Data and Evaluation. Following recent works (Zhang et al. 2021 Magid et al. 2021 Liang al. 2021] we choose DIV2K (Timofte et al. (017) and Flickr2K (Lim et al. (017]) as the training data. For testing, we use five standard benchmark datasets: Set5 (Bevilacqua et al., 2012), Set14 (Zeyde) et al. 20TO,, B100 (Martin et al. 2001). Urban 100 (Huang et al. 2015), and Manga109 Matsuu jet al. 2017). We conduct experiments with three upscaling factors: x2, x3, and x4. The low- resolution images are generated by Bicubic (BI) downsampling. To evaluate our method, we use the metries PSNR and SSIM (Wang et al. 20044) on the Y channel of the YCbCr space.
Implementation Details. We set two version models, RGT-S and RGT, with different computational complexity. For RGT-S, we set the residual group (RG) number N₃ as and the Transformer block number N2 for each RG as 6 for L-SA. The channel dimension number, attention head number, and mlp expansion ratio (Dosovitskiy et al.] 202I) are set as 180, 6, and 2, respectively. The window size is set as 8232. The 8, (stride size) and G, (adjustment factor) The representative map size is set as 4 for training, and 16 for testing. For RGT, we increase the number of RG from 6 8, while other settings remain the same as RGT-S.
set as 4 and 0.5 for RG-SA.
Training Settings. We train our models with batch size 32, where each input image is randomly cropped to 64x64 size, and the total training iterations are 500K. Training patches are augmented using random horizontal flips and rotations with 90", 180", and 270". To keep fair comparisons, we adopt Adam optimizer (Kingma Ba 2015) with 31 =0.9 and 00=0.99 to minimize the L1 loss function following previous works (Zhang et al. 20 Dai et al. 2019 Liang et al. 202T) I) The 14440000KKK,,,,,0
initial learning rate set as 2x 10- and reduced by half the milestone We use PyTorch (Paszee et al. 2019) to implement our models with 4 Nvidia A100 GPUs.
4.2 ABLATION STUDY
In this section, we study the effects of different components of our method. For fair comparisons, all models adopt the same basic architecture and settings RGT-S. We conduct experiments on the x2 factor. We adopt the dataset DIV2K (Timofte et al., 2017) and Flickr2K (Lim et al., (017) to train models, and the iterations are 200K. The dataset Urban 100 (Huang et al. 2015) is applied for testing. When we calculate the FLOPs, the input size is 3x 128x 128.
Effects of each component. We conduct break-down ablation experiment to investigate the effects of each component on SR performance. The results are listed in Tab. Ta First, baseline. The baseline model derived by replacing all Transformer blocks in RGT-S with local self-attention (L- SA) (Liang et al., 1 CCen et al., 20226) block and removing HAI. We set the window size of L- SA as 8x32, which is consistent with RGT-S. Second, applying RG-SA. We introduce the recursive- generalization self-attention (RG-SA) into the baseline, and alternately arrange L-SA and RG-SA in successive Transformer blocks. Without changing the structure of the network, the model achieves a 0.09 dB improvement. Meanwhile, compared with the baseline, the FLOPs and parameters of the model are slightly reduced. It shows that our proposed RG-SA is effective regarding parameters and computational complexity. Third, applying HAI We further adopt the hybrid adaptive integration (HAI) and get the final version of our RGT-S. The model obtains the best performance of 33.68 dB. These results demonstrate the effectiveness of the RG-SA and HAT.

---

Published as a conference paper at ICLR 2024
RGT-S w/o HAI
RGT-S w/ HAI
35
0.9
0.040
25
Blacks
20
9.7
0.030
10
0.5
0.000
0.4
0.030
30
5
20
25
25
25
Block Index
Blocks
Blocks
Figure 4: CKA similarity between all pairs of Trans- former blocks in RGT-S without or with HAI.
The value (al//C C=180) distribution of a of RGT-S.
Figure 5:
We investigate the design of our recursive- generalization self-attentiins (RG-SA). We implement ablation experiments on the recursive operation and channel adjustment factor. The results are reported in Tab. Tb First, the impact of recursive operation. We build the model without recursion (w/o Recur) by removing the recursive operation of RGM in RGT-S. Namely, the stride depth-wise convolution is only utilized once to generate the representative maps. We keep s, and 0,5 unchanged. Compared with RGT-S, it can be observed that the usage of recursive operation can effectively reduce the FLOPs by 30%, while achieving better performance. Second, the impact of the channel adjustment factor Cr. We build and train the model without channel scaling (w/o Scale) by setting the adjustment factor C, in RGT-S as (0.5 by default). Compared with RGT-S, discover that scaling the channel dimension yields 0.14 dB performance gain, since the smaller C mitigates the redundancy between channels, thus enhancing the feature expression.
Effects of RG-SA.
Effects of HAL We show the influence of the hybrid adaptive integration (HAI) in Tab. Tc We compared three models: without HAI (w/o HAI), with vanilla skip connection (He et al. 2016) (w/ Skip), and with HAI (w/ HAI). Both skip connection and HAI act on the outside of each Transformer block. First, the impact of vanilla skip connection. Comparing the model w/o HAI and the model w/ Skip, we can find that the simple application of skip connection seriously degrades the model performance by 0.81 dB. This may be due to the misalignment between different level (global or local) features, which prevents their direct fusion. Second, the impact of HAI. In contrast, our proposed HAI adaptively adjusts the input features through learnable adaptor o, thus achieving valid feature integration. With the HAI, the model obtains 0.16 dB gain.
We also apply HAI only models only with L-SA in Tab. Id Since there is no feature misalignment, the performance does not change much with HAI. These results are consistent with analysis in Sec. 3.3 and demonstrates the effectiveness of HAI in solving feature misalignment.
Furthermore, we further introduce centered kernel alignment (CKA) (Cortes et al.) 2012 Kornblith 2019 Raghu et al. (02I)) to study the internal representation structure of the model. The higher the CKA score, the higher the representation similarity. We calculate CKA similarities between all Transformer blocks in RGT-S without and with HAI. The results are shown as heatmaps in Fig. 4 First, enhance module integration. There are obvious differences before and after the 30th block, in the model without HAI. On the contrary, equipping with HAI, it can be found that the transition is more gradual, which indicates that the integration between the modules is more effective. Second, encourage information flow. We observe that there is still high similarity between the initial blocks (0th -2nd and the very deep blocks (28"h-29th in the model with HAI. It means that through HAI, more information can flow to the deep layers of the network.
We also visualize the value distribution of a of RGT-S in Fig. 5 We find that a is diverse based on different blocks, indicating its adaptability. Meanwhile, the value of a increases at the end block in each RG, i.e., blocks 5th 1h 17th, and 23th, It indicates that HAI integrates different modules.
4.3 COMPARISONS WITH STATE-OF-TIEE- ART METHODS
We compare our two models, RGT-S and RGT, with recent state-of-the- art methods: EDSR (Lim) et al.] 2017). RCAN (Zhang et al., 2018a), 8a), SRFBN (Li et al.) 2019). SAN (Dai et al. 2019) Niu et al. 2020). CSNLN (Mei et al. 2020). NLSA (Mei et al. 2021). CRAN (Zhang et al.
HAN
2021 DFSA (Magid et al. 2021 ELAN Zhang et al. 2022). SwinIR (Liang et al. 202 and
CAT-A (Chen et al.) 2022c). Similar ar to previous works (Lim et al. 2017 Zhang et al.
2018a), we
use self-ensemble strategy in testing and mark the model with the symbol
"4"
7

---

Published as a conference paper at ICLR 2024
Method
Scale
Set5
Set14
B100
Urban
Manga
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
EDSR
2017
x2
38.11
0.9602
33,92
0.9195
32.32
0.9013
32.93
0.9351
39.10
0.9773
RCAN
Zhiig et al 2018a
x2
38.27
0.9614
34.12
0.9216
32.41
0.9027
33.34
0.9384
39.44
0,9786
SRFBN
Li el al. 2019
x2
38.11
0.9609
33.82
0.9196
32.29
0.9010
32.62
0.9328
39.08
0.9779
SAN
Da 2019
X2
38.31
0.9620
34,07
0.9213
32.42
0.9028
33.10
0.9370
39.32
0,9792
HAN
Nia et al 2022
X2
38.27
0.9614
34.16
0.9217
32.41
0.9027
33.35
0.9385
39.46
0.9785
CSNLN MMe
nl 2020
38.
0.9616
34.12
0.9223
32,40
0.9024
33.25
0.9386
39.37
0.9785
NLSA
Mai et al. X021
X2
38.34
0.9618
34.08
0.9231
32.43
0.9027
33.42
0.9394
39.59
0,9789
CRAN
Zhang al. 2002
38.31
0.9617
34,22
0.9232
0.9029
33,43
0.9394
39.75
0,9793
DFSA
Mapid el al
2021
X2
38.38
0.9620
34.33
0.9232
32.50
0.9036
33.66
0.9412
39.98
0.9798
ELAN
Ziang et al.
(0122
X2
38.36
0.9620
34.20
0.9228
32.45
0.9030
33.44
0.9391
39.62
0.9793
SwinIR Laang
2021
X2
38.42
0.9623
34,46
0.9250
32.53
0.9041
33.81
0.9427
39.92
0.9797
CAT-A
Chen ot al
21222
x2
38.51
0.9626
34.78
0. 9.925
32.59
0.9047
34,26
0.9440
40.10
0.9805
RGT-S (ours)
38.56
0.9627
34.77
0.9270
32.59
0.9050
34,32
0.9457
40.18
0,9805
RGT (ours)
x2
38..99
0.9628
34,85
0.9272
32.62
0.9050
34.47
0.9467
40.34
0,9808
RGT+ (ours)
38.62
0.9629
0.9474
40.45
0.9810
EDSR Lim al. 2017
X3
34.65
0.9280
30.52
0.8462
29.25
0.8093
28.80
0.8653
34.17
0.9476
RCAN Zhang
201
34.74
0.9299
30 30.65
0,8482
29.32
0.8111
29.09
0.8702
34,44
0,9499
SREBN Li of cJ 2019)
x3
34.70
0.9292
30.51
0.8461
29.24
0.8084
28.73
0.8641
34.18
0,9481
2019
34.75
0.9300
30.59
29.33
0.8671
34.30
0,9494
HAN Niu et al Aiza
x3
34.75
0.9299
30.67
0.8483
29.
0.81
29.10
0.8705
34,48
0,9500
CSNLN
Mei er al 2020
x3
34.74
0,9300
30 66
0. 0.84
29.33 33
0.8105
29.13 13
0.8712
34.45
0,9502
NLSA MS èt al 2021
x3
34,85
0.9306
30.70
0.8485
29.34
0.8117
29.25
0.8726
34.57
0,9508
CRAN
Zhang at. 2202
X3
34.80
0.9304
30.73
0.8498
29.38
0.8124
29.33
0.8745
34.84
0.9515
DFSA
Mapid
34.92
0.9312
30.83
0,8507
29.42
0.8128
29.44
0.8761
35.07
0,9525
ELAN
Zhang et al
2022
X3
34.90
0.9313
30.80
0.8504
29.38
0.8124
29.32
0.8745
34.73
0.9517
SwinIR [Lanng
34.97
0.9318
30,93
29.46
0.8145
29.75
0.8826
35.12
0.9537
CAT-A
Then of al
21226
x
35.06
0.9326
31 ...44
0.85338
29.52
0.8160
30.12
0.8862
35.38
0.9546
RGT-S cours)
x3
35,11
0.9328
31.05
0.8548
29.53
0.8164
30.18
0.8884
35,39
0.9548
RGT (ours)
x3
35,15
0.9329
31.13
0,8550
29.55
0.8165
30.28
0.8899
35.55
0,9553
RGT+ (ours)
X3
38.18
0.9331
31.16
0.8558
29.57
0.8170
30.40
0.8914
35.69
0.9559
EDSR Lim et 2017
X4
32.46
0.8968
28.80
0.7876
27.71
0.7420
26.64
0.8033
31.02
0,9148
RCAN Zhang er al 201 8a
x4
32.63
0.9002
28.5
0.7889
27.77
0.7436
26.83
0,8087
31.22
0.9173
SREBN of 20
32:47
0.8983
28.81
0.7868
27.72
0.7409
26.60
0,8015
31.15
0,9160
SAN Da el all 2 2019
x4
32.64
0.9003
28. 92
0. 0.7888
27.78
0.7436
26.79
0.8068
31.18
0.9169
HAN Niu
32.64
0.9002
28.90 90
0. 0.7890
27,80
0.7442
26.85
0,8094
31.42
0.9177
CSNLN (Mer et al 2020
x4
32.68
0.9004
28. 95
0.7888
27.80 80
0.7439
27.22
0.8168
31 43
0.9201
NLSA
Mei
2021
32.59
0.9000
28.
0.7891
27.78
0.7444
26.96
0.8109
31.27
0.9184
CRAN
Ziang et at 2021
x4
32.72
0.9012
29.01
0.7918
27.86
0.7460
27.13
0.8167
31.75
0.9219
DFSA
Magid at
1021
x4
32.79
0.9019
29,
0.7922
27.87
0.7458
27.17
0.8163
31.88
0.9266
ELAN
Zhang al.
2022
32,75
0,9022
28.96
0.7914
27.83
0.7459
27.13
0.8167
31.68
0,9226
SwinIR
LLung el al
2021
X4
32.92
0.9044
29.09
0.7950
27.92
0.7489
27.45
0.8254
32.03
0.9260
CAT-A
Then
211246
33.08
0.9082
29.18
0.7960
27.99
0.7510
27.89
0.8339
32.39
0.9285
RGT-S cours)
X4
32,98
0.9047
29.18
0.7966
27.98
0.7509
27.89
0.8347
32.38
0.9281
RGT (ours)
33.12
0.9060
29.25
0.7972
28.00
0.7513
0.8369
32.50
0.9291
RGT+ (ours)
X4
33.16
0.9066
29. 28
0.7979
28.03
0.7520
28.09
0.8388
32.68
0.9303
Table 2: Quantitative comparison (PSNR/SSIM) with state-of-the methods. Best and second best results are colored with red and blue. Our methods outperforms other competitors.
Quantitative results. We show the quantitative comparisons for x2, x3, and x4 image SR in Tab. [ We also report the comparisons of computational complexity (e.g.. FLOPs), and parame- ter numbers in Tab. 3 As we can see, our proposed RGT significantly outperforms other methods on all datasets with all scaling factors. Compared with recent Transformer-based methods, such as SwinIR (Liang et al., (021) and CAT-A (Chen et al., 2022c). our proposed RGT achieves better results, particularly on the Urban 100 and Mangal09 datasets. For instance, on the Urban 100 dataset (x2), RGT outperforms CAT-A by 0.21 dB. Meanwhile, the model size and computational com- plexity are lower than CAT-A. Even the small vision model, RGT-S, obtains comparable or better results than compared methods. These comparisons indicate that our proposed RGT can capture more global information compared with previous CNN-based and Transformer- based methods.
Visual Results. We show visual comparisons (x4) in Fig. 6 We can observe that most compared methods suffer from blurring artifacts and cannot recover accurate textures in some representative challenging cases. In contrast, our RGT can alleviate the blurring artifacts better and recover more image details. For instance, in image img 059, some methods fail to reconstruct most of the strips correctly (e.g., SAN and DFSA), while some only restore part stripes (e.g., SwinIR and CAT-A). In contrast, method recovers more precise structures. These visual comparisons demonstrate that our RGT is capable of reconstructing high-quality images by modeling global information. Com- bining with the quantitative comparisons, we further demonstrate the effectiveness of our method.
8

---

Published as a conference paper at ICLR 2024
HR
Bicubic
RCAN
SAN
HAN
Urban 100: img. 059 (x44)
CSNLN
DFSA
SwinIR
CAT-A
RGT (ours)
Urbun 100: O9SS (X4)
CSNLN
DFSA
SwinIR
CAT-A
RGT (ours)
IIR
Bicubic
RCAN
SAN
HAN
Mangal09: Momoyana. (x4)
CSNLN
DFSA
SwinlR
CAT-A
RGT (ours)
Figure
Visual comparison for image SR (x4) in some challenging cases.
Method
EDSR
RCAN
HAN
CSNEN
SwinTR
CAT-A
RGT-S (ours)
RGT (ours)
Params(M)
43.09
15.59
FLOPxGG)
823.34 34
261.01
269.1
84,155.24
215.32
360.67
193.08
251.07 .07
U0ban100
26.64
26.82
26.85
27.22
27.45
27.89
27.98
Manga109
31.02
31.22
31.42
31.43
32.03
32.39
32.38
32.50
Table 3: Comparison of parameters, FLOPs, and PSNR (dB) values on Urban and Manga with scaling factor x4. When we calculate FLOPs, the output size is 31512x112
4.4 MODEL SIZE ANALYSES
We further show the comparison of parameter numbers, FLOPS, and performance with recent image SR methods in Tab. FLOPs are measured when the output size is set as 3x512x512, and PSNR values are tested on Urban 100 and Manga 109 (x4). Our RGT has lower computational complexity and model size than CNN-based methods, EDSR (Lim et al. 2017) and RCAN (Zhang et al. 201 8a). Compared with CSNLN (Mei et al.] (020). our RGT only requires 0.3% computational complexity (i.e., FLOPs). Meanwhile, compared with the recent Transformer-basse performs better, while FLOPs decreased by 30.39% (109.6G) and parameters decreased by 19.46% (3.23M). Compared with SwinIR (Liang et al. (022), RGT has comparable computational complex- ity and model size. Furthermore, to further demonstrate the effectiveness of our method, we provide another version of the model, RGT-S, with lower FLOPs and parameters than SwinIR. Our RGT- still obtains notable SR performance gains compared with other methods. These comparisons indicate that method achieves better trade-off between model complexity and performance.
model. CAT-A, our RGT
CONCLUSION
We propose a new Transformer model, named Recursive Generalization Transformer (RGT), for accurate image SR. Our RGT is capable of modeling global spatial information while maintaining low computational costs. Specifically, we design the secusive-generalizze SA) to extract global information effectively in linear complexity. The RG-SA computes cross- attention between the input features and the representative maps recursively aggregated from the input. Meanwhile, the channel dimensions of attention matrices are further scaled to mitigate the redundancy in the channel domain. Furthermore, to improve the exploitation of the global context, we combine RG-SA with local self-attention, and propose the hybrid adaptive integration (HAT) for module integration. The HAI acts the outside of each Transformer block to directly fuse features at different levels (local or global). Extensive experiments on image SR demonstrate that proposed RGT achieves superior performance recent state-of-the-ar methods.
self-attention (RG-

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported in part by NSFC grant (62141220, U19B2035) and Shanghai Municipal Science and Technology Major Project (2021SHZDZZ01022,,
REFERENCES
Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, al. Xcit: Cross-covariance image transformers. In NeurlPS, 2021.
Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel. Low- complexity single-image super-resolution based on nonnegative neighbor embedding. In BMVC, 2012.
Richard Chen, Rameswar Panda, and Quanfu Fan. Regionvit: Regional-to-local attention for vision transformers. In ICLR, 2022a.
Xiangyu Chen, Xintao Wang. Jiantao Zhou, and Chao Dong. Activating more pixels in image super- resolution transformer. arXiv preprint arXiv:2205.04455) 2022b.
Zheng Chen, Yulun Zhang, Jinjin Gu, Yongbing Zhang, Linghe Kong, and Xin Yuan. Cross aggre- gation transformer for image restoration. In NeurIPS, 2022c.
Zheng Chen, Yulun Zhang, Jinjin Gu, Linghe Kong, Xiaokang Yang, and Fisher Yu. Dual aggrega- tion transformer for image super-resolution. In ICCV, 2023.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang. Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. In NewrIPS, 2021.
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered alignment. JMLR, 2012.
Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang. Second-order attention network for single image super-resolution. In CVPR, 2019.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning deep convolutional network for image super-resolution. In ECCV, 2014.
Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. CVPR, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom,, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. In ICLR, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from trans- formed self-exemplars. In CVPR, 2015.
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In CVPR, 2016.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Simon Komblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In NewrlPS, 2019.
10

---

Published as a conference paper at ICLR 2024
Yawei Li, Yuchen Fan, Xiaoyu Xiang, Denis Demandolx, Rakesh Ranjan, Radu Timofte, and Lue Van Gool. Efficient and explicit modelling of image hierarchies for image restoration. In CVPR, 2023.
Zhen Li, Jinglei Yang. Zheng Liu, Xiaomin Yang, Gwanggil Jeon, and Wei Wu. Feedback network for image super-resolution. In CVPR, 2019.
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In ICCVW, 2021.
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep resid- ual networks for single image super-resolution. In CVPRW, 2017.
Jie Liu, Wenjie Zhang, Yuting Tang, Jie Tang, and Gangshan Wu. Residual feature aggregation network for image super-resolution. In CVPR, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Salma Abdel Magid, Yulun Zhang, Donglai Wei, Won-Dong Jang, Zudi Lin, Yun Fu, and Hanspeter Pfister. Dynamic high-pass filtering and multi-spectral attention for image super-resolution. In ICCV, 2021.
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV, 2001.
Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-based manga retrieval using mangal09 dataset. Multimedia Tools and Applications, 2017.
Yiqun Mci, Yuchen Fan, Yuqian Zhou, Lichao Huang, Thomas Huang, and Humphrey Shi. Image super-resolution with cross-scale non-local attention and exhaustive self-exemplars mining. In CVPR, 2020.
Yiqun Mei, Yuchen Fan, and Yuqian Zhou. Image super-resolution with non-local sparse attention. In CVPR, 2021.
Ben Niu, Weilei Wen, Wenqi Ren, Xiangde Zhang, Lianping Yang, Shuzhen Wang, Kaihao Zhang, Xiaochun Cao, and Haifeng Shen. Single image resoltiino via a holistic attention network. In ECCV, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In NeurIPS, 2019.
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? In NeurlPS, 2021.
Wenzhe Shi, Jose Caballero, Ferene Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using efficient sub-pixel convolutional neural network. In CVPR, 2016.
Radu Timofte, Eirikur Agustsson, Lue Van Gool, Ming-Hsuan Hsuan Yang. Lei Zhang, Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, Kyoung Mu Lee, al. Ntire 2017 challenge single image super-resolution: Methods and results. In CVPRW, 2017.
Fu-Jen Tsai, Yan-Tsung Peng, Yen-V Yu Lin, Chung- Chi Tsai, and Chia- Wen Lin. Stripformer: Strip transformer for fast image deblurring. In ECCV, 2022.
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In ECCV, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.

---

Published as a conference paper at ICLR 2024
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021.
Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: A general u-shaped transformer for image restoration. In CVPR, 2022.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility structural similarity. TIP. 2004.
Enze Xic, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. In NeurIPS, 2021.
Rui Yang, Hailong Ma, Jie Wu, Yansong Tang. Xuefeng Xiao, Min Zheng, and Xiu Li. Scalablevit: Rethinking the context-oriented generalization of vision transformer. ECCV, 2022.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming- Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In CVPR, 2022.
Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse- representations. In Proc. 7th Int. Conf. Curves Surf. 2010.
Xindong Zhang, Hui Zeng, Shi Guo, and Lei Zhang. Efficient long-range attention network for image super-resolution. In ECCV, 2022.
Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang. Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention networks. ECCV, 2018a.
Yulun Zhang, Yapeng Tian, Yu Kong. Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In CVPR, 2018b.
Yulun Zhang, Kunpeng Li, Kai Li, Bineng Zhong, and Yun Fu. Residual non-local attention net- works for image restoration. In ICLR, 2019.
Yulun Zhang, Donglai Wei, Can Qin, Huan Wang, Hanspeter Pfister, and Yun Fu. Context reasoning attention network for image super-resolution. ICCV, 2021.
Shangchen Zhou, Jiawei Zhang, Wangmeng Zuo, and Chen Change Loy. Cross-scale internal graph neural network for image super-resolution. In NeurIPS, 2020.
12


Reviews:
[{"review_id": "eecFebdpB6", "reviewer": "Reviewer_B1Rr", "summary": "The authors propose the Recursive Generalization Transformer (RGT) for image super-resolution. The network can capture global spatial information and is more suitable for high-resolution images. It introduces Recursive-generalization self-attention (RG-SA) to recursively aggregate input features and Hybrid Adaptive Integration (HAI) to integrate global and local modules. Experiments show that the proposed method outperforms recent methods quantitatively and qualitatively.", "questions": "1. The authors should compare RG-SA with other global attention, such as sparse attention proposed in ART (Accurate Image Restoration with Attention Retractable Transformer).\n2. It is suggested to add more comparisons with more SR methods like ART.\n3. Providing visual comparison results on more datasets to strengthen the effectiveness of the model.\n4. Compared with other Transformer methods like SwinIR, are their training datasets the same? Are there any differences in the training strategy?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of using global attention in the Transformer is widespread. But, the authors effectively maintain low computational complexity, which is meaningful in image SR.\n2. Additionally, the proposed HAI is simple yet effective. Both the ablation experiments (Table 1 (c), (d)) and the visual results (Figs. 3, 4, 5) strongly support the authors' claim: integrate global and local modules.\n3. The main comparisons with recent methods demonstrate the superiority of this method. I also notice that the authors provide model size comparisons and a more fair comparison with SwinIR in the supplementary materials.\n4. The paper is carefully organized. The paper is well-written and easy to read. The proposed methods are easy to follow.", "weaknesses": "1. The experiments on RG-SA are not enough. The authors claim the superiority of RG-SA, but it is not compared with other global attention mechanisms.\n2. The authors only provide visual comparisons on Urban100 and Manga109 datasets. Comparisons on other datasets are lacking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5J0Qct1VXG", "reviewer": "Reviewer_fY4p", "summary": "This paper presents the Recursive Generalization Transformer (RGT) for image SR. Specifically, the recursive-generalization self-attention (RG-SA) is proposed to capture global spatial information. And the hybrid adaptive integration (HAI) is designed for module integration. Experiments show that the proposed RGT achieves improvements over recent methods.", "questions": "1. Some details in the paper are not clear. For example, the representative map size \"h\" is set as 4 for training but 16 for testing. Why not use the same settings?\n2. The RGT-S and RGT all adopt a larger window size than SwinIR. To establish a fairer comparison, it is recommended to use the same window size.\n3. It would be beneficial to include comparisons with more recent methods, such as RGT, to evaluate the effectiveness of the proposed method.\n4. A comparison of running times should be given, which is important for practical applications of the model.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors propose the recursive-generalization self-attention (RG-SA), which controls computational complexity while achieving global modeling.\n2. They also design the hybrid adaptive integration (HAI). It is a simple yet effective.\n3. The paper's experiments are comprehensive. The ablation study demonstrates the effects of each component.\n4. Quantitative and qualitative results indicate that the proposed method outperforms SwinIR and CAT-A.\n5. The authors provide various visual results: feature maps and CKA heatmap, which greatly support the author's claims and enrich the overall quality of the paper.\n6. The paper is well-organized, with clear and readable content.", "weaknesses": "1. Some details in the paper are not clear. For example, the representative map size \"h\" is set as 4 for training but 16 for testing. Why not use the same settings?\n2. The RGT-S and RGT all adopt a larger window size than SwinIR. To establish a fairer comparison, it is recommended to use the same window size.\n3. It would be beneficial to include comparisons with more recent methods, such as RGT, to evaluate the effectiveness of the proposed method.\n4. A comparison of running times should be given, which is important for practical applications of the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wvtA4cWNLy", "reviewer": "Reviewer_Pj4p", "summary": "This paper proposed a novel image SR method, Recursive Generalization Transformer (RGT). The RGT uses recursive-generalization self-attention (RG-SA) and hybrid adaptive integration (HAI) to model global information. Extensive experiments demonstrate that the proposed RGT achieves state-of-the-art performance on image SR.", "questions": "- The author mentioned in the paper that the proposed RG-SA realizes linear complexity, which is better than other global methods. So, has the author tried other global attention, such as the attention proposed in PVT or XCiT? How do the computational complexity and performance of RG-SA compare with these methods?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper's writing and organization are good. All illustrations, tables, and visual results are intuitive and clear.\n- The motivation for the proposed method is reasonable. The global information in SR is important while reducing the complexity of global attention is crucial for its application in SR tasks. \n- The proposed components RG-SA and HAI are novel and valuable.\n- The ablation study is extensive. The effectiveness of each part in RGT is demonstrated.\n- The authors provide multiple model versions and compare them with state-of-the-art methods. All results prove that the RGT proposed in this paper is a promising SR method.", "weaknesses": "- When compared with CAT-A, the improvements of RGT on some datasets (Set5, Set14) are not very obvious (< 0.1 dB).\n- Although FLOPs are provided in Sec. 4.4, the running time of the model on real devices should also be provided.\n- The primary evaluation metrics used in the paper are PSNR and SSIM. However, these metrics may not reflect actual SR performance. Some perceptual metrics, such as LPIPS, should be evaluated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2wb1nTmnFz", "reviewer": "Reviewer_bq6U", "summary": "This paper proposes a recursive generalization transformer for image SR. In which, the recursive-generalization self-attention (RG-SA) recursively aggregates input features into representative feature maps, and then extract global information via the cross-attention. Moreover, the authors combine the RG-SA with local self-attention to enhance the exploitation of the global context, and further propose the hybrid adaptive integration (HAI) for module integration. Experiments demonstrate that the proposed RGT achieves good performance.", "questions": "See the weaknesses part", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper is well writting and easy to following. \n2.The author proves the effectiveness of the proposed components by sufficient experiments.", "weaknesses": "1. The comparisons with recent ViT-based methods are insufficient.\n2. The author should also make comparisons on inference speed and GPU usage to evaluate the model computational efficiency.\n3. What is the performance of a larger window SA with HAI?\n4. In the ablation of RG-SA, the authors state that \"a smaller cr reduces redundancy between channels\". Why does fewer channels reduce feature redundancy? \n5. What are the implementation details of dynamically choosing the recursion time T in RGM？", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)