Cross Aggregation Transformer for Image Restoration
Authors: Zheng Chen
Yulun Zhang
Jinjin Gu
Yongbing Zhang
Linghe Kong
Xin Yuan
Venue: NeurIPS 2022 Accept (2022)
Keywords: Cross Aggregation Transformer
Locality Complementary Module
Image Restoration

Abstract: Recently, Transformer architecture has been introduced into image restoration to replace convolution neural network (CNN) with surprising results. Considering the high computational complexity of Transformer with global attention, some methods use the local square window to limit the scope of self-attention. However, these methods lack direct interaction among different windows, which limits the establishment of long-range dependencies. To address the above issue, we propose a new image restoration model, Cross Aggregation Transformer (CAT). The core of our CAT is the Rectangle-Window Self-Attention (Rwin-SA), which utilizes horizontal and vertical rectangle window attention in different heads parallelly to expand the attention area and aggregate the features cross different windows. We also introduce the Axial-Shift operation for different window interactions. Furthermore, we propose the Locality Complementary Module to complement the self-attention mechanism, which incorporates the inductive bias of CNN (e.g., translation invariance and locality) into Transformer, enabling global-local coupling. Extensive experiments demonstrate that our CAT outperforms recent state-of-the-art methods on several image restoration applications. The code and models are available at https://github.com/zhengchen1999/CAT.

Cross Aggregation Transformer for Image Restoration
Zheng Chen', Yulun Zhang?, Jinjin Gu Yongbing Zhang", Linghe Kong', Xin Yuan 6 'Shanhhai Jiao Tong University, 'ETH Zürich, 'Shanghai Al Laborator,, 'The University of Sydney, 'Harbin Institute of Technology (Shenzhen), "Westlake University
Abstract
Recently, Transformer architecture has been introduced into image restoration to replace convolution neural network (CNN) with surprising results. Considering the high computational complexity of Transformer with global attention, some methods use the local square window limit the scope self-attention. However, these methods lack direct interaction among different windows, which limits the establishment of Long-range dependencies. To address the above issue, we propose a new image restoration model, Cross Aggregation Transformer (CAT). The core BeetangleWWindow of our CAT is the Rectangle- Window Self-Attention (Rwin-SA), which utilizes horizontal and vertical rectangle window attention in different heads parallelly to expand the attention area and aggregate the features cross different windows. We also introduce the Axial-Shift operation for different window interactions. Furthermore, we propose the Locality Complementary Module to complement the self-attention mechanism, which incorporates the inductive bias of CNN (e.g.. translation invariance and locality) into Transformer, enabling global-local coupling. Extensive experiments demonstrate that our CAT outperforms recent state-of-the- art methods on several image restoration applications. The code and models are available https: /// //gttttt.t/t////////////tt//bh com/ 2hengchen1999/CAT
Introduction
Image restoration is a long-term low-level vision problem, aimed at recovering high-quality (HQ) images from low-quality (I-Q) counterparts. Depending on the type of degradation, there many sub-problems, such as super-resolution (SR), image denoising, and JPEG compression artifact reduction. In recent decades, many researchers have proposed various theories and methods solve this problem, e.g., inverse filtering and wiener filtering. Meanwhile, with the increase of computing resources, neural networks have become the mainstream of image restoration, especially convolutional neural network (CNN) [10, 11, 50], which achieve impressive performance over traditional methods.
Recently, some researchers have been using Transformer proposed in natural language processing (NLP) to replace CNN and achieving excellent performance in multiple vision tasks, such as image classification [13], semantic segmentation [12], and object detection [41]. The core component of Transformer is the multi-head self-attention (MHSA), which can effectively model long-range dependencies, while CNN is more inclined extract local features. Although Transformer can effectively handle dependencies between tokens, its complexity is quadratic with the image resolution (H x W), i.e., (( )²), which limits its application to high-resolution image tasks (including image restoration). To apply Transformer in image restoration, some methods [5, 21, 44] propose to divide the image into smaller independent patches (e.g., 8x8) and perform attention separately or apply self-attention (SA) in the feature dimension instead of the spatial dimension, thus achieving linear complexity. However, the square window lacks wieeww interaction, leading the slow
CCorespondiig author: Linghe Kong,
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

increase of the receptive field. Moreover, the channel- attention mechanism may lose some spatial information. All these problems restrict the performance of the Transformer in image restoration.
To solve the above problems of Transformer application in image restoration tasks, we propose new image restoration Transformer model named Cross Aggregation Transformer (CAT). Our CAT utilizes local window self-attention to restrict the computational complexity and aggregates features cross different windows to expand the receptive field. The key part of our CAT is new self-attention mechanism, named Rectangle-Wing Window Self-Attention (Rwin-SA). Specifically, the Rwin-SA performs the attention operation in a separate rectangle window (shfsw) rather than the square (sh=sw) one, where sh and sw are window height and width. And we divide multiple heads into two groups and perform horizontal (sh<sw) and vertical (sh>sw) rectangle window self-attention in parallel. This method effectively expands the attention area and aggregates different features in horizontal and vertical directions while keeping the computational complexity unchanged. Moreover, inspired by CSwin [12], which performs cross-shaped window self-attention, we fix the length of one side of the rectangle window to be or W (image resolution). Then the window becomes a stripe along the axis, called axial rectangle window. It can generate more window interactions and obtain larger attention area than the regular rectangle window.
We also propose a new shift operation named Axial-Shift Operation, which is applied between consecutive Rwin-SA, to further increase the interaction of the different windows. Like our Rwin-SA, axial-shift is also divided into horizontal and vertical directions. which acts on the corresponding window respectively. Compared with the shift operation in Swin Transformer [23], our axial-shift explicitly realizes the interaction between horizontal-horizone or vertical-vertical windows, and implicitly enables the interaction between horizontal and vertical windows.
Furthermore, we propose Locality Complementary Module (LCM), convolution operation that it performed on the value (V) in the self-attention mechanism in parallel with the attention part. Compared with Transformer that mainly builds global long-range dependencies, CNN has the characteristies of translation invariance and locality, and can effectively capture 2D local structures of the image (e.g., corners and edges). The LCM can complements Rwin-SA with locality information, enabling the coupling of global (self-attention) and local (convolution) information.
Based on the above three designs, our CAT realizes the feature aggregation cross different windows, thus possessing long-range dependencies modeling capability. We apply our CAT to several classic image restoration tasks: image SR, JPEG compression artifacts reduction, and real image denoising. Extensive experiments demonstrate the superiority of our method. Our contributions are as follows:
We propose a new image restoration Transformer model named Cross Aggregation Trans- former (CAT). The CAT utilizes the window self-attention and aggregates the features cross different windows, thus achieving a large receptive field with linear complexity.
We propose a novel self- atffnattention mechanism, named Rwin-SA, using rectangle window with axial-shift operation. Moreover, we propose the locality complementary module to realize the coupling of global and local information.
We apply our CAT to classic image restoration tasks: image SR, JPEG compression artifact reduction, and real image denoising. Extensive experiments show that our proposed model achieves state-of-the of-the-art performance both quantitatively and visually.
2 Related Work
Image Restoration. CNNs have been achieving impressive performance over the traditional ap- proaches and becoming the mainstream method in the field of image restoration since Dong et al. proposed SRCNN [1i] and ARCNN [10] for image SR and JPEG compression artifact re- duction, respectively. Currently, many efforts have been devoted to building very deep CNNs to improve performance through multiple methods, e.g., residual learning strategies [15], dense con- nection [16], dropout [19], and UNet architecture [34]. Meanwhile, spatial and channel attention mechanisms [51, 31, 52, 20, 6] are utilized to make the network more focused on specific information in the feature space. However, most CNN-based methods are still hard to capture global dependencies.
Vision Transformer. Vision Transformer (ViT) achieves surprising results high-level vision tasks [13, 41]. Meanwhile, many efficient attention mechanisms have been proposed to improve the performance of ViT. Swin Transformer [23] uses local window attention and realizes the interaction between windows through shift operations. CSwin [12] proposes cross-shaper window attention, and

---

Residual Group (RG)
MLP
Reconstruction
LayerNorm
Conv 3x3
Conv 3x3
Conv 3x3
CATB
Rwin-SA
Ng
LayerNorm
(a) Architecture of CAT
(b) CATB
Figure (a) The architecture of our CAT. (b) Illustaation of CATB.
Twins [7] adopt combination of global and local attention. Inspired by the success of Transformer in high-level tasks, some works have been trying to apply Transformer to low-level vision tasks, such as IPT [5]. However, employing original ViT [13] for restoration tasks is hard since the complexity of full self-attention is quadratic to image size. Some methods have been proposed to solve this problem. SwinIR [21]. based on Swin Transformer, utilizes an 8x8 window to divide the image and executes self-attention in each window separately. It also uses the shift operation. Uformer [39] also applies 8x8 local window and introduces UNet architecture to capture much more global dependencies. Restormer [44] calculates cross-covariance across channel dimensions rather than spatial ones, making the complexity of SA linear with image resolution. However, the 8x8 square window limits the receptive field of the Transformer even with the shift operation. And the channel-wise attention mechanisms may lose spatial information. Furthermore, above methods do not effectively combine convolution with attention mechanisms, thus limiting model capabilities.
3 Method
To build an efficient Transformer model suitable for image restoration, we propose a rectangle- window self-attention (Rwin-SA) to replace vanilla multi-head self-attention mechanism in ViT [13] and form the cross aggregation Transformer block (CATB). Then we use RCAN [51] as the backbone and replace the residual channel attention block (RCAB) proposed in RCAN with our CATB to build the model cross aggregation Transformer (CAT), which is illustrated in Fig. 1. We first describe the overall structure of CAT and then introduce the three core designs of the CATB, including rectangle-window self-attention, axial-shift operation, and locality complementary module.
3.1 CAT Architecture
As shown in Fig. 1, following RCAN [51], our proposed CAT consists of three modules: shallow feature extraction, deep feature extraction, and reconstruction part. For low-quality input image ILQER H xW xC.,, we leverage one convolution layer as shallow feature extraction to get the low- level feature H xc where H, W, Cin: C are the image height, width, input channel, and the feature number, respectively. Then, the shallow feature is processed by the deep feature extraction module, which is composed of Nj residual groups (RG) [51] and one convolution layer, and extracted the deep feature (denoted as F XC). The convolution layer here is employed to aggregate previously extracted features from RG. Moreover, a residual connection is applied in the deep feature extraction part to make the training stable. We use CATB to replace RCAB [51] in RG, as we mentioned above. So, RG consists of N₂ cross aggregation Transformer blocks and convolution layer that can introduce the unique properties (translation invariance and locality) of CNN into the output of Transformer blocks. In addition, residual strategy is also applied in each RG.
Finally, can get the high-quality output image ERQ H XC..r through the reconstruction part, where Cout is the output dimension. And following SwinIR [21]. the composition of reconstruction module is different for different image restoration tasks. More specifically, for image SR, a sub-pixel convolution layer [36] is used to upsample the deep feature EDF to the same size of the high- resolution output and there is convolution layer before and after the upsampling module to aggregate the features. For JPEG compression artifact reduction, the reconstruction is just convolution layer adjust the channel dimension of the For from Cout. The low-quality input InQ is then added to the convolution output to produce a high-quality output IHQ This residual learning can accelerate network model convergence. Moreover, for real image denoising, following Restormer [44], we apply our proposed CATB to the UNet architecture [34] to construct CAT.
3

---

002
64/2
V-Shift
V-Rwin
Project
Concal
12/2
61/2
H-Shift
H-Rwin
(a) Rectangle-Window Self-Attention
(b) Axial-Shift Operation
Figure 2: (a) Illustration of rectangle- windww self-attention mechanism. The sh and the window height and width for H-Rwin and V-Rwin. For pixel (red dot), the attention area is the pink region of the middle feature map. (b) Illustration of axial-shift operation.
3.2 Cross Aggregation Transformer Block
The cross aggregation Transformer block (CATB) the key part of our CAT, shown in Fig. 1, which is more suitable for image restoration tasks than the previous Transformer structures [21, 44J. utilizes a new attention mechanism with three novel designs: rectangle-windov self-attention, axial-shift operation, and locality complementary module. We will describe these parts in detail below.
Rectangle-Winme Window Self-Attention. To alleviate the drawbacks previous methods [5, 21, 44], we propose new window attention mechanism shown in Fig. 2(a), called rectangle-window attention (Rwin-SA). Rwin-SA uses the rectangle window (sh=sw) rather than the square (sh=sw)) one, where sh and represent the height and width of the rectangle. Moreover, divide the rectangle window into the horizontal one (sh<sw, denoted as H-Rwin) and the vertical one (sh>sw, denoted as V-Rwin), and employ them to different attention heads in parallel.
window self-
Specifically, given the input X H we execute the attention operation M (the heads number) times in parallel. For each attention head, we split X into non-overlapping sh shw rectangle windows and denote the i-th rectangle window feature as (thxeexX) where i=1,... H shxsw"
Then the
self-attention of X; for the m-th head can be computed as
(Q, m K, m Vi = (X, K,W, ? X,W ",WW,, V),
2 (KK, myT
(1)
Y Attention(Q K, V = SoftMax
+ B)
Vd
where Y (shxx the attention feature X; in the m-th head, and Q." K2 V," "RR" represent the projection matrices of query, key, and value for m-th head. D=d= C is the channel dimension in each head and B is the dynamic relative position encoding [38]. Performing the attention H skxsw and reshaping and merging them the order division, we can get the attention feature n ER #xWx of X. In general, the above attention operation is the same for wiRwin and V-Rwin, where the sh and sw take different values.
operation on all X;
Assuming that the number of attention heads M even, we divide the heads equally into two parts and execute H-Rwin for the first part and V-Rwin for the second part in parallel. Finally, the outputs of two parts are concatenated along the channel dimension. The process is formulated as Rwin-SA(X) Conact(Y' y ,?....,YMM WWP.
where yl V the output of the head using H-Rwin, and # YM are from the V-Rwin. The WPERCXC represents the projection matrix for feature fusion. As shown in Fig. 3, through H-Rwin and V-Rwin, attention area without increasing computational complexity. Moreover, the rectangle window can capture different features in horizontal and vertical directions for each pixel, which is hard for square window to handle with comparable computation resource. This property is crucial for some datasets (e.g., Urban 100 [17]) that contain many directional and repetitive texture features.
aggregate features cross different windows and expand the

---

Furthermore, as shown in Fig. 3, in- spired by CSwin [12], we fix the length of one side of the rectangle to be the image resolution H or W (the other side length denoted as sl). Then the window becomes a stripe along the axis, called axial rectangle win- dow (denoted as axial-Rwin). Since the axial window changes as the im- age changes, the axial-Rwin is more flexible than the regular rectangle window, which side length is [sh, sw] or [sw, sh] (denoted as regular-Rwing The computational complexity of regular-Rwin and axial-Rwin are
Figure 3: Illustration of attention expansion, directional fea- tures aggregation, and axial-Rwin. H, W, and st mean height, width, and the other side length.
O (regular-Rwin) = HWC C' x (4C + 2sh x sw),
(3)
O(xxial-Rwin) HWC x (4C + sl H sl x W).
When H=W (square input), the complexity of regular-Rwin is O and axial-Rwin is O(H)). Al- though axial-Rwin is more complex than regular-Rwin, is still smaller than full-attention (O (O(H))),
and applicable for high-resolution images. And axial-Rwin has a larger attention area than that in regular- Rwin, which can capture more information, especially on the axial direction.
Axial-Shift Operation. To further expand the receptive field of Rwin-SA so that each pixel can aggregate more information, we propose a new shift operation called axial-shift, for R Rwin-SA, As shown in Fig. 2(b), axial-shift includes two shift operations, the horizontal one is called H-Shift and the vertical one is called V-Shift, corresponding to H-Rwin and V-Rwin, respectively. The axial-shit operation moves the window partition down and left by sh and pixels, where sh and sw are the window height and width of H-Rwin and V-Rwin. In the implementation, we cyclically shift the feature map to down-left direction. Then we execute H-Rwin and V-Rwin on corresponding shifted feature maps and use a masking mechanism to avoid false interactions between non-adjacent pixels.
The axial-shift operation can be regarded a more general shifted window operation in Swin Transformer [23]. When sh and sw in H- H-Rwin and V-Rwin are all the same, it degenerates into the swin shift operation. Compared with the swin shift operation, range of window interaction: (1) Rwwi-Rwin with H(V)-Rwin. Intuitively, the new window partition enables the interactive fusion of different rectangle window of the same type (horizontal vertical just just like the swin shift operation. (2) H-Rwin with V-Rwin. The Rwin-SA performs W-Rwin and V-Rwin simultaneously and fuses them through the projection in Eq. (2). For this reason, axial-shift can implicitly implement interaction between horizontal and vertical rectangle windows.
axial-shift enables a broader
Locality Complementary Module. Transformer can effectively capture global information and model long-range dependencies between pixels. How- ever, the inductive bias (translation invariance and locality) of CNN are still indispensable in image restoration tasks. It can aggregate local features and extract the basic structures of an image (e.g., corners and edges). In order to complement the Transformer with the locality and achieve global and local coupling, we propose add independent convolution operation when cal- culating self-attention, called locality complementary module. As shown in Fig. 4, different from the previous practice in Transformer [42, 8], we directly operate convolution on value V. The process is formulated as
Projection
Rwin-SA
LCM
QKI
V
Figure 4: Illustra-
tion of locality com- where y1 ,...... YMM and W are the same as Eq. (2). The VER#*WxC plementary module.
Rwin-SA(X) = (Conact(Y'. y ..... MM + Conv(V))
(4)
the value projected directly from X without window partition (V," is the partition of V in m-thh attention head defined in Eq. (I)), and Conv(.) represents the 3x3 depth-wise convolution.
This operation has two advantages over executing convolution sequentially or using convolution on X: (1) Using convolution as a parallel module allows Transformer block to adaptively choose to adopt attention or convolution operations, which is more ficxible than sequential execution; (2) From Eq. (1), we can find that self-attention can be regarded as a content-dependent and dynamic weight acting on V. The convolution operation is equivalent to a learnable and static weight. Therefore, the convolution operation on V is conducted in the same feature domain as the attention operation.
5

---

Network
PSNR
SSIM
FLOPs
Network
PSNR
SSIM
FLOPs
Netwerk
PSNR
SSIM
FLOPS
Sq. w/o slift
52.50
09325
28188
LCM
3291
09360
28188
C-R
0.936
282.7G
Sq. w/ shift
32.75
0.9347
281.8G
C-R w/ LCM
32.98
0.9361
282.7G
C-A-1
32.97
0.9353
323,3G
Rc. wio axinl
32.66
0.9334
281.8G
C-A LCM
LCM
33.01
0.9354
349.7G
C-A-2
33.11
0.9363
350.7G
Re. w/ axsiall
32.91
0.9360
281.8G
C.A W/CM
33.11
0,9363
350.7G
C-A-3
33.20
[0.3376
377.9G
(a) Beetangle-Window Self-Attention
(b) Locality Complementary Module
(c) Window Size Impact
Table I: Ablation study. (a) Sq.: square window (8x8) self-attention; Re.: regular rectangle window
(4x 16) self-attention; shift: swin shift operation; axial: axial-shift operation. (b) C-R: CAT-R (regular-
Rwin) model, [sw, sh] is [4, 16] (or 16, 4D; C-A: CAT-A (axial-Rwin) model, sl is [2, 2, 2, 4, 4, 4); LCM: locality complementary module. (c) The sl of C-A-1, C-A-2, and C-A-3 are [2, 2, 2, 2, 2, 2. [2, 2, 2, 4, 4, 4], and [4, 4, 4, 4,4, 4, 4), respectively.
Cross Aggregation Transformer Block. CATB is composed of Rwin-SA and MLP, which has two linear projection layers with GELU non-linearity between them. We formulate the block
X' (LLN + Xin.
(5)
X out MLP(LN(X')) + X',
where X,n and Xout are the input and output feature maps of CATB; LN(.) is the LayerNorm operation. And axial-shift operation is used on the interval between two consecutive cross aggregation Transformer blocks to enhance the interaction among different windows.
Experiments
4.1 Experimental Settings
Data and Evaluation. For image SR, we choose DIV2K [37] and Flickr2K [22] as the training data. And benchmark datasets for evaluation include Set5 [3], S [48], B100 [27], Urban 100 [17], and Mangal 09 [28] with three upscaling factors: x2, x3, and x4. The low-resolution images are generated by Bicubic downsampling. For JPEG compression artifact reduction, training set consists of DIV2K [37], Flickr2K [22]. BSD500 [2], and WED [26]. And we have two testing datasets: Classic5 [14] and LIVEI [35], with JPEG compression qualities of 10, 20, 30, and 40. For real image denoising, we train CAT on SIDD [1] dataset. And we have two testing datasets: SIDD validation set [1] and DND [33]. We use the metrics PSNR and SSIM [40] to evaluate the results.
Implementation Details. We set the residual group (RG) number as N₁=6 and the cross aggregation Transformer block (CATB) number as N₂= for each RG. The channel dimension, attention head number, and MLP expansion ratio for each CATB are set as 180, 6, and 4, respectively. For image SR, we build two versions of CAT, called CAT-R and CAT-A. CAT-R uses the regular rectangle window (regular-R Rwin), and CAT-A uses the axial rectangle window (oxial-Rwin). The regular- Rwin and the axial-Rwin are defined in Sec. 3.2. For CAT-R, we set [sh, as [4, 16] (or [16, 4)) for each CATB in each RG, and for CAT-A, we set al as [2, 2, 2, 4, 4, 4] for different residual groups. For JPEG compression artifact reduction, we also employ CAT-A with the same st setting in image SR. For real image denoising, following Restormer [44], we employ a 4-eeel encoder-decoder. The number of Transformer blocks is [4. 6, 6, 8), from level-1 to level-4. We set attention heads as [2, (2,2, 4, 8] and MLP expansion ratio as 4. We apply axial-Rwin here and set sl as 4 for all CATB.
Training Settings. For image SR, we train the model with batch size 32, where each input image is randomly cropped to 64 64 size, and the total training iterations are 500K. We adopt Adam optimizer [1 [18] with 3, =0.9 and 3290 to minimize the Lı loss following the work [51, 21J. The initial learning rate is set as 2 10-4 and reduced by half at the milestone [250K,4000 400KKK00KKK4555 For JPEG compression artifact reduction, the batch size and total iterations are 8 and 1600K, where the input image size is 128 x 128. Same as image SR, we adopt Adam optimizer to minimize weeiiiioiiiiii
Charbonnier Loss [4] with e=10- 3 and the learning rate is initialized to xx 10 which is halved at (200K, 1400K, For real image denoising, we use the progressive Iearning proposed by Restormer [44]. The total training iterations
300K. We adopt AdamW optimizer [25] with $,=0.9 and 32=0.99 to minimize the L1 loss. The initial learning rate is 3 and gradually reduced to 1 10-6 with the cosine annealing [24]. Furthermore, we randomly utilize rotation and flipping to augment the training data for image SR, JPEG compression artifact reduction, and real image denoising. We use PyTorch [32] to implement our models with 4 Tesla V00 GPUs.?
2 All the code and trained models are available https :ttp::///gttttt wttp////wgcccnnn99999/////

---

HQ
Bicubic
EDSR 122]
RCAN 1511
SAN 19]
Urban100: ing 024 (x4)
IGNN 1541
HAN 1311
CSNLN [30]
SwintR 1211
CAT (ours)
HQ
Bicubic
EDSR [22]
RCAN [5]]
SAN [9]
U0ban100: img 074 (x44)
IGNN [5=]]
HAN [31]
CSNLN [30]
SwinIR [21]
CAT (ours)
(11
HO
Bicubic
EDSR [22]]
RCAN [SL]
SAN [9]
1111
U0han100:
img. 099 (x44
IGNN [54]
HAN [31]
CSNI.N [30]
SwinTR [21]
CAT (ours)
Figure 5: Visual comparison about image SR (x4) in some challenging cases.
4.2 Ablation Study
For the ablation study, we use the dataset DIV2K [37] and Flickr2K [22] to train CAT on the image SR (x2) task, where the input image patch size 64 x 64 and the iterations are 150K. The testing dataset is 000 [17]. FLOPs are calculated on input image size 3 x 128x
Rectangle- Winddw Self-Attention. Table la demonstrates that gectangle-window self-attention is more effective than the square window attention (e.g., SwinIR [21]). With no increase in compu- tational complexity (e.g., FLOPs), the rectangle window can improve the performance from 32.50 dB to 32.66 26 dB compared to the square one. Moreover, when we adopt the axial-shift operation in Rwin-SA, PSNR can reach 32.91 dB. And the square window attention of the swin shift operation only obtains 32.75 dB. It can be found that our Rwin-SA can aggregate more information and is more suitable for image restoration than attention mechanisms Swin Transformer [23].
Locality Complementary Module. Table lb shows the impact of locality complementary module on CAT-R (regular-Rwin) and CAT-A (axial-Rwin). With the LCM, CAT-R and CAT-A can achieve 0.07 dB and 0.10 dB gains over the cases without LCM, respectively. These results show that the inductive bias of CNN can effectively improve the performance of Transformer in image restoration. And comparing the complexity, it can be found that LCM only increases FLOPs by 0.26 0.26%-0.32%) which will not affect the model efficiency too much. Furthermore, LCM is embedded into self-attention as a new module without changing the implementation of the attention part.
Window Size Impact. From above analyses, we know that our Rwin-SA is more suitable for image restoration tasks than the swin attention mechanism. We further investigate the impact of different window sizes. The results are provided in Table Ic. Comparing regular- Rwin (CAT-R) and axial-Rwin (CAT-A), we can find that the performance of axial-Rwin is worse than regular- -win (32.97 dB vs. 32.98 dB), when the side of axial-Rwin small (s/=2), despite its higher complexity (323.5G vs. 282.7G). It may be because the side of axial-Rwin (sl) is too narrow to capture enough information and may even introduce noise. Therefore, the result of axial-Rwin rises rapidly as increase. These results show that both side length and window size of Rwin are crucial for performance.
4.3 Image Super-Resolutie
We compare our two models CAT-R and CAT-A with state-of-the-art methods: EDSR [22], RCAN [51], SAN [9], IGNN [54], HAN [31], CSNLN [30], NLSA [29], IPT [5], and SwinIR [21]. Following the previous work [22, 21, 51], we use self-ensemble strategy in testing and mark the model with symbol "+". We use CAT-A for visual comparisons, abbreviated as CAT.
7

---

Method
Scale
SetS
Set14
B100
Urban100
MangaTUs
PSNK
SSIM
PSNK
SSIM
PSNR
SSIM
PSNK
SSIM
PSNK
SSIM
EDSR [2]]
X2
38.11
0.9602
33.92
0.9195
32.32
0.9013
32.93
0.9351
39.10
0,9773
[SIN
X2
38.27
0.9614
34.12
0.9216
32.41
0,9027
0.9384
39.44
0,9786
SAN [9]
x2
38.31
0.9620
34,07
0.9213
32.42
0,9028
33.10
0.9370
39.32
0.9792
IGNN [54]
X2
38.24
0.9613
34.07
0.9217
32.41
0.9025
33.23
0,9383
39.35
0.9786
HAN [3]]
x2
38.27
0.9614
16,16
0.9217
32.41
0.9027
33.35
0.9385
39.46
0.9785
CSNLN [30]
X2
38.28
0.9616
34,12
0.9223
32,40
0,9024
33.25
0.9386
39.37
0,9785
NLSA 129]
X2
38.34
0.9618
34,08
32.43
0.9027
33.42
0.9394
39.59
0.9789
IPT [5]
X2
38.37
34.43
32.48
33.76
SwinIR [2]]
X-
38.42
0.9623
0.9250
32.53
0.9041
33.81
0.9427
39,92
0,9797
CAT-R (ours)
X2
38.48
0.9625
34,53
0.9251
32.56
0,9045
34.08
0.9443
40.09
0,9804
CAT-A (ours)
X2
38.51
0.9626
34.78
0.9265
32.59
0.9047
34,26
0,9440
40.10
0.9805
CC (ours)
X2"
338.52
069627
"44.5
0:9257
32.5%
0699047
34.19
'0.9450
10.18
0,9805
CAT-A+ (ours)
X2
38.55
0.9628
34.81
0.9267
32,60
0.9048
34.34
0.9445
40.18
0,9806
EDSR [22]
x3
34.65
0.9280
30.52
0.8462
29.25
0,8093
0.8653
34.17
0,9476
RCAN[S [5]]
x3
34,74
0.9299
30.65
0.8482
29.32
0.8111
29.09
0,8702
34.44
0,9499
SAN [9]
x3
34.75
0.9300
30.59
0.8476
29.33
0.8112
28.93
0,8671
34.30
0.9494
IGNN [54]
X3
34.72
0.9298
30.66
0.8484
29.31
0.8105
29.03
0.869%
34.39
0.9496
HAN [31]
X3
34.75
0.9299
30.67
0.8483
29.32
0.8110
29.10
0,8705
34.48
0.9500
CSNL. [30]
X3
34.74
0.9300
30.66
0.8482
29.33
0.8105
29.13
0.8712
34.45
0,9502
NLSA [29]
X3
34,85
0.9306
30,70
0.8485
29. 34
0.8117
29.25
0,8726
34.57
0,9508
IPT [5]
X3
34.81
30.85
29.38
29.49
SwinIR [21]
X3
34.97
0.9318
30,93
0,8534
29.46
0.8145
29.75
0,8826
35.12
0,9537
CAT-R (ours)
X3
34,99
0.9320
31,00
0.8539
29.
0.8154
29.91
0,8848
35.29
0.9542
CAT-A (ours)
X3
35.06
0.9326
31,04
0.8538
29.52
0.8160
30.12
0,8862
35,38
0,9546
CAT=== (ours)
X3"
"33.097
0.33324
31.006
0:83444
29.52"
00.8150
"30000
"0.8864
"35'44"
'0.35488"
CAT-A+ (ours)
x
35.10
0.9327
31.09
0.8545
29. 39.55
0.8166
30.21
0.8872
35.48
0,9550
EDSR |22
X4
32.46
0.8968
28,80
0.7876
27.71
U.7420
26.64
0,8033
31.02
0,9148
RCAN [51]
x4
32.63
0.9002
28.87
0.7889
27.77
0.7436
26.82
0, 0,8087
31.22
0.9173
SAN 19]
x4
32.64
0.9003
28.92
0.7888
27.78
0.7436
26.79
0.8068
31.18
0,9169
IGNN [54]
X4
32.57
0.8998
28.85
0.7891
27.77
0.7434
26.84
0,8090
31.28
0.9182
HAN [31]
X44
32.64
0.90022
28.90
0.7890
27.80
0.7442
26.85
0,5094
31.42
0,9177
CSNLN [30]
X4
32.68
0,9004
28.95
0,7888
27. 80
0.7439
27.22
0.8168
31 14
0,9201
NLSA [29]
X4
32.59
0.9000
0.7891
27.78
0.7444
26.96
0,8109
31.27
0,9184
[5]
X4
32.64
29.01
27.82
27.26
SwinIR [21]
X4
32.92
0.9044
29.09
0.7950
27.92
0.7489
27.45
0,8254
32.03
0.9260
CAT-R (ours)
X4
32,89
0.9044
29.13
0.7955
27.95
0,7500
27.62
0.8242
32.16
0,9269
CAT-A (ours)
X4
33.08
0.9052
29.18
0.7960
27.99
0.7510
27.89
0,8339
32.39
0.9285
"CAF-R (ours)
"4""
392.99
0.9699
19911
0070663
21.99%
0.3300
10.8310"
"32.35
0039280"
CAT-A+ (ours)
X4
33.14
0.9059
29.23
0.7968
28.01
0
27.99
0,8356
32.52
0,9293
Table 2: Quantitative comparison (PSNR/SSIM) with state-of-the- methods for image SR. Best and second best results are colored with red and blue.
RNAN [52]
RDN
[53]
DRUNeI [49]
SwinIR [2]]
CAT (ours)
CAT+ (ours)
Dataset
PSNR
SSIM
PSNK
SSIM
PSNR
SSIM
PSNK
SSIM
PSNR
SSIM
PSNR
SSIM
10
29.63
0.8239
29.67
0.8247
29.79
0.8278
29.86
0.8287
29.89
0.8295
29.92
0.8299
20
32.03
0.8877
32.07
0.8882
32.17
0.8899
32.25
0.8909
32.30
0.8913
32.32
0.8915
LIVEI
30
33.45
0.9149
33.51
0.9153
33.59
0,9166
33.69
0.9174
33.73
0.9177
33.75
0.9179
40
34.47
0.9299
34.51
0.9302
34.58
0,9312
34.67
0.9317
34.72
0.9320
34.74
0.9322
10
29.96
0.8178
30.00
0.8188
30,16
0.8234
30.27
0,8249
30.26
0.8250
30.30
0.8257
20
32
0.8693
32.15
0.8699
32.39
0.8734
32.52
0,8748
32.57
0.8754
32.60
0.8756
ClassicS
30
33.38
0.8924
33.43
0.8930
33.59
0.8949
33.73
0.8961
33.77
0,8964
33,80
0.8966
40
34.27
0.9061
34.27
0.9061
34.41
0.9075
34.52
0.9082
34.58
0.9087
34.60
0,9088
Table 3: Quantitative comparison (PSNR/SSIM) with state-of the methods for JPEG compression artifact reduction. Best and second best results are colored with red and blue.
Quantitative Results. Table 2 shows PSNR/SSIM comparisons for x2, x3, and x4 image SR. As we can see, our CAT-R (regular- Rwin) and CAT-A (axial-Rwin) significantly outperform other methods on all datasets with all scale factors. Among these five datasets, CAT-R and CAT-A have the most noticeable improvement in Urban100 Compared with the recent Transformer model (e.g., SwinlR), for CAT-R. the PSNR gain reaches 0.27 dB for scale factor 2, and for CAT-A, the PSNR increases by 0.45 dB. It shows that our CAT can capture more global information than previous CNN-based and Transformer-base a large number of repetitive texture structures. Moreover, compared with CAT-R, CAT-A yields 0.18-0.27 dB improvements on Urban100. All these results indicate the effectiveness of our method.
models, which is very effective for images in Urban with
Visual Results. We show visual comparisons (x4) in Fig. 5. We can observe that most compared methods can hardly recover accurate textures and suffer from blurring artifacts in some challenging cases. In contrast, our CAT can reconstruct more high-frequency structural details and alleviate the blurring artifacts. For example, in img 092, our CAT can recover correct and sharp textures, while
8

---

LIVEI: buildings
HQ
JPL3 (g=10)
RNAN |521
RDN [53]
SwintR [211
CAT' (ours)
LIVEI: camivaldolls
HQ
JPEG (q=10)
RNAN [52]
RDN [53]
SwinIR [21]
(ours)
Figure 6: Visual comparison about JPEG compression artifacts reduction (g=10).
Dataset
DANet+ [43]
Cycle[SP [45]
MIRNet [46]
MPRNe1 [77]
Uonner [39]
Restormer [44]
CAT (ouns)
CAT+ (ours)
9.15
31.79
15.74
50.88
26.11
25.77
25.77
SIDD*
PSNR
39.47
39.52
39.72
39.71
39.89
40.02
40.01
40.05
SSIM
0,9570
0.9571
0.9586
0,9586
0.9594
0.96033
0.9600
0,9602
DND
SSIM
0.9545
0.9564
0.9563
0.9540
0.9554
0.9564
0.9561
0,9563
Table 4:
Quantitative comparison (PSNR/SSIM) for real image denoising. Best and second best results are colored with red and blue. * We re-test the SIDD with all official pre-trained models.
most compared methods fail to recover image details. These results demonstrate that our CAT has more robust representational ability to recover structural contents and textural details. It also shows that, compared with other models, our CAT is more effective in modeling long-range dependencies, which are crucial for reconstructing high-quality images.
4.4 JPEG Compression Artifact Reduction
We compare our CAT with state-of-the-art image compression artifact removal methods: RNAN [52], RDN [53], DRUNet [49], and SwinIR [21]. Here, only focus on the restoration of Y channel (in YCbCr space). We also use self-ensemble strategy and mark the model with a symbol "+". Quantitative and visual comparisons are shown in Table 3 and Fig. 6, respectively.
Quantitative Results. Table 3 shows quantitative comparisons with four JPEG quality settings: 10, 20, 30, and 40. Our CAT+ performs better than other compared methods on LIVEI and Classic5 with all JPEG qualities. Even without self-ensemble, our RCAN also outperforms other compared methods, except for PSNR on Classic5 (q=10). Especially when 4040, CAT achieves 0.05 and 0.06 dB improvements over the Transformer-based model SwinlR. Even in a very low compression quality (e.g., 10), CAT achieves 29.89 dB in LIVEI, higher than previous methods.
Visual Results. We show visual comparisons very low image quality (g=10) Fig. 6. For previous methods, RNAN [52], RDN [53]. and SwinIR [21], blocking artifacts are hardly to be totally removed and some structures are over-smoothed some challenging cases. In contrast, our CAT can recover more details and remove blocking artifacts, resulting in sharper edges and more explicit textures. These visual comparisons further demonstrate the effectiveness of our proposed CAT.
4.5 Real Image Denoising
We further conduct real image denoising experiments. We compare our CAT with state-of-the- methods: DANet+ [43], CycleISP [45], MIRNet [46], MPRNet [47], Uformer [39], and Restormer [44]. We use self-ensemble [22] strategy and mark the model with symbol "4"
Quantitative Results. Table 4 shows quantitative comparisons for real image denoising. We re-test the SIDD results with all official pre-trained models on previous state-of-the-art methods, Our CAT performs better than other compared methods on SIDD and DND, except Restormer [44]. Meanwhile, our method has comparable performance to Restormer with fewer parameters. Our CAT achieves 0.02 dB improvements over Restormer DND. All these show the superiority of our method.

---

Method
EDSR [22]
RCAN [51]
HAN
CSNLN [30]
SwinlR [2]]
CAT-R (ours)
CAT-A (ours)
CAT-R-2 (ours)
PSNR (dB)
26.64
26.82
26.85
27.22
27.45
27.62
27,89
27.59
FLOPs (G)
823.3
261.0
269.1
84.155.2
215.3
292.7
360.7
216.3
Parameters (M)
43.09
15.59
16.07
6.57
11.90
16.60
16.60
11.93
Table 5: Model complexity comparisons (x4). Output size is 3x512x512 to calculate FLOPs.
4.6
Model Size Analyses
Table 5 shows the comparison of performance, computational complexity (e.g., FL.OPs,, and parame- ter numbers on image SR. FLOPs are measured when output size is set to 3x512x512 and PSNR values are tested on Urban 100 (x4). Although our CAT achieves excellent performance, it has less computational complexity and parameters than EDSR [22]. And the computational complexity of our CAT is much lower than CSNLN [30]. Compared to other CNN-based and Transformer-baset methods, our CAT still has comparable complexity and model size. To demonstrate the effectiveness of our method, we provide another variant of CAT (CAT-R-2) with similar computational complexity and parameters to SwinlR. Our CAT-R-2 still outperforms SwinIR and other state-of-the-an methods. More details about C-T-R-2 are provided in the supplementary material.
5 Conclusion
In this paper, we propose new Transformer model named cross aggregation Transformer (CAT) for image restoration with the rectangle-wing window self-attention (Rwin-SA) mechanism as the key part. Specifically, Rwin-SA performs H-Rwin and V-Rwin attention parallelly in different heads. This mechanism can aggregate the features cross different windows and increase the receptive field without additional computational cost. Meanwhile, we propose axial-shift operation according to the property of rectangle- window to further fuse different windows, which complements Rwin- SA. Furthermore, the locality complementary module (LCM) introduces locality into self-attention, adding the inductive bias of CNN to Transformer. Through the usage of LCM, Rwin-SA can realize the coupling of global and local information, which is more conducive image restoration. Extensive experiments on image SR, JPEG compression artifact reduction, and real image denoising demonstrate that our proposed CAT outperforms current state-of-the-art methods.
Acknowledgments. This work is partly supported by NSFC (62141220, 61972253, U1908212, 72061 127001), the Program for Professor of Special Appointment (Fastern Scholar) at Shanghai Institutions Higher Learning. Xin Yuan acknowledges the support of NSFC (62271414), Westlake Foundation (2211B1501-2) and the Research Center for Industries of the Future (RCIF) at Westlake University.
References
[1] Ahdelrahman Abdelhamed, Stephen Lin, and Michael S Brown. A high-quality denoising dataset for smartphone cameras. In CVPR, 2018.
[2] Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. TPAMI, 2010. 6
[3] Marco Bevilacqua. Aline Rouny, Christine Guillemot, and Marie Line Alberi-Moel. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In BMVC, 2012. 6
[4] Pierre Charbonnier, Laure Blanc-Feraud, Gilles Aubert, and Michel Barlaud. Two deterministic half- quadratic regularization algorithms for computed imaging. In ICIP, 1994. 6
[5] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In CVPR, 2021. 1,3, 7,8
[6] Haoyu Chen, Jinjin Gu. and Zhi Zhang. Attention in attention network for image super-resolution. arXiv preprint arXiv:2104.09.0 094977 2021. 2
[7] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. In NeurIPS, 2021.
[8] Xiangxiang Chu. Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Conditional positional encodings for vision transformers. arXiv preprint 200iv: 10882, 2021. 5
[9] Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang, Sccond-order attention network for single image super-resolude resouuiiso In CVPR, 2019. 7,8
10

---

[10] Chao Dong, Yubin Deng, Chen Change Loy, and Xiaoou Tang. Compression artifacts reduction by deep convolutional network. In ICCV, 2013. 1,2
[11] Chao Dong Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning deep convolutional network for image super-resolution. ECCV, 2014. 1,2
[12] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In CVPR, 2022. 2,
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021. 2,
[14] Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Pointwise shape-adaptive det for high-quality denoising and deblocking grayscale and color images. TIP, 2007. 6
[15] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 2
[16] Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten, Densely connected convolu- tional networks. In CVPR, 2017. 2
[17] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In CVPR, 2015. 4, 6,
[18] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015, 6
[19] Xiangtao Kong, Xina Liu, Jinjin Gu, Yu Qiao, and Chao Dong. Reflash dropout in image super-resolution. In CVPR, 2022. 2
[20] Zheyuan Li, Yinggi Liu, Xiangyu Chen, Haoming Cai, Jinjin Gu, Yu Qiao, and Chao Dong. Blueprint separable residual network for efficient image super-resolution. CVPR, 2022.
[21] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Lue Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In ICCVW, 2021. 1, 3, 4, 6, 7, 8, 9, 10
[22] Bee Lim, Sanghyun Son, Heewon Kim, Scungjun Nah, and Kyoung Mu Lec. Enhanced deep residual networks for single image super-resolution. In CVPRW, 2017. 6, 7, 8, 9, 10
[23] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021. 5,
[24] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2017. 6
[25] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 6
[26] Kede Ma, Zhengfang Duanmu, Qingbo Wu, Zhou Wang, Hongwei Yong, Hongliang Li, and Lei Zhang. Waterloo exploration database: New challenges for image quality assessment models. TIP, 2016. 6
[27] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms measuring ecological statistics. In ICCV, 2001. 6
[28] Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-based manga retrieval using mangal 109 datasct. Multimedia Tools and Applications, 2017.
[29] Yigun Mei, Yuchen Fan, and Yuqian Zhou. Image super-resolution with non-local sparse attention. In CVPR, 2021. 7,
[30] Yiqun Mei. Yuchen Fan, Yuqian Zhou, Lichao Huang, Thomas S Huang, and Humphrey Shi. Image super-resolution with cross-scale non-local attention and exhaustive self-exemplars mining. CVPR.
2020. 7, 8, 10
[31] Ben Niu, Weilei Wen, Wengi Ren. Xiangde Zhang, Lianping Yang, Shuzhen Wang, Kaihao Zhang. Xiaochun Cao, and Haifeng Shen. Single image super resolutoo via a holistic attention network. In ECCV, 2020. 2, 7, 8, 10

---

[32] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Viti,o Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017. 6
[33] Tobias Plotz and Stefan Roth. Benchmarking denoising algorithms with real photographs. In CVPR, 2017.
[34] Olaf Ronneberger. Philipp Fischer, and Thomas Brox, U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015. 2,3
[35] Hamid Sheikh, Muhammad F Sabir, and Alan C Bovik. A statistical evaluation of recent full referencc image quality assessment algorithms. TIP, 2006.
[36] Wenzhe Shi, Jose Caballero, Ferene Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR, 2016.
[37] Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, Lei Zhang, Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, Kyoung Mu Lee, et al. Ntire 2017 challenge single image super-resolution: Methods and results. In CVPRW, 2017. 6,
[38] Wenxiao Wang, Lu Yao, Long Chen, Binbin Lin, Deng Cai, Xiaofci He, and Wei Liu. Crossforme:: A versatile vision transformer hinging cross-scale attention. In ICLR. 2022. 4
[39] Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer:
A general u-shaped transformer for image restoration. In CVPR, 2022. 3,9
[40] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from visibility to structural similarity. TIP, 2004.
[41] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal self-attention for local-global interactions in vision transformers. In NeurlPS, 2021. 1,2
[42] Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. In CVPR, 2021. 5
[43] Zongsheng Yue. Qian Zhao, Lei Zhang, and Deyu Meng. Dual adversarial network: Toward real-world noise removal and noise generation. In ECCV, 2020. 9
[44] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. CVPR, 2022. 1, 3, 4, 6,
[45] Syed Wagas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hsuan Yang, and Ling Shao. Cycleisp: Real image restoration via improved data synthesis. In CVPR, 2020. 9
[46] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsee Hing-Hsuan Yang, and Ling Shao. Learning enriched features for real image restoration and enhancement. In ECCV, 2020. 9
[47] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hssan Yang, and Ling Shao. Multi-stage progressive image restoration. In CVPR, 2021. 9
[48] Roman Zeyde, Michael Flad, and Matan Protter. On single image scale-up using sparsc-represennttiins. In Proc. 7th Int. Conf. Curves Surf.. 2010.
[49] Kai Zhang, Yawei Li, Wangmeng Zuo. Lei Zhang, Lue Van Gool, and Radu Timofte. Plug-and-play image restoration with deep denoiser prior. TPAMI, 2021. 8,9
[50] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep enn for image denoising. TIP, 2017. 1
[51] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super-resolution using very deep residual channel attention networks. In ECCV, 2018. 2, 3, 6, 7,8, 8, 10
[52] Yulun Zhang, Kunpeng Li, Kai Li, Bineng Zhong, and Yun Fu. Residual non-local attention networks for image restoration. In ICLR, 2019. 2, 6, 8,
[53] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image restoration. TPAMI, 2020. 8, 9
[54] Shangchen Zhou, Jiawei Zhang, Wangmeng Zuo, and Chen Change Loy. Cross-scate internal graph neural network for image super-resolution. In NewrIPS, 2020. 7, 8
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
(b) Did you describe the limitations your work? [Yes] the supplementary
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See the supplemen-
tary
(d) Have you read the ethics review guidelines and ensured that your paper conforms them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes] See See 3.2
(b) Did you include complete proofs of all theoretical results? [Yes] See Sec 3.2
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [Yes]
(h) Did you specify all the training details (e.g., data splits, hyperparamcters, how they were chosen)? [Yes] See Implementation Details in Sec. 4.1 and Training Settings in See. 4.1
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [No]
(d) Did you include total amount of compute and the type resources used (e.g.. type GPUs, internal cluster, or cloud provider)? [Yes] See Training Settings in Sec. 4.1
4. If you using existing assets (e.g.. code, data, models) curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're us- ing/curating? [Yes] The used data
publicly available.
(c) Did you discuss whether the data you are using/curating contains personally identifiable infor- mation or offensive content? [Yes] The used data doesn't 't contain any personally identifiable information or offensive content.
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "nvOE5jXoIR6", "reviewer": "Reviewer_LKyp", "summary": "The authors proposed cross aggregation Transformer (CAT) for image restoration, where the long-range dependencies are well considered. The proposed CAT uses the rectangle window self-attention (Rwin-SA) with axial-shift and aggregates the features cross different windows. Plus, they proposed a locality complementary module (LCM) to capture both local and global information. The main results on classic image restoration applications, like image SR and JPEG artifact reduction, show the superior performance of CAT quantitatively and visually.", "questions": "(1) There are image restoration (i.e., image SR and JPEG artifact reduction) applications are conducted. The pipeline in Figure 1 mainly takes image SR as an example. Can the authors give more details about the pipeline for JPEG artifact reduction? Or the only difference is that the upscaling module is removed? Please clarify this.\n\n(2) Is the axial-shift in CAT exactly the same as the shift operation in SwinIR? If not, please tell the differences.\n\n(3) Can the method be applied to other image restoration tasks well? Like image denoising and deblurring. If so, how is the performance of CAT? Will CAT still achieves SOTA performance?", "limitations": "In supplementary file, the authors have discussed the limitations and potential negative societal impact. Overall, I agree with the views of the authors.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n(1) Firstly, the paper writing, organization, method, illustration figures (e.g., the pipeline in Figure 1), results are pretty good, which impress me a lot! I believe this paper is carefully prepared. \n\n(2) The idea of CAT is novel and a promising step to investigate more powerful Transformer based methods for high-quality image restoration.\n\n(3) The authors proposed cross aggregation Transformer block as the basic unit to form the image restoration networks. In the block, there three key parts: rectangle-window self-attention, axial-shift operation, and locality complementary module. The effect of each part has been demonstrated with ablation study experiments.  \n\n(4) The main results with other leading methods are also extensive. The authors conducted experiments on image SR and JPEG artifact reduction. According to the quantitative (i.e., Tables 2 and 3) and visual (i.e., Figures 5 and 6) comparisons, we can further see the superior performance of the CAT.\n\n(5) The authors provided more variant models of the proposed one and achieved further improvements, also indicating the promising representation ability of CAT.\n\n(6) In supplementary file, the authors provided convergence analyses on CAT-A-CAT-R, CAR-R-2, and SwinIR using multiple testing sets. These results show that various CAT models performs better than the basic SwinIR. \n\nWeaknesses:\n\n(1) For the model size analyses in Table 4, CAT-R and CAT-A perform better than other compared ones, but their parameter numbers are not the smallest or still higher than SwinIR. In supplementary file, the authors gave a more reasonable comparison (see Table 2), which should be included in the main paper. \n\n(2) For JPEG artifact reduction, the quantitative results do not show too much improvement of CAT over SwinIR. It would be better if the authors can give more analyses.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Z27T25YoQdA", "reviewer": "Reviewer_mWKh", "summary": "A new Transformer based method, named as cross aggregation Transformer (CAT), was proposed for high-quality image restoration. The authors proposed three main components: rectangle-window self-attention, axial-shift operation, and locality complementary module (LCM). Their effects are demonstrated by extensive ablation study results. The main comparisons show the superior of the proposed CAT.", "questions": "- For JPEG artifact reduction, the performance improvements of CAT over SwinIR are not very obvious, although the visual differences are obvious. Please give some explanations.\n\n- The authors claim that the method is for image restoration. Then, how would the CAT perform for other image restoration tasks? Will it still achieve SOTA performance?\n\n- Do all the training and testing details between SwinIR and CAT for image SR and JPEG artifact reduction keep the same? If there has some differences, please clarify them.\n\n- Please tell the differences between CAT and CSwin [10].", "limitations": "In supplementary file, the limitations and potential negative societal impact of the work have been briefly discussed.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n- The authors proposed to aggregate deep features across different windows with expanded sensing area, which is reasonable and effective for image restoration. \n\n- A new shift operation named axial-shift operation was proposed to further increase the interaction of different windows.\n\n- A Conv operation named locality complementary module (LCM) was proposed to make good use of its translation invariance and locality, enabling the model to capture both global and local information.\n\n- Experiments are extensive. The ablation study shows the effect of each proposed new component. The main quantitative and visual comparisons with recent methods, like SwinIR, demonstrate that the proposed CAT is a promising Transformer based method.  \n\n- The authors provided source code and pretrained models for results reproduction. These materials could help other researchers for further investigations and improvements. For example, it would be easy to apply for this method for image denoising, derain, dehazing et al. with the provided code. \n \n- The authors also show some visualization results using LAM and compare with SwinIR. According to the visualization results (Fig. 2 in supp.), the proposed CAT captures a larger receptive field. This observation is consistent with the claim in ‘Abstract’ about the long-range dependencies estimation. \n\n- The main paper and supplementary file are well prepared. The writing is pretty good and easy to read. The motivation is clear and reasonable. The paper is carefully organized. \n\nWeakness\n\n- In Table 4, the parameter number and FLOPs of CAT are larger than those in SwinIR. A better and more reasonable way is to keep similar model size and FLOPs and then show comparisons.\n\n- The detailed differences between axial-shift and the shift option in Swin Transformer are not very clear to me.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VzjEgWVanXS", "reviewer": "Reviewer_RMwD", "summary": "This paper proposes an image restoration model, named Cross Aggregation Transformer (CAT). Specifically, this paper designs the Rectangle-Window Self-Attention (Rwin-SA), which utilizes horizontal and vertical rectangle window multi-head attention parallelly. The proposed Locality Complementary Module incorporates the inductive bias of CNN into Transformer to complement the self-attention mechanism.  Ablation experiments prove that the LCM improves model performance with negligible computational cost. A series experiments on different datasets demonstrate that the CAT outperforms recent state-of-art methods on two image restoration tasks: image SR and JPEG compression artifacts reduction.\n", "questions": "1. In Swin Transformer [19], Hu et al. proposed a shifted window operation, whose shifting directions include horizontal and vertical, and the shifting range of each direction is [0, k], where k is the size of the window. In this paper, author proposes a axial-shift (the horizontal one is called H-shift and the vertical one is called V-shift) operation for Rwin-SA, whose shifting range is set as sh/2 and sw/2, where sh and sw are the size of the rectangle window. The difference between shifted window operation [19] and axial-shift operation is caused by the shape of the window, so I personally think axial-shift and shifted window operation [19] are the same. Please explain in more detail the difference between V-shift and H-shift, and the difference between axial-shift and shifted window operation [19].\n\n2.this paper proposes a Rectangle-Window Self-Attention (Rwin-SA), which utilizes horizontal and vertical rectangle window (H-Rwin and V-Rwin) attention in different heads parallelly. Furthermore, authors also proposes an axial rectangle window (axis-Rwin), in which the length of one side is fixed as the image resolution H or W. However, the axial rectangle window division method has been proposed in CSWin [10], so this part should be marked in reference [10].  \n\n3. More recent methods should be compared to demonstrate the superiority of the proposed method, such as Uformer and Restormer.\n4. In related work, the discussion of the existing tranformer methods is insufficient, and the author should analyze CSWin and Uformer.\n5. More experiments on color JPEG image restoration should be conducted.", "limitations": "The contribution and novelty of this paper are not convincing enough. More recent methods should be compared to demonstrate the superiority of the proposed method.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper proposes an image restoration model, named Cross Aggregation Transformer (CAT). Specifically, this paper designs the Rectangle-Window Self-Attention (Rwin-SA), which utilizes horizontal and vertical rectangle window multi-head attention parallelly.\n2. The proposed Locality Complementary Module incorporates the inductive bias of CNN into Transformer to complement the self-attention mechanism.  Ablation experiments prove that the LCM improves model performance with negligible computational cost.\n\nWeakness:\n1. the contribution and novelty of this paper are not convincing enough. The proposed axial rectangle window has been designed in other work. Besides, the proposed axial-shift operation is the same as shifted window operation in Swin transformer.\n2. In this paper, the discussion of the existing transformer methods is insufficient.\n3. More recent methods should be compared to demonstrate the superiority of the proposed method.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Lg5AqAG0VoI", "reviewer": "Reviewer_J9TR", "summary": "The authors propose a new image restoration model, cross aggregation Transformer (CAT). The key idea is the rectangle-window self-attention, which uses horizontal and vertical rectangle window attention in different heads parallelly. The axial-shift operation is further introduced to different window interactions. They further propose a locality complementary module to incorporate some useful properties of CNN into Transformer. The ablation study and main results support the contributions well. ", "questions": "1. Please discuss the differences between the key ideas of the work (e.g., axial-Rwin) and CSWin.\n2. The authors said in the ‘Conclusion’ part that the proposed CAT could also be used for other image restoration tasks. Did the authors try those tasks and/or get some preliminary results?\n", "limitations": "No, the authors did not discuss too much about the limitations or potential negative societal impact. It would be better if the authors can discuss some limitations.\n", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The proposed method cross aggregation Transformer (CAT) is simple yet efficient. It has been a good Transformer based method for high-quality image restoration applications.  \n2. All proposed components are stated clearly and logically, like rectangle-window self-attention, axial-shift operation, and locality complementary module. \n3. The ablation study results are extensive and demonstrate the effects of each proposed component.\n4. The main comparisons with recent leading methods (e.g., SwinIR) further show the superior of the proposed one. According to Table 2, we can see the best and second-best results are almost achieved by the proposed method CAT. The visual differences between the proposed CAT and other methods are very obvious, and further show the effectiveness of the proposed CAT. Similar observations happen for image JPEG compression artifact reduction.\n5. The overall writing and organization are pretty good. \n6. The authors also provide code and pre-trained models for reproduction, which further shows the solidness of the work.\n\nWeaknesses\n1. In Figure 3, the meanings of H, W, sl should be given in the caption.\n2. In Table 4, the authors provide model size, FLOPs, and PSNR comparisons. The proposed CAT obtains the highest performance. However, the proposed CAT has a larger model size and FLOPs than the related work SwinIR. It would be much better if the authors provide comparisons with SwinIR using similar parameter number and FLOPs. \n3. Some details should be made more clear. For example, for image SR, the authors provide two versions of CAT: CAT-R and CAT-A. For JPEG compression artifact reduction, the authors only show CAT. It is not very clear about its model size and FLOPs. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept