Accurate Image Restoration with Attention Retractable Transformer
Authors: Jiale Zhang
Yulun Zhang
Jinjin Gu
Yongbing Zhang
Linghe Kong
Xin Yuan
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Image restoration
Dense and sparse attention

Abstract: Recently, Transformer-based image restoration networks have achieved promising improvements over convolutional neural networks due to parameter-independent global interactions. To lower computational cost, existing works generally limit self-attention computation within non-overlapping windows. However, each group of tokens are always from a dense area of the image. This is considered as a dense attention strategy since the interactions of tokens are restrained in dense regions. Obviously, this strategy could result in restricted receptive fields. To address this issue, we propose \textbf{A}ttention \textbf{R}etractable \textbf{T}ransformer (ART) for image restoration, which presents both dense and sparse attention modules in the network. The sparse attention module allows tokens from sparse areas to interact and thus provides a wider receptive field. Furthermore, the alternating application of dense and sparse attention modules greatly enhances representation ability of Transformer while providing retractable attention on the input image.We conduct extensive experiments on image super-resolution, denoising, and JPEG compression artifact reduction tasks. Experimental results validate that our proposed ART outperforms state-of-the-art methods on various benchmark datasets both quantitatively and visually. We also provide code and models at~\url{https://github.com/gladzhang/ART}.

Published as a conference paper at ICLR 2023
ACCURATE IMAGE RESTORATION WITH ATTENTION RETRACTABLE TRANSFORMER
Jiale Zhang', Yulun Zhang²; Jinjin Gu Yongbing Zhang', Linghe Kong',, Xin Yuan Shanghai Jiao Tong University. "ETH Zürich, 'Shanghai Al Laboratory, "hhe University of Sydney, HHarbin Institute of Technology (Shenzhen), "Westlake University
ABSTRACT
Recently, Transformer-bast based image restoration networks have achieved promising improvements over convolutional neural networks due to parameter-independert global interactions. To lower computational cost, existing works generally limit self-attention computation within non-overlapping windows. However, each group tokens are always from dense area the image. This is considered as a dense attention strategy since the interactions of tokens are restrained in dense regions. Obviously, this strategy could result in restricted receptive fields. To address this issue, we propose Attention Retractable Transformer (ART) for image restoration, which presents both dense and sparse attention modules in the network. The sparse attention module allows tokens from sparse areas to interact and thus provides a wider receptive field. Furthermore, the alternating application of dense and sparse attention modules greatly enhances representation ability of Transformer while pro- viding retractable attention on the input image. We conduct extensive experiments on image nuper-resolution, denoising, and JPEG compression artifact reduction tasks. Experimental results validate that our proposed ART outperforms state- of- the-art methods on various benchmark datasets both quantitatively and visually. We also provide code and models at https: / / thub. com/ g a aaazanggg/aa ART.
INTRODUCTION
Image restoration aims to recover the high-quality image from its low-quality counterpart and includes a series of computer vision applications, such as image super-resolution (SR) and denoising. an ill-posed inverse problem since there are huge amount of candidates for any original input. Recently, deep convolutional neural networks (CNNs) have been investigated to design various models Kim et al. (2016b); Zhang et al. (2020; 2021b) for image restoration. SRCNN Dong et al. (2014) firstly introduced deep CNN into image SR. Then several representative works utilized residual learning (e.g., EDSR Lim al. (2017)) and attention mechanism (e.g., RCAN Zhang al. (2018h)) to train very deep network image SR. Meanwhile, a number of methods were also proposed for image denoising such as DnCNN Zhang al. (2017a), RPCNN Xia & Chakrabarti (2020), and BRDNet Tian et al. (2020), These CNN-based networks have achieved remarkable performance.
However, due to parameter-dependee tions of convolutions, CNN has limited ability to model long-range dependencies. To overcome this limitation, recent works have begun to introduce self-attention into computer vision systems Hu et al. (2019); Ramachandran et al. (2019); Wang et al. (2020); Zhao et al. (2020). Since Transformer has been shown to achieve ofate-of-the-arr performance in natural language processing Vaswani et al. (2017) and high-level vision tasks Dosovitskiy et al. 1);21); Touvron et al. (2021); Wang et al. (202 I); Zheng et al. (2021): Chu et al. (2021), researchers have been investigating Transformer-basee image restoration networks Yang et al. (2020); Wang et al. (2022b). Chen et al. proposed pre-trained image processing Transformer named IPT Chen et al. (2021a). Liang et al. proposed a strong baseline model named SwinIR Liang et al. (2021) based on Swin Transformer Liu et al. (2021) for image restoration. Zamir et al. also proposed an efficient Transformer model using U-net structure named Restormer Zamir et al. (2022) and achieved state-of-ther -hrtt--rtttttttoe results on several image restoration tasks. In contrast, higher performance can be achieved when using Transformer.
receptive field scaling and content-independent local interac-
"Corresponding authors: Yulun Zhang, yulun 100@ gmail.com; Linghe Kong, linghe.kog@6 @jnn@sju..dd.cn

---

Published as a conference paper at ICLR 2023
Despite showing outstanding performance, isting Transformer backbones for image restora- tion still suffer from serious defects. As know, SwinIR Liang et al. (2021) takes advan- tage of shifted window scheme to limit self- attention computation within non-overlapping windows. On the other hand, IPT Chen et al. (2021a) directly splits features into PxP patches to shrink original feature map pa times, treating each patch as a token. In short, these methods compute self-attention with shorter to- ken sequences and the tokens in each group are always from a dense area of the image. It is considered as dense attention strategy, which obviously causes a restricted receptive field. To address this issue, the sparse attention strategy is employed. We extract each group of tokens from sparse area of the image provide interactions like previous studies (e.g., OG-Transformer Yu al. (2021), Max ViT Tu al. (2022b), Cross- Former Wang et al. (2022a)), but different from them. Our proposed sparse attention module focuses on equal-scale features. Besides, We pay more attention to pixel-level information than semantic-level information. Since the sparse attention has not been well proposed to solve the problems in low-level vision fields, our proposed method can bridge this gap.
Sparre
VS.
do SwinlTt: Dimse-Attetionn with Shilled Wirdowe
Figure 1: (a) Dense attention and sparse attention strategies our ART. (b) Dense attention strategy with shifted window of SwinIR.
We further propose Attention Retractable Transformer named ART for image restoration. Following RCAN Zhang et al. (2018b) and SwinlR Liang et al. (2021), reserve the residual in residual structure Zhang ct al. (2018b) for model architecture. Based on joint dense and sparse attention strategies, we design two types of self-attention blocks. We utilize fixed non-overlapping local windows to obtain tokens for the first block named dense attention block (DAB) and sparse grids to obtain tokens for the second block named sparse attention block (SAB). To better understand the difference between our work and SwinIR, we show a visual comparison in Fig. 1. As we can see, the image is divided into four groups and tokens in each group interact with each other. Visibly, the token in our sparse attention block can learn relationships from farther tokens while the one in dense attention block of SwinlR cannot. At the same computational cost, the sparse attention block has stronger ability to compensate for the lack of global information. We consider our dense and sparse attention blocks as successive ones and apply them to extract deep feature. practice, the alternating application of DAB and SAB can provide retractable attention for the model to capture both local and global receptive field. Our main contributions can be summarized as follows:
We propose the sparse attention to compensate the defect of mainly using dense attention in existing Transformer-based image restoration networks. The interactions among tokens extracted from a sparse area of an image can bring a wider receptive field to the module.
We further propose Attention Retractable Transformer (ART) for image restoration. Our ART offers two types of self-attentio blocks to obtain retractable attention on the input fea- ture. With the alternating application of dense and sparse attention blocks, the Transformer model can capture local and global receptive field simultaneously.
We employ ART train an effective Transformer-basd experiments on three image restoration tasks: image super-resolution, denoising, and JPEG compression artifact reduction. Our method achieves sfate-of sate-of--hhe-ar the--art performance.
network. We conduct extensive
RELATED WORK
Image Restoration. With the rapid development of CNN, numerous works based on CNN have been proposed to solve image restoration problems Anwar & Barnes (2020); Dudhane et al. (2022); Zamir et al. (2020; 2021), 1); Li et al. (2022): Chen et al. (2021b) and achieved superior performance over conventional restoration approaches Timofte et al. (2013); Michaeli & Irani (2013); He et al. (2010). The pioneering work SRCNN Dong et al. (2014) was firstly proposed for image SR. DnCNN Zhang al. (2017a) was a representative image denoising method. Following these works, various model
2

---

Published as a conference paper at ICLR 2023
Input
Residual Group
OUEDUE
MLP
MLP
XM
LN
LN
Conv Sxa
3x3 3x3
Conv
Resiore
D-MSA
S- MSA
EXR
LN
LN
(a) The architecture of ART for image restoration
(b) Two successive attention blocks
Figure 2: (a) The architecture of our proposed ART for image restoration. (b) The inner structure of two successive attention blocks DAB and SAB with two attention modules D-MSA and S-MSA.
designs and improving techniques have been introduced into the basic CNN frameworks. These techniques include but not limit to the residual structure Kim et al. (2016a); Zhang et al. (2021a), skip connection Zhang ct al. (2018b; 2020), dropout Kong ct al. (2022), and attention mechanism Dai et al. (2019); Niu et al. (2020). Recently, due to the limited ability of CNN to model long-range dependencies, researchers have started to replace convolution operator with pure self-attention module for image restoration Yang al. (2020); Liang et al. (2021); Zamir et al. (2022); Chen et al. (2021a).
Vision Transformer: Transformer has been achieving impressive performance in machine translation tasks Vaswani et al. (2017). Due to the content- derreneed global receptive field, it has been introduced to improve computer vision systems in recent years. Dosovitskiy et al. Dosovitskiy et al. (2021) proposed ViT and introduced Transformer into image recognition by projecting large image patches into token sequences. Tu et al. proposed Max ViT Tu et al. (2022b) as an efficient Vision Transformer while introducing multi-axis attention. Wang et al. proposed CrossFormer Wang et al. (2022a) to build the interactions among long and short distance tokens. Yu et al. proposed GG- GG-Traasformer Yu et al. (2021), which performed self-attention on the adaptively-dilan by the strong ability to learn long-range dependencies, researches have also investigated the usage of Transformer for low-level vision tasks Yang et al. (2020); Chen al. (2021a); Liang et al. (2021); Zamir et al. (2022); Wang et al. (2022b). However, existing works still suffer from restricted receptive fields due to mainly using dense attention strategy. Very recently, Tu et al. proposed a MLP- based network named MAXIM Tu et al. (2022a) to introduce dilated spatial communications into image processing. It further demonstrates that the sparse interactions of visual elements are important for solving low-level problems. In our proposed method, we use dense and sparse attention strategies to build network, which can capture wider global interactions. As the sparse attention has not been well proposed solve the low-level vision problems, our proposed method can bridge this gap.
partitions of the input. Inspired
3
PROPOSED METHOD
3.1 OVERALL ARCHITECTURE
The overall architecture of our ART is shown in Fig. 2. Following RCAN Zhang et al. (2018b), ART employs residual in residual structure to construct deep feature extraction module. Given degraded image LLQ E DxCCn (H, D. and Cin are the height, width, and input channels of the input), ART firstly applies 3x3 convolutional layer (Conv) to obtain shallow feature FD E IR 1 DxC where C is the dimension size of the new feature embedding. Next, the shallow feature is normalized and fed into the residual groups, which consist of core Transformer attention blocks. The deep feature is extracted and then passes through another 3x3 Conv to get further feature embeddings F1. Then we element-wist sum to obtain the final feature map FR Fo + F1. Finally, we employ the restoration module to generate the high-quality image IHQ from the feature map Fnz-
Residual Group. We No successive residual groups extract the deep feature. Each residual group consists of Ng pairs of attention blocks. We design two successive attention blocks shown in Fig. 2(b). The input feature _ı-1 passes through layer normalization (LN) and multi-hear self- attention (MSA). After adding the shortcut, the output x't is fed into the multi-layer perception (MLP). 27 is the final output at the 1-th block. The process is formulated
z't = (ISA(L((r1))) 1)) + I1-1,
x7 MLP(LN D) + z't.
Lastly, we also apply a 3x3 convolutional layer to refine the feature embeddings. As shown in Fig 2(a), a residual connection is employed to obtain the final output in each residual group module.
3

---

Published as a conference paper at ICLR 2023
1=2
I=2
W=4
Split
Split
(a) Dense Attention
(b) Spurse Attention
Figure 3: (a) Dense attention strategy. Tokens of each group from a dense area of the image. (b) Sparse attention strategy. Tokens of each group are from a sparse area of the image.
The restoration module is applied as the last stage of the framework to obtain the reconstructed image. As we know, image restoration tasks can be divided into two categories according to the usage of upsampling. For image super-resolution, convolutional layer Shi et al. (2016) to upsample final feature map FR- Next, we use a convolutional layer to get the final reconstructed image LиQ- The whole process is formulated Ilq = Conv Conv(Upsample (Upsample(FR))
Restoration Module.
take advantage of the sub-pixo
For tasks without upsampling, such as image denoising, we directly use a convolutional layer to reconstruct the high-quality image. Besides, we add the original image to the last output of restoration module for better performance. We formulate the whole process as
IHQ= = Conv + JLQ.
(3)
Loss Function. We optimize our ART with two types of loss functions. There are various well- studied loss functions, such as L2 loss Dong al. (2016); Sajjadi al. (2017); Tai et al. (2017), L1 loss Lai et al. (2017); Zhang et al. (2020), and Charbonnier loss Charbonnier et al. (1994). Same with previous works Zhang et al. (2018b); Liang et al. (2021), we utilize I, loss for image super-resolution (SR) and Charbonnier loss for image denoising and compression artifact reduction. For image SR. the goal of training ART is to minimize the L1 loss function, which is formulated as
(4) where TQ is the output of ART and Ig is the ground-truth image. For image denoising and JPEG compression artifact reduction, we utilize Charbonnier loss with super-parameter as 10-3, which is
L = / - Ialli,
= Vll - lal +
(5)
3.2 ATTENTION RETRACTABLE TRANSFORMER
We elaborate the details about our proposed two types of self-attention blocks in this section. As plotted in Fig. 2(b), the interactions of tokens concentrated on the multi-head self-attention module (MSA). We formulate the calculation process in MSA as
QKT
MSA(X) Softmax
VC V
(6)
where Q, K, V E NxC are respectively the query, key, and value from the linear projecting of input X NxC N the length of token sequence, and C is the dimension size of each token. Here we assume that the number of heads is 1 to transfer MSA singe-head self-attention for simplification.
Multi-head Self Attention. Given an image with size DD, vision Transformer firstly splits the raw image into numerous patches. These patches projected by convolutions with stride size P. The new projected feature map X x is prepared with h = P and P. Common MSA uses all the tokens extracted from the whole feature map and sends them to self-attention module to learn relationships between each other. It will suffer from high computational cost, which is
((MSA) 4hwC²² + 2(hw))²C.
(7)
To lower the computational cost, existing works generally utilize non-overlapping windows to obtain shorter token sequences. However, they mainly consider the tokens from dense area of an image. Different from them, we propose the retractable attention strategies, which provide interactions of tokens from not only dense areas but also sparse areas of an image to obtain wider receptive field.
Dense Attention. As shown in Fig. 3(a), dense attention allows each token to interact with a smaller number of tokens from the neighborhood position of non-overlapping W x W window. All tokens
4

---

Published as a conference paper at ICLR 2023
Methods
Solving problems
Structure
Interval of extracted tokens
Representation of tokens
Using long long-distance
residual connection
GG-Iranstoemer al. (2021)
Hige-level
Pyramid
Changed
Semanti-level Somantic-level
No
MaxViT To ct al. (20226)
High-lovel
Pyramid
Changed
No
CrossFormrr Wang al. (2022k)
High-level
Pyramid
Changed
Semantic-level
No
ART (Ouns)
Low-level
Iostropic
Unchamged
Pixel Jevel
Yes
Table I: Comparison to related works. The differences between our ART with other works.
are split into several groups and each group has W W tokens. We apply these groups to compute self-attention for x w. times and the computational cost of new module named D-MSA is
R(D-MSA) = C4 + +2 x
w W W
= 4hwC2 + hwC.
(8)
Sparse Attention. Meanwhile, as shown in Fig. 3(b), we propose sparse attention to allow each token to interact with a smaller number of tokens, which are from sparse positions with interval size I. After that, the updates of all tokens are also split into several groups and each group has 4 z tokens. We further utilize these groups to compute self-attention for I Ix/ times. We name the new multi-head self- module as S-MSA and the corresponding computational cost is
((S-MSA) -47 h x C2
0² + 2 7
h w h W ())))) x I I 4h + #I harC.
(9)
< By contrast, our proposed D-MSA and S-MSA modules have lower computational cost since W hw and hs m < lew. After computing all groups, the outputs are further merged to form original-size T feature map. In practice, we apply these two attention strategies to design two types of self-attention blocks named as dense attention block (DAB) and sparse attention block (SAB) as plotted in Fig. 2.
Successive Attention Blocks. We propose the alternating application of these two blocks. As the local interactions have higher priority, we fix the order of DAB in front of SAB. Besides, we provide the rongdditance residual connection between each three pairs of blocks. We show the effectiveness of this joint application with residual connection in the supplementary material.
Attention Retractable Transformer. We demonstrate that the application of these two blocks enables our model to capture local and global receptive field simultaneously. We treat the succes- sive attention blocks as a whole and get a new type of Transformer named Attention Retractable Transformer, which can provide interactions for both local dense tokens and global sparse tokens.
3.3 DIFFERENCES TO RELATED WORKS
We summarize the differences between our proposed approach, ART with the closely related works in Tab. We conclude them as three points. (1) Different tasks. GG- Transforme Yu al. (2021), ViX Tu et al. (2022b) and CrossFormer Wang et al. (2022a) are proposed to solve high-level vision problems. Our ART is the only one to employ the sparse attention in low-level vision fields. (2) Different designs of sparse attention. In the part of attention, GG-Transformer utilizes the adaptively-dilee dilated partitions, Max ViT utilizes the fixed-size grid attention and CrossFormer utilizes the cross-scale long-distance attention. As the layers get deeper, the interval of tokens from sparse attention becomes smaller and the channels of tokens become larger. Therefore, each token learns more semantic-level information. In contrast, the interval and the channel dimension of tokens in our ART keep unchanged and each token represents the accurate pixel-level information. (3) Different model structures. Different from these works using Pyramid model structure, our proposed ART enjoys an Isotropic structure. Besides, we provide the long- distance residual connection between several Transformer encoders, which enables the feature of deep layers to reserve more low-frequency information from shallow layers. More discussion can be found in the supplementary material.
3.4 IMPLEMENTATION DETAILS
Some details about how to apply our ART to construct image restoration model are introduced here. Firstly, the residual group number, DAB number, and SAB number in each group are set as 6, 3, and 3. Secondly, all the convolutional layers are equipped with 3x3 kernel, 1-length stride, and 1-length padding, so the height and width of feature map remain unchanged. In practice, we treat X patch as a token. Besides, we set the channel dimension as 180 for most layers except for the shallow feature extraction and the image reconstruction process. Thirdly, the window size in DAB is set as and the interval size in SAB is adjustable according to different tasks, which is discussed in Sec. 4.2. Lastly, to adjust the division of windows and sparse grids, we use padding and mask strategies to the input feature map of self-attention so that the number of division is always an integer.
5

---

Published as a conference paper at ICLR 2023
132
DAB and SAB Choices
Impact of Interval Size
24.26
Comparanon of Varient Models
13.0
34.30
34.00
12.8
13.75
18.
SAB
DAB+SAB
12.9
Small
Medium
25.55
13.90
ART ART-S SWii
312
DAB
22.08
Large
12.50
256
190
260 s60
Bandion 00
Boration (N)
Itrrtion (1)
Figure 4: Left: PSNR (dB) comparison of our ART using all dense attention block (DAB), using all sparse attention block (SAB), and using alternating DAB and SAB. Middle: PSNR (dB) comparison of our ART using large interval size in sparse attention block which (8,8, 8, 8, 8, 8) for six residual groups, using medium interval size which is (8, 8, 6, 6, 4,4), 4), and using small interval size which is (4, 4, 4, 4, 4, 4). Right: PSNR (dB) comparison of SwinlR, ART-S, and ART.
4
EXPERIMENTAL RESULTS
4.1 EXPERIMENTAL SETTINGS
Data and Evaluation. We conduct experiments three image restoration tasks, including image SR, denoising, and JPEG Compression Artifact Reduction (CAR). For image SR, following previous works Zhang et al. (2018b); Haris et al. (2018), we use DIV2K Timofte et al. (2017) and Flickr2K Lim et al. (2017) as training data, Set5 Bevilacqua et al. (2012), Set14 Zeyde et al. (2010), B B100 Martin et al. (2001), Urban Huang et al. (2015), and Mangal 109 Matsui al. (2017) as test data. For image denoising and JPEG CAR, same as SwinlR Liang et al. (2021), we use training data: DIV2K, Flickr2K, BSD500 Arbelaez et al. (2010), and WED Ma et al. (2016). We use BSD68 Martin et al. (2001), Kodak24 Franzen (1999), McMaster Zhang et al. (201 1). and Urban 100 as test data of image denoising. Classic5 Foi et al. (2007) and LIVEI Sheikh et al. (2006) are test data of JPEG CAR. Note that we crop large-size input image into 200x200 partitions with overlapping pixels during inference. Following Lim et al. (2017), we adopt the self-ensemble strategy to further improve the performance of our ART and name it as ART+, We evaluate experimental results with PSNR and SSIM Wang et al. (2004) values on Y channel of images transformed to YCbCr space.
Training Settings. Data augmentation is performed on the training data through horizontal flip and random rotation of 90º, 180", and 2 Besides, we crop the original images into x64 patches as the basic training inputs for image SR, 128 128 patches for image denoising, and 126x 126 patches for JPEG CAR. We resize the training batch to 32 for image SR, and 8 for image denoising and JPEG CAR in order to make a fair comparison. We choose ADAM Kingma optimize our ART model with 31 = 0.9, 82 = 0.999, and zero weight decay. The initial learning rate set as 2x10-4 and is reduced by half as the training iteration reaches a certain number. Taking image SR as example, we train ART for total 500k iterations and adjust learning rate to half when training iterations reach 250k, 400k, 450k, and 475k, where 1K means one thousand. Our ART is implemented on PyTorch Paszke et al. (2017) with 4 NVIDIA RTX8000 GPUs.
Ba (2015) to
4.2 ABLATION STUDY
For ablation experiments, we train our models for image super-resolution ( (x2) based on DIV2K and Flicke2K datasets. The results are evaluated on Urban benchmark dataset.
Design Choices for DAB and SAB. We demonstrate the necessity for simultaneous usage of dense attention block (DAB) and sparse attention block (SAB) by conducting ablation study. We set three different experiment conditions, which are using 6 DABs, 6 SABs, and 3 pairs of alternating DAB and SAB. We keep the rest of experiment environment the same and train all models within 100k iterations. The experimental results are shown in Fig. 4(Left). As we can see, only using DAB or SAB suffers from poor performance, because they lack either global receptive field or local receptive field. On the other hand, the structure of SAB following DAB brings higher performance. It validates that both local contextual interactions and global sparse interactions are important for improving strong representation ability of Transformer by obtaining retractable attention the input feature.
Impact of Interval Size. The interval size in sparse attention block has a vital impact on the performance of our ART. In fact, if the interval size is set as 1, it will be transferred to full attention. Generally, a smaller interval means wider receptive fields but higher computational cost. We compare the experimental results under different interval settings in Fig. 4(Middle). As we can see, smaller
6

---

Published as a conference paper at ICLR 2023
Melhel
Stale
PSNR
Sec5
SSIM
PSNR
Setld
SSIM
FSNK
B100
SSIM
PSNR
UrbanT00
SSIM
PSNR
MangwlOS
SSIM
EDSR Lim et al. (2017)
X2
38.11
0.5002
33.92
0.9195
09011
32.93
0:9351
39.10
0.9723
RCAN Zhang et all (2018))
x2
38.3 27
0.9614
34.12
0.9216
32.41
0.9027
35.34
0,9384
39.44
0.9786
SAN Daiet al. 2l.19)
x2
38.31
0.9620
54.07
0.9213
32.42
0.0088
33.10
0.9570
39.32
0.9792
SR (2019)
X2
38.11
0,9609
29,08
BAN Niu all (202g)
X2
38.27
0.9614
34.16
0.9217
32.41
0.9027
35.35
0.9385
39.46
0.9785
IGNN IENN al. (1.222))
X2
18.24
0.9611
$4.07
0.9217
32.41
05025
33.23
0.9383
39.35
0.9786
CSNLN et. (2020)
X2
38.26
0.9616
34.12
0.9223
32.40
0.9024
33.25
0.9386
39.37
0.9785
RFANet ei (2020)
0.9615
0.9220
NLSA Met al. (2821)
X2
38.
0.9618
0.9231
0.0027
33.42
0.9794
39.59
0.9789
IPT Chen al. (202
X2
38.37
NIA
34.43
N/A
32,48
NIA
33.76
N/A
N/A
N/A
SwinTR Linng et al. (2021)
X2
38.42
0.9623
34,46
0.9250
32.53
0.9041
33.81
0.9427
79.92
0.9797
ART-S (ours)
X2
38.
0.9625
0.9258
40.11
0.9804
ART (ours)
X2
38.56
0626
34.39
0.9267
32,58
0,9048
34.30 30
0.9452
40.24
0.9808
ART- (ours)
X2
38.59
0.9630
34.68
0,9269
32.60
0.0050
34.41
0.9457
40.33
0
EDSK Lim et al. (2017)
X33
34,65
0.9230
30.52
0,8482
29.25
0.809
25.30
0.M633
34,17
0.9476
RCAN et
0.9299
34.44
SAN Daiettl (2019)
X3
34.25
0.9300
30.59
0.4466
29.33
0.8122
28.93
0.8671
34.30
0.494
SRFaNT2 (2019)
x3
34.70
0.9292
50.51
0.8461
29.24
0,5084
28.73
0.864]
34.18
0.9481
HAN Nia al. (2020)
X3
34.25
0.9299
50.67
0.8483
29.32
0.81D
29 10
0.8705
34.48
0,9500
IGNN Zhou (2020)
34.72
0.2298
0.8484
0.869%
34.39
0.94%
CSNLN at, (2020)
X3
34.24
0.900
30.66
0.8482
29.33
0.800
13
0.8712
34.45
0,9502
RFANe: ct (202))
x3
34.79
0.300
50.67
0.8487
29.34
0,8115
29.15
0.8720
34.59
0.9506
NLSA ct.l (2021)
34,85
0.9306
30.70
0,8485
29,34
0.8117
29.25
0.8726
34,57
0L9508
(2021s)
34.81
NIA
SwintR Llang et al. (2021)
X3
34.97
09318
50.23
0,8544
0.8145
29.75
0.8826
35.12
0,9537
ART:S (ours)
X3
34.98
0.3188
3094
0,8530
29.45
0.8146
29.86
0,8830
35.22
0.9599
ART (ours)
x3
35.07
9325
31.00
0.8541
29.51
0.815
30.00
0.8871
35.30
0,9548
35.11
30.22
Lim (2017)
X4
32.46
0.3988
0.7876
27.21
31.02
0
RCAN Zhang et al. (20188)
X-4
32.65
0.5022
28.87
0,7889
22.77
0.7436
26.82
0.8057
31.22
0.173
SAN Dai al. (2.019) (2019)
X44
32.64
0.9003
28.92
0.7888
27.78
0.7436
26.79
0.805S
31.18
0.9199
SRFBN (2001)
32.47
0.8983
28.51
0.7868
27.72
0.7409
26.50
0.8015
31,15
0,9160
HAN et (2020)
X4
050022
28.00
0.7890
27,80
0.7442
0.8034
31,42
0.1777
JGNN 2hou at. (2020)
X-4
32,57
0.8998
25.35
0.7891
27.77
0.7434
26.84
0.80+0
31.28
099182
CSNLN Mei et al. (2020)
X4
32,68
0.9064
25.05
0,7888
27.50
0.7439
27.22
0.8168
31,45
0.9281
KHANe: al. (2120)
32,66
0.5064
0.7894
0,7442
26.91
31.41
0.9187
NLSA Mei (2021)
X4
32.59
0,5000
28.57
0.7891
27.78
0.7444
26.96
0,8109
31,27
0.9184
IPT Chen at.l. (20216)
X4
32.64
N/A
29.01
N/A
27.82
N/A
27.26
N/A
N/A
N/A
SwiniR Liang et al. (2021)
X4
32.92
0.5044
29.09
0,7950
27.5
0.7489
27.45
0.8254
32.05
QL5260
ART'S (ours)
32,86
0.9029
29.09
0.7942
27.91
0.7489
27.54
32.13
0.9263
ART (ours)
X4
33.04
0.9051
29.16
0.7958
27.97
0.7510
27.77
0,8321
32.91
0,9283
AKT' (ours)
x4
33.07
0.5055
24:20
0,7964
27.99
0.7513
27.89
0.8389
32,45
0U9291
Table 2: PSNR (dB)/SSI comparisons for image super-resolution on five benchmark datasets. We color best and second best results in red and blue.
Method
EDSR
RCAN
SRFBN
HAN
CSNLN
SwiTR
ART-S (ours)
ART (ours)
Params (M)
43.09
15.59
3.63
16.07
7.16
11.90
11.87
16.55
Mull Addls (G)
1,286
407
103,640
336
392
PSNR on Urban (dB)
26.64
26.82
26.60
26.85
27.22
27:45
27.34
27.77
PSNR on Mangal09 (dB)
31.02
31.22
31,15
31.42
31.43
32.03
32.13
32.31
Table 3: Model size comparisons (x4 SR). Output size is 3 3x640x660
640 for Mult-Adds calculation.
intervals bring more performance gains. To keep the balance between accuracy and complexity, we set the interval size of 6 residual groups as (4, 4, 4, 4, 4,4, 4) for image SR, (16, 16, 12, 12, 8, 8) for image denoising, and (18, 18, 13, 13, 7,7) for JPEG CAR in the following comparative experiments.
Comparison of Variant Models. We provide new version of our model for fair comparisons and name it ART-S. Different from ART, the MLP ratio in ART-S is set to 2 (4 in ART) and the interval size is set 8. We demonstrate that ART-S has comparable model size with SwinlR. We provide the PSNR comparison results in Fig. 4(ight,. As we can see, our ART- achieves better performance than SwinIR. More comparative results can be found in following experiment parts.
4.3 IMAGE SOPEERRRSOOUTION
We provide comparisons of our proposed ART with representative image SR methods, including CNN-based networks: EDSR Lim et al. (2017), RCAN Zhang et al. (2018b), SAN Dai et al. (2019), SRFBN Li et al. (2019), HAN Niu et al. (2020), IGNN Zhou et al. (2020), CSNLN Mei et al. (2020), RFANet Liu et al. (2020), NLSA Mei et al. (2021), and Transformer-based networks: IPT Chen ct al. 1202 and SwinlR Liang et al. (2021). Note that IPT is pre-trained model, which is trained on ImageNet benchmark dataset. All the results are provided by publicly available code and data. Quantitative and visual comparisons are provided in Tab. 2 and Fig. 5.
Quantitative Comparisons. We present PSNR/SSIM comparison results for x2, x3, and x4 image SR in Tab. 2. As we see, our ART achieves the best PSNR/SSIM performance all five benchmark datasets. Using self-ensemble,, ART+ gains even better results. Compared with existing Mfte-oo-the-an method SwinlR, our ART obtains better gains across all scale factors, indicating that
7

---

Published as a conference paper at ICLR 2023
HQQ/ PSNR (UR)
Bicubic/ 15.31
RCAN 16.36
SRFAN / 18.26
SANY 18.26
Udbanto: imp 092 (X4))
IGNN 18.51
CSNLN/ 18.69
REANe 18.09
SwinlR
ART 19.56
PSNK
Bicubic 18:28
RCAN
SKEWN 19.55
SAN 19.66
Urbun100: 10g (X-4))8))
BGNN 19:70
CSNEN/ 1982
RFANK / 19.72
SwintR 20.00
AKT/ 20.10
Figure 5: Visual comparison with challenging examples on image super-resolution (x4),
proposed joint dense and sparse attention blocks enable Transformer stronger representation ability. Despite showing better performance than CNN-based networks, another Transformer -based network IPT is not good as It validated that our proposed ART becomes new promising Transformer-base network for image SR.
Retractable Dense Attention. We fur- ther show typical visual comparison with SwinlR in Fig. 6. As SwinlR mainly uti- lizes dense attention strategy, it restores wrong texture structures under the influ- ence of close patches with mainly vertical lines. However, our ART can reconstruct the right texture, thanks to the wider recep- tive field provided by sparse attention strategy. Visibly, the patch is able to interact with farther patches with similar horizontal lines so that it can be reconstructed clearly. This comparison demonstrates the advantage retractable attention and its strong ability to restore high-quality outputs.
SwintR
Attentonn
Sparse
Figure 6: Visual comparison (x4) of SwinIR and Ours.
Model Size Comparisons. Table 3 provides comparisons of parameters number and Mult-Adds of different networks, which include existing state-of-the methods. We calculate the Mult-Adds assuming that the output size is 3 x40x 640 under x4 image SR. Compared with previous CNN- based networks, our ART has comparable parameter number and Mult-Adds but achieves high that ART-S has less parameters and Mult-Adds than most
performance. Besides, of the compared methods. The model size of ART-S is similar with SwinIR. However, ART-S still achieves better performance gains than all compared methods except our ART. It indicates that our method is able to achieve promising performance at an acceptable computational and memory cost.
Visual Comparisons. We also provide some challenging examples for visual comparison (x4) in Fig. 5. We can see that our ART is able to alleviate heavy blurring artifacts while restoring detailed edges and textures. Compared with other methods, ART obtains visually pleasing results by recovering more high-frequency details. It indicates that ART preforms better for image SR.
4.4 IMAGE DENOISING
We show color image denoising results compare ART with representative methods in Tab. 4. These methods are CBM3D Dabov et al. (2007), IRCNN Zhang et al. (2017b), FFDNet Zhang al. (2018a), DnCNN Zhang al. (2017a), RNAN Zhang et al. (2019), RDN Zhang al. (2020), IPT Chen et al. (2021a), DRUNet Zhang et al. (2021a), P3AN Hu et al. (2021). )), SwinIR Liang et al. (2021), and Restormer Zamir al. (2022). Following most recent works, we set the noise level to 15, 25, and 50. We also shows visual comparisons of challenging examples in Fig. 7.
Quantitative Comparisons. Table shows PSNR results of color image denoising. As we can see, our ART achieves the highest performance across all compared methods three datasets except Kodak24. Even better results are obtained by ART+ using self-ensemble. Particularly, it obtains better gains than the state-of-ther the-hhhttttt model Restormer Zamir et al. (2022) by up 0.25dB Urban100 Restormer also has restricted receptive fields and thus has difficulty in some challenging cases. In conclusion, these comparisons indicate that our ART also has strong ability in image denoising.
8

---

Published as a conference paper at ICLR 2023
BSD68
Kodak24
McMaster
Urban 100
Method
0=
0=
or=50
d==15
0==5
0=
0==15
0=
a=
0=15
o=
o0=50
CBM3D Dabov cl. (2007)
N/A
NIA
27.38
N/A
N/A
28.63
N/A
N/A
N/A
N/A
N/A
27.94
IRCNN Zhang et al. (2017b)
33.86
31.16
27.86
34.69
32.18
28.93
34.58
32.18
28.91
33.78
31.20
27.70
FRDNet Zhang et (2018a)
33.87
31.21
27.96
34.63
32.13
28.98
34.66
32.35
29.18
33.83
31.40
28.05
DrCNN Zhang ct al. (2017a)
33,90
31.24
27.95
34.60
32.14
28.95
33.45
31.52
28.6
32.98
30.81
27.59
RNAN Zhang al. (2019)
N/A
NIA
28.27
N/A
N/A
29.58
N/A
N/A
29.72
N/A
N/A
29.08
RDN Zhang et al. (2020)
N/A
NIA
28.31
N/A
N/A
29.66
N/A
NIA
N/A
N/A
N/A
29.38
IPT Chen et (2021a)
NIA
NIA
28.39
N/A
N/A
29.64
N/A
N/A
29.98
N/A
N/A
29.71
DRUNet Zhang el al. (2021a)
34,30
31.69
28.51
35.31
32.89
29.86
35,40
33,14
30.08
34,81
32,60
29.61
P3AN Hu et al. (2021)
N/A
N/A
28.37
N/A
N/A
29.69
N/A
N/A
N/A
N/A
N/A
29.51
SwinIR Liang et al. (2021)
34,42
31.78
28.56
35.34
32.89
29.79
35.61
33.20
30.22
35.13
32.90
29.82
Restormer Zamir et al, (2022)
34.40
31.79
28
35.47
33.04
30.01
35.61
33.34
30.30
35.13
32,96
30.02
ART (ours)
34.46
31.84
28.63
35.39
32.95
29.87
35.68
33.41
30.31
35,29
33.14
30.19
ART+ (ours)
34.47
31.85
28.65
35,41
32.98
29.89 89.89999
35.71
33.44
30.35
35.34
33.20
30.27
Table 4: PSNR (dB) comparisons. The best and second best results are in red and blue.
Dataset
RNAN
RDN
DRUNet
SwinIR
ART (ours)
ART
(ours)
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
29.96
0.8178
30.00
0.8188
30.16
0.8234
30.27
0.8249
30.27
0.8258
30.32
0.8263
Classic5
30
33,38
0.8924
33.43
0.8930
33.59
0.8949
33.73
0.8961
33.74
0.8964
33.78
0.8967
40
34.27
0.9061
34.27
0,9061
34.41
0.9075
34.52
0,9082
34.55
0,9086
34,58
0.9089
29.63
0.8239
0.8247
29.79
0.8278
29.86
0.8300
29.92
0.8305
LIVEI
30
33.45
0.9149
33.51
0.9153
33.59
0,9166
33.69
0,9174
33.71
0.9178
33.74
0.9181
40
34.47
0.9299
34.51
0.9302
34.58
0,9312
34.67
0,9317
34.70
0.9322
34.73
0.9324
Table 5: PSNR (dBy/SIM comparisons. The best and second best results are in red and blue.
Visual Comparisons. The visual comparison for color image denoising of different methods is shown in Fig. 7. Our ART preserve detailed textures and high-frequency components and remove heavy noise corruption. Compared with other methods, it has better performance to restore clean and crisp images. It demonstrates that our ART is also suitable for image denoising.
PSNR (JB)
Noiay/ 15.15
CRM3D/ 28.22
IRCNN 28.57
DoCNN 29.13
Usban100: 1m3
RNAN/ 30.28
RDN / 30.48
Swin /1RR//1... 31.16
Restomer 31.81
ART/ 31,94
Figure 7: Visual comparison with challenging examples on color image denoising (a=50).
4.5 JPEG COMPRESSION ARTIFACT REDUCTION
We compare our ART with state-of-the-art JPEG CAR methods: RNAN Zhang et al. (2019), RDN Zhang et al. (2020), DRUNet Zhang et al. (2021a), and SwinIR Liang al. (2021). Fol- lowing most recent works, we set the compression quality factors of original images to 40, 30, and 10. We provide the PSNR and SSIM comparison results in Table 5.
Quantitative Comparisons. Table 5 shows the PSNR/SSIM comparisons of our ART with existing state-of oh-the-art methods. We can see that our proposed method has the best performance. Better results are achieved by ART+ using self-ensemble. These results indicate that our ART also performs outstandingly when solving image compression artifact reduction problems.
5 CONCLUSION
In this work, we propose Attention Retractable Transformer for image restoration named ART, which offers two types of self-attention blocks to enhance the Transformer representation ability. Most previous image restoration Transformer backbones mainly utilize dense attention modules to alleviate self-attention computation within non-overlapping regions and thus suffer from restricted receptive fields. Without introducing additional computational cost, we employ the sparse attention mechanism to enable tokens from sparse areas of the image to interact with each other. In practice, the alternating application dense and sparse attention modules is able to provide retractable attention for the model and bring promising improvement. Experiments on image SR, denoising, and JPEG CAR tasks validate that our method achieves ofate-of-the-a results on various benchmark datasets both quantitatively and visually. In future work, we will try to apply our proposed method to more image restoration tasks, like image deraining, deblurring, dehazing, and so on. We will further explore the potential of sparse attention in solving low-level vision problems.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported in part by NSFC grant 62141220, 61972253, U1908212, 62172276, 61972254, the Program for Professor of Special Appointment (Eastern Scholar) Shanghai In- stitutions of Higher Learning, the National Natural Science Foundation of China under Grant No. 62271414, Zhejiang Provincial Natural Science Foundation of China under Grant No. LR23F010001. This work was also supported by the Shenzhen Science and Technology Project (JCYJ202001091 1111202200091428000544 428080344, and in part by Guangdong Special Support (2019TX05X187). Xin Yuan would like to thank Research Center for Industries of the Future (RCIF) at Westlake University for supporting this work.
REPRODUCIBILITY STATEMENT
We provide the reproducibility statement of our proposed method this section. We introduce the model architecture and core dense and sparse attention modules in Sec. 3. Besides, we also give the implementation details. In Sec. 4.1, we provide the detailed experiment settings. To ensure the reproducibility, provide the source code and pre- prainee models the website'. Everyone can run our code to check the training and testing process according to the given instructions. At the website, the pre-trained models provided to verify the validity of corresponding results. More details please refer to the website or the submitted supplementary materials.
REFERENCES
Saced Anwar and Nick Barnes. Densely residual laplacian super-resolution. TPAMI, 2020.
Pablo Arhelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. TPAMI, 2010. 6
Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel. Low-complexity single- image super-resolution based nonnegative neighbor embedding. In BMVC, 2012. 6
Pierre Charbonnier, Laure Blanc-Feraud, Gilles Aubert, and Michel Barlaud. Two deterministic half-quadratic regularization algorithms for computed imaging. In ICIP, 1994.
Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng. Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In CVPR, 2021a. 1, 2, 3. 7,7,8
Haoyu Chen, Jinjin Gu, and Zhi Zhang. Attention in attention network for image super-resolution. arXiv preprint arXiv:2104.09997 2021b. 2
Xiangxiang Chu, Zhi Tian, Yuging Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. In NeurlPS, 2021.
Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen O. Egiazarian. Color image denoising via sparse 3d collaborative filtering with grouping constraint luminance-chrominanc space. In ICIP, 2007. 8,
Tao Dai, Jianrui Cai, Yongbing Zhang. Shu-Tao Xia, and Lei Zhang. Second-order attention network for single image super-resolution. In CVPR, 2019. 3.7
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning a deep convolutional network for image super-resolution. In ECCV, 2014. 1,2
Chao Dong, Chen Change Loy, and Xiaoou Tang. Accelcrating the super-resolution convolutional neural network. In ECCV, 2016. 4
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov. Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x words: Transformers for image recognition scale. In ICI.R, 2021. 1,
Akshay Dudhane, Syed Waqas Zamir, Salman Khan, Fahad Khan, and Ming- Hsuan Yang. Burst image restoration and enhancement. In CVPR, 2022. 2
10

---

Published as a conference paper at ICLR 2023
Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Pointwise shape-adaptive det for high-quality denoising and deblocking grayscale and color images. TIP, May 2007.
Rich Franzen. Kodak lossless true color image suite. source: htpp://k.. usfgraphics/koda, 1999. 6
Muhammad Haris, Greg Shakhnarovich, and Norimichi Ukita. Deep back-projection networks for super- resolution. In CVPR, 2018. 6
Kaiming He, Jian Sun, and Xiaoou Tang. Single image haze removal using dark channel prior. TPAMI, 2010.
Han Hu, Zheng Zhang. Zhenda Xie, and Stephen Lin. Local relation networks for image recognition. In ICCV, 2019. 1
Xiaowan Hu, Ruijun Ma, Zhihong Liu, Yuanhao Cai, Xiaole Zhao, Yulun Zhang, and Haoqian Wang. Pseudo 3d auto-correlation network for real image denoising. In CVPR, 2021.
Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self- exemplars. In CVPR, 2015.
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In CVPR, 2016a. 3
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Deeply-recursive convolutional network for image super- resolution. In CVPR, 2016b.
Diederik Kingma Jimmy Ba. Adam: method for stochastic optimization. ICLR, 2015.
Xiangtao Kong, Xina Liu, Jinjin Gu, Yu Qiao, and Chao Dong. Reflash dropout in image super-resolution. In CVPR, pp. 0002-6012, 2022.
Wei-Sheng Lai, Jia- Bin Huang. Narendra Ahuja, and Ming- fing-Hsuan Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. CVPR, 2017.
Zhen Li, Jinglei Yang, Zheng Liu, Xiaomin Yang, Gwanggil Jeon, and Wei Wu. Feedback network for image
super-resolution. In CVPR, 2019. 7
Zheyuan Li, Yingqi Liu, Xiangyu Chen, Haoming Cai, Jinjin Gu, Yu Qiao, and Chao Dong. Blueprint separable residual network for efficient image super-resolution. In CVPR, 2022. 2
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Lue Van Gool, and Radu Timofte, Swinir: Image restoration using swin transformer. In JCCVW, 2021. 1, 2, 3, 6, 7,
Bee Lim, Sanghyun Son, Heewon Kim, Scungjun Nah, Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In CVPRW, 2017. 1, 6,
Jie Liu, Wenjie Zhang, Yuting Tang, Jie Tang, and Gangshan Wu. Residual feature aggregation network for image super-resolution. CVPR, 2020. 7
7e I Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Kede Ma, Zhengfang Duanmu, Qingbo Wu, Zhou Wang, Hongwei Yong, Hongliang Li, and Lei Zhang. Waterloo exploration database: New challenges for image quality assessment models. TIP, 2016. 6
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV, 2001.
Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-based manga retrieval using manga109 dataset. Multimedia Tools and Applications, 2017. 6
Yiqun Mei. Yuchen Fan, Yuqian Zhou, Lichao Huang. Thomas Huang, and Humphrey Shi. Image super- resolution with cross-scale nocalocal attention and exhaustive self elf-exemplars mining. In CVPR, 2020. 7
Yiqun Mei, Yuchen Fan, and Yuqian Zhou. Image resolutisot with non-local sparse attention. In CVPR, 2021.7 7
Tomer Michaeli and Michal Irani. Nonparametric blind super-resolution. ICCV, 2013.
Ben Niu, Weilei Wen, Wenqi Ren, Xiangde Zhang, Lianping Yang. Shuzhen Wang, Kaihao Zhang, Xiaochun Cao, and Haifeng Shen. Single image super-resolution via holistic attention network. In ECCV, 2020. 3,

---

Published as a conference paper at ICLR 2023
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation pytorch. 2017.
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Manddalona self-attention in vision models. In NewrIPS, 2019.
Mchdi SM Sajjadi, Berhhard Schölkopf, and Michael Hirsch. Enhancenet: Single image super-resolution through automated texture synthesis. In ICCV, 2017. 4
Hamid R Sheikh, Muhammad F Sabir, and Alan C Bovik. A statistical evaluation of recent full reference image quality assessment algorithms. TIP, 2006. 6
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Toiz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using efficient sub-pixel convolutional neural network. In CVPR, 2016. 4
Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu. Memnet: A persistent memory network for image restoration. In ICCV, 2017. 4
Chunwci Tian, Yong Xu, and Wangmeng Zuo. Image denoising using deep enn with batch renormalization.
Neural Networks, 2020. 1
Radu Timofte, Vincent De, and Luc Van Gool. Anchored neighhorhood regression for fast example-based super-resolution. In /CCV, 2013. 2
Radu Timofte, Eirikur Agustsson, Lue Van Gool, Ming-Hsuan Yang, Lei Zhang, Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, Kyoung Mu Lee, et al. Ntire 2017 challenge single image super-resolution: Methods and results. In CVPRW, 2017.
Hugo Touvron, Matthieu Cord. Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In JCML, 2021. 1
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxim: Multi-axis mlp for image processing. In CVPR, 2022a. 3
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In ECCV, 2022b. 3,5
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017. 1,3
Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Axial-dceplah: Stand-alone axial-attention for panoptic segmentation. In ECCV, 2020. 1
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang. Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021.
Wenxiao Wang, Lu Yao, Long Chen, Binbin Lin, Deng Cai, Xiaofci Hc, and Wei Liu. Crossformer: A versatile vision transformer hinging cross-scale attention. In ICLR, 2022a. 2, 3,5
Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: A general u-shaped transformer for image restoration. In CVPR, 2022b. 1,3
Zhou Wang, Alan Bovik, Hamid R Sheikh, and Fero Simoncelli. Image quality assessment: from error visibility to structural similarity. TIP, 2004. 6
Zhihao Xia and Ayan Chakrabarti. Identifying recurring patterss with deep neural networks for natural image
denoising. In WACV, 2020. 1
Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo. Learning texture transformer network for image super-resolution. In CVPR, 2020. 1.
Qihang Yu, Yingda Xia, Yutong Bai, Yongyi Lu, Alan L Yuille, and Wei Shen. Slance-and-gaze vision transformer, In NeurIPS. 2021. 3,5
Syed Wagas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. learning enriched features for real image restoration and enhancement. In ECCV, 2020.
12

---

Published as a conference paper at ICLR 2023
Syed Waqas Zamir, Adilya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hssua Yang, and Ling Shao. Multi-stage progressive image restoration. CVPR, 2021.
Syed Waqas Zamir, Aditya Arora, Salman H. Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer high-resolution image restoration. CVPR, 2022. 1, 9
Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse-representtaioons In Proc. 7th Int Conf. Curves Surf., 2010.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng. and Lei Zhang. Beyond gaussian denoiser: Residual learning of deep enn for image denoising. TIP, 2017a. 1, 9
Kai Zhang. Wangmeng Zuo, Shuhang Gu, and Lei Zhang. Learning deep denoiser prior for image restoration. In CVPR, 2017b. 8,
Kai Zhang, Wangmeng Zuo, and Lei Zhang. Ffdnet: Toward fast and flexible solution for enn-based image denoising. TIP, 2018a. 8,9 9
Kai Zhang. Yawei Li. Wangmeng Zuo, Lei Zhang, Luc Van Gool, and Radu Timofte. Plug-and-pla image restoration with deep denoiser prior. TPAMI, 2021a. 3, 8,
Lei Zhang, Xiaolin Wu, Antoni Buades, and Xin Li. Color demosaicking by local directional interpolation and nonlocal adaptive thresholding. Electron Imaging, 2011. 6
Yulun Zhang, Kumpeng Li, Kai Li. Lichen Wang. Bineng Zhong, and Yun Fu. Image super-resolution using very deep residual channel attention networks. In ECCV, 2018b. 1,2, 3,
Yulun Zhang, Kunpeng Li, Kai Li, Bineng Zhong, and Yun Fu. Residual non-local attention networks for image restoration. In ICI.R, 2019. 8,9
Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu, Residual dense network for image restoration. TPAMI, 2020. 1,3,4, 4, 8,9
Yulun Zhang, Huan Wang, Can Qin, and Yun Fu. Aligned structured sparsity learning for efficient image super-resolution. NeurlPS, 2021b.
Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring self-attention for image recognition. In CVPR, 2020.
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zckun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from seguence-to-sqqueeee perspective with transformers. In CVPR, 2021. 1
Shangehen Zhou, Jiawei Zhang, Wangmeng Zmo, and Chen Change Loy. Cross-scale internal graph neural network for image super-resolution. In NeurlPS, 2020. 7
13


Reviews:
[{"review_id": "p0NEbYL25t", "reviewer": "Reviewer_dTRi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Attention Retractable Transformer (ART) for image restoration, which integrates both dense and sparse attention modules in the transformer-based network. The dense attention block and sparse attention blocks emerge alternately to enable interactions among tokens extracted from a sparse area during restoration. The paper claims that the proposed ART enhances representation ability of Transformer while providing retractable attention on the input image. Extensive experiments are conducted on image super-resolution, denoising, and JPEG compression artifact reduction tasks.", "summary_of_the_review": "The paper proposed sparse attention module and design interlaced attention blocks to leverage near and distant feature attention. The designed ART could achieve remarkable improvements on the images with repeat patterns, but it cannot prove its advantages on other general datasets.  The non-local attention modules have been proposed by other recognition works.", "strength_and_weaknesses": "Strength:\n+  This paper introduces sparse attention mechanism to low-level vision task and enable tokens from sparse areas of the image to interact with each other to enlarge the receptive field of transformer.\n+ The paper is well organized and clearly presented. In the paper, the differences and relations with other related works are discussed and analyzed.\n\nWeakness:\n- The improvement in performance is limited. In the SR results. The ART-s has similar parameter size and FLOPS as SwinIR.  The gain obtained by ART-s is insignificant in terms of PSNR and SSIM on dataset Set5, Set14, and B100. ART and ART+ outperforms the SwarIR by 0.1dB, but they are equipped with more parameters. For the image denosing task, the gap between the SwinIR and restormer is not remarkable. \n- The proposed scheme introduces the features from dilated position to enable distant feature access, which benefits restoration of the images with repeated pattern, such as buildings in Urban100.  Table 2 and figure 4, the Urban100 are chosen to be compared with peer works.  The paper should provide more comprehensive result comparison to verify the advantage of ART.\n- The description of the method could be further refined. E.g. The symbol “I” represents original input and intervals in the proposed method. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Tjy2ot4GigR", "reviewer": "Reviewer_eLG6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an Attention Retractable Transformer (ART), which incorporates sparse and dense attention to widen the receptive field size. The ART method achieves pretty good results in several image restoration tasks, like super-resolution, denoising, compression artifact reduction.", "summary_of_the_review": "The authors propose a new Transformer-based image restoration method with several new components, which are demonstrated by the extensive ablation study and main experiments. Furthermore, the paper is well prepared and reproducible based on the submitted code and models.", "strength_and_weaknesses": "The idea about using sparse and dense self-attention in Transformer is simple but effective. It is good to see such an investigation in image restoration and good performance for different tasks.\n\nAlso, the alternating application of sparse and dense attention blocks help capture the local and global information. The corresponding ablation study supports the claim well.\n\nThe main comparisons with recent methods for different tasks are extensive. The ART achieves quantitative gains and also show obviously better visual results in some challenging cases.\n\nThe supplementary file provides more experiments to show the effectiveness of the ART. They also submit the demo code and pretrained models, which make their work reproducible and more solid.\n\nThe paper is prepared pretty well and organized clearly. It is also easy to read and follow with the provided code.\n\nWeaknesses:\n\nThe proposed method is claimed for image restoration. The authors show experiments on several image restoration tasks, like image super-resolution, color, and real image denoising, and JPEG compression artifact reduction. How would be the method performed for other restoration tasks? Like image deblurring, deraining, dehazing, and so on. It would be good if the authors can provide some additional results for one/some of those tasks. Or at least, the authors can briefly discuss them in the conclusion part.\n\nIn image denoising, the main architecture of ART is different from that of Restormer. Also, Restormer uses a progressive training strategy, which is not used in this paper. Do the authors think that ART would further benefit from the U-Net structure and progressive training used in Restormer? If so, why not use a similar structure and progressive training? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "R3bXuRvsd0", "reviewer": "Reviewer_wRmT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates an efficient Transformer based method for high-quality image restoration. Specifically, the proposed Attention Retractable Transformer (ART) incorporates sparse and dense attention to sense larger receptive field. The alternative manner of those two attention blocks help capture global and local information. The ablation study and main comparisons support the claimed components well with SOTA performance.", "summary_of_the_review": "Overall, the paper is high-quality with a novel method (ART). The experiments are very extensive and support the contributions well. The submitted code and models make the method more solid.", "strength_and_weaknesses": "Strengths\n\n1. The paper is well-written and easy to follow. The idea about attention retractable Transformer is simple yet effective. The combination of sparse and dense attention help sense global and local information for high-quality image restoration.\n\n2.The authors provide extensive experiments to show the effects of each proposed component and the whole method. They also discuss the differences between the proposed method and other related works, which highlight the novelty of the paper.\n\n3.Some experimental curves (e.g., Figs. 4, 8, and 9) provide good visualization during the training. Those curve comparisons make the method more convincing.\n\n4.The quantitative results are good. In Tabs. 2, 4, and 5, the proposed ART achieves the best PSNR/SSIM values in most cases. Such improvements over the previous methods are notable, especially for image SR. The authors also provide model size and flops comparisons, trying to keep fair comparisons with similar parameters and flops.\n\n5.The visual comparisons (e.g., Figs. 5 and 7) show impressive results of the proposed method, which obtains obviously better outputs than others. Such comparisons are also provided in the supplementary file.\n\n6.The supplementary file provides many useful experiments, like more ablation study results, analyses and performance comparisons with related work, real image denoising, and other quantitative and visual comparisons. All those supplements further demonstrate the effectiveness of the proposed method.\n\n7.The authors provide demo code on an anonymous website, where the pretrained models and testing scripts are provided to reproduce results in the main paper and supplementary file. The code availability makes the work more solid and convincing.\n\nWeaknesses\n\n1.Some details are not clear enough. For example, in Tab. 7 of the supplementary file, real image denoising part, did the authors use the same pretrained model to test on SIDD and DND data? Or use different pretrained models for each dataset? Also, did the authors still use the architecture shown in Fig. 2(a) for real image denoising? (As I know, Restormer uses a different structure.) There are some training strategies in Restormer, like progressive training. Did the authors also use such a strategy?\n\n2.It would be much better if the authors can provide some discussions or/and experiments of the method applied for other image restoration applications, like deraining and deblurring.\n\n3.It would be much better if the authors could provide more curve and/or visual comparisons. For example, how about the validation curves and visual results between the proposed method and others, like Restormer.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "N8DUp5EmGZy", "reviewer": "Reviewer_f4x2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed an attention retractable Transformer (ART) for accurate image restoration, which consists of dense and sparse attention modules. The experiments are extensive and include several general image restoration tasks, where the proposed method achieves SOTA performance.", "summary_of_the_review": "A well-prepared paper with solid and extensive experiments.", "strength_and_weaknesses": "Strength:\n\n1. The paper is well-written and easy to follow. It is obvious that the authors pay much attention to most details.\n\n2. The idea about attention retractable Transformer (ART) is motivated by some observations from image restoration tasks and is more specifically designed. The authors provide illustrations of differences between the proposed method and others, like in Figure 1 and Table 1.\n\n3. They propose a sparse attention to compensate the defect of dense attention used in most Transformer-based image restoration methods. In the sparse area of an image, the token interactions help enlarge the receptive field of the module.\n\n4. Alternating sparse and dense self-attention blocks helps provide retractable attention. Then, the ART model can capture both local and global information simultaneously.\n\n5. The ablation study experiments well support the claimed contributions. The main results with promising performance show the superior and effectiveness of the proposed method.\n\n6. In the supplementary file, the authors not only provide more results (e.g., real image denoising), but also provide the source code, pretained models, and detailed readme information (e.g., scripts). Those supplements make the paper stronger and more solid.\n\nWeakness:\n\n1. For image denoising, the authors mainly provide color image denoising results in the main paper and real image denoising in supplementary file. It would be better to include real image denoising in the paper too.\n\n2. In the demo code, the authors mainly provide testing scripts. Will it be possible to give all the training details in the future? That would be good to the image restoration community.\n\n3. Some details are not very clear. For image denoising, did the authors use the framework and/or training strategy (e.g., progressive training) as used in Restormer?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%