Adaptive Cross-Layer Attention for Image Restoration
Authors: Yancheng Wang
Yingzhen Yang
Chong Chen
Ning Xu
Venue: ICLR 2022 Submitted (2022)
Keywords: image restoration
neural architecture search
non-local attention

Abstract: Non-local attention module has been proven to be crucial for image restoration. Conventional non-local attention processes features of each layer separately, so it risks missing correlation between features among different layers. To address this problem, we propose Cross-Layer Attention (CLA) module in this paper. Instead of ﬁnding correlated key pixels within the same layer, each query pixel is allowed to attend to key pixels at previous layers of the network. In order to mitigate the expensive computational cost of such hierarchical attention design, only a small ﬁxed number of keys can be selected for each query from a previous layer. We further propose a variant of CLA termed Adaptive Cross-Layer Attention (ACLA). In ACLA, the number of keys to be aggregated for each query is dynamically selected. A neural architecture search method is used to ﬁnd the insert positions of ACLA modules to render a compact neural network with compelling performance. Extensive experiments on image restoration tasks including single image super-resolution, image denoising, image demosaicing, and image compression artifacts reduction validate the effectiveness and efﬁciency of ACLA.

Under review as a conference paper at ICLR 2022
ADAPTIVE CROSS-LAYER ATTENTION FOR IMAGE RESTORATION
Anonymous authors Paper under double-blind review
ABSTRACT
Non-local attention module has been proven to be crucial for image restoration. Conventional non-local attention processes features of each layer separately, so it risks missing correlation between features among different layers. To address this problem, we propose Cross-Layer Attention (CI.A) module in this paper. Instead of finding correlated key pixels within the same layer, each query pixel is allowed to attend to key pixels at previous layers of the network. In order to mitigate the expensive computational cost of such hierarchical attention design, only small fixed number of keys can be selected for each query from a previous layer. We further propose variant of CLA termed Adaptive Cross-Layed Attention (ACLA). In ACLA, the number of keys to be aggregated for each query is dynamically selected. A neural architecture search method is used find the insert positions of ACLA modules to render a compact neural network with compelling performance. Extensive experiments on image restoration tasks including single image super- resolution, image denoising, image demosaicing and image compression artifacts reduction validate the effectiveness and efficiency of ACLA.
INTRODUCTION
Image restoration algorithms aim to recover a high-quality image from contaminated input image by solving an ill-posed image restoration problem. There are various image restoration tasks depending on the type of corruptions, such image denoising (Zhang al. [019 Liu et al. 2018b). demosaicing
(Zhang et al.) 2017b) 2019), single image super-resolution (Fan et al.) 2019: Lai et al., 2017 Tai
al. 2017), and image compression artifacts reductionZZhang et al. 201 7a). To restore corrupted
information from the contaminated image, variety of image priors Buades et al., 2005 Zoran &
Weiss 2011 Zontak et al., 2013) were proposed.
Recently, image restoration methods based on deep neural networks have achieved great success. Inspired by the widely used non-local prior, most recent approaches based on neural networks (Zhang et al. 2019: Liu et al. 2018b) adapt non-local attention into their neural network to enhance the representation learning, following the non-local neural networks (Wang et al. 2018). In a non- local block, response is calculated as weighted sum over all pixel-wise features the feature map to account for long-range information. Such a module was initially designed for high-level recognition tasks such image classification, and has been proven be beneficial for low-level vision tasks (Zhang et al. 2019 Liu et al. 2018b).
Though attention modules have been shown to be effective in boosting the performance. Most attention modules only explore the correlation among features at the same layer. Actually, features at different intermediate layers encode variant information at different scales, and might be helpful to augment the information used in recovering the high-quality image. Motivated by the potential benefit of exploring feature correlation across intermediate layers, Holistic Attention Network (HAN) (Nuu et al. 2020) is proposed to find interrelationship among features at hierarchical levels with Layer Attention Module (LAM). However, LAM assigns a single importance weight to all features at the same layer and neglects the difference of spatial positions of these features. Recent research in omnidirectional representation (Ta et al. 2)2 suggests that exploring the relationship among features at different layers benefit the representation learning of neural networks. Nevertheless, calculating correlation among features at hierarchical layers is computationally expensive due to the quadratic complexity of dot product attention. The complexity of such cross-layer attention design is

---

Under review as a conference paper at ICLR 2022
increased from L to (HWL)?, where H, W are the height and width of the feature map and is the number of layers.
1.1 CONTRIBUTIONS
Our contributions are listed as follows.
First, in order to address the problem caused by only referring to keys within the same layer in most attention modules, we propose novel attention module termed Cross-Layer Attention (CLA), which searches for keys across different layers for each query feature. With the help of the deformation mechanism, CLA only attends to small set of keys at different layers for each query feature.
Second, we propose an improved CLA termed Adaptive Cross-Layer Attention (ACLA) which selects adaptive number of keys each layer for each query, and searches for the optimal insert positions of ACLA modules. We deploy ACLA modules on commonly used neural network models, e.g. EDSR (Lim et al.] 2017a), for image restoration. Extensive experiments single image super-resolution, image denoising, image compression artifacts reduction, and image demosaicing demonstrate the effectiveness of our approach.
2
RELATED WORKS
2.1 NEURAL NETWORKS FOR IMAGE RESTORATION
Adopting neural networks for image restoration has achieved great success by utilizing their power in representation. ARCNN (Dong et al. 2015a) was first proposed use CNN for compression artifacts reduction. Later, DrCNN (Zhang et al. 2017a) uses residual learning and batch normalization to boost performance of CNN for image denoising. In IRCNN (Zhang et al. 20176), learned set of CNNs are used as denoising prior for other image restoration tasks. For single image super- resolution[Lai et al. 2017 /Zhang et al., [208d] Haris et al.) 20 Fan et al. 20.920 even more efforts have been devoted to designing advanced architectures and learning methods. For example, RDN (Zang et al.] 2018d) and CARN (Ahn et al., 2018) fuse low-level and high-level features with dense connection to provide richer information and details for reconstructing. Recently, non-local attention (Liu et al. [018b) Dai et al.) 2019; Mei et al. 2020 is also used to further boost the peformance of CNN for image restoration.
2.2 ATTENTION MECHANISM
Attention mechanism has been applied many computer vision tasks, such image captioning (Xu et al.] 2015 Chen et al. 2017) and image classification (Hu et al., 2018 Wang et al. 2017). Non-local attention (Wang al., 2018) was first proposed capture long-range dependencies for high-level recognition tasks. Recently, several works propose to leverage non-local attention for low-level vision tasks. In NLRN (Liu al. (018b) a recurrent neural network proposed incorporate non-local attention. RNAN (Zhang et al. (2019) proposed residual local and non-local mask branch to obtain non-local mixed attention. RCAN (Zhang et al. 2018c) exploits the interdependencies among feature channels by generating different attention for each channel- wiiteell-weeeeeee feature. HAN (Niu et al.) 2020) is proposed to find interrelationship among features at hierarchical levels with a layer attention module. Besides, some recent works attempt explore the benefits of transformer based models for image restoration. IPT (Chen et al. 2021) is proposed to solve various restoration problems in a multi-task learning framework based on visual Transformer. SwinIR (Liang al.) (202T) adopt the architecture of Swin Transformer. However, compared with methods using CNN architecture, mansformer-basee image restoration methods usually use large datasets for training. Specifically, IPT uses ImageNet to pretrain the model. SwinIR adapts a combination of four datasets consisting of over 8000 high-quality images as training set for the tasks of denoising and compression artifact reduction.
2.3 NEURAL ARCHITECTURE SEARCII
Neural Architecture Search (NAS) has attracted lots of attention recently. Early works of NAS adopt heuristic methods such reinforcement learning (Zoph & Le| 2016) and evolutionary algorithm (Xie & Yuille, 2017). The search process with such methods requires huge computational resources. Recently, various strategies are designed to reduce the expensive costs including weight sharing (Pham)
2

---

Under review as a conference paper at ICLR 2022
et al. 2018), progressive search (Liu al.
2018a and one-shot search (Lu et al. 2018c Xie et al. 201 firstly relaxes the search space to continuous
2018). For example, DARTS (Liu et al. and conducts differentiable search. The architecture parameters and network weights trained simultaneously by gradient descent to reduce the search time.
Despite the success of NAS methods for classification, dense prediction tasks such semantic image segmentation and image restoration, usually demand more complicate network architectures. Some recent works have been devoted explore hierarchical search space for dense prediction tasks. For example, Auto-DeepLab (Liu et al. (019)) introduces a hierarchical search space for semantic image segmentation. DCNAS (Zhang al. 2021) build densely connected search space extract
multi-level information. HNAS (Guo et al. 2020 also adopts hierarchical search space for single image super-resolution.
METHOD
3.1
CLA: CROSS-LAYER ATTENTION
Non-Local Attention. Non-local attention (Wang et al.) 2018) is designed to integrate self-attention mechanism into convolutional neural networks for computer vision tasks. It is usually applied on an input feature map x R xC to explore self similarities among all spatial positions of that
feature map. For illustration purpose, reshape N x C, where N H x W. A generic non-local attention can be formulated as
N Σ nm
y/
[E,, (n)g((a)
(1)
C(a:
where indexes the spatial position of feature maps. y is the output of non-local attention with the same size as f(;; xn)) outputs the pairwise affinity between the query feature Hę and its key feature xa. g(x,)) computes an embedding of feature In C(z) normalization term
Query Feature
[xl Conv
Softmax
Attention
Weights
|xl Conv
Input Feature Map
Soft Gumbel Hard
Gather
Officts
Gating Softmax Gating
Mask
Mask
M
1x1 Conv
Gather
Aggregate
Sampled
Feature Map of Referred Layer
Keys
Output Feature Map
Figure 1: Illustration of Adaptive Cross-Layer Attention (ACLA) module. For each query pixel, 1x1 convolution layer used to obtain the offsets of the positions of keys sampled from the referred layer. Then, convolution layer and Softmax are applied to the query feature to generate attention weights for sampled keys. To adaptively find the number of informative keys, a mask unit M together with the gumbel-softmax operation is used to generate hard gating mask for the sampled keys. After the Hard Gating Mask applied, the features of the selected keys are weighted by the corresponding attention weights. During inference time, gather operation is applied on the selected keys.
3

---

Under review as a conference paper at ICLR 2022
High
Reshlock 3
Reshlock 12
Resblock 26
Resblock 31
Figure 2: Visualization of selected keys by ACLA for query feature from the 31st resblock. The first row shows the positions of the keys selected by ACLA with
16. For comparison, the positions of keys with top-16 16 attention weights following the vanilla Cross-Layer Non-Local attention formulation Equation 33) is diplayed in the second row. From left the right are the sampled key positions from the 3rd, 12nd, 26th, and 31st resblock. The query feature is shown green cross marker. Each sampled key feature is marked as a circle whose color indicates its attention weight. It can be observed that ACLA adaptively selects semantically similar key features for the query feature, and its vanilla counterpart lacks such capability. More visualization results can be found in Appendix A.5
Non-local attention in Equation T is usually wrapped into a non-local block (Wang et al. 2018) with residual connection from the input feature The mathematical formulation given
2
h(v) +
where h denotes a learnable feature transformation.
Cross-Layer Attention. To search for keys from different layers for each query feature, we propose Cross- Attention (CLA). To derive the formulation of CLA, we start by adapting non-local attention in Equation (1 to a cross-layer design. Suppose that of is the output of the i-th layer in CNN backbone for image restoration, where (1...., ...,,,,) L} and L is the number of layers. A non-local attention with cross-layer design can be formulated as
1 C(x2)
==
where the superscripts j. index the layer and the subscripts i, index the spatial locations of features. y. x denote the output feature and input feature respectively. Query feature in this formulation can thus refer features in previous layers. However, given the quadratic complexity of correlation computation, the complexity of such adaption of non-local attention increased from 2 to (NL)². In order mitigate the expensive computation cost, we leverage deformable mechanism proposed in deformable convolution (DCN) (Dai et al. 2017) to design our proposed CLA as
[(,) (p. + (pw))g(a' + Apx)), 1=1k=1
C(x)
where i indexes the sampled keys. p; represents the 2-d spatial position of the query in the feature map, and Apa is the 2-d offset from the position of query p: to the corresponding sampled key. As p: + Apu can be fractional, bilinear interpolation is used as (Dai et al. 2017) to compute a(p:+Appt To further reduce the computation complexity in computing attention weights, attention weights can be generated from query feature alone. Thus, our proposed CLA can be simplified as
K
-
C(r))
(a))g(a' + Apa)),
(5)
1=1 k=1
where f is the function to generate attention weights from the query feature. Similar to the design for non-local attention module in (Wang et al. 2018), when deploying CLA in CNN backbones, we also wrap into non-local block with residual connection.
4

---

Under review as a conference paper at ICLR 2022
With our proposed CLA, each query feature now can refer to a fixed number of keys from each previous layer. However, query features at different spatial positions may have different preferences on keys sampled from different layers. Especially for image restoration tasks, the restoration process at different spatial position may vary significantly due to the diversity of textures in an image. Besides, when deploying CLA in CNN backbones for image restoration, we find that increasing the number of inserted CLA will not constantly improve the performance, but result in much heavier computation cost. This observation motivates us search for an optimal configuration of deploying CLA in neural networks to efficiently explore its representation learning capacity.
3.2 ACLA: ADAPTIVE CROSS-LAYED ATTENTION
To achieve adaptive key selection and the search for the optimal configuration of CLA, propose Adaptive Cross-Layer Attention (ACLA). Specifically, for each query feature, we dynamically search for the keys sampled from previous layers with ACLA. Besides, when deploying ACLA in CNN backbones, a neural architecture search method is used to search for the insert positions of ACLA. An objective based on the computation cost of inserted ACLA modules is used to supervise the search procedure.
Adaptive Cross-Layer Attention. To adaptively search for the informative sampled keys for query feature from its previous layers, we first follow the method in CLA to get a fixed number of sampled keys from each layer. Next, a hard gating mask applied on each of the sampled key as
K
v
m (2)) + Ap:s)):
(6)
C(x))
R=1
where m is the hard gating mask for the k-th sampled key from x' for query feature x/. whose value either 1 or O. We further relax the hard gating mask continuous domain with the simplified binary Gumbel-Softme (Verelst & Tuytelaars, 2020). Therefore, the hard gating mask mř, is approximated by
m of ( + 5,,,1 -(122)
ck 2,1,2 and temperature т. o is the sigmoid function. The sampling parameter Bf, can be regarded as soft gating mask. To make the gating decisions on sampled keys input-dependent, mask unit M is used to generate the soft gating mask / from the features of the sampled keys as
with sampling parameter Bh Gumbel noise 1,1,1 .,
B/ = MM= + Ap:)))
Following the design in Verelst & Tuytelaars 2020), a 1 x convolution layer is used as the mask unit M in our model. To train the mask unit M, the straight- estimator from (Bengio et al.) 2013 Verelst & Tuytelaars 2020) is used for m In particular, during backward pass, we set m mi while in forward pass we
m > 0.5, [1
m
m S 0.5.
Besides, Gumbel noise t and & L2 are set to during inference. demonstrate the effectiveness of adaptive key selection in ACLA, we visualized keys selected by ACLA at different layers for query feature Figure
Insert Positions. To search for the insert positions of ACLA, we first densely insert ACLA after each layer of the CNN backbone as in Figure build the supernet. Similar to the gating formulation in ACLA, define hard decision parameter 8, {0, 1} for the j-th inserted ACLA in the supernet. 8y indicates that a ACLA module is inserted after the j-th layer, and otherwise for 8j = 0. The simplified binary Gumbel-Softmal (Verelst & Tuytelaars 2020) can also be used here approximate the hard decision parameter sy by
aj ol
$3 = σ
(10)
5

---

Under review as a conference paper at ICLR 2022
with sampling parameter a.j. Gumbel noise and temperature Thus, the output of ACLA in the supernet can be expressed as
K m
+ Ap.)).
(1I)
C(2))
Different from data-dependent design of the gating mask ACLA, aj here is defined as architecture parameter and be directly optimized by stochastic gradient descent (SGD) during the search process. By gradually decrease the value of temperature, aj will be optimized such that sj will approach 1 or 0.
Conv
ResBlock
ACLA
ResBlock
ACLA
ResBBuck
ACLA
ResBlock
ACLA
Figure 3: Illustration of the search for insert positions in ACLA
Search Procedure. We propose to optimize both the accuracy and the computation cost (FLOPs) of the ACLA modules inserted. Thus, the cost of the ACLA modules inserted needs be estimated during the search phase. Following the formulation of the ACLA in the supernet, can estimate the cost of the ACLA inserted after the j-th residual block as
K
cost = Σe 1=1
Σ
(2 N 2NC + 6KNC),
(12)
where is the number of spatial positions. of sampled keys. 2mf is the FLOPs for the convolution generating the gating masks. 2NC is the FLOPs for the convolution on sampling keys. 6KNCC is the FLOPs for the aggregation process. Then we can get the estimation on the computation cost all inserted ACLA modules as
the number of channels. is the maximal number
cost
8, costj,
(13)
which is the summation of the cost of each layer, weighted by corresponding decision parameters. As mentioned before, with the help of the continuously relaxed representation in the supernet, can search for the architecture by updating the architecture parameters using SGD. To supervise the search process, we design loss function with the cost-based regularization achieve the multi-objective optimization:
C(w, a) = LasSE + X log cost,
(14)
where À is the hyper-pane ppper-parameters that controls the magnitude of the term.
We find that at the beginning of the search process, ACLA modules inserted at shallow layers are more likely to be strengthened. To solve this problem, split search procedure into two stages. In the first stage, we only optimize the parameters of the network for enough epochs to get network weights sufficiently trained. In the second stage, we activate the architecture optimization. We alternatively optimize the network weights by descending Vw-rrain (w, ar) on the training set, and optimize the architecture parameters by descending VaLral a) on the validation When the search procedure terminates, we derive the insert positions based on the architecture parameters a.
EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
We select DIV2K (imofte al. 2017) as the training set for experiments as like in (Dai et al.) 2019 Zhhng et al. 2018d), which include 800 images for training and 100 images for validation. We augment the training images by randomly rotating 90", 180", 270" and horizontally flipping. In each

---

Under review as a conference paper at ICLR 2022
Table 1: Quantitative results on benchmark datasets for single image super-resolution
Method
Scale
Params(M)
Set5
Set14
BT00
Urban100
Manga109
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNK
SSIM
PSNR
SSIM
Bieubic
x2
33.66
0.9299
30.24
0.8688
29.56
0.8431
26.88
0,8403
30,80
0.9339
SRCNN
X2
0.244
36.66
0,9542
32.45
0.9067
31.36
0,8879
29.50
0.8946
35.60
0.9663
VDSR
x2
0.672
37.53
0.9590
33.05
0.9130
31.90
0.8960
30.77
0.9140
37.22
0.9750
MomNet
x2
0.677
37.78
0.9597
33.28
0.9142
32.08
0.8978
31.31
0.9195
37.72
0.9740
SRMDNF
x2
5.69
37.79
0.9601
33.32
0.9159
32.05
0.8985
31.33
0.9204
38.07
0.9761
RDN
x2
22.6
38.24
0.9614
34.01
0.9212
32.34
0.9017
32.89
0.9353
39.18
0.9780
HAN
X2
17.3
38.27
0,9614
34.16
0.9217
32.41
0.9027
33.35
0.9385
39.46
0.9787
EDSR
X2
40.7
38.
0.9602
33,92
0.9195
32.32
0.9013
32.93
0.9351
39,10
0.9773
EDSR+CLA
X2
42.1
38.24
0,9613
34,08
0,9214
32.41
0,9028
33.28
0.9367
39.23
0.9780
EDSR-ACLA
X2
42.3
38.31
0.9617
34.10
0,9221
32.43
0.90130
33.35
0.9385
39.42
0.9787
RCAN
X2
13.3
18.27
0.96114
34.12
0.9216
32.41
0.50227
35.34
0.9384
39.44
0.9786
RCAN+CLA
x2
16.5
38.27
0,9615
34,14
0.92118
32.43
0.9013
33.34
0.9385
39.46
0.9785
RCAN+ACLA
X2
16.7
38.30
0,9615
34,15
0.9217
32.45
0.9029
33.39
0.9387
39.48
0.9789
Bicubic
X3
M0.39
0.8682
27.55
0.7742
27.21
0.7385
24.46
0.7349
26.95
0.8556
SRCNN
x3
0.244
32.75
0,9090
29.30 30
0.8215
28.41
0.7863
26.24
0.7989
30.48
0.9117
VDSR
X3
0.672
33.67
0.9210
29.78
0.8320
28.83
0.7990
27.14
0,8290
32.01
0.9340
MemNNet
X3
0.677
34.09
0,9248
30.00
0.8350
28.96
0.8001
27.56
0.8376
32.51
0.9369
SRMDNF
X3
5.69
34.12
0.9254
30.04
0.8382
28.97
0.8025
27.57
0,8398
33,00
0.9403
RDN
X3
22.6
34.71
0,9296
30.57
0.8468
29. 29
0.8093
28, 80
0.8653
34.13
0.9484
HAN
X3
17.3
34.75
0,9299
30.67
0,8483
29.32
0.8110
29.10
0.8705
34.48
0.9500
EDSR
X3
40.7
34,65
0.9280
30.32
0.84162
29.25
0.8093
28.80
0.8653
34.17
0.9476
EDSR+CLA
42.1
34.75
0.9297
30.66
0.8481
29.30
0.8113
29.05
0.8700
34,33
0.9492
EDSR+ACIA
x3
42.3
34.76
0.9303
30.69
0.8484
29.34
0.8115
29. 12
0.8706
34.40
0.9498
RCAN
X3
15.3
34.74
0.9299
30.65
0.8482
29.32
0.8111
29.09
0.8702
34.44
0.9499
RCAN-CLA
X3
16.5
34.75
0.9301
30.67
0.8485
29.31
0.8114
29.11
0,8705
34.46
0.9499
RCAN+ACLA
X3
16.7
34.74
0,9304
30.68
0.8485
29.33
0,8115
29.14
0.8709
34.48
0.9503
Bicubic
X4
42
0.8104
26.00
0.7027
36,96
0.6675
23.14
0.6577
24.89
0,7806
SRCNN
0.244
30.48
0.8628
0.7513
26.90
0.7101
0.7221
27.58
0.8555
VDSR
X4
0.672
31.35
0.8830
28.02
0.76800
27.29
0.0726
25.18
0.7540
28.83
0.8870
Memieet
0.677
31.74
0.8893
28.26
0.7723
27.40
0.7281
0.7630
29.42
0.8942
SRMDNF
X4
5.69
31.96
0.8925
28.35
0.7787
27.49
0.7337
25.68
0.7731
30,09
0.9024
RDN
22.6
32.47
0,8990
0.7871
0.7419
0.8028
31,00
0.9151
HAN
X4
17.3
32.64
0.9002
28,90
0,7890
27.80
0.7442
26.85
0,8094
31.42
0.9177
EDSR
X4
40.7
32.46
0,8968
28, 80
0.7876
27.71
0.7420
26.64 64
0.8033
31,02
0.9148
EDSR
42.1
32.64
0.9001
28.83
0,7880
27.79
0.7435
26.79
0. 8083
31.16
0.9159
EDSR-ACLA
X4
42.3
32.64
0,9003
28.88
0.7883
27.85
0.7443
26.87
0. 0,8087
31.24
0.9174
RCAN
15.3
32.63
0.9002
28.87
0.7889
27.77
0.7436
26.82
0.8087
31.22
0.9173
RCAN-CLA
X4
16.5
32.65
0.9002
28.88
0.7891
27.80
0.7441
26. 86
0. 0.8087
31.27
0.9177
RCAN+ACLA
16.7
32.64
0,9002
28.90
0.7892
27.
0.7445
26.89
31.29
0.9179
mini-batch, 16 low-quality patches with size 48 x 48 are provided as inputs. ADAM optimizer is used for both the search phase and training phase. Default values of 31 and 83 are set as 0.9 and 0.999, respectin
respectively, and we set e = 10- 8 Before the search, we perform a cross-validation on 20% of the
training data to decide the value of X. In search phase, the learning rate is initialized as 10-4 and cosine learning rate schedule is used. The search process takes 200 epochs. In the training phase, the learning rate is initialized as 10-4 and then reduced to half every 200 epochs. The model trained for 1000 epochs in total. K = 8 is sef for all CLA modules by default. For all ACLA modules, the value of K are initialized as 16.
4.2 SINGLE IMAGE SUPER-RESOLO RESSOLUTION
We test our methods on top of the widely used super-resolution backbone EDSR (Lim et al. 2017b) and RCAN (Zhang al. 2018b). The LR images are obtained by the bicubic downsampling of HR images. Our methods are evaluated on five standard datasets: Set5 (Bevilacqua et al. 2012). S (Zeyde et al. 2010)) B100 (Martin al. 2001). Urban (Huang et al. 2015), and Mangal Matsui et al. 2017). The reconstructed results by our model are first converted to YCbrr space, PSNR and SSIM in the luminance channel is calculated in our experiments. We compare our methods with 6 other methods: SRCNN (Dong et al.) 2015b), VDSR (Kim et al. (016)). MemNet ( (Ti et al. al. 2017). SRMDNF (Zhng et al., 2018a), RDN (Zhang et al. (0T 2018d), and HAN (Niu et al. 2020).
The quantitative results are shown in Table D Some visual results can be found in Appendix A.6 Our methods greatly improves the performance of EDSR and RCAN on all benchmarks and all upsampling scales. Besides, our methods outperform current sate-of-the-aa method HAN, which is also an attention based method, demonstrating the superiority of our proposed attention modules.
4.3
IMAGE DENOISING
We evaluate our proposed CLA and ACLA module on standard benchmarks for image denoising: KCLDAK24 BSD68 (Martin et al. 2001), and 1rban100 (Huang al., 2015). The noisy images
7

---

Under review as a conference paper at ICLR 2022
created by adding AWGN noises with 10,30, 50, 70. We compare approach with methods, namely DrCNN (Zhagg et al. 2017a). MemNet (Tai et al. 2017). RNAN (Zhang et al.)
2019), and PANet (Mei et al. 2020). For fair comparisons. 16-layer EDSR is used as the baseline CNN backbone. As shown in Table 2 our methods are also beneficial for representation learning
with neural network image denoising.
Table 2: Quantitative results benchmark datasets for single image denoising
KCLDAE
BSD68
Urban
100
Method
(M)
10
50
70
10
30
50
70
10
30
50
70
MemNet
0.677
N/A
29.67
27.65
26.40
N/A
28.39
26.33
25.08
N/A
28.93
26.53
24.93
DrCNN
0.672
36.98
31.39
29.16
27.64
36.31
30.40
28.01
26.56
36.21
30.28
28.16
26.17
RNAN
7.409
37.24
31.86
29.58
28. 16
36.43
30.63
28.27
26.83
36.59
31.50
29.08
27.45
PANet
5.957
37.35
31.96
29.65
28.20
36.50
30,70
28.33
26.89
36.80
31.87
29.47
27.87
baseline
5,430
37.21
31.85
29.60
28.15
30.60
28.28
26.84
36.63
31.64
22
27.54
CLA
5.896
37.37
31.97
29.67
28.23
36.52
30.74
28
26.91
36.79
31.85
29.43
27.88
ACLA
5.914
37.38
31.97
29.70
28.25
36.54
30.77
28.36
26.94
36.85
31.90
29.49
27.91
4.4
IMAGE COMPRESSION ARTIFACTS REDUCTION
For the task of image compression artifacts reduction (CAR), compare method with approaches: DnCNN (Zhang et al. 2017a). RNAN (Zhang et al. 2019). and PANet (Mei et al. 2020). All methods evaluated LIVET (Sheikh al., 2005) and Classic5 (Foi et al. 2007). To obtain the low-quality compressed images, we follow the standard JPEG compression process and use Matlab JPEG encoder with quality q 10, 20, 30, 40, For fair comparison, the results are only evaluated on Y channel in YCbCr Space. The quantitative results are shown in Table 3 16-layer EDSR used our baseline CNN backbone. Both CLA and ACLA improves the performance of the neural network baseline.
Table 3: Quantitative results on benchmark datasets for image compression artifacts reduction
LIVEI
Classic5
Method
Params (M)
30
40
10
20
30
40
JPBG
27.37
30.07
31.41
32,35
27.82
30.12
31.48
32,43
DrCNN
0.672
29.19
31.59
32.98
33.96
29.40
31.63
32.91
33.77
RNAN
7.409
29.63
32.03
33.45
34.47
29.96
32.11
33.38
34.27
PANet
5.957
29.69
32.10
33.55
34.55
30.03
32.36
33.53
34.38
baseline
3,430
29.63
32.04
33.50
35.51
29.99
32.22
33,43
34.31
CLA
5.896
29.73
32.13
33.57
35.54
30.05
32.38
33.55
34.42
ACLA
5.914
29.73
32.17
33.63
35.55
30.07
32.42
33.58
34.44
4.5 ABLATION STUDY AND DISCUSSION
ACLA vs. Non-local attention To verify the effectiveness of our proposed methods, we compare CLA and ACLA with Non-Local attention (Wang et al.) 2018) and vanilla Cross-Layer Non-Local attention in terms of computational efficiency and performance. The vanilla Cross-Layer Non-Local attention follows the formulation in Equation (3. The comparison is performed on Set5 for single image super-resolution with EDSR backbone. The Non-Local attention modules and vanilla Cross- Layer Non-Local attention modules are inserted evenly after every 8th residual blocks. All the FLOPs in our ablation study are calculated for input of size 48 x 48. Results are presented in Table 4 where NI stands for Non-Local attention and CINL stands for vanilla Cross-Layer Non-Local. shown in Table with less computation cost, CLA and ACLA achieve much better performance compared to standard Lon-Local attention module.
Table 4: Efficiency and performance comparison with Non-Local attention on Set5
Method
PSNR
FLOPs(G)
Params(M)
EDSR
38.11
93.97
40.73
NL
38.15
109.38
43.56
CLNL
38.14
122.67
45.87
CLA (Ours)
38.24
96.93
42.13
ACLA (Ours)
38.31
96.97
42.29
Number of inserted CLA modules As stated before, the computation cost of the cross-layer design in CLA is quadratic to the number of inserted CLA modules. To verify that dense insertion of CLA
8

---

Under review as a conference paper at ICLR 2022
modules is not necessary, we perform ablation study on the number of inserted CLA modules Set5 (x2) for single image super-resolution We use EDSR as our backbone where CLA modules evenly inserted. As shown in Table 5 with more CLA modules inserted, the performance slightly improved. However, the computation cost and parameter size of the model are also greatly increased. While with insert positions searched as shown in T model can reach comparable performance with much less computational resources.
be
Table 5: Ablation study on number of inserted CLA modules on Set5
Method
L
PSNR
Params(M) FLOPs(G)
CLA
4
38.24
96.93
42.13
CLA
8
38.27
101.37
44.35
CLA
16
38.27
118.48
51.47
CLA
32
38.26
182.93
79.29
Ablation study on ACLA As explained in Section 3.2
ACLA further improves CLA by two adaptive designs: selecting an adaptive number of keys at each layer for non-local attention and searching for optimal insert positions of ACLA modules. To verify the effectiveness of the two adaptive designs in ACLA, we perform an ablation study on their influence on top of CLA. The comparison is also performed Set5 (x2) for single image super-resolution with EDSR backbone. The results shown in 6 CLA-I stands for CLA is deployed with the search for insert positions as in ACLA. CLA-K stands for CLA that adaptively select sampled keys in ACLA, We can clearly see that both adaptive designs are beneficial to the performance of CLA.
Table 6: Ablation study on the effectiveness of insertion position search and sampled keys selection
Method
Description of Methods
PSNR
FLOPs(G)
Params(M)
CLA
38.24
96.93
42.13
CLA-I
scarch for insert positions
38.27
96.93
42.13
CLA-K
select aggregated keys
38.28
96.87
42.29
ACLA
38.31
96.98
42.29
Number of sampled keys K for CLA and ACLA As discussed before, the key point sampling strategy in CLA plays vital role to reduce the computation cost. To verify that small number of sampled keys K can be sufficient, we perform experiments on CLA with different value of K. The comparison is performed on Set5 (x2) for single image super-resolution with EDSR backbone. Besides, we also compare ACLA with different value K, which is the maximal number of sampled keys. The results are displayed in Table and Table g With increased value of K, the performance of CLA and ACLA does not constantly improve. CLA with K 8 and ACLA with K 16 can already achieve comparable performance to those with larger K. This also consistent with studies (Zhang al. 2015 Elad & Aharon 2006) the power of sparse representation learning for image restoration.
Table 7: Ablation study on number of sam- pled keys in CLA Set5
Table 8: Ablation study on maximum number of sampled keys in ACLA on Set5
Method
K
PSNR
FLOPs(G)
Params(M)
Method
K
PSNR
FLOPs()
Params(M) M)
CLA
4
38.22
96.71
42.09
ACLA
8
38.28
96.78
42.18
CLA
8
38.24
96.93
42.13
ACLA
16
38.31
96.98
42,29
CLA
16
38.25
97.38
42,21
ACLA
32
38.30
97.56
42.41
CLA
32
38.23
97.90
42,39
ACLA
64
38.31
98.03
42.69
CLA
64
38.25
98.92
42.74
ACLA
128
38.29
99.17
43.02
CONCLUSIONS
In this paper, we first propose novel attention module Cross-Layer Attention (CLA) to search for informative keys across different layers for each query feature. We further propose Adaptive CLA, or ACLA, which improves CLA by two adaptive designs: selecting adaptive number of keys at each layer and searching for insert positions of ACLA modules. Experiments on image restoration tasks including single-imag super resolution, image denoising, image compression artifacts reduction and image demosaicing validate the effectiveness and efficiency of CLAA and ACLA.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Namhyuk Ahn, Byungkon Kang, and Kyung-Ah Sohn. Fast, accurate, and lightweight super- resolution with cascading residual network. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 252-268, 2018.
Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoe V Le. Attention augmented convolutional networks. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 22266-3295, 2019.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. 2013. URL httpttttp///ii../// abs/ 0o0s/130..m 3432
Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel. Low-complexity
single-image super- super-resolution based on nonnegative neighbor embedding. 2012.
Antoni Buades, Bartomeu Coll, and J-M Morel. A non-local algorithm for image denoising. In CVPR, 2005.
Hanting Chen, Yunhe Wang. Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp- 12299-12310, 2021.
Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei Liu, and Tat-Seng Chua. Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning. Proceedings of the IEEE conference computer vision and pultern recognition, pp. 5659-5667, 2017.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 764-773, 2017.
Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang. Second-order attention network for single image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 11065-11074, 2019.
Chao Dong, Yubin Deng, Chen Change Loy, and Xiaoou Tang. Compression artifacts reduction by deep convolutional network. In ICCV, 2015a.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. IEEE transactions pattern analysis and machine intelligence, 38(2): 295-307, 2015b.
Michael Elad and Michal Aharon. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions Image processing, 15(12):3736 3745, 2006.
Yuchen Fan, Jiahui Yu, Ding Liu, and Thomas S Huang. Scale-wise convolution for image restoration. arXiv preprint arXiv: 1912.09022, 2019.
Alessandro Foi, Vladimir Katkovník, and Karen Egiazarian. Pointwise shape-adaptive det for
high-quality denoising and deblocking of grayscale and color images. TIP, May 2007.
Yong Guo, Yongsheng Luo, Zhenhao He, Jin Huang, and Jian Chen. Hierarchical neural architecture search for single image super-resse resortresolution. IEEE Signal Processing Letters, 07:1255-1255 2020.
Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita. Deep back- poojecroeettion networks for super-resolution. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1673, 2018.
Jie Hu, Shen, and Gang Sun. Squeeze-and-excinee conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
networks. In Proceedings of the IEEE
Hia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolutie resoluucsolution from transformed self-exemplars. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5197-5206. 2015.
10

---

Under review as a conference paper at ICLR 2022
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1646-1654, 2016.
Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming- Husuan Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. In CVPR, 2017.
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. arXiv preprint arXiv:2108. 10257, 2021.
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah. and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017a.
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE conference computer vision and pattern recognition workshops, pp. 136-144, 2017b.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceedings of the European conference on computer vision (ECCV), pp. 19 19-34, 2018a.
Chenxi Liu, Liang-Chie Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L Yuille, and Li Fei- Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
Ding Liu, Bihan Wen, Yuchen Fan, Chen Change Loy, and Thomas S Huang. Non-local recurrent network for image restoration. In Advances in Neural Information Processing Systems, pp. 1673- 1682, 2018b.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In International Conference on Learning Representations (ICLR), 2018c.
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2, pp. 416-423. IEEE, 2001.
Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-based manga retrieval using mangal09 dataset. Multimedia Tools and Applications, 76(20):218) 81 1-21838, 2017
Yiqun Mei, Yuchen Fan, Yulun Zhang, Jiahui Yu, Yuqian Zhou, Ding Liu, Yun Fu, Thomas Huang, and Honghui Shi. Pyramid attention networks for image restoration. arXiv preprint arXiv:2004. 13824, 2020.
Ben Niu, Weilei Wen, Wenqi Ren, Xiangde Zhang, Lianping Yang, Shuzhen Wang, Kaihao Zhang, Xiaochun Cao, and Haifeng Shen. Single image super-resolution via holistic attention network. In European Conference on Computer Vision, pp. 191-207. Springer, 2020.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International Conference on Machine Learning (ICML), 2018.
Hamid R Sheikh, Zhou Wang, Lawrence Cormack, and Alan C Bovik. Live image quality assessment database release 2 (2005), 2005.
Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu. Memnet: A persistent memory network for image restoration. In Proceedings of the IEEE international conference on computer vision, pp. 4539 4539- 2017.
Yi Tay, Mostafa Dehghani, Vamsi Aribandi, Jai Gupta, Philip Pham, Zhen Qin, Dara Bahri, Da-Cheng Juan, and Donald Metzler. Omninet: Onnidirectional representations from transformers. arXiv preprint arXiv:2 103.0/ 03055 2021.
11

---

Under review as a conference paper at ICLR 2022
Radu Timofte, Eirikur Agustsson, Lue Van Gool, Ming-Hsuae Yang, and Lei Zhang. Ntire 2017 challenge on single image super-resolution: Methods and results. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. pp. 114-125, 2017.
Thomas Verelst and Tinne Tuytelaars. Dynamic convolutions: Exploiting spatial sparsity for faster in- ference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2320-2329, 2020.
Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang. Cheng Li, Honggang Zhang, Xiaogang Wang. and Xiaoou Tang. Residual attention network for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3156-3164, 2017.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7194-8803,
2018.
Lingxi Xie and Alan Yuille. Genetic cnn. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2017.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search. In International Conference on Learning Representations (ICIRR) 2018.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference machine learning, pp. 2048-24420002 2057.. PMLR, 2015.
Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse-
representations. In International conference on curves and surfaces, 2010.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng. and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep enn for image denoising. TIP, 2017a.
Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. Learning deep enn denoiser prior for image restoration. In CVPR, 2017b.
Kai Zhang, Wangmeng Zuo, and Lei Zhang. Learning single convolutional super-resolution network for multiple degradations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3262-3271, 2018a
Xiong Zhang, Hongmin Xu, Hong Mo. Jianchao Tan, Cheng Yang, Lei Wang, and Wenqi Ren. Denas: Densely connected neural architecture search for semantic image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13956-13966 2021.
Y Zhang, K Li, K Li, Zhong, and Fu. Residual non-local attention networks for image restoration. In International Conference on Learning Representations, 2019.
Yulun Zhang, Kumpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super-resolution using very deep residual channel attention networks. Proceedings of the European Conference on Computer Vision (ECCV), pp. 286-301,70 2018b.
Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 286-301, 2018c.
Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2472-2481, 2018d.
Zheng Zhang, Yong Xu, Jian Yang, Xuelong Li, and David Zhang. A survey of sparse representation: algorithms and applications. IEEE access, 3:490-530, 2015.
Maria Zontak, Inbar Mosseri, and Michal Irani. Separating signal from noise using patch recurrence across scales. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1292- 2013.
12

---

Under review as a conference paper at ICLR 2022
Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations (ICLR), 2016.
Daniel Zoran and Yair Weiss. From learning models of natural image patches to whole image restoration. In 2011 International Conference on Computer Vision, pp. 479-486. IEEE, 2011.
13

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 MORE IMPLEMENTATION DETAILS
We summarize the value of X, i.e. hyper- syper-parameter that controls the magnitude of the cost term, for different experiments in Table 9 The insert positions of ACLA in the searched models are also displayed in the same table. Since RCAN constitutes 10 residual groups, we insert ACLA module after each of the residual group in the corresponding super network. For experiments with EDSR, ACLA modules are inserted after each residual block the super network. Note that 16-layer EDSR is used for Image Denoising, Image Demosaicing, and Image Compression Artifacts Reduction.
Table 9: Search settings for ACLA in different experiments
Task
Backbone
Value
Insert Positions
Single-Image Super-Resooution
EDSR
0.15
3, 12, 26, 31, 32
Single-Image Super-Resolution
RCAN
0.3
1,3,5,9
Image Denoising
EDSR
0.2
2, 9, 13,
Image Demosaicing
EDSR
0.2
2. 2, 14, 16
Image Compression Artifacts Reduction
EDSR
0.2
2. 10.
13.0113.10
A.2 IMAGE DEMOSAICING
To demonstrate the effectiivnees of our proposed CLA and ACLA on various image restoration tasks, we add an experiment image demosaicing. The evaluation is conducted on Kodak24, McMaster (Zhang al., 201. BSD68, and Urban 100, following the settings RNAN Zhang et al. 2019). We compare our approach with IRCNN (Zhang et al., 2017b), RNAN (Zhang et al. 2019). and PANet (Mei et al., 2020). We also use 16-layer EDSR as baseline CNN model. As shown in Table 10 our approach yields the best reconstruction result for image demosaicing.
Table 10: Quantitative results on benchmark datasets for image demosaicing
McMaster
Kodak24
BSD68
Urban 100
Method
Params(M)
PSNK
SSIM
FSNN
SSIM
FSNK
SSIM
FSNN
SSIM
Mosaieed
9.17
0.1674
8.36
0.0682
8.43
0.0850
7.48
0.1195
IRCNN
0.731
0.9807
RNAN
7.409
39.71
0.9725
43.09
0.9902
42.50
0.9929
39.75
0.9848
Baseline
5.430
39.81
0.9730
43.18
0.9903
42.66
0.9931
40.23
0.9852
ACLA
5.914
40.08
0.9742
43.38
0.9908
42.90
0.9936
40.55
0.9857
A.3 COMPARISON WITH OTHER ATTENTION METHODS
In our paper, we have compared CLA and ACLA with non-local attention in section 4.5. Here, further compare our proposed ACLA against other forms of attention modules that are widely used in the CV community, including Squeeze-and-Exciian Multi-Head Attention (MHA) (Bello et al.) 2019). SE aim at reweighting the channel-wise responses by using soft self-attention to model interdependencies between the channels of the convolutional features. MHA actually variant of self-attention from the NLP domain. Specifically, MHA can be regarded special kind of non-local attention that takes account of the relative position information. For the experiments with SE and MHA, we insert four SE blocks and MHA blocks evenly in the EDSR backbone, respectively. The comparison is performed on 2x single-image super-resolution following the settings section 4.2. The results are displayed in Table 12 Although MHA and SE bring improvements over the EDSR baseline. The best results achieved by our proposed ACLA. Besides, we also perform an experiment that combines ACLA with SE, where SE blocks are inserted after each ACLA module.
(SE) (Hu et al. 2018) attention and
14

---

Under review as a conference paper at ICLR 2022
Table 11: Efficiency and performance comparison with Squeeze-and-Exciialle (SE) attention and
Multi-Head Attention (MHA)
Methods
Params(M)
FLOPs(G)
Set
14
100
Urban
Manga 100
EDSR
40.73
93.97
38.11
33.92
32.32
32.93
39.10
EDSR MHA
42.17
100.21
38.23
34.01
32.39
33.07
39.29
EDSR + SE
41.79
96.14
38.19
34.03
32.36
33.06
39.22
EDSR ACLA
42.29
96.97
38.31
34.10
32.43
33.35
39.42
EDSR ACLA SE
43.47
99.32
38.33
34.09
32.44
33.38
39.46
A.4 PERFORMANCE COMPARISON WITH OTHER ATTENTION-BASEO METHOOSS
As image restoration is regarded as an ill- ill-posed problem, improving the performance of CNN backbones for image restoration has always been challenging task. Recently, attention methods have been widely used to improve performance. Here, we compare our ACLA with HAN and SAN, which are also attention-basee SAN are based on the previous ofate-of-the-arts (SOTA) method RCAN. In the table below, we compared the improvements of ACLA, HAN, and SAN. The improvements over RCAN the parentheses after the PSNR results. For our ALCA, the percentage comparisons of improvement also calculated. For example, the improvement of ACLA over RCAN on B 5100 is 0.04, which is 400% of the improvement SAN over RCAN. Besides, we have also calculated the improvement by ACLA on the EDSR backbone. As shown in the table, compared HAN and SAN which are competitive baselines representing the recent progress of this literature, our method makes significant improvements in this literature.
methods for single image super-resolution. Both HAN and
listed in
Table 12: Efficiency and performance comparison with previous SOTA methods HAN and SAN for single-image super-resolution
Methods
Inforence
Urbon
Mangal 19.14
RCAN
$2.7
38.27
34.12
32.41
$1.34
HAN
38.9
38.22 (0.00)
34,16 (0.04)
32.41(0.00)
33.35 (0.01)
39.46 (0.02)
(-0.05)
RCAN-ACLA
36.9
38.30 (0.03 75%)
34.15 (0.03 75%))
32.45 (0.04 400%)
33.39 (0.05 500%)
39.49 (0.04/ 200%)
EDSR
16.2
38.11
33.92
32.32
32.93
39.
(0.18)
(0.42)
(0.32)
Besides, we also compare the inference time between our proposed ACLA, HAN, and SAN. The running time is the average of 1000 runs input of size 48 x 48. The running time is evaluated on a single 16G Tesla V100. As shown Table [2 our method EDSR+ACLA achieves even better performance than HAN with much less inference time.
A.5 VISUALIZATION OF SELECTED KEYS BY ACLA
We present more examples on visualization of selected keys by ACLA in Figure demonstrate the superiority of our method in searching for informative keys for query feature. The visualization is based on our results for 2x image super-resolution. Similar to Figure 2, the first shows the positions the keys selected by ACLA with K = 16. For comparison, the positions of keys with top-16 attention weights following the vanilla CLNL attention formulation in Equation (3) is displayed in the second row. From left to the right are the sampled key positions from the 3rd, 12nd, 26th, and 3st resblock.
The visualization results show that ACLA adaptively selects semantically similar keys for the query feature, and its vanilla counterpart CLNL lacks such capability. For instance. in Figure [ the query is from the ear of the elephant on the right side. With ACLA, 60% of the selected keys across are also from the ear of the same elephant. Besides, among the keys selected outside the ear of the same elephant, 5 out of 11 are from the ear of the elephant on the left, which have similar textures as the ear of the elephant on the right. While with CLNL, only 39% of the selected keys are from the the ear of the elephant on the right. Similar observations can also be found in Figure 4 In Figure 4 (a), pick a query point from the frame structure at the top of gate. With ACLA, 90% of the keys selected are distributed on the frame structures the top of gates. While with CLNL, positions from the gates and the frame structure at the balcony are also given high attention weights. Only 61% of the selected keys distributed the frame structures the top of gates, which may limit the power of attention
15

---

Under review as a conference paper at ICLR 2022
modules. Similar observations also be found Figure 4 (b) and Figure 4 (c). In Figure 4 (b), the query is from the bridge in the middle of the image. All the keys selected by ACLA are also from the bridge. In Figure I (c), the query is from the back of yak. Most of the keys selected by ACLA are also located on the body of yaks. While as shown in the second row, CLNL even assigns large attention weights to positions from the grass and the background. Such observations strongly demonstrate the power of ACLA in searching for informative keys across different layers.
Resblock
Resblock
Resblock
Resblock 31
Low
Resblock
Resblock 12
Resblock 26
Resblock 31
b
High
Resblock
Resblock
Resblock
Resblock
(c)
Figure 4: Visualization of selected keys by ACLA
16

---

Under review as a conference paper at ICLR 2022
A.6 VISUAL RESULTS
We present some visual results of our proposed methods for 4x SR with BI degradation model in Figure The visual results of CLA and ACLA are from our experiments with EDSR backbone.
HR
Bicubic
SRCNN
FSRCNN
VDSR
EDSR
RDN
RCAN
CLA
ACLA
PowerPoint
How to Everything
2002
Make poiitit
HR
Bicubic
SRCNN
FSRCNN
VDSR
EDSR
RDN
RCAN
CLA
ACLA
HR
Bicubic
SRCNN
FSRCNN
VDSR
EDSR
RDN
RCAN
CLA
ACLA
Figure 5; Visual comparison for 4x SR with BI degradation model.
17


Reviews:
[{"review_id": "iZ_KJqWcGY6", "reviewer": "Reviewer_6N93", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Attention is a powerful mechanism and has found utility in numerous tasks in both high and low level computer vision as effectively argued in the paper.  Therefore paper is addressing an interesting topic relevant to the ICLR community.\n\nNote however, the idea of exploiting correlations across layers in convolutional networks isn’t new, for example, the papers\n\n•\tChang et al., “EPSNet: Efficient Panoptic Segmentation Network with Cross-layer Attention Fusion,” ACCV 2020 also applies non-local attention at different layers, in a hierarchical network architecture used for image segmentation.\n\n•\tMei et al., “Image Super-Resolution with Cross-Scale Non-Local Attention and Exhaustive Self-Exemplars Mining,” CVPR 2020 from the authors of the PANet (Mei et al., 2020) referenced and compared to in the paper; has similar motivations for image restoration tasks.\n\nHowever, the approach taken in this paper appears to be novel through its use of deformable convolution, gating, and neural architecture search to reduce the computational complexity.  \n\nThe paper is clearly described.  It would be helpful if the paper commented on the availability of source code in order for others to replicate the results.  The approach is fairly generic so the method may find applicability in many tasks beyond image restoration.\n\nExperimental results cover a wide range of image restoration tasks to prove the effectiveness and efficiency of the proposed module.  Although the method often produces the best results, the incremental improvement is arguably small; for example in Table 4 the PSNR increases by 0.2 dB compared to EDSR.  However, the visual results are compelling particularly in the appendix.  It seems somewhat surprising to this reviewer that the PSNR improvement is so small yet visually the results appear to be more pronounced (for example, in Figure 5).  \n\nStrengths: \n•\tWhile cross-layer attention isn’t a new idea, the approach taken in this paper has novelty and provides a fairly generic approach that could be used with different backbones.\n•\tThe method carefully addresses the issue of computational complexity to produce a practical working solution.\n\n•\tComprehensive experiments including quantitate analysis and ablation study shows how the proposed algorithm performs efficiently.\n\n•\tThe paper is clearly written and includes useful figures, e.g. Figure 2.\n\nWeaknesses.\n•\tThe primary disadvantage is the small incremental benefit for some of the steps of the method.  For example, the ablation study in Table 6 shows only a 0.07 dB improvement on Set5 for ACLA compared to CLA.  Given that ACLA requires neural architecture search, dynamic key selection and more it may be a lot of work for limited benefit to squeeze out less than 0.1 dB improvement.\n\nSmall corrections and suggestions:\n\n•\tThe first sentence in the introduction describes degradation as “irreversible”.  But the idea of the image restoration is to restore the degradation.  If the image is restored, isn’t the degradation reversed?  Perhaps this qualifying statement could be removed or clarified. \n\n•\tPage 1 argues that most CNN-based image restoration methods tend to produce smooth results, potentially due to only referring to keys in the same layer in their attention modules.  This seems highly speculative – it would be better to show this is the case or possibly remove this.  It’s well known there is a perception-distoration tradeoff in image restoration; e.g. Blau et al., “The Perception-Distortion Tradeoff,” CVPR 2018 and blurring often results in minimizing distortion using standard losses and can be addressed through generative methods that hallucinate detail at the expense of PSNR.\n\n•\tPage 4, please change “To find for keys” to “To search for keys”\n\n•\tPage 5, please change “ALDA” to “ACLA”\n\n•\tPage 5, please change “top op CLA” to “top of CLA”\n", "summary_of_the_paper": "This paper proposes a novel Cross Layer Attention (CLA) module for image restoration tasks. The CLA module doesn’t look for correlated key pixels within the same layer as other algorithms do; instead query pixels attend to key pixels at previous layers of the network. To reduce computational complexity deformable convolution is used to reduce the number of sampled keys.  To further reduce computational complexity, the paper proposes a modified CLA called Adaptive Cross Layer Attention (ACLA). In ACLA, the number of collected keys for each query is dynamically selected through a gating mechanism.  Further, a neural architecture search method was used to find the proper positions to insert the ACLA modules in the backbone network.  The paper provides comprehensive experiments on image restoration tasks to validate the effectiveness and efficiency of ACLA. Conducted experiments include single image super resolution, image denoising, image demosaicing and image compression artifact reduction.  Contributions are the CLA and ACLA modules designed to efficiently perform attention across layers in a neural network.", "summary_of_the_review": "Overall, this paper address an important problem of performing attention across different layers in a neural network and proposes a novel approach that carefully handles the computational complexity.  The method is demonstrated to work effectively in several image restoration methods.  Numerically the results are only slightly better than competitors, but it does reflect an advance.  This reviewer is somewhere between marginally above and accept for this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "h2bPEmrAb8n", "reviewer": "Reviewer_HQYo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n\n- The proposed method seems quite new, and properly designed.\n\n- The proposed method is applied to various image restoration tasks to show the generality power of the method.\n\nWeakness\n\n- Isn't just nonlocal attention good enough? What is the exact problem of nonlocal attention?  An experimental analysis is needed to determine the specific benefits of using cross-layer nonlocal attention. In particular, what are the advantages of using it for image restoration tasks?\n\n- It would be better to mention differences from LAM more specifically.\n\n- In Table1, why is there no CLA in RCAN? \n\n- In Table1, compared to RDN or HAN, the performance is not very good. How different is the inference time? If the speed is not very fast, I think that the meaning of the proposed method is not great.\n\n- In Table2, and Table3, similar to SR, there is not much difference in performance compared to the existing method, and the baseline model is already quite good. It is interpreted that the effect of the proposed method is not large.\n\n- Why is the order of HAN and RDN in x4 different from x2 and x3?\n\n- In Table 5, the difference in performance according to L is not very large, but on the contrary, FLOP and Pram increase significantly. It seems that the effect of the CLA module is not so great.\n\n- In other ablation studies, various experiments are performed, but there is little difference in performance between each, making meaningful analysis difficult.\n", "summary_of_the_paper": "This paper proposes a cross-layer attention module for the image restoration tasks. Unlike previous conventional non-local attention approaches that find correlated keys within the same layer, the proposed method selects keys across the layers. In order to prevent expensive computational costs, the authors propose adaptive cross-layer attention modules. The proposed method is validated on various image restoration tasks including SR, denoising, demosaicing, and compression.\n", "summary_of_the_review": "- This paper is technically sound and quite new. However, the performance difference according to the proposed method is not significant in most experiments.\n\n- In other words, it is difficult to say that the value of the proposed method is that high. Even technically, I think that difference with the existing methods is not significant. (Of course, the proposed method have different points compared to existing methods)\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Y3ytzQl_nkX", "reviewer": "Reviewer_g1nJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n(1)\tThis work presents cross-layer attention (CLA) modules and adaptive cross-layer attention (ACLA) to model non-local pixel correlations by considering feature correlations among different layers.\n(2)\tExperimental results show that ACLA works well for different image restoration tasks.\nWeaknesses:\n1.\tIn Tables 1, 2, & 3, the authors are suggested to add an experiment to replace CLA with the classical non-local block.\n2.\tIt is unclear why finding the correlated key pixels from previous CNN layers is capable to enhance the image restoration performance.\n3.\tCompared to CLA, the ACLA dynamically selects key pixels by using a NAS. However, according to Table 1, ACLA only slightly improves CLA in terms of super-resolution accuracy. It tends to degrade the effectiveness of the NAS of ACLA. Please discuss it.\n", "summary_of_the_paper": "This work presents cross-layer attention (CLA) modules to find informative keys across different CNN layers for each query feature. Furthermore, an adaptive cross-layer attention (ACLA) is also formulated to dynamically select keys from different CNN layers by using a NAS method. After that, the authors embedded the presented CLA or ACLA modules into EDSR to formulate a deep model for image restoration. Experimental results on several image restoration tasks show that the developed deep model outperforms state-of-the-art methods.", "summary_of_the_review": "Please refer to the weaknesses of the main review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "A9JWCOQRavY", "reviewer": "Reviewer_Xgo2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Extensive experiments are conducted to validate the proposed method both in quality and quantity.\n\n(1)\tThe effective of layer attention is a commonly used method in high-level and low-level computer vision task. There are various layer attention module was proposed in the past three years. So, the main contribution of the paper is lack of novelty. More importantly, the findings of the paper has been general knowledge in computer vision community. The authors may need to focus on more important and valuable problems in image restoration field.\n\n(2)\tThe claim of “However, LAM neglects the difference of spatial positions of features.” is not strictly correct. The authors have used the CSAM to exploit the spatial positions of features in their paper. The authors may also need to compare the proposed to LAM. \n\n(3)\tThe claim of “both IPT and SwinIR requires large-scale datasets for good performance.” is not exactly correct. SwinIR only use a small training set to train their network. \n\n(4)\tMore importantly, in the experiments, there are only the proposed CLA and ACLA results ,which can’t present the comparative experiment with other attention mechanisms. The author may need explain the difference between the proposed CLA and other attention mechanisms in computer vision field.\n\n(5)\tThe equation (13) should be further expained clearly.\n\n(6)\tIn the experiments ,there are lack of params such as single image super-resolution、image denoising and image compression artifacts reduction. Also,the RCAN+CLA may neglect by the authors. \n\n", "summary_of_the_paper": "The paper proposes a Cross-Layer Attention(CLA) module in order to capture the correlations. Between features among different layers. Besides, an Adaptive Cross-Layer Attention (ACLA) is proposed to reduce the computational cost of the Cross-Layer Attention(CLA) module. Lastly,a neural architecture search method is used to find the insert positions of ACLA modules to further improve performance. ", "summary_of_the_review": "Please see the paper weakness. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces an cross-layer attention mechanism for image restoration. To reduce the computational complexity, the framework uses deformable convolutions and an adaptive selection for reducing the number of keys, as well as a neural architecture search. The paper received three borderline reject recommendations and a clear accept. After reading the reviews, responses, and the paper in details, the area chair agrees with Reviewer 6N93 that the paper has some merit. Unfortunately, he/she also agrees with the fact that the proposed framework is quite complicated with many components for a marginal improvement (something that also Reviewer 6N93 has mentioned in the discussion between reviewers). Overall, this points towards rejection, which is the final recommendation of the area chair.

Another point that would be helpful, in case this paper is resubmitted elsewhere, is to release the code for the method, given its complexity.
Decision: Reject