Year: 2021

Abstract:
Non-local attention module has been proven to be crucial for image restoration. Conventional non-local attention processes features of each layer separately, so it risks missing correlation between features among different layers. To address this problem, we propose Cross-Layer Attention (CLA) module in this paper. Instead of finding correlated key pixels within the same layer, each query pixel is allowed to attend to key pixels at previous layers of the network. In order to mitigate the expensive computational cost of such hierarchical attention design, only a small fixed number of keys can be selected for each query from a previous layer. We further propose a variant of CLA termed Adaptive Cross-Layer Attention (ACLA). In ACLA, the number of keys to be aggregated for each query is dynamically selected. A neural architecture search method is used to find the insert positions of ACLA modules to render a compact neural network with compelling performance. Extensive experiments on image restoration tasks including single image superresolution, image denoising, image demosaicing and image compression artifacts reduction validate the effectiveness and efficiency of ACLA.

heading: 1 INTRODUCTION
text: Image restoration algorithms aim to recover a high-quality image from a contaminated input image by solving an ill-posed image restoration problem. There are various image restoration tasks depending on the type of corruptions, such as image denoising (Zhang et al., 2019; Liu et al., 2018b), demosaicing (Zhang et al., 2017b; 2019), single image super-resolution (Fan et al., 2019; Lai et al., 2017; Tai et al., 2017), and image compression artifacts reduction(Zhang et al., 2017a). To restore corrupted information from the contaminated image, a variety of image priors (Buades et al., 2005; Zoran & Weiss, 2011; Zontak et al., 2013) were proposed.
Recently, image restoration methods based on deep neural networks have achieved great success. Inspired by the widely used non-local prior, most recent approaches based on neural networks (Zhang et al., 2019; Liu et al., 2018b) adapt non-local attention into their neural network to enhance the representation learning, following the non-local neural networks (Wang et al., 2018). In a nonlocal block, a response is calculated as a weighted sum over all pixel-wise features on the feature map to account for long-range information. Such a module was initially designed for high-level recognition tasks such as image classification, and it has been proven to be beneficial for low-level vision tasks (Zhang et al., 2019; Liu et al., 2018b).
Though attention modules have been shown to be effective in boosting the performance. Most attention modules only explore the correlation among features at the same layer. Actually, features at different intermediate layers encode variant information at different scales, and might be helpful to augment the information used in recovering the high-quality image. Motivated by the potential benefit of exploring feature correlation across intermediate layers, Holistic Attention Network (HAN) (Niu et al., 2020) is proposed to find interrelationship among features at hierarchical levels with a Layer Attention Module (LAM). However, LAM assigns a single importance weight to all features at the same layer and neglects the difference of spatial positions of these features. Recent research in omnidirectional representation (Tay et al., 2021) suggests that exploring the relationship among features at different layers can benefit the representation learning of neural networks. Nevertheless, calculating correlation among features at hierarchical layers is computationally expensive due to the quadratic complexity of dot product attention. The complexity of such cross-layer attention design is
increased from (HW )2L to (HWL)2, where H,W are the height and width of the feature map and L is the number of layers.

heading: 1.1 CONTRIBUTIONS
text: Our contributions are listed as follows.
First, in order to address the problem caused by only referring to keys within the same layer in most attention modules, we propose a novel attention module termed Cross-Layer Attention (CLA), which searches for keys across different layers for each query feature. With the help of the deformation mechanism, CLA only attends to a small set of keys at different layers for each query feature.
Second, we propose an improved CLA termed Adaptive Cross-Layer Attention (ACLA) which selects an adaptive number of keys at each layer for each query, and searches for the optimal insert positions of ACLA modules. We deploy ACLA modules on commonly used neural network models, e.g. EDSR (Lim et al., 2017a), for image restoration. Extensive experiments on single image super-resolution, image denoising, image compression artifacts reduction, and image demosaicing demonstrate the effectiveness of our approach.

heading: 2 RELATED WORKS

heading: 2.1 NEURAL NETWORKS FOR IMAGE RESTORATION
text: Adopting neural networks for image restoration has achieved great success by utilizing their power in representation. ARCNN (Dong et al., 2015a) was first proposed to use CNN for compression artifacts reduction. Later, DnCNN (Zhang et al., 2017a) uses residual learning and batch normalization to boost performance of CNN for image denoising. In IRCNN (Zhang et al., 2017b), a learned set of CNNs are used as denoising prior for other image restoration tasks. For single image superresolution(Lai et al., 2017; Zhang et al., 2018d; Haris et al., 2018; Fan et al., 2019), even more efforts have been devoted to designing advanced architectures and learning methods. For example, RDN (Zhang et al., 2018d) and CARN (Ahn et al., 2018) fuse low-level and high-level features with dense connection to provide richer information and details for reconstructing. Recently, non-local attention (Liu et al., 2018b; Dai et al., 2019; Mei et al., 2020) is also used to further boost the peformance of CNN for image restoration.

heading: 2.2 ATTENTION MECHANISM
text: Attention mechanism has been applied to many computer vision tasks, such as image captioning (Xu et al., 2015; Chen et al., 2017) and image classification (Hu et al., 2018; Wang et al., 2017). Non-local attention (Wang et al., 2018) was first proposed to capture long-range dependencies for high-level recognition tasks. Recently, several works propose to leverage non-local attention for low-level vision tasks. In NLRN (Liu et al., 2018b) a recurrent neural network is proposed to incorporate non-local attention. RNAN (Zhang et al., 2019) proposed a residual local and non-local mask branch to obtain non-local mixed attention. RCAN (Zhang et al., 2018c) exploits the interdependencies among feature channels by generating different attention for each channel-wise feature. HAN (Niu et al., 2020) is proposed to find interrelationship among features at hierarchical levels with a layer attention module. Besides, some recent works attempt to explore the benefits of transformer based models for image restoration. IPT (Chen et al., 2021) is proposed to solve various restoration problems in a multi-task learning framework based on visual Transformer. SwinIR (Liang et al., 2021) adopt the architecture of Swin Transformer. However, compared with methods using CNN architecture, transformer-based image restoration methods usually use large datasets for training. Specifically, IPT uses ImageNet to pretrain the model. SwinIR adapts a combination of four datasets consisting of over 8000 high-quality images as training set for the tasks of denoising and compression artifact reduction.

heading: 2.3 NEURAL ARCHITECTURE SEARCH
text: Neural Architecture Search (NAS) has attracted lots of attention recently. Early works of NAS adopt heuristic methods such as reinforcement learning (Zoph & Le, 2016) and evolutionary algorithm (Xie & Yuille, 2017). The search process with such methods requires huge computational resources. Recently, various strategies are designed to reduce the expensive costs including weight sharing (Pham
et al., 2018), progressive search (Liu et al., 2018a) and one-shot search (Liu et al., 2018c; Xie et al., 2018). For example, DARTS (Liu et al., 2018c) firstly relaxes the search space to be continuous and conducts differentiable search. The architecture parameters and network weights are trained simultaneously by gradient descent to reduce the search time.
Despite the success of NAS methods for classification, dense prediction tasks such as semantic image segmentation and image restoration, usually demand more complicate network architectures. Some recent works have been devoted to explore hierarchical search space for dense prediction tasks. For example, Auto-DeepLab (Liu et al., 2019) introduces a hierarchical search space for semantic image segmentation. DCNAS (Zhang et al., 2021) build a densely connected search space to extract multi-level information. HNAS (Guo et al., 2020) also adopts a hierarchical search space for single image super-resolution.

heading: 3 METHOD

heading: 3.1 CLA: CROSS-LAYER ATTENTION
text: Non-Local Attention. Non-local attention (Wang et al., 2018) is designed to integrate self-attention mechanism into convolutional neural networks for computer vision tasks. It is usually applied on an input feature map x ∈ RH×W×C to explore self similarities among all spatial positions of that feature map. For illustration purpose, we reshape x to N × C, where N = H ×W . A generic non-local attention can be formulated as
yi = 1
C(x) N∑ n=1 f(xi, xn)g(xn), (1)
where i indexes the spatial position of feature maps. y is the output of non-local attention with the same size as x. f(xi, xn) outputs the pairwise affinity between the query feature xi and its key feature xn. g(xn) computes an embedding of feature xn. C(x) is a normalization term.
Non-local attention in Equation (1) is usually wrapped into a non-local block (Wang et al., 2018) with a residual connection from the input feature x. The mathematical formulation is given as
z = h(y) + x, (2)
where h denotes a learnable feature transformation.
Cross-Layer Attention. To search for keys from different layers for each query feature, we propose Cross-Layer Attention (CLA). To derive the formulation of CLA, we start by adapting non-local attention in Equation (1) to a cross-layer design. Suppose that xi is the output of the i-th layer in a CNN backbone for image restoration, where i ∈ {1, . . . , L} and L is the number of layers. A non-local attention with cross-layer design can be formulated as
yji = 1
C(xj) j∑ l=1 N∑ n=1 f(xji , x l n)g(x l n), (3)
where the superscripts j, l index the layer and the subscripts i, n index the spatial locations of features. y, x denote the output feature and input feature respectively. Query feature in this formulation can thus refer to features in previous layers. However, given the quadratic complexity of correlation computation, the complexity of such adaption of non-local attention is increased fromN2L to (NL)2. In order to mitigate the expensive computation cost, we leverage deformable mechanism proposed in deformable convolution (DCN) (Dai et al., 2017) to design our proposed CLA as
yji = 1
C(xj) j∑ l=1 K∑ k=1 f(xji , x l(pi + ∆pik))g(x l(pi + ∆pik)), (4)
where k indexes the sampled keys. pi represents the 2-d spatial position of the query in the feature map, and ∆pik is the 2-d offset from the position of query pi to the corresponding sampled key. As pi + ∆pik can be fractional, bilinear interpolation is used as in (Dai et al., 2017) to compute x(pi+∆pik). To further reduce the computation complexity in computing attention weights, attention weights can be generated from query feature alone. Thus, our proposed CLA can be simplified as
yji = 1
C(xj) j∑ l=1 K∑ k=1 f(xji )g(x l(pi + ∆pik)), (5)
where f is the function to generate attention weights from the query feature. Similar to the design for non-local attention module in (Wang et al., 2018), when deploying CLA in CNN backbones, we also wrap it into a non-local block with residual connection.
With our proposed CLA, each query feature now can refer to a fixed number of keys from each previous layer. However, query features at different spatial positions may have different preferences on keys sampled from different layers. Especially for image restoration tasks, the restoration process at different spatial position may vary significantly due to the diversity of textures in an image. Besides, when deploying CLA in CNN backbones for image restoration, we find that increasing the number of inserted CLA will not constantly improve the performance, but result in much heavier computation cost. This observation motivates us to search for an optimal configuration of deploying CLA in neural networks to efficiently explore its representation learning capacity.

heading: 3.2 ACLA: ADAPTIVE CROSS-LAYER ATTENTION
text: To achieve adaptive key selection and the search for the optimal configuration of CLA, we propose Adaptive Cross-Layer Attention (ACLA). Specifically, for each query feature, we dynamically search for the keys sampled from previous layers with ACLA. Besides, when deploying ACLA in CNN backbones, a neural architecture search method is used to search for the insert positions of ACLA. An objective based on the computation cost of inserted ACLA modules is used to supervise the search procedure.
Adaptive Cross-Layer Attention. To adaptively search for the informative sampled keys for a query feature from its previous layers, we first follow the method in CLA to get a fixed number of sampled keys from each layer. Next, a hard gating mask is applied on each of the sampled key as
yji = 1
C(xj) j∑ l=1 K∑ k=1 mki,lf(x j i )g(x l(pi + ∆pik)), (6)
wheremki,l is the hard gating mask for the k-th sampled key from x l for query feature xji , whose value is either 1 or 0. We further relax the hard gating mask to continuous domain with the simplified binary Gumbel-Softmax (Verelst & Tuytelaars, 2020). Therefore, the hard gating mask mki,l is approximated by
m̂ki,l = σ (βki,l + ki,l,1 − ki,l,2
τ
) , (7)
with sampling parameter βki,l, Gumbel noise k i,l,1, k i,l,2, and temperature τ . σ is the sigmoid function. The sampling parameter βki,l can be regarded as a soft gating mask. To make the gating decisions on sampled keys input-dependent, a mask unitM is used to generate the soft gating mask β from the features of the sampled keys as
βkj,l =M(xl(pi + ∆pik)). (8)
Following the design in (Verelst & Tuytelaars, 2020), a 1× 1 convolution layer is used as the mask unitM in our model. To train the mask unitM, the straight-through estimator from (Bengio et al., 2013; Verelst & Tuytelaars, 2020) is used for mki,l. In particular, during backward pass, we set mki,l = m̂ k i,l, while in forward pass we set
mki,l =
{ 1 m̂ki,l > 0.5,
0 m̂ki,l ≤ 0.5. (9)
Besides, Gumbel noise ki,l,1 and k i,l,2 are set to 0 during inference. To demonstrate the effectiveness of adaptive key selection in ACLA, we visualized keys selected by ACLA at different layers for a query feature in Figure 2
Insert Positions. To search for the insert positions of ACLA, we first densely insert ACLA after each layer of the CNN backbone as in Figure 3 to build the supernet. Similar to the gating formulation in ACLA, we define a hard decision parameter sj ∈ {0, 1} for the j-th inserted ACLA in the supernet. sj = 1 indicates that a ACLA module is inserted after the j-th layer, and otherwise for sj = 0. The simplified binary Gumbel-Softmax (Verelst & Tuytelaars, 2020) can also be used here to approximate the hard decision parameter sj by
sj = σ (αj + j1 − j2
τ
) , (10)
with sampling parameter αj , Gumbel noise , and temperature τ . Thus, the output of ACLA in the supernet can be expressed as
yji = 1
C(xj) j∑ l=1 sl K∑ k=1 mki,lf(x j i )g(x l(pi + ∆pik)). (11)
Different from data-dependent design of the gating mask in ACLA, αj here is defined as architecture parameter and can be directly optimized by stochastic gradient descent (SGD) during the search process. By gradually decrease the value of temperature, αj will be optimized such that sj will approach 1 or 0.
Search Procedure. We propose to optimize both the accuracy and the computation cost (FLOPs) of the ACLA modules inserted. Thus, the cost of the ACLA modules inserted needs to be estimated during the search phase. Following the formulation of the ACLA in the supernet, we can estimate the cost of the ACLA inserted after the j-th residual block as
costj = j∑ l=1 sl K∑ k=1 (2mkj,lNC 2 + 2NC2 + 6KNC), (12)
where N is the number of spatial positions. C is the number of channels. K is the maximal number of sampled keys. 2mkj,lNC 2 is the FLOPs for the convolution on generating the gating masks. 2NC2 is the FLOPs for the convolution on sampling keys. 6KNC is the FLOPs for the aggregation process. Then we can get the estimation on the computation cost of all inserted ACLA modules as
cost = L∑ j=1 sjcostj , (13)
which is the summation of the cost of each layer, weighted by corresponding decision parameters. As mentioned before, with the help of the continuously relaxed representation in the supernet, we can search for the architecture by updating the architecture parameters using SGD. To supervise the search process, we design a loss function with the cost-based regularization to achieve the multi-objective optimization: L(w,α) = LMSE + λ log cost, (14) where λ is the hyper-parameters that controls the magnitude of the cost term.
We find that at the beginning of the search process, ACLA modules inserted at shallow layers are more likely to be strengthened. To solve this problem, we split our search procedure into two stages. In the first stage, we only optimize the parameters of the network for enough epochs to get network weights sufficiently trained. In the second stage, we activate the architecture optimization. We alternatively optimize the network weights by descending∇wLtrain(w,α) on the training set, and optimize the architecture parameters by descending∇αLval(w,α) on the validation set. When the search procedure terminates, we derive the insert positions based on the architecture parameters α.

heading: 4 EXPERIMENTS

heading: 4.1 IMPLEMENTATION DETAILS
text: We select DIV2K (Timofte et al., 2017) as the training set for our experiments as like in (Dai et al., 2019; Zhang et al., 2018d), which include 800 images for training and 100 images for validation. We augment the training images by randomly rotating 90◦, 180◦, 270◦and horizontally flipping. In each
mini-batch, 16 low-quality patches with size 48×48 are provided as inputs. ADAM optimizer is used for both the search phase and training phase. Default values of β1 and β2 are set as 0.9 and 0.999, respectively, and we set = 10−8. Before the search, we perform a cross-validation on 20% of the training data to decide the value of λ. In search phase, the learning rate is initialized as 10−4 and cosine learning rate schedule is used. The search process takes 200 epochs. In the training phase, the learning rate is initialized as 10−4 and then reduced to half every 200 epochs. The model is trained for 1000 epochs in total. K = 8 is sef for all CLA modules by default. For all ACLA modules, the value of K are initialized as 16.

heading: 4.2 SINGLE IMAGE SUPER-RESOLUTION
text: We test our methods on top of the widely used super-resolution backbone EDSR (Lim et al., 2017b) and RCAN (Zhang et al., 2018b). The LR images are obtained by the bicubic downsampling of HR images. Our methods are evaluated on five standard datasets: Set5 (Bevilacqua et al., 2012), Set14 (Zeyde et al., 2010), B100 (Martin et al., 2001), Urban100 (Huang et al., 2015), and Manga109 (Matsui et al., 2017). The reconstructed results by our model are first converted to YCbCr space, PSNR and SSIM in the luminance channel is calculated in our experiments. We compare our methods with 6 other methods: SRCNN (Dong et al., 2015b), VDSR (Kim et al., 2016), MemNet (Tai et al., 2017), SRMDNF (Zhang et al., 2018a), RDN (Zhang et al., 2018d), and HAN (Niu et al., 2020). The quantitative results are shown in Table 1. Some visual results can be found in Appendix A.6. Our methods greatly improves the performance of EDSR and RCAN on all benchmarks and all upsampling scales. Besides, our methods outperform current state-of-the-art method HAN, which is also an attention based method, demonstrating the superiority of our proposed attention modules.

heading: 4.3 IMAGE DENOISING
text: We evaluate our proposed CLA and ACLA module on standard benchmarks for image denoising: KCLDAk24, BSD68 (Martin et al., 2001), and Urban100 (Huang et al., 2015). The noisy images
are created by adding AWGN noises with σ = 10, 30, 50, 70. We compare our approach with 4 methods, namely DnCNN (Zhang et al., 2017a), MemNet (Tai et al., 2017), RNAN (Zhang et al., 2019), and PANet (Mei et al., 2020). For fair comparisons, a 16-layer EDSR is used as the baseline CNN backbone. As shown in Table 2, our methods are also beneficial for representation learning with neural network on image denoising.

heading: 4.4 IMAGE COMPRESSION ARTIFACTS REDUCTION
text: For the task of image compression artifacts reduction (CAR), we compare our method with 3 approaches: DnCNN (Zhang et al., 2017a), RNAN (Zhang et al., 2019), and PANet (Mei et al., 2020). All methods are evaluated on LIVE1 (Sheikh et al., 2005) and Classic5 (Foi et al., 2007). To obtain the low-quality compressed images, we follow the standard JPEG compression process and use Matlab JPEG encoder with quality q = 10, 20, 30, 40. For fair comparison, the results are only evaluated on Y channel in YCbCr Space. The quantitative results are shown in Table 3. 16-layer EDSR is used as our baseline CNN backbone. Both CLA and ACLA improves the performance of the neural network baseline.

heading: 4.5 ABLATION STUDY AND DISCUSSION
text: ACLA vs. Non-local attention To verify the effectiveness of our proposed methods, we compare CLA and ACLA with Non-Local attention (Wang et al., 2018) and vanilla Cross-Layer Non-Local attention in terms of computational efficiency and performance. The vanilla Cross-Layer Non-Local attention follows the formulation in Equation (3). The comparison is performed on Set5 for single image super-resolution with EDSR backbone. The Non-Local attention modules and vanilla CrossLayer Non-Local attention modules are inserted evenly after every 8th residual blocks. All the FLOPs in our ablation study are calculated for input of size 48× 48. Results are presented in Table 4, where NL stands for Non-Local attention and CLNL stands for vanilla Cross-Layer Non-Local. As shown in Table 4, with less computation cost, CLA and ACLA achieve much better performance compared to standard Non-Local attention module.
Number of inserted CLA modules As stated before, the computation cost of the cross-layer design in CLA is quadratic to the number of inserted CLA modules. To verify that dense insertion of CLA
modules is not necessary, we perform an ablation study on the number of inserted CLA modules on Set5 (×2) for single image super-resolution. We use EDSR as our backbone where L CLA modules are evenly inserted. As shown in Table 5, with more CLA modules inserted, the performance can be slightly improved. However, the computation cost and parameter size of the model are also greatly increased. While with insert positions searched as shown in Table 6, our model can reach comparable performance with much less computational resources.
Ablation study on ACLA As explained in Section 3.2, ACLA further improves CLA by two adaptive designs: selecting an adaptive number of keys at each layer for non-local attention and searching for optimal insert positions of ACLA modules. To verify the effectiveness of the two adaptive designs in ACLA, we perform an ablation study on their influence on top of CLA. The comparison is also performed on Set5 (×2) for single image super-resolution with EDSR backbone. The results are shown in Table 6. CLA-I stands for CLA is deployed with the search for insert positions as in ACLA. CLA-K stands for CLA that adaptively select sampled keys as in ACLA. We can clearly see that both adaptive designs are beneficial to the performance of CLA.
Number of sampled keys K for CLA and ACLA As discussed before, the key point sampling strategy in CLA plays a vital role to reduce the computation cost. To verify that a small number of sampled keys K can be sufficient, we perform experiments on CLA with different value of K. The comparison is performed on Set5 (×2) for single image super-resolution with EDSR backbone. Besides, we also compare ACLA with different value K, which is the maximal number of sampled keys. The results are displayed in Table 7 and Table 8. With increased value of K, the performance of CLA and ACLA does not constantly improve. CLA with K = 8 and ACLA with K = 16 can already achieve comparable performance to those with larger K. This is also consistent with studies (Zhang et al., 2015; Elad & Aharon, 2006) on the power of sparse representation learning for image restoration.

heading: 5 CONCLUSIONS
text: In this paper, we first propose a novel attention module Cross-Layer Attention (CLA) to search for informative keys across different layers for each query feature. We further propose Adaptive CLA, or ACLA, which improves CLA by two adaptive designs: selecting adaptive number of keys at each layer and searching for insert positions of ACLA modules. Experiments on image restoration tasks including single-image super resolution, image denoising, image compression artifacts reduction and image demosaicing validate the effectiveness and efficiency of CLA and ACLA.

heading: A APPENDIX
text: A.1 MORE IMPLEMENTATION DETAILS
We summarize the value of λ, i.e. hyper-parameter that controls the magnitude of the cost term, for different experiments in Table 9. The insert positions of ACLA in the searched models are also displayed in the same table. Since RCAN constitutes 10 residual groups, we insert ACLA module after each of the residual group in the corresponding super network. For experiments with EDSR, ACLA modules are inserted after each residual block in the super network. Note that 16-layer EDSR is used for Image Denoising, Image Demosaicing, and Image Compression Artifacts Reduction.
A.2 IMAGE DEMOSAICING
To demonstrate the effectiveness of our proposed CLA and ACLA on various image restoration tasks, we add an experiment on image demosaicing. The evaluation is conducted on Kodak24, McMaster (Zhang et al., 2017b), BSD68, and Urban100, following the settings in RNAN (Zhang et al., 2019). We compare our approach with IRCNN (Zhang et al., 2017b), RNAN (Zhang et al., 2019), and PANet (Mei et al., 2020). We also use 16-layer EDSR as the baseline CNN model. As shown in Table 10, our approach yields the best reconstruction result for image demosaicing.
A.3 COMPARISON WITH OTHER ATTENTION METHODS
In our paper, we have compared CLA and ACLA with non-local attention in section 4.5. Here, we further compare our proposed ACLA against other forms of attention modules that are widely used in the CV community, including Squeeze-and-Excitation (SE) (Hu et al., 2018) attention and Multi-Head Attention (MHA) (Bello et al., 2019). SE aim at reweighting the channel-wise responses by using soft self-attention to model interdependencies between the channels of the convolutional features. MHA is actually a variant of self-attention from the NLP domain. Specifically, MHA can be regarded as a special kind of non-local attention that takes account of the relative position information. For the experiments with SE and MHA, we insert four SE blocks and MHA blocks evenly in the EDSR backbone, respectively. The comparison is performed on 2x single-image super-resolution following the settings in section 4.2. The results are displayed in Table 12. Although MHA and SE bring improvements over the EDSR baseline. The best results are achieved by our proposed ACLA. Besides, we also perform an experiment that combines ACLA with SE, where SE blocks are inserted after each ACLA module.
A.4 PERFORMANCE COMPARISON WITH OTHER ATTENTION-BASED METHODS
As image restoration is regarded as an ill-posed problem, improving the performance of CNN backbones for image restoration has always been a challenging task. Recently, attention methods have been widely used to improve performance. Here, we compare our ACLA with HAN and SAN, which are also attention-based methods for single image super-resolution. Both HAN and SAN are based on the previous state-of-the-arts (SOTA) method RCAN. In the table below, we compared the improvements of ACLA, HAN, and SAN. The improvements over RCAN are listed in the parentheses after the PSNR results. For our ALCA, the percentage comparisons of improvement are also calculated. For example, the improvement of ACLA over RCAN on B100 is 0.04, which is 400% of the improvement of SAN over RCAN. Besides, we have also calculated the improvement by ACLA on the EDSR backbone. As shown in the table, compared to HAN and SAN which are competitive baselines representing the recent progress of this literature, our method makes significant improvements in this literature.
Besides, we also compare the inference time between our proposed ACLA, HAN, and SAN. The running time is the average of 1000 runs on input of size 48 × 48. The running time is evaluated on a single 16G Tesla V100. As shown Table 12, our method EDSR+ACLA achieves even better performance than HAN with much less inference time.
A.5 VISUALIZATION OF SELECTED KEYS BY ACLA
We present more examples on visualization of selected keys by ACLA in Figure 4 to demonstrate the superiority of our method in searching for informative keys for query feature. The visualization is based on our results for 2× image super-resolution. Similar to Figure 2, the first row shows the positions of the keys selected by ACLA with K = 16. For comparison, the positions of keys with top-16 attention weights following the vanilla CLNL attention formulation in Equation (3) is displayed in the second row. From left to the right are the sampled key positions from the 3rd, 12nd, 26th, and 31st resblock.
The visualization results show that ACLA adaptively selects semantically similar keys for the query feature, and its vanilla counterpart CLNL lacks such capability. For instance, in Figure 2, the query is from the ear of the elephant on the right side. With ACLA, 60% of the selected keys across are also from the ear of the same elephant. Besides, among the keys selected outside the ear of the same elephant, 5 out of 11 are from the ear of the elephant on the left, which have similar textures as the ear of the elephant on the right. While with CLNL, only 39% of the selected keys are from the the ear of the elephant on the right. Similar observations can also be found in Figure 4. In Figure 4 (a), we pick a query point from the frame structure at the top of a gate. With ACLA, 90% of the keys selected are distributed on the frame structures at the top of gates. While with CLNL, positions from the gates and the frame structure at the balcony are also given high attention weights. Only 61% of the selected keys are distributed on the frame structures at the top of gates, which may limit the power of attention
modules. Similar observations can also be found Figure 4 (b) and Figure 4 (c). In Figure 4 (b), the query is from the bridge in the middle of the image. All the keys selected by ACLA are also from the bridge. In Figure 4 (c), the query is from the back of a yak. Most of the keys selected by ACLA are also located on the body of yaks. While as shown in the second row, CLNL even assigns large attention weights to positions from the grass and the background. Such observations strongly demonstrate the power of ACLA in searching for informative keys across different layers.
A.6 VISUAL RESULTS
We present some visual results of our proposed methods for 4× SR with BI degradation model in Figure 5. The visual results of CLA and ACLA are from our experiments with EDSR backbone.

Review summaries:
This work presents cross-layer attention (CLA) modules to find informative keys across different CNN layers for each query feature. Furthermore, an adaptive cross-layer attention (ACLA) is also formulated to dynamically select keys from different CNN layers by using a NAS method. After that, the authors embedded the presented CLA or ACLA modules into EDSR to formulate a deep model for image restoration. Experimental results on several image restoration tasks show that the developed deep model outperforms state-of-the-art methods.

This paper proposes a novel Cross Layer Attention (CLA) module for image restoration tasks. The CLA module doesn’t look for correlated key pixels within the same layer as other algorithms do; instead query pixels attend to key pixels at previous layers of the network. To reduce computational complexity deformable convolution is used to reduce the number of sampled keys.  To further reduce computational complexity, the paper proposes a modified CLA called Adaptive Cross Layer Attention (ACLA). In ACLA, the number of collected keys for each query is dynamically selected through a gating mechanism.  Further, a neural architecture search method was used to find the proper positions to insert the ACLA modules in the backbone network.  The paper provides comprehensive experiments on image restoration tasks to validate the effectiveness and efficiency of ACLA. Conducted experiments include single image super resolution, image denoising, image demosaicing and image compression artifact reduction.  Contributions are the CLA and ACLA modules designed to efficiently perform attention across layers in a neural network.

This paper proposes a cross-layer attention module for the image restoration tasks. Unlike previous conventional non-local attention approaches that find correlated keys within the same layer, the proposed method selects keys across the layers. In order to prevent expensive computational costs, the authors propose adaptive cross-layer attention modules. The proposed method is validated on various image restoration tasks including SR, denoising, demosaicing, and compression. 

The paper proposes a Cross-Layer Attention(CLA) module in order to capture the correlations. Between features among different layers. Besides, an Adaptive Cross-Layer Attention (ACLA) is proposed to reduce the computational cost of the Cross-Layer Attention(CLA) module. Lastly,a neural architecture search method is used to find the insert positions of ACLA modules to further improve performance.