Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work presents cross-layer attention (CLA) modules to find informative keys across different CNN layers for each query feature. Furthermore, an adaptive cross-layer attention (ACLA) is also formulated to dynamically select keys from different CNN layers by using a NAS method. After that, the authors embedded the presented CLA or ACLA modules into EDSR to formulate a deep model for image restoration. Experimental results on several image restoration tasks show that the developed deep model outperforms state-of-the-art methods.
main_review: Strengths:
(1)	This work presents cross-layer attention (CLA) modules and adaptive cross-layer attention (ACLA) to model non-local pixel correlations by considering feature correlations among different layers.
(2)	Experimental results show that ACLA works well for different image restoration tasks.
Weaknesses:
1.	In Tables 1, 2, & 3, the authors are suggested to add an experiment to replace CLA with the classical non-local block.
2.	It is unclear why finding the correlated key pixels from previous CNN layers is capable to enhance the image restoration performance.
3.	Compared to CLA, the ACLA dynamically selects key pixels by using a NAS. However, according to Table 1, ACLA only slightly improves CLA in terms of super-resolution accuracy. It tends to degrade the effectiveness of the NAS of ACLA. Please discuss it.

summary_of_the_review: Please refer to the weaknesses of the main review.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a novel Cross Layer Attention (CLA) module for image restoration tasks. The CLA module doesn’t look for correlated key pixels within the same layer as other algorithms do; instead query pixels attend to key pixels at previous layers of the network. To reduce computational complexity deformable convolution is used to reduce the number of sampled keys.  To further reduce computational complexity, the paper proposes a modified CLA called Adaptive Cross Layer Attention (ACLA). In ACLA, the number of collected keys for each query is dynamically selected through a gating mechanism.  Further, a neural architecture search method was used to find the proper positions to insert the ACLA modules in the backbone network.  The paper provides comprehensive experiments on image restoration tasks to validate the effectiveness and efficiency of ACLA. Conducted experiments include single image super resolution, image denoising, image demosaicing and image compression artifact reduction.  Contributions are the CLA and ACLA modules designed to efficiently perform attention across layers in a neural network.
main_review: Attention is a powerful mechanism and has found utility in numerous tasks in both high and low level computer vision as effectively argued in the paper.  Therefore paper is addressing an interesting topic relevant to the ICLR community.

Note however, the idea of exploiting correlations across layers in convolutional networks isn’t new, for example, the papers

•	Chang et al., “EPSNet: Efficient Panoptic Segmentation Network with Cross-layer Attention Fusion,” ACCV 2020 also applies non-local attention at different layers, in a hierarchical network architecture used for image segmentation.

•	Mei et al., “Image Super-Resolution with Cross-Scale Non-Local Attention and Exhaustive Self-Exemplars Mining,” CVPR 2020 from the authors of the PANet (Mei et al., 2020) referenced and compared to in the paper; has similar motivations for image restoration tasks.

However, the approach taken in this paper appears to be novel through its use of deformable convolution, gating, and neural architecture search to reduce the computational complexity.  

The paper is clearly described.  It would be helpful if the paper commented on the availability of source code in order for others to replicate the results.  The approach is fairly generic so the method may find applicability in many tasks beyond image restoration.

Experimental results cover a wide range of image restoration tasks to prove the effectiveness and efficiency of the proposed module.  Although the method often produces the best results, the incremental improvement is arguably small; for example in Table 4 the PSNR increases by 0.2 dB compared to EDSR.  However, the visual results are compelling particularly in the appendix.  It seems somewhat surprising to this reviewer that the PSNR improvement is so small yet visually the results appear to be more pronounced (for example, in Figure 5).  

Strengths: 
•	While cross-layer attention isn’t a new idea, the approach taken in this paper has novelty and provides a fairly generic approach that could be used with different backbones.
•	The method carefully addresses the issue of computational complexity to produce a practical working solution.

•	Comprehensive experiments including quantitate analysis and ablation study shows how the proposed algorithm performs efficiently.

•	The paper is clearly written and includes useful figures, e.g. Figure 2.

Weaknesses.
•	The primary disadvantage is the small incremental benefit for some of the steps of the method.  For example, the ablation study in Table 6 shows only a 0.07 dB improvement on Set5 for ACLA compared to CLA.  Given that ACLA requires neural architecture search, dynamic key selection and more it may be a lot of work for limited benefit to squeeze out less than 0.1 dB improvement.

Small corrections and suggestions:

•	The first sentence in the introduction describes degradation as “irreversible”.  But the idea of the image restoration is to restore the degradation.  If the image is restored, isn’t the degradation reversed?  Perhaps this qualifying statement could be removed or clarified. 

•	Page 1 argues that most CNN-based image restoration methods tend to produce smooth results, potentially due to only referring to keys in the same layer in their attention modules.  This seems highly speculative – it would be better to show this is the case or possibly remove this.  It’s well known there is a perception-distoration tradeoff in image restoration; e.g. Blau et al., “The Perception-Distortion Tradeoff,” CVPR 2018 and blurring often results in minimizing distortion using standard losses and can be addressed through generative methods that hallucinate detail at the expense of PSNR.

•	Page 4, please change “To find for keys” to “To search for keys”

•	Page 5, please change “ALDA” to “ACLA”

•	Page 5, please change “top op CLA” to “top of CLA”

summary_of_the_review: Overall, this paper address an important problem of performing attention across different layers in a neural network and proposes a novel approach that carefully handles the computational complexity.  The method is demonstrated to work effectively in several image restoration methods.  Numerically the results are only slightly better than competitors, but it does reflect an advance.  This reviewer is somewhere between marginally above and accept for this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a cross-layer attention module for the image restoration tasks. Unlike previous conventional non-local attention approaches that find correlated keys within the same layer, the proposed method selects keys across the layers. In order to prevent expensive computational costs, the authors propose adaptive cross-layer attention modules. The proposed method is validated on various image restoration tasks including SR, denoising, demosaicing, and compression.

main_review: Strength

- The proposed method seems quite new, and properly designed.

- The proposed method is applied to various image restoration tasks to show the generality power of the method.

Weakness

- Isn't just nonlocal attention good enough? What is the exact problem of nonlocal attention?  An experimental analysis is needed to determine the specific benefits of using cross-layer nonlocal attention. In particular, what are the advantages of using it for image restoration tasks?

- It would be better to mention differences from LAM more specifically.

- In Table1, why is there no CLA in RCAN? 

- In Table1, compared to RDN or HAN, the performance is not very good. How different is the inference time? If the speed is not very fast, I think that the meaning of the proposed method is not great.

- In Table2, and Table3, similar to SR, there is not much difference in performance compared to the existing method, and the baseline model is already quite good. It is interpreted that the effect of the proposed method is not large.

- Why is the order of HAN and RDN in x4 different from x2 and x3?

- In Table 5, the difference in performance according to L is not very large, but on the contrary, FLOP and Pram increase significantly. It seems that the effect of the CLA module is not so great.

- In other ablation studies, various experiments are performed, but there is little difference in performance between each, making meaningful analysis difficult.

summary_of_the_review: - This paper is technically sound and quite new. However, the performance difference according to the proposed method is not significant in most experiments.

- In other words, it is difficult to say that the value of the proposed method is that high. Even technically, I think that difference with the existing methods is not significant. (Of course, the proposed method have different points compared to existing methods)
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a Cross-Layer Attention(CLA) module in order to capture the correlations. Between features among different layers. Besides, an Adaptive Cross-Layer Attention (ACLA) is proposed to reduce the computational cost of the Cross-Layer Attention(CLA) module. Lastly,a neural architecture search method is used to find the insert positions of ACLA modules to further improve performance. 
main_review: Extensive experiments are conducted to validate the proposed method both in quality and quantity.

(1)	The effective of layer attention is a commonly used method in high-level and low-level computer vision task. There are various layer attention module was proposed in the past three years. So, the main contribution of the paper is lack of novelty. More importantly, the findings of the paper has been general knowledge in computer vision community. The authors may need to focus on more important and valuable problems in image restoration field.

(2)	The claim of “However, LAM neglects the difference of spatial positions of features.” is not strictly correct. The authors have used the CSAM to exploit the spatial positions of features in their paper. The authors may also need to compare the proposed to LAM. 

(3)	The claim of “both IPT and SwinIR requires large-scale datasets for good performance.” is not exactly correct. SwinIR only use a small training set to train their network. 

(4)	More importantly, in the experiments, there are only the proposed CLA and ACLA results ,which can’t present the comparative experiment with other attention mechanisms. The author may need explain the difference between the proposed CLA and other attention mechanisms in computer vision field.

(5)	The equation (13) should be further expained clearly.

(6)	In the experiments ,there are lack of params such as single image super-resolution、image denoising and image compression artifacts reduction. Also,the RCAN+CLA may neglect by the authors. 


summary_of_the_review: Please see the paper weakness.
review_title: null

Meta-review: The paper introduces an cross-layer attention mechanism for image restoration. To reduce the computational complexity, the framework uses deformable convolutions and an adaptive selection for reducing the number of keys, as well as a neural architecture search. The paper received three borderline reject recommendations and a clear accept. After reading the reviews, responses, and the paper in details, the area chair agrees with Reviewer 6N93 that the paper has some merit. Unfortunately, he/she also agrees with the fact that the proposed framework is quite complicated with many components for a marginal improvement (something that also Reviewer 6N93 has mentioned in the discussion between reviewers). Overall, this points towards rejection, which is the final recommendation of the area chair.

Another point that would be helpful, in case this paper is resubmitted elsewhere, is to release the code for the method, given its complexity.
Decision: Reject