Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel method for cross-layer interaction, which complements current mainstream networks emphasizing the interaction within a layer. Taking advantage of the attention mechanism, the proposed method enhances the layer interaction via attention. An efficient implementation is also introduced to avoid the vanilla quadratic complexity. Experimental results demonstrate the effectiveness of the proposed method.
strength_and_weaknesses: [ Strength ]
+ The motivation is very attractive. It is common that most of existing networks, including Transformers, only focus on the interaction within a certain layer. Even though ResNet and DenseNet, as analyzed by the authors, put some emphasis on layer interaction, the way they used (i.e., addition and/or concatenation) is a little bit naive and hard. The proposed method takes advantage of the attention mechanism and makes the cross-layer interaction more technically rational. 
+ The method is elegant and the presentation logic is clear. It is quite straightforward to understand each part of the method section. The proposed efficient implementation of layer attention, just like linear attention in efficient Transformers, is useful and necessary. 
+ Sufficient experiments demonstrate the effectiveness of the proposed method. The performance on image classification, object detection and instance segmentation indicates that the method is indeed effective and efficient. 

[ Weakness ]
- On image classification, the input resolution is identically set as 224. What if enlarging the resolution? In fact, it is quite important to compare the accuracy and efficiency with various resolutions to reveal the robustness. Of course, it is necessary to check whether the complexity is linear (or nearly linear) to the input resolution.
- Visual comparison is missing. The proposed cross-layer attention can enhance the interaction between layers, but there is no strong evidence specially for this interaction. For example, are local features from shallow layers transferred to top layers? How do high-level features facilitate the feature representation in shallow layers? These would need visual comparison.
clarity,_quality,_novelty_and_reproducibility: The paper is of high quality including its clear motivation, good novelty and originality, and logical presentation.
summary_of_the_review: I think the paper is quite good, and I recommend to accept the paper if the authors can address my two primary concerns above. Also, I would like to have further communication with other reviewers and the AC.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a module to perform layer attention for CNNs and Vision Transformers in linear-complexity. The core idea is to leverage the representations of previous layers in a recurrent form so that the quadratic-complexity self-attention is avoided. Experiments on image classification, object detection, and instance segmentation are performed to validate the effectiveness of the proposed module.
strength_and_weaknesses: Strength
- The core idea is straightforward to implement.
- A wide range of experiments (image classification, object detection, and instance segmentation) are performed on large-scale datasets including ImageNet-1K and COCO. Also, the same module has been tested on quite a few backbone networks including many common variants of ResNet and Vision Transformers.
- The paper is clearly written.

Weakness
- Honestly, I reviewed this paper in the last NeurIPS 2022 and thought the work quality, e.g. solid experiments, well-written,  reached the weak acceptance bar at that time. Given the edited version, the authors have addressed my previous concerns about the experiments. However, my main concerns about this work are still reserved. The novelty of applying layer-wise connection to improve attention is limited, which has been proved by the existing works, e.g, denseNet. And the marginal improvement over existing methods makes it stay slightly above the borderline.
- The ablation study of different layer connections impacting performance is missing.

clarity,_quality,_novelty_and_reproducibility: It is straightforward to implement MRLA. And the code is provided. The novelty is minor.

summary_of_the_review: Though the novelty and performance improvement over SOTA is minor, the work contains solid experiments and comes with good presentation, which provides insights into designing layer-wise connected attention. The idea is straightforward and easy to implement.  Overall, I think it reaches the acceptance bar.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a network family named Multi-head Recurrent Layer Attention (MRLA). It has two module variants: MRLA and MRLA-light. Both of the variants take a self-attention format. A major difference is that MRLA looks back at its previous module’s keys and values, while the light variant only uses the input features for the current layer. The self-attention takes a straightforward format where keys and queries are after global average pooling, and values are pixel features. The modules are tested with both convolutional networks and the vision transformers, and show consistent improvements over the baselines.
strength_and_weaknesses: The paper presents a network family that combines the attention, densely connected networks, SENet into one unified framework. The overall performance is good, and it’s nice to see this method can work fine with the recent vision transformers given their basic building block is already self-attention. The idea seems simple and straightforward, yet providing performance improvements at little additional costs.

There are some problems about how this paper is formed and how the claims are supported by the experimental evidence.

The first concept I think is `recurrent`. The overall framework could be implemented as a recurrent module. Ignore this comment if I’m wrong but I think the codes show that each building block has its own MRLA instance meaning that the variables are not shared. This leaves a question: does stacking up building blocks make them `recurrent`?

By the way, I think the paper could do better to give us a clearer picture of the underlying implementation by providing some pseudo codes for the two variants in the main paper or appendix. Fortunately, the authors uploaded their codes. So understanding them hasn’t been a problem.

The second problem is the position of the MRLA. The MRLA-light is a self-attention module implemented in the SENet style using sigmoid to replace softmax. It does not look back explicitly, does not compute features recurrently, yet performs similarly as or better than MRLA. This put MRLA in an awkward position that the method following the main topic of the paper more does not show strong advantages over the light variant that follows less.

Finally, I think it will strengthen the paper by showing performance improvements for larger models. The claims are mostly based on experimental results from models at small model scales. Larger model comparisons are necessary for methods that add complexity to the baselines.

clarity,_quality,_novelty_and_reproducibility: The paper is clear to read and easy to follow. Although I would appreciate it if the authors provide pseudo codes. The overall quality meets the conference standard. The novelty is not particularly strong. The reproducibility seems ok as the authors provided their codes.
summary_of_the_review: The paper proposes a network family for cross-layer attention. The paper is clearly written and the results look solid. What bothers me is that the major topics and claims are not well supported by the network implementation and the experimental evidence.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a multi-head recurrent layer attention (MRLA) mechanism, which uses the feature of the current layer to query all previous layers to retrieve query-related information of different levels. The paper further proposes a light-weighted MRLA to reduce the computational cost. The proposed method can be applied to both CNN and transformer-based networks. Experiments show its effectiveness in several vision tasks including image classification, object detection, and instance segmentation.
strength_and_weaknesses: Strong points.

1. The paper proposes a new attention mechanism by formulating each layer as a token and querying all previous layers.
2. The paper formulates extends the proposed layer attention to have multi-head and be light-weighted.
3. Code is submitted.


Weak points.

1. It is not clear to me why Eq.5 can be derived from Eq.3. More explanation would be better. Besides, it seems there are some symbol ambiguities, for example, it is not clear what the differences are between f and F. in Eq.3 and Eq.5, and what is s in Eq. 1 and Eq. 4.

2. Section 3.3 mentions an assumption that query vectors at two consecutive layers have a similar pattern, based on which the linearization is performed by using a learnable parameter. However, such an assumption is not justified. The paper should also compare the method proposed by (Katharopoulos et al. 2020) to demonstrate its own advantages.

3. Almost all papers referred to are published before/in 2021. A more comprehensive literature review and experimental comparisons are essential in order to evaluate its novelties.
clarity,_quality,_novelty_and_reproducibility: The paper is generally written well and easy to read. The formulation of the proposed layer attention is similar to that of token attention in transformers. The code is provided for reproduction.
summary_of_the_review: I am on the negative side at this point mainly due to the out-of-date literature review, and some experiments that support key assumptions/claims are missing.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduced a multi-head recurrent layer attention (MRLA) mechanism, the core idea of which is to leverage the representations of previous layers in a recurrent form so that the quadratic-complexity self-attention is avoided.  The authors further proposed a light-weight MRLA to reduce the computational cost. The resulting algorithm can be applied to both CNN and transformer-based networks, and the authors have done basic experiments to evaluate the performance of the proposed algorithms. 


justification_for_why_not_higher_score: All reviewers expressed their concerns about the limited novelty of this paper.  A similar idea has already been widely explored in the field, but this paper provided a different implementation. Though the authors mentioned relevant works that share a similar idea, e.g., TDAM (Jaiswal et al., 2022), BANet (Zhao et al. , 2022), and ACLA (Wang et al., 2022b), they only provide a rough comparison at the moment. In the final version, the authors are strongly suggested including a complete comparison with these most related works. 
justification_for_why_not_lower_score: All reviewers well recognized the contribution of this paper. Three of them would like to support the acceptance of this paper, while the remaining reviewer expressed his concerns about the out-of-the-date literature review and comparison, which can be addressed in the final version.
Decision: Accept: poster