Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper is inspired by the layer aggregation concept which is also used in DenseNet to better reuse the information from previous layers and extract features at the current layer. Then the authors propose a light-weighted module, called recurrent layer aggregation (RLA), and prove it is compatible with many common structures in ResNets, Xception and MobileNetV2. Finally, they use the controlled experiments to show the effectiveness of the RLA modules on classification and detection tasks.

main_review: This paper has the good motivation to discuss better utilize the common information among adjacent layers to improve the feature extraction capability of the deep networks.
From the clarity aspect, I would suggest the authors to put the Figure B.4: (RLA modules with different types of connections) from the supplementary to the main paper. Just use the formula to express the module structure and its variants is not as clear as the images.
The main concern about this paper is the real performance during training and inference. Because of the introducing the recurrent layer aggregation module, layers need to also consider the previous states of adjacent layers. So similar to RNN modules, it is not friendly to the parallel acceleration like GPU device. I am curious about the real performance number when adding the RLA module.
limitations_and_societal_impact: The authors seem not to provide enough limitations and potential negative social impact of their work.
I suggest the authors to provide a description of the potential performance regression due to the introduction of the RLA module.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: A CNN architecture with two network paths (one is non-recurrent and another is recurrent) is introduced in this paper. The features from the two paths are aggregated through the RLA module (with residual connection) and redistributed to the individual paths. This brings performance improvements on image classification, object detection, instance segmentation at very small increases of parameter and computation costs.
main_review: Strengths
+ Lightweight design and works well with modern CNN architectures.
+ Good discussion of existing work.
+ Comprehensive experiments on several tasks with good results. RLA can be combined with ECA to achieve stronger performance.

Weaknesses
- The writing for the motivation behind the design of RLA is overly lengthy and not easy to follow. This part does not have major implications on RLA-Net as a whole yet it occupies a large portion of the paper. It is only relevant to Eq.8 in Sec 3.3 which is similar to vanilla RNN (already well known to the community).
- "applying RLA modules to CNNs is more than a simple combination of RNN and CNN (e.g., [39])" - Since RLA is different from [39] in terms of mutual information exchange and spatial feature preservation, there should be experiments showing the effects of these differences. Otherwise, it would be hard to justify the technical significance of RLA.
- The motivation behind using recurrent/shared weights for the second network path is not clear. There is nothing a recurrent network can model that a non-recurrent network cannot model. It can be simply replaced by a non-recurrent network. It seems to me that using recurrent/shared weights is just for keeping the parameter counts low. 

Minor issue
Line 132: "DensNets" -> "DenseNets"
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a module called Recurrent Layer Aggregation (RLA) to improve the performance of general CNN backbones. This module is motivated by the analysis of both layer aggregation and the time series method. Experiments on tasks like classification, detection, and segmentation validate the effectiveness of this RLA module.
main_review: Originality:
The proposed RLA seems new to me, especially considering its relation with the time series analysis. The recurrent module is not simple usage of the off-the-shelf modules like LSTM or GRU but a custom one, which is interesting.

Quality:
The paper is technically sound. The proposed module is motivated by both layer aggregation and the time series method. The implementation of RLA is just a slight variant of the inspired formula Eq. (8), which is clean.
However, despite the custom recurrent module implementation, I think there should be an experimental comparison to show how the LSTM and GRU perform when they are used as the recurrent aggregation module (as Eq. 11). This will show us whether the custom recurrent module in the RLA is required or just an option.

Clarity:
There are some issues to address.
1. Line 138: I think there should be some more introductions about the autoregressive (AR) models and autoregressive moving average (ARMA) models, e.g. what they are handling, what their benefits are.
2. Line 116-129: Is there any specific reason to show the relationship between the layer aggregation and the channel attention? Does this motivate the proposed method?
3. Line 139: What is the meaning of the additive error $i^t$?
4. Line 191: Are the batch-norm layers share weights in the recurrent layers?

Significance:
I believe this work shows a certain level of significance as it highlights the importance of layer aggregation in a deep network, and the implementation of using a recurrent aggregation mechanism is new and simple.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposed a recurrent layer aggregation (RLA) structure. The RLA structure proposes a recurrent module to aggregating information from the previous layers. The proposal is inspired by an additive form of describing the information aggregations in deep neural network and its similarity to autoregressive models. The structure is compatible with many popular CNN models including the ResNets, Xception and MobileNet. Extensive sets of experiments have been conducted to demonstrate the effectiveness of the proposed method on different vision tasks.
main_review: Pros
1. The proposed RLA is a class of modules compatible with different popular neural networks and other methods like channel attention. Experiment shows consistent performance improvements on different backbone models and tasks over model without RLA.
2. The motivation of the proposed RLA based on time series model is strong and well supported by experiment results in Table 2 and Fig. 2.
3. The structure of the paper is well-organized and easy to follow.

Cons

There’re no significant shortcomings in this work but two additional comments that might help improve the paper. 
1. The paper shows the robustness of performance improvements to different RLA vairants on cifar10 datasets. Extension of such experiments to ImageNet and MSCOCO with similar results would support this conclusion significantly better.
2. The purpose of the paragraphs about interpretation of densenet as channel attention and the connection between this interpretation and motivation of RLA are not clear. 
limitations_and_societal_impact: There’s no discussion of limitations or negative social impacts in the paper. In general, concerns about negative social impacts of vision models are related to security issues like vulnerability to adversarial attacks and privacy issues due to the abuse of these models.
review_title: null

Meta-review: The paper presents a new design for aggregating information from shallower to deeper layers of a neural network (i.e., an alternative to ResNet or DenseNet-style connections).  Following the author response and discussion, 1 reviewer rating is negative, 3 are positive, with overall sentiment being marginally positive.  The paper presents a clear design and provides sufficient experimental evidence of benefits.  A downside is that performance benefits, even if consistent, are of an incremental nature over existing network designs.
Decision: Accept (Poster)