Recurrence along Depth: Deep Convolutional Neural Networks with Recurrent Layer Aggregation
Year: 2021

Abstract:
This paper introduces a concept of layer aggregation to describe how information from previous layers can be reused to better extract features at the current layer. While DenseNet is a typical example of the layer aggregation mechanism, its redundancy has been commonly criticized in the literature. This motivates us to propose a very light-weighted module, called recurrent layer aggregation (RLA), by making use of the sequential structure of layers in a deep CNN. Our RLA module is compatible with many mainstream deep CNNs, including ResNets, Xception and MobileNetV2, and its effectiveness is verified by our extensive experiments on image classification, object detection and instance segmentation tasks. Specifically, improvements can be uniformly observed on CIFAR, ImageNet and MS COCO datasets, and the corresponding RLA-Nets can surprisingly boost the performances by 2-3% on the object detection task. This evidences the power of our RLA module in helping main CNNs better learn structural information in images.

text: This paper introduces a concept of layer aggregation to describe how information from previous layers can be reused to better extract features at the current layer. While DenseNet is a typical example of the layer aggregation mechanism, its redundancy has been commonly criticized in the literature. This motivates us to propose a very light-weighted module, called recurrent layer aggregation (RLA), by making use of the sequential structure of layers in a deep CNN. Our RLA module is compatible with many mainstream deep CNNs, including ResNets, Xception and MobileNetV2, and its effectiveness is verified by our extensive experiments on image classification, object detection and instance segmentation tasks. Specifically, improvements can be uniformly observed on CIFAR, ImageNet and MS COCO datasets, and the corresponding RLA-Nets can surprisingly boost the performances by 2-3% on the object detection task. This evidences the power of our RLA module in helping main CNNs better learn structural information in images.

heading: 1 Introduction
text: Convolutional neural networks (CNNs) have achieved notable success in computer vision tasks, crediting to their ability to extract high-level features from input images. Due to rapid growth in the depth of CNNs in recent years, the problem of how to pass information efficiently through layers often arises when designing deep architectures. Residual connections [20, 21], or skip connections, are now cornerstone components that act as information pathways to give layers direct access to previous ones and make training feasible with hundreds of layers. In a deep feature hierarchy, higher-level features learned by the network are built upon simpler ones, but not necessarily on the layer right before it [17]. This conjecture can be supported by the stochastic depth training method [27], where layers randomly gain access to previous ones. Furthermore, it is discovered that entire layers can be removed from ResNets without impacting the performance. This observation incubated the dense connectivity in DenseNets [28, 29], where layers in a stage have direct access to all previous layers through fully connected skip connections.
We introduce a concept of layer aggregation to systematically study network designs on feature reuse, and a typical example is DenseNet. However, within a densely-connected stage, the number of network parameters grows quadratically with respect to the number of layers. These highly redundant parameters may extract similar features multiple times [7], and limit the number of channels to store new information. This motivates us to propose a very light-weighted recurrent layer aggregation (RLA) module with much fewer parameters. Specifically, we use recurrent connections to replace dense connectivity and achieve a parameter count independent of the network depth. The RLA
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
module preserves the layer aggregation functionality, and it can be added to existing CNNs for better feature extraction.
Our RLA module is compatible with many CNNs used today. A schematic diagram is provided in Figure 1, where x represents hidden layers in a deep CNN and h represents hidden states in RLA modules. LeCun et al. [33] pointed out that unfolded RNNs can be seen as very deep feedforward networks where all the hidden layers share the same weights. When examined alone, an RLA module behaves similar to an unfolded RNN with layers of a deep CNN as its inputs. But applying RLA modules to CNNs is more than a simple combination of RNN and CNN (e.g., [41]). As the key idea behind layer aggregation is to provide the current layer with a version of layer history, information exchange between main CNN and RLA modules is a must. This results in a connection from the hidden unit of the RNN back to its inputs, which is hardly observable in other deep architectures. Moreover, we do not use global average pooling when passing information from the main CNN to RLA, so that the historical information is enriched and contains spatial information.
Despite its recurrent design, a convolutional-RNN-based RLA module can be easily implemented using standard convolutions with parameter sharing. Empirically, RLA modules are computationally light-weighted and impose only a slight increase in model parameters and computational cost. In terms of functionality, it serves purposes beyond channel attention and can be applied on top of channel attention modules. We perform extensive experiments across a variety of tasks using different deep CNN architectures, including ResNets [20, 21], ECANets [54], Xception [10] and MobileNetV2 [43]. Our experimental results show that RLA consistently improves model performances on CIFAR, ImageNet and MS COCO datasets. On MS COCO, our RLA module can achieve 2-3% gains in terms of average precision, which significantly outperforms other state-of-the-art networks with remarkable performances.
The main contributions of our work are below: (1) We introduce a definition of layer aggregation for analyzing CNN architectures. (2) We propose a novel recurrent layer aggregation module, with motivation from the layer aggregation mechanism and time series analysis. (3) We investigate detailed RLA module designs through ablation study and provide guidelines on applying RLA modules. (4) We show the effectiveness of RLA across a broad range of tasks on benchmark datasets using several popular deep CNN architectures.

heading: 2 Related work
text: CNN-RNN hybrid models Deep CNNs with RLA modules should be easily distinguishable from other CNN-RNN hybrid networks, as RLA features information exchange between CNN and RNN at each layer and the corresponding recurrent update. In visual description tasks, RNNs are adopted to process the outputs of CNNs for sequence generation [14, 51, 44, 53, 5]. For video super-resolution, convolutional LSTMs can be used first to capture the temporal dependence between frames and then pass outputs to CNNs [38]. There are only a few papers that apply the concept of recurrence to CNNs. In RecResNet [1], a whole CNN is used recurrently for disparity map refinement. Moreover, in [36], convolutional layers are used recurrently to produce "deep" CNNs before the debut of ResNets. Dense-and-Implicit Attention Network [30] possesses a similar high-level CNN-RNN combination structure compared with ours, while their design is within the attention framework.
CNNs with branching paths CNNs with our RLA module have similar structures compared to those with branching paths, e.g., Inception networks [46, 31, 47, 48], or networks with grouped convolutions, e.g., AlexNet [32] and ResNeXt [56]. The main difference is that our RLA module has
shared convolutional kernels along depth. When RLA is applied to ResNets, the resulting networks can be viewed as compressed Dual Path Networks [7].
Attention mechanisms in CNNs Attention mechanisms have been incorporated into CNNs mainly in the form of spatial attention and channel attention. Successful applications include image captioning [57, 5], visual question answering [62], image classification [52, 16], semantic segmentation [34], face verification [6], and video frame interpolation [9]. An outstanding design of channel attention would be the Squeeze-and-Excitation (SE) block [26], which is later adopted by MobileNetV3 [25] and EfficientNet [49] and developed into compressed variants [54]. The proposed layer aggregation mechanism has a connection with channel attention across multiple layers, and meanwhile, it can be applied to networks with channel attention modules.
RNN variants Since RLA modules adopt the form of a convolutional RNN, RNN variants give rise to different RLA designs. For example, the hidden feature maps in RLA can also be updated by the convolutional counterparts of LSTM [24, 45] or GRU cells[8]. In terms of architectural connectivity, skip connections can be introduced to RNN hidden states [40, 60, 3, 13]. Moreover, for deep RNNs, different layers of state variables can be updated at different frequencies [15]. Correspondingly, we may derive deep RLA modules that formulate stage aggregation on top of layer aggregation.

heading: 3 Recurrent layer aggregation modules
text: This section first introduces a concept of layer aggregation and some parsimonious models in time series analysis, which together motivate a type of light-weighted recurrent layer aggregation (RLA) modules by making use of the sequential structures of deep CNNs.

heading: 3.1 Layer aggregation
text: Consider a deep CNN with xt being the hidden features at the tth layer and x0 being the input, where L is the number of layers, and 1 ≤ t ≤ L. The CNN is said to have the layer aggregation mechanism if At = gt(xt−1, xt−2, . . . , x0) and xt = f t(At−1, xt−1), (1) where f t denotes the transformation that produces new feature map at layer t of the CNN, At represents the aggregated information up to (t−1)th layer, and gt is the corresponding transformation that summarizes these layers. For CNNs without skip connections, according to our definition, they do not involve layer aggregation since xt = f t(xt−1), i.e., the information in xls with l < t− 1 is only available to xt through xt−1. On the other hand, the Hierarchical Layer Aggregation [59] can be shown to have such mechanism since it satisfies Eq. (1).
It is usually more convenient to consider an additive form for the summarizing transformation, i.e. At = ∑t−1
l=0 g t l (x l), (2)
where an additive constant c may also be included, and gtl can be one or more convolutions, e.g., a set of 1x1-3x3-1x1 convolutions that compose a bottleneck block. Consider a DenseNet layer
xt = Conv3t [ Conv1t(Concat(x0, x1, ..., xt−1)) ] , (3)
where Conv1t and Conv3t denote 1x1 and 3x3 convolutions, respectively. It can be verified that Conv1t(Concat(x0, x1, ..., xt−1)) = ∑t−1
l=0 Conv1 t l(x l), (4)
where the kernel weights of Conv1tl form a partition of the weights in Conv1 t. As a result, At = ∑t−1
l=0 Conv1 t+1 l (x l) and xt = Conv3t [ At−1 + Conv1tt−1(x t−1) ] ,
i.e., it is a typical example of the layer aggregation mechanism.
We next consider the pre-activated ResNets [21]. On the one hand, according to the ResNet update xt = xt−1 + gt−1(x
t−1), it may be concluded to involve no layer aggregation. On the other hand, by recursively applying the update, we have xt = ∑t−1 l=0 gl(x
l) + x0, which satisfies Eq. (1) since At = ∑t−1 l=0 gl(x
l) + x0 and xt = At−1 + gt−1(xt−1). It is noticeable that gl depends only on l, the ordinal number of previous layers, and we will discuss such a pattern in the next subsection.
Connection with channel attention The layer aggregation mechanism in the original DenseNet can be interpreted as channel attention across layers. Specifically, the multi-head masked self-attention [50] mechanism can be written as
Attentiont = Concat(headt1, ..., head t m) with head t i = ∑t−1 l=0 s t l,i x l, for i = 1, ...,m, (5)
where query, key and value are all set to be xt, stl,is are scalar similarity measures and m is the number of attention heads.
Note that, when considering DenseNet layers, feature maps are tensors instead of matrices. Suppose that all xl’s have the same number of channels, denoted by k, and then Conv1t has tk input and m output channels. Denote the tk × 1 × 1 × m kernel weights of Conv1t by {W tj,i}. Let head t i be the ith channel in Attentiont, and xlc be the cth channel of the feature tensor x l, then it holds that
headti = ∑ l,c W t j(l,c),i x l c, where j(l, c) = lk+ c is an index. As a result, DenseNet updates coincide with a natural extension of Eq. (5) to tensor variables.
However, the above attention mechanism is too meticulous in a CNN context. For example, it will collapse when simply changing Conv1t in DenseNets to a 3x3 convolution. In contrast, the layer aggregation interpretation of DenseNet is robust to such a modification.
Our purpose of the discussions above is two-fold: (1) we build the relationship that layer aggregation generalizes channel attention across layers based on the DenseNet architecture; and thus, (2) layer aggregation is not a substitute to channel attention modules, but complementary to them, which inspires us to add the proposed RLA modules to ECAnets in Section 4.

heading: 3.2 Sequential structure of layers in deep networks
text: The complexity of summarizing transformations can drop dramatically from (1) to (2), while it may still be complicated. Actually DenseNets have been frequently criticized for their redundancy. Note that the layers {xt} form a sequential structure, and hence this subsection will introduce some related concepts in time series analysis such that we can further compress the summarizing transformations at (1) and (2). More background on time series is available in the supplementary material A.2.
We first simplify Eq. (2) by assuming that gtl depends on the lag t− l only, and this is a typical setting in time series analysis. If all xts are scalars and all related transformations are linear, we have
xt = β1x t−1 + ...+ βt−1x 1 + βtx 0 = ∑t−1 l=0 βt−lx l. (6)
It can also be rewritten into ∑∞
l=1 βlx t−l with xs = 0 for s < 0, which has a form of autoregressive (AR) models in time series, xt = ∑∞
l=1 βlx t−l + it, with it being the additive error.
In time series analysis, the above AR model, which is usually referred to as the AR(∞) model, is mainly for theoretical discussions, and the commonly used running model is the autoregressive moving average (ARMA) model since it has a parsimonious structure. Consider a simple ARMA(1, 1) model, xt = βxt−1 + it − γit−1. It can be verified to have an AR(∞) form of
xt = βxt−1 + it − γit−1 = βxt−1 − γ(xt−1 − βxt−2) + it − γ2it−2 = · · · = ∑∞ l=1(β − γ)γl−1xt−l + it, (7)
where equation is = xs−βxs−1+γis−1 is recursively applied for all s ≤ t−1, while there are only two parameters involved. This motivates us to use a similar pattern to construct a more parsimonious form for Eq. (6). Let
ht = αxt−1 + γht−1 and xt = β1xt−1 + β2ht−1. (8) Assuming h0 = 0, we then have ht = ∑t
l=1 αγ l−1xt−l and xt = β1xt−1 + β2α ∑t−1 l=1 γ
l−1xt−1−l, which has a form of (6) but involves at most four parameters. It is worthy to point out that ht plays a role similar to that of it, and Eq. (8) is similar to the principle of RNNs [11].
We may also consider the case that gtl at (2) depends on l only, i.e. x t = f t( ∑t−2 l=0 gl(x
l), xt−1), which leads to a ResNet-type layer. From (8), for s > 0, the information in xt−s will be reused when extracting features xt by an amount that depends on lag s only and decays at a rate of γs. However, as mentioned in Section 3.1, ResNet can be viewed as layer aggregation in a different pattern, and the amount of information in xl reused when extracting features xt depends on the ordinal number l only. We next conduct a small experiment to compare the ResNet-type simplification with that at (6).
Comparison of two simplifications The experiment is carried out based on the 100-layer DenseNetBC (k = 12) [28]. According to the two simplifications, we modify the DenseNet update (3) into the following versions:
(Shared-Lag) xt = Conv3t(Conv11(xt−1) + · · ·+ Conv1t−1(x1) + Conv1t0(x0)), (9) (Shared-Ordinal) xt = Conv3t(Conv1t−1(xt−1) + · · ·+ Conv11(x1) + Conv1t0(x0)), (10)
which correspond to (6) and ResNet, respectively. The above formulations require Conv1l to be compatible with all previous layers. For DenseNet-BC (k = 12), the input to dense layers x0 has 3 + 2 × k = 27 channels, while all later layers xl, 1 ≤ l ≤ 16, have k = 12 channels. Thus, we keep the convolutions applied to x0 unchanged and only modify the other 1x1 convolutions. Detailed explanations of the network structure of (9) are provided in the supplementary material A.2.
Table 1 presents the comparison of parameter count and accuracy on CIFAR-10. Detailed training settings are the same as other experiments on CIFAR-10 in Section 4. Sharing the 1x1 convolutions reduces the number of parameters by 25% (0.20M), while the accuracy only drops by about 0.5%. This observation is consistent with the parameter sharing experiment by [37], where they found that a ResNet with shared weights retains most of its performance on CIFAR-10. And the better performance of Shared-Lag encourages us to pursue a more parsimonious structure along the direction of (6).
Furthermore, we extract the weights learned by the shared 1x1 convolutions and produce Figure 2, following Figure 5 in [28]. A first observation is that the strongest signal in Shared-Lag (lag=1) is almost twice as strong as that in Shared-Ordinal (layer=1). Secondly, we spot a quick decaying pattern in the plot for Shared-Lag, which is similar to the exponential decay pattern in an ARMA model or an RNN [61]. This observation provides us empirical evidence to adopt the principle at (8) to design a light-weighted layer aggregation mechanism.

heading: 3.3 Recurrent layer aggregation
text: By adopting the idea at (8), we introduce a recurrent layer aggregation (RLA) mechanism below,
ht = gt(ht−1, xt−1) and xt = f t(ht−1, xt−1), (11)
where At = ht represents the recurrently aggregated information up to (t− 1)th layer, and it can be shown to have an additive form of (2) under a linearity assumption; see supplementary material A.3. By considering different forms of gt, we can achieve various RLA modules to help the main CNN better extract features, and weight sharing can be further used to regularize the layer aggregation. The form of f t is mainly determined by the currently used mainstream CNNs, and the combination of aggregated information may be implemented in different ways.
Figure 3 gives an example for a CNN block with a residual connection, and legends are provided at the bottom of Figure 4. Its update can be formulated as
ht = g2[g1(y t) + ht−1] and xt = yt + xt−1,
where yt = f t1[Concat(h t−1, xt−1)]. Specifically, inputs xt−1 and ht−1 are combined via concatenation and then passed to a residual unit to extract new features. One copy of this new feature map is joined with the residual connection, and the other copy is compressed via the shared 1x1 convolution and then added to ht−1. Then, the RLA path goes through a pre-activated ConvRNN unit comprised of an unshared batch normalization, a tanh activation, and a recurrent 3x3 convolution.
Practically, backbone CNNs often comprise of several stages, and RLA modules can cater to various stage structures following the guidelines below:
1. Group main CNN layers/blocks into stages so that feature maps within the same stage have the same resolution, e.g., ResNet-50/101/152 has 4 stages and MobileNetV2 has 5 stages.
2. For each stage, define (a) a shared recurrent unit, allowing information to be compressed differently at various stages, and (b) a shared 1x1 convolution, to change the channels of the residual feature maps for adding it to the RLA hidden state.
3. (RLA path) RLA hidden state h0 is initialized as zero. Before entering the next stage, the RLA hidden states are downsampled to the correct resolution via average pooling.
4. (RLA output) For classification, the last hidden state is concatenated with the main CNN output and passed to the classifier. For detection and segmentation, RLA output can be discarded.

heading: 4 Experiments
text: We verify the effectiveness of our RLA module in Figure 3 on image classification, object detection and instance segmentation tasks using CIFAR, ImageNet and MS COCO datasets. All experiments are implemented on four Tesla V100 GPUs. 1 More implementation details and tabular descriptions of network architectures are provided in Section B of the supplementary material.

heading: 4.1 CIFAR and analysis
text: The two CIFAR datasets, CIFAR-10 and CIFAR-100, consist of 60k 32× 32 RGB images of 10 and 100 classes. The training and test sets contain 50k and 10k images. We adopt a train/validation split of 45k/5k and follow the widely used data preprocessing scheme in [20, 28]. All the networks are trained from scratch using Nesterov SGD with momentum of 0.9, l2 weight decay of 10−4, and a batch size of 128 for 300 epochs. The initial learning rate is set to 0.1 and is divided by 10 at 150 and 225 epochs. We choose the model with the highest validation accuracy.
We report the performance of each baseline model and its RLA-Net counterpart on CIFAR-10 and CIFAR-100 test sets in Table 2. We observe that RLA-Net consistently improves performance across various networks with different depths, blocks and patterns of stages. For the three models in Table 2, adding the RLA module introduces about a 1%-4% increase in parameters and FLOPs. Meanwhile,
1Our implementation and weights are available at https://github.com/fangyanwen1106/RLANet.
our RLA-Net improves the accuracies of ResNet-110 and ResNet-164 by about 0.5% and 1.0% on CIFAR-10 and CIFAR-100, respectively.

heading: 4.2 ImageNet classification
text: This subsection reports experiments on the ImageNet LSVRC 2012 dataset [12]. We employ our RLA module on the widely used ResNet [20] architectures and the light-weighted MobileNetV2 [43], as well as their counterparts with the state-of-the-art channel attention ECA modules [54]. For MobileNetV2, depthwise separable convolutions are used for the shared 3x3 convolutions to adapt to the inverted residual blocks of MobileNetV2. When both RLA and ECA modules present, ECA modules are placed according to [54], and the output of the ECA module is then passed to our RLA
module through the shared 1x1 convolution. We compare our RLA-Nets with several state-of-the-art attention-based CNN architectures, including SENet [26], CBAM [55] and ECANet [54].
For training RLA-ResNets, we follow the same data augmentation and hyper-parameter settings as in [20, 28]. All the networks are trained using SGD with momentum 0.9, l2 weight decay of 10−4, and a mini-batch size of 256 on 4 GPUs. We train models for 120 epochs from scratch and use the weight initialization strategy described in [18]. The initial learning rate is set to 0.1 and decreased by a factor of 10 every 30 epochs. For the light-weighted model MobileNetV2, we train the model on 2 GPUs within 400 epochs using SGD with weight decay of 4e-5, momentum of 0.9, and a mini-batch size of 96, following the settings in [43]. The initial learning rate is set to 0.045, decreased by a linear decay rate of 0.98. All the parameters and FLOPs shown in the tables are computed by our devices.
Table 3 shows that our RLA-Nets obtain better results compared with their original counterparts while introducing small extra costs. Though RLA modules introduce more FLOPs when compared with channel attention modules, we find that the extra training time per epoch is almost the same. Specifically, adding SE, ECA or RLA module to ResNet-101 costs about 15%, 15% or 19% more training time. RLA-ResNets introduce about 1% increase in the number of parameters, leading to 1.9%, 2.1% and 1.8% increases in top-1 accuracy for RLA-ResNet-50/101/152, respectively. Similar results can be observed on the light-weighted MobileNetV2 architectures. Furthermore, combinations of RLA and ECA modules yield remarkably better results, achieving the best performances among these models. These further improvements based on ECANets verify our perspective that the functionalities of channel attention modules and our proposed RLA module are complementary.
Last but not least, our model with some tricks in [23] can perform remarkably better than many popular networks and state-of-the-art networks with delicate designs, including DPN [7], AOGNet [35] and HCGNet [58], which are designed to combine the advantages of ResNet and DenseNet. The notable and consistent improvements demonstrate that any tricks or refinements used to further improve the original ResNets can be directly applied to our model to further improve the model performance. And RLA modules can be treated as one of the beneficial architectural refinements to be adopted by state-of-the-art CNNs as well.

heading: 4.3 Object detection and instance segmentation on MS COCO
text: To show the transferability and the generalization ability, we experiment our RLA-Net on object detection task using Faster R-CNN [42], Mask R-CNN [22] and RetinaNet [39] as detectors. For Mask R-CNN, we also show instance segmentation results. All detectors are implemented by the open
source MMDetection toolkit [4]. We employ the same settings as in [54] to finetune our RLA-Nets on COCO train2017 set. Specifically, the shorter side of input images are resized to 800. We train all detectors within 12 epochs using SGD with weight decay of 1e-4, momentum of 0.9 and mini-batch size of 8. The learning rate is initialized to 0.01 and is decreased by a factor of 10 after 8 and 11 epochs, respectively, i.e., the 1x training schedule. Different from the classification task, we do not pass the hidden states of our RLA module to the FPN. Thus, the gains are solely from the more powerful representations learnt in the main CNNs.
We report the results on COCO val2017 set by standard COCO metrics of Average Precision (AP). Tables 4 and 5 report the performances of the object detectors. We can observe that our RLA-Net outperforms the original ResNet by 2.4% and 2.5% in terms of AP for the networks of 50 and 101 layers, respectively. More excitingly, our RLA-ResNet-50 outperforms ResNet-101 on these three detectors. In particular, our RLA module achieves more gains in AP75 and APL, indicating the high accuracy and effectiveness for large objects. Remarkably, exploiting RLA on ECANets can surpass all other models with large margins. We hypothesize that, with the help of RLA modules, positional information from low-level features is better preserved in the main CNN, leading to these notable improvements. In summary, the results in Tables 4 and 5 demonstrate that our RLA-Net can well generalize to various tasks with extraordinary benefits on the object detection task.

heading: 4.4 Ablation study
text: We conduct an ablation study on ImageNet with ResNet-50 as the baseline network. To validate the effectiveness of the detailed inner structure, we ablate the important design elements of RLA module and compare the proposed structure with the following variants:
(a) channels increased by 32;
(b) RLA without parameter sharing;
(c) RLA without the information flow from RLA to the main CNN (as in [41]);
(d) RLA with ConvLSTM cell;
(e) RLA with the post-activated hidden unit; and
(f-j) RLA with different connectivity between two paths, see Figure 4.
The results are reported in Table 6. We first demonstrate that the improvement in accuracy is not fully caused by the increment in network width by comparing to the variant with 32 additional channels in each layer. We then investigate the necessity of recurrence in our module by comparing it with its fully unshared counterpart. Results show that the recurrent design can reduce parameter count and slightly increase accuracy at the same time. Removing the information flow from RLA to the main CNN results in a decrease in model performances, validating that the information exchange between RLA and the main CNN is a must. Furthermore, using ConvLSTM cells may be unnecessary since
each stage only has 3 to 6 time steps in ResNet-50, and the post-activated recurrent unit is not very compatible with the pre-activated network.
We also compare different integration strategies to combine our RLA module with the main CNN. In addition to the proposed design, we consider five variants (v2-v6), as depicted in Figure 4. The performances of the variants are reported in Table 6. We observe that the variants v1, v3 and v5 perform similarly well, as they all integrate new information from the main CNNs before entering into the recurrent operation. Compared with variants v3 and v5, v1 obtains the learned residual as new information, including less redundancy. The performance of our RLA modules is relatively robust to the integration strategies, as long as new information is added before the recurrent operation.
In summary, the proposed RLA module structure in Figure 3 achieves the smallest top-1 error and very competitive top-5 error with the minimum number of additional parameters. This structure can strike a good balance between the increase of computational cost and performance gain. We also conduct a more comprehensive ablation study on the CIFAR-10 dataset with ResNet-164 as the baseline network. Due to limited space, we present details of the ablation studies and the investigated module structures in the supplementary material B.4.

heading: 5 Conclusion and future work
text: This paper proposes a recurrent layer aggregation module that is light-weighted and capable of providing the backbone CNN with sequentially aggregated information from previous layers. Its potential has been well illustrated by the applications on mainstream CNNs with benchmark datasets. On the other hand, we mainly focused on controlled experiments to illustrate the effectiveness and compatibility of the RLA modules with settings adopted from existing work. Our first future work is to consider hyperparameter tuning, advanced training tricks [23, 2], or architecture search [49]. Moreover, in terms of RLA design and application, it is also an interesting direction to apply it at the stage level on top of the layer level, which leads to deep RLA modules.
Acknowledgments and Disclosure of Funding
We thank the anonymous reviewers for their constructive feedback and Jinxu Zhao for technical support. This project is supported by National Supercomputer Center in Guangzhou, China.

Review summaries:
A CNN architecture with two network paths (one is non-recurrent and another is recurrent) is introduced in this paper. The features from the two paths are aggregated through the RLA module (with residual connection) and redistributed to the individual paths. This brings performance improvements on image classification, object detection, instance segmentation at very small increases of parameter and computation costs.

The paper proposed a recurrent layer aggregation (RLA) structure. The RLA structure proposes a recurrent module to aggregating information from the previous layers. The proposal is inspired by an additive form of describing the information aggregations in deep neural network and its similarity to autoregressive models. The structure is compatible with many popular CNN models including the ResNets, Xception and MobileNet. Extensive sets of experiments have been conducted to demonstrate the effectiveness of the proposed method on different vision tasks.

This paper is inspired by the layer aggregation concept which is also used in DenseNet to better reuse the information from previous layers and extract features at the current layer. Then the authors propose a light-weighted module, called recurrent layer aggregation (RLA), and prove it is compatible with many common structures in ResNets, Xception and MobileNetV2. Finally, they use the controlled experiments to show the effectiveness of the RLA modules on classification and detection tasks. 

This paper proposes a module called Recurrent Layer Aggregation (RLA) to improve the performance of general CNN backbones. This module is motivated by the analysis of both layer aggregation and the time series method. Experiments on tasks like classification, detection, and segmentation validate the effectiveness of this RLA module.