Recurrence along Depth: Deep Convolutional Neural Networks with Recurrent Layer Aggregation
Authors: Jingyu Zhao
Yanwen Fang
Guodong Li
Venue: NeurIPS 2021 Poster (2021)
Keywords: Convolutional Neural Networks
ResNet
DenseNet
Recurrent Structures
Layer Aggregation

Abstract: This paper introduces a concept of layer aggregation to describe how information from previous layers can be reused to better extract features at the current layer. While DenseNet is a typical example of the layer aggregation mechanism, its redundancy has been commonly criticized in the literature. This motivates us to propose a very light-weighted module, called recurrent layer aggregation (RLA), by making use of the sequential structure of layers in a deep CNN. Our RLA module is compatible with many mainstream deep CNNs, including ResNets, Xception and MobileNetV2, and its effectiveness is verified by our extensive experiments on image classification, object detection and instance segmentation tasks. Specifically, improvements can be uniformly observed on CIFAR, ImageNet and MS COCO datasets, and the corresponding RLA-Nets can surprisingly boost the performances by 2-3% on the object detection task. This evidences the power of our RLA module in helping main CNNs better learn structural information in images.


Recurrence along Depth: Deep Convolutional Neural Networks with Recurrent Layer Aggregation
Jingyu Zhao, Yanwen Fang and Guodong Li
Department of Statistics and Actuarial Science
The University of Hong Kong
fgladys17, u3545663))cconnect hku.hk.. gdl gdli@hku.ha hk
Abstract
This paper introduces concept of layer aggregation to describe how information from previous layers be reused better extract features at the current layer. While DenseNet is a typical example of the layer aggregation mechanism, its redundancy has been commonly criticized in the literature. This motivates us propose very light-weighted module, called recurrent layer aggregation (RLA), by making use of the sequential structure of layers is compatible with many mainstream deep CNNs, including ResNets, Xception and MobileNetV2, and its effectiveness verified by our extensive experiments on image classification, object detection and instance segmentation tasks. Specifically, improvements can be uniformly observed on CIFAR, ImageNet and MS COCÓ datasets, and the corresponding RLA-Nets can surprisingly boost the performances by 2-3% on the object detection task. This evidences the power our RLA module in helping main CNNs better learn structural information in images.
deep CNN. Our RLA module
Introduction
Convolutional neural networks (CNNs) have achieved notable success in computer vision tasks, crediting to their ability to extract high-level features from input images. Due to rapid growth depth of CNNs in recent years, the problem of how to pass information efficiently through layers often arises when designing deep architectures. Residual connections [20 21), or skip connections, are now cornerstone components that act information pathways to give layers direct access previous ones and make training feasible with hundreds of layers. In a deep feature hicrarchy, higher-level features learned by the network built upon simpler ones, but before [17]. This conjecture can be supported by the stochastic depth training method (27) where layers randomly gain access previous ones. Furthermore, it is discovered that entire layers be removed from ResNets without impacting the performance. This observation incubated the dense connectivity in DenseNets (28 29 where layers stage have direct access all previous layers through fully connected skip connections.
necessarily the layer right
We introduce concept of layer aggregation to systematically study network designs feature reuse, and a typical example is DenseNet. However, within a densely-connecte stage, the number of network parameters grows quadratically with respect to the number of layers. These highly redundant parameters may extract similar features multiple times 77. and limit the number of channels to store new information. This motivates to propose very light- weighted recurrent layer aggregation (RLA) module with much fewer parameters. Specifically, we use recurrent connections to replace dense connectivity and achieve parameter count independent of the network depth. The RLA
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

---

CNN lsyer(s)
time delay
recurrent connection
tansor
AveragePooling
0
Output
cal(a.
Global PageePdinngg
Deneet Layer
DownSampligg
Input
Repeat to create several stages
Figure 1: Schematic diagram of CNN with recurrent layer aggregation for image classification.
module preserves the layer aggregation functionality, and it can be added to existing CNNs for better feature extraction.
Our RLA module is compatible with many CNNs used today. A schematic diagram provided in Figure I where 2 represents hidden layers in deep CNN and h represents hidden states in RLA modules. LeCun et al. [33] pointed out that unfolded RNNs can be seen as very deep feedforward networks where all the hidden layers share the same weights. When examined alone, RLA module behaves similar to an unfolded RNN with layers of a deep CNN as its inputs. But applying RLA modules CNNs is behind layer aggregation to provide the current layer with a version of layer history, information exchange between main CNN and RLA modules is must. This results in a connection from the hidden unit of the RNN back to its inputs, which is hardly observable in other deep architectures. Moreover, we do use global average pooling when passing information from the main CNN to RLA, so that the historical information is enriched and contains spatial information.
than simple combination of RNN and CNN (e.g.. 41) As the key idea
Despite its recurrent design, convolutional -aaiiiioonl-NNN------- based RLA module can be easily implemented using standard convolutions with parameter sharing. Empirically, RLA modules are computationally liigt-we and impose only a slight increase in model parameters and computational cost. In terms of functionality, it serves purposes beyond channel attention and can be applied on top of channel attention modules. We perform extensive experiments across variety of tasks using different deep CNN architectures, including ResNets (1) ECANets [54], Xception [10] and MobileNetV2 43]. Our experimental results show that RLA consistently improves model performances on CIFAR, ImageNet and MS COCO datasets, On MS COCO, our RLA module can achieve 2-3% gains in terms of average precision, which significantly outperforms other state-of- of-the-art networks with remarkable performances.
The main contributions of our work are below: (1) We introduce a definition of layer aggregation for analyzing CNN architectures. (2) We propose a novel recurrent layer aggregation module, with motivation from the layer aggregation mechanism and time series analysis. (3) We investigate detailed RLA module designs through ablation study and provide guidelines on applying RLA modules. (4) We show the effectiveness of RLA across broad range of tasks on benchmark datasets using several popular deep CNN architectures.
2 Related work
CNN-RNN hybrid models Deep CNNs with RLA modules should be easily distinguishable from other CNN-RNN hybrid networks, RLA features information exchange between CNN and RNN each layer and the corresponding recurrent update. In visual description tasks, RNNs are adopted to For video super-resolution, convolutional LSTMs can be used first to capture the temporal dependence between frames and then pass outputs to CNNs 38] There are only few papers that apply the concept of recurrence CNNs. In RecResNet [. a whole CNN is used recurrently for disparity map refinement. Moreover, in (36, convolutional layers are used recurrently to produce "deep" CNNs before the debut of ResNets. Lense-and-Implee Attention Network [30 possesses similar high-level CNN-RNN combination structure compared with ours, while their design within the attention framework.
process the outputs of CNNs for sequence generation [14 51 44
53 5
CNNs with branching paths CNNs with RLA module have similar structures compared to those with branching paths, e.g., Inception networks [46] 31 47 48). or networks with grouped convolutions, e.g., AlexNet (32) and ResNeXt (56). The main difference is that our RLA module has

---

shared convolutional kernels along depth. When RLA is applied to ResNets, the resulting networks can be viewed as compressed Dual Path Networks 7
Attention mechanisms in CNNs Attention mechanisms have been incorporated into CNNs mainly
the form of spatial attention and channel attention. Successful applications include image captioning [57 5), visual question answering [62,, image classification [52, 16], semantic segmentation [34], face verification [6]. and video frame interpolation 9 An outstanding design of channel attention would be the Squeeze-and
(SE) block [26]. which is later adopted by MobileNetV3 [25]
and EfficientNet 49) and developed into compressed variants [54]. The proposed layer aggregation mechanism has connection with channel attention across multiple layers, and meanwhile, it can be
applied to networks with channel attention modules.
RNN variants Since RLA modules adopt the form of a convolutional RNN, RNN variants give rise different RLA designs. For example, the hidden feature maps in RLA can also be updated by the convolutional counterparts of LSTM [24 45] or GRU cells 8 In terms of architectural connectivity,
skip connections can introduced RNN hidden states 40 60 S B Moreover, for deep RNNs, different layers of state variables can be updated at different frequencies
[5]. Correspondingly, we may derive deep RLA modules that formulate stage aggregation on top of layer aggregation.
Recurrent layer aggregation modules
This section first introduces a concept of layer aggregation and some parsimonious models in time series analysis, which together motivate type of light-weighted recurrent layer aggregation (RLA) modules by making use of the sequential structures of deep CNNs.
3.1 Layer aggregation
Consider deep CNN with xf being the hidden features the /th layer and x0 being the input, where is the number of layers, and 1 < L. The CNN is said to have the layer aggregation mechanism if
(1) where f² denotes the transformation that produces new feature map layer t of the CNN, A' represents the aggregated information up to (t 1)th layer, and g" is the corresponding transformation that summarizes these layers. For CNNs without skip connections. according to our definition, they
A' = g'x"(x""
and f"
do not involve layer aggregation since x² =f' fxxx'") ), i.e.. the information in z's with / < is
only available to xf through x²-1, On the other hand, the Hierarchical Layer Aggregation
59 can
be shown to have such mechanism since it satisfies Eq. (I.
is usually more convenient consider an additive form for the summarizing transformation, i.e. (2)
A' = Ei= 9 (z²),
where an additive constant may also be included. and of can be one more convolutions, c.g., set of 1x1-3x3-1x1 convolutions that compose bottleneck block. Consider a DenseNet layer
x¹ Conv3' [Conv1' "Coovv'(Coom)) (Concat(æ" .o t [..,]==))]]
where Conv1' and Conv3' denote 1x1 and 3x3 convolutions, respectively. It can be verified that Convi' 1 (Concat(x*) o (!.,,""!!)) Σtz (on)
(4)
where the kernel weights of Conv form a partition of the weights in Conv1', S As a result, A' - Conv1 1 (a') and x² = Conv3' [A'-1 + Conv (o
ie., it is typical example of the layer aggregation mechanism.
We next consider the are-activated ResNets (2]. On the one hand, according the ResNet update it may be concluded to involve no layer aggregation. On the other hand, by recursively applying the update,
xf = _ + St - (xx-11)
have x² = Σ gi (a²) Iº, which satisfies Eq. since A = Σーー d (ar) 2 and x¹ = A + St- 9t-1 1 (2:-11). is noticeable that şi depends only on i, the ordinal number of previous layers, and we will discuss such a pattern in the next subsection.
3

---

Connection with channel attention be interpreted channel attention across layers. Specifically, the multi-head masked self attention 50
The layer aggregation mechanism in the original DenseNet can
mechanism can be written as
Attention' Concat(headd; head?)) with head; E'= of, x0, for i ,,,, m,
(5)
where query, key and value are all set to be x², sf,r are scalar similarity measures and m is the number of attention heads.
Note that, when considering DenseNet layers, feature maps are tensors instead of matrices. Suppose that all x have the same number channels, denoted by k and then Conv has t. input and m output channels. Denote the th x 1 x 1 x m kernel weights of Conv1' by {W), Let head, be the ith channel in Attention', and 2t be the eth channel of the feature tensor x', then it holds that head?- Lie Wt a(L.o). tee where j [[, c) lk + is index. As result, DenseNet updates coincide with natural extension of Eq.
tensor variables.
However, the above attention mechanism is too meticulous in a CNN context. For example, it will collapse when simply changing Conv 1' in DenseNets to 3x3 convolution. In contrast, the layer aggregation interpretation of DenseNet robust to such a modification.
Our purpose of the discussions above is two-fold: (1) we build the relationship that layer aggregation generalizes channel attention across layers based on the DenseNet architecture; and thus, (2) layer aggregation is not a substitute to channel attention modules, but complementary to them, which inspires us to add the proposed RLA modules to ECAnets in Section
3.2 Sequential structure of layers in deep networks
The complexity of summarizing transformations can drop dramatically from ①1 to (2) while it may still be complicated. Actually DenseNets have been frequently criticized for their redundancy. Note that the layers (xr) ) form sequential structure, and hence this subsection will introduce some related concepts time series analysis such that we can further compress the summarizing transformations at ① and (2). More background on time series is available in the supplementary material A.2.
We first simplify Eq. (2) by assuming that of depends on the lag - / only, and this is a typical setting in time series analysis. If all x's are scalars and all related transformations are linear, we have
x² B,x²"1 +... + 31-1x' + 3,00 -E
It can also be rewritten into Ee 1,x with x" for O, which has a form of autoregressive (AR) models in time series, x² Ei= 3az- + it. with 1 being the additive error.
In time series analysis, the above AR model, which is usually referred to as the AR(co) model, mainly for theoretical discussions, and the commonly used running model is the autoregressive moving average (ARMA) model since it has a parsimonious structure. Consider a simple ARMA(1, 1) model, x. Ba--11 +
7-1-1 It can be verified to have an AR(x0) form of
x² = -r1 + 1t 71t = --1
(ax -8xx 2) + 2
=... = Ei= (3 7)n²
where equation 1 = x² 3x + 7-i==1 is recursively applied for all s < tt 1, while there are only two parameters involved. This motivates us to use similar pattern to construct more parsimonious form for Eq. (6). Let
-==x---- + 7 and x² = 310' 12/12111
(8)
Assuming ho O, then have h²' = Et_ -y and x² 3z B2a 2
which has a form of (6) but involves at most four parameters. It is worthy to point out that J1 plays role similar to that of 7. and Eq. (8) is similar the principle of RNNs (II).
We may also consider the case that of at depends on l only, i.c. x' = -' (1 which leads to a ResNet-type layer. From (8) for s > 0, the information in x=== will be reused when extracting features x' by an amount that depends on lag s only and decays at a rate 7², However, as mentioned in Section 3.1, ResNet can be viewed as layer aggregation in different pattern, and the amount of information in x¹ reused when extracting features x" depends on the ordinal number only. We next conduct small experiment to compare the ResNet-type simplification with that at (

---

Table 1: Classification error of DenseNet and its parameter sharing variants on the CIFAR-10 dataset.
Model
Params
Error (%)
DenseNet
0.80M
5.80+t 0.33
Shared-Lag
0.60M
6.22
Chared-Ordinal
0.60M
6.29
The experiment carried out based on the 100-layer DenseNet- BC (k 12) [88] According to the two simplifications, modify the DenseNet update B into the following versions:
Comparison of two simplifications
(Shared-Lag)
x' = Conv3' (Conv (x--1) ) Convo, + Conv
(9)
(Shared-Ordinal x² Conv3' (ConvI,-1) (x- +... Conv ( l
(10)
which correspond to 6 and ResNet, respectively. The above formulations require Convl, 1 to be compatible with all previous layers. For DenseNet-BC (k 12), the input to dense layers x has
3 + 2 k 27 channels, while all later layers x' 1 < < 16, have k = 12 channels. Thus, keep the convolutions applied to x. unchanged and only modify the other 1x1 convolutions. Detailed explanations of the network structure of are provided in the supplementary material A.2.
Table D presents the comparison of parameter count and accuracy on CIFAR-10. 10. Detailed training settings are the same other experiments on CIFAR-10 in Section + Sharing the 1xl convolutions reduces the number of parameters by 25% (0.20M). while the accuracy only drops by about 0.5%. This observation is consistent with the parameter sharing experiment by (37,, where they found that a ResNet with shared weights retains most of its performance on CIFAR-10, 10. And the better performance of Shared-Lag encourages us pursue a more parsimonious structure along the direction of (6).
Furthermore, we extract the weights learned by the shared 1x1 convolutions and produce Figure 2. following Figure in [28). A first observation is that the strongest signal in Shared-Lag (lag=1) almost twice strong as that in Shared-Ording (layer= Secondly, we spot quick decaying pattern in the plot for Shared-Lag. which similar to the exponential decay pattern an ARMA model RNN [61]. This observation provides empirical evidence to adopt the principle at to design a weightee layer aggregation mechanism.
Lag
80
Residual
Stage
norm
4
txt.
HoWsk
BN
20
2
tanh
3x3, H.Wik
z
A
Lag
Layer
Figure 2: L1 norm of the shared weights.
Figure 3: Diagram of an RLA module.
3.3 Recurrent layer aggregation
By adopting the idea at 8 we introduce recurrent layer aggregation (RLA) mechanism below,
h == g' (h'-1, xf-1) and x' ='= -
(11)
where A' At represents the recurrently aggregated information up to (t 1)th layer, and can be shown to have an additive form of (2) under linearity assumption; see supplementary material A.3. By considering different forms of 9¹, we can achieve various RLA modules to help the main CNN better extract features, and weight sharing can be further used to regularize the layer aggregation. The form of f is mainly determined by the currently used mainstream CNNs, and the combination of aggregated information may be implemented in different ways.
5

---

Table 2: Classification (%) CIFAR- 10 and CIFAR- 100 test sets. Results highlighted in
bold if RLA-Net outperforms our re-implemented baseline model shown in the column "Re-impl.".
Dataset
Model
Params
FLOPs
Original
Re-impl.
RLA-Net
Params
AFLOPs
ResNet-110
1.73M
8.67M
6.37
6.35
5.88
+0.07M
+0.37M
C-10
ResNet-164
1.72M
8.55M
5.46
5.72
4,95
+0.02M
+0.19M
Xception
2.70M
13.54M
6,03
5,78
+0.07M
+0,33M
ResNet-110
1.74M
8.69M
26.88
28.51
27.44
+0.07M
+0.38M
C-100
ResNet-164
1.74M
8.67M
24.33
25.22
23.78
+0.02M
+0.19M
Xception
2.73M
13.77M
24.31
24.00
+0.06M
+0.33M
Figure gives an example for CNN block with a residual connection, and legends are provided at the bottom of Figure Its update can be formulated as
h' = 92 92 (v') + h=1 and =²= + x'_1
where yf = f [Concat 1 ,-,=,,=)) Specifically, inputs x'-1 and h²-1 are combined via concatena- tion and then passed to residual unit extract new features. One copy of this feature map is joined with the residual connection, and the other copy is compressed via the shared Xx1 convolution and then added to 1 Then, the RLA path goes through pre-activated ConvRNN unit comprised of an unshared batch normalization, a tanh activation, and recurrent 3x3 convolution.
Practically, backbone CNNs often comprise of several stages, and RLA modules can cater to various stage structures following the guidelines below:
1. Group main CNN layers/blocks into stages so that feature maps within the same stage have the same resolution, e.g., ResNet-500101/152 has 4 stages and MobileNetV2 has 5 stages.
2. For each stage, define (a) shared recurrent unit, allowing information to compressed differently at various stages, and (b) a shared 1xl convolution, to change the channels of the residual feature maps for adding it to the RLA hidden state.
3. (RLA path) RLA hidden state 0 initialized as zero. Before entering the next stage, the RLA hidden states are downsampled to the correct resolution via average pooling.
4. (RLA output) For classification, the last hidden state is concatenated with the main CNN output and passed to the classifier. For detection and segmentation, RLA output can be discarded.
Experiments
We verify the effectiveness of our RLA module in Figure on image classification, object detection and instance segmentation tasks using CIFAR, ImageNet and MS COCO datasets. All experiments are implemented on four Tesla V100 GPUs. More implementation details and tabular descriptions of network architectures are provided in Section B of the supplementary material.
4.1 CIFAR and analysis
The two CIFAR datasets, CIFAR-10 10 and CIFAR- 100, consist of 60k 32 RGB images of 10 and 100 classes. The training and test sets contain 50k and 10k images. We adopt a train/validation split of 45k/5k and follow the widely used data preprocessing scheme 20 28). All the networks are trained from scratch using Nesterov SGD with momentum of 0.9, la weight decay of 10-4, and batch size of 128 for 300 epochs. The initial learning rate is set to 0.1 and divided by 10 150 and 225 epochs. We choose the model with the highest validation accuracy.
We report the performance each baseline model and its RLA-Net counterpart on CIFAR-10 and CIFAR-100 test sets in T We observe that RLA-Net consistently improves performance across various networks with different depths, blocks and patterns of stages. For the three models Table 2 adding the RLA module introduces about 1% increase in parameters and FLOPs. Meanwhile,
'Our implementation and weights available htttts://ggthhue http:s////sttuub.ccm////aggaaarr

---

Table 3: Comparisons of single-crop error on the ILSRRC 2012 validation set with center crop size 224 x 224. All models trained with crop size of 224 224. The last column report increases in Top-1 accuracy by adding RLA. Trained with some training tricks and architectural refinements in [3 see Section B2.2.
Baseline+Modulee
Params
FLOPs
Top-1 err.
Top-S err.
AAce1
ResNet-50 19
24.37M
3,83G
24.70
7.80
+RLA (Ours)
24.67M
4.17G
22.83
6,58
1.87
+SE 126
26.77M
3.84G
23.29
6.62
CBAM 55
26.77M
3.84G
22.66
6.31
+ ECA +
24,37M
3.83G
22.52
6.32
+ECA+RLA (Ours)
24.67M
4.18G
22.15
6.11
0.37
ResNet-101 19
42,49M
7.30G
23.60
7.10
+RLA (Ours)
42.92M
7.79G
21.48
5.80
2.12
+SE 26
47.01M
7.31G
22.38
6.07
+CBAM 55
47.01M
7.31G
21.51
5.67
+ECA 64
42.49M
7.30G
21.35
5.66
RECA+RLA (Ours)
42.92M
7.80G
21.13
5.61
0.22
ResNet- 152 [19
57.4 57.40M
10.77G
23.00
6.70
+RLA (Ours)
57.96M
11,47G
21.22
5.65
1.78
+SE (26)
63.68M
10.78G
21.57
5.73
+ECA [4
57.41M
10.78G
21.08
5,45
+ECA+RLA (Ours)
$7.96M
11,48G
20.66
5.49
0.42
MobileNetV2 S4 43
3,34M
299.6M
28.36
9.80
+RLA (Ours)
3,46M
351.8M
27.68
9.18
0.68
+SE 26
3.51M
300.3M
27.58
9.33
+ECA 54
3.34M
300
27.44
9.19
+ECA+RLA (Ours)
3,46M
352.4M
27.07
8.89
0.37
ResNet-50-DJ
24.37M
3.83G
20.57
5.31
+RLA (Ours)
24.67M
4.17G
20.09
5.11
0.48
+ECA
24.37M
3.83G
19.88
4.76
+ECA+RLA (Ours)
24.67M
4.18G
19.28
4.63
0.60
ResNet-101-D'
42.49M
7.30G
18.89
4.51
+RLA (Ours)
42.92M
7.79G
18.46
4.25
0.43
+ECA
42.49M
7.30G
18.34
4.01
+ECA+RLA (Ours)
42.92M
7.80G
18.03
3.89
0.31
DenseNet-161 48)
S4
27.35M
7.34G
22.35
6.20
DenseNet-264
32)
31.79M
5.52G
22.15
6.22
ResNet- 54
74.45M
14.10G
21.80
6.00
DPN-68 [58
12.80M
2.50G
23.60
6.90
DPN-92 558
38.00M
6.50G
20.70
5,40
DPN-98 58
61.60M
11.70G
20.20
5.20
AOGNet-
11.90M
2.40G
22.30
6.10
A0GNeI-40M
58
40.30M
8.90G
19.80
4.90
HCGNet-B 58
12.90M
2.00G
21.50
5,90
HCGNet-C
[58
42.20M
7.10G
19.50
4.80
our RLA-Net improves the accuracies of ResNet-110 10 and ResNet-164 by about 0.5% and 1.0% on CIFAR-10 10 and CIFAR-100, respectively.
4.2 ImageNet classification
This subsection reports experiments on the ImageNet LSVRC 2012 dataset [12] We employ our
RLA module on the widely used ResNet [20] architectures and the liightee MobileNetV2 4 as well as their counterparts with the state-of-the-art channel attention ECA modules [54] For
Mob1leNerV2, depthwise separable convolutions are used for the shared 3x3 convolutions to adapt
to the inverted residual blocks of MobileNetV2. When both RLA and ECA modules present, ECA
modules are placed according to [54], and the output of the ECA module is then passed to our RLA

---

Table 4: Object detection results of different methods on COCO val2017.
Methods
Detector
AP
Al'so
APTs
APs
APы
APr
1 AP
ResNet-50
36.4
58.2
39.2
21.8
40.0
46.2
+RLA (Ours)
38.8
59.6
42.0
22.5
42.9
49.5
2,4
+SE
37.7
60.1
40.9
22.9
41.9
48.2
+ECA
38.0
60.6
40,9
48.0
+ECA+RLA (Ours)
39.8
61.2
43.2
23,9
43.7
50.8
1.8
Faster R-CNN
ResNet-101
38.7
60.6
41.9
22.7
43.2
50.4
+RLA (Ours)
41.2
61.8
44.9
23.7
45.7
53.8
2.5
+SE
39.6
62.0
43.1
23.7
44.0
51.4
+ECA
40.3
62.9
44.0
24.5
44.7
51.3
+ECA+RLA (Ours)
42.1
63.3
46.4
54.8
ResNet-50
35.6
55.5
38.2
20.0
39.6
46.8
+RLA (Ours)
37.9
57.0
40.8
22.0
41.7
49.2
2.3
+SE
37.1
57.2
39.9
21.2
40.7
49.3
+ECA
37.3
57.7
39.6
21.9
41.3
48.9
+ECA+RLA (Ours)
38.9
58.7
41.7
23.9
42.7
49.7
1.6
RetinaNet
ResNet-101
37.7
57.5
40.4
21.1
42.2
49.5
+RLA (Ours)
40.3
59.8
43.5
24.2
43.8
52.7
2.6
38.7
59.1
41.6
22.1
50.9
+ECA
39.1
59.9
41.8
22.8
43.4
50.6
+ECA+RLA (Ours)
41.5
61.6
44.4
25.3
45.7
53.8
2.4
module through the shared 1x1 convolution. We compare our RLA-Nets with several state-of-the-art attention- based CNN architectures, including SENet (26), CBAM (55) and ECANet [54].
For training RLA-ResNets, follow the same data augmentation and hyper-parameter settings in 20 28] All the networks are trained using SGD with momentum 0.9, la weight decay of 10-4, and mini-bated size of 256 GPUs. We train models for 120 epochs from scratch and use the weight initialization strategy described in 18]. The initial learning rate set to 0.1 and decreased by a factor of 10 every 30 epochs. For the light-weighted model MobileNetV2, we train the model 2 GPUs within 400 epochs using SGD with weight decay of 4e-5, momentum of 0.9, and a mini-batch size of 96, following the settings in 4 The initial learning rate set to 0.045, decreased by linear decay rate of 0.98. All the parameters and FLOPs shown in the tables are computed by our devices.
Table shows that RLA-Nets obtain better results compared with their original counterparts while introducing small extra costs. Though RLA modules introduce more FLOPs when compared with channel attention modules, find that the extra training time per epoch is almost the same. Specifically, adding SE, ECA or RLA module to ResNet- 101 costs about 15%, 15% or 19% more training time. RLA-ResNets introduce about 1% increase in the number of parameters, leading to respectively. Similar
1.9%. 2. % and 1.8% increases in top-1 accuracy for L...-ReeNee-50/10111 results be observed on the light-weighted MobileNetV2 architectures. Furthermore, combinations of RLA and ECA modules yield remarkably better results, achieving the best performances among these models. These further improvements based ECANets verify our perspective that the functionalities of channel attention modules and our proposed RLA module are complementary.
Last but not least, model with some tricks in 23 popular networks and state-of-the-r networks with delicate designs, including DPN 77. AOGNet [35] and HCGNet (58), which designed to combine the advantages of ResNet and DenseNet. The notable and consistent improvements demonstrate that any tricks or refinements used to further improve the original ResNets be directly applied to model to further improve the model performance. And RLA modules can be treated as one of the beneficial architectural refinements to be adopted by state-of-the-arr CNNs well.
perform remarkably better than many
4.3 Object detection and instance segmentation on MS COCO
To show the transferability and the generalization ability, we experiment our RLA-Net on object detection task using Faster R-CNN 42 Mask R-CNN [2] and RetinaNet (39] as detectors. For Mask R-CNN, also show instance segmentation results. All detectors are implemented by the open

---

Table 5: Object detection and instance segmentation results of different methods using Mask R-CNN on COCO va12017. A Ab and APM denote AP of bounding box detection and instance segmentation.
Methods
A pli
A 5
A P4
A pm
A
Ap
APm
ResNet-50
37.2
58.9
40.3
34.1
53.5
36.2
+RLA (Ours)
39.5
60.1
43,3
35.6
56.9
38,0
2.3
1.5
+SE
38.7
60.9
42.1
35.4
57.4
37.8
NL
38.0
59.8
41.0
34.7
56.7
36.6
+GC Block
39.4
61.6
42.4
35.7
58.4
37.6
+ECA
39.0
61.3
42.1
35.6
57.1
37.7
+ECA+RLA (Ours)
40.6
61.8
44.0
36.5
58.4
38,8
1.6
0.9
ResNet-101
39.4
60.9
43.3
35.9
57.7
38.4
+RLA (Ours)
41.8
62.3
46.2
37.3
59.2
40.1
2.4
1.4
+SE
40.7
62.5
44.3
36.8
59.3
+ECA
41.3
63.1
44.8
37.4
59.9
39.8
+ECA+RLA (Ours)
42.9
63.6
46.9
38.1
60.5
40.8
1.6
0.7
source MMDetection toolkit 4 We employ the same settings in 154 finetune our RLA-Nets on COCO train2017 set. Specifically, the shorter side of input images are resized to 800. We train all detectors within epochs using SGD with weight decay of le-4, momentum of 0.9 and mini-bate size of 8. The learning rate is initialized to 0.01 and is decreased by factor of 10 after 8 and 11 epochs, respectively, i.e., the Ix training schedule. Different from the classification task, we do not pass the hidden states of our RLA module to the FPN. Thus, the gains are solely from the more powerful representations learnt the main CNNs.
We report the results on COCO v 7 set by standard COCO metries of Average Precision (AP). Tables and 3 report the performances of the object detectors. We can observe that our RLA-Net outperforms the original ResNet by 2.4% and 2.5% in terms of AP for the networks of 50 and 101 layers, respectively. More excitingly, our RLA-ResNet-500 outperforms ResNet- 101 on these three detectors. In particular, our RLA module achieves more gains in APrs and APL. indicating the high accuracy and effectiveness for large objects. Remarkably, exploiting RLA on ECANets can surpass all other models with large margins. We hypothesize that, with the help of RLA modules, positional information from low-level features better preserved the main CNN, leading to these notable improvements. In summary, the results in Tables 4 and demonstrate that our RLA-Net can well generalize to various tasks with extraordinary benefits on the object detection task.
4.4
Ablation study
We conduct an ablation study on ImageNet with ResNet-50 as the baseline network. To validate the effectiveness of the detailed inner structure, ablate the important design elements of RLA module and compare the proposed structure with the following variants:
(a) channels increased by 32;
(b) RLA without parameter sharing:
(c) RLA without the information flow from RLA to the main CNN (as in 41
(d) RLA with ConvL.STM cell;
(c) RLA with the post-activated hidden unit; and
(f-j) RLA with different connectivity between two paths, see Figure
The results are reported in Table We first demonstrate that the improvement in accuracy is not fully caused by the increment in network width by comparing to the variant with 32 additional channels in each layer. We then investigate the necessity of recurrence in our module by comparing it with its fully unshared counterpart. Results show that recurrent design can reduce parameter count and slightly increase accuracy at the same time. Removing the information flow from RLA to the main CNN results a decrease model performances, validating that the information exchange between RLA and the main CNN is must. Furthermore, using ConVLSTM cells may be unnecessary since

---

203
(+1. ours)
(x2)
(+4)
[v5)
(v6)
Concatenatinn
Copy
Elementwise Sum
AEF
Convolatio with
temel sire
Shored convotation with karnel
Figure 4: RLA modules with different types of connections between two paths.
Table Classification errors on the ImageNet validation set using ResNet-50 as the baseline model.
Model
Params (M)
FLOPs (G)
Top-1 err.
Top-5 err.
ResNet-50
24.37
3.83
24.70
7.80
(a) channel
25.55
4.24
23.54
6.72
(b) RLA-vI (k 32, unshared)
25.12
4.17
22.93
6.61
(c) RLA-vI (A: 32, no exchange)
24.56
4.11
23.42
6.90
(d) RLA-v1 (& 32, STM)
24.92
5,00
22.92
6.53
(e) RLA-v1 (k 32, PostAct.)
24.67
4.17
23.11
6.65
Proposed RLA-v1 (k: 32)
24.67
4.17
22,83
6.58
(0) RLA-v2 v (k 32)
24.67
4.17
23.24
6.61
(g) RLA-v3 (k- = 32)
24.67
4.17
22.95
6.54
(h) RLA-v4 (k- 32)
24.67
4.17
23.36
6.72
(i) RLA-v5 (k 32)
24.67
4.17
23.30
6.60
(i) RLA-v6 =32)
24.67
4.17
23.50
6.82
each stage only has to 6 time steps ResNet-50, and the post-activated recurrent unit is not very compatible with the pre-activated network.
We also compare different integration strategies to combine our RLA module with the main CNN. In addition to the proposed design, we consider five variants (v2-v6), as depicted in 4 The performances of the variants are reported in Table We observe that the variants vl, v3 and v5 perform similarly well, as they all integrate new information from the main CNNs before entering into the recurrent operation. Compared with variants v3 and v5, v1 obtains the learned residual as new information, including less redundancy. The performance of our RLA modules is relatively robust the integration strategies, as long as new information added before the recurrent operation.
In summary, the proposed RLA module structure in Figure Eegure3achhin Jaciiees the smallest top-1 error and very competitive top-5 error with the minimum number of additional parameters. This structure can strike a good balance between the increase of computational cost and performance gain. We also conduct more comprehensive ablation study on the CIFAR-10 dataset with ResNet-16A the baseline network. Due to limited space, we present details of the ablation studies and the investigated module structures in the supplementary material B.4.
Conclusion and future work
This paper proposes recurrent layer aggregation module that is light-weighted and capable of providing the backbone CNN with sequentially aggregated information from previous layers. Its potential has been well illustrated by the applications on mainstream CNNs with benchmark datasets. On the other hand, we mainly focused on controlled experiments to illustrate the effectiveness and compatibility of the RLA modules with settings adopted from existing work. Our first future work is to consider hyperparameter tuning, advanced training tricks (3 2
or architecture search [49] Moreover, terms of RLA design and application, it is also an interesting direction to apply the stage level top of the layer level, which leads to deep RLA modules.
10

---

Acknowledgments and Disclosure of Funding
We thank the anonymous reviewers for their constructive feedback and Jinxu Zhao for technical support. This project is supported by National Supercomputer Center in Guangzhou, China.
References
[1] Konstantinos Batsos and Philippos Mordohai. RecResNet: A recurrent residual CNN architec- ture for disparity map enhancement. In 2018 International Conference on 3D Vision (3DV), pages 238-247 IEEE, 2018
[2] Irwan Bello, William Fedus, Xianzhi Du, Ekin D Cubuk, Aravind Srinivas, Tsung- Yi Lin, Jonathon Shlens, and Barret Zoph. Revisiting ResNets: Improved training and scaling strategies. arXiv preprint arXhv:2103.07579 2021.
[3] Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark Hasegawa-Johnson, and Thomas Huang. Dilated recurrent neural networks. In Advances in Neural Information Processing Systems, 2017.
[4] Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai. Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint
2019.
[5] Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei Liu, and Tat-Seng Chua. SCA-CNN: Spatial and channel wise attention in convolutional networks for image captioning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5659-5667, 2017.
[6] Sheng Chen, Yang Liu, Xiang Gao, and Zhen Han. Mobilefacceeets: Efficient CNNs for accurate real-time face verification on mobile devices. Chinese Conference on Biometric Recognition, pages 428-438. Springer, 2018
[7] Yunpeng Chen, Jianan Li, Huaxin Xiao, Xiaojie Jin, Shuicheng Yan, and Jiashi Feng. Dual path networks. In Advances in Neural Information Processing Systems, 2017.
[8] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder- decoder for statistical machine translation. In EMNLP. 2014.
[9] Myungsub Choi, Heewon Kim, Bohyung Han, Ning Xu, and Kyoung Mu Lee. Channel attention all you need for video frame interpolation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10663-10671, 2020.
[10] François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1251-1258, 2017.
[11] Jerome Connor, R Douglas Martin, and Les E Atlas. Recurrent neural networks and robust time series prediction. IEEE Transactions on Neural Networks, 5(2):240-))) 2244444,, 1994.
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 348-255 IEEE, 2009.
[13] Robert DiPictro, Christian Rupprecht, Nassir Navab, and Gregory D Hager. Analyzing and exploiting recurrent neural networks for long-term dependencies. arXiv preprint arXiv: /702.0700.07880 2017.
[14] Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2625-2634, 2015.
11

---

[15] Salah EI Hihi and Yoshua Bengio. Hierarchical recurrent neural networks for long- teng-term dependencies. In Advances in Neural Information Processing Systems, pages 493-499, 1996.
[16] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3146-3154, 2019.
[17] Klaus Greff, Rupesh K Srivastava, and Jürgen Schmidhuber. Highway and residual networks learn unrolled iterative estimation. In International Conference on Learning Representations, 2017.
[18) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Sur- passing human-level performance on imagenet classification. In Proceedings of the IEEF/CVF International Conference Computer Vision (ICCV), pages 1026-1034, 2015.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. /ttp:::///ttuu.ccom/// wtw.aatw...canaaaaaaiiaaaaeeeeee
residuaaleeeiiuuaa-aeeeea--ll networks 2015.
[20] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 630-645. Springer, 2016.
[22] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pages 2961-2969, 2017.
[23] Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks the
for image classification with convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 558-567, 2019.
[24] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9 (8):1735-1780, 1997.
[25] Andrew Howard, Mark Sandler, Grace Chu, Liang- Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for MobileNetV3. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1314-1324, 2019.
[26] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitatie networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 7132-7141, 2018.
[27] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinherger. Deep networks with stochastic depth. In Proceedings of the European Conference on Computer Vision (ECCV), pages 646 661. Springer, 2016.
[28] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4700-4708, 2017.
[29] Gao Huang, Zhuang Liu, Geoff Pleiss, Laurens Van Der Maaten, and Kilian Weinberger. Convolutional networks with dense connectivity. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.
[30] Zhongzhan Huang, Senwei Liang, Mingfu Liang, and Haizhao Yang. DIANet: Dense-and- implicit attention network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4214-4214, 2020.
[31] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pages 448-456 PMLR, 2015.
12

---

[32] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep con- volutional neural networks. Advances in Neural Information Processing Systems, volume 25, pages 1097-1105, 2012.
[33] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521555))44755) 2015.
[34] Hanchao Li, Pengfei Xiong, Jie An, and Lingxue Wang. Pyramid attention network for semantic segmentation. In British Machine Vision Conference, 2018.
[35] Xilai Li, Xi Song, and Tianfu Wu. Aognets: Compositional grammatical architectures for deep learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2019.
[36] Ming Liang and Xiaolin Hu. Recurrent convolutional neural network for object recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3367-3375, 2015.
[37] Qianli Liao and Tomaso Poggio. Bridging the gaps between residual learning, recurrent neural networks and visual cortex. arXiv preprint vrXiv:160.00360 2016.
[38] Bee Lim and Kyoung Mu Lee. Deep recurrent ResNet for video super-resolution In 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 1452-1455. IEEE, 2017.
[39] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming Hc, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2980-2988, 2017.
[40] Tsungnan Lin, Bill G Horn,, Peter Tino, and C Lee Giles. Learning long-term dependencies in narx recurrent neural networks. IEEE Transactions on Neural Networks, (600:1329-1333 1996.
[41] Joel Moniz and Christopher Pal. Convolutional residual memory networks. arXiv preprint 16Xiv:1666.05222 2016.
[42] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, NIPS' 15, page 91-99, Cambridge, MA, USA, 2015. MIT Press
[43] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Liang-Chich Chen. MobileNetV2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4510-4520, 2018
[44] Baoguang Shi, Xiang Bai, and Cong Yao. An end-to-end trainable neural network for image- hased sequence recognition and its application to scene text recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(1 (9(11):22)) 1):2998 2304, 2016.
[45] Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In Advances in Neural Information Processing Systems, 2015.
[46] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1-9, 2015.
[47] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 281 1826826, 2016.
[48] Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander Alemi. Inception-v4, Inception- Res and the impact of residual connections on learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
13

---

[49] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. International Conference on Machine Learning, pages 6105-614 6114. PMLR, 2019.
[50] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Informa- tion Processing Systems, pages 5998-6008, 2017.
[51] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 3156-3164, 2015.
[52] Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang. Residual attention network for image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3156 3164, 2017.
[53] Jiang Wang, Yi Yang, Junhua Mao, Zhiheng Huang. Chang Huang, and Wei Xu. CNN-RNN: A unified framework for multi-label image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2285-2294, 2016.
[54] Qilong Wang, Banggu Wu, Pengfei Zhu, Peihua Li, Wangmeng Zuo, and Qinghua Hu. ECA- Net: Efficient channel attention for deep convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[55] Sanghyun Woo, Jongchan Park, Loon-Young Lee, and In So Kweon. CBAM: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV), pages 3-19, 2018.
[56) Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1492-15000 2017.
[57] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning, pages 2048-24480002 2054--057. PMLR, 2015.
[58] Chuanguang Yang, Zhulin An, Hui Zhu, Xiaolong Hu, Kun Zhang, Kaiqiang Xu, Chao Li, and Yongjun Xu. Gated convolutional networks with hybrid connectivity for image classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12581-12588, 2020.
[59] Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 2403-2412, 2018
[60] Saizheng Zhang, Yuhuai Wu, Tong Che, Zhouhan Lin, Roland Memisevic, Ruslan Salakhutdi- nov, and Yoshua Bengio. Architectural complexity measures of recurrent neural networks. In Advances in Neural Information Processing Systems, 2016.
[61] Jingyu Zhao, Feiqing Huang, Jia Lv, Yanjie Duan, Zhen Qin, Guodong Li, and Guangjian Tian. Do RNN and LSTM have long memory? In International Conference on Machine Learning, pages 11365-1 1375. PMLR, 2020.
[62] Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4995-5004, 2016.
14


Reviews:
[{"review_id": "psJROffdOJr", "reviewer": "Reviewer_Amr2", "summary": "A CNN architecture with two network paths (one is non-recurrent and another is recurrent) is introduced in this paper. The features from the two paths are aggregated through the RLA module (with residual connection) and redistributed to the individual paths. This brings performance improvements on image classification, object detection, instance segmentation at very small increases of parameter and computation costs.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n+ Lightweight design and works well with modern CNN architectures.\n+ Good discussion of existing work.\n+ Comprehensive experiments on several tasks with good results. RLA can be combined with ECA to achieve stronger performance.\n\nWeaknesses\n- The writing for the motivation behind the design of RLA is overly lengthy and not easy to follow. This part does not have major implications on RLA-Net as a whole yet it occupies a large portion of the paper. It is only relevant to Eq.8 in Sec 3.3 which is similar to vanilla RNN (already well known to the community).\n- \"applying RLA modules to CNNs is more than a simple combination of RNN and CNN (e.g., [39])\" - Since RLA is different from [39] in terms of mutual information exchange and spatial feature preservation, there should be experiments showing the effects of these differences. Otherwise, it would be hard to justify the technical significance of RLA.\n- The motivation behind using recurrent/shared weights for the second network path is not clear. There is nothing a recurrent network can model that a non-recurrent network cannot model. It can be simply replaced by a non-recurrent network. It seems to me that using recurrent/shared weights is just for keeping the parameter counts low. \n\nMinor issue\nLine 132: \"DensNets\" -> \"DenseNets\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "YG6CNKrQ_9V", "reviewer": "Reviewer_ngdy", "summary": "This paper is inspired by the layer aggregation concept which is also used in DenseNet to better reuse the information from previous layers and extract features at the current layer. Then the authors propose a light-weighted module, called recurrent layer aggregation (RLA), and prove it is compatible with many common structures in ResNets, Xception and MobileNetV2. Finally, they use the controlled experiments to show the effectiveness of the RLA modules on classification and detection tasks.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper has the good motivation to discuss better utilize the common information among adjacent layers to improve the feature extraction capability of the deep networks.\nFrom the clarity aspect, I would suggest the authors to put the Figure B.4: (RLA modules with different types of connections) from the supplementary to the main paper. Just use the formula to express the module structure and its variants is not as clear as the images.\nThe main concern about this paper is the real performance during training and inference. Because of the introducing the recurrent layer aggregation module, layers need to also consider the previous states of adjacent layers. So similar to RNN modules, it is not friendly to the parallel acceleration like GPU device. I am curious about the real performance number when adding the RLA module.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors seem not to provide enough limitations and potential negative social impact of their work.\nI suggest the authors to provide a description of the potential performance regression due to the introduction of the RLA module.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {"ethical_concerns": "This paper only did the ablation study on the CIFAR-10 dataset with ResNet-164 as the backbone network. Because the CIFAR-10 dataset is very small, while ResNet-164 is more capable for a larger dataset. There are maybe over-fitting situations and lead to bias in the ablation conclusion."}}, {"review_id": "Fk3-JpQHp7h", "reviewer": "Reviewer_uWZq", "summary": "The paper proposed a recurrent layer aggregation (RLA) structure. The RLA structure proposes a recurrent module to aggregating information from the previous layers. The proposal is inspired by an additive form of describing the information aggregations in deep neural network and its similarity to autoregressive models. The structure is compatible with many popular CNN models including the ResNets, Xception and MobileNet. Extensive sets of experiments have been conducted to demonstrate the effectiveness of the proposed method on different vision tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n1. The proposed RLA is a class of modules compatible with different popular neural networks and other methods like channel attention. Experiment shows consistent performance improvements on different backbone models and tasks over model without RLA.\n2. The motivation of the proposed RLA based on time series model is strong and well supported by experiment results in Table 2 and Fig. 2.\n3. The structure of the paper is well-organized and easy to follow.\n\nCons\n\nThere’re no significant shortcomings in this work but two additional comments that might help improve the paper. \n1. The paper shows the robustness of performance improvements to different RLA vairants on cifar10 datasets. Extension of such experiments to ImageNet and MSCOCO with similar results would support this conclusion significantly better.\n2. The purpose of the paragraphs about interpretation of densenet as channel attention and the connection between this interpretation and motivation of RLA are not clear. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There’s no discussion of limitations or negative social impacts in the paper. In general, concerns about negative social impacts of vision models are related to security issues like vulnerability to adversarial attacks and privacy issues due to the abuse of these models.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "7s7qfZ64P_", "reviewer": "Reviewer_jn5x", "summary": "This paper proposes a module called Recurrent Layer Aggregation (RLA) to improve the performance of general CNN backbones. This module is motivated by the analysis of both layer aggregation and the time series method. Experiments on tasks like classification, detection, and segmentation validate the effectiveness of this RLA module.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nThe proposed RLA seems new to me, especially considering its relation with the time series analysis. The recurrent module is not simple usage of the off-the-shelf modules like LSTM or GRU but a custom one, which is interesting.\n\nQuality:\nThe paper is technically sound. The proposed module is motivated by both layer aggregation and the time series method. The implementation of RLA is just a slight variant of the inspired formula Eq. (8), which is clean.\nHowever, despite the custom recurrent module implementation, I think there should be an experimental comparison to show how the LSTM and GRU perform when they are used as the recurrent aggregation module (as Eq. 11). This will show us whether the custom recurrent module in the RLA is required or just an option.\n\nClarity:\nThere are some issues to address.\n1. Line 138: I think there should be some more introductions about the autoregressive (AR) models and autoregressive moving average (ARMA) models, e.g. what they are handling, what their benefits are.\n2. Line 116-129: Is there any specific reason to show the relationship between the layer aggregation and the channel attention? Does this motivate the proposed method?\n3. Line 139: What is the meaning of the additive error $i^t$?\n4. Line 191: Are the batch-norm layers share weights in the recurrent layers?\n\nSignificance:\nI believe this work shows a certain level of significance as it highlights the importance of layer aggregation in a deep network, and the implementation of using a recurrent aggregation mechanism is new and simple.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a new design for aggregating information from shallower to deeper layers of a neural network (i.e., an alternative to ResNet or DenseNet-style connections).  Following the author response and discussion, 1 reviewer rating is negative, 3 are positive, with overall sentiment being marginally positive.  The paper presents a clear design and provides sufficient experimental evidence of benefits.  A downside is that performance benefits, even if consistent, are of an incremental nature over existing network designs.
Decision: Accept (Poster)