Recurrence along Depth: Deep Convolutional Neural Networks with Recurrent Layer Aggregation

This paper introduces a concept of layer aggregation to describe how information from previous layers can be reused to better extract features at the current layer . While DenseNet is a typical example of the layer aggregation mechanism , its redundancy has been commonly criticized in the literature . This motivates us to propose a very light-weighted module , called recurrent layer aggregation ( RLA ) , by making use of the sequential structure of layers in a deep CNN . Our RLA module is compatible with many mainstream deep CNNs , including ResNets , Xception and MobileNetV2 , and its effectiveness is verified by our extensive experiments on image classification , object detection and instance segmentation tasks . Specifically , improvements can be uniformly observed on CIFAR , ImageNet and MS COCO datasets , and the corresponding RLA-Nets can surprisingly boost the performances by 2-3 % on the object detection task . This evidences the power of our RLA module in helping main CNNs better learn structural information in images . 1 Introduction . Convolutional neural networks ( CNNs ) have achieved notable success in computer vision tasks , crediting to their ability to extract high-level features from input images . Due to rapid growth in the depth of CNNs in recent years , the problem of how to pass information efficiently through layers often arises when designing deep architectures . Residual connections [ 20 , 21 ] , or skip connections , are now cornerstone components that act as information pathways to give layers direct access to previous ones and make training feasible with hundreds of layers . In a deep feature hierarchy , higher-level features learned by the network are built upon simpler ones , but not necessarily on the layer right before it [ 17 ] . This conjecture can be supported by the stochastic depth training method [ 27 ] , where layers randomly gain access to previous ones . Furthermore , it is discovered that entire layers can be removed from ResNets without impacting the performance . This observation incubated the dense connectivity in DenseNets [ 28 , 29 ] , where layers in a stage have direct access to all previous layers through fully connected skip connections . We introduce a concept of layer aggregation to systematically study network designs on feature reuse , and a typical example is DenseNet . However , within a densely-connected stage , the number of network parameters grows quadratically with respect to the number of layers . These highly redundant parameters may extract similar features multiple times [ 7 ] , and limit the number of channels to store new information . This motivates us to propose a very light-weighted recurrent layer aggregation ( RLA ) module with much fewer parameters . Specifically , we use recurrent connections to replace dense connectivity and achieve a parameter count independent of the network depth . The RLA 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . module preserves the layer aggregation functionality , and it can be added to existing CNNs for better feature extraction . Our RLA module is compatible with many CNNs used today . A schematic diagram is provided in Figure 1 , where x represents hidden layers in a deep CNN and h represents hidden states in RLA modules . LeCun et al . [ 33 ] pointed out that unfolded RNNs can be seen as very deep feedforward networks where all the hidden layers share the same weights . When examined alone , an RLA module behaves similar to an unfolded RNN with layers of a deep CNN as its inputs . But applying RLA modules to CNNs is more than a simple combination of RNN and CNN ( e.g. , [ 41 ] ) . As the key idea behind layer aggregation is to provide the current layer with a version of layer history , information exchange between main CNN and RLA modules is a must . This results in a connection from the hidden unit of the RNN back to its inputs , which is hardly observable in other deep architectures . Moreover , we do not use global average pooling when passing information from the main CNN to RLA , so that the historical information is enriched and contains spatial information . Despite its recurrent design , a convolutional-RNN-based RLA module can be easily implemented using standard convolutions with parameter sharing . Empirically , RLA modules are computationally light-weighted and impose only a slight increase in model parameters and computational cost . In terms of functionality , it serves purposes beyond channel attention and can be applied on top of channel attention modules . We perform extensive experiments across a variety of tasks using different deep CNN architectures , including ResNets [ 20 , 21 ] , ECANets [ 54 ] , Xception [ 10 ] and MobileNetV2 [ 43 ] . Our experimental results show that RLA consistently improves model performances on CIFAR , ImageNet and MS COCO datasets . On MS COCO , our RLA module can achieve 2-3 % gains in terms of average precision , which significantly outperforms other state-of-the-art networks with remarkable performances . The main contributions of our work are below : ( 1 ) We introduce a definition of layer aggregation for analyzing CNN architectures . ( 2 ) We propose a novel recurrent layer aggregation module , with motivation from the layer aggregation mechanism and time series analysis . ( 3 ) We investigate detailed RLA module designs through ablation study and provide guidelines on applying RLA modules . ( 4 ) We show the effectiveness of RLA across a broad range of tasks on benchmark datasets using several popular deep CNN architectures . 2 Related work . CNN-RNN hybrid models Deep CNNs with RLA modules should be easily distinguishable from other CNN-RNN hybrid networks , as RLA features information exchange between CNN and RNN at each layer and the corresponding recurrent update . In visual description tasks , RNNs are adopted to process the outputs of CNNs for sequence generation [ 14 , 51 , 44 , 53 , 5 ] . For video super-resolution , convolutional LSTMs can be used first to capture the temporal dependence between frames and then pass outputs to CNNs [ 38 ] . There are only a few papers that apply the concept of recurrence to CNNs . In RecResNet [ 1 ] , a whole CNN is used recurrently for disparity map refinement . Moreover , in [ 36 ] , convolutional layers are used recurrently to produce `` deep '' CNNs before the debut of ResNets . Dense-and-Implicit Attention Network [ 30 ] possesses a similar high-level CNN-RNN combination structure compared with ours , while their design is within the attention framework . CNNs with branching paths CNNs with our RLA module have similar structures compared to those with branching paths , e.g. , Inception networks [ 46 , 31 , 47 , 48 ] , or networks with grouped convolutions , e.g. , AlexNet [ 32 ] and ResNeXt [ 56 ] . The main difference is that our RLA module has shared convolutional kernels along depth . When RLA is applied to ResNets , the resulting networks can be viewed as compressed Dual Path Networks [ 7 ] . Attention mechanisms in CNNs Attention mechanisms have been incorporated into CNNs mainly in the form of spatial attention and channel attention . Successful applications include image captioning [ 57 , 5 ] , visual question answering [ 62 ] , image classification [ 52 , 16 ] , semantic segmentation [ 34 ] , face verification [ 6 ] , and video frame interpolation [ 9 ] . An outstanding design of channel attention would be the Squeeze-and-Excitation ( SE ) block [ 26 ] , which is later adopted by MobileNetV3 [ 25 ] and EfficientNet [ 49 ] and developed into compressed variants [ 54 ] . The proposed layer aggregation mechanism has a connection with channel attention across multiple layers , and meanwhile , it can be applied to networks with channel attention modules . RNN variants Since RLA modules adopt the form of a convolutional RNN , RNN variants give rise to different RLA designs . For example , the hidden feature maps in RLA can also be updated by the convolutional counterparts of LSTM [ 24 , 45 ] or GRU cells [ 8 ] . In terms of architectural connectivity , skip connections can be introduced to RNN hidden states [ 40 , 60 , 3 , 13 ] . Moreover , for deep RNNs , different layers of state variables can be updated at different frequencies [ 15 ] . Correspondingly , we may derive deep RLA modules that formulate stage aggregation on top of layer aggregation . 3 Recurrent layer aggregation modules . This section first introduces a concept of layer aggregation and some parsimonious models in time series analysis , which together motivate a type of light-weighted recurrent layer aggregation ( RLA ) modules by making use of the sequential structures of deep CNNs . 3.1 Layer aggregation . Consider a deep CNN with xt being the hidden features at the tth layer and x0 being the input , where L is the number of layers , and 1 ≤ t ≤ L. The CNN is said to have the layer aggregation mechanism if At = gt ( xt−1 , xt−2 , . . . , x0 ) and xt = f t ( At−1 , xt−1 ) , ( 1 ) where f t denotes the transformation that produces new feature map at layer t of the CNN , At represents the aggregated information up to ( t−1 ) th layer , and gt is the corresponding transformation that summarizes these layers . For CNNs without skip connections , according to our definition , they do not involve layer aggregation since xt = f t ( xt−1 ) , i.e. , the information in xls with l < t− 1 is only available to xt through xt−1 . On the other hand , the Hierarchical Layer Aggregation [ 59 ] can be shown to have such mechanism since it satisfies Eq . ( 1 ) . It is usually more convenient to consider an additive form for the summarizing transformation , i.e . At = ∑t−1 l=0 g t l ( x l ) , ( 2 ) where an additive constant c may also be included , and gtl can be one or more convolutions , e.g. , a set of 1x1-3x3-1x1 convolutions that compose a bottleneck block . Consider a DenseNet layer xt = Conv3t [ Conv1t ( Concat ( x0 , x1 , ... , xt−1 ) ) ] , ( 3 ) where Conv1t and Conv3t denote 1x1 and 3x3 convolutions , respectively . It can be verified that Conv1t ( Concat ( x0 , x1 , ... , xt−1 ) ) = ∑t−1 l=0 Conv1 t l ( x l ) , ( 4 ) where the kernel weights of Conv1tl form a partition of the weights in Conv1 t. As a result , At = ∑t−1 l=0 Conv1 t+1 l ( x l ) and xt = Conv3t [ At−1 + Conv1tt−1 ( x t−1 ) ] , i.e. , it is a typical example of the layer aggregation mechanism . We next consider the pre-activated ResNets [ 21 ] . On the one hand , according to the ResNet update xt = xt−1 + gt−1 ( x t−1 ) , it may be concluded to involve no layer aggregation . On the other hand , by recursively applying the update , we have xt = ∑t−1 l=0 gl ( x l ) + x0 , which satisfies Eq . ( 1 ) since At = ∑t−1 l=0 gl ( x l ) + x0 and xt = At−1 + gt−1 ( xt−1 ) . It is noticeable that gl depends only on l , the ordinal number of previous layers , and we will discuss such a pattern in the next subsection . Connection with channel attention The layer aggregation mechanism in the original DenseNet can be interpreted as channel attention across layers . Specifically , the multi-head masked self-attention [ 50 ] mechanism can be written as Attentiont = Concat ( headt1 , ... , head t m ) with head t i = ∑t−1 l=0 s t l , i x l , for i = 1 , ... , m , ( 5 ) where query , key and value are all set to be xt , stl , is are scalar similarity measures and m is the number of attention heads . Note that , when considering DenseNet layers , feature maps are tensors instead of matrices . Suppose that all xl ’ s have the same number of channels , denoted by k , and then Conv1t has tk input and m output channels . Denote the tk × 1 × 1 × m kernel weights of Conv1t by { W tj , i } . Let head t i be the ith channel in Attentiont , and xlc be the cth channel of the feature tensor x l , then it holds that headti = ∑ l , c W t j ( l , c ) , i x l c , where j ( l , c ) = lk+ c is an index . As a result , DenseNet updates coincide with a natural extension of Eq . ( 5 ) to tensor variables . However , the above attention mechanism is too meticulous in a CNN context . For example , it will collapse when simply changing Conv1t in DenseNets to a 3x3 convolution . In contrast , the layer aggregation interpretation of DenseNet is robust to such a modification . Our purpose of the discussions above is two-fold : ( 1 ) we build the relationship that layer aggregation generalizes channel attention across layers based on the DenseNet architecture ; and thus , ( 2 ) layer aggregation is not a substitute to channel attention modules , but complementary to them , which inspires us to add the proposed RLA modules to ECAnets in Section 4 .

Summary: This paper proposes a module called Recurrent Layer Aggregation (RLA) to improve the performance of general CNN backbones. This module is motivated by the analysis of both layer aggregation and the time series method. Experiments on tasks like classification, detection, and segmentation validate the effectiveness of this RLA module.