RMNet: Equivalently Removing Residual Connection from Networks

1 INTRODUCTION . Since AlexNet ( Krizhevsky et al. , 2012 ) came out , the state-of-the-art CNN architecture has become deeper and deeper . For example , AlexNet only has 5 convolutional layers , it is soon extended to 19 and 22 layers by VGG network ( Simonyan & Zisserman , 2015 ) and GoogLeNet ( Szegedy et al. , 2015 ; 2016 ; 2017 ) , respectively . However , deep networks that simply stack layers are hard to train because of the gradient vanishing and exploding problem — as the gradient is back-propagated to earlier layers , repeated multiplication may make the gradient infinitely small or large . This problem has been largely addressed by the normalized initialization ( LeCun et al. , 2012 ; Glorot & Bengio , 2010 ; Saxe et al. , 2013 ; He et al. , 2015 ) and intermediate normalization layers ( Ioffe & Szegedy , 2015 ) , which enable networks with tens of layers converging . Meanwhile , another degradation problem has been exposed : with the increase of the network depth , accuracy gets saturated and then degrades rapidly . ResNet ( He et al. , 2016a ; b ) addresses the degradation problem and achieves 1K+ layers models by adding a residual connection from the input of a block to the output . Instead of hoping each stacked layer directly fit a desired underlying mapping , ResNet lets these layers fit a residual mapping . When the identity mapping is optimal , it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers . As ResNet gains more and more popularity , researchers propose many new architectures , e.g . ( Hu et al. , 2018 ; Li et al. , 2019 ; Tan & Le , 2019 ; 2021 ) , based on ResNet and interpret the success from different aspects . However , ResDistill ( Li et al. , 2020 ) points that the residual connections in ResNet-50 account for about 40 percent of the entire memory usage on feature maps , which will slow down the inference process . Besides , residual connections in the network are not friendly for ‘ network pruning ’ ( Ding et al. , 2021b ) . By contrast , the VGG-like model , also called the plain model in this paper , has only one path and is fast , memory economical , and parallelization friendly . RepVGG ( Ding et al. , 2021b ) proposes a powerful approach to remove residual connections via re-parametrization at inference . Specifically , RepVGG adds 3×3 convolution , 1×1 convolution , and identity together for training . The BN layer was added at the end of each branch and the ReLU was added after the addition operation . During training , RepVGG only needs to learn the residual mapping , while in the inference stage , re-parameterization was utilized to transform the basic block of RepVGG into a stack of 3× 3 convolution layers plus ReLU operation , which has a favorable speed-accuracy trade-off compared to ResNet . However , we find that the performance of RepVGG suffers severe degradation when the network goes deeper , we provide experiments to verify this observation in Section 4.1 . In this paper , we introduce a novel approach named RM operation , which can remove the residual connection with non-linear layers inside it and keep the result of the model unchanged . RM operation reserves the input feature maps by the first convolution layer , BN Layer and ReLU Layer , then merges them with the output feature maps by the last convolution in the ResBlock . With this method , we can equivalently convert a pre-trained ResNet or MobileNetV2 to an RMNet model to increase the degree of parallelism . Besides , the architecture of RMNet makes it friendly for pruning since there are no residual connections . We summarized our main contributions as follows : • We find that removing residual connection by re-parameterization methods has its limitation , especially when the depth of the model is large . It lies in the non-linear operation that can not be put inside of residual connection for re-parameterization . • We propose a novel method named RM operation which can remove residual connection across non-linear layers without changing the output by reserving the input feature maps and merging them with the output feature maps . • With RM operation , we can convert the ResBlocks to a stack of convolutions and ReLUs , which achieves a better accuracy-speed trade-off and make it very friendly for pruning . 2 RELATED WORK . Advantage of Residual Networks over Plain Model . He et al . ( He et al. , 2016a ) introduce ResNet for addressing the degradation problem . In addition , the gradient of layers in PreActResNet ( He et al. , 2016b ) does not vanish even when the weights are arbitrarily small , which leads to nice backward propagation properties . ( Weinan , 2017 ) shows the connection between dynamical systems and deep learning . ( Lu et al. , 2018 ) regards ResNet as an Euler discretization of Ordinary Differential Equations ( ODE ) . NODEs ( Chen et al. , 2018 ) replaces residual block with neural ODEs for better training extremely deep networks . ( Balduzzi et al. , 2017 ) identifies shattered gradients problems making the training of DNNs difficult . They show that the correlation between gradients in the standard feedforward network decays exponentially with depth . In contrast , the gradients in ResNet are far more resistant to shattering , decaying sublinearly . ( Veit et al. , 2016 ) shows that residual networks can be seen as a collection of many paths of differing lengths . From this view , n-blocks ’ ResNet have O ( 2n ) implicit paths connecting input and output , and that adding a block doubles the number of paths . DNNs Without Residual Connection . A recent work ( Oyedotun et al. , 2020 ) combined several techniques including Leaky ReLU , max-norm , and careful initialization to train a 30 layer plain ConvNet , which could reach 74.6 % top-1 accuracy , 2 % lower than their baseline . A mean-field theory-based method ( Xiao et al. , 2018 ) is proposed to train extremely deep ConvNets without branches . However , 1K-layer plain networks only achieve 82 % accuracy on CIFAR-10 . ( Balduzzi et al. , 2017 ) use a ” looks linear ” initialization method and CReLUs ( Shang et al. , 2016 ) to train a 198 layer plain model to 85 % accuracy on CIFAR-10 . Though the theoretical contributions were insightful , the models were not practical . One can get DNNs without residual connection in a reparameterization manner : re-parameterization ( Ding et al. , 2019 ; 2021a ) means using the params of a structure to parameterize another set of params . Those approaches first train a model with residual connection and remove residual connection by re-parameterization when inference . DiracNet ( Zagoruyko & Komodakis , 2017 ) uses the addition of an identity matrix and convolutional matrix for forwarding propagation , the parameters of convolution need only to learn the residual function like ResNet . After training , DiracNet adds the identity matrix to the convolutional matrix and uses the reparameterized model for inference . However , DiracNet can only train up to 34 layer plain model which has 72.83 % accuracy on ImageNet . RepVGG ( Ding et al. , 2021b ) deploys residual neural network only at the training time . At the inference , RepVGG can transform the residual block into a plain module consisting of a stack of 3 × 3 convolution and ReLU via re-parameterization . The difference between DiracNet and RepVGG is each block in DiracNet has two branches ( identity without BN and 3 × 3 ConvBN ) while RepVGG has three branches ( identity with BN , 3 × 3 ConvBN , and 1 × 1 ConvBN ) . However , those re-parameterization methods that used commutative property can only apply to linear layers , i.e. , the non-linear layers have to be outside of the residual connection , which limits the potential of neural networks for large depths . Filter Pruning : Filter pruning is a promising solution to accelerate CNNs . Numerous inspiring works prune the filters by evaluating their importance . Heuristic metrics are proposed , such as the magnitude of convolution kernels ( Li et al. , 2017 ) , the average percentage of zero activations ( APoZ ) ( Hu et al. , 2016 ) . There also exists some work to let networks automatically select important filters . For example , ( Liu et al. , 2017a ) sparsify the weights in the BN layer to automatically find which filters contribute most to the network performance . However , For ResNet-based architecture , the existence of residual connection limits the power of pruning since the dimension of the input and output through the residual connection must keep the same . Thus pruning ratio of ResNet is not larger than the ratio of the plain model . Since RM operation can equivalently transform ResNet to a plain model , the transferred model ( RMNet ) also has a great advantage on pruning . 3 RM OPERATION AND RMNET : TOWARDS AN EFFICIENT PLAIN NETWORK . 3.1 RM OPERATION . Figure 1 shows the process of equivalently removing residual connection by RM Operation . For simplicity , we do not show BN layer in the figure , and the number of input channels , medium channels , and output channels are the same and are assigned as C. •Reserving : We first insert several Dirac initialized filters ( output channels ) in Conv 1 . The number of Dirac initialized filters is the same as the input channels ’ number in the convolution layer . The Dirac initialized filters are defined as 4-dimensional matrix : Ic , n , i , j = { 1 if c = n and i = j = 0 , 0 otherwise ( 1 ) We can view these filters in Figure 1 ( b ) . Every filter only has one element to be 1 , which can reserve the corresponding channel ’ s information of the input feature map via convolution . For the BN layer , to reserve the input feature map , weight w and bias b in BN need to be adjusted to make the BN layer behave like an identity function . Suppose the running mean and the running variance of the feature maps are µ and σ2 separately , we set w = √ σ2 + and bias b = µ . Then for any input x through the BN layer , the output is : y = w × ( x− µ ) √ σ2 + + b = x ( 2 ) where = 10−5 is added to the σ2 in case of the divisor to be zero . For the ReLU layer , There are two cases to be considered : • When the input value through residual connection is non-negative ( ie.e , in ResNet , every ResBlock has a following a ReLU layer , which keeps input values are all non-negative ) , we can directly use ReLU to reserve the information . Since ReLU does not change the non-negative values . • When the input value through residual connection can be negative ( e.g . in MobileNetV2 , ReLU is only located in the middle of the ResBlock ) , we use PReLU instead of ReLU to reserve the information . The parameters of PReLU regarding to the additional channels are set to one . In this way , PReLU behaves as Identity function . From the above analysis , the input feature map can be well reserved by Conv 1 , BN , and ReLU in the ResBlock . • Merging : We extend input channels in Conv 2 and dirac initialize these channels . The value of i-th channel of z is the sum of the original i-th filters ’ output xR2i and the i-th input feature map x + i , which is equal to yi in the original ResBlock . The whole merging process can be formulated as : z+c , h , w =Max ( 2C∑ n K∑ i K∑ j ( [ WR2 , I ] c , n , i , j × [ xR1+ , x+ ] n , h+i , w+j + [ BR2 , 0 ] c ) , 0 ) =Max ( C∑ n K∑ i K∑ j ( WR2c , n , i , j × xR1+n , h+i , w+j +B R2 c ) + 2C∑ n=C+1 K∑ i K∑ j ( Ic , n , i , j × x+n , h+i , w+j + 0 ) , 0 ) =Max ( C∑ n K∑ i K∑ j ( WR2c , n , i , j × xR1+n , h+i , w+j +B R2 c ) + x+c , h , w , 0 ) = y+c , h , w ( 3 ) where [ ∗ , ∗ ] means the concatenation of the two elements . Thus , by reserving and merging , we can remove the residual connection without changing the original output of ResBlock .

Summary: Update: I have read the rebuttal, see below. ----------- While residual connections are a key component in today's deep learning architectures, they can be problematic in some settings, e.g., in pruning. This paper presents an improved method (RMNet) for removing residual connections from a ResNet - type neural network after training. It improves over related work and in contrast to those, it works on some typical ResNet variants. The paper also discusses fine-tuning, pruning and more efficient architectures.