RMNet: Equivalently Removing Residual Connection from Networks

1 INTRODUCTION . Since AlexNet ( Krizhevsky et al. , 2012 ) came out , the state-of-the-art CNN architecture has become deeper and deeper . For example , AlexNet only has 5 convolutional layers , it is soon extended to 19 and 22 layers by VGG network ( Simonyan & Zisserman , 2015 ) and GoogLeNet ( Szegedy et al. , 2015 ; 2016 ; 2017 ) , respectively . However , deep networks that simply stack layers are hard to train because of the gradient vanishing and exploding problem — as the gradient is back-propagated to earlier layers , repeated multiplication may make the gradient infinitely small or large . This problem has been largely addressed by the normalized initialization ( LeCun et al. , 2012 ; Glorot & Bengio , 2010 ; Saxe et al. , 2013 ; He et al. , 2015 ) and intermediate normalization layers ( Ioffe & Szegedy , 2015 ) , which enable networks with tens of layers converging . Meanwhile , another degradation problem has been exposed : with the increase of the network depth , accuracy gets saturated and then degrades rapidly . ResNet ( He et al. , 2016a ; b ) addresses the degradation problem and achieves 1K+ layers models by adding a residual connection from the input of a block to the output . Instead of hoping each stacked layer directly fit a desired underlying mapping , ResNet lets these layers fit a residual mapping . When the identity mapping is optimal , it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers . As ResNet gains more and more popularity , researchers propose many new architectures , e.g . ( Hu et al. , 2018 ; Li et al. , 2019 ; Tan & Le , 2019 ; 2021 ) , based on ResNet and interpret the success from different aspects . However , ResDistill ( Li et al. , 2020 ) points that the residual connections in ResNet-50 account for about 40 percent of the entire memory usage on feature maps , which will slow down the inference process . Besides , residual connections in the network are not friendly for ‘ network pruning ’ ( Ding et al. , 2021b ) . By contrast , the VGG-like model , also called the plain model in this paper , has only one path and is fast , memory economical , and parallelization friendly . RepVGG ( Ding et al. , 2021b ) proposes a powerful approach to remove residual connections via re-parametrization at inference . Specifically , RepVGG adds 3×3 convolution , 1×1 convolution , and identity together for training . The BN layer was added at the end of each branch and the ReLU was added after the addition operation . During training , RepVGG only needs to learn the residual mapping , while in the inference stage , re-parameterization was utilized to transform the basic block of RepVGG into a stack of 3× 3 convolution layers plus ReLU operation , which has a favorable speed-accuracy trade-off compared to ResNet . However , we find that the performance of RepVGG suffers severe degradation when the network goes deeper , we provide experiments to verify this observation in Section 4.1 . In this paper , we introduce a novel approach named RM operation , which can remove the residual connection with non-linear layers inside it and keep the result of the model unchanged . RM operation reserves the input feature maps by the first convolution layer , BN Layer and ReLU Layer , then merges them with the output feature maps by the last convolution in the ResBlock . With this method , we can equivalently convert a pre-trained ResNet or MobileNetV2 to an RMNet model to increase the degree of parallelism . Besides , the architecture of RMNet makes it friendly for pruning since there are no residual connections . We summarized our main contributions as follows : • We find that removing residual connection by re-parameterization methods has its limitation , especially when the depth of the model is large . It lies in the non-linear operation that can not be put inside of residual connection for re-parameterization . • We propose a novel method named RM operation which can remove residual connection across non-linear layers without changing the output by reserving the input feature maps and merging them with the output feature maps . • With RM operation , we can convert the ResBlocks to a stack of convolutions and ReLUs , which achieves a better accuracy-speed trade-off and make it very friendly for pruning . 2 RELATED WORK . Advantage of Residual Networks over Plain Model . He et al . ( He et al. , 2016a ) introduce ResNet for addressing the degradation problem . In addition , the gradient of layers in PreActResNet ( He et al. , 2016b ) does not vanish even when the weights are arbitrarily small , which leads to nice backward propagation properties . ( Weinan , 2017 ) shows the connection between dynamical systems and deep learning . ( Lu et al. , 2018 ) regards ResNet as an Euler discretization of Ordinary Differential Equations ( ODE ) . NODEs ( Chen et al. , 2018 ) replaces residual block with neural ODEs for better training extremely deep networks . ( Balduzzi et al. , 2017 ) identifies shattered gradients problems making the training of DNNs difficult . They show that the correlation between gradients in the standard feedforward network decays exponentially with depth . In contrast , the gradients in ResNet are far more resistant to shattering , decaying sublinearly . ( Veit et al. , 2016 ) shows that residual networks can be seen as a collection of many paths of differing lengths . From this view , n-blocks ’ ResNet have O ( 2n ) implicit paths connecting input and output , and that adding a block doubles the number of paths . DNNs Without Residual Connection . A recent work ( Oyedotun et al. , 2020 ) combined several techniques including Leaky ReLU , max-norm , and careful initialization to train a 30 layer plain ConvNet , which could reach 74.6 % top-1 accuracy , 2 % lower than their baseline . A mean-field theory-based method ( Xiao et al. , 2018 ) is proposed to train extremely deep ConvNets without branches . However , 1K-layer plain networks only achieve 82 % accuracy on CIFAR-10 . ( Balduzzi et al. , 2017 ) use a ” looks linear ” initialization method and CReLUs ( Shang et al. , 2016 ) to train a 198 layer plain model to 85 % accuracy on CIFAR-10 . Though the theoretical contributions were insightful , the models were not practical . One can get DNNs without residual connection in a reparameterization manner : re-parameterization ( Ding et al. , 2019 ; 2021a ) means using the params of a structure to parameterize another set of params . Those approaches first train a model with residual connection and remove residual connection by re-parameterization when inference . DiracNet ( Zagoruyko & Komodakis , 2017 ) uses the addition of an identity matrix and convolutional matrix for forwarding propagation , the parameters of convolution need only to learn the residual function like ResNet . After training , DiracNet adds the identity matrix to the convolutional matrix and uses the reparameterized model for inference . However , DiracNet can only train up to 34 layer plain model which has 72.83 % accuracy on ImageNet . RepVGG ( Ding et al. , 2021b ) deploys residual neural network only at the training time . At the inference , RepVGG can transform the residual block into a plain module consisting of a stack of 3 × 3 convolution and ReLU via re-parameterization . The difference between DiracNet and RepVGG is each block in DiracNet has two branches ( identity without BN and 3 × 3 ConvBN ) while RepVGG has three branches ( identity with BN , 3 × 3 ConvBN , and 1 × 1 ConvBN ) . However , those re-parameterization methods that used commutative property can only apply to linear layers , i.e. , the non-linear layers have to be outside of the residual connection , which limits the potential of neural networks for large depths . Filter Pruning : Filter pruning is a promising solution to accelerate CNNs . Numerous inspiring works prune the filters by evaluating their importance . Heuristic metrics are proposed , such as the magnitude of convolution kernels ( Li et al. , 2017 ) , the average percentage of zero activations ( APoZ ) ( Hu et al. , 2016 ) . There also exists some work to let networks automatically select important filters . For example , ( Liu et al. , 2017a ) sparsify the weights in the BN layer to automatically find which filters contribute most to the network performance . However , For ResNet-based architecture , the existence of residual connection limits the power of pruning since the dimension of the input and output through the residual connection must keep the same . Thus pruning ratio of ResNet is not larger than the ratio of the plain model . Since RM operation can equivalently transform ResNet to a plain model , the transferred model ( RMNet ) also has a great advantage on pruning . 3 RM OPERATION AND RMNET : TOWARDS AN EFFICIENT PLAIN NETWORK . 3.1 RM OPERATION . Figure 1 shows the process of equivalently removing residual connection by RM Operation . For simplicity , we do not show BN layer in the figure , and the number of input channels , medium channels , and output channels are the same and are assigned as C. •Reserving : We first insert several Dirac initialized filters ( output channels ) in Conv 1 . The number of Dirac initialized filters is the same as the input channels ’ number in the convolution layer . The Dirac initialized filters are defined as 4-dimensional matrix : Ic , n , i , j = { 1 if c = n and i = j = 0 , 0 otherwise ( 1 ) We can view these filters in Figure 1 ( b ) . Every filter only has one element to be 1 , which can reserve the corresponding channel ’ s information of the input feature map via convolution . For the BN layer , to reserve the input feature map , weight w and bias b in BN need to be adjusted to make the BN layer behave like an identity function . Suppose the running mean and the running variance of the feature maps are µ and σ2 separately , we set w = √ σ2 + and bias b = µ . Then for any input x through the BN layer , the output is : y = w × ( x− µ ) √ σ2 + + b = x ( 2 ) where = 10−5 is added to the σ2 in case of the divisor to be zero . For the ReLU layer , There are two cases to be considered : • When the input value through residual connection is non-negative ( ie.e , in ResNet , every ResBlock has a following a ReLU layer , which keeps input values are all non-negative ) , we can directly use ReLU to reserve the information . Since ReLU does not change the non-negative values . • When the input value through residual connection can be negative ( e.g . in MobileNetV2 , ReLU is only located in the middle of the ResBlock ) , we use PReLU instead of ReLU to reserve the information . The parameters of PReLU regarding to the additional channels are set to one . In this way , PReLU behaves as Identity function . From the above analysis , the input feature map can be well reserved by Conv 1 , BN , and ReLU in the ResBlock . • Merging : We extend input channels in Conv 2 and dirac initialize these channels . The value of i-th channel of z is the sum of the original i-th filters ’ output xR2i and the i-th input feature map x + i , which is equal to yi in the original ResBlock . The whole merging process can be formulated as : z+c , h , w =Max ( 2C∑ n K∑ i K∑ j ( [ WR2 , I ] c , n , i , j × [ xR1+ , x+ ] n , h+i , w+j + [ BR2 , 0 ] c ) , 0 ) =Max ( C∑ n K∑ i K∑ j ( WR2c , n , i , j × xR1+n , h+i , w+j +B R2 c ) + 2C∑ n=C+1 K∑ i K∑ j ( Ic , n , i , j × x+n , h+i , w+j + 0 ) , 0 ) =Max ( C∑ n K∑ i K∑ j ( WR2c , n , i , j × xR1+n , h+i , w+j +B R2 c ) + x+c , h , w , 0 ) = y+c , h , w ( 3 ) where [ ∗ , ∗ ] means the concatenation of the two elements . Thus , by reserving and merging , we can remove the residual connection without changing the original output of ResBlock .

Summary: This paper proposes to convert residual network architectures to equivalent "plain" networks after training.  This is accomplished by augmenting convolutional layers with Dirac initialized filters (folding extraction of the residual signal into the conv layer), and modifying the subsequent Batch Normalization and ReLU layers (converting ReLU to PReLU) accordingly.  Motivations for doing this are to increase inference throughput, and allow additional pruning of the resulting plain network.