RMNET: EQUIVALENTLY REMOVING RESIDUAL CON-
Year: 2021

Abstract:
Although residual connection enables training very deep neural networks, it is not friendly for online inference due to its multi-branch topology. This encourages many researchers to work on designing DNNs without residual connections at inference. For example, RepVGG re-parameterizes multi-branch topology to a VGG-like (single-branch) model when deploying, showing great performance when the network is relatively shallow. However, RepVGG can not transform ResNet to VGG equivalently because re-parameterizing methods can only be applied to linear Blocks and the non-linear layers (ReLU) have to be put outside of the residual connection which results in limited representation ability, especially for deeper networks. In this paper, we aim to remedy this problem and propose to remove the residual connection in a vanilla ResNet equivalently by a reserving and merging (RM) operation on ResBlock. Specifically, the RM operation allows input feature maps to pass through the block while reserving their information and merges all the information at the end of each block, which can remove residual connections without changing the original output. RMNet basically has two advantages: 1) it achieves a better accuracy-speed trade-off compared with ResNet and RepVGG; 2) its implementation makes it naturally friendly for high ratio network pruning. Extensive experiments are performed to verify the effectiveness of RMNet. We believe the ideology of RMNet can inspire many insights on model design for the community in the future.

heading: 1 INTRODUCTION
text: Since AlexNet (Krizhevsky et al., 2012) came out, the state-of-the-art CNN architecture has become deeper and deeper. For example, AlexNet only has 5 convolutional layers, it is soon extended to 19 and 22 layers by VGG network (Simonyan & Zisserman, 2015) and GoogLeNet(Szegedy et al., 2015; 2016; 2017), respectively. However, deep networks that simply stack layers are hard to train because of the gradient vanishing and exploding problem — as the gradient is back-propagated to earlier layers, repeated multiplication may make the gradient infinitely small or large. This problem has been largely addressed by the normalized initialization (LeCun et al., 2012; Glorot & Bengio, 2010; Saxe et al., 2013; He et al., 2015) and intermediate normalization layers (Ioffe & Szegedy, 2015), which enable networks with tens of layers converging. Meanwhile, another degradation problem has been exposed: with the increase of the network depth, accuracy gets saturated and then degrades rapidly. ResNet (He et al., 2016a;b) addresses the degradation problem and achieves 1K+ layers models by adding a residual connection from the input of a block to the output. Instead of hoping each stacked layer directly fit a desired underlying mapping, ResNet lets these layers fit a residual mapping. When the identity mapping is optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers. As ResNet gains more and more popularity, researchers propose many new architectures, e.g. (Hu et al., 2018; Li et al., 2019; Tan & Le, 2019; 2021), based on ResNet and interpret the success from different aspects.
However, ResDistill (Li et al., 2020) points that the residual connections in ResNet-50 account for about 40 percent of the entire memory usage on feature maps, which will slow down the inference process. Besides, residual connections in the network are not friendly for ‘network pruning’ (Ding et al., 2021b). By contrast, the VGG-like model, also called the plain model in this paper, has only one path and is fast, memory economical, and parallelization friendly. RepVGG (Ding et al., 2021b) proposes a powerful approach to remove residual connections via re-parametrization at inference.
Specifically, RepVGG adds 3×3 convolution, 1×1 convolution, and identity together for training. The BN layer was added at the end of each branch and the ReLU was added after the addition operation. During training, RepVGG only needs to learn the residual mapping, while in the inference stage, re-parameterization was utilized to transform the basic block of RepVGG into a stack of 3× 3 convolution layers plus ReLU operation, which has a favorable speed-accuracy trade-off compared to ResNet. However, we find that the performance of RepVGG suffers severe degradation when the network goes deeper, we provide experiments to verify this observation in Section 4.1.
In this paper, we introduce a novel approach named RM operation, which can remove the residual connection with non-linear layers inside it and keep the result of the model unchanged. RM operation reserves the input feature maps by the first convolution layer, BN Layer and ReLU Layer, then merges them with the output feature maps by the last convolution in the ResBlock. With this method, we can equivalently convert a pre-trained ResNet or MobileNetV2 to an RMNet model to increase the degree of parallelism. Besides, the architecture of RMNet makes it friendly for pruning since there are no residual connections. We summarized our main contributions as follows:
• We find that removing residual connection by re-parameterization methods has its limitation, especially when the depth of the model is large. It lies in the non-linear operation that can not be put inside of residual connection for re-parameterization.
• We propose a novel method named RM operation which can remove residual connection across non-linear layers without changing the output by reserving the input feature maps and merging them with the output feature maps.
• With RM operation, we can convert the ResBlocks to a stack of convolutions and ReLUs, which achieves a better accuracy-speed trade-off and make it very friendly for pruning.

heading: 2 RELATED WORK
text: Advantage of Residual Networks over Plain Model. He et al. (He et al., 2016a) introduce ResNet for addressing the degradation problem. In addition, the gradient of layers in PreActResNet (He et al., 2016b) does not vanish even when the weights are arbitrarily small, which leads to nice backward propagation properties. (Weinan, 2017) shows the connection between dynamical systems and deep learning. (Lu et al., 2018) regards ResNet as an Euler discretization of Ordinary Differential Equations (ODE). NODEs (Chen et al., 2018) replaces residual block with neural ODEs for better training extremely deep networks. (Balduzzi et al., 2017) identifies shattered gradients problems making the training of DNNs difficult. They show that the correlation between gradients in the standard feedforward network decays exponentially with depth. In contrast, the gradients in ResNet are far more resistant to shattering, decaying sublinearly. (Veit et al., 2016) shows that residual networks can be seen as a collection of many paths of differing lengths. From this view, n-blocks’ ResNet have O(2n) implicit paths connecting input and output, and that adding a block doubles the number of paths.
DNNs Without Residual Connection. A recent work(Oyedotun et al., 2020) combined several techniques including Leaky ReLU, max-norm, and careful initialization to train a 30 layer plain ConvNet, which could reach 74.6% top-1 accuracy, 2% lower than their baseline. A mean-field theory-based method (Xiao et al., 2018) is proposed to train extremely deep ConvNets without branches. However, 1K-layer plain networks only achieve 82% accuracy on CIFAR-10. (Balduzzi et al., 2017) use a ”looks linear” initialization method and CReLUs (Shang et al., 2016) to train a 198 layer plain model to 85% accuracy on CIFAR-10. Though the theoretical contributions were insightful, the models were not practical. One can get DNNs without residual connection in a reparameterization manner: re-parameterization (Ding et al., 2019; 2021a) means using the params of a structure to parameterize another set of params. Those approaches first train a model with residual connection and remove residual connection by re-parameterization when inference. DiracNet (Zagoruyko & Komodakis, 2017) uses the addition of an identity matrix and convolutional matrix for forwarding propagation, the parameters of convolution need only to learn the residual function like ResNet. After training, DiracNet adds the identity matrix to the convolutional matrix and uses the reparameterized model for inference. However, DiracNet can only train up to 34 layer plain model which has 72.83% accuracy on ImageNet. RepVGG (Ding et al., 2021b) deploys residual neural network only at the training time. At the inference, RepVGG can transform the residual block into a plain module consisting of a stack of 3 × 3 convolution and ReLU via re-parameterization.
The difference between DiracNet and RepVGG is each block in DiracNet has two branches (identity without BN and 3 × 3 ConvBN) while RepVGG has three branches (identity with BN, 3 × 3 ConvBN, and 1 × 1 ConvBN). However, those re-parameterization methods that used commutative property can only apply to linear layers, i.e., the non-linear layers have to be outside of the residual connection, which limits the potential of neural networks for large depths.
Filter Pruning: Filter pruning is a promising solution to accelerate CNNs. Numerous inspiring works prune the filters by evaluating their importance. Heuristic metrics are proposed, such as the magnitude of convolution kernels (Li et al., 2017), the average percentage of zero activations (APoZ) (Hu et al., 2016). There also exists some work to let networks automatically select important filters. For example, (Liu et al., 2017a) sparsify the weights in the BN layer to automatically find which filters contribute most to the network performance. However, For ResNet-based architecture, the existence of residual connection limits the power of pruning since the dimension of the input and output through the residual connection must keep the same. Thus pruning ratio of ResNet is not larger than the ratio of the plain model. Since RM operation can equivalently transform ResNet to a plain model, the transferred model (RMNet) also has a great advantage on pruning.

heading: 3 RM OPERATION AND RMNET: TOWARDS AN EFFICIENT PLAIN NETWORK

heading: 3.1 RM OPERATION
text: Figure 1 shows the process of equivalently removing residual connection by RM Operation. For simplicity, we do not show BN layer in the figure, and the number of input channels, medium channels, and output channels are the same and are assigned as C.
•Reserving: We first insert several Dirac initialized filters (output channels) in Conv 1. The number of Dirac initialized filters is the same as the input channels’ number in the convolution layer. The Dirac initialized filters are defined as 4-dimensional matrix:
Ic,n,i,j = { 1 if c = n and i = j = 0, 0 otherwise
(1)
We can view these filters in Figure 1(b). Every filter only has one element to be 1, which can reserve the corresponding channel’s information of the input feature map via convolution.
For the BN layer, to reserve the input feature map, weight w and bias b in BN need to be adjusted to make the BN layer behave like an identity function. Suppose the running mean and the running variance of the feature maps are µ and σ2 separately, we set w = √ σ2 + and bias b = µ. Then for any input x through the BN layer, the output is:
y = w × (x− µ)√ σ2 + + b
= x
(2)
where = 10−5 is added to the σ2 in case of the divisor to be zero.
For the ReLU layer, There are two cases to be considered:
• When the input value through residual connection is non-negative (ie.e, in ResNet, every ResBlock has a following a ReLU layer, which keeps input values are all non-negative), we can directly use ReLU to reserve the information. Since ReLU does not change the non-negative values.
• When the input value through residual connection can be negative (e.g. in MobileNetV2, ReLU is only located in the middle of the ResBlock), we use PReLU instead of ReLU to reserve the information. The parameters of PReLU regarding to the additional channels are set to one. In this way, PReLU behaves as Identity function.
From the above analysis, the input feature map can be well reserved by Conv 1, BN, and ReLU in the ResBlock.
• Merging: We extend input channels in Conv 2 and dirac initialize these channels. The value of i-th channel of z is the sum of the original i-th filters’ output xR2i and the i-th input feature map x + i , which is equal to yi in the original ResBlock. The whole merging process can be formulated as:
z+c,h,w =Max( 2C∑ n K∑ i K∑ j ( [WR2, I]c,n,i,j × [xR1+, x+]n,h+i,w+j + [BR2, 0]c ) , 0)
=Max( C∑ n K∑ i K∑ j ( WR2c,n,i,j × xR1+n,h+i,w+j +B R2 c )
+ 2C∑ n=C+1 K∑ i K∑ j ( Ic,n,i,j × x+n,h+i,w+j + 0 ) , 0)
=Max( C∑ n K∑ i K∑ j ( WR2c,n,i,j × xR1+n,h+i,w+j +B R2 c ) + x+c,h,w, 0)
= y+c,h,w
(3)
where [∗, ∗] means the concatenation of the two elements. Thus, by reserving and merging, we can remove the residual connection without changing the original output of ResBlock.

heading: 3.2 CONVERT RESNET TO VGG
text: In this section, we show how to apply RM operation to convert ResNet into VGG-like RMNet. Notice that there exist some DownSample ResBlocks at the first block of layer2, layer3, and layer4 in ResNet. The function of ‘downsample’ is for:
• doubling the number of channels;
• reducing the depth and width of the feature map by half.
Even though there are only a few DownSample ResBlocks in ResNet, removing residual connection in such blocks is not as direct as in basic ResBlocks because there exist 1 × 1 convolution in the
downsample branch and it will change the input value in the path of residual connection. We propose two solutions in Figure 2 to convert DownSample ResBlock to RMBlock.
As shown in Figure 2(a), the original 1×1 downsample kernel is padding with zeros to 3×3 kernel. we replace ReLU in the block with PReLU, whose parameters for each channel can control the scale of negative activation, as analyzed in Section 3.1. For the left (residual) branch, we set the PReLU parameters as zeros, in this way, the function of the PReLU is equal to ReLU. PReLU is added after the downsample convolution, the parameters of which are initialized by one. In this way, PReLU is equal to the Identity Mapping function. New additional channels of the second convolutional layer are Dirac initialized (similar to Figure 1(b)), which keep the feature map after downsampling convolution unchanged. Then, all the layers in the left and right branches can be merged together into one sequence shown in Figure 2(a).
In Figure 2(b), we separate the function of downsampling convolution into two convolutions in downsample branch. The first module is 3×3 convolution with Dirac initialized filters and the stride is 2 for reducing the depth and width of the input feature map by half. The feature map through this Conv layer is still non-negative and ReLU can reserve the value unchanged. The second module is 3×3 convolution, whose weights are transferred from original 1×1 filters in downsample branch with 0 paddings. Since the original filters’ kernel size is 1×1, reducing the size of feature maps before doubling the number of channels will not change the receptive field in the original downsample branch.
Both methods can equally convert the DownSample Blocks to RMBlocks. The number of parameters of the first method is 108C2 + 4C, including Conv 1 (C × 4C × 3× 3), PReLU (4C), Conv 2 (4C × 2C × 3× 3). The number of parameters of the second method is 81C2, consisting of Conv 1 (C × 3C × 3× 3), Conv 2 (3C × 2C × 3× 3). We can see the number of parameters of the second method is only 0.75 times of the first method. Thus we use the second method to convert ResNet to RMNet in our experiment.

heading: 3.3 CONVERT MOBILENETV2 TO MOBILENETV1
text: Technically, there is no difference between converting ResNet and converting MobileNetV2 by using RM operation. However, the specialty of the structure of MobileNetV2 allows to further decrease the inference time by using parameter fusion after RM operation.
From Figure 3, after applying RM operation on MobileNetV2, the residual connections are removed, which leaves two Pointwise-ConvBN layers shown in the dashed box. Since convolution and batchnormalization can be both presented by matrix multiplication and there exists no non-linear layer between them, these two Pointwise-ConvBN layers can be fused. Mathematically, let X , Y , W , B be the input feature map, output feature map, weight, and bias of the first Pointwise-ConvBN layer in the dashed box and Z, W ′ , B ′ be the output feature map, weight and bias of the second Pointwise-ConvBN layer in the dashed box. The process of fusing the parameters can be represented as:
MobileNetV2 RMNetV1 MobileNetV1
Z =W ′ Y +B ′
=W ′ (WX +B) +B ′
= (W ′ W )X +W ′ B +B ′
(4)
The fused weight is W ′ W ∈ Rn,c,1,1 and the fused bias is W ′B + B′ ∈ Rn. Thus, RM operation provides another opportunity to further decrease the inference time by fusing the parameters. After parameter fusion, the architecture of RMNet is identical to MobileNetV1 which is very interesting since the presence of MobileNetV2 is to increases the generalization ability of MobileNetV1 by utilizing residual connections. However, we show RM operation can inverse this process, i.e., converting MobileNetV2 into MobileNetV1 to make MobileNetV1 great again. We provide experiments in Section 4.3 to verify the effectiveness of this transformation.

heading: 3.4 PRUNING RMNET
text: Since RMNet does not have any residual connections, it is more friendly for filter pruning. In this paper, we adopt Network slimming (Liu et al., 2017b) to prune RMNet because of its simpleness and effectiveness. Specifically, we first train ResNet and sparsity the weight in the BN layer. Note an additional BN layer should be added in the residual connection during training since we also need to determine which additional filters are important after RM operation. After training, we convert ResNet into RMNet and prune the filters according to the weights in BN layers, which is identical to vanilla Network slimming. Figure 4 shows the overall process of pruning. Different from the traditional approach, RMNet can serve as a transition to gain a larger pruning ratio.

heading: 4 EXPERIMENTS
text: This section aranges as follows: in Section 4.1, we show RMNet has great advantage over RepVGG for deeper networks; in Section 4.2. we show RMNet can achieve better speed-accuracy tradeoff on CIFAR10, CIFAR100 and ImageNet datasets; in Section 4.3, we show RMNet is applicable for light-weight models; in Section 4.4, we verify the effectivenss of RMNet on network pruning task.
The speed of each model is tested on a Tesla V100 GPU with a batch size of 128 by first feeding 50 batches to warm the hardware up, then 50 batches with time usage recorded.

heading: 4.1 REMAINING HIGH ACCURACY FOR DEEPER NETWORK
text: One power of DNN is that the representation ability increases as we raise the network depth. Thus we examine how network depth influences network performance on RMNet and RepVGG. Figure 5 shows the network performance on CIFAR-10 and CIFAR-100. For a fair comparison, the same hyper-parameters are deployed for each method: mini-batch size (256), optimizer (SGD), initial learning rate (0.1), momentum (0.9), weight decay (0.0005), number of epochs (200), learning rate decay 0.1 at every 60 epochs.
From Figure 5, as the depth increases, RMNet can get better accuracy. This is because, unlike RepVGG, RMNet can equivalently convert ResNet into a plain model. Thus the performance will not decrease as the network goes deeper. In contrast, the accuracy of RepVGG will decrease (the reasons are analyzed at Appendix A.2 ), e.g. compared to RMNet 133 that achieves 79.57% accuracy on CIFAR-100, RepVGG 133 only has 41.38% accuracy.

heading: 4.2 BETTER ACCURACY-SPEED TRADEOFF
text: From Section 3, RMNet removes residual connections in the cost of bringing additional parameters. For example, in Figure 1, RM operation doubles the number of parameters of the original ResBlock. To alleviate this issue, we use RM operation on the ResNet-based architectures with Inverted Residual Block for designing RMNet. Inverted Residual Block (IRB) mainly contains three consecutive steps: point-wise convolution, group-wise convolution, and point-wise convolution. The first ‘pointwise convolution‘ will increase the number of channels of the input feature map by T times and the second ‘point-wise convolution‘ will reduce the number of channels to the original number. For point-wise convolution, RM operation only increases the number of parameters and FLOPs by 1T times. Suppose the size of the input feature map is H×W ×C and the parameters of the first pointwise convolutional layer are expressed as T ×C×C×1×1. We only need additional C×C×1×1 parameters to reserve the input feature map, since RM operation is only to reserve the input feature map. ‘Group-wise convolution’ is also parameter-friendly. For example, for a normal convolution, we needC×K×K parameters to reserve the information of an input channel. While for group-wise convolution, we only need 1G times of parameters to reserve the information where G is the group number in group-wise convolution. Besides, the cost of storage and calculation does not change whether or not to use group-wise convolution. Based on the advantage of IRB, we design a series of RMNet models. We first determine the input and output width by the classic width setting of [64, 128, 256, 512]. We replace 7 × 7 Conv and MaxPooling on the head with two sequences of 3 × 3 Conv, BN, and ReLU (the same approach used in RepVGG). The tail of RMNet is the same as
ResNet. For RMNet 50x6 32, ‘50’ indicates the number of all the Conv layers; ‘6’ indicates the multiple of classic width setting, for example, the width of the grouped convolution layer in RMNet 50x6 32 is [6×64, 6×128, 6×256, 6×512]; ‘32’ indicates the number of channels of each group in RMBlock. We print out the detailed architecture of RMNet in the Appendix A.4. Next, we compare RMNet with SOTA models on CIFAR10, CIFAR100 and ImageNet to test the speed and accuracy. For a fair comparison, we train RMNet following the official implementations of RepVGG 1.
From Table 1, the accuracy of RMNet 50×6 32 is 2.3% higher than ResNet 101 and 0.59% higher than RepVGG B2 with the nearly same speed. It is worth noting that RMNet 101×6 16 reaches over 80% top-1 accuracy without using any tricks, which is the first time for a plain model, to the best of our knowledge. Also, increasing the depth to 152 still has a benefit to RMNet, which shows the great potential of our method.

heading: 4.3 FRIENDLY FOR LIGHT-WEIGHT MODELS
text: We conduct an experiment to verify our analysis in Section 3.3. We first train a MobileNetV2 from scratch and convert it to RMNetV1 following the process introduced in Section 3.3. Note that the initial architecture of MobileNetV2 has to be designed to make sure the transformed RMNetV1 has the same architecture (depth and width) of MobileNetV1 from the original paper (Howard et al., 2017). Thus the MobileNetV2 in Figure 3 is different from vanilla MobileNetV2(Sandler et al.,
1https://github.com/DingXiaoH/RepVGG/blob/main/train.py
2018). We also train a MobileNetV1 trained from scratch for comparison. The result are show in Table 2.
From the experiment in Table 2, we can see the speed of RMNetV1 (equivalently converted from MobileNetV2) is faster than vanilla MobileNetV2 and the performance of RMNetV1 is higher than MobileNetV1, which shows RM operation can be very friendly for light-weight models.

heading: 4.4 FRIENDLY FOR PRUNING
text: We conduct an experiment in Figure 6 to verify the effectiveness of pruning RMNet. We use L1 norm multiply a certain sparsity factor to punish the weight of BN layers, and regard the channels whose weight is smaller than a certain threshold as invalid. The sparsity factor is selected from 1e-4 to 1e-3, and the threshold is selected from 5e-4 to 5e-3. A larger degree of sparsity will lead to a larger pruning ratio.
From the experiments, RMNet retains higher accuracy than ResNet under a larger pruning ratio and when the speed of pruned RMNet is nearly the same with pruned ResNet, because of more reasonable structure. Thus RMNet has a better accuracy-speed tradeoff over ResNet architecture on pruning tasks.

heading: 5 CONCLUSION AND DISCUSSION
text: In this paper, We propose RM operation to remove residual connections and perform RM operation to convert ResNet and MobileNetV2 into plain models (RMNet). RM operation allows input feature maps to pass through the block while reserving their information and merges all the information at the end of each block, which can remove residual connection without changing original output. Experiments have shown that RMNet can achieve a better speed-accuracy trade-off and is very friendly for network pruning.
There are some future directions to be considered: 1) Analyzing the residual connection on the Transformer based architectures and removing such connections via RM operation. 2) Searching residual networks with less residual connection with NAS and converting them to efficient RMNets.

heading: A APPENDIX
text: The appendix is arranged as follows: in Section A.1, we perform ablation studies; in Section A.2, we further analyze why RepVGG does not have high performace for deeper network; in Section A.3, we show fine-tuning RMNet can achieve higher performance; in Section A.4, we show the detailed architecture of RMNet.
A.1 ABLATION STUDY
In this section, we perform an ablation study to show how batch-size and hyper-parameters of network structure affect inference time for RM operation. The results are shown in Figure 7. As shown in the left bottom sub-figure, the exact inference speed grows linearly with batch size when the batch size is relatively small, indicating the resource are sufficient. From the experiments, we can find that under the condition of sufficient resources, i.e., lower batch-size and lower complexity of network structure, RMNet has a great advantage over ResNet. This is because RMNet is a plain model without residual connections. The degree of parallelism can be greatly increased when the resources are sufficient. Thus we believe RMNet has great potential in the future.
A.2 COMPARISON OF REPVGG AND RESNET
We depict the basic blocks used in ResNet and RepVGG in Figure 8. In ResBlock (Figure 8(a)), two ReLUs are inside and outside of the residual connection, respectively. While in RepBlock (Figure 8(b)), since re-parameterization is based on the commutative law of multiplication, two ReLUs have to be both outside of the residual connection. One can build RepVGG from a basic ResNet architecture by replacing each ResBlock with two RepBlocks.
Next, We analyze why RepVGG can not be trained very deep like ResNet from both forward and backward passes:
• forward path: (Veit et al., 2016) assumes that the success of ResNet can be contributed to the ”model ensemble”. we can regard ResNet as an ensemble of many paths of different lengths. Thus, n-blocks’ ResNet have O(2n) implicit paths connecting input and output. However, unlike ResNet that two branches in the block are separable and can not be merged, multi-branches in RepVGG can be represented by one branch, which can be shown as:
xi+1 = ReLU(BN(xi) +BN(Conv1×1(xi)) +BN(Conv3×3(xi)))
= ReLU(BN(Conv ′ 3×3(xi))) (5)
where Conv ′
is the merged convolution of each Conv in the branches. Thus RepVGG does not have the implicit ”ensemble assumption” of ResNet, and the representation gap between RepVGG and ResNet increases as the number of blocks raises.
• backward path: (Balduzzi et al., 2017) analyzes ”shattered gradients problem” in deep neural networks. The correlation of gradients behaves like ‘White Gaussian Noise’ when there are more ReLUs in the backward path. Suppose ResNet and RepVGG both have n layers. From Figure 8(a), Information in ResNet can pass through the residual without going through inside ReLU in each block. However, each ReLU in RepVGG is located in the main path. Thus the number of ReLUs for ResNet in the backward path is n2 , while the number of ReLUs for RepVGG in the path is n which suggests that the gradients in ResNet are far more resistant to shattering when the depth is large, leading to better performance than RepVGG.
Table 3 shows the performance of RepVGG and ResNet on ImageNet dataset. The setting of this experiment is followed by the official implementation of RepVGG. We can see under the same network structure, ResNet-18 is 0.5% higher than RepVGG-18, and ResNet-34 is 0.8% higher than RepVGG in terms of top-1 accuracy. Thus, RepVGG increases the speed at the cost of losing representation ability.
Compared to other approaches (Li et al., 2020; Ding et al., 2021b; Zagoruyko & Komodakis, 2017) that remove residual connections, RM operation can equivalently convert ResBlock to a plain module without re-training. However, if we fine-tune the pre-trained network, the performance may slightly improve since RM operation brings additional parameters into the network. It is worth noting that we need to carefully design the BN layer of additional channels (brought by RM operation) during fine-tuning since the running mean and running var in BN may lead to an unstable state of DNN. Thus We statistics the mean and variance by inputting the training dataset through the network, which will serve as the initial values of running mean and running var of BN layers for better stableness. In addition, the weight and bias in BN are set by Equation 2 to guarantee the equality of the model. Of course, there exists another solution that we do not add BN layers when fine-tuning, i.e., fusing the BN with convolution before fine-tuning. However, it will lead to worse performance.
Figure 9 shows the accuracy of ResNet 18/34 on CIFAR-10/100 with/without fusing the BN layer when training. We can see the network without fusing BN has better performance and the accuracy is better than the pre-trained model.
A.4 DETAILED STRUCTURE OF RMNET
We print out the detailed network structure of RMNet 26 × 3 32 in Table 4. From Table 4, the most components in RMNet are ‘Conv’ and ‘ReLU’, making it very efficient on inference.

Review summaries:
Update: I have read the rebuttal, see below. ----------- While residual connections are a key component in today's deep learning architectures, they can be problematic in some settings, e.g., in pruning. This paper presents an improved method (RMNet) for removing residual connections from a ResNet - type neural network after training. It improves over related work and in contrast to those, it works on some typical ResNet variants. The paper also discusses fine-tuning, pruning and more efficient architectures.

This work focuses on removing residual connection in network via  reserving and merging (RM) operations on the ResBlock. The author has tested the approach on classification tasks on several networks with skip connection, e.g., resnet, mobilenet v2, etc. The experiments demonstrated the good performance on the listed benchmarks. 

This paper proposes to convert residual network architectures to equivalent "plain" networks after training.  This is accomplished by augmenting convolutional layers with Dirac initialized filters (folding extraction of the residual signal into the conv layer), and modifying the subsequent Batch Normalization and ReLU layers (converting ReLU to PReLU) accordingly.  Motivations for doing this are to increase inference throughput, and allow additional pruning of the resulting plain network. 

This paper focuses on an important direction of removing residual connections from ResNet-based architectures and proposes RM operation to achieve this goal. RM operation consists of two steps: reserving and merging. The reserving operation allows input feature map passing through the Conv, BN,ReLU layers without changing their values and the merging operation adds the output feature map and the reserved input feature map together with the last Conv layer. Unlike re-parameterization method in RepVGG, RM Operation can remove residual connection across non-linear layers, resulting in equivalent transformation from ResNet to plain models. Authors also design a series of plain models with RM Operation named RMNet, which outperforms previous plain models such as DiracNet, ResDistill and RepVGG.