RMNet: Equivalently Removing Residual Connection from Networks
Authors: Fanxu Meng
Hao Cheng
Jia-Xin Zhuang
Ke Li
Xing Sun
Venue: ICLR 2022 Submitted (2022)
Keywords: Efficient Network
Residual Connection

Abstract: Although residual connection enables training very deep neural networks, it is not friendly for online inference due to its multi-branch topology. This encourages many researchers to work on designing DNNs without residual connections at inference. For example, RepVGG re-parameterizes multi-branch topology to a VGG-like (single-branch) model when deploying, showing great performance when the network is relatively shallow. However, RepVGG can not transform ResNet to VGG equivalently because re-parameterizing methods can only be applied to linear Blocks and the non-linear layers (ReLU) have to be put outside of the residual connection which results in limited representation ability, especially for deeper networks. In this paper, we aim to remedy this problem and propose to remove the residual connection in a vanilla ResNet equivalently by a reserving and merging (RM) operation on ResBlock. Specifically, RM operation allows input feature maps to pass through the block while reserving their information and merges all the information at the end of each block, which can remove residual connection without changing original output. RMNet basically has two advantages: 1) it achieves a better accuracy-speed trade-off compared with ResNet and RepVGG; 2) its implementation makes it naturally friendly for high ratio network pruning. Extensive experiments are performed to verify the effectiveness of RMNet. We believe the ideology of RMNet can inspire many insights on model design for the community in the future.

Under review as a conference paper at ICLR 2022
RMNET: EQUIVALENTLY REMOVING RESIDUAL CON- NECTION FROM NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Although residual connection enables training very deep neural networks, is not friendly for online inference due to its multi-branch topology. This encour- ages many researchers work on designing DNNs without residual connections at inference. For example, RepVGG re- re-parameterizes multi-branch topology to a VGG-like (single-branch) model when deploying, showing great performance when the network is relatively shallow. However, RepVGG can not transform ResNet to VGG equivalently because re-paramcterizing methods can only be ap- plied to linear Blocks and the non non-linear layers (ReLU) have to be put outside of the residual connection which results in limited representation ability, especially for deeper networks. In this paper, aim to remedy this problem and propose to remove the residual connection in a vanilla ResNet equivalently by a reserving and merging (RM) operation ResBlock. Specifically, the RM operation allows input feature maps to pass through the block while reserving their information and merges all the information at the end of each block, which can remove residual connections without changing the original output. RMNet basically has two ad- vantages: 1) it achieves better accuracy-speet trade-off compared with ResNet and Rep VGG; 2) its implementation makes it naturally friendly for high ratio net- work pruning. Extensive experiments are performed to verify the effectiveness RMNet. We believe the ideology of RMNet can inspire many insights on model design for the community in the future.
INTRODUCTION
Since AlexNet (Krizhevsky et al. 2012) came out, the state-of- trrte- CNN architecture has become deeper and deeper. For example, AlexNet only has 5 convolutional layers, it is soon extended to 19 and 22 layers by VGG network (Simonyan & Zisserman (2055) and GoogLeNet(Szegedy 2015 2016; 2017), respectively. However, deep networks that simply stack layers are hard to train because of the gradient vanishing and exploding problem - as the gradient
al.]
back-propagated to earlier layers, repeated multiplication may make the gradient infinitely small or large. This problem has been largely addressed by the normalized initialization (LeCun et al.) 2012] Glorot & Bengio,
2010 Saxe al., 2013; He et al., 2015) and intermediate normalization layers (loffe & Szegedy, 2015). which enable networks with tens of layers converging. Meanwhile, another degradation problem has been exposed: with the increase of the network depth, accuracy gets saturated and then degrades rapidly. ResNet (He et al. (2166a addresses the degradation problem and achieves IK+ layers models by adding a residual connection from the input of block the output. Instead of hoping each stacked layer directly fit desired underlying mapping, ResNet lets these layers fit residual mapping. When the identity mapping is optimal. would be easier to push the residual to zero than to fit an identity mapping by stack of nonlinear layers. As ResNet gains more and more popularity, researchers propose many new architectures, e.g. (Hu et al. 2018 Li et al. 2019 Tan &
Le 2019 2) based ResNet and interpret the success from different aspects.
However, ResDistill (Li et al. 2020) points that the residual connections in ResNet-50 account for about 40 percent of the entire memory usage on feature maps, which will slow down the inference process. Besides, residual connections the network are not friendly for 'network pruming' (Ding et al. 20211b. By contrast, the VGG-like model, also called the plain model in this paper, has only one path and is fast, memory economical, and parallelization friendly. VGG (Ding et al. 2021b) proposes powerful approach to remove residual connections via re-parametrization inference.

---

Under review as a conference paper at ICLR 2022
Specifically, RepVGG adds 3x3 convolution, 1x1 convolution, and identity together for training. The BN layer was added at the end of each branch and the ReLU was added after the addition operation. During training, RepVGG only needs to learn the residual mapping. while in the inference stage, re-parameterization was utilized to transform the basic block of RepVGG into a stack of 3 x convolution layers plus ReI.U operation, which has favorable speed-accuracy trade-off compared to ResNet. However, we find that the performance of RepVGG suffers severe degradation when the network goes deeper, we provide experiments verify this observation in Section 4.1
In this paper, we introduce a novel approach named RM operation, which can remove the residual connection with non-linear layers inside it and keep the result of the model unchanged. RM op- eration reserves the imput feature maps by the first convolution layer, BN Layer and ReLU Layer, then merges them with the output feature maps by the last convolution in the ResBlock. With this method, we can equivalently convert pre-trained ResNet or MobileNetV2 to an RMNet model to increase the degree of parallelism. Besides, the architecture of RMNet makes friendly for pruning since there are residual connections. We summarized our main contributions as follows:
We find that removing residual connection by re-parameterization methods has its limita- tion, especially when the depth of the model is large. It lies in the non-linear operation that can not be put inside of residual connection for re-parameterization.
We propose novel method named RM operation which can remove residual connection across non-linear layers without changing the output by reserving the input feature maps and merging them with the output feature maps.
With RM operation, we can convert the ResBlocks to a stack of convolutions and ReLUs, which achieves a better accuracy- -speed trade-off and make very friendly for pruning.
RELATED WORK
Ad Advantage of Residual Networks over Plain Model. He et al. (He et al. 20166) introduce ResNet for addressing the degradation problem. In addition, the gradient of layers in PreActResNet (He et al. 2016b) does not vanish even when the weights are arbitrarily small, which leads to nice back- ward propagation properties. (Weinan 2017) shows the connection between dynamical systems and deep learning. (Lu et al., 2018) regards ResNet as Euler discretization of Ordinary Differential Equations (ODE). NODES (Chen et al., 2018) replaces residual block with neural ODEs for better training extremely deep networks. (Balduzzi et al., (017) identifies shattered gradients problems making the training of DNNs difficult. They show that the correlation between gradients in the stan- dard feedforward network decays exponentially with depth. contrast, the gradients in ResNet are far more resistant to shattering, decaying sublinearly. (Veit et (0166) shows that residual networks can be seen as a collection of many paths of differing lengths. From this view, n-blocks' ResNet have O(2") implicit paths connecting input and output, and that adding block doubles the number of paths.
DNNs Without Residual Connection. A recent work(Oyedotun et al. 2020) combined several techniques including Leaky ReLU, max-norm, and careful initialization to train a 30 layer plain ConvNet, which could reach 74.6% top-1 accuracy, 2% lower than their baseline. A mcan-field theory-based method (Xiao et al. 2018) is proposed to train extremely deep ConvNets without branches. However, laylr plain networks only achieve 82% accuracy on CIFAR-10. (Balduzzi et al. 2017) use a "looks linear" initialization method and CReLUs (Shang et al., 2016) to train a 198 layer plain model 85% accuracy on CIFAR-10. Though the theoretical contributions were insightful, the models were not practical. One can get DNNs without residual connection in a re- parameterization manner: re-paaamttrrization (Ding et al. 2019 (021a) means using the params of a structure to parameterize another set of params. Those approaches first train a model with resid- ual connection and remove residual connection by re-parameterization when inference. DiracNet Zagoruyko & Komodakis 2017) uses the addition of an identity matrix and convolutional matrix for forwarding propagation, the parameters of convolution need only to learn the residual function like ResNet. After training, DiracNet adds the identity matrix to the convolutional matrix and uses the reparameterized model for inference. However, DiracNet can only train up to 34 layer plain model which has 72.83% accuracy on ImageNet. RepVGG (Ding et al., 2021b) deploys residual neural network only at the training time. At the inference, RepVGG can transform the residual block into plain module consisting of stack of 3 x 3 convolution and ReLU via re-parameterization.
2

---

Under review as a conference paper at ICLR 2022
The difference between DiracNet and RepVGG is each block in DiracNet has two branches (iden- tity without BN and ConvBN) while RepVGG has three branches (identity with BN, ConvBN, and ConvBN). However, those re parameterization methods that used commutative property can only apply to linear layers, i.e., the non-linear layers have to be outside of the residual connection, which limits the potential of neural networks for large depths.
Filter pruning promising solution to accelerate CNNs. Numerous inspiring works prune the filters by evaluating their importance. Heuristic metrics proposed, such as the magnitude of convolution kernels (Li et al. 2017), the average percentage of zero activations (APoZ) (Hu et al. 2016). There also exists some work let networks automatically select important filters. For example, (Liu et al.) 2017a) sparsify the weights in the BN layer to automatically find which filters contribute most to network performance. However, For ResNet-based architecture, the existence of residual connection limits the power of pruning since the dimension of the input and output through the residual connection must keep the same. Thus pruning ratio of ResNet not larger than the ratio of the plain model. Since RM operation equivalently transform ResNet to plain model, the transferred model (RMNet) also has a great advantage on pruning.
Filter Pruning:
3
RM OPERATION AND RMNET: TOWARDS AN EFFICIENT PLAIN NETWORK
3.1 RM OPERATION
Figure T shows the process of equivalently removing residual connection by RM Operation. For simplicity, we do not show BN layer in the figure, and the number of input channels, medium channels, and output channels are the same and are assigned as C.
X-R
R2
y
y"
ReLU
Rell 2
Conv
Conv 2
(a) Training Phase
RI+
X
θ
z'
(b) Inferencing Phase
Figure 1: The upper figure shows a ResBlock during the training phase. The lower figure is the converted RMBlock for inference, which has no residual connections. Both blocks have equal output given the same input.
Reserving: We first insert several Dirac initialized filters (output channels) in Conv 1. The number of Dirae initialized filters the same as the input channels' number the convolution layer. The Dirac initialized filters are defined as 4-dimensional matrix:
{
if and i= O, otherwise
Lem,
(I)
We can view these filters in Figure l(b) Every filter only has one element to be 1, which can reserve the corresponding channel's information of the input feature map via convolution.
3

---

Under review as a conference paper at ICLR 2022
For the BN layer, reserve the imput feature map, weight and bias b in BN need be adjusted to make the BN layer behave like an identity function. Suppose the running mean and the running variance the feature maps are μ and o² separately, we set w = vo² e and bias = μ.. Then for any input through the BN layer, the output is:
(x
v +
(2)
where 10-5 is added to the o² in case of the divisor be zero.
For the ReLUU layer, There are two cases to be considered:
When the input value through residual connection non-negative (ie.e, in ResNet, every ResBlock has following ReLU layer, which keeps input values are all non-negative), we can directly use ReLU to reserve the information. Since ReLU does not change the non-negative values.
When the input value through residual connection can be negative (e.g. in MobileNetV2, ReLU is only located in the middle the ResBlock), we use PReLU instead of ReLU to reserve the information. The parameters of PReLU regarding to the additional channels are set to one. this way, PReLU behaves as Identity function.
From the above analysis, the input feature map can be well reserved by Conv 1, BN, and ReLU in the ResBlock.
Merging: We extend input channels in Conv and dirac initialize these channels. The value of i-th channel of a is the sum of the original i-th filters' output x! and the i-th input feature map 2t. which is equal to y' in the original ResBlock. The whole merging process can be formulated as:
Max
Σ
Σ
(WW R2 Meniss X R1+ xt "e Inghtiwts + R2 [,,)), 00)
C
K
K
Max
Σ
(WR22..
Gmnin x R1+
nnitiwt + B2
n
20
K
K
(3)
+ Σ
Σ
Σ
(Lenis x
-u + +))
C=C+1
j
M (
ΣΣΣ
m+ Inh4i + BR + x 0)
c,
where [+, *] means the concatenation of the elements.
Thus, by reserving and merging, we can remove the residual connection without changing the orig- inal output of ResBlock.
3.2 CONVERT RESNET TO VGG
In this section, we show how to apply RM operation to convert ResNet into VGG-like RMNet. Notice that there exist some DownSample ResBlocks at the first block of layer2, layer3, and layer4 in ResNet. The function of 'downsample" is for:
doubling the number of channels;
reducing the depth and width of the feature map by half.
Even though there are only few DownSample ResBlocks in ResNet, removing residual connection in such blocks is not as direct as in basic ResBlocks because there exist 1 x 1 convolution in the
4

---

Under review as a conference paper at ICLR 2022
Conea AKU
92
(a) DownSample RMBlock (type 1)
(b) DownSample RMBlock (type 2)
Figure 2: (a) and (b) show two methods of removing downsample branch in the original Down- Sample ResBlock.
downsample branch and it will change the input value in the path of residual connection. We propose two solutions in Figure to convert DownSample ResBlock to RMBlock.
As shown in Figure 2(a) the original 1 downsample kernel is padding with zeros to 3x3 kernel. we replace ReI.U in the block with PReI.U, whose parameters for each channel can control the scale of negative activation, as analyzed in Section 3.1 For the left (residual) branch, parameters as zeros, in this way, the function of the PReLU is equal to ReLU. PReLU is added after the downsample convolution, the parameters of which are initialized by one. In this way, PReLU is equal to the Identity Mapping function. New additional channels of the second convolutional layer are Dirac initialized (similar Figure [(b)), which keep the feature map after downsampling convolution unchanged. Then, all the layers in the left and right branches can be merged together into one sequence shown in Figure 2(a)
set the PReLU
In Figure 2(b) we separate the function of downsampling convolution into two convolutions in downsample branch. The first module 3x3 convolution with Dirae initialized filters and the stride is 2 for reducing the depth and width of the input feature map by half. The feature map through this Conv layer is still non non-negative and ReLU can reserve the value unchanged. The second module is 3 convolution, whose weights transferred from original x filters in downsample branch with 0 paddings. Since the original filters' kernel size is Ix 1, reducing the size of feature maps before doubling the number of channels will not change the receptive field in the original downsample branch.
Both methods equally convert the DownSample Blocks RMBlocks. The number of parame- ters of the first method is 108C + 4C, including Conv (C x 4C x 3 x 3), PReLU (4C), Conv (4C 2C 3 3). The number of parameters of the second method is 81C" consisting of Conv (C x 3C x 3 x 3), Conv 2 (3C x 2C x 3 x 3). We can see the number of parameters of the second method is only 0.75 times of the first method. Thus we use the second method to convert ResNet to RMNet in our experiment.
3.3 CONVERT MOBILENETV2 TO MOBILENETVI
Technically, there is no difference between converting ResNet and converting MobileNetV2 by using RM operation. However, the specialty of the structure of MobileNetV2 allows to further decrease the inference time by using parameter fusion after RM operation.
From Figure 3 after applying RM operation on MobileNetV2, the residual connections are removed, which leaves two Pointwise-ConwBN layers shown the dashed box. Since convolution and batch- normalization can be both presented by matrix multiplication and there exists no non-linear layer between them, these two Pointwise- Pointwise-Con-BN ConvBN layers can be fused. Mathematically, let X. Y. W, B be the input feature map, output feature map, weight, and bias of the first Pointwise-ConvBY layer in the dashed box and Z, W', B' be the output feature map, weight and bias of the second Dointwise-Come ConvBN layer in the dashed box. The process of fusing the parameters can be represented
5

---

Under review as a conference paper at ICLR 2022
MobileNetV2
RMNetV1
MobileNetV1
Pointwise ConvBN
Pointwise ConvBN
Pointwise ConvBN
ReLU
ReLU
RelU
Depthwise ConvBN
Depthwise
ReLU
ReLU
RetU
Pointwise ConvEN
Pointwise ConvEN
RM
Fuse
Pointwise ConvBN
Pointwise ConvaN
ReLU
ReLU
ReLU
Depthwise ConvEN
Depthwise ConvBN
Depthwise ConvBN
ReLU
ReLU
ReLV
Pointwise ConvEN
Pointwise ConvBN
Pointwise ConvBN
Figure 3: The process of converting MobileNerV2 into MobileNetV1.
2 = 2=WY + B'
=W + B) BB
=(W'W)X + B W
The fused weight is W W E R.n.l.11 and the fused bias is W B B' E R?. Thus, RM operation provides another opportunity to further decrease the inference time by fusing the parameters. After parameter fusion, the architecture of RMNet is identical to MobileNetVI which is very interest- ing since the presence of MobileNetV2 to increases the generalization ability of MobileNetV1 by utilizing residual connections. However, show RM operation inverse this process, i.e., converting MobileNetV2 into MobileNetV1 to make MobileNetVI great again. We provide experi- ments in Section 4.3 to verify the effectiveness of this transformation.
3.4
PRUNING RMNET
Reserving
RM
Prune
Shortcut
Merging
Figure 4: The process of pruning on ResNet. During the process, RMNet can serve as transition to gain larger pruning ratio.
Since RMNet does not have any residual connections, more friendly for filter pruning. this paper, we adopt Network slimming (Liu et al. 2017b) to prune RMNet because of its simpleness and effectiveness. Specifically, we first train ResNet and sparsity the weight in BN layer. Note an additional BN layer should be added in the residual connection during training since we also need to determine which additional filters are important after RM operation. After training, we convert ResNet into RMNet and prune the filters according to the weights in BN layers, which is identical to vanilla Network slimming. Figure 4 shows the overall process of pruning. Different from the traditional approach, RMNet serve transition to gain larger pruning ratio.

---

Under review as a conference paper at ICLR 2022
4 EXPERIMENTS
This section aranges as follows: in Section 4.1 we show RMNet has great advantage over RepVGG for deeper networks; in Section 4.2)
we show RMNet can achieve better speed-accuracy tradeoff on CIFARIO, CIFAR 100 and ImageNet datasets; in Section 4.3 show RMNet is applicable for
light-weight models; in Section 4.4 we verify the effectivenss of RMNet on network pruning task.
The speed of each model is tested Tesla V100 GPU with batch size of 128 by first feeding 50 batches to warm the hardware up, then 50 batches with time usage recorded.
4.1 REMAINING HIGH ACCURACY FOR DEEPER NETWORK
One power of DNN is that the representation ability increases as we raise the network depth. Thus we examine how network depth influences network performance on RMNet and RepVGG. Figure 5 shows the network performance on CIFAR-10 10 and CIFAR-100. For a fair comparison, the same nyper-parameters are deployed for each method: mini-batch size (256), optimizer (SGD), initial learning rate (0.1), momentum (0.9), weight decay (0.0005), number of epochs (200), learning rate decay 0.1 at every 60 epochs.
CIFAR1O Accuracy
CIFAR100 00RR100 Accuracy
13
88-
$6
RepWGG
RepiViCC
RecMel
Rendar
Depth
Depth
Figure This figure shows the influence of depth on RMNet and RepVGG.
the depth increases, RMNet get better accuracy. This is because, unlike Rep RMNet can equivalently convert ResNet into a plain model. Thus the performance will not decrease as the network goes deeper. In contrast, the accuracy of RepVGG will decrease (the reasons are analyzed at Appendix(A.2).. e.g. compared to RMNet 133 that achieves 79.57% accuracy on CIFAR-100, RepVGG 133 only has 41.38% accuracy.
From Figure 5
4.2 BETTER ACCURACY-SPEED TRADEOFF
From Section RMNet removes residual connections in the cost of bringing additional parameters. For example, in Figure [
RM operation doubles the number of parameters of the original ResBlock. To alleviate this issue, we use RM operation on the ResNet-based architectures with Inverted Resid- ual Block for designing RMNet. Inverted Residual Block (IRB) mainly contains three consecutive steps: point- wise convolution, group-wise convolution, and point-wise convolution. The first 'point- wise convolution' will increase the number of channels of the input feature map by T times and the second "point-wise convolution' will reduce the number of channels to the original number. For point-wise convolution, RM operation only increases the number of parameters and FLOPs by 7 times. Suppose the size of the input feature map is H x W x C' and the parameters of the first point- wise convolutional layer are expressed T parameters to reserve the input feature map, since RM operation is only to reserve the input feature also seaamter--rrenddy For example, for normal convolution,
C x 1. We only need additional C
1
map. 'Group- Group-wise wise convolution' we need C x K x K parameters to reserve the information of an input channel. While for group-wise convolution, we only need & times of parameters to reserve the information where G is the group number in group-wist convolution. Besides, the cost of storage and calculation does not change whether or not to use group-wise convolution. Based on the advantage of IRB, we design a series of RMNet models. We first determine the input and output width by the classic width setting of [64, 128, 256, 512]. We replace
Conv and MaxPooling on the head with two sequences of 3 x 3 Cony, BN, and ReLU (the same approach used in RepVGG). The tail of RMNet is the same as

---

Under review as a conference paper at ICLR 2022
ResNet. For RMNet 50x6_32, '50' indicates the number of all the Conv layers; '6' indicates the multiple of classic width setting, for example, the width of the grouped convolution layer in RMNee 50x6 32 is [6x64, 6x 128, 6 6525,, 6x512]; '32' indicates the number of channels of each group in RMBlock. We print out the detailed architecture of RMNet in the Appendix A..4 Next, we compare RMNNt with SOTA models CIFARIO, CIFARIO0 and ImageNet test the speed and accuracy. For fair comparison, we train RMNet following the official implementations of RepVGG
Table 1: Comparing RMNet with other SOTA models. RMNet 26 have [2. 2, 2] RMBlocks for each stage; R.MNet 41 have [2, 3, 5, 3] RMBlocks for each stage; RMNet 50 have [3, 4. 6, 3] R.Blocck for each stage; RMNet 101 have [3, 4, 23, 3] RMBlocks for each stage; RMNet 152 have [3, 8, 36, 3] RMBlocks for each stage.
Dataset
Backbone
Params(M)
FLOPS(G)
Accuracy(%)
Imgs/sec
ResNet 34
21.2
1.16
95.64
8208.4
DiracNet 34
21.1
1.15
94.83
6299.64
CIFARIO
ResDistill 34
21.1
1.16
94.62
9652.8
Rep A2
24.1
1.67
95.35
8099.7
RMNet 26x3332
8.8
0.51
95.81
10078.2
ResNet 34
21.3
1.16
78.61
8205.2
DiracNet 34
21.1
1.15
76.04
6285.3
CIFARI00
ResDistill 34
21.2
1.16
78.42
9621.9
RepVGG A2
24.2
1.67
78.41
8084.3
RMNet 26x3 32
8.9
0.51
79.16
10065.5
ResNet 50
25.6
4.11
76.31
1252.1
ResDistill 50
25.6
4.11
76.08
1494.4
RMNet41 x4
11.2
1.9
77.80
1460.4
ResNeXt 50
25.0
4.26
78.41
916.9
RepVGG B1
51.8
11.82
78.31
1123.2
RMNet 41 x5_16
23.9
3.79
78.5
1185.8
ResNet 101
44.5
7.83
77.21
729.9
VGG 16
138
15.48
72.21
782.1
ImageNet
RepVGG B2
80.3
18.38
78.78
712.9
RMNet 50 32
39.6
6,88
79.08
849.8
ResNet 152
60.19
11.56
77.78
512.8
RepVGG B3
111.0
26.21
79.26
543.5
RMNet x0.6_32
47.6
8.26
79.57
713.8
ResNeXt 101
88.8
16.48
79.88
308.4
RMNet 101 66166
59.45
11.11
80.07
461.5
RMNet 152x6_32
126.92
25.32
80.36
272.9
From Table I the accuracy of RMNet 50x6 322 is 2.3% higher than ResNet 101 and 0.59% higher than RepVGG B2 with the nearly same speed. worth noting that RMNet 101 x6.16 reaches over 80% top-1 accuracy without using any tricks, which is the first time for a plain model, to the best of our knowledge. Also, increasing the depth to 152 still has a benefit to RMNet, which shows the great potential of our method.
4.3 FRIENDLY FOR LIGHT- WEIGHT MODELS
We conduct an experiment to verify our analysis in Section 3.3 scratch and convert RMNetV1 following the process introduced in Section 3.3 initial architecture of MobileNetV2 has to be designed to make sure the transformed RMNetVI has the same architecture (depth and width) of MobileNetVI from the original paper (Howard al. 2017). Thus the MobileNetV2 in Figure is different from vanilla MobileNetV2 Sandler et al.
We first train a MobileNerV2 from
Note that the
8

---

Under review as a conference paper at ICLR 2022
2018 8 Table
We also train MobileNetV1 trained from scratch for comparison. The result show in
Table 2: Converting MobileNetV2 MobileNetV1 by RM operation and Fuse Operation on CIFAR- 10 and CIFARI00
Dataset
Backbone
Initial
w/wo Training
FLOPS(M)
Imgs/Sec
Acc(%)
MobileNetV2
Scratch
83.4
19119
92.07 0..17
CIFAR-I0
RMNeVVI
MobileNetV2
47.1
27720
92.07 1.118
MobileNetVI
Scratch
47.1
27720
91.31110.11
MobileNetV2
Scratch
V
83.5
19076
70.57:10.09
CIFAR-100
RMNetVI
MobileNetV2
47.2
27712
70.57+:0.09
MobileNetV1
Scratch
47.2
27712
68.75:+0.37
the speed of RMNetV1 (equivalently converted from MobileNerV2) is faster than vanilla MobileNetV2 and the performance of R.MNeVV is higher than MobileNetV1, which shows RM operation be very friendly for light-weight models.
From the experiment in Table 2
4.4 FRIENDLY FOR PRUNING
We conduct an experiment in Figure 6 to verify the effectiveness of pruning RMNet. We use L norm multiply a certain sparsity factor to punish weight BN layers, and regard the channels whose weight is smaller than certain threshold as invalid. The sparsity factor is selected from le-4 to 1e-3, and the threshold is selected from 5c-4 to 5e-3. A larger degree of sparsity will lead to larger pruning ratio.
CFAR 10 Accuracy (%)
44
92
CHAR 10
10
RM
20000
33000
50
16
63
56
180
Speed (Imgs/Sec)
Prumed Params Ratio (%)
Figure 6: (a) and (b) show the CIFARIO accuracy with respect network speed and pruned
parameters ratio in pruning task.
From the experiments, RMNet retains higher accuracy than ResNet under larger pruning ratio and when the speed of pruned RMNet is nearly the same with pruned ResNet, because of more reasonable structure. Thus RMNet has better accuracy-speed tradeoff over ResNet architecture on pruning tasks.
5 CONCLUSION AND DISCUSSION
In this paper, We propose RM operation to remove residual connections and perform RM operation convert ResNet and MobileNetV2 into plain models (RMNet). RM operation allows input feature maps to pass through the block while reserving their information and merges all the information the end of each block, which can remove residual connection without changing original output. Experiments have shown that RMNet can achieve better speed-accuracy trade-off and is very friendly for network pruning.
There are some future directions to be considered: 1) Analyzing the residual connection the Transformer based architectures and removing such connections via RM operation. 2) Searching residual networks with less residual connection with NAS and converting them to efficient RMNets.

---

Under review as a conference paper at ICLR 2022
REFERENCES
David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: If resnets are the answer, then what is the question? In Pro- ceedings of the International Conference on Machine Learning, pp. 342-350, 2017.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Proceedings of the Advances in Neural Information Processing Systems, pp. 6571-6588 2018.
Xiaohan Ding, Yuchen Guo, Guiguang Ding, and Jungong Han. Acnet:
Strengthening the ker- nel skeletons for powerful enn via asymmetric convolution blocks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1911-1920, 2019.
Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Diverse branch block: Building a convolution as an inception-like unit. arXiv preprint arXiv:2 13425, 2021a.
Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021b.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the International Conference on Artificial Intelligence and Statistics, pp. 249-2542266 256, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Proceedings of the European Conference on Computer Vision, pp. 630-645, 2016b.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 17X4::7704.04611, 2017.
Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi- Chi-Keung Tang. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv, fbs/1607.032550 2016.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitatio networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132-7141, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by
reducing internal covariate shift. In Proceedings of the International Conference Machine Learning, pp. 448-456, 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. In Proceedings of the Advances Neural Information Processing Systems, pp. 1097-1105, 2012.
Yann A LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Rober Müller. Efficient backprop. In Neural networks: Tricks of the trade, pp. 9-48. 2012.
Guilin Li, Junlei Zhang, Yunhe Wang, Chuanjian Liu, Matthias Tan, Yunfeng Lin, Wei Zhang, Jiashi Feng, and Tong Zhang. Residual distillation: Towards portable deep neural networks without shortcuts. In Proceedings of the Advances in Neural Information Processing Systems, 2020.
Hao Li, Asim Kaday, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference Learning Representations (ICLR), 2017.
10

---

Under review as a conference paper at ICLR 2022
Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 510-519, 2019.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang. Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision, pp. 2736-2744, 2017a.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In IEEE International Confer- ence on Computer Vision (ICCV), pp. 2755-2763, 2017b.
Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In Proceedings of the Interna- fional Conference on Machine Learning, pp. 3282-3291 2018.
Oyebade K Oyedotun, Djamila Aouada, Björn Ottersten, et al. Going deeper with neural networks without skip connections. In Proceedings of the International Conference on Image Processing, pp. 1756-1760, 2020.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 4510-4520, 2018.
Andrew M Saxe, James McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013.
Wenling Shang, Kihyuk Sohn, Diogo Almeida, and Honglak Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units. In Proceedings of the Inter- national Conference on Machine Learning, pp. 2217-2225, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations, 2015.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1-9, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818 2886-282,, 2016.
Christian Szegedy. Sergey loffe, Vincent Vanhoucke. and Alexander Alemi. Inception-v4, inception- resnet and the impact of residual connections on learning. In Proceedings of the Association for the Advancement of Artificial Intelligence. 2017.
Mingxing Tan and Quoe Le. Efficientnet:
Rethinking model scaling for convolutional neural net- works. In International Conference on Machine Learning, pp. 6105-6114, PMLR, 2019.
Mingxing Tan and Quoc V Le. Efficientnetv2: Smaller models and faster training. arXiv preprint arXiv:2104.0020 2021.
Andreas Veit, Michael Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. arXiv preprint 1rXiv:1605.064: 2016.
Weinan. proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 5(1::1111, 2017.
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel Schoenholz, and Jeffrey Pennington. Dynamical isometry and mean field theory of cnns: How train 10,000-layer vanilla convo- lutional neural networks. In Proceedings of the International Conference on Machine Learning, pp. 5393-5402, 2018.
Sergey Zagoruyko and Nikos Komodakis. Diracnets: Training very deep neural networks without skip-connections. arXiv preprint arXiv: 7736008888 2017.

---

Under review as a conference paper at ICLR 2022
A APPENDIX
The appendix is arranged as follows: in Section A.I
perform ablation studies; in Section A.2 we further analyze why RepVGG does not have high performace for deeper network; in Section A.3 we show fine-tuning RMNet can achieve higher performance; in Section A.4 we show the detailed architecture of RMNet.
A.1 ABLATION STUDY
In this section, we perform an ablation study to show how batch-size and hyper-parameters of net- work structure affect inference time for RM operation. The results are shown in Figure As shown in the left bottom sub-figure, the exact inference speed grows linearly with batch size when the batch size is relatively small, indicating the resource are sufficient. From the experiments, we can find that under the condition of sufficient resources, i.e. lower batch-size and lower complexity of network structure, RMNet has great advantage over ResNet. This is because RMNet a plain model with- out residual connections. The degree of parallelism can be greatly increased when the resources sufficient. Thus we believe RMNet has great potential in the future.
griie
Spredt Up
26
25
215
10
Channelsirroup
374
1901
314
Steo
Basic WWith
Mulisyy Times
Channess/Grupp
Figure 7: The first row shows the speed-up ratio of RMNet over ResNet. The second row shows the exact inference speed for ResNet and RMNet. Basic width, Multiply times and Stannelss/Grous are
of Network Structure introduced in Section 4.2 The larger the value, the more complex the network structure.
A.2 COMPARISON OF REPVGG AND RESNET
First Conv
Residual Connection
Conv
3x3 Conv
Residual Connection
Input
+
Output
Input
Output
Outside
Outside
(a) ResBlock ResNet
(b) RepBlock in RepVGG
Figure 8: Comparison of ResBlock in ResNet and RepBlock RepVGG.
12

---

Under review as a conference paper at ICLR 2022
We depict the basic blocks used in ResNet and RepVGG in Figure In ResBlock (Figure 8(a) two ReLUs are inside and outside of the residual connection, respectively. While in RepBlock (Figure 8(b)), since eepprrameterization is based on the commutative law of multiplication, two ReLUs have to be both outside of the residual connection. One can build RepVGG from basic ResNet architecture by replacing each ResBlock with two RepBlocks.
Next, We analyze why RepVGG can not be trained very deep like ResNet from both forward and backward passes:
forward path: (Veit et al. 2016) assumes that the success of ResNet can be contributed to the "model ensemble". we can regard ResNet ensemble of many paths of different lengths. Thus, n-blocks' ResNet have ((2") implicit paths connecting input and output. However, unlike ResNet that branches in the block are separable and can not be merged, multi-branches in RepVGG can be represented by one branch, which can be shown as:
ReLU + (N((UUL + BN(ConUsx3(z.)) 2(+1
(5)
ReLU (BLUUBNNCCoO (BN(Con)xxa(::))
where Cone' is the merged convolution of each Cone in the branches. Thus RepVGG does not have the implicit "ensemble assumption" of ResNet, and the representation gap between RepVGG and ResNet increases as the number of blocks raises.
backward path: (Balduzzi et al. 2017) analyzes "shattered gradients problem" in deep neural networks. The correlation of gradients behaves like 'White Gaussian Noise' when there are more ReLUS in the backward path. Suppose ResNet and RepVGG both have n layers. From Figure 8(a) Information in ResNet can pass through the residual without going through inside ReLU in each block. However, each ReLU in RepVGG is located in the main path. Thus the number of ReLUs Us for ResNet in the backward path is 2 while the number ReLUs for Rep in the path is n which suggests that the gradients in ResNet are far more resistant to shattering when the depth is large, leading to better performance than RepVGG.
n
Table shows the performance of RepVGG and ResNet ImageNet dataset. The setting of this experiment is followed by the official implementation of RepVGG. We can see under the same network structure, R8 is 0.5% higher than RepVGG-18, and ResNet-34 is 0.8% higher than RepVGG in terms of top-1 accuracy. Thus, RepVGG increases the speed at the cost of losing representation ability.
Table 3: Comparison of ResNet and RepVGG with the same depth and width on ImageNet.
Backbone
Params(M)
FLOPS(G)
Top-1 Acc(%)
Top-5 Acc(%)
Imgs/sec
ResNet 18
11.7
1.82
71.146
90.008
4548.59
RepVGG 18
11.5
1.80
70.638
89.608
4833.36
ResNet 34
21.8
3.67
74.442
91.89
2634.93
RepVGG 34
21.6
3.65
73.67
91.564
2772.17
A.3 FINETUNE
Compared to other approaches (Li et al. 2020 Ding et al. 2021b Zagoruyko & Komodakis 2017) that remove residual connections, RM operation can equivalently convert ResBlock to a plain mod- ule without re-training. However, if we fine-tune the pre-trained network, the performance may slightly improve since RM operation brings additional parameters into the network. It is worth not- ing that we need to carefully design the BN layer of additional channels (brought by RM operation) during fine-tuning since the running mean and running var in BN may lead to an unstable state of DNN. Thus We statistics the mean and variance by inputting the training dataset through the net- work, which will serve as the initial values running mean and running var of BN layers for better stableness. In addition, the weight and bias in BN are set by Equation to guarantee the equality of the model. Of course, there exists another solution that we do not add BN layers when fine-tuning, Le. fusing the BN with convolution before fine-tuning. However, it will lead to worse performance.
13

---

Under review as a conference paper at ICLR 2022
Figure shows the accuracy of ResNet 18/34 CIFAR-10/100 with/without fusing the BN layer when training. We can see the network without fusing BN has better performance and the accuracy is better than the pre-trained model.
00.50
29.00
My
84.75
CIPARIO
CIFARIO
04:25
54.0
Re5Net10 Baseline finetune with BN
ResNet33 Baseline w/o
03.8
93.75
Finetuune with UN
25
75
100
125
175
280
1:50 375
250
Epoch
Epoch
CIFAR100 Accuracy
CIPARI00 Accusey
77-
76
26.0
ResNet18 Baseline
Reshit33 Baseline
finetune w/o BN
finetune with BN
74
125
200
Epoch
Epoch
Figure 9: In each sub-figure, green line and blue line the accuracies the ResNet model with or without the BN layer, respectively.
A.4 DETAILED STRUCTURE OF RMNET
We print out the detailed network structure RMNet 26 3.32 Table From Table the most components in RMNet are 'Conv' and 'ReLU', making it very efficient on inference.
14

---

Under review as a conference paper at ICLR 2022
Table 4: The detailed network structure of RMNet 26 x 3_32
Sequential(
(0): Conv2d(3, 64, kemel_size==,, 3), stride=(1, 1), padding=(1, 1))
(I): ReLU(inplace=Trru))
(2): Conv2d(64, 192, kernel size=(1, 1), stride=(1, 1))
(3): Rel.U(inplace-TTue)
(4): Conv2d(192, 192, kir 3), stride=(1, 1), padding=(1, 1), groups=6)
(5): ReLU(inplacc--Tru))
(6): Conv2d(192, 64, kizel 1), stride=(1, I))
(7): ReLU(inplacc--Tuu))
(8): Conv2d(64, 192, kereel 1), stride=(1, 1))
(9): ReLU(inplace=Trre
(10): Conv2d(192, 192, kemnel 3), stride=(1, 1), padding=(1, 1), groups=6)
(11): ReLU/inplacc==Tue)
(12): Conv2d(192, 64, kernel size=(1, 1), stride=(1, 1))
(13): ReL.U(inllace//rue)
(14): Conv2d(64, 320, kernel_size=(( 1), stride=(1, 1))
(15): ReLUGinplace=True)
(16); Conv2d(320, 320, (3 3), stride=(2, 2), padding=(1, 1), groups=5)
(17): ReL.U(nplace=True)
(18): Conv2d(320, 128, kim=l 1), stride=(1, 1))
(19): ReL-Uinllace-True)
(20): Conv2d(12) 128, 384, kemel size=(1, 1), stride=(1, 1))
(21): Rel.U(inplace=/Tue)
(22): Conv2d(384, 384, kemel 3), stride=(1, 1), padding=(1, 1), groups=6)
(23): ReLU(inllace=True)
(24): Conv2d(384, 128, kemeL 1), stride=(1, 1))
(25): ReLU((mplace=True)
(26): Conv2d(128, 640, Gemmel stride=(1, 1))
(27): ReI.U(inllace=True)
(28): Conv2d(640,, 640, kemel_size= 3), stride=(2, 2), padding=(1, 1), groups=5)
(29): ReLU(inplace=True)
(30): C600 256, kernel size=(1, stride=(1, 1))
(31): ReL.U(inlaace=True)
(32): Conv2d(256, 768, siz= 1), stride=(1, 1))
(33): ReLUU(nplace==Tru))
(34): Conv2d(768, 768, k3mel 3), stride=(1, 1), padding=(1, 1), groups=6)
(35): ReL.U(inplace=Truee)
(36): Conv2d(768, 256, kemeeLsize==,, I), stride=(1, 1))
(37): ReLU(inplace=:ruc)
(38): Conv2d(256, 1280, kernel size=(1, 1), stride=(1, 1))
(39): Rel.U(inplace=TTre)
(40): Conv2d(1280, 1280, kernel 3), stride=(2, 2), padding=(1, 1), groups=5)
(41): ReLUGinlaac==True))
(42): Conv2d(1280, 512, keznelssiee=1,, 1), stride=(1, 1))
(43): ReLUÚnnlaac=-ruu))
(44): Conv2d(512, 1536, kernel size=(1, 1), stride=(1, 1))
(45): Rel.U(inplace=Truu)
(46): Conv2d(1536. 1536, kernel_size= 3), stride=(1, 1), padding=(1, 1), groups=6)
(47): ReLU(iplaae==Tue))
(48): Cony2d(1536. 512, kemnelssiz==() 1), stride=(1, 1))
(49): ReLUU(nplace==Tuu))
(50): AdiptivelygPPol22loouttttszzss
(51): Flatten(start dim=l, end end_dim=-1)
(52): Linear(in features=512, out features=10, bias=True))
15


Reviews:
[{"review_id": "qxRHdj4NCj", "reviewer": "Reviewer_Dt6G", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strength:\n1. The approach has showed constant classification performance improvements on the benchmarks. \n2. The idea is straightforward and easy to follow. \n\nThe weakness:\n1. More experiments are needed to validate the performance, like the performance on detection, segmentation benchmarks. \n2. Could the author list more inference comparison results across different hardware platforms, e.g,. cpu, gpu, popular embedded devices ? \n3. Beside the RM steps, there are also other additional operations like pruning operations, it would be good to see the improvements from different operations.\n\n", "summary_of_the_paper": "This work focuses on removing residual connection in network via  reserving and merging (RM) operations on the ResBlock. The author has tested the approach on classification tasks on several networks with skip connection, e.g., resnet, mobilenet v2, etc. The experiments demonstrated the good performance on the listed benchmarks. ", "summary_of_the_review": "In general, the paper has showed the strength in removing the residual connections on the classification tasks, it would be much more convincing to have more experiments on other popular tasks.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bl100tl2XT4", "reviewer": "Reviewer_oEx8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: \n\n1. The motivation of this paper is clear. Plain model can reduce off-chip memory access and avoid point operations like residual plus which is helpful for model deployment on specific platforms. This paper is also well-written and easy to follow.\n\n2. The method of re-parameterization is very useful in model design and network pruning. However, the re-parameterization has limitations such as the difficulty of removing residual connections across non-linear layers. This paper analyzes the limitation of re-parameterization method to remove residual connections such as RepVGG from both forward paths and backward paths. \nTo overcome the limitation, authors propose RM operation to equivalently remove residual connections from ResNet-based architectures which enables the plain models to keep higher performance for deeper networks. I think this ideology is very inspiring and may have an interesting combination with other research areas in the future.\n\n3. I especially like the part of network pruning. It is known that ResNet is harder to be pruned compared to plain model because of the existence of residual connections. This paper proposes a alternative way to prune ResNet. First, use RM operation to convert ResNet to RMNet, then perform pruning on RMNet. The result shows the speed is much faster than vanilla pruning on ResNet architecture. \n\n4. The part of the experiment for transforming MobileNetV2 is interesting. By further utilizing fusing operation, RM can transform MobileNetV2 into MobileNetV1 and the performance of MobileNetV1 keeps the same as MobileNetV2 which suggests that RM is applicable for light-weight models.\n\nWeakness:\n\nRM operation adds extra parameters and FLOPs for achieving equivalent transformation. Thus the benefit of the speed may be subject to the specific platforms, or additional operation like fusing two 1x1 Conv when converting MobileNetV2 into MobileNetV1.", "summary_of_the_paper": "This paper focuses on an important direction of removing residual connections from ResNet-based architectures and proposes RM operation to achieve this goal. RM operation consists of two steps: reserving and merging. The reserving operation allows input feature map passing through the Conv, BN,ReLU layers without changing their values and the merging operation adds the output feature map and the reserved input feature map together with the last Conv layer. Unlike re-parameterization method in RepVGG, RM Operation can remove residual connection across non-linear layers, resulting in equivalent transformation from ResNet to plain models. Authors also design a series of plain models with RM Operation named RMNet, which outperforms previous plain models such as DiracNet, ResDistill and RepVGG.", "summary_of_the_review": "The paper puts forward a novel method: RM Operation, which can equivalently remove residual connection across non-linear layer in ResNet-based architecture and shows great power in network pruning. I think RM operation is novel and can inspire future works on model design.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "E9p2kqCrYcB", "reviewer": "Reviewer_BQPh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "As the proposed conversion from residual network to plain network exactly preserves function, any throughput benefits are entirely determined by which form of operations is more efficient for the underlying combination of software library and compute hardware running the network.  Absent knowledge of such interactions, there does not appear to be any intrinsic reason for preferring the proposed form for rewriting residual layers.  In fact, the proposed approach seems to add computational work (more convolutional filters, Dirac initialized) for the sole purpose of taking advantage of fast implementations of convolutional layers.  Would a better approach simply be to write a custom fused low-level kernel for executing an entire residual block?  What form is actually better if one considers the optimal implementation possible for the underlying hardware (e.g., CPU, GPU, or TPU)?\n\nA similar question arises for the pruning approach.  If the functional forms of the residual and plain networks are equivalent, then any pruning operations on one form should have an equivalent expression in the other.  Why is it necessary to first convert to a plain network, instead of simply considering pruning of residual connections in the original form?\n", "summary_of_the_paper": "This paper proposes to convert residual network architectures to equivalent \"plain\" networks after training.  This is accomplished by augmenting convolutional layers with Dirac initialized filters (folding extraction of the residual signal into the conv layer), and modifying the subsequent Batch Normalization and ReLU layers (converting ReLU to PReLU) accordingly.  Motivations for doing this are to increase inference throughput, and allow additional pruning of the resulting plain network.\n", "summary_of_the_review": "I am not convinced that the core idea of the paper -- converting residual networks to an equivalent plain network -- addresses any fundamental issue.  The argument for speed of one vs the other is empirical, and may merely depend on what is optimized in the underlying software libraries.  To make a case here, the paper should provide analysis in terms of achievable parallel efficiency by an optimal implementation.  Similarly, the argument made for pruning RMNet seems to be one of convenience rather than fundamental difference -- is there not an equivalent (though perhaps not off-the-shelf) approach in terms of of pruning components of the original network?\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6P_marhQKpv", "reviewer": "Reviewer_ybcv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Main Review (+ = positive comment, o = neutral comment, - = negative comment)\n------------------\n\n\\+ The strength of the paper is the relatively easy to implement method that can be applied to many ResNet variants in order to remove the residual connections. The results and discussion on finetuning, pruning etc. are also quite useful. \n\n\\+ The paper has good empirical part that covers quite well comparison to related work, results on fine-tuning and pruning.\n\no While the paper is somewhat incremental in nature, it generalizes the previously proposed approaches to removing residual connections to other ResNet architecture variants, so it has importance in that sense.\n\no While the paper is relative practical in nature, it is fine, since as far as I understand the resulting RMNet is mathematically equivalent to the corresponding ResNets, it inherits all analysis from ResNets.\n\n\\- Some details are missing in the paper. For example in section 3, e.g., Fig. 1 and the text describe the setup with the traditional ResNet with only one relu inside the block, while MobileNetV2 has a different architecture (resembling more preactivation resnet), where the residual branch has no nonlinearities. It is mentioned that PReLU is used in the case of MobileNetV2, but it not clear weather the parameters of PReLU fixed with some pattern (identity for new and relu for others?) or learned using SGD. The paper only mentions that the PReLU for the additional channels uses weights “set to one”. \n\n\\- Could the authors discuss whether it is possible to use fixed filters in part of the network and train the final RmNet from scratch? If I understand correctly, RmNet is mathematically equivalent to the corresponding ResNet and this should suggest that this is possible. This would remove the need for multi-step setup, where first there is a resnet trained, then the RM operation is applied, then the network is potentially fine-tuned and pruned. Since this paper is relatively practical work, it would be good to describe how to do this in typical deep learning framework as well.\n\n\\- Applicability to typical preactivation PreActResNet is missing (although there is MobileNetV2)\n\n\\- Applicability to some other popular ResNet variants, such as shufflenet is missing\n\nMinor problems\n------------------------------\n\n\\- Merging operation could be shown in pseudo-code in addition to the mathematical description, but this is a minor point, since there is also a figure about it.\n\n\\- Spelling mistakes, page 4. “(ie.e, in ResNet, every ResBlock has a following a ReLU layer, which keeps input values are all non-negative)”\n\n\\- Grammar error page 13 A.3. Finetune, the sentence “Thus We statistics the mean and variance by inputting ...” is wrong and sound probably by “Thus we computed the mean and variance by inputting…”\n", "summary_of_the_paper": "Update: I have read the rebuttal, see below.\n-----------\nWhile residual connections are a key component in today's deep learning architectures, they can be problematic in some settings, e.g., in pruning. This paper presents an improved method (RMNet) for removing residual connections from a ResNet - type neural network after training. It improves over related work and in contrast to those, it works on some typical ResNet variants. The paper also discusses fine-tuning, pruning and more efficient architectures.", "summary_of_the_review": "\\+ Generalizes previous methods to other typical ResNet types\n\n\\+ Extensive Empirical evaluation\n\no/- Somewhat incremental, quite empirical\n\n\\- Some details missing", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper propose a reparametrization approach for pruning residual networks. The proposed approach replace the skip layer connections with feedforward layers, and show the equivalence to the original network. However, the current presentation is not very clear on the advantage of the proposed approach for pruning. As two networks are equivalent, pruning the reparameterized network can be transferred to pruning the residual network. The authors need to clarify how their reparametrized network is different from the residual network when being pruned. More ablation studies are also need to better justify their claim.
Decision: Reject