Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to convert residual network architectures to equivalent "plain" networks after training.  This is accomplished by augmenting convolutional layers with Dirac initialized filters (folding extraction of the residual signal into the conv layer), and modifying the subsequent Batch Normalization and ReLU layers (converting ReLU to PReLU) accordingly.  Motivations for doing this are to increase inference throughput, and allow additional pruning of the resulting plain network.

main_review: As the proposed conversion from residual network to plain network exactly preserves function, any throughput benefits are entirely determined by which form of operations is more efficient for the underlying combination of software library and compute hardware running the network.  Absent knowledge of such interactions, there does not appear to be any intrinsic reason for preferring the proposed form for rewriting residual layers.  In fact, the proposed approach seems to add computational work (more convolutional filters, Dirac initialized) for the sole purpose of taking advantage of fast implementations of convolutional layers.  Would a better approach simply be to write a custom fused low-level kernel for executing an entire residual block?  What form is actually better if one considers the optimal implementation possible for the underlying hardware (e.g., CPU, GPU, or TPU)?

A similar question arises for the pruning approach.  If the functional forms of the residual and plain networks are equivalent, then any pruning operations on one form should have an equivalent expression in the other.  Why is it necessary to first convert to a plain network, instead of simply considering pruning of residual connections in the original form?

summary_of_the_review: I am not convinced that the core idea of the paper -- converting residual networks to an equivalent plain network -- addresses any fundamental issue.  The argument for speed of one vs the other is empirical, and may merely depend on what is optimized in the underlying software libraries.  To make a case here, the paper should provide analysis in terms of achievable parallel efficiency by an optimal implementation.  Similarly, the argument made for pruning RMNet seems to be one of convenience rather than fundamental difference -- is there not an equivalent (though perhaps not off-the-shelf) approach in terms of of pruning components of the original network?
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work focuses on removing residual connection in network via  reserving and merging (RM) operations on the ResBlock. The author has tested the approach on classification tasks on several networks with skip connection, e.g., resnet, mobilenet v2, etc. The experiments demonstrated the good performance on the listed benchmarks. 
main_review: The strength:
1. The approach has showed constant classification performance improvements on the benchmarks. 
2. The idea is straightforward and easy to follow. 

The weakness:
1. More experiments are needed to validate the performance, like the performance on detection, segmentation benchmarks. 
2. Could the author list more inference comparison results across different hardware platforms, e.g,. cpu, gpu, popular embedded devices ? 
3. Beside the RM steps, there are also other additional operations like pruning operations, it would be good to see the improvements from different operations.


summary_of_the_review: In general, the paper has showed the strength in removing the residual connections on the classification tasks, it would be much more convincing to have more experiments on other popular tasks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Update: I have read the rebuttal, see below.
-----------
While residual connections are a key component in today's deep learning architectures, they can be problematic in some settings, e.g., in pruning. This paper presents an improved method (RMNet) for removing residual connections from a ResNet - type neural network after training. It improves over related work and in contrast to those, it works on some typical ResNet variants. The paper also discusses fine-tuning, pruning and more efficient architectures.
main_review: Main Review (+ = positive comment, o = neutral comment, - = negative comment)
------------------

\+ The strength of the paper is the relatively easy to implement method that can be applied to many ResNet variants in order to remove the residual connections. The results and discussion on finetuning, pruning etc. are also quite useful. 

\+ The paper has good empirical part that covers quite well comparison to related work, results on fine-tuning and pruning.

o While the paper is somewhat incremental in nature, it generalizes the previously proposed approaches to removing residual connections to other ResNet architecture variants, so it has importance in that sense.

o While the paper is relative practical in nature, it is fine, since as far as I understand the resulting RMNet is mathematically equivalent to the corresponding ResNets, it inherits all analysis from ResNets.

\- Some details are missing in the paper. For example in section 3, e.g., Fig. 1 and the text describe the setup with the traditional ResNet with only one relu inside the block, while MobileNetV2 has a different architecture (resembling more preactivation resnet), where the residual branch has no nonlinearities. It is mentioned that PReLU is used in the case of MobileNetV2, but it not clear weather the parameters of PReLU fixed with some pattern (identity for new and relu for others?) or learned using SGD. The paper only mentions that the PReLU for the additional channels uses weights “set to one”. 

\- Could the authors discuss whether it is possible to use fixed filters in part of the network and train the final RmNet from scratch? If I understand correctly, RmNet is mathematically equivalent to the corresponding ResNet and this should suggest that this is possible. This would remove the need for multi-step setup, where first there is a resnet trained, then the RM operation is applied, then the network is potentially fine-tuned and pruned. Since this paper is relatively practical work, it would be good to describe how to do this in typical deep learning framework as well.

\- Applicability to typical preactivation PreActResNet is missing (although there is MobileNetV2)

\- Applicability to some other popular ResNet variants, such as shufflenet is missing

Minor problems
------------------------------

\- Merging operation could be shown in pseudo-code in addition to the mathematical description, but this is a minor point, since there is also a figure about it.

\- Spelling mistakes, page 4. “(ie.e, in ResNet, every ResBlock has a following a ReLU layer, which keeps input values are all non-negative)”

\- Grammar error page 13 A.3. Finetune, the sentence “Thus We statistics the mean and variance by inputting ...” is wrong and sound probably by “Thus we computed the mean and variance by inputting…”

summary_of_the_review: \+ Generalizes previous methods to other typical ResNet types

\+ Extensive Empirical evaluation

o/- Somewhat incremental, quite empirical

\- Some details missing
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper focuses on an important direction of removing residual connections from ResNet-based architectures and proposes RM operation to achieve this goal. RM operation consists of two steps: reserving and merging. The reserving operation allows input feature map passing through the Conv, BN,ReLU layers without changing their values and the merging operation adds the output feature map and the reserved input feature map together with the last Conv layer. Unlike re-parameterization method in RepVGG, RM Operation can remove residual connection across non-linear layers, resulting in equivalent transformation from ResNet to plain models. Authors also design a series of plain models with RM Operation named RMNet, which outperforms previous plain models such as DiracNet, ResDistill and RepVGG.
main_review: Strength: 

1. The motivation of this paper is clear. Plain model can reduce off-chip memory access and avoid point operations like residual plus which is helpful for model deployment on specific platforms. This paper is also well-written and easy to follow.

2. The method of re-parameterization is very useful in model design and network pruning. However, the re-parameterization has limitations such as the difficulty of removing residual connections across non-linear layers. This paper analyzes the limitation of re-parameterization method to remove residual connections such as RepVGG from both forward paths and backward paths. 
To overcome the limitation, authors propose RM operation to equivalently remove residual connections from ResNet-based architectures which enables the plain models to keep higher performance for deeper networks. I think this ideology is very inspiring and may have an interesting combination with other research areas in the future.

3. I especially like the part of network pruning. It is known that ResNet is harder to be pruned compared to plain model because of the existence of residual connections. This paper proposes a alternative way to prune ResNet. First, use RM operation to convert ResNet to RMNet, then perform pruning on RMNet. The result shows the speed is much faster than vanilla pruning on ResNet architecture. 

4. The part of the experiment for transforming MobileNetV2 is interesting. By further utilizing fusing operation, RM can transform MobileNetV2 into MobileNetV1 and the performance of MobileNetV1 keeps the same as MobileNetV2 which suggests that RM is applicable for light-weight models.

Weakness:

RM operation adds extra parameters and FLOPs for achieving equivalent transformation. Thus the benefit of the speed may be subject to the specific platforms, or additional operation like fusing two 1x1 Conv when converting MobileNetV2 into MobileNetV1.
summary_of_the_review: The paper puts forward a novel method: RM Operation, which can equivalently remove residual connection across non-linear layer in ResNet-based architecture and shows great power in network pruning. I think RM operation is novel and can inspire future works on model design.
review_title: null

Meta-review: This paper propose a reparametrization approach for pruning residual networks. The proposed approach replace the skip layer connections with feedforward layers, and show the equivalence to the original network. However, the current presentation is not very clear on the advantage of the proposed approach for pruning. As two networks are equivalent, pruning the reparameterized network can be transferred to pruning the residual network. The authors need to clarify how their reparametrized network is different from the residual network when being pruned. More ablation studies are also need to better justify their claim.
Decision: Reject