Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new way to measure the 'importance' of a neuron, by re-parameterizing the scaling parameters of the batchnorm layers. The proposed method is to use a sigmoid on the scaling parameter, to ensure that it lies in [0,1], and the value of the sigmoid reflects the importance. Using this value (value closer to 1 means important), the 'unimportant' neurons are pruned in a one-shot manner. The paper also proposes a new regularizer for this scaling parameter, to control the amount of neurons to be pruned. The results on simpler image classification tasks like CIFAR-10, -100 suggest that this method indeed outperforms other methods, particularly at higher pruning levels. However, on more complex task like Imagenet, it seems to perform worse than the UCS method. Further, the algorithm is only evaluated on image tasks (CIFAR and ImageNet), and two architectures (ResNet and VGG).
strength_and_weaknesses: Strengths:
1. The proposed measure and regularizer seem to be a more principled magnitude based pruning method than existing methods that directly work on the magnitude and apply an L1 regularizer. 

2. The results on CIFAR-10, 100 suggest that indeed this method has potential, since it outperforms existing methods on higher pruning ratios.

Concerns:
1. The proposed method is only tested on 2 architectures: ResNet and VGG, and on image tasks CIFAR and ImageNet. 
           
- Can the method be applied to Transformer architecture on language tasks? 
- The paper applies the method only for one-shot pruning. Can this be applied in multiple iterations, pruning a certain fraction of neurons in each round?
- Can this method be used to prune pre-trained networks? For example, take a pretrained network and fine-tune if for a few epochs on a downstream task, and then apply this pruning method.

2. Although the method performs well on simpler tasks like CIFAR, it performs worse than UCS on difficult tasks like ImageNet. The paper says that this is because "This again indicates that for more difficult datasets, global methods over-prune the wrong layers". I think this is a major limitation. However, I think this might be remedied, for example, by a per-layer re-normalization of the importance scores, so that a similar ratio of neurons are pruned in every layer.

3. (Minor) The paper promotes the proposed method by claiming that it measures the 'importance' of neurons by bounding the scaling parameter of batchnorm in the range [0, 1]. However, I am not fully convinced by this. Consider the case when a neuron has a low scaling parameter, but in its outgoing weights to the next layer, a few (say 10%) of the weights are large. That means that even though the output magnitude of that neuron is low, some neurons in the next layer give high weightage to its output. In that case, this neuron can still be important. In short, what I mean to say is that just the magnitude of a neuron's output might not necessarily capture its 'importance'. While this method still seems more principled than previous methods at using the scaling parameter's magnitude for pruning, I do not agree that it captures the importance of the neuron. I would suggest clarifying this in the paper.

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. I have not tried to run the provided code to reproduce the results.
summary_of_the_review: I think the proposed method to measure the 'importance' of a neuron is more principled than previous methods that use magnitude, but I think the method can be more refined to deal with the problem of over-pruning the wrong layers for complex tasks, and by adding more experiments to show how the method performs for different architectures and tasks.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors proposed a channel-pruning method by regularizing the BatchNorm scaling parameters, and proposed a BatchNorm variation suitable for pruning. The authors point out that the previous BatchNorm-based pruning methods shrink all parameters with an equal gradient irrespective of their importance. Thus, the authors propose to remove the offset $\beta$ in BatchNorm and bound the scale parameter $\gamma$ by applying the sigmoid function before multiplication, and the Receding Neuron Importance regularization for pruning. However, the design is lacking in theoretical support and the authors didn’t prove the proposed method achieved better accuracy than previous methods.
strength_and_weaknesses: Channel pruning is an important task in neural network compression and the effort towards better channel pruning methods is appreciable.

Experimental results are weak. The results on the ImageNet dataset are much worse than the previous state-of-the-art results.

[ref1] Soft filter pruning for accelerating deep convolutional neural networks

[ref2] Pruning filter via geometric median for deep convolutional neural networks acceleration.

[ref3] Metapruning: Meta-learning for automatic neural network channel pruning

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and easy to follow and the quality is good.

Novelty is limited. The original idea of regularizing BatchNorm parameters for channel pruning is proposed in Slimmable Network (Liu et al., 2017). Based on that, this paper proposed the adjustment for BatchNorm, which seems marginal and not well-supported by either mathmatical proof or strong experimental results.
summary_of_the_review: The paper is well-written, however, the proposed method seems incremental, the design is lacking in theoretical support and the experimental results are weak.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a simple variation of batch normalization with a bounded scaling parameter to estimate the importance of each channel. To only suppress unimportant units while preserving important units, the authors design a regularization loss on these bounded scaling factors, with additional hyperparameters. The proposed approach is validated on CIFAR and ImageNet with VGG, ResNet, and MobileNet models.
strength_and_weaknesses: Strength
- The idea that shrinks only the outputs of unimportant neurons while maintaining the others is well-motivated.
- The sigmoid batchnorm is sufficiently simple for people to build on, although the proposed regularization loss involves additional hyperparameters.
- I believe the experimental validation is quite strong and extensive; various architectures with different capacities were used, and the baselines are recently proposed methods and competitive.
- The authors provide detailed analyses on the hyperparameters and where to be pruned, which help understanding the network behavior. 
- The paper is generally quite thorough and generally feels complete: I believe it is generally ready for publication if it is decided by the reviewers that significance is sufficient to warrant publication.

Weakness
- As the authors described, this paper and Polarization Regularizer (Zhuang et al., 2020) share the same motivation and take a similar approach in terms of optimizing batchnorm scaling factors in a desired way. Although the authors briefly mentioned the difference in the related work section (i.e., While the method effectively increases the margin between important and unimportant neurons, it does so by both shrinking and expanding weights.), I am not sure why the proposed method should be better than Polarization Regularizer. Could the authors provide the merits of your work over Polarization Regularizer in more detail? 
- I am not sure whether the proposed method is effective for ResNets and is generally applicable to resblock-equipped networks. For instance, in the right part of Table 2, RNI slightly outperforms the baselines but with more FLOPs at 90% pruning ratio. In the right part of Table 3, at 90% pruning ratio, the accuracies are very similar. Furthermore, in Table 4, the superiority of RNI is more questionable: Slimming and Polarization yield better results with less FLOPs at 50% ratio.
- I think the comparison in Table 5 should be improved; matching the number of parameters may be too naive. I would recommend the authors to add FLOPs to Table 5 and perform additional experiments using the network with similar FLOPs or latency. 

clarity,_quality,_novelty_and_reproducibility: - The idea to obtain scaling factors that enable better discrimination between unimportant and important neurons is well-motivated and intuitive.
- The paper is clearly written and well polished. I think most members of the community would be able to easily dissect it without requiring substantial prior knowledge and without being misled.

summary_of_the_review: In general, I believe this work did a good job in terms of designing interesting sparsity-inducing regularization for structured pruning. Although the in-depth comparison with Polarization Regularizer is necessary and some results are questionable, the experiments and analyses are quite good and thorough.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: **Summary:** This method proposes a slight tweak to the way we do channel pruning in convolutional neural networks. Rather than pruning based on the batchnorm gamma parameter, it adds a sigmoid around that parameter (both in the network and for pruning purposes) and prunes based on that. This ensures that all of these parameters are within the same scale range. The authors claim that the method "significantly outperforms existing approaches," but the evaluation doesn't seem to support that claim.

**Strengths:** The modification is simple and easy to implement.

**Weaknesses:**
* This method doesn't conclusively do better than other methods. The authors tout the smallest-scale experiments on VGG-like networks for CIFAR, but the method is not definitively better than others on ResNet-50 on ImageNet (which is the de facto standard task in pruning right now). Several of the comparisons are at very different FLOP numbers and accuracies for different methods, so it's difficult to make an apples-to-apples comparison. Pareto curves showing the tradeoff at many different points for these methods would be more helpful. Despite the authors' claim that this method "significantly outperforms existing approaches," that is not apparent from the results. There are other methodological weaknesses in the paper, such as a lack of error bars, that suggest the evaluation is incomplete.
* The method is really simple. There isn't much novelty. A single sigmoid is added in one place. No, a method doesn't have to be complicated to be effective (and many of the best tricks we have in deep learning are very simple), but this method (a) is a minimal change, (b) isn't clearly justified, and (c) doesn't actually help very much. This is in contrast to similarly small changes like residual connections, batchnorm, or dropout, which are small but very significant in their impact on the resulting network.

**Recommendation:** Reject
justification_for_why_not_higher_score: The evaluation does not support the claim that this method is an improvement in the most challenging settings.
Decision: Reject