Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on preserving the trainability of network after pruning. As proposed by the authors, the trainability of pruned networks can be affected with the phenomenon of being less robust to finetuning learning rate. To alleviate this issue, the authors proposes the trainability preserving pruning (TPP) that can maintain the trainability during pruning. The authors suggest that the dependency between weights may cause the issue. So in this paper, the authors propose to decorrelate the pruned weights and the kept weights. In detail, the authors propose to regularize the gram matrix of weights in the way that the correlation of between the kept and pruned weights approaches 0. In addition, the authors suggest that the parameters of BatchNormalization should be explicitly considered for pruning by regularizing the parameters. Experiments on MNIST, Cifar and ImageNet verifies the efficacy of the proposed methods. On ImageNet classification task  with ResNet50, the proposed TPP  illustrates significant improvement compared with other SOTA methods.
strength_and_weaknesses: Pros
This paper proposes a post process for pertained network to get  more trainable sparse weights for finetuning. Experiments verifies the trainability of the sparse weight via comparisons with different finetuning  learning rate.  Extensive experiments on several datasets illustrates the efficacy of the proposed method. In addition, the authors also consider the effect of batch normalization parameters and adds to the regularization term.

Cons
The authors illustrate the ablation with regularizing  BN weight and without regularizing BN weight. However, I do not find the results of using the regularization term of BN parameters alone. Could the authors show this result?


Question
As shown in the paper, the proposed TPP method is a post-process algorithm for a pertained model. After the TPP process, the sparse weight is finetuned to get a sparse model with decent performance. I am wondering whether the TPP algorithm can be combined with the pretraining? In addition,  the TPP post process also takes the classification loss into consideration, so it can be considered as the part of finetuning. It would be better to have the experiments with similar finetuning cost to compare TPP with other methods in a more fair way. Whether the proposed TPP can be useful for finding lottery ticket subnetwork in a filter level?
clarity,_quality,_novelty_and_reproducibility: this paper is well written and of good novelty.
summary_of_the_review: please refer the comments listed above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present trainability prerseving pruning (TPP), considering to maintain trainability for the pruned networks. The authors construct two regularization terms to achieve this goal. TTP decorrelates the pruned weights from the kept weights, thus achieves non-trivial improvement in pruning process. 
strength_and_weaknesses: Strength:
1. The trainability of pruned models is interesting, and may provide a new perspective to pruning, and even to the whole deep learning field.
2. The proposed pruning method is easy to implement, and produces good experimental results.

Weakness:
1. The relationship between trainability and dynamical isometry is not clear. I find no evidence to indicate this point, i.e. how can trainability be expressed by the orthogonal property of channels?
2. The necessity of contraining BN parameters is not shown. The reason in the paper is that "Although unimportant weights are enforced with regularization for sparsity, their magnitude can barely be exact zero, making the subsequent removal of filters biased". However, if the pruned weights are not zeros, how to achieve the speedup of models during  the inference phase?
clarity,_quality,_novelty_and_reproducibility: The logic of the paper is easy to follow, but some motivations are not clear.
The method proposed in the paper is novel.
The work is original.
summary_of_the_review: The paper focuses on an interesting topic, and proposes a working solution. However, the motivation of the methods and some specific techniques are not very clear. Therefore, I think this paper should be weakly rejected.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a neural network pruning method which perserves trainability after pruning. The authors argue that the proposed approach will make the network easier to train after pruning therefore lead to a better finetuned model. First, the proposed method use L1 norm to select important and unimportant filters. Then,  a trainability regularization term is induced during the finetuning step which enforces the important filters to be orthogonal to unimportant ones. In addition, parameters in BN layers are regularized by L2 norm for those unimportant filters. The authors validate the proposed method on CIFAR-10/100 and ImageNet-1k.
strength_and_weaknesses: [Strength]

* The paper is overall well-written and easy to follow.
* The authors compared their method to multiple strong baseline methods, such as OrthConv and KernOrth.
* In the finetuning, the authors compared two different learning rates. All methods are trained under the same settings for fairness.
* Experiment results on ImageNet are reported, in addition to the CIFAR-10/100, including a dozen baseline methods.
* The accuracy improvement is significant on CIFAR-10/100

[Weakness]
* The novelty is incremental. Orthogonal regularization is not new in training deep network. Dating back to years ago, there were lots of discussions of weight regularization to make network more trainable, such as dynamic isometry constraint or nuclear norm regularizer. This work extends these ideas to network pruning, with some heuristic adaptations. For example, the correlation between selected important filters is not penalized in the proposed regularizer. BN norms of unimportant filters are considered. However, all these adaptations lack strong theoretical motivations. The authors did not provide any theoretical insight of why the proposed decorrelation plays a critical role in deep neural network pruning. From the numerical results, it seems that the improvement of the proposed method is also marginal on large-scale datasets.

* Marginal improvement on large-scale datasets such as ImageNet. The difference sof the proposed method v.s. previous SOTA methods are around or less than 0.5% in most cases. This is not significant, especailly when the training setting in this work is not SOTA. For example, using a better training recipe in "Resnet strikes back: An improved training procedure in timm“, it is possible to improve ResNet-50 from 76% to 80% on ImageNet-1k. The 0.5% improvement is hardly to say a significant number.

* No trained long enough. In SOTA training setting, ResNet-50 on ImageNet requires 360 epochs of training. Training even longer can still improve accuracy a bit. In this paper, most training experiments are early-stopped. So it is hard to tell the final accuracies when all models are trained by 360 epoches.

* Training curves not reported. As the key argument of this work is trainability, the convergence curve should be plotted. Again, please consider to train long enough such that all models achive their stable convergent points. Then please compare the convergence speed.

clarity,_quality,_novelty_and_reproducibility: * The paper is presented clearly, with sufficient related works and related discussions. 
* Experiments come with sufficient details. Training parameters are provided in appendix. It should not be difficult to reproduce results in this work.
* This work lacks novelty in both theory and methodology. The practical impact may be very limited.
summary_of_the_review: The majority concern is lacking novelty. Some improvments in the experiments would make this work more stronger and more convincing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper propose a regularization method to maintain the trainability of the model during the filter pruning process. The regularization is applied on both the weight and the BN parameters. Experiment results on multiple dataset and model architectures show promising performance comparing to other SOTA pruning methods.
strength_and_weaknesses: ## Strength
1. The paper makes novel and significant contribution by introducing the concept of trainability preservation into the neural network pruning process.
2. The proposed method is well motivated and theortically sound
3. Thorough emperical evaluation is provided to prove the effectiveness of the proposed method

## Weakness
1. The paper only demonstrates the model performance during the finetuning process. As a regualrization paper, it would be interesting to show how the regularization perform during the regularized training process. Things like how does the regularization affect model accuracy before pruning, and how it changes the mean JSV. I would encourage the author to repeat Figure 3 with the regualrized training epochs before the pruning.
2. To my understanding the regularizer requires additional training epochs before pruning to take effect, however there's no discussion on how many epochs is needed, and how it's decided. This may also cause unfair comparison for the results in Table 1 and 2, as some baselines may be trained with less epochs in total (both before and after pruning), and may benefit from more training epochs
3. The paper limits the exploration on L1 megnitude-based pruning with fixed pruning precentage on each layer. However there are more advanced filter pruning techniques like Taylor importance-based global pruning[1], or dynamically determining the filters to be pruned during regularized training process[2,3]. It would be interesting to see if the proposed method works under these pruning schemes.

[1] https://arxiv.org/abs/1906.10771

[2] https://proceedings.neurips.cc/paper/2016/file/41bfd20a38bb1b0bec75acf0845530a7-Paper.pdf

[3] https://arxiv.org/pdf/1908.09979.pdf
clarity,_quality,_novelty_and_reproducibility: The paper makes novel and significant contribution by introducing the concept of trainability preservation into the neural network pruning process. The paper is overall clearity written and easy to follow. The quality of the paper is generally good, but can be improved by further showing the model behavior during the regularized training phase with the proposed method, and consider the additional training steps needed by the proposed method in discussing the experiment results.
summary_of_the_review: Generally the paper proposes an interesting and strong method for improving the finetuning trainability of the pruned model. The method is novel and solid. Meanwhile there are some concern on the behavior of the model during the regularized training phase and the fairness of the experiment. I would recommend a weak acceptance for now, expecting the author to clear my doubt in the rebuttal.

## Post rebuttal

I have read other reviews and all responses from the author. Most of my concerns are resolved. The paper provides a novel prespective for improving DNN pruning, where is shows small but consistent improvement in accuracy under practical sparsity, and larger accuracy improvement under high sparsity. I would keep my score as weak accept.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes the trainability preserving pruning (TPP) that can maintain the trainability during pruning. This is achieved by regularizing the gram matrix of convolutional filters to decorrelate the pruned filters from the retained filters. In addition to the convolutional layers, the batch normalization parameters are also regularized to preserve the trainability of the network. There was an extensive discussion over the paper, and I communicated with the reviewers multiple times during the discussion period. The authors eagerly addressed most of the concerns about novelty and provided new large-scale experimental results on ImageNet which further confirmed the effectiveness of the proposed method. Hence, I recommend acceptance.
justification_for_why_not_higher_score: The proposed method is not backed up by theory.
Decision: Accept: poster