Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper presents a training-time pruning method for deep neural network. The main idea is to include network elements only if they could improve the predictive mean of the saliency, where the saliency measures the efficiency of the network elements in terms of minimizing the loss function. 
The paper presents a clear objective of the early pruning, which is to preserve the sub-network that can maximize the saliency function. This optimization problem is NP-hard, and even approximation is very expensive. The authors proves that one can simplify the approximation by ranking the network element by predictive mean of the saliency function.

The authors evaluated their method empirically and showed that it is superior than the GP modeling, and provides a trade-off between accuracy and the training time.

+ This paper is well organized. The theoretical analysis is well written and provides a good review to readers who lack relevant background.
+ The idea is interesting and Lemma 1 should be of value to researchers working on the similar problem.
+ Reading from the review section, the problem of training-time pruning is not well studied yet, whereas this paper could be seen as an important contribution.
- The experiment section is a bit hard to follow. Among other problem, I find it hard to understand the intuition behind the dynamic penalty scaling.
- It is not clear to me why the dynamic sparse reparameterization methods are not listed as baseline in evaluations.

Overall I think this is a good paper. The authors could improve it by addressing the two problems in the evaluation section.
review_title: Well organized, interesting contribution, experiments could be improved

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces a new method to accelerate training by saliency-based pruning. The method predicts future saliency for neurons based on observed saliency with a multi-output Gaussian process (MOGP), then greedily prunes neurons with least saliency at fixed intervals during training. The authors provide extensive mathematical analysis to show that the algorithm produces pruning mask solutions that are close to the optimum of the formulated optimization (the reviewer is unable to verify). The experimental results showed improvements in task accuracies of trained models but with longer training times. 

The reviewer believes that the proposed method is novel, as it considers historical statistics during training to provide a more accurate saliency prediction. While this is interesting, the reviewer is concerned with the practicality. The paper can be improved with answers to the following questions:
* The mathematical analysis showed that the algorithm produces pruning mask solutions that are close to the optimum. Is it possible to quantify? The reviewer worries that under a series of approximations and heuristic-based modeling below, the solution no longer aligns with the goal of pruning optimality, and thus makes the mathematical proofs irrelevant:
	* 3.1 problem statement as the pruning objective,
	* 3.2 saliency as MOGP with exponential kernel,
	* 3.3 subsequent simplifications,
	* 4 variational approximation of MOGP.
* What is the overhead introduced by the maximum likelihood estimation of the MOGP, and Bayesian early pruning (BEP, Algorithm 1)? Does the benefit of having a pruned model always outweigh the costs of pruning?
* How accurate is the saliency prediction? Can this be quantified and illustrated somehow, e.g. with the predicted values on Figure 2 in the appendix? If this is not accurate that one may expect the components in the pruning procedure can be replaced with simpler variants without a detrimental impact. For instance, instead of using MOGP for prediction, one may consider static saliency. (Or is this identical to traditional approaches used by SNIP and GRASP?)

Additionally, experiments should not only have averaged results but also provide standard deviations.

The paper is in general well-written, the reviewer has minor complaints:
* In Algorithm 1, `\mu_{T \mid 1:t}` is not assigned value anywhere.
review_title: Review - Bayesian early pruning

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary

This paper introduces a method for pruning during the training process in order to filter out unimportant/redundant components of the network continuously to speed up training and perform gradual pruning over the training process. The proposed approach is novel in the sense that the vast amount of prior work on pruning has focused on either (i) pruning on network initialization (e.g., SNIP, etc.) or (ii) pruning after the network has been fully trained (e.g., Magnitude Pruning, among many others). The introduced method uses the Taylor-series based saliency criterion (of Molchanov et al., 2017) and uses a multi-output Gaussian process to predict future saliencies and to determine whether a parameter can be safely removed early on during training.


Rationale for Score
	
As far as the negatives go, I take issue with the fact that the proposed approach seems to be highly complicated -- requiring a multitude of hyper-parameters/design choices, and tuning functions/ablation studies (e.g., for lambda). The empirical results are also not very compelling -- the proposed approach requires many more training hours than compared approaches that prune on initialization (cf., SNIP or GRASP in Table 3) and attains only a modest (.3% to .5%) improvement in pruning performance as measured by test accuracy.

With that said, I recommend weak acceptance with the hope that this work inspires more research in this area and that the shortcomings will be remedied in subsequent works that build upon the techniques introduced in this paper. I believe that this work has merit in pushing the community in the direction of trying to achieve one of the overarching goals of pruning: an efficient way to simultaneously train and search for an optimized network architecture for a particular application.


Strengths

- The paper is highly relevant to the ML and optimization communities; the premise that, e.g., filters that would have been pruned anyway after training, should not be trained to save computation time (and improve pruning performance) is very intuitive and appealing.

- It is commendable that the authors tackle the very difficult problem of pruning during training and try to model the interdependencies/future uncertainty in a principled way (using MOGP). To my knowledge, there is no other work that attempts to tackle this problem as rigorously this paper does.

- The method is overall motivated by principled insights and there is some analysis to justify parts of the method (Lemma 1)

- The authors perform evaluations on appropriate benchmarks (ResNet50 trained on ImageNet) and achieve superior pruning results (in terms of test accuracy after training/pruning) relative to those of recently-proposed, state-of-the-art approaches (SNIP and GRASP)


Weaknesses

- The proposed algorithm is not parameter-free (unlike SNIP, which is virtually parameter-free), is quite complicated (and I imagine difficult to implement), and there is little justification for certain components of the method, e.g., the dynamic scaling function (and choices of lambda), whether the simplification of m_{t-1} = … = m_{T} is mild enough. It is not clear to me how a practitioner can run the proposed algorithm in a parameter-free way without having to conduct ablation studies of their own first, especially since, as the authors note, “We observed that the penalty parameter was difficult to tune properly, either being too aggressive at pruning, or too passive” as the justification for the dynamic scaling function

- Parts of the paper are too dense and notation-heavy, and this hurts readability and understanding significantly, e.g., Lemma 1, paragraph regarding the introduction of the saliency function on pg. 2.

- The presented experimental results are not very compelling. For example, in Table 3, we see that BEP 1e-4 achieves a ~.4% improvement over SNIP and GRASP, at the cost of ~7-8.4 more hours of training time. This calls into question the effectiveness of the proposed approach -- which is, at the end of the day, meant to *speed up* training + pruning by removing unnecessary components of the network early on.


Clarity 

- The paper is reasonably well-written and organized overall. It was clear that the authors compressed some of the mathematical expressions/lemmas (e.g., statement of Lemma 1), which is somewhat understandable given the page limit, but this hurt readability and understandability.



review_title: A push in the right direction

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes a Bayesian-based approach to early prune parameters, which are predicted to have low saliency/importance, with the goal of accelerating the training of deep neural networks. The predictor is a "multi-output Gaussian process" which is computation expensive.

The writing quality and clarity of this paper is OK, but I recommend the authors to include a table of mathematical notations.

The idea of using a Gaussian process based predictor to predict the importance of parameters during the training process is interesting. However, this paper only compares with some methods (e.g. SNIP and GraSP) which prune parameters before training, inspired by the lottery ticket hypothesis paper. Lots of pruning methods can prune parameters during training before the lottery ticket hypothesis paper appears. For example, PruneTrain https://arxiv.org/abs/1901.09290 can reduce end-to-end training time of ResNet-50 by 39% without losing accuracy by simply using a previous sparsity regularization method (while this paper has a significant accuracy drop to 72.5% for ResNet-50). The paper should compare with those more superior methods, regardless of the fact that in Table 3 it is unclear if BEP can outperform SNIP/GraSP or not under the same "Time".

Moreover, the method introduces new hyperparameters. To tune the hyperparameters, the method should be run multiple times. I cannot see how it will make training a neural network faster, unless the hyperparameters are super robust to generate, which is not deeply discussed.
review_title: A complex approach without significant improvement

Meta-review: This paper considers the problem of pruning deep neural networks (DNNs) during training. The key idea is to include DNN elements only if they improve the predictive mean of the saliency (efficiency of the DNN elements in terms of minimizing the loss function). The objective of early pruning is to preserve the sub-network that can maximize saliency. This optimization problem is NP-hard, and even approximation is very expensive. The paper proves that one can simplify the approximation by ranking the network element by predictive mean of the saliency function.

The proposed approach is novel as most of the prior work on pruning has focused on either (i) pruning on network initialization  or (ii) pruning after the network has been fully trained. 

Couple of issues with the paper are:
1. Current approach is somewhat complicated with many hyper-parameters
2. Experimental results are not very compelling when compared to pruning on network initialization

Overall, my assessment is that the paper takes a new research direction and has the potential to inspire the community, and followup work may be able to overcome the above two issues in future. However, due to the remaining shortcomings, the paper is not judged ready for publication in its present form. I strongly encourage to resubmit the paper after addressing the above two concerns.
Decision: Reject