Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
- This paper conducted a detailed study on how does the loss modeling affects the final performance of the pruned model. The authors first provided a unified view of various pruning algorithms (e.g., Magnitude Pruning, SNIP, OBD, and OBS), which can be categorized into three classes: weight magnitude, linear and quadratic models of the loss function. In the experiments, the authors seek to answer the questions: 1) how well do each criterion preserve the loss; 2) how does the locality assumption affect the final performance; and 3) how does the loss relate to the final performance? Empirical, the authors found that the quadratic model preserves the loss the best, as expected. Also, the loss after pruning seems not strongly correlated with the performance after fine-tuning.

Overall:

This paper is well-written and easy to follow. The authors did a great job of unifying the analysis of several pruning algorithms. More importantly, revisiting the loss modeling of network pruning is interesting, and it might invoke further research efforts in better understanding the pruning techniques developed in the past and also inspire researchers in designing improved pruning algorithms. However, I still have the following questions:

- The authors show that the loss after pruning does not correlate strongly with the accuracy after fine-tuning. In Figure 3, the change in the loss ranges from 0 to 5. Can you show the plot with a smaller range of the change in loss, e.g., 0~0.5? I believe a large change in loss means that the pruning results are very close to random, so the comparisons in this regime may not be meaningful.

- For testing the locality assumption, you introduce an L2 penalty on the changes. To me, this is more like a weighted combination of the original pruning criteria and the magnitude pruning criteria. Why not using some other techniques, such as backtracking line search for determining the pruning ratio at each iteration? 

- In equation (5), why do we need to take the absolute value? I think preserving the loss is only meaningful when the network is converged. If the model is not converged, then it would be preferable to prune those weights whose removal will decrease the loss. In this sense, the sign of the loss change should not be ignored. 

- The third plot in the second row of Figure 1 shows that OBD with more iterations has a larger change in the loss. Do you have any explanation for this?

Rating:
- I vote for a weak acceptance due to the above reasons. I believe the studied topic in this paper is important and impactful for the pruning community. In the meantime, it would be great if the author can propose some hypotheses on this phenomenon.  To me, preserving the loss is a way to enforce the pruned network to stay close to the original solution, and in this regime, it's easy for the optimization algorithm to find a good enough solution. I will raise my rating if the authors can address my concerns well during the rebuttal period.

==========================After rebuttal=================================

Thanks very much for your efforts to address my concerns. I kept my score unchanged. I agree with most of the responses, except for the response to "L2 penalty, backtracking line search for determining the pruning ratio". This paper is not proposing a practical algorithm but a revisiting, so I don't think the computational cost is a bottleneck in preventing you from using more advanced methods to get more robust conclusions.
review_title: Well-written paper with interesting results.

final_score: rating: 3: Clear rejection
review_content: Although the paper is covering an interesting topic, much of what's in the paper can be found in other works, and there's not a lot of novelty to the insights, nor a large breadth of experiments to justify it as a survey paper.
- The linear and quadratic loss functions are not new.
- The enforcing locality part is essentially why the pruning strength annealing schemes exist, this insight is not new and can be found in Zhu&Gupta, or in Bayesian settings like in the Molchanov paper, they suggest annealing from a Bayesian statistical perspective
- The fact that this leads to multiple stages of pruning to be a good idea, is also known in the literature this paper cites.
- The most meat of the paper is in the 'survey' part of it, investigating the results... but this section feels lacking since there are not a lot of experiments, and the insights of e.g. post-finetuning can be largely found in e.g. Blalock et al. I'm missing deeper insights/analysis here. What is the reason for this? How do we remedy this? What are the characteristics of networks that lead to this behavior? 
- It would have been great to see a lot more insight/experiments on this topic. The authors throw up a lot of hypothesis and suggestions/ideas throughout the paper, but don't back them up. E.g. If the ||Theta|| term is better of to be constant throughout the pruning procedure... can we somehow make an annealing scheme that keeps ||Theta|| small and constant throughout the pruning process, and show that that works well? This can be proven/shown somehow. 

I do think the paper is well written; and I encourage the authors to look further into this topic and come up with more novel insights/results and methods to improve pruning

Other things/questions/suggestions:
- In formulations (1), (2) and (5), (6). Why are the absolute brackets necessary? Especially for models that have not converged, why would you want to stay close to the original loss, as opposed to just decreasing the overall loss of the model? 
- For most of the discussion in section 3.2, the authors talk about the norm of the delta theta squared being large. But this largeness is relative to the 0th and first order term which the authors glance over. Under 'other considerations' for example, if the weights theta are large, the gradients likely follow suit. Thus the absolute magnitude of the weights might not matter, as it's the relative size of this to the gradient terms that should be considered. 
- Constraining the step size in section 3.2. Interestingly, if you take the assumptions that for each layer, the hessian of the loss w.r.t. the output is sqrt(lambda/2), and your input distributions to a layer are normalized, you end up with the weaker norm penalty. This is a crude approximation of the second order term, which would give this method a bit more of a theoretical foundation than just a regularization term.
- 5.1 Convergence assumption. I don't get this part, both the OBD method, and the linear and quadratic loss terms depend on the locality, so all will also depend on the amount of steps taken. For OBD, as long as you recalculate the Gauss-Newton matrix, I don't see why this method is different when not doing fine-tuning. 
- 5.1 Convergence assumption. The result cited in appendix A is a very well-know result. How could this link explain the OBD performance on the VGG network? 'Could' is not strong enough to make it into a paper. 

Small tidbits:
3. Do pruning criteria better at preserving the loss lead to better fine-tuned networks? <- this sentence doesn't flow nicely. I would add a 'that' so you have do pruning criteria that are better ...
review_title: A paper that combines many different things from other works, and runs a few extra experiments, but in the end doesn't feel like it has enough body and depth for a full-conference paper.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper develops two modified version of Optimal Brain Damage (OBD) criteria, namely LM and QM (linear and quadratic model) to measure the importance/saliency of model weights. It then compares these three together with Magnitude Pruning (MP), and show that these among four criteria: 1. for the first three, using iterative pruning to enforce locality of the gradient calculation is important, 2. the best method for training loss before fine-tuning does not necessarily lead to best validation accuracy after fine-tuning. 

The paper's empirical investigation is valuable and appreciated. It helps me understand OBD more throughly and the assumptions behind it. It is also useful to know that using iterative pruning can improve these gradient approximation-based methods because of locality.

1. My primary concern is the experiments does not seem to lead to a useful guidance for future practice. The paper does conclude for the first three using iterative pruning is useful, but these three criteria are rarely used nowadays and MP is the mainstream, and the simplest method. The paper also didn't conclude which of the four criteria is in general best and recommended. From table 2 it seems to be LM but the paper did not conclude this way. This is also possibly due to that the experiments are not run extensively on different datasets and architectures. 

2. The paper compared the training loss before fine-tuning, and validation acc. after fine-tuning. But I think validation acc. before fine-tuning is also a quantity worth investigating. 

3. More importantly, the paper shows the training loss (before fine-tuning) and valid acc. (after fine-tuning) are not necessarily correlated, but did not give explanation on why this could be the case through experiments, or give useful suggestions to achieve a good valid acc. after fine-tuning.

Overall I appreciate the empirical study but I suggest conducting the experiments on more datasets and architectures, and extract a useful conclusion to guide future practices.

+++++++++++++++++

I appreciate the clarified messages of the paper, and would like to see them emphasized more clearly in the next version of the paper. But due to the limited experimental scale on ImageNet (added in rebuttal, and in my understanding, it only verifies one of the multiple observations mentioned in the paper), I'm still leaning on rejection. I updated my score from 4 to 5.
review_title: Appreciate the empirical study but cannot find substantial useful conclusions

final_score: rating: 7: Good paper, accept
review_content: Summary:

The authors study the use of loss-modeling to maintain model quality when inducing unstructured sparsity in deep neural networks. They study a range of different approximations and modifications that can help improve the quality of the approximation (taking local steps, avoid large changes in weight magnitude, avoiding assumptions about convergence). The authors conduct a thorough empirical investigation that yields practical observations for the design of future pruning techniques.

Pros:

The paper is well written and well organized and the empirical investigations are well done. The observations made by the authors are interesting and practically useful for the development of future pruning techniques. Namely,
1. That including first order terms in loss approximations can relax the convergence assumption behind some existing loss modeling approaches to enable more flexible application of the pruning algorithm.
2. The quality of the local loss approximation can be improved by taking a series of smaller pruning steps.
3. That loss-preservation does not necessarily translate into accuracy preservation.

Cons:

It would be nice to see experiments in domains other than computer vision. For example, language modeling with RNNs or Transformers. Results at a wider range of sparsity levels for ImageNet would also have been useful, as it seems possible that these techniques could perform differently for high sparsity (>90%) than they do for moderate sparsity (e.g., the 70% sparsity reported in Figure 4).

Comments:

Another ICLR 2021 submission is highly relevant to your investigation: https://openreview.net/forum?id=rumv7QmLUue. Their theoretical/empirical results appear to corroborate your conclusions that loss preservation is not necessarily the best metric to optimize for when you care about accuracy preservation.
review_title: Interesting paper with solid empirical evaluation

Meta-review: This paper presents a systematic breakdown and evaluation of several assumptions and algorithmic choices for pruning algorithms. As covered in the reviews, the evaluation and its conclusion offers a timely contribution to the broader community.

In particular, this paper uncovers the observation that precisely modeling the loss (and hence minimizing the drop in loss after pruning) may not in fact yield improvements in pruning. This is an important observation as the community continues to propose new techniques with the justification that their improved performance results from improved loss modeling.

A significant concern on the part of the reviewers is the limited practical prescription offered by the paper. Specifically, the paper does not propose a new algorithm. It also doesn’t necessarily identify why this interesting phenomenon emerges.  For example, to the latter, it doesn't articulate what features of the network or loss landscape is indicative of this property.

Ultimately, the decision for this paper is very challenging given the reviews. Whether or not a phenomena is interesting is an inherently subjective consideration. Moreover, without a clear technical prescription or path forward that can be evaluated on its merits, the reviews fall into two categories of either 1) those that --- by my estimation --- felt personally inspired by the work and 2) those that could not intuit the impact of the observation.  

A significant complication is that the narrative of the paper includes claims around addressing locality and convergence which, if not read with the understanding that contributions here are simply a synthesis of current work, appear as claims to novelty (when these techniques have no or limited novelty). This is a source of contention in at least one review.

Given this partition, my recommendation is Reject.

For future versions of this paper, I recommend that the authors narrow the claimed contributions to exclusively focus on the final observation that modeling the loss may not be as important as thought. 

The work in this paper on developing the ideas around convergence and locality can, instead, be cast as efforts to provide best available baselines for the topline claim.  I believe these changes will eliminate a significant source of distraction, enabling readers (and reviewers) to avoid any attempt to evaluate the novelty of  the locality and convergence narratives, which have indeed been considered in other work in various ways.

An additional step that I highly recommend for this paper to unambiguously clear the bar is to identify with what the performance of pruning does correlate. Appendix C.4 provides an evaluation of two recent gradient preservation methods. Unfortunately, the paper did not present if, instead, the preservation of the gradient correlated with additional performance. 

In essence, the paper need not solve the mystery by providing a SoTA algorithm that exploits the right features of the problem for pruning. However, it would be valuable to provide a roadmap for future directions along with an articulation of the challenges down those directions.

Decision: Reject