Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors address the problem of network pruning under a fixed training budget.  Earlier work on network pruning relied on computationally expensive iterative training & pruning regimes.  The authors show that when operating under a fixed training regime, a lot of savings can be found by eschewing complex learning rate schedules for a simple but aggressive linear learning rate schedule.  The authors further propose improvements to the initial choice of learning rate in the schedule, as well as adding more structure to the budget of the dense network training, which together result in real value per unit of computation.  They demonstrate good performance against alternative methods which induce sparsity during training, and challenge the community to reconsider the commonly held tenet that retraining is wasteful. 
strength_and_weaknesses: ### Strengths
- The authors demonstrate a solid understanding of the literature on pruning
- The figures and tables are quite clear, establishing the benefit of adaptive LLR in certain settings.
- The experiments, within their original parameters, are extensive enough to demonstrate the advantage of LLR and ALLR over the competing methods.

### Weaknesses
- There are a number of claims in the paper that do not agree with the published literature.  For instance, in the first paragraph of the introduction, they claim that a heavily pruned model will normally be less performant than its dense (or moderately pruned) counterpart.  But this isn't always true.  For example, Lottery-Ticket hypothesis [works](https://proceedings.mlr.press/v119/frankle20a.html) show that pruning even up to 80 to 90% of the weights of an original network.  I think the authors should either be more careful in their claims, or more precise in their writing.
- At the end of the introduction, the authors state their results build upon work by Renda et al, Le & HUa by proposing  how to choose the initial value of the learning rate, a problem not previously addressed.  Key works missing here are: *in a pruning context*.  This is a fine point to make, but a necessary one to avoid claiming that the problem of choosing an initial value of the learning rate for retraining a network has not previously been addressed.  For instance, every simple transfer learning problem has to solve this issue, by fine-tuning new classification weights (or more output-proximal representation layer weights) to solve a new task, which i do not see as fundamentally different than retraining following pruning.  I would walk back this claim. I also note that the reviewers of Le and Hua found the same criticism. 
- In section 2.1, the authors declare themselves for Li et al in the battle of ideas about re-training a learned network.  That may be so, but what I have yet to see here, (and to be fair in the short readings of any of the competitions like Renda et al., or Le & Hua), is the question of how this problem differs form the fine-tuning problem of any base model (or foundation model).  

I think the authors would help separate their work from these other lines by clearly delineating what aspects of the retraining problem they think are the most salient, and then arguing why the budgeted approach yields the best results.

- The connection of the learning rate scheduler to generalization (in Li et al as well) is empirical, and not well established.  Appealing to this as justification is wishful thinking. Over all, section 2.1 feels poorly structured.  Where does the related work summary end, and the authors’ own work begin?  It could be made more clear by the use of sub-sectioning to delineate the concepts of pruning, learning rate schedules, and generalization.

- A minor quibble about equation (1): you cannot have the interval of $s$ include $0$, as that leaves $d_1$ undefined by virtue of division by zero.

- It’s also unclear how this is supposed to stand as a proxy for the degradation in on-task training accuracy, which the authors claim in the preamble to equation (1). There might be a more grounded argument to be made here (perhaps approximating the function of $W$ -> accuracy for a fixed evaluation set?

- The final paragraph of section 2 cursorily sketches out ideas in selecting weights to be pruned, but only considers magnitude based pruning.  There are competing approaches that are sufficiently different, and deserve to be mentioned in here, such as  [Tanaka et al.](https://proceedings.neurips.cc/paper/2020/file/46a4378f835dc8040c8057beb6a2da52-Paper.pdf) who propose a flow-based method for pruning weights.

- Finally, I am sure I’m not the first to raise this issue, but there is surely an in-built bias resulting from the authors’ choice to study only multi-class classification in image data sets, using one family of convolutional networks.  It would be more convincing if the authors could have included some diversity in either their model families chosen, or their tasks considered.  The authors acknowledge this failing in their work  as a footnote in the concluding discussion, but if they are to follow in the steps of  Li et al 2020 (who use tasks from four different domains), then they should present more evidence that their method has an empirical benefit beyond that of this narrow domain.
clarity,_quality,_novelty_and_reproducibility: ### Clarity
There are a few areas in the paper that suffer from a lack of clarity:
- Point 2 of the major takeaways is awkwardly phrased.  If there is more evidence supporting a linear LR schedule, why not come out and say so?  Putting it in parentheses weakens the idea.
- The first sentence after stating the major takeaways states the authors' claim that retraining should be considered in a budgeted scenario, but at this stage, It’s not clear why.  Do the authors believe this because budgeted results give the best improvement per parameter update?  Do they have better generalization performance?  They should call out 

This section has laid out the cases for not using a truncated learning rate schedule, and not using a fixed learning rate schedule, but it has not argued successfully for why the budgeted learning rate schedule is more advantageous.

- Section 2.2 lays out two commonly stated advantages of pruning-stable methods, and goes on to argue why IMP should prevail.  I think the authors can make the case for Budgeted IMP more clearly.  They cast doubt on the potential of pruning stable algorithms, and propose that instead of aiming at a complicated method to prune in one shot, instead the cost of IMP should be reduced by training on a budget.  This way, the main drawback of IMP can be removed, and the network can achieve good performance though a more careful choice of an initially aggressive learning rate coupled with the right choice of initial LR in each subsequent phase of  post-pruning recovery training.

Figure 2 repeats much of the story of Table 1, which is that the task only gets hard enough to differentiate the methods at 80% sparsity and up.  It is somewhat interesting that there is a noticeable difference in behaviour between the top performers (ALLR,  LLR, CLR) between the 80% sparsity task and the 90% sparsity task.  In the former, ALLR is universally dominant, while in the latter, it is indistinguishable from LLR and CLR until at least 20 epochs of retraining are budgeted.  Is this to be expected?  And what does this say about  how the advantage of adjusting the initial learning rate carefully (as ALLR enjoys over LLR) interacts with retraining budget to and task difficulty? 

### Novelty 

- The main novelty here is in extending the definition of what constitutes a training interval under Budgeted training, and of adaptively choosing a new learning rate after each phrase of retraining pursuant to magnitude based pruning.  It's clear that the development of ALLR is an improvement on LLR, but I'm not sure that the addition of a heuristic-based adaptive learning rate scheme is truly novel, given the extensive literature on adaptive learning rates for neural networks.



summary_of_the_review: While the authors convinced me that the addition of budgeting can save IMP, I am less than convinced of the generality of their results to other settings than well-studied vision data sets.  I think that the authors should either extend their set of empirical results to encompass different problems (and different networks), or to reach into theory of generalization papers to try and support their claims for adaptive learning rates and budgeted retraining.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies the retraining steps in iterative magnitude pruning from the "budgeted training" perspective. This is used to derive an efficient learning rate schedule for the retraining phase.
strength_and_weaknesses: ### Strengths

- The paper does a very good job of explaining existing work and positioning itself in that context. This makes the work accesible to readers not familiar with minute details of pruning methods.

- While one could see the proposed technique as only a slight variation of some existing methods, the simple linear step size schedule is well-motivated from the "budgeted training" perspective. It is simpler than the schedules used in prior work while performing at least as good, and often better, across a wide range of "retraining budgets" and target sparsities.

- The experimental comparison is conducted with great care. The experimental set-up is motivated and explained in detail, including hyperparameter settings and/or tuning protocols. The results are presented in very clearly and their significance is discussed in detail.

- I want to explicitly commend the authors for making clear the limitations of the proposed method and to focus on generating useful insights rather than trying to coin another acronym.

### Weaknesses

- I find the motivation for **A**LLR quite unsatisfying. The relative distance computed in Eq. (1) strikes me as quite a poor proxy for what we are really interested in, which is to quote the paper "how much of an increase in loss do we have to compensate for". At the same time, this increase in loss could actually be computed at the expense of an additional validation epoch. In contrast to what's stated in the paper, I don't think that would be a big computational burden. The evaluation could be done on the validation set (or possibly even a subset thereof) and requires only a forward pass. Its cost should therefore be minor compared to the actual retraining for multiple passes over the training set.

- The experiments are restricted to image classification tasks. For breadth, it would have been desirable to include another data modality and/or task.
clarity,_quality,_novelty_and_reproducibility: Quality: The experiments are very well thought out and conducted with great care. I do miss some variety in the selected models/datasets.

Clarity: In my opinion, this paper is very well-written. It does a great job of positioning itself in the context of related work. Most steps are very well motivated.

Originality: There is little technical/methodological novelty in the proposed approach. However, the budgeted training perspective and the careful experimental comparison gives very useful insights.
summary_of_the_review: I think this is a very well-written paper with a solid contribution to the model pruning literature. I was slightly disappointed by the motivation for **A**LLR and would encourage the authors to respond to this during the rebuttal.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose that recent work on budgeted training in dense models applies in the context of both the dense training and sparse re-training phases of Iterative Magnitude Pruning (IMP) - perhaps the most common form of pruning. In particular the authors propose two methods of pruning/re-training consisting of relatively short total training duration compared to existing work, and yet achieving strong results compared to those much longer-trained baselines.
strength_and_weaknesses: # Strengths
* Well-written and well-organized paper, with a clear motivation, excellent background/references
* Method is simple and explained well overall
* Figures and results are presented well
* Empirical evaluation is (mostly) on reasonable datasets/models (exception being VGG, see below)
* Questions existing wisdom in the sparse neural network community that "pruning stable" approaches (i.e. methods without hard/fixed masks during training) to the finding sparse neural networks are preferable to "pruning unstable" approaches (i.e. methods with hard masks).
* Strong results mean this could be a new baseline for "pruning unstable" methods

# Weaknesses 
* The authors significantly weaken the impact of their work by essentially over-claiming that their work shows that "pruning stable" methods are pointless, without the results to back it up. In a notable and questionable omission in the results, the authors do not compare to the state-of-the-art dynamic sparse training methods despite citing them, for example RiGL. On ImageNet/ResNet-50 the BIMP results are massively below those of RiGL. For example, in Table 2, at 90% sparsity, BIMP has a Top-1 accuracy of 70.25%, while RiGL achieves 72-76.4% (saying based on ERK/non-ERK and length of training). 
* The authors point out rightly that RiGL and other such methods are massively more computationally complex and require longer training - and if they instead focused on this argument in their results, rather than generalization alone while omitting RiGL, this would be a much stronger paper.
* Intuition/reasoning behind the ALLR method of using the normalized distance between weight vectors to decide an initial learning rate is not made clear, and this reads as quite arbitrary/heuristic as-is.
* VGG is much too over-parameterized to be in a sparsity-related paper in 2022.
clarity,_quality,_novelty_and_reproducibility: This paper is clearly up to standards in paper quality, with it being very well written and organized. As far as I'm aware applying such a training budget and achieving such good results is novel, and of interest to the broader research community. Although I worry the originality of the work will be debated due to its simplicity, the simplicity of the method and strength of the results make this a stronger paper.
summary_of_the_review: Overall the paper is well-written, with a good background, experimental setup and analysis. The paper sets strong new baselines for "pruning unstable" methods, such as Iterative Magnitude Pruning. Although these results are used by the authors to compare with "pruning stable" methods, such as Dynamic Sparse Training methods (in general), in not comparing to the widely known DST methods with the best generalization results (in particular RiGL on ImageNet/ResNet-50) weakens this argument significantly, and avoiding comparing to such a well-known and strong baseline in general when its cited by the authors raises questions. The authors do make very good points on the relative computational complexity of such methods however which even in the face of RiGL results still leaves this work motivating, and of interest to the sparse neural network research community.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a simple and efficient method for improving iterative prunning. The method focuses on tuning the learning rate schedule during the retraining phase, which has been shown to significantly impact the performance of pruned networks. The experiments show improvement upon some much more complex methods. 

The simplicity of the method and its impact on the robustness of the field are the main strong sides. However, some reviewers noted a lack of clarity in the writing and poor motivation behind the learning rate scheduling.

After discussion, all reviewers were supportive of accepting the paper. All in all, it is my pleasure to recommend acceptance. Thank you for your submission. Please make sure to address the authors’ feedback in the camera-ready version.
justification_for_why_not_higher_score: This is a solid contribution but cannot say currently it is high impact, but at the same time hard to rule out.
Decision: Accept: poster