Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: # After Rebuttal: Score Lowered from 7 to 6

## Concerns Addressed

I appreciate the effort the reviewers put into revising the paper to include the settings I suggested.

I am generally pleased with the revisions the authors made to the paper (especially Section 3), and I appreciate their attention to these details.

## Remaining Concern: Settings in the Main Body are Poorly-Tuned and May Overstate Results

I am concerned by one aspect of Figure 2: the unpruned accuracies for VGG-16 and ResNet-20 are much lower than they should be. VGG-16 should get 94% accuracy on CIFAR-10 (vs. 91% in the plot), and ResNet-20 should get 92% accuracy (vs. 86% accuracy in the plot).  This is because the paper uses Adam to train all networks without any learning rate drops, whereas the typical learning rate schedules for VGG-16 and ResNet-20 use SGD with momentum with learning rate drops.

This important difference raises the concern of whether the results shown in the paper will translate into fully-tuned, large-scale settings. As evidence of this concern, Appendix G does show a fully-tuned VGG-16 getting standard accuracy. In this setting, LAMP is no better than global magnitude pruning until very extreme sparsities.

**I have lowered my score on the basis that the results in Figure 2 may overstate the value of LAMP in well-tuned settings. I no longer have unequivocal confidence that LAMP is an improvement that should be adopted in general. I implore the authors to replace the experiments in Figure 2 with well-tuned versions of these networks that achieve SOTA accuracies.**

## Overall: Score Lowered from 7 to 6

I am less confident in the method's significance in well-tuned settings, and I can no longer unequivocally trust the empirical evaluation in the paper. I still support acceptance, but only tentatively.

# Overall

I think the technical results are excellent. The authors should be commended for making a productive contribution to the pruning literature. My biggest concerns are :
(1) The mathematical derivations are unreadable and impossible for future researchers to build on, so the authors *must* expand this section and make it crystal clear if I am to continue to recommend acceptance (they should use the extra page for this)
(2) The authors should include a network that isn't severely overparameterized for CIFAR-10. I recommend ResNet-20 or ResNet-56.
(3) The authors need to verify that their statements about prior papers are correct. I have noted a few things that need to be addressed below. In many cases, prior work was unclear about how it handled global/layerwise pruning decisions, and the authors should mention when there is ambiguity.

In addition, other smaller (but important) changes I would like to see include:
(4) The authors should ideally include results on all of ImageNet, since subsets of ImageNet tend to show very different results from the full task. However, I understand that this is very expensive and that not everyone has the resources to do so, and I understand if the authors are not in a position to accomplish this.
(5) The authors should use weight-rewinding as described below in the manner of "Linear Mode Connectivity and the Lottery Ticket Hypothesis" (Frankle et al.). Conv-6 is not a setting whose results correspond to larger-scale settings.
(6) I think the authors should revise the title to clarify that they are proposing a better way to select the layerwise rates for magnitude pruning. This paper makes a valuable contribution, and that should come through from the title so readers can determine this easily.

# Score

I have issued the paper a 7 on the assumption that the authors will make substantial revisions to Section 3 and take advantage of the extra page of space to do so. If they do not address my major concerns (and ideally my minor concerns), I will lower my score.

# Questions

Why is "minimizing the l2 distortion for the worst-case input signal" a reasonable design choice to make? The authors assert that it is what they aim to do, but they never explain why this is a good idea. I see why this makes sense at a high level, but adding a couple of sentences to this effect would be valuable, and I would like to hear from the authors directly.

"Minimizing l2 distortion for the worst-case input signal is equivalent to minimizing the spectral norm distortion." Please justify this. It's not immediate. I assume that the norm on the right side of (4) is the spectral norm, but please clarify.

"The optimization (4) can be relaxed to a Frobenius distortion minimization." Please justify this. It is not immediate. In general, this math is dense and the paper goes through it too quickly, and I wasn't able to follow it.

Where does the sparsity constraint k come from? 

If I understand "MP: layerwise distortion minimization" correctly (which I don't think I do), it's meant to say that pruning the lowest-magnitude weights is a way of minimizing the l2 distortion for the worst-case input signal for a single layer in isolation? If this is the statement, I'm concerned that there's a <= in the derivation, since this suggests these two are not necessarily equivalent. And I don't understand why k is in there if we're pruning to a specific target sparsity. In general, I'm quite confused by (1) what you're trying to show here and (2) the actual details of this derivation.

The section on the LAMP score is completely uninformative. You need to include more of the derivation in the main body of the paper, especially explaining why it's appropriate to use the relaxation from the generalization theory literature (that again includes a <= suggesting that the two sides are not equivalent), what these quantities mean, what a "damage score" is, and how this leads to the LAMP scores. Right now, this paper suffers from - at best - obfuscation via math; worse, I highly doubt it would be possible for a reader (or someone working on follow-up work) to understand or reproduce your derivations/rationale. This is a crucial flaw in the paper.

The models you use are quite overparameterized in general. I would prefer to see a model like ResNet-20 or ResNet-56 on CIFAR-10, which is much less overparameterized for the specific task. 

In general, the results look very impressive.

I am concerned that these graphs zoom in on the lowest sparsities. Importantly, it is difficult to tell what the unpruned accuracy looks like for these models.

For weight-rewinding, do not use Conv-6. It is a toy network with little correspondence to real behavior. Use a larger-scale network like VGG-1`6 or ResNet-20 and rewind to iteration 1000 as Frankle et al. do in "Linear Mode Connectivity and the Lottery Ticket Hypothesis."

# Things to Address

Verify that the details of the way that prior work has pruned networks is correct.

Ensure the purpose and derivations in Section 3.1 are explained in a way that is crystal clear to a reader. I'm a mathematically well-informed reader who is an expert on pruning, and I can't follow the derivations or what you're trying to say with those derivations. This is due to unclear writing; I'm not a poorly-informed reader.

# Other Notes

"The iterative pruning scheme of Han et al. uses global magnitude pruning." I believe this is incorrect. Han et al say that "the pruning threshold is chosen as a quality parameter multiplied by the standard deviation of a layer's weights." 

"Frankle & Carbin (2019)...employ uniform layerwise sparsity." I believe this is incorrect. Frankle & Carbin use uniform sparsity for MNIST but use global pruning for larger-scale networks (ResNet-18 and VGG-19).
review_title: Great Results; Section 3 Needs to be Rewritten

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper presents a novel technique  (layer-adaptive magnitude based pruning, or LAMP) for pruning neural network weights (pruning can be beneficial in terms of overfitting prevention as well as other practical considerations).  LAMP evaluates weights in each layer in terms of the ratio of the magnitude of the weight to the sum of magnitudes of all surviving weights in the layer. The weight which evaluates as least important across all layers is pruned and then the process is repeated until the desired sparsity is achieved. The method is motivated theoretically as minimizing the distortion in the input/output mapping implemented by the weights of the layer. Experimental results on several benchmarks are presented.

Pros:

The experimental results are strong,  with LAMP consistently winning vs. competing techniques on 4 benchmarks problems. 

The method is elegant, requiring no hyperparameter tuning and minimal computation. 

The theoretical justification (mapping-distortion-minimization) makes a lot of sense. 

Cons:

My only objection is that all of the experiments seem to be done on image datasets. It seems possible that deep learning networks applied to non-image data might not do as well under LAMP as they do for images. 'Under diverse datasets' in the abstract seems like an exaggeration. 

Further comments:

I found a couple of typos.

P3 while such unstrutured pruning -> while such unstructured pruning
P5 Global ‘on every layers’ -> on every layer


review_title: Strong experimental results and an elegant technique- more experimental diversity would be beneficial

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary
- The authors propose LAMP, a layerwise adaptive magnitude-based pruning method.  The authors conduct extensive experiments on CIFAR10/CIFAR100/SVHN and Penn Treebank to validate the method.

Pros
- somewhat novel pruning method, based on new weight score
- extensive experiments on image and language datasets

Cons
- In Equation (2), the authors point out that LAMP score is align with the order of weight squares. Therefore, one can directly prune the network based on weight squares. Why is it necessary to prune the network based on LAMP?
- The comparisons are not sufficient. The authors should compare other "pruning-retraining" methods, like network slimming [1], soft filter pruning [2], etc. Though they focus on structured pruning, the core idea can be borrowed and adapted for unstructured pruning.
- Lacking of experiments on large-scale datasets and large models, for example, on ImageNet. The performance of pruning methods  can be very sensitive and versatile when only evaluating on small datasets and models. And usually, a pruning method can be invalid when testing on ImageNet models.
review_title: Reviews

final_score: rating: 7: Good paper, accept
review_content: # Summary

The paper proposes LAMP, an importance score for unstructured pruning that incorporates layerwise statistics such that the resultant scores for each connection can be compared globally, cutting down on the hyperparameter space for magnitude pruning from the relatively standard practice of requiring hand-specified layerwise pruning rates. LAMP is motivated with a distortion analysis: LAMP is shown to be equivalent to minimizing an upper bound on the supremum of the change in model predictions of unit vectors. LAMP is compared against layerwise pruning rates obtained by standard uniform layerwise and global pruning, along with the less standard Erdos-Renyi kernel method, showing that LAMP can achieve higher accuracy for equivalent pruning rates in a specific experimental setup across several different networks.

# Strengths

- The problem the paper addresses is well-motivated: magnitude pruning rests on a poorly motivated and poorly understood set of heuristics.
- The approach proposed by the paper has several desirable properties: it is hyperparameter-free, relatively cheap to compute, and handles some corner cases of global magnitude pruning (e.g., LAMP is invariant to rescaling of a layer in a net with BatchNorm).
- Within the scope of the empirical evaluation in the paper, LAMP is demonstrated to outperform uniform and global magnitude pruning, the two most common approaches in the literature.

# Weaknesses

- Theoretical justification of LAMP: it's unclear how useful or strong the bound in Equation 7 is, especially given that the network is re-trained after pruning. These types of bounds often seem to be weak or vacuous in practice; further, there's no explicit connection between the bound and the loss, let alone the loss after re-training.
- Empirical evaluation: the training and re-training regimes used by the paper are very non-standard, raising questions about the generality of the technique and the comparison to prior work. Specifically, the choice of AdamW with a fixed learning rate and the seemingly arbitrarily chosen durations of re-training are out of line with standard practice for many of these networks; these choices mean that it is impossible to compare the accuracy/sparsity tradeoffs achieved by LAMP to the results of various other papers that claim state-of-the-art pruning results with local/global magnitude pruning.

# Overall recommendation

6: Weak accept

# Questions / suggestions for authors

I would be willing to raise my score to a 7 with a more thorough comparison against prior work with more standard hyperparameters. Specifically, if the authors reported the results a comparison of using LAMP with the same networks and training/re-training schedule as any of [1,2,3,4], or any other specific setting that claims to be state-of-the-art, I would raise my score.

# Other comments and suggestions

- The notation of Equation 1 is a little bit confusing: I'm assuming that $W[i]$ is the $i$th element in some unrolled version of $W$, and that $W^2$ denotes element-wise squaring, but it'd be good to clarify this non-standard notation.
- It would be good to have some more details on the networks, especially since many are being used in non-standard settings (e.g., VGG and ResNet-18 on CIFAR). Specifically: how are these networks adapted to these datasets, and what are the base accuracies for each networks on each dataset?
- A deeper comparison to the layerwise sparsities (and overall pereformance) of [2] would be appreciated. For example: Section 5 of this paper notes that "LAMP tends to keep the number of nonzero weights relatively uniform throughout the layers at extreme sparsity level." Section 4.2 of [2] notes "The peaks and crests show that the RL agent automatically learns to prune $3 \times 3$ convolutional layers with larger sparsity." Is there a reason for this discrepancy between the findings of each paper?
- More details on the experimental methodologies of Figures 4(b,c), along with motivations for these experiments, would be appreciated. I am particularly confused by 4(c) -- I do not see standard SNIP scores on this plot, unless "Global" here refers to SNIP scores and not global magnitude pruning (in which case, how does standard global magnitude pruning perform here?). It would be worthwhile to include full methodological details for each of these experiments, including the training and re-training (where applicable) schemes, the specific point in training at which the importance scores are calculated, and the specific importance score used by each line on the plot.
- Minor point: I don't believe that [1] used global magnitude pruning; I believe a better citation would be [5] (though there are also other prior/contemporary examples of global heuristics, like [6] and [7])

# References used in review:

[1] Song Han, Jeff Pool, John Tran, William Dally. "Learning both weights and connections for efficient neural networks".
[2] Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, Song Han. "AMC: AutoML for Model Compression and Acceleration on Mobile Devices"
[3] Trevor Gale, Erich Elsen, Sara Hooker. "The State of Sparsity in Deep Neural Networks"
[4] Alex Renda, Jonathan Frankle, Michael Carbin. "Comparing Rewinding and Fine-tuning in Neural Network Pruning"
[5] Jonathan Frankle, Michael Carbin. "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks"
[6] Yann Le Cun, John S. Denker, Sara A. Solla. "Optimal Brain Damage"
[7] Namhoon Lee, Thalaiyasingam Ajanthan, Philip H. S. Torr. "SNIP: Single-shot Network Pruning based on Connection Sensitivity"


# Update post author response

Thanks to the authors for the detailed response. The authors have satisfactorily responded to my main criticisms of the paper (primarily about the non-standard evaluation regime, and secondarily about the motivation and strength of the theoretical results), so I’m raising my score to a 7, though I do think the paper would be further improved with references to Appendix G in the main body of the paper.

review_title: Well-motivated paper proposing hyperparameter-free layerwise pruning rate heuristic, though theoretical justification and evaluation could both be strengthened.

Meta-review: The paper proposes a layer-wise magnitude-based tuning method through the adoption of LAMP score, motivated by minimizing the model output distortion. The new importance score differs from vanilla magnitude-based score in that it incorporate more layer-wise information. Extensive experiments are conducted on image and language models to show the improved accuracy upon prior arts under same model compression ratio. Ablation study is also provided to further explain the intuition and comparison of LAMP with other pruning methods. 

Though the experiments are extensive, some reviewers raised questions that only image datasets are tested. In the rebuttal, the authors addressed more on Appendix D which provides non-image results, and also modified the abstract to highlight the efficacy on image data. In all, given the extensive empirical evaluation on various datasets and model architectures, the improvement of LAMB over prior methods seems convincing. Nevertheless, we urge the authors to include more experimental results, for example ResNet-18 on ImageNet as promised to Reviewer 1, to make the results more solid. It is also suggested to include and discuss some relevant papers mentioned by the reviewers in the final version.  
Decision: Accept (Poster)