Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper addresses the important research question of to prune neural networks such that the gain in inference is maximized. Authors to convert the neural network pruning problem into a resource allocation problem and solve it using Knapsack strategy. For estimation latency for different parts of the network, authors employ a lookup table. Experiments are conducted on multiple hardwares and authors demonstrate improved performance with the proposed method.
strengths_and_weaknesses: Strengths: 
The approach of optimizing the performance of neural networks w.r.t latency budget is interesting and a very relevant research problem from a practical point of view. The strategy presented by the authors permits designing pruned networks for different choices of latency budget which makes this approach quite useful. Further, the lookup table approach of the authors makes it very easy to include the d latency parameter into the optimization problem. Authors demonstrate the efficacy of the method on various choices of hardwares and present an extensive ablation study around the proposed method. Overall, the research content seems original, well written and the message is clear.


Weaknesses: The presented paper in its current format has certain weaknesses and I would like to request the authors to respond with their remarks to the weaknesses listed below.

1.	A major concern for me is the use of lookup table to construct the latency of the overall network. In my opinion, latency cannot be computed as a sum of latency from different parts, rather it should also take into account the parallelizability across the cores of the GPU. Although authors demonstrate strong correlation between the computed latency and actual latency of the model, I am still skeptical about how well the look up table can model the latency. I would like the authors to present a discussion on the limitations of using the lookup table, if any.

2.	The approach of pruning the model while taking latency into account is not something new and has been addressed in previous works. Moreover, the use of lookup table itself is not very new. In this regard, the novelty of this paper is limited. 

3.	The baselines presented in the paper seem outdated. Authors present a comparison with EagleEye, however, there are more recent pruning methods such as BAR [] and ChipNet [2] and CHIP [3] which perform very efficient pruning. Although these methods do not take latency into consideration, it is of interest to compare with these methods. These methods have been shown to reduce the FLOPs remarkably, thus authors should compare if their latency-aware method can outperform such recent methods. 


##############################
Post rebuttal: Increased the rating from 5 to 6.

questions: Authors need to provide justifications/or address the points stated in the points mentioned in the weaknesses of the paper.
limitations: This work, in my opinion, does not have ant direct negative impact on the society. The limitations are not discussed thoroughly, but I hope with addressing the weaknesses outlined above, those as well can be included.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes latency-aware structural pruning (LASP), which performs pruning as a resource allocation optimization problem to maximize the accuracy while maintaining a latency budget. The proposed methods achieve good performance on the ImageNet dataset for compressing ResNet, VGG, and MobileNet models.

strengths_and_weaknesses: Originality: Each part of the technique is not that novel. It is like a combination of several existing tricks to perform pruning, not enough analysis of the convergence or correctness. 

Quality: 
1) The method lacks mathematical proof. 
2) Evaluation is not comprehensive. It only performs on one dataset, ImageNet. Most of the experiments are on ResNet models. It will be better if more model architectures, like densenet and shufflenet, and more datasets, like cifar10 and cifar100, are included. More related works should be compared in evaluation, like [1-3].

2) In related works, authors mentioned that one limitation of the related works is that a compression ratio does not directly translate into a computation reduction ratio. I do not think it is true. A lot of previous work has studied computation reduction ratio. For example, they studied FLOPs, inference time on hardware.

Clarity: Writing is clear.

Significance: This paper addresses a difficult structuring pruning task in a slightly better way than previous works.

[1] Layer-adaptive Sparsity for the Magnitude-based Pruning
[2] GDP: Stabilized Neural Network Pruning via Gates with Differentiable Polarization
[3] The lottery ticket hypothesis: Finding sparse, trainable neural network

questions: Is the average importance score of each neuron affected by the different batches of data? For example, for different batches of data, the score might be different.

limitations: The authors addressed some limitations of the previous work. It's good that they evaluted the efficiency of the pruned model on real hardware.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: - The author defines an optimization problem that maximizes accuracy with constraints on latency. 
- The author estimates the importance score of neurons based on Taylor Expansion (Molchanov, 2019) and uses the sum of the importance score as the proxy objective of the optimization problem. 
- Moreover, the author estimates latency as the sum of latency contributions of neurons. 
- Using these estimations, the author could view the optimization problem as a knapsack problem and solves the problem by using the knapsack solver augmented by the author.
strengths_and_weaknesses: # Strengths
- The idea is simple and intuitive.
- The proposed latency proxy, the sum of latency reduction potential, seems an efficient and simple estimator. 
- The method does not require additional regularization term during the training. Thus the pruning process can start from the pretrained network. (Some existing pruning methods require regularization term in training step before pruning, so these methods usually start from scratch.)

# Weaknesses
- The novelty of the paper seems insufficient.
- The effectiveness of the proposed latency proxy needs more evidence. 

POST REBUTTAL COMMENTS:
The authors suggest a novel and simple latency estimation technique that can consider latency staircase pattern. They show the effectiveness of proposed latency estimation method with some evidence.
questions: - As the author said in supplementary material, solving pruning problems with a knapsack problem is an existing idea (`KP`). Thus, the main novelty of this paper seems to be in the latency modeling method. However, there are some existing filter pruning methods that regress the latency with a simple model and utilize the model to solve constrained optimization problem. For example, `QCQP` models latency using a quadratic equation and solve a latency constrained optimization problem using QCQP. Hence, LASP seems to require more evidence that shows its benefit.
- For example, I want to see the correlation between real latency and the sum of latency reduction potential, which LASP proposed as a proxy of real latency (I'm curious about the real latency vs the sum of latency reduction potential plot for some pruning recipes). 
- Also, if possible, I want to see the comparison between the sum of latency reduction potential and the quadratic modeling in `QCQP`.
- For the training step, the training recipe and update steps greatly affect the performance. This method requires a few training steps for iterative pruning steps and 90 additional fine-tuning epochs. I would like to see the comparison with the baseline methods (such as `GBN` and `QCQP`) under the training recipe and the number of update steps as unified as possible.

To sum up, my main concern is (1) the analysis on latency proxy, and (2) fair comparison results.

`KP` Knapsack Pruning with Inner Distillation, Aflalo et al.

`QCQP` Optimal channel selection with discrete QCQP, Jeong et al.

`GBN`   Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks, You et al.

POST REBUTTAL COMMENTS : 
The authors answer the questions. My concerns have been addressed and I adjust my score. I recommend highlighting the strength of the latency estimation method with evidence in your responses in the revised version.
limitations: This paper deals with a socially important topic, filter pruning method, that can reduce the computational resources during the inference step of DNN-based APIs.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a method for latency-driven filter pruning, by considering both hardware and importance scoring. To that extent, the authors formulate the pruning problem as a knapsack problem, where the goal is to maximize the importance given the latency constraints, which is optimized using an efficient problem-specific knapsack solver. The authors proceed to show the effectiveness of their pruning method on various networks and hardware configurations.
strengths_and_weaknesses: Strengths:
* The paper is well-written, highly-motivated, and easy to follow
* Results are shown on different devices (including both GPU and CPU, and quantized networks as well)
* The work focuses on real run-times rather than theoretical speedups
* While most of the experiments focus on Imagenet classification, the inclusion of object detection is highly appreciated

Concerns:
* Figure 2 shows LASP+distillation, but I can't find information of it in the paper
* It should be made clearer if Subsection 4.1 include the neuron grouping variation
questions: I have no further questions, and will appreciate addressing my concerns from the main review.
limitations: The authors did not discuss limitations in their main paper, but did refer to a section in the appendix.
I see no potential negative impact from this work.
review_title: null

Meta-review: metareview: Structured neural network pruning is an important problem for efficient inference. Reducing the inference time of the network has many downstream impacts for practical deployments of learned neural networks. The paper tackles the problem of reducing the GPU wall clock latency with structured pruning is a difficult problem due to difficulties in modeling the GPU latency.

The paper proposes latency aware structural pruning (LASP). The idea is to prune less important channels subject to latency budget constraints. The paper seems to build on top of [1] and is closely related to [2]. The main contribution is a) modeling the hardware latency via the lookup table modeling the staircase pattern present in GPUs and b) corresponding resource allocation optimization problem. I think the LASP method provides a clear contribution to the research community in structured neural network pruning and the systematic experiments back up the claim with actual wall clock speedup results on GPUs.

Overall, I would like to suggest the authors to include the baseline comparison results w.r.t GBN and QCQP in the main paper. Also, as a possible future work, I suggest the authors to consider switching the linear objective to the quadratic objective suggested in [2]. The quadratic objective takes into account the coupling effect between neighboring layers: pruning output channels leads to inactive weights along the corresponding input channels in the subsequent layer.

[1] Aflalo et al. Knapsack pruning with inner distillation. 2020
[2] Jeong et al. Optimal channel selection with discrete QCQP. AISTAT 2022
confidence: Certain
Decision: Accept