Structural Pruning via Latency-Saliency Knapsack

Abstract: Structural pruning can simplify network architecture and improve inference speed. We propose Hardware-Aware Latency Pruning (HALP) that formulates structural pruning as a global resource allocation optimization problem, aiming at maximizing the accuracy while constraining latency under a predefined budget on targeting device. For filter importance ranking, HALP leverages latency lookup table to track latency reduction potential and global saliency score to gauge accuracy drop. Both metrics can be evaluated very efficiently during pruning, allowing us to reformulate global structural pruning under a reward maximization problem given target constraint. This makes the problem solvable via our augmented knapsack solver, enabling HALP to surpass prior work in pruning efficacy and accuracy-efficiency trade-off. We examine HALP on both classification and detection tasks, over varying networks, on ImageNet and VOC datasets, on different platforms. In particular, for ResNet-50/-101 pruning on ImageNet, HALP improves network throughput by $1.60\times$/$1.90\times$ with $+0.3\%$/$-0.2\%$ top-1 accuracy changes, respectively. For SSD pruning on VOC, HALP improves throughput by $1.94\times$ with only a $0.56$ mAP drop. HALP consistently outperforms prior art, sometimes by large margins. Project page at \url{https://halp-neurips.github.io/}.

Reviews and author responses:

 Thank you for your time and consideration. We would like to highlight one more time the contributions of this work:

1. We use the latency-aware grouping, which assigns different grouping sizes to each layer according to the latency traits rather than predefined fixed values. To the best of our knowledge, we are the first to incorporate the latency traits into pruning, enabling full exploitation of the latency landscape, thus maximizing the inference speedups.

2. We formulate the pruning into a knapsack problem that is simple yet effective, as pointed out by reviewers, and propose an augmented solver to make it work for the interdependent neuron latency. Our channel selection process is efficient, and our experiments show that it yields a good trade-off between accuracy and latency.

3. Compared to existing methods, we achieve better accuracy with a lower inference time for several architectures, datasets in classification, and object detection.

We reveal that existing methods neglect to consider the actual latency behavior on GPU; thus, they do not maximize the actual computation utilization. Our method focuses on the practical side of pruning neural networks by directly optimizing for the actual latency on the target platform.

 Thank you for your efforts to answer my questions. It is good to add more experimental results here. But I think the novelty of the proposed method is limited. It is like a combination of several existing tricks to perform pruning, not enough analysis of the convergence or correctness. More evaluation should have been done in the first version.


 Dear Reviewer 6smA,

As the deadline for discussion is approaching, we would appreciate it if you could check our responses to your comments. Thank you!

Best Regards,

Authors of the paper

 Thank you for addressing my concerns. With the answers to my problems, I believe the paper is complete. Although there are limitations of the work proposed, I believe it is a good step towards including latency in model compression. I am happy with the response from the authors. Hence I am raising my rating of this paper.

 Dear Reviewer NWGG,

Following your suggestion, we added the one-shot pruning experiment on ResNet50 on Imagenet. For a fair comparison, we use the PyTorch pretrained model and the same finetuning recipe as in QCQP. Table 3 below summarizes the results of this experiment. As shown, LASP consistently outperforms QCQP providing better accuracy and inference speedups. From these results, we can conclude that, compared to QCQP, the channel selection process in our approach yields a better trade-off between accuracy and latency.

Table 3. One-shot pruning results on ResNet50 on ImageNet. “-” means the number is not provided in QCQP paper.
| method | Remaining FLOPs (%) | Top1 drop (%) | Inference speedup |
| ----- | ----- | ----- | ----- |
| QCQP | 59.0 | 0.15 | - |
| **LASP** | **58.7** | **-0.21** | 1.43x |
| QCQP | - | 0.32 | 1.52x |
| **LASP** | 49.4 | 0.32 | **1.79x** |
| **LASP** | 52.6 | **0.03** | **1.56x** |
|||||



 Dear author

Thank you for your nice response. 
Many of my questions have been resolved, and I have a better understanding of LASP's contribution.

But I have one question yet.

If I understood correctly, the difference between QCQP and LASP is summarized in two parts: 

(1) channel selection technique (QCQP selects channels by QCQP with its own latency estimator based on quadratic modeling and LASP selects channels by Knapsack pruning with its own latency estimator based on a lookup table.),

(2) pruning procedure (QCQP prunes the network by one-shot but LASP iteratively prunes the network. ).

So, I wonder which part of the two parts leads to a performance gap. If possible, I want to see the performance of LASP without iterative pruning. In other words, I want to see the one-shot pruning accuracy of LASP to compare the effect of only the channel selection technique directly.

Thank you for your helpful responses. 

 Dear reviewers and ACs,

We sincerely thank all the reviewers for their valuable comments and suggestions to help improve the paper. We have integrated all the comments into our main paper and the appendix. The main changes in the paper are summarized as follows:

* Comparison with more recent pruning methods was added in Fig.2 and Tab.1 of the main paper, and Section C of the appendix. We showed consistent improvements over these new additional baselines.

* In addition to ImageNet and VOC datasets, experiments and comparisons on CIFAR10 for three different architectures (DenseNet40-12, ResNet-56 and ResNet-50) were added as requested to Section C of the appendix. Compared to other methods, we get up to +1.17% higher accuracy under the same FLOP constraints.

* We rephrased the related works in the main paper to add clarity; added more details of experiment setting in Appendix B.

* We added the discussion about the latency lookup table and the comparison with quadratic latency modeling in Appendix M.

* We added the discussion about the augmented knapsack solver in Appendix R.

 Dear Reviewer NWGG,

Thank you for the valuable comments and suggestions. Below are our answers to your concerns.

1. We had a brief description of the teacher model used for distillation in line 232-234. Now we also added the following details regarding the method and the hyperparameters to Appendix B. 

    "For the result in Fig.1 that is trained with knowledge distillation, we use RegNetY-16GF (top1 82.9%) as the teacher model when finetuning the pruned model. We use hard distillation on the logits and the final training loss is calculated as $L = (1-\alpha) L_{base} + \alpha L_{distil}$ where $\alpha = 0.5$ to balance between the original loss $L_{base}$ and the distillation loss $L_{distil}$."

2. Thanks for the suggestion. We have clarified in the experimental section that all our main results are obtained with latency-aware grouping.

 Dear Reviewer NWGG,

Thank you for the valuable comments and suggestions. Below are our answers to your comments and details regarding the modifications to the main paper and appendix.

Fig.11 in Appendix M shows the correlation between the predicted latency reduction and the real latency reduction. As we can see in that figure, predicted and real latency reductions are linearly correlated and, hence, the look up table is a good approximation (the Pearson Correlation Coefficient is 0.995 for ResNet50). The gaps between the predicted and the real latency of the model are likely because we only consider convolution operations to build the lookup table. Given this correlation, it would be possible to use a linear coefficient to calibrate the latency estimation and obtain a better estimation. Compared to the estimation in QCQP, our method gives smaller estimation error as shown from [the figure comparison](https://drive.google.com/file/d/1mc0DuUBAcxaMvxfZl13f6VdaoTh068FF/view?usp=sharing).

The main difference between our estimation from a lookup table and the QCQP[1]’s estimation from quadratic modeling is that we use different ways to model each layer’s latency. QCQP[1] uses $\alpha_l + \beta_l ||r^{(l-1)}||_1 + \delta_l ||r^{(l-1)}||_1 ||r^{(l)}||_1$ to model the layer latency where $||r||_1$ is the the number of remaining channels in the corresponding layer, $\alpha$, $\beta$ and $\delta$ are the coefficients that need to be optimized for the targeting platform. Note that QCQP also needs to profile latency for different samples of each layer, like what we do to create the lookup table but with less samples, in order to optimize $\alpha$, $\beta$ and $\delta$.

It is also important to note that QCQP uses a linear model between the layer latency and FLOPs (the quadratic part) and memory. Therefore, QCQP fails to capture the latency staircase pattern (see [Fig. 3a](https://drive.google.com/file/d/18k9ZGHYZbCpv-GfHwUow9CnpIBAI9RlQ/view?usp=sharing) in the QCQP paper) which is the key to maximize GPU utilization (see latency surface in Fig. 1 in our main paper). As shown in [Fig.3a](https://drive.google.com/file/d/18k9ZGHYZbCpv-GfHwUow9CnpIBAI9RlQ/view?usp=sharing) in the QCQP paper, the quadratic modeling gives different latency estimations to layers with different number of input and output channels. However, these layers have the same real inference time. As a result, the larger the number of input and output channels, the larger the error in the estimation of QCQP. We have included this discussion in Appendix M. 

$~$

We also added an additional comparison with GBN[2] and QCQP[1] to Appendix C and summarize the main results in Tables 1 and 2 below. In Table 1 we report top-1 accuracy and FLOPs while in Table 2 we report the accuracy drop and inference speedup for ImageNet as it is reported in QCQP. As shown in these new results, our method consistently outperforms GBN and QCQP yielding lower FLOPs and higher Top1 accuracy and yields better accuracy and a higher latency speed up when using the same latency constraint.

Table 1. Pruning results comparison with QCQP [1] and GBN [2] on ImageNet and CIFAR10 dataset. Remaining FLOPs (%) are relative to those of the unpruned network
| dataset | model | method | Remaining FLOPs (%) | Top1 (%) |
| ----- | ----- | ----- | ----- | ----- |
| CIFAR10 | ResNet56 | GBN-30 [2] | 29.7 | 93.07 |
| | | **LASP (Ours)** | **26.8** | **93.22** |
| | | QCQP [1] | 47.4 | 93.36 |
| | | **LASP (Ours)** | **33.7** | **93.68** |
| ImageNet | ResNet50 | GBN-60 [2] | 59.46 | 76.19 |
| | | QCQP [1] | 59.0 | 76.00 |
| | | **LASP (Ours)** | **58.12** | **76.93 (76.49&ast; / 76.53&ast;&ast;)** |
| | | GBN-50 [2] | 44.94 | 75.18 |
| | | **LASP (Ours)** | **42.05** | **76.09 (75.27&ast;)** |
||||||

**&ast;** using the same finetune recipe and epochs as GBN; 
**&ast;&ast;** using the same finetune recipe and epochs as QCQP

$~$

Table2. Comparison of pruned model inference speedup with QCQP [1]
| dataset | model | method | Inference speedup | Top1 drop (%) |
| ----- | ----- | ----- | ----- | ----- |
| ImageNet | ResNet50 | QCQP [1] | 1.52x | 0.32 |
| | | **LASP (Ours)** | **1.60x** | **-0.22 (0.16&ast;&ast;)** |
||||||

**&ast;&ast;** using the same finetune recipe and epochs as QCQP

$~$

[1] Optimal channel selection with discrete QCQP, AISTATS’22

[2] Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks, NeurIPS’19.

 Dear Reviewer 6smA,

Regarding the question of whether the important score is affected by different batches of data, we think this might happen for the absolute score value given varying samples for gradient calculation. However, we use global neuron ranking and observed this ranking is in general very stable. To verify this hypothesis, for a ResNet50 on ImageNet, we repeated three times the ranking process of all the neurons with random batches and calculated the spearman’s rank correlation. We obtained an average correlation coefficient of **0.999**, meaning that the neuron ranking is not impacted by potential changes in the absolute importance score. 

 Dear Reviewer 6smA,

Thank you for the valuable comments and suggestions. Below are our answers to your comments and details regarding the modifications to the main paper and appendix.

1. Inspired by the standard dynamic programming solution for the 0-1 knapsack problem [7, 8, 9], we augment the solution to make it work for the interdependent neuron latency, converting it to a greedy approximation selection to retain the same time complexity as the original problem. We tested the greedy method in our paper and showed consistent improvements over prior arts. With the augmented solver, each solution is solvable in $O(N\times C)$ where $N=\sum_{l=1}^L N_l$ is the number of neuron groups and $C$ is the latency constraint. During the experiments we have also tested a more comprehensive, non-greedy method as added in Appendix R (Algo.2), of complexity $O(\sum_{l=1}^L N_l^2 \times C)$. We observed similar pruning performance on ImageNet. We hypothesize that the efficacy of the greedy approach suffices from the already decreasingly ranked neurons feeding into the solver and the iterative nature during pruning. We have added the discussion and the complexity analysis in Appendix R.

2. We test our method on multiple model architectures including ResNet50/101, VGG16, MobileNet-V1 and MobileNet-V2, in the main results section 4.1. We further test our method on the object detection task using Single Shot Detector (SSD) structure on the PASCAL VOC dataset in section 4.5. We use ResNet50 for the platform scalability analysis of our method and ablation study, as it is the most widely adopted setting for papers so it is easier to have fair comparisons.

    As requested, we have added additional experiments on CIFAR10 for three different architectures (DenseNet40-12, ResNet-56 and ResNet-50) with comparisons to GPD [2], CHIP [4], ChipNet [5] and QCQP [6], and a comparison to GPD [2] on ImageNet with a ResNet50. LAMP [1] and lottery ticket [3] focus on unstructured pruning (pruning single parameters) and therefore are not comparable to our method (focused on pruning entire channels). For a fair comparison, for these experiments, we used the same learning rate scheduler as the method to compare. Results are summarized in Table. 1 below and added to Tab.1 and Appendix C of the paper. As shown, our method consistently yields better accuracy than other methods both on ImageNet and CIFAR10 datasets. For DenseNet, we observe competitive results compared to QCQP even though the training hyperparameters of our algorithm tested only once and were not tuned amid timeline.

Table 1. Additional pruning results comparison on ImageNet and CIFAR10 dataset. Remaining FLOPs (%) are reported relative to those of the unpruned network 
| dataset | model | method | Remaining FLOPs (%) | Top1 drop (%) |
| ----- | ----- | ----- | :-----: | :-----: |
| CIFAR10 | ResNet56 | CHIP [4] | 27.7 | 1.21 |
| | | **LASP (Ours)** | **26.8** | **0.46** |
| | | GDP [2] | 34.36 | 0.35 |
| | | **LASP (Ours)** | **33.72** | **0.0** |
| | ResNet50 | ChipNet [5] | 17.7 | 0.5 |
| | | **LASP (Ours)** | **13.7** | **0.3** |
| | DenseNet40-12 | QCQP [6] | 29.2 | 1.21 |
| | | **LASP (Ours)** | 29.2 | 1.25 |
| ImageNet | ResNet50 | GDP [2] | 47.62 | 0.30 |
| | | **LASP (Ours)** | **47.59** | **-0.04** |
|||||

3. In the related work, we use “compression ratio” for channel pruning ratio specifically. In that case, the pruning of a neuron at different location leads to different reduction on FLOPs or latency. We have rephrased this part in the main paper to add clarity. Prior methods using FLOP constraints are limited as FLOPS and actual latency are not linearly correlated (lines 90-65). Therefore, these methods are limited to considering the FLOP constraint as a proxy of the actual latency and do not optimize for the actual GPU utilization. The last two paragraphs of the related work section also include a discussion regarding methods focused on inference time. 

$~$

[1] Layer-adaptive Sparsity for the Magnitude-based Pruning, ICLR’21

[2] GDP: Stabilized Neural Network Pruning via Gates with Differentiable Polarization, ICCV’21

[3] The lottery ticket hypothesis: Finding sparse, trainable neural network, ICLR’19

[4] CHIP: CHannel independence-based pruning for compact neural networks, NeurIPS’21

[5] Chipnet: Budget-aware pruning with heaviside continuous approximations, ICLR’21

[6] Optimal channel selection with discrete QCQP, AISTATS’22

[7] Unbounded knapsack problem: Dynamic programming revisited, 2000

[8] Dynamic programming and strong bounds for the 0-1 knapsack problem, 1990

[9] Knapsack problems: algorithms and computer implementations, 1990



 Dear Reviewer 7C8F,

Thank you for the valuable comments and suggestions. Below are our answers to your comments and details regarding the modifications to the main paper and appendix.

1. We added the discussion about the layer-wise latency lookup table in Appendix M and summarized as below:

    We apply layer-wise latency lookup table in our method, which does not consider the caching and parallelization among layers, currently in line with prior literature and reliable for a wide range of networks over off-the-shelf platforms such as GPUs, CPUs, and mobile DSPs as also detailed by ChamNet [CVPR’19], FBNet[CVPR'19] and OFA [CVPR’20]. For networks like VGG without bypass paths, the layer operations will be executed sequentially, in which case, the latency lookup table models the actual latency. For models with parallel paths, it depends on the hardware implementation whether there will be parallel execution in practice. When there is parallelization, e.g., in accelerators, LUTs for models like ResNets can still work well as skip connection only takes small portion of computation (E.g., 9% in ResNet50 and 5% in ResNet101); for models like InceptionNet, it is good a suggestion to take the parallelization into consideration when generating the lookup table to better estimate the latency. For wider applications in the future, the more domain knowledge we have about the GPU execution and improve accordingly, the more accurate estimation we will obtain using the latency lookup table.

2. The main novelty of the paper, instead of LUT, is that 1) we use the latency-aware grouping which assigns different grouping sizes to each layer according to the latency traits rather than predefined fixed values. This allows full exploitation of the latency landscape of the underlying hardware (GPU) thus maximizing the inference speedups. 2) We formulate the pruning into a knapsack problem and propose the augmented solver to make it work for the interdependent neuron latency.

3. We appreciate the feedback and next add additional comparison with CHIP [3] on ResNet50 on ImageNet dataset (in Fig.2 and Tab.1 of the main paper); and CHIP [3] and ChipNet [2] on CIFAR10 (in Appendix C). BAR [1] uses knowledge distillation in training to improve accuracy, therefore, it is not possible to have a fair comparison.

      The summary of these results is shown in Table.1 below where Remaining FLOPs (%) are relative to those of the unpruned network. As shown, our approach consistently yields better Top1 accuracy with less FLOPs.

$~$

Table 1. The pruning results comparison with ChipNet [2] and CHIP [3] on ImageNet and CIFAR10 dataset.
| dataset | model | method | Remaining FLOPs (%) | Top1 (%) |
| ------ | ----- | ----- | :-----: | :-----: |
| ImageNet | ResNet50 | CHIP [3] | 51.3 | 76.15 |
|                |                | **LASP (Ours)** | **50.3** |  **76.46** |
| | | CHIP [3] | 23.3 | 73.30 |
| | | **LASP (Ours)** | **22.8** | **73.69** |
| CIFAR10 | ResNet56 | CHIP [3] | 27.7 | 92.05 |
| | | **LASP (Ours)** | **26.8** | **93.22** |
| CIFAR10 | ResNet50 | ChipNet [2] | 17.7 | 92.8 |
| | | **LASP (Ours)** | **13.7** | **93.2** |
|||||

$~$

[1] Structured Pruning of Neural Networks with Budget-Aware Regularization, CVPR’19

[2] Chipnet: Budget-aware pruning with heaviside continuous approximations, ICLR’21

[3] CHIP: CHannel independence-based pruning for compact neural networks, NeurIPS’21



 This paper addresses the important research question of to prune neural networks such that the gain in inference is maximized. Authors to convert the neural network pruning problem into a resource allocation problem and solve it using Knapsack strategy. For estimation latency for different parts of the network, authors employ a lookup table. Experiments are conducted on multiple hardwares and authors demonstrate improved performance with the proposed method. Strengths: 
The approach of optimizing the performance of neural networks w.r.t latency budget is interesting and a very relevant research problem from a practical point of view. The strategy presented by the authors permits designing pruned networks for different choices of latency budget which makes this approach quite useful. Further, the lookup table approach of the authors makes it very easy to include the d latency parameter into the optimization problem. Authors demonstrate the efficacy of the method on various choices of hardwares and present an extensive ablation study around the proposed method. Overall, the research content seems original, well written and the message is clear.


Weaknesses: The presented paper in its current format has certain weaknesses and I would like to request the authors to respond with their remarks to the weaknesses listed below.

1.	A major concern for me is the use of lookup table to construct the latency of the overall network. In my opinion, latency cannot be computed as a sum of latency from different parts, rather it should also take into account the parallelizability across the cores of the GPU. Although authors demonstrate strong correlation between the computed latency and actual latency of the model, I am still skeptical about how well the look up table can model the latency. I would like the authors to present a discussion on the limitations of using the lookup table, if any.

2.	The approach of pruning the model while taking latency into account is not something new and has been addressed in previous works. Moreover, the use of lookup table itself is not very new. In this regard, the novelty of this paper is limited. 

3.	The baselines presented in the paper seem outdated. Authors present a comparison with EagleEye, however, there are more recent pruning methods such as BAR [] and ChipNet [2] and CHIP [3] which perform very efficient pruning. Although these methods do not take latency into consideration, it is of interest to compare with these methods. These methods have been shown to reduce the FLOPs remarkably, thus authors should compare if their latency-aware method can outperform such recent methods. 


##############################
Post rebuttal: Increased the rating from 5 to 6.
 Authors need to provide justifications/or address the points stated in the points mentioned in the weaknesses of the paper. This work, in my opinion, does not have ant direct negative impact on the society. The limitations are not discussed thoroughly, but I hope with addressing the weaknesses outlined above, those as well can be included.

 This paper proposes latency-aware structural pruning (LASP), which performs pruning as a resource allocation optimization problem to maximize the accuracy while maintaining a latency budget. The proposed methods achieve good performance on the ImageNet dataset for compressing ResNet, VGG, and MobileNet models.
 Originality: Each part of the technique is not that novel. It is like a combination of several existing tricks to perform pruning, not enough analysis of the convergence or correctness. 

Quality: 
1) The method lacks mathematical proof. 
2) Evaluation is not comprehensive. It only performs on one dataset, ImageNet. Most of the experiments are on ResNet models. It will be better if more model architectures, like densenet and shufflenet, and more datasets, like cifar10 and cifar100, are included. More related works should be compared in evaluation, like [1-3].

2) In related works, authors mentioned that one limitation of the related works is that a compression ratio does not directly translate into a computation reduction ratio. I do not think it is true. A lot of previous work has studied computation reduction ratio. For example, they studied FLOPs, inference time on hardware.

Clarity: Writing is clear.

Significance: This paper addresses a difficult structuring pruning task in a slightly better way than previous works.

[1] Layer-adaptive Sparsity for the Magnitude-based Pruning
[2] GDP: Stabilized Neural Network Pruning via Gates with Differentiable Polarization
[3] The lottery ticket hypothesis: Finding sparse, trainable neural network
 Is the average importance score of each neuron affected by the different batches of data? For example, for different batches of data, the score might be different.
 The authors addressed some limitations of the previous work. It's good that they evaluted the efficiency of the pruned model on real hardware.


 - The author defines an optimization problem that maximizes accuracy with constraints on latency. 
- The author estimates the importance score of neurons based on Taylor Expansion (Molchanov, 2019) and uses the sum of the importance score as the proxy objective of the optimization problem. 
- Moreover, the author estimates latency as the sum of latency contributions of neurons. 
- Using these estimations, the author could view the optimization problem as a knapsack problem and solves the problem by using the knapsack solver augmented by the author. # Strengths
- The idea is simple and intuitive.
- The proposed latency proxy, the sum of latency reduction potential, seems an efficient and simple estimator. 
- The method does not require additional regularization term during the training. Thus the pruning process can start from the pretrained network. (Some existing pruning methods require regularization term in training step before pruning, so these methods usually start from scratch.)

# Weaknesses
- The novelty of the paper seems insufficient.
- The effectiveness of the proposed latency proxy needs more evidence. 

POST REBUTTAL COMMENTS:
The authors suggest a novel and simple latency estimation technique that can consider latency staircase pattern. They show the effectiveness of proposed latency estimation method with some evidence. - As the author said in supplementary material, solving pruning problems with a knapsack problem is an existing idea (`KP`). Thus, the main novelty of this paper seems to be in the latency modeling method. However, there are some existing filter pruning methods that regress the latency with a simple model and utilize the model to solve constrained optimization problem. For example, `QCQP` models latency using a quadratic equation and solve a latency constrained optimization problem using QCQP. Hence, LASP seems to require more evidence that shows its benefit.
- For example, I want to see the correlation between real latency and the sum of latency reduction potential, which LASP proposed as a proxy of real latency (I'm curious about the real latency vs the sum of latency reduction potential plot for some pruning recipes). 
- Also, if possible, I want to see the comparison between the sum of latency reduction potential and the quadratic modeling in `QCQP`.
- For the training step, the training recipe and update steps greatly affect the performance. This method requires a few training steps for iterative pruning steps and 90 additional fine-tuning epochs. I would like to see the comparison with the baseline methods (such as `GBN` and `QCQP`) under the training recipe and the number of update steps as unified as possible.

To sum up, my main concern is (1) the analysis on latency proxy, and (2) fair comparison results.

`KP` Knapsack Pruning with Inner Distillation, Aflalo et al.

`QCQP` Optimal channel selection with discrete QCQP, Jeong et al.

`GBN`   Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks, You et al.

POST REBUTTAL COMMENTS : 
The authors answer the questions. My concerns have been addressed and I adjust my score. I recommend highlighting the strength of the latency estimation method with evidence in your responses in the revised version. This paper deals with a socially important topic, filter pruning method, that can reduce the computational resources during the inference step of DNN-based APIs.

 The paper proposes a method for latency-driven filter pruning, by considering both hardware and importance scoring. To that extent, the authors formulate the pruning problem as a knapsack problem, where the goal is to maximize the importance given the latency constraints, which is optimized using an efficient problem-specific knapsack solver. The authors proceed to show the effectiveness of their pruning method on various networks and hardware configurations. Strengths:
* The paper is well-written, highly-motivated, and easy to follow
* Results are shown on different devices (including both GPU and CPU, and quantized networks as well)
* The work focuses on real run-times rather than theoretical speedups
* While most of the experiments focus on Imagenet classification, the inclusion of object detection is highly appreciated

Concerns:
* Figure 2 shows LASP+distillation, but I can't find information of it in the paper
* It should be made clearer if Subsection 4.1 include the neuron grouping variation I have no further questions, and will appreciate addressing my concerns from the main review. The authors did not discuss limitations in their main paper, but did refer to a section in the appendix.
I see no potential negative impact from this work.

Meta-review: Structured neural network pruning is an important problem for efficient inference. Reducing the inference time of the network has many downstream impacts for practical deployments of learned neural networks. The paper tackles the problem of reducing the GPU wall clock latency with structured pruning is a difficult problem due to difficulties in modeling the GPU latency.

The paper proposes latency aware structural pruning (LASP). The idea is to prune less important channels subject to latency budget constraints. The paper seems to build on top of [1] and is closely related to [2]. The main contribution is a) modeling the hardware latency via the lookup table modeling the staircase pattern present in GPUs and b) corresponding resource allocation optimization problem. I think the LASP method provides a clear contribution to the research community in structured neural network pruning and the systematic experiments back up the claim with actual wall clock speedup results on GPUs.

Overall, I would like to suggest the authors to include the baseline comparison results w.r.t GBN and QCQP in the main paper. Also, as a possible future work, I suggest the authors to consider switching the linear objective to the quadratic objective suggested in [2]. The quadratic objective takes into account the coupling effect between neighboring layers: pruning output channels leads to inactive weights along the corresponding input channels in the subsequent layer.

[1] Aflalo et al. Knapsack pruning with inner distillation. 2020
[2] Jeong et al. Optimal channel selection with discrete QCQP. AISTAT 2022

Decision: Accept