Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposed a post-training pruning framework for Transformers, without retraining. It prunes of both heads in MHA and filters in FFN layers in a structured  way. The process is done by applying a lightweight Fisher based mask search along with a Fisher mask rearrangement and mask tuning. The results are comparable or even better FLOPs-accuracy trade-off than prior methods.
strengths_and_weaknesses: Strengths: The paper is well written, and the authors proposed the method in quite a detail. The post-training pruning framework does not require retraining, which is very good. Adding latency-constrained in consideration is also good.
The experiments are quite sufficient and are able to support their claims and conclusions. The results show the effectiveness of the proposed methods. The paper also compared existing structured pruning works on GLUE


questions: 1. The reason to use second-order Taylor. It seems that first-order Taylor can be faster and doesn't lose much accuracy? Some other works use first-order [1][2]


[1]Accelerating Sparse DNN Models without hardware-Support via Tile-Wise Sparsity
[2]Chasing sparsity in vision transformers: an end-to-end exploration
limitations: Please see the above comments. I will consider changing my ratings based on the author's rebuttal.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes three  techniques to obtain a high accuracy transformer without  retraining. A search algorithm to find which heads and filters are need to prune based on the Fisher information. An algorithm which rearrange the mask that complements the search algorithm. And tuning the mask which reconstructs the output activations for each layer. The experiments show the author can get better results than compare methods.
strengths_and_weaknesses: Strengths:

This paper is well-written, well-motivated, and clear presentation.

The proposed algorithm improves transformer throughout efficiency with competitive accuracy and small latency, outperforming prior pruning and distillation approaches.

Formulating the hardware-aware structural pruning as a knapsack problem is interesting.

Weaknesses:

This paper should compare their algorithms with some SOTA pruning methods on transformer such as COFI[1]

[1]Xia M, Zhong Z, Chen D. Structured pruning learns compact and accurate models[J]. arXiv preprint arXiv:2204.00408, 2022.
questions: 1、How to obtain latency exactly? Is the latency become different when obtain them from different hardware and inference engines


limitations: Please see my weaknesses
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors proposed a fast post-training pruning framework for transformer-based language models which does not require retraining. The algorithm takes a model, sample dataset, and compression constraint to generate the compressed model. It introduces three techniques to retain high accuracy: 1. mask search; 2. mask rearrangement; 3. mask tuning. Experiments show that the proposed method achieves 2x FLOPs reduction and 1.6x speed up within 1% accuracy drop.
strengths_and_weaknesses: Strengths:
1. Firstly, the paper is well written and easy to follow. 
2. The proposed method solves a complex optimization problem by introducing several approximations and using a multi-step approach. The process is introduced clearly. The effectiveness of each step is demonstrated by ablation studies. 
3. The experimental results are solid. The proposed method achieves similar performance compared to existing work without a large training cost, which could be valuable to real-life applications. 
4. I like the discussion of latency-aware compression, where the authors used a piece-wise linear function to approximate the latency LUT, which is integrated into the optimization objective. It is a smart design to fit both settings under the same optimization framework. 

Weakness:
1. Firstly, it seems that the proposed method is not limited to pruning transformers; it can be also applied to other models like CNNs by just using the channel mask. Is there any consideration why the work is limited to transformers? Will the algorithm perform well on other models like CNNs? 
2. There are some other works on post-training channel pruning (e.g., [a]). It would be better if the authors can show the proposed method can also outperform general data-free pruning methods when applied to transformers. But I also agree the current experimental results are already solid.

[a] Lazarevich et al., Post-training deep neural network pruning via layer-wise calibration, ICCVW. 

questions: 1. The authors may discuss if the proposed method is only restricted to transformers. Can it be applied to other models like CNNs? Is it possible that the post-training method only works well here due to the large redundancy of BERT models on a specific downstream dataset?

limitations: There lacks a discussion of potential negative societal impacts. It might be acceptable due to the nature of the work; but a discussion is still encouraged.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a three-stage post-training pruning framework for Transformers.  It first uses Fisher information to search the binary mask, i.e, layer-wise pruning rate. Then, the framework modifies the binary mask in a layer-wise manner. Lastly, it tunes the non-zero valued mask for minimizing the layer-wise reconstruction error.

Such a framework could retain the performance of the model without retraining, thus it can finish the pruning in less than 3 minutes on a single GPU and can obtain actual speedup in inference because of structured pruning.

Extensive experiments show that the proposed post-training pruning framework has a comparable performance with prior methods.

strengths_and_weaknesses: 1. To my knowledge, this is the first work to use post-training pruning in Transformer. I recognize the contribution of applying technologies to new areas.

2. Proposing simple mask search solutions based on FLOPs and Latency,  which avoids user intervention.

3. Experiments show the efficacy  of the proposed framework, which could retain high accuracy without retraining.
questions: 1. Figure 3 (Left) only shows that  the actual speed is linear to the number of filters, which does not support the claim in line206-207. I suggest authors re-plot Figure 3 (Left) with fewer filter number.

2. Technical merits are limited.  Each component used in the three-stage pruning has been used in prior works. For example,   taking Fisher Information Matrix as an importance score and  tuning the mask variables toward minimizing the layer-wise reconstruction error.

3. I can not understand why the mask by Eq. (4) and Eq. (11) is different because the importance score is the same.   Line 237 claims that $||m_l||_0$ is  equal to $||m_l^*||$. Line 242 claims that $\mathcal{I}_l$ is the  $l$-th diagonal block of $\mathcal{I}$. In such situation, $m_l^*$ selected by $\mathcal{I}$  and $m_l$ selected by $\mathcal{I}_l$ should be the same. This can be proved by contradiction.

4. The ablation experiments are not clear enough. There are two ablative modules in Figure 6, including importance score and mask search algorithm. Such an ablation study with one more ablative module can not support the claim of Line 319.  The ablation experiment in Figure 7 should be divided into two experiments. 1) The importance score. Keeping the three-stage pruning pipeline and just modifying the importance score, which can reflect the efficacy of the importance score. 2) Mask search and rearrangement. Keeping the importance score and using uniformly prune, can reflect the efficacy of mask search and re-arrangement.


typos:

- Line 613: Eq. ?? -> Eq. 9
- Algorithm 2.2 pf->of

==================post rebuttal============
My concerns are addressed. I raise up my score. Thanks.
limitations: No societal impact discussion needed, in my opinion.
review_title: null

Meta-review: metareview: The authors deliver on what they promise: a fast post-training pruning framework for transformers. It reduces the inference costs of deploying transformers while preserving much or all of their accuracy on the standard range of academic downstream tasks. Moreover, it does so without the hefty costs that typically come with prune-and-retrain cycles. The paper is clearly written and well-presented, and the technique seems to work quite well. The authors seemed to satisfactorily address all reviewer concerns, and those concerns were minor at best. What more can you ask for? I look forward to visiting the poster at NeurIPS and trying this technique myself.

The authors are to be especially commended for focusing on real-world speedup on real hardware. That's (sadly) still a rarity in pruning papers. This is something that appears genuinely useful, today, by practitioners.
confidence: Certain
Decision: Accept