Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This work proposes to prune ViT from all components comprehensively, which considers the interactions between different components in pruning ViTs. Different from the homogeneous components of CNN, the components of ViT are always heterogeneous, thus this work constructs a Taylor-based optimization function to take full advantage of the interactions between heterogeneous components. To avoid the huge computation cost of the Hessian matrix, this work derives an approximation to transform the Hessian matrix into pruning ratios. Finally, it solves the optimization target towards the optimal trade-off between accuracy and computational cost. This work is validated on DeiT and Swin on ImageNet and also on detection experiments.
strengths_and_weaknesses: Strengths:
1. This work is well-written and easy to follow.
2.  The motivation of this work is clear.


Weaknesses：

1. Please verify the definition of ∆w, as ∆w = b ⊙ w − w, while the optimization target (eq.1) is: min ∆L = L(w + ∆w) − L(w), thus ∆L = L(w + ∆w) − L(w) = L(b ⊙ w) - L(w). And  C(w + ∆w) = C(b ⊙ w). I think this target is not correct, which maybe cuased by the definition of ∆w.
2. Based on the definition of ∆w, I think we cannot get eq2 from eq1.
3. The pruning results on Swin are directly trained on ImageNet-1k, while Swin can achieve much better performance after pretraining on ImageNet-22k. So the author should also provide experimental results on it. No need to train Swin on ImageNet-22k, but directly load the pretrained weights on ImageNet-22k then do pruning along with finetuning on ImageNet-1k.

questions: Overall, I think this work has a reasonable motivation and good idea, but the definition of ∆w confused me.
limitations: Please refer to Weaknesses.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a model pruning method for vision Transformers by jointly considering multiple possible pruning dimensions. To address the problem of joint optimization, a new collaborative pruning is designed. Experiments on multiple backbones (i.e., DeiT and Swin) and multiple tasks (i.e., ImageNet classification and COCO detection) show the effectiveness of the method.
strengths_and_weaknesses: Strengths:

- The idea of jointly considering multiple pruning dimensions is natural and well-motivated. 

- The method is tested and works well on multiple tasks and backbones.

Weaknesses:

- The idea of joint optimization of the vision Transformer architecture is not very new. Many previous methods have explored the joint optimization problem for network acceleration. Recent work like AutoFormer also considers multiple dimensions for vision Transformers.

- Table 3 presents an important experiment to compare with previous state-of-the-art pruning methods. Since ViT/DeiT-B/S are considered as the standard models in many previous papers, it is better to provide the results on multiple model sizes (e.g., ViT-B/S/T) to clearly show the effectiveness of the proposed method.

- According to Table 6, pruning multiple dimensions may not lead to ideal actual speedup on GPUs.

- The method introduces a new pruning algorithm, which may not be easy to implement. Since the code is not available, I am a bit worried about the reproducibility of the method.


-----------------
Post rebuttal:

I would like to thank the authors for the detailed feedback and additional results. The response addressed my concerns about the insufficient experiments and actual speedup on GPUs. I would like to upgrade my rating to 5.

questions: The paper presents a thorough study of the emerging area of efficient vision Transformers. The method is tested on multiple backbones and datasets. However, the joint optimization framework is similar to previous methods for CNNs, which makes the novelty of the pruning algorithm relatively low. Besides, I still have some concerns about the insufficient experiments, actual speedup on GPU, and reproducibility.  This paper can be stronger if the issues mentioned in the weaknesses subsection can be addressed.
limitations: Limitations of the proposed method are not discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a new neural network pruning method for vision transformers.  The proposed technique can effectively accelerate most vision transformers such as ViT and Swin Transformer by collaboratively pruning components such as multi-head self-attention, hidden neurons, and embedding neurons. Extensive experiments show that the proposed technique is efficient yet competitive in accuracy as compared with the state-of-the-art. 
strengths_and_weaknesses: The manuscript has the following pros as follows: 1) The paper has a clear motivation and the proposed technique is based on the theoretical analysis. 2) The paper is clearly written, and easy to follow in general; 3) A large number of experiments have been performed to validate the performance in various aspects.
 
I have some small concerns about this paper as listed. First, in Sec 4.4, the author said that the proposed method can bring accuracy gains when compressing the DeiT-Base. I think this conclusion is a bit inappropriate. As described in Sec. 4.1 and 4.2, the pruned models will further fine-tune 300 epochs, which means that the pruned models have a longer training schedule than baseline models. The accuracy gains may come from the longer training schedule.
 
In addition, the pseudo-code for the whole pruning algorithm in the Appendix is the core contribution of this work, which should be moved to the main paper.

questions: No questions.
limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes SAViT,  a structure-aware pruning method for transformer-based architecture, which jointly prunes parameters in different components by considering interactions between these components. Experiments on different ViT architectures and vision tasks demonstrate the effectiveness of SAViT.
strengths_and_weaknesses: Strengths
- The idea of collaboratively pruning all components of a model is interesting.
- The performance gain is impressive when compared to sufficient state-of-the-art methods.
- This paper is well-written and easy to follow.

Weaknesses
- It's not clear how much computation cost and data is needed for the proposed method during pruning, when compared to state-of-the-art network pruning methods.
- The main novelty of this work is the interaction of different components during pruning, so the ablation study on this design is important. In my understanding, Line 297-305 and Table 7 aim to give such ablation studies, while it's not clear whether the setting of "without second-order interations" (Line 300) in this ablation study means dropping all Hessian-based terms in Eq. (4) or only drop the cross-components terms (green blocks in Figure 2(b))? I think the latter one can better reflect the main contribution of the proposed method.
questions: See "Weaknesses"
limitations: The authors haven't discussed any limitations.
review_title: null

Meta-review: metareview: The paper received three positive reviews and one negative review. The raised issues contain technical correctness, ImageNet-22K pertaining, insufficient experiments and speedup on GPUs, computational cost, clarity on ablation studies. During the rebuttal and discussion phases, most of the issues are addressed and reviewers are willing to upgrade. After checking all the reviews, rebuttals, and discussions, the AC agrees with the reviewers that the raised issues are well addressed. The authors shall revise according to the suggestions to further improve the current manuscript in the camera-ready submission. Also, the comparison to token selection-based ViT acceleration methods [a] shall be included in the experiments. 

[a]. Not All Patches Are What You Need: Expediting Vision Transformers via Token Reorganizations. Liang et al. ICLR 2022.
confidence: Certain
Decision: Accept