Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors proposed the pruning method with low-frequency for ViTs. The metrics named low-frequency sensitivity for channel pruning, and low-frequency energy for token pruning are proposed, where the low-frequency image and feature are introduced. Besides, a bottom-up cascade pruning is proposed to prune the channels and tokens gradually. The experiments show a good reduction ratio of FLOPs and throughput with less performance drop, and the effectiveness of each proposed components.
strengths_and_weaknesses: Strengths:
1. The authors introduced low-frequeny image and feature to prune the ViT models. It is the first work to compress the model in the frequency domain.
2. The improvement on the throughput is significant with less performance drop.
3. The formulas of the proposed pruning method are clear and correct.
4. The paper is well written.

Weaknesses:
1. The performance metric is not sufficient. The pruned model can be evaluated on ImageNet-Real/V2 and downstream classification datasets, like CIFAR-10/100.
2. In Table 1, there is no comparison on the number of the finetuning epochs.
3. In Table 5, the accuracy of the model without LFS and BCP after finetuning is close by 0.2 to VTC-LFC. The difference is too small to show the effectiveness of the proposed method.
questions: 1) In the section 4.1 Line 228, the pruned models DeiT-Tiny/DeiT-Small/DeiT-Base are finetuned for 300/150/75 epochs, respectively.
     - Why the number of epochs for the three models are different?
     -  How many epochs to finetune the pruned models for other pruning methods, like SPViT, S2ViTE? Is it comparable to that in this paper?
     -  It usually takes 300 epochs to train DeiT-Tiny on ImageNet-1k. Why the pruned DeiT-Tiny needs finetuning for 300 epochs? 

2) In the section 3.4 Bottom-up Cascade Pruning, the model is pruned from the first block to the last one. Is there any comparison to the method from the last block to the first one?
3) In line 164, how to set the hyper-parameter λ in the experiment? Is there any ablation study on it?
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces a model compression approach based on low-frequency components for vision transformers. Channel pruning based on low-frequency sensitivity, token pruning based on low-frequency energy, along with bottom-up cascade pruning scheme effectively reduce the model complexity while maintaining high model accuracy. 
strengths_and_weaknesses: strengths: 
1. it proposes a novel approach for model compression with low-frequency component, by leveraging the intrinsic property of self-attention modules as low-pass filters. 
2. it introduces the bottom-up cascade pruning framework for model compression.  
3. it outperforms many SOTA vision transformer compression methods with higher accuracy at reduced model complexities 

weaknesses: 
it's unclear how effective this approach would be when applied to other ViT models such as LV-ViT. 
questions: 1. How effective would this approach be when applied to other ViT models such as LV-ViT? 
2. Would it be useful for window-based attention and hierarchical model structure such as Swin?
limitations: adequate
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The major contributions of this paper are 2-fold:
1) By leveraging on the understanding regarding the low-frequency information preference of ViTs, this work proposes improved channel pruning and token pruning schemes to compress ViTs. This is achieved using 3 components below:
- **Channel pruning** : A quantitative metric called LFS (Low Frequency Sensitivity) is proposed to estimate the importance of model parameters by largely considering low-frequency components in images. This is used for channel pruning.
- **Token pruning** : A quantitative metric called LFE (Low Frequency Energy) is proposed to estimate the percentage of low-frequency information in tokens. 
- **A bottom-up cascaded pruning framework** is proposed to jointly compress channels and tokens of ViTs.

2) The proposed method obtains large improvements in ViT compression (i.e.: reduce FLOPs by 40%-60% at the expense of only 1% top1 accuracy) in ImageNet based classification setups.

strengths_and_weaknesses: **Strengths:**
1) This paper is written well. It is easy to follow.
2) The improvements in terms of FLOPs and parameter reduction using VTC-LFC is impressive. VTC-LFC has potential to be useful in edge-intelligence applications (where ViTs are superior to CNN based architectures).

**Weaknesses:**
1) **The overall compression procedure is unclear**. I.e.: There is also a knowledge distillation component (Line 159). Can the authors show the overall algorithm (a pseudocode will be useful)? 
2) **Only final top1 accuracy of VTC-LFC is insufficient to understand the significance of the proposed method / More analysis is needed**: This paper could significantly benefit from more analysis / statistics regarding channel pruning and token pruning. Given that LFS and LFE are scalar values, can the authors show the distribution of these values in Baseline / VTC-LFC ViTs? (A simple histogram could be useful). 
3) **The cutoff point between low-frequency and high frequency for the experiments is not clear**: How is Figure 4 ( c ) obtained? Is it obtained by radial averaging of 2D Fourier spectrum similar to these deepfake works ( [1, 2, 3, 4] )?
4) The caption of Figure 3 is too plain. Please consider explaining the details as Figure 3 is very important in this paper.

Overall I enjoyed reading this paper. In my opinion, the strengths of this paper outweigh the weaknesses by a small margin. I’m happy to change my opinion based on the rebuttal. 

=================

[1] Dzanic, T., Shah, K., & Witherden, F. (2020). Fourier spectrum discrepancies in deep network generated images. Advances in neural information processing systems, 33, 3022-3032.

[2] Durall, R., Keuper, M., & Keuper, J. (2020). Watch your up-convolution: Cnn based generative deep neural networks are failing to reproduce spectral distributions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 7890-7899).

[3] Chandrasegaran et al., 2021: "A closer look at fourier spectrum discrepancies for cnn-generated images detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[4] Schwarz et al., 2021: "On the Frequency Bias of Generative Models." Advances in Neural Information Processing Systems 34 (2021).


questions: Please see Weaknesses section above for a list of questions. Further please consider answering the following questions:

1) Can the authors include Top-5 accuracies for Tables 1, 2?

limitations: Briefly mentioned in Checklist 1(C).
review_title: null

Meta-review: metareview: The authors present a method to improve ViT efficiency by pruning channels and tokens using a selection mechanism that emphasizes low spatial frequency information. In particular they propose two measures: Low Frequency Sensitivity (LFS) and Low Frequency Energy (LFE).

1) LFS comprises two parts (Eq 3), the contribution between is controlled by a weighting hyper-parameter (though ultimately a Taylor approximation is used): 
	a. The difference in the loss function of the model on the low-pass image with and without a removed weight. 
	b. The difference in KL divergence of the class token before and after low-pass filtering with and without the weight in question. 
2) LFE measures the proportionate energy of a token among the energy of all tokens after low-pass filtering (Eq 6), combined with a measure of the attention weights (Eq 8).

Pruning is carried out via a bottom-up cascade mechanism (Sec 3.4). 

Performance is evaluated on a variety of model sizes on ImageNet 1K.

Pros:
- [R] First work to compress a model emphasizing low-frequency information
- [AC/R] Throughput improvement is significant with minimal performance drop.
- [R] Clear and correct formulas.
- [AC/R] Well written.
- [AC/R] Outperforms other compression methods.

Cons:
- [R] More thorough evaluation needed (ImageNet Real/V2 and CIFAR 10/100). Authors followed up and provided this requested evaluation data.
- [R] No comparison on the impact of the number of epochs. Authors supplied information.
- [R] No ablation on hyperparameter for LFS. Authors supplied information.
- [R] Unclear how the method might work with window based ViT such as Swin. Authors provided these additional experiments.
- [R] Unclear what is the cutoff for low/high frequency components. Authors provided ablation of varying cutoffs.

Paper has unanimous accept ratings. Authors addressed reviewer concerns, and reviewers complimented authors for a good job. AC recommends accept.

AC Rating: Strong Accept
confidence: Certain
Decision: Accept