Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Vision Transformers (ViT) have attracted much attention recently; however their large model sizes and high training cost pose practical problems for computer vision tasks. This paper introduces sparse ViT exploration algorithms ($SViTE$, $S^2ViTE$, $SViTE+$) to explore sparse patterns in the ViT architectures. The goal is to alleviate the training memory bottleneck and achieving high inference efficiency. The authors conducted comprehensive experiments on ImageNet dataset to validate their approach.
main_review: **Strengths:**

- The paper is clear, well-written and well organized.
- The authors included different illustrations and descriptions of the proposed algorithms. In addition to including several visualizations such as the sparse connectivity patterns and self-attention heatmaps.
- The experimental results show the efficiency of the proposed methods. The gain in the classification accuracy and the runtime savings are significant in many cases.
- The proposed methods could open a way for training and utilizing Vision Transformer models in an efficient way for complex computer vision tasks especially for low resources devices and real-time applications. 


**Weaknesses:** 

- The novelty of the proposed methods seem to be highly incremental. It seems the methods/ideas presented here were borrowed from the prior related work ([24], [25]); the difference seems to be applying those methods on Vision Transformers. 
- In the visualization Figure 5, the dropped/selection patches patterns seem to be random. There is high similarity between many of the background patches and other patches that weren't dropped. 
- Some visualizations are not clear enough. For example, in Figure 4, I couldn't notice the black squares mentioned in the caption.


limitations_and_societal_impact: There is no potential negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes sparse ViT algorithms, named as SViTE and its variants S^2ViTE, and SViTE+, to explore high-quality sparse patterns in both ViT’s architecture and input token embeddings. The proposed methods can jointly optimize model parameters and explore connectivity throughout training. Experiments are conducted on ImageNet to validate the effectiveness of the proposed methods and find that SViTE+ can mine crucial connections and input tokens.
main_review: This work aims to explore the sparse vision transformers by unstructured and structure pruning methods. The proposed SViTE and its variants S^2ViTE, and SViTE+ achieve impressive results on baselines.
limitations_and_societal_impact: 1. The reproduced result on DeiT-base is much lower than the reported results (80.98 v.s. 81.80). What's the potential reason? DeiT has opensource the training scripts and pretrained weights. If considering the reported results of DeiT as the baseline, SViTE and its variants do not achieve such impressive improvements as stated in the paper.
2. The proposed method is only validated on the DeiT but not validated on more advanced image transformer architecture such as TNT[1], T2T[2], PVT[3], etc. It would be better to validate the effects of the proposed methods (at least S^2ViTE) on more architectures.   


[1] Han, Kai, et al. "Transformer in transformer." arXiv preprint arXiv:2103.00112 (2021). 
[2] Yuan, Li, et al. "Tokens-to-token vit: Training vision transformers from scratch on imagenet." arXiv:2101.11986 (2021). 
[3] Wang, Wenhai, et al. "Pyramid vision transformer: A versatile backbone for dense prediction without convolutions." arXiv preprint arXiv:2102.12122 (2021).
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The author proposes three methods for achieving sparse vision transformers to reduce the number of parameters and FLOP. By using a dynamic structured sparsity injecting method, the authors are able to achieve promising speedup without loss of accuracy. For an example, the author claims to achieve 14.7% run time reduction with 0.42% accuracy improvement. However, the training settings of the author's experiments and the DeiT baseline are different. Thus, there are some ambiguities exits.
main_review: Strength:

1. The paper is well organized and easy to follow.
2. The sparsity in vision transformers is an interesting problem to explore as it could be potentially used to apply ViTs on large input images which otherwise will cause a huge cost of the computation memory.
3. The token selector is an interesting propose to dynamically select the top-k tokens that are more important to a given input sample.

Weakness:

1. The proposed methods are trained with longer epochs than the baseline models (600 VS 300). It has been proven that training with longer epochs will further increase the performance of the ViTs with proper augmentation. Thus, it is not clear if the proposed method is able to achieve better accuracy with less parameters and computations or the method is actually applied on a stronger baseline and the the accuracy is actually reduced from the stringer baseline.

2. The idea of selecting Top-k tokens is interesting but the current analysis is not enough to support its efficiency. There is no ablations on the impacts of the selector. For example, as shown in Fig. 6 in [1], the sparsity of the attention map is increasing from bottom layers to the top layers. As the attention map sparsity is related to the token sparsity, it may be more convincing to show if following a decaying sparsity could better preserve the model accuracy and the proposed method could learned a layer-wise sparsity that is following such kind of observations.

3. Based on the results in Tab. 4 in the paper, it seems that the accuracy drop is more significant on smaller models. Considering that the proposed methods are all trained with 2x longer epochs, it is not clear what is the performance drop on large models given a well trained baseline.

Overall, I agree that injecting sparsity into ViTs is an interesting and meaningful topic to explore. However, based on the current experiments results and ablation analysis, I am not clear the impacts of the proposed method on the accuracy.  I am willing to increase my score if the author could address my concerns sufficiently.

[1] Zhou, D., Kang, B., Jin, X., Yang, L., Lian, X., Jiang, Z., ... & Feng, J. (2021). Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103.11886.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work pioneered to study of various sparse training and pruning techniques to the recently proposed vision transformers (ViT). It shows the effectiveness of pruning/sparse training/data selection in such settings. This is a  interesting work for both ViT and sparse learning communities. 
main_review: Positive 
+	This paper presents a particularly comprehensive study for bringing sparsity into ViTs. Sparsity itself is addressed from various angles, e.g., data sparsity and sparse training connectivity (structured and unstructured). 
+	The authors studied a structured version of sparse training. Unlike CNNs where structural sparsity is often enforced by channel prunin, the authors leveraged a BERT-like idea to defined the importance of self-attention heads, by first-order loss approximation. 
+	The authors experimented with various ViT and DeitT models on ImageNet. Their results are very useful and interesting. Sparsely trained VIT can outperform the dense baseline and other state-of-the-art methods to obtain sparsity. 
+	Overall, this paper establishes so far the most promising trade-off between ViT accuracy and efficiency.  Interestingly, with similar parameter counts, a large sparse ViT consistently outperforms the corresponding smaller dense ViT.
+	The work suggests sparsity conveys not only higher efficiency for transformer, but even sometimes improved accuracy. That potentially will lend a better understanding of the optimization difficulty in training ViTs, which recently received much attention.
+	Figures, algorithms, pseudocodes, and summary-of-takeaway windows help understand this paper.

Negative
-	I am not impressed by SViTE+ results. Look at Table 6, only 5- 10% of tokens can be removed without compromising accuracy. 30% removed would drop accuracy to unacceptable. This is surprising to me, since apparently many tokens would be redundant for making predictions on an image. The author might have to explain why. 
-	Also, SViTE+ seems not compatible with structural sparsity.
-	The abstract highlights two results, both improving test accuracy while saving parameters and running time. Whereas those are good, the authors organized their statements in a quite confusing way, e.g., I was only expecting some counterintuitive results after “Perhaps most surprisingly,..”, but the authors laid another similar-flavor result before that one. That caused a negative reading experience (especially it is in the abstract). 
-	Also, since most results still mildly sacrificed some accuracy, it is inappropriate or misleading to highlight only two rare results with occasional accuracy gains. This looks like an accidental writing inconsistency issue but must be fixed.
-	The paper overall has a quite clear and logical outline. But it, unfortunately, suffers from a number of annoying typos such as “ scarifying” (many times!) and “ typologies”. More careful proofreading would have helped. 

limitations_and_societal_impact: Efficient training and inference can save ML costs, leading to more economic ML deployment in real-world applications.
review_title: null

Meta-review: Four experts reviewed this paper and gave ratings 6, 6, 6, and 7, respectively. The reviewers expressed concerns about novelty, writing, and some details, but they generally appreciated the study of sparsity in Transformers. Some reviewers especially liked the results. AC felt that the concerns can be addressed in a light revision of the paper after reading the rebuttal and the post-rebuttal discussions. Hence, the decision is to recommend the paper for acceptance. The authors are encouraged to make necessary changes in the camera-ready to address the reviewers' questions to the best of their ability. We congratulate the authors on the acceptance of their paper!

Decision: Accept (Poster)