SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 5: Fine-tuning performance drop of OPT-350M at GLUE dataset wrt. dense counterpart on multiple N:M sparsity patterns [80] masks.
various downstream datasets with multiple N:M sparse mask  $(m\cdot \theta_p^{x\%})$ . It can be in general observed essential sparsity holds valid for structured N:M sparsity patterns, and a large fraction of low-magnitude weights can be removed at free without significant drop in downstream performance.
Figure 6: Essential Sparsity and performance comparison ViT-base and DINO-base which share the same architecture but pre-trained using supervised (SL) and self-supervised learning (SSL) objectives. It can be observed that the SSL induces a better sparsification ability in the pre-trained checkpoint.
Figure 7: Layer-wise weight distribution of ViT-base and DINO-base trained using supervised and self-supervised learning objective. Note that the weights of both pre-trained models are normalized using sklearn for fair comparison. Additionally, DINO has  $14.37\%$  more zero weights than ViT.
Essential Sparsity and Sharp Turning Behaviour: In this section, we attempt to bring attention towards the sharp turning point behavior of essential sparsity. Across all our vision and language models and related downstream tasks (Figure 2, 3, & 4), we observed that after a certain removal of a certain proportion of low-magnitude weights, the downstream fine-tuning ability of pre-trained models sharply drops. This is a clear indication that the pre-training knowledge resides in a fraction of high-magnitude weight regions and if we our one-shot pruning touches that region, it non-linearly impacts the transfer learning ability of the pre-trained checkpoint. Our experiments reveal that this sharp-turning behavior is not dependent on model size but on the downstream task. Larger model doesn't implicate that it can be pruned for a higher proportion of low-magnitude weight, without observing the sharp drop. For example, bert-base observes the sharp turning point at around  $60\%$  sparsity while OPT-1.3B can not be pruned beyond  $40\%$  without observing the sharp performance drop on RTE task, although it has  $\sim 10$  times more parameters than bert-base. Also, it is interesting to observe that although OPT-125M and bert-base have similar performance count, bert-base illustrate more friendly behavior to pruning, and can be pruned to comparatively higher sparsity ratio than OPT-125M on all our evaluation downstream tasks.
Influence of Supervised versus Self-supervised Learning Objectives: With the recent success of self-supervised learning (SSL) in pre-training large transformer models and its ability to scale to enormous internet-size data, it is interesting to understand how SSL learning objectives favor sparsity. Due to the unavailability of supervised datasets to pre-train BERT-scale models, we switch to Vision transformers, and use ViT-base and its self-supervised version DINO-base which inherit exactly the same architecture. Figure 6 illustrates fine-tuning performance drop of ViT-base and DINO with an increase in the proportion of low-magnitude weight removal from pre-trained checkpoints. Across all tasks, fine-tuning performance doesn't suffer any drop till  $30 - 40\%$  weights are removed.
One interesting observation is that DINO-base tends to be more robust to low-weight removal, and has comparatively better essential sparsity across all tasks. To further investigate why we can prune
Figure 8: Plot Description in order left-right. (i) Zero-weight count of 5 pre-training experiments of Bert-base using bookcorpus dataset from HuggingFace with varying percentages of randomly selected data volume with exactly the same pre-training setting. (ii) Downstream performance of the pre-trained Bert-base models with varying data volume across different sparsity ratios on MNLI. (iii) Downstream performance of 5 dense pre-trained models on QNLI, QQP, RTE.
DINO-base more than ViT-base, we examine the weight magnitude distribution across the layers of the transformer blocks from the pre-trained checkpoints. Figure 7 presents the normalized layer-wise weight distribution of DINO-base and ViT-base. It can be clearly observed that SSL-based DINO tends to have a weight distribution more friendly to pruning with a significantly large amount of weights concentrated around zero magnitudes. More concretely, we found that DINO-base have  $\sim 14\%$  more zero weights than ViT-base, which justifies its higher essential sparsity.
5 How Essential Sparsity Emerges during the Pre-Training Dynamics
Scaling the volume of pre-training data volume is widely believed to favor transfer performance in many downstream tasks in a desirable way [81]. Although, this relationship has been extensively studied recently in many works [81, 82, 83, 84], pre-training data volume role in inducing sparse properties in the large transformers is still unexplored. In this section, we ask an important question: How does the volume of pre-training data impact the emerging sparse patterns in large transformers, and if it improvises their prunability?
To this end, we designed custom experiments to pre-train bert-base from scratch using HuggingFace bookcorpus with a vocabulary size of 30,522. We created 5 different pre-training datasets by randomly selecting  $100\%$ ,  $90\%$ ,  $80\%$ ,  $70\%$ ,  $60\%$  of the training samples from bookcorpus and pretrain for 50k iteration each to ensure that all models receive the same amount of gradients. Note that we maintain exactly same training settings for all models to retain fair comparison and save the pre-training checkpoints every 1000 iterations. We now enlist our key findings related to pre-training data volume and induced sparse patterns in transformers:
- We observe an interesting new phenomenon of abrupt sparsification, i.e., the introduction of sudden high sparsity, during pre-training bert-base models, regardless of data size.
- We additionally observed a counter-intuitive finding that bert-base trained with a larger amount of pre-training data tends to have better emergence of induced sparsity.
- Across all sparsity level, we found bert-base trained with a larger volume of training data, enjoys better pruningability, and achieve better performance on the downstream task (MNLI).
Figure 8(i) illustrate the emergence of sparse patterns during pre-training of bert-base with different pre-training data volume. We plotted the number of zero weights that emerged in the pre-training checkpoints every 1k iterations. It can be clearly observed that in all settings, the number of zero weights suddenly grow up, indicating the models start abstracting pre-training data knowledge into fewer parameter set. For instance, we find this sharp turn around 22-25k iterations for  $100\%$ ,  $90\%$ ,  $80\%$  settings while around 40k iterations for  $70\%$ ,  $60\%$  settings. In addition, we found that bert trained with a larger amount of pre-training data tends to have better emergence of induced sparsity. We argue that bert-base tend to learn more generalizable features and develop the capability to abstract knowledge. To further investigate how this sparsity emergence further influences the prunability of the pre-trained models, we examined their performance across multiple sparsity ratios (Figure 8(ii)) on MNLI and found it to be in harmony with the magnitude of induced sparsity, i.e., models with high induced sparsity patterns during pre-training tend to perform better when we remove the existing low magnitude weights and fine-tune them on downstream tasks. In dense
Figure 9: Performance difference comparison of fine-tuning of masks identified by LTH and OMP from bert-base (left) and ViT-base (right) across multiple downstream tasks.
settings on QNLI, QQP, RTE (Figure 8(iii)), we found that an increase in pre-training data volume has favorable benefits on the downstream performance with similar fine-tuning hyperparameters.
6 Essential Sparsity and Lottery Ticket Hypothesis
Lottery Ticket Hypothesis deviates from the convention of after-training pruning, and points to the existence of independently trainable sparse subnetworks from scratch that can match the performance of dense networks. Since its introduction, it has been very successful and widely adopted in compressing small-scale (eg. ResNet family) models. However, it is significantly limited in the practical adoption for large pre-trained models due to train-prune-retrain routine of IMP. In this section, we ask: How effective LTH is within the essential sparsity range, and does it bring any additional privilege which can substantiate its computational cost? Our key takeaway can be summarized as:
- Within the essential sparsity range of bert-base and ViT-base, we do not find any significant fine-tuning performance difference of the mask identified by LTH and one by removing lowest magnitude weights (OMP) across multiple downstream tasks.
- We surprisingly found the existence of high cosine similarity  $(>96\%)$  across the masks identified by LTH and OMP within the essential sparsity range.
- Mask similarity between LTH and OMP decreases with an increase in the sparsity ratio. This corroborates with the benefit of LTH in the high sparsity range, where LTH is able to identify task-fitting masks to retain performance due to repetitive prune and retrain routine.
Figure 9 illustrates the performance difference comparison of fine-tuning of masks identified by LTH and OMP from bert-base and ViT-base across multiple downstream tasks. It can be clearly observed that for sparsity ratio below  $50\%$ , we do not find any significant difference in performance between expensive LTH and free-of-cost OMP pruning within the essential sparsity range. A deeper analysis (Figure 10) across the masks from LTH and OMP unveils a surprising observation about the existence of significantly high cosine similarity across. This observation supports the findings in Figure 9 about the matching performance of LTH and OMP and convey a strong and impressive message that with the essential sparsity range, LTH doesn't provide any additional privilege. However, it is important to note that LTH is very effective in the high sparsity range beyond essential sparsity, due to its ability to find task-fitting mask using train-prune-retrain procedure but they tend to be non-transferable across different downstream tasks [20]. Moreover, it can be observed from Figure 10, even in the essential sparsity range, mask similarity across tasks by LTH is comparatively low than OMP based masks due to the strong intertwine of LTH with the downstream tasks.
7 Scaling Essential Sparsity to Modern-Scale LLMs: A Case Study
Large Language Models (LLMs) recently reshape the field of NLP with remarkable performance benefits across a range of complex language benchmarks. However, due to their gigantic size and computational costs; they still remain out of reach for many researchers and small industries. In this section, we investigate the presence of essential sparsity in two popular LLM (Vicuna-7B & 13B) with one-shot magnitude pruning. Note that unlike our previous setting, where we perform sparse fine-tuning after the mask is identified; here we do not perform downstream task-specific fine-tuning and evaluate the performance directly "zero-shot" on the sparsified pre-trained checkpoint.
Figure 11(a) illustrates the performance drop of Vicuna-7B and Vicuna-13B on popular MMLU benchmark (Stem, Humanities, Social Science, Others) when  $x\%$  of the lowest magnitude weights
Figure 10: Cosine similarity between the masks obtained by LTH (depending on downstream task) and OMP on bert-base (Row 1) and ViT-base (Row 2) for sparsity ratio  $s \in \{10\%, 20\%, 30\%, 40\% \}$ . High cosine similarity indicate masks identified by LTH and OMP are significantly similar.
Figure 11: Performance drop of Vicuna-7B and 13B on MMLU benchmark [31] w.r.t. the dense counterpart, using OMP and the recently proposed SparseGPT [85]. This indicates a notable fraction of weights of Vicuna-7B/13B can also be removed at free without any significant drop in performance.
are removed from the dense pre-trained checkpoint. It is interesting to see that our essential sparsity observations hold true even for modern LLMs, sending a favorable signal about the hidden existence of high-quality sparse subnetwork which can be identified free of calibration data or post-optimization, in dense pre-trained checkpoints. To further enrich our study, we replaced OMP with the recently proposed SparseGPT [85] and found it to have generally consistent trends with OMP (Figure 11(b)).
Compared to SparseGPT, our research uncovers the surprising simplicity of LLM pruning within a specific sparsity range. The remarkable effectiveness of one-shot, magnitude-based pruning not only establishes a robust baseline but also suggests considerable practical value, for instance, in enabling economical "on-the-fly" LLM pruning that adapts to fluctuating resource availability during testing. Nonetheless, it's worth noting that more refined pruning strategies like SparseGPT can further push the boundary of essential sparsity and identify better sparse subnetworks at comparatively higher sparsity ratios, albeit at relatively higher expenses. Bridging the performance divide between OMP and SparseGPT remains an intriguing avenue for future investigation.
We comprehensively study induced sparse patterns across large pre-trained vision and language transformers. We experimentally validated the ubiquitous existence of essential sparsity across large pre-trained transformer models of varying scales for vision and language (including LLMs), irrespective of the training strategy used for pre-training them. We also present an intriguing emerging phenomenon of abrupt sparsification during the pre-training of transformers and its ability to abstract knowledge within few parameter subsets with increasing pre-training data volume. Lastly, we studied the performance of LTH with respect to essential sparsity. Our future work will aim to extend our essential sparsity observations to more gigantic models, and to push for higher sparsity ranges.
The research is in part supported by the Intelligence Advanced Research Projects Activity (IARPA) under Contract No. 2022-21102100004. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of IARPA or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.
[2] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
[3] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, Zhaohui Yang, Yiman Zhang, and Dacheng Tao. A survey on visual transformer. ArXiv, abs/2012.12556, 2020.
[4] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv'e J'egou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
[5] Zhiyuan Mao, Ajay Jaiswal, Zhangyang Wang, and Stanley H. Chan. Single frame atmospheric turbulence mitigation: A benchmark study and a new physics-inspired transformer model. ArXiv, abs/2207.10040, 2022.
[6] Minghang Zheng, Peng Gao, Renrui Zhang, Xiaogang Wang, Hongsheng Li, and Hao Dong. End-to-end object detection with adaptive clustering transformer. ArXiv, abs/2011.09315, 2021.
[7] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam M. Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In ICML, 2018.
[8] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
[9] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
[10] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937, 2018.
[11] Ajay Jaiswal, Liyan Tang, Meheli Ghosh, Justin Rousseau, Yifan Peng, and Ying Ding. Radbert- cl: Factually-aware contrastive learning for radiology report classification. Proceedings of machine learning research, 158:196-208, 2021.
[12] Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. End-to-end open-domain question answering with bertserini. arXiv preprint arXiv:1902.01718, 2019.
[13] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.
[14] Ming Ding, Chang Zhou, Qibin Chen, Hongxia Yang, and Jie Tang. Cognitive graph for multi-hop reading comprehension at scale. arXiv preprint arXiv:1905.05460, 2019.