The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter
Authors: AJAY KUMAR JAISWAL
Shiwei Liu
Tianlong Chen
Zhangyang Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Pre-trained Models
Sparsity
Emergence
Transformers
Pruning
TL;DR: Our work study induced sparse patterns across multiple large pre-trained vision and language transformers, and preposes the existence of essential sparsity.

Abstract: Large pre-trained transformers are $\textit{show-stealer}$ in modern-day deep learning, and it becomes crucial to comprehend the parsimonious patterns that exist within them as they grow in scale. With exploding parameter counts, Lottery Ticket Hypothesis (LTH) and its variants, have lost their pragmatism in sparsifying them due to high computation and memory bottleneck of repetitive $\textit{train-prune-retrain}$ routine of iterative magnitude pruning (IMP) which worsens with increasing model size. In this paper, we comprehensively study $\textit{induced sparse patterns}$ across multiple large pre-trained vision and language transformers. We propose the existence of -- $\textbf{essential sparsity}$ defined with a $\textbf{sharp dropping point}$ beyond which the performance declines much faster w.r.t the rise of sparsity level, when we directly remove weights with the smallest magnitudes in $\textbf{one-shot}$. We also present an intriguing emerging phenomenon of $\textbf{abrupt sparsification}$ during the pre-training of BERT, i.e., BERT suddenly becomes heavily sparse in pre-training after certain iterations. Moreover, our observations also indicate a $\textbf{counter-intuitive}$ finding that BERT trained with a larger amount of pre-training data tends to have a better ability to condense knowledge in comparatively relatively fewer parameters. Lastly, we investigate the effect of the pre-training loss on essential sparsity and discover that self-supervised learning (SSL) objectives trigger stronger emergent sparsification properties than supervised learning (SL). All our codes will be publicly available.

The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter
Ajay Jaiswal', Shiwei Liu!?, Tianlong Chen', Zhangyang Wang' 'University of Texas at Austin, 'Einhhoven University of Technology {ajayjaiswal, shiwei. liu, tianlong. chen, atlasvang)Outexas edu
Abstract
Large pre-trained transformers are show-stealer in modern-day deep learning, and it becomes crucial to comprehend the parsimonious patterns that exist within them as they grow in scale. With exploding parameter counts, Lottery Ticket Hypothesis (LTH) and its variants, have lost their pragmatism in sparsifying them due to high computation and memory bottleneck of repetitive train-prume-retrain routine of iterative magnitude pruning (IMP) which worsens with increasing model size. This paper comprehensively studies induced spurse palterns across multiple large pre-trained vision and language transformers. We propose the existence of "essential sparsity" defined with a sharp dropping point beyond which the performance declines much faster w.r.t the rise of sparsity level, when directly remove weights with the smallest magnitudes in one-shot without re-training. We also find essential sparsity to hold valid for N:M sparsity patterns well as on modern-scale large language models (Vicuna-7B/13B) We also present intriguing emerging phenomenon of abrupt sparsification during the pre- training of BERT, i.e., BERT suddenly becomes heavily sparse in pre-training after certain iterations. Moreover, our observations also indicate counter-intuitin finding that BERT trained with a larger amount of pre-training data tends to have a better ability condense knowledge in comparatively relatively fewer parameters. Lastly, we investigate the effect of the pre-training loss essential sparsity and discover that self-supervised learning (SSL) objectives trigger stronger emergent sparsification properties than supervised learning (SL). Our codes are available at /ttpps://githann com/VITA http/VVTAAGroou///ooe wrw.poeessentiiiiii essentiall aalaallsssstttttaaaa
Introduction
Enormous increases in scale often permeate systems with unique new behavior. Transformers [ swiftly after their introduction scaling every day, dramatically improving the state-of-the-art per- formance on a wide array of real-world computer vision 2 3 6 7. natural language processing
10 12 [4 [5] [6] applications and leaderboards. With the astonishing explosion of parameter counts (millions to billions) in the past few years, while chasing performance gains, fine- tuning these large pre-trained models with non-industry standard hardware becoming seemingly impossible, in addition to expensive inference and steep environmental cost. For instance, GPT-3
8
17
contains 175 billion parameters and requires at least five 80GB A A100 GPUs for inference [18]
In the hustle of building gigantic models, a parallel and growing field of model compression has been exploring the prospects to compress these gigantic models at the cost of marginal/no sacrifice in performance. Among many efforts for compressing models and accelerating inference, network pruning [19 20
25 261, which removes the least important components from the model, varying from the low granularity (e.g., individual weights) to the high granularity (such as blocks, filters, and attention heads), stands out of the most effective techniques. Despite the enormous success of the Lottery Ticket Hypothesis (LTH) 27 and its variants for small-scale neural
21
22
23
24
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

networks, there still exists large gap in its practical adoption for large preOtrained transformers with billions of parameters because of the fully dense training routine of iterative magnitude pruning (IMP), which exacerbates with an increase in model capacity. Recently, many works 28
29
30 20)
are investigating strategies to make IMP pragmatic for large pre-trained models. However, they still rely on train-prune-retrain routine of IMP and downstream tasks to identify subnetworks. While chasing for gold-standard sparsity masks with computationally expensive approaches, it is critically important to understand pre-existing high-quality sparse patterns in these large transformers.
In our work, we reveal and study an intriguing, previously overlooked property of large pre-trained transformers "eessential sparsity". In plain words (formal definition in Sec. B3. we define essential sparsity as sharp dropping point, beyond which the fine-tuning performance after one-shot pruning declines much faster w.r.t. the sparsity level rise. We directly remove weights with the smallest magnitudes in one-shot from the pre-trained checkpoints, thereby the identified subnetwork mask requires no extra computational overhead to spot and remains identical across all downstream tasks.
Essential sparsity is an important yet understudied induced property of large pre-trained transformers. indicating that at any time, a significant proportion of the weights in them can be removed free of any calibration data or post-optimization, although the proportion may vary depending on the complexity of the downstream task. One important practical implication conveyed by this observation, as overlooked by prior work, is summarized as: within the sparsity range induced by "essential sparsity", ", the simplest possible pruming technique as aforementioned performs the same well as any fancy technique such as LTH, and even their identified sparse masks are highly similar.
Our work, for the first time, conducts a comprehensive study of the existence of essential spar- sity across multiple vision and language transformers with varying scales and training strategies. Besides, we observe essential sparsity to hold valid for various N M sparsity patterns with hard- ware speedup potentials. In addition, our experiments using the popular MMLU benchmark [31) on Vicuna-7B// -BB/ma illustrate essential sparsity observations remain true for modern-scale large language models (LLMs), indicating the existence of high-quality sparse subnetworks within dense pre-trained checkpoint.
Next, we study the emergence of those sparsification properties during the pre-training dynamics, using BERT as the focused subject of study. We found an intriguing phenomenon of abrupt sparsification, i.e., BERT suddenly becomes heavily sparse after certain number of training iterations. As we vary the pre-training dataset size. our observation indicates another counter-intuitive finding that BERT trained with larger amount of pre-training data tend to have better ability condense knowledge in relatively fewer parameters. We also dive deep into the effect of the pre-training loss on essential sparsity and discover that self-supervised learning (SSL) objectives trigger stronger emergent sparsification properties than supervised learning (SL).
Key contributions for our work can be summarized as:
We found the ubiquitous existence of essential sparsity across large pre-trained transformer models of varying scales for vision and language, irrespective of the training strategy used for pre-training them. High-quality sparse masks comparable lottery tickets [27] within the essential sparsity range can be spotted free of inference or ppst-optimizatio by merely selecting the lowest magnitude weights, that requires no expensive repetitive train-prunce retrain routine. The observation holds true for both unstructured and N : sparsity patterns.
Our comparison of the sparse masks obtained by selecting the lowest magnitude weights with the lottery tickets within the essential sparsity range, surprisingly unveils a notably high cosine similarity (>98%) across various downstream tasks from NLP and CV. This observation sends a strong message that in large pre- rre-trained models, LTH perhaps enjoys little additional privilege despite utilizing enormous computational costs.
While studying the emergence of sparsification properties during the pre-training dynamics in BERTs, found an intriguing phenomenon of abrupt sparsification, i.e., BERT suddenly becomes heavily sparse after certain number of training iterations. We additionally found a counter-intuitive observation that BERT pre- pre-trained with larger amount of data tends to be more sparser, i.e., they become better at knowledge abstraction with fewer parameters.
We additionally study the effect of the pre-training loss on the emerging sparsity of FMs. When we switch between supervised learning (SL.) and self-supervised learning (SSL.) objectives on VIT, we observed that SSL tends to have better emergent sparsification
2

---

properties, thereby more friendly to pruning. We further provide layer-wise visualization to understand what the sparsity learned by SSL vs SL looks like.
2 Related Work
Sparse Neural Networks (SNNs). Sparsity in deep neural networks is usually introduced by model compression 32 33 which removes the redundant parameters. Based on the type of sparsity patterns,
can be categorized into two families: (i) unstructured sparsity [33] 34 33] where the non-zero elements irregularly distributed; (it) structured sparsity [36
38 where group of parameters is eliminated like a convolutional kernel in convolutional neural networks or an attention head in the transformer. In general, the former sparsity pattern obtains better performance thanks to its flexibility, while the latter sparse pattern tends to be more hardware friendly. Many important SNN works start by studying the former and then turn the latter as a special case.
37
Meantime, according to the timing to perform dimensionality reduction, sparsity can be obtained in the post-training, during-training, and pursue inference time efficiency, trained models can be pruned dramatically with marginal loss of performance with respect to certain heuristies, including zero-order [33), first-order (39 40 41], and
of deep neural networks. (1) Post- Post-Training. To
approach (e.g. iterative magnitude pruning) is a popular option. Especially, it is frequently adopted by the Lottery Ticket Hypothesis [19 to produce sparse NNs with undamaged trainability and expressiveness. (ti) During-Training. On the contrary to pruning well trained model for inference efficiency, during- training sparsification [44] also enjoys computational savings for model training. It normally first optimizes a dense network for several iterations, then gradually sparsifies it with pre-defined schedule, and finally leads to lightweight sparse NNs. For example, [45] 46 leverage Eo or & penalties to encourage weight sparsity during the network training. hoping to zero out unimportant parameters. [48] 491
second-order 33
43 criteria. Among these algorithms, the weight magnitude-
cast it as an optimization problem with reparamcterization and bi-level forms,
respectively. Another fashion is dynamic sparsity exploration (50 51 52 53 54 55 56 57 58 59]
which allows pruned connections can be re-activated in the latter training stage. (iii) IntorTTriniin.. Identifying the critical sparsity patterns at the initialization is one exciting rising sub-field.
determines the sparse connectivities in a very early stage, like one iteration [25] 60] or a few
iterations 61 62]. In this paper, we mainly investigate post-training unstructured SNNs.
Sparsity in Pre-Trained Transformers. Pre-trained Transformers have become de facto choice for numerous applications of natural language processing (NLP) [8] 9 I0 [5] [6] and computer vision 2
Their impressive performance partially credited to their tremendous learning capacity empowered by huge amounts of model parameters. Unfortunately, such successes come with burning thousands of GPUs/TPUs for thousands of hours (64 17 even for a single round of model training. To address this resource-intensive concern and improve the affordability of these transformers, plenty of pioneering researchers devote themselves in this area 440 20 30 65 66
67 68 69 700. Rather than proposing new sparsification method, this paper reveals the blessings
of induced essential sparsity during the pre-training and how we can capitalize it to prune large pre-trained models without any computational overhead.
3 Experimental Settings
Network and Pre-Trained Transformers. We consider (BERT 64
BERT BERT Large [64].
OPT (T], 3PT (7T), and 0PT1.3B (7I)) and {ViTBase 22. Large DiNOBase
[2]. and
6 for NIP and CV applications respectively. Their officially pre-trained model are adopted in our experiments. To be specific, let f(;; 0p) be the output of a transformer with pre-trained model parameters 0, and input sample x.
Datasets, Training, and Evaluation. For downstream tasks in NLP, consider (MNLI, QNLI, QQP, SST-2) from GLUE [3), RTE from SuperGLUE 72), and SQUAD v1.1 77). As for vision downstream applications, use the Arithmetic Reasoning task from the recently proposed SMC-Benchmark 75 and consider
examine (CIFAR-10, CIFAR- Tiny-ImageeNet 774) We also
popular math word problem datasets: (1) the widely used MAWPS benchmark 776 composed of 2,373 problems; (2) the arithmetic subset of ASDiv-A 77 with 1 1,218 math problems; (3) the more
'Check supplementary materials for pre-trained model details.
3

---

DONGO VAN
(Seara Waggee
Quarre Coparety Whegte
60
Figure 1: Naturally induced sparsity patterns of bert bastt-bse---ae uncased across the components of transformer blocks. The pre-trained model is pruned by 21 2105 using one-shot-magnitude pruning. Yellow dots indicate the location of pruned low- -o--aanit weights.
Table 1: Downstream tasks fine-tuning details. Learning rate decay linearly from initial value to 0.
Sertings
Natural Langiage Progessing
Computer Vision
MNLI
QNLI
QQP
RTE
SST-2
SQuAD
CIFAR-I0
CIFAR-100
Fashion MNIST
TmageeeeeNNet
Training
392,701
104,768
363,872
2.196
67,360
15,000
45,000
55,000
90,000
Epoch
5
Batch Size
32
32
32
32
32
16
64
64
64
64
Learning Rate
2c
20
5
20
5
20
20
5
30
5
2c
2r
5
20
6
2
5
Optinizer
AdamnW decay (0)
10-1
AdamnW with deeay (o)
10-1
Pval. Metric
Musched Ace.
Accuracy
F1-score
Accuncy (Top-1)
challenging SVAMP (78 dataset which is created by applying complex types of variations to the samples from ASDiv-A. The task difficulty monotonically increases from MAWPS to ASDiv-A, and to SVAMP. We adopt the default dataset split for training and evaluation for our downstream application. More detailed configurations are collected in Table [ followed the settings proposed in the official implementation
For SMC-benchmark, we strictly
Sparse Neural Networks (SNNs). The weights of a SNN can be depicted as m @ 9, where m E 1).1) is a binary mask with the same dimensionality as 0 and @ is the elment-wise product. Let 8T (f(z; 0)) denotes the evaluation function of model outputs f(z; () on the corresponding task T (which might involve fine-tuning). Pp (p(.) is the sparsification algorithm which turns a portion p of "1" elements in the sparse mask m into "O's. Here is formal definition of Essential Sparsity below.
Essential Sparsity. If ET (f(a; e 0)) 2 ET (f(=;)) c, and ET (f (m(m) C 0)) ET (T( 0)) -c e where the value of and e are small. Then, the according sparsity ||mllo is named as Essential Sparsity for the model f on task T.
Im
As detailed above, the model at the essential sparsity usually has a turning point performance, which means further pruning even a small portion of weights leads to at least c performance drop, compared to its dense counterpart ET (f(x; 8)). In our case, 6 is set as 1%. In plain language, the turning point of essential sparsity defines sparsity range with two characteristics: (i) within this range, the one-shot pruned model performs as well as or better than dense baseline, without re-training; (ii) beyond this range, notable accuracy drop becomes observable after one-shot magnitude pruning.
Pruning Methods. To find the desired sparse mask m. we use two classic weight magnitude pruning algorithms 33 19]. One-shot Magnitude Pruming (OMP): we directly eliminate a pre-defined portion of parameters from Op with the least absolute magnitude. Lottery Tickets Pruning (LTP): (i) we first train the model to converge on a downstream task T; (1i) then remove a portion of the smallest weights and reset the remaining weight to their initialized value from pre-training 8p: (tit such processes will be repeated until reaching the desired sparsity.
Two crucial facts are noteworthy. First, starting from the pre-trained model, unlike LTP which seeks nownstream-speeiff. masks with additional training, the OMP sparse masks require no additional training to identify and are agnostic to downstream applications. Since they are estimated directly from the pre-trained weight checkpoint, the OMP sparse mask remains the same for all downstream applications (we only continue to fine-tune the non-zero weights with the same mask, over different downstream datasets), Second, for the same above reason, a higher-sparsity OMP mask is naturally nested within lower-sparsity mask, as long as they are pruned from the same pre-trained checkpoint using OMP. The same nested property does not hold for LTP-obtainer masks.
PMMC Benchmark tuning setting details: Arithmetic Reasoning

---

MALI Masthe Facurity
Sparsty
CHA AODZANO D
Sporsty
Q22 SCc-unccn Drop
Sporsty
SST Accuricy Crop
EspITRE Spersit
Esseraa Sparnhy
Exseral Sperety
SQ Drop
RIC Accunic
BAR BUSE
Beri BasE
Benl Baze
OPT-1.38
OFT-1.36
Figure 2: Fine-tuning performance drop estimated with respect to dense counterpart for various downstream tasks of NLP pre-trained models (bert-base, OPT-125m, OPT-350m, OPT-1. (B)). Note that for fair evaluation, have used exactly same fine-tuning settings across all pruning ratios.
Tayy aageeeee Top Acturazy Drop
Drop
CHH-M-1 Actuncyy
Essential Sparsily
Espentisl Sponsly
Esseniaa Sparsty
-10
1:0 Top
-10
-15
Vi-Baaee
VIT-Base VIT-Large
Vil-Dase VIT-Large
VIT-Lärge
Figure 3: Fine-tuning performance drop estimated with respect to dense counterpart for various downstream tasks of CV pre-trained models (ViT-base & ViT-large).
4 Essential Sparsity Exists in Pre-trained Language and Vision Models
The unconstrained growth in parameters has resulted in significant computational costs and excessive memory demands. The trend undoubtedly continues with transformers on the forefront, where more and more layers are stacked with dense attention blocks (eg. GPT-3 has surprisingly 175 billion parameters) necessitating substantial compu- tational resources and extended training or fine-tuning durations. Unlike extensive efforts explored in the past for pruning ResNets and related families of networks, pruning large-scale pre-trit pre-trained transformer models hasn't received enough attention.
Revisiting sparsity in Pre-trained Transformers:
Lottery Ticket Hypothesis (LTH) based approaches have been recently explored to prune BERT-bas. size pre-trained models. However, they lose their pragmatism due to the expensive dense train- main--rune- retrain routine of IMP and non-transferable characteristics of identified subnetworks across similar tasks [20). oBERT [66] proposes second-order to BERT-level scale by allowing for pruning blocks of weights. Note that, although these methods provide interesting approaches to prune BERT scale transformers, they fail to sufficiently recognize the strength of pre-existing, more easily accessible high-quality sparse patterns in pre-trained large models invariant of the scale, training data modality, and strategy. This paper's s primary goal is to bring the sparse community's 's attention towards the strength of ubiquitously induced sparse patterns during the pre pre-training of large transformers and encourage them to effectively capitalize it during the design of more sophisticated pruning algorithms.
Essential Sparsity: As defined in Section 3 essential sparsity indicates the presence of naturally induced sparse structures in large pre-trained models with respect the lowest magnitude weights. Figure represents the induced sparsity distribution of bert uncased pre-trained model with 21. 21.50% close to zero weights, which we have been able to remove without ANY drop performance across any of our evaluation downstream datasets. This sought a strong message about the existence of free, universal, and available to consume sparse mask (m 0p) without any computational overhead induced by sophisticated pruning algorithms. In addition, a closer observation of the distributed sparsity ratios across different modules of the transformer block indicates that the majority of the low-magnitude weights are concentrated in the dense feed-forward layers. Such findings conform
5

---

with the success of Mixture-of-Exee Expertss [79 and provide cues for future development of pruning algorithms exploit the over- vrreprramettiiinionnn of dense ceed-forward layers.
We now enlist our key findings related to essential sparsity in large pre-trained transformer models:
In all vision and language models, we find the existence of essential sparsity and the sharp turning point of the munsstt-performancee curve.
The sharp turning point of essential sparsity is downstream task-dependent and can vary depending on the task complexity.
The sharpness behavior essential sparsity is more profound in vision pre-trained models in comparison with language models.
Essential sparsity holds valid for recently proposed pruning benchmark SMC-bench 77
Essential sparsity holds valid for N:M structured sparsity patterns (80 with potential speedup.
Self-supervised learning objective triggers stronger emergent sparsification properties than supervised learning in models having exactly the same architecture.
Universal Existence of Essential Sparsity: We com- prehensively evaluate the extent to which essential spar- sity exists in the large pre-trained language models (CV and NLP) with a standard pre-trained initializa- tion p.. For NLP, studied bert-base OPT-125M, OPT-350M, OPT-1. and used MNLI, QNLI, QQP, SST-2, RTE, and SQuAD1. 1 estimate the effect the removing low-magnitude weights on downstream performance. On the other hand, for CV, we rely ViT-Base and ViT-Large models and use CIF 10-1, CIFAR-100, TinyImageNet datasets. Figure 2 illus trate the performance drop (y-axis) of various NLP downstream task fine-tuning with mask (m--02 (m-0=%%) when we remove 5% of lowest magnitude weights. Similarly, Figure EigureBlilusse Billustrates the effect of pruning x% low-magnitude weights the fine-tuning performance of CV downstream tasks. Moreover, preseets the existence of essential sparsity for a recently proposed sparsity benchmark, named SMC-Bench [75]. is interesting to observe that essential sparsity exists in all CV and NLP pre-trained models, and ~ 30 50% of weights can be removed free without any significant drop in performance. Note that these masks are identified prior to fine-tuning, on the pre-traine trained weights and thereby remain the same for all the downstream tasks, indicating no requirement for bookkeeping LTH-type task -ase-spec masks. It is also important to appreciate the significance of removing ~ 40% (which translates to > 500 million parameters) of O.T-1.3B no cost without any significant performance drop.
Performance drop
Essenbal Sparsity
MAWPS
ASDiv-A
SVAMP
0 10
20 30
40
50
60
70
80
90
Figure Fine-tuning performance drop of bert-base on Arithmetic Reasoning datasets in the SMC-benchmark (75).
Eseential Sparsity for Structured N:M Sparse Patterns: Considering the demand for practi- cal speedup, we also conduct evaluations to un- derstand the essential sparsity for the hardware- friendly structured N:M sparsity. A neural net- work with N:M sparsity satisfies that, each group of M consecutive weights of the network. there are at most N weights have non-zero val- ues. We studied 0PT-350M and used MNLI, QNLI, RTE, SST-2 to estimate the effect of removing low-magnitude weights in structured N:M fashion following 80] Figure Fllluuutase the performance drop (y-axis) of 0PT-350M
Ins.
500 Mil Falaen Shans
Fine-tuning performance drop of 0PT-350M at GLUE dataset wrt. dense counter- part on multiple N:M sparsity patterns [8] masks.
Figure 5:
various downstream datasets with multiple N:M sparse mask (m 85%) It can be in general ob- served essential sparsity holds valid for structured N:M sparsity patterns, and a large fraction of low-magnitude weights be removed at free without significant drop in downstream performance.

---

Accorayy Drop
lnngingeeeee top- Pors- Drop
Essental Spaesiyy
Esoinial Sparrtty
Eseerdal Sparsit
CIARR 10 Top A.ccuu
-10
-30
-1
Vi Mi-teee-atchhl
Vit Bost patcht
Vit ther patchts
DIN DiNC-Beee-aicc16
ONO--est-piichh
PINO-Boe-pucii
Figure 6: Essential Sparsity and performance comparison ViT-base and DINO-base which share the same architecture but pre-trained using supervised (SL) and self-supprvised learning (SSL) objectives. It can be observed that the SSL induces better sparsification ability in the pre-trained checkpoint.
Loger
Loges
Aguanneat
Ligas
105
Sejemoan
307
10P
112
31
107
107
107
10
107
106
30
101
101
102
107
Layer
Siperredd
107
Layer
10
107
Lagus
108
Liget
10
107
102
10P
10
10a
3.3
0.1
3.0
12.3
-300 11.5
0203
0.5
03.3
Figure 7: Layer-wise weight distribution of ViT- and DINO-base trained using supervised and self-supervised learning objective. Note that the weights of both pre-trained models are normalized using sklearn for fair comparison. Additionally, DINO has 14.37% more zero weights than VIT.
Essential Sparsity and Sharp Turning Behaviour: In this section, we attempt to bring attention towards the sharp turning point behavior of essential sparsity. Across all our vision and language models and related downstream tasks (Figure 2 3 & (4), we observed that after a certain removal of certain proportion of low- low-magnitude weights, the downstream fine-tuning ability of pre-trained models sharply drops. This is clear indication that the pre-training knowledge resides in a fraction of high-magnitude weight regions and if we our one-shot pruning touches that region, it non-linearly impacts the transfer learning ability of the pre- traiiriddd checkpoint. Our experiments reveal that this sharr-turning behavior is not dependent on model size but the downstream task. Larger model doesn't implicate that it can be pruned for a higher proportion of low- magnitude weight, without observing the sharp drop. For example, bert-base observes the sharp turning point at around 60% sparsity while 1.T-1.3B can not be pruned beyond 40% without observing the sharp performance drop RTE task, although it has 10 times more parameters than bert-base. Also, it is interesting to observe that although 0PT-125M and bert-base have similar performance count, illustrate more friendly behavior pruning, and can be pruned to comparatively higher sparsity ratio than 0PT-125M on all our evaluation downstream tasks.
Learning Objectives: With the recent success of self-supervised learning (SSL) in pre-training large transformer models and its ability to scale to enormous internet-size data, it is interesting to understand how SSL learning objectives favor sparsity. Due to the unavailability of supervised datasets to pre-train BERT-scale models, we switch to Vision transformers, and use ViT-base and its self-supervised version DINO- base which inherit exactly the same architecture. Figure illustrates fine-tuning performance drop of ViT-base and DINO with an increase in the proportion of low-magnitude weight removal from pre-trained checkpoints. Across all tasks, fine-tuning performance doesn't suffer any drop till 30- 40% weights are removed.
Influence of Supervised versus Self-supervised
One interesting observation is that DINO-base tends to be more robust to low-weight removal, and has comparatively better essential sparsity across all tasks. To further investigate why we can prune
7

---

1100%
PAN Monechi Acsuannc Drop A4
Daen Viiae Vikie: 70%
7emn Welgge Count
05000
50000
00000
3.
8
40403
40000
2
20
50
10
20
30
60 Sparsity Rib0
80
90
70
80
80
OCP (MT (Arz))
ATE (Aos)
Traiiig Beratons
Figure 8: Plot Description in order left-right. (i) wero- count of 5 pre-training experiments of Bert- base using bookcorpus dataset from HuggingFace with varying percentages of randomly selected data volume with exactly the same pre- pre-training setting. (ïi) Downstream performance of the pre-trained Bert-base models with varying data volume across different sparsity ratios on MNLI. (iii) Downstream performance of dense pre-trained models on QÑLI, , QQP, RTE.
DINO- base more than the transformer blocks from the pre-trained checkpoints. Figure presents the normalized layer-wise weight distribution of DINO-base and ViT-base. It can be clearly observed that SSL-based DINO tends to have weight distribution more friendly to pruning with significantly large amount of weights concentrated around zero magnitudes. More concretely, we found that DINO-base have 14% more zero weights than ViT-base, which justifies its higher essential sparsity.
we examine the weight magnitude distribution across the layers of
5 How Essential Sparsity Emerges during the Pre- Pre-Training Dynamics
Scaling the volume of pre-training data volume is widely believed favor transfer performance in many downstream tasks in a desirable way 81 Although, this relationship has been extensively studied recently many works properties in the large transformers is still unexplored. In this section, we ask an important question: How does the volume of pre- pre-training data impact the emerging sparse patterns large transformers, and if it improvises their prunability?
81
pre-training data volume role in inducing sparse
To this end, designed custom experiments pre-train bert-base from scratch using HuggingFace bookcorpus with vocabulary size of 30,522. We created 5 different pre-training datasets by randomly selecting 100%, 90%, 80%, 70%, 60% of the training samples from bookcorpus and pre- train for 50k iteration each to ensure that all models receive the same amount of gradients. Note that we maintain exactly same training settings for all models retain fair comparison and save the pre-training checkpoints every 1000 iterations. We now enlist our key findings related to pre-training data volume and induced sparse patterns transformers:
We observe an interesting new phenomenon of abrupt sparsification, i.e., the introduction of sudden high sparsity, during pre-training bert-base models, regardless of data size.
We additionally observed a counter-intune inntuir-i finding that bert base trained with a larger amount of pre-training data tends to have better emergence of induced sparsity.
Across all sparsity level, we found bert-base trained with a larger volume of training data, enjoys better prunability, and achieve better performance on the downstream task (MNLI).
Figure illustrate the emergence of sparse patterns during pre-training of bert with different pre-training data volume. We plotted the number of zero weights that emerged in the pre-training checkpoints every 1k iterations. It can be clearly observed that in all settings, the number of zero weights suddenly grow up. indicating the models start abstracting pre-training data knowl- edge into fewer parameter set. For instance, we find this sharp turn around 22-25k iterations for 100%, 90%, 80% settings while around 40k iterations for 70%, 60% settings. In addition, we found that bert trained with a larger amount of pre-training data tends to have better emergence of induced sparsity. We argue that bert-base tend learn more generalizable features and develop the capa- bility to abstract knowledge To further investigate how this sparsity emergence further influences the prunability of the pre-trained models, we examined their performance across multiple sparsity ratios (Figure(3tin ie., models with high induced sparsity patterns during pre-training tend to perform better when remove the existing low magnitude weights and fine-tune them downstream tasks. In dense
on MNLI and found it to be in harmony with the magnitude of induced sparsity,

---

2.5
Per (OMP) Per (LTH))
-2.5
MNLI
-2.
Per (OMP)
-0.0
QNLI
-9.0
CIFAR-10
QQP
CIFAR-100
-75
-7.5
SST-2
Fashion-MINST
SQUAD
-10.0
TirylmageNet
50
Sparsity Ratio
Sparsny Rati
Figure 9: Performance difference comparison of fine-tuning of masks identified by LTH and OMP from bert-base (left) and ViT-base (right) across multiple downstream tasks.
settings on QNLI, QQP, RTE (Figure fii)), we found that an increase in pre-training data volume has favorable benefits on the downstream performance with similar fine-tuning hyperparameters.
6 Essential Sparsity and Lottery Ticket Hypothesis
Lottery Ticket Hypothesis deviates from the convention of training pruning, and points to the ex- istence of independently trainable sparse subnetworks from scratch that can match the performance of dense networks. Since its introduction, it has been very successful and widely adopted compressing small-scale (eg. ResNet family) models. However, it is significantly limited in the practical adoption for large pre-trained models due to wrii--ruue--eervle routine of IMP. In this section, we ask: How effective LTH within the essential sparsity range, and does it bring any additional privilege which can substantiate its computational cost? Our key takeaway can be summarized as:
Within the essential sparsity range of bert-base and ViT-base, we do not find any significant fine-tuning performance difference of the mask identified by LTH and one by removing lowest magnitude weights (OMP) across multiple downstream tasks.
We surprisingly found the existence high cosine similarity(> 96%) across the masks identified by LTH and OMP within the essential sparsity range.
Mask similarity between LTH and OMP decreases with an increase in the sparsity ratio. This corroborates with the benefit of LTH in the high sparsity range, where LTH is able to identify task- fitting masks to retain performance due to repetitive prune and retrain routine.
Figure 9 illustrates the performance difference comparison of fine-tuning of masks identified by LTH and OMP from bert-base and ViT-base across multiple downstream tasks. It can clearly observed that for sparsity ratio below 50%, we do not find any significant difference in performance between expensive L.TH and free-of-cost OMP pruning within the essential sparsity range. A deeper analysis (Figure T0) across the masks from LTH and OMP unveils surprising observation about the existence of significantly high cosine similarity across. This observation supports the findings in Figure abbuut the matching performance of LTH and OMP and convey strong and impressive message that with the essential spansity range, LTH doesn't I provide any additional privilege. However, that LTH very effective in the high sparsity range beyond essential sparsity,
it is important due to its ability to find task-fitting mask using train-prune-retrain procedure but they tend to be non con-transferable across different downstream tasks (20). Moreover, can be observed from Figure 10 even in the essential sparsity range, mask similarity across tasks by LTH is comparatively low than OMP based masks due the strong intertwine of LTH with the downstream tasks.
7 Scaling Essential Sparsity to Modern-Scale LLMs: A Case Study
Large Language Models (LLMs) recently reshape the field of NLP with remarkable performance benefits across a range of complex language benchmarks. However, due to their gigantic size and computational costs; they still remain out of reach for many researchers and small industries. In this section, we investigate the presence of essential sparsity in two popular LLM (Vicuna-7B & 13B) with one-shot magnitude pruning. Note that unlike our previous setting, where we perform sparse fine-tuning after the mask is identified; here we do not perform downstream task-specific fine-tuning and evaluate the performance directly "zero-shot" on the sparsified pre-trained checkpoint.
Figure TTaa a) illustrates the performance drop Vicuna-7B and Vicuna-13 -13B on popular MMLU benchmark (Stem, Humanities, Social Science, Others) when x% of the lowest magnitude weights

---

Apaanityy Rodio
Sepuriity Rodio
Scuntry Ror
LSG00
com
UPV qua
coes
Lave
COCD
LICO
Cane
EXNOT
com
1.000
COO
CAO
coo
OWN
17% 11-FFemmeep
Ufis reppent
can
kiOW
Coice
View
4.000
Figure 10: Cosine similarity between the masks obtained by LTH (depending downstream task) and OMP on (Row 1) and ViT-base (Row 2) for sparsity ratio E {10% 20%, 30%, 40% High cosine similarity indicate masks identified by LTH and OMP are significantly similar.
Magnitude
SparseGPT
Magnitude Vicuna-133
SperseGPT
Essendial Sparsity
Eccential Sparsity
Eccential Sparsity
Performance Drop
Escentaal Sparsity
Performance
STEW
Ilrranttees
itummnttie
Ilumantties
Duranniee
Social
Sclence Others
Others
Othe's
29 in 14 DEGG
12 it
2915 in 2010 Sparsity
Sparsity
Sparsity Ratio
Figure 11: Performance drop of Vicuna-2 7B and 13B MMLU benchmark 33 w.r.t. the dense coun- terpart, using OMP and the recently proposed SparseGPT 855. This indicates a notable fraction of weights of Vicuna-7B/ 13B also be removed at free without any significant drop in performance.
are removed from the dense pre-trained checkpoint. It is interesting to see that our essential sparsity observations hold true even for modern LLMs, sending a favorable signal about the hidden existence of high-quality sparse subnetwork which can be identified free of calibration data or post-optimization, in dense pre-trained checkpoints. To further enrich our study, we replaced OMP with the recently proposed SparseGPT
85 and found it to have generally consistent trends with OMP (Figure 11 b)).
Compared to SparseGPT, specific sparsity range. The remarkable effectiveness of one-shot, magnitude-basee pruning not only establishes a robust baseline but also suggests considerable practical value, for instance, in enabling economical "on-the-ffy" LLM pruning that adapts to fluctuating resource availability during testing. Nonetheless, it's worth noting that more refined pruning strategies like SparseGPT can further push the boundary of essential sparsity and identify better sparse subnetworks at comparatively higher sparsity ratios, albeit at relatively higher expenses. Bridging the performance divide between OMP and SPT remains an intriguing avenue for future investigation.
research uncovers the surprising simplicity of LLM pruning within
Conclusion
We comprehensively study induced sparse patterns across large pre-trained vision and language transformers. We experimentally validated the ubiquitous existence of essential sparsity across large pre-trained transformer models of varying scales for vision and language (including LLMs), irrespective of the training strategy used for pre- pre-training them. We also present an intriguing emerging phenomenon of abrupt sparsification during the pre-training transformers and its ability abstract knowledge within few parameter subset with increasing pre-training data volume. Lastly, we studied the performance of LTH with respect to essential sparsity. Our future work will aim extend our essential sparsity observations to more gigantic models, and to push for higher sparsity ranges.
10

---

Acknowledgement
The research is in part supported by the Intelligence Advanced Research Projects Activity (IARPA) under Contract No. 2022-21 102200004, The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of IARPA or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
References
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar. Jakob Uszkoreit, Llion Jones, Aidan N Gomez. Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998 5998-6008, 2017.
[2] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
[3] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jiamyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, Zhaohui Yang, Yiman Zhang, and Dacheng Tao. A survey on visual transformer. ArXiv, abs/2012. 2020.
[4] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv'e J' J'egou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
[5] Zhiyuan Mao, Ajay Jaiswal, Zhangyang Wang, and Stanley H. Chan. Single frame atmospheric turbulence mitigation: A benchmark study and new mmyysic--inspied transformer model. ArXiv, abs/2207.10040. 2022.
[6] Minghang Zheng, Peng Gao, Renrui Zhang, Xiaogang Wang, Hongsheng Li, and Hao Dong. End-to-end object detection with adaptive clustering transformer. ArXiv, 0ix/2011.09311 2021.
[7] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam M. Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In ICML, 2018.
[8] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information prcntttiooooooossso processing systems, 32, 2019.
[9] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
[10] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A ques- tion answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.009 2018.
[11] Ajay Jaiswal, Liyan Tang, Meheli Ghosh, Justin Rousseau, Yifan Peng, and Ying Ding. Radbert- cl: Factually-awar contrastive learning for radiology report classification. Proceedings of machine learning research, 158:196-201 208, 2021.
[12] Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. End-to-end open-domain question answering with bertserini. arXiv preprint arXiv:1902 1902.01718, 2019.
[13] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 180..0444477 2018.
[14) Ming Ding, Chang Zhou, Qibin Chen, Hongxia Yang, and Jie Tang. Cognitive graph for multi-hop reading comprehension at scale. arXiv preprint arXiv: 190.00009, 2019, 2019.
11

---

[15] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham. Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprini (rxXix:2044002111 2022.
[16] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le. and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. /903, 2022.
[17] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint orXiv:2005.14100 2020.
[18] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned one-shot, 2023.
[19]
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
[20] Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. Advances in neural information processing systems, 33: 11534-11866,, 2020.
[21] Ajay Jaiswal, Haoyu Ma, Tianlong Chen, Ying Ding, and Zhangyang Wang. Spending your winning lottery better after drawing it, 2022.
[22] Lu Yin, Shiwei Liu, Fang Meng, Tianjin Huang, Vlado Menkovski, and Mykola Pechenizkiy. Lottery pools: Winning more by interpolating tickets without increasing training or inference cost. arXiv preprint arXiv:2208 0222208.1844 10842, 2022.
[23] Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing early-bird tickets: Towards more efficient training of deep networks. arXiv preprint arXiv: 1957, 2019.
[24] Ajay Kumar Jaiswal, Haoyu Ma, Tianlong Chen, Ying Ding, and Zhangyang Wang. Training your sparse neural network better with any mask. In International Conference on Machine Learning, pages
PMLR, 2022.
[25] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations, 2019.
[26] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 5858824-5866, 2020.
[27] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. The lottery ticket hypothesis at scale. arXiv preprint arXiv: 190..099300111 2019.
[28) Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Zhangyang Wang, and Jingjing Liu. Earlybert: Efficient bert training via early-bird lottery tickets. arXiv preprint arXiv:2 100.0003,, 2020.
[29] Sai Prasanna, Anna Rogers, and Anna Rumshisky. When bert plays the lottery, all tickets are winning. arXiv preprint 20X.iv:200000056)), 2020.
[30] Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end-to-end exploration. Advances in Neural Information Processing Systems, 14.19994-19988, 2021.
[31] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint WwXiv:2009.03300 2020.
[32] Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. Advances in neural information processing systems, pages 598-605, 1990.
12

---

[33] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruming. trained quantization and huffiman coding. In International Conference on Learning Representations, 2016.
[34] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 1135-1143. Curran Associates, Inc., 2015.
[35] Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In D. S. Touretzky, edi- tor, Advances in Neural Information Processing Systems 2, pages 598-605. Morgan-Kaufmann, 1990.
[36] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, pages 2736-2744 2017.
[37] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. Proceedings of the IEEE International Conference on Computer Vision, 2017.
[38] Hao Zhou, Jose M Alvarez, and Fatih Porikli. Less is more: Towards compact cnns. In European Conference on Computer Vision, pages 662-677. Springer, 2016.
[39] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint 1rX1v:1611.06440, 2016.
[40] Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning, In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 20378-20389. Curran Associates, Inc., 2020.
[41] Hao Jiang, Ke Zhan, Jianwei Qu, Yongkang Wu, Zhaoye Fei, Xinyu Zhang, Lei Chen, Zhicheng Dou, Xipeng Qiu, Zikai Guo, et al. Towards more effective and economic aparsely-activatee model. arXiv preprint arXiv:2110.. 1rXiv:2110.07431, 2021.
[42] Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992.
[43] Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. Advances in Neural Information Processing Systems, 30, 2017.
[44] William Finnoff, Ferdinand Hergert, and Hans Georg Zimmermann. Improving model selection by nonconvergent methods. Neural Networks, 6(6):711-783, 1993.
[45] Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through |_0 regularization. arXiv preprint 172iv:17 2017.
[46] Jion-Hao Luo and Jianxin Wu. Autopruner: An end-to-end trainable filter pruning method for efficient deep model inference. Puttern Recognition, 107: 2020.
[47] Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsifi- cation. Advances in Neural Information Processing Systems 33 pre-proceedings, 2020.
[48] Vikash Sehwag, Shiqi Wang, Prateek Mittal, and Suman Jana. Hydra: Pruning adversarially robust neural networks. Advances in Neural Information Processing Systems, 33, 2020.
[49] Yihua Zhang, Yuguang Yao, Parikshit Ram, Pu Zhao, Tianlong Chen, Mingyi Hong, Yanzhi Wang. and Sijia Liu. Advancing model pruning via bi-level optimization. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[50] Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint 11Xiv:1710001878, 2017.
13

---

[51] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.095744 2019.
[52] Tao Lin, Sebastian U. Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. In International Conference on Learning Representations, 2020.
[53] Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang. and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with neuroregeneraaion. Advances in Neural Information Processing Systems (NeurlPs).. 2021.
[54] Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibeseu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 9(1):2383, 2018.
[55] Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, 2019.
[56] Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXivv:907.00m /9070048400 2019.
[57] Utku Evci, Trevor Gale, Jacob Mcnick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pages 2943-2952. PMLR, 2020.
[58] Shiwei Liu, Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do we actually need dense over-parameetrizzatonn in-time paramtttettiiiiioonnn in sparse training. arXiv preprint arXiv:2102.002887, 2021.
[59] Jonathan Schwarz, Siddhant Jayakumar, Razvan Pascanu, Peter Latham, and Yee Teh. Power- propagation: A sparsity inducing weight reparameterisation. Advances in neural information processing systems, 14:28889-289035 2021.
[60] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020.
[61] Hidenori Tanaka, Daniel Kunin, Daniel LK Yamins, and Surya Ganguli, Pruning neural networks without any data by iteratively conserving synaptic flow. In Advances in Neural Information Processing Systems 33 pre-proceedings, 2020.
[62] Pau de Jorge, Amartya Sanyal, Harkirat S Behl, Philip HS Torr, Gregory Rogez, and Puneet K Dokania. Progressive skeletonization: Trimming more fat from network at initialization. arXiv preprint 2xXiv:2006.09081 2020.
[63] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
(64] Jacob Devlin, Ming- King-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810. 04800, 2018.
[65] Ofir Zafrir, Ariel Larey, Guy Boudoukh, Haihao Shen, and Moshe Wasserblat, Prune once for all: Sparse pre-trained language models. arXiv preprint arXiv:21 1.05754, 2021.
[66] Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models. arXiv preprint arXiv:2003.07259, 2022.
[67] Dongkuan Xu, Ian EH Yen, Jinxi Zhao, and Zhibin Xiao. Rethinking network pruning-under the pre-train and fine-tune paradigm. arXiv preprint (.Xiv:210400866 2021.
[68] François Lagunas, Ella Charlaix, Victor Sanh, and Alexander M Rush. Block pruning for faster transformers, arXiv preprint arXiv:2109.0488. 2021.
14

---

[69] Qingru Zhang, Simiao Zuo, Chen Liang, Alexander Bukharin, Pengcheng He, Weizhu Chen, and Tuo Zhao. Platon: Pruning large transformer models with upper confidence bound of weight importance. In International Conference on Machine Learning, pages (6809-2682)) PMLR, 2022.
[70] Elias Frantar, Eldar Kurtic, and Dan Alistarh. M-fac: Efficient matrix-free approximations of second-order information. Advances in Neural Information Processing Systems, 34:14873
14886, 2021.
[71] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprini arXiv:2205.01668 2022.
[72] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9938-997,, 2020.
[73] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension text. arXiv preprint arXiv:1606.0522 05250, 2016.
[74] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
[75] Shiwei Liu, Tianlong Chen, Zhenyu Zhang, Xuxi Chen, Tianjin Huang, Ajay Jaiswal, and Zhangyang Wang. Sparsity may cry: Let us fail (current) sparse neural networks together! arXiv preprint arXiv:2303.02/444 2023.
[76] Rik Koncel- Kedzioskk Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. MAWPS: math word problem repository. In Proceedings of the 2016 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 1152-1157, San Diego, California, June 2016. Association for Computational Linguistics.
[77] Shen- Yun Miao, Chao-Chun Liang, and Keh-Yin Su. A diverse corpus for evaluating and developing english math word problem solvers. arXiv preprins arXiv:2106.15772 2021.
[78] Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07199) 2021.
[79] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1):5232-522) 5270, 2022.
[80] Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning n: m fine-grained structured sparse neural networks from scratch. arXiv preprint NXX::22022000100 2021.
[81] Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre-training arXiv preprint arXiv:2110.0209 2021.
(82) Donghyun Kim, Kaihong Wang, Stan Sclaroff, and Kate Saenko. A broad study of pre-training for domain generalization and adaptation. In European Conference on Computer Vision, 2022.
[83] Simon Kornblith, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2661-2671, 2019.
[84] Ziquan Liu, Yi Xu, Yuanhong Xu, Qi Qian, Hao Li, Antoni Chan, and Rong Jin. Improved fine- tuning by leveraging pre-training data: Theory and practice. arXiv preprint arXiv:2111.12292. 2021.
[85] Elias Frantar and Dan Alistarh. Massive language models can be accurately pruned in one-shot. arXiv preprint 00X10:2301.00774, 2023.
15


Reviews:
[{"review_id": "OGJE5hjxTU", "reviewer": "Reviewer_XMXq", "summary": "This paper delves into the natural sparsity of large-scale pretrained transformers that could be found without additionally training the models with sparsity optimization targets. The authors find that when simply low-magnitude weights are zeroed out, 1) up to a certain sparsity level,  the pruned transformers do not exhibit performance drop on downstream tasks after fine-tuning. 2) the sparsity of weights abruptly increases at certain training iterations 3) simple magnitude-based pruning shows similar performance as the Lottery Ticket Hypothesis method up to a certain sparsity level. The phenomenon is observed across many different LLM and vision transformer architectures. Also interestingly, self-supervised weights are more sparse than supervised weights.\n", "questions": "Please refer to the weaknesses.", "limitations": "No negative societal impact is expected.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper presents interesting findings on the nature of pretrained weights of large-scale models based on substantial amount of experimental grounds. The authors emphasize that the use of essential sparsity to prune weights can be done without repetitive train-prune-retrain routine. Considering the massive scale of recent large-scale transformers, it is a promising and practically efficient approach to find and utilize the sparsity of transformers. \n\nIn addition, this paper opens up several research topics to be studied. I believe they will be impactful.\n- Sparsity pattern from self-supervised learning and supervised learning\n- Sparsity pattern change by training on larger-scale datasets\n- Acceleration of large-scale transformers by pruning weights\n", "weaknesses": "- Practical gains after finding the sparsity pattern\n\nIn contrast to learning-based pruning methods (such as LTH), the essential sparsity has the raw pattern of weight magnitude. Although it is interesting to find the sparsity is there by nature, as mentioned in L90-91, such patterns can’t be directly used to bring actual acceleration of neural networks. Training-based methods can set optimization goal on sparsity patterns to alter the sparsity pattern to make the neural network accelerable, however, such approach seems to be difficult for essential sparsity. While it is easier to find the sparsity pattern with essential sparsity, what would be the practical gains of the proposed method after finding the sparsity?\n\n- Why does sparsity occur?\n\nThe authors present interesting comparison of impact of sparsity on model accuracy on various conditions - model size & architecture, supervised vs self-supervised learning. It is interesting to know of such results, but why does such differences happen? How can we understand the phenomenon? \n\n- typos\n\nL36 pre0trained -> pre-trained\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for identifying the significance of our work and finding it promising and practical considering the massive scale of the recent large-scale models. We additionally appreciate that you found that our experiments are quite solid and our observations can lead to opening up several research topics. To further address some weaknesses pointed out by you, we would like to address them point-by-point as below:\n\n**1. Limited practical acceleration due to unstructured sparse patterns?** Thank you for bringing up this point and we are glad to update you that we have additionally explored fine-grained N:M structured sparsity (https://arxiv.org/abs/2102.04010) (including widely accepted 2:4 sparsity pattern with real hardware acceleration) which suggests each contiguous block of M values, N values must be non-zero. \n\n * To the surprise in our favor, we found that essential sparsity **still holds for N:M sparsity** which can also be identified in a training-free and data-free manner at FREE COST bringing actual acceleration for large transformers. We have included our results on N:M sparsity in Figure 2 of the rebuttal pdf. \n\n* We would like to address that although optimization goals on sparsity patterns can be set up during training; it is impractical for this to be adopted to current gigantic transformers without industry-scale hardware. \n\n* On the other hand, the pivotal contribution of the essential sparsity is to bring attention to FREE of COST pre-existing sparse patterns without any training requirements.\n\n**2. Why does sparsity occur?** Thank you for raising this question. Prior literature has consistently observed that higher sparsity comes at a natural consequence of model size. [https://openreview.net/pdf?id=TJ2nxciYCk-] found that large transformers often have unactivated sparse neurons (activation space) and our results can be viewed as a counterpart in weight space. Similarly, [https://cbmm.mit.edu/sites/default/files/publications/Theoretical_Framework__How_Deep_Nets_May_Work_15.pdf] argues that certain deep architectures – such as CNNs and transformers work very well because they significantly exploit the general property of compositional sparsity. In addition, in deep learning theory [https://arxiv.org/abs/2112.11027 , https://arxiv.org/abs/1909.05122 , https://arxiv.org/abs/2002.09277, https://arxiv.org/abs/1903.09367 ] there are many work using the sparsity modeling to understand the implicit regularization impact and over-parameterization of DNN with growing scale. Regarding your point about supervised vs self-supervised, our results are consistent with prior work [https://arxiv.org/abs/2012.06908 ] on small-scale models, illustrating a important signal that self-supervised learning is more sparsity friendly. We conjecture that sparsity is one of the key structure prior for unsupervised learning [https://openreview.net/pdf?id=TJ2nxciYCk-, https://arxiv.org/pdf/2207.04630.pdf ]; and while we do not have a theoretical explanation yet (leaving it for future work), we believe that self-sup learning may inherently induce better sparse patterns during training. \n\nFinally, thank you for pointing out some typos, and we promise to correct them all in the camera ready version along with a more detailed related section discussing the above prior works.\n\nWe hope our responses have clarified many of your concerns, and please do not hesitate to let us know what else we could do in order to convince you for a rating upgrade."}, {"review_id": "BD3DjBqdbr", "reviewer": "Reviewer_AZn4", "summary": "The paper postulates existence of \"Essential Sparsity\" in large pre-trained transformer models. The \"Essential Sparsity\" is defined by the paper as a sparsity threshold beyond which further pruning of weights leads to a large performance drop. It considers (1) one-shot pruning and (2) lottery ticket pruning of networks with fine-tuning. Experiments are conducted on both large language and vision transformer models.", "questions": "While some of the analyses of the paper is interesting, I would appreciate authors comments on above weaknesses.", "limitations": "The paper doesn't cover discuss the limitations of the study in details. For example, while the sparsity and efficiency of models both during training and inference is an important problem, it should be kept in mind that unstructured sparsity alone is not good enough to obtain speed gains. There is no discussion of potential speed gains with the unstructured sparsity (this doesn't necessarily mean FLOPs but speed improvement during inference/training), or training time of different networks, etc.\n", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "While the sparsity has been studied a lot in the neural network literature, this work studies the sparsity for pre-trained language and vision models. Given the wide adoption of these networks, the study is important. The paper covers a number of experiments for large vision as well as language models. The comparison of self-supervised vs fully supervised models is interesting.", "weaknesses": "1. Existence of sparsity in transformers in both language [a,b] and vision [c] has been discussed in the prior literature. The drop of performance in networks beyond certain threshold has also been covered in the literature. I do believe the current work is the first one to formally define it, however it will be good if the work can put the contributions in proper context.\n\n2. Many of the \"surprising\" findings of the work appear to be over-claimed. For instance,\n\n    i. The abrupt drop in the performance of networks beyond certain sparsity thresholds is shown by the original LTH paper and the current work extends it to pre-trained network.\n\n    ii. The similarity of sparse masks for various downstream tasks is also not very surprising given that the networks was pre-trained on a large massive dataset and the downstream tasks comprise of very small similar sets of data (often even the subset of pre-training data).\n\n    iii. L50 (and L66) says that \"a significant proportion of the weights in them can be removed for free\" while it is hardly free given the network has been heavily pre-trained.\n\n3. The mathematical definition of Essential Sparsity is somewhat unclear (what exactly does $1-\\frac{||m||_0}{|m|}$ represent?). Also calling it essential for the network is a bit misleading. I agree dropping it would hurt the performance of the network but it is not \"essential\" for the network to perform well. A fully dense network usually performs the best.\n\n4. Minor fix - Labels for x-axis is missing in most of the figures.\n\nReferences:\n\na. Chen, Tianlong, et al. \"The lottery ticket hypothesis for pre-trained bert networks.\" NeurIPS 2020\n\nb. Dettmers, Tim, et al. \"Llm. int8 (): 8-bit matrix multiplication for transformers at scale.\" NeurIPS 2022\n\nc. Girish, Sharath, et al. \"The lottery ticket hypothesis for object recognition.\" CVPR. 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your time to review our work and glad that you find our work important and that some of our observations are very interesting. We would like to address all the weaknesses pointed out by you point-by-point below:\n\n**1. Existence of sparsity in transformers in both language and vision has been discussed in the prior literature?** Thank you for outlining your concern. We would like to emphasize that although there exist several works discussing the existence of sparsity in transformers:\n* Our work primarily focuses to bring attention to the pre-existing sparse patterns in a **TRAINING-FREE** and **TASK-FREE** setting which is critically important considering the exploding scale of transformers. As pointed out by reviewers XMXq and dYgU, our observations provide a promising and practically efficient approach to finding and utilizing the pre-existing sparsity of transformers and potentially open up discussion around many research topics (eg. phase transition, etc). \n* In addition, we are the first to scale up our observations for sparsity **at billion-level transformers** (additional results in rebuttal pdf **include Vicuna 7B results - check Figure 3**) and illustrate that you do not require any iterative pruning and retraining in the essential sparsity range. We for the first time provide empirical evidence that sparse masks obtained by expensive iterative prune and retrain procedures and simple OMP without any retraining are significantly similar indicating LTH is not doing anything magical in essential sparsity regimes and OMP can identify matching subnetworks. \n* Lastly, our observations related to abrupt sparsification during the pre-training process, supervised vs self-supervised, etc also open exploration playgrounds to theoretically understand and design efficient pre-training strategies for researchers.\n\n**NOTE:** We also include new results on Vicuna-7B to illustrate the existence of essential sparsity within them; indicating a sound message that there potentially **exists an easy way to sparsify modern-day LLMs without access to any training data, no retraining, in one-shot fashion** before observing abrupt performance drop.  \n\n**KEY POINT** of this work is not to demonstrate sparsity, **BUT** it is to demonstrate how easily this good sparsity could be achieved in the large model at no cost. We are not competing with LTH or any other pruning work, but we are interested in whether it is necessary to use LTH or other expensive methods.\n\n**2. Over-claimed observations:**\n* **Abrupt drop is observed by original LTH paper:** We would clarify that although an abrupt drop in the performance of networks beyond certain sparsity thresholds is shown by the original LTH paper; our key point is to study this observation in a multi-dimensional way wrt. training strategy, pruning strategy, data modality, and model-scale. Not just extending to pre-trained models, our multi-dimensional study reveals many interesting findings such as sparsity ratios below the abrupt drop marker are agnostic to pruning strategies (iterative vs OMP) and you do not require any fancy and expensive method to identify high-quality sparse subnetworks. At a glance, it might look simple, but this has a huge practical implication considering the impracticality to perform iterative prune and retrain strategy with modern scale LLMs.\n* **Similarity of sparse mask for various downstream tasks is not interesting:** We would like to highlight that unlike developing downstream task-dependent sparse mask (https://arxiv.org/pdf/2303.14409.pdf, https://arxiv.org/abs/2012.06908, etc), our work again brings focus on the key question: *Is always required to have a task-dependent mask which only works for single task considering the computational cost involved in identifying them?* Our observations find that within the essential sparsity range, it is not required to search for a task-dependent mask and a cheap one-shot mask is as good as an expensive task-dependent mask identified by IMP of LTH.\n* **Our argument of free is hardly free given the network has been heavily pre-trained?** We apologize for the confusion, and we think there is a misunderstanding of the term “for free” in the wrong context. When we say “for free”, given a pre-trained model, with our OMP settings you do not need an additional computational budget to iteratively prune and retrain the model weights to identify sparse subnetworks (impractical for modern LLMs without industry-standard hardware), thereby no pruning overhead.\n \n**3. Limitation [unstructured sparsity is not good enough]:** Thank you for bringing up this point and we are glad to update you that we have **additionally explored fine-grained N:M structured sparsity** (https://arxiv.org/abs/2102.04010) (including widely accepted 2:4 sparsity pattern with real hardware acceleration) which suggests each contiguous block of M values, N values must be non-zero. To our favor, we found that **essential sparsity still holds for N:M sparsity** which can also be identified in a training-free and data-free manner at FREE COST bringing actual acceleration for large transformers. We have included our results on N:M sparsity in **Figure 2 of the rebuttal pdf**.\n\n**4. Essential for network is misleading?** We would like to clarify the confusion. When we say essential sparsity it does not mean it is essential for the network. However, it means that the network reveals robustness to sparsification and its performance doesn’t significantly drop wrt. pruning with the essential sparsity range. We follow previous works (https://arxiv.org/abs/1901.09181, https://arxiv.org/abs/1902.09574, https://arxiv.org/abs/2101.09048) and define the model sparsity as  $1-\\frac{\\|m\\|_0}{|m|}$, where $|m|_0$ is the number of non-zero weights and $|m|$ refers to the total number of weights. \n\nHope our rebuttal clarifies all your doubts and please let us know what else we could do to convince you of a rating upgrade."}, {"review_id": "pLhhuXVyIP", "reviewer": "Reviewer_Vvp4", "summary": "This paper defines essential sparsity and conducts various experiments to analyze the sparsity property of pre-trained models.", "questions": "What about the conclusions on large models like LLaMA 65B or Bloom 175B?", "limitations": "The insights brought from this paper is limited.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper investigates the potential of directly sparsifying the pre-trained models\n2. CV and NLP models are both explored.", "weaknesses": "1. Evidence on large models is lacking.\n2. Some conclusions are not new, e.g., sharp dropping point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your time to review our work. We appreciate your feedback and would like to address the concerns you raised regarding our work's weaknesses. However, we believe that there might be some key contributions that may not have been fully acknowledged in your assessment. We kindly suggest taking into consideration the viewpoints of other reviewers as well, as they might provide a more comprehensive perspective on our work. \n\n1. To address your concerns related to evidence on large-scale models, we have included **new experiments on Vicuna-7B in the rebuttal pdf Figure 3** (note we have OPT-1.3B results in the submitted draft) which aligns with our observations in the paper. Secondly, we would clarify that although the abrupt drop phenomenon is observed previously; we would like to highlight that our key point is to study this observation in a multi-dimensional way wrt training strategy, pruning strategy, model-scale, data modality, and dataset size. Not just extending to pre-trained models, our multi-dimensional study reveals many interesting findings such as sparsity ratios below the abrupt drop marker are agnostic to pruning strategies (iterative vs OMP) and you do not require any fancy and expensive method to identify high-quality sparse subnetworks. At a glance, it might look simple, but we believe this has a huge practical implication considering the impracticality to perform iterative prune and retrain strategy with modern scale LLMs. \n\nFinally, we kindly hope you understand that the request for conclusions on LLaMa 65B and Bloom 175B couldn’t be afforded without large-scale industry-scale hardware.\n\n2. **Conclusions are not new?** We respectfully disagree. As nicely summarized by other reviewers: **(a) Reviewer XMXq:**  *“...this paper opens up several research topics to be studied. I believe they will be impactful…”* **(b) Reviewer Azn4:** *“...comparison of self-supervised vs fully supervised models is interesting…”* **(c) Reviewer QH66:** *“...paper makes a number of interesting observations about the notion of essential sparsity…”* **(d) Reviewer dYgU:** *“...the authors present several quite surprising findings related to the existence of essential sparsity…”*. Almost all other reviewers found that our empirical observations have wide practical importance to find and utilize the pre-existing sparsity of transformers. \n\nAdditionally, we believe our observations related to abrupt sparsification during the pre-training process, supervised vs self-supervised, etc also open exploration playground to theoretically understand and design efficient pre-training strategies for researchers.\n\nWe sincerely hope our responses have clarified many of your concerns, and please do not hesitate to let us know what else we could do in order to convince you of a rating upgrade."}, {"review_id": "4qLQtAjl4s", "reviewer": "Reviewer_QH66", "summary": "This research paper focuses on the following notion for large pre-trained models: \"essential sparsity\", the idea that a sharp drop in fine-tuning performance occurs after one-shot pruning relative to the level of sparsity. The authors propose that large and overparameterized models can be pruned without additional computational expense, showing this to be true across a range of tasks in both computer vision and natural language processing. Additionally, the authors found an interesting occurrence of abrupt sparsification during pre-training, indicating that models trained with larger datasets tend to achieve knowledge abstraction with fewer parameters. Their findings also revealed that self-supervised learning objectives tend to trigger stronger emergent sparsification properties than supervised learning. They argue that identifying and understanding these inherent high-quality sparse patterns could make fine-tuning large models more practical and environmentally friendly.", "questions": "There is a completely different approach to get the benefits of essential sparsity -- namely to start with and train a sparse network. That would be even less computationally intensive than training a dense network and then doing OMP on that. \nAre the authors familiar with methods such as SynFlow, PHEW, Synflow++, etc that do pruning before training? If not, I suggest that they also consider those and compare the performance they get from OMP versus those networks. \n\n The two more interesting observations of the paper (the idea that better knowledge abstraction may be feasible with fewer parameters (as long as we train with more data), and that self-supervised learning tends to have better emergent sparsification than supervised learning) are presented very briefly, without a sufficiently deep analysis in my opinion. ", "limitations": "One limitation of course is that the claims of the paper are not substantiated in the case of really large foundation models used today -- but of course it is hard for academic researchers to experiment with the training of such models.\n\nAnother issue is that the paper does not explicitly state the limitations of the proposed method. For example, in the end of section 6 there are some rather \"hidden statements\" about the benefits of LTH and IMP, which is a very important point. ", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper makes a number of interesting observations about the notion of essential sparsity, its relation to \"winning ticket\" networks (see LTH), the idea that better knowledge abstraction may be feasible with fewer parameters (as long as we train with more data), and that self-supervised learning tends to have better emergent sparsification than supervised learning. \n", "weaknesses": "Unfortunately, these interesting observation mentioned in the Strengths section, are also too \"empirical\" or anecdotally shown in the paper. It is hard to be convinced that these are general results -- and they are not substantiated with any form of of theory or at least some explanations/justifications based on results from other papers. \n\nIn general, I would describe this paper as \"it is moving in a good direction but it is not ready for publication yet\" -- at least not at the top-tier conference of deep learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your time to review our work.  We would like to address all the weaknesses pointed out by you point-by-point below:\n\n**1. Experiments are too experimental and hard to be convinced that these are general results:** We would like to highlight that sparse neural networks are new frontier for deep learning theory, and limited theoretical works are available to augment experimental observations and algorithm designs. \n* Although we are heavily empirical, we would like to bring your attention to recent theoretical works  in deep learning theory [https://arxiv.org/abs/2112.11027 , https://arxiv.org/abs/1909.05122 , https://arxiv.org/abs/2002.09277, https://arxiv.org/abs/1903.09367 ] which use the sparsity modeling to understand the implicit regulaization impact and over-parameterization of DNN with growing scale supporting our empirical observations. We promise to cite them in our final version to provide enough theoretical support for our observations. \n* About supervised vs self-supervised findings, our results are again consistent with prior work [https://arxiv.org/abs/2012.06908] on small-scale models, illustrating an important signal that self-supervised learning is more sparsity friendly. We conjecture that sparsity is one of the key structure prior for unsupervised learning [https://openreview.net/pdf?id=TJ2nxciYCk-, https://arxiv.org/pdf/2207.04630.pdf ]; and while we do not have a theoretical explanation yet (leaving it for future work), we believe that self-sup learning inherently induces better sparse patterns during training. \n* To further augment that our finding can be generalized to even modern-day large-scale models, we have attached **additional experiments with Vicuna-7B in the rebuttal pdf Figure 3**. The need to find a high-quality sparse subnetwork in a training-free and data-free approach is significantly important considering the exploding size of LLMs, where the conventional iterative prune-retrain strategy becomes impractical. \n* We have also attached **new results exploring fine-grained N:M structured sparsity** (https://arxiv.org/abs/2102.04010) (including widely accepted 2:4 sparsity pattern with real hardware acceleration) which suggests each contiguous block of M values, N values must be non-zero. To our favor, we found that essential sparsity still holds for N:M sparsity which can also be identified in a **training-free and data-free manner at FREE COST** bringing actual acceleration for large transformers. Please check **Figure 2 of the rebuttal pdf**.\n\n\n**2. Less computationally intensive way: start with a sparse network and perform training?** \n\nWe apologize for the confusion, and we would like to highlight that **our work focuses on identifying the sparse patterns in pre-trained models**. Note that we perform OMP on a pre-trained checkpoint and this is *a training-free and data-free approach* (cheapest possible wrt. SynFlow or SynFlow++: Yes we are very familiar with Synflow) to sparsify the pre-trained checkpoint. Note that, first, we are not doing pre-training but simply pruning pre-trained models in resource-constrained environments to identify the sparse subnetwork for free. \n\nSecondly, during fine-tuning of the identified sparse subnetwork, we only do sparse training (as you suggested). We want to clarify that like OMP,  SynFlow can be smoothly integrated with our settings. To your interest, *we applied Synflow and found that Synflow barely outperforms our extremely cheap approach* as shown in the **Figure 1 of our rebuttal pdf**. \n\n**KEY POINT** of this work is not to demonstrate sparsity, BUT it is to demonstrate *how easy this good sparsity could be achieved in large model at no cost* due to emergent behaviors. We are not competing with LTH or any other fancy pruning work, but we are interested in whether it is necessary to LTH or other expensive methods within the essential sparsity range. \n\n**3.Claims of the paper are substantiated for really large foundational models used today?** Thank you for bringing up this point and we provide **additional experiments with Vicuna-7B on popular MMLU benchmark in the rebuttal pdf Figure 3**. Based on our results in Figure 3, it is interesting to observe that our essential sparsity observations hold true even for modern LLMs, sending a favorable signal about the hidden existence of high-quality sparse subnetwork which can be identified at free in dense pre-trained checkpoints. We also provide new results which extend our observations for fine-grained N:M structured sparsity with real hardware potential.\n\n**4. Limitations are not explicitly outlined?** We really appreciate your concerns and we promise to add a separate limitation section in the paper explicitly mentioning the limitations of our work (such as the benefits of IMP in high sparsity regime, theoretical evidence, scaling up to 10B+ model parameters, etc.).\n\nWe sincerely hope our responses have clarified many of your concerns, and please do not hesitate to let us know what else we could do in order to convince you of a rating upgrade.\n  \n\n  "}, {"review_id": "NUA9nas0Zi", "reviewer": "Reviewer_dYgU", "summary": "This paper presents a comprehensive study on the induced sparse patterns across multiple large pre-trained vision and language transformers. The authors propose the existence of \"essential sparsity\" and present intriguing findings on abrupt sparsification during the pre-training of transformers and the effect of pre-training data on knowledge condensation.", "questions": "Please see the Weaknesses", "limitations": "Not discussing more hardware-friendly sparse patterns", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The main strength of this paper is that it validates the ubiquitous existence of essential sparsity across large pre-trained transformer models of varying scales for vision and language, irrespective of the training strategy used for pre-training them.\n\nThe authors used various datasets and models to validate the existence of \"essential sparsity\" and presented intriguing findings on abrupt sparsification during the pre-training of transformers and the effect of pre-training data on knowledge condensation. The authors also compared the performance of models with and without sparsification and found that one-shot sparsification without re-training does not significantly affect the performance of downstream tasks. In Section 6, the authors further carefully analyzed the connection between LTH and essential sparsity, and showed the former to become potentially less necessary or relevant in larger models. Those are valuable insights of broad interest to the sparsity research community.\n\nBased on the methodology and results presented in the paper, the experiment presented is sound enough to support the idea of the existence of essential sparsity in large pre-trained models (bert-base, OPT-125m, OPT-350m, OPT-1.3B). In particular, the authors present several quite surprising findings related to the existence of essential sparsity in large pre-trained transformer models. \n-\tFirstly, the authors found that BERT suddenly becomes heavily sparse after a certain number of training iterations, which is unobserved before and not well-understood. This finding suggests that there may be underlying “phase transition”-like mechanisms in the pre-training dynamics of transformers that are responsible for inducing sparsity, and further research is needed to understand it.\n-\tSecondly, the authors also found that BERT trained with a larger amount of pre-training data tends to have a better ability to condense knowledge in relatively fewer parameters. This finding is counter-intuitive because one would expect that increasing the amount of pre-training data would lead to an increase in the number of parameters required to capture the additional information.\n-\tThirdly, the authors found that self-supervised learning (SSL) objectives trigger stronger emergent sparsification properties than supervised learning (SL). This finding is also intriguing because one would expect that supervised learning, which provides more explicit information to the model, would lead to better knowledge condensation.", "weaknesses": "-\tOne main downside of this paper is that the paper did not report any result on hardware-friendly sparsity such as N:M, nor it discussed any GPU run time benefit from the induced essential sparsity (if any). While one can understand the study is mainly conceptual, just like the original LTH, I believe the real hardware support is especially important for LLM pruning/sparsity research due to their exploding costs\n-\tMore baselines are desired – currently only LTH is reported. Would essential sparsity meaningfully outperform the simplest baseline of random pruning? How it compares with the pruning result of SparseGPT [18]?\n-\tIn both SparseGPT [18] and Sparsity-May-Cry [75], it was found that larger LLMs are harder to prune. This paper seems to pinpoint the opposite conclusion. Could the authors elaborate why their conclusions seem to contradict [18,75], or not?\n-\tSparsity has more benefits beyond efficiency, such as few-shot transfer, robustness to noisy label or other distribution shifts. Many of those were previously demonstrated under LTH or dynamic sparse training settings. I will be curious to see if essential sparsity will own the same merits too.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for identifying the significance of our work and finding it promising and practical considering the massive scale of the recent large-scale models. We additionally appreciate that you found that our experiments are quite insightful, have surprising findings, and counter-intutive observations and can lead to opening up several research topics (phase transition etc.). To further address some weaknesses pointed out by you, we would like to address them point-by-point as below:\n\n**1. No results for hardware-friendly sparsity such as N:M:** Thank you for bringing up this point and we are glad to update you that we have additionally **explored fine-grained N:M structured sparsity** (https://arxiv.org/abs/2102.04010) (including widely accepted 2:4 sparsity pattern with real hardware acceleration) which suggests each contiguous block of M values, N values must be non-zero. To our favor, we found that **essential sparsity still holds for N:M sparsity** which can also be identified in a *training-free and data-free manner at FREE COST* bringing actual acceleration for large transformers. We have included **our results on N:M sparsity in Figure 2 of the rebuttal pdf**.\n\n**2. Large models are harder to prune?** \n\nWe would like to clarify the confusion that Sparsity may cry [75] states that *on hard tasks (arithmatic reasoning, protein stability etc.), even large models can not be effectively pruned to high sparsity*. It doesn’t mean that the larger the model becomes, the harder it is to prune them. Instead, they claim that the more challenging the task, the more difficult it becomes to prune the model.  Our work observations align with [75] without any contradiction, where we state that the essential sparsity range is dependent on the task complexity (line 193-197 of submitted pdf). \n\nSimilarly for SparseGPT [18], they mention “*In general, there is a clear trend of larger models being easier to sparsify, which we speculate is due to overparametrization (paragraph 3 section 4.1)*” which have **no contradiction with our claims**. Moreover, another piece of evidence from Figure 2 from SparseGPT -  \"*One key positive finding, illustrated in Figure 2, is that larger models are more compressible: they drop significantly less accuracy at a fixed sparsity, relative to their smaller counterparts*\" which doesn't contradict our observation.\n\n**3. More baselines are required like random pruning?** Thank you very much for raising this point and we completely agree that comparison with Random pruning is important. We have **uploaded new results of random pruning, random erk pruning in Figure 1 of the rebuttal pdf**. Note that random pruning performs significantly bad wrt. our one-shot magnitude pruning approach. We promise to include these results in the final version.\n\nIn addition to your interest, **Figure 3(a) in our rebuttal draft** illustrates that our essential sparsity observations hold **true even for modern LLMs (Vicuna-B)**, sending a favorable signal about the hidden existence of high-quality sparse subnetwork which can be identified at free in dense pre-trained checkpoints. \n\nTo further enrich our study, we replaced OMP with the recently proposed SparseGPT and found it to have generally consistent trends with OMP (**Figure 3(b) in our rebuttal draft**). In addition, it is interesting to observe that better-designed pruning strategies such as SparseGPT can further push the boundary of essential sparsity and identify better sparse subnetworks at comparatively higher sparsity ratios: yet at higher compute costs. We leave as future work on how to close the performance gap between OMP and SparseGPT.\n\n**4. Sparsity beyond Efficiency?** Thank you for bringing it up and we agree that sparsity has benefits beyond efficiency (eg. robustness, few-shot, etc). Unfortunately due to limited time for rebuttal, we leave this experiment for future (camera-ready) as it is not directly related to the primary scope of this work.\n\nWe sincerely hope our responses have clarified many of your concerns, and please do not hesitate to let us know what else we could do in order to convince you of a rating upgrade."}]

Author rebuttal:
We would like to thank all the reviewers for their time to review our work and offering important suggestions. In this pdf, we attach some key experiments requested by reviewers which further strengthen the impact of our work. We summarize our results as follows:

* **[Requested by Reviewer QH66 and dYgU] Figure 1** provides additional results related to the performance comparison of SynFlow, and the important baseline of Random Pruning + Random pruning with ERK; and illustrates that SynFlow doesn't bring any additional benefit despite being more expensive than OMP.  
* **[Requested by Reviewer XMXq, AZn4, dYgU BUT for all] Figure 2** provides additional results exploring fine-grained N:M structured sparsity (https://arxiv.org/abs/2102.04010) (including widely accepted 2:4 sparsity pattern with real hardware acceleration) and show our essential sparsity observations holds true for N:M sparse patterns.
* **[Requested by Reviewer Vvp4, QH66, dYgU BUT for all] Figure 3** provide additional experiments with *Vicuna-7B on popular MMLU benchmark*. Based on our results in Figure 3, it is interesting to observe that our essential sparsity observations hold true even for modern LLMs, sending a favorable signal about the hidden existence of high-quality sparse subnetwork which can be identified at free in dense pre-trained checkpoints. 

In last, we again thank all the reviewers and hope our additional results + rebuttal responses can clarify their doubts. We are more than happy to provide any further explanations required.

Best,
Authors 9125
 

Meta-review: There was a wide spread in initial ratings. Author rebuttal helped address majority of the concerns, though there are still some lingering questions from the reviewers in the discussion. After reading through the discussion and taking the author rebuttal and reviewer opinion into account, I recommend acceptance. Many of the pending concerns can be addressed in text, and I strongly recommend that authors incorporate these changes to further improve their manuscript.
Decision: Accept (poster)