Title: The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter
Venue: NeurIPS 2023
Accepted: yes
Keywords: Pre-trained Models

Sparsity

Emergence

Transformers

Pruning
Abstract: Large pre-trained transformers are $\textit{show-stealer}$ in modern-day deep learning, and it becomes crucial to comprehend the parsimonious patterns that exist within them as they grow in scale. With exploding parameter counts, Lottery Ticket Hypothesis (LTH) and its variants, have lost their pragmatism in sparsifying them due to high computation and memory bottleneck of repetitive $\textit{train-prune-retrain}$ routine of iterative magnitude pruning (IMP) which worsens with increasing model size. In this paper, we comprehensively study $\textit{induced sparse patterns}$ across multiple large pre-trained vision and language transformers. We propose the existence of -- $\textbf{essential sparsity}$ defined with a $\textbf{sharp dropping point}$ beyond which the performance declines much faster w.r.t the rise of sparsity level, when we directly remove weights with the smallest magnitudes in $\textbf{one-shot}$. We also present an intriguing emerging phenomenon of $\textbf{abrupt sparsification}$ during the pre-training of BERT, i.e., BERT suddenly becomes heavily sparse in pre-training after certain iterations. Moreover, our observations also indicate a $\textbf{counter-intuitive}$ finding that BERT trained with a larger amount of pre-training data tends to have a better ability to condense knowledge in comparatively relatively fewer parameters. Lastly, we investigate the effect of the pre-training loss on essential sparsity and discover that self-supervised learning (SSL) objectives trigger stronger emergent sparsification properties than supervised learning (SL). All our codes will be publicly available.

The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter

Abstract

Large pre-trained transformers are show-stealer in modern-day deep learning, and it becomes crucial to comprehend the parsimonious patterns that exist within them as they grow in scale. With exploding parameter counts, Lottery Ticket Hypothesis (LTH) and its variants, have lost their pragmatism in sparsifying them due to high computation and memory bottleneck of repetitive train-prune-retrain routine of iterative magnitude pruning (IMP) which worsens with increasing model size. This paper comprehensively studies induced sparse patterns across multiple large pre-trained vision and language transformers. We propose the existence of -'essential sparsity' defined with a sharp dropping point beyond which the performance declines much faster w.r.t the rise of sparsity level, when we directly remove weights with the smallest magnitudes in one-shot without re-training . We also find essential sparsity to hold valid for N:M sparsity patterns as well as on modern-scale large language models ( Vicuna-7B/13B ). We also present an intriguing emerging phenomenon of abrupt sparsification during the pretraining of BERT, i.e., BERT suddenly becomes heavily sparse in pre-training after certain iterations. Moreover, our observations also indicate a counter-intuitive finding that BERT trained with a larger amount of pre-training data tends to have a better ability to condense knowledge in comparatively relatively fewer parameters. Lastly, we investigate the effect of the pre-training loss on essential sparsity and discover that self-supervised learning (SSL) objectives trigger stronger emergent sparsification properties than supervised learning (SL). Our codes are available at https://github.com/VITA-Group/essential_sparsity .

1 Introduction

Enormous increases in scale often permeate systems with unique new behavior. Transformers [1], swiftly after their introduction are scaling every day, dramatically improving the state-of-the-art performance on a wide array of real-world computer vision [2, 3, 4, 5, 6, 7], natural language processing [8, 9, 10, 11, 12, 13, 14, 15, 16] applications and leaderboards. With the astonishing explosion of parameter counts (millions to billions) in the past few years, while chasing performance gains, finetuning these large pre-trained models with non-industry standard hardware is becoming seemingly impossible, in addition to expensive inference and steep environmental cost. For instance, GPT-3 [17] contains 175 billion parameters and requires at least five 80GB A100 GPUs for inference [18].

In the hustle of building gigantic models, a parallel and growing field of model compression has been exploring the prospects to compress these gigantic models at the cost of marginal/no sacrifice in performance. Among many efforts for compressing models and accelerating inference, network pruning [19, 20, 21, 22, 23, 24, 25, 26], which removes the least important components from the model, varying from the low granularity (e.g., individual weights) to the high granularity (such as blocks, filters, and attention heads), stands out as one of the most effective techniques. Despite the enormous success of the Lottery Ticket Hypothesis (LTH) [27] and its variants for small-scale neural

networks, there still exists a large gap in its practical adoption for large pre0trained transformers with billions of parameters because of the fully dense training routine of iterative magnitude pruning (IMP), which exacerbates with an increase in model capacity. Recently, many works [28, 29, 30, 20] are investigating strategies to make IMP pragmatic for large pre-trained models. However, they still rely on train-prune-retrain routine of IMP and downstream tasks to identify subnetworks. While chasing for gold-standard sparsity masks with computationally expensive approaches, it is critically important to understand pre-existing high-quality sparse patterns in these large transformers.

In our work, we reveal and study an intriguing, previously overlooked property of large pre-trained transformers 'essential sparsity' . In plain words (formal definition in Sec. 3), we define essential sparsity as sharp dropping point , beyond which the fine-tuning performance after one-shot pruning declines much faster w.r.t. the sparsity level rise. We directly remove weights with the smallest magnitudes in one-shot from the pre-trained checkpoints, thereby the identified subnetwork mask requires no extra computational overhead to spot and remains identical across all downstream tasks.

Essential sparsity is an important yet understudied induced property of large pre-trained transformers, indicating that at any time, a significant proportion of the weights in them can be removed free of any calibration data or post-optimization, although the proportion may vary depending on the complexity of the downstream task . One important practical implication conveyed by this observation, as overlooked by prior work, is summarized as: within the sparsity range induced by 'essential sparsity', the simplest possible pruning technique as aforementioned performs the same well as any fancy technique such as LTH, and even their identified sparse masks are highly similar.

Our work, for the first time, conducts a comprehensive study of the existence of essential sparsity across multiple vision and language transformers with varying scales and training strategies. Besides, we observe essential sparsity to hold valid for various N:M sparsity patterns with hardware speedup potentials. In addition, our experiments using the popular MMLU benchmark [31] on Vicuna-7B/13B illustrate essential sparsity observations remain true for modern-scale large language models (LLMs) , indicating the existence of high-quality sparse subnetworks within dense pre-trained checkpoint.

Next, we study the emergence of those sparsification properties during the pre-training dynamics, using BERT as the focused subject of study. We found an intriguing phenomenon of abrupt sparsification , i.e., BERT suddenly becomes heavily sparse after certain number of training iterations. As we vary the pre-training dataset size. our observation indicates another counter-intuitive finding that BERT trained with a larger amount of pre-training data tend to have a better ability to condense knowledge in relatively fewer parameters . We also dive deep into the effect of the pre-training loss on essential sparsity and discover that self-supervised learning (SSL) objectives trigger stronger emergent sparsification properties than supervised learning (SL).

Key contributions for our work can be summarized as:

Wefound the ubiquitous existence of essential sparsity across large pre-trained transformer models of varying scales for vision and language, irrespective of the training strategy used for pre-training them. High-quality sparse masks comparable to lottery tickets [27] within the essential sparsity range can be spotted free of inference or post-optimization by merely selecting the lowest magnitude weights, that requires no expensive repetitive train-pruneretrain routine. The observation holds true for both unstructured and N:M sparsity patterns.

Our comparison of the sparse masks obtained by selecting the lowest magnitude weights with the lottery tickets within the essential sparsity range, surprisingly unveils a notably high cosine similarity (>98%) across various downstream tasks from NLP and CV. This observation sends a strong message that in large pre-trained models, LTH perhaps enjoys little additional privilege despite utilizing enormous computational costs.

While studying the emergence of sparsification properties during the pre-training dynamics in BERTs, we found an intriguing phenomenon of abrupt sparsification , i.e., BERT suddenly becomes heavily sparse after certain number of training iterations. We additionally found a counter-intuitive observation that BERT pre-trained with larger amount of data tends to be more sparser, i.e., they become better at knowledge abstraction with fewer parameters .

We additionally study the effect of the pre-training loss on the emerging sparsity of FMs. When we switch between supervised learning (SL) and self-supervised learning (SSL) objectives on ViT, we observed that SSL tends to have better emergent sparsification

properties, thereby more friendly to pruning. We further provide layer-wise visualization to understand what the sparsity learned by SSL vs SL looks like.

2 Related Work

Sparse Neural Networks (SNNs). Sparsity in deep neural networks is usually introduced by model compression [32, 33] which removes the redundant parameters. Based on the type of sparsity patterns, it can be categorized into two families: ( i ) unstructured sparsity [33, 34, 35] where the non-zero elements are irregularly distributed; ( ii ) structured sparsity [36, 37, 38] where a group of parameters is eliminated like a convolutional kernel in convolutional neural networks or an attention head in the transformer. In general, the former sparsity pattern obtains a better performance thanks to its flexibility, while the latter sparse pattern tends to be more hardware friendly. Many important SNN works start by studying the former and then turn to the latter as a special case.

Meantime, according to the timing to perform dimensionality reduction, sparsity can be obtained in the post-training, during-training, and prior-training of deep neural networks. ( i ) Post-Training . To pursue inference time efficiency, trained models can be pruned dramatically with marginal loss of performance with respect to certain heuristics, including zero-order [33], first-order [39, 40, 41], and second-order [32, 42, 43] criteria. Among these algorithms, the weight magnitude-based approach ( e.g. iterative magnitude pruning) is a popular option. Especially, it is frequently adopted by the Lottery Ticket Hypothesis [19] to produce sparse NNs with undamaged trainability and expressiveness. ( ii ) During-Training . On the contrary to pruning a well-trained model for inference efficiency, duringtraining sparsification [44] also enjoys computational savings for model training. It normally first optimizes a dense network for several iterations, then gradually sparsifies it with a pre-defined schedule, and finally leads to lightweight sparse NNs. For example, [45, 46, 47] leverage ℓ 0 or ℓ 1 penalties to encourage weight sparsity during the network training, hoping to zero out unimportant parameters. [48, 49] cast it as an optimization problem with reparameterization and bi-level forms, respectively. Another fashion is dynamic sparsity exploration [50, 51, 52, 53, 54, 55, 56, 57, 58, 59] which allows pruned connections can be re-activated in the latter training stage. ( iii ) Prior-Training . Identifying the critical sparsity patterns at the initialization is one exciting rising sub-field. It determines the sparse connectivities in a very early stage, like one iteration [25, 60] or a few iterations [61, 62]. In this paper, we mainly investigate post-training unstructured SNNs.

Sparsity in Pre-Trained Transformers. Pre-trained Transformers have become de facto choice for numerous applications of natural language processing (NLP) [8, 9, 10, 15, 16] and computer vision [2, 7, 63]. Their impressive performance is partially credited to their tremendous learning capacity empowered by huge amounts of model parameters. Unfortunately, such successes come with burning thousands of GPUs/TPUs for thousands of hours [64, 17], even for a single round of model training. To address this resource-intensive concern and improve the affordability of these transformers, plenty of pioneering researchers devote themselves in this area [40, 20, 30, 65, 66, 67, 68, 69, 70]. Rather than proposing a new sparsification method, this paper reveals the blessings of induced essential sparsity during the pre-training and how we can capitalize it to prune large pre-trained models without any computational overhead.

3 Experimental Settings

Network and Pre-Trained Transformers. We consider { BERT Base [64], BERT Large [64], OPT 125M [71], OPT 350M [71], and OPT 1.3B [71]} and { ViT Base [2], ViT Large [2], and DiNO Base [63] for NLP and CV applications respectively. Their officially pre-trained models 1 are adopted in our experiments. To be specific, let f ( x ; θ p ) be the output of a transformer with pre-trained model parameters θ p and input sample x .

Datasets, Training, and Evaluation. For downstream tasks in NLP, we consider {MNLI, QNLI, QQP, SST-2} from GLUE [13], RTE from SuperGLUE [72], and SQuAD v 1 . 1 [73]. As for vision downstream applications, we examine {CIFAR-10, CIFAR-100, Tiny-ImageNet [74]}. We also use the Arithmetic Reasoning task from the recently proposed SMC-Benchmark [75] and consider popular math word problem datasets: (1) the widely used MAWPS benchmark [76] composed of 2,373 problems; (2) the arithmetic subset of ASDiv-A [77] - with 1,218 math problems; (3) the more

Output Dense (SparsityWeight:7.32%)

Intermediate Dense (Sparsity Weight:6.82%)

Attention Output Dense (Sparsity Weight:2.20%)

Layer

1

1.0

Layer 1

1.0

Layer1Uu

1.0

Layer 4

Layer 4

Layer 4

Layer 8

0.5

Layer 8

0.5

Layer 8

0.5

Layer 12

0.0

Layer 12

0.0

Layer 12

0.0

Attention Value (Sparsity Weight: 2.09%)

Attetion Query (Sparsity Weight: 1.54%)

Attention Key (Sparsity Weight: 1.55%)

Layer 1

1.0

Layer 1

1.0

Layer 1

1.0

Layer 4

Layer 4

Layer 4

Layer 8

0.5

Layer 8

0.5

Layer 8

0.5

Layer 12

0.0

Layer 12

0.0

Layer 12

0.0

challenging SVAMP [78] dataset which is created by applying complex types of variations to the samples from ASDiv-A. The task difficulty monotonically increases from MAWPS to ASDiv-A, and to SVAMP. We adopt the default dataset split for training and evaluation for our downstream application. More detailed configurations are collected in Table 1. For SMC-benchmark, we strictly followed the settings proposed in the official implementation 2

Sparse Neural Networks (SNNs). The weights of a SNN can be depicted as m ⊙ θ , where m ∈ { 0 , 1 } | θ | is a binary mask with the same dimensionality as θ and ⊙ is the elment-wise product. Let E T ( f ( x ; θ )) denotes the evaluation function of model outputs f ( x ; θ ) on the corresponding task T (which might involve fine-tuning). P ρ ( · ) is the sparsification algorithm which turns a portion ρ of ' 1 " elements in the sparse mask m into ' 0 "s. Here is a formal definition of Essential Sparsity below.

Essential Sparsity . If E T ( f ( x ; m ⊙ θ )) ≥ E T ( f ( x ; θ )) -ϵ , and E T ( f ( x ; P ρ ( m ) ⊙ θ )) < E T ( f ( x ; θ )) -ϵ where the value of ρ and ϵ are small. Then, the according sparsity 1 -∥ m ∥ 0 | m | is named as Essential Sparsity for the model f on task T .

As detailed above, the model at the essential sparsity usually has a turning point performance, which means further pruning even a small portion ρ of weights leads to at least ϵ performance drop, compared to its dense counterpart E T ( f ( x, θ )) . In our case, ϵ is set as 1% . In plain language, the turning point of essential sparsity defines a sparsity range with two characteristics: (i) within this range , the one-shot pruned model performs as well as or better than dense baseline, without re-training; (ii) beyond this range , a notable accuracy drop becomes observable after one-shot magnitude pruning.

Pruning Methods. To find the desired sparse mask m , we use two classic weight magnitude pruning algorithms [33, 19]. One-shot Magnitude Pruning (OMP): we directly eliminate a pre-defined portion of parameters from θ p with the least absolute magnitude. Lottery Tickets Pruning (LTP): ( i ) we first train the model to converge on a downstream task T ; ( ii ) then remove a portion of the smallest weights and reset the remaining weight to their initialized value from pre-training θ p ; ( iii ) such processes will be repeated until reaching the desired sparsity.

Two crucial facts are noteworthy. First, starting from the pre-trained model, unlike LTP which seeks downstream-specific masks with additional training, the OMP sparse masks require no additional training to identify and are agnostic to downstream applications . Since they are estimated directly from the pre-trained weight checkpoint, the OMP sparse mask remains the same for all downstream applications (we only continue to fine-tune the non-zero weights with the same mask, over different downstream datasets). Second, for the same above reason, a higher-sparsity OMP mask is naturally nested within a lower-sparsity mask, as long as they are pruned from the same pre-trained checkpoint using OMP. The same nested property does not hold for LTP-obtained masks.

Essential Sparsity

QNLI Accuracy Drop

EssentialSparsity

Essential Sparsity

4

OPT-125M

OPT-125M

OPT-125M

-10

OPT-350M

OPT-1.3B

OPT-350M

OPT-1.3B

OPT-350M

OPT-1.3B

-12

Essential Sparsity

Essential Sparsity

Essential Sparsity

2

8

OPT-125M

OPT-125M

OPT-125M

-10

■

OPT-1.3B

OPT-350M

OPT-1.3B

OPT-350M

OPT-1.3B

OPT-350M

12

0

10

20

30405060708090

0

10

20

3040

50

60

70

80

90

0

10

20

30405060708090

Sparsity Ratio

Sparsity Ratio

Sparsity Ratio

0

0

EssentialSparsity

EssentialSparsity

EssentialSparsity

-5

-5

-5

10

-10

-10

15

ViT-Base

-15

ViT-Base

-15

ViT-Base

ViT-Large

ViT-Large

ViT-Large

-20

0

10

20

30

40

50

607080

06

20

0

10

20

30

40

50

60

70

80

06

-20

0

10

20

30

40

50

60

7080

06

4 Essential Sparsity Exists in Pre-trained Language and Vision Models

Revisiting sparsity in Pre-trained Transformers: The unconstrained growth in parameters has resulted in significant computational costs and excessive memory demands. The trend undoubtedly continues with transformers on the forefront, where more and more layers are stacked with dense attention blocks (eg. GPT-3 has surprisingly 175 billion parameters) necessitating substantial computational resources and extended training or fine-tuning durations. Unlike extensive efforts explored in the past for pruning ResNets and related families of networks, pruning large-scale pre-trained transformer models hasn't received enough attention.

Lottery Ticket Hypothesis (LTH) based approaches have been recently explored to prune BERT-base size pre-trained models. However, they lose their pragmatism due to the expensive dense train-pruneretrain routine of IMP and non-transferable characteristics of identified subnetworks across similar tasks [20]. oBERT [66] proposes second-order to BERT-level scale by allowing for pruning blocks of weights. Note that, although these methods provide interesting approaches to prune BERT scale transformers, they fail to sufficiently recognize the strength of pre-existing, more easily accessible high-quality sparse patterns in pre-trained large models invariant of the scale, training data modality, and strategy. This paper's primary goal is to bring the sparse community's attention towards the strength of ubiquitously induced sparse patterns during the pre-training of large transformers and encourage them to effectively capitalize it during the design of more sophisticated pruning algorithms.

Essential Sparsity: As defined in Section 3, essential sparsity indicates the presence of naturally induced sparse structures in large pre-trained models with respect to the lowest magnitude weights. Figure 1 represents the induced sparsity distribution of bert-base-uncased pre-trained model with 21.50% close to zero weights, which we have been able to remove without ANY drop of performance across any of our evaluation downstream datasets. This sought a strong message about the existence of free, universal, and available to consume sparse mask ( m · θ p ) without any computational overhead induced by sophisticated pruning algorithms. In addition, a closer observation of the distributed sparsity ratios across different modules of the transformer block indicates that the majority of the low-magnitude weights are concentrated in the dense feed-forward layers. Such findings conform

with the success of Mixture-of-Experts [79] and provide cues for future development of pruning algorithms to exploit the over-parameterization of dense feed-forward layers.

We now enlist our key findings related to essential sparsity in large pre-trained transformer models:

In all vision and language models, we find the existence of essential sparsity and the sharp turning point of the sparsity-performance curve.

The sharp turning point of essential sparsity is downstream task-dependent and can vary depending on the task complexity.

The sharpness behavior of essential sparsity is more profound in vision pre-trained models in comparison with language models.

Essential sparsity holds valid for recently proposed pruning benchmark SMC-bench [75].

Essential sparsity holds valid for N:M structured sparsity patterns [80] with potential speedup.

Self-supervised learning objective triggers stronger emergent sparsification properties than supervised learning in models having exactly the same architecture.

Universal Existence of Essential Sparsity: We comprehensively evaluate the extent to which essential sparsity exists in the large pre-trained language models (CV and NLP) with a standard pre-trained initialization θ p . For NLP, we studied bert-base, OPT-125M, OPT-350M, OPT-1.3B and used MNLI, QNLI, QQP, SST-2, RTE, and SQuAD1.1 to estimate the effect the removing low-magnitude weights on downstream performance. On the other hand, for CV, we rely on ViT-Base and ViT-Large models and use CIFAR-10, CIFAR-100, TinyImageNet datasets. Figure 2 illustrate the performance drop (y-axis) of various NLP downstream task fine-tuning with mask ( m · θ x % p ), when we remove x % of lowest magnitude weights. Similarly,

Performance drop

0

EssentialSparsity

-5

MAWPS

ASDiv-A

-10

SVAMP

0

102030405060708090

Figure 3 illustrates the effect of pruning x % low-magnitude weights on the fine-tuning performance of CV downstream tasks. Moreover, Figure 4 presents the existence of essential sparsity for a recently proposed sparsity benchmark, named SMC-Bench [75]. It is interesting to observe that essential sparsity exists in all CV and NLP pre-trained models, and ∼ 30 -50% of weights can be removed at free without any significant drop in performance. Note that these masks are identified prior to fine-tuning, on the pre-trained weights and thereby remain the same for all the downstream tasks, indicating no requirement for bookkeeping LTH-type task-specific masks. It is also important to appreciate the significance of removing ∼ 40% (which translates to > 500 million parameters) of OPT-1.3B at no cost without any significant performance drop.

Eseential Sparsity for Structured N:M Sparse Patterns: Considering the demand for practical speedup, we also conduct evaluations to understand the essential sparsity for the hardwarefriendly structured N:M sparsity. A neural network with N:M sparsity satisfies that, in each group of M consecutive weights of the network, there are at most N weights have non-zero values. We studied OPT-350M and used MNLI, QNLI, RTE, SST-2 to estimate the effect of removing low-magnitude weights in structured N:M fashion following [80]. Figure 5 illustrates the performance drop (y-axis) of OPT-350M on

1.5

...

I7INO

1.0

RTE

SST-2

0.5

Performance drop

0.0

0.5

Essential Sparsity

-1.0

1.5

2.0

9:10 (10.49)

5:6 (16.69%)

8:10 (-20.08%)3:4 (25.01%)

4:6 (33.38%)

2:4 (50.00%)

N:M Sparsity Patterns (OPT-350M)

various downstream datasets with multiple N:M sparse mask ( m · θ x % p ). It can be in general observed essential sparsity holds valid for structured N:M sparsity patterns, and a large fraction of low-magnitude weights can be removed at free without significant drop in downstream performance.

0

0

EssentialSparsity

EssentialSparsity

EssentialSparsity

-5

-5

10

10

-10

15

ViT-Base-patch16

-15

ViT-Base-patch16

-15

ViT-Base-patch16

DINO-Base-patch16

DINO-Base-patch16

DINO-Base-patch16

-20

0

10

20

30

40

50

60

70

80

90

20

0

10

20

30

40

50

60

70

80

90

-20

0

10

20

30

40

50

60

70

80

06

105

Layer 1

VT-Self-Supervised

VT-Supervised

105

Layer 2

VT-Self-Supervised

VT-Supervised

105

Layer 3

VT-Supervised

VfT-Self-Supervised

105

Layer 4

ViT-Self-Supervised

VIT-Supervised

103

103

103

103

101

101

101

101

105

Layer 5

VT-Self-Supervised

VT-Supervised

105

Layer6

VT-Supervised

ViT-Self-Supervised

105

Layer 7

VT-Supervised

VfT-Sel-Supervised

105

Layer 8

VT-Sel-Supervised

VIT-Supervised

103

103

103

103

101

101

101

101

105

Layer9

VfT-Self-Supervised

VIT-Supervised

105

Layer10

VT-Self-Supervised

VIT-Supervised

105

Layer 11

VfT-Sef-Supervised

VT-Supervised

105

Layer 12

VfT-Sef-Supervised

VIT-Supervised

103

103

103

103

101

101

101

101

1.0

0.5

0.0

0.5

1.0

1.0

-0.5

0.0

0.5

1.0

1.0

0.5

0.0

0.5

1.0

1.0

-0.5

0.0

0.5

1.0

Essential Sparsity and Sharp Turning Behaviour: In this section, we attempt to bring attention towards the sharp turning point behavior of essential sparsity. Across all our vision and language models and related downstream tasks (Figure 2, 3, & 4), we observed that after a certain removal of a certain proportion of low-magnitude weights, the downstream fine-tuning ability of pre-trained models sharply drops. This is a clear indication that the pre-training knowledge resides in a fraction of high-magnitude weight regions and if we our one-shot pruning touches that region, it non-linearly impacts the transfer learning ability of the pre-trained checkpoint. Our experiments reveal that this sharp-turning behavior is not dependent on model size but on the downstream task . Larger model doesn't implicate that it can be pruned for a higher proportion of low-magnitude weight, without observing the sharp drop. For example, bert-base observes the sharp turning point at around 60% sparsity while OPT-1.3B can not be pruned beyond 40% without observing the sharp performance drop on RTE task, although it has ∼ 10 times more parameters than bert-base . Also, it is interesting to observe that although OPT-125M and bert-base have similar performance count, bert-base illustrate more friendly behavior to pruning, and can be pruned to comparatively higher sparsity ratio than OPT-125M on all our evaluation downstream tasks.

Influence of Supervised versus Self-supervised Learning Objectives: With the recent success of self-supervised learning (SSL) in pre-training large transformer models and its ability to scale to enormous internet-size data, it is interesting to understand how SSL learning objectives favor sparsity. Due to the unavailability of supervised datasets to pre-train BERT-scale models, we switch to Vision transformers, and use ViT-base and its self-supervised version DINO-base which inherit exactly the same architecture. Figure 6 illustrates fine-tuning performance drop of ViT-base and DINO with an increase in the proportion of low-magnitude weight removal from pre-trained checkpoints. Across all tasks, fine-tuning performance doesn't suffer any drop till 30 -40% weights are removed.

One interesting observation is that DINO-base tends to be more robust to low-weight removal, and has comparatively better essential sparsity across all tasks. To further investigate why we can prune

Count

70000

65000

Data Volume:80%

Data Volume:100%

Data Volume:90%

-1

1

0

Data Volume: 100%

Data Volume:90%

Data Volume:80%

80

75

Data Volume 100%

Data Volume 90%

Data Volume 80%

Data Volume 70%

60000

Data Volume:70%

Data Volume: 70%

Performance

Data Volume 60%

Weight

55000

Data Volume:60%

-2

-3

DataVolume:60%

70

Zero

50000

-4

65

45000

-5

60

40000

-6

0

10

20

30

40

50

-7

0

10

20

3040506070

80

90

55

QNLI (AcC.)

QQP (F1)

RTE (Acc.)

TrainingIterations

Sparsity Ratio

DINO-base more than ViT-base , we examine the weight magnitude distribution across the layers of the transformer blocks from the pre-trained checkpoints. Figure 7 presents the normalized layer-wise weight distribution of DINO-base and ViT-base . It can be clearly observed that SSL-based DINO tends to have a weight distribution more friendly to pruning with a significantly large amount of weights concentrated around zero magnitudes. More concretely, we found that DINO-base have ∼ 14% more zero weights than ViT-base , which justifies its higher essential sparsity.

5 How Essential Sparsity Emerges during the Pre-Training Dynamics

Scaling the volume of pre-training data volume is widely believed to favor transfer performance in many downstream tasks in a desirable way [81]. Although, this relationship has been extensively studied recently in many works [81, 82, 83, 84], pre-training data volume role in inducing sparse properties in the large transformers is still unexplored . In this section, we ask an important question: How does the volume of pre-training data impact the emerging sparse patterns in large transformers, and if it improvises their prunability?

To this end, we designed custom experiments to pre-train bert-base from scratch using HuggingFace bookcorpus with a vocabulary size of 30,522. We created 5 different pre-training datasets by randomly selecting 100% , 90% , 80% , 70% , 60% of the training samples from bookcorpus and pretrain for 50k iteration each to ensure that all models receive the same amount of gradients. Note that we maintain exactly same training settings for all models to retain fair comparison and save the pre-training checkpoints every 1000 iterations. We now enlist our key findings related to pre-training data volume and induced sparse patterns in transformers:

We observe an interesting new phenomenon of abrupt sparsification , i.e., the introduction of sudden high sparsity, during pre-training bert-base models, regardless of data size.

We additionally observed a counter-intuitive finding that bert-base trained with a larger amount of pre-training data tends to have better emergence of induced sparsity.

Across all sparsity level, we found bert-base trained with a larger volume of training data, enjoys better prunability, and achieve better performance on the downstream task ( MNLI ).

Figure 8(i) illustrate the emergence of sparse patterns during pre-training of bert-base with different pre-training data volume. We plotted the number of zero weights that emerged in the pre-training checkpoints every 1k iterations. It can be clearly observed that in all settings, the number of zero weights suddenly grow up, indicating the models start abstracting pre-training data knowledge into fewer parameter set. For instance, we find this sharp turn around 22-25k iterations for 100% , 90% , 80% settings while around 40k iterations for 70% , 60% settings. In addition, we found that bert trained with a larger amount of pre-training data tends to have better emergence of induced sparsity. We argue that bert-base tend to learn more generalizable features and develop the capability to abstract knowledge To further investigate how this sparsity emergence further influences the prunability of the pre-trained models, we examined their performance across multiple sparsity ratios (Figure 8(ii)) on MNLI and found it to be in harmony with the magnitude of induced sparsity, i.e., models with high induced sparsity patterns during pre-training tend to perform better when we remove the existing low magnitude weights and fine-tune them on downstream tasks. In dense

2.5

2.5

(H17)

(H17)

0.0

0.0

Per

-2.5

MNLI

P

-2.5

(OMP)

1

-5.0

INO

(OMP)

-5.0

CIFAR-10

QQP

CIFAR-100

7.5

7.5

SST-2

Fashion-MINST

P

-10.0

SQuAD

P

-10.0

TinylmageNet

0

10

20

30

40

50

60

70

80

90

0

10

20

30

40

50

60

70

80

90

SparsityRatio

SparsityRatio

settings on QNLI, QQP, RTE (Figure 8(iii)), we found that an increase in pre-training data volume has favorable benefits on the downstream performance with similar fine-tuning hyperparameters.

6 Essential Sparsity and Lottery Ticket Hypothesis

Lottery Ticket Hypothesis deviates from the convention of after-training pruning, and points to the existence of independently trainable sparse subnetworks from scratch that can match the performance of dense networks. Since its introduction, it has been very successful and widely adopted in compressing small-scale (eg. ResNet family) models. However, it is significantly limited in the practical adoption for large pre-trained models due to train-prune-retrain routine of IMP. In this section, we ask: How effective LTH is within the essential sparsity range, and does it bring any additional privilege which can substantiate its computational cost? Our key takeaway can be summarized as:

Within the essential sparsity range of bert-base and ViT-base , we do not find any significant fine-tuning performance difference of the mask identified by LTH and one by removing lowest magnitude weights (OMP) across multiple downstream tasks.

We surprisingly found the existence of high cosine similarity( > 96% ) across the masks identified by LTH and OMP within the essential sparsity range.

Mask similarity between LTH and OMP decreases with an increase in the sparsity ratio. This corroborates with the benefit of LTH in the high sparsity range, where LTH is able to identify task-fitting masks to retain performance due to repetitive prune and retrain routine.

Figure 9 illustrates the performance difference comparison of fine-tuning of masks identified by LTH and OMP from bert-base and ViT-base across multiple downstream tasks. It can be clearly observed that for sparsity ratio below 50% , we do not find any significant difference in performance between expensive LTH and free-of-cost OMP pruning within the essential sparsity range . A deeper analysis (Figure 10) across the masks from LTH and OMP unveils a surprising observation about the existence of significantly high cosine similarity across. This observation supports the findings in Figure 9 about the matching performance of LTH and OMP and convey a strong and impressive message that with the essential sparsity range, LTH doesn't provide any additional privilege. However, it is important to note that LTH is very effective in the high sparsity range beyond essential sparsity, due to its ability to find task-fitting mask using train-prune-retrain procedure but they tend to be non-transferable across different downstream tasks [20]. Moreover, it can be observed from Figure 10, even in the essential sparsity range, mask similarity across tasks by LTH is comparatively low than OMP based masks due to the strong intertwine of LTH with the downstream tasks.

7 Scaling Essential Sparsity to Modern-Scale LLMs: A Case Study

Large Language Models (LLMs) recently reshape the field of NLP with remarkable performance benefits across a range of complex language benchmarks. However, due to their gigantic size and computational costs; they still remain out of reach for many researchers and small industries. In this section, we investigate the presence of essential sparsity in two popular LLM ( Vicuna-7B & 13B ) with one-shot magnitude pruning. Note that unlike our previous setting, where we perform sparse fine-tuning after the mask is identified; here we do not perform downstream task-specific fine-tuning and evaluate the performance directly 'zero-shot" on the sparsified pre-trained checkpoint.

Figure 11(a) illustrates the performance drop of Vicuna-7B and Vicuna-13B on popular MMLU benchmark (Stem, Humanities, Social Science, Others) when x % of the lowest magnitude weights

1.000

1.000

1.000

1.000

Sparsity Ratio: 0.1

Sparsity Ratio: 0.2

Sparsity Ratio: 0.3

Sparsity Ratio: 0.4

LTH-MNLI

1.0000

0.995

LTH-MNL1

1.0000

0.995

LTH-MNLI

1.0000

0.995

LTH-MNLI

1.0000

0.995

0.990

0.990

0.990

0.990

LTH-QNLI

0.9847

1.0000

0.985

LTH-QNLI

0.9838

1.0000

0.985

LTH-QNL1

0.9826

1.0000

0.985

LTH-QNLI

0.9814

1.0000

0.985

LTH-QQP

0.9805

1.0000

0.980

LTH-QQP

0.9793

0.9781

1.0000

0.980

LTH-QQP

0.97780.9766

1.0000

0.980

LTH-QQP

0.97620.9749

1.0000

0.980

LTH-SST2

0.98790.9861

0.9815

1.0000

0.975

LTH-SST2

0.98720.9853

0.9804

1.0000

0.975

LTH-SST2

0.9862

0.9842

0.9791

1.0000

S/60

LTH-SST2

0.9852

0.9830

0.9774

1.0000

0.975

0.970

0.970

0.970

0.970

OMP

0.9901

0.9880

0.9829

0.9928

1.0000

0.965

OMP

0.9895

0.9874

0.9818

0.9923

1.0000

0.965

OMP

0.9887

0.9865

0.9807

0.9917

1.0000

0.965

OMP

0.9879

0.9854

0.9792

0.9910

1.0000

0.965

LTH-MNLI

NO-H1

Z1SS-H1T

0.960

NNW-HL1

INNO-H11

d00-H11

Z1SS-H1T

WO

0.960

NNW-H1T

NO-H11

LTH-QQP

LTH-SST2

dWO

0.960

NNW-HL1

NO-H11

71SS-H1T

0.960

1.000

1.000

1.000

1.000

Sparsity Ratio: 0.1

Sparsity Ratio: 0.2

Sparsity Ratio: 0.3

Sparsity Ratio: 0.4

0.995

966'0

966°0

966°0

LTH-CIFAR10

1.0000

LTH-CIFAR10

1.0000

LTH-CIFAR10

1.0000

LTH-CIFAR10

1.0000

0.990

0.990

0.990

0.990

LTH-CIFAR100

0.9734

1.0000

0.985

LTH-CIFAR100

0.9705

1.0000

0.985

LTH-CIFAR100

0.9692

1.0000

0.985

LTH-CIFAR100

0.9676

1.0000

0.985

0.980

0.980

0.980

0.980

LTH-Tinylmagenet

0.9679

0.9668

1.0000

0.975

LTH-Tinylmagenet

0.9627

0.9607

1.0000

0.975

LTH-Tinylmagenet

0.9607

88960

1.0000

0.975

LTH-Tinylmagenet

0.9585

0.9567

1.0000

0.975

OMP

0.9819

0.9770

0.9705

1.0000

0.970

OMP

0.9808

0.9754

0.9669

1.0000

0.970

OMP

0.9798

0.9746

0.9651

1.0000

0.970

OMP

0.9785

0.9733

0.9632

1.0000

0.970

0.965

0.965

0.965

0.965

LTH-CIFAR10

LTH-CIFAR100

OMP

0.960

LTH-CIFAR10

LTH-CIFAR100

LTH-Tinylmagenet

OMP

0.960

LTH-CIFAR10

LTH-CIFAR100

LTH-Tinylmagenet

dWO

0.960

LTH-CIFAR10

LTH-CIFAR100

LTH-Tinylmagenet

WO

0.960

0

5 10 15 20 25 30 35 40 45 50 55 60 65 70 75

Sparsity Ratio

50

40

30

20

10

0

10

Essential Sparsity

Magnitude | Vicuna-7B

STEM

Humanities

Social Science

Others

0

5 10 15 20 25 30 35 40 45 50 55 60 65 70 75

Sparsity Ratio

50

40

30

20

10

0

10

Essential Sparsity

SparseGPT | Vicuna-7B

STEM

Humanities

Social Science

Others

0

5 10 15 20 25 30 35 40 45 50 55 60 65 70 75

Sparsity Ratio

50

40

30

20

10

0

10

Essential Sparsity

Magnitude | Vicuna-13B

STEM

Humanities

Social Science

Others

0

5 10 15 20 25 30 35 40 45 50 55 60 65 70 75

Sparsity Ratio

50

40

30

20

10

0

10

Essential Sparsity

SparseGPT | Vicuna-13B

STEM

Humanities

Social Science

Others

are removed from the dense pre-trained checkpoint. It is interesting to see that our essential sparsity observations hold true even for modern LLMs, sending a favorable signal about the hidden existence of high-quality sparse subnetwork which can be identified free of calibration data or post-optimization, in dense pre-trained checkpoints. To further enrich our study, we replaced OMP with the recently proposed SparseGPT [85] and found it to have generally consistent trends with OMP (Figure 11(b)).

Compared to SparseGPT, our research uncovers the surprising simplicity of LLM pruning within a specific sparsity range . The remarkable effectiveness of one-shot, magnitude-based pruning not only establishes a robust baseline but also suggests considerable practical value, for instance, in enabling economical 'on-the-fly" LLM pruning that adapts to fluctuating resource availability during testing. Nonetheless, it's worth noting that more refined pruning strategies like SparseGPT can further push the boundary of essential sparsity and identify better sparse subnetworks at comparatively higher sparsity ratios, albeit at relatively higher expenses. Bridging the performance divide between OMP and SparseGPT remains an intriguing avenue for future investigation.

8 Conclusion

We comprehensively study induced sparse patterns across large pre-trained vision and language transformers. We experimentally validated the ubiquitous existence of essential sparsity across large pre-trained transformer models of varying scales for vision and language (including LLMs), irrespective of the training strategy used for pre-training them. We also present an intriguing emerging phenomenon of abrupt sparsification during the pre-training of transformers and its ability to abstract knowledge within few parameter subset with increasing pre-training data volume. Lastly, we studied the performance of LTH with respect to essential sparsity. Our future work will aim to extend our essential sparsity observations to more gigantic models, and to push for higher sparsity ranges.

References

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems , pages 5998-6008, 2017.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, Zhaohui Yang, Yiman Zhang, and Dacheng Tao. A survey on visual transformer. ArXiv , abs/2012.12556, 2020.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv'e J'egou. Training data-efficient image transformers & distillation through attention. In ICML , 2021.

Zhiyuan Mao, Ajay Jaiswal, Zhangyang Wang, and Stanley H. Chan. Single frame atmospheric turbulence mitigation: A benchmark study and a new physics-inspired transformer model. ArXiv , abs/2207.10040, 2022.

Minghang Zheng, Peng Gao, Renrui Zhang, Xiaogang Wang, Hongsheng Li, and Hao Dong. End-to-end object detection with adaptive clustering transformer. ArXiv , abs/2011.09315, 2021.

Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam M. Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In ICML , 2018.

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems , 32, 2019.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937 , 2018.

Ajay Jaiswal, Liyan Tang, Meheli Ghosh, Justin Rousseau, Yifan Peng, and Ying Ding. Radbertcl: Factually-aware contrastive learning for radiology report classification. Proceedings of machine learning research , 158:196-208, 2021.

Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. End-to-end open-domain question answering with bertserini. arXiv preprint arXiv:1902.01718 , 2019.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461 , 2018.

Ming Ding, Chang Zhou, Qibin Chen, Hongxia Yang, and Jie Tang. Cognitive graph for multi-hop reading comprehension at scale. arXiv preprint arXiv:1905.05460 , 2019.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 , 2022.

Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 , 2020.

Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot, 2023.

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations , 2019.

Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. Advances in neural information processing systems , 33:15834-15846, 2020.

Ajay Jaiswal, Haoyu Ma, Tianlong Chen, Ying Ding, and Zhangyang Wang. Spending your winning lottery better after drawing it, 2022.

Lu Yin, Shiwei Liu, Fang Meng, Tianjin Huang, Vlado Menkovski, and Mykola Pechenizkiy. Lottery pools: Winning more by interpolating tickets without increasing training or inference cost. arXiv preprint arXiv:2208.10842 , 2022.

Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing early-bird tickets: Towards more efficient training of deep networks. arXiv preprint arXiv:1909.11957 , 2019.

Ajay Kumar Jaiswal, Haoyu Ma, Tianlong Chen, Ying Ding, and Zhangyang Wang. Training your sparse neural network better with any mask. In International Conference on Machine Learning , pages 9833-9844. PMLR, 2022.

Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations , 2019.

Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems , 33:5824-5836, 2020.

Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. The lottery ticket hypothesis at scale. arXiv preprint arXiv:1903.01611 , 2019.

Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Zhangyang Wang, and Jingjing Liu. Earlybert: Efficient bert training via early-bird lottery tickets. arXiv preprint arXiv:2101.00063 , 2020.

Sai Prasanna, Anna Rogers, and Anna Rumshisky. When bert plays the lottery, all tickets are winning. arXiv preprint arXiv:2005.00561 , 2020.

Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end-to-end exploration. Advances in Neural Information Processing Systems , 34:19974-19988, 2021.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 , 2020.

Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems , pages 598-605, 1990.

Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In International Conference on Learning Representations , 2016.

Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28 , pages 1135-1143. Curran Associates, Inc., 2015.

Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In D. S. Touretzky, editor, Advances in Neural Information Processing Systems 2 , pages 598-605. Morgan-Kaufmann, 1990.

Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision , pages 2736-2744, 2017.

Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision , 2017.

Hao Zhou, Jose M Alvarez, and Fatih Porikli. Less is more: Towards compact cnns. In European Conference on Computer Vision , pages 662-677. Springer, 2016.

Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440 , 2016.

Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 20378-20389. Curran Associates, Inc., 2020.

Hao Jiang, Ke Zhan, Jianwei Qu, Yongkang Wu, Zhaoye Fei, Xinyu Zhang, Lei Chen, Zhicheng Dou, Xipeng Qiu, Zikai Guo, et al. Towards more effective and economic sparsely-activated model. arXiv preprint arXiv:2110.07431 , 2021.

Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems , 5, 1992.

Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. Advances in Neural Information Processing Systems , 30, 2017.

William Finnoff, Ferdinand Hergert, and Hans Georg Zimmermann. Improving model selection by nonconvergent methods. Neural Networks , 6(6):771-783, 1993.

Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through l _ 0 regularization. arXiv preprint arXiv:1712.01312 , 2017.

Jian-Hao Luo and Jianxin Wu. Autopruner: An end-to-end trainable filter pruning method for efficient deep model inference. Pattern Recognition , 107:107461, 2020.

Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsification. In Advances in Neural Information Processing Systems 33 pre-proceedings , 2020.

Vikash Sehwag, Shiqi Wang, Prateek Mittal, and Suman Jana. Hydra: Pruning adversarially robust neural networks. Advances in Neural Information Processing Systems , 33, 2020.

Yihua Zhang, Yuguang Yao, Parikshit Ram, Pu Zhao, Tianlong Chen, Mingyi Hong, Yanzhi Wang, and Sijia Liu. Advancing model pruning via bi-level optimization. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878 , 2017.

Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09574 , 2019.

Tao Lin, Sebastian U. Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. In International Conference on Learning Representations , 2020.

Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. Advances in Neural Information Processing Systems (NeurIPs). , 2021.

Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications , 9(1):2383, 2018.

Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning , 2019.

Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.04840 , 2019.

Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning , pages 2943-2952. PMLR, 2020.

Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do we actually need dense over-parameterization? in-time over-parameterization in sparse training. arXiv preprint arXiv:2102.02887 , 2021.

Jonathan Schwarz, Siddhant Jayakumar, Razvan Pascanu, Peter E Latham, and Yee Teh. Powerpropagation: A sparsity inducing weight reparameterisation. Advances in neural information processing systems , 34:28889-28903, 2021.

Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations , 2020.

Hidenori Tanaka, Daniel Kunin, Daniel LK Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In Advances in Neural Information Processing Systems 33 pre-proceedings , 2020.

Pau de Jorge, Amartya Sanyal, Harkirat S Behl, Philip HS Torr, Gregory Rogez, and Puneet K Dokania. Progressive skeletonization: Trimming more fat from a network at initialization. arXiv preprint arXiv:2006.09081 , 2020.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Ofir Zafrir, Ariel Larey, Guy Boudoukh, Haihao Shen, and Moshe Wasserblat. Prune once for all: Sparse pre-trained language models. arXiv preprint arXiv:2111.05754 , 2021.

Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models. arXiv preprint arXiv:2203.07259 , 2022.

Dongkuan Xu, Ian EH Yen, Jinxi Zhao, and Zhibin Xiao. Rethinking network pruning-under the pre-train and fine-tune paradigm. arXiv preprint arXiv:2104.08682 , 2021.

François Lagunas, Ella Charlaix, Victor Sanh, and Alexander M Rush. Block pruning for faster transformers. arXiv preprint arXiv:2109.04838 , 2021.

Qingru Zhang, Simiao Zuo, Chen Liang, Alexander Bukharin, Pengcheng He, Weizhu Chen, and Tuo Zhao. Platon: Pruning large transformer models with upper confidence bound of weight importance. In International Conference on Machine Learning , pages 26809-26823. PMLR, 2022.

Elias Frantar, Eldar Kurtic, and Dan Alistarh. M-fac: Efficient matrix-free approximations of second-order information. Advances in Neural Information Processing Systems , 34:1487314886, 2021.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022.

Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4938-4947, 2020.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 , 2016.

Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N , 7(7):3, 2015.

Shiwei Liu, Tianlong Chen, Zhenyu Zhang, Xuxi Chen, Tianjin Huang, Ajay Jaiswal, and Zhangyang Wang. Sparsity may cry: Let us fail (current) sparse neural networks together! arXiv preprint arXiv:2303.02141 , 2023.

Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. MAWPS: A math word problem repository. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 1152-1157, San Diego, California, June 2016. Association for Computational Linguistics.

Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing english math word problem solvers. arXiv preprint arXiv:2106.15772 , 2021.

Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191 , 2021.

William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research , 23(1):5232-5270, 2022.

Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning n: m fine-grained structured sparse neural networks from scratch. arXiv preprint arXiv:2102.04010 , 2021.

Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre-training. arXiv preprint arXiv:2110.02095 , 2021.

Donghyun Kim, Kaihong Wang, Stan Sclaroff, and Kate Saenko. A broad study of pre-training for domain generalization and adaptation. In European Conference on Computer Vision , 2022.

Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2661-2671, 2019.

Ziquan Liu, Yi Xu, Yuanhong Xu, Qi Qian, Hao Li, Antoni Chan, and Rong Jin. Improved finetuning by leveraging pre-training data: Theory and practice. arXiv preprint arXiv:2111.12292 , 2021.

Elias Frantar and Dan Alistarh. Massive language models can be accurately pruned in one-shot. arXiv preprint arXiv:2301.00774 , 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a comprehensive study on the induced sparse patterns across multiple large pre-trained vision and language transformers. The authors propose the existence of \"essential sparsity\" and present intriguing findings on abrupt sparsification during the pre-training of transformers and the effect of pre-training data on knowledge condensation.\n\nSTRENGTHS:\nThe main strength of this paper is that it validates the ubiquitous existence of essential sparsity across large pre-trained transformer models of varying scales for vision and language, irrespective of the training strategy used for pre-training them.\n\nThe authors used various datasets and models to validate the existence of \"essential sparsity\" and presented intriguing findings on abrupt sparsification during the pre-training of transformers and the effect of pre-training data on knowledge condensation. The authors also compared the performance of models with and without sparsification and found that one-shot sparsification without re-training does not significantly affect the performance of downstream tasks. In Section 6, the authors further carefully analyzed the connection between LTH and essential sparsity, and showed the former to become potentially less necessary or relevant in larger models. Those are valuable insights of broad interest to the sparsity research community.\n\nBased on the methodology and results presented in the paper, the experiment presented is sound enough to support the idea of the existence of essential sparsity in large pre-trained models (bert-base, OPT-125m, OPT-350m, OPT-1.3B). In particular, the authors present several quite surprising findings related to the existence of essential sparsity in large pre-trained transformer models. \n-\tFirstly, the authors found that BERT suddenly becomes heavily sparse after a certain number of training iterations, which is unobserved before and not well-understood. This finding suggests that there may be underlying “phase transition”-like mechanisms in the pre-training dynamics of transformers that are responsible for inducing sparsity, and further research is needed to understand it.\n-\tSecondly, the authors also found that BERT trained with a larger amount of pre-training data tends to have a better ability to condense knowledge in relatively fewer parameters. This finding is counter-intuitive because one would expect that increasing the amount of pre-training data would lead to an increase in the number of parameters required to capture the additional information.\n-\tThirdly, the authors found that self-supervised learning (SSL) objectives trigger stronger emergent sparsification properties than supervised learning (SL). This finding is also intriguing because one would expect that supervised learning, which provides more explicit information to the model, would lead to better knowledge condensation.\n\nWEAKNESSES:\n-\tOne main downside of this paper is that the paper did not report any result on hardware-friendly sparsity such as N:M, nor it discussed any GPU run time benefit from the induced essential sparsity (if any). While one can understand the study is mainly conceptual, just like the original LTH, I believe the real hardware support is especially important for LLM pruning/sparsity research due to their exploding costs\n-\tMore baselines are desired – currently only LTH is reported. Would essential sparsity meaningfully outperform the simplest baseline of random pruning? How it compares with the pruning result of SparseGPT [18]?\n-\tIn both SparseGPT [18] and Sparsity-May-Cry [75], it was found that larger LLMs are harder to prune. This paper seems to pinpoint the opposite conclusion. Could the authors elaborate why their conclusions seem to contradict [18,75], or not?\n-\tSparsity has more benefits beyond efficiency, such as few-shot transfer, robustness to noisy label or other distribution shifts. Many of those were previously demonstrated under LTH or dynamic sparse training settings. I will be curious to see if essential sparsity will own the same merits too.\n\nQUESTIONS:\nPlease see the Weaknesses\n\nLIMITATIONS:\nNot discussing more hardware-friendly sparse patterns", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis research paper focuses on the following notion for large pre-trained models: \"essential sparsity\", the idea that a sharp drop in fine-tuning performance occurs after one-shot pruning relative to the level of sparsity. The authors propose that large and overparameterized models can be pruned without additional computational expense, showing this to be true across a range of tasks in both computer vision and natural language processing. Additionally, the authors found an interesting occurrence of abrupt sparsification during pre-training, indicating that models trained with larger datasets tend to achieve knowledge abstraction with fewer parameters. Their findings also revealed that self-supervised learning objectives tend to trigger stronger emergent sparsification properties than supervised learning. They argue that identifying and understanding these inherent high-quality sparse patterns could make fine-tuning large models more practical and environmentally friendly.\n\nSTRENGTHS:\nThe paper makes a number of interesting observations about the notion of essential sparsity, its relation to \"winning ticket\" networks (see LTH), the idea that better knowledge abstraction may be feasible with fewer parameters (as long as we train with more data), and that self-supervised learning tends to have better emergent sparsification than supervised learning.\n\nWEAKNESSES:\nUnfortunately, these interesting observation mentioned in the Strengths section, are also too \"empirical\" or anecdotally shown in the paper. It is hard to be convinced that these are general results -- and they are not substantiated with any form of of theory or at least some explanations/justifications based on results from other papers. \n\nIn general, I would describe this paper as \"it is moving in a good direction but it is not ready for publication yet\" -- at least not at the top-tier conference of deep learning.\n\nQUESTIONS:\nThere is a completely different approach to get the benefits of essential sparsity -- namely to start with and train a sparse network. That would be even less computationally intensive than training a dense network and then doing OMP on that. \nAre the authors familiar with methods such as SynFlow, PHEW, Synflow++, etc that do pruning before training? If not, I suggest that they also consider those and compare the performance they get from OMP versus those networks. \n\n The two more interesting observations of the paper (the idea that better knowledge abstraction may be feasible with fewer parameters (as long as we train with more data), and that self-supervised learning tends to have better emergent sparsification than supervised learning) are presented very briefly, without a sufficiently deep analysis in my opinion.\n\nLIMITATIONS:\nOne limitation of course is that the claims of the paper are not substantiated in the case of really large foundation models used today -- but of course it is hard for academic researchers to experiment with the training of such models.\n\nAnother issue is that the paper does not explicitly state the limitations of the proposed method. For example, in the end of section 6 there are some rather \"hidden statements\" about the benefits of LTH and IMP, which is a very important point.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper defines essential sparsity and conducts various experiments to analyze the sparsity property of pre-trained models.

STRENGTHS:
1. This paper investigates the potential of directly sparsifying the pre-trained models
2. CV and NLP models are both explored.

WEAKNESSES:
1. Evidence on large models is lacking.
2. Some conclusions are not new, e.g., sharp dropping point.

QUESTIONS:
What about the conclusions on large models like LLaMA 65B or Bloom 175B?

LIMITATIONS:
The insights brought from this paper is limited.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper postulates existence of \"Essential Sparsity\" in large pre-trained transformer models. The \"Essential Sparsity\" is defined by the paper as a sparsity threshold beyond which further pruning of weights leads to a large performance drop. It considers (1) one-shot pruning and (2) lottery ticket pruning of networks with fine-tuning. Experiments are conducted on both large language and vision transformer models.\n\nSTRENGTHS:\nWhile the sparsity has been studied a lot in the neural network literature, this work studies the sparsity for pre-trained language and vision models. Given the wide adoption of these networks, the study is important. The paper covers a number of experiments for large vision as well as language models. The comparison of self-supervised vs fully supervised models is interesting.\n\nWEAKNESSES:\n1. Existence of sparsity in transformers in both language [a,b] and vision [c] has been discussed in the prior literature. The drop of performance in networks beyond certain threshold has also been covered in the literature. I do believe the current work is the first one to formally define it, however it will be good if the work can put the contributions in proper context.\n\n2. Many of the \"surprising\" findings of the work appear to be over-claimed. For instance,\n\n    i. The abrupt drop in the performance of networks beyond certain sparsity thresholds is shown by the original LTH paper and the current work extends it to pre-trained network.\n\n    ii. The similarity of sparse masks for various downstream tasks is also not very surprising given that the networks was pre-trained on a large massive dataset and the downstream tasks comprise of very small similar sets of data (often even the subset of pre-training data).\n\n    iii. L50 (and L66) says that \"a significant proportion of the weights in them can be removed for free\" while it is hardly free given the network has been heavily pre-trained.\n\n3. The mathematical definition of Essential Sparsity is somewhat unclear (what exactly does $1-\\frac{||m||_0}{|m|}$ represent?). Also calling it essential for the network is a bit misleading. I agree dropping it would hurt the performance of the network but it is not \"essential\" for the network to perform well. A fully dense network usually performs the best.\n\n4. Minor fix - Labels for x-axis is missing in most of the figures.\n\nReferences:\n\na. Chen, Tianlong, et al. \"The lottery ticket hypothesis for pre-trained bert networks.\" NeurIPS 2020\n\nb. Dettmers, Tim, et al. \"Llm. int8 (): 8-bit matrix multiplication for transformers at scale.\" NeurIPS 2022\n\nc. Girish, Sharath, et al. \"The lottery ticket hypothesis for object recognition.\" CVPR. 2021.\n\nQUESTIONS:\nWhile some of the analyses of the paper is interesting, I would appreciate authors comments on above weaknesses.\n\nLIMITATIONS:\nThe paper doesn't cover discuss the limitations of the study in details. For example, while the sparsity and efficiency of models both during training and inference is an important problem, it should be kept in mind that unstructured sparsity alone is not good enough to obtain speed gains. There is no discussion of potential speed gains with the unstructured sparsity (this doesn't necessarily mean FLOPs but speed improvement during inference/training), or training time of different networks, etc.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper delves into the natural sparsity of large-scale pretrained transformers that could be found without additionally training the models with sparsity optimization targets. The authors find that when simply low-magnitude weights are zeroed out, 1) up to a certain sparsity level,  the pruned transformers do not exhibit performance drop on downstream tasks after fine-tuning. 2) the sparsity of weights abruptly increases at certain training iterations 3) simple magnitude-based pruning shows similar performance as the Lottery Ticket Hypothesis method up to a certain sparsity level. The phenomenon is observed across many different LLM and vision transformer architectures. Also interestingly, self-supervised weights are more sparse than supervised weights.\n\nSTRENGTHS:\nThis paper presents interesting findings on the nature of pretrained weights of large-scale models based on substantial amount of experimental grounds. The authors emphasize that the use of essential sparsity to prune weights can be done without repetitive train-prune-retrain routine. Considering the massive scale of recent large-scale transformers, it is a promising and practically efficient approach to find and utilize the sparsity of transformers. \n\nIn addition, this paper opens up several research topics to be studied. I believe they will be impactful.\n- Sparsity pattern from self-supervised learning and supervised learning\n- Sparsity pattern change by training on larger-scale datasets\n- Acceleration of large-scale transformers by pruning weights\n\nWEAKNESSES:\n- Practical gains after finding the sparsity pattern\n\nIn contrast to learning-based pruning methods (such as LTH), the essential sparsity has the raw pattern of weight magnitude. Although it is interesting to find the sparsity is there by nature, as mentioned in L90-91, such patterns can’t be directly used to bring actual acceleration of neural networks. Training-based methods can set optimization goal on sparsity patterns to alter the sparsity pattern to make the neural network accelerable, however, such approach seems to be difficult for essential sparsity. While it is easier to find the sparsity pattern with essential sparsity, what would be the practical gains of the proposed method after finding the sparsity?\n\n- Why does sparsity occur?\n\nThe authors present interesting comparison of impact of sparsity on model accuracy on various conditions - model size & architecture, supervised vs self-supervised learning. It is interesting to know of such results, but why does such differences happen? How can we understand the phenomenon? \n\n- typos\n\nL36 pre0trained -> pre-trained\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nNo negative societal impact is expected.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-29"}