Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!
Authors: Shiwei Liu
Tianlong Chen
Zhenyu Zhang
Xuxi Chen
Tianjin Huang
AJAY KUMAR JAISWAL
Zhangyang Wang
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Sparse Neural Networks
Benchmark
Sparsity
Neural Network Pruning

Abstract: Sparse Neural Networks (SNNs) have received voluminous attention predominantly due to growing computational and memory footprints of consistently exploding parameter count in large-scale models. Similar to their dense counterparts, recent SNNs generalize just as well and are equipped with numerous favorable benefits (e.g., low complexity, high scalability, and robustness), sometimes even better than the original dense networks. As research effort is focused on developing increasingly sophisticated sparse algorithms, it is startling that a comprehensive benchmark to evaluate the effectiveness of these algorithms has been highly overlooked. In absence of a carefully crafted evaluation benchmark, most if not all, sparse algorithms are evaluated against fairly simple and naive tasks (eg. CIFAR-10/100, ImageNet, GLUE, etc.), which can potentially camouflage many advantages as well unexpected predicaments of SNNs. In pursuit of a more general evaluation and unveiling the true potential of sparse algorithms, we introduce “Sparsity May Cry” Benchmark (SMC-Bench), a collection of carefully-curated 4 diverse tasks with 10 datasets, that accounts for capturing a wide range of domain-specific and sophisticated knowledge. Our systemic evaluation of the most representative sparse algorithms reveals an important obscured observation: the state-of-the-art magnitude- and/or gradient-based sparse algorithms seemingly fail to perform on SMC-Bench when applied out-of-the-box, sometimes at significantly trivial sparsity as low as 5%. The observations seek the immediate attention of the sparsity research community to reconsider the highly proclaimed benefits of SNNs. We further conduct a thorough investigation into the reasons for the failure of common SNNs. Our analysis points out that such failure is intimately related to the “lazy regime” of large model training, which hints us with stronger pruning recipes that alleviate the failure on SMC-Bench (though still more or less suffering). By incorporating these well-thought and diverse tasks, SMC-Bench is designed to favor and encourage the development of more scalable and generalizable sparse algorithms. We open-source SMC-Bench to assist researchers in building next-generation sparse algorithms that scale and generalize: https://github.com/VITA-Group/SMC-Bench.

Published as a conference paper at ICLR 2023
SPARSITY MAY CRY: LET Us FAIL (CURRENT) SPARSE NEURAL NETWORKS TOGETHER!
Shiwei Liu',, Tianlong , Zhenyu Zhang', Xuxi Chen', Tianjin Huang", Ajay Jaiswal'. Zhangyang Wang'
'University of Texas at Austin shiwei. liueaustin utexas eduu; t.huang@tue.ni ftianleng. chen, zhenyu. zhang, xxchen, a jay jasswal,wwwa,,laoo attpaallaalleeee atw.aaa.t....ssaaeetttt....o..
"Eindhoven University of Technology
ABSTRACT
Sparse Neural Networks (SNNs) have received voluminous attention predominantly due to growing computational and memory footprints of consistently exploding parameter count in large-scale models. Similar to their dense counterparts, recent SNNs generalize just as well and are equipped with numerous favorable benefits (e.g., low complexity, high scalability, and robustness), sometimes even better than the original dense networks. As research effort is focused on developing increas- ingly sophisticated sparse algorithms, it is startling that a comprehensive benchmark to evaluate the effectiveness of these algorithms has been highly overlooked. absence of a carefully crafted evaluation benchmark, most if not all, sparse al- gorithms are evaluated against fairly simple and naive tasks (eg. CIF1R- ImageNet. GLUE, etc.), which can potentially camouflage many advantages well unexpected predicaments of SNNs. In pursuit of a more general evaluation and unveiling the true potential of sparse algorithms, we introduce "Sparsity May Cry" Benchmark (SMCC-ench), a collection of carefully- curated 4 diverse tasks with 10 datasets, that accounts for capturing a wide range of domain-specific and sophisticated knowledge. Our systemic evaluation of the most representative sparse algorithms reveals an important obscured observation: the state-of-the-an magnitude- and/or gradient-basee sparse algorithms seemingly fail to perform on SMC-Bench when applied out-of-the-box, sometimes at significantly trivial sparsity as low as 5%. The observations seek the immediate attention of the sparsity re- search community to reconsider the highly proclaimed benefits of SNNs. We further conduct thorough investigation into the reasons for the failure of common SNNs. Our analysis points out that such failure is intimately related to the "lazy regime" large model training, which hints us with stronger pruning recipes that alleviate the failure on SMC-Bench (though still more or less suffering). By incorporating these well-thought and diverse tasks, SMC-BBech is designed favor and encourage the development of more scalable and generalizable sparse algorithms. We open-source SMC-Bench assist researchers building next-generation sparse algorithms that scale and generalize: https:/ //ggtgttt.b////////////. www.tttw.ooooooo.tooooooootttttt www.////////.. GreaAAGrruup/ae SMC SMC- Bench.
INTRODUCTION
Sparse Neural Networks (SNNs) are no stranger to the deep learning community (Liu & Wang, 2023). but recently they have received stupendous attention in the era of transformers (eg. BERT, GPT, ViT, CLIP, etc.), when the parameter count frequently measured billions rather than millions. Due to the consistent efforts of sparsity researchers. SNNs have ushered enormous breakthroughs and can generalize just as well as original dense networks, and it feasible to procure them after training (Frankle & Carbin, 2019; Sanh al., 2020; Chen et al., 2020; Frankle et al., 2020), during training (Zhu & Gupta, 2017: Gale et al., 2019; Liu et al., 20)1b), and even before training (Mocanu al., 2018; Lee et al., 2019; Liu et al., 2022) their dense counterparts using pruning. Apart from well-known efficiency benefits, surprisingly, SNNs also enjoy auxiliary benefits such as adversarial robustness (Guo al., 2018; Özdenizei & Legenstein, 2021; Chen al., 2022), out-of-distribution
These authors contributed equally this work.

---

Published as a conference paper at ICLR 2023
generalization (Zhang et al., 2021; Diffenderfer et al., 2021), and uncertainty estimation (Liu et al., 2021a), etc. Despite the multi-dimensional success of numerous sparse algorithmss, startlingly, our extensive survey across over 100 recent SNN papers within 2015-2022 unveils multiple daunting issues regarding evaluation datasets and protocols blindly followed within the sparsity community, that may significantly impede future progress if left unacknowledged.
Firstly, the vast majority of current work on SNNs is narrowly evaluated, i.e., only targeting a single or a few tasks (usually on image classification and sometimes on language understanding) on which SNNs have already proven their proficiency (Gale et al., 2019; Frankle & Carbin, 2019). Surprisingly. 79 papers out of our carefully selected 100 papers on SNNs, evaluate sparse models merely on a single task, where 72 out of them evaluate image classification. Secondly, people are obsessed with evaluating SNNs on well-understood datasets, including but not limited to MNIST (LeCun, 1998) (26 papers), CIFAR- 10/100 (Krizhevsky et al., 2009) (59 and 37 papers, respectively). ImageNet (Deng et al., 2009) (62 papers), and GLUE (Wang et al., 2018) (9 papers), where deep neural networks have already exceeded the human-equivalent performance (refer to Appendix D for more details). For instance, even though ImageNet has been considered rather challenging task over years, very high accuracy (>90%) has been reported many times (Yu et al., 2022; Wortsman et al., 2022; Zhai et al., 2022). Such relatively restricted evaluations with "nearly saturated performance" limit the scope of sparse neural networks and are potentially ill-suited to identify new and unexpected capabiliites of SNNs.
Issues with current evaluation paradigm:
Addressing the aforementioned limitations of current SNN evaluation protocols is pressing need for the community. To this end, we assemble a large-scale, fairly arduous, and diverse benchmark for sparse neural networks- "Sparsity May Cry" Benchmark (or briefly SMC-Bench). Specifically, we consider a broad set of tasks including complex reasoning, multilingual translation, and protein prediction, whose content spans multiple domains. Those tasks require a vast amount of commonsense knowledge, solid mathematical and scientific backgrounds to solve even for humans. Note that none of the datasets in SMC-Bench was created from scratch for the benchmark, we rely on pre-existing datasets as they have been agreed by researchers as challenging, interesting, and of high practical value. We rigorously measure the performance of state-of-the-at (SOTA) pruning and sparse training approaches (in their most common, basic settings) on SMC-Bench, to understand the potential of SNNs to scale and generalize. Our key observations and contributions can be unfolded as:
We present "Sparsity May Cry" Benchmark, to re-define the evaluation protocols for sparse neural networks and facilitate a comprehensive assessment of SOTA sparse algorithms. The premise of SMC-bench is to develop suite of large-scale, challenging, realistic, and diverse tasks and datasets that can empower the rigorous advancements in the community.
SMC-Bench unveils a critical and startling observation all of the SOTA sparse algorithms seem to fail on SMC-Bench "out-of-the-box", sometimes at significantly trivial sparsity egg., 5%. Note that the failure does not appear specific to one sparsification approach but unanimously across all approaches we evaluated. This observation alarmingly demands the attention of the sparsity community to reconsider the highly proclaimed benefits of SNNs.
We conduct extensive experiments across representative SNNs produced by various SOTA pruming and sparse training approaches on S.C--Bench and we summarize our findings: 1 Model prunability is intimately related task difficulty: models trained difficult tasks suffer more from pruning compared to easier tasks; @ The success of before-training sparsi- fication (sparse training or pruning at initialization) hard to generalize in more complex scenarios; 2 Iterative magnitude pruning (IMP) does not necessarily generalize better than one-shot pruning (OMP) or during-training pruning: O Despite performance difference, different magnitude- based pruning approaches lead to extremely similar layerwise sprasities.
We further carry a comprehensive investigation into the potential causes of SNN failures on SMC-Bench. Our analysis suggests that the failure of the existing sparse algorithms might be due the "lazy regime" dynamics emerging in sufficiently overparameterized models (Chizat et al., 2019; Malladi et al., 2022). Inspired by this finding, we hypothesize and confirm that the second-order pruning approaches, i... oBERT (Kurtic et al., 2022), are more reliable pruning approaches for SMC-Bench, which yield relatively more promising performance on SMC- MMC-Bench in Appendix C.
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORK
2.1 ADVANCES IN SPARSE NEURAL NETWORKS
Post-Training, SNNs refer neural network where certain portion of its components (e.g.. weights, neurons, filters, and attention heads) have exactly zero values. The initial purpose of SNNs retrospectively to accelerate model at inference time (a.k.a., post-training sparsification; Mozer & Smolensky (1989); LeCun et al. (1990)). Thanks to the over-parameterizzttio property of deep neural networks, we can dramatically prune deep neural networks to smaller sizes with marginal loss of performance. Post-training sparsification has been well studied and results in various mature criteria that can be generally categorized into zero-order methods (magnitude-baseed Han et al. (2015)), first-order methods (gradient-based, Molchanov et al. (2016); Sanh et al. (2020); Jiang et al. (2021)). )), and second-order methods (Hessian-basede LeCun et al. (1990); Hassibi & Stork (1992); Dong et al. (2017)). Second-order sparsification usually achieves higher performance than the other two but is also more expensive due to the full Hessian calculation. Fortunately, many approaches have been proposed to efficiently approximate Hessian (Zeng& & Urtasun, 2018; Wang et al., 2019; Singh & Alistarh, 2020). The Lottery Ticket Hypothesis (LTH) adopts iterative magnitude pruning (IMP) on fully trained networks and finds subnetwork initialization that can be re-trained in isolation to match the original dense networks. Renda et al. (2020) further found that instead of re-training with the initial weights, re-training with the final weights achieves better performance. With the rise of large language models (LLMs), newer post-training pruning methods have emerged which aim to improve the affordability of these models (Sanh et al., 2020; Chen al., 2020; Zafrir et al., 2021; Kurtic et al., 2022; Xu et al., 2021; Lagunas et al., 2021; Zhang et al., 2022; Frantar et al., 2021).
During-Traininn Training. During-training sparsification (Finnoff et al., 1993) a cheaper option, compared to sparsify a fully converged model. Approaches of during-training sparsification usually train dense network for some time and then gradually sparsify the network with some schedules and end up with sparse model with target sparsities. Zhu & Gupta (2017); Gale et al. (2019); Lin et al. (2020); Liu et al. (2021b) are highlight approaches that gradually prune networks during training and meanwhile allow the pruned weights to be reactivated in case of inaccurate pruning. Another direction of during- training sparsification is adding sparsifying penalties such as (grouped) Lu and L, norm to the loss function, which will punish the unimportant weights to zero, leading to sparse weights (Louizos et al., 2018; Luo & Wu, 2020; Savarese et al., 2020).
Prior-Training. Recently, foundation models (Brown al., 2020; Chowdhery et al., 2022; Ramesh et al., 2022) have demonstrated promising quantitative improvement and new qualitative capabilities with increasing scale (Zhang et al., 2020b). Along with the scaling of model size and data size, the training resources of these foundation models also get outrageous. To accelerate training, we need to sparsify models before training. LTH unveils the possibility to find SNNs at initialization that can match their dense counterparts, even though it uses post- training pruning to find them. At the same time, sparse training (Mocanu et al., 2018; Mostafa & Wang, 2019; Dettmers & Zettlemoyer, 2019; Evci et al., 2020; Liu et al., 2021c; Schwarz et al., 2021) was proposed that can train randomly- initialized sparse neural network from scratch while dynamically optimizing the sparse connectivity with promising performance. Instead of randomly initializing sparse networks, one iteration (Lee et al., 2019; Wang et al., 2020) or a few iterations (Tanaka et al., 2020; de Jorge et al., 2021) of training can be utilized to guide the search for sparse networks before training.
2.2 BENCHMARKING IN SPARSE NEURAL NETWORKS
Gale et al. (2019) rigorously evaluated variational dropout (Molchanov et al., 2017), 4n regular- izaion (Louizos et al., 2018), and GMP (Zhu & Gupta, 2017) two large-scale tasks. They demonstrated that the appealing performance advantages of variational dropout and lo regularization cannot generalize to large-scale tasks whereas simple magnitude pruning performs surprisingly well. Liu et al. (2018) examined two pipelines: training from scratch and fine-tuning, concluding that fine-tuning pruned model only gives comparable or worse performance than training from scratch. Blalock et al. (2020) provided a comprehensive literature review on SNNs and found that pruning papers rarely make direct and controlled comparisons. Frankle et al. (2021) assessed the efficacy of various pruning at initialization approaches and attribute their inferior performance to their insensitivity to weight shuffling and re-initialization. Liu al. (2022) re-evaluated the performance of various random pruning before training and found that sparsely training randomly pruned network from scratch can surprisingly match the performance of its dense equivalent. These papers shed light on the behavior of SNNs and discover important research problems for future work.
3

---

Published as a conference paper at ICLR 2023
3 SMC-BENCH
SMC-Bench is crafted for evaluating if all proclaimed benefits of SNNs can "scale and generalize". It consists of 4 diverse and difficult tasks, including commonsense reasoning, arithmetic reasoning, multilingual translation, and protein prediction, with 10 datasets collected from prior work and open- source GitHub repositories. To investigate if there is a strong correlation between model prunability and task difficulty, we choose multiple datasets with different degrees of difficulty.
3.1 COMMONSENSE REASONING
Commonsense reasoning task asks commonsense questions about the world involving complex semantics that often require rich common sense and background knowledge. We consider three commonly used datasets for commonsense reasoning. (1) RACE (Lai al., 2017) contains near 28,000 passages and 100,000 questions collected from the English exams for Chinese students in middle (RACE-M) and high school (RACE-H). (2) WinoGrande (Sakaguchi et al., 2021) is a modified version of the Winograd Schema Challenge (WSC) benchmark (Levesque et al., 2012) with enhanced scale and hardness, containing 44k problems. (3) Commonsense QA (CSQA) (Talmor et al., 2018) 8) is challenging dataset containing 12,247 multiple-choice questions where one source concept and three target concepts are first extracted from ConceptNet (Speer et al., 2017) based on which the Crowd-works In general, CSQA harder than WinoGrande and RACE, with ceiling human performance of 89%, 94%, and 95%, respectively.
asked to author multiple-choice questions with two additional distractors.
3.2 ARITIMETIC REASONING
Arithmetic reasoning poses question of math problem and the model is asked to generate a mathematical equation that can be evaluated to get the answer. We consider the following three math word problem datasets: (1) the widely used MAWPS benchmark
et al., 2016) composed of 2,373 problems; (2) the arithmetic subset of ASDiv (Miao et al., 2021) ASDiv-A with 1,218 math problems; (3) the more challenging SVAMP (Patel et al., 2021) dataset which is created by applying complex types of variations to the samples from ASDiv-A. The task difficulty monotonically increases from MAWPS to ASDiv-A, and to SVAMP.
3.3 PROTEIN THERMOSTABILITY PREDICTION
Maintaining a stable 3D structure is an essential pre-condition for protein to function correctly in biological phenomena. Numerous efforts are devoted modeling and predicting protein's stability against pH, salinity, and temperature. We consider the tasks of protein thermostability prediction on two representative datasets: (1) HotProtein (Chen et al., 2023) is recently proposed as large-scale, standardized protein benchmark with organism-lewe temperature annotations, which contains 182K protein sequences and 3K folded structure from 230 different species. Three dataset variants, HP-S, HP-S² C5, and HP-S² C2, are adopted to examine sequence- and structure-base -based methods, respec- tively. HP-S has {6, 390, 3, 4946, 30, 333, 79, 087, 31, 549} protein sequences from five categories of [Cryophilic, Psychrophilic, Mesophilic, Thermophilic, Hyperthermephillc]) HP-S" C5 has both se- quences and structures for {73, 387, 195, 196, 189) proteins from the same live classes ordered from Cryophilic to Hyperthermophilic 2 has both sequences and structures for 026, 939) pro- teins from ("hot" (2 45 C), "cold" (< 45c))) two classes. (2) Meltome Atlas (Jarzab et al., 2020) another challenging test bed for protein's 's thermostability. has {7, 902, 15, 833, 10, 518) protein sequences from three of the five aforementioned classes, from Mesophilic to Hyperthermophillc. All samples are annotated with their melting temperature.
3.4 MULTILINGUAL TRANSLATION
Multilingual translation processes multiple languages using a single language model and requires the model to have the ability to perform translation across languages. We follow Liu et al. (2020); Tang et al. (2020) and choose 10 English-centric language pairs (Fr, Cs, De, Gu, Ja, My, Ro, Ru, Vi, Zh < En) from open source parallel corpus OPUS (OPU, 2020). We follow Arivazhagan et al. (2019) and use pivot data through English to create 3 Many-to-Mane multilingual translation fine-tuning settings including 2-1o-2 (Fr, Cs), 5-to-5 (Fr, Cs, De, Gi, Ja), and 10-to-10.
4

---

Published as a conference paper at ICLR 2023
4 EVALUATION ON SMC-BENCH
Models. Despite we are aware that performing few-shot prompting on large-scale pre-training language models with billions of parameters is able solve these tasks (Wei al., 2022; Srivastava et al., 2022), we choose to focus on fine-tuning or training with pre-trained mid-scale models with millions of parameters, improve the accessibility of our Benchmark. Specifically, we choose to fine- tune the popular RoBERTa (Liu et al., 2019) for commonsense reasoning; to fine-tune mBART (Liu et al., 2020) for multilingual translation; train GTS (Xie & Sun, 2019) and Graph2Tree (Zhang et al., 2020a) with RoBERTa's ' pre-trained embedding for arithmetic reasoning: to fine-tune Transformer- based (Vaswani et al., 2017) for protein property prediction. See Appendix for full details.
Sparse Neural Networks. We select the most representative magnitude- and/or muddeent-based approaches where the prune operation is performed before, during, or after training. Formally, given a dense network Of E IRd with a dimension of di in each layer / E {1,..., L), pruning generates binary masks my E 10,
di yielding sparse neural networks with sparse weights 0: O my. The sparsity level is the fraction of the weights that are zero-valued, calculated as s = 1 - E -Following a mainstream convention in many sparse training papers (Frankle & Carbin, 2019; Gale al., 2019; Evci et al., 2020; Lee et al., 2019; Liu et al., 2021c), we sparsify most layers in the model including embedding layers and classifier heads, and we do not apply advanced techniques such as Iterative Learning Rate Rewinding (Renda et al., 2020) and Knowledge Distillation (Hinton et al., 2015) in our main evaluations. even if we observe that they help alleviate accuracy drops as in Appendix C.
E mi
Lottery Ticket Hypothesis (LTH) (Frankle & Carbin, 2019) is strong post- test-traniing pruming baseline that iteratively adopts magnitude pruning after training to produce binary masks and re-tral re-train together with weights from step f. We set & = 0 in this paper, since rewinding to early training does not notably improve the performance of Transformer models (e.g., BERT) for downstream tasks (Chen et al., 2020). The pruning rate of each IMP is set as 20%.
Magnitude After Truining is a strong baseline for prune after training, which has demonstrated strong results in various regimes. After training fine-tuning models on the specific task, we prune the model with one-shot magnitude pruning and re-train it with the full learning rate schedule from the beginning, dubbed "OMP (After)" in our experiments.
Random After Training (Mittal et al., 2019) is the most naive baseline for post-training pruning. It uniformly samples random score s Uniform(0. 1) for each weight and prunes the weights with the lowest scores. After pruning, we also re- re-train with the entire training schedule.
Gradual Magnitude Pruning (GMP) (Zhu & Gupta, 2017; Gale et al., 2019) gradually sparsifies networks during training according to a pre-defined sparsification schedule with sorting-based weight thresholding. The starting and the ending iteration of the gradual sparsification process set 10% and 80% of the entire training iterations. The frequency of sparsification steps is tuned among 500, 1000, and 4000, depending on the specific tasks. While we are aware of the advanced gradual pruning methods- movement pruning (Sanh et al., 2020), it usually exceeds GMP only at high sparsities (e.g.. 990%), which is interesting but not within the scope of this paper.
Magnitude Before Training (Frankle et al., 2021) simply removes weights with the lowest magnitude initialization. Since we inherit weights from pre- pre-trained models, the initial weights actually refer to the weights that are learned on the pre-trained tasks. We abbreviate this approach to "OMP (Before)" as we use one-shot magnitude pruning.
Random Before Training (Liu et al., 2022) is the most naive baseline for prior-training pruning. We randomly sample scores for each weight and removes the weights with the lowest scores. Different from Random After Training, the pruning operation is performed before fine-tuning.
SNIP (Lee et al., 2019) is prior-training pruning technique that globally removes weights with the lowest connection sensitivity score C w|. SNIP is a strong bascline that consistently performs well among various training approaches (Frankle et al., 2021).
Rigging the Lottery (RigL) (Evci et al., 2020) is leading sparse training method that updates the topology of sparse neural networks during training via a prune-and grow scheme. To evaluate combine RigL with the other three prior-training
its effectiveness on downstream fine-tuning, methods. The update interval of RigL is set the same as the ones used for updating sparsity in GMP, following Liu et al. (2021b).
5

---

Published as a conference paper at ICLR 2023
RoBERTa on CommonsenseeO
RoBERTa on WinoGrande
RoBERTS on RACE (Middlle)
RoBERTa on RACE (High)
80
-0
Accursyy
50
40
-0
59
20
0.99
04.05
0.003
0.300
0.80
0.69
Sparsity
Sparsity
Sparsity
Sparsity
Rundon
Figure 1: Commonsense reasoning performance of various sparse RoBERTa.
4.1 COMMONSENSE REASONING
Implementation Details. We follow the training settings of sequence modeling toolkit Fairseq (Ott et al., 2019) and fine-tune the pre-trained RoBERTa on our datasets with standard cross-cntropy loss. Specifically for each question, we also construct five inputs, one for each of the five candidate answer choices. Each input is constructed by concatenating the question and candidate answer together. We then encode each input and pass the resulting "[CLS]" representations through classifier to predict the correct answer. All models are trained with the Adam (Kingma & Ba, 2014) optimizer with learning rate of 1 x 10 using an A 100 GPU. For CSQA, we train the model for 3000 steps with a linear warmup of 150 steps and batch size of 16. The dropout rate is set as 0.1. This gives us test accuracy of 77.3% with dense RoBERTa. For RACE, we train each model for 3 epochs with batch size of 16. This gives 86.6% and 82.0% dense accuracy RACE (H) and RACE (M), matching the ones reported in Fairseq (86.5% and 81.3% 81.3%). Models on WinoGrrande are trained for 23, 750 steps with 2, 735 warmup steps and 32 batch size, reaching 76.3% accuracy.
Results and Analyses. The results of various sparse neural networks are demonstrated in Figure 1. We summarize our main observations below:
I All sparse algorithms seemingly fail to find matching SNNs, even at trivial sparsities such as 36%. While several methods maintain the dense performance at 20% sparsity, their performance starts to drop significantly after that, and will undergo a catastrophic failure as the sparsity continues increasing. It is difficult even for the top-perforning LTH to maintain the matching performance after the 2nd IMP iteration. This is in stark contrast with the behavior of SNNs on the image classification task, where LTH can gracefully preserve the matching performance even very extreme sparsities (>95% on CIFAR- 10/10001//01 100 (Yin et al., 2022) and >80% on ImageNet (Renda et al., 2020)).
② The quality of SNNs on harder tasks suffers more from sparsity. Models trained on the hardest task, CSQA, undergo a larger accuracy loss at the same sparsity than the other two datasets. For instance, all the SNNs on CSQA suffer from a catastrophic accuracy drop (up to 74%) and become no better than the random prediction at 59% sparsity. Meanwhile, when trained on WinoGrande and RACE at 59% sparsity, two sparse algorithms (LTH and GMP) can maintain relatively good performance with smaller performance loss (i.e., 3% ~ 10%).
K Post- Post-training pruning consistently outperforms prior-training pruning. LTH achieves the best performance across datasets, GMP performs well, and OMP (After) follows behind. However, prior- training pruning achieves worse performance. OMP (Before) performs closely behind OMP (After), whereas SNIP performs of SNIP, we find SNIP aggressively prunes the embedding layers to more than 99% sparsity even with mild overall sparsity of 20%. Surprisingly, the leading dynamic sparsity approach RigL does not bring significant gains to these prior-training approaches, and sometimes even hurts the performance.
better than the naive random pruning. After digging deeper into the case
4.2 ARITHMETIC REASONING
Implementation Details. We follow SVAMP (Patel et al., 2021) and choose the two top-performing tree-based models for arithmetic reasoning: GTS (Xie & Sun, 2019) and Graph2Tree (Zhang et al., 2020a). Graph2Tree in general performs slightly better than GTS. GTS adopts LSTM to encode input sequences and a tree- based Decoder to generate questions. Graph2Tree uses a graph transformer to learn the latent quantity representations from data, and tree structure decoder generate solution expression tree. We follow exactly the training settings of Patel et al. (2021). The embedding weights are inherited from THE pre- preitre RoBERTa, All models are trained with Adam for 50 epochs.

---

Published as a conference paper at ICLR 2023
ASDiV-A
SVAMP
87.5
65.0
70
02.5
Accuracy
Accuracy
$0.0
77.5
75.0
40
723
700
8.66
2.40
8.19
0.92
0.728
0.991
6.125
0.806
0.893
0.38
0.00
0,50
0.57
0,500
0.297
ear25
0.815
0.69
0.36
0.40
0.59
6,62
0.72
a.ml
0.529
Graph2Tree MAWPS
Graph2Tree ASDiviA
Graph2Tree SVAMP
80
87.5
83.0
Accuracy
82.5
Accuracy
160
80.0
65.0
77.
82.5
75.0
10
30.9
72.5
27.5
70.0
0.58
0.403
0.19
0.772
0.700
0.991
0.605
0.00
0127
30
0.28
0.000
0.09
D872
07.00
0.20
04223
0.000
0.005
0.36
0468
0.59
0.672
0.266
0.791
0.6229
0.000
Spanity
Spars
Sparaty
Denge mode:
LTH (Afte1
CVP (Aito))
Random (Aftor)
Random-RiAL (Belrr)
GMP DDuing!
SNIP (Boforc)
- ShIP Rigt afora)
Rondom (Bcforc)
Figure 2: Arithmetic reasoning performance of various sparse GTS and Graph2Tree.
On MAWPS and ASDiv-A, models are trained with the training data and then evaluated on 5-fold cross-validation based pre-assigned splits. For SVAMP, train the models on combination of MAWPS and ASDiv-A and test them on SVAMP, following Patel et al. (2021).
Results and Analyses. The performance arithmetic reasoning is reported in Figure 2. The overall performance trend is very similar to the commonsense reasoning: SNNs can only match the dense performance when the sparsity level is lower than 48.8% with the exception of Graph2Tres on the relatively simple MAWPS dataset whose failing sparsity 59%; SNNs are prone to sacrifice more performance on harder datasets (i.e., ASDiv-A and SVAMP) than the easier MAWPS dataset; prior-training methods perform no differently than random pruming. Moreover, we want to highlight that LTH surprisingly reaches lower accuracy than OMP and GMP high sparsity levels, indicating that iterative magnitude pruning may not necessarily generalize better than on more complex tasks. Moreover, Magnitude Before Training (OMG (Before)) consistently causes severe layer collapse in the non-embedding layers, leading to zero accuracies. Since including the results of OMG (Before) will significantly dilute the performance difference different sparsification methods, we choose to report it in Appendix B.
4.3
PROTEIN THERMAL STABILITY PREDICTION
4.3.1
ESQUENCE-BASER MODELS
Implementation Details. We examine two classic sequence-based approaches in protein property
prediction, i.e., TAPE (Rao et al., 2019) and ESM-IB (Rives al., 2021). For TAPE, we fine-tune from the official pre- pre-training (Rao et al., 2019) for 4 epochs with an initial learning rate of 1 x 10-4 and linear decay scheduler together with 100 warm-up steps. As for ESM-1B (Rives al., 2021). starting from the official pre-trained checkpoint, we fine-tune the backbone with learning rate of 1 x 10-6 and the linear classification head on top of the backbone with a learning rate of 10-2 The learning rate schedulers used for both backbone and linear head are OneCycle (Smith & Topin, 2019) decay schedulers. The training batch size is 2 for Meltome Atlas and 3 for HotProtein (HP-S). Classification accuracy test sets is collected measure the model performance.
Results and Analyses. this section, we examine diverse sparse neural networks of sequence- based models (i.e., transformers) protein thermostability prediction benchmarks. ESM-1B (Rives et al., 2021), 1), a SOTA approach in protein property modeling, is evaluated on both HotProtein (HP-S) and Meltome Atlas datasets. TAPE (Rao et al., 2019) is classic transformer-basee model adopted on HotProtein (HP-S). Extensive results of both static and dynamic sparsifications are collected in Figure 3. We observe that: ① For ESM-1B, 1B, all extracted sparse neural networks incur significant performance degradation whenever the sparsity level larger than 20%. Note that here we only sparsify the fully connected layers in multi-head self-attentions & feed-forward networks of each
7

---

Published as a conference paper at ICLR 2023
ESM-1F1 on CPS5
ESM-1B on HP-S
Tape on HP-S
ESM-1B on Allas
c0
20-
Accuracy [%%]
Accursa
-00
12
15-
10
0.00
0.29
ax20
cale
0.791
cabe
0.811
Dx35
9.86
0.2
2,34
Cadle
6.
6:291
call
Sparsiy
Sparsity
Sparsity
Sparsity
Figure 3: Protein prediction performance of various sparse models.
transformer layer, and leave all other modules unpruned. Even under this loose condition, ESM- IB still fails after pruning both HP-S and Meltome Atlas, which indicates the low parameter redundancy in ESM-1B for modeling protein thermal stability. @ In general, static and dynamic pruning algorithms achieve similar performance with ESM-IB. SNIP (Before) and SNIP + RIGL (Before) deliver relatively better accuracies, especially for the high sparsity (2 48.80%) subnetworks on HP-S. As for the worse backbone TAPE compared with ESM-IB, IB, magnitude- like LTH (After), OMP (After), and GMP (During) show satisfactory results before 59% sparsity.
prunings
Furthermore, we conduct a more fine-grained pruning schedule to investigate the tolerance of ESM- 1B against sparsification. In de- tail, we prune 5% weights with OMP (after) on different modules in ESM-1B and present the obtained accuracies in Table 1. Q K.
Table 1: OMP (after) pruning 5% weights of ESM-IB IB on different modules with CP.
Pruned Modules
Accuncy (*)
V, O, and FFN represent the fully connected layer the query, key. value, & output of the self-attention module and feed-forward net- works, respectively. The results show that whatever modules we select, 5% sparsity damages the ESM-IB performance of protein thermostabilityy prediction on C22
Nome (Dense)
91.68
Q. Q.K. V,
92 19
K.
92.55
O.K
Q.
K
92.56
4.3.2 STRUCTURE-BBAT BASED MODELS
Implementation Details. We further consider a representative structure-based algorithm for ther- mostability prediction, i.e., ESM-IFI (Hsu et al., 2022). Specifically, for ESM-IFI, we train the backbone and its linear head with learning rates of x 10-4 and x 10-2, respectively. A batch size of 4 is used for both models on HotProtein (HP-s Classification testing accuracy is reported to reflect the model performance.
Results and Analyses. In this section, we study structure-based models and their sparse variants on HotProtein (HP-S?C5). ESM-IFI (Hsu et al., 2022), a recent SOTA approach, is chosen for benchmarking. It takes the 3D structure of proteins as input and predicts its thermal stable temperature. As shown in Figure 3, ESM-IF1 produces inferior sparse neural networks with all pruning mechanisms of both static and dynamic, where OMP (after) and GMP (During) present relatively better accuracies.
4.4 MULTILINGUAL TRANSLATION
Implementation Details.
We choose the official multilingual model mBART' (Liu et al., 2020), which was originally pre-trained on 25 languages using masked language modeling (MLM), following the fine-tuning setting of Tang et al. (2020). We first choose 10 languages from the language pools used for MLM pre-training; create three sub-groups containing 2, 5. 10 languages; and fine-tune referring to 2-to-2, 5-to-5, and 10-to-10 multilingual translation fine-
mBART on each tuning, respectively. During inference, we report the averaged BLEU (Tang et al., 2020; Liu et al., 2020) scores of bi-directional translation across 10 languages to measure the translation performance. Hence, the task difficulty monotonically decreases from 2-to-2 to 5-to-5, and 10-to- 100 fine-tuning as more languages are involved during training. The default fine-tuning configurations in Tang et al. (2020) are adopted for 40K iterations with an Adam optimizer and a learning rate of 1 x 10 1
Results and Analyses. Intuitively, fewer languages involved during fine-tuning leads to a more difficult translation for all languages. As demonstrated in Figure 4, several consistent conclusions can be drawn: ! Besides OMP (After) and LTH (After), all other produced sparse subnetworks perform
httpp:://gi tttp:///ttttutttc com wttp://///.ccccccceeeeeeeeeeeeee
rseq
8

---

Published as a conference paper at ICLR 2023
worse than the dense baseline when the sparsity is larger than 20%. The BLEU scores of OMP (After) and LTH (After) also decline and fail to match at 2 20%, 2 48.8%. 2 59% sparsity levels for 2-to-2, 5-to-5, and (0-to-10 fine-tuning, respectively. @ Magnitude--aaseer sparsifications like OMP, LTH, and GMP are comparably robust across all three translation settings, while other pruning methods have negligible advantages compared to random pruning. @ While the overall tendency of SNNs is quite consistent across different tasks, the prunability of mBART increases as more languages are involved during fine-tuning, It seems that multilingual translation has already been challenging task for pruning, and involving more languages in inference causes extra obstacles. This is why in the hardest scenario of fine-ture tine-tuning on 2-to-2 and evaluating with 10 languages, all sparse subnetworks suffer from substantial performance degradation.
Trained on 2-to-2
Trained on 5-to-5
Trained on 10-to-10
BLEU
OLEU
0.59
2,791
0.000
Denas mozel
5387
(3etoro)
CNP (Abbe))
Tandom (Aber)
CM7 (uring))
RIP-RIGL (Bstor))
1TH (Ater)
548+
MGL Seloool
Bundom
(Bolore)
Tandoms R CR (Boford)
CM7 (Satrrc)
Figure 4: Multilingual performance of various sparse mBART. All models are tested on 10-to-10 multilingual translation and the averaged BLEU are reported.
4.5 WHY SNNS FAIL ON SMC-BENCH
We conduct a thorough investigation into the reasons why most SNNs struggle to perform on SMC- Bench. Our analysis identifies two possible causes for the failure: (1) the "lazy regime" in LLMs, and (2) the specific model components that are pruned. Based on these findings, we discover a set of stronger pruning recipes that alleviates (though still more or less suffering from) the failure on SMC-Bench, by breaking down the performance of the state-of-the-a BERT-pruning framework oBERT (Kurtic et al., 2022) on SMC-Bench (note that most models evaluated in this paper are also BERT-based). Due to the limited space, we present our full investigation in Appendix C, and briefly present our sanity check of layer collapse below.
Does layer collapse occur unexpectedly on SMC-Bench? Layer collapse is the most common cause that blocks the information flow (signal propagation) of sparse neural networks, leading to a catastrophic performance drop (Tanaka et al., 2020). We plot the layerwise sparsity ratios of various sparse models in Appendix C.I. We do not observe severe layer collapse across methods except for SNIP which specifically removes nearly entire embedding layers. However, we do observe an unexpected phenomenon: layerwise sparsities of different magnitude- badee pruning approaches (i.e., IMP, OMP, and GMP) extremely similar, all overlapped on one line, despite the significant performance gap among them (up to 42.3%), small differences only start to appear until reaching very deep layers (e.g., classification heads) (see Appendix C. for more details). This abnormal behavior is highly correlated with the "lazy regime" (Neyshabur et al., 2020; Malladi et al., 2022) where the model stays in the same basin during fine-tuning with fairly small weight changes, and hence all magnitude- based pruning approaches, before, during, and after fine-tuning, tend to collapse to the same solution.
5 CONCLUSION
Given the enormous breakthroughs and the fruitful results that sparse neural networks have achieved in numerous fields, it is necessary to rethink the sufficiency of current evaluation protocols and introduce more difficult and diverse benchmarks to explore the limitation of sparse neural networks. In this paper, we assemble a large-scale, challenging, and more diverse benchmark, BMC-Bench. Through our thorough evaluation across various leading sparsifications, we confirm that SMC-Bench notably challenges the capability of most magnitude- or/and gradient-based sparse algorithms. We further dig deeper into the behavior of SNNs, and observe several surprising phenomena that are absent in the current evaluation. Our analysis points out that such failure is intimately related to the "lazy regime", which leads us to a suite of strong pruning recipes that alleviates (yet still more or less suffering from) the failure on AMC-Bench. Our subsequent effort will focus on exploring stronger sparse training algorithms that can scale and generalize on SMC-Bench, and meanwhile will consider the training costs of different sparse algorithms for more holistic picture of their efficiency benefits.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
We thank Dan Alistarh and Eldar Kurtic for the extremely helpful discussions about the implementa- tion details of oBERT as well as our benchmark's claims; and Zhangheng Li for helping run extra experiments with oBERT. S. Liu and Z. Wang are in part supported by the NSF Al Institute for Foun- dations of Machine Learning (IFML). Part of this work used the Dutch national e-infrastructure with the support of the SURF Cooperative using grant no. NWO2021.06 10000, EINF-2694 and EINF-2943/LI.
REFERENCES
Open source parallel corpus of opus. https: /opus hallpleeeoppus-t111 php.ppp.pppppppppp.... 2020.
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. Massively multilingual neural machine translation in the wild: Findings and challenges. arXiv preprint arXiv: 00079,0519,..0 2019.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? Proceedings of machine learning and systems, 2:129-146, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Ben- jamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL WtLLhhtts:://ooooto http::::s////cceedepe/ee http://roooed.nn.....r.......... neurips. /ttpppee2200//l1eee1555//22
967 (wa888fffacl110 42f1442f4442 -aapeeefeeaaaeaaaeea pddf.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. Advances in neural information processing systems, 33: 18:115334-5844 2020.
Tianlong Chen, Zhenyu Zhang, Santosh Balachandra, Haoyu Ma. Zehao Wang, Zhangyang Wang, et al. Sparsity winning twice: Better robust generalization from more efficient training. In International Conference Learning Representations, 2022.
Tianlong Chen, Chengyue Gong, Daniel Jesus Diaz, Xuxi Chen, Jordan Tyler Wells, qiang liu, Zhangyang Wang, Andrew Ellington, Alex Dimakis, and Adam Klivans. Hotprotein: A novel framework for protein thermostability prediction and editing. In The Eleventh International Conference Learning Representations, 2023. URL http:::ss///peeeeeee... net forum? Ed-YYDJEEWBNbb
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in neural information processing systems, 32, 2019.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023111 2022.
Pau de Jorge, Amartya Sanyal, Harkirat Behl, Philip Torr, Grégory Rogez, and Puneet K. Dokania. Progressive skeletonization: Trimming more fat from a network at initialization. In International Conference on Learning Representations, 2021. URL https: htRp://tt/////eeeeeeee /ttpss:////p/eeeeeeeeeee meee/forum id- Ld=9GsFOUyUPi.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv: 1et..044704400 2019.
10

---

Published as a conference paper at ICLR 2023
James Diffenderfer, Brian Bartoldson, Shreya Chaganti, Jize Zhang, and Bhavya Kailkhura. A winning hand: Compressing deep networks can improve out- of-sittribttiin robustness. Advances 2021.
in Neural Information Processing Systems,
Xin Dong. Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. Advances in Neural Information Processing Systems, 30, 2017.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943-2952. PMLR, 2020.
William Fimnoff, Ferdinand Hergert, and Hans Georg Zimmermann. Improving model selection by nonconvergent methods. Neural Networks, 6(6):771-783, 1993.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https:
Jonathan Frankle, David J. Schwab, and Ari S. Morcos. The early phase of neural network training. In International Conference on Learning Representations, 2020. URL https://openeeviiiiee net / srrum?id=Hkl1.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Pruning neural
networks at initialization: Why missing the mark? In International Conference Learning Representations, 2021. URL https: openrevieeveeeeee eee net...ewweeeeeeeeeeeeee..e..ee forum? fog-
Elias Frantar, Eldar Kurtic, and Dan Alistarh. M-fac: Efficient matrix-free approximations of second-order information. Advances in Neural Information Processing Systems, 34:14873-14886, 2021.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks, arXiv preprint arXiv:1902.09574 2019.
Yiwen Guo, Chao Zhang, Changshui Zhang, and Yurong Chen. Sparse dans with improved adversarial robustness. Advances in neural information processing systems, 31, 2018.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruming, trained quantization and huffman coding. International Conference Learning Representations, 2015.
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.0253: 2015.
Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. bioRxiv, 2022. doi: 10.1101/2022.04 10. 10.497779. URL httpt: //www./wwww//.. /022/04/10
httpps///w...bbr..iiii/ biorxiv org/ content/eart 10..4777777
Anna Jarzab, Nils Kurzawa, Thomas Hopf, Matthias Moerch, Jana Zecha, Niels Leijten, Yangyang Bian, Eva Musiol, Melanie Maschberger, Gabriele Stoehr, et al. Meltome atlas tthermal proteome stability across the tree of life. Nature methods, 17(5):95-5503, 2020.
Hao Jiang, Ke Zhan, Jianwei Qu, Yongkang Wu, Zhaoye Fei, Xinyu Zhang, Lei Chen, Zhicheng Dou, Xipeng Qiu, Zikai Guo, et al. Towards more effective and economic sparsely-activated model. arXiv preprint arXiv:21 10.07431, 2021.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 44426988, 2014.
11

---

Published as a conference paper at ICLR 2023
Rik
Subhro Roy, Aida Amini, Nate Kushman, and Hannanch Hajishirzi. Mawps: A math word problem repository. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1152-1157, 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models. arXiv preprint Sxiv:22033.02559, 2022.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity, In International Conference on Machine Learning, pp. 1555. PMLR, 2020.
François Lagunas, Ella Charlaix, Victor Sanh, and Alexander M Rush. Block pruning for faster transformers. arXiv preprint arXiv:2109.0m. 04838,, 2021.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang. and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint
2017.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mmist// 1998.
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances neural information processing systems, pp. 598-605, 1990.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. SNIP: SINGLE-SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIVITY. In International Conference Learning Representations, 2019. URL https: httpt:ttt/s///eeeee http///ppeeeeveeeeeeeeeeeee net/ forum? ? d= VZ
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning. 2012.
Tao Lin, Sebastian U. Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. In International Conference on Learning Representations, 2020. URL https:/ /ttp:////peeeviieeeee
net www.:eettooouttmeeeeee forum?i dd-Jem 15FFR.
Shiwei Liu and Zhangyang Wang. Ten lessons we have learned in the new" sparseland": short handbook for sparse neural network researchers. arXiv preprini urXiv:2302.02555 2023.
Shiwei Liu, Tianlong Chen, Zahra Atashgahi, Xiaohan Chen, Ghada Sokar, Elena Mocanu, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Deep ensembling with no overhead for either training testing: The all-round blessings of dynamic sparsity, arXiv preprint arXiv:2106. 14568, 202
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with neuroregeneraaioo. Advances in Neural Information Processing Systems (NeurlPs). 2021b.
Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do we actually need dense w in-time over- wver-parameterization in sparse training. In Proceedings of the 39th International Conference on Machine Learning, pp. 6989-7000. PMLR, 2021c
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, and Mykola Pechenizkiy. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. International Conference on Learning Representations, 2022.
URL https::///// / apenreview wwt.veewweee...ee.ee.ee.......ee forum? id=VB2J_330 1t1e22_3_t2222
Yinhan Liu. Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. / 2019.
12

---

Published as a conference paper at ICLR 2023
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre- pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742, 2020.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint 110.05280.005270, 2018.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1.0 regularization. International Conference on Learning Representations, 2018.
Jian-Hao Luo and Jianxin Wu. Autopruner: An end-to-end trainable filter pruning method for efficient deep model inference. Pattern Recognition, 107:10746. 2020.
Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning, arXiv preprint arXiv:2210.05644 2022.
Shen- Yun Miao, Chao-Chun Liang, and Keh- Yih Su. A diverse corpus for evaluating and developing english math word problem solvers. arXiv preprint arXiv:2106.15772, 2021.
Deepak Mittal, Shweta Bhardwaj, Mitesh M Khapra, and Balaraman Ravindran. Studying the plasticity in deep convolutional neural networks using random pruning. Machine Vision and Applications, 30(2):203-216, 2019.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9(1)23383, 2018.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In International Conference on Machine Learning, pp. 2498 2507. PMLR, 2017.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. International Conference on Learning Represen- tations, 2016.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, pp. 4646 4655. PMLR, 2019.
Michael C Mozer and Paul Smolensky. Using relevance to reduce network size automatically. Connection Science, 1(1):3-16, 1989.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? Advances in neural information processing systems, 03:512-522, 2020.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Ozan Özdenizei and Robert Legenstein. Training ad adversarially robust sparse networks via bayesian connectivity sampling. In International Conference on Machine Learning. pp. 2314-8324, PMLR, 2021.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able solve simple math word problems? arXiv preprint drXiv:2103.07191, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song. Evaluating protein transfer lcarning with tape. Advances in neural information processing systems, 32, 2019.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations, 2020. URL https: /ttp:///eeeeeeeeeeeee / forum? d= gs OONKRR.
13

---

Published as a conference paper at ICLR 2023
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, Lawrence Zitmick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15), 2021.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106, 2021.
Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning. In H. Larochelle, M. Ranzato, Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 20378-20389, Curran As- sociates, Inc., 2020. URL http: httl:ttts//prrccddnnggg wtt.ssss.rr.s...c.pppree.eeee wttpcc/pppp/////.../ file/ pdf.
Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsification. Advances in Neural Information Processing Systems, 1180-11390000 2020.
Jonathan Schwarz, Siddhant Jayakumar, Razvan Pascanu, Peter E Latham, and Yee Teh. Powerpropa- gation: A sparsity inducing weight reparameterisation. Advances in neural information processing systems, 01122889-22990 2021.
Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order upproximation for neural network compression. Advances in Neural Information Processing Systems, 33:18098-18109 2020.
Leslie N Smith and Nicholay Topin. Super-conve Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi- domain operations applications, volume 1006, pp. 1100612. International Society for Optics and Photonics, 2019.
Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In Thirty-first AAAI conference on artificial intelligence, 2017.
Aarohi Srivastava, Abhinay Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Farriga-Alonss et Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.0460 04615, 2022.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: question answering challenge targeting commonsense knowledge. arXiv preprint arXiv: 2018.
Hidenori Tanaka, Daniel Kunin, Daniel LK Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances Neural Information Process- ing Systems. arXiv:2006. 05467, 2020.
Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. Multilingual translation with extensible multilingual pretraining and finetuning. arXiv preprint arXiv:2008.0m 00401, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 17X4::880.004400 2018.
Chaoqi Wang, Roger Grosse, Sanja Fidler, and Guodong Zhang. Eigendamage: Structured pruning in the kronecker- factorre cigenbasis. In International Conference on Machine Learning, pp. 6566-6577 PMLR, 2019.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020. URL https: http///pppe//eeeeeeee//e/eeeeeee net www.:/wwwwwwetmteeee/ee.. foorumiid=Skgsaaeee
14

---

Published as a conference paper at ICLR 2023
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Korblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pp. 33965-23998, PMLR, 2022.
Zhipeng Xie and Shichao Sun. A goal-driven tree-structured neural model for math word problems. In IJCAI, pp. 5299-5305, 2019.
Dongkuan Xu, Ian EH Yen, Jinxi Zhao, and Zhibin Xiao. Rethinking network pruning-unde under the pre-train and fine-tune paradigm. arXiv preprini urXiv:2104.08665 2021.
Lu Yin, Shiwei Liu, Fang Meng, Tianjin Huang, Vlado Menkovski, and Mykola Pechenizkiy. Lottery pools: Winning more by interpolating tickets without increasing training or inference cost. arXiv preprint arXiv:2208. 10842, 2022.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini. and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint 27X11:2205.01917, 2022.
Ofir Zafrir, Ariel Larey, Guy Boudoukh, Haihao Shen, and Moshe Wasserblat. Prune once for all: Sparse pre-trained language models. arXiv preprint arXiv:2111.05754 2021.
Wenyuan Zeng and Raquel Urtasun. Miprune: Multi-layer pruning for automated neural network compression. 2018.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12104-12113, 2022.
Dinghuai Zhang, Kartik Ahuja, Yilun Xu, Yisen Wang, and Aaron Courville. Can subnetwork structure be the key out-of-distribtioon generalization? International Conference on Machine PMLR, 2021.
Learning, pp.
Jipeng Zhang, Lei Wang, Roy KeiWei Lee, Yi Bin, Yan Wang, Jie Shao, and Ee-Peng Lim. Graph- to-tree learning for solving math word problems. Association for Computational Linguistics, 2020a.
Qingru Zhang, Simiao Zuo, Chen Liang, Alexander Bukharin, Pengcheng He, Weizhu Chen, and Tuo Zhao. Platon: Pruning large transformer models with upper confidence bound of weight importance. In International Conference on Machine Learning, pp. 26809-26823. PMLR, 2022.
Yian Zhang, Alex Warstadt, Haau-Sing Li, and Samuel R Bowman. When do you need billions of words of pretraining data? arXiv preprint arXiv:201 / .0496,, 2020b.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint 1710017710.0188,, 2017.
15

---

Published as a conference paper at ICLR 2023
A SUMMARY OF TASKS, MODELS, DATASETS, AND TRAINING
We summarize the combinations of models and configurations that we used to evaluate SNNs on SMC-Bench.
Table 2: Summary of models and datasets that we used to evaluate on SMC-Benn
Task
Datasets
Models
Source
CSQA
RoBERT, Large
Facebook Al Reseuch Sequence Sequence Toolkit (Onl al. 2019)
Commonseuss Reasoning
WinoKrande
RoBERT Lange
Facehook Roseanch Sequenes-to- Sequene Toolki
2019)
RoBERT: Large
Facebook Reseorch SespeencsttoSSqqueeee
al., 2019)
Gitlluh Repository (Paiel et 2021)
MAVPS
Graph2TTvee
GilHub Repository (Paiel et 2021)
2021)
Arithmetic Recasoning
ASDivAA
GTS Graph2Tree
GitHuuh Repasitory (Pasel et al. 2021)
GilHuuh Repository (Pasel
Repusitury
Graph2Tree
GiiHHhh Repository (Parel 2021)
HotProtein (HI--)
ESM 1B
TAPE
Gillluh Repository (Rives al., 2021)
GitHub Repository (Kao et al., 2019)
Protein Thermoslability Prediction
HotProtcin (HP SPC5)
FSM-IF1
GiiHuu Repository (Rives ol. 2021)
HolProtein (MP-S-C2)
ESM-IB
GitHlub Repository (Rives al., 2021)
Repository
2021)
2-10-2
mBART
Fucebook Resurch Sequence- to Sequence Toolkit all. 2019)
Multilingual Transsation
5-0-5
mBART
Facebook Al Resemnch Sequenes-to- Se-euue-e---e-ee Tholki (On al., 2019)
10-t-100
mBARTT
Facebook Al Research Sequence-tos So-uqueece Toolkit (On al., 2019)
We strictly follow the training configurations reported in the original source to replicate the results of each task. The hyperparameters and configurations used for each model this paper are shared below.
A.1
COMMONSENSE REASONING
Table 3: Hyperparameters and training configurations used for models on Commonsense Reasoning.
Models
RoBERTa
RoBERTa
RoBERTa
Dataset
CSQA
WinoGrande
RACE
Pre-trained Models
RoBERTa
RoBERTa
RoBERTa
Hidden Size
[1024]
[1024]]
[1024]
FEN Inner Hidden Size
[4096]
[4096]
[4096]
Number of Layers
[24]
[24]
[24]
Learning Rate
[lc-5]
[1c-5]
[1c-5]
Weight Decay
[0.0]]
[0.01]
[0.01]
Batch Size
[16]
[32]
[16]
Dropout
[0.1]
Attention Dropout
[0.1]
[0.1]
[0.1]
Clip Norm
[0.0]
[0.0]
[0.0]
Adam
[1c-06]
[1e-06]
[1c-06]
Adam &
[0.9]
[0.9]
[0.9]
Adam 3
[0.98]
[0.98]
[0.98]
# Parameters
355M
355M
355M
Training Time
3000 steps
23750 steps
epochs
Wramup Time
150 steps
2375 steps
500
16

---

Published as a conference paper at ICLR 2023
A.2
ARITHMETIC REASONING
Table 4: Hyperparameters and training configurations used for models on Arithmetic Reasoning.
Models
GTS
Graph2Tree
Dataset
MAVPS, ASDiv-A, SVAMP
MAVPS,ADDiv SVAMP
Pre-trained Embedding
RoBERTa
RoBERTa
Embedding Size
[768]
[768]
Hidden Size
(512)
[384]
Number of Layers
[2]
[2]
Learning Rate
[le-3]
[8c-4]
Weight Decay
[le-5]
[le-5]
Embedding LR
[8c-6]
[lc-5]
Batch Size
[4 (MAVPS, ASDiv-A), 8 (SVAMP))
[4 (MAVPS, ASDiv-A),
(SVAMP)
Dropout
[0.5]
[0.5]
Adam
[1c-08]
[1c-08]
Adam 31
[0.9]
[0.9]
Adam
[0.999]
[0.999]
# Parameters
140M
143M
Training Time
50 epochs
50 epochs
A.3 PROTEIN THERMOSTABILITY PREDICTION
Table 5: Hyperparameters and training configurations used for models on Protein Thermostability Prediction.
Models
TAPE
ESM-1B
ESM-IFI
Dataset
HP-S
HP-S Meltome
HP-S*C5
Atlas, HP-S
Hidden Size
[768]
[1280]
[512]
Number of Layers
[12]
[33]
[20]
Learning Rate
[le-4]
[2e-2 (head), 1c-6
[2e-2 (head), le-4
(backbone)]
(backbone))
Weight Decay
[le-2]
[1e-2]
[5e-2]
Batch Size
[16]
[3,2,3]
[4]
Attention Dropout
[0.1]
[0:0]
[0.1]
Dropout
[0.1]
[0.0]
[0.1]
Adam
[le-08]
[1e408]
[1e-08]
Adam 3
[0.9]
[0.9]
[0.9]
Adam 3
[0.999]
[0.999]
[0.999]
# Parumetens
92M
650M
124M
Training Time
epochs
epochs
epoccs
A.4 MULTHIINGUAL TRANSLATION
Table 6: Hyperparameters and training configurations used for models on Multilingual Translation.
Models
mBART
mBART
mBART
Dataset
2-1o-2
5-10-5
10-to-10
Pre-trained Models
mBART
mBART
mBART
Hidden Size
[1024]
[1024]
[1024]
Number of Layers
[24]
[24]
[24]
Learning Rate
[3e-5]
[3e-5]
[3e-5]
Weight Decay
[0.0]
[0.0]
[0.0]
Batch Size
[16]
[32]
[16]
Dropout
[0.3]
[0.3]
[0.3]
Attention Dropout
[0.1]
[0.1]
[0.1]
Clip Norm
[0.0]
[0.0]
[0.0]
Adam
[le-06]
[1e-06]
[6e
Adam 3
[0.9]
[0.9]
[0.9]
Adam 81
[0.98]
[0.98]
[0.98]
# Purameters
680M
680M
680M
Training Time
40,000 steps
40,000 steps
40,000 steps
Wramup Time
2,500 steps
2,500 steps
2,500 steps
17

---

Published as a conference paper at ICLR 2023
B RESULTS OF ARITHMETIC REASONING WITH MAGNITUDE BEFORE TRAINING
In this appendix, we report the performance of SNNs arithmetic reasoning including Magnitude before Training (OMP (Before)). We can clearly observe that the accuracy of magnitude pruning before training dramatically falls from 80% to nearly 0% when sparsity larger than 36%. After checking the corresponding layerwise sparsity, we find that OMP (Before) completely removes all the weights from eobeedidd and non-encoder layers, leading to severe layer collapse.
GTS on MAWPS
GTS on ASDiv-A
GTS on SVAMP
40
Accuracy (96)
60-
Accuracy [%6]
60
Accuracy [96]
30
40
40
0.38
0.488
0.59
0,072
0.738
0.791
0.825
0.806
0.093
0.36
0.489
0.59
0,772
0.728
0.791
0.3225
6.006
0.002
0.38
0-429
0.50
0.622
0.736
0.701
0.325
0.866
0.893
Draph2Tren MAWPS
Graph2Tree ASDiv-A
Graph2Tree SVAMP
80
80
40
Accuracy (%)
Accuracy [%]]
Accuracy [%]
60-
20
20
10
0.38
0.488
0.59
0.872
0.738
0.791
0.822
0,0ed
0.89
0.36
0.489
0.59
0,872
0,738
0.791
0.8329
0.66
0.892
0.
0.36
0.402
0.39
0.672
0.238
0.791
6,6325
0.668
0,893
Sparsg.
Sparsi
Sporsity
Denise
LTH
Random (After
GMP
(Befons)
SNIP (Before)
SNI--RIEE [Befcre]
Random [Before]
Random- RIF (Before)
DMP (Befone)
Figure 5: Arithmetic reasoning performance of various sparse GTS and Graph2Tree on MAWPS, ASDiv-A, and SVAMP.
18

---

Published as a conference paper at ICLR 2023
C AN INVESTIGATION OF WHY SNNS FAIL ON SMC-BENCH
In this section, we conduct a full investigation, attempting to open the box for the potential causes of SNN failures on SMC-Bench. Our analysis reveals two possible causes: (I) the "lazy regime" in fine- tuning LLMs, and (2) the model components to prune. Due to the "lazy regime" phenomenon (Chizat al., 2019; Malladi et al., 2022), commonly used pruning techniques that rely on magnitude and gradient can be very uninformative. Therefore, we turn to the latest strong second-order pruning framework - oBERT (Kurtic al., 2022), which utilizes inverse- Hesssan approximations to guide pruning decisions. We choose RoBERTa. large on CSQA to conduct this investigation. The roadmap of our full investigation presented below.
Layerwise sparsity ratios of various magnitude-based pruning methods are strikingly similar, suggesting that "lazy regime" may occur during fine-tuning, In this regime, the most common pruning criteria such as magnitude and gradient can be rather unreliable.
Second-order pruning approaches like oBERT provide more faithful signals than magni- tudes and gradients for LLM pruning, and hence achieve significantly higher accuracy at high sparsities.
C.I LAYERWISE SPARSITY RATIOS ON SMC-BENCH
To check if severe layer collapse occurs on SMC-Bench, we plot the per-layer sparsity ratios discovered by various sparsification approaches at three sparsity levels: 36%, 64%, and 83%. Layers are ordered from input to output on the X-axis. We respectively report the layerwise sparsity of commonsense reasoning with RoBERTa on CSQA and RACE in Figure 6 and 7, and arithmetic reasoning with GTS on SVAMP in Figure 8. We summarize main findings here.
Layerwise sparsities of magnitude- based pruning approaches are extremely similar. IMP, OMP, and GMP that rely weight magnitude for pruning share an extremely similar set of layer- wise sparsities. Especially, sparsity values of magnitude pruning on commonsense reasoning are completely identical, all overlapped blue lines, except for the tiny difference in classification heads. This phenomenon indicates that weights of RoBERTa excluding classifiers remain rather stable during commonsense reasoning fine-tuning so that all the magnitude pruning variants (both before and after) discover the same sparsity pattern. The sparsity difference of arithmetic reasoning more distinguishable than commonsense reasoning. Still, sparsities in the encoder (pre-trained RoBERTA) of IMP, GMP, and OMP (After) largely overlap. Until reaching the later layers, the sparsity ratios of different approaches start be distinct. @ SNIP tends prune all the weights in embedding layers aggressively. Even at the mild 36% sparsity, SNIP prunes weights of embedding layers to 99.7% sparsity, which may explain why SNIP struggles on SMC-Bench. 2 OMP (Before) suffers from layer collapse arithmetic reasoning. We empirically find that OMP (Before) leads to completely empty deep layers when sparsity is larger than 36,, indicating the limitation of only considering pruming with the magnitude before fine-tuning or re-training.
The ncar-identical layerwise sparsity ratios across various magnitude-bas basse methods remind us the "lazy training" regime (Chizat et al., 2019; Malladi et al., 2022) which was revealed to occur during the fine-tuning of LLMs. Under this regime, weight changes during fine-tuning are negligible, hence non-informative and more "noisy". Consequently, various magnitude-based pruning approaches, regardless of their timing, all tend to converge to the same solution.
19

---

Published as a conference paper at ICLR 2023
NARERT COOA Overcl Sparty-O0t
1.0
08-
66-
04
1.0
Sparit
CFF [Dustgg)
(Duten))
Random (Delanu)
Figure 6: Layerwise sparsity of RoBERTa CSQA sparsity levels [36%, 67%, 83%].
BEK RACE, Oenrsl Sparty--oee -115
06
1.0
Connel Spontyet3
CMP (Dunrig)
(Butend)
Figure 7: Layerwise sparsity of RoBERTa RACE sparsity levels [36%, 67%, 83%].
20

---

Published as a conference paper at ICLR 2023
1.0
04
Uh Jahen
CMP (heives
DMP (hwelog
SAP (eelang
Randivm (helwo)
CMP johwi
Figure Layerwise sparsity of GTS on SVAMP at sparsity levels E [36%, 67%, 83%].
C.2 EVALUATION OF OBERT ON SMC-BENCH
Based on our conjecture that magtitudes/rradeens become unreliable for pruning LLMs, we hy- pothesize that the second-order pruning approaches with approximated Hessian matrix would be more accurate options. To verify framework, oBERT (Kurtic et al., 2022). Specifically, we follow Kurtic et al. (2022) and replace the magnitude pruning criterion of the LTH framework with the second-order oBERT criterion; adopt Learning Rate Rewinding (LRR) (Renda et al., 2020) and Knowledge Distillation (KD) (Hinton et al., 2015) during each pruning iteration; and keep the embeddings and classification heads dense.
conjecture, turn to the latest stronger second-order pruning
Figure 9 support our hypothesis and demonstrates that oBERT notably outperforms the zero-order and first-order sparsification approaches, and substantially improves the accuracy to competitive level. More importantly, oBERT produces a completely different layerwise sparsity pattern from magnitude- based pruning approaches, which is consistent with the patterns that are commonly observed in sparse computer vision models: deeper layers tend to have higher sparsities than lower layers (Evci et al., 2020; Kusupati et al., 2020; Tanaka et al., 2020; Liu al., 2021b).
44g
629
600
6923
Drpnn model
649
Randem (0h1
681R 001 pan enb ard risotferr
Figure 9: A roadmap of accuracy recovering via a suite of stronger pruning recipes with RoBERTa- Large on CSQA.
So far, investigation has discovered that the pruning recipe used in (Kurtic et al., 2022) can remarkably improve the LLM pruning performance on AMC-Bench Nevertheless, such a well-tuned pruning recipe both time- and resource-intensive (9x more fine-tuning time, besides Hessian matrix
21

---

Published as a conference paper at ICLR 2023
148 en CSGA, Overal S2arty-351
www.b.ppprt 3-4
Figure 10: Layerwise sparsity comparison among LTH, LRR, oBERT, and KD with RoBERTa-Large on CSQA.
approximation); and even so, strongest SNNs still fall short of their dense counterpart by around 10% accuracy at sparsities between 60% 80%, in contrast to "normal" SNNs that easily match their dense models on CIFAR, ImageNet, or GLUE. Therefore, the main claim our paper still holds, that is, SMC-Bench indeed provides new benchmark that is way more challenging for SOTA sparse algorithms, than existing testbeds.
22

---

Published as a conference paper at ICLR 2023
D SUMMARY OF EVALUATION TASKS AND DATASETS IN 100 PAPERS
Table 7: Summary of Evaluation Tasks and Datasets Used in 100 Recent SNN Papers.
TASK
TOTAL SPAPER
DATASETS
SPAPER
IMAGENOT
62
CIFAR-I0
59
CIFAR-100
37
MNIST
26
IMAGE CLASSUNCATION
FASHION MNIST
10
BIRDS-20
PLOWERS-102 102
EMNIST
GLUE SQUAD
9
WIKTTOxI-103
WMT
5
TASK
IMDB AAN
LO
OPENWER
BILLION WORD BUNCHMARK
LFW YOUTURE FACTES
FACH REDOENITION
2
CASIA-WEHFACH
COCO DATASET PASCAL-VOL-2107
2
OBJECT DETECTION
GOOGLE-12
SPInH REcOGNITION
TIMIT
SETŠ
SE114
HIGH RESOLUTION RECONSIBUCTION
2
B100
2
URBAN100
MANGA109
CIFAR-10
IMAGE GENERATION
IMAGNNN
STL-10
JIMAN ACTIVITY RECOGNITION
HAR-2
LEUKEMIA CLL-SUB-
MICROARRAY CLASSIRCATION
SMK-CAN-18
GLI-85
HAND GESTURE RECONSTRUCTION
NYGESTURE
REGRESSION TASK
NYU DEPTH
OBJECT PARI SEOMENIATION
SHAPUNET
CARIPOLE
RL TASK
ACRUBOT MOUSTAINAR
DVD GOPRO REAL BLUERY VIDEOS
DEBLURRING
VS
VOCABULARY SPEECH RECOGNITION
SWB
23


Reviews:
[{"review_id": "WLJDqRFb_9", "reviewer": "Reviewer_11zQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a new benchmark and results for evaluating sparsity methods on diverse tasks called SMC-Bench.\nThroughout difficult tasks the paper finds that existing works on sparsity may fail at very early in the level of sparsity in comparison to what is known previously in the literature for rather simpler tasks such as imagenet classification or language modeling tasks.\nThis result can imply a few different things including ineffectiveness of existing methods and the need for new ones or even the need of re-thinking the benefit of sparsity itself.\n", "summary_of_the_review": "This paper develops a new benchmark for evaluating pruning. There still remain concerns to be resolved for the paper to serve as a new pointer for sparsity research.\n", "strength_and_weaknesses": "Strength\n- This paper provides a range of new and difficult evaluation tasks for pruning and serves as a new benchmark.\n- Overall the paper is well written.\n\nWeaknesses\n- The results achieved do not seem to provide significantly new or unknown information in the community. They basically show that iterative magnitude based pruning after training works well. The discovery is where pruning methods start to fail, but for more complex and difficult tasks it is quite obvious and expected that they will fail earlier. In this sense, the paper quite exaggerates the benefit of this new benchmark.\n- This paper serves to provide empirical results but also draw comparisons and conclusions based only upon it rather than any analysis of methods themselves; it is like A is better than B because of the performance, but without providing why. Perhaps it suits the purpose of creating a benchmark though, but to point out a limitation.\n- All evaluated methods are saliency-based pruning methods, among which all methods are magnitude based approaches except for random and SNIP. This is definitely a degrading factor, considering that throughout the paper the authors address in words a lot of different pruning approaches even including variational approaches and sparsity-inducing regularization based approaches.\n- The benchmark is missing one critical aspect of evaluating sparsity methods: lack of consideration of the total computations needed to achieve the trained performant sparse networks. Precisely, given a network model, how many forward/backward passess (or gradient computations) are needed for the entire process of finding and training the sparse network result, for each and every method being compared? Why is this important? It is because the benchmark claims to serve as a new standard evaluation protocol providing comparisons that one is better than the other. With different budgets or without even mentioning it properly is simply not fair for that matter. Potential evaluation protocols may include (1) for a fixed budget of computations, how good one method compares to the others in terms of final accuracy or sparsity level, or (2) for a fixed target accuracy/sparsity, how efficiently/faster one method achieves the goal compared to the others. It would also be better to include, at least in words level, how simple or difficult it is to tune the algorithms or hyperparameters (which includes the winning process in the LTH). Without these considerations, this benchmark may only provide a partial result.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "P0yGHJ2KWC", "reviewer": "Reviewer_JUeu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a new set of benchmarks dubbed \"SMC-Bench\" (Sparsity May Cry). The authors correctly remark that the ever-growing research in sparsifying neural networks is often (if not always) evaluated on limited set of benchmarks which are often one of the very well understood, and \"relatively easy\" datasets.\n\nThe authors then walk us over the different ways of sparsifying a neural network - providing a good summary of the current state of the art.\n\nThe paper continues with the introduction of the dataset, and providing experiments that author use to support their claims. ", "summary_of_the_review": "Please see above.", "strength_and_weaknesses": "The paper attaches an obvious question that seems not to have been addressed yet.  It's obvious that the authors spent an enormous amount of time and effort to collect all the results.\n\nThe authors promise to open-source their dataset, however no more details are provided. Additionally one would expect a well organized list of the \"10 open source repositories\" used for creating the benchmark. The open-sourcing and hopefully stream-lining future benchmarks is the most valuable part of the authors' work.\n\nI would also expect the authors to evaluate the networks on the \"weak\"/\"easy\" datasets - to confirm that their implementations/runs are consistent with what was claimed in the intro", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OX3dyJWw2RX", "reviewer": "Reviewer_yFvQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces “Sparsity May Cry” Benchmark (SMC-Bench), a collection of carefully curated 4 diverse tasks with 12 datasets, for a more general evaluation and unveiling the true potential of sparse algorithms. Evaluation reveals several important and unusual findings. ", "summary_of_the_review": "Overall, I would tend to accept this paper. If the author can address my concerns. I would be more convinced.", "strength_and_weaknesses": "Overall, I enjoy this work a lot and think it will make a big/good splash in the sparsity community. But I also believe its scientific rigor needs to be enhanced in several aspects before being treated as an indeed solid benchmark. See details below.\n \nFirst, it might be too early to declare sparsity will cry! For example, only first-order, unstructured pruning has been so far tested, which represent a strong yet still restricted subclass of modern pruning methods. I would definitely suggest the authors to pick representatives from other pruning families, such as second-order pruning.\n\nSecond, how the authors could justify their chosen datasets and tasks are indeed more challenging than ImageNet? IMHO, the sole fact that current pruning methods work better on ImageNet than those cannot directly support the claim, since it could also be some task types that are just more “pruning (un)friendly”. \n\nSimilarly, how the authors can justify their chosen models (admittedly very large already) are sufficiently “overparameterized” for those new difficult tasks? For example, would it be possible that, if continuing to increase the model size on those tasks, the generalization performance will keep improving, while pruning algorithms also start to gain back their effectiveness?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "9zYno96vWH", "reviewer": "Reviewer_wzLQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors assemble a large-scale, challenging, and more diverse benchmark, SMC-Bench, for pruning and sparse training algorithms. A careful evaluation indicates that SMC-Bench significantly challenges current SOTA sparse algorithms, exposes their limitations and points to new research opportunities. ", "summary_of_the_review": "This paper proposes an interesting method and the major concerns lie on more comparisons in the experiments.", "strength_and_weaknesses": "Strength:\n- This paper raises a very interesting new angle on the uprising field of sparse neural networks. Given the enormous that sparse neural networks have achieved, the authors ask a timely question: are current benchmarks sufficiently challenging for them, and (if indeed so) are we too optimistic in declaring the prevailing win of sparsity? The authors choose a few latest datasets and tasks that require substantial reasoning and structure information. This new benchmark will make an exciting addition to several existing sparse NN benchmarks such as Frankle et al. (2021).\n\n- Comparing a large range of before-training, during-training and after-training pruning methods (including both static and dynamic), the authors observe several surprising phenomena that are absent in the current evaluation. Most alarmingly, all of the SOTA sparse algorithms bluntly fail to perform on SMC-Bench, sometimes at significantly trivial sparsity e.g., 5%. The failure does not appear specific to one approach but unanimously across all sparse NN approaches that were evaluated. That points us to some rather fundamental rethinking of both sparse NN training and evaluation currently.\n\n- The authors also identified that model “prunability” is intimately related to task difficulty: those models trained on difficult tasks suffer more from pruning compared to easier tasks. Also, some best pruning algorithms start to perform indistinguishably from the random pruning baseline on the new challenging benchmark, and iterative pruning does not necessarily generalize better than one-shot pruning either. All those findings offer brand-new knowledge of broad interest to the sparse NN community.\n\nWeaknesses:\n- The paper did not identify a root cause for the failure of SNNs on SMC-Bench, although exploring then excluding a few factors. I feel one main reason is probably intimately related to the “lazy training” dynamics emerging in sufficiently overparameterized models, e.g., after training weights will only change very little from initialization. \nMoreover, in one of their sanity checks on whether pruning embedding layers or not, the authors found SNIP to significantly underperform others because it prunes too much of the pre-trained embedding while others not. I am wondering whether we could ad-hoc skip that embedding, for all pruning methods and then compare them?  \n\n- I strongly suggest the authors to add to their comparison one or more second-order pruning methods. such as Optimal Brain Damage, Fisher Pruning, or the recent Optimal BeRT Surgeon. \n\n- Also missed in the current benchmark is the structured pruning family: will they be more robust or fragile when task difficulty and data volume scale up? How about sparse MoE methods?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%