Dynamic Sparse No Training:  Training-Free Fine-tuning for Sparse LLMs
Authors: Yuxin Zhang
Lirui Zhao
Mingbao Lin
Sun Yunyun
Yiwu Yao
Xingjia Han
Jared Tanner
Shiwei Liu
Rongrong Ji
Venue: ICLR 2024 poster (2024)
Keywords: Large Language Models
Network Sparsity
Primary area: general machine learning (i.e., none of the above)
TL;DR: We introduce DSNT: a training-free fine-tuning approach to enhance the performance of sparse LLMs without weight update.

Abstract: The ever-increasing large language models (LLMs), though opening a potential path for the upcoming artificial general intelligence, sadly drops a daunting obstacle on the way towards their on-device deployment. As one of the most well-established pre-LLMs approaches in reducing model complexity, network pruning appears to lag behind in the era of LLMs, due mostly to its costly fine-tuning (or re-training) necessity under the massive volumes of model parameter and training data. To close this industry-academia gap, we introduce Dynamic Sparse No Training ($\texttt{DSNT}$), a training-free fine-tuning approach that slightly updates sparse LLMs without the expensive backpropagation and any weight updates. Inspired by the Dynamic Sparse Training, $\texttt{DSNT}$ minimizes the reconstruction error between the dense and sparse LLMs, in the fashion of performing iterative weight pruning-and-growing on top of sparse LLMs. To accomplish this purpose, $\texttt{DSNT}$ particularly takes into account the anticipated reduction in reconstruction error for pruning and growing, as well as the variance w.r.t. different input data for growing each weight. This practice can be executed efficiently in linear time since its obviates the need of backpropagation for fine-tuning LLMs. Extensive experiments on LLaMA-V1/V2, Vicuna, and OPT across various benchmarks demonstrate the effectiveness of $\texttt{DSNT}$ in enhancing the performance of sparse LLMs, especially at high sparsity levels. For instance, $\texttt{DSNT}$ is able to outperform the state-of-the-art Wanda by 26.79 perplexity at 70% sparsity with LLaMA-7B. Our paper offers fresh insights into how to fine-tune sparse LLMs in an efficient training-free manner and open new venues to scale the great potential of sparsity to LLMs.  Codes are available at https://github.com/zyxxmu/DSnoT.

Published as a conference paper at ICLR 2024
DYNAMIC SPARSE No TRAINING O: TRAINING-FREE FINE-TUNING FOR SPARSE LLMS
Yuxin Zhanglun 1.2f Lirui Zhao Mingbao Lin
Yunyun Sun Rongrong J
Yiwu Yao
Xingjia Han Jared Tanner
Shiwei L,63,6,7
Key Laboratory of Multimedia Trusted Perception and Efficient Computing. Ministry of Education of China, Xiamen University 2 Pengcheng Lab Tencent Youtu Lab "Huawei Technologies, University of Oxford, University Texas at Austin Eindhoven University of Technology, Innttittee of Artificial Intelligence, Xiamen University
ABSTRACT
The ever-increasing large language models (LLMs), though opening a potential path for the upcoming artificial general intelligence, sadly drops daunting obstacle on the way towards their on-device deployment. As one the most well- established pre-LLMs approaches in reducing model complexity, network pruning appears to lag behind in the era of LLMs, due mostly to its costly fine-tur seee-tuning (or re-training) necessity under the massive volumes of model parameter and training data. To close this industry-academia gap, we introduce Dynamic Sparse No Training (DS@T"). a training-free fine-tuning approach that slightly updates sparse LLMs without the expensive backpropagation and any weight updates. Inspired by the Dynamic Sparse Training, DSOT minimizes the reconstruction error between the dense and sparse LLMs, in the fashion of performing iterative weight
on top of sparse LLMs. accomplish this purpose, DSOT particularly takes into account the anticipated reduction in reconstruction error for pruning and growing, as well as the variance w.r.f different input data for growing each weight. This practice can be executed efficiently in linear time since its obviates the need of backpropagation for fine-tuning LLMs. Extensive experiments ILLMA-VI/V2, Vieuna, and OPT across various benchmarks demonstrate the effectiveness of DSOT in enhancing the performance of sparse LLMs, especially at high sparsity levels. For instance, DSOT is able outperform the state-of-the-art Wanda by 26.79 perplexity at 70% sparsity with LLaMA-7B. Our paper offers fresh insights into how fine-tune sparse LLMs in an efficient training-free free manner and open new venues to scale the great potential of sparsity to LLMs. Codes are available at /ttttt:://giiiiiiis
com/ wwxxxm/DSSoo
INTRODUCTION
Large language models (LLMs) (Zhang et al. 2022a Touvron al.) 2023a Brown al. 2020 have recently emerged as the new favorite in various domains of natural language processing (NLP) (Wei et al. 2022b Bubeck al. 2023). Nevertheless, L.MSs face a significant constraint:
their extensive parametenization and computational demands present substantial challenges in terms of storage and deployment. For example. the GPT-175B model (Brown al.) 2020) eats up 320G of memory to load its parameters in FP16 precision, requiring at least five A100-80G GPUs for inference (Frantar & Alistarh 2023). In response to this issue, there has been surge of interest in compressing LLMs, as it holds the promise of LLMs while remarkably reducing memory usage and computational costs. To date, the majority of current effort for LLM compression falls into quantization (Yao et al.) 2022. Lin et al. 2023 Frantar et al. 2022 Dettmers et al. 2023 2022] Xiao et al., 2023] Shao et al.] 2024 Ma et al. (024), which compresses LLMs by diminishing the number of bits employed to represent weights hidden states.
"*Equal contribution &Corresponding author: mrj@@ rrji@xmu.comm xmm.cdu.cc
Pronnuuceed "DS No T".

---

Published as a conference paper at ICLR 2024
Wanda
Wanda
Wanda DSOT
165
DSØT
SparseGPT
SparseGPT
Perplexity
SparseGPT w. DSØT
Running Time
le2
8-
lcl
6-
Model
Model
Figure 1: Perplexity WikiText-2 (left) and running time (right) of different methods for pruning LLaMA-VI model family at 60% sparsity rate. Without any training, DSO consistently improves the performance of sparse LLMs, all within a linear time spectrum.
On the other hand, network pruning (LeCun et al. 1989 Han et al. 2015 Mocanu et al. 2018).
technique that removes superfluous weights create a sparse and lightweight model, has received relatively little attention (Frantar & Alistarh 2023 Sun et al., (023). The plausible reason is that, network pruning usually appreciates least one, usually many, iterations of fine-tuning or re-training to guarantee top performance (Frankle & Carbin 2019 Yin et al., (203)). This fine-tuning step would cause a significant amount compute and memory footprints due the colossal model size and massive training data of modern LLMs, which even unnerves large corporations, let alone individual researchers.
Two previous arts have explored the possibility to scale pruming billion-level LLMs without any fine-tuning. SparseGPT (Frantar Alistarh 2023) formulates LLM pruning as a layer-wise weight reconstruction problem, where the target falls into mitigating the output discrepancy, w.E., recon- struction error, between dense and sparse LLMs. To solve the row- oww-Hessian challenge, i.e., the need for calculating the expensive inversion of huge matrix for each row individually, SparseGPT itera- tively applies OBS (Hassibi et al. 1993) to individually prune and updates weights in column-wise manner, ultimately reaching the same optimal solution applying the Elosed-form regression recon- struction. Wanda (Sun et al. 2023) proposes new pruming metric that takes both weight magnitude and their corresponding input activations into consideration, performing on part with SparseGPT without the need for the expensive second-order information. The intuition behind Wanda lies in the existence of emergent outlier feature dimensions in large-scale I.I.Ms which are significantly larger than typical features and meanwhile are essential for the optimal performance of L.MMs (Dettmers) cet al. 2022). While these two approaches enable LLM pruning without performing fine-tuning, their performance is still far from satisfactory, e.g., starting lose performance 20% sparsity with LLaMA-30B. Therefore, is imperative to enable fine-tuning for sparse LLMs to fully unlock the potential of sparsity escalate the affordability of LLMs.
In a parallel vein, Dynamic Sparse Training (DST), as outlined previous research (Mocanu et al, 2018 Liu al. 2019; Evci et al. 2020), has garnered considerable attention recently due to its significant saving potentials in the context neural network training. Instead of training an entire network, DST selectively updates and maintains subset of the network throughout the training pro- cess, while allowing the sparse network topology to dynamically evolve via a weight operation (Mo- canu al. 2018). Given its demonstrated efficacy in achieving efficient training, DST seems be promising candidate for efficient LLMs fine-tuning. However, it is essential to note that DST in- trinsically requires the training of subnetworks via backpropagation, and the effectiveness of mask adaptation highly relies on sufficient number of weight updates (Liu et al. 2021). Moreover, prior studies have indicated its failure when employed for fine-tuning small-scale BERT-level language models (Liu et al. 2023).
Fortunately, is noteworthy that the mowning-nnd-roowine step employed DST solely stands as fraining-free methodology, enabling sparse mask adaptation based on certain weight status, e,g. magnitude (Mocanu et al. 2018). This offers an alternative perspective for addressing the aforemen- tioned challenge: While tinin sparse LLMs through backpropagation computational overhead, we can explore the possibility of iteratively updating sparse mask in a training-free fashion viable alternative. Based this intuition, introduce training-free
result in substantial
2

---

Published as a conference paper at ICLR 2024
fine-tuning approach- Dynamic Sparse No Training (DSOT). This approach empowers the fur- ther refinement of sparse LLMs without any weight updates. To facilitate mask adaptation in favor of the sparse reconstruction problem, propose new criteria for mask pruning and growing, by considering both the expectation and variance of the reconstruction error reduction when recovering specific weight. It is worth emphasizing that the DSQT functions independently of the need for computationally intensive operations, such as gradient or Hessian matrices, Instead, it exclusively relies on a singular matrix multiplication operation to assess the reconstruction error.
We conduct comprehensive experiments to evaluate the effectiveness of DSOT with a variety of LLMs. including LLaMa-VI (Touvron et al.] 2023a) and LLaMa-V2 (Zhang al. (022a), Vi- cuna (Chiang et al.] 2023), and OPT families (Zhang et al.] (2022a). from 7 billion to 70 billion parameters. Our results demonstrate that D consistently improves the performance of sparse L.MMs by good margin, especially at high sparsity levels > 50%. For instance, DS@T is able to improve the performance over Magnitude pruning, SparseGPT, and Wanda by 1. 4.31, and 1.87 perplexity with OPT-13B WikiText-2 at 60% sparsity only using 7.3s on a single NVIDIA A100 GPU. Our work provides fresh insights in efficient sparse LLM fine-tune without weight updates and we hope to encourage more research in exploring benefits of sparsity in LLMs.
2 RELATED WORK
Network Sparsification. The process of eliminating redundant weights, known as network sparsi- fication or network pruning, has served as practical strategy to diminish the complexity of deep neural networks over the past decades (LeCun al.. 1989 Han al. 2015). Despite the substantial body of literature, network pruning can be roughly classified based on the granularity of sparsity and the dependency of the pre-trained dense models. I. Granularity of Sparsity: The granular- ity of sparsity varies from coarse grains to fine grains. The coarse-grained granularity can be a group of weights (Gray et al. 2017; Ding et al.) 2017), a complete neuron (Jiang et al.) 2018); a filters/channels (Li et al. (0017). or an attention head (Voita et al.) 2019), etc. On the other hand, fine-grained granularity eliminates the least important weights based on the selected criteria, regard- less of where they are (Gale et al. 2019). The advantage of coarse- coarse-grained sparsity is its pronounced acceleration effect, which yet typically suffers from larger performance loss. Fine- fine-grained sparsity enjoys performance superiority compared other more structured forms of sparsity but receives limited support in common hardware. Nonetheless, recent advancements of dedicated fine-grained sparse patterns, such as N:M sparsity (Zhou et al. 2021 Zhang et al. 2022b), can be effectively accelerated. As such, this paper focuses on fine-grand tine-grined network pruning. I1. Dependency of Pre- trained Networks: In parallel, sparsification techniques can be grouped into dense-to-sparse and sparse-to-spane methods based on the necessity of an over-parameterized dense network. The for- mer entails embarking from pre-trained dense model and discovering a sparse network (Han et al.) 2015 Wen et al. 2016 Molchanov et al. 2017: Gale et al., 2019 Kurtic et al.) (022). usually fol- lowed by a retraining process to recover the optimal accuracy. On the other hand, sparse-to-sparse methods aim to train sparse neural networks from scratch, omitting any preliminary steps involving
dense pre-training (Mocanu et al.) 2018 Lee et al.] 2019; Evei et al.] 2200;; Wang et al. [2020] Liu et al. 2021). Among them, Dynamic Sparse Training (DST) (Mocanu al. 2018, Evei et al. 2020
Liu et af. 2021) stands out and receives upsurging interest due to its promise in saving both
training and inference phases. In contrast the conventional practices of pre-training followed by pruning, DST distinguishes itself by commencing with a randomly initialized sparse neural net- work. During a single training run, dynamically adjusts the sparse network topology by such as pruming-and -odggrooing, without the need for pre-training, while maintaining moderate training costs by. for example, keeping the similar sparsity ratios across all varying masks (Mostafa & Waan 2019
Dettmers& & Zeutlemyeer 2019: Yuan et al. 2021 Jayakumar et al. 2020).
While the crux of this paper focuses on the first category, Le., pruning a pre-trained LLM model, our proposed method is mainly inspired by the muuing-and--rooingg utilized in DST to iteratively refine the binary masks in a training-free free manner, even though we do not conduct weight training as such. Another line of research, akin to our approach, demonstrates the existence of "supermasks" within randomly initialized network (Zhou et al.) 2019; Ramanujan an et al. 2020 Huang et al.) 2022) or pre-trainn trained networks (Mallya et al.) 20188 Wortsman et al. 2020 Zhang al. (202)), exhibiting the capacity to achieve commendable performance solely by seeking binary masks. However, it is imperative to note that these methods heavily rely on backpropagation, which is ill-suited for L.Mss.
3

---

Published as a conference paper at ICLR 2024
Pruning of LLMs. Compared to the well-established promise of pruning in pre-LLM small-scale models, the advancement of pruning in the context of LLMs appears to exhibit relatively modest progress. Firstly, traditional pruning generally requires least iteration of re-training to recover performance. Considering the substantial model size and massive datasets associated with LLMs, the prospect of conducting such resource-intensivv re-training becomes formidable challenge. To mitigate the above challenge, researchers have introduced pruning algorithms specifically devised for LLMs compression. Ma al. (022)) explored structured sparse LLM by applying Taylor prun- ing [Molchanov et al. (017) ) to remove entire weight rows, followed by the parameter efficient fine- tuning (PEFT) technique (Hu et al. 2021) fine-tuning. However, the fine-tuning phase still demands considerable amount of data while the performance suffers significant degradation, attributed primarily the garse-grained level of sparsity. Recent research endeavours have evolved towards the direction of unstructured pruning in one-shot without fine-tuning, demonstrating significant pro- gresses. SparseGPT (Frantar & Alistarh 2023) incorporates the Hessian inverse for pruning and subsequent residual weight updates, whereas Wanda (Sun et al. (203)) directly arrives at sparse LLM model by a criterion depicted by the multiplication of the absolute values of weights and their activations with the aim to preserve outliers (Dettmers et al. 2022) emerged in LLMs. DSOT serves as an orthogonal perspective and can be organically integrated on top of them.
DYNAMIC SPARSE NO TRAINING - DSOT
Preliminary. LLM pruning entails the removal of certain proportion of pre-trained weights to obtain sparse LLM, with the objective of achieving minimal discrepancy between the output of sparse and dense models (Hassibi al. 1993). Solving this problem can very arduous given the immense scale of LLMs. Therefore, it more practical to formalize LLM pruning as layer- wise reconstruction problem (Hubara et al. 2021 Frantar Alistarh, 2023). Denote the weights of one dense LLM layer W E Ro where Cour and Cin stand for the number of output and input channels respectively, Supposing we have N calibration samples, the input activation can be represented A XxL with L be the sequence length. Pruning be viewed as devising binary mask M E {0, Ca, to indicate whether weights are removed or not. Hence, the problem of LLM pruning given a specifie pruning rate p can be formalized
|
min M.W
W * A (M
W) * A I|2, s.f. 1
(1)
Cout Cin
where O. ||. 1112 denote matrix multiplication, dot product operation, and &2 norm, respectively. Note we refer A E N- to the reconstruction enror for ease of the following text.
Dynamic Sparse No Training. The problem defined in Eq. (1) can be addressed from two complementary per- spectives. Firstly, can be resolved through the initial- ization of sparse networks i.e., devising criteria to prune weights that exhibit minimal impact on model output. For instance, SparseGPTT (Frantar & Alistarh, 2023) employs second-order Hessian inverses, while Wanda (Sun et al., 2023) considers products of weight and activation norm the guide for weight removal. Secondly, for the obtained sparse networks, the remaining weights fine-tuned to further compensate for the reconstruction er- ror (Han et al., 2015). Unfortunately, this requires sub- stantial training resources, which is not practical given the large volumes L.MMs. Therefore, SparseGPT adjusts the remaining weights via iterative OBS update (Hassibi & Stork 1992), which as a consequence remarkably reduces the computing demands.
Weights
Pruning
Reconstruction Error
Update
Growing
Pruning
be naturally
Figure 2: Framework of DSØT.
In this work, our focus is on the second part, i.e., how to efficiently reduce the reconstruction error of given pruned sparse network to its dense counterpart? Instead of fully fine-tuning (Han al. 2015) or partially updating the pruned LLMs (Frantar & Alistarh (023) to recover performance, we intro- duce ultra-efficient yet effective alternative refine the sparse mask after pruning based on their
4

---

Published as a conference paper at ICLR 2024
contribution to the reconstruction error. Our approach is inspired by the pruning-and and--andddddn growing opera- tion used in Dynamic Sparse Training (Mocanu et al. 2018 Evci et al.) 2020). DST incorporates the processes of weight pruning and weight growing within the framework of sparse network training, contributing to the discovery of improved sparse topologies. Note that this muning-and--rowwe operation solely serves training-free approach that is able to adapt sparse masks towards de- sirable perspective, e.g,, loss minimization. Based on this insight, we propose DSOT, a training- fine-tuning method for sparse LLMs that strips weights updating in DST and keeps the pruning-and- growing by converting the optimization objective to the reconstruction enror of each weight row. We isolate pruning- pruning-and-grew and-growing from network training, and formulate it as an iterative approach to progressively optimize sparse masks towards the desirable ones achieving minimal reconstruction error represented by Eq. (
Specifically, DSOT starts with a sparse L.M
Algorithm 1:
Pseudocode of DSOT.
which can be pruned by any existing crite-
ria (Jaiswal et al. [0233 [Snn et al. [023 Fran-
Input: A sparse layer with weight WO, maximum cycle T, update threshold
tar & Alistarh| (002)) Then, it performs itera- tive weight growing and pruning by looking at
Workflow DSßT:
Initialize reconstruction error A via Eq.
the reconstruction error as defined in Eq. (I). with especially-des -eesiggdd criteria decrease the output discrepancy between sparse L.MSs and their dense counterparts. The framework of DS@T is illustrated in Figure its main parts are detailedly described below.
for = 1 to Cout do
for f= to T do
Obtain the growing index via Eq. Obtain the pruning index j via Eq.
M =1
M.j. =
Update reconstruction error A. via
Growing Criterion. As each output neu- ron is computed independently, we use one weight row W, and the corresponding mask M, for illustration. Given sparse weight row M,&W we attempt to revive pruned weight that leads to the most decrease on A, across different input activations. Therefore, ance of the reconstruction error change when recovering a weight back. In particular, the index of the revived weights is derived as follows:
Eq.
if A, then
break
return Fine-tuned sparse weights W M.
growing criterion considers both the expectation and vari-
arg max -M,,k Wr. [[AA]/Var( Var(A,), i[ > O, k arg arg min -M,, W.. /arr//VarAAA otherwise,
(2)
where EE()) and Var(.) stand for the expectation and variance of given inputs across N I. different tokens. To explain, [[A,] Wr represents the expected influence of weight growing on A,- Thus, based on the sign of the reconstruction error A,, we can determine which weight should be restored to approach the decrease of Ar. Furthermore, we consider introducing the variance of the input activation to achieve a more robust revival. This intuitive because if the influence of weight on A, exhibits high variance across different inputs, restoring it may not result in stable error reduction.
Pruning Criterion. After choosing revived weights, we need to select another weight for pruning in order to maintain fixed sparsity rate. However, the circumstances here are distinct: if we prune weights based on the impact of reconstruction error change as per Eq. ( there is a risk of removing weights that significantly influence the output. This concern becomes especially critical when prun- ing LLMs due to the presence of emergent large magnitude features within them (Dettmers et al.) 2022 Wei et al.) 2022a; Schaeffer et al.) 2023). To alleviate this, we utilize a transformed version of the Wanda metric (Sun et al. 2023). In addition to its standard criterion for pruning weights, we mandate that the selected weights should also contribute positively towards the reduction of recon- struction error when being pruned. This helps in preserving critical weights from removal without compromising the stable decrease of reconstruction error during the training-free fine-tuning pro- cess. Therefore, the pruning index j is obtained as follows:
arg min k.M. W W,, ElAA arg min M.M. (.A..
M, W. ||A, 112, E( > 0,
j
(3)
M2 Ww ||Ar otherwise.
5

---

Published as a conference paper at ICLR 2024
Table 1: WikiText-2 Perplexity comparison for pruning LLMs at 60% sparsity rate.
LLaMA-VI
LLaMA-
Vicuna
OPT
Method
7B
13B
30B
65B
7B
13B
70B
13B
13B
Dense
5.68
5.09
4,10
3.56
5.47
4.88
3.32
5.94
10.12
Magnitude
5.6c2
2.3e2
15.97
8.18
6.9e3
10.11
13.35
14.39
1.1c6
w. D
66.70
30.71
7.37
40.01
9.41
6.77
12.02
2,4c2
SparseGPT
10.41
8.43
6.81
5.83
10.14
7.88
5.10
10.02
21.23
DSQT
9.65
7.73
6.69
5.64
9.67
7.57
5.07
9.38
16.92
Wanda 10.69
8.75
6.56
5.90
10.79
8.40
5.25
9.54
15.88
DSQT
10.22
8.46
6.44
5.75
10.59
8.18
5.20
9.18
14.01
Workflow. Given the criteria depicted above, the workflow of DS@T is outlined in Algorithm In particular, it iteratively performs weight growing and pruning with respect to Eq. (2) and Eq. (3) with the reconstruction enror updated until it reaches pre-defined threshold. Meanwhile, we set a maximum pruning-and--rowing cycle T to prevent certain rows from being unable to reach the settled threshold
Remark, It's noteworthy that Aggorthmm[Toutines the processing of each row a sequential man- ner, primarily for the sake of simplicity. However, it's imperative acknowledge that each row can, in fact, undergo parallel processing by employing a binary indicator to assess whether particular row has satisfied the termination condition. Furthermore, the DSOT process eliminates the neces- sity for resource-intensive procedures such as backpropagation or the computation of gradient and Hessian matrices. Instead, it relies solely on several matrix multiplications calculate the recon- struction error, a task that can be executed efficiently on GPUs. Subsequently, during each iteration of the DS@T process, the only operation is to update the reconstruction error through straightfor- ward addition and subtraction operations during the pruning-ande -and-aadddiiggg process. This approach effectively circumvents the introduction additional algorithmic complexity. In summary, DS@T preserves the simplicity associated with pruning LLMs, akin the approaches employed in Wanda and Magnitude pruning.
EXPERIMENTAL RESULTS
4.1 SETTINGS
Implementation details. The implementation details our proposed DSOT are presented as fol- lows, mostly conforming to the existing setups (Frantar & Alistarh 2023 Sun et al. 2023). In context to pruning configuration, we adhere to SparseGPT (Frantar & Alistarn 2023), where a uni- form sparsity is imposed for all layers with the first embedding layer and the final classification head skipped. Meanwhile, the calibration data consists of 128 segments, each with 2048 tokens. These segments randomly selected from the first shard of the C4 dataset (Raffel al. 2020). For the parameter settings, we set the maximum cycle T = 50 and the update threshold 0.1 in all experiments. Given sparse LLMs, we apply D to fine-tune each layer in progressive man- ner. We implement T in PyTorch (Paszke et al. 2019) and use the HuggingFace Transformers library (Wolf et al. 2019) for handling models and datasets. All pruning experiments are conducted on NVIDIA 1100 GPUs with 80GB of memory.
hyper-parameter
Baselines. We principally work with the LLaMA-VI (Touvron et al.) 2023a), LLaMA-V2 Touvron et al. 2023b). Vicuna (Chiang et al. 2023). and OPT families (Zhang al.) 2022a), from 7 billion to
70 billion parameters, which are among the most powerful and open-source Large Language Models
(LLMs) in the field today. We run DS
sparse LLMs pruned by various methods including (I)
Magnitude- based pruning (Han et al.) 2015) that discards weights based on their magnitudes. (2) SparseGPT (Frantar & Alistarh 20233 that utilizes second-order Hessian inverses ascertain unim- portant weights. (3) Wanda (Sun et al. 2023) that removes weights with the smallest magnitudes
multiplied by the corresponding input activation norms.
Evaluation. In accordance with prior studies (Frantar et al. 2022 Dettmers et al. 2023 Yao
al. 2022 Frantar & Alistarh, 2023), assess the performance of pruned models by caleu-

---

Published as a conference paper at ICLR 2024
Table 2: WikiText-2 perplexity performance of DSØT for fine-tuning sparse LLaMA-VI-7B65B pruned by the Wanda metric at varying sparsity rates.
L.AMA-VI-7B
ILaMA-VI- 1--5BB
Sparsity 50%
60%
70%
80%
90%
50%
60%
70%
80%
90%
Wanda
7.26
10.69
88.84
4.80e3
6.41e5
4.57
5,90
15.24
2.06e3
3.21e4
W. DSOT
7.12
10.22
62.05
4.12e3
8.43e4
4.54
5.75
12.93
1.82e3
2.09c4
lating the perplexity of language generation experiments separate validation sets derived from WikiText2 (Merity et al.) all|[20111 (2016). While perplexity has served as a stable and robust indicator of the generative performance of models (Dettmers & Zettlemoyer 2023) we also examined the zero-shot capabilities of pruned models. In detail, we report the accuracy in six zero-shot tasks including PIQA (Bisk al. 2020). StoryCloze (Mostafazadch al. 2017). ARC Easy and Challenge (Clark) et al. 2018). HellaS (Zellers et 2019) and Open (Mihaylov et al. 2018). We imple- ment the an-eval-harness (Gao al. (022)) for the execution of all zero-shot tasks, with the report including both the accuracy results on each benchmark and overall average accuracy.
4.2 LANGUAGE MODELING
Quantitative results. The results for fine-tuning sparse LLM models at a uniform sparsity rate of 60% are presented in Table Irrespective of the datasets used for evaluation, DS@T consis- tently delivers performance improvement for sparse LLMs with their original sizes varying from 7B to 70B. For instance, when pruning LLaMA-VI V1 with 7B parameters, DSOT is able to enhance the performance of Magnitude (Jaiswal et al.) 2023), SparseGPT (Frantar & Alistarh (023). and Wanda (Sun et al., 2023) by 4.94e2, 0.76, and 0.47 perplexity on the Wikitext-2 validation sets, respectively. is worth noting that, without any weight updating, DS@T consistently demonstrates better performance than SparseGPT, which requires expensive second-order Hessian inverses to up- date the sparse model. For larger models, the efficacy of DSOT is still hold with performance gain from 13.35 to 6.77 perplexity when fine-tuning sparse LLaMA-VA V2-700B obtained by magnitude pruning (Han et al. 2015). These findings suggest DSOT's versatility, being adaptable to boost the performance of sparse LLMS with different parameter budgets.
Varying Sparsity Rates. We further investigate the efficacy of DSØT when fine-tuning sparse LLMs with varying pruning rates. Table[shows that DS@T offers effective performance enhance- ment across various pruning methods at different sparsity levels. Particularly, this improvement becomes increasingly evident the sparsity level grows.
Table 3: Time overhead (in seconds) for pruning LLaMA-V VI model family.
Table 5: Wikitext-2 perplexity comparison for pruning LLaMA-VI model family with N:M pattern.
Method
7B
13B
30B
65B
Method
Sparsity
7B
13B
30B
65B
SparseGPT
209
337
721
1285
Dense
5.68
5.09
4.10
3.56
Wanda
0.3
0.5
1.1
1.9
SparseGPT
4:8
8.61
7.40
6.17
5.38
Wanda+DSOr
4.3
7.4
15.7
23.7
DSOT
4:8
8.32
7.05
6.10
5.12
Wanda
4:8
8.57
7.40
5.97
5.30
Table 4: Comparion with LoRA fine-
w. DSOT
4:8
8.45
7.25
5.91
5.26
tuning using 50% sparse LLaMA-7B.
SparseGPT
2:4
11.00
9.11
7.16
6.28
Method
Time Cost
Perplexity
w. DSOT
2:4
10.03
8.36
6.82
5.80
Wanda+LoRA
4h
6.87
Wanda
2:4
11.53
9.58
6.90
6.25
Wanda+DSQT
4.3s
7.12
w. DSDST
2:4
10.89
9.05
6.76
6.14
Computing efficiency. We further demonstrate the efficiency of DS@T. Following Wanda, we only report the total pruning time and exclude the forward pass process shared by all methods. Table compares the quantitative wall-clock overhead evaluated NVIDIA A100 GPUs. It is indeed encouraging to observe that, as a fine-tuning approach, DSØT maintains a comparable computing time to Wanda, while demonstrating significantly higher efficiency compared to SparseGPT
Comparison with LoRA Fine-tuning, To further demonstrate the ultra efficiency of DSØT in terms of fine-tuning, we also compare DSOT with parameter efficient fine-tuning (PEFT) method

---

Published as a conference paper at ICLR 2024
Table Zero-shot Accuracy comparison for pruning LLaMA-VI model family 60% sparsity rate.
Params
Method
PIQA
HellaSwag
StoryCloze
ARC-e
ARC-c
OBQA
Mean
Dense
78.7
56.9
76.8
75.3
41.8
34.0
60.6
7B
SparseGPT
73.1
44.8
71.5
62.6
30.2
24.4
51.1
w. DSOT
73.7
47.2
72.3
62.8
30.9
29.4
52.7
Wanda DSOT
73.0
43.6
69.7
62,8
30.3
25.0
50.7
73.2
43.7
70.0
63.6
30.8
25.8
51.2
Dense
79.1
59.9
78.4
77.4
46.5
33.2
62.4
13B
SparseGPT
75.6
49.0
74.8
68.4
36.2
27.6
55.2
w. DSØT
75.8
51.5
75.8
69.8
36.3
28.8
56.3
Wanda
74.9
48.9
74.5
68.9
34.9
27.6
54.9
w. DSOT
75.0
49.1
75.1
69.2
35.4
28.0
55.3
Dense
81.1
63.3
79.1
80.4
52.9
36.0
65.4
30B
SparseGPT
76.8
55.0
78.4
74.7
43.3
32.2
60.1
w. DSØT
77.3
58.0
78.8
74.8
45.6
32.8
61.2
Wanda
77.7
56.7
79.1
76.2
46.5
31.6
61.3
w. DSOT
78.1
56.7
79.7
76.8
46.6
32.6
61.7
Dense
81.2
64.6
80.2
81.3
52.9
38.2
66.4
65B
SparseGPT
79.6
58.3
80.5
77.4
46.6
33.4
62.6
w. DSOT
79.9
59.8
80.4
78.1
46.9
34.6
63.3
Wanda
79.9
58.9
80.6
47.1
34.8
63.3
W. DSOT
80.9
59.6
80.2
78.2
47.7
36.0
63.7
LoRA (Hu et al. 2021). Table 4 presents presentss a comparison of the time and performance of both methods fine-tuning sparse LLaMA-74 7B. LoRA leverages the complete C4 dataset for 5-hour fine-tuning and achieved perplexity of 6.84. In stark contrast, DSOT only requires brief duration of 4.3s and 128 samples to deliver a comparable performance, 7.12 perplexity. Taking into consideration the additional parameter burden incorporated by LoRA, the efficiency and practicality of DSOT is hold.
N:M Fine-grained Sparsity. Compared with unstructured sparsity. N:M fine- tiee-grrindd sparsity offers more practical speedup on the NVIDIA Ampere sparse tensor core (Nvidia, 2020). Thus, we also evaluate the effectiveness DS@r on N:M fine-grained sparsity. Given the unique pattern of N:M sparsity that stipulates N non-zero components within M consecutive weight block, implemen- tation of D involves a restriction on the position of saniiingand-growinn weights. In particular, select the pruned weight within the same block as the revived weight, thus the N.M charac- teristic is still maintained after fine-tuning. Table lists the results for pruning LLaMA-VI model family at 2:4 and 4:8 sparse patterns. Interestingly, even with the aforementioned extra restriction, DSOT can achieve more significant performance improvement compared to previous methods. For instance, when pruning LLaMA-VI with 7B parameters, DSØT archives perplexity of 10.89, hancing Wanda (1 (11.53) by noticeable 0.64 ppl. Similar findings can be concluded when it comes other models and sparse patterns. These results highlight the effectiveness of DSOT in boosting the performance of sparse LLMs, even with more complex sparsity constraints.
4.3 ZERO-SHIOT TASKS
Following Frantar & Alistarh 2023 Sun et al. 2023), we also provided the accuracy performance of the ILaMA- V1 model family pruned at 50% sparsity rate on seven downstream zero-shot tasks. Av- eraging the accuracy over all tasks suggests DSOT's efficacy for enhancing sparse LLMs of any size. Particularly, DSOT improves the average accuracy of SparseGPT by 1.6% when pruning LLaMA- V1-7B (52.7% for DSØT and 51.1% for SparseGPT), For task-wise performance, DS@T is benefi- cial on all tasks, while there is not a fixed superiority for fine-tuning models obtained by different pruning methods. This phenomenon may evidence the reported relatively noisy evaluation results from these zero-shot experiments (Dettmers et al. 2022). However, the advantages of consistent performance improvement and efficiency of D for zero-shot tasks obvious.
8

---

Published as a conference paper at ICLR 2024
10.7
E-0
Wanda
10.6
E-0.1
26
Wanda DS2T
Perplexity
10.5
E-1
Perplexity
22
SparseGPT
SparseGPT w. DSØT
10.4
18
10.3
10.2
16a
10a
10
256
512
Maximum Cyele (T)
Calibration Samples
and Figure 3: (left) Effect of the update schedule (7, and (right) number of calibration sequences.
4.4 PERFORMANCE ANALYSIS
Next, investigate the influence of the components within DS@T, unfolds its update schedule, pruning-and- growigg criteria, and robustness to calibration samples. All experimental setups are based on the LLaMA-7B model pruned by the Wanda metric (Sun al. 2023) with 60% sparsity.
Update schedule. In Figure (left), we examine the performance of DSOT under different hyper- parameter setting for the update schedule, including the maximum cycle C' and stop threshold The best performance is obtained with 50 cycles and 0. updating threshold. To analyze, smaller C and larger both lead an insufficient procedure for the decrease in reconstruction error. In contrast, running DS@T without termination conditions also resulted in poor performance, most likely due to over-fitting of calibration data.
Robustness to calibration samples. In Figure (right), we show the performance of pruning meth- ods with varying numbers of sampled sequences for calibration. As can be observed, SparseGPT suffers serious performance degradation when calibration samples are limited, mostly due to the difficulty in estimating Hessian inverses such cases. Fortunately, DS@T consistently the perfor- mance of SparseGPT, robustness of DSØT for mitigating the reconstruction error.
if only very few samples given. These results further highlight the
aoaniing-an-grooiiin criteria. We further investigate the influence on criteria for prune and grow in Table Note that when transfer Eq. (2) to the prune criteria, the elec- tion of extreme values is also correspond- ingly reversed, As for the prune criterion, can be seen that pruning weights that could bring the most reduction in reconstruction enror actually led to a significant perfor- mance decrease. This indicates that while pursuing the reduction of reconstruction error, it is also essential to keep weights that exhibit an extremely large influence on the output, e.g., weights within outlier channel. On the other hand, proposed criteria based on the expectation and variance of the reconstruction error reduction achieved the best results among all growing criteria.
Table 7: Effect of the pruning and growing criteria.
Growing
|W..l ||Ar||2
Eq.
Eq.
Pruning
|W..l ||A
10.72
10.49
10.27
Eq.
11.24
10.61
10.84
Eq.
10.52
10.37
10.22
5 CONCLUSION
In this work, we introduce DS@T, a training-free fine-tuning approach that enhances the perfor- mance of sparse LLMs without the expensive backpropagation or any weight updates. Taking spiration from the success of sparse training in the pre-LLM pruning age, DSØT adapts iterative weights growing and pruning in sparse LLM, with a transferred target for minimizing the recon- struction error between dense and sparse LLMs outputs. To furnish guidance in the selection of weights to be pruned and grown, we introduce novel criteria that take into account the expectation and variance of the reconstruction puts. Extensive experiments on pruning representative LLMs across various language benchmarks demonstrate the efficiency and effectiveness of DS@T in boosting the performance of sparse L.MMs.
reduction by growing each weight concerning different in-

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work was supported by National Science and Technology Major Project (No. 2022ZD01 8200), the National Science Fund for Distinguished Young Scholars 6o.62025603), the National Natural Science Foundation of China (No. U21B2037, No. U22B2051, No. 62176222, No. 62176223, No. 62176226, No. 62072386, No. 62072387, No. 62072389, No. 62002305 and No. 62272401), and the Natural Science Foundation of Fujian Province of China (N0.2021J01002 No.2022106001).
REFERENCES
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical com- monsense natural language. Proceedings of the AAAI conference on artificial intelligence (AAAI), volume 34, pp. 7432-7439, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems (NeurIPs), 33:1877-1901, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka- mar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* a chatgpt quality. See https://vicuna, Imsys. org (accessed 14 April 2023), 2023.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1XX3::8800.05455 2018
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiiv1907.0484// 2019.
Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. In International Conference on Machine Learning (ICML), pp. 7750-7774. PMLR, 2023.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 O: 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems (NeurIPs), 2022.
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashk- boos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized repre- sentation for near-lossless lim weight compression, arXiv preprint arXiv:2306.00 203002306.03078,, 2023.
Caiwen Ding, Siyu Liao, Yanzhi Wang, Zhe Li, Ning Liu, Youwei Zhuo Chao Wang, Xuehai Qian, Yu Bai, Geng Yuan, et al. Cirenn: accelerating and compressing deep neural networks using block-circulant weight matrices. In Proceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture, pp. 395-408, 2017.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference Machine Learning (ICML), pp. 2943- 2952, 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations (ICI.), 2019.
Elias Frantar and Dan Alistarh. Massive language models can be accurately pruned in one-shot. In International Conference Machine Learning (ICML), 2023.
10

---

Published as a conference paper at ICLR 2024
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training compression for generative pretrained transformers. In International Conference on Learning Representations (ICLR), 2022.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity deep neural networks. arXiv preprint arXiv: 0957049797 2019.
Trevor Gale, Matei Zaharia, Cliff Young, and Brich Elsen. Sparse gpu kernels for deep learning. In International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-14, 2020.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Version vO. 0.1. Sept, 2021.
Scott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights. arXiv preprint arXiv: 3:2, 2017.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems (NeurIPS), pp- 1135-1143, 2015.
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in Neural Information Processing Systems (NeurIPS), pp. 164-171, 1992.
Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In JEEE international conference on neural networks, pp. 293-299. IEEE, 1993.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0968 2021.
Tianjin Huang, Tianlong Chen, Meng Fang, Vlado Menkovski, Jiaxu Zhao, Lu Yin, Yulong Pei, Decebal Constantin Mocanu, Zhangyang Wang, Mykola Pechenizkiy, et al. You can have bet- ter graph neural networks by not training weights at all: Finding untrained gnns tickets. arXiv preprint arXiv:2211.15333 2022.
Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Joseph Naor, and Daniel Soudry. Accel- erated sparse neural training: A provable and efficient method to find n: m transposable masks. Advances in Neural Information Processing Systems (NeurIPs), 34:2 34:21099-22111, 2021.
Ajay Jaiswal, Shiwei Liu, Tianlong Chen, and Zhangyang Wang. The emergence of essential spar- sity in large pre-trained models: The weights that matter. arXiv preprint arXhv:2306.03805, 2023.
Siddhant Jayakumar, Razvan Pascanu, Jack Rae, Simon Osindero, and Erich Elsen. Top-kast: Top- k always sparse training. Advances in Neural Information Processing Systems (NeurlPs), 33: 20744-20754, 2020.
Chunhui Jiang, Guiying Li, Chao Qian, and Ke Tang. Efficient dnn neuron pruning by minimizing layer-wise nonlinear reconstruction error. In IJCAL, volume 2018, pp. 2-2, 2018.
Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models. arXiv preprint arXiv:2203.07259 2022.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In Advances in Neural Informa- tion Processing Systems (NeurIPS), pp. 598-605, 1989.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based on connection sensitivity, In International Conference on Learning Representations (ICLR), 2019.
Hao Li, Asim Kaday, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference Learning Representations (ICLR), 2017.

---

Published as a conference paper at ICLR 2024
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.00977 2023.
Liu, DC Mocanu, ARR Matavalam, Y Pei, and M Pechenizkiy. Sparse evolutionary deep learn- ing with over one million artificial neurons commodity hardware. arxiv. arXiv preprint arXiv: 1xXiv:/901.00 190119991,.,,, 2019.
Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do actually need dense over wver-aarmtterizationn! in-time over-parameterization in sparse training. In International Conference Machine Learning, pp. 7909-7000. PMLR, 2021.
Shiwei Liu, Tianlong Chen, Zhenyu Zhang, Xuxi Chen, Tianjin Huang, Ajay Jaiswal, and Zhangyang Wang. Sparsity may cry: Let us fail (current) sparse neural networks together! arXiv preprint arXiv:2303.0214/ 2023.
Xinyin Ma. Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. arXiv preprint arXiv:2305. 1627, 2023.
Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang, Shilei Wen, Fei Chao, and Rongrong Ji. Affinequant: Affine transformation quantization for large language models. In International Conference Learning Representations' (ICLR), 2024.
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to mul- tiple tasks by learning to mask weights. In Proceedings of the European conference computer vision (ECCV), pp. 67-82, 2018.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint NEGV:1600000843 2016.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.0277 027899 2018.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature Communications, 9:1-12, 2018.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In International Conference on Learming Repre- sentations (ICLR), 2017.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural net- works by dynamic sparse reparameterization. In International Conference on Machine Learning (ICML), pp. 4646-4655, 2019.
Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. Lsdsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pp. 46 51, 2017.
Nvidia.
Nvidia a100 tensor core gpu architecture, 2020.
https://
www. http://www.....coecoo//////eee/a
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), pp. 8026-8037, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2185555855- 2020.
12

---

Published as a conference paper at ICLR 2024
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Raste- gari. What's ' hidden in randomly weighted neural network? In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11893-11902, 2020.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? arXiv preprint arXiv:2304. 15004, 2023,
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li. Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. In International Conference on Learning Representations (ICLR), 2024.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1695, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 13977
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0921 092888, 2023b.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.094/1 2019.
Chaoqi Wang. Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations (ICLR), 2020.
Jason Wei, Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems (NeurIPs), 15:24824-248871 2022b.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances in Neural Information Processing Systems (NeurlPs), 29, 2016.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's 's transformers: State-of-the-ar natural language processing. arXiv preprint arXiv:1910.03771 2019.
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. Advances in Neural Information Processing Systems (NeurIPs), 33:15173-15184, 2020.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning (ICML), pp. 38087-38099. PMLR, 2023.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems (NeurlPs), 35:27168-2787 2022.
Lu Yin, Shiwei Liu, Meng Fang, Tianjin Huang, Vlado Menkovski, and Mykola Pechenizkiy. Lot- tery pools: Winning more by interpolating tickets without increasing training or inference cost. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 37, pp. 10945- 10953, 2023.
Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, et al. Mest: Accurate and fast eccoomic sparse training framework on the edge. Advances in Neural Information Processing Systems (NeurIPs), 34: 20838-20850 2021.
13

---

Published as a conference paper at ICLR 2024
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint arXiv:195.007830, 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. 2022a.
Yuxin Zhang, Mingbao Lin, Zhihang Lin, Yiting Luo, Ke Li, Fei Chao, Yongjian Wu, and Rongrong Ji. Learning best combination for efficient M sparsity. Advances in Neural Information Processing Systems (NeurIPS), 2022b.
Yuxin Zhang, Mingbao Lin, Fei Chao, Yan Wang, Ke Li, Yunhang Shen, Yongjian Wu, and Ron- grong Ji. Lottery jackpots exist in pre-trained models. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2023.
Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hong- sheng Li. Learning n: M fine-grained structured sparse neural networks from scratch. In Interna- tional Conference on Learning Representations (ICLR), 2021.
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: zeros, signs, and the supermask. In Advances in Neural Information Processing Systems (NeurIPS), pp- 3597-3607, 2019.
14

---

Published as a conference paper at ICLR 2024
A
APPENDIX
A.1
COMPLEMENTARY EXPERIMENTAL RESULTS
In this section, we supplement the main paper with more experimental outcomes, including wider spectrum of results at varying sparsity rates, robustness analysis under random seeds, and quantita- tive comparison with varying numbers of calibration sequences.
Varying Sparsity Rates. This part delivers extended results of DS@T when fine-tuning sparse LLMs at alternating sparsity rates as a supplement to Section 4 The performance of various LLMs with sparsity rates oscillating between 10% and 90%, are presented Table Beneficial enhance- ments are consistently observable at all examined sparsity levels when employing DS@T, with the significance of improvements escalating concurrently with the increase in sparsity. is notewor- thy that the acceleration resultant from unstructured sparsity comes into play predominantly at high sparsity levels (exceeding 60%) Gale et al. (2020). thereby accentuating the indispensable efficacy of DSOT.
Table 8: WikiText-2 perplexity performance for fine-tuning LLMs at varying sparsity rates.
Model
Method
10%
20%
30%
40%
50%
60%
70%
80%
90%
LLaMA-VI-78
Wanda
5.70
5.82
6.00
6.39
7.26
10.69
88.84
4.80c3
6.41e5
LLaMA-VI-7B
w. DS@T
5.68
5.73
5,89
6.28
7.12
10.22
62.05
4.12e3
8,43e4
LLaMA-VI-13B
Wanda
5.10
5.13
5.25
5.51
6.15
8.75
55.89
3.66e3
1.54c6
LLaMA-VI-13B
w. DSOT
5,09
5.11
5.05
5.29
6.08
8.46
43.31
1.12e3
1.95e5
LLaMA-V2-7B
Wanda
5,49
5.59
5.74
6.06
6.92
10.79
75.01
2.36e3
7.87e3
LLaMA-V2-7B
w. DSOT
5.48
5.49
5.65
5.85
6.81
10.59
53.12
1.12e3
2.35e3
LLAMA-V2-13B
Wanda
4.91
4.99
5.13
5.37
7.88
8.30
46.05
1.06c3
1.22e5
LLaMA-V2-13B
w. DSOT
4.89
4.91
5.01
5.25
7.57
8.13
33.19
2.59e2
3,49e4
OPT-13B
Wanda
10.13
10.09
10.12
10.63
11.92
15.88
55.07
13722
7.61e5
OPT-13B
w. DSØT
10.12
10.08
10.11
10.41
11.28
14.01
45.10
8.43e3
2.33e5
Varying Number of Sample Sequences. Table shows the quantitative results of different methods with varying numbers of calibrated sequences in complementary with Figure Indeed, SparseGPT largely outperforms Wanda when the sample number starts to exceed 512. The performance gap gets larger with the length of 2048. It is worth mentioning that the efficacy of DS@T is indeed obvious when very limited numbers of calibration samples are given. Mcanwhile, it is also encouraging to that DSOT can consistently improve the performance of SparseGPT and Wanda even with 2048 calibrated sequences. This highlights the effectiveness of DSOT even when the pruning baseline is considerably strong, i.e., SparseGPT with long imput length.
Table 9: WikiText validation perplexity for different methods in pruning LLaMA -1-7B at 50% sparsity with varying number of calibration sequences.
Sample Length
8
16
32
64
128
256
512
1024
2048
Wanda
13.18
12.11
11.29
11.04
10.83
10.68
10.69
10.65
10,54
10.68
10.77
w. DSOT
12.52
11.22
10.91
10.71
10.62
10.42
10.22
10.15
10.12
10.38
10.41
SparseGPT
30.23
26.04
12.92
11.87
11.45
10.79
10.40
10.41
10.39
9.99
w. DSØT
17.19
15.61
11.62
11.02
10.33
10.04
10.04
10.03
10.02
9.70
15

---

Published as a conference paper at ICLR 2024
Robustness Analysis. We further perform robustness analysis of DS@T. Given that the results in Table evaluated under a fixed calibration set, Table I0 show the results with different calibration sets under 5 random seeds. The variance across random seeds is very low, suggesting the stability of DSOT, corroborating its efficacy as a tool in fine-tuning sparse L.Mss.
Table 10: Wiki'Text validation perplexity for pruning LLaMA-VI and LLaMA-V2 models at 60%
sparsity. We report the mean and standard deviation under 5 random seeds.
LLaMA-VI
ILaMA-V2
Method
13B
7B
13B
Dense
5.68 (+:0.00)
5.09 (+0.00)
5.47 (=00000)
4.88 (=10.00)
SparseGPT
10.42(=0.04)
8.43(+0.02)
10.14 (4-0.03)
7.88(40.01)
w. DSOT
9.64(10.03)
7.73(10.02)
9.68(10.03)
(..001
Wanda
10.69(=0.01)
8.75(10.01)
10.79(100001
8,40(10.01)
w. DS@T(+0001)
(0.22(-0.01))
8.46(+-0011)
10.59(+0.01)
8.18(+0.01)
16


Reviews:
[{"review_id": "UNWADn63cj", "reviewer": "Reviewer_CfsK", "summary": "This paper introduces Dynamic Sparse No Training, a training-free fine-tuning approach for sparse Language Model (LLM) deployment. It minimizes the reconstruction error between dense and sparse LLMs through iterative weight pruning-and-growing. This approach allows for updating sparse LLMs without the expensive backpropagation and weight updates, making it more efficient for on-device deployment. The paper demonstrates the effectiveness of the proposed method on several benchmark datasets, achieving comparable or better performance than traditional fine-tuning approaches while requiring significantly less computation. The contributions of this paper include the introduction of a training-free fine-tuning approach for sparse LLMs, and the demonstration of its effectiveness on several benchmark datasets.", "questions": "please follow weaknesses", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In terms of originality, the paper introduces a novel approach to fine-tuning sparse LMs, called Dynamic Sparse No Training. This approach does not require backpropagation or weight updates, making it more efficient for on-device deployment.\n\nIn terms of quality, the authors provide detailed descriptions of the datasets and experimental setup, as well as a thorough analysis of the results. The paper also includes a comprehensive review of related work, highlighting the strengths and weaknesses of existing approaches.\n\nIn terms of clarity, this paper is well-written, with clear explanations of the proposed approach and experimental results.\n\nIn terms of significance, the paper addresses an important problem in the field of LMs, namely the challenge of deploying large models on resource-constrained devices.", "weaknesses": "1. For the inner loop, how does the threshold affect the final performance of the model, for both perplexity and efficiency?\n\n2. The paper assesses the methods using a consistent sparsity rate of 60%. However, a 50% sparsity rate is more commonly employed in previous baselines. It would be beneficial to see the outcomes at this 50% sparsity level for comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BJtLWNLT7I", "reviewer": "Reviewer_s71w", "summary": "This paper presents DS$\\oslash$T, a novel training-free fine-tuning approach for sparse LLMs which edits sparse mask configuration inspired from DST. DS$\\oslash$T revives weights which negatively contribute to reconstruction error between dense and sparse LLMs, and prunes weights based on Wanda metric and the sign of reconstruction error. By conducting experiments across a wide range of tasks, authors show that DS$\\oslash$T can be seamlessly integrated with existing LLM-pruning techniques, achieving state-of-the-art results at  >50% sparsity regime with minimal computational overhead.", "questions": "- How many random seeds are used throughout the experiments? \n- Why is LLM-pruner [2] not included in the baselines while N:M structured pruning is included? \n\n[1] Sun et al., “A simple and effective pruning approach for large language models.” 2023.\\\n[2] Ma et al., “Llm-pruner: On the structural pruning of large language models.” 2023.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper tackles a timely and practically-relevant problem supported by a fair amount of experiments conducted spanning different tasks and domains. Notably, this is the first work tackling the LLM pruning problem at >50% sparsity regime.\n- The proposed method has two distinctive advantages: (i) it doesn't require gradient computation, resulting in minimal overhead costs, and (ii) it reconfigures existing sparse masks, making it compatible with existing LLM pruning methods.\n- In general, the paper is well-written and easy to follow.", "weaknesses": "- Throughout the paper, the authors report performance at >50% sparsity regime. However, in most cases, DS$\\oslash$T does not bring the performance of sparse neural networks even remotely close to that of the original dense network. To better understand DS$\\oslash$T, I recommend reporting experimental results at a lower sparsity regime. For instance, as mentioned in the introduction, the authors stated that baselines start to lose performance at 20% sparsity with LLaMA-30B. Does the use of DS$\\oslash$T preserve performance at 20% sparsity regime?\n- The paper notes that the overhead cost of using Wanda+DS$\\oslash$T is approximately 15 times greater than using Wanda alone in Table 3. However, it appears that the performance gain achieved in Tables 5 and 6 is relatively marginal. For instance, while a previous work [1] argues that the zero-shot classification performance of Wanda and SparseGPT is similar (as seen in Table 2 in [1]), the application of DS$\\oslash$T to Wanda or SparseGPT does not consistently result in substantial improvements over the respective baselines. This observation raises questions about the trade-off between computational cost and performance improvement when employing DS$\\oslash$T in combination with existing methods. \n- In Table 4, the authors make a comparison between DS$\\oslash$T and LoRA. Since DS$\\oslash$T alters network structure (pruning mask), the paper would benefit from analyzing whether the resulting sparse network structure from DS$\\oslash$T can be further optimized with full-finetuning or LoRA. I wonder whether Wanda+DS$\\oslash$T can achieve better fully fine-tuned accuracy compared to that of Wanda.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4qA6uBfiUI", "reviewer": "Reviewer_nrCN", "summary": "Inspired by the weight pruning-and-growing method in dynamic sparse training, the authors propose a training-free fine-tuning method to sparsify LLMs. In practice, the proposed method iteratively performs weight pruning and growing with new importance metrics that take into account the expectation and variance of the reconstruction error reduction. The proposed metrics allow to eliminate the expensive backpropagation or any weight update in the original dynamic sparse training to enable training-free fine-tuning for LLMs. In the experiments, the authors conduct experiments on multiple benchmarks and show better performance when migrating into prior training-free methods.", "questions": "Detailed questions regarding Weakness:\n\n1. It's unknown how the algorithm is used for sparsifying an entire LLM. \n\n    1.1  Regarding the entire LLM, since the proposed method aims to reduce the reconstruction error in layer-wise, will the proposed method progressive prune each layer or jointly prune all the layers for an entire LLM?\n\n    1.2 How to assign the sparsity rate for each layer? \n\n2. It seems the proposed method is an improved technique for existing training-free methods. In the related work, it seems that SparseGPT and Wanda have different importance metrics to sparsify LLM, and this work proposes an orthogonal method. \n\n    2.1 What is the difference between SparseGPT and Wanda when integrating the proposed method? \n\n     2.2 Why the proposed method can not be considered as an independent method for training-free LLM sparsification?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing is clear and easy to follow.\n2. Although the idea of pruning-and-growing is not new, the proposed method is novel to eliminate backpropagation and weight update with new importance metrics to enable training-free LLM sparsification\n3. The proposed method consistently shows a clear performance gap on multiple benchmarks with varied sparsity rates compared to prior works.", "weaknesses": "1. It's unclear how to use the proposed dynamic sparse no training in the whole fine-tuning process. In the paper, the authors mainly illustrate the training-free pruning-and-growing method for one specific layer. It's unknown how the algorithm is used for sparsifying an entire LLM. \n\n2. It seems the proposed method is an improved technique for existing training-free methods. From the Related work, it can not tell the difference between SparseGPT and Wanda when integrating the proposed method. \n\n3. The proposed method involves extra computing and running time compared to Wanda.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)