Fast and Effective Weight Update for Pruned Large Language Models
Authors: Vladimír Boža
Venue: Accepted by TMLR ()

Abstract: Pruning large language models (LLMs) is a challenging task due to their enormous size. The primary difficulty is fine-tuning the model after pruning, which is needed to recover the lost performance caused by dropping weights. Recent approaches have either ignored fine-tuning entirely, focusing on efficient pruning criteria, or attempted layer-wise weight updates, preserving the behavior of each layer. However, even layer-wise weight updates can be costly for LLMs, and previous works have resorted to various approximations. 

In our paper, we propose a fast and effective weight update algorithm for pruned layers based on the Alternating Direction Method of Multipliers (ADMM). We further extend it with a simple gradual pruning mask selection and achieve state-of-the-art pruning performance across a wide range of LLMs.

Published in Transactions on Machine Learning Research (07/2024)
Fast and Effective Weight Update for Pruned Large Lan- guage Models
Vladimír Boža
boza @fmph.uniba... uniba.
Faculty of Mathematics, Physics and Informatics, Comenius University, Bratislavo, Slovakia
Reviewed on OpenReview: https: // openreview. net/ fttl/form? id- 1hcpXd9Jir
Abstract
Pruning large language models (LLMs) a challenging task due to their enormous size. The primary difficulty fine-tuning the model after pruming, which is needed to recover the lost performance caused by dropping weights. Recent approaches have either ignored fine-tuning entircly, focusing efficient pruning criteria, attempted layer-wise weight updates, preserving the behavior of each layer. However, even layer-wise weight updates can be costly for LLMs, and previous works have resorted to various approximations.
In paper, propose fast and effective weight update algorithm for pruned layers based on the Alternating Direction Method of Multipliers (ADMM). We further extend it with a simple gradual pruning mask selection and achieve state-of-the-art pruming performance across a wide range of LLMs. The code is available at htttp::/ggiii http://itttuu...o/fftf/o. com/fmfi compbio/ admm- Sham--pruiing
1 Introduction
Large language models (LLMs) (Brown et al., 2020; Zhang et al., 2022 Touvron et al.) 2023a b) have displayed impressive performance in different tasks, but deploying them can be challenging due to their large size and high memory demands. In this work, introduce one-shot pruning and weight update algorithm for LLMs that is both fast and effective. Our algorithm produces stato-of-tho-art results for LLM pruning while imposing minimal computational overhead (Table 1
Neural networks are usually compressed by either quantization or weight pruning. LLM quantization
[Dettmers et al. 2022 Dettmers & Zettlemoyer 2023 Ahmadian et al. 2023 Xiao et al. 2023 compresses
LLMs by storing weights using a small number of bits. On the other hand, pruning compresses models by dropping irrelevant weights (LeCun et al. 1989 Han al., [2155] Zhu & Gupta 2018). Pruning can be helpful for LLMs since, during inference, the main bottleneck memory bandwidth for loading weights to processing unit (Xia et al.) 2023 However, the main challenge in deploying LLM pruning is that the network needs to be fine-tuned BBlloock et al.] 2020 Liu et al. 2018), which is not feasible with LLMss due to extensive computational and memory footprint. For example, Agarwalla et al. (2024) needed retraining on 45 100 billion tokens to recover lost performance by pruning. Also, memory-effcient fine-tuning like LoRA (Hu et al. 2021) not applicable for L.LM weight pruning since we cannot, easily merge the low-rank update with the sparsified matrix.
A feasible alternative is one-shot pruning, where one is given trained model with small amount of cali- bration data, and has to compress the model in single forward pass using limited computational resources. This is typically done via layer-wisc pruning, where the pruning problem is split into layer-wise subprob- lems. In each layer, one aims to select a pruning mask and update weights to minimize reconstruction error. Adaprune (Hubara et al. 2021) solves layer-wise reconstruction by updating weights directly via gradient descent (using Adam optimizer). However, noods many iterations to achieve convergence. Optimal brain compression (OBC) (Frantar & Alistarh 2022) removes weights one by one. In each step, caleulates the optimal weight to remove and also the optimal update. However, this approach is also very time-consuming

---

Published in Transactions on Machine Learning Research (07/2024)
Table 1: WikiText perplexity of pruned LLaMA-7B. Our ADMM-based methods are superior previous ones.
Method
Sparsity
Perplexity
Dense
0 %
5.68
Wanda
50 %
7.26
SparseGPT
50 %
7.22
ADMM1
50 %
7.20
ADMM-Gras
50 %
7.06
Wanda
60 %
10.66
SparseGPT
60 %
10.51
ADMM1
60 %
9.96
ADMM-Grad
60 %
9.22
Wanda
70 %
85.77
SparseGPT
70 %
26.30
ADMM1
70 %
26.31
ADMM-Grad
70 %
18.66
Wanda
80 %
5e3
SparseGPT
80 %
154.75
ADMMI
80 %
202.04
ADMM-Grad Grad
80 %
69.46
Wanda
2:4
11.53
SparseGPT
2:4
11.00
ADMM1
2:4
10.38
ADMM-Grad
2:4
9.90
The first practical approach applicable to LLMs was SparseGPT Frantar & Alistarh, 2023) using approximations on top of the OBC approach to make the problem feasible, albeit at the cost of decreased reconstruction quality.
for pruning LLMs.
Recently, Wanda (Sun al., 2023) showed that LLMs can be pruned by removing weights with the smallest product of weight magnitude and corresponding input activation norm. This selection approach without the weight update is competitive with SparseGPTT on lower sparsities (up to 60%).
Our results. In this paper, we introduce efficient layer-wite wise weight update algorithm based on alternating direction method of multipliers (ADMM) (Boyd et al. 2011). Our algorithm sidesteps all of the problems of previous solutions. We do not need many gradient descent iterations, nor do we need any heuristic approximation for calculating the weight update. We only need a single inversion matrix similar the weight matrix and very few simple iterations to achieve accurate weight updates for given pruning mask. Furthermore, step, prune more and more weights. This simple extension allows us to obtain stale-ol-the-arl pruning results at a very low additional cost.
extend our algorithm with gradual pruning (Zhu & Gupta 2018), where in cach
2 Preliminaries
2.1 Large language models and transformers
Large language models (like Llama) use transformer (Vaswani et al. 2017 architecture and are trained predict the next word in the text. Transformer consists multiple repeating blocks. Each block has multihead attention and a foed-forwer subblock, which contain multiple linear transformations. Our work focuses pruning weights in these linear transformations.
2

---

Published in Transactions on Machine Learning Research (07 (224))
2.2 One- and layer-wise pruning
We consider scenario of post-training pruning, where we prime already trained model to a desired sparsity (we assume that the sparsity is the same in each pruned layer).
Since manipulating the whole LLM at once leads to huge computational and memory requirements, we follow the works of Hubaaa et al. (202T); ) Frantar Alistarh (2022) 2023). We prune the LLM during one forward pass (one-shot pruning) and split pruning into multiple layer-wist wise subproblems. During the forward pass, we capture the calibration inputs for each layer and then prune and update each layer accordingly. More specifically, for each block in the model, run forward pass through it, capture imputs for each layer, prine and update weights, and then rerun a forward pass through the whole block to get outputs after pruning.
We are given the original weights We for each layer and calibration inputs Xz. Our goal is to find a binary weight mask Me and updated weights We such that the following reconstruction error is minimized:
|| Xe @ WollB
For now, we assume that pruning mask Mj was found via a separate method and focus only on finding updated weights We.
Assuming that our layer has output neurons and inputs, regressions to solve the problem optimally. Since the mask for each output is different, cach one of n outputs requires a separate matrix inversion of the relevant submatrix of X Xx, which in total takes O(m²n) time. This is infeasible even for small neural networks. It is possible to use various approximations to compute updates faster, as done in SparseGPT (Frantar Alistarh 2023). However, we demonstrate our experiments that this compromises the quality of the solution. Another approximation to not update weights and prune weights with the lowest product of magnitude and input activation norm, done in Wanda (Sun et al. 2023).
just solve independent linear
Another possible solution is to update weights iteratively via gradient descent as in Adaprune Hubara et al. 2021). Here, one update step is proportional to X O We Wz). Assuming X²', X2 is precomputed, one update step takes O(m"n) time. While this looks much better than solving linear regressions, Frantar & Alistarh (2023) as well as our own experiments show that one needs many iterations to achieve reasonable convergence.
2.3 Alternating Direction Method of Multipliers
Alternating direction method of multipliers (ADMM) (Boyd et al. 2011 is an optimization method for solving problems in the form:
minimize [(x++ + g(z)
subject to Ax + Bz
where f(x) and g(z) are convex functions. ADMM forms the augmented Lagrangian with dual variables y and penalty factor p:
Lo 2, = f(x) + g(z) + vJ (Ax + B2 + GllAr Bz c|1
Typically ADMM is given using scale dual variable u = y/p a form:
2,(,, f()) + g()) + (11A + B2 + u1 -1u|11)
The Lagrangian is then optimized via. the following iterations:
3

---

Published in Transactions on Machine Learning Research (07 (224))
k+1 arg mig f(x) (p/2)] + Bz - u 12
arg min g (p/2) BB c u0
k+1 u *+1 + B2* k+1 -
It can shown that ADMM converges to the optimal solution when f and g are convex and some other mild assumptions hold Boyd et al. (2011]. Assumption The (extended-reals real-uuuedd functions f IR" RU +00 and g: R RU+00 are closed, proper, and conver.
Assumption 2. The wnaugmented Lagrangian Lo has saddle point, i.e. there erists (a* ,,",,""" where for all x, 2, y: Lo(x", (""",==",,,"00"""""""""""""""" ,),,) S. Lo(x", 2² v²" S Lo(x, 2,y")
Theorem 1. Let Assumplions and hold. Then:
Ar + Bz + as k - oa, ie. ilerales approach fensibilily.
f(*²) + g(²*) approach optimal value as k
One application of ADMM is solving constrained optimization over convex set C, i.e.:
minimize f(x) subject to x EEC
This problem be rewritten into ADMM form using indicator function 9, where g(z) = 0 if 2 E C, and g(2) oc otherwise:
minimize f(x) + g(=) subject to x -- =
(1)
In this case, the ADMM update becomes:
x k+1 = arg min f(x) + (p/2)]lx 2h + +
x
k+1 [x( + )²
(2)
w *+1 w² + xxtt1 k+1
Here, He is an Euclidean projection operation onto set C. Also, note that the update just the original unconstrained problem with simple quadratic penalty term.
3 Methods
Here, we propose alternative solution to finding updated weights in the layer-wise pruning problem. Our solution will have same iteration complexity as gradient descent, but will converge much faster. Recall, that given set of calibration inputs Xe and mask Mt and are looking for W,, such that reconstruction error || - (M O WolB is minimized.
We observe that when a set zeroed weights fixed, valid weight matrices form a convex set C. In other words, we are solving the following constrained optimization problem (we omit subscript for clarity):
min W
|XW - -
subject to
W ④ (1 - M) =
4

---

Published in Transactions on Machine Learning Research (07/2024)
Our objective is also convex and thus we use ADMM to solve our optimization problem. We denote our objective as f(W) ||X - xWI2 and we will use indicator function g(Z) which takes value 0 if Z о (1 18-MM M) = 0 and oc otherwise. Using formulation [1] and updates 2 we get following iterations:
Wh+1 arg min f(W) (p/2) 2 + 1
w
zk+1 = II U*)
(3)
U = U Wh+1 zk+1
In our case, Z-update is just a projection onto the set of valid matrices, thus:
z = (1 + U
(4)
Updating W is very similar to ridge regression and can be computed as:
Wh+1 = (x - - pX)- XW + p( -U*))
(5)
Corollary 1. For fixed calibration imput X and mask M iterates 3 (with updates 4 solution for weight update.
5
converge to optimal
Proof. Since algorithm uses ADMM iterations, we only need to prove that assumptions 1 and 2 hold. f and g are clearly closed, proper, and convex functions; thus, assumption holds.
To show that assumption holds, we need to prove that there exists (W*, ,²"""" that for all (W, W, LoW",, 3) LWW v²,v") I LoW,, Z,y") where Lo(W, Z, f(W) + g(Z) y (W- Z).
There a globally optimal solution W Z (can be found by independent lincar regressions), where: Lo
,2) "0'"""""","" f(W) (() and thus LW 7" .y) S (W y²"
If Múj =1 (Wij is unmasked and can have any value), then we set vtg = 0. II Mij = 0, then Zi Wij and we set Viy = 0ل_ aw.
Then all W and Z derivatives of Lo are zero (or Zij must be 0 due to masking) at Lo LoW",, "W*"***"0"" z²,y") and since Lo is convex in W and Z, then we have a global optimum for given y" and thus (oW", ,"")= I Lo(W (W, Z, y And thus, assumption holds.
We can precompute and cache (X"X + pf)-1 and X XW, and then one update iteration has O(m"n) complexity, which the same as the complexity of gradient descent. Note that theoretical results do not say anything about the speed of convergence. In the experimental section, we will show that, in practice we get high-quality solutions after very few itcrations.
One can also view W update as way of pulling pruned weights towards zero. Note that for unpruned weights, the penalty term (p/2)] - -1=* only limits the step size, but for pruned weights, the value of Z T will have different sign than the value of W and thus they will be strongly pulled towards zero.
3.1 Mask selection and preconditioning
In the previous section, we described how to update weights when we are given the spansity mask. Now, we will discuss how to select the mask for pruning.
Wanda (Sun et al. 2023 a simple rule to select a high-quality mask for pruming LLMs. Instead of selecting weights with the largest value (magnitude pruning), they select weights with the highest product of weight absolute value and input neuron norm, i.e. [Wal] ||Xxl12- In implementation, we follow this selection
5

---

Published in Transactions on Machine Learning Research (07/2024)
rule, but use the norm of imputs as preconditioning. We multiply the weight matrix by feature norms and divide calibration inputs by their feature norms, run the ADMM algorithm, and then normalize the weight matrix back. Note that after the preconditioning, selecting the mask by weight magnitude is equivalent to the Wanda algorithm and that the diagonal of X contains only oncs.
Wanda paper also suggests keeping a constant number of weights per output. We found that our case with weight update, this constraint is actually slightly detrimental, and in our work, we select the top weights for the whole layer.
3.2 Gradual pruning
Until now, we considered scenario where one selects the pruming mask first and then updates weights. Here, propose simple extension to algorithm, which progressively prunes more and more weights and simultancously computes the weight update. Note, that this still happens during one forward pass, we will just apply multiple iterations to one layer-wise problem.
We adopt cubic sparsity schedule from Zhu & Gupta, 2018 where sparsity at step t is computed as 9 = (R.) ) where sy is final sparsity and hg is the number of sparsification steps. In each step, set st weights to zero and then proceed with the ADMM update. Note, that only overhoad of gradual pruming is just a mask selection added into cach step. While Z represents the current valid solution, we found that it is slightly better to use Wi+11 + I for selecting weights to prune. This is actually the optimal choice our constraint (function g) was specific sparsity, not predefined mask. We summarize our pruning algorithm in Algorithm 1
We also extend gradual pruming to structured 2:4 sparsity using the following straightforward idea. Our final sparsify will be sy 0.5. If in step t our target sparsity is st, then we always keep the two highest elements from each group of four and then prune 28, weights from the remaining ones.
Algorithm Layerwise gradual pruning with ADMM. Given weight matrix W, calibration input X, desired sparsity 8f, number of iterations k number of sparsification steps k,, dampening factor X (usually 0.1) and penalty factor (usually 1), we prune matrix W to desired sparsity and accuratelly update weights for the given weight mask.
norm + || +
W < W norm
X + X/norm
XX + X + +X
1/ X is usually 0.1
XXW + +XX. W
// precomputation
XX-1 =(XX +l)
for step 1..k do
if step <= h then
3
8; 8f
M +- mask lowest indices from IW + U
end if
+ (W + U) * *M
U- U (W- Z)
W WX XXX-IXXXW
pZZ -U)
end for
W + (W + U)+ M/norm
3.3 Comparison with SparseGPT and Wanda
Compared to SparseGPT (Frantar & Alistarh 2023). our algorithm does a more accurate weight update since it does not rely on approximation (we also verify this later in the experimental section). Tt is difficult

---

Published in Transactions on Machine Learning Research (07/2024)
to say which mask selection algorithm is better in theory. We gradually prune the whole weight matrix while SparseGPT does optimal selection on group columns of the weight matrix iteratively. But in our experiments our mask selection leads to better results.
Our algorithm can also be thought of as Wanda (Sun et al. 2023 with added weight updates and gradual pruning.
3.4 Note on using ADMM with Lo penalty
It is possible to use ADMM to optimize functions under Lo constraint heuristically. This was previously done by Zhang et al. (2018); Ye et al. (2019); Gui et al. (2019). While some the papers claim that this reality, using ADMM with Lo constraint is just heuristic since the constraint is not convex. Moreover, in our preliminary experiments, we found that ADMM with Lo constraint. is very sensitive to the choice of p. and for some choices, it will actually run in cycles and not converge.
approach is "systematic',
4 Experiments
General setup. We implement our algorithms by extending the Wanda (Sun et al. 2023 codebase, which relies on Pytorch and the Huggingface library, Similarly to Wanda and SparseGPTT, we use 128 calibration samples from the C4 training dataset (Raffel et al. 2020). We pruning RTX 5000 GPUs (each with 16GB of GPU memory). Since prune layers sequentially in order, need only to load one layer to GPU memory at one time. This allows us to prune 70B parameter LLaMA models using a relatively small GPU. Unless stated otherwise, we prune for k = 20 iterations, using k. = 15 sparsification steps, and set the dampening factor to À = 0.1 and ADMM penalty factor p p=
machine with two Quadro
We compare our methods to Wanda (Sun et al., 2023), which does not do weight update and just prunes weights with the lowest product of magnitude and activation norm, and SparseGPT (Frantar & Alistarh 2023), which uses multiple approximations to select pruned weight and calculating weight updates. For both methods, use their public implementation and default hyperparameter settings.
Models and evaluation. We test our methods on LLaMA Touvron et al. 2023a) and LLaMA2 Touvron
et al. 2023b models. Similarly to previous works Frantar & Alistarh, 2023 Sun et al. 2023 we measure the performance of pruned models on language modeling and zero-shot tasks. Our main focus is perplexity on held-out WikiText (Merity et al., 2016),, considered a goto metric for evaluating language model compression (Frantar & Alistarh 2023). As additional verification and testing, we the same seven tasks as Wanda
uses from EleutherAT LM Harness (Gao et al. 2021
4.1 Reconstruction error convergence
As first experiment, we study the quality of our update algorithm. We use fixed sparsity mask derived using Wanda with 50% sparsity and observe reconstruction error convergence in one layer. We compare our algorithm to gradient-based approaches using Adam and SGD optimizers with varying learning rates. We also compare it to the SparseGPT update (without mask selection) used in the Wanda paper.
The results for selected layers of LLaMA-7b are presented in Figure/ 1. Our ADMM-based algorithm is superior to both gradient-based algorithms and SparseGPT as it converges almost instantly after computing the initial x matrix inverse. We also note that ADMM works well with the default setting of p = 1 and does not require learning rate tuning, which starkly contrasts with SGD and Adam, which have different optimal learning rates in different layers.
4.2 Weight update quality comparison
In this experiment, we first prune cach layer of LLaMA-7B 60% or 80% sparsity using Wanda mask selection and then update weights either using gradient- based (via Adam) or ADMM update. We select the pruning mask in single step, i.e. do not do any gradual mask selection. We test using 1, 10, 20, 50,
7

---

Published in Transactions on Machine Learning Research (07/2024)
0.mlp.gate proj
1.self attn.o__pp ___rj
15.0
3000
12.5
2000
0
2
0.0
0.5
1.0
1.5
1.mlp.down _roj
2.self attn.k proj
25000
20000
15000
10000
10
0.5
1.0
admm
adam_0.003
adam_ 0.003
adam_ 0.1
.....:
sgd_0.00 0.0_
adam 0.0003
adam_ 0.01
sgd_ 0,000,01
[gd_0.03
adam 0.001
adam 0.03
0.003
SparseGPT
Figure Reconstruction error over time (in seconds) during optimizatiin of weights in selected layers of LLaMA-7B. The mask was derived by Wanda using 50% sparsity. We compare our proposed ADMM algo- rithm to SGD with momentum and Adam using various learning rates. We also compare to the SparseGPT update. Our ADMM update converges much faster than other methods and is better than the SparseGPT update.
Table 2: Comparision of weight update quality between ADMM and SparseGPT on Llama-70 using 60% sparsity.
Mask selection
Weight update
Perplexity
Wanda
SparseGPT
10.86
Wanda
ADMM
9.96
SparseGPT
SparseGPT
10.51
SparseGPT
ADMM
9.92
and 100 update steps. We also test the performance of SparseGPT weight update and, for reference, include results running SparseGPT with its own gradual mask selection.
We measure perplexity on Wikitext and time overhead (over forward pass) for cach update option. Using just one update step, we can almost beat. SparseGPT and all gradient-based algorithms (Figure 2 The ADMM update almost converges with ten update steps, while the gradient-bases algorithms need steps. ADMM is thus clcarly a faster and superior weight update algorithm compared to the gradient-based update. Our algorithm also provides a better weight update than SparseGPT weight update, and at 60% sparsity, is even better than SparseGPT with its own iterative mask selection.
than 100
Furthermore, we explicitly compare SparseGPT and ADMM weight updates over differret weight masks. We select either Wanda or SparseGPT mask and apply SparseGPT or ADMM weight update (in the case of SparseGPT mask, SparseGPT update is no-op, and for ADMM update, we rewind weights and keep the selected mask). Results are summarized in Table 2 Our ADMM weight update is always better than SparseGPT update. Note that, our mask selection is also better than SparseGPT one (9.22 9.92 perplexity).
4.3 Pruning LLaMA-7B
Based on previous observations, we the number of update iterations 20, which should provide pruming overhead similar to SparseGPT (Table 3) and also guarantee reasonable convergence of weight updates. We compare weight update after mask selection without gradual pruning (ADMM1), our gradual pruning
8

---

Published in Transactions on Machine Learning Research (07/202) /024)
20
18
Perplexity
16
14
12
10
400
600
800
1000
1200
1400
2500
2000
Perplexity
1500
1000
500
400
600
800
1000
1200
1400
Time [s]
ADMM
Adam 0.003
SparseGPT finetune SparseGPT full
Adam 0.001
Adam 0.01
Figure 2: WikiText perplexity vs time overhead for ADMM, Adam, and SparseGPT weight update on LLaMA-7B. We run ADMM and Adam for 1, 10, 20, 50 and 100 update steps and test Adam with various learning rates. The top plot shows 60% sparsity. The bottom one uses 80% sparsity. SparseGPT full refers to normal SParseGPT, which also selects the pruning mask gradually. All other options just update weights over fixed mask selected by Wanda. Our weight update is better than the one in SparscGPT and better than gradient-based methods.
Table 3: Total pruning time for Llama-7B
Method Wanda
Total seconds
245
SparseGPT
850
ADMM1
832
ADMM-Grad
869
algorithm, which computes the mask over 15 iterations (ADMM-Grad) with Wanda and SparseGPT pruning. We prune LLaMA-7b various sparsities and also with 2:4 structured sparsity, First, we measure Wikitext perplexity (Table We that our weight update over fixed Wanda mask (ADMM1) produces better results than any other algorithm at 50%, 60%, and 2:4 sparsities. Note that SparseGPT generates the pruning mask iteratively, which gives it a slight edge in higher sparsities. When selecting the mask gradually, we are superior to all previously developed algorithms, especially at higher sparsities.
Finally, we measure performance on seven zero-shot tasks (we use the same selection as the authors of Wanda): BoolQ (Clark et al., 2019), RTE (Wang al., 2018). HellaSWAG (Zellers al. 2019), WinoGrande (akaauuhi et al. 2021) ARC easy and challenge (Clark et. al. 2018), and OpenbookQA (Mihaylov et al. 2018).
Our results (Table 4 show that our algorithm is superior to the previous ones except for the RTE task. Wo note that results for the RTE task are slightly erratic (e.g. there is betler performance at 60% sparsity than at 50%). We attribute this to the small RTE dataset size (277 samples). Notably, we recover 30-40% of the performance drop of SparseGPT on the BoolQ task at 50-70% sparsities and also on WinoGrande task using 50-60% sparsities. When using 2:4 sparsity, we recover 20- 25% of the performance drop on WinoGrande and ARC-e tasks.

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Table Zero shot accuracies on various tasks during pruning of LLaMA-7B
Sparsity
Method
BoolQ
RTE
HellaSwag
WinoGrande
ARC-e
ARC-c
OBQA
Mean
0 %
Dense
75.05
66.43
56.92
69.93
75.34
41.89
34.40
59.99
Wanda
71.22
55.60
51.85
66.06
69.11
36.86
28.80
54.21
50%
SparseGPT
73.05
52.34
51.21
68.42
70.70
36.43
28.60
54.39
ADMM-Grad
73.63
52.34
52.33
69.13
70.74
37.88
30.20
55.18
Wanda
69.26
59.56
43.76
62.35
62.58
30.29
25.20
50.43
60%
SparseGPT
70.7
62.09
44.84
65.58
64.14
30.97
25.20
51.93
ADMM-Grad
72.41
58.84
46.61
66.77
64.52
31.65
26.20
52.43
Wanda
59.78
58.12
28.81
50.82
32.40
18.85
14.20
37.57
70%
SparseGPT 62.35
55.95
33.77
59.35
45.70
23.97
17.20
42.61
ADMM-Grad
66.05
53.79
36.29
59.74
50.84
25.50
18.60
44.40
Wanda
37.82
48.37
26.29
48.77
27.23
20.56
13.00
31.72
80%
SparseGPT 41.89
52.70
27.83
48.38
30.30
18.77
13.40
33.32
ADMM-Grad
56.14
52.70
28.75
50.74
31.56
18.94
12.40
35.89
Wanda
69.3
51.99
42.06
62.75
60.94
28.07
24.60
48.53
2:4
SparseGPT 70.46
60.65
42.99
64.88
61.49
30.12
23.60
50.60
ADMM-Grad
70.27
35.59
44.88
66.14
64.18
30.97
25.20
51.03
Table 5: Perplexity of pruned LLaMA-2 variants on WikiText
Method
Sparsity
7B
13 B
70B
Dense
0 %
5.12
4.57
3.12
Wanda
50 %
6.42
5.56
3.98
SparseGPT
50 %
6.51
5.63
3.98
ADMM-Grad
50 %
6.33
5.52
3.95
Wanda
60 %
9.71
7.75
4.98
SparseGPT
60 %
9.58
7.80
4.98
ADMM-Grad
60 %
8.70
7.09
4.81
Wanda
80 %
5e3
2e3
1e2
SparseGPT
80 %
108.87
94.23
25.86
ADMM-Grad
80 %
55.93
43.58
18.84
Wanda
2:4
11.02
8.27
5.16
SparseGPT
2:4
10.17
8.32
5.40
ADMM-Grad
2:4
9.74
7.78
5.19
4.4 Pruning LLaMA-2 variants
Our algorithm generalizes and scales to bigger LLMs. We test it on variants of LLaMA-2 at various sparsity
levels. Table 5 shows that our method is superior to previous ones, except at 2:4 sparsity on I.AMA2-70B.. We note quite a substantial improvement of our algorithm over previous ones at 60% sparsity and also at 2:4 sparsity on and 13B models.
5 Related Work
General neural network pruning. Post-training network pruning aims to compress neural networks
by removing some of their parts (weights, neurons, layers) (LeCun et al. 1989 Han al. 2015 Blalock
et al. 2020 Liu et al. 2018p. Pruning criteria vary from simple magnitude pruning (Zhu & Gupta 2018
to sophisticated second-order approximations (Singh & Alistarhi 2020 Nowadays, there is also focus
10

---

Published in Transactions on Machine Learning Research (07/2024)
methods that use limited calibration data and do very little fine-tuning Frrntar & Alistarh 2022 Hubara et al. 2021).
LLM pruning algorithms. Due to sheer LLM size, weight pruning algorithms focused mainly on pruming with limited calibration data and fine-tuning. SparseGPT (Frantar & Alistarh 2023) solves layer-wiss wise pruning problem using multiple approximations. Wanda (Sun et al., 2023) shows that a simple product of
weight magnitude and input activation norm provides competition pruning criterion. DSOT (Zhang al.
2023
provides an iterative mask improvement algorithm.
Another possibility for LLM pruning is structured pruning. One can either remove individual neurons et at...
Ma
2023
Ashkboos et al. 2024), or remove whole layers (Men et al. 2024 Gromov et al. 2024
Target specific distillation and tuning. One can also make neural networks smaller by using knowledge distillation (Hinton et al. 2015). In L.M context, this is usually done with specific task in mind (Hsich)
ct al. 2023 Fu al., 2023 Gu et al., 2023: Ko al.) 2024), where large general model knowledge (logits) is distilled into smaller lask-specific model. This is in contrast with method, which aims to preserve the general ability of the original LLM.
6 Conclusions and Future Work
In this work, we presented simple, fast, and effective post-pruning weight update algorithm based the alternating direction method of multiplicrs. We showed that algorithm converges much faster than any previously available option. Our weight update method is also theoretically sound and does not rely on any heuristical decisions or approximations.
We further improved the pruning performance by doing gradual mask selection and weight updates. This achieves state-of-the-art performance in the layer-wise pruning setting, much better than previous solutions like Wanda or SparseGPT.
Our main limitation is that our update rule runs dense matricos, and thus, during update computation, we have no lime or space savings from potential sparsity. We hope to address this in future work.
Another limitation is that one-shot pruned large models are still inferior to smaller dense ones. The pruning results can certainly be improved by using nonuniform sparsity across layers (Yin et al. 2023); for now, we leave this as a future work. Another option for improvement is use a more accurate mask selection rule, such as one in Optimal brain surgcon (Hassibi al. 1993).
Finally, our algorithm provides efficient update rule for sparse matrices and can be used in some advanced optimizers like FOOF (Benzing 2022
Acknowledgments
This research was supported by grant 1/0538/22 from Slovak research grant agency VEGA.
References
Abhinav Agarwalla, Abhay Gupta, Alexandre Marques, Shubhra Pandit, Michael Goin, Eldar Kurtic, Kevin Leong, Tuan Nguyen, Mahmoud Salem, Dan Alistarh, et al. Enabling high-sparsity foundational Ilama models with efficient pretraining and deployment. arXiv preprint arXiv:2405.032: 2024.
Arash Ahmadian, Saurabh Dash, Hongyu Chen, Bharat Venkitesh, Stephen Gou, Phil Blunsom, Ahmet Üstün, and Sara Hooker. Intriguing properties of quantization at scale. arXiv preprint or arxi:2305 19268, 2023.
Salch Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXio preprint arXiv:2/01 15024. 2024.
11

---

Published in Transactions on Machine Learning Research (07/2024)
Frederik Benzing. Gradient descent neurons and its link to approximate second-order optimization. In International Conference on Machine Learning, pp. 1817-1853. PMTR, 2022.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? Proceedings of machine learning and systems, 2:129-146 2020.
Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trendso8 in Machine learning, 3(1):1-122, 2011.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1904 2020.
Christopher Clark, Kenton Lee, Ming-Wa Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arxiv preprint arXiv:1905.1004 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schocnick, and Oyvind Tafford. Think you have solved question answering? try arc, the ai2 reasoning challenge. orXiv preprint arXiv:1803 1803000573 2018.
Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. In Interna- tional Conference on Machine Learning, pp- 7750-7774. PMLR, 2023.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. aixie preprint arXiv:2208.0733 07339, 2022.
Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quan- tization and pruning. Advances in Newral Information Processing Systems, 35:4475-4488, 2022.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp.
PMLR, 2023.
Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. Specializing smaller language models towards multi-step reasoning. International Conference on Machine Leurning, pp. 10421- 10430. PMLR, 2023.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jef- frey Hsu, Kyle McDonell, Niklas Mucnnighoff, et al. A framework for fow-shot language model evaluation. Version vO. 0. Sept, 2021.
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. The unrea- sonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403.17888 2024.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference Learning Representations, 2023.
Shupeng Gui, Haotao Wang, Haichuan Yang, Chen Yu, Zhangyang Wang, and Ji Liu. Model compression with adversarial robustness: A unified optimization framework. Advances Neural Information Processing Systems, 32, 2019.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruming. In IEEE international conference on neural networks, pp. 293-299. IEEE, 1993.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge a neural network. arXiv preprint 1oX3.2500302250 2015.
12

---

Published in Transactions on Machine Learning Research (07/2024)
Cheng- Yu Hsich, Chun-Liang Li, Chih-Kuan Ych, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ran- Jay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.02300 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv prepriut arXiv:2106. 096685 2021.
Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Joseph Naor, and Daniel Soudry. Accelerated sparse neural training: A provable and efficient method to find m transposable masks. Advances in neural information processing systems, 34:21099 21111, 2021.
Jongwoo Ko, Sungayun Kim, Tianyi Chen, and Se- Young Yun. Distillm: Towards streamlined distillation for large language models. Xrxiv preprint orx: 03883888 2024.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint DrXfc:1810.0522 2018.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702 21720, 2023.
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojic Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. arXio preprinl WrXiv:2403.03852 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint Xic::1660900 07843, 2016.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arxiv: 1809.800000 000900279, 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2155:55455-5511 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99 106, 2021.
Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. Advances in Neural Information Processing Systems, 33:18098 18109, 2020.
Mingjic Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruming approach for large language models. arXiv preprint Xrx::22306. 1 11695, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Amn Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. orXiv preprint urxiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Ashish Vaswani, Noam Shazoer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
13

---

Published in Transactions on Machine Learning Research (07/2024)
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprinl arXiv:1804.074611 2018.
Haojun Xia, Zhen Zheng, Yuchao Li, Donglin Zhuang, Zhongzhu Zhou, Xiafei Qiu, Yong Li, Wei Lin, and Shuaiwen Leon Song. Flash-llm: Enabling cost-cffective and highly-efficient large generative model inference with unstructured spansity. arXiv preprinl ar Xiv:2300. 10285, 2023.
Guangxuan Xiao, Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In Internalional Conference Machine Learning, pp. 38087-38099. PMLR, 2023.
Shaokai Ye, Kaidi Xu, Siji Liu, Hao Cheng, Jan- Henrik Lambrechts, Huan Zhang, Aojun Zhou, Kaisheng Ma, Yanzhi Wang, and Xue Lin. Adversarial robustness vs. model compression, or both? In Proceedings of the IEEE/CVE International Conference on Compuler Vision, pp. 111 120, 2019.
Lu Yin, You Wu, Zhenyu Zhang, Cheng-Ye Yu Hsieh, Yaqing Wang, Yiling Jia, Mykola Pechenizkiy, Yi Liang, Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning lhms high sparsity. arXiv preprint arXiv:2310.05115 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint Xiv: 070.000000 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXivv preprint Xiv:22200 01068, 2022.
Tianyun Zhang, Shaokai Ye, Kaiqi Zhang, Jian Tang, Wujie Wen, Makan Fardad, and Yanzhi Wang. A systematic dnn weight pruning framework using alternating direction method of multipliers. In Proceedings of the European conference computer vision (ECCV), pp. 184 199, 2018.
Yuxin Zhang, Lirui Zhao, Mingbao Lin, Yunyun Sun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, and Rongrong Ji. Dynamic sparse no training: Training-free finc-tuning for sparse Ilms. arXiv preprint arXiv:2310. 089115 2023.
Michael Zhu and Suyog Gupta. To prune, or not to prune: Exploring the efficacy of pruning for model compression. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Workshop Track Proceedings. OpenReview.net, 2018. URL https:
14


Reviews:
[{"review_id": "Sqk7j6C1q5", "reviewer": "Reviewer_tt5c", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The proposed algorithm works very well, outperforming state-of-art pruning algorithms like Wanda and SparseGPT in producing models with better text perplexity and zero-shot prediction accuracies at different levels of sparsity. \n\n- In terms of speed the proposed method is also fast, on par with the layerwise optimization approach SparseGPT but not as fast as Wanda. \n\n- One potential weakness of the paper is its novelty. The paper largely follows the optimization framework of SparseGPT, replacing the inner optimizer with ADMM. The mask selection heuristic largely follows from the Wanda algorithm. There are relatively few new technical contributions, even as the method is very effective.  \n\n- The title of the paper says fast and optimal weight updates, but in what sense are the weight updates \"optimal\"? I don't see any explanation of this in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose the use of Alternating Direction Method of Multipliers(ADMM) for layerwise pruning of large language models(LLM). They show that their method is more effective than previous state-of-art approaches like sparseGPT and Wanda in pruning, giving models with better text perplexity and zero-shot accuracies on various tasks at different levels of sparsity.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Explanation or modification of the term \"optimal\" in the title of the paper.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PGWCGPkYaK", "reviewer": "Reviewer_qDYv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThis paper shows how to adopt ADMM for weight pruning of LLM and provide experimental results showing its effectiveness compared to other approaches. It is a theoretically more sound approach compared to other iterative methods such as gradient-based methods. It also practically works very well -- faster and more accurate.\n\nIt is tested on several benchmarks and it supports their claim.\n\nA simple extension to ADMM (gradual pruning) improves the naive application of ADMM significantly. It is simple and effective.\n\n\nWeaknesses\n\nAlthough it is clearly better than existing methods evaluated in the paper, but the gap to the original dense model seems to be large. It is not clear if the number is good enough and the resulted model is useful for some real applications.\n\nAll of the details of experiments are not explained in the paper. For example, how to obtain the calibration input for each weight is not very clear. The architecture of LLaMA is not explained at all. Readers could go the original paper to understand it, but I think it is an important detail to be explained in the paper, too.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper adopted ADMM for weight pruning of LLM. ADMM has advantages over existing methods both in terms of the speed and accuracy. For example, ADMM has the same complexity as gradient descent-based methods for one step, but it converges much faster in practice. The optimization problem is cast as an ADMM problem and the update equation and algorithm are presented. The experimental results show the superior performance to other existing weight pruning methods in a similar setting.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It would be good to mention if the performance degradation by the pruning method is acceptable for downstream applications.\n\nIt would be good to have discussions on the comparison with distillation/SFT. There are methods to distill and finetune LLM and it could potentially provide better model with a smaller size. It is probably the case that the proposed method is best-in-class in the particular setting, but it would also be important to compare it with methods that are possible with a milder condition.\n\nIt would be good to have more detailed explanations on baseline methods to further contrast the differences between methods and to make sure that the comparisons are apples-to-apples and it would also be good to have more detailed explanations on experimental conditions including points mentioned in the weaknesses section above.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FivY4Y6qYx", "reviewer": "Reviewer_FKAh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The evaluations provided of the method are comprehensive, done on several\n  model sizes (7B, 13B, 70B), and show that the method is effective at\n  performing weight updates after pruning LLMs, while maintaining decent performance.\n- The method outperforms similar methods in terms of performance (as measured by perplexity)\n  on evaluation datasets, and seems to generalize well as model size increases.\n- The method is intuitive and well described, including an algorithm in\n  pseudo-code presented in the main body.\n\n## Weaknesses\n\n- The paper claims that fine-tuning an LLM after pruning is not feasible due to the high computational cost.\n  However, the paper does not provide any empirical evidence to support this claim, nor cite any work that has shown this to be the case.\n  Contrary to this claim, there is work that shows how to fine-tune LLMs efficiently, e.g., [1, 2, 3].\n- The work continues to use the term \"optimal\" to describe their post-pruning method.\n  There is no evidence provided that the method is optimal. Optimal in this setting may mean that it is the best possible method.\n- The paper could be better structured and be more concise.\n  For instance, providing the tables after the conclusion makes it harder to follow the results.\n  Additionally, I found the conclusion section to be a bit verbose and could be shortened.\n  The paper also does not provide a related works section, however they do\n  dive into related works in the introduction and throughout the paper.\n  A related works section would help to summarize the related works in one place.\n\n### References\n\n- [1] \"LIMA: Less Is More for Alignment\": https://arxiv.org/abs/2305.11206\n- [2] \"LoRA: Low-Rank Adaptation of Large Language Models\": https://arxiv.org/abs/2106.09685\n- [3] \"LoRAPrune: Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning\": https://arxiv.org/abs/2305.18403", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides an algorithm to perform weight updates post-pruning of a\nlarge language model, in order to maintain performance while reducing the\neffective model size. This method does not require fine-tuning or\ncontinual training, and is evaluated on the Llama model at three different\nsizes (7B, 13B, 70B) and compared to other methods. The\nresults show that the method is effective at maintaining performance while\nreducing the model size, and outperforms other methods in terms of\nperformance on evaluation datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please either reduce the claim that fine-tuning an LLM after pruning is not feasible or provide empirical evidence to support this claim.\n  - There are several other reasons for not fine-tuning after pruning, for instance, catastrophic forgetting after\n    fine-tuning [4]. It would be beneficial to discuss these reasons in the paper.\n- Please refrain from using the term \"optimal\" to describe the method unless there is evidence to support this claim.\n- Some of the results suggest that this method (Table 5), and other post-pruning weight update methods,\n  lead to inferior performance compared to utilizing a smaller unpruned model.\n  It would be beneficial to discuss this in the paper, and why pruning is still a useful technique.\n- While the supplemental material provides all the details needed to reproduce the results, it would be beneficial to include\n  some of the details in the main body of the paper, such as the hyperparameters used for inference\n  in the experiments.\n\n### References\n\n- [4] \"Understanding Catastrophic Forgetting in Language Models via Implicit Inference\": https://arxiv.org/abs/2309.10105\n\n### Nitpicks\n\n- \"showed that LLMs can pruned by removing weights\" -> \"showed that LLMs can be pruned by removing weights\"\n- \"We prune LLM during one forward pass\" -> \"We prune the LLM during one forward pass\"\n- \"state-of-the-art performance in layer-wise pruning setting\" -> \"state-of-the-art performance in the layer-wise pruning setting\"", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree that the accuracy and efficiency claims are supported with strong empirical evidence. The authors already made some changes to the paper based on the feedback from the reviewers (e.g., better related work, changing some terms around optimality). Below I summarize some additional updates that are important to implement for the final version of the paper to improve clarity and readability. 

As the sections 2.3 and 3 are written right now, it is difficult for the reader to separate precise mathematical statements and informal interpretations and discussion:

 - Section 2.3 quotes results that are not directly used: only the key results that you rely on from Boyd et al. should be quoted. For example, there is no point in introducing function g(x) since it just maps to zero. I would suggest having a proposition statement in a form that you use (you are welcome to include the full theorem in the appendix). It is also crucial to state the exact conditions under which the algorithm converges to the optimum. 

 - The application of the result by Boyd et al to pruning, as in Section 3, should be stated as a precise corollary (with a proof). Then the discussion can follow. 


As mentioned by other reviewers, it would be great in Section 3 to explicitly contrast your proposed method to SparseGPT and Wanda (explaining how similar it is to SparseGPT and which component is being replaced).

Regarding the optimality and accuracy statements:  from what I can tell, there is no “accuracy” guarantee (no theorem saying that within k updates, one will be epsilon far from the most “accurate” solution, where the latter implicitly implies connections to generalization). Thus I do not think that replacing “optimal” with “accurate” is any better. Going back to optimality, even ignoring convergence, the optimal solution might only be reached for the objective based on the subsample of the training data, and also only for the convex per-layer subproblem which is not the same as optimality of pruning and updating the entire network. So I do agree with the reviewers that saying that the proposed algorithm is optimal in any sense is not accurate.

Some additional minor comments: (1) the way that the abstract is currently written makes it sound like you are proposing a better fine-tuning technique only (that should be applied to an already-sparse model). (2) In the introduction, under “Our results”, you say “Our algorithm sidesteps all of the problems of previous solutions.”. Please list these problems explicitly, as they are entangled in the previous paragraph and hard to separate.
Decision: Accept (poster)