The LLM Surgeon
Authors: Tycho F. A. van der Ouderaa
Markus Nagel
Mart Van Baalen
Tijmen Blankevoort
Venue: ICLR 2024 poster (2024)
Keywords: large language model
LLM
compression
pruning
fisher
Primary area: generative models

Abstract: State-of-the-art language models are becoming increasingly large in an effort to achieve the highest performance on large corpora of available textual data. However, the sheer size of the Transformer architectures makes it difficult to deploy models within computational, environmental or device-specific constraints. We explore data-driven compression of existing pretrained models as an alternative to training smaller models from scratch. To do so, we scale Kronecker-factored curvature approximations of the target loss landscape to large language models. In doing so, we can compute both the dynamic allocation of structures that can be removed as well as updates of remaining weights that account for the removal. We provide a general framework for unstructured, semi-structured and structured pruning and improve upon weight updates to capture more correlations between weights, while remaining computationally efficient. Experimentally, our method can prune rows and columns from a range of OPT models and Llamav2-7B by 20\%-30\%, with a negligible loss in performance, and achieve state-of-the-art results in unstructured and semi-structured pruning of large language models. We will open source our code on GitHub upon acceptance.

THE LLM SURGEON
Tycho F.A. der Ouderaa' 1+ Markus Nagel', Mart van Baalen",
Yuki M. Asano,, Tijmen Blankevoort IImperial College London, "Qualcomm Al Research!, 'QUVA Lab, University of Amsterdam
ABSTRACT
State-of-the-art language models are becoming increasingly large in an effort to achieve the highest performance on large corpora of available textual data. How- ever, the sheer size of the Transformer architectures makes it difficult to deploy models within computational, environmental or device-specific constraints. We explore data-driven compression of existing pretrained models as an alternative to training smaller models from scratch. To do so, we scale Kronecker-factoree curvature approximations of the target loss landscape to large language models. In doing so, we can compute both the dynamic allocation structures that can be removed as well as updates of remaining weights that account for the removal. We provide general framework for unstructured, semi uumi-structured and structured pruning and improve upon weight updates capture more correlations between weights, while remaining computationally efficient. Experimentally. our method can prune rows and columns from a range of OPT models and Llamav2-7B by 20%-30%, with a negligible loss in performance, and achieve statc-of-the-a- re- sults in unstructured and semi-structured pruning of large language models. Code is available at: https://github.com http:/gthhbbccomuuuuloooomo Al-research/llmmsurrre
INTRODUCTION
Recent advancements in language modeling (Vaswani et al. 2017) allow fitting large language mod- els (LLMs) with millions even billions of parameters (such as OPT (Zhang al., 2022) and Llama 2 (Touvron et al.] 2023)) on big text corpora achieving high performance. Unfortunately, the size of these LLMs often makes it hard to deploy them within practical constraints. Cloud- based deployment can get very expensive for larger models, and efficient devices such as phones are frequently limited in the memory size to host a model.
A body of literature extending back to the late 1980s, c.g., Optimal Brain Damage (OBD, LeCun ott al. (1989)) and Optimal Brain Surgeon (OBS, Hassibi Stork (1992), phrases pruning as constraint optimization problem to reduce a model's footprint and runtime requirements. The Hes- sian required for this approach grows with the square of the number of parameters, and can only be computed in practice for unrealistically small networks. To overcome this issue, Eigendamage blockwise-diagonal approximation of
(Wang et al. 2019) introduces a Kronecker factorization the Hessian. Recent works, like Optimal Brain Compression (Frantar & Alistarh 2022), SparseGPT (Frantar & Alistarh, 2023), demonstrate practical post-training pruning of LLMs, but only consider loss curvature of a pruned layer's ss squared output reconstruction error, ignoring gradients that relate local removal costs to the target loss. As a result, their approximation to the target loss landscape inaccurate, leading to significant performance degradation for pruned L.LMs. Further, these methods do not readily extend to structured pruning.
"Work done while doing an internship Qualcomm Al Research "Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.
Structured compression (rows and columns)
Unstructured compression (matrix elements)
pretrained
pretrained SparseGPT LLM Surgean
test set perplexily
test set perplexity
K- OBD
Surgeon
14.62
14.62
12.47
12.47
10.86
10.86
8-o---
model sizes:
1.3b
2.7b
6.7b
model sizes: 1.3b
2.7b
6.7b
Figure LLM Surgeon allows interpolation of model size between existing pretrained models.

---

This work introduces LLM Surgeon, general framework for unstructured, semi-structured and structured pruning of LLMs. At paper submission, we deemed this the first method to successfully perform structured pruning of LLMs. Concurrent work by AAhkboos et al. (2024) also considers structured pruning of LLMs but ignores gradient information, resulting in lower final performance. The superior performance of LLM Surgeon is achieved by scaling up the block-diagonal Kronecker- factorized approximations to the empirical Fisher from Eigendamage to LLMs. We further expand upon the work by deriving OBS-like weight pruning costs and updates for structured pruning of multiple rows and columns, and provide general framework that also incorporates semi-structured and unstructured pruning. Instead of treating individual weight updates independently, we strive to consider many correlations between weights as practically possible and derive joint weight updates for pruning multiple weights (or multiple sets of structured weights) at once. Unlike prior work in LLM pruning, LLM Surgeon prunes in multiple shots, updating weights and curvature es- timates between shots. We use global thresholding for unstructured, semi-structured and structured, i.., instead of pruning layers by a fixed amount, more sensitive layers are pruned less than those that are more robust. Lastly, we propose to mitigate possible first-order gradients not being zero by using optional low-rank first-order updates between shots. A key advantage of LLM Surgeon is that it allows trading off additional compute during compression for better accuracy by increasing the number of correlations and/or shots. Our method gives the first practically usable results for struc- tured pruning of LLMs they can be pruned by up to 30% with minor performance degradation. Furthermore, we achieve state-of-the the results unstructured and semi-structured LLM pruning.
BACKGROUND AND RELATED WORK
Neural network pruning aims to remove parameters from model while minimizing negative im- pact on final performance. More formally, we denote the P model parameters as vector 0' = vec (Wr: W ..... Wi) E IR., by flattening the L weight matrices of attention and fully-connected blocks, with already fitted 00 arg ming L(0) to data 'D to minimise a negative likelihood loss [(0)= log ()) To compress the model, we are looking for a pruned vector θ:
= arg ming (0) s.t. pruning constraints based on 0
where chosen constraints determine the structure of compressed weights Ó. In unstructured prun- ing, a fraction of total weight elements is set to zero. In semi-structured pruning of M:N we have that M weights of every N consecutive weights are zero (Zhou et al. 2021 Hubara et al. [021]. And in structured pruning (Louizos et al., 2017). entire rows and columns are set to zero Struc- tured pruning leads to the most immediate gains in memory and computing, as it directly reduces the dimensions of matrices that need to be represented explicitly but is regarded as the most difficult to compress. Maintaining high performance is often easier in the other schemes but requires spe- cialised arithmetic exploiting the sparsity structure to benefit at deployment. We consider all pruning types above, with focus on structured pruning for LLMs.
Typically, eq. (1) can not be solved directly, as the space of possible pruning configurations exceeds what can be evaluated in practice. To illustrate, search over all possible unstructured pruning masks of a 125 million parameter LLM would require 2' 2215mm 102 1037628749 evaluations. The idea, therefore, is to find Ô using a surrogate of the loss landscape that is easier to work with:
[()) = - log p( (D 0) ~ - log (0)
(2)
If one chooses particular Gaussian form for our surrogate q, then solutions for unstructured, semi- structured, and structured pruning constraints can be derived in closed-form (appendix A
2.1 TAYLOR EXPANSION
How do we obtain good surrogate of the loss q? One of the easiest approaches is to locally expand the log loss through second-order Taylor expansion around the pretrained weights 0", yielding:
1 (0 0-) Ho- o*)
log (0)) log -logp(D(e)) (e- V'(O'
(3)
where [V((0')] = a (())) denotes the Jacobian and [Halıs
8² C(O,)) denotes the Hessian. J0,0,
The first-order term vanishes [VL(0))) 0 at the optimum. Note that in practice the first order
2

---

Loss decape view (Geometricc perspective)
Liellnood densty view (Pronabil perspective)
log 9)
log of
(bpp(()
p())
P(O)
Aorb
p(o)
8ran
x001
Bun
Boes
6
Second order Taylor approximat
Constraired Optimization
Laplace approsmmation
Forward- Backwardk-
Gauseian KI ryenoe
Figure 2: Pruning equality constrained optimization of quadratic approximation of the loss land- scape (left), or equivalently. maximising the likelihood under Laplace approximation (right).
term may not vanish. While we follow this assumption initially, we consider interleaved first-order corrections to mitigate the issue in section 3.6 The quadratic expansion of eq. (3) forms the basis of the optimal brain damage (LeCun et al. 1989) and optimal brain surgeon (Hassibi & Stork 1992) pruning methods. Note that from probabilistic perspective, quadratic approximation of the log likelihood implies a Gaussian approximation of the likelihood, as also observed by <Wang et al. 2019) and illustrated in fig. 2 This well-known (Bishop Nasrabadii (2066), (MacKay) 2003) as the Laplace approximation (()) N 0- VZ("", Hg-"), with pretrained weights are the mean and the local inverse Hcssian is the covariance matrix capturing correlations between weights.
2.2 BLOCK FISHER INFORMATION MATRIX
For network trained with negative log-likehood loss, the Hessian is identical to the Fisher matrix:
H Fg Σ,
E Ey>po(vlan (vlm..)
V log (,) V log po (v(
(4)
which has the benefit of always being positive semi-definite. with the inverse thus forming proper covariance matrix for 9, and can be approximated with Monte Carlo samples of most I.LMs, this would be treating the softmax output of the network categorical distribution po))), and sampling from that. In practice, we use the 'empirical Fisher" replacing the exper- tation over y with target data yn. (Kunstner al. 2019). The full (empirical) Fisher Fe R scales quadratically in the number of parameters P. To overcome this, the Fisher is often written in terms of layer-wise blocks Fix- = Σn E |ve( V w; log po (yl vec]) w, log pa (y) x ) approximated by only treating layers independently (Martens Grosse 2015:
For
and
Botev al. 2017):
Fe diag(F F1,, F22.... ................................ FLL).
F
E (gl.ni @ (ai,n af
(5)
==1
RC
where @ denotes the Kronecker product and vec(-) the matrix vectorisation operation. Because we disregard cross-layer interactions we write F; instead of Fú for Fisher blocks associated with the weight matrix W.ERRXC producing outputs ft,, = Wiai,, ER from inputs at for each layer and datapoint n. Consequently, we can compute Fisher blocks from input activations ar, ERC of forward-passed data v, and output gradients - VV. CER from backpropagation.
2.3 PRUNING AS CONSTRAINED OPTIMIZATION
Optimal brain surgery relies on removing and adapting weights such that the loss least negatively affected, thus it behooves us to write the problem constrained optimization problem. From the Gaussian approximation discussed in section 2.1 obtained by quadratically expanding the log likelihood loss log pT } TT F0, the optimal update 2 (and thus also 0-0+A0) becomes the following equality constrained quadratic optimization problem (Hassibi & Stork 1992.-
1 - - FA0
arg min 20
6
2
A0
c
0 0, VK K.
where F is positive semi-definite and K is the of K indices that are pruned (i.e., zero).
3

---

Algorithm LLM Surgeon (structured)
Input:
initial weights 0 target size a, and data For shot / in [I. .....
Compute: approximate curvature G, A from data D Compute: costs per row/column L,, Le from G. A
Di section
3.
section
Compute: threshold 7 using L, and Le given target size at
>> section 3.3
Select: rows and columns to remove En. Eg based on
section
3.3
Compute: weight update A based on En. Ec and G, A remaining weights / <- f + A00-1
D section
3.4
Update:
section
3.5
0f < low-rank ) Output: compressed weights 0 0
Optionally:
section
3.6
General solution We denote EK (ei e2 ex| E [0, [). as matrix of which the
row vectors are canonical basis vectors E R² that select the elements to be pruned. One of the most standard approaches to solve eq. (6) using Langrange multipliers, which results in general closed-fomm solution for the expected increase in loss L and optimal weight update A0:
1 2
1 [ExOT )z EEKF-LE E
C
A F E [E 0
(8)
which we use to derive unstructured, stui-structured, structured for modern Fisher approximations (see appendices/A A.2 A.4). The same general form of eqs. (7) and (8) appears in prior LLM pruning
work Kurtic et al. (2022). but only for much simpler layer-wise pruning and no structured pruning.
3 LLM SURGEON
This section describes the components of our method, LLM Surgeon, summarised in algorithm
3.1 ESTIMATING LOSS LANDSCAPE CURVATURE
Accurate pruning relies on approximating the local curvature accurately while overcoming the mem- ory cost associated with storing the true curvature. Specifically, even with the block-wise approxima- tion of eq. (5) F E IR RCx RC requires summing N large RC x RC matrices, too large to practically fit in memory. Instead, we adapt the KFAC approximation (Martens & Grosse 2015) that assumes independence of activations and derivatives, approximating an expectation of Kronecker products as a Kronecker product of two expectations Elg....... ( a1,n (f.g.n.. s Elai,na....)
allowing layer-wise Fisher blocks to be approximated Fi Fj. where
N 1 1 E n= g. and A Σ VN n=1 at, ai
F = Gi A with G: =
VN
constructed from activations ai,n E C from forward passes and gradients 9t.n R. from back- ward passes (Eschenbagen et al. 2024). The approximation originates from optimization literature, but has recently gained popularity for other problems that require curvature approximations (Immer ct al. 2022 van der Ouderaa al. 2023), including structured pruning in Wang al. (2019).
An additional advantage of approximating Fisher blocks as Kronecker products that the inverse becomes particularly easy to compute F-1 G 69 A-1, thus only requires inverting the factors. This fact allows us never explicitly construct large RCx RC matrices in memory that make up F and F but rather directly work with the much smaller matrices and A.
3.2 COMPUTING COSTS IN FINAL LOSS
The number of possible combinations in which weights can be removed grows (supra-)exponenttaaly in parameter count, making it infeasible to estimate a separate cost L for each such removal. A common strategy, therefore, is to treat weights independently when computing removal costs L. We also follow this strategy, but note that this does not necessarily imply that we have to make such same strong independence assumption for the weight updates A0 after selecting weights be removed.
4

---

Unlike most prior work, we present correlated weight updates by taking into account off ofggdiaa elements of the Fisher approximation section 3.4
For semi-structured and unstructured ke[1, RCJ, and for structured use independent costs for all rows rE[1, R] and columns c C]. We find that can derive the appropriate costs from the general cost formula eq. 7 by letting E=c, E RRCC where the single one-hot element at index of canonical basis vector e₁ selects the weight to remove. For structured pruning, we similarly select rows and columns c, by setting
independent costs for individual weight elements
E E- @ I CRRC or 1-,,,,EE RxRC with e, " C,ER". Plugging into eq. 77. we find:
1 (0x)² Lx 2 G A
C, 1 A0,
L. 1 GO. [o-
(10)
2 IG-
Full derivations can be found in appendices A.2 and A.3 The costs for single elements Cs are equivalent to those found in optimal brain surgeon (Hassibi & Stork| (992) and L. and C. closely resemble structured brain surgeon of (Wang et al., 2019). but in our case derived for matrix rows and columns (see appendix A.3)). Given curvature estimates, costs for either removing all weights or all rows and columns can be computed in parallel. In addition, we derive costs for the more
general of Kronecker factor approximation F == G1 & A1 G2 @ A2 in appendix I through an eigendecomposition.
3.3 DYNAMIC WEIGHT ALLOCATION WITH GLOBAL THRESHOLD
Unlike prior works that compress layer-by- layer (Frantar & Alistarh, 2023). we use a global threshold enabling dynamic alloca- tion of sparsity levels across layers, pruning most where hurts the least. Our method can compress a model to specifically chosen target size a, defined as the fraction of weights that should remain, i.e. stay non-zero after com- pression. In all structured, semi-structured, and unstructured pruning (fig. we select as many weights for removal so that the target size a is reached that inflict the least possible costs C, as com- puted according to section 3.22 For unstructured pruning, this is as simple as sorting the costs for all weights L1 in the network and setting a global threshold 7 such that a fraction of weights fall within the threshold L; S For M:N semi-structured pruning, we sort the M costs of each N consecutive weights and select the M weights with lowest cost. In case of a multi shot schedule (see section 3.5) also sum the M lowest costs in each block find a cost per block, sort costs per block across the entire network, and similar to the unstructured case set a global threshold 7 such that an fraction of weights fall within threshold. Lastly for structured pruning, we perform sorting appropriately weighted by the number of elements that make up row or column and set the global threshold such that a fraction of all weights fall within the threshold. Then we remove rows and columns that fall within the threshold Cr,, Le
structured
semi-structured
unstructured
original
W
cost &.
costs E
element costs
weights
Updated remaining weights
Figure 3: General framework for structured, semi-
structured and unstructured compression.
3.4 CORRELATED WEIGHT UPDATES
Like most other pruning methods, we prune multiple weights at once (Frantar Alistarh 2023 Wang et al. 2019. To arrive at pruning costs and weight updates for pruning multiple weights, is common to compute costs and updates for individual weights (or sets of weights) independently and add them together to arrive to consider weight updates jointly instead of independently. After selecting the set of weights for pruning, we can often afford compute single correlated weight update associated to the joint removal of multiple weights, instead of naively summing weight updates associated to individual removals. We derive such correlated weight updates below. Note that, for the expected cost compu- tation, we do assume that the row, column or weight costs are independent, as the number of possible combinations of weights to prune grows too large compute within reasonable time.
joint pruning cost. In LLM Surgeon,
argue that it's better
Fast unstructured / semi-structured correlated weight updates Mathematically, we represent pruned weights Ex [e1 e2 ... exd KxRRS where e, CR are one-hot canonical ba-
5

---

sis vectors selecting the weights for removal. As each element & has unique associated row and column e index, we can co also use canonical basis vectors for these respective rows ERER KxR and columns E (i.e., we have RE @ [c],=[Ex]] is satisfied for all i).
We derive unstructured weight updates in appendix A.2, by considering cigendecompositions K KT A K2 of the Fisher approximation F = G @ A, which from eq. (8 yields:
AW = G K
K
W
K2 3
K2) K2) A-
(11)
Kxk
where @ is element-wise division, and for brevity use bar notation Kj=Ek K, K₂ Eg K2 0 and S S=dia((S))diag(S)) EER RxC and diag(-) vectorises matrix diagonals.
Programmatically, we always avoid explicitly representing large matrices F and F-1 in memory, but rather compute relevant quantities from their factors. Likewise, we never represent sparse ma- trices Ex. En or Ec in memory, but instead work with a lists of indices of the one-hot elements directly. For example, we can cheaply construct K = E E RKXR and K2=EcKK E RXxC, by copying row vectors, and the vector 0- 0-0--Eeppp E E RRK by indexing all pruned weights.
Maximum number of correlated weights The main computational bottleneck the x ma- trix inverse in eq. (T.. To control compression speed, we split pruned weights into disjoint such that each subset K; does not exceed the set maximum number of correlated weights K,Sm, and sum associated independent updates. Using less correlation by set- ting a lower m allows trading compression quality for speed.
subsets K-KUUKGU...
Fast structured correlated weight updates Unlike the general case which requires inverting K K matrix for K correlated weights, we find that weight updates with the Kronecker factored Fisher approximation F = G @ A only require inverting Ht' x R'' matrix when removing Hi' rows or C' x C" matrix when removing C' columns. The updates are much cheaper than we would have expected based on the effective number of weights in those rows and columns, which would imply inverting R'C x R'C or RC' RC" matrices. In practice, this leads to a significant speed-up for structured pruning and weight updates that take into account correlations between rows or columns. When removing R' rows, r2,,2,.... TR', or the C' columns C1 C,,..... CC',
CC', with 1<R < and <C""<<, we denote one-hot vectors selecting all rows and columns to be removed respectively ) E R and Ec: [ei e2
as Es-- = [ei e2
RC''C, We find weight updates associated to removing the R' rows by setting Ex = ER & or Ex = I 2 E
ec
remove multiple R' rows:
AW = W(EE A A ( (A ET)
(12)
remove multiple C" columns:
AW G G W
From here, it is clear that the special case of removing single row or column under Kronecker approximation involves inverting matrix, and thus only requires scalar division:
G- e, & Or or single column e: AO 0. X A
remove single row A0
(13)
G-
in accordance to independent structured updates in Wang et al. 2019). for convolutional filters. We have thus extended existing structured weight updates to rows and columns, and derived update rules that also consider correlation between structured groups (in our case the rows and columns).
3.5 MULTI SHOT PRUNING SCHEDULE
ratio, we propose pruning in multiple shots. We theoreti- cally justify this multi-shot approach by noting that the surrogate loss landscape relies on Taylor expansion (eq. 33) that only holds locally and thus becomes unreliable for larger jumps AO in pa- rameter space. We mitigate this by pruning in multiple 7>1 shots, & E 2,, T",, each resulting in a smaller weight update A after which the curvature of the loss surface can be re-estimated. When pruming to target size a, ie. removing 1-a of total weights, at ag = 1 and ends with =a, such that after shots, exactly a fraction of the total weight remain. Empirically, we find that linear schedule for (4, formulated in section monotonically improves
To improve the performance-to-spaae
choose schedule a starting

---

Table 1: Structured compression of large language models wikitext-2 data.
Test performance (PPL)
Method
Target size
OPT (125m)
OPT (1.3b)
OPT (2.7b)
OPT (6.7b)
Llama-v2 (7b)
Baseline
100%
27.65
14.62
12.47
10.86
5.12
Magnitude
90%
767.2
894.4
1229
3464
36746
80%
(1278)
2788
16747
347960
70%
17970
(3098)
9255
17312
41373
L-OBD
90%
20.76
17.69
27.20
14259
diag(I & A)
80%
94.14
1392
3236
15630
multi
70%
545.6
2147
7233
7628
21386
K-OBD
90%
27.97
14.68
11.96
10.53
5.48
diag( G & A)
80%
29.89
15.63
12.47
11.28
9.14
multi shot
70%
36.54
18.29
14.53
13.03
15.43
60%
47.54
24.65
18.09
16.21
28.03
50%
75.95
37,68
26.68
25.54
46.64
LLM Surgeon (ours)
90%
28.29
14.73
12.00
10.82
5.43
G & A
80%
29.37
15.27
12.37
11.22
7.29
within row/col cor.
70%
32.46
16.60
13.16
11.83
10.85
60%
39.82
19.40
14.79
12.94
16.67
50%
51.48
23,81
18.01
15,38
25.62
LLM Surgeon (ours)
90%
28.01
14.70
12.02
10.77
5,25
G A
80%
28.73
15.12
12.27
11.02
6.18
full cor. A
70%
31.82
16.24
12.92
11.64
7.83
60%
38.47
18.45
14.23
12.58
10.39
50%
49.78
22.95
17.15
14.90
15.38
pruning performance with more shots, and that higher sparsity levels typically require more shots (see appendix F.I. Multi-shot pruning allows one to spend (linearly in ?") more computation to improve the final compression performance.
3.6 INTERLEAVED LOW-RANK FIRST-ORDER CORRECTIONS
We propose optional interleaved low-rank first-order corrections to further improve compression performance. So far, we assumed parameters are in local optimum when finding a closed-form solution to the quadratic constraint problem. In practice, however, this assumption likely does not hold since (i) the neural network may not be optimised the minimum, (ii) a different loss may (iii) we prune in multiple shots (section 3.5)
be used for compression than used for training, inevitably causing weights to diverge from the optimum. To mitigate this, we consider first-order corrections by interleaving pruning shots with low-rank adaptations of weights Wi+UV (LoRA, by (Hu et al. 202T)), commonly used LLM finetuning. We always absorb updates after each shot, so that the next loss estimate is closer to the optimum and underlying assumptions likely hold more closely. By absorbing LoRA updates between shots, the sum of low-rank updates can have higher rank than individual updates. That is, have rank(U" (V)) for the updates U'V' at any shot f, with equality only arising if updates lie exactly in the subspace which unlikely regular LoRA finetuning and may therefore be useful outside the context of model compression to allow more expressive low-rank model adaptation, negligible cost.
+ +UTTT >
occur in practice. This insight could also be used during
RESULTS
We compare compression performance of LLM Surgeon on language modeling tasks on OPT (Zhang et al. 2022) and Llama-v2 (Touvron et al. 2023) model families, using data from wikitext-2 dataset (appendix B.2).. For compression, we use 128 sequences with a sequence length of 2048 tokens from the training data set and evaluate test perplexity (PPL) on the standard test split. In our experiments, we use a linear sparsity schedule a;=1-t(
at each shot s before reaching the final sparsity We use 40 shots a=0.5 sparsity and report intermediate compression rates, effectively using T-8 shots for a=0.9, T-16 for a=0.8, T-24 for a=0.7,
7, and T=32 for a00.6. We com- pare against magnitude pruning, L-OBD, GparseGPT and K-OBD baselines. The K-OBD and LLM Surgeon use the multi shot procedure of section 3.5 using T-40 shots for structured pruning and T=5 shots for semistructured and unstructured pruning. Further details are found in appendix

---

4.1 STRUCTURED COMPRESSION
Structured compression of rows and columns enables direct savings in memory and compute through straight reduction of matrix dimensions in the model. For LLM surgeon, consider in section 3.4 weight updates with different levels of correlations: limited to correlations within rows and columns, and correlations both within and between rows and columns. We further compare against magnitude pruning. which only uses weight magnitudes, L-OBD, which only uses activations, and K-OBD, which also uses Kronecker-factoree curvature but assumes full independence and thus only prunes without updating remaining weights. We report results in table] | and observe that more correlations results in better performance, with the largest improvements for the Llama-v2 model family.
While 50% structured compression is not better than smaller model of similar size, LLM Surgeon allows us to reduce model size by up to 30% with minimal loss, without training a smaller model from scratch fig. In structured compression experiments our proposed LLM Surgeon method outperforms baselines and achieves the best performance for each compression target size.
4.2 INTERLEAVED LOW-RANK UPDATES
Additionally, assess compression performance in conjunction with the proposed first-order cor- rections using the interleaved low-rank adaptation described in section 3.6 We find that LoRA im- proves compression performance in the smallest 125m model, but not in larger models. We hypoth- esise that larger models are more prone to overfit- ting the relatively few batches of wikitext-2 data used to compress the model. Nevertheless, we con- clude that interleaved LoRA can be useful in cases, and recommend first using the proposed method without interleaved updates and, if enough data is available for compression, optionally using it if it improves performance.
Table 2: Structured compression of OPT- 125m
wikitext-2 using interleaved LoRA A updates
Target Size
without LoRA
with
LoRA
Pretrained
100%
27.65
23.35
LLM Surgeon
90%
28.01
24.16
(ours) G & full cor. A
80%
28.73
25.25
70%
31.82
28.86
60%
38.47
31.26
50%
49.78
36.50
4.3 SEMI-STRUCTUREL COMPRESSION
For 2:4 semi-structured pruning, we compare LLM Surgeon with magnitude pruning, which only uses weight magnitudes, single-shot L-OBD, which only uses activations, and single-shot K-OBD, which also uses Kronecker-factorn curvature but assumes full independence and thus only prunes without updating remaining weights as well as the recent state-of-the-arr SparseGPT (Frantar & Alis- tarh 2023). We report test performance after 50 % (2:4) semi-structured compression on wikitext-2 data in table 3 We empirically find that considering more weight correlations results in improved final performance after compression. Our proposed LLM Surgeon is competitive with prior work outperforming all baselines in terms of test set perplexity (PPL).
Table 3: Semi-structured 2:4 compression for large language models wikitext-2 data.
Target size
Test performance (PPL)
Method
F Rs
OPT (125m)
OPT (1.3b)
OPT (2.7h)
OPT (6.7b)
Baseline
100%
27.65
14.62
12.47
10.86
Magnitude
I
50%
342.04
379.57
1106.01
187.29
L-OBD
diag(
I
A)
50%
87.26
44.92
41.40
27.36
K-OBD
diag I
A)
50%
68.74
27.22
20.23
15.55
SparseGPT
50%
45.51
29.44
14.92
13.01
L.M Surgeon (ours)
50%
44.64
25.10
14.64
12.10
4.4 UNSTRUCTURED COMPRESSION
For unstructured pruning, we repeat the same experiments as structured pruning case described in section 4.1 In table 4 we report final test performance in terms of perplexity (PPL) on wikitext- 2 after compressing LLMs of different sizes of OPT and Llama-v2 family. Overall, we find that methods with more accurate approximations of the curvature landscape and that account for more correlations perform better. The proposed LLM Surgeon outperforms all baselines, reaching the highest test performance across target sizes.
8

---

Table 4: Unstructured compression of large language models wikitext-2 data.
Target size
Test performance (PPL)
Method
OPT (125m)
OPT (1.3b)
OPT (2.7b)
OPT (6.7b)
Llama-v2 (7b)
Baseline
100%
27.65
14.62
12.47
10,86
5.12
Magnitude
90%
27.62
14.69
12.60
10.88
5.18
80%
28.53
15.68
13.18
11.26
5.37
52.88
15.22
12.22
L-OBD
90%
29.70
16.24
14.44
13.43
6.09
diag(1 I @ A)
80%
32.18
21.92
23.35
39.85
116.2
single shot
70%
49.08
204.7
274.8
810.4
6549
K-OBD
27.64
14.62
12.09
36.89
5.13
G & A single shot
80%
27.62
14.37
130220
39928
5.19
70%
27.92
220.1
23097
19506
5.60
60%
29.24
13783
10331
33896
9.20
50%
34.43
7311
10495
91506
118.6
SparseGPT
90%
27.93
14.69
12.00
10.86
5.49
(
80%
28.18
15.07
12.05
10.86
5.58
28.93
22.77
12.17
10.89
60%
30.20
25.07
12.37
10.98
5,94
50%
33.17
26.77
12.88
11.92
6.51
LLM Surgeon (ours)
90%
27.69
14.62
12.01
10.86
5.13
G1 A
80%
27.83
14.66
12.14
10.87
5.20
full A
70%
28.35
14.81
12.25
10.82
5.36
multi shot
28.98
12.28
5.66
50%
30.30
15.47
12.68
10.97
6.08
4.5 LEARNED SPARSITY STRUCTURE
The proposed method can dynamically allocate sparsity across layers through global thresholds described in section 3.3 Fig. 4.5 we compare total allocated sparsity levels per layer depth and layer type after compressing pretrained OPT-125m model. We find that the LLM Surgeon prunes relatively more in the first layer and less middle layers. Further, we observe that a larger portions of weights are removed in fully-connected compared to attention blocks, but deviations less compared to other methods. Dynamic allocation allows for most pruning where it hurts least.
Sparsity per layer depth
Sparsity per layer type
nn weights
LLM Surgeon Ifull, cor A)
80%
80%
Magribude
60%
L-OBD
40%
20%
0%
0%
10
11
12
FC1
FC2
layer
layer type
Figure 4: Sparsity levels obtained with structured pruning on OPT-125m by layer depth and type.
CONCLUSION
In this work, we have introduced the LLM Surgeon algorithm for unstructured, semi-structured and structured compression of neural networks. The work builds upon classic neural network compres- sion approaches originating from the early 1990's that aim to find optimal pruning by expanding the curvature of the loss landscape. The method utilises modern Fisher approximations scale accurate pruning to the realm of large language models (LLMs) with billions of parameters, while remaining practical in both memory and compute. Unlike most prior work on data-based LLM compression, we not only use weight magnitude and activations from forward passes, but also use gradient information from backward passes to relate weight removal costs to the true final objective. We improve upon prior work through more accurate approximations to the loss landscape curvature and considering more weight correlations to update remaining weights. Increasing the number of correlations and using multiple shots allows trading off additional compute for better accuracy. Lastly, LLM Surgeon gives the first practically usable results for structured pruning of LLMs and achieves state-of-fe -feth-------ttttt- results in unstructured and semi-structured large language model pruning.

---

REFERENCES
Salch Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.15022 2024.
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, vol- ume 4. Springer, 2006.
Aleksandar Botev, Hippolyt Ritter, and David Barber. Practical gauss-newton optimisation for deep learning. In International Conference on Machine Learning, pp. 557-565. PMLR, 2017.
Runa Eschenhagen, Alexander Immer, Richard Tumnrr Frank Schneider, and Philipp Hennig. froneccke-facctord approximate curvature for modern neural network architectures. Advances in Neural Information Processing Systems, 36, 2024.
Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems, 35:4475 4488, 2022.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. 2023.
Gene H Golub and Charles F Van Loan. Matrix computations. JHU press, 2013.
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprini arXiv:2106.0966 09685, 2021.
Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Accurate post training quantization with small calibration In International Conference on Machine Learning, pp- 4466-4475. PMIR, 2021.
Alexander Immer, Tycho der Ouderaa, Gunnar Rätsch, Vincent Fortuin, and Mark der Wilk. Invariance learning in deep neural networks with differentiable laplace approximations. Advances in Neural Information Processing Systems, 35:12449 1441144-112633, 2022.
Abdoulaye Koroko, Ani Seriaux-SSedraian, Ibtihel Ben Gharbia, Valérie Garès, Mounir Haddou, and Quang Huy Tran. Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition. arXiv preprint MXXi:22211100280 2022.
Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical fisher approx- imation for natural gradient descent. Advances in neural information processing systems, 32, 2019.
Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models. arXiv preprint arXiv:2203.07255 2022.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1.0 regularization. arXiv preprint arXiv:1712.01312. 2017.
David JC MacKay. Information theory, inference and learning algorithms. Cambridge university
press, 2003.
James Martens and Roger Grosse. Optimizing neural networks with chonecker-factorree approximate curvature. In International conference machine learning, pp. 2408-2417. PMLR, M. 2015.
10

---

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv: arXiv:1609.0784. 1684900077,,,, 2016.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307. 092888, 2023.
Tycho van der Ouderaa, Alexander Immer, and Mark van der Wilk. Learning layer-wise equivari- ances automatically using gradients. Advances in Neural Information Processing Systems, 36, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Chaoqi Wang, Roger Grosse, Sanja Fidler, and Guodong Zhang. Eigendamage: Structured pruning in the conecker-factoree eigenbasis. In International conference on machine learning, pp. 6566 6575. PMLR, 2019.
Wikipedia. Wikipedia. PediaPress, 2004.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transformers: State-of-the the-art art natural language processing. arXiv preprint 1r1iv:1910.0377,, 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0106: 2022.
Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hong- sheng Li. Learning fin-nnn structured sparse neural networks from scratch. arXiv preprint arXiv:2102.04010 2021.
11


Reviews:
[{"review_id": "TtxnA4nbva", "reviewer": "Reviewer_zF62", "summary": "This paper proposes a way to prune the weights of a pretrained LLM with a negligible loss in performance by iteratively solving a quadratic optimization problem using the curvature of a local minimum. To save the memory of materializing hessian, the work calculates the covariance layer-wise with Kronecker factorization. Also, the curvature is calculated incrementally and the remaining weights are corrected with a first-order term as more weights are pruned so that the weight remains in a local minimum.", "questions": "Equation 2 should read `- log(D | theta)`. There is a typo in `General solution  We denote ... e_{q_k}` => `e_{k_q}`.\n\nRelated Work could include more previous works on LLM compression.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The approach is sound and is presented well. Not having to materialize the hessian makes this amenable for LLM pruning.", "weaknesses": "While it is believable that this method would generalize, the setup is unsatisfying in that the dataset used for compression is drawn from wikitext-2 which is a narrow domain (meaning the loss landscape may be easier to optimize and prune than a broader distribution), and the final model is evaluated only on the test perplexity of the same dataset. SparseGPT uses C4 and reports downstream performance on various standard benchmarks. The paper could really strengthen its claim by repeating the same setup as SparseGPT. Downstream benchmarks are required for a fair comparison and acceptance of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a4WMew260w", "reviewer": "Reviewer_zzzW", "summary": "The paper proposes a method called \"LLM Surgeon\" for efficient pruning and compression of large pretrained language models like OPT and LLAMa. It scales up second-order Hessian-based pruning methods like Optimal Brain Surgeon using Kronecker-factored approximations of the Fisher information matrix. It derives closed-form solutions for removal costs and correlated weight updates when pruning multiple weights jointly. Experiments show the method can prune OPT and LLAMa models by 20-30% with minor performance loss and outperforms prior work.", "questions": "- Could the authors provide both inference speed and additional cost for approximation and updating the offsets?\n- Could larger model sizes also be included and evaluated throughout?\n- Besides the PPL, could the author also provide the 0shot performance degradation regarding the OPT/LLaMA models for a comprehensive evaluation;", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and clearly presented; \n- The paper provides a general pruning framework applicable to different structured and unstructured schemes. \n- The paper provides careful derivation of update rules that consider correlations between weights, theoretically principled and extends classical segundo-order pruning methods.\n- The proposed methods achieves state-of-the-art pruning results on large language models, especially for structured pruning. \n- Detailed ablation regarding the low-rank components, approximation methods, as well as the qualitative sparsity level analyses are provided to show the comprehensiveness of the proposed methods and design choices;", "weaknesses": "- The paper still uses approximations for computational tractability which limits pruning performance.\n- Structured pruning leads to irregular sparsity patterns which are difficult to accelerate. The real inference speedup or memory savings after pruning is unknown; \n- Additional FLOPs for approximation and updating offsets gains during deployment are needed, while the detailed comparison and discussion might be missing.\n- Some related works might also be good to include [1];\n\n[1] Yu, Shixing, et al. \"Hessian-aware pruning and optimal neural implant.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OqD3oiG1WH", "reviewer": "Reviewer_2GDn", "summary": "This paper introduces LLM Surgeon, a method that enhances the scalability of Kronecker-factored curvature approximations of the targeted loss landscapes, designed for pruning LLMs at arbitrary sparse patterns. The authors demonstrate that this proposed methodology consistently improves the PPL performance of current methods across a range of LLMs.", "questions": "Please see the weakness part.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-organized and easy to comprehend.\n\n2. The fundamental idea for estimating the curvature of the loss landscape is reasonable and innovative.", "weaknesses": "1. Since the authors adopt a multi-shot sparse approach, it would be beneficial to quantitatively compare the time costs and GPU memory consumption with SparseGPT.\n\n2. While the authors emphasize sparsity in large models, the largest model they utilize is of 7-billion parameters. It might provide readers with a clearer view if experiments involving larger model sizes were included.\n\n3. Global rank ordering is a sound strategy, but there seems to be a lack of an ablation experiment, that is, whether the suggested method outperforms SparseGPT under the same layer-wise sparsity situation.\n\n4. Although the authors underlines that LLM surgeon can be migrated to structured pruning, and as the authors stated, \"To the best of our knowledge, this is the first method to successfully perform structured pruning for LLMs,\" they do not discuss nor compare their approach to the already presented structured LLM pruning method[1]. This lack of discussion appears less meticulous. Furthermore, the authors should also consider comparing their work with another state-of-the-art large model sparse method, Wanda [2], which can also be adapted for pruning LLMs at any patterns.\n\n[1] LLM-Pruner: On the Structural Pruning of Large Language Models. In NeurIPS, 2023\n[2] A Simple and Effective Pruning Approach for Large Language Models. In Arxiv, 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)