SliceGPT: Compress Large Language Models by Deleting Rows and Columns
Authors: Saleh Ashkboos
Maximilian L. Croci
Marcelo Gennari do Nascimento
Torsten Hoefler
James Hensman
Venue: ICLR 2024 poster (2024)
Keywords: compression
sparsification
large language models
Primary area: generative models
TL;DR: We compress large language models by replacing the weight matrices with smaller (dense) weight matrices, which leads to speedup and storage reduction.

Abstract: Large language models have become the cornerstone of natural language processing, but their use comes with substantial costs in terms of compute and memory resources. Sparsification provides a solution to alleviate these resource constraints, and recent works have shown that trained models can be sparsified post-hoc. Existing sparsification techniques face challenges as they need additional data structures and offer constrained speedup with current hardware. In this paper we present SliceGPT, a new post-training sparsification scheme which replaces each weight matrix with a smaller (dense) matrix, reducing the embedding dimension of the network. Through extensive experimentation we show that SliceGPT can remove up to 25% of the model parameters (including embeddings) for LLAMA-2 70B, OPT 66B and Phi-2 models while maintaining 99%, 99% and 90% zero-shot task performance of the dense model respectively. Our sliced models run on fewer GPUs and run faster without any additional code optimization: on 24GB consumer GPUs we reduce the total compute for inference on LLAMA-2 70B to 64% of that of the dense model; on 40GB A100 GPUs we reduce it to 66%. We offer a new insight, computational invariance in transformer networks, which enables SliceGPT and we hope it will inspire and enable future avenues to reduce memory and computation demands for pre-trained models.

Published as a conference paper at ICLR 2024
SLICEGPT: COMPRESS LARGE LANGUAGE MODELS BY DELETING ROWS AND COLUMNS
Saleh Ashkboos
Maximilian L. Croci* Microsoft Research
Marcelo Gennari do Nascimento Microsoft
ETH Zurich
Torsten Hoefler
James Hensman Microsoft Research
ETH Zurich
ABSTRACT
Large language models have become the cornerstone of natural language process- ing, but their use comes with substantial costs in terms of compute and memory resources. Sparsification provides solution to alleviate these resource constraints, and recent works have shown that trained models can be sparsified post-hoc. Exist- ing sparsification techniques face challenges as they need additional data structures and offer constrained speedup with current hardware. In this paper we present SliceGPT, a new post-training sparsification scheme which replaces each weight matrix with a smaller (dense) matrix, reducing the embedding dimension of the network. Through extensive experimentation we show that SliceGPT can remove up to 25% of the model parameters (including embeddings) for LLAMA-2 70B, OPT 66B and Phi-2 models while maintaining 99%, 99% and 90% zero-shot task performance of the dense model respectively. Our sliced models run on fewer GPUs and run faster without any additional code optimization: consumer GPUs we reduce the total compute for inference on LLAMA-2 70B 64% of that of the dense model; on 40GB A100 GPUs reduce it to 66%. We offer a new insight, computational invariance in transformer networks, which enables SliceGPT and we hope it will inspire and enable future avenues reduce memory and computation demands for pre-trained models. Code is available at: https:/ tttps://githuse /tthhtttt..ug.ggg com crosoft
24GB
INTRODUCTION
Large language models (LLMMs) are neural networks with billions of parameters, trained on trillions of tokens (Zhao et al., 2023). The cost of training LLM has caused shift to re-using pre-trained models for multiple tasks, the foundation model paradigm. The size of LLMs makes deploying a pre-trained model expensive undertaking. Many models require multiple GPUs to be able to compute prediction, and because the models are autoregressive, multiple forward passes of the neural network needed generate text responses. is therefore of widespread interest to reduce the computational requirements of these models, usually performed via post-training techniques referred as model compression.
A majority of model compression techniques fall into one of four categories: distillation, tensor decomposition (which includes low-rank factorization), pruning and quantization (Hoefler et al., 2021; Gholami et al., 2021; Zhu et al., 2023; Gupta & Agrawal, 2021). In this work we focus pruning. though we hope that our methodology may influence future work on other areas. Whilst pruning methods have been around for some time, many approaches require recovery fine-tuning (RFT) after pruning to maintain performance, making the overall process an expensive and hard-to-scale task. With SliceGPT we compress large models using single GPU in just a few hours and maintain competitive performance on generation and downstream tasks even without RFT.
* Equual contribution
Work completed as an intern at Microsoft.
i ssalle, leh. ashkboos, torsten. hhe
(t-mcroci, marceloge,
j jameshensman) amicrooott.cmm

---

Published as a conference paper at ICLR 2024
Unstructured sparsity
2:4 Structured sparsity
Slicing (ours)
W
W
W
X
X
XQ
Matrix multiplication of the signal X and weight matrix W under different types of sparsity. Left: unstructured sparsity, where some elements of W are zero, and X is dense. Middle: 2:4 structured sparsity, where each block of four weight matrix entries contains two zeros, and X is dense. Right: SliceGPT, where after introducing transformation Q, all the sparsity is arranged to the bottom rows of W and the corresponding columns of X are removed.
Figure 1:
Pruning methods work by setting some elements of the weight matrices in an LLM to zero, and (optionally) updating the surrounding elements of the matrix to compensate. The result is sparse pattern which means that some floating point operations can be skipped in the matrix multiplications required in the forward pass of the neural network. The relative speedup of the operations depends on the level of sparsity and the sparsity pattern: more structured sparsity associated with more computational gain. In contrast other pruning methods, SliceGPT prunes away (slices off!) entire rows or columns of the weight matrices. Before slicing, we perform single transformation of the network which leaves the predictions invariant, but allows the slicing to have only small effect.
The result is that weight matrices are smaller, and the signals passed between blocks of the neural network are smaller too: we reduce the embedding dimension of the neural network.
Figure compares our approach with existing sparsity methods. Our contributions are as follows:
1. We introduce the idea of computational invariance: we show that we can apply orthogonal- matrix transformations each weight matrix in a transformer without changing the model.
2. We use this to edit each block in a transformer architecture, such that we are projecting the signal matrix' between blocks onto its own principal components. We remove columns or rows of the transformed weight matrices to reduce the model size. We call the transformation and removal of weights SliceGPT.
3. We conduct multiple experiments on OPT (Zhang et al., 2022) and LLAMA-2 (Touvron et al., 2023) LLMs, demonstrating that SliceGPT is able to compress these models by up to 30% with superior perplexity to the state the art 2:4 scheme. On downstream tasks we additionally experiment with Phi-2 and show that all models can be sliced by up 30% while maintaining >90% of the dense performance.
BACKGROUND
In this section, we first describe some necessary background on transformer architectures, which allows to introduce notation which will use to prove main results. Then we describe related work on sparsification for compressing such architectures.
2.1 TRANSFORMER NETWORKS
Transformer networks (Vaswani et al., 2017) are a class of neural networks that have been shown be effective at wide range of tasks including language modeling. The transformer architecture is composed of a series of layers, each of which is composed of a multi-head self-attention block followed by a feed-forward network block. Between each block, there is a LayerNorm (Ba al., 2016) (or RMSNorm (Zhang & Sennrich, 2019)) block. Figure 2 illustrates part of transformer network: an attention block connected Feed Forward Network (FFN) block through a LayerNorm block, with residual connections. The following describes the operations of each component (ignoring dropout, which is not applied post-training).
'The signal matrix is sometimes referred activation matrix.
2

---

Published as a conference paper at ICLR 2024
Embeddings Let D be the embedding dimension of our transformer, N be the sequence length. The transformer model takes as input a sequence of token IDs and position IDs, and uses them to index the embedding matrices, producing the initial signal X with shape N x D. In what follows consider, without loss of generality, a single embedding matrix Wemba indexed by input sequence 8.
LayerNorm After embeddings, the signal matrix is passed through a LayerNorm operation, which subtracts the mean from each row of the matrix, divides the row by its standard deviation, rescales (columnwise), and adds an offset. We write the LayerNorm block as
(I) where RMSNorm X) applies² x: <- zz/ [ to each row of X. The vector parameter a and offset (vector) parameter are learned independently at each LayerNorm instance. The constant matrix M = 111 is a D X D matrix which subtracts the mean from each row of X.
LayerNorm X) = ZMMSNrr(XMMddinee (e) (t) DDVVD++ 1
Attention Blocks The attention block has four matrices: Wk. W W. and W.. o. each of dimen- sion D D. The input signal arriving into the block is projected into the Key (XWa) Query (XW,, x). and Value (XW,,) matrices, which are then split into multiple heads. A nonlinear opera- tion is applied at each head before the signals are combined and multiplied by the output weight matrix W.. Since the first three weight matri- applied separately the inputs, can concatenate them and perform a single matrix multiplication (denoted by the white box around these matrices in Figure 2). We can consider the concatenation of these matrices to be a single lin- ear layer, which we denote Win. We also refer to the output matrix as W... om. We treat the attention block as a WW in + b W out + b where represents the multi-head attention operation.
Activation Function
FFN
FRTT
diag(r))
LayerNorm
Attention
w.
W,
w
Muti--Had Allention
W
FFN Blocks The other type of block that ap- pears in transformer architectures is Feed For- ward Network (FFN) block. In many cases, this is Multi-layer Perceptron (MLP), which consists of a linear layer W,, followed by an element- wise operation a, followed by a second linear W2 + b2. Some architec- tures have adopted the gated format, where an additional matrix is used, and the operation is (o (X(W + bı) (XW2)) W3. where o is an element- wise product. Much like the first three linear layers in the attention module, we can con- sider the concatenation of W, and Wa to be single linear operation, and denote it Win. We can therefore denote the operation of MLP or gated FFN layers a (XW,,, in W out. where takes different meaning to that in an attention.
Inputs
A single layer in a transformer net- work. The signals (inputs) arising from the pre- vious blocks of the networks arrive at the bottom of the figure, before being passed through atten- tion, LayerNorm, and FFN. The attention and FFN blocks both have input and output linear op- erations (blue) which we denote in the text as Win: W Wour- The linear operations of LayerNorm M and diag(a) are highlighted. This and subse- quent figures do not show biases.
Figure 2:
layer: (XW, +
Language Modelling (LM) Head All of the transformer networks to which we apply SliceGPT in this paper have decoder-only structure following (Radford et al., 2018): after multiple layers applying alternating attention and FFN blocks, a head block computes logits which are used to compute the loss during training and token prediction on deployment. The head operation is X head bread where X is the output of the last transformer block.
in some implementations special instances of LayerNorm and handle them accordingly.
RMSNorm block may contain scale parameters. We consider these be
Forr ease of notation here and throughout this paper, we abuse notation slightly and omit the broadcasting of the bias terms across the sequence length dimension. The complete notation for the operation of an attention block is ooX W., + 1 b W out 1 b
3

---

Published as a conference paper at ICLR 2024
Forward pass Once the model is trained and all of the parameters are set, the computations required in a transformer network to produce predictions involve passing signal matrices from one block to the next until the head node is reached. Since we able to define both FFN and attention blocks in the form a (XWin + bmn) W out + Doat where we understand that a represents either a point-wise or multi-head-attention nonlinearity,
are able to describe the forward pass using Algorithm 1.
Algorithm 1 The forward pass of transformer network
Require: {Win bin W blut M
weights and biases of FFN and attention blocks I/ nonlineurity associated with each block LayerNorm or RMSNorm instances to perform between blocks // embedding and head matrices If input sequence If index embeddings normalize
Require:
{az}
Require: (Norme]és
Require: Wemta, Whead, bhend
Require:
1: X Wantd s
<- Normo (X)
3: for = I... L do
4: Z - de (x WT + big
W + bun
Il apply FFN or attention
5: X Norm, +
I/ normalize and apply residual connection
6: end for
7: return XW head
Dhead
I| apply model head
2.2 RELATED WORK
In the simplest setting, one can employ magnitude-based sparsification, which involves setting the smallest weights in the model to zero (Han et al., 2016; Zhu & Gupta, 2017; Gale et al., 2019). Although magnitude sparsification is scalable, its application to LLMs gives too strong a degradation in performance (Frantar & Alistarh, 2023). Optimal Brain Surgeon (OBS) (Hassibi et al., 1993; LeCun et al., 1989), a more sophisticated method, systematically removes weights that have the least impact the loss function. The method compensates for the error introduced by weight removal by updating the un-pruned weights using the inverse of the Hessian matrix. Unfortunately, OBS impractical for models with a few million parameters due to the need to calculate and store the inverse of the Hessian matrix. To address the computational limitation posed by OBS, recent research has explored two approaches: approximating the inverse of the Hessian matrix such as WoodFisher (Singh & Alistarh, 2020) or applying it separately to each layer such as in Optimal Brain Compression (OBC, Frantar & Alistarh, 2022), known as layer-wis, pruning. While these techniques have proven effective for medium-sized networks, they are not practical for large language models, where individual layer weight matrices typieally contain more than 10 parameters.
GPTQ (Frantar et al., 2022) has solved this issue by quantizing (representing the parameter using lower precision) the weight matrix of LLMs using a column-by-column scheme and updating all not-yet-quantize weights in the next columns. SparseGPT (Frantar & Alistarh, 2023) applied the same idea for pruning and sparsifies the LLMs using unstructured and semi-structured pruning, and Sun al. (2023) simplified the idea by using only the diagonal of the Hessian. Since achieving end- to-end speed improvements through unstructured pruning is a demanding task, they also attempted similar technique induce sparsity with semi-structured patterns like 2:4 and 4:8 (Mishra et al., 2021). However, implementing such structures does not maintain the accuracy of the model.
Another approach compression is low-rank approximation, where each weight matrix replaced with the product of two matrices with a smaller inner dimension, usually followed by fine-tuning step (Hu et al., 2021; Mahabadi et al., 2021; Noach & Goldberg, 2020; Tukan et al., 2020). To achieve compression, the inner dimension must be smaller than half of the original dimension. In contrast, our method replaces each weight matrix with a single smaller one, reducing the embedding dimension without the need for fine-tuning.
We propose to delete rows and columns of weight matrices, which is similar to pruning of filters and channels in the convnet literature. There, -induiiiin regularization is added to batch-norm factors (Liu et al., 2017) or network structures (Huang & Wang. 2018), and the network is trained or fine-tuned, resulting in the pruning of channels or parts of the network. Perhaps the most analogous methods to ours are ThiNet (Luo et al., 2017; He et al., 2017), which apply linear operations between layers (as will we), interleaved with more fine-tuning with regularization. In this literature, the model sizes are typically several orders of magnitude smaller than in LLMs, for example the VGG16 network has 138M parameters, comparable with the very smallest OPT model that consider. The
4

---

Published as a conference paper at ICLR 2024
huge size of LLMs makes methods that involve extensive fine- tine-tuning unappealing, especially when outer-loops are needed to select regularization parameters.
Recently, some works have been proposed that apply structured pruning to LLMs, followed by continued training (or fine-tuning) to recover the performance that is lost. For example LLM-pruner (Ma al., 2023a) removes connected structures from LLM before further training. Contemporarily with our work, LLM Surgeon (van der Ouderaa et al., 2023) interweaves recovery fine-tuning with pruning. We provide results for SliceGPT as a single-shot method and with post-slicing recovery fine-tuning.
3 SLICEGPT
Our SliceGPT method relies on computational invariance that is inherent in the transformer architecture. By this, we mean that it possible to apply an orthogonal transformation the output of one component, so long as it is undone in the next. Our key insight is that the RMSNorm operation which is performed between blocks of the network does not affect the transformation: the operations commute. In this section, we first describe how the invariance occurs in RMSNNor--connccted transformer networks, then we note how networks trained with LayerNorm connections can be converted to RMSNorm. Next, we describe our method to compute transformations at each layer using Principal Component Analysis (PCA), such that the signal between blocks is projected onto its principal components. Finally, describe how deleting the minor principal components corresponds to slicing away rows or columns of the modified network.
3.1 COMPUTATIONAL INVARIANCE IN TRANSFORMER NETWORKS
Let Q denote an orthogonal matrix: we have QTQ QQ = I. Note that multiplying a vector æ by Q does not change the norm of the vector, since ||a|| Vx T Q Qæ Vx x ||=]| In this work, the dimensions of Q will always match the embedding dimension of the transformer D.
Suppose that X₃ is the output of one block of the transformer, which is then processed by RMSNorm, and then inputted to the subsequent block as RMSNorm(X;)) If we insert linear layers with the orthogonal matrix Q before RMSNorm and Q T after RMSNorm, the network remains unchanged, since each row of the signal matrix is multiplied by Q, normalized and multiplied by Q We have
(o,Norm(X;Q00
RMSNorm(X.)..
(2)
A proof of this relation appears in Appendix A. Now, since each attention or FFN block of the network has a linear operation on both the input and output, we can absorb the additional operations Q into the linear layers of the blocks. Since the network contains residual connections, must also apply Q to the output of all previous layers (all the way back to the embedding) and to all subsequent layers (all the way up the LM Head).
An invariant function is one for which a transformation to the input does not result in a change to the output. In our case, we can apply any orthogonal transformation Q to the weights of the transformer without changing the result, so the computation can be performed in any transformed state. We refer this computational invariance, and define it in the following theorem.
Theorem 1. Let W and W!. out be the weight matrices of the linear layers of the l-th block of an RMSNorm-ccnnected transformer network, and b b be the corresponding biases. if any, and let emiut and W head be the embedding and head matrices. Let Q be an orthogonal matrix of dimension D. Then the following network is equivalent the original transformer network:
W embd W
VebbaQ W!
(3)
6 = Q Te our
(6)
W
in Q
(4)
W head
Q T W head
(7)
W
W Q
(5)
The input and head biases are copied: b/ by bread
-head-
Proof. We can show that the transformed network computes the same results as the original by
stepping through Algorithm 1. Suppose that line 1, the original network has computed X, then
5

---

Published as a conference paper at ICLR 2024
the modified network has computed X XQ, using Equation 3. Applying RMSNorm on line 2, see that the operation of the two networks matches: by Equation 2 we have RMSNorm (X) (MSN = R.MSNorm (.)Q. Applying the nonlinearity line 4,
that XW
XW in. using Equation and it follows that Z ZQ. On line the residual connection means we have (X + 2) = (X + Q) and applying RMSNorm results in assignment of X = XQ This follows through to the end of the loop. Finally, on line 7, the transformations are undone as X bead XWheca head using Equation 7.
3.2 LAYERNORM TRANSFORMERS CAN BE CONVERTED TO RMSNORM
The computational invariance of the transformer network applies only to RMSNorm-ccmmected networks. Before working those with Layer- Norm, we convert the network to RMSNorm by absorbing the linear blocks of LayerNorm into the adjacent blocks. Figure shows such a trans- formation on the transformer network (see Figure 2). In each block, we multiply the output matrix W out by the mean-subtraction matrix M, which accounts for the mean subtraction that would hap- pen in the subsequent LayerNorm. The input matrices Win are pre-multiplied by the scales of the preceding LayerNorm blocks. The em- bedding matrix W amdd must be mean-subtracted, and W hend must be re-scaled by the last Layer- Norm scales. This is straightforward change in the order of operations and does not affect the network output.
[o/W;
Activation Function
WaM
FFN
1-1
RMSNom
Atteniion
(o') J
(o')W
(e
Multi-Head Attention
W.M
Inputs
3.3 A TRANSFORMATION PER BLOCK
Now that every LayerNorm in the transformer has been converted to RMSNorm, we can select any Q to modify the model. Our initial plan was to collect signals from the model, construct an orthogonal matrix using those signals and to delete parts of the network. We quickly saw that the signals different blocks of the network were not aligned, and that we would need to apply different orthogonal matrix each block, Qe-
Converting a transformer network from LayerNorm to RMSNorm: the scale matrix diag(a)) is absorbed into the subsequent matrix Win- Figure shows the block in combined colors. We use (ar for brevity. The mean-subtraction matrix M is applied to each matrix W Layer- norm becomes RMSNorm, up to constant VD (not shown). Here, the scaling (a') comes from the previous block.
Figure 3:
Allowing the orthogonal matrix used in each block to differ can be shown to leave the model unchanged using the same proof as Theorem 1, with the exception of line 5 of Algorithm 1. Here we see that the residual connection and the output of the block must have the same rotation. To fix this, modify the residual connection by applying the linear transformation Q Q₂ to the residual. Figure shows how different rotations can be applied to different blocks with the additional linear operation in the residual connection. Unlike the modifications to the weight matrices, these additional operations cannot be pre- pre-computed and add a small (D x D) overhead to the model. Nonetheless, they are needed allow slicing the model (Section 3.4) and we see real speedup overall (Section 4).
To compute the matrices Qe. we use PCA. We select a calibration dataset from the training set, run it through the model (after converting LayerNorm operations into RMSNorm), and extract the orthogonal matrix of the layer. We use the output of the transformed network to calculate the orthogonal matrices of the next layers. More precisely, if Xt,, is the output of the 4th RMSNorm block for the 7thh sequence in the calibration dataset, we compute
C EX,
(8)
and set Qi the be the eigenvectors of C₁, sorted by decreasing eigenvalues.

---

Published as a conference paper at ICLR 2024
Figure 4: With the network converted to RMSNorm (see Figure 3), we apply the computationallinvaiiane idea. The input weight matrices diag(or) W are pre-multiplied by Q The output matri- ces o are post-multiplied by Q. In the skip-connection, new linear layer is added Q Qe+1. After these modifica- tions, the matrices can be sliced (hatched areas).
Q.a
Co
Acthvaton
0.0
Qu Jea vw.
Ra we
Mali-Hesd Attettion
W.MO
Anserion
eogtmgitggto by Qu and mineeatd
3.4 SLICING
The goal of Principal Component Analysis is usually to take data matrix X and compute lower dimensional representation Z, and an approximate reconstruction X:
XQD,
X= x=ZD Q
where Q is the eigenvectors of X X X. and D is D D small deletion matrix (containing D columns of the D) x 1 identity matrix), which removes some of the columns of the matrix to the left. The reconstruction is L2 optimal, in the sense that QD is a linear mapping that minimizes ||X - -
When apply PCA to the signal matrix X between blocks, never materialize the N D signal matrix, but we apply the deletion matrix to the operations preceding and succeeding the construction of that matrix, which have already been multiplied by in the above. We delete rows of Win and columns of Waut and Wenhd- We also delete both rows and columns of the matrix Q Or that we have inserted into the residual connection (see Figure 4).
EXPERIMENTAL VALIDATION
Setup We Hugging Face Transformers (Wolf et al., 2019) to implement code with PyTorch (Paszke et al., 2019). The computation of Q is performed on a single H100 GPU with 80GB of memory, taking approximately 3.5 hours complete for the LLAMA-2 70B model. We use double precision for the PCA calculation because using single precision for eigenvector calculations in PyTorch leads discrepancy in the final accuracy, as detailed in Appendix A.2.
We experiment with two different calibration sets: the WikiText-2 training dataset (Merity et al., 2016) and the Alpaca training dataset (Taori al., 2023). An ablation study on the calibration set size and sequence length is presented in Appendix A.3. We apply a small amount of RFT to sliced LLAMA-2 and Phi-2 models using LoRA (Hu al., 2021), following the idea from Ma al. (2023a). For models sliced with WikiText-2 we use approximately 1k sequences, for those sliced with the Alpaca dataset we use 5k. We use LoRA with r = 32, a 10 and sequence length 1024, and defaults for all other hyperparameters in PEFT (Mangrulkar al., 2022).
evaluate all our experiments on OPT (Zhang et al., 2022), LLAMA- 2 (Touvron et al., 2023) model families, and additionally evaluate Phi-2 (in our zero-shot task) experiments. We exclude OPT 175B, as it is outperformed by smaller LLAMA-2 models. Nonetheless, anticipate that this larger model will yield improved results, as larger models typically offer more promising opportunities for compression (see Section 4. 4).. We evaluate our scheme on both language generation well as popular zero-shot tasks. To demonstrate the comprehensive speedup achieved by SliceGPT we use: Quadro RTX6000 GPUs with 24GB of memory as a representative example of consumer-level GPUs; 40GB A100s and 80GB H100s provide datacenter-level benchmarks.
Models, Tasks, and GPUs
Baseline Setup We initially planned to compare our results against a scheme that pruned columns (or rows) with the smallest norm but found that this baseline was very poor, with the WikiTexxt2
7

---

Published as a conference paper at ICLR 2024
perplexity of the model soaring into the 1000s after pruning just few columns. Instead, we compare SliceGPT against SparseGPT (Frantar & Alistarh, 2023) employing a 2:4 sparsity ratio, as this is the only sparsity scheme which achieves speedup (Mishra et al., 2021).
4.1 RESULTS
Generation Task We begin by showcasing our findings using the WikiText-2 dataset. In this context, we evaluate the performance of both the OPT and LLAMA-2 model families across different sizes when using this dataset for slicing. Table shows the perplexity obtained by various slicing levels. SliceGPT exhibits superior performance when applied to OPT models compared to LLAMA-2 models which matches our intuition from the spectrum analysis of those models (see Appendix A.4 for our discussion). The performance of SliceGPT improves as the model size increases. Comparing SliceGPT with SparseGPT, we see that that SparseGPT 2:4 performs worse than SliceGPT with 25% slicing in all LLAMA-2 models. For OPT, we see that 30% sliced models beat 2:4 sparsity for all model sizes except 2.7B.
Table 1: OPT and LLAMA-2 perplexity results on WikiTex12. The calibration set size and sequence length are 1024 and 2048, respectively.
OPT
LLAMA-2
Method
125M
1.3B
2.7B
6.7B
13B
30B
66B
7B
13B
70B
Dense
27.64
14.61
12.46
10.85
10.12
9.56
9.33
5.47
4.88
3.32
SparseGPT 2:4
45.07
29.61
14.90
13.00
11.80
10.53
10.22
8.69
7.07
4.98
SliceGPT (10%)
29.34
15.10
12.75
10.92
10.27
9.65
9.43
5.89
5.21
3.69
SliceGPT (20%)
34.26
16.43
13.73
11.48
10.66
9.87
9.57
6.64
5.81
4.25
SliceGPT (25%)
37.74
17.46
14.56
11.90
10.94
10.04
9.68
7.24
6.30
4.60
SliceGPT (30%)
43.98
19.09
15.83
12.51
11.33
10.27
9.85
8.12
6.99
5.05
Zero-shot Tasks We assess SliceGPT across five well-known zero-shot tasks: PIQA (Bisk et al., 2020); WinoGrande (Sakaguchi et al., 2021); HellaSwag (Zellers et al., 2019); ARC-e and ARC- (Clark et al., 2018) using the LM Evaluation Harness (Gao et al., 2021). Figure shows the results. We marked difference between the datasets, with the Alpaca dataset giving much higher performing models. We attribute this difference to the similarity between Alpaca and the benchmark tasks. For LLAMA-2 70B sliced 30%, with RFT Alpaca accuracy of 74.3%. compared to 76.6% on the dense model. The sliced model has approximately 51.6B parameters and considerably improved throughput and for all models post-pruning without RFT are shown in Appendix A.5.
able to achieve average
demonstrate later. Results for OPT
We that Phi-2 is not able to recover the drop in accuracy from slicing using only the WikiText-2 dataset, but using Alpaca we are able to recover several percentage points. The average accuracy of Phi-2 with 25% slicing and RFT 65.2%. compared to 72.2% with the dense model. The sliced model has approximately 2.2B parameters and retains 90.3% of the accuracy of the 2.8B model. This shows that even small LMs can benefit from post-training pruning. Tables of accuracies across each task are provided in Appendix A.6.
WikiText2 cal. RFT
LLAMA-2 Family
Phi-2
LLAMA-2 Family
Phi-2
80
Alpaca cal. & RFT
80
Mcan Accuracy
Mean Accuracy
70
70
Dense
20% sliced
60
60
25% sliced
50
50
D- 30% sliced
40
7B 13B
70B
2.78B
7B
13B
70B
2.8B
#params in the original model
#params in the original model
Figure 5: Mean zero-shot accuracy on LLAMA-2 and Phi-2 across multiple tasks after slicing and recovery fine-tuning (RFT). Left: WikiText-2 used for calibration and RFT. Right: Alpaca used for calibration and RFT.
8

---

Published as a conference paper at ICLR 2024
Benchmarking Throughput Unlike conventional sparsity methods, which introduce sparsity in Win and Wou.. SliceGPT also introduces (structured) sparsity in X: entire columns of X are sliced off, reducing the embedding dimension. This enhances both the computational complexity (in flops) and data movement within our compressed model.
The token throughput of models sliced 25% and 50% are compared to the dense model on 80GB H100 GPUs. We set the sequence length to 128 and find the maximum throughput by doubling the batch size until the GPUs run out of memory or the throughput drops off. The 25% sliced models achieve up to I x.55x throughput improvement over the dense model. At 50% slicing the largest models require only one GPU instead of two, with large increases in throughput: 3.13x and 1.87 This means that for fixed number of GPUs, these models achieve 6.26 x and 3.75x throughput of a dense model. We note that the WikiText2 perplexity of SliceGPT at 50% is worse than SparseGPT 2:4, but the throughput is much higher than could be achieved with sparse method that does not slice X. For full details see Appendix A.7.
Inference Time Table 2 compares the time of generating a single token in OPT 66B and LLAMA-2 70B models on Quadro RTX6000 and A100 GPUs. We observe 16-17% speedup on RTX6000 GPUs when employing 25% slicing, and 1 11-13% on A100s. We reduce the number of GPUs used in both cases, providing energy and cost savings relative deployment of the dense model. For LLAMA-2 70B, the compute required using RTX6000 GPUs is reduced to 64%, from 1764 GPUms to 1075 GPUms*. We attribute this improvement to our approach of substituting weight matrices with smaller ones in our compressed models, which is infeasible with other pruning schemes.
Table 2: Average per-token inference time of SliceGPT when generating sequences of length 128 (with batch size of I). In each case, we show the time taken in ms, the number of GPUs required and the total compute in GPUms.
GPU Type
Slicing
OPT 66B
LLAMA-2 70B
Dense
114ms on 4 GPUs 456 GPUms
125ms on 4 GPUs|
500 GPUms
A 100 (40GB)
25%
102ms on 3 GPUs
306 GPUms
110ms on GPUs
330 GPUms
Quadro RTX6000
Dense
237ms on 6 GPUs
1422 GPUms
252ms on GPUs
1764 GPUms
(24GB)
25%
204ms on 5 GPUs
1020 GPUms
215ms on GPUs 1075 GPUms
End-to-end performance gains are not feasible with the SparseGPT baseline at the time of writing. Instead, we compare SliceGPT with SparseGPT by comparing the relative timing of each operation involved in a transformer layer. We find that SliceGPT (25%) is competitive with SparseGPT (2:4) in terms of speedup and perplexity for large models. For further details see Appendix A.8.
5 CONCLUSION AND FUTURE WORK
We've introduced SliceGPT which allows for structured pruning for large language models. We reduce the cost of inference of LLAMA-2 70B on 40GB A100 GPUs to 66% of that of the dense model without any additional code optimization, requiring fewer GPUs (from to 3) while maintaining better held-out perplexity than SparseGPT 2:4. On 24GB RTX6000 GPUs, the cost of inference is reduced to 64%, requiring fewer GPUs (from 7 to 5). On zero-shot downstream tasks, slicing OPT 66B, LLAMA-2 70B and Phi-2 at 25% maintains 99%, 96% and 87% of the dense performance. With recovery fine-tuning 25 25%csl LLAMA-2 70B and Phi-2 increase to 99% and 90% respectively.
Opportunities remain to build on method. Smaller but dense LMs perform better than LMs with 13B parameters or less pruned to similar sizes, though we do not expect this to remain the case for long. Our pruned models have more parameters than those pruned with SparseGPT but fit larger batches in GPU memory with no overhead for sparsity structure: perhaps a combined method could obtain the best of both. Other methods of computing Q could improve the results. To further decrease the inference time and GPU count, complementary methods including quantization (Xiao et al., 2023; Dettmers et al., 2022; Ashkboos et al., 2023; Dettmers al., 2023; Frantar et al., 2022), and structural pruning (e.g. Ma et al., 2023b) could be used.
'Our Hugging Face-based testing does not enjoy continuous batching or model sharding. This means that in terms of inference time, the dense-model could be improved more than our sliced model in terms of GPUms. Nonetheless, our measurements do reflect the energy-usage per token in such deployment.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We thank Dmitry Kats, Pashmina Cameron, Pavel Myshkov, Elena pochernina and Liana Mikaelyan for their invaluable contributions to the source code. We additionally thank Pashmina Cameron for her helpful feedback when reviewing early versions of the paper.
REFERENCES
Saleh Ashkboos, Ili Markov, Elias Frantar, Tingxuan Zhong, Xincheng Wang, Jie Ren, Torsten Hoefler, and Dan Alistarh. Towards end-to-end 4-bit inference on generative large language models. arXiv preprint arXiv:2310.0922 2023.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint 1rXiv:1677.06440 2016.
Yonatan Bisk. Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. ArXiv, abs/1 00/188030545, 2018. URL https:/ /api semant ascho aa. .rrg/rrg/ce Co ID 3922816.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. LLM. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.073339 2022.
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashk- boos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized represen- tation for near-lossless LLM weight compression. arXiv preprint arXiv:2306003077 2023.
Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training tratnin quantization and pruning. Advances in Neural Information Processing Systems, 35:4475 4488, 2022.
Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in one-shot. 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint 2...::2210.17222 2022.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks, 2019.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. framework for few-shot language model evaluation. Version vO. 0.1. Sept, 2021.
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. CoRR, a33/2103.13630, 2021. URL https: http//rxvvvv./// org abs 2202..1... 13630.
Manish Gupta and Puneet Agrawal. Compression of deep learning models for text: A survey, 2021.
Song Han, Huizi Mao, and William J.
Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, 2016.
Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning, In IEEE international conference on neural networks, pp. 293-299, IEEE, 1993.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference computer vision, pp. 1389-1397, 2017.
Torsten Hoefler. Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. CoRR, abs/2102.00554, 2021. URL https: //tp:///aiiii www.voorg/aaaoo /bts//a////// 02. 00554.
10

---

Published as a conference paper at ICLR 2024
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Alu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. In Proceedings of the European conference on computer vision (ECCV), pp. 304-320 2018.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang. Shoumeng Yan, and Changshui Zhang. Learn- L ing efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision, pp- 2736-2744, 2017.
Jian- Han Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference computer vision, pp- 5058-5066, 2017.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. arXiv preprint arKiv:2305. 16227, 2023a. URL htt.httss///raaw..orrg//r/// 2305. 11627, pdf.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. L.MM-prune:: On the structural pruning of large language models, 2023b.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers, 2021.
Sourab Mangrulkar, Sylvain Gugger, I Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-arr parameter-efficieet fine-tuning methods. sttps://gittnn com/ htt.g/iggggaee/eeetttt 2022.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint (oXXi:1609.075505 2016.
Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosie, Ganesh Venkatesh. Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. arXiv preprint arXiv:2/04.088 083788 2021.
Matan Ben Noach and Yoav Goldberg, Compressing pre-trained language models by matrix decom- position. In Proceedings of the Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Pro- cessing, pp. 884-889, Suzhou, China, December 2020. Association for Computational Linguistics. URL wwt.htttp///mmm http://aaaaaatoooooa.o// org/ Nafrg/20200 2020 aacl waaecllmainn -main. 88.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, perf-peerormance deep learning library. Advances in neural information processing systems, 32, 2019.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre- training. 2018.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106. 2021.
Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. Advances in Neural Information Processing Systems, 33:18098-18109 2020.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1695, 2023.

---

Published as a conference paper at ICLR 2024
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsumori B. Hashimoto. Stanford alpaca: An inarucctio-followwie llama model. https:: /ttp:/gggtttb/tt// www.eto//////eeeee ttttttttssaaaa// stanford alpaca, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Goodbye SVD, hello robust low-rank approximation. arXiv preprint arXiv:2009.055 050470 2020. Murad Tukan, Alaa Maalouf, Matan Weksler, and Dan Feldman. Compressed deep networks:
Tycho FA van der Ouderaa, Markus Nagel, Mart van Baalen, Yuki M Asano, and Tijmen Blankevoort. The Ilm surgeon. arXiv preprint arXiv:2312.174 17244, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the -hrt natural language processing. arXiv preprint 1rXiv:1910.03771, 2019.
Guangxuan Xiao, Ji Lin, Mickael Seznec. Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099. PMLR, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07888 2019.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre- vre-rra transformer language models. arXiv preprint wrX.h:2205.00666 2022.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv::303. 18223, 2023.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression, 2017.
Xunyu Zhu. Jian Li, Yong Liu, Can Ma. and Weiping Wang. A survey model compression for large language models. arXiv preprint arXiv:2308.07633 2023.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 PROOF OF EQUATION 2
An orthogonal matrix Q is square matrix that satisfies the relation QTQ Q QQ = I. The norm
of vector is the square-root of the sum of squares of the elements: | Σ, x Vx x.
Multiplying vector by Q does not change the norm since |Qælll - x Q Qx ||a]||
The RMSNorm operation divides each row of the input matrix X by its norm. By the basic rules of linear algebra, æ is a row X, then Q x is the corresponding row of XQ. Applying RMSNorm
to XQ, said row will now be equal [[a:] Q x. After RMSnorm, we can multiply by Q
our row
is now equal to
22 QQ
T a. Thus we have the relation
RMSNorm(XQJQ RMSNorm(X).
(10)
A.2 SINGLE PRECISION EIGENVALUE CALCULATION
As previously noted in Section 4, we employ double precision when performing the PCA algorithm. This choice is made order to mitigate potential numerical errors that may arise during the computa- tion of the orthogonal matrix in SliceGPT. Nevertheless, it is intriguing investigate the impact of employing lower precision for PCA calculations on the ultimate accuracy.
Table 3 shows the perplexity of all our models when apply FP32 PCA in our scheme. It shows that the accuracy of larger models could be affected by numerical errors during the PCA calculation PyTorch (torch. linalg) for calculating the eigenvectors
phase. It should be noted that and eigenvalues.
Table 3: OPT and LLAMA-2 perplexity results on WikiText2 using FP32 PCA calculation. The calibration set size and sequence length are 128 and 2048, respectively.
OPT
LLAMA-2
Method
125M
1.3B
2.7B
6.7B
13B
30B
66B
7B
13B
70B
Dense
27.64
14.61
12.46
10.85
10.12
9.56
9.33
5.47
4.88
3.32
SparseGPT 2:4
45.07
29.61
14.90
13.00
11.80
10.53
10.22
8.69
7.07
4.98
SliceGPT 10%
29.48
15.15
12.83
11.05
10.28
9.68
9.45
6.51
5.64
4.20
SliceGPT 20%
34.12
16.51
13.87
11.64
10.73
9.94
9.80
7.30
6.07
5.82
SliceGPT 25%
38.25
17.67
14.78
12.14
11.08
10.15
9.81
8.52
6.65
7.01
SliceGPT 30%
44.17
19.33
16.20
12.82
11.53
10.43
9.99
10.41
7.49
8.75
A.3 SENSITIVITY THE CALIBRATION SET SIZE AND SEQUENCE LENGTH
We present an ablation study examine the role of the WikiText-2 calibration set. We focus on the generation task with 25% sparsity using OPT 6.7B and LLAMA-2 models.
WikiText2 PPL
14
12
OPT 6.7B
LLAMA-2 7B
8
16
32
64
128
256
512
1024
128
256
512
1024
2048
4096
Calibration set
Calibration sequence length
Figure 6: The effect of the calibration size and sequence length perplexity of WikiText2.
13

---

Published as a conference paper at ICLR 2024
Figure 6 (left) shows the result of varying the size of the calibration the perplexity. It shows that sample sizes at least 128 provide sensible choices for our calibration set.
Next explore the effect of using different sequence lengths N in the calibration set. Given fixed number of samples, the PCA imput matrix computed using B embedding vectors, and understanding the tradeoff between having larger B or larger N is interesting. Figure 6 (right) shows the results of varying the sequence length in the calibration set from 128 to 4096: we conclude that having larger sequence length can result in better perplexity.
Using these insights, we use a calibration set size of 1024 and sequence length 2048 in our main experiments (Table 1). In Table 4 below evaluate the perplexity of OPT and LLAMA-2 models WikiText-2 with smaller calibration set size, which confirms the trend that decreasing this degrades the perplexity across all models and sizes.
Table OPT and LLAMA-2 perplexity results on WikiText2, The calibration set size and sequence length are 128 and 2048, respectively.
OPT
LLAMA-2
Method
125M
1.3B
2.7B
6.7B
13B
30B
66B
7B
13B
70B
Dense
27.64
14.61
12.46
10.85
10.12
9.56
9.33
5.47
4.88
3.32
SparseGPT 2:4
45.07
29.61
14.90
13.00
11.80
10.53
10.22
8.69
7.07
4.98
SliceGPT (10%)
29.33
15.15
12.82
11.00
10.30
9.66
9.43
5.96
5.29
3.78
SliceGPT (20%)
34.53
16.58
13.89
11.62
10.75
9.91
9.61
6.86
6.04
4.46
SliceGPT (25%)
38.13
17.78
14.84
12.12
11.08
10.10
9.76
7.56
6.61
4.89
SliceGPT (30%)
44.61
19.61
16.30
12.81
11.55
10.32
9.95
8.64
7.44
5.42
A.4 SPECTRUM ANALYSIS OF LLAMA-2 AND OPT MODELS
The figure below shows the eigenvalue distribution for the OPT 6.7B and LLAMA-2 7B models. Although both models have comparable parameter count, the LLAMA-2 model has more tightly compressed distribution in its embeddings spectrum. This observation shows that there are no dominant principal components with significantly more information, making the pruning of these components more challenging task.
Nomalizee Spectrum of the MLP Input.
OPT (6.78)
LLAMA-2 (7B)
10"
10"
1
10"
10"
2
6
10
12
16
18
20
22
24
26
28
30
0
2
6
10
12
14
16
18
20
22
24
26
28
30
Num.
Layer Num.
Figure 7: Normalized (by maximum) spectrum of the MLP inputs (log scale) using 64 samples. Except for the first layer in the LLAMA-2 model, the eigenvalue distributions for both models show faster decay in early layers compared to later ones. This suggests that greater amount of slicing could be applied after the orthogonal transformation these early layers.
We can use the above insights to slice different layers by different amounts. Instead of specifying the slicing level upfront, we set the fraction of the total variance to discard during each PCA calculation, which sets the number of rows and columns to slice off from each matrix. For each model, three experiments with varying target variances to obtain total reduction on the network close to 25%.
14

---

Published as a conference paper at ICLR 2024
The results shown in Table below. Varying the slicing level by layer improves the WikiText-2 perplexity OPT models, but has the opposite effect in LLAMA-2 models.
Table 5: Evaluating the effects of varying slicing level by layer. The calibration set size is 128 and the sequence length is the maximum for each model.
WikiText-2 PPL
WikiText-2 PPL
Model
Improvement
(25% constant slicing)
(varying slicing layer)
OPT 6.7B
12.10
11.94, 24.7% total slicing
0.16
OPT 13B
11.04
10.76, 24.2% total slicing
0.28
OPT 30B
10.13
9.95, 24.8% total slicing
0.18
OPT 66B
9.75
9.63, 24.1 % total slicing
0.12
LLAMA-2 7B
6.84
7.63, 24.1% total slicing
-0.79
LLAMA-2 13B
6.00
6.17, 23.3% total slicing
-0.17
LLAMA-2 70B
4.44
4.63, 25.5% total slicing
-0.19
A.5 ZERO-SHOT ACCURACY ABLATION OVER CALIBRATION DATASET
Figure shows the average scores achieved by the sliced models across the zero-shot tasks. The top row of the plot shows the mean accuracy when WikiText-2 used for calibration, and the bottom row shows the accuracy when Alpaca is used for calibration. We observe a similar pattern to the generation task the results: the OPT models are more amenable to compression than the LLAMA-22 models, and the reduction in accuracy is less pronounced in the larger models. Here we also include the Phi-2 model: we see that sliced versions the Phi-2 model are comparable with sliced versions of the LLAMA-2 7B model. The largest OPT and LLAMA-2 models can be compressed very effectively, with just a few percentage points loss when removing 30% of the 66B OPT model.
OPT Family
LLAMA-2 Family
Phi-2
80
WikiText-2 calib.
Mcan Accuracy
70
Dense
20% sliced
60
25% sliced
30% sliced
30
Mcan Accuracy
Alpaca calib.
70
60
50
40
1.3B
2.7B
6.7B
13B
30B
66B
7B
13B
70B
2.8B
#params in original model
#params the original model
Figure 8: Mean zero-shot accuracy on OPT, LLAMA-2 and Phi-2 across multiple tasks after slicing with the WikiText-2 (top) and Alpaca (bottom) datasets for calibration.
Recovery fine-tuning (RFT) can be applied LLAMA-2 and Phi-2 models to improve their performance further (Figure main text). Despite an extensive search, we were not able find RFT parameters that enabled improved performance in the OPT models.
15

---

Published as a conference paper at ICLR 2024
A.6 DETAILED ZERO-SHOP RESULTS
In this section, we provide the detailed results of the zero-shot tasks we presented in the paper.
Table 6: Downstream zero-shot task performance of OPT, LLAMA-2 and Phi-2 models when slicing
using the WikiText2 dataset.
Model
Slicing
PIQA WinoGrande
HellaSwag
ARC-e
ARC-e
Avg. Score
Dense
72.42
59.27
53.72
50.97
29.52
53.18
20%
65.34
54.85
45.39
46.04
26.96
47.72
OPT 1.3B
25%
62.30
53.83
42.91
45.45
27.22
46.34
30%
60.77
54.70
39.81
43.90
25.77
44.99
Dense
74.81
61.01
60.58
54.42
31.14
56,39
20%
68.23
57.93
51.38
51.81
28.50
51.57
OPT 2.7B
25%
65.29
57.22
47.85
49.79
27.99
49.63
30%
62.35
57.22
44.18
46.72
27.05
47.50
Dense
76.39
65.19
67.16
60.14
34.64
60.70
20%
72.74
61.09
61.04
55.89
30.80
56.31
OPT 6.7B
25%
70.35
60.62
58.15
52.78
29.52
54.28
30%
68.61
60.69
54.56
$2.15
29.01
53.00
Dense
76.82
64.80
69.81
61.87
35.67
61.79
20%
74,48
64.96
65.42
60.90
35.24
60.20
OPT 13B
25%
73.67
64.25
63.28
60.52
34.64
59.27
30%
71.82
62.90
60.66
58.80
32.94
57.42
Dense
78.07
68.19
72.27
65.24
38.23
64.40
20%
76.50
66.61
70.61
64.18
35.75
62.73
OPT 30B
25%
75.30
66.61
69.42
63.55
35.67
62.11
30%
74.97
65.04
68.15
63.55
34.64
61.27
Dense
79.82
68.90
74.85
67.21
40.02
66.16
20%
78.73
67.88
73.79
68.81
39.51
65.74
OPT 66B
25%
78.40
67.09
73.33
67.89
39.16
65.17
30%
77.42
66.30
72.62
66.90
37.97
64.24
Dense
79.11
69.06
75.99
74.58
46.25
69.00
20%
69.42
65.
59.04
59.76
37.54
58.18
LLAMA-2 7B
25%
66.87
63.38
$4.16
58.46
34.56
55.48
30%
63.55
61.33
49.62
51.77
31.23
51.50
Dense
80.47
72.22
79.39
77.48
49.23
71.76
20%
71.87
69.38
63.04
69.87
43.09
63.45
LLAMA-2 13B
25%
68.55
67.48
58.10
62.50
37.88
58,90
30%
66.10
65.11
52.69
$6.82
35.07
55.16
Dense
82.70
77.98
83.84
80.98
57.34
76.57
20%
76.61
76.40
72.98
80.51
55.20
72.34
LLAMA-2 70B
25%
74.92
75.37
68.84
77.90
51.71
69.75
30%
72.31
73.56
63.69
73.40
47.61
66.11
Dense
79.11
75.77
73.83
78.32
54.18
72.24
20%
71.87
67.80
57.76
$8.00
35.32
58.15
Phi-2
25%
69.21
65.35
$2.40
$3.70
31.66
$4.46
30%
65.94
63.14
47.56
$3.03
30.29
$1.99
16

---

Published as a conference paper at ICLR 2024
Table Downstream zero-shot task performance of OPT, LLAMA-2 and Phi-2 models when slicing using the Alpaca dataset.
Model
Slicing
PIQA
WinoGrande
HellaSwag
ARC-e
ARC-c
Avg. Score
Dense
72.42
59.27
53.72
50.97
29.52
53.18
20%
69.91
55.49
47.88
49.66
27.05
50.00
OPT 1.3B
25%
69.37
55.72
45.82
48.70
26.62
49.25
30%
68.55
55.33
43.92
47.26
26.45
48.30
Dense
74.81
61.01
60.58
54.42
31.14
56.39
20%
71.87
58.09
54.98
54.04
29.44
53.68
OPT 2.7B
25%
70.95
58.09
52.62
53.03
29.61
52.86
30%
69.64
56.43
49.45
51.81
28.33
51.13
Dense
76.39
65.19
67.16
60.14
34.64
60.70
20%
74.54
62.67
62.84
59.18
33.36
58.52
OPT 6.7B
25%
73.78
62.59
60.99
$9.01
33.70
58,01
30%
73.34
61.80
58.93
58.33
32.85
$7.05
Dense
76.82
64.80
69.81
61.87
35.67
61.79
20%
76.01
65.19
66.15
61.57
34.73
60.73
OPT 13B
25%
74.65
64.64
65.02
60.65
35.07
60.00
30%
74.86
63.46
63.16
61.36
34.56
59.48
Dense
78.07
68.19
72.27
65.24
38.23
64.40
20%
78.35
66.61
70.64
65.19
37.46
63.65
OPT 30B
25%
77.48
65.82
69.58
65.91
37.63
63.28
30%
76.93
64.96
68.66
65.70
37.12
62.67
Dense
79.82
68.90
74.85
67.21
40.02
66.16
20%
79.49
68.19
73.69
67.26
39.25
65.58
OPT 66B
25%
79.11
68.35
73.30
67.00
38.74
65.30
30%
79.05
68.75
72.62
66.29
38.31
65.00
Dense
79.11
69.06
75.99
74.58
46.25
69.00
20%
76.50
65.51
65.20
69.99
41.21
63.68
LLAMA-2 7B
25%
74.21
64.01
60.55
66.88
38.91
60.91
30%
72.25
59.83
55.86
63.93
37.80
$7.93
Dense
80.47
72.22
79.39
77.48
49.23
71.76
20%
77.97
68.90
69.64
74.71
45.99
67.44
LLAMA-2 13B
25%
76.88
67.40
65.85
72.52
44.54
65.44
30%
74.10
65.82
60.91
68.43
42.41
62.34
Dense
82.70
77.98
83.84
80.98
57.34
76.57
20%
81.99
76.87
78.93
80.26
$4.10
74.43
LLAMA-2 70B
25%
80.69
77.98
76.97
79.67
52.65
73.59
30%
79.33
77.27
73.11
77.44
51.19
71.67
Dense
79.11
75.77
73.83
78.32
54.18
72.24
20%
76.17
68.75
61.95
72.18
45.48
64.90
Phi-2
25%
75.68
64.88
58.19
70.41
43.43
62.52
30%
74.05
62.12
53.31
67.26
39.42
63.47
17

---

Published as a conference paper at ICLR 2024
Table 8: Downstream zero-shot task performance of LLAMA-2 and Phi-2 models when slicing and recovery fine-tuning using the Wiki'eet22 dataset.
Model
Slicing
PIQA
WinoGrande
HellaSwag
ARC-e
ARC-c
Avg. Score
Dense
79.11
69.06
75.99
74.58
46.25
69.00
20%
69.86
64.72
61.07
54.25
36.43
57.27
LLAMA-2 7B
25%
69.26
64.96
58.65
52.36
35.75
56,20
30%
67.41
63.22
55.65
50.76
34.13
54.23
Dense
80.47
72.22
79.39
77.48
49.23
71.76
20%
74.10
68.51
66.94
70.54
43.77
64.77
LLAMA-2 13B
25%
71.27
68.98
64.12
63.76
40.87
61.80
30%
69.64
66.85
59.93
59.55
38.65
58.93
Dense
82.70
77.98
83.84
80.98
57.34
76.57
20%
77.86
76.16
72.91
81.27
55.89
72.82
LLAMA-2 70B
25%
76.71
73.72
71.41
79.88
54.69
71.28
30%
75.14
73.56
69.91
74.79
51.71
69.02
Dense
79.11
75.77
73.83
78.32
54.18
72.24
20%
71.27
67.17
54.86
56.61
38.91
57.76
Phi-2
25%
69.91
65.19
52.48
$2.78
35.49
55.17
30%
66.16
63.54
49.72
46.38
32.68
51.70
Table 9: Downstream zero-shot task performance of LLAMA-2 and Phi-2 models when slicing and recovery fine-tuning using the Alpaca dataset.
Model
Slicing
PIQA
WinoGrande
HellaSwag
ARC-e
ARC-c
Avg. Score
Dense
79.11
69.06
75.99
74.58
46.25
69.00
20%
76.55
65.59
68.26
71.84
45.05
65.46
LLAMA-2 7B
25%
75.79
63.22
65.12
68.22
42.83
63.04
30%
74.59
61.64
63.06
66.54
40.87
61.34
Dense
80.47
72.22
79.39
77.48
49.23
71.76
20%
79.27
68.27
73.21
74.37
49.83
68.99
LLAMA-2 13B
25%
78.84
67.64
71.21
73.57
49,66
68.18
30%
76.11
68.03
68.58
71.42
47.10
66.35
Dense
82.70
77.98
83.84
80.98
57.34
76.57
20%
81.94
77.74
79.39
81.57
58.45
75.82
LLAMA-2 70B
25%
81.88
77.11
79.04
81.36
58.70
75.62
30%
80.30
75.85
77.13
80.05
58.19
74.30
Dense
79.11
75.77
73.83
78.32
$4.18
72.24
20%
77.42
72.14
65.33
74.20
49.91
67.80
Phi-2
25%
76.17
68.75
63.39
70.45
47.44
65,24
30%
75.24
65.59
60.10
70.16
46.25
63.47
18

---

Published as a conference paper at ICLR 2024
A.7 BENCHMARKING THROUGHPUT EXPERIMENT
Table 10: Benchmarking throughput for OPT and LLAMA-2 models 80GB H100 GPUs. We the sequence length to 128 and find the maximum throughput by doubling the batch size until the GPUs run out of memory the throughput drops off.
Model
Slicing
GPUs
Batchsize
Tokens/s
Dense
1
512
2518
OPT 13B
25%
I
512
2846 (1.13x)
50%
512
3071 (1.22x)
Dense
2
16
141
OPT
25%
2
152 (1.08x)
50%
1
32
441 (6.26x)
Dense
1
512
2707
LLAMA-2 13B
25%
1
512
2878 (1.06x)
50%
1
512
3122 (1.15x)
Dense
2
128
541
LLAMA-2 70B
25
2
256
839 (1.55x)
50%
128
1014 (3.75x)
A.8 BENCHMARKING INFERENCE TIME OF SLICEGPT AGAINST SPARSEGPT
We the CuSparseLT 0.5 library to sparse matrix multiplications replicating the size the matrix-mane multiplications in three different-sizer LLAMA-2 models. We used PyTorch (which also straightforward dense matmul, but smaller). We chose a sequence length of 2048, and took the matrix sizes from the HuggingFace config files. We took the median runtime over 10 attempts.
80 GB A100 GPU,
similar matrix multiplications for the dense equivalent, and for SliceGPT
Each LLAMA-2 layer requires a gated FFN with one up projection, one down projection, and a gated projection. In attention, the architecture of the model means that the query matrix multiplication is different size the key and value matrix multiplications. The following table shows the time taken in ms to run each matrix multiplication in the model, plus a "total" time and relative speedup.
Table 11: Results timing the matrix multiplications involved in each layer of LLAMA-2 models. For larger models, SliceGPT (25%) gives the same speedup as SparseGPT 2:4 but with better WikiTexx-2 perplexity. For smaller models SparseGPT 2:4 provides better speedup albeit at worse perplexity. Slicing 50% trades off perplexity for even greater speedups.
Operation (ms)
Total in ms (speedup)
Model
Method
PPL
Down Proj
Up/Gate Proj
K.V
Out
Dense
5.47
0.89
0.87
0.34
0.34
0.34
3.99
SparseGPT 2:4
8.69
0.56
0.61
0.23
0.23
0.23
2.70 (1.48x)
LLAMA-2 7B
SliceGPT (25%)
0.67
0.64
0.26
0.25
2.99 (1.33x)
SliceGPT (50%)
17.17
0.46
0.44
0.18
0.18
0.18
2.06 (1.9x)
Dense
4.88
1.29
1.28
0.52
0.52
0.52
5.93
SparseGPT 2:4
7.07
0.81
0.95
0.31
0.31
0.31
3.95 (1.50x)
LLAMA-2 13B
SliceGPT (25%)
6.30
1,03
0.98
0.39
0.39
0.41
4.57 (1.30x)
SliceGPT (50%)
13.71
0.68
0.67
0.26
0.27
0.30
3.11 (1.91x)
Dense
3.32
4.63
4.27
0.21
1.27
1.27
16.13
SparseGPT 2:4
4.98
2.87
3.69
0.14
0.84
12.20 (1.32x)
LLAMA-2 70B
SliceGPT (25%)
3.4
3.26
0.16
0.96
1.00
12.20 (1.32x)
SliceGPT (50%)
8.86
2,28
2.34
0.15
0.69
0.68
8.63 (L.87))
We also benchmarked the OPT architecture in the same way. In this case, the matrix multiplications associated with Key, Value, Query and Out are all the same size, and there are just two matrix multiplications in the MIP section (FC1 and FC2).
19

---

Published as a conference paper at ICLR 2024
Table 12: Results of timing the matrix multiplications involved in each layer of OPT models. For larger models, SliceGPT (25%) gives slightly better speedup than SparseGPT 2:4, and with better WikiText-2 perplexity. For smaller models SparseGPT 2:4 provides better speedup albeit worse perplexity. Slicing at 50% trades off perplexity for even greater speedups.
Operation (ms)
Total in ms (speedup)
Model
Method
PPL
FC2
FC1
K.V.Q.Out
Dense
10.12
1.89
1.89
0.52
7.75
SparseGPT 2:4
11.80
1.18
1.50
0.31
5.42 (1.43x)
OPT 13B
SliceGPT (25%)
10.94
1.50
1.45
0.38
5.92 (1.31x)
SliceGPT (50%)
15.39
0.96
0.99
0.26
3.98 (1.9x)
Dense
9.56
10.29
1.28
5,93
SparseGPT 2:4
10.53
0.81
0.95
0.31
3.95 (1.00x
OPT
SliceGPT (25%)
10.04
1.03
0.98
0.39
4,55 (1,30x)
SliceGPT (50%)
12.47
0.68
0.67
0.26
3,06 (1.94x)
Dense
9.33
4.63
4.27
0.21
14.01
SparseGPT 2:4
10.22
2.87
3.69
0.14
10.81 (1.30x)
OPT 66B
SliceGPT (25%)
3,40
0.16
10.56 (1.33x)
SliceGPT (50%)
11.39
2.28
2.34
0.15
7.56 (185x)
A.9 RECOVERY FINE-TUNING COST
All LLAMA-2, OPT and Phi-2 models can be sliced on a single GPU in 1 to 3 hours. With recovery fine-tuning compress all LMs in I to hours total, as shown in Table 13.
Table 13:
Compute cost of slicing 30% with SliceGPT and performing recovery fine-tuning using calibration set size of 1024 for LLAMA-2 models and 2048 for Phi-2, and calibration sequence length 2048 in all cases.
Alpaca dataset. Here
SliceGPT 30%
Recovery finc-tuning
Model
Total
Time
GPUs
Time
GPUs
LLAMA-2 7B
0h44m
1xH100 80GB
0h23m
1xH100 80GB
1h07m
LLAMA-2 13B
1h08m
1xH100 80GB
0h44m
1xH100
80GB
1h52m
LLAMA-2 70B
3h31m
1xH100
80GB
1h35m
40
80GB
5h06m
Phi-2
0h49m
IxV100 32GB
1h59m
1xV100 32GB
2h48m
20


Reviews:
[{"review_id": "wzanu4qE6s", "reviewer": "Reviewer_7EAi", "summary": "The paper proposes to idea of using computation invariance for row-/column-wise sparsification. The authors leverages the idea of pre- and post-multiplying each block in a transformer model by orthogonal matrices that warrants computational invariance of each block. On the surface, adding new operations increases the raw FLOPs. However, following this technique, the authors show that they can sparsify most of operations in a transformer models, including attention and FFN layers.", "questions": "I think if the authors could clarify the following questions/comments and include few additional results, the quality of the paper could significantly increase:\n\n(Q1) Show latency comparison across different baselines, (a) Dense, (b) SliceGPT, (c) SparseGPT. \n\n(Q2) I may have missed this in the paper, but can you please clarify how you decide on row/column sparsity and how you select them? If the sparsed rows/columns are spread across the matrix, how do you manage to do the multiplication while getting latency benefits? or the overall benefits are derived from memory savings?\n\n(Q3) Do you have any insights as which operation/layer is more sensitive to sparsification? Have you thought of not uniformly sparsifying all the layers? Can looking into the range of values in the weight matrices provide insights on how to apply the sparsificiation (both degree and pattern)?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "$\\mathtt{+}$ The idea of computational invariance and re-purposing additional computation for higher opportunity for sparsification is interesting and warrants further investigation.\n\n$\\mathtt{+}$ The results are promising and show the benefits across a range of SOTA models. The comparison with SparseGPT technique is also valuable.", "weaknesses": "$\\mathtt{-}$ The paper lacks sufficient insights of how the rows and columns are sparsified. It was not clear whether some operations are friendlier to row vs. column sparsification or this is a byproduct of the computational invariance approach.\n\n$\\mathtt{-}$ The paper compares accuracy with 2:4 structured sparsity but does not provide head-to-head comparison with SparseGPT (2:4) in terms of latency. \n\n$\\mathtt{-}$ One of the premises of the paper is memory saving, but going through the results it is not clear how the memory savings are in comparison to 2:4 sparsity. Showing a trade-off possibly can clarify this point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NbEcY7ak4n", "reviewer": "Reviewer_gLsa", "summary": "This paper introduces SliceGPT - a new approach for compressing large language models. By deleting rows and columns based on computational invariance, SliceGPT can significantly reduce the computation and memory required for inference while maintaining high accuracy. The evaluation demonstrates that this method is effective for large models such as OPT-66B and Llama-70B.", "questions": "1. In Table 2, it is not fair to multiply the number of GPUs by the total latency and get \"GPUms\". Huggingface Transformers implements naive model parallelism (or device placement, or pipeline parallelism without pipelining) method to parallelize the models, which means that only one GPU is active at a time. A correct implementation of tensor parallelism or pipeline parallelism will give different results. Considering this, the latency speedup is less impressive.\n2. Give the same parameter count budget or inference latency budget, how does this method compare to quantization-based method?\n3. The \"computational invariance\" trick is similar to a trick in SmoothQuant[1] (equation 3). Both of them multiplicate some matrices between the X and W, so it is good to do some comparison here.\n\n[1] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A novel method of compression based on computational invariance\n- No special code is required to run the compressed models and achieve speedup and memory savings\n- Works for Llama-70B and OPT-66B\n- It is well-written and easy to follow", "weaknesses": "- The accuracy loss is not \"negligible\". With 25% sparsity, the perplexity of Llama-2-70B on WikiText2 increases from 3.32 to 4.89, which is similar to a dense Llama-2-13B. However, a 25% sparse Llama-2-70B has much more parameters than a dense Llama-2-13B.\n- The speedup is not impressive.\n- Compared to the quantization-based method, there is no advantage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lOwJWQiDrC", "reviewer": "Reviewer_96nz", "summary": "The authors propose a technique for pruning neurons in Tranformer architectures based on a clever application of orthogonal matrices, which enables PCA-based elimination of rows and columns throughout the architecture. The authors evaluate their technique on a range of large language models from the OPT and Llama-2 families and demonstrate improvements in inference runtime on GPUs.", "questions": "I have no additional question.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper was very well written and organized. I found the method easy to understand. The insights that underpin the method (e.g., invariance to repeated application of orthogonal matrices) are clever and I think the PCA-based pruning of rows and columns in weight matrices is nicely grounded relative to other neuron pruning techniques.", "weaknesses": "I think there are two main weaknesses in this paper. First, the authors don’t acknowledge prior work on neuron pruning. Admittedly most of the papers that I’m aware of on this topic focus on convolutional neuron networks. But, some of the methods are likely to provide a reasonable baseline for the proposed technique. I’ve cited some potentially relevant papers below [1, 2, 3, 4, 5].\n\nSecond, the results in Table 1 suggest to me that 2:4 sparsity is preferable to the proposed technique? If I understand correctly, 2:4 will remove 50% of the weights in the model and the results in Table 1 show that it suffers less quality degradation than removing 30% of the parameters with SliceGPT. Based on this, I expect 2:4 sparsity would show larger inference runtime savings for a given quality than the results in Table 2.\n\n[1] https://arxiv.org/abs/1708.06519\n\n[2] https://arxiv.org/abs/1707.06342\n\n[3] https://arxiv.org/abs/1707.01213\n\n[4] https://arxiv.org/abs/1707.06168\n\n[5] https://arxiv.org/abs/1810.05270", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JC2HR8LnSv", "reviewer": "Reviewer_fXjP", "summary": "This paper introduces SliceGPT, a method to reduce the size of matrices for inference of LLMs. The method uses orthogonal matrices to project to a lower-dimensional space the weight matrices, these orthogonal matrices being constructed using PCA.", "questions": "- Do the authors plan to release the code? I think open sourcing it is very important for the community.\n- The latex is broken, citations are not redirecting, I think your should recompile the pdf.\n- Could the authors comment on the use of a random projection (which is orthogonal in expectation, as in sketching methods) compared to $Q_\\ell$ computed using by PCA, which is more expensive?\n- In practice, not all layers may be equivalent signal-wise, could the authors comment on the possible use of weight watcher ( https://github.com/CalculatedContent/WeightWatcher ) to analyze how to select a different projection dimension for each layer? This question is purely curiosity but I think, combining both SliceGPT and weight watcher could greatly improve the method.\n- p8: what do the authors mean by \"using dense kernels in our compressed models\"? Did they code specific kernels for SliceGPT?\n- I think the authors should write a small proof of Equation (2) to increase the readability of the paper. Can the authors provide it in their answer?\n- \"Theorem\" is too strong for Theorem 1, I suggest \"Proposition\" or \"Remark\".\n- p4: typo: OBC instead of OBS.\n\nOverall I liked the paper and the method, and satisfying answers to my questions and weaknesses would make me consider increase my score.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and pleasant to follow. Ideas are simply explained and figures are helping the understanding. \n- Experimental results are convincing.\n- I think this method could be really used in practice to reduce inference time.", "weaknesses": "- I think section \"layernorm transformers can be converted to RMSnorm\" is not well motivated. Could the authors explain more in details the subtleties of this section and why it was written? I may have missed the point.\n- I'll wait for other reviewers weaknesses to see whether I agree with them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "51aEAegduG", "reviewer": "Reviewer_jAXk", "summary": "The authors of this paper describe their methodology as a transformation of a Transformer network from LayerNorm to RMSNorm. They implement an approach involving the application of orthogonal-matrix transformations and the selective removal of columns and rows from the transformed weight matrices. This process is aimed at reducing the overall model size while preserving performance integrity. The results of their research demonstrate a significant improvement in perplexity on benchmark datasets, OPT and Llmas2, aligning with the 2:4 scheme and underscoring the substantial enhancement in model efficiency and accuracy.", "questions": "* Can you show the performance (perplexity, inference time) of SliceGPT at 50% sparsity?\n* Can you show the inference time of SparseGPT in comparsion to SliceGPT under the same experimental setup?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The formulation is clear and enhanced by illuminating diagrams for better comprehension.\n* The experimental results illustrate the method's effectiveness, establishing a well-defined trade-off between accuracy and sparsity.", "weaknesses": "The experimental section has certain shortcomings:\n\n1. The experiment section does not comprehensively address the comparison between SliceGPT and SparseGPT. While 2:4 sparsity implies a 50% compression rate, Table 1 exclusively showcases SliceGPT with up to 30% compression. This limitation hinders a clear conclusion regarding the superior performance of SliceGPT over SparseGPT.\n\n2. The absence of inference time data for SparseGPT in the experiments makes it challenging to convincingly demonstrate the superior efficiency of SliceGPT.\n\n3. The paper lacks a comparative analysis with state-of-the-art pruning methods such as low-rank approximation, unstructured sparsity, and block sparsity. The omission of these comparisons limits the paper's ability to establish the competitiveness of SliceGPT within the broader context of pruning techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)