Olica: Efficient Structured Pruning of Large Language Models without Retraining
Authors: Jiujun He
Huazhen Lin
Venue: ICML 2025 poster (2025)
Keywords: Model compression
structured pruning
large language models
principle component analysis
Primary area: deep_learning->large_language_models
TL;DR: We propose an efficient method for structured pruning of large language models that does not require retraining.

Abstract: Most existing structured pruning methods for  Large Language Models (LLMs) require substantial computational  and data resources for retraining to reestablish the corrupted correlations, making them prohibitively expensive. To address this, we propose an efficient pruning framework for LLMs called Orthogonal Neuron Decomposition and Linear Calibration  (Olica),  which eliminates the need for retraining.  A key observation is that the multi-head attention (MHA) layer depends on two types of matrix products (i.e., ${\rm W}_q{\rm W}^{\top}_k$ and ${\rm W}_v{\rm W}^{\top}_o$). By treating these matrix products as unified entities and applying principal component analysis (PCA), we extract the most important information to compress LLMs without sacrificing accuracy or disrupting their original structure. Consequently, retraining becomes unnecessary. Moreover, a fast decomposition method is  devised, reducing the  complexity of PCA by a factor of the square of the number of attention heads. Additionally,  to mitigate error accumulation problem caused by pruning the feed-forward network (FFN) layer, we introduce a linear calibration  method to reconstruct the residual errors of a pruned layer using two low-rank matrices. By leveraging singular value decomposition (SVD) on the solution of the least-squares problem, these matrices are obtained without requiring  retraining.  Extensive experiments show that the proposed Olica is efficient in terms of data usage, GPU memory, and running time, while delivering superior performance across multiple benchmarks.

Olica: Efficient Structured Pruning of Large Language Models without Retraining
Jiujun He Huazhen Lin"
Abstract
Table I. We compare the resource consumption of different prun- ing methods on the LLaMA-7B model, focusing on the number of data usage, peak GPU memory consumption, and the runtime quired for pruning (or retraining). The performance of the pruned model is evaluated based on perplexity (PPL.) on the WikiText- 2 dataset and accuracy averaged across the following datasets: BoolQ, PIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, and OBQA. "SR" indicates the sparsity ratio of the pruned model.
Most existing structured pruning methods for Large Language Models (LLMs) require substan- tial computational and data resources for retrain- ing to reestablish the corrupted correlations, mak- ing them prohibitively expensive. To address this, we propose a pruning framework for L.MSs called Orthogonal decomposition and linear calibration (Olica), which eliminates the need for retraining. A key observation is that the multi-head atten- tion (MHA) layer depends products (i.e., W, W & and W. W. ). By treat- ing these matrix products as unified entities and applying principal component analysis (PCA), we extract the most important information to com- press LLMs without sacrificing accuracy or dis- rupting their original structure. Consequently, re- training becomes unnecessary. A fast decomposi- tion method is devised, reducing the complexity of PCA by a factor of the square of the num- ber of attention heads. Additionally, to mitigate error accumulation problem caused by pruning the feed-rd network (FFN) layer, we intro- duce linear calibration method to reconstruct the residual errors of pruned layerS using low-rank matrices. By leveraging singular value decompo- sition (SVD) on the solution of the least-squares problem, these matrices are obtained without re- quiring retraining. Extensive experiments show that the proposed Olica efficient terms of data usage, GPU memory, and running time, while delivering superior performance across multiple benchmarks.
SR: 25%
SR: 33%
Method Sam. Time Mem. PPL (1) Acc. (f) PPL (L) Acc.
LLM-Pruner
50K
3h 30GB
20.57
58.67
24.50
55.39
types of matrix
SlimGPT
50K
1h 20GB
18.45
62.45
22.43
61.41
Olica (Ours) 256
7min 3GB
16.69
63,53
19.83
61.21
1. Introduction
Since the introduction of transformer architecture (Vaswani et al., 2017), the field of natural language processing has witnessed a surge of unsupervised pre-training models, such as BERT (Devlin al., 2019) and GPT series (Brown et al., 2020). Following the scaling law (Kaplan al., 2020), the model parameters have expanded from hundreds of millions to hundreds of billions (Chowdhery et al., 2023; Touvron et al., 2023a; OpenAI, 2024), earning them the label of Large Language Models (LLMs). Despite emerging abil- ities such as in-context leaning and instruction following (Wei et al., 2022a) with increasing scale, the sheer size of LLMS makes their deployment and inference on edge de- vices highly challenging. To address this, techniques such as network pruning (Han al., 2015; Frankle & Carbin, 2019), knowledge distillation (Hinton et al., 2015; Sun et al., 2019), and quantization (Frantar et al., 2023; Dettmers et al., 2024) have been proposed.
In this work, we focus mainly on network pruning, which aims to eliminate redundant parameters in neural networks while preserving their performance. Before the of LLMs, network pruning approaches primarily relied on Taylor ex- pansion of the loss function or regularization techniques. For instance, Optimal Brain Damage (LeCun et al., 1989), Optimal Brain Surgeon (Hassibi & Stork, 1992), and later works (Liu al., 2021; Kwon et al., 2022) assess the the
'Center of Statistical Research, School of Statistics and Data Science, and New Cornerstone Science Laboratory, Southwestern University of Finance and Economics, Chengdu, China. Corre- spondence to: Huazhen Lin swa.ii@uswcc......n....
Proceedings of the 4 International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Olica: Efficient Structured Pruning of Large Language Models without Retraining
importance of parameters using the Hessian matrix of the objective function. Despite their success, it is prohibitively expensive to acquire the gradient of full model parame- ters for LLMs. On the other hand, regularization bbased approaches (Han et al., 2015; Wen et al., 2016; Tan et al., 2024) impose ( or l2 penalties on the parameters of neu- ral networks, thereby pruning those with relatively smaller magnitudes. However, the pre-training process of LLMs is so resource-intensive that developers often avoid imposing structured penalties. Genalty-based pruning methods have become less effective in the context of LLMss, as evidenced by empirical results shown in (Frantar& & Alistarh, 2023).
the complexity from O(h de) to /hd))), where d and h are the dimensionality of the embedding and the number of heads, respectively. In addition, pruning a layer can alter its output. and these changes may be amplified as layers stack. To mitigate error accumulation problem caused by pruning, we introduce a linear calibration strategy to recon- struct the residual errors of the pruned feed-forward network (FFN) layers. Our approach leverages the closed-for solu- tion of ridge regression to model the reconstruction process, thereby alleviating the need for retraining. Additionally, to reduce the number of additional parameters introduced by the linear calibration, we apply a low-rank approximation to the ridge regression solution.
Recent pruning works on LLMs can be categorized into unstructured and structured approaches. Unstructured prun- ing (Frantar& & Alistarh, 2023; Sun et al., 2024) removes individual weights but often struggles to achieve significant speedup without specialized libraries or hardware. In con- trast, structured pruning (Ma et al., 2023; Li et al., 2024; Zhao et al., 2024a; Ling et al.. 2024; Gao et al., 2024) regularly reduces the dimensionality of intermediate fea- tures, allowing to benefit from most GPU devices. How- ever, existing structured pruning methods typically require substantial computational resources and large amounts of data for retraining to reestablish the corrupted correlations. For instance, DISP-LLM (Gao et al., 2024) necessitates 4 NVIDIA A 1000 80GB GPUs to prune a 13B LLaMA model, while methods like LLM-Pruner (Ma et al., 2023), LoRAP (Li et al., 2024), and SlimGPT (Ling et al., 2024) demand tens of thousands of well-annotl annotated instruction data (i.e., Alpaca (Taori et al., 2023)) for retraining to recover the model's 's performance. These significant computational and data requirements present huge challenges in resource- constrained settings. For example, when pruning LLMs for a specific domain, annotating a large amount of instruc- tion data from scratch is a prohibitively labor-intensive and time-consuming process.
To summarize, our contributions are as follows: i) We pro- pose applying PCA the matrix product in the MHA layer, which compresses the model without the need for retraining. Moreover, we devise fast decomposition method, reducing the complexity of PCA by factor of the square of the num- ber of attention heads. ii) We introduce linear calibration to model the residual errors of pruned FFN layers. Along with the proposed weighted SVD and layer selection crite- rion, this method adds negligible extra parameters. iii) Our approach is efficient in terms of data usage, GPU memory consumption, and runing time, while achieves performance that is either better or on par with existing methods across several benchmarks. For example, as shown in Table 1, com- pared to state-of-the-art structured pruning methods (e.g.. LLM-Pruner (Ma et al., 2023) and SlimGPT (Ling et al., 2024)), Olica requires only hundreds of calibration samples, consumes less GPU memory, and significantly reduces the running time while delivering better performance.'
2. Related Work
Large Language Models (LLMs). As demonstrated in (Zhao et al., 2024b), the development of Language Models (LMs) follows four major stages: statistical- based, pre-training- based, and scaling-based based LMs. The rise of LLMs is primarily concentrated in the third stage. Pre- training- based LMs, such as BERT (Devlin et al., 2019; Liu, 2019), GPT-1 (Radford et al., 2018), and GPT-2 (Rad- ford et al., 2019), typically follow a paradigm where the model first pre-trained on a large corpus in task-agnostic manner, and then fine-tuned for specific downstream tasks. Subsequently, according to the scaling law (Kaplan et al., 2020), researchers have found that by simply scaling up the size of the pre-trained model and the amount of pre-training data, and with sufficient computational resources, the per- formance of LMs increases exponentially (Wei et al., 2022a; Schaeffer et al., 2023). Through scaling, LMs have demon- strated exceptional abilities in handling downstream tasks,
neural-
To address these challenges, we propose an efficient struc- tured pruning framework for LLMs that eliminates the need for retraining, called Orthogonal Decomposition and Lin- car Calibration (Olica). The key observation motivating our method is that the core design of transformers, i.e., the multi-head attention (MHA) layer, involves two types of ma- trix products (i.e., W, W and W. .. We treat these products as unified entities and apply Principal Component Analysis (PCA) to extract the most important information for compressing L.M.s. The benefit of this approach is that it eliminates the data requirements for reestablishing corre- lations within the MHA layer, as it directly operates on the parameter matrices. Furthermore, to reduce the complex- ity of PCA, we devise a fast-approach that only performs SVD on one of the product matrices based on the observa- tion of similar distributions of singular values for W,, and W₁ & (also for W. and W.). The fast-approach reduces
Code is available at attps://githusl Better caa
2

---

Olica: Efficient Structured Pruning of Large Language Models without Retraining
Orihogonal Neuron Decompositon
Transformer Block
Linear Calibration
sibi & Stork, 1992; Frantar & Alistarh, 2023) structured pruning scenarios. Additionally, Compresso (Guo al., 2023), LoRAPrune (Zhang et al., 2024), and APT (Zhao et al., 2024a) proposed single-stage pruning approaches, but these still require LoRA to retrain the L.LMs.
FEN Layer
FEN Layer
Ieoredan Savve
Layer
MITA Layer
As mentioned in Section 1, retraining LLMs using LoRA is a tedious and resource-intensive process. especially for models with tens of billions of parameters. In contrast, our proposed approach has the distinct feature of single-stage pruning without the need for retraining.
K
Figure 1. (a): Orthogonal neuron decomposition. Neurons that carry shared information preserve the core projection information from all the original neurons, while neurons that carry residual information can be pruned with minimal impact on accuracy. (b): Linear calibration. This is example of reconstructing the resid- ual errors of pruned FFN Layers.
3. Methodology
3.1. Preliminaries
Transformer. The transformer model consists of L blocks, each of which contains two consecutive layers: a multi- head self-attention (MHA) layer and feed-forward network (FFN) layer. Let X Raxd represent the inputs to the 1th transformer layer, where n is the length of the token sequence and d is the dimensionality of a token. The forward pass of the 1th transformer block is then given by:
such as in-context learning (Brown al., 2020), instruc- tion following (Ouyang et al., 2022), and chain-of -h-huutt prompting (Wei et al., 2022b). As result, the parameters of current LLMs generally range from tens of billions to hundreds of billions (Touvron et al., 2023a; Bai et al., 2023; OpenAI, 2024: Team, 2024).
X MITA
X1 + MHA(XI),
(1)
X X
M + FFN(XMHA)
Despite LLMs surpassing human performance across vari- ous tasks, their large parameter scale makes deployment and inference highly challenging, hindering researchers from conducting further studies on these models.
where MHA(-) and FFN(.) are defined as:
XW WT,
MHA(X) Σ SM
XW WT. W. (2)
Network Pruning on LLMs. Network pruning aims to remove redundant parameters from neural networks while maximally maintaining their performance. Pioneering prun- ing works (Ma al., 2023; Frantar& & Alistarh, 2023) have identified two major challenges in pruning L.MMs: the over- whelmingly large number of parameters and the lack of readily available training data. These challenges make con- ventional gradient-based (LeCun et al., I 1989; Hassibi & Stork, 1992; Molchanov et al., 2019; Sanh et al., 2020) and penalty-based (Han et al., 2015; Wen et al., 2016; Tan et al., 2025) methods inefficient in the context of LLMs. To ad- dress these issues, Ma et al. (2023) proposed two-stage pruning framework: first, using a small amount of calibra- tion data to eliminate unimportant components of LLMs (fast), and then leveraging Low-rank Adaptation (LoRA) (Hu et al., 2022) to retrain the pruned model and recover its performance (time-consuming). Most subsequent works on structured pruning (Ashkboos et al., 2024; Ling et al., 2024: Li al., 2024) have followed this paradigm, with differ- ences in the first phase. For instance, SliceGPT (Ashkboos et al., 2024) uses PCA to reduce the dimensionality of the hidden representations; LoRAP (Li et al., 2024) employs low-rank approximation to replace the weight matrices in the self-attention layer, and SlimGPT (Ling et al., 2024) extends the optimal brain surgeon (OBS) framework (Has-
Vdn
(3)
where SM is the softmax operation. W a. We, K. W W., E Rdxd, denote the 7t head's query, key, value, and output matrices, respectively. h and d, are the number of heads and the dimensionality of each head (generally d dp x h), correspondingly. W., Wa E Rdxid and a is an activation function, such ReLU, GeLU, SiLU, etc. Some LLMs, like LLaMA, adopt gated activation: FFN(X) (XW,, @ () where @ denotes the Hadamard product.
Importance Scores. In network pruning works, the most common criterion for evaluating the importance of elements in the weight matrix W is the first-order Taylor expan- sion of the loss function L(-): [C(O[W ej 0) L(0) x ac(a)) | fj JW, where 0 is the parameter set that consists of all weight matrices. This criterion reflects how much the loss function changes when setting W.j = 0.
The above criterion requires computing the gradient of all parameters in the model, which is computationally expen- sive for LLMss. To mitigate this, Wanda (Sun et al., 2024)
3

---

Olica: Efficient Structured Pruning of Large Language Models without Retraining
Doper lomilt aer! curt (hd bert Chope Lookk leet
and W vo. Here, we use PCA extract the most important features from yk and W.,. no to compress the models while maintaining the performance of the MHA layer. Specifically, taking W so as an example, we apply SVD to Wed. Weo- +- UE and W. <- V. Clearly, . = W. W.W. Since the columns of U and V are orthogonal, the resulting output neurons, such as XW carry distinct information, as shown in Figure I (a). This ensures that the information is captured as fully as possible within the given dimensionality. We refer to this method as Orthogonal Neuron Decomposition (OND), One of the advantages of OND is that it can extract the key information from the MHA layer without requiring additional data and retraining.
Talio
UEV and define W,
we have W.W. T
(a) W. 0
(b) W. 0
(c) Wa
(d) We
Figure 2. The distributions of singular values of different weight matrix in the MHA layer (LLaMA-7B). Wa and W& exhibit strongly similar distribution of singular values, and similar results can be observed from w. and W.. Notably, W, and We show more significant low-rank property.
MHA Pruning. A direct approach for pruning the MHA layer is to preserve the first eigenvectors corresponding to the largest eigenvalues when performing SVD. This is known as the magnitude- based pruning method. However, eigenvalues with smaller magnitudes may still contain im- portant information. Therefore, we use (5) to evaluate the importance of each eigenvector, resulting in the following criteria:
proposed using the magnitudes of weights and activations to measure the contribution of a parameter Way: iy:
I Ig = || 12 (W.l.
(4)
I(ncuron,) = Σ(I(W.)) (.,)),) + I(Wo,,))
(6)
where ||x is the le norm of the 7t column of X, i.c., the vector comprised of 7th feature of all n tokens. Formula (4) is intuitive: if both the 7th feature of the input samples and Wiy ij have larger magnitudes, removing the parameter W., sj will result in drastic change in the projection out- put XW, so it should be kept. Formula (4) is fast and memory-efficient for assessing the importance of parame- ters in LLMs, as it only requires the forward pass of the model. Therefore, in this work, we use (4) by default to evaluate the importance scores of the model's parameters. Although originally designed for unstructured pruning, the subsequent work LoRAP (Li et al., 2024) has demonstrated that it is also effective for structured pruning. Inspired by these empirical results, we measure the importance of a group of parameters, such as the jth intermediate neuron of the FFN layer, as follows:
Based on (6), we prune the neurons with the least impor- tance, thereby reducing the dimensionality of each attention head while maintaining performance.
Fast OND. The complexity of performing SVD on W to Rddd is O(d2). Moreover, there are h heads in a MHA layer, resulting in a complexity of (hdd). This complexity requires about an hour to prune a 7B model. Fortunately, we observe similar distributions of singular values for W. and W. (also for Wg and Ws) as shown in Figure 2. This means that the number of retained singular values required to preserve a certain energy ratio for W, and W. is also similar. Therefore, we can only perform SVD on one of W. and W. to roughly determine the redundancy of the unified entity W Weo Specifically, for instance, letting W. UEV tten we have W., < U and W. a + W. V.V Since typically W. E Rdxd//, the SVD complexity is O /h²), leading to a complexity of O(d (h) for a entire MHA layer. Fast-OND reduces the complexity by factor of 122 In the I.LaMA-7B model, the number of heads h is 32, and pruning it requires only several minutes of runtime.
I(neuron,) = E(z(Wer) + I(Wa,))]
(5)
3.2. Orthogonal Decomposition for MHA
In this section, we consider the compression of the MHA layer. From (2), we can see that the MHA layer, the key design of transformers, depends on matrices W., Wa, k. W. and W. o. through two kinds of matrix product: W gb: W, , W T and W.. W. W. where the head index is omitted for simplicity. Thus, W qh and W.. can be treated as unified entities. The most important information about a specific MHA layer then can be derived by analyzing W ye
3.3. Pruning and Linear Calibration for FFN
FFN Pruning. As illustrated in Figure (b), our goal is to prune the intermediate neurons of each FFN layer, making the prumed FFN layer slimmer. To achieve this, we evaluate their importance scores using (5) and prune the neurons with the lowest importance scores. For example, with a specified
4

---

Olica: Efficient Structured Pruning of Large Language Models without Retraining
0:0
Laper
RxE is defined as follows:
073
RXE 7
R
(9)
2
0.4
where R2 is the pearson correlation coefficient:
13
R. (j= 10 -E))) - -e())
(a) of LLaMA-7B
(b) RXE
(10)
(1) E (ee
Figure 3. (a): Perplexity (PPL, lower better) on WikiTex12 with different number of FPN layers calibrated. (b): Each FFN layer's MC2 of LLaMA-7B and LLaMA-13B models.
e is the tth column of E, e indicates its mean, and similarly for ed and 5()) when E replaced by É. If R is large, linear model can effectively approximate e e(i) se Xw(() where w(() indicates the 7th column of W. Therefore, a large value of (9) indicates that the RE of the pruned layer f is able to be calibrated. Figure 3 (b) shows the RXE values for each FFN layer of the LLaMA-7B and LLaMA-133 13B models, demonstrating that the RE of the FFN layer in the shallower blocks are highly calibratable.
sparsity ratio s, the weights of the pruned FFN layer can be represented as W.. u, W E Rdxd where == 1(1-s) [)]-s)=ss))))) x 4d], and the original weights are W., Wa E Rdxid.
Linear Calibration. Pruning layer generally leads to significant changes in its output. Conventional methods typ- ically require retraining to restore the correlation between layers. Here, we propose a linear calibration strategy to re- construct the residual errors (RE) of the pruned FFN layers, eliminating the need for retraining. Let f denote a FFN layer, and let f represent the pruned version of f. To mit- igate these changes, we use linear model to recover the alterations by solving the ridge regression loss:
Low-rank Approximation. The linear calibration intro- duces matrix W E idddd with the number of parameters being d², This accounts for 1/8 of the number of parameters for FFN layer. To reduce the number of extra parame- ters introduced, we propose to use a low-rank approxima- tion of W. Specifically, let W = UEV and define W1 U,E, and W₂ V,, where we preserve the first r eigenvectors with the largest eigenvalues. Then, the number of parameters, i.e., W, and W2, introduced by the linear calibration is 2dr, which is far less than de if we take < d. The forward pass of (8) can then be modified as:
W = arg min |E- XW 12 + W WeRoxd
(7)
where E- f(x) - f(X) E Raxd indicates the RE of a pruned layer f. || is the Frobenius norm, and indi- cates the penalty strength, which helps avoid the irreversible problem. Then, the forward pass of our linear calibration for the 1th layer is as follows:
X f(Xi)) + X,W,, W
(11)
X = f(X)) + H,W,
(8)
4. Experiments
4.1. Experimental Settings
The loss of ridge regression (7) has a closed-form solution W = (X X AX)- where is an identity matrix. Therefore, can small amount of calibration data to obtain the solution, avoiding the need for retraining.
Models and evaluation protocols. Following the previous works (Ma et al., 2023; Li ct al., 2024), we mainly focus on the evaluation of 1 (Touvron et al., 2023a), LLaMA- (Touvron et al., 2023b), and Vicuna (Chiang al., 2023) models. We assess the performance of the pruned models WikiText2 (Merity et al., 2017) with sequence length of 128 tokens, and on the following downstream tasks: BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2020), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021) ARC-easy (Clark et al., 2018), ARC-challenge (Clark et al., 2018), and OpenbookQA (Mihaylov et al., 2018). We employ Im-eval-harness framework (Gao et al., 2021) to evaluate the pruned model performance on these tasks?
Layer Selection. Figure 3
(a) shows that perplexity (PPL,
lower is better) decreases, then stays stable as the number of calibrated FFN layers increases. This implies that more cali- bration is not necessarily better, especially it may introduce more parameters. Hence, it is necessary to select the FFN layers that should be calibrated, Here, we choose layers whose RE are linearly recoverable, as we are using the lin- ear calibration strategy. To this end, we propose criterion based on the Multiple Correlation Coefficient (MC2), which is used to express the linear correlation and is denoted as RxE. Let = XW E Raxd represents the prediction of a linear model that fits the residual data (X, E). Then, the
'The version of Im-eval-harne used in this paper the as SlimGPT (Ling et al., 2024), which can be found their the sup-
5

---

Olica: Efficient Structured Pruning of Large Language Models without Retraining
Table Zero-shot performance and PPI. on WikiText2 of pruned I.LaMA family. "SR" indicates sparsity ratio, and "FT" denotes whether LoRA fine-tuning is used to recover the model's performance. represents results that are reproduced by this paper. The best and the second best performances are marked as bolded and underlined. correspondingly.
Model
SR%
Method
FT
PPL (1)
BoolQ
PIQA
HellaS
WinoG
ARC-e
ARC-c
OBQA
Avg. (t)
0%
Dense
x
12.63
75.08
79.16
76.20
70.00
72.89
44.88
44.40
66.09
LLM--runer
18.01
66.76
78.45
71,44
63.77
66.41
39.85
43.80
61.50
DISP-LLM
x
76.66
68.39
64.72
64.81
37.12
Compresso
79.08
75.46
53.44
67.80
68.64
37.97
34.20
59.51
20%
FLAP
X
17.00
69.40
74.70
66.90
66.30
64.60
36.50
38.20
59.50
LoraPrune
16.80
65.62
79.31
70.00
62.76
65.87
37.69
39.14
60.06
LoRAP
X
15.84
73.88
76.71
72.94
68.03
69.70
40.87
42.40
63.60
SlimGPT
16.99
75.93
77.58
73.07
67.96
68.60
41.72
41.80
63.81
Olica (Ours)
X
15.35
71.59
77.91
73.30
70.01
72.10
42.66
44.20
64.54
LLaMA-7B
LLM- LLM-Pruner
20.57
62.81
76.93
69.21
60.46
63.34
38.14
39.80
58.67
Compresso
73.55
73.07
49.16
64.80
66.20
37.20
29.80
56.25
25%
SlimGPT
x
19.11
75.11
76.77
70.60
67.25
66.75
40.40
40.40
62.47
LoRAP
X 17.40
72.14
77.20
71.30
68.75
68.48
39.16
41.00
62.57
Olica (Ours)
X
16.69
72.54
76.88
71.40
68. 19
70.83
41.72
43.20
63.54
LLM-Pruner
24.50
62.02
74.92
64.41
61.80
53.79
32.00
38.80
55.39
Compresso
68.69
72.85
47.18
63.38
65.99
35.07
29.00
54.59
33%
LoRAP
x
21.66
66.54
74.70
67.11
65.98
63.97
35.41
39.40
59.02
SlimGPT
x
24.55
72.72
75.68
68.10
66.54
62.29
37.03
40.20
60.37
Olica (Ours)
19.83
72.87
75.55
67.95
67.01
66.25
37.63
41.20
61.21
LLaMA-133 13B
0%
Dense
x
11.58
77.89
80.14
79.06
72.85
74.75
47.61
44.80
68.16
LLM-Pruner
/ 16.62
79.38
77.36
71.47
70.32
70.54
44.88
45.80
65.68
SlimGPT
X 14.87
77.06
79.82
76.94
72.61
69.78
44.80
43.60
66.37
20%
LoRAP
X
13.84
78.87
79.05
77.54
71.35
73.57
43.00
44.54
66.84
Olica (Ours)
13.68
78.32
78.89
77.21
74.11
73.99
46.59
44.60
67.67
Implementation details. We randomly select 256 samples from Bookcorpus (Zhu et al., 2015) and Alpaca (Taori et al., 2023) datasets, each of which is truncated to sequence length 128 tokens, as the calibration data. The number of calibrated FFN layers is selected from (6. 12, 16) for mod- els with different parameter sizes. the linear calibration, we retain the top 3% cigenvectors for low-rank approxima- tion, i.e., r/d- 0.03. Following LoRAP (Li et al., 2024), compress the MHA and FFN layers, until achieving specified sparsity ratio of the entire model (i.e., including the token embedding layer and the final projection layer). All the experiments are conducted on a single NVIDIA A100 80GB GPU. More details can found in Appendix A, and detailed algorithm can be found in Appendix B.
Baselines. We select state-of-the-ar structured pruning approaches for comparisons: LLM-Prunce (NeulPS'23) 23) (Ma et al., 2023), LoraPrune (Findings of ACL'23) (Zhang et al., 2024), Compresso (Guo et al., 2023), FALP (AAAI' 24) (An et al., 2024), SliceGPT (ICLR'24 (Ashkboos et al., 2024), LLM-Surgeon (ICLR'24) (van der Ouderaa al.,
2024), LoRAP (ICML'24) (Li et al., 2024), DISP-LLM (NeulPS'24) 24) (Gao al., 2024), and SlimGPT (NeulPS'24 24) (Ling et al., 2024).
4.2. Main Results
Performance. The experimental results presented in Table
2 and Table 3 highlight the superior performance of our Olica method across multiple models and sparsity ratios. In particular, for LLaMA-7B in Table 2, Olica achieves the highest average accuracy of 64.54%, 63.54%, and 61.21% %, while achieving the lowest perplexity (PPL) values of 15.35,
plementary material: https: // /oppnreviiix net forum? 1d=MxF0IIIH xw. Therefore, the baseline results this paper, not specified, are cited from (Ling et al., 2024).
6

---

Olica: Efficient Structured Pruning of Large Language Models without Retraining
Table Zero-shot performance and PPI. WikiText2 pruned L.-2 and Vicuna. "SR" indicates sparsity ratio, and "FT" denotes whether LoRA fine-tuning is used to recover the model's performance. represents results that are reproduced by this paper. The best and the second best performances are marked as bolded and underlined, correspondingly.
Model
SR%
Method
FT
PPL (L)
BoolQ
PIQA
HellaS
WinoG
ARC-c
ARC-c
OBQA
Avg. (1)
0%
Dense
x
12.19
77.71
79.05
76.00
68.98
74.58
46.33
44.20
66.69
LLaMA-2-7B
SliceGPT
x
63.55
49.62
61.33
51.77
31.23
DISP-LLM
X
73.72
62.87
63.93
60.10
37.03
30%
LLM-Surgeon
X
61.25
73.56
60.72
61.09
63.09
36.69
38.80
56.56
LoRAP f
X
19.42
68.93
75.46
66.66
66.30
62.37
35,49
37.00
58.89
Olica (Ours)
18.54
71.19
75.35
67.04
67.88
68.60
37.71
39.20
61.14
0%
Dense
X
16.11
78.41
78.56
74.68
70.09
72.01
43.77
43.40
65.85
Vicuna- 7B
LLM--runer
19.11
61.96
76.88
69.18
63.30
61.83
37.88
39.40
58.63
SlimGPT
X
21.14
75.41
77.09
72.34
68.43
69.23
41.47
43.40
63.91
20%
LoRAP f
X
20.74
78.01
76.12
72.62
66.93
71.17
43.09
42,80
64.39
Olica (Ours)
X
20.23
75.81
76.66
72.82
68.67
72.81
42.41
45.00
64.88
Table Statistics of the pruned model LLaMA-7B, including the parameters, MACs, memory, and inference latency.
Table 5. Compare Fast-OND with S.D,,sVViiaalttl Activation Weinoon-Weiieee SVD (AWSVD) (I.i al., 2024), and Wanda (Sun al., 2024), under scenarios with and without linear calihration (I.C).
SR%
Params
MACs
Memory
Latency
0%
6.74B
424.02G
12884.5MiB
46.95s
Setting
Method
PPL (L)
Mean Accuracy (f)
20%
5.39B
373.23G
10464.3MiB
40.62s
SVD
71.01
47.62
25%
5.01B
360.25G
9696.3MiB
39.32s
w/o LC
AWSVD
25.78
59.93
33%
4.52B
339.53G
8718.
35.78s
Wanda
20.94
59.82
Fast-OND
20.34
60.68
SVD
63.48
48.62
AWSVD
24.72
60.38
16.69, and 19.83 at sparsities of 20%, 25%, and 33%, respec- tively. Additionally, the improvement of Olica over other methods increases with higher sparsity. These demonstrate its ability to maintain both modeling and reasoning perfor- mance under compression and pruning. For LLaMA-13B at 20% sparsity, Olica outperforms all baseline methods with the highest average accuracy of 67.67% and the lowest PPL of 13.68, demonstrating its scalability to larger models. In Table 3, for LLaMA-2-7B at 30% sparsity, Olica achieves the best average accuracy of 61. 14% and the lowest PPL of 18.54, outperforming structured pruning methods like LLM- Surgeon and LoRAP across key benchmarks. Similarly, for Vicuna-7B at 20% sparsity, Olica leads with an average accu- racy of 64.88%, excelling on datasets like WinoG (68.67%) and OBQA (45.00%), while maintaining the lowest PPL of 20.23. These results demonstrate that Olica consistently outperforms baseline pruning methods across diverse tasks, higher sparsity ratios, and multiple model architectures, all while requiring no retraining to recover performance. This makes Olica highly effective, robust, and scalable solu- tion for compressing L.LSs without significant performance
LC
Wanda
20.40
60.04
Fast-OND
19.83
61.21
degradation and resource consumption.
Inference cost of the pruned models. Table 4 presents the statistics of the pruned LLaMA-7B model, including sparsity ratio (SR), number of parameters (Params), MACs, memory consumption, and inference latency. As the sparsity ratio increases, both the model size (Params) and computa- tional requirements (MACs) decrease, For instance, at 33% sparsity, the number of parameters decreases from 6.74B (at 0% sparsity) to 4.52B, and MACs drop from 424.02G to 339. 339.53G, indicating a significant reduction in compu- tational complexity. Similarly, memory usage decreases from 12884.5 MiB to 8718.1 MiB, demonstrating improved memory efficiency. Inference latency is also notably re- duced, from 46.95s at 0% sparsity to 34.28s at 33% sparsity (measured on the WikiText2 test set using single NVIDIA

---

Olica: Efficient Structured Pruning of Large Language Models without Retraining
SK-55% SR.20
54.35%
12
50..0%
Parpetty (PP
(a) PPL on WikiText2
(b) Ratio of Parameters (%)
64 120
312
2700
120
K20
Figure (a): PPL on WikiText2 datatset (after minus for better presentation), where we vary the number of calibrated FFN layers and the retained ratio ranks in the linear calibration strategy. (b) each cell represents the ratio (%) of additional parameters introduced by the linear calibration strategy, corresponding to (a),
(a) Number of Samples
(b) Sequence Length
Figure 5. (a): Vary the number of samples while fixing the length of sequence as 128. (b) Vary the length of sequence while fixing the number of samples 128. To meet the length of sequence, these samples are sampled from C4 dataset (Raffel et al., 2020).
GeForce RTX 4090). These results show that higher sparsity ratios effectively reduce resource consumption and enhance inference efficiency, making the pruned models more suit- able for resource-constrained environments without signifi- cant performance loss.
Table 6. Pruning runtime of OND and fast-OND (LLaMA-7B).
Method
SR
Runtime
PPL (4)
Mcan Accuracy (t)
OND
20%
2910s
15.17
64.32
Fast-OND
20%
413s
15.35
64.54
4.3. Ablation Study
Effectiveness of the proposed modules. In this experiment, we aim to investigate the effectiveness of the proposed Fast- OND and the linear calibration strategy. The experiment is arranged as follows: first, we replace Fast-OND with stan- dard SVD (directly performing SVD both W, and W.), Activation Aetivation-Weighted SVD (performing weighted SVD both W. and W. by the magnitudes of input features), and Wanda (directly remove neurons based on (5) for W
and the retained rank ratio is 0.15, the additional parameters account for around 1%, while the corresponding PPL, as shown in Figure 4 (a), decreases significantly.
4.4. Efficiency Analysis
Sample efficiency. From Figure 5, we observe that our approach is not sensitive to the amount of calibration data. Specifically, both the number of samples and the sequence length vary from 8 to 2048 (2$ times larger), yet the PPI. changes by at most 2.4.
and W, ); second, all these experiments are conducted un- der the scenarios with and without linear calibration. The results are reported in Table We can observe that the proposed Fast-OND is more effective when compared with baselines. Moreover, the linear calibration strategy can be easily coupled with existing pruning methods, and improves their performance.
Runtime analysis. We present the runtime of OND and Fast-OND in Table 6, where we observe that while both OND and Fast-OND achieve similar performance in terms of PPL and accuracy, the runtime of Fast-OND is signifi- cantly shorter than that of OND.
Parameters introduced by the linear calibration. In this experiment, we examine the effect of the number cali- brated FFN layers and the retained ratio r/d for low-rank approximation in linear calibration strategy. From Figure 4 (a), where each cell represents the model performance (measured by PPL) under different scenarios. we observe that both factors affect the PPL. Specifically, when the num- ber of calibrated FFN layers is fixed and the retained ratio increases, the PPL gradually decreases. Similarly, when the retained ratio is fixed and the number of calibrated FFN layers increases, similar trend is observed. However, the performance gain gradually diminishes once the rank ratio exceeds 0.15 or the number of calibrated FFN layers ex- ceeds 20. In Figure 4 (b), we observe that within the region showing performance gains in Figure 4 (a), the linear cali- bration strategy introduces very few additional parameters. Specifically, when the number of calibrated FFN layers is 20
5. Conclusion
In this paper, proposed efficient pruning method for LLMs, called Orthogonal Neuron Decomposition and Lin- ear Calibration (Olica), which eliminates the need for re- training. One of the core designs in the Olica is Orthogo- nal Neuron Decomposition (OND), which treats the matrix products the MHA layer as unified entities. Thus, we can use PCA to extract the most information from the MHA layer without sacrificing accuracy or disrupting their origi- nal structure. We also devised a fast-OND method, reducing the complexity by a factor of the square of the number of attention heads. In addition, we introduced a linear calibra- tion approach to mitigate the problem error accumulation in the FFN layer. Using the closed- forrm solution of ridge
8

---

Olica: Efficient Structured Pruning of Large Language Models without Retraining
regression, We model the residual errors of the pruned FFN layers with two low-rank matrices, without retraining. Fi-
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan,
J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
nally, we conducted extensive experiments, showing that the proposed Olica is efficient in terms of data usage, GPU memory, and running time, while delivering superior perfor- mance across multiple benchmarks.
Askell, A., Agarwal, S., Herbert-Voss. A., Krueger, G., Henighan. T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
J. Winter, C., Hesse, C.,, Chen, M., Sigler, E.,, Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners, 2020. URL https:// arxiv. arxiv.org/aii.. wts//a///////eaeeeeee.ee./eeeaee
Acknowledgements
The research was partially supported by National Key R&D Program of China (No.2022YFF 1003702), National Nat- ural Science Foundation of China (Nos.12426309) New
Chiang, W.-L.., Li, Z., Lin, Z.,, Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality. See https://vicuma Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
Comerstone Science Foundation.
Impact Statement
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M.. Gur-Ari, G., Yin, P., Duke, T.,, Levskaya, A., Ghemawat, S., Dev, S., Michalewski, H., Garcia, X., Misra, V., Robinson, K., Fedus, L.,, Zhou, D., Ippolito, D., Luan, D., Lim, H., Zoph, B., Spiridonov, A.. Sepassi, R.. Dohan, D., Agrawal, S., Omernick, M., Dai, A. M., Pillai, T. S., Pel- lat, M., Lewkowycz, A., Moreira, E., Child, R., Polozov, O., Lee, K., Zhou, 7.,, Wang, X., Saeta, B., Diaz, M., Firat, O., Catasta, M., Wei, J., Mcier-Hellsterm, K., Eck, D., Dean, J., Petrov, S., and Fiedel, N. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023. URL http:
Our work focuses structured pruning of LLMs, which improves efficiency by reducing computational and memory requirements, making LLMs more suitable for deployment on resource-constrained devices. The proposed method is efficient in data utilization, GPU memory, and runtime, of- fering equitable opportunities for the use and development of LLMs. Furthermore, the core idea of Olica replacing complex structures with simple, explicit expressions couud improve both robustness and interpretability of LLMs, war- ranting further investigation.
References
An, Y., Zhao, X., Yu, T., Tang, M., and Wang, J. Fluctuation- based adaptive structured pruning for large language mod- els. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 10865-10873, 2024.
lllmmr.orr////./ www.orgppgg/////..... pwtperrs/v24/22 -144 html.
Ashkboos, S., Croci, M. L., do Nascimento, M. G., Hoefler, T., and Hensman, SliceGPT: Compress large language models by deleting rows and columns. In The Twelfth International Conference Learning Representations, 2024. URL https: : httppppeeeviieeeeee net/ forum? i id=vxxardq6cb.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Burstein, J., Do- ran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Pupers), pp. 2924-2936, Minneapolis, Minnesota, June 2019. Associ- ation for Computational Linguistics. doi: 18.13v///v// N19- 1300. URL. https: hatps://acaaattthoi .orllggyyoro/ooooooooooooo N1 9 1300/.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., Hui, B., Ji, L., Li, M., Lin, J., Lin, R., Liu, D., Liu, G., Lu, C., Lu, K., Ma, J., Men, R., Ren, X., Ren, X., Tan, C., Tan, S., Tu, J., Wang, P. Wang. S., Wang, W., Wu, S., Xu, B., Xu, J., Yang. A., Yang, H., Yang, J., Yang, S., Yao, Y., Yu, B., Yuan, H. Yuan, Z., Zhang, J., Zhang, X., Zhang, Y., Zhang, Z., Zhou, C., Zhou, J., Zhou, X., and Zhu, T. Qwen technical report. arXiv preprint arXiv:2309. 16609,99.666...... 2023.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 180300545, 2018
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reason- ing about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 34, pp. 7432-7439, 2020.
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. int8() 8-bit matrix multiplication for transformers at scale. In Proceedings of the 36th International Confer- ence on Neural Information Processing Systems, NIPS

---

Olica: Efficient Structured Pruning of Large Language Models without Retraining
'22, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 9781713871088.
Processing Systems, NIPS'92, pp. 164-171, San Fran- cisco, CA, USA, 1992. Morgan Kaufmann Publishers Inc. ISBN 1558602747
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 1 (Long and Short Pupers), pp- 4171-4186 86, Min- neapolis, Minnesota, June 2019. Association for Compu- tational Linguistics. doi: 10. 10.18653/v1/N19-11 1423. URL https: / /acl antho Saaly.ogg/N/ wwwg/19--110000
Hinton, G.,
Vinyals, O., and Dean, J. Distilling the knowledge in a neural network, 2015. URL https: / //arxiv. www./oor///ooo abs 100.. 02531.
Hsu, Y.-C., Hua, T., Chang, S., Lou, Q., Shen, Y., and Jin, H. Language model compression with weighted low-rank factorization. In International Conference on Learning Representations, 2022. URL https: oppnrevien.eeeeeeeeeee net / wwt.ooommmiouuyeaeemmmmm
Frankle, J. and Carbin, M. The lottery ticket hypothe- sis: Finding sparse, trainable neural networks. In In- ternational Conference on Learning Representations, forrrrrruiweee/ooomm
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adap- tation large language models. International Confer- ence on Learning Representations, 2022. https: / //openrevies ew net/ forum? d=nZeVKeeFYFS
2019. URL
id-r -33RcF
Frantar, E. and Alistarh, D. Sparsegpt: massive language models can be accurately pruned in one-shot. In Proceed- ings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
Kaplan, J., McCandlish, S., Henighan, T., Brown, B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language mod- els, 2020. URL https::/p/// /awxiv. org/ hbsg//bb//00 20011 08361.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https: //openeeview..o.mmmm///// net. forum? chBPnnws
Kwon, W., Kim, S., Mahoney, M. W., Hassoun, J., Keutzer, K., and Gholami, A. A fast post-training pruning framework for transformers. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 24101-24116. Curran Associates, Inc., 2022. URL https: pttttps:/prroceeddiii hew.riinnnnn.n.......ee..m.... cc htppaapr__fff fi / /eaperr_filee/pppe///2//////// 98 9877ed997ab668f9111337 bbt.:eeee.....2000eeeeeeeeeeee 9bce3ea1221 Paper ConeerrCoofeeecce..dde
Gao, L.. Tow, J.. Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., McDonell, K., Muennighoff, N., et al. A framework for few-shot language model evaluation. Version vO. O. Sept, 10:8-9, 2021.
Gao, S., Lin, C.-H., Hua, T., Tang, Z., Shen, Y., Jin, H., and Hsu, Y.C. DISP-LLM: Dimension Dimension-independeel struc- tural pruning for large language models. In The Thirty- eighth Annual Conference on Neural Information Pro- cessing Systems, 2024. httts::: operreview. net/ net/forum? forum? finnnidd==xxxa Y
LeCun, Y., Denker, J., and Solla, S. Optimal brain damage. In Touretzky, D. (ed.), Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmann, 1989. URL https::/prooceedingg.. http:/pproceeednnn...eieuup... neurips. cc/ pa/paper__. f paappeeefflleepppppe//////////ie 2c982bbeecc700hh2250118822222222 pdf.
Guo, S., Xu, J., Zhang, L., and Yang. M. Com- presso: Structured pruning with collaborative prompting learns compact large language models. arXiv preprint arXiv:2310.05015, 2023.
Li, G., Tang, Y., and Zhang, W. LoRAP: Transformer sub-layers deserve differentiated structured compression for large language models. In Forty-first International Conference Machine Learning, 2024. URL https: / / //operreview eww net forumfrrmiooo ioorrmmiddmmhee wwc5mmmmmIInn.....m
Han, S., Pool, J., Tran, J., and Dally, W. Learning both weights and connections for efficient neural networks. In Proceedings of the 29th International Conference on Neu- ral Information Processing Systems Volume 1, NIPS' 15, pp. 1135 143, Cambridge, MA, USA, 2015. MIT Press.
Ling, G., Wang, Z.,, Yuliang Yan, and Liu, Q. SlimGPT: Layer-wise structured pruning for large language mod- els. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: / //oppreevl ew net/ forum? d=MxF0 KK....
Hassibi, B. and Stork, D. G. Second order derivatives for net- work pruning: optimal brain surgeon. Proceedings of the 6th International Conference on Neural Information
10

---

Olica: Efficient Structured Pruning of Large Language Models without Retraining
Liu, L.,, Zhang, S., Kuang, Z., Zhou, A., Xue, J.-H., Wang, X., Chen, Y., Yang, W., Liao, Q., and Zhang, W. Group fisher pruning for practical network com- pression. In Meila, M. and Zhang, T. (eds.), Pro- ceedings of the 38th International Conference on Ma- chine Learning, volume 139 Proceedings of Machine Learning Research, pp. 7021-7032. PMLR, 18-24 Jul 2021. URL https: httpps:///proooeeo /ptp//eeeoeeggsssee/ee mlr. press/ v1 #139/1iu2lab html.
Radford, A.,
Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., al. Language models unsupervised multitask learners. OpenAl blog, 1(8)99, 2019.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S.,
Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with unified text-to-text transformer. Jourmal of machine learning research, 21 (140):1-67, 2020.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9))99-106, 2021.
Liu, Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: /907. 1692, 364, 2019.
Ma, X., Fang, G., and Wang, X. LLM-pruner: On the structural pruning of large language models. In Thirty- seventh Conference on Neural Information Processing Systems, 2023. URL https: http:///ppeeeeeeeeeeeooee n Forum?id=J8AA
Sanh, V., Wolf, T., and Rush, A. Movement pruning: Adaptive sparsity by fine-tuning. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 10378-20388 Curran Associates, Inc., 2020. URL https::/proceedins.. neerere cc/pappee_fflleeeppper///////1 cc/ aaalsabaa77Gacc4399ttafff. pdf.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. URL https:// openreview net http://orummmmmmme
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? a new dataset for open book question answering. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (eds.), Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 2381-239) Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10. 10.18653/11/D8-- 1260. URL ht https: //acl placlantholsge o-tpa//chhhoogo..oo///--1-1-111-
Schaeffer, R., Miranda, B., and Koyejo, S. Are emergent abilities of large language models mirage? In Thirty- seventh Conference Neural Information Processing Systems, 2023, URL https: openreview nnt// Torum?1d=ITW9edRDD
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:1 17063, 2024.
Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. In The Tweifth International Conference on Learning Representations, 2024. URL htttps://openrevieet net wwwoooormdddooooououo
Molchanov, P., Mallya, A., Tyree, S., Frosio, I., and Kautz, J. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11264-11272, 2019.
OpenAl. Gpt-4 technical report, 2024. URL https:// arxiv. or..orgg/aooo abs wtt.b////3333.m
Sun, S., Cheng, Y., Gan, Z., and Liu, J. Patient knowl- edge distillation for bert model compression, 2019. URI. https: /ttp///axvvvvv/vv org abs 1908. 09355.
Ouyang. L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Gray, A., Schulman, J.. Hilton, J., Kelton, F., Miller, L., Simens,
Tan, Z.. Zhou, L., and Lin, H. Generative adversarial leamn ing with optimal input dimension and its adaptive genera- tor architecture, 2024. URL https: /tttps://axxv...oo http/aaxvvvvrr/////e abs abs/2405.0 03723.
M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instruc- tions with human feedback. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https: //openreview net / forum? TGBKACXEON.
Tan, Z., He, J., and Lin, H. AMSC: Adaptive multi- dimensional structured compression with theoretical guar- antees, 2025. URL https:
/ hwtp//winneeeeeeeeeeeeeee net/ Dorumiid=cSboBrSTK:
Radford, A., Narasimhan, K., Salimans, T., and Sutskever, Improving language understanding with unsupervised learning. Technical Report, OpenAI, 2018.
Taori, R. Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An -nstruction-followinng llama

---

Olica: Efficient Structured Pruning of Large Language Models without Retraining
model.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can machine really finish your sen- tence? In Korhonen, A., Traum, D., and Mårquez. L. (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791- 4800, Florence, Italy, July 2019. Association for Compu- tational Linguistics. doi: 10. 10.186533w//119-1447 URL
Stanford_alpacte 2023.
Team, T. The falcon 3 family of open models, December 2024.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models, 2023a. URL https://azxa org/ abs/ 2302. 13971.
Zhang, M., Chen, H., Shen, C., Yang, Z., Ou, L., Yu, X., and Zhuang, B. LoRAPrune: Structured pruning meets low- rank parameter-efficient fine-tuning. In Ku, L.-W., Mar- tins, A., and Srikumar, V. (eds.), Findings of the Associa- tion for Computational Linguistics: ACI. 2024, pp. 3013- 3026, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 18653/v1/2024. findings- acl. URL https: aalaccaann /atp//haaaan..oy....oo org/ org/2024, 2024. f findiniinnninnne acl. 178/.
Touvron, H., Martin, L.., Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
van der Ouderaa, F. A., Nagel, M., Baalen, M. V., and Blankevoort, T. The LLM surgeon. The Twelfth International Conference on Learning Representations, 2024. URL https: https:///ppeeeeee http:///pppeeeeeeeeeeeeeeeeeeeee wttp://eeeeerrooreeeeeeeeoeeeeee id-DYIIRgwg21.
Zhao, B., Hajishirzi, H., and Cao, Q. Apt: Adaptive prun- ing and tuning pretrained language models for efficient training and inference. In ICML, 2024a. URL https: / / /poperevvi ew net. forum? d=sb8 1teeb811110000
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X.,
Liu, Z., Liu, P., Nie, J.-Y., and Wen, J.-R. A survey of large language models, 2024b. URL https: /arxiv. org/ abs 20ls/2330 22033/333.000 18223.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Met- zler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022a. ISSN 2835-8856. URL https:// openreview net vey Certification.
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urta- sun, R., Torralba, A., and Fidler, S. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the 2015 IEEE International Conference on Com- puler Vision (ICCV), ICCV '15, pp. 19-27, USA, 2015. IEEE Computer Society. ISBN 9781467383912. doi: 10. 10.1109/CCCV2211.111 URL https: / ///oi.oorrr 1t.1109/cccv.20555
wwt.torummeeeeeee forum? idyyzk0522dDD Sur-
Wei, J., Wang, X., Schuurmans, D., Bosma, M., brian ichter, Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022b. URL httph:tt////// httppp://ppereeeiiee http//ppnnrvvieereeeeee net/ forum?ide Biliddd____MMM_B______
Wen, W., Wu, C., Wang, Y., Chen, Y., and Li, H. Learn- ing structured sparsity in deep neural networks. In Lee, D., Sugiyama, M., Luxburg, U., Guyon, and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc.,
2016. URL ppaper_ffiees/ppper////////Le cc cc/paper / Liff@03Bbbb00eeeraaar333353-3--3 pdf.
12

---

Olica: Efficient Structured Pruning of Large Language Models without Retraining
35
40
Perplexity (PPL)
30
Perplexity (PPL)
25
30
25
20
15
15
10
0.1
0.2
0.3
0.4
0.5 0.6
0.1 0.2
0.3
0.4
0.5
0.6
Spansity Ratio
Sparsity Ratio
(a) Bookcorpus + Alpaca
(b) C4 Alpaca
Figure 6. Perplexity (PPL.) on WikiText2 of LLaMA-30B under different sparsity ratio, where different calibration datasets used. (a): 256 samples with 128 sequence length are randomly selected from Bookcorpus and Alpaca datasets. (ab): 256 samples with 128 sequence length are randomly selected from C4 and Alpaca datasets.
A. Implementation Details
We randomly select 128 samples from each of Bookcorpus (Zhu et al., 2015) and Alpaca (Taori et al., 2023) datasets, each of which is truncated to a sequence length of 128 tokens, as the calibration data. The number of calibrated FFN layers selected from {6, 12, 16) for models with different parameter sizes. In the linear calibration, low-rank approximation, i.c., r/d 0.03. Following LoRAP (Li et al., 2024), we compress the MHA and FFN layers, until achieving specified sparsity ratio of the entire model (i.e., including the token embedding layer and the final projection layer). Following (Frantar & Alistarh, 2023; Frantar et al., 2023), we set the À in 7 as: À = A Mean(diag(2 X)) where An is fixed as 0.5.
retain the top 3% eigenvectors for
The form MHA layer formulated in (2) is one the most popular transformer architectures. However, the popular open- sourced LLaMA models employ rotary position embedding (RoPE) (Su et al., 2024) in the MHA layer: x; W & W W zJ. i.e., it inserts rotary matrix P.j that is based the position of input variables before the product of W and W inject the positional information of tokens, which means that there is direct product between W, and W.. k. Fortunately, from Figure 2 (c) and (d), we observe significant low-rank property for W Wa and W.. R. which means that they are suitable for low-rank approximation. Thus, we apply PCA to W, y and W₁ & respectively, approximating each by two low-rank matrices. The approximation are obtained by retaining the first principle components. Moreover, inspired by the success of weighted SVD (Hsu et al., 2022; Li et al., 2024), we replace the standard SVD with weighted SVD by the magnitudes of input features with the following objective function:
arg min |[ W. W2) W.
(12)
where D ((aa((Ia:())) ..., standard SVD WD = T W, UΣ and W VD-1 Despite performing separate PCA on W and W (this is not our contribution), in the ablation study (i.e., Table 5), we have demonstrated that the Fast-OND is still very important for pruning W, and W., which highlights the contribution of our proposed Fast-OND.
... ||
and ( is the tth column of input matrix X. This be solved by performing
Mi where Mj is the Parameter Allocation. Letting denotes the sparsity ratio (SR), we first modify the SR as $ = s M, number of parameters of the entire model and M2 is the number of parameters of all the MHA and FFN layers. Then, the for the query and key matrices 28; the SR for value and output matrices is 6/2; the SR for the FFN layers determined by the remaining parameter budgets that ensures the SR of the entire model is $.
13

---

Olica: Efficient Structured Pruning of Large Language Models without Retraining
Algorithm 1 Overview of the proposed Olica
Input: Calibration data D, model M (e.g., LLaMA-7B), and the number of FFN layers needed to calibrate K, sparsity
ratio (SR) 8;
Output: A pruned model M
1: # Step I: Layer selection
2: Indexes +- FFN indexes with Topk MC2 using (9);
3: Step 2: Start pruning
4: input + Preprocess(D);
5: for block in M.blocks do
6: outputs t- block(inputs))
7: # Pruning MHA layer with SR
8: Wg. WE. W., + wwt.bbll.....aatii..............
9: Wa. W& +- WeighteddSVD(W. WR);
W.. W. + Fast OND(W,,, Wo):
11: # Pruning and calibrating FFN layer with SR
12: FEEN using (5);
13:
if block.FFN.inde. in Indexes then
14:
W,, W₂ t Performing SVD on the solution of (7);
15: Modifying Balock.FFN.forvaan as (11);
16: end if
17: inputss + outputs;
end for
19: return The pruned model M
B. Algorithm.
Algorithm. As an example, we use LLaMA models to present the final algorithm in Algorithm 1.
C. Pruning Results of LLaMA-30B
Here, provide the pruning results of a larger model, i.e., LLaMA-30B. The results for the pruned LLaMA-30B model presented in Figure 6. We report results for sparsity ratios of 0.1, 0.2, 0.3, 0.4, 0.5, and 0.6. Additionally, we evaluate different combinations of datasets for calibration data, namely Bookcorpus + Alpaca and C4 + Alpaca. It can be observed that without retraining, the LLaMA-30B model can tolerate maximum sparsity ratio of 40%. Beyond this threshold, when the model is pruned with a higher sparsity ratio, its performance deteriorates significantly. Furthermore, when the sparsity ratio is below 40%, the choice calibration dataset has a minimal impact on the pruning results.
D. Results of 50% Sparsity Ratio
We observe that at high sparsity ratios (e.g., 50%), the performance of pruned LLMs deteriorates significantly. For instance, in the results reported by (Ma et al., 2023; Li et al., 2024; Ling et al., 2024), the performance of the pruned LLaMA- 13B model with a 50% sparsity ratio is notably lower than that of LLaMA-7B. even with retraining. This suggests that a 50% sparsity ratio may exceed the tolerance limit of these models. However, we believe that when pursuing a high sparsity ratio, the primary prerequisite is to maximally maintain the performance LLMs. Therefore, we provide the results of LLMs with 50% sparsity ratio in the Appendix.
Table 7 presents the zero-shot performance and perplexity (PPL) results the WikiText2 dataset for various models with 50% sparsity ratio. The table compares the performance of different pruning methods, including Olica (our proposed method), LLMM-runee, and LoRAP.
-LLaMA-7B: At 50% sparsity. Olica achieves an average accuracy of 50.68%, which is higher than both LLM-Pruned Pruuer (48.35%) and LoRAP (46.86%). Olica also performs well on BoolQ (65.32%) and ARC-e (51.98%), with a competitive PPL. of 49.39. This demonstrates that Olica balances compression with high performance across several reasoning tasks.
14

---

Olica: Efficient Structured Pruning of Large Language Models without Retraining
Table 7. Zero-shot performance and PPI. wikitex12 with 50% sparsity ratio.
Model
Method
FT
PPL (4)
BoolQ
PIQA
HcllaS
WinoG
ARC-e
ARC-c
OBQA
Avg. (t)
Dense
x
12.19
77.71
79.05
76.00
68.98
74.58
46.33
44.20
66.69
LLaMA-7B
LLM-Pruner
40.64
60.21
68.88
47.86
54.62
43.94
27.73
35.20
48.35
LoRAP |
54.19
53.85
64.74
48.58
57.30
44.28
27.30
32.00
46.86
Olica (Ours)
49.39
65.32
66.87
47.80
58.72
51.98
30.72
33.40
50.68
Dense
X
11.58
77.89
80.14
79.06
72.85
74.75
47.61
44.80
68.
LLaMA-13B
LLM-Pruner
74.62
62.35
72.74
58.43
55.88
51.89
33.02
38.20
53.22
L
34.93
57.68
70.24
59.93
62.19
54.46
31.83
36.00
53.18
Olica (Ours)
X
35.35
70.03
71.16
58.63
65.35
61.41
34.47
33.80
56.41
Dense
12.19
77.71
79.05
76.00
68.98
74.58
46.33
44.20
66.69
LLaMA2-7B
LoRAP i
X
70.33
51.87
62.08
43.78
57.54
40.78
26.79
32.20
45.00
Olica (Ours)
52.82
64.07
62.73
43.34
53.91
49.49
28.24
31,20
47.57
Dense
10.98
80.55
80.52
79.37
72.21
79.38
48.98
45.20
69.46
LLaMA2-13B
LoRAP
37.55
67.13
69.04
55.60
58.96
53.83
31.14
34.00
52.81
Olica (Ours)
X
34.21
65.96
70.67
54.63
60.30
60.31
34.04
33.60
54.21
Dense
16.11
78.41
78.56
74.68
70.09
72.01
43.77
43.40
65.85
Vicuna-7B
LLM-Prunerr
43.96
40.76
67.08
46.64
53.28
43.98
27.56
34.00
44.76
LoRAP +
82.18
48.62
63.71
47.26
55.96
42.34
28.33
31.00
45.31
Olica (Ours)
56.35
56.02
66.27
51.11
56.99
52.31
31.91
32.20
49.54
Dense
13.50
85.29
79.11
77.51
71.59
78.66
50.77
45.40
69.76
Vicuna-
LoRAP i
45.58
70.61
69.10
58.15
60.62
56.27
34.04
36.40
55.02
Olica (Ours)
39.22
72.02
70.29
57.59
61.80
62.16
36.95
35.20
56.57
LLLMM-13B: For this larger model, Olica again outperforms LLM--Puner and LoRAP, with an average of 56.41% 16, which is higher than both methods (LLM-Prumm) 53.22%, LoRAP: 53.1.%). Olica also delivers superior performance datasets like BoolQ (70.03%) and WinoG (65.35%), with relatively low PPL of 35.35.
ILLMA2-7B: Olica shows strong performance with an average score of 47.57%, significantly surpassing LoRAP (45.00%) and providing solid results across the tasks. achieves low PPL. of 52.84.
Vicuna-7B: Olica performs well here too, with an average score of 49.54%, surpassing LLM-Pruner (44.76%) and LoRAP (4). It also maintains a lowest PPL 56.35.
Vicuna-13B. At 50% sparsity, Olica outperforms LoRAP (55.02%) with an average score of 56.57%. It achieves low PPL of 39.22, showing the robustness of Olica across larger models as well.
In summary, Olica consistently outperforms both L.L.MPruner and LoRAP across all tested models, achieving better or comparable performance while maintaining competitive perplexity. These results demonstrate the effectiveness of Olica as a pruning method that preserves model performance while significantly reducing model size and computational complexity.
15


Reviews:
[{"review_id": "nPX5gjH7jp", "reviewer": "Reviewer_5bY6", "summary": "This paper proposed a new structured pruning method by treating the matrix products WqWk and also WvWo as unified entities and applying PCA, and pruning the unimportant information. They also pruned the FFN layer and introduced a linear calibration method to reconstruct the residual error with two low-rank matrices. The experimental results show that the method achieved a SOTA result.\n\n## update after rebuttal\nThe author addresses my concerns. I will keep my score as positive.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your time and efforts in reviewing our paper. We will fully address your concerns below.\n\n**Q1:** The product of $W_{q_i}W_{k_i}^{\\top}$  has already shown the low-rank property since $d_h$ is smaller than $d$. Based on this, are you trying to reduce $d_h$ into an even smaller $r$ using PCA ?\n\n**A1:**  Thanks for your question. Indeed, our orthogonal neuron decomposition (OND) method is to reduce the dimensionality of each attention head, i.e, reduce $d_h$  into a smaller $r$ using PCA.\n\n**Q2:** I am confused about Eq.7. Is $f(X)=XW$ in Eq.7? If so, this means that your target is to learn  $\\hat{f}(X)=0$, which is $\\hat{W}=0$. If not, what are you trying to learn in Eq.7? The authors should further explain this.\n\n**A2:** Sorry for the confusion. In Eq.7, the $f(X)$ is denoted as the original FFN layer. Our objective  is to recover the residual errors $E$ of pruned layers $\\hat{f}$, that is, $E=f(X)-\\hat{f}(X)$, where $f$ is the original version of $\\hat{f}$. We approximate $E$ by a linear model: $E\\approx X\\hat{W}$, and thus the pruned layer $\\hat{f}$ can be calibrated to approximate its original version: $f(X) \\approx \\hat{f}(X) + X\\hat{W}$. The $\\hat{f}$ is obtained by the algorithm detailed in the **FFN Pruning** section of our paper and both $f(X)$ and $\\hat{f}(X)$ are  fixed in Eq.7. So the objective of Eq.7 is to learn the parameters $\\hat{W}$ of the liner model  given both $f$ and $\\hat{f}$ fixed. \n\n**Q3:** There is a lack of some ablation studies. For example, adjusting the pruning ratio of FFN layers and MHA layers.\n\n**Table1:** Ablation study of different pruning ratios of FFN layers and MHA layers (LLaMA-7B).\n| MHA | FNN | PPL ($\\downarrow$) |Avg. Acc. ($\\uparrow$) |\n|-------|-------|-------|-------|\n| 20% | 20% | 15.35 |64.54 |\n| 30% | 20% | 16.68 |63.21 |\n| 20% | 30% | 19.11 |61.57 |\n| 40% | 20% | 20.85|60.82 |\n| 20% | 40% | 24.21 |58.07 |\n\n**A3**:  Thanks for this valuable suggestion. We conducted ablation studies with different pruning ratios of FFN layers and MHA layers in the Table 1. We observe that the MHA layers can tolerate a larger pruning ratio than the FFN layers, meaning that the core design of Transformers, i.e., the multi-head attention layer, is particularly redundant. This observation coincides  with existing works such as [1].\n\n[1] What matters in transformers? not all attention is needed. https://arxiv.org/abs/2406.15786.\n\n**Q4:**  There is a lack of comparison to other SOTA methods, such as OSSCAR and FLAP.\n\n**Table2:** Comparisons with FLAP (Accuracy  ($\\uparrow$)).\n|Method | PPL | BoolQ  | PIQA| HellaS |WinoG| ARC-e|ARC-c| OBQA |Avg.|\n|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|\n|FLAP | 17.0 | 69.4| 74.7| 66.9| 66.3| 64.6| 36.5 |38.2|59.5|\n| Olica (Ours) |**15.4**  | **71.6** | **77.9** |**77.3** |**70.0** |**72.1** |**42.7** |**44.2** |**64.5** |\n\n**Table3:** Comparisons with OSSCAR (PPL ($\\downarrow$) on WikiText).\n| Method | OPT-1.3B | OPT-2.7B |OPT-6.7B |\n|-------|-------|-------|-------|\n| OSSCAR | **15.38** | 13.17 | 12.79 |\n| Olica (Ours) | 16.14 | **12.78** |**11.63** |\n\n**A4:** We compare Olica with FLAP  and OSSCAR in Table 2 and Table 3 in terms of accuracy and PPL, respectively. We can see that the proposed Olica consistently outperforms FLAP with large margins. Besides, our approach surpasses OSSCAR on the larger models, i.e., OPT-2.7B and OPT-6.7B.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem or application at hand.", "theoretical_claims": "There is no proof or theoretical claim.", "experimental_designs_or_analyses": "I check the soundness and validity of experimental designs and analyses, and they are correct.", "relation_to_broader_scientific_literature": "This paper is related to structure pruning and LoRA.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\n1. This paper is well-written and technically sound.\n\n2. The way of using PCA to directly prune the output of WqWq and WvWo is somewhat novel.\n\n3. The experimental results show that the proposed method performs well.\n\nWeaknesses:\n\n1. The product of $W_{q_i}W_{k_i}^\\top$ has already shown the low-rank property since $d_h$ is smaller than $d$. Based on this, are you trying to reduce $d_h$ into an even smaller $r$ using PCA?\n\n2. I am confused about Eq.7. Is $f(X)=XW$ in Eq.7? If so, this means that your target is to learn $\\hat f(X)=0$, which is $\\hat W=0$. If not, what are you trying to learn in Eq.7? The authors should further explain this.\n\n3. There is a lack of some ablation studies. For example, adjusting the pruning ratio of FFN layers and MHA layers, \n\n4. There is a lack of comparison to other SOTA methods, such as OSSCAR [1] and FLAP [2].\n\n5. Typo error: at the end of page 1, the the -> the.\n\n[1] OSSCAR: One-Shot Structured Pruning in Vision and Language Models with Combinatorial Optimization.\n\n[2] Fluctuation-based Adaptive Structured Pruning for Large Language Models.", "other_comments_or_suggestions": "See weaknesses above.", "questions_for_authors": "See weaknesses above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RhAHmOk80I", "reviewer": "Reviewer_ZyT3", "summary": "Olica is a retraining-free structured pruning framework for Large Language Models (LLMs), with orthogonal decomposition and linear calibration. It unifies MHA matrix products and applies PCA to preserve essential information while compressing the model. A fast decomposition method reduces PCA complexity, and a linear calibration technique reconstructs residual errors in pruned FFN layers using SVD.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing the insightful comments. We will try our best to address your concerns as follows.\n\n**Q1:** The key observation of this article is that MHA depends on $W_{q_i}W_{k_i}^{\\top}$, which is not true on llama, because of RoPE. However, all experiments in this article are done on llama. This means that all the discussion in Section 3.2 doesn’t apply to llama. In Appendix A, authors claim that they apply PCA separately on $W_q$ and $W_k$. This part is quite unclear. And If PCA is separately applied on $W_q$ and $W_k$, what is the innovation of this compared with SVD-LLM [1], ASVD [2]? \n\n**A1:** Despite the separate estimation of $W_q$ and $W_k$, it cannot diminish our contribution to the estimation of $W_v$ and $W_o$. \nOur approach is different from both SVD-LLM and ASVD in the following aspects.  First, both SVD-LLM and ASVD treat the matrices $W_v$ and $W_o$ independently and propose different variants of SVD to reconstruct $W_v$ and $W_o$ individually. \nConsidering that  the reconstruction loss  $||XW_{vo} - X\\hat{W_{vo}}||$ relies on $W_{vo}=W_vW_o^{\\top}$, rather than each $W_v$ and $W_o$ individually, the proposed Olica regards the product $W_{vo}=W_vW_o^{\\top}$ as an unified entity. Although \nSVD-LLM and ASVD can gain more accurate estimations of  $W_v$ and $W_o$, the reconstruction loss of the product  $W_{vo}$ can not be guaranteed, because  it may incur the error accumulation issue induced by the product of separately estimated $\\hat{W_{v}}$ and $\\hat{W_{o}}$. As evidenced by Table 5 of our paper, the proposed method can achieve significantly better performance than separate estimations of $W_v$ and $W_o$. Moreover, we  compare the proposed Olica with SVD-LLM based on the same evaluation settings.  We obtain the following performance on three datasets: ARC\\_easy (69\\%), PIQA (78\\%) and WinoG (70\\%) using Olica, whereas the performance of SVD-LLM(W) is: ARC\\_easy (62\\%),  PIQA (71\\%), and  WinoG  (61\\%). These results clearly demonstrated the advantages of our proposed method. \n\n[1] Wang, Xin, et al. \"Svd-llm: Truncation-aware singular value decomposition for large language model compression.\" arXiv preprint arXiv:2403.07378 (2024). \n\n[2] Yuan, Zhihang, et al. \"Asvd: Activation-aware singular value decomposition for compressing large language models.\" arXiv preprint arXiv:2312.05821 (2023).\n\n**Q2:** The linear calibration turns the skip connection into weighted low-rank layer, which is quite similar to SliceGPT. SliceGPT also converts the skip connection into a multiplication of two low-rank matrices. What is the contribution of this part?\n\n**A2:**  Thanks for your insightful comments. The proposed linear calibration (LC)  is designed to calibrate the residual information of pruned layers by linear models, that is, $E=f(X)-\\hat{f}(X)\\approx X\\hat{W}$, such that the pruned layer $\\hat{f}$ can be calibrated to approximate its original version: $f(X) \\approx \\hat{f}(X) + X\\hat{W}$. To further reduce the number of parameters of $\\hat{W}$, we performed SVD on $\\hat{W}$. The SliceGPT inserts an identity matrix $I=UU^{\\top}$ into the intermediate layers of transformer so that to reduce its hidden dimension, where the orthogonal matrix $U$ is obtained by perform SVD on the feature matrix $X$. We emphasize that the SVD is a general technique and can be used for different purposes. Here, we use it for reducing the number of parameters of $\\hat{W}$, whereas SliceGPT use it for  reducing the feature dimension of $X$.\n\n**Q3:**  In paragraph of “Fast OND”, authors claim that Figure 2 shows that the low-rank structure of the product $W_v W_o^T$ can be determined by one of the $W_v$ and $W_o$. But Figure 2 only shows that the trend of singular value in $W_v$ and $W_o$ but doesn’t show that they have similar singular vectors. So the motivation of performing SVD on one of $W_v$ and $W_o$ seems not very solid.\n\n**A3:** Thanks for your insightful comments.   Since $rank(W_v W_o^T)< min(rank(W_v), rank(W_o))$, the low-rank property of $W_v W_o^T$ is upper bounded by the ranks of $W_v$ and $W_o$. If either $W_v$ or $W_o$ exhibits the low-rank structure, we immediately know that $W_v W_o^T$ is also a low-rank matrix. As a result, we only need to examine the distribution of the singular values of $W_v$ and $W_o$ to determine their low-rank structures. We will state these more clearly in the final manuscript.", "claims_and_evidence": "1.\tThe key observation of this article is that MHA depends on W_q * W_k^T, which is not true on llama, because of RoPE. However, all experiments in this article are done on llama. This means that all the discussion in Section 3.2 doesn’t apply to llama. In Appendix A, authors claim that they apply PCA separately on W_q and W_k. This part is quite unclear. And If PCA is separately applied on W_q and W_k, what is the innovation of this compared with SVD-LLM [1], ASVD [2]?\n2.\tThe linear calibration turns the skip connection into weighted low-rank layer, which is quite similar to SliceGPT. SliceGPT also converts the skip connection into a multiplication of two low-rank matrices. What is the contribution of this part?\n3.\tIn paragraph of “Fast OND”, authors claim that Figure 2 shows that the low-rank structure of the product W_v W_o^T can be determined by one of the W_v and W_o. But Figure 2 only shows that the trend of singular value in W_v and W_o but doesn’t show that they have similar singular vectors. So the motivation of performing SVD on one of W_v and W_o seems not very solid.\n\n[1] Wang, Xin, et al. \"Svd-llm: Truncation-aware singular value decomposition for large language model compression.\" arXiv preprint arXiv:2403.07378 (2024).\n[2] Yuan, Zhihang, et al. \"Asvd: Activation-aware singular value decomposition for compressing large language models.\" arXiv preprint arXiv:2312.05821 (2023).", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "This article proposes a new way of retraining-free structured pruning, which could be benefit for model compression.", "essential_references_not_discussed": "[1] Wang, Xin, et al. \"Svd-llm: Truncation-aware singular value decomposition for large language model compression.\" arXiv preprint arXiv:2403.07378 (2024).\n[2] Yuan, Zhihang, et al. \"Asvd: Activation-aware singular value decomposition for compressing large language models.\" arXiv preprint arXiv:2312.05821 (2023).", "other_strengths_and_weaknesses": "no", "other_comments_or_suggestions": "1.\tLine 25, “extracte” -> “extract”", "questions_for_authors": "1. If PCA is separately applied on W_q and W_k, what is the innovation of this compared with SVD-LLM [1], ASVD [2]?\n2.\tThe linear calibration turns the skip connection into weighted low-rank layer, which is quite similar to SliceGPT. SliceGPT also converts the skip connection into a multiplication of two low-rank matrices. What is the contribution of this part?\n3.\tIn paragraph of “Fast OND”, authors claim that Figure 2 shows that the low-rank structure of the product W_v W_o^T can be determined by one of the W_v and W_o. But Figure 2 only shows that the trend of singular value in W_v and W_o but doesn’t show that they have similar singular vectors. So the motivation of performing SVD on one of W_v and W_o seems not very solid.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cj1r3UaIFC", "reviewer": "Reviewer_9AzP", "summary": "This paper proposes Olica, a structured pruning method for large language models that eliminates the need for retraining. The approach introduces Orthogonal Neuron Decomposition to compress the multi-head attention layer using PCA-based factorization and Linear Calibration to mitigate pruning-induced errors in feed-forward networks using ridge regression. Experimental results suggest that Olica achieves competitive performance while reducing computational costs.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly thank you for the detailed reviews and helpful suggestions. We reply point-by-point here.\n\n**Q1:** The claim of \"no retraining\" is misleading.\n\n **A1:** Thanks for your question. In deep learning,  \"training'' or \"fine-tuning'' generally require a series of forward and backward processes to compute the gradients so as to update the parameters of neural networks. However, Our Olica only takes one forward process and does not involve the  backward process. In this sense, our method does not require retraining. In the literature [1, 2], they also refer this paradigm as **no retraining**.\n\n[1] Sparsegpt: Massive language models can be accurately pruned in one-shot. ICML, 2023.\n\n[2] A simple and effective pruning approach for large language models. ICLR, 2024.\n\n\n**Q2**: Comparions to one-shot magnitude-based or Taylor expansion-based structured pruning. \n\n**A2**: Sorry for the confusion. We  clarify that the baselines **LLM-Pruner** and **LoRAP** are one-shot Taylor expansion-based and magnitude-based methods, respectively. Their results have been included in Table 2 and Table 3 of  our paper.\n\n**Q3:** The performance of 30B+ model and higher pruning ratio are unknown.\n\n**Table 1**: Results of larger models and higher sparsity ratios (SR), where A=LLaMA-30B, B=LLaMA-2-70B.\n| Model (SR) | PPL ||  Avg. Acc. ||\n|-------|-------|-------|-------|-------|\n| A (0%) | 9.8 | | 72.1| |\n|  | LoRAP | Olica |LoRAP | Olica |\n| A (20%)  | 11.6 | **11.1** | 70.3 |**71.1**|\n| A (50%)  | 23.4 | **20.9** | 61.9 |**63.6**|\n| B (0%) | 9.2 | | 73.6| |\n| B (20%)  | 9.9 | **9.2**| 71.3 | **72.4**|\n| B (50%)  | 16.3 |**14.8**|65.1|**67.3**|\n\n**A3:** We further conducted experiments on models LLaMA-30B and LLaMA-2-70B. As shown in Table 1, we see that our proposed Olica consistently achieves better performance with different pruning raios and model scales,  which clearly demonstrates the scalability of our porposed Olica. \n\n**Q4**: No evaluation on challenging datasets. Throughput in tokens per second is not benchmarked.\n\n**Table 2:** Results (accuracy) on MMLU of Llama-2-13b-chat (20% sparsity).\n|Method|Humanities|Social Sciences |STEM|Other|Avg.|\n|-------|-------|-------|-------|-------|-------|\n|Dense|49.5|62.1|44.0|59.9|53.9|\n|LoRAP|41.2|49.7|36.8|48.9|44.2|\n|Olica|**43.7**|**52.5** |**38.2**|**50.8**|**46.3**|\n\n**Table 3:** Throughput in tokens per second of pruned LLaMA-30B tested on 3 RTX 4090 GPUs using vanilla HuggingFace.\n\n |Sparsity |0%| 20%| 30%| 50|\n|-------|-------|-------|-------|-------|\n|Tokens/s |14.2|16.2|16.9|18.3|\n\n**A4:** We exend the evaluation results to Massive Multi task Language Understanding (MMLU) task, which is a quiz bank covering 57 subjects, presenting a greater challenge compared to the Commonsense Reasoning datasets. From Table 2, we can still obtain better performance. We further tested the throughput in tokens of pruned LLaMA-30B. As shown in Table 3,  under 50% sparsity, the throughput in tokens of pruned model can enhance about 30%.\n\n**Q5:** Several key ideas, e.g., how ridge regression is applied, are underexplained.\n\n**A5**: As for the ridge regression, since our target is to estimate the the pruning errors by a linear model: $E=f(X)-\\hat{f}(X) \\approx X\\hat{W}$, where $\\hat{f}$ is the pruned version of $f$ and $\\hat{W}$ is the parameter matrix to estimate, a natural solution \nis the least-square estimation: $\\hat{W}=(X^{\\top}X)^{-1}E$. However, in the modern LLMs, the input matrix $X$ is extremely high-dimensional, leading to  the irreversibility of $X^{\\top}X$. To solve the problem, following the ridge regression, we add a $l_2$ penalty to the regression loss: $ ||E - XW|| + \\lambda || W  ||^{2}_{F}$. This leads\na closed-form solution: $\\hat{W}=(X^{\\top}X+\\lambda I)^{-1}E$, where $I$ is an identity matrix.\n\n**Q6:** Confidence intervals or significance tests should be provided.\n\n**Table 4:** t-test of comparison experiments of our paper.\n |Experiments| LLaMA-7B (20%) | LLaMA-7B (25%) | LLaMA-7B (33%) | LLaMA-13B (20%) | LLaMA-2-7B (30%) | Vicuna-7B (20%) |\n|-------|-------|-------|-------|-------|-------|-------|\n| p-value | 2.221$\\times 10^{-5}$ |6.358$\\times 10^{-5}$ | 2.368$\\times 10^{-6}$|2.093$\\times 10^{-4}$| 2.943$\\times 10^{-8}$| 6.036$\\times 10^{-4}$|\n\n**A6:**   To conduct significance tests, we set the null hypothesis as \"the baseline methods and the proposed Olica have the same accuracy $\\mu$\".  We conduct five random experiments by independently selecting calibration datasets, over which we record the mean accuracy, denoted as $\\hat{\\mu}$. The significance tests of p-values are reported in Table 4, and we can see that all the null hypotheses are rejected when p-value $<$ 0.01.\n\n**Q7:** Some recent works are missing. For example, [1] Search for Efficient Large Language Models.\n\n**A7:** We directly cite the results from [1]. Under the 20% sparsity, our average accuracies are: 61.10% (LLaMA-7B) and 63.73% (LLaMA-13B), whereas [1] are 59.71%  (LLaMA-7B) and 62.10% (LLaMA-13B).", "claims_and_evidence": "The paper emphasizes that Olica eliminates retraining, but the linear calibration step implicitly fine-tunes the model using calibration data. While this is not full retraining, it is still an optimization step requiring data, making the claim of \"no retraining\" misleading.\nThe authors compare Olica against pruning methods that explicitly use LoRA fine-tuning, but they do not evaluate against one-shot pruning methods that also do not retrain the model (e.g., magnitude-based or Taylor expansion-based structured pruning).\n\nSome claims, such as \"significantly reduces running time while delivering better performance,\" are not statistically verified. Confidence intervals or significance tests should be provided.", "methods_and_evaluation_criteria": "It makes sense for the application.", "theoretical_claims": "There is no theoretical claim in this paper.", "experimental_designs_or_analyses": "I check the soundness/validity of all experimental designs or analyses.", "relation_to_broader_scientific_literature": "There is no contribution of the paper related to the broader scientific literature. This paper focus on the application aspects.", "essential_references_not_discussed": "Some recent works are missing. For example, \"Search for Efficient Large Language Models\" published in NeurIPS 2024. Please compare with works published in NeurIPS 2024 or ICLR 2025 if possible.", "other_strengths_and_weaknesses": "**Strengths:**\n\n1. The authors conduct extensive experiments across multiple LLMs and benchmarks, comparing Olica against state-of-the-art methods.\n\n**Weaknesses:**\n\n1. The paper emphasizes that Olica eliminates retraining, but the linear calibration step implicitly fine-tunes the model using calibration data. While this is not full retraining, it is still an optimization step requiring data, making the claim of \"no retraining\" misleading.\nThe authors compare Olica against pruning methods that explicitly use LoRA fine-tuning, but they do not evaluate against one-shot pruning methods that also do not retrain the model (e.g., magnitude-based or Taylor expansion-based structured pruning).\n\n2. All experiments are conducted on LLaMA-7B and 13B. The scalability of Olica to models with 30B+ parameters is not demonstrated.\nThe pruning ratios tested (up to 33% sparsity) are moderate. In real-world applications, structured pruning often targets higher sparsity ratios (e.g., 50–75%), and the performance at these levels is unknown.\n\n3. The results rely on WikiText-2 and a small set of multiple-choice benchmarks. There is no evaluation on open-ended tasks, instruction following, or reasoning-heavy datasets, where pruning might degrade model coherence.\nWhile FLOP reductions are reported, actual inference speedup (e.g., throughput in tokens per second) is not benchmarked on real-world workloads.\nNo ablations compare Olica against simpler structured or semi-structured pruning baselines.\n\n5. Several key ideas (e.g., how ridge regression is applied) are underexplained. The methodology section assumes familiarity with pruning literature but does not adequately define important concepts for a broad ICML audience.\nSome claims, such as \"significantly reduces running time while delivering better performance,\" are not statistically verified. Confidence intervals or significance tests should be provided.", "other_comments_or_suggestions": "Please refer to my previous comments.", "questions_for_authors": "Please refer to my previous questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ogpa3cl9hy", "reviewer": "Reviewer_uBxH", "summary": "This work proposes using PCA to compress the matrix product in MHA with a fast computation approach for LLM compression. Additionally, to address errors caused by pruning, a reconstruction method based on ridge regression is introduced for FFN. The experiments cover LLaMA-based models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really  appreciate the time and efforts you extended in reviewing our paper. Below please find our responses regarding your concerns.\n\n**Q1**: SVD and AWSVD treat each of $W_v$ and $W_o$ independently, which might provide more accurate estimates than treating the product of the two matrices, $W_vW_o^T$.\n\n**A1**: Thanks for your question. The reconstruct loss $|| XW_{vo} - X\\hat{W_{vo} }||$ relies primarily on $W_{vo}=W_vW_o^{\\top}$, rather than each $W_v$ and $W_o$ individually. Although $W_v$ and $W_o$ can be individually estimated accurately by SVD and AWSVD, the reconstruction loss of  the product $W_vW_o^T$ can not be guaranteed, as it may incur the error accumulation issue introduced by the product of separately estimated $\\hat{W_v}$ and $\\hat{W_o}$. \n\n**Q2:** Is the gate projection pruned in the same manner as the up projection? What are the differences compared to LoRAP?\n\n**A2:** Yes, the gate projection of the FFN layer is pruned in the same manner as the up projection and this is coincide with LoRAP. Our purpose here is to design the linear calibration, i.e., approximating the residual errors $E=f(X)-\\hat{f}(X) \\approx X\\hat{W}$ by a linear model, such that $f(X)\\approx \\hat{f}(X)+ X\\hat{W}$, no matter how $f$ is pruned to $\\hat{f}$. In principle, any efficient pruning methods for $f$ can be adopted here. \n\n**Q3:** Could you provide additional experimental results on higher pruning ratio?\n\n**Table 1:** Results of larger models with 50% sparsity ratios.\n|Method|PPL|Avg. Acc.|\n|-------|-------|-------|\n|LLaMA-30B (dense)|9.8| 72.1|\n|LoRAP|23.4| 60.9 |\n|Olica (Ours) |**20.9**|**63.6**|\n|LLaMA-2-70B (dense)|9.2 |73.6|\n|LoRAP|16.9|64.7|\n|Olica (Ours)|**14.8**|**67.3**|\n\n**A3:** We have demonstrated the results of 50\\% sparsity ratio in the Appendix D of our paper, including 7B and 13B models. Here, we further scale the model parameters up to 30B and 70B. From Table 1, we can observe that our proposed method still works better than LoRAP under higher pruning ratio and larger models. As for the fine-tuned results of these pruned larger models, we leave to future works due to the resource constraints.  \n\n**Q4:** Results of other models (e.g., OPT, Qwen, and Phi) and MoE-based architectures.\n\n**Table 2:** Results of 20% sparsity ratio.\n |Method|PPL (A)| Avg. Acc. (A)|PPL (B)| Avg. Acc. (B)| PPL (C)| Avg. Acc. (C)|\n|-------|-------|-------|-------|-------|-------|-------|\n|Dense|23.73|69.58 |11.91|73.18|9.33|74.49|\n|LoRAP|42.78| 58.57|25.34|64.23|12.87|69.21|\n|Olica (Ours)| **39.77**|**61.78**|**21.15**|**66.67**|**11.28**|**70.95**| \n\n**A4:** We further extended the evaluation results to LLMs Phi-2 (A), Qwen2.5-14B (B), and Mixtral-8x7B-v0.1 (C), where Mixtral-8x7B-v0.1 is a MoE-based model. As shown in Table 2, we see that our approach consistently outperforms LoRAP, showing high applicability and superiority. \n\n**Q5:** What are the details of calculating the latency? Can this method achieve speedups with a batch size of 1 for both prefilling and decoding stages?\n\n**A5:** Indeed, following the evaluation protocols of works LLM-Pruner and LoRAP, we mainly tested the speedup of the prefilling state, where the vanilla HuggingFace is used and the batch size is set to 2. As for the decoding stage, please see Table 3 in the **Q4** raised by **Reviewer 9AzP**, we can see that under 50% sparsity, the throughput in tokens of pruned model can enhance about 30\\%. \n\n**Q6:**  Could you explain why the proposed calibration method is considered superior to A, B, and C?\n\n**A6:** Both A and B propose to adjust the combination of the remaining filters to minimize the reconstruction error. However, if the pruned filters contain information not carried by any remaining filters, this adjustment is insufficient. Our proposed approach directly models the residual errors, complementing the remaining filters by introducing new information they do not capture. As for C, it uses a constant baseline to compensate for the pruned filters, which may reduce the accuracy. As shown in Table 3  presented in **A7**,  FLAP performs significantly worse than the proposed Olica.\n\n\n**Q7:** Comparisons with Shortened LLaMA, FLAP, and SLEB (LLaMA-7B).\n\n**Table 3:**  Comparison results.\n|Method |PPL |Avg. Acc.|\n|-------|-------|-------|\n|FLAP|17.0|59.5|\n|SLEB|18.5|57.6|\n|Shortaned LLaMA: Grad+|20.2|63.5|\n|Shortaned LLaMA: PPL|17.7|61.9|\n|Olica (Ours)|**15.4**|**64.5**|\n\n**A7:**  From Table 3, we see that our proposed Olica consistently outperforms these baselines. We do not compare with methods Sheared LLaMA and Minitron due to the unfairness, as they require huge computation and data resources to retrain the pruned model. For example, to retrain a 2.7B model, Sheared LLaMA requires 50B tokens and 16 GPUs. In contrast, our proposed method is highly efficient, enabling pruning models with more than 70B parameters on a single NVIDIA GeForce RTX 4090 GPU with less than an hour runtime.", "claims_and_evidence": "I think the claims of this paper are well supported by clear and compelling evidence. This work introduces a computationally efficient pruning method for compressing LLMs. The concept of applying PCA to matrix products in MHA is interesting, and further performing SVD on one of the product matrices for fast computation appears effective. I appreciate the interesting comparison in Tables 5 and 6.", "methods_and_evaluation_criteria": "I believe the proposed method and setup are well-aligned with previous LLM pruning studies. Although numerous methods exist for evaluating LLMs, considering this work's focus on compression, employing metrics such as PPL along with LLaMA/LLM-Pruner's benchmarks appears to be sufficient.", "theoretical_claims": "I believe this paper clearly presents the method in general, including matrix products for MHA and related SVD-based methods, as well as regression-based FFN compression.", "experimental_designs_or_analyses": "* I am a bit unclear why OND and FastOND outperform SVD and AWSVD in Table 5. In my understanding, SVD and AWSVD treat each of W_v and W_o independently, which might provide more accurate estimates than treating the product of the two matrices, W_vW_o^T. Could you provide additional explanation and/or analysis for this?\n* Why is the importance score for the gate projection of the FFN missing in Equation (5)? How is the pruning of the FFN’s gate projection handled in the LLaMA family (e.g., is it pruned in the same manner as the up projection)? Could you also elaborate on the differences and merits of this approach compared to LoRAP?\n* The pruning ratios explored in this work appear relatively low. Is this approach still effective at higher pruning ratios, for instance, exceeding 50%? Based on my experience, pruning only 20-30% of weights typically does not yield significant computational efficiency gains. Therefore, I personally prefer the approach of extreme compression (beyond 50%), along with subsequent retraining, as it is more effective for achieving notable speedups and substantial memory reductions. Could you provide additional experimental results or insights on this topic?\n* The models used for experimental validation in this work appear quite limited, as they are confined to the LLaMA family. I would recommend including experimental comparisons with other models (e.g., OPT, Qwen, and Phi) and MoE-based architectures alongside existing methods. This would help verify the broad applicability and superiority of this work.\n* I think the methodology for calculating latency gains in Table 4 should be described in more detail, as width pruning has been reported to be challenging for achieving actual speedups in certain setups (e.g., ZipLM, Shortened LLaMA). I suspect that the speedup reported in Table 4 may result from measurements taken only during the prefill stage, excluding decoding, and possibly under a large batch size. Could you specify what kind of framework (such as Vanilla HuggingFace, VLLM, etc.), batch size (1 or more), and output generation length were used? Additionally, can this method achieve speedups with a batch size of 1 for both prefill and decoding stages?\n  - [ZipLM] https://arxiv.org/abs/2302.04089\n  - [Shortened LLaMA] https://arxiv.org/abs/2402.02834\n* I believe that the baseline comparison could be enhanced by including the studies I outlined in the section 'Essential References Not Discussed' below.", "relation_to_broader_scientific_literature": "I think compressing LLMs, particularly with reduced computation, is a hot topic, and this work has done a good job addressing it. Several ablations presented in this work demonstrate that naively applying SVD-based methods does not perform well, while employing a Wanda-based importance criterion and considering matrix products appears effective. Although the concept of FFN compression with reconstruction-based loss seems quite similar to previous work, the further developments with layer selection and low-rank approximation are interesting.", "essential_references_not_discussed": "* In terms of FFN pruning, several retraining-free methods (e.g., A, B, and C below) have already been proposed. Particularly, the concept of reconstruction recovery, defined as reducing the difference between activations from the original weights and those from compressed weights, is widely applied these days (e.g., B and C). Could you explain why the proposed calibration method is considered superior to existing methods?\n  - [A] A Fast Post-Training Pruning Framework for Transformers https://arxiv.org/abs/2204.09656\n  - [B] Gradient-Free Structured Pruning with Unlabeled Data https://arxiv.org/abs/2303.04185\n  - [C] Fluctuation-based Adaptive Structured Pruning for Large Language Models https://arxiv.org/abs/2312.11983\n\n* This work concentrates on the width pruning of LLMs. However, it appears to lack a thorough discussion or comparison with other pruning methods: width pruning (FLAP, Minitron), depth pruning (ShortGPT, Shortened LLaMA, SLEB, Minitron), and hybrid width-depth pruning (Sheared LLaMA). A comparison with FLAP, in particular, seems essential given that the primary advantage of this work is its retraining-free approach, and the concept of reconstruction loss recovery looks similar.\n  - [FLAP] https://arxiv.org/abs/2312.11983\n  - [ShortGPT] https://arxiv.org/abs/2403.03853\n  - [Shortened LLaMA] https://arxiv.org/abs/2402.02834\n  - [SLEB] https://arxiv.org/abs/2402.09025\n  - [Sheared LLaMA] https://arxiv.org/abs/2310.06694\n  - [Minitron] https://arxiv.org/abs/2408.11796", "other_strengths_and_weaknesses": "* Proper analyses have been conducted with in-depth ablation studies, though the experiments cover solely the LLaMA family.\n* Overall, the paper is well-written and easy to follow.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please refer to <Experimental Designs Or Analyses> and <Essential References Not Discussed>.\n\nI appreciate the authors' efforts to compress LLMs with reduced computation and generally like the main idea and flow of this paper. However, my overall opinion of this work is borderline, primarily due to some unclear aspects of the results and method, coupled with limited experimental validation. While I do not favor reviews that demand excessive experiments (often appearing to seek reasons for rejection), I believe that additional results could significantly strengthen the value of this work. Given that light computation for compression is a major advantage of this work, extending the experiments would be feasible. I hope my comments will contribute to enhancing the impact of this study.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This manuscript proposes a methodology for compressing transformer architectures based on SVD compression of certain matrices. To correct for errors introduced during the compression a calibration step is used. The method is efficient and performs well on the, somewhat limited, set of evaluations. Nevertheless, I am sensitive to the issues of requiring/requesting excessive experiments given the breadth of work in the area. Moreover, the rebuttal phase did partially alleviate some of these concerns.

The reviewers do not identify major shortcomings (rather the manuscript will simply get stronger with more experiments), and the primary strength of the manuscript is as a well executed empirical demonstration of a simple methodology that leverages common tools. Therefore, my inclination is positive—there are good ideas here that may form the basis for future study and incorporation into other methods. 

Assorted notes:

The section on MHA pruning conflates eigenvectors and singular vectors. This needs to be corrected. 

As the reviewers have pointed out, the fast OND part needs some clarification. The suggestions below address the concerns, but as written the procedure could certainly over-estimate the numerical rank.

Per the reviewers comments, some of the presentation can be improved and it should be.
Decision: Accept (poster)