Streamlining Redundant Layers to Compress Large Language Models
Authors: Xiaodong Chen
Yuxuan Hu
Jing Zhang
Yanling Wang
Cuiping Li
Hong Chen
Venue: ICLR 2025 Spotlight (2025)
Keywords: large language models
model compression
structured pruning
Primary area: generative models

Abstract: This paper introduces LLM-Streamline, a pioneer work on layer pruning for large language models (LLMs). It is based on the observation that different layers have varying impacts on hidden states, enabling the identification of less important layers to be pruned. 
LLM-Streamline comprises two parts: layer pruning, which removes consecutive layers with the lowest importance based on target sparsity, and layer replacement, a novel module that trains a lightweight network to replace the pruned layers to mitigate performance loss. Additionally, a new metric called stability is proposed to address the limitations of the widely used accuracy metric in evaluating model compression. Experiments show that LLM-Streamline outperforms both previous and concurrent state-of-the-art pruning methods in terms of both performance and training efficiency. Our code is available at \href{https://github.com/RUCKBReasoning/LLM-Streamline}{this repository}.

less important Published as a conference paper at ICLR 2025
STREAMLINING REDUNDANT LAYERS TO COMPRESS LARGE LANGUAGE MODELS
Xiaodong Chen Yuxuan Hu? Jing Zhang Yanling Wang", Cuiping Li Hong Chen
1 Engineering Research Center Database and Business Intelligence, MOE, China
2 School of Information, Renmin University Chijing China
3 Key Laboratory of Data Engineering and Knowledge Engineering, MOE, China
4 Zhongguancun Laboratory. China
cchenxiadoong huyuxuan1 999, zhang- jing, licuiping, chong) @ruc. edu. en wangy1ezgelab. edu.
ABSTRACT
This paper introduces LLM--Steamllnn, pioneer work on layer pruning for large language models (LLMs). It is based on the observation that different layers have varying impacts on hidden states, enabling the identification layers to be pruned. LLM-Streamline comprises two parts: layer pruning, which removes consecutive layers with the lowest importance based on target sparsity, and layer replacement, novel module that trains lightweight network replace the pruned layers to mitigate performance loss. Additionally, a new metric called stability is proposed to address the limitations of the widely used accuracy metric in evaluating model compression. Experiments show that MLMM-rreamline outper- forms both previous and concurrent state-of-the-arr pruning methods in terms of both performance and training efficiency. Our code is available at this repository.
100
Ma
Laveri
Highewight
Lighteightt Network
78.0
78.0
Redutive
Layer
Loss
(SE
(1) Lap Praning
Traising Lightweight Network
(3) Layer Reglacement
The left side of the figure illustrates the LLM-Streamline workflow, which includes layer pruning to remove consecutive layers and layer replacement where a lightweight network is trained replace the pruned layers. The right side of the figure presents the comparison results of LLM-Streamline with the statc-of-the-an (SOTA) pruning methods on 12 classification benchmarks (details in Section 4.2) after pruning about 25% of the parameters on Llama2-7B. L.MM-Stremlinee achieves 1 11.2% higher relative accuracy than these methods, where the relative accuracy represents the percentage of the original model's accuracy retained by the pruning method.
Figure 1:
INTRODUCTION
Large language models (LLMs) built on the Transformer architecture (Vaswani et al., 2017) have gained widespread attention and are applied across diverse domains and tasks. However, as LLMs increase in size, their hardware requirements escalate substantially, thereby constraining their applica- bility and impeding their deployment real-world scenarios. To reduce the hardware requirements
These authors contributed equally.
Jing Zhang the corresponding author.

---

Published as a conference paper at ICLR 2025
0PT-1.88
1-10
0PT-278
Llma2-78
Sinallerty
OPT-77
1-10"
1-10
10
15
20
25
30
Layer
Figure 2: The cosine similarity between the input and output hidden states of each layer in OPT-1.3B, OPT-2.7B, OPT-6.7B 7B, and Llama2-7B.
for deploying LLMs, research efforts have been devoted to developing compact models that maintain high performance through model compression (Zhu et al., 2023; Wang et al., 2024). Currently, model compression techniques are widely categorized into knowledge distillation (Hinton et al., 2015; Gou et al., 2021; et al., 2022; Huang et al., 2022; Ho al., 2022), quantization (Liu et al., 2021; Gholami et al., 2022: Dettmers et al., 2022), and pruning (Louizos al., 2017: Chen et al., 2023; Frantar Alistarh, 2023; Das et al., 2023; Sun et al., 2023; Xia et al., 2023). Knowledge distillation achieves compression by transferring the capabilities of a larger teacher model to smaller student model. Quantization compresses the model by quantizing the weights to lower precision. Alternatively, pruning compresses the model by eliminating unimportant parameters and modules.
This work focuses on the popular pruning methods. Previous approaches for LLM primarily prune dense matrices (Ashkboos al., 2024), attention heads (Michel et al., 2019; Voita et al., 2019), filters (McCarley et al., 2019; Prasanna et al., 2020), or prune parameters to reduce an L.M's hidden dimension (Xia et al., 2023; van der Ouderaa et al., 2023; Hu et al., 2024). Despite their effectiveness, these methods often result in structural irregularities, making it inflexible to store and deploy the pruned models. In contrast, layer pruning method simply reduces the depth of LLMs. As the layers of LLMs are stored in data structures like nn. nn. ModduleLisst layer pruning only requires removing elements from this list, making more flexible for application. Therefore, exploring an effective layer-wise pruning method is crucial.
The core idea of layer pruning to identify and remove less important layers in an L.M.M Specifically, the effect of each layer can be viewed as a transformation of the hidden states. If the input and output hidden states of a particular layer are highly similar, such as exhibiting high cosine similarity, we can say that the layer has small impact on adjusting the hidden states. As illustrated in Fig. 2, our pilot study shows that certain contiguous layers indeed have smaller impact on the hidden states, indicating they are less important and suitable for pruming. Some concurrent works (Song et al., 2024; Kim et al., 2024; Yang et al., 2024; Men et al., 2024; Gromov et al., 2024) also explore layer pruning. These studies either prune unimportant layers directly without further training (Song al., 2024; Men et al.. 2024) or fine-tune the pruned model to enhance performance (Kim et al., 2024; Yang et al., 2024; Gromov et al., 2024). Directly removing layers can lead more performance degradation. While parameter-efffcciet fine-tuning techniques like LoRA (Hu et al., 2021) are used to train the pruned LLM, fine-tuning the model make the original non-contiguous layers compensate for the performance degradation is not an easy task (details in Section 2.3).
In this work, we propose layer pruning method called LLM- LLM-Streamline, which exhibits advantages in both performance and training efficiency while requiring less training data. I.LM-Streamline comprises two components: layer pruning and layer replacement. According to a certain target sparsity, the first step removes consecutive layers with the lowest importance from the original LLM. Subsequently, we train a lightweight network to replace the pruned layers, aiming to recover the performance degradation caused by pruning. We employ various architectures for this lightweight network, including a feed-forwar forward neural network (FFN), a SwiGLU-based feed-forward neural network (SwiGLU), and Transformer layer.
Additionally, we find that existing accuracy metrics for evaluating model compression methods have limitations. Specifically, in natural language understanding tasks that involve multiple-choice
2

---

Published as a conference paper at ICLR 2025
classification, compressed model may guess correct answers for samples which the original model was uncertain, resulting in an overestimation of the compression performance. To address this issue, propose new metric named stability, which measures the consistency of predictions before and after pruning, considering the prediction confidence of the original model.
Overall, this paper makes the following contributions:
We propose LLM--Stramlline, a layer-wise pruning algorithm that demonstrates superior effectiveness and efficiency compared to concurrent methods. To mitigate the potential performance degradation caused by pruning, we propose to use a lightweight network to approximate the functionality of the pruned layers.
We propose a new metric called stability, which considers both the prediction confidence of the original model and the consistency of predictions before and after pruning. Stability provides more accurate reflection of the pruned model's performance in classification tasks compared to the widely used accuracy metric.
We conduct experiments on 12 well-known classification benchmarks and generation benchmarks. Our results show that for an LLM with 7B or 13B parameters and a 25% pruning rate, we can maintain 93% performance in classification tasks and 77% in generation tasks without requiring lot of training data, outperforming existing SOTA pruning methods.
2 LLM-STREAMLINE
The workflow of the LLM- Streaammeee framework, shown in Fig. 1 (a), comprises two main steps: layer pruning and layer replacement. First, we prune redundant layers from the LLMs. Then, we train lightweight network replace the pruned layers restore the model's performance.
2.1 LAYER REDUNDANCY LLMS
LLMs primarily utilize Transformer architecture, consisting of series of Transformer decoder layers. These layers adopt a residual structure, so the effect of each Transformer layer can be viewed transformation of the input hidden states. Assuming that the parameters of the ć-th layer f are denoted as ((0) and its input hidden states represented by x(f), the layer f can be expressed
x (f+1) x + f))() ,000).
(I)
In Eq. 1. the L-th layer| f contributes a transformation (xx(11 g(0) to the input (() Therefore, we assess the importance of each layer LLMs by evaluating its impact on the input hidden states. We
use the cosine similarity cos (-.)) between input (*) and output (f++1) as a metric. Essentially, higher cosine similarity between the input and output of a layer indicates lower importance, and vice
versa. This interpretation arises from the observation that a high cosine similarity suggests the layer's
transformation is minimal, making it more amenable to pruning.
To measure the importance of different layers in LLMs, we randomly select samples from the pre- training data (details in Section 4.1). We then record the hidden states generated by the LLMs for these samples and compute the cosine similarity between the input and output hidden states each layer. The computation of cosine similarity can be formalized as follows,
(E+1)
cos (1) x (f+1)) E
(e ((+1))ED
1| (2+1)
1
where D denotes the recorded hidden states from different samples, x²0) x (l+1) RdxL denotes the input and output hidden states of the i-th sample respectively, denotes the hidden size and denotes the sequence length of each sample.
To mitigate the effects of model size and model structure, we conduct experiments on four models OPT-1.3B, OPT-2.7B, OPT-6.7B (Zhang et al., 2022), and Llama2-7B (Touvron et al., 2023). The results, illustrated in Fig. 2, show that there high cosine similarity between the input and output of several consecutive layers in all models, indicating low level of importance.
3

---

Published as a conference paper at ICLR 2025
Discussion I: Why not use other similarity to measure the importance of layers? In deep learning, cosine similarity is widely employed to measure the similarity between two vectors (Chen et al., 2020; Chen & He, 2021; Reimers, 2019). Alongside it, dot product and Euclidean distance are also utilized, but they additionally consider vector magnitude. Current research suggests that the hidden states of Transformers with the Pre-Norm architecture tend to grow the depth of layers increases (Liu et al., 2023). This trend leads to a bias where deeper layers in the model have higher dot product similarity, while earlier layers have smaller Euclidean distances. Consequently, we opt for cosine similarity, which is agnostic to the magnitude of the vectors.
Discussion II: Why not use perplexity as the metric measure the importance of layers? Some concurrent layer pruning work uses perplexity as the metric to measure the importance of layers (Song et al., 2024; Kim et al., 2024). Specifically, they remove each layer one at time, measuring the change in perplexity of the model on the pre-training data, and eliminate the layer that causes the least change. This process is repeated multiple times remove several layers. However, we think perplexity is highly data-sensitive metric, which results in different layers being removed when pruning with different pre-training data. This also results in a situation where, although the perplexity of the pruned model on the pre-training data used for pruning is low, it performs poorly on other datasets. In contrast, the cosine similarity is highly stable and always leads to the same pruned layers on different pre-training data. We conduct detailed experiments in the Appendix A demonstrate that perplexity is a highly data-sensitive metric and performs poorly on downstream tasks.
2.2 LAYER PRUNING
As Fig. 2 shows, the less important layers are often contiguous. Hence, given number of pruned layers n determined by a target sparsity, we remove n contiguous layers by finding the initial layer (n) corresponding to the highest cosine similarity for pruning:
E (m) = arg max cos( x (t+*)))
where randomly select samples from the pre- pre-training data compute the cosine similarity between (t))) and (f-mn), as outlined in Eq. 2.
2.3 LAYER REPLACEMENT
After the layer pruning process, we aim to replace the pruned layer with a lightweight network that has much fewer parameters. The rationale is that these layers contribute only minor transformations to the input. Therefore, we hypothesize that the cumulative effect of these layers can be approximated by a lightweight network. Specifically, after identifying the initial layer E (n) for pruning, we use (ax(")) (g((+na) as the training data train the lightweight network using mean squared error (MSH) loss, which can be formalized as follows:
min E (o!! ((**))) MSE(H(z) (7)) x )z
((*+n)).
where h denotes the lightweight network, D denotes the recorded hidden states of samples.
Discussion:
Layer Replacement or Fine-Tuning Pruned LLMs? Here, we discuss why opt for layer replacement, instead of using common Crrameter-Efticienn Fine-Tuning (PEFT) methods such as LoRA (Hu et al., 2021) and QLoRA (Dettmers et al., 2023) after layer pruning.
First, from the perspective of resource overhead, layer replacement is more adaptable to hardware resource constraints compared to other methods. Fine-tuming the model using the PEFT methods requires storing the model's 'ss weights, activation values, and the optimizer state of the PEFT module in the GPU during training. In contrast, layer replacement involves two stages: dataset construction and model training. The first stage only requires storing the model's weight and the forward computation overhead, and the second stage only requires storing of the lightweight network's 's weight, activation values of lightweight network, and the optimizer state of lightweight network. Therefore, layer replacement can also be implemented under conditions of hardware resource constraints.
Second, layer replacement uses lightweight network to replace the pruned layer. and distills the knowledge of the pruned layer into the lightweight network using the MSE loss function. Unlike layer replacement, we speculate that training the model after pruning with LoRA is process of
4

---

Published as a conference paper at ICLR 2025
Table 1: (a) Number of samples in TP, FN, and TN. (b) The PPL standard deviation results (x (x10-3) for Llama2-78 7B and its pruned version on Race-H.
Datasei
#TP
#FN
#FP
#TN
Model
TP
FN
FP
TN
C3
543
257
210
815
CHID
269
177
993
Llama2-78
1.12
0.87
0.94
1.02
w/ pruniigg
1.13
0.84
0.88
0.92
Race-H
938
305
383
1902
redistributing the function of the pruned layers across the remaining layers. Therefore, substituting the pruned layers with lightweight network could be a less challenging training task than redistributing the function of the pruned layers across the remaining layers. the experiments Section 4.7, we demonstrate that layer replacement has better performance compared to LoRA.
3 METRICS FOR EVALUATING PRUNED MODELS
Accuracy is the most commonly used metric for evaluating LLMs in classification tasks. However, accuracy may overestimate the performance of the model after compression, since does not take into account the consistency of the model's ' answers before and after compression. In this section, we analyze such limitation and propose novel metric for evaluating compressed models.
3.1 SHORTCOMING OF ACCURACY METRIC
When evaluating the natural language understanding capabilities of L.Mss, most existing benchmarks frame the task classification task. A classification task with k: choices and comprising N samples is denoted as T = {TT; Ci.1, C2,2, .... Cr,:: 34 )i) where represents the question in the i-th sample, Crj represents the j-th choices, and yi represents the correct choice. The input the classification task consists of a question accompanied by multiple choices, and the LLM is required to select the correct answer from these choices. Typically, each choice concatenated with the question to form multiple sentences, and the perplexity (PPL) of each sentence is computed. The choice corresponding to the sentence with the lowest PPI. is selected as the answer.
Typically, model pruning results in decreased model performance. However, when we evaluate the model pruned by the method described in See 2.2, we unexpectedly observe the accuracy of the pruned model has been improved on some classification tasks. We define M to denote the original LLM, M to denote the compressed LLM, and ((M) to denote the choice predicted by the model M. To further investigate this phenomenon, matrix (Li et al., 2024). Specifically, we count the number samples and average standard deviation (std) for the PPL of the samples for each term of the confusion matrix. The calculation of the std for the PPL of the i-th sample is defined as follows:
analyze the experimental results using the confusion
E PPT... k
(PPL.. E
PPT.,)
(5) where PPL,,, denotes the PPL for the sentence created by question 2; and choice C, of the model before pruning, std, denotes the std for PPL of the i-th sample. A higher std, value indicates the LLM exhibits greater confidence in answering the question Tj.
PPL., PPL(M(Es c.))),
std,
Each term of the confusion matrix defined as follows,
TP (M) y4 A g. (M) = ys] is set of samples where the model answers correctly both before and after pruning.
FN (M) y A (M()) * yi] is set of samples where the model answers correctly before pruning but incorrectly after pruning.
FP (M) * is A (M()) = ys] is set of samples where the model answers incorrectly before pruning but correctly after pruning.
TN (M) # y; A (M) * yi] is a set of samples where the model answers incorrectly both before and after pruning.
5

---

Published as a conference paper at ICLR 2025
Table I presents the counts of samples in TP, FN, FP, TN in sevaral datasets, and also the PPL standard deviation in Race-H H. dataset. We can observe that the std for TP and TN is significantly higher than that for FN and FP. This indicates that the model is more uncertain about the FN and FP samples. In addition, the samples in FP constitute a considerable proportion of the total samples, implying that the model may guess the correct answer for significant portion after pruning. This phenomenon suggests that the accuracy metric may overestimate the performance of the compressed model.
3.2 STABILITY METRIC
We propose a novel metric stability to evaluate the performance of LLMs after pruning, i.e.,
Stability.M, E (exp (std.) (ETPUTN)) Σ exp (std;)
(6)
where the identifier [[jETPUTN]] is used to indicate whether the i-th sample belongs to TP and TN. We use std, the weight of the i-th sample. Because the std of different samples varies significantly, to mitigate the influence of samples with excessively large standard deviations, we apply the exp function to moderate the weight differences among samples. Different from accuracy, stability focuses on the model's ss confidence in its answers and the consistency between the model before and after pruning on task, aligning more closely with the goal of model pruning, i.e., ensuring the pruned model remains as similar as possible to the original model.
4 EXPERIMENTS
In this section, first compare our proposed method, LLM-Streamline, with several popular pruning methods to demonstrate its effectiveness (4.4). Next, we analyze the impact of different sizes and structures of lightweight networks on model performance (4.5) and evaluate performance under various pruning ratios (4.6), Finally, we compare our layer replacement approach with the well- known PEFT method, LoRA (Hu al., 2021) (4.7), showing that layer replacement offers superior performance and reduced memory overhead.
4.1 SETUP
We conduct experiments on popular open- source LLMs, including Llama2-7B and Llama2- 13B (Tou- vron et al., 2023). Following previous work (Men et al., 2024; Yang et al., 2024), we perform experiments with 25% pruning rate and extract data from pre-training dataset SlimPajama which contains data from different domains for layer pruning and layer replacement. Sheared LLaMa (Xia al., 2023) finds that the performance degradation of pruned models varies across different domains, and proposes determining the distribution of data from different domains based on the degree of performance degradation. Therefore, we randomly sample the data based on the distribution used by Sheared LLaMa (Xia et al., 2023), finally constructing the dataset containing 30,000 pieces of data. We randomly select 500 samples from this dataset and input them into LLMs, generating Fig. and use these 500 data samples for layer pruning. All 30,000 pieces of data are used to train the lightweight network. We utilize two types of lightweight networks: (FFN), referred to as Ours (FFN), and Transformer Layer, referred to as Ours (Layer). The FFN is randomly initialized, while the Transformer Layer inherits the parameters from the first pruned layer. Additionally, we explore a purely pruning approach without incorporating any lightweight network, denoted as Ours (None). Further experimental details are available in the Appendix C.I.
Neural Network
4.2 BENCHMARK
We use 12 natural language understanding benchmarks for evaluation: CMNLI (Xu et al., 2020), HellaSwag(HeSww (Zellers et al., 2019), PIQA (Bisk et al., 2020),CHID (Zheng et al., 2019), WSC (Levesque et al., 2012),0 CommonSenseeQAACCCCc. MMM (Hendrycks et al., 2020), CMMLU (Li et al., 2017), C3 (Sun et al., 2020). The tasks in these benchmarks are formalized as classification tasks, so refer to these benchmarks classification benchmarks. For these benchmarks, we use both
(Talmor et al., 2018), BoolQ (Clark et al., (Lai et al.,

---

Published as a conference paper at ICLR 2025
Table Accuracy pruning methods on classification benchmarks. "** indicates that we refer to the results in the original paper. Retained performance (RP) represents the percentage of the original model's performance retained by the pruning method.
LLM
Method
Ratio
CMNTI
CHID
BuslQ
WSC
CoQA
Bonchmeek H:SW
P9QA
Racp-M
Rasll
MMLII
CMMLU
Average
RP
76.
49.3
100.
33.4
25.4
78,0
ShasciP
370
Ubnu2-78
LaCa*
21.0%
10.7
34.4
36.1
64.1
-0.4
45.3
55.3
698
25.6
22.5
26.5
25.3
40.3
61.9
Ous (None)
24.0%
402
36.4
21.5
62.3
30.5
51.3
59.3
690
25.2
34,7
44.5
24.5
44.8
89.4
Ous (FEN)
25.0%
40.7
3500
22A
05.5
35.5
60.6
01.2
71.2
3K0
367
47.0
31.7
45.8
93.1
(Lays)
33.0
24.1
67.5
61.1
21.5
34.8
33.3
45.5
45.2
91.9
34.4%
39.5
33
29.5
580
47.1
43.3
$4.3
72.7
31.
22.5
25.2
34.9
18.6
62.3
SlkeeGFT
23.6%
35.6
30.5
15.3
37.8
42.3
38.3
45.5
61.9
26,0
25.0
33.6
25.6
34.9
61.1
LaCo?
24.0%
32.
46.1
64.0
529
52.7
64.4
74.3
56.6
34.5
45.9
31.8
51.3
900
Oux
54.5
95.6
33
74.0
350
751
3KD
531
9822
Table 3: Stability of pruning methods on classification benchmarks. The stability of the original model is 1.0, because stability is measured by comparing the prediction results of the original model.
LLM
Method
Ratio
CMNILI
BoulQ
CoQA
Benchmacks
Hu:W
PIQ
Rax--M
CMMLU
Average
1amu2-78
24.0%
87.5
74.7
80.7
81.0
70.5
73.2
Ours (ITN)
25.0%6
79.5
160
83.1
83.3
60.7
$5.7
64.5
35.1
$7.5
82.5
83.7
Oun (Layer)
24.09
79.8
100
64.4
86.3
952
BL.7
85.3
85.6
31.3
7070
$2.4
71.0
X2.2
1LMPanee
24.4%
71.6
100
69.2
70.5
65.4
66.5
77.8
86.7
42.3
35.6
48.1
53.3
65.8
SlicGGPT
23.6%5
62.2
59.5
51.8
27.1
68.3
55.5
60.9
75.8
45.3
43.4
52.7
52.9
54.1
Limma 130
Ous (Nusg)
24.68
$4.2
90.9
7188
77.4
46.2
82.2
85.7
86.5
53.3
$3.6
39.1
83.8
81.1
Dur (FFN)
25.45
55.7
100
72.5
79.8
59.6
89.2
39.4
89.7
54.8
53.3
93.6
90.7
M4.9
Dur (Lager)
24.05
87.4
100
58.6
89.0
90.3
983
54.3
92.5
63.5
accuracy and stability metrics for evaluating the models. Additionally, we include 3 benchmarks: XSum (Narayan et al., 2018), GSM8K (Cobbe et al., 2021) and StrategyQA (Geva et al., 2021), to demonstrate the LLM's performance on generation tasks after pruning. We refer these tasks as generation benchmarks. Following the evaluation framework of OpenCompass (Contributors, 2023), we use accuracy evaluate Strategy QA and GSM8K, and use ROUGEI to evaluate Xsum.
4.3 BASELINE
We compare several pruning methods that prune the attention heads, the filters of the FFN layer, and hidden dimension, as well as the concurrent layer-pruning methods LaCo. addition, ShortGPT and UIDL (Men et al., 2024; Gromov et al., 2024) be considered as variant of our approach, i... Ours (None). We also discuss layer pruning methods which use perplexity as the metric, such as SLEB (Song et al., 2024), in Appendix A.
LLM-Pruner (Ma et al., 2023) prunes attention heads, FFN layer filters, and hidden dimensions by using gradients and activations to estimate the importance of these modules.
SliceGPT (Ashkboos et al., 2024) prunes hidden dimensions. It inserts dimensionality reduction matrices into the model and employs Principal Component Analysis (PCA) initialize and compress the matrices, and then merge them with the original weight matrix to reduce the model's ''s size.
LaCo (Yang al., 2024) prunes layers by dividing the layers into groups, each consisting of multiple consecutive layers, and compresses them separately, whereas our method simply compresses a piece of consecutive layers. LaCo merges consecutive layers by averaging their parameters whereas train an additional lightweight network to replace these layers.
4.4
MAIN RESULTS
We present accuracy and stability for different methods on the classification benchmarks in Table and Table 3, respectively, and Table 4 for the generative benchmarks. The results demonstrate that our proposed LLM S.MM-Stramline consistently outperforms the baseline methods. Specifically, in classification tasks, LLM-Streamline surpasses LLM-Pruner by 7% in accuracy and 12% in stability on Llama2-7B, and by 16% in accuracy and 19% in stability on Llama2- 13B. LLM-Streamline also surpasses LaCo by 5% in accuracy Llama2-7B. For generation tasks, --te-atra retains
7

---

Published as a conference paper at ICLR 2025
Table 4: Evaluations on generation benchmarks. "** original paper.
indicates that we refer to the results in the
Benchmarks GSMSK
LLM
Method
Ratio
Xsum
StrategyQA
Average
RP
Dense
0.00%
19.4
16.5
60.2
32.0
100.0
LLMPruner
24.8%
16.4
0.61
44.2
20.4
63.8
SliceGPT
25.4%
12.4
3.34
20.5
64.1
Llama2-7B
I IaCo*
27.1%
15.6
Ours (None)
24.0%
14.8
1.97
41.8
19.5
60.9
Ours (FFK)
25.0%
18.6
2.16
46.5
22.4
70.0
Ours (L.ayer)
77.2
Dense
0.00%
23.7
29.0
58.1
36.9
100.0
LLMPruner
24.4%
$6.9
SliceGPT
23.6%
1.9
38.3
15.1
40.9
Llama2-13B
LaCo*
24.6%
14.5
Ours (None)
24.6%
17.7
2.35
46.0
22.0
59.6
(FFN)
25.4%
28.4
77.0
Ours (Layer)
24.6%
21.8
4,70
57.3
27.9
75.6
Table 5: Comparison of different lightweight networks on classification benchmarks in terms of average accuracy and stability metrics, where "f" indicates that the intermediate size of the added lightweight network is half that of the default LLM's intermediate size.
Layer-Finst
Layer-Last
Layer-Avg
FFN
SwiGLUt
SwiGL.
Accuracy 45.1
45,2
45.6
44.4
46.0
45.8
43.8
44.2
Stability 81.2
82.7
81.9
79.2
80.7
83.7
82.6
83.3
nearly 77% of Llama2-7B and Llama2- 13B's capabilities, significantly outperforming other pruning methods. We find that almost all of the pruning methods fail on the GSM8K dataset. However, sufficient training can gradually restore the model's performance on math tasks, and the specific experimental results are shown in Table 28 of Appendix E.8.
Additionally, comparing the average stability (Average) in Table 3 with the retrained performance (RP) in Table reveals that stability is often much lower than accuracy. We also observe that accuracy on Race-M and R even increases after model pruming. Furthermore, we find that without using any lightweight network, Llama2- 13B achieves the highest accuracy on classification benchmarks, but its stability on classification benchmarks and performance on generation benchmarks are lower. These results indicate that pruned models tend to make correct guesses on some classification questions that they are uncertain, highlighting the limitations of accuracy as sole measure of pruning method performance. We also conduct experiments on OPT-1.3B, OPT-2.7B, OPT-6.78 6.7B, Baichuan- 7B, Baichuan- 13B, Baichuan2-77 7B, Baichuan2- 13B (Yang al., 2023), Llama3. 1-8B, Llama3. 1-BB (Dubey et al., 2024) and Mixtral-8x7B-v00 (Jiang et al., 2024). Details can be found in Appendix F, Appendix E.2, Appendix F.3, Appendix E.4, Appendix E.5 and Appendix F.6. In addition, we compare the performance of L.LM-Streamline with other methods at a higher pruning ratio of approximately 50%, and the results can be found Appendix E.7.
4.5 IMPACT OF DIFFERENT LIGHTWEIGHT NETWORKS
While FFN achieves the best result, Transformer layer still has performance potential. We perform experiments with Llama2-7B using various lightweight network, including Feed-Forward Neural Networks (FFN), SwiGLU-base Feed-Forward Neural Networks (SwiGLU), and Transformer layers. We also explore various initialization methods for the Transformer Layer, including random inheritance of the first pruned layer (Layer-First), inheritance of the last pruned layer LLaty,-Lass),, and averaging the pruned layers (Layer-Avg). The average accuracy and stability metrics across all the classification benchmarks are presented in Table 5, with detailed results on each benchmark available Appendix E.9. The results show that FFN achieves the best results. Meanwhile, for the Transformer Layer, inheriting the pruned first layer yields the best results. In contrast, the performance Layer-Avy Avg. inspired by LaCo, shows that averaging weights does not achieve the same effectiveness the pruned first layer. In addition, for different lightweight networks, as shown in Fig. 3. We can observe that FFN and SwiGLU have already converged by the I 10th epoch, whereas the loss of Transformer Layer is still decreasing. This
initialization
plot the validation loss curves
8

---

Published as a conference paper at ICLR 2025
Layen Layendtre LagtnLat
TFN
ValGHite Iae
10
Yalidatte Iun
Loyes-eg
60
0.55
0.5
0.90
Figure 3: Validation loss curves during training of (a) FFN and SwiGLU; (b) Transformer layer.
100
Ourr prased Liuext-78
Opea HigLhantLii Okes prened Liand-11
Stability
Accurac
85
306
25%
5.00
5.25
5.50
5.75
6.00
6.25
6.50
6.75
&Params (B)
PParams
Figure 4: (a) Stability of the pruned Llama2-7B at different pruning ratios. (b) Accuracy of the pruned Llama2-7B at different pruning ratios, compared the original Llama2-7B, OpenLlama-3B- v2, and TamaL-aama1.. 1.1.a 1. Metrics are averaged across classification benchmarks.
indicates that the Transformer layer still has potential, and further training could yield better results, this would require more computing resources.
4.6 IMPACT OF DIFFERENT PRUNING RATIOS
The performance of the pruned model is linearly correlated with the number of parameters at modest pruning ratios. To verify the model's performance at various modest pruning ratios, we evaluate our method not only the approximately 25% pruning ratio but also at ratios of around 2%, 1 11%, and 20% on Llama2-7B. The average stability and accuracy metrics across all the classification benchmarks are shown Fig. 4, with details on each benchmark presented in Appendix 10 By comparing the performance of the original Llama2-7B, TinyLlama-I.I 1.1.m OpenLlama-3BB Llama2-7B pruned at various ratios, we observe linear correlation between the performance of both the pruned models and the pre-trained original models relative to the number of parameters. This suggests that the performance of models pruned using our method is comparable to that pre-trained models with the same number of parameters.
and
4.7 COMPARISON OF LAYER REPLACEMENT AND LORA
Layer Replacement outperforms LoRA in both performance and GPU memory consumption. We compare the performance of layer replacement with LoRA. Since layer replacement is trained based on hidden states with a different training objective than LoRA, we additionally train one epoch using the language model loss for layer replacement when comparing it with LoRA. The training details can be found in the Appendix C.2. For layer replacement, we freeze the original model's weights and train only the lightweight network. In the case of LoRA, we set the rank to 128 to align the number of parameters trained with those of the lightweight networks. We randomly extract 30,000 samples from SlimPajama-6B for layer replacement training and also test with the entire dataset to evaluate the limited impact of extensive data performance (details in Appendix E.8). For LoRA, we use 300,000 samples from SlimPajama-68 Table 6 presents the average accuracy and stability across all classification benchmarks, with detailed results available in Appendix E. The findings indicate that layer replacement surpasses LoRA in both accuracy and stability, while also requiring significantly less GPU memory and training data.

---

Published as a conference paper at ICLR 2025
Table 6: Comparison of layer replacement and LoRA on classification benchmarks in terms of average accuracy metrics across all benchmarks, where "I" indicates that the intermediate size of the added lightweight network is half that of the default LLM's intermediate size.
Finst
Layer-Last
Layer-Avg
FFN
FFN
SwiGLU
SwiGLU
LoRA
Accuracy
46.7
46.8
46.2
45.8
46.3
44.4
45.5
44.5
Stability
85.7
85.6
83.9
83.4
85.2
84.7
84.7
82.1
GPU Memory (G)
27.8
27.8
27.8
25.6
27.0
25.3
26.4
56.4
Table 7: Comparison of concurrent layer pruning methods, with the metric indicating the importance of layers. Shortened Llama consists of two training stages: initial continual pre-training the SlimPajama dataset, followed by LoRA fine-tuning on the Alpaca dataset.
Method
Metrie
Need Training
Training Data
Data Size
Training Module
Trainig Method
SLEB
Perplexity
No
None
None
None
None
ShonGGP
Cosine Similarity
None
None
None
None
UIDL
Cosine Similarity
Yes
C4
164M
LRRA-Adapter
QLoAA
LaCO
Cosine Similarity
Yes
Unpublisbed
1B
Full Parameters
Fine-tuning
Shortened Llama
Taylor
Yes
SlimPujama Alpaca
627B
Full Parameters
Fine-tuning
Perplexity
50%
LoRA-Adapter
LoRA
Lighteight
Training
LLM-Streamllnee
Cosine Similarity
Yos
SlimPnjama
30k
Network
Lightweight Network
RELATED WORK
Previous pruning methods for LLMs primarily focus on pruning dense matrices (Ashkboos et al., 2024), attention heads (Michel et al., 2019; Voita et al., 2019), filters (McCarley et al., 2019; Prasanna et al., 2020), or hidden dimension (Xia et al., 2023; van der Ouderaa et al., 2023). These approaches often lead to structural irregularities, making pruned models less flexible for deployment. In contrast, layer pruning, which only alters the model's depth, is easier to deploy. Concurrent works in layer pruning alongside I. L.M--Stramlline include LaCo ( (Yang et al., 2024), ShortGP (Men et al., 2024), UIDL (Gromov et al., 2024), SLEB (Song et al., 2024), and Shortened Llama (Kim et al., 2024).
LaCo (Yang al., 2024) divides layers into groups of consecutive layers and compresses them by replacing the consecutive layers with averaged parameter weights. ShortGPT (Men et al., 2024) uses BI score, equivalent to cosine similarity, assess layer importance and remove less important layers. Similarly, UIDL (Gromov et al., 2024) uses angular distance, also equivalent to cosine similarity, to determine and remove less important layers, and employs QLoRA to enhance performance. SLEB (Song et al., 2024) calculates layer importance using perplexity and discards those deemed insignificant. Shortened Llama (Kim et al., 2024) explores various layer selection metrics and examines the effectivvness of using continual pre-training and LoRA after pruning. The differences between these layer pruning methods and LLM- LLM-Strealline are summarized in Table 7.
Unlike traditional layer pruning methods, LLM-Streamline fundamentally differs by retraining lightweight model to replace the pruned layers, rather than removing them directly with or without training the pruned model. compared to layer pruning methods (Shortened Llama, LaCo, UIDL) that necessitate retraining. Additionally, I. L.LM-Streamline better preserves the performance of the original L.LM compared to concurrent layer pruning methods.
reduces both computation time and resource consumption
6 CONCLUSION
In this paper, propose LLM-Streamline, layer pruning-and-repllcemen algorithm for LLMs. We also identify shortcomings in the existing accuracy metric and propose a new metric called stability for evaluating model compression. Extensive experiments show that this layer replacement method using a lightweight network outperforms previous state-of-the-an pruning methods and demonstrates superior effectiveness and efficiency compared to concurrent layer pruning methods.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported by the National Key Research & Develop Plan (2023YFF0725100) and the National Natural Science Foundation of China (62322214, U23A20299, U24B20144, 62172424, 62276270).
REFERENCES
Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint ar arXiv:2401.15022 2024.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Tianyi Chen, Tianyu Ding, Badal Yadav, Ilya Zharkov, and Luming Liang. Lorashear: Efficient large language model structured pruning and knowledge recovery. arXiv preprint 1rX1v.2310.18356, 2023.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp- 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation leamiing In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 15750-15758, 2021.
Christopher Clark, Kenton Lee, Ming-Wor Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 1rXiv:1905.1004 2019.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14100 14168, 2021.
OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. GitHuu repository, 2023.
Rocktim Jyoti Das, Liqun Ma, and Zhiqiang Shen. Beyond size: How gradients shape pruning decisions in large language models. arXiv preprint arXiv:231 2023.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 (): 8-bit matrix multiplication for transformers scale. Advances in Neural Information Processing Systems, 35: 30318-30333 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized 1lms, 2023.
Abhimanyu Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 1783, 2024.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 10323-10337. PMLR, 2023.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use laptop? a question answering benchmark with implicit reasoming strategies. Transactions of the Association for Computational Linguistics, 9:346-361, 2021.
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. In Low-Pows Power Computer Vision, pp. 291-326 Chapman and Hall/CRC, 2022.

---

Published as a conference paper at ICLR 2025
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, (296)):189-1811 2021.
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403. 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.00 03300, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1303.0253:3 2015.
Namgyu Ho. Laura Schmid, and Se- Young Yun. Large language models reasoning teachers. arXiv preprint 11xXi::2212.10777 2022.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
Yuxuan Hu, Jing Zhang, Zhe Zhao, Chen Zhao, Xiaodong Chen, Cuiping Li, and Hong Chen. sp Enhancing structured pruning via pea projection, 2024. URL https: /arpiii.////i..oo/////e/ org abs/ 2308. 16455.
Yukun Huang, Yanda Chen, Zhou Yu, and Kathleen McKeown. In-context learning distillation: Trans- ferring few-shot learning ability of pre-trained language models, arXiv preprint arXiv:2212.10670 2022.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint TiXiv:2401.04000 2024.
Bo-Kyeong Kim, Geonmin Kim, Ta Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung-Kye Song. Shortened llama: simple depth pruning for large language models. arXiv preprint arXiv:2402.0288. 2024.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.0466 04683, 2017.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference the principles of knowledge representation and reasoning, 2012.
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint 2rX1v:2306.09212, 2023.
Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang. Zekun Li, Hong Wang, Jing Qian, Baolin Peng. Yi Mao, et al. Explanations from large language models make small reasoners better. 2022.
arXiv preprint
Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Evaluating quantized large language models, 2024.
Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quantization for vision transformer. Advances Neural Information Processing Systems. 34:28092-28103, 2021.
Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Contextual sparsity for efficient llms at inference time. In International Conference Machine Learning, pp. 22137-22176, PMLR. 2023.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1_ regularization. arXiv preprint arXiv: 1712.01312, 2017.
12

---

Published as a conference paper at ICLR 2025
Xinyin Ma, Gongfan Fang, and Xinchao Wang.
On the structural pruning of large language models. Advances in neural information processing systems, 36:21702-21722 2023.
JS McCarley, Rishav Chakravarti, and Avirup Sil. Structured pruning bert-based question answering model. arXiv preprint 1xX0::910.066. 2019.
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojic Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint Stt::24003008553, 2024.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019.
Shashi Narayan, Shay Cohen, and Mirella Lapata. Don't give me the details, just the sum- mary! topie-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745 08745, 2018.
Sai Prasanna, Anna Rogers, and Anna Rumshisky. When bert plays the lottery, all tickets are winning. arXiv preprint (xXix:2005.00561 2020.
N Reimers. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908. 10084, 2019.
Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, and Jae-Joon Kim. Sleb: Streamlining Ilms through redundancy verification and elimination of transformer blocks, 2024.
URL https:://// / arxiiv. org/ abs/2402. 09025.
Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. Investigating prior knowledge for challenging chinese machine reading comprehension. Transactions of the Association for Computational Linguistics, 8:141-155, 2020.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. /695, 2023.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.009% 2018.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Tycho FA van der Ouderaa, Markus Nagel, Mart Van Baalen, Yuki M Asano, and Tijmen Blankevoort. The Ilm surgeon. arXiv preprint arXiv:2312. 17244, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned, 2019.
Wenxiao Wang, Wei Chen, Yicong Luo, Yongliu Long, Zhengkai Lin, Liye Zhang, Binbin Lin, Deng Cai, and Xiaofei He. Model compression and efficient inference for large language models: A survey, 2024.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06644 2023.
Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao. Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al. Clue: A chinese language understanding evaluation benchmark. arXiv preprint arXiv:2004. 059966 2020.
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin. Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309. 10305, 2023.
13

---

Published as a conference paper at ICLR 2025
Yifei Yang, Zouying Cao, and Hai Zhao. Laco: Large language model pruning via layer collapse. arXiv preprint arXiv:2402111877, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv: 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0/ 01068, 2022.
Chujic Zheng, Minlie Huang, and Aixin Sun. Chid: A large-scale chinese idiom dataset for cloze test. arXiv preprint arXiv: arXiv: /906.0/266.00... 2019.
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. survey model compression for large language models, 2023.
14

---

Published as a conference paper at ICLR 2025
A COMPARISON OF COSINE SIMILARITY AND PERPLEXITY
To demonstrate the sensitivity of perplexity, referencing the SLEB (Song et al., 2024), we prune the Llama2-7B model with different pre-training datasets, including SlimPajama, C4 and wikitext. The experimental results are presented in Table 8. When pruning with cosine similarity, the layers pruned are consistent across different datasets, whereas when pruning with perplexity, the layers vary, indicating the sensitivity of perplexity. In addition, we evaluate the model after pruning with the SlimPajama dataset, and the experimental results are shown Table This indicates that the model pruned with perplexity shows lower perplexity on the dataset used for pruning, but performs worse on downstream tasks.
Table 8: Pruned layers using perplexity and cosine similarity for pruning.
Pruned Layers
LLM
Datasct
Perplexity
Cosine Similarity
SlimPujama
9.10:11.1..1:2:222277
22.25.24.25.66.2..22.2.
Llama2-7B
wikitext
9.10.1.12.212322..7
22.23..4.25.26.22..222
8:9.11.12.222332442255
22.25.24.2522622222...
Table 9: Detailed results of accuracy of using perplexity and cosine similarity for pruning. "Perplex- ity*" refers to the Perplexity of the pruned model SlimPajama. Using perplexity the metric can be considered as SLEB, while using cosine similarity as the metric can be considered as variant of our approach, i.e., Ours (deta)(details in Section 4.1).
LLM
Mark
Paplocty
Becckals
Avarue
Peplicat
37.6
342
413
1.24
35.8
B DATA DISTRIBUTION
We extract the training data from different domains based on the data distribution strategy proposed in Sheared-LLaMa (Xia et al., 2023). The detailed data distribution is shown in Table 10.
Table 10: The proportion of different domains randomly selected from the SlimPajama-66 dataset.
CC
GitHub
Book
StackExchange
Wiki
ArXiv
C4
SlimPajama-6B
54.1%
4.2%
3.7%
2.8%
3.1%
3.4%
28.7%
Ours
36.1%
0.8%
9.1%
1.0%
3.1%
0.7%
49.2%
C TRAINING IMPLEMENTATION DETAILS
C.I LIGHTWEIGHT NETWORK TRAINING DETAILS
For both the FFN structure and the SwiGLU structure, the learning rate is set to le-3 and the weight decay is le-4, For the Transformer layer, the learning rate is le-5 and the weight decay le-3. The model is trained using a batch size of 32 over 20 epochs. On a single A800 GPU, the training duration for the lightweight network is approximate hours (for the Transformer layer).
C.2 POST TRAINING DETAILS
For layer replacement, in order to have fairer comparison with LoRA, conduct epoch of post-training with learning rate of 5e-5, a weight decay of 1e-3, and batch size 32. This process takes less than hour single A800 GPU. For LoRA, the model is trained one epochs with
15

---

Published as a conference paper at ICLR 2025
learning rate of le-4, weight decay of 1e-3, and batch size of 32. Since the amount of training data used is ten times that of layer replacement, it take approximately 10 hours to complete the training on single A800 GPU.
D INFERENCE SPEED COMPARISON
As shown the Table 11, we evaluate the inference speed of various pruning methods at similar pruning ratio when generating sequences of length 128. The results indicate that the acceleration effect of L.M-Streamline is slightly inferior to that of SliceGPT and L.MM-Pruner.
Table 11: The inference speed of models pruned using different methods.
Llama2 7B
Dense
LLM Pruner
SliceGPT
Ours(None))
Ours(FFN)
Ours(Layyr)
Pruning Ratio (%)
0.00
24.8 25.91
25.4
24.0
25.0
24.0
Inference Speed (tokens/s)
19.87
27.20
25.68
25.88
25,68
E DETAILED EXPERIMENTAL RESULTS
E. EXPERIMENTAL RESULTS OF OPT-6.7B
We conduct experiments on OPT-6.7B 7B. The experimental results are shown in Table 12, Table 13 and Table The results indicate that our proposed L.LM-Streamline is superior to the previous SOTA method.
Table 12: Accuracy of different pruning methods classification benchmarks by pruning OPT-6.78 7B.
Matied
BoolQ
CoQA
Benchmarts H&SW
PIQA
Race-M
Race-3
Averg:
Dome
0.00%:
S4.7
12.9
21.5
64.6
41.4
$4.8
53.3
15.4
25.1
25.4
24.7
23.5
41.1
100
SliceGPT
25.8
(Monc)
20.6
22.1
22.4
Oum (FFN)
25.0%
57.6
12.1
18.7
63.7
37.5
41.8
55.9
33.2
22.7
22.3
26.4
24.0
37.9
93.2
25.2
92.9
Table 13: Stability of different pruning methods on classification benchmarks by pruning OPT-6.7B.
LLM
Methed
Ratio
Benchmarks IeeSW
Avenge
CMNLI
CIII
BoolQ
WSC
CoQA
PIQA
Race M
Race II
MMLU
CMMLU
SliceGPT
25.6%
66.4
39.8
73.1
30.1
79.4
75.2
73.7
82.5
72.5
69.1
68.9
67.2
56.5
OPT6.78
Durs (Nenc)
34.0%
56.2
62.5
71.4
41.5
87.5
48.6
50.5
57.1
63.4
61.2
62.3
62.7
60.5
(FRN)
82.4
74.7
Oues (Layes)
24.0%
72.1
35.5
76.1
72.1
91.4
83.6
37.0
91.6
72.9
71.2
75.1
70.1
75.1
Table 14: Accuracy of different pruning methods on generation benchmarks by pruning OPT-6.7B.
Benchmarks GSM8K
LLM
Method
Ratio
Average
RP
Xsum
StrategyQA
Dense
0.00%
13.4
2.2
54.3
23.3
100.0
SliceGPT
19.4
83.3
OPT.6.B
Ours (None)
24.0%
4.9
1.6
6.87
Ours (FFN)
25.0%
14.8
0.8
43.6
19.7
84.5
Ours (Layer)
24.0%
18.4
44.4
21.8
93.6
E.2 EXPERIMENTAL RESULTS OF BAICHUAN-7B AND BAICHUAN-13BB
We conduct experiments Baichuan-7B and Baichuan-133 The experimental results are shown in Table 15, Table 16 and Table 17. The results indicate that our proposed LLM--Stramlline is superior the previous SOTA method.
16

---

Published as conference paper at ICLR 2025
Table 15: Accuracy of different pruning methods on classification benchmarks by pruning Baichuan- 7B and Baichuan- 13B.
11M
Methed
Ratin
Banchmeks HeSW
Avangg
RP
UMNNII
CHIB
BoolQ
CoKA
PSQA
Race-Mi
MMLUU
UMMLUU
29.5
11MPnmer
65.9
20.5
12.6
24.3
25.3
Becchon-78
Ous (Nooe)
24.25
33.2
32.7
25.8
69.5
36.5
36,0
34,5
55.7
22.1
21.5
25.7
18.8
35.5
67.4
23.2
Ous
35.0
43.1
22.9
Dase
0.005
61.5
364
91.5
63.5
49.0
64.2
69.1
78.2
48.1
46.0
54.5
55.3
60.0
100
Iuichan--1R
Oun. (Niee))
24.75%
68.8
34.8
50.2
62.2
40.4
65.4
$6.7
68.2
36.6
23.3
52.9
55.1
47.6
79.7
25.55
26.5
Okex (layer)
67.0
27.8
Table 16: Stability different pruning methods on classification benchmarks by pruning Baichuan- and Baichuan-131 -3BB.
LLM
Method
Ratio
Benchmaeks
Boool)
LLMPPuner
26.2%
70.2
40.0
71.4
24.9
91.4
75.3
75.9
62.4
65.8
66.8
56.7
53.6
64.6
Buichun-7B
26.2%
55.5
60.8
25.1%
$4.1
77.8
75.6
65.4
98.3
24.6
Ouzi (Layer)
24.2%
$6.3
79.3
$2.3
40.7
93.3
77.6
81.3
83.5
75.8
71.0
75.9
75.6
76.9
(None)
Baichuan 13B
Ours (FTN)
25.55
35.7
57.3
82.1
81.3
43.3
81.2
31.7
79,8
46.6
61.1
$4.3
83.7
243
Ours
26.2%
88.
92.7
$9.3
72.9
$3.0
83.5
89.2
90.4
766
Table 17: Accuracy of different pruning methods generation benchmarks by pruning Baichuan-7B and Baichuan-IB
Benchmarks GSMSK
LLM
Method
Rasio
Average
RP
Xsum
StalegyQA
Dease
0,00%
19.1
9.84
55.5
28.1
100
LLMPhuner
Raichuan-7B
Ours (None)
24.2%
0.3
0
0
0.1
0
Ours (HFN)
25.1%
19.3
2.11
41.1
20,8
74.0
Ouns (Layer)
24.2%
18.2
1.36
38.7
19.4
69.0
0.00%
37.6
100
Ours (None)
24.7%
2.1
1.2
12.3
5.2
13.8
Ours (PFN)
25.5%
23.1
2.1
47.3
24.2
64.4
Ouns (Layer)
24.7%
22.2
2.4
43.2
22.6
60.1
E.3 EXPERIMENTAL RESULTS OF BAICHUAN2-7B AND BAICHUAN2-133
We conduct experiments on Baichuan2-7B and Baichuan2-13B. The experimental results are shown in Table 18, Table 19 and Table 20. The results indicate that our proposed L.M-Streamllne is superior to the concurrent SOTA method, LaCo.
Table 18: Accuracy of different pruning methods on classification benchmarks by pruning Baichuan2- 7B and Baichuan2-13B. indicates that we refer to the results in the original paper.
11M
Mahsod
Rañio
CMNLI
Benchmark He5W
MMLU
CMMLU
Axcragg
RP
53.1
57.6
11MPrner
26.2%
39.9
35.0
70.6
50.0
41.3
38.7
52.7
7014
22.3
22.8
26.9
24.9
41.1
60.4
Reidwar2-78
LaCu*
24.2%
50.9
35.0
76.2
36.2
42.3
47.3
52.3
68.5
27.7
29.0
31.5
31.2
45.5
76.9
Ouz (PFN)
47.6
29.1
30.5
Oun (layer)
26.2%
60.4
34.9
722
62.7
36.5
48.5
52.5
67.0
35.
36.5
54,0
56.3
31.5
$7.0
Denue
0.00%
65.6
35.2
$6.7
66.8
42.3
65.6
71.1
78.1
68.9
67.2
99.6
61.3
63.9
100
Raichan2-13B
Oan- (Nam))
$1.9
60.4
Dirs (FTN)
25.5%
A5.0
13.01
81.7
60.1
36.5
56.7
52.1
70.5
71.1
68.2
52.1
55.2
19.7
21.8
(laper)
36.2%
433
15.0
841
18.5
$6.9
K30
12,1
10.2
66.1
99.1
40.7
900
17

---

Published as a conference paper at ICLR 2025
Table 19: Stability of different pruning methods on classification benchmarks by pruning Baichuan2- 7B and Baichuan2-13B.
1LM
Method
Raso
CMNLI
Dencimarks H::W
Race-M
Ruce-H
CMMLU
Average
LLMirrunr
24.2%
62.2
50.6
15.1
55.5
63.3
05.0
88.3
54.2
51.7
50.0
46.9
Inichoan2-71
Ouns (Nome))
24.2%
65.5
98.1
63.8
69.1
$4.6
61.3
67.3
72.0
50.6
48.0
69.3
74.5
68.9
61.2
2222
Ouss (Layer)
24.2%
83.3
92.8
74.9
67.6
$4.6
65.0
77.2
80.8
64.1
62.9
89.2
53.6
77.5
Oun (None)
34.7%
85.0
85.7
96.5
85.1
50.7
79.5
81.2
85.1
84.6
53.0
74.2
65.6
83.5
37.2
77.5
83.2
Oues (Layer
34.7%
87.0
99.0
$9.0
87.1
$4.7
69.2
84.9
86.9
89.4
87.0
91.7
92.5
88.4
Table 20: Accuracy of different pruning methods generation benchmarks by pruning Baichuan2-78 and Baichuan2- 13B. "*" indicates that we refer to the results in the original paper.
Benchmarks
LLM
Method
Ratio
Avemae
RP
Xsum
GSMBK
StrategyOA
L.MMPrner
24.2%
14.5
8.9
25.2
LaCo*
24.2%
12.0
Baichman2-7B
Ours (Nonc)
24.2%
1.7
30.7
14.8
11.9
52.7
Ours (Layer)
24.2%
16.8
2.3
34.8
18.0
51.0
Dense
0.00%
25.3
53.2
65.9
48.1
100
LaCo*
24.7%
12.3
Ruichmam2-138
(None)
3.3
Ours (FFN)
25.5%
21.3
3.1
48.8
24.4
50.7
Ours (Layer)
24.7%
20.9
5.5
51.3
25.9
538
E.4 EXPERIMENTAL RESULTS OF LLAMA3.1-8B AND ALAMA3.1-70B
We conduct experiments on Llama3. 1-8B and Llama3.1-700 The experimental results are shown in Table 21 and Table 22. The results indicate that our proposed LLMM-Streamln is superior to the previous SOTA method.
Table 21: Accuracy different pruning methods on classification benchmarks by pruning Llama3. 8B and 1-0ma3.1-70B.
Benchmurks
LLM
Methed
BoolQ
PIQA
Desse
0.005
65.3
33.0
73.8
68.2
36.5
69.8
74.7
$1.1
71.6
66.5
52.5
63.2
100
Lluma3.1 SB
ShockaP
23.99
39.4
32.11
21.3
37.8
36.5
35.0
39.9
55.7
21.9
21.3
25.5
25.2
33.4
52.8
52.3
58.9
Ours (Layer)
Danse
0.00%
74.8
33.0
$1.6
76.5
37.5
73.0
70.9
$3.9
$6.8
80.3
70.3
68.8
71.3
100
1-302.1-708 1-708
SlaeelPP
29.19
40.4
31.9
18.9
37.8
37.5
41.0
45.3
61.0
24.1
24.8
37.5
30.5
35.9
50.4
Our (layor)
79.3
Table 22: Stability of different pruning methods on classification benchmarks by pruning Llama3. 1-8B and 1lama3.1-7088
Benchroaks HeSW
CMNLI
CHID
Booll)
WSC
CoQA
PIQA
Race M
Race
MMLUU
CMMLU
Avcrage
SticelPP
23.9%
63.2
57.0
38.5
35.6
58.1
39.0
47.4
53.2
(Nons)
59.9
47.0
$7.0
$3.4
61.4
$3.7
Ouss (Layen)
24.4%
78.5
49.7
59.9
75.0
100
S0.3
$4.8
86.5
87.3
86.1
90.8
89.1
30.7
SässsPT
Unmi 1-208
Ours (None)
30.3%
77.6
43.5
64.5
75.0
91.4
76.8
$4.5
92.4
89.3
81.4
78.6
Durs (1 aperi
10.9%
86.7
98.7
76.1
71.7
95.2
80.5
92.6
93.4
97.2
96.3
95.9
94.6
90.9
18

---

Published as a conference paper at ICLR 2025
E.5 EXPERIMENTAL RESULTS OF MIXTRAL-8X7B-VO1.
We conduct experiments Mixture of Experts(MoFF) model Mixtral-8x7B--v0.1. The experimental results are shown in Table 23 and Table 24.
Table 23: Accuracy of LLM-Streamline on classification benchmarks by pruning Mixtral-8x7B-vO..
LIM
Metbod
Ratio
CMNI
(3I1)
BuolQ
WSC
CHQA
Berchrrnts HeSW
PIQA
Rac-M
Raced1
MMIL
CMMIE
Arerae
RP
Dorsa
0.005
54.1
13.0
65.6
56.4
66-3
76.
10.
71.3
52.8
63.
100.0
Mixtul 8xTB s0.1
(Layes)
Table 24: Stability of LLM-Streamline on classification benchmarks by pruning Mixtral-88 Mixtral-&x7B-vei -817-B00.
LLM
Methed
Rutio
43
CMNNI
UHID
Bool)
WSC
CoO24
Bendhmarks
He5W
PRQA
Race M
Race
MMLUU
UMMLU
Average
Mistro--677---0.1
(Nosz)
24.9%
989
75.4
69.3
72.4
79.1
68.9
(1 (lager)
E.6 EXPERIMENTAL RESULTS OF OPT- 1.3B AND OPT-2.7B
also conduct experiments on small models (OPT-1.3B and OPT-2.7B). The experimental results are shown in Table 25, The results indicate that our proposed LLM-Streamline is superior to the previous SOTA method, across different pruning rates.
Table 25: Accuracy of different pruning methods by pruning OPT-1.3B and OPT-2.7B.
Beuchmarks ARC-easy
L.M
Method
Ratio
Average
RP
PIQA
HellaSwag
ARC-challenge
OpenBookQA
Dense
0.00%
72.4
59.3
29.5
23.4
48.2
100.0
SliceGPT
23.1
20.2
Oone)
19.4%
57.2
51.7
29.1
32.5
22.7
13.2
34,4
71.4
Ouss(FEN)
18.1%
68.8
58.4
39.1
54.3
23.3
23,3
44.5
92.3
Dense
0.00%
72.4
59.3
53.7
$1.0
29.5
23.4
48.2
100.0
SliceGPT
25.8%
65.5
52.8
34.2
48.8
24.4
17.0
40.5
$4.0
OPT-L.3B
Ours(None)
52.2
Ours(FFN)
Dense
0.00%
72.4
59.3
53.7
51.0
29.5
25.4
48.2
100.0
SliceGPT
33.6%
62.4
S2.6
12.2
45.4
23.1
16.6
38.7
80.3
Ours(Nonc)
34.8%
50.5
51.5
25.8
26.2
20.3
14.6
31.5
65.4
Ours(FFN)
33.6%
62.9
52.1
33.9
48.3
20.8
20.6
39.8
82.6
Dense
0.00%
73.8
61.0
45.9
60.9
26.8
25.0
48.9
100.0
SliceGPT
92.0
Ouns)None)
Ours(TNN)
16.8%
70.7
60.4
57.8
25.3
24.4
46.9
95.9
Dense
0.00%%
73.8
61.0
45.9
60.9
26.8
25.0
48.9
100.0
SliceGPT
25.7%
69.1
55.0
37.9
$3.9
26.7
18.2
43.5
$9.0
OPT-117B
Ouns(None)
26.7%
59.7
53.4
33.5
38.1
24.3
15.4
37.4
76.5
Ours(FFN)
SliceGPT
34.6%
64.8
$4.1
35.6
50.0
26.5
18.0
41.5
$4.9
OunseNone)
35.6%
56.6
52.9
31.5
37.6
24.1
14.9
36.3
74,2
Ours(EFN)
34.6%
65.3
55.3
36.3
51.4
24.5
21.0
42,3
86.5
E.7 EXPERIMENTAL RESULTS OF LLAMA2-7B AT AROUND 50% PRUNING RATIO
We conduct experiments Llama2-7B higher pruning ratio of approximately 50%. The experimental results are shown in Table 26 and Table 27. The results indicate that our proposed L.MM-Streamline is superior the previous SOTA method.
19

---

Published as a conference paper at ICLR 2025
Table 26: Accuracy of different pruning methods on classification benchmarks by pruning Llama2-7B at higher pruning ratio of approximately 50%.
Bendhmarkc HeS
LLM
Method
Rasio
CMNUI
Roce-H
MMLLU
(MMLLU
Aneriie
RP
LLMPrune
49.2%
26.4
33.2
17.3
43.5
35.5
25.4
32.7
52.2
22.3
21.7
23.4
24.8
30.7
63.4
1lama2 7B
ShacCPP
48.3%
26.5
32.1
15.4
38.1
42.3
28.1
30.9
53.6
23.6
23.1
25.2
25.3
30.4
61.8
Our (Layes)
48.0%
39.7
33.0
27.7
62.1
36.5
46.3
45.
63.6
25.6
25.3
13.2
28.1
38.5
78.3
Table 27: Stability of different pruning methods classification benchmarks by pruning Llama2-7B at a higher pruning ratio of approximately 50%.
LLM
Method
Ratio
Benchmarks
Averae
LLMPruser
49.2%
63.6
96.2
00.8
33.8
97.5
51.5
55.5
70.9
47.8
43.1
56,9
35.1
50.9
Liams2-78
ShicePT
48.3%
34.9
54.7
19.0
83.7
52.7
63.4
71.2
61.5
52.2
S4.8
48.0%
61.2
65.2
48.2
61.4
58.8
Our (Layer)
48.0%
88.8
100
67.7
79.5
95.2
65.3
66.1
76,8
38.7
57.3
63.8
59.4
72.5
E.8 RESULTS OF SUFFICIENT POST-TRAINING
Following the method outlined in Section 4.7, We conduct experiments using the entire SlimPajama- 6B for post-training, and the results are presented in Table 28. As shown, using the entire dataset resulted in slight improvement, but significant computational cost, requiring 100 times the computational time.
Table 28: Detailed accuracy results with different training data volumes.
LLM
Mateel
Tuiday data
CHD
CI0A
Rax-M
CMNNU
GEMB:
SeragOh
Avarue
Desi
65.8
11.5
165
457
Lacestna
23k
2,5
E.9 DETAILED RESULTS OF DIFFERENT LIGHTWEIGHT NETWORKS
The detailed results of accuracy and stability of different lightweight networks on different classifica- tion benchmarks are shown Table 29 and Table 30. We can observe that FFN achieves the best results. Meanwhile, for the Transformer Layer, inheritance of the pruned first layer yields the best results.
Table 29: Detailed accuracy results of different lightweight networks on different classification benchmarks, where "f" indicates that the intermediate size of the added lightweight network half that the default LLM'' intermediate size.
1LM
Mathol
Ratio
C3
CMNLI
CHID
BoxtQ
WSC
CaQA
Benchmack BLSW
PSQA
Race-M
Racz-Et
MM.LU
CMMELL
Average
RP
100.0
Layes
24.0%
42.5
LayesFint
34.0%
43.3
33.0
34.1
67.5
16.5
92.3
61:1
21.5
34.8
37.0
45.5
20.4
45.3
91.8
Layer-Laa
24.0%
43.5
33.0
2801
64.5
41.4
56.8
61.5
71.
34.5
35.0
46.0
30.8
45.6
92.3
Llanx2-7B
Laye--Ag
FIN"
35.0%
40.6
67.5
53.4
41.4
46.0
93.5
FFN
25.09
9037
35.0
223
65.9
22.5
68.
61.1
71.3
28.0
18.7
47,0
31.7
55.8
911
SuGGLUT
26.03
41.9
68.5
55.8
76.9
33.3
41.
W00
20

---

Published as a conference paper at ICLR 2025
Table 30: Detailed stability results of different lightweight networks on different classification benchmarks, where "f" indicates that the intermediate size of the added lightweight network is half that of the default LLM's intermediate size.
Mathod
CMNI
CHID
BooiQ
WSSC.
CoQA
Benelmnkks HeSW
PIQA
Race-M
Race-Hf
MMLJ
CMMILU
Average
Laym-Rendom
77.4
Layer Fiast
24.0%
79.3
64.4
36.3
95.2
81.7
35.3
85.6
31.8
79.0
82.4
71.0
82.7
Layer-Lase
34.0%
$1.0
13.5
36.0
82.7
81.
35.8
85.4
82.3
26.3
66.9
81.9
L-m22788
Layer
PPN)
26.0%
79.9
100
55.4
$2.1
98.2
78.7
80.7
81.7
76.7
70.3
$4.9
74.5
80.7
ETN
82.5
SwiGLUT
26.0%
78.3
64.5
78.9
75.4
80.9
82.1
90.7
78.4
82.6
SwiGLU
25.0%
78.9
100
63.5
$4.4
95.2
77.0
82.2
82.3
85.7
12.4
$7.9
79.7
83.3
E.10 DETAILED RESULTS OF DIFFERENT PRUNING RATIO
The detailed results accuracy and stability on LLMs under different pruning ratios on different classification benchmarks are shown in Table 31 and Table 32. The experiment results show that the performance the pruned model linearly correlated with the number of parameters, demonstrating the effectiveness of our method.
Table 31: Detailed accuracy results of different pruning ratios on different classification benchmarks, where "t" indicates that the intermediate size of the added lightweight network is half that of the default LLM's intermediate size.
Benchmanks HeS
CMNEI
CHID
Bool()
WSC:
CoQAA
PIOA
Race-M
Race-It
MMLU
CMMLU
Awarage
Dence
0/00%
43.31
33.11
41.5
70.8
37.5
66.7
71.3
7K.1
33.1
35.5
66.3
31.8
45.2
10010
FFN)
2,05
43.6
33.01
41.0
71.5
56.1
65.4
7007
77.5
16.1
58.1
46.1
31.6
49.3
10KI
Llmm2--88
62.6
$7.2
10.0
HNN
20.%
423
2910
70.2
02.7
63.8
72.3
37.8
37.7
45.7
46.7
FFNT
26.0%
40.6
33.0
24.2
67.5
36.5
55.4
59.5
71.4
41.9
41.4
66.1
20.8.
46.0
93.5
Toyl Jame-1.11 1B
100.0
OperLiuna
55.7
Table 32: Detailed stability results of different pruning ratios on different classification benchmarks, where "t" indicates that the intermediate size of the added lightweight network is half that of the default LLM's intermediate size.
Benchmarks HeSW
Mathod
Ratio
CMNLI
CHID
BoolQ
WSC
CoOA
PIQA
Roce M
Race
MMLU
CMMLU
Avirage
FFNt
2.0%
95.8
100
86.3
95.2
97.8
97.1
93.0
89.5
$9.7
Llama2: 70
F9N1
11.0%
90.4
100
K2.5
91.4
94.2
90.2
93.4
92.1
59.6
N2.7
84.6
72.00
88.6
PFN1
20.0%
91.6
86.0
95.2
$4.4
87.3
87.9
83.2
78.6
83.7
19N1
26.0%
79.9
100
65.4
82.1
95.2
78.7
80.7
81.7
74.7
70.3
84.9
74.6
80.7
E. DETAILED RESULTS OF LAYER REPLACEMENT AND LORA
The detailed results of accuracy and stability of different layer replacement strategies and LoRA different benchmarks are shown in Table 33 and Table 34. The results show that layer replacement outperforms LoRA in both accuracy and stability.
21

---

Published as a conference paper at ICLR 2025
Table 33: Detailed accuracy results of layer replacement and LoRA on different classification benchmarks, where "f" indicates that the intermediate size of the added lightweight network is half that of the default LLM's intermediate size.
CMENLI
CHID
BoolQ
WSC
CoOA
HcSW
PIQA
Ruc M
Race-Ei
MMLU
CMMLU
Average
Dense
0.0016
41.8
41.6
17.5
75.1
15.1
35.3
46.3
31.3
64.2
10310
Layer
21.3
Layer-Last
24.0%
44.9
33.0
29.4
60.2
36.5
58.9
63.5
74.1
37.8
37.8
46.5
30.4
46.8
95.1
Layes-Arg
24.0%s
43.9
3308
30.1
67.5
36.5
58.3
62.5
72.3
36.6
36.1
46.2
31.9
46.2
93.9
1lmss2-7B
FFN
HFN
25.16
81.8
3230
36.5
03.5
72.
374
35.4
45
$1.5
66.3
SwIGEL!
26.0%
44.0
3300
27.9
61.2
36.5
57.2
61.7
71.2
30.3
32.9
45.0
31.9
44.4
90.2
SwiOLU
43.2
LakA
26.0%
43.2
32.6
71.3
12.
10.8
Table 34: Detailed stability results of layer replacement and LoRA on different classification bench- marks, where "1" indicates that the intermediate size of the added lightweight network half that of the default LLM's intermediate size.
Benchmarks HeSS
Methed
CMNLI
CITI
BoelQ
WSC
CrQA
PIQA
Race-M
Race-If
MMLU
CMMLU
Averge
Layer-Fint
24.0%
82.7
74.3
84.7
95.3
85.7
89.1
58.6
54.1
$2.3
64.5
71.2
83.7
LayasLas
24.0%
73.8
Layer-Ang
24.0%
81.0
100
72.3
67.0
95.2
$4.1
$7.4
56.6
$7.3
52.2
90.5
73.7
$3.9
Clamod-78
FFN:
25.0%
FTN
25.05
80.6
100
72.0
83.5
95.2
$5.4
$2.2
87.2
$4.5
$1.0
83.4
79.3
85.2
SwiCi
SwiGiLU
25.0%
81.6
72.9
67.3
95.2
$1.8
36.5
55.3
53.7
54.8
90.0
13.3
$4.7
LoRA
$1.6
F
LIMITATION
Our method achieves SOTA results compared to existing pruning methods, but its performance still falls short of other commonly used model compression methods, e.g., quantization. Therefore, plan to enhance the performance of pruning method and explore combining it with other compression and inference acceleration techniques to make more practical.
22


Reviews:
[{"review_id": "vTRpXaUW9j", "reviewer": "Reviewer_YVX4", "summary": "This submission presented LLM-Streamline, a new model pruning method for LLMs. Besides traditional pruning, this method also proposed layer replacement, a novel module that trains a lightweight network to replace the pruned layers to mitigate performance loss. In addition, the authors also proposed stability as a new metric. Experimental results show the superiority of this method.", "questions": "I'm not an expert of LLM pruning/compression field. I'm curious why you didn't compare with some knowledge distillation method for LLM, for example \nhttps://proceedings.mlr.press/v235/ko24c.html", "limitations": "", "rating": 8, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Basically, this submission has two major innovations:\n1. layer replacement\n2. new metric named stability.\n\nThe first one is a very good contribution that mitigates the loss of pruning only.", "weaknesses": "I didn't find any weakness of this submission.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fJoSIy7693", "reviewer": "Reviewer_wDBW", "summary": "In this work, the authors propose LLM-Streamline, a new approach for compressing large language models (LLMs) by pruning layers based on cosine similarity and replacing the pruned layers with a lightweight network to preserve or even enhance model performance. This two-step method is shown to effectively reduce model size with minimal loss in accuracy, as demonstrated by experiments across various classification and generation benchmarks. Additionally, the work introduces a stability metric to address limitations of traditional accuracy metrics in evaluating pruned models. Experimental results highlight the superiority of LLM-Streamline over other state-of-the-art pruning methods in terms of accuracy retention and computational efficiency, especially under hardware constraints. Overall, this work offers a novel technique with practical implications for large-scale LLM deployments.", "questions": "1. Can you provide more detail on why cosine similarity was chosen as the primary measure for layer redundancy? How does it compare to using perplexity or other metrics?\n2. For the stability metric, is there a specific scenario or task where it offers clear advantages over traditional accuracy measures? An illustrative example would help clarify this.\n3. Have you considered a detailed comparison with other model compression methods, like LoRA? This would help highlight what sets LLM-Streamline apart.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Originality: This paper combines layer pruning with lightweight network replacement in a novel approach for compressing LLMs. This method effectively maintains model performance even after significant pruning.\n2. Significance: The proposed stability metric enhances LLM compression evaluation by addressing limitations in standard accuracy metrics, providing a potentially more reliable measure of retained model performance.\n3. Technical Quality: The experiments are comprehensive, covering various benchmarks that showcase the model’s effectiveness in classification and content generation. Each step of the pruning and replacement process is clearly explained, highlighting the technical soundness of the approach.\n4. Practical Implications: LLM-Streamline improves model efficiency while retaining performance, making it valuable for real-world applications where computational resources are limited.", "weaknesses": "1. Metric Justification: While cosine similarity is chosen as the primary metric for layer redundancy, additional justification for this choice over other metrics (e.g., perplexity, Euclidean distance) would be beneficial.\n2. Comparison with Other Methods: While the authors mention alternative approaches, such as LoRA, they do not provide a detailed comparison. A more thorough discussion of how LLM-Streamline performs relative to other popular fine-tuning and compression techniques would give readers a clearer perspective.\n3. Explanation of Stability Metric: The paper could simplify its explanation of the stability metric, making it more accessible to readers new to model compression. Clarifying why stability offers practical advantages over accuracy would also be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ac8ox2N0q5", "reviewer": "Reviewer_M4rG", "summary": "The paper presents LLM-Streamline, a novel layer-wise pruning and replacement framework aimed at compressing large language models (LLMs) by identifying and removing less significant layers. The process is divided into two stages: (1) Layer Pruning to identify and prune layers with minimal impact based on cosine similarity, and (2) Layer Replacement, which trains a lightweight network to compensate for the removed layers, mitigating performance loss. Additionally, a new stability metric is introduced to more accurately reflect performance post-pruning, which is compared against existing methods across multiple benchmarks.", "questions": "Please refer to weaknesses section for questions.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* This paper uses a lightweight network and training based on hidden states before and after compression to compensate for the loss caused by pruning, reducing the need for computing resources and leading to better precision recovery.\n\n* The continuous layer pruning used in this paper reduces the complexity of the compressed model and is easier to accelerate on hardware than unstructured pruning and other methods.\n\n* The paper provides an in-depth analysis of the limitations within traditional accuracy metrics, offering a well-thought-out solution to address it. Additionally, the ablation studies are exceptionally comprehensive, with thorough and insightful analysis throughout.", "weaknesses": "* The sparsity levels explored in the paper do not exceed 25%, leaving higher sparsity scenarios untested. In contrast, methods like LLMPruner evaluate and compare performance at a 50% sparsity level. This omission raises concerns about whether the proposed contiguous layer pruning approach would remain effective at higher sparsity levels.\n\n* The models selected for evaluation are all based on the LLaMA architecture, which limits the assessment of the proposed method’s generalizability. Testing on more diverse architectures (e.g., MoE, GQA) and larger models with greater sparsity requirements (such as LLaMA-30B, LLaMA-70B) would provide a more comprehensive validation of LLM-Streamline’s effectiveness across various model types and scales.\n\n* The study can add comparisons of real hardware performance (e.g., inference speed, FLOPs) with other structured pruning methods, where actual hardware constraints play a critical role in model selection and pruning effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FtVyI3uSeH", "reviewer": "Reviewer_MF5H", "summary": "This work addresses the compression of LLMs through layer pruning and layer replacement. A lightweight module is trained to substitute for the pruned layers. Additionally, a new evaluation metric is proposed to mitigate the limitations of accuracy-based metrics.", "questions": "Please see the above weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The concept of layer replacement for performance recovery is intriguing.\n* I appreciate the detailed comparison between perplexity and cosine similarity as pruning metrics, effectively revealing the data-sensitive drawbacks of perplexity.\n* The paper comprehensively covers various types of LLMs and evaluation benchmarks.\n* I like the exploration of multiple designs for lightweight networks.\n* Overall, the paper is well-written and easy to follow.", "weaknesses": "* I think that replacing layers with FFN or Transformer layers may not provide the full inference speed benefits typically gained through layer pruning, even if parameter counts remain similar. Could you analyze the speedup of layer replacement and compare it with pure layer pruning?\n* It’s unclear how to set the number of contiguous layers (n) in Equation (3). How many layers are typically removed at once during pruning? Could you elaborate on this aspect?\n* Is this method intended as one-shot pruning (pruning-replacement once) or iterative pruning (pruning-replacement, then pruning-replacement again)? Would it be possible to compare the two approaches?\n* Why is MSE chosen as the loss function for the replacement module in Equation (3)? What would happen if it were trained using the original language model loss?\n* A 25% compression rate seems relatively modest. Could you provide a comparison at higher pruning ratios (e.g., 50% layer removal) against baseline methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)