A Simple Linear Patch Revives Layer-Pruned Large Language Models
Authors: Xinrui Chen
Haoli Bai
Tao Yuan
Ruikang Liu
Kang Zhao
Xianzhi Yu
Lu Hou
Tian Guan
Yonghong He
Chun Yuan
Venue: NeurIPS 2025 poster ()
Keywords: Layer Pruning
Large Language Models
Model Compression
Primary area: deep_learning
TL;DR: This paper introduces LinearPatch, a plug and play technique that compensates for activation magnitude mismatches in layer-pruned large language models (LLMs), improving compression efficiency and performance with negligible inference overhead.

Abstract: Layer pruning has emerged as a widely used technique for compressing large language models (LLMs). However, existing layer pruning approaches often incur substantial performance degradation. We identify the majority of this degradation to a single yet previously overlooked issue: \textit{the mismatch of activation magnitudes at the pruning interface}. 
The pre-interface activations exhibit significantly different scales from the post-interface ones, causing the distributional shift as it propagates through the remaining layers. 
To address this issue, we introduce \textsc{LinearPatch}, a lightweight and plug-and-play technique that fuses two operations into one matrix multiply at the pruning interface: (i) a Hadamard transformation that suppresses massive outliers at particular tokens and (ii) a channel-wise scaling that aligns activation statistics. 
On LLaMA-3-8B, \textsc{LinearPatch} preserves up to \textbf{94.15\%} of the original model's performance when pruning 5 out of 32 layers, outperforming the previous state of the art by \textbf{4\%}. 
The patch can be further refined with 5K unlabeled samples via memory-efficient offline distillation, pushing the retention to 95.16\% within only 30 minutes on a single GPU.
Code is available at \url{https://github.com/chenxinrui-tsinghua/LinearPatch}.

A Simple Linear Patch Revives Layer-Pruned Large Language Models
Xinrui Chen', Haoli Bai²; Tao Yuan', Ruikang Liu', Kang Zhao?, Xianzhi Yu²,
Lu Hou?, Tian Guan', Yonghong He!, Chun Yuan
*Shenzhen International Graduate School, Tsinghua University
Huawei Technologies
baihaol @huawei.com yuancusz. tsinghua. .dd.. cn
Abstract
Layer pruning has emerged as widely used technique for compressing large language models (LLMs). However, existing layer pruning approaches often incur substantial performance degradation, We identify the majority of this degradation to a single yet previously overlooked issue: the mismatch of activation magnitudes at the pruning interface. The pre-interface activations exhibit significantly different scales from the post-interface ones, causing the distributional shift as propagates through the remaining layers. To address this issue, we introduce LINEAR LINEARPATCH, lightweight and plug-and-play technique that fuses two operations into one matrix multiply at the pruning interface: (i) a Hadamard transformation that suppresses massive outliers at particular tokens and (ii) chamnel-win scaling that aligns activation statistics. On LLaMA-3 3-BBB, LINEARPATCE preserves up to 94.15% of the original model's performance when pruning 5 out of 32 layers, outperforming the previous state of the art by 4%. The patch be further refined with 5K unlabeled samples via memory-efficient offline distillation, pushing the retention 16% within only 30 minutes single GPU. Code is available at /httpps //github. com/ ttwpoohhtttttttinnuussaaaaaaaaee
Introduction
Recent large language models (LLMs) have achieved remarkable progress toward artificial general intelligence 2 25 55 16 46,
31 19 49 50). However, their massive scale also incurs substantial computational and memory costs during deployment. To address this issue, variety of model compression techniques have been proposed, including quantization [5] 52
33 29 and
pruning 3
Among these, layer pruning has emerged as an attractive approach since it can be readily applied without relying on hardware-specifee optimizations or low-level kernel modifications [42] 26 & 37
18. contrast, unstructured pruning 20
17 44 is difficult accelerate efficiently due irregular
memory access patterns, while structured sparsity 3 48 and N:M sparsity [17 [7 often require modifications model architectures or specialized kernels. Layer pruning, on the other hand, simply removes redundant layers without additional dependencies. However, despite the simplicity, a critical challenge the sharp drop of performance.
In this work, we uncover a new phenomenon that explains this degradation: the mismatch of activation magnitudes across layers and tokens at the pruning interface. Specifically, when layers are pruned, the activations from the remaining layers often exhibit different scales, and the activations from the layer preceding the pruning interface may not align with those from the subsequent layer. This
"Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

mismatch is further exacerbated by the presence of massive outliers in the activations of special tokens (e.g.. [BOS] or delimiter tokens), which are commonly observed in LLMs [22 4 As result, pruned LLMs suffer from severe activation mismatch, ultimately leading to performance degradation.
To address this issue, we propose LINCARPATCH, a plug-and-play technique designed compensate for activation mismatches. The method can be seamlessly integrated with various pruning metrics. Concretely, LINEARPATCH first applies Hadamard transformation to suppress massive outliers in the activations of special tokens 32 43. It then introduces a channel-wise scaling parameter to bridge the gap in activation magnitudes. Leveraging spectral theory, both the Hadamard transformation and the diagonalized channel-wise scaling can be combined into a real synmetric matrix, which we insert at the pruning interface as LINEARPATCH. This approach incurs negligible inference overhead while effectively aligning activation magnitudes. Beyond alignment, we further enhance pruned LLMs through memory-efficient knowledge distillation. In particular, we fine-tune only the LINEAR PATCH matrix while freezing all other model parameters. This efficient training process requires only 5K samples and can be completed within 30 minutes on a single GPU for a 7B model.
Our empirical results demonstrate the effectiveness of LINEARPATCH across diverse LLMs and tasks. For example, on the question answering benchmark. LINEARPATCII preserves up 94.15% of the performance when pruning 5 layers from LLaMA-3-8B, significantly outperforming state-of-the-art methods such as -..M-ttremlliee (90.84%). Moreover, with our efficient knowledge distillation, the retained performance can be further boosted to 95.16%. These results highlight LINEARPATCH as a simple yet powerful solution for reviving layer- pryeer-runed LLMs with minimal overhead.
2
Related Work
Weight pruning for LLMs can be categorized into structured and unstructured pruning, depending on the regularity of the pruning pattern. Unstructured pruning removes individual weights based on their importance, without considering structural organization. A pioneering work 20 prunes weights with the smallest magnitudes, followed by retraining to restore accuracy. Wanda 44 extends this idea by pruning weights according to the product of their magnitudes and the corresponding input activations, outperforming standard magnitude- based pruning. While unstructured pruning is relatively straightforward and can achieve higher compression ratios, it cannot be efficiently accelerated due to irregular memory access.
Weight Pruning.
In contrast, structured pruming climinates entire groups of weights, thereby preserving computational efficiency. In addition, N:M sparsity only keeps N elements out of every M contiguous weights, which yields more flexibility compared to structured pruning [17 [7]. However, such approach typically requires architectural modifications customized low-level kernels for acceleration. Beyond N:M sparsity, other structured pruning techniques typically remove entire groups of parameters, such as attention heads, MLP neurons, or hidden dimensions [3] 48 40 24]. Although more
hardware- frrienly- than unstructured pruning, width pruning still introduces structural irregularities and inevitable requires re-training.
Layer Pruning. More recently, layer pruning has emerged as promising direction for compressing LLMs. Unlike width pruning. which often produces irregular architectures, layer pruning removes entire Transformer layers-including both attention and MLP modules, making it easier to deploy and accelerate. series of recent works have explored this direction. ShortGPT [37 evaluates layer importance via cosine similarity between layer inputs and outputs, pruning the least critical layers. SLEB [42) iteratively prunes redundant layers by measuring perplexity degradation on a calibration set. Shortened LLaMa (26) explores Taylor-based metrics and perplexity as pruning metrics, and employs LoRA (Low-rank adaptation) fine-tuning to recover accuracy. UIDL [18] introduces an angular distance metric to identify and remove consecutive layers, followed by QLoRA fine-tuning to mitigate accuracy loss. -tte-mmmemlline [8] identifies the least important consecutive layers using cosine similarity and replaces them with lightweight network, reporting that fine-tuning this lightweight network with MSE loss outperforms LoRA fine-tuning.
Despite their effectiveness, these approaches consistently overlook critical issue: the magnitude mismatch that arises at the pruning interface. We demonstrate that this mismatch highly detrimental to performance. In contrast, our proposed LINEARPATCH explicitly addresses this challenge by

---

introducing a lightweight yet effective maanitude-alignneen patch, achieving superior performance under both training-free free and post-training settings compared with existing layer pruning methods.
Method
3.1 Preliminaries on LLM Layer Pruning
LLMs primarily adopt the Transformer architecture, consisting of stack of Transformer decoder layers, each with residual structure. We denote the E-th Transformer layer as f f(X()) (( (()), where x(O and g represent its input activations and parameters, respectively. Under the prevalent pre-norm architecture, the input to the + 1) layer can thus be obtained by:
( x f(x(0) ((9)).
(1)
Pruning Metries. To identify redundant layers, several metries are commonly used, including cosine similarity (37 18 8) gradient-based scores [26 35 and perplexity scores [26] 42). For example, Lit--ttramlinne [8] removes contiguous layers with the highest cosine similarities among their input activations. The optimal pruning index e is determined by:
((+))
= arg max E (x(0, x ((ttl)eD
(+n)
where denotes the calibration set used to compute layer activations, and ( BXXXCC the input of the E-th layer, with batch size B, sequence length L, and hidden dimension C.
Layer Pruning. Suppose we remove consecutive layers starting from l" The (e* + n))thh layer then takes the input of the 8 -th layer as its own, i.e.,
(()+ +2) x (() f(x(() gi +n)).
In this work we adopt cosine similarity the pruning metric for most cases, though our approach is agnostic to the choice of metric. Additional analyses of pruning metrics are provided Appendix
However, we observe that layer pruming introduces large mismatches in channel magnitudes at the pruning interface, which severely degrade model performance, shown in Figure [ In Sections 3.2 and 3.3
we investigate two root causes of this issue and propose corresponding remedies.
3.2 Channel Magnitude Alignment
Layer-wise Channel Mismatch. We find that mismatched channel magnitudes directly impair the performance of pruned LLMs. As illustrated in Figure [a), hidden state magnitudes vary substantially across layers and channels. Removing layers exacerbates this discrepancy. Additional visualizations across different LLMs are provided in Appendix
To mitigate this, we statistically compute channel-wise scaling factors. For each channel k, we calculate the ratio of mean activation magnitudes between the inputs of the (e* + n)-th and e th layers over a calibration set, yielding scaling vector e RC with entries
dx E +n) = |X
(e" +R) .
ln.
Quantitative Evaluation. We further perturb (!") by a scaling factor a around d, i.c.,
xn) -o .d d f(ax(() d +n)).
(5)
Figure bb) shows that ( .d best restores magnitude alignment and improves perplexity. Deviating from the optimal scaling (a # 1) results in substantial performance degradation.

---

Original X (PPL: 35.89
S (PPL: 20.62)
250.03
190.40
187.20
244.85
Perplexity
Magnitrts
1.
pruned
10
35.68
88.83
w/o Scaling
755
10088 Channel
10
1/5
1/4
1/2
layor
(a) Layer-wist Channel Magnitude
(b) Channel Magnitude Scaling
Figure I: Visualization of layer-wise channel mismatch in pruned LLMs. Removing layers introduces
magnitude mismatches, which we address using channel magnitude alignment.
OF
6,-230.2
channel
charnel
(a) Distribution of token scalings
(b) Distribution token-wisee scalings in rotation space
Figure 2:
Violin plot of token-wise scaling mismatch in pruned LLMs. The violin width represents the estimated probability density of token-witg -ise scalings. After applying the Hadamard transformation, the scaling distributions become more concentrated, indicating reduced variance across tokens.
3.3 Token Magnitude Smoothing
Token-wise Scaling Mismatch. As suggested by recent research 132 53), there are massive outliers specific to particular tokens (c.g.. [BOS] and delimiter tokens) with magnitudes over 10a. Consequently, single channel scaling dy may fail to fit all tokens within that channel, illustrated in Figure [a). a). We quantify this mismatch by computing the standard deviation of token- Risce scalings:
+n) A.
aall" lt + + BC
(6)
where Xi,x E R² denotes the activations of channel k for batch i, and a(.) is the standard deviation. A small σa indicates consistent scaling across tokens. However, observe aa 2137.75 when pruning 9 layers from LLaMA-2-7B, reflecting severe token-level mismatch.
Hadamard Transformation. Recent works 30 34
45
show that applying Hadamard trans-
formations suppresses outliers. A Hadamard matrix 23 of size C' = 2n can be constructed recursively:
1 H2
v2
H₂n = H2 & H
For C' C' 2", we follow [4] and factorize C 2"m construct H. H2 H The orthogonality of Hadamard matrix (i.e., H H I) makes the following transformations equivalent:
x (e") H)H (x
(
where ((") 'H denotes the rotated activations of (* layer. The rotation effectively redistributes outliers among all channels and encourages more balanced distribution of activation across channels. We also provide mathematical proofs Appendix G With the rotated activations, thus more friendly to share the same scaling parameter d for all tokens, with aa down to 230.32.

---

LLM
Pruned LLM
Pruned LLM
Layer
Hadamard
replace
merge
Scalings D
Linear Patch
Hadamard H!
H+
L.1
Step1:
Layer Pruning
Step2: Channel Scaling in the Rotation Space
Step3: Matrix Fusion
Figure 3: Overview of LINEARPATCH. First, layers are pruned using a specified metric. Next, space. Finally, the scalings are
channel-wise scalings are estimated in the fused with Hadamard transformations to form LINEARPATCII, which supports efficient fine-tuning.
3.4 LINEARPATCH: the Ultimate Recipe
LINEARPATCH acts as a plug-and-play module that can be easily integrated to improve the layer- pruned LLMs. Specifically, we first apply Hadamard rotation to X (("), then scale in the rotated space with D = diag(d) d) E R CxC, The operations are fused into a single symmetric matrix P S
Xnew (8)) = (x"))
(x
where the last equality stems from the spectral theorem 21 i.e., any real symmetric matrix can be decomposed into an orthogonal matrix of eigenvectors (H) and diagonal matrix of real eigenvalues (D). The process is also visualized in Figure 3, where the patch matrix P can effectively bridge the gap for layer-prumed LLMs. In addition, LINEARPATCH also reduces transformation overhead and enables efficient fine-tuming since only single GEMM operation for the matrix multiplication is required, rather than three distinct GEMM operations.
Hemory-Efficieen Offline Knowledge Distillation. Conventional knowledge distillation requires loading both the teacher and student models into GPU memory, which prohibitive for LLMs due to the extreme memory overhead. In contrast, building on Equation (9), LINEARPATCH supports memory-efficient offline distillation strategy. By storing only the inputs and outputs of the teacher model, we keep it offline during knowledge distillation.
Given small training corpus X T (e.g., 5,000 samples), extract the top-K output logits probability distribution and their indices from the teacher's vocabulary. We set K = 100 in practice, which reduces memory usage by 320x compared storing the full 32K vocabulary. An ablation study on K is provided in Appendix F
Similarly, collect the top-K logits probability distribution o, from the student model (the pruned LLM) using the same indices as the teacher. We then optimize the patch matrix P by minimizing the Kullback-Leible (KL) divergence [ between the two distributions:
min ExeT KL 0.).
(10)
During fine-tuning, we remove the desitive-effinte constraint on P to allow greater flexibility, while freezing all other model parameters to minimize memory usage. The entire process is lightweight: for example, fine-tuning LLaMA-2.2 2-BBB requires only 30 minutes on a single NVIDIA V100 GPU.
We also experimented with minimizing the mean squared error (MSE) between the pruned and teacher layers following [8) but found that MSE led to overfitting and consistently inferior results compared to KL-based logits distillation.
5

---

Experiments
4.1 Setup
Models and Baselines. We evaluate LINEARPATCII on several open-source LLMs, including
LLaMA-2-7B/13B 47), LLaMA-3-8B [66. BBacchua2--BB (54), and DeepSeek-RI-Distill [19].
We compare against
layer pruning methods with diverse pruning metrics: gradient-
based LLM-Prunee [35]. perplexity-based SLEB 42 Taylor-based Shortened LLaMA [26], and
cosine similarity-base -hased ShortGPT 37 and L. -Stetttmmle [8. For LL-M-Punner, we follow the
block-wise pruning setup with the C4 calibration set using the official code. For SLEB and Shortened
LLaMA, we adopt the official implementation or use the released pruned-layer indices 2 For ShorGGPT and LLM--Stramllne, we reproduce the methods strictly following their published descriptions.s
Evaluation. We use three benchmarks for evaluation: perplexity (PPL), Massive Multitask Lan- guage Understanding (MMLU), and commonsense question answering (QA). For PPL. we evaluate language modeling on WikiText-2 (WIKI-2) (38), C4 [9], and PTB [3.. For MMLU, we report five-shot accuracy on the full benchmark [22]. For QA, we evaluate on nine commonsense QA tasks, reporting the average (Avg.). the weighted average (Weighted avg.) for MMLU, and both the average and retained performance (RP) for QA. Further details are provided in Appendix B
4.2 Implementation Details
Calibration and Fine-tuning. To determine pruned layers and initialize channel-wise scaling parameters, we use 128 randomly sampled sentences with sequence length 2048 from WikiText-2 for calibration. Ablation studies the calibration set size and distillation dataset size are reported in Appendix C calibration dataset in Appendix D and Appendix [E] respectively. For fine-tuning LINEARPATCII, we use AdamW with a learning rate of 1e-4, training for one epoch on 5,000 WikiText-2 sentences of length 2048.
Resource Consumption. Our implementation is based on PyTorch. All experiments are conducted on a single NVIDIA V100 GPU with 24GB memory. For a 7B model, initialization of LINEARPATCI takes ~0 seconds, while the optional fine-tuning process completes within 30 minutes.
Pruning Configurations. We follow prior work and restrict pruning ratios below 30%. Detailed layer pruning configurations, including the officially released model links, the number and indices of pruned layers for each baseline, are listed in Appendix A
4.3
Main Results
Our proposed LINEARPATCH can be seamlessly integrated with existing layer-wise pruning methods. Throughout the experiments, denote the combination of LINEARPATCH with ShorGPT and L.M-- Streamline as LINEARPATCH s and LINEARPATCHIJ]: respectively. In most cases, these two ap- proaches prune similar sets of layers, therefore, collectively refer them as LINEARPATCHE (SLL)
4.3.1 Comparison on Training-free Methods
We begin by evaluating the effectiveness of LINEARPATCH under training-free settings. Specifically, foeus commonsense question answering (QA) benchmarks and perplexity (PPL.) benchmarks, which are widely adopted to measure the retained reasoning and language modeling capabilities of pruned large language models (LLMs). To ensure faimesss comparison, none of the considered approaches are allowed to perform fine-tuning, In particular, for LLM-Pruner we exclude its LoRA- based fine-tuning stage, while for LLM-Streamline we follow its official protocol and denote the variant that discards layer replacement and offline distillation as -.MM-ttramliee (None). Results on more LLM backbones and additional benchmarks can be found in Appendix !
iwonsong- dev/SLEB Shortened http//orrssoo//LLL
https://github
SLEB: https:///ithuu com/

---

Table 1: Comparison on QA benchmark with training-free methods. Lp denotes the number of pruned layers and L, denotes the total number of layers of the model. The Ratio column represents the proportion (%) of pruning parameters to the total parameters of the model. Avg. column denotes the average accuracy (%) and RP column denotes the retained performance (%). Same interpretation is adopted in all the tables.
Model 1////
Method
Ratio
ARC-
ARC-e
BouiQ
HeSw
PIQA
WG
WSC
Race-h
CaPa
Avg.
RP
0/2
Donse
46.25
74.58
77.74
75.97
79.11 11
68. 18.98
50.59
39.62
87,00
690.98
100
LLMPruner
26.99
31:91
52.90
62.43
$4.41
57
SLEB
27.03
31.91
52.31
69.59
58.25
32.25
9/32
ShotGGTT
27.03
32.76
18.61
62.17
56.17
64.36
6/133
71.06
32:25
77,00
56.52
80.29
1L.M4 2.77
9/32
LLM--ireunline (Nome)
27.03
32.76
48.61
62.17
56.17
64.33 36
64.33
71. 16.06
32.25
77.00
56.52
80.29
9/32
E LINEARPATCM
26.78
13.45
55.22
62.14 14
57.67
67.48 46
65:11
77.29
34.93
79.00
59.14
X4.108
MPriner
20.56
35.24
60.61
62.47
61 61.66
75.41
54.78
31.67
90.000
83,80
SLEB
21.02
33.02
56.57
73.07
32.06
83.66
7/32
ShoeGGPT
21.02
36.18
$3.89
62.17
62,66
70. 40
65.98
77.29
33.78
81
60.39
86.06
7/32
LLM-Streamlig (Nooe)
21.02
36. 18
55.89
62:17
62.66
70 40
65.99
77.
33.78
81 K1.30
60.59
86.06
7/32
E LINLARPATCH
20.78
37.63
61.24
62. 14
63.49
70:46
65:90
79.49
36.46
85.00
62,42
88.88
Dense
49.06
77.40
80.61
79.35
80.52
72.38
86.81
40.48
91.00
100
10/40
MPMPreeer
23.00
39.51
67.26
72.24
77.91
57. $7.30
73.33
33.59
85,00
86.32
10/40
SLEB
24.37
39.93
66.01
66.76
68.24
75.63
63.61
75.16
36.91
84,00
61.07
87,51
L.aMA-2.10 138
10/40
ShortGP
24.57
43.00
63.51
58.20
69.33 69.29
72.52
69. 85
32.32
36.84
87.00
64.61
88.45
1Q140
I.I M---rmaaalinee (None)
24.37
43.00
63.51
58.30
699.29
72.52
69.88
81.32
36,84
87.00
64.61
88.45
10/40
LINEARPATC
24.17
44.20
65.53
62.39
70.15
69.61
81.68
38.09
8/40
LLMPruner
19.48
41.98
67.51
76.50
56.51
32.06
94.78
8/40
SLEB
19.50
36.43
61.83
62:32
67.03
75.08
62.51
78.02
34.83
83.00
62.34
84.74
K440)
ShoeGGPT
19.50
44.03
67.38
57.00
72.38
75.24
69.61
79.85
38.18
89.00
65.85
90.27
840
I. SiSreaamline (None)
19.50
44.03
67.35
57.00
72.58
75.24
69.61
79.85
38.18
89.00
65.85
90.27
LINLARPAYCHH MA7
19.30
44.54
69.53
67.74
75.02
75.68
69.38
39.71
92.00
68.26
93.45
Dense
53.41
77.78
72.85
86.45
40:19
89.00
73.44
100
7/32
LLMPriner
19.37
35.32
59.30
55.23
51 $1,48
72.58
59.98
67.03
31.39
81.00
$7.03
77.12
7/32
SLEB
19.01
34.04
60.06
45.17
62.01
74.05
55.01
67.40
32.82
74.00
56.06
76.08
7/32
ShorGGP
19.01
42.41
56.65
65.26
64.70
70.89
71.19
73.63
34.16
75.00
61.55
83.79
1LaMA-3. si8
7/32
L.M-Streamline (None)
18.80
19.01
43:17
28.92
39.56
33.26
72.85
70:17
55.56
59.71
75.82
24.02
37.51
60.00
77.00
44.29
7/32
LINEARPATCHI
18.80
34.39
51.26
57.52
49.31
63.33
63.22
72:53
29:95
67.00
54.28
73.57
5/32
LLMPtuner
13.59
39.51
68,10
71.28
64.69
76,33
64,48
74.36
35.60
78.00
63.59
86,23
5/32
SLEB
13.58
39.68
66,
54.71
67.39
75.90
62.51
73.63
34.16
83.00
61,90
83.88
ShorGGP
13.58
63.51
71.19
75.09
36.94
5/32
LLM-Streamllnn (None)
13.58
47.35
66
73.52
71,10
74.27
71.03
76.
36.65
84.00
66.74
90.84
5/32
LINEARPATCHE
13.37
45.73
68.60
73:30
70.71
76.01
73:09
79.85
38.18
82,00
67.50
91.91
5/32
LINLARPATCIEE
13:37
48.55
70.71
74.25
72.52
76.71
73.95
81.32
38.37
86.00
69.15
94.15
Results on QA Benchmarks. As presented in Table 1 LINEARPATCH consistently achieves improvements over prior training-free free pruning methods across multiple backbones. For instance, on the L-MM--2-7B model with 7 out of 32 layers pruned, LINEARPATCH attains a retained performance ratio of 88.88% clearly outperforming L.LM-Pruner (83.80%) and SLEB (83.66%). On the more recent LLMM--3888 model with 5 out of 32 layers pruned, our method yields an even more substantial improvement, achieving 94.15% retained performance, while LLM--runer and SLEB fall behind at 86.23% and 83.88% respectively. When the baseline pruning method, LINEARPATCH achieves the state-of-the-art performance. For example, on LLaMA-2-7B with 9 out of 32 layers pruned, LINEARPATCU reaches 84.08% retained performance, representing 3.79% gain over both ShorGGT and LLM--Sreamline. Similarly, on the larger LLaMA- 13B model with 8 out of 40 layers pruned, LINEARPATCHIS/AA secures 93.45% retained performance, outperforming both baselines by 3.18%. These results highlight that LINEARPATCH provides consistent benefits when incorporated into existing pruning pipelines.
SWAGGG/T/LLM-Sttrammiii is employed as
Results on PPL Benchmarks. We further examine the performance of pruned models on PPL. benchmarks, which offer a direct measure of language modeling ability. Lower perplexity values correspond to stronger generative modeling performance. As shown in Table 2 SLEB equipped with a PPL-based pruning metric displays moderate advantages in this evaluation but struggles to maintain accuracy on QA tasks. By contrast, approaches using cosine cemilarity-based pruning are more balanced, and among them, LINEARPATCH consistently achieves the best overall performance. For example, on LLaMA- --BB with out of 40 layers pruned, LINEARPATCH achieves an average perplexity of 18.10, 10, dramatically surpassing LLM-Pruner (35.06) and SLEB (36.61). A particularly striking case occurs on the LLaMA-3-8P model with 7 out of 32 layers pruned: LLMM-Stramllnn nearly collapses, producing an unacceptably high average PPL of 2839.3, indicating severe failure in retaining generative capability. sharp contrast, LINEARPATCHIL) effectively rescues the model without any additional training, restoring its performance to functional and competitive level. This
7

---

Table 2: Comparison on PPL benchmark with training- methods over LLaMA-2-7B and LLaMA- 3-8B with 7 out of 32 layers pruned.
Model
Method
WIKI-2
C4
PTB
PPI. avg.
Dense
5.47
6.97
22.51
11.65
SLEB
9.14
11.21
38.45
19.60
+LINEARPATCH
8.77
10.66
38,30
19.24
Taylor+
18.45
20.99
18
33.87
LLaMA-2-7B
ALINEARPATCH
13.84
15.28
48.26
ShonGPT
18.45
20.99
62.18
33.87
+LINEARPATCH
13.22
14.38
45,97
24.59
LtM-Streamline (None)
18.45
20.99
62.18
33.87
*LINEARPATCH
13.22
14.58
45.97
24.59
Dense
6.14
8.88
10.59
8.54
SLEB
13.12
16.76
21.04
16.97
+LINEARPATCH
11.97
15.74
19.55
15.75
Taylor+
2287.86
1.91.38
4741.90
2840.38
LLAMA-3-88
+1 +LINFARPATCH
208.88
235.66
264.97
236.44
ShonGPT
57.76
50.13
67.39
58.43
#LINEARPATCH
25.67
28.38
31.22
28.42
LLM-Streaminn Streamline (None)
2287.73
1491.37
4738.81
28.39.30
+LINEARPATCII
69.82
96.68
88.79
85.10
Table 3: Comparison QA benchmark with the SOTA post-training method LLM ----Sttramline (FFN).
Model
L,/L, Method
ARC-c
ARC-
BoolQ
HeSw
PIQA
WG
WSC
Race-h
Cul'a
Ang.
RP
79.11
39.62
L. LMM-22278
7/32
7/32
LINEARPATCH(;
Saemaiinaminn FT
37.63
60:48
61.24
62.14
70.18
63.49
63.75
69.86
70.46
65 90
67.48
30.95
79.49
37.51
36.46
79.00
85.00
62.42
90.00
88.88
7/32
LINEARPATCH
38.23
64.35
65.32
69.33
73.23
67.40
83.88
38.37
87.00
65.23
92.83
Dense
81.28
79.16
72.85
86.45
40.19
73.44
100
LLaMA-3-8B
5/32
4/32
LLM-Srneumlines PT
LINEARPATCINgg
48.55
30.03
39 39.94
70.71
74.25
63.32
72.52
49.19
59.
76.71
67.50 07.80
73.95
81.32
81.32
31.39
38.37
71.00
86.00
55.00
69.15
94,15
74.34
LINEARPATCH
demonstrates the robustness of LINEARPATCH in mitigating instability issues inherent in pruning strategies, ensuring stable recovery of language modeling performance.
Results on PPL Benchmarks. We further assess the effect pruning on PPL benchmarks, which provide direct evaluation of the language modeling ability of pruned LLMs. Lower perplexity values indicate stronger generative capacity. As summarized Table LineARPATCH consistently improves the robustness and performance of diverse pruning baselines across both LLaMA-2-7B and LLaMA-3-8B models. On LLaMA-2-7B with 7 out of 32 layers pruned, LINEAR PATCH yields significant improvements for all baseline methods. For instance, when combined with ShortGP and L.MM-Stramllnee (None), LINEARPATCH decreases the average PPL from 33.87 to 24.59, highlighting its ability consistently mitigate the performance degradation caused by pruning. On LLaMA-3-8B with 7 out of 32 layers pruned, the advantage of LINEARPATCH is even more pronounced. When applied to SLEB, LINEARPATCH improves average PPL from 16.97 to 15.75; for Taylor+, it reduces catastrophic degradation (2840.38) to a reasonable 236.49; and for ShortGPT, it lowers average PPL from 58.43 to 42.4.. These results clearly demonstrate that LINEARPATCH serves as general and effective enhancement across different pruning strategies.
4.3.2 Comparison on Post-training Methods
Beyond training-fin free pruming, we next compare LINEARPATCH with the state-of-the-an post-training method I.LM-Streamline, under identical fine-tuning configurations (i.e., same dataset, sample count, and learning rate). This setup allows us to fairly isolate the benefit introduced by our lightweight LINEARPATCH mechanism.
Results on QA Benchmarks. Table Billustrates that LINEARPATCH provides clear advantages when combined with post-training fine-tuning. On LLaMA-2-7B with out of layers pruned, LINEARPATCHIgH achieves 65.23% accuracy, outperforming LLM LLM--ttrammliee (63.05%). Sim- ilarly, on LLaMA-3-8B with 5 out of 32 layers pruned, our method achieves 70.01 % accuracy, fails to deliver competitive results due to its reliance on randomly ini-
whereas

---

Table 4: Comparison on PPL benchmark with SOTA post-training method LLM-Streamline.
Model
1u/L2
Method
Ratio
WIK1-2
C4
PTB
Avg.
0/32
Dense
0
5.47
6.97
22.51
11.65
7/32
LLM Streamlie (FFN) + FT
19.01
9.60
17.10
47.04
24.58
LLaMA-2-7B
7/32
LINEARPATCHIy
20.78
13.22
14.58
45.97
24.59
7/32
LINEARPATCH +FT
20.78
8.09
11.25
32.48
17:27
0/32
Dense
6.14
8.88
10.59
8.54
5/32
LLM LLM-Streamline (FFN) +
11.39
383.15
201.60
101.35
228.70 70
LLaMA-3-8B
5/32
LINEARPATCHIL
13.37
15.13
17.41
19.30
17.28
5/32
LINEARPATCHII FT
13.37
9.00
13.34
14.34
12.23
Table 5: Comparisons on tunable parameters, distillation type and loss functions. FT denotes fine- tuning. When distilling feature, we optimize the parameters of the replaced layer by aligning its output with MSE loss as ALM--treamline. When distilling logits, we optimize the parameters of the replaced layer by aligning the model output logits with KL loss.
Model
Parameters
Distillation
Loss
WIKI-2
C4
PTB
PPL Avg.
QA Avg.
QA RP
Dense
5.47
6.97
22.51
11.65
69.98
100
LLM-Streamline (FFN) FT
FFN FFN
Feature Logits
MSE
13.00
27.22
68.97
36.40
59.44
84.74 NaN
KI.
Symmetric Diugonal matrix Symmetric
MSE
15.26
19.54
54.33
L FT
Logils
KL
17.51
18.64
$1.64
29.26
59.40
84.45
Logits
8.60
12.98
37.16
tialized replacement layers. Notably, the retained performance (RP) of LINEARPATCHigj on LLaMA-3-8B reaches 95,16, underscoring its ability to maintain nearly all original model accuracy despite aggressive layer pruning. These findings reinforce that LINEARPATCH is well suited for post-training recovery, offering both stability and efficiency advantages.
Results PPL Benchmarks. As reported in Table 4 LINEARPATCH consistently matches or exceeds LLM-Streamline across different pruning configurations. For instance, LLaMA-2-7B with out of 32 layers pruned, our method achieves an average PPL of 17.27 compared to 24.58 for L.MM-Streamline. Likewise, on ALaMA-3-8BD with pruned layers, LLNEARPATCH(L)+FF attains 12.23 perplexity, reflecting its robustness and effectiveness in preserving language modeling quality under post-training scenarios.
The superiority of LINEARPATCH is also evident in evaluaaiais
4.4 Discussions and Ablation Studies
Tunable Parameters and Loss Functions. To further understand the design choices, conducted comprehensive experiments on LLaMA-2-7B with 9 out 32 layers pruned, systematically varying training settings such loss functions and distillation targets (see Table (5). Results reveal that training configurations have a marked impact on performance. In particular, using model logits as the distillation target combined with KI. divergence the loss function yields the best results: average PPL of 19.55 58 on language modeling tasks and an average QA accuracy of 61.71%, 17,, representing 2.13% improvement over Str--ttramline. Conversely, using replaced layer outputs for distillation tends to cause overfitting, resulting in inferior performance. Additionally. LLM-Streamling 's strategy of employing randomly initialized lightweight network introduces instability during training and can lead to degraded outcomes. These observations highlight the advantage of LINEARPATCH, which offers principled and stable initialization, enabling more reliable performance recovery through lightweight fine-tuning.
The Ingredients of LINEARPATCH. We further ablate the contribution of individual components of LINEARPATCH, as reported in Table Without any additional mechanism, a pruned model suffers severe degradation, with average PPL increasing 56.10 and QA retained performance dropping to 80.29%. Introducing scaling parameters d already mitigates the degradation, improving PPL to 33.70 and QA retention to 83.56%. Incorporating the Hadamard rotation to construct the full LINEARPATCH P further enhances results, reducing PPL to 30.29 and pushing QA retention to 84.08% corresponding to relative improvement of 3.8% over baseline pruning. Finally, applying fine-tuning top of LINEARPATCH provides additional substantial gains, with RP improved by

---

Table 6: Ablation study on the ingredients of LINEARPATCH over LLaMA-2-7B with 9 out of 32 layers pruned. +d applies channel scaling. +P (i.e., H refers the LINEARPATCH, and +FT denotes fine-tuning with knowledge distillation. Note that we omit ablating H since Hadamard transformation alone is an equivalent operation.
WIK1-2
Avg.
BoolQ
PIQA
Avg.
Dense
3.47
6.97
22.51
11.63
46.25
74.58
71.74
75.97
79:11
68.99
80.39
39.62
87.00
69.98
100
Vanilla
35.68
56.10
32.76
48.61
62.17
56.17
64.36
64.33
71.06
32.25
77.00
56.52
80.29
-d 20.68
22.75
$7.67
33.70
35.07
54.97
62.17
56.93
66.76
63.77
75.09
34.83
78.00
58.62
83.56
-P 18.60
19.28
53.00
30.29
53.53
55.
62.14
57.69
67.41
65.04
77.29
54.95
79.00
59.14
84.08
+FT 8.60
12.98
37.16
19.58
34.81
60.65
62.48
64.52
70.29
66.69
76.92
39.01
80.00
61.71
88.15
4.06%. These ablations confirm that each design element contributes meaningfully to the strong overall performance our approach.
Online Inference Overhead. We also evaluate the online inference overhead of LINEARPATCIL Experiments are conducted using batch size 16, sequence length 2048, 1000 iterations, hidden size 4096 (aligned with LLaMA-2-77B) and float16 precision. Our method, implemented linear layer inserted at the pruning interface, introduces virtually no latency end-to-end inference compared baseline methods. Empirical results indicate no measurable overhead relative to LLM- Streamline with its feed-forward network (FFN). Furthermore, LINEARPATCH is approximately 8x smaller in parameter size and around 190x faster than LL-M-Srramlline(FFN) making it significantly more efficient for deployment.
single
Offline Storage Overhead. Finally, examine the offline storage overhead. With Top-K logits distillation, LINEARPATCH reduces storage requirements by up to 40x for hidden size 4096, com- pared with L.M--Stramelie during offline fine-tuning. This efficiency stems from the much smaller intermediate tensors that need to be stored by our approach. Such reduction not only improves practical usability but also substantially lowers the resource cost of post-training adaptation, which is critical in large-scale deployment scenarios.
5 Conclusion
In this work, we introduce LINEARPATCH, simple yet effective aldg-and-play technique that addresses the critical issue of activation magnitude mismatch in layer-pruned large language models (LLMs). By leveraging the Hadamard transformation and channel-wist scaling, LINEARPATCH efficiently aligns activations across layers, significantly enhancing model performance with negligible inference overhead. Extensive empirical evaluations are conducted demonstrate the effectiveness of the proposed approach. We hope the proposed LINEARPATCH can shed more light on simple and lightweight algorithms of LLM compression without compromising the performance.
Limitation and Broader Impact
Limitation. Layer pruning may unevenly degrade model performance across different tasks. For instance, while some question answering tasks might remain robust, complex reasoning or context- dependent tasks could suffer. Future work should establish a framework to evaluate trade-offs between efficiency gains and task-specific performance.
Broader Impact. Layer pruning methods significantly reduce the computational costs of deploying large language models, making them more accessible to broader range of users. However, these methods do not address the social biases embedded in LLMs, which often stem from the training data and can affect fairness and inclusivity. It is crucial to ensure ethical deployment of LLMs.
7 Acknowledgment
This work was supported by the National Key R&D Program of China (3022YTB44014004470444)) SSTIC Grant (KJZD20230923)
019000023922115100011 15106011, KJZD202309231 1e000311199999300 14916032, 1HH220240218113604400
10

---

References
[1] Winogrande: An adversarial winograd schema challenge scale. 2019.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint MrXiv:2333.08774 2023.
[3] Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler. and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint axiv:2401.15024, 2024.
[4] Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian Croci, Bo Li, Martin Jaggi. Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. arXiv preprint ar xrXw::2404.00556 2024.
[5] Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang. Jing Jin, Xin Jiang. Qun Liu, Michael Lyu, and Irwin King. Binaryhert: Pushing the limit of bert quantization. arXiv preprint arXiv:2012.15701, 2020.
[6] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical common- sense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages
2020.
[7] Jerry Chee, Yaohui Cai, Volodymyr Kuleshow, and Christopher M De Sa. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024.
[8] Xiaodong Chen. Yuxuan Hu, and Jing Zhang. Compressing large language models by streamlining unimportant layer. arXiv preprint arXiv:2403. 19135, 2024.
[9] Xinrui Chen, Yizhi Wang, Yao Li, Xitong Ling, Mengkui Li, Ruikang Liu, Minxi Ouyang, Kang Zhao, Tian Guan, and Yonghong He. Low bit-width zero-shot quantization with soft feature-infused hints for iot systems. IEEE Internet of Things Journal, 2024.
[10] Xinrui Chen, Yizhi Wang, Renao Yan, Yiqing Liu, Tian Guan, and Yonghong He. Texq: zero-shot network quantization with texture feature distribution calibration. Advances in Neural Information Processing Systems, 36, 2024.
[11] Xinrui Chen, Renao Yan, Junru Cheng, Yizhi Wang, Yuqiu Fu, Yi Chen, Tian Guan, and Yonghong He. Adeq: Adaptive diversity enhancement for zero-shot quantization. In International Conference Neural Information Processing, pages 53-64. Springer, 2023.
[12] Xinrui Chen, Renao Yan, Yizhi Wang, Jiawen Li, Junru Cheng. Tian Guan, and Yonghong He. Hiq: One-shot network quantization for histopathological image classification. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1916-19200 IEEE, 2024.
[13] Xinrui Chen, Hongxing Zhang, Fanyi Zeng. Yongxian Wei, Yizhi Wang, Xitong Ling, Guanghao Li, and Chun Yuan. Prune&comp: Free lunch layer-pruned ilms via iterative pruning with magnitude compensation. arXiv preprint arXiv:2507.1822 2025.
[14] Christopher Clark, Kenton Lec, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 1rXiv:1905.1000 2019.
[15] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv:18000 netii-803005555 05457, 2018
[16] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Al-Dahle, Aiesha Letman. Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.2178 217.27733,, 2024.
[17] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can accurately pruned one-shot. In International Conference on Machine Learning, pages 10323-10337. PMLR, 2023.
[18] Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403. 21..22443..777777 17887, 2024.
11

---

[19] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, al. Deepseck-rl: Incentivizing reasoning capability in ilms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
[20] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in Neural Information Processing Systems, 28, 2015.
[21] Henry Helson. The spectral theorem. The Spectral Theorem, pages 23-41, 2006.
[22] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song. and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint MXXi:2009.033000 2020.
[23] Kathy J Horadam. Hadamard matrices and their applications. Princeton University Press, 2012.
[24] Yuxuan Hu, Jing Zhang, Zhe Zhao, Chen Zhao, Xiaodong Chen, Cuiping Li, and Hong Chen. Sp3: Enhancing structured pruning via pea projection. In Findings of the Association for Computational Linguistics ACL 2024, pages 3150-3170, 2024.
[25] Albert Q Jiang, Alexandre Sablayrolles. Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel. Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 2023.
[26] Bo-Kyeong Kim. Geonmin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung-Kyu Song. Shortened Ilama: A simple depth pruning for large language models. arXiv preprint arXiv:2402.02834, 11, 2024.
[27] Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations, arXiv preprint
04683, 2017.
[28] Hector Levesque, Ernest Davis, and Leora Morgenstem. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning, 2012.
[29] Changhao Li, Xinrui Chen, Ji Wang, Kang Zhao, and Jianfei Chen. Task spekispecific zero-shot quantization- aware training for object detection. arXiv preprint arXiv:2507. 16782, 2025.
[30] Haokun Lin, Haobo Xu. Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Lingi Song, Zhenan Sun. and Ying Wei. Rotation and permutation for advanced outlier management and efficient quantization of Ilms. arXiv preprint arXiv:2406.0177 2024.
[31] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 194377, 2024.
[32] Ruikang Liu, Haoli Bai, Haokun Lin, Yuening Li, Han Gao, Zhengzhuo Xu, Lu Hou, Jun Yao, and Chun Yuan. Intactkv: Improving large language model quantization by keeping pivot tokens intact. arXiv preprint wxXiv:2403.01444 2024.
[33] Ruikang Liu, Yuxuan Sun, Manyi Zhang, Haoli Bai, Xianzhi Yu, Tiezheng Yu, Chun Yuan, and Lu Hou. Quantization hurts reasoning? an empirical study on quantized reasoning models. Conference on Language Modeling, 2025.
[34] Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant-lIm quantization with learned rotations. arXiv preprint arXiv:2405. 16406, 2024.
[35] Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances in Neural Information Processing Systems, 36:21702 2222222222,700 2023.
[36] Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2)3113330, 1993.
[37] Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang. Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint 3XXix22403.038533 2024.
[38] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.078.. 2016.
12

---

[39] Colin Raffel, Noam Shazeer. Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal Machine Learning Research, 21(140):1-67, 2020.
[40]
Anthony Sarah, Sharath Nittur Sridhar. Maciej Szankin, and Sairam Sundaresan. Alama-nas: Efficient neural architecture search for large language models. arXiv preprint arXiv:2405.18377) 2024.
[41] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4938-4947, 2020.
[42] Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, and Jae-Joon Kim. Sleb: Streamlining Ilms through redundancy verification and elimination of transformer blocks. arXiv preprint arXiv:2402.00025 2024.
[43] Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. arxiv preprint arXiv:2402. 17762. 2024.
[44] Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter A simple and effective pruming approach for large language models. arXiv preprint Xiv:vv2306. 1695, 2023.
[45] Yuxuan Sun, Ruikang Liu, Haoli Bai, Han Bao, Kang Zhao, Yuening Li, Jiaxin Hu, Xianzhi Yu, Lu Hou, Chun Yuan, et al. Flaiquant: Flatness matters for Ilm quantization. arXiv preprint arXiv:2410.09426 2024.
[46] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1.5: Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501 1255999991119999 2025.
[47] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 230.::3300.09888. 2023.
[48] Tycho FA der Ouderaa, Markus Nagel. Mart Van Baalen, Yuki M Asano, and Tijmen Blankevoort. The IIm surgeon, arXiv preprint arXiv:2312.17244 2023.
[49] Yongxian Wei, Runxi Cheng, Weike Jin, Enneng Yang, Li Shen, Lu Hou, Sinan Du, Chun Yuan, Xiaochun Cao, and Dacheng Tao. Unifying multimodal large language model capabilittes and modalities via model merging. arXiv preprint 2rxiv:2505. 19892, 2025.
[50] Yongxian Wei, Anke Tang. Li Shen, Zixuan Hu, Chun Yuan, and Xiaochun Cao. Modeling multi-task model merging as adaptive projective gradient descent. In Forty-sccond International Conference on Machine Learning, 2025.
[51] Mengzhou Xia. Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared Ilama: Accelerating language model pre-training via structured pruning. arXiv preprint 1xXiv:23100066/0 2023.
[52] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference Machine Learning, pages 99089938099. PMLR, 2023.
[53] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. 1 17453, 2023.
[54] Aiyuan Yang. Bin Xiao, Bingning Wang. Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang. Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint Xiv:2333 10305. 2023.
[55] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint (ttv:2407.1067) 2024.
[56] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint Xiv: 1905.07830, 2019.
[57] Yingtao Zhang. Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. Plug-and- play: An efficient post-training pruning method for large language models, The Twelfth International Conference on learning Representations, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have summarized our contributions and concrete numbers of improvement in the abstract and introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer Conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
The paper does not include theoretical results. We conduct rigorous experi- ments detailed in Section 4 to support the results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We disclose all the information needed to reproduce the results in Section and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We utilize publicly available datasets for our experiments and plan to release the code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
ollicy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.ccpublle pwwppppguuuiieepeep Sodi:////uuddee/deeuuuubbiii
(CodeSubiisss wwtplubmmmsssiiii ssiniiolccyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Section 3 and Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps://euurips...... http://aaaapp..ccooooo...actttta
Answer: [Yes]
Justification:
We obey the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
In the foresecable future, there is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This work poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The datasets and models in this work and properly cited and used.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
20

---

A Details on Pruned Models
For our experiments, adopt the officially released LLMs from the sources listed in Table The corresponding pruned layer indices for different configurations and models are comprehensively presented in Table g
Table 7: Download links to officially released LLMs.
Model
Download Link
LLaMA-2-7B
https//huggingface
http:/giinnnacccccoommtt---iaa Jama-2-7B
LLaMA-2-13B
hitps//hugginggfaee ww/////oo/olamaaee.... mameettllaa/lllmmaa 1amaa2131m
LLaMA-3-8B
hitps:/hugginggfcc..o/o//o/ haammaaallmaa/llammaaaa 1-8B
DeepSeek-
BarsSeek-R--Diise R1-Distill
II-Distill-Qwen-P71 Qwen-7B
htpp://huggingface.com
R1-Distill Qwen-7B
DeepSeek- PoopSeek-RI-DDee
htts://hugginggfce.
Baichuan2-7B
Table Details of pruning settings.
Model
Lp/ L
Method
Pruned layer index
9/32
LINEARPATCH(S/A)
[21,30)
7/32
LINEARPATCH(S/0)
[23,30)
9/32
Taylor+
22,,7.225,2,,22222232211
LLaMA-2-7B
7/32
Taylor+
(28,27,25,22,26,2))
9/32
SLEB
14423,11.2211222:1552...11
7/32
SLEB
[14,2,11,22110227,155]]
7/32
(INEARPATCH
(22,29)
5/32
LINEARPATCI
[24,29)
7/32
LINEARPATCH
(23,30)
5/32
LINEARPATCHILI
(23,28)
LLaMA-3-8B
7/32
Taylor+
[24.25.26.22.22222227
5/32
Taylor+
(24,25,26,28,29)
7/32
SLEB
(10,25,11.261122922)
5/32
SLEB
[10,25,11,26,12]]
10/40
LINEARPATCH(S/1)
(26,36)
LLaMA-2-13B
8/40
LINEARPATCU(S//11
(28,36)
9/32
LINEARPATCH(S/0)
[22,31)
Baichuan2-7B
7/32
LINEARPATCH(S/21)
(23,30)
B Details of Evaluation Benchmarks
We use variety benchmarks for model evaluation, including the perplexity (PPL. benchmark (measured by the average perplexity score), the Massive Multitask Language Understanding (MMLU) benchmark and the question answering (QA) benchmark for model evaluation.
PPL.
For PPL benchmarks, we report the perplexity of language generation on WikiText2 [8 and PTB (36] datasets.
39
MMLU. For MMLU benchmark, we test the five-shot performance on the Massively Multitask
Language Understanding (MMLU) datasets 22
Commonsense QA. For QA benchmark, evaluate methods on commonsense QA tasks:
CRC-Challenge (ARC-e). ARC-Easy (ARC-e) [15], BoolQ (14). HellaSwag (HeSw) [56), PIQA [6)
WinoGrande (WG) M1. WSC273 (WSC) 228. Race-high (Race-h) 27 and CoPA 41).
For MMLU benchmark, we use the official code. For PPI. and QA benchmarks, we use the lm_eval harness.
library (version 0.4.4) from http:///ghhhhoccaoo/////ee//aa/e
21

---

C Ablation on Size of Calibration Set
We vary the size of the calibration (a subset of WIKI-2) evaluate its impact on the performance of LINEARPATCH in Table 2 The results show that a larger calibration set leads to better scaling parameters and improved performance, but the gains diminish beyond a certain size. calibration set of 128 samples provides good balance between computational efficiency and performance.
Table Ablation on the number of calibration samples for scaling parameters statistics over LLaMA- 2-7B with 9 out of 32 layers pruned.
Num of samples
WIKI-2
C4
PTB
64
18.61
19.29
53.03
128
18. 18.60
19.28
53.00
256
18.60
19.28
53,00
512
18.61
19.28
53.01
D Ablation on Calibration Data
To thoroughly investigate the impact of different calibration datasets and validate the robustness of our method, we provided experimental results of different calibration sets in Table 10 findings are follows:
The key
Domain-Specifer Calibration Boosts Performance: When PTB is used as the calibration set, the perplexity (PPL) on PTB drops to 52.25, significantly outperforming other calibration sets. This indicates that demain-specific calibration enhances performance in that domain.
Stability Under Domain Mismatch: It is common that using domain-matee matched calibration data yields the best results (diagonal entries in Table I0). We note that even with domain- mismatched calibration data, the increase in PPL is marginal. For example, calibrating with WIKI-2 results in the PPL of 19.28 C4, only 0.03 higher than the comhin-matched case (19.25). This confirms the stability of our method.
Robustness to Data Quality: Despite C4 being the lowest-quality calibration set, the average PPL (30.52) is only slightly higher than the best result (30.29). This minimal degradation demonstrates the robustness of our method to calibration data quality.
Table 10: Ablation on calibration datasets over LLaMA-2-7B with 9 out of 32 layers pruned.
Calibration set
Data quality
WIKI-2
PTB
C4
PPL avg.
WIKI-2
best
18.60
53.00
19.28
30.29
PTB
mediate
19.31
52.24
19.66
30.40
C4
poor
19.11
53.21
19.25
30.52
E
Ablation on Size of Distillation Dataset
To provide deeper analyses on how dataset size impacts recovery during knowledge distillation, we ablated the dataset size in the setting of removing 7 layers of LLaMA-2-7B, As shown in Table TT larger dataset for knowledge distillation tends to produce better performance, the benefits are limited above 5000 samples. The adopted setting weighs the performance and efficiency.
Table 11: Ablation study on the size of distillation dataset.
Cache
Avg.
ARC-
BoolQ
PIQA
Avg.
wlo FT
13.22
14.58
45.97
24.59
37.63
61.24
62.14
63.19
70.16
65.90
79.19
36.46
85.00
59.60
2500
8.22
11.25
32.79
17.42
38.82
63.89
64.86
69.12
73.01
68.43
81.32
37.89
68.00
62.82
5000
10G
800
11:25
32.48
17.27
38.23
64.35
69.53
73.23
38.3
87.00
7500
15G
8.07
11.24
33.37
17.56
38.91
61.14
66.06
69.49
73.45
68.03
80.95
38.09
88.00
65.24
22

---

Ablation on K in Top-K Logits for Knowledge Distillation
We ablate {= 100, 200) in top-K logits to show how K influence on knowledge distillation for pruned model. We removed 7 layers on LLaMA-2-78 and performed ablation experiments according to the fine-tuning settings. As shown in Table 12 benefits and costs are well balanced when K=100.
Table 12: Ablation study on on K in lop-K logits for knowledge distillation.
Cache
WIK1-2
Avg.
ARCe
BoolQ
PIQA
WG
CoPa
w/o
13.22
14.58
45.97
24.59
37.63
61.24
62.14
63.49
70.46
65.90
79.49
36.45
85.00
59.60
50
56
8.73
11.63
34.68
18.35
38.14
63.30
67.37
69.90
72.36
68.19
80.95
38.47
88.00
65.19
200
200
7.78
11.03
31.51
16.77
38.05
64.73
64.71
69.02
73.07
67.56
82.78
37.42
87.00
64.93
G Mathematical Proof on Outlier Reduction of Hadamard Transformation
We introduce incoherence processing 4 into activation with Hadamard transformation. A activa- tion matrix X E R=xm is y-incoherent all i and j. X = |e Xeyl S [[]]X|ll/mmmm Then, we obtain:
mar X): S |||X|/// //mm
(1I)
where mar is the Sement-wiss max of the matrix, and mn the number of elements.
H Results on Combination LINEARPATCE with More Layer Pruning Metric
LINEARPATCH allows for the combination of non-contiguous layer pruning settings, gradient-based and PPL-based layer pruning metric, combined with any pruning metric results in the improvement of performance. We demonstrate these in the following experiments.
H.1 Combination with Non-contiguous Layer Pruning Metric
We integrated LINEARPATCH with non-contiguous layer pruning settings. ShortGPT 337 accesses the importance of each layers with the prevalent cosine similarity. We replaced each removed layer in ShorGPT with a LINEARPATCH, As shown in Table 13 combining LINEARPATCH with non-contiguous layer pruning still significantly improves the performance of the pruned LLM. For example, on the LLaMA-3-8B with 7 layers pruned, LINEARPATCE achieved a 3.51% improvement performance on the QA task benchmark compared to ShonGPT.
H.2 Combination with Gradient-based Layer Pruning Metric
We integrated LINEARPATCH with gradient-based layer pruning metric taylor+ [26). We identify the redundant layers with gradient information and replaced each redundant layer with LINEARPATCH. As shown in Table [4, combining LINEARPATCH with gradient-based -based layer pruning metric signifi- cantly improves the performance. For example, on the LLaMA-3-8B model with layers pruned, LINEARPATCH achieved 7.64% improvement in performance on the QA task benchmark, with average PPL reducing from 2840.38 to 236.49.
H.3
Combination with base Layer Pruning Metric
We combined LINEARPATCH with the SLEB 42
which uses PPL-based pruning metric. SLEB selects layers with the least performance drop based on the PPL metric, and the removed layers are usually non-contiguous. We replaced each removed layer in SLEB with single LINEARPATCH. As shown in Table T5] LINFARPATCH consistently improves the pruned model's performance when combined with bas-based pruning metries, without any fine-tuning.
23

---

Table 13: Combination with non-contiguous cosine similarity- based pruning metric.
k,
WIKIZ
AARC
ARCr
BoolQ)
PIQA
Cofo
AAPE
Dens
11:65
74.55
77.74
79.11
$0.59
39.62
$7.00
09.96
L
7032
ShentiFT
1845
20:99
02.18
33.87
16.18
55.89
02.17
62.05
70,40
06
77.29
33.77
51.00
00.59
16.15
7/32
-LINEARPATCH
1384
15:25
68.26
25.79
37.71
60.04
00.37
63.32
70.76
665.9
78.15
35.96
$6.00
62.05
88.35
Dane:
10.59
8.54
| 53.41
77.78
81.28
79.15
80.85
72.85
96.43
40.19
89.00
73.44
3-4M
SbenGPT
57.76
50.13
56.65
63.26
70.89
71.19
75.63
34.16
75.00
61.54
$3.79
1253
14.27
3545
47.16
3687
41
60.05
06.35
72.83
70,09
78
46
35.89
10.00
64.30
87.30
Table 14: Combination with gradient-based layer pruning metric.
Model
Method
WIKI22
PTB
AAYG
ARC-
ARCie
Hooo))
Hebe
PIQA
WG
WSC
Raceh
CoPs
Apg
992
6.97
11.65
74.58
LLIMN-2-78
7/33
Tayfor
18:45
20.59
62.18
3337
35.18
55.89
62.17
62.65
70.80
65.96
77.25
33.78
81.00
50.59
$6/06
7135
13.84
15:28
08.26
25.79
37.71
60114
02:17
63.31
70.7%
65.59
76.75
35.94
84.00
$2.05
88.36
1614
LLMMA-3-8
1/3
Tayfoe-
3387
1491 34
4761 90
7540.76
16
7061
39.56
18.04
51.24
59.30
55.89
99.71
3400
60.00
46.76
99497
7132
"LoEAAPATCS
208.88
235.63
264.0
236.49
31.65
45:20
48.07
43.13
63.57
$7.54
63.37
27.85
67200
49.74
67.43
Table 15: Combination with PPL-bbased layer pruning metric.
Model
WIK1-
ARCC
Bod0
P1QA
WSC
032
Dense
5.47
6.97
22.51
11,65
| 46
74.55
77.34
75.97
79.01
68.95
$0.59
39.52
$7.00
09.95
100
117MA-2-77
7732
SLEB
9.14
11:21
35.45
19.60
| 33.002
56.57
63.91
62.49
13.07
58.96
60
32.56
$6.00
99.26
$3.65
-LINFARPATCH
8.77
10.55
36.10
19.28
12.13
57.55
66:19
61.71
71.79
99.43
69
37.83
K309
99.54
84314
Danc:
10.59
77.78
81.28
72.65
40.19
1.3
1890
SLFR
13
16.55
21.04
1697
14.04
60.06
45.17
62011
74.05
55.01
67 40
92.60
74.00
$6,06
76.08
732
"LINEAHPATCH
11.97
1574
19355
1575
33.79
01.33
43.53
61.95
74.37
35.49
65.37
33,49
78.00
57.00
77.30
I Results on More Models and Benchmarks
I.1 Comparison on QA Benchmarks with Training-free Methods
Table 16 shows more results on QA benchmarks. On the Baichuan2-78 model with 7 out of layers achieves retained performance ratio of 87.27% leading both ShortGPT
pruned, LINEARPATCH and LLMM S.tem--nre with 3.39%. When comes out of 32 layers pruned, LINEARPATCH (S/CH(/ still maintains the 81.66% of the original performance, outperforming ShortGPT and LLM-Streamline by 4.56%. We also provide the results of the latest DeepSeek-R --Dissii--i---iss-----aaaaaaaaa Distill-Qwen-71 models. The results show that [INEARPATCII is applicable for latest GQA-based large language models.
and DeepSeek-RI
I.2 Comparison on MMLU Benchmarks with Training-free Methods
We evaluate the proposed LINEAR method on the MMLU tasks across multiple models in Table 17 Overall, LINEARPATCH demonstrates significant improvements in weighted average accuracy across different models and pruning ratios. For example, it attains weighted average accuracy of 63.84% for LLaMA-3-S88 with 5 out of 32 layers pruned, outperforming the best results from other methods. Similarly, on LLaMA-2-13BB it reaches 53.96% and 54.01 % for 10 and 8 out of 40 layers pruned, respectively, where SL.B almost collapsed in the same case. These results highlight the robustness and effectiveness of LINEARPATCH in enhancing the performance of layer-prumed large language models on MMLU tasks, demonstrating its potential as simple yet powerful solution for reviving pruned models.
Visualization of Activation Magnitude of LLMs
See Figure for more visualization of the magnitude of L.M layer output activations. All layer- pruned model exhibit magnitude mismatch.
24

---

Table 16: Comparison on QA benchmark with training-free free methods. denotes the DeepSeek R1-- Distill version.
Ly/L,
BoolQ
PIQA
Avg.
032
Dense
42.49
72.98
73.91
77.20
79.85
38.28
9/3
LLMPrrner
23.29
32.42
56
59.83
54.11
69.70
$3.20
59.
28.04
78.00
54.55
79.64
9/32
SLEB
24.26
29.
48.91
62:29
52.14
68.88
55.09
66.30
30.43
75.00
54.25
79.19
Baichuan-2-1 2-10am---778
9/52
ShorGPT
24.26
28.67
12:55
67.19
47.09
62.68
62.19
68.23
29.38
65.00
52.66
77.10
LLM
LINTARPATCH
24.04
65.72
71.43
732
LLMPraner
18:47
36.86
62.63
61.25
72.03
$4.06
63.24
29.00
80.00
57.96
84.85
7/32
SLEB
18.87
31.31
55.39
65:47
$6.93
11.65
59 59.12
72.89
33.21
73.00
57.66
$4.46
7/52
ShorGGPT
18.87
34.90
51.81
62.39
55:27
65.56
61.72
74.73
31.77
72.00
57.02
83.88
MiM--nncalline (Nonc)
18.87
LINEARPATCH
0/28
Dense
44.62
06.84
78.38
60.77
12.42
00.77
67.03
36.36
74.00
62.35
100
Qwenn-7th
3/28
SLEB
21.42
32.17
52.82
39.42
46.08
66.97
53.20
56.41
23.
62.00
50.25
79.39
7/28
SharGPTT
21.42
33.62
58.00
56.24
45.89
66.97
53.67
58.61
30.72
60.00
51.52
82.58
Strem-iinll (None)
21.42
32.00
53.20
46.21
51.22
54.95
29.95
65.00
49.05
79.63
LINEARPATCHR
LLMA 3B
0/2
Dense
42.49
65.91
82.91
74:35
77.58
67.80
82.78
41.53
89.00
09.37
100
7/32
SLEB
19.01
34.30
54.38
62.75
57.27
70.08
55.33
66.67
32.92
76.00
56.63
81.45
7/52
ShorGGP
19.01
37.12
49.03
77.09
$5.95
66.49
63.14
70.33
33.21
71.00
58.13
83.72
Lira-Sreamllne (Nonc)
19.01
37.12
63.14
70.33
33.21
83.72
LINEARPATCUig
79.72
71.06
37.51
Table 17: Comparison on MMLU benchmark with training-
taining-free free methods.
Model
Lp/L,
Method
STEM
Humanities
Social sciences
Others
Weighed avg.
0/40
Dense
44.14
54.35
63.44
60.80
55.63
8/40
ShorGGT
42.80
50.13
62.78
61.19
53.88
LLaMA-2-13B
8/40
ILNEARPATCH (None)
43.07
50.61
62.56
61.04
54.01
8/40
42.80
50.13
62.78
61.19
53.88
+LINEARPATCH
43.07
50.61
62.56
61.04
54.01
Dense
55.20
$9.00
75.95
71.56
84.80
5/32
ShortGPT
46.92
53.92
63.65
65.42
57.64
LLMMA-3-88
5/32
+1 INFARPPACH
44.67
50.31
65.91
61.91
55.19
5/32
LLM- Streamliee (None)
53.47
$6.08
74.58
68.32
62.40
5/32
+LINEARPATCH
57.15
75.40
71.50
63.84
0/32
Dense
51.30
61.23
60.85
54.23
7732
ShorGGPT
42.01
45.48
58.17
55.71
49,88
Baichuan2-7B
7/32
+LINEARPATCH
42.84
48.42
59.60
58.70
52.00
7/32
LLM Strcmmiie (None)
42.01
45.48
$8.17
$5.71
49.88
7/32
+LINEARPATCH
48.42
58.70
52.00
0/32
Dense
43.25
51.77
52.47
45.90
7/32
ShonGPT
31.75
37.90
44.72
46.18
39.98
LJaMA-2-7B
7/32
HLINEARPATCH
31.71
39.26
45.82
47.07
40.88
7/32
LLM Streanline (None)
31.75
37.90
44.72
46.18
39.98
7/32
+LINEARPATCH
31.71
39.26
45.82
47.07
40.88
25

---

2011
1014
(a) LLaMA-2-78 Pruned (21-30)
LLaMA-2-78 Pruned layers (23-30)
Co11
10
LLaMA-2-138 Pruned layers (26-36)
LLaMA-2-138 Pruned layers (28-36)
(e) LLaMA-3-8B Pruned layers (23-30)
(f) LLaMA-3-88 Pruned 5 layers (23-28)
(g) Baichuan-2-78 Pruned layers (22-31)
(h) Baichuan-2-78 Pruned 7 layers (23-30)
30
10 Salle Channel
(1) DeepSeek-R1 Ri-iseii-RR---iiil--A-- Distill-Qwee-77 Pruned layers (13-20)
DeepSeek R1-------iii Distill-Llaam-BB Pruned 7 layers (22-29)
Figure 4: Visualization of the magnitude of LLM layer output activations, where pruned layers of LINEARPATCHIL) represented in grey. All layer-pruned model exhibit magnitude mismatch.
26


Reviews:
[{"review_id": "DYQsiumttA", "reviewer": "Reviewer_xhnK", "summary": "This paper proposes a novel framework to improve the performance of layer pruning in LLMs. This method is compatible with layer pruning methods that remove one or more consecutive layers. A patch, with a Hadamard transformation based initialization, is introduced at the location of the removed layers and then subsequently finetuned. Additionally, token-wise scaling is performed to bridge the gap between massive outliers at the different layers. The LinearPatch framework is compatible with layer pruning methods and outperforms the original versions of the layer pruning algorithms. The method is relatively efficient and lightweight, and a set of ablation studies demonstrate the contribution of each component to overall performance gains.", "questions": "N/A", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Overall I think that this is a good paper with a nice method. However, the writing needs to be improved and I would recommend copy editing of the entire paper. \n\nStrengths:\n- The method is compatible with existing layer pruning methods.\n- The LinearPatch method outperforms baselines on a range of tasks.\n- Ablation studies demonstrate the impact of each component of the pipeline.\n- The main figure and description offer a clear presentation of the main method.\n\nWeaknesses:\n- The language throughout the entire paper needs to be improved. Several sentences are quite unclear and hinder understanding of key concepts. For example: \"As a result, the pruned LLMs may suffer from efficient forward propagation as\n35 before, which ultimately leads to the drop in performance.\" in lines 34-35 are awkward, use non standard terminology, and are unclear.", "was_revised": true, "final_justification": "The authors addressed my concerns and I would like to maintain my original score.", "rebuttal": ">Q1: The language throughout the entire paper needs to be improved. Several sentences are quite unclear and hinder understanding of key concepts. For example: \"As a result, the pruned LLMs may suffer from efficient forward propagation as 35 before, which ultimately leads to the drop in performance.\" in lines 34-35 are awkward, use non standard terminology, and are unclear.\n\nA1: We sincerely appreciate the reviewers' recognition of the plug-and-play LinearPatch method proposed in this paper, as well as the reviewer's careful observation and constructive feedback. \n\n* We thoroughly revise the entire manuscript, focusing on refining sentence structures, eliminating awkward phrasing, and ensuring consistent terminology usage. \n\n* Regarding the specific example cited in lines 34-35, the revised text now reads: \"As a result, the pruned LLMs may suffer from $\\textit{activation mismatch}$, which ultimately leads to the drop in performance.\" This revision clarifies the technical concept and uses standard terminology aligned with the field."}, {"review_id": "dU3hAFMeWe", "reviewer": "Reviewer_oSnA", "summary": "This paper proposes LINEARPATCH, a simple and efficient plug-and-play method to restore the performance of layer-pruned large language models by addressing activation magnitude mismatches at the pruning interface, using a combination of Hadamard transformation and channel-wise scaling, and further improves results through lightweight offline knowledge distillation—demonstrating strong gains across multiple LLM architectures and benchmarks with minimal inference overhead.", "questions": "While the empirical benefits of Hadamard transformation and scaling are clear, can you provide more intuition or theoretical justification for why this combination is effective for aligning activation magnitudes after pruning? Additionally, are there identified scenarios where this approach fails or is suboptimal?", "limitations": "yes, authors addressed the limitations and potential negative societal impact of their work", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n1.LINEARPATCH is a lightweight, plug-and-play method that combines Hadamard transformation and channel-wise scaling into a single matrix, making it easy to implement and introducing negligible inference overhead.\n\n2.LINEARPATCH is shown to be agnostic to the pruning metric and compatible with multiple pruning strategies, and can be further improved with efficient, memory-light knowledge distillation.\n\n3.The method is extensively evaluated on multiple LLM architectures (e.g., LLaMA-2/3, Baichuan, DeepSeek) and various tasks (QA, PPL, MMLU), consistently outperforming prior state-of-the-art layer pruning approaches both in zero-shot and post-training scenarios.\n\n\nWeakness:\n\n1.The channel-wise scaling parameters rely on a calibration set. The impact of poor or domain-mismatched calibration data on final performance is not fully analyzed.\n\n2.The paper briefly acknowledges that certain tasks, especially those requiring complex reasoning, might still be more affected by layer pruning, but does not provide a systematic analysis or mitigation strategies.\n\n3.While the empirical results are strong, the paper does not provide theoretical analysis on why Hadamard transformation and scaling are sufficient for all types of activation mismatches.", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for your positive review and constructive comments. We have performed supplemented experiments and will include them in the camera-ready version.**\n\n>Q1: The channel-wise scaling parameters rely on a calibration set. The impact of poor or domain-mismatched calibration data on final performance is not fully analyzed.\n\nA1: We sincerely appreciate the reviewer's insightful comment. In layer pruning, WikiText-2 is a commonly used calibration dataset. **To thoroughly investigate the impact of different calibration datasets and validate the robustness of our method, we have supplemented experimental results using C4 and PTB as calibration sets. The results are presented in Table 1.**\n\n* Datasets Used in Experiments:\n\n1. WikiText-2: Extracted from Wikipedia's \"Good\" and \"Featured\" articles, it is one of the most widely used and reliable small-scale benchmarks in current language modelling evaluations. It has **high quality** due to manual verification and cleaning.\n\n2. PTB (Penn Treebank): Small in scale (approximately 1 million words), narrow in domain (Wall Street Journal articles from the 1980s), outdated, and limited in vocabulary, resulting in **moderate quality.**\n\n3. C4: Filtered from Common Crawl, it is large in scale but contains a significant amount of repetitive and low-quality web text, suffering from high noise, leading to relatively **low quality.**\n\n---\n\nTable 1: Ablation on calibration datasets over LLaMA-2-7B with 9 out of 32 layers pruned.\n| Calibration set | Data quality | WIKI-2 | PTB | C4 | PPL avg. |\n|---|---|---|---|---|---|\n| WIKI-2 | best | **18.6** | 53.00 | 19.28 | **30.29** |\n| PTB | mediate | 19.31 | **52.24** | 19.66 | 30.40 |\n| C4 | poor | 19.11 | 53.21 | **19.25** | 30.52 |\n\n* Key Findings:\n\n1. **Domain-Specific Calibration Boosts Performance**: When PTB is used as the calibration set, the perplexity (PPL) on PTB drops to 52.25, significantly outperforming other calibration sets. This indicates that calibrating with domain-specific data enhances performance in that domain.\n\n2. **Robustness to Data Quality**: Despite C4 being the lowest-quality calibration set, the average PPL (30.52) is only slightly higher than the best result (30.29). This minimal degradation demonstrates the robustness of our method to calibration data quality.\n\n3. **Stability Under Domain Mismatch**: Using domain-matched calibration data yields the best results (diagonal entries in Table 1). Even under domain mismatch, the increase in PPL is marginal. For example, calibrating with WikiText-2 on C4 results in a PPL of 19.28, only 0.03 higher than the domain-matched result (19.25). This confirms the stability of our method.\n\n\n>Q2: The paper briefly acknowledges that certain tasks, especially those requiring complex reasoning, might still be more affected by layer pruning, but does not provide a systematic analysis or mitigation strategies.\n\nA2: We thank the reviewer for highlighting this important point.\n\n* Currently, applying pruned models to complex reasoning tasks under training-free or lightweight training settings remains challenging. As shown in recent literature [1], pruning Llama-3.1-8B in a 2:4 format reduces its average sorting performance from 60.11 to 15.89. Similarly, our preliminary experiments using ShortGPT [2] to remove just three layers from DeepSeek-R1-Distill-Qwen-7B resulted in a significant drop in reasoning performance, with MATH-500 accuracy declining from 92.8 to 43.4.\n\n* We hypothesise that performance can be restored through retraining or constructing high-quality task-specific training data. Systematic analysis and mitigation strategies for complex reasoning tasks are currently unexplored and will be a key focus of our future work.\n\n---\n\n[1] Srivastava, G., Cao, S., & Wang, X. (2025). Towards reasoning ability of small language models. arXiv preprint arXiv:2502.11569.\n\n[2] Men, X., Xu, M., Zhang, Q., Wang, B., Lin, H., Lu, Y., ... & Chen, W. (2024). Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853.\n\n>Q3: While the empirical results are strong, the paper does not provide theoretical analysis on why Hadamard transformation and scaling are sufficient for all types of activation mismatches.\nQ4: While the empirical benefits of Hadamard transformation and scaling are clear, can you provide more intuition or theoretical justification for why this combination is effective for aligning activation magnitudes after pruning? Additionally, are there identified scenarios where this approach fails or is suboptimal?\n\nA3: We thank the reviewer for highlighting the theoretical analysis on Hadamard transformation.\n\n* In the revised manuscript, we have added detailed explanation of theoretical analysis on Hadamard transformation in the Hadamard Transformation paragraph of Section 3.3. Briefly, multiplication with the Hadamard matrix encourages a more balanced distribution of activation across channels.\n\n* We provide mathematical proofs below.\nA symmetric Hessian matrix $H\\in\\mathbb{R}^{n\\times n}$ is μ-incoherent if it has an eigendecomposition $H=Q\\Lambda Q^{\\top}$ such that for all $i$ and $j$, $|Q_{ij}|=|e_{i}^{\\top}Qe_{j}|\\leq\\mu/\\sqrt{n}$. One straightforward way to make a symmetric matrix incoherent is to conjugate it by a uniform random orthogonal matrix: this will result in each of its eigenvectors being a random unit vector, whose entries will concentrate around magnitude $n^{-1/2}$. [3][4]\n \nIn this work, we introduce incoherence processing into activation with Hadamard transformation. A activation matrix $X\\in\\mathbb{R}^{n\\times m}$is μ-incoherent if all $i$ and $j$,  $|X_{ij}|=|e_{i}^{T}We_{j}|  \\leq\\mu/||X||_F \\sqrt{nm}$.\n \nThen, we obtain:\n \n$max(X) \\leq\\mu||X||_F /\\sqrt{nm},$\n \nwhere $max$ is the element-wise max of the matrix, and $mn$ is the number of elements.\n \nIncoherence in activation $X$ can be viewed as a form of outlier reduction: a small bound on the magnitude of its entries across tokens means that we do not need to tailor a specific scaling parameter for each token and therefore eliminate the token-wise magnitude mismatch.\n\n---\n\n[3] Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Cameron, P., Jaggi, M., ... & Hensman, J. (2024). Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems, 37, 100213-100240.\n\n[4] Chee, J., Cai, Y., Kuleshov, V., & De Sa, C. M. (2023). Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 4396-4429."}, {"review_id": "5H9U3QN5mO", "reviewer": "Reviewer_9aSJ", "summary": "This paper proposes LINEARPATCH, a simple and plug-and-play method for pruning large language models (LLMs). The paper highlights that the mismatch in channel magnitudes directly impacts the performance of pruned LLMs and proposes a dimension-wise scaling $d$. Considering that massive outliers specific to particular tokens further degrade pruning effectiveness, this paper uses Hadamard transformations $H$. Finally, a Memory-Efficient Offline Knowledge Distillation strategy is employed to further enhance the performance. Experimental results demonstrate that, compared to baselines, LINEARPATCH achieves superior performance in preserving accuracy and computational efficiency.", "questions": "See weaknesses.", "limitations": "yes", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written.\n2. The proposed method is applicable to both training-free settings and fine-tuning.\n3. In simple terms, the paper highlights the problem of norm inconsistency after pruning and introduces a method for initializing the parameter matrix. Dimension-wise scaling and the proposed method are both intuitive.\n\nWeaknesses:\n1. As shown in Tables 15 and 16, the proposed method often fails to outperform the baselines. Additionally, the datasets and results in Table 1 differ from those reported in ShortGPT and LLM-Streamline. Therefore, I have serious doubts about the effectiveness of the proposed method.\n2. In the fine-tuning setting, does the proposed method outperform a randomly initialized P?\n3. In Table 1, why are the results of ShortGPT and LLM-Streamline identical for LLaMA-2-7B and LLaMA-2-13B, but different for LLaMA-3-8B?\n\n\nMinor:\n1. The left side of Figure 1(a) appears somewhat cluttered. Given the significant magnitude differences, the alignment lines seem unnecessary and make the overall figure cluttered.\n\n2. Changing \"channel\" to \"dimension\" may also improve readability.", "was_revised": true, "final_justification": "I acknowledge the methodological contributions of this paper; my concerns mainly focus on the evaluation. This paper lacks many of the datasets used in baselines, and the experimental results are inconsistent, which raises concerns about the effectiveness of the methods presented in this paper.", "rebuttal": "Thanks for your reviews. We provide point-by-point responses below.\n\n>Q1: As shown in Tables 15 and 16, the proposed method often fails to outperform the baselines. Additionally, the datasets and results in Table 1 differ from those reported in ShortGPT and LLM-Streamline. Therefore, I have serious doubts about the effectiveness of the proposed method.\n\nA1:\n\n**Baseline Comparison Issues**\n\n1. **Plug-and-play method**. We highlight that LinearPatch is a plug-and-play method that can be combined with different pruning metrics for further improvement. By **reorganzing the results** of the corresponding baselines from Table 11, 12, 13, and 15, it can be found the **consistent improvement** of LinearPatch relative to baselines, as shown below in Table A and Table B.\n\n2. **Corresponding baselines**. The baseline of ${LinearPatch_{[L]}}$ is LLM-Streamline (None), and the baseline of ${LinearPatch_{[S]}}$ is ShortGPT.   We have claimed this at the beginning of Section 4.3 Main Results (lines 216-218). Therefore, the proposed method **always outperform the baselines in Tables 15 and mostly outperform the baselines in Table 16 as presented in Table A, Table B and Table C.**\n\n3. **The overfitting of SLEB**. Tables 15 and 16 are placed together to illustrate that SLEB [1] with PPL pruning metric tends to overfit, resulting in suboptimal performance on QA and MMLU tasks, as also claimed in Appendix G.2 (line 494).\n\n\n**Reproducibility Issues**\n\nThe original papers of ShortGPT, LLM-Streamline and  Shortened LLaMA [2-4] do not claim the versions of lm_eval for the evaluation, as different versions of lm_eval may influence the results. For fair comparisons, **we consistently adopt lm_eval (v0.4.4) throughout our experiments**, and used **the same models and pruning indices as the baseline methods to ensure the reproduction.** All the configurations and code will be released for the community.\n\n---\n\n[1] Song, J., Oh, K., Kim, T., Kim, H., Kim, Y., & Kim, J. J. (2024, July). SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks. In International Conference on Machine Learning (pp. 46136-46155). PMLR.\n\n[2] Men, X., Xu, M., Zhang, Q., Wang, B., Lin, H., Lu, Y., ... & Chen, W. (2024). Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853.\n\n[3] Chen, X., Hu, Y., Zhang, J., Wang, Y., Li, C., & Chen, H. (2024). Streamlining redundant layers to compress large language models. arXiv preprint arXiv:2403.19135.\n\n[4] Kim, B. K., Kim, G., Kim, T. H., Castells, T., Choi, S., Shin, J., & Song, H. K. (2024). Shortened llama: A simple depth pruning for large language models. arXiv preprint arXiv:2402.02834, 11, 1.\n\n\n---\n\n\n**Tabel A.  Comparison on PPL benchmark over LLaMA-2-7B with 9 out of 32 layers pruned. (reorganized)**\n\n| Method | WIKI-2 | C4 | PTB | AVG |\n|---|---|---|---|---|\n| Dense | 5.47  | 6.97  | 22.51  | 11.65  |\n| SLEB | 9.14  | 11.21  | 38.45  | 19.60  |\n| +$LinearPatch$ | 8.77  | 10.66  | 38.30  | 19.24  |\n| Taylor+ | 18.45  | 20.99  | 62.18  | 33.87  |\n| +$LinearPatch$ | 13.84  | 15.28  | 48.26  | 25.79  |\n| ShotGPT | 18.45  | 20.99  | 62.18  | 33.87  |\n| +$LinearPatch$ | 13.22  | 14.58  | 45.97  | 24.59  |\n| LLM-Streanline(None) | 18.45  | 20.99  | 62.18  | 33.87  |\n| +$LinearPatch$ | 13.22  | 14.58  | 45.97  | 24.59  |\n \n---\n\n**Tabel B.  Comparison on PPL benchmark over LLaMA-3-8B with 7 out of 32 layers. (reorganized)**\n\n\n| Method | WIKI-2 | C4 | PTB | AVG |\n|---|---|---|---|---|\n| Dense | 6.14  | 8.88  | 10.59  | 8.54  |\n| SLEB | 13.12  | 16.76  | 21.04  | 16.97  |\n| +$LinearPatch$ | 11.97  | 15.74  | 19.55  | 15.75  |\n| Taylor+ | 2287.86  | 1491.38  | 4741.90  | 2840.38  |\n| +$LinearPatch$ | 208.88  | 235.63  | 264.97  | 236.49  |\n| ShotGPT | 57.76  | 50.13  | 67.39  | 58.43  |\n| +$LinearPatch$ | 25.67  | 28.38  | 31.22  | 28.42  |\n| LLM-Streanline(None) | 2287.73  | 1491.37  | 4738.81  | 2839.30  |\n| +$LinearPatch$ | 69.82  | 96.68  | 88.79  | 85.10  |\n \n\n---\n\n\n**Tabel C.  Comparison on MMLU benchmark. (reorganized)**\n\n| Model | Puned/total layers | Method | STEM | Humanities | Social sciences | Others | Weighed average accuracy |\n|---|---|---|---|---|---|---|---|\n| LLaMA-2-13b | 0/40 | Dense | 44.14% | 54.35% | 63.44% | 60.80% | 55.63% |\n|  | 8/40 | ShotGPT | 42.80% | 50.13% | 62.78% | 61.19% | 53.88% |\n|  | 8/40 | +$LinearPatch$ | 43.07% | 50.61% | 62.56% | 61.04% | 54.01% |\n|  | 8/40 | LLM-Streanline(None) | 42.80% | 50.13% | 62.78% | 61.19% | 53.88% |\n|  | 8/40 | +$LinearPatch$ | 43.07% | 50.61% | 62.56% | 61.04% | 54.01% |\n| LLaMA-3-8b | 0/32 | Dense | 55.20% | 59.00% | 75.95% | 71.56% | 64.80% |\n|  | 5/32 | ShotGPT  | 46.92% | 53.92% | 65.65% | 65.42% | 57.64% |\n|  | 5/32 | +$LinearPatch$ | 44.67% | 50.31% | 65.91% | 61.91% | 55.19% |\n|  | 5/32 | LLM-Streanline(None) | 53.47% | 56.08% | 74.58% | 68.32% | 62.40% |\n|  | 5/32 | +$LinearPatch$ | 54.24% | 57.15% | 75.40% | 71.50% | 63.84% |\n| Baichuan2-7b | 0/32 | Dense | 44.53% | 51.30% | 61.23% | 60.85% | 54.23% |\n|  | 7/32 | ShotGPT | 42.01% | 45.48% | 58.17% | 55.71% | 49.88% |\n|  | 7/32 | +$LinearPatch$ | 42.84% | 48.42% | 59.60% | 58.70% | 52.00% |\n|  | 7/32 | LLM-Streanline(None) | 42.01% | 45.48% | 58.17% | 55.71% | 49.88% |\n|  | 7/32 | +$LinearPatch$ | 42.84% | 48.42% | 59.60% | 58.70% | 52.00% |\n| LLaMA-2-7b | 0/32 | Dense | 36.98% | 43.25% | 51.77% | 52.47% | 45.90% |\n|  | 7/32 | ShotGPT | 31.75% | 37.90% | 44.72% | 46.18% | 39.98% |\n|  | 7/32 | +$LinearPatch$ | 31.71% | 39.26% | 45.82% | 47.07% | 40.88% |\n|  | 7/32 | LLM-Streanline(None) | 31.75% | 37.90% | 44.72% | 46.18% | 39.98% |\n|  | 7/32 | +$LinearPatch$ | 31.71% | 39.26% | 45.82% | 47.07% | 40.88% |\n\n \n---\n\n\n>Q2: In the fine-tuning setting, does the proposed method outperform a randomly initialized P?\n\nA2: **LinearPatch addresses the issue of layer-wise and token-wise mismatch in activation magnitudes**, and can quickly restore performance combined with lightweight training. **In contrast, inserting a random matrix into the LLM exacerbates such mismatch.** A randomly initialized P makes layer-pruned LLM diverge with loss NAN during the output distillation fine-tuning process.  In comparison, our work offers an effective initialization method for the patch.\n\n\n\n\n>Q3: In Table 1, why are the results of ShortGPT and LLM-Streamline identical for LLaMA-2-7B and LLaMA-2-13B, but different for LLaMA-3-8B?\n\nA3: \n \nShortGPT [2] and LLM-Streamline [3]  share the same set of pruned layers, and thus have the same performance. For LLaMA-2-7B and LLaMA-2-13B, the pruning indices of the two methods are the same, whereas they differ for LLaMA-3-8B. We have reported the pruned layer indices of these methods in Appendix A.\n\n\n>Q4: The left side of Figure 1(a) appears somewhat cluttered. Given the significant magnitude differences, the alignment lines seem unnecessary and make the overall figure cluttered.\n\nA4: We thank the reviewer for pointing out the clarity issue in Figure 1(a). The alignment lines were intended to illustrate the gap before and after alignment, as well as the effect of alignment. We believe these auxiliary lines effectively help readers understand the process of activation scaling.\n  \n \n>Q5: Changing \"channel\" to \"dimension\" may also improve readability.\n\nA5: In the manuscript, **the term \"channel\" follows the conventions established in references [5]–[10], which are widely cited in the model compression community.** This term is commonly used in structured pruning techniques, such as \"channel-wise pruning\" described in [5], and \"scale activations  described channel-wise\" in [8]. \n\n---\n\n\n[5] Ma, X., Fang, G., & Wang, X. (2023). Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36, 21702-21720.\n\n[6] Nrusimha, A., Mishra, M., Wang, N., Alistarh, D., Panda, R., & Kim, Y. (2024). Mitigating the impact of outlier channels for language model quantization with activation regularization. arXiv preprint arXiv:2404.03605.\n\n[7] Sun, M., Liu, Z., Bair, A., & Kolter, J. Z. (2023). A Simple and Effective Pruning Approach for Large Language Models. In The Twelfth International Conference on Learning Representations.\n\n[8] Frantar, E., & Alistarh, D. (2023, July). Sparsegpt: Massive language models can be accurately pruned in one-shot. In International conference on machine learning (pp. 10323-10337). PMLR.\n\n[9] Zhang, Y., Bai, H., Lin, H., Zhao, J., Hou, L., & Cannistraci, C. V. (2024). Plug-and-Play: An Efficient Post-training Pruning Method for Large Language Models. In The Twelfth International Conference on Learning Representations.\n\n[10] Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Cameron, P., Jaggi, M., ... & Hensman, J. (2024). Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems, 37, 100213-100240."}, {"review_id": "J4PDzjchjl", "reviewer": "Reviewer_e7ph", "summary": "This work identifies activation magnitude mismatch before/after pruning site as a cause for performance drop observed during layer pruning. The authors proposed Hadamard Transofmration as a remedy. The proposed technique was extensively validated and well ablated.", "questions": "Can you comment on the mechanism with which Hadamard transformation can mitigate outlier activations? I think this is assumed knowledge in this paper, but is worth explaining to make this work self-contained.\n\nReading the ablation studies, it seems like this Hadamard transformation is not really introducing significant benefits (mostly helps language modling metrics, not downstream task performance). It would be interesting to see this technique without Hadamard transformation, but finetuned. I believe the significance of this work will be further enhanced if the solution/mitigation can be further simplified.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n- Compelling motivation and analysis of the pathology from layer pruning.\n- The designed method is simple and effective.\n- The authors presented compelling experimental evaluation, as well as thorough ablation.\n- The subject of study -- layer pruning -- is conceptually simple and easy to deploy and has potential for wide real world impact.\n- Overall I think this paper presents a solid engineering contribution.\n\nWeakness:\n- Biggest weakness is writing -- for example, the phrase \"pruning interface\" is used without definition; in abstract, the author claims that they can remove 5 layers from llama-3 8B model, but does not mention how many layers in total are there in this model.\n- Appears that the pruning regime studied is fairly conservative (<=30%), which limits the significance of the proposed technique in real-world settings.", "was_revised": true, "final_justification": "I appreciate the authors' thorough follow-up and my questions and concerns are addressed. I maintain my recommendation of acceptance.", "rebuttal": "Thanks for your invaluable reviews. We provide point-by-point responses below.\n\n>Q1: Biggest weakness is writing -- for example, the phrase \"pruning interface\" is used without definition; in abstract, the author claims that they can remove 5 layers from llama-3 8B model, but does not mention how many layers in total are there in this model.\n\nA1: We sincerely appreciate the reviewer's feedback on the manuscript's clarity.\n* To address the concerns, we have revised the terminology and provided explicit definitions for technical terms such as \"pruning interface\" in the context.\n* Additionally, we have included the total number of layers in the Llama-3 8B model (32 layers) when discussing the removal of 5 layers, both in the abstract and relevant sections, improving the reader's understanding of the experimental setup.\n \n>Q2: Appears that the pruning regime studied is fairly conservative (<=30%), which limits the significance of the proposed technique in real-world settings.\n\nA2: Unlike other pruning methods, **layer pruning exhibits an intuitive positive correlation between pruning ratio and speedup**, and does not rely on additional software or hardware support (such as CUDA operators or device computing cores), offering advantages in practical applications.\n\n* Structured sparsity like 2:4 sparsity, even with a **50% pruning ratio**, achieves only a **1.24× speedup** on LLaMA-7B and requires specific hardware devices [1].\n\n* Layer pruning with a **30% pruning ratio** can reduce latency by nearly 30%, equivalent to a **1.43× speedup**.\n\nFurthermore, this compression ratio strikes a balance between practical acceleration and model performance. For instance, when the pruning ratio of LLaMA-2-13B is 24.17%, it retains 90.49% of the QA performance.\n\n---\n[1] Sun, M., Liu, Z., Bair, A., & Kolter, J. Z. A Simple and Effective Pruning Approach for Large Language Models. In The Twelfth International Conference on Learning Representations.\n\n\n>Q3: Can you comment on the mechanism with which Hadamard transformation can mitigate outlier activations? I think this is assumed knowledge in this paper, but is worth explaining to make this work self-contained.\n\nA3: We thank the reviewer for highlighting the mechanism of Hadamard transformation.\n\n* In the revised manuscript, we have added detailed explanation of mechanism of Hadamard transformation in the Hadamard Transformation paragraph of Section 3.3. Briefly, multiplication with the Hadamard matrix encourages a more balanced distribution of activation across channels.\n\n* We provide mathematical proofs below.\nA symmetric Hessian matrix $H\\in\\mathbb{R}^{n\\times n}$ is μ-incoherent if it has an eigendecomposition $H=Q\\Lambda Q^{\\top}$ such that for all $i$ and $j$, $|Q_{ij}|=|e_{i}^{\\top}Qe_{j}|\\leq\\mu/\\sqrt{n}$. One straightforward way to make a symmetric matrix incoherent is to conjugate it by a uniform random orthogonal matrix: this will result in each of its eigenvectors being a random unit vector, whose entries will concentrate around magnitude $n^{-1/2}$. [2][3]\n \nIn this work, we introduce incoherence processing into activation with Hadamard transformation. A activation matrix $X\\in\\mathbb{R}^{n\\times m}$is μ-incoherent if all $i$ and $j$,  $|X_{ij}|=|e_{i}^{T}We_{j}|  \\leq\\mu/||X||_F \\sqrt{nm}.$.\n \nThen, we obtain:\n \n$max(X) \\leq\\mu||X||_F /\\sqrt{nm},$\n \nwhere $max$ is the element-wise max of the matrix, and $mn$ is the number of elements.\n \nIncoherence in activation $X$ can be viewed as a form of outlier reduction: a small bound on the magnitude of its entries across tokens means that we do not need to tailor a specific scaling parameter for each token and therefore eliminate the token-wise magnitude mismatch.\n\n---\n\n[2] Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Cameron, P., Jaggi, M., ... & Hensman, J. (2024). Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems, 37, 100213-100240.\n\n[3] Chee, J., Cai, Y., Kuleshov, V., & De Sa, C. M. (2023). Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 4396-4429.\n\n\n\n \n> Q4: Reading the ablation studies, it seems like this Hadamard transformation is not really introducing significant benefits (mostly helps language modling metrics, not downstream task performance). It would be interesting to see this technique without Hadamard transformation, but finetuned. I believe the significance of this work will be further enhanced if the solution/mitigation can be further simplified.\n\nA4: \n* Ablation Study on Hadamard Transformation\n\nWe have supplemented the ablation experimental results based on the LLaMA-3-8B model in Table 1, which confirm that the Hadamard transformation remains effective in downstream task like QA benchmark. Under the training-free configuration, the Hadamard transformation brings a 1.05% relative performance improvement, and this gain is of practical value. It should be noted that the equivalent transformations formed by Hadamard orthogonal matrices do not affect performance, while the core goal of introducing the Hadamard transformation is to achieve the smoothing of activations.\n\n---\nTable 1: Ablation study on the ingredients of LINEARPATCH over LLaMA-3-8B with 5 out of 32 layers pruned. +d applies channel scaling, +P(i.e., ${HDH^\\top}$) refers to the LINEARPATCH.\n|  | ARC-c | ARC-e | BoolQ | HeSw | PIQA | WG | WSC | Race-h | CoPa | AVG | RP |\n|---|---|---|---|---|---|---|---|---|---|---|---|\n| Dense | 53.41 | 77.78 | 81.28 | 79.16 | 80.85 | 72.85 | 86.45 | 40.19 | 89.00 | 73.44 | 100.00 |\n| Vanilla | 47.35 | 66.20 | 73.52 | 71.10 | 74.27 | 71.03 | 76.56 | 36.65 | 84.00 | 66.74 | 90.84 |\n|  +d | 48.63 | 69.99 | 74.07 | 72.63 | 75.95 | 72.61 | 79.12 | 37.51 | 85.00 | 68.39 | 93.10 (+2.26%) |\n|  +P | 48.55 | 70.71 | 74.25 | 72.52 | 76.71 | 73.95 | 81.32 | 38.37 | 86.00 | 69.15 | 94.15 (+1.05%) |\n\n\n* Explanation of Finetuned Parameter\n\nFurther simplifying the approach is worthy of exploration and will also be the focus of our subsequent research. In the current study, we found that when only the scaling parameters are trained while removing the Hadamard transformation, the model performance will be at a suboptimal level (see Table 2 for details). In future research, we plan to integrate the Hadamard transformation into the model weights through equivalent transformations, thereby reducing the computational complexity of this part.\n\n---\nTable 2: Ablation on finetuned  parameters over LLaMA-2-7B with 9 out of 32 layers pruned. RP denotes the retained performance. d denotes channel scaling, +P(i.e., ${HDH^\\top}$) refers to the LINEARPATCH.\n\n| Parameters | WIKI-2 | C4 | PTB | PPL avg. | QA RP |\n|---|---|---|---|---|---|\n| w/o (Dense) | 5.47 | 6.97 | 22.51 | 11.65 | 100 |\n| w/o (Vanilla) | 35.68 | 36.10 | 96.52 | 56.10 | 80.29 |\n|  d | 17.51  | 18.64  | 51.64  | 29.26 | 84.45 (+4.16%)|\n|  P | 8.60  | 12.98  | 37.16  | 19.58 | 88.15 (+7.86%)|"}]

Author rebuttal:

Meta-review: This paper proposes LinearPatch, a lightweight and plug-and-play framework to mitigate activation mismatches in large language models subjected to layer pruning. The method combines Hadamard transformation, channel-wise scaling, and optional lightweight distillation to restore performance. It is pruning-metric agnostic and compatible with multiple pruning strategies, showing consistent improvements across various LLMs and benchmarks. Extensive ablation studies highlight the contributions of each component, and the approach achieves practical efficiency gains without requiring specialized hardware. The work identifies an important pathology in pruning and proposes a simple, deployable remedy with broad empirical validation.

During the review process, three reviewers provided positive scores (4, 4, and 5). Their main concerns have been addressed during the rebuttal. Reviewer 9aSJ raised several critical points, but many of them were also addressed by the authors (e.g., reproducibility, ablation, evaluation setup). Some of this reviewer’s additional demands (such as reproducing baselines from unspecified versions or providing links/emails outside the permitted review scope) are not reasonable within the review process.

Overall, the paper offers a technically sound, practically useful, and well-validated contribution to LLM pruning. Given the majority of positive reviews, the successful addressing of concerns, and the broad empirical validation of the approach, I recommend acceptance.
Decision: Accept (poster)