Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation
Authors: Fei Wang
Li Shen
Liang Ding
Chao Xue
Ye Liu
Changxing Ding
Venue: NeurIPS 2025 poster ()
Keywords: Structured Pruning; Post Training
Primary area: deep_learning

Abstract: Large Language Models (LLMs) excel at natural language processing tasks, but their massive size leads to high computational and storage demands.
Recent works have sought to reduce their model size through layer-wise structured pruning.
However, they tend to ignore retaining the capabilities in the pruned part. 
In this work, we re-examine structured pruning paradigms and uncover several key limitations: 1) notable performance degradation due to direct layer removal, 2) incompetent linear weighted layer aggregation, and 3) the lack of effective post-training recovery mechanisms.
To address these limitations, we propose CoMe, including a progressive layer pruning framework with a Concatenation-based Merging technology and a hierarchical distillation post-training process. 
Specifically, we introduce a channel sensitivity metric that utilizes activation intensity and weight norms for fine-grained channel selection. 
Subsequently, we employ a concatenation-based layer merging method to fuse the most critical channels in the adjacent layers, enabling a progressive model size reduction. 
Finally, we propose a hierarchical distillation protocol, which leverages the correspondences between the original and pruned model layers established during pruning, enabling efficient knowledge transfer.
Experiments on seven benchmarks show that CoMe achieves state-of-the-art performance; when pruning 30% of LLaMA-2-7b's parameters, the pruned model retains 83% of its original average accuracy.

Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation
Fei Wang!.2 1.2, Li Sh Liang Ding', Chao Xue?, Ye Liu', Changxing Ding!:b.
South China University of Technology 2D Explore Academy
SShenzhen Campus of Sun Yat-sen University UUiversity of Sydney Paahou Lab Center for AI Theoretical Foundation and Systems, Shenzhen Loop Area Institute ft feiw @ scut.e..u., chxding@scut.edu.cg
Abstract
Large Language Models excel at natural language processing tasks, their mas- sive size leads to high computational and storage demands. Recent works have sought to reduce their model size through layer-wise structured pruning. However, they tend to ignore retaining the capabilities in the pruned part. In this work, re-ex re-examine structured pruning paradigms and uncover several key limitations: 1) notable performance degradation due to direct layer removal, 2) incompetent linear weight layer aggregation, and 3) the lack of effective post-training recovery mech- propose CoMe, including progressive
anisms. To address these limitations, layer pruning framework with Concatenation--ased Merging technology and a hierarchical distillation post-training process. Specifically, we introduce channel sensitivity metric that utilizes activation intensity and weight norms for fine-grained channel selection. Subsequently, employ concatenation-basse layer merging method to fuse the most critical channels across adjacent layers, enabling progres- sive model size reduction. Finally, we propose hierarchical distillation protocol that leverages the correspondences between the original and pruned model layers established during pruning. thereby enabling efficient knowledge transfer. Experi- ments on seven benchmarks show that CoMe achieves state-of-the-ane performance; when pruning 30% of LLaMA-2-7b's parameters, the pruned model retains 83% of its original average accuracy
Introduction
Large Language Models (LLMs) [ 41 9 have become the cornerstone of modern natural language processing, enabling start-of-the-un performance in tasks such text generation [20] [2]
21 32), machine translation (45), question answering [35 13), and a variety of other challenging Their success is primarily attributed scaling up model parameters, which enhances their representational capacity [13] However, the rapid growth model size comes at a
tasks 29
30 38
cost: LLMs' computational and storage demands have become significant obstacle for practical deployment, especially in resource-constrainee environments.
Recent works resort model compression [4 39 to reduce the resource footprint of LLMs, with
mainstream approaches encompassing model pruning [24] 34 27 [6] and knowledge distillation [I 5). Among these, structured layer pruning desirable for its hardware efficiency, it removes entire modules and reduces computational complexity [24] 34 [6). While direct layer pruning reduces model size, often leads to performance degradation. To mitigate this, several studies have
proposed merging adjacent layers through linear aggregation of their weights (27
22 These
"Correspondigg author.
"Our code is available at https:/giihab.cco///PAAAACCCC
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

approaches aim to better preserve model capacity by combining information from multiple layers, yet important questions remain regarding their effectiveness and the underlying assumptions.
Despite the efficiency of layer pruning, we rethink and identify several critical limitations in existing approaches. Direct layer pruning assumes that specific layers are redundant and can be removed without harming model performance. However, our analysis reveals that different methods yield highly inconsistent rankings of layer importance, and pruning different layers leads to performance degradation on different benchmarks. This suggests that each layer contributes meaningfully to the model, and preserving the mapping functions of pruned layers essential. In addition, linear weight aggregation methods rely on the assumption of distributional similarity among layer weights, which does not hold for ferd-fo networks; as result, linear aggregation can further exacerbate information loss. Finally, most current pruning methods lack integrated post-training recovery schemes. While some studies combine pruning and knowledge distillation, these stages are typically handled separately, resulting in suboptimal knowledge transfer and limited performance recovery.
Motivated by these insights, we propose CoMe, a structured compression framework for LLMs that integrates progressive layer pruning with a Concatenation- bbased Merging technique and a hierarchical distillation protocol. Our approach consists of three key components. First, we introduce channel- level sensitivity metric that quantifies the importance of each channel according to activation response intensity and weight norms, providing a principled basis for fne-grained pruning. Second, we present progressive merging method that concatenates important channels from adjacent layers, reconstructing compact fusion layers and minimizing information loss. Iteratively applying this strategy yields a pruned model with significantly reduced complexity. Third, we propose a hierarchical distillation protocol that utilizes layer correspondences established during pruning to guide efficient knowledge transfer via decoupled layer-wise feature alignment. This protocol accelerates post- depicts the the overall procedure.
training and reduces computational overhead. Fig. [
We evaluate CoME on seven LLMs, three sparsity levels (10% 20%, 30%), seven NLP benchmarks, and two datasets, comparing against nine competitive baselines. Experimental results show that CoMe consistently outperforms existing methods: (1) During pruning, it achieves superior performance across all settings; (2) After post-training, it surpasses ofate-of-the-art art by over 2.4% in average accuracy; (3) Compared to linear weight aggregation methods, concatenation- based merging improves average accuracy by more than 2% and reduces perplexity by over 4.7 across benchmarks.
In summary, our main contributions are as follows:
We introduce CoMe, a novel framework for structured layer pruning and recovery in LLMs, which preserves critical channels and enables efficient post-training restoration.
We propose concatenation-base merging method that reconstructs layers using the most informative channels, which minimizes performance loss caused by pruning.
We develop a hierarchical distillation protocol that exploits pruning priors to efficiently transfer knowledge through decoupled layer-wise feature alignment.
We conduct extensive experiments, reveal the inherent limitations of linear weight aggre- gation in layer pruning, and demonstrate the effectiveness of our approach across multiple LLMs and benchmarks.
2
Related Work
Structured Layer Pruning. L.MS are typically constructed by stacking multiple transformer layers, and substantial research has been devoted to compressing these models via layer pruning, Existing methods can be broadly grouped into three categories. First, heuristic layer pruning approaches, such as Magnitude 16) and Taylor [23 16). evaluate the layer importance based on weight norm the error induced by removing specific weights. While straightforward, these methods often fail to capture the complex, nonlinear dependencies across layers. Second, redundancy- bbased pruning estimates block importance based on activation patterns. For example, Men et al. [24] identify redundant layers via cosine similarity between output features, and Song et al. (4] select sub-models by evaluating their performance on calibration set. Although these methods incorporate global performance metrics, their discrete search spaces make it challenging to preserve the essential feature encoding capacities of pruned layers. Third, layer fusion methods aim to improve parameter utilization by
2

---

Step Channel Importance BI/SBI Score
Step Concatenation- Layer Merge
Layer!
Layer m
Lagar!
Bliam
New Layer
M1M
Channel importance
Importance
Ratio margel o.. Blum)
Layer! -
Layer!
(((-
10-1
Layari
SBI Score
Layer
Layer
Optimize
Layer!
H(lm)
H
Laperi
oyar m1
Ohiginel Modd
Step Hierarchical Distillation Strategy
I. Lightweight Model
Figure I: Overview of CoMe. Steps and 2 are iteratively executed until the model achieves the target size. Following this, Step 3 utilizes efficient post-training through feature-level distillation to recover the performance. The resulting lightweight model incorporates several newly replaced layers.
reconstructing layers through linear aggregation of weights. Techniques such as MKA use manifold learning to guide layer fusion, while LaCo [42] quantifies parameter differences for better information retention. However, these approaches often overlook the position-sense Position-wiss wise Feed-Forward Networks (FFN), leading suboptimal performance on tasks requiring fine- graigga semantic understanding. Despite these advances, current pruning methods are limited by the difficulty of accurately quantifying parameter importance and the lack of effective mechanisms for preserving layer capacity, which restricts further improvements both compression efficiency and pruned model performance.
properties of
Knowledge Distillation. Knowledge distillation widely used transfer the capabilities of large models to smaller ones, emphasizing efficient and effective knowledge transfer. However, aligning features between layers and ensuring training efficiency remains significant challenge. Early works such as DistilBERT B3 uses layer-wise sampling to initialize student models, but this approach disrupts inter-layer knowledge flow and requires additional alignment losses during training. More recent methods, including MiniL.M [10] and DistiL.LM [17]. adopts global feature alignment, simplifying mapping but increasing computational costs due the involvement of the entire model. L.M--Sreamline [5] introduces dynamic layer replacement, yet its reliance on single-layer mappings limits model expressiveness. Overall, current distillation techniques face two main challenges: Insufficient use of structural priors from pruning, which leads to feature mismatching between the pruning and distillation stages, and the lack hierarchical knowledge transfer results in a trade-off' between resouree efficiency and performance recovery.
Our Contributions. To overcome these challenges, we introduce channel sensitivity metric for fine-grained assessment of parameter importance, reducing discretization issues in existing pruning strategies. We also propose progressive channel concatenation strategy as principled alternative to linear aggregation, which mitigates over- over-smoothing in layer fusion. For knowledge distillation, our hierarchical protocol leverages layer correspondences established during pruning to facilitate efficient and continuous multi-level feature alignment between the original and pruned models. This framework enhances the efficiency of knowledge transfer in compressed LLMs.
3 Rethinking the Layer-based Structured Pruning
Based on how parameters are manipulated, layer pruning methods can be divided into two main paradigms: Direct Layer Pruning (DIP) and Weighted Sum-based Layer Pruning (WSLP). DIP is coarse-grained approach that removes entire layers based predefined importance metrics [34 24

---

16]. In contrast, WSLP reduces the number of layers by linearly combining multiple layers into one
layer [22 27 42 This section analyzes the limitations of both paradigms from the perspectives of layer importance, pruning performance, and the effectiveness of linear aggregation and motivates the design principles behind CoMe.
Core Issue 1: Are any layers in LLMs truly "redundant"? DLP methods typically rely on importance metrics, such as weight norms or cosine distance, to identify and remove suppos- edly redundant layers [24 [ They implic- itly assume that some layers make negligible contributions to overall model performance. To investigate this assumption, we compare layer importance scores from various DLPP methods on the LLaMA-2-77 model (Fig. 2) The dis- tributions of importance scores differ markedly across methods, with only a few shallow or deep layers consistently identified as highly impor- tant. In some cases, layer importance correlates with depth, while in others, it does not. This inconsistency indicates that the notion of "redundant" layers depends highly on the metric. Further, DLP methods yield varied performance across benchmarks (please refer to Tab. 8 in the appendix.). For example, pruning 10% of parameters with ShonGPT 24 maintains performance on WinoG and MMLU, but leads significant degradation others across all benchmarks and sparsity levels. These findings suggest that pruning different layers impairs different capabilities. Removing entire layers risks eliminating intermediate feature mappings that are critical for downstream tasks, which can result in notable performance loss. Therefore, preserving the mapping capacity of pruned layers is crucial for mitigating such degradation.
Vage
SLEB
Coffe/ / ShortSPT (8)
1.2
Taylor-
RusedPT (M0
0.8
0.6
0.4
0.2
0.0
Endex
Figure 2: Comparative analysis of normalized layer importance score across different methods. The score of SLEB and FuseGPT is derived from the first-round pruning.
other tasks. No method consistently outperforms
Core Issue 2: Does linear weight aggregation preserve hierarchical knowledge and mapping capability? WSLP methods merge adjacent lay- ers by linearly aggregating their weights, under the assumption that channel-wise alignment ex- ists not only in Multi- Head Attention (MHA) and normalization modules, but also in Feed- Forward Networks (FFN) 2X 2 However, while residual connections in Transformers do facilitate channel aligmment in MHA and nor- malization, this property does not extend to FFN modules, which lack explicit feature correspon- dences across layers [23]. As result, linear aggregation in FFN modules can lead to over- smoothing weights and disrupt layer-specific knowledge preservation. To empirically exam- ine this issue, we conduct fusion experiments by applying linear aggregation to adjacent layers under various fusion ratios (Figs. andd 13). We evaluate the resulting models using perplexity on multiple datasets, considering three module types: FFN, MHA, and the complete Transformer layer. Across all settings, models obtained via linear aggregation consistently underperform those produced by directly pruning layers ("L20"), as indicated by higher perplexity scores. These results confirm that weight distributions in adjacent layers are not sufficiently aligned and that WSLP fails to maintain the original network's hierarchical knowledge and mapping capacity.
11c
WSLP
WSLP-FFN
100
WELP-MH
Perplesity VMMG
07355
2894
8.25
8404
8.d77
dadd
61dp
u8
L19
(20
Figure 3: Merge adjacent layers with linear weight aggregation. First, we reduce LLaMA-2-7b's layer number to 23 with ShortGPT. Second, the compo- nents in layers 19 and 20 are merged at different ratios. When merging MHA or FFN, the other component retained from layer 20.
In summary, DLP and WSLP have fundamental limitations: DLP risks discarding meaningful intermediate representations, while WSLP introduces over-smoo and fails to preserve essential knowledge structures. These observations highlight the need for alternative pruning strategies that better retain LLMs' representational and mapping capabilitics.

---

W
W
W
Wy Wylnegd
Charnd Impostarce
Chareet begostane
PAOM +337-033
2
Figure 4: Channel importance calculation pro- cess. The importance scores reflect the expected changes of output caused by different channels.
Figure 5: Concatenation-basel weight merge process. W(O) and W(++1) are derived from the same positions in adjacent layers.
4
Methodology
This section details the technical innovations of CoMe. During the iterative layer pruning phase, motivated by insights discussed in Section 3 aim to minimize performance degradation by preserving the most critical parameters from pruned layers. We introduce a Channel Sensitivity Metric that quantifies the influence of individual weight channels the module's s output, thereby faciliitaiig the identification of the most essential channels within layers. Then, we extend the Block Influence (BI) score [24) to the cross-layer setting by introducing the Skip-BI BI (SBI) score, which enables us to quantify the extent to which group of adjacent layers alters intermediate features. Based on SBI score, we identify groups of layers that minimally affect feature transformations and implement our concatenation-base merging strategy with the channel importance information. The concatenation-hist -base merging strategy retains the essential channels, thereby mitigating the propagation of disruptive changes from shallow to deeper layers. By iteratively applying the above process, multiple layers are fused into a single layer, thereby progressively pruning the model. post-training stage, we further exploit the layer correspondences established during the pruning phase to fine-tune the merged layers in the pruned model. We employ two distinct knowledge distillation strategies, enabling the merged layers to recover the mapping capacities of the original layer groups more effectively. The complete CoMe pipeline is depicted in Fig. [ and the algorithm framework is shown in Algs. and
4.1 Channel Sensitivity Metric
To enable concatenation baase layer merging, we first establish a channel importance metric that quantifies the impact of pruming individual neural pathways. Consider linear mapping characterized by weight matrix W Ru' The input is vector X = [x1, ........ Iul, resulting in an output We express the linear transformation as O W X. As shown in Fig. 4
O [01, ..,..., 0g). pruning the weights W at the channel level alters the output O. We formulate channel sensitivity through output perturbation analysis. Pruning the i-th weight channel (achieved by setting x; = 0) induces perturbation ( O O' W|: .r,, As shown in Fig. we compute the expected
& fi-normo of this perturbation across the calibration dataset D as the quantizer of channel importance:
8; = ED |100 = E Σ bel
where the expectation over the calibration dataset D decouples input statistics from static weights. A larger score indicates that the weight channel more significant, and discarding would cause more substantial damage to the block's functionality.
4.2 Progressive Concatenation--asse Layer Merge
In existing work, evaluating the importance of blocks involves analyzing model weights, activations, gradients, and differences input and output. Men et (24] posit that the cosine distance between the hidden features of block's 's input and output measures the block's redundancy. Consequently, Men al. (24 define the Block Influence (BI) score for the /-th layer as:
H (
Bī: = - Ep
(2)
H
H
2
5

---

where H I represents the output hidden features of the 1 layer in the model. S is the sequence length, and d the hidden dimension. D the calibration dataset used.
The objective CoMe is to replace multiple original blocks with fused blocks while minimizing performance loss. Therefore, extend the BI concept layer groups as novel Skip-Block Influence (SBI) metric. For the module containing layers / to / + m, its SBI score is defined as:
H
((m))
SBlu+m
Ep
H
where + 2 indicates the number of layers be merged. A smaller SBI score indicates that the block group spanning multiple blocks induces more minor perturbations to the hidden features, implying that manipulating these groups will have minor impact on model performance. We factorize block weights into channel-level components, treating channels as atomic units and concatenating the parameters sequentially to increase the importance of each channel. The concatenation- -based merge (Fig. 5 implements parameter preservation through:
1+m
W (merge)
( Tk),
where Ta denotes the top-k channels selected via our sensitivity metric, and @ indicates column-wi -wise concatenation. It ensures the retention of the most impactful parameters in merged layers. For the parameter preservation ratio, we employ heuristic approach. The parameter preservation ratio for layer/ Ž in group {l..../ + m) follows:
BIP ΣE E p>
F1
(5)
where controls the distribution skewness, and larger p value emphasizes the preservation of layers with high BI scores. Through constrained concatenation where 2n; = I, the merge module keeps the hidden state dimension the size of the origin module. We provide the calculation details for channel importance and the merge rules for different structures Section C
4.3 Post-Traie Dost-Training via Hierarchical Distillation Strategy
After a progressive merging process, aim to replace the original layers with the fused ones to alleviate representational gaps induced by pruning. Thus, each fused layer should maintain equivalent feature representation capabilities to its original group, enhancing model performance. We align their feature representations through post-training. Our progressive layer merging process creates direct mapping between pruned and original layers, formalized as: P =[K1,, b1):-... ..... {aN, by) }}, where is a layer index from origin model, b is from the pruned model and N is the number of merged layers. provides priori conditions for efficient beasere-based post-training.
We use the original model as the teacher model and the pruned lightweight model as the student model, transferring knowledge from the teacher model to the student model. We employ feature-level distillation using symmetric Kullback-Leibe divergence (KL) to mitigate distribution shift:
CKL
ED [DLL (o(H(t)a) || [ ((()))))
(6)
where (t.a) denotes the output features of the a-th layer of the teacher model, and (s.6)) denotes the output features of the b-th layer of the student model. The element pair {a,b) E P. denotes the softmax activation function. Dtrain is the training dataset. The KL-divergence term expands as:
Derain
P. Q
D ( (1
Σ
P. Po
= (H())
(7)
The optimization process requires iterating through all mapping pairs P from shallow to deep layers. post-training (CoMe-mp). While CoMe-mp updates one layer
This process is named per iteration and uses minimal resources, its sequential nature prevents joint optimization of shallow and deep layers. To address this, we consolidate multiple training processes into one, iterating over

---

ELPAlgg DUT-Tglar
EUISUCE
WELPARA
Aeoeage Store plgg
LLMM-2116
Morrs-h
410%
10%
40%
10%
LLaMA-2.7b
Danns 976
Figure 6: Comparison of different layer-wise pruning, including CoMe (red), DLP (blue series), and WSLP (orange series), on various models with 10%, 20%, and 30% sparsity.
all mapping relationships in simultancously, and name it single-process post-training (CoMe-sp). It minimizes the KL divergence across multiple mappings:
SKL-S
E DKI. (o(H))))) (H((b))
(8)
P
[a,b]EP
CoMe-sp enables the joint optimization of multiple fusion layers using global information, which helps capture inter-layer dependencies and allows for more coordinated parameter updates. Al- though CoMe-sp requires more storage for optimization parameters, increasing by factor of more than IP compared to CoMe-mp, it often achieves more effective training and better overall model performance due its holistic optimization strategy.
5
Experiments
5.1 Experimental Setting
Models, Datasets, and Metries. We evaluate our method on several widely used open-source models, including LLLMA-2-7b,, LLaMA-2-131
13b 377, and LLaMA-3-8h [9] from the LLaMA series, Qwen- 2.5-7b 41, Qwen-3-4b, Mistral-7b 14, and Vicuna-7b 44 Model performance is assessed using
the in-evvllatioon-haree 8 framework seven standard benchmarks commonly adopted in model compression research: ARC-challenge (ARC-c), ARC-casy (ARC-e) [6]. HellaSwag (HellaS) [43], QA (OBQA) (26), PIQA 4.. Winoground (WinoG) (36 under the zero-shot setting, and MMLU M under the five-shot setting. Accuracy is reported with normalized option lengths to ensure comparability. We report the average accuracy (Avg) across all datasets and the Retained Performance (RP), defined as the percentage of the original model's i's accuracy preserved after pruning. Perplexity (PPL) is measured the C4 (28) and Wikitext-2 (Wiki-2) [25] datasets.
Baselines. Eight ofate-of-the-art layer pruning methods are selected as baselines. DDa-based methods include Magnitude+ (Mag+) 19 16], Taylor+ (23] 16. ShortGPT (24. SLEB [34 and
FUseGPT-MI (27). WSLP-based methods include FuseGPT-MI-F (27), MKA [22], and LaCo 42
For fair comparison, unless otherwise noted, the post-training phase is excluded from FuseGPT. FuseGPT-MI refers to pruning with Macro Influence (MI) without fusion, while FuseGPT-MI-F includes layer fusion. Tables distinguish between DLP and WSLP methods with horizontal lines. For post-training comparisons,
(3] and FuseGPT with training are also evaluated. Detailed descriptions of all baseline methods are provided in Section
Implementation. All methods are evaluated under sparsity levels of 10%, 20%, and 30%. Tab. summarizes model configurations for each sparsity setting, while Tab. 4 lists the pruned layer sequences for all DLP methods. Unless otherwise specified, CoMe is conducted on --aMM-2-7b using the Wiki-2 calibration set (256 samples) with a default sparsity of 30%. During the layer
7

---

Table 1: The Post-training Experiment on the LLaMA-2-7b and Qwen3-4b.
Method
ARC-e
ARC-e
HellaS
BenchmarkT OBQA
PIQA
WineCi
(S)
Avgt
RP1
PPL.
Wiki-
LLaMA-2-7b Denee
46.33
74.54
75.99
44,20
79.05
69.06
45.60
62.11
100.00
7.27
5.47
FuseGPT-MI
30,20
50.59
52.98
33.60
69.37
54.54
25.17
45:21
71.53
17.60
14,94
FusGPPTMMI+F
30
50.13
55,08
31.80
68.50
55.11
27.01
15.88
72.82
17.80
14.34
(110))
LLM-stramnline
31.60
76.53
CoMe
35.24
54.46
56.56
35.40
68.88
61.17
25.50
48.17
76.47
19.93
16.53
FuseGPT-MI
30.55
56.52
55.42
36.6
71.11
54,46
24.95
47.09
74,43
13.31
9.74
FusGPPT-MM+F
31.23
57.41
56.26
35.6
71.22
55.8
26.72
47.75
75.50
12.96
8.85
Post-training
M-strcandime
33,60
81.92
CaMe-np
35.24
60.65
61.14
37,80
70.57
64.56
25.35
76x77 6n
80.25
13.01
9.72
CoMespp
35.58
63.51
65.83
39.20
74,05
63,38
26.18
52.5% (448)
82,98
11.45
8.54
Qwen3-4b Dense
$1.54
76.43
73.70
41.20
77,80
71.03
73.01
66.39
100,00
13.31
7,90
Prune
LL -treamlaeee
26,71
40.57
38.31
28.20
61.81
$3.91
23.54
39.01
58.99
200.93
228.89
CoMe
28.67
47.10
43.95
29.60
63.00
51.46
32.67
42:35
63.84
56.13
37.14
LLM streamline
30.40
41.96
w/ post training
CaMe-mp mmp
30.97
51.98
47.02
30.80
65.23
51.51
36.41
45. (5.28
68.17
32.02
22.09
CaMe sp
31,40
$3.91
48.33
33,80
65.23
56.75
41.10
47.22
71.30
29.28
20.25
pruning process, we set the number of layers merged per iteration to 2 (i.e., = 1). Comprchensive experimental settings and implementation details are available in Section B
5.2 Main Result
CoMe Outperforms DLP and WSLP Across Benchmarks. Fig. summarizes the average accuracy of various layer pruning methods. In most settings, CoMe consistently ranks first. When pruning 30% of the parameters from LLaMA-3-8b, CoMe achieves the recond-highen average accuracy. When pruming Qwen-3-4b, CoMe achieves comparable performance compared to the best method. These results indicate that CoMe effectively preserves model capability and enhances the performance of pruned models. Notably, the performance of CoMe declines approximately linearly as sparsity increases, suggesting that important parameters are selectively retained even at high sparsity levels. On the larger LLaMA-2-113 model with 30% sparsity, CoMe surpasses all other methods by at least 4% in average accuracy, highlighting its particular advantage for large-scale models.
CoMe-sp Surpasses Existing Trst-iraining Baser- Methods. Tab. presents the effective- ness of various post-training strategies in restoring performance after layer pruning. CoMe achieves the highest average accuracy after pruning, outperforming L.M-Streamline by 1% LLaMA-2-7b and by 2.3% on Qwen-3-4b and exceeding FuseGPT-MI++ by 2%. Its PPL is also comparable to the best-performing method, After post-training the relative ranking of methods remains consistent, with CoMe-sp achieving the top performance, surpassing LLM-Strammline by 2.4% on LLaMA- 2-7b, and by 5.2% on Qwen-3-4b, while also achieving lower PPL. These results indicate that the hierarchical distillation strategy in CoMe-sp accelerates performance recovery, even with limited training data (Tab. [7). In contrast, CoMe-mp, which relies only on local layer feature alignment, performs similarly to LLMMStreamling indicating that relying solely on local layer feature alignment is insufficient for restoring model performance effectively.
5.3 Ablation Study
Ablation on Concatenated Structure. The results shown in Figs. and indicate that using the CoMe to merge the MHA module, FFN module, or entire layer structure of adjacent layers can achieve better PPL metrics at specific fusion ratios compared to simply retaining layer 20. It demonstrates that the concatenation-base layer merging strategy effectively preserves the model's 's language model- ing capability. Notably, merging the entire layer structure outperforms merging the MHA or FFN modules individu- ally. It indicates that the collaborative fusion of MHA and FFN modules can produce a complementary effect on the parameters. The results validate the effectiveness of the fusion ratio allocation strategy.
35.0
Colle Cille-TT
02:0
Galde-dii
45
880
1000
2000
Figure 11: Merge layers with CoMe.
The setting is the same as Fig
8

---

NVG. Wik2
04
14
48.0
2.8
WIKI C4 PP flog)
47-
2.5
AVG. Acc.
47.5
222
2.0g
1.8
43-
1.5
1.2
9
10
11
Hyperparameter P
Layers per Merge
Figure 7: Effect of in heuristic merge ratio. As p grows, more parameters from layers with higher BI scores are merged.
Impact of merge step granularity. Merging more layers degrades performance due to parameter distribution differences.
Figure 8:
MAG.
W
50
48.0
WAKI Cd PPL (log)
1.5
40
47.5
Wiki
2.5
C4
30
AVG
20
1.5
46.5
10
46.0
1910
128 256
512
Wi6-2
C4
PG11
MMLU
Alpaca
0.0
concat
Caibration Detaset
Callbratian Dataset Scale
Figure 9: Impact of the calibration dataset. We use two samples PG19 and 256 in the others.
Figure 10: Effect of calibration data scale. CoMe achieves optimal performance with sam- ples more than 128.
Effectiveness of Concatenation-Basee Merge. As shown in Figs. I1 and 14 merging adjacent layers using the concatenation-be based strategy CoMe yields lower PPL than simply retaining layer 20, demonstrating superior language modeling capability. Merging the entire layer structure consistently outperforms merging only the MHA or FFN modules, suggesting that the joint fusion of MHA and FFN produces complementary effect. However, the optimal fusion ratio varies across the three types structures, suggesting that the heuristic fusion ratio allocation strategy has limitations and may not fully unlock the potential of CoMe.
Impact of p in Heuristic Merge Ratio. Fig. iillustrates the impact of p on the performance of models pruned using CoMe. For p < 2, the model maintains optimal performance after pruning, with average accuracy exceeding 48%. As increases, retention becomes more skewed toward layers with higher BI scores, leading to performance degradation. When p inf (i.e., without merging), performance drops sharply (PPL rises to 25.50/ 21.21 21.21). These findings confirm that: (1) the channel sensitivity-basee based selection mechanism effectively identifies key parameters; (2) progressive layer merging is essential for maintaining performance; and (3) BI scores are positively correlated with the number of essential parameters, validating the skewness control strategy.
Effect of Merge Step Granularity. The analysis in Fig. reveals that merging more than two layers at time leads to a drop in average accuracy of over 3% and a substantial increase in PPL. This suggests that merging multiple disparate layers increases parameter distribution discrepancies. The progressive merging of fewer layers helps mitigate this issue and preserves model performance better.
Robustness Calibration Dataset. Cross-dataset experiments in Fig. demonstrate that CoMe is robust to the choice of calibration dataset. When using pre-training style data, such Wiki-2, C4, the model achieves low PPL and strong language modeling performance. In contrast, using QA datasets such as MMLU, which differ significantly from the pre- praiirain dataset, increases PPL by over 25%. However, the impact on downstream task average accuracy remains limited (fluctuation 1. 1.5%), indicating that the channel importance measurement mechanism effectively decouples input distribution from static weight features. These suggest that calibration data similar to the training distribution is preferable.

---

Table Comparison of CoMe and WSLP, w/ and w/o the merge process. For the methods w/o Add,
we retain the layers deemed most important by the method (LaCo w/o Add retains shallow layers).
ARC-c
ARC=
HellaS
Beachmack 6o CRQA
PIQA
Winof
MMLLU (5)
Aughui
RP".:
PPL
MKA
34.04
49.58
48.12
35,00
63.00
99.12
35.64
46 36
wio Add
33.9
49-45-049
48.02:4.0
34.80-120
62.79
59.08-0ue
35.64.1
28.41
74.95.0.09
808,74.0.00
454.70
Lacaa
10:37
43.79
SOTT4
35.00
68.34
5391
24.84
44.71
71.11
50.18
42.07
wio Add
4m
50.38:009
50.90.009
34.60-144
69.0
55.0
24.7
45 55:0
71.53-04
25.51
28:18:2140
FuseuPT
70.20
50113
35.08
34,80
08.30
25.11
27,01
45.88
72.82
17.30
14.34
win Add
30.20
50.59,001
52.98.3.0
33,60
69.37
54.54,000
45 45.21,007
71.53
17,60,090
14 14.04
$5.54
44.40
35,40
68388
25.00
1993
Coscat
48.00.600
68.12,470
26.710133
72944,59
Calibration Data Scale. As shown in Fig. 10 channel importance stabilizes when calibration samples exceed 128 (PPL fluctuation WikiText-2 is less than 1.3). When fewer than eight samples are used, parameter merging degenerates into random selection, with PPL exceeding that of the no-fusion strategy (p inf in Fig. 7 This highlights the necessity of accurate channel importance evaluation, as incorrect estimates can impair pruning performance.
5.4 Weight Sum-Based Merge vs. Concatenation -Based Merge
Both WSLP and CoMe aim to mitigate the loss of layer mapping functionality caused by DLP. Tab. compares these approaches without the merging process. For WSLP, removing the merge step changes average accuracy by less than 0.7% and RP by less than 1.3%, indicating a negligible effect. In contrast, removing the concatenation-bat average accuracy and RP (both decrease by more than 2%) and an increase in PPL (by more than 4.7), demonstrating that the concatenation- based merge plays a critical role in preserving model performance during pruning. Further analysis is provided in Section
merge CoMe leads to a significant drop in
6
Conclusion
This paper addresses the challenge of layer pruning in LLMs, focusing on preserving model per- formance while reducing computational complexity. Our proposed framework, CoMe, introduces three key innovations: channel sensitivity metric quantify the importance according to acti- merging strategy to retain critical information during
vation and weight, a concatenation-base pruning effectively, and a hierarchical distillation protocol for efficient post-training recovery. Exten- sive experiments across multiple models, sparsity levels, and benchmarks demonstrate that CoMe achieves superior performance compared existing pruning approaches in maintaining accuracy after compression.
Limitations
CoMe adopts a heuristic and uniform parameter preservation ratio for merging all Transformer components, which limits its adaptability to different architectures. In Section [ we present posterior-based solution for adaptively merging two adjacent layers to alleviate the limitations of CoMe. In future work, we will explore adaptive methods for merging multiple layers once and extend CoMe to expert merge.
Acknowledgements
This work was supported by the Guangdong Provincial Key Field R&D Program Project (2024B0101040004) the National Natural Science Foundation of China under Grants 62476099, 62076101, and 62576364, the Guangdong Basic and Applied Basic Research Foundation under Grants 2024B151 15 and 2023A1515010007, the Guangdong Provincial Key Laboratory of Human Digital Twin under Grant 2022B 1201000, and the TCL Young Scholars Program.
10

---

References
[1] Josh Achiam, Steven Adler. Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint ariiv::3033007770 08774, 2023.
[2] Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations, 2023.
[3] Jinze Bai, Shuai Bai, Yumfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Gc, Yu Han, Fei Huang. al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[4] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqu: Reasoning about physical common- sense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
2020.
[5] Xiaodong Chen, Yuxuan Hu, Jing Zhang, Yanling Wang. Cuiping Li, and Hong Chen. Streamlining redundant layers to compress large language models. In The Thirteenth International Conference on Learning Representations, 2025.
[6] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803. 1803. 000557, 2018.
[7] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv: 1902.09544 2019.
[8] Leo Gao. Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac h. Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, I Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024.
[9] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint 247.:2247..2188., 2024.
[10] Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2024.
[11] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song. and Jacob Steinhardt. Measuring massive multitask language understanding, In International Conference on Learning Representations, 2021. URL httpps:/
[12] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL https
/oppnreviee wtwp:e/wwwneeetotooiidiiddededee hett//forurridrryyyyyrttt
[13] Zhiwei Huang, Juan Li, Long Jin, Junjic Wang, Mingehen Tu, Yin Hua, Zhiqiang Liu, Jiawei Meng, and Wen Zhang. Reliable academic conference question answering: A study based on large language model. In China Conference on Knowledge Graph and Semantic Computing, pages 181-193. Springer, 2024.
[14] Fengqing Jiang, Zhangehen Xu, Luyao Niu, Boxin Wang, Jinyuan Jia, Bo Li, and Radha Poovendran. Identifying and mitigating vulnerabílitics in lmm-integrated applications. In NeurlPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023.
[15] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001. 083611. 2020.
[16] Bo-Kyeong Kim, Geonmin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung- Kyu Song. Shortened LLaMA: A simple depth pruning for large language models. In ICLR 2024 Workshop Mathematical and Empirical Understonding of Foundation Models, 2024.
[17] Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se- Young Yun. DistiLLM: Towards streamlined distillation for large language models. In Forty-first International Conference on Machine Learning, 2024.
[18] Jaeho Lee, Sejun Park. Sangwoo Mo. Sungsoo Ahn, and Jinwoo Shin. magnitude-base based pruning. In International Conference Learning Representations, 2021.
sparsity for the
11

---

[19] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations. 2017.
[20] Junyi Li, Tinyyi Tang, Wayne Xin Zhao, Jian- Yun Nie, and Ji-Rong Wen. Pre-trained language models for text generation: A survey. ACM Computing Surveys, 56(9): 39, 2024.
[21] Xun Liang, Hanyu Wang, Yezhaohui Wang, Shichao Song, Jiawei Yang. Simin Niu, Jie Hu, Dan Liu, Shunyu Yao, Feiyu Xiong, al. Controllable text generation for large language models: A survey. arXiv preprint arXiv:2408 / 1299,, 2024.
[22] Deyuan Liu, Zhanyue Qin, Hairu Wang, Zhao Yang. Zecheng Wang, Fangying Rong, Qingbin Liu, Yanchao Hao, Bo Li, Xi Chen, Cunhang Fan, Zhao Lv, Dianhui Chu, Zhiying Tu, and Dianbo Sui. Pruning via merging: Compressing LLMs via manifold alignment based layer merging. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17817-17829. Association for Computational Linguisties, 2024.
[23] Xinyin Ma. Gongfan Fang, and Xinchao Wang. Lim- Lim-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702-211111 2023.
[24] Xin Men. Mingyu Xu, Qingyu Zhang. Bingning Wang, Hongyu Lin. Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853, 2024.
[25] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. URL https://openreeraaii net/forum?
id=By]72udxe 72udxe
[26] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages
2018
[27] Zehua Pei, Hui-Ling Zhen, Xianzhi Yu, Sinno Jialin Pan, Mingxuan Yuan, and Bei Yu. Fusegpt: Learnable layers fusion of generative pre-trained transformers. arXiv preprint arXiv:241.114007, 2024.
[28] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21((40))1-67, 2020.
[29] Jun Rao, Fei Wang, Liang Ding, Shuhan Qi, Yibing Zhan, Weifeng Liu, and Dacheng Tao. Where does the performance improvement come from? -a reproducibility concern about image-text retrieval. In Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval, pages 2727-2737, 2022.
[30] Jun Rao, Liang Ding, Shuhan Qi, Meng Fang, Yang Liu, Li Shen, and Dacheng Tao. Dynamic contrastive distillation for image-text retrieval. IEEE Transactions on Multimedia, pages 1-13, 2023. doi: 109/ 223.333377....
[31] Jun Rao, Xv Meng. Liang Ding, Shuhan Qi, Xuebo Liu, Min Zhang, and Dacheng Tao. Parameter-efficient and student-friende knowledge distillation. IEEE Transactions Multimedia, 26:4230-4222 42411 2023.
[32] Jun Rao, Xuebo Liu, Lian Lian, Shengjun Cheng, Yunjie Liao, and Min Zhang. CommonIT: Commonality- aware instruction tuning for large language models via data partitions. In Yaser AI- Onaizaan Mohit Bansal, and Yun-Nung Chen, editors, EMNLP, pages 10064- 10083, Miami, Florida, USA, 2024, Association for Computational Linguistics. doi: 10. 10.18653/v1/2224.com 1tt.3//111222..mmp--mm m61n mat.444.eemmmmmmi.5........ URL https: httpp:///alanthhooogy anthology org/ 2024.002.ooo emalp-main.56i
[33] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version hert: smaller. faster, cheaper and lighter. arXiv preprint 108.v:1910.01108, 2019.
[34] Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, and Jae-Joon Kim. Sleb: streamlin- ing iims through redundancy verification and elimination of transformer blocks. In Proceedings of the 41st International Conference on Machine Learning, pages 46136 46155, 2024.
[35] Mingxu Tao, Dongyan Zhao, and Yansong Feng. discussion: A multi model framework for complex evidence- based question answering. In Proceedings of the 31st International Conference on Computational Linguistics, pages 11070- 11000, 2025.
12

---

[36] Tristan Thrush. Ryan Jiang. Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionnllty. In Pru- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5238-5248.
2022.
[37] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava. Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0999
2023.
[38] Fei Wang, Liang Ding, Jun Rao, Ye Liu, Li Shen, and Changxing Ding. Can linguistic knowledge improve multimodal alignment in vision language pretraining? ACM Transactions on Multimedia Computing. Communications and Applications, 20(12)1-22 22, 2024.
[39] Wenxiao Wang, Wei Chen, Yicong Luo, Yongliu Long, Zhengkai Lin, Liye Zhang, Binbin Lin, Deng Cai, and Xiaofci He. Model compression and efficient inference for large language models: A survey, arXiv preprint Xiv:22402. 09748. 2024.
[40] Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng. Shuxin Zheng. Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architccture. International conference on machine learning, pages 10524-10533. PMLR, 2020.
[41] Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Zheng, Bowen Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et Qwen2. technical report. arXiv preprint 15,,, 2024.
[42] Yifci Yang, Zouying Cao, and Hai Zhao. Laco: Large language model pruning via layer collapse. In Findings of the Association for Compitational Linguistics: EMNLP 2024, pages 6401-6417, 2024.
[43] Rowan Zellers, Holtzman, Yonatan Bisk. Ali Farhadi, Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800, 2019.
[44] Lianmin Zheng, Wci-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging Ilm-as-a-judge with mt-bench and chathot arena. Advances in Neural Information Processing Systems, 36:46595-46600 46623,, 2023.
[45] Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. Multilingual machine translation with large language models: Empirical results and analysis. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2765-2781, 2024.
[46] Xunyu 7hu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey model compression for large language models. Transactions of the Association for Computational Linguistics, 12:1556-1577, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, clearly demonstrate the contribution and scope this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
Our paper is empirical and does not include new theoretical results proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide comprehensive implementation and experimental details in Sec- stion B and 4 and Subsection 5.1 sufficient for reproduction.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification: Our code is included in the supplementary materials for this submission, and we will release it publicly after publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nips.cc/public/giue guwde//ddee/// Cote/ooodeeeeeseeeeeeee wwtpiaimsssinoooiooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All training and evaluation details, including hyperparameters and data splits, are specified in Secstion B and and Subsection 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to computational constraints, we did not report error bars or statistical significance; results are based on single runs with fixed random seed for all experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
16

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors Is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: We provide the type of GPU used (NVIDIA A100), but did not report the runtime or memory usage for different methods. This because the code implementations of the compared methods differ significantly, and made modifications to the model files, making it difficult to provide fair and accurate comparisons of runtime and memory consumption.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp::/neurips.. htt.://ps...pplbtttttccccccccaae
Answer: [Yes]
Justification:
Our research complies with the NeurTPS Code of Ethics in all respects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: We did not include broader impacts section because research is method- ological and does not have immediate or direct societal applications.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not introduce or release models or datasets with high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [No]
Justification: We properly cited all external code, datasets, and models used in our work and ensured our usage complies with their terms. However, we did not explicitly state the licenses or terms of use the paper.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, batasets.papersstt paperswi thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will make code publicly available after publication. The license and terms of use will be clearly stated in the ReadMe file accompanying the code
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [No]
Justification:
Our research does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification:
No human subjects were involved in our research.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: Our method primarily implemented on large language models (LLMs), and we conduct extensive experiments comparing different LLMs throughout the paper. All LLM usage is clearly described the main text.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy for what should or should be described.
20

---

Table 3: Model Parameters and Sparsity Settings Across Different Models.
Misel
Size
# Block.
=105 Speriity
& RRmssna
# Pansmtic
=000% Sparisity
RRenmeed
Phrancizn
v3375 Sputiblt
RRmoved
4 Permetees
11MMA-3-78
LL.MAN-211 13B
1.3
11:7b
10.36/3
19,01%
29
Miuuna7hh 7th
6.1b
21 1..02%
30,0395
10
Miveal-78
7.26
32
9,04%
6.66
21 31,08%6
5.76
30.128
10
5.1b
Uwen-125-bb
7:0
78
1896
6,6
21 42%
0.00
10.005
10
53t
Qwes-3-4b
4.00
36
10,04%
3,6b
20.07%
3.226
30.10%
12
2.3b
A The Details of Comparison Methods
This paper compares three types of structured pruning paradigms: (1) DLP, which includes Mag+
[6
Taylor+ [16], ShorGPT [24], and SLEB [4] (2) WSLP, which includes LaCo [22 and MKA 22
and (3) methods combining layer pruning with post-training, such as L.M-streamline
5 and
FuseGPT 277. We implemented each method using publicly available code.
Magnitude+ (Mag+), I Kim et al. [16] use this method as a baseline in the pruning method
comparison conducted. Initially proposed by Li al. [19], it assumes that weights with smaller norms contain less information. For block-level analysis, the importance of the 1-th layer is calculated as the sum of the first-order norms of all weight parameters: IMag+! Ewew() Ewew (w), where W represents the set of all weight matrices in the I-th layer. We follow popular heuristic algorithms 7 [8]. Kim al. [16] further mitigated performance degradation by retaining the first four and last two layers (23].
Taylor+. B This method is also baseline in the pruning method comparison by Kim et al. [16]. assumes that the error introduced by removing weight parameters indicates their importance. Given calibration dataset D, this error can be expressed a change in the training loss L:
(C(W;D) L(W D;D) BCDP. Following Kim al. [66] and Ma et al.
2 we de-
aW
fine the layer importance parameter as /Tayorr+ EwewO Eweu heuristic optimization methods to retain the first four and last two layers.
@C(D)
w/. We use similar
ShortGPT. Proposed by Men et al. [24] this method assumes redundancy in the model layers and defines redundancy as layers that minimally alter the hidden embeddings. To measure change, they use the cosine distance dataset for calibration, and we control the size of the calibration dataset 256 samples. The results are consistent with those reported by Men et al. [24]. Men et al. [24] compute the importance scores for models in the LLaMA family, and we directly use the layer importance order provided in the paper. When the number of pruned layers exceeds the required number, we append the least important layers from our calculated importance scores. For models where Men et al. [24) do not provide layer importance sorting, we estimate it using the settings in this paper.
metric, as shown in Eq. ( Men et al. (24) use the PG19 long-document
SLEB.
Song al. 34 propose using posterior method verify the redundancy of specific layers. SLEB uses the exponential part of the PPL score of the pruned model on a specified dataset as the redundancy score: ISLEB, = ExED K En= log (Ia 24 E X), where M' denotes the
smaller model obtained from previous pruning steps, M2 denotes the model obtained after pruning the 1-th layer, and = Ii, ..., Zú, ..., In represents a sample. ISLEB, is the exponential part of the PPL score, which positively correlates with the PPL score; hence, ISLEB positively correlates with the PPL score. The SLEB method a progressive structural search optimized for the PPL on the specified dataset.
LaCo. 0 Proposed by Yang et al. 422 this method uses the weight differences between layers as important information for layer retention. LaCo groups several adjacent layers and performs wwweeeernneessile-eeeeeeeeeeeeee Seeell--Seeiig--oooeee Common layer merge. For weight fusion from layer / l + m, the fused weight represented as W w + (W(++) w + ... + (W(++m) W W + Σ. (W((++) W()). It fuses the differences between deeper and shallow layers into the shallow layers. LaCo assesses the redundancy of pruned groups using the cosine similarity of output
http:://gthab..ooo/yyyyy/fff///
21

---

Table The layer importance ranking of different DLP methods.
Model
LLaMA-2-7B
1LaMA-2-13B
LL-MA-3-85
Sparisity
Mags
7.
6,
1
8.
4,
10,
9/
14, 13
4.
6
1/10,
8.
9.
13/
12,
11, 14, 16
5. 8.
11/
6.
107
9.
13,
12, 14
Taylor+
ShonGPT
27.
28,
231
22.122.230
30
33,
32
/9.
36,
37
25,
26.
23
29.
21.
SLED
14,
23,
11
24,
10,
25/
25,
33,
29,
12
26
31
14,
32
11
10,
25, 35
10,
26.
11,
12/
9,
19
22,
25.
PuseGPT-MI
19
Midel
ViammeTh
S-W6
MOPE / /300% 3005
90% 1281/2118//22
9.2%/21 (2.4%
10,0%
Mage
L1F
9. 10, 4/12 14,13
4.
5712,
T. 9. 9.10/11
13
14,4
15
7/12,
10
21, 20, 18/22.
22222.10
15.
23.
9.
Daleew
10.
21727,
2112
30
16,
15717
1399
4.
5,21/92
22 M
18 19
16
76. 25.
19
10
ShanGPT
27.
23
21.30
25.26,
24/27,
27.222
22
25/21,
30
15.
29. 26,
28,
25
20
10.
SLES
20.27,10
2X
PI
12.
13, (4/11,
2.27,
19.1777
17.17.
18/
H HeELs
features between the pruned and unpruned models: LaCo = N EXED TH lall where H H!!)
and (L')
M represent the output features of the last layer of the model. Due the threshold adjustment for cosine similarity in LaCo and the need adjust the starting and ending layers for pruning, as well as the number of layers in each group. the excessive parameter settings made it challenging to optimize performance for each model. Therefore, we implement this method only on models in the LLaMA-2 family.
MKA. Proposed by 22 this method uses manifold learning and the Normalized Pairwise In- formation Bottleneck (NPIB) measures to assess layer similarity and fusion. MKA progressively fuses deeper into shallower layers, merging the last two adjacent layers each time. In the code implementation, we find that MKA calculates the NPIB scores for two layers as approximately equal: INPIB. INPIB,141 0.5 0.5. An exponential mapping increases the fusion proportion of the Jnorm
shallower (-th layer: IMKA, = T-elnerm where Inorm Txezs. is the normalized NPIB score: [MKA1++ 1 IMKAI- After mapping, the similarity ratio between the two layers approaches MMKA,, AM1+1 RO 0.6 0.4.
INPIB,
FuseGPT. Proposed by et al. 27 FuseGPT hypothesizes that layer pruning causes performance loss and uses FFN parameter fusion to integrate layer capabilities into adjacent blocks, as Pei et al. (27 hypothesizes that FFN layers concentrate the main capabilities. Low-rank learnable weight matrices disperse the capabilities of pruned layers, optimizing multiple layers at once to reduce the gap caused by pruning. To better study the effectiveness of fusion, we remove parameter adjustment part of FuseGPT in pure pruning experiments, using randomly initialized low-rank matrix products to fuse weights. post-training comparison experiments, we use the complete FuseGPT method. Pei et al. [27 propose a Macro Influence (MI) score to measure the global-level impact of
HLAT mll' As'
removing model layer: JMI = 1 EXE [
=1 - ILaCo-
all wr
LLM LLLM-streamline. Proposed by Chen et al. (5). LLM-streamline uses SBI (Eq. 33 to measure redundancy of multiple consecutive layers, replacing these layers with the shallowest layer among them, and fine-tuning this shallowest layer post-training restore model performance.
B The Details of Experiment Setting
The settings for the experiment methods follow mainly those in the original papers. All experiments conducted using A100-40G GPU, We conducted pruning experiments on LLaMA-2-7b
ine
22

---

Table 5: Experimental setting for pruning methods. idenote methods whose hyperparameters adjusted to satisfy the sparsity ratio constraints in our implementation. Complete implementation details documented in Subsection B.1
Methods
Calibration
# data
seed
Mag+
Wiki2
128
10
Taylor+
Wiki2
128
10
ShonGGPT
PG19
256
10
SLEB
Wiki2
128
10
FuseGPT
Wiki2
32
10
MKAi
MMLU
50 subtask *
10
Mouron () is commune in the Arde
Torreorgaz is a municipality in the
LaCor
The 81st Mechanised Brigade () is mechanised hrigade of the Romanian I Land Force There are 18 National Natural Landmarks in the U.S. state of Washington, out of nearly Copa Libertadores 1973 was won by defending champions Independiente of A 256
CoMe
Wiki2
10
Table 6: The Hyyerr-aramend used in LaCo 42
is the number of layers to be merged during each merging optimization. Z is the minimum interval of layers between two merging operations. L and H. are the minimum and maximum indices of the range of layers for merging. is similarity threshold.
Sparisity
C
H
I
9.01%
1
32
2
0.85
LLaMA-2-7b
21.02%
1
32
2
0.65
30.03%
1
32
2
0,55
9.75%
40
2
0.85
LLaMA-2-13b
19.49%
2
0.70
29.24%
2
0.55
1LaMA-2-135 LLaMA-3-8) 3-8b and performed post-training experiments on LLaMA-22 A-2-b and Qwen-3-4b.
Vicuna-7b 13
Mistral-7b Qwen-2.5-7b 15
and Qwen-3-4b 16
We modify some settings based on the original implementations and develop an open-source project with multiple pruning methods. Our project code can be found at htttts//gghhubbooo///MLLLLCoo///
B.1 Implementation of Pruning Methods
Tab. shows the calibration datasets, the dataset number, and the random seeds used in the pruning methods. Tab. 4 presents the pruned layers' index order for the pruning method. We implement the Mag+, Taylor+, and SLEB using our reproduced code.
For the ShortGPT method, we follow the layer BI score for LLaMA-2-7B provided in the original article. For the LLaMA-2-13H model, the original paper provides only the pruning order for the first 10 layers. We the open-source project reproduction code to caleulate the remaining layers' BI scores and place the two with the smallest BI scores at the end of the given pruning order. For other models, we obtain the BI scores for each layer entirely through the reproduction method. PG19 is long- documentt dataset, and the training set contains 28,602 training samples. Using all samples to get the model's 's BI scores would consume significant training resources, so we randomly selected 256 training samples from PG19 for calibration. Even with a small amount of data, the ShortGPT method takes much longer to calculate BI scores than other methods.
httpp://wuppiggeee...c//eeaaaaaa 13b-
8B
23

---

When selecting MMLU data, MKA randomly samples five samples from 50 sub-tasks. In our imple- mentation, we uniformly sample 250 samples from each sub-task. We reproduce the experimental results for the LLaMA family using the original MKA code, while we obtaín the sparse results for other models using our reproduced code. The Qwen-2.5-7b model contains bias weights, and fusing these weights would degrade the performance of the pruned model, so we do not fuse the bias weights.
The calibration samples for the LaCo method are sourced from the open-source project code, and we fully reproduce the process using the original code. To achieve the number of pruned layers consistent with the settings in this paper, we make simple parameter adjustments to the LaCo method, with the detailed parameter settings shown in Tab. For other models, adjusting the LaCo code too complex, so we do not reproduce it.
The FuseGPT method is implemented using the original code. To compare different categories of methods, we comment on the post-training code of FuseGPT for the pruning method comparison experiment. In the pure pruning method comparison experiment, FuseGPT-MI++ means that we mask the post-training code, while FuseGPT-MI implies that we additionally mask the fusion code. Implementing this method on the LLaMA-2-13b model with NVIDIA A A100-40G GPU resulted in a memory overflow, so we do not implement it.
In the layer pruning process of CoMe, fuse two layers of the model per iteration, meaning that we reduce one layer per iteration. When pruning models from the LLaMA family, Vicuna-7b and Mistral-7b, the hyperparameter p is 7b, Qwen2.5-7h, and LLaMA-3-8b models have high knowledge density and less redundancy in parameters, making them very sensitive to hyperparameter settings. To further mitigate performance degradation caused by merging channels with different distributions, we set minimum parameter retention ratio p. meaning the proportion of parameters from the more critical layer cannot be less than during the fusion of two layers. The values of for the Mistral-7b, Qwen2.5-7b, and LLaMA-3-8b models set to 0.97, 0.85, and 0.97, respectively.
1. For the Qwen2.5-7b model, p is 32. The Mistral-
Table 7: Experimental setting for post-training methods.
Method
# Tterations
Epochs
# Steps
Batch size
Token length
FuseGPT -tre-mtreml
10
20
128
2048
1
5
938
32
2048
CoMe-mp
7
1
2000
32
512
CoMe-sp
1
10000
32
512
B.2 Implementation of Post-Training teaaTrainn
Tab. 7 summarizes the post-training settings for all methods. Based on these settings, we quantify the resource consumption each method by calculating the total number of tokens required to train single layer, which we denote as Tazer. This metric is computed as follows:
Tlayer Iterations # Layers # Epochs # Steps Batch size x Token length,
where "# Layers" indicates the number of layers updated in each iteration.
The post-training process for the FuseGPT method is synchronized with the pruning process, utilizing 1,024 samples from the Wiki-2 dataset, in accordance with the settings of Pei et al. (27) In each iteration, the parameters of one layer are merged into seven adjacent layers. Pruning ten layers requires ten iterations, with seven layers updated in each iteration. For FuseGPT, Tlayr 2.93B.
The StM--ttramline trains a merged layer using 30,000 samples and employs five epochs, following the settings of Chen et al. [5] For LLM L.MM-ttramlie, Tlager R 0.31B.
We carry out the post-training process of CoMe after completing the pruning process. After pruning 10 layers, the pruned model has seven layers corresponding to multiple layers of the original model. Therefore, CoMe-mp, there are seven training iterations requiring minimal training resources. CoMe-sp trains seven layers in one training round, requiring more training resources. For optimization, we utilize the AdamW optimizer with a weight decay coefficient of le 2 and implement cosine decay for learning rate scheduling. The CoMe-sp employs a fixed learning rate of le 5. The
24

---

CoMe-mp adopts spyerfspecific decaying rates during multi-layer distillation, with learning rates
progressively decreasing from the shallow to the deep layers as follows: 5e 4, 2.5e 4, le
7.5e - 5, 5e - 5, 2.5c 5, and le - 5 for L.aMMA2-77b; 5e - 4, 2.5e- 4, 5e - 5, 2.5e - 5. le- 5,
and 7.5e- 6 for Qwen-3-4b. For CoMe-mp, T Tlayer Rs 0.23B. For CoMe-sp, Tager 2 1.15B.
C Channel importance and Concatenation-based Merge
We use the channel importance for parameter division in the concatenation- -based merge strategy; thus, we need to analyze the channel importance calculation for different transformer parts. Ma et al. [23] highlight that in transformer- based models, certain correspondence exists in feature dimensions during forward propagation due to residual connections. For instance, the positional correspondence of output features from Norm, MHA, and FFN is fixed.
In the Norm part, we use the weights to scale the feature inputs. Xiong et al. 40 note that the parameters of deep layer Norms need significant enlargement to stabilize training. which is closely related to the distribution of input features. Our objective to minimize changes the output features of each module; therefore, we average the Norm parts of adjacent layers to maintain stability, [ 7,0), where m + denotes the number of merge layers.
as 7
m++
The MHA module generates three feature vectors: Query, Key, and Value. These vectors are concatenated and undergo matrix multiplication for cross-information fusion. Consequently, the weights within the heads used to generate Query, Key, and Value are tightly coupled, making it difficult to make finer divisions. Thus, we consider each head in MHA the basic unit for concatenation.
We ignore the coupling between heads to further simplify the calculation of channel importance. Pruning a single head structure reduces the input dimension of the o p_project weight (using the transformer structure in LLaMA as an example), leading to changes in the MHA output. We take the average channel importance of the reduced dimensions as the importance corresponding to each head structure.
The FFN module usually contains three weight matrices: up_ projecet,t, sat_project, and downn_rrojee By neglecting the coupling caused by activation functions, the information loss from channel weight pruning in up_ project and gat gat_project maps to a reduction in intermediate feature dimensions. Therefore, we use the intermediate features and
to calculate channel importance.
D Posterior-based CoMe
When applying CoMe to different model architectures, is often necessary to adjust the hyperparam- eter p to control the parameter preservation ratio. However, the optimal ratio can vary significantly across models, which reduces the convenience and usability of CoMe. Inspired by SLEB [34] and LaCo 42] we propose an adaptive, posterior-based strategy for determining the parameter preservation ratio within CoMe, referred to as Posterior- based CoMe (CoMe-P).
CoMe-P replaces the parameter preservation ratio calculation in the layer merging process of CoMe (Eq. (5)) with posterior-drye driven approach. Specifically, consider the case of merging two adjacent layers in iteration. Let the parameter preservation ratio of the lower-indexed layer be and that of the other layer be - r. We define a candidate set for r as = ( i 0, 1, 2,,,,....,, n). where n. determines the granularity of the search. CoMe-P iteratively applies different preservation ratios from I' to generate compressed models, evaluating each candidate model on a calibration dataset using the PPL metric. The compressed model yielding the lowest PPL is selected for the final merging. The detailed algorithm of CoMe-P is presented Alg. 2
We set = 20, with all other parameters kept consistent with the default settings of CoMe. Tabs. g to I0 present comparisons between CoMe-P and other methods across different models. C achieves performance comparable to CoMe, and yields higher average accuracy and lower PPL on Qwen3-4b, Vicuna-7b, and Mistral-7B, demonstrating the effectiveness of the posterior-based approach. However, since CoMe-P is posterior search method, the search space grows exponentially when merging more than two layers in each iteration, resulting in exponentially increased resource consumption.
25

---

E Analysis of CoMe-mp and CoMe-sp
To evaluate the effectiveness of CoMe-mp and CoMe-sp during the post-training process, we examine the cross-entropy loss between the stu- dent and teacher models, as shown in Fig. 12 When using CoMe-sp, the cross-entropy loss converges rapidly and stabilizes within the first 4000 steps. This phenomenon indicates an effec- tive alignment of feature representations, as the hierarchical distillation strategy facilitates rapid convergence. In contrast, CoMe-mp shows more linear convergence pattern, suggesting that aligning features layer by layer significantly hances the student model's performance. How- ever, because shallow features require process- ing by deeper layers, training one layer at a time results in slower convergence.
Figure 12: Cross Entropy loss curves for using CoMe-mp and CoMe-sp during post-training. The loss curves for multiple subprocesses of CoMe-mp are concatenated in the order of training.
Subsection B.2 details the number of tokens used during the post-training phase. Although CoMe-sp uses fewer tokens, it requires updating seven times more parameters per step than CoMe-mp, necessitating greater memory resources. The overhead of memory resources is due to CoMe-sp's simultaneous optimization of multiple layers, which, while essurce-intensive allows for more efficient global information updates compared to the sequential approach of CoMe-mp.
Sum-based Merge vs. Concatenation-based Merge?
The role of Weight Sum-based Merge in both pruning and post-training processes is LESS. Tab, and Tab. provide the effects of using parameter fusion (Fusion-MI+F) and only layer pruning without parameter fusion (Fusion-MI) in the FuseGPT method when pruning the LLaMA-2-7b model. In both pruning and post-training, the average accuracy score differences do not exceed 1.2 points, and PPL score differences do not exceed 0.5 points. Moreover, in pruning experiments, the performance of parameter fusion methods is worse when 20% of the parameters are pruned. It indicates that additive lnter-layer parameter fusion is ineffective. In all experiments, Fusion-MI+F does not significantly improve pruning performance compared to Musion-MI.
The Weight Sum-based Merge does not exhibit significant differences from the DLP methods, but the Concatenation- Merge can improve the performance of DLP methods. In Tab, after removing parameter fusion, the MKA method exhibits only a slight decrease in average accuracy scores and a slight increase in PPL, which is almost negligible. With the removal of parameter fusion, LaCo shows slight rise in average scores and significant decrease PPL, indicating that the parameter fusion has a negative impact. For FuseGPT, removing parameter fusion results in notable reduction on some datasets, such as HellaS and MMLU, slight decrease in PPL on the C4 dataset, and slight increase on the Wiki-2 dataset. It is difficult to conclude that parameter fusion further enhances model performance beyond layer pruning, but previous analyses suggest that FuseGPT has minimal effect. The Weight Sum-based Merge method does not significantly differ from Direct layer pruning methods. However, when CoMe removes parameter fusion, it shows a noticcable performance decline across all test benchmarks, except for the MMLU dataset, with significant increases in PPL on the Wiki-2 and C4 datasets. It strongly indicates that Concatenation Merge can further enhance model performance based on DLP.
Concatenated-basee Merge is Effective, but Weight Sum-based Merge NOT. In Figs. 13 and [4] we apply both a. + (1 - a:) B (WSLP) and Concatenation-ban based Merge to blend parameters of two layers in varying proportions. The a-Add method, whether merging the Atte-Ante structure, the FFN structure, or the entire model layer, consistently results in significantly increased PPL. on the Wiki-2 and C4 datasets. shows that the Weight Sum-based Merge method harms model performance, degrading performance as the fusion ratio approaches equality. Conversely, the
26

---

WELP
WELP-FFM FFN
WSSU MIAA
COMA--FN
CoMo-MMA
00
L20
Perpleiity
Pepleaty
110
6381
8802
ar9a
0604
6207
a286
up
U5p
0882
62833
6804
0900
0408
6387
6:00
e12a
Figure 13: Merge adjacent layers with linear weight aggregation at different ratios, using the
Figure 14: Merge adjacent layers with CoMe at different ratios, using the C4 calibration dataset.
C4 calibration dataset.
Concatenation -aase Merge method can reduce PPL specific fusion ratios, preserving the model's language modeling ability.
G Detailed Experimental Results
In this section, present comprehensive experimental data. The specific outcomes corresponding to Fig. are detailed in Tabs. g to
I0
Additionally, the results associated with Figs. to I0 are
thoroughly documented in Tabs. II 14 respectively.
27

---

Table 8: The Layer Pruning Experiment on the LLaMA Family.
Model
Spumity
Mathods
ARC-c
ARC-
HellsS
Benchmork OBIQA
PIQA
WinoG
MMLU (5)
AvgT
RP1
C4
PPL.
Wiks-2
Derse
46.33
74.54
75.99
44.20
79.05
69.06
45.60
66:11
100,00
7.27
5.47
Mags
57.97
66-20
67.65
39.80
75.44
59.83
26.
53.53
84-56
9.21
7.01
Taylane
42.06
69.07
73.00
42,00
75.68
66.19
42.22
58.99
94.51
10.21
7.74
ShanGPT
43.00
68.77
71.61
40.40
76.44
68.67
45.56
59.21
05.25
9.33
7.43
SLEB
38.57
65.82
70.69
39.30
77.26
63.45
33.83
55.63
89.35
8.71
6.47
9.0%
PI
39.65
68.86
70.11
40.40
77.45
61.88
31.95
55.77
03.49
9.73
6.55
F
39.51
65.57
71.03
40.50
77.37
52.13
55.62
8.73
6.47
MKAA
44.54
64.98
67.09
37.80
72.80
62.43
45.64
56.47
91.39
44.50
25,41
LaCo
43.43
68.60
71.78
40.60
70.39
05.51
45.39
59.24
95.35
9.38
7.46
CaMe
44.11
70.96
42.00
77.04
69.19
46.04
60.31
97.11
8.58
LLaMA 76
34.52
Tay Tore
36,19
52.47
ShonGPT
36.25
62.62
39.85
52.64
23.31
SLEB
33.02
71:29
PascCPPM-I
34.64
37.00
73.67
57.14
25.05
50.13
79.36
9.46
PuseGPT MH+F
51.92
50.21
388.57
247.36
26.79
49.87
33.50
71.55
70.90
39.59
54.35
23.45
39.31
27.20
51.70
22,96
37.52
39.49
52.35
Taytor-
32.68
36.06
31.60
62.67
42.89
47.35
76.75
63.08
50.96
ShanGPT
31.91
47.39
34.80
63.26
46.21
75.07
49.56
SLEB
30,80
51.81
54,14
45.31
30.0%
PiseGPTMI
30.20
50.59
33.60
54.54
45.21
FoseGPT-MI-H
30.20
50.13
55.06
34.80
63,50
55.41
27.01
45:88
72.82
17.50
14.34
MKA
34.04
49.56
46.12
35.00
63.00
59,13
35.64
46.36
75.14
610.04
455.34
LaCo
30.97
49.79
50.14
35.00
68.34
$3,91
24.84
44.71
71.11
39.18
42.67
CoMe
35.24
$4.46
56.56
35.40
68.85
62.17
25.50
48.17
76.47
19.93
16.53
CaMe-P
34.64
54.58
58.35
35.40
67.95
61.00
26.89
48.41
76.89
18.87
14.74
49:15
77.53
45.20
50.52
72.14
55.16
65:58
100.00
6.73
4.89
Mrge
62.84
59.55
20:71
ShinGPT
72.85
SLEB
42.69
72.31
60.05
7.81
5.64
MI
40.95
69.02
46.00
79.11
42.52
59.84
7.83
5.65
MKA
47.01
45.00
60.77
CoMe
47.53
75.25
76.33
43.20
79.49
71.96
55.34
64.43
98.10
7.61
5.36
LLLMA 2 1356
Mags
23.12
46.42
37.81
29.50
65.83
50,91
23.65
39.65
59.38
125.08
278.40
Taylore
43,26
65
72.09
40.80
75.30
70.45
47.34
59.76
90.00
13.37
12.12
ShonGPT
43,94
67.36
72.39
41.00
75.24
69.69
53,83
60.19
92.25
11.36
8.30
SLEB
37.88
64.65
70.59
42.40
76.82
64.64
30.32
55.61
83.85
9.47
6.85
19.5%
PINGGPTEM
38.91
63.72
70.97
40.60
76.93
67,32
42.57
57.29
36.66
9.69
7.04
MKA
40.61
$9.60
57.18
41.40
68.03
62.90
53.04
54.81
$4.58
219.41
206.12
LaChn
34.83
$4.97
64.67
39.20
74.32
63.61
23.53
50.73
76.14
13.04
10,86
CoMe
45.14
72.52
75.87
42,80
75,50
70,80
50.35
62.00
94.30
9.17
6.29
Mag+
23.12
33.16
30.27
25.40
$6.15
$2.17
25.37
35:12
53.23
317.35
$09.77
Tayfor-
35,91
$4.97
62.24
37.20
70.75
89.61
48.20
54.55
33.21
23.96
28.38
ShonGPT
35.75
52.82
57.94
36:20
19.91
59.06
47.78
53.07
$1.06
29.37
39.61
SLEB
$4.04
58.59
63.18
36.60
75.35
62.35
26.72
51.29
76:94
11.64
8.60
29.24
PENICiPT-MI
37.29
56.90
64.80
36.60
74.45
65.04
30.75
52.76
78.61
12.65
9.46
MKA
36.95
53.07
48.67
36,00
63.56
60.46
50.72
50.211
77.39
759.76
633.25
LaCo
33.19
51.43
54.88
39.00
65.39
60.77
24.55
47.46
71.85
27.43
23.81
CoMe
42.69
67.05
69.87
42.60
73.34
68.95
51.17
39.36
90.67
12.64
8.85
CoMe P
43.43
66.96
69.38
40.20
73.50
69.77
51.81
59.29
90.43
12.32
N.56
Dense
$3.33
27.74
79.17
45.00
80.79
72.93
65,29
67.75
100.00
9.45
6.14
Mags
34.73
64.02
49.37
36.00
74.16
54.76
25.35
48.34
71.50
25.93
20:44
Tuylure
47.70
70.92
66.31
41120
76.01
72.69
30,57
57.34
35.00
20.59
14.88
ShonGPT
47.44
69.99
73.63
39.80
76.26
71.43
63.67
63.18
92.90
20.58
15.07
9.2%
FRNCCPPEM
SLRR
41.30
42.83
70.29
67.47
70.79
69.05
38.50
39.00
77.80
77.53
64.33
69.14
66.20
30.01
62.12
55.53
81.18
91.05
13.65
13.68
8.93
8.85
PuseCiPT MH+F
40.10
67.26
68.47
38.20
75.66
62.43
29,97
54.73
79.93
13.72
8.95
MKA
44.71
63:45
62.75
41.20
72.96
64.09
53.65
58.97
37.42
307.03
191,80
LLAMA 3 86
GoMe
47.70
72.81
72.78
40,60
76.50
74.19
$3.65
63.96
94.08
14.84
9.52
Mage
25:77
46.02
43.40
30.40
65.29
53.20
25.16
41.32
601.2
43.24
40.83
Tustore
31
43.60
35.50
32.20
60.17
55.80
33.05
42.18
62.58
1549.77
1294.94
ShoRGPT
43.43
56.52
64.65
33.40
70.89
71.11
61.73
57.24
$3.09
63.81
17.64
21.4%
PascCIPT-MI
SLEB
34.55
35.75
59.51
58.42
59.03
62.20
34.00
34.00
74.37
73.83
57.85
$6.67
27.43
50.43
52.70
50005
76.98
72.09
19.38
18.67
13.35
13.44
PaseGPT MH+F
33.20
53.91
61.17
35.30
72.74
57,93
26.42
48.81
71:35
19.03
13.42
MKA
42,55
60.10
35.90
40.40
65.85
62.04
59.27
35.60
$2.66
1 1168.37
1004.27
CoMe
35.60
73.50
70.96
$6.95
58.17
N5.12
23.10
17.15
Magt
22.15
24.19
35:87
242,47
254.76
Taylare
27.99
30.44
37.40
57.73
$2.17
47.05
30.53
58.67
44214.20
50035.02
30.20
38.13
30.20
59.09
41:31
15690.99
SLEB
27.73
27:80
66.97
51.46
25.83
42.47
30,90
28.27
PISECIPEEM
29:01
43.25
29.20
66.05
54.14
40.95
45.09
05.82
37.93
FIseGPT
38.14
50.52
15.02
7447.81
CoMe
33.70
50.67
67.08
30,99
46.38
28

---

Table 9: The Layer Pruning Experiment on the Vicuna-7b and Mistral-7h.
Model
Spasity
Metheds
ARC c
A
HellaS
Benchmark OBQA
PIQA
WinoG
MMLU (5)
AugT
RP1
C4
PPL.)
Wilki
Dense
43.90
71.30
73.78
45.00
75.02
69.85
49.89
61.91
100,00
9.19
6.73
Mage
Tayline
58.81
SainGPT
PiseCIPTEMI
41.30
41.98
92.15
72.33
Vicuma 7b
46.91
ShanGPT
$9.09
10
FINCCPTEMM
36.25
51.02
FiscGIPT-Mi4F
35.00
54.17
335.40
CoMe
40.96
64.52
42.40
72.47
68.11
35.43
55.77
89.43
16.86
Mags
24.15
40.78
36.13
27.60
62.30
50.83
25.06
36:12
60148
72.72
92.79
Taylore
33.96
46.55
49.17
31.60
60.85
62.12
52.13
45.20
72.57
62.01
183.82
ShonGPT
33.11
48.40
48.95
34.60
64.25
62.90
41.04
47.61
76.92
61.91
59,84
30.0%
FANCCPPEM
FI
33.62
30.00
57.45
56 56.10
53.01
53.56
33.00
36.40
68.88
70.24
53.75
56.30
25.25
24.38
47.02
46.42
74.86
73.34
18.61
NaN
15.63
NaN
MiseGPT-MMiE
33.35
53.11
53.02
34.00
65.25
55.09
25.07
45.99
73.09
NGN
19,85
MKA
34.94
48.15
47.02
25.80
61.97
61.17
47.99
48.02
75.38
996.16
650,93
CoMe
36.35
58.84
56.48
42.40
68.66
62.96
25.53
50.15
$0.28
29.55
18.00
CoMe-P
$4.56
57.97
56.42
36.80
69.53
67.51
78.45
49.3 16
78.59
22.32
18.97
5.25
49.13
Tuytore
44,85
70335
41160
62.09
ShonGPT
43.35
73.40
79.96
62.26
66.95
95.02
7.14
SLEB
43.09
21.09
76.33
41.40
79.16
59.39
36.36
9.76
PoNGGPTM
45.39
72.10
74.38
41.60
80.41
64,80
60.08
87.65
6.25
FINCCIPT-MI4F
42.49
70.75
73.49
41.00
79.76
06.45
30.62
59.08
35.08
9.86
6.31
MKA
43.43
61.05
62.75
55.32
82.35
214.34
200.48
CoMee
48.55
74.37
76.93
41.00
79.60
72.61
61.52
64.94
95.00
10.04
6.52
Misral To
Mage
23.12
38.47
33.44
25.00
59.85
52.09
23.53
36.39
$3.08
876.24
1409.19
Taytore
35.24
$1.46
66.30
33.80
73.72
61.64
25.05
49.74
71.87
19.58
15.34
ShonGPT
40.54
57.66
65.53
32.30
72.14
67.85
59.95
56.19
$2.43
33.21
24.01
21.19
SLEB
PiseCIPEEI
35.177
36.95
55.88
61.36
65.71
66.51
39,00
37.00
75.08
75.24
51.45
55.96
24.95
28.83
50.47
52.52
73.13
76.44
14.16
13.55
10.00
9.35
PiseGITTMM+F
$4.55
57.20
65.94
36.60
74.76
$7.30
25.65
50.29
72.87
21.73
15.29
MRA
35.84
42.47
40.42
33.80
55.49
$7.14
$3.95
46.02
68.75
36779.48
30245.43
CiMee
40.61
63.50
67.54
36.20
74.21
67.64
53.50
$7.64
34.06
14.33
16.01
Mage
25
32.07
30.74
28.20
54.08
50.25
25.03
35.09
51.36
233.79
258.66
Taste-
29:01
35.75
32.20
60.91
54.62
24.70
40.26
112.23
121.24
ShonGPT
32.00
29:71
31,60
$7.51
56.91
22.72
37.72
36.10
760.27
881.51
30.1%
SLEB
PI
30,00
32.25
51.08
47.69
36.23
57.00
34.20
33.80
65.44
70.78
57.22
52.96
25.10
25.92
45.7%
46.25
66.56
67.17
$2.31
21.19
47.95
16.46
FUNGGPT-MM+F
48.68
44:18
20.62
33.54
30.50
$4.70
25.00
56.72
33005.64
37735.00
29

---

Table 10: The Layer Pruning Experiment on the Qwen-2.5-7b and Qwen-3-4b.
Sparity
Methods
UBQQ
PIQA
(MLUU())
AugT
PPL.
77,36
78.95
47.20
79.65
14,16
56.75
6:85
Mage
43,00
68.48
62.44
38.40
75.14
50.54
49.64
56.31
52.47
15.73
9.23
Tuylee
47.70
71.84
07.62
37.80
73.01
67.01
65.90
61.44
55.93
16.96
11.16
ShorkiiP
46.99
72.43
72.23
44.00
79.00
58.85
55.88
62.15
90.42
13,64
8.13
9.29%
FuseGPT MI
SLEB
43.09
45.59
72.43
07.22
72,90
72.27
44.00
44.00
78,89
79,00
02.90
04.85
03.05
55.88
61.90
62.15
90.43
39.36
13.64
14.42
8.13
8.78
64.15
48.63
25339.47
56126.31
CaMe:
79.22
$4.85
8.17
Quand
Taylor
Mags
33.02
20.55
45.12
51.52
05.30
50.35
12.00
35.40
02.77
67.90
51.85
55.88
28.19
08.73
44.65
46.11
077
64.69
72.43
29.12
162.15
16.90
SharkCP
34.98
62:50
69.41
27.40
73.94
$4.38
27.87
50.21
72.84
17.98
11.37
21.416
SLEB
34.98
33.87
62.50
69.41
00.79
37.40
73.94
71:45
54.38
27.87
50.21
72.84
17.98
17.16
11.37
27,34
50.82
27.85
34.10
50
73.61
53.00
Mage
25.51
47.52
39.40
31.20
62.57
48.54
26.01
40.11
59.21
58.89
36,02
Thyle-
25.85
35.00
51.57
15.60
56.76
51.30
25.99
16.55
59.81
919
432.97
ShorsCPP
50.55
52.53
43.90
32.00
66.76
53.67
25:32
44.11
63.06
32.87
22:51
10.058
SLEB
[MI
26.02
27.30
52.57
43.81
43.21
23.26
28.80
32.00
06.10
08.44
53.45
51.78
25.94
20.06
41.00
43.85
59.21
03:30
55
26.35
51.52
17.18
50.71
24.21
32.59
17641898.00
CoM::
26.78
00
75.79
57.57
Thylor-
71.05
70.35
59.46
53.599
21.55
ShanGPP
42.58
59.60
64.92
33.80
71.11
56.05
70.20
58.45
$7.50
23.06
15.88
SLEB
43.43
67.09
63.57
39.00
75.30
50.35
48.53
56.77
55.91
15.83
9,69
10.06
48.03
23.59
63.54
39.60
75.79
52.15
55.63
68.16
91.01
16.05
9.45
PUse M-M
18:00
75.53
51.95
55.55
16.03
9.46
67.19
71.39
497.26
Mags
35.49
61.70
56.05
37.00
72.42
54.54
24.96
48.35
74.13
344
16.60
Thy
30.55
42.00
47.63
29.40
63.82
59.27
23.69
42.34
64.02
$1.53
72.02
SheeLaPT
35.35
45.12
52.10
31 10
65.13
49.75
38.11
46.79
70.79
96.83
130.50
SLEB
38.65
64.48
55.04
37.50
70.84
57.85
50.82
50.95
37.41
21
13:01
FuseCPTMMII
35,09
59.85
56.41
37.50
72.31
54.38
31.70
40.58
75.81
20.77
13.27
65.23
52.71
538.13
913.36
55.10
51.37
Mage
27.56
45.45
43.50
33.60
65.18
50.99
25.23
41.52
63.20
56.14
47.50
Tuylee
25.58
31.23
52.73
30.60
$1.90
50.20
23.24
35.95
55.44
3961.42
7781.82
32.17
50.81
45.73
31.40
62.46
55.76
22.75
41. 50
63.72
457.05
$1982
SLEB
30.97
53.32
46.42
31.00
03.67
53.83
27.16
44.85
66.50
19.00
28.11
PuseGPT MI
30.83
55.47
23,90
30.00
03.51
52.80
27.35
43.96
06.17
33.53
23.49
FusCCPT
CoMie
47.10
63.95
79
61.00
51.46
32.67
42.15
63.84
56.13
32.14
CoME-- P
27.56
47.73
43.41
32.60
63.98
$4.06
29.35
42.58
64.43
40.83
28.18
Table 11: The detail experiment result of Fig.
Effect of p in heuristic merge ratio.
Benchmarkt OBQA
PPI.
P
ARC-c
ARC-e
HellaS
PIQA
WinoG
MMLU (5)
Avgt
RP
C4
Wiki-2
35.24
54.46
56.56
35.40
68.8% 68.88
61.17
25.50
48.17
76.47
19.93
16,53
35.75
54.84
56.27
35.40
68.93
60.77
25.30
48.18
76.51
20.70
16.57
4
34.98
54.76
57.86
36.40
67.79
59.75
24.22
47.97
76.12
19.87
17.40
34.22
54.17
56.41
36.60
68.12
58.48
26.47
47.78
76.07
20.94
19.19
16
32.08
53.62
56.07
33.60
68.34
60.62
26.81
47.31
74.86
20.30
17.82
32
30.89
52.78
55.87
33.40
68.77
59.43
26.93
46.87
74.10
21.43
18.64
64
32.34
53.28
56.31
34.60
67.95
59.91
26.91
47.33
75.06
21.69
18.01
128
32.25
52.48
56.51
33.20
69.31
57.38
27.47
46.94
74.36
23.29
18.64
256
32.00
50.80
55.99
34,80
67.74
60.22
27.16
46.96
74.58
23.97
19.72
512
31.66
49.33
55.79
33.20
67.79
59.04
26.82
46.23
73.30
25.03
20.97
inf
31.31
49.07
55.69
32.80
68.01
60.30
26.63
46.26
73.24
25.50
21.21
30

---

Table 12: The detail experiment result of Fig. 10
Effect of calibration data scale.
Benchmarkt OBQA
PPI.
Num
Avg
RP
ARC-e
ARC-e
HellaS
PIQA
WinoG
MMLU (5)
Wiki-2
2
33,87
52,40
56.42
36.60
66.97
59.19
25.27
47.25
75.19
29.35
43,70
35,58
54.25
56.45
37.00
68.23
59.59
27.05
48.31
77.07
23.45
31.42
33.28
53.16
54.85
37.00
67.63
59.98
26.95
47.55
75.79
21.08
21.01
16
35.41
55.93
56.83
36.20
66.97
60.14
26.30
48.25
76.80
20.26
16.99
32
33,36
53.49
55.64
36.00
66.54
61.33
27.38
47.68
75.92
22.67
17.26
64
33.19
53.03
56.58
35.80
67.46
60.46
26.87
47.63
75.72
20.61
16.11
128
34.56
53.96
56.37
36.80
68.44
61.56
25.22
48.13
76.49
20.88
17.03
256
35.24
54.46
56.56
61.17
25.50
48.17
76.47
19.93
512
33.79
54.04
56.79
36.60
67.52
60.14
27.23
48.02
76.45
19.60
Table 13: The detail experiment result of Fig.
Impact of merge step granularity.
Benchmark? OBQA
PPI.].
m
ARC-c
ARC-e
HellaS
PIQA
WinoG
MMLU (5)
Avgt
RP
C4
Wiki-2
35.24
54.46
56.56
35.40
68.88
61.17
25.50
48.
76.47
19,93
16.53
34.30
48.48
54.74
33.60
65.07
63.69
37.69
48.22
77.76
29.56
45.18
32.59
48.19
46.94
34.80
65.29
63.22
24.78
45.12
72.00
53.32
109.74
32.34
42.21
38.03
34.00
59.30
57.70
28.12
41.67
67.66
91.96
668.17
30.63
43.10
40.36
35.00
60.39
58.64
22.57
41.53
66.72
128.37
551.57
32.00
41.25
37.14
33.40
58.22
59.51
29.73
41.61
67.70
97.65
268.47
33.11
47.22
40.69
35.40
63.49
58.41
28.76
43.87
70.92
96.44
276.09
29.69
43.39
41.67
33,80
63.60
57.46
25.56
42.17
67.62
65.33
193.89
10
30.29
43.18
41.13
33.60
62.89
57.06
28.38
42.36
68.27
78.85
303.17
11
30.46
41.58
40.24
34.20
61.37
59.91
24.71
41.78
67.20
104.82
470.91
Table 14: The detail experiment result of Fig.
Impact of calibration datasets.
Benchmarkt OBQA
PPL.
Dataset
ARC-c
ARC-e
HellaS
PIQA
WinoG
MML (5)
Avgt
RP
C4
Wiki-2
wiki2
35.24
54.46
56.56
35.40
68.88
61.17
25.50
48.17
76.47
19.93
16.53
C4
34.30
52.95
57.25
35.60
69.80
62.04
26.39
48.33
76.71
19.32
21.05
PG19 (2)
34.39
50.21
53.74
34.00
67.25
60.69
27.90
46.88
74.77
21.28
21.15
MMLU
33,70
50.80
$4.13
33.20
66.00
63.77
27.20
46.97
74.67
28. 18.12
35.79
Aplaca
34.39
51.77
53.54
35.00
67.25
63.77
29.87
47.94
76.61
25.65
30.57
31

---

Algorithm Progressive Concatenation-basee Layer Merging Strategy (CoMe)
Input: calibration dataset D, original model M, the number of layers skipped in SBI m, skewness exponent p. target layer number L, minimum retention ratio p (0,1) Output: Pruned model M layer mapping P = [{a1. b1) bi)...., ............. fax by
Initialize + 0, M' <- M
2: while NUMLAYERS( (M') > L do
3: m' <- min(m, NUMLAYERS(M") L)
4:
{5 -- CCOMPUTCCHANNEESSSSITTYYYYYC D)
5:
{BI} -COAPUTEBISCORESSHH
6:
(SBlut+m²) COMPUTESBISCORESS (H m')
7:
(1",1" +m²) + arg min SBIg+
8:
W/(nerge)) +- -ONGATEEATOOTAASSDLLEVEEEEEE PP.
fri) < AADUSTRETENTTORRRTOSS
9:
10: Replace layers , 1" + m'] with W in M
11: P+PU PK( + m', new layer index in M']
12: end while
13: return ...
14:
15: function COMPUTTCHAINN..SSSSTVVYYYY D)
for each layer M' do
17:
H FORWARDPASS(AA D.I)
18: 500 < [E>[[>; > [wi
b Eq.
19: end for
20: return {8())
21: end function l1=1
22: function COMPUTE
23: for / = to NUMLAYERS M") do
H 1) H
24:
Bh: + Ep
D Eq.
H
H
25: end for
26:
return {BI,}
27: end function
28: function COMPUTESBISCORES/ H H m²)
29:
for 1= NUMLAYERS M")
m do
H H (+mm²
30:
SBI <- Ep
>> Eq. (3)
31: end for
32: return (SBIutm')
33: end function
function ADUUSREEENNTONNAIIX((....
P..P)
35: Ti + BI/ TRR/E,II, BI, for e [[" mm]
b> Eq.
if maxx; < then
37:
Farg max BI, + arg max BIg -
39:
40:
Tt + p)r\/ E' for +
41: end if
42:
return {r.} normalized to Σr 11
43:
end function
44: function CONCAEENNTTOBBBBBLLLLEEEEEEEECET (1) r.)) 4m
45:
for each layer / 11", + m') do
46:
l rt x |5(0] Te Top-k, indices sorted by ((t)
47:
48:
end for (m(merg))
+ +' W T.)
49:
Eq.
50:
return (iterrg))
51:
end function
32

---

Algorithm 2 Progressive Posterior-based CoMe (CoMe-P)
Input:
Calibration dataset D, original model M, target layers number L,, search granularity
Output: Pruned model M' layer mapping P = [[a1, b1),..,, {ax, bx Initialize 0, Ad' M.
2: Generate parameter preservation ratio candidate set I' = { i= = 0, 1, n
3: while NUMLAYERS(A"') > do
4: {5(9] +- COMPUTECHAANEESSSSTTTVVYYYY D)
5: {SBIa441} + COMPUTESBISSCRESS/ 1.1)
6: (1",1" +1) < arg min SBI.++
7:
for each r in r do
8:
Wimnrge) &-
" + 1] with Wi in ).)
9:
+ Replace layers ,I","" ppl - PPL(M")
10:
11:
end for
12: M <- <The M" has the lower ppl
P-PU
new layer index in M')
14:
end while
return M1, P
Algorithm 3 CoMe Single-Process Post-training (CoMe-sp)
Input: training data Durain layer mapping P Kar, by ..., (aN, by }, teacher model M, student model M', learning rate 7), optimizer S2, batch size B
Output: Optimized student model M'
1: Initialize <- ADAM (0,,. Kai, bi} E P),
Optimize merged layers only
2: for epoch to Egiobal do
3: for + BATCHLOADERR (Duain : B) do
4:
H (t.
J + GETACTIVATIONS M, x, {ailla, bi) P)) GETACTIVATIONS M x, (bilfa, b,} E P))
(
5:
6:
return {H (Ga:)) H( l l==1
7:
Listal f- 0
8:
for i
1 to N do
9:
L N
(o ( (H(()))
Eq. (6)
10:
(1) Lisal < Cioual + LKL end for U.zero (aeeo_grad))
b Eq.
11:
12:
13:
L BACKWARD() 14: S2.slep()
15:
end for
16:
end for
33

---

Algorithm 4 CoME Multi Multi-Process Process Post-training (CoMe-mp)
Input: training data Dirain. layer mapping P = [Kr,, bn) bi..... (aN, by }}, teacher model M, student
model M", learning rate (m....>//p)) optimizer f batch size
1: for k to [P] do
LLyerwise progression
2:
{ax. by + P[k]
3:
S4 f ADAM( 77 = 17)
4:
for epoch = to Erocal do
5:
for B BATCHLOADER(( B) do
6:
(tt,au) GETSINGLEACTTIVTIONN 0k)
7:
H t FORWARDTOLAYER(MM x, bg)
8:
LXL. (KL(o(H((.a)) (HH())))
Eq. (6)
9:
Dg.zero_gradd
10:
BACKWARC Sg.step()
11:
end for
13:
end for
end for
15:
return M
34


Reviews:
[{"review_id": "wsuYHCndEu", "reviewer": "Reviewer_SDDe", "summary": "This paper introduces CoMe, a new framework for structured compression of Large Language Models that addresses the limitations of traditional layer pruning and linear weight merging. Instead of removing layers directly, CoMe employs a channel sensitivity metric to identify important channels based on activation intensity and weight norms. These channels are then merged via concatenation across adjacent layers, preserving useful information and enabling progressive pruning.", "questions": "1.\tAll figure used raster image instead of vector image.\n2.\tI’m confused about the concept of “adjacent layers”. For example, if W^l refer to one up_proj, do W^(l+1) refer to next transformer block’s up_proj or this block’s down_proj?\n3.\tWhy does it called “Hierarchical” Distillation Strategy? In Table 1, I found “sp” performs better. And Equation 8 includes all KL divergence in one loss. I didn’t find any hierarchical distillation.\n4.\tm in Figure 1 is just 1 in practical right? Because only two layer are merged in one time?", "limitations": "Addressed", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1.\tThis paper focused on an important problem of structured pruning for model compression.\n2.\tThis paper propose a framework to iteratively merge layers.\n\nWeakness:\n\n1.\tThe concatenation-based weight merge process in Figure 5 doesn’t seem make sense to me. The concatenation disrupts the channel index of original weight and new weight. Then the new input completely mismatches the original input. I think there lacks some theoretical explanation or motivation about the effectiveness of this new algorithm and why this should be better than others.\n2.\tPlease check “Question”.", "was_revised": true, "final_justification": "Thanks for the authors' response. Most of my concerns have been addressed. I would like to raise my score.", "rebuttal": "Thank you for taking the time to review our manuscript. We greatly appreciate your valuable feedback. \n\nBelow, we first address [Question 2] and [Question 4] to set the stage for reiterating our method and resolving the concerns you mentioned under weaknesses. We then explain the meaning of the \"Hierarchical\" Distillation Strategy and address the remaining minor issues.\n\n## **[Question 2] Clarification on \"Adjacent Layers\"**\n>In our manuscript, the term “adjacent layers” refers to consecutive layers within the model that are directly connected through input-output relationships. This is a widely used term in academic literature. For example, if the output of layer $l$ serves as the input to layer $l+1$, and the output of layer $l+1$ serves as the input to layer $l+2$, then layers $l$ and $l+1$ are considered “adjacent layers.” Similarly, layers $l$, $l+1$, and $l+2$ (or more consecutive layers) are referred to as “a group of adjacent layers.” \nIn Figure 5, $W^{(l)}$ and $W^{(l+1)}$ denote the weight matrices with the same function name (e.g., up\\_proj) in adjacent layers. \nSpecifically, $W^{(l)}$ corresponds to the up\\_proj weight in layer $l$, while $W^{(l+1)}$ corresponds to the up\\_proj weight in layer $l+1$. \nFigure 5 illustrates only the fusion method. Depending on the type of matrices, the fusion needs to be performed along different channels (either rows or columns).\n\n>To reduce potential misunderstandings, we will revise the caption of Figure 5 in future versions to:\n“Concatenation-based weight merge process between weight matrices with the same function name in adjacent layers. $W^{(l)}$ and $W^{(l+1)}$ denote the weight matrices with the same function name in two consecutive layers.”\n\n## **[Question 4] Explanation of Hyperparameter $m$**\n>The hyperparameter $m$ in Figure 1 can be any value less than or equal to the total number of layers in the model. CoMe is capable of merging an arbitrary number ($m$) of layers simultaneously. \nIn Figure 9, we conduct an ablation study on $m$, and the results show that merging too many layers at once degrades the performance of CoMe. Therefore, as stated in the experimental details (line 582), we default to merging two layers at a time.\n\n>To reduce potential misunderstandings, we will revise the manuscript as follows:\nAt the first mention of $m$ (line 207), we will add the clarification \"$m>=2$ and less than or equal to the total number of layers in the model.” In the implementation details (line 269), we will specify \"During the layer pruning process, the merging layers number $m$ in each iteration is set to 2.”\n\n## **[Weakness 1]** \n> We thank the reviewer for pointing out this issue. In future versions of the manuscript, we will include a more detailed theoretical analysis in the Appendix to further clarify our method.\n\n### * **Explanation of Channel alignment**\n>When designing CoMe, we carefully considered the challenge of channel alignment in the layer. Below, we clarify the intrinsic rationale of the CoMe and explain why it does not lead to a mismatch between the original and the new input.\nWe decompose the transformer into two main components (Appendix C): the FFN and MHA. In the FFN, we define the input as $X \\in \\mathbb{R}^{d}$, the intermediate feature as $I \\in \\mathbb{R}^{r}$, and the output as $O \\in \\mathbb{R}^{d}$. The weight matrices $W^{U} \\in \\mathbb{R}^{d \\times r}$, $W^{G} \\in \\mathbb{R}^{d \\times r}$, and $W^{D} \\in \\mathbb{R}^{r \\times d}$ correspond to up\\_proj, gate\\_proj, and down\\_proj, respectively. Thus, the FFN can be formulated as:\n\n>$$ I_{[i]} = \\sigma(W^{G}\\_{[i,:]} X) (W^{U}\\_{[i,:]} X)$$\n>$$ O = \\sum W^{D}\\_{[:,i]} I\\_{[i]} + X$$\n\n>From the above equations, it is evident that permuting the channel indices ($i$) of the intermediate feature $I$ does not affect the channel indices of the input $X$ or the output $O$. It only changes the internal channel order within the FFN. \nWhen we concatenate the intermediate features from different layers at the corresponding channel indices, we are simply reorganizing the submodules responsible for feature processing within the FFN. \nRegarding channel mapping in the model, we strictly follow the detailed analysis of channel alignment provided in LLM-Pruner *[1]* (Line 612). Therefore, we partition and concatenate the channels in the FFN according to the intermediate size dimension.\n\n>For the same reason, we believe that the WSLP, which directly fuses weights, may result in incorrect weight fusion, as it does not ensure channel correspondence across different layers, potentially leading to smoothed weight distributions.\n\n>Similarly, for the MHA structure, we partition channels based on the input dimension of O\\_proj. Since the head structure in MHA is inherently indivisible, we further treat each head (comprising $W^Q$, $W^K$, and $W^V$) as a “channel group.” Changing the order of heads or the input order of O\\_proj does not affect the overall input or output of the MHA.\n\n\n### * **Theoretical Basis of Concatenation-Based Merging**\n>Layer pruning achieves higher efficiency by reducing the number of layers, which inevitably leads to input space shifts between layers. This issue is common to DLP, WSLP. The key lies in mitigating such misalignments.\n\n>The core idea of CoMe is to divide each layer into finer-grained functional blocks. From the perspective of the FFN module, after partitioning parameters into more granular channels, each channel can be regarded as a small expert. Fusing two adjacent layers is thus equivalent to selecting and retaining the most important experts. Several recent works *[2,3]* have adopted similar perspectives to analyze the FFN (or experts). Likewise, in the MHA structure, each head can naturally be viewed as an independent feature processor.\n\n>Furthermore, we employ a Channel Sensitivity Metric to select which channels to retain or discard based on their contribution to the output. Channels with larger products of activation and weight magnitude are considered to have a greater impact, which avoids the need to compute gradients. This approach is consistent with the perspectives of several model sparsification methods *[4,5]*.\n\n## **[Question 3] Definition of \"Hierarchical\" Distillation**\n>We provide a detailed description of the Hierarchical Distillation Strategy in Section 4.3 of our manuscript. Lines 238–245 specifically explain how the CoMe-mp process is transformed into CoMe-sp by modifying the training approach and loss function. **Our hierarchical distillation strategy leverages layer correspondences $\\mathcal{P} = \\\\{\\\\{a_1, b_1\\\\}, ..., \\\\{a_N, b_N\\\\}\\\\}$ established during pruning**, where $a_i$ and $b_i$ are indices of corresponding layers in the teacher and student models, respectively. For example, if $\\mathcal{P} = \\\\{\\\\{1, 3\\\\}, \\\\{2, 6\\\\}, \\\\{5, 8\\\\}\\\\}$, then the output of the 1st layer in the student is aligned with the output of the 3rd layer in the teacher, the 2nd layer in the student is aligned with the 6th layer in the teacher, and the 5th layer in the student is aligned with the 8th layer in the teacher.\nThe loss function (Equation 8 in the manuscript):\n>$$\\mathcal{L}\\_{\\text{KL-sp}} = \\frac{1}{|\\mathcal{P}|} \\sum\\_{\\{a,b\\} \\in \\mathcal{P}} \\mathbb{E}\\_{\\mathcal{D}\\_{\\text{train}}} \\left[ D\\_{\\text{KL}} \\left( \\sigma(H^{(t,a)}) \\| \\sigma(H^{(s,b)}) \\right) \\right]$$\nenforces alignment of **layer-wise features** across all hierarchical levels, rather than only the final logits. \n\n>**Therefore, both CoMe-mp and CoMe-sp distill knowledge from the original model to restore the performance of the pruned model by aligning the outputs at the “layer\" level. For this reason, our approach is referred to as a “Hierarchical Distillation Strategy.”**\n\n## **[Question 1] Format of Figures**\n>Our manuscript was prepared using Overleaf, with all images provided in .pdf format. Nevertheless, following your helpful comments regarding the quality of our figures, we also noticed that some images appear blurry when zoomed in. We will provide clearer vector images in future versions of the manuscript.\n\n## Reference\n>*[1] Ma, Xinyin, Gongfan Fang, and Xinchao Wang. \"Llm-pruner: On the structural pruning of large language models.\" Advances in neural information processing systems 36 (2023): 21702-21720.*\n\n>*[2] Lo, Ka Man, et al. \"A Closer Look into Mixture-of-Experts in Large Language Models.\" Findings of the Association for Computational Linguistics: NAACL 2025. 2025.*\n\n>*[3] Zhu, Tong, et al. \"LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-Training.\" Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.*\n\n>*[4] Sun, Mingjie, et al. \"A Simple and Effective Pruning Approach for Large Language Models.\" The Twelfth International Conference on Learning Representations, 2024.*\n\n>*[5] Xie, Yanyue, et al. \"Moe-pruner: Pruning mixture-of-experts large language model using the hints from its router.\" arXiv preprint arXiv:2410.12013 (2024).*\n\nThank you again for your valuable feedback, and we hope our responses address your concerns. If you have further questions, we would be happy to provide additional clarification."}, {"review_id": "zPzBjH4xzM", "reviewer": "Reviewer_ACeT", "summary": "This paper introduces CoMe, a novel structured compression framework for Large Language Models (LLMs) that addresses key limitations in existing pruning and knowledge distillation methods. The authors propose a progressive layer pruning framework with a Concatenation-based Merging technology and a hierarchical distillation post-training process to reduce model size while minimizing performance degradation", "questions": "- Could an adaptive or learned ratio improve results further, and has this been explored?\n- Have the authors considered the computational cost implications in real-world deployment scenarios, particularly concerning hierarchical distillation?\n- Can the concatenation-based merging approach be generalized or adapted to non-transformer architectures or other modalities beyond NLP?", "limitations": "See strengths and weaknesses", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strength**\n- The paper empirically demonstrates its superiority in preserving model capacity and hierarchical knowledge.\n- CoMe integrates pruning and distillation into a unified, progressive framework, which is crucial given that previous approaches often handle these stages separately, leading to suboptimal knowledge transfer.\n- The \"Rethinking the Layer-based Structured Pruning\" section (Section 3) provides strong motivation for their proposed approach.\n- The introduction of a channel-level sensitivity metric provides a more principled basis for fine-grained pruning.\n\n**Weaknesses**\n- The channel sensitivity metric and BI/SBI scores rely on a calibration dataset. The impact of the size and representativeness of this dataset on CoMe's performance is briefly discussed (Fig. 8 ), but further analysis of its sensitivity to dataset characteristics could be beneficial (eg, impact of adding noise, reasoning capabilities, etc).\n- While the concatenation-based strategy is effective, its reliance on heuristic parameter preservation ratios limits adaptability across diverse model architectures.\n- The computational overhead introduced by hierarchical distillation, especially CoMe-sp needs to be discussed,\n- Limited exploration on the interpretability of the retained parameters and their roles post-concatenation.", "was_revised": true, "final_justification": "The authors have addressed my concerns.", "rebuttal": "Thank you for taking the time to review our manuscript. We greatly appreciate your valuable feedback. Below, we provide our responses to your comments:\n\n## **[Weakness 1] Impact of Calibrating Dataset Types, Scales, and Adding Noise on CoMe**\n>Thanks very much for your insightful suggestion. Both the BI/SBI metrics and the channel sensitivity metric indeed rely on a calibration dataset. We present the impact of the calibration dataset size and type on CoMe’s performance in Figure 8 and Figure 11, and provide analysis in Lines 324–340. As shown in Figure 11, CoMe is not highly sensitive to the calibration dataset. Different calibration datasets lead to only minor fluctuations in performance, and the average accuracy is largely unaffected in terms of reasoning ability. \nHowever, in terms of PPL, using pre-trained data (e.g., Wikitext-2, C4, PG19) as the calibration set yields better results than using instruction-tuning datasets in QA format (e.g., MMLU, Alpaca).\nAs shown in Figure 8, when the sample size of the Wikitext-2 calibration dataset exceeds 128, the pruning performance of CoMe becomes stable.\n\n>To further investigate the impact of dataset noise on CoMe’s performance, we conducted an additional experiment in which we randomly shuffled the word order in a proportion of sentences in the calibration set, thereby disrupting the linguistic structure. Using the default experimental settings from the main paper (256 calibration samples), we found that shuffling up to 60\\% of the sentences does not significantly affect CoMe’s performance. However, when more than 80\\% of the sentences are shuffled, the average accuracy drops by more than two points.\n\n| noise ratio | ARC-c | ARC-e | HellaS | OBQA  | PIQA  | WinoG | MMLU  |  Avg  | Wiki-2 |  C4   |\n|:-----------:|:-----:|:-----:|:------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|\n|     0.0     | 35.24 | 54.46 | 56.56  | 35.40 | 68.88 | 61.17 | 25.50 | 48.17 | 16.53 | 19.93 |\n|     0.1     | 34.98 | 54.88 | 56.29  | 34.40 | 69.37 | 60.22 | 25.65 | 47.97 | 16.30 | 20.02 |\n|     0.2     | 34.13 | 56.19 | 55.75  | 36.20 | 68.06 | 59.98 | 23.50 | 47.69 | 15.46 | 19.58 |\n|     0.4     | 34.56 | 55.35 | 54.40  | 38.00 | 68.06 | 59.19 | 26.64 | 48.03 | 15.53 | 20.57 |\n|     0.6     | 35.07 | 55.60 | 55.95  | 36.20 | 69.59 | 60.06 | 22.94 | 47.92 | 14.94 | 22.92 |\n|     0.8     | 32.00 | 52.48 | 53.68  | 34.80 | 65.89 | 60.54 | 22.92 | 46.04 | 19.14 | 26.01 |\n|     1.0     | 33.02 | 50.42 | 51.81  | 32.40 | 66.92 | 59.67 | 23.06 | 45.33 | 19.91 | 26.01 |\n\n## **[Weakness 2 \\& Question 1] Limitation of the Heuristic Parameter Preservation Ratios**\n>Thank you for your insightful comments. We fully agree that the use of heuristic parameter preservation ratios poses limitations in terms of adaptability across different structures, as also noted in our Limitations section (Lines 357–359). Our experiments in Figures 10 and 14 demonstrate that the optimal merging ratio differs between MHA and FFN, indicating that a fixed ratio may not be universally optimal. Furthermore, as shown in Figure 7, we investigated the impact of the fusion ratio hyperparameter and found that heavily prioritizing one layer while neglecting others leads to significant performance degradation. These results suggest that adopting adaptive or learned fusion ratios for different structures could potentially yield better results than our current heuristic approach. We consider this an important direction for future work and plan to explore adaptive strategies in subsequent research.\n\n## **[Weakness 3 \\& Question 2] Analysis of Computational Overhead Introduced by Hierarchical Distillation**\n>Our post-training hierarchical distillation strategy is based on the layer pruning information obtained from the first stage of CoMe. As a result, the number of optimized layers in CoMe-mp is not fixed. In CoMe-mp, each sub-step only updates the parameters for a single layer, whereas in CoMe-sp, layers that are not involved in the merging process during layer pruning (i.e., layers between merged layers) are kept frozen but still included in the training computation. This makes it challenging to directly compare the computational overhead of CoMe-sp and CoMe-mp. In Appendix D, we analyze the differences between CoMe-mp and CoMe-sp and provide a rough comparison of their computational costs. In Table 7, we further compare the number of tokens used by CoMe and single-layer updating methods. Specifically, the computational cost of CoMe-mp is 74\\% that of LLM-Streamline and 50\\% that of FuseGPT. Under our default experimental settings, for LLaMa2-7B pruned with CoMe, CoMe-sp incurs over 7 times the computational cost of CoMe-mp in terms of parameter updates and forward passes, but requires fewer training tokens (about 70\\% of CoMe-mp). Overall, CoMe-sp requires more than 5 times the computational cost of CoMe-mp, but achieves better performance (see Table 1).\nCompared to full fine-tuning, the computational cost of CoMe-mp is approximately $1/M$, where $M$ is the number of layers in the model.\n\n## **[Weakness 4] Theoretical Basis of Concatenation-Based Merging**\n>The theoretical foundation for parameter retention in CoMe is based on two main aspects, as discussed in Section 3 of our manuscript. \n\n>**First**, the DLP method demonstrates that pruning any single adjacent layer results in minimal performance degradation, which is elaborated in Line 127 (Core Issue 1). This observation led us to consider that while some important feature-processing modules may be discarded during pruning, the model’s intermediate representations can still be effectively processed by subsequent layers even after one layer is removed.\n\n>**Second**, the WSLP method, which blindly merges the weights of adjacent layers, tends to result in over-smoothed parameter distributions. This issue is discussed in Line 142 (Core Issue 2). To address this, we decompose each layer into finer-grained submodules, such as channels in the FFN structure and heads in the MHA structure. These submodules are characterized by the property that adding or removing some of them does not affect the input and output of the remaining modules. Building on this insight, we split layers into feature-mapping submodules and concatenate the submodules from adjacent layers. In this way, the input and output domain distributions of each submodule remain unchanged, and the more important submodules are preserved. This forms the basis of the “Layer as Puzzle Pieces” concept in our title, highlighting that, compared to existing layer pruning methods (DLP and WSLP), CoMe provides a novel framework for layer merging. Our approach addresses layer pruning at a finer granularity and yields a pruned model structure that is more hardware-friendly and easier to deploy.\n\n>We present ablation studies comparing WSLP and various CoMe merging schemes in Table 2, with detailed analyses provided in Section 5.4 and Appendix E. These results demonstrate that our method effectively resolves the issue of mismatched weight mapping between different layers in WSLP. Furthermore, the performance improvements over DLP indicate that CoMe is more effective than both WSLP and DLP.\n\n## **[Question 3] Generalization of CoMe**\n>In principle, CoMe can be applied to other model architectures. As discussed in our response to [Weakness 4], the CoMe approach decomposes each layer into submodules whose inputs and outputs are independent of one another, and then merges these submodules. Therefore, as long as a model architecture possesses such modularity, CoMe can be adapted for model merging, layer merging, or the fusion of other decomposable structures.\n\n>For example, with minor modifications, CoMe can be extended to convolutional neural networks (CNNs) by pruning layers through the concatenation of convolutional kernels from adjacent layers.\n\nWe hope these responses can address your concerns. If you have further questions, we would be happy to provide additional clarification."}, {"review_id": "qf8cMVf5i2", "reviewer": "Reviewer_8TsA", "summary": "This paper proposes CoMe, a structured pruning framework for large language models that improves upon existing methods by avoiding performance loss from direct layer removal and the limitations of linear layer fusion. It introduces a channel sensitivity metric, a concatenation-based merging strategy, and a hierarchical distillation approach to preserve critical information and support effective post-training recovery. Experiments across multiple models and benchmarks demonstrate consistent performance gains over prior work.", "questions": "- The evaluation focuses exclusively on multiple-choice benchmarks such as ARC, PIQA, and MMLU. Could the authors clarify whether the proposed compression method has been tested on open-ended generation tasks or real-world downstream applications?\n\n-  Could the authors clarify why this particular formulation based on expected output perturbation was chosen over more established alternatives such as gradient-based or second-order importance metrics? Additionally, has the metric been evaluated in isolation across diverse settings to assess its general applicability beyond the CoMe framework?\n\n- The channel retention ratio p seems to be set empirically. Please explain the selection principle of p and the adaptive strategy at different layers and different model sizes. If p is not properly set, how much performance will be degraded?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- The paper introduces a channel-sensitivity metric, a concatenation-based layer merge, and a hierarchical distillation scheme, providing a well-structured solution to common limitations in LLM pruning.\n\n- CoMe achieves top performance across seven benchmarks and multiple model scales.\n\n- The authors test CoMe's sensitivity to calibration data size, merge granularity, and data distribution mismatch. The method shows stable performance and favorable perplexity compared to baselines.\n\n- CoMe evaluates pruning at the channel level using a custom sensitivity metric and extends the Block Influence score across layers, improving information retention over prior techniques.\n\n---\n\n### Weaknesses\n\n- The evaluation is limited to multiple-choice benchmarks such as ARC, PIQA, and MMLU. While these benchmarks are commonly used to assess factual knowledge and reasoning, they do not capture important aspects of language modeling such as open-ended generation, contextual coherence, or task generalization. As a result, it is difficult to determine whether the proposed compression method preserves the full capabilities of the original model. Including a more diverse set of benchmarks that reflect real-world downstream tasks would provide a more comprehensive evaluation.\n\n- While the proposed channel sensitivity metric offers an intuitive formulation based on expected output perturbation when individual channels are ablated, it remains fundamentally heuristic and lacks rigorous theoretical grounding. The paper does not explain well why this specific formulation is preferable over well-established alternatives such as gradient-based (for example, Taylor or saliency) or second-order importance metrics. \n\n- Ablation studies in the paper show that the heuristic fusion ratio is not consistently effective. As reported in the authors’ own analysis (page 8), the optimal ratio varies across different layer structures, and fixed heuristics may lead to performance degradation.", "was_revised": true, "final_justification": "The author's explanation effectively addresses my concern regarding the diversity of benchmarks and the evaluation scope in Layer Pruning. I find this clarification convincing and therefore have decided to raise my score.\n\nHowever, I believe that Weakness 3 still requires additional experimental support. The current response does not fully alleviate my concerns in that regard.", "rebuttal": "Thank you for taking the time to review our manuscript. We greatly appreciate your valuable feedback. Below, we provide our responses to your comments:\n## **[Weakness 1 \\& Question 1] Benchmark Diversity and Evaluation Scope in Layer Pruning**\n>Thank you very much for your thoughtful suggestions. We agree that evaluating on open-ended generation and task generalization benchmarks would provide a more comprehensive assessment of model performance. However, in the domain of layer pruning, it is standard practice to use pretrained models as the base for pruning. As the number of layers is reduced, the model’s feature representations are often quickly degraded, making fine-grained evaluations such as open-ended generation, contextual coherence, or generalization tests less meaningful. The resulting scores tend to approach random chance. Therefore, in the layer pruning literature, two main evaluation schemes are commonly adopted. The first is measuring PPL on datasets *[1,2]* such as WikiText2 and C4, which reflects the coarse-grained language modeling capability of the pruned model. The second is to use a range of multiple-choice benchmarks (e.g., SuperGLUE, ARC, MMLU) *[3]*, which serve to assess the model’s natural language understanding ability after pruning. For a fair comparison with prior work, we report both types of evaluation results in our paper.\n\n## **[Weakness 2 \\& Question 2] Rationale for Channel Sensitivity Metric**\n>Our choice of the channel sensitivity metric is motivated by prior work and empirical analysis. Specifically, we refer to Wanda *[4]* and MOE-PRUNER *[5]*, which focus on weight-level sparsity and the importance of individual weights. In contrast, CoMe assesses channel importance by considering both weight magnitude and the influence of input activations on layer output, inspired by findings that large activations tend to have a greater impact on model output *[6]*. We provide relevant explanations in Section 4.1 of our manuscript.\n\n>Moreover, we chose not to adopt gradient-based (first-order or second-order) importance metrics for two main reasons: (1) Methods such as Magnitude and Taylor, which rely on gradients, often result in the removal of initial or final layers. To ensure fair comparison, we prohibit pruning of the first and last several layers for these methods (see Appendix A), highlighting the suboptimality of gradient-based approaches for layer pruning. (2) Gradient-based methods require additional storage for gradient information during pruning, which increases computational and memory overhead, contrary to our goal of lightweight pruning and training under resource constraints.\n\n## **[Regarding Weakness 3 \\& Question 3] Selection and Impact of the Channel Retention Ratio**\n>We propose an adaptive parameter fusion ratio formula (Equation 5), which automatically adjusts the fusion ratio between adjacent layers based on their BI scores. Layers with higher BI scores are considered more important and thus retain more parameters. Within the same layer, the FFN and MHA structures share the same fusion ratio. However, our ablation studies in Figures 10 and 14 reveal that the optimal fusion ratios differ between structures (FFN and MHA). We have addressed this issue in the limitations section (line 357) and provided further analysis on page 8.\n\n>The hyperparameter $p$ is used to control the scaling relationship for parameter retention between more important and less important adjacent layers. A larger $p$ means that more important layers retain a higher proportion of parameters. We conduct an ablation study on $p$ in Figure 7 and provide analysis in lines 312–319. Our results show that as $p$ approaches infinity, the pruned model’s performance degrades and converges to that of the DLP method. In other words, an inappropriate choice of $p$ can lead to performance as poor as DLP. Since the optimal value of $p$ varies across different models, we use hyperparameter search to determine the most suitable $p$ for each model. Currently, we do not have a universal setting for $p$ across different models, which is why we have highlighted this limitation in our manuscript.\n\n>In future work, we plan to investigate more effective and adaptive fusion ratios tailored to different structures, layers, and models.\n\nReferences:\n>*[1] Song, Jiwon, et al. \"SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks.\" Forty-first International Conference on Machine Learning.*\n\n>*[2] Pei, Zehua, et al. \"FuseGPT: Learnable Layers Fusion of Generative Pre-trained Transformers.\" arXiv preprint arXiv:2411.14507 (2024).*\n\n>*[3] Men, Xin, et al. \"ShortGPT: Layers in Large Language Models are More Redundant Than You Expect.\" Findings of the Association for Computational Linguistics: ACL 2025, edited by Wanxiang Che et al., Association for Computational Linguistics, 2025, pp. 20192–20204.*\n\n>*[4] Sun, Mingjie, et al. \"A Simple and Effective Pruning Approach for Large Language Models.\" The Twelfth International Conference on Learning Representations, 2024.*\n\n>*[5] Xie, Yanyue, et al. \"Moe-pruner: Pruning mixture-of-experts large language model using the hints from its router.\" arXiv preprint arXiv:2410.12013 (2024).*\n\n>*[6] Sun, Mingjie, et al. \"Massive Activations in Large Language Models.\" First Conference on Language Modeling, 2024.*\n\nWe hope these responses can address your concerns. If you have further questions, we would be happy to provide additional clarification."}, {"review_id": "YASYAmaips", "reviewer": "Reviewer_1RXD", "summary": "This paper introduces CoMe, a structured compression framework for large language models that integrates: a channel sensitivity metric that utilizes activation intensity and weight norms for fine-grained channel selection, a concatenation-based layer merging method and a hierarchical distillation protocol. Extensive experiments on multiple LLMs (e.g. LLaMA‑2‑7b, Qwen‑3‑4b) and three sparsity levels (10/20/30%) demonstrate that CoMe consistently outperforms existing layer‑pruning baselines.", "questions": "Please elaborate on the difference between CoMe and Laco [1].\n\n[1] Yang, Yifei, Zouying Cao, and Hai Zhao. \"Laco: Large language model pruning via layer collapse.\" arXiv preprint arXiv:2402.11187 (2024).", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "- Strengths\n  - This paper is easy to follow.\n  - The ablation experiments are very detailed.\n  - Progressive Concatenation-based Layer Merge: Instead of simply linearly weighting or averaging layer weights, authors fuse high-value channels of adjacent layers according to sensitivity to construct an optimally aligned subspace, thereby maximizing the representation capability of the original model under the same parameter budget.\n- Weaknesses\n  - Only the results of the 7B system model are provided. The conclusions may not be transferable to other larger-sized models (e.g. LLAMA3-70B).\n  - There is a lack of theoretical evidence to prove why merging is better than direct pruning.\n  - The paper does not conduct a quantitative comparative analysis between the proposed knowledge distillation method and traditional LoRA fine-tuning in terms of downstream task performance, training time, number of parameters, and computing resource consumption, which makes it impossible for readers to evaluate the effectiveness of hierarchical distillation protocol.", "was_revised": true, "final_justification": "The current manuscript has deficiencies in experimental comparisons, which make the experimental conclusions less credible and fair, so I maintain the original score. The main issues are as follows:\n\n- The computational cost comparison is not uniform (unfair): The table sets the sequence length of FuseGPT/LLM-Streamline to 2048, while that of CoMe is 512. Direct comparison under such non-uniform input lengths is unfair.\n- Lack of comparison with LoRA: The paper uses Hierarchical Distillation for fine-tuning and accuracy recovery, but does not compare with LoRA (a representative method for efficient parameter fine-tuning). LoRA is often used as a strong baseline for resource-constrained fine-tuning.", "rebuttal": "Thank you for taking the time to review our manuscript. We greatly appreciate your valuable feedback. Below, we provide our responses to your comments:\n## **[Question 1] Difference between CoMe and LaCo**\n>LaCo belongs to the category of WSLP methods, while CoMe differs fundamentally from LaCo in terms of layer selection, fusion strategy, and post-training protocol.\n\n> * **Fusion Strategy**:\nCoMe employs a concatenation-based merging at the sub-module level, whereas LaCo essentially utilizes a weighted sum of layer weights.\n\n> * **Layer Group Selection**:\nCoMe uses the SBI score to select a group of adjacent layers for merging, requiring only a single forward pass on a calibration dataset in one iteration. In contrast, LaCo sets a threshold $\\mathcal{T}$ and traverses from shallow layer index $\\mathcal{L}$ to deep layer index $\\mathcal{H}$, merging $\\mathcal{I}$ layers at a time. If the similarity between the pruning model’s output and the original model’s output is within a threshold $\\tau$, the fused layer will be adopted. Otherwise, it rejects the merging process and retries with different layers until the target number of layers is reached or all layers are traversed. LaCo is a post-hoc pruning method and, due to multiple hyperparameters, it is challenging to obtain a model with the required number of layers. Therefore, we only reproduced LaCo experiments on the LLaMA-2 series models.\n\n>* **Post-training**:\nCoMe employs a fine-grained, layer-wise post-training protocol, which operates by aligning the output distributions of individual layers during the training process. In contrast, LaCo adopts a model-wise post-training approach that aligns the logits of the original model and the pruned model through conventional knowledge distillation. Due to our resource constraints (2 × A100 40G), we only compared CoMe with layer-wise post-training methods. We were unable to reproduce LaCo’s post-training results and thus excluded it from the comparative analysis.\n\n## **[Weakness 2] Theoretical Evidence for Merging**\n>The theoretical motivation for CoMe is grounded in two key observations from previous layer pruning studies, as detailed in Section 3 of our manuscript. First, DLP directly prunes entire layers, which can result in the loss of important weights. Second, WSLP ignores the correlation between the weights of adjacent layers, and its blind weighted fusion often leads to over-smoothed weight distributions. Based on these insights, we propose a novel layer pruning framework. CoMe decomposes each layer into submodules whose inputs and outputs are independent, so that removing or adding submodules does not affect the input or output of the remaining submodules. For example, as analyzed in Appendix  C, channels in the FFN structure and heads in the MHA structure can be treated as such submodules. By concatenating these submodules, CoMe avoids weight smoothing and more effectively preserves important weights or functional components.\n\n>It is important to note that layer pruning is inherently a lossy process, and there is currently no effective theoretical method to quantify the extent of this loss. This limitation is common across many layer pruning approaches (e.g., LaCo, MKA, SLEB, ShortGPT, etc.), making the field largely empirical.\n\n>In our manuscript, we provide detailed experimental comparisons between DLP and WSLP (see Figure 3 and Figure 13), as well as between DLP and CoMe (see Figure 10 and Figure 14), clearly illustrating the effects of direct layer pruning, weighted sum fusion, and concatenation-based fusion on model performance. Ablation studies of the fusion operation in Table 2 further demonstrate that CoMe is more effective than WSLP.\n\n## **[Weakness 1] Experiment on Larger Models**\n>Due to limited computational resources (2 × A100 40G), we were unable to conduct experiments on models larger than 20B parameters. We have provided experimental results on models of various scales, including 4B, 7B, 8B, and 13B. We plan to extend our experiments to 70B and even larger models when resources permit in the future.\n\n## **[Weakness 3] Lack of Quantitative Comparison with LoRA Fine-tuning**\n>We appreciate the reviewer’s suggestion. Our method is orthogonal to LoRA fine-tuning, as the two approaches serve fundamentally different purposes. LoRA aims to reduce the number of trainable parameters through low-rank decomposition for parameter-efficient fine-tuning. CoMe’s post-training protocol is designed to rapidly align the merged layers using information obtained during pruning, thereby restoring model performance. \nAdditionally, because CoMe’s pruning process is dynamic, the number of layers and training time required for post-training also vary accordingly. \nIn our paper, we compared post-training methods that are more closely related to CoMe’s motivation (Table 1) and provided comparisons of training processes and resource consumption (Appendix B.2 and Table 7).\n\nWe hope these responses can address your concerns. If you have further questions, we would be happy to provide additional clarification."}]

Author rebuttal:

Meta-review: This paper proposes CoMe, a layer pruning framework for large language models that addresses limitations of direct layer removal and linear weight layer fusion. CoMe selects channels from adjacent layers for weight merging and applies knowledge distillation to restore performance. Most reviewers gave positive ratings to this paper. Reviewer 1RXD noted that the paper does not provide a fair comparison of computational costs and that LoRA was not included as a baseline. Other Reviewers acknowledged that most of their responses have been resolved during the rebuttal period. 

After carefully considering all reviews, the AC believes that the weakness identified by Reviewer 1RXD does not undermine the paper’s main contribution. Since this work focuses on layer pruning and merging, it is not necessary to compare with PEFT methods such as LoRA.

The authors are encouraged to incorporate all new results from the rebuttal, along with the reviewers’ suggestions—such as providing a more rigorous comparison of computational costs with other methods—into the camera-ready version.
Decision: Accept (poster)