Title: Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Structured Pruning; Post Training
Abstract: Large Language Models (LLMs) excel at natural language processing tasks, but their massive size leads to high computational and storage demands.
Recent works have sought to reduce their model size through layer-wise structured pruning.
However, they tend to ignore retaining the capabilities in the pruned part. 
In this work, we re-examine structured pruning paradigms and uncover several key limitations: 1) notable performance degradation due to direct layer removal, 2) incompetent linear weighted layer aggregation, and 3) the lack of effective post-training recovery mechanisms.
To address these limitations, we propose CoMe, including a progressive layer pruning framework with a Concatenation-based Merging technology and a hierarchical distillation post-training process. 
Specifically, we introduce a channel sensitivity metric that utilizes activation intensity and weight norms for fine-grained channel selection. 
Subsequently, we employ a concatenation-based layer merging method to fuse the most critical channels in the adjacent layers, enabling a progressive model size reduction. 
Finally, we propose a hierarchical distillation protocol, which leverages the correspondences between the original and pruned model layers established during pruning, enabling efficient knowledge transfer.
Experiments on seven benchmarks show that CoMe achieves state-of-the-art performance; when pruning 30% of LLaMA-2-7b's parameters, the pruned model retains 83% of its original average accuracy.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CoMe, a structured compression framework for large language models that integrates: a channel sensitivity metric that utilizes activation intensity and weight norms for fine-grained channel selection, a concatenation-based layer merging method and a hierarchical distillation protocol. Extensive experiments on multiple LLMs (e.g. LLaMA‑2‑7b, Qwen‑3‑4b) and three sparsity levels (10/20/30%) demonstrate that CoMe consistently outperforms existing layer‑pruning baselines.\n\nSTRENGTHS AND WEAKNESSES:\n- Strengths\n  - This paper is easy to follow.\n  - The ablation experiments are very detailed.\n  - Progressive Concatenation-based Layer Merge: Instead of simply linearly weighting or averaging layer weights, authors fuse high-value channels of adjacent layers according to sensitivity to construct an optimally aligned subspace, thereby maximizing the representation capability of the original model under the same parameter budget.\n- Weaknesses\n  - Only the results of the 7B system model are provided. The conclusions may not be transferable to other larger-sized models (e.g. LLAMA3-70B).\n  - There is a lack of theoretical evidence to prove why merging is better than direct pruning.\n  - The paper does not conduct a quantitative comparative analysis between the proposed knowledge distillation method and traditional LoRA fine-tuning in terms of downstream task performance, training time, number of parameters, and computing resource consumption, which makes it impossible for readers to evaluate the effectiveness of hierarchical distillation protocol.\n\nQUESTIONS:\nPlease elaborate on the difference between CoMe and Laco [1].\n\n[1] Yang, Yifei, Zouying Cao, and Hai Zhao. \"Laco: Large language model pruning via layer collapse.\" arXiv preprint arXiv:2402.11187 (2024).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes CoMe, a structured pruning framework for large language models that improves upon existing methods by avoiding performance loss from direct layer removal and the limitations of linear layer fusion. It introduces a channel sensitivity metric, a concatenation-based merging strategy, and a hierarchical distillation approach to preserve critical information and support effective post-training recovery. Experiments across multiple models and benchmarks demonstrate consistent performance gains over prior work.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The paper introduces a channel-sensitivity metric, a concatenation-based layer merge, and a hierarchical distillation scheme, providing a well-structured solution to common limitations in LLM pruning.\n\n- CoMe achieves top performance across seven benchmarks and multiple model scales.\n\n- The authors test CoMe's sensitivity to calibration data size, merge granularity, and data distribution mismatch. The method shows stable performance and favorable perplexity compared to baselines.\n\n- CoMe evaluates pruning at the channel level using a custom sensitivity metric and extends the Block Influence score across layers, improving information retention over prior techniques.\n\n---\n\n### Weaknesses\n\n- The evaluation is limited to multiple-choice benchmarks such as ARC, PIQA, and MMLU. While these benchmarks are commonly used to assess factual knowledge and reasoning, they do not capture important aspects of language modeling such as open-ended generation, contextual coherence, or task generalization. As a result, it is difficult to determine whether the proposed compression method preserves the full capabilities of the original model. Including a more diverse set of benchmarks that reflect real-world downstream tasks would provide a more comprehensive evaluation.\n\n- While the proposed channel sensitivity metric offers an intuitive formulation based on expected output perturbation when individual channels are ablated, it remains fundamentally heuristic and lacks rigorous theoretical grounding. The paper does not explain well why this specific formulation is preferable over well-established alternatives such as gradient-based (for example, Taylor or saliency) or second-order importance metrics. \n\n- Ablation studies in the paper show that the heuristic fusion ratio is not consistently effective. As reported in the authors’ own analysis (page 8), the optimal ratio varies across different layer structures, and fixed heuristics may lead to performance degradation.\n\nQUESTIONS:\n- The evaluation focuses exclusively on multiple-choice benchmarks such as ARC, PIQA, and MMLU. Could the authors clarify whether the proposed compression method has been tested on open-ended generation tasks or real-world downstream applications?\n\n-  Could the authors clarify why this particular formulation based on expected output perturbation was chosen over more established alternatives such as gradient-based or second-order importance metrics? Additionally, has the metric been evaluated in isolation across diverse settings to assess its general applicability beyond the CoMe framework?\n\n- The channel retention ratio p seems to be set empirically. Please explain the selection principle of p and the adaptive strategy at different layers and different model sizes. If p is not properly set, how much performance will be degraded?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CoMe, a novel structured compression framework for Large Language Models (LLMs) that addresses key limitations in existing pruning and knowledge distillation methods. The authors propose a progressive layer pruning framework with a Concatenation-based Merging technology and a hierarchical distillation post-training process to reduce model size while minimizing performance degradation\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n- The paper empirically demonstrates its superiority in preserving model capacity and hierarchical knowledge.\n- CoMe integrates pruning and distillation into a unified, progressive framework, which is crucial given that previous approaches often handle these stages separately, leading to suboptimal knowledge transfer.\n- The \"Rethinking the Layer-based Structured Pruning\" section (Section 3) provides strong motivation for their proposed approach.\n- The introduction of a channel-level sensitivity metric provides a more principled basis for fine-grained pruning.\n\n**Weaknesses**\n- The channel sensitivity metric and BI/SBI scores rely on a calibration dataset. The impact of the size and representativeness of this dataset on CoMe's performance is briefly discussed (Fig. 8 ), but further analysis of its sensitivity to dataset characteristics could be beneficial (eg, impact of adding noise, reasoning capabilities, etc).\n- While the concatenation-based strategy is effective, its reliance on heuristic parameter preservation ratios limits adaptability across diverse model architectures.\n- The computational overhead introduced by hierarchical distillation, especially CoMe-sp needs to be discussed,\n- Limited exploration on the interpretability of the retained parameters and their roles post-concatenation.\n\nQUESTIONS:\n- Could an adaptive or learned ratio improve results further, and has this been explored?\n- Have the authors considered the computational cost implications in real-world deployment scenarios, particularly concerning hierarchical distillation?\n- Can the concatenation-based merging approach be generalized or adapted to non-transformer architectures or other modalities beyond NLP?\n\nLIMITATIONS:\nSee strengths and weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper introduces CoMe, a new framework for structured compression of Large Language Models that addresses the limitations of traditional layer pruning and linear weight merging. Instead of removing layers directly, CoMe employs a channel sensitivity metric to identify important channels based on activation intensity and weight norms. These channels are then merged via concatenation across adjacent layers, preserving useful information and enabling progressive pruning.

STRENGTHS AND WEAKNESSES:
Strengths:

1.	This paper focused on an important problem of structured pruning for model compression.
2.	This paper propose a framework to iteratively merge layers.

Weakness:

1.	The concatenation-based weight merge process in Figure 5 doesn’t seem make sense to me. The concatenation disrupts the channel index of original weight and new weight. Then the new input completely mismatches the original input. I think there lacks some theoretical explanation or motivation about the effectiveness of this new algorithm and why this should be better than others.
2.	Please check “Question”.

QUESTIONS:
1.	All figure used raster image instead of vector image.
2.	I’m confused about the concept of “adjacent layers”. For example, if W^l refer to one up_proj, do W^(l+1) refer to next transformer block’s up_proj or this block’s down_proj?
3.	Why does it called “Hierarchical” Distillation Strategy? In Table 1, I found “sp” performs better. And Equation 8 includes all KL divergence in one loss. I didn’t find any hierarchical distillation.
4.	m in Figure 1 is just 1 in practical right? Because only two layer are merged in one time?

LIMITATIONS:
Addressed
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-02