Title: Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Structural Pruning

Large Language Models

Sparsity Distribution Optimization
Abstract: Structural pruning enhances hardware-agnostic inference efficiency for large language models (LLMs) yet often fails to maintain comparable performance. Local pruning performs efficient layer-by-layer compression but ignores global topology. Although global pruning aims to identify an optimal sparse model, intuitive methods typically adopt a two-stage paradigm that first evaluates substructure saliency and then applies global pruning, which ignores inter-structure dependencies and fails to achieve end-to-end optimization. To address these limitations, we propose Týr-the-Pruner, an efficient end-to-end search-based global structural pruning framework. This framework constructs a supernet by repeatedly applying local pruning across a range of sparsity ratios to each layer in an LLM, with the core goal of determining the optimal sparsity distribution under a target overall sparsity ratio. Concretely, we introduce an effective local pruning and an expectation error accumulation approach to improve supernet construction. Furthermore, we employ an iterative prune-and-search strategy with coarse-to-fine sparsity granularity to ensure efficient search convergence. Experimental results show that Týr-the-Pruner achieves state-of-the-art structural pruning, retaining 97% of the dense model's performance while removing a challenging 50% of Llama-3.1-70B's parameters.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Týr-the-Pruner, a global structural pruning framework for LLMs that optimizes sparsity distribution through an iterative prune-and-search approach. It achieves state-of-the-art performance with significant parameter reduction while maintaining high accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The method overcomes the limitations of conventional two-stage pruning pipelines by integrating a locally pruned supernet with a global evolutionary search that is guided by expected error accumulation.\n2. The proposed method is extensively evaluated on LLMs ranging from 7B to 70B parameters and achieves up to 97% of original accuracy even with 50% sparsity. It outperforms strong baselines in both perplexity and downstream QA tasks, demonstrating robustness and generalizability.\n\nWeaknesses:\n1. There are instances of carelessness in writing. For example, in line 71, the phrase “search the optimal sparsity distribution” is used, but the final letter ‘n’ in “distribution” is not bolded consistently with the rest of the word.\n2. The search process incurs significantly higher computational cost compared to other structured pruning approaches such as LLM-Pruner and SliceGPT. Although the authors acknowledge this limitation and mention plans to improve efficiency in future work, the current method may be less practical in resource-constrained environments due to the heavy search overhead.\n\nQUESTIONS:\n1.\tIn Equation (3), the Taylor expansion of the error function $E=\\left \\| \\mathbf{XW-X\\widehat{W} } \\right \\| _2^{2}$ is expressed in terms of derivatives with respect to $\\mathbf W$. However, $\\mathbf W$ is fixed and the actual variable is either $\\delta \\mathbf W$ or equivalently $\\widehat{\\mathbf W}$. Could the authors clarify this?\n2.\tEquation (6) introduces two key hyperparameters, $\\alpha_l$ and $\\beta$. How sensitive is the final performance to different choices of $\\alpha_l$ and $\\beta$? Can the authors provide empirical results or practical guidance on setting these parameters?\n3.\tHow does the search cost compare quantitatively to existing structured pruning methods such as LLM-Pruner and SliceGPT?\n4.\tCould the approach be integrated with other compression methods like quantization or unstructured pruning? If so, have the authors conducted experiments or analyses to evaluate the compatibility or additive effects of such combinations?\n\nLIMITATIONS:\nLimitations are discussed. It would be better if the authors could optimize search time cost", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-14"}

comments: SUMMARY:
This paper proposes a new global structural pruning approach called Tyr-the-Pruner. This method utilizes a supernet to determine the optimal sparsity distribution under a target overall sparsity ratio. An iterative prune-and-search strategy with coarse-to-fine sparsity granularity is used to reduce the search cost. The experimental results show that the proposed method can achieve state-of-the-art results.

STRENGTHS AND WEAKNESSES:
Strengths:

1. This paper is technically sound and easy to understand.

2. The paper can discard 50% of channels without sacrificing much accuracy, which achieves state-of-the-art results.

3. The overall method shows some novelty.

Weaknesses:

1. Eq.4 is not very easy to understand. What is the intuition behind Eq.4? What does ~p mean?

2. How does random error accumulation and abandoned error accumulation do in Lines 149-150?

3. The authors use evolutionary search, which has many operations not optimized for GPU. What is the total search cost compared to sparsity-aware training methods and post-training sparsity methods?

4. Is this method comparable to or exceeding the sparsity-aware training methods?

5. You use 4 iterations during search. What if you use fewer or more iterations? Will that influence the search results and search time?

QUESTIONS:
See weaknesses above.

LIMITATIONS:
yes.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-16

comments: SUMMARY:
This paper proposes a method for choosing the optimal sparsity values for different layers under the structured pruning setup. This is done by first pruning different layers to different sparsity levels, and then choosing the optimal sparsity allocation.

STRENGTHS AND WEAKNESSES:
I think this is an important problem to study, and I commend the authors for their work. The numerical experiments show the benefits of this type of sparsity allocation which is positive. However, I also have some concerns. 

1- I believe the models that do not have uniform sparsity will need custom code, which will limit the application of this method in practice. Specially these custom models need to be implemented in serving engines such as vLLM and SGLang, which are used in practice for serving. This is missing from the paper.

2- The authors provide inference speed results for the models from their method, but how does this compare to a uniformly pruned model?   A non-uniform model will require invoking several different kernels on the GPU, which will create inference time overheads.

3- Are the accuracy numbers reported after distillation using (6)? How about the baselines? The consistency of the baselines need to be discussed. 

4- What does (5) mean? Can you please provide some intuition?

5- (4) is very close to OBS/OSSCAR/ZipLM. Some discussion is needed here.

QUESTIONS:
See the comments above.

LIMITATIONS:
See the comments above.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-22

comments: SUMMARY:
This paper introduces an efficient end-to-end search-based global structural pruning framework for LLMs. It proposes a method to construct a supernet by applying local pruning across various sparsity ratios to each layer of an LLM, aiming to identify the optimal sparsity distribution under a target overall sparsity ratio. The framework employs an iterative prune-and-search strategy with coarse-to-fine sparsity granularity to ensure efficient search convergence. The proposed method achieves state-of-the-art structural pruning results, retaining 97% of the dense model’s performance while removing 50% of Llama-3.1-70B’s parameters.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The paper is well-written and well-motivated. 
2. The proposed method seems to be novel.
3. The experimental results presented are promising, showing the effectiveness and robustness of the proposed framework.

Weaknesses:
1. The proposed method requires an additional calibration dataset, however, some baselines do not rely on such a dataset, this makes the comparison unfair.
2. The method part could be polished to facilitate comprehension.

QUESTIONS:
Given that the matrix \hat{W} shares the same dimensionality as W, how can this sparser \hat{W} speed up inference? I have this concern because GPUs are optimized for tensor calculation and \hat{W}X and WX require the same number of computation operations.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02