ICML 2025

Let LLM Tell What to Prune and How Much to Prune
Mingzhe Yang * 1 Sihao Lin * 2 Changlin Li 3 Xiaojun Chang 1
Abstract
Large language models (LLMs) have revolution-
ized various AI applications. However, their bil-
lions of parameters pose significant challenges
for practical deployment. Structured pruning is
a hardware-friendly compression technique and
receives widespread attention. Nonetheless, exist-
ing literature typically targets a single structure
of LLMs. We observe that the structure units of
LLMs differ in terms of inference cost and func-
tionality. Therefore, pruning a single structure
unit in isolation often results in an imbalance be-
tween performance and efficiency. In addition,
previous works mainly employ a prescribed prun-
ing ratio. Since the significance of LLM modules
may vary, it is ideal to distribute the pruning load
to a specific structure unit according to its role
within LLMs. To address the two issues, we pro-
pose a pruning method that targets multiple LLM
modules with dynamic pruning ratios. Specifi-
cally, we find the intrinsic properties of LLMs
can guide us to determine the importance of each
module and thus distribute the pruning load on de-
mand, i.e., what to prune and how much to prune.
This is achieved by quantifying the complex inter-
actions within LLMs. Extensive experiments on
multiple benchmarks and LLM variants demon-
strate that our method effectively balances the
trade-off between efficiency and performance.
1. Introduction
Large language models (LLMs) have demonstrated excep-
tional performance in the field of natural language process-
ing (Mann et al., 2020; Thoppilan et al., 2022; Zhu et al.,
2024). Nonetheless, the billions of parameters (Zhao et al.,
2023) characteristic of LLMs substantially increase storage
requirements and computational overhead. Therefore, de-
*Equal contribution 1University of Science and Technology of
China 2RMIT University 3Stanford University. Correspondence
to: Xiaojun Chang <xjchang@ustc...