ICML 2025

SkipGPT: Each Token is One of a Kind
Anhao Zhao 1 2 Fanghua Ye 3 Yingqi Fan 1 Junlong Tong 1 4 Jing Xiong 5 Zhiwei Fei 6 Hui Su 7 Xiaoyu Shen 1
Abstract
Large language models (LLMs) achieve remark-
able performance across tasks but incur substan-
tial computational costs due to their deep, multi-
layered architectures. Layer pruning has emerged
as a strategy to alleviate these inefficiencies, but
conventional static pruning methods overlook two
critical dynamics inherent to LLM inference: (1)
horizontal dynamics, where token-level hetero-
geneity demands context-aware pruning decisions,
and (2) vertical dynamics, where the distinct func-
tional roles of MLP and self-attention layers ne-
cessitate component-specific pruning policies. We
introduce SkipGPT, a dynamic layer pruning
framework designed to optimize computational
resource allocation through two core innovations:
(1) global token-aware routing to prioritize crit-
ical tokens and (2) decoupled pruning policies
for MLP and self-attention components. To miti-
gate training instability, we propose a two-stage
optimization paradigm: first, a disentangled train-
ing phase that learns routing strategies via soft
parameterization to avoid premature pruning de-
cisions, followed by parameter-efficient LoRA
fine-tuning to restore performance impacted by
layer removal. Extensive experiments demon-
strate that SkipGPT reduces over 40% model pa-
rameters while matching or exceeding the perfor-
mance of the original dense model across bench-
marks. By harmonizing dynamic efficiency with
preserved expressivity, SkipGPT advances the
practical deployment of scalable, resource-aware
LLMs. Our code is publicly available at: https:
//github.com/EIT-NLP/SkipGPT.
1Ningbo Key Laboratory of Spatial Intelligence and Dig-
ital Derivative, Institute of Digital Twin, Eastern Institute of
Technology, Ningbo 2Southwest Jiaotong University 3Tencent
Inc.
4Shanghai Jiao Tong University 5The University of Hong
Kong
6Nanjing University
7Meitu...