SkipGPT: Each Token is One of a Kind
Authors: Anhao Zhao
Fanghua Ye
Yingqi Fan
Junlong Tong
Jing Xiong
Zhiwei Fei
Hui Su
Xiaoyu Shen
Venue: ICML 2025 poster (2025)
Keywords: compression
pruning
layer skipping
efficiency
large language models
model optimization
Primary area: deep_learning->large_language_models
TL;DR: We propose SkipGPT, a dynamic pruning framework that adapts to token complexity, decouples MLP and attention, and uses a two-stage training paradigm, achieving 40% fewer parameters while preserving or improving performance.

Abstract: Large language models (LLMs) achieve remarkable performance across tasks but incur substantial computational costs due to their deep, multi-layered architectures. Layer pruning has emerged as a strategy to alleviate these inefficiencies, but conventional static pruning methods overlook two critical dynamics inherent to LLM inference: (1) *horizontal dynamics*, where token-level heterogeneity demands context-aware pruning decisions, and (2) *vertical dynamics*, where the distinct functional roles of MLP and self-attention layers necessitate component-specific pruning policies. We introduce **SkipGPT**, a dynamic layer pruning framework designed to optimize computational resource allocation through two core innovations: (1) global token-aware routing to prioritize critical tokens and (2) decoupled pruning policies for MLP and self-attention components. To mitigate training instability, we propose a two-stage optimization paradigm: first, a disentangled training phase that learns routing strategies via soft parameterization to avoid premature pruning decisions, followed by parameter-efficient LoRA fine-tuning to restore performance impacted by layer removal. Extensive experiments demonstrate that SkipGPT reduces over 40% model parameters while matching or exceeding the performance of the original dense model across benchmarks. By harmonizing dynamic efficiency with preserved expressivity, SkipGPT advances the practical deployment of scalable, resource-aware LLMs. Our code is publicly available at: https://github.com/EIT-NLP/SkipGPT.

SkipGPT: Each Token is One of a Kind
Anhao Zhao 12 Fanghua Ye
Yingqi Fan
Junlong Tong'
Jing Xiong
Zhiwei Fei
Hui Su
Xiaoyu Shen
Abstract
SkipGPT
Static Structured Pruning
Large language models (LLMs) achieve remark- able performance across tasks but incur substan- tial computational costs due their deep, multi- layered architectures. Layer pruning has emerged strategy to alleviate these inefficiencies, but conventional static pruning methods overlook two critical dynamics inherent to LLM inference: (I) horizontal dynamics, where token-level hetero- geneity demands conte pruning decisions, and (2) vertical dynamics, where the distinct func- tional roles of MLP and self-attention layers ne- cessitate component-specifie pruning policies. We introduce SkipGPT, dynamic layer pruning framework designed to optimize computational resource allocation through two core innovations: (1) global token-aware routing to prioritize crit- ieal tokens and (2) decoupled pruning policies for MLP and self-attention components. To miti- gate training instability, we propose two-stage optimization paradigm: first, disentangled train- ing phase that learns routing strategies via soft parameterization to avoid premature pruning de- LoRA
"the"
"and"
"the
"2"
"and"
Redundant
Important
Self Attention
FFN
Figure I. An overview of SkipGPT. Unlike conventional static structured pruning, SkipGPT dynamically prunes layers by con- sidering both horizontal and vertical dynamics. In horizontal dynamics, different tokens receive varying computational alloca- tions. vertical dynamics, the MLP and attention modules decoupled to account for their distinct roles within each layer.
1. Introduction
Large language models (LLMs) are built on a layer- wise Transformer architecture, where each layer consists a self-attention mechanism followed by a multi-layer percep- (MLP) (Vaswani et al., 2017). Scaling up model size has driven significant breakthroughs across wide range of tasks (Brown, 2020; Bommasani et al., 2022; Wei et al., 2023; Zhao et al., 2024; Xin et al., 2025; Chen et al., 2025). However, this progress comes at a steep computational cost, requiring vast resources for inference (Chowdhery et al., 2022; Wan al., 2024; OpenAl al., 2024). In contrast, the human desinte its 100 trillion synaptic connections, far surpassing
cisions, followed by parameter-effie fine-tuning to restore performance impacted by layer removal. Extensive experiments demon- strate that SkipGPT reduces over 40% model pa- rameters while matching exceeding the perfor- mance of the original dense model across bench- marks. By harmonizing dynamic efficiency with preserved expressivity, SkipGPT advances the practical deployment of scalable, resource-aware I.LMs. Our code is publicly available at: https: / //github.cc com EIT SETTTNLPPSKiPCC SKiPGPT.
the largest LLMSs operates operatess efficiently just 30 watts of power (Bartol et al., 2015; Samsi et al., 2023). This stark disparity underscores fundamental inef- ficiency in current L.LM architectures, highlighting the gap between artificial intelligence and human cognition.
'Ninhoo Key Lahoratory of Spatial Intelligence and Dig- ital Derivative, Institute of Digital Twin, Eastern Institute of Technology, Ningbo 'Southwest Jiaotong University 'Tencent Inc. "Shanghai Jiao Tong University 'The University of Hong Kong "Nanjing University 'Meituan Inc.. Correspondence to: Fanghua Ye <fanghua.ye.21 @ com,, Xiaoyu Shen <xyshenn eitech.edu.en>.
Given their sequential layer-wise structure, LLMs struggle to fully leverage parallelism, even with abundant compu- tational resources. This limitation makes layer pruning crucial strategy for accelerating inference and improving efficiency (Men et al., 2024; Kim et al., 2024; Gromov et al., 2024; Chen et al., 2024b). While existing pruning meth- ods offer some improvements, they often overlook two key aspects of pruning dynamics (see Figure 1):
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

SkipGPT: Each Token is One of Kind
1. Horizontal Dynamics: Different tokens in an input se- quence require varying levels of computation. Current methods either allocate resources to the top-k most rel- evant tokens per layer (Raposo et al., 2024; Zeng et al., 2023) or enforce a fixed computation ratio across all tokens (Jiang et al., 2024). These rigid approaches fail to adapt to token complexity, leading to suboptimal ef- ficiency. To address this, we introduce global sparsity mechanism, allowing computation budgets to be flexi- bly distributed across the entire forward pass rather than imposing fixed layer-wise or token-wise constraints.
Training Paradigm. First, in Router Tuning, we freeze the model parameters and optimize only the router, allowing it to identify the most critical computations without disrupt- ing the model's pretrained knowledge. Second, in LoRA Fine-Tuning, we freeze the router and fine-tune the model using LoRA to compensate for any performance degrada- tion caused by pruning. Our results establish SKipGPT as a highly effective pruning strategy model to fully restore its performance, even surpassing the original model, despite a 40% reduction in parameters.
the pruned
Furthermore, since router tuning does not modify the pre- trained model parameters, allows for a direct analysis of module importance in the original model. Through detailed router behavior analyses, we uncover two key insights: (!) Attention modules exhibit greater redundancy than MLP modules. (2) As context length increases, later tokens de- mand more attention computation bui less ML.P processing. These findings highlight inherent inefficiencies in current large model architectures, providing valuable insights for future architectural design and inference optimization.
Vertical Dynamics: The MLP and self-attention com- ponents within each layer serve distinct functions, yet most pruning methods treat them uniformly. Research suggests that MLPs function like localized neural pro- cesses, capturing task- task-specific interactions (Geva et al., 2021; Meng et al., 2023; Merullo et al., 2024), whereas attention mechanisms resemble higher-level cognitive functions, managing contextual relevance and informa- tion flow (Olsson et al., 2022; Kobayashi al., 2020). In- spired by how the human brain activates different regions for different tasks, we propose decoupling the pruning of MLP and self-attention, enabling more targeted and efficient computation reduction.
2. Motivation
Measuring module importance using cosine similarity Recent work has demonstrated that the cosine similarity between modulc's input and output serves as a reliable metric for evaluating the importance of each module (Men et al., 2024; Gromov et al., 2024). Notably, these studies often define a "module" as an entire Transformer layer. To enable more fine-grained analysis, we refine the definition of a module to refer to either an MLP block or an attention block within a layer. The underlying hypothesis of using cosine similarity is that redundant modules generate out- puts that closely resemble their inputs, indicating minimal transformation. Conversely, important modules substan- tially alter their inputs, suggesting they play a critical role in the model and should be preserved. Prior approaches typically average the cosine similarity over all tokens to derive general importance metric for each module. How- ever, this aggregation may obscure the variability of module importance across different tokens and contexts. To better understand this variability, we analyze the cosine similarity of each module at the token level. Specifically, the cosine similarity Ct, of the th module at token t computed as:
To achieve dynamic pruning, few recent works have ex- plored adaptive computation methods that introduce router at each Transformer layer (Zeng et al., 2023; Raposo et al., 2024; Jiang et al., 2024). This router functions as a decision module, determining whether specific network units should be executed or skipped during inference. However, these approaches typically optimize the router and model param- eters simultaneously in a joint training paradigm, similar to offturee-f-Expertt. (MoE) (Lepikhin et al., 2020; Cai et al., 2024; Zhu et al., 2024). However, this method fails to account for fundamental difference between pruning and pretraining- the pruming context, the router starts from random initialization, while the model parameters have already converged to an optimal or locally optimal distri- bution through extensive pretraining. This mismatch may make joint training unstable and prevent dynamic pruning from reaching its full potential.
In this work, we first provide empirical evidence demonstrat- ing the significance of both horizontal dynamics and vertical dynamics in LLMs. To incorporate these two dynamics, we propose SkipGPT, a novel upproach that dynamically prunes layers on a per-token basis, adapting the pruning pro- cess to the complexity of each token. Furthermore, SkipGPT decouples the MI. and self-uttention components within each layer, enabling more granular control over which parts of the model are pruned, thereby optimizing both computa- tional efficiency and model performance. To fully unlock the potential of dynamic pruning, we introduce a Two-stage
C, ||
(1)
where || ||2 is the L2-norm and Vi,t denotes the hidden state before module i at token t. To illustrate how module importance varies, we conduct a case study using randomly selected sentence from the BookCorpus dataset (Zhu et al., 2015). We analyze cosine similarity distributions across 15 consecutive tokens in LLaMA-2-7B (Touvron et al., 2023a), with the results visualized in Figure 2.
2

---

SkipGPT: Each Token is One of Kind
1.0
MLP 32
0.00
0.01
0.07
0.10
0.04
0.05
0.12
0.04
0.01
0.08
0.08
0.04
0.05
0.05
0.09
Attn 32
0.94
0.97
0.97
0.96
0.95
0.94
0.97
0.97
0.96
0.96
0.94
0.96
0.96
0.95
0.96
MLP 31
0.77
0.80
0.85
0.81
0.75
0.75
0.85
0.68
0.58
0.74
0.78
0.66
0.67
0.67
0.70
Alln
0.99
0.99
0.99
0.98
0.95
0.92
0.98
0.98
0.98
0.98
0.99
0.99
0.97
0.97
0.95
0.8
MLP 30
0.35
0.87
0.88
0.89
0.90
0.87
0.90
Attn 30
1.00
1.00
0.99
0.97
0.99
0.98
0.99
0.99
0.97
0.98
-0.6
MLP
0.32
0.50
0.50
0.59
0.51
0.31
0.43
0.38
0.34
0.44
0.49
0.50
0.42
0.42
0.38
Attn
0.92
0.77
0.67
0.63
0.18
0.68
0.44
MLP
0.56
0.56
0.46
0.59
0.29
0.53
0.40
0.46
0.60
0.65
0.57
0.60
0.50
0.51
0.54
00.
Attn
0.82
0.80
0.77
0.77
0.88
0.77
0.64
0.70
0.71
0.75
0.66
0.77
0.76
0.72
0.74
MLP
0.52
0.58
0.59
0.67
0.15
0.63
0.41
0.56
0.70
0.71
0,64
0.60
0.58
0.62
0.66
Attn
0.92
0.89
0.64
0.86
0.69
0.76
0.51
0.75
0.79
0.79
0.83
0.58
0.68
0.87
0.87
0.2
MLP
0.31
0.51
0.59
0.42
0.37
0.54
0.42
0.56
0.70
0.66
0.58
0.57
0.45
0.46
0.51
Attn
0.06
0.12
0.09
0.29
0.17
0.34
0.26
0.38
0.37
0.31
0.21
0.24
0.30
0.35
0.23
MLP
0.21
0.50
0.45
0.49
0.71
0.66
0.72
0.70
0.72
0.62
0.63
Attn
0.01
0.09
0.03
0.12
0.22
0.17
0.36
0.05
0.0
that
had
been
meg
plan
when
she
got
him
dressed
earlier
Figure 2. Token- Wise Cosine Similarities Across Modules in LLMMA-2-7, which consists of 32 layers, corresponding to 64 modules total. Due space constraints, we showease only the results for the initial and final modulcs. Higher values indicate greater redundancy.
The Necessity Vertical Dynamics Existing pruning methods, whether dynamic or static, typically treat an entire transformer layer as the smallest pruning unit. However, as illustrated in Figure 2, we find that within the same layer, the cosine similarity distributions of attention and MLP modules can vary significantly. For example, in the second layer, the cosine similarity of most tokens in the attention module ranges from 0.1 to 0.4, while the MLP module predominantly falls within the range of 0.4 to 0.7. This indicates that for this layer, the MLP module is almost universally more redundant than the attention module. In the final layer, however, the situation is entirely reversed: all attention modules exhibit cosine similarity values between 0.9 and 1, while MIP modules fall within a completely different range, from 0 to 0.2, making the MLP module far more critical than the attention module this stage. These findings reveal that even within the same layer. redundancy levels between attention and MLP can vary and shift across layers, underscoring the necessity of decoupling attention and MLP modules for more effective pruning.
these assumptions not hold in practice. First, module importance varies across layers. For instance, at cosine similarity threshold of 0.8, the attention module in layer 31 is redundant for 15 tokens, whereas in layer 4, it is redundant for only 3 clclearly disproving uniform impor- tance. Second, token importance is not uniform within a sequence. our case study, we analyze the number of mod- ules with a cosine similarity below 0.6 concisiidgiing them significant for each token. The results reveal that the token "plan" is associated with 13 important modules, while "an" has only 8, confirming that different tokens require varying levels of computation.
3. SkipGPT: Dynamic Layer Pruning
After establishing the necessity horizontal and verti- cal dynamics, we now introduce SkipGPT, our proposed framework for dynamic layer pruning. In this section, first outline the necessary preliminaries for understanding SkipGPT's optimization process. followed by an explana- tion of its sparsity mechanism, routing implementation. loss function, and finally, our two two-stage training paradigm for stable and effective learning.
The Necessity of Horizontal Dynamics Static pruning methods rely on two key assumptions: (1) the distribution of important modules is uniform across all tokens, and (2) each token is associated with the same number of important modules (Men et al., 2024; Kim et al., 2024; Gromov et al., 2024). Existing dynamic pruning methods also adopt these assumptions when setting compute budgets (Zeng et al., 2023; Raposo et al., 2024; Jiang et al., 2024). However,
3.1. Preliminaries: Gumbel-Softmax and STE
Gumbel-Softmax Reparametrization To optimize SkipGPT's dynamic pruning decisions, we formulate the pruning process as discrete optimization problem,
3

---

SkipGPT: Each Token is One of Kind
where each module (MLP or attention) is either executed or skipped based on its computed importance. However, directly optimizing discrete decisions is non-differentiable, making standard gradient- optimization infeasi- ble. The Gumbel-Softmax distribution is a continuous approximation of the categorical distribution, enabling differentiable sampling (Jang et al., 2022). This is achieved via reparameterization, which transforms discrete samples into differentiable continuous ones for gradient-based optimization. Let II, 22..... TT& represent the class probabilities of a k-class categorical distribution. To sample from this distribution, the Gumbel-Max trick (Gumbel, 1954; Maddison et al., 2015) selects the category with the highest value of log ,, + gi, where 9i are i.i.d. samples from the Gumbel distribution Gumbel(O, 1)):
sparsity is achieved through dynamic routing, where only subset of modules is selected for computation in each forward pass. Specifically, assuming that in a forward pass, an L-layer LLM (which consists of 2L modules, each layer containing one attention module and one MI.P module) pro- cesses a sequence of length S, a dense transformer would compute 2L x modules, corresponding to a sparsity of O. When sparsity greater than 0, only (1 -paarstyy x 2L S modules are computed.
Unlike previous work, which defines the compute budget by restricting computation to the top-k tokens at each layer (Raposo et al., 2024; Zeng et al., 2023) or enforcing the same sparsity for each token (Jiang et al., 2024), we allow the computational load to be dynamically allocated across both width (the number of tokens participating computation at each layer) and height (the number of modules involved in the computation for each token).
one_hot aarg max [g: + log ma)
Since arg max is non-differentiable, the Gumbel-Softman reparamctrization replaces it with softmax function, pro- ducing continuous samples approximating the categorical distribution:
3.3. Routing Implementation
To enable dynamic allocation, SkipGPT assigns a router before each individual module. Specifically, we route to- kens to two computational paths: (1) self-attention (for the attention router) or FFN modules (for the FFN router), and (2) a residual connection. The latter is computationally in- expensive, producing an output determined entirely by the input, while the former incurs a high computational cost.
tui
exp
y/
i= 1, 1. h
log Fo
j= exp
where controls the sharpness of the distribution. As O, the samples resemble one-hot vectors, recovering the origi- nal arg max operation. This differentiable approximation enables standard backpropagation for optimization.
Suppose that we have token embedding xę prior to the I-th transformer module fi, where this module can either be self-attention or an FFN. Before passing through the module fi, z first undergoes a router function, which is a simple linear projection, yielding a categorical distribution ri = WT x E IR,, where the first element represents the probability of skipping the module, and the second element represents the probability of executing it.
Straight-Througs Estimator The Straight-Through (ST) Estimator (Bengio et al., 2013) enables discrete sampling while preserving differentiability for backpropagation. In the forward pass, we use Gumbel-SSfttax to generate con- tinuous samples. To discretize them, we apply arg max, but in the backward pass, gradients computed as if using the continuous approximation. This is achieved via:
Once this categorical distribution is obtained, a natural rout- ing strategy is the Top-1 I routing. Specifically, we have:
[7[1] (fi(a)) + mi), if arg max ri = 1, m:(0] 21,
Yhard = Yhard - Ysoft detach() + Ysoft:
(4)
01+1 =
(5)
otherwise,
where Vsoft is the continuous Gumbel-Softmal sample. This ensures a one-hot output while gradients follow ysofts en- abling efficient optimization despite discrete sampling.
such that the gradients can be backpropagated. However, this routing strategy challenges precise sparsity control, as Ti is merely soft approximation of binary selection.
3.2. The Concept of Sparsity
By leveraging Gumbel-Softmax and the ST Gumbel Esti- mator, we effectively address this issue. Specifically, after applying Gumbel-Softmal and the ST Gumbel Estimator to Tt, we obtain a one-hot vector gi E {0,1)². Thus, the input to the next module is computed as:
To control the total FLOPs, we introduce the concept of sparsity, defined as the fraction of module computations (attention or MLP) skipped in a forward pass, relative to the total computations in fully dense transformer, account- ing for all layers and sequence positions. In our method,
x = gi [1] (fi(z)) + x)) + gi 00 . F/-
(6)
can be obtained via inverse transform sampling: u Uniform(O, 1), g = - -log(- log(u))-
During the forward pass, discrete binary values are sam- pled, ensuring clear pruning decisions. In the backward pass,
4

---

SkipGPT: Each Token is One of Kind
Trainable
Frozen
Router Tuning In the initial stage of training, we focus exclusively tuning the router while keeping all other model parameters frozen. This stage is highly efficient, as it requires adding only a lightweight linear layer before each module, with all router parameters combined accounting for just 0.007% of the total parameters in LLaMA2-7B. Through this stage, we achieve over 90% of the model's original performance even after discarding 25 25% of the parameters, shown in Table 1. The forward and backward passes during router tuning are illustrated in Figure 3.
Attention
Attention
Layer Norm
271
Router
0.18
Rouuer 0.22
9.78
4
Figure 3. Illustration of the forward and backward passes during the router tuning stage. In the forward pass, the router makes hard decisions either execute (1) or skip (0). In the backward pass, the gradients are propagated back using soft probabilities.
LoRA Fine- Tine-Tuning (Optional) While the router tuning stage is sufficient to preserve most of the model's perfor- mance, we provide an optional LoRA fine-tuning stage for those aiming to fully restore performance to the original model level. Low-Rank Adaptation (LoRA) (Hu et al., 2021) enables efficient refinement of L.MSs with minimal computational overhead. Previous works, such as Ma et al. (2023); Kim et al. (2024), have demonstrated LoRA's ef- fectiveness in enhancing statically pruned models. this study, we show that LoRA can also effectively recover the performance of dynamically pruned models. For a clearer understanding of the entire training process, please refer to the algorithm illustrated in Appendix B.
soft probabilities facilitate gradient propagation, allowing the router weights to be updated effectively.
With this routing design, in principle, we can train the router independently without altering the model parameters to ob- tain the optimal routing solution for a given sparsity ratio.
3.4. Loss Function
With our definition in Section 3.2, the sparsity is given by:
E.,gi10 8 x 2L
4. Experimental Configuration
4.1. Models and Benchmarks
where t is the token index, I represents the module index, L is the total number of layers in the LLM, and S denotes the length of the sequence. To meet different computational demands, we introduce a sparsity regularization term:
Models We conduct experiments utilizing LLaMA2-7B, LLaMA2-13B, and LLaMA3.1-8B (Touvron et al., 2023a;b; Dubey al., 2024).
Data During both router and LoRA tuning, we use the RedPajama- Daatua--Dat-----Sappee IT-Sample dataset (Computer, 2023)², which contains 850,000 samples (1 billion tokens) trun- cated to 4096 tokens each. This dataset serves two roles: (1) as a calibration set (100 random samples) to compute block-level significance for pruning redundant layers (static methods), and (2) as training set for dynamic methods and for recovering static method performance (the specific details of static and dynamic methods will be introduced later Section 4.2).
Csparsity IT - rr,
(8)
where |-| denotes absolute value, and T is the user-defined target sparsity. The overall loss function is then given by:
Lan Lim + a cppuriity:
where Zim is the standard language modeling loss, which represents the negative log- likelihoooo of predicting the next token average. The hyperparameter a controls the strength of the sparsity penalty in the overall loss f ?
Training Procedure Each model is trained for 10,000 steps with next-token prediction, using a batch size of 16 in both router and LoRA tuning. In the router tuning stage, we use a constant learning rate of 2e-34. Additionally, the softmax temperature r of the Gumbell-Sftmax linearly annealed from 5 to 1. In the LoRA tuning stage, the learning
3.5. Two-stage Training Paradigm
As the router starts from random initialization and the LLM has already been pretrained, direct training can lead to un- stable and suboptimal pruning decisions. To address this issue, we propose a two-stage training paradigm.
/ cttpa/aaaaccccctttat////// jamaa jama--ataa- Sample
IIf is too small, it may fail to enforce the desired sparsity. If is too large, the model may compromise the optimization of Lim- Based on our experiments, a value of 8 works well.
'A grid-search was conducted to determine that leaming rate of 2e-3 optimally ensures training stability.
5

---

SkipGPT: Each Token is One of Kind
Model
Method
Ratio
OBQA AccNorm
WinGGrndde
PIQA Acc
HeSw AccNorm
BoolQ Acc
ARC-E AccNorm
ARC-C
Asg. Acc.
WT2
PTB
Avg. PPL,
Acc
AceNorm
PPL.
PPL.
Dense
0.00%
44.20
78.07
78.93
68.69
5.47
20.83
13.15
ShonGGT
25.0%
34.80
68.43
67.68
60.77
62.17
59.34
38.40
55.94
25.42
70.97
48.20
Shortened-PPL
25.0%
33.60
52.88
70.40
55.12
61.07
55.05
29.44
51.08
11.15
49.07
30.11
Shortened- Taylor
25.0%
35.40
66.30
66,97
59,90
62.17
59.76
38.65
55.59
23.75
69.60
46.68
LLaMA2-7B
Joint Layer Drup
23.9%
38.60
72.38
70.08
67.93
40.37
65.57
44.54
57.07
29. 19
64.69
46.94
LaCo
25.0%
36.60
67.32
65.72
62.76
74.37
58.92
38.14
57.69
18.96
47.58
33.27
LLM-Pruner
25.3%
39.00
58.56
73.45
60.77
54.71
58.63
37.03
54,59
14.
65.09
39.60
SliceGPT
25.4%
66.38
10
7.56
76.29
41,93
SKipGPT-RT
25.5%
39.60
63,54
72.20
70.96
68.81
76.52
44.37
62.29
7.81
30.58
19.20
Dense
0.00%
45.20
76. 16
79.11
82.23
80.52
84.68
59.47
72.48
4,88
28.92
16.90
ShonGPT
25.0%
40.60
70.80
71.38
72.59
62.69
69.19
45.31
61.79
20.05
49.52
34,79
Shortenad-PPL
25.0%
39.40
67.17
73.12
69.31
60.27
8.45
54.62
31.54
Shortened- Taylor
25.0%
41.80
70.80
70.78
62.58
38.10
61.99
38,31
$4,90
24.46
59.34
41.90
LLaMA2-13B
Layer Drop
24.3%
73.37
74.43
13.31
27.97
LaCo
25.0%
38.80
62.75
72.74
63.11
44,46
62. 84
35.07
54.25
13.40
53.92
33.66
L.M. Prumer
24.9%
44.00
65.82
76.99
74.15
59.88
72. 60
45.82
62.75
9.82
71.49
40.66
SliceGPT
24.7%
38.60
68.43
64.42
50.97
38.87
69.36
40.36
53.00
7.43
99.89
53.66
SKipGPT-RT
25.4%
46.00
71.90
76.88
74.33
74.37
77.69
47.08
66.89
6.78
41.69
24.24
Dense
0.00%
44.80
77.51
80.03
81.95
82.14
84.85
57.59
72.69
6.24
10.58
8.41
ShonGPT
25.0%
28.00
54.14
58,76
31.50
37.77
0..05
31.40
39.95
2796.24
2799.46
2797.
Shortened-PPL
25.0%
33.60
53.51
71.87
57.98
42.08
57.07
31.74
49.69
15,00
23,86
19,43
Shortened- Taylor
25.0%
28,20
$4.06
58.87
31.53
37.77
38.05
31.31
39.97
2690.34
2793.25
2,741
LLAMA3.1-8B
Joint Layer Dmo
24.2%
33.00
55.56
61.26
42.26
37.31
35.06
30.12
42.08
32.32
51.40
41.86
LaCo
24.5%
31.20
65.11
66.16
55.77
71.13
48.65
37.03
$3.5K
30.
50.35
40,25
L.M.
24.5%
37.20
72.03
54.75
51.47
25.21
35.40
SliceGPT
24.6%
30.40
55.17
57.83
38.19
37.83
38.64
25.85
40.56
17.25
63.13
40.19
SkipGPT-RT
25.5%
44.20
75.69
78.07
76.87
74.06
82.44
53.41
69.25
16.47
26.91
21.69
Dense
0.00%
44.80
77.51
80.03
81,95
82.14
84.85
57.59
72.66 72.69
6.24
10.58
8.41
ShonGPT
40.6%
30.00
$7.62
58.54
34.55
62.29
34.81
29.10
43.84
79856.66
125507.27
102681.9 1.97
Shortened-PP
40.6%
27.00
52.41
61.48
41.74
57.13
39.69
26.45
43.70
157.01
196.04
176.53
Shortened Taylor
40.6%
28.80
59.68
37.89
58.53
43.28
78424.35
125435.41
101,929.90
LLaMA3.1-88
Joint Layer Drop
39.9%
27.60
50.12
53.48
26.65
37.86
26.35
25.77
35.40
251.52
341.25
296.39
LaCo
40.7%
27.60
$1.54
56.31
31.53
55.11
30.22
25.85
39.74
269.24
392.50
330.87
J.M-Pruner
39.9%
29.20
51.30
64.09
36.23
50.52
36.87
23.46
41.67
152.98
161.49
157.24
SliceGPT 39.9%
25. 60
51.62
53.92
30.64
37.83
31 82
22.10
36.22
143 24
183.21
163. 163.23
SKipGPT-RT
40.2%
38.00
77.53
59.80
48.05
59.65
Table 1. Comparison of SkipGPT-RT and static pruning baselines without LoRA, where the ratio denotes the proportion of parameters (averaged per token) that not participate in computations relative to the original model's total parameter count.
rate is set to 2e-4 with a warmup ratio of 0.1 and a cosine learning rate scheduler. The AdamW optimizer (Loshchilov & Hutter, 2019) with 31 0.9 and B2 0.95 is used for gradient backpropagation.
LaCo (Yang et al., 2024) collapses layers progressively from deep to shallow, employing a threshold to prevent excessive merging. Joint Layer Drop (He et al., 2024) is fine-grained variant of ShortGPT that separately removes attention and MLP layers based on the same BI metric. -LM--runer (Ma et al., 2023) selectively removes non-critical structures using gradient- criteria. SliceGPT (Ashkboos et al., 2024) shrinks the embedding dimension by replacing large weight matrices with smaller dense matrices. Unlike static pruning methods, SkipGPT considers horizontal dynamics, offering more adaptive approach.
Benchmarks Following Touvron et al. (2023a), we evalu- ate accuracy scores variety of commonsense reasoning datasets, including BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2020), HellaSwag (HeSw) (Zellers et al., 2019), Wino- Grande (Sakaguchi et al., 2021), ARC-easy (ARC-E) (Clark et al., 2018), ARC-challenge (ARC-C) (Clark et al., 2018), and OpenbookQA (OBQA) (Mihaylov al., 2018), using the in-evaluation-hurreee (Gao al., 2024). Additionally, report zero-shot PPL scores on WikiTex12 (WT2) (Merity et al., 2016) and PTB (Marcus et al., 1993).
Dynamic Pruning Baselines Dynamic pruning allows for the selective activation of model components during infer- ence. MoD (Raposo et al., 2024) employs a top-k routing mechanism to dynamically activate layers for each token. To ensure fair comparison, we introduce a new variant, MoD-D, which differs from MoD by decoupling attention and MLP modules, whereas MoD treats them as a single unit. D-LLM (Jiang et al., 2024) introduces dynamic deci- sion module at each transformer layer, determining whether a layer should be executed or skipped. Additionally, we in- clude SkipGPT-Joint which adopts a joint training paradigm instead of two-stage training. From the perspective of hori-
4.2. Baseline
Statie Pruning Baselines Static pruning involves the per- manent removal of redundant model components. Short- GPT (Men et al., 2024) removes redundant layers based on Block Influence (BI) scores. Shortened LLaMA (Kim et al., 2024) prumes layers using PPL and Taylor Expansion, result- ing in two variants: Shortened-PPI. and ciaetennd-Tayloor
6

---

SkipGPT: Each Token is One of Kind
zontal dynamics, MoD-D allocates its computation budget layer-wise, while D-LLM allocates it token-wise. Regard- ing vertical dynamics, D-LLM does not decouple attention and MLP modules. Please refer to Appendix C for detailed descriptions of these baselines.
with prior work (Men et al., 2024). This strongly supports focusing on depth pruning when designing SkipGPT.
Joint Layer Drop, the fine-grained variant of ShontGP that independently removes attention and MI.P, outper- forms ShorlGPT in most cases. This result validates our motivation for emphasizing the necessity of Vertical Dy- namics to achieve more effective pruning.
5. Results
This section provides a detailed analysis of the models after each training stage. For clarity, we define the model ob- tained after the first stage (Router Tuning) as SkipGPT-RT and the final model after the second stage (LoRA Fine- Tuning) as SkipGPT-RT-LL
SkipGPT-RT significantly outperforms Joint Layer Drop across all models in accuracy (reflecting L.LMs' ability as general-purposs task solvers) and perplexity (involving the capability to generate coherent and fluent sentences). This strongly supports our motivation for highlighting the necessity of horizontal dynamics.
5.1. Comparison Between SkipGPT-RT and Baselines
We present the accuracy and PPL for baseline pruning meth- ods and SkipGPT-RT in Table 1. For all models we evaluate, including LLaMA2-7B, LLaMA2-132 and LLaMA3. 1-8B, the attention module contains approximately half as many parameters as the MLP module. Thus, in Joint Layer Drop and SkipGPT-RT, we prune proportionally more modules to maintain consistent average purameter ratio across methods. For long-contex: tasks, the FLOPs of the atten- tion module surpass those of the MLP, making SkipGPT-RT achieve lower computational overhead compared to other baselines. For LLaMA2-7HB and ILL.MA2-3BB we report evaluation results under 25% parameter reduction setting, while for LLaMA3.1 1B, we provide results for both 25% and 40% parameter reduction. Additional results under varying pruning ratios are presented in Appendix D.
5.2. Comparison between SkipGPT-RT-L and Baselines
SkipGPT-RT-L R original model performance Fig- ure 4 shows training curves for SkipGPT-RT, SkipGPT- RT-L, static pruning methods with LoRA, and joint router+LoRA training for dynamic pruning for LLaMA2- 7B and I.LMM2-13BB Results are summarized in Table 2. Notably, SkipGPT-RT achieves performance comparable to the best fine-tuned baseline using router tuning alone. After LoRA fine-tuning, our method not only fully restores the model's performance the original level but even surpasses it (without fine-tuning), ranking second only to directly ap- plying LoRA to the original model for LLaMA2-7B. For those aiming to quickly build a high-performing pruned model, router tuning is highly effective. Meanwhile, to fully match or exceed the original model, LoRA fine-tuning is an excellent choice. It is worth reiterating that the entire process for all models requires only single A800 GPU.
The results demonstrate that with router tuning alone (with- out modifying model parameters), SkipGPT-RT signifi- cantly outperforms baseline methods. Specifically, for LLaMA2-7B and LLaMA2-13B, SkipGPT-RT retains over 90% of the dense model's ' performance even under 25% parameter pruning. For LLaMA3. I-BB, it retains over 95% performance at 25% pruning and over 80% performance at 40% pruning -a a level at which nearly all baselines col- lapse catastrophically. Notably, while router tuning involves training the router parameters, unlike static pruning, which permanently removes layers based on predefined metrics, its cost is minimal. This is because the router parameters constitute less than 0.01% % of the total model parameters and converge rapidly (Figure 4). In practice, the tuning process requires only a single A800 (80GB) GPU and com- pletes within four hours. Furthermore, the results in Table highlight three key observations:
The Effectiveness of Two-Stage Training Paradigm We designate SkipGPT-Joint, which refers to the variant where both the router parameters and model parameters are trained simultancously, as the ablation baseline for the two-stage training paradigm. As shown in Table 2, its performance is significantly worse than SkipGPT. While may seem intu- itive that directly adopting MoE's joint training paradigm could outperform two-stage training-sine the router and LoRA parameters can gradually adapt to each other's s rep- resentations the results tell a different story. Not only does SkipGPT-Join underperform, but dynamic pruning methods following the joint training paradigm consistently fail achieve satisfactory results. This finding reinforces our argument in Section 3.5: in the joint training paradigm, the randomly initialized router forces model parameters to adapt early on to a suboptimal, random routing strategy. This misalignment disrupts the model's established parame- ter distribution, making it increasingly difficult for the router to identify critical modules. Over time, this leads to a rein- forcing feedback loop: as the model adapts to poor routing,
Layer-pruning methods-sun as ShortGPT, LaCo, Short- ened LLaMA, and Joint Layer Drop- generallyyll match or surpass embedding dimension reduction approaches like and SliceGPT. This suggests that LLMs have greater redundancy in depth than width, aligning
7

---

SkipGPT: Each Token is One of Kind
Model
Method
Ratio
OBQA AccNorm
WinoGrande
PIQA Acc
HeSw AccNorm
BoolQ Acc
ARC-E AccNorm
ARC-C AccNorm
Avg. Acc.f
WT2
PTB
Avg. PPL.
Acc
PPL.
PPT.
Dense
0.00%
44.20
74.19
78.07
78.93
71.62
52.47
5.47
13.15
LoRA LoRA
0.00%
44.80
74.27
78.02
78.96
79.02.
81.73
53.07
69.98
5.48
20.58
13.03
ShonGPT. LoRA
25.05
39.40
71.74
71.38
70.12
73.91
70.37
41.72
62.66
8.45
35.96
22.21
Shortened-PPP. LoRA
25.0%
36.40
57.38
74.59
64.58
63.21
69.70
39.16
57.86
7.71
31.26
19.49
Shortened Taylor, LoRA
25.0%
37.20
70.24
71.38
70.38
72.35
70.24
43.43
62.17
8.38
33,60
20.99
Joint Layer Drop LoRA
23.9%
41.40
72.38
74.32
71.25
73.85
75.17
45.99
64.91
9,04
32.49
20.77
LLaMA2-7B
LaCo, LoRA
25.0%
39.20
70.40
70.95
69:40
76.73
69.74
42,66
62.73
8,38
28.77
18.58
LoRA
25.3%
73.50
54.65
58.59
37.03
54,73
14.10
SliceGPT, LoRA
25.4%
38.60
67,88
72.58
69. 34
71.13
75.29
45.39
62.89
7.19
38.91
23.05
MoD-D
25.0%
28.00
50.28
69.86
58.80
62.29
72.81
43.34
55.05
33.55
125.62
79.59
D-LLM
25.6%
25.60
52.88
51.14
40.81
57.91
38.09
27.13
41,95
34.75
143.85
89.30
SkipGPT Joint
25.3%
24.40
50.83
50.98
62. 81
43.36
66.50
38.23
48. 16
10.12
346.05
178.09
SKipGPT-RT-L
25.5%
44,00
74.90
78.24
77.80
77.58
81.40
52.56
69.50
5.82
21.26
13.54
Dense
0.00%
45.20
76.16
80.52
$4.68
59.47
4.88
28,92
16.90
Dense, LoRA LoRA
0.00%
45.20
76.24
11
82.26
80.06
84
59.72
72.51
4,80
28.90
16.85
ShonGPT, LaRA
25.0%
42.20
74.90
74.54
76.60
64.37
76.94
49.74
65.61
6.78
34.48
20.63
Joint Layer Drop. LoRA
24.3%
44,80
73.56
77.48
77.20
77.83
79.97
51.96
68.97
5:58
27.34
16.46
LLaMA2-13B
LaCo, LoRA
25.0%
40.60
65.98
76.99
72.19
67.37
75.63
45.31
63.44
6.81
29.53
18.17
LLM-Prruer. LoRA
24.9%
65.82
76.999
74.13
72.64
45.82
9.83
71.49
40.66
MoD.D
25.0%
37.20
67.25
71.49
$1.83
54,13
78.24
$0.34
58.64
39.31
39.69
39.50
D.LLM
28,00
$7.06
60.35
62.78
35.92
13:09
SkipGPT-Joint
25.2%
38.00
62.90
60.83
71.44
68.32
66.50
46,93
59.27
8.49
27.97
18.23
SkipGPT RT
25.4%
46.20
76.32
79.38
82.14
80.31
83.88
57.25
72.21
5.00
28.59
16.80
Dense
0.00%
44.80
77.51
80.03
81.95
82,
84.85
$7.59
72,69
6.24
10.58
8.41
Dense, LoRA
0.00%
77.68
80.22
82.23
$7.89
72.81
6.13
8.29
ShonGPT, LoRA
25.0%
38.40
70.96
73.94
69.23
72.05
68.01
43,86
62.35
11.13
16.64
13.89
Shortened-PPL.. LoRA
25.0%
39.00
60.22
75.95
67.92
63.46
68.98
39.76
59.33
8.17
13.22
10.20
Shortened- Taylor, LoRA
25.0%
37.40
71.82
73.72
69.56
71.19
66.88
44.45
62.15
10.32
14.97
12.65
Joint Layer Drop. LoRA
24.2%
35.00
69.30
72.47
64.11
67.95
59.81
38.23
58.12
14.32
20.14
17.23
LLMMA3.1-B
LaCo, LoRA
24.5%
36.40
69.46
71.93
66.50
76.24
64
41.13
60.91
10.77
17.23
14.00
Pruner, LoRA
24.5%
37.20
64,09
75.44
65.72
64.55
62.42
35.67
57.87
20.42
34.87
27.65
SliceGPT, LoRA
24.6%
61.56
66.70
56.96
72.39
31.48
53.37
9.22
26.42
17.82
MoD-D
25.0%
31,60
52,41
64.25
50.44
50.28
37.67
28.24
44.98
34.21
43.29
38.75
D-LLM
25.0%
30,20
$2.49
$7.40
37.64
50.36
37.12
28.16
41.91
40.12
132.44
86.28
SLipGPT Joint
25.3%
31.50
52.34
60.13
50.87
50.74
37,21
28.66 66
44.49
9.87
31.28
20.58
SkipGPT RT.
25.5%
42.60
77.03
79.97
82.13
82.84
84.47
$7.08
72.30
7.00
11.70
9.40
Dense
0.00%
44.80
77.51
80.03
81.95
82,
84.85
$7.59
72,69
6.24
10.58
8.41
Dense, LoRA
0.00%
44,90
77.68
80.22
81.86
82.23
84
$7.89
72.81
6.13
8.29
ShonGPT, LoRA
40.6%
32.00
67.32
68.61
58.43
65. 38
$3.37
35,32
$4.35
18.35
30.65
24.50
Shortened PPI., LoRA
40.6%
33.80
54.78
71.16
54.43
60.58
55.51
31.31
51.65
12.77
20.54
16.66
Shortened- Taylor, LoR A
40.6%
32.40
64.64
68.01
57.55
65.02
53.11
33.02
$3.39
17.22
28.79
23.01
Joint Layer Drop, LoRA
39.9%
28.60
$2,96
60.23
35.10
56.91
36.99
36.99
43.97
21.65
33.25
27.45
LLaMA3.1-8B
LaCo, LoRA
40.7%
56.27
65.13
43.99
26.79
16.43
LLM Pruner, LoRA
39.9%
31.80
$5.01
70.24
$1.34
56.
50.08
27.90
48.94
30.43
40.67
35.55
SliceGPT, LoRA
39.9%
55.41
60.61
15
67.52
25.34
45,99
14.87
33.24
MoD-D
40.0%
33.00
51.38
65.56
54.01
50.28
38.09
30,20
46.07
40.42
52.76
46.59
D.LLM
40.0%
31.80
$1.78
58.54
48.30
50.00
44.82
26.88
44.59
52.78
231.66
142,22
SkipGPT-Joint
40.3%
32.94
51.23
60.41
51.24
50.21
39.75
30.76
45.22
13.28
45.63
29.46
SkipGPT-RI-L
40.3%
40.80
74.98
79.16
80.33
80.00
82.74
54.01
70.29
7.70
13.10
10.40
Table Comparison of SkipGPT-RT-L against LoRA finetuned static pruning and dynamic pruning baselines. LoRA results for shortened- PPL., Shortened-Taye Taylor,,,,,, and SliceGPT on LLaMA2-13B are omitted these methods fail to converge during recovery training.
the router struggles to refine its decisions, further degrad- ing both its effectiveness and overall model performance. Ultimately, this prevents the joint training paradigm from achieving strong results. In contrast, our two-stage training paradigm effectively mitigates these challenges, enabling the pruned model to achieve superior performance compared to existing dynamic pruning methods.
Comparison of Redundancy between Attention and MLP Modules Recent studies suggest that LLMs are more resilient to the removal of self-attention layers than feed-forward layers, indicating that the attention modules exhibit higher redundancy compared to MLP modules (Sid- diqui et al., 2024; He et al., 2024; He et al., 2025). To ex- plore this further, we analyze five pruned models of LLaMA- 2-13B, generated through router tuning under different tar- get sparsity levels T. For each pruned model, we measure the average sparsity of the attention and MLP modules us- ing 50 randomly selected sentences from the WT2 dataset (Merity et al., 2016). Here, sparsity is defined as the ra- tio of pruned modules to the total number of modules in either the attention or MLP. As shown in Figure 5, SkipGPT.
6. Routing Behavior Analysis
Beyond efficiency, we explore if router tuning in SkipGPT- RT reveals deeper insights into the original LLMs. By ana- lyzing the router's 's behavior, we aim to uncover meaningful patterns in how different modules contribute to inference.
8

---

SkipGPT: Each Token is One of Kind
5.0
5.0
Language Modeling Loss
LaCo, LoRA
LLM-Pruner, LoRA
4.5
Layer Drop, LoRA
MoD-D
lodeling g Los
4.5
Joint Layer Drop, LoRA
Shortened PPL LoRA
D-LLM
ShortGPT, LoRA
4.0
Shortened Taylor. LoRA
SNppGPTjoint
4.0
LLM Pruner,
ShortGPT. LoRA
Dense, Lora
MoD-D
3.5
SlicePPT, LoRA
SNpGPPTRT
3.5
DILLM
SkipGPT-RT-
MipGPPJjiim
3.0
3.0
Dense, Lora
SKIpGPT-RT
2.5
2.5
SkipGPT-RT-L
2.0
2.0
1.5
1.5
0
2000
4000
6000
8000
10000
0
2000
4000
6000
8000
10000
Training Steps
Training Steps
(a) LLaMA2-7B
(b)
LLaMA2-13B
Figure Training loss curves of AoRA-finetuned static and dynamic pruning baselines, and two-stage training of SkipGPT.
Attention and MLP Sparsity (%6
80
0.80
Attention Sparsity
54.35
0.75
Average Attention Activation Average MLP Activation
Sparsity
60
Activation Ratio
0.70
22.57
0.65
0.60
33.18
0.55
20
0.50
0.45
30
40
50
60
70
0.40
0.0
0.5
1.0
1.5 2.0 2.5
3.0 3.5 4.0
User-Defined Target Sparsity (r)
Position Index (Start Sliding Window,
Figure 5. Average sparsity of the attention and MLP modules in five pruned models of LLaMA-2-13B, generated through router tuning under different target sparsity levels T.
Figure 6. Redundancy shifts in attention and MLP modules of 45% sparsity LLaMA2-133 13B model obtained via router tuning as context length grows.
RT consistently prunes more attention than MLP modules, across both low and high pruning rates. This observation aligns with previous findings and may indicate structural inefficiency in the current Transformer architecture, which enforces fixed pairing of one attention and one MLP mod- ule per layer. We hypothesize future LLMs could achieve greater efficiency by revisiting this design and potentially reducing the proportion of attention modules.
tokens require more attention computation but less MLP computation. We hypothesize that in the early stages, the model has not yet determined the task and has limited con- textual information. As result, it relies more on complex MI. computations for task identification and less on at- tention for context extraction. Later, as the task becomes clear and context accumulates, the model shifts to increased attention computation while reducing MI.P computation. Developing techniques that dynamically adjust computation presents an exciting direction for future research.
Redundancy Shifts Attention and MLP Modules Del Corro et al. (2023) argues that as the context grows with the sequence, later tokens become more predictive and therefore require less computation. Meanwhile, He al. (2025) shows that the MLP module has higher redundancy during the decoding phase compared to the pre-filling phase, while attention redundancy remains nearly the same in both phases. We investigate how redundancy shifts within the attention and MLP modules by analyzing a 45% sparsity model obtained through router tuning on LLaMA2-133 We randomly select sentences from RedPajama, truncate them to the model's maximum context length, and apply 100-token sliding window across each sentence. At each step, we record the activation ratios of attention and MLP within the sliding window and average the results over 50 examples. The final results appear in Figure 6. Contrary to previous findings, we observe that as context grows, later
7. Conclusion
In this work, introduce SkipGPT, novel dynamic prun- ing framework that addresses the inefficiencies of static layer pruning by incorporating horizontal and vertical dy- namics. SkipGPT dynamically allocates computation token and decouples attention and MLP pruning, enabling more fine-grained optimization. To stabilize training, we propose two-stage training paradigm, where the router is first tuned alone before fine-tuning the model with LoRA. Experiments show that the pruned model can fully restore its performance, even surpassing the original model de- spite a 40% parameter reduction. Furthermore, our router analysis reveals key insights into module redundancy and token-level compute allocation, highlighting potential di- rections for future model efficiency improvements.

---

SkipGPT: Each Token is One of Kind
Impact Statement
X. L., Li, X., Ma, T., Malik, A., Manning, C. D., Mirchan- dani, S., Mitchell, E.,, Munyikwa, Z., Nair, S., Narayan, A., Narayanan, D., Newman, B., Nie, A., Niebles, J. C., Nilforoshan, H.. Nyarko, J., Ogut, G., Ort. L.,, Papadim- itriou, I., Park, J. S., Piech, C., Portelance, E.,, Potts, C., Raghunathan, A., Reich, R., Ren, H., Rong. F,, Roohani, Y., Ruiz, C.. Ryan, J., Ré, C., Sadigh, D., Sagawa, S., Santhanam, K., Shih, A., Srinivasan, K., Tamkin, A.. Taori, R., Thomas, A. W., Tramèr, F., Wang, R. E., Wang, W., Wu, B., Wu, J., Wu, Y., Xie, S. M., Yasunaga, M., You, J., Zaharia, M., Zhang, M., Zhang. T., Zhang, X., Zhang, Y., Zheng, L., Zhou, K., and Liang, P. On the opportunities and risks of foundation models, 2022. URL https: /aaxiv.viivv.///.i///aa/a//eaavv org aass///////// 20080000000 07258.
Our work, SkipGPT, introduces a novel dynamic pruning framework that significantly improves the efficiency of large language models (LLMs) by adapting computation to token complexity. By decoupling MLP and attention pruning and introducing two-stage training paradigm, our approach enhances computational efficiency while preserving, and even surpassing, original model performance. This reduces the energy demands of LLMs. making them more sustain- able and accessible for deployment in resource-constrained environments, while also providing new insights into model redundancy and architectural optimization.
References
Brown, T. Language models are few-shot learners. arXiv preprint arXiv:2005.44165 2020.
An, Y., Zhao, X., Yu, T., Tang, M., and Wang. J. Fluctuation- based adaptive structured pruning for large language mod- els. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 10865-10873, 2024.
Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., and Huang, J. A survey on mixture of experts, 2024. URL https: / //aaxivv.oooo wwwpxiivorgg/bboo absg////////// 2407.06204
Ashkboos, S., Croci, M. L., Nascimento, M. G. d., Hoefler, T., and Hensman, J. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401. 200224401.502.... 2024.
Chen, X., Hu, Y., and Zhang, J. Compressing large language models by streamlining the unimportant layer. arXiv preprint arXiv:2403. 19135, 2024a.
Bartol, Thomas M, J.. Bromer, C., Kinney, J., Chirillo, M. A., Bourne, J. N., Harris, K. M., and Sejnowski, T. J. Nanoconnectomic upper bound on the variabil- ity of synaptic plasticity. eLife, 4:e10778, nov 2015. URL
Chen, X., Hu, Y., and Zhang, J. Compressing large language models by streamlining the unimportant layer. arXiv preprint arXiv:2403. 19135, 2024b.
ISSN 2050-084X. doi: 10.7554/eLife.10777 https: //d///ddo /ttp///oo.ooo////////ooo org/l0. 7554/55ife. 10778.
Chen, X., Sun, Z., Guo, W., Zhang, M., Chen, Y., Sun, Y., Su, H., Pan, Y., Klakow, D., Li, W., and Shen, X. Unveiling the key factors for distilling chain-of-thought reasoning, 2025. URL https: /tt///axvvv..///// org.iioorrrrgeeee ors/ 2502.18001
Bengio, Y., Léonard, N., and Courville, A. Estimating propagating gradients through stochastic neurons for con- ditional computation. arXiv preprint arXiv: 330..3432, 2013.
Chen, Y., Pan, X., Li, Y., Ding, B., and Zhou, J. Ee- 11m: Large-scale training and inference of early-exit large language models with 3d parallelism, 2024c. URL https: /tas:///rxvv.... org/oogg/////. hts//bbs/2311/ 2212. 04916.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense natural language. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 34, pp. 1439.7439, 2020.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N. Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S., Dev, S., Michalewski, H., Garcia, X.. Misra, V., Robinson, K., Fedus, L., Zhou, D., Ippolito, D., Luan, D., Lim, H., Zoph, B., Spiridonov, A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A. M., Pillai, T. S., Pellat, M., Lewkowycz, A., Moreira, E., Child, R., Polo- zov, O., Lee, K., Zhou, Z., Wang, X., Saeta, B., Diaz, M., Firat, O., Catasta, M., Wei, J., Mellrt-Hl K., Eck, D., Dean, J., Petrov, S., and Fiedel, N. Palm:
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosse- lut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N., Chen, A., Creel, K., Davis, J. Q. Demszky, D., Donahue, C., Doumbouya, M.. Durmus, E., Ermon, S., Etchemendy, J.. Ethayarajh, K., Fei-Fei, L., Finn, C. Gale, T., Gillespie, L., Goel, K., Goodman, N., Grossman, S., Guha, N., Hashimoto,
T., Henderson, P., Hewitt, J., Ho, D. F., Hong, J., Hsu, K., Huang. J.. Icard, T., Jain, S., Jurafsky, D., Kalluri, P., Karamcheti, S., Keeling, G., Khani, F., Khattab, O., Koh, P. W., Krass, M., Krishna, R., Kuditipudi, R., Kurnar A., Ladhak, F., Lee, M., Lee, T., Leskovec, J., Levent, I., Li,
10

---

SkipGPT: Each Token is One of Kind
Scaling language modeling with pathways. 2022. URL https: http///avvvvvv.////v//ee///..vvv org/ abs///b////// 2204.. 02311.
He, S., Sun, G., Shen, Z., and Li, A. What matters in transformers? not all attention is needed, 2024. URL https:
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint Axii:1905.100444 2019.
He, Z., Yizhen Yao, Pengfei Zuo, Bin Gao, Qinya Li, Zhen- zhe Zheng, and Fan Wu. Adaskip: Adaptive sublayer skip- ping for accelerating long-context Ilm inference, 2025, URL hwwps: httpp//////.wi....o/a///////////
02336.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv: arXiv:/803.000 1108.03355577 2018.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, s., Wang, I.,, and Chen, W. Lora: Low-rank adaptation of large language models, 2021. URL https: /arxiv. org/ abs /ww.. 09685.
Computer, T. Redpajama: an open dataset for training large language models. 2023. htts:: / / com/ wtwpeeteeercoomuteeeeeeeeeaappae
Jang, E.,, Gu, S., and Poole, Categorical reparameteriza- tion with gumbel-softmax. In International Conference on Learning Representations, 2022.
Data.
Del Corro, L., Del Giorno, A., Agarwal, S., Yu, B., Awadal- lah, A., and Mukherjee, S. Skipdecode: Autoregressive skip decoding with batching and caching for efficient Ilm inference. arXiv preprint arXiv:2307.0226: 2023.
Jiang, Y., Wang, H., Xie, L., Zhao, H., zhang chao, Qian, H., and Lui, J. C. D-llm: A token adaptive comput- ing resource allocation strategy for large language mod- els. In Proceedings of the 38th Conference on Neu- ral Information Processing Systems (NeurIPS 2024), 2024. URL https///eeeris.ccc/eeeeee 4024/poster/94977
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang. A., Fan, A., et al. The llama herd of models. arXiv preprint urxiv:2407.217880 2024.
Kim, B.-K., Kim, G., Kim, T.-H., Castells, T., Choi, S., Shin, J., and Song, H.-K. Shortened Ilama: simple depth pruning for large language models, arXiv preprint arXiv:2402.02834, 11, 2024.
Fan, S., Jiang, X., Li, X., Meng, X., Han, P. Shang, S., Sun, A., Wang. Y., and Wang, Z. Not all layers of llms are necessary during inference, 2024. URL https:// arxiv. org. abs/ 2403. 02181.
Kobayashi, G., Kuribayashi, T., Yokoi, S., and Inui, K. Attention is not only weight: Analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7057-7075, 2020.
Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. In Inter- national Conference on Machine Learning, pp. 10323- 10337. PMLR, 2023.
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi,
A., Foster, C., Golding, I.,, Hsu, J., Le Noac"h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H.. Skowron, A., Sutawika, L., Tang, F.,, Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 07 2024. URL https::t/////. //ee:///////ooo////// org/ wwtg//eeoors/// 12 82608602.
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Gshard: Scaling giant models with conditional computation and automatic sharding, 2020. URL https: http::///xxxvvvvooee abs/ 2006 16668.
Geva, M., Schuster, R., Berant, J., and Levy, O. Transformer feed-forward layers are key-value memories, 2021. URL https: /ates///arxiii /aaxiiiiv.. org/ abs 2012.0022222 14913.
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization, 2019, URL https: http:///axxvvvooooee /bs// 1 7711.05100
Gromov, A., Tirumala, K., Shapourian, H., Glorioso, P., and Roberts, D. A. The unreasonable ineffectiveness of the deeper layers. arXiv preprint (tt::2403.78887) 2024.
Ma, X., Fang, G., and Wang, X. LIm-pruner: On the struc- tural pruning of large language models. Advances in neural information processing systems, 36:21702-2111 2023.
Gumbel, J. Statistical theory of extreme values and some practical applications: a series of lectures, volume 33. US Government Printing Office, 1954.

---

SkipGPT: Each Token is One of Kind
Maddison, C. J., Tarlow, D., and Minka, T. A sam- pling, 2015. URL https://arxiv..rrg abs/
Greene, R., Gross, J., Gu, S., Guo, Y., Hallacy, C.,, Han,
J., Harris, J., He, Y., Heaton, M., Heidecke, J., Hesse, C., Hickey, A., Hickey, W., Hoeschele, P., Houghton, B., Hsu, K., Hu, S., Hu, X., Huizinga, J., Jain, S., Jain, S., Jang, J., Jiang, A., Jiang, R., Jin, H., Jin, D., Jomoto, S., Jonn, B., Jun, H., Kaftan, T., Lukasz Kaiser, Kamali, A., Kanitscheider, I., Keskar, N. S., Khan, T., Kilpatrick, L., Kim. W., Kim, C.,, Kim, Y., Kirchner, J. H., Kiros, J., Knight, M., Kokotajlo, D., Lukasz Kondraciuk, Kondrich, A., Konstantinidis, A., Kosic, K. Krueger, G., Kuo, V., Lampe, M., Lan, I., Lee,, T., Leike, J., Leung, J., Levy. D., Li, C. M., Lim, R., Lin, M., Lin, S., Litwin, M., Lopez, T., Lowe, R., Lue, P., Makanju, A., Malfacini, K., Manning, S., Markov, T., Markovski, Y., Martin, B., Mayer, K., Mayne, A. McGrew, B., McKinney, M., McLeavey, C., McMillan, P., McNeil, J., Medina, D., Mehta, A., Menick, J., Metz, L.,, Mishchenko, A., Mishkin, P., Monaco, V., Morikawa, E.,, Mossing, D., Mu, T., Murati, M., Murk, O., Mély, D., Nair, A., Nakano, R., Nayak, R., Neelakantan, A., Ngo, R., Noh, H., Ouyang, L., O' Keefe, C., Pachocki,
1411.0030.
Marcus, M., Santorini, B., and Marcinkiewicz, M. A. Build- ing large annotated corpus of english: The penn tree- bank. Computational linguistics, 19(2):313-338) 1993.
Men, X., Xu, M., Zhang, Q.. Wang, B., Lin, H., Lu, Y., Han, X., and Chen, W. Shortgpt: Layers in large language mod- els are more redundant than you expect. arXiv preprint arXiv:2403. 03853, 2024.
Meng, K. Bau, D., Andonian, A., and Belinkov, Y. Locat- ing and editing factual associations in gpt, 2023. URL https: //arxvv..vvvv///a///// org/orggaooooo abs/ 220222222222222 . 05262.
Merity, S., Xiong. C., Bradbury. J., and Socher, R. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016.
Merullo, J., Eickhoff, C., and Pavlick, E. Language models implement simple word2vec-style vector arithmetic, 2024. https::
J., Paino, A., Palermo, J., Pantuliano, A.. Parascandolo,
G., Parish, J., Parparita, E.,, Passos, A., Pavlov, M., Peng, A., Perelman, A., de Avila Belbute Peres, F., Petrov, M., de Oliveira Pinto, H. P., Michael, Pokomny, Pokrass, M., Pong, V. H., Powell, T., Power, A., Power, B., Proehl, E., Puri, R., Radford, A., Rae, J., Ramesh, A., Raymond, C., Real, F., Rimbach, K., Ross, C., Rotsted, B., Roussez, H., Ryder, N., Saltarelli, M., Sanders, T., Santurkar, S., Sastry, G., Schmidt, H. Schnurr, D., Schulman, J., Selsam, D., Sheppard. K., Sherbakov, T., Shieh, J.. Shoker, S., Shyam, P., Sidor, S., Sigler, E.,, Simens, M., Sitkin J., Slama, K., Sohl, L. Sokolowsky, B., Song, Y., Staudacher, Such, F. P., Summers, N., Sutskever, I., Tang, J., Tezak, N., Thompson, M. B., Tillet, P., Tootoonchian, A., Tseng, E., Tuggle, P., Turley, N., Tworek, J., Uribe, J. F. C., Vallone, A., Vijayvergiya, A., Voss, C., Wainwright, C., Wang, J. Wang, A., Wang. B., Ward, J., Wei. J.. Weinmann, C., Welihinda, A., Welinder, P., Weng, J., Weng, L., Wi- ethoff, M., Willner, D., Winter, C., Wolrich, S., Wong, H., Workman, L., Wu, S., Wa, J., Wu, M., Xiao, K., Xu, T., Yoo, S., Yu, K., Yuan, Q., Zaremba, W., Zellers, R., Zhang, C., Zhang, M., Zhao, S., Zheng, T., Zhuang. J., Zhuk, W., and Zoph, B. Gpt-4 technical report, 2024. URL https: /alp://arrae /ttp://aaxvv.v/..//ee www. /as///b///// www.//3333333 22/22333300777
wttp://ggabb//.oo// 16130.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018.
Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann. B., Askell, A., Bai, Y., Chen, A. Conerly, T., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D.. Johnston, S., Jones, A., Kernion, J., Lovitt, I.., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. In-context learn- ing and induction heads. Transformer Circuits Thread, 2022. hutps//transfoomercrcriiiic http:rrrrrrcccisssup////o//ooooo wwwp/0222/incoottttt hww.:/innnnnnndd/eee..ee..
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Bal- aji, S., Balcom, V. Balteseu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shapiro G., Berner, C., Bogdonoff, L., Boiko, O., Boyd, M., Brakman, A.-L., Brockman, G.. Brooks, T., Brundage, M., Button, K., Cai, T., Campbell, R., Cann, A., Carey, B., Carlson, C. Carmichael, R., Chan, B., Chang, C., Chantzis, F., Chen. D., Chen, S. Chen, R., Chen, J.. Chen, M., Chess, B. Cho, C., Chu, C., Chung, H. W., Cummings. D., Cur- rier, J., Dai, Y., Decareaux, C., Degry, T., Deutsch, N., Deville, D., Dhar, A., Dohan, D., Dowling, S. Dunning, S., Ecoffet, A., Elcti, A., Eloundou, T., Farhi, D., Fedus, L., Felix, N., Fishman, S. P., Forte, J., Fulford, I., Gao, L.,, Georges, E., Gibson, C., Goel, V. Gogineni, T., Goh, G., Gontijo-Lopes, R., Gordon, J., Grafstein, M., Gray, S.,
Raposo, D., Ritter, S., Richards, B., Lillicrap, T. Humphreys, P. C., and Santoro, A. Mixture-of deptts Dynamically allocating compute in transformer-based lan- guage models. arXiv preprint N.Nv:22404.02258, 2024.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106 2021.
12

---

SkipGPT: Each Token is One of Kind
Samsi, S., Zhao, D., McDonald, J., Li, B., Michaleas, A., Jones, M., Bergeron, W., Kepner, J., Tiwari, D., and Gadepally, V. From words to watts: Benchmarking the energy costs of large language model inference, 2023. URL https::////......... /aaiiv.iv.. org/ abs/ 03003.
networks, 2018. URL 1711.09485.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F,, Chi, E., Le. Q. and Zhou, D. Chain-of- thought prompting elicits reasoning in large language models, 2023. URL https: /ppps///axiv...// org/ /bba/ 11903.
Schuster, T., Fisch, A.. Gupta. J.. Dehghani, M., Bahri, D., Tran, V., Tay, Y., and Metzler, D. Confident adaptive language modeling, Advances in Neural Information Processing Systems, 35:17456-1744 17472, 2022.
Xia, M., Gao, T., Zeng, Z., and Chen, D. Sheared llama: Accelerating language model pre-trainn pre-training via structured pruning. arXiv preprint arXiv:23 10.::3300006944 2023.
Siddiqui, A., Dong, X., Heinrich, G., Breuel, T., Kautz, J., Krueger, D., and Molchanov, deeper look at depth pruning of Ilms. arXiv preprint arXiv:2407.16286 2024.
Xin, H., Sun, Y., Wang, C., and Xiong, H. Llmedsr: Enhanc- ing cross-domain sequential recommendation with large language models. ACM Trunsuctions on Information Systems, 2025.
Song, J., Oh, K., Kim, T., Kim, H., Kim, Y., and Kim, J.JJ. Sleb: Streamlining Ilms through redundancy verification and climination of transformer blocks. arXiv preprint DrXiv:2402.09025. 2024.
Yang, Y., Cao, Z., and Zhao, H. Laco: Large lan- guage model pruning via layer collapse, arXiv preprint arXiv:2402.1 11877, 2024.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, F., and Lam- ple, G. Llama: Open and efficient foundation language models, 2023a. URL pectps://arxint org/ abs/ 2302. 13971.
Yom Din, A., Karidi, T., Choshen, L., and Geva, M. Jump to conclusions: Short-cutting transformers with linear transformations. In Calzolari, N., Kan, M.-Y., Hoste, V., Lenci, A., Sakti, S., and Xue, N. (eds.). Proceedings the 2024 Joint International Conference Computational Linguistics, Language Resources and Evaluation (LREC- COLING 2024), pp. 9625, Torino, Italia, May 2024. ELRA and ICCL. URL https: / / aacllanhhoooy.. org/ 202/ Irec main. 840.
Touvron, H., Martin, L.., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint Xiiv:2307.00288. 2023b.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sentence? 2019.
Varshney, N., Chatterjee, A., Parmar, M., and Baral, C. Accelerating Ilama inference by enabling intermediate layer decoding via instruction tuning with lite, 2023. URL https: /araiiiivvvv//..//aaaaaaaaaaavva org/ abs
arXiv preprint
Zeng, D., Du, N., Wang, T., Xu, Y., Lei, T., Chen, Z., and Cui, C. Learning to skip for language modeling, 2023, URL https://aatt http///axivv.vv orq/
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N.. Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (cds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https::/prcceeedngg neurips. ec /t/ppppee_f1lee/pppp////1/////11 Paper.
Zhang, Y., Li, Y., Wang, X.. Shen, Q., Plank, B., Bischl, B., Rezaei, M., and Kawaguchi, K. Finercut: Finer- grained interpretable layer pruning for large language models, 2024. URL https: http:///xxivvorr/ee /bss/ 2405.18218.
Zhao, A., Ye, F., Fu, J., and Shen, X. Unveiling in-context learning: A coordinate system to understand its working mechanism. arXiv preprint 2rXiv:2407.70111 2024.
pdf.
Wan, Z. Wang, X., Liu, C., Alam, S., Zheng, Y., Liu, J.,
Qu, Z., Yan, S., Zhu, Y., Zhang. Q., Chowdhury, M., and Zhang, M. Efficient large language models: sur- vey, 2024. URL https: wtwpvv.iiiiv////ii//...//aaaaavv wwtpriii.w...........eeeeaaaa a23//22b 03863.
Zhu, T., Qu, X., Dong, D., Ruan, J., Tong, J., He, C., and Cheng, Y. Llama-moe: Building mixture-of-expent from Ilama with continual pre- trainnnnn 2024. URL https: // wwwxiv..rrg//o abs/ 2406. 1655.
Wang, X., Yu, F., Dou, Z.-Y., Darrell, T., and Gonzalez, E. Skipnet: Learning dynamic routing in convolutional
Zhu, Y., Kiros, R., Zemel. R. Salakhutdinov, R., Urta- sun, R., Torralba, A., and Fidler, S. Aligning books and
13

---

SkipGPT: Each Token is One of Kind
Towards story-like visual explanations by watch- ing movies and reading books. In The IEEE International Conference Computer Vision (ICCV), December 2015.
movies:
14

---

SkipGPT: Each Token is One of Kind
A. Related Work
Static Pruning. Static pruning refers to a kind of approach where the computation of the pruned LLMs remains invariant to the imput instances. SparseGPT (Frantar & Alistarh, 2023) simplifies the pruning problem by turning it into large-scale sparse regression tasks, which are efficiently solved using a novel approximate solver. FLAP (An et al., 2024) and LLM-Pruner (Ma et al., 2023) reduce network width by eliminating coupled structures while keeping the number of layers unchanged. Sheared-LLaMA (Xia et al., 2023) learns pruning masks at various granularities, from global ones like layers and hidden dimensions to local ones like attention heads and intermediate dimensions. SliceGPT (Ashkboos et al., 2024) reduces the network's embedding dimension by replacing each weight matrix with a smaller dense matrix. Recent works (Song et al., 2024; Gromov et al., 2024; Kim et al., 2024; Chen et al., 2024a; Yang et al., 2024; Siddiqui et al., 2024) demonstrate that is possible to selectively drop blocks from a range of pretrained language models, sparking community interest in depth pruning. In this context, Zhang et al. (2024) and He et al. (2024) investigate the decoder layers, treating the self-attention layers and FFN layers independent components to be pruned separately, and observe a preference for pruning the self-attention layers. Despite significant advancements in static pruning, substantial recovery fine-tuning is often necessary to preserve performance post-pruning, making the process both costly and challenging to scale.
Dynamic Pruning. Dynamic pruning refers to another kind of approach where the pruming of unimportant layers is dynamically determined based on the specific imput instance. common technique in this context is Early Exit (Schuster et al., 2022; Varshney al., 2023; Del Corro al., 2023; Yom Din et al., 2024; Chen al., 2024c: Fan et al., 2024). This approach dynamically evaluates whether to continue processing subsequent transformer blocks. Notably, transformer blocks that produce predictions matching the final token output of LLMs are typically located towards the end of the model. As result, extensive training is often required to adapt LLMs for the effective use of early exit mechanisms. Therefore, earty exit strategies have been rarely explored in larger SOTA LLMs. Another prominent technique is Skip Layer, which dynamically skips the execution of intermediate layers (or modules) for a given input token. This is achieved through mechanisms such as gating funetion (Wang et al., 2018; Raposo et al., 2024) or a binary router (Zeng et al., 2023; Jiang et al., 2024). For instance, Mixture-of-Deplte (MoD) (Raposo et al., 2024) determines which tokens to process using top-k routing mechanism. Essentially, SkipGPT aligns with the Skip Layer paradigm, where the execution of layers for each input token is dynamically determined. To the best of our knowledge, ours is the first work to simultaneously address both horizontal and vertical dynamics.
B. Training Algorithm of SkipGPT
The detailed two-stage training paradigm is outlined in Algorithm
C. Detailed Descriptions of All Baseline Methods
ShortGPT is a structured pruning method that removes redundant layers in LLMs based on a novel importance metric called Block Influence (BI). By analyzing hidden state transformations, ShorGGP assigns BI scores to measure each layer's contribution to model performance. Layers with lower BI scores are identified as redundant and pruned in ascending order. This simple yet effective approach significantly reduces model size and inference cost while maintaining competitive performance. Unlike complex pruning techniques, ShortGPT demonstrates that LLMs contain substantial layer-wise redundancy, enabling efficient compression without additional fine-tuning.
is a depth pruning method that reduces the computational cost of LLMs by removing entire Trans- former layers while keeping the remaining architecture intact. It determines layer importance using Perplexity (PPL) and Taylor Expansion, pruning less significant layers in a one-shot manner. Unlike width pruning, which reduces weight matrices but struggles to accelerate inference under memory constraints, Shortened LLaMA achieves significant speedups, particularly in resource-limited settings. To recover pruned models, it employs LoRA-based fine-tuning for moderate pruning and Continued Pretraining (CPT) for aggressive pruning. This method provides an efficient way to compress L.Mss while maintaining strong performance.
Shortened LLaMA
LaCo (Layer Collapse) is a structured pruning method that progressively merges deeper layers into earlier ones, reducing model depth while preserving output representations. Instead of removing layers outright, LaCo employs Reserving- Differences- wew---eeeiinnnomnen (RDSC) Layer Merge, which integrates parameter differences from consecutive layers
15

---

SkipGPT: Each Token is One of Kind
Algorithm I Training Process of SkipGPT
Require: Pretrained model M, dataset D, target sparsity T, router parameters 0, learning rates ry (router) and 7/2 (LoRA),
maximum steps S1 and 2
1: Initialize router parameters 0 for each module
2: Initialize LoRA parameters (Optional)
3: Stage 1: Router Tuning
4: for step = to S₁ do
5: X (ample(D) (Sample batch)
6: for token in X do
7: for module / - 1 to LoLLon do
8:
r <- W with &Lw being active oW
9:
gi Gumbel-Softmaxx(ff
10: 2|41 g (fi(af) =f) of [0] of
11: end for
12:
end for
Ex, 9/10
SXL
14: Call f- Lim T -r
15: Update 8 t- 0 VoLall
16: end for
17: Stage 2: LoRA Fine-Tuning (Optional)
18: for step = to S2 do
sample(D) 20: for token in X do
21:
for module / = 1 to L do
22:
acm. of + W of with oW
0 (routers frozen)
of t argmax(r()
24:
if of then
25:
x|+1 + li(af) #
26:
else
27:
end ri+1 + of
28:
29:
end for
end for
31: Compute Zim
32:
Update LoRA parameters using 722 and LORA
33: end for
Return: Pruned model M'
into a prior layer, maintaining structural integrity. To minimize performance degradation, it uses few-shot calibration samples to ensure representation similarity. This approach enables 30-50% layer reduction without retraining, significantly lowering computational costs while retaining strong performance. Additionally, post-training on pruned models further restores accuracy, making LaCo a highly effective structured pruning technique for large language models.
Joint Layer Drop is a structured pruning method that enhances the efficiency of Transformer based LLMs by jointly pruning Attention and MLP layers based similarity- based metric. This method first removes redundant Attention layers, as they exhibit significant redundancy while maintaining performance. Once the least important Attention layers are pruned, it selectively removes MLP layers to further compress the model. By dynamically balancing the pruning of both components, Joint Layer Drop achieves higher compression ratios with minimal accuracy degradation, making a highly effective approach for structured model reduction.
LLM-Pruner is structured pruning method designed for task-agnostic compression LLMs, aiming to reduce model size while preserving their general-purpose capabilities. employs dependency- structural pruning, where
16

---

SkipGPT: Each Token is One of Kind
interdependent components are grouped and pruned together to minimize disruption. Importance estimation is performed using gradient-based and Hessian-based metrics, ensuring the least impactful components are removed. Unlike traditional compression methods that require extensive retraining, LLM-Pruner enables efficient post-training recovery using LoRA- based fine-tune tuning, requiring only 50K public samples. This approach significantly reduces computational overhead while maintaining strong zero-shot performance across multiple tasks.
SliceGPT is a post-training sparsification method that reduces the computational and memory demands of LLMs by removing entire rows and columns of weight matrices, effectively shrinking the embedding dimension while maintaining dense matrix operations for efficient execution. It leverages computational invariance, applying orthogonal matrix trans- formations to ensure minimal performance degradation. Using Principal Component Analysis (PCA), SliceGPT projects activation signals onto their principal components, allowing redundant dimensions to be pruned. This method achieves up to 30% compression on LLaMA-2, OPT, and Phi-2 models while retaining over 90% of the original model's 's accuracy, leading to significant inference speedups and reduced hardware requirements without additional fine-tuning.
Mixture-of-Depths (MoD) is a conditional computation method that dynamically allocates compute across model depth, reducing unnecessary computations in transformer-basee LLMs. Unlike standard transformers, which apply the same amount of compute to all tokens every layer, Mol) employs top-k routing mechanism selectively process only the most important tokens in each layer, skipping others via residual connections. This allows the model to optimize compute expenditure per token, maintaining performance while significantly reducing FL.OPs per forward pass. By ensuring static computation graph with predictable efficiency gains, MoD achieves up to 50% faster inference while matching or surpassing the performance of equivalent full-compute transformers.
D-LLM is a dynamic inference framework for LLMs that adaptively allocates computational resources at the token level, optimizing efficiency without compromising performance. It introduces dynamic decision module before each transformer layer, determining whether token should execute the layer be skipped, thereby reducing unnecessary computation for less critical tokens and simpler tasks. To maintain compatibility with KV-cache mechanisms, D-LLM employs KV-cache eviction policy, excluding skipped layers from subsequent attention calculations, which not only reduces storage overhead but also ensures smooth deployment in real-world applications. Experimentally, D-LLM achieves up to 50% reduction in computational cost across various NLP tasks while maintaining strong accuracy, making it a highly effective solution for resource-constrainet environments.
D. Exploring Pruning Scaling Laws with SkipGPT-RT
Due to its significantly superior performance over the baseline, we believe that SkipGPT-RT is capable of identifying optimal or ncar-optimal routing solutions for a given sparsity ratio. This capability establishes SkipGPT not only as an effective pruning method but also as a reliable probing tool for exploring the pruning scaling laws in existing LLMs. To this end, in this section, we employ SkipGPT-RT as a "probe" to analyze both LLaMA2-7B and LLaMA2-13B, using the SOTA static pruning method, Joint Layer Drop, for comparison, Figure highlights several key findings:
Static pruning methods lack scalability: While static pruning methods are simple and effective in some scenarios, their ability preserve model performance diminishes significantly under high sparsity levels. For instance, in the case of 1.LMMA--13B, SkipGPT-RT demonstrates significantly lower PPI. even at 70% sparsity, outperforming Joint Layer Drop, which struggles to maintain comparable performance even at a much lower sparsity of 50%.
The surprising redundancy in LLMs: Our analysis reveals that LLMs exhibit far greater redundancy than expected. For instance, LLaMA2- 13B only begins to show a noticeable increase in PPL at an 80% sparsity level. This phenomenon may stem from two key factors: (1) the inefficiency of the current transformer architecture, which applies uniform computation to all tokens regardless of their importance, and (2) the concentration of critical information within small subset of modules during pretraining. We hope this finding offers valuable insights for designing future architectures and pretraining strategies.
17

---

SkipGPT: Each Token is One of Kind
Ze4
1eJ
14387.0
8425.
8495.4
1.4
Joint Layer Drop
Joint Layer Drop
1.2
SkipGPT-RT
SkipGPT-RT
1.0
0.6
1870.1
1540.7
0.2
2044.0
510,2
0.0
126.916:4
54.6
148.4
21.3
33.6
1-11.5
20
30
40
50
60
30
40
50
60
70
80
Sparsity
Sparsity
(a) LLaMA2-7B
(b) LLaMA2-13B
Figure 7. Perplexity (PPL) of Joint Layer Drop and SkipGPT-RT under different sparsity levels.
MLP
0.05
0.16
1.0
Attn 32
0.91
0.92
0.95
0.97
0.90
0.98
0.98
0.92
0.96
0.97
0.90
0.98
0.97
0.96
0.92
0.80
0.81
0.69
0:90
0.75
0.63
0.83
0.67
0.71
0.75
0.75
0.85
0.87
Attn 31
0.98
0.96
0.98
0.98
0.98
0.98
0.97
0.98
0.98
0.98
0.98
0.99
0.98
0.95
0.99
MLP 30
0.86
0.88
0.90
0.93
0.97
0.91
0.87
0.92
0.89
0.90
0.93
0.88
0.92
0.94
0.90
0.99
0.98
0.99
0.99
0.98
0.99
0.99
0.99
-0.6
MLP
0.29
0.42
0.54
0.41
0.42
0.51
0.49
0.36
0.41
0.46
0.43
0.47
0.57
0.60
0.55
Attn
0.93
0.78
0.72
0.62
0.55
0.80
0.58
0.75
0.57
0.65
0.72
0.66
0.73
0.76
0.82
MLP
0.60
0.51
0.41
0.49
0.61
0.56
0.46
0.38
0.58
0.51
0.59
0.56
0.59
004
Attn
0.84
0.67
0.61
0.60
0.49
0.63
0.72
0.59
0.51
0.72
0.62
0.52
0.62
0.64
0.73
MLP
0.56
0.60
0.47
0.48
0.78
0.71
0.74
0.49
0.53
0.67
0.55
0.60
0.66
0.66
0.58
Attn
0.81
MLP
0.44
0.50
0.43
0.54
0.48
0.51
0.49
0.61
0.51
0.54
0.50
0.68
0.67
0.54
0.44
Attn
0.14
0.21
0.14
0.20
0.44
0.32
0.41
0.32
0.28
0.25
0.36
0.38
0.29
0.35
0.31
MLP
0.22
0.55
0.52
0.48
0.65
0.63
0.74
0.69
Attn
0.01
0.02
0.05
0.20
0.00
0.24
0.33
0.02
0.35
0.07
0.34
0.04
0.04
0.01
0.25
he
seen
the
movie
almost
by
mistake
considering
he
was
little
0,0
Figure 8. Token- Wise Cosine Similaritics Across Modules in LLaMA-2-7B.
E. Additional Case Studies on Motivation
In this section, present additional case studies that illustrate Token-Wiss Cosine Similarities Across Modules in LLaMA2-7B, as shown in Figures 8 and 9, For I.L.MA2-133 133,, see Figures 10 and 11.
18

---

SkipGPT: Each Token is One of Kind
MLP 32
0.04
0.02
0.04
0.03
0.08
0.08
0.06
0.17
0.04
0.06
0.06
0.24
0.04
0.10
0.09
1.0
Attn 32
0.92
0.97
0.96
0.97
0.95
0.92
0.94
0.97
0.98
0.90
0.88
0.95
0.98
0.91
0.96
0.70
Attn 31
0.98
0.99
1.00
0.98
0.99
0.97
0.99
0.99
0.98
0.97
0.99
0.96
0.98
0.98
0.98
MLP 30
0.84
0.94
0.95
0.91
0.93
0.87
0.95
0.93
0.89
0.94
0.94
0.85
0.89
0.93
0.93
-0.6
MLP
0.30
0.51
0.44
0.37
0.48
0.46
0.47
0.50
0.46
0.55
0.46
0.41
0.43
0.49
0.45
Attn
0.92
0.84
0.67
0.71
0.61
0.83
0.71
0.81
0.66
0.49
0.61
0.56
0.45
0.32
0.57
MLP
0.43
0.4
Attn
0.77
0.73
0.69
0.66
0.68
0.72
0.62
0.81
0.65
0.69
0.78
0.77
0.71
0.60
0.78
MLP
0.54
0.53
0.39
0.47
0.51
0.66
0.39
0.75
0.59
0.72
0.63
0.68
0.61
0.62
0.56
Attn
0.81
MLP
0.47
0.53
0.53
0.50
0.55
0.52
0.59
0.42
0.34
0.61
0.57
0.68
0.23
0.67
0.60
0..2
Attn
0.13
0.11
0.25
0.21
0.23
0.32
0.25
0.37
0.09
0.46
0.44
0.45
0.22
0.40
0.34
MLP
0.23
0.61
0.61
0.79
Attn
0.02
0.16
0.00
0.12
0.13
0.35
0.01
0.26
0.02
0.00
0.30
0.07
0.16
0.04
0.10
-she
Liked
to
think
being
surrounded
by
adult
aan
ooldr
ids
was
one
Figure 9. Token- Token-Wise Cosine Similarities Across Madules in I.J.MA-2-7B.
1.0
MLP
0.40
0.38
0.42
0.53
0.41
Attn 40
0.97
0.95
MLP 39
0.81
0.87
0.84
0.90
0.82
0.86
0.87
0.85
0.86
0.79
0.88
0.82
Attn 39
0.98
0.98
0.96
0.96
0.99
0.98
0.98
0.97
0.98
0.96
0.98
0.98
MLP 38
0.91
0.92
0.91
0.98
0.93
0.94
0.94
0.93
0.95
0.89
0.92
0.93
Attn 38
0.95
0.98
0.99
0.96
0.99
0.98
0.98
0.97
0.98
0.96
0.96
0.98
0.6
MLP
0.55
0.55
0.65
0.64
0.55
0.47
0.57
0.57
0.45
0.46
0.54
0.57
Attn
0.94
0.81
0.83
0.83
0.82
0.78
0.85
0.70
0.73
0.82
0.69
0.86
MLP
0.34
0.49
0.57
0.57
0.40
0.47
0.47
0.57
0.56
0.53
0.64
0.54
0.4
Attn
0.72
0.44
0.69
0.64
0.75
0.58
0.58
0.46
0.56
0.71
0.62
0.63
MLP
0.49
0.51
0.66
0.34
0.36
0.35
0.45
0.53
0.61
0.36
0.58
0.60
Attn
0.81
0.60
0.59
0.51
0.37
0.51
0.67
0.60
0.65
0.46
0.61
0.66
0.2
MLP
0.59
0.47
0.64
0.47
0.24
0.62
0.57
0.65
0.63
0.56
0.59
0.56
Attn
0.72
0.60
0.61
0.36
0.40
0.61
0.55
0.56
0.54
0.54
0.62
0.43
MLP
0.28
0.11
0.15
0.12
0.19
0.22
0.35
0.18
0.44
0.42
0.38
0.28
Attn
0.03
0.01
0.05
0.01
0.02
0.03
0.12
0.02
0.16
0,20
0.16
0.16
.are
you
being
.good
boy
2
Figure 10. Token- Wise Cosine Similarities Across Modules in LLaMA-2-13B
19

---

SkipGPT: Each Token is One of Kind
MLP
0.46
0.62
0.24
0.64
1.0
Attn
0.97
0.95
0.96
0.93
0.92
MLP 39
0.87
0.86
0.83
0.83
0.71
0.84
0.91
0.84
0.83
0.86
0.87
0.89
0.88
0.82
0.82
Attn 39
0.99
0.99
0.99
0.99
0.98
0.99
0.99
0.98
0.99
0.99
0.98
0.99
0.98
0.95
0.98
0.8
MLP 38
0.94
0.94
0.95
0.94
0.93
0.95
0.90
0.94
0.91
0.94
0.86
0.94
0.81
0.83
0.94
Attn 38
0.98
0.99
0.98
0.98
0.99
0.98
0.99
0.97
0.98
0.99
0.98
0.98
0.95
0.94
0.96
0.6
MLP
0.57
0.53
0.46
0.58
0.59
0.52
0.43
Attn
0.90
0.78
0.83
0.65
0.75
0.64
0.75
0.89
0.89
0.85
0.87
00.
MLP
0.36
0.45
0.39
0.36
0.37
0.46
0.59
0.43
0.53
0.53
0.56
0.51
0.59
0.47
0.46
Attn
0.72
0.66
0.63
0.72
0.62
0.51
0.77
0.79
0.67
0.61
0.55
0.72
0.60
0.76
0.72
MLP
0.49
0.50
0.44
0.46
0.42
0.61
0.65
0.24
0.64
0.68
0.66
0.54
0.67
0.15
0.21
Attn
0.82
0.71
0.71
0.74
0.51
0.61
0.75
0.74
0.67
0.61
0.65
0.66
0.76
0.53
0.53
0.2
MLP
0.46
0.53
0.48
0.58
0.52
0.69
0.63
0.26
0.63
0.71
0.53
0.16
0.55
0.34
0.36
Attn
0.52
0.29
0.21
0.34
0.32
0.62
0.65
0.36
0.63
0.53
0.63
0.24
0.63
0.49
0.37
MLP
0.21
0.26
0.24
0.33
0.30
0.24
0.35
0.48
0.44
0.48
0.41
0.46
0.44
0.49
0.58
Attn
0.01
0.03
0.04
0.10
0.02
0.01
0.16
0.10
0.21
0.17
0.32
0.18
0.26
0.07
0.31
but
Just
Lone
Jook
min
ion
sent
him
pract
Ically
Cat
at
onic
Figure II. Token-Wize Cosine Similarities Across Modules in LLaMA-2-13B.
20


Reviews:
[{"review_id": "O6n3I8tJEM", "reviewer": "Reviewer_hvhB", "summary": "The paper proposes a method to prune LLMs in horizontal (per token processing) and vertical (layer-wise) dimensions. It also provides a two-stage pruning pipeline in which first it trains a router given a fixed pre-trained LLM and then fine-tunes the model with LoRA adapters to recover the performance. Experiments with various benchmarks demonstrate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** Is the definition of sparsity in the paper appropriate and reflective of practical inference efficiency?\n\n**Response:** Our sparsity metric, based on skipped modules, is a simple and consistent proxy for dynamic pruning, though it doesn't directly reflect FLOPs. While actual efficiency depends on sequence length and module type, MLP and attention FLOPs are roughly comparable on average (e.g., in LLaMA2-7B), making our proxy a reasonable simplification. For fair comparisons, we match parameter sparsity with the pruning baseline. Since attention has fewer parameters and is skipped more often, this leads to lower actual FLOPs—especially for long sequences. We plan to include FLOP and latency metrics in future versions.\n\n**Q2:** Can the proposed method demonstrate real-world inference speedup, and are latency measurements compared to dense baselines reported?\n\n**Response:** Thank you for highlighting this important point. In practice, LLM inference consists of two phases: prefilling (processing the initial prompt) and decoding (generating tokens step by step).\n\n**(1) Prefilling phase**\n\nTo quantify the latency overhead introduced by our method, we conduct detailed timing analysis on an A800 GPU using a SkipGPT model (LLaMA2-7B) with 25% sparsity. For an 80-token input, the additional operations per layer introduced by SkipGPT include attention/MLP routing, argmax, and slicing. The average per-layer latency (in seconds) is as follows:\n| Operation                          | Avg. Time per Layer (s) |\n|-----------------------------------|--------------------------|\n| Attention Router                  | 0.000314                 |\n| Argmax                            | 0.000154                 |\n| MLP Router                        | 0.000202                 |\n| Slicing                           | 0.000140                 |\n| Grouped Computation (post-slice) | 0.057000                 |\n| **Total (SkipGPT)**              | **0.057964**             |\n| **Total (Dense)**                | **0.064000**             |\n\nThis yields an average ~10% reduction in per-layer latency during prefilling. We also report end-to-end prefilling latency across sequence lengths for both dense and 25% sparse models, showing the sparse model’s runtime as a fraction of the dense model’s:\n| Input Tokens | Dense (s) | SkipGPT (25%) (s) | Ratio (SkipGPT / Dense) |\n|--------------|-----------|-------------------|--------------------------|\n| 80           | 2.048     | 1.854             | 0.905                    |\n| 800          | 3.964     | 3.369             | 0.850                    |\n| 2000         | 22.618    | 20.862            | 0.922                    |\n\n**(2) Decoding phase**\n\nDue to token-by-token generation, SkipGPT often achieves greater-than-theoretical speedups. FlashAttention’s per-step overheads—like linear projections, RoPE, KV cache updates, and kernel setup—can’t be amortized, making attention the dominant cost. Since our router frequently skips attention, SkipGPT yields significant latency gains. The table below shows results for a 25% sparsity SkipGPT model (LLaMA2-7B) with an 80-token prompt.\n| Generated Tokens | Dense (s) | SkipGPT (s) | Theoretical Ratio | Actual Ratio |\n|------------------|-----------|-------------|--------------------|--------------|\n| 40   | 5.688| 4.402 | 0.75 | 0.77|\n| 200 | 13.298| 9.920       | 0.75  | 0.75         |\n| 800| 37.089    | 26.383      | 0.75 | 0.71         |\n\nAs shown above, the actual speedup closely matches or even exceeds the theoretical upper bound, particularly as the sequence length increases.\n\n**(3) Hardware-aware Optimization (Ongoing FPGA Work)**\n\nWe recognize that current GPUs struggle with token-wise dynamic sparsity due to memory bottlenecks and kernel overhead. To address this, we're developing a custom FPGA backend for SkipGPT that separates compute- and memory-bound phases for targeted optimization:\n- Prefilling (Compute-bound): We use a dataflow PE array with sparsity-aware scheduling to boost sparse matrix throughput and reduce idle cycles.\n\n\n- Decoding (Memory-bound): We design a hierarchical memory system with local buffers, lightweight scheduling, and KV cache-aware prefetching, outperforming GPU’s unified memory pipeline.\n\nUsing a 25% sparse SkipGPT (LLaMA2-7B), our FPGA prototype achieves the following normalized speedups:\n| Input Tokens | Relative Time |\n|--------------|----------------|\n| 40 | 0.753|\n| 200 | 0.732 |\n| 800| 0.715  |\n\nFor a fixed prompt length of 80 tokens, we measure the total decoding time for generating different output lengths:\n| Output Tokens | Relative Time |\n|---------------|----------------|\n| 40 | 0.723 |\n| 200 | 0.719|\n| 800 | 0.712 |\n\nLastly, we have shown that the sparsity ratio can be further reduced to **40%** without significantly copromising the model performance (See our response to RGDzU Q3 and Reviewer kVh7 Q1). We will add latency measurements and provide results under higher sparsity ratio in the revision.", "claims_and_evidence": "Yes, the claims about the need for vertical, horizontal, and two-staged pruning pipeline are properly supported by the experiments.", "methods_and_evaluation_criteria": "* I think definition of sparsity in the paper is problematic. In Eq. (7), the number of active blocks determines sparsity. However, there are more practical and principled metrics of sparsity like number of active parameters in MoE models or MACs/FLOPs in the model pruning literature that can better reflect inference efficiency. \n\n* Following the previous point, if the definition of in Tab. 1 be the same as Eq. (7), it makes it difficult to determine practical usefulness of the proposed method. Although the idea of fine-grained resource allocation to tokens is intuitive, it is usually hard to gain proportional inference latency speed up due to the required slicing operations in practice. Also, the inference latency for different blocks like attention vs MLPs depends on factors like the number of tokens (like the fact that attention is quadratic w.r.t the sequence length) and the software package (for instance Flash Attention is highly efficient on H100 GPUs) for the implementation. Therefore, I believe that the paper needs to report inference latency values for the SkipGPT vs the dense baseline to better demonstrate the practical significance of the proposed method.", "theoretical_claims": "The paper has no theoretical claims.", "experimental_designs_or_analyses": "Yes, the paper experiments with two variants of LLaMA models on well-known benchmarks.", "relation_to_broader_scientific_literature": "The paper makes the pruning more fine-grained in terms of depth pruning and token processing. On the depth pruning side, it prunes the attention layers and MLP layers in the transformer blocks separately. On the token processing side, it selects whether each token be processed by a layer or not.", "essential_references_not_discussed": "I don't know any essential reference not discussed.", "other_strengths_and_weaknesses": "Please check the sections above.", "other_comments_or_suggestions": "I recommend that the paper provide inference latency numbers for the pruned model vs the baseline dense model, and if possible with baseline methods. As I mentioned above, the reduction in per token computation may not necessarily lead to proportional inference latency reduction. I will raise my score if the authors can provide compelling evidence in this regard.", "questions_for_authors": "Please check the sections above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pqnmmxDv1z", "reviewer": "Reviewer_GDzU", "summary": "The authors propose SkipGPT by addressing the challenges of existing dynamic pruning methods, namely horizontal dynamics, vertical dynamics, and the training paradigm. In other words, they redefine sparsity, separate MLP and self-attention within layers, and train routing and LoRA in a two-stage process.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: The phrase \"novel sparsity concept\" seems exaggerated.\n\n**Response**: We appreciate the reviewer’s feedback regarding the “novel sparsity concept.” Our intention was not to overstate novelty in this regard. Rather, we aimed to highlight that SkipGPT enables fully dynamic layer skipping, where both the number of layers each token passes through and the participating tokens in each module can vary dynamically across the sequence.\n\n That said, we agree that similar ideas of global token- and layer-level budget allocation have been explored in prior works such as PyramidKV. In our case, this sparsity definition is primarily used as a control mechanism during training to specify and regulate the sparsity target, and is not the main technical contribution of our paper. We will revise the wording in the final version to better reflect this and avoid overstating novelty.\n\n**Q2:** Can the method be evaluated on recent LLMs such as LLaMA-3 or Gemma-2?\n\n**Response:** Thank you for the suggestion. We refer you to our response to Reviewer kVh7 Q1, where we present detailed results and analysis on LLaMA3-8B.\n\n**Q3**: Does the method maintain strong performance under higher pruning ratios beyond 30%?\n\n**Response:** We appreciate the reviewer’s attention to the method’s robustness under higher pruning ratios. While we have already demonstrated in Q2 that SkipGPT remains effective at pruning ratios beyond 30% using LLaMA3-8B, we further conducted a comprehensive set of experiments on LLaMA2-7B, the model originally used in our paper, under a 40% sparsity setting.\n\nThe results after router tuning are shown below:\n\n|Model|OpenBookQA|Winogrande|PIQA|HellaSwag|BoolQ|ARC-Easy|ARC-Challenge|Avg.Acc|WikiText|PTB|Avg.PPL|\n|------------------------|------------|------------|-------|-----------|--------|-----------|----------------|-----------|----------|---------|-----------|\n|Dense|44.2|74.19|78.07|78.93|71.62|81.36|52.47|68.69|5.47|20.83|13.15|\n|ShortGPT|34.4|62.83|59.58|45.43|62.17|44.82|30.80|48.58|79.47|156.67|118.07|\n|JointLayerDrop|30.2|57.85|60.61|41.32|62.17|36.91|30.72|45.68|126.88|302.04|214.46|\n|LLM-Pruner|32.2|53.35|65.67|42.41|59.97|32.24|26.37|44.60|46.34|191.31|118.83|\n|SkipGPT-RT|34.2|55.72|63.87|51.09|56.30|56.94|31.83|50.00|16.69|91.00|53.85|\n\n\nWe observe that our method outperforms all pruning baselines on both downstream accuracy and language modeling perplexity under 40% sparsity, demonstrating its superior adaptability and robustness. To further validate effectiveness, we also apply LoRA fine-tuning under the same 40% sparsity setting. The results are summarized below:\n\n|Model|OpenBookQA|Winogrande|PIQA|HellaSwag|BoolQ|ARC-Easy|ARC-Challenge|Avg.Acc|WikiText|PTB|Avg.PPL|\n|------------------|------------|------------|-------|-----------|--------|-----------|----------------|-----------|----------|---------|-----------|\n|DenseLoRA|44.8|74.27|78.02|78.96|79.02|81.73|53.07|69.98|5.48|20.58|13.03|\n|ShortGPT,LoRA|34.0|65.90|64.91|57.30|63.46|55.60|33.02|53.46|14.78|49.71|32.25|\n|JointLayerDrop,LoRA|35.4|63.22|69.42|62.06|69.91|62.12|36.18|56.90|11.08|42.19|25.64|\n|LLM-Pruner,LoRA|32.2|53.51|65.72|42.42|60.00|32.24|26.37|44.64|46.33|191.31|118.82|\n|SkipGPT-RT-L|43.0|72.93|77.09|76.63|76.88|81.48|52.39|68.63|6.00|31.05|18.53|\n\nAs shown, after LoRA fine-tuning, our method nearly fully recovers the performance of the dense model.\n\n**Q4:** How does the method perform on long-context benchmarks such as LongBench?\n\n**Response:** Thank you for the question. To evaluate our method on long-context understanding, we conducted experiments on LongBench. Specifically, we applied router tuning followed by supervised fine-tuning (SFT) on the LLaMA 3.1 8B base model using the Alpaca dataset. We compare our model (SkipGPT-RT-SFT, with 40% sparsity) against the official LLaMA-3.1-8B-Instruct, as well as a strong pruning baseline ShortGPT, which uses the same SFT configuration as our method. The following table summarizes the results (without CoT prompting):\n\n|Model|Overall(%)|Easy(%)|Hard(%)|Short(%)|Medium(%)|Long(%)|\n|----------------------|-------------|----------|----------|------------|--------------|-----------|\n|LLaMA-3.1-8B-Instruct|29.8|30.7|29.6|35.0|27.9|25.9|\n|SkipGPT-RT-SFT|28.6|29.3|28.3|32.7|26.5|26.3|\n|ShortGPT+SFT|25.5|25.7|26.1|24.3|26.3|25.3|\n\nAlthough LongBench remains a challenging benchmark, our method significantly outperforms the pruning baseline (ShortGPT) across all difficulty levels and context lengths. While SkipGPT-RT-SFT does not fully match the dense model’s performance, it shows strong capability in long-context scenarios—even under 40% sparsity—highlighting the effectiveness of our compressed approach.\n\n**Q5**: Can you provide actual execution time comparisons rather than just theoretical speedup ratios?\n\n**Response:** Thank you for your question. Please refer to our response to Reviewer hvhB Q2.", "claims_and_evidence": "Yes, but certain parts appear to be somewhat overextended.\n\n- For instance, The phrase \"novel sparsity concept\" seems exaggerated. It appears to be merely a removal of certain constraints. In the field that deals with token eviction, it is already standard practice to consider the token axis as a whole. Moreover, the papers like pyramidKV [1] even addresses the layer axis in relation to the overall budget.\n\n\n---\n[1] Cai, Zefan, et al. \"Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling.\" arXiv preprint arXiv:2406.02069 (2024).", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "None.", "experimental_designs_or_analyses": "- Please provide the results of recent LLMs such as llama-3, gemma-2, etc.\n- I understand that the structured pruning method maintains meaningful performance up to approximately 30%. However, it would be preferable if it could deliver performance at even higher ratios.\n- It would be beneficial to include a comparison of the performance for long context like LongBench.\n- Please specify the actual execution time of each algorithm (beyond the theoretical ratio).", "relation_to_broader_scientific_literature": "This is a study related to the lightweight optimization of large language models (LLMs). But for me, proposed three approach (redefined sparsity, decoupling, two-stage training) are not novel, but it is practical.", "essential_references_not_discussed": "Please refer to \"Claims and Evidence\"", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iLwH5vhUoT", "reviewer": "Reviewer_kVh7", "summary": "The paper focuses on speeding up inference of large language models (LLMs) by skipping different layers, at the granularity of attention or feed forward network (FFN) layers, for different tokens. It does this by introducing a router (a simple MLP layer with Gumble-Softmax) for each attention and FFN layer. It first trains the router while freezing the model, then introduces LoRA adapters and trains both LoRA and router parameters. The results show that when skipping 25% of parameters, the proposed solution is better than many existing pruning or layer skipping approaches on various tasks and model sizes (Llama2 7B and 13B).\n\n## Update after Rebuttal \nI have read all the reviews and rebuttals and I would like to keep my score. With a very limited training budget, the paper exceeds SOTA in layer skipping and the insights and proposal in the paper are important for the academic community. \n\n2 questions just popped to me while doing the final revision are:\n1. How KV cache is handled for skipped layers? When token i skips a certain layer but token i+1 runs the same layer, does the query of token i+1 attend to the key and value of token i for that layer?\n2. For prefill (that I assume was used to measure perplexity as well as multiple choice tasks like HellaSwag, etc.), were you skipping different layers for different tokens as well?", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: The reviewer suggests including experiments on LLaMA3, which was released in April 2024. \n\n**Response**:  To address the reviewer’s suggestion, we conducted additional experiments on LLaMA3-8B, evaluating SkipGPT under both 25% and 40% sparsity settings. Due to space constraints, we compare SkipGPT with two of the most competitive baselines: ShortGPT and Shortened LLaMA (PPL). The table below presents the results under 25% sparsity, comparing SkipGPT-RT with the baselines across several downstream tasks and language modeling benchmarks:\n\n|Model|OpenBookQA|Winogrande|PIQA|HellaSwag|BoolQ|ARC-Easy|ARC-Challenge|Avg. Acc|WikiText|PTB|Avg. PPL|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|Dense|44.8|77.51|80.03|81.95|82.14|84.85|57.59|72.69|6.24|10.58|8.41|\n|ShortGPT|29.2|52.25|60.44|30.80|37.64|36.70|30.80|39.69|2796.24|2799.46|2797.85|\n|SkipGPT|44.2|75.69|78.07|76.87|74.06|82.44|53.41|69.25|16.47|26.91|21.69|\n|Shortened LLaMA (PPL)|33.6|58.64|71.54|55.94|39.69|59.81|31.48|50.10|15.00|23.86|19.43|\n\nAs shown, static pruning methods like ShortGPT suffer significant degradation—e.g., ShortGPT shows extremely poor language modeling (PPL > 2700). In contrast, SkipGPT-RT maintains performance close to the dense model, highlighting the importance of dynamic pruning for large, well-trained models like LLaMA3-8B, where full layer removal is no longer viable.\nWe observe the same trend under 40% sparsity, where SkipGPT continues to outperform static baselines by a large margin:\n\n|Model|OpenBookQA|Winogrande|PIQA|HellaSwag|BoolQ|ARC-Easy|ARC-Challenge|Avg. Acc|WikiText|PTB|Avg. PPL|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|Dense|44.8|77.51|80.03|81.95|82.14|84.85|57.59|72.69|6.24|10.58|8.41|\n|ShortGPT|29.1|53.51|60.28|35.49|56.02|34.56|30.34|42.76|79856.66|125507.27|102681.97|\n|SkipGPT|38.0|59.35|73.34|64.36|60.37|77.53|45.65|59.80|71.25|48.05|59.65|\n|Shortened LLaMA (PPL)|31.3|57.45|62.35|50.52|36.93|53.76|31.45|46.23|157.01|196.04|176.53|\n\nHere, ShortGPT fails completely as a language model, producing meaningless outputs. In contrast, SkipGPT-RT maintains over 80% of the dense model’s performance, despite having no fine-tuning beyond router tuning.\nFor the LoRA fine-tuning phase, Shortened LLaMA (PPL) fails to converge and is thus omitted. We compare SkipGPT-RT-L (router tuning + LoRA) with ShortGPT under 25% sparsity:\n\n|Model|OpenBookQA|Winogrande|PIQA|HellaSwag|BoolQ|ARC-Easy|ARC-Challenge|Avg. Acc|WikiText|PTB|Avg. PPL|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|Dense LoRA|44.9|77.68|80.22|81.86|82.23|84.92|57.89|72.81|6.13|10.44|8.29|\n|ShortGPT|37.8|74.27|72.63|70.53|71.19|69.21|47.78|63.34|11.13|16.64|13.89|\n|SkipGPT-RT-L|42.6|77.03|79.97|82.13|82.84|84.47|57.08|72.30|7.10|11.70|9.40|\n\nSkipGPT-RT-L effectively recovers nearly all the performance of the dense model after LoRA adaptation.\nThis conclusion holds under 40% sparsity as well:\n\n|Model|OpenBookQA|Winogrande|PIQA|HellaSwag|BoolQ|ARC-Easy|ARC-Challenge|Avg. Acc|WikiText|PTB|Avg. PPL|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|Dense LoRA|44.9|77.68|80.22|81.86|82.23|84.92|57.89|72.81|6.13|10.44|8.29|\n|ShortGPT|31.0|69.13|67.57|67.24|65.84|62.31|37.20|57.18|18.35|30.65|24.50|\n|SkipGPT-RT-L|40.8|74.98|79.16|80.33|80.00|82.74|54.01|70.29|7.70|13.10|10.40|\n\n**Q2**: The reviewer recommends including results on generation tasks to further evaluate the effectiveness of the method.\n\n**Response**: Thank you for the helpful suggestion. To evaluate our method on generation tasks, we conducted zero-shot experiments on GSM8K (flexible match) and MBPP using LLaMA3-8B under a 40% sparsity setting. We compared SkipGPT-RT-L (router tuning + LoRA) with a strong baseline ShortGPT + LoRA, using the same sparsity level. The results are as follows:\n\n|Models|GSM8K (%)|MBPP (%)|\n|-|-|-|\n|LLaMA-3.1-8B|26.23|30.6|\n|SkipGPT-RT-L|15.34|21.5|\n|ShortGPT + LoRA|3.42|2.12|\n\nWhile there is some drop in performance compared to the dense model, SkipGPT still significantly outperforms the pruning baseline, demonstrating its advantage in generation tasks. \n\n**Q3:** Can you measure end-to-end latency with and without the router?\n\n**Response:** Please refer to our response to Reviewer hvhB Q2. The router adds negligible overhead, as shown in our end-to-end latency and module-level breakdown. We’ll highlight this in the revision.\n\n**Q4:** The reviewer asks for clarification on the LoRA configuration used in the experiments.\n\n**Response:** Thank you for the suggestion. In our experiments, we use a LoRA rank of 16, applied to \\( W_q \\), \\( W_v \\), and \\( W_{\\text{gate}} \\). We will clearly specify these details in the main text in the revision.\n\n**Q5:** Do Table 1 ratios include router/LoRA parameters?\n\n**Response:** Table 1 includes router parameters. Table 2 includes both router and LoRA. We'll clarify in the revision.", "claims_and_evidence": "- Authors made detailed comparison with large number of static and dynamic pruning methods on 2 model sizes and show they perform better than most of them on most tasks\n- Authors provided multiple ablation studies to support their claims", "methods_and_evaluation_criteria": "- Since this paper was submitted in 2025, I believe it should also consist of experiments on Llama3 that was open sourced in April 2024.\nI know there is a risk of an accuracy drop because Llama3 is harder to compress but that's fine.\n- I also recommend to add results for experiments on generation tasks such as GSM8K, Natural Questions, HumanEval, MBPP", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "- Paper provided detailed ablation studies to prove the importance of each component of the solution: effectiveness of transformer routing vs attention/ffn routing, effectiveness of two-stage training, routing analysis\n- From my previous experience, perplexity is a better evaluation metric of a language model's performance compared to accuracies of so-called commonsense reasoning tasks that are multiple-choice questions. Hence, I would pay more attention to perplexity\n- Line 312 (Second Column): \"For long contexts, the FLOPs of attention surpass those of the MLP. As a result, SkipGPT-RT achieves lower computational overhead than other baselines.\" Where in the Tables do we see results of long-context evaluation?\n- Please also specify the context length when perplexity was evaluated", "relation_to_broader_scientific_literature": "- The paper made a concise summary of related work and compared with a large number of static layer pruning, dynamic layer pruning, as well as embedding dimension pruning", "essential_references_not_discussed": "- The paper made sufficient references and discussed a relatively large number of papers", "other_strengths_and_weaknesses": "- Strengths:\n  - Novelty: \"To overcome this limitation, we introduce a novel sparsity concept that defines computation budgets across the entire forward pass, rather than being confined to layer-wise or token-wise constraints\"\n  - Approach:\n     - Very lightweight router\n     - Easy and fast: requires a single A800 GPU and 4 hours of finetuning\n  - Useful insights presented, such as :\n    - \"(1) Attention modules exhibit greater redundancy than MLP modules. (2) As context length increases, later tokens demand more attention computation but less MLP processing.\"\n    - Instability introduced by joint training of untrained routers with trained parameters\n  - Writing style:\n    - Authors build the motivation of their approach in an intriguing and clear way.\n    - Each component of the solution is explained in a clear and detailed manner to build up the full picture", "other_comments_or_suggestions": "- Figure 1: \"Static Structured Pruning\" sub-figure doesn't seem static. Different tokens skip different sub-layers in both \"SkipGPT\" and \"Static Structured Pruning\" sub-figures, so they both seem dynamic. In fact, the 2 sub-figures are almost identical to each other.\n- Line 190: \"Gumbel distribution Gumbel(0, 1)\". Is the additional \"Gumbel\" here a typo?\n- Equation 4: I suggest to write down the explicit definitions of y_soft and y_hard\n- Line 210 (Right Column): I recommend adding subscript l to W_{theta}\n- Line 226: please write the definition of g_l as a function of r_l\n- Can you measure end-to-end latency with and without the router?\n- Line 246 (Right Column): Please briefly specify in the main body of the paper basic configuration of LoRA finetuning: the rank used, and which weight matrices (Wk, Wv, Wq, Wo, Wup, Wdown?) it was applied on\n- For Static Pruning Baselines, you may consider comparing with the pretrained checkpoints of LayerSkip Llama2 7B and Llama2 13B checkpoints ( https://huggingface.co/collections/facebook/layerskip-666b25c50c8ae90e1965727a ) by removing the last 25% of the layers\n- Figure 4: Too many curves with similar colors. To make it easier, I suggest to order the plots in the legend in the same order as the Language Modeling Loss in the last training step", "questions_for_authors": "- Table 1: Does the ratio of parameters used put into consideration the router and LoRA parameter count?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}]

Author rebuttal:

Meta-review: This paper introduces a dynamic pruning framework for LLMs that addresses three key challenges: redefining sparsity through global optimization across token and layer dimensions, decoupling MLP and attention components for selective pruning, and implementing a two-stage training paradigm with router optimization followed by LoRA fine-tuning.

During the rebuttal period, reviewers raised concerns about the sparsity definition, practical efficiency measurements, evaluations on recent models, performance at higher sparsity ratios, and long-context capabilities. The authors responded comprehensively with detailed latency measurements showing reduction in prefill and decoding latency, extensive experiments on Llama 8B demonstrating superior performance over baselines, evaluations on Long-Bench showing strong long-context understanding, and additional tests on generation tasks like GSM8K. Reviewer hvhB updated their score positively.

Key strengths include its novel integration of different pruning strategies, strong empirical results across multiple model sizes and tasks, measurable efficiency gains in real-world scenarios, lightweight training requirements, comprehensive ablations, and valuable insights into component redundancy. While the approach shows limitations at higher sparsity ratios, builds upon some existing techniques, and initially lacked evaluations on newer models, it ultimately represents a good paper worth sharing with the broader community.
Decision: Accept (poster)