Pruning as a Cooperative Game: Surrogate-Assisted Layer Contribution Estimation for Large Language Models
Authors: Xuan Ding
Pengyu Tong
Ranjie Duan
Yunjian Zhang
Rui Sun
Yao Zhu
Venue: ICLR 2026 Poster (2026)
Keywords: Layer-wise Pruning
Cooperative Game Theory
Shapley Value Approximation
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We frame LLM pruning as a cooperative game and use surrogate-assisted Shapley value estimation to identify and remove redundant layers efficiently while preserving performance.

Abstract: While large language models (LLMs) demonstrate impressive performance across various tasks, their deployment in real-world scenarios is still constrained by high computational demands. Layer-wise pruning, a commonly employed strategy to mitigate inference costs, can partially address this challenge. However, existing approaches generally depend on static heuristic rules and fail to account for the interdependencies among layers, thereby limiting the effectiveness of the pruning process. To this end, this paper proposes a game-theoretic framework that formulates layer pruning as a cooperative game in which each layer acts as a player and model performance serves as the utility. As computing exact Shapley values is computationally infeasible for large language models (LLMs), we propose using a lightweight surrogate network to estimate layer-wise marginal contributions. This network can predict LLM performance for arbitrary layer combinations at a low computational cost. Additionally, we employ stratified Monte Carlo mask sampling to further reduce the cost of Sharpley value estimation. This approach captures inter-layer dependencies and dynamically identifies critical layers for pruning. Extensive experiments demonstrate the consistent superiority of our method in terms of perplexity and zero-shot accuracy, achieving more efficient and effective layer-wise pruning for large language models.

Published as a conference paper at ICLR 2026
PRUNING AS A COOPERATIVE GAME: SURROGATE- ASSISTED LAYER CONTRIBUTION ESTIMATION FOR LARGE LANGUAGE MODELS
Xuan Ding
Shenzhen Future Network of Intelligence Institute Guangdong Provincial Key Laboratory of Future Networks of Intelligence The Chinese University of Hong Kong (Shenzhen) 20232201111 90mail wn...du..u.....ee... .c..
Pengyu Tong Beijing Normal University 202522011202685 bnu. edu. on
Ranjie Duan Alibaba Group ran (ennjiddun@@gaaaa com
Yunjian Zhang
University of Chinese Academy of Sciences sdtczyj jaally/@mma.....mm
Rui Sun
Shenzhen Future Network of Intelligence Institute Guangdong Provincial Key Laboratory of Future Networks of Intelligence
The Chinese University of Hong Kong (Shenzhen) rui munelink... cuhk. edu.
Yao Zhu
Zhejiang University ee ee.zhuy@zju. edu. cn
ABSTRACT
While large language models (LLMs) demonstrate impressive performance across various tasks, their deployment in real-world scenarios is still constrained by high computational demands. Layer-wisc pruning, commonly employed strategy mitigate inference costs, can partially address this challenge. However, existing approaches generally depend static heuristic rules and fail to account for the interdependencies among layers, thereby limiting the effectiveness of the prun- ing process. To this end, this paper proposes game-theoretic framework that formulates layer pruning as a cooperative game in which each layer acts as a player and model performance serves as the utility. As computing exact Shap- ley values is computationally infeasible for large language models (LLMss), we propose using lightweight surrogate network estimate layer-wise marginal contributions. This network can predict LLM performance for arbitrary layer combinations at a low computational cost. Additionally, we employ stratified Monte Carlo mask sampling to further reduce the cost of Sharpley value esti- mation. This approach captures inter-layer dependencies and dynamically identi- fies critical layers for pruning. Extensive experiments demonstrate the consistent superiority of our method in terms perplexity and zero-shot accuracy, achiev- ing more efficient and effective layer-wise pruning for large language models. The code is available at: bittps://qit / thub. com/ 920927 / /rrun/ ng_ As A L.Coopprraivve Game
corresponding author

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Large language models (LLMs) achieve state-of-the-art performance across a wide range of tasks (Chen et al. 2023: Duan et al. 2024 Zhu et al. 2025). but their massive computational and memory requirements pose significant challenges for practical deployment (Wang et al., 2024aj [unn et al. 2024a). This has prompted extensive research into model compression techniques. Among these, layer pruning, which removes entire transformer layers, stands out as an effective method for re- ducing inference cost. Compared to width pruning, depth pruning offers superior throughput and inference speed, while also being easier to implement than other compression methods, making an attractive approach for large-scale model compression (Kim al. 2024).
Existing deep pruning methods typically assign an importance score to each layer to determine the pruning order. These scores are often based on heuristics such weight magnitudes, activation norms (Filters' Importance 2016). or sensitivity analysis (Men et al., 22224] Kim et al. 2024). as- suming that layer importance is fixed and independent. However, our experiments reveal that layer importance is context-dependent. In single-layer pruning (Fig. [a), the rankings of early and late layers remain relatively stable, whereas the rankings of middle layers fluctuate significantly. When extended to multi-layer pruning (Fig. (F) this volatility is further amplified, with some layers show- ing strong fluctuations throughout the pruning process. These observations highlight the dynamic interdependencies between layers: pruning one layer can alter the relative importance of others, and evaluating layers in isolation often results in suboptimal pruning decisions. Previous studies have partially considered interactions between layers, such merging redundant layers (Ding al., 2025b) or progressively pruning less important ones during training (Song et al.] 2024). However, these approaches often fail to find the globally optimal layer set. For example, pruning layers quentially based on individual importance may not yield the best two-layer combination, because the optimal strategy could involve simultaneously removing a pair of layers that are not the least important individually (see Tab.
Single-Layer Pruning
Multi-Layer Pruning
Pruiing Cases
TO JOYES
Figure 1: Layer importance is context-dependent under pruning. Both plots are based on the change in PPL before and after pruning on the BookCorpus dataset rank layer importance. (Left) Bar plot showing the ARank changes for random single-layer pruning, highlighting that some layers' importance shifts more dramatically when others are pruned. (Right) Line plot showing the ARank changes during multi-layer pruning, where the lowest-ranked layer is removed at each step. The five most volatile layers are highlighted in darker colors, reflecting their fluctuating importance.
Scheme
Deleted Layers
Post-deletion PPL
Explanation
Layer 27
14.9845
Delete the least important layer
Single Deletion
Layer 10
14.9915
Delete the second least important layer
Layer 11
15.0154
Delete the third least important layer
Scheme
Layer 27 Layer 10
15.4535
Delete the two least important layers
Scheme
Layer 27 Layer
15.4535
Delate the least important layer, re-test, then delete the
Optimal Combo
Layer Layer
15.4279
Delete a pair layers accounting for inter-layer interactions
Table 1:
Summary of layer deletion schemes and their corresponding PPL values on Book- Corpus. Both pruning based on static importance and re-calculated importance after each deletion may not always lead to optimal performance.
2

---

Published as a conference paper at ICLR 2026
To address the limitations of static heuristics and explicitly capture the dynamic interdependencies among layers, we formulate LLM pruning as a cooperative game, where each Transformer layer is treated player with the model's performance defining the utility. In this setting, a layer's contri- bution is inherently context-dependent, shaped by its interactions with other layers. While Shapley values offer principled way quantify such contributions, their exact computation is intractable for large-scale models due to the exponential number of possible layer combinations. To make this feasible, we propose a two-stage approximation strategy aligned with cooperative game theory. In the first stage, we generate diverse pruning masks through stratified Monte Carlo sampling with con- trolled Hamming weights, and evaluate them on calibration data to measure perplexity (PPL). The performance gap between each pruned model and the original model provides supervision signals for learning. In the second stage, we train a lightweight surrogate network to predict these per- formance drops for unseen masks, enabling efficient estimation of marginal contributions without repeated full-model evaluations. Once trained, the surrogate allows us to estimate Shapley values from large pool of candidate masks. This design preserves inter-layer dependencies, adaptively identifies critical layers, and scales effectively to pruning large language models.
Extensive experiments language modeling and downstream tasks demonstrate the effectiveness of our method. Specifically, we report perplexity on WikiText, PTB, and C4, and assess inference performance across eight zero-shot benchmarks and adversarial reasoning robustness metric. Compared to depth-wise and width-wise pruning baselines, our method achieves lower perplex- ity, higher accuracy, and favorable trade-offs in speed and throughput. Furthermore, we show that our framework extends beyond Transformer-base LLMs, demonstrating strong generality on non- Transformer architectures, and can be seamlessly combined with quantization deliver additional efficiency gains.
In summary, our contributions are as follows:
We rethink LLM pruning from a game-theoretic perspective, treating layers as interdepen- dent players and revealing inter-layer dependencies that static heuristics fail capture.
We propose scalable approximation framework that leverages stratified Monte Carlo mask sampling and a lightweight surrogate network, enabling efficient Shapley-based estimation of layer contributions in large LLMs.
We validate our method on language modeling tasks and zero-shot benchmarks, showing consistent improvements over strong pruning baselines across diverse architectures.
2 RELATED WORK
2.1 PRUNING METHODS FOR LARGE LANGUAGE MODELS
The rapid growth of large language models (LLMs) has led to the development of various com- pression techniques, including quantization (Fratarr al. 2023 Dettmers al. 2022). knowledge distillation (Fu et al., 2023] Hsich et al., (023), tensor decomposition (Wang et al. 2024b; Ding et al. 2025a). and pruming. Pruning reduces inference costs by removing model components with- out complex retraining. It has evolved from unstructured sparsity (removing individual weights) to structured sparsity (pruning entire neurons, attention heads, layers). For example, SparseGPT uses the OBS error formula to assess weight importance and decide whether to prune, while ad- dressing the challenge of non-structured sparsity on real hardware through semi-structured pruning (Frantar & Alistarh (023). Methods like Wanda combine weight magnitudes with input feature norms for layer selection (Sun al., (023). LLM-Pruner and FLAP focus on pruning coupled structures, such attention heads, to reduce network width while maintaining the number of layers (Ma et al. 2023) An et al. (0244). These methods show pruning's 's potential for deploying LLMS on resource-constrained devices with minimal performance loss.
2.2 MEASURING LAYER CONTRIBUTIONS
Direct layer- layer-wise pruning methods are straightforward and effective, offering better inference speed and throughput. Most of these methods in LLM pruming use heuristics like weight magnitude, gradi- ent sensitivity, or activation statistics to assess layer importance. ShortGPT introduces Block Influ- ence (BI) to quantify the importance of each layer and prunes redundant layers (Men et al. 2024
3

---

Published as a conference paper at ICLR 2026
while LAYERIF tracks the sensitivity of different layers to training data using influence function Askari et al. 2025). Shortened-LLaMA on the other hand, calculates weight importance scores at the output neuron level using Taylor+ and PPL metrics, assessing block-level importance (Kim et al. 2024). SI.BB prunes redundant transformer blocks iteratively using Metric3, integrating smoothly into the forward pass (Song al. 2024). In contrast, CALM, Mixture-of-Depths, and SkipDecode dynamically allocate computation resources based on context, adjusting compute expenditure to optimize efficiency (Del Corro et al. 2023 Raposo al. 2024 Schuster al. 2022)
Cooperative game theory studies how multiple rational decision-makers form alliances to achieve common goals. This makes it well-suited for assessing the importance of different layers LLMs. The GTAP method, for instance, treats neurons cooperative agents and uses power indices to eval- uate importance (Diaz-Ortiz Jr et al., 2023). While computational complexity limits its application to ultra-large models, the principles offer valuable perspective for pruning. /Zhang et al. (2024) applied the Shapley value to model interpretation, noting its computational infeasibility for LLMs. They proposed using early truncation similar SV-NUP methods, considering only the influence of adjacent layers for non-uniform pruning (Sun et al. 2025).
METHOD
3.1 LAYER PRUNING AS A COOPERATIVE GAME
Layer pruning in large language models (LLMs) is inherently challenging because the contribution of each Transformer layer is not independent: conventional importance rankings often ignore inter- layer dependencies, leading to suboptimal pruning. To address this, we formulate layer pruning as cooperative game, where each layer acts player and the model's performance defines the utility function.
Formally, let C 21. L} denote the set of layers in LLM with L layers. For any subset of layers $ _ L, let (S) be the model obtained by retaining only layers S, and let u(S) denote its performance measured by perplexity (PPL) calibration data, where lower PPL corresponds to higher utility. The marginal contribution of layer i to subset $ is
- (i)) u(S).
(I)
While the Shapley value provides theoretically grounded measure of layer importance, computing exactly is infeasible due to the exponential number of subsets. This motivates our swo-stage approximation framework, which efficiently estimates layer contributions while preserving inter- layer dependencies.
3.2 ALGORITIIM OVERVIEW
Figure 2 illustrates the overall pipeline of our approach. The goal is efficiently estimate the contribution of each layer to the model's performance, enabling informed pruning decisions while preserving inter-layer dependencies. Our framework proceeds in two stages:
1. Mask Generation and Performance Evaluation: In the first stage, generate diverse pruning masks using stratified Monte Carlo sampling with controlled Hamming weights. Each mask defines a subset of layers to retain, and we evaluate the corresponding pruned models on calibration data to obtain performance scores (perplexity differences). These supervision signals for learning.
scores
2. Surrogate Training and Shapley Value Estimation: the second stage, we train a lightweight surrogate network fe to predict the performance of unseen masks. Once trained, the surrogate enables efficient estimation of each layer's marginal contribution, which is then aggregated compute approximate Shapley values. Finally, layers with the lowest contributions are pruned.
For clarity, the complete procedure is also provided in pseudo-code in Algorithm 1 in Appendix, B.3
4

---

Published as a conference paper at ICLR 2026
Large Language Model
PPhorig
Remove
Calibration Data
Score; PPhariy PPlyney
Scorey PPlusg PPlyeee
Score, PPhari / PPlpoens
Modd
MSE Loss
Binary Mask
iila.
Score
donninngnggee fonctte aumber uecaned byans
Surrogute Model
marginnal contribation of layer 4,55) wtS (() u(5)
Mante re
Mask oz
and Stuge
Figure 2: Framework of our method. Both stages use stratified Monte Carlo masks with con- trolled Hamming weight. Stage one uses calibration data to compute PPL-based scores for training lightweight surrogate network, and stage two uses the surrogate to efficiently compute Shapley- based layer importance for scalable LLM pruning.
3.3 STAGE ONE: MASK GENERATION AND PERFORMANCE EVALUATION
Directly computing Shapley values requires evaluating all 2 layer subsets, which is computation- ally infeasible for LLMs. In the first stage, we approximate layer contributions via stratified Monte Carlo sampling of binary pruning masks. Each mask m E {0, 1) denotes subset of retained lay- ers, where m; indicates that layer preserved. The corresponding pruned model M (mm)
To quantify the contribution of each mask, we compute a performance score as the ratio of the original perplexity to the pruned model's perplexity:
PPLorig
(m)
PPL(M
where smm) closer to indicates better preservation of the original model's performance.
To ensure balanced exploration across pruning ratios, we design a stratified sampling strategy based on Hamming weight (i.c., number of retained layers k(m) = -'e m.). Let K. = {(I,.... | be the set of target weights. For each weight ky E K. we draw Nk, masks that retain exactly ky layers. Given total budget of N masks enforce Σ} NN, N and in practice set NN, [N/||]] (distributing any remainder the first few strata). The t-th mask sampled within the stratum of Hamming weight k, is denoted by m (k) and drawn as
m (k) Uniform m E 10. k(m) = ky) t= Nay, == 1.... K.]
(3)
This stratified sampling ensures balanced coverage across pruning ratios, while Monte Carlo ran- domness captures diverse interactions among layers. The resulting masks and their corresponding scores form the training data for the surrogate network in Stage Two.
3.4 STAGE Two: SURROGATE TRAINING AND SHAPLEY VALUE ESTIMATION
Direct evaluation of s(m) for every pruned model is computationally prohibitive. To address this, introduce lightweight surrogate network fe fo(m). implemented two-layer ferw-forward network, that predicts the performance score of any binary mask m. This surrogate decouples expensive full-model inference from large-scale Shapley value estimation.
5

---

Published as a conference paper at ICLR 2026
Training the surrogate The surrogate is trained on the limited set of masks and their true scores obtained in Stage One, denoted as {(m s (+))))))) We optimize the mean squared error:
(()) Σ (J((m))ss(m)))
where N is the total number of training masks. Once trained, So generalizes to unseen masks, enabling efficient prediction of model performance without costly full-model evaluations.
Approximating Shapley values Using the surrogate, we approximate each layer's Shapley value via stratified Monte Carlo sampling in Eq. (3) For layer i, we repeatedly sample Q binary masks m (kg.9) and compute the marginal contribution of layer
Σ (Jo(m( Uti)) - (m)))))))))))))
(5)
This efficiently estimates layer-wise contributions while preserving inter-layer dependencies, as the surrogate captures performance shifts under diverse layer coalitions.
Layer pruning Finally, layers are ranked by their estimated Shapley values ( We remove the ceast-contributing layers until the target compression ratio. The resulting pruned LLM retains critical layers while eliminating redundant ones, maintaining overall model performance.
EXPERIMENTS
4.1 EXPERIMENTAL. SETUP
Foundation LLMs. We evaluate open-source LLMs,, including Transformer models (LLaMA2-
(7B, 13B) (Touvron et al. 2023), Meta-LLaMMA3S8. Vicuna-7B-v1.3 CChingg et al. 2023)) and
non-Transformer models (RWKV-7B (Peng et al. 2023). Mamba-2.8B (Gu Dao 2023
Benchmarks. Model performance is assessed on language modeling and zero-shot reasoning tasks.
For language modeling. we measure perplexity on WikiTex12, PTB, and C4 to quantify generative
quality after pruning. Zero-shot reasoning is evaluated on nine datasets: PIQA (Bisk et al.) 2020). HellaSwag (Zellers al., (2019). ARC-easy and ARC-challenge (Clark al. 2018), OpenbookQA (Mihaylov et 2018). RACE (Lai et al. 2017). WSC273 (Levesque et al. 2012)) LAMBADA (Pa-
perno et al. 2016), and MMLU (Hendrycks et al. 2021), using the 1m-eva uat ion raesss
(Gao et al. 2024). We further test adversarial reasoning robustness on ANLI (Nie et al.
2020) across
its three rounds (RI-R3).
Baselines.
We compare against width pruning methods (LLM-Pruner (M et al.
2023), Wanda-sp,
FLAP An et al. 2024)) and depth pruning methods (SliceGPT (Ashkboos et al. 2024), SLEB (Sng
al. 2024), Shortened-LLM. -LMM (Kim et al. 2024), ShortGPT (Men al. 2024)).
Implementation Details. Experiments are implemented in PyTorch (Paszke et al. 2019) using HuggingFace Transformers (Wolf et al. 2020). Following Ma et al. (2023), randomly sample 10 BookCorpus (Zhu (015) examples for pruning calibration, using the same set across all baselines for fair comparison. We provide additional experiments LoRA finetuning in Appendix. [ ablation
studies Appendix. F and an analysis of the computational cost our method in Appendix.
4.2 LANGUAGE MODELING
Tab, compares compares the language modeling performance of our method with depth-wise prun- ing baselines. Our approach consistently yields the lowest near-lowest perplexity across models and pruning ratios, with the advantage becoming more pronounced under aggressive pruning. No- tably, while baseline methods Meta-LLaMA--3-88 degrade sharply at high pruning ratios, method maintains stable and significantly lower perplexity, confirming its effectiveness in preserv- ing generative performance under substantial compression.

---

Published as a conference paper at ICLR 2026
Method
PRLPF
Remone 6
P9L.44
PRLPTB
WALTON
Recone
PELPIS
PRLCC4
213 8967
151.5134
353,5550
529183
1229822
1946766
63,8323
16,3935
22,537
704239
21,3939
58.194
43,8205
16.515
54.987
1380
105 2007
79:3345
1556115
63.8876
StanGPT
16.0515
54,5822
13.5905
36,3700
105 2407
29.2345
81,M15
1566155
61,9676
157.9550
205 1568
9645
Oups
14.0942
252511
12,9682
18,5806
013877
16.1392
24
76,9957
28.7233
56,1577
105.2417
26.7712
VinrE 78
StonGiP
1514
185.7901
15:0
201 1763
110 330
350.5719
401 139
475,3755
241.530
315.5617
19.7741
16:528
16.1992
16,5001
$7.2415
31.3859
115.5943
306256
65,879
19L6155
Shattened MMM
95:8545
37,5800
143.410
48.1827
344,5519
35600477
157,
48.1927
155.1243
SHiGPPT
516.1966
9025
00295
647 2052
15% 1307
316/5689
106.008
962.4747
590.2691
1182 3573
65613488
SLEB
20.4013
38.1157
20.72 7132
13,5300
$3,75/3
30.1520
633313
115.5943
40.7122
136.9445
100,6663
90,1171
20:72 737711
57:5287
70 1331
1941396
I0N 2007
50.4000
9936 5438
9724 1949
2948.7800
1513K 9538
2113:8197
23.852
41.3128
22 7999
$4.5655
189.2533
61,8678
7455
2714 1931
1364.7820
15138 5538
40630 HCRS
2113.8157
Duts
18.5761
34.7477
19.1658
25,3966
4539763
24.9969
45.3635
76.1644
33.0055
334
173.5126
6578579
Table 2: Perplexity results of different pruning methods on WikiText2, PTB, and C4 for LLaMA-2- 7B-hf, Vicuna-7B-vl.. and Meta-LLaMAA3-881
4.3 INFERENCE PERFORMANCE
Zero-shot Performance We compare depth-wise pruning methods on LLaMA-2-7B-hr across eight zero-shot multiple-choice tasks in Tab. 3 Across all model sizes, our method consistently achieves the highest or near-highest average performance. For instance, when pruning the model to 5.5B parameters, our approach reaches 0.5227 average accuracy, outperforming SliceGPT (0.3865) and Shortened-LLaMA (0,5050). Similar trends hold for 4.9B and 4.3B models, where baselines degrade more sharply, particularly reasoning- focused tasks such as ARC-challenge and WSC273. These results indicate that our method effectively preserves layers critical for downstream reasoning, complementing the generative capability retention observed in the language modeling experiments.
Params
Method
PIQA
HcSw
ARC-
ARC-c
OBQA
Race
WSC273
LAMBADA
MMLU
Average
6.7B
LLMMA-2-7B-hr
0.7807
0,7602
0.7630
0,4625
0.4420
0.3962
0,8059
0,7388
0.4177
0.6186
SliceGPT
0.6676
0.5299
0.4663
0.3020
0,3140
0.3397
0,8022
0,3152
0,2500
0.4430
SLEB
0.7644
0,7180
0.7138
0.3985
0,4080
0,3550
0.7839
0.6216
0.3084
0.5635
6.1B
Shortencd-LLaMA
0.7497
0,7298
0.7201
0,4360
0,4040
0.3799
0.8278
0,6301
0,3572
0,5816
ShorGGP
0.7573
0,7162
0.7104
0,4292
0,4040
0.3847
0.7692
0.6167
0.351
0.5709
Ours
0,7709
0,7185
0,7155
0.4096
0,3940
0.3694
0,7912
0,6682
0,3663
0.5782
SliceGPT
0.6077
0,4270
0.3590
0,2756
0,2700
0.3081
0.7473
0,2395
0,2441
0.3865
SLEB
0,7301
0,6656
0.6700
0.395
0,3880
0,3502
0.7399
0,4477
0.2378
0.5138
5.5B
Shortened-LLaMA
0.7095
0.6528
0.5934
0.3797
0,3740
0,3330
0.7692
0,4747
0,2589
0.5050
SharGGPT
0.7095
0.6528
0.5934
0.3797
0,3740
0.3330
0,7692
0,4747
0,2589
0.5050
Ours
0,7372
0.6719
0.6473
0.3729
0,3860
0.3646
0.7546
0,4739
0,2955
0.5227
SliceGPT
0.5887
0,3856
0.3245
0.2619
0,2700
0. 0.2947
0.7253
0,16
0.2452
0.3645
SLEB
0.6910
0,5640
0.5947
0.3251
0,3520
0.3263
0.6850
0,3134
0.2372
0.4543
4.98
Shortened LLaMA
0.6485
0,5617
0.4802
0.3276
0,3280
0.3225
0.7143
0,2915
0,3811
0.4506
SharGGPT
0.6485
0.5617
0.4802
0.3276
0.3280
0.3225
0.7143
0,2915
0,3811
0.4506
Ours
0,7046
0,5840
0,5821
0.3404
0.3600
0,3501
0,7033
0,3427
0.2725
0.4689
SliceGPT
0.5718
0,3475
0.3077
0.2500
0,2560
0.2775
0.6923
0, 1450
0.2493
0.3441
SLEB
0.61.86
0.4665
0.4491
0.3020
0,3100
0.3024
0,5788
0,1527
0.2507
0.3812
4.3B
LLaMM LLaMA
0.6023
0.4430
0.3611
0.3063
0,2760
0.2909
0.6081
0.0505
0,3373
0.3640
ShorGPT
0,5952
0.4371
0.4158
0,3003
0.3500
0.2900
0.6886
0,1100
0.3532
0.3911
Ours
0.6659
0.4618
0.4937
0.2876
0.3260
0.2989
0.6117
0.1731
0.2370
0.3951
Table 3: Performance comparison of models with different retained parameters using various prun- ing methods on eight zero-shot benchmark datasets. Higher values indicate better performance.
Adversarial Reasoning Robustness To evaluate robustness under adversarial reasoning, we test pruned models on the ANLI dataset (Nie et al., 2020), which consists of three increasingly difficult rounds (RI-R3). As shown in Fig. our method achieves the highest average accuracy across all rounds and consistently surpasses baselines, ranking first on RI (36.7%) and R3 (36.6%). These results indicate that by explicitly capturing inter-layer dependencies, our method better preserves adversarial reasoning robustness, highlighting its potential for deployment in scenarios requiring resilience to distribution shifts and adversarial perturbations.
7

---

Published as a conference paper at ICLR 2026
Comparison Different Methods
Comparison
AML16
(.41
4211
532
032
ANLRD
ANL-WAarag
Purentt Scal 301
Figure 3: Adversarial reasoning accuracy ANLI (a) Comparison of pruned models RI-R3 and average; only top two values per x are labeled. (b) Accuracy across parameter scales (3-12 layers, 6.IB 4.33); only the average is labeled.
Performance on larger-scale models further demonstrate the generality of our method, we extend experiments to larger-scale models. Tab. reppots average zero-shot accuracy across eight tasks for Meta-LLaMA-3-1 matches depth-wise baselines across pruning levels. For instance, at 9.2B parameters on LLaMA- 2- 2-13B-hf, our method achieves 0.5327 accuracy, compared to 0.3950 for SliceGPT and 0.4825 for Shortened-LLaMA, demonstrating robust generalization at high pruning ratios.
and LLaMA-2-13B-M
Our approach consistently outperforms or
M3ta-LLaMMA-3SB
LLaMA-2-13RB
Method
7.4B
6.1B
5,4B
Method
11.8B
10.5B
9.2B
SliceGPT
0.4465
0.3620
0.3296
SliceGPT
0.4959
0.4286
0.3950
SLEB
0.6109
0.4526
0.3696
SLEB
0.6393
0.5762
0.5289
Shortened-LLawy LLaMA
0.6299
0.3323
0.3576
Shortened-LLaM..
0.6349
0,5340
0.4825
ShonGPT
0.6054
0.3598
0.3576
ShorGGT
0.6456
0.5878
0.4825
Ours
0.6354
0.4676
0.3912
Ours
0.6470
0.5970
0.5327
Table 4: Average zero-shot accuracy on eight datasets for Meta-LLaMA-3-SB (a) and LLaMA-2- 13B-hf (b) with different parameter sizes. Higher values indicate better performance.
4.4 COMPARISON WITH WIDTII-WISE PRUNING METIIO
We extend our analysis width-wise pruning methods on LLaMA-2-7B-hf. as shown in Fig. Using the PTB dataset as a representative case, our method consistently achieves lower perplexity than width-wise pruning across different sparsity levels, confirming its superior ability to preserve generative capacity. For example, at 4.3B parameters, our model achieves PPL of 105.2, substan- tially outperforming Wanda-sp and LLM-Prune. In terms of efficiency, depth-wise pruning proves more favorable: removing layers leads consistent improvements in both throughput and latency as pruning ratios increase, while width-wise pruning exhibits limited gains. These improvements are realized without additional memory overhead, with usage remaining in the range of 8.3-11.8 GB. Overall, our method outperforms width-wise pruning both preserving language modeling quality and in delivering scalable runtime efficiency.
4.5 PERFORMANCE NON-TRANSFORMER LLM
We apply our pruning strategy to RWKV-4-World778 and Mamba-2.8B. and evaluate perplexity on WikiText2, PTB, and C4 under varying pruning scales Tab. 5 Despite progressive layer reduc- tion, method maintains the overall generative performance of these non-Transformer models. For example, RWKV-7B retains PPL of 56.3313 at 5.6B parameters before degradation becomes significant at more aggressive pruming. Similar robustness is observed Mamba-2.88 indicat- ing that our pruning method generalizes beyond Transformer architectures and effectively preserves language modeling quality across diverse backbones.
8

---

Published as a conference paper at ICLR 2026
Language
(b) Speed
Processing Efficiency
Usage
113
L10
2001
Figure 4: Comparison with structured width pruning methods (Wanda-sp. FLAP, LLM-Pruner) on PTB and system efficiency metrics across progressively reduced parameter budgets. Our method consistently achieves the best trade-off between perplexity, latency, throughput, and GPU memory.
(a)
RWKY-4-Wortd-7B
(b) Mamba-2.8B
Params
Method
PPL. WikiText2
PPL PTB
PPL C4
Paramns
Method
PPL WikiTex12
PPL
PPL
ShenGPT
38,7159
61.8678
31,5990
ShortGP
378.9863
1865,4358
391.0166
6.2B
2.5B
34.1666
65,8579
32,0966
24.2278
43.1907
22.05%
SherGGP
90.0171
179.0204
67,9485
ShortGP
4074.4865
15138.5538
4074.4865
5.6B
2.3B
Ours
56,3313
105.2407
48.9415
Ours
31,1911
53.7517
ShonGPT
252.4593
471.6560
179.0204
SharGGT
98715.7710
143630.5993
49637
4.9B
2.0B
Ours
130.9742
252.4593
95.8227
Ours
41.2128
72.3308
33,6369
Table 5: PPL on WikiTex12, PTB and C4 for non-Transformer models RWKV-4-World-7B (a) and M.mba-2.8B (b) with different parameter sizes. Lower values indicate better performance.
4.6 COMPATIBILITY WITH POST-TRAINING QUANTIZATION
Post-training quantization (PTQ) is common technique to reduce memory usage during LLM in- ference. We evaluate its compatibility with our method by applying GPTQ (Frantar et al.) 2023) to LLaMA-2-7B-ht at different pruning scales. As illustrated in Fig. 5 pruming and quantization demonstrate strong compatibility: quantization incurs only a modest increase in perplexity, while their combination effectively improves throughput and further amplifies memory savings- for ample, reducing usage from 12.9 GB to 4.8 GB on the 6.7B variant. We further consider the in- fluence by different integration orders of our pruning strategy and 4-bit quantization in Tab. 6 The results show that the language modeling performance differences across orders are generally small, confirming that our method highly compatible with PTQ. Interestingly, we observe that placing the pruning step last often achieves the lowest perplexity. We attribute this to our method's explicit consideration of inter-layer dependencies: by analyzing the layer contributions after quantization, pruning decisions can be made on representation that is closer to the model's final inference form, thereby yielding better retention of critical capacity.
Scheme
PPL. WikiText2
PTB
PPL C4
Remove 3 layers by ours then 4-bit quantization
15.4001
56.3313
13.3799
4-bit quantization then
layers by
14,9263
56,3313
13.1724
Remove layers by ours then 4-bit quantization
19.7741
65.8579
16.9137
Remove 3 layers by ours then 4-bit quantization then remove layers by ours
19.4676
67.9485
16.3933
4-bit quantization then
layers by
18.5761
65.8579
15,8890
Table 6: Perplexity results LLaMA-2-7B-hf under different integration orders of our pruning strategy and 4-bit quantization.
5 CONCLUSION
We formulate model compression cooperative game among layers, enabling principled esti- mation of later-layer dependencies via a lightweight surrogate model. Extensive benchmarks show that approach consistently outperforms depth-wise and width-wise pruning baselines, achiev-

---

Published as a conference paper at ICLR 2026
(a) PPL Model Size
Latency Throughput
Dalore Ousntaation
Afar Quantaationn
214
4.33 1057
06
Tro.ghe-tthooms
102
(c) GPU Memory vs Model Size
11.
Memsrry (MD)
Min Ooottaaion
20043
304
6.75
6443
Vodd Sice
Figure 5: Evaluation before and after quantization across model sizes: (a) PPL (left right bars), (b) Latency vs Throughput (circles / squares), and (c) GPU Memory (solid / dashed lines).
ing lower perplexity and higher zero-shot accuracy while delivering superior efficiency in latency, throughput, and memory usage. Evaluations on larger-scale models and non-Transformer architec- tures further underscore the generality of our method. Moreover, compatibility with post-training quantization highlights the potential for practical deployment. By introducing a game-theoretic per- spective to model compression, our approach provides a novel framework for systematically and efficiently pruning large language models, achieving a balance between performance preservation and practical efficiency.
Ethics Statement This work adheres to the ICLR Code of Ethics. Our study exclusively uses publicly available datasets (e.g., WikiText2, PTB, C4, BookCorpus) that do not contain personally identifiable information. We believe that our work, which focuses on efficient pruning, primarily contributes to reducing the computational cost of training and deploying large models. We have disclosed all relevant details and ensured research integrity in accordance with the Code of Ethics.
Reproducibility Statement We have taken multiple steps to ensure reproducibility. All experimen- tal settings, including datasets, hyperparameters, model configurations, and evaluation metrics, detailed in the main text and Appendix. Algorithmic details and ablation studies are also included. provide anonymized code repository in the supplementary
To further facilitate reproduction, material, containing scripts for data preprocessing, model pruning, training, and evaluation. These resources allow others to reproduce our results and verify the claims made in this paper.
REFERENCES
Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation-baser adaptive struc- tured pruning for large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 10865-10873, 2024.
Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.15022 2024.
Hadi Askari, Shivanshu Gupta, Fei Wang, Anshuman Chhabra, and Muhao Chen. Layerif: Es- timating layer quality for large language models using influence functions. arXiv preprint
arXiv:2505.238/1 2025.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Zhipeng Chen, Kun Zhou, Beichen Zhang, Zheng Gong. Wayne Xin Zhao, and Ji- Ji-Rong Wen. Chat Cot: Tool-augmented offthougght reasoning on chat-based large language models. arXiv preprint arXiv:2305. 2023.
10

---

Published as a conference paper at ICLR 2026
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
chatbot
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprini arXiv: 034557 2018.
Luciano Del Corro, Allie Del Giorno, Sahaj Agarwal, Bin Yu, Ahmed Awadallah, and Subhabrata Mukherjee. Skipdecode: Autoregressive skip decoding with batching and caching for efficient IIm inference. arXiv preprint arXiv:2307.02662. 2023.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 O: 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems, 35; 30318-30332, 2022.
Mauricio Diaz-Ortiz Jr, Benjamin Kempinski, Daphne Cornelisse, Yoram Bachrach, and Tal Kach- man. Using cooperative game theory to prune neural networks. arXiv preprint arXiv:2311.10468. 2023.
Xuan Ding, Rui Sun, Yunjian Zhang, Xiu Yan, Yueqi Zhou, Kaihao Huang, Suzhong Fu, Chuanlong Xie, and Yao Zhu, Dipsvd: Dual-importance protected svd for efficient Ilm compression. arXiv preprint arXiv:2506.20 20353, 2025a.
Xuan Ding, Yao Zhu, Yunjian Zhang, and Chuanlong Xie. A sliding layer merging method for efficient depth-wise pruning in llms. arXiv preprint arXiv:2502.19155 2025b.
Jinhao Duan, Renming Zhang, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Elias Stengel- Eskin, Mohit Bansal, Tianlong Chen, and Kaidi Xu. Gtbench: Uncovering the strategic reasoning limitations of llms via game-theoretic evaluations. arXiv preprint arXiv:2402.1 12348, 2024.
Determine Filters' Importance. arXivv1608.08710, 2016.
Pruning filters for efficient convnets.
arXiv preprint
E. Frantar, S Ashkboos, T Hoefler, and D Alistarh. Optq: Accurate quantization for generative pre-trained transformers. 2023. In URL. htpp:///ppeerreiie. net/forum, 2023.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International conference on machine learning, pp. 10323-10337. PMLR, 2023.
Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. Specializing smaller language models towards multi-step reasoning. In International Conference on Machine Learning, pp- 10421-10430. PMLR, 2023.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL 12608602
org records/
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding, 2021. URL https:
03300
Cheng-Yo Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Rat- ner, Ranjay Krishna, Chen-Yu Yu Lee, and Tomas Pfister. Distilling step-by-step! outperform- ing larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.0245 02300, 2023.
11

---

Published as a conference paper at ICLR 2026
Bo-Kyeong Kim, Geonmin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung-Kyi Kyu Song. Shortened llama: A simple depth pruning for large language models. arXiv preprint 4rXhv:2402.02834, 11, 2024.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:/704. 2017.
Hector J Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. KR, 2012(1 2012.
Xinyin Ma. Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702-2111 2023.
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprini arXiv:2403.038.3 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.022 2018.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adver- sarial NLI: A new benchmark for natural language understanding. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Associ- ation for Computational Linguistics, pp. 4885-4901, Online, July 2020. Association for Compu- URL httpss:/acllathhll.
tational Linguisties. doi: 10. 11111866/1122222aclllll
orgg22200000. acl 4417
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint 1606::6606000033 2016.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing mns for the transformer era. arXiv preprint arXiv:2305. 2023.
David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute in transformer-hase lan- guage models. arXiv preprint 240.::22200...2558... 2024.
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. Advances in Neural Information Processing Systems, 35:17456-1177 2022.
Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, and Jae-Joon Kim. Sleb: Streamlining Ilms through redundancy verification and elimination of transformer blocks. arXiv preprint arXiv:2402.0990 2024.
Chuan Sun, Han Yu, Lizhen Cui, and Xiaoxiao Li. Efficient shapley value-based non-uniform pruning of large language models. arXiv preprint arXiv:2505.01331 2025.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effecttve pruning approach for large language models. arXiv preprint arXiv:2306. 1695, 2023.
Peng Sun, Yao Zhu, Yunjian Zhang, Xiu Yan, Zizhe Wang, and Xiangyang Ji. Unleashing the potential of large language models through spectral modulation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 3892-3911, 2024a.
Qi Sun, Mare Pickett, Aakash Kumar Nain, and Llion Jones. Transformer layers as painters. arXiv preprint ar arXiv:2407.0920 2024b.
12

---

Published as a conference paper at ICLR 2026
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 20233
Xin Wang, Zhongwei Wan, Arvin Hekmati, Mingyu Zong, Samiul Alam, Mi Zhang, and Bhaskar Krishnamachari. lot in the era of generative ai: Vision and challenges. IEEE Internel Computing, 2024a.
Xin Wang, Yu Zheng. Zhongwei Wan, and Mi Zhang. Svd-llm: Truncation-awar singular value decomposition for large language model compression. arXiv preprint arXivv2403.00337, 2024b.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference empirical methods in natural language processing; system demonstrations, pp. 38-45, 2020.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint /009509950.78 2019.
Yang Zhang, Yanfei Dong, and Kenji Kawaguchi. Investigating layer importance large language models. arXiv preprint arXiv:2409. het::2409.1488... 2024.
Yao Zhu, Yunjian Zhang, Zizhe Wang, Xiu Yan, Peng Sun, and Xiangyang Ji. Patchwise cooper- ative game-based interpretability method for large vision-language models. Transactions of the Association for Computational Linguistics, 13:744-759, 2025.
Yukun Zhu, Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. arXiv preprint drXin:15506.0622,,200 2015.
13

---

Published as a conference paper at ICLR 2026
A USAGE OF LARGE LANGUAGE MODELS
We employed large language model to assist in polishing the language of this paper. Its use was restricted to improving linguistic fluency and reducing grammatical inaccuracies, with the goal of providing a clearer and more accessible reading experience. All research ideas, experimental designs, and conclusions
conceived and validated solely by the authors.
B METHOD DETAILS
B. SURROGATE MODEL
As shown Fig. 6 we propose using lightweight surrogate network to estimate layer-wise marginal contributions. The surrogate network takes binary mask input, projects it into hid- den dimension twice as large (input dimension = number layers, hidden width = 2 x input), applies a CELU activation, and finally outputs single scalar through sigmoid activation.
Performance Score
nn.Sigmoid)
Fully Connected Layer
nn. CELU()
Fully Connected Layer
Binary Sequence
Figure 6: Framework of our surrogate network.
We stochastic gradient descent with momentum 0.9, initial learning rate of 0.008, and step- decay scheduler that reduces the learning rate by factor of 0.1 every 100 epochs in the training process. The model optimized with mean squared error loss, trained with batch size of 300, and produces both the learned parameters and the training loss curve outputs (see Tab. 7).
Setting
Value
Setting
Value
Input Dim
32
Hidden Dim
64 (CELU)
Output
(Sigmoid)
Optimizer
SGD + Momentum 0.9
Learning Rate
0.008 (decay x0.1 every 100 epochs)
Loss
MSE
Table 7: Summary of hyperparameters used for our surrogate network.
We tested the surrogate model's 's performance using the LLaMA-2-7B- model as an example (see T.b(8). By calculating error metrics and R² values, we found that the surrogate model performs stably when changing the random seed or the number of test samples. For different mask configura- tions, the model maintains high predictive accuracy when the test mask closely matches the training mask. However, if the test mask differs significantly from the training mask, the model's predictive ability declines, which is expected. In practical experiments, the training mask is adjusted according to the desired pruning ratio, meaning that the test mask is similar to the training mask. This ensures the usefulness of the surrogate model's predictions.
14

---

Published as a conference paper at ICLR 2026
Case
Test Samples
Seed
Test Mask
R²
200
42
ks (30, 27, 24, 21, 18)
0.9360
Same Seed,
500
42
ks (30, 27, 24, 21, 18)
0.9492
Same Mask
1000
42
ks (30, 27, 24, 21, 18)
0.9464
500
500
ks (30, 27, 24, 21,
0.9359
Different Seeds,
500
1234
ks (30, 27, 24, 21, 18)
0.9402
Same Mask
500
99999
ks (30, 27, 24, 21, 18)
0.9400
500
42
ks (16, 27, 24, 21,
0.9230
Same Seed,
500
42
ks (16, 14, 24, 21, 18)
0.8658
Different Masks
500
42
ks (16, 14, 12, 21, 18)
0.2466
Table 8: Surrogate Model Performance on LLaMA-2-7B-hr Model.
B.2 HYPERPARAMETERS SETTING
We use BookCorpus the calibration dataset, sampling 10 prompts of up 128 tokens, and com- pute bascline perplexity as the normalization reference for evaluating masked models. All mod- els in float16 precision with fixed random seed of 42. The pruning pipeline follows three stages: Step evaluates 8,000 randomly masked sub-networks, Step 2 trains the surrogate for 200 epochs, and Step scores 80,000 masks via the surrogate. Batch sizes are set to 45 for mask evaluation and 300 for surrogate training. While in principle different architectures may adopt distinet configurations, our experiments follow the unified settings summarized in Tab, 9 The only variation lies in the predefined Hamming weight sets, which are scaled to match model depth, e.g., {30, 27, 24, 21, 18) for 32-layer models, [36, 32, 28, 24, 20} for 40-layer models, and (60, 56, 52, 48, 42, 38, 34) } for 64-layer models. This design ensures fair comparison across models while preserving flexibility for adaptation to other architectures.
Component
Setting
Value
Setting
Value
Dataset
BookCorpus (10 samples, 128 tokens)
Random Seed
Batch size of Mask Evaluation
45
Batch size of training
300
Number of Layers
32 {30. 27, 21, 18)
Precision
Blaa16
LLaMA-2-7B-hf
Hamming Weights
Step Samples
8,000
Step Epochs
200
Step 3 Samples
80.000
Number of Layers
32
Precision
float16
Vicuna-7B-v1.3
Hamming Weights
[30, 27, 24, 21, 18)
Step 1 Samples
8,000
Step Epochs
200
Step Samples
80,000
Number Layers
Precision
float16
Meta-LLaMA-3-SB
Hamming Weights
{30, 27, 24, 21, 18)
Step 1 Samples
8,000
Step Epochs
200
Step 3 Samples
80,000
Number of Layers
40
Precision
floa116
LLaMA-2-13B-hf
Hamming Weights
[36, 32, 28, 24, 20)
Step Samples
8,000
Step Epochs
200
Step 3 Samples
80,000
Number of Layers
32
Precision
float16
RWKV.4-World77h
Hamming Weights
(30, 27, 24, 21) 200
Step Samples
12.000
Step Epochs
Step Samples
80,000
Number Layers
Precision
foaa
Mamiba-2.8B
Hamming Weights
(60, 56, 52, 48, 42, 38, 34) 200
Step Samples
8,000
Step Epochs
Step 3 Samples
80.000
Table 9: Summary of hyperparameters used in our experiments.
15

---

Published as a conference paper at ICLR 2026
B.3
ALGORITHM PSEUDOCODE
Algorithm 1 provides the main procedure of our method, where layer contributions are estimated through mask perturbation, surrogate training, and Monte Carlo approximation. To support this process, A describes the auxiliary strategy used to generate binary masks with predefined Hamming weights, ensuring sufficient diversity pruning patterns for stable surrogate learning. Together, these components form the backbone of our pruning framework.
Algorithm Layer Contribution Estimation via Mask Perturbation and Surrogate Learning
Require: Pretrained model M with L layers; validation set D; number of masks N; Monte Carlo
samples M.
Ensure: Estimated layer contribution scores C' R
1: Stage I: Data Generation (Mask Evaluation)
2: for each mask m, sampled by stratified Hamming weight do
3: Apply mask m, to M and compute masked model perplexity PPI masked (m,)) (my)
4: Compute performance score:
PPL PPLeuseline
m
PPLmaseed (my)
5: end for
6: Stage 2: Surrogate Training and Layer Contribution Estimation
7: Train surrogate network fo to predict mask scores with MSE loss:
1 Σ (fo(m)) s(m)))
(())
8:
for = to L do
Estimate contribution for each layer
9:
A +- 0
10:
for m = to M do
Monte Carlo approximation over random masks
Sample base mask m Compute marginal contribution:
12:
A = (m(+)) - fo (mm
A A+ A
14: end for
15: C. + A/M
16: end for
17: return C (C1, C2,...,Cy) CL)
C EXPERIMENTAL SETTING
C.I BASELINE METHODS
We primarily consider the width-wise pruning methods and depth-wise pruning methods as our baseline methods in our analysis. The specific information of baseline methods are described below, where we use their official code for implementation. To ensure a fair comparison, we employ the same calibration dataset across all methods.
C.1.1 WIDTH- WISTH-WISE METHOD
The width-wise pruning baselines considered include Wanda-sp, FLAP, and LLM-Pruner. Wanda- sp is structured variant of Wanda (Sun et al. 2024b), where the original metric-based on the product of weight magnitudes and input activation norms is extended to exploit shared dimensions across modules (An et al. 2024). FLAP (An et al. 2024) is retraining- ffre structured pruning framework that evaluates the recoverability of feature maps via the fluctuation pruning index. adaptively determines the compressed structure using normalized importance scores and introduces
16

---

Published as a conference paper at ICLR 2026
Algorithm 2 Stratified Mask Sampling by Hamming Weight
Require: Number of layers L, total samples M. predefined Hamming weights K
Ensure: A set of binary masks MM
if per k is not given then
2:
+ [M/ < M mod [K]
3: per + lq + for first values, q for others)
4: end if
5: M +
6: for each (k, in (K, per_k) do
7: for i = 1 to do
8:
idx -- randomly select is distinct indices from (1...., L)}
9: m f- binary vector of length with m[idx] = 1, others = 0
M + M J {m}
11: end for
end for
13:
return M
bias correction to pruned feature maps to mitigate accuracy loss. As in the original paper, we adopt the default configuration: pruning metric WIFV (among [IFV, WIFV, WIFN, N/A)) and global structure AL-AM (among [UL-UM, UL-MM, AL-MM, AL-AMJ). LLM-Pruner (Ma et al., 2023 employs a Taylor-basee based importance metric prune attention heads in MHA and neurons in FFN, operating locally within modules while preserving dimension consistency across blocks; we follow the original setting of keeping the first four and last two layers intact. The specific pruning ratios applied to LLaMA-2-7B-hh are detailed in Tab. 10
Remove 3 layers
Remove layers
Remove 9 layers
Remove 12 layers
Method
Pruned Ratio
Params
Pruned Ratio
Params
Pruned Ratio
Params
Pruned Ratio
Params
Wanda-sp
0.1
6104813568
0.19
551 3809920
0.29
4879552512
0.38
4288548864
FLAP
0.1
6091 898880
0.19
55095787522
0.28
4924891 136
0.38
4279099392
LLM. Pruner
0,12
6152794112
0.24
5512646656
0.35
4907642880
0.46
4357165056
Table Pruned ratio settings of wisth-wise method on LLaMA-2-7B-hf.
C.1.2 DEPTH-WISH METHOD
Depth pruning methods adopt the Transformer block as the basic pruning unit. We evaluate four representative approaches: SliceGPT, SLEB, ShortGPT, and Shortened-LLaMee SliceGPT (Ashk-) boos et al. 2024) is a post-training sparsification technique that replaces each weight matrix with smaller dense matrix, thereby reducing the embedding dimension; the pruning ratios used in our experiments are summarized in Tab. TT SLEB (Song et al. 2024) employs a logit-based criterion to identify redundant blocks and iteratively updates importance scores after block removal. Although designed for a no-retraining scenario, SLEB suffers from noticeable performance degradation at higher pruning rates. ShortGPT (Men et al.) 2024) introduces the Block Influence (BI) metric, which quantifies the contribution of each block by measuring the similarity between its input and output representations. Shortened-LLaMA (Kim al. 2024) determines block importance based on perplexity (PPL) sensitivity and removes low-importance layers accordingly. The specific pruned layer indices for SLEB, ShorGPPT, and Shortened-LL across four different model architectures
are provided in Tab. 12
Tab. 13,
Tab. 14 and Tab. 15
Remove layers
Remove 6 layers
Remove 9 layers
Remove 12 layers
Method
Pruned Ratio
Parums
Pruned Ratio
Params
Pruned Ratio
Parums
Prumed Ratio
Purams
LLaMA-2-7B. 2f
0.2
6105941928
0.3
$29291 4432
0.35
4886502400
0.4
4500862400
Vicuna.7b. vb.
0.2
6105940928
0.3
529291 4432
0.35
4886502400
0.4
4500862400
MeuaLlama- Llama---a-- 3-8B
0.19
7309430528
0.25
6775635968
0.33
6057857888
0.33
6057H33886
Table 11: Pruned ratio settings of SliceGPT.
17

---

Published as a conference paper at ICLR 2026
Params
Method
Remove Layers Index
SLEB
[15, 24]
Shortened Ilama
[28, 27, 25]
Remove layers
6.1B (6131265536)
ShortGPT
[25, 27, 26]
Ours
23, 11]
SLEB
[15, 14, 24, 13, 25, 22]
Shortened
[28, 27, 25, 29, 26, 24]
Remove layers
5.5B (55241
ShorGPT
[25,
29]
Ours
[21,23, 11, 12, 18, 24]
SLEB
[15, 14, 13, 25, 22, 8, 12]
Shortened. Ilama
Ilama
27, 29,
23, 22]
Remove 9 layers
4.9B (4916965376)
ShorGPT
27, 24, 29, 23, 22, 21)
Ours
[21, 11, 12, 18, 24, 10, 27, 25]
SLEB
[4, 14, 13, 25, 22, 8, 23, 12, 21,
Shortened Iaaa
26, 24, 23, 21, 22, 20. 19, 18] [25, 27, 26, 24, 28, 29, 23, 22, 21, 19, 30, 20]
27, 27, 25,
Remove 12 layers
4.3B (4309815296)
ShorGPP
Ours
23, 11, 12, 18, 24, 10, 27, 25, 14, 8,
Table 12: Pruned layer index of depth wise wise method on LLaMA-2-7B-hr.
Params
Method
Remove Layers Index
SLEB
[7. 27, 24]
Shortened Ilama
[27, 26, 24]
Remove layers
6.1B (6131265536)
ShorGPT
[27, 28, 26]
Ours
[26, 24, 29]
SI.BB
[7, 24, 17, 22, 10]
Shortened_ Ilama
[27, 26,
29, 28]
Remove 6 layers
5.5B (5524115450)
ShorGPT
127, 28, 26, 29, 25, 24]
Ours
[26, 24, 29, 27. 11, 10]
SI.BB
17, 24, 17, 22,
13, 14]
Shortemed Ilama
167, 24, 25,
28, 23,
Remove layers
4.9B (4916965376)
ShorGGPT
[27.28, 26, 29. 25, 23, 22,
Ours
24,. 29. 11. 12, 22,
SI.EB
[7, 17, 24, 17, 22, 10, 26, 13, 14,8, 25
Shortened laama
[27, 26, 24, 25, 29, 28, 23, 21, 22, 20. 19, 18)
Remove 12 layers
4.3B (4309815296)
ShorGGTT
27. 26, 29, 25, 24, 23, 22, 21, 30. 20, 19]
Ours
Table 13: Pruned layer index of depth-wise method on Vicuna-7B-vl.3.
C.2 SELECTED LAYERS OF NON- TRANSFFRRRR MODELS
We evaluate our method on non-Transformer architectures, including RWKV-4-World-7B and Mamba-2.8B. Tab. 16 reports the specific layer indices removed by our approach.
C.3 SELECTED LAYERS OF QUANTIZED MODEL
Our pruming method can be combined with quantization to further decrease memory usage. To validate this aspect, we apply 4-bit GPTQ to our pruned models, using 128 randomly sampled sequences with 2048 tokens from the C4 dataset calibration data. The specific layer indices removed under different integration orders of pruning and quantization are summarized in Tab. 17
D DETAILED RESULTS ZERO-SHOT EVALUATION
In the main text, reported the average performance of our method on eight zero-shot tasks for different model architectures. To provide a more comprehensive view, we include the detailed per- task results in Tab. 18 and Tab. 19
18

---

Published as a conference paper at ICLR 2026
Params
Method
Remove Layers Index
SLEB
25, 18]
Shortened
[24, 25, 26]
Remove layers
7.4B (7375925248)
ShorGGP
[25. 26]
Ours
25, 26]
SL.B
[7. 25, 18, 23, 28, 26]
Shortened Ilama
[24, 26, 28, 29, 23)
Remove layers
6.7B (6721589248)
ShorGPT
27, 24, 28, 23)
Ours
18. 26, 10, 11,
SLEB
[7,25, 18,
23, 28,
14, 13,
Shortened Ilama
[24, 26, 28,
Remove layers
6.1B (6067253248)
ShonGPT
125, 28,
29]
Ours
[8, 25, 26, 10,
24,
SI.FB
[7, 18, 23, 28, 26, 14, 13, 22, 10, 2,
Shortened Ilama
[24, 25, 26, 28. 29, 23, 27, 22, 20, 19, 21, 18]
Remove layers
5.4B (5412917248)
ShonGPT
125, 27, 26, 24,
28,
23, 22, 29, 21, 20, 19, 18]
Ours
25, 26, 10, 11,
12, 23, 21, 22]
Table 14: Pruned layer index of depth-wise wise method on Meta-I -MA-3-88
Params
Method
Remove Layers Index
SLEB
[15, 28. 31, 29]
Shortened Ilama
[35, 33, 34, 36]
Remove 4 layers
11.7B (11747046400)
ShorGGP
[33, 32, 30]
[31, 26, 29|
SLEB
[15, 28, 31, 29, 22, 13, 18, 8]
Shortened Jlama
36, 37, 31, 32, 30]
Remove layers
10.5B (10478228480)
ShortGPT
31,
34, 35, 29, 28]
[31, 26,
29, 25,
24]
SLEB
[15, 31, 29, 22, 13, 18, 8, 30, 27, 19,
Shortened Iaama
33, 34, 36, 37, 31, 32, 30, 28, 27,
29, 26]
Remove 12 layers
9.2B (9209410560)
ShorGPT
333. 32,
30, 34, 35, 29, 28, 27, 36,
37, 26]
[31, 26, 29, 25, 28, 24,
22,
23, 30]
Table 15: Pruned layer index of depth-wise method on LLaMA-2-13B-hr.
E FURTHER RESULTS OF LORA RETRAINING
LoRA provides an efficient approach to refining large language models (LLMs) with significantly reduced computational overhead. In our experiments, we follow the setup in Ma et al. (0023) and insert LoRA adapters into every projection weight matrix of the Transformer blocks. Specifically,
we adopt LoRA rank of 8, train with a learning rate of lc-4, batch size of 64, and run for epochs. The entire fine-tuning process lightweight: it requires only a single GPU and incurs negligible retraining cost compared to full model fine-tuning.
As shown in Fig. LoRA fine-tuning consistently lowers perplexity across different pruning ra- tios, and the beneñt becomes more pronounced as pruning intensifies. When compared at the 4.3B scale, our method not only maintains the lowest perplexity prior to fine-tuning but also preserves this advantage after LoRA is applied, underscoring both its robustness and its compatibility with lightweight adaptation strategies.
F ABLATION STUDY
F.I CALIBRATION DATASET
We analyze the effect of calibration settings on pruning robustness in Table 20 ing, different datasets lead to comparable outcomes, but discrepancies become evident under more aggressive pruning: WikiText2 calibration degrades PTB perplexity more severely, while C4 shows less stability on own domain. Increasing the number of calibration samples (from 10 to 50 or 100) delays degradation in the moderate pruning, yet more than ten layers removed, all
With mild prun-
19

---

Published as a conference paper at ICLR 2026
Model
Params
Remove Layers Index
Remove layers
6.2B (6208757760)
[12. 20, 13, 29, 23, 5]
RWKV-4-World7BB
Remove layers
5.6B (5554311168)
[12, 20, 13, 29, 23, 5, 17,
Remove 12 layers
4.9B (4899864576)
[12. 13, 23, 18.
Remove layers
(2520880640)
3,, 13]
Mamba-2.8B
Remove 12 layers
2.3B (2273415680)
[8,5,3,7,4,15 6,,33,
Remove 18 layers
2.0B (2025950720)
18,5,377,4 13, 10, 1,,,,31,6, 32, 21, 14, 22, 12, 35, 34, 27]
Table 16: Pruned layer index of our pruning method on non- no-Trrnsformer model.
Scheme
Pruned layer index
Remove layers by ours then 4-bit quantization
[21. 23, II]
bit quantization then remove 3 layers by
[21. 11, 12]
Remove 6 layers by ours then 4-bit quantization
[21, 23, 11, 12, 18, 24]
Remove layers by ours then 4-bil quantization then remove 3 layers by ours
[21, 23, 11] + [11, 10, 8]
4-t quantization then remove layers by ours
[21. 11, 12, 25, 23, 10]
Table 17:
Perplexity results on LLaMA-2-7B-hf under different integration orders of our pruning
strategy and 4-bit quantization.
settings deteriorate similarly. We attribute this to distributional biases across datasets, as well as the surrogate model's tendency overfit noisy calibration signals when exposed larger and more het- erogeneous sets. These results indicate that pruning robustness depends little dataset choice, benefits only marginally from sample size, and is fundamentally limited by the depth of pruning.
F.2 SIMULATION NUMBER FOR LAYER PRUNING
To examine the effect of the number of Monte Carlo simulations (Simu-Num) on pruning per- formance, we conduct an ablation study on LLaMA-2-7B-hr with simulation counts ranging from 3,000 to 15,000, shown in Tab. 21 In the lightly pruned regime, all settings yield similar perplex- ity across WikiText2, PTB, and C4, indicating robustness to simulation count. As pruning deepens, schemes with larger Simu_Nuuu achieve slightly lower perplexity, reflecting more accurate estima- tion of layer importance. For example, Scheme 9 (Simu_Nurr) Num=15,000 consistently outperforms Scheme 7 Siimu_Num==,,
when 12 layers are pruned. Although perplexity increases with prun- ing depth across all settings, the relative ranking of masks remains stable, suggesting that our method reliably identifies critical layers even with fewer simulations.
F.3 HAMMING WEIGHT CONSTRAINT FOR MASK GENERATION
We analyze the effect of incorporating Hamming Weight constraint in Step during mask gen- eration. Tab. 22 compares Scheme 1, which adopts stratified sampling over predefined Hamming Weights ks = (30, 27, 21, 18), with Scheme 13, which generates masks fully at random. The results show a clear advantage of using the Hamming Weight constraint. In the lightly pruned regime (first pruning step). both approaches achieve similar PPL. but as pruning proceeds, random mask generation in Scheme 10 quickly leads to sharp performance degradation. For instance, af- ter pruning 9 layers, Scheme yields PPLs of 24.6, 77.0, and 20.7 on WikiText2, PTB, and C4, respectively, while Scheme 10 under the same pruning depth reaches much higher values of 60.0, 190.6, and 43.9. The gap further widens when pruning 12 layers, where random sampling results in catastrophic degradation (PPL. 100 on WikiText2).
To further explore the impact of the Hamming weight constraint, we tested several additional weight constraints (Schemes 10-12). The results show that differentt Hamming weight ranges lead to vary- ing performance. In the case of light pruning (c.g., pruning layers), the differences are minimal. However, as pruning depth increases, the effects become more pronounced. As we intervene more with the Hamming weights (from Scheme 10 to Scheme 12), the pruning performance progressively worsens, and this effect becomes more pronounced the pruning depth increases.
The optimal performance of Scheme further highlights that the choice of Hamming weight range should be tailored to the desired pruning ratio. A balanced Hamming weight range that matches
20

---

Published as a conference paper at ICLR 2026
Params
Method
PIQA
HcSw
ARC-c
ARC-c
OBQA
Race
WSC237
LAMBADA
SliceGPT
0.6371
0.5076
0.4512
0.2841
0.2980
0.3273
0.7436
0.3227
SLEB
0.7726
0.6993
0.7643
0,4556
0,4160
0,3818
0.7949
0.6026
7.4B
Shortened Ilama
0.7726
0.7569
0.7584
0.4761
0.4020
0.3837
0.8425
0.6472
ShonGPT
0.7644
0,7566
0.7517
0,4863
0.4220
0.3990
0.8095
0.4535
Ours
0.7797
0.7399
0.7466
0.4710
0.4240
0.3914
0.8315
0.6994
SliceGPT
0.6034
0.4440
0,3830
0.2585
0.2740
0.3110
0.7106
0.2827
SI.BB
0.7394
0.6352
0.6856
0.4215
0.3660
0.3770
0.7143
0.5447
6.7B
Shortened llama
0.7155
0.6419
0.6557
0.4411
0.3760
0.3627
0.7399
0.3776
ShonGPT
0.7247
0,6815
0,6263
0.4352
0.3700
0,3569
0.7399
0.3433
Ours
0.7459
0.6473
0.6662
0.3840
0.3560
0.3292
0.7582
0.5581
SliceGPT
0,5773
0,3612
0.3249
0.2381
0.2540
0.2756
0,6593
0.2057
SLEB
0.6768
0.5218
0.5362
0.3268
0.3280
0.2900
0.6484
0.2930
6.1BB
Shortened Ilama
0.5881
0.2912
0.3746
0.3038
0.2800
0.2536
0.5385
0.0287
ShonGPT
0.6143
0.3159
0.3994
0.3183
0.3000
0.2440
0.6374
0.0487
Ours
0.7138
0.5561
0.5934
0.3242
0.3140
0,2995
0.6447
0,2952
SliceGPT
0.5571
0.3106
0.3114
0.2312
0.2560
0.2584
0,5897
0.1225
0.6246
0.4063
0.4125
0.2884
0.2780
0.2766
0.5934
0.0768
5.4B
Shortened Iaaaa
Ilama
0,5876
0,3774
0.3737
0.3038
0.2800
0.2699
0.6337
0.0349
ShortGPT
0.5876
0.3774
0.3737
0.3038
0.2800
0.2699
0.6337
0.0349
Ours
0.6474
0,4530
0.4583
0,2807
0.2940
0.2842
0.5971
0.1149
Table 18: Detailed Zero-shot Downstream Task Performance of Meta-LJaMMA-3-8h.
Params
Method
PIQA
HcSw
ARC-c
ARC-c
OBQA
Race
WSC237
LAMBADA
SLEB
0.7709
0.7593
0.7567
0.4608
0.4360
0,4038
0.8352
0.6918
Shortened Jaama
0.7709
0,7694
0.7563
0.4727
0.4420
0.4048
0.8278
0.6350
11.7B
ShonGPT
0.7726
0.7662
0.7614
0.4770
0.4460
0.4029
0.8645
0.6738
SliceGPT
0.6801
0,5654
0.5189
0.3328
0,3380
0.3435
0.8535
0,3350
Ours
0.7731
0,7733
0.7567
0.4667
0.4280
0.3971
0.8535
0.7279
SLEB
0.7470
0.6851
0.6902
0.4070
0,3840
0.3636
0.7729
0,5595
Ilama
0.7291
0,6536
0.6246
0.3857
0.4320
0.3455
0.7949
0,3062
10.5B
ShonGPT
0.7399
0.7240
0.6852
0.4377
0.4120
0.3828
0.7985
0.5224
SliceGPT
0.6219
0,4525
0.4015
0.2910
0.2960
0.3177
0,7875
0,2608
Ours
0.7443
0.7257
0.6890
0.4155
0.3840
0.3694
0.8095
0.6389
SLEB
0.7193
0.6295
0.6132
0.3643
0,3580
0.3445
0.7289
0.4735
Shortened__lame
0.6801
0.5793
0.5535
0.3575
0.3780
0.3062
0.7839
0.2212
9.2B
ShonGPT
0.6801
0.5793
0.5535
0.3575
0.3780
0.3062
0.7839
0.2212
SliceGPT
0.5963
0,4096
0.3561
0.2790
0.2960
0,2852
0.7253
0.2123
Ours
0.7013
0.6399
0.5930
0.3567
0,3740
0.3636
0.7289
0.5044
SLEB
0.6823
0.5485
0.5311
0.3328
0.3300
0,3081
0.6557
0.3569
Shortened Jlama
0.6230
0.4718
0.4524
0.3174
0.3560
0.2775
0.6630
0.1304
7.9B
ShonGPT
0.6230
0.4718
0.4524
0.3174
0.3560
0.2775
0.6630
0.1304
SliceGPT
0.5642
0.3309
0.2976
0.2585
0.2500
0.2612
0.6520
0.1285
Ours
0.6567
0,5265
0.4566
0.3046
0.3220
0.3263
0.6520
0,3419
Table 19: Detailed Zero-shot Downstream Task Performance of LLaMA-2-13B-hf..
the pruning requirements is essential to maintain performance while ensuring diverse and effective pruning patterns.
Overall, all constrained sampling schemes (Schemes 10, 11, and 12) outperform random sampling (Scheme 13), underscoring the advantages of Beeamiing-weiht-ggeee guided pruning. These findings sug- gest that stratified sampling by Hamming Weight stabilizes the Monte Carlo estimation layer con- tributions, ensuring that sampled masks cover diverse pruning patterns in more balanced manner. Without this constraint, random sampling can generate unbalanced or extreme masks, which may bias importance estimation and lead to suboptimal pruning. Ultimately, Hamming Weight -uided sampling significantly improves the robustness and effectiveness of our framework, particularly un- der deep pruning.
21

---

Published as a conference paper at ICLR 2026
(a) PPL before/after LoRA under different pruning ratios
(b) PPL before/after LoRA with different methods at 4.30
Relom lo A0ter LoRA
300
20150
Betere LoAA Aiter Lo
250
16.41 23.09
137 93
50
27.33
30.32
Figure 7: Effeet of LoRA fine-tuning on pruned models. (a) PPL before and after LoRA fine- tuning under different pruning ratios using our method. (b) PPL before and after LoRA fine-tuning with different pruning methods 4.3B parameters.
Pruned Layer
WikiTest2
PPL.PTR
PPI.
Setting
121, 23, 11]
14,0949
53.7517
12,9682
[21. 23,
11, 12, 18, 24]
18.86
61.8678
16.1392
Schemel
Bookeorpus, num=10
231,23, 11,
25)
76.9057
23, 11,
24, 27, 84
38.1157
(0572407
28,7712
[11,21, 34]
14:4671
53,7517
12,7671
[11,21,24, 25,
Scheme2
C4, num-10
[11,
23,
21.9969
79.4398
20,4018
[11,21,24 24, 14,
23,
13.22,.
38, 11.5
14k 4132
26. 29.1965
[24, 12]
52,0979
simu
[34, 11, 12, 27, 23,8]
17.7254
61 8678
15.1614
epoch-300
Scheme3
Wikitexi2, nume:10
[24, 11, 12, 27, 20,
10,14)
25,3905
79.4398
20.4018
8000
27,
(0,22,22.22)1111
[1, 13]
14.9263
$2,9183
12,9682
Schemet
Bookcerpus, num=50
(22,21, 12, 13]
19.7741
61,8678
15,8890
[22,
76.9957
20.7231
[22, 21, 12, 24, 11, 14,25, 10, 18, 17.8)
$2.0979
139.4213
37.5247
[23, 11]
14,6949
53,7517
12,9682
Scheme5
Bookcorpus,
12,
[23, 11, 21, 12,
18, 10) 25,
14,
24.9969
36.9987
20.7231
[23, 11,
25,
18, 22, 13)
61.8678
7415
34.1666
Table 20: Ablation study on calibration dataset for layer pruning LLaMA-2-7B-hf.
G COMPUTATIONAL COST AND PRACTICAL OVERHEAD
We provide overview of the computational cost of pruning framework to give readers sense of its efficiency in Tab. 23 Our method consists of two stages:
Stage 1: Mask Evaluation. this stage, evaluate the contribution each layer by generating and scoring set of pruning masks. For the LLaMA-2-7B 2-7BBhf model, evaluating 8,000 randomly generated masks requires approximately 15 minutes on a single NVIDIA V100 GPU with 32GB of memory. This step establishes the performance landscape needed to estimate layer importance.
Stage Surrogate Model Training and Contribution Estimation. Once the evaluation data is collected, we train lightweight surrogate model to predict the performance of unseen masks. Training the surrogate model is extremely fast, taking less than one minute. Using the surrogate model to score 80,000 masks and estimate Shapley-based importance values for all layers requires around 15 minutes on the same V100 GPU.
Without the surrogate model (i.e., directly estimating Shapley values by evaluating each of the 80,000 masks across all 32 layers), the computation would require roughly 80,000 x 32 forward passes, taking about 320 hours the same hardware. Furthermore, computing exact Shapley values without Monte Carlo approximation would require evaluating all possible combinations layers, ie., 232 masks for 32-lr model, which computationally infeasible. This highlights the
22

---

Published as a conference paper at ICLR 2026
Scheme
Simu. Num
Pruned
Layer Index
PPL. WikTTesf2
PPL..ITB
PPI.C4
Other Setting
[21.
111
14.6949
53.7517
12.9682
[21.
18,
16.1392
Schemel
8000
24, 25]
24.6093
76,9957
20.7231
121. 11,
18,
10, 25, 84.
38.1157
105.2407
28.7712
[11, 21]
14,6949
53.7517
12,9682
[11, 23,
12, 23,
18.5761
61.8678
15,8890
Schemn66
5000
[11, 12, 12, 27, 10,
24.6093
76.9957
20.7231
[11,
12,25,27,
10, 14]
54.5982
229
[21. 24]
14.4671
12.7671
cpoch=200
[21. 24, 10,
18.0046
63.8317
15.6426
Seheme7
3000
mc-80000
[21. 11, 24, 10, 12,
25,3905
76,9957
20:4018
ks==(30,7,22421,,15
[21. 11, 24, 10,8,27, 18, 23.
45,9763
112.0281
31,5990
[11, 21, 23)
14.6949
53.7517
12.9682
[11, 23, 12, 18]
18.0046
61.8678
15.8890
Scheme8
10000
[11, 12,
10,
24.6093
76.9957
21,
10,
43,8708
126.9445
32,0966
[1I, 21]
14,6949
53.3517
12.9682
[11, 21, 12, 10, 27]
17.1801
61,8678
15.1614
Schemee9
15000
24, 251 [11, 23, 22, 12, 10,
24.6093
76.9957
20.7231
[11. 23, 21, 12, 10, 18, 24, 25, 9,
43.8708
126.9445
32.0966
Table 21: Ablation study on the number of Monte Carlo simulations in -LaMA-2-7B-hi
for layer pruning
Scheme
Hamming Weight
Pruned Layer Indes
Wik.Wik
PPL_C4 PPL_PTB
Ofher Setting
23. III
14,0919
53,7517
12,9682
Schemel
(3-22,27,242111188
221,23
18,
18.8686
61,8678
16,1392
[21, 23,
11, 12, 18, 24, 10, 251
24,6093
76.9957
20.7231
22,,23,
18,
8.
105.2407
28.2712
[11,24,13 12]
1366844
51,9857
12.2754
[11, 24, 12, 23, 10, 20)
1844744
60.9857
15,5452
Scheme10
k==(10,7,24421,100
[4,
[11, 24, 23, 10, 20, 25, 21, 84
48.6813
114,6653
33,0519
[iI. 12.
13.8152
52,4852
12,3442
simu, num- num-8000
(2,227,14,12,100
spoch=2000
[LL, 13, 23, 24, 10,20, 18,
30,7862
84,7057
24.0289
[11, 21, 24.
25. 10, 22.
$5.5611
159.2974
37,2783
15.455
55.6454
13.1504
[6, 14, 20, %.9.21 21]
21.5396
71,4415
16,8722
Scheme12 1==10,16,,4.122100
8,
88.0215
21.
15, 29.
74.3637
149,7612
48.9448
[1.,8, [6]
16.3933
59.9643
14.0220
Scheme
Generate Mask Randomly
[19,
95.8227
23.4824
[19.8. X. 28,
59.9643
190.5663
43,8708
[9, 16, 17, 28,
30,
12, 25.
115.5843
334,4542
76.9957
Table 22: Ablation study on Hamming weight constraint for mask generation in LLaMA-2-7B-hf.
cessity of our two-stage approximation method, combining stratified Monte Carlo sampling with lightweight surrogate model, to estimate layer importance efficiently.
Scheme
Method
Computation
Approx. Time
Scheme
Stage Stage 2 (Our method)
8,000 surrogate for 80,000
15 minutes 15 minutes x 32 hours
Scheme
Direct evaluation (Monte Carlo Shapley)
80,000 32 forward passes
Scheme
Exact Shapley computation
2 forward passes
Infeasible
Table 23: Computational overhead for estimating layer importance LLaMA-2-7B-hf using 32GB
V100 GPU.
H INTEGRATION OF STRUCTURED PRUNING AND UNSTRUCTURED PRUNING
Here we demonstrate the effectiveness of combining structured and unstructured pruning methods, which leverages the strengths of both approaches. Specifically, unstructured pruning, exemplified by SPT excels maintaining high post-pruning accuracy but results irregular sparse ma- trices, making it more suited for storage compression than inference acceleration. On the other hand,
23

---

Published as a conference paper at ICLR 2026
structured pruning such as depth pruning providdes regular sparsity pattern that can signif- icantly accelerate inference. Our results in Tab[24 show that integrating these two methods strikes a balance between model performance and computational efficiency. Specifically, using LLaMA2-7B model as an example, we divide the pruning process into unstructure pruning and structure prun- the SparseGPT method to prune weights. In the second stage,
ing. In the first stage, compress the model obtained in the first stage by our proposed method. The experiment controlled the total pruning ratio at 37.5%, with the proportion of structure and non-structure being adjusted. The results show that increasing SparseGPT's pruning ratio while reducing our deep pruning ratio decreases perplexity (PPL) but reduces efficiency. Conversely, reducing PPL typically enhances efficiency.
Integrated Method
PPL
Efficiency
Unstructured Ratio
Structured Ratio
WikiTex12
PTB
C4
Latency(sec)
Throughout((nens/s))
0%
100%
38.1157
105.2407
28.7712
2.2141
57.8891
28%
72%
24.7917
76.1376
20.0237
2.4163
52.9739
50%
50%
18.2917
66.1394
16,1004
2.7269
46,9398
72%
28%
14.4598
53,9301
12.9936
3.044
42.1448
100%
13.5921
50.4833
11.8936
3.3142
38.6315
Table 24: Integration of structured pruning and unstructured pruning. Key metrics such Perplexity (PPL.), Latency (sec), Throughput (tokens/s), and Number of Parameters (nparam) are analyzed across different pruning configurations.
is worth noting that during our experiments, we also discovered that, while ensuring approximate performance and efficiency requirements, a combined approach can achieve a higher compression rate compared to using a single pruning method. For instance, when we prune the model using 100% structured pruning (without any unstructured pruning), the resulting model with approximately 5.5B parameters achieves perplexity (PPL) values of 18. 8.8666, 61.8678, and 16. 16.1392 on the WikiText2, PTB, and C4 datasets, respectively, with inference latency of 2.7554 seconds and throughput of 46.455 tokens/s. However, when we apply a combination of 26% unstructured pruning and 74% structured pruning, reducing the model to 4.6B parameters, we are able to maintain similar PPL values and inference efficiency.
Integrated Prune
PPL
Efficiency
Params
Unstructured Ratio
Structured Ratio
PP.__WikTeex12
PPI_PPTB
PPL_C4
Latency(sec)
Throughont(nkenssk)
5.5B
0%
100%
18. 18.8686
61.8678
16.1392
2.7554
46.455
5.0B
14%
86%
18.3585
64.8451
15.6334
2.7678
46.249
4.6B
26%
74%
18.2917
66.1394
16.1004
2.7269
46.9398
4.1B
44%
56%
19.6860
63,9139
16.2838
2.7237
46,9953
Table 25: Integration of structured pruning and unstructured pruning. Key metrics such as Perplexity (PPL), Latency (sec), Throughput (tokens/s), and Number of Parameters (nparam) are analyzed across different pruning configurations.
We have added a comparison with the SLEB method in Tab Specifically, using the LLaMA2-7B model as example, first sparsified the model using SparseGPT and then performed further depth-wise pruning on the sparsified model using both our method and SLEB. To ensure the robust- ness of the experimental results, used four models with sparse ratios of 0. 0.18, 0.27, and 0.36, and performed depth pruning with 6, 9, and 12 layers pruned. For a fair comparison, the experiments were conducted under the same settings. The experimental results in the table below show that our method outperforms SLEB when integrated with the unstructured pruning approach.
24

---

Published as a conference paper at ICLR 2026
Unstructured Method
Structured Method Remove layer counts
PPL WikiText2
PPL C4
Sparse Rate
Ours
SLEB
Ours
SLEB
18.3585
19.4312
15,6334
16.3469
24.7917
27.3805
20.0237
21.5788
52.8705
58.9879
32.1339
44.3139
18.2917
19.7478
16.1004
16.5431
0.18
27.1824
27.8619
21.8909
21,9230
53.7073
$9.6368
37.2207
44.6520
19.6860
23.8295
16.2838
18,8977
0.27
26.6423
59.7167
21.5267
43,5821
41.7826
189.4915
30,9364
135.0492
20.6773
22.9696
17.472
18.6676
0.36
30.6545
33.6778
23.3234
25,4039
12
59.6735
75.7457
37.5587
55,5516
Table 26: Integration of structured pruning and unstructured pruning. Key metrics such Perplexity (PPL), Latency (sec), Throughput (tokens/s), and Number of Parameters (nparam) are analyzed across different pruning configurations.
25


Reviews:
[{"review_id": "1DjMAzAXjW", "reviewer": "Reviewer_gSFA", "summary": "This is a well-executed paper with a creative idea (cooperative game formulation + surrogate Shapley approximation) and extensive empirical validation. While more theoretical analysis of the surrogate approximation would strengthen the paper, the methodological novelty and practical effectiveness make it strong.", "questions": "1. What is the computational overhead (GPU hours) compared to simpler pruning baselines like ShortGPT or SliceGPT?\n2. How sensitive is the method to the number of sampled masks or surrogate capacity?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Viewing layer pruning as a cooperative game is original and well-motivated. It captures inter-layer dependencies often ignored in prior pruning methods based on static heuristics. The surrogate-assisted estimation is elegant and computationally practical, bridging theory and application.\n2. Experiments are thorough, covering multiple models (transformer and non-transformer), datasets, and both generative and reasoning tasks, and generalization to quantization. \n3. Consistently outperforms strong baselines (SliceGPT, SLEB, ShortGPT, Shortened-LLaMA) across tasks and pruning ratios. The improvements are meaningful, especially at high pruning levels.", "weaknesses": "1. While inspired by cooperative game theory, the connection remains mostly heuristic. The surrogate model approximates marginal contributions but lacks analysis of approximation error or variance bounds.\n2. There is limited discussion of the surrogate’s accuracy or potential biases (e.g., overfitting to sampled masks). Reporting R² or correlation between predicted and true perplexities would strengthen the claim.\n3. Although the method reduces evaluation costs compared to naive Shapley computation, 8k–80k mask evaluations and 200-epoch surrogate training are still substantial for large models. Quantitative runtime comparisons to baselines would help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YXHsJaX6G4", "reviewer": "Reviewer_FgAt", "summary": "This paper proposes a game-theoretic approach to layer-wise pruning in large language models, casting the problem as a cooperative game where each layer is a player and model performance serves as utility. Since computing exact Shapley values to measure each layer’s marginal contribution is infeasible at scale, the authors design a two-stage approximation using stratified Monte Carlo mask sampling and a lightweight surrogate network to efficiently estimate layer importance. The framework reportedly captures inter-layer dependencies better than prior methods and consistently surpasses strong depth-wise and width-wise pruning baselines across multiple downstream and generative benchmarks for Transformer and non-Transformer models.", "questions": "- How robust is the surrogate model to mismatches between the calibration data distribution and the actual deployment or test distribution? Have the authors quantified the surrogate’s error rate, particularly for masks not seen during training?\n\n- Can the authors provide theoretical or empirical insights into the approximation gap between surrogate-predicted and true Shapley values, especially regarding stability of the ranking under different sampling or masking regimes?\n\n- Can authors provide results for SparseGPT as well and acknowledge or compare against the newer pruning methods that were left out.\n\n- Could the authors provide more interpretability on which types of layers (e.g., attention, feed-forward, early vs. late) are most frequently pruned, and relate this to observed task degradations", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- **Principled Formulation & Theoretical Motivation**: The paper introduces a compelling game-theoretic framing for layer pruning, challenging the prevalent assumption of independent layer importance and instead recognizing context-dependent inter-layer dynamics. This principled approach addresses a core limitation of widely-used heuristics.\n- **Efficient Approximation of Shapley Values**: By incorporating a lightweight surrogate network trained on stratified Monte Carlo mask samples, the method approximates Shapley values efficiently, enabling practical application to large-scale LLMs—this is articulated with clear algorithmic details and demonstrated scalability (see Algorithm 1 & Figure 2).\n- **Conceptual Generality**: The approach is validated across Transformer and non-Transformer architectures, and shown to integrate compatibly with quantization and LoRA fine-tuning (Figure 7), suggesting broad applicability for LLM deployment.", "weaknesses": "- **Surrogate Network Limitations and Validation**: While Figure 6 and Table 7 specify the surrogate’s structure, the paper lacks a rigorous quantitative evaluation of its prediction fidelity, especially for masks far from the training distribution. There is little discussion of failure modes, e.g., overfitting to calibration samples, brittleness under extreme masking, or calibration data misspecification (see Appendix F.1), limiting confidence for highly compressed regimes or out-of-domain settings. For instance, the impact of surrogate error on Shapley ranking stability is not systematically assessed.\n- **Potential Optimization and Mask Generation Shortcomings**: The stratified Monte Carlo mask sampling strategy (Section 3.3, Table 21) is justified primarily by ablation, with theoretical explanations on sampling sufficiency or representativeness lacking. While empirical results (Table 21) suggest an advantage over random sampling, the method’s robustness to choice of Hamming weights, number of samples, and potential for bias due to nonuniform coverage of important layer subsets remains underexplored. The mask set’s coverage and the potential for missed critical coalitions, particularly as $L$ increases, are not discussed in depth.\n- **Missing Related Work and Baselines**: The paper mentions SparseGPT but the citation to SparseGPT is pointing to the wrong paper (from the same research group).  Adding SparseGPT to the results would also make the experiments more complete. Furthermore, there has been research using Shapely values [1,2] and Influence Functions [3] for LLM pruning and LLM layer importance estimation that the paper fails to acknowledge.\n- **Ambiguity in Theoretical Guarantees**: While the game-theoretic formulation is elegant, there is no quantification of the approximation gap between true Shapley values and those estimated by the surrogate. No guarantees or bounds are provided regarding the surrogate’s reliability or the stability of resulting pruning strategies as masking regimes change. This undermines full confidence in the method’s reliability for critical compression tasks.\n- **Additional Minor Concerns**: Certain tables (such as Table 2) require careful reading to parse due to excessive fragmentation of sub-columns; legend clarity could be improved (see also Figure 3, Figure 4); some experimental settings (e.g., LoRA fine-tuning, ablation tasks) are relegated to the appendix without high-level results in the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7AqW49WTLR", "reviewer": "Reviewer_CfBF", "summary": "This paper introduces a game-theoretic framework for pruning LLMs, aiming to reduce computational cost while preserving model performance. Instead of treating layers independently, the authors model pruning as a cooperative game, where each transformer layer is a “player” and the model’s performance serves as the utility function.", "questions": "1. The paper uses 10 BookCorpus samples for calibration. How was this number chosen, and how does increasing or diversifying the calibration set affect Shapley estimation quality?\n2. Have you explored whether minimal fine-tuning after pruning further improves performance?\n3. Have you profiled the wall-clock speedups and memory reductions on real hardware (e.g., A100, H100, or consumer GPUs)? How does the pruning affect model latency when combined with quantization in real-time inference settings?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper reformulates LLM pruning as a cooperative game, capturing inter-layer dependencies ignored by static heuristics.\n2. This paper proposes a scalable Shapley-based pruning framework using stratified sampling and a surrogate model for efficient layer contribution estimation.\n3. This paper demonstrates consistent improvements over depth- and width-wise pruning baselines on multiple benchmarks, including WikiText2, PTB, C4, and zero-shot reasoning tasks.", "weaknesses": "1. While the paper proposes a surrogate-assisted approach to estimate Shapley values efficiently, it lacks a clear theoretical analysis quantifying how well the surrogate approximates true layer contributions.\n2. The method relies on a small calibration set, which may not adequately represent diverse data distributions or downstream task requirements. The resulting Shapley estimates might therefore be dataset-dependent and unstable across domains.\n3. The study focuses solely on one-shot pruning without retraining, which may restrict achievable performance. Many recent works (e.g., ShortGPT) benefit from lightweight fine-tuning. It will be helpful to investigate how minor retraining after pruning interacts with the cooperative-game framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3YboPo2Jyr", "reviewer": "Reviewer_jT3z", "summary": "The paper points out that existing layer pruning methods mostly rely on static heuristic rules, overlooking the dynamic interdependencies between layers, which leads to suboptimal results. To address this issue, the authors propose a game-theoretic approach that formulates the layer pruning problem as a cooperative game. In this game, each Transformer layer is treated as a “player,” and the overall model performance (measured by perplexity, PPL) represents the collective “utility” produced through cooperation among all players. Since computing the exact contribution of each player (i.e., the Shapley value) is computationally intractable, the authors further design an efficient two-stage approximation framework to estimate these contributions and prune layers with lower importance. Experimental results show that this method consistently and significantly outperforms existing depth-wise and width-wise pruning baselines on both language modeling (PPL) and zero-shot reasoning tasks. Moreover, it generalizes well to non-Transformer architectures and demonstrates strong compatibility with quantization techniques such as GPTQ.", "questions": "1. Why did the authors choose to evaluate on the ANLI benchmark?\n2. Why was MMLU not included in the evaluation? According to the results reported in the ShortGPT paper, its pruning method performs well on MMLU. Given that MMLU is a standard benchmark for evaluating reasoning and knowledge retention in large language models, the authors should at least include results on this task to make the evaluation more complete.\n3. Regarding the implementation of iterative pruning, the paper mentions “iteratively removing the least contributive layers.” Could the authors clarify how this iterative process is carried out?\n(A) Are all layers’ Shapley values computed once, and then layers are removed in batches (e.g., first pruning the three least contributive layers, then the next three)?\n(B) Or after each batch of layers is removed (e.g., three layers), are both Stage 1 and Stage 2 rerun to recompute the Shapley values for the remaining layers?\nIf the process follows (A), it seems inconsistent with the paper’s main motivation that pruning changes the relative importance of other layers. If it follows (B), the overall computational cost would increase significantly.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation is clear, and the results in Figure 1 clearly demonstrate the interdependence among layers.\n2. The paper proposes an efficient two-stage approximation framework that significantly reduces the computational complexity of solving the cooperative game.\n3. The experimental evaluation is comprehensive, covering both Transformer and non-Transformer model architectures.", "weaknesses": "1. The paper’s core innovation is insufficient. The idea of viewing pruning from a cooperative game theory perspective has already been explored in prior work, such as “Using Cooperative Game Theory to Prune Neural Networks.” In addition, “Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding” also formulates the layer pruning problem as an optimization task through binary pruning masks.\n2. The experimental validation is not sufficiently comprehensive. The benchmarks used are limited to perplexity (PPL) and multiple-choice tasks, without evaluation on generation benchmarks such as GSM8K. In the generalization experiments on non-Transformer architectures, only PPL was reported, lacking broader task evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)