Reassessing Layer Pruning in LLMs: New Insights and Methods
Authors: Yao Lu
Hao Cheng
Yujie Fang
Zeyu Wang
Jiaheng Wei
Dongwei Xu
Qi Xuan
Zhaowei Zhu
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
Layer Pruning
Model Compression
Primary area: foundation or frontier models, including LLMs
TL;DR: This paper presents a theoretical and empirical analysis of layer pruning in Large Language Models, aiming to improve and refine pruning strategies.

Abstract: Although large language models (LLMs) have achieved remarkable success across various domains, their considerable scale necessitates substantial computational resources, posing significant challenges for deployment in resource-constrained environments. Layer pruning, as a simple yet effective compression method, removes layers of a model directly, reducing computational overhead. However, what are the best practices for layer pruning in LLMs? Are sophisticated layer selection metrics truly effective? Does the LoRA (Low-Rank Approximation) family, widely regarded as a leading method for pruned model fine-tuning, truly meet expectations when applied to post-pruning fine-tuning? To answer these questions, we dedicate thousands of GPU hours to benchmarking layer pruning in LLMs and gaining insights across multiple dimensions. Our results demonstrate that a simple approach, i.e., pruning the final layers followed by fine-tuning the lm\_head and the remaining last three layers, yields remarkably strong performance. These pruning strategies are further supported by theoretical analyses based on the gradient flow. Following this guide, our method surpasses existing state-of-the-art pruning methods by $5.62\%$–$17.27\%$ on Llama-3.1-8B-It, by $2.36\%$–$19.45\%$ on Llama-3-8B and by $4.34\%$–$9.59\%$ on Llama-3-70B. The code is available at at https://github.com/yaolu-zjut/Navigation_LLM_layer_pruning.

Published as a conference paper at ICLR 2026
REASSESSING LAYER PRUNING IN LLMs: NEW IN- SIGHTS AND METHODS
Yao Lu Hao Chengè Yujie Fang'
Zeyu Wang
Jiaheng Wei Dongwei Xu Xuan 1.2
Zhaowei Zhu
l Institute of Cyberspace Security, Zhejiang University of Technology
2Binjiang Institute of Artificial Intelligence, Zhejiang University of Technology 'Hong Kong Baptist University
'Hong Kong University of Science and Technology (Guangzhou) $D5 Data
yaolu. z @mat com, xuanqi @ juju.. edu. cn, zzwl 22wed5datar
ABSTRACT
Although large language models (LLMs) have achieved remarkable success across various domains, their considerable scale necessitates substantial computational resources, posing significant challenges for deployment environments. Layer pruning, as simple yet effective compression method, re- moves layers of a model directly, reducing computational overhead. However, what are the best practices for layer pruning in LLMs? Are sophisticated layer selection metrics truly effective? Does the LoRA (Low-Rank Approximation) family, widely regarded as leading method for pruned model fine-tuning, truly meet expectations when applied to post-pruning fine-tuning? To answer these questions, we dedicate thousands of GPU hours benchmarking layer pruning in L.MMs and gaining insights across multiple dimensions. Our results demon- strate that simple approach, i.e., pruning the final layers followed by fine-tuning the 1m_head and the remaining last three layers, yields remarkably strong per- formance. These pruning strategies are further supported by theoretical analy- ses based on the gradient flow. Following this guide, our method surpasses ex- isting state-of-the-art pruning methods by 1.62%-17..2. 27% on L.ama-3.1-8B- by 2.36%-19.453 on Llama-3-8B and by 4.34% 9.59% on Llama-3-700 The code is available hhtpp:: //tp:////ttttt com cwm/yaololl yawluu-ouulueeo wtt.llliittt//aiiiittttt
resutree-costtrained
LLM
INTRODUCTION
In recent years, large language models (LLMs) have achieved unprecedented success in many fields,
such as text generation (Achiam et al. 2023 Touvron et al.) 2023 Dai et al. (025), semantic analy-
sis (Zeng al. 2025a b) Chu al. 2025) and machine translation (Zhang al. 2023 Wang al.,
2023). However, these achievements come with massive resource consumption, posing significant challenges for deployment on resource-cone con.urrc-contttieeeedeeeeee devices and scenarios (Ke al.) 2024 2025). To
address these challenges, numerous techniques have been developed to create more efficient LLMs.
including pruning (Ma al.] 2023 Sun al. [2023] Lu al. 2024 Li et al., 2025a Lu al.
2025
knowledge distillation (Xu et al. 2024 Huang et al.) 2022 Yang et al. 2024b; Wu et al.
2025
quantization (Zhou al. 2025
al.) 2025). low-rank factorization (Saha et al. [2023]
Zhao
al. 2024a), and system-leve inference acceleration (Shah et al. 2024 Lee al. (024).
Among these methods, pruning has emerged as a promising solution to mitigate the resource de- mands of L.MMs. By selectively removing redundant patterns-sude as parameters (Sun et al.. 2023), attention heads (Ma et al. 2023) and layers (Men et al. (2024) prruning aims to slim down the model while maintaining its original performance as much possible. Among different types of pruning, layer pruning (Kim et al., 2024, Siddiqui et al., (2244) has garnered particular interest due to its direct impact pruning the model's 's depth, thereby decreasing both computational complexity and mem- ory usage. Additionally, thanks to the nice structure the existing LLMs such as Llama (Dubey
Corresponniig author: Qi Xuan and Zhaowei Zhu.

---

Published as a conference paper at ICLR 2026
et al. 2024), whose transformer blocks have the exactly same dimension of input and output, layer pruning becomes a straightforward and simple solution. Therefore, in this paper, we focus on layer
pruning. Unlike existing studies (Men al. 2024 Yang et al. 2024c Chen et all 2024 Zhong et al. 2024, Liu et al., 2024b) that aim to propose various sophisticated pruning methods, we take step back and focus the following questions:
Q1. Layer Selection: Are sophisticated metries essential for identifying redundant layers?
Q2. Fine-Tuning: Is the LoRA family the best choice for post-pruning fine-tuning?
To address the aforementioned questions, we conduct extensive benchmarking on layer pruning. dedicating thousands of GPU hours to systematic experiments. Our study spans 7 layer selection metrics, satte-of-the-are open-source LLMs, 6 fine-tuning methods and common datasets. From these efforts, we have distilled a practical set of key insights for LLM layer pruning:
1). Reverse-order pruning is simple yet effective,, i.e., simply pruning the last several layers performs better than many complex pruning metries (Kim al. 2024 Men et al. 2024).
2). LoRA performs worse than expected, i.c., LoRA, the most commonly used fine-tuning method in existing pruning approaches (Sun et al. [023] Ma et al.
[023 Kim et al.) 2024 Men et al. 2024). J is the best choice for post-pruning performance recovery. contrast, freezing the other layers and fine-tuning only the last few remaining layers and Im_head,
also known as partial-layer fine-tuning, can achieve higher accuracy while reducing the training time.
Subsequently, we present an analytical framework based on the gradient flow that demonstrates how gradient weakening in Pre-Layer Normalization (Pre-LN) Transformers reduces the contributions of deep layers. This framework theoretically explains why pruning the final layers and fine-tuning the last few layers of the pruned model are effective. Then, we apply these insights and practices and Llama-3-6.3B-AApp
to develop ing state-of-the-art pruning methods by 5.62%-17.27%% on Llama-3.1-8B-Ir and by 2.36%-19.453% on Llama-3-8B. To validate the scalability of our method, further extend evaluation to the Llama-3-708 model. Extensive experiments demonstrate that our method achieves stronger gen- eralization and consistent gains over previous pruning techniques. Finally, hope our work will help guide future efforts in LLM layer pruning and inform best practices for deploying LLMs in real-world applications. In nutshell, we make the following contributions:
Our method surpasses exist-
Novel Best Practices:
Through detailed and extensive experiments, we identify reverse- order as a simple and effective layer selection metric and find that partial-layer fine-tuning outperforms LoRA-basee techniques.
Theoretical Insights: We present an analytical framework based on the gradient flow, demonstrating how gradient weakening in Pre-LN Transformers reduces the contributions of deep layers, thereby explaining the two key practices identified above.
State-of-the-Art Performance: We carry out extensive experiments mid-scale (Llama-3.1-8B-1 and Llama-3-8B) and large-scale (Llama-3-70B) models. The ex- perimental results consistently demonstrate that our method outperforms existing state-of-the-art pruning methods.
2 RELATED WORK
LLM Layer Pruning. LLM layer pruming is a technique used to reduce the number of layers in LLMs, aiming lower computational costs without significantly degrading performance. Specif- ically, it evaluates the contribution of each layer to the model's 's overall performance, using criteria such as gradients, activation values, parameter weights, or the layer's 's influence on the loss function. Layers that contribute the least are then pruned to reduce complexity. For example, LaCo (Yang) et al.) 2024c) achieves rapid model size reduction by folding subsequent layers into the previous layer. effectively preserving the model structure. Similarly, MKA (Liu al. 2024b) uses manifold learning and the Normalized Pairwise Information Bottleneck measure (Tishby et al. 2000) to iden- tify the most similar layers for merging. ShortGPT (Men al. 2024) Block Influence (BI) to
2

---

Published as a conference paper at ICLR 2026
measure the importance of each layer in LLMs and remove layers with low BI scores. Kim et al. (2024) utilize Magnitude, Taylor and Perplexity (PPL) to evaluate the significance of each layer. Recently, (Song al. 2025) find that deeper layers are crucial for reasoning and that performance can be recovered through distillation.
Normalization in Language Models. Layer Normalization (Lei Ba al. 2016) plays crucial role in Transformers (Vaswani 2017) by directly estimating the normalization statistics from the summed imputs to the neurons within a hidden layer. The original Transformer (Vaswani 2017) uses Post-LN, which applies layer normalization after the residual connections. However, subsequent research (Baevski & Auli 2018 Nguyen & Salazar 2019: Li et al.) 2023b 2025b) shows that putting normalization before the residual connections (Pre-L. can significantly improve training
stability, especially in large language models (Brown et al. 2020; Chaplot 2023; Touvron et al., 2023). Xiong et al. (2020) theoretically prove that Post-LN results in larger gradients near the output layer, making the use of warm-up essential to prevent instability. By contrast, Pre-LN scales down gradients with the depth of the model, which ensures more stable gradients during initialization.
3 AN EMPIRICAL EXPLORATION
This paper aims to contribute to the community the best practice of layer pruning such that practi- tioners prune LLM affordable size and desired performance with minimal exploration effort. Specifically, we explore which metric is most effective for identifying unimportant layers and investigate which fine-tuning method most effectively restores model performance after prun- ing, helping researchers make informed choices.
3.1 LAYER PRUNING METRICS
Layer pruning aims to find a subset of unimportant layers such that the pruned model maintains acceptable performance while reducing the model's complexity. Numerous methods have proposed various metrics to identify and prune unimportant layers. Herein, we include 7 popular metries: Random, Reverse-order, Magnitude-11 (Kim et al. 2024), Magnitude-12 (Kim et al. 2024). Taylor, PPL and BI (Men et al. 2024).
Specifically, for the random selection bascline, we randomly select several layers to prune. Reverse-order (Men et al. 2024) posits that importance inversely proportional to the sequence order. It assigns lower importance scores to the deeper layers and prune them. Magnitude was first introduced by Li et al. (016) and subsequently adopted by Kim et al. (2024). which as- deemed less informative. Following Kim sumes that weights exhibiting smaller magnitudes where W denotes the weight matrix of op- et al. (2024), we compute Magnitude Ex [|W I eration l within the n-th transformer layer. In this paper, we uniformly set = [1, As a re- sult, we term these methods as Magnitude-11 and Magnitude-12. For a given calibration dataset D. the significance of removing weight parameters is indicated by the change in training loss f. IC(WE. D) L(WF 0, D)1 2024), we omit the second-order derivatives in this assessment. Then we define the Taylor score of the n-th transformer layer as Faylor = Ee
@L(D) WT Following Ma et al. 2023 Kim et al.
"aw
BW BC(D) wPl. Following Kim et al. 22024) we remove
single layer and assess its impact the perplexity (PPL.) of the pruned model using the calibration dataset D. We then prune those layers that lead to a smaller degradation of the PPI.. Men et al.) (2024) introduce metric called Block Influence (BI) as an effective indicator of layer importance. Specifically, the BI score of the i-th layer can be calculated as follows:
BIG = En,,
||os,ell2 |
where 2; denotes the input of the i-th layer and w,, is the t-th row of x4.
3.2 EVALUATION AND DATASETS
To assess the performance of the model, follow the evaluation of Ma et al. (2023) to per- form zero-shot task classification on 8 common sense reasoning datasets using the im-evaluation: harness (Gao al. 2023) package: MMLU (Hendrycks et al. 2021), CMMLU (Li et al. 2023a
3

---

Published as a conference paper at ICLR 2026
Renchmisks
Model
Metric
PighT
HellaSwagf
OpenbooisAA7
ARC=ef
ARC.tt
MMLUT
CMMLU?
WineGrannde
Aug Acc1
Random
0.2233
0.2607
0.1380
0.2614
0.2176
0.2295
0.2500
0.4672
0.2953
PPL
0.7361
0.1344
0,2760
0.6755
0,3156
0.2943
0,2569
0,5896
0,4553
Magninudes11
0.529
0.2586
0,144
0.2609
0,2253
0.2297
0.2514
0.4893
0.2986
Magnitude-2 12
0.5256
0.2578
0,1340
0.2622
0.2108
0.2296
0.2515
0.4838
0.2944
MBuuaa7B-x11.3
Bi
0,6910
0.3987
0,2100
0.5829
0.2654
0.2389
0.2513
0.5036
0.3927
Taylor
0.5250
0.2581
0.1360
0.2584
0.2048
0.2318
0.2526
0.4972
0.2955
Reverse order
0.7171
0.5005
0,2508
0.6221
0.3848
0.4737
0.3417
0.6267
0.4909
Ramlom
0,5408
0,3682
0.1240
0.2630
0.2039
0.2366
0.2457
0.4807
0.2954
0.7089
0.2240
0.5960
0.2944
0.3457
0.5185
0.4078
Magnitude-1
0,578
0,2040
0,5244
0,290]
0.2574
0.5249
0.3890
Magninuds-12
0.5903
0.9657
0,1640
0.2381
0.2502
0,2513
0.5312
0.3567
Quen1.5-78 7,577
0.7220
0.4190
0.2440
0.5972
0.2671
0.2456
0.2536
0.5383
0,4190
Taylor 0.6970
044284
0,2060
0,5160
0,3140
0.523
0.6079
0.6046
0,4671
Reverse-order
0.692
0.4444
0,2280
0:5143
0.3302
0.5101
0,7171
0.5912
0,5037
Kandom
0.563
0.2886
0,1400
0.3169
0.1890
0.2275
0.2559
0,5075
0.3110
IPL
0,7628
0.4931
0,2640
0,7290
0.3805
0.3367
0.2724
0,5793
0,4772
Magnitud-I11
0.408
0.2634
0.1300
0.2845
0.2014
0.2504
0.2503
0.4878
0,3018
Magnitude-1 12
0,5413
0.2638
0.1340
0.2841
0.2014
0.2198
0.2504
0.4870
0.3015
1aams-3 1-18--
0.7176
04196
0,2020
0.6107
0.2841
0.2417
0.2494
0.5391
0,4080
Taylor
0.7138
0.2740
0.6848
0.2861
0.479
urder
0,7002
0.2940
0.6170
0.3985
0.6243
0,5268
Table 1: Zero-shot performance of the pruned models (25% pruning rate, fine-tuning using LoRA). "Avg Acc" denotes the average accuracy calculated among eight datasets. The best pruning results marked in boldface. The sub-optimal ones are underlined. See Table [ more pruning rates.
PIQA (Bisk et al.. 2020), HellaSwag (Zellers et al. (019)). WinoGrande (Sakaguchi et al. 2021), ARC-ean -aay (Clark al. 2018), ARC-challenge (Clark et al. 2018) and Openbook (Mihaylov et al. 2018 See the Appendix for more details on the dataset description.
3.3
ARE SOPHISTICATED METRICS ESSENTIAL FOR IDENTIFYING REDUNDANT LAYERS?
The first question is to find the most "redundant" layers to prune. As discussed in the previous subsection, there various metrics for layer selection, which can be as straightforward as reverse- order, or as complicated as BI. However, does a complicated metric always contribute to a better performance? Probably not. We find that simple metric, i.e., reverse-order, is competitive among these metrics. Specifically, we conduct comprehensive experiments on Vicuna-7B-v1.5 (Zheng al. 2024), Qwen1.5-7B (Yang al. 2024a) and Llama-3.1 1---..88---stuuue (Dubey al., 2024). We unifrmlyy prune 8 layers (25% pruning ratio) for Vicuna-7B-vl.. Qwen1.3-7B and Llama-3.1- 8B-Instruct. Experiments with a 50% pruning ratio are provided in T In the fine-tuning stage, we use LoRA with a rank d of 8 and batch size of 64, and the AdamW optimizer. The learning rate is set to x 10 with 100 warming steps.
As shown in Table I we find that the reverse-order metric delivers stable and superior results across various models under the 25% pruning rate, making it a reliable choice for pruning. On average, it outperforms the second-best PPL metric by 6.04% across three models. The result also holds for the 50% pruning rate, as shown in T We hope our insights can help researchers make informed choices when selecting the most suitable pruning metrics for their specific models.
Results.
Insight #1: The reverse-order are simple yet foolproof metrics for pruning, providing stable and reliable results across different models and pruning rates.
3.4 Is THE LORA FAMILY THE BEST CHOICE FOR POST-PRUNING FINE-TUNING?
In previous studies (im et al. 2024 Men et al., 2024), LoRA is often used to restore the perfor- mance of pruned models. This raises question: Is the LoRA family the best choice for post-pruning fine-tuning? To answer this question, we further use QLoRA (Dettmers et al. 2024) and partial-layer fine-tuning techniques to conduct experiments. We briefly introduce these methods follows:
LoRA Fine-tuning, LoRA is one of the best- pest-performing parameter-efficient fine-tuning paradigm that updates dense model layers using pluggable low-rank matrices (Mao al. 2024). Specifically,
4

---

Published as a conference paper at ICLR 2026
Beschmakks
Model
Method
Layer
19241
HellsSwog!
ARC-e1
ARRCet
[MMLL
WasCinise1
Ang
LeRA
03111
0.9005
0.288
0.031
0.3448
04737
0.5417
06767
0,099
QUaRA
0.6499
0.4677
0.700
05345
0.349
0498
0,5433
0,0044
04461
In food only
A.057
0.4665
0.2800
06301
0.4000
04819
0.3320
06156
0,R51
7B-v1.5
Phriik-lyyar
Jnr Jour two lover
Ive Je.jeraf lor layar
00715
27314
0.5054
0.50m
0.3620
0.000
06611
06833
04113
0.4022
0463:
04658
0,3188
0,330
06233
06267
0.504
0,5060
In Jenatelorr Niw levvr
67247
0,310
0.000
08338
0.983
0,482
0L384
0.9011
LoRA
A6942
0/4444
0.230
05148
0.3302
0.510l
0.7191
0,9912
0,5007
QLARA
06667
0.402
0.400
04250
0.200
04797
0.014
0,828
047W
In final only
A7149
0,4355
0.400
0,549
0.3524
0.5467
0.736
0.9967
0,5259
Qve71
Phnisk-byer
Jn Jon two lover
Je.Jaasf layar
A7214
0.220
0:4015
0.4890
02440
0,2540
0.783
0.500
0.354
0.3584
6.579
0.534
0L7283
0.7275
0.028
0.223
0.557
0.5418
In Innatelorr siww koowe
67726
0,4244
02530
0.506
0.3618
0,599
0.222
0.223
6.5445
LoRA
A.702
0,4010
0.2940
0.600
0.3965
0.542
0.5449
0.243
0,5266
QLARA
0.000
0.1915
03000
06183
0.344
0,61
0,980
06787
0.5171
Iy final only
A7334
0.4996
0.2000
0.7012
0.4411
6,122
0.5442
0.077
0,5999
Je wwwaaaoooo forn lapar
00330
0.5107
0,2000
0.709
0.431
0,6600
0.5876
0,682
0.5731
Phrisk-byar
Jn Jeadd-llrr tare lnver
27361
0.5304
0,3680
0.7111
0.4633
0,6888
0.5543
0.6367
0,5766
levad three layeer
0:738
0,309
0.7260
0.4684
0.6607
0.0015
0.5907
Table 2: Zero-shot performance of pruned models using various fine-tuning methods under 25% pruning rate (using reverse-order). The best results are marked in boldface, and the sub-optimal ones are underlined.
LoRA
QLoRA
Imch only
Im..kodd+lart leyer
Int Int.hrad+ last layers
Im.liead last thrree layers
Trainable parameters
15.73M
15.75M
525. 225.34M
743.45M
961. .66
1179.68M
GPU memory
45.83G
14,26G
39.82G
42.12G
44.41G
48.02G
Training (2 epoch)
10440 30s
17249.01s
6952.92s
7296.76s
7616.83s
7931.366
Table 3: The training cost of fine-tuning the pruned Llama-3. 1.ae-3.1-888 I-8B-Instruct (with 8 layers removed in reverse-order) using different methods on 2 NVIDIA A100 GPUs.
for a pre-trained weight matrix Wo, LoRA constrains its update by representing the latter with low-rank decomposition Wo + AW Wo + BA. At the beginning of training, is initialize with random Gaussian initialization, while B is initialized to zero. During training, Wo is frozen and does not receive gradient updates, while A and B contain trainable parameters. Then the forward pass can be formalized as:
Wh + AWx = Wox + BAr.
(2)
QLoRA Fine-tuning. QLoRA builds on LoRA by incorporating quantization techniques to further reduce memory usage while maintaining, or even enhancing the performance.
Partial-layer Fine-tuning. Compared to LoRA and QLoRA, which inject trainable low-rank factor- ization matrices into each layer, partial-layer fine-tuning simply freezes the weights of some layers while updating only the specified layers to save computing resources and time (Shen et al.) 2021 Ngesthi et al. 1021; Peng & Wang] 2020). Following by the common practice of previous stud- ies (Khan & Fang 2023). we choose to fine-tune only the later layers that are closer to the output, while keeping the earlier layers, which capture more general features, frozen. Specifically, we use two different fine-tuning strategies: one is to finetune only the model head (lm_head only), and the other is to finetune the im_head plus the last layer (Im + last layer), the last two layers hlm_ + last two layers), and the last three layers (Im head + last three layers).
In view of the superiority of the reverse-order metric in Section Vicuna- 7B-ra- Qwen1,5-7B, and 3lama-3. 1--BBBBBBBBBBB--- 1-88--Instuc models, we prune 8 layers. Subsequently, we utilize LoRA. QLoRA and lartial-layer fine-tuning methods to restore performance. To verify the generalizability of these fine-tuning methods, we provide additional fine-tuning results using the Taylor metric in Table M For fine-tuning with LoRA and partial-layer methods, we utilize the AdamW optimizer, while for QLoRA, opt for the Bademddddmw_BS00 optimizer. All other hyperparameter settings are the same as in the previous subsection.
we use it to prune here. For the
Results. As shown in the Table 2 and Table M we find that fine-tuning with QLoRA slightly hurts the performance of pruned models compared to LoRA. Excitingly, the effect of partial-layer fine- tuning is significantly better than LoRA, providing viable new direction for fine-tuning models after pruning. When considering fine-tuning methods for LLMs, in addition to performance, the training cost is also significant factor to take into account. Therefore, compare the training
5

---

Published as a conference paper at ICLR 2026
cost of these fine-tuning methods, including training time, gpu memory and trainable parameters. Specifically, we conduct experiments on 2 NVIDIA A100 GPUs using the pruned Llama-3.1-8P 3.1- Instruct model (with 8 layers removed in reverse order). Table 3 shows the comparison among these fine-tuning methods. We find that compared to LoRA, partial-layer fine-tuning involves more trainable parameters but maintains comparable GPU usage and achieves faster training time. Addi- tionally, partial-layer fine-tuning outperforms LoRA in effectiveness. In contrast, although QLoRA consumes less GPU memory, has much longer training time and yields poorer performance. In summary, we conclude that eartial-layer fine-tuning is an effective approach to restoring the perfor- mance of pruned models.
Insight #2: Partial-layer fine-tuning can serve as an alternative to LoRA, achieving better performance recovery for pruned models while reducing the training time.
4 THEORETICAL ANALYSIS
While our extensive experiments in the previous section demonstrate the surprising effectiveness of pruning the last several layers (reverse-order) and fine-tuning the last few remaining layers of pruned model, these empirical successes beg fundamental question: Why do these strategies work? We argue that the answer lies in the gradient propagation landscape shaped by the LayerNorm. In this section, theoretically amalyze these key insights
Notably, Most state-of-the-ar LLMs, like GPT, LLaMA, and Mistral, adopt the Pre-Layer Normal- ization (Pre-LN) Transformer architecture. Our goal is examine whether this architecture weakens the gradients deep layers during backpropagation. To investigate this, we formally characterize the gradient flow behavior in Pre-LN Transformers and analyze how the gradient norm evolves propagates from deep layers to shallow layers.
analytical framework of the gradient flow, explain key insights: (1) in Pre-LN models, the deeper layers contribute less to overall performance, and (2) fine-tuning the last layer (after pruning several final layers) can recover performance more effectively than applying LoRA updates on all remaining layers.
Specifically, based
Consider L-layer Transformer with a Pre-LN (ubeyy al.
2024 scheme:
F1+1 = + F(LN(x1); 0:),
where LN is a normalization operator (LayerNorm (Lei Ba al. 2016) or RMSNorm (Zhang & Sennrich 2019)), is the input of the [-th layer, #1+1 is the output of the I-th layer (a.k.a. the input of the (7+ 1)-th layer), O:) the learnable sub-layer. We can calculate the derivatives of
Equation 3 follows:
0z I OF(LN(Z)).) O.N(m)) 0x:
(4)
aLN(x1)
Oz
BP LN
In terms of B3 B.Pte-LN, we consider two normalization methods: LayerNorm and RMSNorm, with their corresponding Jacobian matrices (w.r.t. the input vector æ) denoted by BLN (x:) and BRMS For RMSNorm, we have Theorem 4.1
Theorem 4.1 (Spectral Norm of RMSNorm Jacobian). Let RMSNorm(z)) = T IRMS-e where
||ø]|SS = v E x? then the spectral norm of the RMSNorm Jacobian satisfies:
RBM (x)]1
1
(5)
| +
Proof. Given RMSNorm(Z)
RM its Jacobian w.r.t. is
BRMS (x) | fl|lRM + d [[| [[]]]|RMM + 2)
20
(6)

---

Published as a conference paper at ICLR 2026
where d is the model dimension. Denote by d([ te)a.
We have
BRMS
[' a x..
| RMS +
Note that the matrix
has:
z-orthogonal subspace (dim d)1): each vector v orthogonal to x an eigenvector with eigenvalue 1.
the direction of has eigenvalue |ar|| which is most 1.
Hence its operator norm (largest singular value) is Multiplying by the scalar || gives the final spectral norm:
|BRMS (x)
(8)
2 RMS +e
For LayerNorm, we have Theorem 4.2
Theorem 4.2 (Spectral Norm LayerNorm Jacobian). Let LayerNorm(z) =
2-
where
Va
== E F₁, o² = E (o+ - 4)? then the spectral norm of the LayerNorm Jacobian
satisfies:
|BL (x)]l2 = ve || V
(9)
Proof. Given (ay)
a-u1 its Jacobian w.r.t. x is vo*tc"
BLN (x) Vo2 M,
(10)
where
M 11 (0--1)(6 (1) (1)1((1-11² d +
(11)
Notice that M is rank-2 modification of the identity. On any vector orthogonal both to and
(x-w1), the matrix M acts as the identity. Meanwhile, in the 2D subspace spanned by {1, 11lz-u1,,
have a certain negative correction. The largest singular value of M is still 1 (or less), so conclude
||
(12)
Hence,
(B = V02 +e
x ||
(13)
V02 +
Following the proof of Xiong al. (2020) and Takase et al. (2033) with the assumption that follow normal distribution. Therefore, we have | | RMS 0² + ut According to Theorem 4.1
and Theorem 4.2) we can derive | (x (z) < |LL (z) In practice, we usually observe o² > 1 in later training stage Li et al. (2024a). Therefore, we have
BBMS (x) I ||BLN (x)lla 1.
(14)
4.1 LARGER GRADIENTS SHALLOWER LAYERS
For the Pre-LN model, we have:
ac
OL
0a111
00,
0z41
00,
ac
T-
dzk+1
011+1
0rl
Ox
00
(15)
k=l+1
ac
L-1
0m111
+1+
Pre- Pre-LN LN 1
LNN)
00,-
Ak=l+1
7

---

Published as a conference paper at ICLR 2026
According to Equation
13]. the norm of the gradient factor can be upper bounded by
0c 00:
11
dc
II k=1+1
L-1
B.e- 1))
ax1+1
||.
(16)
dIL
00,
According to Equation 4.. the derivative of the residual connection in Pre-LN is decoupled from the term associated with the derivative of LNN which prevents the vanishing gradient problem in early layers. According to Theorem 4.1 and Theorem 4.2) we have demonstrated that |BRMS (z)ll2 S [|BLN(x)) < 1, i.e., || P LK < 1. Therefore, when I close to L, we are more likely to have ITk (I) + 1 Pre-LN 1 close to 1 compared to shallower layers, indicating that deep layers are less important than shallow layers due their gradients do not contribute much. The above derivation theoretically explains why it is reasonable for us to prune the last several layers.
4.2 FINE-TUNING THE LAST REMAINING LAYERS AFTER PRUNING OUTPERFORMS LORA
When pruning the top (or last few) layers and freezing the rest, a key decision is whether to apply LoRA (Hu et al. T02T) across all remaining layers to fully fine-tune only the last layer. This scenario differs from standard fine-tuning settings, where LoRA applied to the full model is often preferred due to its parameter efficiency and ability to distribute adaptation across all layers.
From Equation (15), we observe that LoRA tends to have a more pronounced effect on shallower layers. However, in large pretrained models, lower-layer representations already robust and broadly transferable to various downstream tasks, often requiring minimal modification. When the model is not pruned, fine-tuning the full model with LoRA generally more effective since the entire network participates in adaptation. In contrast, after pruning, the distribution shift in the last remaining layer is more severe because it must align the modified feature space with the new output distribution. If LoRA exerts weaker influence on deeper (remaining) layers, the final alignment with the task output may still be suboptimal. By fully fine-tuning only the last remaining layer, we directly adapt the part of the model that most critically shapes the final output distribution. As result, this approach can yield better alignment and higher performance than a distributed LoRA update across all remaining layers, particularly in scenarios where deeper layers undergo significant pruning-induced induced shifts.
OBTAINING THE BEST PRUNED MODELS
In the previous sections, we have gained some valuable non-trivial practices and insights on LLM layer pruning through systematic experiments and provided theoretical proofs that underpin its fea- sibility. Herein, we use these practices obtain the Llama-3.1-6.3B-I-AAApl
and Llama-3-6.3B. Alpaca models and compare their performance against pruned models obtained by various state- of-the-art LLM pruning methods, including ShortGPT (Men et al. 2024), Shortened LLaMA (Kim)
et al. 2024). SLEB (Song et al.) 2024). PIP (Cao et al. 2025), ILM-Prunet (Ma al. 2023)
SliceGPT (Ashkboos et al. 2024).
LaCo (Yang et al. (024c), DeeperLayers Gromov et al.
2024)
FINERCUT (Zhang et al. 2024b LLM-Streamline (Chen et al.
2024) and GRASP (Liu et al.
2025).
obtained by pruning 8 layers of Llama-3.1 -IB-It and Llama-3-8B using the reverse-order metric. The pruned models are then fine-tuned with partial-layer fine-tuning (Im_head last three layers) on the Alpaca-cleaned dataset. As shown in Table 4 experimental results show that our 1.ama-3.1-66 1-6.3B-It-AAlaaee and Llama-3- 6.3B-Alpaca outperform all baselines in terms of average accuracy. Specifically, as for Llama- 3. 1-t.Bi, our method achieves an average accuracy of 58. 58.07%, which is 5.62% higher than the best existing baseline GRASP and 17.27% higher than the porst-pe method ShortGPT. As for Llama-3-8B, our method achieves an average accuracy of 56.33%, which is 2.36% higher than the best existing baseline FINERCUT and 19.45% higher than the worst-performing method ShortGPT. It is worth noting that SLEB (Song al. 2024 processed based on the inference-only approach without any fine-tuning, so we fine-tune the pruned model using LoRA on Alpaca-cleanle for fair comparison. We also present the results of pruning without fine-tuning in Table KJ where reverse-order pruning still outperforms SLEB, highlighting its superior performance even without additional fine-tuning. Finally, provide the generation results of our Llama-3. 1-6.3B-It-AAl
Specifically, Llama-3. 1.1..3.....-3---111 It-Apacaaa and -lamm-3-6633-AAppaat
8

---

Published as a conference paper at ICLR 2026
Beschmarks
Model
PR
Method
PIGAT
HellaSwee
OpenkookQA
ARC-et
ARC-E1
MMLUT
CMMLU?
Win-Crmke
Arg Aeet
016
Origiall Model
0.7987
0,5910
0,3380
0.890
0.5171
0.6004
0,5543
0.7312
0,029
25%
Shool3PT (30
0.717
0.4196
0.2020
0.6007
0.2541
0.417
0,2494
0.539
0,4080
Showicnel LLMA (PPL)
0.7628
0.7200
0.3805
0.3367
0.2724
0.5793
0.4772
25%
Shericual HLMA (Taykor)
0.740
0.648
0.261
0.2544
0L7135
0.4796
Lhamned, 1-8B-I)
25%
SI FR (wl line-tuning)
0.571
044733
02680
0.920
0. 3.695
04305
03318
0.6885
0.5011
2016
LLMM-Phnner
0.700
0,540
02530
0,2500
0.642)
20%
SliceGiPT
0.030
0,6150
0,2880
0,2480
0.4233
0,2600
0.2530
0.4408
20%
Stcandile"
0.715
D,4550
0.2940
0.5185
20%
GRASP
0.730
0.6270
0,4300
0.3070
0.5345
25%
Lanoe 1-6 Al-n--Alpuss
0,713
0,832
0.3080
0,7300
0.4684
RU6667
0.5818
016545
0.500
0%
Origiaal Model
0.796
0,0014
0.3480
0.8005
0.4983
0.6212
0,4752
0.7332
0.6093
(BI)
0.1980
0.5825
0.2662
0.2310
0.2522
0.3194
0.3683
Shomicral (PP1.)
0.2044
0.5493
15%
Shertenod LLMMA (Taylor)
0.7188
0,4919
0.2400
0.6995
0.4018
0.3270
0.11
0.6832
0.6341
25%
SLUB (wl tine tuning)
0.7114
0,5026
0,2780
0,201
0,3720
0.3115
0,2683
0,5967
0,3947
20%
MI-M-Preeer
|
0.6050
0.4299
02700
0.5330
0.2950
0.6240
0.6721
20%
SliceGPP!
0.6540
0,3980
0.2380
0.5950
0.2940
0U6320
0.4683
20%
PIP
0.6960
0.4470
0,2680
0.5790
0.3510
(L6940
0.5058
25%
DesperLayens?
0.6260
0,3790
0.2820
0.3720
0.3120
0.3570
0,5300
0,0096
FINERCUTS
0,7480
0.6200
0.3500
0.4120
0.6590
0.5197
1umo-116, MI-Alppea
Table 4: Comparison with pruned models obtained through state-of-the-ard LLM pruning methods. Results marked with s, and & are reported from Liu et al. (2025), Cao et al. (2025) and Zhang al. (2024b), respectively. All other evaluations are run by us. PR denotes the pruning rate. The best results are marked in boldface, and the sub-optimal ones are underlined.
Benchmarks
Method
PIQAT
Helllawaggt
OpenbaokQA1
ARC-et
ARC-cf
MMLUT
CMMLU1
WinoGrande?
Avg Accf
0,5805
0.3360
0.5604
0.7238
0.5878
SticeGPT
0.6960
0.4440
0.3020
0.6970
0.4110
0.7210
0.5452
ShorGPP
0.7600
0.5710
0.1900
0,7740
0.4790
0.5050
0.5632
PIP
0.7860
0.5610
0.2960
0.7570
0.4530
0.7330
0,5977
Our
0.7612
0,5887
0.3580
0.7837
0.5606
0.7483
0.6288
0.6993
0.6411
Table 5: Comparison with pruned Llama-3-70B obtained through state-of-the-art LLM pruning methods under 20% pruning rate. Results marked with f are reported from Cao et al. (2025)
and the original Llama-3. 1.ama-3.1-8B-1-. 1-B--It in Table Q It is evident that the sentences generated by Llama- B.I-6.3B-It-Alpaee are comparable to those produced by the original model. They exhibit fluency, relevance, and informativeness regarding the given topic.
Generalization of the proposed method on large-scale models. To further demonstrate the gen- eralization of the proposed method, we conduct experiments the LLaMA-3-70B model. Specifi- cally, we compare our method with SliceGPT (Ashkboos et al.] 2024). ShortGP (Men et al. 2024). SLEB (Song al. 2024) and PIP (Cao et al. 2025) under 20% pruning rate. As shown in Ta- ble 5) our method achieves the highest overall performance with an average accuracy of 64.11%, outperforming the strongest prior baseline (PIP) by 4.34%.
Varying Pruning Ratios. In this subsection, we explore the performance of our method at different pruning ratios. We conduct experiments on Llama-3.1-8B, controlling the pruning ratios approxi- mately 10%, 20% and 35% and compare our method with ShortGPT, LaCo, SliceGPT, 2SSP (Sandri al. 2025) and COMPACT (Kwek & Ym, 2023). All benchmark experimental results are reported from Kwek & Yin (0025). Since Kwek & Yin (2025) primarily tests datasets such as MMLU, PIQA, WinoCrande, ARC-e, and ARC-e, our comparisons also focus on these evaluation tasks to ensure that the comparisons are fair. The experimental results are shown in Table 6 Our method achieves the highest average accuracy under each pruning rate (10% 20%, 35%), further demon- strating the effectiveness of our method.
Joint Pruning and Quantization. In this subsection, we carry out joint pruning and quantization of the model, to see if their effects can be combined. We follow the setting of SparseGPT (Frantar & Alistarh (023) and MixGPT (Shao et (0.4), where pruning is applied prior to quantization. SSpecifically, apply 4-bit GPTQ (Frantar et al. 2022) quantization the Llama-3. 3lama-3.1-663B.11 1-6.3BB---

---

Published as a conference paper at ICLR 2026
PR
Method
MMLUT
PIQAT
WinoCrandde
ARC-o7
ARC-c7
Avg Acet
10.86%:
ShoRGPT
0.5800
0.7750
0.7020
0.7120
0.4740
0.6486
10.86%
LaCo
0.5880
0.7630
0.7230
0.7370
0.4890
0,6600
10.16%
SliceGPT
0.4390
0.7100
0.6720
0.6620
0.3940
0.5754
10.86%
0.7970
0.7160
10%
COMPACT
0.5960
0.7840
0.7370
0.7490
0,5030
0,6738
12.50%
0.6785
0,7742
0.6993
0.7686
0,4949
0.6825
19.02%
SherGPT
0.5860
0.7160
0.6840
0.5833
0.4220
0.5982
19.02%
LaCo
0.2410
0.7240
0.5530
0.5110
0.2920
0,4642
20.12%
SliceGPT
0.2450
0.6190
0.6180
0.4900
0.3030
0,4550
19,99%
2SSP
0.3740
0,7680
0.6840
0.6180
0.3810
0.5650
20%
COMPACT
0.5070
0.7590
0.7010
0.6600
0,4280
10
21.88% %
Qurs
0.6378
0.7432
0.6654
0.7445
0.4744
0.6831
35,31% %
ShorGPT
0.2320
0.5720
0.5910
0.3690
0.2970
0,4122
35.31 %
LaCo
0.2310
0.5880
0,5310
0.3150
0.2730
0.3876
35.16%
SliceGPT
0.5500
0.5430
34.77%
2SSP
0.2530
0.6870
0.5930
0.4430
0.2710
0.4494
34.99%
COMPACT
0.3590
0.7060
0.6330
0,4840
0.3080
0.4980
34.38%
Ours
0.6630
0.7067
0.6772
0.6705
0.4198
0.6278
Table 6: Comparison with pruned Llama-3.1 obtained through state-of of- the LLM pruning methods under different pruning rates. The best results are marked in boldface.
Method
Memory:
PIQAT
HellaSwagT
OpenbookQAT
ARC-eT
ARC-eT
MMLUT
CMMLU1
WinoGrundee
Avg Aoet
Liama 1.1 6,30 In Alpoca
11963.40 60 MD
0.2383
0.5
0,3090 0,2860
0.7260
0. 0.4684
0.6567
0.55
0,6546 0.8267
0.5807 0.5380
Ll.ma-3.1-6.3 AlpaziCCPTQ
1326 93 MB
0,7190
0 0.5060
0,6822
0.3965
0.6093
0.497
Table 7: Zero-shot performance of quantized Llama-3.1-6.3B-I--ND
Alpaca. As shown in Table 7 after applying GPTQ for quantization, the model's memory footprint decreased significantly from 11988.40 MB 1326.93 MB, reduction of over 89%. Although this extreme compression leads to performance degradation, the significant reduction in mem- ory footprint and acceptable performance retention strongly demonstrate that our layer pruning and quantization, used in combination as orthogonal techniques, are highly effective.
Performance comparison on GSM8K. To further illustrate the reasoning capability of the pruned model obtained by our method, we evaluate the performance of pruned models obtained by various pruning methods on GSM8K (Cobbe et al.) 2021) using Llama- 3.1-8B. All benchmark results are reported from Kwek Yin 2025). As shown in Table 8 despite the high pruning rate of our method, the accuracy is significantly higher than all baselines, reaching 29.19%. This further demonstrates the effecttiveness of our proposed method.
Method
Pruning Rate
Acc (5)
ShonGPT
19.02%
0.60
LaCo
19.02%
0.40
SliceGPT
20. 12%
0.00
2SSP
19.99%
4.30
COMPACT
20.00%
10.80
Table 8: Performance com- parison on GSMSK.
6 CONCLUSION
In this paper, we revisit LLM layer pruning, focusing on pruning metrics and fine-tuning methods. From these efforts, we have developed practical list of best practices for LLM layer pruning- These pruning strategies are further supported by theoretical analyses based on the gradient flow. Finally, we use these practices and insights guide the pruning of --ama-3.1-8B-Insttteee 3-8B and Llama-3-70B. Extensive experiments demonstrate that our pruning method outperforms various state-of saate-of-the-ar pruming methods. Finally, we hope our work will inform best practices for pruning LLMs in real-world applications
Llama-
(Li ct al. (2024b) Shen & Zhang 2025 Wang et al.) 20255
Liu et 2026 Feng et
2026 Lu al. 2026 Zhao et al. 2026).
Acknowledgments. This work was supported by National Key R&D Program of China under Grant 2025YFA1510900, by the Baima Lake Laboratory Joint Fund of Zhejiang Provincial Natural Sci- ence Foundation of China under Grant LBMHZ25F02000. by the Key R&D Program of Zhe- jiang under Grant 2024C01025, by Key Technology Research and Development Program Project of Hangzhou under Grant 2024SZD1 A23 and 2025SZD1A41 by the National Natural Science Foun- dation of China under Grant 62373325 and by the Zhejiang Provincial Natural Science Foundation of China under Grant LY21F030016.
10

---

Published as a conference paper at ICLR 2026
Ethics Statement. In this paper, we carefully consider ethical concerns related to research and ensure that all methodologies and experimental designs adhere to ethical standards. Our study focuses layer pruning enhance the efficiency of LLMs and reduce computational resource quirements, thereby promoting sustainable Al development. Furthermore, all models and datasets used in our research sourced from publicly available and accessible origins, ensuring no infringe- ment on intellectual property or personal privacy.
Reproducibility Statement. The authors have made great efforts ensure the repro- ducibility of the empirical results reported in this paper. Firstly, the experiment settings, evaluation metrics, and datasets are described in detail in the main text. Secondly, the code to reproduce the results is available at https: hatp::://nnoymouuu 4cpen. Navigati -aan----e--aaerraaaaaaa innaingg-EB7
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303. 08774, 2023.
Saleh Ashkboos, Maximilian Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.15022 2024.
Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. arXiv preprint arXiv: 10853, 2018.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma. Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang. Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang. Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Yi Cao, Wei-Jie Xu, Yucheng Shen, Weijie Shi, Chi-Min Chan, and Jiajie Xu. Pip: Perturbation- based iterative pruning for large language models. arXiv preprint AXXV:22001.12278 2025.
Devendra Singh Chaplot. Albert q. jiang, alexandre sablayrolles, arthur mensch, chris bamford, devendra singh chaplot, diego de las casas, florian bressand, gianna lengyel, guillaume lample, lucile saulnier, lélio renard lavaud, marie-anne lachaux, pierre stock, teven Ic scao, thibaut lavril, thomas wang, timothée lacroix, william el sayed. arXiv preprint arXiv:2310.066. 068225,, 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards. Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings. Mathhis Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code, 2021.
11

---

Published as a conference paper at ICLR 2026
Shi Chen and Qi Zhao. Shallowing deep networks: Layer-wiss pruning based on feature repre- sentations. IEEE transactions on pattem analysis and machine intelligence, 41(12):3048-300 2018.
Xiaodong Chen, Yuxuan Hu, and Jing Zhang. Compressing large language models by streamlining the unimportant layer. arXiv preprint wrXv:2403.119135 2024.
Yuetan Chu, Yilan Zhang, Zhongyi Han, Changchun Yang, Longxi Zhou, Gongning Luo, Chao Huang, and Xin Gao. Improving representation of high-frequency components for medical visual foundation models. IEEE Transactions on Medical Imaging, 44(8):319633209 2025. doi: 10.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1803.05005 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. arXiv preprint arXiv: 10. 14168, 2021.
Lu Dai, Yijie Xu, Jinhui Ye, Hao Liu, and Hui Xiong. Seper: Measure retrieval utility through the lens of semantic perplexity reduction. arXiv preprint arXiv:2503.01475 2025.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition scale, 2021. URL /ttp:/tp///raa...orrr//a//////// /2000.
http://arxivi wrw.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Chen Feng, Minghe Shen, Ananth Balashankar, Carsten GernerBeeuere and Miguel RD Rodrigues. Noisy but valid: Robust statistical evaluation of LLMs with imperfect judges. In The Fourteenth International Conference on Learning Representations, 2026. (tRLhttps://
net www.foommmmii-hhhereeeeeeeee
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 10323-10337. PMLR, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.1332. 2022.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL htt.:tttt:://eeeeee
org/ records/
10256836
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403.17887, 2024.
Valentin Frank Ingmar Guenter and Athanasios Sideris. Concurrent training and layer pruning of deep neural networks. arXiv preprint arXiv:2406.04549 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition, 2015. http:///pp////aiii..rrr///////// 03385
12

---

Published as a conference paper at ICLR 2026
Shwai He, Daize Dong, Liang Ding, and Ang Li. Towards efficient mixture of experts: A holistic study of compression techniques. arXiv preprint FrXiv:2406.025000 2024.
Yang He and Lingao Xiao. Structured pruning for deep convolutional neural networks: A survey. IEEE transactions pattern analysis and machine intelligence, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- fional Conference on Learning Representations (ICLR), 2021.
Geoffrey Hinton. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.02531, 2015.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09688 2021.
Zhenhua Huang, Shunzhi Yang, MengChu Zhou, Zhetao Li, Zheng Gong, and Yunwen Chen. Fea- ture map distillation of thin nets for low-resolution object recognition. IEEE Transactions Image Processing, 31:13644113111 1379, 2022.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral experts. arXiv preprint arXiv:2401.04088 2024.
Zhaokang Ke, Dingyi Kang, Bo Yuan, David Du, and Bingzhe Li. Improving the sustainability of solid-state drives by prolonging lifetime. In 2024 IEEE Computer Society Annual Symposium VISI (ISVLSI), pp. 502-507. IEFF, 2024.
Zhaokang Ke, Haoyu Gong, and David HC Du. Pm-dedup: Secure deduplication with partial mi- gration from cloud to edge servers. arXiv preprint arXiv:2501.022. 002350, 2025.
Muhammad Osama Khan and Yi Fang. Revisiting fine-tuning strategies for self self-supervised medical imaging analysis. arXiv preprint 0rXiv2237..09115 2023.
Bo-Kyeong Kim, Geonmin Kim, Tac-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung-Kyu Song. Shortened llama: A simple depth pruning for large language models. arXiv preprint arXiv:2402.02834. 2024.
Eugene Kwek and Wenpeng Yin. Compact: Common-toker optimized model pruning across chan- nels and tokens. arXiv preprint arXiv:2509.0688: 2025.
Jungi Lee, Wonbeom Lee, and Jaewoong Sim. Tender: Accelerating large language models via tensor decomposition and runtime requantization, arXiv preprint arXiv:2406. 222406612999 12930, 2024.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. ArXiv e-prints, pp. arXiv-1607, 2016.
Hector Levesque, Emest Davis, and Leora Morgenstem. The winograd schema challenge. In Thir- teenth international conference on the principles of knowledge representation and reasoning, 2012.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint 160iv:160..08771 2016.
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timo- thy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint 2XXx:23066092122, 2023a.
Pengxiang Li, Lu Yin, and Shiwei Liu. Mix-ln: Unleashing the power of deeper layers by combining pre-In and post-In. arXiv preprint 2rX1v:24/2.13795, 2024a.
Yuqi Li, Yao Lu, Junhao Dong, Zeyu Dong, Chuanguang Yang, Xin Yin, Yihao Chen, Jianping Gou, Yingli Tian, and Tingwen Huang. Sglp: similarity guided fast layer partition pruning for compressing large deep models. arXiv preprint arXiv:2410. 14720, 2024b.
13

---

Published as a conference paper at ICLR 2026
Yuqi Li, Kai Li, Xin Yin, Zhifei Yang, Junhao Dong, Zeyu Dong, Chuanguang Yang, Yingli Tian, and Yao Lu. Sepprune: Structured pruning for efficient deep speech separation. arXiv preprint MrXiv:2505.12079 2025a.
Yuyuan Li, Chaochao Chen, Yizhao Zhang. Weiming Liu, Lingjuan Lyu, Xiaolin Zheng, Dan Meng, and Jun Wang. Ultrare: Enhancing receraser for recommendation unlearning via error decompo- sition. Advances in Neural Information Processing Systems, 36: 16:126612 1-12625, 2023b.
Yuyuan Li, Yizhao Zhang, Weiming Liu, Xiaohua Feng, Zhongxuan Han, Chaochao Chen, and Chenggang Yan. Multi-objectiive unlearning in recommender systems via preference guided pareto exploration. IEEE Transactions on Services Computing. 2025b.
Chengyuan Liu, Shihang Wang, Yangyang Kang, Lizhi Qing, Fubang Zhao, Changlong Sun, Kun Kuang, and Fei Wu. More than catastrophic forgetting: Integrating general capabilities for domain-specifie llms. arXiv preprint arXiv:2405. /7830, 2024a.
Deyuan Liu, Zhanyue Qin, Hairu Wang, Zhao Yang, Zecheng Wang, Fangying Rong, Qingbin Liu, Yanchao Hao, Xi Chen, Cunhang Fan, et al. Pruning via merging: Compressing Ilms via manifold alignment based layer merging. arXiv preprint arXiv:2406. /6330, 2024b.
Dingyuan Liu, Qiannan Shen, and Jiaci Liu. The health-wealth gradient in labor markets: Integrating health, insurance, and social metrics predict employment density Computation, 14(1):22, 2026. doi: 0..399/compuutaiion10000 4010022. URLhttps:: //wp: mdpi com/ 20tom//079-330 2112200999111111111 22
Open Access.
Kainan Liu, Yong Zhang, Ning Cheng, Zhitao Li, Shaojun Wang, and Jing Xiao. Grasp: Replace redundant layers with adaptive singular parameters for efficient model compression, 2025. URL
https://arxiv.cnm org/abs//////ooooooooooo...oss
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows, 2021. URL https:
14030
Yao Lu, Wen Yang, Yunzhe Zhang. Zuohui Chen, Jinyin Chen, Qi Xuan, Zhen Wang, and Xiaoniu Yang. Understanding the dynamics of dnns using graph modularity. In European Conference Computer Vision, pp. 225-242. Springer, 2022.
Yao Lu, Yutao Zhu, Yuqi Li, Dongwei Xu, Yun Lin, Qi Xuan, and Xiaoniu Yang. A generic layer pruning method for signal modulation recognition deep learning models. IEEE Transactions on Cognitive Communications and Networking, 2024.
Yao Lu, Yuqi Li, Wenbin Xie, Shanqing Yu, Qi Xuan, Zhaowei Zhu, and Shiping Wen. The struc- tural scalpel: Automated contiguous layer pruning for large language models. arXiv preprint 1.Xiv25100223653 2025.
Yao Lu, Ji Zhaiyuan, Jiawei Du, Yu Shanqing, Qi Xuan, and Joey Tianyi Zhou. Autoannotator: A collaborative annotation framework for large and small language models. Transactions on Machine Learning Research, 2026.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702-217222 2023.
Yuren Mao, Yuhang Ge, Yijiang Fan, Wenyi Xu, Yu Mi, Zhonghao Hu, and Yunjun Gao. A survey on lora of large language models. arXiv preprint arXiv:2407. 1046, 2024.
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.038.. 2024.
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models, 2024. TRLLhttss://axxii http://wwwwoo..////////000
02 948
14

---

Published as a conference paper at ICLR 2026
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018.
Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation. arXiv preprint arXiv:2407.1667, 2024.
Stephany Octaviani Ngesthi, Iwan Setyawan, and Ivanna K Timotius. The effect of partial fine tuning on alexnet for skin lesions classification. In 2021 13th International Conference on Information Technology and Electrical Engineering (ICITEE), pp. 147-152. IEEE, 2021.
Toan Q Nguyen and Julian Salazar. Transformers without tears: Improving the normalization of self-attention. arXiv preprint 1rXiv:19/0005895, 2019.
Peng Peng and Jiugen Wang. How fine-tune deep neural networks in few-shot learning? arXiv preprint arXiv:20/2.00204 2020.
Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Nikolai Gerasimenko, Ivan Os- eledets, Denis Dimitrov, and Andrey Kuznetsov. Your transformer secretly linear. arXiv preprint arXiv:2405. 12250, 2024.
Rajarshi Saha, Varun Srivastava, and Mert Pilanci. Matrix compression via randomized low rank and low precision factorization. Advances in Neural Information Processing Systems, 36, 2023.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Fabrizio Sandri, Elia Cunegatti, and Giovanni lacca. 2ssp: A two-stage framework for structured pruning of Ilms. arXiv preprint arXiv:2501.1771 2025.
Jay Shah, Ganesh Bikshandi. Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3:: Fast and accurate attention with asynchrony and low-precision, arXiv preprint arXiv:2407.08608, 2024.
Hang Shao, Bei Liu, and Yanmin Qian. One-shot sensitivity-awarr mixed sparsity pruning for large language models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 11296-11300. IEEE, 2024.
Qiannan Shen and Jing Zhang. Ai-enhanced disaster risk prediction with explainable shap anal- ysis: A multi-class classification approach using xghoost, December 2025. URL https:
7/www. //www.resear www.rrreeaaccsuuueeeeeeee com/ httmm//aiicceeee---3111121 December 31, 2025.
Preprint, Version I, posted
Zhiqiang Shen, Zechun Liu, Jie Qin, Marios Savvides, and Kwang-Ting Cheng. Partial is better than all: Revisiting fine-tuning strategy for few-shot learning. Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 9594-9602, 2021.
Shoaib Ahmed Siddiqui, Xin Dong, Greg Heinrich, Thomas Breuel, Jan Kautz, David Krueger, and Pavlo Molchanov. A deeper look at depth pruning of llms. arXiv preprint arXiv:2407.16286, 2024.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2015. URL /BRK:/ssss//axxii.orrraa///////s
1556
Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, and Jac-Joon Kim. Sleb: Streamlining Iims through redundancy verification and elimination of transformer blocks. arXiv preprint 200i::2402.000222 2024.
Xinyuan Song, Keyu Wang, PengXiang Li, Lu Yin, and Shiwei Liu. Demystifying the roles of I1m layers in retrieval, knowledge, and reasoning. arXiv preprint arXiv:2510.02091, 2025.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1695,, 2023.
15

---

Published as a conference paper at ICLR 2026
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions, 2014.
Sho Takase, Shun Kiyono, Sosuke Kobayashi, and Jun Suzuki. Spike more: Stabilizing the pre-training of large language models. arXiv preprint arXiv:2312.1690. 2023.
Chong Min John Tan and Mehul Motani. Dropnet: Reducing neural network complexity via iterative pruning. In International Conference on Machine Learning, pp. 9356-9366, PMLR, 2020.
Hui Tang. Yao Lu, and Qi Xuan. Sr-init: An interpretable layer pruning method. In ICASSP 2023- 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 1-5. IEEE, 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-followee foolownngg Ilama model, 2023.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu. Document-level machine translation with large language models. arXiv preprint arXiv:330..02210, 2023.
Maoyu Wang, Yao Lu, Bo Zhou, Zhuangzhi Chen, Yun Lin, Qi Xuan, and Guan Gui. Hsep: two- stage spectral clustering framework for resource-constrained uav identification. arXiv preprint arXiv:2512.08983, 2025.
Wenxiao Wang, Shuai Zhao, Minghao Chen, Jinming Hu, Deng Cai, and Haifeng Liu. Dbp: Discrimination based block-level pruning for deep model acceleration. arXiv preprint 11111:19/2.10178 2019.
Shuang Wu, Heng Liang, Yong Zhang, Yanlin Chen, and Ziyu Jia A cross-modal densely guided knowledge distillation based on modality rebalancing strategy for enhanced unimodal emotion recognition. In Proceedings of the Chirty-Fourlh International Joint Conference on Artificial Intelligence, pp. 4236-4244 2025.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International Conference on Machine Learning, pp. 10524-10533. PMLR, 2020.
Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng. Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. survey on knowledge distillation large language models. arXiv preprint arXiv:2402. 131 16, 2024.
An Yang, Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren. Xinyu Zhang, Xipin Wei. Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan. Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report, arXiv preprint 440i 10671, 2024a.
16

---

Published as a conference paper at ICLR 2026
Shunzhi Yang, Jinfeng Yang. MengChu Zhou, Zhenhua Huang, Wei-Shi Zheng, Xiong Yang, and Jin Ren. Learning from human educational wisdom: A student-centered knowledge distillation method. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(6):4188-4200) 2024b.
Yifei Yang, Zouying Cao, and Hai Zhao. Laco: Large language model pruning via layer collapse. arXiv preprini 1.X:::2022.11887, 2024c.
Yong Yu, Sifan Zhou, Dawei Yang, Shuoyu Li, Shuo Wang, Xing Hu, Chen Xu, Zukang Xu, Changyong Shu, and Zhihang Yuan. Mquant: Unleashing the inference potential of multimodal large language models via static quantization. In Proceedings of the 33rd ACM International Conference on Multimedia, pp. 1783-1792 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi Hellaswag: Can ma- chine really finish your sentence? arXiv preprint 100500900000730 2019.
Shuang Zeng, Xinyuan Chang, Mengwei Xie, Xinran Liu, Yifan Bai, Zheng Pan, Mu Xu, and Xing Wei. Futurexightdrive: Thinking visually with spatio-temporal cot for autonomous driving. arXiv preprint arXiv:2505. 2025a.
Shuang Zeng, Dekang Qi. Xinyuan Chang, Feng Xiong, Shichao Xie, Xiaolong Wu, Shiyi Liang, Mu Xu, and Xing Wei. Janusvin: Decoupling semanties and spatiality with dual implicit memory for vision- lasgun-lnnguage navigation. arXiv preprint (tti::5002225445. 2025b.
Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma. Investigating the catastrophic forgetting in multimodal large language model fine-tuning. In Conference on Parsimony and Learning, pp. 202-227. PMLR, 2024.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Infor- mation Processing Systems, 32, 2019.
Biao Zhang, Barry Haddow, and Alexandra Birch. Prompting large language model for machine translation: A case study. In International Conference on Machine Learning, pp. -1092-41110 41110. PMLR, 2023.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024a.
Yang Zhang, Yawei Li, Xinpeng Wang, Qianli Shen, Barbara Plank, Bernd Bischl, Mina Rezaei, and Kenji Kawaguchi. Finercut: grained interpretable layer pruning for large language models. arXiv preprint arXiv:2405.18811, 2024b.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient IIm training by gradient low-rank projection. arXiv preprint 1xXiv:2403.03507 2024a.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient 1lm training by gradient low-rank projection, 2024b. URL
https::/arxizs //ssvv///xv.... org/ abs/ 2403. 03507
Sichen Zhao, Zhiming Xue, Yalun Qi, Xianling Zeng, and Zihan Yu. Non-intrusive graph-based bot detection for e-commerce using inductive graph neural networks, 2026. URL https://
Lrxiv.org/abs/2601. 22579
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Longguang Zhong, Fanqi Wan, Ruijun Chen, Xiaojun Quan, and Liangzii Li. Blockpruner: Fine- grained pruning for large language models. arXiv preprint arXiv:2406. 105544 2024.
Sifan Zhou, Shuo Wang, Zhihang Yuan, Mingjia Shi, Yuzhang Shang, and Dawei Yang. Gsq-tuning: Group-shared exponents integer in fully quantized training for llms on-device fine-tuning. arXiv preprint arXiv:2502. 129/3, 2025.
17

---

Published as a conference paper at ICLR 2026
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference computer vision, pp. 19-27, 2015.
18

---

Published as a conference paper at ICLR 2026
APPENDIX
A PROOFS
This proof provides a theoretical analysis, which starts from the chain rule with respect to model parameters, to explain why, in very deep Pre-LN (pre-layer normalization) Transformers, the per- formance gain from deeper layers might become marginal. The analysis can also be generalized to prove why, in very deep Post-LN (post-layer normalization) Transformers, the performance gain from shallow layers might become marginal. For simplicity, we consider single residual path per layer, which contains a LayerNorm operator and learnable sub-network F. We omit multi-head attention and FFN specifics to maintain clarity.
A.1 MODEL STRUCTURE AND NOTATION
Consider L-layer network (a simplified Transformer). where all layers follow Pre-LN residual blocks (or Pos-IN residual blocks). Let:
Zo be the initial input (e.g., an embedding or the output of a preceding network).
x7 IRd denote the input layer with 1 = 0, 1.... L 1.
11+1 denote the output of layer
Pre-LN residual block has the form:
I1-1 I1 F (LN(Z)):01).
where:
LN(.) is LayerNorm,
F(- :() (0) is the learnable sub-network at layer (e.g. including parameters for attention, feed beed-forward, etc.),
0t collects all parameters at layer I,
The residual part is represented by zq-
Similarly, a Post-LN residual block has the form:
2 LN LN ay + ((m);))))
The final layer's output æ goes through Head function produce prediction y, which is used to compute the loss L(y, target), where target stands for the next token. Denote the final prediction by
Head ( Ohead).
Our goal to examine the gradient
for each layer = 0, 1, 1...... T. 1.
00:
A.2 CHAIN RULE: GRADIENT
According to the standard backpropagation (chain rule):
ac
dc
0x
a0,
0z
a6:
incoming gradicnt
layer w.t. parameters
where
801+1
describes how the layer output #1+1 depends the layer parameters 0;.
a6;
19

---

Published as a conference paper at ICLR 2026
is the "incoming" gradient from deeper layers (from layer / + 1 to and finally to ac 0m++1
the loss). By chain rule:
ac
Oc
dEL
dL
L-1
dmk+1
II
0z1+1
dz
0m1+1
d
Ork
k=1+1
To understand the gradient flow through layer I, we need to analyze both above factors.
A.3 SINGLE-LAYER JACOBIAN
Recall the Pre-LN block:
3441 = xı + (LN(a)):..)
Taking the derivative w.r.t 21, we get
0+1+1 T + OF(LN (1). 01) OLN(a)) dr
OLN(x.)
A APM-IN
BPIc-LN LN
Denote
(F(LN(Z1).0)
B BPre-L 0LN(z))
A
(L
Oxi
Thus,
J.Ie-LN= = 001+1
I + A/ A
LN B.Ie-LN.
Ora
Similarly, for the Post-LN block:
R1+1 = LN(20 21 Fla1;6)).
Taking the derivative w.rt. 21, we get
0rt1
JNN( + F(IR01)
I (F(1))
dzi
0(z + F(zn0))
Ox
B BP IN
Apont LN
Denote
OLN(ZI + F(21;(1)
B Post-IN LN
0(x + F 01)
A Po 1 aF(x))
8x1
Thus,
Ox1+1 Ox
Post-LLN
BBPost-LN A LO
In terms of B;, we consider two normalization methods: LayerNorm and RMSNorm, with their corresponding Jacobian matrices (w.r.t. the input vector æ) denoted by BLN (z) and BRMS (z).
The RMSNorm defined as follows.
RMSNorm(Z)
J [[|ls +
where ||æ||RMS = V dd x.. Thus its Jacobian w.r.t. is
(R) a (RMSNorm(z)
dx
[T
xxT
| RMS + ([|x]||MM + t
20

---

Published as a conference paper at ICLR 2026
where d is the model dimension.
Consider BBMSS/ (z)l|2. the maximum singular value. Denote by a = a(I=]Rmss We have
RRMS
| + [I a xx_] (c)
Note that the matrix
[
has:
z-orthogonal subspace (dim d- 1): each vector v orthogonal to x an eigenvector with eigenvalue 1.
the direction of has an eigenvalue a |0/| which is at most 1.
Hence its operator norm (largest singular value) is Multiplying by the scalar | gives the final spectral norm:
|BRMS (x) | +e
The LayerNorm is defined as follows.
#1
LayerNorm(())
Va² +c
where μ D Tú, 0²
1 (z+ 2)2. Its Jacobian w.r.t is i (LayerNorm(z))
Σ
(1)
Vo2 + M,
dr
where
M 11 (x - (1)(1 -1=
d
Notice that M is a rank-2 modification of the identity. On any vector orthogonal both to and (ar -1), the matrix M acts as the identity. Meanwhile, the 2D subspace spanned by 1,æ-=1); we have a certain negative correction. The largest singular value of M is still 1 (or less), so we conclude
|| = 1.
Hence,
1
| (x)]l2
|| V0 +
Vo +
Note that | o² + 2 We know RMS
(x)
2
|BLN (x)ll2- In practice, usually
observe o² > 1 in later training stage Li et al. |BLN (x)]l2 < 1.
(002aa).
Therefore, have [| (z)]l2 z
A.4 MULTI-LAYER CONCATENATION
Checking the partial derivative w.r.t. F1+1, have:
ac
AL
L-1
0rs+1
0L
L-1
II Jr. k=1+1
dr
dz
Ork
DEL
k=1+1
For Pre-LN, we have
0C
Oc
0x1+1
00,
0zi41
00,
ac
8r
J. Pre-LN
0.L
30;
W=1+1
0c
T--1
II
(1+ A
Pre-LNN LN L 00:
0z1+1
011.
k==++
21

---

Published as a conference paper at ICLR 2026
Meleed
Meiil
Maiic
Jaxie
MOA
Oprieegg
CMNE
6:00-3210
6365-300
8538
10203-00001
Toke
03100000
0-2N600001
wedd::
8500
118
(0200-0000
Pald kym
Takk
01179100011
0.2751000011
(.515
3339
Table A: Zero-shot performance of pruned models (25% pruning rate) using different pruning strate- gies. The best results are marked in boldface. "1:1:8" refers to an iterative pruning process where layer is pruned time, and total of layers are pruned by the end of the process.
Due the existence of I, the gradients will not vanish when larger number of layers concatenated. Besides, for deeper layers, when / is close to L, we are more likely to have a I1k=4 (I)4 + A ers are less important than shallow layers due to their gradients do not contribute much.
LN 1e than is close to / than shallow layers, indicating that deep lay-
For Post-LN, we have
0C
OL
0r1+1
00,
dx1-1
00:
0C
L
JP
0z4+1
d.
001
k=l+1
0L
L-1
0m1+1
II
B
Post-IN LN
A
Po
INN
dæg
00,
We know ||BPoottLN < 1. From the definition of A Post we know ||Af is usually close to 1. Then, we usually have [| Post-INN| < 1. This indicates that shallow layers, which are followed by larger number of subsequent layers, tend to receive smaller gradients and may even experience gradient vanishing. Therefore, with Post-LN, shallow layers are not important.
B WILL ITERATIVE PRUNING OUTPERFORM ONE-SHOP PRUNING?
In this subsection, we provide insights into the optimal pruning strategy for LLMs. Although Mu- ralidharan al. 2024) have explored pruning strategies and concluded that iterative pruning offers no benefit, their study focuses on utilizing knowledge distillation (Hinton 2015) for performance recovery. In contrast, this paper concentrates on layer pruning with LoRA and partial-layer fine- tuning, thereby broadening the scope of pruning strategies evaluated. We briefly introduce the one- shot pruning and iterative pruning:
One-shot Pruning. One-shot pruning scores once and prunes the model to target prune ratio.
Iterative Pruning. Iterative pruning alternately processes the ccore-prune-update cycle until achiev- ing the target prune ratio.
and Sattmma--2B-Instteeee as the base models. For one- shot pruning, we prune layers from the Llama-3.1 -88-t-rttuttt and 6 layers from the Gemma2-2B- Instruct in single step, guided by the reverse-order and taylor metries. For iterative pruning with LoRA, we begin by scoring all layers using these metrics. Subsequently, we set the pruning step to and for Llama-3.1 1------IInstrctt and and for Gemma2-2B-Instmun step, we fine-tune the model with LoRA and merge LoRA weights back into the fine-tuned model. This pwwpprrnnnn--uee--nteee 3.1-SB-Instruct and layers for Gemmaa-2B-Insiee tuning, we fin-tune the model using partial-layer fine-tuning (Im + last three layers) after each pruning step, and then repeat the score ceree-prueefeeeee fine-tune cycle. To avoid the fine-tuned layers being pruned completely, we set the pruning step size to 1. All hyperparameter settings are the same in Section 3.3 Experiments with iterative pruning of more layers provided in Table N
Specifically, we select J-ama-3.1-8B--nnst
After each pruning
cycle is repeated until a total of layers are pruned for Llama-
For iterative pruning with partial-layer fine-
22

---

Published as a conference paper at ICLR 2026
Results. By comparing the results of iterative and one-shot pruning in Table A and Table N find that unlike traditional CNN pruning, which often yields significant performance improvements through iterative pruning (Tan & Motani 2020; He & Xiao 2023), the iterative approach for LLMs may not provide the same benefits and can even lead to performance degradation. We believe that is because too much training causes the model to suffer from catastrophic forgetting (Zhai al.) 2024 Liu et al. 2024a), which is evidenced by FigureBi It visualizes the representational similarity of different pruning strategies. From this, we observe that different pruning strategies yield sig- nificantly different representations, highlighting the impact of each strategy on the model's 's learned features. Besides, iterative pruning requires more computational overhead than one-shot pruning,
which is not cost-effective with limited performance gains.
Insight: Considering both performance gain and computational overhead, iterative pruning has no benefit.
C DIFFERENCES FROM TRADITIONAL LAYER PRUNING
Unlike traditional Deep Neural Networks (Szegedy et al. 2014 Simonyan & Zisserman 2015 He
et al. 2015; Dosovitskiy et al. 2021 Liu et al., 2021) (DNNs), typically trained for a single, spe- cific task, LLMs designed to handle wide range of tasks and are structured with billions of parameters. These differences in model scale and task complexity fundamentally alter the chal- lenges associated with layer pruning. For example, traditional DNN layer pruning (Chen & Zhao 2018 Wang et al. 2019. Lu et al. 2022, Tang et al. 2023 Guenter & Sideris 2024), assessing the importance of each layer is relatively straightforward, as it is tied to a single task. In contrast, the parameters of LLMs are optimized across diverse tasks, complicating the evaluation of layer impor- tance. Furthermore, traditional DNN pruning commonly involves full parameter fine-tuning after pruning, while LLMs often employ Parameter-Efficient Fine-Tuning (PEFT) techniques (Hu et al.) 2021 Meng et al. 2024 Zhao et al. 2024b Dettmers et al. (204)) such as Low-Rank Approximation (LoRA) (Hu al. 2021] to accommodate their massive parameter space. Consequently. traditional
DNN pruning methods may not adequately address the unique challenges posed by LLMss, high-
lighting the need for specialized pruning strategies.
D EXPERIMENTAL DETAILS
2023 Muralidharan et al. 2024 and use Wiki'Text2 for calculation. Following (Ma al. 2023). randomly select 10 samples from BookCorpus (Zhu ct al. 2015) to compute Taylor and BI, truncating each sample to a sequence length of 128. Unless otherwise specified, we utilize the Alpaca-cleaned (Taori et al. 2023) with LoRA recover the performance. Uniformly, we set the training epoch to 2 and batch size to 64. All experiments are conducted on NVIDIA A100 GPUs with 40 GB of memory and NVIDIA RTX A5000 GPUs with 24 GB of memory.
For the PPL metric, we follow (Ma et al.
D.1 DATASETS DESCRIPTION
MMLU (Hendryeks et al., 2021) is massive multitask dataset consisting of multiple-choice ques- tions from various branches of knowledge. The dataset spans subjects in the humanities, social sciences, hard sciences, and other areas and covers 57 tasks including elementary mathematics, US history, computer science, law, and more.
CMMLU (Li et al. 2023a) is a comprehensive Chinese assessment suite specifically designed to evaluate the advanced knowledge and reasoning abilities of LLMs within the Chinese language and cultural context. It covers wide range of subjects, comprising 67 topics that span from elementary to advanced professional levels. CMMLU includes subjects that require computational expertise, such physics and mathematics, well disciplines within humanities and social sciences.
PIQA Bisk et al., 2020) is a dataset for commonsense reasoning, and is created to investigate the physical knowledge of existing models in NL..
23

---

Published as a conference paper at ICLR 2026
Beachmalks
PIQA
HellaSang
ARC-2
ARC-
MMLU
CMMLU
WinGGanle
Aug Aue
Alpaca cleaned
00758:000105
0.32500000
0.3000000077
O 00 0L00
0.000
0,0666100133
0,807
MMLU
06012200114
0271420004
0.3430
0,2457400012
0,5888400000
0,5896300138
04165
Table B: The effeec of SFT datasets LLM layer pruning.
HellaSwag (Zellers et al. 2019) a challenge dataset for evaluating commonsense NLI that is specially hard for state-ol-the-art art models.
OpenBookQA (Mihaylov et al., 2018) is new kind of question-answering dataset modeled after open book exams for assessing human understanding of subject.
ARC-easy Clark et al. 2018) and ARC-challenge Clark et al. (2018) are two subsets of the A12's Reasoning Challenge (ARC) dataset, which is multiple-choie question-answerin dataset containing questions from science exams from grade 3 to grade 9.
WinoGrande (Sakaguchi et al., 2021) is new collection of 44k problems, inspired by Winograd Schema Challenge (Levesque et al. 2012), but adjusted to improve the scale and robustness against the dataset-specific bias. Formulated as fill-in-a-blank task with binary options, the goal is to choose the right option for a given sentence which requires commonsense reasoning.
Benchmarks
Propartion
PIQA
HellaSwag
OpenbookKAA
ARC-e
ARC-
MMLU
CMMLL
WinGGrade
Aug Ace
1.0
0,0000
0.3231000000
0.0080
0.726010.0092
046641000166
0.6567_.00008
10,055
0,6666100133
0.507
0.8
0.7372-0.0105 0.5279000000
0:300000.00000
0:733--00002
04565+00146
0.6915-00088
0.4777000045
066667000138
0.764
0.6
0.739900.0012 01524240.0000
0.3040
00000
0,497100115
0.6511000008
0,6922000133
0.5447
0.519410,0000 0,0000
04568100146
0,6500:000088
0.551
0.0600100013
0.574
0,788900.010
0.5077
0.20000800206
04360400145
0.6455-000035
0.5458:0000055
0:690000133
0.5063
Table C: The number of samples used in SFT.
Model
Parmms
MAC'S
Memory
Latency
Jama-
Alppa-
6.291
656
2398-MMBB
Table D: Statistics of -..ma--.--6.3--I--AA
E SENSITIVITY ANALYSIS
In this section, we conduct sensitivity analyses the number of calibration samples, the choice of SET dataset and various pruning rates for LLM layer pruning.
The effect of number of calibration samples LLM layer pruning. It is worth noting that some data-driven layer pruming methods, such as BI and Taylor, rely upon calibration samples to generate layer activations. Therefore, we explore the effect of the number of calibration samples on pruning. Specifically, we calculate BI and Taylor metrics using 1, 5, 10, 30, and 50 calibration samples, prune layers based on these metrics, finetune the pruned Llama-3. 1-8B--Instruct models using LoRA, and evaluate their performance through Im-evaluation-haares package. For ease of comparison, we report the average accuracy on datasets. For more details, see Table O Besides, we report the model perplexity on the WikiText and Penn Treebank test set. As shown in Table D we observe that the prumed models, obtained using varying numbers of calibration samples, do affect the model complexity and zero-shot performance, which suggests that for data-driven pruning methods, performance stability should also considered a key criterion when evaluating the quality of pruning technique. It is worth noting that the reverse-order pruning method can complete the pruning without any calibration samples.
The effect of SFT datasets LLM layer pruning. In the main text, we uniformly utilize Alpaca- cleaned TToor et al.) 2023) to fine-tune the pruned models. Herein, we aim to assess how fine- tuning a pruned model using different SFT datasets affects its performance. Specifically,
conduct experiments using the Reverse-orde metric to remove 8 layers from the Llama-3. 1-8-11-----I--- and fine-tune the pruned model using Im head last three layers MMLU (training set) (Hendrycks)
24

---

Published as a conference paper at ICLR 2026
et al. 2021). We set the maximum sequence length to 512 for MMLU. From Table B we observe that fine-tunin tie-tining with different SFT datasets significantly affects the performance of pruned models, highlighting the need for further exploration to identify the most suitable datasets for fine-tuning pruned models.
The effect of different pruning rates on LLM layer pruning. We investigate the impact of pruning the LLM at various pruning rates in Figure [A] Specifically, we conduct one-shot pruning on Llama- 3.1-SB-Instruet using reverse-order and taylor metrics and evaluate their effects on the model's per- formance with LoRA. All hyperparameter settings remain consistent with those in Section [.3] As shown in Figure A we observe that as the number of pruned layers increases, the performance the model on all datasets tends to decrease and eventually converges. However, certain datasets, espe- cially MMLU, CMMIJ,, and ARC-c, are highly sensitive to layer changes and degrade faster than others. Besides, after cutting off about 16 layers, the model was damaged, so we set the maximum pruning rate in the paper to 16 layers.
The effect of different numbers of samples used in SFT. According to Table B the choice of SFT dataset significantly impacts model performance. To investigate further, we conduct additional experiments analyzing the effect of the number of samples used in SFT. Specifically, we use 20%, 40%, 60%, 80% and 100% of the Alpaca-cleaned dataset for partial fine-tuning. As shown in Ta- ble[c] we find that the number of samples used in SFT indeed affects the performance of the pruned model. Using only 20% of the dataset leads to substantial performance decline.
Pruning without any subsequent training. To demonstrate the effectiveness of our reverse-order pruning method, we compare it with SLEB, a well-known retraining- -free pruning method in a set- ting without post-pruning training. To further illustrate the effectiveness of our method and provide a more comprehensive comparison, we also add LLM-Pruneer to this "no-fine-tuning" comparative experiment. Specifically, we apply SLEB,
and Reverse-order algorithms to prune the Llama-3.1-8BB and Llama-3-8B models, respectively, and then evaluate their zero-shot perfor- mance directly on benchmarks without any fine-tuning. As shown in Table KJ even without any fine-tuning recovery, our simple reverse-order pruning significantly outperforms SLEB and LLM- Pruner in terms of average accuracy on Llama-3.1-8B-- and Llama-3-8B. This further demonstrates the effectiveness of our method, proving its strong competitiveness even pure, retraining-free compression technique.
Generality of partial-layer fine-tuning. In Table 2 of the main text, we have demonstrated the effectiveness of partial-layer fine-tuning on reverse-order pruning. To verify the robustness of our partial-layer fine-tuning method, further apply it to a completely different pruning metric, Tay- lor, in Table H. Experimental results show that after using Taylor pruning, the average accuracy using traditional LoRA for fine-tuning is only 0.4796. However, when switch to our proposed partial-layer fine-tuning on the same model using Taylor pruning, the average accuracy significantly improves 0.5701. This experiment fully demonstrates that our partial-layer fine-tuning strategy is not only applicable to reverse-order pruning, but is also a universally effective performance recovery method.
How many layers are appropriate for partial-layer fine-tuning? In this subsection, we aim to extend partial-layer fine-tuning beyond the last three layers. Herein, we include experiments for more settings (e.g., last 5, 6, 7, 8, 9, 10 layers) on ---ma-.1188--nnstttt the number of fine-tuned layers increases, the average accuracy improves slightly, but the increase is very limited. However, during this process, GPU memory usage increases from 48.02GB to 69.99GB, and training time also increases from 7931.36s to 10316.53s. Therefore, considering both performance and computational cost, fine-tuning the last 3 layers can achieve most of the performance improvement with relatively low training resource consumption.
As shown in Table E as
Partial-layer Fine-tuning vs. Full Fine-tuning. In this subsection, we explore the effects of the proposed partial-layer fine-tuning and full fine-tuning (training from scratch). Due to the extremely larger models, we choose TinyLlama-I.11 (Zhang et al.) 2024a) as our test model. Specifically, we prune the last four layers. Then, we compare two ap- proaches: fine-tuning only the last three layers of the remaining layers and training from scratch. As shown in Table F. fine-tuning the last 3 layers achieves an average score of 37.62, while training from scratch achieves an average score of 37.84. This experiment strongly demonstrates that our
high cost of "training from scratch"
25

---

single NVIDIA Published as a conference paper at ICLR 2026
Type
MMLU
CMMELU
PIQA
OpenboolQA
WineGrande
HellSwaag
ARC
ARC
Inx Jestllllat 3 luyers
0.6567
0.5515
0.7383
0.308
0.6646
0.5323
0.7250
0.4654
0.3907
48,020
7931 36.
In Ausaf lest 4 laners
0,6437
0,5530
0.7405
0.2540
0.6585
0.5370
0.7285
0.662
0.5791
49.500
8242.46a
to iu.hrssfoout 5 uperrs
0.639
0.5545
0.2448
0.3.80
016811
0.591
0.342
04669
0.3848
51.4963
6372 15x
In ia.hhosfeeott upprr
06614
0.5495
0.3476
0.3:40
B16998
0.5418
0.7452
04633
0.5855
58.18K3
6900 82x
In in.Acad-fos lest 7 lspers
0.6877
0.5530
0,7425
0.2940
816685
0.5370
0.7255
0.4635
0.5791
58,896
9259 695
ln.Aradf- leat ispers
0.6455
0.5431
0.7897
0.2540
0.6899
0,545
0.7572
0.4382
0.5836
62.596
9607 296
in.Aesul-lo lest liynery
0.6457
0.5433
0.7519
0.2960
0.6835
0.7929
0.4642
0.5860
66
in Aw..vaateii laywer
0.6576
0.7470
0.3000
0.6748
0.7521
0.4753
0.5875
69.990
10316.53
Table E: Zero-shot performance of pruned Jlama-3.1-8B- models using various fine-tuning meth- ods under 25% pruning rate (using reverse-order).
Method
MMLU1
CMMLUf
PIQA"
OpenbookkQA
WinoCirandet
HellaSwagt
ARC-ef
ARC-cT
Avg Accf
brad fayers
3497
0.2479
0.6480
0.1940
0.5541
0.3937
0.3762 0,520
0:2708 0.2491
Pull line muning
0.3478
0.2497
0.6670
0.3879
0,4714
0.3784
Table F: Zero-shot performance of the pruned model using partial-layer and full fine-tuning.
method can achieve performance almost equivalent to training a model of the same size from scratch with a much lower computational cost than training from scratch.
Perplexity of the pruned models. Perplexity is an important metric for evaluating the language modeling capabilities of prumed models. Therefore, we evaluate the performance of various pruned models (20% pruning rate, Llama-3-8B) on WikiText-2. As shown in Section E. our method achieves the lowest perplexity value of 19.16, significantly outperforming all other pruning methods. This further demonstrates that proposed method can maximally preserve the general generative capabilities of the pruned model.
Method
Wanda
LLM-Pruner
ShortGPT
OWL
GISP
Ours
Perplexity|
29.92
23.21
118.62
29.49
24.18
19.16
Table G: Comparison of perplexity of models obtained by different pruning methods.
(MoE) models. To further demonstrate that our pruning algorithm can be applied different model architectures, especially MoE models, we pruned the Mixtral 8x7B (Jiang et al. 2024) model. Specifically, we prune the last 5 blocks of the Mixtral 8x7B and compare with the MoE pruning baseline UMD (He et al. 2024) with the same pruning rate. As shown in Table H our method achieves average accuracy of 0.6673, which is higher than UMD's 0.6558. This further demonstrates that our method has good architectural generality: even in complex MoE models, simple reverse pruning strategy still outperform existing MoE pruning baselines and achieve better performance.
Pruning Mixture-of-
Performance comparison HumanEval. To further illustrate the reasoning capability of the pruned model obtained by our method, we evaluate the performance of pruned models obtained by various pruning methods on HumanEval (Chen et al. [ (02TT) using CodeQwen1.5-785 5--7B-hhat (Bai al., 2023). We set the pruning rate to 20% and compare it with ShortGPT, UIDL (Gromov et al. 20224) Linearity (Razzhigaev et al., 2024). SLEB, and LLM-Pruner. As shown in Table [] our method achieves the highest PassI at 48.72, significantly exceeding ShortGPT (42.68) and LLM--runer (15.85). Finally, the superior performance in code benchmarks further validates the effectiveness and generality of our method.
Statisties of Llama-3.1-6.3-II--AAaa
Table D presents the statistic of
Alpaca, including parameters, MACs, memory requirements and latency. Following Ma et al. (202), the statistical evaluation conducted in inference mode, where the model is fed a sentence consisting of 64 tokens. The latency is tested under the test set of WikiText2 A100 GPU.
Limitations and Future Work. In this paper, we focus primarily on layer pruning due to the straightforward nature of pruning layers in LLMs, where the input and output dimensions are iden- tical. However, we plan further investigate weight pruning (Sun al. 2023 Frantar & Alistarh, 2023) and width pruning (Ma et al., (023) in future experiments. Besides, in this paper, we gain insights by focusing solely on the upper bound of the f--norm of the gradient flow. We acknowl-
26

---

Published as a conference paper at ICLR 2026
Method
MMLU
PIQA
OpenbookQA
WinoGrande
HellaSwag
ARC-c
Avg Acc
UMD
0.6790
0.7930
0.4200
0.7430 0.7674
0.7870
0.5130
0.6558 0.6673
Ours
0.6957
0.7890
0.4321
0.7981
0.5216
Table H: Performance comparison of pruned MoE models.
Method
ShortGPT
UIDL
Linearity
SLEB
LLM-Pruner
Ours
HumanEval(Pass@@] I)
42.68
0.00
0.00
20.73
15.85
48.72
Table I: Performance comparison on HumanEval.
edge that matrix multiplicatton is significantly more complex than simple norm, particularly in the context of LLMs. comprehensive analysis beyond the scope of this paper and is left for future work. Besides, since quantization may alter gradient flow activation distribution, making otherwise unimportant layers crucial. Therefore, we will investigate the impact of quantization on layer importance in the future.
Figure A: The effect of different pruning rates on LLM layer pruning.
27

---

Published as a conference paper at ICLR 2026
Verifeation
PPL
Maine
Bi
Tayler
Tayler
Tayyker
51.06
65:43
90.97
94.35
0.40
0.36
61:41
Calibation Samplex
55.55
50.00
55.42
0.55
50
59.73
35.42
103.19
77.63
0:41
0.55
Table J: The effect of number of calibration samples on LLM layer pruning. For more details, please refer to Table O
Model
Minhod
PNAA
Belliwag
OpenhonrgA
ARC-
ARDS
MNLU
CMMLA
Wawkione
Arg Aro
ST3R (AR tre-
1050
0730050099
6.70666 003
94160
Revere enter twb Raeresing
1049
SLB8 (wil lne-tassg?
64401
090145500000
022077903631
0.26149950001
0.200a55 1040
0.469
Reieriorele tolo Werreiing
0.975500155
6.000
Table K: Comparisons with retraining frreen (pruning without fine-tuning) pruning method.
Figure B: Visualization of the layer similarity matrix of 16-layer pruned 1lama-3.1-8B- models (using Taylor) obtained by different pruning strategies. Left: one-shot pruning; Middle: iterative pruning with pruning step 1; Right: iterative pruning with pruning step 8.
Berchmarks
Model
Metiic
P924
HellSwong
OpecbockgA
ARC-
ARC-
MMLLI
CMNERU
WinCCroke
Arg. Are
Rundors
05733000115
0 10300
0. 15600000162
0.500/2000111
0L323
Magitwdell
0523-00011
0.505 0.066
B1 11000-000155
0 (2635-100000)
0.770--01115
0.970
Meguitude12
0:000:000004
110000001151
020502000001
0.4991000040
BC
05520=000177
0. 0.2044
0. 1404:001117
0 022405000022
0.19282000115
02476600
0.7688000144
02965
Teylor
05331-00110
01000001155
02572500000)
22167200012
02014400007
01
Resene-sriicr
165622000166
02009:500045
0 FW0e86018
0.245:200229
#3669588943
83872590036
0.685
Dane
00006
Rondor
U6IN-0011
IL366 8.8647
IL1948-68811
02261-00022
07711-0006
PPL
66529991111
03293-00042
01100000016
0.430-000022
02099-000199
0254180044
0.925400140 40
03498
05452=00010
Qweil 7B
Maguunak-22 05548:00116
1.26111100044
B 15000100161
O 10050000000
0.1845200113
035191000000
03357
Teylor
05223-00117
02540:000043
8 1189=000154
0 02433-500065
021766400121
0247860000 0340
0.541
0.5731000155
0.3100:000046
1600:000116
02333000124 0.044
Dane
83001-90005
015900:000099
0338902006212
0.555220
073051100123
06399
Rondon
01523 009841001110
0.770
0/0044
0.3019
94110
6.4688
0.1999068178
O
033855000226
026660090
05458500140
Magrnukel1
83516
0001
0.225
61190
(8020)
09556118
Limm--3 1-80
2agrindee2
03516-00116
01576-00004
1300-00115
0 0057
1960H00116
Bi
15773-00015
0 015000001161 038755000045
03674200009)
023442000066
02406200000
0.5066200044
0.174
Revero-crttrr
(1576 00112
11166 0.0066
IL (L198: 0.817
(000000000011
0.318K.
0,2422 nURdS
039621000400
BAT0L10ULE OUILTT
6,3714
Table L: Zero-shot performance of the pruned models (50% pruning rate, fine-tuning using LoRA). "Avg Acc denotes the average accuracy calculated among eight datasets. The best results are marked in boldface, and the sub-optimal ones underlined.
28

---

Published as a conference paper at ICLR 2026
Seachenat
Mode
Metied
Taper
Aug Mg
LabA
0103601005
DAIRTACA1
64W
Moll
81466200011
0463122200
0.394
0.49%
Neadivaat Aayer
0.350
0.7888990027
0.501
Table M: Zeoo performance of the pruned models using various fine-tuning methods under 25% pruning rate (using Taylor metric). "Avg Acc" denotes the average accuracy calculated among eight datasets. The best results are marked in boldface, and the sub-optimal ones are underlined.
Bockrat
RRA
QptbeedA
ABCe
ABCe
MILU
OBLE
WaaKrt::
wedla:
0335500006
019010013
0315000115
0.3230005
Goll G0ll
037H
Beresoanzrr
1816
0400-0001
05190-006
0199-0005
(199-0001)
0599-0001
0265:006)
0000-00117
8
0033J002
030100191
0390000
(39100641
09:20007
L2708
Lew 31 SkD
Tyyke
0-1770004)
10tt
0.19000011:
0.501 20012
GADE 200135
LGH
1::16
(n (1946-00ms
5-0000
0113)-001s1
1993-00tt
0-955-00011
Table N: Zero-shot performance of pruned models (50% pruning rate) using different pruning strate- gies. "Avg Acc" denotes the average accuracy calculated among eight datasets. The best results are marked in boldface. "1:1:12" refers to an iterative pruning process where layer is pruned at time, and a total of 12 layers are pruned by the end of the process.
Layss
Ang
0342
(20411.00)
02008 weres
0422 (0099
62171
1204
144010064
044
30,
(1200-03131
63001 800
E50010003
103
Tabe
3.804401116
0"
27188-01209
030014589 8m
015
Table O: The effect of number of calibration samples on LLM layer pruning.
Benchmmakk
Reseline
Parimeters (TTakms)
HellasSag
OpenbookQA
Arg Ace
(370M)
ChaGLM2
6.24B (1.41)
0.2411
Bukchun2:78
7.518 (2.01)
0.7666
0.5024
Qwen).5-77 5-78
(18T)
0.7845
0.4053
0.5967
0.7777
LLaMA3-8B
(15T4)
0.7965
0.6114
0.5005
0.6212
0.752
0,4993
Gamma2-78
8.548 (6T)
0.6039
0.110
0.5009
06143
0.601
(15T+)
0.8003
0.5910
0.5179
0.6299
Llama 3.ama334.38-Appscc
6.298 (12.7MM)
Llano-3.1-6.380 1...0..3.16.38.ttAAppa..
6,298 (12.74M)
0.5807
Table P: Performance of our pruned models with respect to similarly-sized community models. "TTokens" denotes the training tokens. The best results are marked in boldface, and the sub-optimal ones are underlined.
29

---

Published as a conference paper at ICLR 2026
Model
Generated Sentences
To be or not to be, that is the question that has been asked thousands of times throughout history. but have you ever wondered what would be like to not exist? what would feel like to not have consciousness or awareness of any kind? would be like being asleep or combed hair?
would be something else entirely. it would be like not being here at all, like being tree rock, piece of furniture or piece of trush, nothingness is not something we can really comprehend because we can't experience it. we can't even imagine because we can't imagine.
It fills me with such pride and joy knowing that have been able to help shaped the lives of countless stu- dents over the years, am proud of the numerous accomplishments and nchievements of my students, and I am proud of the role have played in shaping them into the people they are today. As teacher, I believe it is my duty provide optimal learning environment that fosters growth and development of my students. believe in making learning fun and engaging, and I am dedicated to pushing the bounds of innovation in the classroom. am dedicated to ensuring that my students leave my classoom with the knowledge and skills they need... ...
Llama-3.116.22 1-6..BB It-lpaca
What's great about the huliday season? Christmas lighus, Christmas shows, Christmas presents, Christmas holiday traditions. But what's 's not great about the holiday season? crowds, stress, Santa Claus, Christmas holiday stress, Christmas holiday stressors. It's important remember to do things that help you relax during the holiday season, such as taking time for yourself, engaging in reloxation techniques, practicing mindfulness, engaging in physical activity, practicing gratitude, practicing self-care, engaging in activities that bring you joy, and spending time with loved ones. These are all important components of stressors prevention during the holiday season. Here some tips help you...
To be or not to be, that is the question. How does one decide whether a being is alive or not? The issue of what constitutes life has puzzled philosophers and scientists alike for centuries. A being is alive if it meets the criteria of life, which are: Metabolism: The ability to carry basic functions such respiration. photosynthesis, and nutrient uptake. Homerstasis: The ability to maintain a stable internal environmnnt, despite changes in the external environment. Growth and Development: The ability to incrense in size and complexity over time. Reproduction: The ability to produce offspring, either sexually or asex
It fills me with such pride and joy tn see my friends' passions ignite into flame and to watch them soar to great heights! It was an absolutely honour to have been able to share this experience with you and to hear witness to your hard work dedication! was an absolute pleasure to get to know you and share our experiences. but I know that this just the beginning of an amazing jourcy! I feel so grateful to have you in my life, and I am constantly in awe of your love, kindness, and generosity! It's been such an incredible journey watching you discover and grow, and feel so lucky to have...
L.ama-3.1-8B-4
What's great about the holiday season? In addition being able spend time with loved and get some much-needed R&R, many people enjuy the idea of giving back to others. Whether it's volunteering. donating to charity, or participating in a Socret Santa gift exchange, the holiday season can be time kindness and generosity. But have you ever thought about how you might be able to combine your love cooking and giving back this holiday senson? If so, you might be interested in hosting charity-theme potluck dinner or bake sale. Here are a few ideas to get you started: Host a potluck dinner to...
Table Q: Generated Examples from the Llama-3. 6.3B-It-Alpaca and Llama-3, 1.mma-3.1-80 18B-II.
30


Reviews:
[{"review_id": "mCAFX6HKkP", "reviewer": "Reviewer_tCfY", "summary": "The paper is about empirical benchmarking and methodological clarification for layer pruning.\n\nBenchmarks 7 layer-selection metrics and 6 fine-tuning methods across Vicuna-7B, Qwen-7B, and Llama-3.x models.\n\nFinds that reverse-order pruning (dropping last layers) consistently outperforms complex importance metrics.\n\nShows partial-layer fine-tuning (LM head + last 1–3 layers) surpasses LoRA/QLoRA for accuracy and training cost.\n\nExtends tests to Llama-3-70B.\n\nReports 2-19 pp improvement over prior layer-pruning baselines.\n\nAdds a gradient-flow derivation explaining why deep layers matter less.\n\nNotes that iterative prune–tune cycles provide no benefit over one-shot pruning.", "questions": "Ought to address:\n\n- Clarify that the theoretical contribution is an application of prior analyses, not new theory.\n- Discuss why reverse-order pruning should be a baseline control for future pruning papers.\n\nI think the following is really future work:\n\n- Benchmark versus quantized models at equal memory budgets.\n- Investigate interaction of quantization and depth--does aggressive quantization change which layers are dispensable?\n- Compare against GPU-usable sparsity (2:4) and width pruning for completeness.\n- Compare with models trained from scratch at the final depth, as well as models incrementally grown.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "Comprehensive and reproducible experimental design.\n\nHonest ablations revealing when complexity adds no value.\n\nSimple, clearly-defined recipe that practitioners can reproduce in hours.\n\nReally primarily illustrates a weakness in all the other papers on layer pruning: they ought to have used final layer pruning as the obvious control experiment and have failed to do so. Providing this missing baseline is probably important within the narrow domain of layer pruning.\n\nExperimentally verifies a fact that is part of the design of LLM architectures and their understanding as unrolling, and has also been examined theoretically and by other experimental methods before.", "weaknesses": "Scope: confined to layer pruning; ignores dominant GPU-friendly methods (structured width pruning, 2:4 sparsity, quantization).\n\nNovelty: theoretical component re-derives known results; empirical finding is mainly that others’ metrics fail.\n\nPractical relevance: minimal for most users in practice. For people training from scratch, incremental deepening is probably preferable. For people trying to squeeze a large model into a slightly smaller GPU, quantization and GPU-friendly sparsity are probably preferable even if smaller models aren't just available. The primary use case is where a user has an unusual model, cannot control training, but quickly wants to squeeze it into an existing GPU with somewhat more limited space.\n\nThe paper really ought to have compared final layer pruning with models of the same final size trained from scratch, since they are architecturally identical. This would indicate whether final layer pruning could be a useful shortcut for generating a simple multi-depth collection of models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nII3u1uhJm", "reviewer": "Reviewer_mE3b", "summary": "This paper re-evaluates layer pruning for Pre-LN LLMs and shows that a simple strategy that prunes layers in reverse order and then fine-tune only the LM head plus the last 1-3 layers consistently matches or even outperforms more complicated pruning methods on a few standard benchmarks (PIQA, HellaSwag, WinoGrande, ARC-e/c, OBQA, MMLU, CMMLU). The empirical study is broad (several LLaMA and Qwen-style models) and scales up to LLaMA-3-70B. The authors give gradient-flow explanation for why deeper layers in Pre-LN are matter less, and they also find that this approach can beat the usual \"prune + LoRA\" recovery. This makes the paper especially useful for users who just want a reliable pruning recipe without complex per-layer scoring.", "questions": "- Can you add some generation/reasoning benchmarks (e.g. GSM8K) to verify that reverse-order pruning still holds outside multiple choice tasks?\n- Please also report the perplexity on some perplexity based data sets e.g. wikitext to see how that varies across varies different techniques.\n- Do the results apply for different model architectures as well? (it would be interesting to see the results on some models mixture of experts models e.g. Mixtral 8×7B)\n\nI'd be happy to increase my score if these experiments are included!", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Clear recipe to prune layers in reverse order and fine-tune only the LM head alongside the last 1–3 layers.\n- Reasonable empirical baking, tested on several LLaMA-3 and Qwen-style models at several pruning ratios, and several standard benchmarks, and it still works at 70B scale.\n- Practical impact, simple post-pruning FT outperforms the common \"prune + LoRA\" setup.\n- Plausible architectural explanation, the Pre-LN gradient-flow analysis motivates why late layers are safer to drop.", "weaknesses": "- They don’t evaluate on generation or reasoning datasets (e.g. GSM8K), so the conclusions are validated only on specific LM-harness-style multiple-choice tasks.\n- Prior work shows that layer importance depends on the nature of the task. Without generation tasks, the paper assumes task-invariance of the \"prune-from-the-top\" rule. Later layers tend to be more critical for perplexity, so pruning them first might hurt exactly the tasks they didn’t test.\n- As a result, the current recipe is a strong default for classification-style LLM evals, but its generality to generation remains unproven.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ULnrI4m9Iy", "reviewer": "Reviewer_Ncfm", "summary": "This paper re-evaluates layer pruning methods for Large Language Models (LLMs), addressing whether complex metrics are needed to identify redundant layers and if LoRA is the optimal fine-tuning choice after pruning. Through extensive experiments across various metrics, LLMs, and fine-tuning methods, the paper reveals that a simple \"backward pruning\" (removing the last few layers directly) often outperforms more complex indicators. Furthermore, \"partial layer fine-tuning\" (tuning only the last few layers and the output layer) is found to be more effective and faster than LoRA for performance recovery. This paper provide a theoretical framework based on gradient flow to explain why deeper layers in Pre-LN Transformers contribute less, validating their approach. Pruned models based on these findings significantly surpass existing methods across benchmarks.", "questions": "1.Could you show the results of different pruning metrics without any subsequent training?\n\n2.Have you compared pruning using other metrics (e.g., cosine similarity, perplexity) followed by fine-tuning only the layers immediately surrounding the pruned sections?\n\n3.Recent work suggests deeper LLM layers are crucial for reasoning[1]. Does direct pruning of the final layers impact reasoning capabilities? It would be beneficial to evaluate this method on mathematical and code-related tasks to assess its performance in reasoning.\n\n[1] Song, Xinyuan, et al. \"Demystifying the Roles of LLM Layers in Retrieval, Knowledge, and Reasoning.\" arXiv preprint arXiv:2510.02091 (2025).\n\nIf the author's response addresses my questions, I will consider increasing my rating.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.Comprehensive experimental design covering diverse pruning metrics, fine-tuning methods, and models.\n\n2.The proposed \"backward pruning + partial layer fine-tuning\" strategy is simple yet effective.\n\n3.Theoretical analysis using gradient flow provides a rationale for the method's efficacy.\n\n4.Achieves significant performance gains across multiple models, outperforming other methods.", "weaknesses": "1.Inconsistent calibration datasets and data volumes were used for different pruning metrics, which could affect experimental fairness.\n\n2.The performance of the pre-pruned models should be included in the results tables.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "US7LMRU6C4", "reviewer": "Reviewer_PD7z", "summary": "The paper relates to the pruning of LLM layers. The paper consists of three main parts:\n1. Discussion of criteria for identifying prunable layers\n2. Comparison between LoRA and partial fine-tuning methods for recovering accuracy after pruning\n3. Theoretical analysis of gradient flow in the presence Pre-Layer Normalization, and how this affects layers by depth\n\nThe main observation in the paper is the relative unimportance of deep layers, and the fact that pruning the last layers is a more useful heuristic than other more elaborate importance estimators (c.f. Magnitude, Taylor, PPL, BI).\nThis claim is supported by Table 1, which shows superior results for the \"reverse order\" method, at a 20% pruning ratio, for Qwen1.5-7B, Llama-3.1-8B-It and Vicuna-7B-v1.5\n\nA parallel finding is the fact that partial fine-tuning of the last one or two layers yields a greater accuracy recovery than full LoRA fine-tuning.\nThis claim is supported by Table 2.\n\nIn the last paragraph of the main body of the paper, the theoretical analysis of gradient flow and show that Pre-LN architectures inherently weaken the gradients and contributions of deeper layers due to the normalization step scaling them down.", "questions": "\"reverse order\" wins the benchmark at 25% pruning ratio, but does not perform well at 50% pruning ratio, would you be able to do a comprehensive sweep of pruning ratios in order to collect more data points? For example, from 5% to 75%, by increments of 5%.\n\nCan you repeat the experiments in Table 2 with partial-to-full fine-tuning, so we can see which setting is optimal in experimental results?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "The paper is nicely written, and very well laid out, making it an enjoyable read.", "weaknesses": "The paper focuses on depth pruning, however there is abundant evidence in the literature that layer-wise pruning is not as efficient as width pruning. For example, this claim is made in Muralidharan et al. (2024), which is cited in this paper.\n\nIn the main body of the paper, results are shown for a pruning ratio of 25%. We need to read the appendix to see results for 50% pruning ratio in Table G, and these results seem to contradict the main finding of the paper, since the \"reverse order\" method yields inferior results there. The PPL method appears to dominate at 50% pruning ratio, and the \"random\" method even wins the benchmark for Qwen1.5-7B, which raises questions about the relevance of the results.\n\nThe the LoRA vs. partial fine-tuning experiments, the study is limited to partial fine-tuning of the last few layers. Table 2 shows that fine-tuning the last three layers is better than fine-tuning the last two layers, which is better than fine-tuning the last layer. Thus, why stop at three layers? It would seem like if the trend follows, fine-tuning all layers would be optimal?\n\nThe theoretical analysis builds upon prior analyses of Pre-LN vs. Post-LN Transformers (e.g., Xiong et al., 2020; Liu et al., 2020). It's known that Pre-LN helps with training stability by damping gradients as depth increases, avoiding explosions near the output (which Post-LN can cause without warmups). However the theoretical analysis falls short of proving the optimality of fine-tuning just the last three layers.\n\nI could not access any of the files behind the URL (https://anonymous.4open.science/r/Navigation-LLM-layer-pruning-DEB7/README.md) due to \"The requested file is not found\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)