Entropy-Based Block Pruning for Efficient Large Language Models
Authors: Liangwei Yang
Yuhui Xu
Juntao Tan
Doyen Sahoo
silvio savarese
Caiming Xiong
Huan Wang
Shelby Heinecke
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
Efficiency
Primary area: foundation or frontier models, including LLMs

Abstract: As large language models continue to scale, their growing computational and storage demands pose significant challenges for real-world deployment. In this work, we investigate redundancy within Transformer-based models and propose an entropy-based pruning strategy to enhance efficiency while maintaining performance. Empirical analysis reveals that the entropy of hidden representations decreases in the early blocks but progressively increases across most subsequent blocks. This trend suggests that entropy serves as a more effective measure of information richness within computation blocks. Unlike cosine similarity, which primarily captures geometric relationships, entropy directly quantifies uncertainty and information content, making it a more reliable criterion for pruning. Extensive experiments demonstrate that our entropy-based pruning approach surpasses cosine similarity-based methods in reducing model size while preserving accuracy, offering a promising direction for efficient model deployment.

Published as a conference paper at ICLR 2026
ENTROPY-BASED BLOCK PRUNING FOR EFFICIENT LARGE LANGUAGE MODELS
Liangwei Yang, Yuhui Xu Juntao Tan, Doyen Sahoo, Silvio Savarese, Caiming Xiong, Huan Wang, Shelby Heinecke Salesforce AI Research
ABSTRACT
As large language models continue to scale, their growing computational and storage demands pose significant challenges for real-world deployment. In this work, we investigate redundancy within Transformer- entropy-based pruning strategy to enhance efficiency while maintaining per- formance. Empirical analysis reveals that the entropy of hidden representations decreases in the early blocks but progressively increases across most subsequent blocks. This trend suggests that entropy serves as a more effective measure of information richness within computation blocks. Unlike cosine similarity, which primarily captures geometric relationships, entropy directly quantifies uncertainty and information content, making more reliable criterion for pruning. Extensive experiments demonstrate that our entropy-based pruning approach surpasses co- sine similarity- -assd methods in reducing model size while preserving accuracy, offering a promising direction for efficient model deployment.
models and propose
INTRODUCTION
The emergence of large language models (LLMMs) has reshaped current research landscape as well as empowering applications (Dubey et al. [2024] Team et al., 2024 Yang et al. 2025 7hhu et al.) 2025)
Scaling in size, they demonstrate remarkable performance across a wide range of domains/tasks such as chatbot (Achiam et al.] (202)), code generation (Nijkamp et al. 2022). recommendation (Liang
al. |025|/Zaag et al. 2025), etc. Hidden behind these striking achievement, Transformer-basee models WWawwani et al. parameter size from millions to billions and research continues to explore even larger architec- tures (Liu al. 2024) to further enhance their capabilities. However, the increasing scale in sizes result in substantial computational and storage costs, posing significant challenges for deployment.
et al., 2023 Jiang al. 2023 Xue et al. 2024) scale their
Recent researches have detected the inherent redundancy of these pre-trained L.Mss, especially on the layer level (Gromov et al. 22024] Men et al. [2024] Yang et al.] 2024 Xu et al.) 2022; Song et al.) 2024 Chen et al.) 2024 Kim al. (0224.. Models can maintain competitive performance even after a significant number of layers are removed, indicating that not all layers contribute equally. This observation has spurred extensive research on layer pruning techniques, which focus on eliminating redundant layers while retaining the model's core functionalities. LLMDrop (He et al. (2024) further discovered that the Attention block more redundant than the MLP block, highlighting the need for more fine-grained pruning approach to remove redundant components within each block rather than pruning entire layers. This redundancy provides new insights for optimizing model deployment, enabling more efficient acceleration strategies while maintaining performance.
For both layer and attention pruning, existing methods (Men et al., 2024 Yang et al. 2024 He et al., 2024 Mao al., 2024) adhere the practice of using cosine similarity to measure the redundancy between computation blocks. Redundant blocks with high similarity scores are identified and re- moved by comparing adjacent layers or selected layer pairs. However, cosine similarity primarily captures the geometric alignment of hidden representations, which does not necessarily reflect the actual information contribution of each layer. Consequently, relying solely cosine similarity for pruning may lead to suboptimal decisions, potentially compromising model performance.
"Corresooniing Author: yuhui www.sforc.cocooommmmmmmm.

---

Published as a conference paper at ICLR 2026
In this paper, reconsider the use of cosine similarity the criterion for pruning and propose EntroDrop. a novel approach that leverages entropy increase to assess the importance of computa- tion blocks. Empirical analysis reveals that the entropy of hidden representations initially decreases in the early layers but progressively rises across subsequent layers. It suggests that entropy can serve effective indicator of information richness within each block. Unlike cosine similarity, which primarily captures geometric relationships, entropy directly quantifies the information con- tent of a block's oulput, providing a more reliable basis for pruning decisions. Extensive experiments comparing entropy-based and cosine similarity- bbseed pruning demonstrate that our entropy-drive driven approach more effectively preserves model accuracy while reducing computational costs. The code open-sourced to facilitate further research. Our key contributions are summarized as:
We conduct an empirical analysis of entropy dynamics hidden representations across LLM blocks during inference, offering new insights into information flow.
We propose a novel entropy-based pruning strategy to effectively reduce model size and preserve performance.
Extensive experiments demonstrate the superiority of EntroDrop over cosine similarity- based pruning methods.
PRELIMINARY
Transformer-based architectures consist of two primary computational blocks: the Attention and the MLP Block. They process hidden states and enrich them sequentially.
2.1 COMPUTATION BLOCKS
Attention Block enables each token the input sequence interact with others. Given an input
X, Layer Normalization (LayerNorm) operation is applied before the self-attention computation
X norm LayerNorm(X), (X). Then, the attention mechanism computes as:
QKT
Y
Softmax
V.
dd
where Q = X norm W K =
X norm W V
X norm W and Vdl is scaling factor. The output
Y represents the transformed hidden states.
MLP Block further transforms the output of Attention block. Assume the input for MLP block is also X. firstly applies layer normalization to stabilize the output as X LayerNorm Then two-layer feedforward network is calculated to process Xnom as:
Y ReLU norm Wi + b,) + b2,
where W,, 22, bj and b2 are learnable parameters. There are also other variants (Touvron et al., 2023) for this feedforward network. Together, the Attention Block and MI.P Block form a complete Transformer Block. which can be stacked build deep Transformer models. Each Transformer Block refines and enriches the hidden states, enabling hierarchical learning across multiple layers.
2.2 BLOCK- WISE PRUNING
Block-wise pruning aims to determine the importance of each computation block by analyzing the relationship between its input X and output Y. The goal define effective metric that identifies less informative blocks for removal while preserving essential model functionality. To quantify the importance of a block, an importance criterion is often calculated as:
gX, Y)
(3)
where g(.) is a function measuring the information contribution of the block and we prioritize the pruning on blocks with less importance score. No matter which computation blocks, current methods (He et al. 2024] Men et al. 2024) judge the importance by cosine similarity and the impor- tance criterion is calculated as 9(X
X In this paper, we propose entropy increase,
X
2

---

Published as a conference paper at ICLR 2026
Liama3.11
Llarna3.1-8
Llamn3.11
4.50
4.4 012
4.4
4.2
4.00
2.
3.75
5 3.6
3.0
3.25
Law
Low
Medicine
3.2
3.4
Law
Modicine
Wikitext2
Wikhtext2
Wikitext2
2.75
03
10
20
30
40
50
60
01
10
15
20
30
10
20
Block Index
MLP Block Index
Attention Black Index
Mistral-7B-v0.3
Mistral-78-v0.3
Mistral-7B-v0.3
4.50
4.0
4.25
4.2
4.2
4.00
4.0
3.8
8
3.6
C4
C4
3.50
Law
3.4
Medicine
3.2
Medicine
3.2
Medicine
3.00
Wikhtext2
3.0
Wikiteot2
3.0
Wikitext2
2.75
2.8
20
30
Block Index
MLP Index
Attention Block Index
Figure 1: Entropy dynamics among layers during inference
new importance criterion based empirical observations of entropy change across the layers. En- tropy increase can better capture the information flow within the model, providing a more effective metric for identifying redundant blocks.
2.3
ENTROPY DEFINITION
For discrete variable Z with probability mass function p(2), its Shannon entropy is defined as:
(Z) = -===>(())ogp))
which measures the average uncertainty information content of Z. Higher entropy indicates that takes more uniformly distributed values (i.e., richer information), while lower entropy suggests more concentrated representations. In this work, firstly discretize the hidden activations of large language models and compute their entropy layer by layer. This allows us to systematically ob- serve how entropy evolves the model depth increases, revealing two-stage pattern of entropy decrease followed by entropy increase. This empirical finding serves as the foundation for our pro- posed entropy-basee based pruning strategy.
3 METHOD
3.1 OBSERVATIONS ON ENTROPY DYNAMICS
To investigate entropy across different layers of Transformer models, we conduct experiments
on Llama3.1-8B (Dubey al. 2024 and Mistral-7B-v0.3 (Jiang et al. 2023) | entropy trends during inference across Transformer Blocks, Attention Blocks, and MLP Blocks using four datasets: C4, Law, Medicine, and Wikitext2. We compute the entropy of hidden states at each block level and track its evolution across the network.
We analyze the
The experimental results, shown in Fig. reveal consistent two-stage behavior:
Stage Entropy Decrease (Layers 1-3). Early layers progressively reduce entropy, indi- cating strong information compression, noise filtering, and formation of compact represen- tations.
Stage 2: Entropy Increase (Layers 3-32), Subsequent layers gradually increase entropy, suggesting progressive contextual expansion and feature enrichment.
Phttps//huggingfacccco hagp:/gginnaaecccooomaaaaaalaaaa
https://huggingffcccccm http://gggg.atttooommmmmmms/s/aa
3

---

Published as a conference paper at ICLR 2026
Stage 1 Stage
Dalesot
Callbration
Blook
Book
Block
2
AH*
AAH
Am1
AAA
(a) Entropy Increase Calculation (b) Pruning Order Decision
Rank
N
Figure 2: Overview of the EntroDrop framework. Stage keeps intact, while Stage 2 exhibits in- creasing entropy. Blocks Stage 2 are ranked based on their entropy increase, and those with the lower entropy increase pruned earlier.
This pattern appears robust across all four datasets and aligns with prior findings Yang al. 2024 Men et al.) 2024), which highlight the importance of early layers for information preservation. Our results further suggest that later layers contribute more uniformly representation expansion, mak- ing them better candidates for pruning.
Entropy-lens Ali al. (0255) studies entropy dynamics from prediction logits derived from hid- den states, showing that different model families may exhibit distinct output-ievel entropy pat- terns. In contrast, we directly analyze the entropy of raw hidden representations, providing more fine-grained view of information evolution inside the model. Across several mainstream ALMM--ncluding 1lama3.1-8B, Mistral-7B-v0.33 Qwen3-14B, and DeepSeek- V2-Lite- we we consistently observe an early-layer entropy decrease followed by a later-layer entropy increase (Ap- pendix A.2). Although this two-stage trend is not universal across every architecture examined, recurs across many widely used Transformer families, suggesting that such compressioneexpasion dynamics are a common emergent behavior in modern decoder-only LLMs.
3.2 ENTRODROP
Based empirical observations of entropy dynamics across Transformer models, propose EntroDrop, a novel entropy- bbsedd pruning method that leverages entropy increase in later layers to identify and remove redundant computation blocks while preserving essential model performance. The framework is shown in Fig. 2
We consider a pre-trained Transformer model consisting of L computation blocks, each responsible for transforming hidden states as the input propagates through the network. Given a calibration dataset D, we pass input samples through the model and collect the hidden states each block:
(4) where Z represents the hidden at the 1-th block, and fi(-) denotes the computation block function (e.g., Attention, MLP). Once the hidden states at all blocks are obtained, we estimate the entropy of each block and rank them according entropy increases. The lowest K blocks, which exhibit minimal entropy increase, are selected for pruning.
z' Ziz'-1 1 1, L,
EntroDrop leverages two-stage pruning strategy based on entropy observations. Stage com- presses the information, and no computation blocks are pruned in this stage. Stage 2 gradually increases the entropy, suggesting that these blocks perform similar hidden state enrichments. The transition point between the two stages, denoted S is determined using a calibration dataset.
To effectively estimate the importance of computation blocks, we define entropy increase as:
AH'S = H('z) - (Z
(5)
where H(.) represents the entropy estimation function. Blocks in Stage 2, indexed by Saan S are ranked in ascending order based on their entropy increase:
L.
Rank(A
(rgsort(AAHT)) for I. Sann
Finally, the K blocks with the smallest entropy increase within Stage 2 are selected for pruning:
Sprune
{fe fa Rank (a')(AHl)... I K]}.
4

---

Published as a conference paper at ICLR 2026
Method
MQQA HellaSwag
WSC273
CSQA
WinoGrande
ARC-E
Dataset
ARC-C
OBQA
CMMLU
XSim
GSMIL
Average
0.503
0.6091
0.8864
0,5741
0.7388
0,7963
0,4898
0,3300
0,5908
0,3830
0,4086
0.0315
0.3715
0.5697
LaCo
Da501
0.2915
0,6996
0.1196
0,6255
0,2500
0.2568
0.2400
0.1817
0.2070
0.2746
0.0284
0.0543
0.3216
ShetGGT
0.7557
0.5158
0,8352
0,4908
0,7048
0,7104
0,4181
0.2620
0,1887
0,3596
0,3828
0.091
0,0591
0,4662
Ours (Layer)
0,7524
0,5467
0,8278
0.4865
0.7214
0,7079
0.4266
0.2700
0.4954
0.345K
0,3914
0.0445
0.0326
0
0.6051
0.5717
0.7285
0,4872
0,3878
0,3528
Ours (Arta)
0,8020
0.6062
0,7755
0,5725
0,7309
0.7984
0,4889
0,3380
0,5888
0.3819
0,4019
0.0571
0,3510
0.5568
LaCin
0.5582
0.3180
0.7035
0.3090
0.637
0,3914
0.3029
0.154M
0.2802
0.2602
D,3062
0.0055
0.007X
0.3298
ShonGPT
0.6627
0.3960
0.7143
0.184
0,6988
0,5025
0.3294
0.2100
0.5086
0.3259
0,3167
0,0078
0,0099
0,3749
Ours (Layer)
0.6627
0.3960
0.7143
0.5181
0,6998
0,5025
0.5294
0,2100
0.5086
0.3259
0,3167
0.0078
0.0099
0.5749
LLMMDrop
0,7998
0.5970
0,7788
0.5766
0.7364
0,7944
0,4753
0.3520
0.5917
0,3659
0,3952
0.0126
0.2815
0,5314
(Art))
0,8003
0.5991
0.8681
0.5763
0.7332
0,7050
0,4855
0.5240
0.5902
0,3752
0.3952
0.0165
0,3495
0.5443
LaCo
0.2937
0.1630
0.5825
0,2671
0.2020
0.2810
0,2214
0.2584
0,0054
0.3110
ShorGGP
0,5702
0.2795
0,6007
0.1971
0.5612
0.3367
0.2858
0.2080
0.2261
0.2126
0.2287
0.0083
0.0099
0.3014
Ours (Layer)
0.3415
0.2424
0.2969
0,2505
0.3074
0.0050
0.3359
M1.oor
0,7742
0.5614
0,3888
0.4054
0.7277
0.7483
0.4437
0.2821
0.5551
0.3145
0.3722
0.01
0.0826
0.5071
Ours (Atta)
0,7802
0,5749
0,8498
0.5446
0.7222
0,7546
0,4693
0.3080
0,5857
0.3636
0,3799
0,0147
0.0811
0.5255
LuCo 0.5777
0.2764
0.5665
0.2146
0.5367
0.3266
0.2509
0.1520
0.3637
0.1549
0.2641
0.0012
0.0025
0.2988
SherGPT
0,5403
0.2704
0,5421
0.1949
0.5501
0.3068
0,2619
0.1540
0.2367
0.2539
0.2488
0.0043
0.0059
0.2944
16
Ours (Layer)
0.5272
0,2760
0.5275
0.1900
0.5067
0.2955
0,2491
0.1720
0.2473
0.2509
0.2411
0.0032
0.0020
0.2914
L.MDDrop
0.6926
0.4272
0.7875
0.2121
0.7017
0.5640
0.3328
0.2220
0.2735
0.2819
0.2938
0.0099
0.0126
0.4000
Ours (Artn)
0.7514
0.4481
0.7656
0.3022
0.7048
0.6995
0.3925
0.2700
0.3586
0.2784
0,3282
0.0117
0.0511
0.4407
Table 1: Experiment Results on Mistral-7B--O.33 L indicates the number of pruned blocks.
where Sprnne denotes the set of pruned blocks and AH' represents the entropy increase of l com- putation block. The bottom i ranked layers are pruned to optimize efficiency. To estimate entropy efficiently, we explore multiple techniques:
Bucket-based Estimation: Discretize activation values into bins and estimate based on fre- quency distribution.
K-Nearest Neighbors (KNN): Computes entropy by estimating local density using KNN,
Renyi Entropy: A generalization of Shannon entropy that provides tunable parameter to control sensitivity to distribution variations.
Regardless of the estimation method used, entropy computation remains efficient. Our experimental results demonstrate that selecting an appropriate entropy estimation method is crucial for achieving optimal pruning performance. Among the approaches tested, Bucket-based estimation and KNN- based estimation were found be particularly effective.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Models We conduct experiments on two state-of-the-art decoder-only Transformer models: Llama3.1-8B and Mistral-7B-v0.3. To make fair comparison, all experiments are finished on single 40G A100 GPU device.
Benchmarks To evaluate the effectiveness of EntroDrop, we test on a diverse set of reason-
ing and comprehension benchmarks: Commonsense Reasoning: PIQA (Bisk et al. 2020). Hel-
laSwag (Zellers al. 2019). WSC273 (Sakaguchi et al., 2021), CSQA (Talmor al. (2019). Wino-
Grande (Sakaguchi et al. (222T) Scientific and Knowledge based QA: ARC-E (Clark et al. 2018),
ARC-C (Clark et al. 2018). OBQA (Mihaylov et al.
2018). General and Subject-specific Knowl edge: MMLU (Hendrycks et al. 202 CMMLU (Li et al.) 2024), RACE (Lai et al. 2017 In addition, we include two generation
benchmarks: XSum (Narayan et al.) 2018) for ab- stractive summarization and GSM8K (Cobbe et al.) 2021) for multi-step mathematical reasoning, allowing us to evaluate EntroDrop on both long-form and procedural generation tasks.
Baselines We compare EntroDrop against ofte-offthheart pruning techniques in two categories: (1) Layer Pruning Methods that directly prune the whole transformer block: LaCo (Yang et al., 2024) and ShortGPT (Men et al. 2024). (2) Attention Pruning Method that only prunes the attention block: LLMDrop (He et al.) 2024). These baselines allow us to assess how EntroDrop compares against existing pruning methods in terms of performance preservation under different pruning granularity.
5

---

Published as a conference paper at ICLR 2026
Method
HellaSwag
WSC273
CSQA
WinoGrande
ARC-E
Dataset
ARC-C
OBQA
CMMLU
XSim
GSMSL
Average
0.7998
0.6003
0,8008
0.7166
0.7316
0,8148
0,5102
0.3340
0.6332
0,5090
0,3923
0.1302
0,5011
0.5872
LoCo
0.7628
0,5166
0,8059
0.6806
0,710
0,7302
0,4462
0.2840
0,5949
0.4370
0.3761
0,1025
0.285
0.5002
ShetGGP
0.7557
0.5504
0.7949
0.6921
0,0017
0,4420
0.3120
0,5802
0,4160
0,3818
0.1078
0,242
0,5054
Ours (Layer)
0,7573
0,5407
0,8242
0,7027
0.7088
0,7504
0.4215
0.2860
0,6212
0,4918
0,3818
0.1215
0.0844
0.51
0.0025
0.5965
0.518
0,5955
Ours (Arta)
0,8003
0.6022
0.8498
0,7158
0.7364
0,8157
0.5179
0.3420
0.6238
0,5044
0,3894
0.1315
0,5057
0.5949
LaCin
0.6197
0.3098
0.6007
0,4005
0/0227
0,3552
0.2156
0.255l
0,4463
0,3405
D:2478
0.0280
00113
0,3007
ShonGPT
0.6045
0.2825
0.5971
0,4046
0.5422
0,4289
0.2739
0.1820
0.5226
0,3153
0,2526
0,0310
0,1100
0,3015
Ours (Layer)
0.6795
0,1384
0,7509
0.6216
0,8888
0,5641
0.3532
0.2120
0.5581
0,6008
0,3578
0.0837
0,0152
0,4533
LLMMDrop
0,7954
0.5877
0,888
0.7174
0.7443
08119
0.5068
0.3560
0.6338
0,5073
0,4010
0.1268
04575
0.5932
(Art))
0.7954
0.5921
0,8322
0.7183
0,7411
0,8186
0.5154
0.3540
0.6901
0,5041
0.3876
0.1263
0,4439
0,5000
LaCo
0.3312
0,6337
0.1966
0,219
0.2705
0.1980
0,2571
0,0014
0.3141
ShorGGP
0.6007
0,3066
0.5961
0.5160
0.5901
0,4007
0.2765
0.1780
0,3605
0.3252
0,2660
0.0083
0.0015
0.3346
Ours (Layer)
0.30166
0.5%61
0.5160
0.2765
0.2560
0.0015
0.3346
M1.oorrp
0.7867
0,3584
0,8608
0.6790
0.7253
0.7807
0,4753
0.5100
0.5992
0,4511
0.3799
0.1195
0.2002
0.5467
Ours (Atta)
0.7867
0.5584
0.3608
0.6790
0,7253
0,7807
0,4753
0.3100
0,5992
0.4511
0,3799
0,1195
0.2002
0.5467
LuCo 0.5854
0.2904
0.6447
0.1957
0.5612
0,3443
0.2338
0.1600
0.2295
0.2527
0.246/
0.0024
0.0000
0.3071
SherGPT
0.5647
0.2754
0,5421
0.1949
0.5501
0.3194
0.2440
0.1540
0.2295
0.2529
0.2488
0.0023
0.0000
0.2956
16
Ours (Layer)
0.5729
0.2705
0.5238
0.2113
0.5099
0,3165
0.2321
0.1380
0.2627
0.2538
0.2278
0.0145
0.0000
0.2980
1..MDrop 0.6926
0.4272
0.7875
0.2121
0.7017
0.5640
0.3328
0.2220
0.2735
0.2819
0.2938
0,0595
0.1020
0.4307
Ours (Artn)
0.7514
0.4481
0.7656
0,3022
0,7048
0.6695
0.3925
0.2700
0.3586
0.2784
0,3282
0.0936
0.1207
0.4663
Table 2: Experiment Results on Llama3.1-8B. L indicates the number of pruned blocks. The best
performance is marked in bold.
Ilama 3,1-88
Mistral-78.v0.3
1.0
0.6
0.6
0.4
0.2
0.2
Datasets
Datasets
Figure 4: Impact of Calibration Datasets.
4.2 OVERALL PERFORMANCE
Our experimental results on Llama3.1-88 (Table 2 and 7Bstral-7B-v0.3 (Table demonstrate the effectiveness of EntroDrop. We summarize the key findings follows:
EntroDrop is effective across multiple models. Our method consistently achieves the best performance across both L-ama3.1-8B and Mistral- This demonstrates that EntroDrop is generalizable pruning strategy applicable to different pre-trained LLMs.
EntroDrop outperforms both layer pruning and attention pruning baselines. Com- pared to LaCo and ShortGP (layer pruning) and LLMDrop (attention pruning), our method consistently achieves superior results. This suggests that our entropy-based metric effectively identifies and prunes redundant computation blocks at different granularities.
Pretrained Transformer models contain significant redundancy, especially in atten- tion layers. Our experiments show that removing up to 12 layers (37.5% of total attention layers) in Alama3.1-8P still retains over 95% of the model's original performance. This in- dicates that modern Transformers are often over- parameterized and that structured pruning can significantly improve efficiency without major performance degradation.
Overall, these findings confirm that entropy- pruning an effective and generalizable strategy for reducing redundant computation in large Transformer models. By leveraging entropy dynamics, EntroDrop enables efficient pruning while maintaining competitive performance.
4.3 IMPACT OF CALIBRATION DATASET

---

Published as a conference paper at ICLR 2026
Ulmm331-40
Mistral-78-
Bacat
Buckd
0.8
0.8
0.4
Datasels
Datarets
Figure 5: Impact of Entropy Estimate Methods.
Our pruning method relies calibration dataset to estimate entropy dynamics across Transformer layers. We investigate how differrnnt calibration datasets affect pruning results. Specifically, we evaluate two general- domain datasets (C4 (Raffel al., 2020) and Wiki- text (Merity et al.] (016)) and two specific-domain
Uama333 3-99
Desaeess
Danaress
25
20
22 20
Nederee
Las
datasets (Medicine (Cheng al. 2023 and Law (Cheng
et al. ).
20
202
mated using different calibration datasets on Llama3.1- Figure 3: Calibration Datasets Heatmap
Figure presents the entropy increase heatmaps esti-
8B and Mistral-7B-v0.3, Across all models, entropy increase is smaller in deeper layers, indicating that these layers contribute less to new information processing and are more redundant. This suggests that deeper layers are natural candidates for pruning. Furthermore, despite differences calibration datasets, the estimated entropy increase trends remain largely consistent. The relative importance of layers preserved across general and domain-speant specific datasets, suggesting that our entropy-based pruning approach is robust to calibration dataset variations.
To further examine the impact of calibration datasets on model performance, Figure 4 presents the evaluation results of Llama3. 1-BB and Mistral-7B-v0.3 after pruning 12 attention layers (37.5%), The results show that different calibration datasets lead to minimal differences performance across all benchmark datasets, reinforcing the robustness of our entropy-basee -based pruning strategy. Notably, even with domain-speciff datasets (Medicine, Law), the average accuracy remains stable, indicat- ing that the entropy estimation process generalizes well across different calibration datasets. These findings confirm that EntroDrop remains effective regardless of the calibration dataset, making it flexible and generalizable pruning strategy.
4.4
ENTROPY ESTIMATION SENSITIVITY
HellsSwag
WinoGrande
Arcurac
0.4
0.5
Bucket
Accurscy
0.6
Bucket
Accursc)
3.7
0.5
Cosine
Reny
0.4
0,3
10
Llama3.1 0B Attention
Uama3.1 Attention
Llama3.1 Attention
Liama3.1
ARC-C
HellaSwng
MNLU
WinaGrande
2..40
Figure 6: Attention Deletion Experiments
Estimation Method. Entropy estimation plays a crucial role in our pruming framework, as it di- rectly influences the selection of redundant computation blocks. We evaluate three entropy esti- mation methods: Bucket, KNN and Renyi. To analyze the impact different entropy estimation methods, compare pruning results using these approaches on Llama3. 1-8B and Mistral-7B-v0.3.

---

Published as a conference paper at ICLR 2026
Figure presents the evaluation results across multiple benchmark datasets when deleting 12 lay- ers of attention blocks using our method. The results indicate that the choice of entropy estimation method significantly affects performance. Both Bucket-based and KNN-based estimation methods yield stable and high accuracy across all datasets, demonstrating their effectiveness in preserving es- sential model capabilities after pruning. In contrast, Renyi entropy estimation consistently underper- forms, leading to noticeable accuracy degradation. This suggests that Renyi entropy may introduce excessive sensitivity certain probability distributions, making less suitable for pruning decisions of pre-trained Transformer blocks.
To investigate the redundancy in Transformer models, we analyze the impact of attention layer deletion across multiple datasets, including ARC-C, HellaSwag, MMLU, and WinoGrande. Fig- ure presents the performance degradation trend on MMLJ as attention layers are progressively removed from Mistral-7B-v0.3. The results indicate that model performance remains stable until approximately 12 attention layers are removed, after which accuracy begins to degrade. This sug- gests that significant portion of attention layers redundant and can be pruned without substantial performance loss. Additionally, we compare different importance estimation methods for attention pruning. Both Bucket and KNN-based estimation methods consistently outperform Cosine Similar- ity, demonstrating their effectiveness in identifying unimportant attention layers. In contrast, Renyi entropy performs poorly, further confirming its limitations in guiding structured pruning.
Estimation Hapee-par To further ana- lyze the robustness of entropy estimation meth- ods, we investigate the impact of different hy- perparameter settings. We tune the following parameters: Bucket-based Estimation: Number of bins selected from {20, 40, 80, 160). KNN- based Estimation: Number of nearest neigh- bors selected from (25, 50, 75, 100). Figure 7 presents the estimated entropy values across Transformer layers using different hyperparam- eter settings. The results indicate that while different entropy estimation methods (Bucket KNN) yield significantly different absolute entropy values, the relative importance rank- ing of layers remains largely unchanged within same estimation method. This suggests that the choice of hyperparameter (c.g.. number of bin- s/neighbors) does not significantly impact the identification of redundant layers.
Llama3.1-8B
25
30
Estimation
Succut 80
Buceet 40-
KNN 25
KNN 75
KNN 100
Layers
Mistral-7 B-v0.3
Mistral-7B-v0.3
10
15
20
25
30
Estimation
Bucket 160
Succet 80
Bucant 20-
KNNN
KNN 75
Layers
100
200
300
400
500
Figure 7: Hyperparameter Impact
The findings highlight the importance of selecting entropy estimation method while also reinforc- ing the stability of entropy- based pruning. Although different methods may compute varying abso- lute entropy values, the pruning decisions remain consistent. From our experiments, Bucket-based and KNN-based methods provide reliable performance, whereas using an inappropriate method like Renyi entropy could lead to suboptimal pruning outcomes.
4.5 SPEEDUP TEST
To evaluate the efficiency gains from pruning attention blocks, we conduct inference speed tests on Llama3.1-88 and Mistral-7B--0.33 We prune attention layers progressively and measure both model performance and inference time. The speed test performed by fixing the input sequence length to 1024 tokens and generating an output of 1024 tokens. Each experiment is repeated 10 times, and the average inference time reported.
shown in Fig. inference time decreases nearly linearly with the number of pruned blocks, while accuracy remains stable for the first 12 pruned layers. Beyond this point, further pruning leads to no- ticeable degradation. Bucket- and KNN-based EntroDrop consistently outperform cosine-similarii pruming. offering better accuracy sccuracy-speed trade-offs. Since EntroDrop removes entire attention blocks, the observed latency gains align closely with the theoretical FLOPs reduction (e.g., pruning 12 of 32 blocks leads to about 37.5% FLOPs reduction). These results demonstrate that EntroDrop yields substantial acceleration while preserving strong task performance.
8

---

Published as a conference paper at ICLR 2026
Model Parforance (Average)
Llama3.1-88
Model Performance (Average)
Mistral-7B-vOl
0.6
Inference Time
37.5
0.6
Inference Time
Inference Time (s)
0.5
35.0
Inference Timee
0.5
32.5
KNN
0.4
KNH
Reryi Cosine
30.0
Renyl
27.5
Cosine
0.3
5
10
15
20
25
10
15
20
25
Number of Dropped Layers
Number of Drapped Layers
Figure 8: SpeedUp Experiments
RELATED WORK
LLM Pruning, In the era of large language models (LLMs), various methods have been proposed
to reduce model size and accelerate inference (Frantar et al.) 2022] Lin al. [024] Xiao et al.
2023 [Shao et al. 2023 Zhu et al. 2023; Xu et al.) 2023] Dettmers et al. 2023 Liu et al.) [023] Huang et al. 2025 Shen et al. 2025). Recent advances focus on post-training pruning techniques that eliminate redundant parameters structures. (Gromov et al. 2024) empirically study the di- minishing contribution of deeper layers in large Transformers and highlight the potential for layer pruning. While their work focuses on diagnosing redundancy, our method operationalizes this obser- vation by quantifying information change through entropy and proposing a concrete, entropy-guided pruning strategy. SparseGPT (Frantar & Alistarh (023) leverages second-order information to iden- tify unimportant parameters in LLMs. Wanda (Sun et al., 2023) introduces a pruning matrix that considers both weight magnitude and corresponding input activations. NEPENTHE (Liao et al., 2024) introduces a method that utilizes entropy to identify and remove low-entropy layers in deep neural networks, effectively reducing model depth while maintaining performance. E-Sparse (Li) et al., (023) introduces an
pruning method that enhances inference speed and reduces memory usage large language models by leveraging information richness to guide N:M sparsity. SPP (Lu et al. 2024b) designs an efficient fine-tuning method to recover model performance post- pruning while maintaining sparsity. Beyond parameter pruning, structural pruning of LLMs has also gained popularity. LLM-Pruner (Ma al., 2023) and SaaaredLLaMA (Xia et al., 2023 remove unimportant structures such as layers and attention heads. Additionally, (Lu et al. (2024)) finds that certain experts in mixture-of-expee offexperts (MoE) LLMs can also be pruned. Among structural pruning methods, layer pruning is particularly relevant. Laco (Yang et al.) 2024) reduces model depth by merging adjacent layers from the topmost layer downward. ShortGPT (Men et al. 2024) prunes unimportant layers based on a cosine similarity criterion. L.LMDrop (He et al. (2024) finds that at- tention layers are more redundant than MLP layers but also relies on cosine similarity for pruning. Yang et al. (Yang et al.) also adopt an information-thcorete lens, using transfer entropy to measure how masking block changes the model's output distribution. Their method evaluates output-level sensitivity via additional masked forward passes. In contrast, EntroDrop focuses on hidden-state entropy dynamics, requiring only single unmasked forward pass and revealing a consistent com- pression-expansion pattern across layers. This representation-level perspective leads to a distinct and more efficient pruning criterion.
Different from prior approaches, firstly analyze layer-wise entropy dynamics in LLMs and reveal a two-stage pattern of information compression followed by expansion. Based on this insight, we propose EntroDrop, which prunes blocks with the smallest entropy increase between layers.
6 CONCLUSION
In this paper, we present the first systematic study of layer-wise entropy dynamics in pretrained large language models. Our analysis reveals two-stage pattern characterized by early-layer information compression followed by late-layer information enrichment. Building on this observation, we de- sign EntroDrop. entropy-based pruning framework that removes blocks contributing the least additional information. Extensive experiments show that EntroDrop achieves substantial parameter reduction and inference acceleration while maintaining high accuracy across multiple benchmarks.

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint WwXXxx23333087770 2023.
Riccardo Ali, Francesco Caso, Christopher Irwin, and Pietro Liò. Entropy-lens: The information signature of transformer computations. arXiv preprint 1rXiv:5502.16500 2025.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. Thirty- AAAI Conference on Artificial Intelligence, 2020.
Xiaodong Chen, Yuxuan Hu, Jing Zhang, Yanling Wang, Cuiping Li, and Hong Chen. Streamlining redundant layers to compress large language models. arXiv preprint arXiv:2403. 19135, 2024.
Daixuan Cheng, Shaohan Huang, and Furu Wei. Adapting large language models via reading com- prehension. In The Twelfth International Conference on Learning Representations, 2023.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803 000580000.055....0 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Refichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2 10, 14168, 2021.
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Salch Ashk- boos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantize repre- sentation for near-lossless IIm weight compression. arXiv preprint FrXxi:3306003075 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint 2XXv:2407.21788 2024.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp. (0323-10337. PMLR, 2023.
Elias Frantar, Salch Ashkboos. Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.1732. 2022.
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel Roberts. The
unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403. /887,, /78877 2024.
Shwai He, Guoheng Sun, Zheyu Shen, and Ang Li. What matters in transformers? not all attention is needed. arXiv preprint arXiv:2406.15788 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Stein- hardt. Aligning with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021b.
Weiyu Huang, Yuezhou Hu, Guohao Jian, Jun Zhu, and Jianfei Chen. Pruning large language models with semi-structural adaptive sparse training. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 24167-24175, 2025.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 2023.
10

---

Published as a conference paper at ICLR 2026
Bo-Kyeong Kim, Geonmin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung-Kyi Kyu Song. Shortened llama: simple depth pruning for large language models. arXiv preprint 2022:22402.02844,, 11, 2024.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 785-794, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-10022 URL
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Tim- othy Baldwin. CMMLU: measuring massive multitask language understanding in chinese. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Com- putational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 11260-1 1 285, Association for Computational Linguistics, 2024. doi: 10.18653/V1/2022 FINDINGS-ACL.671. URL httpss//ddoi.org//0.. 1ttp//10001110111 18653/v hnnn://////...nnnnnaaa.......
Yun Li, Lin Niu, Xipeng Zhang, Kai Liu, Jianchen Zhu, and Zhanhui Kang. E-sparse: Boost- ing the large language model inference through entropy-based n: sparsity. arXiv preprint arXiv:2310. 1.529,, 2023.
Yueqing Liang, Liangwei Yang, Chen Wang, Xiongxiao Xu, Philip Yu, and Kai Shu. Taxonomy- guided zero-shot recommendations with llms. In Proceedings of the 3 3tt International Conference on Computational Linguistics, pp. 1520-1530, 2025.
Zhu Liao, Victor Quétu, Van-Tam Nguyen, and Enzo Tartaglione. Nepenthe: Entropy-based pruning as a neural network depth's 's reducer. arXiv preprint arXiv:2404. 16890, 2024.
Lin, Jiaming Tang, Haotian Tang, Shang Yang. Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6: 87-100, 2024.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305.17888 2023.
Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. Not all experts are equal: Efficient expert pruning and skipping for mixture-of-expere large language models. arXiv preprint arXiv:2402.1 14800, 2024a.
Xudong Lu, Aojun Zhou, Yuhui Xu, Renrui Zhang, Peng Gao, and Hongsheng Li. Spp: Sparsity-preserved parameter-effiiiet fine-tuning for large language models. arXiv preprint 10055 2024b.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702-21720 2023.
Yu Mao, Weilan Wang, Hongchao Du, Nan Guan, and Chun Jason Xue. On the compressibility of quantized large language models. arXiv preprint arXiv:2403.01384, 2024.
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.0388.0 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint 16X9v:1609007843 2016.

---

Published as a conference paper at ICLR 2026
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? A new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pp. 2381-2391 Association for Computational Linguistics, 2018. doi: 1118833////DDL 1/D18-1260. URL https://doi.org/ 1653/v 1
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. ArXiv, abs/1808 008.08745, 2018.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang. Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-tum program synthesis. arXiv preprini arXiv:2203. 1374, 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-61 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications the ACM, 64(9):99-106, 2021.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308.13137 2023.
Xuan Shen, Zhao Song. Yufa Zhou, Bo Chen, Jing Liu, Ruiyi Zhang, Ryan Rossi, Hao Tan, Tong Yu, Xiang Chen, et al. Numerical pruning for efficient autoregressive models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 20418-20426, 2025.
Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, and Jae-Joon Kim. Sleb: Streamlining ilms through redundancy verification and elimination transformer blocks. arXiv preprint arXiv:2402. 09025, 2024.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.1 1695, 2023.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseeQ:: A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pp. 4149-4158, Minncapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/N19-11 1421.. URI hutps://acanmttoogg..og/////
Gemini Team, Petko Georgiev. Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.05530 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023,,
A Waswani, N Shazeer, N Parmar, J Uszkoreit, L Jones, A Gomez, L Kaiser, and Polosukhin. Attention is all you need. NIPS, 2017.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared Ilama: Accelerating language model pre-training via structured pruning. arXiv preprint urXiv:2310.066// 2023.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference Machine Learning, pp. 38087-38099. PMI.R, 2023.
12

---

Published as a conference paper at ICLR 2026
Rumxin Xu, Fuli Luo, Chengyu Wang, Baobao Chang, Jun Huang. Songfang Huang, and Fei Huang. From dense to sparse: Contrastive pruning for better pre-trained language model compression. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 11547-11555, 2022.
Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xi- aopeng Zhang, and Qi Tian. Qa-lora: Quantization aware low-rank adaptation of large language models. arXiv preprint vxXi:2239..4717, 2023.
Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael Ryoo, et al. xgen-mm (blip-3); A family open large multimodal models. arXiv preprint 240.::2408.0887. 2024.
Liangwei Yang, Jing Ma, Jianguo Zhang. Zhiwei Liu, Jielin Qiu, Shirley Kokane, Shiyu Wang, Haolin Chen, Rithesh Murthy, Ming Zhu, et al. Geognn: Quantifying and mitigating semantic drift in text-attributed graphs. arXiv preprint arXiv:2511.0900 1.0942,, 2025.
Mingzhe Yang, Sihao Lin, Changlin Li, and Xiaojun Chang. Let Ilm tell what prune and how much to prune. In Forty-second International Conference on Machine Learning.
Yifei Yang. Zouying Cao, and Hai Zhao. LaCo: Large language model pruning via layer collapse. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 6401-6417, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: /t0.6//////222...nnnnggee/////// URL https:/acaantooooy.oo//224444444 (0nng///...nddnnnneemm3337717
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, 2019.
Weizhi Zhang. Liangwei Yang, Wooseong Yang, Henry Peng Zou, Yuqing Liu, Ke Xu, Sourav Me- dya, and Philip Yu. Llminit: A free lunch from large language models for selective initialization of recommendation. arXiv preprint ar XrXiv:2503.01814, 2025.
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. survey on model compression for large language models. arXiv preprint arXiv:2308 2xXv:23008.06333, 2023.
Yuanjie Zhu, Liangwei Yang. Ke Xu, Weizhi Zhang, Zihe Song, Jindong Wang, and Philip S Yu. Llm-memcluster: Empowering large language models with dynamic memory for text clustering. arXiv preprint arXiv:2511.15424 2025.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 THE USE OF LARGE LANGUAGE MODELS
We used large language models (LLMs) solely for grammar correction and language polishing of the manuscript. No part of the research ideation, experimental design, implementation, or analysis relied on LLMs. All methodological contributions and results were produced entirely by the authors.
A.2
ENTROPY DYNAMICS DIFFERENT MODEL.S
Owen3 148
Owen3 148
Qwen3 148
4.4
4.40
4.35
Entropy
4.3
Entropy
4.2
C4
4.2
C4
C4
Law
Law
Medicine
Medicine
4.10
Medicine
Wikitest2
Wikitxt2
4.05
Wikitext2
4.00
50
Block index
MLP Block Index
Attention Block Index
DeepSeek-V2-Liteee
DeepSeek-V2-Lae -iteekk------------
DeepSeek-V2-Lites
3.5
3.5
C4
3.0
Law
3.0
Law
2.5
Wikitext2
Medicine
Medicine
Wikitext2
2.5
Wikitext2
Medicine
10
20
30
40
50
2
10
15
20
25
0.2
10
15
20
25
Block Index
MLP Block Index
Altention Block Index
Figure 9: Entropy dynamics among layers during inference
To assess whether the observed entropy dynamics generalize beyond the models reported in the main text, we further evaluate two widely used open-source LLMs: Qwen3-14B and DeepSeek-V V2- Lite-16B, As shown in Fig. / both models exhibit a similar two-stage trend: an initial entropy de- crease in the first few layers, followed by gradual increase across the remaining depth. This pattern appears consistently across all four datasets. These results indicate that the compression-expie expaansion transition is not specific to Llama- or Mistral-style architectures. While we also tested several ad- ditional models (not all shown), and the phenomenon not universal, it is prominently present in many of the most commonly used open-source LLMs (Llama3, Mistral, Qwen, DeepSeek). This suggests that the entropy dynamics we report reflect a broadly shared behavior across mainstream Transformer models rather than artifact of a single architecture.
Overall, the additional experiments reinforce our key conclusion: early layers tend to compress and stabilize representations, whereas later layers expand contextual information, making them more suitable for depth- based pruning.
A.3 PRUNING EXPERIMENT RESULTS ON QWEN3-144 14B MODEL
As shown in Table 3 pruning Qwen3-14B 14B under different budgets reveals clear differences among baseline methods. LaCo and ShortGPT degrade rapidly as L increases, confirming that distance- and gradient-free heuristics are insufficient for identifying compressible layers in larger modern LLMs. LLMDrop (Attention) is consistently stronger, especially under higher pruning ratios. However, our method achieves the best performance across all pruning depths, with notable gains on challenging tasks such as WSC273, MMLU, GSM8k, and XSum. These results indicate that entropy entropy-guided guided layer selection preserves essential reasoning and knowledge pathways more effectively than existing heuristics, leading to more robust model compression on Qwen-series architectures.
A.4 POST-TRAINING EXPERIMENT
To further evaluate whether entropy-guides pruming preserves recoverable structure more effectively than cosine-simin similarity pruning, conduct lightweight post-training experiments the HellaSwar wag training split. For both Llama3.1-8B and Mistral-7B--0..3 we prune the models with three budgets (T. (4, 12}) and apply short finetuning stage using only the HellaSwag training data. Since
14

---

Published as a conference paper at ICLR 2026
Method
Average
PIQA
HellaSwag
WSC:273
CSQA
WinoCirndd ARC-E
ARC-C
ORQA
MMLJU
CMMLU
RACE
XSam
GIMS
0.8014
0.6090
0.8498
0,8018
0.7309
0.8122
0,5845
0.3520
0.7714
0,8019
0.4335
0.1196
0.8802
0.6599
LaCo
0.7220
0.1872
0,7875
0.6732
0.6985
0.7189
0.4573
0.2760
0.7564
0.7179
0.3770
0.0961
0.2163
0,5374
ShomGaPT
0.7334
0.5309
0,7912
0,7830
0.6819
0.7319
0.4855
0.2740
0.7587
0,7364
0.4048
0.1145
0.3397
0.5666
(Layer)
0.7704
0.5370
0.6559
0.4644
0.3180
0,6299
0,6966
0:4077
0.1196
0.5648
0.5750
L.MDioop (Atin)
0.7549
0.0066
0.8315
0.7854
0,7182
0.8262
0.5759
0.3560
0.7644
0.7806
0.4316
0.1194
0.6027
0.6303
Ours (Amm)
0.7565
0.6022
0.8498
0.7879
0.7135
0,833
0,5742
0.3400
0.7664
0.7884
0:4354
0.1171
0.8446
0.6503
LaCo
0.7367
0.4622
0,7020
0.4283
0.5983
0.7231
0.4053
0.2760
0.3583
0.3976
0.3550
0.1227
0.0238
0.4806
ShomtaPT
0.6097
0.4188
0.7766
0.6626
0,6740
0,5219
0.3507
0.2300
0.7302
0.7217
0.3167
0,0308
0.0000
0.4698
(Layer)
0,6872
0,440
0,7179
0.6847
0,6306
0.6646
0,4019
0,2600
0.7131
0,3043
0.1036
0.0341
0.4946
LLMDrop (Aitn)
0.7775
0.5771
0,7985
0.7396
0.7245
0.8039
0.5538
0.3320
0.7598
0.6735
0.1067
0.1028
0.1054
0.565K
Ours (Altn)
0.7862
0.5717
0.8132
0,7186
0,7061
0,8308
0,5717
0,3180
0.7168
0,7080
0.4351
0.1161
0.6133
0.6128
LaCo
0.5604
0.2930
0.6374
0.2498
0.5841
0.3624
0.2790
0.1840
0.3126
0.2546
0,2555
0.0162
0.0000
0,306K
ShurGPT
0.6627
0.3866
0,6410
0.1957
0.5406
0.9972
0,2978
0,2200
0.2490
0.2646
0.2986
0.0898
0.0190
0,3433
12
Ours (Layer)
0.6161
0.3706
0.6007
0,4349
0.5635
0.5311
0,3106
0,2300
0,5755
0.5343
0.3167
0,0198
0,0008
0,3973
LLMDrop (Ailn)
0,2637
0,7018
0.5862
0,2833
0.0534
Ours (Alta)
0.7709
0.5090
0,7729
0.5807
0,6527
0,7567
0:4518
0.3260
0.5823
0.5875
0,3895
0.0909
0.0197
0.4995
Table 3: Experiment Results Qwen3-14B. L indicates the number of pruned blocks. The best performance (per block group) is marked bold.
Entropy (Ours)
Recovery
Cosine
Recovery
Model
L
Base
Pruned
Finetuned
(%)
Pruned
Finetuned
(%)
4
0.6003
0.5947
0.6002
98.8%
0.5965
0.5953
30.0%
1lama3.1-8B
8
0.6003
0.5921
0.5937
79.4%
0.5877
0,5897
52.6%
12
0.6003
0.5708
0.5836
42.5%
0.5584
0.5594
4.6%
0.6091
0.6062
0.6080
74.5%
0.6051
0.6080
75.6%
Mistral-7B-v0.3
8
0.6091
0.5991
0.6045
46.3%
0.5970
0.6036
62.4%
12
0.6091
0.5749
0,5857
37.7%
0,5614
0,5726
30.6%
Table 4: Post-training recovery on HellaSwag for antropy-based pruning (ours) vs cosine similarity pruning. Base accuracy is shared across pruning methods.
all pruning methods begin from the same dense checkpoint, we report shared baseline accuracy for each model and compare how much performance can be recovered after pruning. All results are evaluated on the HellaSwag test set, enabling direct and controlled comparison between entropy- based and cosine-based criteria.
Results. Across both models, entropy- guided pruning shows markedly stronger recoverability than cosine similarity. For Llama3. EntroDrop preserves most of the recoverable structure: pruning layers leads to minimal degradation (0. (0.6003-<0) +0.0344), and post-training restores 98.8 % of the original accuracy, compared cosine pruning which restores noticeably less. Even larger budgets (L (L=88, 12), EntrooDrop-pruned models consistently regain a substantial portion of the lost accuracy (79.4% and 42.5% recovery), while pesune-pruned models exhibit weaker and more unstable recov- ery. Mistral-7B--0.33 demonstrates the same pattern. Under EntroDrop, the model recovers 74.5% of the gap L 4, and still restores 46.3% and 37.7% L = 8 and L 12, respectively-agant exceeding the recovery obtained by cosine pruning. These results suggest that entropy reliably iden- tifies layers whose removal can be compensated through small-scale post- post-training, indicating that entropy-based pruning preserves the model's intrinsic plasticity more effectively than cosine simi- larity, especially under deeper pruning budgets.
To evaluate whether post-training can recover the general capabilities of structurally pruned models, we design a controlled study that jointly varies the pruning criterion and the semantic relevance of post-training data. Experiments are conducted on two representative base models (Llama-3.1- 8B and Mistral-7B), each pruned with a fixed 12-layer budget. To study capability recovery. we finetune each pruned model on 100k randomly sampled tuning examples drawn from datasets with progressively increasing relevance to the evaluation tasks: (a) Alpaca (general instruction tuning, least related), (b) Flan v2 (broad mixture of reasoning and knowledge tasks, moderately related),
15

---

Published as a conference paper at ICLR 2026
Table 5: Ero vs Cosp post-training results on 7 key benchmarks (Llama-3.1, 12-layer
pruning).
Orig
Pruned
Alpaca
Flan v2
Mixed
Task
Base
EropoDrop
Cosine
EntroDrop
Cosine
ErtroDrop
Cosine
EntroDrop
Cosine
HellaSwag
0,6003
0,5708
0,5584
0.5778
0,5643
0.5741
0,5599
0.5731
0,5588
WinoGrande
0.7316
0.7340
0.7253
0.7253
0.7261
0.7293
0.7324
0.7301
0.7245
RACE
0.3923
0.3703
0.3799
0_3837
0.3732
0.3780
0.3713
0.3770
0.3789
CSQA
0.7166
0.6740
0.6790
0.6388
0.6495
0.6978
0.6658
0.6978
0.6609
ARC-E
0.8148
0.7980
0,7807
0.7997
0.7824
0.7950
0.7811
0.7988
0.7824
ARC-C
0.5102
0.4957
0.4753
0.4940
0.4855
0.4889
0,4821
0.5000
0.4838
OBQA
0,3340
0,3540
0,3100
0,3460
0,3140
0.3500
0,3080
0.3540
0,3100
Avg (7 tasks)
0,5857
0,5710
0,5584
0.5665
0,5564
0,5733
0.5572
0.5758
0,5570
Table 6: EntroDrop vs Cosine pruned model post-training experiment across datasets (Mistral-7B,
12-layer pruning. 7-task subset).
Orig
Pruned
Alpaca
Flan v2
Mixed
Task
Base
EropoDrop
Cosine
EntroDrop
Cosine
EntroDrop
Cosine
EntroDrop
Cosine
HellaSwag
0.6091
0.5749
0.5614
0,5815
0.5686
0.5780
0.5645
0.5796
0.5678
WinoGrande
0.7388
0.7222
0.7277
0.7174
0.7253
0.7324
0.7309
0.7316
0,7411
RACE
0,4086
0.3799
0,3722
0.3856
0.3780
0,3799
0.3828
0.3732
0,3789
CSQA
0.5741
0,5446
0,4054
0.5127
0.3841
0.5152
0,3784
0,5651
0.4472
ARC-E
0.7963
0.7546
0.7483
0.7614
0,7593
0.7374
0.7260
0.7626
0.7500
ARC-C
0.4899
0,4693
0.4437
0.4556
0.4369
0.4556
0.4420
0.4599
0.4437
OBQA
0.3300
0,3080
0,2820
0.3060
0.2820
0,3240
0.2880
0.3020
0,2800
Avg tasks)
0.5638
0,5502
0,5083
0.5315
0.5049
0.5310
0.5124
0.5380
0,5184
and (c) Mixed (most related), constructed by sampling from the training data of the evaluation benchmarks (HellaSwag, WinoGrande, RACE, CSQA, ARC, and OBQA). This yields controlled semantic gradient (Alpaca + Flan v2 + Mixed) that allows us to study how recovery improves as post-training data becomes more aligned with the evaluation tasks. For each (criterion, dataset) configuration, we report per-task results, macro-average accuracy across all 11 tasks, and direct comparison between EntropyDrop and Crop under matched post-training conditions, using the Orig model as the common upper bound.
Across both 1lama-3.1 (Table 5p and Mistral-7B (Table the 7-task evaluation reveals a consistent pattern in post-pruning capability recovery. First, EntropyDrop consistently outperforms cosine- based pruning across all post-training settings, including the no-training case (PRUNED). This indi- cates that entropy-based saliency preserves functionally essential layers more effectively than repre- sentational similarity measures. Second, the extent recovery strongly depends on the semantic relevance of post-training data. While Alpaca provides limited gains, Flan v2 yields moderate improvements, and the Mixed dataset- dataset-samples sampled from the training distributions tasks producess the largest recovery. This alignment-driven trend holds for both pruning criteria and model architectures. bosine-based pruning, however, exhibits a larger irrecoverable gap even under Mixed training. Third, lightweight post-training (100k examples) provides substantial but diignment--eeendenn recovery. When the finetuning data are semantically related to the evaluation tasks, performance improves markedly; however, unrelated datasets can introduce negative transfer and further degrade accuracy. Even under the aest-aligned Mixed dataset, cuningg-nduued loss is only partially recoverable, indicating that post-training is highly effective but cannot fully compen- sate for information discarded by pruning.
the evaluation
We additionally evaluate post-training recovery under 8-layer Transformer pruning on Llama-3.1 8B. Using the same setup in the attention study, we finetune each pruned model with 100k examples from three datasets with increasing semantic relevance (Alpaca + Flan v2 - Mixed). This allows to examine how recovery depends on data alignment when entire Transformer layers are removed. Experiment results are shown in Table 7 Across all settings, EntropyDrop again outperforms CosineDrop- botth immediately after pruning and after post-training confirming the stability of catrooy-ggideer layer selection. Recovery improves steadily with data relevance, but the pruning loss remains only partially recoverable, especially for cosine-based saliency.
16

---

Published as a conference paper at ICLR 2026
Table 7: EropoDrop vs Cosine post-training results on 7 key benchmarks (Llama-3.1, 8-layer prun-
ing).
Orig
Pruned (8L)
Alpaca
Flan v2
Mixed
Task
Base
ErtroDrop
Cosine
EntroDrop
Cosine
EntroDrop
Cosine
EntroDrop
Cusine
HellaSwag
0.6003
0.4384
0.2825
0.4420
0.2903
0.4409
0.2866
0.4417
0.2922
WinoGrande
0.7316
0,6898
0,5422
0.6898
0,5414
0,7009
0.5430
0.6993
0,5446
RACE
0.3923
0.3378
0,2526
0.3445
0.2517
0.3349
0.2517
0.3426
0.2545
CSQA
0.7166
0.6216
0.4046
0.4963
0,4128
0.5872
0.4087
0.5512
0.4095
ARC-E
0.8148
0,5644
0.4289
0.5968
0,4411
0.5981
0.4297
0.5985
0.4335
ARC-C
0.5102
0,3532
0.2739
0.3592
0.2807
0.3575
0.2816
0.3490
0.2756
OBQA
0.3340
0.2120
0.1820
0.2640
0.1800
0,2640
0.1780
0.2600
0.1840
Avg tasks)
0.5857
0.4596
0.3381
0.4561
0.3426
0.4691
0.3399
0.4632
0.3420
17


Reviews:
[{"review_id": "CWyjJc7R5B", "reviewer": "Reviewer_vwSb", "summary": "This paper proposes EntroDrop, an entropy-based pruning method for large language models. It computes the Shannon entropy of each layer’s hidden representations and measures the entropy increase between consecutive layers to estimate information contribution. Layers with smaller entropy gains are considered redundant and pruned. Experiments on Llama-3.1-8B and Mistral-7B-v0.3 show that EntroDrop can remove about 37.5% of layers while keeping around 95% of the original performance and greatly improving inference efficiency.", "questions": "see above", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces an entropy-increase–based pruning metric that more directly reflects information changes across layers, offering a more principled alternative to cosine similarity.\n2. It identifies a clear “entropy-decrease–then–increase” pattern in Transformer representations during inference, providing new empirical insight into information flow in LLMs.\n3. The method performs consistently well across Llama-3.1-8B and Mistral-7B-v0.3, showing robustness to different calibration datasets.", "weaknesses": "1. The connection between entropy and information flow is mainly empirical, lacking a rigorous theoretical justification from an information-theoretic perspective.\n2. Experiments are limited to two models. More experiments on larger models and different model sizes can be included.\n3. The paper should include some post-training experiments to verify whether the performance of the pruned model can be recovered. Although the pruned model outperforms the baselines, there is still a gap compared to the dense model, which post-training could help address.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b96TJEUsWC", "reviewer": "Reviewer_AtZA", "summary": "This paper proposes EntroDrop, an entropy-based block pruning strategy for Transformer language models. By monitoring the block-wise entropy variation during inference, the paper empirically identifies two distinct stages: the entropy of early layers gradually decreases, while that of later layers rises again. EntroDrop interprets this late-stage entropy increase as redundancy, and therefore prunes the k blocks with the smallest entropy change as a criterion. Experiments based on Llama3.1-8B and Mistral-7B-v0.3 on reasoning and comprehension benchmarks demonstrate that, under the same pruning ratio, EntroDrop achieves higher accuracy than baseline methods.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper provides new insights into the information flow within large language models (LLMs) through an empirical analysis of the entropy dynamics across Transformer blocks.\n2. The authors propose a simple yet effective entropy-based pruning method, shifting from cosine similarity that captures geometric relationships to an entropy criterion that better reflects information content.", "weaknesses": "1. The experiments in this paper primarily focus on multi-choice question answering. However, previous studies (e.g., LLM-Streamline [1]) have shown that similar pruning methods may cause significant performance degradation in generation tasks. It remains unverified whether EntroDrop suffers from the same issue. For instance, how does it perform on generative benchmarks such as XSum or GSM8K?\n2. EntroDrop is built upon the empirical observation of entropy dynamics, where entropy first decreases and then increases across layers. However, it is questionable whether such a pattern universally holds across different model families. Entropy-Lens [2], which also analyzes models using entropy, finds that entropy dynamics vary significantly among different architectures.\n3. One of EntroDrop’s main contributions is the use of entropy as a criterion to evaluate block importance. However, Yang et al. [3] also leveraged transfer entropy to assess the importance of blocks for pruning. Clarifying the similarities and differences between these two approaches would help better highlight EntroDrop’s unique contributions.\n\n[1] Compressing Large Language Models by Streamlining the Unimportant Layer. https://arxiv.org/abs/2403.19135\n\n[2] Entropy-lens: The information signature of transformer computations. arXiv preprint arXiv:2502.16570.\n\n[3] Let LLM Tell What to Prune and How Much to Prune. In Forty-second International Conference on Machine Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rtZ2cMWx5p", "reviewer": "Reviewer_Z5K4", "summary": "This paper focuses on pruning for large language models for better model efficiency. Authors analized the LLM models and found the entropy dynamics across different layers are significantly different from different layers, which provide sound basis for the introduction of using entropy dynamics for model pruning. Authors have included this in the model design and compared with other pruning methods on multiple datasets, where the introduced method is showing very promising results for both Llama3.1-8B and Mistral-7B-v0.3 for both compresson on both whole transformer block and only on attention. Authors also show very detailed ablation studies.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "+ Authors have included a complete analysis and solution pipeline for finding the possible reduction for compression and then introduce the solution to the pipeline.\n\n+ Authors have conducted extensive experiments on mulitple datasets for benchmark as well as providing comparison for two different LLM models, which provide the generalizability of the proposed compression method.\n\n+ Authors have provided comprehensive analysis for the sensitive analysis for different datasets on different model, which provide a pretty comprehensive analysis for the overall results.", "weaknesses": "- I do not have a big concern, one of the small intersting fact - the proposed model seems to have different performance behavior for Llama3.1-8B and Mistral-7B-v0.3, where Mistral-7B-v0.3 is showing better performance when the number for the pruned blocks is limited, while it is not observed in Llama3.1-8B. This could be some internal model difference. It would be interesting if authors can apply this to some more models for further analysis across different models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s8F2oOjN73", "reviewer": "Reviewer_Mwbs", "summary": "This paper proposes EntroDrop, an entropy-based pruning method for large language models (LLMs) that identifies and removes redundant computational blocks (e.g., Attention or MLP blocks) within Transformer architectures. The authors first analyze the entropy dynamics of hidden representations across model layers, observing a consistent two-stage pattern: entropy decreases in early layers (information compression) and increases in later layers (information enrichment). Based on this finding, they introduce entropy increase as a pruning criterion to identify less informative blocks, arguing it better captures information richness compared to the commonly used cosine similarity. Extensive experiments on models like Llama3.1-8B and Mistral-7B-v0.3 across various benchmarks demonstrate that EntroDrop effectively reduces model size and accelerates inference while maintaining competitive performance, outperforming existing layer and attention pruning baselines.", "questions": "Has the author explored training the pruned models? Because a model that performs better after pruning does not necessarily mean it will still perform better after training. I believe such experiments are crucial to more definitively prove that entropy-based layer pruning is superior to cosine similarity-based pruning.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The empirical analysis of layer-wise entropy dynamics provides a fresh and well-motivated perspective for structured pruning, moving beyond geometric similarity measures like cosine similarity.\n\n\n2.The paper includes thorough experiments analyzing different aspects, including the impact of calibration datasets, comparisons of entropy estimation methods (Bucket, KNN, Renyi), hyperparameter robustness, and actual inference speedup, providing strong empirical support.", "weaknesses": "1.The paper omits citations of some layer pruning methods, such as LLM-Streamline[1] and Shortened llama[2].\n\n2.The proposed entropy-based pruning method does not consistently demonstrate clear superiority over cosine similarity-based approaches across all scenarios. While their performance is often comparable, and sometimes one method significantly outperforms the other, the authors could provide further insight into the specific conditions under which entropy-based pruning is more advantageous.\n\n3.Based on the above experimental observations, expanding the evaluation to include more models would be highly beneficial. This would help clarify the circumstances in which entropy-based pruning is preferable to cosine similarity-based methods.\n\n[1]Chen, Xiaodong, et al. \"Streamlining redundant layers to compress large language models.\" arXiv preprint arXiv:2403.19135 (2024).\n\n[2]Kim, Bo-Kyeong, et al. \"Shortened llama: A simple depth pruning for large language models.\" arXiv preprint arXiv:2402.02834 11 (2024): 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)