Rethinking the Value of Training-Free Structured Pruning of LLMs
Authors: Nahush Lele
Arnav Chavan
Aryamaan Thakur
Deepak Gupta
Venue: Accepted by TMLR ()

Abstract: This paper investigates the effectiveness of training-free structured pruning techniques for Large Language Models (LLMs), with a particular focus on depth and width pruning strategies. Through an extensive empirical evaluation across a diverse range of tasks, datasets and modalities, we reveal critical limitations in current pruning methods. While some tasks exhibit minimal performance degradation, others face significant deterioration, even at low pruning rates, contradicting prior findings that often rely on selective benchmarks. Our analysis also finds that depth pruning, despite its simplicity, usually outperforms the more granular width pruning approaches in maintaining downstream task performance. Our findings highlight that existing evaluations of pruned LLMs often overstate their effectiveness due to incomplete or limited evaluation tasks, necessitating a critical reassessment of the true value of pruning and emphasizing the need to explore more robust pruning algorithms.

Published in Transactions on Machine Learning Research (5/2025)
Rethinking the Value of Training-Free Structured Pruning of LLMs
Nahush Lele"...2 Arnav Chavan Aryamaan Thakur Deepak Gupta*
Indion Institute of Technology (ISM), Dhanbad, India
Nyyun AI, India
"Amazon Lab120
Transmate AT Lab (Termin Hub)
Reviewed on OpenReview: https: // openreview. net/ forum? id- 7RkkytYYNM YYNMu
Abstract
This paper investigates the effectiveness training-free structured pruning techniques for Large Language Models (LLMs), with a particular focus on depth and width pruning strategies. Through an extensive empirical evaluation across diverse range of tasks, datasets and modalities, reveal critical limitations in current. pruning methods. While some tasks exhibit minimal performance degradation, others face significant deterioration, pruning rates, contradicting prior findings that often rely selective benchmarks. Our analysis also finds that depth pruning, despite its simplicity, usually outperforms the more granular width pruning approaches maintaining downstream task performance. Our findings highlight that existing evaluations of prumed LLMs often overstate their effectiveness due to incomplete or limited evaluation tasks, necessitating a critical reassessment of the true value of pruning and emphasizing the need to explore more robust pruning algorithms.
at low
1 Introduction
Due to their large size and complexity, Large Language Models (LLMs) face several challenges in deployment for real-world applications (Thompson et al.) 2022 This issue has sparked significant rescarch focused on
creating lightweight. LLMs (Mehta al. 2024 Thawakar el al. 2024 Abdin el al. 2024) or compressing
2024 to enable efficient use in resource-constrained settings, such mobile devices, edge servers, and embedded systems. The primary goal of these efforts is to balance model efficiency with minimal performance degradation, ensuring that LLMs remain functional across various applications while reducing their resource demands.
existing models (Gu al. 2024 Srrenivas
To achieve this balance, multiple compression techniques such as pruning (Ma et al. 2023 An et al. 2023a
quantization (Lin et al. 2024a Frantar et al.] 2023). ) matrix decomposition (Lin et al. 2024b Chavan et al, 2024a b) and knowledge distillation (Sanh et al. 2020 Jiao et al. 2020 Gu et al. 2024) have been explored
that modify the structure and operation of LLMs reduce their size and computation needs (Chavan et al. 2024c). Structured pruning (Molchanov al. 2016), in particular, has emerged
key method for compressing LLMs by selectively removing model components sucoonhentssschhhae layers neurons neurons-without extensive retraining. Unlike unstructured pruning (Han et al. 2015a), which targets individual weights, structured pruning offers the advantage of removing entire groups of parameters, thus yielding greater efficiency gains and simplifying deployment. However, traditional structured pruning methods often rely on retraining
Equal contribution. Work done while Arnav was CTO at. Nyun AT.

---

Published in Transactions on Machine Learning Research (5/2025)
fine-tuning to recover any performance loss, which can be computationally costly and time-consuming, especially for very large models (Frankle et al.
2020). In response to these limitations, there is a growing interest in training-free free structured pruming techniques, which avoid the need for additional retraining (Ma ct al. 2023 An et al.) 2023a Kim et al. 2024 Men et al.) 2024h. Despite their potential to enable faster deploymente training-Tres free methods remain underexplored, particularly in the context of how they impact. model performance and generalization across various downstream tasks.
In this study, rigorously investigate training- free structured pruning techniques in the context of LLMs, examining the performance implications of both depth pruming (removing entire layers) and width pruning (removing neurons or channels) across diverse tasks. Our contributions are threefold:
Comprehensive Evaluation of Training- Free Structured Pruning of LLMs: We present an extensive empirical analysis of recent training- free structured pruming methods applied to LLMs, extending the evaluation to additional tasks and datasets beyond those previously reported. This broader assessment allows us to identify patterns and limitations in pruning that were not apparent in prior studies.
Critical Benchmarking of Pruming Effects: Our findings reveal that even at minimal pruning rates, significant performance degradation can occur critical benchmarking tasks, which contrasts with prior work that often reports results on selectively curated benchmarks. By evaluating the impact of pruning across a representative set of standard benchmarks, we highlight the potential risks and limitations associated with training-froe structured pruning, underscoring the need for cantion when applying these methods broadly.
Insights inlo verious Pruning Stralegies: We provide novel insights into the prunability of LLMs across both depth and width pruming regimes. Our analysis reveals distinct pruning dynamics across model sizes, challenging conventional assumptions by showing that depth pruning, despite its simplicity, can sometimes outperform width pruming in maintaining task-specific performance. We also explore the bias and fairness implications of pruning, highlighting ethical considerations that have yet be fully addressed in existing research.
Our findings raise important questions about the broader applicability of current pruning techniques and contributes to the growing understanding of efficient LLM compression, providing actionable insights for rescarchers and practitioners focusod on deploying these models across resourco-limited environments.
2 Related Work
Model Compression Techniques for LLMs have played an important role in enabling the deployment of LLMs resource constrained hardware. Traditional methods, such quantization (Han et al.) 2015b), reduce the precision of weights and activations thereby reducing the memory and computational requirements Lin et al. 2024a Frantar et al., 2023). Matrix decomposition techniques, such as low-rank factorizations, approximate large weight matrices with smaller, more efficient representations, and have been utilized in LLMs to reduce the number of parameters while maintaining model quality (Chavan et al.) 2024a b). Knowledge distillation, aims transferring knowledge from a larger teacher model to a smaller student model, allowing the creation of smaller networks which inherently use fewer resources (Gu al. 2024 Sreenivas et al., 2024). While these approaches have demoustrated effectiveness in reducing the size and complexity of LLMs, they often require additional training or fine-tuning steps, which can be computationally costly, especially for very large models.
Structured pruning has gained prominence a particularly effective technique for reducing model size and improving practical efficiency (Molchanov et al. 2016). By removing entire structures such as channels, or layers structured pruning maintains a more interpretable and farddare-friendly model structure, unlike unstructured pruning, which involves removing individual weights and can lead sparse matrices that are challenging to deploy (Li et al., (017).. Studies have shown that structured pruning can effectively compress models without significant, performance loss, especially when combined with fine-tuning (He et. al. 2017a).
2

---

Published in Transactions on Machine Learning Research (5/2025)
Training-Fres Free Structured Praning: While structured pruning methods have shown success, they often rely retraining to recover lost performance, a process that is resource-intensive for large models. To mitigate this, recent research has explored training-free structured pruning techniques that aim to simplify models without roquiring additional training stops. Approaches in this domain include pruning based on weight magnitudes,
neuron activat ions, or structural properties of the model (Ma et al. 2023 Dery et al. 2024 et al. 2024 An et al. 2023a).
Bias and Fairness in Pruned Models: The impact of model pruning on bias and fairness has become area of growing concern, especially as LLMs are increasingly applied to sensitive domains. Studies have shown that pruning can unintentionally affect a model's learned representations, potentially amplifying biases or reducing performance on underrepresented groups (Hooker et al.) 2020 Ramesh et al. 2023). Fairness-aware pruning strategies have been proposed to mitigate these effects, typically by incorporating fairness constraints during the pruning process (Dai et al. 2023 Lin et al., 2022). However, little work has examined these impacts specifically for pruning, leaving an important gap the literature. In our study, we extend the analysis of training-free pruning techniques to include the effects on bias and fairness, providing novel insights into the ethical implications of model compression.
3
Background
This study investigates the impact of training- free structured pruning on large language models (LLMs). We evaluate two main pruming strategies atrategies-eephh and width pruning- across a variety of LLMs and downstream tasks. Our analysis spans both standard and lessor-explored tasks, emphasizing the importance of comprehen- sive task coverage when assessing pruning effects. Additionally, we examine the influence of pruning on model fairness and bias, providing a holistic view of its implications on model performance, resource utilization, and equitable treatment across demographic groups.
manning-ccross
3.1 Training-Free Structured Pruning
Structured pruning has emerged as an effective technique for compressing deep neural networks by selectively removing parts of the model in an interpretable manner. Unlike unstructured pruning, which removes individual weights and can lead to sparse connections, structured pruning targets groups of parameters, such as channels, entire layers. This approach simplifies the final model structure, making it more compatible with hardware accelerators and easier to deploy in real-world scenarios (Liu et al. 2018 He et al.) 2017b). However, structured pruning presents several challenges, particularly structured pruning is followed by resource intensive fine-tuning in order to recover lost performance. This is especially problematic in the case of billion-scale LLMs where retraining based on full-model gradients is limited to large enterprises (Liu et al., 2018). Training-fue free structured pruning bypasses the need for retraining by directly pruning the model without any subsequent. optimization steps, making it highly suitable for scenarios with limited computational resources. They rely intrinsic properties of the model, such weight magnitudes and activation fluctuations, to identify and remove less critical components, often substituting the pruned components with additional bias terms (An) et al. 2023a). Training-free structured pruning is often complemented with parameter-efficion fine-tuning (PEFT) techniques (Hu et al.] [202]] Lian et al. (022), however the improvements with PEFT are inferior as compared to full-fine Ramesh et al. (2023). Nevertheless, training free structured pruning reduces memory and computational costs, enabling deployment on resource-consttrineel devices or environments where retraining infrastructure may not be available.
However, training-fres free pruning comes with significant trade-offs. Because these methods do not allow the model to adapt post-pruning, they can lead to more pronounced performance degradation, especially for complex tasks that rely on nuanced representations. Without the corrective phase of fine-tuning, training-free pruned models may experience reduced accuracy, affecting their reliability across diverse applications. Unlike smaller models, LLMs have extensive feature hierarchies and complex dependencies that may be disrupted by pruning, particularly without the restorative benefits of fine-tuning. As such, while training-free pruning provides efficiency, may compromise task-specific performance, highlighting need to balance compression gains with acceptable accuracy loss.
3

---

Published in Transactions on Machine Learning Research (5/2025)
3.2 Pruning Techniques
Our chosen pruning approaches encompass both depth and width pruning techniques independently, targeting different aspects of model structure. Depth pruming involves the selective removal of entire model blocks, guided by an activation similarity based importance criterion (Men et al. 2024 Gromov et al. 2024 Jha el al.)
2024 Liu et al,, 2023), whereas width pruning leverages activation fluctuations to evaluate and adaptively compress individual weight columns within model layers (An et al. 2023a Ma et al. 2023). More details on
selected pruning methods are provided in Appendix.
Depth Pruning: We adopt activation similarity based depth pruning method (Men et al. 2024 Gromov ct al. 2024). ShortGPT calculates the cosine similarity between the input and output emboddings of the ith block, using cosine similarity as a global pruning criterion across model layers. The rationale behind this approach is that if a block's input and output features show minimal difference- as indicated by high cosine similarity then that block contributes less significantly, relative to other blocks with lower cosine similarity.
Width pruning involves selectively pruning heads and channels from the attention and MLP layers respectively within each module. In the attention layer, pruning is performed at the gramularity of heads, meaning that entire Key-Value (K-V) heads are removed. As a result, any associated Query heads are also pruned, effectively reducing the dimensionality of the attention mechanism. This is especially relevant in models which employ group-query attention wherein multiple query heads are attended by the same key-value heads thus leading to the pruning of multiple query heads on the removal of a single key-value head. In the case of MLP layers, traditional channel- prruning is adopted for the fully connected layers.
Width Pruning:
We adopt activation fluctuation based FLAP (An et al. 2023a) which is an approximate metric for structured rocoverability, referred to as the fluctuation metric. This metric is calculated by taking the sample variance of each input feature and weighting it by the squared norm of the corresponding column in the weight matrix. This approach allows us to estimate the recoverability of specific features in structured manner.
The chosen pruning methods, ShortGPT for depth pruning and FLAP for width pruning, fairly represent the state-of-the-art approaches the literature for training-fres free structured pruning. ShortGPT captures the essence of depth pruning by leveraging a straightforward yet effective activation similarity metric, which aligns with the broader class of techniques that aim to identify and eliminate redundant layers based on their contribution to the overall model representation. Similarly, FLAP embodies the principles of width pruning by assessing the structured recoverability of channels and heads, which is consistent with other methods that focus on pruning individual components within a layer. To ensure that FLAP adequately represents the stale-of-the-art in width pruning, we compare it against LLM-Pruner (Ma et al. 2023), a competitive method in this domain, across our benchmark suite. The results, presented in the Appendix 6.5, consistently demonstrate the superiority of FLAP over LLM-Pruner, thereby supporting our choice of width pruning technique. By selecting these representative methods, we ensure that our evaluation encompasses the primary paradigms of structured pruning while remaining computationally feasible for large-scale models.
3.3 Extended Evaluation
We present an extensive empirical analysis across multiple LLM application areas. Specifically, we focus on the primary common sense reasoning tasks reported in existing works but extend them to more niche and practical application areas such mathematics and coding ability. Additionally, also focus instruction following ability of pruned LLMs and the influence of in-context prompts in model predictions. We also investigate the impact of pruning on multimodal understanding of LLMs. Finally, we present. a thorough investigation on the bias and fairness of pruned LLMs. We compare all the above mentioned tasks across depth and width pruning regimes and multiple compression ratios. This extended evaluation provides insights into real world performance of pruned LLMs. Finally, we provide insights on the appropriate task-specific pruning methodology.
4

---

Published in Transactions on Machine Learning Research (5/2025)
4 Empirical Analysis
4.1 Choice Datasets
This section outlines datasets for evaluating pruned LLMs across reasoning, language modeling, code generation, instruction following, fairness, and multimodal tasks. Key benchmarks include ARC, GSM8k, MMLU, Wikitext-2, HumanEval, IFEVAL, StereoSet, POPE, Tex,VQA, and MMMU. These datasets assess linguistic, logical, and multimodal capabilities comprehensively. The details of each of the mentioned tasks are provided in the Appendix.
4.2 Choice of LLMs
We selected a diverse range of large language models for pruming and evaluation on reasoning-based tasks, including LLaMA-3-8D (Dubey et al. 2024), Gemma-2-99B, Gemma-2-22 2-22 (Team et al. 2024), and Qwen-2- 1.5B YYagg et al. 2024p. These models were chosen for their mass adoption in real-world applications and parameter scale where pruning matter much more as compared to >50B parameters. By applying pruning techniques, assess cach model's efficiency in terms of throughput and their performance selection of lasks. For lasks requiring instruction following, and Gemma-2-9B. These instruction-tunee models are optimized to better understand and follow task-specific instructions.
diverse instruction-luned versions of LLaMA-3-8B
4.3 Incompleteness of current evaluation schemes for LLM pruning
Random
58-
Ransom
Baccline
Raselre
Puyfor mannce
Performonn
BoolQ
Winsdrande
ARC-Eaay ARC-Challenge MHW Thocs
GSMBk
BoolQ)
WineGrande
ARC-Easy ABC-Chalenge Tasks
MNL
GSMBK
(a) Width Pruned Gemma-2-9n
(b) Width Pruned HLMA-33SB
Confguintion
Confgurat one
Nar
2.
20%
30%
30/
hefermancc co
Bool
WinoGrande
Eacy Ce MMW
GSMBk
B630 WineGrande
ARC Easy Chalenee MHW
GSMBK
(c) Depth Pruned Gemma-2-99
(d) Depth Pruned LLaMA-3-8B
Figure 1: Evaluations on Extended Benchmarks for Width and Depth Pruned Models The figure compares the performance of width-pruned 1a 1b) and depth-pruned [Ic 1d) versions of Gemma-2-9B and LLaMA-3-8B across varying compression ratios (10%, 20% and 30%) on extended reasoning benchmarks: BoolQ, WinoGrande, ARC-Fasy, ARC-Challenge, MMLUU and GSM8K. Baseline performances and random thresholds are included for reference.

---

Published in Transactions on Machine Learning Research (5/2025)
This section highlights the impact of depth and width pruning numerous downstream tasks across varying budgets. BoolQ, ARC-Challenge, ARC-Fasy, and WinoGrande are widely recognized common sense reasoning benchmarks, frequently reported in the majority of pruning literature as performance metrics. In addition to these, wC propose evaluating models on MMLU and GSMsK, which are knowledge-based reasoning tasks. MMLU serves as a benchmark for assessing the knowledge acquired by language models during pre-training, covering 57 categories and making a knowledge-intenes redge--nteensie task. In contrast, GSM8K is designed to evaluate high-scchol-level mathematical reasoning skills. While foundational models frequently report performance on these benchmarks, we argue that it is equally important to include these evaluations in pruning literature. Doing so provides more comprchensive understanding of the capabilities of compressed models. Papers introducing foundational models Dubey et al. (2024); Yang et al. (2024) typically present comprehensive suite of benchmarks to remonstraae the model's versalility across diverse lasks. In contrast, prusing literature often focuses on a limited set of simpler benchmarks,, primarily in the domain of common sense reasoning. In rare instances, datasots like MMLU are included, highlighting a narrower scope of evaluation. Our findings Figure [ support (iidiiqui al. (2024), which shows significant performance drops in depth-pruned models GSM&K even at low compression ratios. Extending this trend, find that width-pruned models exhibit similar decline. The complete evaluations of Qwen-2-1.35 1.5B and Gemma-2-22 across varying budgets using both depth and width pruning are provided in the Appendix.
4.4 Evaluation beyond the norm
Coding Ability
We evaluate the code completion ability using the HumanEval dataset, focusing on functional correctness. For each sample in the test. set, we generate five completions with a temperature setting of 0.8 and compute the pass@1 metric across the entire dataset to assess performance. We conduct evaluations on the Gemma-2-9B and LLaMA-3-SB models, compressed by 10% and 20% using both depth and width pruning methods. Our findings as shown in Figure indicate that pruning has a profoundly negative impact on the code completion abilitics of models. Regardless of the pruming method employed, results in significant decline in pass@1 performance. This pattern is consistent across both the Gemma-2-99 observe that both depth and width pruning methods have a
modest compression ratio of 10%
and L.MMA-3-8B model series. Furthermore, similarly detrimental effect on the models' code completion capabilities. At higher compression ratios, the models lose their ability to maintain proper syntax and begin generating outputs with random, repetitive tokens, further highlighting the adverse effects of aggressive pruning on their functionality. These findings underscore that existing evaluations of pruned LLMs often overstate their effectiveness due incomplete limited evaluation tasks, necessitating a critical reassessment of the true value of pruning.
Compression Beseline
Compresssion
10%
10%
20%
20%
Performance
Purfomannce
30
Gemmal-2-4b-lt
Liima-1-8--it
Gemma-2-99bu
Models
Models
(a) Pass@1 performance of Depth Pruned Models
(b) Pass@1 performance Width Pruned Models
Figure 2: The figure illustrates the Pass@1 performance of depth-pruned 2a and width-pruned 2b versions of L.L.AA3-8B and Gemma-2-0B models on Human-Fae Eval different compression ratios (10% and 20%), alongside their respective baselines.
Instruction Following

---

Published in Transactions on Machine Learning Research (5/2025)
The evaluations in Table I reveals that the performance degradation becomes significant compression ratios increase, with width pruning generally leading to sharper declines compared to depth pruning. While depth pruming retains better performance at lower compression levels, the gap narrows at higher ratios, where both strategics exhibit severe losses. This benchmark is often overlooked in pruning literature, as our ovaluations reveal a far greater loss in performance compared to the more commonly reported metrics.
Table 1: IFEVAL scores for LLaMA-3-8B and Gemma-2222 Instruction Tuned Model computed across varying budgets (10%, 20% and 30% specifically for LLaMA-3-8B)
Model
Strategy
Compression (%)
Score
Baseline
0.410
10
0.356
Depth
20
0.216
LLMMA-3-BB
30
0.130
10
0.301
Width
0.258
30
0.110
Baseline
0.321
0.255
Gemma-2-2B
Depth
20
0.144
0.125
Width
20
0.097
Table 2: Depth and width pruned LLaMA-3-70B results on 20% and 40% compression on reasoning and Wikilext
Method
Compression (%)
ARC
ARC_
Winogrande
WikiText
BoolQ
MMLU
Baseline
0
86.91
64.33
80.58
2.91
85.17
75.02
20
80.85
37.25
79.40
9.02
85.56
74.65
Depth
40
58.04
42.15
76.08
> >1e
66.85
68.07
20
32.42
42.13
65.35
2.9e5
84.34
62.99
Width
40
31.65
29.60
65.75
>> le6
46.36
72.08
4.5
Depth vs. Width pruning of LLMs
We now investigate how depth and width pruning affects model performance, focusing on their unique impact across various tasks. Based better overall compared to width pruning. Depth pruning, while reducing the number of layers, appears to preserve the model's ability to extract meaningful hierarchical features, which are crucial for tasks involving logical reasoning and factual knowledge. ARC-Easy, ARC-Challenge and BoolQ consistently benefit from depth pruning as compared to width pruning while on the other hand MMLU and GSM8k show mixed results depending on the model and compression level. When we run standard evaluations on Llama-3-70B for reasoning tasks, we obscrve a similar trend where depth pruned models consistently outperform those that are width pruned on average. This observation, detailed in Table reinforces our claim that depth pruning better preserves the model's critical reasoning capabilities while achieving compression. We find that the Gemma-2 series of models is particularly susceptible to significant performance deterioration introduced by width pruning. As observed in our evaluations in Figure 3a and Figure 3c the degradation becomes more pronounced as the pruning ratio increases, with tasks like ARC-Easy, ARC-Challenge, and MMLU showing clear advantages for depth pruning over width pruning.
evaluations presented in Figure 3 find that depth pruning performs
However, we also find that perplexity reflects higher degradation upon depth pruning compared width pruning shown in Figure| This trend indicates that depth pruning, while preserving performance
7

---

Published in Transactions on Machine Learning Research (5/2025)
5.5
Whrggran ands
0.04
0.02
6.94
Wirogmanc
0.02
0.24
-0.01
1.03
ARC Eaiy
0.36
0.05
3.05
0.01
lange
0.02
MNLU
4.001
00.36
0.38
0.01
-0.03
-0.20
-0.1
0.28
GOMRE
3.01
622
(a) Gemma-2-98 2-9B
(b)
LLaMA-3-88
0.01
0.35
5.03
Minogradee
0.21
0.5
0.12
Wirogrince
0.02
+0.64
2.01
5.00
0.09
0.09
-0.35
lange
lange
0.03
-0.10
NN
<0.01
-0.00
0.18
19H1
0.01
40.01
3.00
0.07
-4.45
0.01
-0.1
(c) Gemma-2-2B
(d) Qwen-2-1.5B
Figure Difference in Performance of Depth and Width Pruned Models The figure showcases the performance difference between depth-pruned and width-pruned models (Depth Width) across varying compression ratios (10% 20%, and 30%) on extended reasoning benchmarks.
Meta Llame-3 5-19 busding
Qwen2-136 Baseins
Meta Lane- 3-15 Dupth
Qwen2-158 Death
Meta 3one- 3-58 widh
gwen2-155 Widt
Gomma-22 90 Sapelie
Gamma: 2:20 Bossliiee
Gamma-2 3:30 Depth
Gemme-2-2 2-2h
Gamma-3
Gemma
Compessdon
Compres on (4)
(a) Compressed LLaMA-3-SB and Gemma-2-99B
(b) Compressed Qwen2-1.5B and Cemma-2-2B
Figure 4: Perplexity Performance on WikiText-2 for Compressed Models The figure compares the natural log of perplexity for depth- and width-pruned models at compression ratios of 10%, 20%, and 30% on the WikiText-2 dataset. Subfigure 4a)) shows the results for LLaMA-3-8B and (emmma-2-913 while subfigure (4b) presents those for Qwen2-1.5B and Cemma-2-2B. The baseline perplexity for each model is represented by dashed lines.
certain reasoning tasks, significantly disrupts the model's ability to generalize across broader language modeling objectives. At higher pruning ratios, the difference in perplexity between depth-pruned and width- pruned models becomes substantial, often reaching an order or even multiple orders of magnitude higher. This highlights a critical tradc-off: while depth pruning retains task-spocific performance more effectively, it imposes severe penalty on perplexity, suggesting greater loss of overall language modeling capacity compared to width pruning.

---

Published in Transactions on Machine Learning Research (5/2025)
4.6 Throughput Measurement of Depth and Width Pruned Models
To evaluate the efficiency of depth and width prumed models across different pruning budgets, measure throughput in terms of tokens generated per second with batch size of 1. This setup as shown in Table 3 allows us to obscrve the effect of varying pruning strategics on single-token generation speed. Depth pruming and width pruning achieve computational reductions in fundamentally different ways. Depth pruning focuses on removing entire transformer blocks, significantly decreasing both memory usage and inference latency, as it directly reduces the number of sequential operations in the forward pass. In contrast, width pruming targets the model's internal structures, such reducing the number of attention heads or the size of MLP layers within transformer blocks. While width pruning reduces the per-layer computation cost and memory requirements, it does not reduce the depth of the model, meaning the number of sequential layers remain constant. As a result, depth pruning offers more pronounced reductions in latency as compared to width pruning.
Table 3: Throughput in tokens/second for LLaMA-3-SB and Gemma-2-212 The table reports the throughput (measured in tokens per second) for LLaMA-3-SB and Gemma-2-2B models under different pruning st staaeggees (Depth and Width) and compression ratios (10%, 20%, and 30%). The baseline throughput without pruning included as reference.
Model
Pruning Strategy
Compression %
Value (Tokens/Second)
Baseline
37.16
10%
41.90
LLAMA-3-8B
Depth
20%
45.72
30%
54.40
10%
38.26
Width
20%
37,98
30%
39.12
Baseline
31.35
10%
31.95
Gemma-2-2B
Depth
20%
35.87
30%
44.54
10%
28.65
Width
20%
28.48
30%
28.66
4.7 Effect of Pruning on Bias and Fairness
For a holistic evaluation scheme, we evaluate stereotypical biases LLaMA-3-8B (Figure and Gemma-2-9B models Figure 7 both depth and width pruned, by reporting the ICAT scores on the StereoSet dataset. The evaluation includes bias measurements across categories such as race, religion, gender, and profession, along with the computation of the overall ICAT score for each model. It has been noted that biases tend to decrease language models undergo higher compression Ramesh et al. (2023). Through experiments, find that the overall stereotypical bias remains intact consistently across budgets and models. Interestingly, in certain individual categories such as race, we observe an improvement in ICAT scores when compared to the baseline after pruning, indicating a reduced stereotypical bias post-pruning.
4.8 Effect of pruning multimodal ability
Vision-Language Models (VLMs) integrate visual and textual modalities to perform tasks requiring cross- modal understanding, such as image captioning and visual question answering. To evaluate the effect of depth and width pruning on the multimodal performance of VLMs, we first isolate the language model of Idefics3-8B [Laurençon al.] 2024), which is a fine-tuned LLaMA3-8B model, and prune under various budgets. The pruned language model is then reconnected the vision encoder, and the full model's performance is assessod across multiple multimodal benchmarks as shown in Table 4 Through this experiment find that depth

---

Published in Transactions on Machine Learning Research (5/2025)
Profession
Profession
Race
Race
Gender
Gender
Rellgion
Rellgion
Overall
Overall
Baseline
Compressed
Compressed
Compressed
Compressed
(a) Depth Pruned LLaMA-3-88
(b) Width Pruned LLaMA-3-8B
Figure 5: ICAT Scores for Depth and Width Pruned LLaMA-3-8B. The figure presents the ICAT scores across five domains- Gender, Profession, Race, Religion, and Overall for depth-pruned (5a) and width-pruned (5b) versions of LLAMA-3-8B varying compression levels (10% and 20%) and the baseline.
pruning performs better overall which aligns with our previous findings. Moreover, we observe that while performing width pruning, as increase the compression ratio POPE score decreases drastically suggesting that the model begins to hallucinate although it's performance other benchmarks is retained comparatively. However, during depth pruning the model loses its Text Understanding capabilities as the pruning ratio increases.
Table 4: The table presents the performance of Idefics3-BB across three benchmarks POPE, Text VQA, and MMMU under varying pruning strategies (Depth and Width) and compression ratios (10% and 20%).
Pruning Strategy
Compression %
РОPE
Tex
MMMU
Baseline
86.72
60.42
10%
87.17
54.66
41.56
Depth
20%
72.71
25.38
38.44
10%
86.99
50.16
42.56
Width
20%
54.24
37.20
4.9 Effect of pruning on long-conter handling ability
We present evaluations on long-con cong-context benchmarks using the MLLaMA3-BB-Insruuee model within the LEval framework (An et al.) 2023b) in Table L.val includes multiple challenging tasks: QuALITY, which consists of questions based on long literary stories from Gutenberg; Coursera, designed assess the reasoning ability of LLMs complex and tiffieulttto-compreeeed course materials; SFiction, which features questions that contradict real-world principles, testing the model's ability to rely on input context rather than pretrained knowledge; and GSM100, subset of 100 grade school math problems from the GSM8K datasct, aimed at evaluating few-shot learning capabilities in very long-context scenarios. The table below presents the performance different pruning strategies at 10% and 20% compression on long-context benchmarks, compared the Bascline. Depth pruning initially improves performance on SFiction but degrades on Coursera and GSM100 as pruning increases. Width pruning shows trade-off, with 10% pruning slightly improving overall performance but 20% pruning significantly reducing accuracy, especially GSM100.
10

---

Published in Transactions on Machine Learning Research (5/2025)
Overall in the case of long context understanding, we do not see clear pattern with respect performance degradation.
Table 5: Comparison of pruning strategics different compression levels across long-context
evaluation benchmarks
Strategy Pruning (%)
Quality
Coursera
SFcition
GSM100
Baseline
62.38
52.33
69.53
80.00
10%
62.38
47.24
75.00
74.00
Depth
20%
62.87
42.88
67.19
39.00
44.19
70.31
Width 20%
60.40
51.45
67.97
39.00
4.10 Choosing the right pruning strategy
Depth pruning has been shown to outperform width pruning at moderate comprossion ratios, particularly for reasoning-inlensive lasks such GSM8K and knowledge benchmarks like MMLU. However, perplexity evaluations demonstrate that depth pruning can significantly degrade language modeling capabilities. The removal of entire layers disrupts the model's depth-driven representation learning, which essential for capturing sequential dependencies in language tasks.
In contrast, width pruning achieves compression by reducing the dimensionality of individual layers. This method is more effective for maintaining language modcling performance, preserves the model's depth. However, at higher compression levels, width pruning disproportionately reduces the model's representational capacity, leading to sharper decline in downstream task performance, particularly for reasoning and knowledge-intensive tasks.
Depth pruning is therefore more suitable for scenarios that involve reasoning-inted intensive tasks, require moderate compression ratios of 10 10-20%, and prioritize significant inference speedups. Tt. less effective when language modeling capabilities are central to task performance. Conversely, width pruning is more appropriate preserve language modeling capabilities with minimal performance loss.
In cases where both efficiency and hierarchical depth are critical, combining depth and width pruning in a hybrid approach can help balance compression performance and retention of downstream task accuracy. By strategically leveraging the strengths of both methods, it is possible to mitigate the trade-offs associated with either pruning strategy.
4.11 Fairing Against Unstructured Pruning on Extended Benchmarks
While focus evaluating structured pruning methods, it is important to understand how these fair against unstructured pruning methods such as Wanda and SparseGPT on our extended benchmarks. Our experiments with SparseGPT (Frantar and Alistarh 2023) and Wanda (Sun et al. 2024) on LLaMA3-SB, Table 6 span common sense reasoning tasks, mathematical reasoning (GSM8K), Code Eval, and Instruction Following (IFEVAL). Our findings suggest that unstructured and semi-structured pruming methods, which allow for finer granularity, generally preserve ream task performance better than structured pruning approaches that reduce model depth or width. The only task where these methods exhibit significant performance degradation is mathematical reasoning. These results offer further insight into the impact of unstructured pruning LLMs and highlight that, when the target hardware supports unstructured sparsily, these methods may be preferable over structured pruning techniques.
5 Conclusions
Training-free structured pruning of LLMs presents significant challenges, evidenced by our comprehensive evaluations. Our results demonstrate that such pruning methods lead to substantial performance
11

---

Published in Transactions on Machine Learning Research (5/2025)
Table 6: Results for unstructured pruning methods on extended benchmarks with different
compression ratios.
Pruning Method
Compression (%)
ARC
ARC
Winogrande
BoolQ
MMLU
GSM8K
CodeEval
WikiText
IFEVAL
10%
18.11
5.58
Wanda
20%
79.71
49.14
72.61
81.87
61.11
47.83
49.90
5.71
39.18
30%
78.57
18.38
13.10
81.80
59.31
11.39
18.11
6.03
11.03
10%
30.38
81.71
61.69
19.14
5.58
11.22
SpaneGFT
21%
79.59
29.31
72.85
81.68
60.95
48.06
18.28
5.72
41.77
30%
78.
47.61
74.11
81.89
50.88
13.59
47.93
6.05
39.74
degradation, particularly on benchmarks that are often overlooked. Highlighting these underperforming benchmarks is essential, as it encourages the development of new methods that address these shorteomings and improve overall model robustness.
We have showeased the differences between depth and width pruning strategies to gain deeper understanding of their disparate impacts on downstream tasks. This disparity emphasizes the need for careful selection of pruning strategies based on the specific requirements of the task at hand.
By bringing attention to these challenges and disparitics, we hope this paper inspires future research in the compression literature evaluate their methods across a wider variety of tasks. Comprehensive evaluations will not only illuminate the limitations of current approaches but also guide the development of more effective pruning techniques that maintain performance while achieving desired levels of model compression.
References
Neil C. Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel Manso. The computational limits of deep learning, 2022. URL https://arxiv. org/ ab0//007 05558
Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, et al. Openelm: An efficient language model family with open-source training and inference framework. arXiv e-prints, pages arXiv 2404, 2024.
Omkar Thawakar, Ashuaal Vayani, Salman Khan, Hisham Cholakal, Rao M Anwer, Michael Felsberg, Tim Baldwin, Eric P Xing, and Fahad Shahbaz Khan. Mobillama: Towards accurate and lightweight fully transparent gpt. XrXiv preprint arXiv:2402.165/ 2024.
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmod Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, al. Phi-3 technical report: A highly capable language model locally your phone. arXiv preprint arXiv:2404114219, 2024.
Yuxian Gu, Li Dong, Furn Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: //openreview
Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Lim pruning and distillation in practice: The minitron approach. arXio preprint 10Xiv:2408.111 111669 2024.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models, 2023. URL https://arxxv..ot http//axxv..org/bbb www./ogggg/....e..oeo...e..eeeee 220. 11627
Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation-based adaptive structured pruning for large language models, 2023a. URL
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for 1lm compression and acceleration, 2024a. URL https: atttp://arxiis: https://arrkv.orrraaa//33.....9
12

---

Published in Transactions on Machine Learning Research (5/2025)
Elias Frantar, Salch Ashkboos, Torsten Hoefter, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023. URL
wttpg////g///ooo///////////// 17323.
Chi-Heng Lin, Shangqian Gao, James Seale Smith, Abhishek Patel, Shikhar Tuli, Yilin Shen, Hongxia Jin, and Yen-Chang Hsu. Modegpt: Modular decomposition for large language model compression. arXiv preprint ar Xrx::2208.. 09632, 2024b.
Arnav Chavan, Nahush Lele, and Deepak Gupta, Rethinking compression: Reduced order modelling of latent features in large language models. In ICLR 2024 Tiny Papers, 2024a.
Arnay Chavan, Nahush Lele, and Deepak Gupta, Surgical feature-space decomposition of Ilms: Why, when and how? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2389 2400, 2024b.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020. URL
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding, 2020. URL
wrg./bs///////// 10351
Arnav Chavan, Raghav Magazine, Shubham Kushwaha, Merouane Debbah, and Deepak Gupta. Faster and lighter 1lms: A survey on current challenges and way forward. In Proceedings of the Phirty-Third International Joind Conference on Arlificial Intelligence (IJCAI), pages 7980 7988, 2024c.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611 16110004144, 2016.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient. neural network. Advances in neural information processing systems, 28, 2015a.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis, 2020. URL
Bo-Kyeong Kim, Geommin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung-Ky Song. Shortened Ilama: Depth pruning for large language models with comparison of retraining methods, 2024. URL https: /taps:///arxxii org/ abs/
wwwpaa/....aeeeeee/. 2402.4422.0 02834
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect, 2024. URL
https://arxiv. www.orrgg/.o..o.o...e harg/abb//// 2403/2433.0 018553853
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint 15100000000 2015b.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets, 2017. URL https:://axxi...rrr http://wrrrrbb///..../... abs/1608.08710
Yihui Hc, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks, 2017a. URL
06668
Lucio Dery, Steven Kolawole, Jean- -ran-FFançoos Kagy, Virginia Smith, Graham Neubig, and Ameet Talwalkar. Everybody prune now: Structured pruning of lims with only forward passes, 2024. URL hnttt:t////// https://arxiv org/
05406
Jianwei Li, Yijun Dong, and Qi Lei. Greedy output approximation: Towards efficient structured pruning for Imms without retraining, 2024. URL https://arxiv/ http::///xxvvvvoo////// wttggg///////../// 19126
Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily Denton. Characterising bias in compressed models, 2020. URL http:://aardv.rrrga////00000000
13

---

Published in Transactions on Machine Learning Research (5/2025)
Krithika Ramesh, Arnav Chavan, Shrey Pandit, and Sunayana Sitaram. A comparative study the impact of model compression techniques on fairness in language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15762-15782, Toronto, Canada, July 2023. Association for Computational Linguisties. doi: v11886531122223.../ acl- mol.-oon..78. URL
acl-long. 878/
Yucong Dai, Gen Li, Feng Luo, Xiaolong Ma, and Yongkai Wu. Coupling fairness and pruning in a single run: a bi-level optimization perspective, 2023. URL
10181
Xiaofeng Lin, Soungbac Kim, and Jungscock Joo. Fairgrape: Fairness-aware gradient pruning method for face attribute classification, 2022. URL
10888
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. IEEE Transactions on Pattern Analysis and Machine Intelligence, ((10))25433225))) 2018.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pages 1389-1397, 2017b.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 09685, 2021.
Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: A new baseline for efficient model tuning. Advances in Neural Information Processing Systems, 35:109-123, 2022.
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A. Roberts. The unreasonable ineffectiveness of the deeper layers, 2024. URL hctps://arzvv..og//abb wwwpg//aaa//40....a 17887
Ananya Harsh Jha, Tom Sherborne, Evan Pete Walsh, Dirk Groeneveld, Emma Strubell, and Iz Beltagy. Just chop: Embarrassingly simple Ilm compression, 2024. URL https://arxiv. wtw./////.........a///// 10822305..8844444
Wei Liu, Zhiyuan Peng, and Tan Lee. Comflp: Correlation measure based fast. search on asr layer pruning, 2023. URL https://arxii wwtezivv.rr//aae abs//309//2111111 11768
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint ar 2278833 2024.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408. 0001000 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024.
Shoaib Ahmed Siddiqui, Xin Dong, Greg Heinrich, Thomas Breuel, Jan Kautz, David Krueger, and Pavlo Molchanov. A deeper look at depth pruning of llms, 2024. URL
www./ogg///bbbbeeeee..eeeee abs/2407 16286
Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better understanding vision-language models: insights and future directions. In Workshop on Responsibly Building the Next Generation of Multimodal Foundational Models, 2024.
Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models, 2023b.
Elias Frantar and Dan Alistarh. Sparsogpt: Massive language models can be accurately pruned in one-shot, 2023. URT. https://arxiv. wttp://wwww.g///////..oooo
14

---

Published in Transactions on Machine Learning Research (5/2025)
Mingjic Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models, 2024. URL https://arxiv htt.aaaawv..vvrrrrrraaaaaaaaaaee orggabs/2306...o.. 11695
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schocnick, and Oyvind Tafjord. Think you have solved question answering? try arc, the al2 reasoning challenge. ar 103::0803.0557vl. 05457v1, 2018.
Christopher Clark, Kenton Lee, Ming-Was Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapler of the Association for Compulational Linguislies: Human Language Technologies, Volume 1 (Long and Short Papers), pages
2019.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732-8740, 8740, 2020.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. 2021.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou, estruuction-oolloinng evaluation for large language models, 2023. URL https: /ttp////vvv/////////////////aav/ wwerrvi.org//oeee /bs// 2811.07911
Moin Nadcem, Anna Bethke, and Siva. Reddy. Stereoset: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages
5356 5371, 2021.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint 2tl::330551035555 2023.
Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE Conference on Computer Vision and Pallern Recognilion, pages 8317 8326, 2019.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision 2024.
and Pattern Recognition, pages
15

---

Published in Transactions on Machine Learning Research (5/2025)
6 Appendix
6.1 Details on ShortGPT
SPT calculates the cosine similarity between the input and output embeddings of the ith block, using cosine similarity as a global pruning criterion across model layers.
B, = E (x x
(1)
N
In expression 1 B, represents the pruning criterion for the i-th block, X; denotes the output embedding of the i-th layer, X refers the output embodding of the (i (i-1)-4t 1)-th layer (serving as the input to the i-th layer), N is the total number of samples used in calculating cosine similarity, and Σド=1 (x xx()
represents the summation over all samples, with x and x being the embeddings from the (i - 1)-th
and i-th layers for the j-th sample. After calculating the B, for each block, we sort them in an ascending
order based on the B; values and prunc blocks with the lowest B, l is docided deponding on the target pruning ratio.
6.2 Details on FLAP
Fluctuation metric S,. for the jth channel of the Ith layer is defined as
1 S1 N - Σ x - X1z)² |Wi,
(2)
Here, N denotes the total number of calibration samples, and x is the value of the jth channel of the input features layer for the ith sample. The term X1,; represents the mean of Xt, across all N samples, allowing to calculate the variance of this feature. Additionally, ||W,, refers to the squared norm of the j-th column of the weight matrix in layer which weights the variance term in the metric calculation. Together, these components form estimate of each channel's recoverability by quantifying its variance and weighting it according to the model's structure.
To ensure that the score can serve as a global approximation, the metric is standardized for each layer to a common mean and standard deviation.
St E[St,]
S V E[ 1[
(3)
where E[ denotes the expected value of the vector St,j, and v [[ 1[ represents the standard deviation, calculated the square root of the variance.
6.3 Datasets
In this section, we describe the datascts used as part of our extended evaluation of pruned LLMs:
Common Sense Reasoning and Factual Question Answering
ARC Challenge and ARC Easy (Clark al., 2018): These are standard benchmarks for common sense reasoning and are widely referenced in pruning literature, providing basis for comparison with cxisting models.
BoolQ (Clark et al. 2019) and Winograndc (Sakaguchi al.) 2020): Both datasets are used for binary question-answering tasks that involve common sense and linguistic reasoning, making them essential for understanding how pruning impacts nuanced reasoning abilities.
16

---

Published in Transactions on Machine Learning Research (5/2025)
GSM8k (Cobbe et al., 2021): A dataset designed to test mathematical and logical reasoning skills. This dataset's 's complexity surpasses that of traditional common sense tasks, offering insight into how pruning affects tasks with higher-order reasoning requirements.
MMLU (Hendrycks et al,, 2021): Used alongside GSM8k to evaluate factual knowledge combined with reasoning, allowing us to examine pruning performance on tasks requiring an integration of knowledge domains.
Wikitext-2 (Perplexity): Wikitext-2 serves as a standard language modoling benchmark for measuring model perplexily, providing insights into the fluency and coherence of pruned models. Perplexity a critical metric here, it quantifies the model's ability to predict word sequences effectively, highlighting any degradation in linguistic capability due pruning.
Code Completion on HumanEval (Chen et al. 20210: :: The HumanEval dataset is benchmark for evaluating the functional correctness of code generated by AT models. The primary purpose of the dataset is to test the ability of models to generate syntactically correct and semantically meaningful code that passes all provided unit tests, thus HumanEval emphasizes correctness over superficial metrics like BLEU or code similarity.
Instruction Following on IFEVAL (Zhou et al., 2023): The dataset comprises prompts utilized in the Instruction-Follone Evaluation (IFEVAL) benchmark, specifically designed for evaluating the performance of large language models. It includes approximately 500 "verifiable instructions", which are carcfully constructed tasks that allow for objective and heuristic-based verification.
Fairness Evaluation on Stereoset (Nadeem et al., 2021): StereoSet is a benchmark designed to evaluate the social bias present in language models across dimensions like gender, race, religion, and profession. It tests model's tendency to exhibit stercotypical associations in natural language understanding tasks. A combined metric, the Idealized Context Association Test (TCAT) Score, rewards models that exhibit low bias while maintaining high language quality.
Multimodal Evaluation: To evaluate the multimodal capabilities of Vision-Language Models (VLMs), we use three datasets: POPE (Li et al., 2023) for detecting object hallucinations; Text (Singh et al., 2019) for assessing the ability to read and reason about text in images; and MMMU (Yue et al.
2024), which
lests college-level subject knowledge and deliberate reasoning covering six core disciplines Art. Design, Business, Science, Health & Medicine Humanities & Social Science and Tech & Engineering.
6.4 Analysis of Pruned Models Under Varying Budgets
Here, present the complete evaluations of Qwen2-1.5B and Cemma-2-2B (Figure G to assess the effects of depth and width pruning on smaller-sized models. The results reveal a consistent trend of performance degradation, particularly on the GSM8K benchmark, even at low compression ratios.
6.5 Comparing FLAP with LLM-Pruner
To establish FLAP as state-of-the-arl width pruning method, we compare it against LLMPruner (Ma et al. 2023) across our benchmark suite. Table 7 presents a comparison the zero-shot performance of LLaMA-7B pruned using L.MM-Pruner and FL.A at 20% and 50% pruning ratios, consistently demonstrating FLAP's superiority across tasks.
Table 7: Comparison of methods at different compression levels across multiple benchmarks.
Method
Compression (%)
ARC
ARC
BuolQ
Winogrande
LLM-Prruer 20%
37.18
39.18
61.33
FLAP 20%
39.25
69.91
69.63
68 08.85
LLM-Pruner
50%
25.51
32.83
52.57
19.01
FLAP
50%
29.95
49.66
60.21
57.54
17

---

Published in Transactions on Machine Learning Research (5/2025)
20%
20%
bontg Ainooande
AlC-Eaty Al--Chilenge MHI
65ARK
Boot0
WhncGrarde
ARC Easy ARC-Chillnge NVEU
(a) Width Pruned Gemma-2-22 2-2B
(b) Width Pruned Qwen2-1.58
Randon
Lub
Perfomaree
Winudiand ABC Ahalchdlangge NHW
GUMKK
Sool3
Whscrandd AFC. Bayy WWC-Chalenge NYRU
GSHSK
lsece
bokss
(c) Depth Pruned Gemma-2-21
(d) Depth Pruned Qwen2-1.5B
Figure 6: Performance of Smaller Models on Extended Benchmarks The figure presents the performance of Gemma-2-2B (6c 6a] and Qwen2-1.5B 6d 6b) on exlended reasoning benchmarks,
comparing depth and width pruning
6.6 Bias and Fairness Evaluation in Gemma- 2-9B
We evaluate Gemma-2-99 2-BB model (Figure 7 on the StereoSet dataset to analyze its behavior regarding stereotypical biases under the influence of depth and width compression across varying budgets.
Profession
Profession
Gender
Gender
Rellgion
Religion
Overall
Overall
Baseline
10% Compressed
20% Compressed
Baseline
10% Compressed
20% Compressed
(a) Depth Pruned Gemma 2emma-2-9B
(b) Width Pruned Gemma-2.
Figure 7: Bias Evaluations for Gemma-2-9BB The radar plots illustrate the ICAT across domains Gender, Profession, Race, Religion, and Overall- for depth-prumed 7a and width-
pruned 7b Gemma-2-9B
18


Reviews:
[{"review_id": "62wC8BTd6U", "reviewer": "Reviewer_dot3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The paper is well written\n- The experiments are well designed and clear in terms of their conclusions\n- The topic is important and relevant to a wider audience\n\n\n## Weaknesses\n\n- There have been some recent attempts to analyze the true impact of depth pruning (e.g., see https://arxiv.org/abs/2407.16286). They also showed that having information about the exact task being solved can be leveraged to further boost performance via a more careful selection of model blocks to prune. This makes the contribution in 4.3 not truly novel. The only aspect that I found novel is the fact that you attempted to look at chat-tuned models instead of just pretrained models, which makes it possible to compare on instruction-tuning benchmarks.\n- Why not consider one shot approaches such as SparseGPT (https://arxiv.org/abs/2301.00774) for width pruning, which assumes the calibration set you already have access to for estimating importance? That seems to be aligned with the overall theme of the paper. While ShortGPT is close to the best methods we have for depth pruning, the literature on width pruning is significantly richer.\n- There have been other attempts to construct useful metrics for importance estimation of different attention heads and neurons (e.g., see https://arxiv.org/pdf/2406.10260). Why didn't the authors compare against anything else beyond FLAP? Seems like an unfair comparison between depth and width. While there are only limited useful metrics for depth, a significant amount of work has gone into estimating the importance of parameters along the width dimension. I won't feel comfortable with such a strong general statement while looking at just one particular method.\n\n\n## Questions\n\n- Page 11; Fig 5a. Religion results with 10% depth pruning are worse than the baseline, but almost on par with the baseline at 30%. Seems weird. Any particular reason for that? This is also consistent with Fig. 7a.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- Compared width vs. depth pruning of language models without finetuning\n- Analyzed instruction-tuned models instead of just base models that have been the focus of prior work\n- Showed that despite claims of retaining performance (such as ShortGPT claiming MMLU to be the retrained), performance is sacrificed when looking at a more holistic suite of tasks for evaluation", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Page 3: \"however the improvements with PEFT are inferior as compared to full-finetuning; especially in the case of pruned LLMs\". Add citation to back this strong claim -- this paper is certainly relevant, but doesn't talk about pruning: https://arxiv.org/abs/2405.09673\n- Weird heading at the end of page 8 (\"Throughput Measurement of Depth and Width Pruned Models:\") -- neither a paragraph header, nor a heading. Maybe make it a subsection? You don't seem to be short on space.\n- Page 10: table should be set to be on the top rather than weirdly placed after a couple of lines.\n- Include other methods for width pruning beyond the current FLAP method", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6TblyE5fOv", "reviewer": "Reviewer_wMGw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nDeeper evaluations are always valuable.\n\nWeaknesses:\nI do not like the writing; the authors spend half of the paper on the setup and introduction.\n\nI also do not like the fact, that figure 1 and 4 are far away. Wikitext perplexity is the main measure used in evaluating pruned models. If you want to show, that this measure is not enough and we need to do deeper evaluations, showing lack of correlation between benchmarks and wikitext perplexity would be very valuable.\n\nThis statement from the abstract does not hold  \"Unexpectedly, our analysis finds that depth pruning, despite its simplicity, consistently\noutperforms the more granular width pruning approaches in maintaining downstream task performance.\"\nJust look at Llama-3 numbers.\n\nAlso, I think the actual information provided is limited, and the analysis is simplistic. It would be great to see either the effects of fine-tuning or the performance of other methods (unstructured pruning/quantization). There are also long-range benchmarks; it would be interesting to see if performance depends on the task length.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Authors evaluate width and depth pruned model on variety of tasks and show, that pruned models do not perform well on GSM8k and other harder benchmarks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "See weaknesses.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jaKV85kbOL", "reviewer": "Reviewer_EncY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- “The chosen pruning methods, ShortGPT for depth pruning and FLAP for width pruning, fairly represent the state-of-the-art approaches in the literature for training-free structured pruning.” \n\n    Could the authors provide some supporting evidence for this? For example, some reference works that discuss ShortGPT and FLAP.\n\n    Similarly, claims like this should be supported with concrete evidence:  \n    “By selecting these representative methods, we ensure that our evaluation encompasses the primary paradigms of structured pruning while remaining computationally feasible for large-scale models.”  \n    “In contrast, pruning literature often focuses on a limited set of simpler benchmarks, primarily in the domain of common sense reasoning”\n\n- Smaller models may be prone to accuracy degradation during pruning. It would be good to results on models with sizes larger than 8b, e.g., llama-3.3-70b?  \n\n- The results are interesting, showing that pruning can lead to catastropic drop of performance in certain task, e.g., coding. However, it would be good to add some analysis or discussions on this observation.  \n\n- I am not sure about the focus of “Training-free” in the paper. In principle, pruning is a technique that often involves fine-tuning [1].  \n\n- It would be also to see if similar observations can be found in sparse pruning methods [2,3].  \n\n[1] Rethinking the value of network pruning. ICLR, 2019.  \n[2] SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. ICML 2023.  \n[3] A simple and effective pruning approach for LLMs. ICLR 2024.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper evaluates training-free structured pruning techniques for Large Language Models (LLMs), comparing depth and width pruning strategies. It finds that while some tasks handle pruning well, others degrade significantly, challenging previous studies that used selective benchmarks. The paper shows that depth pruning consistently works better than width pruning, highlighting the need for better pruning methods and more comprehensive evaluations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses above.", "broader_impact_concerns": "I do not have concerns regarding broader impacts of this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides a detailed analysis of training-free structural pruning methods for LLMs and highlights shortcomings in their evaluation within the literature.

Reviewers acknowledge that the analysis is generally accurate, though minor improvements could be made. Reviewer wMGw noted that the paper focuses only on small pruning rates, which may not yield significant practical gains, and suggested that additional pruning methods could be further explored. However, in my view, the paper does not aim to provide a large-scale study of existing pruning methods but rather to highlight current limitations in the field.

Overall, I believe the paper has the potential to improve current practices and guide future research.
Decision: Accept (poster)