2SSP: A Two-Stage Framework for Structured Pruning of LLMs
Authors: Fabrizio Sandri
Elia Cunegatti
Giovanni Iacca
Venue: Accepted by TMLR ()

Abstract: We propose a novel Two-Stage framework for Structured Pruning (2SSP) for pruning Large Language Models (LLMs), which combines two different strategies of pruning, namely Width and Depth Pruning. The first stage (Width Pruning) removes entire neurons, hence their corresponding rows and columns, aiming to preserve the connectivity among the pruned structures in the intermediate state of the Feed-Forward Networks in each Transformer block. This is done based on an importance score measuring the impact of each neuron over the output magnitude. The second stage (Depth Pruning), instead, removes entire Attention submodules. This is done by applying an iterative process that removes the Attention submodules with the minimum impact on a given metric of interest (in our case, perplexity). We also propose a novel mechanism to balance the sparsity rate of the two stages w.r.t. to the desired global sparsity. We test 2SSP on four LLM families and three sparsity rates (25%, 37.5%, and 50%), measuring the resulting perplexity over three language modeling datasets as well as the performance over six downstream tasks. Our method consistently outperforms five state-of-the-art competitors over three language modeling and six downstream tasks, with an up to two-order-of-magnitude gain in terms of pruning time.

Published in Transactions on Machine Learning Research (09/2025)
2SSP: A Two-Stage Framework for Structured Pruning of LLMs
Fabrizio Sandri"
sandri.fabr Batarrr@@mmll.. com
University of Trento, Italy
Elia Cunegatti"
ekaa.cunegaathikkmiin..
University of Trento, Itay
Giovanni Iacca
iomaanii.acco @wmitn..
University of Trento, Italy
Reviewed on OpenReview: https: // optpreveeewr..ew...mmmm net/ forum? id= Q67LxxB921
Abstract
We propose a novel Two-Stage framework for Structured Pruning (2SSP) for pruning Large Language Models (LLMs), which combines two different strategics of pruning, namely Width and Depth Pruning. The first stage (Width Pruning) removes entire neurons, hence their corresponding rows and columns, aiming to preserve the connectivity among the pruned structures in the intermediate state of the Feed-Forward Networks in each Transformer block. This is done based output magnitude. The second stage (Depth Pruning). instead, removes entire Attention submodules. This is done by applying an iterative process that removes the Attention with the minimum impact on a given metric of interest (in our case, perplexity). We also propose a novel mechanism to balance the sparsity rate of the two stages w.r.t. to the desired global sparsity. We test 2SSP on four LLM familics and three sparsity rates (25%, 37.5%, and 50%), measuring the resulting perplexity over three language modeling datasets as well as the performance over six downstream tasks. Our method consistently outperforms five state-of-the-art competitors over three language modeling and six downstream tasks, with an up to ow gain in terms of pruning time. The code is available at http:s//gghhbbbccc///bbbtaabll//
importance score measuring the impact of each neuron the
1 Introduction
The sheer size of the recent, billion-scale Large Language Models (LLMs) is one of the main reasons for their successful performance. However, it comes at the cost of computational budget in terms of required GPUs well as time for pre-training and inference, which in turn has serious economic and environmental impacts. Therefore, studying approaches to reduce the computational burdon of such models while minimizing their performance degradation has become a pressing matter.
One of the main approaches address this issue is through Network Pruning (Frantar & Alistarh 2023 Ma
et al. 2023), which mainly focuses reducing the size of pre-trained LLMs as well as their inference time. Among the several pruning methods available in the literature, reliable inference speed-ups (Kurtic et al. 2023 Ashkboos et al. 2024) have been achieved mainly through structured pruning, i.c., approaches that remove entire portions of the model. Different strategies to select which portions of the network to remove have been proposed, see Figure [ identifying Width pruning, which removes rows/columns and/or single layers, Depth Pruning (Blocks), which removes entire Transformer Blocks, and Depth Pruning (Submodules), which removes entire submodules (i.c., Attention submodules-in the following, referred as "Attentions",
"These authors contributed equally this work.

---

Published in Transactions on Machine Learning Research (09/2025)
Feed-Forward Networks (FFNs)) from the Transformer Blocks. width pruning has the main advantage of having a lower granularity level in the removal search space, which leads to a more refined identification of unimportant components of the model. On the other hand, the advantage of depth pruning lies in the lower computation time required to obtain the sparse structure as well as the larger inference speed-up that comes from the removal of entire blocks/submodules.
for
Width Pruning
Depth Pruning Blocks
Depth Pruning Submodiules
Width Depth Pruning (Ours) RoowsColbmnn Submochuie
Rows/Colunne
Figure 1:
The four calegories of structured pruning discussed. Each squared box indicates Transformer block, with the Attention on the left. and the FFN the right. The red color indicates the elements pruned in each category, ranging from wwws/cllume to entire blocks or submodules therein. Our method 2SSP is the first one to combine width and depth (submodule) pruning.
Contributions So far, the aforementioned categories of structured pruming have been investigated inde- pendently of one another, and their combination is currently a relatively unexplored research direction. In this paper, we propose a Two-Stage Framework for Structured Pruning (2SSP), a new structured pruning approach that, to our knowledge, is the first lo combine width and depth pruning, hence exploit ing the advantages of both approaches. The proposed 2SSP works in two stages. The first stage works at a lower granularity level, i.c., it uses width pruning to remove neurons from the intermediate state of FFNs basod on the output's magnitude. This is done while preserving the network connectivity, which is a critical measure for reducing performance degradation in sparse structures (Vysogorets & Kempe] 2023 Cunegatti et. al. 2024; lurada et al. 2024). Moreover, the first stage is applied only to the FFN parts of the model, for which pruming is known to be more difficult than Attention blocks (Siddiqui et al. 2024). The second stage complements the effect of the first one by iteratively applying depth praning on Attentions, based on the minimization of given performance metric (in our case, perplexity).
We tested our proposed method over four different LLM families, for three different sparsity rates (25%, 37,5%, and 50%) over three language modeling datasets, and six downstream tasks, showing that it consistently outperforms the five most recent state-of-the-art basclines while also having the best performance vs. pruning runtime trade-off. We also included experiments on higher spansity levels, up to 70%, to further validate the effectiveness of our proposed method w.r.t. the baselines.
2 Related Work
In this section, we discuss the different techniques proposed in the literature to remove parameters from pre-trained LLMs (i.e., for pruning after training). Methods for removing parameters from a model, while minimizing its performance degradations, can be roughly categorized into unstructured and structured pruning. The main difference between these two approaches that for a given sparsity and task, unstructured pruning allows for achieving better performance for the task, but the real speed-up is mainly theoretical (since the parameters are only masked, rather than practically removed); whereas, using structured pruning (the focus of this paper), the speed-up at inference/ training time is substantial (because of actual structure removals), but the performance degradation on the task may be higher.
2.1 Unstructured Pruning
Unstructured pruning aims to reduce the model's parameters by identifying critical weights (inside a weight matrix W), in any position, e.g., based on weight magnitude (Jaiswal el al. 2024), Hessian matrix aFrantar & Alistarh 2023), or activations (Sun et al. 2024 Zhang et al.) 2024a Farina et al. 2024). On top of these mothods, several techniques have been proposed either to select the best flock-wisc sparsity allocation, c.g., based on outliers (Yin et al.) 2024), activation alignment (Cunegatti et al. 2025), and optimal allocation
search (Li et al. 2024). further minimize the reconstruction error (Zhang et al. 2024b Xu et al. 2024
2

---

Published in Transactions on Machine Learning Research (09/2025)
2.2 Structured Pruning
Structured pruning removes entire parameter groups or structures from the LLM. Different categories of structured pruning algorithms have been proposed, which can be roughly categorized w.r.t. the gramularity of the structures they remove. The earliest approach is named Width Pruning, which encompasses methods that remove specific portions of weight matrices (such as rows and/or columns), or layers inside each Transformer block (such as a single Linear layer). The pruning decision can be based on gradient information (Ma et al. 2023 Fang et al. 2024), computational invariance (Ashkboos et al. 2024)) perturbative pruning (Dery et al. 2024), hardware-aware inference vs. performance trade-off (Kurtic et al. 2023). Fisher
information (van der Ouderaa al. 2024 or knowledge-disiillaiinn (Muralidharan al. 2024].
Recently, new paradigm of pruning, namely Depth Pruning, has emerged, shifting the pruning focus from weight matrices and layers to either entire Transformer blocks or entire Transformer submodules (Attention and/or FFN). The main idea behind this category of pruming algorithms is to assign an importance score to each block/ blooc//sbboode /submodule and then prune it. w.r.t. to its given score. These scores can be given either based on the change in hidden representation between blocks (similarity-bassed) or by computing each block's importance w.r.t. the model performance on given task (performanco-baasee
Among the Depth Pruning (Blocks), most of the existing methods are similarity-based (Samragh et al., 2023
2024 Kim et al. 2024 Song et al., 2024), with a few performance- based exceptions Motivated by the promising results of these algorithms, the more recent category of Depth Pruning (Submodules) further refined the granularity of the pruned structures,
Gromov et al.
(Ma et al. 2023 Song et
al. 2024).
proposing both performance-basse (Zhong ct al. 2025) and similarity-based (Siddiqui et al. 2024 Sicberling
et al. 2025
approaches.
3 Methods
We now introduce 2SSP, a novel pruning framework that combines two stages of structured pruming: a first stage (sD), which prunes at the level of entire neurons by removing rows and columns from the FFN within the Transformer blocks; and a second stage (s2) ), which performs submodule-basee depth pruning by removing entire Attentions.
Notations Given a Transformer-based LLM M, let B denote the number of identical blocks it is made of, and b represents a generic block. We define a subroddue of block b as either the Atlention or the FFN of that block. The Attention mechanism involves three linear projections, namely queries W quuery, keys Wkey, and values W and an output projection W Wout, which for a given input X outputs a final reprosentation computed as softmax ((X W query (X /) /z (XW value) Wout- For the most recent LLMs, such as the one tested in this paper, the FFN consists of three linear projections: gate W Wgate: up Wups and down Wdown; where the input is processed sequentially through the first two before the down projection. For a given input X, the FFN forward output is given by ( gXte) W down: where is the activation function.
The hidden dimension of the model is represented by dmodel, while dim denotes the intermediate dimension of the FFN, corresponding to the output dimension of the gate and up projections, and to the input dimension of the down projection. For any Linear layer within these submodules, let Wt E Rdoa xd denote the weight matrix of the layer. We use Deal to denote the calibration dataset used for pruning.
Neuron Pruning (s1) ) The method aims to prune the neurons of the intermediate representation within the Linear layers of the FFN that follows the Attention in each Transformer block. The rationale is that the FFN's hidden state generates an intermediate representation that can be compressed by removing entire neurons, obtaining a lower-dimensiona representation that preserves the most important features of the input sequence. We show that a high fraction of neurons in this representation created within the FFN is irrelevant, and thus can be pruned without significantly impacting performance. Moreover, our proposed neuron pruning scheme goes beyond simple segnitudo-basokl pruning by aiming to preserve the whole information path, which has been theoretically shown to be one of the aspects correlated with better performance in sparse models Hoang al. (2023); Cunegatti et al. (2024). Tn contrast to conventional approaches that simply eliminate
3

---

Published in Transactions on Machine Learning Research (09/2025)
Neuron Pruning Stage
Attention Pruning Stage
FPN
Dense FFN
ungnioigph.... Jsfel
Attentios
ppl
224a
Up
Duee
W
W up
W
step,
Astenriinn
Sparse FFN
VS E (1 Atn
Atteatiin
InPp
IPPr
FPP
MN
gaie
Wap
W
Atleako
Figure 2: Conceptual scheme of the proposed 2SSP method. The left part of the image indicates the LLM given as input to the algorithm; the central part indicates stage 81, which focuses on FFNs; the right part indicates stage which focuses on the Attention modules.
the highest magnitude weights, method specifically targets the pruning of pathways connecting the three distinet FFN components (gate W gatee up W ups and down W Woown projections).
Specifically, our algorithm prunes an equal number of neurons from each FFN, removing the neurons in the hidden state that have the lowest. impact, measured as the magnitude of their activated output. The magnitude calculated as the average C2 norm across the tokens in a set of calibration samples from a given calibration datasot Deal- The top-K neurons are then retained in the FFN of cach block.
Let the intermediate representation of the FFN in block b be denoted Zb RT''ddu, where T is the sequence length and dng is the dimension of the intermediate representation of the FEN in block b. For each neuron j, we compute an importance score 8j across the calibration datasct Deal:
Dewl
8j
1
(1)
D
( is the activation of the j-th neuron for the c-th sequence in the calibration dataset, and || where 26
1|2
denotes the La norm over the tokens in the calibration sequence.
Formally, let Win E X and W out E Rddodaa be the input and output projection matrices of the FFN, respectively, where dmodel is the model's hidden dimension. We define a binary mask m E {0, 1 that selects the top- neurons to preserve based on their importance scores 8j. The pruned weight matrices are then computed as:
Wi. = Win(m 1,:]
(2)
W out = Wonls m = 1]
(3)
To illustrate the mechanism, let us consider a simple FFN with two Linear layers, denoted as 4n and
bodd Removing a neuron from the hidden state of the FFN necessitates the removal of all the associated weights in both 4n and lout connected to that particular neuron. More precisely, given the weight matrices Win E Ru xdmoial and Wout Rddmoda respectively of 4m and lout, the pruning of a hidden state ncuron involves eliminating (in this order, to preserve work connectivity) the corresponding row in Wi, in and the associated column in W Wout.
Attention Pruning (s2) Pruning only the FFN parameters limits the effectiveness of the algorithm, as only a restricted fraction of the total number of parameters can be pruned, leaving all the Attention
Without loss of generality, we represent the standard phoee--rojections FFN (gate, up, down) two-layer representation where in combines the gate and up projections, and loue handles the down-projection.
4

---

Published in Transactions on Machine Learning Research (09/2025)
parameters intact. To address this limitation, we get inspiration from the observation derived in (Siddiqui et al. 2024), where it has been shown that Attentions can be removed to certain degree (~ 33%) with almost no performance degradation, while the FFNs are more sensitive to depth removal. Based on this, we propose a second pruning stage that, after removing a certain fraction of FFN neurons in the first stage, applics depth pruning to Attention We address this by adopting the submodules pruning mechanism proposed in (Zhong ct al. 2025 by removing only Attentions (and not also FFNs as the original mechanism) from the sparse network Mi obtained after the first stage. Specifically, we iteratively remove the Attentions leading to the lowest perplexity the calibration dataset until the target sparsity is reached.
Formally, let A = fai....,ay) ag} be the set of Attentions across all the B blocks of the Transformer. At cach step 4, we select the Attention module a' whose removal minimizes the perplexity on the calibration dataset:
a" = argmin (PL(M, \ a, Deal)
(4)
aEA.
where A is the set of the remaining Attention modules at step t, Mr \ a denotes the model at stop t with the Attention module a removed, and PPL represents the perplexity metric, calculated the exponential of the negated average log-likelihood over the calibration samples of Deat-
Balancing the Sparsity Rate As explained above, the proposed 2SSP algorithm works in two stages. Hence, given a target sparsity rate s, selecting the pruning rate allocated for each of the two stages is also required.
Through empirical analysis, we found out that the following equation provides a reliable metric for determining the optimal number of Attention modules to prune at any given sparsity rate (see Figure 6
N round
B
क Attn
(5)
where a = 1.3 (see Section 4.3) WFFN| represents the number of FFN parameters per block, and WAuta represents the number of Attention parameters per block. This equation captures two critical aspocts of this pruming process. First, it ensures that the number of pruned Attention parameters scales with increasing sparsity rates. Second, it adjusts the Attention pruning rate based on the relative sizes of the FFN and Attention modules. Specifically, when [Wees] is large, indicating that FFN parameters dominate the block structure, the equation reduces the proportion of Attention parameters to be pruned. This adaptive behavior helps maintain the balance between the number of Attentions and the number of FFN parameters pruned.
WALnl
4 Experiments
Setup We implement 2SSP using PyTorch (aaske et al. 2019) with HuggingFace's Transformer library (Wolf et al. 2020) for model and dataset management. All experiments are conducted on a cluster of four NVIDIA A30 GPUs (24GB memory each).
Models To lest the effectiveness of our approach across different LLM families, we tested the 7B-parameter variants of Mistral-v0.3 [Tiang et al. 2023), Llama-2 (Touvron et al. 2023 and Qwen-2.5 (Yang al. 2024
while using the 14B- 1aB-parmettrr version Phi-3 (Abdin et al. 2024).
Tasks and Datasets We employ perplexity over language modeling tasks as the primary evaluation metric.
given its established usage in assessing pruning algorithms (Frantar & Alistarh, 2023 Sun et al. 2024
Ashkboos et al.) 2024 Sieberling et al.) 2025) and the fact that it has been empirically proved to be a reliable metric for evaluating compressod models (Jin et al.] 2024). We measure perplexity across three datasets:
the full 2ixi2 dataset. (Merity et al., 2017), along with subsets of samples from C4 (Raffel et al.) 2020)
and FineWeb (Penedo al.) (024)) datasets, which are all popular benchmarks in the pruning literature (Sieberling et al. 2025). We also conduct downstream evaluations using LM Eval Harness (Gao el. al. 2024h.
PIt should be noted that, in our method, width pruning is applied only to FFNs because, while information paths can be computed in FFNs (taking as neurons the activations the hidden states); this not possible for Attentions, because of the scaled product between K, Q and V the different heads. This makes pruning entire neurons impractical in Attentions.
5

---

Published in Transactions on Machine Learning Research (09/2025)
including MMLU (Hendryeks et al. 2021
WinoGrande (WQ) (Sakaguchi et al. 2021 PiQA (Tata & Patel
2003), HellaSwag (HS) (Zellers et al. 2019), and ARC (easy and challenge, in the following referred to "ARC-e" and 'ARC-c", respectively) (Clark et al. 2018).
Table 1: Perplexity for 2SSP vs. the compared praming algorithms over three different sparsity rates across four L.Mss. The boldface and underline indicate, respectively, the best and second-best value per dataset. (excluding the dense baseline).
Mistral-vo.J 7B
LLama 7B
Qwen-2.5 2.5 7D
Phi-3 140
Spursily
Mathod
Wikefeext2
Pineweb-Edo
WikdFextz
Fineweb-EEd
WikIText2
Pineweb-Bed
Finewee-EEdu
0%
Demie
5:35
8:333
6:40
5:47
7:33
6:44
1835
13:85
1.7
4:31
8:51
6:41
ShoetGPPT
64.85
38
32.51
20.43
31.04
22.5
12.09
19.35
14:24
123. 70
124.02
120.00
Stiling Window
57.75
52:26
42:49
18.25
21.71
17.05
11.37
17.54
12.76
31.13
50.28
24.39
Block Pruner
10.53
13:45
10.95
19.00
12.96
11.01
11.57
17:45
12:35
9.76
15.10
9:85
ProPrese
9.35
12:86
10.57
11.74
17.25
12.34
9.53
Sli
11.81
13.00
11.52
12.57
12.72
17:40
(Ours)
0.24
9.25
15.07
10:43
ShortGP
41
Stiliix Window
054.31
225.83
21.78
30.85
57.5%
BlockFruner
ReoPros
27.00
21,31
24.10
25.06
23:29
19.95
21.522
19.00
21.47
20.22
18.13
17.04
22.05
22:17
28.06
29.40
21 19
21.76
19.91
15.02
21.51
19.59
17:15
15.18
ShtroPT
21.21
21.41
19.99
30:25
18.71
25.05
25.17
25.31
17.00
15.51
11.52
253P (Ours;
14.92
17:15
15.08
14.04
14.05
13.36
15.26
20.85
16.89
9.79
12.88
10.91
ShueGGPT
1.01
963.5
$80.31
238.38
187.46
160.56
9.30c
3,27
3.636
4.40c
2.79c
3,07c
Shiling Windoor
3.51e
1.50-1
3.51
2.18-
9.63>
213.96
137.08
142.71
1.00
1.66
9.077
BlockFrnner
$1.00
66.85
71.06
55.46
54.97
63.00
57.36
ExoPros
01.83
73.71
57.95
50.22
$6.95
SlroGPP
10.114
02.005
31.115
21.00
20.12
253P (Ours)
28.77
31:40
18.82
17.07
Baselines We compare 2SSP against a broad set of state-of-the-art structured pruning methods for LLMs. Since proposed approach works by combining width pruning with depth submodule pruning, designed the experimental setup to include baselines from the three different categories of pruning granularity mentioned
above:
Depth Pruning (Blocks): This category includes ShortGPT (Men et al. 2024 and Sliding Window
Cosine Similarity (Gromov et al. 2024 which perform pruning at the coarsest level by removing entire Transformer blocks.
2025) and EvoPress (Sieberling et al. 2025), which shift the pruning target from entire Transformer blocks to Attention and/or
Depth Pruning (Submodules): This category includes BlockPruner (Zhong
FFN submodules.
Width Pruning: This category includes SliceGPT (Ashkboos et al. 2024 which operates at the finest. granularity by pruning individual rows and columns within Transformer weight matrices.
For all the pruming algorithms, as done for 2SSP, we use a calibration dataset made of sequences of 2048 tokens each, taken from the C4 dataset. For 2SSP, we use 32 sample sequences for the first stage (see Section 4.2) and one sample for the second stage. As done in the original papers, we set the number of samples for ShortGPT and Sliding Window Cosine Similarity to 256. For BlockPruner and EvoPress, calibration sample. Since these two methods are based on removing submodules and evaluating the solutions in an itcrative manner (as our $2 stage), the number of samples highly influences the pruning runtime. For this reason, following the findings in (Sieberling et al. 2025), which demonstrate robust performance even with minimal calibration data, we set the calibration set size to one sample, as we have done for our second stage, aiming to obtain fair comparison. For SliceGPT, we utilize 256 samples based on the empirical analysis showing performance convergence beyond 128 samples reported in (Ashkboos al. 2024).
single
4.1 Experimental Results
In this section, we evaluate 2SSP w.r.t. the selected baselines in terms of performance over both language modeling and downstream tasks. We also include experiments about the runtime required by our approach to obtain the final sparse models. We show how our approach outperforms the competitors in terms of task performance and when confronting the pruning runtime vs. performance trade-off. We also evaluate the quality of the sparse models w.r.t. their inference speed-up w.r.t. the corresponding dense models.
6

---

Published in Transactions on Machine Learning Research (09/2025)
Numerical Results The first task we evaluate is language modeling over WikdText2, C4, and Fine at three different sparsity rates, namely 25%, as in (Men et al. 2024 Zhong et al. 2025 Ashkboos et al. 2024), 37.5% taken as intermediate value, and 50%, as in (Sieberling et al. 2025), avoiding higher values as it is established that structured pruning algorithms struggle to obtain reasonable performance above such sparsity rales.
Table ! reports all the numerical results in terms of perplexity across the tested dataset and sparsity rates. It is clear how our proposed approach outperforms the baselines in all test cases. Even more interesting is the robustness of our approach w.r.t. both models and sparsity rates. Table ! also highlights, for each setting, the second best (underline). Apart from the 50% case, where SliceGPT consistently ranks second, there are no baselines that outperform all the others after 2SSP. On the other hand, different baselines take the second place in different settings. In contrast, our approach provides stable results across all models, sparsity rates, and datasets tested.
We also tested if our approach could still outperform all the baselines at sparsity rales different from 25%, 37.5%, and 50%. For doing so, we computed the perplexity over WikiText2 for different sparsity rates ranging from zero to 70%. The results for Mistral-v0.3 7B and LLama-2 7B are shown in Figure 3 where the performance trend of 2SSP w.r.t. the baselines indicated in Table D is confirmed even across such broader set of sparsity rates.
In order to assess the performance of our approach not only in terms of perplexity, we considered six different downstream tasks. In this case, we tested 2SSP, well as the baselines, at 37.5% sparsity (the intermediate value among those con- sidered), considering the average zero-shot and few-shot accuracy as the main met- ric. In detail, we conduct 3-shot evalu- ations on the same benchmark datasets from Table 2 namely MMLU (Hendrycks
ShortGPT Sliding Window MV0 7B
BlockPruner Evopress
SliceGPT 2SSP (Ours)
Liama 2 7B
10
Perplexity
102
ee al. 2021
WinoGrande (WQ) (Sak-
0.0
0.3
0.4 Sparsity
077
0.1
0.2
0.5
0.5
aguchi et al. 2021). PiQA (Tata & Pa- tel| (2003), HellaSwag (HS) (Zellers et al.
Sparsily
2019), and ARC (easy and challenge, in
Figure 3: Perplexity for 2SSP vs. the compared pruning algo- rithins and the dense baseline when varying the sparsity rate on Llama-2 7B and Mistral-v0.3 7B.
the following referred to as "ARC-e" and
"ARC-c", respectively) (Clark et al., 2018). Table the numerical results of the zero-shot experimental setting, showing once again how 2SSP outperforms the tested baselines in most of the cases (and always on average across all tasks). Also in this case, no fixed second-best. exists among the competitors, while our approach consistently outperforms (on average across all tasks) all the other methods, regardless of the model. Table g demonstrates once more that 2SSP consistently outperforms the baseline models across most tasks and maintains superior average performance in the few-shot (3-shot) regime across all models.
Pruning Runtime For any pruning algorithm to be considered effective, along with the performance obtained by the generated sparse model, also the time required to obtain such sparse models is a critical metric. For this reason, we compare the performance (in terms of perplexity over WikiText2) vs. the pruning runtime of 2SSP and the baselines. Figure 4 shows the trade-off between these two metrics for the three sparsity rates tested in Table [ for the case LLama-2 7B (for the same analysis on the other models, please see the Appendix). The results clearly show how 2SSP is the best algorithm in terms of performance vs. pruning runtime trade-off. As expected, the fastest algorithms in terms of pruning runtime are the ones belonging to the Depth Pruming (Blocks) category, since they apply pruning in one-shot, the first stage, w.r.t. the similarity among blocks. On the other hand, the algorithms from the Depth Pruning (Submodules)
Given Qwen's 's architecture of blocks, instead of 37.5% we select sparsity rate of 39.29% represents the nearest. achievable rate through whole-block pruning (L.e., blocks removed).
7

---

Published in Transactions on Machine Learning Research (09/2025)
Table Zero-shot performance for 2SSP vs. the compared pruning models at 37.5% sparsity. The boldface and underline indicate, respoctively, the best and sccond-best value per datasct (excluding the dense baseline).
MMLL
PIQA
ARCA
ARC-
wQ
115
Model
Algorithm
Average
Dense
59.08
73.72
60.30
60.91
70.67
45.61
67
05
ShortCiPT
58.56
56.96
29.27
Sliding Window
25.54
57.22
50.19
29.41
34.51
26.55
Mistral v0.3 7D
BlockPruner
23.50
54.02
66.16
37.52
46.22
24.322
42.14
RvoPwes
SlieeCPP
23.15
61.98
65.34
36.99
42.668
21. 42
41.50
2S8P
24.49
63.
70.20
41.99
19.96
241.49
45.78
78:07
67.00
43.34
ShortGPT
32 25
80.54
59.63
33.54
41.33
28.50
42.63
Sliding Window
Llama: 2
BlockPruner
23.59
55.09
66.87
36.92
50.80
21.49
42.96
EvoPress
25.86
52.01
68.61
37.15
53.20
25.04
43.76
288P
27.91
61.33
70.29
42.78
155.93
27.30
47.61
46.20
ShortGPT
23.02
51.48
63.98
33.
50.55
21.74
41.27
Sliding Window
23.70
52.49
60.02
34.30
54.46
22.12
42:20
Qwen:
BlockPruner
36.97
EvoPress
24.21
55.17
67.30
36.90
69.776
27:13
45.00)
SlireOPP
22.91
57.70
65.94
34.32
48.02
20.48
41.55
70.29
52.45
26.70
Dease
67.81
75.17
61.01
64.04
$4.05
60.67
72.19
ShortGPT
Pla-3 14B
Sliding Window
BlockPrunez
25.17
27.90
50.04
61.40
52.77
68.12
25.60
42.00
022775
26.22
23.04
37.37
353.82
49.93
61.74
SlirOPP
27.21
66.61
71.16
45.45
54.34
20. 78
49.00
2SSP
51.85
68.82
74.97
$1.60
27
38.99
$8.92
Table 3: Few-shot performance (3-shot) for 2SSP the compared pruning models 37.5% sparsity. The boldface and underline indicate, respectively, the best and second-best value per dataset (excluding the dense bascline).
MMLU
ARC-
ARC-
WQ
Algoritbun
Shor:GPT
23.84
55.72
55.71
27.
32.20
27.3
37
Sliding Window
Mistral-vi. v0.3 7B
BloekPrumet
26.81
56.75
65.59
37.64
49.18
24.32
43.55
EvoPress
25.37
67.14
68.12
39.62
51..48
27.05
44.70
SleesGPT
27.12
258P
29.84
64.64
70.08
40.97
43.12
24.32
441.49
47.87
ShorGGPT
20.16
62.50
60.34
33.85
433.73
29.86
44.92
Sliding Window
39.27
58.64
58.32
32.
38.01
29.15
12.64
Lluma-2
BlockPraner
24.78
54.54
66.59
36.00
52.78
EvoPress
25.54
51.53
68.17
36.80
54.66
27.
44.03
SlieoGP
258P
28.38
00
70.08
41.00
57.11
29.78
47.001
Dense
74.01
08.79
ShuroGI'T
24.02
50:67
61.2
34.14
53.06
25.
12.29
Sliding Window
26.58
50.67
64.04
34.12
54.00
22.06
42.06
Qwen-2.5
BlockPrunet
EvoPress
26.63
52.40
66.97
36.63
60.27
27.05
45.04
SlieeGPT
27.00
55.25
63.82
32.99
10.52
288P
70.29
42.09
52.57
27.82
Dense
76.51
73.85
51.12
64.54
87.25
62.37
74.33
Phi-3
BlockPrmmet
Sliding Window
24.21
51.62
52.39
25.77
26.05
22.95
33.83
EvoPress
35.47
61.96
70.13
11.69
65.00
37.5 17.88
52.19
SIiesCPT
37.24
00.02
71.22
45.30
M6.14
30.03
61.70
8

---

Published in Transactions on Machine Learning Research (09/2025)
category the slowest ones, since they evaluate each possible submodule removal combination. The pruming runtime of SliceGPT lies somewhere in between and is mostly due to the PCA step employed for reducing the matrix dimension. To conclude, our approach, which is a combination of one-shot removal (sl and is able to achieve the best performance in terms of perplexity while requiring
submodule evaluation
limited pruning runtime.
25% sparsity
37.5% sparsily
50% sparsity
293P (Dund)
14
9/2-4
-0.20
57.50
Evcthees
10ea0 34
12/9
10
2342
190 I
7100
8kiig Window
82.7700
$3.54
200.10
Shent271
Figure 4: Pruning runtime (red, left side of the x-axis, log scale) vs. perplexity (green, right side of the x-axis, log scale) for 2SSP the compared pruning algorithms over LLama-2 7B pruned at three different sparsity rates.
Sparse Inference Speed-Up To complete the analysis of 2SSP and the baselines, study the inference runtime speed-up of the sparse models generated with the different pruning algorithms under examination over three different NVIDIA GPU models. Table reports the inference runtime of LLama-2 7B (including its dense version) computed as the average inference GPU time (in ms) over 10 runs on single sample of 2048 tokens. The speed-up of the structured pruning algorithms w.r.t. the dense model is evident and, as expected, increases with the sparsity rate. The advantage is maximum for the depth pruning approaches, while the width pruning methods achieve minimum speed-ups. As expected, 2SSP positions itself between the two categories.
Table 4: Inference time (ms) of pruned models on different GPUs.
Pruning Method
GPU Sparsity
ShortGPT Sliding Window
BlockPruner
Evopress
SliceGIT
2SSP
Dense
457.59
RTX
3090
25%
375.86
362.25
357.31
352.95
410.33
381.22
24GB
37.5%
318.26
314.61
301.84
297.03
330.92
316.66
50%
258.46
253.46
245.30
242.01
286.33
270.82
Dense
317.39
A30
25%
241.85
241.82
243.45
242.47
269.56
255.09
24GB
37.5%
203.07
203.66
204.80
204.49
228.48
223.41
50%
163.96
163.86
167.15
166.65
206.30
186.92
Dense
158. 17
A100 80GB
25%
120.49
120.48
120.21
121.43
129.17
127.53
37.5%
100.44
100.94
100.92
102.09
113.47
111.11
50%
81.22
81.39
81.40
83.09
99.17
92.59
4.2 Ablation Studies
In this section, we conduct different ablation studies to test the robustness of our proposed approach. In particular, we focus our analysis on the first stage (sl) and on the combination of depth pruning and width pruning, which are the distinctive aspects of our method.
Pruning Rows-Columns vs. Columns-Rows in s1 We conduct an ablation study to analyze the first stage of 2SSP, hence the neuron-based pruning approach. As discussed carlier, given the intermediate representation of an FFN, method removes entire neurons by pruning their corresponding rows and

---

Published in Transactions on Machine Learning Research (09/2025)
columns in the imput and output projection matrices, respectively. We explore "inverted" approach that, instead, prunes columns in the input projection matrix and rows in the output projection matrix, thereby preserving all neurons in the intermediate representation while reducing the dimensionality dmodel of the hidden state. The perplexity results in Table 5 show how the row-columns strategy employed by 2SSP the most reliable choice since inverting the order or pruning leads to worse results even by orders of magnit terms of perplexity. This can be explained by the fact that pruning rows first and then columns (but not the other way around) preserves the network connectivity.
Neuron Selection based on L1 vs. Le in s1 To evaluate the robustness of our neuron selection criterion, we study the impact of using L1 norm as an alternative magnitude metric for neuron selection, comparing it against the L2 norm used in Eq. [ In this variation, we modify the imporlance score šj, by replacing the L2 norm with the & norm. The perplexity results in Table T show that using &1 leads to worse performance across all models and sparsity rates compared to the main 2SSP version basod on L2.
Running s1 only vs. s1
+
We also separate the two stages included in 2SSP, to show the effectiveness of their combination. However, while it is possible to ablate on the first stage only, we exclude the case of having the second stage only since this stage's applicability is constrained by the model's architecture, hence it is not applicable to all models. In fact, the number of Attention parameters an LLM ranges from approximately 19% in Llama-2 to 33% in Mistral-v0.3, which limits the possibility of conducting this ablation across the three different sparsity rates chosen throughout the whole paper.
On the other hand, this does not hold for the first stage alone, for which results are reported in Table An interesting trend emerges in this case: while 2SSP (combining both stages) always outperforms the first stage only, it is visible how the gap between these two approaches in- creases while increasing the sparsity rate. It should be noted, however, that also the first stage alone cannot be applied to extreme sparsity rates such as > 70% due to the same limitations of the second stage only, in this case related to the number of FFN parameters.
Table 5: Numerical results of the ablation studies over Mistral-vO.3 7B and T.ama-2 7B for three different sparsity rates. The results correspond to the perplexity computed over the WikiText2 datasct.
Sparsity
Model
2SSP variant
25%
37.5%
50%
81 inverted +
257 .30
1.23c
5.80e
Li norm
9.41
17.65
48.52
Mistral-v0.3 7B
s1 only
9.51
15.58
30.16
9.24
14.92
23.77
s1 inverted
8.27e
1.02e*
5.51e"
s1 L1 norm
11.37
23.64
66.56
LLama-2 7B
s1 only
9.88
19.94
41.73
s1
9.19
18.16
32.85
4.3 Hyperparameter Tuning
2SSP relies on two main hyperparameters, namely used in Eq. 5 and the calibration set size |Deall- We detail their selection choices in this section. Note that for the calibration set size of the second stage, we set it to one sample, as done in (Sieberling et al. 2025), to balance pruning runtime vs. performance and provide a fair comparison, as also explained in Section Therefore, we limit our analysis to the calibration set size applied to the first stage.
Choice of a parameter A crucial aspect that makes our method effective an accurate balancing between the number of parameters pruned in the first and the second stages. To achieve this balance, we introduced Eq. 5 which, given desired sparsity rate for the whole model, controls the number of Attentions to prunc (N-TiA) This equation uses parameter to regulate the rale of Attention pruning w.r.t. the overall sparsity: lower values of result in a more gradual increase in the number of Attentions w.r.t. s, while higher values of a produce higher values of AAtta already at lower sparsity rates. Through empirical evaluation across multiple sparsity rates and models, we determine a = 1.5 to be near-optimal, as shown in Figure 5
10

---

Published in Transactions on Machine Learning Research (09/2025)
Mistral-vO. 7B
Liama-2
670
Sparsit Atn E51 4.6 020
Perpledly
255P
Both
031 2.47
122
Sparsity Rate
Sparsity Rata
Figure 6: Perplexity WikiText2 for various model sparsities (steps of 0.03) and Attention sparsities (pruned/total Attentions). The cells with orange (gray) border indicate the optimal of Attention modules to prune for each FPN sparsily, obtained by grid search (by 2SSP with Eq. 5 The red border indicates that the two methods find the same value. Eq. i a valid proxy for the optimal value. The white cells correspond to sparsity values where 2SSP cannot be applied due to the thcoretical maximum sparsity constraint.
Figure G shows, for Mistral-v0.3 7B and Llama-2 7B, the perplexity across various combinations of overall sparsity rates (s) and Attention sparsities. The figure highlights the importance of our proposed sparsity balancing approach. When pruning only FFNs (i.e., when the Attention sparsity zero), per- plexity deteriorates rapidly higher sparsity rates, particularly when approaching the theoretical max- imum sparsity of B- [-Eel [Weotal This limit is imposed by the total number of FFN parameters available for pruning. However, we observe that gradually incorporating Attention pruning alongside neuron pruning leads to improved perplexity scores. The figure also shows the robustness of our selection choice. Comparing the value of AAtt found by Eq. 5 setting = 1.5 with the optimal value found by an cxhaustive grid search spanning all possible choices of Allention sparsity, it can be seen there is in most
25%
37.5%
2SSP
Mistral-vO.3 7B
LLama-2 7B
100
35
Perplexity
30
20
0.5
1.0
1.5
2.0
0.5
1.0
1.5
20
Figure 5: Perplexity on WikiTText2 of models pruned
using 2SSP different sparsity rates when varying in Eq. 5
cases an almost exact match. is also noteworthy observe that the optimal number of Attention parameters to prune highly dependent the model; nevertheless, our approach reliably provides near-optimal choice.
Choice of Calibration Set Size Finally, we inves- tigate how the number of calibration samples for s1 alfects model performance while maintaining a fixed sequence length 2048 tokens. We conduct this analysis on Wikitext2 with both Mistral-v0.3 and Llama-2 50% sparsity, using perplexity to evaluate the performance of the final pruned models after ap- plying 2SSP with varying calibration set sizes. The results shown in Figure reveal that 2SSP achieves strong performance even with a limited number of calibration samples (at least 16). Given that larger calibration sets incrcase computational overhead due additional forward passes, we stick calibra- tion samples for the main experiments, mentioned earlier in Section 4.1
Mistral-v0.3
Liama-2 7B
288P
Perplexity
30
16
128
256
Calibration set size
Figure 7: Perplexity on WikiTexi2 of models pruned using 2SSP at 50% sparsity when varying the calibra- tion set size.
11

---

Published in Transactions on Machine Learning Research (09/2025)
5 Conclusions
In this paper, introduced 2SSP, new structured pruning algorithm that aims to combine Width Pruming for FFNs with Depth Pruning for Attentions. Our approach works in two stages by firstly pruming neurons in the intermediate state of FFNs, and then iteratively removing Attentions based on the model performance computed as perplexity. We tested 2SSP over three different families of LLMs, ranging from 7B to 14B at three different sparsity rates. The results demonstrate how our proposed algorithm consistently outperforms the state-of- -fate-tf baselines on both language modeling and downstream tasks. 2SSP achieves these results while requiring limited pruning runtime; which positions our method state-of-the-art over the performance pruming runtime trade-off. Finally, conducted in-depth ablation and tuning studies to demonstrate the robustness of our proposed method, focusing in particular on the neuron pruning mechanism of the first stage and the sparsity rate balancing between the two stages.
References
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL hhttss//arxiii http:///xxvvvoo////e//
www./ogg//////aoo.ooo.ooooe /a//abba/24400 2404. 14219
Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. SliceGPT: Compress Large Language Models by Deleting Rows and Columns. In International Conference on Learning Representations, 2024. URL
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge, 2018. URL
https://arxiv: http:////aarii...g/a///////// argg/bs//803 05457
Elia Cunegatti, Matteo Farina, Doina Bucur, and Giovanni Tacca. Understanding Sparse Neural Networks from their Topology via Multipartite Graph Representations. Transections on Machine Learning Research, 2024. ISSN 2835-8856. URL http://pp...peeeeeeeeeeeeeeeeeen fetf://o
Elia Cunegatti, Leonardo Lucio Custode, and Giovanni Iacca. Zeroth-Order Adaptive Neuron Alignment Based Pruning without Re-Training. In Sporsily in LLMs (SLLM): Deep Dive inlo Mixlure of Experts, Quantization, Hardware, and Inference, 2025. URL
Lucio Dery, Steven Kolawole, Jean-Françua Kagy, Virginia Smith, Graham Neubig, and Ameet Talwalkar. Everybody prune now: Structured pruning of LLMs with only forward passes, 2024. URL https: : //arxiv. wwrxiv.orrg/bs/240022
Gonglan Fang, Hongxu Yin, Saurav Muralidharan, Greg Heinrich, Jeff Pool, Jan Kautz, Pavlo Molchanov, and Xinchao Wang. MaskLLM: Learnable Seni--tructured Sparsity for Large Language Models. In Advances in Neural Information Processing Systems, 2024. URL httt:s//ppeeeeeeeee.ee////ordddr
Matteo Farina, Massimiliano Mancini, Elia Cunegatti, Gaowen Liu, Giovanni lacca, and Elisa Ricci. MULTI- FLOW: Shifting Towards Task-Agnostic Vision-Languag Pruning. In IEEE/CVF Conference on Computer Vision and Pollern Recognilion, pp. 16185 16185-16195, 16195, 2024. URL https:///oi. lesscooppterroccittyyyoo
10. 1119/CVPR52233.... 200222333.2222..... 2024.01532
Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in one- shot. In International Conference on Machine Learning, pp. 10323 10337. PMLR, 2023. URL https:
//dl. acm. org/
36188222
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPoli, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailcy Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for sew-shot language model evaluation, 2024.
URL httpss//zeendo. htccpp////eedddooorrrreeeea////s org/reecords/1280862
12

---

Published in Transactions on Machine Learning Research (09/2025)
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arx::2403. 17887, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. Mcasuring Massive Multitask Language Understanding. In International Conference on Learning Representations, 2021. URL http::////paaaeeeeeeeeee////raaa http://wweeetoormmmmaaaaaaaaaaaa
Duc N.M Hoang, Shiwei Liu, Radu Marculescu, and Zhangyang Wang. Revisiting Pruning at Initialization Through the Lens of Ramanujan Graph. In International Conference on Learning Representations, 2023.
URL httts::/ http://www..ee.ooom/////////////
Leonardo Lurada, Marco Ciccone, and Tatiana Tommasi. Finding Lottery Tickets in Vision Models via Data-driven Spectral Foresight Pruning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16142 16151, 2024.
Ajay Jaiswal, Shiwei Liu, Tianlong Chen, Zhangrang Wang, et al. The emergence of essential sparsity in large pre-trained models: The weights that matter. In Advances in Neural Information Processing Systems, volume 36, 2024. URL https::/opeenreview wwt.://ooooom///nnusWssssssss
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B, 2023. URL https://arxiv: http:////xxxw.oo/aabb/////////// htte/abb/2311...e 23110068220
Renren Jin, Jiangeun Du, Wuwei Huang, Wei Liu, Jian Luan, Bin Wang, and Deyi Xiong. A Comprehensive Evaluation of Quantization Strategies for Large Language Models. In Findings of the Association for Compu- tational Linguisties, pp. 12186-12215 12215, Bangkok, Thailand, 2024. Association for Computational Linguistics. URL htttps://aclanthologyy wwwp:004...///22...../ http://o/////...iii....... findings-acl. /266
doi: 10.18653/ v
Bo-Kyeong Kim, Geonmin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung Kyu Song. Shortened LLaMA: A Simple Depth Pruning for Large Language Models. In ICLR Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024. URL |atpss://ppenrevlew.nr net/
forumtid=13VGGuuddpp
Eldar Kurtic, Elias Frantar, and Dan Alistarh. ZipLM: Inference- Structured Pruning of Language Models. In Advances in Neural Information Processing Systems, 2023. URL httptp::///pppreeeee..... net/
forum frrum?dd=d831saa
Lujun Li, Peijie Dong, Zhenheng Tang, Xiang Liu, Qiang Wang, Wenhan Luo, Wei Xue, Qifeng Liu, Xiaowen Chu, and Yike Guo. Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models. In Advances in Neural Information Processing Systems, 2024. URL htttp:: http://pparrriieeeeeeeerrrrttrt net/formmmddd IgtrYVC9n4
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LLM-pruner: On the structural pruning of large language models. In Advances in Neural Information Processing Systems, volume 36, pp. 21702-21720, 2023. URL
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. ShortGPT: Layers in large language models are more redundant than you expect, 2024. URL
https://arxivv.rr//aee wwtpa///.rrg///////003 03853
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer Sentinel Mixture Models. In International Conference Learning Representations, 2017. URL http::///openneeeeeeeeee.ooooort
72udxe
Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Bhuminand Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shocybi, Bryan Catanzaro, Jan Kautz, and Paylo Molchanov. Compact Language Models via Pruming and Knowledge Distillation. In Advonces in Neural Information Processing Systems, 2024. URI https://openreviert httpp////pppaaeeeee.eeee/rrrrtrp hww.t//orrum//..-000nnnnn
13

---

Published in Transactions on Machine Learning Research (09/2025)
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Porformance Deep Learning Library, 2019. URL 01703
Guilherme Penedo, Hynek Kydlfčck, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The Wib Datasets: incanting the Web for the Finest Text Data at Scale. In Advances in Neural Information Processing Systems, 2024. URL https://openrevver htt/fforumii=musSCkn2aaa
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-61 67, 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: an adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99- 100, 2021. URL https:
//doi.org/10 10.
Mohammad Samragh, Mehrdad Farajtabar, Sachin Mchta, Raviteja Vemulapalli, Fartash Faghri, Devang Naik, Oncel Tuzel, and Mohammad Rastegari. Weight subcloning: direct initialization of transformers using larger pretrained ones, 2023. URL https://arxii: org/ argg/bs//20 wttp//12..0e..
Shoaib Ahmed Siddiqui, Xin Dong, Greg Heinrich, Thomas Breuel, Jan Kautz, David Krueger, and Pavlo Molchanov. A deeper look at depth pruning of LLMs. In ICML 2024 Workshop on Theoretical Foundations of Foundation Models, 2024. URL httts:://ppereviiee wttp://www..e/ooorrrro///////ooo
Oliver Sieberling, Denis Kuznedelev, Eldar Kurtie, and Dan Alistarh. EvoPress: Accurate Dynamic Model Compression via Evolutionary Search. In International Conference on Machine Learning, 2025. URL
https://openreviio wttp://ww...toooor///d///ooooo/.
Jiwon Song, Kyungscok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, and Jac-Joon Kim. SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks. In International Conference on Machine Learning, 2024. URL.
net/ met:/foruun?dd-fuX4yyypP
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A Simple and Effective Pruning Approach for Large Language Models. In International Conference Learning Representations, 2024. URL https:///openrevier wwtp://to///odddddooootooooe
Sandeep Tata and Jignesh M. Patel. PiQA: an algebra for querying protein data sets. In International Conference Scientific and Statistical Database Management, pp. 141-150, 2003. URL https://api
semanticscholar. www. Catg/CorpusID:.. 1545102
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Tlama 2: Open foundation and finc-tuned chat models, 2023. URL
Tycho F. A. van der Ouderaa, Markus Nagel, Mart Van Baalen, and Tijmen Blankevoort. The LLM Surgeon. In International Conference on Learning Representations, 2024. URL htttp:://ppnneriie.nnttt// net/forummm
id=DYIIRgwg21
Artem Vysogorets and Julia Kempe. Connectivity matters: Neural network pruning through the lens of effective sparsity. Journal of Machine Learning Research, 24(99): 2023.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Authony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. HuggingFace's Transformers: State-of-the-art Natural Language Processing, 2020. URL https:://ar:::: wwwpd//..rg//bb//////ee/e httt::://axxx..orr///////110 1910.037711
14

---

Published in Transactions on Machine Learning Research (09/2025)
Peng Xu, Wenqi Shao, Mengzhao Chen, Shitao Tang, Kaipeng Zhang, Peng Gao, Fengwei An, Yu Qiao, and Ping Luo. BESA: Pruning Large Language Models with Blockwise Parameter-Fificient Sparsity Allocation. In International Conference on Learning Representations, 2024. URL https://pperrriiew.eee/// new.:ewwwwee//ormmmm
id MLggGGTTEEJ]]
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Poi Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ron, Xuancheng Ren, Yang Pan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 Technical Report, 2024. URT https: htaps://aaxxiv org/ abs 1abss244 2212. 15115
Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yo Yu Hsieh, Yaqing Wang, Yiling Jia, Mykola Pechenizkiy, Yi Liang, Zhangyang Wang, and Shiwei Liu. Outlier Weighed Layerwise Sparsity (OWL)): A Missing Secret Sauce for Pruming LLMs to High Sparsity, 2024. URL hattps://openrevieee www/ferummnmeee het//foruniicrp0BrrPPPPR
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can Machine Really Finish Your Sentence? In Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, 2019. URL. https://aclamthologyy wwt.
Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. Plug-and-Play: An Efficient Post-training Pruning Method for Large Language Models. In International Conference Learning Representations, 2024a. URL
Yuxin Zhang, Lirui Zhao, Mingbao Lin, Sun Yunyun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, and Rongrong Ji. Dynamic Sparse No Training: Training-Fre Free Finc-tuning for Sparse LLMs. In International Conference on Learning Representations, 2024b. URL htttps://openreeiinn http:////ppppppeeeeeeooooo/oonae wwtp://ooorriiidnnnnnnidiiididid
Longguang Zhong, Fanqi Wan, Ruijun Chen, Xiaojun Quan, and Liangshi Li. BlockPruner: Fine-grained Pruning for Large Language Models. In Findings of the Association for Computational Linguistics, 2025. URL http:://aclanntaooogy.oooo org/2025. findings-ach .62.
15

---

Published in Transactions on Machine Learning Research (09/2025)
A Details on Pruning Rows-Columns vs. Columns-Rows in s1
For each dimension in the inputs' and outputs' hidden state, we compute importance scores across the calibration dataset. cat:
Datl
1
$%
in
l
(6)
DDaal
rall
out
1l% 1|
(7)
Deall
where e and (k) denote the k-th dimension of the input and output hidden states respectively. We define binary masks min: Mout E 10, 10model to select the top-K dimensions based these importance scores. The pruned weight matrices are computed as:
W Win min = 1]
(8)
W out = W Wont Mout 1,:]
(9)
We evaluate both pruning strategies on the WikiText2 dataset across various sparsity rates as shown Our experimental results demonstrate that pruming entire neurons consistently outperforms the dimension-based pruning approach in terms of perplexity.
Table 5]
B
Additional Results
B.1
Performance vs. Pruning Runtime
Here present additional results of performance (perplexity WikiTex(2) vs. pruning runtime of 2SSP and the baselines Mistral-v0.3 7B (Figure D Qwen-2.5 7B (Figure g and Phi-3 14B Figuure 10 Across all models and sparsity rates, our 2SSP consistently demonstrates superior performance compared to existing pruning methods.
25% sparsity
37.5% sparsity
50% sparsity
57
2.24
14:02
2:10
23.77
288P (0urs)
01
4114
450213
2000
4328 kb
Boddhure
20:3
27.75
$0.00 bi
234-31
94
2010.06
StonSPT
Figure 8: Pruning runtime (left side of the x-axis, log scale) vs. perplexity (right side of the x-axis, log scale) for 2SSP vs. the compared pruning algorithms over Mistral-vO.3 7B pruned at three different sparsity rates,
25% sparsity
37.5% spersily
50% sparsity
203P (Dund
1354
12.02
15V
12
30:47
Evethees
102.8:01
11.7
107.11 it
20177
100.00 34
5497
Bking Window
81:20000
12:0011
$2.00
50000
Short29
Figure 9: Pruning runtime (left side of the x-axis, log scale) vs. perplexity (right side of the x-axis, log scale)
for 2SSP the compared pruning algorithms over Qwen-2.5 7B pruned at three different sparsity rates.
16

---

Published in Transactions on Machine Learning Research (09/2025)
25% sparsity
37.5% sparsily
50% sparsily
203P (Dung)
1PH4 16
10.01
1008 34
12.04
161437
StoeOPT
19090
17948
12332 27
5443
Evstress
950 1t
ANG 84
1737206 43
1014463 56
Sking
StonSP7
Figure 10: Pruning runtime (left side of the x-axis, log scale) perplexity (right side of the x-axis, log scale) for 2SSP vs. the compared pruning algorithms over Phi-3 14B pruned at three different sparsity rates.
C Experimental Details
We implement our experiments using the HuggingFace Transformers library (Wolf et al. 2020 The models we use are sourced from the HuggingFace model hub, with their corresponding repositories listed in Table 6
Model
HuggingFace Repository
Mistral-v0.3 7B
mistralaa/MMittrll-BB-000
Llama-2 7B
meta-llamm//llame2--7-hh-
Qwen-2.5 2.5
Qwen/Qxen2.5-7B
Phi-3 14B
Table 6: Models and their corresponding HuggingFace repositories.
For the calibration dataset, we use the Colossal Clean Crawled Corpus (C4) (Raffel et al., 2020), specifically the Hugging Face repository allenai/c4. Due to the substantial size of the training split, we fetch only a subset of samples extracted from the first shard of the dataset. Since sequence lengths in C4 vary, we first concatenated all sequences into single corpus, tokenized it, and then divided it into segments of exactly 2048 tokens each.
For evaluation, we use the vikitext-2-raw-ve split available from the wikitext repository on HuggingFace. Additionally, for perplexity evaluation, we include two other datasets: a subset the validation split of C4 and samples from the sample-10BT subset of the HugginngFaceF///inneeeeeeee calibration dataset, the evaluation datasets are processed by concatenating their sequences, tokenizing the resulting corpus, and splitting it into sequences of 2048 tokens each. Specifically, Wikitext consists of 163 sequences, C4 contains 288 sequences, and Feb comprises 259 sequences.
repository. Similar to the
For reproducibility, we set fixed random seed of O across all randomization sources in our experiments by enabling deterministic mode for CUDA operations and setting the seed for Python's 's random module, NumPy, and PyTorch. This ensures consistent results across different runs of our experiments.
Qualitative Results
In this section, we present some qualitative examples obtained from pruned models by evaluating their performance on concrete generation tasks. Specifically, we examine the Mistral-v0.3 model pruned using 2SSP at three distinct sparsity rales: 25%, 37.5%, and 50%. The pruned models are tested on the content. generation task starting from two distinct prompts: "Who is Albert Einstein?" and "Explain the theory of relativity". We compare the quality of their generated responses at each sparsity rate, providing examples for each case. The results for the first prompt are shown in Table 7 while those for the second prompt are shown in Table K
The qualitative results confirm that, as expected, higher sparsity leads to a docrease the quality of generatod responscs. However, it is noteworthy that despite the reduced accuracy, 2SSP is capable of maintaining the structural coherence of the sentences, often preserving the flow of the narrative. This coherence is
17

---

Published in Transactions on Machine Learning Research (09/2025)
achieved at the cost of factual correctness, with pruned models occastonally introducing incorrect irrelevant information.
Table 7: Qualitative examples of text generated by Mistral-v0.3 7B pruned at different sparsity rates with 2SSP. The responses are based on the prompt "Who is Albert Einstein?".
Sparsity
Generated Sentences
Dense
Albert Einstein was a German-born theoretical physicist who developed the theory of relativity, one the two pillars of modern physics (alongside quantum mechanics). His work is also known for its influence on the philosophy of science. He is best known for his mass enesy equivalence formula E mc2 (which has been dubbed "the world's most famous equation"). He received the 1921 Nobel Prize in Physics for his services to theoretical physics, and especially for his discovery of the law of the photoelectric effect
25%
Albert Einstein was born on January 1, 1879, in Switzerland. He was physicist and won the Nobel Prize in 1991. He is best known for his theory of relativity.
Einstein was born in Switzerland, but he grew up in Germany. He studied at the University Zurich, where he earned his doctorate in physics. He then went on to work the Swiss Federal Institute of Technology Zurich. There, he worked on number of projects, including the development of the atomic bomb.
37.5% Albert Einstein was born on January 21, 1905, and died on April 28, 2021. He was a Nobel Nobel Prize winner and the winner of Nobel's Nobel Award for Nobel Science. Einstein was born in the United States, and he was the first Nobel win Nobel and Nobel awards. He won Nobel in Nobel, and his Nobel award Nobel. He also received Nobel Awards for Noble Science and Noble Noble Awards. Albert Finstein
50%
Albert Einstein was born in 1990 in the United States Albert Einstein well-known artist and sculptor. He is famous for his sculpting skills. He has been sculpting since 2010. Albert is sculptor by profession. ## Albert Einstein Biography He was born on the 17th of July, 1890, in the USA. Albert has been working in the sculpting industry since he was 10 years old. He started sculpting at the age of 15.
18

---

Published in Transactions on Machine Learning Research (09/202))
Table 8: Qualitative examples of text generated by Mistral-v0.3 7B pruned at different sparsity rates with 2SSP. The responses are based on the prompt "Explain the theory of relativily".
Sparsity
Generated Sentences
Dense
The theory of relativity is of the most important theories in the history of science. It was developed by Albert Einstein in 1905 and published in his paper "On the Electrodynamics of Moving Bodics." In this paper, Einstein proposed that the laws of physics are the same for all observers, regardless of their state of motion. This theory has revolutionized our understanding of the universe and has had a profound impact on many fields of science, including physics, astronomy, and cosmology. One of the key concepts in the theory of relativistic is the concept of spacetime.
25%
The theory of relativity is one of the most important thcorics in physics. It is based on the idea that the speed of light. the same for all observers, regardless of their distance from each other. This means that no matter how far apart two people are, they will still be able to cach other at the same time.
The theory first proposed by Albert Einstein in 1905, and it has since been used to explain many different phenomena. For example, it can explain why some objects appear to be moving faster than others, even though they are not actually moving at all.
37.5%
The thcory of relativity is one of the most important concepts in the world of physics. It is a concept that has been around since the beginning of the 20th century, and it has been used to explain how the universe works.
The theory of physics is a very important concept in physics. Physics field of physics that deals with the physics of the universe. deals with physics, physics, and physics. The physics of physics are very important because they help us to understand the universe better. Physics is very important in physics because it helps to better understand physics. For example, physics
50%
The theory of relativity in the United States of America (US) has been the topic of discussion for long time now. The US is the only country in the world that has reputation for being the best in the US.
The US is a country that has been in existence since the 1950s. It is the US that has the reputation of being one the most popular countries to live in. This because of the US' reputation for the US to be a country where the US can be found to the best country be in. In this article, we will discuss the US
19


Reviews:
[{"review_id": "hu874tdOVO", "reviewer": "Reviewer_sCDe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n- The paper considers both depth and width pruning together, applying them to different modules (attention and MLP) in Transformer-based LLMs. This joint perspective is relatively underexplored in the literature.\n- The proposed pruning approach is evaluated using both perplexity and downstream task performance, providing empirical evidence for its effectiveness.\n\n**Weaknesses**:\n- Limited novelty beyond prior work: The technical contributions are incremental. Both width pruning and depth/layer pruning have been widely explored in prior works such as SliceGPT [1] and LLM-Pruner [2]. The paper does not introduce fundamentally new mechanisms or insights beyond these.\n- Weak baseline comparisons: The paper lacks comparisons with several strong structured pruning baselines, such as: 1. OBD based methods: LLM-Surgeon [3] 2. Decomposition-based: MoDeGPT [4] 3. Layer pruning: SLEB [5] 4. Hypernetwork-based: DISP-LLM [6]\nThese methods represent their respective categories and have reported stronger performance on both perplexity and downstream tasks, making them valuablel baselines for a fair comparisons.\n- Insufficient justification for module-specific pruning: The rationale behind applying width pruning only to MLP layers and depth pruning to attention layers is unclear. The paper does not provide sufficient theoretical motivation or ablation studies to support this design.\n- No large-scale model validation: The experiments are limited to small and mid-sized LLMs. It remains unclear whether the proposed method can scale effectively to larger models (e.g., 30B+ parameters), which is important in applications.\n\nReferences\n\n[1] Saleh Ashkboos, Marco López-Croci, Matheus G. de Nascimento, Torsten Hoefler, James Hensman. SliceGPT: Compress Large Language Models by Deleting Rows and Columns. arXiv:2401.15024, 2024.\n\n[2] Ma, Xinyin, Gongfan Fang, and Xinchao Wang. \"Llm-pruner: On the structural pruning of large language models.\" Advances in neural information processing systems 36 (2023): 21702-21720.\n\n[3] van der Ouderaa, Tycho FA, et al. \"The llm surgeon.\" arXiv preprint arXiv:2312.17244 (2023).\n\n[4] Lin, Chi-Heng, et al. \"Modegpt: Modular decomposition for large language model compression.\" arXiv preprint arXiv:2408.09632 (2024).\n\n[5] Song, Jiwon, et al. \"Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks.\" arXiv preprint arXiv:2402.09025 (2024).\n\n[6] Gao, Shangqian, et al. \"Disp-llm: Dimension-independent structural pruning for large language models.\" Advances in Neural Information Processing Systems 37 (2024): 72219-72244.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates the joint application of width and depth pruning for structured compression of large language models (LLMs). While it claims to be the first to explore this combination, similar ideas have been studied in prior work under the frameworks of global and local sparsity allocation. The proposed method is empirically evaluated using both perplexity metrics and performance on downstream tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**1. Could the authors include comparisons with more state-of-the-art structured pruning methods?**\nA more comprehensive evaluation against recent strong baselines—such as score-based, decomposition-based, or hypernetwork-based approaches—would strengthen the empirical validity of the paper.\n\n**2. Could the authors provide more justification for the choice of specific width and depth pruning techniques, and their application to particular modules?**\nIt would be helpful to include theoretical motivations and/or thorough ablation studies to clarify why width pruning is applied to MLPs and depth pruning to attention layers.\n\n**3. Could the authors investigate the scalability of the proposed method to larger-scale LLMs?**\nDemonstrating the method’s effectiveness on models with tens of billions of parameters would provide a more comprehensive study on *Large* language models.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "S0ZoOXMnzR", "reviewer": "Reviewer_QcEY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nS1. Seems the first work to combine width and depth pruning of LLMs.\n\nS2. Extensive experimental setup, covering multiple LLM architectures, datasets, sparsity levels, and downstream tasks.\n\nS3. Significant improvements demonstrated in both pruning runtime and model performance relative to the selected baseline methods.\n\nWeaknesses:\n\nW1. Technical novelty appears incremental, primarily combining existing methods without introducing fundamentally new insights or theoretical advancements.\n\nW2. Motivation behind selecting specifically width and depth pruning methods is not clearly justified. Why not include other structured pruning methods or combinations?\n\nW3. Empirical analysis could be more robust, notably the absence of comparisons with state-of-the-art pruning methods and recent and popular LLM architectures (e.g., Llama 3 & 4, Qwen 3, etc).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a two-stage structured pruning framework (2SSP) combining width pruning (removing neurons from Feed-Forward Networks) and depth pruning (removing Attention submodules) for LLMs. The method balances pruning between these two stages using a novel sparsity allocation strategy based on perplexity. Experimental evaluations on four LLMs across various datasets and tasks demonstrate that 2SSP consistently outperforms several recent structured pruning methods, showing improved pruning runtime and model performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Clarify the motivation and theoretical reasoning behind selecting width and depth pruning specifically, rather than alternative structured pruning methods or combinations.\n\n2. Expand comparisons to include recent state-of-the-art structured pruning methods [1-4] and newer LLM architectures like Llama 3 & 4 and Qwen 3.\n\n[1] Dery, Lucio, et al. \"Everybody prune now: Structured pruning of llms with only forward passes.\" arXiv preprint arXiv:2402.05406 (2024).\n\n[2] Sengupta, Ayan, Siddhant Chaudhary, and Tanmoy Chakraborty. \"You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning.\" The Thirteenth International Conference on Learning Representations 2025.\n\n[3] Kong, Jun, et al. \"Sample-aware Adaptive Structured Pruning for Large Language Models.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 17. 2025.\n\n[4] Le, Qi, et al. \"Probe pruning: Accelerating LLMs through dynamic pruning via model-probing.\" The Thirteenth International Conference on Learning Representations 2025.\n\n3. I suggest that the authors should also evaluate the method on \"real\" large LLMs (e.g., 60B–70B parameters) where model compression is more practically necessary. Additionally, assess higher sparsity levels (>50%) to determine if 2SSP remains effective under more extreme compression settings. Sparsity < 50% does not produce much practical usage IMO.\n\n4. (minor) Consider conducting additional ablation studies with newer models to confirm robustness and effectiveness across contemporary architectures.", "broader_impact_concerns": "No significant ethical concerns arise directly from the presented pruning method.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mmC2CzxJen", "reviewer": "Reviewer_Ngzt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n- Novelty: Interesting idea to combine width and depth pruning strategies.\n- Solid performance:  Comprehensive evaluation across multiple models, datasets, and metrics with proper baseline comparisons.\n- Practical efficiency: Achieves strong performance with reasonable computational overhead, balancing one-shot removal (s1) with iterative evaluation (s2).\n- Extensive ablation study: Extensive ablation studies examining neuron selection criteria, pruning order, hyperparameter sensitivity, and calibration set size.\n- Nice presentation and clarity: Clear algorithmic description, simple importance metric, analytically motivated balancing rule.\n\n### Weaknesses:\n- Fairness: The authors utilize an ablation study to choose the size of the calibration set (Fig. 7). However, for the rest of the baselines, the authors use different calibration set sizes for different baselines. While authors did provide rationale for why they do so (For BlockPruner and EvoPress the authors trade away calibration data to hold wall-clock time in check, whereas for SliceGPT they trade in more data to maximise accuracy once its curve “converges beyond 128.”), It mixes two objectives (speed vs. quality) across baselines. It would be good to give both views: (a) accuracy-max settings (each method’s recommended sample count, even if slower, like for BlockPruner, the original setting is 256) and (b) matched-runtime settings (few samples for all). Readers can then see the speed/accuracy frontier.\n- Limited theoretical analysis: While the idea is interesting, the paper lacks theoretical justification for why combining width and depth pruning should work better than individual approaches. Adding this would provide interesting insights to the readers.\n- Limited sparsity exploration: As acknowledged by the authors themselves, due to limitations of the methodology, the maximal sparsity that can be achieved is 70%, which may not be satisfied in certain usage scenarios. Even so, the maximum presented sparsity in this paper is 50%, which is even lower.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces 2SSP, a two-stage structured pruning framework that combines width and depth pruning strategies for LLM. The key contributions are:\n- Two-stage pruning framework: Stage 1 (width pruning) removes whole neurons in the FFN based on L2 norm importance scores. Stage 2 (depth pruning) iteratively removes attention modules that minimize perplexity impact. \n- Sparsity balancing mechanism: A sparsity-balancing rule (Eq. 5) automatically allocates how much to prune in each stage.\n- Comprehensive evaluation: Testing across four LLM families (Mistral, Llama-2, Qwen-2.5, Phi-3) at multiple sparsity rates (25%, 37.5%, 50%) on both language modeling and downstream tasks.\n- Useful ablation study: Ablations justify the row-then-column order, the L2 importance metric, and the need to combine both stages﻿.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I do not find any changes that seem to be **critical** for this paper's acceptance. However, there are a few optional changes that can increase the chance of acceptance including: a) fairness of experiments about the calibration set size, b) some theoretical analysis, c) boarded sparsity exploration (as explained above), d) a broader impact statement section (as explained below).", "broader_impact_concerns": "The paper does not raise major ethical red flags. However, there may always be concerns about the fairness, security, and hallucination issues for a pruned LLM. While I do not think it is always necessary to add this broader impact statement, it would be welcomed if authors were willing to discuss it.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)