Zeroth-Order Adaptive Neuron Alignment Based Pruning without Re-Training
Authors: Elia Cunegatti
Leonardo Lucio Custode
Giovanni Iacca
Venue: Accepted by TMLR ()

Abstract: Network pruning focuses on algorithms that aim to reduce a given model's computational cost by removing a subset of its parameters while having minimal impact on performance. Throughout the last decade, the most widely used pruning paradigm has been pruning and re-training, which nowadays is inconvenient due to the vast amount of pre-trained models, which are, in any case, too expensive to re-train. In this paper, we exploit functional information from dense pre-trained models, i.e., their input activations, to obtain sparse models that maximize the activations' alignment with respect to their corresponding dense models. Hence, we propose \algname, a \emph{top-up} algorithm that can be used on top of any given pruning algorithm for LLMs, which modifies the block-wise and row-wise sparsity, exploiting information from both the dense model and its sparse version to maximize the \emph{neuron alignment} among activations. Different from existing methods, our approach adaptively selects the best hyperparameters for the block-wise and row-wise sparsity ratios w.r.t. the model and the desired sparsity, and requires \emph{no re-training}. We test our method over $\sim$300 test cases with four LLM families, three sparsity ratios, and ten language tasks (three language modeling and seven zero-shot datasets), showing how it consistently outperforms the latest state-of-the-art methods in terms of performance-runtime trade-off.

Published in Transactions on Machine Learning Research (10/2025)
Zeroth-Order Adaptive Neuron Alignment Based Pruning without Re-Training
Elia Cunegatti University of Trento, Italy
sio..cunegaatii@vmii...t
Leonardo Lucio Custode
leonardo.costtolee@gaaa
Independent Researcher
Giovanni Iacca University of Trento, Italy
giovanni. iwwpraniiicccaaaahaaaaaa www.ttccccccciccccccom
Reviewed on OpenReview: https: // optp//veeeer..eww..mmmm net/ forum? id= uPyNNaNFFF2
Abstract
Network pruning focuses on algorithms that aim to reduce a given model's computational cost by removing subset of its parameters while having minimal impact on performance. Throughout the last decade, the most widely used pruning paradigm has been pruning and re-training, which nowadays is inconvenient due to the vast amount of pre-trained models, which are, in any case, too expensive to re-train. In this paper, we exploit functional infor- mation from dense pre-trained models, i.e., their input activations, to oblain sparse models that maximize the activations' alignment with respect to their corresponding dense models. Hence, we propose NEURONAL, top-up algorithm that can be used on top of any given pruning algorithm for LLMs, which modifies the block-wise and row-wise sparsity, exploiting information from both the dense model and its sparse version to maximize the neuron align- ment among activations. Different from existing methods, our approach adaptively selects the best hyperparameters for the block-wise and row-wise sparsity ratios w.r.t. the model and the desired sparsity, and requires no re-truining. We test our method over <300 test cases with four LLM familics, three sparsity ratios, and ten language tasks (three language modeling and seven zero-shot datasets), showing how it consistently outperforms the latest state-of-the-ae methods in terms of performance-runting trade-off. The code is available at litpps: httplluu..oo//lllaaaaea
NeuroAL.
1 Introduction
In recent times, Large Language Models (LLMs) have shown incredible performance over several language tasks (Wei et al. 2022 Min al. 2023 Chang et al.) 2024).. However, their performance usually improves with their sizes (i.e., the number of trainable parameters), which in turn proportional to the computational cost of training and inference. One way to reduce this cost is network pruning, which studies algorithms that remove parameters while minimizing performance degradation. This approach, extensively studied on
Convolutional Neural Networks (CNNs) (Frankle & Carbin 2019 Lee et al. 2019 Wang et al.] 2020 Evei
et al. 2020), nowadays is mainly applied to pre-trained models (Touvron et. al. 2023a Jiang et. al. 2023
This shift has required change of paradigm in pruning techniques: in fact, while in CNNs the main paradigm is iterative pruning (with re-training) (Frankle & Carbin 2019), with pre-trained models (such as LLMs), it is not possible in most cases to perform a full re-training, because (1) training data are often not accessible, and (2) full re-training would be anyway too expensive. This calls for "exploiting" as much as possible the information contained in a pre-trained model to obtain a performant sparse version of it, using weights' information (Jaiswal al.] 2024), activations (Sun et al. 2023 2024). or reconstruction crror (Frantar & Alistarh 2023)), without the need for re-training. More recently, a new category of pruning algorithms, which may call top-up algorithms (i.e,, methods that be applied on top of given pruning algorithm for

---

Published in Transactions on Machine Learning Research (10/2025)
LLMs), has emerged, aiming further improving pruning performance. Such approaches can be divided
into two categories: those that minimize the reconstruction error (Guo et al. 2024 Xu et al. 2024 Zhang
et al.
2024
and those that impose non- non-uniform sparsity distribution, modifying the block-wiss sparsity (Yin
ct al.
2024
Lu et al. 2024 Li et al.) 2024p. The latter category is extremely effective on CNNs (Fanklle
et al.
2020 Su et al.
2020), while its application to LLMs has only recently emerged.
Contributions In this paper, we first analyze the major limitations of current top-up algorithms. To do so, we carefully analyze the state-of-the-arl top-up methods, highlighting their limitations in terms of sensitivity to hyperparameters, the required computational budget, and their block-importance metric. Leveraging this knowledge, we introduce NEURONAL, a novel top-up pruning algorithm that outperforms, in most cases, the previous mrate-of-the-art approaches over both Language Modeling datasets and Zero-Shot tasks, while providing hyperparameter adaptation and reduced runtime to obtain the non-uniform sparsity allocation.
The algorithm consists of two-step approach that re-distributes the biock-wise sparsity, Le., the spar- sity among Transformer blocks, and the row-wise sparsity, i.., the sparsity for each row of a given layer's s matrix, maximizing a metric which exploits information from both the dense and sparse model, namely the neuron alignmmet between dense and sparse activations. NEURONAL does not require the user to specify any hyperparametor-tuning as it automatically selects the most-performing values from a suitable set,, hence adapting to the underlying model and the target sparsity. Another advantage is that the neuron alignment only requires the com- putation of the activations of the dense and sparse models, which reduces the computation budget re- quired, compared to other top-up approaches.
DSA [NEURIPS 24]
BESA [ICLR 24]
10
Alpha (NEURIPS 24)
Adoptive
Runlimm
10a
Hyperparametar
NEUROA
(Orrs)
Depondent
Reconstruction
OWL IICML 24]
Error
10
Gradient- Based
10'
DSriOT [ICLR 24]
20
30
40
50
60
70
Perplexily
Figure 1: Perplexity vs. Runtime (seconds) trade-off among different top-up algorithms and our proposed NEURONAL based on LLama-1 7B with sparsity of 70%, evaluated on WikiTex12.
We test our approach three Language Modeling datasets and seven Zero-Shot tasks over four differ- ent LLM families from 7B to 70B parameters, to show its ability to outperform, in the majority of the cases, the most recent state-of-the-art techniques, in- cluding OWL (Yin et al. 2024h, DsNoT (Zhang et al.) 2024). and AlphaPruning (Lu et al. 2024) over three different high sparsity values (60%, 70%, and 80%) for a total of 276 test-cascs. The performance results clearly show that our proposed NEURONAL stands out as the most performing lop-up pruning algorithm, outperforming in most cases all baselines, and average by large margin, over both Language Modeling and, especially, Zero-Shot cases. Furthermore, the runtime analysis indicates that our proposed approach outperforms the state-of-the than the closest competitor over 70B models. To assess the robustness of our approach, we also conduct an in-depth sensitivity analysis.
baselines while also being more time efficient, being up to ~20 faster
2 Related Work
In this section, we provide a comprehensive discussion about network pruning applied to LLMs. We first introduce structured and unstructured network pruning; then, we focus on the latter, introducing the latest approaches proposed for improving sparse model performance.
2.1 Structured Network Pruning
Given a layer's weight matrix W E Raxm to sparsify, structured pruning removes either entire rows (n) or columns (m) (see the next section), aiming at speeding up both training and inference time. The first approach that applies structured pruning to LLMs has been proposed by Ma et al. (2023), and focuses on the dependency of Transformers, i.c., removes components of the networks while maximizing their original functionality. In (Kurtić et al., 2004)) a pruning mechanism has been devised to remove components with the worst balance between loss and runtime. Other structured pruning approaches have been proposed based
2

---

Published in Transactions on Machine Learning Research (10/2025)
combinatorial optimization (Meng et al., 2024). perturbative forward-pass only (Dery et al. 2024 and reduction of the embedding dimension through PCA AAhkboos et. al. 2023). Finally, in (Gromov et al. 2024) it has been found that the last Transformer blocks are redundant, hence they can be completely removed with minor performance drops. The reason behind this phenomenon lies in the similarity between the learnable representation of consecutive blocks, which turns out to increase when the block depth increases. While all these approaches can achieve valuable inference spced-ups, the performance of the resulting sparse models w.r.t. their dense counterparts can be matched only at low sparsity values, such as 20% in (Ma et al. 2023) or 30% in [Ashkboos et al. 2023). This somehow limits the applicability of these methods, since in the case of models with billions of parameters, one may need more aggressive pruning strategies to meet stringent hardware requirements.
2.2 Unstructured Network Pruning
Differently from structure pruming, unstructured pruming works by removing weights in scattered (i.e., non- structured) way. While in this setting the inference speed-up is limited (although techniques for reordering weights are available (Li et al.) 2019 Mishra et al.) 2021 Zhou et al. (02I)), the performance w.r.t. the dense model can be preserved also at high sparsity ratios (i.e,, above 50%), with the performance at lower sparsity being almost always completely preserved. The first approach of this kind has been proposed in (Frantar & Alistarh, (023), where weight pruning and reconstruction are combined based on the Hessian matrix. Even simple magnitude-baser approach turned out to perform well (Jaiswal et al. 2024), also when integrated with information on the neuron activations (Sun et al., 2023 Farina et al., 2024). These approaches compute score for each weight and then remove the ones with the lower scores for each layer, with a uniform sparsity across layers.
2.3 Top-Up Algorithms
To improve the performance of unstructured pruning, several top-up algorithms have been devised. These approaches can be categorized into two distinct groups: methods that minimize the reconstruction error, keeping the sparsity uniform for each block, and methods that modify the block-wise sparsity of the model, resulting in non-uniform sparsity distribution across blocks.
The first group firstly sparsifies the model using pruning algorithm and then, either dynamically (Zhang et al. 2024] or by backpropagation (Guo et al.] 2024), updates the pruning mask. The second group (lo which our method belongs) modifies the block- wise sparsity (obtained by a given pruming algorithm) based
either on activations" outliers (Yin et al. 2024). Empirical Spectral Distance (ESD) (Lu el al. 2024). or
allocation functions in a gradient-free manner (Li et al. 2024), while in BESA (Xu et al. 2024 gradient information used to set layer-wise sparsity using block-wiss reconstruction error.
The idea of simply redistributing the layer-wise sparsity is known to be extremely well-performing on Multi- Layer Perceptrons (MLPs) and Convolutional Neural Networks (CNNs). The first approach of this kind, based on the Erdös-Rémn (ER) model, has been proposed by Mocanu et al. (2018) for MLPs and then adjusted for CNNs in (Evci et al. 2020), while an empirical study about the effect of layer-wise pruning using different sparsity ratios has been done in (Liu et al.) 2021). Regarding Transformers (both for vision and text), the state-of-the-art algorithms (Frantar & Alistarh 2023 Sun et al.] 2023) have been devised to set the block-wise sparsity across the Transformer blocks in uniform way. Later on, OWT. (Yin et al. 2024 AlphaPruning (Lu et al. 2024), and DSA (Li et al. 2024) have been proposed to build upon scoring-basee pruning algorithrss, adjusting the block-wise sparsity in a non-uniform way. These approaches improve the performance of several pruning algorithms, e.g. (Frantar & Alistarh 2023 Sun et al.) (023),, especially at sparsity above 60%. On the same line, BESA (Xu et al. (024) allocates layer-wise sparsity across each block's layer using gradient information. Recently, modalityywis sparsity distribution has been investigated
in the case of multimodal tasks in (Farina el. al. 2024 He & Chen 2024).
3 Current limitations of top-up algorithms
We discuss below the three main limitations of the statc-of-tho-art approaches for redistribution of non- uniform sparsity in LLMs, namely 1) their need for hyperparameter tuning, 2) their large runtime, and 3) their block importance metric calculation (hence their block sparsity allocation).
3

---

Published in Transactions on Machine Learning Research (10/2025)
3.1
Need for Hyperparameter Tuning
We analyze the sensitivity of the hyperparameters used by OWL, namely À and M, and by AlphaPruning, namely e. Concerning OWL, the first hyperparameter is used to set how much the sparsity can vary across blocks (i.c., [s - A, + Al) while keeping the overall sparsity fixed as s. The second hyperparameter, M, defines the outliers' threshold: namely, for each block, the number of outliers is computed as the number of activations that are M times greater than the block's activations' mean. For AlphaPruning, instead, a hyperparameter called c is used and mamually tuned to set two tunable hyperparameters ($1; s2) that control the sparsity across blocks. We test the sensitivity of OWL and AlphaPruning to their hyperparameters, using three different sparsity ratios, two LLMs, and Wanda as the underlying pruning algorithm. Fig. 2 displays the perplexity on WikiText2 of the different hyperparameter settings obtained with OWL (first two rows) and AlphaPruning (last row); the gray square corresponds to the best a-posteriori hyperparameter selection. It can be seen that no single hyperparameter value achieves the best performance in all settings, which entails that careful tuning is required for these approaches to be effective.
3.2 Large Runtime
Sparsty Rato
D7
Another main limitation of some of the current ap- proaches for non-unine non-uniform distribution is their com- putational runtime. This holds mainly for BESA [uu et al., 2024) and DSA (Li et al., )024). On LLama-1 7B, the first approach, which relies on gradient information, requires ~ hours to find the best non-uniform distribution configuration. On the other hand, DSA uses an evolutionary approach to find the best combination of set of allocation func- tions, requiring 12 hours to find the best distribu- tiou]] This high runtime is due to the evaluation of each sparse model obtained by applying all the pos- sible combinations of sparsity allocation functions.
Phi-2.78
LLama-78 7B
Parplccty
Proploaily
10P
0656
0.00
6100
0.10
0076
0.000
0.00
100
Papplcity
220
3.3 Block Importance Metric
Almost all the top-up algorithms select the block- wise sparsity w.r.t. the block importance given by scoring criteria computed either from the dense model or from the evaluation of the sparse model. In the first case, the block importance is computed on the dense models on a block-wise view using ei- ther outlier information, as in OWL, or the ESD, as in AlphaPruning, without focusing on how the selected non-uniform sparsity, once applied, could change the block importance of the successive layers.
Figure 2: Perplexity for various hyperparameter set- lings of OWT (M, and AlphaPruning (e) using Phi-2 and LLama-1 7B for three sparsity ratios. The gray square corresponds to the hyperparameter values that lead to the best performance.
On the other hand DSA only uses the per- plexity of the sparse model as the evaluation metric. The only exception is BESA, which, similarly to NEURONAL, uses information from both the dense and the sparse model. However, it does so using gradient information, which leads to a high computational runtime. Also, DSnoT mininizes the reconstruction error us- ing information from both the dense and sparse models. However, it only updates each layer's binary mask without changing the sparsity dis- tribution across blocks.
Table 1: Properties of state-of-the-art top-up algorithmss vs. NEURONAL. The runtime is expressed as orders of magni- tude computed on LLama-1 7B. For the metrics, indi- cate whether they are computed over the dense (D) and/or sparse (S) models.
Top-Up
Hyper, Tuming
Runtime (s)
Metric
DS Zhong al.
2024
10a
Reconstruction Error (D.S)
BESA
Xu
2024
Recmetruction Loes
OWL Vin
2024
Outlies (1)
AppaPradia La
2014
10º
ESDs (2)
DSA La
2024
10'
Porpleslly (8)
NEURONAL (Ours)
10a
Nemon Aligament (D.S)
The runtime has been Laken from the original papers, where authors used 1 GPUs for BESA, and H800 GPUs for DSA.
4

---

Published in Transactions on Machine Learning Research (10/2025)
Table summarizes the different properties of the current top-up algorithms and shows how our proposed approach positions w.r.t. the previously proposed approaches.
4 Methodology
We now describe our proposed NEURONAL.
Preliminaries. Given dense model D, a pruning algorithm P, and larget sparsity 8 E [0, 1], unstructured pruning assigns saliency score y to each weight D and use top,(-) function to retain the (1 - [P] weights with the highest values. This function returns a binary mask M, such that the pruned model can be obtained as S = D @ M (where @ represents the Hadamard product). In the case of LLMs, since these models are composed of stacked Transformer blocks (each one denoted as Bi), i.e., sets of linear layers (each one defined by weight matrix W that implement the self-attention mechanism followed by an MLP, the mask is usually computed with uniform sparsity level across each layer e (Frantar & Alistarh
2023
M = top of (WL, W
Overview of NeuronAl. Our proposed NEURONAL based on two principles: (1) neuron alignment, which compares internal input activations rather than layer outputs as in (Frantar & Alistarh, 2023) (sce Appendix A) and (ii) adoplive sparsily reallocalion, which removes the need to manually tune the algorithm hyperparameters. NEURONAL relies neuron alignment to reassign sparsity in two stages: firstly across blocks (via vectors )B), and then across rows (via vectors s"). Both steps aim to minimize the discrepancy between the dense and sparse models' internal input activations, as formalized in Section 4.1 Importantly, the method requires no weight updates, gradients, or retraining. It simply reuses the scoring function V from the selected pruning method P and adjusts accordingly the sparsity ratios.
4.1 Neuron Alignment
The rationale behind NEURONAL is to adapt the sparsity distribution by comparing how well the sparse model preserves the internal representations of the dense one, concept that refer to neuron alignment. Rather than mcasuring the discrepancy in layer outputs, as in (Frantar & Alistarh, 2023 our method compares the impul aclivations to each projection matrix in a Transformer block.
Here, we formalize the proposed neuron alignment metric, which relies on the block and row-wis row-wise sparsity allocation. Given D and its sparse version D M, obtained via a pruning method P at sparsity ratio s, firstly perform forward passes over calibration set Cx and collect the activations Ap and As. Then, for each Transformer block B,, extract the input activations to all linear projection layers (i.e., the vectors a% such that the corresponding layer computes a w during the forward pass). This includes projections from the attention module (WQ, WK, Wv, Wo) and the MLP module (W gate. Wup Wdown.
To make the comparison consistent across layers, we normalize each input activation vector as a a For a given projection layer é, given an input x E Cx, we define its neuron alignment score as:
(x)))
(5( Σ
neuron (AB.
(2)
IA (z)
(1)
and then the neuron alignment across the whole model as:
NeuronAL(D,S,C))
ΣΣΣ ZECa BeB
neuronal (AIIOnn((A4.4 10 (5
(3)
The input activations a used calculating the neuron alignment either taken right after the LayerNorm (for Wq: up ) from the intermediate values within each sub-block (for Wo. and W down).
WK. Wv, gases and Wap).
5

---

Published in Transactions on Machine Learning Research (10/2025)
4.1.1 Block-wiss Sparsity Ratio
First, we optimize sparsity at the gramularity of transformer blocks. Given a set of candidato sparsity window teet, we generate multiple vectors s = {>x; S ' $ / J. that define how the larget sparsity 8 is distributed across blocks using linear schedule in [s - AE, s + Aa]. Formally, each sh a vector of length [B] where each element s (i) is the block-wise sparsity for the i-th block. Each vector s is computed using the following function:
-1-(2x
$h GetDist(s,Auiv) = -1- -22 Umax Umin
Umin
g + (1 (1- 5)
(4)
where [Ui,...,,e ug] is vector of block indices (with Vmin and Umax |B]), and ÿ mean 2
Umax Umiin For the block-wise case, since we employed linear schedule, we set 2 increase lincarly with the block index i (i.e., 2/2 = Vi E [1, [B1)), which yields s (i) = A + 2 $
V-Ymin
For cach vector A alignment score from Eq. 3 We then select the best configuration as:
apply pruning at the block level using the base pruning method P, and compute the
arg mrg NeuronAL (D,S,C.)
(5)
best
E 1 (EE
where D
+ i- D tops, (ve, W with + indicating a mask concatenation operation. This sparsity configuration is then applied the dense model to obtain Sg that has the best alignment across blocks. This step captures variations in block importance without modifying individual neurons.
4.1.2 Row-Wise Sparsity Ratio
We then redistribute the sparsity ratios across the rows of each projection matrix. For each layer & with weight matrix W Tx : we gemerate a set of row-wise sparsity vectors $ using linear schedule in [sB. Ak. SB, + Ax), with A E Aser Different from the block step, the initial sparsity of layer j is given from its parent block i from the sparsified model Sg. Importantly, each sx, defines a vector of row-wise sparsity values that is inversely proportional to the neuron alignment of each corresponding row, i.e., the rows that deviate more from the dense model are set as less sparse. Hence, the row-wise sparsity distribution is computed using the same function introduced in Eq. % = GetDist(sg, Agi viow) where voww is the vector of neuron alignment values associated with the rows of w For each vector in set Stot = (s,, prune rows using the original saliency scores and compute the neuron alignment score from Eq. 3 applied over Sg. We then select the configuration that minimizes the neuron alignment metric:
Sestt
arg min NeuronAL (D,S',Ca)
(6)
8 (AE
The full procedure, composed of block-wise and row-wise sparsity reallocation, shown in Fig. where align = True for NeuroAl(-) returns, for each layer of E B the row-wise alignment vector row betwoen D and Sg.
where S' Sg + f top (
w
4.2 Non-Uniform Block-Wise Sparsity Distribution
Given the sparsity values for each block in & and a selected sparsity window [s 1, 8 A). the non-uniform block-wise sparsity schedule redistributes the sparsity across blocks in a monotonically linear way (i.e., the sparsity of block is always larger than the sparsity of layer i 1, > 1) via Eq. 5 We select this sparsity schedule for three main reasons: (1) shown below such straight sparsity schedule is already able to achieve similar results w.r.t. statc-of-the art approaches, (2) to align with the latest discoverics in the literature of structured pruming where is consistently demonstrate how deeper blocks are redundant and can be removed with marginal performance degradation (Gromov et al. 2024 Men et al. 20244] Kim et al. 2024), and (3) to avoid having another sub-routine to select the best sparsity schedule, that, if linked with the block-wise and row-wise À selection, would have lead to have a combinatorial search space.

---

Published in Transactions on Machine Learning Research (10/2025)
Input: D, P.P.s. Cx. Asee // Block step
Function
GetBesttNeuronll (D, A Cx, P. v):
vblk II.... B]:
s" + 0; neur at tc; = =PD));
(s < GetBestNeeuoonA((.. A C2, P, vblk);
Sg - D +i top (s..) -
W
foreach À E A do
// Row step
SA - GetDist(s, v) // Eq.
vrow NeuronAL(D, SB. Cxi align=True)
S (D + i- + j-1 tops, a* ( W
( + GetBestNeuronAL( s,, s,0 Cx. P, vlon);
neur <- NeuronAL Cx) // Eq.
Sfinal + Do Do θ +
top
we)
if neurai < neur then
8xi
neur f neurali
end end return
Figure Left: Overall NEURONAL top-up pruning procedure. Right: GETBESTNNURONAL sub-routine used in both block- and row-selection stages.
We motivated the choice of linear schedule by testing three straightforward non-uniform sparsity schedules (namely linear, exponential, and logarithmic), which do not require any block scoring for sparsity allocation. Table 2 displays the improvement, w.r.t. uniform distribution (avcraged across three diffcrent Language Modeling datasets, namely Wiki- Text2, C4, PTB), achieved by the three spar- sity schedules using Wanda pruning algo- rithm with À 0.08. The results highlight how non-uniform sparsity schedules, without any block-based scoring, lead to performance improvement close to OWL's. Overall, the linear schedule turns out to be the most reliable one since it does not show oscillations in performance across the different sparsity ratios (while this happens for the logarithmic and exponential schodules).
Table 2: Performance improvement w.r.t. uniform distri-
bution averaged across three different datasets (Wiki C4, and PTB) using Wanda as pruning algorithm.
Schedule
Sparsity
Model
OWL
Exp
Log
Linear
Phi-2.7B
+3.4%
+2.4%
+7.8% 115.7%
+7.7%
60%
LLamna-1 7B
16.5%
13.4%
+ +1811%
Phi-2.7B
45.8%
+47.7%
+44.9%
+52.5%
70%
LLama-1 7B
+ 66.8%
+28.2%
+53.9%
+ +63.5%
Phi-2.7B
87.8%
89.3%
55..7%
82.8%
80%
LLama-1 7B
+ 81.5%
63.6%
-4.4%
08.1%
Mean
+ 50.3%
+39.1%
+28.9%
+ 18.8%
5 Experiments
We apply our proposed NEURONAL to different statc-of-tho-art pruning algorithms tailored for LLMs. Specif- ically, we test how it compares in terms of performance over Language Modeling datasets and Zero-Shot tasks w.r.t. the most recent. top-up algorithms for pruning. We also perform scalability and sensitivity analyses to show the effectiveness our NEURONAL.
5.1 Experimental Setup
Language Modeling Datasets To measure the models' perplexity on Language Modeling datasets, we use the following three datasets: (1) WikiTert2 (Merity et al. 2017), (2) Colossal Clean Common Crawl (C4)
[Raffel et al. 2020), and (3) Penn Treebank (PTB).
Zero-Shot Tasks To assess more thoroughly how the different pruning algorithms affect the models' capa-
bilities, we employ the following datasets: (1) Recognizing Textual Entailment (RTE) (Dagan et al. 2006
Bar Haim et al. 20066] Giampiccolo et al.) 2007 Bentivogli et al. 20099 (2) WinoGrande (Sakaguchi ct al.
2021), (3) BoolQ CClark el al.] 2019), (4) HellaSwag (Zellers al.] 2019 (5) ARC-e CClrr el al. 2018
(6) ARC-c (Clark et al. 2018). (7) OBQA (Mihaylov et al. 2018)
Models and Sparsity Since one of the distinctive features of NEURONAL is its adaptability w.r.t. sparsity and models, test. four different LLM families, namely LLama. 7B (both v1 and v2) (Touvron et. al.
7

---

Published in Transactions on Machine Learning Research (10/2025)
Table 3: Perplexity on the three Language Modeling datasets computed over five different LLMs for four different top-up algorithms (Uniform, DSnoT, OWL, and NEURONAL) on three pruning algorithms (Mag- nitude, MULTIFLOW, and Wanda) at 70% sparsity.
Phi-2.7B
LLamm-
LEamo-2 -
Mistrol-P
OPT-6.70
Algorithus
Top-Up
Wiklhbes2
C4
PTR
Wikillet2
C1
PTB
WikeTbee22
C4
PTB
WikiThst2
C4
PTB
WikifTeer2
C4
PTR
Disis
8.7
14.1
18.2
5.7
7.5
10.1
5.5
7.5
32.9
5.3
5.4
56.5
10.3
12.7
15.8
Unitern
764.5
384.4
983.9
2,5364
1,20c4
3.25ed
1.4265
1,10266
2,00e5
221.9
232.9
748.7
1.0001
5.09cl
6.5661
DSuTT
650.0
258.0
696.2
140k7
2.7706
4.9067
1.3188
2,9067
2.2008
152.7
180.9
560.2
0.10c3
3.03c3
4.36k3
Maguitule
OWL
419.5
362.7
358.3
1.30e1
0.5863
5.30d
3.30
1.34ml
3.36-5
111.7
131.2
515.5
1.5ted
6.8563
Aisharoniig
2.524
1.60ed
2.34d
424.5
231.5
1.7965
91.3
100.5
717.1
7.22cl
NEURONAL
261.7
Unifora
258.5
610.8
71.9
1,2963
9:3762
0.5062
1.25cl
341.1
DSu/
325.5
65.0
0,50v2
57.4
2.65v2
263.0
MULNIRL.W
280.0
AlshePrmning
0.5244
12.2
297.8
230.5
202.5
1.72
NEUROWAL
87.1
179.5
20.7
21:2
265.5
200.5
359.1
81.1
227.6
152.7
346.2
56.2
109.3
60.7
DSaIT
221.9
257.6
72.9
76.0
191.8
81.3
79.9
330.8
173.3
Wanda
DWL
131.7
136.2
280.7
20.6
27.3
61.2
20.5
36.6
3313.
41.0
53.8
3533.5
54.4
69.7
100.7
Arpharrming
4.22ed
3.05ed
2.15
26.9
31.1
77.4
30.0
37.
873.8
39.4
09.8
146
93.8
51.7
1919
NEURORAL
58.3
77.7
129.5
21.5
25.2
44.2
24.0
27.4
207.0
28.8
$3.7
282.0
172.0
84.0
152.7
2023a by, Phi-2, Mistral-7B (Jiang et al. 2023), and OPT-6.7B (Zhang et al. 2022).
To scale up the model
size, we also test LLama 13B (both v1 and v2) and LLama 70B (v2). In the paper, we mainly present results at 60%, 70%, and 80% sparsity, for fair comparisons with (Yin al. 2024 Lu et al. 2024). To assess the generalization to different sparsity ratios, we present the results on broader sparsity ratios (from 10% 80%) in Appendix D.7
Baselines As pruning algorithms, we test Magnitude, MULTIFLOW Farina et al. 2024). and Wanda (Snn
ct al. 2023 All are tested with four different top-up algorithms (besides ours): (1) Uniform distribution,
(2) DsnoTT (Zhang et al. 2024) (dynamic training-free uniform distribution with mask update), (3) OWL
(Yi et al. phaPruning (Lu al., 2024) (block-wiss training-fre free non-uniform distribution based on ESD 3 Further
2024) (block-wise wise training-fre free non-uniform distribution based on outliers scores) and (4) Al-
details on the setup are Appendix C
Table 4: Average accuracy the seven Zero-Shot tasks using Wanda as pruning algorithm.
Model
Sparsity
Tpp-Up
Phi-2.7B
LLama-1 7B
LLama-2 7B
Mistral-7B
OPT-6.7B
Dense
50.97
59.71
64.21
51.52
Uniform
50.39
49.97
51.03
46.24
DSnoT
49.36
49.12
48.93
50.51
45.75
60%
OWL
51.18
30.93
51.68
52.19
16.05
AiphaPruning
43.14
51.08
51.21
49.87
44.50
NECRONAL
51.41
51.99
46.10
Uniformm
36.00
34.37
36.85
36.31
DSnoTT
38.76
36.26
34.09
36.53
36.35
70%
OWL
11.20
43.31
10.57
38.17 77
38, 77
AlphaPruning
35.01
41.08
42.03
39.05
39.53
NECRONAL
41.87
44.57
43.10
41.56
Uniform
36.21
31.66
31.81
33, 48
33.34
D
32.11
80%
OWL
36.26
31.43
32.48
32.02
32.10
AlphaPruning
30..74
35.34
32.09
32. 29
32.16
NECRONAL
33.356
36.31
32.74
35.08
33.05
5.2
Experimental Evaluation
In this section, we show the numerical results of our proposed NEURONAL w...t. the baselines for Language Modeling and Zero-Shot tasks.
5.2.1
Language Modeling and Zero-shot Tasks
Concerning the Language Modeling datascts, the numorical results in terms of perplexity computed over the three Language Modeling datasets at 70% sparsity are shown in Table It can be seen how NEURONAL is
3BESA and DSA are included in these experiments due their large runtime. Testing them on combinations spansities and pruning algorithms is unfeasible with our GPU resources.
8

---

Published in Transactions on Machine Learning Research (10/2025)
able in almost all cases outperform all the other baselines by large margin. In does NEURONAL perform worse w.r.t. the uniform distribution. The only model for which NEURONAL is not the best top- up algorithm for all pruning algorithms is OPT. In all other cases, NEURONAL outperforms OWL and AlphaPruning for all models and pruning algorithms. The results at 60% and 80% sparsity shown Tables 17 in the Appendix confirm this trend.
As for the Zero-Shot tasks, the numerical results are shown in Tables the seven Zero-Shot tasks, while the results for each task are available in Tables 21 in the Appendix. Again, NEURONAL turns out to outperform in the majority of cases all the basclines. In 20 cases out 30 (w.r.t. the mean accuracy across all Lasks), NEURONAL the one that reaches the best performance, and in cases, the second best.
We display only the mean across
Table 5: Average accuracy on the seven Zero-Shot tasks using MULTIFLOW pruming algorithm.
Model
Spansity
Tpp-Up
Phi-2.7B
LLama-1 1 7B
LLama-2 7B
Mistral-7B
OPT-6.7B
Dense
64.35
50.97
59.71
64.21
51.52
Uniform
53.34
49.60
48.76
32.51
45.76
DSmo'T
48.90
49.57
49.62
$1.93
45.75
60%
OWL
53.04
49.70
49.12
34.70
46.15
AlphaPruning
34.97
45.01
NECRONAL
53.77
50.57
51.08
34.07
45.51
Uniform
38.40
38.51
36.98
32.76
33.91
DSmo'T
38.76
37.51
36.25
37.83
36.33
70%
OWL
42.46
43.23
40.60
32. 63
35. 69
AipbaPruning
31.81
41.07
11.50
31.95
NEURONAL
39.34
Uniform
31.32
32.10
63
31.98
32.32
DSuoT
32.20
31.45
32.25
32,
31.84
80%
OWL
35.99
33.25
32.19
32.04
$2.67
AiphaPruning
30.72
32.30
32.77
NEURONAL
32.32
36.63
5.3 Aggregate Comparison of Top-up up Strategies
already show that NEU- RONAL consistently outperforms the baselines, one of the main strengths of our proposed method lies in its ability to adapt to any sparsity, model, pruning algorithm, and tested datasot. For showing that, we design two metrics that aggregate, for each lested lop- up algorithm, the results sparsity ratios E {60, 70, 80)% by calculating the geometric mean of the results of different models (LLM), pruning methods (P) and datasets (D) at each sparsity. The two metrics are: (1) the accuracy retention, defined as:
While Tables
g 5
Table 6: Summary of accurocy retention (% w...t. dense) and perplerity degradation factor (x dense) at 60%, 70%, and 80% spansity for up to 7B models.
Acc.
retention (%) 1
PPL factor (x)
Top-Up
60 %
70% %
80 %
60 %
70 %
80 %
Uniform
79.6
61.0
55.0
9.0
62.2
786.7
DSnoT
81.8
61.6
54.7
7.0
96.9
746.3
OWL
80.7
66.2
55.2
6.1
35.5
392.4
AlphaPruning
76.2
64.3
54.7
8.3
66.0
505.7
NeuronAL
58.0
4.1
14.6
201.8
(m)).p(m)
1/(ICCM) (P))
AccRet(s)
=
II
II
AcCitenso(m)
(7)
mE PEP
and (2) the perplezity factor, defined as:
II IIII
PPL (m, d) PPLdense (m, d))
1/(ICCM) ) (P)))
PPLFac(s)
(8)
MECCM dED PEP
Magnitude omitted from these tables, due to space constraints. Results are available in Tables 22 24 in the A Appendix.

---

Published in Transactions on Machine Learning Research (10/2025)
The data used to compute the values shown in Table 6 are as follows. For the accuracy retention, used the average performance across wero-shot tasks and took the data from Tables 45 for the perplexily faclor we took the perplexity results from Table 3 and Tables [7 [7 reported in the Appendix.
The results clearly show that, when aggregating all the performance across pruning methods, datasets, and language models (in the order 7B scale), NEURONAL always achieves better performance w.r.t. the top-up basclines. These results, apart from providing additional evidence about the robustness of our approach, also highlight its adaptability
of the main strengths.
5.3.1 Scalability Study
To assess if the NEURONAL performance scales to bigger models, we apply it to LLama 13B (v1 and v2) and LLama 70B (v2) on both the Language Modeling datasets and Zero-Shot tasks. Also in this case, we provide,
Table 7: Summary of top-up accuracy retention (% w.r.t. dense) and perplezity degradation factor (x dense) at 70% and 80% sparsity for 13B and 70B models.
Acc. retention (%)
PPL factor +
Table 7 the aggregated results using the perplexity Jactor and accuracy relention defined in Sec- tion 5.3. These aggregated results clearly show that, On average, NEURONAL is in line with AlphaPruning in terms of perplexity, but out- performs in the Zero-Shot tasks, even when scaling up the LLM size, while roquiring 20x less time to obtain the non-uniform sparsity schedule. Looking at the detailed results in Table 8 9 it further becomes clearer that NEURONAL and AlphaPruning both provide advantages in this experimental sctting. When evaluating over Language Mod- eling datasets, see Table g our proposed approach provides better performance over the 13B models, while AlphaPruning over the 70B case. On the other hand, when evaluating over Zero-Shot tasks, the mean ac- curacy across the seven tasks shows that NEURONAL outperforms AlphaPruning in 10 out of 12 cases, see shown in Table (for the individual tasks' results see Appendix D.5). However, it worth mentioning that, in contrast with AlphaPruning, which in the 70B requires hours (~12 hours computing the non- non-uniform sparsity schedule, NEURONAL requires 35 minutes to obtain it, shown in Table 10
Top-Up
70%
80%
70%
80%
Uniform
67.9
52.2
13.1
219.4
DSno'T
67.5
51.5
6.3
229.6
OWL
77.8
53.7
6.0
82.8
AlphaPruning
78.8
60.4
3.8
35.9
NeuronAL
80.9
62.4
3.6
47.0
A100 80GB) for
Table 8: Perplexity of LLama models (vl 13B and v2 of both 13B and 70B) on the three Language Modeling datasets at 70% (top) and 80% sparsity (bottom).
Laaaa-1 13B
LLana-2 13B
LLama-2 2 70B
Algorithm
Upp-Up
WikiText-2
C4
PTB
WikiText-2
C4
PTB
WikiText-2
C4
PTB
Dense
5.09
6,80
28.11
1.88
6.73
48.82
3.32
5.71
20.76
Uniform
49.4
45.3
277.8
144.3
112.4
623.2
11.59
15.09
216.34
DSnoTT
46.2
48.9
240
45.8
54.2
611.5
9.19
13.22
86.17
MULTIPLOW
OWL
16.6
17.7
132.2
51.0
56.2
9.32
12.13
85.16
AlphaPruning
16.1
26.9
32.6
7.98
NEURONAL
13.8
15.6
101.7
20.1
23.1
318.9
9.03
11:75
120.84
Uniform
54.4
55.3
309.2
45.7
56.2
571.0
10.59
14.17
88.01
DSnoT
47.8
54.2
248.6
46.6
57.7
555.5
10.14
13.97
74.87
Wanda
OWL
16.3
18.9
147.6
18.0
21.8
315.1
9.01
11.92
5-1.96
AlphaPruning
14.6
17.3
126.7
13.2
18.8
271.1
7.82
10.78
39.21
NEURONAL
97.9
16.5
19.3
237.6
8.46
11.18
Uniform
3.71e8
1.70k3
3.59e3
4.48r3
2.41c3
5.21e8
266.20
175.07
956.49
DSno'T
2.56c3
1.94e3
1.67c3
180.76
698.30
MULTIFLOW
OWL
813.8
376.7
2.14e3
1.50e3
1.01e3
4.39e3
84.52
74.02
727.40
AlphaPruning
210.7
147.8
1.19
458.1
279.7
1.93e3
33.69
37.23
241.23
NEURONAL
126.8
123.7
901.6
1ible3
864
2 2.96e3
52.73
44.07
747.53
Uniform
3.48o3
1.96c3
3.5763
1.12c3
870.5
5.55c3
151.80
122.17
606.57
DSSoTT
4.37e4
2.44r4
3.22p4
4.44c3
3.96:3
4. 4.09x3
193.28
137.61
620.97
Wanda
368.1
1.93e3
248.0
204.2
2. 2.03e3
56.07
368.97
AlphaPruning
209.6
148.6
973.3
165.1
158.2
1.53e3
31.10
36.19
162.65
NEURONAL
156.4
132.4
1.47e3
185.7
143.1
1.25e3
17.72
40.28
212.02
10

---

Published in Transactions on Machine Learning Research (10/2025)
Table 9: Average accuracy on the seven Zero-Shot tasks using both Wanda and MULTIFLOW as pruning algorithm on larger-scale LLama models (13B, both v1 and v2, and 70B v2) at 70% (top) and 80% (bottom) sparsity.
LLama-1 13B
LLama-2 13B
LLama-2 70B
Sparsity Tpp-Up
Wanda
MULTIFLWW
MULTIFLLW
Wanda
MILTIFLOWW
Unilorm
37.54
32.73
56.73
55.02
DSpoT
39.93
37.31
36.96
56,09
57.36
70%
OWL
46.52
17.31
46.33
38,05
57.91
56.60
AlphaPruning
47.41
48.22
47.46
44,95
37.60 37.62
56.83
NEURONAL
49.34
50.79
48.28
47.82
58.98
56.95
Uniform
32.76
32.78
32.82
32.4
34.95
35.18
DSuoT
32.41
32.50
80%
OWL
32.25
32.26
32.49
32.36
39.21
38.18
AlphaPruning
38.07
38.56
36.59
32.28
42.00
41.63
NEURONAL
37.98
40.72
36.19
32.51
46.79
45.32
5.4
Efficiency Analysis
In this section, we analyze the efficiency of NEURONAL in terms of pruming runtime and inference speed-up.
5.4.1 Runtime vs. Perplexity
NEURONAL provides a good trade-off bet ween performance and runtime. In Table I we show for all baselines (here we also include BESA and DSA), the runtime in seconds required to obtain the non-uniform sparsity distribution for the given model (in this case, LLama-1 7B) as well as the performance computed as the perplexity over WikiText2. The results confirm how NEURONAL AL can achieve, best results in terms of perplexity while maintaining low computational budget. In terms of runtime, the only comparable methods performance. On the other hand, DSA is the closest in terms of perplexity to NEURONAL, while requiring four orders of magnitude more time to obtain the best sparsity distribution. Overall, the performance-runtee trade-off of NEURONAL improves when increasing the sparsity ratio.
out of 4 cases, the
DSno'T and OWL, compared to which, however, NEURONAL achieves better
Moreover, Table 10 we report the runtime of the baselines used as comparison in the paper over the
Table 10: Runtime (scconds) for obtaining the non- uniform sparsity allocation among different lop-up algorithms over LLama 13B and 70B 70% sparsity using Wanda.
13B and 70B LLama. modcls. As expected, the re- sults of the runtime scale up with the model's size. Overall, DsNoT and OWT. display the lowest runtime, while NEURONAL tends to be <~3.5x ~3.5x-4x slower, even when scaling the model size. On the other hand, NEU- RONAL drastically faster than AlphaPruning, from 6x the 7B case, up to ~20 in the 70B case. This runtime advantage indicates that NEURONAL not only provides better performance, which is especially evi- dent on larger models (see Tables but also achieves a better performance-runtime trade-off.
Top-up pruning algorithms
Metric
DaNoT
OWL
AlphaPruning NeuronAL
LLama-1 13B
129.61s
140.55e
5443.440
525.81s
I.aaa-22 13B
129.966
140.466
5379.42s
524.41s
LLama-2
417.92s
488.766
41890.97s
2110.20s
Table 11: Runtime (seconds) vs. perplexity trade-off comparison among different top-up algorithms over LLama-1 7B pruned at different sparsity ratios using Wanda.
Top-up pruning algorithms
Metric
Uniform
DsNoT
OWL
BESA
DSA
AlphaPruning
NeuronAL
Runtime
4.5s
73.3s
1.8
x10
4.3
x10
1479.4s
237.1s
Perplexity @ 65%
20.9
19.1
13.1
18.5
12.6
14.0
12.8
Perplexity @ 70%
85.1
72.9
24.6
42.6
22.6
26.9
20.7
Perplexity @ 75%
927.4
646.7
152.5
257.9
103.3
110.2
61.2
Perplexity @ 80%
5.22e3
3.71e3
986.5
2.21e3
736.81
768.4
302.8
11

---

Published in Transactions on Machine Learning Research (10/2025)
5.4.2 Inference Speed-up
Here, we evaluate the speed-up accel- eration of both dense and sparse mod- els obtained with NEURONAL, using the same inference pipeline based on DeepSparse [NeuralMagic 2021 ON- NXRuntime backends. The evaluation consists of the end-to-end token genera- tion and has been done over an Intel i9- 10980XE CPU using 18 cores. Table 12
Table 12: End-to-end inference speed-up (throughput gain) for Phi-2 and LLama-2 7B (v1) at different sparsity ratios. Through- put is measured in tokens/sec.
Model
Matric
Dense
20%
40%
Thronghpat (tokens/s)
0.1306
0.1307
0.1325
0.1488
0.1770
Phi-2
Speed-up f
1.00x
1,00x
1.01x
1.14x
1.35x
Thronghput n (tokens/s)
0.1498
0,1506
0.1354
0,1714
0.2309
LLama-2
Speed-up t
1.00x
1.01x
1.0xx
1.14x
1.51x
shows the throughput, measured in terms of tokens generated per second over Phi-2, using sequence length of 2024, and LLama-2 7B, using a sequence length of 1024, with four different sparsity ratios, along with the speed-up w.r.t. the generation time required by the dense model.
5.5 Ablation Studies
In this section, we provide a set of ablation studies that further show the robustness of our proposed method.
5.5.1 NeuronAL À Selection
In this section, we report an analysis of the ability of NEURONAL. to pick the best À parameters (i.e., the parameters for which the performance is the best one, hence the lowest value if computed over perplexity). To do this, we evaluate NEURONAL for all the À parameters (in the block-only setting, to simplify the visualization of results) over the three Language Modeling datasets. Fig. 4 reports the perplexity at 70% sparsity across different values of À (black dots connected by solid lincs), while the dots highlighted in orange indicate the perplexity achieved with the À value selected by NEURONAL. These results highlight how NEURONAL, in the majority of the cases, can pick the best value of X both with data knowledge, as in the C4 dataset (from which the calibration data is sampled), as well as on unseen datasets such as WikiText22 and PTB. Fig.s in the Appendix show the results at 60%-80% sparsity.
Figure 4: Perplexity over different values of at 70% sparsity. The orange dot indicates the value selected by NEURONAL using neuron aligument. The green dot indicates the value selected by NEURONAL using the reconstruction error rather than the neuron alignment. (see Section 5.5.2).
5.5.2 NeuronAL vs. Reconstruction Error and Distance Metrics
Our proposed approach is based on the activation alignment, defined in Eq.s 2 In Appendix A provide a detailed comparison of the formal difference between the definition of neuron alignment and that of Reconstruction Error (Rec-Error) (Franaar & Alistarh 2023 In this section, we demonstrate, empirically, how such a difference translates into an improved performance of neuron alignment w.r.t. Rec-Error. To do so, we conduct an ablation study of NEURON AL where the best for the block and row cases is selected based on the values that minimize the Rec-Erro rather than the neuron alignment.
Table 13 shows the results, in terms of perplexity neuron alignment and the setting based Rec-Error. As clearly visible, neuron alignment always leads to better performance. In addition, we highlight in Fig. [ the À value (for the block-only case) selected using Rec-Error. As displayed in the figure, while NEURONAL AL picks the À that minimizes the neuron aligmment, which in this case correlates with the lower (hence better) perplexity performance, Rec- Error does not provide the same ability.
WikiText2, of NEURON AL. for the standard case with
12

---

Published in Transactions on Machine Learning Research (10/2025)
Moreover, include in the same table an additional ablation study about the newron aligament formulation (Lqq 2). Specifically, we tested two different variations over Eq. 2 where the L2-norm is replaced by Cosine- Similarity (Cosine) and gence, respectively. In both cases, the procedure for the X selection remains the same for NEURONAL. The results in Table 13 show that the original NEURONAL formula- tion with the 1.2norm provides, in the majority of the cases (10 out of 12), the best results in terms of perplex- ity over W1k This further ablation confirms the effectiveness of the proposed formulation.
Table 13: Perplexity of NEURONAL vs. Rec- Error and different distance metrics (Cosine Sim- ilarity and KI-Divergence) across three sparsity ratios and four LLMs.
(KT) diver-
Model
Tpp-Up
60%
70%
80%
Ree-Erroe
10.78
79.40
3.29c3
LLama-1 7B
NEURONAL w. Cosine
9.66
25.01
536.33
NEURONAL w. KL
10:12
6.18
NEURONAL
9.59
21.53
302.82
Hec-Error
10.96i
67.25
2.92<3
LLama-2 7B
NEURONAL w. Cosine
9.56
27. 913
448.76
NLUROOAL w. KL
10.36
66.93
1.54e3
NEURONAL
9.38
24.05
557.17
Ree-Error
24.21
202.73
1.042e4
5.5.3 Sensitivity to Calibration Data
Phi-2
NBURONAL w.
24.57
106.18
3.03<3
Since NEURONAL. works uses a calibration set, we test its sensitivity on the calibration settings in terms of seed, data source, and size. The results in Table 14 show the mean and standard deviation of perplexity at 70% spar- sity over the three Language Modeling datasets for dif- ferent seeds and when using different datasets to extract the calibration data. Furthermore, the results in Fig. 5 report the perplexity at 70% sparsity over the three Language Modeling datasets when changing the number of calibration samples (using C4 as calibration source and 0 as seed).
NEURONAL KL
24.99
2:7. 23
1.92el
NEURONAL
25.26
88.32
2.19e3
Rey-Erroe
10.95
56.07
200.73
Mistral-7B
NEURONAL w. Cosine
10.78
52..22
214.54
NEURONAL w. KL
10.97
55.01
311.45
NEURONAL
9.94
28.78
219.47
Perplexity (avg. + std. dev.) achieved when using NEURONAL AL with different calibration data seeds (top), and when using different datasets as the source for the calibration data (bottom) on the three Language Modeling datasets at 70% sparsity.
Table 14:
Calibration source(s), sced(s)
Model
Dataset
Phi-2.78
LLamm-1 7B
LLamm-2
Mistnui-B
OPT-6.7B
Wikilest2
86.7
22.7
31.3
105.5 GI.7
Calibration: (C4)
C4
78.8 = 1.7
24.3 à: 1.6
27.3 1 0.8
15.3 2.4
6L9 16.6
Scedi
PTD
108.5 = 11.8
46.8 1 4.6
201.0 1 5,3
2393 2 6.1
1224 1: 59.7
WikTTett2
91.8 4.7
26.7 + 0.9
25.5 + 41
30.1 + 53
125.5 70.5
Cullbratton: (WarTer2, CA. PTB)
C4
8:1
120.0
Sceds: (0)
PTB
380. 10.6
41.7
213.6 19.3
289.5
163.2 90.7
Overall, it can be seen that our method is fairly robust w.r.t. the calibration source, seeds, and number of samples. The only exception is the OPT model, where the sensitivity to the calibration data turns out to be higher. This sensitivity analysis reveals a possible explanation for the results presented in Section 5.2] where our proposed approach consistently outperformed the basclines over all models and tasks, except for the OPT family.
Phi-2.7B
Mistral-7B
Llama-1
Llama
OPT-6.7B
WikiTexth
C4
PTB
500
Perpleccty
200
400
300
1:0
300
200
50
100
100
Number of Samples ((C.)
Number of Samples ((C.)
Number Samples (CC.)
Figure Perplexity over different values of [Cxl (size of the calibration data) when using NEURONAL on the three Language Modeling datasets at 70% sparsity.
13

---

Published in Transactions on Machine Learning Research (10/2025)
6 Conclusion and Limitations
In this paper, we have proposed NEURONAL, between sparse and dense activations. The main novelty of our approach is that it exploits information from both the dense and the sparse models while also being adaplive since it is designed to automatically select the best hyperparameters for a given model, pruning algorithm, and target sparsity. Throughout extensive experiments, we showed how our approach outperforms, in most cases, the latest state-of-tho-art methods both on Language Modeling datasets and Zero-Shor tasks, with different LLM families and sparsity ratios, while requiring minimal time overhead w.r.t. the base pruning algorithm applied. We also included extensive sensitivity analysis to show the robustness of our approach to the calibration data, and its capability to select X, as well as runtime comparison with the most recent competitors.
approach to pruning LLMs based on the neuron alignment
The present version NEURON AL. has two main limitations. The first one derives from the algorithm setup. In fact, NEURONAL requires selecting the size of A and Cx, both affecting the computational cost of the forward slep. In order to alleviale this limitation and provide a fair comparison w.r.t. the base pruning methods and the top-up baselines, we set |Cx| 8 (the closest power of to [CI/[***), so that NEURONAL incurs only two extra forward passos compared to the basc methods, and one more pass than OWL (Yin et al., 2024h and AlphaPruning (Lu et al., 2024). In Appendix B analyze the NEURONAL complexity w.r.l. A and Cx. The second limitation (which should be noted, however, to be common to all top-up pruming algsrithrms). is the inability to make use of optimized semi-structured sparsity inference implementations (e.g, the NVIDIA N:M sparsity (Pool, 2020 In fact, for a given sparsity, NEURONAL, as well as OWL (Yin al., 2024), AlphaPruning (Lu al. 2024), and DSA (Li et al. 2024), produce customized sparsity constraints for each layer in block. Therefore, these semi-structured sparsity implementations cannot be employed they often require continuity in the sparsity (N:M) across all matrices in the model.
14

---

Published in Transactions on Machine Learning Research (10/2025)
References
Salch Ashkboos, Maximilian I. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. SliceGPT: Compress Large Language Models by Deleting Rows and Columns. In International Conference on Learning Represenlations, 2023.
Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa. Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second PASCAL recognising textual entailment challenge, 2006.
Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The fifth PASCAL recognizing textual entailment challenge. In Text Analysis Conference, 2009.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang. Kaijic Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1-45, 2024.
Christopher Clark, Kenton Lee, Ming-Was Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In Conference of the North American Chapter of the Association for Compulational Linguisties: Human Language Tech- mologies, Volume 1 (Long and Short Papers), pp. 2924 2--699366 2019.
Peter Clark, Isaac Cowhey, Oren Etxioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have Solved Question Answering? Try ARC, the A12 Reasoning Challenge. arXiv 2018.
preprint ar Xic::8003005577
Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment chal- lenge. In Machine learning challenges. Evalualing prediclive uncertainly, Visual object classification, and Recognizing tertual entailment, pp. 177-190. Springer, 2006.
Lucio Dery, Steven Kolawole, Jean-Francois Kagey, Virginia Smith, Graham Neubig, and Ameet Tal- walkar. Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes. arXiv preprint arXiv:2402.054 2024.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943 22-222222222 PMI.R, 2020.
Matteo Farina, Massimiliano Mancini, Flia Cunegatti, Gaowen Liu, Giovanni Tacca, and Elisa Ricci. MUL-- TIFLOW: Shifting Towards Yask-Agnostic Vision-Language Pruning. In IEEE/CVE Conference on Com- puter Vision and Pattern Recognition, pp. 16185-16195, 2024.
Jonathan Frankle and Michael Carbin. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. In International Conference on Learning Representations, 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Pruning Neural Networks at Initialization: Why Are We Missing the Mark? In International Conference on Learning Representations, 2020.
Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in onc-shot. In International Conference on Machine Learning, pp. 10323 10337. PMLR, 2023.
Danilo Giampiccolo, Bernardo Maguini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pp. 1 1-9, Association for Computational Linguisties, 2007.
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. The unrea- sonable ineffectiveness of the deeper layers. arXiv preprint 2rXiv:2408.17887 2024.
Song Guo, Fan Wu, Lei Zhang, Xiawu Zheng, Shengehuan Zhang, Fei Chao, Yiyu Shi, and Rongrong Ji. FBFT: Effective and Block-Wise Fine-Tuning for Sparse LLMss Xix preprint ar arXio:2402.0 12419 2024.
15

---

Published in Transactions on Machine Learning Research (10/2025)
Shwai He and Tianlong Chen. RESSA: Repair Sparse Vision-Language Models via Sparse Cross-Modale Adaptation. arviv preprint or 0et:22444.2244. 2024.
Ajay Jaiswal, Shiwei Liu, Tianlong Chen, Zhangyang Wang, et al. The emergence of essential sparsity in large pre-trained models: The weights that matter. Advances in Neural Information Processing Systems, 36, 2024.
Albert Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B. Xrviv preprint ar Xix:223100 060062 2023.
Bo-Kyeons Kim, Geonmin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung- Kyu Song. Shortened LLaMA: Simple Depth Pruning for Large Language Models. ICLR Workshop on Mathematical and Empirical Underslanding of Foundation Models (ME-FoMo), 2024.
Eldar Kurtić, Elias Frantar, and Dan Alistarh. ZipLM: Inference- Structured Pruning of Language Models. Advances in Neural Information Processing Systems, 36, 2024.
Namhoon Lec, Thalaiyasingam Ajanthan, and Philip Torr. SNIP: Single-Shot Network Pruning based on Connection Sensitivily. In International Conference on Learning Representations, 2019.
Jiajia Li, Bora Uçar, Umit V Çatalyürek, Jimeng Sun, Kevin Barker, and Richard Vuduc. Efficient and effective sparse tensor reordering. In ACM International Conference on Supercomputing, pp. 227-237, 2019.
Lujun Li, Peijic Dong, Zhenheng Tang, Xiang Liu, Qiang Wang, Wenhan Luo, Wei Xue, Qifeng Liu, Xiaowen Chu, and Yike Guo. Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models. In Advances in Neural Information Processing Systems, 2024.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, and Mykola Pechenizkiy. The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training. In International Conference on Learning Representations, 2021.
Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W. Mahoney, and Yaoqing Yang. Al- phaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models. In Advances in Neural Information Processing Systems, 2024.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LLM-pruner: On the structural pruning of large language models. Advances in Neural Information Processing Systems, 36, 2023.
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. ShortGPT: Layers in large language models more redundant than you expect. arXiv preprint w:Xiv:2403.03853, 2024.
Xiang Meng, Shibal Ibrahim, Kayhan Behdin, Hussein Hazimeh, Natalia Ponomareva, and Rahul Mazumder. OSSCAR: One-Shot Structured Pruning in Vision and Language Models with Combinatorial Optimization. In International Conference on Machine Learning, 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer Sentinel Mixture Models. In International Conference on Learning Representations, 2017.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In Conference on Empirical Methods in Natural Language Processing, pp. 2381-2391, 2018
Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veysch, Thien Hun Nguyen, Oscar Sainz, Encko Agirre, Ilana Heintz, and Dan Roth. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Compuling Surveys, 56(2):1-40) 2023.
16

---

Published in Transactions on Machine Learning Research (10/2025)
Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. arXio preprinl or orXi::104.088378 2021.
Decebal Constantin Mocamu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madcleine Gibeseu, and An- tonio Liotta. Scalable training of artificial neural networks with adaplive sparse connectivily inspired by network science. Nature Communications, (1):23383, 2018.
NeuralMagic. DeepSparse Inference Engine.
httes///ghhabb.cc///rrrraagggggg 2021.
GitHub repository.
Jeff Pool. Accelerating Sparsity in the NVIDIA Ampere Architecture, 2020. GTC 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1 67, 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communications of the ACM, 64(9):90-106, 2021.
Jingtong Su, Yihang Chen, Tianle Cai, Tianhao Wu, Ruiqi Gao, Liwei Wang, and Jason Lee. Sanity- checking pruming methods: Random tickets can win the jackpot. Advances in Newral Information Pro- cessing Systems, 33, 2020.
Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter, A simple and effective pruning approach for large language models. In International Conference on Learning Representations, 2023.
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive Activations in Large Language Models. In Conference on Language Modeling, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint orXiv:2307.0925: 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Sounya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arxiv preprint rxXi:2337.09288 2023b.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking Winning Tickets Before Training by Preserving Gradient Flow. In International Conference on Learning Representations, 2020.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, al. Thain-of-thougee prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824 24837, 2022.
Peng Xu, Wenqi Shao, Mengzhao Chen, Shitao Tang, Kaipeng Zhang, Peng Gao, Fengwei An, Yu Qiao, and Ping Luo. BESA: Pruning large language models with blockwise barameter-efficion sparsity allocation. In International Conference on Learning Representations, 2024.
Lu Yin, You Wi,, Zhenyu Zhang, Cheng- Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, AJAY KUMAR JAISWAL, Mykola Pechenizkly, Yi Liang, Michael Bendersky, Zhangyang Wang, and Shiwei Liu. Out- lier weighed layerwise sparsity (OWL): A missing secret sauce for pruning LLMs to high sparsity. International Conference on Machine Learning. PMLR, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a Machine Really Finish Your Sentence? In Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800 2019.
17

---

Published in Transactions on Machine Learning Research (10/2025)
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open Pre-trained Transformer Language Models. arXiv preprint Xr::222005.010 010668 2022.
Yuxin Zhang, Lirui Zhao, Mingbao Lin, Sun Yunyun, Yiwn Yao, Xingjaa Han, Jared Tanner, Shiwei Liu, and Rongrong Ji. Dynamic sparse no training: Training-free finc-tuning for sparse LLMs. In International Conference Leurning Represenlations, 2024.
Aojun Zhou, Yukum Ma, Junnan Zhu, Jianbo Liu, Zhijic Zhang, Kun Yuan, Wenxiu Sun, and Hongshong Li. Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch. In Indernational Conference on Learning Representations, 2021.
18

---

Published in Transactions on Machine Learning Research (10/2025)
A Comparison with Reconstruction Error-Based Pruning
Here, we discuss in detail the main difference between our proposed neuron alignment metric and the well- established reconstruction error (Franaar & Alistarh 2023). Reconstruction error (Frantar & Alistarh, 2023; Guo et al., 2024 Xu et al.) 2024; Zhang et al.] 2024) relies on minimizing the L2-norm difference bet the outputs the dense and sparse models at the layer level, following the formula:
min
w'x - (M' w' M
(9)
where W E RRdua is the weight matrix of layer L, X E RR x is the input data to the layer l, and M E [0,1]douexda, is the binary pruning mask for the weights of layer &.
However, in recent decoder-only Transformers [Touvron et al.) 2023a Jiang et al. 2023 cach block applics residual connections and intermediate LayerNorms. Hence, preserving the output of single projection may not be sufficient since it could either be passed to normalization layer or be summed with the residual stream. Specifically, cach module operates as follows:
x (ho)))
a1 = SelfAttention(x))
h1 ha + a1
X2 LayerNorm(h:)
a2 MLP(x²)
h2 h1 +
This means that the input activations to each projection (i.e., the inputs to Wo. WK, Wv, Wo, W Wgate: up: and W down) are the real "control points" of information flow. Modifying the weights without preserving these activation patterns can distort the layer's internal behavior.
To address this, NEURONAL directly compares the input activations to each projection in the dense and sparse models. The resulting neuron aligument metric is computed as:
(x)
(x)
min ΣEΣ M REC, bEB FEC (x)
(10)
E
E 2
where Axxx) represents the projection input to layer for input and L includes all projection matrices in the Transformer Block B.
To conclude, unlike reconstruction--rror--e method evaluates what each layer receives as imput. This distinction leads to a more fine-grained, neuron- level preservation of the behavior of the dense model. Another key difference is given by the fact that the neuron alignment metric produces one numerical value for the activations' alignment between the sparse and the dense model, while the reconstruction error provides separale values of errors for each sparse layer. This allows provide metric that depicts the global alignment between the sparse and dense models, rather than local (layer-wise) perspective.
methods, which measure what a layer produces as output, our
B NeuronAL Complexity Analysis
The space complexity of NEURONAL primarily depends on the size of the calibration set C2 and 1 i.e., the number of sparsity schedules to evaluate. For each input x E C2 and each candidate À E s we perform one forward pass through the dense model and two forward passes through the sparse model (one for the block step and one for the row step of NEURONAL.).
Let B be the number of Transformer blocks, n the input sequence length, and d the hidden dimension. Then, the complexity of one forward pass is O(Bn2 dd). As result, the overall complexity of NEURONAL becomes:
O ((1 2/ |Cal Bndd),
(11)
19

---

Published in Transactions on Machine Learning Research (10/2025)
From practical standpoint, optimize memory usage by processing each block independently and in parallel for the sparse and dense models. Specifically, we collect and compare the input activations to each projection matrix. Once the alignment score is computed for a block, the corresponding activations are discarded to save memory.
As a result, the actual memory complexity at runtime becomes:
O([C)|nīd).
(12)
since inference and neuron alignment are performed block-by-block, independently of [x and the number of layers.
B.1 Complexity NeuronAl vs. OWL
All base pruning methods require a single forward pass over the full calibration set C. On the other hand, top-up algorithms, such as OWT. (Yin et al. 2024) and AlphaPruning (Tu al. 2024), require one additional forward pass for computing block information for their non non-uniform distribution metric. NEURONAL, instead, performs one forward pass per cach À A over a smaller subset Cx. To ensure a comparable cost w.r.t. the top-up baselines which use one additional forward over a calibration set of |C| samples, we set |Cxl = [. so that the overall number of tokens processed remains roughly the same. As result, NEURONAL requires 2. |Cx| : ||A0tt| = 2. |C'| forward passes-whin which is two times more than base pruning methods, and one time more than OWL and AlphaPruning. The runtime comparisons can be found in Table [1 in the main text.
[Cl
C Experimental Setup
Baselines. For OWT., we set the hyperparameters to the values that are used mostly in the original paper, hence M = 5 and À = 0.08; we do the same for AlphaPruning, sctting = 0.3. All these baselines are tested considering each row as a comparison group: in other words, for a given layer, the spansity for each row of each matrix rather than uniform across matrices. This is done for two main reasons: 1) mentioned carlier, it is established that row-wise pruning LLMs leads to better performance w.r.t. layer-wise pruning (Sun et al. 2023), and 2) since approach relies on a row-wise step, for fairness we also use each row (rather than layer) as comparison group on all the other methods, to appreciate the benefit of our approach. We also test our approach on SparseGPT (Frantar & Alistarh 2023), using in this case only the block step, since SparseGPT relics on a weight reconstruction mechanism that pruncs columns first and then adjusts the rows of pruned cells using Hessian information, which makes unfeasible to apply our row step. The results can be found in Tables 28 30 For all the pruning algorithms that use calibration data (i.e., MULTIFLOW, Wanda, and SparseGPT) we use 128 samples from the C4 dataset, in (Frantar &
uniform
Alistarh 2023 Sun et al. 2023 Yin et al. 2024
NcuronAL Sctup. Our method takes as input an LLM model, target sparsity, a scoring-based pruning algorithm, and two sets of À parameters (for the block and the row steps, respectively). In the experiments, we set A = [0.01, 0.02, 0.08, 0.05, 0.06, 0.07, 0.08, 0.09, 0.1, 0.12, 0.15, 0.20,0.25] for the block step, while for the row step, we also added 0.0 (in case of no performance improvement). Each value in 1 has been evaluated (as described in Algorithm B over calibration data Cx. Since all the base pruning algorithms require single forward over C. (with C containing 128 sequences of 2048 tokens each), while OWT. requires second forward always over C, to make the computational runtime similar set C3 8 (the closest power of 2 w.r.t. IC)/ In essence, NEURONAL only requires two forward steps more than the basc pruning algorithrs, and one forward step more than 5 All the experiments have been run on NVIDIA A100 GPUs, both with 40 and 80 GB.
For both C and Cx. we use the same seed (0) for the calibration set, iee. C2 contains the first elements of
20

---

Published in Transactions on Machine Learning Research (10/2025)
D Additional experiments
Here include the results of the experiments that, due space limits, could not include in the main text. Specifically, we report: the results of NEURONAL over more recent model, namely LLama-3 8B; the results on the Language Modeling datasets at 60% and 80% sparsity and the full results on the Zero-Shot lasks at 60%, 70%, and 80% sparsity; the results for the Zero-Shot lasks with Magnitude pruning; the full results of Zero-Shot tasks over LLama 13B and 70B at 70% sparsity; the results NEURONAL (block-only) applied SparscGPT FFrantar Alistarh 2023
D.1 Results on LLama-3 8B
In this section, we include further experi- ments over more recent model: LLama- 3 8B. We stick to the same experimental setup of Appendix C and compare NEU- RONAL to the same baselines of the main text. Specifically, we report the Lan- guage Modeling results Table 15 and Zero-Shot results in Table 16 As we saw in the main text for the case of LLama-2 70B, also in this case, method offers competitive results (although it does not. always result in being the best algorithm in all cases) over the Language Modeling datasets. However, is by a large margin the most performing algorithm, on average, across the Zero-Shot tasks, as clearly shown Table T6. Overall, the points of strength of NEURONAL stated in the main text also apply the most recent LLM models.
Table Perplexity on the three Language Modeling datasets computed over LLama-3 SB) for four different top-up algorithms (Uniform, DSnoT, OWL, and NEURONAL) on three pruning al- gorithms (Magnitude, MULTIFLOW, and Wanda) at 70% sparsity.
60%
70%
80%
Tup-Up
WileeTwet2
CH
WikITest2
C41
WikkText2
PTR
Unifors
50.4
101.3
108.1
156.
1297.5
1038.5
53.8
32.11
119.0
152.0
1:17.0
1086.1
MUSTIPLOW
OWL
27.7
33.00
138.0
155.9
62007
716.5
AlphaPrenteg
22.6
31.5
151.0
170.6
715.0
1309-1
23.2
30..2
38.7
127.9
128.4
5033.5
3075
6395.1
Uniforn
30.6
35.3
41.5
155
103
117.6
1051.5
606.1
1029
Wanda
ORT
21-4
18.0
30.6
20.6
81.7
36.1
193
92.
124.8
156
182.8
152.5
721.3
11:85.9
001.1
717.9
913.0
1613
AlplaPruaing
10.2
28.4
03.1
115.7
167.1
367.7
1110.3
1079.0
1144.
NEtHOOAA
18:4
25.6
30.4
13.1
98.1
100.0
232.
101.2
1032
Table 16: Accuracy the seven Zero-Shot Tasks, computed LLama-3 8B for four different top-up pruning algorithms (DSnoT, OWL, AlphaPruning, and NEURONAL) two pruming algorithms (MULTIFLOW and Wanda) at 70% and 80% sparsity. "Average" indicates the mean accuracy across tasks. The rows corresponding to the pruning algorithms refer to the uniform distribution.
WhnGGrunde
SellaSwaut
BoolQ
ARC--
ARC-C
OBQA
RTE
Sparsity Algorithm
Average
Dense
69.68
72.77
81.35
60.19
80.09
50.433
34.8
64.19
MULTIFLOW
52.71
48.56
51.90
27.55
31.19
18.26
12.20
34.67
w. DSnoT
52.71
48.46
50.00
27.31
31.36
17.15
13.40
34.35
OWL
52.71
51.62
27.84
34.05
12.20
36.92
AlphaPrmiig
52.71
50.99
61.19
27.62
32.03
17.24
12.80
36.40
NEURONAL
62.71
53.12
62.17
29.21
37.12
17.83
15.20
38.19
70%
Wanda
52.71
49.01
51.38
27.27
32.63
17.58
12.00
34.64
w. DSmo'
52.71
18.38
51.50
27.
31.06
17.58
13.60
34.57
w. OWL
52.71
48.86
28.29
35.09
17.41
12.60
36.76
AlphaPeuning
52.71
50.51
53.58
27.69
31.61
17.15
13,20
35.61
w. NEURONAL
52.71
54.06
62.20
31.27
41.71
20.05
16.20
39.74
MULTIPLOW
52.71
26.46
11.00
32.26
DSmoT
52.71
16.57
37.95
26.36
28.03
20.48
11.80
31.99
w. OWL
52.71
48.02
40.76
26.38
28.24
20.22
13:00
32.85
AlphaPruning
52.71
14.20
34.00
NRURONAL
52.35
40.01
62..14
26.54
28.41
19.45
13.00
35.99
80%
Wanda
52.71
47.99
37.88
26.68
27.53
19.11
12.40
32.04
DSmoT
52.71
37..80
26.21
12.60
32.33
OWT.
52.71
47.53
38.
26.63
28.58
19.54
13.60
32.44
AlpbaPeuning
53.07
19.41
26.39
27.15
19.88
13.60
33.93
NRURONAL
52.71
49.57
60.31
26.65
28.41
18.60
13.40
35.68
21

---

Published in Transactions on Machine Learning Research (10/2025)
D.2 Language Modeling at 60% and 80% Sparsity
In Table 17 we report the results of NEURONAL over the 3 Language Modeling datasets (WikiTex12, C4, and PTB) with the five different. L.Mss considered in the main text, for 60% and 80% sparsity. In the first case, our approach turns out to be the best one in 23 out of 45 cases, while for 80% sparsity in 20 out of 45, while is second best in 15 cases. is interesting to notice how at medium sparsity (60%) all the top-up algorithrs, including ours, provide similar results, while the improvement provided by NEURON AL at 80% (w.r.t. the top-up competitors) in some cases reaches a factor of 2-3x (e.g., with LLama-1 1 7B for MULTIFLOW and Wanda).
Table 17: Perplexity on the three Language Modeling datasets computed over five different LLMs for four different top-up algorithms (Uniform, DSnoT, OWL, and NEURONAL) on three pruning algorithms (Mag- nitude, MULTIFLOW, and Wanda) at 60% sparsity.
Phi-2.78
Maams-1 7B
Mlamm22 TR
Mistral-TR
OPT 0.78
Algocithm
Top Up
WildText22
WildTTexr2
C4
PTD
WildToot2
WikdiText2
Dane
9.7
14.1
15.2
5.7
7.3
10.1
5.5
7.3
32.9
5.3
5:4
36.6
10.9
12.7
15.8
Unilarm
61.3
45.9
06.9
152.4
152.8
3,02x3
6.80d
4.20vl
1.71w5
19.0
21.4
189.5
9:00d
6.3068
6.76d
DSm/f
181.6
1.46e3
6.76ed
7.3066
14662
8.16d
6.06
6,95e3
Maguitude
42.2
240.4
1.94e4
2,3066
12.0
109.1
6.81e3
8.6703
Alplac Pruning
203.7
1.1563
153.0
49.4
5.4365
13.1
16.2
185.4
6.07c3
2.83-3
3.03id
NEtRoNAA
18.5
16.8
65.1
65.8
03.6
163.1
58.4
70.3
2.4864
12.5
17.0
1.50.0
1.20us
547.6
982.8
Uniform
25.4
28.3
54.7
11.6
13.9
25.0
11.0
13.7
106.9
167.7
311.0
884.3
16.3
19.7
20.8
DSmdi
87.2
42.1
50.2
10.1
12.7
18.2
10.5
13.4
137.6
10.9
14.5
86.0
16.9
19.2
26.5
NULTIFLOW
OWL
28.8
26.7
12.7
106,0
64.1
641.
16.1
13.8
53.4
116.4
454.6
30.9
NEURONAL
27.2
12.1
72.2
112.4
168.0
505.9
16.6
Uniforms
225.8
20.3
J5.9
10.7
13.7
24.0
10.8
14.0
122.2
11.3
15.9
101.6
15.2
17.9
23.7
DSmdlT"
12:2
38.0
50.6
10.4
13:2
20.8
10.8
14.1
100.6
11.4
15.9
96..
15.8
19.1
24.6
Wizdda
OWL
24.8
28.2
45.6
9.4
11.8
18.5
9.2
11.9
15.1
20.3
14.5
84.5
16.7
17.8
24.5
AlphaPrumttgg
165.3
166.5
609.7
12.6
09.3
10.8
15.3
05.5
27.9
Table 18: Perplexity on the three Language Modeling datasets computed over five different LLMs for four different top-up algorithms (Uniform, DSnoT, OWL, and NEURONAL) on three pruning algorithms (Mag- nitude, MULTIFLOW, and Wanda) at 80% sparsity.
Ph
LLama-1 7B
LLama-2 7U
Mistral-77 7B
OPT 6.7U
Algortehm
Wik/Texs2
C4
PTB
WikIText2
C4
PTB
WikiText2
C4
PTB
WikiText2
C4
PTB
WikiTous2
C4
PTB
9.7
12:7
Uniooras
1.5364
1.7te4
3.20c4
1.1365
1.1460
1.4000
5:30el
3.12c4
7.5863
4.29c1
2,3134
2.33e1
3.4984
2.17e5
1
3.126
3,M883
1.19ee
Meguitude
OWL
6.63c8
5.60cB
8.30c3
1.065
1.5066
2.09c4
1.79c4
5.79c4
5,5065
5.70c3
0.50pl
1.7644
2.166
AlpiaPruning
5.5404
9,8704
7,53c4
5.78c4
LG464
9.2264
2.86o4
5,0764
2.74:3
3,30c3
2.52c4
L.314k
NETRONAL
9.665
53365
5.574
5.664
S11ee1
2.0864
7.53
1.15g3
2.20e1
Unitoras
2.5344
1,28c4
2,20c4
4.868
200188
1.4543
3,88:8
2.0833
3.51c3
4.42a3
2.3863
3.9065
3,005
1:23el
57565
2.8+55
8.20
1728
1.5463
5.546
327.0
270.0
752.8
1.36el
9.72-3
1.18vl
MULTIFTOW
OWL
256.2
2.80
1.50el
928.5
563.1
1.765
314.2
414.5
2.83-5
S.366
2.2188
3.56e5
1.35el
1.11e
AlpiaPuuning
2.1264
1.97el
2.0044
0330
841.6
1.50p5
070.9
2.0k53
3.8k33
3,0565
6.0Ge3
4.21e3
2,00-3
34483
NECHONAL
2.3468
892.4
4.2265
250.8
200.2
013.8
378.5
456.5
2.0963
1.023
710.1
1,56:3
1.20c5
721.0
1.36c3
Unitorss
2.00e4
1.34e5
3.38c4
5/2263
3.9733
1.00c4
1933
3.1263
5,563
350.9
177.2
763.7
4.2663
2.35<3
2.363
DiStil
1.53ed
6.90
1.00e1
371
7.7065
5.963
6. ..663
5,0955
3.0..
977.3
735.1
7.75c3
0.16-3
7.7%c
Wando
OWL
2.55c8
1.21c3
7.0633
936.5
654.5
2,00:5
063.0
456.2
2:2508
206.3
157.8
603.0
1.30c4
1.0034
1.42a4
AlpisPomiing
4.5te4
5:53et
2.60e1
768.1
65419
1.20
963
670.0
2.16v
201
152.8
774.2
5.61e3
4.99a5
5.2
NECRONAA
2.5055
1,00
4.00u5
302.8
272.2
785.8
887.2
000.2
2.00..1
240.5
177.8
765.9
1.04u5
032.8
1.15c5
22

---

Published in Transactions on Machine Learning Research (10/2025)
D.3 Zero-Shot at 60%. 70%. and 80% Sparsity
report the results at 60%, 70%, and 80% sparsity of NEURONAL over Zero-Shot tasks with the five different LLMs tested in the main text. In particular, we report the detailed results for each task, while in the main text, we only report the average results across the seven tasks.
In Tables 19 21
Table 19: Accuracy on the seven Zero-Shot Tasks, computed over five different LLMs for three different top-up pruning algorithms (DSno'T, OWL, and NEURONAL) on two pruning algorithms (MULTIFLOW and Wanda) at 60% sparsity. "Average" indicates the mean accuracy across tasks. The rows corresponding to the pruning algorithms refer to the uniform distribution.
WinoGrande
HellsSwag
BoolQ
ARC-
RTE
Model
Algorithum
Average
MULTIFLOW
53.31
DSaoT
63.18
66.77
13.49
40.84
66.62
35.41
26.0
18.90
OWL
AlphaPruning
62.45
12.96
50.83
29.03
48,27
22.18
16.0
40.25
NEURONAL
02.82
67.4
65.87
42.96
70.88
38.65
27.8
53.77
Phi-2.7B 7B
Wanda
63.54
64.8
69:08
40.16
08.64
34.9
25.4
52.36
w. DSaoT
62:45
64.33
59.17
39.25
64.18
33.53
22.6
49.36
OWL
64.62
64.30
64.83
39.50
67.63
34.98
24.2
51.48
AlphaPruning
61.26
55.33
62.39
31.07
47.9
23.81
17.2
43.14
NEURRONAL
64.26
66.38
67.55
40.63
86.5
34.39
24.6
52.04
MULTIFLOW
57.04
62.38
67. 19
45.31
59.55
30.97
24.6
19.
w. DSaoT
68.
OWL
51.15
63.16
66 06.51
16.53
60.65
31.14
25.1
19.70
AlphnPruning
49.13
NEURONAL
50.9
63.54
68.35
47.91
63.89
32.17
27.2
50.57
LLama=1 7B Wanda
59.57
62.67
68.81
13.64
62.84
30.38
24.8
50.39
DSmoT
51.62
61.
67
43.39
63.89
30.55
25.4
19.12
w. OWL
55.60
64.17
70.61
16.63
62.96
31.74
24.8
50.93
AlpanPruning
59.57
65.27
68.81
46.09
80.4
32.85
24.6
51.08
w. NETRON As.
58.48
63.61
70.55
46.53
63.8
30.88
26.0
51.41
MULTIFLOW
18.76
DSwoT
51.15
63.77
63. 63.91
13.12
66,25
31.83
21.0
19.62
OWL
51.87
AlphaPruning
52.71
64.96
64.96
44.63
61.32
30.8
23.4
48.05
NEURONAL
51.08
LLama-2 7B
Wanda
54.15
64.48
65.44
43.85
65.19
30.46
26.2
49.97
w. DSaoT
53.79
61.09
61.83
12.39
63.89
30.03
22.8
18.53
OWL
53.79
66.61
66.76
46.63
67.63
32.34
28.0
51.08
AlphsPruning
51.15
67.4
66.07
16.17
61.98
33.28
25.8
51.21
NEURONAL
52.71
66.77
71.99
46.85
66.33
32:08
27.2
$1.99
MULTIFFOW
51.62
48
39.17
12.44
290.67
18:77
11.0
392.01
TSSoT
54.87
66.61
70.86
45.99
68.27
51.90
OWL
53.07
50.
16.33
28.
32.55
19.28
13.2
31.70
AlphaPruning
53.79
34.97
NEURONAL
52.35
51.90
43.13
28.08
32.28
18.52
12.0
34.07
Mistral-7B Wonda
54.87
51.03
DSaoT
54.15
60. 03.
70:43
44.5
06.88
31.40
20.60
D0.51
w. OWL
37.04
67.17
73.85
15.66
67.89
32.50
23.20
52.49
AlphnPruning
57.4
68.54
64.98
44.02
65.32
30.03
20.8
19.87
w. AEURONAL
567.01
66.06
70.34
16.37
08.6
33.00
23.2
52.09
MULTIFLOW
52.71
58.25
62.69
42.24
56.52
25.68
22.20
45.76
w. D6moT
53.07
58. 68.48
62:
42.13
57.79
23. 98
222.20
45.76
OWL
53.07
AlpiaPruning
58. 12
58.64
62
40.83
51.68
24.40
19.0
45.01
NEURONAL
52.71
25.0
15.51
OPT-6.7B
Wanda
52.71
50.67
62.29
42.80
58.00
25.60
22.6
46.24
w. DSaOT
52.71
58.17
62.11
25.40
45.75
OWL.
52.71
58.72
62.66
42.14
58.33
25.17
22.6
46.05
AlphnPruning
56.23
58.
62.17
39.71
51.85
21.53
19.6
11.50
NEURONAL
52.71
81.25
62.39
41.95
57.28
25.08
22.0
16.10
23

---

Published in Transactions on Machine Learning Research (10/2025)
Table 20: Accuracy on the seven Zero-Shot Tasks, computed over five different LLMs for three different top-up pruning algorithms (DSnoT, OWT, and NEURONAL) on two pruning algorithms (MULTIFLOW and Wanda) at 70% sparsity. "Average" indicates the mean accuracy across tasks. The rows corresponding to the pruning algorithmns refer to the uniforın distribution.
WinoGrundh
HellaSwan
BoolQ
ARC-c
ARC-c
OBQ4
Model
Algorithu
Average
MULTIFLOW
53.79
53.49
28.
46 16.59
20:45
15.6
38.
DSuo'T
52.71
54.06
54.92
28.
44.15
21.5
15.6
38.
w. OWL
52.35
55.33
61.99
30.31
54.59
21.23
18.4
42.
AlphmPraning
47.29
51.3
37.71
25.85
26.61
20.9
13.0
31.81
w. NEURONAL
53.07
58.61
62.17
33.45
52.19
25.31
17.4
43.18
Phi-2.7B
Warda
52.35
53.3
62.11
28.51
41.87
20.99
17.4
39.69
W. DSuoT
52.35
51.54
60.98
28.3 28.33
11. 11.02
21.05
15.4
38. 36.76
OWL
52.71
53.59
62.05
30.09
48.61
22.53
18.5
41.20
w. AlphaPruning
17.29
29.58
60.86
25.52
25, 05.55
22 22.01
14.00
35.02
NEURONAL
52.71
56.33
62.17
32.47
49.422
22.95
16.8
41.87
MULTIFLOW
55.96
52.57
61.96
29.77
34.64
19:45
15.2
38.51
DSuooT
51.15
50.43
59.33
29.3 99.33
36.45
19. 19.28
13.6
37.51
W. OWL
52.35
58.64
62.63
36.74
47.43
26.62
18.2
43.23
AlphaPruning
55.6
63.54
64.1
36.37
43.22
26.79
18.6
44.07
w. NELRONAL
57.76
61.72
63.3
38.19
50.88
26.88
22.8
45.93
Laamn-1 Wanda
55.25
52.8
57.46
28.54
32.2
18.0
13.8
36.9
w. DSnoT
54.15
51.22
54.56
28. 28.97
33.08
18.
13.6
36.
OWL
58.48
58.56
62.60
34.71
47.35
21.06
17.4
13.31
w. AlphaPruning
58.68
63.14
63.85
35.78
44.82
26.71
17.6
44.08
NEURONAL
55.96
59.27
63.12
36.91
50.0
26.11
20.6
44.37
MULTIFLO
52.71
50.00
62.05
28.52
33.04
17.92
13.6
36.98
DS
52.71
50.99
59.72
27.92
32.58
16.81
13.0
36.25
W. OWL
52.71
56.12
62.05
32.40
42.42
19.88
18.6
40.6
AlphaPruning
52.71
58.64
35.17
22.87
17.66
41.5
w. NEURONAL
53.45
58.09
62.35
35.24
48.32
22.44
20.2
42.87
LLamm-2 7B
Warda
52.71
48.46
49.04
28.
30-30
11.88
34.37
W. D85eTT
52.71
50.36
47.77
27.67
30.6
17.32
12.22
34.00
OWL
52.71
55.96
62.11
31.56
43.70
20.65
17.0
40.57
w. AlphaPraning
52.71
61.33
62.2
34.82
43.43
22.1
17.46
42.00
NECRONAL
$0.07
$7.85
63.27
355.42
49.62
22.44
20.0
43.1
MULTIFLOW
49.82
30.75
41.19
26:
26.
21.84
12.46
33,77
DSreTT
36.66
18.
112.0
w. OWL.
53.79
49.17
38.90
26.77
27.78
19:20
12.8
32.63
AlphmPraning
11.88
W. NEURONAL
52.71
50.75
38.29
27.16
25.75
17.75
13.00
32.63
Mistral-BB
Wonda
52.71
51.62
59.79
28.
34.18
18.17
12.6
36.85
W. DSmoTT
52.71
50.28
58.62
28.51
33.54
18.86
13.2
36.53
OWT.
52.71
33.91
62.20
13.6
w. AlphoPruning
52.71
56.27
62.2
31.47
38.55
18.52
13.6
39.05
NEURONAL.
52.71
60.62
34.8
16.0
41.56
MUNTIFLOWW
53.79
40.72
43.0
26.44
30:51
20.05
13.8
33.91
DSaoT
12.2
OWL
48.74
48.62
61.56
27.18
35. 63)
16.47
11.46
35.69
AlphssPrnnig
22.
NEURONAL
50.54
50.99
62.17
31.3
40.74
22. 27
17:44
39.34
OPT-6.7B
Wardda
26.91
35.
17.75
36.31
w. DSnoTT
52.71
49.57
60.61
25.91
35.06
17.58
12:00
36 96.35
OWL
29.
Al AphaPrrniing
51.62
51.7
62.17
353.13
40.19
22.77
15.22
39.53
NEURONAL
24

---

Published in Transactions on Machine Learning Research (10/2025)
Table 21: Accuracy on the seven Zero-Shot Tasks, computed over five different. LLMs for three different top-up pruning algorithms (DSnoT, OWL, and NEURONAL) on pruning algorithms (MULTIFLOW and Wanda) at 80% sparsity. "Average" indicates the mean accuracy across tasks. The rows corresponding to the pruning algorithms refer to the uniform distribution.
WinoCrands
HellaSwag
BoolQ
ARC-6
OBQA
RTB
Model
Algorithm
Average
MULTIPLOW
32.71
50.12
51.04
26.15
27.44
19.37
13.4
34.32
w. DSaoTT
30.18
49.49
37.83
26.32
27.82
19.97
13.5
32.2
OWL
53.07
51.22
56.76
26.30
30.85
18.52
15.2
$5.99
AlphaPruming
44.77
48.78
37.83
25.74
26.18
20.31
11.4
30.72
NEURONAL
52.71
50.59
43.0
26.66
33.85
21.16
16.0
34.96
Phi-2.7B
Wands
63.07
49.26
02.17
26. 26.93
27.
20.90
14.6
36.21
DSmoT
50.99
38.04
27.10
OWL
52.35
51.22
60.21
26. 51
29 888
20.005
13.6
36.26
AlphaPruning
43.32
49.5
30.74
NEURONAL
$2.71
50.2
02.17
26. 26.96
32.32
20.56
16.8
37.36
MULTIFLOW
47.29
50.91
40.03
26.17
26.77
21.16
12.4
32.1
DSnoTT
46.57
50.43
37.83
26.02
27.06
20.05
12.2
31.45
w. OWL
50.18
50.04
45.47
26. 74
27.05
20.48
12.2
33.25
AlphaPruning
36.08
27.
36.41
w. NEURONAL
54.51
51.14
62.2
28.82
32.77
19:45
14.4
37.61
LLama-1 7B Wanda
DSmoT
46.93
50.36
37.83
26.03
26.56
21.33
13.4
31.78
47.29
49.88
37.83
26.67
27.19
AlphaPrusing
32.35
19.17
61.19
26.58
26.77
20.77
10.6
35.34
NEURONAL
49:57
60.03
30.85
345.3
MULTIFLOW
53.43
18.5
37,6 37,83
26.35
27.15
21.25
13.2
32.63
w. DSmo'T
52.71
48.86
37.86
26.
26.80
20.30
13.0
32.23
OWT
26. 22.62
26.04
32.11
AlphxPrwning
52.71
50.67
38.01
26. 62
26.77
20.14
11.2
32.30
NEURONAL
25.1
LLama-2 Wanda
47.65
09.01
37.83
26.82
26.52
20.52
14.6
31.81
DSaoTT
47.91
20.73
32.27
OWT
32.71
50.88
37.
26. 06.52
27.
19.37
12.8
392-48
AlphaPruming
52.71
18.51
37.89
26.42
27.11
19:45
12.2
32.09
NEURONAL
52.71
50.04
37.77
27.3
28.45
18.94
14.0
32.74
MUTTIPLOW
50.18
48.15
37. 80
26.67
26.18
22.70
13.20
31. .08
DSmoTT
52.71
47.36
37.55 83.88
26.58
28.03
32.18
OWL
48.38
19.09
38. 34
25.88
25.59
23.29
13.6
392 04
AlphaPruning
NEURONAL
52.71
48.46
37.83
26.07
27.57
19.03
14.8
02.12
Mistral-7B Wonds
53.79
48.
37.
26.
27.88
19.8
12.8
312-48
DSw/T
32.35
48.30
37. 833
26.55
27.44
19.54
13.0
32.14
w. OWL
52.71
17:13
37, 83.
26.08
27.78
18.52
13.2
32.00
AlphaPruning
52.71
47.99
37.63
26.73
28.11
19.54
12.8
32.29
w. NEURONAL
52.71
51.14
38.04
27.24
28.06
20.31
13.4
33.08
25.87
32.34
DSaoT
52.71
50.83
57.31
26.00
25.00
20.22
11.80
34.84
OWL
51.07
33.67
AJphaPruning
52.71
49.01
37.83
26.
27.31
20.73
14.8
32.65
NEURONAL
OPT-6.7B
Wands
51.15
52.09
41.53
26.47
28.15
18.86
11.80
33.31
w. DSmoT
52.71
51.38
55.32
17
27.06
19.37
13.80
35.12
OWL
52.71
19.33
37.83
25.81
25.67
20.31
13.00
32.10
AlphaPruning
52.71
48.86
37.98
26.09
26.73
20.14
12.6
32.16
NEtBONAL
52.71
50.67
41.83
26.43
30.06
11.0
33300
25

---

Published in Transactions on Machine Learning Research (10/2025)
D.4
at 60%. 70%. and 80% Sparsity with Magnitude
In Tables 22 24
report the results of NEURONAL over Zero-Shot tasks using Magnitude pruning.
The results provided by NEURONAL turn out to be the best in 10 out of 15 cases, while being the second best in 3 cases. is also worth noticing that the performance gap between the Magnitude pruning and score-based pruning algorithms (such as Wanda or MULTIFLOW) is generally quite high. Hence, NEURONAL can improve the performance of Magnitude (in the standard setting with uniform distribution) only to a certain degree, since high sparsity ratios (as the ones we test) the performance of Magnitude has been shown to be poor (Jaiswal et al. 2024
Table 22: Accuracy on the seven Zero-Shot Tasks, computed over five different LLMs for three different top- up pruning algorithms (DSnoT, OWL, and NEURONAL) on Magnitude 60% sparsity. "Average" indicates the mean accuracy across tasks. The rows corresponding to the pruning algorithms refer the uniform distribution.
WhnGGrnnd
HellaSwox
ARC-
ARCe
RIE
Model
Algorithm
Average
Magnitade
67.04
02.83
51.38
42.58
66.33
35.41
25.2
49..1
DS
54.51
64.09
42.32
41.00
66.25
$4.98
26.6
47.12
Phi 2.7B
OWL
w. AlphaPruuiing
57.76
56.2
47.34
35.16
58.46
33.22
23.2
44.49
NEURONAL
$4.10
05.07
47.43
42.38
05.30
36.77
25.8
48.45
Magnitade
52.64
DSw/T
38.3
1.Lama-1
w. OWL
52.55
58.41
51.8
42.02
56.31
29.78
25.5
44.59
AlpanPruming
57.14
56.36
40.53
56.52
25
NEURONAL
Magnitade
$1.26
55.8
41.19
36.97
50.17
26. 96.
16.2
39.79
DSaoT
$3.79
58.04
42.87
38.3
53.25
27.9
41.71
L.aam-2
w.
w. Pruning
52.35
61.88
58.1
46.13
58.50
31.65
24.8
47.66
NEURONAL
23
59.59
83
46.15
58.06
32.85
27.6
48.00
Magnicade
w. DSaoTT
35.6
62.35
68.53
48.28
67.51
$3.11
25.2
$1.23
Mistrs-7B
w. OWL
58, 71
64.48
72.17
413.39
68.14
33..5
23.8
32:225
AlpiaPruninn
w. NEURONAL
64.06
82
65.45
02
24.6
52.36
Magnisade
53.43
50.59
37.55 37.96
26.38
26.6
21.42
13.2
52. 78
DSaeT
32.71
49.25
37.55 36
26.14
27.27
21.5
15.2
32.56 $2.56
OPT-6.7B
20.77
18.02
w. Alpha Pruning
02.71
61.22
37.83
26.54
29.84
10.8
13.4
33.05
NEURONAL
32.71
53.01
39
33.23
37.71
24.06
16.8
36.79
26

---

Published in Transactions on Machine Learning Research (10/2025)
Table 23: Accuracy on the seven Zero-Shot Tasks, computed over five different LLMs for three different top- up pruning algorithms (DSnoT, OWL, and NEURONAL) on Magnitude 70% sparsity. "Average" indicates the mean accuracy across tasks. The rows corresponding to the pruning algorithms refer to the uniform distribution.
WinoGrande
BoolQ
ARC-c
OBQA
RTB
Model
Algoritiam
Averaage
Magniisnde
46.93
53.50
47.22
30.45
47.85
24.57
10.2
38.54
Pli-2.7B3
w. OWL
45.13
52.85
19.2
32.25
51.04
27.56
21.4
40.01
AlphaPPruingg
47.65
52:41
38.01
26.06
25.25
22.53
15.2
32.
68.87
Magnisade
53.43
49.90
37.92
27.50
31.73
22.44
16.6
34.24
DS0TT
52.71
51.7
37.83
27. 71
30.26
22.7
15.4
34.00
Llame-1 7H
w. AlphinPraning
52 71
53.04
30.48
36. 18
421.56
24.83
21.2
38.
NEURONAL
52.71
$4.62
52.8
39.88
06.13
26.79
23.00
4.2.24
Maguitand
51.25
49.06
37.86
25.9
32.88
w. D880T
53.79
40.88
86
25.42
28.83
20.56
16.6
33.22
OWI
AlphuPruming
52.71
50.85 83
43.91
41.67
25.34
NEURONAL
$1.51
55. 41
64.86
33.25
(2.10
27.47
21.2
68
Maguitude
41.13
42.77
36.42
DSnoT
53.07
42.61
36.44
Mistral-7B
w. OWL
07.76
56.30
40.17
36.48
45.76
22.01
18.8
40.99
AlphaProingg
68.96
42.3
NEURONAL
$3.79
58.32 36
62.8
38.6
04.23
28. 28.28
21.0
Magnitude
02.71
49.8
37.83
25.88
26.68
21.33
12.4
32.338
OPT-6.7B
w. OWL
02.71
50.59
37.83
25.88
25. 46
21.25
12.8
32.35
AlphnProming
32.71
02.00 01
37.83
26.21
27.74
20.9
13.0
32.91
NEURONAL
Table 24: Accuracy on the seven Zero-Shot Tasks, computed over five different LLMs for three different top- up pruning algorithms (DSnoT, OWL, and NEURONAL) Magnitude at 80% sparsity. "Average" indicates the mean accuracy across tasks. The rows corresponding to the pruning algorithms refer to the uniform distribution.
WinGGrando
HellaSwag
ARC-c
ARC.
Model
Algorithm
Average
Magnitade
45.13
50.96
20.9
32.30
w. DS5m/T
46.93
52.38
30.63
25.9
28.32
21.25
13:4
$2.54
OWL
49.46
50.91
42.35
26.71
35.27
21.67
13.4
34.25
Prunnnn
33.72
w. NRURRON NEURONAT
30.54
62.25
42.60
26.21
25.7
22.
12.9
33.61
Magnicade
46.21
49.96
53.98
25.00
24.55
84
33.76
DSaoT
L.ama-11 7B
w. OWL
48.38
48.93
44.74
25.76
26.35
21.08
15.8
33.01
AlpiuPruning
26
50.2
50.54
26. 25
25,
20:59
15.8
33.11
NEURORAL
Magnitade
$2.35
49.57
46.18
25.94
26.14
23.11
16.0
$1.19
w. DSaoT
$2.71
51.54
37.55
25. 46
27.10
22. 44
15.4
33.22
1..ama22
OWL
48.70
w. Alpiu Proming
23
48.33
40.57
26. 16.56
22.00
16.2
w. NEURONAL
$4.15
50.56
$2.69
26. 52
29.67
20.14
15.6
35.49
Magnicade
$1.26
50.99
41.16
25.90
27.48
21.884
33.32
Mistral 7B
w. OWL
52.35
50.20
41.04
26.55
27.78
19.57
13.8
33.10
Promingg
33.07
47.67
37 56
26.4
25,
19.54
13.4
32:34
NEURONAL
Magnitude
52.77 71
49.40
37.83
25.79
20.39
21.25
13.00
32.35
w. DSaoTT
52.77 71
49.67
37.83
25.7%
25.06
20.65
12.8
32.18
OPT 6.7B
OWL
32.71
w. AlpanPruniig
52.71
40.09
37.83
25. 84
26.14
20.52
12.8
32.18
NEURONAL
52.71
60.01
37.83
25.74
20.73
20.9
12.4
32.47
27

---

Published in Transactions on Machine Learning Research (10/2025)
D.5 Zero-Shot at 70% and 80% Sparsity for LLama 13B (both v1 and v2) and 70B (v2)
report the single-task detailed results for the Zero-Shot tasks, for which only the averaged results are reported in the main text in Table g These detailed results clearly show the gap between NEURONAL and the baselines: as mentioned in the main text, NEURONAL turns out to be the top-up algorithm that, in the majority of the cases, provides the strongest results across all tested models.
In Tables 25 27
Table 25: Accuracy on the seven Zero-Shot Tasks, computed over LLama-1 13B for four different top-up pruning algorithms (DSnoT, OWL, AlphaPruning, and NEURONAL) on two pruning algorithms (MULTIFLOW and Wanda) 70% and 80% sparsity. " "Average" indicates the mean accuracy across tasks. The rows corresponding to the pruning algorithms refer to the uniform distribution.
WhnGGande
HellaSwag
BoolQ
ARC-c
OBQA
RTE
Sparsity
Amgorithm
Average
Dense
71.48
72.53
78.04
59.84
77.31
46.25
33.0
62.64
MULTIPLOW
52.35
32.22
16.60
40.43
DSnoT
52.71
53.04
62.17
31.38
43.01
19.97
17.20
39.93
w. OWL
52.71
63.93
67.28
40.63
57.83
27.99
20.80
47.31
MphaPruning
54.15
66.06
43.04
57.70
28.99
21.40
NEURONAL
51.15
65.90
69.72
45.10
62.54
30.89
27.20
50.79
70%
Wanda
52.71
63.43
61.74
30.61
40.87
16.72
16.60
38.95
30.41
30.98
OWL
52.71
63.85
63.55
39.75
67.74
27.22
20.80
46.52
AphaPruming
53.43
65.75
62.78
42.23
58.12
28.33
21.20
47.41
NEURONAL
52.71
64.64
65.76
43.96
26.00
49.34
MULTIFLOW
53.79
40.01
37.83
26.89
26.98
20.99
14.00
32.78
w. DSnoT
52.71
49.64
38. 38.17
26.99
27.31
19.80
12:40
32.43
OWT.
52.71
38.50
28.45
18.17
10.50
32.26
AlphaPruning
52.71
53.04
62.17
30.46
35.75
22.18
15.60
38.56
w. NEURONAL
52.71
55.09
62.17
32, 22.211
39.48
23.81
19.60
40.72
80%
Wanda
52.71
50.67
37.88
26.68
27.23
19.97
14,20
32.76
W. DSuoTT
52.71
50.04
37.88
26.83
27.90
18.94
12.60
32.41
52.71
27.37
28.45
12.20
32.25
AlphaPruning
52.71
55.25
62.17
29.56
34.30
20.31
12,20
38.07
w. NRURONAL
52.71
51.14
62.
20.42
34.72
19.97
15.80
37.98
28

---

Published in Transactions on Machine Learning Research (10/2025)
Table 26: Accuracy the seven Zero-Shot Tasks, computed over LLama-2 13B for four different top-up pruning algorithms (DSno'T, OWL, AlphaPruning, and NEURONAL) two pruning algorithms (MULTIFLOW and Wanda) at 70% and 80% sparsity. "Average" indicates the mean accuracy across tasks. The rows corresponding to the pruning algorithms refer to the uniform distribution.
WinoGrand
HellaSwag
BoolQ
ARC-
ARC-c
OBQA
RTE
Sparsity Algorithm
Average
35.2
MULTIFLOW
52.71
49.25
39.82
27.34
29.21
19.20
11.60
32.73
DS
62.71
61.22
62.05
28.75
34.68
17.49
11.80
36.96
OWL
51.85
39.44
14.00
38.05
AlphaPruniig
52.71
63.85
62.45
31.79
53.58
26.45
20.80
44.95
W. NEURONAL
62.71
62.51
70.06
38.66
57.70
28.07
25.00
47.89
70%
Wanda
62.71
61.78
62.23
29. 00
36.
17.06
12.
37.54
DS
52.71
50.67
62.11
28. 78
36.62
17.66
12.80)
37.34
OWL
62.71
61.25
65.14
38. 15
57, 70
26.88
22.20
16.33
AlphaPruning
67.48
63.64
56.73
29.27
22.20
47.44
NEURONAL
52.71
62.04
70.67
40.28
58.59
29.69
24.00
48.28
MULTIFLOW
52.71
49.96
37.83
26. 16
26.01
20.90
13.20
32.4
w. DSuo'
52.71
50.20
38. 38.26
26.09
28.
20.65
11.80
32.59
OWL
52.71
50.25
37.83
26.41 41
27.36
20.90
11.00
32.36
AlphaPruning
52.71
19.61
37.95
26. 63
27. 10
20.31
11.60
32.28
NEURONAL
48.78
37.92
13.40
80%
Wanda
52.35
52.57
37.83
26.55
26.64
20.05
13.80
32.82
DSaoT
52.71
19.25
37.83
26.02
26.61
20.14
12:40
32.14
OWL
62.71
48.30
41.35
00
27.48
19:11
11.40
32.49
AlphaPruning
52.71
52.64
62.23
27.74
30.13
11.00
36.59
w. NETRONAL
52.35
50.99
61.05
28.16
29.59
18.17
12.40
36.19
Table 27: Accuracy on the seven Zero-Shot Tasks, computed over LLama-2 70B for four different top-up pruning algorithms (DSnoT, OWL, AlphaPruning, and NEURONAL) on two pruning algorithuns (MULTIFLOW and Wanda) at 70% and 80% sparsity. "Average" indicates the mean accuracy across tasks. The rows corresponding to the pruning algorithms refer to the uniform distribution.
WinoGrande
HellaSwaig
BoolQ
ARC-
ARCHE
OBQA
RTE
Sparsity hm
Average
67.87
77.98
66.
37.20
67.10
MULTILLOW
56.32
71.51
73.95
49.00
70.30
37.29
26.50
55.02
w. DS1oTT
59.21
74.43
75.35
51.50
72.45
39.76
28.8
57.36
w. OWL
61.01
74.11
72.40
51.10
72.70
37.71
27.20
56.00
AlphaPrunin
61.37
73.72
74.50
50.25
71.75
38.05
28.20
56.83
w. NRURONAL
61.01
72.77
75.90
52.20
71.25
37. 29.29
25.
56.95
70%
61.01
46.60
27.20
56.73
DSnoT
57.10
72.85
75.65
19.25
71.85
38 82
26.80
56.09
w. OWT.
64.98
76.22
72.90
50.95
73.40
38.14
29.80
57.91
AlphaPruning
63.18
51.30
29.40
NRURONAL.
62.09
75.93
78.90
52.80
744.4
38.90
20.00
588.96
MULTIFLOW
52.35
48.62
62.60
27.35
26.00
18.17
11.20
35.18
27.70
w. OWT.
46.93
55.41
02.75
31.10
37.00
19.80
13.40
38.18
AlphaPruning
46.93
60.93
65
33.30
47.85
21.33
16.40
41.63
NRURONAL
54.51
65.04
65.55
38.40
46.55
27.99
19.20
45.32
Wanda
50.18
40.49
62.30
26.95
26.50
17.92
11.20
34.98
w. DSnoT
16.93
19.25
61.70
27.15
27.10
00
12.00
34.65
OWT.
40.82
57.38
62.70
31.30
40.10
20.11
13.00
30.21
AlphaPeuning
48.01
61.96
02.00 05
33. 80
48.75
22. 24
16.40
42.00
NRURONAL
29

---

Published in Transactions on Machine Learning Research (10/2025)
D.6 NeuronAL SparseGPT at 60%. 70%. and 80% Sparsity
In Tables 28 30
present the results of NEURONAL on SparseGPT the WikiTexi2, C4, and PTB datasets, using the block-only setting. To note: Since SparseGPT prunes and updates the weights from columns to rows, the row step of AL cannot be included: indeed, it would force each row to have a different sparsity ratio, which in contrast with the nature of SparseGPT.
Using SparseGPT, the superiority of NEURONAL less evident than with other pruning algorithms. Never- theless, NEURONAL turns to be the best-performing top-up algorithm in out of 15, 8 out of 15, and 7 out of 15 cases, respectively, for WikiText2, C4, and PTB. Interestingly, for lower sparsity, the gap between uniform and non-uniform distribution (both NEURON AI. and OWL) is less remarkable than at higher spar- sity. We explain these results with the inherent functioning of SparseGPT, which, differently from the other pruning algorithms, includes a weight reconstruction step. However, we can conclude that, also in this case, our proposed approach turns out to be effective in many cases at increasing the task performance.
Table 28: Perplexity on Wik1 using SParseGPT.
Model
Sparsity
Top-Up
Pbi-2.7B
LLama-1
LLama-2
Mistral-BB
OPT--87B
Uniforta
00%
OWL
15.8
9.4
9.2
9.1
14.2
NEURONAL
Uniform
28.9
27.3
27.3
22.0
20.5
70%
OWL
27.7
19.2
20.5
18.6
NEURONAL
27.3
22.6
20.9
17.8
21.8
Uniformm
131.0
207.0
122.1
98.4
95.7
80%
107.5
93.8
84.3
NEURONAL
113.5
144.7
88.7
70.8
$4.0
Table 29: Perplexity C4 using SparseGPT.
Model
Sparsity
Top-Up
Phi-2.7B
LLama-1 7B
LLama-2 7B
Mistral-7B
OPT-6.7B
Uniformm
19.0
12.8
12.9
13.0
15.3
60%
11.7
NEURONAL
19.1
12.1
11.7
12.3
15.5
Uniferu
31.5
70%
OWL
28.2
21.1
22.8
23.7
22.4
NEURONAL
27.8
21.8
22.5
21.9
22.2
Uniform
96.7
136.2
1018
86.5
72.5
OWL
79.7
68.3
73.4
66.2
65.4
Table 30: Perplexity on PTB using SparseGPT.
Sparsity
Top-Up
Phi-2.7B
LLamaa1 7B
LLama-22 7B
Mistral-7B
OPT--6.7
Uniform
26.7
19.5
430.5
73.7
20.3
60%
OWL
29.3
16.9
262.1
70.9
21.0
Uniforma
50.3
52.6
3790.0
153.2
32.0
70%
NEURONAF.
47.3
40.5
861.6
123.4
32.8
Cnlform
195.4
285.6
3201.7
316.2
80%
OWL
141.4
162.3
5666.5
278.5
98.9
NEURONAL
156.7
260:2
3659 8
266.8
105.3
30

---

Published in Transactions on Machine Learning Research (10/2025)
D.7 Results on Broader Sparsity Range
While in the main text, mainly tested high sparsity configurations (hence 2 0.6), here report the results of NEURÓNAL well as the baselines for several sparsity values between 10% and 80%. Table 31 for LLama-2 7B, and Table 32] for Phi-2, show the results of perplexity across three datasets (Wikitext, C4, and PTB) for different sparsity ratios. The algorithm's setup has been kept the same as the experiments in the main lext, see Appendix C The results provide a non-trivial paltern: for up to 50% sparsity, all the top-up algorithms perform similarly (with very small deviations), with DSnoT providing almost in all cases the best results. At such low sparsity values, even Uniform achieves effective results.
However, the pattern completely changes for sparsity above 60%, where, as already shown in the main text, NEURONAL turns out to be the most reliable among all the tested top-up algorithms in achieving the best performance. This trend is in line with the results provided in the original baseline papers (Yin et al. 2024 Lu et al. 2024), where the authors also recognize this top-up algorithms.
pattern for non uniform sparsity
Table 31: Perplexity di 2 7B at different sparsity levels (10% 80%) on the WikiText2, C4, and PTB datasets.
WikiText2
Top-Up
10%
20%
300%
10%
30%
600
70%
80%
Uniform
5.49
5.59
5.74
6.06
6.92
10.77
78.01
4.93c3
OWL
5.49
5.59
5.76
6.10
B.N6
9.17
24.9
BGSLD
DSuoT
5.48
5.52
5.20e3
AlphaPraning
6.49
5.59
5.77
6.14
7.06
9.882
31.97
982 08
NeuronAL
557.16
Top-Up
C4
10%
2005
30%
600%
70%
80%
Uniform
7.28
7.41
7.63
3.10
9.24
13.99
81.01
3.12e3
OWL
7.29
7.43
7.66
8.17
9.22
11.85
20.49
862.95
DSnaT
7.27
7.33
7.50
35.65
4.44e3
AlphaPraniug
7,29
7.42
7.68
8.27
9.50
12.62
37.73
670 03
NeuronAL
Top-Up
PTB
500%
600%
90%
Uniform
33.07
33.65
31.55
38.02
18.20
122.25
599.27
5.29e3
OWL
33.01
33.45
34.54
36.65
43.16
75.09
333.7
2.28cl
DSnoT
32.81
32.98
34.10
37.25
47.08
109.61
491.77
6.89e3
AlphaPraning
33.00
33.83
34.72
37.
43.85
69.34
273.84
2.18el
NeuronAL
48
206.97
2/003
Table 32: Perplexity of Phi-2 at different sparsity levels (10 (10%-80%) on the WikiText2, C4, and PTB datasets.
WikiText2
Top-Up
10%
20%
30%
40%5
50%
60%
70%
80%
Uniform
10:07
11.46
227.
2.04e4
OWL
9.83
10.05
10.18
11.47
14.20
24.80
132 06
2.54.%
DSacT
86.
1.52e1
AlphmPraning
9.87
10.09
10.59
12.05
16.66
165. 29
1.22-4
4.30e1
NeuronAL
10.51
Top-Up
C4
10%
20%
40%
50%
60%
70%
80%
Uniform
14.25
14.48
14.87
15.81
18.25
29.28
182.70
1.24c4
OWT.
14.22
14.46
14.87
15.78
18.24
28. 18
116:21
1.21
DSacT
14.74
15.20
20.25
6.86c3
AJphaPraming
14.25
14.50
14.98
16.31
20.26
166. 48
3.05e4
3.33e1
NeuronAL
PTB
Top-Up
10%
20%
30%
40%
30%
60%
70%
80%
Uniform
18.37
18.70
25.07
48.98
346.16
3.14e4
OWL
18.34
18.70
19.18
21.02
25.65
45. 613
183.66
7.06-3
DS
18.45
18.85
19.75
27.26
50.58
257.60 067.60
1.4064
AlphnPraning
18:35
18.74
19.65
21.55 21.
28. 69
669. 68
2.2464
2.066-
NeuronAL
18.71
25.62
31

---

Published in Transactions on Machine Learning Research (10/2025)
D.8 Sensitivity to Calibration Data
complement the results regarding the seed set of the calibration data at 60% and 80% sparsity. The results are fully in line with the ones presented in the main text. As expected, the standard deviation of the performance increases when increasing the sparsity ratio, and at higher sparsity (80%), it turns out to be model-dependent.
In Tables 33 34
Table 33: Perplexity achicved by NEURONAL with different calibration data seeds (0, 16, 46) at 60% sparsity.
Model
Dataset
Phi-2.7B
LLama-1 7B
LLama-2 7B
Mistral-7B
OPT-6.7B
WikiText2
9.5
=
16.2 0.1
C4
27.0 + 0.2
11.9 + 0.1
11.9 = 0.0
13.8 + 0.1
19.1 + 0.1
PTB
42.3 + 0.7
17.1 + 0.3
65.3 = 0.8
74.9 # 0.2
25.1 + 0.1
Table 34: Perplexity achieved by NEURONAL with different calibration data seeds (0, 16, 46) at 80% sparsity.
Model
Dataset
Phi-2.7B
LLama-1
LLama-2
Mistral-7B
OPT-6.7B
WikiTex12
3654.7 255.1
382.4
216.5 12.6
1284.9
482.5
72323.6
121.7
250.5
27.1
265.3 +: 31.1
171.7
663.5
316.3
PTB
6014.9 788.3
624.4
165.5
11019 + 94.1
706.1 6.9
1056.9 124.5
E NeuronAL À Selection at 60% and 80% Sparsity
In the main text, we presented an experiment regarding the ability of NEURONAL to pick the most performing À parameters (in the block-only case) at 70% sparsity. Here we include the same analysis at 60% and 80% sparsity. In Fig. 0 and Fig. 7 it is clearly visible how NEURONAL still performs correctly over different sparsity ratios. It is also worth noticing that the calibration data always come from the C4 dataset, and then the results are transferred to the other unknown datascts.
WilThest
PTB
Cr
Dr
Figure 6: Perplexity over different values of \ at 60% sparsity. The orange dot indicates the value selected by NEURONAL
Mira-
PTB
Figure 7: Perplexity over different values of À at 80% sparsity. The orange dot indicates the value selected by NEURONAL
32

---

Published in Transactions on Machine Learning Research (10/2025)
F Reproducibiliy:: X Parameters Selected by NeuronAL
Here we show the À parameters selected by our proposed approach for each model, sparsity ratio, and pruning algorithm tested in this work, aiming to facilitate the reproducibility of our results for the community. Please note that such values are the ones used for each combination ion of sparsity-pruning algorit been extracted from Cx from C4 (using as seed and 8 as size), and then transferred to all the other datascts/tasks. We report the final À values for both the block and row steps in Table 35 for the first 5 models tested in the main text, and in Table 36 for the LLama 13B models.
that have
Table 35: parameters selected by NEURONAL (block row) for each combination of charsity-pruming algorithm-model. Note that, for SparseGPT, the row step not possible (see the main text for details).
Model
Sparsity
Top-Up
Phi-2.7B
LLama-1 7B
LLama-2 7B
Mistral-7B
OPT-6.7B
Magnitude
0.01
0.25
0.10
0.20
0.20
0.04
0.15
0.25
0.25
0.25
Wanda
0.10
0.25
0.09
0.25
0.12
0.25
0.08
0.00
0.01
0.15
60%
MULTIFLOW
0.08
0.25
0.12
0.25
0.12
0.25
0.06
0.00
0.01
0.06
SparseGPT
0.01
0.02
0.06
0.09
0.02
Magnitude
0.06
0.25
0.20
0.20
0.25
0.00
0.20
0.25
0.15
0.20
Wanda
0.12
0.25
0.15
0.20
0.15
0.25
0.15
0.25
0.25
0.25
70%
MULTIFLOW
0.15
0.25
0.15
0.25
0.12
0.25
0.15
0.01
0.25
0.25
SparseGPT
0.02
0.04
0.08
0.08
0.05
Magnitude
0.01
0.25
0.03
0.10
0.20
0.05
0.20
0.20
0.20
0.08
Wanda
0.20
0.25
0.20
0.25
0.20
0.25
0.15
0.25
0.20
0.20
80%
MULTIFLOW
0.12
0.20
0.20
0.25
0.20
0.20
0.15
0.20
0.20
0.09
SparseGPT
0.06
0.02
0.07
0.08
0.07
Table 36: À values selected by NEURONAL on each combination of sparsity-pruning algorithm for LLama 13B (vl & v2) (block row).
Model
Sparsity Tpp-Up
LLama-1 13B
LLama-2 13B
Magnitude
0.10
0.25
0.12
0.12
60%
Wanda
0.10
0.20
0.09
0.25
MULTIFLOW
0.15
0.25
0.12
0.25
Magnitude
0.12
0.25
0.25
0.25
70%
Wanda
0.15
0.20
0.12
0.20
MULTIFLOW
0.15
0.20
0.15
0.25
Magnitude
0.20
0.15
0.12
0.05
80%
Wanda
0.15
0.20
0.15
0.25
MULTIFLOW
0.20
0.25
0.04
0.25
33


Reviews:
[{"review_id": "KAxTjXKqOU", "reviewer": "Reviewer_KUSh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on unstructured pruning and proposes a new approach, NeuroAL. This method obtains sparse models by maximizing the activations' alignment with respect to their corresponding dense models. It is superior to baseline approaches in that 1. it requires no further training; 2. it runs faster than the baseline approaches. Empirical results of NeuroAL on language modeling and downstream tasks also demonstrate advantages. \n\nWhile the strengths are already stated above, I have some concerns about the contributions of this work:\n\n1. Across all the evaluations, this sparse model suffers from very significant performance decrease compared to the dense model. This questions the meaning of this line of work in the first place, as the proposed approach does not seem really useful practically given such a  performance loss. I guess maybe this is because the authors perform experiments at relatively large sparsity ratios at 60%, 70%, 80%. I think it is helpful to report results at smaller sparsity ratios to understand better on the sparsity-performance tradeoff. \n\n2. Unstructured pruning is known to provide limited improvements in inference speed, despite achieving seemingly high sparsity levels. This is also reflected in Table 10, which shows only a 1.3x–1.5x speedup at 80% sparsity. Nevertheless, I am curious about which inference engines the authors used for evaluating the dense baseline. Frameworks such as vLLM and sglang have introduced extensive optimizations for dense model inference, while similar optimizations may be either infeasible or highly non-trivial for unstructured-pruned models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors mainly claim the superiority of the proposed approach over the other unstructured pruning baselines, which I think is accurate.", "audience_yn": "Yes", "audience_explanation": "This paper proposes a new approach to benefit unstructured pruning, which should be relevant to some TMLR's audience.", "requested_changes": "1. Report results at more sparsity levels to reveal the full picture of sparsity-performance tradeoff, which can help understand how practical the proposed approach is.  \n2. Include more experiment details on the inference speed evaluation", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VhuZzulnck", "reviewer": "Reviewer_3osa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary**:\n\nThe paper introduces NeuronAl, which is a top-up algorithm designed to improve the performance of existing pruning methods without retraining. The method uses neuron alignment, i.e. a metric comparing input activations between a dense and pruned model, to guide block-wise and row-wise sparsity redistribution. The proposed method is tested extensively in four LLM families with ten language tasks, showcasing the method’s effectiveness.\n\n\n**Strengths**:\n- The paper is well-written and easy to follow.\n- It makes sense to use the activations from both sparse and dense models. Alignment of these metrics is an intuitive idea.\n- The experimentation is really thorough with over 300 test cases.\n- The empirical results are mostly solid.\n\n\n**Weaknesses/Requested Changes**:\n- One major weakness in my opinion is the lack of theoretical motivation for using the alignment as a metric. While the intuition to use alignment is somewhat clear, a small theoretical explanation would benefit this paper. However, I understand if a full theoretical proof would be too much for a rebuttal, but maybe a couple of paragraphs hinting at the theory would be nice.\n- I would also recommend the authors to more explicitly state what are the claims and evidence of the paper in the introduction (for eg. Claims regarding the fact that the method is better/faster,etc.)\n- One recommended ablation would be to compare the proposed alignment metric with other potentially good alignment metrics, e.g. cosine-similarity, KL-divergence, etc. This would strengthen the use of the exact formulation used in the paper.\n\n**Minor Weakness**:\n- Use citep and citet for citations. This differentiates the in-sentence citations with ones that are not. \n- Replacing conference names with references in Figure 1.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper claims that the proposed method, using alignment between activations of dense and sparse models, is better than previous approaches, which is demonstrated in the experiments.", "audience_yn": "Yes", "audience_explanation": "This work will be interesting to the compression community.", "requested_changes": "Please see the Weaknesses", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AWkyJlGmBC", "reviewer": "Reviewer_JQU1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces NeuronAl, a new top-up algorithm for pruning LLMs based on neuron alignment. Neuron alignment compares the dense and sparsified input activations to each projection matrix in a Transformer block on a calibration set $C_\\lambda$. The intuition is that these are the real control points of information flow. The method redistributes block-wise and row-wise sparsity based on neuron alignment.\n\nStrengths:\n- the method doesn't need hyperparameter tuning, only the size of the calibration set $C_\\lambda$ needs to be given\n- the method only requires computing the activations (and not the gradients, for example), so it is fast\n- interesting insight: the linear sparsity schedule is the most reliable and gives good results\n- comprehensive experiments across three language modeling datasets and 7 zero-shot tasks. Three different pruning algorithms and four top-up algorithm baselines are used.\n- comprehensive ablation studies about $\\lambda$ selection, comparing neuron alignment to using reconstruction error, and sensitivity to calibration data\n- the paper measures the inference speed-up after pruning\n- the paper is clearly written\n\nWeaknesses:\n- NeuronAl is outperformed by AlphaPruning on midsize models (LLama-2 70B), and the zero-shot tasks are not measured\n- future work could be discussed", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are well-supported in general, with extensive evaluations. However, there are some gaps.\n\nMy main concern is about the relationship between model size and the performance of NeuronAl. Although the Introduction states that the approach is tested on LLMs from 7B to 70B parameters, most of the results are obtained on the small LLMs (<= 7B parameters). I believe that the only results with Llama-2 70B are in Table 8. NeuronAl - although it is faster - is outperformed by AlphaPruning on the 70B model. I coulnd't find the zero-shot task results for the larger models (I think they are missing).\n\nI believe that the analysis of the runtimes could be expanded. Currently, Table 9 has only the runtime of LLama-7B, but it would be more interesting to see how the runtime scales with the size of the model. Section 5.3.1 mentions that NeuronAl is still about $10\\times$ as fast as AlphaPruning on Llama-2 70B, this could be expanded upon. The algorithmic complexity of the methods could be compared.\n\nLastly, the models examined in the paper (e.g., Llama-1) are dated by LLM standards and have been superseded by newer versions.", "audience_yn": "Yes", "audience_explanation": "I believe that the ideas of the paper are novel and interesting, and its topic is very relevant to TMLR.", "requested_changes": "My main requested changes are about making the claims better supported:\n- include the results of zero-shot tasks for the larger models, like LLama-2 70B\n- provide additional information for the analysis of runtimes\n\nIt would be good to include some more recent LLMs if possible.\n  \nSmall typos:\n- OWL is not cited at its first mention.\n- \"This holds mainly BESA Xu et al. (2024) and DSA Li et al. (2024).\"\n- \"to align with the latest discovering in the literature of structured pruning\"", "broader_impact_concerns": "I don't believe the work has ethical implications that require a Broader Impact statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)