Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization
Authors: Guanchen Li
Yixing Xu
Zeping Li
Ji Liu
Xuanwu Yin
Dong Li
Emad Barsoum
Venue: NeurIPS 2025 poster ()
Keywords: Structural Pruning
Large Language Models
Sparsity Distribution Optimization
Primary area: deep_learning

Abstract: Structural pruning enhances hardware-agnostic inference efficiency for large language models (LLMs) yet often fails to maintain comparable performance. Local pruning performs efficient layer-by-layer compression but ignores global topology. Although global pruning aims to identify an optimal sparse model, intuitive methods typically adopt a two-stage paradigm that first evaluates substructure saliency and then applies global pruning, which ignores inter-structure dependencies and fails to achieve end-to-end optimization. To address these limitations, we propose Týr-the-Pruner, an efficient end-to-end search-based global structural pruning framework. This framework constructs a supernet by repeatedly applying local pruning across a range of sparsity ratios to each layer in an LLM, with the core goal of determining the optimal sparsity distribution under a target overall sparsity ratio. Concretely, we introduce an effective local pruning and an expectation error accumulation approach to improve supernet construction. Furthermore, we employ an iterative prune-and-search strategy with coarse-to-fine sparsity granularity to ensure efficient search convergence. Experimental results show that Týr-the-Pruner achieves state-of-the-art structural pruning, retaining 97% of the dense model's performance while removing a challenging 50% of Llama-3.1-70B's parameters.

Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization
Guanchen Li, Yixing Xu, Zeping Li, Ji Liu, Xuanwu Yin, Dong Li, Emad Barsoum Advanced Micro Devices, Inc. (AMD) (guanchen,yixin xu, zeping, 11, ji, liu, xuansu. yin, a. 1i, emad. barsoum) @amd. com
Abstract
Structural pruning enhances hardware-agnostic inference efficiency for large lan- guage models (LLMs) yet often fails to maintain comparable performance. Local pruning performs efficient layer-by-layer compression but ignores global topology. Although global pruning aims to identify an optimal sparse model, intuitive meth- ods typically adopt a two-stage paradigm that first evaluates substructure saliency and then applies global pruning, which ignores fails achieve end-to-end optimization. To address these limitations, we propose Týr-the-Pruner. an efficient end-to-end search-basee based global structural pruning framework. This framework constructs supernet by repeatedly applying local pruning across a range of sparsity ratios to each layer in LLM, with the core goal of determining the optimal sparsity distribution under a target overall sparsity ratio. Concretely, we introduce an effective local pruning and an expectation error accumulation approach to improve supernet construction. Furthermore, we employ iterative prune-ane seeee-nddsccarde strategy with coarse-to- ceaseeto-ffiee sparsity granularity ensure efficient search convergence. Experimental results show that Týr-the- Týr-the-Pruner achieves state-of-the-ar structural pruning, retaining 97% of the dense model's ss performance while removing a challenging 50% of 3lama-3.1-70BB parameters.
dependencies and
Introduction
Large language models (LLMs) have significantly advanced natural language processing, achieving exceptional performance in tasks such as text understanding, generation, and reasoning [53 7
However, the computational and storage resources required for model deployment incur high costs and environmental impacts, limiting their accessibility in resource-constraine scenarios. Model compression techniques, such quantization [21] 10], pruning (9] 23 39], and low-rank decomposi- tion 45). are essential for reducing LLM size and computational demands. This paper focuses on structural pruning, which enhances inference efficiency
hardware-agnostee manner.
Existing structural pruning methods for LLMs are typically classified into local and global techniques. Local pruning methods [17 26). which prune layers individually, enable efficient compression of hundred-biin siwteredd-bilioo-caaale LLMs on a single GPU via offload approaches. However, they overlook global dependencies model topology and restrict the sparsity to be uniform across layers. Global pruning methods (33 18. 1 alleviate local constraints, facilitating sparsity allocation and the potential for optimal pruning. However, many existing methods estimate the saliency local substructures and prune them accordingly via global ranking, ignoring inter-structure dependencies and hindering end- to-end optimization. Such methods may also suffer from the inefficiency of backpropagation-biie based saliency estimation and overfitting when calibration data is limited. Therefore, a question arises:
How to achieve efficient global structural pruning with end-to-end optimization?
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

To address this challenge, we propose Týr-the-Pruner, an efficient sarch-based global pruning framework with end-to-end optimization. Our framework constructs a supernet by applying local pruning to each layer, producing pruned copies with different sparsity ratios. The objective is to identify an optimal subnet that satisfies the target overall sparsity ratio within the supernet by determining the optimal sparsity distribution across layers. We use evolutionary search (22] to solve this optimization problem. To construct reliable supernets and perform effective and efficient search, we make the following contributions:
To improve supernet construction, we propose an effective local pruning approach for attention heads and feed-forward networks (FFN), using Taylor expansion-basee first- and second-order optimization information to identify redundant structures and adjust remaining weights. Pruning and weight adjustments are applied progressively and finely to preserve accuracy. Additionally, we introduce an expectation error accumulation approach to address the challenge of unclear error propagation caused by the multiple pruned copies within the supernet. This approach ensures balanced mutual awareness across sparse structures during supernet construction.
To enhance the efficacy and efficiency of subnet search, we employ tailored distillation- inspired metric as the optimization objective to guide the search process, aiming to preserve the subnet's generative capability. In general, Týr-the-Prume frther formed as an iterative prune- and-search framework that refines sparsity allocation for each layer with reduced search space and fast convergence. Each iteration prunes and constructs a supernet across a specific range of sparsity ratios, coupled with sparsity-shift-drive random sparsity shifts between layers generate parent candidates, and the best-performing ones are filtered as offspring. The sparsity interval is refined after each iteration.
evolutionary search, where
By making these contributions, Týr-the-Pruner achieves end-to-end global pruning with strong efficacy and efficiency. Notably, the proposed framework only requires 4M tokens for calibration and search. Experimental results demonstrate that Syr-the-Prunce surpasses ootte--f-hh-arr pruning methods. For example, Týr-the- Týr-the-Pruner outperforms the SOTA method FLAP, achieving 3.45 lower perplexity in language comprehension and 10.26% higher average downstream accuracy when pruning 37.5% of the parameters of Llama-3.1 1-8BB. Moreover, it maintains 97% performance with 50% pruning on Llama-3.1 1-70B a sparsity ratio that is considered aggressive for existing methods.
2 Method
This section presents Týr-the- Pruner, novel structural pruning framework for large language models (cf. Section 2.T for preliminaries), as illustrated in Figure [ This framework (1) constructs supernet by applying local pruning across various sparsity ratios each model layer, aiming to (2) search the optimal sparsity distribution under a target overall sparsity ratio. Specifically, we propose an effective local pruning approach (cf. Section 2.2p and an expectation error accumulation approach (cf. Section 2.3) to enhance supernet construction. An iterative prune-and-search strategy with coarse-to-fine sparsity granularity (cf. Section 2.4) ensures efficient search convergence.
2.1 Preliminaries
Large language models typically use the Transformer decoder architecture [42], as shown in Fig- ure [a). Each Transformer layer consists of two key components: the multi-head self-attention (MHA) and the feed-forward network (FFN), followed by a residual connection and layer normaliza- tion. Given the input X the f-th layer, the output X, can be expressed as:
X LayerNorm (X,-1 f-1 MHA(X,-)),
X, LayerNorm (X + FFN(X)).
The MHA mechanism captures dependencies across different positions in the input sequence with multiple attention heads, each with its query (W,). key (W;), value (W,). .,), and out (W.) linear transformations. Modern LLMs typically employ a SwiGLUU-bsed FFN [34, consisting of gate (W gate)), up (W (Wup), and down (W downn) linear transformations, with activation after the gate. This structure aids in extracting non-linear representations.

---

Prune
Sparsity Crunalarity Search
Add & Normalize
Add & Normalize
Feod Forward Network
Forward Nowerrk
Fulher PPL 18.19
Watrum Maste
35%
325
Select Spane Stractue
Souns
Add & Normalio
Add & Normalize
Multt-bead Attention
Attention
Select
Solect Sparso Senicturo
(a)
(b)
(c)
(d)
Figure 1: An overview for Tyr-the-Pruner. Large language models (a) will be effectively locally pruned across multiple sparsity ratios and constructed into supernet (b). An iterative prune-and- search strategy will be used to select the optimal sparse structure for each layer while maintaining target overall sparsity ratio: pruning and sparsity-shift-drive iteratively with a coarse-to-fing sparsity interval granularity (c). Ultimately, the post-pruned LLM with the optimal sparsity distribution (d) is obtained.
evolutionary search are implemented
Structural pruning for LLMs be applied across four key dimensions: (1) attention heads, (2) FFN intermediate neurons, (3) embedding dimension size, and (4) model depth. It can be isotropic (uniform sparsity across layers) attention heads and FFN intermediate neurons with non-uniform sparsity: pruning functionally independent heads and neurons allows for controllable accuracy loss, while layer-specific sparsity further enhances pruning by tailoring compression to each layer's ' characteristics.
non-isotropic (layer-specific sparsity). This paper focuses pruning
2.2 Effective Local Pruning
Redundant structure identification and weight adjustment. When pruning is scoped to the local level, can determine the pruning outcome by eliminating the redundant input channels of each o_proj and down_prog _pon modules, with a consistent sparsity across layers. Assuming the weight of layer is W E dd don and its input activation is X dd the pruned weight W satisfies the sparsity constraint C. The corresponding optimization objective is expressed as:
argminw XW
XWI
s.t. C(W) C.
(2)
The pruning process can be viewed as perturbation applied to the weights: W W JW. There- fore, the error function given by = |XW x = | which can be approximated by Taylor series expansion around W and whose local fluctuations can be defined as:
OE OW
P
8E =
8W +
TT
SW
+0((6W19)
OW
G 20
H
800
5E reflects the effect of 5W on the pruning error, which we aim to minimize. The first-order gradient G cannot neglected, as the calibration samples are inevitably misaligned with the proprietary closed-source pre-training data. The Hessian matrix H helps to identify pruning-sensitive weights from curvature perspective. Considering the sparsity constraint (6W, p.: p. the p-th input channel of W is to be pruned), we design the redundant channels and weight adjustment as follows:

---

W argminw,,,. p.: (IGm. W
IW Pi
WT
8W ~p; =- - Hp Gapay
(4)
P.P
= and G HW (analytic solutions computed without backpropagation, efficient) are used estimates of the local optimization information. The channel p with the least error impact identified and pruned, while oW adjusts the remaining weights to compensate for pruning errors (~ p represents other channels that have not been pruned).
Pruning heads and neurons. In our framework, feed-forward network neurons are pruned based on individual channel saliency computed from the down_pro] layer, where each channel acts the atomic unit for ranking and removal. For multi-hease self-attention, saliency is first computed per output channel of the o_proj layer, then aggregated (averaged) across channels belonging to the same head, which is treated as the atomic unit for pruning.
Progressively pruning and weight adjustment. We adopt progressive pruning with appropriately fine granularity: finer granularity enables unpruned weights to gradually and uniformly compensate for pruning losses in small increments while enabling precise and dynamic redundant channel identification. Reducing gramularity does not significantly complicate pruning, as the key intermediate variable H can be rapidly adjusted to account for partial channel pruning in (dd complexity [8]
Detailed analysis can be found in Section A.I
2.3 Prune-to-supernet across Multiple Sparsity Ratios
As illustrated in Figure [b), super- net will be constructed by repeatedly applying local pruning across range of sparsity ratios to each LLM layer, producing pruned copies with varying sparsity ratios. However, this introduces challenges in error accumulation across layers. Error accumulation introduces an additional forward pass of the post- pruned layer, using its output activation as input for the next layer. The change in the input directly affects the optimiza- tion of the subsequent layer. In the ex- ample shown in Figure 2 pruning half of Llama-3. 3.1ma-3.11-BB. 1-Bss'sss 88's parameters using the local pruning approach with error accumulation results in significantly lower language comprehension perplexity than pruning without it. This performance gap highlights the critical role of error accumulation: it enables deeper layer pruning to be aware shallower layer pruning.
Perplexity@ Wikitexi2
Perplexity@ Wikitex12
Llama-3. 3.188: 5.81
1.1mm-3.1-8BB 5.81
Local pruned 50%: 53823
Local pruned 50%: 58.09
Dense
D Dense
Dense
Dense
Pruned
To-prune
Pruned
To-prune
Layer
Layer
Layer
Layer +
(a) w/o Error Accum.
(b) w/ Error Accum.
Figure 2: Implementing layerwise error accumulation gives a more accurate pruning result than not. Solid lines indicate forward propagation, and dashed lines indicate pruning,
The existence of multiple sparse structures complicates error accumulation, making it unclear which pathway to prioritize. To address this issue, propose an expectation error accumulation approach to enable balanced mutual awareness among the sparse structures in the supernet. Let the output activation of the e-th sparse structure with sparsity S. in layer / be X We define the expectation output activation X for this layer as:
1 1- S. X E S.)
Xet1.
(5)
higher scaling factor is assigned low sparsity weights because their output activations are more stable and reliable. By enabling expectation етrоr accumulation and applying the local pruning approach, we can prune .lama-3.1-88 to create nine sparse structures in each layer, with 12.5% the sparsity interval (covers complete pruning and abandoned pruning). The post-pruned model achieves a language comprehension perplexity of 66.38 on WikiText-2 task, with manually picking the 50% sparse structure as an example. This result is close to the ideal perplexity of 58.09 achieved under full error accumulation and significantly better than the 208.92 perplexity from random error accumulation and 538.23 perplexity with abandoned error accumulation.

---

2.4 Týr-the-Pree Pruner
By introducing effective local pruning and expectation error accumulation approaches, we can construct supernet to tackle the global sparsity allocation problem. Specifically, we address the following issues to achieve efficient and effective sparsity allocation: (1) defining generalizable criteria for selecting better sparse structure, (2) achieving an efficient search-based sparse structure selection while maintaining overall sparsity, and (3) handling the contradiction between fine-grained sparsity intervals and the large search space.
Align to dense model behaviors to win. Towards the definition of better sparse structures, we consider that large language models are designed for multi-task generalization. Thus, guiding sparse structure selection on a single task risks overfitting. To mitigate this, we adopt a distillation-inspired metric to measure the similarity between sparse and dense models. A salient similarity indicates that the current sparse structure is better aligned with the dense model, making more suitable for selection. Specifically, let himmm and he,ee denote the activations of the dense and e-th sparse (structure) models at layer f, and ydease and z spurse represent the logits of the dense model and selected (e)
(fe) (c+)/2) sparse subnet. The optimization objective is formulated follows:
sparss {é) = argmin (e) Σ az hyees h
+ (L dense
Z
sparse (e)
(6)
Sparse structure selection via evolutionary search. Evolutionary search can achieve convergence in model architecture optimization [36] (22). Compared to intuitive router training, evolutionary search requires no additional parameters. It maintains constant overall sparsity by shifting sparsity between sparse structures from different layers, whereas router training relies on penalty terms for suboptimal soft sparsity control. Evolutionary search is efficient, as it allows the just-in- in-time loading (cf. Section 3.5) of sparse structures and leverages the backpropagation free feature.
Mutation (stochastic perturbation) in our evolutionary search arises from sparsity shifts across layers (ef. Select Sparse Structure in Figure D For instance, the sparsity of the /-th layer may decrease by 8%, while the f'-th layer increases by s% (achieved by selecting different sparse structures). In each generation, we randomly generate such a group of sparsity distributions as candidates. Starting from the root generation, the performance of candidates is evaluated, and the best-performing ones are selected to generate new candidates for the next generation. Generations continue to be explored until the optimal sparsity distribution is found.
Iterative prune-to-supernet and evolutionary search. The search space for selecting sparse structures with fine- grained sparsity is enormous. For instance. constructing supernet with a sparsity interval of 1.5625% would result in 65 sparse structures per MHA/FFN layer. For a 40-layer LLM, this would lead over 5K sparse structures, creating 10 search space. Identifying solutions in this large search space is difficult and costly. To address this challenge, Tyr-the-Pruner adopts an iterative prune-and search strategy that progressively refines sparsity granularity. In each iteration, we (1) prune the model, (2) construct supernet with a narrower sparsity interval, and (3) perform evolutionary search to locate the optimal sparsity distribution. This coarse-to- fine paradigm improves both search efficiency and accuracy in sparse structure discovery, as illustrated below.
Example: Týr-the-Prume Pruner
Týr-the-Pruner starts by constructing a coarse-grained supernet (e.g., Figure [b)) using 12.5% sparsity interval, where each MHA/FFN layer contains only nine candidate structures. Evolutionary search is then applied to determine the optimal sparsity pattern at this iteration. In the subsequent iteration, a new supernet (e.g., Figure Tc)) is constructed by halving the sparsity interval to 6.25% and centering it on the current optimum (c.g., 37.5%), yielding nine refined candidates from sparsity 12.5% to 62.5%, followed by evolutionary search to update the optimal sparsity. With the sparsity interval halved at each iteration. Týr-the- Pruner narrows the search space to about 76 per iteration and reaches a 1.5625% sparsity granularity within four iterations. This coarse-to-fin refinement enhances both efficiency and accuracy in identifying the optimal fine-grained sparsity for final model compression (e.g., Figure [
The algorithmic procedures for local pruning, supernet construction, evolutionary search, and the overall Týr-the- Phunerr framework are detailed in Algorithms T to 4 of Section A.2
5

---

Table 1: Post pruning performance comparison of different methods. Language comprehension perplexity is validated on the Wikitext2 test set with a sequence length of 4096, where a lower value reflects better performance. Downstream accuracy (%, higher is better) is averaged across ARC-Ease -asy, ARC-Challenge, BoolQ, HellaSwag, OpenbookQA, RTE, WinoGrande, and MMLU, with MMLU using a 5-shot benchmark and others 0-shot benchmark. The best results are shown in bold.
Perplexity on Wikitext2
Average Downstream Accuracy (%)
Sparsity
Method
Llama-
Llama-3.x
Mistral
Llama-2
Llama-3.x
Mistral
13B
2:30
0.80
7B.v0.3
7B-v0.3
Nemo
0%
N/A
5.12
4.57
7.29
5.76
5.84
4,95
5.35
57.96
62.05
57.01
64.08
64.77
63.72
66.24
ShonGPT
8.86
5.67
12.42
13.90
13:14
7.58
7.72
53.27
59.16
53.13
57.75
58.50
59.49
59.46
LaCO+
7.52
5.69
12.25
10.12
7.46
7.95
53.23
57.26
52,46
59.41
60.36
58.67
59.96
SIsoOPT
8.25
7.19
18.71
20:46
22.10
7,00
9.74
59.70 55.89
51.64
57.55
56.82
59.67
53.27
Wanda-sp sp
6.09
86.91
6.86
57.41
47.89
56.82
12.5%
I.I. M..urrruner
6.11
5.17
11.14
8.24
8.26
6.17
6.79
53.38 59.78
46.98
53.96
54.04
55.26
58.23
ZipLM
5:21
11.32
10.37
5.84
61.91
62.46
60.24
OSSCAR
5.94
5.21
11.11
10.15
9.87
5.75
7.04
55.29
61 61.94
52.23
57.19
58.53
62.06
53.89
FLAP
6.11
5.75
10.25
8.34
8.07
6,18
7 7.68
57.55 54.63
47.04
55.72
56.66
59.51
57.67
PyP-ure
5.84
5,03
9.16
7.39
7.41
5.61
6.31
56,98
62.66
54.78
62.01
63.02
63.05
64.15
ShortGP
23.41
17.94
1464,20
4836.41 41
3418.83 83
35,20
124,20
146.68
$1.86
41.25
38.12
38.62
$1.07
51.68
LaCO+
9.00
128.77
137.17
22.91
53.65
ShscGPT
16.84
12.50
45.44
47.73
55.43
12.08
19.37
51.40
58.04
45.87
50.01
48.49
52.26
46.27
Wa
19.92
94.12
48.95
17.83
49.13
41.30
25%
L.L.-Prune Pruner
11.56
7.11
25.14
18.65
19.35
10.24
11.81
44.09
49.56
39.55
42.36
40.88
46.32
45,26
7.49
6.65
43,50
28.74
52.69
7.39
9.91
52.59
60.50
41.61
38.72
20
58.05
45.59
OSSCAR
7,46
9.19
122.63
17.40
17.03
7.16
9.57
51.99
59.55
33.29
44.27
42.19
55.94
45.95
FLAP
8,31
7.50
15.64
12.65
12.30
8.01
13.59
$4.37 49.36
44.01
47.41 49.20
52.64
48.83
Prrunee
7.51
5.79
12,53
13.14
10.38
7.08
7.87
61.16 54.64
51.72
58.50
58.66
60.22
60.61
ShonGPT
70.96
52.24
554.88
5.18:04
9.3E+04
2347 69
864,38 38
43.66
43
41.28
39. 16 38.97
35 80
42.52
LaCO+
87.77
96.00
494107
1645.83
429.78
462.92
42.88
SlieGPP
35.10
26.22
98.41
176.81
237.50
27.68
38.44
43.80
51.83
37.40
39.96
38.97
43.30
39.55
Wanda-sp
34.70
344.
2422.78
3627.00
31.85
74.87
33.59
37.5%
LLM Pruner
37.75
14.96
161.10
87.93
70.93
24.90
32.10
35.96
40.36
33.26
32.40
32.53
37.94
37.42
Zipl.M
13.01
50.36
125.98
14.01
47.53
34:77
OSSCAR
11.28
12.74
182.00
27.69
28.87
10.43
16,00
47.42
51.74
32.76
40.81
39.87
48.91
45.81
FLAP
12.41
11.33
26.05
22.61
21.54
11.81
27.01
43.51
48.54
39.28
41.51
43.07
44.990
45.57
Tyr-th--runnes
10.29
7.17
27.88
21.64
18.09
10.25
11.47
58.67 52.21
46.11
53.66
53.46
52.34
54.63
ShorGPT
226 40
187.23
23133.30
1473.71
1678.15
5532. 76
6804.52
36.99
39.47
34.09
37.51
36.52
35.05
38.00
LaCO+
256.71
1129.00
6019:01
2.1E-04
5.4E+04
6019.01
5.9E+04
41.79
35.21
33.28
33.93
35,25
SisoCPP
65.34
50.66
205.09
384.04
353.21
$4166
69.15
39.43
43.84 32.26
33.52
34.55
34.32
36.17
34.95
Wanda-sp
122.28
47.89
262.92
188.47
91.34
293.59
32.39
33.59
32.27
50%
I. M.
117.40
53.96
473.50
302.15
288.32
74.04
469.93
31.70
35.17
30.97
31.63
31.58
32.64
32,89
ZipLM
32.91
24.70
356.02
102.76
366.34
24.18
24.96
32.60
42.66
32.51
33 14
34.45
39.93
38.42
OSSCAR
28.41
44.17
320. 14
80.90
198.87
29.58
23.14
39.46
40.40
33.85
32.58
34.16
40.95
37.99
FLAP
25,49
16.89
272.98
82.12
134.28
34.81
79.46
29.84
44.04
33.29
38.68
36.59
40.57
39.34
Tyr-the-Pruner
16.17
9.59
29.84
38.59
30.89
15.53
16.85
47.41
54.58
41.41
47.41
47.79
46.21
47.92
Experiments
3.1 Experimental Settings
Models. We conduct experiments using the widely adopted large language models Llama2, Llama3.x x. and Mistral [4 7 15), focusing on models with over three billion parameters. The pruning targets include attention heads and FFN neurons, which are applied to the Transformer backbone. The embed_tokent and 1m head layers remain unchanged.
Calibration. For calibration, we consider Web [31, a high-quality dataset curated from Common Crawl snapshots with rigorous deduplication and filtering. Specifically, we extract about 4M tokens (about Ik samples for a maximum input length of 4k) from its FineWeb-Edu subset to construct calibration samples, ensuring high data quality and efficiency.
Evaluation. We use perplexity as one evaluation metric for language comprehension performance 19. validated on the WikiTexx2 [27 test set. To evaluate the impact of compression across various downstream tasks, we report 0-shot accuracy on ARC 66. BoolQ [5]. HellaSwag [5T], OpenBookQA
28 RTE 43), and WinoGrande 33 tasks, well as 5-shot accuracy on the MMLU [33 benchmark.
Implementation details. We implement Tyr-the-Prunc with PyTorch 30 and leverage the Hug- gingFace Transformers and Datasets libraries 47 to manage models and datasets. For local pruning,

---

Table 2: Post pruning performance massive language models. Accuracy (%, higher is better) serves as the comparison metric. MMLU employed a 5-shot benchmark, while other tasks used O-shot benchmarks. The percentage of average accuracy maintenance after pruning recorded, with values 295% highlighted in green and values <95% in red. The best results are shown in bold.
Model
Spansity
Methed
Are-C
Arc-R
BoolQ
HellaSwag
OBQA
RTE
WinoGrande
MMLU
AYG
0%
NIA
51,44
82.71
83,73
61.77
37,40
67.87
77,98
68.79
67.22 (100%)
SliceGPT
68.39
38.40
50.20
(78%5
Llama-2-70B
50%
LLM Pruner
Zipl.M
46.67
21.93
29.08
77.61
82.26
43.18
56.94
26.26
34.00
14.00
68,95
51.62
75.61
49.25
23.77
$4.31
62.05 (9258)
32.39 (48%)
OSSCAR
48.21
78.37
81.99
57.00
32,50
67.15
76.64
56.05
62.25 (9356)
FLAP
40,02
70.79
74.74
51.83
32,00
60.29
67.88
39.61
54.65 (8155)
Tyr-the-Pruner
60.04
70.76
60.58
(965))
0%%
NIA
60.58
87.29
85 $5.29
66.50
37,00
70.04
79.64
78.72
70.63 (100%)
SliceGPT
32,08
58.00
63.85
34,02
20.60
53,43
56,99
32,60
43.95 (62%)
3.1-70
50%
LLM- Pruner
OSSCAR
ZipLM
48.55
21.42
25.38
78.54
80.55
38.81
26.22
55.98
32.80
31.60
13.80
54.87
66.79
78.37
50.83
62,73
24.95
62.89 (8955)
32.04 (15%)
(89%))
FLAP
37.54
68.90
67. 34
43.98
26.40
60.65
72.30
$4.40
53.94 (765)
Tot-Ube-Pruncr Pruncr
56.74
85.40
85.20
64.07
36.40
71,48
78.91
70.29
68.56 (97%)
we iteratively prune and adjust weights by removing one attention head or 16 FFN neurons at a time.
The prune-and-seane process consists of 4 iterations, where the sparsity interval at the i-th iteration is set to 12.5% 2/22-/11 In each iteration, we explore 50 generations with 128 offspring candidates per generation. The sparsity shifts of the attention or FFN layers are independent to ensure the consistency of the sparsity interval granularity. Candidate validation is performed using the reciilaaaioniisppiee metric with vocabulary logits. We follow 36 to enhance validation efficiency: the 128 offspring are first validated on 2K tokens, and the top 16 are selected. These 16 survivors are then validated 16K tokens, from which the top 4 are selected, and finally, the best one is validated and selected on 128K tokens. To ensure fair comparison, we use the same Feb-WeebEEd samples for calibration to reproduce the baselines. The benchmark results of the baselines may outperform their reported results due to the improved calibration sample size and data quality. All experiments for Tyr-the- were conducted on 4 AMD IM MI250 (64GB) Accelerators, with models less than 13B parameters running on a single accelerator.
3.2 Performance
Language comprehension and downstream task performance of post-pruned LLMs. We applied structural pruning to various large language models using Týr-the- Prunerr at overall sparsity levels of 12.5%, 25%. 37.5%, and 50%. The performance was benchmarked against state-of-the-art methods, including ShortGPT (layer pruning) [24), LaCO+ (ShonGPT with LaCO layer merging) (50. SliceGPT (embedding dimension pruning) 2 Wanda-SP [. L.M--runer (23), ZipL.M
OSSCAR (26). and FLAP [[] Table I summarizes the comparative results, highlighting post- pruming performance in language comprehension and downstream tasks (cf. Section A.8 for detailed
results within each task).
Týr-the-Prunee Pruner demonstrates competitive performance across various sparsity ratios and LLMs. It consistently achieves 12.5% of Llama-3-8B's parameters yields the lowest perplexity (7.39) and the highest average downstream accuracy by 8.0% and 2.6%. At higher sparsities (237.5%), maintaining performance poses a significant challenge for existing methods, with advanced techniques like OSSCAR often exhibiting perplexities exceeding 100 and accuracies dropping below 40%. Sir-the-Prunee by contrast, excels under these conditions. For example, at 37.5% sparsity, the pruned fistral-Nemi model achieves perplexity of 11.47 and an accuracy of 55.63%, substantially outperforming ZipLM and FLAP.
results at low sparsity ratios (525%). For instance, pruning
surpassing the previous advanced methods, LLM-Pruner and LaCO+,
Scale up to massive language models. Structural pruning of massive language models challenges post-pruned performance and resource budgets. We incorporated CPU offload policy into typical baseline methods to ensure fair comparison on 70B-scale models. Table[ccompare the post- postnigu performance of Llama-2-70B and Llama-3.1 3-70B 50% sparsity.
Experimental results demonstrate Tyr-the- Tyr-the-Pruner's strong scalability under high sparsity for massive models. LLM-Prune' shows clear scaling limitations, maintaining only 48% accuracy when pruning
7

---

Table 3: Inference efficiency of post-pruned LLMs with Týr-the-Pruner. Benchmarks were conducted on a single AMD Instinct MI250 accelerator using PyTorch (HipBlas) for LLM inference, with input and output sequence lengths to 2048.
Table 4: Ablation study on local pruning. Wikitext2 perplexity and 0-shot accuracy on ARC-C, ARC-E, and BoolQ are reported.
Method
Cunliguration
Wikitat22 ARC-C
ARC-E
BoolQ
PLAP
134.25
33,99
43,18
$2.29
Defsallt
58,09
24.06
58.67
63.46
Model
Spursity
PParams
Decode Throughput
Wistterz Calibrated
09:00
30.05
54.54
01.71
0%
8.0B
2.49 000)
12.27 (1.00)
Local Prening
C4 Calibeated
wie grogressive pezniig
73.07
63.48
23.38
21.42
57.58
56,65
62.17
62.17
Llama-3.1-88
25%
6.1B
1,94 (1.28x)
14.13 (1.15x)
wil Howisn
109.88
22.53
51.68
46.45
50%
4.3B
1,42 (1.75x)
16.97 (1.38x)
wip Gradient
67.31
25.60
57.83
62.17
Mistral-Neme
25%
50%
08
14.38
11.08
7,8B
3.34 (1x25x)
4,16 (1.0l))
2.49 (1.67)
8.93 (1.34x)
6.68 (1.00k)
(1.13x)
Build Sugernet
Local Pruning
Defaelt
wel Einor Accun.
Kandem timor Accum.
Unitorm Fimor Accum.
200.92
$38.35
75.10
66.38
23.72
23.05
21.93
22.20
53,03
33.54
58.46
60.6
62.15
40.31
Llama-2-70B. In contrast, Týr-the-Pran Prrnterrrrrrreee achieves 97% accuracy maintenance when pruning Llama- 3.1-70B, outperforming alternative methods.
Inference efficiency of post-pruned LLMs. To evaluate the efficiency gains of post-pruned LLMs, we constructed inference benchmarks summarized in Table 3 For L.ama-3.1-88, 50% sparsity reduces time to first token (TTFT, in seconds) by 43% and boosts decode throughput (tokens/s) by 38%. These results highlight pruning as a key technique for inference optimization in large language models. More detailed efficiency analysis can be found in Section A.4
3.3 Ablation Study
The effectiveness of local pruning and supernet construction depends factors such as calibration samples, the implementation of local pruning, and error accumulation. Table 4 presents ablation study evaluating these factors for pruning Llama-3. 1--BBBBB-BBBB- 50% sparsity. Experimen- tal results show that FineWeb-Edu is consistently preferred as a calibration source, emphasizing the importance of selecting high-quality calibration samples. The presence of both first- and second-order optimization information and progressive pruning significantly impacts accuracy, demonstrating their necessity. Furthermore, the proposed expectation error accumulation approach outperforms alternatives, showcasing its ability to make sparse structures mutually aware appropriately.
Prune-to-supernet.
Evolutionary search direction. To assess the impact of search direction on final performance, we compare the effects of minimizing single- task losses versus our similarity-based metric when pruning 50% of Llama-3.1-8B's parame- ters, as shown in T Experiments show that single-task search underperforms our metric, which achieves optimal accuracy by calculating the simi- larity across activations from the first, median, last, and logits layers, requiring 96 GB for hidden activation checkpointing, Due to this overhead, the logits-only metric was favored, maintaining strong performance with reduced resource demands.
Table 5: Ablation study on search direction. Wikitext2 perplexity and 0-shot accuracy on ARC-C, ARC-E, BoolQ are reported.
Search Direction
Wikitex12
ARC-C
ARC-E
BualQ
Wikitexr2 Perplexity
17.22
29.69
64.06
62.23
Finzweh-FFih Falu Perplexity
31.65
31.06
64.18
62.17
Simillaiity based
65.87
Similarity based Logits only
30.89
31.83
65.36
64.62
Týr-the-Pruner vs. fine- grained search-only strategy. Figure 3 demon- strates the advantages of Týr-the-Pruner over the search-only strategy in efficacy and efficiency in identifying the optimal 50% sparsity distribution on Llama-3.1-8B. In which the search-only strategy uses fine-grained 3. 1.55% sparsity interval. Exper- imental results show that Prr-ter-Pruuerr achieves similar convergence trend as the search-only strat- egy but with faster convergence, fewer generations, and reduced search time. Additionally, the final post-pruned model discovered by Tyr-the-Prumn outperforms the search-only strategy, with an av- erage accuracy of 47.79 compared to 43.58. Our evolutionary search maintains time efficiency, with single generation requiring only 190 seconds.
Effect comparison:
Search Time (Minutes)
192
394
576
758
960
1152
3.4
Tosthollrane Band
Tyndh--Frueer hasc Ti- Pruner Bas-
Soarch Only
Matiic
1.2
Thy lio-2
8.3
300
Generation
Figure Syr-the-Prunce has faster conver- gence, fewer exploration generations, shorter search time, and better search outcomes com- pared to the fine-grained search-only approach.
8

---

Table 6: Týr-the- Prunner progressively refines and optimizes the sparsity distribution in iterations, steadily enhancing performance.
Method
Wikitext2
BoolQ
HellaSwng
OBQA
WinoGrande
MML.U
w/o search
66.38
23.55
58.46
62.35
32.51
16.60
51.26
52.88
28.34
40.74
search-only only
27.96
25.34
59.30
64.71
36.52
22.20
55.23
56.20
29.17
43,58
Týr-the-Prume II
28.92
26.45
56.19
62.17
37.05
22.20
50.54
56.75
29.29
42.58
Týr-the-Prunce Plauerr 12
63.51
23.80
50.54
56.85
30.23
44.37
Týr Pruner 13
29.75
29.86
63.09
64.62
39.28
25.00
$1.62
59,51
31.62
45.58
Týr:
47.79
Týr-the- Sy--the-Pruner! 15
29.56
31.87
65.57
66.69
39.08
26.40
58.84
60.89
33.67
47.88
To further examine how Týr-the-Pruner refines model performance across successive iterations and to highlight its advantage the search-only strategy, report the 50% post-pruned results of Llama- 3. 3.1-8B on multiple tasks: Wikitext2 perplexity (4), O-shot accuracy (f) on Arc, BoolQ, HellaSwag. OBQA, RTE, and WinoGrande, and 5-shot accuracy (1) on MMLU, as summarized in Table
Experimental results underscore the superiority of Týr-the- Prennhreerruerrrre compared to the search-only strat- egy. Under isotropic pruning, the w/o search baseline consistently delivers suboptimal performance across all tasks. By contrast, Tyr-the- Prr-the-PPrueer begins to surpass the search-only approach as early as the second iteration (I2). highlighting the advantage of progressively refining the sparsity distribution. The search-only method, constrained by the vast search space, suffers from extended search times and limited effectiveness. Tyr-the- Pruner achieves an ideal result at the fourth iteration (I4). Although performance continues to improve at the fifth iteration (15), the gains diminish, making four iterations the default configuration.
3.4 Compatibility with Quantization and Unstructured Sparsity
Fýr-the-Prunet is compatible with further compression techniques. Table 7 reports results of applying quantization and unstructured sparsity to the 50% pruned Llama-3. -8B model produced by Týr- the-Pruned The results show that quantization preserves over 99% of the pruned model's 's accuracy, while unstructured sparsity at 50% and 2:4 granularity achieves performance consistent with prior work 9 25). These findings indicate that Mrr-the-Prunce provides a strong foundation for multi-stage compression pipelines, retaining both high accuracy and robustness to further compression.
Table 7: Quantization and unstructured sparsity applied to the 50% pruned Llama-3.1 1-BB with Wyr-the-Prunce Reported Wikitext2 perplexity and accuracy (%) across benchmarks.
Method
Wikitext2
Are-C
Are-E
BoolQ
HellaSwug
OBQA
RTE
WinoGrande
MMLU
AVG
Týe
(100%)
AWQ ETT @ W1A16
34.79
31.06
61.65
66.09
39.34
24.60
60.77
60.77
31.43
17.34 (5699 I)
SmoothQuant (99 @ WBAB
31.31
31.23
64.94
65.50
40.18
15.80
58.84
61.80
32.45
47.59 (5-99.6)
RTN FPSE4M3
31.05
31.31
65.28
66.15
40.17
24.50
58.84
61.18
32.28
17.55 (5699.5)
SparsGPPT [ @ 50%
47.05
27.99
59.39
65.32
37.38
23
53.07
61
28.32
44.57 (5993.3
ALPS MAN2
21.40
52.71
(9687.3)
3.5 Memory/Storage Efficiency Analysis of Tyr-the-Pruner
Tyr-the-Prunce employs a supernet search tech- nique, where storing a large-scale supernet in memory is costly. To address memory concerns, we optimize our approach by storing pruned sub-
Table 8: Resource requirements of Tyr-the-Pruner:
Submodules
Usage
Storage Usage
7-8B
576
14-16GB 26GB
39.6GB
structures disk instead of in high-bandwidtth memory (HBM). An integer Python list is used track the currently selected substructures, en- suring that only one entire LLM is loaded into HBM at any given time (e.g., the 7B model uses approximately 14GB, and the 13B model uses around 26GBB. and disk storage occupancy. Furthermore, since there is no dependency between iterations (iterative prune-and-sed phase), the storage from previous iterations can be cleaned, further minimizing disk usage. Due to the low cost of disk storage, these memory and storage demands are highly acceptable.
66.6GB
70B
1440
140GB
414,7GB
detailed data on HBM

---

4 Related Work
Pruning techniques for compressing large language models. The growing complexity of Transformer- based language models, now reaching hundreds of billions of parameters, has intensified the necessity for effective pruning strategies. Pruning methods are generally divided into unstructural and structural approaches. Unstructural pruning [9] achieves high accuracy by selectively zeroing individual elements in the weight. However, it often requires specialized hardware, such as 2:4 sparse tensor cores 154, for end-to-end acceleration. Structural pruning enables hardware-agnosse acceleration by removing entire weight groups, but it may result in a pronounced loss of accuracy.
Structural pruning of LLMs can be approached as local optimization, alleviating memory constraints from loading the full model. ZipLM [17] accelerates inference by leveraging the Optimal Brain Surgeon (OBS) [12) theory, pruning weights to minimize the impact on the Hessian matrix and adjusting the remaining weights to reduce layerwise loss. Building on ZipLM, OSSCAR (26) introduces a permutation search between pruned and remaining weights within each layer, further reducing pruning- inducedd loss. Some approaches apply global optimization strategies to prune LLMs, overcoming local constraints, enabling customized sparsity distributions, and potentially finding optimal solutions. Fisher information was introduced saliency metric guide structure pruning via global dynamic programming [18]. LLM--runer [23 defines broad substructure dependency groups and then evaluates their saliency to guide pruning. FLAP [ uses global metric that considers both weights and activations for sparsity allocation, followed by layerwise pruning and bias adjustments to mitigate pruning losses.
Additionally, there is growing interest in embedding dimension 2 and depth 24
50 pruning
techniques for LLMs. Some training-aware structural pruning methods [52 20 29 are also gaining attention, as they further enhance pruning effectiveness by considering training dynamics.
Neural architecture search (NAS) for LLM compression. Several studies have applied NAS to compress L.Ms,, seeking architectures that reduce inference costs while maintaining accuracy. multi- objective NAS has been employed explore various search space definitions, identifying compressed LLM architectures that enhance efficiency and accuracy when fine-tuned on specific downstream tasks [16). LLLMAFlex
4 fine-tunes LLMs into supernets with Gumbel softmax- -based trainable subnet router, realized a "rain once, deploy many" model compression. EvoPress (36) proves that evolutionary search can determine suitable layerwise compression configurations and extends this method to support mixed-precision quantization and non-isotropic unstructural sparsity.
This paper presents novel structural pruning framework, Týr-the- -yr-the--runer, for large language models. Unlike conventional methods, this framework searches for the optimal sparsity distribution within a supernet. Through enhanced supernet construction and an iterative prune-and- and-search technique, it achieves end-to-end global pruning optimization with strong efficiency and efficacy, setting a new benchmark for post-pruning accuracy maintenance.
5 Limitations
Týr-the- Prunerrr achieves state-of-the-an structural pruning outcomes by constructing reliable supernets and employing iterative prune-and- ande--and- process. We have significantly reduced the search space and the number of generations explored. However, the search time cost remains non-negligible. Fair time costs in model compression are often considered acceptable, as the goal to achieve sufficiently optimized pruned model. However, we will continue to optimize it in future work.
6 Conclusion
This paper introduces Syr-the-Pruner an end-to-end global structural pruning framework for large language models. By constructing supernet through local pruming across various sparsity ratios and using evolutionary search to identify the optimal subnet, our framework achieves the optimal sparsity distribution under a target overall sparsity ratio. We propose an effective local pruning and an expectation error accumulation approach to enhance supernet construction. Additionally, an iterative ardne-hnd-search strategy with coarse-to-fine sparsity granularity ensures rapid convergence. Extensive experiments show that Tyr-the-Prum outperforms state-of-the-an methods, achieving 50% parameter pruning while retaining 97% accuracy on Llama-3.1
10

---

References
[1] Yongqi An. Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Pluctuation-based adaptive structured pruning for large language models. In Michael J. Wooldridge. Jennifer G. Dy, and Sriraam Natarajan. editors, Thirty-Eighh AAAI Conference on Artificial Intelligence. AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 10865- 10873. AAAI Press, 2024.
[2] Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari Do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
[3] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Lan- guage models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Ticn Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020. NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[4] Ruisi Cai, Saurav Muralidharan, Hongxu Yin, Zhangyang Wang. Jan Kautz, and Pavlo Molchanov. Llamaflex: Many-in-one Ilms via generalized pruning and weight sharing. In The Thirteenth International Conference Learning Representations. ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025.
[5] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLTT 2019, Minneapolis, MN. USA. June 2-7, 2019, Volume (Long and Short Papers), pages 2924-29322 Association for Computational Linguistics. 2019.
[6] Peter Clark, Isaac Cowhey, Oren Etzioni. Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. Stii:110330045574 2018.
[7] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal. Anthony Hartshorn, Aobo Yang. Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang. Aurélien Rodriguez. Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia- Olano, Diego Perino, Dieuwke Hupkes. Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve. Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen. Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jacwon Lec, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadcokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The llama herd of models. CoRR, abs/2440.211 21783, 2024.
[8] Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training guantization and pruning. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35; Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022.
[9] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 10323 10323-10337. 10337. PMLR, 2023.
11

---

[10] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: accurate post-training quantiza- tion for generative pre-trained transformers. CoRR, abs/2210.17323, 2022.
[11] Shangqian Gao, Chi- Heng Lin, Ting Hua, Zheng Tang. Yilin Shen, Hongxia Jin, and Yen-Chang Hsu. DISP-LLM: inteeniionnnddeedeeent structural pruning for large language models. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, edi- tors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
[12] Babak Hassibi and David G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Stephen Jose Hanson, Jack D. Cowan, and C. Lee Giles, editors, Advances in Neural Information Processing Systems 5, [NIPS Conference, Denver, Colorado, USA, November 30 December 3, 1992]. pages 164-171. Morgan Kaufmann, 1992.
[13] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference Learning Representations (ICLR), 2021.
[14] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[15] Alhert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample. Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux. Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/23 10.06825, 2023.
[16] Aaron Klein, Jacek Golebiowski, Xingchen Ma, Valerio Perrone, and Cedric Archambeau. Structural pruning large language models via neural architecture search. AutoML Conference 2023 (Workshop).
[17] Eldar Kurtic, Elias Frantar, and Dan Alistarh. Ziplm: Inference- aware structured pruning of language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[18] Woosuk Kwon, Sehoon Kim, Michael W. Mahoney, Joseph Hassoun. Kurt Keutzer, and Amir Gholami. A fast post-training pruning framework for transformers. In Sanmi Koyejo, Mohamed, A. Agarwal. Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans. LA. USA, November 28- December 9, 2022, 2022.
[19] Qi Le. Enmao Diao, Ziyan Wang. Xinran Wang. Jie Ding, Li Yang. and Ali Anwar Probe pruning: Accelerating Ilms through dynamic pruning via model-prohing. arXiv preprint 255iv:2502.15618, 2025.
[20] Shengrui Li, Xueting Han, and Jing Bai. Nuteprune: Efficient progressive pruning with numerous teachers for large language models. CoRR, abs/2402.09 09773, 2024.
[21] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang. Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: activation-aware weight quantization for on-device LLM compression and acceleration. In Phillip B. Gibbons. Gennady Pekhimenko. and Christopher De Sa, editors. Proceedings of the Seventh Annual Conference on Machine Learning and Systems, MLSys 2024. Santa Clara, CA. USA. May 13-16. 2024. misys.org. 2024.
[22] Yuqiao Liu, Yanan Sun, Bing Xue, Mengjie Zhang, Gary G Yen, and Kay Chen Tan. survey on cvolutionary neural architecture search. IEEE transactions on neural networks and lcarning systems, 34(2):550-570, 2021.
[23] Xinyin Ma, Gongfan Fang, and Xinchao Wang. Ilm-pruner: On the structural pruning of large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NeurIPS 2023. New Orleans, LA. USA, December 10 16, 2023, 2023.
[24] Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang. Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. CoRR. abs/2403.03853, 2024.
12

---

[25] Xiang Meng, Kayhan Behdin, Haoyue Wang, and Rahul Mazumder. ALPS: improved optimization for highly sparse one-shot pruning for large language models. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang, editors, Advances Neural Information Processing Systems 38; Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10- 15, 2024, 2024.
[26] Xiang Meng, Shibal Ibrahim, Kayhan Behdin, Hussein Hazimeh, Natalia Ponomareva, and Rahul Mazumder. OSSCAR: one-shot structured pruning in vision and language models with combinatorial optimization. In Forty-first International Conference on Machine Learning, ICMI 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
[27] Stephen Merity, Caiming Xiong. James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[28] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. EMNLP, 2018.
[29] Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary. Mohammad Shocybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation. CoRR, (bs/2400.114679 2024.
[30]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary Vito,o Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai. and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Hanna M. Wallach. Hugo Larochelle, Alina Beygelzimer, Florence d' AlchéBBuc, Emily B. Fox, and Roman Garnett, editors. Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurlPS 2019, December 8- 8-14, 2019, Vancouver, BC, Canada, pages 8024 8035, 2019.
[31] Guilherme Penedo, Hynek Kydlícek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. CoRR, abs/2406. 17557, 2024.
[32] Baolin Peng. Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao, Instruction tuning with gpt-4. arXiv preprint MXXXx:2304403277 2023.
[33] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint 19X. 10641, 2019.
[34] Noam Shazeer. GLU variants improve transformer. CoRR, abs/2002.05202, 2020.
[35] Xuan Shen, Pu Zhao, Yifan Gong, Zhenglun Kong, Zheng Zhan, Yushu Wu, Ming Lin, Chao Wu, Xue Lin, and Yanzhi Wang. Search for efficient large language models. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10- 15, 2024, 2024.
[36] Oliver Sieberling, Denis Kuznedelev, Eldar Kurtic, and Dan Alistarh. EvoPress: Accurate dynamic model compression via evolutionary search. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste- Julien, Felix Berkenkamp. Tegan Maharaj. Kini Wagstaff. and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 55556-55555 55590. PMLR, 13-19 Jul 2025.
[37] Mingjic Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
[38] Shengkun Tang, Oliver Sieberling, Eldar Kurtic, Zhiqiang Shen, and Dan Alistarh. Darwinlm: Evolutionary structured pruning of large language models. arXiv preprint arXiv:2502.0778 2025.
[39] Yehui Tang, Yunhe Wang. Yixing Xu, Yiping Deng, Chao Xu, Dacheng Tao, and Chang Xu. Manifold regularized dynamic network pruning. In IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2021, virtual, June 19-25, 2021, pages 5018 5028. Computer Vision Foundation IEEE, 2021.
13

---

[40] Yehui Tang, Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing Xu. Chao Xu, and Chang Xu. SCOP: scien- tific control for reliable neural network pruning. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Ilsuan-Tien Lin, editors, Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[41] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton- Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas. Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee. Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Erie Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama Open foundation and fine-tuned chat models. CoRR, 0aw/2307.09285. 2023.
[42]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg. Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-59989999 6008, 2017.
[43] Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans. LA, USA. May 6-9, 2019, OpenRe- view.net, 2019.
[44] Boyao Wang, Rui Pan, Shizhe Diao, Xingyuan Pan, Jipeng Zhang, Renjie Pi, and Tong Zhang. Adapt- pruner: Adaptive structural pruning for efficient small language model training. CoRR, abs/2502.03460, 2025.
[45] Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. SVD-LLM: truncation-aware singular value decomposition for large language model compression. CoRR, abs/2403.00378. 2024.
[46] Yuxin Wang. Minghua Ma. Zekun Wang. Jingchang Chen, Liping Shan. Qing Yang. Dongliang Xu. Ming Liu, and Bing Qin. CFSP: an efficient structured pruning framework for Ilms with coarse-to- fine activation information. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugemio, and Steven Schockaert, editors, Proceedings of the 31st International Conference Computational L Linguistics, COLING 2025, Abu Dhabi, UAF, January 19-24, 2025, pages 9311-9328. Association for Computational Linguistics, 2025.
[47] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander
Rush. Transformers: State-of -he natural language processing. In Qun Liu and David Schlangen, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, EMNLP 2020 Demos, Online, November 16-20, 2020, pages 38 45. Association for Computational Linguisties, 2020.
[48] Shangyu Wu, Hongchao Du, Ying Xiong, Shuai Chen, Tei-Wei Kuo, Nan Guan, and Chun Jason Xue. Evop: Robust LLM inference via evolutionary pruning. CoRR, Abs/2502..19100. 2025.
[49] Guangxuan Xiao, Ji Lin, Mickaël Seznee, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 38087-38099, PMLR, 2023.
[50] Yifei Yang, Zouying Cao, and Hai Zhao. Laco: Large language model pruning via layer collapse. In Yaser Al- Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pages 6401-6417, Association for Computational Linguistics, 2024.
14

---

[51] Rowan Zellers. Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
[52]
Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang. Linlin Ou, Xinyi Yu, and Bohan Zhuang. Loraprune: Structured pruning meets low-rank parameter-efficien fine-tuning. Lun-Wet Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics. ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 3013-3026. Association for Computational Linguisties, 2024.
[53] Wayne Xin Zhao, Kun Zhou, Junyi Li. Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang. Zican Dong. Yifan Du, Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang. Ruiyang Ren, Yifan Li, Xinyu Tang. Zikang Liu, Peiyu Liu, Jian-Yum Yun Nie, and Ji-Rong Wen. A survey of large language models. CoRR, abs/2303. 18223, 2023.
[54] Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning N: fine-grained structured sparse neural networks from scratch. 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction sections offer comprehensive discussion of the manuscript's context, intuition, and ambitions, as well as its contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of this work are discussed the end of the manuscript.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification:
This manuscript uses existing theories as a basis, while providing complete
and correct derivations.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The pipeline the methods and the details experiments are presented with corresponding reproducible credentials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: At this stage, the experimental setup and details are sufficient guarantee reproducibility. Further materials will be made available future updates.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc
aublic/guudees/Coeetbbbmsss wtt.:eeesuumiisioooooooooooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wttps/cccppppclggui.aeaeaeccuuea
Stteeee//ooSSSSeissSioica www.ssssiicggcooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The pipeline of the methods and the details of experiments are presented with corresponding reproducible credentials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
The results contain the standard deviation of the results over several random
runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: The paper provided sufficient information on the computer resources needed reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed: This paper discusses the lightweighting of L.MSs that can reduce the consumption of resources without obvious broader impacts.
19

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The original owners assets, including data and models used in the paper, are properly credited, and the licenses and terms of use explicitly mentioned and properly respected.
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
20

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the palaaets.paperssttt paperssi thcode. com con/datases package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurips: hect::::/eerrrps.ccc//oofeereeee Cc//Confereeeee//222///.... for what should or should not be described.
22

---

A Appendix
A.1 Theoretical Foundations of Local Pruning
Redundant channel identification. We consider first- and second-order terms to minimize Equa- tion (3. For the first-order term, we identify the to-prune channel p by argminw,,. (G,, W
which identifies the weights with the minimal contribution in the gradient direction 23 For the
second-order term, we employ the Optimal Brain Surgeon (OBS) method [2], which optimizes
W by considering the inverse of the diagonal elements of the Hessian matrix.
argminw...
This method measures each channel's contribution to the curvature of the loss function.
The identification metric for redundant channels is derived from manual design that takes into account both first- and second-order optimization information, distinguishing it from previous work. Table demonsteessssssttss the validity of metric by ablation.
Weight adjustment. We minimize Equation (3) by applying the Lagrange multiplier method to impose constraints on the p-th channel should be pruned (6W, W Wp.):
L(oW, A) = G ow oW HoW + X (oW,.: W Wp.)).
Under the constraints, the resulting loss function L(OW, 2) will be differentiated with respect to &W and À to find the minimum value:
OL(oW, A)
G + H&W + E,A 0, OC(OW. A)
OsW
(8)
SW,, p. W. =0
0X
For the G HOW + E, term, we use p and ~ p to denote channels prune and channels to remain. Corresponding variables can be expanded in this way:
Gp. Gipp::
Hp H Happ
H HpPp =0 0 H H.p.op
][
OW p.s
A 0
P.P
0,
(9)
8W
Ap.
where the elements of the Hessian matrix corresponding to the pruned positions can be set to zero (when a channel of the weights is pruned, the same position of the Hessian/invHessian matrix are pruned correspondingly X Overall, the solution is 8W -p,
H Pp.p
- G.p..-
Fast update of inverse Hessian matrix. When the p-th channel pruned, the inverse Hessian matrix H 1 must be updated to account for the removal of the corresponding channel p in W. This update can be efficiently derived by leveraging the properties of partitioned matrices and applying the Maaaa--orriso--orr
formula. The main idea is that the pruning of the p-th channel results in rank-1 update to H which is mathematically represented as:
H -
H H
(10)
By updating the inverse Hessian with rank- I adjustment, the influence of the p-th channel is properly removed through the outer product of the corresponding column and row vectors, using the reciprocal of the p-th diagonal element. The updated H-1 ensures consistency for the remaining channels, enabling efficient and scalable pruning operations. This method has time complexity of ((d)). avoiding full recomputation of the inverse and ensuring computational efficiency.
23

---

A.2 Algorithms for Týr-the-Prunes Pruner
Algorithm Function local pruning
Algorithm 2 Function prune to to_suuneeeett
1: Inputs: to-prune weight W,
Inputs:
LLM weights (W,, W...., Wz}.
input activations sparsity S,
sparsity ratios
{St...... .t.........E)
pruning granularity (pruning times) K
input activations for first weight X,
Mask < ones like(W)
pruning granularity (pruning times) K
3: H + X
2: for l 1 to do
4: G HW
3: X_list +
for <
do
4: for e <- to E do
6: p p argginn,,
GG.. W
W W
5: We local pruming We, X, St K)
2
6:
store(
7: Mask, +- 0
7: X Jistt. append We.e)
W up. f- W Wipp: + H prup G.pp:
8:
end for
9: H -H
H H
X -e 1-Sc.
listle]
H-1
(1-((1
end for
10: end for
11: Return Mask @ W
11: Return (Weelest.es)
Algorithm 3 Function evolutionary_scane
1: Inputs: sparse structures W = [W...... Wi.,, WL.E), sparsity ratios (5.) sparsity interval 8
2: procedure
(5,) Sn)
3: Candidates +
4: for 1 to veurCaniidaees do
5: Candidates. ( suiis (
6: end for
7: end procedure: return Candidates
8: (5e) + {Se)
9: for - to numGenerations do
Offsprings + mmkeCnndiiaessnnnmnCaadnneee W, (, (S:
(8:) +
Return (Se)
Algorithm Function Tyr-the-Prunee
1: Inputs: LLM weights { (W1, W }. imput activations for first weight X, pruning granularity (pruning times) K, overall sparsity S. sparsity interval 8"
sparse structures E. iterations T 2: procedure generateSparrstiis((. E, (S.) E
3: Sparsities = (
4: for & + to range(L) [) do
5: for <- 0 to range( E)) do
6: Sparsities. append(S, St ((e- 1) 0.5) x S 81
7: end
8: end for
9: end procedure: return Sparsities
10: (5ett + (S)z
11: fort +rr++l to d do
12:
Sparsities < generateSSparsities(..... (Se) S
13:
(W...
prune_to_supernee ((W, J/ Sparsities, X, K)
(8:) <- evolutionary search( [ 1.
15:
5" x 0.5
16: end for
Return compress( (W. a (8.)
24

---

A.3 Further Comparisons
To further demonstrate the ef- fectiveness of our proposed method, Týr-the-Pruner, we conducted a more comprehen- sive comparison. The com- petitors include the pure subnet search framework SearchLLM [35) the probe-based dynamic pruning approach ProbePrun- ing [19]. the sparsity distri- bution optimizer Adapt-Pruner [44], the coarse-and-fing com- bined approach CFSP [46], the calibration-free approach PruneNet [19], the structure- independent approach DISP- LLM [... the cluster-based evolutionary pruning approach EvoP 48), and the search- only approach DarwinLLM [8]. The experimental results, with competitor performance taken from their respective pa- pers, are presented in Table 9
Table 9: Further comparisons. Perplexity on Wikitext2 (lower is better) and O-shot accuracy (%, higher is better, DarwinLLM reported the 25-shot Are-C benchmark) serve as the comparison metrics. Optimal results are bolded.
Midel
Sparsity
Method
|Wikites2
BoolQ1 1 WinGGrsnde ARC-R
ARC-C1
0%%
N/A
5.68
71.38
67.01
67.45
41.38
Limr=788
2015
SeachLLM
6,89
70.98
74.92
64.23
36,51
25%
Tyr-ite-runner
7.36
75.81
78.68
66.36
42.06
0%
NDA
5.12
77.68
69.06
76.30
43.43
30%
PruneNet DISP-LLM LLM
6.85
61.09
62.27
59.51
53.20
33.19
53.51
Liana 7B
37.5% 40%
Tyu-teePruner Pruncr
10.29
68,07
66.93
71.13
38,31
ProbePerring
8.01
64.20
58.10
62.50
57,20
DISPLLLM DuwinLL Týr the Pruncr
9.84
58.41
43.06
25.85
50%
62.70
35,10
16.17
65.54
62.12
66.12
33,62
80.61
72.02
EvoP Tyr-tho-runcer
6.33
L-
DISP-EI
50.41
37.5%
Týr the Pruner
7.37
72.06
76.35
CFSP
62.33
50%
DISNLL
7.11
$9:37
52.57
tyr-tee Pruner
9.59
74.06
70.09
72.18
39.85
0%
NIA
5,84
$2.17
73.56
81.31
51.34
Llama 3.1 sB
Adlpe-Prunce
33.75
25.07
50%
Týt-te-Pruner
DunwinLM
30.99
62.20
66.64
61.50
57.30
39,50
65.86
34.20
31,83
0%
N/A
5.76
81.10
73.01
80.05
50.43
It is evident that Tyr-the- Pruner significantly outperforms other structured pruning methods, achieving better performance even at higher sparsities compared to other methods at lower sparsities. In particular, Prr-the-Pruner surpasses the search-based methods SearchI.LM, EvoP, and Darwinl demonstrating the effectiveness of our effective local pruning approach, expected error accumulation, and iterative prune-and ande-annd-searh strategy.
Lama 3 8D
40% MI Pamnly
PrnhePruring Týr the Pruncr
14.00
20.30
67.30
57.40
50.00
25%
13.14
76.02
71.11
75.63
42.15
A.4 Efficiency Analysis on Non-isotropic Structural Pruning
Large language models (LLMs) with non-isotropic pruning may be considered exhibit inferior inference efficiency compared to those with isotropic sparsity across layers. To explore, we provide comparative analysis of inference efficiency for L.lama-31 1-BB and Mistral-Nemo, both pre- and post-50% structural pruning. The evaluation was conducted on an AMD InstinetTM MI250 Accelerator using Pytorch (HipBlas), covering both prefilling and decoding tasks across range of sentence lengths, as illustrated in Figure 4
The variance (Var) quantifies the degree of variation in sparsity under non-isotropic pruning conditions; a larger variance indicates more fluctuation in sparsity across layers. As shown in Figure the 50% structural pruned LLMs achieve up to 1.3x or greater speedup in both prefilling and decoding tasks compared to their dense counterparts across most sentence lengths. Variations in layer sparsity do not have significant impact efficiency. A slight efficiency decrease is only observed when the variance reaches 1. In this case, the reduction in efficiency is likely due to the frequent high sparsity, which leads more bottlecceb "thin" matrix multiplications in the computational graph.
Figure 5 ()) and Figure (bbb compare the sparsity distributions of the MHA and FFN layers in Llama- 3.B after 50% pruning with Týr-the- Pruner and the search-only methods, respectively. The sparsity distribution obtained by Týr-the-Pruun Pruner resembles that of the search-only strategy, yet Týr-the-Pe Pruner performs better. Its search process is more refined, incorporating multiple rounds of expectation error accumulation, ultimately leading to a superior sparsity distribution and higher performance in the pruned model.
Figure 5 c) and Figure 5 d) compare the sparsity distributions of the MHA and FFN layers in the 50% post-pruned Llama-3. -8B across different iterations of Tyr- Tyr-the-Pruner. Týr-the-Prume identifies
25

---

3000
25
Profil Tokens
Deoods Tokenss
2000
1000
Donse
50% Var 0.5
Dense
50% Var
50% Leporopic
30% Var00.1
50% Leporopic
30%
Varm0.1
Var-1
Var-
128
128
2048
Sequenoe Leugth
Sequense
Length
(a) Llama-3.1-88 prefilling benchmarks
(b) Llama-3. 1-8B decoding benchmarks
1500
Decwde Tokess /s8
Profil Tokens
1000
Donse
50% Varmu.5
10
Dense
50% Var
50% lpptropcc
50% Var0.1
50% Loporopic
50% Var-0.1
Var=]
Var-1
128
Sequenoe Length
Sequence Length
(c) Mistral-Nemo prefilling benchmarks
(d)
-iimooal-Nemo decoding benchmarks
Figure 4: Pre- and post-pruning large language model inference benchmarks.
Search
Tyr the Prizer
0.6
Spurvity
0.2
0.2
Tir-he-Prrnne
0.0
10
20
30
10
20
50
Layer
Layer
(a) MHA: Syyr-the-Prnne vs. Search-only
(b) FFN: Týr-the- Pruner vs. Search-only
Dession
1.0
Desaiin
Desation
0.8
Sparitty
0.6
Sparity
0.6
0.2
Tonoioa 0
Torotion Burstor
+
Teestion 4
Binaton
10
20
30
10
20
30
Loyer
(c) MHA: variations during iterations
(d) FFN: variations during iterations
Figure 5: Sparsity distribution of Týr-the-Pruner and the search-only strategy on Alama-3.1-8P
relatively ideal and coarse-grained sparsity distribution in the first search (with a sparsity interval of 12.5%). In the subsequent iterations (2nd, 3rd, and 4th), with sparsity intervals of 6.25%, 3. 1,52%, and 1.5625%, respectively, the sparsity distribution is progressively refined and optimized, ultimately converging to optimal solution.
26

---

A.5 Sparsity Distribution of Different Pruning Methods
Different pruning methods vary in the distribution sparsity. Figure o[a) and Figure b show the sparsity distributions of MHA and FFN of Llama-3. 3.1-8B after 50% pruning by series of LLM structural pruning methods, respectively.
11.M-rruner
ZipLM OSSCAR
1.0
TLAP
Tyrnhh-Prrune
Spersity
0.6
Spunsiy
0.2
LLM--Punce
OSSC
FLAP
Tis Teethe Pruscr
0.0
0.0
10
20
30
20
30
Layer
Layer
(a) MHA
(b) FFN
Figure Sparsity distributions with different structural pruning methods.
ZipLM and OSSCAR maintain isotropic sparsity distribution. LLM-Pruner incorporates prior knowledge, recognizing that the shallow and deep layers of LLMs are more pruning-sensitive and thus preserve them while only isotropically pruning the intermediate layers. These three methods fail account for the unique characteristies of different LLMs, leading to clear suboptimal sparsity assignments. Conversely, FLAP combines local activations and weights to assess the global sparsity distribution, resulting non-isotropic pruning. While this method seeks balance between local and global sparsity, it does not fully address the gap between them, making it challenging to achieve an optimal sparsity distribution.
Týr-the-Pruner's sparsity distribution clearly differs from that of other methods. It directly searches for the optimal sparsity distribution the global level without the local and global gaps. The resulting sparsity distribution does not adhere to prior assumptions: for instance, the 2-nd FFN layer is largely retained, while the 12-th FFN layer is entirely pruned, and there is discernible pattern in the sparsity ratio as layers become deeper or shallower. This demonstrates that model optimization should fully account for the model's unique characteristics.
A.6 Prune and Tune
Table 10: Results of pruning and finetuning. Perplexity Wikitext2 (lower is better) and accuracy (%, higher is better) serve as the comparison metrics. MMLU employed a 5-shot benchmark, while other tasks used 0-shot benchmarks. An asterisk (*) indicates fine-tuned result. Optimal and
suboptimal results are bolded and underlined, respectively.
Sparsity
Method
Wikitex22
Avg. Acc.
Arc-C t Arc-E BoolQ
HellaSwag
OBQA
KTE
WinoGrande"
MMLU
0%
N/A
5.84
64.77
51.54
81.31
82.17
60.04
33,20
71.12
73.56
65.20
LLM- PaMPrunne
70.93
32.87
19.68
32.07
40.03
27.55
13.20
52.71
50.83
26.88
Pruner*
NutePrunc*
14.31
51.33
36.77
62.92
68.78
49.45
29.20
58.12
66.93
38.49
37.5%
FLAP
21.54
43.07
23.98
52,15
64.62
36.50
23.40
55. 60
$8.17
30.17
"yr-the-Prunce
18.09
53.46
39.68
73,53
70:55
47.12
30.00
58. 84
66.54
41.43
Tyr-the Pruner*
12.65
58.22
46.93
77.02
75.75
54.99
32.60
59.57
68.82
30.06
LLM.
LLM-Pruncr*
27.34
40.39
25.68
$3.49
45.84
39.88
22.00
53.79
55.25
27.22
N
23.55
42.71
30.01
53.84
55.61
38.02
24.20
55.23
58.25
26.50
50%
FLAP*
51.29
43.01
29. 18
53.32
60.83
37.15
22.00
56.68
57.14
27.77
Tyr-the-Pruner Pruner
30.89
47.79
31.83
65.36
66.64
39.99
24.80
58.12
61.80
33.76
Tyr-the-Pruner
38.65
67.25
Structured pruning is often followed by parameter-efficient fine-tuning to restore model performance 23 40 20). To evaluate the benefits of our proposed method, Sfr-the-Prunets in the context of
27

---

post-pruning fine-tuning, we conducted fine-tuning experiments on the Llama-3.1-3 3-BB model. The fine-tuning employed the parameter-efficieet method LoRA (rank=16) [14] and the post-prunce
L.M was fine-tuned for three epochs on the Alpaca-GPT4 dataset 22
Table TO presents the experimental results. Týr-the-Prunes demonstrates superior performance at both 37.5% and 50% sparsity, with many of the results without fine-tuning already outperforming those fine-tuned by other methods. Furthermore, Syr-the-Pruner surpasses the state-of-the-art training- aware pruning method, NutePrune 20
A.7 Statistical Significance Analysis
To verify the robustness of the pro- posed method, Týr-the-Pruner, we ad- just the random seeds (the change of random seeds triggers the change of calibration samples and the change in random sparsity shift) for multiple (number of tests: n=5) experiments and observe the error bar (# standard deviation), as shown in Table
Table 11: Statistical significance analysis for Týr-the-Prune
Model
Sparsity
Wikitext2
BoolQ
ARC-E1
ARC-C1
Llama-2-78
25%
7:51
0.07
69.45
0.04 75.13
0.10 42.58 0.09
Ilama 13B
5.79
81.35
0.06 77.74
10.38
0.11
76.36
0.12
0.09
45.48
0.06
Llama-3.1-88
50%
30.89
0.21
66.64
0.26
65.86
0.33
31,83 0.16
From the global observation of experimental results, the proposed method performs relatively consistently in multiple randomized trials, with standard deviations within acceptable limits (< 0.21 for Wikitext2 perplexity and < 0.33 for downstream performance). From the local observation of experimental results, it can be seen that pruning yields more stable performance for larger models or under lower sparsity ratios.
28

---

A.8 Detailed Downstream Task Results
Table 12: 0-shot ace (%) on ARC-Challenge.
Table 13: 0-shot ace (%) on ARC-Easy.
Sparsity
Methed
7B
2
136
2:3
LLaMA-3.
0:86
3.x
1-R5
7B-v0.
Mastral
Nemo
Spansiry
Method
70
2LaMA
136
2:35
LLaMA
0:80
3.x
1-R6
7B-v0.3 Nemo
Mistral
096
NAA
43.43
48.46
42.32
50.43
51.50]
48.81
55.72
N/A
76.30
79.46
74.49
80005
80.31
79:67
StooGGP
36.18
43.86
37.50
43.94
44.20
43.45
46.16
ShootaPP
65.87
75.55
68.10
71.17
72.18
71.51
75.84
LaCO+
37,97
43.00
37,63
43.00
44.20
42.15
45.14
LaC
66.39
75.04
64,09
73.44
75.67
71.25
75.67
SliceGPP
41.81
15.15
42.15
31.66
SlieKiPT
73.40
77.78
74:71
76.85
Wands-sp
43.34
24.00
19.28
34.04
46.16
48.89
Wanda-sp
74.62
76.43
48.95
28.70
77.61
79.29
12.5%
MMMM-rrmner
38.40
44.45
11.53
58.57
57.97
40.02
43.52
12.5%
M-
72.05
77.
62.90
70.83
72.47
73.32
75.46
7:pl. M
41.55
49.15
18.23
40.19
19
42.49
47.27
$2.30
7ipl M
75.73
79.80
71.51
73.74
75.34
78.62
79.63
OSSCAR
40.78
30,80
OSSCAR
79.
74:37
78.28
FLAP
40,02
42.15
33:45
41.30
41.47
43.86
45.48
ELAP
71.38
72.69
64.44
73.36
74.16
73.76
75.88
T the Pruner
42,05
48.05
38,82
47.44
49.15
48.55
$4.35
Ttr the Pruner
75.84
79.62
72,94
79.08
79.80
79.84
81.61
ShonGPT
32.34
37.55
30.97
26.88
27.39
33,96
38.99
ShonGPT
52.74 61.24
49.58
38.85
43.18
52.57
63.30
LaCOs
30.89
12.34
36.77
33,87
40.10
LaCO+
53.03
64.72
53.41
55.47
52.23
62,50
SlaceGPT
37,88
41.47
28.33
35.32
37.46
38,99
24.49
SaceCPP
71.80
74,92
58.67
67.80
55.60
71.46
47.56
Wands-sp
38.14
20.52
18,09
16.89
19.20
37.57
23.21
Wanda-sp
70.41
33.59
37.46
42.26
25.41
70.83
$3.66
25%
L.MMPrrnner
22:35
24.57
40
M-u-Pruner
59
Zipl M
39.51
46.93
29.52
18:43
30:31
43.69
27. 27.39
Zigl
74.66
78.45
61,33
36.47
75.88
50.04
OSSCAR
40.53
45.65
18:17
27:47
23.4h
42.66
27.22
OSSCAR
74.45
77,57
27.95
53.70
40.03
75.59
51.60
FLAP
31.91
40.78
26.
00.54411
31.91
33.45
36.52
40.30
FLAP
64.23
69.23
53,96
60.31 65.95
67.22
68.69
Tyr the Pruner
42.58
35.41
45.48
48.38
Pruner
75.13
77.74
69.40
75.63
77.23
ShorGPT
28.58
31.14
25.85
25.38
27.56
29.10
27.90
ShonGGP
41.58
48.95
40.07
37.50
39.98
23.88
42.72
LaCO+
28.24
32.06
25.43
27.30
27.05
28.24
31.97
LaCO+
26.11
18.57
40.49
39:27
40.05
35,40
46.09
SPaceGPP
32,00
23:29
27.39
19.45
SaceGPT
62.75
67.47
58,46
Whidss-pp
27.99
21.76
20.30
20.90
19.97
30.65
20.31
Wards-sp
$7.03
32.37
26.94
25.72
25,00
47.90
35.90
37.5%
LLM-Prraer
17.58
24.40
17.49
16.99
16.9%
20.90
19:28
37.5%
LLM-PPrser
28.93
54.76
31.69
32.53
$2.07
47.10
47.9%
7ipl M
33.53
32.08
20.56
19:80
21.16
18.31
37.03
7ipl M
68.45
61.95
27:99
27.10
27.02
70.54
70.79
OSSCAR
23.9%
37.63
OSSCAR
62.16
54:31
71.04
70.08
FLAP
29.18
35.75
24:40
25.17
23.98
20.
32:51
FLAP
53.45
58,
46.55
46:63
55.15
56.65
61.70
Sy-tho-Pruner
38.31
43.26
30.97
38.99
39.68
38.31
42.41
Ty--ihePrunnr
71.13
76.35
64.52
72.56
73.55
71.38
75.51
ShonGiPT
23,46
28.16
21.84
23.29
23.56
26.19
32.22
ShonGaPT
32.45
37.75
30.39
31.23
32.83
32.87
36.28
L:CO+
21.25
22.87
21.67
37.71
28:38
20.18
SisceiPT
24.91
30.63
18.36
20.99
21.50
19.45
18.52
SiceCiP
48.90
54,84
36.20
41.33
41.62
43.56
35.25
Wanda sp
17.58
19.54
20.73
15.00
19:54
18.00
15.00
Wanda
27.95
25.86
26.59
30.98
30.47
12.79
35.94
50%
LLM-Pruaer
21.59
50%
LLM-Piuser
28.70
25.16
ZIPLM
20.14
27.99
19:20
17.15
20.48
23.72
21.16
ZipLM
29.38
54.00
27.57
25:57
25.28
50.84
49.96
OSSCAR
23.81
25.34
20.56
17:58
19.97
27.13
20.82
OSSCAR
50.72 45.92
22.15
28.07
26.05
59.22
41.92
FLAP
29,10
27.47
22.78
21.76
20.99
25.34
28.24
FLAP
47.01 43.18
27.23
42.30
43.18
52.61
52.57
Tyr-hh-Prrnner
25.51
31,83
32.59
Tyr-ihe-runner
72.18
56.23
65.36
66.37
Table 14: 0-shot acc (%) on BoolQ.
Table 15: 0-shot acc (%) on HellaSwag.
Spuriity
Method
78
LLaMA-2
136
2-18
LLaMA-3xx
1-85
7B--0-3 Nama
Matral
Spunity
Method
78
LLaMA-2
138
2-18
LLaMA-3x
(1:15
1-875
7B--90.3
Misrral Nemo
0%
N/A
77.68
73.00
67.17|
$2.17
R5:14
57.14 60.04
55.20
60.11
60.92
62.90
SheetGer
74.77
75.54
66.30
73.70
70.70
77.31
66:21
SheetCiPT
49.85 55.70
49.76
55.12
55.09
54.93
56.26
1aC0+
61.13
68:50
62.72
72.78
70.06
37.19
68.62
1,00+
51.11
56.23
48.94
54.51
54.68
54.79
55.95
SliceGPT
73.12
80.67
68.99
75.75
75.57
$1.19
77.71
SlseCPP
52.32
56.16
47.73
52.03
50.97
54.57
50.66
Wanda sp
71.68
51.90
63.09
77.31
63.04
56.53
53.96
27.29
55.97
12.5%
LLM--Pruner
80.43 76.48
65.72
74.34
71.90
72.72
77.58
12.5%
LLM Pruner
$1.60
37,06
43.62
69.63
50.02
51.28
52.34
ZiPLM
09.36
82.54
05.00
75.03
77.00
82.36
71.8
ZipLM
55.41
59.42
48.85
51.87
52.37
$7.92
54.77
OSSCAR
69.02
83.00
68.59
74.80
79.91
51.53
73.05
OSSCAR
55.39
59.53
48.60
51.25
57,83
54.05
FLAP
70.95
60100
73.49
71.87
77.49
50.24
FLAP
53.95
57.21
43.55
50.97
51.66
54.20
51.15
Wy-the Pruner
70.67
N2.78
72.33
80.12
90.24
$2.11
82.94
WWrthe- Pruncr
55.88
59.39
$1.55
56.52
56.32
58.27
59.31
SheriGPT
62.17
62:54
44.83
37.80
37.60 65
67.25
67.22
ShonGGP
41.94
47.70
37.31
28.59
28.37
42.51
43.61
1,00+
66.70
42.16
39.36
43.07
42.75
SliceGPT
68.93
79.27
65.81
72.02
67.60 68
75.78
63.41
SlicePPT
46.16
49.84
39.29
43:29
42.10
44.24
40.37
Winds-sp
68.96
62.17
46.02
48.90
42.11
60.45
61.93
Wands-sp
51.21
34.47
28.87
28.04
27.25
44.25
54.65
25%
LLM Fruner
62.97
68.25
61.59
60.89
57,89
68.78
64.25
25%
LLM Pruner
38,80
16.81
32.91
33.94
33.05
38.66
33.11
ZipL.M
67.19
59.20
65.1
Zipl.M
51.57
55,91
32:12
51.34
OSSCAR
06.42
79.48
54.28
60.06
68.66
77.13
64.28
OSSCAR
$1.61
35.16
26.55
36.45
36.44
20.89
43.09
FLAP
65.47
68.51
64.89
68.29
67.25
65.14
63.82
FLAP
47.73
51.42
37.30
42.54
43.16
45,80
44.20
69.45
67.89
76.36
82.26
Ty-the- Phuser
46.62
58.27
55004
Shortii''
62.17
68.87
56.57
58.99
ShontaPP
23.53
29.31
31,44
32.11
30.71
27.72
34
1,00+
62.11
62.78
63.50
48.78
45. 36
63.12
64.62
1 1
33.56
41.75
33.12
34.16
54.07
$1.45
32.18
SliceGPT
63.00
71.44
42.008
50.49
46,85
63.41
60.06
SlicePP
37.65
41.27
32.05
34.14
53.37
34,50
32.48
Winda-sp
62.26
52.66
31.13
51.68
49.97
Wanda-pp
35.07
20.
30.63
37.5%
61.74
62.11
50.70
41.31
40.03
63,35
61.87
37.5%
M-Punerr
28.17
33.06
36.68
27.53
27.85
20. 38
26.36
ZrpLM
61.89
76.79
49.76
51.56
61.47
69.91
62.72
ZipLM
38.29
45.79
26.89
29.11
27.43
40.65
35.69
OSSCAR
64.63
74.25
58.01
62.26
62.26
OSSCAR
42.86
48.
41.37
35,09
FLAP
63.46
61.93
64.62
62.54
65,50
32.69
36.48
37.49
Ty-ste-
68.87
80.76
66.33
70.09
70.55
70.N5
74.65
Pu-nhe-Pruner
48.47
$4.11
39.97
47.33
47.12
46.01
48.22
ShoottaPT
62.17
62.30
46.61
62.57
62.17
$1.90
55.29
ShocttarT
28.61
32.44
28.01
27.82
27.87
36.30
30,83
LiCOs
54,83
51.41
46.57
LaC'U+
27.72
31.64
2H.31
27.31
28.28
SliceGPP
57.16
62.36
40.76
41.74
38.56
51.13
$1.55
SliceCPT
30.91
32.35
28.23
28.06
20.07
29.60
29/02
Wamdh-sp
46.91
62.14
41.39
54.77
40.37
48.90
43.06
Wandsepp
26.65
28.50
26.32
26.72
26.73
27.61
21
50%
1. MIM-Prrnner
18.23
61.31
38
90.54
37.83
43.24
43.94
90%
M-M-Priner
26.76
37.7%
76.60
26.43
26.36
27.09
25.06
ZipLM
43.79
41.95
57.43
62.72
62.25
ZipLM
35.84
27.52
26.42
32.17
OSSCAR
61.62
62.94
56.48
53.36
61.04
60.95
67.17
OSSCAR
32.21
32.16
26.58
27:51
96
32.15
30.26
FLAP
58,50
65.14
51.25
61.65
52.29
61.47
48.87
FLAP
27.02
41.13
26.29
32.96
29.43
13.09
32.51
Tye Pruner
65.54
74.46
62.26
65.63
66.64
62,17
65.26
Tyr-tee Pruner
42.62
49.45
33.68
39.71
39.99
38.68
40.24
29

---

Table 16: 0-shot acc (%) on OpenBookQA.
Table 17: O-shot acc (%) on RTE.
Spunsity
Method
LLaMA-2
3LaMA-3x
Mismil
Spunity
Medhod
LLMMA22
[LaMA-3x
Mistnl
7B
2:33
V0.3 Nemo
13B
0:88
18B
v0.3 Nemo
0%
NA
31.40
35.20
31.00
34.80
33.20
33,40
36.40
0%
N/
02.82
05.34
54,57
67.87
71.12
68.95
64.26
ShortiiT
28.20
33.20
26.00
33.00
30.80
27.00
31.60
ShortaPT
55.96
63.90
55.95
57.04
62,82
70.04
57.40
L:CO+
30,00
30.20
25.60
30.30
31.20
25,60
23.80
LaCO+
63.54
58.48
57.40
68.23
70.10
65.34
64.26
SiiceGPP
32.00
34.00
27:20
28.00
26.60
27.20
26.40
SaccCPP
64.26
56.84
58.45
62.62
63.90
66.06
57.40
Wanda sp
31,60
32.00
15:40
13:20
22.8
28.00
23.60
Wanda sp
58.48
06.98
46.93
57.04
57.76
57.76
62.82
12.5%
L.MM-rraser
28,40
34.40
24.00
27.20
26.40
36,80
31,00
12.5%
LLM-Prmner
62.82
61.73
48.35
56.68
61.01
65.34
55.60
ZiLLM
34.00
27.50
26.60)
32
61.37
63.18
50.15
66.06
66.23
OSSCAR
91.20
35,50
27.00
25.40
40
37.60
51,80
OSSCAR
58.84
61.73
56.33
64-62
63.18
60:31
59.57
FLAP
29.20
32.40
27.60
30.60
30,
33.40
$1.60
FLAP
57.76
59.21
50.54
52.35
55.96
67.51
56.68
Pielhe-Pruner
31,20
35,50
29.30
33.40
34,60
34.20
34,80
Tyh-hhePruner
66.06
67.15
56,68
66 79
71.84
65.95
62.82
SheeGGP
23,40
27.00
23:20
19:60
23.00
SheeGGPT
57.76
59.
62.50
64.98
63.54
LaCO+
25.20
25.20
22.40
20.60
20.00
23,40
22.20
LaC0+
53.79
62.45
57.04
63.90
58.12
61.26
58.12
SliceGPT
25,00
30.40
23:00
24.10
22, 60
23.00
23.00
SIseGPT
55.96
66.79
59.21
58.12
57.40
$7.40
52.71
Wunda sp
17.30
14.20
13.40
21.20
Wanda
48.38
52.71
52.31
53.79
53.45
25%
M-
21.00
28.50
15.40
19,80
18.00
30,40
23.001
56.68
50.55
52.35
52:35
53,07
55.23
53007
ZipLM
31.40
34.59
17.00
24.40
18.80
29.20
19:20
ZipLM
55.60
68.23
55.23
50.18
51.99
68.95
53.45
OSSCAR
31,40
34.30
13.00
20:30
21.
34.40
23.00
OSSCAR
52.35
67.51
48.74
54.51
50.54
63.18
53.79
FLAP
27,40
24.60
40
25.00
ELAP
63.90
Tyr-the-rrneer
31.60
34.20
28:30
34.00
31.80
33.40
31.80
Tyr-the-Poee Sy-uhe-Pruner
62.09
69.31
59.57
63.90
63.18
65.34
59.57
SherGGP
21.60
22.00
18,50
18:50
18.40
17.20
17.20
SheeGGPT
62.09
$2.35
58.48
50.54
53.79
49,82
53.79
1,00+
17,00
21.40
17.00
17:80
16.60
17.00
19.80
1a00+
57.40
55.60
$7.04
54-87
53.84
61.37
54.15
SliceGPT
17.60
15.00
17.40
SPicePP
58.48
52.71
53.43
55.23
52.71
Wanda-sp
17,60
13.40
11.30
12.60
11.M0
14,80
12.60
Wanda-sp
49.38
52.71
54.51
46.57
50.54
53,117
49.46
37.5%
LLM Pruner
12,80
17.30
12:20
12:30
13.20
14,90
13.00
37.5%
LLM Pruner
52.71
52.71
52.71
52.71
52.71
51.26
52.71
ZipLM
13.50
14.40
13.20
14.60
Zip.M
58.12
64.26
53.79
54.15
52.35
OSSCAR
25.20
20.40
14:20
15.20
14.80
21.60
17:80
OSSCAR
51.62
63.18
49.82
52.71
50.90
04022
52.71
FLAP
24.20
27:30
21.60
22,80
23,
34.00
25.80
FLAP
54.51 48.36
46.57
51.36
55.60
53.79
55.96
Tyr-the Pruner
31,00
32.40
26.00
29,50
30,00
26.20
29.20
Tyr-the Pruner
61.37
65.70
55,96
60:29
58.84
58.84
54.15
ShenGGP
17.00
14.00
ShenGiT
51.62
60.65
51.26
1,00+
16.20
18.00
14:60
14:00
16.60
16.00
1a00+
$1.62
61.00
57.40
51
45.13
53.43
49.87
SlaceGPT
16,80
22.00
14:20
14.00
12.80
14,80
15.00
SisGGP
53,43
52,71
53.07
53.43
55.96
53.07
52.71
Winda-sp
12.20
11.30
13:00
13:60
13.40
13,60
13.80
Winda-sp
53.07
52.71
54.51
53.07
51.62
52.71
49.16
50%
Lruner
12.60
12.40
13.40
13:40
15:40
50%
LLM Pruner
53.07
52.71
52.71
52.35
52.71
53.07
ZipLM
14,00
19.50
12:40
13.30
11.60
17.40
13.20
2:pL.
52.71
52.71
53.07
52:35
52.71
52.71
51.26
OSSCAR
17,00
20.00
11.50
11.60
10.60
16,00
14.40
OSSCAR
53.43
52,71
50.50
47.65
51.26
53.79
55.96
FLAAP
13:30
21.00
FLAP
45.40
53.07
27.30
30.40
20.40
26,60
34,00
22,80
26.20
Tol-Be-Punner
55.96
59.93
53.43
5834
58.12
53.79
Table 18: 0-shot acc (%) on WinoGrande.
Table 19: 5-shot acc (%) on MMLU.
Sparity
Mathod
733
LLaMA-2
130
2:3
LLaMA-3.x
0.80
1:85
7B v0.3
Misral
Nemo
Sparsity
Method
71
LLaMA-2
136
238
LLaMA-3x
0.89
1 RB
BB v0.3 Nemo
Mustrl
0%
NiA
69.06
72.22
69.06
73.01
73.56]
73.64
73.64
N/A
45.84
55,06
65.27
56.17
63.20
62.18
63.85
ShevtfiPr
68.98
21.08
67.64
71.67
70.09
70.24
74.11
SheesCaPT
46.28
56.33
55,90
68.10
LaCOs
68.35
70.56
69.61
75.16
73.16
71.35
74.05
La00+
45.34
54.71
58.70
53.10
63.54
61.69
67.23
SlicsCPP
67.32
70.48
60.54
67.17
66.61
70.64
64.01
SlieeCPT
42.88
53.43
55.85
47.34
53.22
58.38
64.58
Wands-pp
67.64
52.01
62.27
Wands-up
39.99
44.64
56.42
12.5%
LLM--rrnner
64.25
69.30
60.93
65.35
65.51
66.06
63.51
12.5%
LLM-Pruner
37.40
50.51
48.71
39:10
50.26
54.09
54.59
7ipl M
70.45
72.60
60:77
68.75
66.93
72.53
63.98
7ipl M
41.33
53.57
58.58
48:51
53.95
58.62
63.20
OSSCAR
69.66
73.40
60.30
68.19
68.27
71.59
65.11
OSSCAR
40.02
53.25
58.05
47.31
58.45
58.71
63.83
HLAP
68.03
62.19
69.53
70.40
70.24
63.45
HLAP
40.60
47.66
49.36
41.90
52.29
55.82
52.98
Tye the Pruner
70,09
73,85
67.40
73.24
72.55
73.40
72.69
The Pruncr
44.07
34.61
59.50
09.32
59.06
29.11
64.06
ShontGPT
65.67
70.56
61.40
53.99
56.17
67.25
63.14
Shortaa'
37.38
48,00
36.30
34.20
34.88
59.66
50.66
1,00s
64.25
63.14
71.59
1a00+
39,36
ShicePP
65.27
70.40
57.62
62.67
59.75
61.48
56.25
ShceCPP
40.20
51.21
36.46
35.022
32,30
45.76
Wands-sp
63,00
54.30
51.78
48.78
49.57
60.62
57.06
Wands-sp
20.40
20.
24.47
24.07
25.78
$7.15
25.25
25%
LLM-Prrnner
57.70
61.56
52.57
55.41
55.17
56,20
57.62
25%
LLM-Prmmer
27.37
32.58
30.95
28.49
27.47
33.72
32.60
ZipL.M
51.85
58.25
54.70
Zipl.M M
32.70
45.09
24.79
51.22
OSSCAR
68.11
70.56
52.72
56.91
56.75
64,33
55.41
OSSCAR
21.04
46.25
41.89
24.89
43.06
49.51
51.18
FLAP
64.72
68.00
57.06
62.75
63.46
64.72
64.66
FLAP
20.57
42.66
35.96
34.47
39.18
47.99
31.09
Tyeehe Pruner
68.51
64,01
70.01
Tyr
24,90
52.12
57.68
Shortai'
60.30
61.56
54.54
55.09
59,19
53.99
ShortaaP
39,40
48.00
36.35
25.16
30.64
23.90
50.72
LaCO+
19.04
65.82
58.01
60.38
61.25
59.43
55.19
LaCO+
28.91
52.76
28.57
27.54
27.13
27.50
39.10
SlaceGPT
46.73
65.39
55.09
54.78
53.04
55,80
54.85
Shxc
35.04
46.58
27.21
29.13
25.17
31,35
39.03
Winda-sp
48 08
Wands-pp
25.20
23.44
22.09
37.5%
LLM-Pruner
51,07
52.57
49.96
51.07
50.83
51.78
50.36
37.5%
11M-Pruner
24.65
25.50
24.38
24
36.88
25.97
25.79
ZipUM
60.46
63:46
50.36
53.83
54.38
57.38
51.14
M
30.65
43.60
31.82
24.76
33.97
18.48
34,54
OSSCAR
61.64
63.14
50.04
55.09
54.40
57.36
53.83
OSSCAR
28.95
42.65
34.01
25.05
53.98
35.42
37.06
FLAP
61.72
61.66
52.57
58.17
56.20
26.55
29.24
27.59
37.64
Ty-the-Pruner
66.93
72.06
60:22
66.54
66.54
64.17
65.27
Tyr-the-Prume Pruner
31.56
44.77
43.78
24.92
41.43
42.96
47.63
SherGGPT
56.20
61.50
51,46
54.70
14.
53.12
52.64
SheetGPP
25.77
23:97
23:00
24.24
22.97
23.37
32.22
LaCO+
51.38
59.19
51:22
53.75
52.17
$1.30
50.12
LaCO+
23.81
40.19
25.95
25.48
26.08
24.47
25.35
SliceGPP
54,62
56.99
51.30
49.88
51.38
49.35
SlicKPPT
29.37
38.94
24.03
25.54
25.17
26.34
25.24
Wando-sp
49.17
50.04
51.07
47.43
51.22
49.88
48.07
Wanda-sp
24.55
25.97
25.72
24:21
25.76
25.15
25.72
50%
11 M-Mannr
49.58 50.43
51.07
52
49.64
50,20
49.80
50%
M.MMPrrneer
25.81
24.63
25.24
23.31
24.70
25.89
25.17
ZipLM
48.54
51.54
50.59
25.70
24:87
27.92
28.46
OSSCAR
51.54
52.50
$1.70
48.86
51.14
$2,49
50.45
OSSCAR
25.35
31.36
25.71
25
26.33
25.27
27.97
FLAP
56.51
61.72
50.51
52.80
$4.14
$2.88
$2.09
FLAP
23.87
20 (0
23.49
23
40
23.18
25.07
24.23
Tyr-tbe Pruner
62.12
70.89
53.28
60.30
61.80
39,43
59.04
Tyr-tbe Pruner
26.06
40.29
30.46
26.46
33,76
33,51
33.34
30


Reviews:
[{"review_id": "ap7J9IfRur", "reviewer": "Reviewer_qkjw", "summary": "This paper introduces an efficient end-to-end search-based global structural pruning framework for LLMs. It proposes a method to construct a supernet by applying local pruning across various sparsity ratios to each layer of an LLM, aiming to identify the optimal sparsity distribution under a target overall sparsity ratio. The framework employs an iterative prune-and-search strategy with coarse-to-fine sparsity granularity to ensure efficient search convergence. The proposed method achieves state-of-the-art structural pruning results, retaining 97% of the dense model’s performance while removing 50% of Llama-3.1-70B’s parameters.", "questions": "Given that the matrix \\hat{W} shares the same dimensionality as W, how can this sparser \\hat{W} speed up inference? I have this concern because GPUs are optimized for tensor calculation and \\hat{W}X and WX require the same number of computation operations.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written and well-motivated. \n2. The proposed method seems to be novel.\n3. The experimental results presented are promising, showing the effectiveness and robustness of the proposed framework.\n\nWeaknesses:\n1. The proposed method requires an additional calibration dataset, however, some baselines do not rely on such a dataset, this makes the comparison unfair.\n2. The method part could be polished to facilitate comprehension.", "was_revised": true, "final_justification": "My original score is already pretty positive,", "rebuttal": "Dear Reviewer qkjw:\n\nThanks for your thoughtful and encouraging feedback. We are glad that you found our work valuable. In response to your concerns, we offer the following responses.\n\n---\n\n**Weakness 1**: \"The proposed method requires an additional calibration dataset, however, some baselines do not rely on such a dataset, this makes the comparison unfair.\"\n\n**Response to Weakness 1**: We understand the your concern regarding the fairness of comparisons due to the use of a calibration dataset. However, we would like to clarify that **(1)** all baselines in our main experiments were re-evaluated using the same calibration samples as our method, ensuring fair comparisons. This setup is described in *Section 3.1, line 225*. **(2)** For baselines in *Table 6 (Appendix A.3)*, whose official implementations were either released very recently or inaccessible to reproduce, we report results directly from their papers. Since these methods were already carefully tuned, we believe the comparison remains relatively fair. **(3)** Only a few methods—such as *magnitude pruning* and *PruneNet (a baseline in Table 6)*—are not calibration-dependent, but either perform poorly or are designed specifically in a calibration-free manner, making their inclusion reasonable for a comparison.\n\n---\n\n**Weakness 2**: \"The method part could be polished to facilitate comprehension.\"\n\n**Response to Weakness 2**: We agree that the method section could be made clearer to improve reader comprehension. In particular, we realize that adding an overview of the overall algorithmic process and providing richer illustrative examples would significantly enhance clarity. We will refine this part accordingly in future revisions.\n\n---\n\n**Question**: \"Given that the matrix $\\widehat{\\mathbf{W}}$ shares the same dimensionality as $\\mathbf{W}$, how can this sparser $\\widehat{\\mathbf{W}}$ speed up inference? I have this concern because GPUs are optimized for tensor calculation and $\\widehat{\\mathbf{W}}\\mathbf{X}$ and $\\mathbf{W}\\mathbf{X}$ require the same number of computation operations.\"\n\n**Response to Question**: Thank you for the insightful question. We believe the concern about pruning and its impact on actual inference speed is understandable but unnecessary in our case. In our method, the pruning process is initially represented via setting zeros, with weight dimensions kept unchanged during our iterative prune-and-search process (as shown in *Equations* and *Figure 2(b-c)*). After the algorithm is finalized, we perform **true structural compression** by removing the zeroed **rows and columns**—specifically, reducing the **output** dimensions of weights of *q*, *k*, *v*, *up*, and *gate*, and the **input** dimensions of weights of *o* and *down*. Other modules such as RMSNorm, Embeddings, and the LM-head remain unchanged. The resulting model, illustrated as *Figure 2(d)*, achieves genuine reductions in parameter count and FLOPs, yielding **practical speedup without relying on hardware-specific support**. \n\nYou can view detailed information about the actual acceleration of the post-pruned model in *Table 3 (Subsection 3.2)* and *Figure 4 (Appendix A.5)*. Or refer to the following simple table to quickly summarize the post-pruning acceleration effect.\n\n**Table**: Inference efficiency of post-pruned LLMs with Týr-the-Pruner. Benchmarks were conducted on a single AMD Instinct™ MI250 accelerator using PyTorch (HipBlas) for LLM inference, with input and output sequence lengths set to 2048.\n\n| Model         | Sparsity | #Params | TTFT        | Decode Throughput |\n|---------------|----------|---------|-------------|-------------------|\n| Llama-3.1-8B  | 0%       | 8.0B    | 2.49 (1.00×) | 12.27 (1.00×)     |\n| Llama-3.1-8B  | 25%      | 6.1B    | 1.94 (1.28×) | 14.13 (1.15×)     |\n| Llama-3.1-8B  | 50%      | 4.3B    | 1.42 (1.75×) | 16.97 (1.38×)     |\n\n---\n\nPlease let us know if any specific aspect could be further elaborated. We thank you again for your constructive review and your support of our work.\n\nSincerely,\n\nAuthors 11393."}, {"review_id": "s6tppKZoBm", "reviewer": "Reviewer_Atmm", "summary": "This paper proposes a method for choosing the optimal sparsity values for different layers under the structured pruning setup. This is done by first pruning different layers to different sparsity levels, and then choosing the optimal sparsity allocation.", "questions": "See the comments above.", "limitations": "See the comments above.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I think this is an important problem to study, and I commend the authors for their work. The numerical experiments show the benefits of this type of sparsity allocation which is positive. However, I also have some concerns. \n\n1- I believe the models that do not have uniform sparsity will need custom code, which will limit the application of this method in practice. Specially these custom models need to be implemented in serving engines such as vLLM and SGLang, which are used in practice for serving. This is missing from the paper.\n\n2- The authors provide inference speed results for the models from their method, but how does this compare to a uniformly pruned model?   A non-uniform model will require invoking several different kernels on the GPU, which will create inference time overheads.\n\n3- Are the accuracy numbers reported after distillation using (6)? How about the baselines? The consistency of the baselines need to be discussed. \n\n4- What does (5) mean? Can you please provide some intuition?\n\n5- (4) is very close to OBS/OSSCAR/ZipLM. Some discussion is needed here.", "was_revised": true, "final_justification": "I think the lack of widely available inference implementation limits the applicability of the work. Given that this is not the first work to study non-uniform sparsity, I think the paper's contribution remains limited.", "rebuttal": "Dear Reviewer Atmm:\n\nWe are grateful for your careful and insightful comments. Please find our detailed responses to your observations and concerns below.\n\n---\n\n**Weakness 1**: \"I believe the models that do not have uniform sparsity will need custom code, which will limit the application of this method in practice. Specially these custom models need to be implemented in serving engines such as vLLM and SGLang, which are used in practice for serving. This is missing from the paper.\"\n\n**Weakness 2**: \" The authors provide inference speed results for the models from their method, but how does this compare to a uniformly pruned model? A non-uniform model will require invoking several different kernels on the GPU, which will create inference time overheads.\"\n\n**Response to Weakness 1 and 2**: We appreciate your concerns regarding the practicality and efficiency of serving non-uniformly pruned LLMs. Our responses are as follows: **First**, our results demonstrate that non-uniform sparsity consistently outperforms uniform sparsity under the same compression ratio, achieving SOTA accuracy. This performance gap validates a hypothesis that weight importance is not evenly distributed across layers in LLMs. Thus, exploring non-uniform pruning is not merely a technical curiosity but a meaningful direction for understanding and optimizing LLMs.\n\n**Second**, while we acknowledge that current deployment frameworks like vLLM and SGLang primarily support uniform sparsity out of the box, it is the practical value of non-uniform sparsity that motivates extending these frameworks. A representative example is the recent push to support hybrid and non-uniform architectures in TensorRT-LLM, driven by large-scale Nemotron series models [1, 2]. Our work aligns with this trend and contributes to the ecosystem by identifying practical needs and performance benefits of such models.\n\n**Third**, we are actively working to extend vLLM to support non-uniform sparsity. This effort involves custom model architecture registration, tailored weight loading, and heterogeneous key-value (KV) cache management. Our support may be adopted by vLLM in the future. A preliminary result from this ongoing work is shown in the Table below. \n\n**Table**: Inference efficiency of uniform and non-uniform pruned Llama-3.1-8B. Benchmarks were conducted on a single AMD Instinct™ MI350 (288GB) accelerator using vLLM (extended to support non-uniform LLMs, with the temporary unified kv cache space allocation method) for inference, with input and output sequence lengths set to 256.\n| Sparsity           | TTFT (ms) | Prefill Speedup | Decoding Tpt. | Decoding Speedup |\n|--------------------|-----------|-----------------|---------------|------------------|\n|         0%         |   480.5   |      1.00x      |    7642.04    |       1.00x      |\n|     50% uniform    |   287.93  |      1.67x      |    11935.39   |       1.56x      |\n| 50% Týr-the-Pruner |   291.91  |      1.65x      |    11257.77   |       1.47x      |\n\n**Finally**, as shown in *Appendix A.5*, we compare the inference efficiency between uniformly and non-uniformly pruned models. The results show that inference efficiency are comparable in most cases. Uniform sparsity can offer slightly higher throughput only when the non-uniform sparsity distribution is highly imbalanced (i.e., large inter-layer sparsity variance), which causes multiple GEMM kernels (Including low-utilization \"thin\" MatMul) to be dispatched. Nonetheless, both pruned variants achieve substantial speedup over dense baselines, and the accuracy-efficiency tradeoff remains favorable for non-uniform models.\n\n---\n\n**Weakness 3**: \" Are the accuracy numbers reported after distillation using (6)? How about the baselines? The consistency of the baselines need to be discussed.\"\n\n**Response to Weakness 3**: We would like to clarify that our method does not involve knowledge distillation training. The formulation in Equation (6) is merely inspired by distillation, serving as a training-free scoring function to search for non-uniform sparsity patterns that better approximate the unpruned model’s outputs. All reported accuracy results are based on these searched architectures. Even without search, our method (Týr-the-Pruner with uniform sparsity) achieves competitive performance, as shown in the Table below (*Table 4 and Table 8* also provided relative detailed results). Additionally, to ensure fairness, **all baseline methods were re-evaluated using the same calibration dataset** as ours.\n\n**Table**: Týr-the-Pruner Performance with or without Search when pruning 50% parameters of Llama-3.1-8B.\n| Method                    | Wikitext2 | Arc-C | Arc-E | BoolQ | HellaSwag |  OBQA |  RTE  | WinoGrande |  MMLU |  AVG  |\n|---------------------------|-----------|:-----:|:-----:|:-----:|:---------:|:-----:|:-----:|:----------:|:-----:|:-----:|\n| LLM-Pruner                |   288.32  | 19.62 | 28.70 | 37.83 |   26.36   | 13.40 | 52.35 |    49.64   | 24.70 | 31.58 |\n| ZipLM                     |   366.34  | 20.48 | 28.28 | 57.43 |   26.42   | 11.60 | 52.71 |    52.49   | 26.16 | 34.45 |\n| FLAP                      |   134.28  | 20.99 | 43.18 | 52.29 |   29.43   | 16.80 | 52.71 |    54.14   | 23.18 | 36.59 |\n| Týr-the-Pruner w/o search |   66.38   | 23.55 | 58.46 | 62.35 |   32.51   | 16.60 | 51.26 |    52.88   | 28.34 | 40.74 |\n| Týr-the-Pruner            |   30.89   | 31.83 | 65.36 | 66.64 |   39.99   | 24.80 | 58.12 |    61.80   | 33.76 | 47.79 |\n\n---\n\n**Weakness 4**: \"What does (5) mean? Can you please provide some intuition?\"\n\n**Response to Weakness 4**: Certainly. *Equation (5)* captures the idea of error accumulation during calibration-included layer-wise pruning. In this setting, pruning is performed layer by layer, where the input to each layer $X_l$ is affected by the pruning of the previous layer. After pruning layer $l$, the output becomes $X_{l+1}=W_l^{\\text{pruned}}X_l$, which propagates errors forward. In our method, each weight generates multiple pruned versions with different sparsity levels. *Equation (5)* fuses the outputs from these variants using a **weighted combination**, where the weights are normalized sparsity scores—assigning **higher weight to lower-sparsity outputs**, which tend to be closer to the reliable original dense outputs. This fusion provides a more reliable input for pruning the next layer and better simulates cumulative pruning effects.\n\n---\n\n**Weakness 5**: \"(4) is very close to OBS/OSSCAR/ZipLM. Some discussion is needed here.\"\n\n**Response to Weakness 5**: We acknowledge that our method builds upon the same Taylor expansion principle as OBS, which has been widely adopted and validated in recent pruning works such as OSSCAR, ZipLM, SparseGPT, and LLM-Pruner. While large theoretical deviations are unlikely at this stage, our contribution lies in fully leveraging both first-order gradients and second-order Hessian information to guide fine-grained pruning and weight adjustments, pushing the performance boundary further. As shown in *Table 4*, our ablation study confirms the effectiveness of each component in our local pruning method. Most importantly, our core innovation is the iterative prune-and-search framework, within which the enhanced local pruning serves to build a more reliable supernet for sparsity distribution optimization.\n\n---\n\n[1] Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models. 2025.\n\n[2] Llama-Nemotron: Efficient Reasoning Models. 2025.\n\n---\n\nWe sincerely thank you again for your thoughtful feedback and support. If there are any particular points that would benefit from further clarification, we would be glad to elaborate.\n\nSincerely,\n\nAuthors 11393."}, {"review_id": "WFtoiIokfK", "reviewer": "Reviewer_xbcw", "summary": "This paper proposes a new global structural pruning approach called Tyr-the-Pruner. This method utilizes a supernet to determine the optimal sparsity distribution under a target overall sparsity ratio. An iterative prune-and-search strategy with coarse-to-fine sparsity granularity is used to reduce the search cost. The experimental results show that the proposed method can achieve state-of-the-art results.", "questions": "See weaknesses above.", "limitations": "yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. This paper is technically sound and easy to understand.\n\n2. The paper can discard 50% of channels without sacrificing much accuracy, which achieves state-of-the-art results.\n\n3. The overall method shows some novelty.\n\nWeaknesses:\n\n1. Eq.4 is not very easy to understand. What is the intuition behind Eq.4? What does ~p mean?\n\n2. How does random error accumulation and abandoned error accumulation do in Lines 149-150?\n\n3. The authors use evolutionary search, which has many operations not optimized for GPU. What is the total search cost compared to sparsity-aware training methods and post-training sparsity methods?\n\n4. Is this method comparable to or exceeding the sparsity-aware training methods?\n\n5. You use 4 iterations during search. What if you use fewer or more iterations? Will that influence the search results and search time?", "was_revised": true, "final_justification": "The rebuttal has addressed all my concerns, so I suggest the acceptance of the paper.", "rebuttal": "Dear Reviewer xbcw:\n\nThank you for your thoughtful and encouraging feedback. Below, we provide detailed responses to each of your comments and concerns.\n\n---\n\n**Weakness 1**: \"Eq.4 is not very easy to understand. What is the intuition behind Eq.4? What does ~p mean?\"\n\n**Response to Weakness 1**: *Equation (4)* is derived based on the Taylor expansion for minimizing the pruning error (as shown in *Equations. (2-3)*). $W_p$ denotes the candidate channel selected for pruning, while $\\delta W_{\\sim p}$ represents the weight adjustment applied to the remaining unpruned channels ($\\sim p$) to compensate for the error introduced by zeroing out $W_p$. For selecting which channel $p$ to prune, we prioritize those with both small gradient magnitude and low curvature (i.e., small entries in the Hessian, c.f., *Appendix A.1*). Intuitively, a small gradient implies that the weight contributes little to the loss, while a small Hessian entry indicates that the loss is relatively insensitive to perturbations in this weight. Therefore, pruning such a channel is expected to introduce minimal loss increase. For the remaining unpruned weights ($\\sim p$), we derive an adjustment $\\delta W_{\\sim p}$ to compensate for the loss incurred by pruning $W_p$. This is formulated as a constrained optimization problem: minimizing the pruning-induced loss subject to the constraint that $W_p$ is set to zero. Using the method of Lagrange multipliers, we solve for the optimal $\\delta W_{\\sim p}$ in closed form, as shown in the right-hand expression of *Equation. (4)* and *Appendix A.1*. Overall and Intuitively, this formulation ensures that pruning and compensation jointly minimize the estimated error impact.\n\n---\n\n**Weakness 2**: \"How does random error accumulation and abandoned error accumulation do in Lines 149-150?\"\n\n**Response to Weakness 2**: In the context of layer-wise pruning with calibration, error accumulation reflects how pruning decisions in earlier layers affect the inputs to later layers. Specifically, after pruning layer $l$, the input to layer $l+1$ becomes $X_{l+1} = W_l^{\\text{pruned}} X_l$, thus propagating pruning-induced errors forward. Since our method explores multiple pruned candidates with different sparsity ratios per layer, we must decide how to accumulate errors. The abandoned strategy discards error accumulation entirely by using the dense model's original activations at every layer, ignoring any pruning-induced errors. In contrast, the random strategy selects one pruned candidate output at random and uses it as the input for the next layer's pruning, simulating a single-path forward pass but without fusion. Both are used as baselines to highlight the effectiveness of our proposed weighted accumulation in *Equation (5)*, which provides a more stable and realistic pruning signal.\n\n---\n\n**Weakness 3**: \"The authors use evolutionary search, which has many operations not optimized for GPU. What is the total search cost compared to sparsity-aware training methods and post-training sparsity methods?\"\n\n**Weakness 4**: \"Is this method comparable to or exceeding the sparsity-aware training methods?\"\n\n**Response to Weakness 3 and 4**: To evaluate the trade-offs between efficiency and effectiveness, we conducted a comprehensive comparison targeting 50% sparsity on LLaMA-2-7B using four representative approaches: (1) *FLAP*, a previous SOTA post-training sparsity (PTS) method; (2) *Týr-the-Pruner without search*, which applies our local pruning but skips search; (3) *NutePrune [1]*, a training-aware sparsity method based on LoRA and multi-stage knowledge distillation; and (4) *Týr-the-Pruner* framework. As shown in the table below, PTS methods (FLAP and our non-search variant) complete in under 30 minutes, offering high efficiency but generally lower accuracy compared to training-aware/search methods. Nevertheless, our non-search variant consistently outperforms FLAP, demonstrating the strength of our local pruning strategy. While our full method incurs a higher search cost (~612 minutes), it significantly improves accuracy—surpassing both PTS and training-aware approaches. In particular, it outperforms the training-aware method NutePrune (649 minutes) in most tasks despite requiring no training. Moreover, our method does not rely on large-scale training data (requiring calibration samples only), making it more practical in scenarios with limited supervision. These results highlight that our approach strikes a strong balance between accuracy and efficiency while maintaining minimal data and computational requirements.\n\n**Table**: Comparison with post-training sparsity and training-aware sparsity methods for pruning 50% of Llama-2-7B’s parameters.\n| Method                    | Type                       | Optimization time cost | Wikitext2 | Arc-C | Arc-E | BoolQ | HellaSwag | OBQA  | RTE   | WinoGrande |\n|---------------------------|----------------------------|------------------------|-----------|-------|-------|-------|-----------|-------|-------|------------|\n| Dense                     | NA                         | 0 minutes              |    5.12   | 43.43 | 76.30 | 77.68 |   57.14   | 31.40 | 62.82 |    69.06   |\n| FLAP                      | post-training sparsity     | 22 minutes             |   25.49   | 29.10 | 47.01 | 58.50 |   37.02   | 21.20 | 45.49 |    56.51   |\n| Týr-the-Pruner w/o search | post-training sparsity     | 26 minutes             |   21.37   | 25.09 | 54.17 | 62.63 |   35.83   | 20.80 | 55.60 |    56.12   |\n| NutePrune                 | training-aware sparsity    | 649 minutes            |   12.90   | 35.41 | 62.46 | 63.64 |     -     |   -   |   -   |    61.25   |\n| Týr-the-Pruner            | iterative prune-and-search | 612 minutes            |   16.17   | 33.62 | 66.12 | 65.54 |   42.62   | 27.20 | 55.96 |    62.12   |\n\n---\n\n**Weakness 5**: \"You use 4 iterations during search. What if you use fewer or more iterations? Will that influence the search results and search time?\"\n\n**Response to Weakness 5**: Thanks for the meaningful question. We conducted an ablation study by varying the number of search iterations from 0 (i.e., no search, equivalent to uniform pruning) to 5, while fixing the final iteration sparsity granularity to 1.5625% in each setting. As shown in the Table below, increasing the number of iterations consistently improves the performance of the searched sparsity pattern, with a smaller number of performance enhancements beyond 3–4 iterations. This suggests that 3–4 iterations are generally sufficient to reach near-optimal sparsity configurations. Each iteration takes approximately 150±10 minutes, and our default setting of 4 iterations offers a good trade-off between search cost and pruning quality. Intermediate results from each iteration in the 4-iteration setting can be found in *Table 8* in *Appendix A.6*.\n\n**Table**: Ablation Study of Search Iterations when pruning 50% parameters of Llama-3.1-8B.\n| Iterations | Wikitext2 | Arc-C | Arc-E | BoolQ | HellaSwag |  OBQA |  RTE  | WinoGrande |  MMLU |  AVG  |\n|------------|-----------|:-----:|:-----:|:-----:|:---------:|:-----:|:-----:|:----------:|:-----:|:-----:|\n| 0          |   66.38   | 23.55 | 58.46 | 62.35 |   32.51   | 16.60 | 51.26 |    52.88   | 28.34 | 40.74 |\n| 1          |   35.91   | 26.62 | 61.11 | 64.58 |   37.01   | 22.60 | 55.23 |    57.05   | 29.29 | 43.58 |\n| 2          |   37.16   | 31.66 | 65.61 | 63.61 |   37.79   | 23.00 | 58.84 |    56.83   | 30.65 | 46.00 |\n| 3          |   32.34   | 31.83 | 65.49 | 64.59 |   39.11   | 23.80 | 58.48 |    59.67   | 30.57 | 46.69 |\n| 4          |   30.89   | 31.83 | 65.36 | 66.64 |   39.99   | 24.80 | 58.12 |    61.80   | 33.76 | 47.79 |\n| 5          |   29.56   | 31.87 | 65.57 | 66.69 |   39.08   | 26.40 | 58.84 |    60.89   | 33.67 | 47.88 |\n\n---\n[1] NutePrune: Efficient Progressive Pruning with Numerous Teachers for Large Language Models. 2024.\n\n---\n\nWe truly appreciate your valuable feedback and continued support. Should any aspect require further clarification, we would be happy to provide additional details.\n\nSincerely,\n\nAuthors 11393."}, {"review_id": "9xmaan7nXO", "reviewer": "Reviewer_DF7P", "summary": "The paper introduces Týr-the-Pruner, a global structural pruning framework for LLMs that optimizes sparsity distribution through an iterative prune-and-search approach. It achieves state-of-the-art performance with significant parameter reduction while maintaining high accuracy.", "questions": "1.\tIn Equation (3), the Taylor expansion of the error function $E=\\left \\| \\mathbf{XW-X\\widehat{W} } \\right \\| _2^{2}$ is expressed in terms of derivatives with respect to $\\mathbf W$. However, $\\mathbf W$ is fixed and the actual variable is either $\\delta \\mathbf W$ or equivalently $\\widehat{\\mathbf W}$. Could the authors clarify this?\n2.\tEquation (6) introduces two key hyperparameters, $\\alpha_l$ and $\\beta$. How sensitive is the final performance to different choices of $\\alpha_l$ and $\\beta$? Can the authors provide empirical results or practical guidance on setting these parameters?\n3.\tHow does the search cost compare quantitatively to existing structured pruning methods such as LLM-Pruner and SliceGPT?\n4.\tCould the approach be integrated with other compression methods like quantization or unstructured pruning? If so, have the authors conducted experiments or analyses to evaluate the compatibility or additive effects of such combinations?", "limitations": "Limitations are discussed. It would be better if the authors could optimize search time cost", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The method overcomes the limitations of conventional two-stage pruning pipelines by integrating a locally pruned supernet with a global evolutionary search that is guided by expected error accumulation.\n2. The proposed method is extensively evaluated on LLMs ranging from 7B to 70B parameters and achieves up to 97% of original accuracy even with 50% sparsity. It outperforms strong baselines in both perplexity and downstream QA tasks, demonstrating robustness and generalizability.\n\nWeaknesses:\n1. There are instances of carelessness in writing. For example, in line 71, the phrase “search the optimal sparsity distribution” is used, but the final letter ‘n’ in “distribution” is not bolded consistently with the rest of the word.\n2. The search process incurs significantly higher computational cost compared to other structured pruning approaches such as LLM-Pruner and SliceGPT. Although the authors acknowledge this limitation and mention plans to improve efficiency in future work, the current method may be less practical in resource-constrained environments due to the heavy search overhead.", "was_revised": true, "final_justification": "The author provides more detailed hyperparameter analysis and comparison on the running cost. Everything is clear.", "rebuttal": "Dear Reviewer DF7P:\n\nWe greatly appreciate your insightful and constructive feedback. Please find our point-by-point responses to your concerns and questions below.\n\n---\n\n**Weakness 1**: \"There are instances of carelessness in writing. For example, in line 71, the phrase “search the optimal sparsity distribution” is used, but the final letter ‘n’ in “distribution” is not bolded consistently with the rest of the word.\"\n\n**Response to Weakness 1**: Thank you for pointing this out. We will correct the formatting inconsistency and carefully proofread the manuscript to eliminate similar and other typographical issues throughout the paper.\n\n---\n\n**Weakness 2**: \"The search process incurs significantly higher computational cost compared to other structured pruning approaches such as LLM-Pruner and SliceGPT. Although the authors acknowledge this limitation and mention plans to improve efficiency in future work, the current method may be less practical in resource-constrained environments due to the heavy search overhead.\"\n\n**Response to Weakness 2**: We acknowledge that there exists the cost of search in our method compared to some fast pruning approaches such as LLM-Pruner and SliceGPT. However, these methods often sacrifice accuracy and rely on costly post-pruning fine-tuning to recover performance. For example, as shown in *Table 2*, pruning 50% of LLaMA-3.1-70B results in SliceGPT and LLM-Pruner retaining only 62% and 45% of the dense model’s accuracy, respectively, whereas our method preserves up to 97%. For the former methods, substantial fine-tuning or knowledge distillation is typically required after pruning to restore accuracy—introducing additional computation and time overhead that can outweigh the initial pruning efficiency. Our method avoids gradient backpropagation during pruning (c.f., *line 158, Section 2.4*) and adopts an optimized supernet storage strategy (c.f., *Appendix A.4*), making the primary overhead come from search time rather than resource costs. The entire pruning process can be executed on only a single GPU for an 8B model, which ensures its applicability in resource-constrained environments. Overall, we believe that spending additional time to obtain a substantially higher-quality pruned model is a worthwhile trade-off, particularly in scenarios where inference efficiency and accuracy are both critical.\n\n---\n\n**Question 1**: \"In Equation (3), the Taylor expansion of the error function $E=|\\mathbf{X} \\mathbf{W}-\\mathbf{X} \\widehat{\\mathbf{W}}|_2^2$ is expressed in terms of derivatives with respect to $\\mathbf{W}$. However, $\\mathbf{W}$ is fixed and the actual variable is either $\\delta \\mathbf{W}$ or equivalently $\\hat{\\mathbf{W}}$. Could the authors clarify this?\"\n\n**Response to Question 1**: Thank you for the thoughtful question. While it is true that $\\widehat{\\mathbf{W}}$ is the optimization variable, our analysis adopts a perturbation-based perspective, where the focus is on how small deviations $\\delta \\mathbf{W} = \\widehat{\\mathbf{W}} - \\mathbf{W}$ around a fixed reference point $\\mathbf{W}$ influence the error function. The Taylor expansion is thus naturally formulated around $\\mathbf{W}$. This approach is standard in model compression literature [1], where the goal is to assess the impact of weight perturbations. We will clarify this interpretation in the future version for better readability.\n\n---\n\n**Question 2**: \"Equation (6) introduces two key hyperparameters, $\\alpha$ and $\\beta$. How sensitive is the final performance to different choices of them? Can the authors provide empirical results or practical guidance on setting these parameters?\"\n\n**Response to Question 2**: We express Equation (6) in a general form where the hyperparameter $\\alpha_l$ controls the layer-wise alignment between the sparse and dense hidden representations, and $\\beta$ governs the alignment between the output logits. In practice, we recommend setting the weights $\\alpha_l$ uniformly across layers and normalizing them such that $\\sum_l \\alpha_l = \\beta = 1$, following practices commonly used in typical knowledge distillation literature [2]. To assess the sensitivity to different supervision strategies, we conducted an ablation study comparing three configurations: (1) using only hidden-state alignment across layers (first, middle, and last layer), (2) using only output logits alignment, and (3) combining both types of supervision. We found that the combined supervision yields the best overall performance, indicating the complementary benefits of guiding both intermediate and final representations. However, layer-wise supervision requires intermediate activations checkpointing in memory. For practical purposes, we adopt logits-only supervision as it achieves a favorable trade-off between effectiveness and resource efficiency.\n\n**Table**: Ablation on Search Direction for Pruning 50% of Llama-3.1-8B     \n| Search Direction| Perplexity | ARC-C | ARC-E | BoolQ | HellaSwag | OBQA | RTE | WinoGrande | MMLU | Average |\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| No Search | 66.38 | 23.55 | 58.46 | 62.35 | 32.51 | 16.60 | 51.26 | 52.88 | 28.34 | 40.74 |\n| Hidden States Only | 37.86 | 30.97 | 62.05 | 63.16 | 33.40 | 21.80 | 53.79 | 57.38 | 29.99 | 44.07 |\n| Logits Only  | 30.89 | 31.83 | 65.36 | 66.64 | 39.99 | 24.80 | 58.12 | 61.80 | 33.76 | 47.79 |\n| Hidden States & Logits | 28.56 | 32.51 | 65.87 | 63.12 | 40.35 | 26.2 | 58.12 | 63.14 | 34.21 | 47.94 |\n\n---\n\n**Question 3**: \" How does the search cost compare quantitatively to existing structured pruning methods such as LLM-Pruner and SliceGPT?\"\n\n**Response to Question 3**: Thank you for the helpful question. As shown in the Table below, LLM-Pruner, FLAP, and SliceGPT are representative post-training sparsity (PTS) methods that feature low optimization cost, making them attractive for fast deployment. However, they typically suffer from limited accuracy. In contrast, our method Týr-the-Pruner without search already outperforms all these PTS baselines in accuracy, while maintaining a similar or even lower optimization time. Furthermore, Týr-the-Pruner achieves the best overall performance across all benchmarks. Although it incurs a search cost, we believe this is a worthwhile trade-off: it produces a significantly more accurate and efficient sparse model without retraining, which is crucial in resource-constrained settings.\n\n**Table**: Cost and Performance Comparison when Pruning 50% Parameters of Llama-3.1-8B.\n| Method | Time cost | Wikitext2 | Arc-C | Arc-E | BoolQ | HellaSwag | OBQA | RTE | WinoGrande | MMLU | Average |\n|---|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| LLM-Pruner | 18 minutes | 288.32 | 19.62 | 28.70 | 37.83 | 26.36 | 13.40 | 52.35 | 49.64 | 24.70 | 31.58 |\n| SliceGPT | 134 minutes | 353.21 | 21.50 | 41.62 | 38.56 | 29.07 | 12.80 | 55.96 | 49.88 | 25.17 | 34.32 |\n| FLAP | 22 minutes | 134.28 | 20.99 | 43.18 | 52.29 | 29.43 | 16.80 | 52.71 | 54.14 | 23.18 | 36.59 |\n| Týr-the-Pruner w/o search | 26 minutes | 66.38 | 23.55 | 58.46 | 62.35 | 32.51 | 16.60 | 51.26 | 52.88 | 28.34 | 40.74 |\n| Týr-the-Pruner | 612 minutes | 30.89 | 31.83 | 65.36 | 66.64 | 39.99 | 24.80 | 58.12 | 61.80 | 33.76 | 47.79 |\n\n---\n\n**Question 4**: \"Could the approach be integrated with other compression methods like quantization or unstructured pruning? If so, have the authors conducted experiments or analyses to evaluate the compatibility or additive effects of such combinations?\"\n\n**Response to Question 4**: Thanks for the insightful question. Our method is indeed compatible with additional compression techniques. As shown in the Table below, we present results of applying quantization and unstructured sparsity on the 50% pruned Llama-3.1-8B model produced by Týr-the-Pruner. These results suggest that our pruned model is highly amenable to further compression: quantization maintains over 99% of the original pruned model's accuracy, while unstructured sparsity at 50% and 2:4 granularity achieves performance levels that are consistent with prior work (e.g., [SparseGPT: 91%@50%, ALPS: 83%@2:4] on 7B models). This compatibility indicates that Týr-the-Pruner can serve as a strong foundation in multi-stage compression pipelines, offering both high accuracy and robustness to further compression.\n\n**Table**: Integrated with quantization and unstructured pruning for 50% pruned Llama-3.1-8B.\n| Method | Wikitext2 | Arc-C | Arc-E | BoolQ | HellaSwag | OBQA | RTE | WinoGrande | MMLU | AVG |\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| Týr-the-Pruner@FP16 | 30.89 | 31.83 | 65.36 | 66.64 | 39.99 | 24.80 | 58.12 | 61.80 | 33.76 | 47.79 |\n| Awq[3]@W4A16 | 34.79 | 31.06 | 64.65 | 66.09 | 39.34 | 24.60 | 60.77 | 60.77 | 31.43 | 47.34 (99.1%) |\n| SmoothQuant[4]@W8A8 | 31.31 | 31.23 | 64.94 | 65.50 | 40.18 | 25.80 | 58.84 | 61.80 | 32.45 | 47.59 (99.6%) |\n| RTN@FP8E4M3 | 31.05 | 31.31 | 65.28 | 66.45 | 40.17 | 24.60 | 58.84 | 61.48 | 32.28 | 47.55 (99.5%) |\n| SparseGPT[5]@50% | 47.05 | 27.99 | 59.39 | 65.32 | 37.38 | 23.60 | 53.07 | 61.48 | 28.32 | 44.57 (93.3%) |\n| ALPS[6]@M4N2 | 70.22 | 23.04 | 53.62 | 63.43 | 34.22 | 21.40 | 52.71 | 58.17 | 27.34 | 41.74 (87.3%) |\n\n--- \n\n[1] Optimal Brain Surgeon: Extensions and performance comparisons. 1993.\n\n[2] Sparse Fine-tuning for Inference Acceleration of Large Language Models. 2023.\n\n[3] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. 2024.\n\n[4] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. 2024.\n\n[5] SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. 2023.\n\n[6] ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models. 2024.\n\n---\n\nWe sincerely appreciate your thoughtful feedback and support. If there are any points that would benefit from further explanation, we would be glad to elaborate.\n\nSincerely,\n\nAuthors 11393."}]

Author rebuttal:

Meta-review: The paper presents an end‑to‑end, search‑based global structural pruning framework that builds a supernet of locally pruned sub‑models and efficiently optimizes the layer‑wise sparsity distribution under a target overall compression ratio. The method introduces an expectation‑error accumulation scheme for reliable supernet construction and an iterative prune‑and‑search strategy with coarse‑to‑fine granularity, which together enable state‑of‑the‑art results: \approx 97 % of dense performance is retained while removing 50 % of the parameters of Llama‑3.1‑70B (and comparable gains on smaller models). Extensive experiments support the approach and look promising. Overall, the contribution is technically solid, well‑validated, and of high relevance to the community working on efficient LLM deployment. The majority of strong, high‑confidence reviews support acceptance, and the single borderline reject raises only practical concerns that seem to be mostly addressed in the rebuttal. I concur and recommend acceptance.
Decision: Accept (poster)