Computation and Memory-Efficient Model Compression with  Gradient Reweighting
Authors: Zhiwei Li
Yuesen Liao
Binrui Wu
Yuquan Zhou
Xupeng Shi
Dongsheng Jiang
Yin Li
WEIZHONG ZHANG
Venue: NeurIPS 2025 poster ()
Keywords: model compression
deep neural networks
sparse training
deep learning
Primary area: deep_learning

Abstract: Pruning is a commonly employed technique for deep neural networks (DNNs) aiming at compressing the model size to reduce computational and memory costs during inference. In contrast to conventional neural networks, large language models (LLMs) pose a unique challenge regarding pruning efficiency due to their substantial computational and memory demands. Existing methods, particularly optimization-based ones, often require considerable computational resources in gradient estimation because they cannot effectively leverage weight sparsity of the intermediate pruned network to lower compuation and memory costs in each iteration. The fundamental challenge  lies in the need to frequently instantiate intermediate pruned sub-models to achieve these savings, a task that becomes infeasible even for moderately sized neural networks.  To this end, this paper proposes a novel pruning method for DNNs that is both computationally and memory-efficient. Our key idea is to develop an effective reweighting mechanism that enables us to estimate the gradient of the pruned network in current iteration via reweigting the gradient estimated on an outdated intermediate sub-model instantiated at an earlier stage, thereby significantly reducing model instantiation frequency. We further develop a series of techniques, e.g., clipping and preconditioning matrix, to reduce the variance of gradient estimation and stabilize the optimization process. We conducted extensive experimental validation across various domains. Our approach achieves 50\% sparsity and a 1.58$\times$ speedup in forward pass on Llama2-7B model with only 6 GB of memory usage, outperforming state-of-the-art methods with respect to both perplexity and zero-shot performance. As a by-product, our method is highly suited for  distributed sparse training  and can achieve a 2 $\times$ speedup over the dense distributed baselines.

Computation and Memory-Efficient Model Compression with Gradient Reweighting
Zhiwei Li Yuesen Liao'r, Binrui Wu', Yuquan Zhou' Xupeng Shi?, Dongsheng Jiang?, Yin Li,, Weizhong Zhang 1.4 Fuudan University, "Northeastern University, Huawei Inc. "Shanghai Key Laboratory of Intelligent Information Processing {zwli23, ysliao24, brwu23, yqzhou21)@m fudan. edu. binpcheng.jinggoouttue com, liyin9@huawei. com weizhongzhang@fuue edu.
Abstract
Pruning commonly employed technique for deep neural networks (DNNs) aiming at compressing the model size to reduce computational and memory costs during inference. In contrast to conventional neural networks, large language models (LLMs) pose unique challenge regarding pruning efficiency due to their substantial computational and memory demands. Existing methods, particularly criimization-based ones, often require considerable computational resources in gradient estimation because they cannot effectively leverage weight sparsity the intermediate pruned network lower compuation and memory costs in each iteration. The fundamental challenge lies in the need to frequently instantiate inter- mediate pruned sub-models to achieve these savings, task that becomes infeasible even for moderately sized neural networks. To this end, this paper proposes a novel pruning method for DNNs that is both computationally and memory-efficient Our key idea is to develop an effective reweighting mechanism that enables us to estimate the gradient of the pruned network in current iteration via reweigting the gradient estimated on an outdated intermediate sub-model instantiated at an earlier stage, thereby significantly reducing model instantiation frequency. We further develop series of techniques, e.g., clipping and preconditioning matrix, to reduce the variance of gradient estimation and stabilize the optimization process. We conducted extensive experimental validation across various domains. Our approach achieves 50% sparsity and 1.58x speedup in forward pass on Llama2-7B model with only 6 GB of memory usage, outperforming state-of-the-- methods with respect to both perplexity and zero-shot performance. As by-product, our method is highly suited for distributed sparse training and can achieve 2 x speedup over the dense distributed baselines.
1 Introduction
The explosive growth in the scale of deep neural networks (DNNs), especially large language models (LLMs) 6 48), has introduced practical challenges, including computational expense, memory
usage, and latency in inference. Model pruning [13 [4] 21 28 34], particularly structured pruning, has emerged as a promising solution, aiming to obtain sparse neural network by removing redundant components. For conventional DNNs, has been reported that existing methods g 23 can enhance
Equal Contribution. Conrespnnding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

the inference efficiency and reduce the memory consumption by orders of magnitudes, with only a slight performance degradation, facilitating the deployment of DNNS on low-power devices.
In contrast to conventional DNNs, LLMs have substantial computational and memory demands. The users in downstream applications always have significantly less computational and data resources for pruning compared to those used for training. Consequently, LLMs pose a unique challenge for pruning. Metric-based pruning methods 2 [5] are proposed overcome this challenge; however, their manually designed rules can adversely affect the performance of the pruned model, which is more evident higher sparsity levels. Moreover, due the differing parameter redundancy across layers, manually setting the pruning rate for each layer is almost impossible. Thus, these methods struggle to achieve global heterogeneous pruning and are often forced to apply uniform pruning across layers [3] 46] Optimization iiriiitttionbbsedd pruning methods [88 [53, which better preserve model performance, typically rely on training with backpropagation, making them difficult to apply in resource-constrained applications
31 57.
Trininng menory (CB)
Derse Memrey Retuinal
12.0
WRITAOD Pepteslty
Readen HAP M
9.0
1.4
6.0
Speed
3.0
35%
30%
Osly suborade for saps. (2) returtstion prine
pruze
1.0
(onoorettin prone
Takaing FECEARY
Treas sponiag
Infornce
Speet(X)
(a) Algorithm Overview Diagram.
(b) Memory Cost & Inference Speed.
(c) PPL Inference Speed.
Figure 1: (a) In the case of T iterations, vanila implementation need to instantiate sub-model times. In contrast, with the reweighting technique, single sub-model can be used continuously for r steps. This is a schematic; in practice, multiple sub-models can be sampled (see Algorithm [ (b) With our method, higher prune rates lead lower training memory cost and faster forward computation. Notably, with 50% prune rate, we prune Llama2-7B (12.6 GB in BF16) using only 6.3 GB memory, while achieving 1.58 x speedup in the forward pass. (c) Circle size is proportional to pruned model's memory usage. We achieve substantial inference speedup while maintaining a low perplexity.
We argue that leveraging the sparse structure of the model is natural idea develop computationally and memory-efficiee pruning algorithms. This can be implemented by instantiating the sparse subnetwork during pruning. However, as the subnetwork structure is updated in each iteration during pruning, the model instantiation needs to be performed frequently, which could leads to unbearable prun-
time cost. To this end, we propose a computationally and memory-efficient optimization- ing algorithm with a novel reweighting technique to reduce the model instantiation frequency. This approach effectively leverages the sparsity by completely discarding pruned parameters, preventing them from participating in computation and storage. Our carefully designed reweighting mechanism enables us to estimate the gradient of the pruned network in current iteration via simply reweighting outdated intermediate submodel instantiated at earlier stage, in this way the model instantiation frequency can be reduced. Our algorithm is illustratively shown in Fig Tal We further integrate reweighting technique with policy gradient estimator to bypass the computationally expensive backpropagation to update the sparse sub-model structural parameters. Finally, considering the additional variance introduced by reweighting and policy gradient estimator, we develop a series of variance reduction techniques to stabilize training, including clipping and preconditioning matrix, which are detailed in Section 4.2
the gradient estimated
Extensive experiments on LLM demonstrate that our method reduces memory requirements during the pruning process and provides substantial computational speedup, particularly under high pruning rate. On the Llama2-7B model, our algorithm achieved 50% sparsity with only 6 GB of memory usage and delivered 1.58 speedup in forward pass, shown in Fig. lb while maintaining state-of-the-att arr performance in terms of perplexity and zero-shot capabilities. In Fig. Icl we present a visual comparison of the model's speed and perplexity after pruning with different methods.
Additionally, our algorithm is highly compatible with distributed sparse training [4] In this setting, we can send the instantiated sub-models to the respective clients. With the reweighting technique, each client can work on its sub-model for long period, reducing the frequency of model weights transmission. During training, by passing only two real numbers, i.c., the reweighting value and

---

the training loss, we can provide clients with essential structural information and update structural parameters on the server based on the latest client results in real time.
Our main contributions can be summarized as follow:
We propose a novel pruning approach that, for the first time, instantiates intermediate pruned sub-models throughout the pruning process. This technique effectively reduces compu- tational and memory costs by limiting the frequency of instantiation, thereby enhancing overall efficiency in pruning.
Our reweighting technique enables unbiased policy gradient estimation based on outdated structural information and reduces instantiation frequency. addition, we design variance reduction techniques to stabilize the pruning process.
Our approach achieves 50% sparsity and 1.58x speedup in forward pass on Llama2-7B with only 6 GB memory usage, outperforming state-of-the-ard methods with respect to both perplexity and zero-shot performance.
As byproduct, our algorithm is inherently well-suited for distributed sparse training, reducing communication overhead during training process.
Related Work
2.1
Conventional Neural Network Pruning
Neural network pruning (20 is a technique aimed at reducing model parameters, focusing on reducing storage costs and inference time. Conventional pruning algorithms [T3] 19
29
38 51 generally
operate on traditional network, establishing criteria (e.g. weight magnitude, importance score) to assess and prune the parameters modules, while minimizing performance degradation. These pruning methods make the deployment of some neural networks on resource-limited devices possible.
2.2 LLM Pruning
For LLMs, due to the large number of parameters and the resource mismatch between training and pruning, the discussed conventional pruning algorithms are often difficult to apply. An increasing variety of pruning algorithms are being designed specifically for LLMs, which can be divided into two main types based whether structural parameters optimization is involved: metric-based and optimization-based
46
63 65
Metric-based pruning is similar to traditional methods mentioned above but is tailored to accommodate specific characteristics of LLMs, such as large number of parameters and the outlier activation in certain channels [ [59] These metries are often more meticulously designed: for instance, Wanda [46] incorporates activations into the importance score: SparseGPT [15] efficiently calculates Hessian's inverse set up metrics and parameter reconstruction; L.MM- Pruner (34] groups the module by graph information, and deletes them together after obtaining the scores; FLAP 2 extends the metric globally through score normalization. These methods are typically limited to layers or matrix, and their performance often deteriorates under high sparsity.
Metric-Based Pruning.
Optimization- Based Pruning. By training the structural parameters, the optimization based prun- ing algorithm learn more effective sparse structure. For instance, [54] [57] employs reparameter- ization to learn masks, pruning the model down to a fixed size, (57) additionally continues pretraining to restore model capability; APT (66] introduces dynamic low-rank structure similar to LoRA (25]. allowing adaptive fine-tuning and pruning. NutePrune B1 employs the model with lower sparsity as teacher for distillation during pruning. Overall, optimization based LLM pruning methods typically require gradient computation through the chain rule, which inevitably demands higher computational costs, making practical application more challenging.
Most L.LM pruning methods fail to leverage sparse structures, and the resource mismatch between training and pruning calls for computationally and memory-efficient algorithm.

---

3 Preliminary
Framework of Probabilistic Masking for Pruning. We adopt the sparse training framework for neural networks proposed in [67]. Given neural network parameterized by W { } 1 where K is the number of modules and w, represents the parameter matrix a module within the network, e.g., head in Attention, binary mask for W. The module wi is pruned when m, and retained when m. = 1. By further reparameterizing the mask with Bernoulli random variabels, i.e., m, Bernoulli (s:)) with the probability 8, to be 1, the structured pruning framework can be formalized as follows:
channel in CNNs. Let I....... mx denotes the corresponding
min E ppim L(W m) -
1 N
Σ ((((; W o om) vi),
s.t. || - (1 - k) and s E [0.
where {a;, y:) the training samples, p denotes the prune rate, ((; W m) represents the sparse neural network, ((:,)) is the loss function. Here, the optimization of W is omitted as we focus on pruning instead of training.
The base solver proposed in [67] optimizes above problem using policy gradient estimator, which enables unbiased gradient estimation of using only forward propagation. After sampling mini batch and m from p( ss), the specific update procedure for each iteration is as follows:
f projs( (* '/9=), where Js = (m(m) V log(p(m)s)),
where - (s |||| < (1 p) K and & E [0, ()) is the feasible domain and projs is the m-a can be computed explicitly and thus
projection operator on Notably, v. log(p(mlss) s(1-8) the entire process is more efficient compared to traditional backpropagation--ased algorithms.
4 Method
In this section, we introduce method in three parts: the reweighting technique to reduce the instantiation frequency Section [.I the detailed framework of efficient pruning in Section 4.2; and finally, the extension to distributed sparse training in Section 4.3
4.1 Reweighted Policy Gradient Estimator
Sub-model Instantiation. We first give the definition of instantiation follows:
Definition 4.1. Given the parameters W of DNN and the corresponding masks we instantiate a compact sub-model by creating a new neural network to delete the pruned modules, which can be written as f (:; W.. with the parameters W.. [wihee and I {i: m, 1, 1..., K}.
It can be seen that if we utilize m to instantiate compact sub-model at each step, we can reduce mem- ory consumption and accelerate forward speed. However, frequent instantiations lead to unbearable time costs. Considering this issue, we propose a new reweighting technique to reduce instantiation frequency, allowing to use outdated compact sub-model to optimize the structural parameters s.
Basic Idea. Our reweighting technique is inspired by importance sampling, and its general form is:
Ex-n() - fxx)p(x)xx ((x))(()) p(x) dx = Ex-o()
f(x) p(x)
(3)
g(x)
q(x)
This equation indicates that the expectation of f(x) over distrbution p(-) equals to that of f(x) p(x)
g(x)
over q(-). The item p(x) is the importance weight. This means one can estimate the mean of random
gix) variable in another probability space by choosing proper distribution.
'The computation of the projection operator is detailed in Appendix D

---

Reweighting Technique Derivation. Based on the above analysis, we develop our reweighted policy gradient estimator below. Firstly, have
V ppim a) ((m)
=V. / ((m)s)) [(mm)
p(mls)m ( +
dm
= / [(m))(mls)) [l log(p)))
P ( 8)dm
p(m]s)
log(om())))
p(m]33)
= [s) V lo (m))
(4)
where & S is the outdated structural parameter. w(m) that transforms the measure p(m)s 8) to p(m) 8). Therefore,
p(m(a) p(mla) denotes the reweighting function
9s w(m) LB(m log(p(mlss)
is an unbiased estimator of V V. (())) m is sampled from the outdated distribution p(m) 8).
Discussion. Let and be the outdated and current structural parameters, respectively. Eqn. demonstrates that timated on an intermediate sub-model m instantiated at an earlier stage from an outdated distribution p(m|s), ()) In this way, we reduce the instantiation frequency. Moreover, the estimation of gs can be computationally and memory-efficient due to the compactess of sub-model f(; Wm).
estimate the gradient of in current iteration via reweigting the gradient
4.2 Framework of Efficient Pruning
Variance Reduction. Before giving the framework of our efficient pruning method, need to address the issue of high variance in our reweighted policy gradient estimator in Eqn. as follows:
(I) Variance of w(m): The reweighting function ((m) can be unstable due to large divergence between the different distributions p(m ss) and p(m](). We clip w(m) [1 e, 1]
following PPO 42), and abbreviate as clip(w((m)),
(2) Variance of (m(m): The training loss Cs(m)'s variation comes from the randomness of the mask m. We minus (m(m) V. (og(p(mm))) s)) by highly correlated and several zero meaned items in form of La(m') V. log(p(m)(s)) where the mask m' is sampled independently with
(3) Variance of V which takes the form of V. log(p(m(ss) .(--))
The potentially small denominator in V log(p(m(s))
m could introduce additional variance. We multiply the gradient with preconditioning matrix H diag(s (1 s))" with a E [2, 1) to offset this impact.
After applying these variance reduction techniques, our reweighted policy gradient estimator becomes:
M
M
Σ
clip(w(m )))
[m( (1)
M (
1
Σ
H (s) log po (s). (6)
M M
where the masks m (,, 1,..., M are sampled independently from the distribution p(m) $).
We have the following theoretical guarantee for variance reduced gradient g²r Detailed theoreti- cal explanation is provided in the Appendix E
Theorem 1. Suppose the masks (ml0)M
j= are independently sampled from the distribution p(m 8),
thenfoo any a E 12, 1) and E [0, 1]K, the variance of g is bounded.
Pruning Algorithm. Finally, we present the complete algorithm in Algorithm Based on reweighted policy gradient estimator with variance reduction, this algorithm enables efficient prun- ing using only the forward pass of the completely sparsified model, thereby avoiding the need for expensive backpropagation.
5

---

Remark 1. Though we have M sub-models, during implementation, they can be processed se- quentially order to save the memory cost. Concurrently, we assign distinct mini batches to each sub-model to ensure equitable allocation of computational resources with the baselines. See Appendix A.2 for details. Experiments under such fair setting verify the superiority of our approach.
Algorithm Efficient Pruning with Reweighted Policy Gradient Estimator
Input: prune rate p. structural parameters 8, instantiation steps т, dense DNN W, learning rate fj-
1: Initialize with certain method.
2: for = 2........ do
3: Set s 8, sample masks (m()) Mi=1 from p(mla), instantiate sub-model [(; W Wm()).
4: for .,2. do
5: Sample mini batch B = { y1), -.. ya))-
6: Get (m(m from each sub-model f(: W ()) and compute (mm
p(m p(m
7: Calculate 9-" according to Eqn.
8: Update projs(s "19"".
9: end for
end for
11:
return Pruned DNN W Wm, where mask m is sampled from the distribution p(m|s).
Discussion. We aummarize the appealing features of our algorithm as follows:
Instantiating a compact sub-model loads only active components into memory, reducing memory usage and accelerating forward passes, unlike existing methods [57 54 58
that
merely mask parameters with zeros and miss the full benefits of sparsity.
If we consider training model weights through backpropagation:
W + W - Twww: where = [p(mma)((( V w [(m) ~ W CB(m), (7)
the instantiation method ensures that gradient information maintains a sparsity level of granting an efficient backpropagation.
We find that our algorithm inherently well-suited for distributed sparse training. Through the use of reweighting technique, it reduces the overhead caused by frequent communication of model parameters in training process. Therefore, we present the application of our algorithm within the framework of distributed sparse training below.
4.3 Extension to Distributed Sparse Training
Distributed training requires the frequent passing of gradients and parameters between the server and clients, which imposes high demands on transfer efficiency. As discussed above, algorithm is inherently well-suited for extending to distributed sparse training. During training process, the server updates the structural parameters based on policy gradient and the client is responsible for training its own sub-model, which only requires the latest structural information from the server. Since the weight parameters are trainable now, we use clearer notation Lg( W (o instead of (m) to distinguish this setting from the pruning task above. Our training process can be divided into four stages:
(I) Instantiating sub-models. To start the training round, the server samples (m 1 from p(m]s) to instantiate {- W m/)) ) and sends them to each client.
(2) Updating sub-model paras. At each iteration, client; receives reweighting value w wmm()) ()) from the server, which provides real-time structural information and is used for updating the current parameters W not via Eqn.(7).
(3) Updating structural paras. Each client returns the loss L W m()) By collecting these losses, the server estimates the policy gradient 9 via Eqn. (6) and updates s.
(4) Aggregating updates. Repeat (2) and (3) for steps. Then each client returns the parameter updates AW m)) and the server aggregates them to update the full model.

---

Throughout the training process, only two scalars, the reweighting function w(m! and the training loss Ls( W m()) of each client are communicated in real time, while the model weight parameters are passed only after every steps. Therefore, in this framework, the communication costs between the server and clients, as well as the instantiation overhead, are greatly reduced. Fig. 4 visually
illustrates the flow of our algorithm. The detailed algorithm is shown in Appendix B
Discussion. We genuinely believe that the distributed sparse training extension we propose is highly non-trivial and elegant. However, its advantages are challenging to fully convey through text alone. We strongly encourage readers to dedicate additional time to review its details along with Fig. and Algorithm in appendix.
5
Experiments
We conduct extensive experiments in this section, which demonstrate the superior performance of our algorithm. In Section [.1] we introduce the experimental setups. In Section 5.2 we compare our method with other pruning methods for LLMs and conduct several ablation experiments in Section Section presents experiments using vision models in distributed sparse training scenario.
5.3
5.1 Experimental Setups
Model Configuration. We utilize the Llama and OPT families including Llama2-(7B. 13B) (48) Llama3Y3. 1-8B [2]. Llamma.2-33B, and OPT-(2.7B, 6. 6.7B) [64]. We apply our pruning algorithm across various structural granularities of the model, including attention heads and MLP channels.
Prune Rate. We target high sparsity levels (30%-50%) to better highlight our method's advantages. Under these settings, instantiated method achieves higher acceleration and memory efficiency.
Baseline. Due to resource limits, we only include limited comparisons with optimization-base methods (e.g.. Compresso [8], NutePrune (31) in Section 5.3 Our main focus on metric-based SOTA methods, including LLM-Prunee [84), SliceGPT
B Wanda-sp, and FLAP 2 No weight
fine-tuning performed post-pruning for fairness.
Training Details. We follow (34], using C4 [40] for training and Wikitext2 (36), PTB [35] for
testing, which reflect the generalization of the pruned model. Adam (26) is used to optimize structural parameters 8 with a learning rate of 5e-3 and batch size 8. Training is conducted for one epoch with frozen model weights. Further details are provided in Appendix A
5.2 Pruning Results on LLMs
The main experimental results are presented in Table At various sparsity levels, our pruning algorithm achieves lower perplexity compared to other tatte-oftth-ane methods. This advantage is more pronounced at 50% sparsity, highlighting the superior performance of our optimization- based algorithm over other metric- based methods.
Table 1: Performance comparison across methods and models under different prune rates. The lowest and second lowest PPL are in bold and underlined, respectively. denotes unsupported model types.
Prune rale
Method
Llama2-78
Llama2-138 138
LlumaJ-a 3-B
Lium631-55
OPT-2788
GPT4.78
WikFTott
WilTeett2
Ptb
WilTeett2
WilTTest2
WildTest2
WilTess22
Ptb
WaiTeu22
2,67
7.61
15.16
LLM Pruncr
27.15
111.16
15.19
125.95
20.15
30.32
19.23
31.30
SikeOPT I
23.31
45.14
17.35
72.58
30.25
15.37
19.70
25.49
Wastbopp
76
29.50
23.71
23.85
11 M-Phirnre
43.79
171.91
21.51
181.0%
45.30
40.55
01.11
06
SikeOPT I
41.16
148.59
33.30
103.45
30.07
125.08
32.06
44.10
Wasde-pp
31.72
30.06
76.47
20.1
17.40
$6.18
73.76
46.70
35301
11 M-PPinee
125.33
474-44
48.44
769
170 16
310.77
141 24
151.44
SikaCCPP
16
60.97
$0.01
Wasds-pp
112.30
102.22
160.47
71.92
89.04
175.58
173
163.0
Chirs
26.83
44.35
21.21
34.47
$1.14
74.13
$7.80
98.38
R3.6
11641
65.86
$3.75
65.96

---

We evaluate the generalization performance of our pruned models based on the settings from [4]. using five zero-shot tasks from LM Evaluation Hareess [6]: PIQA [.. HellaSwag 662, ARC-e ARC-c [ and OBQA (37] and recorded their average performance. We use Llama2-7B and Llama3-8B as baseline models. As shown in the Table structured pruning does have significant impact on the model's s generalization ability. At high sparsity levels, datasets like ARC-e and ARC-c show a notable performance decline. However, our algorithm still maintains the best among all methods, demonstrating its effectiveness in preserving model capacity.
Table 2: Comparison of zero-shot performance between Llama2-77 2-BB and Llama3-8B
Prune rute
Method
Lluma2-7B
Lhama-8BB
1 PIQA
HellaSwag
ARC-
ARC-
0BQ4
Average
PIQA
HellaSwag
ARC-
ARC-
OBQA
Average
Danse
78.74
76.00
74.62
46.33
44.50
63.98
800.79
70.19
77:69
53.41
45.00
67.22
LLM Fruner
72.09
58.51
57,07
33.45
38.00
52.06
69.37
44.63
54.08
30.63
31.20
45.98
30%
SlacsCiPT
73.39
60.41
52.36
12.25
52.50
50.24
20:02
57.34
29.54
30.30
32.00
47.82
Ours
63.06
34.90
38,80
72.74
56.52
36.89
33.60
1M-Praser SlxcGPPT
43.39
42.98
41.96
23.63
77
37.63
Curs
53.89
48.70
29.89
32.20
46.69
65.94
45.03
46.13
36.02
30.40
42,70
To demonstrate the effectiveness of our algorithm in reducing memory usage and accelerating the forward pass during the pruning process, we present the instantiated model parameter nums, multiply- accumulate counts (MAC), memory usage, and inference speed different prune rates in Table It is clear that the resource savings achieved by our algorithm are proportional to the model's sparsity.
Table Statistics on pruned model. Inference Speed is evaluated by generating 400 tokens.
Table 4: Comparison of initialization methods for on Llama2-7B, using PPL.
Prune rate
Params(B)
MACx(G)
Memory(Mill)
Speed(x)
Datuset
Prune rate
Random
1 1-p
Wanda-sp
FLAP
0%
6.61
845.71
12607.57
1,00
50%
30%
4.66
596.06
8888.2 24
L.24
35%
4.32
552.57
8239.75
1.30
Average
30:01
20.19
1905
21.39
40%
4.01
512.85
469.90
6999.97 97
7648.41
1.55
1.43
Pb
50%
41%
137.33
66.41
24.31
44.58
22.18
73.51
35.64
44.25
1588
61.22
10.00
18.10
50%
3.35
429.09
6389.62
1,58
Average
36.47
5.3 Furthur Analysis
We conduct further studies on the following aspects: 1) different initialization methods for the structural parameters s; 2) effectiveness the reweighting and instantiation technique; 3) ablation study for the variance reduction techniques; 4) comparison with the optimization-bast methods.
Initialization Methods Comparsion. Four methods are employed to initialize the structural param- eters including: 1) Uniform(O, 1) random values; 2) 1- - p directly; 3) the Wanda-sp score; 4) the FLAP score"] From the results in Table 4 it can be observed that using Wanda-sp initialization yields the best performance, followed closely by FLAP initialization. This indicates that initializing the structural parameters s with heuristic metrics can facilitate the optimization process. Additionally, we find that random initialization is already sufficient to surpass metric-basee based pruning algorithms.
Effects of Reweighting and Instantiation. As shown in Fig. 2 as the number of instantiation steps increases, perplexity gradually rises due to the accumulation of errors from the outdated sub-model Wm and gradients 9a- However, by using the rewcighting function, this issue can be effectively alleviated. Even with 100 steps, the perplexity than SliceGPT's 71.72. Meanwhile, in (c), we observe that after applying reweighting, all models in the experiment exhibit significant reduction in perplexity. Therefore, we believe the reweighting function makes infrequent instantiation possible, which helps us fully leverage the sparse structure, resulting in memory savings and forward acceleration.
WikiText2 still reaches 71.01, which is better
'Since the scores of Wanda-Si SP and FLAP do not fall within the range (0, 1). normalize these scores to form probability distribution. For different prune rates, scale them to the desired sparsity level.
8

---

150 Ne Suggtee
WilaTboo Puplosty
120
sovouit
Miaed 19
sovegdi
:00
Wanlesp
via sovcldi
200
0976.7
StazfPT
0772770
15
OPT VMM
50
30
150
Inskatiation stepo
WAkTent Pepplosay
Thrining stgge
(a) Effect of T on pruning.
(b) Results at 10.
(c) Reweighting on training.
Figure 2: (a) For different instantiation step т, reweighted version consistently outperforms the w/o reweighted baseline. Considering the trade-off between instantiation time and pruning performance. we fix r 10 throughout all experiments. (b) Under our fixed setting T = 10 for all experiments, reweighting technique improves pruning across all models. Notably, on OPT-2.7B, the impact of reweighting can reach up to 11.7 compared to w/o reweighting case. (c) Reweighting technique stabilizes the training process, leading to better pruning. (a), (b), (c) are based on the 50% prune rate.
Variance Reduction, We employ clipping and preconditioning matrix to stabilize the training process and conducted ablation experiments verify their contribution. The results in Table shows that omitting either the clipping or the preconditioning matrix negatively impacts the performance of the pruned model. Among these, the clipping has more evident effect on performance. Compared to our final algorithm, not using clipping increases the perplexity on the Wikitext2 dataset by 12.44, indicating that our variance reduction techniques enhance the stability of the training process. Details of the variance reduction analysis can be found in Appendix C
shows
Table 5: Variance reduction study, 50% Llama2-7B.
Table 6: Overhead test, 50% Alama2-7B.
Method
WileTess2
PPQA
HellaSwng
ARC-
ARCx
Average
Methol
WilTeest2
Fervardis)
Bodivardis
MenoryMMil
62,89
25,00
42.02
Gun
26.33
Now
6750 21
24.40
40,35
Compesso!
49.28
0004
33296.74
cli
69.28
60,83
29.11
38.88
23.04
40.42
NucPone
0,00
57460778
Note: evaluation metries: Perplexity (WikiText2, PTB) and zero-shot accuracy
Note:
pruning with LoRA adaption. co-training the masks and LoRA modules, also involves distillation.
(PIQA, HellaSwag, ARC-o, ARC-e and their average).
Comparison with optimization- methods. Our method outperforms other optimization-
based pruning algorithms by significantly reducing memory usage and training time. To demonstrate the superiority of our algorithm, we select Compresso [18] and NutePrune [31] for comparison. Since these methods use different training settings, focus on memory usage and time consumption in single optimization iteration. Table 6 shows that our algorithm achieves comparable performance while consuming less memory, offering faster forward time, and requiring no backpropagation.
5.4 Distributed Sparse Training
Based on the method described in Section
we conduct extensive experiments within the framework the distributed backend and built an 8-node cluster (each node equipped with a GPU). The worker nodes connected via 40 Gbps (5000 Mb/s) Ethernet interface. The baseline algorithms include
4.3
of distributed sparse training. We gloo 43
for training on the ImageNet- 1K dataset 41
LocalSGD [5), PowerSGD [50]. TSNNS 10]. and our base solver EffTrain [67] (which performs sparse training locally without using reweighting techniques, described in Section 3). We set the instantiation sampling interval r to 20 steps. The detailed experimental configuration and baseline
description can be found in Appendix A. The specific comparison results are shown in Table 7
Table 7: Comparison of methods across ResNet-50, MobileNet-V V1 and DeiT-Base ImageNet-IKe IK.
Method
ResNet-50
MobileNet-VI
XX]
DeiT-
Params(()))
FL.PP((S)).
Time|
Acc(%)
Params(%)
FLOPPsS))
Time
Acc(S6)
Parmme/S5 (6)
FLOPP495)
Time
Dense Dist.
76.9
100
100
23.00
72.0
100
100
8.66
$1.8
100
100
$3.05
SGI
25.0
100
100
100
40.41
PowerSGi
26.0
100
19.25
71.3
100
5.50
79.5
100
100
TSNNS
74.5
61.2
77.4
19.49
70.4
88.5
87.9
3.69
78.5
80.3
79.4
30:37
ElTrain
76.1
48.2
46.8
115.44
71.5
68.1
69.2
61.36
80.1
67.1
68.2
288-24
Ours
75.2
48.1
86.5
9.79
71.6
67.4
68.9
4.15
$0.9
66,5
67.3
26.70
Our method achieves excellent performance across three different models, surpassing other distributed training algorithms, while also reducing model params and FLOPs. The base solver EffTrain also

---

achieves superior performance and model compression; however, due to the frequent instantiation operations, its training time is significantly longer compared distributed algorithms. In conclu- sion, our algorithm integrates well into the framework of distributed sparse training. By using the reweighting technique to reduce instantiation frequency, achieves promising results in less time.
Conclusion
This paper introduces novel DNN pruning algorithm that is computationally and memory--fficient For the first time in pruning research, we define model instantiation to leverage sparsity for memory savings and forward acceleration. Additionally, we propose reweighting technique that reduces instantiation frequency. making it feasible to prune LLMs under resource constraints. As a byprod- uct, algorithm seamlessly integrates into distributed sparse training environments. Extensive experiments demonstrate the excellent practical performance of our pruning algorithm, and numerous ablation studies validates our proposed techniques' effectiveness. Besides, we also provide thorough theoretical analysis to support our algorithm.
7
Acknowledgements
This work was supported by the National Nature Science Foundation of China (62472097), Shanghai Municipal Science and Technology Commission (Grant No.245111 1061102) Al for Science Foundation of Fudan University (FudanX24A1028) and Fudan Kunpeng& Ascend Center of Cultivation. The computations in this research were performed on the CFFF platform of Fudan University.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.088: 087744 2023.
[2] Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation-base adaptive structured pruning for large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10865-10873, 2024.
[3] Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations, 2024.
[4] Tal Ben-Nun and Torsten Hoefler. Demystifying parallel and distributed deep learning: An in-depth concurrency analysis. ACM Computing Surveys (CSUR), 52(4):1-43, 2019.
[5] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piga: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages
2020.
[6] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 1877-1901, 2020.
[7] Tianyi Chen, Tianyu Ding, Badal Yadav, Ilya Zharkov, and Luming Liang. Lorashear: Ef- ficient large language model structured pruning and knowledge recovery. arXiv preprint arXiv:2310.18355 2023.
[8] Hongrong Cheng, Miao Zhang, and Javen Qinfeng Shi. A survey on deep neural network pruning: Taxonomy, comparison, analysis, and recommendations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[9] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1803.00800.0045555 2018.
10

---

[10] Selima Curci, Decebal Constantin Mocanu, and Mykola Pechenizkiyi. Truly sparse neural networks at scale. arXiv preprint 20X.v22102.01722. 2021.
[11] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers scale. Advances in Neural Information Processing Systems, 11:0018-303322 2022.
[12] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
[13] Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International conference on machine learning, pages 2943 2952. PMLR, 2020.
[14] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference Learning Representations, 2018.
[15] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323-10337. PMLR, 2023.
[16] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf. Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou, A framework for shw-shot language model evaluation, 07 2024.
[17] Yuan Gao, Zujing Liu, Weizhong Zhang, Bo Du, and Gui-Song Xia. Optimization- based structural pruning for large language models without back- pocck-ropagation. arXiv preprint arXiv:2406. I0576, 2024.
[18] Song Guo, Jiahang Xu, Li Lyna Zhang, and Mao Yang. Compresso: Structured pruning with col- laborative prompting learns compact large language models. arXiv preprint arXiv:2310.05015, 2023.
[19] Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. Advances neural information processing systems, 29, 2016.
[20] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
[21] Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992.
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[23] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241):1-124, 2021.
[24] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: efficient convolutional neural networks for mobile vision applications (2017). arXiv preprint arXiv:/704.04861/ 126, 2017.
[25] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[26) Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Interna- tional Conference on Learning Representations (ICLR 2015), 2015.
11

---

[27] Jongwoo Ko, Seungjoon Park, Yujin Kim, Sumyeong Ahn, Du-Seong Chang, Euijai Ahn, and Se- Young Yun. Nash: simple unified framework of structured pruning for accelerating encoder-decoder language models. In Findings of the Association for Computational Linguisties: EMNLP 2023, pages 6076-6093, 2023.
[28] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
[29] Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. Layer-adaptive sparsity for the magnitude-baas based pruning. In International Conference on Learning Representations, 2021.
[30] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations, 2017.
[3]] Shengrui Li, Junzhe Chen, Xueting Han, and Jing Bai. Nuteprune: Efficient progressive pruning with numerous teachers for large language models. arXiv preprint arXiv:2402.09772 2024.
[32] Yun Li, Lin Niu, Xipeng Zhang, Kai Liu, Jianchen Zhu, and Zhanhui Kang. E-sparse: Boosting the large language model inference through entropy-based N: M sparsity. arXiv preprint arXiv:2310. 15929, 2023.
[33] Lucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Daniela Rus. Provable filter pruning for efficient neural networks. In International Conference on Learning Representations.
[34] Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702-21122 2023.
[35] Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):3133300 1993.
[36] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2022.
[37] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381-2391, 2018.
[38] Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In CVPR, pages 11264-11222 1172, 2019.
[39] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026-8037, 2019.
[40] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
[41] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 11:2111-252 2015.
[42] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint
2017.
[43] Alexander Sergeev and Mike Del Balso. Horovod: fast and easy distributed deep learning in tensorflow. arXiv preprint arXiv: 1802.009000 2018.
[44] Karen Simonyan. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1409. 1556, 2014.
[45] Sebastian U Stich. Local sgd converges fast and communicates little. arXiv preprint 1rXiv:1805.09767, 2018.
12

---

[46] Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2024.
[47] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
[48] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arviv preprint S.KY::2307.02288, 2023.
[49] Tycho FA van der Ouderaa, Markus Nagel, Mart van Baalen, Yuki M Asano, and Tijmen Blankevoort. The Ilm surgeon. In International Conference of Learning Representations (ICLR), 2024.
[50] Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. Powersgd: Practical low-rank gradient compression for distributed optimization. Advances in Neural Information Processing Systems, 32, 2019.
[51] Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Neural pruning via growing regularization. In International Conference on Learning Representations (ICLR), 2021.
[52] Weiran Wang and Miguel A Perpiia.. Projection onto the probability simplex: An efficient algorithm with a simple proof, and an application. arXiv preprint 2013.
1541,,
[53] Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), Jan 2020.
[54] Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6151-6162, 2020.
[55] Boyi Wei, Kaixuan Huang, Yangsibo Huang. Tinghao Xie, Xiangyu Qi. Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications. arXiv preprint 240.0022002..1122.. 2024.
[56]
Wikipedia. Var reduction. http:////////..kk......aao./a//a
[57] Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared Ilama: Accelerating language model pre-training via structured pruning. In The Twelfth International Conference on Learning Representations, 2024.
[58) Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume I: Long Pupers), pages 1518-1528, 2022.
[59] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087-38099, PMLR, 2023.
[60] Lu Yin, You Wu, Zhenyu Zhang, Cheng Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, AJAY KU- MAR JAISWAL, Mykola Pechenizkiy, Yi Liang, et al. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning Ilms to high sparsity. In Forty-first International Conference on Machine Learning, 2024.
[61] Xin Yuan, Pedro Henrique Pamplona Savarese, and Michael Maire. Growing efficient deep networks by structured continuous sparsification. In International Conference on Learning Representations, 2021.
[62] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800 2019.
13

---

[63] Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, and Bohan Zhuang. Pruning meets low-rank parameter-efficient fine-tuning, 2023.
[64] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
[65] Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. Plug-and alayyay An efficient post-training pruning method for large language models. In The Twelfth International Conference on Learning Representations, 2024.
[66] Bowen Zhao, Hannaneh Hajishirzi, and Qingqing Cao. Apt: Adaptive pruning and tuning pretrained language models for efficient training and inference. Forty-first International Conference on Machine Learning. 2024.
[67] Xiao Zhou, Weizhong Zhang. Zonghao Chen, Shizhe Diao, and Tong Zhang. Efficient neural network training via forward and backward propagation sparsification. Advances neural information processing systems, 34:15216-15222 2021.
[68] Xiao Zhou, Weizhong Zhang, Hang Xu, and Tong Zhang. Effective sparsification of neural networks with global sparsity constraint. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2021.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main contributions of this paper are the high-performance and high- efficiency pruning method, which is elaborated in detail in the method section. Additionally, extensive experimental validation has been conducted, effectively reflecting the contributions of this paper.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitations of our work in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
Justification: We provide a full derivation of our theoretical results, including assumptions and proofs, in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide detailed descriptions of all experimental configurations. and we believe the results can be reproduced regardless of whether the code and data are provided. Additionally, we will make our experimental code available in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
16

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: Yes, we will make our experimental code available in the supplementary materials. The data used in our experiments is open source.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https::/niis.cc.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips. wwwpppppccggpuueeeeeeeeooopppppe /odeeeeeeoodSeumiiiccc cetbmmmsssinnp ssinnooiccyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental configuration has been thoroughly described throughout the manuscript and supplementary materials (see Section 5.1 and Appendix A for complete details).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
The statistical metrics we used are all officially defined and well-established.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The computing platform and training resources employed in our experiments have been thoroughly documented (See Appendix A for details).
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: This work has been conducted in full accordance with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed the broader impacts in Section G
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our study does not carry these risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We used public data and models under the license and terms, which were also properly cited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
latasets.paperssit
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
We did not release any new assets in this work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our experiments didn't 't include the crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our experiments did not include the crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: LLMs serve as the primary experimental subjects in our study. As detailed in Section we provide comprehensive documentation (1) the methodology for pruning
LLMs, (2) evaluation protocols for pruned models, and (3) performance analysis of the
models' generative capabilities post-pruning.
Guidelines:
The answer NA means that the core method development in this research does not non Please refer to our LLM policy (https: /tttps:/neeurps... www.eeuripss..ooooeeeeeeeeeeeeee www.on/eeeecncc//2222222222 for what should or should not be described.
involve LLMMs as any important, original, non- non-standard components.
21


Reviews:
[{"review_id": "Uhi5I0JOYK", "reviewer": "Reviewer_FimZ", "summary": "This work introduces a reweighting-based mask updating method for pruning DNNs, particularly Large Language Models (LLMs), with a focus on computational and memory efficiency. The primary challenge addressed is the computational complexity and memory redundancy associated with instantiating intermediate pruned sub-models. Instead of relying on back-propagation for the entire neural network, this method estimates the gradient of the pruned network. The authors highlight that the DNN feed-forward-only mechanism can also be generalized to sparse distributed training. The results demonstrate state-of-the-art performance in LLM and DNN compression compared to existing pruning algorithms.", "questions": "Super LLMs and Pruning Limitations\n* Resource Requirements: What are the potential resource requirements for super LLMs to demonstrate the limitations of pruning?\n* Network Growth and Redundancy: Theoretically, at what point does network growth indicate sufficient redundancy to warrant pruning?\n\nDistributed Sparse Training and Asynchronous Pruning\n* Full-Sync Optimization: Does the current distributed sparse training framework necessitate full-synchronization for optimization?\n* Asynchronous Training and Framework Adjustments: If asynchronous training is implemented, what adjustments are required? Would a separate framework be necessary to accommodate asynchronous pruning?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths \n* The reweighted policy gradient estimator approximates the gradient of the mask distribution parameter using outdated sub-model instantiation results, significantly reducing model instantiation frequency.\n* Variance reduction in the pruning framework is mathematically proven to be bounded, demonstrating the framework's stability for practical application.\n* The framework runs in parallel with model training, making it compatible with existing methods for simultaneous model finetuning and compression.\n* As a byproduct, the framework is also compatible with sparse distributed training.\n* The reweighting-based pruning framework achieves state-of-the-art performance compared to existing pruning frameworks for DNNs and LLMs.\n* The presentation is logical, smooth, and clear, making it easy for future researchers to extend the work. Detailed discussions on theorems, implementation, and code are provided in the appendix.\n\n\n\n\nWeakness\n* The study highlights that pruning performance on Large Language Models (LLMs) remains largely unexplored. While the proposed method outperforms other LLM pruning techniques, the pruned network significantly underperforms the original LLM. This disparity might stem from LLMs' inherent sensitivity to noise. However, computational resource limitations prevented the application of larger, more redundant networks to validate this hypothesis. Consequently, further research is needed to determine if the trade-off between LLM complexity and efficiency justifies this approach to LLM pruning.\n* minor: line 159 compactess -> compactness", "was_revised": true, "final_justification": "All of my concerns has been addressed by the rebuttal. After combining all the comments, I'd like to keep the rating 4 for this work.", "rebuttal": "## W1: Further research is needed to determine if the trade-off between LLM complexity and efficiency justifies this approach to LLM pruning.\n\nWe'd like to clarify as follows.\n\n**Reasons on Unexplored Pruning for LLMs.** We observe that LLMs often exhibit performance degradation after pruning, especially under high sparsity structural pruning. We argue this phenomenon may be attributed to two main factors:\n\n1. A mismatch between pruning and training resources. The data scale and computing resources used for pruning are much lower than those used for training, which hinders its ability to achieve the desired effectiveness.\n2. Potentially lower redundancy in LLMs compared to vision models. One possible explanation is that language data contains less redundancy than image data, in which local pixel regions typically exhibit high similarity. However, this remains unclear under the current limited pruning resources.\n\nBased on the above background, we emphasis the key contributions of our work as follows:\n\n1. The memory cost during pruning is significantly reduced by our method, making it more feasible to prune larger models with the same memory resource.  This is important for the downstream practioners of LLMs, where the resource is always limited. As shown in **Table R1**, our method incurs very low memory overhead.\n\n2. Our method has higher effiency. We emphasize that improving pruning performance  is closely tied to the efficiency of the algorithms, especially for the large scaled networks like LLMs. A more efficient pruning algorithm allows for more iterative updates, enabling more sufficient exploration and refinement on sparse structures. As shown in **Table R1**, extending the training time to match that of the baselines leads to more significant performance superority.\n\n3. Our pruning method achieves SOTA performance compared to the baselines.\n\n**Table R1. Comparison of pruning performance, training time, and memory overhead under 50% prune rate of Llama2-7B. To ensure fair experimentation, we extended the training schedule of our method to match comparable time budgets.**\n|Method|WikiText2 PPL↓|Ptb PPL↓|Time cost|Memory(MiB)|\n|-|:-:|:-:|:-:|:-:|\n|Ours(1 Epoch)|26.83|44.23|3.6h|6390.21|\n|Ours(2 Epochs)|23.46|41.79|7.2h|6390.21|\n|Ours(3 Epochs)|**21.52**|**38.69**|**10.7h**|**6390.21**|\n|Compresso|49.28|78.90|16.8h|33298.74|\n|NutePrune|24.89|40.83|13.4h|37460.78|\n\n## Q1: What are the potential resource requirements for super LLMs to demonstrate the limitations of pruning?\n\nFor medium-sized networks, i.e., ResNet for image classification, existing pruning algorithms often require resource investments comparable to full training in order to achieve satisfactory pruning results, like 4 GeForce RTX 4090. By analogy, for LLMs with more complex parameter structures and larger parameter scales, it is reasonable to hypothesize that approaching the limit of pruning performance similarly demands computational resources and data volumes on par with pre-training.\n\n## Q2: Theoretically, at what scale does network growth imply sufficient redundancy to make pruning worthwhile?\n\nMost existing neural network inherently contain a certain degree of redundancy, as they are manually designed and cannot guarantee that each parameter is optimally utilized to represent knowledge. Consequently, pruning is essential for improving efficiency in applications where extreme time efficiency is required.\n\n## Q3: Does the current distributed sparse training framework necessitate full-synchronization for optimization?\n\nOur framework can be integrated with both  synchronized and asynchronized optimization flexibly. It is currently designed as a synchronized optimization process, since synchronized optimization is widely used in distribued training systems. \n\nTo support asynchronous optimization, only Step 8 in Algorithm 2 needs to be modified. Instead of waiting for all workers to return their parameter updates $\\Delta W_{m(i)}$ before updating the global model, the server can update the global weights using each received gradient individually as follows:\n\nFor each incoming $\\Delta W_{m(i)}$ from worker i, update the global weights as:\n\n$$\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad W \\leftarrow W + \\eta_{3} \\Delta W_{m(i)},$$\n\nwhere $\\eta_{3}$ is a small learning rate that controls the pace of asynchronous updates.\n\nThis minor modification would allow the framework to run fully asynchronously without requiring a complete redesign, while still benefiting from the lightweight communication scheme enabled by the sparse structure and reweighting strategy.\n\n## Q4: If asynchronous training is implemented, what adjustments are required? Would a separate framework be necessary to accommodate asynchronous pruning?\n\nThe asynchronous version can be implemented following the answer in **Q3**, without the need for a separate training framework. We provide experimental results of asynchronous sparse training on ResNet-50, as shown in **Table R2**.\n\n**Table R2. Comparison of asynchronous sparse training results**\n|Method|Acc(%)↑|Params(%)↓|FLOPs(%)↓|Time↓|\n|-|:-:|:-:|:-:|:-:|\n|Dense Dist.|76.9|100|100|23.00|\n|Local SGD|75.0|100|100|20.52|\n|PowerSGD|76.0|100|100|19.25|\n|TSNNS|74.5|63.2|77.4|19.49|\n|EffTrain|76.1|48.2|46.8|115.04|\n|Ours(Syn)|75.2|48.1|46.5|9.79|\n|Ours(Asyn)|74.7|48.2|46.7|9.13|"}, {"review_id": "1AKoYk3qfV", "reviewer": "Reviewer_bDVm", "summary": "This paper addresses the challenges of pruning large language models (LLMs), where conventional methods struggle due to the high computational and memory overhead of repeated sub-model instantiations. The authors propose a novel, resource-efficient pruning approach that introduces a reweighting mechanism to estimate gradients of the current pruned model using gradients from previously instantiated sub-models. This significantly reduces the frequency of model instantiation. Additional techniques such as gradient clipping and preconditioning are introduced to stabilize optimization and reduce variance. The method is empirically validated across domains, achieving 50\\% sparsity and 1.58$\\times$ forward-pass speedup on LLaMA2-7B using only 6 GB of memory, with improved perplexity and zero-shot performance compared to prior work. Furthermore, the technique demonstrates strong potential for distributed sparse training, achieving 2$\\times$ speedup over dense baselines.", "questions": "Is the m in Equation 1 either 0 or 1? Can we prune part of the each sub-module?", "limitations": "See above.", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper presents a clear and concise mathematical formulation, including only the essential equations—particularly for the variance reduction mechanism—without unnecessary complexity.\n2. The ablation studies are thorough and well-structured, effectively isolating the impact of each proposed component on addressing key bottlenecks in pruning.\n3. The experimental evaluation is comprehensive, covering both algorithmic metrics (e.g., perplexity, zero-shot performance) and practical system-level outcomes (e.g., memory usage and actual speedup).\n\nWeaknesses:\n1. The manuscript repeatedly emphasizes the three main contributions throughout multiple sections, leading to redundancy and detracting from the overall clarity and flow of the writing.\n2. The empirical comparison with prior optimization-based pruning methods is limited, relying on only two baseline works. A broader and more diverse set of baselines would strengthen the validity of the claims.\n3. The trade-off introduced by the variance reduction techniques appears to slightly degrade model performance in Table 5. It would be helpful to evaluate whether variance reduction is strictly necessary—can the method maintain acceptable accuracy while further improving throughput and memory consumption if the variance reduction is omitted?", "was_revised": true, "final_justification": "The rebuttal addresses my confusion and makes the paper solid. I will raise my score to 4.", "rebuttal": "## W1: Repeated emphasis on three main contributions.\n\nWe will revise the manuscript to improve clarity and make it easier for readers to follow.\n\n## W2: Lack of comparison with optimization-based methods.\n\nThanks for your suggestion. We additionally include experiments based on the optimization-based pruning method Compresso, NutePrune, LoRAShear [r1], MaskPrune [r2] with different models and prune rates. More comparison will be included in the revised version if accepted. For a fair comparison under equal training time, we extend the training epochs of our method to fully exploit its effienciy. We additionally report the time and memory overhead incurred by each method during the pruning process. The results are summarized in **Table R1** and **Table R2**.\n\n**Table R1. Comparison with various optimization-based methods on Llama2-7B. To ensure a consistent training time budget, we extend the training epoch accordingly. Meanwhile, we also record the memory overhead for reference. The fine-tuned version is trained on the Alpaca dataset.**\n|Prune rate|Method|WikiText2 PPL↓|Ptb PPL↓|WikiText2 PPL(&Fine-tune)↓|Ptb PPL(&Fine-tune)↓|Time cost|Memory(MiB)|\n|-|-|:-:|:-:|:-:|:-:|:-:|:-:|\n|0%|-|5.47|8.39|-|-|-|-|\n|20%|Ours(1 Epoch)|9.67|13.94|8.35|13.18|4.8h|10224.34|\n|20%|Ours(2 Epochs)|9.44|13.70|8.02|12.90|9.6h|10224.34|\n|20%|Ours(3 Epochs)|**9.35**|**13.48**|**7.94**|**12.57**|14.5h|10224.34|\n|20%|Compresso|11.47|21.25|9.92|18.39|16.8h|33298.74|\n|20%|NutePrune|9.88|16.64|8.74|14.90|13.4h|37460.78|\n|50%|Ours(1 Epoch)|26.83|44.23|14.39|30.07|3.6h|6390.21|\n|50%|Ours(2 Epochs)|23.46|41.79|12.79|25.42|7.2h|6390.21|\n|50%|Ours(3 Epochs)|**21.52**|**38.69**|**12.55**|**25.20**|10.7h|6390.21|\n|50%|Compresso|49.28|78.90|40.83|67.08|16.8h|33298.74|\n|50%|NutePrune|24.89|40.83|12.94|26.04|13.4h|37460.78|\n\n**Table R2. Comparison with various optimization-based methods on Llama-7B. Zero Shot task includes BoolQ, PIQA, HellaSwag, Winogrande, ARC-e, ARC-c and OBQA.**\n|Prune rate|Method|Zero Shot Avg. Score↑|Zero Shot Avg. Score(&Fine-tune)↑|Time cost|Memory(MiB)|\n|-|-|:-:|:-:|:-:|:-:|\n|0%|-|63.25|-|-|-|\n|20%|Ours(1 Epoch)|60.37|61.71|4.8h|10173.46|\n|20%|Ours(2 Epochs)|**61.34**|**61.97**|9.6h|10173.46|\n|20%|Compresso|-|60.75|16.8h|33298.74|\n|20%|NutePrune|59.46|61.46|13.4h|37460.78|\n|20%|LoRAShear|-|60.63|-|-|\n|20%|MaskPrune|61.17|-|-|-|\n|50%|Ours(1 Epoch)|51.73|53.26|3.6h|6239.06|\n|50%|Ours(2 Epochs)|**51.94**|**53.68**|7.2h|6239.06|\n|50%|Compresso|-|46.87|16.8h|33298.74|\n|50%|NutePrune|50.35|52.13|13.4h|37460.78|\n|50%|LoRAShear|-|50.39|-|-|\n|50%|MaskPrune|49.92|-|-|-|\n\n## W3: The variance reduction in Table 5 seems to degrade the performance.\n\nThere may be some misunderstanding. As shown in Table 5, removing the components of our variance reduction technique (either the clipping operation or the preconditioning matrix H) leads to a performance degrade. \"Ours\" in Table 5 refers to our pruning method equipped with the full variance reduction technique. It is important to note that for WikiText2 and Ptb, the evaluation metric is perplexity, where lower values indicate better performance, whereas for PIQA, HellaSwag, ARC-e, and ARC-c, the metric is accuracy, where higher values are better. Taken together, these results demonstrate that the variance reduction strategies contribute positively to performance and remain an integral part of our method.\n\n## Q1: Is the $m$ in Equation 1 either 0 or 1? Can we prune part of the each sub-module?\n\nFirst, yes, in Equation 1, $m$ is either 0 or 1, representing the retention state of a sub-module. Then, we can prune part of each sub-module by setting finer-grained mask. In existing works (e.g., Sheared LLaMA [r3] and FLAP [r4]), structural pruning is typically applied at the level of attention heads or the intermediate dimensions of FFNs, as such sparsity patterns can be efficiently implemented on various hardware platforms to achieve speedup. Therefore, our method adopts these default settings. However, thanks to the flexibility of our masks, our approach can also be extended to support finer-grained pruning. For instance, within each attention head, we can assign individual masks to the intermediate dimensions corresponding to Q, K, and V, enabling more fine-grained pruning strategies.\n\n## Reference\n\n[r1] Chen T, Ding T, Yadav B, et al. Lorashear: Efficient large language model structured pruning and knowledge recovery. ArXiv 2023.\n\n[r2] Qin J, Tan J, Zhang K, et al. MaskPrune: Mask-based LLM Pruning for Layer-wise Uniform Structures. ArXiv 2025.\n\n[r3] Xia M, Gao T, Zeng Z, et al. Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning. ICLR 2024.\n\n[r4] An Y, Zhao X, Yu T, et al. Fluctuation-based adaptive structured pruning for large language models. AAAI 2024."}, {"review_id": "HRCl97Abre", "reviewer": "Reviewer_ubKW", "summary": "This paper introduces a computationally and memory-efficient pruning algorithm for LLMs, which addresses a key practical challenge in optimization-based structured pruning — the high overhead of sub-model instantiation. The core idea is a reweighting scheme, which reuses the old sub-model gradients to estimate the current gradient. The paper integrates multiple variance reduction techniques and demonstrates the method’s advantages in both standalone and distributed pruning scenarios.", "questions": "See weaknesses.\n\nIf the authors can solve some of my confusion in experiments, the practical significance of the paper will be greatly improved, and I will be happy to increase my score.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. Novel reweighting-based pruning technique with strong theoretical and practical grounding.\n2. Extensive experimental coverage across LLaMA, OPT, and vision models.\n\nWeaknesses\n1. I'm wondering about the results beyond 50% sparsity and robustness under extreme pruning, which is lacking in the paper.\n2. SliceGPT reported that the WikiText2 of Llama2-7B at 30% sparsity is 8.12, which significantly differs from the one reported in this paper (23.31). The WikiText2 of Llama2-13B at 30% sparsity is also larger than the one in the original paper. The difference also occurs in the results of LLM-pruner, I'm wondering why there is such a gap. Is there some difference in the experimental setup? If so why not use the same setup as the original paper?\n3. Comparison with optimization-based baselines is limited. Only 50% sparsity is compared, more sparsity levels should be compared. Nuteprune reported that the WikiText2 of Llama2-7B at 50% sparsity is 12.29, which significantly differs from the one reported in this paper. Does this difference come from the lack of fine-tuning?\n4. Since the other methods do not appear to match the results presented in their own original papers. The quality of these results in a scientific context is low, because it is difficult to draw any conclusions from them. I hope the author can provide some explanation for the difference. \n5. I want to know why the authors only compare the results without fine-tuning? If the models pruned by gradient reweighting can't beat those SOTA methods after fine-tuning, the practical significance of gradient reweighting would seem to be less. I recommend that the authors compare the performance after fine-tuning.\n6. I wish the authors could provide statistical information rather than single experiments, which would help to increase the credibility of the results", "was_revised": true, "final_justification": "The authors have addressed my concerns on the higher sparse level, different results in the original paper, comparison with the results with fine-tuning, and statistical information. I think those improvements increase the value of the paper, and I have changed the rating from 2 to 4.", "rebuttal": "## W1: Results beyond 50% sparsity.\n\nWe provide experimental results under 55% and 60% prune rates, as shown in the Table R1. \n\n**Table R1. Performances with higher prune rates on  Llama2-7B.**\n|Prune rate|Method|WikiText2 PPL↓|Ptb PPL↓|\n|-|-|:-:|:-:|\n|55%|LLM-Pruner|434.51|531.02|\n|55%|SliceGPT|97.69|311.01|\n|55%|Wanda-sp|218.80|164.81|\n|55%|FLAP|363.82|413.05|\n|55%|Ours|**35.34**|**59.83**|\n|60%|LLM-Pruner|761.97|816.07|\n|60%|SliceGPT|139.64|475.73|\n|60%|Wanda-sp|298.84|247.60|\n|60%|FLAP|464.39|532.67|\n|60%|Ours|**53.84**|**72.95**|\n\n## W2: The experimental performance gap in SliceGPT and LLM-Pruner.\n\nThe reason of the performance difference is we use different   experimental setup for fair comparison. \n\nBefore presenting our detailed setting, we'd like to clarify the following three factors:\n \n1. **SliceGPT uses WikiText2 as both the calibration and evaluation dataset, which makes it difficult to fully evaluate the generalization ability of the pruned model.**  Such experimental setup is also not wildely adopted in the literature.\n2. Regarding the performance difference on LLM-Pruner, there may be a misunderstanding: the original paper used the LLaMA-1 with a evaluation context length of 128, whereas our experiments are based on the LLaMA-2 and LLaMA-3 with a context length of 2048.\n3. **Pruning methods such as SliceGPT and LLM-Pruner can have notable performance variation over different calibration datastes and input sequence lengths.** As shown in Table R2 below, it achieves an average score of 51.50 when using WikiText2 as the validation set, compared to 57.93 when using Alpaca. Figure 7 of SliceGPT further illustrates that when varying the length of the calibration dataset, the WikiText2 PPL of OPT-6.7B fluctuates within a range of 12.0 to nearly 20.0.\n\n**Table R2. The results extracted from Tables 7–10 in the original SliceGPT paper demonstrate that the performance of LLaMA2-7B at a 30% prune rate varies significantly depending on the calibration data used. \"-\" means the result is not available in the original paper.**\n|Calibration Data|WikiText2 PPL↓|PIQA↑|WinoGrande↑|HellaSwag↑|ARC-e↑|ARC-c↑|Avg. Score↑|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|WikiText2|8.12|63.55|61.33|49.62|51.77|31.23|51.50|\n|WikiText2&fine-tuning|-|67.41|63.22|55.65|50.76|34.13|54.23|\n|Alpaca|-|72.25|59.83|55.86|63.93|37.80|57.93|\n|Alpaca&fine-tuning|-|74.59|61.64|63.06|66.54|40.87|61.34|\n\n**Our experimental settings.** To ensure fair comparison across all baselines in our experiments, we standardize the calibration dataset to **C4** and use a sequence length of **128**. For evaluation, we use **WikiText2** and **Ptb** as test sets with a length of **2048**, and we **do not apply fine-tuning**. \n\n**Results under the same setting with SliceGPT.** Furthermore, we evaluate our method under the same experimental setting as SliceGPT, where **WikiText2 is used as both the training and evaluation set**. Under this experimental setup, our method also surpasses SliceGPT on multiple models. The results are shown in the following table.\n\n**Table R3. Performance of our method, with training and evaluation conducted on Wikitext2(↓).**\n|Prune rate|Method|Llama2-7B|Llama2-13B|OPT-2.7B|OPT-6.7B|\n|-|-|:-:|:-:|:-:|:-:|\n|20%|SliceGPT|6.64|5.81|13.73|11.48|\n|20%|Ours|**6.32**|**5.63**|**12.69**|**10.94**|\n|30%|SliceGPT|8.12|6.99|15.83|12.51|\n|30%|Ours|**7.75**|**6.61**|**15.02**|**12.07**|\n\n## W3-1: Comparison with optimization-based baselines is limited.\n\nThanks for your suggestion. We additionally include experiments based on the optimization-based pruning method Compresso, NutePrune, LoRAShear [r1], MaskPrune [r2] with different models and prune rates. More comparison will be included in the revised version if accepted. For a fair comparison under equal training time, we extend the training epochs of our method to fully exploit its effienciy. We additionally report the time and memory overhead incurred by each method during the pruning process. The results are summarized below.\n\n**Table R4. Comparison with various optimization-based methods on Llama2-7B. To ensure a consistent training time budget, we extend the training epoch accordingly. Meanwhile, we also record the memory overhead for reference. The fine-tuned version is trained on the Alpaca dataset.**\n|Prune rate|Method|WikiText2 PPL↓|Ptb PPL↓|WikiText2 PPL(&Fine-tune)↓|Ptb PPL(&Fine-tune)↓|Time cost|Memory(MiB)|\n|-|-|:-:|:-:|:-:|:-:|:-:|:-:|\n|0%|-|5.47|8.39|-|-|-|-|\n|20%|Ours(1 Epoch)|9.67|13.94|8.35|13.18|4.8h|10224.34|\n|20%|Ours(2 Epochs)|9.44|13.70|8.02|12.90|9.6h|10224.34|\n|20%|Ours(3 Epochs)|**9.35**|**13.48**|**7.94**|**12.57**|14.5h|10224.34|\n|20%|Compresso|11.47|21.25|9.92|18.39|16.8h|33298.74|\n|20%|NutePrune|9.88|16.64|8.74|14.90|13.4h|37460.78|\n|50%|Ours(1 Epoch)|26.83|44.23|14.39|30.07|3.6h|6390.21|\n|50%|Ours(2 Epochs)|23.46|41.79|12.79|25.42|7.2h|6390.21|\n|50%|Ours(3 Epochs)|**21.52**|**38.69**|**12.55**|**25.20**|10.7h|6390.21|\n|50%|Compresso|49.28|78.90|40.83|67.08|16.8h|33298.74|\n|50%|NutePrune|24.89|40.83|12.94|26.04|13.4h|37460.78|\n\n**Table R5. Comparison with various optimization-based methods on Llama-7B. Zero Shot task includes BoolQ, PIQA, HellaSwag, Winogrande, ARC-e, ARC-c and OBQA.**\n|Prune rate|Method|Zero Shot Avg. Score↑|Zero Shot Avg. Score(&Fine-tune)↑|Time cost|Memory(MiB)|\n|-|-|:-:|:-:|:-:|:-:|\n|0%|-|63.25|-|-|-|\n|20%|Ours(1 Epoch)|60.37|61.71|4.8h|10173.46|\n|20%|Ours(2 Epochs)|**61.34**|**61.97**|9.6h|10173.46|\n|20%|Compresso|-|60.75|16.8h|33298.74|\n|20%|NutePrune|59.46|61.46|13.4h|37460.78|\n|20%|LoRAShear|-|60.63|-|-|\n|20%|MaskPrune|61.17|-|-|-|\n|50%|Ours(1 Epoch)|51.73|53.26|3.6h|6239.06|\n|50%|Ours(2 Epochs)|**51.94**|**53.68**|7.2h|6239.06|\n|50%|Compresso|-|46.87|16.8h|33298.74|\n|50%|NutePrune|50.35|52.13|13.4h|37460.78|\n|50%|LoRAShear|-|50.39|-|-|\n|50%|MaskPrune|49.92|-|-|-|\n\n## W3-2: Performance differences on NutePrune. Does this difference come from the lack of fine-tuning?\n\nYes. To ensure a fair comparison, we did not perform additional fine-tuning when reproducing the experiments of NutePrune, which may lead to discrepancies from the results reported in the original paper. \n\n## W4: Explanation for the difference.\n\nFor the differences in certain experimental results, please refer to the responses in W2 and W3-2.\n\n## W5: Why the authors only compare the results without fine-tuning?\n\nThanks for your suggestions. We provide the fine-tuning results in Table R4 and Table R5. For fair comparison, we adopt the same setup as LLM-Pruner: LoRA fine-tuning is performed on the pruned model using the Alpaca dataset for 2 epochs, with a rank of 8 and a learning rate of 1e-4. \n\n## W6: Statistical information.\n\nThank you for your valuable suggestion. Actually, we observed that the experimental results exhibited low variance in the initial training stage, so we fixed a random seed for all subsequent experiments. To illustrate this, we conducted statistical experiments using five different random seeds on both Llama2-7B and Llama3-8B. The results, shown in the table below, indicate that the standard deviation is relatively small compared to the performance gains. Complete results with statistical information will be included in the revised version if accepted.\n\n**Table R6. Statistical information over 5 random seeds.**\n|Model|WikiText2 PPL↓|Ptb PPL↓|PIQA↑|HellaSwag↑|ARC-e↑|ARC-c↑|OBQA↑|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Llama2-7B|10.58(±0.64)|15.86(±0.73)|73.59(±1.24)|63.09(±0.93)|58.51(±1.53)|32.85(±0.87)|38.62(±0.91)|\n|Llama3-8B|17.88(±0.79)|29.26(±0.96)|72.18(±1.66)|58.68(±0.85)|56.50(±0.56)|30.87(±0.75)|33.68(±0.64)|\n\n## Reference\n\n[r1] Chen T, Ding T, Yadav B, et al. Lorashear: Efficient large language model structured pruning and knowledge recovery. ArXiv 2023.\n\n[r2] Qin J, Tan J, Zhang K, et al. MaskPrune: Mask-based LLM Pruning for Layer-wise Uniform Structures. ArXiv 2025."}, {"review_id": "M6suKvW5UD", "reviewer": "Reviewer_Hb8r", "summary": "This paper proposes an optimization-based pruning method for large language models (LLMs). The key contribution is a reweighting technique that allows the reuse of outdated sub-models to estimate gradients for structural parameters, reducing the need to frequently instantiate new sub-networks. The method also incorporates variance reduction techniques to stabilize training. Experimental results demonstrate memory savings, inference speedup, and competitive performance compared to state-of-the-art pruning methods. The paper also extends the method to a distributed sparse training setting.", "questions": "Please refer to Strengths and Weaknesses", "limitations": "The authors have not discussed the limitations and potential negative societal impact of their work. It is recommanded to be included in the final revision.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The method allows using an outdated sub-model for multiple pruning steps, which significantly reduces computational cost.\n\n2. The method achieves competitive or superior performance compared to several baselines in terms of perplexity, memory usage, and inference speed.\n\n3. The paper is well organized and easy to follow.\n\n**Weaknesses**\n\n1. The introduction states that \"Consequently, LLMs pose a unique challenge for pruning\", but this claim lacks sufficient justification. A clearer explanation should be provided.\n\n2. The theoretical parts in Section 4 are sometimes dense and hard to follow without a strong background in knowledge.\n\n3. Comparisons with more recent optimization-based methods are limited. Most baselines are metric-based.\n\n4. typos: caption of Figure 1 (\"vanila\" -> \"vanilla\")", "was_revised": false, "final_justification": "", "rebuttal": "## W1: Explanation for \"LLMs pose a unique challenge for pruning\".\n\nAs we mentioned in the main text, unlike conventional DNNs, LLMs face unique challenges in pruning, primarily due to the asymmetry in resources (data volume and memory) between training and pruning. Conventional DNNs, e.g., ResNet for image classification, can typically be pruned under resource conditions similar to those used during training, such as comparable datasets and memory (e.g., ImageNet dataset and 4 GeForce RTX 4090), which is always affordable for the downstream practioners. In contrast, training LLMs often relies on extremely large-scale computational resources and massive data. For example, LLaMA3-405B was trained with up to 16K H100 GPUs and approximately 15T of training tokens [r1], which far exceeds what is accessible during the pruning phase. This characteristic of LLMs therefore presents unique challenges for pruning methods, distinct from those encountered in conventional DNNs.\n\n## W2: Theoretical parts in Section 4 are dense and hard to follow.\n\nWe will move some of the theoretical parts to the appendix in the revision.\n\n## W3: Comparison with optimization-based baselines is limited.\n\nThanks for your suggestion. We additionally include experiments based on the optimization-based pruning method Compresso, NutePrune, LoRAShear [r2], MaskPrune [r3] with different models and prune rates. More comparison will be included in the revised version if accepted. For a fair comparison under equal training time, we extend the training epochs of our method to fully exploit its effienciy. We additionally report the time and memory overhead incurred by each method during the pruning process. The results are summarized below.\n\n**Table R1. Comparison with various optimization-based methods on Llama2-7B. To ensure a consistent training time budget, we extend the training epoch accordingly. Meanwhile, we also record the memory overhead for reference. The fine-tuned version is trained on the Alpaca dataset.**\n|Prune rate|Method|WikiText2 PPL↓|Ptb PPL↓|WikiText2 PPL(&Fine-tune)↓|Ptb PPL(&Fine-tune)↓|Time cost|Memory(MiB)|\n|-|-|:-:|:-:|:-:|:-:|:-:|:-:|\n|0%|-|5.47|8.39|-|-|-|-|\n|20%|Ours(1 Epoch)|9.67|13.94|8.35|13.18|4.8h|10224.34|\n|20%|Ours(2 Epochs)|9.44|13.70|8.02|12.90|9.6h|10224.34|\n|20%|Ours(3 Epochs)|**9.35**|**13.48**|**7.94**|**12.57**|14.5h|10224.34|\n|20%|Compresso|11.47|21.25|9.92|18.39|16.8h|33298.74|\n|20%|NutePrune|9.88|16.64|8.74|14.90|13.4h|37460.78|\n|50%|Ours(1 Epoch)|26.83|44.23|14.39|30.07|3.6h|6390.21|\n|50%|Ours(2 Epochs)|23.46|41.79|12.79|25.42|7.2h|6390.21|\n|50%|Ours(3 Epochs)|**21.52**|**38.69**|**12.55**|**25.20**|10.7h|6390.21|\n|50%|Compresso|49.28|78.90|40.83|67.08|16.8h|33298.74|\n|50%|NutePrune|24.89|40.83|12.94|26.04|13.4h|37460.78|\n\n**Table R2. Comparison with various optimization-based methods on Llama-7B. Zero Shot task includes BoolQ, PIQA, HellaSwag, Winogrande, ARC-e, ARC-c and OBQA.**\n|Prune rate|Method|Zero Shot Avg. Score↑|Zero Shot Avg. Score(&Fine-tune)↑|Time cost|Memory(MiB)|\n|-|-|:-:|:-:|:-:|:-:|\n|0%|-|63.25|-|-|-|\n|20%|Ours(1 Epoch)|60.37|61.71|4.8h|10173.46|\n|20%|Ours(2 Epochs)|**61.34**|**61.97**|9.6h|10173.46|\n|20%|Compresso|-|60.75|16.8h|33298.74|\n|20%|NutePrune|59.46|61.46|13.4h|37460.78|\n|20%|LoRAShear|-|60.63|-|-|\n|20%|MaskPrune|61.17|-|-|-|\n|50%|Ours(1 Epoch)|51.73|53.26|3.6h|6239.06|\n|50%|Ours(2 Epochs)|**51.94**|**53.68**|7.2h|6239.06|\n|50%|Compresso|-|46.87|16.8h|33298.74|\n|50%|NutePrune|50.35|52.13|13.4h|37460.78|\n|50%|LoRAShear|-|50.39|-|-|\n|50%|MaskPrune|49.92|-|-|-|\n\n## W4: Typos.\n\nWe will correct these typos in the revision.\n\n## L5: Missing discussion of limitations and broader impacts.\n\nThanks for the reminder. In fact, we discussed the limitations and broader impacts in Appendices F and G. We will move them into main text in the revised version if accepted.\n\n## Reference\n\n[r1] Dubey A, Jauhri A, Pandey A, et al. The llama 3 herd of models. arXiv e-prints, 2024: arXiv: 2407.21783.\n\n[r2] Chen T, Ding T, Yadav B, et al. Lorashear: Efficient large language model structured pruning and knowledge recovery. ArXiv 2023.\n\n[r3] Qin J, Tan J, Zhang K, et al. MaskPrune: Mask-based LLM Pruning for Layer-wise Uniform Structures. ArXiv 2025."}]

Author rebuttal:

Meta-review: This manuscript proposes a method to explore pruned LLMs based on having access to a dense fully trained LLM.  Their approach is novel in that they are able to better make use of the potential pruned intermediate LLMs with less variance in the reweight function, the loss, and the gradient.  These are believed to allow for a more stable training process when searching for the final pruned LLM.  The reviewers initial manuscript was viewed by the reviewers as needing additional experiments, especially needing studies with fine-tuning training.  The authors engaged well with these criticism and the result is that all four reviewers gave scores of 4 with the only variation being on the certainty of the reviewers in their scores; that said the feedback from the less certain reviewers was consistent with that of the more certain reviewers.

The manuscript is a reasonable contribution.  Using variance reduction for these less stable quantities is a natural and good idea.  It has been implemented and shown to give superior performance.  Reviewers have the requisite expertise to review the results and the only remaining question is how well the methods scale to larger problems which is beyond the scope of what the authors would be able to do.  The ideas of the manuscript could be conveyed well through a poster; there isn't need for an oral or spotlight presentation.
Decision: Accept (poster)