You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning
Authors: Ayan Sengupta
Siddhant Chaudhary
Tanmoy Chakraborty
Venue: ICLR 2025 Poster (2025)
Keywords: Model Compression
Large Language Models
Structured Pruning
Primary area: foundation or frontier models, including LLMs

Abstract: The ever-increasing size of large language models (LLMs) presents significant challenges for deployment due to their heavy computational and memory requirements. Current model pruning techniques attempt to alleviate these issues by relying heavily on external calibration datasets to determine which parameters to prune or compress, thus limiting their flexibility and scalability across different compression ratios. Moreover, these methods often cause severe performance degradation, particularly in downstream tasks, when subjected to higher compression rates. In this paper, we propose *PruneNet*, a novel model compression method that addresses these limitations by reformulating model pruning as a policy learning process. PruneNet decouples the pruning process from the model architecture, eliminating the need for calibration datasets. It learns a stochastic pruning policy to assess parameter importance solely based on intrinsic model properties while preserving the spectral structure to minimize information loss. PruneNet can compress the LLaMA-2-7B model in just 15 minutes, achieving over 80\% retention of its zero-shot performance with a 30\% compression ratio, outperforming existing methods that retain only 75\% performance. Furthermore, on complex multitask language understanding tasks, PruneNet demonstrates its robustness by preserving up to 80\% performance of the original model, proving itself a superior alternative to conventional structured compression techniques.

Published as a conference paper at ICLR 2025
You Only Prune Once: DESIGNING CALIBRATION-FREE MODEL COMPRESSION WITH POLICY LEARNING
Ayan Sengupta", Siddhant Chaudhary" & Tanmoy Chakraborty
Department of Electrical Engineering Indian Institute of Technology Delhi, India ayan. sengupta@ee. ietiiee..iiii... aitd.ac. in, urssiddegmail.cmm com, tanchak@iitd. in
ABSTRACT
The ever-increasing size of large language models (LLMs) presents significant challenges for deployment due their heavy computational and memory require- ments. Current model pruning techniques attempt to alleviate these issues by relying heavily on external calibration datasets to determine which parameters to prune compress, thus limiting their flexibility and scalability across differ- ent compression ratios. Moreover, these methods often cause severe performance degradation, particularly in downstream tasks, when subjected to higher compres- sion rates. In this paper, we propose PruneNet, a novel model compression method that addresses these limitations by reformulating model pruning as a pol- icy learning process. P PruneNet decouples the pruning process from the model architecture, climinating the need for calibration datasets. learns stochastic pruning policy to assess parameter importance solely based on intrinsic model properties while preserving the spectral structure to minimize information loss. P PruneNet can compress the LLaMA-2-77 model in just 15 minutes, achieving over 80% retention outperforming existing methods that retain only 75% performance. Furthermore, on complex multitask language understanding tasks, PruneNet demonstrates its robustness by preserving up to 80% performance of the original model, proving itself a superior alternative to conventional structured compression techniques.
zero-shot performance with a 30% compression ratio,
INTRODUCTION
Pre-trained Large Language models (LLMs) (Touvron al., 2023; Zhang al., 2022; Scao al., 2023; OpenAI et al., 2024) have demonstrated exceptional abilities in natural language understand- ing and generation, creating numerous avenues of applications across a wide range of domains such as healthcare, education and finance. These deep neural models are predominantly based on the Transformer architecture (Vaswani al., 2023) and often contain several billions of parameters. Models like GPT-4, and larger variants of (> 65 parameters) LLaMA-2, and OPT can occupy as much as 350GB of memory FP16 format, making them impractical for deploying to resource- constrained environments such as mobile or edge devices. The need for dedicated GPUs, even for inference, restricts their applicability in real-time, low-resource scenarios, creating a barrier for broader adoption in industries where speed and efficiency crucial.
Model compression is a class of techniques that are widely used to reduce the computational over- head of LLMs. Two major subclasses of model compression are quantization and model praming- While quantization is primarily used to reduce the precision points of the saved model weights, thereby reducing the memory footprint of the model, model pruning aims at pruming dropping different neural components to make models smaller and faster during inference. Notable methods like post-training model pruning methods (Ashkboos al., 2024; Yang et al., 2024) usually work by either removing entire components like neurons, attention heads or layers based on specific rules (structured pruning) removing individual parameters resulting in an irregular sparse structure
These authors contributed equally to this work.
'The source code of PruneNet is made public at https://githuu: wtt.:/w....mmm.......... PruneNet.

---

Published as a conference paper at ICLR 2025
Method Sparsity
Effective Sparsity
FL.PPs
Avg. Zero-shot Acc
Dense 0%
0.0%
135e+13
69.0
SlicēGPT
9.4%
1225+f37[.100
38.2
P PruneNet 20%
12.0%
1.18e+13 (1.15x)
61.7
PliceCPT
13.3%
1.14e7 ((.TTK)
35.5
PruneNet 25%
16.0%
1.13e+13 (1.20x)
58.6
SlicēGIT 30%
21.4%
1.07e4
31.5
PruneNet
19.0 %
1.09e+13 (1.24x)
55.5
Table 1: A summary of the experimental results. We highlight the effective sparsity ratio, along with total FLOPs (Floating Point Operations) and average zero-shot accuracy for different spar- sity ratios with the LLaMA-2-7B model (see Table 2 for more details). Effective sparsity can be calculated as the ratio of the total number of parameters in the compressed model and that of the uncompressed model. SliceGPT achieves less effective sparsity with high FLOPs at a lower sparsity ratio (< 25%) as learns the pruned parameters with a Icarnable network and retains them within the LLM. On the other hand, PruneNet decouples the compression process from the LLM,, thereby achieving higher effective sparsity with lower FLOPs. cost-compression performance drop from the dense (uncompressed or compression ratio 0%) LLaMA-2-78 model is also significantly higher for SliceGPT (average drop of 13.9%, as compared to a drop of 10.7% of PruneNet).
(unstructured pruning). pruningg) For instance, SliceGPT (Ashkboos et al., 2024) uses orthogonal transforma- tions on pre-trained model parameters to slice off contiguous block of rows and columns, reducing the overall size of the models. However, these methods face notable limitations, particularly their reliance on calibration data and hardware-speccfie optimizations, constraining their flexibility and scalability. Most notably. the heavy dependence on calibration data limits the applicability of exist- ing pruning methods. Over-reliance on calibration datasets makes the model compression methods vulnerable to data quality issues and reduces their trustworthiness for performance-sensitii appli- cations like finance or healthcare. Furthermore, the need to repeatedly calibrate models for different compression ratios hinders their efficiency, making it difficult to adapt to varying computational resource constraints.
Our Contribution. To address these challenges, introduce PruneNet, novel structured prun- ing technique that eliminates the need for calibration data and enhances flexibility across varying compression ratios. Unlike traditional methods, PruneNet reformulates model pruming a policy learning process. leveraging a reusable policy learner that decouples parameter importance assess- ment from the model architecture itself. This allows for rapid pruning without repeated retrain- ing, enabling the same pruning strategy to be applied at multiple compression ratios. The method also minimizes the information loss after model compression by preserving the uncompressed and compressed models' spectrum structure. Our empirical analysis demonstrates that PruneNet t can compress the LLaMA-2-7B model in just 15 minutes 50% faster than existing methods like SliceGPT while retaining up to 95% of the original model's performance across various tasks. Re- markably, even without recovery fine-tuning, PruneNet preserves 84% of the zero-shot accuracy on commonsense reasoning tasks (c.f. Table 1), outperforming SliceGPT by significant margin. Furthermore, even smaller LLMS such as OPT-125M exhibit post-compression performance stabil- ity, suggesting that P PruneNet is not only efficient but also robust across models of varying sizes.
RELATED WORK
Model pruning is a widely-used technique to reduce the number of parameters in a model, enhancing both speed and efficiency. can be broadly categorized into two classes unstructured and struc- tured pruning. Unstructured pruning removes individual weights, as seen in SparseGPT (Frantar & Alistarh, 2023), which leverages Hessian matrix inversion to identify and eliminate less critical weights. However, unstructured pruning often requires hardware-specifi optimizations and may not always result in significant computational gains (Yang et al., 2024: Wang et al., 2024b). In contrast, structured pruning removes entire channels components. making it more compatible with various hardware setups. For example, --rrrrr (Ma et al., 2023b) evaluates weight group importance and uses LoRA fine-tuning recover lost accuracy. While structured pruning is more hardware- frienddy, it can lead to greater accuracy loss at higher compression ratios. Methods like Layer Collapse (Yang al., 2024) take layer-wise approach, merging parameters of adjacent lay-
2

---

Published as a conference paper at ICLR 2025
to achieve up to 50% compression without extensive retraining. Recent advancements in post- training compression methods, such as SliceGPT (Ashkboos et al., 2024) and SVD-LLM (Wang et al.. 2024b), aim to maintain performance while reducing model size. SliceGPT is structured pruning method that compresses LLMSs by slicing off entire rows and columns of weight matrices, using orthogonal transformations to reduce the embedding dimensions. SVD-LLM, on the other hand, applies singular value decomposition with layer-wise updates, ensuring minimal accuracy loss even under high compression, outperforming previous methods like ASVD (Yuan et al., 2023) and FWSVD (Hsu et al., 2022).
Majority of these structured pruning methods rely heavily on external calibration datasets, making them sensitive to data quality. For instance, the LLaMA-2-7B model, compressed with SliceGPT, drops zero-shot performance by 6% when calibrated with the WikiText2 dataset instead of the Al- paca dataset. In contrast, our proposed P PruneNet method does not rely any calibration data and captures the parameter importance as learnable policy. The policy learner model, being indepen- dent of the original model, offers great flexibility and can be reused to compress different models at different compression ratios.
BACKGROUND
3.1 TRANSFORMER ARCHITECTURE
The Transformer architecture (Vaswani et al., 2023) has been widely adopted to achieve state-of-the art results in wide range natural language tasks, including natural language understanding and generative language modeling. Each layer of Transformer model includes two core components: the multi-headed self-attention layer followed by a ferd-forward layer (FFN), which are separated by LayerNorm (Ba et al., 2016) blocks and residual connections (He et al., 2015). In most Trans- former architectures, the FFN module is a two-layered multi-layer perceptron (MLP), which can be mathematically represented as follows:
FFN(X) o (XWT + bup) WT
up
down
boown
(I)
where a is a non- activation function, X E Bx N xdhata is a batch of inputs with being the batch size, N denotes the sequence length, dhidten is the hidden dimension of the architecture, W up
Rdmemnater dhasknn is the up-projection matrix (i.e., dlintrnediate > daidden), Wdown E R.dsssee Xcrnematiiee is the down-projection matrix, and hup E dhsemasiiee and downn E dusaa are the corresponding biases. The first and second layers of MLP are also referred to FFN, and FFN2, respectively; we point out that all these notations just refer to the two MLP layers, and for our purposes, can be used interchangeably. While most LLMs such as OPT (Zhang al., 2022). Falcon (Almazrouei al., 2023) and Phi (Gunasekar et al., 2023) have two weight matrices in the FFN layers, models like LLaMA (Touvron et al., 2023) have three matrices in their FFN layers: the up-projection matrix (Wap). mp, the down-projection matrix (W and gated-projection matrix W
gate-
3.2 INTRINSIC MODEL COMPRESSION
Several existing structured pruning methods (e.g., SliceGPT) prune matrices by intrinsically com- puting parameter importance. Specifically, for every attention and FFN layer, SliceGPT adds an additional matrix of dimension dhidden X (1 r) dpieten and duidden X (1 - -r) dantemediate- for a compression ratio of It turns out that for small compression ratios, such techniques end up adding more parameters to the model than they remove, leading to an overall increase in the total parameter count of the model. We formalize this fact in the following lemma.
Lemma 3.1 (Limitations of Intrinsic Model Compression). Given an LLM with hidden dimension driddoo and intermediate FFN dimension disermodiates any intrinsic model compression method that introduces new parameters within the model will reduce model size only if the compression ratio
r destiiat doatnaatu Eduisie + 33dsttrnioon
For Phi-2, dustden 2, 560 and dintermediate 10, 240. Therefore, for any compression ratio < 29.4%. the compressed model can have negative effective compression, defying the purpose of model compression.
See Appendix C.1 for the proof.
3

---

Published as a conference paper at ICLR 2025
3.3 EFFECTS OF SLICING ON THE SPECTRUM OF MATRIX
In a slicing operation (popularly used in methods like SliceGPT), drop
(a) Layer
(b) Layer
Cummulative Probability
1.0
entire row(s) or column(s) from the
0.8
weight matrices of model. We high-
0.6
light two key results with slicing on the spectrum of a weight matrix.
0.4
Compression
Compression 0%
Compression 25%
Compression 25%
0.2
Compression 50%
Compression 50%
For any matrix W E R=xm, the matrix WW (also known the Gram Matrix of W) is a positive- semidefinite matrix, thus admitting only real, non-negative cigenval- ues. Morcover, the non-zero singu- lar values of W precisely square roots the non-zero eigenvalues of W W. Throughout the paper, will refer to the set of eigenvalues of W W the spectrum of W.
Compression 75%
Compression 75%
0.00
Singular Values
Singular Values
Empirical cumulative distribution function (ECDF) of singular values of FFN, layer weights at layers (a) and 32 (b) of the LLaMA-2-7BB model. The right skew- ness of the distribution at higher sparsity ratio highlights the diminishing property of singular values at reduced dimen- sions.
Figure 1:
Theorem 3.2 (Poincaré Separation Theorem). Let A E guxa be symmettrcc matrix, and let B E
Raxr be a anti-orthogonal matrix, i.e. B I, where I I n and I, xx is the identity matrix. Let \ I )2 I .... S Xn be the eigenvalues of A and μn I P2 S ... S pr be the eigenvalues of B AB. Then
A S U4 S An-r+t
for SiST. r. In particular,
min ISSn
A min 1515T
max 1SJSP
Py
max X 18i5n
(2)
i.e, the range of eigenvalues of B" AB is subset of the range of eigenvalues of A. Corollary 3.3 (Slicing shrinks the range of the spectrum). Let W E Rxdd be a weight matrix, and let W E R.mxd he matrix obtained by slicing off rows of W so that m S n. Then, the range of singular values of W is subset of the range of singular values of W.
Corollary 3.3 suggests that if we remove any set rows from any weight matrix in an LLM, the distribution of singular values of the matrix becomes more right-skewed. To understand this phe- nomenon, we illustrate the spectrum of FFN, from layer I and 32 of the LLaMA-2-7B model in Figure 1. This observation encourages to learn compression model to minimize the distribu- tional shift after compression to retain the performance the uncompressed model.
4 METHODOLOGY
In this section, we elaborate on PruneNet, our proposed policy-driven gruming-objectio network. Corollary 3.3 lays the foundation of our method, which aims at minimizing the spectral distributional shift between post and pre-compression LLMs, thereby achieving the purpose of model pruning and maintaining the model's 's superiority. Figure 2 illustrates the overall PruneNet framework.
Although PruneNet is flexible and can be used to compress any neural module, we focus only on the FFN layers of LLMs as these layers contribute to the most parameters in LLM with the highest density structure. For instance, LLaMA-2-7B, FFN layers collectively contribute to 64% of the parameters, whereas the self-attention layers contribute to 32% of the parameters. Moreover, due to the multi-headed structure, the density (number of neurons directly connected to a parameter) in self-attention is only 27%, whereas is 100% in FFN layers. Additionally, only FFN layers responsible for non-linear activation. Mirzadeh et al. (2023) highlighted that non-linear activations comprise significant computational cost within an LLM. Therefore, we make logical assumption only to compress the FFN layers of an LLM to achieve the highest effective sparsity.
'See Theorem 11.10 of Magnus (2019) for the proof.
*See Appendix C.2 for the proof.
4

---

Published as a conference paper at ICLR 2025
U(D 13
Reparametrisation
Sampler
Multinomial
Computing importance воoree
Indioes koep
Wing
dowa
W.
Pollcy Learner
W W.
Penalty
Wap
[S,[S.)
Cleas
Distance
Clone
Unaliced MLP Layer
Sliced, MLP Layer
Original Spectrum Distribution
Up-atodd Spectrum Distribution
Figure 2: A schematic diagram of PruneNet. A policy learner is used to learn the row indices of an FFN, weight matrix to prune. The policy learner is trained with policy gradient with penalty calculated with the Kolmogorov-Smirno (KS) distance between the uncompressed and compressed matrix singular value distributions. FFN2 is pruned by columns with the same indices learned by the policy learner. Biases, Dup and baowns are excluded from the diagram, but are also pruned.
4.1 IDENTIFYING PARAMETERS TO COMPRESS
The foundational block of PruneNet is the policy learner model. The policy learner is simple MLP, which is used to determine the row indices of an FFN, weight matrix, Wap to prune (see Section 3.1 for notation). We refer this process as selective pruning it does not assume the continuity of columns or rows in the selection. The policy learner is used to prune the FFN, weight matrices of all MLP layers in the model.
Consider an FFN, weight matrix Wap E 1nxdd of the model. To prune rows of this matrix, the policy learner aims compute vector Winp E R of importance scores for each row; all impor- tance scores are in (0, 1). The policy learner model has two trainable parameters: an intermediate projection matrix W imter xx and final projection matrix Wpoj lxn, The computation of Winp Wimp can be described as follows:
W W inter
W
W imp = o W proj W')
(3)
Note that W' Raxn can capture the interaction between all rows the matrix to assess interdependence among them. We do not assume any bias while computing importance scores; hence, we do not use any bias parameter the policy learner model.
Using the importance scores W Wimp. determine the rows of W., prune. Given compression ratio r E [0, 1]. we sample a set S containing (1 - r) n indices from the set 21,2..... n) from multinomial distribution with event importance" W imp- calculated as follows:
W Imp alog log (1 c) log (Wimp) log ))
where above, e _ is a random vector, each of whose entries are sampled from the uniform distribution, ('0 1). Equation (4) uses the well-known reparametrization trick (Kingma, 2013; Maddison et al., 2016) to ensure that the sampling process is differentiable w.r. the importance scores W Imp-
To get the pruned matrix, we prune all indices not in S:
W W compressed
W [S
'Event probabilities can computed by normalizing event importances.
'The formulation in Equation 4 can be used to simulate Bernoulli random variable with mean given by W , for [n]. See Appendix C.3 for the pronf.
5

---

Published as a conference paper at ICLR 2025
While compressing LLM, only perform the sampling for the FFN1 weight matrix. The same set of sampled indices reused for the FFN₂ matrix by using [ to ensure that the dropped dimensions match between the two matrices. Similarly, the bias terms, Dup and Dowwns pruned accordingly.
4.2 PENALTY COMPUTATION FOR THE POLICY LEARNER
We caleulate the singular values of the original weight matrix Wap as (A,) with A1 2 A2 2..
A Similarly, for the compressed matrix Wup we obtain the singular values { where = (1 r) n. To calculate the distributional difference between the spectrums, we calculate the
Kolmogorov-Smina (KS) distance, defined as:
D = sup (I)) (2))
(6)
where Fi,. and F2,n are the empirical distributions of {Mr)i and [4:رe respectively. The KS distance computes the supremum of the distance between the two distributions at any point in their support. The policy learner uses the distance measure as the penalty to minimize.
4.3 POLICY OPTIMIZATION
The final step in PruneNet is to optimize the policy learner to minimize the expected penalty defined in Equation 6. Given an L-layered LLM, we consider each FFN, matrix weight W; as states and the selected indices S; as the action obtained by the policy learner. Therefore, we can construct a trajectory W1, S1. W2,S2,.... L............ SL. Figure suggests that typically, an LLM, the later layers have higher singular values. Therefore, the penalty could be higher for the later layers. Moreover, the later layers of LLMs contain the most semantic information (Yang et al., 2024), needing them to preserve the most. Thus, for each LLM layer I, we calculate the discounted future penalty as:
G G== k=
* Di+K+1
For given state W, (FFN, weight at layer 1), ability using Equation 4, sample the action St (set of rows to select) and obtain the compressed matrix W-oopprssedd Using Equations 6 and 7 we calculate the penalty. Finally, using the REIN- FORCE (Williams, 1992) algorithm, we compute the gradient and learn the policy model.
the policy learner to calculate action prob-
EXPERIMENTAL RESULTS
5.1 EXPERIMENTAL SETUP
We use PruneNet for compressing different LLMs, including LLaMA-2-7B (Touvron al., 2023), LLaMA-1-7B, Phi-2 (Gunasekar et al., 2023) and OPT (Zhang et al., 2022) (125M, 2.7B and 6.7B variants). For the zero-shot performance evaluation,
five commonsense reasoning tasks - PIQA (Bisk et al., 2020), WinoGrande (Sakaguchi et al., 2021), HellaSwag (Zellers et al., 2019), ARC-e and ARC-c (Clark al.. 2018), using the LM Evaluation Harness suite (Gao et al., 2024) and the MMLU benchmark (Hendrycks et al., 2020) 7 For the policy learner model, we consider the discount factor, ? 0.99 and use the AdamW (Loshchilov. 2017) optimizer with a learning rate of 5e and a maximum of 20 episodes. The total number of trainable parameters in the policy learner is 45M (~ 0.67%) for the LLaMA-2-7B model. We evaluate the compression performance of PruneNet primarily against the state-of-the-art baseline SliceGPT (Ashkboos et al., 2024). Additionally, we also consider other competitive structured pruning methods - LaCo (Yang et al., 2024), SVD-LLM (Wang et al., 2024b), ASVD (Yuan et al., 2023), LLM Pruner (Ma al., 2023b) and ShonGPT (Men et al., 2024) for our evaluation.* All the experiments were performed on a single Nvidia A A100-40GB GPU.
7 the task descriptions are provided in Section D of the Appendix. Due to unavailability of LaCo and SPT baseline source codes, we were able to compare these methods only on selected tasks.

---

Published as a conference paper at ICLR 2025
Model
Comp, Ratio
Method
PIQA
WinoGrande
HellSSaag
ARC-
ARC-e
Avg.
0%
Deuse
69.06 19.11
(1000)
(438(1008)
46.25 (100%)
(5000
LLaMA-2-18
20%
SlisGPPT
69.42 (88%)
65.11 (94%)
(78%)
59.76 (80%)
37,34
(844)
PruneNlet
75.30
65.51 (95%)
(87%)
63.80 (85%)
37.29 (81%)
61.67 (89%
SlicGPT
(848)
63.38 (92%)
(7
58.46 (78%)
34.56 (759)
55.48
25%
PruneNet
(910)
62.43 (908)
(2.33
60.14
36.18 (78%)
58,6308599
"SisGPTT
(803))
61.33 (84))
75.62 (65%)
$1.77 (00%)
31223 (619)
(1530
30%
PruneNet
71.11 (90%)
61.09 (88%)
58.30 (77%)
53.20 (71%)
33.53 (72%)
55.45 (80%)
0%
Demse
19.11(1003)
15.77 (D03))
73 (1OP5)
(802)
14.18
12.24
(100%)
201
(913)
67.80 (896)
37.76
35.32 T633
(5000
Phu-2
25%
"S&OOPT"
PruneNet
(8996
74.37 (94%)
70.80 (93%)
65.33 186%)
65.53 (89%) %)
32.R077151
74.71 (95%)
33,7 (195))
47,53 (88%)
31.68 [586T
66.59 (92%)
PruneNet
74.37 (94%)
68.98 (91%)
62.18 (84%)
70.54 (90%)
44.45 (82%)
64.10 (89%)
30%
SlicoGPT
65.94 (839))
63.14 (83%)
47.56 (64%)
53.03 (68%)
30.29 (565))
51,99 (72%)
PruneNet
72.80 (92%)
67.48 (89%)
56,80 (77%)
67.55 (86%)
40.61 (75%)
61.05 (84%)
Table 2: Comparison of SliceGPT and PruneNet on generative tasks without recovery fine-t We report the performance recovery from the uncompressed model in parenthesis. Tables 21 and 22 in Appendix E.3 highlight the zero-shot performance for the other LLMs compressed with PruneNet and SliceGPT.
Model
Comp. Ratio
RFT
PIQA
WineGrande
HellaSwag
ARC-
ARC-c
Avg.
L.aMA 2-7B
20%
75.30
65.51
66.43
62.25
36,26
61.15
74.76
66.22
68.37
63.93
38.40
62.34
25%
72.09
62.43
62.33
60.14
36,18
58.63
74.37
63.77
65.71
60.65
35,75
60.05
71.II
61.00
5836
53220
32,93
5533
30%
72.20
62.90
63.21
51.37
33.70
57.08
20%
74.37
70.80
63.53
74.71
47.53
66.59
76.17
71.51
63.28
70.50
46.42
65.58
Phi-2
74.37
68.8
62718
70351
34.45
64.70
25%
73.39
69.22
61.53
71.38
45.73
64.25
72.80
67.48
56.80
67.53
30.61
61.05
30%
71.49
63.93
58.18
61.78
37.80
58.34
Table 3: Performance of LLaMA-2-7B and Phi-2 under different compression ratios without
(marked as x) and with (marked as v/ recovery fine-tuning (RFT) on the WikiText2 dataset. Results without RFT are the same as the ones reported in Table 2.
5.2 RESULTS
Table 2 reports" the zero-shot performance of LLaMA-2-7B and Phi-2 models after being com- pressed with PruneNet and SliceGPT (the best baseline) at different compression ratios. The average performance of the uncompressed LLaMA model is 69%, out of which 85% is preserved after compression with P PruneNet with maximum drop by 20%. On the other hand, at 30% com- pression ratio, SliceGPT drops the performance by 25%. Similarly, for the Phi-2 model, the maxi- mum performance drop is 16% with P PruneNet, which is significantly higher (28%) for SliceGPT. We report the zero-shot performance of other compressed LLMs. i.c., OPT-125M, OPT-2.7B and OPT-6.7B in Table 21 of Appendix E.3. A one-sided Kolmogorov-Smirnov (KS) test suggests (p- value 0.05) that the performance drop exhibited by PruneNet is significantly lower than that of SliceGPT across all the LLMs. Interestingly, for different LLMs, the performance remains stable across different compression ratios (standard deviation of 2.03), whereas, for SliceGPT, the per- formance drops significantly with higher compression rate (standard deviation of 2.54). Even on complex multitask language understanding tasks (c.f. Table 23 in Appendix E.3), LLMs compressed with P PruneNet exhibit high quality stability at different compression ratios. Interestingly, tasks like formal logic and global facts, compressed LLaMA models with PruneNet even outperform the uncompressed ones.
Recovery fine-tuning (RFT) is a common trick to regain performance drop after compression. To understand the importance of RFT the effectiveness of PruneNet, we report the zero-shot performance of compressed LLaMA and Phi-2 models after fine-tuning on the WikiText2 (Merity al., 2016) dataset in Table 3. For fine-tuning, we use LoRA adapters (Hu et al., 2022) with rank 8. Interestingly, RFT has only marginal impact of 1.5% on the compressed LLaMA model, which highlights the robustness of our method. Remarkably, the importance of RFT remains the same for a higher compression rate. On the other hand, with Phi-2, the performance drops after RFT in several cases. This result validates the robustness of PruneNet but also an appreciation of the pre-
Further ablation studies and detailed experimental analysis can be found in section E of the appendix.
7

---

Published as a conference paper at ICLR 2025
Comp. Ratio
Method
PIQA
Hellaswag
Average
Dense
77.91
71.26
74.58
"EGneNet
7C.1
58.36
64.70
LLM Pruner (Ma et al., 2023a)
71.22
56.46
63.84
30%
SliceGPT (Ashkboos et al., 2024);
66.21
50.27
58.24
LaCo (Yang al., 2024yf
69.80
55.69
62.74
ShortGPT (Men et al., 2024)
66.43
53.02
59.72
Table 4: Comparison of different model compression methods generative tasks with the LLaMA-
2-7B model without recovery fine-tuning. Results marked with 'f" are taken from Men et al. (2024).
Comp. Ratio
Method
PIQA
HellaSwag
WinoGrande
ARC-e
ARC-c
Average
PruneNet
62
36
61.0
SliceGPT (Ashkboos al., 2024)
61
62
35
59.0
20%
LLM--runer (Ma et al. 2025b)
76
67
61
61
37
60.4
ASVD (Yuan et al., 2023)
68
41
64
53
27
50.6
SVD-LLM (Wang et al., 2024a)
63
29
52.4
"Pttneeet"
70
57
39
32
31
33.8
ShiceGPT (Ashkhoos et al., 2024)
69
56
61
49
32
53.3
30%
LLM-Prrune (Ma el ul., 2023b)
69
$7
59
48
30
52.6
ASVD et al., 2023)
53
44.6
SVD-LLLM (Wang al., 2024a)
65
37
59
48
26
47.0
Table Comparison different model compression techniques with the LLaMA-1-7B model with- out recovery fine-tuning.
training objective of the small language models such as Phi-2 that uses specialized curated datasets for pre-training.
Table 4 reports the performance of the LLaMA-2-7B model compressed with different structured pruning methods. At the same compression ratio, I.L.MA-2-77 compressed with PruneNet achieves 1.6% higher performance than the other methods like L.LM-Pruner, SliceGPT and LaCo. Our comparison against other contemporary structured pruning methods like SVD-LLM and ASVD on a different model - LLLMA-1-7B (c.f. Table 5) also highlights similar trend. ASVD and SVD- LLM work on a similar spectral preservation objective while compressing LLMs. However, it is worth noting that these methods preserve the information loss the output rather than the com- pressed model, therefore needing access to calibration datasets. Even after calibration, these meth- ods significantly underperform P PruneNet (average difference of 8.5%). Typically, SliceGPT and LLM-Prune: perform better than these SVD-based model compression techniques, with SliceGPT being most robust baseline. However, it worth noting that, our method surpasses all these baselines with consistent margin across different compression ratios.
Can we reuse the policy learner? We evaluate if an LLM trained with the policy learner at a higher compression ratio can perform well in zero-shot commonsense tasks lower compression ratios. Precisely, we assess the flexibility of P runeNet at different compression ratios. Table 6a highlights the zero-shot performance of the compressed LLaMA model, reusing the policy learned at a compression ratio of 40%. It turns out that PruneNet is robust in terms of the learned policy,
Comp, Rañio
PIQA
HellaSwag
WinoGrande
ARC-
ARC-
Average
10%
76.88 (77.09)
71.06 (71.06)
63.67 (66.14)
68. (68. (68.14)
40.87 (40.87)
64.52 (64.66)
20%
75.30 (75.30)
66. (66.43)
65.50 (65.51)
60.30 (63.80)
36.30 (37.29)
60.70 (61.67)
30%
69.21 (71.11)
57.92 (58.30)
61.09 (61.09)
51.89 (53.20)
32.94 (33.53)
54.61 (55.45)
(a) High compression
low compression policy transfer
Comp. Ratio
PIQA
HellaSwag
WinoGrande
ARC-e
ARC-c
Average
20%
(75.30)
6664(66.43)
63.9 (63.51)
63.8 (03.80)
37.29(37.29)
(61.67) 54.89 (55.45)
30%
70.18 (71.1))
57.81 (58.30)
59.83 (61.09)
53.11 (53.20)
33.53 (33.53)
40%
65.13 (66.32)
48.59 (48 22)
55.25 (55.80)
41.67 (48.61)
26.96 (28.41)
47.52 (49.48)
(b) Low compression > high compression policy transfer
Table 6: Zero-shot performance of the LLaMA-2-7B model compressed with the policy learned at 40% (a) and 10% (b) compression ratios. The numbers reported in parentheses are the results obtained with the model compressed with the same ratio as the policy learner.
8

---

Published as a conference paper at ICLR 2025
Model
PIQA
HellaSwag
WinoGrande
ARC-e
ARC-c
Average
LLaMA-2-7B
73.18 (75.30)
65.35 (66.43)
(65.06)))3
58.38 (63.80)
35.07 (37.29)
(6.52
Phi-2 74.65 (74.37)
65.88 (65.53)
69.53 (70.80)
(74.71)
47.87 (47.53)
66.35 (66.59)
Table 7: Zero-shot performance of the LLaMA-2-7B and Phi-2 models when different layers com- pressed at different compression ratio with maximum being 40%. For a fair comparison, we compare these results with models compressed at fixed compression ratio of 20%.
Comp. Ratio
RIT Dataset
PIQA
WinoCiandd
HellaSwag
ARC-e
ARC4
Average
Alpaca
72.73
61.52
42.15
20%
PTB
73.07
63.38
65.05
64.56
36.69
60.55
WikiTexu2
Alpiaca
75.79
62.35
63.45
60.94
39.16
60.74
25%
PTB
62.79
WikiText2
74.37
66.46
65.71
60.82
36.60
60.79
Alpaca
72.14
55.43
37.03
30%
PTB
71.11
62.67
59.60
58.12
35.07
57.31
WikiTexr2
73.01
63.46
63.21
60.14
35.92
59.15
Table Results obtained by LLaMA-2-7B after recovery finetuning on Alpaca, PTB and WikiTex12 datasets with different compression ratios.
where the performance only drops marginally (maximum drop < 1% with p-value < 0.05) with transferable policy. Similar observations are found (c.f. Table 6b) when the policy learned at lower compression ratio is reused to perform compression at higher compression ratio. Remarkably, the average drop with the different policy is only 0.96%, with maximum margin observed only at higher compression ratios. Interestingly, even under transferred policies, the results are still 3% better than SliceGPT. Therefore, if we cannot retrain the policy learner at different compression ratios under certain cireumstances, reusing another policy trained for compressing the same model various compression ratios is still feasible. This transferable property of P PruneNet can have tremendous practical implications that most existing contemporary compression methods fail display.
Can we use different compression ratios in different layers? The policy learner of P Net enables us to use different compression ratios at different L.LM layers. To empirically validate its effectiveness, we evaluate I.LMMA2-7B and Phi-2 models compressed at varying compression ratios between (0%- - 40%] (average compression ratio 20%) at different layers. Table 7 highlights that varying compression ratios have mixed effect on the two models. Phi-2 is highly stable, and the average accuracy drops only by 0.24% with varying compression. However, for the LLaMA-7B model, the drop is more prominent, 2.15%.
Does the RFT dataset matter? Table 3 high- light that RFT has very little requirement with Pr PruneNet. However, it would be interest- ing understand whether the RFT datasets have any particular impact on the fine-tuned compressed model. Table 8 reports the zero- shot performance of the LLaMA-2-7B model after being fine-tuned on different datasets. It turns out that the standard deviation of perfor- mance after RFT on different datasets is only 1.3 and go down to even 0.52 for tain compression ratios. Although it is as- sumed that instruction-cuning datasets like Al- paca are more suitable for acquiring knowledge on instruction-tuning commonsense tasks, in reality, all the datasets have a similar impact on the compressed model. This observation sug- gests that the robustness of PruneNet preserves the key information within the compressed model, therefore minimizing the need for fine-tuning on specialized datasets.
Performance Recovery (%)
95-
90
85
80-
Compression 20% Compression
75
Comprassion 30%
100
1000
2000
4000
RFT Training Data Size
Figure We highlight the performance recov- ery (w.r.. the dense uncompressed model) for the LLaMA-2-7B model under different compres- sion ratios after recovery fine-tuning with varying training data sizes.
Does RFT data size matter? The previous experiments suggest that RFT datasets do not signif- icantly affect the compressed dataset's zero-shot performance. The pertinent research question is whether the RFT training data size immediately affects the compressed model's downstream perfor-

---

Published as a conference paper at ICLR 2025
mance. To answer this, perform RFT with varying training data sizes. Figure 3 highlights that the training data size also minimally impacts the compressed model's zero-shot downstream per- formance. Even with 100 training samples, compressed LLaMA-2-7B with < 25% compression ratio can preserve up to 85% of the original performance. However, 100 samples may be insufficient higher compression ratio of 30%. At higher compression rates, the compressed model quickly regains performance with more than 1000 training samples.
5.3 ANALYSIS
Does sparsity help recovery fine-
12.5
Compression 0%
Compression 0%
tuning? Figure 4, shows the impact of model compression on recovery fine-tuning any fine-tuning post- compression. Interestingly, compres- sion can make LLMs more efficient during later fine-tuning. A highly compressed LLM usually has high training and validation loss (high per- plexity) initially but can regain per- formance quickly even in just 50 training iterations. With a higher compression ratio, LLMs can elimi- nate more redundant and low-importance features, making future training and inference efficient
Training Loss
10.0
Compression 20%
Compression
Compression 20%
Compression 25%
10.
7.3-
Compression 30%
Compression 30%
7.5
valliattio
5.0
5.0
2.5
2.5
0
200
400
600
0
200
400
600
Training Step
Validation Step
Figure 4: LLaMA-2-78 2-BB RFT training and validation loss curves for different compression ratios. We observe that sparser models achieve faster convergence and generaliza- tion.
without significant performance drops.
Model
Params 6.78
Comp. Modules
Runtime (In soc)
Computational analysis. Table 9a exhibits the runtime complexity of PruneNet. compress LLaMA-7B, PruneNet requires 15 minutes, whereas, for the same compres- sion ratio, SliceGPT requires 29 minutes. At 30% compression rate, a compressed LLaMA- 7B model exhibits 73% better efficiency (to- kens generated per second) during inference. For similar compression rate, SliceGPT exhibits very marginal (7%) inference effi- ciency over the dense uncompressed model. Interestingly, for the smaller Phi-2 model, SliceGPT has poorer throughput even than the dense model, whereas, PruneNet manages to achieve 46% higher throughput than the un- compressed model.
L-MM--2-B
Up. Down & Catc Proy
916 1 15
Phi-2
FCI,
(a) Runtime PruneNet
Model
Method
Throughput (Token/sec)
Danse
11.96
LLaMA-2-78
SlicGGPT
FruneNet
20.74
Dcinse
20220
Phi-2
SliceGPT
18.48
PruneNat
29.50
(b) Inference throughput
Table 9: (a) Time taken in seconds for com- pressing LLaMA-2-7B and Phi-2 models with PruneNet at 30% compression ratio. (b) Infer- ence throughput (tokens generated per second) of different compression methods at 30% compres- sion ratio. The experiments were conducted on single Nvidia A100-40GB GPU.
CONCLUSION
This paper introduced novel structured pruning method, PruneNet for compressing LLMs. Un- like the existing compression methods, PruneNet learns general policy for a given LLM and, therefore, can be reused to compress the model at any given compression rate without needing the compression model to rerun. Moreover, by utilizing the intrinsic properties of the LLM, P PruneNet can get rid of the over-reliance on external calibration datasets and improve the stability of the compressed model on downstream tasks. While our current work explores the effectiveness of calibration-ffes learnable compression policy and applicable only for sparsifying weight matrices, evaluating its effectiveness on activation sparsity would be interesting. Even higher inference effi- ciency can be achieved with activation sparsity, where a similar formulation can be utilized to prune the non-linear activations with low information gain. Moreover, in the current setting, PruneNet is orthogonal to the concepts of quantization and, therefore, can easily be integrated with quantization methods to enhance the computational efficiency of LLMs further. In cooperation with quantization and mixed- precision training, P PruneNet can achieve higher inference yield, with marginal perfor- mance drop, paving the way for better utilization of large language models in real-life applications.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
T. Chakraborty acknowledges the support of the IBM-IITD AI Horizons network and Rajiv Khemani Young Faculty Chair Professorship in Artificial Intelligence.
REFERENCES
Ebtesam Almazrouci, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Co- jocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series of open language models, 2023. URL httpt:://aaaattttttt .wrg///i.ooro.ooooooo.o....oo abs /2111211111 16867.
Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns, 2024. URL petps::/arxxr wwt.oogg//sso.. abs 2401 15024.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL https: /ttppss///axvv..//e org/ a /1000 06450.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint
0.00000055557.. 2018.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in 33301007744
one-shot, 2023. URL https: http/axxvvvv..///////e///..a org/
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter; Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL httts::/zenodo.orttt 12608602.
http://eeoodo..oo org records/
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allic Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, 2023. URL https:/ /arxiv.org/ abs/2300 1444
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition, 2015. URL https: www.xii.oooraaaroo /111.1111111 033.5.1115........
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.088 03300,, 2020.
Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. arXiv preprini 22Xiv:2207.001 12, 2022.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL https:
/ http://pwweeeeeeeeeeeeeee net forum?
iddnnee/eeeyyf,
Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
Hector J. Levesque, Emnsst Davis, and Leora Morgenstern. The winograd schema challenge. In Pro- ceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, 12' pp. 552-561. AAAI Press, 2012. ISBN 9781577355601.

---

Published as a conference paper at ICLR 2025
I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.051000 2017.
Xinyin Ma. Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances in neural information processing
processinnnpoocsssinngg systems, 36:21702-21720, 2023a.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. In Advances in Neural Information Processing Systems, 2023b.
Chris J Maddison, Andriy Mnih, and Yes Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.007/ 2016.
J.R Magnus. Matrix Differential Calculus with Applications in Statistics and Econometrics. Wiley Series in Probability and Statistics, 2019.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):113-330, 1993. URL ht https: hatps////aaattooooy....o... org/ 290-2004.
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect, 2024. URL htwps::////m.. / org/ abs /2403.4000000 03853.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXi::1609.000 07843, 2016.
Iman Mirzadeh, Keivan Alizadeh, Sachin Mehta, Carlo C Del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, and Mehrdad Farajtabar. Relu strikes back: Exploiting activation sparsity in large language models, 2023. URL https: / httpxxivvv.o/ org/ abs 2oloo./221.. 04564.
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Moham- mad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapigg Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brock- man, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Alty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gib- son, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hal- lacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billic Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Ka- mali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, tukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger. Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning. Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Rejichrr Nakano, Ra- jeev Nayak, Arvind Neelakantan, Richard Ngo. Hyeonwoo Noh, Long Ouyang, Cullen O' Keeff, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl,
12

---

Published as a conference paper at ICLR 2025
Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross. Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Sel- sam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher. Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vi- jayvergiya, Chelsea Voss. Carroll Wainwright, Justin Jay Wang, Alvin Wang. Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng. Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Work- man, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL picts:://ariii org/ hatp//a////// 220.//033.00777
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexan- der M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Baw- den, Stas Bekman, Angelina SiMMillan-Majjo Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christo- pher Klamm, Colin Leong, Daniel van Strien, David Ifcoluwa Adelani, Dragomir Radev, Ed- uardo González Ponferrada, Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli, Hady Elsahar, Hamza Benyamina, Hieu Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios, Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jörg Frohberg, Joseph Tobing, Joydeep Bhat- tacharjee, Khalid Almubarak, Kimbo Chen, Kyle Lo. Leandro Von Werra, Leon Weber, Long Phan, Loubna Ben allal, Ludovic Tanguy, Manan Dey, Manuel Romero Muñoz, Maraim Ma- soud, María Grandury, Mario Šaško, Max Huang, Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Mohammad A. Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulaqilla Khamis, Olivier Nguyen, Omar Espejel, Ona de Gibert, Paulo Vil- legas, Peter Henderson, Pierre Colombo, Priscilla Amuok, Quentin Lhoest, Rheza Harliman, Rishi Bommasani, Roberto Luis López, Rui Ribeiro, Salomey Osei, Sampo Pyysalo, Sebas- tian Nagel. Shamik Bose, Shamsuddeen Hassan Muhammad, Shanya Sharma, Shayne Longpre, Somaieh Nikpoor, Stanislav Silberberg, Suhas Pai, Sydney Zink, Tiago Timponi Torrent, Timo Schick, Tristan Thrush, Valentin Danchev, Vassilina Nikoulina, Veronika Laippala, Violette Lep- ercq. Vrinda Prabhu, Zaid Alyafeai, Zeerak Talat, Arun Raja, Benjamin Heinzerling, Chenglei Si, Davut Emre Taşar, Elizabeth Salesky. Sabrina J. Mielke, Wilson Y. Lee, Abheesht Sharma, An- drea Santilli, Antoine Chaffin, Arnaud Stiegler, Debajyoti Datta, Eliza Szezechla, Gunjan Chh- ablani, Han Wang, Harshit Pandey, Hendrik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Saiful Bari, Maged S. Al-shaibani, Mattoo Manica, Nihal Nayak, Ryan Tee- han, Samuel Albanie, Sheng Shen, Srulik Ben-David, Stephen H. Bach, Taewoon Kim, Tali Bers, Thibault Fevry, Trishala Neeraj, Urmish Thakker, Vikas Raunak, Xiangru Tang. Zheng-Xin Yong, Zhiqing Sun, Shaked Brody, Yallow Uri, Hadar Tojarich, Adam Roberts, Hyung Won Chung, Jaesung Tae. Jason Phang, Ofir Press, Conglong Li. Deepak Narayanan, Hatim Bourfoune, Jared Casper, Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi, Myr- iam Peyrounette, Nicolas Patry, Nouamane Tazi, Omar Sanseviero, Patrick von Platen, Pierre Cornette, Pierre François Lavallée, Rémi Lacroix, Samyam Rajbhandari, Sanchit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil, Tim Dettmers, Ahmed Baruwa, Amanpreet Singh, Anas- tasia Cheveleva, Anne-Laure Ligozat, Arjun Subramonian, Aurélie Névéol, Charles Lovering, Dan Garrette, Deepak Tunuguntla, Ehud Reiter, Ekaterina Taktasheva, Ekaterina Voloshina, Eli Bogdanov, Genta Indra Winata, Hailey Schoelkopf, Jan- -hrriiistoph Kalo, Jekaterina Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuat,
13

---

Published as a conference paper at ICLR 2025
Miruna Clinciu, Najoung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, Shachar Mirkin, Shani Pais, Tatiana Shav- rina, Thomas Scialom, Tian Yun. Tomasz Limisiewicz, Verena Rieser, Vitaly Protasov, Vladislav Mikhailov, Yada Pruksachatkun, Yonatan Belinkov, Zachary Bamberger, Zdeněk Kasner, Alice Rueda, Amanda Pestana, Amir Feizpour, Ammar Khan, Amy Faranak, Ana Santos, Anthony Hevia, Antigona Unldreaj, Arash Aghagol, Arezoo Abdollahi, Aycha Tammour, Azadeh Haji- Hosseini, Bahareh Behroozi, Benjamin Ajibade, Bharat Saxena, Carlos Muñoz Ferrandis, Daniel McDuff, Danish Contractor, David Lansky, Davis David, Douwe Kiela, Duong A. Nguyen, Ed- ward Tan, Emi Baylor, Ezinwanne Ozoani, Fatima Mirza, Frankline Ononiwu, Habib Rezane- jad, Hessie Jones, Indrani Bhattacharya, Irene Solaiman, Irina Sedenko, Isar Nejadgholi, Jesse Passmore, Josh Seltzer, Julio Bonis Sanz, Livia Dutra, Mairon Samagaio, Maraim Elbadri, Mar- got Mieskes, Marissa Gerchick, Martha Akinlolu, Michael McKenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok, Nafis Abrar, Nazneen Rajani, Nour Elkott, Nour Fahmy,, Olanre- waju Samuel, Ran An, Rasmus Kromann, Ryan Hao, Samira Alizadeh, Sarmad Shubber, Silas Wang, Sourav Roy, Sylvain Viguier, Thanh Le, Tobi Oyebade, Trieu Le, Yoyo Yang, Zach Nguyen, Abhinav Ramesh Kashyap. Alfredo Palasciano, Alison Callahan, Anima Shukla, An- tonio Miranda-Escaladte Ayush Singh, Benjamin Beilharz, Bo Wang, Caio Brito, Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel León Periñián Daniel Molano, Dian Yu, Enrique Manjavacas, Fabio Barth, Florian Fubrimann, Gabriel Altay, Giyaseddin Bayrak, Gully Burns, Helena U. Vrabec. Imane Bello, Ishani Dash, Jihyun Kang, John Giorgi, Jonas Golde, Jose David Posada, Karthik Rangasai Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Mare Pàmies, Maria A Castillo, Marianna Nezhurina, Mario Sänger, Matthias Samwald, Michael Cullan, Michael Weinberg, Michiel De Wolf, Mina Mihaljeic, Minna Liu, Moritz Freidank, Myungsun Kang. Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Muellner, Pascale Fung, Patrick Haller, Ramya Chandrasekhar, Renata Eisenberg. Robert Martin, Rodrigo Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok S Deshmukh, Shubhanshu Mishra, Sid Kiblawi, Simon Ott, Since Sang-aroonsiri, Srishti Kumar, Stefan Schweter, Sushil Bharati, Tanmay Laud, Théo Gi- gant, Tomoya Kainuma, Wojeiech Kusa, Yanis Labrak, Yash Shailesh Bajaj. Yash Venkatraman, Yifan Xu, Yingxin Xu, Yu Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf. Bloom: A 176b 166b-parameter open-access multilingual language model, 2023. URL https:ttp////cccm /ttp:////axivv.e/ org/ abt./////112 21t...221..... 05100.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-follloinng llama model, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL http://///// /ttp//awwwrraaaaaaae/ee/aeeeea/a wwtgg/////////////
13971.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https: //axxiv. org/abs abs//bb//// / 1706. 03762.
Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-lim: Truncation- singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07337 2024a.
Xin Wang, Yu Zheng. Zhongwei Wan, and Mi Zhang. Svd-llm: Truncation-awarc singular value decomposition for large language model compression, 2024b. URL https://axxii...oo abs/2403. 07378.
Ronald J. Williams. Simple statistical gradient-following algorithms for cenprecciioiittrinnnnccccmm learning. Mach. Learn, 8:229 256, 1992. doi: 10.1007/BF000996666 URL https: //oi..
Yifei Yang, Zouying Cao, and Hai Zhao. Laco: Large language model pruning via layer collapse, 2024. URL https://arkee wwg/abs/////o..oooo.ooo.o..... /24022422200 11187.
Zhihang Yuan, Yuzhang Shang, Yue Song. Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation-awan aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05881, 2023.
14

---

Published as a conference paper at ICLR 2025
Cummulative Probabilts)
1.0
(a) Layer
(b) Layer 32
Cummulative Probatallty
1.0
Layer
(b) Layer
0.8
0.6
0.6
0.4
Compression
0.4
Compression 06
Compressiinn 25%
Campression 55%.
Compression 25%
Campression 25%
Comgression 50%
Campression 50%
02:
Compression 50%
Compression 50%
Compression
0.00
10
0.0
Singular Values
Singular Values
Singular Values
Singular Values
(a) Phi-2
(b) OPT-2.7B
Figure 5: Spectrum of FFNI layer for Phi-2 (a) and OPT-2.7B (b) models at different compression ratios.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- 2019.
chine really finish your sentence? arXiv preprint
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettle- moyer. Opt: Open pre-trained transformer language models, 2022. URL https://arxits org/abs/2205 01068.
A BACKGROUND
A. EFFECTS OF SLICING ON THE SPECTRUM OF A MATRIX
Weight matrices of LLMs have been known to exhibit variation in the distribution of singular val- ues across layers (Yuan et al., 2023). To further verify our observations from Figure 1, we conduct similar study the FFNI matrices of the OPT and Phi-2 models. We plot the cumulative dis- tribution of the singular values of these matrices and observe consistent behaviour across all three model architectures. Figure 5a and 5b highlights the spectrum of the FFNI module different lay- ers for Phi-2 and OPT-2.7B models, respectively. This further strengthens our assumption of using discounted rewards in our policy learning approach.
A.2 SPARSITY VS. EFFECTIVE SPARSITY
In structured model compression, we often choose predetermined compression ratio to deter- mine how much to compress pre-trained model. We refer to this predetermined compression ratio as sparsity ratio. Based on the sparsity ratio, structured pruning methods prune the model components subjected to compression. Usually, the self-attention and FFN blocks of Transformer models are pruned. However, it is worth understanding that model compression does not af- fect all the components equally. For instance, the token embedding and final output layers of a pre-trained L.M are usually not pruned during compression. Therefore, to understand the ef- fective impact of model compression method the pre-trained model calculate 1
whole, can #parameters in uncompressed model referred as effective sparsity. It is easy to notice that effective sparsity & sparsity ratio. Lemma 3.1 highlights that for certain model compression meth- ods and sparsity ratios, effective sparsity can even be negative, where the number of parameters in the compressed model is higher than the uncompressed model.
# parametees in compressed model
B METHODOLOGY
We highlight the compression process with PruneNet in Algorithm 1.
15

---

Published as a conference paper at ICLR 2025
Algorithm Policy- Model Compression Framework (P (PruneNet)
Require: LLM with L layers, FFN1 weight matrices {Wi) compression ratio policy learner
parameters Witter: Wpejj: discount factor y
Ensure:
Compressed LLM with pruned FFN layers
1: Initialize policy learner parameters
2: for each training step do
3: for each layer / = ... L do
4:
Compute importance scores:
5: W < W₁ WT inter
Interaction of all rows
6:
Wimp
imp < --o proj
W")
Apply activation function
7:
Sample row indices:
8:
(0, 1)
Sample random vector
9:
Wimp + (log(c) log(1 e) + log( W Wimp) - -log(1 Wimp)) b Reparametrization
trick
10:
Normalize Wimp to obtain probabilities
11:
S; < Sample Multinomial W imp: (1 - -r) m))
Prune FFN;: W compressed
Wi[S, :]
13:
Prune FFN:: W coompeessed + WLFFNa Si]
1,FEN,
14:
Adjust biases bup. Udown accordingly end for
16:
Compute penalty:
for each layer 1 1.2...... do
18:
Di + sup, (x) (x)) with Fi,. and F2,a being the ECDF of singular values of compeessed and Wi, respectively.
D Kolmogorov-Smimoo distance Discounted future penalty
20:
end for
Optimize policy learner:
22:
Compute gradient using REINFORCE:
VE x V log P(S)WW) Gi
23:
Update W Vitter: Wpro) proj
24:
end for
16

---

Published as a conference paper at ICLR 2025
C PROOFS OF THE THEORETICAL RESULTS
C.I PROOF OF LEMMA 3.1
Proof. Suppose LLM has dhidten the hidden size and dintemediate the intermediate FFN size. Suppose the decoder vocabulary V. Therefore, the total number of parameters introduced in the decoder imput embedding and output heads [V] dhddenn Each self-attention query, value and key projection matrices has daidden chidden parameters and the attention output projection has dhidden Unddeen parameters. Each FFN layer has total 2chnudenn Ummermediate parameters (here we assume that the LLM has only two FFNs in the MLP layer, however similar formulation is applicable for models like LLaMA-2 that uses three FFNs). Therefore, the total number of parameters in the uncompressed model with L layers is
P Puncompressed
2 dnidden + (4d (4diideen + Eddiddeannneereeeiii L..
For the compressed model with compression ratio r, the total number of parameters without the parameter learning component
Pcmmpressed V |dhidden + (4pd + 2neemdite
Where p --r r. Here, we assume that compression will take place only on one dimension of Q,K, K, V, FFN, and FFN2, The parameter learning component has pdlideen parameters for each attention module and (thiitten parameters for each FFN layer. Therefore, the total number of parameters in the compressed model is
Poompressed 2 [dniatten + (5pdhaddon + S L.
Therefore, to ensure Poompressed
< P Puncompressed we must ensure 5pd + 3pddnidden Chntermedinee 4d2 + dh
4d + 2thnidden inntermediates
i.e., p 5ch +-3djades dencuectiee
4dmatos +2dmsr 5d +30meemtii
Therefore,
1 4dhasto - 2dgernotatt
duattn + -mmrmnatn
Schatee Momarmatttt
5dhase -3dmmamsiiee
C.2 PROOF COROLLARY 3.3
Ruxx and (W") (W')' Rmmxm are both symmetric matrices. It is easy to see that (W')(W")" is obtained from WWT by removing rows and columns of WWT corresponding to the row indices which have been sliced off: we'll make this formal below.
Proof. Note that WWT I'
Let A = WWT and A' (W')"")) Let S' be the set of row indices not sliced (we assume that the set arranged in sorted order). In particular, we have that m. Now, for S, let p(i) be the index of i in S. In particular, as iterates through elements in S in order, p(i) iterates through the integers in the set [m).
Next, consider the matrix B E R given by
Bij {
if S and j = p(i)
otherwise
We claim that B semi-orthogonal, i.e B²B Im. where In E mm identity matrix. But this is straightforward; note that the mapping i + p(i) is one-to-one mapping. In particular, this implics that the columns of are distinct one-hot vectors, thus forming a set of orthonormal vectors, and hence 8 B Im. follows easily.
Next, we claim that
A' = B AB
17

---

Published as a conference paper at ICLR 2025
The above equality can be derived follows:
JAB]s =
Σ
(Ba A)aBi
1=1
a
B ARI
Bys AuA Bij
Ap 16,)
1=1
Ap-1(0))-10 1()
A
So, from Theorem 3.2, we know that the range of eigenvalues of B AB a subset of the range of eigenvalues values of A. Finally, since singular values of W and W are just square roots of the eigenvalues of Td AB and A respectively, our claim follows.
C.3 THE REPARAMETRIZATION TRICK IN EQUATION
In this section, we motivate the formulation in Equation 4 by showing that reparametrizes the Bernouli distribution by using thresholding trick. So, let E (0, 1), and consider the distribution Bernoulli(a) (a). Consider the following experiment: we sample e (O, 1). and define the following random variable:
a
Y In
In
1- a
ln In particular, if is the sigmoid function, then
c) - a))
oYY
1+ (1-0(1-a)
60
+ (1-c)(1
Next, consider the random variable Z defined as:
if oY) 2 0.5
Z = 0 otherwise
We claim that Z ~ Bernoulli(a). This is straightforward; note that
P(z = 1] Poo (Y) )) 2 0.5)
(ca + (1 c) a1)=)))
P >
2
IP [ea 2 (1 <)(1 - a)] c Ea
P fea 2
PPc 2 1 - a]
PP|e a]
(1 1 a)
completing the proof of the claim.
18

---

Published as a conference paper at ICLR 2025
D DATASET AND TASK DESCRIPTIONS
Zero-shot evaluation datasets. PIQA (Bisk et al., 2020) is a physical common-sens, reasoning dataset focusing on everyday situations with a preference for atypical solutions. Each example of the dataset provides users with instructions on how to build, craft, bake or manipulate objects using everyday materials. The common-sense reasoning task formulated as an MCQ-based question- answering task: given a question q and two possible solutions $1 and $2, model/human must choose the most appropriate solution, of which exactly one is correct. The WinoGrande dataset (Sakaguchi et al., 2021) is a large-scale dataset of problems from the Winograd Schema Challenge (Levesque et al., 2012) consisting of pronoun resolution problems that are designed to be trivial for humans but complex for AI systems. The HellaSwag (Zellers et al., 2019) dataset consists of the issues describing common-sense natural language inference (NLI) tasks: given a sentence, a model/human must predict the most likely follow-up. The A12 Reasoning Challenge (Clark al., 2018) dataset contains natural, grade-school science question-answw answering problems (authored for human tests) requiring powerful knowledge and reasoning. The MMLU benchmark (Hendrycks et al., 2020) contains problems covering 57 subjects across STEM, the humanities and social science. measures the knowledge models acquired during pre-training by evaluating them in zero-shot and few-shot settings.
Recovery fine-tuning datasets. The WikiText (Merity et al., 2016) dataset is a commonly used benchmark for language modelling, consisting of articles from Wikipedia which satisfy the Good or Featured article criteria specified by editors on the platform. These articles, reviewed by hu- mans, considered well-written, factually correct and neutral in point of view. The dataset is available in WikiText-2 and WikiText-103; our experiments use the WikiText-2 dataset. The Penn Treebank (PTB) (Marcus al., 1993) dataset is large annotated corpus containing over 4.5 mil- lion words of American English. The section of the corpus corresponding to articles from the Wall Street Journal mainly known be used to evaluate models for sequence labelling. The Alpaca (Taori et al., 2023) dataset consists of 52000 instructions and demonstrations generated by OpenAl's text davinci 3 model, which is widely used for instruction tuning of language models. We use only up to 8000 samples from these datasets for recovery fine-tuning
E ADDITIONAL RESULTS
E.1 DETAILED COMPARISON WITH SLICEGPT WITH AND WITHOUT RFT
Table 10 contains comparison of the performance of the LLaMA-2-7B and Phi-2 pruned with SliceGPT using the Alpaca calibration dataset. While LLaMA-2-7B pruned with SliceGPT exhibits better average performance for all compression ratios, Phi-2 exhibits opposite trend, wherein PruneNet beats SliceGPT by a consistent margin of at least 2% for all compression ratios.
We report the results with LLaMA-2-7B with RFT on Wikitext2 and Alpaca datasets in Table 11 and Table 12, respectively. With RFT on Wikitext2 dataset, PruneNet achieves on average > 5% accuracy than SliceGPT. However, SliceGPT outperforms PruneNet when fine-tuned on the Alpaca dataset, with an average margin of 3%. As SliceGPT uses the same datasets for calibration and RFT, it typically has access to more instruction-tuning datasets than PruneNet allowing to do better when fine-tuned on the Alpaca dataset. However, it is worth noticing that the average standard deviation in the performance of SliceGPT after RFT is significantly high (5.5) compared to PruneNet (1.5). The low standard deviation highlights the robustness of P PruneNet when fine-tuned on different datasets for recovering the information loss during compression.
Table 13 highlights the performance of the I.LaMA-2-7B model at 50% compression ratio with PruneNet and SliceGPT. While both the methods can regain only 60% of the performance of the original uncompressed model, P PruneNet demonstrates 2% better than the SliceGPT, showcasing its effectiveness over the baseline, even at a very high compression rate.
In Table 14, we highlight the results with PruneNet and SliceGPT for different compression ratios for the LLaMA-2- 2-1BB model. The performance drops drastically for larger models like LLaMA- 13B at high compression ratio. However, the results in Table 15 highlight that after recovery fine-tuning, the compressed models regain the performance quickly and can preserve up to 84% of the original uncompressed model performance, even at high compression rate of 30%. PruneNet
19

---

Published as a conference paper at ICLR 2025
also outperforms SliceGPT with margin of 2%, showcasing similar trend as the smaller LLMs used previously.
E.2 ABLATION STUDY
We perform an ablation study understand the importance of different components of PruneNet.
Importance of Learnable Policy for Model Compression. We conduct experiments with ran- dom selection process, where the pruned parameter indices are chosen randomly. We highlight the results with random policy in Table 16. We observe an average 2% drop with a random selection method, justifying the need to learn which parameters to compress for more effective model com- pression.
Importance of Stochastic Policy. Table 17 highlights the results with LLaMA-2-7B with deter- ministic and stochastic (policy learned with P PruneNet in Equation 5) policies. In the deterministic policy, chose only the topk parameters based on the importance metric defined in Equation 3. We observe that deterministic policy often underperforms the stochastic policy with an average margin of 4%. The results highlight that parameter importance alone cannot determine which parameters to compress. Preserving the spectral structure between the compressed and uncompressed models is critical ensure minimal performance drop post-compression.
Results with Different Reward Functions. To further understand the effectiveness of PruneNet under different distance measures, we evaluate the LLaMA-2-7B model compressed using PruneNet with non-parametric Anderson-Darling measure of agreement. Table 18 high- lights the effectiveness of PruneNet with both Kolmogorov-Sminng Smirnov (highlighted as KS) and An- derson-Darlin (highlighted as AD) distance measures. Under both reward functions, we observe similar performance of PruneNet for different compression ratios with the LLaMA-2-7B model. The results further emphasize the stability of our proposed compression method under different choice metrics.
Importance of Policy State for Model Compression. Table 19 highlights the zero-shot perfor- mance of the LLaMA-2-7B model compressed with PruneNet with policy learned on different FFN matrices. In most cases, observe marginal differences in the result (< 1%) when the policy learned with FFN2 instead of FFN1. The observations emphasize that P PruneNet is invariant to the choice of parameter used for learning the policy.
Results with Pruned Self-Attention Blocks. We highlight the pruning results for LLaMA-2-7B model with compressed self-attention layers in Table 20. The performance drop with the compressed models suggests that compressing self-attention layers intrinsically is harder than compressing FFN layers. However, around 50% of the performance drop can be recovered with recovery fine-tuning.
E.3 ADDITIONAL RESULTS WITH OPT AND LLAMA-1 MODELS
Table 21 highlights the zero-shot performance of all the LLMs without RFT. OPT-125M can retain nearly 96% of its original performance even 25% compression, suggesting strong stability under compression. Larger OPT models are more sensitive to different compression ratios where the performance drops by 8% at higher compression. Table 22 highlights the zero-shot performance of the LLMs after being compressed by SliceGPT. An one-sided Golmogorov-Sminne (KS) test suggests (p-value < 0.05) that the performance drop exhibited by PruneNet significantly lower than that of SliceGPT across all the L.Mss.
Table 23 reports the zero-shot performance on multitask language understanding (MMLU) tasks. The OPT family models show a steady performance across all compression ratios. At 10% com- pression, the average MMLU accuracy score is 23.4 with the OPT-6.7B model, and maintains a comparable performance of 24.5 even at 30% compression, indicating good resilience. Similar per- formance stability can also observed with OPT- and OPT-2.7B models. However, LLaMA and Phi model performances drop significantly with higher compression ratios. However, the com- pressed models show strong performance on selected tasks like College Computer Science, College Mathematics, Formal Logic, and Global Facts, even outperforming the uncompressed models.
20

---

Published as a conference paper at ICLR 2025
Model
Comp.
Rálio
Melhod
PIQA
WinoGrande
Hellaswag
ARC-
ARC-
Avg.
Dense
75.99
69.00
LLAMA-2-78
SliceGPT
76.60
63.31
68.20
69.99
41.21
63.68
20%
PruneNet
75.53
65.51
66.43
63.80
37.29
61.71
SliceGPT
77421
64.01
60.35
66.88
38,91
60.91
25%
PruneNet
72.09
62.43
62.33
60.14
36.18
58.63
SticeGPT
72.25
59.93
55.86
61.93
37.80
57.93
30%
Prunstee
71.11
61.09
58.30
53.20
33.53
55.45
Dense
19.11
15.77
1383
78.32
54.18
12.24
20%
SliveCM
766.7
668.15
61.95
"22.18"
43.45"
64.30
Phi-2
Pruaeklet
74.37
70.80
65.53
74.71
47.53
66.59
25%
SliveGPP
75.68
64.88
58.19
70.41
43.43
62.32
PruneNit 74.37
68.98
62.18
70.54
44,45
64.10
30%
SlieeGPT 74.05
62.12
53.31
67.26
39.42
59.23
PruneNet 72,80
67.48
56.80
67.55
40.61
61.05
Table 10: LLaMA-2-7B results compressed with PruneNet without RFT. SliceGPT uses Alpaca dataset for calibration.
Comp. Ratio
Method
PIQA
WineGrande
TellaSwag
eRC
cRC
Avg.
0%
79.11
75.99
74.58
46.25
69.00
SlieeGPT
"69.86
"4.72
66.07
34.25
36.45
57.27
20%
PruneNet
74.76
66.22
69.38
65.61
39.25
63.04
SleeGGPT
69.26
64
38.05
32.36
33.75
56.20
25%
PruneNat 74.37
66.46
65.71
60.82
36,60
60.79
SlicEGPT 667A
63.222
33.65
30.76
33.15
54.23
30%
PruneNet 73.01
63.46
63.21
60.14
35.92
59.15
Table 11: LLaMA-2-7B results compressed with PruneNet and SliceGPT with recovery fine- tuning on WikiText2 dataset.
Comp. Ratio 0%
Method
PIQA
WinoGrande
HellaSwag
ARC-
ARC-c
Avg.
Dense
79.11
69.06
75.99
74.58
46.25
69.00
SliceGPT
76.55
65.59
45.05
20%
PruneNle
72.73
62.25
66.45
61. 52
42.15
61.02
SlEeGGP
75.77
63.12
66.T2
66..22
44.88
63.D
25%
Prunaliet
75.79
62.35
65.48
60.94
39.16
60.74
SUEEGTP "74.35
66.84
63.06
666.4
40..8
-6.34
30%
Pruneliet
72.14
62.75
62.38
55.43
37.03
57.95
Table 12: LLaMA-2-7B results compressed with PruneNet and SliceGPT with recovery fine- tuning on Alpaca dataset.
Method
PIQA
Winokinnnee
HellaSwag
ARC.e
ARC-C
Dense 79.11
69.06
75.99
74.58
46.25
69.00
S
"3.D9
"33.04"
"2.65
39.76
233.7
1933
Prunakiet
59.68
$2.09
35.21
34.89
25.43
41.46
Table 13: LLaMA-2-7B results compressed with PruneNet without RFT at 50% compression ratio.
Comp. Ratio 0%
Method PIQA
WinoGrande
HellaSwag
ARC-c
ARC-c
Avg.
80.47
79.39
49.23
20%
SiseeGPP
71.87
69.38
63.04
69.87
43.09
63.45
ProneNet
72.90
70.50
25%
SIiceUPP
68.55
67.48
5K.1
62.5
33.888
58.90
25%
70.89
62,43
58.67
58.63
34.04
56.93
30%
SlieeGPT
66.
65.11
52.69
56.82
35.07
55.16
30%
61.92
56.99
35.65
46.34
28.33
Table 14: I.L.MA--133B results compressed with P PruneNet and SliceGPT without RFT.
Comp. Ratio
Method
PIQA
WinoGrande
HellaSwag
ARC-c
ARC-c
Avg.
0%
Dense
80.47
72.22
79.39
77.48
49.23
71.76
20%
SliceGPT
74.10
68.51
66.94
70.54
43.77
64.77
PruneNet
70.72
65.02
25%
SlicsGPT
77.22
68.98
64.12
63.76
40.87
61.88
PruneNet 76.93
70.44
30%
SticeCPPT 69.64
66.85
39.93
39.55
18.65
58.92
Prunetiet 73.45
60.73
38.57
Table 15: LLaMA-2-13B results compressed with PruneNet and SliceGPT with RET on Wiki- Text2 dataset.
21

---

25% Published as a conference paper at ICLR 2025
Comp. Ratio
Selection
PIQA
WinoGirande
HellaSwag
ARC-e
ARC-c
Average
20%
Policy-bused
Random
72.36
75.3
63.14
65.5
61.18
66.43
60.31
63.8
36.52
37.29
61.66
58.70
Pulicy-hased
72.09
62.43
62.33
60.14
36.18
58.63
25%
Random
70.13
60.38
58.38
56.27
35.67
56.20
Policy-bused
71.13
58,30
53.20
33.53
30%
Random
73.13
59.35
55.15
49.83
31.66
53.90
Table 16: Effect learnable policy for compressing an LLaMA-2-7B model.
Comp. Policy
PIQA
WinoCirande
HellaSwag
Average
Stochastic
15.3
65.5
66.43
63.8
37.29
61.66
Deterministic
72.91
61 64
61.05
56.69
36.52
57.76
Stöchastie"
72.09
662.4 23
6233
56.18
38.63
Deterministic
69.97
59.27
59.39
56.69
33.53
55.77
Stöchustic"
71.13"
"61.09
58.30
53,20
33333
35.45
30%
Deterministic
69.64
58.25
54.45
54.97
31.23
53.71
Table 17: Effect of stochastic policy on compressed I.AMA-2-7B model.
Comp. Ratio
Reward Function
PIQA
WinoGrande
HellaSwag
ARC-
ARC-
Averuge
KS
75.3
66.43
63.8
37.29
61.66
20%
AD
73.01
63.3
65.7
60.4
37.46
59.97
RS
72.09
62.43
62.33
60.14
56..8
38.63"
25%
AD
73.88
61.17
63.98
61.62
35.84
59.30
RS
7I.
"6.099
58.30
5322
53333
35.45
30%
AD
72.13
61.88
60.18
58.00
33.62
57.16
Table 18: Zero-shot performance of LLaMA-2-775 --BBB compressed with PruneNet with different ward functions.
Comp.
Laver
PIQA
HellaSwag
ARC=
Avg.
FENT
15.30
65,50
66.43
63.30
37.29
61.66
20%
PPNT
72.04
62.43
62,33
60.14"
"36.18"
5835
25%
70.13
HENT 71II
61.09
38.30
33.20
"33.53
53.45
30%
72.20
33.70
Table 19: comparison of using FFNI vs FFN2 modules for learning policy with PruneNet.
Comp. Rulio
RFT
PIQA
WinoGrande
HellaSwag
ARCa
ARC-c
Avg.
20%
None
50.11
52.49
26.24
27.31
28.75
36.98
20%
WikiText2
71.98
56.83
53.04
55.13
33.45
$4.09
50.38
26.25
28.
25%
WikiText2
69.42
54.93
40.24
49.87
32.68
49.43
None
49.57
30%
WikrText2
62.46
52.01
47.53
38.55
28.75
45.86
Table 20: Results with LLaMA-2-7B model with pruned self-attention modules.
22

---

Published as a conference paper at ICLR 2025
Model
Comp. Ratio
PIQA
WinoGrande
HellaSwag
ARC-e
ARC-C
Average
0%
61,97
50.28
31.35
39,90
22.78
41.26
20%
58.49
52.49
30.43
37.46
21.50
40.07
OPT-125M
25%
59.41
51.78
30.19
35.31
2193
39.72
30%
58.75
45.88
29.03
34.15
20.79
37.72
74.81
61.09
60.63
54.25
31.31
56.42
20%
70.02
58.33
50.52
46.30
27.73
50.58
OPT-2.7B
25%
68.88
57.46
47.91
45,83
27.82
49.58
30%
66.21
56.99
46.61
43.56
27.05
48.08
0%
76.55
65.35
67.22
60.06
34.81
60.80
20%
72.80
61.64
59.08
49.49
31.06
54.81
OPT-6.7B
25%
72.69
60.38
$7.01
30.38
53.81
30%
70.84
58.25
55.50
46,93
29.95
52.29
0%
79.1
69.30
76.00
74,62
46.25
69.06
20%
75.30
65,51
66.43
63.80
37.29
61.67
LLaMA-2-7B
25%
72,09
62.43
62.33
60.14
36.18
58.63
30%
71.11
61.09
58.30
53,20
33.53
55.45
0%
79.16
76.01
73.84
34.01
72.25
20%
74.37
70.80
65.53
74.71
47.53
66.59
Phi-2
25%
74.37
68.98
62.18
70.54
44.45
64.10
30%
72,80
67.48
56.80
67.55
40.61
61.05
0%
78,24
667,4
63.78
67.38
38.14
6373
20%
75.51
62.12
65.40
64.65
36.52
60.82
LLaMA-1-7B
25%
60.40
59.55
32.37
$7.29
30%
69.91
59.27
$7.19
52.23
30.80
53.88
Table 21: Zero-shot performance of LLMs compressed with PruneNet without recovery fine-
tuning.
Comp.
Ratio
PIQA
WinoGrande
HellaSwag
ARC-
ARC-C
Average
20%
55.19
46.57
27.23
36.86
21.24
37.42
OPT-125M
25%
53.35
44.10
25.72
35.07
20.99
35.85
30%
$2.79
43.57
24.91
35.13
19.35
35.15
20%
66..33
5793
51.38
51387
"RR
51.37
OPT-2.7B
25%
65.29
$7.22
47.85
49.79
27.99
49.63
30%
62.35
57.22
44.18
46.72
27.05
20%
72.74
61.79
61.0
55.89
30.3
56.T
OPT-6.7B
25%
70.35
60.62
58.15
$2.78
29.52
54.28
30%
68.61
60.69
$4.56
$2.15
29.01
$3,00
20%
669.42
65.
5970
59.76
37.5
58.78
LLAMA-2-7B
25%
66.87
63.38
54.16
58.66
34.56
55.48
30%
63.55
49.62
$1.77
31.23
20
7T.87
67.00
51.16
58700
35.32
58.15
Phi-2
25%
69.21
65.35
$2.40
53.70
31.66
54.46
30%
65.94
63.14
47.56
53.03
30.29
51.99
20%
722.4""
6235
61.30
63.59
35.09
58.97
LLAMA-I-7R
25%
70.19
61.60
58.58
60.08
33.56
56.80
30%
68.57
56.26
49.14
31.95
53.33
Table 22: Zero-shot performance of LLMs compressed with SliceGPT without recovery fine-tuning.
23

---

Published as a conference paper at ICLR 2025
Model
Comp, Ratão
Abs, Alg.
Bus, Eth.
Clg. CS.
Cig. Math
Con, Ply.
Frm, Log.
Glb, Fet.
ML.
Misc.
Phil.
LLLMM-1 Phi ILLaMA 2/ COPT-S.T 7.70 2.7. OPP. 125M
20%
21
30
26
28
29
33
23
19
25
25%
25
26
31
23
18
24
0%
26
25
31
27
20%
21
23
28
23
24
29
25
30
24
20
25
25%
30%
19
0%
24
20
26
30
25
20%
26
27
24
25%
30%
26
33
24
19
24
0%
28
46
36
10
40
55
51
58
25%
26
30%
27
24
30
26
24
29
70%
45
17
19
57
20%
38
30
25%
20
48
29
42
33
37
52
50
37
30%
25
25
32
30
20%
26
27
20
31
26
20
25
25%
25
21
Table 23: Zero-shot performance compressed LLMs on MMLU (Massive Multitask Language Understanding) tasks without RFT. Task deseriptions: Abs. Alg: Abstract Algebra, Bus. Eth. Business Ethics, Clg. CS.: College Computer Science, Clg. Math: College Mathematics, Con. Phy.: Conceptual Physics, Frm. Log.: Formal Logic, Glb. Fet.: Global Facts, MI.:: Machine Learning, Misc.: Miscellaneous, Phil.: Philosophy.
24


Reviews:
[{"review_id": "qeckXggy0A", "reviewer": "Reviewer_ZGwT", "summary": "This paper introduces PruneNet, a policy-based pruning method designed to learn the removal of redundant parameters without the need for calibration data. PruneNet’s policy is trained based on intrinsic model properties to minimize information loss, quantified using the KS distance between the singular value distributions of the uncompressed and compressed matrices. To make policy learning differentiable, the paper employs the reparameterization trick, introducing a random variable into the sampling process and training the policy module by evaluating various pruning outcomes. The proposed method is primarily evaluated on Llama-2 7B and Phi-2, with comparisons to SliceGPT, a technique that directly compresses parameter matrices by training on datasets. Results indicate that PruneNet achieves superior performance on several popular benchmarks.", "questions": "Please see weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* One of the key strengths of this paper is a data-independent importance scorer, derived through a policy-learning process. The policy focuses mainly on individual linear layers, making the method theoretically efficient. This efficiency is demonstrated by its successful application to Llama-7B, where pruning requires only 15 minutes, illustrating its practicality for large-scale models.\n* The authors conducted extensive experiments, exploring factors such as compression ratio, RFT dataset, and computational costs. \n* The policy learner can be applied across different compression ratios, which offers practical value for determining optimal pruning ratios in large language models (LLMs). This flexibility could be particularly advantageous for users looking to fine-tune pruning ratios without retraining the policy for each setting.", "weaknesses": "* The pruning approach is focused on single layers, potentially overlooking structured interdependencies with subsequent layers. In structured pruning, parameters in one layer impact those in the next; by not accounting for this, the proposed method risks removing essential parameters from interconnected layers. This oversight could inadvertently affect model performance, as it might miss opportunities for more coordinated pruning across layers.\n* Although the method is designed to optimize the KS distance before and after pruning, it lacks a detailed analysis of its policy-learning approach. A simple random sampling approach might also work: sampling multiple randomly pruned matrices and selecting the one with the lowest KS distance might be able to provide good results. Including more ablation studies to compare these methods would enrich the analysis. Additionally, further discussion on the design and benefits of the policy approach would add valuable insights.\n* While the method shows promise on Llama-7B, its efficacy on larger models, such as Llama 13B, remains untested. An evaluation on larger LLMs would help establish its robustness and scalability, further highlighting its applicability to a broader range of large models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "miHjoVYis6", "reviewer": "Reviewer_bk7b", "summary": "This paper proposes a FFN-layer pruning algorithm for large language models that does not require a calibration dataset. The idea is to prune the weight matrices so that the spectrum of the matrices before and after pruning remains similar in the Kolmogorov-Smirnoff distance. To hasten the pruning procedure, one trains a policy that predicts the importance scores of the first layer of FFN. The empirical results show that the proposed method outperforms SliceGPT, in terms of prediction quality and the actual speedup.", "questions": "Discussed in 'weaknesses'", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The central research question of training a policy to prune LLMs is quite intriguing, with some practical importance. Although the paper mainly claims the benefit on the data side, I personally find that the method might have more future uses on low-memory or compute scenarios where it is very difficult to run usual pruning algorithms that require much gradient-based computations.\n\n- The empirical performance seems to be reasonably good.\n\n- The paper asks many interesting questions in section 5.2, including the transferability and the layerwise compression ratio.", "weaknesses": "- One nitpick is that the comparison against SliceGPT, reported on Table 2, can be quite misleading. The figures seems to be the one for SliceGPT with WikiText2 calibration data, which is much worse than the SliceGPT with Alpaca calibration data. In fact, with Alpaca, I believe that the SliceGPT works much better than the proposed PruneNet. Of course, this is somewhat expected because it utilizes more data than PruneNet; making a comparison with both WikiText and Alpaca case does not diminish the usefulness of the proposed method, as they assume different amount of usable resources. Thus I recommend including both SliceGPTs, for a more comprehensive comparison.\n\n- Another weakness is that the proposed method may not work better than the methods that use calibration data, as assuming no access to the calibration data also makes it impossible to use RFT. It seems like Table 3, ironically, can also be interpreted as a limitation of the proposed method, in a sense that the model pruned with PruneNet cannot recover much with RFT, while LLMs pruned with other methods can recover much with fine-tuning.\n\n- Also, the motivation is quite unclear to me. At least for the text data, I am not sure why it is practical to assume no access to the calibration data. We already have quite abundant text data (crawled from web), even as our benchmark. Why should we suddenly assume that those data are not available? Can you give any further justifications?\n\n- The idea of spectrum matching is also not justified well. First, I do not see how corollary 3.3 can be used to predict that the singular values of the matrix become more right-skewed. Doesn't the corollary simply states that the spectrum will be about subsampling, rather than predicting anything about the skewness? Also, the fact that there can be some spectrum shift does not fully mean that it will be an effective criterion for deciding how to prune. Perhaps more comparison with other criterion (e.g., minimizing Frobenius norm distortion) may help us understand whether the spectrum matching part is indeed an essential and nontrivial component.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G6ZxLPyYGN", "reviewer": "Reviewer_ooUi", "summary": "This paper introduces PruneNet, a structured pruning technique that deploys a policy learning process to identify important parameters. The proposed method is calibration-free, and once completed, can be applied to different pruning ratios without repeated assmentments of weights and data. The proposed method can be applied to popular Llama-2-7B in just 15 minutes, which is efficient enough for LLM pruning. The author conducted experiments on Llama-2 7B and Phi-2, with the compression ratio from 20% to 30%. Compared to SliceGPT, the method archives better accuracy on 5 zero-shot tasks, keeping >80% performance of the original LLMs. Besides, even without fine-tuning, the method is still able to preserve a good average accuracy.", "questions": "May I ask about the motivation behind Equation 3 (the importance score)? It appears that the equation is designed to project the parameter $W$ into a score $d$-dim vector. Could you clarify why this approach was chosen over other designs, such as directly optimizing a d-dimensional vector as the importance score? Additionally, will the learned matrix be shared across all linear weight layers?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "The idea of calibration-free pruning is interesting.  The policy learned on one compression ratio is scalable and can be directly transferred to higher or lower ratios. This is a useful feature if one would like to evaluate the performance of pruned models for a good trade-off between performance and efficiency. In addition, Extensive experiments were conducted to evaluate the effectiveness of the policy learning, with different finetuning settings and compression ratios, The proposed method is able to achieve better average accuracy (+3%~+4%) on zero-shot tasks.", "weaknesses": "1. It appears that most of the experiments in this paper were conducted without fine-tuning. It would be insightful to examine how the proposed method compares to other baselines when fine-tuning is enabled, as this could provide a more complete picture of its performance under optimal conditions.\n2. The efficiency of policy learning has not been studied. Providing more analysis on the efficiency of policy learning would enhance the understanding of its practical impact.\n3. Based on point 3, if we allocate the same computational budget across all baselines, such as assigning the training cost of policy learning to LoRA fine-tuning in other baselines, does the proposed method still achieve superior performance?\n4. Including pseudocode for the entire pipeline would improve clarity and reproducibility.\n5. On of my main concerns is the lack of clarity behind the motivation for each design choice. For example, Equation 3 is introduced without sufficient context or explanation. The KS Distance introduced in Equation 6 also reveals weak insights about its motivation. Why KS distance instead of other distribution distance? In addition, other paragraphs could also benefit from refinement. In particular, the background section, especially sections 3.2 and 3.2, is densely packed with technical details and formulas about SliceGPT and eigenvalues. However, the relevance to the proposed method remains unclear until the final paragraph, creating a disconnect that might hinder reader understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gG1zU21h0p", "reviewer": "Reviewer_onBH", "summary": "This paper describes a technique -- PruneNet that avoids the usage of calibration data for structured pruning. They instead reformulate it as a policy learning process via (prun)ing-object(e)d (net)work. They does this by first making a corollary that the range of eigenvalues of the compressed model weights is a subset of the original model weights (corollary 3.3). Following that, the key idea is to identify (using an MLP/policy learner) and remove rows/columns in such a way that the distribution (and distance using Kolmogorov Smirnoff (KS) Distance) of the eigenvalues for original and compressed weights remain the same. \n\nThe method seems to achieve robust results across various models/datasets and compression ratios without a calibration dataset.", "questions": "***Model design***\n\n1. The sampling is done on FFN1 and complementary columns are chosen in FFN2 to match the dimensions. This makes sense logically/intuitively but did the authors experiment with sampling FFN2 separately and/or any other design choices. Does the indices match the intuitive selection. Or does the authors have an explanation on why this choice is better compared to other intuitive selections? \n2. Did the authors perform experiments on usage of other penalty functions (KS)? That can be a good ablation study to have. \n3. L292: The assumption of discounted penalty works if the LLM has higher singular values at later layers. Has there been any (previous) analysis on existing LLMs whether that’s the scenario? A reference would be great if it exists. If not, having plots similar to Figure 1 for other LLMs at appendix would be great. \n\n***Experiments***\n\n4. L361: Are the results for the one-sided test and additional details available? A reference would be great.\n5. L375 (RFT): Can the authors add more insights on why RFT on model X might reduce the performance?\n6. L375 (RFT): Can the RFT be done on SliceGPT as well (i.e Table 3 looking similar to Table 2). That will help to understand whether RFT helps SliceGPT more (>1.5%) and PruneNet less (marginal 1.5%) or less for both methods (in either case, it will be a useful finding to understand the effects of RFT)\n\n***Suggestion for additional experiment on design choice***\n\n1. Can this policy be extended to attention matrices as well? From my understanding, the key idea seems to be applicable on any matrix, so this experiment will be super helpful to understand the design space more clearly? There has been some studies [1] done to understand the effects of compression on different modules in a transformer, so the authors might take some inspiration to understand the effects on compressing on different modules.\n2. Can the authors perform an experiment with >40% compression ratio? The performance drops most likely with higher compression ratios [1, 2, 3] but it would be great to know how much the PruneNet can achieve for reasonable compression. \n\n***Possible relevant citation***\n1. The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models - https://arxiv.org/abs/2312.00960\n2. Are sixteen heads really better than one? - https://arxiv.org/abs/1905.10650\n3. Compressing bert: Studying the effects of weight pruning on transfer learning - https://arxiv.org/abs/2002.08307\n\n***Format***\n\n1. Line 37 - Can be more descriptive about the model sizes rather than a simple statement (eg: Llama 2 can be of different sizes)\n2. Can the authors give clear explanations on different between Effective Sparsity and Sparsity (maybe in Appendix if possible)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The removal of dependence on calibration dataset while pruning. As noted by authors in L115, calibration dataset has an impact on performance of models, so it has important effects for applications. \n2. The intrinsic model compression is an interesting observation. This helps practitioners understand the dependency of compression methods (whether method X is adding parameters or not) with respect to the compression ratios\n3. The reusability of policy learner i.e train a learner (MLP) for one compression ratio and use it for another ratio as the authors have noted down is a huge strength (L412)\n4. Very well written with clear presentation.\n\nI believe the paper would be a great addition with some clarifications needed in design space and experiments.", "weaknesses": "I don't have a list of \"weakness\" i.e substantial missing components. Please refer the questions section for potential ideas and clarifications needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)