Beware of Calibration Data for Pruning Large Language Models
Authors: Yixin Ji
Yang Xiang
Juntao Li
Qingrong Xia
Ping Li
Xinyu Duan
Zhefeng Wang
Min Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: calibration data
post-training pruning
large language models
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: As large language models (LLMs) are widely applied across various fields, model
compression has become increasingly crucial for reducing costs and improving
inference efficiency. Post-training pruning is a promising method that does not
require resource-intensive iterative training and only needs a small amount of
calibration data to assess the importance of parameters. Recent research has enhanced post-training pruning from different aspects but few of them systematically
explore the effects of calibration data, and it is unclear if there exist better calibration data construction strategies. We fill this blank and surprisingly observe that
calibration data is also crucial to post-training pruning, especially for high sparsity. Through controlled experiments on important influence factors of calibration
data, including the pruning settings, the amount of data, and its similarity with
pre-training data, we observe that a small size of data is adequate, and more similar data to its pre-training stage can yield better performance. As pre-training data
is usually inaccessible for advanced LLMs, we further provide a self-generating
calibration data synthesis strategy to construct feasible calibration data. Experimental results on recent strong open-source LLMs (e.g., DCLM, and LLaMA-3)
show that the proposed strategy can enhance the performance of strong pruning
methods (e.g., Wanda, DSnoT, OWL) by a large margin (up to 2.68%).

Published as a conference paper at ICLR 2025
BEWARE OF CALIBRATION DATA FOR PRUNING LARGE LANGUAGE MODELS
Yixin Ji Yang Xiang Juntao Li Qingrong Xia?
Ping Li Xinyu Duan', Zhefeng Wang'. Min Zhang 1.2
1 Schooo of Computer Science and Technology. Soochow University
"Key Laboratory of Data Intelligence and Advanced Computing, Soochow University
Huawei Cloud, China
(jiyixin16,, baw.ww022112 @emaill.@mmm.m.mmm com;
mit.nmizzaaa edu. en duanxinyu,
(xiagingrong, innn wangzhefeenJ com;
ABSTRACT
As large language models (LLMs) are widely applied across various fields, model compression has become increasingly crucial for reducing costs and improving inference efficiency. Post-training pruning is a promising method that does not require resource-intensive iterative training and only needs small amount of calibration data to assess the importance of parameters. Recent research has en- hanced post-training pruning from different aspects but few of them systematically explore the effects of calibration data, and is unclear if there exist better calibra- tion data construction strategies. We fill this blank and surprisingly observe that calibration data is also crucial to post-training pruning, especially for high spar- sity. Through controlled experiments on important influence factors of calibration data, including the pruning settings, the amount of data, and its similarity with pre-training data, we observe that small size of data adequate, and more simi- lar data to its pre-training stage can yield better performance. As pre-training data is usually inaccessible for advanced I.L.s, we further provide self- self-generating calibration data synthesis strategy to construct feasible calibration data. Experi- mental results on recent strong open-source LLMs (e.g.. DCLM, and LLaMA-3) show that the proposed strategy can enhance the performance of strong pruning methods (e.g., Wanda, DSnoT, OWL) by a large margin (up to 2.68%
INTRODUCTION
Recently, Large Language Models (LLMs) have exhibited remarkable performance and enormous potential in Natural Language Processing (NLP) and Artificial Intelligence (AT) (OpenAT, 2022
2023 Bubeck al.
2023 Yang et al.) 2023). The success of LLMs is closely tied to scaling 2020: Hotimann et al. (002):: training language models with more parameters, using more data and greater computational resources leads to more powerful capabilities. However, LLMS with more parameters increase the difficulty and cost of deployment and inference. Therefore, much work has been devoted to compressing LLMs to achieve trade-off' between efficiency and performance, such as pruning (Ma el al.) 2023 Xia et al. 2024) and quantization (Frantar et al., Shao al. (024).
laws Kaplan et al.
2023 Huang et 2024
Pruning is model compression technique that has evolved over many years (LeCun et al. 1989) and remains full of potential and challenges. Based on the over parameterizaaiion of neural networks, it aims to remove redundant parameters while minimizing the degradation of model performance. Pruning has been successfully applied to compress small to medium-sized sistuum--izedddd neural networks. Through sparse training (Lee et al.) | Frankle & Carbin] 2019 Yuan et al.) [021] Lasby et al.) (2024) or pruning-aware training (Sanh et al.] 2020; Lagunas et al. 2022] Jiang et al. 2023) methods, it can achieve performance comparable to dense models with high sparsity ratio (2 70%). However, these
Corresponding author.
'Code is available at httpts:///tthhheeeee httcpp////stthaa..cc///errreeees bbn.aitoaratiiotdataaaaaa

---

Published as a conference paper at ICLR 2025
Willpedia
Method
DonoT
Wends
66
Stip-sjama
DCLM
Dara
64
A2
A2
(a) Peformance differences of repre- sentative pruning methods with the commonly-used C4 calibration data.
(b) Performance differences of vari- calibration data on SparseGPT.
(c) Method differ- enees data dif- ferences.
Figure 1: The effects of pruning methods and calibration data commonsense reasoning tasks.
methods require iterative training, which is costly and time-consuming for LLMs with billions of parameters. As result, post-training pruning that does not require iterative training has become the preferred approach for pruning LLMs.
The challenge of post-trainin post-training pruning is how to perform training-free parameter importance estima- tion. Frantar Alistarh| (2023) note that simple parameter magnitude-based metrics perform poorly in post-training pruning with over 20% sparsity. Therefore, they use a small amount of calibration data to compute the inverse Hessian matrix, estimating parameter importance through second-order gradient information. Sun et al. (2024) propose simpler method by using the product of weight corresponding input activations. [Dong et al. (2024) utilize the genetic algorithm to search for the optimal combination of information from magnitude, activation, and gradient as an importance metric. Overall, current advanced parameter importance metrics rely on calibration data. Although most papers claim their pruning methods are robust to calibration data, Williams & Aletras (2024)s empirical study challenges this view. They demonstrate the per- formance differences of various methods using different calibration data. Our experiments further revealed that the performance gains from selecting better calibration data can even surpass those of advanced pruning methods (Figure
magnitudes and the 1.2 norm
To learn more about calibration data, we design experiments to explore (1) the impact of calibration data with increased sparsity and varied pruning types, (2) the influence of the amount of calibra- tion data, and (3) the selection strategy of calibration data. Our empirical results demonstrate that as sparsity increases, the performance differences among different calibration data become more pronounced, and simply increasing the data volume does not reduce this disparity. We further find that calibration data similar to pretraining data yields better performance. Based on this, we propose the self-generation strategy to construct appropriate calibration data for pruning in practical settings with unavailable training data. To evaluate effectiveness of our proposed calibration data sampling method, conduct experiments results show that our proposed method performs better than the commonly used calibration data and compatible with strong pruning methods by substantially improving their performance.
DCLM, LLaMA-2, and LLaMA-3 models. The
2 BACKGROUND
Model compression is crucial way to improve inference efficiency by reducing the required mem- ory, including pruning (Guo et al.) 2023 ZZhang al. 2024b) Xia et al.) 2024). quantization (Xiao) ct al. Wang et al. 2024 Ji et al. 2024), etc. The enormous memory requirements and inefficient infer- ence speeds for LLMs urgently necessitate model compression. However, many successful model compression methods have required substantial computational resourees for retraining, which limits their application for LLMs in low-resource settings. Therefore, post-training compression, which does not require retraining, has become current research focus.
2023 Lin al. 2024), low-rank decomposition (Kaushal et al. 2023 Yuan et al., [024]]
2

---

Published as a conference paper at ICLR 2025
Post-training compression methods typically approximate model compression optimization problem for layer-wist wise compression (Frantar& & Alistarh 2022 :
min W W/X,ll,,
(1)
where Wi, Wi are the original and compressed 1-th linear layer, respectively, and X, is the input fea- ture activation. For post-training pruning. to optimize the objective, OBC (Frantar & Alistarh 2022)
and SparseGPT (Frantar & Alistarh| (023) utilize second-order gradient information to measure pa- rameter importance and propose efficient algorithm for computing the inverse Hessian matrix. Wanda (Sun et al. 2024) evaluates weight importance by combining their magnitudes with input activations without requiring backpropagation. Zhang al. (2024c) propose the relative importance and activation metric (RIA), which integrates weight, input, and output activation. They also utilize the channel permutation to minimize pruning loss under N:M semi-structured pruning. Pruner- Zero (Dong et al. 2024) designs a genetic algorithm-based framework to automatically search the best pruming metric. Recently, several studies (Sung et al. 2024, Xu et al., 2024a; Yin et al. 2024) indicate that layer-wise compression, which typically applies a uniform sparsity rate across all lay- ers and evaluates weight importance within the layer, often results in suboptimal performance due to the lack of overall consideration. Specifically, Xu et al. (2024a) proposes a differentiable pruning framework designed to search for optimal pruning rates for each layer. OWL (Yin al. (2024) in- troduces outlier weighed layerwise sparsity, which relates the sparsity of each layer to the observed outliers in a proportional manner.
In the aforementioned post-training compression methods, calibration data is an indispensable com- ponent. Calibration data a small subset randomly sampled from unlabeled pretraining text. Many methods (Frantar& & Alistarh| 2023 [Sun al., (2024) Dettmers et al., (2024) claim their robustness to the quantity and distribution of calibration data, requiring only dozens or hundreds of samples with 2,048 sequence length. However, this conclusion is based the perplexity of certain datasets (such as Wikitext2), which does not fully reflect the true capabilities of the LLMs. Even if perplexity shows no significant change, the compressed model may still experience substantial performance declines in downstream tasks (Jaiswal et al. 2024). Khanal & Capone (2024) suggest that using task-specific calibration data helps improve performance on specific downstream tasks. [Williams & Aletras (0024) observe in extensive experiments that the selection of calibration data in post-training pruning and quantization methods significantly impacts downstream tasks' performance, especially post-training pruning, which is highly sensitive to calibration data. [Shin et al. (2024) notice that the reconstruction error objective (Eq, M leads to overfitting on calibration data and that self geff-generated calibration data can effectively mitigate the overfitting. Nevertheless, current research on calibration data remains under-explored, with few studies providing guidelines for selecting calibration data. Different from previous works, our paper (1) explores the impact of calibration data under varying sparsity ratios and types, (2) investigates the effect of data amount on various calibration data, not limited the widely used C4 calibration data, (3) further addresses which calibration data is suitable for LLM pruning and provides a practical and effective method.
3 THE IMPACT OF CALIBRATION DATA FOR PRUNING
Though Williams & Aletras 2024 have noted that calibration data significantly impacts post- training pruning, there exist many open questions. How much does calibration data affect prun- ing performance? How does the amount of calibration data affect compressed model performance? What data sources are more suitable for calibration? We investigate these questions in this section.
3.1 EXPERIMENTAL DETAILS
Dense Model To study the impact of data from different sources on post-training pruning methods, we need comprehensive knowledge of the data used in model training. We select the powerful and fully open-source LLM (including training data), DCLM-7E (Li et al. 2024), as the dense model and conduct post-training pruning with different calibration data on it.
3

---

Published as a conference paper at ICLR 2025
2.5
1.5
1.5
1.0
0.5
30%
40%
50%
60%
50%
4:8
2:4
sparsity ratio
(b) sparsity type
Figure 2: Pruning performance range (Max.-Mén.) of different datasets (C4, Wikipedia, Slimpa- jama, DCLM) under various sparsity ratios (a) and sparsity types (b) on Wanda.
Post-training Pruning Methods We choose three competitive and representative post-training pruning methods for evaluation: Wanda (Sun al.) 2024), DSnoT (Zhang al. 2024d) and
OWL Yin et al. 2024). These methods apply to both unstructured and semi-structured pruning.
Calibration Data We consider various data sources to be calibration data. Following the main- stream works, the calibration data sources are all from the unlabeled pre-trained corpus:
C4 (Raffel et al. 2020 is a widely used calibration data source, consisting a large amount of multilingual web text filtered from Common Crawl. We sample from the English training set. Wikipedia is a source of high-quality encyclopedic text. We use the first shard of the cleaned English version until 2023-11-01.
Slimpajama a cleaned and deduplicated version of RedPajama. It is a high-quality pre-training corpus with diverse sources, including C4, ArXiv, GitHub, Books, etc.
DCLM LLi et al.) 2024) is the pre-training data of DCLM-7B model. It includes 2.6T tokens extracted from Common Crawl. We sample from a subse
2 of the DCLM.
Aside from the experiments in Section 3.3 we follow prior works and randomly sample 128 se- quences with 2048 tokens calibration data. To mitigate the impact of sampling randomness, all experiments repeat the calibration data sampling 20 times with different random seeds and report average performance.
Evaluation Tasks Some pruning works focus the perplexity of certain datasets while neglect- ing performance on various downstream tasks, which often fails to fully reflect the capabilities of compressed models. Therefore, choose multiple widely used and challenging commonsense reasoning tasks for evaluation, including BoolQ (Clark et al. 2019), Winogrande (Sakaguchi et al. 2019), ARC-e, ARC-c (Clark al. 2018)
2021). PIQA [Bisk al. 2020). Hellaswag (Zellers and MMLU (Hendrycks et al. 2021). For MMLU, we use a 5-shot setting, while all other tasks are evaluated zero-shot setting. Our evaluation code is based on the 1m-eval uat iontthrreesssss repository We report the average performance of these seven tasks.
3.2 How MUCH DOES CALIBRATION DATA AFFECT PRUNING PERFORMANCE?
practical applications, evaluating and comparing the impact different calibration data on pruned models inevitably consumes time and computational resources. Therefore, wonder how signif- icant the impact of calibration data is on pruning performance and whether it's worth our effort seek optimal calibration data in research and practice. We consider different sparsity ratios and
'htps://uuggingffcc.. co/dattseessalllnniiie
bttps://huggingffcc.. wo/datasets//D Vooa
"httss//huggingfac.coo
hhtpp:///gthub....
4

---

Published as a conference paper at ICLR 2025
C4
Siimajjama DCLM
C4
Simmajjaaa
Wikipedia
Wilipedin
DCLM
62.5
62.0
62.0
61.5
61.5
61.0
61.0
60.5
61.5
1024 60.0
128
128
1024
(a) Wanda
(h) DSnoTT
Figure 3: The impact calibration data amount for different pre- pre-training data resources (i.e., C4, Wikipedia, Slimpajama, DCLM) and pruning methods, i..., Wanda (a) and DSnoT (b). Shaded areas represent the standard deviations of 20 random seeds.
sparsity types. Our experiments cover sparsity ratios ranging from 30% to 60%, and at 50% sparsity ratio, we further compare unstructured, 4:8 semi-structured, and 2:4 semi-structured sparsity types.
example illustrate the model's performance range, defined the difference between the maximum and minimum values, after pruning with four calibration data sets, as shown in Figure More details on the performance
We Wanda
the different calibration data can be found in Figure 6 in Appendix A Specifically, at low sparsity ratios (<50%), the performance difference between different calibration data is minimal, less than 0. 1. sparsity increases, the impact of calibration data on pruning gradually amplifies, rising from 0.5% difference at 50% sparsity to 2.3% at 60% sparsity. Notably, as shown in Figure 6 inappropriate calibration data can even have negative effect at moderate sparsity levels. For instance, at 60% sparsity, using Wikipedia and Slimpajama as calibration data performs worse than magnitude pruning without any calibration data. For sparsity types, observe that as the sparsity pattern becomes more structured, the choice of calibration data becomes increasingly important, with the maximum difference reaching 1.5% 1.8%. We also report results DSnoT and OWL in Appendix A Although different pruning methods exhibit varying performance, they show similar trends regarding the impact of calibration data. Overall, moderate to high sparsity ratios and with semi-structured sparsity types, different calibration data significantly affect the performance of pruned LLMs. For all pruning methods, higher sparsity ratios and more structured sparsity types are key to achieving effective inference acceleration. Therefore, paying more attention to the choice of calibration data is crucial.
3.3
Is CALIBRATION DATA FROM DIFFERENT SOURCES EQUALLY ROBUST TO DATA AMOUNT?
Currently, almost all post-training pruning methods for LLMs have empirically demonstrated ro- bustness in terms of the amount of calibration data they use. Typically, model performance reaches plateau when the data amount reaches 128, and more calibration data do lead to additional performance gains. We wonder whether these methods are equally robust to the amount of data for calibration data from different sources. Can certain calibration data that lead to poorer pruned models be improved by increasing the data amount?
We perform Wanda and DSnoT pruning on DCLM-7B the 2:4 semi-structured pruning setting- We randomly sample 64, 128, 256, 512, 1024, and 2048 samples from different data sources as calibration data. Figure 3 shows how the performance of pruned models changes with increasing data amount using different calibration data. We observe that the average performance of pruned models is robust to data amount, regardless of the calibration data source, with fluctuations of only 0.1 0..%%. Therefore, cannot expect that increasing the amount of calibration data will narrow the performance gap between different calibration data. Additionally, as the data amount increases, the standard deviation of the pruned model's performance decreases.
5

---

Published as a conference paper at ICLR 2025
3.4 WHAT CALIBRATION DATA SUITABLE FOR PRUNING?
Since the choice of calibration data is crucial and cannot be improved by increasing the amount alone, we have to figure out what calibration data is more suitable for pruning. We propose two reasonable hypotheses: (1) The more similar the calibration data is to the training data of the LLMs, the better the pruning performance. (2) The higher the quality of the calibration data, the better the pruning performance.
To verify the hypotheses, we perform three post-training pruning methods on DCLM- 7B with various calibration data in the 2:4 semi-structured pruning setting. We report our results in Table Among these data, using DCLM from the training data as cal- ibration data consistently achieves the best performance. C4 and Slimpajama, which are also extracted from Common Crawl,
Table 1: Pruning performance of three pruning meth- ods with four different sources of calibration data.
Method
C4
Wikipedia
Slimpajama
DCLM
Wanda
62.52p.21
61.030.21
62.31..22
62.88
DSnoT
61.7lp.2 60. 00.488.24
61.20p.21
g2.25
OWL
63. 03.40p..9
62.230.19
63.10p.22
63.60
perform slightly worse. In contrast, the source of Wikipedia differs significantly from the other three datasets. Although Wikipedia is recognized as high-quality data, shows the worst perfor- mance, falling short of DCLM by 1.3% to 1.8%. Therefore, we assert that the quality of calibration data is not the primary factor affecting pruning performance. We further quantify the similarity between different calibration data and the training data. We utilize the MinHash-LSH algorithm to encode the 3-grams of C4, SlimPajama, Wikipedia, and DCLM, calculating their Jaccard similar- ities. The results show that the Jaccard similarity between C4 and DCLM is 0.070, SlimPajama is 0.016, and Wikipedia is 0.008. This indicates that C4 is the most similar to the training data, followed by SlimPajama, while Wikipedia has the lowest similarity. This ranking aligns with their performance as calibration data in pruning. Therefore, we believe that the similarity of calibration data the training data has more significant impact on pruning performance than the qual- ity of the calibration data. Training data or data similar to the training data is better suited as calibration data. We conjecture that this may be due to LLMs learning the patterns in the training data better. Therefore, using data with similar patterns as calibration data during the pruning process can more accurately reflect the importance of model parameters.
4 CALIBRATION DATA SAMPLING METHOD
In the Section our empirical study of the open-source DCLM-7B model demonstrates that se- lecting calibration data similar to the training data can yield better pruning performance. However, in practical scenarios, the training data of many L.MMs is not publicly available to users. In this section, we will propose the "self self-ggenerating then sampling" strategy for sampling calibration data when the training data is unavailable. Formally, given a dataset D the source of calibration data and an LLM M pre-trained on an inaccessible dataset De, we aim to sample n instances from as calibration data D. that has similar distribution to D,
Recently, Xu et al. 2024b) disclosed that LLMS internalize patterns such as language structure, word distribution, and commonsense knowledge from the training data during the training process. Due to their auto-regressive nature, LLMs leverage these internalized patterns when predicting the next token, producing the generated text similar to the training data. Thus, we propose using self- generated synthetic data as a proxy for the training data for calibration in post-training pruning. Specifically, for sample from the source of calibration data D, we truncate the first tokens as the prefix and then allow the LLM M to generate contextually relevant subsequent content:
Iy ~ (M((F)) L... N.
After generating the data, we filter the synthetic data to prevent low-quality generated data from negatively impacting pruning effectiveness. We calculate each generated sample's 's perplexity and filter the 1% samples with the highest perplexity. Higher perplexity indicates that the patterns not well- fittef by the LLM and may differ significantly from the training data, making them unsuitable calibration data.

---

Published as a conference paper at ICLR 2025
5 EXPERIMENTS
5.1 EXPERIMENTAL DETAILS
To evaluate the effectiveness of our proposed calibration data sampling method, we apply it vari- ous LLMs, including DCLM-7B, 1LL...7,,LLMM22211LLLAAAA22 LLaMA-2-13B (Touvron et al. (2033) and LLaMA- 3-8B (Dubey et al., 2024). As described in Section 3.1 we use C4, Wikipedia, Slimpajama, and DCLM as baselines for calibration data, employing three post-training pruning methods: Wanda, DSnoT, and OWL, to prune the dense models. In the main experiments, we report performance at the 60% sparsity ratio. We follow previous work to evaluate the compressed LLMs' language modeling and commonsense reasoning capabilities. We do not use the Wikitext2 dataset, which is common in most papers for evaluating language modeling ability, as its similarity to Wikipedia may introduce bias when assessing the impact of different calibration data on language modeling ability. Instead, we choose the Alpaca (Taori et al. 2023) dataset, distinct from all four calibration data sources, as our language modeling test data.
When replicating DSnoT and OWL, we follow the hyperparameter settings detailed in their papers. During the self- self-generation process, we use Top-k and Top-p sampling to improve the diversity of the generated data. Specifically, we set the p-value to 0.95, the k-value to 50, and the temperature to 0.6. We apply the repetition penalty of 1.2 to avoid repeatedly generating low-quality fragments. We randomly sample 5,000 examples from C4, Slimpajama, Wikipedia, and DCLM respectively for generation. In the filtering phase, we eliminate the top 20% of samples based on their perplexity.
5.2 OVERALL PERFORMANCE
We report the main results in Table and Table 5 Overall, our self gelf--genrated synthetic calibra- tion data exceeds other baseline calibration data in language modeling and commonsense reasoning tasks and is compatible with different pruning methods. On DCLM-7B, Wikipedia, which is not part of the pretraining data, achieves the greatest performance improvement through self-generating synthetic data. It improves performance in commonsense reasoning tasks by an average of 2.2% to 2.6% compared to the original Wikipedia data, and even surpasses the commonly used C4 calibra- tion data, achieving an average increase of 0.8% to 1.2%. For C4 and Slimpajama, which partially overlap with the pretraining data, the self seff-eenerttion strategy also yields 0.9-1.5% improvement. On LLaMA family models, the self- geferatee synthetic data also performs better than the origi- nal data, with improvements ranging from approximately 0.9% to 1.1%, and surpasses the C4 data by about 0.3% to 0.5%. Surprisingly, the performance of the self-generated calibration data even exceeds that of calibration data sampled from the DCLM-7B training set, with an average improve- ment of 0.3% to 0.7%. We think this may be due to certain patterns in the calibration data that LLMs have not adequately learned. Using these patterns as calibration data may misestimate the impor- tance of parameters. In contrast, due to the nature of maximum likelihood training, self- gelf-generated calibration data typically generates patterns that LLMs have better learned, thus avoiding using un- derrepresented patterns as calibration data. Additionally, we observe that regardless of the source of synthetic data, the pruned models' performances are similar. indicates that self- -eferaeedd cal- ibration data is versatile, it can generate suitable calibration data even when the available data is significantly different from the pretraining data.
DISCUSSION
6.1 IS THE SYNTHETIC CALIBRATION DATA SUITABLE FOR OTHER PRUNING SETTINGS?
We further validate the effectiveness of geff-generated synthetic calibration data across more pruning settings. Table 3 illustrates the commonsense reasoning perfor- mance of DCLM-7B during Wanda pruning using dif- ferent calibration data at unstructured 50% and 65% sparsity ratios, as well as seri-sttuctured 4:8 and 2:4 settings. In all pruning settings, our synthetic calibra- tion data either matches or exceeds the performance of the optimal calibration data from the training set
Table 3: Pruning performance of different calibration data.
Setting
Wiki
Slim
DCLM
Syn
50%
69.43
69.07
69.26
69.62
69.64
65%
57.22
53.97
56.10
58.14
58.11
4:8
66.27
64.82
66.17
66.28
67.02
2:4
62.52
61.03
62.31
62.88
63.61
7

---

Published as a conference paper at ICLR 2025
Table 2: Pruning performance of different calibration data DCLM-7B in 60% sparsity ratio. The best performance method is indicated in bold. Wiki, Slim, and Syn are abbreviations for Wikipedia, SlimPajama, and our synthetic data, respectively. Underline means the improved performance of synthetic calibration data over the original calibration data for certain task. A denotes the average performance change of pruned models on commonsense reasoning tasks. v. X and A indicate that the calibration data belongs, does not belong, or partially belongs to DCLM-7B's pretraining data,
respectively.
Pretrain
Alpaca (4)
BoolQ
Winogrande
PIQA
Hellaswag
ARC-c
MMLU
Avg.
Wonda
Wiki w/ Syn
9.99
72.05
68.40
74.33
64.79
73.14
39.91
42.20
62.12
9.40
78.73
70.06
75.78
66.16
74.34
42.83
45.04
64.71
+2.59
C4
d 9.67
78.47
70.27
75.12
66.32
72.84
40.84
43.31
63.88
Syn
9.57
78.81
70.52
75.95
66.35
74.23
42.01
45.64
64.78
+0.90
Slim
9.76
78.56
70.16
74.27
65.07
72.37
39.94
43.40
63.40
w/ Syn
9.58
78.51
70.02
75.63
65.90
74.12
42.13
45.26
64,51
+1.11
DCLM
9.54
79.11
70.51
75.13
66.25
73.37
41.66
44.58
64.37
Syn
9.59
79.23
70.69
75.64
66.17
74.04
42.01
45.42
64.74
+0.37
DSnoT
Wiki
X
10.16
69.97
68.08
73.95
63.23
72.09
38.69
41.63
61.09
Syn
77.58
75.38
73.27
44.53
C4
9.81
76.11
69.
74.76
00.08
72.10
39.08
41.62
62.60
w/ Syn
9.56
75.61
69.30
75.56
65.13
73.06
41.11
45.24
63.57
+0.97
Slim
K
9.87
75.58
69.21
73.80
63. 88
71.37
38.63
42.25
62.10
Syn
76.08
69.27
75.09
64.57
73.16
40.97
44.57
63.39
DCLM
9.70
77.39
69.36
74.63
64.89
72.06
39.83
43.73
63.13
w/ Syn
9.52
76.56
68.35
25.55
64.70
23.43
41.43
44.81
63.55
+0.42
OWL
Wiki
9.96
75.27
67.11
74.25
63.07
73.01
38.35
38.75
61.40
Syn
9.20
78.45
68.92
76.03
65.18
73.72
40.29
42.73
63.61
+2.21
C4
x
9.52
78.14
68.90
75.55
65 25.22
72.46
38.24
39.04
62.51
Syn
28.55
76.38
65.45
24.05
40.03
42.94
63.72
+1.21
74.56
72.35
37.95
39.84
62.21
Syn
9.32
78.56
68.71
75.83
64.47
73.81
40.44
43.61
63.64
+1.43
DCLM
9.38
78.45
69.47
75.10
65.07
72.76
38.81
40.73
62.91
Syn
9.28
78.80
67.77
75.90
64.77
73.84
40.56
43.67
63.61
+0.70
DCLM. Notably, the synthetic data improve perfor-
mance by approximately 0.8% in the two semi-structured pruning settings. Since semi-structured pruning can achieve practical inference acceleration and advanced GPUs already support 2:4 sparse tensor cores. Thus, we think the self self- generated synthetic calibration data will effectively enhance the performance of pruned models in real-world deployment.
6.2 HOw DOES PREFIX LENGTH AFFECT THE PERFORMANCE OF SYNTHETIC DATA?
The prefix length during self- seffeeenrrttinn a crucial hy- perparameter. If the prefix is too short, the synthetic text 65 is likely to be far from the semantics of the original text; if it is too long, the synthetic calibration data may retain excessive patterns from the original text. Therefore, it essential explore the selection prefix length. Our experiments range from 0 to 1024 prefix lengths, where a prefix length of 0 indicates only special token rep- shows the trend
Wikipedia
resenting the start of the text. Figure 4 of commonsense reasoning performance as the prefix length varies. Once there is a prefix, the performance exceeds that of the original calibration data. However, longer prefixes do not yield better results, perfor- mance gradually declines with increased prefix length. The results indicate that using to tokens pre-
61
DCLM
Synthetic
32
512
Figure 4: Wanda pruning performance using self-generated synthetic calibration data with different prefix lengths.
8

---

Published as a conference paper at ICLR 2025
fix is optimal. This suggests that semantic consistency with the original text is not critical in synthetic calibration data; instead, the key is to avoid retaining patterns that could have negative effects.
6.3 How DOES TERPLEXITY-BASER DATA FILTERING AFFECT PRUNING PERFORMANCE?
After generating synthetic data, we employ a simple perplexity- based method to filter low-quality data. Is this perplexity- crpeexity-based based filtering method effective, and what should the filtering rate be? We conduct experiments on the DCLM-7B model. As shown in Table [ even without any filtering strategy. the synthetic data outperforms the original data. The perplexity- bassed filtering has proved to be a simple yet effective approach, with the best pruning performance at
Table 4: Impact of perplexity- based data filtering.
Data
Alpaca (s)
Commonsense
9.99
w/o fiter
64,49
10% fiter
9.42
64,76
a filtering rate of 10%-20% As the filtering rate increases, prun- ing effectiveness gradually declines, ultimately matching the per- formance of the unfiltered data. Therefore, we recommend filtering only the outliers based perplexity. as overly aggressive filtering may compromise the diversity the calibration data, negatively impacting pruning performance.
2016 fiterr
9.40
64.71
30% Gler
9.40
64.49
fitter
9.47
64.51
6.4 WHETHER SELF-GENERATED SYNTHETIC CALIBRATION DATA IS MORE SIMILAR TO TRAINING DATA?
we assert that data similar to the train- ing data is more suitable as calibration data for post- training pruning. Based on the rego-segressive gen- eration characteristies of LLMs, propose using self- self-generated data as an approximation of the train- ing data. But is the self-generated synthetic data truly similar to the model's training data than other calibration data? We use an efficient and effective Min-K%++ method (Zhang et al. 2024a) for mea- suring. H notes that after maximum like- lihood training. the probability distribution of the training data always lies at local maxima along the input dimensions. Therefore, for given token se- quence (z),, if the sequence is belong to the training data, the p( 2z) should be higher than that of other candidate tokens in the vocabulary. The Min-K%++ is formulated follows:
In Section 3.4
milapajmaa Synthetic
Wikipedia
Density
-0.8
-0.4
-0.2
Figure 5: The Min-50%++ score distribu- tion of C4, Wikipedia, Slimpajama and self- generated synthetic data.
W W(ren,))) logp(æe (x'r)) Pace
Ozei
Min-K%++(z)
Σ
W(ree,
|min-A%
( k-%
where Prgee Once is the expectation and standard deviation of the next token's log probability given the prefix Tet, respectively. min-k% refers to choosing the bottom 1% of subsequences based on scores from the sequence x. Thus, the higher a sample's 's Min-K%++ score, the more likely it is to appear in the training data. Figure uses kernel density estimation to show the distribution of Min-K%++ values for C4, Wikipedia, SlimPajama and our self-generated synthetic data. We can clearly observe that the self-generated synthetic data has higher Min-50%++ scores than the other calibration data. It indicates that the self- self-generated synthetic calibration data is indeed similar to the training data, confirming the validity of using self- aef-ggeneated data as a proxy for the training data.
7 CONCLUSION AND FUTURE WORK
In this paper, we highlight the critical role that calibration data plays in post-training pruning for LLMs. Through systematic exploration, we demonstrate that calibration data similar to the origi- nal training data leads to superior pruning performance. To address the challenge of inaccessible

---

Published as a conference paper at ICLR 2025
training data in practical scenarios, propose self-generating synthetic calibration data strategy, which effectively samples suitable calibration data for LLMs. Experimental results on the DCLM, 2 and LLaMA-3 models demonstrate that method significantly outperforms existing common-used calibration data. We firmly believe that calibration data, as an essential part of post- training pruning, still holds significant potential for further research.
Our work still has some limitations that are worth exploring further. First, we do not fully optimize the hyperparameters when generating synthetic calibration data, such as using more advanced de- coding strategies or refined filtering methods. We believe that improving these details could further enhance the effectiveness of the synthetic calibration data. Second, our experiments are limited to pruning on 7B-13B LLMs. In future work, we will validate our
unstructured and method on 70B L.LMs and in structured pruning scenarios. Additionally, we will continue to explore how to synthesize high-quality instruction data as calibration data to help compress aligned LLMss.
ACKNOWLEDGMENTS
We want to thank all the anonymous reviewers for their valuable comments. This work was sup- ported by the National Science Foundation of China (NSFC No. 62206194), the Natural Science Foundation of Jiangsu Province, China (Grant No. BK20220488), the Young Elite Scientists Spon- sorship Program by CAST (203QNNRC01), and the Priority Academic Program Development of Jiangsu Higher Education Institutions.
REFERENCES
Yonatan Bisk, Rowan Zellers, Ronan bras, Jianfeng Gao, and Choi Yejin. Piqa: Reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference Artificial Intelligence, 34:7432-74 7439, 04 2020. doi: 11.60//aai.v341005.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.
Christopher Clark, Kenton Lee, Ming-Woi Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume (Long and Short Papers), pp. 2924-22 2236,93,, Minneapolis, Min- nesota, June 2019, Association for Computational Linguistics. doi: 10. 10.18653/1//119-13000 URL https: http:://acaatttaa hogy /arg/N19-i 1300
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc. the ai2 reasoning challenge, 2018.
Tim Dettmers, Ruslan A. Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. SpQR: A sparse-quantized representation for near-lossless LLM weight compression. In The Twelfih International Confer- ence on Learning Representations, 2024. URL http:::://///..... http:///ppeeeviii/e net centtfoorunt foroomiiiii
/ttps:://ppeeeeeeeww
Qlu25ahSuy
Peijie Dong, Lujun Li, Zhenheng Tang, Xiang Liu, Xinglin Pan, Qiang Wang, and Xiaowen Chu. Pruner-zero: Evolving symbolic pruning metric from scratch for large language models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings the 4 Ist International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 11346-11374 1374. PMLR, 21-27 Jul 2024. URL pettps://proceeii:
.ll. http:/ppesssev2333/2
html
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Leuman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint 207.22477.2188.. 2024.
10

---

Published as a conference paper at ICLR 2025
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations. 2019. URL https:// openreview net/ ceee//oorum20 /ormmmmmdd=d111-bee
Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post- training quantization and pruning- In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL.
https://openrev//
net
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 10323-10337, PMLR, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization
for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023. URL URLhttps http:tttsssssseeeee
Song Guo, Jiahang Xu, Li Lyna Zhang, and Mao Yang. Compresso: Structured pruning with collab-
orative prompting learns compact large language models, 2023. UtL_httts::/arxivvcoo
abs/2310. 05015
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt, Measuring massive multitask language understanding. In International Confer- Learning Representations, 2021. URL https: /tttps:://ppeneeieww net fott/ffoum?icc
d7KB jmI 3CmQ.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen- nigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022.
Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and Xiaojuan Qi. BiLLM: Pushing the limit of post-training quantization for LLMs. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings the 4 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 20023-20042. PMLR, 21-27 Jul 2024. URL htttps::/proceedingg. mlr pttpmmm.....e////////aaaaa
html
Ajay Jaiswal, Zhe Gan, Xianzhi Du, Bowen Zhang, Zhangyang Wang, and Yinfei Yang. Com- pressing LLMs: The truth rarely pure and never simple. In The Twelfth International Confer- Learning Representations, 2024. URL httph:htp///. http:::://peereeiewe net forum?id? ddd=
Yixin Ji, Yang Xiang, Juntao Li, Qingrong Xia, Zi Ye, Xinyu Duan, Zhefeng Wang. Kehai Chen, and Min Zhang. Adaptive feature-based low-rank compression of large language models via Bayesian optimization. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 4152-4168, Miami, Florida, USA, November 2024. Association for Computational Linguisties. doi: 10.18653/v1/2024 findings-emnlp...
URL https: https::/accant //clanthooollthooogggooooo wrg.:lllyy.o..oooooooooooooo 2024. findings emmip.
24C/
Ting Jiang, Deqing Wang, Fuzhen Zhuang, Ruobing Xie, and Feng Xia. Pruning pre-trained lan- guage models without fine-tuning. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 594-605, Toronto, Canada, July 2023. Association for Compu- tational Linguistics. doi: 10.18653/v1/200 10.165555/120222acllmm...... org/2023. acl-lcn.l..nng.....
URL https::/accaaa:
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
11

---

Published as a conference paper at ICLR 2025
Ayush Kaushal, Tejas Vaidhya, and Irina Rish. Lord: Low rank decomposition of monolingual code Iims for one-shot compression, 2023. URL httph: httpaa//aaxxvvvvvo..o. http://too...bb//.......00
Bishwash Khanal and Jeffery Capone. Evaluating the impact of temppreessionncchiiiunnneen on task- specific performance of large language models, 2024. URL. htttps://arxiv..rrg////
2409.11233
François Lagunas, Ella Charlaix, Victor Sanh, and Alexander Rush. Block pruning for faster transformers. In Marie-Francin Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10619-10629, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 2..18653/v1/20/ 1tt.53//1222..mmmmmmmma..33333 .emnlp-main.829. URL
829
Mike Lasby, Anna Golubeva, Utku Evci, Mihai Nica, and Yani loannou. Dynamic sparse training with structured sparsity. In The Twelfth International Conference on Learning Representations,
2024. URL BRLLttps:: awrpeerreieew.nnt//oouuuaa.....e
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In D. Touretzky (ed.), Advances in Neural Information Processing Systems, volume 2. Morgan- Kaufmann, 1989. URL Ibttpss///pocceedddngg neurips wwtpaari.s..ccpppeeeeeeeeeee
cc/paper files/ Files/paper/1965
File file//c8822baacc0009322333121223 Pawp77766668p-pppppp.p Paper.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr.
SNIP: SINGLE NETWORK
PRUNING BASED ON CONNECTION SENSITIVITY. In International Conference on Learn- ing Representations, 2019. httptts::/// /ttps/s//////eeeeeeeeeeeeeeeee/e net/ wttp//ooruummdd111//////-
Jeffrey Li, Alex Fang, Georgios Smymis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha Sedrick Kch, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Rein- hard Heckel, Jean Mereat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Al- balak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng- Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang. Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel II- harco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Se- woong Oh, Luke Zettlemoyer, Kyle Lo. Alaacldin El-Nouby. Hadi Pouransari, Alexander Toshev, Stephanic Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kol- lar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. Datacomp-Im: In search of the next generation training sets for language models, 2024. URL https::/aaae
o abs/ / 2006 11794
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. In MLSys, 2024.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LLM-pruner: On the structural pruning of large language models. In Seirth- Conferee on Neural Information Processing Systems, 2023. URL httpttts///peeeeeieettee wtw.::wwww..e/eeemne.m....a.
forum? id= J8A E9WEXP
OpenAl. Chatgpt: Optimizing language models for dialogue. Open AI, blog, 2022.
OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http:
7/ 1/jmlr. httpm//.rrrgppppee/////////////e
074 html
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: ad- versarial winograd schema challenge at scale. Commun. ACM. 6409)99-106, aug 2021. ISSN 0001-0782. doi: 10.1 10/3444/3474381. http:htttt:://...oom 0.11//111144444443333
12

---

Published as a conference paper at ICLR 2025
Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 20378-20389 Curran Associates, Inc., 2020. URL. htttps://proceedd neuri htwp://snnnnnniiscc/ceeeeeeeeeee http:///5...aeeeeaaaaaaaaaaaaaa3
files/
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Ommidirectionally calibrated quantization for large language models. In The Tweifth International Conference on Learning Representations, 2024.
net forum?id WwlhhhlwuwwhhY
Sungbin Shin, Wonpyo Park, Jacho Lee, and Namhoon Lee. Rethinking pruning large language models: Benefits and pitfalls of reconstruction error minimization. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, pp. 1182-1191, Miami, Florida, USA, November 2024.
Association for Computational Linguistics. URL emm.lpmmain. 68
Mingjic Sun, Zhuang Liu, Anna Bair, and Zico Kolter. simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2024.
Yi-Lin Sung, Jaehong Yoon, and Mohit Bansal. ECoFLap: Efficient coarse-to-fine layer-wise prun- ing for vision-language models. In The Twelfth International Conference on Learning Represen- tations, 2024.
http::::///pppeevieieee net w
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction- following model. Stanford Center for Research on Foundation Models. https:/crrfi. stanford. edu2022/003/13alppaca html, 3(6):7, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-Ilm: Truncation-aware singular value decomposition for large language model compression, 2024. URL https::/arxiv.or
abs/2403. 07378
Miles Williams and Nikolaos Aletras. On the impact of calibration data in post-training quantization and pruning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 10100-101 18, Bangkok, Thailand, August 2024. Association for Computational Linguistics. wwwphhoooyyooooooooooooooooooooe 2024 .acl - 544
URL
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared LLaMA: Accelerat- ing language model pre-training via structured pruning. In The Twelfih International Confer- ence on Learning Representations, 2024. https:/ http:////ppeeveieieee net. http://www..eeoooouummamema formmorrrmm- 0910dae0zp
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. Proceedings of the 40th International Conference on Machine Learning, 2023.
Peng Xu, Wenqi Shao, Mengzhao Chen, Shitao Tang, Kaipeng Zhang, Peng Gao, Fengwei An, Yu Qiao, and Ping Luo. BESA: Pruning large language models with blockwise parameter-efficient sparsity allocation. In The Twelfth International Conference on Learning Representations, 2024a. URL
www.ppenneiiiee net httpreieeweeeetttteeeeeeeee forum? wetrummid-oo 3dc.unTE333
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned 1lms with nothing, 2024b. htt.tt httppp:::///xi.i.o/ wttp:/awww......aaaaaa/aaaaaaae/
13

---

Published as a conference paper at ICLR 2025
Zhengyuan Yang. Linjie Li, Kevin Lin, Jianfeng Wang, Chung Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of Imms: Preliminary explorations with 4pt-iv()sion), 2023.
Lu Yin, You Wu, Zhenyu Zhang, Cheng- Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, AJAY KUMAR JAISWAL, Mykola Pechenizkiy, Yi Liang Michael Bendersky, Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (OWL): A missing secret sauce for pruning LLMs to high sparsity. In Forty-first International Conference on Machine Learning, 2024. URL https: //openeeview.o net wtrtt//oommmiimm www.ummmnddddm22222222
Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, et al. Mest: Accurate and fast memory-cconomi sparse training framework on the edge. Advances in Neural Information Processing Systems, 34, 2021.
Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation-aware singular value decomposition for compressing large language models, 2024.
URL httph: htttt://aaxxxv org/ ats//////333333 05821
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10. 18653/v 1/tp://///91111111 1472. URL htt.t:t/////// http://caaaooooooooore/eeeee////
Jingyang Zhang. Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang. and Hai Li. Min-k%++ Improved baseline for detecting pre-training data from large language models, 2024a. RtLhtttss///xxxi/> / /2w. 2404 02936
Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang. Linlin Ou, Xinyi Yu, and Bohan Zhuang. LoRAPrune: Structured pruning meets low-rank parameter-efficiee
fine-tuning. In Lun-Wet Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguisticy ACL 2024, pp. 3013-3026, Bangkok, Thailand and virtual meeting, August 2024b. Association for Computational Linguistics. URL https: accl wotthelogy.org//// findings-acl.117
Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. Plug- and-play: An efficient post-training pruning method for large language models. In The Twelfth International Conference on Learning Representations, 2024c. URL http////ppe/eieeeeeee/eeeee
Yuxin Zhang, Lirui Zhao, Mingbao Lin, Sun Yunyun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, and Rongrong Ji. Dynamic sparse no training: Training-free fine-tuning for sparse LLMs. In The Twelfth International Conference on Learning Representations, 2024d. URL https:
/ppenzeview wet./oooommddddaddiieeeeeeeeeeee
14

---

Published as a conference paper at ICLR 2025
A
MORE STUDIES ON DIFFERENT SPARSITY
72.00
72.25
1035
Stsrpejeze DCLM
Stanpsjana DCLM
Sinpejons
Wildpedia
72.00
Wikipeii
Wilipedl
DCLM
71.90
09.58
71.25
6925
11300
11.00
78.75
66.75
78.50
68.51
11.28
78.25
04.35
71.00
70.00
08.1
sparsity ratio: 0.3
(b) sparsity ratio: 0.4
(c) sparsity ratio: 0.5
3.500
67.50
6431
C4 Wikieelia
Stimpajaam DCEM
C4
stimp-jama DCLM
Ci
Simpaaana
Wilpedii
6555
Wilupoll
DCLM
66.90
63,1
$4.0
66.00
62.08
$3.00
65.00
61.50
64.50
61.08
41.50
63.50
60.00
6031
(d) sparsity ratio: 0.6
(c) sparsity type: 4:8
(f) sparsity type: 2:4
Figure 6: Pruning performance of different datasets (C4, Wikipedia, Slimpajama, DCLM) under various sparsity ratios (a-d) and sparsity types (e-f) on Wanda. The gray dash lines represent the performance of magnitude-base pruning.
TMM
Slaspsjame
Simpujsen
73.75
00.55
12.50
71.50
09.51
79.25
69.35
78.75
68:75
11.90
78.00
68.50
11.35
78.25
08.35
68,0
(a) sparsity ratio: 0.3
(b) sparsity ratio: 0.4
(c) sparsity ratio: 0.5
64.8
Stimpejane
Stinpsjans DCLM
Simppejoa DCLM
Wikipedia
Wakip
Wikipoda
66.50
63,11
66.00
63.58
66.50
62:08
65.00
61.08
$1.00
64.50
60.58
51.0
64.00
0033
(d) sparsity ratio: 0.6
(c) sparsity type: 4:8
(f) sparsity type: 2:4
Figure 7: Pruning performance of different datasets (C4, Wikipedia, Slimpajama, DCLM) under various sparsity ratios (a-d) and sparsity types (e-f) on DSnoT. The gray dash lines represent the performance of magnitude-based pruning.
B MORE RESULTS OF SYNTHETIC CALIBRATION DATA
15

---

Published as a conference paper at ICLR 2025
13,0
12.50
Toat
C4
Strmpajama DCEM
C4 Waipedia
Stimps/ama DCLM
Ci
Simpsjana DCLM
12.75
Wikgeeii
6975
Wiligalle
12.00
09.51
71.50
7200
09:3
71.75
71.00
64,75
1.50
68.58
78.50
78,00
060
(a) sparsity ratio:
(b) sparsity ratio: 0.4
(c) sparsity ratio: 0.5
67.00
64:0
Staspsjana DCLM
Stmpejane DCLM
66.50
63.50
Wilegeole
66.00
63.00
66.50
62.50
64.50
61.58
61.00
61.01
60.59
00.53
(d) sparsity ratio: 0.6
(c) sparsity type: 4:8
(1) sparsity type: 2:4
Figure& 8: Pruning performance of different datasets (C4, Wikipedia, Slimpajama, DCLM) under various sparsity ratios (a-d) and sparsity types (e-f) on OWL. The gray dash lines represent the performance of magniuue-based pruning.
Table 5: Pruning performance of different calibration data on LLaMA-2-7B in 60% sparsity tio. The best performance method is indicated in bold. Wiki, Slim, and Syn are abbreviations for Wikipedia, SlimPajama, and our synthetic data, respectively. Underline means the improved perfor- mance of synthetic calibration data over the original calibration data for a certain task. A denotes the average performance change of pruned models on commonsense reasoning tasks.
Method
Data
Alpaca (1)
BoolQ
Winogrande
PIQA
Hellaswag
ARC-e
ARC-c
MMLU
Avg.
Wiki
10.42
66.80
63.84
70.55
56.69
64.78
34,23
22.94
54.26
w/ Syn
9.62
68.29
64.40
71.49
58.89
64.73
35.41
24.01
55.32
+1.06
C4
10.42
66.30
64.50
71.12
58.92
64.92
33.91
23.06
54.68
Wanda
w/
10.07
67.46
64.15
71.38
59.05
65.64
33.83
23.92
55.06
+0.38
Slim
10.23
66.83
63.68
71.10
57.54
64.68
33.98
22.95
$4.39
w/ Syn
9.92
67.91
64.63
71.45
58.52
65.28
33.93
23.29
55.00
+0.61
DCLM
71.15
58.72
64.81
33,98
23.65
55.07
w/ Syn
9.77
68.90
64.56
21.71
58.90
65.24
34.47
23.61
55.34
+0.27
Wiki
10.92
66.24
62.72
70.55
55.55
64.10
33.16
23.05
53.62
w/ Syn
10.40
65.44
64.01
71.49
57.77
64.86
34,30
23.90
54.54
+0,92
C4
10.88
65.25
64.04
71.22
57.15
64.40
32.82
23.45
$4.05
DSnoT
w/ Syn
9.90
18
64.72
71.00
$7.19
64.86
33.45
24.87
54.61
+0.56
Slim
10.76
65.66
63.66
70.82
56.17
64.43
32.51
23.15
$3.77
w/ Syn
10.04
65.23
63.22
70.84
56.56
65.11
33.11
23.67
53.97
40,20
DCLM
10.37
66.65
63.99
71.44
56.77
64.56
33,30
23.73
$4.35
Syn
66.24
71.00
$7.64
64.86
33.70
24.09
54.51
Wiki
9.30
66.50
66.05
71.82
61.90
67.57
35.89
26.07
$6.54
w/ Syn
9.13
69.85
66.38
73.18
62.86
67.89
35.07
26.34
57.37
+0.83
C4
9.19
66.73
67.34
72.74
62.87
67.54
35.68
26.20
$7.02
OWL
w/ Syn
9.11
68.47
67.40
72.52
62.99
66.75
35.41
27.28
57.26
40.24
Slim
9.21
67.52
66.91
72.32
62.25
66.70
34.91
26.05
56.67
Syn
69,30
67.80
72.31
62.56
67.17
35.75
26.86
$7.39
DCLM
9.08
69.79
67.94
72.39
62.73
67.06
35.85
26.45
57.46
w/ Syn
9.10
69.57
67.72
72.63
62.60
67.59
35.84
26.32
57.47
+0.01
16

---

Published as a conference paper at ICLR 2025
Table 6: Pruning performance different calibration data on LLaMA-2-13B 60% sparsity ra-
tio. The best performance method is indicated in bold. Wiki, Slim, and Syn are abbreviations for Wikipedia, SlimPajama, and our synthetic data, respectively.
Method
Data
Alpaca
BoolQ
Winogrande
PIQA
Hellaswag
ARC-e
ARC-c
MMLU
Avg.
C4
8.99
77.36
68.68
75.45
66.51
69.18
39.74
26.80
60.53
Wiki
9.21
74.39
67.97
74.97
64.39
68.66
38.62
24.96
59.14
Wanda
Slim
8.76
76,82
68.42
75.25
65.18
69.03
39.56
28.01
60.32
DCLM
8.73
77.50
68.37
75.16
66.34
69.95
40.15
27.98
60.78
Syn
8.73
77.06
68.68
75.19
66.25
70,03
40.19
29.06
60.92
C4
9.03
77.16
66.60
74.92
65.76
69.81
38.45
25.73
59.77
Wiki
9.34
76.02
65.89
74.43
63.84
68.93
37.95
25.19
$8.89
DSnoT
Slim
9.03
76.31
66.79
74.84
64.44
70.13
38.33
26.97
59.69
DCLM
9.04
77.22
67.56
74.52
65.38
69.94
38.72
26.97
60.04
Syn
8.96
77.09
67.64
74.54
65.33
70,29
39.68
27.08
60.23
C4
7.56
78.92
70.02
75.95
69.12
70.90
41.14
32.75
62.69
Wiki
8.25
77.93
69.47
75.23
68.13
71.20
39.23
31.75
61.85
OWL
Slim
7.68
79.41
69.69
75.55
68.42
70.60
40.19
32.47
62.33
DCLM
7.33
79.85
70.23
75.57
69.21
71.62
40.48
33.77
62.96
Syn
7.35
79.05
69.61
76.50
69.11
71.51
41.55
31.19
62.65
Table 7: Pruning performance of different calibration data on LLaMA-3 --aMA-3-8B in 60% sparsity ra- The best performance method is indicated in bold. Wiki, Slim, and Syn are abbreviations for Wikipedia, SlimPajama, and our synthetic data, respectively.
tio.
Data
BoolQ
Winogrande
PIQA
Hellaswag
ARC-e
ARC-c
MMLU
Avg.
C4
69.02
60.55
67.98
49.47
59.95
30.59
23.60
51.59
Wiki
66.82
59.02
67.40
47.14
59.79
29.67
24.14
50.57
Slim
66.86
60.11
67.53
48.07
59.38
29.96
23.52
50.77
DCLM
70.14
61.17
67.83
49.97
60.04
31.16
23.22
51.93
Syn
70.03
61.88
68.06
50.11
59.85
31.66
23.19
52.11
17


Reviews:
[{"review_id": "Zhv6PjSvmC", "reviewer": "Reviewer_mY5n", "summary": "This paper investigates the role of calibration data in post-training pruning for large language models (LLMs). The authors find that calibration data similar to the training data yields better performance when pruning LLMs for model compression. As many training datasets for LLMs are inaccessible, the authors propose a strategy to create synthetic calibration data, which outperforms commonly used datasets in experiments. This strategy involves generating synthetic text using the LLM and then filtering out low-quality data. This synthetic data is more similar to the training data and ultimately leads to better performance for pruned LLMs.", "questions": "- What are the main differences between this work and the work by [1]?\n\n- The authors say that \"We can clearly observe that the self-generated synthetic data has higher Min-50%++ scores than the other calibration data. It indicates that the self-generated synthetic calibration data is indeed similar to the training data, confirming the validity of\nusing self-generated data as a proxy for the training data.\". The conclusion is not entirely clear to me, can you explain how to conclude that synthetic calibration data is similar to the training data in this figure?\n\n- While the paper aims to enhance general capabilities, the impact of using domain-specific calibration data for pruning models intended for specialized tasks remains unclear. do the authors have any intuition for that?\n\n[1] Miles Williams and Nikolaos Aletras. On the impact of calibration data in post-training quantization\nand pruning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the\n62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),\npp. 10100–10118, Bangkok, Thailand, August 2024. Association for Computational Linguistics.\nURL https://aclanthology.org/2024.acl-long.544.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper effectively challenges the common assumption that post-training pruning methods are robust to the choice of calibration data. Recognizing the challenge of inaccessible training data, the paper introduces a \"self-generating then sampling\" strategy for constructing suitable calibration data. The paper provides a detailed examination of various aspects related to the self-generating calibration data strategy", "weaknesses": "While the paper shows a correlation between training data similarity and pruning performance, it doesn't explain why this connection exists. The paper's evaluation primarily centers on overall model performance. Investigating how calibration data affects the pruning of individual model components like attention heads or specific layers could be beneficial. This granular analysis would offer a more complete picture of how calibration data impacts different parts of the LLM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xmo7P6RRQX", "reviewer": "Reviewer_9hYo", "summary": "Large language models with numerous parameters substantially increase deployment and inference complexity and costs. To mitigate this, post-training parameter pruning can be used which exploits the fact that neural networks are often over-parametrized. It operates selectively removing redundant parameters while aiming to preserve performance as measured using a sample of calibration data.\nThe key contributions of this paper are: (i) a (plausibly) novel data synthesis strategy for calibration data, and (ii) an investigation into the effects of size, quality, and distribution of calibration data, across different pruning hyperparameters.\nAdditionally, the paper examines major hyperparameter choices within their strategy and perform additional analysis to show that their synthesis method generates data that is distributed similar to the training data.", "questions": "Suggestion (I): decompose or improve the table to highlight matching or exceeding the performance of using calibration data from the actual training set and exceeding the performance compared to calibration datasets belonging to other distributions. \n\nSuggestion (II): avoid redundancy in repeating the literature review and possibly summarize the questions in the introduction. In the literature review, the name of the technique corresponding to each citation could be mentioned as well.\nSuggestion (III): improve the abstract to better reflect the outcomes of the paper and be easier to read.\nSuggestion (IV): Mention somewhere that the paper will first proceed by answering the calibration data related questions and then propose a new a novel technique for its generations. Typically, one expects the main novel contribution to come first.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper productively expands on prior work to answer unanswered follow up questions related to the influence of calibration data on pruning and delivers insightful findings through a set of reliable experiments.\nIt proposes a novel and intuitive approach for the synthesis of calibration data and evaluates it empirically and theoretically while experimentally justifying major hyperparameter choices. They show that the approach can improve by up to 2.6% over using an out-of-distribution calibration dataset.\nThe paper also clearly describes background, relevant pruning approaches, the problem statement and proposed approach for calibration data synthesis as well as experimental results.", "weaknesses": "The main results are not so well represented. In Table 2, the proposed calibration data synthesis approach frequently falls behind other sources of calibration data. It’s not highlighted in the table (e.g., using colors or otherwise) whether each source was present in the training set of the evaluated LLM. That is, it makes sense to have separate comparisons for the proposed approach with each of (i), data the model was not trained on and (ii), data the model was trained on, but these seem to be mixed up in one table making it hard to interpret the quality of the results by looking at the table. The statement “Overall, our self-generated synthetic calibration data outperforms other baseline calibration data in language modeling and commonsense reasoning tasks and…” is not well justified because the remaining of the paragraph focuses on Wikipedia and C4 and its not obvious from the table that it outperforms all sources consistently over all tasks. \n\nThe paper involves some redundancies. For instance, the introduction as well as background seem to closely repeat the literature review. The questions are mentioned in the introduction then later again in section 3. Moreover, the choice of words in some of the sentences used is inadequate. For instance, the use of “value more” in “We fill this blank and surprisingly observe that the effects of calibration data even value more than designing advanced pruning strategies.” Take note as well that the paper does not convey that this “values more” than designing more advanced pruning strategies and that’s nontrivial to prove. Constructs such as “while different calibration data’s impact on pruning performance still lacks systematical exploration.” also make the abstract harder to read compared to if it was something like “while the impact of calibration data used has been…”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k01oV2Pt4i", "reviewer": "Reviewer_4EBf", "summary": "This paper studies the impact of calibration data in the post-training pruning of LLMs, which shows that calibration data significantly affects pruned model performance as pruning difficulty increases, surpassing the improvements from advanced pruning algorithms. The authors also find that using training data or data similar to it as calibration data significantly boosts pruned model performance. Since pre-training data is often unavailable for advanced LLMs, the paper proposes a strategy for self-generating calibration data. Extensive experiments on multiple LLMs and pruning methods confirm the effectiveness of the proposed synthetic calibration data.", "questions": "1. Where does Figure 6 reflect the results of magnitude-based pruning?\n2. Are the conclusions and method presented in this paper applicable to LLM quantization?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces a criterion and construction strategy for choosing calibration data in post-training pruning, supported by extensive experimental validation.\n2. The authors conduct experiments on various LLMs and pruning methods, with multiple repetitions, to eliminate the effects of randomness.\n3. The paper is well-organized, clearly presenting the empirical studies, methodology, experiments, and results, making it easy for readers to follow the authors' arguments.", "weaknesses": "1. This paper only conducts experiments on unstructured and semi-structured pruning settings and does not validate the effectiveness of synthetic calibration data in more practical structured pruning.\n2. The synthetic calibration data is not a method first proposed by the authors. A recent work by Shin et al.[1] also proposed synthetic calibration data. However, the authors do not discuss the differences between that work and the others.\n3. This paper only uses data from Wikipedia to generate synthetic data. Why do you not validate the effectiveness of synthetic data generated from other sources?\n\n[1] Shin, Sungbin, et al. \"Rethinking Pruning Large Language Models: Benefits and Pitfalls of Reconstruction Error Minimization.\" arXiv preprint arXiv:2406.15524 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xDl36f4F3f", "reviewer": "Reviewer_4fLd", "summary": "This paper investigates the impact of calibration data in the pruning of large language models (LLMs). This work mainly repeats some work that has been done by Williams & Altetras (EMNLP2024), which investigates the impact of calibration data in the pruning and quantization of large language models. The authors present evidence that the quality and type of calibration data can impact pruning performance, at times more so than advanced pruning methods themselves, reflecting the results done by Williams & Altetras (EMNLP2024). They propose a self-generating calibration data synthesis strategy to create effective calibration datasets when access to training data is limited.", "questions": "Could the authors provide further clarification on how efficient their proposed calibration data synthesis method is, e.g., what are the minimum data points it needs to generate for calibration?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "1. **Originality:** In addition to the models included by Williams & Altetras (EMNLP2024), the authors also tested with DCLM-7B. This model is designed to showcase the effectiveness of systematic data curation. They propose a self-generating calibration data synthesis strategy. \n2. **Quality:** The paper provides a systematic exploration, supported by experimental results demonstrating how different calibration datasets affect the performance of pruned models.\n3. **Clarity:** The writing is reasonably clear and easy to follow. The objective is straightforward. \n4. **Significance:** The findings have significant implications for practitioners in the field, although it has been highlighted by previous work already.", "weaknesses": "1. **Lack of Novel Contribution:** The study builds on important findings by Williams & Aletras (EMNLP2024), and the findings are already been proven and previous work has done further comparing quantization and pruning. \n2. **Lack of downstream tasks experiments:** the authors only consider pruning performance and it does not necessarily reflect the downstream tasks. Previous work done by Williams & Altetras (EMNLP2024) has done a much more comprehensive evaluation of a wide range of downstream tasks. \n3. **No explanation on pruning performance:** The paper primarily evaluates \"pruning performance,\" but fails to provide a clear explanation of this metric. It's unclear whether this refers to pruning error, signal-to-noise ratio (SNR), or another measure. The authors neither explain their calculation method nor cite a source for this metric. \n4. **Experimentation with Diverse Datasets:** The experiments predominantly focus on a narrow range of calibration datasets and models. Including a broader set of datasets could provide more generalizable results and strengthen the conclusions drawn about the effectiveness of their proposed methods.\n5. **Validation or discussion of choices in methods:** There are some variables actually can be potentially impact the results, such as why 5000 samples from the Wikipedia data for generation, and why eliminate the top 20%.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)