Beware of Calibration Data for Pruning Large Language Models

## Abstract

Abstract As large language models (LLMs) are widely applied across various fields, model compression has become increasingly crucial for reducing costs and improving inference efficiency.
Post-training pruning is a promising method that does not require resource-intensive iterative training and only needs a small amount of calibration data to assess the importance of parameters.
Previous research has primarily focused on designing advanced pruning methods, while different calibration data’s impact on pruning performance still lacks systematical exploration.
We fill this blank and surprisingly observe that the effects of calibration data even value more than designing advanced pruning strategies, especially for high sparsity.
Our preliminary exploration also discloses that using calibration data similar to the training data can yield better performance.
As pre-training data is usually inaccessible for advanced LLMs, we further provide a self-generating calibration data synthesis strategy to construct feasible calibration data.
We conduct experiments on the recent strong open-source LLMs (e.g., DCLM, and LLaMA-3), and the results show that the proposed method outperforms commonly used calibration data and can effectively enhance strong pruning methods (e.g., Wanda, OWL).

## 1 Introduction

Recently, Large Language Models (LLMs) have exhibited remarkable performance and enormous potential in Natural Language Processing (NLP) and Artificial Intelligence (AI) (OpenAI, ; ; Bubeck et al., ; Yang et al., ).
The success of LLMs is closely tied to scaling laws (Kaplan et al., ; Hoffmann et al., ): training language models with more parameters, using more data and greater computational resources leads to more powerful capabilities.
However, LLMs with more parameters increase the difficulty and cost of deployment and inference.
Therefore, much work has been devoted to compressing LLMs to achieve a trade-off between efficiency and performance, such as pruning (Frantar & Alistarh, ; Ma et al., ; Xia et al., ) and quantization (Frantar et al., ; Huang et al., ; Shao et al., ).

Pruning is a model compression technique that has evolved over many years (LeCun et al., ) and remains full of potential and challenges.
Based on the over-parameterization of neural networks, it aims to remove redundant parameters while minimizing the degradation of model performance.
Pruning has been successfully applied to compress small to medium-sized neural networks. Through sparse training (Lee et al., ; Frankle & Carbin, ; Yuan et al., ; Lasby et al., ) or pruning-aware training (Sanh et al., ; Lagunas et al., ; Jiang et al., ) methods, it can achieve performance comparable to dense models with a high sparsity ratio ($\geq$70%).
However, these methods require iterative training, which is costly and time-consuming for LLMs with billions of parameters. As a result, post-training pruning that does not require iterative training has become the preferred approach for pruning LLMs.

The challenge of post-training pruning is how to perform training-free parameter importance estimation.
Frantar & Alistarh () note that simple parameter magnitude-based metrics perform poorly in post-training pruning with over 20% sparsity. Therefore, they use a small amount of calibration data to compute the inverse Hessian matrix, estimating parameter importance through second-order gradient information.
Sun et al. () propose a simpler method by using the product of weight magnitudes and the L2 norm of the corresponding input activations.
Dong et al. () utilize the genetic algorithm to search for the optimal combination of information from magnitude, activation, and gradient as an importance metric.
Overall, current advanced parameter importance metrics rely on calibration data. Although most papers claim their pruning methods are robust to calibration data, Williams & Aletras ()’s empirical study challenges this view.
They demonstrate the performance differences of various methods using different calibration data.
Furthermore, our experiments revealed that the performance gains from selecting better calibration data can even surpass those of advanced pruning methods (Figure 1).
Therefore, it is time to focus more research on calibration data.

However, many open questions regarding calibration data remain under-explored.
For example, how does the impact of calibration data change with increased sparsity and structure of pruning? Can increasing the amount of calibration data narrow the performance gap between various datasets? What type of data is suitable for calibration? How do we select the appropriate calibration data in practice?
In this paper, we investigate these questions. Our empirical results demonstrate that as sparsity and structure increase, the performance differences among different calibration data become more pronounced, and simply increasing the data volume does not reduce this disparity.
We further find that the selection of calibration data is closely related to the LLM’s training data, with calibration data similar to the training data yielding better performance.
Based on this, we propose two strategies, detection and self-generation, aimed at sampling appropriate calibration data for pruning in practical settings with unavailable training data.
To evaluate the effectiveness of our proposed calibration data sampling method, we conduct experiments on DCLM, LLaMA-2, and LLaMA-3 models.
The results show that our proposed method performs better than the commonly used calibration data and is compatible with strong pruning methods by substantially improving their performance.

Figure: (a) Peformance differences of representative pruning methods with the commonly-used C4 calibration data.
Refer to caption: x1.png

## 2 Background

Model compression is a crucial way to improve inference efficiency by reducing the required memory, including pruning (Frantar & Alistarh, ; Sun et al., ; Ma et al., ; Yin et al., ; Guo et al., ; Zhang et al., ; Xia et al., ), quantization (Frantar et al., ; Xiao et al., ; Lin et al., ; Huang et al., ; Shao et al., ), low-rank decomposition (Kaushal et al., ; Yuan et al., ; Wang et al., ; Ji et al., ), etc.
The enormous memory requirements and inefficient inference speeds for LLMs urgently necessitate model compression.
However, many successful model compression methods have required substantial computational resources for retraining, which limits their application for LLMs in low-resource settings. Therefore, post-training compression, which does not require retraining, has become a current research focus.

Post-training compression methods typically approximate model compression as an optimization problem for layer-wise compression (Frantar & Alistarh, ):

$$ $\underset{\bm{\hat{W}}_{l}}{\text{min}}||\bm{W}_{l}\bm{X}_{l}-\bm{\hat{W}}_{l} \bm{X}_{l}||_{F},$ (1) $$

where $\bm{W}_{l}$, $\bm{\hat{W}}_{l}$ are the original and compressed $l$-th linear layer, respectively, and $\bm{X}_{l}$ is the input feature activation.
For post-training pruning, to optimize the objective, Frantar & Alistarh (; ) utilize second-order gradient information to measure parameter importance and propose an efficient algorithm for computing the inverse Hessian matrix.
Sun et al. () evaluate weight importance simply by the product of weight magnitudes and the $L2$ norm of the corresponding input activation without requiring backpropagation.
Zhang et al. () propose the relative importance and activation metric, which integrates weight, input, and output activation.
They also utilize the channel permutation to minimize pruning loss under N:M semi-structured pruning.
Dong et al. () propose a search framework that employs the genetic algorithm to discover the optimal pruning metric for LLMs automatically.
Recently, several studies (Sung et al., ; Xu et al., ; Yin et al., ) indicate that layer-wise compression, which typically applies a uniform sparsity rate across all layers and evaluates weight importance within the layer, often results in suboptimal performance due to the lack of overall consideration.
Specifically, Xu et al. () proposes a differentiable pruning framework designed to search for optimal pruning rates for each layer.
Yin et al. () introduce outlier weighed layerwise sparsity, which relates the sparsity of each layer to the observed outliers in a proportional manner.

In the aforementioned post-training compression methods, calibration data is an indispensable component.
Calibration data is a small subset randomly sampled from unlabeled pretraining text.
Many methods (Frantar & Alistarh, ; Sun et al., ; Dettmers et al., ) claim their robustness to the quantity and distribution of calibration data, requiring only dozens or hundreds of samples with 2,048 sequence length.
However, this conclusion is based on the perplexity of certain datasets (such as Wikitext2), which does not fully reflect the true capabilities of the LLMs.
Even if perplexity shows no significant change, the compressed model may still experience substantial performance declines in downstream tasks (Jaiswal et al., ).
Williams & Aletras () observe in extensive experiments that the selection of calibration data in post-training pruning and quantization methods significantly impacts downstream tasks’ performance, especially post-training pruning, which is highly sensitive to calibration data.
Nevertheless, current research on calibration data remains under-explored, with few studies providing guidelines for selecting calibration data.
Khanal & Capone () suggest that using task-specific calibration data helps improve performance on specific downstream tasks.
Unlike their research, this paper aims to provide guidance on selecting calibration data to enhance the general capabilities of compressed models.

## 3 The Impact of Calibration Data for Pruning

Though Williams & Aletras () have noted that calibration data significantly impacts post-training pruning, there exist many open questions. How much does calibration data affect pruning performance? How does the amount of calibration data affect compressed model performance? What data sources are more suitable for calibration? We investigate these questions in this section.

### 3.1 Experimental details

#### Dense Model

To study the impact of data from different sources on post-training pruning methods, we need a comprehensive knowledge of the data used in model training.
We select the powerful and fully open-source LLM (including training data), DCLM-7B(^1^11https://huggingface.co/apple/DCLM-7B) (Li et al., ), as the dense model and conduct post-training pruning with different calibration data on it.

#### Post-training Pruning Methods

We choose three competitive and representative post-training pruning methods for evaluation: Wanda (Sun et al., ), DSnoT (Zhang et al., ) and OWL (Yin et al., ).
These methods apply to both unstructured and semi-structured pruning.

#### Calibration Data

We consider various data sources to be calibration data.
Following the mainstream works, the calibration data sources are all from the unlabeled pre-trained corpus:

- •
C4 (Raffel et al., )(^2^22https://huggingface.co/datasets/allenai/c4) is a widely used calibration data source, consisting of a large amount of multilingual web text filtered from Common Crawl.
We sample from the English training set.
- •
Wikipedia(^3^33https://huggingface.co/datasets/wikimedia/wikipedia) is a source of high-quality encyclopedic text. We use the first shard of the cleaned English version until 2023-11-01.
- •
Slimpajama(^4^44https://huggingface.co/datasets/DKYoon/SlimPajama-6B) is a cleaned and deduplicated version of RedPajama.
It is a high-quality pre-training corpus with diverse sources, including C4, ArXiv, GitHub, Books, etc.
- •
DCLM (Li et al., ) is the pre-training data of DCLM-7B model. It includes 2.6T tokens extracted from Common Crawl.
We sample from a subset(^5^55https://huggingface.co/datasets/robbiegwaldd/dclm-micro) of the DCLM.

Aside from the experiments in Section 3, we follow prior works and randomly sample 128 sequences with 2048 tokens as calibration data. To mitigate the impact of sampling randomness, all our experiments repeat the calibration data sampling 20 times with different random seeds and report the average performance.

#### Evaluation Tasks

Some pruning works focus on the perplexity of certain datasets while neglecting performance on various downstream tasks, which often fails to fully reflect the capabilities of compressed models.
Therefore, we choose multiple widely used and challenging commonsense reasoning tasks for evaluation, including BoolQ (Clark et al., ), Winogrande (Sakaguchi et al., ), PIQA (Bisk et al., ), Hellaswag (Zellers et al., ), ARC-e, ARC-c (Clark et al., ) and MMLU (Hendrycks et al., ).
For MMLU, we use a 5-shot setting, while all other tasks are evaluated in a zero-shot setting.
Our evaluation code is based on the lm-evaluation-harness repository(^6^66https://github.com/EleutherAI/lm-evaluation-harness).
We report the average performance of these seven tasks.

### 3.2 How much does calibration data affect pruning performance?

Figure: (a) sparsity ratio
Refer to caption: x4.png

In practical applications, evaluating and comparing the impact of different calibration data on pruned models inevitably consumes time and computational resources.
Therefore, we wonder how significant the impact of calibration data is on pruning performance and whether it’s worth our effort to seek optimal calibration data in research and practice.
We consider different sparsity ratios and sparsity types. Our experiments cover sparsity ratios ranging from 30% to 60%, and at 50% sparsity ratio, we further compare unstructured, 4:8 semi-structured, and 2:4 semi-structured sparsity types.

We use Wanda as an example to illustrate the model’s performance range, defined as the difference between the maximum and minimum values, after pruning with four calibration data sets, as shown in Figure 2.
More details on the performance of the different calibration data can be found in Figure 6 in Appendix A.
Specifically, at low sparsity ratios ($<$50%), the performance difference between different calibration data is minimal, less than 0.1%.
As sparsity increases, the impact of calibration data on pruning gradually amplifies, rising from a 0.5% difference at 50% sparsity to 2.3% at 60% sparsity.
Notably, as shown in Figure 6, inappropriate calibration data can even have a negative effect at moderate sparsity levels.
For instance, at 60% sparsity, using Wikipedia and Slimpajama as calibration data performs worse than magnitude pruning without any calibration data.
For sparsity types, we observe that as the sparsity pattern becomes more structured, the choice of calibration data becomes increasingly important, with the maximum difference reaching 1.5% to 1.8%.
We also report results on DSnoT and OWL in Appendix A.
Although different pruning methods exhibit varying performance, they show similar trends regarding the impact of calibration data.
Overall, at moderate to high sparsity ratios and with semi-structured sparsity types, different calibration data significantly affect the performance of pruned LLMs.
For all pruning methods, higher sparsity ratios and more structured sparsity types are key to achieving effective inference acceleration.
Therefore, paying more attention to the choice of calibration data is crucial.

### 3.3 Is Calibration Data from Different Sources Equally Robust to Data Amount?

Figure: (a) Wanda
Refer to caption: x6.png

Currently, almost all post-training pruning methods for LLMs have empirically demonstrated robustness in terms of the amount of calibration data they use.
Typically, model performance reaches a plateau when the data amount reaches 128, and more calibration data do not lead to additional performance gains.
We wonder whether these methods are equally robust to the amount of data for calibration data from different sources. Can certain calibration data that lead to poorer pruned models be improved by increasing the data amount?

We perform Wanda and DSnoT pruning on DCLM-7B in the 2:4 semi-structured pruning setting.
We randomly sample 64, 128, 256, 512, 1024, and 2048 samples from different data sources as calibration data.
Figure 3 shows how the performance of pruned models changes with increasing data amount using different calibration data.
We observe that the average performance of pruned models is robust to data amount, regardless of the calibration data source, with fluctuations of only 0.1%-0.2%.
Therefore, we cannot expect that increasing the amount of calibration data will narrow the performance gap between different calibration data.
Additionally, as the data amount increases, the standard deviation of the pruned model’s performance decreases.

### 3.4 What Calibration Data is Suitable for Pruning?

Since the choice of calibration data is crucial and cannot be improved by increasing the amount alone, we have to figure out what calibration data is more suitable for pruning.
We propose two reasonable hypotheses: (1) The more similar the calibration data is to the training data of the LLMs, the better the pruning performance. (2) The higher the quality of the calibration data, the better the pruning performance.

To verify the hypotheses, we perform three post-training pruning methods on DCLM-7B with various calibration data in the 2:4 semi-structured pruning setting. We report our results in Table 1.

**Table 1: Pruning performance of three pruning methods with four different sources of calibration data.**
| Method | C4 | Wikipedia | Slimpajama | DCLM |
| --- | --- | --- | --- | --- |
| Wanda | $62.52_{0.21}$ | $61.03_{0.21}$ | $62.31_{0.22}$ | $\textbf{62.88}_{0.20}$ |
| DSnoT | $61.71_{0.21}$ | $60.48_{0.24}$ | $61.20_{0.21}$ | $\textbf{62.25}_{0.22}$ |
| OWL | $63.40_{0.19}$ | $62.23_{0.19}$ | $63.10_{0.22}$ | $\textbf{63.60}_{0.16}$ |

Among these data, using DCLM from the training data as calibration data consistently achieves the best performance.
C4 and Slimpajama, which are also extracted from Common Crawl, perform slightly worse.
In contrast, the source of Wikipedia differs significantly from the other three datasets.
Although Wikipedia is recognized as high-quality data, it shows the worst performance, falling short of DCLM by 1.3% to 1.8%.
Therefore, we assert that the quality of calibration data is not the primary factor affecting pruning performance.
We further quantify the similarity between different calibration data and the training data. We utilize the MinHash-LSH algorithm to encode the 3-grams of C4, SlimPajama, Wikipedia, and DCLM, calculating their Jaccard similarities. The results show that the Jaccard similarity between C4 and DCLM is 0.070, SlimPajama is 0.016, and Wikipedia is 0.008. This indicates that C4 is the most similar to the training data, followed by SlimPajama, while Wikipedia has the lowest similarity. This ranking aligns with their performance as calibration data in pruning.
Therefore, we believe that the similarity of calibration data to the training data has a more significant impact on pruning performance than the quality of the calibration data. Training data or data similar to the training data is better suited as calibration data.
We conjecture that this may be due to LLMs learning the patterns in the training data better. Therefore, using data with similar patterns as calibration data during the pruning process can more accurately reflect the importance of model parameters.

## 4 Calibration Data Sampling Method

In the Section 3, our empirical study of the open-source DCLM-7B model demonstrates that selecting calibration data similar to the training data can yield better pruning performance.
However, in practical scenarios, the training data of many LLMs is not publicly available to users.
In this section, we will propose the “self-generating then sampling” strategy for sampling calibration data when the training data is unavailable.
Formally, given a dataset $\mathcal{D}$ as the source of calibration data and an LLM $\mathcal{M}$ pre-trained on an inaccessible dataset $\mathcal{D}_{t}$, we aim to sample $n$ instances from $\mathcal{D}$ as calibration data $\mathcal{D}_{c}$ that has a similar distribution to $\mathcal{D}_{t}$.

Recently, Xu et al. () disclosed that LLMs internalize patterns such as language structure, word distribution, and even commonsense knowledge from the training data during the training process.
Due to their auto-regressive nature, LLMs leverage these internalized patterns when predicting the next token, producing the generated text similar to the training data.
Thus, we propose using self-generated synthetic data as a proxy for the training data for calibration in post-training pruning.
Specifically, for a sample from the source of calibration data $\mathcal{D}$, we truncate the first $t$ tokens as the prefix and then allow the LLM $\mathcal{M}$ to generate contextually relevant subsequent content:

$$ $x_{i}\sim p_{\mathcal{M}}(x_{<i}),i=t\cdots N.$ (2) $$

After generating the data, we filter the synthetic data to prevent low-quality generated data from negatively impacting pruning effectiveness. We calculate each generated sample’s perplexity and filter the $k$% samples with the highest perplexity. Higher perplexity indicates that the patterns are not well-fitted by the LLM and may differ significantly from the training data, making them unsuitable as calibration data.

## 5 Experiments

### 5.1 Experimental Details

To evaluate the effectiveness of our proposed calibration data sampling method, we apply it to various LLMs, including DCLM-7B, LLaMA-2-7B, LLaMA-2-13B (Touvron et al., ) and LLaMA-3-8B (Dubey et al., ).
As described in Section 3.1, we use C4, Wikipedia, Slimpajama, and DCLM as baselines for calibration data, employing three post-training pruning methods: Wanda, DSnoT, and OWL, to prune the dense models.
In the main experiments, we report performance at the 60% sparsity ratio.
We follow previous work to evaluate the compressed LLMs’ language modeling and commonsense reasoning capabilities.
We do not use the Wikitext2 dataset, which is common in most papers for evaluating language modeling ability, as its similarity to Wikipedia may introduce bias when assessing the impact of different calibration data on language modeling ability.
Instead, we choose the Alpaca (Taori et al., ) dataset, distinct from all four calibration data sources, as our language modeling test data.

When replicating DSnoT and OWL, we follow the hyperparameter settings detailed in their papers.
During the self-generation process, we use Top-$k$ and Top-$p$ sampling to improve the diversity of the generated data. Specifically, we set the $p$-value to 0.95, the $k$-value to 50, and the temperature to 0.6 for the DCLM model, and 0.8 temperature for the LLaMA-series model. We apply the repetition penalty of 1.2 to avoid repeatedly generating low-quality fragments. To demonstrate the generalization of our self-generated calibration data, we randomly sample 5,000 examples from the Wikipedia data for generation, as Wikipedia performs poorly among the LLMs we used. In the filtering phase, we eliminate the top 20% of samples based on their perplexity.

### 5.2 Overall Performance

**Table 2: Pruning performance of different calibration data on DCLM-7B and LLaMA-2-7B in 60% sparsity ratio. The best performance method is indicated in bold. Wiki, Slim, and Syn are abbreviations for Wikipedia, SlimPajama, and our synthetic data, respectively.**
| Method | Data | Alpaca ($\downarrow$) | BoolQ | Winogrande | PIQA | Hellaswag | ARC-e | ARC-c | MMLU | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DCLM-7B |  |  |  |  |  |  |  |  |  |  |
| Wanda | C4 | 9.67 | 78.47 | 70.27 | 75.12 | 66.32 | 72.84 | 40.84 | 43.31 | 63.88 |
| Wiki | 9.99 | 72.05 | 68.40 | 74.33 | 64.79 | 73.14 | 39.91 | 42.20 | 62.12 |  |
| Slim | 9.76 | 78.56 | 70.16 | 74.27 | 65.07 | 72.37 | 39.94 | 43.40 | 63.40 |  |
| DCLM | 9.54 | 79.11 | 70.51 | 75.13 | 66.25 | 73.37 | 41.66 | 44.58 | 64.37 |  |
| Syn | 9.40 | 78.73 | 70.06 | 75.78 | 66.16 | 74.34 | 42.83 | 45.04 | 64.71 |  |
| DSnoT | C4 | 9.81 | 76.11 | 69.44 | 74.76 | 65.08 | 72.10 | 39.08 | 41.62 | 62.60 |
| Wiki | 10.16 | 69.97 | 68.08 | 73.95 | 63.23 | 72.09 | 38.69 | 41.63 | 61.09 |  |
| Slim | 9.87 | 75.58 | 69.21 | 73.80 | 63.88 | 71.37 | 38.63 | 42.25 | 62.10 |  |
| DCLM | 9.70 | 77.39 | 69.36 | 74.63 | 64.89 | 72.06 | 39.83 | 43.73 | 63.13 |  |
| Syn | 9.40 | 77.58 | 69.20 | 75.38 | 64.76 | 73.27 | 41.66 | 44.53 | 63.77 |  |
| OWL | C4 | 9.52 | 78.14 | 68.90 | 75.55 | 65.22 | 72.46 | 38.24 | 39.04 | 62.51 |
| Wiki | 9.96 | 75.27 | 67.11 | 74.25 | 63.07 | 73.01 | 38.35 | 38.75 | 61.40 |  |
| Slim | 9.59 | 78.09 | 68.69 | 74.56 | 64.00 | 72.35 | 37.95 | 39.84 | 62.21 |  |
| DCLM | 9.38 | 78.45 | 69.47 | 75.10 | 65.07 | 72.76 | 38.81 | 40.73 | 62.91 |  |
| Syn | 9.20 | 78.45 | 68.92 | 76.03 | 65.18 | 73.72 | 40.29 | 42.73 | 63.61 |  |
| LLaMA-2-7B |  |  |  |  |  |  |  |  |  |  |
| Wanda | C4 | 10.42 | 66.30 | 64.50 | 71.12 | 58.92 | 64.92 | 33.91 | 23.06 | 54.68 |
| Wikipedia | 10.42 | 66.80 | 63.84 | 70.55 | 56.69 | 64.78 | 34.23 | 22.94 | 54.26 |  |
| Slimpajama | 10.23 | 66.83 | 63.68 | 71.10 | 57.54 | 64.68 | 33.98 | 22.95 | 54.39 |  |
| DCLM | 9.88 | 68.92 | 64.25 | 71.15 | 58.72 | 64.81 | 33.98 | 23.65 | 55.07 |  |
| Syn | 9.62 | 68.29 | 64.40 | 71.49 | 58.89 | 64.73 | 35.41 | 24.01 | 55.32 |  |
| DSnoT | C4 | 10.88 | 65.25 | 64.04 | 71.22 | 57.15 | 64.40 | 32.82 | 23.45 | 54.05 |
| Wikipedia | 10.92 | 66.24 | 62.72 | 70.55 | 55.55 | 64.10 | 33.16 | 23.05 | 53.62 |  |
| Slimpajama | 10.76 | 65.66 | 63.66 | 70.82 | 56.17 | 64.43 | 32.51 | 23.15 | 53.77 |  |
| DCLM | 10.37 | 66.65 | 63.99 | 71.44 | 56.77 | 64.56 | 33.30 | 23.73 | 54.35 |  |
| Syn | 10.40 | 65.44 | 64.01 | 71.49 | 57.77 | 64.86 | 34.30 | 23.90 | 54.54 |  |
| OWL | C4 | 9.19 | 66.73 | 67.34 | 72.74 | 62.86 | 67.54 | 35.68 | 26.20 | 57.02 |
| Wikipedia | 9.30 | 66.50 | 66.05 | 71.82 | 61.90 | 67.57 | 35.89 | 26.07 | 56.54 |  |
| Slimpajama | 9.21 | 67.52 | 66.91 | 72.32 | 62.25 | 66.70 | 34.91 | 26.05 | 56.67 |  |
| DCLM | 9.08 | 69.79 | 67.94 | 72.39 | 62.73 | 67.06 | 35.85 | 26.45 | 57.46 |  |
| Syn | 9.13 | 69.85 | 66.38 | 73.18 | 62.86 | 67.89 | 35.07 | 26.34 | 57.37 |  |

We report main results in Table 2 and Table 5.
Overall, our self-generated synthetic calibration data outperforms other baseline calibration data in language modeling and commonsense reasoning tasks and is compatible with different pruning methods. On DCLM-7B, the synthetic calibration data improves performance in commonsense reasoning tasks by an average of 2.2% to 2.6% compared to the original Wikipedia data. Additionally, it surpasses the commonly used C4 calibration data, achieving an average increase of 0.8% to 1.2%. On LLaMA family models, the self-generated synthetic data significantly outperforms the original data, with improvements ranging from approximately 0.9% to 1.1%, and surpasses the C4 data by about 0.3% to 0.5%.
Surprisingly, the performance of the self-generated calibration data even exceeds that of calibration data sampled from the DCLM-7B training set, with an average improvement of 0.3% to 0.7%.
We think this may be due to certain patterns in the calibration data that LLMs have not adequately learned. Using these patterns as calibration data may misestimate the importance of parameters. In contrast, due to the nature of maximum likelihood training, self-generated calibration data typically generates patterns that LLMs have better learned, thus avoiding using underrepresented patterns as calibration data.

## 6 Discussion

### 6.1 Is the synthetic calibration data suitable for other pruning settings?

**Table 3: Pruning performance of different calibration data.**
| Setting | C4 | Wiki | Slim | DCLM | Syn |
| --- | --- | --- | --- | --- | --- |
| 50% | 69.43 | 69.07 | 69.26 | 69.62 | 69.64 |
| 65% | 57.22 | 53.97 | 56.10 | 58.14 | 58.11 |
| 4:8 | 66.27 | 64.82 | 66.17 | 66.28 | 67.02 |
| 2:4 | 62.52 | 61.03 | 62.31 | 62.88 | 63.61 |

We further validate the effectiveness of self-generated synthetic calibration data across more pruning settings. Table 3 illustrates the commonsense reasoning performance of DCLM-7B during Wanda pruning using different calibration data at unstructured 50% and 65% sparsity ratios, as well as semi-structured 4:8 and 2:4 settings. In all pruning settings, our synthetic calibration data either matches or exceeds the performance of the optimal calibration data from the training set DCLM. Notably, the synthetic data improve performance by approximately 0.8% in the two semi-structured pruning settings.
Since semi-structured pruning can achieve practical inference acceleration and advanced GPUs already support 2:4 sparse tensor cores.
Thus, we think the self-generated synthetic calibration data will effectively enhance the performance of pruned models in real-world deployment.

### 6.2 How Does Prefix Length Affect the Performance of Synthetic Data?

Figure: Figure 4: Wanda pruning performance using self-generated synthetic calibration data with different prefix lengths.
Refer to caption: x8.png

The prefix length during self-generation is a crucial hyperparameter. If the prefix is too short, the synthetic text is likely to be far from the semantics of the original text; if it is too long, the synthetic calibration data may retain excessive patterns from the original text. Therefore, it is essential to explore the selection of prefix length. Our experiments range from 0 to 1024 prefix lengths, where a prefix length of 0 indicates only a special token representing the start of the text. Figure 4 shows the trend of commonsense reasoning performance as the prefix length varies. Once there is a prefix, the performance exceeds that of the original calibration data. However, longer prefixes do not yield better results, as performance gradually declines with increased prefix length. The results indicate that using 1 to 4 tokens as a prefix is optimal. This suggests that semantic consistency with the original text is not critical in synthetic calibration data; instead, the key is to avoid retaining patterns that could have negative effects.

### 6.3 How does perplexity-based data filtering affect pruning performance?

**Table 4: Impact of perplexity-based data filtering.**
| Data | Alpaca ($\downarrow$) | Commonsense |
| --- | --- | --- |
| Wiki | 9.99 | 62.12 |
| w/o filter | - | 64.49 |
| 10% filter | 9.42 | 64.76 |
| 20% filter | 9.40 | 64.71 |
| 30% filter | 9.40 | 64.49 |
| 40% filter | 9.47 | 64.51 |

After generating synthetic data, we employ a simple perplexity-based method to filter low-quality data. Is this perplexity-based filtering method effective, and what should the filtering rate be? We conduct experiments on the DCLM-7B model. As shown in Table 4, even without any filtering strategy, the synthetic data outperforms the original data. The perplexity-based filtering has proved to be a simple yet effective approach, with the best pruning performance at a filtering rate of 10%-20%. As the filtering rate increases, pruning effectiveness gradually declines, ultimately matching the performance of the unfiltered data. Therefore, we recommend filtering only the outliers based on perplexity, as overly aggressive filtering may compromise the diversity of the calibration data, negatively impacting pruning performance.

### 6.4 Whether self-generated synthetic calibration data is more similar to training data?

Figure: Figure 5: The Min-50%++ score distribution of C4, Wikipedia, Slimpajama and self-generated synthetic data.
Refer to caption: x9.png

In Section 3.4, we assert that data similar to the training data is more suitable as calibration data for post-training pruning.
Based on the auto-regressive generation characteristics of LLMs, we propose using self-generated data as an approximation of the training data.
But is the self-generated synthetic data truly similar to the model’s training data than other calibration data?
We use an efficient and effective Min-K%++ method (Zhang et al., ) for measuring.
Min-K%++ notes that after maximum likelihood training, the probability distribution of the training data always lies at local maxima along the input dimensions. Therefore, for a given token sequence $(x_{<t},x_{t})$, if the sequence is belong to the training data, the $p(x_{<t},x_{t})$ should be higher than that of other candidate tokens in the vocabulary.
The Min-K%++ is formulated as follows:

$$ $\begin{gathered}W(x_{<t},x_{t})=\frac{logp(x_{t}|x_{<t})-\mu_{x_{<t}}}{\sigma_ {x_{<t}}},\\ \mathrm{Min}\mbox{-}\mathrm{K\%\mbox{++}}(x)=\frac{1}{|min\mbox{-}k\%|}\sum_{( x_{<t},x_{t})\in min\mbox{-}k\%}W(x_{<t},x_{t}),\end{gathered}$ (3) $$

where $\mu_{x_{<t}}$, $\sigma_{x_{<t}}$ is the expectation and standard deviation of the next token’s log probability given the prefix $x_{<t}$, respectively. $min\mbox{-}k\%$ refers to choosing the bottom $k\%$ of subsequences based on scores from the sequence $x$.
Thus, the higher a sample’s Min-K%++ score, the more likely it is to appear in the training data.
Figure 5 shows the Min-50%++ score distribution of C4, Wikipedia, Slimpajama and our self-generated synthetic data.
We can clearly observe that the self-generated synthetic data has higher Min-50%++ scores than the other calibration data.
It indicates that the self-generated synthetic calibration data is indeed similar to the training data, confirming the validity of using self-generated data as a proxy for the training data.

## 7 Conclusion and Future Work

In this paper, we highlight the critical role that calibration data plays in post-training pruning for LLMs. Through systematic exploration, we demonstrate that calibration data similar to the original training data leads to superior pruning performance.
To address the challenge of inaccessible training data in practical scenarios, we propose a self-generating synthetic calibration data strategy, which effectively samples suitable calibration data for LLMs.
Experimental results on the DCLM, LLaMA-2, and LLaMA-3 models demonstrate that our method significantly outperforms existing common-used calibration data.
We firmly believe that calibration data, as an essential part of post-training pruning, still holds significant potential for further research.

Our work still has some limitations that are worth exploring further. First, we do not fully optimize the hyperparameters when generating synthetic calibration data, such as using more advanced decoding strategies or refined filtering methods.
We believe that improving these details could further enhance the effectiveness of the synthetic calibration data. Second, our experiments are limited to unstructured and semi-structured pruning on 7B-13B LLMs. In future work, we will validate our method on 70B LLMs and in structured pruning scenarios.
Additionally, we will continue to explore how to synthesize high-quality instruction data as calibration data to help compress aligned LLMs.

## Appendix A More studies on different sparsity

Figure: (a) sparsity ratio: 0.3
Refer to caption: x10.png

Figure: (a) sparsity ratio: 0.3
Refer to caption: x16.png

Figure: (a) sparsity ratio: 0.3
Refer to caption: x22.png

## Appendix B More Results of synthetic calibration data

**Table 5: Pruning performance of different calibration data on LLaMA-2-13B in 60% sparsity ratio. The best performance method is indicated in bold. Wiki, Slim, and Syn are abbreviations for Wikipedia, SlimPajama, and our synthetic data, respectively.**
| Method | Data | Alpaca | BoolQ | Winogrande | PIQA | Hellaswag | ARC-e | ARC-c | MMLU | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Wanda | C4 | 8.99 | 77.36 | 68.68 | 75.45 | 66.51 | 69.18 | 39.74 | 26.80 | 60.53 |
| Wiki | 9.21 | 74.39 | 67.97 | 74.97 | 64.39 | 68.66 | 38.62 | 24.96 | 59.14 |  |
| Slim | 8.76 | 76.82 | 68.42 | 75.25 | 65.18 | 69.03 | 39.56 | 28.01 | 60.32 |  |
| DCLM | 8.73 | 77.50 | 68.37 | 75.16 | 66.34 | 69.95 | 40.15 | 27.98 | 60.78 |  |
| Syn | 8.73 | 77.06 | 68.68 | 75.19 | 66.25 | 70.03 | 40.19 | 29.06 | 60.92 |  |
| DSnoT | C4 | 9.03 | 77.16 | 66.60 | 74.92 | 65.76 | 69.81 | 38.45 | 25.73 | 59.77 |
| Wiki | 9.34 | 76.02 | 65.89 | 74.43 | 63.84 | 68.93 | 37.95 | 25.19 | 58.89 |  |
| Slim | 9.03 | 76.31 | 66.79 | 74.84 | 64.44 | 70.13 | 38.33 | 26.97 | 59.69 |  |
| DCLM | 9.04 | 77.22 | 67.56 | 74.52 | 65.38 | 69.94 | 38.72 | 26.97 | 60.04 |  |
| Syn | 8.96 | 77.09 | 67.64 | 74.54 | 65.33 | 70.29 | 39.68 | 27.08 | 60.23 |  |
| OWL | C4 | 7.56 | 78.92 | 70.02 | 75.95 | 69.12 | 70.90 | 41.14 | 32.75 | 62.69 |
| Wiki | 8.25 | 77.93 | 69.47 | 75.23 | 68.13 | 71.20 | 39.23 | 31.75 | 61.85 |  |
| Slim | 7.68 | 79.41 | 69.69 | 75.55 | 68.42 | 70.60 | 40.19 | 32.47 | 62.33 |  |
| DCLM | 7.33 | 79.85 | 70.23 | 75.57 | 69.21 | 71.62 | 40.48 | 33.77 | 62.96 |  |
| Syn | 7.35 | 79.05 | 69.61 | 76.50 | 69.11 | 71.51 | 41.55 | 31.19 | 62.65 |  |

**Table 6: Pruning performance of different calibration data on LLaMA-3-8B in 60% sparsity ratio. The best performance method is indicated in bold. Wiki, Slim, and Syn are abbreviations for Wikipedia, SlimPajama, and our synthetic data, respectively.**
| Data | BoolQ | Winogrande | PIQA | Hellaswag | ARC-e | ARC-c | MMLU | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C4 | 69.02 | 60.55 | 67.98 | 49.47 | 59.95 | 30.59 | 23.60 | 51.59 |
| Wiki | 66.82 | 59.02 | 67.40 | 47.14 | 59.79 | 29.67 | 24.14 | 50.57 |
| Slim | 66.86 | 60.11 | 67.53 | 48.07 | 59.38 | 29.96 | 23.52 | 50.77 |
| DCLM | 70.14 | 61.17 | 67.83 | 49.97 | 60.04 | 31.16 | 23.22 | 51.93 |
| Syn | 70.03 | 61.88 | 68.06 | 50.11 | 59.85 | 31.66 | 23.19 | 52.11 |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: - What are the main differences between this work and the work by [1]?

- The authors say that "We can clearly observe that the self-generated synthetic data has higher Min-50%++ scores than the other calibration data. It indicates that the self-generated synthetic calibration data is indeed similar to the training data, confirming the validity of
using self-generated data as a proxy for the training data.". The conclusion is not entirely clear to me, can you explain how to conclude that synthetic calibration data is similar to the training data in this figure?

- While the paper aims to enhance general capabilities, the impact of using domain-specific calibration data for pruning models intended for specialized tasks remains unclear. do the authors have any intuition for that?

[1] Miles Williams and Nikolaos Aletras. On the impact of calibration data in post-training quantization
and pruning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the
62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),
pp. 10100–10118, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
URL https://aclanthology.org/2024.acl-long.544.
soundness: 3
strengths: The paper effectively challenges the common assumption that post-training pruning methods are robust to the choice of calibration data. Recognizing the challenge of inaccessible training data, the paper introduces a "self-generating then sampling" strategy for constructing suitable calibration data. The paper provides a detailed examination of various aspects related to the self-generating calibration data strategy
summary: This paper investigates the role of calibration data in post-training pruning for large language models (LLMs). The authors find that calibration data similar to the training data yields better performance when pruning LLMs for model compression. As many training datasets for LLMs are inaccessible, the authors propose a strategy to create synthetic calibration data, which outperforms commonly used datasets in experiments. This strategy involves generating synthetic text using the LLM and then filtering out low-quality data. This synthetic data is more similar to the training data and ultimately leads to better performance for pruned LLMs.
weaknesses: While the paper shows a correlation between training data similarity and pruning performance, it doesn't explain why this connection exists. The paper's evaluation primarily centers on overall model performance. Investigating how calibration data affects the pruning of individual model components like attention heads or specific layers could be beneficial. This granular analysis would offer a more complete picture of how calibration data impacts different parts of the LLM.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: Suggestion (I): decompose or improve the table to highlight matching or exceeding the performance of using calibration data from the actual training set and exceeding the performance compared to calibration datasets belonging to other distributions. 

Suggestion (II): avoid redundancy in repeating the literature review and possibly summarize the questions in the introduction. In the literature review, the name of the technique corresponding to each citation could be mentioned as well.
Suggestion (III): improve the abstract to better reflect the outcomes of the paper and be easier to read.
Suggestion (IV): Mention somewhere that the paper will first proceed by answering the calibration data related questions and then propose a new a novel technique for its generations. Typically, one expects the main novel contribution to come first.
soundness: 3
strengths: The paper productively expands on prior work to answer unanswered follow up questions related to the influence of calibration data on pruning and delivers insightful findings through a set of reliable experiments.
It proposes a novel and intuitive approach for the synthesis of calibration data and evaluates it empirically and theoretically while experimentally justifying major hyperparameter choices. They show that the approach can improve by up to 2.6% over using an out-of-distribution calibration dataset.
The paper also clearly describes background, relevant pruning approaches, the problem statement and proposed approach for calibration data synthesis as well as experimental results.
summary: Large language models with numerous parameters substantially increase deployment and inference complexity and costs. To mitigate this, post-training parameter pruning can be used which exploits the fact that neural networks are often over-parametrized. It operates selectively removing redundant parameters while aiming to preserve performance as measured using a sample of calibration data.
The key contributions of this paper are: (i) a (plausibly) novel data synthesis strategy for calibration data, and (ii) an investigation into the effects of size, quality, and distribution of calibration data, across different pruning hyperparameters.
Additionally, the paper examines major hyperparameter choices within their strategy and perform additional analysis to show that their synthesis method generates data that is distributed similar to the training data.
weaknesses: The main results are not so well represented. In Table 2, the proposed calibration data synthesis approach frequently falls behind other sources of calibration data. It’s not highlighted in the table (e.g., using colors or otherwise) whether each source was present in the training set of the evaluated LLM. That is, it makes sense to have separate comparisons for the proposed approach with each of (i), data the model was not trained on and (ii), data the model was trained on, but these seem to be mixed up in one table making it hard to interpret the quality of the results by looking at the table. The statement “Overall, our self-generated synthetic calibration data outperforms other baseline calibration data in language modeling and commonsense reasoning tasks and…” is not well justified because the remaining of the paragraph focuses on Wikipedia and C4 and its not obvious from the table that it outperforms all sources consistently over all tasks. 

The paper involves some redundancies. For instance, the introduction as well as background seem to closely repeat the literature review. The questions are mentioned in the introduction then later again in section 3. Moreover, the choice of words in some of the sentences used is inadequate. For instance, the use of “value more” in “We fill this blank and surprisingly observe that the effects of calibration data even value more than designing advanced pruning strategies.” Take note as well that the paper does not convey that this “values more” than designing more advanced pruning strategies and that’s nontrivial to prove. Constructs such as “while different calibration data’s impact on pruning performance still lacks systematical exploration.” also make the abstract harder to read compared to if it was something like “while the impact of calibration data used has been…”.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. Where does Figure 6 reflect the results of magnitude-based pruning?
2. Are the conclusions and method presented in this paper applicable to LLM quantization?
soundness: 4
strengths: 1. This paper introduces a criterion and construction strategy for choosing calibration data in post-training pruning, supported by extensive experimental validation.
2. The authors conduct experiments on various LLMs and pruning methods, with multiple repetitions, to eliminate the effects of randomness.
3. The paper is well-organized, clearly presenting the empirical studies, methodology, experiments, and results, making it easy for readers to follow the authors' arguments.
summary: This paper studies the impact of calibration data in the post-training pruning of LLMs, which shows that calibration data significantly affects pruned model performance as pruning difficulty increases, surpassing the improvements from advanced pruning algorithms. The authors also find that using training data or data similar to it as calibration data significantly boosts pruned model performance. Since pre-training data is often unavailable for advanced LLMs, the paper proposes a strategy for self-generating calibration data. Extensive experiments on multiple LLMs and pruning methods confirm the effectiveness of the proposed synthetic calibration data.
weaknesses: 1. This paper only conducts experiments on unstructured and semi-structured pruning settings and does not validate the effectiveness of synthetic calibration data in more practical structured pruning.
2. The synthetic calibration data is not a method first proposed by the authors. A recent work by Shin et al.[1] also proposed synthetic calibration data. However, the authors do not discuss the differences between that work and the others.
3. This paper only uses data from Wikipedia to generate synthetic data. Why do you not validate the effectiveness of synthetic data generated from other sources?

[1] Shin, Sungbin, et al. "Rethinking Pruning Large Language Models: Benefits and Pitfalls of Reconstruction Error Minimization." arXiv preprint arXiv:2406.15524 (2024).

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: Could the authors provide further clarification on how efficient their proposed calibration data synthesis method is, e.g., what are the minimum data points it needs to generate for calibration?
soundness: 2
strengths: 1. **Originality:** In addition to the models included by Williams & Altetras (EMNLP2024), the authors also tested with DCLM-7B. This model is designed to showcase the effectiveness of systematic data curation. They propose a self-generating calibration data synthesis strategy. 
2. **Quality:** The paper provides a systematic exploration, supported by experimental results demonstrating how different calibration datasets affect the performance of pruned models.
3. **Clarity:** The writing is reasonably clear and easy to follow. The objective is straightforward. 
4. **Significance:** The findings have significant implications for practitioners in the field, although it has been highlighted by previous work already.
summary: This paper investigates the impact of calibration data in the pruning of large language models (LLMs). This work mainly repeats some work that has been done by Williams & Altetras (EMNLP2024), which investigates the impact of calibration data in the pruning and quantization of large language models. The authors present evidence that the quality and type of calibration data can impact pruning performance, at times more so than advanced pruning methods themselves, reflecting the results done by Williams & Altetras (EMNLP2024). They propose a self-generating calibration data synthesis strategy to create effective calibration datasets when access to training data is limited.
weaknesses: 1. **Lack of Novel Contribution:** The study builds on important findings by Williams & Aletras (EMNLP2024), and the findings are already been proven and previous work has done further comparing quantization and pruning. 
2. **Lack of downstream tasks experiments:** the authors only consider pruning performance and it does not necessarily reflect the downstream tasks. Previous work done by Williams & Altetras (EMNLP2024) has done a much more comprehensive evaluation of a wide range of downstream tasks. 
3. **No explanation on pruning performance:** The paper primarily evaluates "pruning performance," but fails to provide a clear explanation of this metric. It's unclear whether this refers to pruning error, signal-to-noise ratio (SNR), or another measure. The authors neither explain their calculation method nor cite a source for this metric. 
4. **Experimentation with Diverse Datasets:** The experiments predominantly focus on a narrow range of calibration datasets and models. Including a broader set of datasets could provide more generalizable results and strengthen the conclusions drawn about the effectiveness of their proposed methods.
5. **Validation or discussion of choices in methods:** There are some variables actually can be potentially impact the results, such as why 5000 samples from the Wikipedia data for generation, and why eliminate the top 20%.

Meta-review: Post-hoc pruning methods are a common and efficient approach for compressing pretraining language models; however, these approaches rely on some auxiliary data, or "calibration data," to estimate parameter importance. This work tests how the choice of calibration data affects the performance of post-hoc pruning methods on various downstream tasks. They find that the choice of data does affect performance: data similar to the original training data performs better, and this effect becomes stronger at higher rates of sparsity in the pruned model. The work then proposes a method for generating synthetic calibration data from the model directly (for the common case where an LLM's training data is not freely available), which is also effective for model pruning.

Strengths:
- This work provides several useful and actionable insights into how calibration data affect post-hoc pruning methods, and in my opinion after reading both papers, significantly builds on the insights in [1].
- The paper follows up on the presented analysis of calibration data to propose a general method for obtaining effective calibration data from the model, even when the underlying training distribution isn't known.
- The paper presents extensive experiments to support the findings in the paper across multiple models, data sources, and tasks. 

Weaknesses:
- The paper doesn't explain why data that is more similar to the training distribution works better, though they present a hypothesis and reasoning as to why this occurs in the paper. Experiments testing this and other analyses—such as the results on domain-specific calibration data—would flesh out the findings significantly.
- The paper is, at some points, confusing and redundant. The authors should ensure that the next version of the paper clears up any lingering points of confusion and redundancies raised by the reviewers.

The reviewers bring up other weaknesses—the presentation of the main results (9hYo), description of the paper's novelty and contributions (mY5n, 9hYo), and synthetic data source ablation (4EBf), etc.—that are addressed in the author response and the revised paper.

[1] Miles Williams and Nikolaos Aletras. On the impact of calibration data in post-training quantization and pruning. In ACL 2024.

Decision: Accept (Poster)