On the Impact of Calibration Data in Post-training Quantization and Pruning (2024)

Abstract: Quantization and pruning form the foundation of compression for neural networks, enabling efficient inference for large language models (LLMs). Recently, various quantization and pruning techniques have demonstrated remarkable performance in a post-training setting. They rely upon calibration data, a small set of unlabeled examples that are used to generate layer activations. However, no prior work has systematically investigated how the calibration data impacts the effectiveness of model compression methods. In this paper, we present the first extensive empirical study on the effect of calibration data upon LLM performance. We trial a variety of quantization and pruning methods, datasets, tasks, and models. Surprisingly, we find substantial variations in downstream task performance, contrasting existing work that suggests a greater level of robustness to the calibration data. Finally, we make a series of recommendations for the effective use of calibration data in LLM quantization and pruning.1

heading: 1 Introduction
text: x
Pre-trained Language Model
Calibration Data
Post-training Compression
Quantized Language Model
Pruned Language Model
In this paper, we present the first empirical study on the impact of calibration data used in posttraining LLM compression. We offer an extensive study with several quantization and pruning methods, across a range of tasks, datasets, and models. Surprisingly, we find that downstream task performance can vary substantially according to the selected calibration data. This contrasts existing work, which suggests a high level of robustness. Finally, we offer a series of recommendations for the effective use of calibration data.

heading: 2 Related Work

heading: 2.1 Model Compression
text: Model compression is the process of reducing the memory requirements of a model, usually enabling improved inference efficiency (Treviso et al., 2023). In the case of neural networks, model compression has a rich history, with origins in seminal work from LeCun et al. (1989).2 Quantization and pruning are two widely adopted approaches for model compression (Gholami et al., 2021; Hoefler et al., 2021). Pruning seeks to remove redundant weights, while quantization aims to represent the weights (and possibly activations) in lower precision. Applying these techniques to LLMs presents significant challenges, such as large-magnitude outlier features and high computational requirements (Dettmers et al., 2022; Frantar and Alistarh, 2023).
Post-training compression considers the scenario where a model must be compressed without retraining, instead relying upon a small amount of calibration data (Nagel et al., 2020; Hubara et al., 2021). While quantization and pruning are distinct methods, they are connected in a post-training setting via the layer-wise compression problem (Frantar and Alistarh, 2022). This involves selecting compressed weights for each layer that function closely to the original weights, with respect to the calibration data. More formally, given layer ℓ with weights Wℓ and inputs Xℓ, the aim is to minimize ||WℓXℓ−ŴℓXℓ||22 with respect to the compressed weights Ŵ. This is subject to a given compression constraint C(Ŵℓ) > C, which differs between quantization and pruning.
Recently, a range of approaches have been proposed for the layer-wise compression problem. SparseGPT (Frantar and Alistarh, 2023) enables LLM pruning up to 60% sparsity, with little im-
2We refer interested readers to Frantar and Alistarh (2022) for a detailed treatment of quantization and pruning.
pact upon perplexity. This sequentially prunes the weights in each column of the weight matrix using a series of Hessian inverses, followed by updating the remaining weights. Wanda (Sun et al., 2024) significantly improves upon the efficiency of SparseGPT, avoiding the expensive computation of the inverse Hessian. Instead, each layer is pruned according to the product of the weight magnitudes and ℓ2 norm of the input activations. For quantization, GPTQ (Frantar et al., 2023) enables storing weights in 3- or 4-bit precision, similarly relying upon inverse Hessian information. SpQR (Dettmers et al., 2024) further enables practically lossless quantization of LLMs through identifying and holding outlier weights in higher precision. Other approaches include SmoothQuant (Xiao et al., 2023), which enables 8-bit quantization of weights and activations through shifting the complexity from the activations to the weights.

heading: 2.2 Sampling Calibration Data
text: Calibration data is customarily sampled from either web text (Frantar et al., 2023; Frantar and Alistarh, 2023; Sun et al., 2024) or model pre-training data (Xiao et al., 2023; Dettmers et al., 2024). The aforementioned approaches use little calibration data, commonly 128 examples, each comprising 2,048 tokens. The addition of further calibration samples provides diminishing gains (Frantar and Alistarh, 2023; Sun et al., 2024). Frantar and Alistarh (2023) demonstrate that SparseGPT reaches an optimal point with only 128 examples, while Sun et al. (2024) demonstrate that Wanda requires as few as eight examples.
Post-training compression methods are generally understood to be robust to the exact distribution of calibration examples (Frantar and Alistarh, 2023; Sun et al., 2024; Dettmers et al., 2024). However, concurrent to our own study, there has been a variety of recent work examining the impact of calibration data. Lee et al. (2023) note that questionanswering performance can vary according to the calibration data sequence length when using GPTQ with 8-bit activation quantization. Moreover, Wu et al. (2023) find that GPTQ tends to overfit to the calibration data when assessing perplexity on several datasets. Separately, Jaiswal et al. (2024) suggest that curated calibration data could play an essential role in the design of improved LLM compression methods. Finally, Zeng et al. (2024) propose a novel calibration data sampling method to improve multilingual LLM compression.

heading: 2.3 Evaluating Compressed Models
text: The recent demand for LLM compression methods has motivated a body of work investigating their broader effects upon model behavior and performance. Gonçalves and Strubell (2023) examine the issue of social bias in pre-trained language models, finding that post-training quantization has a regularizing effect. In a separate direction, Chrysostomou et al. (2023) investigate the impact of pruning upon hallucinations. Kuzmin et al. (2023) offer the first study that directly compares pruning against quantization. Excluding setups with extreme compression ratios, they find that quantization outperforms pruning. Perhaps the closest work to our own is from Jaiswal et al. (2024), which explores the failure modes of compressed LLMs. Notably, they observe that model compression has a substantial impact upon knowledge-intensive tasks.

heading: 3 Methodology
text: Our aim is to investigate the robustness of LLM pruning and quantization methods to the calibration data. To this end, we experiment with four compression methods and nine LLMs (i.e. three different sizes from three model families). We vary only the calibration data, trialing five source datasets, each comprising ten distinct calibration sets. This provides a total of 1,800 compressed models. We then evaluate each model across 11 standard NLP tasks, resulting in a total of 19,800 model evaluations.

heading: 3.1 Model Compression
text: As it is impractical to exhaustively test every model compression method, we select four popular approaches. Unless otherwise stated, we match the compression setup from the original work. We list complete hyperparameters in Appendix A, Table 3.
Quantization. For quantization, we consider GPTQ (Frantar et al., 2023) and SpQR (Dettmers et al., 2024). We follow Dettmers et al. (2024) in using 4-bit weight quantization, which offers an optimal balance between model size and performance (Dettmers and Zettlemoyer, 2023). Since GPTQ was proposed prior to the release of LLaMA, we use the hyperparameters from the AutoGPTQ library used by Hugging Face Transformers.3
Pruning. For pruning, we trial SparseGPT (Frantar and Alistarh, 2023) and Wanda (Sun et al.,
3https://github.com/AutoGPTQ/AutoGPTQ
2024). Since the goal of model compression is typically to enhance inference efficiency, we avoid unstructured pruning, which is challenging to accelerate (Wen et al., 2016). Instead, we opt for semi-structured sparsity, enabling significant inference speedups. We use the 2:4 sparsity pattern that is required for GPU acceleration, resulting in a sparsity ratio of 50% (Mishra et al., 2021).

heading: 3.2 Evaluation Tasks
text: For a fair selection of evaluation tasks, we include all zero-shot tasks used in the original work of GPTQ, SpQR, SparseGPT, and Wanda. These are: (1) ARC easy (ARC-e) and (2) ARC challenge (ARC-c) sets (Clark et al., 2018); (3) BoolQ (Clark et al., 2019); (4) HellaSwag (Zellers et al., 2019); (5) LAMBADA (Paperno et al., 2016); (6) OpenBookQA (Banerjee et al., 2019); (7) PIQA (Bisk et al., 2020); (8) RTE (Dagan et al., 2006); (9) StoryCloze (Mostafazadeh et al., 2016); and (10) WinoGrande (Sakaguchi et al., 2021).
These zero-shot tasks primarily assess commonsense reasoning abilities, using binary, multiple choice, Cloze, and Winograd style questions. We report the evaluation set sizes in Appendix B.
Finally, following the evaluation protocol used by our selected model compression methods, we also report the model perplexity on the WikiText test set (Merity et al., 2017).

heading: 3.3 Calibration Data Sources
text: We explore a diverse variety of data sources to create our calibration sets. Following previous work (see Section 2), we include random web text and curated model pre-training datasets. To maintain the integrity of the zero-shot evaluations, we avoid using evaluation data as a source of calibration data, following Frantar et al. (2023). We therefore consider the following five data sources:
• C4 (Raffel et al., 2020): We use the Colossal Clean Crawled Corpus as our baseline, following Frantar et al. (2023). This consists of web text from Common Crawl, filtered with multiple heuristics to form a subset of clean English text.
• CNN-DM (Hermann et al., 2015; See et al., 2017): The CNN/Daily Mail corpus consists of news articles from both publishers, covering a broad range of topics. We include this corpus since it provides a focused yet distinct genre of high-quality long-form text.
• RedPajama (Together Computer, 2023): Since the pre-training data for LLaMA is not publicly available, we instead use an open-source reproduction. This mainly consists of web text (Common Crawl and C4), in addition to selected highquality sources such as arXiv, GitHub, Stack Exchange, and Books3 (Gao et al., 2020).
• RefinedWeb (Penedo et al., 2023): Assembled through stringent filtering and deduplication of Common Crawl, RefinedWeb is a curated model pre-training dataset. Penedo et al. (2023) find that models trained with this dataset exhibit superior zero-shot generalization abilities compared to alternatives such as The Pile (Gao et al., 2020).
• Wikipedia: We select English Wikipedia as a source of high-quality encyclopedic text. Specifically, we use a preprocessed and cleaned version of the dump from 2022-03-01, prior to the “knowledge cutoff” of our selected models.

heading: 3.4 Models
text: We use three popular ‘open-source’ LLM families: (1) LLaMA (Touvron et al., 2023); (2) Vicuna (Chiang et al., 2023); and (3) OPT (Zhang et al., 2022). This includes base models (LLaMA and OPT) as well as instruction-tuned models (Vicuna).
Additionally, we select these three LLM families since they offer models of comparable sizes. For LLaMA and Vicuna, we select the 7B, 13B, and 33B model sizes. In the case of OPT, we select the closest comparable sizes (6.7B, 13B, and 30B).

heading: 3.5 Implementation Details
text: To create the calibration sets, we use the publicly available version of each source dataset from Hugging Face Datasets (Lhoest et al., 2021). Similarly, we use the weights and implementation of each model from Hugging Face Transformers (Wolf et al., 2020). To ensure that our model evaluations are robust and reproducible, we use the EleutherAI Language Model Evaluation Harness (Gao et al., 2021). Each model is compressed and evaluated using a single NVIDIA A100 (SXM 80GB) GPU.

heading: 3.6 Data Sampling
text: Previous work has demonstrated that increasing the number of calibration examples offers diminishing gains in language modeling performance (Frantar and Alistarh, 2023; Sun et al., 2024). We therefore follow existing work and randomly sample 128 calibration examples, each consisting of 2,048 tokens
(Frantar et al., 2023; Frantar and Alistarh, 2023; Sun et al., 2024). This offers a total of 262,144 tokens in each calibration set. We provide a detailed analysis of how the quantity of calibration examples impacts performance in Section 4.
The use of random sampling avoids selection bias and ensures that each calibration set is representative of the source dataset. Similarly, we sample without replacement, to ensure that each calibration example appears only once. To examine the variability introduced by random sampling, we repeat the sampling process to create ten nonoverlapping calibration sets for each source dataset. This provides a total of 50 distinct calibration sets.
Due to the vast size of C4, we follow Frantar et al. (2023) in sampling data from the first shard only. We use the same strategy for RefinedWeb, although for RedPajama we use the existing 1B token extract.4

heading: 4 Results & Analysis
text: Our principal aim is to investigate the impact of calibration data upon LLM performance. To this end, we examine the model performance across calibration sets, their source datasets, and evaluation tasks. Additionally, we explore trends in overall performance throughout the different configurations.
Performance can vary between calibration sets. Figure 2 shows the distribution of accuracy across ten calibration sets sampled from the same source dataset (C4) for each model in the LLaMA family. Across several tasks, we observe a substantial level of dispersion. In the most extreme cases with LLaMA-7B, we observe that the accuracy with SparseGPT ranges from 52.7% to 61.7% for RTE, and from 66.4% to 73.0% for BoolQ. We observe comparable levels of dispersion for the Vicuna and OPT model families, presented in Appendix C (Figures 5 and 6, respectively).
The degree of dispersion differs between tasks. We also observe that the dispersion of accuracy across calibration sets is elevated for certain tasks (Figure 2). For example, BoolQ and RTE present the highest levels of dispersion, with ranges up to 6.6% and 9.4%, respectively. However, we note that RTE has considerably fewer examples than the other tasks (Appendix B, Table 4). Consequently, RTE may offer a less stable estimate of dispersion.
4https://huggingface.co/datasets/ togethercomputer/RedPajama-Data-1T-Sample
Certain data sources may outperform others. In the case of pruning, we observe that some calibration data sources achieve greater zero-shot performance for the same model. Table 1 shows the mean zero-shot accuracy and standard deviation across ten calibration sets sampled from each source dataset. For SparseGPT, we observe that RefinedWeb reaches the highest level of accuracy for eight of the nine models. In contrast, we observe that Wikipedia achieves the lowest performance across eight of the nine models. Finally, we note that Vicuna-7B demonstrates the greatest range between highest and lowest performing source dataset, with a mean zero-shot accuracy of 52.7% for CNN-DM versus 56.3% for RefinedWeb.

heading: No data source consistently outperforms others.
text: Although we observe that RefinedWeb generally performs well across various models, this is not universally the case. For example, LLaMA-7B pruned with Wanda achieves the highest mean zero-shot accuracy (52.7%) with RedPajama, yet the lowest (52.2%) with RefinedWeb (Table 1). Additionally, we emphasize that the difference between the source datasets with the highest and lowest mean zero-shot accuracy is often very small. Therefore, it is frequently unclear whether a source dataset outperforms its counterparts. This is especially relevant to quantization, where the difference is typically lowest. For example, this is usually around 0.2% for SpQR, across all model families and sizes.
LLaMA Vicuna OPT
Method Dataset 7B 13B 33B 7B 13B 33B 6.7B 13B 30B
- - 64.3 66.7 69.1 64.3 66.5 68.8 57.4 58.3 60.4
GPTQ
C4 63.2 0.2 66.2 0.2 68.5 0.2 62.9 0.2 66.1 0.2 68.3 0.2 56.6 0.4 57.7 0.3 59.6 0.2 CNN-DM 63.1 0.3 66.1 0.2 68.5 0.2 62.7 0.3 66.1 0.2 68.4 0.2 56.5 0.3 57.8 0.3 59.6 0.1 RedPajama 63.2 0.3 66.2 0.2 68.6 0.2 63.2 0.2 66.2 0.2 68.4 0.2 56.7 0.3 57.9 0.3 59.8 0.3 RefinedWeb 63.3 0.3 66.1 0.2 68.6 0.2 63.1 0.2 66.0 0.2 68.4 0.2 56.5 0.4 57.8 0.2 59.7 0.2 Wikipedia 63.2 0.2 66.0 0.2 68.6 0.2 62.9 0.3 66.0 0.1 68.1 0.2 56.7 0.3 57.9 0.2 59.7 0.2
SpQR
C4 64.1 0.2 66.4 0.1 69.0 0.2 63.9 0.1 66.4 0.2 68.7 0.1 57.1 0.1 58.3 0.1 60.3 0.2 CNN-DM 63.8 0.2 66.5 0.2 69.0 0.1 64.1 0.1 66.3 0.1 68.7 0.2 57.5 0.2 58.3 0.3 60.3 0.1 RedPajama 64.0 0.1 66.4 0.2 68.9 0.1 64.1 0.2 66.5 0.2 68.7 0.1 57.2 0.2 58.3 0.2 60.3 0.2 RefinedWeb 64.1 0.2 66.4 0.2 69.0 0.1 64.0 0.2 66.5 0.2 68.7 0.2 57.1 0.1 58.3 0.1 60.3 0.2 Wikipedia 63.9 0.2 66.5 0.2 69.0 0.1 64.0 0.2 66.3 0.2 68.8 0.1 57.4 0.1 58.3 0.2 60.3 0.2
SparseGPT C4 53.9 0.4 58.2 0.3 63.7 0.5 55.7 0.4 59.5 0.4 64.9 0.3 52.9 0.3 54.8 0.3 57.3 0.3 CNN-DM 52.5 0.4 57.5 0.2 63.0 0.4 52.7 0.6 58.9 0.3 63.8 0.3 51.8 0.3 54.0 0.4 56.3 0.2 RedPajama 53.3 0.4 57.6 0.2 63.2 0.4 55.5 0.5 59.3 0.4 64.5 0.3 52.6 0.2 54.4 0.3 56.9 0.1 RefinedWeb 54.0 0.4 58.2 0.2 63.4 0.6 56.3 0.4 59.9 0.3 65.0 0.5 53.0 0.2 55.1 0.1 57.5 0.1 Wikipedia 52.0 0.5 56.3 0.3 61.8 0.4 54.3 0.6 57.6 0.6 62.8 0.3 51.2 0.2 52.7 0.2 55.2 0.2
Wanda
C4 52.4 0.3 56.2 0.2 63.4 0.3 53.4 0.4 56.9 0.2 64.0 0.2 50.6 0.2 52.6 0.2 54.6 0.3 CNN-DM 52.4 0.2 56.3 0.1 63.1 0.1 53.7 0.3 56.8 0.2 63.7 0.1 49.6 0.2 51.3 0.4 54.0 0.3 RedPajama 52.7 0.2 56.3 0.1 63.1 0.1 53.3 0.2 57.0 0.1 63.8 0.2 50.5 0.1 52.4 0.1 54.8 0.2 RefinedWeb 52.2 0.3 56.3 0.2 63.2 0.1 53.4 0.3 57.1 0.1 63.7 0.2 50.8 0.1 52.7 0.1 55.1 0.3 Wikipedia 52.4 0.2 56.1 0.2 62.6 0.2 53.4 0.3 57.0 0.1 63.5 0.2 48.9 0.2 50.5 0.2 53.1 0.2
Additional calibration examples offer diminishing gains in zero-shot performance. Figure 3 also presents the mean zero-shot accuracy across tasks (Section 3.2) for LLaMA-7B compressed using an increasing number of calibration examples. For each compression method, we observe an identical trend to perplexity, with performance plateauing after only a small number of examples. Remarkably, GPTQ, SpQR, and Wanda demonstrate a consistent mean zero-shot accuracy with only a few examples. However, SparseGPT performance continues to improve beyond 128 examples, with quadruple the number of examples offering a 0.4% increase in mean zero-shot accuracy. This suggests that SparseGPT is best used with an expanded calibration set to maximize performance. However, it should be noted that increasing the number of calibration examples also increases the computational cost of compression (Frantar and Alistarh, 2023). Perplexity can be challenging to interpret. Prior work relies on perplexity to assess robustness to the calibration data (Frantar and Alistarh, 2023; Sun et al., 2024). In practice, this consists of compressing a given model using a few different calibration sets, sampled from the same source dataset. Then, the difference in perplexity is measured using a standard dataset (often WikiText). Although this is an entirely reasonable approach, it can be challenging to interpret these results in the context of downstream task performance. Table 2, shows the WikiText test set perplexity and standard deviation across ten distinct calibration sets sampled from each source dataset. For example, applying SparseGPT to Vicuna-7B with the calibration sets from CNN-DM achieves a seemingly robust perplexity of 12.72±0.18. In contrast, the same models achieve 66.7%±4.7 on BoolQ, with accuracy ranging from 57.0% to 71.6%.
SparseGPT typically outperforms Wanda. Although the purpose of our study is not to compare model compression methods, we observe that SparseGPT mostly outperforms Wanda. Considering the C4 calibration data source that was used in both original works, we observe that SparseGPT achieves a higher mean zero-shot accuracy across all models. Moreover, we observe that the mean zero-shot accuracy achieved by SparseGPT is 2.2- 2.6% higher across the OPT family, compared to 0.3-2.0% and 0.9-2.6% for LLaMA and Vicuna, respectively (Table 1). Although Sun et al. (2024) report mixed results for Wanda versus SparseGPT in the 2:4 semi-structured setting, we were somewhat surprised by the consistency and margin that SparseGPT outperforms Wanda in our experiments.

heading: 5 Recommendations
text: Our results suggest that calibration data used in post-training quantization and pruning can influence LLM performance. Consequently, we offer several recommendations concerning the use of calibration data to researchers and practitioners alike:
1. Releasing calibration data: Research relying upon calibration data for post-training model compression should release the data. As calibration data can ultimately affect model performance, this serves to improve reproducibility by removing a source of randomness.5
2. Varying calibration data: Evaluating downstream task performance across several calibration sets during model development can offer an insight into the sensitivity to the calibration data. This provides an opportunity to identify any issues with the compression setup or calibration data that may compromise model performance.
3. Inspecting calibration data: Randomly sampled calibration data can be manually inspected, to remove anomalous examples. Given the small number of examples comprising the calibration set, this may offer a practical way to maximize the performance of the compressed model.

heading: 6 Conclusion
text: In this paper, we presented the first extensive empirical study on calibration data for LLM quantization and pruning. We examined the downstream task performance across a variety of models, compression methods, and calibration data sources. Our results suggest that calibration data can substantially influence the performance of compressed LLMs. We supplement our findings with several recommendations for the effective use of calibration data.
We hope that our work will inspire further research surrounding the use of calibration data in LLM compression, an area which has seen limited attention. For future work, we are particularly interested in exploring how choices in the training protocol may influence the sensitivity to calibration data in LLM compression (Ahmadian et al., 2023).
5Many existing studies have released code to generate the calibration data, but not the calibration data itself. This is not robust since using a seed for random sampling does not guarantee consistency across software or platform versions.
Limitations
In this study, we rely entirely on English-language models, evaluation tasks, and calibration data. We operate under the assumption that the performance of the LLM compression methods we trial is generally language-agnostic. Nevertheless, we recognize the importance of linguistic diversity. We therefore hope to explore the performance of LLM compression methods across diverse language families (including low resource settings) in a future work.

heading: Acknowledgments
text: We are grateful to George Chrysostomou, Huiyin Xue, and the anonymous reviewers for their invaluable feedback. MW is supported by the Centre for Doctoral Training in Speech and Language Technologies (SLT) and their Applications funded by UK Research and Innovation grant EP/S023062/1. NA is supported by EPSRC grant EP/Y009800/1, part of the RAI UK Keystone projects.

heading: A Hyperparameters
text: Table 3 lists the complete hyperparameters used for each compression method. For SparseGPT, Wanda, and SpQR, these are taken from the original work. In the case of GPTQ, which was proposed prior to LLaMA, these are derived from AutoGPTQ.

heading: B Datasets
text: Table 4 shows the number of examples for each evaluation task from the appropriate split (either test or validation).

heading: C Complete Results
text: In addition to the summarized results (Section 4), we provide a complete tabulation of results for all models, tasks, compression methods, and source datasets. We present the mean accuracy and standard deviation across ten distinct calibration sets for each model family: LLaMA 7B, 13B, and 33B (Tables 5, 6 and 7); Vicuna 7B, 13B, and 33B (Tables 8, 9, and 10); and OPT 6.7B, 13B, and 30B (Tables 11, 12, and 13). Finally, we present the distribution of accuracy across calibration sets sampled from C4 (Figure 2) for the Vicuna and OPT model families in Figures 6 and 5, respectively.
Method Dataset ARC-e ARC-c BoolQ HS LMBD OBQA PIQA RTE SC WG Mean
- - 75.3 41.9 75.1 57.0 67.8 34.4 78.7 66.4 76.7 69.9 64.3
GPTQ
C4 74.4 0.3 40.7 0.2 74.4 0.7 56.1 0.1 65.4 0.6 32.0 0.7 78.3 0.2 65.6 2.1 76.0 0.3 69.2 0.3 63.2 0.2 CNN-DM 74.3 0.3 40.9 0.5 74.1 0.5 56.1 0.1 64.7 0.6 32.1 0.9 78.3 0.3 65.7 1.3 75.8 0.2 69.2 0.8 63.1 0.3 RedPajama 74.1 0.4 40.5 0.7 74.0 0.5 56.2 0.1 65.3 0.6 31.7 0.8 78.4 0.2 66.5 2.6 76.0 0.4 68.8 0.5 63.2 0.3 RefinedWeb 74.0 0.5 40.9 0.6 74.6 0.6 56.3 0.1 65.6 0.4 32.0 1.0 78.4 0.3 65.4 2.2 76.0 0.2 69.5 0.4 63.3 0.3 Wikipedia 74.4 0.3 41.4 0.9 74.4 0.5 56.3 0.2 65.0 0.4 31.7 0.8 78.5 0.2 64.6 2.3 76.2 0.4 69.5 0.4 63.2 0.2
SpQR
C4 75.6 0.2 41.7 0.2 74.9 0.5 56.7 0.1 67.2 0.3 33.9 0.6 78.6 0.2 65.7 1.4 76.6 0.3 70.0 0.4 64.1 0.2 CNN-DM 75.2 0.2 41.7 0.4 74.5 0.5 56.7 0.1 66.9 0.1 33.4 0.6 78.5 0.2 65.4 1.0 76.3 0.2 69.7 0.5 63.8 0.2 RedPajama 75.3 0.3 41.8 0.4 74.8 0.5 56.7 0.1 67.1 0.4 33.0 0.4 78.6 0.2 66.1 1.0 76.5 0.2 70.2 0.4 64.0 0.1 RefinedWeb 75.3 0.4 41.5 0.3 74.7 0.6 56.7 0.1 67.3 0.3 33.9 0.5 78.6 0.3 66.1 1.4 76.7 0.3 70.1 0.4 64.1 0.2 Wikipedia 75.3 0.3 41.5 0.4 74.7 0.8 56.7 0.1 67.2 0.3 33.0 0.7 78.4 0.3 66.0 1.3 76.8 0.2 69.9 0.4 63.9 0.2
SparseGPT C4 62.0 1.0 28.8 0.7 70.4 1.9 43.3 0.3 48.7 0.7 23.4 0.7 70.9 0.8 57.0 2.4 70.2 0.4 64.7 1.0 53.9 0.4 CNN-DM 58.3 1.1 27.0 0.4 69.5 1.3 42.2 0.2 47.9 1.1 22.1 1.0 69.0 0.4 54.4 3.2 69.8 0.5 64.7 0.9 52.5 0.4 RedPajama 61.4 0.8 28.4 0.7 70.1 2.6 42.2 0.4 48.5 1.6 22.9 0.9 69.5 0.4 57.0 3.1 69.3 0.4 64.2 0.9 53.3 0.4 RefinedWeb 60.9 0.8 28.1 0.8 70.5 1.9 43.0 0.5 51.3 1.2 22.5 1.1 70.8 0.6 57.2 2.1 70.1 0.7 65.3 0.8 54.0 0.4 Wikipedia 60.7 1.1 28.6 0.7 65.7 1.7 41.2 0.4 45.5 1.7 23.2 0.7 69.3 0.5 54.4 3.2 67.9 0.4 63.6 0.6 52.0 0.5
Wanda
C4 60.6 0.5 27.9 0.5 69.5 0.4 42.0 0.1 44.3 0.4 24.3 0.5 70.5 0.4 53.8 1.9 68.6 0.3 63.0 0.6 52.4 0.3 CNN-DM 59.5 0.6 27.0 0.3 69.3 0.4 41.9 0.1 44.1 0.3 23.8 0.6 69.8 0.3 57.3 1.5 67.9 0.3 63.1 0.5 52.4 0.2 RedPajama 61.2 0.5 28.0 0.4 69.3 0.4 41.7 0.1 44.5 0.3 24.3 0.3 70.1 0.2 57.0 1.6 68.4 0.2 62.2 0.5 52.7 0.2 RefinedWeb 60.6 0.5 27.8 0.4 69.1 0.2 41.9 0.2 45.0 0.3 24.0 0.5 70.4 0.5 52.6 2.1 68.4 0.2 62.2 0.5 52.2 0.3 Wikipedia 60.0 0.5 27.8 0.6 69.0 0.4 41.6 0.2 43.4 0.3 24.0 0.6 69.8 0.2 58.2 1.2 68.0 0.4 62.4 0.4 52.4 0.2
GPTQ
C4 77.0 0.3 46.1 0.4 77.6 0.8 59.6 0.1 70.3 0.4 33.5 0.6 79.1 0.2 68.3 1.5 78.4 0.3 72.2 0.5 66.2 0.2 CNN-DM 76.9 0.4 46.0 0.7 77.8 0.4 59.4 0.2 70.4 0.3 33.1 0.6 78.9 0.2 68.0 2.0 78.4 0.3 72.4 0.4 66.1 0.2 RedPajama 77.2 0.4 46.2 0.7 77.4 0.6 59.6 0.2 70.3 0.2 33.5 0.6 79.1 0.2 68.0 1.9 78.4 0.2 72.4 0.5 66.2 0.2 RefinedWeb 77.0 0.4 46.2 0.4 77.6 0.6 59.5 0.1 70.4 0.4 33.6 0.3 79.2 0.3 67.3 1.5 78.3 0.3 72.4 0.4 66.1 0.2 Wikipedia 76.9 0.4 46.1 0.4 77.0 0.7 59.4 0.2 70.4 0.6 33.4 0.5 79.0 0.3 67.4 1.8 78.1 0.3 72.2 0.5 66.0 0.2
SpQR
C4 76.8 0.2 46.0 0.5 77.6 0.6 59.7 0.1 70.7 0.3 33.3 0.4 79.1 0.2 69.9 0.8 78.6 0.3 72.8 0.5 66.4 0.1 CNN-DM 77.0 0.2 46.1 0.4 78.0 0.4 59.7 0.1 70.9 0.3 33.2 0.5 79.0 0.2 69.6 1.7 78.4 0.3 72.6 0.4 66.5 0.2 RedPajama 76.8 0.2 46.3 0.3 77.8 0.2 59.6 0.1 70.8 0.2 33.3 0.4 79.1 0.2 69.3 1.6 78.3 0.2 72.9 0.5 66.4 0.2 RefinedWeb 77.0 0.3 46.3 0.4 77.9 0.4 59.7 0.1 70.9 0.3 33.6 0.4 79.2 0.2 68.7 1.6 78.4 0.3 72.8 0.4 66.4 0.2 Wikipedia 77.0 0.2 46.2 0.5 78.1 0.4 59.7 0.1 70.9 0.4 33.7 0.5 79.0 0.1 69.4 1.7 78.5 0.2 72.8 0.4 66.5 0.2
SparseGPT C4 68.1 0.7 35.3 1.0 72.7 1.9 48.2 0.2 58.5 0.6 26.7 0.4 74.4 0.2 54.8 0.5 73.8 0.3 69.2 0.5 58.2 0.3 CNN-DM 65.5 0.6 34.4 0.7 72.8 1.5 47.7 0.2 58.9 0.5 26.8 0.7 73.4 0.7 53.5 0.5 73.5 0.4 68.5 0.7 57.5 0.2 RedPajama 67.6 0.8 34.7 0.8 72.7 1.4 47.1 0.4 58.5 0.6 26.2 0.8 73.1 0.5 54.5 1.1 73.2 0.3 68.6 0.7 57.6 0.2 RefinedWeb 67.4 0.6 34.7 0.9 73.1 0.9 48.0 0.3 61.6 0.8 26.5 0.8 74.0 0.6 54.7 1.0 73.7 0.3 68.7 0.6 58.2 0.2 Wikipedia 66.6 0.9 33.3 0.9 69.9 1.7 46.4 0.2 53.9 0.7 26.8 0.9 72.5 0.6 53.1 0.4 72.2 0.3 67.8 0.9 56.3 0.3
Wanda
C4 66.1 0.5 33.6 0.5 71.3 0.5 46.8 0.1 49.2 0.5 26.9 0.5 73.9 0.2 53.8 0.4 72.6 0.3 67.9 0.6 56.2 0.2 CNN-DM 66.2 0.4 33.5 0.5 71.9 0.4 46.8 0.1 50.9 0.3 27.0 0.6 73.1 0.3 53.5 0.3 72.8 0.2 67.8 0.5 56.3 0.1 RedPajama 66.2 0.4 33.9 0.5 71.6 0.5 46.5 0.1 49.6 0.3 27.1 0.5 73.2 0.3 54.4 0.4 72.4 0.2 68.0 0.4 56.3 0.1 RefinedWeb 65.9 0.4 33.5 0.5 71.9 0.4 46.7 0.2 49.4 0.4 27.0 0.6 73.6 0.3 54.2 0.2 72.5 0.3 68.1 0.6 56.3 0.2 Wikipedia 66.8 0.4 34.1 0.2 71.2 1.0 46.2 0.1 48.9 0.5 26.4 0.5 73.1 0.2 53.9 0.5 72.4 0.2 67.8 0.5 56.1 0.2
Table 6: Mean accuracy across ten calibration sets for LLaMA-13B, with standard deviation denoted in subscript.
Method Dataset ARC-e ARC-c BoolQ HS LMBD OBQA PIQA RTE SC WG Mean
- - 80.4 52.8 82.6 63.3 73.4 36.0 81.0 66.8 79.1 75.9 69.1
GPTQ
C4 80.1 0.3 50.9 0.4 81.9 0.5 62.8 0.1 72.8 0.5 35.4 1.0 80.7 0.3 66.9 1.3 79.3 0.2 74.5 0.6 68.5 0.2 CNN-DM 79.7 0.5 50.4 0.4 82.2 0.7 62.6 0.1 73.0 0.4 35.2 0.7 80.7 0.2 67.2 1.9 79.2 0.3 75.2 0.4 68.5 0.2 RedPajama 79.9 0.4 50.7 0.8 81.7 0.7 62.7 0.1 73.1 0.2 35.2 0.5 80.6 0.2 68.0 1.5 79.1 0.2 74.9 0.4 68.6 0.2 RefinedWeb 79.5 0.3 50.7 0.4 82.3 0.4 62.7 0.1 72.9 0.5 35.2 0.7 80.8 0.3 68.1 1.5 79.3 0.2 74.9 0.3 68.6 0.2 Wikipedia 80.1 0.4 50.6 0.7 82.0 0.7 62.7 0.1 73.0 0.4 35.0 0.6 80.7 0.3 67.4 1.7 79.4 0.4 75.4 0.4 68.6 0.2
SpQR
C4 80.3 0.2 51.8 0.4 82.9 0.2 63.2 0.1 73.5 0.3 35.6 0.6 80.9 0.2 67.2 0.9 79.3 0.3 75.2 0.6 69.0 0.2 CNN-DM 80.4 0.3 51.7 0.5 82.7 0.3 63.1 0.1 73.4 0.2 35.8 0.6 81.0 0.2 67.7 0.5 79.2 0.1 75.3 0.5 69.0 0.1 RedPajama 80.4 0.1 51.9 0.5 82.6 0.3 63.1 0.1 73.4 0.2 35.6 0.5 80.9 0.2 66.8 0.8 79.2 0.1 75.3 0.3 68.9 0.1 RefinedWeb 80.6 0.2 51.8 0.4 82.7 0.3 63.1 0.1 73.6 0.2 35.7 0.5 80.9 0.2 67.3 0.4 79.2 0.1 75.1 0.2 69.0 0.1 Wikipedia 80.5 0.3 51.7 0.3 82.6 0.3 63.2 0.1 73.4 0.3 35.9 0.3 80.9 0.2 67.4 1.0 79.2 0.2 75.2 0.3 69.0 0.1
SparseGPT C4 75.1 0.8 42.7 0.9 76.7 1.2 53.4 0.3 66.9 0.6 30.9 0.4 77.2 0.3 63.8 3.2 76.8 0.4 73.2 0.6 63.7 0.5 CNN-DM 73.4 0.3 40.8 0.6 76.7 1.5 53.1 0.2 67.3 0.5 30.2 1.1 76.5 0.4 63.5 2.0 76.5 0.5 72.1 0.9 63.0 0.4 RedPajama 74.5 0.5 42.0 0.7 76.9 1.8 52.4 0.2 67.9 0.5 30.4 1.1 76.4 0.5 62.9 3.4 75.9 0.4 72.8 0.6 63.2 0.4 RefinedWeb 74.6 0.6 42.2 1.0 75.9 2.0 53.1 0.3 68.9 0.7 30.9 0.9 77.2 0.3 61.8 4.0 76.5 0.5 73.1 0.7 63.4 0.6 Wikipedia 74.0 0.6 40.8 0.7 71.2 1.2 51.7 0.2 65.5 0.6 30.1 0.9 75.7 0.2 63.1 2.2 74.9 0.4 71.5 0.8 61.8 0.4
Wanda
C4 74.8 0.3 42.0 0.6 75.7 1.0 54.5 0.1 64.1 0.2 31.4 0.5 77.8 0.2 62.9 2.2 78.1 0.3 72.2 0.4 63.4 0.3 CNN-DM 73.5 0.3 41.2 0.6 75.5 0.6 54.2 0.1 64.4 0.2 31.3 0.6 77.2 0.2 62.8 2.2 77.8 0.2 72.7 0.5 63.1 0.1 RedPajama 74.7 0.4 42.0 0.6 74.8 0.5 54.0 0.1 64.4 0.4 30.4 0.5 77.4 0.2 64.3 1.7 77.6 0.3 71.7 0.6 63.1 0.1 RefinedWeb 74.5 0.3 41.6 0.4 75.3 0.6 54.3 0.1 64.7 0.3 31.0 0.8 77.7 0.2 63.2 1.4 77.9 0.3 72.2 0.3 63.2 0.1 Wikipedia 73.8 0.3 41.3 0.3 72.2 0.9 53.7 0.1 64.1 0.3 30.3 0.5 77.2 0.3 64.9 1.7 77.0 0.2 71.7 0.3 62.6 0.2
GPTQ
C4 73.4 0.3 41.5 0.6 76.4 1.2 55.6 0.2 60.3 0.4 32.3 0.7 76.7 0.4 66.8 2.4 77.8 0.3 68.5 0.4 62.9 0.2 CNN-DM 73.3 0.5 40.7 0.9 75.3 1.0 55.2 0.1 60.1 0.7 32.0 0.7 76.4 0.5 68.7 1.7 77.9 0.6 67.7 0.8 62.7 0.3 RedPajama 73.3 0.5 41.4 0.9 76.5 1.0 55.8 0.1 61.5 0.6 32.5 0.6 76.9 0.5 67.0 1.1 78.3 0.5 68.8 0.7 63.2 0.2 RefinedWeb 73.0 0.5 41.4 0.8 76.6 0.9 55.6 0.2 61.1 0.7 33.1 0.9 76.9 0.2 66.8 1.3 78.1 0.4 68.4 0.4 63.1 0.2 Wikipedia 73.1 0.2 41.6 0.7 75.6 1.5 55.6 0.2 60.5 0.4 32.4 0.8 77.0 0.2 66.1 1.8 78.1 0.5 68.4 0.9 62.9 0.3
SpQR
C4 74.2 0.2 42.2 0.3 77.9 0.8 56.3 0.1 62.6 0.4 33.6 0.9 76.9 0.3 67.1 1.4 79.0 0.3 69.4 0.5 63.9 0.1 CNN-DM 74.2 0.2 42.3 0.7 77.9 0.6 56.2 0.2 62.4 0.4 34.0 0.5 77.0 0.2 68.6 1.4 78.9 0.3 69.4 0.3 64.1 0.1 RedPajama 74.4 0.4 42.1 0.4 78.2 1.0 56.1 0.1 62.6 0.3 33.8 0.5 77.1 0.3 67.8 1.7 79.1 0.3 69.6 0.6 64.1 0.2 RefinedWeb 74.1 0.3 42.3 0.5 78.0 0.6 56.1 0.1 62.6 0.5 33.8 0.9 77.1 0.3 67.9 1.5 79.1 0.3 69.3 0.6 64.0 0.2 Wikipedia 74.1 0.2 42.3 0.5 78.5 0.8 56.3 0.1 62.2 0.3 33.6 0.7 77.1 0.5 67.6 1.8 79.1 0.3 69.3 0.7 64.0 0.2
SparseGPT C4 63.1 0.5 31.7 0.8 71.5 1.2 45.0 0.3 47.4 1.1 24.5 0.9 71.0 0.7 64.2 1.9 73.2 0.5 65.1 0.8 55.7 0.4 CNN-DM 58.9 1.6 29.2 0.9 66.7 4.7 43.2 0.3 46.8 1.8 22.8 1.0 68.0 0.6 55.7 4.4 72.1 0.6 63.5 0.9 52.7 0.6 RedPajama 62.9 0.6 31.2 0.5 72.4 1.7 44.0 0.3 48.7 1.2 23.9 0.9 69.9 0.6 65.0 3.1 72.7 0.4 64.1 0.8 55.5 0.5 RefinedWeb 62.7 1.0 31.5 1.2 73.6 1.2 44.8 0.4 50.8 1.5 24.1 1.1 70.7 0.4 67.0 0.8 73.4 0.6 64.4 0.6 56.3 0.4 Wikipedia 61.7 1.4 30.4 0.9 72.3 1.0 42.8 0.2 45.7 1.2 23.6 1.4 69.2 0.4 63.2 4.2 70.8 0.5 63.1 0.7 54.3 0.6
Wanda
C4 61.5 0.5 31.8 0.5 59.0 2.2 44.2 0.2 42.9 0.4 23.9 0.8 70.6 0.2 66.6 1.3 71.6 0.4 62.3 1.0 53.4 0.4 CNN-DM 61.2 0.5 31.5 0.4 63.5 1.9 44.0 0.1 43.8 0.5 24.5 0.7 69.7 0.3 64.7 1.9 71.0 0.3 63.0 0.9 53.7 0.3 RedPajama 61.6 0.3 31.5 0.5 59.9 1.5 44.0 0.1 43.1 0.5 23.5 1.0 70.0 0.4 66.0 1.6 71.6 0.3 61.9 0.7 53.3 0.2 RefinedWeb 61.5 0.5 32.1 0.8 58.4 1.1 44.1 0.1 43.7 0.3 23.7 0.4 70.3 0.2 66.6 2.7 71.4 0.4 62.1 0.7 53.4 0.3 Wikipedia 60.3 0.4 31.0 0.5 64.6 1.9 43.8 0.1 43.3 0.2 24.5 0.7 69.9 0.3 63.0 1.3 71.7 0.3 62.2 0.6 53.4 0.3
Table 8: Mean accuracy across ten calibration sets for Vicuna-7B, with standard deviation denoted in subscript.
Method Dataset ARC-e ARC-c BoolQ HS LMBD OBQA PIQA RTE SC WG Mean
- - 75.4 45.6 82.8 59.7 67.2 36.4 78.3 67.5 80.8 71.2 66.5
GPTQ
C4 75.3 0.5 45.0 0.7 82.0 0.3 59.0 0.2 66.8 0.4 35.1 0.7 78.2 0.3 67.8 1.4 80.8 0.5 70.7 0.6 66.1 0.2 CNN-DM 75.5 0.4 45.1 0.6 82.6 0.4 59.0 0.2 66.7 0.5 34.8 0.9 78.1 0.5 67.2 1.7 80.9 0.3 71.0 0.4 66.1 0.2 RedPajama 75.6 0.3 45.3 0.4 82.3 0.4 59.1 0.2 66.7 0.5 34.8 0.5 78.2 0.4 68.5 1.6 80.8 0.4 70.2 0.4 66.2 0.2 RefinedWeb 75.7 0.2 45.3 0.8 82.1 0.3 59.0 0.1 66.4 0.3 34.4 0.8 78.3 0.3 67.3 2.0 80.7 0.5 70.8 0.6 66.0 0.2 Wikipedia 75.6 0.5 45.0 0.5 82.0 0.5 59.0 0.2 66.3 0.5 34.5 0.5 78.2 0.3 68.2 1.2 80.8 0.4 70.2 0.6 66.0 0.1
SpQR
C4 75.5 0.4 45.5 0.5 82.8 0.3 59.4 0.1 67.3 0.4 35.7 0.6 78.2 0.4 67.9 1.6 80.8 0.3 71.2 0.5 66.4 0.2 CNN-DM 75.5 0.3 45.7 0.5 82.8 0.3 59.4 0.2 67.3 0.3 35.1 0.5 78.3 0.2 67.7 1.0 80.6 0.2 71.0 0.4 66.3 0.1 RedPajama 75.6 0.3 45.6 0.4 82.8 0.3 59.4 0.1 67.6 0.4 35.5 0.4 78.1 0.3 68.4 1.3 80.8 0.2 70.7 0.4 66.5 0.2 RefinedWeb 75.5 0.3 45.5 0.4 82.7 0.2 59.4 0.1 67.6 0.3 35.6 0.4 78.1 0.3 68.3 1.8 80.7 0.3 71.2 0.5 66.5 0.2 Wikipedia 75.7 0.3 45.6 0.7 82.8 0.4 59.4 0.1 67.6 0.2 35.7 0.8 78.0 0.3 67.1 1.3 80.7 0.3 70.8 0.4 66.3 0.2
SparseGPT C4 68.0 0.4 37.9 0.7 77.6 1.0 49.2 0.3 56.8 0.8 28.4 0.9 73.8 0.6 58.8 2.7 76.3 0.3 68.2 0.4 59.5 0.4 CNN-DM 66.4 0.7 36.2 0.7 77.5 0.9 48.5 0.2 56.4 0.5 28.0 1.1 72.9 0.4 58.3 2.1 76.3 0.3 68.0 0.8 58.9 0.3 RedPajama 67.8 0.7 37.2 1.0 78.0 1.0 48.1 0.3 56.2 0.9 27.8 0.8 72.8 0.6 61.8 2.8 75.8 0.3 67.5 0.5 59.3 0.4 RefinedWeb 67.9 0.8 37.2 0.3 78.4 0.4 49.0 0.3 59.1 1.2 28.5 0.9 73.9 0.5 61.3 3.0 76.2 0.4 68.1 0.6 59.9 0.3 Wikipedia 66.4 0.9 35.9 1.3 76.0 1.3 47.3 0.3 51.8 1.1 28.5 0.8 72.5 0.6 57.4 2.5 74.4 0.5 66.3 0.5 57.6 0.6
Wanda
C4 65.9 0.4 35.6 0.5 76.7 0.4 48.0 0.2 46.4 0.4 28.6 0.5 72.2 0.4 54.5 0.8 75.2 0.3 66.2 0.4 56.9 0.2 CNN-DM 64.9 0.5 35.0 0.4 76.8 0.3 47.8 0.1 47.3 0.3 27.6 0.5 71.5 0.4 55.2 0.9 75.3 0.4 66.6 0.6 56.8 0.2 RedPajama 65.9 0.3 36.0 0.6 76.9 0.5 47.7 0.1 46.5 0.3 27.9 0.5 72.2 0.4 55.5 0.8 75.2 0.3 66.4 0.6 57.0 0.1 RefinedWeb 65.9 0.3 36.1 0.5 76.8 0.7 48.1 0.1 46.7 0.2 28.2 0.6 72.2 0.4 55.7 0.6 75.3 0.2 66.0 0.6 57.1 0.1 Wikipedia 66.2 0.5 35.1 0.7 77.6 0.4 47.5 0.2 46.3 0.4 28.8 1.0 72.1 0.4 55.3 0.9 74.8 0.5 66.1 0.4 57.0 0.1
GPTQ
C4 77.8 0.3 49.5 0.5 83.7 0.4 61.3 0.2 69.9 0.4 35.2 1.0 78.7 0.2 73.0 1.7 81.1 0.3 73.0 0.5 68.3 0.2 CNN-DM 77.5 0.5 49.2 0.9 83.2 0.5 61.2 0.1 69.4 0.4 35.7 0.7 78.8 0.4 73.4 1.2 81.5 0.2 73.7 0.7 68.4 0.2 RedPajama 77.9 0.3 49.7 0.4 84.0 0.4 61.3 0.2 69.9 0.4 35.4 0.4 78.8 0.4 73.0 1.4 81.2 0.2 73.3 0.6 68.4 0.2 RefinedWeb 77.8 0.3 49.7 0.4 83.7 0.4 61.3 0.1 69.6 0.4 35.4 0.8 78.9 0.3 73.0 1.1 80.9 0.3 73.6 0.5 68.4 0.2 Wikipedia 77.5 0.5 48.8 0.6 83.1 0.3 61.2 0.1 69.4 0.4 35.3 0.8 78.8 0.2 72.9 1.7 81.1 0.4 73.4 0.4 68.1 0.2
SpQR
C4 78.0 0.2 50.2 0.5 83.8 0.4 61.6 0.1 70.6 0.2 34.4 0.6 79.2 0.3 74.4 0.9 81.0 0.3 74.0 0.5 68.7 0.1 CNN-DM 78.2 0.3 50.1 0.5 83.8 0.2 61.6 0.1 70.5 0.3 34.7 0.7 79.2 0.2 73.8 1.0 81.0 0.3 74.2 0.5 68.7 0.2 RedPajama 78.2 0.5 50.0 0.5 83.8 0.2 61.6 0.1 70.7 0.3 34.7 0.7 79.3 0.3 73.8 0.8 81.0 0.3 74.2 0.3 68.7 0.1 RefinedWeb 78.2 0.4 50.2 0.4 83.8 0.1 61.6 0.1 70.5 0.3 34.7 0.3 79.1 0.3 73.5 1.0 81.0 0.3 74.1 0.4 68.7 0.2 Wikipedia 78.2 0.3 50.0 0.6 84.2 0.3 61.6 0.1 70.7 0.1 34.7 0.3 79.0 0.3 74.0 1.1 81.1 0.2 74.2 0.3 68.8 0.1
SparseGPT C4 73.4 0.6 43.7 1.2 77.9 1.1 54.4 0.3 67.1 0.8 32.9 0.6 75.6 0.3 72.8 2.0 79.5 0.3 71.7 0.9 64.9 0.3 CNN-DM 71.6 0.7 41.7 0.8 77.4 0.8 53.8 0.2 67.1 0.7 32.2 1.3 74.8 0.4 70.0 2.1 78.7 0.4 70.6 0.6 63.8 0.3 RedPajama 72.8 0.5 42.5 1.0 78.4 0.9 53.6 0.2 67.8 0.9 32.0 1.3 74.8 0.6 72.3 2.7 78.5 0.6 71.7 0.4 64.5 0.3 RefinedWeb 73.2 0.6 43.4 1.1 77.0 1.2 54.2 0.4 69.2 0.7 32.7 1.0 75.7 0.4 73.1 3.1 79.3 0.5 72.0 0.6 65.0 0.5 Wikipedia 72.2 0.6 41.8 0.6 74.5 1.2 52.8 0.2 65.4 0.6 31.6 0.9 74.2 0.6 68.2 2.5 77.7 0.5 69.7 0.9 62.8 0.3
Wanda
C4 71.4 0.4 43.0 0.5 78.8 0.3 54.9 0.1 62.7 0.2 32.8 0.7 75.4 0.4 70.8 1.4 80.5 0.3 69.5 0.2 64.0 0.2 CNN-DM 70.9 0.5 40.9 0.7 77.7 0.6 54.7 0.1 64.1 0.2 33.1 0.9 75.5 0.4 70.2 0.9 80.5 0.3 69.8 0.3 63.7 0.1 RedPajama 71.4 0.5 42.7 0.5 79.3 0.3 54.7 0.1 63.2 0.4 32.3 0.7 75.0 0.4 70.0 1.4 80.5 0.2 69.1 0.6 63.8 0.2 RefinedWeb 71.4 0.3 42.5 0.6 78.5 0.6 54.9 0.2 63.3 0.2 32.6 0.7 75.3 0.3 68.6 1.2 80.5 0.3 69.4 0.6 63.7 0.2 Wikipedia 71.1 0.3 42.0 0.5 78.3 0.4 54.4 0.1 62.6 0.2 32.2 0.5 74.3 0.3 69.9 1.3 80.2 0.2 69.7 0.7 63.5 0.2
Table 10: Mean accuracy across ten calibration sets for Vicuna-33B, with standard deviation denoted in subscript.
Method Dataset ARC-e ARC-c BoolQ HS LMBD OBQA PIQA RTE SC WG Mean
- - 65.6 30.5 66.1 50.5 63.3 27.6 76.3 55.2 73.6 65.2 57.4
GPTQ
C4 65.2 0.4 30.9 0.5 63.6 2.6 49.9 0.1 60.3 1.6 26.8 0.6 76.1 0.3 54.5 1.2 73.3 0.3 65.1 0.4 56.6 0.4 CNN-DM 65.0 0.4 30.9 0.5 63.5 1.9 49.9 0.1 60.7 1.1 26.6 0.7 76.1 0.3 55.1 1.6 73.4 0.3 64.3 0.7 56.5 0.3 RedPajama 65.1 0.3 30.9 0.6 63.7 1.8 49.9 0.1 62.4 0.9 26.3 0.9 76.0 0.3 54.9 1.2 73.2 0.2 64.4 0.6 56.7 0.3 RefinedWeb 65.1 0.5 31.1 0.7 63.3 2.2 49.9 0.1 60.8 1.7 26.4 0.9 76.1 0.2 55.2 0.9 73.1 0.5 63.9 0.7 56.5 0.4 Wikipedia 65.1 0.4 31.0 0.3 64.4 1.6 49.8 0.1 61.0 1.4 26.7 0.7 76.1 0.3 55.3 1.4 73.2 0.3 64.7 0.6 56.7 0.3
SpQR
C4 65.1 0.2 30.2 0.1 66.0 0.4 50.3 0.1 63.0 0.2 26.7 0.5 76.4 0.2 55.5 0.5 73.2 0.3 65.0 0.5 57.1 0.1 CNN-DM 65.6 0.2 30.5 0.4 66.7 0.5 50.3 0.1 63.7 0.3 27.5 0.5 76.4 0.2 55.4 0.7 73.6 0.2 64.9 0.5 57.5 0.2 RedPajama 65.6 0.2 30.5 0.4 65.6 0.8 50.2 0.1 63.2 0.3 26.9 0.5 76.4 0.2 55.4 1.3 73.4 0.3 64.8 0.4 57.2 0.2 RefinedWeb 65.3 0.3 30.2 0.4 65.9 0.6 50.3 0.1 63.1 0.5 26.9 0.6 76.3 0.2 55.1 0.5 73.4 0.2 64.7 0.4 57.1 0.1 Wikipedia 65.5 0.3 30.6 0.4 66.8 0.4 50.2 0.1 63.4 0.5 26.8 0.5 76.3 0.2 55.5 0.6 73.6 0.1 64.9 0.4 57.4 0.1
SparseGPT C4 59.8 0.8 25.5 0.6 62.8 0.4 43.2 0.1 55.6 1.1 24.4 0.7 72.4 0.4 53.1 0.5 69.9 0.4 62.0 0.7 52.9 0.3 CNN-DM 57.8 0.6 24.7 0.5 62.7 0.2 42.1 0.4 53.1 0.7 23.5 0.6 71.0 0.6 53.6 0.5 69.1 0.4 60.7 0.5 51.8 0.3 RedPajama 59.6 0.8 26.0 0.7 62.7 0.3 42.2 0.3 56.2 0.8 24.3 0.6 71.4 0.5 52.9 0.8 69.3 0.4 61.1 0.6 52.6 0.2 RefinedWeb 59.5 0.5 25.7 0.3 63.6 0.7 42.9 0.1 58.1 0.6 23.8 0.7 72.4 0.5 53.1 0.7 69.9 0.3 61.5 0.7 53.0 0.2 Wikipedia 59.0 0.4 25.5 0.6 62.1 0.1 40.8 0.1 48.0 0.5 24.1 1.0 70.1 0.3 52.9 0.3 68.1 0.4 61.2 0.6 51.2 0.2
Wanda
C4 56.8 0.4 24.8 0.3 62.3 0.1 41.7 0.1 44.1 0.7 23.1 0.7 71.3 0.3 53.4 0.5 68.5 0.4 60.5 0.6 50.6 0.2 CNN-DM 54.5 0.6 24.0 0.4 62.2 0.0 40.5 0.3 43.6 0.5 22.1 0.3 69.7 0.6 52.9 0.3 67.2 0.2 59.7 0.6 49.6 0.2 RedPajama 56.7 0.4 25.2 0.6 62.3 0.1 41.0 0.2 45.2 0.6 23.4 0.5 70.1 0.5 53.1 0.5 67.8 0.2 60.2 0.6 50.5 0.1 RefinedWeb 56.3 0.6 24.9 0.5 62.6 0.3 41.5 0.2 46.6 0.6 22.9 0.4 71.1 0.3 53.4 0.5 68.1 0.3 60.5 0.4 50.8 0.1 Wikipedia 56.5 0.4 24.5 0.6 62.2 0.0 39.3 0.1 37.2 0.6 22.4 0.8 68.8 0.4 53.1 0.3 66.3 0.3 58.8 0.6 48.9 0.2
GPTQ
C4 66.9 0.3 33.1 0.3 64.9 2.1 51.6 0.1 61.4 0.8 26.4 0.6 75.9 0.3 56.9 1.8 75.0 0.4 65.2 0.5 57.7 0.3 CNN-DM 67.1 0.4 33.0 0.5 65.7 1.4 51.6 0.1 61.6 0.6 26.7 1.1 76.0 0.3 56.2 1.7 75.1 0.4 64.8 0.6 57.8 0.3 RedPajama 67.1 0.6 33.2 0.5 65.5 1.6 51.6 0.1 61.9 0.6 26.8 0.6 75.7 0.2 56.4 1.1 75.3 0.2 65.4 0.3 57.9 0.3 RefinedWeb 66.7 0.4 32.8 0.4 66.0 1.6 51.6 0.1 61.6 0.5 26.5 0.3 75.8 0.4 57.0 1.3 75.2 0.2 64.9 0.5 57.8 0.2 Wikipedia 67.2 0.4 32.9 0.3 65.2 1.6 51.6 0.1 61.9 0.6 26.7 0.9 75.7 0.2 57.9 1.4 75.2 0.2 64.3 0.6 57.9 0.2
SpQR
C4 66.9 0.2 33.0 0.4 66.2 0.7 52.2 0.1 63.1 0.2 26.7 0.3 75.9 0.1 57.7 0.6 76.1 0.2 65.1 0.3 58.3 0.1 CNN-DM 67.1 0.1 33.1 0.3 65.4 0.7 52.1 0.1 63.1 0.3 26.8 0.9 76.0 0.2 58.7 1.5 76.0 0.2 65.0 0.3 58.3 0.3 RedPajama 67.0 0.2 32.9 0.3 66.0 0.3 52.1 0.0 63.2 0.3 27.0 0.5 75.8 0.4 58.2 1.6 76.0 0.2 65.2 0.7 58.3 0.2 RefinedWeb 67.2 0.2 32.9 0.4 66.0 0.9 52.1 0.1 63.2 0.2 26.7 0.5 75.8 0.3 58.3 1.2 76.1 0.2 65.0 0.4 58.3 0.1 Wikipedia 67.3 0.2 33.0 0.5 66.5 0.7 52.1 0.1 63.3 0.2 27.2 0.4 75.9 0.2 57.2 1.4 75.8 0.1 65.2 0.5 58.3 0.2
SparseGPT C4 61.8 0.4 29.0 0.7 66.8 1.1 45.1 0.2 58.8 1.1 24.6 0.7 73.3 0.3 53.8 0.9 72.0 0.5 63.3 1.2 54.8 0.3 CNN-DM 59.1 0.4 27.2 0.9 66.8 1.5 44.1 0.3 57.1 0.5 22.9 0.6 72.1 0.5 55.0 2.4 71.6 0.4 63.7 0.6 54.0 0.4 RedPajama 61.4 0.5 28.8 0.5 65.8 1.1 44.2 0.2 59.4 0.7 23.1 0.8 72.2 0.4 54.7 2.0 71.3 0.4 63.0 0.7 54.4 0.3 RefinedWeb 61.2 0.5 28.8 0.4 67.7 0.7 44.9 0.2 61.4 0.5 23.5 0.9 73.3 0.6 54.8 0.9 72.1 0.4 63.1 0.4 55.1 0.1 Wikipedia 60.9 0.8 28.4 0.5 62.2 0.0 43.0 0.2 51.1 0.8 23.0 0.4 71.6 0.3 53.6 1.2 70.4 0.3 62.7 0.6 52.7 0.2
Wanda
C4 58.4 0.5 27.1 0.2 66.1 0.6 44.7 0.1 48.8 0.8 21.9 0.6 71.8 0.3 53.8 0.3 71.0 0.4 62.0 0.5 52.6 0.2 CNN-DM 55.4 0.6 26.2 0.2 65.5 1.2 43.9 0.2 45.9 0.5 20.2 0.5 70.2 0.6 53.4 0.5 70.0 0.3 62.4 0.6 51.3 0.4 RedPajama 58.5 0.3 27.0 0.4 65.7 0.4 44.4 0.2 49.0 0.6 21.6 0.5 70.8 0.5 53.9 0.4 70.7 0.3 62.6 0.6 52.4 0.1 RefinedWeb 58.3 0.3 26.9 0.4 65.7 0.6 44.6 0.1 50.8 0.7 21.4 0.4 71.5 0.4 53.9 0.5 71.1 0.3 62.4 0.5 52.7 0.1 Wikipedia 57.9 0.5 26.9 0.4 62.2 0.0 43.1 0.1 39.7 0.5 21.6 0.5 69.8 0.5 52.6 0.6 68.9 0.3 62.1 0.6 50.5 0.2
Table 12: Mean accuracy across ten calibration sets for OPT-13B, with standard deviation denoted in subscript.
Method Dataset ARC-e ARC-c BoolQ HS LMBD OBQA PIQA RTE SC WG Mean
- - 70.0 34.6 70.5 54.3 64.3 30.2 77.6 57.4 76.6 68.4 60.4
GPTQ
C4 69.5 0.3 34.0 0.6 69.7 0.7 53.7 0.1 62.8 0.8 29.2 0.5 77.4 0.2 55.9 1.8 76.3 0.2 68.0 0.4 59.6 0.2 CNN-DM 69.4 0.2 33.7 0.4 69.4 0.7 53.6 0.1 63.3 0.8 29.4 0.6 77.4 0.2 55.8 1.3 76.4 0.3 67.9 0.3 59.6 0.1 RedPajama 69.5 0.4 33.4 0.4 70.1 0.3 53.6 0.1 63.5 0.4 29.5 0.5 77.4 0.3 56.4 2.1 76.3 0.2 67.7 0.3 59.8 0.3 RefinedWeb 69.5 0.4 33.5 0.4 69.8 1.1 53.7 0.1 63.1 0.6 29.3 0.4 77.3 0.2 56.1 1.5 76.3 0.2 67.9 0.9 59.7 0.2 Wikipedia 69.5 0.3 33.5 0.3 69.6 0.7 53.6 0.1 62.8 0.8 29.6 0.4 77.5 0.2 56.5 1.1 76.4 0.3 67.9 0.4 59.7 0.2
SpQR
C4 69.8 0.3 34.3 0.4 70.2 0.4 54.2 0.1 63.8 0.7 29.9 0.4 77.6 0.2 58.7 0.9 76.8 0.2 68.1 0.6 60.3 0.2 CNN-DM 69.6 0.1 34.2 0.3 70.5 0.5 54.1 0.1 63.9 0.7 30.1 0.3 77.4 0.3 58.0 1.2 76.7 0.2 68.0 0.3 60.3 0.1 RedPajama 69.8 0.2 34.3 0.4 70.2 0.5 54.2 0.1 64.1 0.3 30.2 0.4 77.6 0.2 58.0 1.4 76.8 0.2 67.8 0.3 60.3 0.2 RefinedWeb 69.9 0.3 34.5 0.3 70.2 0.3 54.1 0.0 63.6 0.7 29.9 0.4 77.4 0.2 58.2 1.4 76.8 0.2 68.1 0.5 60.3 0.2 Wikipedia 69.8 0.3 34.3 0.3 70.2 0.3 54.1 0.1 63.7 0.8 30.0 0.4 77.5 0.2 58.6 1.3 76.9 0.1 68.0 0.4 60.3 0.2
SparseGPT C4 66.5 0.3 31.3 0.4 66.1 0.9 48.7 0.1 63.5 1.0 26.7 0.8 75.3 0.2 54.5 1.0 73.9 0.4 66.4 0.5 57.3 0.3 CNN-DM 64.6 0.6 30.2 0.4 65.0 0.6 47.6 0.3 62.3 0.3 26.8 0.6 74.0 0.5 53.8 1.0 73.0 0.3 65.8 0.6 56.3 0.2 RedPajama 66.0 0.4 30.7 0.6 65.2 0.5 47.9 0.2 64.8 0.8 26.1 0.4 74.5 0.4 54.7 1.0 73.3 0.3 66.1 0.8 56.9 0.1 RefinedWeb 66.1 0.6 31.2 0.4 67.4 1.2 48.4 0.2 65.6 0.4 26.3 0.8 75.1 0.3 54.8 0.8 73.5 0.3 66.5 0.7 57.5 0.1 Wikipedia 65.5 0.6 30.4 0.7 62.2 0.0 46.5 0.2 57.5 0.6 25.4 0.6 73.7 0.3 53.9 0.9 71.6 0.4 65.1 0.5 55.2 0.2
Wanda
C4 64.3 0.3 29.1 0.3 63.4 0.7 47.9 0.1 50.9 0.6 24.8 0.5 74.8 0.3 53.5 1.7 73.4 0.4 64.2 0.4 54.6 0.3 CNN-DM 62.9 0.5 27.9 0.5 63.4 0.6 46.8 0.4 51.6 0.8 24.1 0.6 73.5 0.5 52.8 0.5 72.2 0.2 64.4 0.4 54.0 0.3 RedPajama 65.0 0.6 29.1 0.3 64.2 0.6 47.4 0.2 52.8 0.5 24.2 0.3 74.3 0.3 54.3 2.0 72.7 0.3 64.2 0.5 54.8 0.2 RefinedWeb 64.2 0.4 29.2 0.5 65.4 0.6 47.8 0.1 53.4 1.3 24.9 0.6 74.8 0.3 54.6 1.7 73.3 0.3 63.9 0.6 55.1 0.3 Wikipedia 63.6 0.4 29.1 0.3 62.2 0.0 45.6 0.2 46.4 0.8 23.4 0.5 73.1 0.4 53.0 0.6 70.6 0.3 64.0 0.6 53.1 0.2
Table 13: Mean accuracy across ten calibration sets for OPT-30B, with standard deviation denoted in subscript.

heading: Future Work
text: For future work, we are particularly interested in exploring how choices in the training protocol may influence the sensitivity to calibration data in LLM compression (Ahmadian et al. , 2023). 5Many existing studies have released code to generate the calibration data, but not the calibration data itself. This is not robust since using a seed for random sampling does not guarantee consistency across software or platform versions. Limitations
In this study, we rely entirely on English-language models, evaluation tasks, and calibration data. We operate under the assumption that the performance of the LLM compression methods we trial is generally language-agnostic. Nevertheless, we recognize the importance of linguistic diversity. We therefore hope to explore the performance of LLM compression methods across diverse language families (including low resource settings) in a future work