Title: Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Calibration Data

Large Language Model

Model Compression

Capability Preservation
Abstract: Post-training compression has been a widely employed approach to scale down large language model (LLM) and facilitate efficient inference. In various proposed compression methods, including pruning and quantization, calibration data plays a vital role by informing the weight importance and activation dynamic ranges. However, how calibration data impacts the LLM capability after compression is less explored. Few of the existing works, though recognizing the significance of this study, only investigate the language modeling or commonsense reasoning performance degradation from limited angles, like the data sources or sample amounts. More systematic research is still needed to examine the impacts on  different LLM capabilities in terms of compositional properties and domain correspondence of calibration data. In this work, we aim at bridging this gap and further analyze underlying influencing mechanisms from the activation pattern perspective. Especially, we explore the calibration data's impacts on high-level complex reasoning capabilities, like math problem solving and code generation. Delving into the underlying mechanism, we find that the representativeness and diversity in activation space more fundamentally determine the quality of calibration data. Finally, we propose a calibration data curation framework based on such observations and analysis, enhancing the performance of existing post-training compression methods on preserving critical LLM capabilities. Our code is provided in [Link](https://github.com/BokwaiHo/COLA.git).


Peer reviews:
comments: SUMMARY:
This paper studies how calibration data affects the ability of quantized LLMs to keep their skills. The authors test different data properties on pruning and quantization methods. They propose a three-stage COLA framework to pick the best samples by using model activations, random projection, and K-means clustering. They show that COLA improves performance over random sampling on many benchmarks.

STRENGTHS AND WEAKNESSES:
Strengths:
- The study covers language modeling, reasoning, math, and code tasks and is complete with consistent gains.
- The activation-space analysis gives insight into why calibration data matters.

Weaknesses: 
 - The paper does not test if the compressed model keeps its abilities under domain drift like further SFT. 
 - The sample selection step (random projection + K-means) adds compute and storage cost. The paper does not give any analysis on this overhead.

QUESTIONS:
I don't have any specific questions; I would like to hear other reviewers’ opinions and update my score based on the discussion.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-01

comments: SUMMARY:
This paper explores how calibration data impacts Large Language Model (LLM) capabilities after compression (like pruning and quantization). It identifies that the representativeness and diversity of calibration data in the activation space are key to preserving complex reasoning abilities like math and code generation. Based on this, the authors propose a framework to curate calibration data to improve post-training compression.

STRENGTHS AND WEAKNESSES:
Strengths:

Timely Topic: The paper addresses a highly relevant and timely problem: the impact of calibration data on Large Language Model (LLM) capabilities after post-training compression. 

Comprehensive Empirical Exploration: The paper conducts extensive experiments using two powerful open-source LLMs (LLaMA3-8B-Instruct and Qwen2.5-7B-Instruct) and multiple representative compression schemes (SparseGPT, Wanda, GPTQ, AWQ). Evaluation covers a wide range of capabilities, including language modeling, commonsense reasoning, mathematical problem solving, and code generation.

Weaknesses:

A potential limitation is the calibration data curation is not fundamentally different from Pretraining or SFT data curation.

QUESTIONS:
What are the fundamental differences between the calibration data curation method proposed in this paper and pre-training or SFT data curation?

LIMITATIONS:
Please see my questions.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes a novel framework for curating calibration data used by model compression techniques such as pruning and quantization. The paper includes a detailed empirical analysis of how various datasets and their combinations, languages, calibration data properties (such as sequence length and number of sequences) affect both general (pretrained) and downstream task accuracy. It also provides a data curation framework for calibration data named COLA based on the insights gleaned from the empirical analysis. The framework is evaluated across multiple pruning and quantization methods on two models:  Llama3-8b and Qwen2.5-7b.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n* The paper addresses an important problem: most pruning and quantization papers gloss over the details of the calibration data. As the authors demonstrate in the paper, various properties of the calibration dataset can significantly affect model accuracy in non-trivial ways.\n* I enjoyed going over the various empirical results in the main paper and appendix - the paper does a good job analyzing the various effects calibration dataset properties have on task accuracy.\n\n**Weaknesses**\n* Generality: given the hundreds of pruning and quantization methods in existence, I'm not sure if the insights presented in the paper will translate to at least the most common compression methods. As a specific example, the effects of calibration data may differ significantly for structured and layer pruning (which are commonly employed due to their hardware friendliness) compared to the unstructured and semi-structured methods discussed in the paper; it's not clear if/how the observations in the paper extrapolate to these and other cases. At least a brief discussion on this important aspect would make the paper stronger.\n* Also somewhat related to generality, the calibration data curation framework described in Section 4.2 seems to largely rely on manual dataset curation (stage 1) and heuristics (stage 2). Specifically, for datasets that haven't been explored in this work, it's not clear how correlations between datasets and domains can be computed by the end user (stage 1), and how/whether the observations on sequence length and number of sequences will translate to these new datasets (stage 2).\n* Writing: while the paper's main ideas have been expressed with sufficient clarity, the writing overall can be improved; in particular, there are several grammatical errors throughout the paper that need correction.\n\nQUESTIONS:\n* Please provide a brief response to the weaknesses listed above (relating to generality).\n* What is the proportion of each dataset selected by the framework described in Section 4.2 for each compression technique in Table 2?\n\nOther suggestions/comments:\n* Figure 1 could benefit from a more detailed caption.\n* Figure 5: line graph not recommended here since it's not showing a trend.\n\nLIMITATIONS:\nLimitations have been discussed adequately in Appendix K.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper focuses on the calibration data curation in post-training compression, which is claimed to be underexplored.
The study first analyses the length, sample number and data domain of the calibration data and how they would impact the result.
Then proposes a method on how to curate a good dataset by balance data from different source. The method is based on representation methods.
The results are reasonble.

STRENGTHS AND WEAKNESSES:
Strength:

- The paper is well written, clear, and straight-forward. 

- The results are reasonable and expected. 

- The proposed method on data calibration is effective. 

Weakness:

- One of the major weakness on paper presenting is that, please add baseline performance of each model before applied to any compression methods.  This will set a better comparison for demonstration. 

- The paper seems doing a lot of research on the data features in the first part of the paper and in appendix. But they seem separated from the method proposed, for example language domain. And on length and sample size, the maximum length studied is 2048, and the max sample size is 256. In my opinion, under the current trend of long CoT LLM, 2048 is a context length far from enough, and all the base model used in the paper can support upto 32k context. And sample size, I think the author should study much larger sample size, which I believe would make a difference as we alway observed during development, for example, the author can try a stride of 2^3 on sample size instead of 2^1.  This is basically the main drawback of the paper.

- On the proposed method, given the previous problematic analyses, it diminishes the significance and novelty of the method.

QUESTIONS:
As Above.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-19