PASER: Post-Training Data Selection for Efficient Pruned Large Language Model Recovery

Authors: Bowei He, Lihao Yin, Huiling Zhen, Mingxuan Yuan, Chen Ma
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Large Language Model, Model Pruning, Recovery Training, Data Selection

Abstract:
Model pruning is an effective approach for compressing Large Language Models (LLMs) and improving inference efficiency. However, this process often leads to significant degradation of model capabilities. While post-training techniques such as instruction tuning are commonly employed to recover model performance, existing methods often overlook the uneven deterioration of model capabilities and incur high computational costs due to extensive recovery training. Moreover, some instruction data irrelevant to model capability recovery may introduce negative effects. To address these challenges, we propose the **P**ost-training d**A**ta **S**election method for **E**fficient pruned large language model **R**ecovery (**PASER**). PASER aims to identify instructions where model capabilities are most severely compromised within a certain recovery data budget. Our approach first applies manifold learning and spectral clustering to group recovery data in the semantic space, revealing capability-specific instruction sets. We then adaptively allocate the data budget to different clusters based on the degrees of model capability degradation. In each cluster, we prioritize data samples where model performance has declined dramatically. To mitigate potential negative transfer, we also detect and filter out conflicting or irrelevant recovery data. Extensive experiments demonstrate that PASER significantly outperforms conventional baselines, effectively recovering the general capabilities of pruned LLMs while utilizing merely 4\%-20\% of the original post-training data and substantially reducing training computational overhead.

Decision: reject
Reviewer scores: 5, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1729016890843

thread_type:
official_review

<thread_object>
creator: reviewer_2rva

<content>
summary:
PASER is a new method to efficiently recover pruned large language models' abilities. It groups training data by capability, allocates resources based on where the model's performance dropped most, and removes unhelpful data. In tests on various models and pruning methods, PASER outperformed other approaches, restoring model performance using only 4-20% of the original training data and significantly reducing training time. While showing promise across different models and datasets,

strengths:
the PASER framework is thoughtful and comprehensive, including semantic-structural, recovery instruction clustering, capability degradation-aware adaptive recovery instruction selection and negative transfer mitigation.

weaknesses:
1. model recovery should be focused on the pre-training data, not instruction data since that's what the model has seen the most in training. The claim in line 45, that "instruction tuning data has emerged as a more preferred choice due to its superior efficiency in capability recovery", would need more citations and arguments to stand. 

2. Even if, for efficiency reasons, the authors choose to use instruction data for their experiments, they need to experiment and prove that each component in the PASER framework, the semantic-structural, recovery instruction clustering, capability degradation-aware adaptive recovery instruction selection and negative transfer mitigation, are contributing in some ways. And the authors need to tell us how each component improves. Those just need ablation studies but now not in the main paper.

3 in Appendix C, the authors state that logit distillation is better than label distillation when using a full dataset. this kind of contradicts the statement in the main paper, in line 407, that "the full version of data contains some irrelevant or conflicting information for capability recovery, resulting in the negative transfer during the training phase." 

If you find that logit distillation +  full dataset is comparable, or even better than label distillation +PASER selected dataset, you really have to sell hard here.

questions:
I would be happy to revisit my score if weakness 2, the ablation studies for each component is fullfilled.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730420791157

thread_type:
official_review

<thread_object>
creator: reviewer_nhgz

<content>
summary:
PASER proposes a method to recover pruned large language models (LLMs) while maintaining computational efficiency. The challenge in model pruning is performance degradation due to data irrelevance and computational costs. PASER employs manifold learning and spectral clustering to group data by specific capabilities, allocating recovery data budgets based on degradation severity in each cluster. This approach ensures balanced capability recovery, reduces computation, and mitigates negative transfer by filtering irrelevant data.

strengths:
1.	PASER introduces a targeted post-training data selection approach, addressing the issue of uneven capability degradation in pruned models. This is a notable advancement over existing methods, which often lack specificity in data selection for capability recovery.
2.	The proposed method leverages spectral clustering and diffusion kernel methods for data grouping, and introduces a concept consistency graph to minimize negative transfer, highlighting innovative use of these techniques.
3.	Extensive experiments demonstrate PASER's effectiveness across different pruning methods and datasets. This validates the method’s applicability and robustness across a variety of scenarios, contributing to its reliability.

weaknesses:
1.	While PASER is computationally efficient relative to prior approaches, the process of clustering and capability degradation-aware sampling might still pose a bottleneck for large-scale datasets or real-time applications.
2.	While PASER attempts to filter out irrelevant or conflicting data using the Concept Consistency Graph (CCG), the approach for ensuring consistency could still allow for some degree of negative transfer if conflicting concepts are not comprehensively identified. It might be beneficial for the authors to discuss scenarios where CCG may not fully capture semantic conflicts and the resulting effects on model recovery.
3.	Dependence on Capability-Specific Clustering: The effectiveness of PASER hinges on the quality of the clustering step to segregate instructions by capability degradation. Errors in clustering or the inability to accurately map instructions to the affected capabilities could lead to uneven recovery, especially in complex or high-dimensional semantic spaces. Exploring alternative clustering or dimensionality reduction methods could strengthen the paper’s reliability.

questions:
1.	While PASER is computationally efficient relative to prior approaches, the process of clustering and capability degradation-aware sampling might still pose a bottleneck for large-scale datasets or real-time applications.
2.	While PASER attempts to filter out irrelevant or conflicting data using the Concept Consistency Graph (CCG), the approach for ensuring consistency could still allow for some degree of negative transfer if conflicting concepts are not comprehensively identified. It might be beneficial for the authors to discuss scenarios where CCG may not fully capture semantic conflicts and the resulting effects on model recovery.
3.	Dependence on Capability-Specific Clustering: The effectiveness of PASER hinges on the quality of the clustering step to segregate instructions by capability degradation. Errors in clustering or the inability to accurately map instructions to the affected capabilities could lead to uneven recovery, especially in complex or high-dimensional semantic spaces. Exploring alternative clustering or dimensionality reduction methods could strengthen the paper’s reliability.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730630908351

thread_type:
official_review

<thread_object>
creator: reviewer_bamd

<content>
summary:
This paper introduces a novel post-training selection method PASER to improve the recovery of pruned large language models. Specifically, this method first utilizes manifold learning and spectral clustering to cluster recovery data, and then allocate the data budget based on the degrees of model capability degradation. Experimental results show the effectiveness of PASER in pruned LLM recovery.

strengths:
1. The paper is well-organized and easy to read with clear explanations of the technical details.
2. The proposed method PARSER includes semantic structural recovery instruction clustering and capability degradation-aware recovery instructions selection is simple yet effective. Experimental results across seven common sense reasoning datasets show some improvements compared with the full fine-tuning method.
3. The author conducts experiments on various types and scales of LLMs, resulting in a relatively comprehensive set of experimental results.

weaknesses:
1. Cluster and then selection paradigm has been used in various data selection methods such as MoDS, and CaR, but the novelty is limited. And the selected data are limited to embedding models and clustering methods.
2. The author conducts extensive experiments, but these are limited to the common sense reasoning task. How does this method perform in other scenarios, such as code or mathematical reasoning tasks?  In these scenarios, where the probability of most tokens for a sample is high, how can the capability degradation score compare the scores between samples of different lengths balanced?

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730722418660

thread_type:
official_review

<thread_object>
creator: reviewer_9bj7

<content>
summary:
PASER is a new method for efficiently recovering the performance of pruned Large Language Models (LLMs) by carefully selecting the most relevant training data. This three-step process begins by grouping instructions based on the LLM capabilities they target using SentenceBERT embeddings and NMF-based spectral clustering. Then, a Capability Degradation Score (CDS) is calculated for each group to identify the areas where model performance has suffered the most from pruning. The data budget is allocated accordingly, prioritizing these most affected clusters. Within each cluster, data samples with the highest Individual Efficiency Score (IES) are chosen to balance recovery benefits with computational costs. Lastly, a Concept Consistency Graph (CCG) is created to model relationships between concepts and prevent the selection of conflicting or irrelevant samples, mitigating negative transfer during training. Extensive experiments demonstrate that PASER effectively recovers pruned LLMs to near-unpruned performance levels using only part of the original data.

strengths:
PASER addresses the uneven impact of pruning on different LLM capabilities by using a clustering technique to group instructions based on the capabilities they target. Instead of blindly using all available data or randomly selecting a subset, PASER allocates its data budget proportionally to the Capability Degradation Score (CDS) of each cluster. The use of a Concept Consistency Graph (CCG) helps identify and reject samples that might introduce conflicting or irrelevant information.

weaknesses:
The quality of the initial pruning likely influences the effectiveness of PASER's recovery process. A poorly pruned model might present challenges that even a targeted recovery approach like PASER cannot fully overcome. 
Additionally, the novelty is not entirely there as this work fails to position itself in the expensive literature similar to this approach.

There are several works on dataset selection, in particular, instruction dataset selection:

[1] Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew E. Peters, Pradeep
Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hanna Hajishirzi. Camels in
a changing climate: Enhancing lm adaptation with tulu 2.

[2] Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for
alignment? a comprehensive study of automatic data selection in instruction tuning. 

[3] Peiqi Wang, Yikang Shen, Zhen Guo, Matthew Stallone, Yoon Kim, Polina Golland, and Rameswar
Panda. Diversity measurement and subset selection for instruction tuning datasets. 

[4] Simon Yu, Liangyu Chen, Sara Ahmadian, Marzieh Fadaee. Diversify and Conquer: Diversity-Centric Data Selection with Iterative Refinement.

[5] Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David
Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring
the state of instruction tuning on open resources.

[6] Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting
influential data for targeted instruction tuning,

questions:
- The paper mentions that instruction tuning data is preferred for LLM recovery due to its efficiency and ability to maintain general-purpose abilities. However, are there any limitations to using instruction-tuning data for this purpose? For instance, could there be scenarios where pre-training corpora or fine-tuning datasets might be more suitable for recovering specific capabilities heavily affected by pruning?

- The paper briefly mentions knowledge distillation as an alternative to supervised learning for recovery training. What are the potential benefits and drawbacks of using knowledge distillation compared to standard fine-tuning? How does PASER's performance differ when using knowledge distillation versus supervised learning? Could combining both approaches lead to further improvements?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732247414545

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revisions in Manuscript

comment:
Dear Chairs and Reviewers:

Thanks for your insightful feedback and active engagement. Here, we summarize our revisions in the original manuscript to better clarify our effort for addressing reviewers' concerns:
1. **2nd Paragraph in Section 1 Introduction:** We have revised the 2nd paragraph in Introduction to highlight the advantages of intruction tuning data for recovery training compared to pre-training corpora (higher efficiency, explicit supervision) and fine-tuning datasets (avoiding overly specialized recovery). Besides, we have followed the suggestion of Reviewer 2rVA and supplemeted more citations in this paragraph.
2. **Section 5.5. Ablation Study:** To alleviate the Reviewer 2rVA's concern regarding the ablation study, we have supplemeted the Section 5.5. in the main paper to validate the positive contribution brought by our each proposed component. The extensive experiments were conducted under four kinds of different pruning schemes and achieved consistent results.
3. **Appendix C. Extended Experiments on Recovery Training with Knowledge Distillation:** To investigate the potential combination strategy between the supervised fine-tuning and knowledge distillation for recovery training (as suggested by Reviewer 9bj7), we made the first-step exploration and combine them in a cascading way. The corresponding results and analysis are provided in the Appendix C. as an independent paragraph of **exploration on Combined Training Strategies**.
4. **Appendix D. Detailed Ablation Study Results:** Following Section 5.5., we also provide the detailed ablation study results and corresponding analysis in Appendix D.
5. **Appendix F. Evaluation on Mathematical Reasoning Tasks:** In response to Reviewer BaMD's concern on performance in other scenarios, we followed the reviewer's suggestion and conducted the experiments on several mathematical tasks. The corresponding results and analysis are now provided in Appendix F.

Hope these revisions and rebuttals can help reduce your concerns to our paper.

Best

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This submission introduces PASER, a post-training data selection method for efficient pruned LLM recovery. It idea is to restore capabilities of pruned LLMs. The primary challenge is to address performance degradation when pruning models. PASER aims to address this by identifying and prioritizing training data where model capabilities are most impaired, by using manifold learning and clustering to group instructions by capability type. By allocating training resources based on degradation severity and filtering irrelevant data, PASER achieves effective capability recovery using a small portion of conventional training data. 

The reviewers identified the strengths of this work as:
- the proposed method addresses uneven capability degradation through targeted data selection and clustering

The concerns are raised by the reviewers on:
- limited novelty as similar clustering and selection approaches exist in prior work
- incomplete validation and limited analysis as the experiments focus mainly on common sense reasoning and lacks thorough ablation studies to validate the contribution of each component and some contradicting claims about negative transfer effects

During the rebuttal, reviewer 9bj7 argued that they are not convinced of some of the answers, thus decided to keep overall ratings as 5 but increase the contribution score. reviewer BaMD still believed the novelty is limited and thus kept the ratings 5. It is unfortunate that both reviewer nHgZ and reviewer 2rVA did not engage with the authors' rebuttal though with the authors' an AC's reminders. 

The final ratings of this work are 5, 5, 5, 5. The  main concerns are on its technical novelty compared to existing work and the lack of sufficient empirical evidence in the original submission. Though new results and arguments have been added during rebuttal, two of the reviewers remain unconvinced on these two issues above. Therefore, this work would benefit from further revision. Based on these factors, this work in its current form is not recommended for acceptance.