PASER: Post-Training Data Selection for Efficient Pruned Large Language Model Recovery
Authors: Bowei He
Lihao Yin
Hui-Ling Zhen
Xiaokun Zhang
Mingxuan Yuan
Chen Ma
Venue: ICLR 2026 Poster (2026)
Keywords: Model Pruning
Large Language Model
Data Selection
Efficient Recovery
Primary area: foundation or frontier models, including LLMs
TL;DR: To achieve efficient capability recovery for pruned LLMs, we propose the PASER method to conduct the post-training data seletion.

Abstract: Model pruning is an effective approach for compressing large language models (LLMs). However, this process often leads to significant degradation of model capabilities. While post-training techniques such as instruction tuning are commonly employed to recover model performance, existing methods often overlook the uneven deterioration of model capabilities and incur high computational costs. Moreover, some irrelevant instructions may also introduce negative effects to model capacity recovery. To address these challenges, we propose the **P**ost-training d**A**ta **S**election method for **E**fficient pruned large language model **R**ecovery (**PASER**). PASER aims to identify instructions to recover the most compromised model capacities with a certain data budget. Our approach first applies manifold learning and spectral clustering to group recovery instructions in the semantic space, revealing capability-specific instruction sets. Then, the data budget is adaptively allocated across clusters by the degree of corresponding model capability degradation. In each cluster, we prioritize data samples that lead to the most decline of model performance. To mitigate potential negative tuning effects, we also detect and filter out conflicting or irrelevant recovery data. Extensive experiments demonstrate that PASER significantly outperforms conventional baselines, effectively recovering the general capabilities of pruned LLMs while utilizing merely 4\%-20\% of the original post-training data. We provide the anonymous code repository in [Link](https://anonymous.4open.science/r/PASER-E606).

Published as a conference paper at ICLR 2026
PASER: POST-TRAINING DATA SELECTION FOR EFFI- CIENT PRUNED LARGE LANGUAGE MODEL RECOVERY
Bowei Lihao Yin?, Hui-Ling Zhen?,
Z Xiaokun Zhang', Mingxuan Yuan', Chen Ma 1 City University of Hong Kong," 2 Huawei Noah's Ark Lab, Hong Kong,
ABSTRACT
Model pruning is an effective approach for compressing large language models (LLMs). However, this process often leads significant degradation of model capabilities. While post-training techniques such as instruction tuning are com- monly employed to recover model performance, existing methods often overlook the uneven deterioration of model capabilities and incur high computational costs. Moreover, some irrelevant instructions may also introduce negative effects to model capacity recovery. To address these challenges, we propose the Post-training dAta Selection method for Efficient pruned large language model Recovery (PASER). PASER aims to identify instructions recover the most compromised model capac- ities with a certain data budget. Our approach first applies manifold learning and spectral clustering group recovery instructions in the semantic space, revealing cypability-speciine instruction sets. Then. the data budget is adaptively allocated across clusters by the degree of corresponding model capability degradation. In each cluster, we prioritize data samples that lead to the most decline of model performance. To mitigate potential negative tuning effects, we also detect and filter out conflicting irrelevant recovery data. Extensive experiments demonstrate that PASER significantly outperforms conventional baselines, effectively recovering the general capabilities of pruned LLMs while utilizing merely 4%-20% of the original post-training data. We provide the code repository in Link
INTRODUCTION
Model pruning, which aims reducing model parameter amounts while maintaining model capabili- ties, has been a promising approach for large language model (LLM) compression. Mainstream LLM pruning schemes are unstructured (Frantar Alistarh 2023). semi-structured (Sun et al.) 2024), and structured pruning (Ma et al. 2023). Even though effective, the model capability degradation after pruning almost unavoidable, especially under high pruning ratios. This degradation phenomenon is often more severe for the structured pruning scheme (Dong et al., which has been widely adopted in industrial LLM compression thanks to its hardware-friess friendly property. Therefore, recovery post-training after pruning has been a silver bullet to address thisissue (Ma et al., 2023) Zhao et a Among various types of data including pre-training corpora and extensive fine-tuning datasets (Xia) et al., 2024a; Sun et al. 2024). instruction tuning data has demonstrated unique advantages for efficient capability recovery (Ma et al. 2023 Zhao et al.) Zhang et al. 2024 Chen et al.) 2023).
Compared to recovery via continued pre-training which requires massive computational resources, instruction tuning enables effective restoration with a much smaller data scale by explicit supervision. Furthermore, through the diverse task coverage, like language modeling. common sense reasoning, mathematical problem solving, and code generation (Taori et al. 2023] Wu et al. 2024), instruction tuning preserves the model's general-pune -prrpos capabilities while preventing over-specialized recovery.
Conventional schemes ( MM et al. 2023) usually employ the full instruction tuning datasets like Alpaca (Taori et al. (023) to conduct the recovery post-training. However, this can bring significant computation overhead and even unsatisfactory recovery performance (See Appendix A).. An intuitive solution to leverage part of the original data for training. thus consuming less data and correspond- ingly reducing the computation resource demand. Nevertheless, directly utilizing the naively split data
Work done as an intern in Huawei Noah's Ark Lab, Hong Kong, boweihe2 -aceey. cityc. eduu..du..i hkk Corresponning author, chenmal a@ci wec.ttt.eeeee edu.

---

Published as a conference paper at ICLR 2026
subset (e.g., first 20% of the data), can lead to sub-optimal performance, even further performance degradation. Moreover, the recovered performance considerably varies for models trained with different subsets, implying the sensitivity to data constitution. Therefore, selecting the most valuable instruction-tuning data that can contribute to recovery performance positively and reduce training costs becomes crucial. Though previous works have noticed the significance of selecting high-quality data for general instruction tuning (Wang et al., 2024 Cao et al. Li et al., 2024a), few of them are specifically designed for the recovery post-training. Note such general high-quality standards, such as clear structure and natural, human-like expressions, may not effectively target the specific capabilities (e.g., math, code) severely compromised during pruning, which is essential for recovery post-training.
Considering the above limitations, the ideal recovery post-training data selection approach should exhibit the following properties: (1) Targeted and Balanced Capability Recovery: Given the uneven deterioration of different capabilities in the pruning (see Appendix A). the ideal selection method should identify and prioritize the severely-degraded ones, while ensuring balanced recovery of the model's overall functionality. (2) Recovery Training Efficiency: Limited computing resources pose serious efficiency challenges to the LLM recovery post-training. An ideal method should be able to select instructions that are both most beneficial for recovery and light in computational cost, thereby accelerating the training process and optimizing resource utilization. (3) Mitigation of Negative Tuning Effects: Recognizing that not all instruction data is beneficial for model recovery, an optimal approach should not only identify the most advantageous instructions, but also filter out potentially harmful or irrelevant ones. This significantly reduces the risk of negative tuning effects during the recovery training, ensuring that the selected data contributes positively to model recovery.
To achieve such goals, we propose the Post-training dAta Selection method for Efficient pruned large language model Recovery (PASER). First, we perform semantic-structural recovery instruction clustering to obtain data groups corresponding to different LLM capabilities. Second, we select recovery instructions in a capability degradation-awai manner, where the data budget is allocated across different clusters based on their corresponding capability degradation degrees. particular, when selecting samples within each capability cluster, the post-training computation cost of each sample is also taken into consideration to optimize the efficiency of the recovery process. Finally, we construct the concept consistency graph to maintain the semantic consistency across selected instruc- tions, thus preventing the introduction of conflicting or irrelevant samples. Furthermore, we analyze the time complexity (Section (3.5) and error bound (Appendix C) of PASER theoretically, which guarantee the efficiency and effectiveness, respectively. We take the LLaMA 2/3/3.1, Baichuan2, Qwen2.5/3, and Mixtral 8x as the target LLMs and perform the experiments under different LLM pruning schemes and different-sized instruction tuning datasets. The comparison with random and conventional instruction tuning data selection baselines demonstrates that PASER can more effectively enhance the recovered LLM performance on language modeling and various reasoning tasks. Meanwhile, the recovery training overhead can also be reduced significantly.
2 RELATED WORKS
Large Language Model Pruning can be generally divided into three categories: unstructured pruming. stmiistrrcturee pruming, and structured pruning. Unstructured pruning removes individual weights without structural constraints, with representative works including SparseGPT (Frantar & Aliitaarh 2023). Wanda (Sun et al. 2024), BESA (Xu et al. 20245), and OWL (Yin et al. (024))
This technique allows for maximum flexibility in weight selection and can achieve high compression rates while maintaining model performance. However, the resulting irregular sparsity patterns limits the practical acceleration. Semi-structured pruning (Guo et al., 2024, Malla al. 2024 Frantar & Alistarb 2023 Sun et al. 2024) targets specific patterns like N:M sparsity, balancing flexibility and hardware efficiency. Structured pruning approaches like LLM-Pruner& (Ma et al.) (023) and
SliceGPT (Ashkboos et al. (222)) remove entire structural components, offering better hardware compatibility and attracting industry attention (Ko et al. 20233 An et al.) 2024 [Sng et al.) 2024 Xia
cet al., 2024a). However, structured pruning faces more severe performance degradation, highlighting the importance of recovery post-training.
Instruction Tuning has emerged as crucial technique for enhancing LLMs (Wei et al. Wang et al.
2023), improving their adaptability to novel tasks (Sanh et al. Liang et al. Zhou et al. (202)). Recent works have explored instruction tuning post-compression recovery mechanism (Zhao et al. Ma ct al. 2023). While promising, this combination faces challenges from reduced model capacity and computational costs. Most current approaches use general instruction datasets without considering
2

---

Published as a conference paper at ICLR 2026
Porformance
Full Dam
LLM Pruning
Recovery Training
Original LLM
Pruned LLM
Data Selacted with PASER
Performance
Semantic- Strruntt-Sttuettrall Recovery Instruction Clustering
Targeted
Recovered LLM
Recovery
Balanoed
Data
Effficient
Concept Consistency Graph
100
Extracted Concepts
Update Graph
3
Inter Cape--Cplllltty Allocation
Capubility Degradation Asseasment
Intes-Cpabiliit ENTsiemcy-Driven Sample Selection
Chock Consistency
Add Sample
Capubilitt Degradation Aware Instruction Selection
Negative Tuning Effeets Mirigation
Figure 1: Visualization for our proposed PASER recovery post-training data selection framework.
compressed model's characteristics or disproportionately affected capabilities. Our work addresses these gaps by proposing novel framework for post-training data selection in pruned L.M recovery.
Data Selection has been a widely studied approach to improve instruction tuning efficiency and model performance. Instruction Mining (Cao et al.) defines set of rule-based evaluation metrics to assess the quality of instructional data. IFD (Li et al., (024a) and Nuggets (Li et al. (024bb leverage trainable LLMs develop scoring functions for instruction data selection. Other works like LESS (Xia et al. 2024b) and DELIFT (Agarwal et al.) 2025) rely on information value analysis tools like influence function select diverse and informative samples. Earlier methods like similarity- based sampling in embedding space (Hardt & Sun (2024) can also reduce data redundancy and emphasize representative examples. More broadly, Datamodels (yyas et al. 2022) analyze how individual examples shape model predictions, thus identifying the most beneficial training data. Unlike above these methods, our this work focuses specifically on post-prumine capability recovery, where the goal not to select generally helpful instructions but to construct cupability-awarr degradation-sensittee data subset that targetedly restores pruned models.
3 METHODOLOGY
In this section, we first formulate the problem and then introduce three main components of the PASER framework (shown in Figure semantic-structural recovery instruction clustering (SRRIC), capability degradation-awart instruction selection (CDAIS), and negative tuning effects mitigation (NTEM). Furthermore, we provide the time complexity analysis for the PASER process.
3.1 PROBLEM FORMULATION
Let M. denote the original large language model and Mp the pruned version of this model. We define the instruction tuning dataset as D { y: ) /
where represents an instruction and yi is its corresponding output. Our goal to select subset S C D to efficiently recover the performance of optimization task:
Mp. We formulate the problem
S = arg min
((e.)) ((M, v)),
Sc SCD
(I)
s.t. (S) Recovery RecoveryTrain(M... Train(M,. S),
where M,(S) is the recovered model after training subset S, T is the distribution of downstream evaluation tasks, L a loss function. B(B < N) is the recovery data budget, i.c. the maximum number of samples allowed in the selected subset.

---

Published as a conference paper at ICLR 2026
3.2
-STUUICCRT RECOVERY INSTRUCTION CLUSTERING
After the LLM pruning, different model capabilities can be affected unevenly because such capabiliiiess rely on distinct sets of internal modules (Meng al.) 2022 Dai et al. (022). which are selectively removed. To ensure targeted and balanced recovery, we need to identify and group data points related to similar capabilities. To achieve this goal, we hypothesize that distinct geometric structures of recovery instruction data in the high-dimensional semantic space may correspond to different aspects of LLM capabilities. This hypothesis based on the intuition that instructions pertaining similar capabilities are likely to cluster together in the semantic space, forming identifiable topological structures. Thus, we employ a two-step approach on the embedding space of instructions. First, to enhance the natural separation between capability -speciffc instruction clusters, we apply diffusion kernel to SentenceBERT (Reimers & Gurevych 2019) embeddings for manifold learning:
e(z;)=
(2)
Here, e(2;)) is the obtained low- diw-emmnnsioall manifold representation of instruction I;. This process helps uncover the intrinsic geometric structure in the semantic space while reducing dimensionality and preserving non-linear relationships. Then, non-negative matrix factorization (NMF) based spectral clustering (Ding et al. 2005) is conducted based on such e(2;) to identify natural groupings of instructions that potentially correspond to different LLM capabilities, leading to K clusters:
C' = (ei,....,cc' cK)
v:) E D)).
The details are provided as below. In the first step of manifold learning. we first obtain the Sentence- BERT embedding of each instruction 2;. Then, adjacency matrix A is constructed based on the
pairwise Euclidean distances of these embeddings: Au exp(- [|eentenceERRT(a.>>> (aaeiieeeitRtrree--Seeeeenttttt
2n² where a is a scaling parameter, typically set to the median of all pairwise distances. The degree matrix D is then computed as diagonal matrix where each diagonal element is the sum of the corresponding row in A: = Σラ= Aij. Using these matrices, we define the normalized graph Laplacian L = 1 - D-1 AD-2AD-11 where / is the identity matrix. We then apply the diffusion kernel to this Laplacian K, = exp(-1L) where Ke is the diffusion kernel at time L. The diffusion time / is selected using the spectral gap method: topt arg max,
(ogg (1tt)) ) where A2(t) is the second d'log(t) eigenvalue of K,- The low-dimensional manifold representation e(x;) is then obtained by selecting the top d eigenvectors of Kin..' e(z;) = [0] (zz), (2(x:), ((((:)) ]]. where dj are the eigenvectors of Kipp corresponding the d largest eigenvalues.
In the second step, we perform NMF-based spectral clustering on these low-dimensional repre- sentations. Specifically, we construct similarity matrix S from the manifold representations Si, = =exp(- =(=x))=((=>)]""" We then determine the optimal number of clusters K by performing NMF with different values of and selecting the one that minimizes the Frobenius norm of the approximation error K arg ming || Wa [l, where W& and Hş are non-negan non-negative matrices resulting from NMF with & components. With this optimal K, we decompose the similarity matrix S using NMF such that S s WHT, where W and H are non-negative matrices with K columns. Finally, we assign each data point to a cluster based on the maximum value in each row W, where e = arg max; Waj, 1, ...,... N. This results in K clusters C {cI, (cI,..,,ck) J. where the number of clusters K is adaptively determined through the above process.
3.3 CAPABILITY DEGRADATION-AWE INSTRUCTION SELECTION
Capability Degradation Assessment To prioritize the severely affected capabilities and finally achieve the balanced recovery of pruned LLMs, we need a measure of capability degradation to guide the data selection. For each cluster сą obtained in Section 3.2 we define the capability degradation score (CDS) with the Jensen- Shannon divergence (JSD) (Fuglede & Topsoe] [20s), which measures distributional differences between the original and pruned models' outputs follows:
lyl
JSD (SD(P(tml))) D (tt
Ck
y
m==1
4

---

Published as a conference paper at ICLR 2026
Here, P(tm MMp F, Vcm) represents the output probability distribution the m-th token variable of the pruned model Mp given imput x and previous m tokens of output y. Taking a token value em (1 S i S |Voel) in this distribution as an example, its corresponding probability is
login(
exp
MMpaM 9em)
(5)
Vocl 1==1 exp( logis(tin))
where is the softmax temperature and the |ocl indicates the vocabulary size. logit(.) is the logit value for tokens produced by LLM. Similarly, the P(tm [Mo; z, Fem) represents the output probability distribution for the original model Mo. The JSD is actually the symmetrized and smoothed version of the Beilbacck-Libble divergence (KLD) (Kullback & Leibler 1951h: YY) = ( + }KLD(Y)JM)) The distribution M = is the mixed distribution of X and Y. Thus, the obtained CDS quantifies the average performance degradation for data points in each capability cluster. The choice of JSD over simple loss variations as the performance degradation signal is motivated by its unique properties. First, its symmetry ensures consistent comparison between the pruned model Mp and the original model Mo. while its bounded range (0 to provides normalized measure of divergence. This facilitates easier interpretation and comparison across different capability clusters. Moreover, JSD's 's robustness to outliers and its information - theoretic foundation allow for a more nuanced assessment of capability degradation, capturing subtle changes model outputs that might not be apparent from loss or accuracy values alone (Dutta et al.) 2024) due to the sampling uncertainty. In fact, the output distribution reveals the model's s thinking process more comprehensively than just sampling results, which is critical for capability degradation assessment. Besides, the smoothing effect introduced by the mixed distribution in JSD calculation also contributes to more stable assessment across diverse instruction types. Thanks to these properties, we obtain a comprehensive and reliable assessment of capability degradation by employing JSD, enabling more accurate identification and prioritization of the capabilities most severely affected by model pruning.
Cnper-Capability Budget Allocation Sampling a subset of high-quality data S from the original set D to achieve better training performance is the objective of the data selection process. Typically, an instruction data budget B( < N) should be maintained to ensure the efficiency on data utilization and training process. Under this budget, we design an adaptive selection approach based on the above CDS for balanced recovery while focusing on severely affected capabilities. In detail, we allocate the sampling budget to each cluster proportionally to its corresponding CDS:
CDS(Ck)
ni
B
Σド= CDS(c,)
K
n& is the sample number budget allocated to cluster Ck-
Intra- Carraiiip Efficiency-Driven Sample Selection To maximize computational efficiency during the recovery post-training phase, we need to select samples that offer the highest recovery benefit relative to their computational cost. Within each cluster Ck. we select the top n& samples based on their Individual Efficiency Scores (IES):
lel
IES( y)
log ComputationnalCot((1,, y)
Here, ComputationalCost is instantiated with the quadratic term of sequence length (|rr + lv/) as the approximated complexity for LLM training. The use of JSD captures the degree of divergence between the pruned and original models' outputs, indicating areas where capabilities have been most affected and thus offering the highest potential for targeted recovery. The logarithmic term balances the consideration of computational cost, allowing for more careful selection that favors efficient samples without overly penalizing high-potential, moderately costly instances.
3.4 NEGATIVE TUNING EFFECTS MITIGATION
To prevent performance degradation due to confficting or irrelevant information, we need to detect and mitigate the potential negative tuning effects. We introduce a Concept Consistency Graph (CCG) to model relationships between concepts in the selected data. Here, a concept refers to a key knowledge unit semantic element extracted from instruction tuning sample. Concepts play crucial role in
5

---

Published as a conference paper at ICLR 2026
capturing the essential information within instructions and help to identify potential conflicts that could lead to a negative tuning effects. By managing relationships between concepts, we aim to maintain semantic consistency across the selected instruction tuning dataset, thereby reducing the risk of learning conflicting or irrelevant information during the recovery process. The formal definition for CCG is provided as follows:
Definition I (Concept Consistency Graph). A CCG is a graph G = (V, E) where the verlices V represent concepts, and an edge (v,, v3) E exists if concepts vs and vy co-occur in at least one instruction tuning sample without conflict.
For each new sample (zz y), we first extract its concept C(z, y) and then check for consistency: IsConsistent(z; [z, y) = Ve,, U/ E C(a,y) : (v,, v)) E E or {v,, 1y} & V. We only add samples that are consistent with the existing CCG, updating the graph with each addition. This approach ensures that we maintain coherent set of instructions, minimizing the risk of negative tuning effects by avoiding the introduction of conflicting concepts during the recovery training process. Furthermore, to mitigate the impact of rare nuanced conflicts that raw so-occurreceebbaed CCG may miss, we employ several lightweight enhancement schemes: (i) consolidating concept variants with simple semantic normalization (e.g.. merging paraphrases and surface-form variants); (ii) treating low-confidence cases conservatively by softly down- weightigg rather than hard-removing them; and (ii) providing an optional, low-hudget Natural Language Inference reranker for only top-risk pairs flagged by CCG. The full version of our algorithm is provided in Appendix B
3.5 TIME COMPLEXITY ANALYSIS
The theoretical analysis for PASER two folds: 1) time complexity analysis regarding efficiency; 2) eтror bound analysis regarding effectiveness. As the main bottleneck for real-world application. we first explore PASER's time complexity here.
Theorem 1.
The overall time complexity of PASER is ON log N + NC*), where N is the number of instructions in D, and C the maximum number of concepts in any instruction tuning sample.
Proof. We analyze each step of the algorithm in detail. The Cemantic-Structurre Recovery Instruction Clustering, including BenT embedding, Diffusion Kernel computation, and NMF Spectral Clustering, has dominant complexity of O(N log N). For the Capability Degradation Assessment step, computing JSD for each sample and calculating CDS for each cluster take ((N) time in total. The Inter-capability Budget Allocation, which involves allocating the budget to clusters, has a time complexity of O(K), where K is the number of clusters. However, since K < N, this step does Sifcciennc--rrieer Sample Selection,
not dominate the overall complexity. During Intra capabiliyy for each cluster Ck. we perform sorting by JSD (O([c) log (cx l)), iterate through sorted samples (O(Ie))) D), perform consistency checks (IsConsistent, OCC²) per sample), and update the CCG (O(C2) per sample). Considering all clusters, this step's 's total complexity is O(N log N + NC Thus, the overall time complexity is dominated by the clustering step and the intra-capability sample selection step. Therefore, the total time complexity is O log N + NC²).
In practice, C is often much smaller than N (C N) and can be considered as constant factor for large N. Thus, we can simplify the complexity to O(N log N). This analysis demonstrates that PASER is computationally efficient and scalable for large instruction tuning datasets. In addition, to guarantee the performance effectiveness, we also conduct a comprehensive theoretical error bound analysis for PASER Appendix C
EXPERIMENTS
4.1 EXPERIMENT SETUP
Target LLMs The experiments are mainly performed on several open-source popular English LLMs: CDMMAA2-7/13//0000 (Touvron et al., (002) (hf version), LLaMA3-8B (Dubey et al., (224)uintttuec version), and bilingual LLMs: Baichuan2-7B/13. both English and Chinese. Besides, several more recent LLMs are also adopted in Appendix !]
(Yang et al. 2023) version), which support
Instruction Tuning Datasets As for the original recovery post-training data, choose two different- size instruction tuning datasets: Alpaca (Taori et al. 2023) and LaMini (Wu et al., (024).. Alpaca contains 52K satuuctoon-oolooing samples generated using OpenAT's text-davinci-003 model based

---

Published as a conference paper at ICLR 2026
Table Recovery performance of different instruction tuning data selection methods under various pruning schemes on LLaMA2-7B model. The 'bold' represents the best performance under the same pruning scheme. Here, the Alpaca is taken as the original dataset.
Pruning
Post training
Recovery
WikiTes21
PTBL
BoclQ
PIQA
HellsSwag
WinoCrande
ARC-c
ARC-c
OBQA
Average
wio praning
w/o Training
12.02
22.14
71.15
13.40
72.79
67.17
09:36
40170)
21.30
62.91
Training
20.34
38.81
61.37
76.61
65.36
60.22
63.13
37.37
39.40
57.78
Hull Data
736.42
1273.10
37.83
53.21
25.29
29.00
35.64
LLM Pruner
Random
93.77
180.62
57.61
64.37
45.39
55.87
43.78
31.94
34,90
47.69
Instruction Mining
23.31
40.63
61.59
75.68
66.08
60.71
62.34
37.96
59.20
57.65
19.76
63.55
77.23
67.21
63.46
40.00
Nuggets
20.02
35.19
63.62
77.43
67.36
61.08
63.77
37.64
39.90
58.60 69.69
PASER
16.40
26.35
67.25
77.29
68.98
66.97
67.84
39.54
39. 80
61.10
Training
41.53
80.07
65.55
66.87
54.16
63.38
58.46
34.55
36.90
51.27
milio255g
SliceOPT
Instruction Mining
Random
41.86
39.75
71.28
74.92
67.57
66.89
68.21
68.93
56./1
35.79
65.76
64.56
61.99
60.23
36.23
35.47
37.60
37.60
56.39
55.54
IFD
37.75
67.48
69.23
70.54
58.38
67.12
63.75
37.18
38.40
57,80
Nuggets
71.21
38.60
PASER
12.24
21.53
72..75
79.84
73.92
69.18
71
41.82
41.30
64.31
w/o Training
42.10
76.85
09.30
71.99
53.06
62.75
60.94
28.07
34,60
54.39
Hull Data
27.63
65.
Wanda
35.98
36.20
qursiry-2:4
Instruction Mining
31.47
57.17
70.61
73.85
61.27
64.13
64.72
35.79
36,80
57.88
IHD
25.82
46.78
71.00 015
75.57
64.15
65.38
66:55
35.63
37,60
59.42
Nugget PASER
23.98
14.13
27.22
70.77
77.87
71.78
66.26
68.30
39.04
40.10
62.02
w/o Training
19.26
36.41
71.22
75.60
62.85
66.06
69.11
36.86
37,80
59.93
Full Data
25.83
47.26
69.
74.15
59.68
67.76
63.74
39.59
37,80
58.83
sparsht=500
SpansGPTT
Instraction Mining
24.08
45.51
70.50
74.17
61.91
65.40
59.76
67.73
36.49
37,10
59.13
IFD
21.19
40,05
71.06
75.13
62.79
65.72
68.80
36.23
37,20
59.56
Naggets
16.21
28.95
71.64
75.67
63.33
66.05
36.60
37.40
60.03
PASER
13.33
74.79
78.38
39.40
on 175 human-written seed tasks. LaMini contains total of 2.58M pairs of instructions and responses synthesized with gpt-3.5-turre based on several existing resources of prompts, including self-instruct (Wang et al., 2202)), P3 (Sanh et al. 2022). FLAN (Longpre et al.) 2023) and Alpaca [Taori
jet al. 2023). This dataset can help evaluate the effectiveness and efficiency in large-scale settings.
Base Pruning Schemes Different pruning schemes incorporated into experiments to explore the applicability of PASER, ranging from structured pruning methods: LLM--runer (Ma et al. (2023
SliceGPT (Ashkboos et al. 2024), semi-structured pruning method: Wanda (Sun al. (024). and unstructured pruning method: SparseGPT (Frantar & Alistarh (203)).
Instruction Tuning Data Selection Baselines In addition to the random data selection, also
compare PASER with several recent general instruction tuning data selection baselines: Instruction Mining (Cao et al.). IFD (Li et al. 2024a), Nuggets (Li et al. 2024b). Note none of these baselines are customized for post-pruning recovery training scenario. Besides, the evaluation performance of recovery training with the full original dataset also compared.
Evaluation Datasets and Tasks To thoroughly evaluate the performance of recovered LLMs, we employ seven common sense reasoning dotasets:BoolQ (Clark al. 2019 PIQA (Bisk et al. 2020). HellaSwag (Zellers et al. (2019). WinoGrande (Sakaguchi et al. (022). ARC- Easy (Clark et al. 2018). ARC-Chailenge (Clark et 2018), and OpenbookQA (Mihaylov et al.. 2018). the practice, we rely on the open-source library" to implement the evaluation, where the model needs to rank the choices in the multiple choice tasks or generate the answer in the open-ended generation tasks. The whole process is conducted in the zero-shot manner. Besides, follow (Ma et al.) (023) to evaluate the language modeling capability with the zero-shot perplexity (PPL) analysis on WikiText2 (Merity et al. 2022) and PTB (Mareus et al. 1993). For enhancing evaluation comprehensivenesss also conduct experiments on mathematical problem solving and code generation tasks, as shown in Appendix D and respectively.
4.2 EXPERIMENT RESULTS AND ANALYSIS
Recovery Performance for Different Pruning Schemes We evaluate the recovery performance of ILLaMA2-7B using different instruction tuning data selection methods under structured pruning, semi-structured pruning, and unstructured pruning, respectively. According to the results Table
7

---

Published as a conference paper at ICLR 2026
Table 2: Recovery performance of different instruction tuning data selection methods on different target LLMs under LLM-Pruner scheme. The 'bold' represents the best performance on the same target LLM. Here, the Alpaca is taken as the original dataset. The "Reason" indicates the averaged performance on 7 common sense reasoning datasets.
Model
Benchmnark
wio pruning
wio Training
Full Data
Random
Instruction Mining
IFD
Nuggets
PASER
WiRlTex:2]
11.58
73.52
27.74
39.85
44.37
38.61
33.50
21.67
LLMMA--13
mati=50%%
PTBL
20.24
151.19
45.08
76.20
80.82
73.25
61.26
35.09
Reasom"
64.78
48.86
56.40
54,62
54.09
54.77
55.25
57.62
WiRITex12]
8.92
46.81
31.76
25.34
23.16
22.87
19.63
12.35
LLaMA2-70B
ratio=50%
PTBL
15.59
92.36
56.83
48.72
43.45
43.68
36.24
21.82
Reason"
71.72
61.14
65.56
64.03
66.74
67.27
67.73
69.62
LLaMA3-SB
WiklTexr2]
7,36
13.47
9.58
12.52
13.25
11.04
10.31
8.09
ratio=25%
PTBL
16.73
22.17
23.47
Reason"
70.14
63.45
67.84
65, 50
65.47
66.64
67.30
69.83
Baichum22-7B
WikilexiZ
14.42
25.29
27.04
34,24
16.92
ratio=25%
PTB
26.78
53.34
35.81
46.83
60.93
37.81
37.65
30.76
Reason
56.33
$7.39
57.09
54.78
$7.36
57.84
Baichman2-133 13B
WikTTex22]
11.23
58.41
24.35
40.44
36.82
33.45
28.96
14.62
rati==50%
PTBL
42.68
76.57
70.45
Reason
67.25
57.59
61.64
59.12
59.38
60.34
61.09
63.75
directly employing full data can indeed bring the sub-optimal recovery performance, especially under the LLLM-Pruner. This is because the full version of data contains some irrelevant or conflicting information for capability recovery, resulting in the negative tuning effects during the training phase. Besides, considering pruning process has reduced the model's s structural capacity, can hardly satisfy all objectives present in the full dataset simultaneously. Training on full data forces the model to fit tasks beyond its representational bandwidth, which causes: overfitting to easy or dominant domains, forgetting of minority but critical capabilities, and instability in multi objettto optimization. Thus, performance becomes worse than simply leaving the pruned model untouched. Meanwhile, even the general instruction tuning data selection methods like IFD and Nuggets can bring better reasoning and language model performance than full data and random in most cases, validating the necessity of conducting recovery data selection. Furthermore, we can find that previous selection methods can hardly help model recover to the level of unpruned status, under the limited data budget. However, PASER can not only outperform baselines, but also reduce the averaged reasoning performance degradation to less than 3% under M-rrrrr Wanda, and SparseGPT. Especially, when pruning LLaMA2-7B with SliceGPT, our PASER can improve the average reasoning performance 64.31, higher than the unpruned model. Besides, its zero-shot perplexity on WikiText2 and PTB is also lower than unpruned model slightly. This suggests that allocating recovery budget according to capability degradation levels and prioritizing most-affected samples exhibit the potential of help pruned LLM recover to the capability level of unpruned status. Besides, PASER can also be extended other LLM post-compression scenarios, like the post-quantization recovery and post-distillation recovery. The corresponding results and analysis are provided in Appendix [nd [H] respectively. Also, we have explored the recovery effects of different instruction tuning data selection methods on more mnowledge-extensinn common sense reasoning tasks Appendix [T] In Appendix we provided more detailed comparison between our PASER and general data selection methods, including both conceptual differentiation and empirical evidence.
Robustness over Various Target Large Language Models To validate whether PASER maintain the robust effectiveness among various target LLMs, we conduct the experiments on LLaMA2- 7B/13B/70B, LLaMA3-8B, Baichuan2-7B/133. under LLLM-Prunes According to results in Table [ we can first observe the model capability under high pruning ratio (50%) is hard to recover to unpruned level, especially for relatively smaller model like LLaMA2-13B and Baichuan2-13B. Though, PASER can still outperform random and best-performing data selection baseline, Nuggets, by 4.41 and 2.31 points, respectively on average. Especially, for LLaMA2-700 control the averaged reasoning performance degradation to less than 3%. This can be explained that the structure redundancy in 70B model is relatively higher, paving the way for effective recovery through data selection under high pruning ratios. As for the second smallest model, LLaMA3-88P PASER can recover the reasoning performance to the 99.56% of the unpruned status, which further demonstrates the robustness of PASER over different target LLMs. Finally, the performance of various recovery methods including PASER on Baichuan2-1 7B is not satisfying enough given only 25% pruning ratio, which be attributed to that the pruning process has severely damaged the model internal structure (c.g., attention heads specialized for corresponding capabilities). In such cases, recovery via post-training, even with carefully selected instructions, can be less effective
our PASER can
8

---

Published as a conference paper at ICLR 2026
because the underlying model expressivity has already been impaired. Supplementary experiments on more recent LLMs like Qwen2.5 and Qwen3 can be seen in Appendix
Recovery Performance with Different In- struction Tuning Datasets In addition to the recovery performance Alpaca shown in Ta- I we also explore the corresponding re- sults on another larger dataset, LaMini. Es- pecially, considering the space limitation and more severe performance degradation of struc- tured pruning schemes, we provide the exper- iments results on L..M-rrnner and SliceGPT on Table 3 From this table, we can observe that PASER can still consistently outperform other data selection and random methods. Be- sides, comparing the results in Table [ and 3] it can be found that improving the data scale (from 10K to 10K samples) indeed facili- tates the recovery performance though the sig- nificantly increased computational overhead, which is consistent with the Scaling Law (Ka- plan et al. (020). We can also notice that the performance of full data on LaMini is rela- tively competitive, which is because the pro- portion of conflicting or negative data for recovery is much lower than that in Alpaca.
Table 3: Recovery performance of different instruc- tion tuning data selection methods under two struc- tured pruning schemes on LLaMA2-7B model. The 'bold' represents the best performance under the same pruning scheme. Here, the LaMini is taken as the original dataset. The "Reason" indicates the averaged performance on 7 reasoning datasets.
Pruning
Recovery
training
WikiTenr2
PTB.
Reason?
wlo pruning
w/o Training
12662
22.14
62.91
w/o Training
20.34
38.81
57.78
Hull Data
16.28
27.12
62.68
LLM- M--rreerr r5tio-25%
Random
18.40
32.15
60.93
Instruction Mining
17.83
28.87
60.76
IFD
18.54
31,23
60.65
Nuggets PASER
60.99
13:45
22.63
63.79
W/o Triinin
44.51
S0.07
34.27
Full Data
SliceGPT 25t
Random
39.86
70.92
56.68
Instruction Mining
37.75
67.28
57.53
ITD
25.75
53.48
58.94
Nuggets
60
PASER
14.27
23.53
68.74
Ablation Study To validate the contribution of each component in PASER, we conduct comprehensive ablation studies. Specifically, we evaluate three variants: (1) PASER w/o S2RIC: replacing semantic- structural clustering with random clustering while keeping other modules unchanged; (2) PASER w/o CDAIS: randomly sampling equal number of instructions from each cluster instead of using capability degradation-aware selection; (3) PASER w/o NTEM: removing the negative tuning effects mitigation module. The results in Table F demonstrate that all three components contribute positively to model recovery across different pruning schemes. The semantic- struutuclt clustering effectively identifies capability-specifie instruction groups, leading to 0.18-1.43 points improvement in reasoning performance. Its removal causes degradation in both language modeling (increased perplexity) and reasoning tasks, particularly evident under structured pruning schemes like LLM-Pruner and SliceGPT. The capability degradation- aware selection mechanism enhances recovery efficiency through adaptive budget allocation, contributing 0.29-1.63 points improvement in reasoning tasks while maintaining stable language modeling performance. Negative tuning effects mitigation shows significant impact (0.68-2.39 points improvement), especially under high pruning ratios, highlighting its importance in preventing conflicting information during recovery training. These improvements are consistently observed across different pruning schemes, with particularly pronounced effects in structured pruning where capability degradation tends to be more severe and uneven. More detailed ablation study results and analysis are provided in Appendix L Besides, we also empirically explore the technique selection of PASER's 's each component in Appendix
Recovery Post-training Efficiency Analysis To highlight PASER's advantages on recovery post- training efficiency, we conduct the experiments under different data budgets B and different datasets and record the corresponding averaged reasoning performance and training time in Figure 2 From the first and third subfigures, we can observe that PASER can obtain best recovery performance under different B/N on Alpaca and LaMini. Interestingly, in the first subfigure, when rising B/N from 0.3 to 0.4, the reasoning performance of Random even decreases. It is because expanding data scale also introduces the conflicting or negative data existing in the original datasct. From the second and fourth subfigures, PASER consistently consumes the least training time, which can be attributed to the efficiency-driven sample selection process in PASER. This advantage can more obvious under low B/N like 0.02 on LaMini. This is because increasing data budget will force PASER to select some relatively more time-consuming samples given the fixed original dataset, weakening its efficiency superiority. Besides, we also study the time consumption during data selection in Appendix O

---

Published as a conference paper at ICLR 2026
Table 4: Ablation study results LLaMA2-7B for each component under different pruning schemes. The "Reason" indicates the averaged performance on 7 common sense reasoning datasets.
Ablation Variant
LLM Franer( (2516)
SlxeGPTT (25%)
Wanda (2:3)
SpaneePP (5076)
(S07))
PIBL
Reason]
Wikllestz)
PTBB]
Reasoot
FTB
Keason
Wikriesz2
PTB.
Reason
S'RIC:
1483
25.42
15.84
weo NTEM
19.82
35.60
39.25
15.37
27.81
61.92
16.79
31.52
61.34
15.91
PASER
16.40
26.35
61.10
12.24
21.53
64.31
14.13
27.22
62.02
13.33
23.77
62.78
Alpaca: Reasoning Performance
Alpaca: Time Consumption
Reasoning Parformance
Training (hour)))
Time Consumption
Averuge nerformance
60
8G
10 Recovery Training
6
8
8
38
Figure 2: Average reasoning performance and recovery post-training time consumption curves corresponding to different instruction tuning data selection methods. The left two subfigures are for Alpaca while right two subfigures are for LaMini.
5 CONCLUSION AND FUTURE WORKS
Recovery post-training has been an important procedure after large language model pruning to restore the critical capabilities. Previous works directly utilize the full instruction tuning dataset, facing high computation cost, risks of untargeted recovery, and negative tuning effects. In this work, propose the post-training data selection method for efficient pruned model recovery. According to capability degradation degrees, allocate selection budget across different capability data obtained through semantic-stieg -strutuurl clustering. We then select samples where model output behavior has been severely affected while considering computation cost, and introduce concept consistency graph to mitigate negative tuning effects. Extensive experiments on different LLMs and theoretical analysis demonstrate the effectiveness of our framework. Future work will explore other optimization approaches like data augmentation and revision to further improve recovery efficiency.
ETHICS STATEMENT
The development and deployment of technologies like PASER for efficient recovery of pruned large language models necessitates careful consideration of ethical implications. While PASER contributes to reducing environmental impact and potentially democratizing AI access by lowering computational requirements, also raises concerns about potential misuse, bias amplification, and privacy. It's crucial to remain vigilant about these risks, implement robust safeguards, and maintain transparency the recovery process. Continuous monitoring for fairness and bias in model outputs essential, as responsible deployment with appropriate human oversight, especially in high-stakes applications. As the field evolves, ongoing ethical assessment and dialogue with stakeholders are vital to ensure that advancements in large language model efficiency contribute positively to society while minimizing potential harm. Ultimately, the goal should be to harness the benefits of improved model recovery techniques like PASER while proactively addressing the complex ethical challenges they present.
REPRODUCIBILITY STATEMENT
We provide a code repository https: hhg.btthuuuu.... hubb com/ Bokwoowaaa....... wookwaiHo/Pae containing the imple- mentation PASER, along with detailed instructions for reproducing our experiments. All datasets used in this work, including Alpaca and LaMini for instruction tuning and standard reasoning/QA benchmarks (e.g.. BoolQ, PIQA, ARC, HellaSwag, WinoGrande, OpenbookQA), are publicly avail- able. Hyperparameters, pruning schemes, and evaluation protocols described in the main text and appendices to ensure clarity. Our theoretical analysis, ablation studies, and efficiency measurements also documented to facilitate independent verification
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the Early Career Scheme (No. CityU 21219323) and the General Research Fund (No. CityU 11220324) of the University Grants Committee (UGC), the NSFC Young Scientists Fund (No. 9240127), and the Donation for Research Projects (No. 9229166 and No. 9220187).
REFERENCES
Ishika Agarwal, Krishnateja Killamsetty, Lucian Popa, and Marina Danilevsky. Delift: Data efficient language model instruction fine-tuning. In The Thirteenth International Conference on Learning Representations, 2025.
Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation baasee adaptive structured pruning for large language models. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan (eds.), Thirty-Fighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty- Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp. 10865-10873. AAAI Press, 2024. doi: 16.9/ALAAALL, V311102288000
URL httpppps://.o..oo///////
169 aaai. v38110 28960
Saleh Ashkboos, Maximilian L.. Croci, Marcelo Gennari Do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: /ttp:////ppeeeeeeeeeeeeee net forum?id= vxxardq6db
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprini arXiv:2 077332, 2021.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Yihan Cao, Yanbin Kang, Chi Wang, and Lichao Sun. Instruction mining: Instruction data selection for tuning large language models.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
Tianyi Chen, Tianyu Ding, Badal Yadav, Ilya Zharkov, and Luming Liang. Lorashear: Efficient large language model structured pruning and knowledge recovery. arXiv preprint arXiv:2310. 18356. 2023.
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. Proceedings of NAACL-HLT, pp. 2924-2936, 2019.
Ming-Wei
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1803. 00457, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, al. Training verifiers solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 8493-8501 2022.
11

---

Published as a conference paper at ICLR 2026
Chris Ding, Xiaofeng He, and Horst D Simon. On the equivalence of nonnegative matrix factorization and spectral clustering. In Proceedings of the 2005 SIAM international conference on data mining. pp. 606-610. SIAM, 2005.
Harry Dong, Beidi Chen, and Yuejie Chi. Prompt-prompted adaptive structured pruning for efficient Ilm generation. In First Conference on Language Modeling, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Abhinav Dutta, Sanjeev Krishnan, Nipun Kwatra, and Ramachandran Ramjee. Accuracy is not all you need. arXiv preprint arXiv:2407.09141, 2024.
Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems, 35:4475-444 -4488, 2022.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In Andreas Krause, Emma Brunskill, Kyunghyun Cho. Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 10323-10333 PMLR, 2023. URL htwtttss//prrrceednngee ml press press/ v202 2202/Erantar23a.h5
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Optq: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference Learning Representations, 2023.
Bent Fuglede and Flemming Topsoe. Jensen-shannon divergence and hilbert space embedding. In International symposium oninformation theory, 2004. ISIT 2004. Proceedings. pp. 31. IEEE, 2004.
Yuxian Gu, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Miniplm: Knowledge distillation for pre-training language models. In The Thirteenth International Conference Learning Representations.
Zhiyu Guo, Hidetaka Kamigaito, and Taro Wanatnabe. Dependency-awar semi-structured sparsity of GLU variants in large language models, CoRR, abs/22405.01944 2024, doi: 10.48550/ARXIV. 2405.01943. URL httthhht////e/
wrg/ /1. 48550/ 2405. 01943.
Moritz Hardt and Yu Sun. Test-time training on nearest neighbors for large language models. The Twelfth International Conference on Learning Representations, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference Learning Representations.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Andrew Ilyss, Sung Min Park, Logan Engstrom, Guillaume Leclere, and Aleksander Madry. Data- models: Predicting predictions from training data, arXiv preprint arXiv:2202.00622. 2022.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.044 240080 2024.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tenybert: Distilling bert for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4163-4174, 2020.
12

---

Published as a conference paper at ICLR 2026
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint ar XXiv:2001.08361,, 2020.
Jongwoo Ko, Seungjoon Park, Yujin Kim, Sumyeong Ahn, Du-Seong Chang, Euijai Ahn, and Se- Young Yun. NASH: A simple unified framework of structured pruning for accelerating encoder-decoder language models. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, De- cember 6-10, 2023, pp. 6076-6093. Association for Computational Linguistics, 2023. doi: 1116653///2023FFIIIIIGGG SOANNDDINGG-EENLLPAAAA 404.P.404. 2023. f Findiiinnnnniniinn p. 404
URL httpp:://di..... http///...rg/11..1111..
Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se- Yun. Distilim: Towards streamlined distillation for large language models. International Conference on Machine Learning, pp. 24872-24895, PMLR, 2024.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathemati- cal statistics, 22(1):79-86, 1951.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng. Jianzong Wang. Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting IIm performance with self-guided data selection for instruction tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume
1: Long Papers), pp. 7595-7628, 2024a.
Yunshui Li, Binyuan Hui, Xiaobo Xia, Jiaxi Yang, Min Yang, Lei Zhang, Shuzheng Si, Ling-Hao Chen, Junhao Liu, Tongliang Liu, Fei Huang, and Yongbin Li. One-shot learning as instruction data prospector for large language models. Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 4586-4601. Association for Computational Linguistics, 2024b. URL https: http://///caaaahhhhooooeo org 2024. accllclonnnoooo 252.
Shihao Liang, Runchu Tian, Kunlun Zhu, Yujia Qin, Huadong Wang, Xin Cong, Zhiyuan Liu, Xiaojiang Liu, and Maosong Sun. Exploring format consistency for instruction tuning. Transactions on Machine Learning Research.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of machine learning and systems, 87-100, 2024.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pp. 22631-22666 PMLR, 2023.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances neural information processing systems, 36:21702-21720, 2023.
Srikanth Malla, Joon Hee Choi, and Chiho Choi. COPAL: continual pruning in large language generative models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. peattts::/openzeeeee net/
Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):31--330, 1993.
13

---

Published as a conference paper at ICLR 2026
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359- 17372, 2022.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2022.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct clectricity? new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 2381-22222 22991, 2018.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. Crosslingual generalization through multitask finetuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15991-16111, 2023.
Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marein Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Moichanov. Compact language models via pruning and knowledge distillation. arXiv preprint arXiv:2407.16679 2024.
Gunho Park, Minsub Kim, Sungjae Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Byeong- wook Kim, Youngjoo Lee, Dongsoo Lee, et al. Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models. In The Twelfth International Conference on Learning Representations.
Nils Reimers and Iryna Gurevych. Sentence-bert. Sentence embeddings using siamese bert-networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLE 2019, Hong Kong, China, November 3-7, 2019, pp. 3980-3990, Association for Computational Linguistics, 2019. doi: 10.18653/V//D19- 1410. https:///// wtt.://.011...111///11000
1et...1663/v//// 1410
HSVNS Kowndinya Renduchintala, Sumit Bhatia, and Ganesh Ramakrishnan. Smart: Submodular data mixture strategy for instruction tuning. In ACL (Findings), 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106. 2021.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, et al. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. In ICLR 2022-Tenth International Conference on Learning Representations, 2022.
Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, and Jae-Joon Kim. SLEB: streamlining llms through redundancy verification and elimination of transformer blocks. In Forty- first International Conference Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL met//ooumm Poo.ddddxxPPutootoc
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqat A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4149-4158, 2019.
14

---

Published as a conference paper at ICLR 2026
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Stanford alpaca: An instruction-followeee LLaMA model. https: /ttp::///gihhlt gitb... com/
stanford alpaca 2023. Accessed on: Enter the date you accessed or used the dataset.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXhv:2307.09288, 2023.
Leonid Nisonovich Vaserstein. Markov processes over denumerable products of spaces, describing large systems of automata. Problemy Peredachi Informatsii, 5(3):64-7) 1969.
Jiahao Wang, Bolin Zhang, Qianlong Du, Jiajun Zhang. and Dianhui Chu. A survey on data selection for IIm instruction tuning. arXiv preprint orXiv:2402.05122 05100,1233520 2024.
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self eelf-generated instructions. In The 61 Annual Meeting The Association For Computational Linguistics, 2023.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners, In International Conference on Learning Representations.
Minghao Wu, Abdul Waheed, Chiyu Zhang, Muhammad Abdul- Abdul-Mageed, and Alham Aji. Lamini- Im: A diverse herd of distilled models from large-scale instructions. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: 2024.
Long Papers). pp.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRevvew.net, 2024a. URL httts::/
net/ Eorummid=09911 0daee00pp
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: selecting influential data for targeted instruction tuning. In Proceedings of the 41st International Conference on Machine Learning, pp. 54104-54132, 2024b.
Cong Xu, Gayathri Saranathan, Mahammad Parwez Alam, Arpit Shah, James Lim, Soon Yee Wong, Foltin Martin, and Suparna Bhattacharya. Data efficient evaluation of large language models and text-to-image models via adaptive sampling. arXiv preprint arXiv:2406.1553 15527, 2024a.
Peng Xu, Wenqi Shao, Mengzhao Chen, Shitao Tang. Kaipeng Zhang, Peng Gao, Fengwei An, Yu Qiao, and Ping Luo. BESA: pruning large language models with blockwise parameter-efficient sparsity allocation. In The Twelfth International Conference on Learning Representattioss, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024b. URL URUpattss//pppeeeeeet net fert/forum?2 id gC6JTE333111
Wenda Xu, Rujun Han, Zifeng Wang, Long Le, Dhruv Madeka, Lei Li, William Yang Wang, Rishabh Agarwal, Chen-Yo Yu Lee, and Tomas Pfister. Speculative knowledge distillation: Bridging the teacher-student gap through interleaved sampling. In The Thirteenth International Conference Learning Representations.
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10300 2023.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 15X 15/15 15, 2024.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post- bost-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 15:27168-27188 2022.
15

---

Published as a conference paper at ICLR 2026
Lu Yin, You Wu, Zhenyu Zhang, Cheng- Yu Hsieh, Yaqing Wang. Yiling Jia, Gen Li, Ajay Kumar Jaiswal, Mykola Pechenizkiy, Yi Liang, Michael Bendersky. Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (OWL): A missing secret sauce for pruning 1lms to high sparsity. In Forty-first International Conference on Machine Learning. JCML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. utthttss: http::///pppeeeveeeeeee ew. net. Forum?id- ahEm312P6w
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800 2019.
Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, and Bohan Zhuang. Loraprune: Structured pruning meets low-rank parameter- -effetee-efffciett finettning. In Findings of the Association for Computational Linguistics ACI. 2024, pp. 3013-3026, 2024.
Weilin Zhao, Yuxiang Huang, Xu Han, Zhiyuan Liu, Zhengyan Zhang, Kuai Li, Chen Chen, Tao Yang, and Maosong Sun. Ca-lora: Adapting existing lora for compressed Ilms to enable efficient multi-tasking on personal devices. In First Conference on Language Modeling.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
16

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
Related Works
3
Methodology
Problem Formulation
3.2
Semantic-Struce Structural Recovery Instruction Clustering
3.3
Capability Degradation Aware Instruction Selection
3.4
Negative Tuning Effects Mitigation
3.5
Time Complexity Analysis
Experiments
Experiment Setup
Experiment Results and Analysis
Conclusion and Future Works
10
Supporting Materials for Introduction Part
19
Pseudocode
19
Theoretical Error Bound Analysis
20
Problem Formulation Revisited
20
Theoretical Framework
20
Detailed Derivation
21
Interpretation of the Error Bound
23
D Evaluation on Mathematical Problem Solving Tasks
24
Evaluation on Code Generation Tasks
25
Evaluation on Other Knowledge-xxtensive Tasks
25
Extended Experiments on Post-quantization Recovery Training
26
Extended Experiments on Post-distillation Recovery Training
27
Experiments Recovery Training with Knowledge Distillation
27
Experiments on More Recent LLMs
29
Discussion on Comparison with General Data Selection Methods
29
K.1 Q1: Why General Data Selection Methods do not Suffice for Pruned-LLM Recovery?
30
17

---

Published as a conference paper at ICLR 2026
K.2 Q2: How PASER Difters from Previous General Data Selection Methods?
30
Q3: What is Specifie Difference between PASER and Gradient-based Sampling?
K.4 Q4: Can PASER Benefit from Gradient-derived Dense Information?
32
Detailed Ablation Study Results
32
M Hyperparameter Robustness Analysis
33
M. Sensitivity to Embedding Dimension
33
M.2 Robustness Temperature Parameter
33
N
Empirical Exploration for Component Selection
34
Exploration on Other Possible Instruction Embedding Models
N.2
Exploration on Other Possible Dimensionality Reduction Methods
34
N.3
Exploration on Other Possible Clustering Methods
36
N.4
Exploration on Other Possible Capability Degradation Assessment Methods
36
N.5
Exploration on Other Possible Budget Allocation Strategies
37
0 F iine grained Data Selection Time Consumption Analysis
37
Case Study for Recovery Instruction Clustering
38
P.I Cluster I:
Basic Factual Knowledge and Information Retrieval ................................
38
P22
Cluster 2:
Language Understanding and Translation
39
P3
Cluster 3: Logical Reasoning and Problem Solving
39
P.
Cluster 4: Creative Writing and Text Generation
39
P5
Cluster 5: Summarization and Information Extraction
39
Visualization for Recovery Instruction Clustering
39
Case Study for Negative Tuning Effects Mitigation
40
R.
Constructing Initial Concept Consistency Graph
40
R.2 Evaluating New Samples
R.2.1 Sample 1: Accepted
41
R.2.2 Sample Rejected
42
R.
Conclusion
42
Analysis on Pruning Severity and PASER Recoverability
43
Implementation Details and Hyperparameter Settings
43
I Limitation Analysis
44
Declaration LLM Usage
45
18

---

Published as a conference paper at ICLR 2026
A SUPPORTING MATERIALS FOR INTRODUCTION PART
We provide the performance comparison in 3 supports the claim in the second paragraph of Section []] about the recovery performance deterioration. From this figure, employing the full version of recovery data or uniformly split subset to conduct recovery training can hardly achieve satisfying performance.
find that
Besides, we also provide the evidence for the uneven deterioration of different LLM capabilities during the pruning process (corresponding to the third paragraph in Section D From the Figure we can observe that the four critical capabilities: language modeling, common sense reasoning, mathematical problem solving, and code generation exhibit significant difference on the performance degradation degrees. This phenomenon exits widely in the provided four pruning settings, which implies the necessity of performing targeted and balanced capability recovery. In fact, even among the various common sense reasoning tasks, this kind of uneven capability deterioration effect is still evident.
LLaMA2-78 (pruning ratio=25%)
LLaMA2-136 (pruning ratio-50%)
0.800
0.48
8.75
CAT
8.35
0.28
agens
Agaeme OI
Ateore DV
Aieere
Aleene di
Aan Labtte
A8 Ladtoor d
La
Letter 14
1/ castiol 4B
Figure 3: Average performance on seven common LLM reasoning evaluation tasks after recovery post-training with different data. The numbers in brackets represent the group index of the data subset the full dataset. Unprumed indicates the original model and w/o Training indicates the pruned model (using LLM--Pruner (Ma et al. 2023) without the recovery post-training.
Capability Degradation Under Different Pruning Settings
Language Modeling
Normalized Performance Degradation (%)
Common Sense Reasoning
30
Mathematical Problem Solving
Code
25
20
15
10
LLM-Pruner (25%)
SliceGPT (25%)
Wanda (2:4)
SparseGPT
(50%)
Pruning Settings
Figure 4: Normalized performance degradation degree(%) on four various capabilities under four LLM pruning settings.
B PSEUDOCODE
We provide the full version of our pseudocode in Algorithm
19

---

Published as a conference paper at ICLR 2026
Algorithm 1 Post-training data Selection for efficient pruned large language model recovery (PASER)
Input: M,: original model, Mp: pruned model, D: instruction tuning dataset, B: data budget, U: computational cost budget
procedure PASER(M Mp. D, U)
C' XMMFSperraaClusteinn((eeelllll/
e D))
for C₁ C' do
CD)((=)) t- CCmppucccppliityyyggaaaaaaaaaoa Mo: Mp)
end for
{nu} t AllocateSampee(((DS((a))))}
S + 0, G + InitializeCCG(()
for Cj E C* do
L f- SonBByIESDescendinn((a/
i + 0, count + 0
while count < ni and < [Tx] do
(ar, < T [i]
IsConsistentt(z, zz, y, G) and
Σ
ComputationalCost(e/ I U then
(e' (E'
S - Su
G UpdateCCG
count count end if i+1
end while end for return S end procedure
C THEORETICAL ERROR BOUND ANALYSIS
In this section, establish theoretical guarantees for PASER framework by deriving error bounds that characterize how close PASER's 's solution is to the optimal recovery solution.
C.1 PROBLEM FORMULATION REVISITED
Recall from Section 3.1 that our objective is find subset S D of instruction tuning data that minimizes the expected loss on downstream tasks:
S = arg
min SCD,
(())) 2,,))
(8)
where M,( is the recovered model after training on subset T is the distribution of downstream evaluation tasks, and C is a loss function.
C.2 THEORETICAL FRAMEWORK
To establish theoretical guarantees, we introduce the following assumptions: Assumption 1 (Performance Decomposition). The performance downstream tasks can be decom- posed as a weighted sum of performances across different capability clusters:
E Bieg)-r(L(M, = Σ We (ez.,) (S),
(9)
where Ct represents the k-th capability cluster, wi 2 0 its weight in the evaluation distribution, and _ 1.
wk
Assumption 2 (Capability Independence). The recovery effect one capability depends primarily on samples from the corresponding cluster, with limited cross-cluster effects. For distinct clusters c; and cj, and subsets S, and S, cjt
E R( (T(M, (S U.) v)) Beere) (T(M v)]l 8ig.
(10)
20

---

Published as a conference paper at ICLR 2026
where 61z 2 0 is small constant representing the cross-cluster influence.
Assumption 3 (Conceptual Consistency Monotonicity). Within a capability cluster, for conceptually consistent subsets, larger training subsets generally produce better recovery effects. Given Concept Consistency Graph G = (V, E), for any SA SB C Ck where all pairs of samples in SB are conceptually consistent according G:
E (a.))~e (S,, u)] S E(z.g)=cz (S(,, x. v)l.
(11)
Assumption (Sample Selection Submodularity), The recovery benefit exhibits approximate sub- modularity within each capability cluster. For conceptually consistent subsets S T C Ch and an element & E Ck AT that is conceptually consistent with all elements in T:
(elss) 2 A(e(T) ak IT's 8),
(12)
(x (I(M, U {e)),,,y) represents the marginal benefit of adding element e to subset S, and ar > a speeer--peiffcc constant.
where (elS)) (a.y)))=
(o (S,
Assumption 5 (Intra-Capability Selection Effectiveness)) For each capability cluster ct, let Sa be the subset of size ng selected by PASER and S the optimal subset of the same size. The intra-capability selection effectiveness of PASER is characterized by:
Bee.grve [L(M, (Sk), v)) S B(z.a)= [ (S), v)) + (1)+++11100 (1-7k) Ak.
(13)
where "ik E (0, 1] and A = -ec [L(M, (0), x. y)] - ((r.))=vx (S)) x. represents the maximum possible improvement, which guaranteed by above Assumption 4
Assumption (Inter- Capabiiity Budget Allocation Effectiveness). Given the limited cross- cluster
influence established in Assumption 2
suboptimal budget allocation across capability clusters cannot catastrophically impact overall recovery performance. Therefore, the effectiveness of PASER's s budget allocation across clusters can be bounded by:
wk wk- (a..B (MM) S wk BE..( (MM)))))) (1)) (14)
where 3 E (0, 1), St. is the optimal subset of size ny from cluster ck. (tnn is the optimal subset of size nt, from cluster Ck2 and {nf] k= is the optimal budget allocation across clusters under the same total budget B (Ek=1
[ng]k=]
Σ Vi = B). Thus, S (1()) S* n
Based on these assumptions, we establish the following theorem: Theorem 2 (PASER Error Bound). Let SPASER be the subset selected by PASER with budget B, S be the optimal subset of the same size, and 7 miny Ykk Then there exists constant C >> 0 such that:
(1.. x. & B(x.)- v)) -1- (T) C.
(15)
C.3 DETAILED DERIVATION
We now provide a step-by-step derivation of the error bound, explicitly showing how each assumption utilized.
Step 1: Decompose the performance into capability clusters using Assumption I
Bec.)) K
wk (( (SA,SsS)))
(16)
[(ew)-7(((M,(SS))
Σ wk (e (IM,
(17)
Step 2: Apply Assumption (Caaaiilli cluster:
(Capability Independence) to isolate the effect of samples from each
((z,g)~e (LM,)AA))) ((,w))aa (L.
Σ
Öug
(18)
jzk
Blz,g)a, (SM,
Σ
(19)
21

---

Published as a conference paper at ICLR 2026
where S₁ SPASER Ck is the subset selected by PASER from cluster Ck-
Similarly for the optimal subset:
E (a.y)~ou (SM, x. v)1 2 ( 2,
Σ
8kg.
(20)
jykk
Step 3:
Now we analyze the relationship between PASER's selection Sş and the optimal selection St within cluster Cg.
First, we utilize Assumption 3 (Conceptual Consistency Monotonicity). PASER's Concept Con- sistency Graph ensures that all selected samples in S₁ are conceptually consistent. For any subset SCCG C Sy that is conceptually consistent:
(a. (MM, x, v)] (L,,)) v,),
(21)
Now, apply Assumption 4 (Sample Selection Submodularity) analyze the greedy selection process employed by PASER within each cluster. For a conceptually consistent set, the greedy algorithm achieves an approximation ratio that depends on the degree of submodularity.
Let 5h be the set obtained by a greedy selection process that maximizes the marginal gain at each step while ensuring conceptual consistency. From the properties of approximated submodular functions (Assumption 4 and the guarantees of the greedy algorithm for such functions:
(a.,))> (L(M((SE) x, y)] S (x,g)><,, (M), 2; y)l + 77s- A..
(22)
where Th E [0, 1) is constant that depends on the submodularity gap parameter og and the size of the selection and A₃ is as defined in Assumption | [Intra-Capability Selection Effectiveness).
PASER's efficiency- driven selection further refines this greedy selection by prioritizing samples with higher Individual Efficiency Scores. This introduces an additional approximation factor Φ E [0, 1):
B(e,g)~c [L(M,((s) v)) S (ttx))_axL((M,M v)1 + ox AR.
(23)
Combining these equations, we get:
E(r,y//<a [L x, v)] & ((a,y)= (LM, + (M + Φu) D..
(24)
Setting (1 7k.) (1) )x). arrive the form presented in Assumption 5
(xz,)>~e (MM, y)] < -,,,) (MM, y)] + (1 (k) AR.
(25)
This explicitly shows how "/ incorporates both the submodularity properties (Assumption 4 and the conceptual consistency properties (Assumption 3 into our error bound.
Step 4: Combine Steps 2 and 3: (zz,)))e ( aa (.
Σ
(26)
jz
< Eaz (SM, x), + (1 7k) D. +
Σ
8kj-
jzk
(27)
Step Apply Assumption 5 (Inter-Capabilit Budget Allocation Optimality) compare PASER's
allocation with the optimal allocation:
Wk Be (S), <
Wk ((n.)) (5M)))))))
3) C₁,
(28)
where C1 is constant related to the maximum performance difference between different budget allocations. The factor quantifies how well PASER 's capability degradation-ave aware budget allocation approximates the theoretical optimal allocation.
-wwre
22

---

Published as a conference paper at ICLR 2026
Step 6: Combine Steps 1-5 and r ming 7k
K = (CN))-/(((M((SPABBB)).....))
Σ wk (L v,
(29)
K (ec,))=aa44(,,5)),.... + (1- A Σ
8kj
(30)
km
j
K
K
(( [e [MM (1-7))
Wk
A
WF
jkj
(31)
k=1
k=1
j
K
Ba (-- C2
(32)
k=1
Σ wi E (lay)>a (MM) (1--))
(33)
B (2 (0-00 (1- (1-7)) C2
(34)
< (x [L(M, 2C' (C (1- (C1+ C2),
(35)
where:
-2= = -k= wk Du represents the maximum possible improvement across all clusters,
. -3= ===1 Wk Ejtt duj represents the total cross-cluster effects.
Step 7: Cet C1 + C2, considering dæj - 0 as claimed in the Assumption 2
(MM v) S (TM.))-([((M((S)) v) (1 -y).c. C.
(36)
C.4 INTERPRETATION OF THE ERROR BOUND
The error bound provides several key insights, explicitly highlighting the role of each assumption:
This foundational assumption enables modular analysis by decomposing model performance into distinct capabilities, allowing PASER to target specific degraded capabilities with focused recovery resources.
Performance Decomposition (Assumption I):
By establishing bounded crass-capaiility influence, this assumption validates PASER's s independent optimization within each capability cluster, ensuring that improvements in one area don't significantly interfere with others.
Capability Independence (Assumption 2)
Conceptual Consistency Monotonicity (Assumption B) By ensuring monotonicity within concep- tually consistent subsets, PASER's CCG mechanism effectively mitigates negative tuning effects, making the selection problem more tractable.
Sample Selection Submodularity (Assumption 4 The approximated submodularity enables efficient greedy-like selection methods to achieve near-optimal results within each capability cluster.
Intra-Capability Selection Effectiveness (Assumption 5) The parameter y reflects PASER's efficiency in selecting the most valuable samples within each capability cluster. When 7 approaches 1, PASER's intra-cluster selection approaches optimality.
Inter-Capability Budget Allocation Effectiveness (Assumption 6) The parameter 3 reflects PASER's efficiency in allocating the budget across different capability clusters. When 3 approaches 1, PASER's budget allocation approaches the theoretical optimal allocation.
Combined Effect: The error term (1 - 37) indicates that when both selection effectiveness and allocation effectiveness are high, PASER's performance approaches the global optimal solution.
This theoretical framework explains PASER's 's strong empirical performance: by intelligently allocat- ing budget based capability degradation degree (improving (8) and selecting the most valuable samples within each cluster while maintaining concept consistency (improving 7), PASER achieves near-optimal recovery performance.
23

---

Published as a conference paper at ICLR 2026
Table 5: Recovery performance of different instruction tuning data selection methods on mathematical problem solving tasks under various pruning schemes. The 'bold' represents the best performance under the same pruning scheme.
JLM--runer (25%)
SliceGPT (25%)
Wanda (2:4)
SParseGiT (50%)
Recovery Method
GSM8K
Minerva
GSM8K
Minerva
GSM8K
Minerva
GSMSK
Minerva
Training
44.3
17.8
42.5
16.9
43.8
17.4
43.1
17.2
Full Data
46.5
44.8
18.3
45.9
18.7
45.2
18.5
Random
18.4
43.9
44.7
18.1
44.3
17.9
Instruction Mining
18.9
44.5
18.1
45.4
18.5
44.9
18.3
IFD
46.8
19.3
45.1
18.5
45.8
18.8
45.4
18.6
Nuggets
47.1
19.5
45.4
18.7
46.2
19.0
45.7
18.8
PASER
49.4
21.2
47.8
20.5
48.5
20.8
47.2
20.1
D EVALUATION ON MATHEMATICAL PROBLEM SOLVING TASKS
To validate the effectiveness of PASER beyond common sense reasoning tasks, we conduct additional experiments on mathematical problem solving capabilities. Specifically, we employ two widely- adopted mathematical problem solving benchmarks:
GSM8K (Cobbe et al. 2021); A dataset containing 8.5K high-quality grade school math word problems that test various mathematical problem solving capabilities, including arithmetic, algebra, and word problem solving.
Minerva Math (Lewkowycz et al. 2022): A comprehensive mathematical evaluation dataset covering diverse topics in mathematics ranging from arithmetic to calculus, with problems requiring multi-step reasoning.
The recovery performance under different pruming schemes is presented in Table From these results, we can observe that PASER consistently outperforms baseline methods across all pruning schemes on both mathematical problem solving benchmarks. The improvements are particularly significant under the LLM--runer scheme, where PASER achieves 5. and 3.4% absolute improvements over w/o Training GSM8K and Minerva Math, respectively, While different pruning schemes affect base performance levels, PASER maintains its effectiveness in recovery. For example, under more aggressive SparseGPT (50%) setting. PASER still achieves 4.1% and 2.9% improvements GSM8K and Minerva Math over w/o Training. Compared to Full Data training, PASER achieves better performance while using only 20% of the instruction data, demonstrating its efficiency in recovering mathematical problem solving capabilities.
These results, combined with the common demonstrate that PASER is effective across diverse tasks. The strong performance on mathematical tasks particularly noteworthy given that these problems often require precise, step-by-step reasoning and have less tolerance for errors compared to common sense reasoning tasks. This validates the effectiveness of our capability degradation score in identifying and prioritizing recovery for severely affected capabilities, even in domains requiring high precision.
reasoning results presented in the main paper,
Table 6: Recovery performance of different instruction tuning data selection methods on code generation tasks under various pruning schemes. The 'bold' represents the best performance under the same pruning scheme. 'P@k indicates @Pass@k'.
LLM-Pruner (25%)
SliceOPT (25%)
Recovery Method
Human
MBPP
HumunEva
MBPP
P@
P@
P@
P0
Pe 100
P@
P@ 100
P0
Truining
3.4
6.2
10.4
15.3
24.6
3.11
5.9
8.7
6.2
T1.K
21.5
Full Data
7.8
15.1
19.0
15.2
26.3
39.5
2.9
5.1
11.8
9.3
19.7
36.4
3.1
17.2
Instruction Mining
8.9
17.8
28.4
15.7
29.2
43.1
6.3
11.4
23.8
12.8
24.5
45.2
IFD
10.5
21.2
35.6
18.2
34.5
50.7
8.7
16.8
31.2
16.4
30.8
52.4
Nuggets
22.9
9.5
32.6
PASER
14.4
27.6
48.2
23.1
42.6
62.0
12.9
25.2
44,5
22.3
41.0
63.7
24

---

Published as a conference paper at ICLR 2026
E EVALUATION ON CODE GENERATION TASKS
To further explore the PASER's ' effectiveness on recovering code generation capability, we take two structured pruning schemes (LLM-Pruner, SliceGPT) and perform exhaustive evaluations on two major code generation benchmarks:
HumanEval (Chen et al. 2021]: A widely- code generation benchmark consisting of 164 hand- crafted Python programming problems that test various programming concepts. Each problem contains a function signature with doestring and test cases, requiring models to complete the implementation. The benchmark evaluates functional correctness using the Pass@ @ k metric, which measures the percentage of problems where a correct solution appears in k samples.
MBPP (Austin et al. 2021): A programming benchmark containing 974 Python programming prob- lems focused on basic programming tasks. Each problem includes a natural language description, test cases, and reference solution, making it particularly suitable for evaluating language-to code generation capabilities. MBPP uses the same Pass@k k evaluation metric as HumanEval but generally features simpler problems with broader coverage of basic programming concepts.
In our experiments, models are evaluated in zero-shot on HumanEval and 3-shot on MBPP. The results under @ (k= 1, 10, 100) metries present in Table 6 As shown in the table, code generation capability experiences severe degradation after pruning. The Pass@) I performance on HumanEval drops to merely 3.4% under indicates that code generation, as complex reasoning task, is particularly vulnerable during model compression. Through capability degradation-awar budget allocation and targeted sample selection, PASER demonstrates remarkable effectiveness in recovering this severely impacted capability. Under LLM-Pruner, it achieves 14.4% Pass@1 on HumanEval, not only substantially outperforming other recovery methods but also surpassing the full data training baseline. The improvement becomes even more pronounced at higher sampling rates, i.e., PASER reaches 48.2% Pass @ 100 compared to Random's ' 20.5% and Instruction Mining's 28.4%. This significant performance gap validates our approach of prioritizing recovery resources for severely degraded capabilities and selecting the most relevant instruction samples for recovery training. The superiority of PASER remains consistent across different evaluation settings. On MBPP, which features simpler programming tasks, PASER still maintains clear advantage over baseline methods, achieving 23.1% 1 Pass @ and 62.0% Pass @ 100 under LLM--ru When tested with different pruning scheme (SliceGPT), which causes even more severe initial degradation (3.0% Pass on HumanEval), PASER successfully recovers the performance to 12.9% Pass@ I and 44.5% Pass @ 100.
without recovery training. This dramatic decline
These results comprehensively demonstrate that our capability-aware recovery strategy effectively addresses the disproportionate impact of model compression on complex reasoning abilities, enabling targeted and efficient recovery of critical model capabilities.
Table 7: Recovery performance of different instruction tuning data selection methods on other knowledge extensive tasks under different LLM pruning schemes. The 'bold' represents the best performance under the same pruning scheme. Here, the Alpaca is taken as the original dataset. The LLaMA2-7B is taken as the target here ensure consistency.
Benchmark
Pruning
Training
Kandom
Instruction Mining
Nuggets
LLM-Pruner (25%)
25.35
36.27
40.91
41.42
41,86
42.30
46,37
MMLU
SticeGPT (25%)
28.80
40.82
41.76
42.15
42.51
42.64
46.97
Wanda (2:4)
32,58
44.45
42.36
42.53
42.73
43.55
47.04
SparseGPT (50%)
34.13
42.47
LLM-Pruner (25%)
48.47
$7.50
60.21
60.65
61.22
62.59
66.44
CommonsenseQA
SliceGPT (25%)
49.60
61.32
61.09
60.81
62.28
62,23
66.51
Wanda (2:4)
52.65
65.04
61.71
62.13
62.47
62.91
66.79
SparseGPTT (50%)
53.97
64.32
62.21
63.04
62.48
63.17
67.63
F EVALUATION ON OTHER KNOWLEDGE-EXTENSIV TASKS
To supplement more experiments on other chowledge-extensive evaluation tasks, we conduct experi- ments MMLU (Hendrycks al.) and CommonsenseQA (Talmor et al. 2019), with LLaMA2-7B as the target LLM to ensure comparison consistency. As shown in the Table PASER consistently outperforms existing data selection baselines and even surpasses full-data tuning. This demonstrates
25

---

Published as a conference paper at ICLR 2026
that PASER generalizes well beyond the commonsense reasoning benchmarks in the main text and is effective on other knowledge-intene intensive tasks.
Table 8: Recovery performance of different instruction tuning data selection methods under RTN 4bits and GPTQ 4bits schemes LLaMA2-7B model. The 'bold' represents the best performance under the same quantization scheme. Here, the Alpaca is taken as the original dataset.
Quamizztion
Post-training
Recovery
WikiTeu12
PTB.
BoolQ
PIQA
HellaSwag
WineGrande
ARC-e
ARC-c
OBQA
Average
w/o Quant
w/o training
12.02
71.13
18.40
72.79
67.17
09.36
10.70
10.80
62.91
wio training
18.14
33.25
66.52
74.95
69.24
63.91
65.58
38,07
35.10
59.05
Pull Dara
15.81
27.41
64.57
38.48
29.56
bits
Instruction Mining
16.05
27.85
66.73
75.15
69.43
64.08
63.74
38.18
35.30
59.23
IFD
15.21
25.74
68.16
76.40
70.60
65.18
66.83
38.83
37.40
60.49
Nuggets
14.68
PASER
14.21
23.37
70.43
78.41
72.47
66.92
68.54
39.81
41.50
62.58
who
13.96
27.36
67.82
76.15
70.35
61.95
66.39
38.69
36,90
60.21
GPTQ bits
Random
16.31
28.74
66.81
75.24
69.49
64.14
65.79
38.22
35.70
59.34
Instruction Mining
15.37
26.42
68.31
76.58
70.75
65.33
66.96
38.93
37.90
60.68
Nugaets
13.52
22.93
69.74
77.83
71.93
66.43
68.06
39.56
10.20
61.96
PASER
12.95
21.84
71.20
79.12
73.12
67.53
69.14
40.18
42.90
63.31
Table 9: Recovery performance of different recovery data selection methods under RTN 4bits
and GPTQ 4bits quantization schemes on LLaMA2-13B model. The 'bold' represents the best performance under the same quantization scheme. Here, the Alpaca is taken the original dataset.
Quanization
Recovery
Post-training
WikiTex12)
PTB]
BoolQ
HollaSwag
WineCirande
ARC-c
Average
w/o Quant
w/o training
11.58
20.24
69.02
78.73
76.60
69.69
73.23
44,20
42,00
64.78
WAo training
17.51
12.34
63.15
74.39
72.62
63.34
69.17
41,49
37,00
60.37
16.95
RTN
Random
17.86
33.15
62.00
73.48
71.55
64.94
68.13
40.84
35.20
59.45
4 hits
Instruction Mining
17.24
31.68
62.83
74,27
72.32
65.65
68.87
41.29
36.10
60.19
42.46
Nuggets
15.08
27.15
65.45
76.76
76.72
67.84
71.17
42:70
40.20
62.98
PASER
12.34
23.08
67.33
78.50
76.41
69.37
72.78
43.66
41.70
64.25
training
14.74
42.28
41.76
GPTQ
Random
14.58
26.52
64.82
76.17
74.15
67.52
70.61
42,36
39.30
62.10
bits
Instruction Mining
13.67
24.59
66.37
77.58
75.51
68.55
71.91
44.15
41.30
63.63
Nuggets
12.76
22.92
67.21
78.34
76.25
69.24
72.63
43.39
42.70
64.28
PASER
11.25
20.93
68.11
79.16
77.05
69.97
73.39
14.01
44.20
65.13
Table 10: Recovery performance of different recovery data selection methods LLaMA2-7B under AWQ 4bits scheme. The 'bold' represents the best performance among different selection methods. Here, the Alpaca is taken the original dataset. The "Reason" indicates the averaged performance on commonsense reasoning datasets.
Quantizatinn
Benchmark
w/n Training
Full Data
Random
Instruction Mining
IF1)
Nuggets
PASER
WikiText2]
13.52
12.10
13.27
12.02
12.24
11.58
11.05
AWQ 4bits
PTB.
23.29
22.76
22.23
21.51
20.12
19.74
18.48
Reasont
61.47
62.32
62.41
63.19
63.46
63.13
65.54
G EXTENDED EXPERIMENTS ON POST-QUANTIZATION RECOVERY TRAINInG
Though the method descriptions and the experiments in the main body are mainly around the LLM pruning scenario, our PASER framework can actually be extended seamlessly to other LLM compression scenario. To further demonstrate its applicability, we conduct the experiments on post-quantization recovery training and compare our PASER with corresponding instruction tuning data selection baselines. In detail, we choose two most representative methods:
(RTN) (Frantar& & Alistarh, 2022 Yao et al., 2022), GPTQ (Frantar et al. 2023) to perform the LLM quantization. It should be clarified that RTN method, which rounds all weights the nearest quantized value on exactly the same asymmetric per-row grid, is actually the fundamental technique in most works about L.L. quantization (Frantar & Alistarh
2022 Yao al. 2022 Park al.).
26

---

Published as a conference paper at ICLR 2026
Its runtimes scales well to the models with many billion parameters due to the direct rounding. According to the results provided in Table and 9 we can observe that the PASER can still effectively enhance the recovery performance and outperform the data selection baselines averaged reasoning performance and zero-shot perplexity for both L.L.MA-7BB and LLMMA2-13B models. Meanwhile, recovery data selection baselines can indeed achieve the stronger performance than full data and random baselines, which validates the necessity of conducting recovery data selection even in the LLM quantization scenario. Furthermore, comparing these results with Table | and - noticed that the improvement space of PASER in Table 8 and has been reduced to some extent. This because the post-compression performance of such quantization schemes has been competitive enough, which can reflected from the w/o training row. In addition, we also conduct experiments on one of the most robust LLM quantization scheme, AWQ (Lin et al. 2024), and the results are provided in the Table TO Considering the superior performance of AWQ itself on reducing the performance drop brought by the precision loss, our improvement over baselines is still obvious, further demonstrating the generalizability of PASER quantization schemes.
it can be
Table 11: Recovery performance of different instruction tuning data selection methods on LLaMA2- 7B under different LLM distillation schemes. The 'bold' represents the best performance on the same distillation scheme. Here, the Alpaca taken as the original dataset. The "Reason" indicates the averaged performance 7 commonsense reasoning datasets.
Distillation
Benchmark
w/o Training
Full Data
Random
Instruction Mining
IFD
Nuggets
PASER
WikiText2,
23.58
27.31
22.50
21.43
20.32
20.09
18.65
DistiLLM
PTB|
46.83
43.71
44.08
41.61
38.17
36.82
32.86
Reason
55.20
57.12
56.29
57.71
58.55
58.42
60.34
WikiText2,
22.71
25.04
21.37
20.74
19.77
18.92
17.27
MiniPLM
PTB.
43.09
42.02
42.71
41.28
39.03
37.22
33,08
Reason"
56.43
58.21
56.07
$7.50
59.39
59.21
61.42
Text2
24.45
25.82
21.14
20.48
19.20
13.36
17.07
SKD
PTB
44.73
41.49
42.02
39.67
36.52
34.31
30,75
Reason
56.67
59.61
57.38
58.79
59.82
59.50
61.94
H EXTENDED EXPERIMENTS ON POST-DISTILLATION RECOVERY TRAINING
In addition to the post-pruning and post-quantization experiments, we also supplement post-distillation experiments achieve full LLM post-compression recovery training picture. In detail, adopt DistiLLM (Ko et al. 2024). MiniPLM (Gu et al.). SKD (Xu et al.). respectively, to distill LLaMA2- 7B model into I.5B scale small language model. The corresponding post-distillation recovery performance has been provided in the Table IT] From the table, we can observe that our PASER can effectively help recover L.LM capabilities compressed with different recent distillation methods. Therefore, we can conclude that our proposed PASER can generalize well across different kinds of LLM compression paradigms, not limited to the pruning discussed in the main text.
EXPERIMENTS ON RECOVERY TRAINING WITH KNOWLEDGE DISTILLATION
Inspired by Muralidharan et al. 2024 we explore the knowledge distillation as the recovery post-training paradigm instead of the standard supervised learning with the groundtruth label. Here, we set the original model M. as the teacher and the pruned model My as the student. The mean KL divergence (Kullback & Leibler 1951) between the output probability distribution of M, and that of M, is taken as the objective function. Comparing the corresponding results under different pruning schemes in Table [1] with that in Table we can first observe that knowledge distillation can effectively improve the recovery performance on both reasoning and language modeling tasks in most cases. particular, the reasoning performance of PASER is improved by 0.348 points on average among such four pruning schemes. Interestingly. the knowledge distillation recovery performance of Full Data under LLM-Pruner is much better than that with standard label-supervised learning. This demonstrates that knowledge distillation is also a promising approach to avoid the misleading information from the irrelevant or conflicting samples existing in the original dataset. Because its learning process directly imitates the unpruned model behavior instead of the provided labels, thus better preserving the thinking and decision- bocision-making consistency with the original model. As summary, distilling the knowledge of unpruned model into the pruned model can be regarded
27

---

Published as a conference paper at ICLR 2026
Table 12: Knowledge distillation recovery performance of different instruction tuning data selection methods under various pruning schemes on LLaMA2-7B model. The 'bold' represents the best performance under the same pruning scheme. Here, the Alpaca taken as the original dataset.
Pruning
Post-training
Recovery
WikiTex:2
PTB.
BoolQ
PIQA
HellaSwag
WinaGrande
ARC-e
ARC-c
OBQA
Avenage
w/o pouning
w/o Traiing
12.62
22.14
71.15
78.10
12.79
67.17
69.36
10.70
40.80
62.91
Trainin
60.22
37.37
Full Data
24.72
43.91
63.30
76.01
67.18
62.27
64.23
36.86
39.20
58.44
m550-25%%
LLM-Pruuee
Instruction Mining
Random
23.82
41.20
68.03
74.89
66.27
64.51
64.65
18
72.58
38.30
58.46
IFD
19.17
32.30
61.13
77.55
67.89
61.56
61.09
38. 38.19
40.40
59.12
Nuggels
18.64
52.19
64.46
76. 76.66
67.26
64.88
66.50
36.
39.20
59.35
PASER
15.91
25.39
67.89
77.81
69.62
67.63
65.46
39.57
40.20
61.64
wio traniing
66:37
63.38
Full Data
35.48
66.25
69.35
70:34
5% 58.50
66.76
62.95
37. 14
38,70
57.68
SlicGPPT
Random
38.63
65.67
67.19
59
56.21
61.94
60.63
35.61
37.80
55.85
ralio=25%
Instruction Mining
IFD
33.50
61.33
69.51
70.82
58.70
67.49
64.09
37.22
38.50
58.05
Nurgets
21.39
32.83
70.17
71 .49
59.11
67.94
72.51
37.
38.70
59.64
71.95
W/G traiming
42.10
76.85
69.30
71.99
53.06
61.75
00.94
28.07
34.00
$4.39
Full Data
25.92
47.85
71.09
75.14
64.10
65.62
65.64
34.38
37.50
$9.07
Wanda
Random
34,98
63.47
70.18
73.62
59.15
63.83
63.70
32.13
36.50
57.02
spansity-2:4 Instruction Mining
71.03
73.97
64.56
IFD
24.08
41.44
71.78
75.89
64.83
65.72
68.89
35 97
38.00
60.15
Naggets PASER
23.14
40.10
72.26
76.50
65.33
66.03
66.52
37.27
38.60
60.36
66.64
WiG briniigg
19:26
36.41
71:22
75.60
62.85
56.06
69.11
36.35
37,50
59.93
Full Data
28.17
52.82
68.52
75.77
57.84
69.26
60.43
37.72
37.00
58.08
SparseGPT
Random
69.74
74.91
spansity500%
Instruction Mining
21.56
39.61
71.12
74:85
62.53
66.06
68.
36.85
37,80
59.61
IFD
17.76
31.25
71.70
75.76
63.43
66.06
69.14
36.59
37,60
60.04
Nuggets
14.83
25.38
72. 18
75.95
63.91
66.29
69.75
36.
37.70
60.38
PASER
13.00
22.24
75.07
78.66
66.90
69.31
72.85
38 38.89
39.60
63.04
Table 13: Recovery performance comparison between different combinations of knowledge distilla- tion (KD) and supervised fine-tuning (SF) under various pruning schemes. The 'bold' represents the best performance under the same pruning scheme.
Recovery Training
WikiTes121
PTB.
BoolQ
PIQA
HellaSwag
WinoGrande
ARC
ARC
OBQA
Average
Prunnrr (rahio=25%)
KD SF
15.91
25.39
67.89
77.81
67.63
39.87
16.40
26.35
67.25
77.29
68.98
66.97
67.84
39.54
39.80
61.10
First KD. then SF
16.15
25.87
67.57
77.55
69.31
67.30
68.15
39.71
40.00
61.37
First SF, then KD
16.28
26.02
67.41
77.43
69.15
67.11
67.96
39.63
39.90
61.23
SliceGPT (ratio=25%)
KD SF
11.87
20.91
73.43
80.32
74.46
69.76
71.93
42.26
41.70
64,84
12.24
21.53
72.75
79 84
73.92
69.18
71.37
41.82
41.30
64.31
First KD, then First SF, then KD
12.06
21.24
73.12
42.03
12.15
21.38
72.94
79.95
74.05
69.32
71.51
41.95
41.40
64.45
Wanda
(sparsity=2:4)
KD SF
13.84
23.54
71.25
78.15
72.06
66.64
68.66
39.38
40.50
62.38
14.13
27.22
70.77
77.87
71.78
66.26
68.30
39.04
40.10
62.02
First KD, then SF
13.97
25.31
71.02
78.03
71.94
66.47
68.49
39.23
40.30
62.21
First SF, then KD
14.05
26.28
70.89
77.95
71.85
66.35
68.41
39.15
40.20
62.11
SparseGiPT
(5)
KD
13.00
22.24
75.07
78.66
66.90
69.31
72.85
38.39
39.60
63,04
13.33
23.77
74.79
78.38
66.62
69.03
72.57
38.70
39.40
62.78
First KD. then SF
13.15
22.96
74.94
78.53
66.78
69.18
72.72
38.81
39.50
62.92
First SF, then KD
13.24
23.35
74.85
78.45
66.70
69.11
72.64
38.75
39.45
62.85
as an effective way to enhance the recovery performance, though bring more memory overhead. Furthermore, stronger layer-wise distillation also be taken into consideration (Tiao et al. 2020
Exploration on Combined Training Strategies Given the complementary potential of knowledge distillation (KD) and supervised fine-tuning (SF), we further explore whether combining these two approaches could lead to enhanced recovery performance. Specifically, we investigate two cascading strategies: (1) first applying KD followed by SF, and (2) first conducting SF followed by KD. Table 13 presents the results under different pruning schemes.
Interestingly, the results show that neither cascading strategy consistently outperforms individual KD or SF approaches. This suggests that these two training paradigms might actually serve similar functions in recovering model capabilities, making their combination redundant rather than comple-
28

---

Published as a conference paper at ICLR 2026
Table 14:
Recovery performance different instruction tuning data selection methods on several more recent and advanced LLMs under LLM-Pruner scheme. The 'bold' represents the best performance on the same target LLM. Here, the Alpaca taken as the original dataset. The "Reason" indicates the averaged performance 7 common sense reasoning datasets.
Model
Benchmark
wlo pruning
w/o Traiing
Full Data
Ramdom
Instruction Mining
IFD
Nuggels
PASER
LLaMA3. 1-8B
Wikllexr2,
6.63
14.85
9.74
12.16
13.84
11.32
10.81
7.56
ratio=25 %
PTBL
16.72
29.21
18.38
23.47
26.52
21.75
20.46
17.83
Reasent
70.81
63.92
67.95
65.73
65,34
66.82
67.58
70.05
Owen2.5-7B
Wikilext2,
6.94
15.82
11.68
13.94
15.37
10,95
10.23
8.16
m5tio25%
PTB]
20.58
32.95
24.43
29.16
30.54
22.89
21.37
19.85
Reason"
70.49
62.71
67.08
64.29
63.75
66,93
67.42
69.14
QwenG-8B
WikiTex22
6.68
14.29
9.18
11.83
13.26
10.17
9.45
7.24
miti==5%
PTBL
18.45
30,56
20.74
25.41
28.13
22.68
21.05
18.97
Reasont
71.73
66.37
Mixtral-&x7B
WikTlex22,
7.52
15.87
12.83
14.32
14.27
13.65
12.98
9.62
milio=25%
PTBL
17.46
27.13
24.07
Reason
72.04
61.56
64.83
63.75
65.21
66.48
67.24
70.87
mentary. Knowledge distillation shows slightly better performance across all pruning schemes, which could be attributed to its ability to capture the nuanced knowledge encoded in the teacher model's full output distribution. However, the marginal gains from combining approaches do not justify the additional computational overhead required for cascaded training.
J EXPERIMENTS ON MORE RECENT LLMS
In addition the previous experiments conducted on LLaMA2/3 and Baichuan2 families, also supplement some results on more recent and advanced LLMs: LLaMA3.1 1-B. Qwen2.5-7B Yang al. 2024) Qwen3-8B Besides. the Mixture of Experts(MOE) model Mixtral-8x7B (Jiang et al. 2024 is also employed to conduct the experiments. The evaluation results have been provided Table T4 From Table 14] we can observe that PASER consistently outperforms all baseline methods across different recent and advanced LLMs, including the MOE architecture (Mixtral-8x7B), For LLaMA3.1-8D PASER achieves perplexity scores (7.56 on WikiText2, 17.83 on PTB) that remarkably close to those of the unpruned model (6.63 and 16.72, respectively), with only a 0.76 point gap reasoning performance (70.05 vs. 70.81). Notably, PASER demonstrates robust performance across different model architectures. For the MoE-based Mixtral-8x7B, PASER significantly narrows the performance gap from 10.48 points (w/o Training: 61.56 vs. w/o pruning: 72.04) just 1.17 points (PASER: 70.87 w/o pruning: 72.04), representing 88.8% recovery of lost capability. This demonstrates that our capability-aware selection approach effectively addresses the specialized recovery needs of sparse models with expert-based architectures. Among the most recent models, Qwen3-8B shows the strongest recovery with PASER, achieving 98.4% of the unpruned model's reasoning performance (70.58 vs. 71.73). Meanwhile, Qwen2.5-7B demonstrates slightly lower recovery effectiveness, potentially due to differences in its architecture and pretraining approach. These results further validate PASER's 's applicability across diverse and state-of-the- art LLM architectures, confirming that our capability degradation-aware instruction selection and negative tuning effects mitigation strategies generalize well beyond the models in the main paper.
K DISCUSSION ON COMPARISON WITH GENERAL DATA SELECTION METHODS
To further demonstrate our PASER is an unique contribution and solution for pruned-LL.M recovery. discuss the comparison with previous general data selection methods (e.g., LESS (Xia al. 2024b), DELIFT (Agarwal et al. 2025). SMART (Renduchintala et al.) (2024). Datamodels (Iyyss et al. 2022)) from four perspectives: Why general data selection methods do suffice for pruned-LLM
recovery? (Appendix K. 2) How PASER differs from previous general data selection methods?
Owen/
29

---

Published as a conference paper at ICLR 2026
(Appendix K 2
3) What is specific comparison between PASER and gradient-based sampling?
(Appendix
K.3
4) Can PASER benefit from gradient-derived dense information? (Appendix K.44)
K.1
Q1: WHY GENERAL DATA SELECTION METHODS DO NOT SUFFICE FOR PRUNED- LLM RECOVERY?
While existing data selection methods such LESS (Xia et al. 2024b) DELIFT (Agarwal et al.
2025), SMART (Renduchintala et al. 2024), and Datamodels (Ilyas et al. 2022) have demonstrated effectiveness for various training objectives (e.g., improving generalization, robustness, efficiency), they are not specifically tailored to the unique recovery challenges posed by pruning. Notably:
Uneven Capability Degradation: Pruning introduces non-uni non-uniform degradation across different model capabilities (e.g., commonsense, coding, math), as shown in Figure 4 of Appendix A. General selection methods typically assume uniform importance across examples or rely on training dynamics in full-capacity models, failing to prioritize capability-specifie weaknesses that are critical in post-pruning recovery (see Section | and Section 3.3). For example, as discussed the Line 48-50, existing data selection methods generally favor samples with clear structure and natural, human-like expressions. However, these samples may not exactly target specific capabilitiies (e.g., math, code) severely compromised during pruning. In fact, in the post-pruning recovery scenario, even if the expression of samples regarding severely compromised capabilities is not so clear or structured, we should still prioritize them.
As shown empirically in Figure 3 of Appendix A recovery performance varies drastically depending on which subset is selected, issue rarely addressed by general-purpose selection algorithms. This is a unique challenge in pruned model recovery, as pruning disrupts the model's internal structure and weakens its representational capacity, making the choice of recovery data more critical and sensitive than in standard fine-tuning scenarios.
Performance Sensitivity to Instruction Content:
Many existing methods select samples that maximize representativeness or informativeness during pre-training or inadvertently include instructions that are irrelevant or conflicting with each other, thus resulting in harmful impact for recovering pruned capabilities.
Incompatibility with Recovery Training Goals:
which may
Thus, applying these methods directly without modification often leads to inefficient use of recovery budgets and fails to address the nuanced degradation patterns introduced by pruning.
K.2
Q2: HOW PASER DIFFERS FROM PREVIOUS GENERAL DATA SELECTION METHOOS??
PASER is designed from first principles for post-pruning recovery and introduces three key innova- tions absent in prior methods:
Capability-Aware Clustering and Budget Allocation: Unlike data selection methods that focus on sample diversity or informativeness globally, PASER performs semantic-structural clustering (Section (32 uncover latent instruction groups aligned with distinct capabilities. We then assess degradation in each cluster using Jensen-Shannon Divergence between pruned and original model outputs (Section 3.3), and allocate recovery budget proportionally. This enables targeted and balanced recovery, which existing methods do not consider. In contrast, existing methods typically treat all capabilities uniformly and are agnostic which skills the model has lost post-pruning. As result, they may waste recovery budget on less-affected areas while failing to restore critical capabilities.
Efficiency-Driven Sample Selection: PASER integrates computational efficiency by computing a sample wise efficiency score based on degradation severity and expected training cost. This is crucial in recovery settings where data budgets and compute are highly constrained. In contrast, saaamodel-basedd selection methods (e.g.. Datamodels (yyas et al., 2022) ) require training large probes or repeated forward/backwand passes, making them computationally expensive and imprac- tical for efficient recovery. LESS is also time-consuming, because involves gradient computation and LoRA training during the selection process.
Negative Tuning Mitigation via Concept Consistency Graph: General-purpose selection ap- proaches rarely guard against harmful instructions. PASER introduces Concept Consistency
30

---

Published as a conference paper at ICLR 2026
Graph (Section 3.4) to detect and filter irrelevant conflicting samples that might lead to negative tuning effects. This is a particularly pressing issue when the model's 's representational capacity has been impaired by pruning, which results in models more sensitive to conflicting information. However, existing methods rarely address this risk, most of which assume all data in high-quality pools is beneficial. However, our results (e.g., Table T show that even using the full dataset can degrade performance due to negative tuning effects, especially for pruned models with weakened internal representations
Besides, we would like to highlight that across all pruning schemes and models (Tables 1 PASER consistently outperforms random, full data, and several instruction-tuning selection baselines (e.g., Nuggets, IFD), despite using only 4%-20% of the original data. Additionally, our ablation study (Table 4 confirms the importance of each PASER module in achieving superior recovery.
In summary, PASER motivated by and explicitly designed for the unique characteristies and challenges of pruned LLM recovery. Existing general data selection methods neither target uneven capability loss nor mitigate the risk of irrelevant or conflicting data. PASER fills this gap by efficiency-aware and consistency-prn preeseevig data selection, thus introducing capability-driven leading to substantial gains in both recovery quality and training cost.
K.3 Q3: WHAT IS SPECIFIC DIFFERENCE BETWEEN PASER AND GRADIENT-BASED SAMPLING?
We would like to clarify the fundamental differences between PASER and gradient-based data selection strategies such as LESS (Xia et al. 2024b), and supplement this with controlled empirical comparison.
Conceptual Differences Gradient- methods (e.g., LESS) typically rely on computing influence scores or gradient norms to identify impactful training samples. These scores are often computed with respect to specific layers or objectives (e.g., classification loss or instruction adherence) and require gradient backpropagation, which incurs high computational cost-especially when evaluating all samples across large instruction tuning sets. In contrast, PASER is specifically designed for capability recovery after structure/semmisssttuuurr/ttttts struucturee/uussuuuuuurd pruning, where the pruned model exhibits uneven degradation across semantic capabilities. Rather than using gradient signals, PASER leverages: 1) Semantic clustering to disentangle instruction groups aligned with distinct capabilities; 2) Capability Degradation Scores (CDS) based on lensen-Shannon divergence (JSD) between the pruned and original models' output distributions; 3) Efficiency-awarr scoring to optimize recovery benefit per unit of compute cost; 4) Negative tuning effect mitigation using concept consistency graphs. This design enables PASER approximate dense information signals (through token- level divergence) while being more effective, efficient, and robust than backpropagation--eeeedde strategies.
Empirical Comparison To further clarify the performance and efficiency differences, we compare PASER with a -tylee gradient- based sampling baseline, which selects the top 20% samples from Alpaca with the highest gradient norm of loss w.r.t. embedding parameters. As shown in Table 15
Table 15: Recovery performance of different data selection methods on LLaMA2-7B under different LLM distillation schemes. Here, the Alpaca is taken as the original dataset. The "Reason" indicates the averaged performance on 7 commonsense reasoning datasets.
Recovery
Data Selection Time (mins)
Recovery Training
Pruning
Data Selection
Reasoning f
Wiki'Text2 4
PTB
Time (mins) J
PASER
61.10:1.11
16.4010228
26.35:0::00
5.25
28.97
LLMM-Prume (25%)
LESS-style
458.48:10.77
20.2112.16
34.20:1..37
163.10
41.35
PASER
64.311:0.08
12.2411032
21.53::.41
5.24
29.14
SliceGPT (25%)
LESS-style
59.09-1.35
25.83-44.03
38.42445.28
148.51
42,04
PASER
62.02+0.14
14.13+0.21
27.22+0.29
5.35
30.37
Wanda (2:4)
LESS-style
58.64-1.511
21.8413.89
39.25:16.02
152.38
42.43
PASER
62.78:0.110
03.33110.5
23.77::033
5.22
28.63
SpurseGPT (50%)
LESS
59.36+0.43
3.66
37.31+-5.15
170.49
41.16
PASER not only achieves better recovery performance across both language modeling and reasoning tasks, but also incurs significantly less data selection time (up to 30x less) due to its gradient- free design. The recovery training time is also reduced (L- 14.2%), PASER's 's efficiency-drivee sampling
31

---

Published as a conference paper at ICLR 2026
strategy favors shorter and semantically cohesive instructions. Furthermore, PASER consistently exhibits lower performance variance across different pruning settings (c.g., reasoning accuracy std of 0.08-0.14 0.35-0.51 for LESS -stye), indicating greater robustness under heterogeneous post-pruning degradation. summary, PASER offers a more effective, efficient, robust, and pruning- aware alternative to gradient- sampling strategies, especially in scenarios where gradient signals become noisy or unreliable due to structural capacity loss.
K.4
Q4: CAN PASER BENEFIT FROM GRADIENT-DERIVER DENSE INFORMATION?
While PASER does not explicitly rely on gradient information, we acknowledge that gradient- derived signals capture dense local sensitivity and can, in principle, provide useful guidance for data selection. In our current design, we approximate similar effects using token-level output distributional divergence (JSD) reflect fine-grained capability deterioration without requiring backward computation. This makes PASER highly efficient and model- That said, we agree with the reviewer that incorporating lightweight gradient signals could further enhance selection granularity. In future work, we plan to explore hybrid approaches by integrating gradient-bard baased information into PASER in the following ways:
Use average gradient norms within each capability cluster to refine degradation scores alongside our JSD-based measure.
Gradient-Guided Cluster Weighting:
Gradient-Augmenter Efficiency Scoring: Combine gradient- based sample importance with our current -ased efficiency metric, possibly as a weighted fusion.
LoRA-style Gradient Probing: Introduce low-rank adapters to cheaply approximate gradient information in parfmeter-efficient manner during selection, especially useful for large LLMss.
We believe this line of integration can combine the dense expressiveness of gradients with PASER's efficient, capability- aware framework, further improving recovery quality without incurring excessive compute overhead.
Table 16: The detailed ablation study for our proposed three components under various pruning schemes LLaMA2-7B model. The 'bold' represents the best performance under the same pruning scheme. Here, the Alpaca taken as the original dataset.
Pruning
Post Post-training
Recovery
WikiTeat2]
PTB
BoolQ
PIQA
HellaSwag WinoGrande
ARC-
ARC-c
OBQA
Average
LLM-Puuner
RTRIC
32.54
19.20
malio255%
PASER CDAIS
17.56
30:15
66.27
77.03
68.15
65.73
66.58
38.54
39.50
PASER wlo NTEM
19.82
35.60
64.83
77.52
67.34
64.48
63.59
36.78
40.20
59.25
PASER
16.40
26.35
67.25
77.29
68.98
66.97
67.84
39,54
39.80
61.10
SlicePT
PASER RICC
1483
25.42
21.15
78.91
72.25
67.84
69.95
40.22
40.30
63
ralio=25%
PASER wio CDAIS
14.16
21.92
70.89
78.56
71.84
67.45
69.58
40.417
40.00
62.68
PASER WAo NTEM
15.37
27.81
69:97
77.33
70.68
65.92
68.03
39,39
42.10
61.92
Wanda
PASER win S'RIC
15.84
30.25
69:25
77.42
70.31
63.82
63 84
$8.67
39.00
61.19
spuriiy=2:44
PASER WAo CDAIS
15.46
29.48
69.14
77.35
70.27
65.74
67.79
38.75
39.60
61.23
31.52
PASER
14.13
27.22
70.77
77.87
71.78
66.26
68.30
39.04
10.10
62.02
SparseGPT
PASER WAY S S'RIC
14899
26:31
73:25
77.45
71L15
68.47
69. 28
39.82
39.80
62.60
50%%
PASER CIAIS
14,62
25.84
69.05
40.00
PASER NTEM
15.91
28. 19
71.53
78.62
65.48
67.21
69.79
39.18
40.50
61.76
PASER
13.33
23.77
74.79
78.38
66.62
69.03
72.57
38.70
39.40
62.78
L DETAILED ABLATION STUDY RESULTS
In this section, we present comprehensive ablation results of the three key components in PASER: semantic-stmee semantic-structurall recovery instruction clustering (SPRIC), capability degradation-awar instruction selection (CDAIS), and negative tuning effects mitigation (NTEM). Table 16 shows the detailed performance across different evaluation metrics.
The detailed results reveal the distinct contributions of each component under different pruning schemes. For structured pruning like -runeer, removing S'IIC leads to significant degradation in both language modeling (perplexity increases from 16.40 18.73 on WikiTexx2) and reasoning tasks (average score drops by 1.43 points), highlighting its importance in addressing uneven capability degradation. The impact of CDAIS is partieularly evident under SliceGPT, where its removal
32

---

Published as a conference paper at ICLR 2026
causes 1.63-point drop in average reasoning performance while maintaining relatively stable language modeling metrics, demonstrating its effectiveness in balancing recovery priorities. Under structrue pruning (Wanda), all three components show more balanced contributions, with performance drops ranging from 0.68 to 0.83 points when each is removed. This suggests that semi- structured pruning requires more holistic recovery approach. For unstructured pruning (Sparse(iP)) where capability degradation tends to be more uniform, NTEM plays a particularly crucial role- its removal leads the largest drop in language modeling performance (perplexity increases from 13.33 to 15.91 on WikiText22 and affects complex reasoning tasks like WinoGrande and ARC-e significantly. Notably, the full PASER framework consistently achieves the best performance across almost all metrics under various pruning schemes, with only occasional exceptions in individual tasks (e.g., OBQA LLM- M-Pruerr and PIQA SparseGPT). This comprehensive superiority validates our design choice of combining these three components for effective pruned model recovery.
M HYPERPARAMETER ROBUSTNESS ANALYSIS
To demonstrate the robustness of PASER to various hyperparameter settings, we conducted a compre- hensive sensitivity analysis. This analysis helps establish that our method's strong performance is not dependent on specific hyperparameter configurations but remains consistent across a reasonable range of settings.
M.1 SENSITIVITY TO EMBEDDING DIMENSION
The dimensionality of the manifold representation after applying the diffusion kernel is controlled by parameter d. We evaluate PASER's performance with d ranging from 4 to 64 on LLaMA2-7B under LLM-Pruner (ratio=25%). ). As shown in left subfigure of Figure clustering results remains consistent (Rand Index >0.55) across this range, and average reasoning performance variations is controlled within =0.2 points. This indicates that PASER relatively stable to the specific choice of embedding dimension, though we select d = 16 based on the eigenvalue decay pattern, where eigenvalues beyond this dimension contribute negligibly to the representation.
(a) Sensitivity to Embedding Dimension
(b) Robustness to Temperature Parameter
Perforancc decreases with higher lemperature
Averuga Reisoning Performannc
Average Ressoning Performance
Clustering Renn Inndex
0.94
61.0
Vanation within s02 pamt
0.52
61.0
0.90
60.8
80.8
0.88
Detout: d:16
Performance Score
Defout: 1-1.0
1.00
1.25
1.50
Embedding
Temperature (1)
Figure 5: (a) Sensitivity to embedding dimension d after manifold learning, here the clustering results
under 16 taken the reference; (b) Robustness temperature parameter 7 in Equation 5
M.2 ROBUSTNESS TO TEMPERATURE PARAMETER
For the Lonsen-Shanoor divergence (JSD) calculation in the capability degradation score, we investi- gate the effect of temperature r in the softmax function (Equation In right subfigure of Figure 5 we test values in the range 7 E (0.5, 0.75, 1.0, 1.25, 1.53 and observe that while lower temperatures (T 0.75) lead to slightly sharper capability distinctions, the overall performance differences 1.0 provides a good balance between sensitivity to capability differences and stability across pruning schemes.
minimal (<0.3 points). Our default setting
33

---

Published as a conference paper at ICLR 2026
Overall, our hyperparameter sensitivity analysis demonstrates that PASER's performance is robust across a wide range of hyperparameter settings. This stability enhances the practical applicability of method, as it does not require extensive hyperparameter tuning to achieve strong performance in real-world scenarios.
N EMPIRICAL EXPLORATION FOR COMPONENT SELECTION
To demonstrate the selection optimality of each component in our PASER framework, we conduct the empirical exploration on alternative initial instruction embedding models, dimensionality reduction methods, clustering methods, divergence measurement methods, and budget allocation strategy, to investigate if they can achieve better final recovery performance.
Table 17: Recovery performance of multiple PASER versions integrated with initial instruction em- bedding models under various pruning schemes LLaMA2-7B model. The PASER(SenteneeBER) the version we employ the main body. The 'bold' represents the best performance under the same pruning scheme. Here, the Alpaca is taken the original dataset.
Pruning
Post-training
Recovery
WikITesi
RoolQ
PIQA
HellaSwag
WinoCirande
ARC-c
ARC-e
OBQA
Average
wio pruning
w/o Training
12.62
22.14
71.13
78.40
72.79
67.17
69.36
40.70
40.80
62.91
w/o Traimin
20.34
38.81
61.87
76.61
63.86
60:22
63.13
37.37
39.10
S7.7%
LLM-Pruner
PASER/Qwen2.5-70
16.38
26.33
67.27
77.32
69.01
67.00
67.86
39. 99.56
39.80
61.12
25
PASER(11
26.31
PASER(Senceno:BBRR)
77.29
wio Training
44.53
80.07
65.34
06.37
54.16
63:38
S8.46
14.56
36.90
$4,27
SliceGPT
PASER/QWee2.5:11 5B))
12:22
21.51
72.78
79.86
73.94
69.20
71.39
41.86
41.30
64.33
ratio=25g
PASER/LLaMA3-383 3-33-B))
12.21
21.50
72.80
79.86
73.94
69.30
71.40
41.85
41.30
64.34
79.84
w/o Training
42.10
60.9
25.07
Wanda
PASER(Qwen2.55 5.7B)
14.11
27.20
70.79
77.87
71.78
66.26
68.32
39.03
40.10
62.02
sparsity-2:4 2:4
PASER/1 LaMA3
14,10
27.22
70.80
77.90
71.30
66.29
68.33
39.07
40.10
62.04
Training
37.80
SparseGPT
PASER(Qwen2.
13.62
24.36
74.65
78.
66.46
68.90
72.50
38.72
39. 39.30
62.67
500%
PASER/LLaMA LLMMA3-SB
13.50
23.96
74.72
78.31
66.55
68.96
72.50
38.63
39.30
62.71
PASER BeRI)neBERT)
13.33
23.77
74.79
78.35
66.62
69.03
72.57
38.70
39.40
62.78
N.1 EXPLORATION ON OTHER POSSIBLE INSTRUCTION EMBEDDING MODELS
In the Comantic-Sttuctar Recovery Instruction Clustering module our PASER framework, we adopt the SenTenceBERT (Reimers & Gurevych 2019) to obtain the initial instruction embedding, consider- ing that SentenceBERT has demonstrated strong transfer capability across various text semantic tasks. To further validate that SeRtenceBERT is competitive enough here with the empirical evidence, replace with two other advanced LLMs: Qwen2.5-7B (Yang et al. 2024) and LLaMA3-8B (Dubey
al. 2024) here and conduct the experiments. As shown Table 17 using much larger pretrained language models like Qwen2.5-7B and LLaMA3-8B as embedding generators provides marginal performance improvements (less than 0.05 points on average) compared to SentenceBERT. While LLaMA3-8B achieves the highest performance in three configurations, these gains are negligible given the substantially increased computational costs associated with using such large models for initial embeddings. Besides, under the SparseGPT, sparsity=50% setting, SentenceBERRT performs obviously better than LLMMA3-8B and Qwen2.5-7B. This finding validates our design choice of SentenceBERT as the default embedding model for PASER, as it strikes optimal balance between semantic representation quality and computational efficiency. SentenceBERT requires significantly fewer resources while still enabling effective eapability-speece recovery approach. The results demonstrate that while more powerful embedding models may theo- retically provide better semantic representations, in practice, the performance difference is minimal for our specific task of recovery data selection.
clustering that supports targeted
N.2 EXPLORATION ON OTHER POSSIBLE DIMENSIONALITY REDUCTION METHODS
To validate the suitability of diffusion kernel as dimensionality reduction approach which focus on uncovering the instruction geometric structure in the semantic space facilitate the following capability clustering process, we replace it with other alternative schemes and compare the perfor- mance. As shown in Table T8 among different dimensionality reduction methods, diffusion
34

---

Published as a conference paper at ICLR 2026
Table 18: Recovery performance of multiple PASER versions integrated with different dimensionality reduction approaches under various pruning schemes on LLaMA2-7B model. The PASER(Diffusion the version we employ the main body. The 'bold' represents the best performance under the same pruning scheme. Here, the Alpaca is taken the original dataset.
Pruning
Post-training
Recovery
WikiTes221
PTB.
BoalQ
PIQA
HellaSwag
WinoGrande
ARC-e
ARC-c
OBQA
Avenge
w/o pruning
w/o Training
12.62
22.14
71.13
78,40
12.79
67.17
69.36
10.70
20.30
62.9]
w/o Training
20.34
38.81
61.87
76.61
65.86
60.22
63.13
37.37
39.40
57.78
LLM- Luu-rrr
(USPR(UMAP)
16.92
27.83
66.31
76.84
67.59
65.85
66.92
38.
39.20
60.31
25%
PPCE)(PCA)
PAS-R(--SNE)
17.05
17:21
28.16
28.42
66.18
66.15
76.68
76.73
67 34
67.46
65.64
65.73
66.74
38. 18.75
38.87
39.00
60.05
60.18
PASER(Diffssion)
16.40
26.35
67.25
77.29
68.98
66.97
67.84
39.54
39.80
61.10
w/o Training
66.87
SliceGPT
PASFR(L/MAB
1483
25,42
71.15
78.91
72.25
67.84
95
40.82
40. 00
63.03
m5%
PAS-R(-SNN)
(PCA)
13.87
23.75
71.73
79.31
72.79
68.48
70.53
-1..28
40.70
63.55
PASER(Difession)
12:24
21.53
72.75
79.84
73.92
69.18
71.37
41.82
41.30
64.31
w/6 Training
42.10
76.85
69.30
71.99
$3.06
62.75
60:94
28.07
34,60
54.39
Wanda
(UMAP)
69.26
38
sparsity-244
(PCA)
PASE-RI-SNE
14.72
13:46
28.21
29.48
70.14
69.14
77.35
77.62
71.08
70.27
66.01
65.74
68.00
67.79
38. 85
38. 38.75
39.60
80
61.65
61,23
PASER (Diffosio))
14.13
27.22
70.77
77.87
71.78
66.26
65.30
39.04
40.10
62.02
w/o Training
75.60
66.06
SparseGPT
(USMA(UMMP))
14.89
26.31
73:25
77.45
65.15
68.47
71:28
37.82
38.80
61.75
5pansty=50%%
PPCA)
14.62
25.81
73.51
77.68
65.52
68.73
71.61
38.
38.90
62.01
PASERO-SNE.
74.13
78.04
PASER/DiF(ssion)
13.33
23.77
74.79
78.38
66.62
69.03
72.57
38.70
39.40
62.78
kernel approach consistently outperforms alternatives across all pruning schemes. The diffusion kernel's 's ability preserve manifold structure while adapting to the intrinsic geometry of instruction data proves particularly advantagcous for capability-awand clustering. UMAP, PCA, and t-SNE each demonstrate reasonable performance, but they fall short by 0.38-1.28 points compared the diffusion kernel approach on average reasoning tasks. The performance gap is especially pronounced under structured pruning schemes like LLM-Pruner (0.79 points advantage) and SliceGPT (0.52 points advantage), where capability degradation tends to be more severe and uneven. This confirms that the diffusion kernel's 's non-linear dimensionality reduction better captures the complex relationships between instructions targeting similar capabilities, enabling more effective recovery prioritization. While t-SNE shows competitive performance some scenarios, its sensitivity hyperparameters and initialization makes it less stable for diverse instruction sets compared to our approach.
Table 19; Recovery performance of multiple PASER versions integrated with different data clustering approaches under various pruning schemes LLaMA2-7B model. The PASER(S RICC is the version we employ in the main body. The 'bold' represents the best performance under the same pruning scheme. Here, the Alpaca is taken as the original dataset.
Pruning
Post-rrining
WikiTex12
PTBL
BoolQ
PIQA
HellaSwag
WinoGrande
ARC-e
ARC=
OBQA
Average
Wio proning
w/S Training
72.79
67.17
69.36
40.70
62.31
WIG Training
20.4
28.81
61.87
76.61
63.30
60122
63:13
27.37
39.30
37.78
PASERINMP TFIDF)
17.2
29.45
65.93
76.88
67.42
65.19
66.37
38.81
39.60
60.03
rat
L..M-Prnner
PASERILDA TFIDF)
PASER/KMeams TRIDF)
17.56
28.13
28.91
66.18
77.02
67.76
65.58
66.92
67.23
39.12
38.95
39.50
39.70
60.53
60.30
PASER(Spectral
27.24
67.56
39.31
60:80
PASER/SpectraL_Ee BBRR)
16.61
26.79
67.06
77.22 26
68.72
66.68
67.71
39:43
39,50
60.95
PASER(S
16.40
26.35
67.25
77.29
68.98
66.97
67.84
39.54
39,80
61.10
MIG
BU.UT
57.16
58.16
PASERINMF (TFDF)
24.36
70.89
70.12
25%
SliceGPT
PLDD TFIDF)
PA TRIDP)
PASER\Spectral MTEB)
14866
13.58
12.91
25.19
22.47
23.42
70.31
72.08
71.46
79.41
79
72.61
73.18
71.64
68.62
68.14
67.25
70.87
70.48
69.58
41.1 41.18
41.43
40.37
41.00
41.10
40.60
63.81
63.41
62.65
PASERISpectra_
12.58
22.01
72.41
63
73.55
68.91
71.12
41.63
41.20
64.06
PASERUS RIC)
21.53
72.75
73.92
69.18
71.37
64.31
WIS Training
4210
76.35
69.30
71.59
53.06
62.75
60.01
2X.07
34.001
54.39
PASER/NMP TFIDF)
16.18
30.94
70.09
76.68
69.98
64.82
66.92
38.
39.60
60.89
Wanda
PLA TRIDF)
PASER/KMenns TRIDP)
18.74
29.76
34.98
70.92
69.88
77
76. 31
70.51
69.42
64.37
65.28
67.98
66.48
38.47
37.82
40.30
39 20.40
61.41
60.52
PASER(Spectral
28.49
71.12
67.82
61.61
PASERiSpactra__BER
14.47
27.86
70.66
77.65
71.45
66.01
68.06
38.89
40.00
61.82
PASER(S' RIC)
14.13
27.22
70.77
77.87
71.78
66.26
68.30
39.04
40.10
62,02
wis
19.26
36.41
62.85
66.06
69.11
36.86
37.30
59.93
PASERINMP _FFEE
26.13
64.37
70.39
spansit==00
SparseGPT
PASER/Spectral MTE) MTEB)
PLD TFIDF)
PASERKKMemt TFIDF)
14.73
15.11
14.03
27.09
25.91
24.84
74.16
73.61
73.12
77
78.01
77.31
66.02
65.46
61.93
68.54
68.09
67.63
71.48
70.92
71.98
37.86
38. 19
38.44
39.20
39.00
38.50
61.93
61.51
62.34
PASER(SpectaL BERT)
13.68
21.31
74.18
78.21
66.32
68.79
72.28
38.75
39.5
62.59
PASERIS RIC)
23.77
66.62
72.57
38.70
62.78
35

---

Published as a conference paper at ICLR 2026
N.3 EXPLORATION ON OTHER POSSIBLE CLUSTERING METHODS
To discuss the impact of different instruction tuning data clustering approaches, replace our Semantic-structuraal Recovery Instruction Clustering RS²RIC) module with some other common text clustering method: NMF TFIDF, LDA_ TFIDE, KMeans TFIDE, Spectral LMTrE, Spec- EMIDn,
val_BERT (Xu et al.] 2024a). The reasoning performance comparison among different PASER versions with such clustering methods is provided in Table 19 From the table, we can find that inte- grating other instruction clustering methods with PASER can bring the performance decline to some extent among all four pruning schemes. Especially, the clustering method with traditional statistics- based text representation technique, TFIDF, generally behaves worse than semantic embedding-basee based text representation techniques like BERT. Therefore, we can conclude that our semantic- ctattic-tttuturall recovery instruction clustering is at least competitive approach as the clustering component of PASER. Though, comparing these results with those in Table we can observe the advantages of PASER over other general instruction tuning data selection methods can still be stably maintained. This further demonstrates that the potential of the clustering- data selection for effective and balanced LLM capability recovery.
Table 20: Recovery performance of multiple PASER versions integrated with different capability degradation assessment approaches under various pruning schemes on LLaMA2-7B model. The PASER(ISD) is the version we employ in the main body. The 'bold' represents the best performance under the same pruning scheme. Here, the Alpaca is taken the original dataset.
Pruning
Recovery tesi-trinnig
WikiText21
PTB.
BoolQ
PIQA
HellaSwag
WineGrande
ARC-e
ARC-c
OBQA
Average
w/o proning
w/o Training
12.62
22.14
71.13
78:40
72.79
67.17
69.36
40.70
40.80
62.91
Training
20.34
38.81
61.87
76.61
65.86
LLM-Pruner
PASER(KLD)
16.91
27.54
66.52
76.95
68.24
63.84
66.58
38.74
39:70
60.37
25%
PASER WD)
16.73
27.26
66.81
77.10
68.58
66.18
67.08
39.09
39.80
60.59
PASERUSD)
16.49
26.35
67.25
77.29
68.98
66.97
67.84
39.54
39.80
61.10
Wo Traiing
44.33
80.07
65.54
06.87
$4,16
63.38
58.46
34.36
36.90
54.27
SliceGPT
PASER(KLD)
13.81
23.64
71.59
78.87
72.28
67.85
69.94
40.84
40.50
63.12
25t
PASER(WD)
13.10
22.75
72.08
79.25
72.93
68.39
70.48
41.26
40.90
63.61
PASER(SSD)
12.24
21.53
72.75
79.84
73.92
69.18
71.37
41.82
41.30
64.31
Training
42.10
76.85
69.30
71.39
33.06
28.07
Wanda
PASER(KLDD)
15.29
29:08
69 85
77.13
70.65
65.47
67.58
38.52
39:
61.20
sparsity=2:4
PASER WD)
14.64
28.21
70.42
77.54
71.23
65.86
67.95
38.78
39.70
61.64
PASERUSD)
14.13
27.22
70.27
77.87
71.78
66.26
68.30
39.04
40.0
62.02
WO Traning
19.25
36.41
71.22
75.60
62.85
66.06
69.11
36.86
37.80
59,93
SpaneGPT
PKLD
14.76
25.97
73.51
77.71
65.87
68.35
71.82
38.36
38. 90
62.07
sparsily=00%
PASER WD)
14.21
25.18
74.15
78.05
66.24
68.68
72.19
38.53
39.11 39.10
62.42
PASER(ISD)
13.33
23.77
74.79
78.38
66.62
69.03
72.57
38.70
39.40
62.78
N.4 EXPLORATION ON OTHER POSSIBLE CAPABILITY DEGRADATION ASSESSMENT METHODS
To further demonstrate the advantages of utilizing Jensen-Shannen divergence (JSD) as the capa- bility degradation assessment approach (claborated in Section 3.3), we compare it with alternative schemes like Hellback-Leible divergence (KLD) (Kullback & Leibler 1951) and Wasserstein dis- tance (WD) (Vaserstein) 1969). As shown in Table 20 JSD consistently outperforms alternative divergence measures across all pruning schemes. While KLD is widely used dissimilarity measure, its asymmetric nature (KLD(PIIQ) + KLD(Q]IP)) makes it less suitable for comparing output distributions between original and pruned models, resulting in 0.73 points average performance drop compared to JSD. The WD performs better than KLD, but still falls 0.51 points short of JSD on average reasoning performance. Although WD can capture gradual distribution shifts that might be missed by other divergence measures. it appears to be more sensitive to outlier probability dif- ferences that don't necessarily correlate with capability degradation. This suggests that major shifts in probability mass, rather than fine-grained transportation costs, are more indicative of capability deterioration. JSD's symmetry and boundedness (always between 0 and 1) provide a more stable and interpretable measure of capability divergence, enabling more accurate prioritization of severely affected capabilities. Its ability to balance sensitivity to both major and minor distribution shifts makes it particularly effective for detecting subtle capability degradations that might be missed by other measures. The superior performance of JSD across all four pruning schemes validates our design choice in Section 3.3
36

---

Published as a conference paper at ICLR 2026
N.5 EXPLORATION OTHER POSSIBLE BUDGET ALLOCATION STRATEGIES
Beyond our proportional allocation strategy in Equation 6 we explore alternative allocation strategies in- cluding equal allocation, square- root scaling (ng x (CDS(ca). and logarithmic scaling (ng
Table 21: Recovery performance of different budget alloca- tion strategies under various pruning schemes on LLaMA2-7B model. The Linear proportional is the version the main body. The 'bold' represents the best performance under the same pruning scheme. Here, the Alpaca is taken as the original dataset.
employ in
log(1 + (c ))). As shown
Table 21 the linear proportional al-
location consistently outperforms al-
Allocation Stratogy Equal
LLM-Pruner 611.34
SliceGPT
Wanda
SparseGPT
ternatives, showing ~0.4-0.88 points higher average performance across pruning schemes. This validates our design choice while demonstrating that PASER maintains reasonable
63.52
61.28
62.05
Square-root sculing
60.68
61.56
62.31
Logurithmic scaling
60.51
63.65
61.43
62.18
Linear proportional
61.10
64.31
62.02
62.78
performance even with different allocation strategies.
O FINE-GRAINED DATA SELECTION TIME CONSUMPTION ANALYSIS
In addition the recovery post-training efficiency analysis provided in the Figure Section 4.2 we also analyze the fine-graines time consumption of each component provide deeper understand- ing for PASER's efficiency advantages during data selection phase. In the experiments, we follow
Table 22: Time consumption breakdown of PASER components on different-sized datasets.
Alpaca (52K)
LaMini (2.58M)
Component
Time (min)
Percentage
Time (hrs)
Percentage
Recovery Instruction Clustering
SentenceBERT Embedding
0.63
11% %
0.12
10%
Diffusion Kernel
1.64
28%
0.36
31%
NMF Spectral Clustering
0.82
14%
0.15
13%
Capability Degradation-Aware Instruction Selection
JSD Computation
0.35
6%
0.07
6%
Budget Allocation
0.12
2%
0.02
2%
Negative Tuning Effects Mitigation
Concept Extraction
1.22
21% %
0.26
22%
CCG Construction
0.71
12%
0.14
12%
Consistency Checking
0.34
6%
0.06
5%
Total Selection Time
5.83
100%
1.18
100%
Recovery Training (Full Data)
83.63
57.97
Recovery Training (PASER)
26.71
4.29
Time Saving
56.92
68.1%
53.68
92.6%
the standard settings in the main body experiments: [B|=10.4K (20%) for Alpaca and [BI=103.2K (4%) for LaMini. Our component-wise time consumption analysis reveals the relative computational costs across PASER's ' pipeline, detailed in Table 22 The semantic-structure clustering process RICRIC) consumes approximately 54% the total selection time, with the Diffusion Kernel phase (30%) requiring the most computation in this component due to eigendecomposition operations.
The capability degradation assessment and budget allocation constitute about 8% of the selection time. Here, JSD computation is the most intensive operation (6%). while budget allocation using Equation requires minimal overhead (2%). This aligns with our theoretical analysis, as JSD computation involves calculating probability distributions across the vocabulary space for each token.
37

---

Published as a conference paper at ICLR 2026
The negative tuning effects mitigation component, particularly building and maintaining the Concept Consistency Graph, accounts for 39% of the selection process. Concept extraction (22%) and CCG construction during sample selection (12%) are the key operations. For Alpaca, the complete CCG- based filtering process required around 2.27 minutes but prevented negative tuning effects that would have otherwise caused performance degradation of 0.68-2.39 points across tasks.
For larger datasets like LaMini (2.58M samples), we implemented optimizations to maintain scal- ability: (1) approximate k-nearest neighbors with locality-sensitive hashing for manifold learning, reducing complexity from O(N²) to O(N log N); (2) parallel processing for JSD computation across multiple CPU cores; and (3) incremental CCG updates with optimized data structures. With these optimizations, the complete data selection process for LaMini took approximately 1.18 hours, significantly less than the days required for full dataset recovery training.
Despite the overhead of data selection (5.83 minutes for Alpaca and 1.18 hours for LaMini), PASER achieves substantial overall time savings: 68. 18.1% for Alpaca and 92.6% for LaMini compared to full dataset training. These savings stem from the reduced training data volume (20% for Alpaca and 4% for LaMini) and the focus on efficiency-driven sample selection that prioritizes samples with high iiformation-ooccompuaatit ratios.
The empirical measurements confirm our theoretical analysis in Section 3.5 with the number of concepts per sample C' being relatively small (average of 5-7 concepts per instruction), the dominant time complexity factor remains ON log N). This enables PASER to scale efficiently to large instruction tuning datasets while maintaining its performance advantages over conventional recovery approaches.
Scaling Experiments Extremely Large Dataset To further explore PASER's scalability to extremely large datasets, we investigate the online resources and find that the xP3x (Muennighofi al. 2023) which contains 532 million samples is the largest open-sourced instruction tuning dataset we can download. We conduct experiments on a 8-node GPU cluster, where each node is
equipped with dual AMD EPYC 9K84 CPUs (192 cores 384 threads per node) and 8 NVIDIA H20 GPUs. The fine-grained time consumption on xP3x for each component is as follows: SentenceBERT
Embedding: 0.34 hours; Manifold Learing: 0.98 hours: NMF Spectral Clustering: 0.43 hours;
ISD Computation: 0.19 hours; Budget Allocation: 0,06 hours; Concept Extraction: 0.65 hours;
CCG Construction: 0.39 hours; Consistency Checking: 0.17 hours; Total Selection Time: 3.21
hours; Recovery Training (Full Data): 50+ days (estimated): Recovery Training (PASER): 13.9
hours; Time Saving: ~99%. Such results on nearly billion-scale datasets further validates PASER's time efficiency. In the future, beyond above mentioned efficiency optimization methods, we can also consider applying quantization (FP16/INTTS to embeddings during the clustering phase, and calculating JSD over the top-K logits (or low-rank logit projection) instead of full logits during the capability degradation assessment.
P CASE STUDY FOR RECOVERY INSTRUCTION CLUSTERING
To illustrate the effectiveness of our approach for grouping samples focusing on similar capabilities together, we conduct case study of clustered instruction samples from the Alpaca dataset. Specifically, we provide representative samples from several obtained clusters as follows.
Recovery Instruction Clustering RS"RIC)
P.1 CLUSTER 1: BASIC FACTUAL KNOWLEDGE AND INFORMATION RETRIEVAL
Instruction: "Find the five largest cities in France."
Instruction: "What is the capital of France?"
Instruction: "Find the population density of United States."
These instructions primarily test the model's ability to recall basic facts and information, correspond- ing general knowledge capabilities.
38

---

Published as a conference paper at ICLR 2026
P.2 CLUSTER 2: LANGUAGE UNDERSTANDING AND TRANSLATION
Instruction: "Translate the word giraffe' to French."
Instruction: "Pick the correct Spanish translation of "Hello"."
Instruction: "Difference in meaning between "done freely" and "freely done"? For instance, is there any difference in meaning between these two sentences?"
This cluster focuses on language-related tasks, including translation, idiomatic expressions, and grammatical analysis.
P.3 CLUSTER 3: LOGICAL REASONING AND PROBLEM SOLVING
Instruction: "A friend shares the following text with you and asks for your opinion: 'Purple-eyed individuals have stronger psychic connection to the cosmos and have more chances to predict the future.' Analyze the statements and point out logical fallacies or unsupported claims."
Instruction: "Explain how to solve Sudoku puzzle in three steps."
Instruction: "Answer this math question: What is the value the power of 5?"
These instructions test the model's ability to perform mathematical calculations, logical deductions, and pattern recognition.
P.4 CLUSTER 4: CREATIVE WRITING AND TEXT GENERATION
Instruction: "Write a microblog post about a recent experience you had."
Instruction: "Compose haiku about the wonders of technology."
Instruction: "Create illustration of the inside of castle."
This cluster groups tasks that require creative text generation, showcasing the model's ability to produce original content across various formats and topics.
P.5 CLUSTER 5: SUMMARIZATION AND INFORMATION EXTRACTION
Instruction: "Summarize the techniques used for summarizing text."
Instruction: "Extract the main argument from the passage."
Instruction: "Provide brief summary of the article "A Brief History of the Automobile Industry"."
These instructions focus on the model's capability to condense information and identify key points from longer texts.
This study demonstrates that S²RIC effectively groups instructions targeting similar LLM capabilities, enabling PASER allocate recovery efforts, i.c., data budget, more strategically. By focusing intensively on clusters where capabilities have degraded most after the pruning, while maintaining awareness of all capability clusters, PASER achieves both targeted and balanced recovery. This dual approach ensures efficient and comprehensive restoration of the model's diverse functionalities, optimizing the recovery process for pruned LLMs.
Q VISUALIZATION FOR RECOVERY INSTRUCTION CLUSTERING
To better illustrate the semantic relationship of clusters after our Semantic-Structure Recovery Instruction Clustering, samples in each cluster correspond to those in above Appendix P For example, the instructions numbered as 1,2,3 in Language Understanding cluster are the three samples provided in Appendix P.2 From this figure, we can find that our approach successfully identifies meaningful semantic structures in the instruction space that correspond to different L.MM capabilities.
provide the visualization results in the Figure 6 Notably, the numbered
39

---

Published as a conference paper at ICLR 2026
PASER: Semantic-Structtral Clustering of Instruction Data by LLM Capabilities
Manifold Dimension
2
6
-2
Manifold Dimension 1
Language Understanding (Large curved structure) Lagica//athemaaiicl Reasoning (Dense, elliptical structure) Factual Knowledge (Medium density, arc structure)
Creative Writing bimodal structure)
Summarization (Small, compact structure)
Figure 6: Visualization for the
clustering of recovery instruction data by LLM capabilitics. For case of presentation, only the first two manifold dimensions are utilized.
R
CASE STUDY FOR NEGATIVE TUNING EFFECTS MITIGATION
To illustrate the negative tuning effects mitigation process in PASER, present the following study, including both initial concept consistency graph construction and new sample evaluation.
R. CONSTRUCTING INITIAL CONCEPT CONSISTENCY GRAPH
We begin with three accepted iesrruction-uupul pairs construct our initial Concept Consistency Graph (CCG):
1. Instruction: "Explain the basic principles of quantum computing."
"Quantum computing is based on the principles of quantum mechanics. uses quantum bits qubits, which can exist in multiple states simultaneously due to superposition. Quantum entanglement allows qubits to be correlated with each other. These properties enable quantum computers perform certain calculations much faster than classical computers for specific problems."
Output:
Concepts:
(quantum computing, qubit, superposition, entanglement)
Instruction:
"Describe the main components of a classical computer."
Output: "A classical computer typically consists of several key components: the Central Pro- cessing Unit (CPU) which performs calculations and controls other components; Random Access Memory (RAM) for temporary data storage; hard drive for long-term data storage; and various
40

---

Published as a conference paper at ICLR 2026
input/output devices. These components work together to process and store information using binary logic." "
Concepts: (CPU, RAM, hard drive, binary logic)
3. Instruction: "What are the fundamental concepts in deep learning?"
"Deep learning is subset of machine learning based artificial neural networks. Key concepts include: neural networks with multiple layers (deep neural networks), which can learn hierarchical representations of data; backpropagation, an algorithm for training these networks by adjusting weights; and various optimization techniques to improve learning. Deep learning has shown remarkable success tasks such as image and speech recognition." Concepts: (deep learning, neural network, backpropagation. optimization
Output:
Based on these samples, we construct an initial Concept Consistency Graph (CCG):
V {quannum computing, qubit, superposition, entanglement, CPU, RAM, hard drive, binary logic,
deep learning, neural network, backpropagation, optimization)
The edges in the graph represent the co-occurrence concepts within each instruction-output pair.
E (qquantum computing, qubit), (quantum computing, superposition), (quantum computing, entanglement), (qubit, superposition),
(qubit, entanglement), (superposition, entanglement),
(CPU, RAM), (CPU, hard drive), (CPU, binary logic), (RAM, hard drive), (RAM, binary logic), (hard drive, binary logic), (deep learning, neural network), (deep learning, backpropagation), (deep learning, optimization), (neural network, backpropagation). (neural network, optimization), (backpropagation, optimization)}
This explicit definition of E clearly shows the relationships between concepts within each domain, while highlighting the absence of edges between concepts from different domains.
R.2
EVALUATING NEW SAMPLES
now evaluate two new samples using PASER's negative tuning effects mitigation process.
R.2.1 SAMPLE 1: ACCEPTED
Instruction: "How does increasing the number qubits affect quantum computer's performance?"
"Increasing the number of qubits in quantum computer can significantly enhance its performance and capabilities:
Output:
1. Exponential growth in computational power: Each additional qubit doubles the number of quantum states that can be represented, leading to an exponential increase in computational power.
2. Handling more complex problems: More qubits allow quantum computers to tackle larger and more complex problems that are intractable for classical computers.
3. Improved error correction: A larger number of qubits be used implement more robust quan- tum eπrоr correction codes, improving the overall stability and reliability of quantum computations.
4. Enhanced quantum parallelism: More qubits enable greater parallelism in quantum algorithms, potentially leading to faster solution times for certain problems.
5. Challenges in scalability: However, increasing the number of qubits also introduces challenges in maintaining coherence and reducing error rates, which are active areas of research in quantum computing.
In summary, while increasing the number of qubits can dramatically improve a quantum computer's capabilities, it also comes with technical challenges that researchers working to overcome."
41

---

Published as a conference paper at ICLR 2026
Concepts:
(qubit, quantum computer, computational power, quantum states, error correction, quan- tum parallefism, coherence)
Applying PASER's negative tuning effects mitigation process:
1. Concept Extraction: C(z,, y) {qubit, quantum computer, computational power, quantum states, error correction, quantum parallelism, coherence)
2. Consistency Check: For each pair (vi, Uj) E C(x,, v):
(qubit, quantum computer): both E V and (et, 0j) E E
(qubit, computational power): qubit E V, computational power & V
(qubit, quantum states): qubit V, quantum states & V
Other pairs: at least one concept & V
3. Decision; IsConsistent(x,
True, because for all pairs, either (ri, my) E E or (1), & V
PASER accepts this sample as it is consistent with the existing CCG and introduces relevant new concepts without conflicts.
R.2.2 SAMPLE 2: REJECTED
Instruction:
"Explain how quantum computing can be used to significantly speed up deep learning algorithms."
"Quantum computing has the potential to speed up certain aspects of deep learning al- gorithms. Quantum algorithms could potentially optimize the training process of neural networks, especially in handling large-dimensional data. However, it's important tages and applications are still being researched, and classical computers remain more practical for most current deep learning tasks."
Output:
that the exact advan-
Concepts: [quantum computing, deep learning, neural network, speedup)
Applying PASER's negative tuning effects mitigation process:
Concept Extraction: C(z, y) {quantum computing, deep learning, neural network, speedup)
Consistency Check: For each pair (vi, vy) E C(x,,) y):
(quantum computing, deep learning): both V, but (Ni, 05) & E
(quantum computing, neural network): both E V. but (Vi, 03) $ E
(deep learning, neural network): both E V and (e;, 03) E E
(speedup, any other concept): speedup &
Decision: IsConsistent(x, y) = False, because the pairs (quantum computing, deep learning) and (quantum computing, neural network) have both concepts in V, but these edges do not exist in E, This introduces new relationships between existing concepts that are not present in the current CCG.
PASER rejects this sample because it introduces direct relationships between quantum computing and deep learning/neural networks, which were not present in the initial CCG. While these concepts existed separately in the CCG, their combination in this context could lead to potential misunder- standings or oversimplifications about the current state and capabilities of quantum computing in machine learning.
R.3 CONCLUSION
This case study demonstrates PASER's negative tuning effects mitigation process in action. By accepting Sample 1, PASER allows for the introduction of new, relevant concepts that expand the concept consistency graph without introducing conflicts. By rejecting Sample 2, PASER prevents the introduction of potentially misleading relationships between existing concepts from different domains, thus mitigating the risk of negative tuning effects during the recovery process.
42

---

Published as a conference paper at ICLR 2026
S ANALYSIS ON PRUNING SEVERITY AND PASER RECOVERABILITY
This section provides analysis and clarification how different pruning strategies affect model recoverability and how PASER interacts with such degradation patterns.
Sapability-Speccfic Impact of Different Pruning Strategies Pruning does not degrade all capa- bilities uniformly. As discussed in the main text and Figure 4 to diffuse and moderate degradation, leaving many capabilities least partially recoverable. In contrast, semi-structured and structured pruning introduce more localized and capability-selective selletive damage, often disproportionatel affecting reasoning- heavy or composition- heavy capabilities. In some architectures, structured pruning (e.g.. L.LM-Pruney) or high-ratio pruning may disrupt func- tional pathways, such as attention-head specialization or layerwise routing, thereby causing structural degradation rather than a simple reduction in parameter count.
unstructured pruning typically leads
Recoverable vs. Irrecoverable Capability Regions PASER focuses recovery efforts on the parts of the pruned model that remain expressively intact. Capabilities whose representational support is degraded but still present tend to respond well to targeted recovery through our capability-aware data selection. By contrast, capabilities whose computational pathways have been effectively removed by pruning exhibit limited improvement regardless of the quality or quantity of recovery data. This distinction explains why PASER produces substantial gains in most settings but cannot fully restore capability clusters whose underlying structure has been fundamentally pruned away.
Interaction Between PASER's Pruning-Awars Aware Design and Model Degradation PASER's pruning- aware design further shapes its interaction with different levels of model degradation. As described in Sections 3.3 and 3.4 PASER identifies capability clusters that suffer more severe pruning-induced degradation and allocates recovery data accordingly. It filters conflicting or harmful samples that could exacerbate uneven degradation during fine-tuning, and it avoids over-training intact clusters, which helps preserve the remaining expressivity of the pruned model. These mechanisms enable PASER to remain effective across a range of pruning strategies, even when the degradation substantial or uneven. Nevertheless, PASER necessarily operates within the recoverable subspace: once key structural representations are removed by pruning, recovery naturally plateaus. This limitation is inherent to all post-training methods, not specific to our approach.
Why Baichuan2-7B Shows Weaker Recovery in Some Cases The weaker recovery observed in Baichuan2-7B (as shown in Table can be understood in this context. Further inspection suggests that Baichuan2-7B contains more functionally specialized attention heads and feedforward pathways, and some of its reasoning related components exhibit lower redundancy compared to models such as LLaMA or Qwen. As a result, structured pruning causes more irreversible representational loss in this architecture, and the affected capability clusters become inherently harder to restore. Structural pruming therefore removes more of the functional backbone needed for downstream recovery, limiting the improvements achievable during post-training.
In summary, pruning severity sets the upper bound on the recoverability of a pruned model. PASER leverages pruning- aware signals to maximize recovery within the expressive capacity that remains, but structural representational loss cannot be reconstructed through post-training alone. Architectural differences further influence this recoverability boundary, helping to explain why some model families are more sensitive to structured pruning than others.
T IMPLEMENTATION DETAILS AND HYPERPARAMETER SETTINGS
Most of experiments are conducted on the server with 8 x NVIDIA RTX 6000 Ada GPUs. Part of experiments on 70B model and MOE models are conducted on NVIDIA H800 Tensor Core GPU cluster. For the Semantic-Structt across all experiments: diffusion time ł is automatically selected using the spectral gap method, and the embedding dimension is set to 16. The optimal number of clusters K is determined adaptively through NMF approximation error minimization, typically resulting in 8-12 clusters for Alpaca and 15-20 clusters for LaMini. For the JSD calculation in capability degradation score, we use a temperature 7=1.0 for the output probability distribution. The computational cost is approximated using the quadratic term of sequence length with a coefficient of 1.0 across all experiments. For concept extraction, use maximum of 10 concepts per instructioonrespponee pair with minimum
Recovery Instruction Clustering, we use consistent settings
43

---

Published as a conference paper at ICLR 2026
phrase length of 2 words and a maximum of 4 words. The concept similarity threshold for consisteney checking is set to 0.75 across all experiments. We maintain these same hyperparameter settings across all models and pruning schemes to ensure fair comparison. During the recovery post-training phase, we take the the low-rank approximation, LoRA (Hu et a to improve the efficiency. The corresponding hyperparameters
set as following: rank=8, batch size=64, epochs=2, learning rate = =ee-4 (Alpaca series experiments). 5e-5 (LaMini series experiments). As for the structured pruning, we set the pruning ratio as 25% for 13B models. For the other two kinds of pruning schemes, we follow the previous work (Frantar Alistarh 2023). Specifically, we adopt the 2:4 semi-structured sparsity patterns and implement 50% unstructured weight sparsity. Except the experiments for recovery post-training efficiency analysis, we set the ratio of recovery data budget
and 50% for LLaMA2-
original dataset size N as 20% for A Alpaca and 4% for LaMini. As for the implementation of concept extraction in Section 3.4 [
we use the open-source library r in this paper are the averages of five runs with different seeds. Statistical significance is also assessed using two-tailed independent t-tests, with results considered significant when < 0.01. our experiments, find all our reported results for PASER pass this statistical significance test. For facilitating the reproduction of our work, we provide the code in the supplementary materials, also seen in Github wtw.hhttps.... https::// hwtp:///www/wwaa/o/ooo....
To ensure statistical robustness, all the results reported
U LIMITATION ANALYSIS
While PASER demonstrates significant improvements recovery performance and efficiency for pruned large language models, there are several limitations consider:
Computational overhead: Although PASER reduces the recovery training time, the initial cluster- ing and data selection process introduces some computational overhead. For very large instruction tuning datasets, this overhead may become non-trivial.
Dependence on initial pruning quality: The effectiveness of PASER may vary depending on the quality and method of the initial model pruning. Poorly pruned models might not benefit as much from the targeted recovery approach.
Potential bias in capability recovery: While PASER aims for balanced capability recovery, there might still be some bias towards certain capabilities based on the initial clustering results and the composition of the instruction tuning dataset.
The paper primarily demonstrates results on models up to 70B parameters. The scalability and effectiveness of PASER even larger models (e.g., 100B+ parameters) need further investigation.
Scalability to extremely large models:
Long-tern torg-term Stability: continued fine-tuning or adaptation, has not been thoroughly examined this work.
The long-term stability of models recovered using PASER, especially under
Linitations of Concept Consistency Graph. In addition to the above analysis, we further discuss the concept consistency graph's 's potential limitations. Indeed, while CCG helps mitigate negative tuning effect, acknowledge there are scenarios where semantic conflicts might not be fully captured:
Cross-domain Knowledge Integration: When instructions involve integrating knowledge from multiple distinct domains, CCG might miss subtle conflicts in their interactions. For example, when concepts from physics and biology are combined in interdisciplinary problems, their complex relationships and potential incompatibilities may not be fully reflected in simple co-occurrence patterns.
Context-dependent Semantics: The same concept pairs might have different relationships de- pending on context. For instance, terms like "positive" and "negative" could be contradictory in sentiment analysis but complementary in mathematics, making it challenging for CCG to maintain consistent concept relationships across different contexts.
Temporal Version-specific Conflicts: In rapidly evolving domains like technology or scientific research, concept relationships might change over time. An instruction about sstate-of-the-ee
44

---

Published as a conference paper at ICLR 2026
performance" or "current best practices" could contain outdated conflicting information that is not immediately apparent from concept co-occurrence analysis.
Nuanced Conceptual Dependencies: When instructions involve subtle logical dependencies or conditional relationships between concepts, the binary edge representation in CCG might not fully capture these complex interactions. This particularly evident in reasoning tasks where conclusions depend specific combinations of conditions.
Our results acknowledge these inherent limitations while demonstrating CCG's overall effectiveness in practical applications.
V DECLARATION OF LLM USAGE
We used ceneral-purposss large language model assistant solely for language editing (grammar polishing, minor rephrasing, and shortening) and for non-substantive code scaffolding (e.g., generating boilerplate for plotting or CLI argument parsing). All research ideas, algorithmic designs (PASER), theoretical analyses, experiment setups, and reported results were conceived, implemented, and verified by the authors. human verification, and no conclusions rely unverifiable LLM outputs. No training evaluation data were created with an LLM for this work. We rely on publicly available instruction- tuning datasets (c.g., Alpaca and LaMini), whose responses were originally synthesized by third-party LLMs as documented by their providers; we used them as-is without further LLM-based modification. The assistant not used to generate novel technical content unique to this paper (e.g., PASER's methodology, equations, or ablation study design). All code that implements PASER's 's selection pipeline and all experimental decisions were written and reviewed by the authors. The LLM is an author and bears no responsibility for the work; accountability lies with the human authors. When LLM suggestions influenced wording, the authors performed fact-checking and ensured compliance with dataset licenses and double-blin review requirements
figures, tables, proofs, or claims were generated by an LLM without
45


Reviews:
[{"review_id": "ISvtCeSOUZ", "reviewer": "Reviewer_spjs", "summary": "This paper aims to solve the practical problem of post-pruning recovery via data selection. It offers a practical solution by combining clustering and degradation metrics, though it is a bit incremental over prior data selection methods.", "questions": "1. How does PASER substantially differ from the applying existing data selection methods (e.g., LESS, DELIFT) directly to the pruned model recovery scenario? The paper only shows performance comparisons but lacks an in-depth analysis and conceptual differentiation.\n\n2. The concept consistency graph appears to be a bit over-simplistic. How does it handle complex semantic conflicts that cannot be captured by simple co-occurrence patterns?\n\n3. Why JSD was chosen over more sophisticated divergence measures?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) The integration of manifold learning and JSD-based degradation scoring is creative and tailored to the pruning recovery context.\n\n(2) Extensive evaluations on models like LLaMA2 and Baichuan2 show robust performance gains and efficiency improvements. The evaluation section is comprehensive, covering multiple models and pruning schemes.", "weaknesses": "(1) The core methodology seems to originate from various existing works. The clustering approach resembles semantic sampling in SMART, while the degradation scoring echoes influence-based selection in Datamodels. How the proposed method is related and differs from the existing approached should be discussed in the main paper, instead of  in the Appendix as has been done in the current paper.\n\n(2) The computational overhead of manifold learning and JSD calculation is non-trivial, yet the paper provides insufficient analysis of scalability.\n\n(3) While having included recent baselines, the paper is lack of presenting an in-depth analysis about the reasons why PASER differs fundamentally from general data selection methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r0F9LXbpPP", "reviewer": "Reviewer_wdMP", "summary": "This paper introduces PASER (Post-training dAta Selection for Efficient pruned large language model Recovery), a novel framework designed to address the significant performance degradation of Large Language Models (LLMs) after pruning. The core challenge is that pruning affects different model capabilities (e.g., reasoning, coding, math) unevenly, and conventional post-training recovery using full instruction datasets is computationally expensive and can be suboptimal due to irrelevant or conflicting data. PASER tackles this with a three-stage, data-centric approach. Extensive experiments on models like LLaMA2/3 and Baichuan2 under various pruning schemes (LLM-Pruner, SliceGPT, Wanda, SparseGPT) demonstrate that PASER significantly outperforms baseline methods (including random selection and general data selection techniques like IFD and Nuggets).", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper effectively identifies a critical and under-explored gap in the LLM compression pipeline: the need for targeted, efficient recovery post-training that accounts for the uneven degradation of model capabilities after pruning.\n2. The experimental section is a major strength. The authors validate PASER across multiple model families (LLaMA, Baichuan, Qwen, Mixtral), pruning methods (structured, semi-structured, unstructured), and dataset scales (Alpaca, LaMini). The consistent superiority over strong baselines, and the ability to sometimes exceed the performance of the unpruned model, provides compelling evidence for its effectiveness.", "weaknesses": "1. While PASER drastically reduces training time, the data selection process itself (clustering, JSD calculation, CCG management) is non-trivial. A more detailed analysis of this overhead, especially for massive datasets (e.g., billions of samples), and potential optimizations for the selection phase would be valuable for real-world scalability.\n2. The CCG is a heuristic-based method. Its effectiveness is limited by the quality of concept extraction and may not capture complex, context-dependent, or cross-domain semantic conflicts. The paper acknowledges this in the appendix, but it remains a potential point of failure where nuanced conflicts could be missed.\n3.  The method's performance is inherently tied to the initial pruning step. The paper notes that for models where pruning \"severely damaged the model internal structure\" (e.g., Baichuan2-7B), recovery is less effective. A deeper investigation into the relationship between pruning strategy, resultant damage, and PASER's recoverability would be insightful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nhYzloctfA", "reviewer": "Reviewer_LptJ", "summary": "This paper introduces PASER, a novel framework for selecting a small, high-impact subset of instruction-tuning data to recover the performance of pruned LLMs. The authors identify three key challenges with existing recovery methods: high computational cost, failure to address the uneven degradation of capabilities after pruning, and the potential for negative tuning effects from irrelevant data. PASER addresses these challenges through a three-stage process: semantic-structural recovery instruction clustering,capability degradation-aware instruction selection, and negative tuning effects mitigation. The authors conduct extensive experiments on a wide range of LLMs, pruning schemes, and instruction datasets. The results demonstrate that PASER significantly outperforms baselines, recovering model performance to near-original levels while using only 4%-20% of the instruction data and substantially reducing training time.", "questions": "1. In Table 1, the results of training the LLM-Pruner pruned model with full data is even lower than without training, (35.64 vs 57.78) how to interpret this?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. High Relevance and Strong Motivation: The problem of efficiently recovering performance in compressed LLMs is highly practical and significant, while underexplored. The paper does an excellent job of motivating the work by clearly identifying the limitations of naive retraining, particularly the issues of uneven capability degradation and negative tuning effects, which are often overlooked.\n\n2. Novel and Well-Designed Methodology: The proposed PASER framework is technically sound, innovative, and directly addresses the identified problems.\n\n3. Extremely Comprehensive and Rigorous Evaluation: The experimental validation in this paper is exceptional and serves as a model for thoroughness.\n\n4. Strong and Convincing Empirical Results: PASER consistently and significantly outperforms all baselines across nearly all tested configurations. The finding that it can recover performance to a level that is sometimes better than the original unpruned model is particularly striking and suggests that the method effectively selects a high-quality, coherent subset of data that is superior to the full, noisy dataset for tuning. The efficiency gains in terms of training time are also substantial and well-documented.", "weaknesses": "The paper is of very high quality, and it is difficult to find significant weaknesses. The following points are minor.\n1. Methodological Complexity: The PASER pipeline is composed of several sophisticated components (manifold learning, NMF spectral clustering, CCG). While the authors demonstrate its effectiveness and computational efficiency, the overall complexity is higher than simpler data selection heuristics. This might pose a slight barrier to adoption for practitioners looking for a \"plug-and-play\" solution, though the provided code should mitigate this.\n2. The writing of the methodology section can be improved to reduce the cognitive load on the reader.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sHe8b0Zq3E", "reviewer": "Reviewer_bfNy", "summary": "The paper proposes PASER, a data selection framework designed to improve the efficiency and effectiveness of recovery post-training after LLM pruning. PASER employs manifold learning and spectral clustering to group instructions in the semantic space, allocates data budgets based on measured capability degradation, and filters out conflicting or irrelevant data to mitigate negative tuning effects. Experiments on several pruning schemes (structured, semi-structured, unstructured) and multiple LLM frameworks (LLaMA 2/3, Baichuan 2) demonstrate that PASER can recover model performance using only 4%–20% of the original instruction tuning data.", "questions": "1. Could the authors clarify the practical advantages of pruning a large model compared to directly training a smaller one from scratch? \n2. Is it possible for the authors to provide a baseline of directly training or fine-tuning a smaller model with the instruction tuning data?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper tackles a problem about how to efficiently recover LLM capabilities after pruning. This issue has become increasingly relevant for real-world model deployment. The study is supported by comprehensive experiments covering multiple pruning strategies and models of different scales, ensuring good empirical validation. Results demonstrate consistent performance improvements over both random selection and other instruction tuning data selection baselines. Results of the ablation studies helps clarify the contribution of each system component, enhancing the overall credibility and interpretability.", "weaknesses": "1. Some ideas resemble ideas from earlier literature (e.g., similarity-based data sampling). The paper would benefit from a more thorough literature review, though this does not undermine its novelty.\n2. The paper is generally well-written, there are a few minor oversights. For example, there are a few typos (“we relies on…”), and some abbreviations used in the ablation study section (S2RIC, CDAIS, NTEM) are not introduced before but only defined in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)