Restoring Pruned Large Language Models via Lost Component Compensation
Authors: Zijian Feng
Hanzhang Zhou
Zixiao Zhu
Tianjiao Li
Chua Jia Jim Deryl
Mak Lee Onn
Gee Wah Ng
Kezhi Mao
Venue: NeurIPS 2025 spotlight ()
Keywords: large language models
pruning
performance restoration
Primary area: deep_learning
TL;DR: A targeted restoration method to recover pruned models while preserving their low cost and high efficiency

Abstract: Pruning is a widely used technique to reduce the size and inference cost of large language models (LLMs), but it often causes performance degradation. To mitigate this, existing restoration methods typically employ parameter-efficient fine-tuning (PEFT), such as LoRA, to recover the pruned model's performance. However, most PEFT methods are designed for dense models and overlook the distinct properties of pruned models, often resulting in suboptimal recovery. In this work, we propose a targeted restoration strategy for pruned models that restores performance while preserving their low cost and high efficiency. We observe that pruning-induced information loss is reflected in attention activations, and selectively reintroducing components of this information can significantly recover model performance. Based on this insight, we introduce RestoreLCC (Restoring Pruned LLMs via Lost Component Compensation), a plug-and-play method that contrastively probes critical attention heads via activation editing, extracts lost components from activation differences, and finally injects them back into the corresponding pruned heads for compensation and recovery. RestoreLCC is compatible with structured, semi-structured, and unstructured pruning schemes. Extensive experiments demonstrate that RestoreLCC consistently outperforms state-of-the-art baselines in both general and task-specific performance recovery, without compromising the sparsity or inference efficiency of pruned models.

Restoring Pruned Large Language Models via Lost Component Compensation
Zijian Feng'
Hanzhang Zhou'
Zixiao Zhu' Gee Wah Ng²
Tianjiao Li'
Jia Jim Deryl Chua?
Lee Onn Mak
Kezhi Mao"
School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore Home Team Science and Technology Agency (HTX), Singapore
(feng0119, hanzhang001, zixiao001) wed.......... adtu.edu.gg.
(tianjiao. 1i,, ekzmao)0ntu, edu. {deryl_chua, mak_lee_onn, ng-gee_ wah @htx. hht.tgov.stgg. gov.sg
Abstract
Pruning is widely used technique reduce the size and inference cost of large language models (LLMs), but it often causes performance degradation. To mitigate this, existing restoration methods typically employ parameter--fficient fine-tuning (PEFT), such as LoRA, to recover the pruned model's performance. However, most PEFT methods are designed for dense models and overlook the distinct properties of pruned models, often resulting in suboptimal recovery. In this work, we propose targeted restoration strategy for pruned models that restores performance while preserving their low cost and high efficiency. We observe that pruning-inducce information loss is reflected in attention activations, and selectively reintroduc- ing components of this information can significantly recover model performance. Based on this insight, we introduce RestoreLCC (Restoring Pruned LLMs via Lost Component Compensation), critical attention heads via activation editing, extracts lost components from ac- tivation differences, and finally injects them back into the corresponding pruned heads for compensation and recovery. RestoreL.CC is compatible with structured, stri-structured, and unstructured pruning schemes. Extensive experiments demon- strate that RestoreLCC consistently outperforms sate-of-thhear basclines in both general and lask-specific performance recovery, without compromising the sparsity or inference efficiency of pruned models
plug-and-plaa method that contrastively probes
1
Introduction
Large language models (LLMs) have achieved remarkable success in various natural language processing (NLP) tasks like commonsense reasoning, math problem solving, and text completion However, their large parameter counts demand substantial computational resources for deployment and inference. To democratize LLMs, pruning has emerged as a key technique to reduce model size and accelerate inference [4) Pruning typically involves two steps: weight pruning and performance restoration. Weight pruning methods can be roughly categorized into structured pruning, such LLM-Pruner [5] and SlimGPT (6) semi-structured pruning, and unstructured pruning, such as SparseGPT (7] and Wanda [4. These methods estimate the importance of parameters and zero out unimportant ones reduce model size. Performance restoration critical step in mitigating the
1 2 3
Corresponding author.
Code: http:/ttw....o/iiiiaaaa////ttrrr
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Logit Difference of Attention Head Outputs
Final Output Accuracy
Logit Difference
0.5
Accuracy
76
74
-0.5
Dense
Pruned
72
1.0
LCC
15.a20
70
Den Pruned LCC
Attention Head Position
Figure 1: These figures compare the performance of the original dense model, the pruned model, and two recovery methods applied to the pruned model: LoRA and Lost Component Compensation (LCC, ours), which directly adds back components lost due to pruning. The used dataset BoolQ. and please see $ for experimental details. The left figure shows the logit difference (correct minus incorrect) of attention heads with/without recovery, and the right figure shows the accuracies of final model outputs for each method. I/-/ and af-] indicate the layer index and the head index, respectively. For instance, 112.al6 means the 16-th head at the 12-th layer.
performance degradation caused by weight pruning, aiming to recover model capability by adjusting weights through language modeling or instruction tuning datasets [8 T0].
As full fine-tuning (FT) still requires substantial computational resources, parameter-efficient fine- tuning (PEFT) has become the mainstream approach for restoring pruned models. Recent pruning methods, including LLM- LLM-Pruner, SparseGPT, Wanda, and SlimGPT, all adopt LoRA [1] to recover performance. Although seemingly straightforward, applying existing PEFT methods, such as LoRA- based approaches [1 12 13 concerns.
and representation engineering [14] 15 16,, to pruned LLMs raises
These PEFT methods are originally designed for dense models, where they adapt LLMs to downstream tasks by training small subset of parameters. When applied to pruned models, they often overlook pruning-specine characteristics, such as the need to account for lost information, leading to inefficient parameter search and suboptimal restoration. As shown Figure attention heads recovered using LoRA achieve only limited improvement in logit predictions, resulting in suboptimal performance on the final task accuracy. In contrast, directly reintroducing the pruned components into the pruned attention heads (LCC) significantly improves both the outputs of the attention heads and the overall model accuracy. Built on this insight, we propose RestoreL.CC (Restoring Pruned LLMs via Lost Component Compensation), which explicitly reconstructs critical information lost during pruning to bridge the performance gap between pruned and dense models.
Specifically, our study in $ reveals that important information removed by pruning can be captured attention head activations, and reintroducing components this lost information can substantially restore the performance of the pruned model. However, this insight also presents challenges, such as determining which attention heads to select and how estimate component vectors that encode critical information for compensation. To overcome these challenges, RestoreLCC incorporates two main mechanisms: (1) contrastive probing. general approach that leverages activation editing to construct contrastive sample pairs and probes a subset of attention heads critical for performance recovery; and (2) lost component compensation (LCC), which retrieves pruning- induund lost information from these key heads. This information is decomposed into components, represented as vectors that capture the lost information directions. We optimize their magnitudes to enable targeted restoration along these directions, aggregate them into single informative component, and finally inject into the pruned model to recover performance. Unlike existing PEFT methods that restore pruned LLMs in an unguided manner, RestoreLCC explicitly compensates for key components lost during pruning, offering a targeted and effective restoration strategy.
We empirically evaluate RestoreL against other performance restoration methods across all three types of pruned models (e.g., structured, structuruc and unstructured), on both general recovery and task-specifie settings across wide range of LLMs of different sizes. Our results show that RestoreLCC significantly improves pruned model performance under general recovery settings while maintaining similar inference speed and sparsity ratios, outperforming existing PEFT methods. Furthermore, under task-specific recovery settings, Restorel ,CC successfully recovers task-specific information and enables higher pruning ratios, where other PEFT methods often fail.

---

information loss reflected in attention activations, and that selectively restoring key components can significantly recover model performance ($ 3): (2) We propose Restorel Sestorel.CC, a method that learns the magnitudes of important component directions lost during pruning and reintroduces them to restore pruned models effectively ($ : (3) Extensive experiments across various L.LSs and pruning schemes demonstrate
Contributions. Our main contributions are: (1) We observe that
that RestoreLCC consistently outperforms existing restoration baselines 5
Related work
LLM Pruning. Pruning reduces model size and speeds up inference by removing less important weights. Unstructured pruning removes individual weights irrespective of position, as in SparseGPT 77. which uses approximate Hessian-based reconstruction, and Wanda 4. which ranks weights by magnitude and activation norms. DSOT [17] and ALPS [88] further refine sparsity via optimization. Semi-structured pruning enforces patterns like N:M sparsity [9]. and many unstructured methods adapt to this format 77 Structured pruning removes entire blocks (e.g.. rows or columns) to
enhance hardware efficiency. Recent methods include LLM--runnr [3], Compresso 20 LoRAPrune
21. and SlimGPT 6)
Performance Restoration and PEFT. Pruning methods such LLM--rrnner [3] LoRAPrune 21
SparseGPT (7). Wanda [4], and SlimGPT 66] consistently benefit from a performance restoration
step, typically involving fine-tuning on language modeling (e.g., C4 ( WikiText (9) or instruction- tuning datasets (e.g., Alpaca [10). While full fine-tuning (FT) is effective, it remains computationally expensive, often requiring days on large GPU clusters. Parameter-efficient fine-tuning (PEFT) offers
cheaper alternative by updating a small subset of parameters. PEFT techniques include adapter-
based [22] |3 (24], prompt-basee [25] 26). LoRA-based (e.g., LoRA [T] AFLoRA (27). VERA
[1]. DoRA [13 ). and representation-enge eeceenattion-nninnering engineering methods (c.g., RED 14. ReFT [TS], LoFit (16]).
Among them, LoRA is the most widely adopted in modern pruning pipelines. In addition to PEFT
methods, FLAP (28] proposes a bias compensation strategy to reduce the pruning loss. EoRA [29
provides tne-tuning-free approach to recover pruned models by searching low-rank spaces in task-specific eigenspace to minimize compression loss.
3 Insight: injecting pruned components back effectively restores models
Preliminaries. We begin by clarifying several key terms and the scope of this study. Current LLM pruning methods primarily target the weight matrices of both attention and feed-forward (FFN) modules. For example, Wanda removes 50% of the weights in each matrix within both the attention and FFN modules (setting them to zero), resulting in an overall sparsity of 50%. Accordingly, in our work, all weight matrices within the LLM are pruned, and we focus on restoring the pruned model by compensating through attention heads. An activation refers to the output of a specific module within the Transformer architecture. A pruned activation is the output produced by a pruned module. Although its dimensionality remains identical that of the original activation, typically contains less information because the underlying weight matrices have been pruned. Importantly, the activations themselves are not pruned -oneddolyylla the associated weight matrices are. Sparsity denotes the proportion of weight parameters that have been pruned.
Pruning inherently leads to information loss, which becomes more pronounced at higher pruning ratios. In this section, we demonstrate that the pruned activations contain critical information, including discriminative components essential for downstream NLP tasks. By reintroducing the components, the performance of the pruned model can be substantially restored.
Recent studies have shown that different attention heads specialize in distinct functions when per- forming NLP tasks 30 31 32). Motivated by this, we investigate how pruning-induce loss of head activations affects the model's 's retained functional capacity and performance. Let MultiHead denote the multi-head attention output in a Transformer block with H attention heads. The output can be expressed as:
MultiHead (z') concat (2t.0),... ..... (0.,,..(..-))))) ((.-I1)) wo,
where x! is the input to the 1-th layer, z((.)) is the output of the h-th head, and WO is a shared output projection matrix. Denote by 24
(L.h) the activations of the h-th head in the dense and pruned (1.h) and 2p
3

---

models, respectively, for the same input. The lost activation can be computed as Eq. 2 Our objective is to analyze whether (2(1,h) carries critical information that could aid in model recovery.
(1,b) 8z 2d -11A) (1.,))
(1,h)
(2)
Given N samples, we denote the activation loss matrix for samples as AZ
d. 02 :2 better characterize the structure of the lost information, we apply singular value decomposition (SVD)
E RNxd,, where d, is the dimensionality of the head activation. To
to (Z as shown in Eq. 3 which decomposes the activation loss matrix into a set of orthogonal latent components. This formulation enables us to identify the dominant directions of lost activation information and quantify their impact on each sample.
du
K oru
(Z UUAASLh
Σ
oi u
(1,b)
(1,b)
Σ
(L
(3)
For the output 2p (1.h) by
of pruned attention head, the lost principal components can be approximated
K
(1.))
(1.)) (1,))
where c(..)) Rd and a. (1.b) denotes the average of 0gu (L.h)
representing the mean projection (1,b)
coefficients across samples. Finally, the activation output of the pruned attention head 2p can be compensated and recovered by injecting the estimated lost components (1..)) back:
(1.A)
(1,h) cc.) zp
(5)
(LA)
(l.b) for simplicity. To evaluate the predictive
where we omit the sample index i in 2, and 2i, behavior of attention head's activation, follow the theories of LogitLnens (33) and prior work
on interpreting LLMs in embedding space [34]. Specifically, we project the activation into the embedding space and compute its prediction over the vocabulary space:
(L,N) P=c = LM Head ( ( (=(.))wo..a]
where LM Head denotes the LLM's prediction head, p(-) the layer normalization function, W is the output projection matrix in Wo corresponding to the h-th head, pzc
(L.)) RRV, V represents the model's vocabulary, and | is the vocabulary size. Following IOI (35] we compute the logit difference, defined as the logit assigned to the correct token minus the logit assigned to the wrong
token. This difference directly reflects the model's confidence and faithfulness in predicting the correct token over an incorrect alternative.
Empirical Study. To evaluate whether reintroducing lost components can aid in performance recovery. we conduct an empirical study using BoolQ (36), widely used commonsense reasoning dataset. We adopt LLaMA-7B [2] as the dense backbone model and apply Wanda [4]. state-of-the ohe art pruning method, to prune the model to 50% sparsity. We randomly sample 1,000 examples from BoolQ, where both the dense and pruned models are tasked with answering "yes" or "no" questions. We extract the attention head activations from the pruned model and reconstruct the compensated activations using the main components, as described in Eqs. 2-5. To quantify model confidence, we compute the logit difference À using:
(1.h) lyes] - p = p
(... (1h)) [no] or -
p (l.h) [yes].
We also report final accuracy, i.e., the predictions from the last layer, assess the effect of component compensation the model's output.

---

1.8
130.a
Attention Hoad Index
2.5
k=9
component
1.6-
112:a16
k-27
k-13
compoment K
2.0
1.4
112.a19
a27
1.5
Logit
1.2
11 9.a6
226
Logi
1.0
k-2
1.0
15.a20
111.a18
16.025
k=12
0.8
16.a6
110.
18.a10
110
110.a0
112.a24
113.a18
a14.a31
113.a19
0.0
130.a13
119.a6
126 a16
17.a26
120.133
10
15
20
25
30
Layer Index
Figure 2: Logit gain of different attention heads recovered with principal components.
Figure 3: Logit gain of attention heads recovered with different components.
Finding 1. Reintroducing lost components to selected pruned attention heads can significantly restore model performance. As shown in Figure [ we use the top-10 (K 10 in Eq. 4 lost components reconstruct the pruned activations. The left figure demonstrates that this compensation substantially recovers the pruned activations. Specifically, the logit difference is restored to level comparable to that of the original dense model. Furthermore, compensating these attention heads also leads to a notable improvement in the model's 's final output accuracy.
Finding 2. Different attention heads vary in importance and exhibit distinct recovery behaviors. We manually select and examine the outputs of several attention heads, along with their restoration using the top-10 10 lost components. To quantify the direct effect of compensation, logit gains as d' reintroducing the lost components. Figure shows the recovered logit gains for these heads. The results reveal that attention heads respond differently to the compensation process, indicating that not all can be effectively restored using the lost principal components.
calculate the Xrecovered Apruned which directly captures the improvement brought by
Finding 3. Discriminative information may reside in minor components rather than the principal ones. iillustrats the recovery performance using either the top principal component or a selected minor component. Notably, since the coefficients minor components are extremely small, scale them by factor of 1000 for visualization and evaluation. Surprisingly, incorporating certain minor components can lead to substantially better restoration performance compared to using the leading principal component.
Extension to FFN. Each Transformer layer comprises a multi-head attention (MHA) module and a feed-forward network (FFN) module. In the above analysis, we focus on compensating for the MHA rather than the FFN. To further justify this design choice, that is, using MHA instead FFN, we provide additional theoretical and empirical analyses in Appendix [A]
To summarize, the above analysis highlights not only the strong potential of leveraging lost compo- nents for performance restoration, but also several key challenges: (1) How to select pruned attention heads that are important for recovery? (2) How to determine the positions and coefficients of key components? and (3) How to apply the findings to more general tasks beyond BoolQ?
4 RestoreL.CC CC: restoring pruned LLMs via lost component compensation
To address these challenges and enable universal performance restoration for pruned LLMs, propose RestoreLCC, shown as Figure 4 which consists two key mechanisms: (1) contrastive probing, general method that uses activation editing to create contrastive sample pairs and identify critical attention heads; and (2) lost component compensation (I.CC), which optimizes the magnitudes of directional components for the lost information. The optimized components then injected back into the model to restore performance.
4.1 Contrastive probing
Contrastive Sample Construction. We construct contrastive samples for general NL.P tasks to support activation editing and probing to localize important heads. Given dataset, either task-
5

---

Contrastive Probing
Daron Midiel
Phuned Model
m
O
Important
Restoration
Lost Component Compensation
Hewd
Heed
Claarned
pii
AZ:
P2
magh
for Risse
...
V4
P4P
Figure 4: Overview of the Restorel.CC framework. It integrates two key modules, (1) contrastive probing and (2) lost component compensation (LCC), to restore the performance pruned LLMs.
specific (c.g., BoolQ) or general (c.g., the Alpaca dataset), every sample is organized as (g. r+), where 9 is the question and r+ is the correct (positive) response. All responses collected into set Iro vo.,.. -- We use a sentence encoder, such as MiniLM-L6 [37], to encode all responses. For each sample, similarity as the negative response r- This process converts each sample into a contrastive tuple (g, r+, rrrrrr)), ). Note that this method is universal and can be applied to any dataset. We provide examples of constructed samples from the BoolQ and Alpaca datasets Appendix B
select the most similar response (excluding the correct one) based cosine
Activation Editing. Recent studies suggest that high-dimensional activations in LLMs are ap- proximately orthogonal with high probability (38). An activation can guided toward a de- sired generation space by adding steering vector or task-specific function vector [39 32).
For pruned LLMs, we assume the lost principal components can be reintroduced to recover the activation and restore the correct response. Formally: recovered question activation s pruned question activation + lost important component. Based on this, the recovered activation for question is defined
= 2/ c0,
where we omit the layer and head indices (1, h) for simplicity. Here, q indicates the activation is taken from the last token of the question, and the other symbols follow the notation in Eq. 5
Attention Head Probing. Ideally, the recovered activation zg should contain sufficient information generate the correct response. It should be consistent with the activation of the complete and correct sample, denoted as 2 - which obtained from the last token of the full sequence (including both the question q and the positive response r+) the original dense model. Additionally, it should be contrastive with respect to the negative sequence activation 2 However, some attention heads may inactive, and certain components may contribute little performance restoration. In such cases, the recovered activation = may not adequately distinguish between =X+*** and 20+"".
In this way, identifying important attention heads can be formulated as a natural language inference (NLI) task. If an attention head is important and the corresponding component is useful, then the recovered activation =ę should entail the correct sequence activation 2 and contradict the negative sequence activation 2/ 20t Specifically, activation pairs are constructed as m [2g 2/ with label 1 (entailment) and m [=g, = =gtr" with label (contradiction). composed of a linear layer followed by a sigmoid activation, is trained to assess the discriminative power of each recovered head activation. The importance of attention heads is then ranked based on the accuracy of their corresponding probing classifiers.
probing classifier,
4.2 Lost component compensation
We now have list of important attention heads along with their corresponding lost components ;; based on Eq. 3 The goal is to use these components to approximate the missing information and recover the performance of pruned LLMs.

---

Lost Component Estimation. As observed in $ the usefulness of a component does not necessarily correlate with its coefficient. Minor components may also earry critical information and enable more effective recovery of pruned L..Ms. To address this, we consider all possible components. Since the vectors vi are orthogonal and unit-normalized, we treat each o as potential direction lost information. We keep these directions fixed and learn a scalar magnitude for each, representing its importance. Formally, for an attention head, we model its lost component containing important information as:
Clearned
Brus + b.
(8)
Here, V is obtained from Eq. 3 and remains fixed during training, while 3, is trainable sealar indicating the importance of each direction. The term b E Rh is trainable bias vector for the attention head, acting as a hedging vector to provide flexibility in cases where important information lies outside the span of the predefined directions.
Component Compensation. For each pruned attention head, its output activation is recovered as &p 1-=2 2p + Cleamed- It is worth noting that the final learned component Cleaned is constant bias vector, capturing important information that was lost for all samples as a result of pruning. The recovered activation 7p is then passed to the subsequent computations, such as those described in Eq. allowing the model to better approximate the behavior of the original dense LLM.
Extension of RestoreLCC to FFNs. It is worth noting that RestoreLCC can be seamlessly extended to FFN modules. Our objective the attention and FFN modules-hav have been pruned. We provide further theoretical and empirical analyses Appendix A to justify why we apply RestoreL.CC attention heads instead of FFNs.
restore pruned models in which all weight matrices both in
4.3 Overhead analysis of sparsity and inference speed
LLM pruning aims to produce sparse model, while performance restoration seeks close the performance gap between the pruned and dense models. It is important to ensure that performance restoration improves the pruned model's accuracy without compromising its sparsity or inference efficiency.
Sparsity Analysis. Each compensated attention head introduces learned vector Cleamed Rdh, In the worst case, where all heads in layer are compensated, the total number of additional parameters introduced across the projection matrices (q-proj, k k_proj. v_proj. o_proj) within the multi-head attention mechanism is:
Parameter Overhead = 2d 1
4d²
2d,
where d; is the hidden size of the layer and d, = d//H. Since di is typically larger than 1000, the increase in parameters is less than 0.05%, which has minimal impact sparsity.
Inference Speed. maintain inference efficiency, the learned vector Oeamnd absorbed as the constant bias vector in the multi-head attention block. This modification introduces almost no additional computation during inference and preserves the speed of the pruned model. Therefore, RestoreLCC effectively preserves the pruned model's sparsity and inference speed while recovering its performance. Additional empirical evidence is provided Appendix D
5 Experiments
5.1 Experimental settings
Metrics. To evaluate the restoration effectiveness of RestoreLCC, we experiment with representative pruned LLMs from diverse pruning strategies. Specifically, we use Wanda [4] for unstructured pruning, SparserGPT 7 for semi-structured pruning, and SlimGPT (6] for structured pruning. The pruning ratio is set to 50% for unstructured and semi-structured methods, and 20% for structured pruning, with C4 [8] as the calibration dataset. Following these works, we assess perplexity (PPL.) of language modeling the held-out WikiText [9] and accuracy of several commonsense reasoning
7

---

benchmarks, including BoolQ (36,, HellaSwag [4]] WinoGrande 141,
eRC-easy 42), ARC-
challenge 422. RTE [4], and OpenBookQA 44,, all evaluated using the Im-eval- harness framework
45
We consider two restoration settings:
General Recovery. It is widely adopted in previous work. We follow SlimGPT and use the Alpaca instruction dataset [10] for tuning. Evaluations are performed in a zero-shot setting across language modeling and commonsense reasoning tasks.
Task-Specific Recovery. While prior work focuses on general recovery, we highlight the impor- tance of task -ask-specific restoration, as pruned LLMs should also perform effectively when deployed in specialized domains. To evaluate this, we increase the sparsity ratio and restore pruned models using 100 training examples (e.g. 100 BoolQ samples) from the target task (e.g.. BoolQ).
Models and Implementations. We conduct our main experiments using LLaMA-7B/13B models 2] To evaluate the universality and scalability of RestoreL.CC, we provide additional results LLaMA-3-8B (2).
LLMs of varying sizes and families, including LLaMA-30B, LLaMA-2-7B//
Vicuna-7b-yl 1.5 [46). Tulu-2-7B [47], Owen-3-8B/14B [48), and CrepSeck-RI-Qwen3--e
44 in
Appendix
K For RestoreLCC, the number of components K is set to I to identify important attention
heads, and select 10%-25% of them for recovery. Additional implementation details are also
provided in Appendix K
Baselines. To ensure comprehensive evaluation, compare RestoreLCC with comprehensive baselines methods as follows:
LoRA 11 which fine-tunes low-rank adapters to update the parameters of pruned LLMs. It is
widely used for performance restoration in recent SOTA pruning methods 5 which enhances LoRA by decomposing low-rank adaptation.
DORA [13]
FLAP 28
which recovers the pruned model by bias compensation.
EoRA 29
which searches low-rank spaces in the eigenspace minimize compression loss.
LoFiT 16
which is a SOTA representation engineering method that intervenes attention activations to adapt LLMs to downstream tasks.
5.2
Performance of general recovery
Table 1 reports results across three pruning regimes LLaMA-7B. Under unstructured pruning at 50% sparsity, RestoreLCC achieves 58.83% mean accuracy, outperforming the best baseline, LoFiT (56.82%), by +2.01 %, and reduces PPL to 6.93. In the semi-structured pruning setting, RestoreLCC improves mean accuracy to 55.00%, yielding a +2.65% % gain over the best baseline DoRA (52.35%), while also lowering PPL from 9.16 to 8.99. Under structured pruning at 20% sparsity, RestoreLCC reaches 59.76% accuracy, improving over DoRA (58.51%) by +1.25% Across all settings, RestoreL.CC consistently outperforms prior recovery methods in both accuracy and perplexity, recovering performance close to the dense model (59.99%)
Table 1: Performance of general recovery on zero-shot language modeling (PPL.) and commonsense reasoning tasks (accuracy) using LLaMA-7B. Best scores are bolded.
Method
PPL
BoolQ
RTE:
HellaSwag
WinoGrunde
ARC=e ?
ARC-
OBQA
Meun
Donse Model
5.68
75.02
66.79
56.95
60.93
75.29
41.72
34.30
59.99
Unstractured Prining 30% Sparsity
Recovery)
51.86
71.68
DoRA
7.11
71.83
62:45
$4.09
71.13
30.76
36.
FoRA
7.14
74.16
61129
$1.27
68.27
70.96
37.63
28,60
55 88
LoFiT
7.35
72.29
61.26
$1.79
65.19
69.99
35.
32,50
56.82
ResioLLCC (Ours)
6.93
72.84
69.68
56.34
63.98
71.80
40.96
34,20
58.83 (+2.011
Somi SPaneGPT (BBase Modkl Recowrry)
(N.M-2:4) 50% Sparsity
43.12
23,50
48
63.77
DoRA
9.16
71.41
59.21
89.16
62.04
65.99
33.45
25.20
52.35
FLAP
10:57
68.81
54.15
44
64.40
63.97
30.05
24.00
49.98
BoRA
9.87
71.68
59.93
44.65
64.33
63.72
30.38
23.30
51.21
LoFIT
10.02
70124
58.48
49.41
63.06
64.06
32.51
27.00
52.11
RestareLCC (Ours)
63.90
29.00
55.00
Struttored Proning Sporsity
SlintePT (Base Recovery)
75.99
53.73
72.14
LoRA
7.06
75.48
65.70
55:25
66.27
72.01
40.36
32.60
38.45
DoRA
7.54
76.79
64.62
55.
66.77
72.35
40.61
33.00
58.51
LoFIT
7,86
75.00
63.54
56.26
67.80
71.34
40.53
33,00
59 58.42
ResioLLCCC LCC (Ours)
7.53
76.48
68.59
$7.05
69.46
72.01
40.53
34,29
39.76 (+1.25)
We found that 100 samples sufficient for effective restoration.

---

Table 2: General recovery on zero-shot language modeling (PPL) and common- sense reasoning tasks (mean accuracy)
Table 3: Performance of task-specific recovery on indi-
vidual commonsense reasoning tasks with LLaMA-7B.
Method
BealQ
KTE
ARC-et
ARC-et
Mean
with LLaMA-
Dense Model
75.02
66.79
75.29
41.72
64.71
Method
PPL
Acc.
Unstrictured Pruning at 60%
Spursity
Wanda
68.87
59.21
62.67
30.29
55.26
Dense Model
5.09
62.57
LoRA
69.14
67.51
62.08
33,36
58,02
Wanda
Unstructured Proning 50%
6.15
Sparsity
59.43
RestoreLCC (Ours)
DoRA
75.32
67.51
67,87
70.40
61,03
63.89
37,46
33.19
61.77 (+3.56)
LORA
6.08
61.13
DoRA
6.11
61.42
Semi-Stractured Pruning (N:M-
2:44 ar 50% Sporsity
LoFTT
6.29
60.98
SpuneGPP
54.51
60.90
30.20
53.77
RestoreLCC (Ours)
6,08
62.46 (+1.04)
DoRA
LORA
69.72
68.95
64.27
35,67
59.65 65
Somi Struetured Pruming (N:M=2:4) 50% Sporsity
LoFIT
70.92
68.95
66.58
36,26
60.68
SparseGPT
9.08
53.32
RCstoneLCC (Ouns
77.74
69,31
69.32
38,57
63.74 (+3.06)
LoRA
7.75
58.22
Structured Prawing Spursity
DoRA
55.18
SlinGPP
69.57
64.26
55.98
52,94
69
LOFIT
8,10
55.94
LORA
69.02
66.06
56.31
58.19
56.15
RestoreLLCC (Ours)
7.61
57.67 (+1.73)
DoRA
69.20
65.34
56.10
33.19
55.96
LOFT
64.26
Structured Pruming of 20% Sparsity
Restorel (Ouss)
70.95
68.23
62.08
36.95
59.55 (43.40)
SlimGPT
60.21
LORA
6.29
61.12
DoRA
6.10
61,96
LoFIT
6.51
61,22
Restorel CC (Ours)
6:21
63.41 (+11.5)
Table
reports the recovery performance of LLaMA-13B; detailed results are provided in Table Under unstructured pruning, RestoreLCC achieves 62.46% mean accuracy with PPL of 6.08, corresponding to a a+1.a +1.04% % gain over the best baseline. In the semi-structured setting, RestoreLCC outperforms LoFiT by +1.73% For structured pruning, RestoreLCC reaches 63.41 % accuracy, surpassing DoRA by +1.45% These results verify that RestoreLCC generalizes well to larger models. and consistently provides superior recovery across all pruning types.
17
5.3 Task-specific recovery
Table Jevaluates spec-fpe recovery on LLaMA-7B under three sparsity settings. For unstructured pruning at 60% sparsity, Restorel achieves mean accuracy of 61. 61.77%, improving over LoFiT (58.21%) by +3.56% %. In the semi-structured pruning. RestoreL.CC achieves 63.74% mean accuracy, outperforming LoFiT by +3.06%. %. For structured pruning at 40% sparsity, RestoreLCC again yields the best mean accuracy (59.55%) outperforming LoRA by +3.4%. In addition, we increase the prun- ing ratio by a challenging 10-20%. RestoreLCC successfully recovers performance, demonstrating its effectiveness in enabling higher sparsity ratios for LLM pruning.
5.4 Ablation study
Table 4: Ablation study results on a 50% pruned LLaMA-7B model using Wanda.
In this subsection, we analyze the impact of different mech- anisms in RestoreLCC.
Method
Mean Accuracy
Effects of Contrastive Probing on Identifying Impor- tant Attention Heads. We apply contrastive probing to identify attention heads critical for performance recov- ery. comparee performance using heads selected by contrastive probing (RestoreLCC) versus randomly se- lected heads (w/o probing). The 1.26% degradation in the latter case confirms that contrastive probing effectively identifies heads essential for recovery. We also conduct experiments with two alternative head-selection strategies: (1) MSE-sclected heads: selecting heads with the smallest MSE between the outputs of the dense and pruned models. (2) KL--selected heads: selecting heads with the smallest KI. divergence. It can be observed that our probing-base based selection consistently identifies important heads and is more effective than other metric-based approaches.
ResoreL.CC
58.83
w/o probing
57.57 (-1.26)
MSE-sclected
58.14 (-0.69)
KL-selected w/o dr w/o b
57.92 (-0.91)
Biv,
57.13 (-1.70)
58.26 (-0.57)
Effects of dn 3:0.. We estimate the lost information from pruning using Σd= 3,0,, which captures both direction and magnitude. To evaluate its impact, we remove this term and tune Glarne using only the bias vector in Eq. g This results 1.70% performance drop, underscoring the importance of recovering both directional and magnitude information in pruned models.

---

Effects of Bias Vector. We introduce a bias vector in Eq. 8 to serve as a hedging term, allowing flexibility when important information lies outside the span predefined directions. Table reports the results of removing the bias vector (w/o b), showing 0.57% % performance drop, which confirms its role in optimizing the final learned component.
5.5 Interpreting the learned component
Heads
Top-5 Decoded Tokens
130a13
no: 'no", _es, 'No', yes'
To examine the information encoded the learned com-
129a10
yest, young' Young', 'noo, yes'
126a16
yes' yes: lof'' 'no', 'YES'
ponent Cleamed and its role in performance recovery, we illustrate task-specific recovery on BoolQ (binary yes/no answering). Following $ $3 we use LogitLens to project Geamed into the embedding space and show the top-5 de- coded tokens in Figure The results suggest that Cleamed captures task-relevant signals, such as
Figure 5: Visualization of learned com- ponents for different attention heads.
indicating "yes" or "no" answers for BoolQ examples.
5.6 Further analysis
We provide additional analysis to verify the universality and efficiency of RestoreL as follows:
We evaluate Restorel.CCC under varying numbers of attention heads and components Eq. Results demonstrate its stability.
Hyperparameter Sensitivity (Appendix C
Overhead and Efficiency Analysis (S 4.3 and Appendix D): We compare the trainable parameters,
inference speed and overhead sensitivity of RestoreLCC with other baselines. The results verify that RestoreLCC restores pruned LLMs without compromising sparsity inference efficiency. Parameter Visualization (Appendix E): We visualize the trained directions, magnitudes, and biases, offering deeper insight into the internal mechanisms of RestoreLCC.
Comparison with hall-Parameter Tuning (Appendix(F)) We present experimental results that compare RestoreI.CC with full-pant pala-mrrrettrr tuning (FT).
Pross-Task Portability and Generalization of Probing (Appendix Gh: We discuss the cross-task
portability and generalization of the contrastive probing module in Appendix G
Efficiency at Scale (Appendix HD: We demonstrate the efficiency of Restorel on a larger L.MM (LLaMA-70B) and evaluate its latency.
Compatibility with Quantized Models. (Appendix H):
We conducted experiments with 4-bit quantization the pruned model and verify RestoreLCC's compatibility with heavily quantized
models.
Effect of Probing Samples. (Appendix) DD: We study the effect of the number of probing samples on RestoreLCC.
Evaluation on More LLMS (Appendix K: Experiments on LLMs including LLLMAA30B,
I. LLAMA-2-7B/133, LLLMM-3-88 (2). Vicuna-7b-vl.55 [46). Tulu-2-7B [47). Qwen-3-8B/14B 48, and DeepSeek-R1-Q R1-e--33--- further validate the universality and scalability of Re-
storeI.CC.
Conclusion
In this work, we propose RestoreLCC, targeted strategy for restoring the performance of pruned LLMs without compromising their sparsity or inference speed. RestoreLCC integrates two key mechanisms: (1) contrastive probing, which leverages activation editing to probe critical attention heads, and (2) lost component compensation (LCCC, which estimates and restores the lost directional information in pruned heads. Extensive experiments across diverse pruning settings and LLMs demonstrate the effectiveness of RestoreLCC in recovering model performance.
Limitation. We assign learnable magnitudes all components compensate for pruned attention heads, allowing less important ones to be down- weighted. However, this still may cause overfitting of unimportant components. future work, we plan to pre-select relevant components before learning their magnitudes, reducing both overfitting and the number of trainable parameters. In addition, Moore and Chaudhuri [50) utilize activation noise to probe network structure and identify redundant neurons. They also discuss a novel way of leveraging activation noise for neuron identification. We believe activation noise could similarly be used to enhance contrastive probing, and we plan to explore this direction in future work.
10

---

Acknowledgments
We extend heartfelt gratitude to the reviewers for their insightful and constructive feedback. This research was supported by the Home Team Science and Technology Agency (HTX), Singapore under the NTU-HTX collaboration project: Parsimonious Domain Specific Large Language Model Enabled Multimodality Sensemaking. We express our sincere appreciation to HTX for their continued support and collaboration.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023.
[3] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint wrXiv:2412.55111 2024.
[4] Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. The Twelfth International Conference on Learning Representations, 2023.
[5] Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-prumer On the structural pruning of large language models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 21702-21720. Curran Associates, Inc., 2023.
[6] Gui Ling, Ziyang Wang, Yuliang Yan, and Qingwen Liu. Slimgpt: Layer-wise structured pruning for large language models. In A. Globerson, Mackey, Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 107112-107137. Curran Associates, Inc., 2024.
[7] Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in one-shot. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10323-10337. PMLR, 23-29 Jul 2023.
[8] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1667 2020.
[9] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017.
[10] Rohan Taori, Ishaan Guirajani. Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An inslroction-follooinng Ilama model. btttp:///gh thub. com/ tttr/tatsu-lass waw/llaaaaaaatoaa ord_ aapac alpaca
2023.
[11] Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[12] Dawid Jan Kopiezko, Tijmen Blankevoort, and Yuki M Asano. Vera: Vector-based random matrix adaptation. In The Twelfth International Conference on Learning Representations, 2024.
[13] Shih- Shih-Yang Liu, Chien-Yi Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: weight-decompposed low-rank adaptation. In Proceedings of the s1st International Conference on Machine Learning, ICMI'24. JML..org, 2024.
11

---

[14] Muling Wu, Wenhao Liu, Xiaohua Wang. Tianlong Li, Changze Lv, Zixuan Ling, Zhu JianHao, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Advancing parameter efficiency in fine- tuning via representation editing. In Lun-Wet Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 13445-13464, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[15] Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. Reft: Representation finetuning for language models. In A. Globerson, I.. Mackey, D. Belgrave, Fan, U. Paquet, Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 19908-63962. Curran Associates, Inc., 2024.
[16] Fangcong Yin, Xi Ye, and Greg Durrett. Lofit: Localized fine-tuning on Ilm representations. In
A. Globerson, I.. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 9474-9506. Curran Associates, Inc., 2024.
[17] Yuxin Zhang, Lirui Zhao, Mingbao Lin, Sun Yunyun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, and Rongrong Ji. Dynamic sparse no training: Training-free fine-tuning for sparse llms. The Twelfth International Conference on Learning Representations, 2024.
[18] Xiang Meng. Kayhan Behdin, Haoyue Wang, and Rahul Mazumder. Alps: Improved optimiza- tion for highly sparse one-shot pruning for large language models. In The Thirty-eights Annual Conference Neural Information Processing Systems, 2024.
[19] Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. arXiv preprint 1eXiv:2/04.08.3 2021.
[20] Song Guo, Jiahang Xu, Li Lyna Zhang, and Mao Yang. Compresso: Structured pruning with col- laborative prompting learns compact large language models, arXiv preprint arXiv:2310.05015. 2023.
[21] Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, and Bohan Zhuang. LoRAPrune: Structured pruning meets low-rank parameter-efffiien fine-tuning, In Lun-Wer Mun-Wei Ku, Andre Martins. and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACI. 2024, pages 3013-3026, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[22] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov. editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR, 09-15 Jun 2019.
[23] Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. efrfiettr-efffiient multi-task fine-tuning for transformers via shared hypernetworks. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the Ith International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 565-576, Online, August 2021. Association for Computational Linguistics.
[24] Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: efficient low-rank hypercomplex adapter layers. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS '21, Red Hook, NY, USA, 2021. Curran Associates Inc.
[25] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficine prompt tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
12

---

[26] Anastasiaa Razdaibiedina, Yuning Mao, Madian Khabsa, Mike Lewis, Rui Hou, Jimmy Ba, and Amjad Almahairi. Residual prompt tuning: improving prompt tuning with residual reparam- eterization. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACI. 2023, pages 6740-6757, Toronto, Canada, July 2023. Association for Computational Linguistics.
[27] Zeyu Liu, Souvik Kundu, Anni Li, Junrui Wan, Lianghao Jiang, and Peter Beerel. AFLoRA: Adaptive freezing of low rank adaptation in parameter efficient fine-tuning of large models. In Lun-Wa Wei Ku, Andre Martins, and Vivek Srikumar, editors. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 2: Short Papers), pages 161-167, Bangkok, Thailand, August 2024. Association for Computational Linguisties.
[28] Yongqi An. Xu Zhao, Tao Yu, Ming Tang. and Jinqiao Wang. Fluctuation-baseee adaptive structured pruning for large language models. In Proceedings of the Seirty-Eighthh AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial AAAI Press, 2024.
Intelligence,
[29] Shih-Yang Liu, Huck Yang, Chein-Yi Yi Wang. Nai Chit Fung, Hongxu Yin, Charbel Sakr, Saurav Muralidharan, Kwang- Ting Cheng, Jan Kautz, Yu-Chianp Frank Wang, al. Eora: Training- free compensation for compressed Ilm with eigenspace low-rank approximation. arXiv e-prints, pages 2410, 2024.
[30] Wei Zhang, Chaoqun Wan, Yonggang Zhang, Yiu-Ming Cheung, Xinmei Tian, Xu Shen, and Jieping Ye. Interpreting and improving large language models in arithmetic calculation. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver. Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 59932-59950. PMLR, 21-27 Jul 2024.
[31] Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for 1lms. In 12th International Conference on Learning Representations, ICLR 2024, 2024.
[32] Xinyu Ma, Yifeng Xu, Yang Lin, Tianlong Wang, Xu Chu, Xin Gao, Junfeng Zhao, and Yasha Wang. Dressing up Nl:: Efficient stylized guestion-answering via style subspace editing. In The Thirteenth International Conference on Learning Representationss 2025.
[33]
nostalgebraist. Interpreting gpt: the logit lens, 2020. Accessed: 2025-05-03.
[34] Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embedding space. In Anna Rogers, Jordan Boad-Graber, and Naoaki Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages
Toronto, Canada, July 2023. Association for Computational Linguistics.
[35] Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. In The Eleventh International Conference on Learning Representations, 2023.
[36] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924-2936, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics.
[37] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self- attention distillation for task-agnostie compression of pre-trained transformers. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances Neural Information Processing Systems, volume 33, pages 5776-5788. Curran Associates, Inc., 2020.
13

---

[38] Zhichao Wang and Yizhe Zhu. Overparameterized random feature regression with nearly orthogonal data. In Francisco Ruiz, Jennifer Dy, and Jan- Mil-Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 8463-8483 8493. PMLR, 25-27 Apr 2023.
[39] Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. In The Tweifth International Conference on Learning Representations, 2024.
[40] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Mårquez, edi- tors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791 4791-4800, Florence, Italy, July 2019. Association for Computational Linguistics.
[41] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99-106, August 2021.
[42] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.055: 2018.
[43] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint
2018.
[44] Todor Mihaylov. Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun't ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381-2391, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
[45]
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf. Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024.
[46]
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang. Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. Judging Ilm-as-a-judge with mt-bench and chatbot arena. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 46595-46623. Curran Associates, Inc., 2023.
[47] Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in changing climate:
Enhaneing Im adaptation with tulu arXiv preprint XrXiv:2311.10702 2023.
[48] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09 003888 2025.
[49] DeepSeek-AI. Deepseek-rl: Incentivizing reasoning capability in lmms via reinforcement learning, 2025.
[50] Eli Moore and Rishidev Chaudhuri. Using noise to probe recurrent neural network structure and prune synapses. In Proceedings of the 34th International Conference on Neural Information Processing Systems. NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc
[51] Wei Zhang, Chaoqun Wan, Yonggang Zhang, Yiu-ming Cheung, Xinmei Tian, Xu Shen, and Jieping Ye. Interpreting and improving large language models in arithmetic calculation. In Proceedings of the 41st International Conference on Machine Learning, ICMI.24. JMLR.org, 2024.
14

---

[52] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc.
[53] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Crani-Fran Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 5484-5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
15

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction provide an accurate summary of our contributions and the scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification:
We have provided assumptions and proof in section 3.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer sections and 5.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
We will submit our source code as an anonymized zip file the supplemental material.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
httpaic/guides//eeeuuum..ssseee coololoo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssicnyolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The paper does not include error bars because of limited computational resources that prevented extensive experimentation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see details of implementations section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:p/p////.rrr...ooo/oo//////a
Answer: [Yes]
Justification: We have read the NeurIPS Code of Ethics and ensure that our paper strictly adheres to these guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please see section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks as we only used open-sourced models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer:[Yes)
Justification:
All of these are properly credited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
21

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
22

---

A MHA Compensation versus FFN Compensation
Each Transformer layer comprises a mulli-heead attention (MHA) module and feed-forw forward network (FFN) module. In this study, we restore pruned LLMs via attention head compensation instead of FFN compensation, as justified by both theoretical analysis and empirical results.
Theoretical Analysis. Studies mechanistic interpretability show that attention heads specialize in distinct functions for different tasks, while FFNs mainly store knowledge and map inputs to outputs. For example, [51] finds that in arithmetic tasks, attention heads process key information and FFNs then produce the final answer. They also show that fine-tuning only important heads outperforms tuning all parameters (including FFNs). Similarly, [31] demonstrates that different heads play distinct roles, and [32] also supports this observation. By contrast, FFNs store factual knowledge [52) and refine output logits [53]. Hence, compensating attention heads is more appropriate than compensating
FFNs.
Moreover, attention heads provide finer information, whereas FFNs produce high-ding digh-dimensiona,, aggregated outputs. In LLaMA-7B, each head outputs 128 dimensions (32 heads per layer), while an FFN outputs 4096-dimensional vectors. Smaller head outputs allow finer analysis. Moreover, choosing heads can span multiple layers, while the same size in FFN covers only one layer.
Empirical Evidence. It is worth noting that our method can also be directly applied to the FFN modules. Accordingly, we applied RestoredLCC to the FFNs and report the best results in Table [ The results obtained with different numbers of FFN layers are presented in Table Compensating the attention modules achieves score of 58.83, whereas compensating only the FFNs yields 57.00.
Table 5: Performance (accuracy) of general recovery zero-shot commonsense reasoning tasks using LLaMA-7B pruned by Wanda at 50% sparsity.
Method
BoolQ
RTE!
HellaSwag
WinoGrande
ARC-e
ARC-et
OBQA 1 Mean
Recover atin.
72.84
69.68
56.34
65.98 67.17
71.80 70.96
40.96
34.20
58.83
Recover FFN.
69.42
63.54
$5.54
39.59
32.80
$7.00
Table 6: Average accuracy performance with different recovered FFN ratios.
FFN ratio
0.1
0.3
0.5
0.7
1.0
Mean
55.92
57.00
56.24
56.68
56.80
B Examples of Contrastive Samples
aigure/Jand Figure show constructed contrastive sample pairs for the Alpaca dataset and the BoolQ dataset, respectively.
23

---

Positive Sample
Negative Sample
Instruction: Describe traditional wedding ceremony you attended.
Instruction: Describe traditional
wedding
ceremony you attended.
Response: As an Al, do not have physical
Response: Weddings are celebration of love, commitment, and the start of a new life journey for two individuals. These ceremonies steeped in traditions that vary from culture culture, with each having its unique rituals and customs to make friends to witness and share in the union of two people in love...
presence and therefore do not attend events such wedding ceremonies. However, from understanding, traditional wedding ceremonies vary greatly
often
different cultures and religions. Can you please specify cultural religious context for the traditional wedding ceremony you
day special family and
would like me to describe?
Instruction: Come up product solve
Instruction: Come up product solve
a problem.
a problem.
Response: One common problem that many people face is forgetting to take their medication time. This can be due to busy schedules, forgetfulness simply being away from home. solve this problem, medication dispenser that reminds people when it's time to take their medication and...
Sure. Here is prototype of a mobile application for tracking medication compliance: MedicineTracker Mobile Application Prototype. Application Overview: MedicineTracker
Response:
mobile application designed help track their medication compliance. With this app. users can easily log......
could develop smart
Figure 6: Constructed contrastive sample pairs for the Alpaca dataset.
Positive Sample
Negative Sample
Ghost in the Shell Animation studio Production I.G has produced several different anime... Question: is ghost in the shell based on the anime?
Ghost the Shell -- Animation studio Production I.G has produced several different anime... Question: is ghost in the shell based on the anime? yes
Answer:
The Walking Dead (season The eighth season of The Walking Dead, American apocalyptic horror television series on AMC, premiered on October 22, 2017, and concluded on April 15...
The Walking Dead (season The eighth season The Walking Dead, an American post- television series AMC, premiered on October 22, 2017, and concluded on April 15...
apocalyptic
Question: is there gonna be season of the walking dead?
Question: is there gonna be season 8 of the walking dead?
Answer: yes
Answer:
Figure 7: Constructed contrastive sample pairs for the BoolQ dataset.
24

---

C Hyperparameter Sensitivity
58.5
Accuracy (%)
58.0
57.5
57.0
Accuracy Best
56.5
0.0
0.2
0.4
0.6
0.8
1.0
Percentage of Heads
Figure 8: Average accuracy on commonsense reasoning tasks for different numbers of attention heads.
Number of Attention Heads. Figure 8 shows the average accuracy on commonsense reasoning tasks for different numbers of attention heads. We observe that using more than 5% of attention heads allows RestoreL.CC achieve around 58% mean accuracy. Based the results, we recommend using 5%-25% attention heads, striking a balance between strong performance and parameter effi- ciency.
Table 7: Experimental results differ-
ent numbers of components in Eq. 3
Mean Accuracy
1
58.83
58.53
10
58.51
Number of Components. The number of components
K Eq. is used to identify important attention heads. Since the component coefficients from SVD) are ordered by magnitude, and the top components As result, RestoreL,CC achieves
dominate, the selected heads remain consistent for K similar performance across these settings, shown in Table Additionally, RestoreLCCC is highly
stable across different K values, with only around 0.3% difference between K = 1, K = 3, and
K 10. Based on these observations, recommend using K < 3, which already captures the most informative components.
D Overhead Analysis
Table 8: Comparison of trainable parameters and inference speed across different restoration methods. The base model is LLaMA-7B at 50% sparsity pruned by Wanda. The inference delay is computed by comparing the inference speed of the restored model to that of the original pruned model. A value of 1.0x indicates that the inference is performed at the same speed as the original pruned model.
Method
Number of Parameters
Inference Delay
Pruned Model (Base Speed)
1.0x
LoRA (w/ mask)
0.0600%
1.0x
LoRA (w/o mask)
0.0600%
1.1x
LoFiT
0.0005% %
1.0x
RestoreLCC
0.0010%
1.0x
As discussed 4.3 RestoreLCC introduces negligible additional parameters that would impact the sparsity or inference speed of the pruned model. The representative engineering baseline, LoFiT, maintains a similar number of trainable parameters and can be integrated into the pruned LLM the same manner as RestoreL.CC. thus incurring no overhead. In contrast, LoRA and DoRA A have two implementation options: (1) training with masking constraint, where the low-rank matrices and B in LoRA are constrained to have zeros in the same pruned positions. This allows the product B.A
25

---

Table 9: Comparison of training time and GPU memory usage on LLaMA-7B 50% sparsity pruned by Wanda.
Method
Time
Memory
LoRA
4h20min
61GB
DoRA
6h08min
71GB
LoFiT
8h1 Imin
65GB
RestoreL.CC
4h13min
65GB
Table 10: GPU memory usage and training time with different head ratios.
Head Ratio
0.1
0.2
0.3
0.5
1.0
GPU Memory
65GB
65GB
65GB
66GB
68GB
Time
3h40min
4h13min
4h40min
5h30min
7h42min
be merged into the pruned LLM without affecting its sparsity or inference speed; and (2) direct training without masking, which treats A and B as external adapters. In this case, merging them would reintroduce non-zero values in pruned locations. compromising sparsity.
Furthermore, the overhead introduced by the intermediate activations (v:) is negligible and can be ignored. These activations remain fixed (frozen) during training. Taking LLaMA-7B
example, each attention head has 128 dimensions; by the property of singular value decomposition (SVD), this corresponds at most 128 components, resulting in 128 128 16,384 parameters per head. Even in the extreme case where all 1,024 heads are used, this adds only 1,024 x 16,384 z 16.8M parameters-merely about 0.24% of a 7B model. Moreover, shown in Figure 8 only 10-30% of the heads are required for recovery, making the actual overhead substantially smaller.
Table 8 presents the number of trainable parameters and inference speed relative to the original pruned model. RestoreLCC requires significantly fewer parameters than LoRA and does not degrade inference speed. These empirical results support earlier analysis: RestoreLCC effectively restores pruned LLMs without introducing overhead in terms sparsity or efficiency.
Table 9 compares the training time and GPU memory usage on same computing conditions across different methods for recovering LLaMA-7B (1 H100 GPU, Norch.bfloat precision, max max-length=512, batch-size-8, Al Alpaca Dataset). For training time, the methods rank as follows: Restorel < LoRA < DoRA < LoFiT. For GPU memory usage, the order is LoRA < RestoreLCC = LoFiT < DoRA. Compared with DoRA and LoFiT, RestoreLCC exhibits the lowest overall overhead while providing superior performance.
Overhead Sensitivity to the Number of Trainable Parameters. The number of trainable parameters in RestoreLCC depends on the proportion of attention heads selected for tuning. Table 10 reports GPU memory usage and training time for different head ratios. The results show that: (i) GPU memory consumption remains nearly constant as the number of trainable parameters increases and consistently lower than that of DoRA (71 GB); (ii) the training time is generally shorter than both DoRA and LoFiT. Importantly, increasing the number of trainable parameters does not necessarily improve performance, indicating that Restorel.CCC is not sensitive to this factor. This observation also motivates our design choice to first contrastively identify the most informative heads.
Based on the theoretical and empirical analysis, the overhead of our proposed RestoreLCC is smaller than advanced PEFT methods while achieving much better performance.
E
Analysis of Trained Components
Figure 9 and Figure 10 illustrate the training results for both magnitudes and directions. Figure 9 shows that the learned magnitudes diverge notably from the original SVD coefficients, which ranked from largest smallest, supporting Finding minor components can retain valuable pruned information. Figure [0 visualizes the directions (v;), the trained bias vector (6), the combined
direction Edh 1 B,0z), and the final learned component ( Geanned) The final component benefits
26

---

T-SNE Projection Trained Components
Regntude
0.00
-0.02
Directions
-0.04
Combined Direction
20
40
60
80
100
120
c_learned
Component (Direction) Index
Figure 9: Visualization of trained magnitudes.
Figure 10: Visualization of trained directions.
Table 11: Comparison between RestoreLCC and full- sull-parameter tuning (FT) on pruned LLaMA-7B models.
Method
Wanda (avg.)
SparseGPT (avg.)
LoFiT
56.82
52.11
58 ..43
56.44
Ours (RestoreLCC)
58.83
55.00
from both magnitude adjustment and the bias vector, which collectively shift it slightly toward the lower-right.
Comparison with Full-Parameter Tuning
We conduct experiments on LLaMA-7B. Table I1 summmrrzes the average results on commonsense reasoning tasks. On Wanda-pruned models, RestoreLCC achieves slightly better performance than full-parameter tuning (FT), On SparseGPT-prund models, RestoreLCC performs slightly worse than FT. Overall, RestoreLCCC achieves performance comparable to FT while requiring significantly fewer trainable parameters.
G Cross-Task Portability and Generalization of Probing
The probing module RestoreL has negligible overhead. For each attention head, we train only simple linear layer as the probing classifier. This step is performed before the recovery tuning stage (Section 4.2, LCC) and can be completed even on a low-end GPU (e.g.. NVIDIA RTX 1080) within minutes using minimal GPU memory. (ii) It is easy to implement. For new datasets or models, only head-wise activations need to be obtained. This can be achieved easily through three approaches: (a) using Python hook functions to capture the input/output of attention modules; (b) employing package. which offers convenient APIs for activation extraction; or (c) leveraging the Transf ormerLens ppackage, which supports modern architectures such Qwen3. In all cases, only a few lines of Python code are required, and the classifiers can be quickly trained. (iii) Cross-task portability. When dataset is available for new task, the Alpaca instruction-tuning dataset can be used as a seneral-purpost probing dataset. As shown Table T works well for both language modeling and multiple commonsense reasoning tasks, enabling recovery without task-specific data. This makes the Alpaca dataset an effective universal fallback for probing and tuning, substantially improving portability. (iv) Benefits outweigh the cost. Probing focuses on identifying informative attention heads -tyicaly only 10%-30% of all heads are sestored-greatie improving efficiency by avoiding unnecessary tuning. Hence, the computational benefits of probing far exceed its minimal cost.
the open-source Pyvene !
27

---

Table 12: Cross-task evaluation results using probing trained on BoolQ. RestoreLCC generalizes
effectively without retraining.
Tasks
RTE
ARC-e
ARC-c
Pruned Model
59.21
62.67
30.29
RestoreLCC (w/o retraining probing)
68.23
62.92
35.41
Generalization Across Tasks. The probing module generalizes effectively across diverse down- stream tasks without retraining, evidenced by two observations:
(1) Probing on general dataset, testing on diverse tasks, In our general recovery setup, attention heads are probed using the Alpaca dataset, while evaluations span distinct tasks including language modeling and seven commonsense reasoning benchmarks. As shown in Table T heads identified from Alpaca generalize well to diverse tasks such as BoolQ and ARC.
(2) Probing specific dataset, testing others. To further verify scalability, we probe attention heads using the BoolQ dataset and then apply them to recover pruned models on other datasets, including RTE, ARC-e, and ARC-c. Table [2 presents the results. Even without retraining. RestoreLCC successfully restores performance beyond the pruned baseline.
Efficiency at Scale
Scalability Larger Models. conduct task-specific recovery on the BoolQ dataset using LLaMA-70B. As shown in Table 13 our proposed RESTORELCC continues to outperform the LoRA baseline.
Table 13: Performance comparison on LLaMA-70B.
Method
BoolQ
Pruned Model
84.70
LoRA
86.21
RCstoreLCC
87.25
Latency and Throughput. We also measure inference latency and throughput on the BoolQ dataset after recovery. The delay ratio is defined as the total inference time of RestoreL.CC relative that of the original pruned model. This ratio is approximately 1.03, corresponding to only a <~3% slowdown, which is negligible. Thus, RestoreL.CC maintains nearly the same inference speed as the pruned baseline while delivering significantly better performance.
I Compatibility with Quantized Models
Our method can also be effectively applied heavily quantized models. We conducted experiments with 4-bit quantization on the pruned model (using Wanda for task-specific recovery) and report the recovery results in Table 14 As shown, the proposed Restorel.CC successfully recovers the performance of heavily quantized L.MMs.
Table 14: Performance of RestoreLCC on 4-bit quantized models.
Data
BoolQ
RTE
ARC-e
ARC-c
b-t Quantized Model
68.20
57.76
59.89
29.95
RestoreLCC (Ours)
72.97
68.23
61.03
33.02
28

---

Effect of Probing Samples
We investigate the effect of probing sample size on recovery performance. The results are shown in Tables 15 and T6 For general recovery, 1,000 samples are sufficient to achieve stable accuracy (=z 58.8). For task-specific recovery on the BoolQ dataset, only 200 samples are sufficient reach an accuracy of approximately 76.
Table 15: Effect of sample size general recovery.
Samples
100
200
500
1000 (reported)
3000
Mean Accuracy
57.63
14
58.69
58,83
58.81
Table 16: Effect of sample size on BoolQ task-specific recovery.
Samples
10
20
50
100 (reported)
200
500
1000
Accuracy
70.03
73.09
74.98
75.32
76.15
76. 12
76.51
K
Experimental Results on More LLMs
Implementation Details. The batch size is set to The learning rate is (le-4, le-5). The max sequence length is 512. All experiments DoRA, we use the same settings: a 16 and rank = 8. Regarding the applied modules, we try two configurations: (1) ["v hyyprojj". "o o__rojj"], which tumes only the head output matrices; and (2) "roj", "k_proj", "v v__rojj". "o "o_pr"""""""""""""""" ooprrj"],, which tunes all head matrices. For LoFiT, we experiment with 10%, 20%, and 30% of the heads and report the best results.
conducted
single H100 GPU. For LoRA and
For each probing classifier, we use the formulation of y = a (W m b), where m either m or m W is weight matrix (with input dimension equal to the size of and output dimension 1), b is a bias term, and a denotes the sigmoid function. The output y E [0, 1] represents the probability of contradiction or entailment. We employ cross-entropy loss optimized with Adam (learning rate =1x 10-2). The dataset divided into training and validation subsets in a 7:3 ratio, and models trained for 100 epochs based on empirical observations. We report the probing accuracy the validation set and rank the importance of attention heads according to their probing accuracy.
Tables 17 24 present experimental results across broader range of LLMSs from various families and sizes, including LLaMA- 1.AMAA13B/300 13B/3BB I.LaMA-2-7B/13B, LLaMA-3-8B
2,, Vicuna-7B-v1.5 [46], Tulu-
2-7B 47). Qwen-3-8B/14p [48). and DeepSeek-R SrepSeek-R1-Quee 1--Qwen3-88
[49]. Our proposed RestoreL.CC
consistently achieves the highest mean accuracy in all cases, demonstrating strong generalizability and scalability.
29

---

Table 17: Performance of general recovery on zero-shot language modeling (PPL.) and commonsense
reasoning tasks (accuracy) using LLaMA-13B.
Method
PPL
BoolQ
RTET
HellaSwag
WinoGrande
ARC-e
ARC-c
OBQA
Mean
Deuse Model
5.09
77.92
70.40
59.92
72.85
77.31
46.42
33,20
62.57
Unstractured Praning at 50%
Sparsity
Wandn
75.90
63.18
55.73
71.90
73.36
43.77
32.20
59,43
LoRA
6.08
78.56
63.18
59.15
71.11
74.75
46.33
34,80
61.13
64.62
71.43
74.79
35,00
61.42
LoFiT
6.29
73.52
68. 59
59.41
71.11
75.04
45.22
34.00
60.98
RestoreLCC (Ours)
70.04
75.97
35.20
Somi Structured Pruning (N:M=2:4)
50% Sporsity
SparseGPT
9.08
71.59
55 23
48.00
69.93
67.47
35.24
25.80
53.32
LoRA
73.21
57.04
53.57
68.27
38.57
28.80
55.22
DoRA
7.72
73.18
S7.76
$3.69
66.46
68.
38.57
28.40
55.18
53.66
RestoreLCC (Ours)
7.61
74.98
61.37
55.51
68.43
70.33
40.87
32.20
57.67
Structured Pruning at 20% Sparsity
SlimGPT
5.99
76.33
62.45
58.06
75.38
42.32
33,20
60.21
LoRA
6.29
76.39
64.26
60.20
72.14
75.72
45.31
33,80
61.12
60.86
LoFiT
6.51
76.01
66.79
$9.93
72.14
74.75
44.45
34.60
61.24
RestoreLCC (Ours)
71.84
61.88
72.77
75.34
37.40
Table 18: Performance (accuracy) of general recovery on zero-shot commonsense reasoning tasks
using LLaMA-30B.
Method
BoolQ
RTEI
HellaSwag 1
WinoGrande 1
ARC-e 1
ARC-e f
OBQA
Mcan
Dense Model
82.75
67.15
63.32
75.93
80.43
$2.90
36.00
65.50
Unstructured Prining
60% Sparsity
Wanda
76.85
$1.99
56.65
72.22
76.43
46.25
32.00
58.91
LoRA
78.69
57.04
61.13
71.59
78.03
47.95
36.40
61.55
DoRA
64.26
61.55
78.11
49.06
62.72
LoFiT
82.23
61.37
60.52
70.64
76.30
45.99
37.20
62.04
RestoreL.CC (Ours)
83.58
66.06
61.63
73.09
77.61
49.74
36.60
64.04
Table 19: Performance (accuracy) of general recovery zero-shot commonsense reasoning tasks
using LLaMA-2-7B.
Method
BoolQ
RTET
HellaSwag t
WinoGrande t
ARC-e t
ARC-c1
OBQA
Mean
Dense Model
77.74
62.82
57.14
69.14
76.30
43.52
31.40
59.72
Unstructured Pruning at 50% Sparsity
Wanda
53,43
$2.54
72.31
DoRA
76.67
$7.40
53.78
66.54
73.06
40.78
31.60
57.12
LoFiT
74.34
59,57
54.24
67.72
72.81
40.44
32.40
$7.36
RestoreLCC (Ouns)
75.23
63,90
55.08
67.56
73.48
41.21
32.80
58.47
Table 20: Performance (accuracy) of general recovery on zero-shot commonsense reasoning tasks
using LLaMA-2-13B.
Method
BoolQ
RTE!
HellaSwag f
WinoGrande t
ARC-e t
ARC-e1
OBQA
Mean
Dense Model
65.34
60.05
79,42
48.38
63.00
Unstruetured Pruming at 50% Sparsity
Wanda
81.13
59.21
57.01
70.96
75.84
42.92
32.00
59.87
DoRA
79,97
63,54
59.64
71.11
75,84
43,17
36,20
61.35
LoFiT
79.72
64.98
59.00
70.80
75.72
44.11
35.20
61.36
RestoreLCC (Ouns)
64.98
58.97
72.22
77.69
45.31
34.60
62.21
30

---

Table 21: Performance (aceuracy) of general recovery on zero-shot commonsense reasoning tasks
using LLaMA-3-SB.
Method
BoolQ
RTET
HellaSwag 1
WinoGrande
ARC-e
ARC-c
OBQA
Dense Model
81.38
69.68
60.19
80.09
50.43
34.80
64.18
Unstruetured Pruning at 50% Sparsity
Wanda
75.81
59.93
50.74
71.42
40.27
56.85
DoRA
74,46
55,96
55.52
70.01
75.63
45.05
30,80
58.20
LoFiT
78.04
61.01
55.89
67.88
73.61
43.26
31.40
58.73
RestoreLCC (Ours)
73.15
64.62
55.09
70.24
76.52
45.39
29.40
59.20
Table 22: Performance (accuracy) of general recovery on zero-shot commonsense reasoning tasks
using Vicuna-7B-vLe
Method
BoolQ
RTEI
HellaSwag
WinoGrande t
ARC-e1
ARC-c1
OBQA
Mean
Dense Model
80.92
63,90
56.43
69.61
75.59
43.17
33.00
60.37
Unstructured Pruming at 50% Sparsity
Wanda
54.87
53.20
71.46
56.79
DoRA
77.09
56.68
54.13
67.96
70.58
41.81
31.60
57.12
LoFiT
77.16
57.76
54.08
67.40
70.54
41.72
32.00
57.24
RestoreLCC (Ours)
80.83
55.96
53.86
68.90
74.41
41.72
32.40
58.30
Table 23: Performance (accuracy) of general recovery zero-shot commonsense reasoning tasks
using Tulu-2-7B.
Method
BoolQ 1
RTE!
HellaSwag
WinoGrande 1
ARC-e
ARC-c
OBQA 1
Mean
Dense Model
82.48
65.34
58.90
69.85
80.39
48.81
33.40
62.74
Unstructured Praming at 50% Sparviry
Wanda
81.07
55.60
54,31
68.43
74.79
42.58
31.20
58.28
DoRA
$4.09
71.38
43.09
59.15
LoFiT
78.59
68.23
54.06
68.19
71.30
43.43
31.80
59.37
Restorel.CC LCC (Ours)
80.46
67.87
54.56
69.14
74.37
43.17
31.20
60.11
Table 24: Performance (accuracy) of general recovery on zero-shot commonsense reasoning tasks using Qwen3-8B, Qwen3-14B, and DS-R1-8B.
Model
B
RTE
HellaSwag
WinoGrande f
ARC-e
ARC-e1
OBQA t
Mean 1
Qwen3-8B Unstructured Pruning at 50% Sparsity
Dense
86.57
78.34
57.10
67.80
83.54
55.80
31.00
65.74
Wanda
84.86
70.04
50.12
69,46
80.22
50.85
28.20
61.96
DoRA
86.30
77.26
56.13
68.82
82.37
53.58
31.80
65.18
Ours
87.13
79.06
57.29
70.48
84.68
57.59
32.60
66.98
Qwen3-14B Unstructured Pruming 50%
Sparsity
Dense
89.33
77.62
60.97
73.01
84.22
35,00
68.40
Wanda 87.43
72.92
57.69
69.77
83.50
55.97
33.60
65,84
DoRA
88.07
78.70
59.87
74.66
84.09
57.85
34.50
68.25
Ours
89.57
80.14
60.70
74.74
86.15
60.15
35.00
69.49
DS-RI-8P
Unstructured Pruming at 50% Sparsity
Dense
85.87
80.51
58.54
67.01
80.18
51.71
31.60
65.06
Wanda
82.48
77.26
51.53
67.17
75.67
48.12
28.40
61.52
DoRA
84,80
74.37
56.84
68.35
77.90
50.09
32.30
63.52
Ours
87.22
74.01
57.41
69.46
83.84
55.63
33.80
65.91
31


Reviews:
[{"review_id": "AsUWCEBA4t", "reviewer": "Reviewer_Kk22", "summary": "This paper proposes RestoreLCC, a parameter-efficient fine-tuning method designed for post-pruning model recovery. Under several different pruning settings, RestoreLCC demonstrates superior recovery performance compared to directly applying other PEFT methods.", "questions": "1. Could you explain why, in Figure 1, the accuracy on the right increases beyond that of the dense model after applying LCC?\n\n2. This paper seems to lack a comparison with full-parameter fine-tuning for performance recovery.\n\n3. I would like to know the effect of applying the method to the FFN layers.\n\n4. Please clarify where the PEFT methods are added in the model and provide other relevant implementation details (e.g., rank, alpha, learning rate...).", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1. An approach similar to distillation is used to preserve errors in the vectors, which reduces the attention information loss after pruning and thus mitigates the performance degradation caused by pruning.\n\n2. The method is validated under different pruning settings.\n\nWeaknesses\n\n1. The underlying rationale behind the effectiveness of the proposed method seems to lack a deeper explanation. I am curious why the information compensation is specifically applied at the attention module, rather than after the attention or at the intermediate dimensions of the FFN, for example.\n\n2. The method appears to require using a large number of intermediate activations during recovery fine-tuning in order to compute the error vectors. I think this seems to deviate from the original purpose of using PEFT for recovery fine-tuning. Could you provide a comparison of the training overhead of different methods during the recovery fine-tuning stage? In my opinion, the number of trainable parameters is not the main concern in this stage.\n\n3. All compared PEFT methods are limited to very small trainable parameter settings. I believe this setup is favorable to the method. However, it is clear that the training overhead of PEFT is not sensitive to the number of trainable parameters; for example, the overhead difference between rank=1 and rank=16 is minimal. I think that in the recovery fine-tuning stage, performance recovery should be the main focus, and a higher rank may yield better performance with only negligible increases in fine-tuning overhead.", "was_revised": true, "final_justification": "I think the authors have partially addressed my concerns. This method can be applied in scenarios involving pruning recovery and represents a PEFT approach focused on specific contexts. I believe that after incorporating the additional content provided by the authors, this paper can be accepted.", "rebuttal": "Dear Reviewer Kk22,\n\nThank you very much for reviewing our paper and recognizing our method as **effective**. Your concerns mainly focus on the effect of RestoreLCC on **FFNs, its overhead, and clarification of certain details**.  Following your suggestions, we have added further analysis and additional experiments on these points. We are pleased that, with your feedback, the paper has been significantly improved. Our responses (**R**) to your weaknesses (**W**) and questions (**Q**) are as follows.\n\n[**W1**] Explain why RestoreLCC uses attention rather than FFN.\n\n[**R1**] Thank you for this valuable comment. The output of each Transformer block consists of the attention module output (Eq.1 in our paper) and the FFN module output. In our study, we focus on the **attention output matrix** rather than the FFN output matrix for the following reasons:\n\n(1) Studies on mechanistic interpretability show that **attention heads specialize in distinct functions for different tasks, while FFNs mainly store knowledge and map inputs to outputs**. For example, [1] finds that in arithmetic tasks, attention heads process key information and FFNs then produce the final answer. They also show that fine‑tuning only important heads outperforms tuning all parameters (including FFNs). Similarly, [2] demonstrates that different heads play distinct roles, and [3] also supports this observation. By contrast, FFNs store factual knowledge [4] and refine output logits [5]. Hence, compensating attention heads is more appropriate than compensating FFNs.\n\n(2) Attention heads provide **finer information**, whereas FFNs produce high‑dimensional, aggregated outputs. In LLaMA-7B, each head outputs 128 dimensions (32 heads per layer), while an FFN outputs 4096. Smaller head outputs allow finer analysis. Moreover, choosing 32 heads can span multiple layers, while the same size in FFN covers only one layer.\n\n(3) **Empirical evidence**. We applied RestoredLCC to FFNs and report the best results as follows (More results can be found in below reponse[**R6**].) Compensating attention achieves 58.83, while compensating FFNs only reaches 57.00.\n\n|  | BoolQ | RTE   | Hell. | Wino. | ARC-e | ARC-c | OBQA | Mean  |\n|--|---|---|---|--|----|--|--|---|\n| On attn. | 72.84 | 69.68 | 56.34 | 65.98| 71.8 | 40.96 | 34.2 | 58.83 |\n| On FFNs | 69.42 | 63.54 | 55.54| 67.17 | 70.96 | 39.59 | 32.8 | 57 |\n\n[1] Interpreting and improving large language models in arithmetic calculation, ICML 2024\n\n[2] Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs, ICLR 2025\n\n[3] Dressing up llm: Efficient stylized question-answering via style subspace editing. ICLR 2025\n\n[4] Locating and Editing Factual Associations in GPT, NeurIPS 2022\n\n[5] Transformer Feed-Forward Layers Are Key-Value Memories, EMNLP 2021.\n\n[**A2**] Training overhead.\n\n[**R2**] Thank you for the question. We would like to clarify that **the overhead of our proposed RestoreLCC is smaller than advanced PEFT methods while achieving much better performance**. While recovering pruned LLaMA‑7B (1 H100 GPU, torch.bfloat16 precision, 1 epoch, max_length=512, batch size=8, Alpaca Dataset),the empirical and theoretical evidence are:\n\n(1) Empirical Analysis. The table below compares training time and GPU memory usage. For training time, the methods rank as follows: RestoreLCC < LoRA < DoRA < LoFiT. For GPU memory usage: LoRA < RestoreLCC = LoFiT < DoRA. Compared with the DoRA and LoFiT, RestoreLCC has the lowest overall overhead while providing superior performance.\n\n|llama-7B|Time|Memory |\n|----|----|---|\n| LoRA | 4h20min | 61GB |\n| DoRA| 6h08min| 71GB |\n| LoFiT | 8h11min| 65GB |\n| RestoreLCC | 4h13min | 65GB |\n\n(2) Theoretical analysis. For RestoreLCC, the main overhead comes from: (i) trainable parameters (the scaling factor β in Eq. 8 and one bias term per head), (ii) intermediate activations (components vᵢ in Eq. 8), and (iii) coding efficiency due to modifications in the attention head output.\n\n(i) RestoreLCC introduces **fewer trainable parameters** than PEFT methods such as LoRA and DoRA (see Section 4.3 and Appen. C for details)\n\n(ii) **The overhead from intermediate activations (vᵢ) is extremely small and can be ignored**. The activations are fixed (frozen) during training. Each attention head has 128 dimensions; by the property of SVD this corresponds to at most 128 components, i.e., 128 × 128 = 16,384 parameters per head. Even in the extreme case of using all 1,024 heads, the adds only 1,024 × 16,384 ≈ 16.8M parameters - only about 0.24% of a 7B model. Moreover, as shown in Section 3 and Figure 8, only 10–30% of heads are required for recovery, so the actual overhead is much smaller.\n\n(3) **We optimized our code implementation to avoid inefficient loops**. Unlike LoFiT, which iterates over each head in thier code, our method uses only matrix multiplications to update all heads. This eliminates costly looping inside the Transformer. Consequently, RestoreLCC trains much faster than LoFiT; re‑implementing LoFiT with our approach similarly reduces its training time, confirming that the efficiency gain comes from our design.\n\n**In summary, RestoreLCC introduces smaller overhead than advanced PEFT methods.**\n\nThe core code implementation is as follows:\n```\n# (B:batch_size, T: sequence_length \n# H:head number per layer, D: head_dimension)\nout = attn_output.view(B, T, H, D)\n# comp: [H, D, D]:decomposed fixed components for heads\n# beta [H,D]: beta in Eq.8\n# b [H,D]: bias in Eq.8\nadded_info = (\n    torch.bmm(comp.transpose(1, 2),  # (H, D, D)\n              beta.unsqueeze(2))  # (H, D, 1)\n    .squeeze(2)\n)\nb = b.view(1, 1, H, D)\nadd = added_info.view(1, 1, H, D)\n# modify every head in one go\nattn_output = out  + add + b\nattn_output = attn_output.view(B, T, H*D)\n```\nNotably, only a subset of attention heads use non‑zero values for comp (frozen, no gradients), beta (trainable), and b (trainable). GPUs will automatically optimize both the storage and the arithmetic for these zero‑valued vectors.\n\n[**W3**] Overhead sensitivity to number of parameters.\n\n[**R3**] We would like to state that our method is **not sensitive to the number of trainable parameters**.\n\n(1) First, as in our response [**R2**] to your [**W2**], RestoreLCC introduces less overhead than other advanced PEFT methods.\n\n(2) Empirical results. The number of trainable parameters in RestoreLCC depends on the proportion of attention heads selected for tuning. The table below reports GPU memory and training time for different ratios of heads. The results show: (i) The GPU memory consumption remains almost unchanged as trainable parameters increase and constantly lower than DoRA (71 GB). (ii) The training time is mostly smaller than DoRA and LoFiT.\n\n| head ratio| 0.1| 0.2| 0.3| 0.5|1|\n|----|-----|-----|--|-------|-----|\n|GPU| 65GB |65GB| 65GB|66GB|68GB|\n|time| 3h40min | 4h13min | 4h40min | 5h30min | 7h42min |\n\n(3) We would like to highlight that more parameters do not necessarily lead to better performance and thus do not introduce sensitivity. As shown in Section 3, not all attention heads are equally useful for recovery. Also, figure 8 demonstrates that using only 10%–30% of the most discriminative heads is sufficient to achieve the best performance. This is also why our method first contrastively probes important heads.\n\n[**Q1**] Results explanation on Figure 1.\n\n[**R4**] Thank you for raising this point. The improvement are for two reasons:\n\n(1) Task‑specific information. The added components act as task‑specific vectors (often referred to as task function vectors [6] or task steering vectors [7]). Consistent with these prior studies, such vectors provide task‑aware guidance and significantly enhance the model’s performance over the original  model.\n\n(2) Removal of useless or harmful components.Some lost components in each attention head can even be harmful, as they may introduce noise that negatively affects the model’s output. By recovering only the useful components and excluding the harmful ones, the model’s performance improves.\n\n[6] Function Vectors in Large Language Models, ICLR 2024\n\n[7] Semantics-Adaptive Activation Intervention for LLMs via Dynamic Steering Vectors, ICLR 2025\n\n[**Q2**] Comparison with full-parameter tuning.\n\n[**R5**] We conduct experiments on LLaMA‑7B. Code implementation is from Wanda. The average results are as follows. On Wanda‑pruned models, RestoreLCC achieves slightly better performance than FT. On SparseGPT‑pruned models, RestoreLCC is slightly worse than FT. Overall, RestoreLCC achieves comparable to FT.\n\n|      | ave. |    | ave. |\n|-----|---|-----|-----|\n| Wanda | 54.09| SparseGPT | 48.99   |\n| LoFiT | 56.82 | LoFiT| 52.11   |\n| FT | 58.43 | FT | 56.44 |\n| Ours  | 58.83 | Ours | 55|\n\n[**Q3**] Effects of using FFNs.\n\n[**R6**] We apply RestoreLCC to FFNs and evaluated different numbers of FFN layers. The best result obtained was 57.00, which is lower than the 58.83 achieved when using attention heads.\n\n| FFN ratio | 0.1 | 0.3 |0.5 | 0.7 |1 |\n|-----|-----|-----|-------|-------|------|\n| ave | 55.92 | 57  | 56.24 | 56.68 | 56.8 |\n\n[**Q4**] Details for PEFT implementations.\n\n[**R7**] We apologize for not providing these details. For LoRA and DoRA, we use the same settings: α = 16 and rank = 8. Regarding the applied modules, we try two configurations: (1) [\"v_proj\", \"o_proj\"], which tunes only the head output matrices; and (2)[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"]`, which tunes all head matrices. The best results are reported in the paper.\n\nFor LoFiT, we experiment with 10%, 20%, and 30% of the heads and report the best results.\n\nThe batch size is set to 8. The learning rate is 1e-4. The max sequence length is 512. The number of training epochs is 2 for global recovery and 5 for task-specific recovery. All experiments are conducted on a single H100 GPU.\n\n**Thank you again for taking the time to review our paper. We will incorporate these discussions into the revised version and look forward to your feedback.**"}, {"review_id": "rabUVHcmF2", "reviewer": "Reviewer_n2VC", "summary": "This paper introduces RestoreLCC, a novel and targeted restoration strategy for pruned large language models (LLMs). Unlike prior parameter-efficient fine-tuning (PEFT) methods that overlook the unique properties of pruned models, RestoreLCC explicitly identifies and compensates for the lost information caused by pruning. The method consists of two key components: (1) Contrastive Probing, which identifies important attention heads via contrastive activation editing; and (2) Lost Component Compensation (LCC), which reconstructs missing activation components using learned linear combinations of principal directions and injects them back to recover the model’s capability. Extensive experiments across multiple pruning schemes (structured, semi-structured, unstructured) and LLM variants demonstrate that RestoreLCC achieves superior accuracy and perplexity recovery without compromising sparsity or inference efficiency.", "questions": "**Scope of Probing Generalization**: Given that contrastive probing is data/model/pruning-specific, how scalable is RestoreLCC across multiple downstream tasks without retraining the probing module?\n\n**Efficiency at Scale**: While parameter overhead is small, does RestoreLCC scale gracefully with larger models (e.g., >30B)? Are there latency/throughput benchmarks in real-world scenarios?", "limitations": "Yes, but not enough.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- The paper introduces a unique insight—model degradation from pruning is traceable to lost activation components—which leads to a targeted and explainable restoration method.\n\n- The methodology is well-justified with theoretical and empirical analyses. Ablation studies, probing diagnostics, and interpretability results are thorough and convincing.\n\n- The overall exposition is clear, well-structured, and supported with informative figures (e.g., logit gain visualizations, component effects).\n\n- The work addresses a critical bottleneck in deploying sparse LLMs by offering a low-overhead, generalizable recovery mechanism. It shows consistent gains over strong PEFT baselines like LoRA, DoRA, and LoFiT.\n\n### Weaknesses\n\n- The scope of evaluation is currently limited to LLaMA-family models. Including more diverse architectures (e.g., Mistral, Qwen, Baichuan) would strengthen the claim of generalizability.\n\n- The contrastive probing module depends on data/model-specific classifier training, which could affect practical deployment and cross-task portability.\n\n- A few sections (e.g., lines 38–59) in the main text are densely written, with overlapping figure references, which slightly affects clarity.", "was_revised": true, "final_justification": "## Final Justification\n\nI have raised my score to **Accept**. The authors provided an exceptionally strong and thorough rebuttal that has fully addressed all of my initial concerns. The new experiments and clarifications have significantly strengthened the paper's claims and overall contribution.\n\n---\n\n### Resolved Issues\n\n**Model Generalizability**  \nMy primary concern regarding the method's generalizability being limited to LLaMA-family models was convincingly resolved. The authors added extensive experiments on entirely different and recent architectures (**Qwen3**, **DeepSeek-R1-Qwen3**), demonstrating that **RestoreLCC** consistently outperforms baselines. This provides compelling evidence for the method's broader applicability.\n\n**Practicality and Scalability**  \nMy questions about the practical deployment, specifically the portability of the probing module and the method's efficiency at scale, were also fully addressed with new, targeted experiments. The authors demonstrated:\n\n- Successful cross-task performance recovery without retraining the probing module, directly answering my scalability question.\n- Effective performance on a much larger **70B model (LLaMA-70B)**, proving scalability.\n- A concrete latency benchmark showing a negligible inference overhead of ~**3%**, confirming real-world efficiency.\n\n---\n\n### Unresolved Issues\n\nFrom my perspective, there are **no remaining unresolved issues**. The authors' response was comprehensive and satisfying.\n\n---\n\n### Overall Assessment\n\nThe weight of these resolved issues is high, as they were central to validating the paper's core claims of being a **generalizable** and **practical** method. The authors' diligence in running these new experiments has transformed the paper from a promising one with limitations to a **robust and well-supported contribution**.\n\n---\n\n### Key Points for the Authors\n\n- Your rebuttal was exemplary. The new experiments were precisely what was needed to validate your claims and were highly persuasive.\n- The paper is now substantially stronger, with robust evidence backing the **generalizability**, **cross-task portability**, and **efficiency** of RestoreLCC.\n- Please ensure that all the new results and the insightful discussions from the rebuttal are **carefully and clearly integrated** into the final version of the manuscript.", "rebuttal": "Dear Reviewer n2VC,\n\nThank you for taking the time to review our paper and for recognizing it as **novel, with unique insights that are clearly and extensively justified**. The concerns you raised mainly relate to **extending our approach to other LLMs, scalability and the portability of the probing module**. Following your suggestions, we have added experiments and additional analysis to address these points. We are glad to see that, with your feedback, the paper has been greatly improved. Our responses [**R**] to your mentioned weaknesses [**W**] and questions [**Q**] are summarized below.\n\n[**W1**] Evaluate on more LLMs from different families.\n\n[**R1**] Thank you for raising this question. To verify the generalizability of RestoreLCC on more recent LLMs, we evaluated it on the latest **Qwen3 family and DeepSeek‑R1‑Qwen3‑8B**, both released at the end of **May 2025**. Our results on Qwen3‑8B, Qwen3‑14B, and DS‑R1‑8B are as follows. Compared with other baseline methods, RestoreLCC consistently delivers significant improvements.\n\nWe would also like to highlight an interesting observation on the latest Qwen3 models: the recovered model performs even better than the original dense model. We believe this is because, despite the state‑of‑the‑art performance of Qwen3, the model still benefits substantially from additional tuning with the Alpaca instruction‑tuning dataset.\n\n| Qwen3-8B | BoolQ | RTE   | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA  | Mean  |\n|----------|-------|-------|-----------|------------|-------|-------|-------|-------|\n| Dense    | 86.57 | 78.34 | 57.10     | 67.80      | 83.54 | 55.80 | 31.00 | 65.74 |\n| Wanda    | 84.86 | 70.04 | 50.12     | 69.46      | 80.22 | 50.85 | 28.20 | 61.96 |\n| DoRA     | 86.30 | 77.26 | 56.13     | 68.82      | 82.37 | 53.58 | 31.80 | 65.18 |\n| Ours     | 87.13 | 79.06 | 57.29     | 70.48      | 84.68 | 57.59 | 32.60 | **66.98** |\n\n\n| Qwen3-14B | BoolQ | RTE   | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA  | Mean  |\n|-----------|-------|-------|-----------|------------|-------|-------|-------|-------|\n| Dense     | 89.33 | 77.62 | 60.97     | 73.01      | 84.22 | 58.62 | 35.00 | 68.40 |\n| Wanda     | 87.43 | 72.92 | 57.69     | 69.77      | 83.50 | 55.97 | 33.60 | 65.84 |\n| DoRA      | 88.07 | 78.70 | 59.87     | 74.66      | 84.09 | 57.85 | 34.50 | 68.25 |\n| Ours      | 89.57 | 80.14 | 60.70     | 74.74      | 86.15 | 60.15 | 35.00 | **69.49** |\n\n| DS-R1-8B | BoolQ | RTE   | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA  | Mean  |\n|----------|-------|-------|-----------|------------|-------|-------|-------|-------|\n| Dense    | 85.87 | 80.51 | 58.54     | 67.01      | 80.18 | 51.71 | 31.60 | 65.06 |\n| Wanda    | 82.48 | 77.26 | 51.53     | 67.17      | 75.67 | 48.12 | 28.40 | 61.52 |\n| DORA     | 84.80 | 74.37 | 56.84     | 68.35      | 77.90 | 50.09 | 32.30 | 63.52 |\n| ours     | 87.22 | 74.01 | 57.41     | 69.46      | 83.84 | 55.63 | 33.80 | **65.91** |\n\n\n[**W2**] The contrastive probing module depends on data/model-specific classifier training, which could affect practical deployment and cross-task portability.\n\n[**R2**] Thank you for pointing out this concern. We would like to highlight that (1) the overhead of training the classifiers on new models and datasets is negligible, (2) the probing classifier is very easy to implement for new datasets and new models, (3) cross‑task portability can be addressed using the Alpaca dataset when no data is available for a new task, and (4) **the benefits of probing far outweigh its cost since it avoids tuning all heads**. We agree this is an interesting direction and plan to explore task‑agnostic probing strategies in future work to further improve portability. We further clarify these four points as follows:\n\n(1) Negligible overhead. For each attention head, we train only a simple linear layer as the probing classifier. This process is precomputed before the recovery tuning stage (Section 4.2 LCC). It can be completed even on a low‑end GPU (e.g., NVIDIA RTX 1080), takes only a few minutes, and requires very little GPU memory. Therefore, the overhead is negligible.\n\n(2) Easy implementation. For new datasets and models, the only step needed is to obtain head‑wise activations from the new dataset and model. The code implementation is simple and we provide three different ways:\n(i) Use Python “hook” functions to capture the inputs/outputs of attention modules directly.\n(ii) Use the open‑source Pyvene package, which provides convenient APIs to extract activations.\n(iii) Use the open‑source TransformerLens package, which supports easy activation extraction even for the latest models (e.g., Qwen3).\nIn all three cases, only a few lines of Python code are needed, after which the classifiers can be quickly trained.\n\n(3) Cross‑task portability. When there is no dataset for a new task, the Alpaca instruction‑tuning dataset can be used in a general recovery setting (lines 279‑281). As shown in Table 1, this dataset works well for both language modeling and seven commonsense reasoning tasks, enabling recovery without task‑specific data. Therefore, in such cases, **the Alpaca dataset can serve as a universal fallback for probing and tuning across different datasets**. This substantially improves the portability of our method.\n\n(4) **Benefit outweights cost**. Probing allows us to focus on informative heads: in practice, only 10%–30% of heads are restored. This greatly improves efficiency by avoiding unnecessary tuning of all heads. Hence, the efficiency gains from probing far outweigh its small cost.\n\n\n[**W3**] A few sections (e.g., lines 38–59) in the main text are densely written, with overlapping figure references, which slightly affects clarity.\n\n[**R3**] Thank you for pointing this out, and we apologize for the inconvenience caused. We will carefully double‑check and revise all such presentations to ensure clarity and correctness.\n\n[**Q1**] Scope of Probing Generalization: Given that contrastive probing is data/model/pruning-specific, how scalable is RestoreLCC across multiple downstream tasks without retraining the probing module?\n\n[**R4**] Thank you for your question. We would like to clarify that **the probing module generalizes well across downstream tasks without requiring retraining**. This is supported by the following two observations:\n\n(1) Probing on a general dataset, then testing on diverse downstream tasks. Our general recovery setting (lines 279–281) follows this requirement. For the results in Table 1, attention heads are probed using the Alpaca instruction‑tuning dataset, while the evaluation tasks are very different, including a language modeling dataset and seven commonsense reasoning benchmarks. As shown in Table 1, heads probed on general instructions successfully recover performance on diverse tasks such as BoolQ and ARC.\n\n(2) Probing on one specific downstream dataset, then testing on other tasks. To further verify the scalability, we use **probing results obtained on the BoolQ dataset and apply them for recovery on other datasets including RTE, ARC‑e, and ARC‑c**. The below table shows the results. Even without retraining, the pruned model’s performance is successfully recovered on these tasks, achieving better results than the pruned baseline. It is notable that the results of original pruned model are from Table 3 in our paper. \n\n| Tasks                                 | RTE   | ARC-e | ARC-c |\n|---------------------------------------|-------|-------|-------|\n| Pruned Model                          | 59.21 | 62.67 | 30.29 |\n| RestoreLCC without retraining probing | 68.23 | 62.92 | 35.41 |\n\n[**Q2**] Efficiency at Scale: While parameter overhead is small, does RestoreLCC scale gracefully with larger models (e.g., >30B)? Are there latency/throughput benchmarks in real-world scenarios?\n\n[**R5**] Thank you for your question. \n\nFirstly, RestoreLCC scales well to larger models. We conducted task‑specific recovery (lines 282–285 in our paper) on the BoolQ dataset using LLaMA‑70B. The results on the below table show that our proposed RestoreLCC continues to outperform the LoRA baseline.\n\nDue to the significant time and GPU resources required for tuning a 70B model, we were only able to complete the current experiment before July 30, 2025. We will continue this work and include additional results in the revised version of the paper.\n\n| llama-70B    | BoolQ |\n|--------------|-------|\n| Pruned Model | 84.7  |\n| LoRA         | 86.21 |\n| RestoreLCC   | 87.25 |\n\n\nSecondly, regarding latency and throughput, we measured the inference speed on the BoolQ dataset after recovery. The delay is calculated as the ratio of total inference time with RestoreLCC to that of the original pruned model. This ratio is approximately 1.03, indicating only a ~3% slowdown, which can be ignored. Thus, RestoreLCC maintains nearly the same inference speed as the original pruned model while delivering significantly better performance.\n\n[**Q3**] Concerns regarding the figure placement and labels.\n\n[**R6**]  Thank you for pointing out these concerns. We will carefully review all figures and make the necessary adjustments to their placement, labels, and formatting to ensure that they are clear and easy to read.\n\n\n**Thank you once again for reviewing our paper and for your insightful comments. We will incorporate all of these discussions into the revised version. Please let us know if you have any further questions, and we look forward to your feedback.**"}, {"review_id": "G1gUL5rKc3", "reviewer": "Reviewer_oP8x", "summary": "Network pruning reduces the complexity of a trained model, enabling more efficient deployment, but at the cost of potentially reduced quality.  Existing approaches to efficiently recovering quality (i.e., excluding fine-tuning the full network) do not consider the pruning that occurred when performing recovery.  The authors perform an empirical study to gain insight into how leveraging the information in *lost* activations can be used to restore quality.  Using these insights, RestoreLCC injects the lost information back into the model via a learned bias term in a three step process: (1) contrastive probing identifies the important attention heads, (2) Lost Component Compensation (LCC) performs SVD on the difference between the outputs of the important dense and pruned heads and learns optimal magnitudes for the top components before (3) augmenting the important heads with these learned bias terms, \"re-injecting\" the lost information back into the model.  Empirical studies show improvements on most metrics compared to baselines.", "questions": "1. What weight tensors are pruned in your experiments?  My understanding of the submission is that it's only Q, K, V, and dense output projections in attention blocks, not MLP blocks. (Clarity, Significance)\n2. Is there anything fundamentally stopping RestoreLLC from being used on heavily quantized models, or does it only apply to pruned models? (Significance)\n3. How does EoRA compare for some model, given the same sparsities and tasks? (Quality, Significance)\n4. What software was used, and how much computation time was required to probe and learn the compensatory terms for each model? (Quality)", "limitations": "This approach seems to be limited to attention weights, and cannot address MLPs.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\nThe problem statement is well-motivated, and the approach seems novel, is theoretically-grounded, and is generally successful.  The solution's low-overhead is particularly compelling; additional bias terms are a very small tax on model size and deployment efficiency, in contrast to the LoRA family of techniques, which require additional (though relatively small) weight matrices.  The contrastive probing and compensation process is adequately described such that an expert could reproduce the results.  The evaluations are sufficiently broad with respect to models and tasks to inspire confidence that there are no surprises hidden behind under-explored areas.  This particular solution to the problem incorporates existing ideas, but is largely original.\n\t\n## Weaknesses\nClarity is a mixed bag - while the exposition is clear, some details are not.  For example, the precise sparsity imposed is unclear.  At times the activations are pruned (line 113), but later it is the attention heads (I assume weights) that are pruned (line 132).  It's also not clear which weights are pruned; is it only those in the attention blocks Q, K, V and dense output projections, or also weights in the MLP blocks (which can be much larger due to the expansion factor)?  This has direct bearing on the significance of the submission.  Also, consider avoiding the term \"performance;\" it is overloaded and can mean both model quality as well as execution speed or efficiency.  In this context, where both meanings are plausible, it's better just to be explicit.\n\nQuality is also slightly lacking.  \n- The inference speed overheads in Appendix C isn't very useful without knowing the deployment conditions: hardware, software, type of pruning, batch sizes, sequence lengths, etc. all play a large role in both dense and sparse efficiency.  Further, the magnitude of the speedups will influence how important the overheads are.\n- The ablation study comparing identifying important heads using contrastive probing vs. randomly-selected heads.  Surely there's a simpler metric that could perform better than random choice: MSE between the output of dense and pruned heads, K-L Divergence, etc.\n- Figure 5 shows the top-5 decoded tokens, but the text suggests it shows the top-10 (line 347)\n- There are two references that should be considered for discussion and inclusion:\n    - Moore and Chaudhuri probe networks' activations in order to drive pruning decisions directly.  \n    - Liu et al. do take the pruning into account when restoring quality with low-rank adaptation.  This work, in particular, should be included as a baseline, since it stands in contrast to, as claimed in the abstract, \"most PEFT methods are designed for dense models and overlook the distinct properties of pruned models.\"\n\n\n1. Moore and Chaudhuri, Using noise to probe recurrent neural network structure and prune synapses, NeurIPS 2020\n2. Liu et al., EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation, https://arxiv.org/abs/2410.21271", "was_revised": true, "final_justification": "All of my questions were resolved with the authors' response.  Assuming the authors can address the clarity issues raised by myself and the other reviewers, I have no other issues.", "rebuttal": "Dear Reviewer oP8x,\n\nThank you for taking the time to review our paper and for recognizing it as **well‑motivated, novel, theoretically grounded, and successful**. The concerns you raised mainly focus on **comparisons with different head‑selection methods, EoRA, extension to quantized models, and clarifications on pruning**. We have carefully addressed all of your concerns, and we are glad to see that, with your suggestions, the quality of the paper has been greatly improved. Our responses [**R**] to the raised weaknesses [**W**] and questions [**Q**] are summarized as follows.\n\n[**W1**] Clarify pruning-related terms\n\n[**A1**] Thank you for pointing this out. We apologize for not explaining these terms clearly. Below we clarify the terminology:\n\n* LLM pruning:\nCurrent LLM pruning methods focus on pruning all the weight matrices (including attention matrices $Q, K, V, O$ and MLP block matrices) rather than the activations of each Transformer block. For example, Wanda removes 50% of the weights in each matrix of both the attention and MLP modules (setting them to zero), leading to an overall sparsity of 50%. Other methods follow a similar manner. **Therefore, in our paper, all weight matrices in the LLM are pruned, and we focus on restoring them by compensating through attention heads. (The reasons for not using MLPs/FFNs are explained in our response R10 below.)**\n\n* Activation:\n  An activation refers to the output of a specific module.\n\n* Pruned activation: It is the activation produced by a pruned module. While the activations have the same dimension as those from the original model, they may be less informative because the underlying weight matrices are pruned. Note that **the activations themselves are not pruned**; only the weight matrices are. We acknowledge that the term “pruned activation” may be confusing, and will revise this terminology for clarity.\n\n* Sparsity: It refers to the proportion of weight parameters that are pruned. The sparsity settings for our experiments can be found in Section 5. For Wanda and SparseGPT, each weight matrix is pruned with the same ratio (e.g., 50% parameters removed), resulting in an overall sparsity of 50%. For SlimGPT, a dynamic ratio is applied to different layers (e.g., 12.5% in shallow layers and 30% in deeper layers, giving an overall sparsity of 20%) based on a specific ratio-assignment formula.\n\n* Performance: Based on your suggestions, we will explicitly use terms such as quality, execution speed, or efficiency in different contexts when discussing performance.\n\n As the report results in Tables 1 - 3, our proposed method can be applied to all three different pruning paradigms.\n\n[**W2**] Overhead details in Appex. C.\n\n[**R2**] We apologize the for the missed details. For LoRA and DoRA, we use the same settings: α = 16 and rank = 8. Regarding the applied modules, we try two configurations: (1) [v_proj, o_proj], which tunes only the head output matrices; and (2)[q_proj, k_proj, v_proj, o_proj]`, which tunes all head matrices. The best results are reported in the paper.\n\nFor LoFiT, we experiment with 10% - 30% of the heads and report the best results.\n\nThe batch size is set to 8. The max sequence length is 512. All the experiments are conducted on a single H100 GPU.\n\nThe overhead is reported using the best hyperparameter settings, with the base model being LLaMA‑7B pruned by Wanda at 50% sparsity. The inference speed is measured as the ratio of the total testing time for general recovery (lines 279–281) of the restored model to that of the original pruned model.\n\n[**W3**] Compare with other metric-selected heads.\n\n[**R3**] Thank you for your suggestion. We have conducted additional experiments with two alternative head-selection strategies: (1) MSE-selected heads: selecting heads with the smallest MSE between the outputs of the dense and pruned models. (2) KL-selected heads: selecting heads with the smallest KL divergence. The results are as follows: our probing-based selection consistently identifies important heads and is more effective than other metric-based approaches.\n\n|Method| Accuracy |\n|-------|---|\n|RestoreLCC | 58.83|\n|random| 57.57|\n|MSE-selected| 58.14|\n|KL-selected| 57.92|\n\n[**W4**] line 347: top-10 should be top-5.\n\n[**R4**] We apologize for the typo. We will correct “top‑10” to “top‑5” and carefully double‑check the paper for similar issues.\n\n[**W5**] Discuss two references.\n\n[**R5**] Thank you for you suggestions. We have carefully read both references and will add to our revised paper.\n\n(1) (add to related work) Moore and Chaudhuri utilize activation noise to probe network structure and identify redundant neurons. (add to future work) They also discuss a novel way of leveraging activation noise for neuron identification. We believe activation noise could similarly be used to enhance contrastive probing, and we plan to explore this direction in future work.\n\n(2) EoRA. (add to related work) EoRA provides a fine-tuning-free approach to recover pruned models by searching low-rank spaces in a task-specific eigenspace to minimize compression loss.\n(add to Table 1) We also compare our method with EoRA, and the results are summarized below (Detailed results can be found in R8 below). It can be observed that our method significantly outperforms EoRA.\n\n|Wanda| Mean  |\n|------|----|\n| EoRA |55.88|\n| RestoreLCC|58.83|\n| SparseGPT| Mean|\n| EoRA|51.21|\n| RestoreLCC|55|\n\n[**Q1**] What weight tensors are pruned? \n\n[**R6**] In our experiments, all weight matrices in both the attention and MLP blocks are pruned, following the exact pruning settings of the respective methods: Wanda, SparseGPT, and SlimGPT.\n\nThese three approaches are well‑established state‑of‑the‑art pruning methods, each representing a different pruning paradigm. The primary difference lies in the positional constraints of pruned parameters in a matrix. To demonstrate the generalizability of our method across different pruning strategies, we evaluate RestoreLCC on all three types of pruned models, as shown in Tables 1–3.\n\nOur recovery operations focus on the attention modules in order to restore the performance of the entire model. We also discuss the possibility of recovering MLPs (FFNs) in our response [**R10**] below.\n\n[**Q2**] Application to heavily quantized models. \n\n[**R7**] Thank you for the question. Our method can also be applied to heavily quantized models. We conducted experiments with 4‑bit quantization on the pruned model (by Wanda) and report the recovery performance below. The experimental setup follows lines 282–285 and Table 3 in our paper. As shown, the proposed RestoreLCC successfully recovers the performance of heavily quantized LLMs.\n\n|data| BoolQ| RTE| ARC-e| ARC-c |\n|--------|-------|-----|-------|-------|\n| 4-bit quantized model|68.2|57.76| 59.89| 29.95|\n| RestoreLCC (Ours)| 72.97|68.23|61.03|33.02|\n\n[**Q3**] Comparison with EoRA.\n\n[**R8**] Thank you for the question. We compare our method with EoRA, and the results are summarized in the table below. It can be observed that our method significantly outperforms EoRA. The experiment setup follows lines 279-281 and Table 1 in our paper.\n\n|Wanda| PPL↓ | BoolQ | RTE| Hella| Wino| ARC-e | ARC-c|OBQA|Mean|\n|--------|------|----|-------|-----|------|----|-------|------|-------|\n| EoRA |7.14|74.16|60.29| 51.27| 68.27| 70.96| 37.63| 28.6| 55.88|\n| RestoreLCC (Ours) | 6.93 | 72.84 | 69.68|56.34| 65.98|71.8|40.96| 34.2| 58.83|\n| SparseGPT  | PPL↓ | BoolQ | RTE   | Hella| Wino| ARC-e | ARC-c | OBQA | Mean|\n| EoRA | 9.87|71.68|59.93|44.65|64.33|63.72|30.38| 23.8 |51.21|\n| RestoreLCC (Ours)| 8.99 |73.61| 63.9| 51.71| 65.11|68.14|33.53|29|55|\n\n[**Q4**] The used software and other overhead.\n\n[**R9**] Thank you for your question. We report the details of recovering pruned LLaMA‑7B (1 H100 GPU, torch.bfloat16 precision, max_length=512, batch size=8, Alpaca Dataset).\n\nThe **software** we used is Python and torch. All the required packages and required versions can be found in our submitted materials: requirements.txt.\n\nThe **computation time for probing** is 8min23sec.\n\nThe **1-epoch training time for each method** is as follows. For training time, the methods rank as follows:\nRestoreLCC < LoRA < DoRA < LoFiT. For GPU memory usage: LoRA < RestoreLCC = LoFiT < DoRA. Compared with the DoRA and LoFiT, RestoreLCC has the lowest overall overhead while providing superior performance.\n\n| llama-7B|Time|Memory|\n|-------|-----|-------|\n| LoRA| 4h20min| 61GB|\n| DoRA | 6h08min| 71GB|\n| LoFiT| 8h11min| 65GB|\n| RestoreLCC| 4h13min|65GB|\n\n [**Q5**] Limitations of addressing MLPs. \n \n [**R10**] Thank you for raising this concern. **Our goal is to restore pruned models in which all weight matrices in both the attention and MLP modules have been pruned**. To achieve this, our recovery operations focus on the attention modules rather than the MLPs to restore the overall model performance.  We provide both theoretical and empirical analyses to support this choice.\n \n (1) Studies on mechanistic interpretability show that **attention heads specialize in distinct functions for different tasks [1,2,3], while FFNs mainly store knowledge and map inputs to outputs [4,5]**. The detailed discussion and references can be found in our response **R1** to Reviewer Kk22.\n \n(2) Empirical evidence. We would like to highlight that **our method can be directly applied to FFNs**. We select heads instead of FFNs for recovery also because of the empirical results. Compensating attention achieves 58.83, while compensating FFNs only reaches 57.00.\n\n|    | BoolQ| RTE| Hell. | Wino. | ARC-e | ARC-c | OBQA| Mean|\n|------|-----|-----|------|----|-----|-----|-----|------|\n| Recover attn. | 72.84 | 69.68 | 56.34| 65.98 | 71.8 | 40.96 | 34.2 | 58.83|\n| Recover FFNs | 69.42 | 63.54 | 55.54| 67.17|70.96|39.59|32.8| 57|\n\n**Thank you again for the insightful comments. We will incorporate the clarifications and discussions into the revised version and we look forward to your feedback.**"}, {"review_id": "5vCpdYcSOf", "reviewer": "Reviewer_aFYT", "summary": "This paper proposes a targeted restoration strategy for pruned models that restores performance while preserving their low cost and high efficiency. It observes that pruning-induced information loss is reflected in attention activations, and selectively reintroducing components of this information can significantly recover model performance. Based on this insight, it introduces RestoreLCC (Restoring Pruned LLMs via  Lost Component Compensation), a plug-and-play method that contrastively probes critical attention heads via activation editing, extracts lost components from activation differences, and finally injects them back into the corresponding pruned heads for compensation and recovery.", "questions": "see the weakness.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "\\+ It observes that pruning-induced information loss is reflected in attention activations, and that selectively restoring key components can significantly recover model performance.\n\n\\+ It proposes RestoreLCC, a method that learns the magnitudes of important component directions lost during pruning and reintroduces them to restore pruned models effectively\n\n\\+ Extensive experiments across various LLMs and pruning schemes demonstrate that RestoreLCC consistently outperforms existing restoration baselines.\n\n\\- The novelty of this paper may be limited. The idea to compensate the activations for the loss of parameter pruning has been explored in previous works such as [R1]. In [R1], it adds a compensation term to the activations for the loss of pruning. SVD is widely adopted to analyze the difference in pruning works such as [R2]. The technical contribution may be limited. \n\n\\- It mentions to train a probing classifier to assess the discriminative power of each recovered head activation. But it only shows some data examples for the classifier. It is better to provide more details for the training of this probing classifier. \n\n\\- The ablation study can be enhanced. It needs to use multiple samples from datasets for the compensation as discussed in Line 125. But I do not find the specific number of data samples used in the paper. It is better to provide this number. Furthermore, it does not provide the ablation study for the number of samples used in the paper. I am not sure if using more or less samples can perform better or not. Moreover, it needs to construct Contrastive Samples to train the probing classifier. I do not find the details about this contrastive sample set either, such as how many samples are used and ablation study for it. \n\n\\- It is better to provide more details for the baselines. It mainly compares with Lora methods, but I do not find much details for the lora training, for example, how many data samples are used, or how many steps are used for the training. It is better to provide the details to ensure that the comparison is fair.  \n\n\\- In the paper, it mentions that the number of components $K$ is set to 10 which leads to  58.83% mean accuracy for Llama-7B. However, in Table 5 in the appendix, it shows that 58.83 is achieved with $K=1$. This is different with the claim for $K=10$ in the main paper. It also recommend using $K\\le 3$ in Appendix B, but the main paper uses $K=10$. It is better to check this. \n\n\\- Finding 3 may have some issues. It is very weird with so many different values of K for different layers or heads. It does not seem to be a general feature. It seems that some specific K which are carefully selected by authors can lead to Figure 3. And the component are scaled by 1000. We are not sure whether 1000 is a reasonable scale number. The observation and discussion in finding 3 is heuristic and experimental. We are not sure whether it is general enough to lead to the claimed finding 3. It is better to provide more discussions.\n\n\\- In equation (8), the sum is computed over $d_h$. However, in equation (4), the sum is computed over $K$. Since their outputs refer to the same thing for the activation compensation, these two equations are not aligned and it seems to have some errors in the equations. \n\n[R1] Fluctuation-Based Adaptive Structured Pruning for Large Language Models, AAAI 2024\n \n[R2] EoRA: Training-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation", "was_revised": true, "final_justification": "Most of my concerns are addressed. I have no more questions.", "rebuttal": "Dear Reviewer aFYT,\n\nThank you for reviewing our paper. The concerns you raised mainly focus on the **novelty, clarification of our findings, and experimental details**. We have carefully addressed all of these points, and we are glad that your suggestions have helped us improve the paper. Our responses [**R**] to your raised questions [**Q**] and weaknesses [**W**] are as follows.\n\n[**W1**] Novelty clarification.\n\n[**R1**] Thank you for your question. We **respectfully disagree** and would like to emphasize that our proposed RestoreLCC is conceptually and technically distinct from R1, R2, and their combination (R1+R2). We clarify the novelty of our method from both **theoretical and empirical** perspectives as follows.\n\n(1) Unique Insight 1 – Focusing on **a few discriminative heads (ours) rather than all modules** (R1, R2 and other studies). As shown in Section 3 (Findings 1 and 2), we reveal that reintroducing lost components to **selected pruned attention heads** can significantly restore model performance. In other words, not all modules are useful. However, previous approaches that recover all attention heads and all FFNs result in lower efficiency and often limited recovery performance.\n\n(2) Unique Insight 2 – **Recovering discriminative components instead of principal components**. Another key difference from R1, R2, and similar work is that RestoreLCC focuses on discriminative components/information rather than principal components. Previous methods emphasize recovering principal components or minimizing global losses such as MSE of hidden states. In contrast, our study (Findings 3 and Figure 3) shows that discriminative components are more effective than principal components for recovery. For example, Figure 9 shows that some minor components (indices > 20), even with extremely small eigenvalues, retain high magnitudes after tuning and contribute strongly to recovery. These components may not capture principal information, but they are highly discriminative. This phenomenon parallels the difference between PCA (capturing principal variance) and LDA (capturing discriminative directions) in classical machine learning theory. This insight also explains why we train coefficients for all components rather than only the top‑k principal components like previous studies.\n\n(3) Methodological novelty – **Leveraging the above insights and address key challenges**.While these findings reveal promising opportunities, they also **raise challenges**. For example, how to identify discriminative heads? How to find discriminative components (possibly among minor components) and determine their coefficients? To tackle these issues, we design RestoreLCC, which integrates a contrastive probing module and a lost component compensation module. Although some terms may sound similar to those in previous work, the underlying mechanisms and method operations are fundamentally different and novel.\n\n(4) **Significant improvements over previous studies**.\nWe compared RestoreLCC with FLAP (R1) and EoRA (R2). As shown in the below table (same setup as lines 279–281 and Table 1), RestoreLCC outperforms EoRA by **2.95%** on Wanda-pruned models and by **3.79%** on SparseGPT-pruned models.\n\n| Wanda| PPL↓| BoolQ| RTE| Hella| Wino| ARC-e| ARC-c| OBQA| Mean|\n|-|-|--|--|---|-|--|-|--|--|\n| EoRA | 7.14 | 74.16 | 60.29 | 51.27     | 68.27 | 70.96 | 37.63 | 28.6 | 55.88 |\n| RestoreLCC| 6.93 | 72.84 | 69.68 | 56.34  | 65.98 | 71.8| 40.96 | 34.2 | 58.83 |\n| SparseGPT| PPL↓| BoolQ | RTE   | Hella| Wino|ARC-e | ARC-c | OBQA | Mean  |\n| FLAP|10.57| 68.81 | 54.15 | 44.48 | 64.4| 63.97 | 30.03 | 24 | 49.98 |\n| EoRA|9.87| 71.68 | 59.93 | 44.65  | 64.33| 63.72 | 30.38 | 23.8 | 51.21 |\n| RestoreLCC | 8.99 | 73.61 | 63.9|51.71| 65.11| 68.14 | 33.53 | 29 | 55  |\n\n[**W2**] More details of probing classifiers.\n\n[**R2**] Thank you for the question. We apologize for not providing enough details about the probing classifiers. The procedure is described in Section 4.1, and we will add the following clarification for the classifiers:\n\nSpecifically, for each classifier, we use the formulation: $y = \\sigma(Wm + b)$, where $m$ is either $m^+$ or $m^-$, $W$ is a weight matrix (input dimension equal to the size of $m$, output dimension = 1), $b$ is a bias term, and $\\sigma$ is the sigmoid function. The output $y \\in [0,1]$ indicates contradiction or entailment. We use a cross‑entropy loss optimized with Adam (lr = 1e‑2). The dataset is split into training and validation sets with a 7:3 ratio, and models are trained for 100 epochs based on empirical experience. We report probing accuracy on the validation set and **rank the importance of attention heads according to their probing accuracy**.\n\n[**W3**] Ablation study: number and effect of samples.\n\n[**R3**] We apologize for not providing enough details.\n\nFor all experiments, we use the same sample sizes: general recovery – 1,000 Alpaca samples for probing and the full 52K for tuning; task‑specific recovery – 100 samples for both probing and tuning. The number of contrastive pairs equals the number of probing samples.\n\nSample size effect: for general recovery, 1,000 samples are sufficient (accuracy ≈ 58.8); for task‑specific recovery on BoolQ, 200 samples can be sufficient (accuracy≈ 76).\n\n|General Recovery|100|200|500|1000 (reported) |3000|   |   |\n|------|-------|-------|----|---|--|------|------|\n| Mean Accuracy| 57.63 | 58.14 | 58.69 | 58.83  | 58.81 | |    |\n| BoolQ recovery| 10 | 20| 50| 100 (reported)| 200| 500 |1000|\n| Acc.| 70.03 |73.09|74.98|75.32|76.15|76.12|76.51|\n\n[**W4**] Details for baselines.\n\n[**R4**] We apologize for not providing these details. For LoRA and DoRA, we use the same settings: α = 16 and rank = 8. Regarding the applied modules, we try two configurations: (1) [v_proj, o_proj], which tunes only the head output matrices; and (2)[q_proj, k_proj, v_proj, o_proj], which tunes all head matrices. The best results are reported in the paper.\n\nFor LoFiT, we experiment with 10%, 20%, and 30% of the heads and report the best results.\n\nThe training samples are same for all methods. The batch size is set to 8. The learning rate is 1e-4. The max sequence length is 512. The number of training epochs is 2 for global recovery and 5 for task-specific recovery. All experiments are conducted on a single H100 GPU.\n\n[**W5**] CLarification on K.\n\n[**R5**] We apologize for the oversight. The setting of $K$ is clarified as follows, and we will double‑check similar details to ensure correctness.\n\nIn our experiments, we set $K = 1$ (not 10) to obtain the reported results.\nWe recommend using $K \\leq 3$ because performance is consistently better in this range than with $K > 3$. Moreover, after the top 3 components, eigenvalues become very small, so additional components have negligible impact on the final results while increasing computation. Therefore, using $K \\leq 3$ provides both better performance and efficiency.\n\n[**W6**] Clarification on Finding 3.\n\n[**R6**] Thank you for the question. We clarify Finding 3 as follows.\n\nUnlike previous studies that assume principal components are always useful for recovery, we present a different insight: **discriminative information can also reside in minor components rather than principal components**. Principal components may capture high-variance but task‑irrelevant information, while minor components may encode task‑specific signals. This distinction is analogous to PCA vs. LDA in classical machine learning.\n\nTo verify this, we manually inspected the token distributions of each component (Eq. 6). For example, in BoolQ, the top tokens of some principal components (e.g., 1st or 2nd) may include stopwords like *a*, *the*, or unrelated words like *walk*. In contrast, **some minor components have answer tokens like *yes* and *no*, which are highly relevant to the task**. Such task‑specific minor components are more discriminative and recover performance more effectively. We further validated this hypothesis by manually selecting attention heads and comparing recovery using principal vs. discriminative components. Figure 3 illustrates this phenomenon. Figure 9 also validates this: **the trained magnitudes of minor components can become larger than those of principal components due to their discriminative power**.\n\nScaling factor: Eigenvalues of principal components can be more than 1000× larger than those of minor components. To ensure fair comparison, we apply a scaling factor (1000) when adding minor components so their contributions are comparable.\n\nIn summary, the key finding is that task‑specific discriminative information may reside in minor components, not just in principal components. This observation directly motivates RestoreLCC, whose goal is to leverage these discriminative components for effective recovery.\n\n[**W7**] Clarification on Eq. 4 and Eq.8.\n\n[**R7**] Thank you for the question. We would like to clarify that there is no error in either equation. In Eq. 4, the top‑k components are used only for probing (Section 4.1), whereas in Eq. 8 all components are used for recovery tuning.\n\nThis is because, in the first step, we use only the top‑k components to evaluate the discriminative power of each head, as minor components have extremely small eigenvalues and barely affect probing results. However, as highlighted in Finding 3, during recovery for the selected heads, **important discriminative information may also lie in minor components rather than only in the top‑k ones**.\n\nTherefore, our method considers all components and learns a magnitude for each component (total of $d_h$) so that even minor but discriminative components can receive a large weight for recovery. Figure 9 also confirms this behavior. Therefore, in Eq. 8 it should indeed be $d_h$.\n\n**Thank you once again for reviewing our paper. We will incorporate the clarifications and discussions into the revised version. Please let us know if you have any further feedback, and we look forward to your reply.**"}]

Author rebuttal:

Meta-review: (a) Summary 

This paper introduces RestoreLCC, a novel and targeted restoration strategy for pruned large language models (LLMs). Unlike prior parameter-efficient fine-tuning (PEFT) methods that overlook the unique properties of pruned models, RestoreLCC explicitly identifies and compensates for the lost information caused by pruning. The authors perform an empirical study to gain insight into how leveraging the information in lost activations can be used to restore quality. Using these insights, RestoreLCC injects the lost information back into the model via a learned bias term in a three step process: (1) contrastive probing identifies the important attention heads, (2) Lost Component Compensation (LCC) performs SVD on the difference between the outputs of the important dense and pruned heads and learns optimal magnitudes for the top components before (3) augmenting the important heads with these learned bias terms, "re-injecting" the lost information back into the model. Empirical studies show improvements on most metrics compared to baselines.
Extensive experiments across multiple pruning schemes (structured, semi-structured, unstructured) and LLM variants demonstrate that RestoreLCC achieves superior accuracy and perplexity recovery without compromising sparsity or inference efficiency.

(b) Strengths 

The paper introduces a unique insight—model degradation from pruning is traceable to lost activation components—which leads to a targeted and explainable restoration method. The work addresses a critical bottleneck in deploying sparse LLMs by offering a low-overhead, generalizable recovery mechanism. It shows consistent gains over strong PEFT baselines like LoRA, DoRA, and LoFiT.

The paper observes that pruning-induced information loss is reflected in attention activations, and that selectively restoring key components can significantly recover model performance.

The methodology is well-justified with theoretical and empirical analyses. Ablation studies, probing diagnostics, and interpretability results are thorough and convincing. The method is validated under different pruning settings.

The overall exposition is clear, well-structured, and supported with informative figures (e.g., logit gain visualizations, component effects).

The solution's low-overhead is particularly compelling; additional bias terms are a very small tax on model size and deployment efficiency, in contrast to the LoRA family of techniques, which require additional (though relatively small) weight matrices. The contrastive probing and compensation process is adequately described such that an expert could reproduce the results. The evaluations are sufficiently broad with respect to models and tasks to inspire confidence that there are no surprises hidden behind under-explored areas. This particular solution to the problem incorporates existing ideas, but is largely original.

(c) Weaknesses 

All compared PEFT methods are limited to very small trainable parameter settings. I believe this setup is favorable to the method. However, it is clear that the training overhead of PEFT is not sensitive to the number of trainable parameters; for example, the overhead difference between rank=1 and rank=16 is minimal. I think that in the recovery fine-tuning stage, performance recovery should be the main focus, and a higher rank may yield better performance with only negligible increases in fine-tuning overhead.

The contrastive probing module depends on data/model-specific classifier training, which could affect practical deployment and cross-task portability.

Clarity is a mixed bag - while the exposition is clear, some details are not. For example, the precise sparsity imposed is unclear. At times the activations are pruned (line 113), but later it is the attention heads (I assume weights) that are pruned (line 132). It's also not clear which weights are pruned; is it only those in the attention blocks Q, K, V and dense output projections, or also weights in the MLP blocks (which can be much larger due to the expansion factor)? This has direct bearing on the significance of the submission. Also, consider avoiding the term "performance;" it is overloaded and can mean both model quality as well as execution speed or efficiency. In this context, where both meanings are plausible, it's better just to be explicit.

(d) Most important reasons for decision to accept/reject. 

The novel and effective approach to restoring performance after network pruning, and the admirable job the authors did in responding to the reviewers' critiques.

(e) Summary of the discussion and changes during the rebuttal period. 

The authors provided extensive and convincing rebuttal text, with additional experiments as requested by the reviewers. All reviewers raised their scores after the rebuttal period.

Reviewer n2VC stated that the authors provided an exceptionally strong and thorough rebuttal that has fully addressed all of their initial concerns, with no remaining unresolved issues. The new experiments and clarifications have significantly strengthened the paper's claims and overall contribution.
The authors added extensive experiments on entirely different and recent architectures (Qwen3, DeepSeek-R1-Qwen3), demonstrating that RestoreLCC consistently outperforms baselines. This provides compelling evidence for the method's broader applicability.

Questions about the practical deployment, specifically the portability of the probing module and the method's efficiency at scale, were also fully addressed with new, targeted experiments. The authors demonstrated tsuccessful cross-task performance recovery without retraining the probing module, directly answering the scalability question, and showed effective performance on a much larger 70B model (LLaMA-70B), proving scalability. A concrete latency benchmark was provided showing a negligible inference overhead of ~3%, confirming real-world efficiency.

The weight of these resolved issues is high, as they were central to validating the paper's core claims of being a generalizable and practical method. The authors' diligence in running these new experiments has transformed the paper from a promising one with limitations to a robust and well-supported contribution.

Reviewer n2VC stated "Your rebuttal was exemplary. The new experiments were precisely what was needed to validate your claims and were highly persuasive. The paper is now substantially stronger, with robust evidence backing the generalizability, cross-task portability, and efficiency of RestoreLCC."
Decision: Accept (spotlight)