Title: Restoring Pruned Large Language Models via Lost Component Compensation
Venue: NeurIPS 2025
Accepted: yes
Keywords: large language models

pruning

performance restoration
Abstract: Pruning is a widely used technique to reduce the size and inference cost of large language models (LLMs), but it often causes performance degradation. To mitigate this, existing restoration methods typically employ parameter-efficient fine-tuning (PEFT), such as LoRA, to recover the pruned model's performance. However, most PEFT methods are designed for dense models and overlook the distinct properties of pruned models, often resulting in suboptimal recovery. In this work, we propose a targeted restoration strategy for pruned models that restores performance while preserving their low cost and high efficiency. We observe that pruning-induced information loss is reflected in attention activations, and selectively reintroducing components of this information can significantly recover model performance. Based on this insight, we introduce RestoreLCC (Restoring Pruned LLMs via Lost Component Compensation), a plug-and-play method that contrastively probes critical attention heads via activation editing, extracts lost components from activation differences, and finally injects them back into the corresponding pruned heads for compensation and recovery. RestoreLCC is compatible with structured, semi-structured, and unstructured pruning schemes. Extensive experiments demonstrate that RestoreLCC consistently outperforms state-of-the-art baselines in both general and task-specific performance recovery, without compromising the sparsity or inference efficiency of pruned models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a targeted restoration strategy for pruned models that restores performance while preserving their low cost and high efficiency. It observes that pruning-induced information loss is reflected in attention activations, and selectively reintroducing components of this information can significantly recover model performance. Based on this insight, it introduces RestoreLCC (Restoring Pruned LLMs via  Lost Component Compensation), a plug-and-play method that contrastively probes critical attention heads via activation editing, extracts lost components from activation differences, and finally injects them back into the corresponding pruned heads for compensation and recovery.\n\nSTRENGTHS AND WEAKNESSES:\n\\+ It observes that pruning-induced information loss is reflected in attention activations, and that selectively restoring key components can significantly recover model performance.\n\n\\+ It proposes RestoreLCC, a method that learns the magnitudes of important component directions lost during pruning and reintroduces them to restore pruned models effectively\n\n\\+ Extensive experiments across various LLMs and pruning schemes demonstrate that RestoreLCC consistently outperforms existing restoration baselines.\n\n\\- The novelty of this paper may be limited. The idea to compensate the activations for the loss of parameter pruning has been explored in previous works such as [R1]. In [R1], it adds a compensation term to the activations for the loss of pruning. SVD is widely adopted to analyze the difference in pruning works such as [R2]. The technical contribution may be limited. \n\n\\- It mentions to train a probing classifier to assess the discriminative power of each recovered head activation. But it only shows some data examples for the classifier. It is better to provide more details for the training of this probing classifier. \n\n\\- The ablation study can be enhanced. It needs to use multiple samples from datasets for the compensation as discussed in Line 125. But I do not find the specific number of data samples used in the paper. It is better to provide this number. Furthermore, it does not provide the ablation study for the number of samples used in the paper. I am not sure if using more or less samples can perform better or not. Moreover, it needs to construct Contrastive Samples to train the probing classifier. I do not find the details about this contrastive sample set either, such as how many samples are used and ablation study for it. \n\n\\- It is better to provide more details for the baselines. It mainly compares with Lora methods, but I do not find much details for the lora training, for example, how many data samples are used, or how many steps are used for the training. It is better to provide the details to ensure that the comparison is fair.  \n\n\\- In the paper, it mentions that the number of components $K$ is set to 10 which leads to  58.83% mean accuracy for Llama-7B. However, in Table 5 in the appendix, it shows that 58.83 is achieved with $K=1$. This is different with the claim for $K=10$ in the main paper. It also recommend using $K\\le 3$ in Appendix B, but the main paper uses $K=10$. It is better to check this. \n\n\\- Finding 3 may have some issues. It is very weird with so many different values of K for different layers or heads. It does not seem to be a general feature. It seems that some specific K which are carefully selected by authors can lead to Figure 3. And the component are scaled by 1000. We are not sure whether 1000 is a reasonable scale number. The observation and discussion in finding 3 is heuristic and experimental. We are not sure whether it is general enough to lead to the claimed finding 3. It is better to provide more discussions.\n\n\\- In equation (8), the sum is computed over $d_h$. However, in equation (4), the sum is computed over $K$. Since their outputs refer to the same thing for the activation compensation, these two equations are not aligned and it seems to have some errors in the equations. \n\n[R1] Fluctuation-Based Adaptive Structured Pruning for Large Language Models, AAAI 2024\n \n[R2] EoRA: Training-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation\n\nQUESTIONS:\nsee the weakness.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nNetwork pruning reduces the complexity of a trained model, enabling more efficient deployment, but at the cost of potentially reduced quality.  Existing approaches to efficiently recovering quality (i.e., excluding fine-tuning the full network) do not consider the pruning that occurred when performing recovery.  The authors perform an empirical study to gain insight into how leveraging the information in *lost* activations can be used to restore quality.  Using these insights, RestoreLCC injects the lost information back into the model via a learned bias term in a three step process: (1) contrastive probing identifies the important attention heads, (2) Lost Component Compensation (LCC) performs SVD on the difference between the outputs of the important dense and pruned heads and learns optimal magnitudes for the top components before (3) augmenting the important heads with these learned bias terms, \"re-injecting\" the lost information back into the model.  Empirical studies show improvements on most metrics compared to baselines.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\nThe problem statement is well-motivated, and the approach seems novel, is theoretically-grounded, and is generally successful.  The solution's low-overhead is particularly compelling; additional bias terms are a very small tax on model size and deployment efficiency, in contrast to the LoRA family of techniques, which require additional (though relatively small) weight matrices.  The contrastive probing and compensation process is adequately described such that an expert could reproduce the results.  The evaluations are sufficiently broad with respect to models and tasks to inspire confidence that there are no surprises hidden behind under-explored areas.  This particular solution to the problem incorporates existing ideas, but is largely original.\n\t\n## Weaknesses\nClarity is a mixed bag - while the exposition is clear, some details are not.  For example, the precise sparsity imposed is unclear.  At times the activations are pruned (line 113), but later it is the attention heads (I assume weights) that are pruned (line 132).  It's also not clear which weights are pruned; is it only those in the attention blocks Q, K, V and dense output projections, or also weights in the MLP blocks (which can be much larger due to the expansion factor)?  This has direct bearing on the significance of the submission.  Also, consider avoiding the term \"performance;\" it is overloaded and can mean both model quality as well as execution speed or efficiency.  In this context, where both meanings are plausible, it's better just to be explicit.\n\nQuality is also slightly lacking.  \n- The inference speed overheads in Appendix C isn't very useful without knowing the deployment conditions: hardware, software, type of pruning, batch sizes, sequence lengths, etc. all play a large role in both dense and sparse efficiency.  Further, the magnitude of the speedups will influence how important the overheads are.\n- The ablation study comparing identifying important heads using contrastive probing vs. randomly-selected heads.  Surely there's a simpler metric that could perform better than random choice: MSE between the output of dense and pruned heads, K-L Divergence, etc.\n- Figure 5 shows the top-5 decoded tokens, but the text suggests it shows the top-10 (line 347)\n- There are two references that should be considered for discussion and inclusion:\n    - Moore and Chaudhuri probe networks' activations in order to drive pruning decisions directly.  \n    - Liu et al. do take the pruning into account when restoring quality with low-rank adaptation.  This work, in particular, should be included as a baseline, since it stands in contrast to, as claimed in the abstract, \"most PEFT methods are designed for dense models and overlook the distinct properties of pruned models.\"\n\n\n1. Moore and Chaudhuri, Using noise to probe recurrent neural network structure and prune synapses, NeurIPS 2020\n2. Liu et al., EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation, https://arxiv.org/abs/2410.21271\n\nQUESTIONS:\n1. What weight tensors are pruned in your experiments?  My understanding of the submission is that it's only Q, K, V, and dense output projections in attention blocks, not MLP blocks. (Clarity, Significance)\n2. Is there anything fundamentally stopping RestoreLLC from being used on heavily quantized models, or does it only apply to pruned models? (Significance)\n3. How does EoRA compare for some model, given the same sparsities and tasks? (Quality, Significance)\n4. What software was used, and how much computation time was required to probe and learn the compensatory terms for each model? (Quality)\n\nLIMITATIONS:\nThis approach seems to be limited to attention weights, and cannot address MLPs.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces RestoreLCC, a novel and targeted restoration strategy for pruned large language models (LLMs). Unlike prior parameter-efficient fine-tuning (PEFT) methods that overlook the unique properties of pruned models, RestoreLCC explicitly identifies and compensates for the lost information caused by pruning. The method consists of two key components: (1) Contrastive Probing, which identifies important attention heads via contrastive activation editing; and (2) Lost Component Compensation (LCC), which reconstructs missing activation components using learned linear combinations of principal directions and injects them back to recover the model’s capability. Extensive experiments across multiple pruning schemes (structured, semi-structured, unstructured) and LLM variants demonstrate that RestoreLCC achieves superior accuracy and perplexity recovery without compromising sparsity or inference efficiency.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The paper introduces a unique insight—model degradation from pruning is traceable to lost activation components—which leads to a targeted and explainable restoration method.\n\n- The methodology is well-justified with theoretical and empirical analyses. Ablation studies, probing diagnostics, and interpretability results are thorough and convincing.\n\n- The overall exposition is clear, well-structured, and supported with informative figures (e.g., logit gain visualizations, component effects).\n\n- The work addresses a critical bottleneck in deploying sparse LLMs by offering a low-overhead, generalizable recovery mechanism. It shows consistent gains over strong PEFT baselines like LoRA, DoRA, and LoFiT.\n\n### Weaknesses\n\n- The scope of evaluation is currently limited to LLaMA-family models. Including more diverse architectures (e.g., Mistral, Qwen, Baichuan) would strengthen the claim of generalizability.\n\n- The contrastive probing module depends on data/model-specific classifier training, which could affect practical deployment and cross-task portability.\n\n- A few sections (e.g., lines 38–59) in the main text are densely written, with overlapping figure references, which slightly affects clarity.\n\nQUESTIONS:\n**Scope of Probing Generalization**: Given that contrastive probing is data/model/pruning-specific, how scalable is RestoreLCC across multiple downstream tasks without retraining the probing module?\n\n**Efficiency at Scale**: While parameter overhead is small, does RestoreLCC scale gracefully with larger models (e.g., >30B)? Are there latency/throughput benchmarks in real-world scenarios?\n\nLIMITATIONS:\nYes, but not enough.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes RestoreLCC, a parameter-efficient fine-tuning method designed for post-pruning model recovery. Under several different pruning settings, RestoreLCC demonstrates superior recovery performance compared to directly applying other PEFT methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. An approach similar to distillation is used to preserve errors in the vectors, which reduces the attention information loss after pruning and thus mitigates the performance degradation caused by pruning.\n\n2. The method is validated under different pruning settings.\n\nWeaknesses\n\n1. The underlying rationale behind the effectiveness of the proposed method seems to lack a deeper explanation. I am curious why the information compensation is specifically applied at the attention module, rather than after the attention or at the intermediate dimensions of the FFN, for example.\n\n2. The method appears to require using a large number of intermediate activations during recovery fine-tuning in order to compute the error vectors. I think this seems to deviate from the original purpose of using PEFT for recovery fine-tuning. Could you provide a comparison of the training overhead of different methods during the recovery fine-tuning stage? In my opinion, the number of trainable parameters is not the main concern in this stage.\n\n3. All compared PEFT methods are limited to very small trainable parameter settings. I believe this setup is favorable to the method. However, it is clear that the training overhead of PEFT is not sensitive to the number of trainable parameters; for example, the overhead difference between rank=1 and rank=16 is minimal. I think that in the recovery fine-tuning stage, performance recovery should be the main focus, and a higher rank may yield better performance with only negligible increases in fine-tuning overhead.\n\nQUESTIONS:\n1. Could you explain why, in Figure 1, the accuracy on the right increases beyond that of the dense model after applying LCC?\n\n2. This paper seems to lack a comparison with full-parameter fine-tuning for performance recovery.\n\n3. I would like to know the effect of applying the method to the FFN layers.\n\n4. Please clarify where the PEFT methods are added in the model and provide other relevant implementation details (e.g., rank, alpha, learning rate...).\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}