Title: Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samples
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLMs

adapting language models

efficient search

no fine-tuning
Abstract: Recently, Sharma et al. (2024) suggested a method called LAyer- SElective-Rank reduction (LASER) which demonstrated that pruning high‑order components of carefully chosen LLM’s weight matrices can boost downstream accuracy—without any gradient‑based fine‑tuning. Yet LASER’s exhaustive, per‑matrix search (each requiring full‑dataset forward passes) makes it impractical for rapid deployment.  We demonstrate that this overhead can be removed and find that: (i) Only a small, carefully chosen subset of matrices needs to be inspected—eliminating the layer‑by‑layer sweep, (ii) The gradient of each matrix’s singular values pinpoints which matrices merit reduction, (iii) Increasing the factorization search space by allowing matrices rows to cluster around multiple subspaces and then decomposing each cluster separately further reduces overfitting on the original training data and further lifts accuracy by up to 24.6 percentage points, and finally, (iv) we discover that evaluating on just 100 samples rather than the full training data—both for computing the indicative gradients and for measuring the final accuracy—suffices to further reduce the search time; we explain that as adaptation to downstream tasks is dominated by prompting style, not dataset size.  As a results, we show that combining these findings yields a fast and robust adaptation algorithm for downstream tasks. Overall, with a single gradient step on 100 examples and a quick scan of the top candidate layers and factorization techniques, we can adapt LLMs to new datasets—entirely without fine‑tuning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper builds on a pruning technique called LAyer SElective Rank reduction (LASER).\nLASER improves the performance of a model on downstream task by replacing its weight matrices with their SVD approximations with reduced ranks.  LASER removes higher-rank components which correspond to smaller singular values. However, it requires a search for the reduction rate on the rank to prevent performance drop, which is not efficient.\n\nThis paper improves the efficiency of LASER by requiring less number of samples than LASER. Additionally, it provides a more reliable approximation than LASER by using multiple SVD projections per layer. These SVD projections are selected based on score vectors, which are calculated by replacing the sigma matrix in SVD with the gradients of loss with respect to the singular values. \n\nThe proposed method show comparable performance to LASER with certain speedups on GPT-J and RoBERTa. The tradeoff is different for different scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nI can list some of the strengths of the proposed approach as follows:\n- requires a small dataset for adapting models to downstream tasks\n- does not require iteratively updating model weights or adapter weights\n- shows comparable/improved performance to LASER with faster runtime. For instance, Clustering LASER 100 Grads 100 Eval leads to:\n\t- 0.95% improvement in accuracy and 52x speedups in runtime for GPT-J (based on Table 1)\n\t- 0.40% drop in accuracy and 22.2x speedups in runtime for RoBERTa (based on Table 2)\n\nThere are still certain aspects of the paper that can be improved:\n- It wasn't easy to follow different variants of the method. \n\t- This can be improved by including more details in table captions (e.g., describe which configuration a method corresponds to)\n\t- Figure 1 can be explained more. It is not clear what is happening in the Clustering LASER block. It is also not clear how the '100 Evaluation' are used to update the weight matrices. (I understand some of it wouldn't be possible due to the space but would help if you could incorporate more details from Algorithm 1). Perhaps you could also include LASER in the figure and point out the differences between LASER and Clustering LASER.\n- 100 seems like a magic number. More validation is needed to make the paper more convincing (please see my response in questions).\n\nQUESTIONS:\nYou mention that the proposed method is training-free but this is not entirely correct as it still relies on gradients of the loss with respect to the singular values. This part may benefit from clarification. \n\n100 seems like a magic number. Have you done any experiments on why 100 is a good choice? What happens for different number of samples? If you could show these results, that would add more support for the choice of 100. \n\nI think writing needs to be improved too (see the details in weaknesses). \n\nI would be happy to increase my scores (quality for the issues about the number of samples and clarity for the issues regarding the writing) if these changes are done. I would also increase my rating to Borderline accept.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

comments: SUMMARY:
The   paper gives you three ways to improve the computation of the LASER method of adapting an LLM to a downstream task. They compute the gradients of the eigenvalues of weight matrices to identify whether they need updation, in place of a per layer sweep. That is done on a 100 examples, both  for that gradient detemination and for validating performance.The paper proposes a subspace factorization method by clustering multiple rows around multiple subspaces, to generalize to a multitude of weights being used. 
The method is able to improve computation by an average of over 50x with multiple benchmarks and two foundation LMs.

STRENGTHS AND WEAKNESSES:
The way to prune weight matrix under condition    avoids an exhaustive search. 

The clustering uses a heuristic, which at times may not be optimal. A proof is not presented.

QUESTIONS:
Could you  please elaborate on the multiple prompting styles, it being apparently that very significant?

The 'quickscan' proposed is hard to read, being split in two places. Will it be a good idea to bring it upward[

LIMITATIONS:
There are a fair bit of experimental knobs to turn, that, despite the experiments given,  might hinder the  prevalence of the method as a whole in practive.
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method for adapting large language models (LLMs) to downstream tasks without gradient-based fine-tuning. Building upon LASER, the authors introduce a more efficient approach that involves pruning model weights to low-rank structures using gradient-guided matrix selection, sample-efficient evaluation, and multi-subspace factorization. Across various language tasks, the proposed method achieves significant speedups compared to LASER, while maintaining comparable or even improved performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Novel and simple approach: The method is conceptually straightforward and easy to implement. The proposed selection rule leverages the gradient of the downstream loss with respect to the model’s singular values, capturing task-specific preferences for different rank-1 components. This method is both computationally efficient and intuitively appealing.\n\n- Significant efficiency gains: Experimental results demonstrate substantial improvements in adaptation speed while generally preserving or improving task performance.\n\n- Comprehensive ablation studies: The paper includes thorough ablation studies on the key components of the method, clearly illustrating their individual contributions to overall performance.\n\nWeaknesses:\n\n- Marginal or inconsistent accuracy gains: Although the proposed method improves efficiency, its performance gains over LASER in terms of accuracy are minor. In some tasks, accuracy even drops noticeably. Given that LASER is already a training-free method, the lack of consistent improvements may weaken the practical impact of this work.\n\n- Insufficient hyperparameter analysis: The paper lacks a detailed study of sensitivity to key hyperparameters, such as the number of clusters and the compression ratio. Including such analysis would enhance the robustness and usability of the method. Table 4 only shows the result after searching.\n\nQUESTIONS:\n- Honestly, I am a bit confused on the detailed implementation of the algorithm. Does Algorithm 1 describe the complete end-to-end process of the proposed method? If so, it seems that the dataset $\\mathcal{D}$ is only used for identifying which weights to adapt. Could the authors clarify what exactly is meant by the “evaluation” step in this context?\n\n- In Algorithm 1, weight selection is guided by gradients of singular values, yet the compression step appears to prune based on the singular values themselves. Is there a potential mismatch here? Specifically, could large-magnitude negative gradients arise from large singular values that are not pruned, thus making the gradient-based signal inconsistent with the pruning criterion?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
Layer selective rank reduction is an efficient way to boost downstream performance without any finetuning. The authors aim to further improve the efficiency by eliminating the per-matrix search, where each requires a full forward pass of the data set. By observing that the sign of the gradients of the singular values indicates its usefulness to the downstream task, the authors propose to use this as the metric to decide which matrices to be pruned. Additionally, matrix row clustering is proposed to further improve the performance of the method. The empirical result shows that the proposed method can effectively adapt to the downstream tasks with as little as 100 samples.

STRENGTHS AND WEAKNESSES:
This paper studies a novel and interesting research direction. The proposed method is based on an insightful observation, while being a bit simplistic and there seems to be more things to be explored/discussed in the proposed design space.

QUESTIONS:
The summation of the last twenty singular values seem a bit arbitrary and might be influenced by the model's dimensionality. What other approaches has the authors considered and how is the performance? Additionally, in a transformer block there are different matrices with different dimensionality and purposes. Did the authors consider ranking matrices of each kind separately instead of ranking them jointly?

LIMITATIONS:
The authors discuss the limitation on the experiments, while having little discussion on the limitation of the method itself.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-08