Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samples
Authors: Shiva Sreeram
Alaa Maalouf
Pratyusha Sharma
Daniela Rus
Venue: NeurIPS 2025 spotlight ()
Keywords: LLMs
adapting language models
efficient search
no fine-tuning
Primary area: deep_learning
TL;DR: We provide a set of findings on how to adapt large language models efficiently without fine-tuning.

Abstract: Recently, Sharma et al. (2024) suggested a method called LAyer- SElective-Rank reduction (LASER) which demonstrated that pruning high‑order components of carefully chosen LLM’s weight matrices can boost downstream accuracy—without any gradient‑based fine‑tuning. Yet LASER’s exhaustive, per‑matrix search (each requiring full‑dataset forward passes) makes it impractical for rapid deployment.  We demonstrate that this overhead can be removed and find that: (i) Only a small, carefully chosen subset of matrices needs to be inspected—eliminating the layer‑by‑layer sweep, (ii) The gradient of each matrix’s singular values pinpoints which matrices merit reduction, (iii) Increasing the factorization search space by allowing matrices rows to cluster around multiple subspaces and then decomposing each cluster separately further reduces overfitting on the original training data and further lifts accuracy by up to 24.6 percentage points, and finally, (iv) we discover that evaluating on just 100 samples rather than the full training data—both for computing the indicative gradients and for measuring the final accuracy—suffices to further reduce the search time; we explain that as adaptation to downstream tasks is dominated by prompting style, not dataset size.  As a results, we show that combining these findings yields a fast and robust adaptation algorithm for downstream tasks. Overall, with a single gradient step on 100 examples and a quick scan of the top candidate layers and factorization techniques, we can adapt LLMs to new datasets—entirely without fine‑tuning.

Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samples
Shiva Sreeram Alaa Maalouf
Pratyusha Sharma'
Daniela Rus!
MIT CSAIL
University Haifa
Abstract
R SSharma et al. suggested method called LAyer- SElective-Rate reduction (LASER) which demonstrated that pruning high-order components of carefully chosen LLM's weight matrices can boost downstream accuracy-witout any gradient-based fine-tuning. Yet LASER's exhaustive, per-matrix search (each requiring full-datasct forward passes) makes it impractical for rapid deployment. We demonstrate that this overhead can be removed and find that: (i) Only a the
small, carefully chosen subset of matrices needs to be inspected-climinatie layer-by-layer sweep, (ii) The gradient of each matrix's 's singular values pinpoints which matrices merit reduction, (iii) Increasing the factorization search space by allowing matrices rows to cluster around multiple subspaces and then decomposing each cluster separately further reduces overfitting the original training data and further lifts accuracy by up to 24.6 percentage points, and finally, (iv) we discover that evaluating on just 100 samples rather than the full training data-both for computing the indicative gradients and for measuring the final accuracy-suffice downstream
further reduce the search time; explain that as adaptation tasks is dominated by prompting style, not dataset size. As a result, we show that combining these findings yields a fast and robust adaptation algorithm for downstream tasks. Overall, with single gradient step on 100 examples and a quick scan of the top candidate layers and factorization techniques, we can adapt LLMs to new datasets entirely without fine-tuning.
Introduction
Transformer-bast based large language models (LLMs) have rapidly become the backbone of modern natural-lang natural-language systems, scaling from hundreds millions to billions or trillions of parameters and achieving remarkable zero-shot and few-shot performance across wide range of tasks Brownn et al.] [2020), Touvron et [002). Despite their success, adapting these models to domain-specific data remains expensive: standard fine-tuning requires back-propagation through all parameters, large GPU clusters, and hundreds of gradient steps. Even parameter-efficiee methods such as LoRA Hu al. 2022 and prompt-tuning Lester al. [2021] still incur non-negligible compute and storage overhead when multiple tasks or domains must be supported simultaneously.
A complementary approach is to explore post-training interventions that modify a pretrained model without gradient- optimization. Lately, LAyer- SEleotive- Rankk Rank reduction (LASER) of Sharma et al. provided striking result: simply pruning higher-order components of carefully chosen weight matrices can increase downstream accuracy- no additional data, optimizers, or training epochs required. Unfortunately, LASER's exhaustive, per-matrix search demands a large-dataset forward pass per matrix in every layer, making it impractical for rapid deployment or on-device adaptation.
Our contribution. In this paper revisit LASER through the lens of efficiency. Our key insight is that the matrices most responsible for task adaptation can be identified without an exhaustive sweep. By computing the gradient of each matrix's 'ss singular values, without ever updating model weights, on a small validation subset (100 data points only each computed just once) and allowing rows
Cooreepondeenc:: sasreera@mit. edu
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Cradlem Dhan Search
Chawring LADER
"IEE Enkandor
Our Tancha
Rise Gender
W
les
M!
Kcatle
Forumaar:
Figure I: Efficient LLM adaptation. We present a method to adapt LLMs to new styles/domains without fine-tuning. (1) With single gradient step on the target data, we compute gradients of the singular values across all weight matrices; these gradients rank which matrices merit low-rank compression to curb overfitting and align to the new style. (2) We broaden the search by clustering the rows of the selected matrices into multiple (best fitting) subspaces and factor each cluster, capturing heterogeneous structure and reducing noise/overfitting that manifest differently across row groups. (3) We show that both the gradient scoring and evaluation can be done with just 100 examples. (4) Finally, this yields up 52x speedups and up to +24.6-240000 24..nt accuracy gains fine-tuning required.
to be decomposed around multiple subspaces, we both narrow the layers search space and unlock richer factorizations that further reduce overfitting Guided by these observations, we develop a fast, sample-efficien adaptation algorithm that needs only single gradient step roughly 100 examples and quick scan of a handful of candidate matrices.
Our contributions offers three key findings and a complete algorithm they enable:
1. Gradient-guided matrix selection. We show that the gradient of singular values reliably pinpoints which weight matrices merit reduction, eliminating the layer-by-layer sweep required by LASER.
2. Sample-efficient evaluation. We demonstrate that 100 labeled examples suffice for both gradient estimation and accuracy checks, i.c., can be used as the full given training data, indicating that adaptation quality is dominated by prompting style rather than dataset size.
3. Multi-subspace factorization. Clustering matrix rows into several subspaces and then performing rank reduction within each cluster enlarges the factorization search space and thus further mitigates overfitting, raising benchmark accuracy by up to 24.6 percentage points.
4. Adapting LLMs. Taken together, these findings yield a lightweight, training-free free pipeline for adapting pretrained LLMs to new domains efficiently single GPU.
In short, with just one gradient step 100 examples-and and a rapid check of the most promising layers and factorization schemes-L.MMe can be adapted to new datasets without any fine-tuning. We hope these findings and this approach broadens the practical reach of LLMs particularly in settings where compute, bandwidth, or labeled data are scarce.
2 Related work
To our knowledge, Sharma et al. were the first to show that largeted rank-reduction of weight matrices can improve LLMs accuracy on Nownstream-taske Nevertheless, three established research streams highly relevant:() how large language models internally encode factual knowledge, (ii) how over prrmttaramttrii networks can be compressed without sacrificing performance, and (iii) how to adapt LLMs to down stream tasks.
How facts are stored. Early probing work [Ettinger et al. 2016 Adi al. 2016 Hupkes et al., 2018
Conneau et al.) 2018 suggests that factual attributes are distributed across layers. One influential hypothesis posits that entity-specific information is cached in two-layer key-value memories inside MLP blocks (Geva et al., [021] and then propagated forward by self-attention (Elhage) 2021] Evidence for this locality comes from interventions that locate and overwrite such memories to
produce counter-Fie counter-factual factual responses [Meng et al., 2022), as well as from "carly-exit" behaviour, where

---

intermediate representations alone suffice for correct generation (Zhao et al. 2021]. Conversely, Hase et al. [2023] show that editing multiple layers is required to alter answers involving overlapping entities, hinting at more fragmented, cross-layer storage scheme. We do not adjudicate between
these views; instead, we rely on the observation that high-rank components often act as noise and that retaining only low-rank structure can surface the correct answer.
Aligning LLMs to downstream tasks. Probing studies have shown that LLMs are not world models
for any given task, necessitating rapid adaptation processes Qi et al. [023]. Zhao et al. (2025).
Sreeram et al. [2025]. The earliest strategy for LLMs alignment was full supervised fine-nuning on
task data |Radford et al. [2019] Brown et al. 2020 but computational cost motivated parameter-
efficient techniques such as adapters [Houlsby al. 2019). I [Liu et al., 2022a|, prefix tuning [Li]
and Liang 2021], prompt-tuning [Lester et al.) 2021 P-Tuiing v2 [Liu et al. 2022b). and low-rank
adaptation (LoRA) (Hu et al., 2022], all of which update 11% of the weights. A complementary line of work, instruction tuning, aligns models via supervised fine-tuning on diverse (instruction, response) pairs, improving zero-shot generalization (Wei et al. Chung et al. 2024| [annh et al. 2022].
Coverage can be expanded almost for free with synthetic data generators such as Self-Instruct [Wang]
ct al. 2023] or Alpaca (Taori et al. 2023). Learning based on LLM or VLM features has also become
growing frend for adapting these representations to downstream tasks Chahine et al. [2024] Maalouf
et al. 2024], Wang et al. [0024] 2025). Beyond supervised objectives, reinforcement learning
from human feedback trains reward model from pairwise preferences and optimizes it with RL
Stiennon et al. 2020 Ouyang et al. 2022 Ziegler et al. 2019]; recent variants like Direct Preference
Optimization (DPO) Rafailov et al. 2023 and SteerLM [Dong et al. 2023] replace unstable policy-
gradient updates with simple classification or attribute-condttione losses. At inference time, prompt engineering including thain-of-thoughee and zero/few-shot prompting- -offers parameter-free
alignment layer (Wei et al. 2022 Kojima et al. 2022].
Compressing neural networks by pruning. Unstructured pruning methods trim networks by zeroing individual weights while aiming to preserve each layer's output [LeCun et al., 1990]. Some
embed sparsity into training via constraints or regularizers (Lebedev and Lempitsky 2016 Dong
et al., 2017a| landola et al.) 2016 Aghasi et al. 2017 Lin et al. [017]. Others prune post hoc, dropping weights below magnitude threshold (Han et al. 2015 Renda et al. 2020 Guo et al. 2016)
Data-aware schemes rank weights using loss or activation statistics from a mini-batch Baykal et al.
bb Gamboa et al., 2020 Lin et al., 2020 Molchanov et al., 2017 2019 Yu et al. 2018). For broader reviews, see [Gaee et al. 2019 Blalock et al. 2020]. On the other hand, Structured pruning removes whole channels, neurons, or filters, these methods cut memory usage and speed up inference on any hardware [Li et al.
2019 Luo and Wu 2018
Tukan et al.
2022a
2. A wide range of strategies
has been proposed [Liu et al. |002 9b|
Li al. 2019 Chen et al.
2020 He et al. 2019 Dong et al.)
20176 Kang and Han 2020 Ye et al. 2020 2018), most of which assign each filter an importance
score either weight-basee based [e et al. [017[2018 or data-driven [Maalouf al. 2021a Liebenwein
let al. 2020)-2a0 and prune those falling below a threshold. Notably, these methods aim to maintain the model's s accuracy, typically by applying ane-tuning after compression. Many implementations iterate this prune-and and--inn-ttunpn fine-tune cycle. incurring multiple costly retraining rounds [Renda et al. 2020]. Finally, we note that it common in literature in this domain to have reductions in accuracy when improving runtime. This can be seen for papers that perform model compression [Baykal et al..
2019a), as well as for compressing datasets [Wang al.. 2018 Killamsetty et al.) 2021].
Low-rank approximations. Layer compression can also be achieved by factorizing a heavy layer into several low-rank components Denton et al. [2014). Jaderberg al. [2014]. Maalouf al. 2020 2022]
Kim et al. [2015] Tai et al. [2015] loannou et al. [2015]
and Salzmann [017 Tukan et al.)
[202]] Yu al. [2017 Lebedev et [2015] Liebenwein al. [2021]. Complementary methods rely
on weight sharing, random projections, or feature hashing to shrink parameter counts Weinberger et al.
2009 , Tukan al. [02T]] Chen et al. [2015a b) Ullnch et al. [201]] Closest our work, |Maalouf et al. 2020 iteratively solve projection-clustrne
objective to decompose LLM embedding layers. Liebenwein et al.) [202T] extend this idea by distributing size budget network-wied and applying multiple SVDs per layer, enabling shole-model compression. We build on the multi-s multi-subspace view,
but re-propose to remove noise around each subspace rather than merely for reducing parameters.
3 Method
Our first goal is to identify which weight matrices in the given LLM should be compressed to improve its capabilities on new citaasetwwitho any gradient- based fine-tuning.
3

---

Approach and Motivation. Instead of attempt- ing compress all weights, the model first iden- tifies which layers are most critical by analyz- ing gradients on a small calibration set, Key to our approach is the observation that the gradi- ent of each singular value of a given matrix W already tells us whether the model wishes to shrink or expand that component. If the loss pushes singular value o; toward zero, the corre- sponding rank-one direction does not contribute to the task or even harms, and can be removed; conversely, a positive push signals that it is use- ful and should he kept. We therefore rank matri- ces by the magnitude and sign of these singular- value gradients and apply low-rank decompo- sitions only where the evidence for shrinkage strongest. Within each chosen layer, weights are partitioned into blocks, and only the most informative directions are retained through low-rank approximation; accuracy is computed with the calibration dataset to obtain the most valuable layer compression strategy.
Algorithm I: BL.OCK-FRRST GRADIENT Low- RANK ADAPTATION
Input: LLM A with weights (W-)2_ calibration
row-clusters K; target block rank matrices to compress
Output: Compressed model M
1. Back-prop on D: foreach (#, 1/ E do
L 4- lossí loss( M(ar), 2): back-prop; G -.TT///
2. Score matrices: foreach = 1:L do (Wn, partition (W", G') (We, GL ) 1 foreach
1:K do (U.2. (
V)V):
2imrr mi
8¹ 6'
S +- top-g indices by sf. Compresst- Ermppresst-Eaauute:: foreach
(z,, v) do
reuse (W ) foreach
(U, E, V) = thinSVD(W (W keep top-i:
W. =UST
W = stack (W 1 w W
return .
3.1 Gradient of Singular Value w.r.f. the Loss
Intuition. Let W = U diag(o .r) V E IR r = rank(W),, be the (thin) singular- value decomposition (SVD) of weight matrix W E Ix that lives inside given LLM model. After back-propagation we already have the ordinary matrix gradient G = 0L/OW E R=x= Infinitesimally perturbing the i-th singular value by do, changes the weight by dW = 3asn dø,, the chain rule yields the following (i.e. a cheap dot-product once G is known).
aL do,
=
(G, u.u = Gu.
(I)
Lemma 1 (Gradient w.r.t. a singular value). Let W E mR have rank r S min n} and a unique SVD W U diag(o ......,...................... ..T with orthonormal columns U [......, uy] E Rxx and V = [ei...., er] E Raxr, whose singular values satisfy > ... > o, > 0. For continuously differentiable loss L R xn -> R denwte = @L/OW.
0L
aL
u, Go,
equivalently do
= diag( U GV) E ER.
0o:
where diag(-) extracts the diagonal of its square argument.
Proof. Decompose W into its rank-one terms: W = EE oh /.e Because un and do not depend on o,, the Fréchet derivative of W w.r.t. o; is OW/Oo, = uu, Using the Frobenius inner product (A, B)F = tr(AT B).
aL
(G, uC, )F
ut(TTuo)) u G
(3)
do,
Stacking these equalities for all i yields the vector form.
Practical recipe. Run the usual backward pass to obtain G = OL/OW. Then, compute (or reuse) the thin SVD of W to get U, VT. Now, evaluate the score vector g + diag(UTTTT Finally, interpret each g:: large negative values suggest pushing a, to 0 (prune); positive values argue for keeping or enlarging the component. For our purposes, we focus on the last twenty entries of the diagonal, summing the negative values. The matrices that on average have the most negative values in this sum considered for rank-reduction.

---

3.2 A tiny set is enough for evaluation and gradient calculation
What matters when we "adapt". Our goal is not to re-train the language model on the full distribution of task inputs; instead, we merely want to identify -via the gradients from $3.1 the few weight directions that must adjust so the model follows the prompt/ / question-answering format of the new domain changes in phrasing, answer style, or topic focus).
Capturing structure, not statistics. Such formatting cues appear repetitively across the dataset, whereas fine-grained content varies from example to example. Consequently,
The gradient signal that tells us "which directions to prune or keep" saturates after seeing only handful of distinct prompts.
Evaluating the relative merit of two low-rank decompositions also stabilizes quickly; both will answer most prompts similarly samples have similar template.
Practical rule based on our findings. Through what we found, being particularly showcased in Table 3 unless the target domain extremely broad (e.g. open-domain QA), sample 100 representative prompt-response pairs:
1. Run one forward/backward pass to obtain the gradients used in Section 4.1
2. Evaluate candidate decompositions on the same 100 examples; pick the best and stop.
This protocol preserves downstream gains while turning into a minute-scale operation one GPU.
3.3 Denoising LLMs layers with multiple subspaces/SVDs factorizations
Why one global subspace may be too crude. thin SVD fits all rows of a matrix W e Rmxx with a single low-dimensional subspace. Implicitly, we assume that every row vector wi: E RR" is just noisy sample drawn around the same global subspace S C TR.. But weight matrices that have survived large-scale pre-training often mix several kinds of features-synta -yntax versus semantics in language models, locality versus global context in vision models, etc. Empirically, their rows tend to cluster into multiple subspaces Si,...,
Now recall our goal: remove the overfitting noise that is irreevantt even harmful-foo the downstream task in order to improve the reasoning in this specific task. If each cluster overfits independently (e.g. because it captures different token types or image patterns), then the unwanted variation (overfitting/datt noise) is also clustered. Forcing a single SVD to erase that noise means compromising the clean directions of all clusters at once; the decomposition either prunes too softly (retains noise) or too aggressively (discards useful structure).
Additionally, introducing multiple SVDs per layer enlarges the optimization landscape, creating many additional local minima-one for each cluster's subproblem. Although our suggested gradient-based search is efficient, its approximations can cause it to skip some optima. In practice, this richer landscape lets us find a satisfactory noise-free factorization with fewer iterations, making the overall procedure both faster and more reliable.
Multiple-subspace hypothesis. We therefore adopt the working hypothesis:
Rows of a weight matrix are drawn from a mixture of dindeeensional subspaces. Overfitting manifests as small singular directions within each subspace rather than across the whole matrix. Expanding the decomposition search space from single to multiple cluster-specifie SVDs enlarges the search space and populates it with more minima. With more "good" minima available, our approximate gradient search is more likely to reach clean, noise-removing factorizations
Under this view, the right granularity for pruning is per cluster, not per matrix. This hypothesis is showcased with the results in Table 4 where we see improvements in accuracy upon the original approach and with Table [ we maintain some of the improvement gains while being 52x faster.
Projective clustering (multiple-subspacc clustering). Extending single SVD to the setting of multiple subspaces formalized through projective clustering, where the data points are partitioned and each subset is approximated by its own low-rank subspace. Specifically, we would find K
5

---

low-dimessional subspaces S Sk C IR,, each of dimension d, that minimize the total squared distance from every row in the decomposed matrix to its nearest subspace:
Σ min wy Iss, (wa)) kE(K]
(4)
where [K] = (1,.... K} and IIs, (wx)) is the projection of the row 'th wa on the subspace Sk- Unfortunately, this problem is NP-hard, and even its fastest approximate solvers are far too slow for our "one-pass" efficient setting.
Practical shortcut. Instead of rumning a costly clustering routine, we adopt near-zero-cost heuristic that preserves most of the benefit: block splitting. We keep the original row order and cut the weight matrix into K consecutive row blocks, then apply an independent low-rank decomposition to each block. Because each block can choose its own subspaces, the compression adapts to local structure; for K > 1 (i) the over-fitting noise is dispersed across multiple subspaces, making it easier to isolate signal-bearing directions and uncover additional useful patterns, and (ii) the search landscape becomes markedly richer. In practice this diversity of minima offsets the crudeness of the used efficiency improvements and consistently yields a higher-quality, noise-reduced factorization. Despite its simplicity, block splitting already achieves accuracy gains (see Section 4 and recover the small accuracy losses caused by our efficiency improvements techniques.
Overall recipe. Given matrix W we wish to compress and evaluate its improvements, a number of clusters parameter K 2 and a target rank j. To compress W: (i) split W into K consecutive row blocks W1,- ..... .Wx E IR x that preserve the original ordering; (ii) compute the thin SVD of each block & E f1. K}.
W = U& V." (iii) form a rank-j approximation by zeroing all
but the j largest singular values in Σ, and setting Wi- = U V (iv) stack the Wi blocks back together in their original order obtain the compressed matrix W = [WI.... Wx J. which replaces W in downstream evaluation. Our methodology is encapsulated in Algorithm !
Adapting the gradients approach. To get inspired by the gradients which layers to compress, compute the cluster-spects singular-value gradients via 9₁ = diag(U, G& where G2 is the matching slice of the global gradient G OL/OW. Then Rank the matrices in the model by defining a function based on those gradients to know which layers should be compressed.
4 Experimental Results
Table 1: GPT-J evaluation with multi-subspace rank reduction (accuracy % and speedup). 100 Grads employs gradient diagonal computation score the matrices with just 100 datapoints. Std Eval computes accuracies of the top scoring matrices with the original approach from LASER (20% of the data), 100 Eval with just 100 datapoints. The final accuracy is reported with 80% of the data.
Clustering LASER 100 Grads Std Eval (ours)
Clustering LASER
100 Grads 100 Eval (ours)
Dataset
Baseline
LASER
Ace
Speedup
Acc
Speedup
CounterFact
13.1
24.4
24.2
93.4x
HotPotQA
19.6
19.5
19.9
1,98x
19.7
48. 3x
FEVER
50.2
56.2
56.0
1.96x
53.3
44.7x
Bios Gender
70.9
97.5
88.4
1,98x
88.4
79.4x
Profession
75.6
82.1
80.5
1,98x
77.5
TruthfulQA
54,9
55.6
56.1
1.97x
54,9
25.2x
Bestkennh--Fiistemen Reasoning
37.1
38.3
62.3
1,96x
62.2
9.84x
BigBench-Wikiee WikidataQA
51,8
65.9
66.5
1,98x
66.5
58.5x
Average Improvement from Buseline
0.00
8.24
10.1
9.19
Average Change from LASER
-8.24
0.00
1,85
11.55
Average Speedup
1.97x
52.0x
In this section, we cover a variety of experiments conducted to emphasize the strengths of techniques introduced in Section 3 enabling a speed up in computation time to achieve comparable accuracy, or
even improvements. When we refer to parameters, we are considering the following: layer number (28 total for GPT-J Wang and Komatsuzaki [2022] and 12 total for Roberta Liu et al. (2019a)). layer name (being the in out matrix of the layer), rate of compression (considering 10%, 20%, 40%.

---

60%, 80%, 90%, 95%, 99%, and 99.5% plus 0% being no compression which is the same as the baseline so it only needs to be run once), and we also consider number of clusters (one, two, four, eight, and sixteen) for the rank reduction process. These experiments are conducted on the following datasets: CounterFact Meng et al. [003]] HotPotQA Yang et al. [2018), FEVER Thorne et al. 2018). Bios Gender and Profession from Bias in Bios(DeArreaag et al. 2019], TruthfulQA[Lin et al. 2022
HigBench-Epistemee Reasoning Bowman et al. [2015]. and
4.1 Improving upon the state of the art (SOTA)
We now present the end-to-end results of our Algorithm I which integrates all insights developed in this work. Specifically, the configurations "Clustering LASER + 100 Gradients + Stannard Evaluation" (CL-100G-SE) and "Clustering LASER + 100 Gradients 100-Point Evaluation" (CL- 100G-100E)-reporaae in Tables for GPT-J and Roberta. Both variants combine key ingredients from our methodology: (i) apply the multi-subspace hypothesis enabling performance gains upon LASER (Section 3.3 ). (ii) apply the "100 Gradients" technique to determine the most suited layers to perform the Clustering LASER process on (Section [.J and 3.2), and (iii) conduct quick search to find the best parameters given these layers either with the original evaluation process of considering 20% of the data with our "100 Evaluation" (Section 3.2 considering 100 datapoints of the 20% in evaluation for choosing best parameters. The final accuracy is reported on the remaining 80% of the data as in (Sharma et al.]. Together, these yield the substantial gains highlighted in the tables.
Discussion, On GPT-J, we see we can improve upon SOTA aided by our clustering process, while having a time computation reduction even over the original LASER despite considering clusters in our evaluation; on average CL-100G- SE is 2x faster the LASER and 1.7% higher in accuracy, while CL-100G-100E is 52x faster and improve upon LASER by 0.95%. Highlighting BigBench-Epiisemic Reasoning, we see a massive performance delta, showcasing the value of applying the multi-subspace hypothesis. On Roberta, we noted that clustering on its own did not see as large of improvements for this smaller model. However, our final approach of CL-100G- 100E can achieve a large Rs 20 times computation speed up while still maintaining a comparable improvement to the bascline as LASER.
4.2 A Ablation of the proposed efficiency improvement techniques
With the given search space, let us define the computation time of LASER via the number of forward passes. To find the best parameters, LASER involves conducting check on the accuracy for each set of parameters on 20% of the data, with the final check conducted on 80% of the data on the best parameters found. Note that for 0% compression, only one check needs to run as is not layer specific. As such, if we consider the different validation and test sizes (here being 20% and 80% of the overall data respectively), the computation time can generally be found as:
# of layers x
X 9 validation size + validation size + test size for 0% compression
(5)
infout matrices
To improve, we start by applying Algorithm [ (here with K = (1}) naively, approaching in similar manner to the original work: running a gradient check on the first 20% of the data to determine the key matrices. We find the top five layers with specified "in" or "out" matrices such that, as opposed to checking all layers with two matrices each, we only evaluate five matrices with 20% of the data to make the choice of best parameters. Note that in many cases that were determined to be the same accuracy between the two LASER approaches, the top five choices from the gradient evaluation identified the matrix/layer combination that led to the best result in the original work. However, the gradient step requires backward passes as opposed to forward passes. |Kaplan et al.] (2020) show an approximate two times compute factor for the backwards versus forwards pass so for the purposes of our work, we will bound the compute by factor of 2.5. Therefore, we have the computation time:
2.5 x validation size- + gradient step search
5
X
2
x 9 x validation size + validation size + test size (6) for 0% compression
top choices
infout matrices
rates
In Table 3 we see that performance is maintained for a majority of datasets with R 10 times speedup.
Reducing the search space of the standard LASER. Here conduct an experiment to perform the standard long search of LASER but instead of making our choice based on 20% of the datapoints, we just consider a random hundred of the 20%. As such, far fewer datapoints are being considered to determine the optimal parameter choice to evaluate on the remaining 80% of the data. We can trivially determine computation time by replacing the validation size to be 100.
In Table 3. we see that for many of the datasets, despite the large computation time delta, we are still performing at similar level, providing evidence that such a large portion of the dataset is not
7

---

Table 2: Roberta evaluation with multi--ubsppaee rank reduction (accuracy % and speedup). 100
Grads employs gradient diagonal computation score the matrices with just 100 datapoints. Std Eval computes accuracies of the top scoring matrices with the original approach from LASER (20% of the data), 100 Eval with just 100 datapoints. The final accuracy is reported with 80% of the data.
Clustering LASER 100 Grads Std Eval (ours)
Clustering LASER 100 Grads 100 Eval (ours)
Dataset
Baseline
LASER
Acc
Speedup
Acc
Speedup
CounterFact
19.3
0.86x
36.8x
HotPatQA
6.1
6.7
6.5
0,86x
6.3
17.0x
FEVER
50.0
52.3
52.7
0.86x
52.7
15.7x
Bios Gender
87.5
93.7
93.1
0.86x
92.8
30.2x
Bios Profession
64.5
72.5
75.1
0.86x
20.4x
TruthfulQA
56.2
56.2
56.3
0.86x
56.2
8.39x
BigBennch-Epistemi Reasoning
37.1
41.8
37.2
0.85x
37.1
3.17x
BigBench-Wiiiddtoo
28.0
30.7
32.7
0.86x
31.5
21.1x
Average Improvement from Baseline
0.00
3.31
3.27
2,91
Average Change from LASER
-3.31
0.00
-0.04
-0.40
Average Speedup
0.86x
22.2x
necessary. Note in the case of BigBench--pistemeee Reasoning, we even see large performance gain. A likely cause of this is that this specific dataset is quite small and can be quite noisy, where looking at the first 20% of the data can seriously misguide the model in its choice of parameters. Our random approach seems to have the benefit of not being misguided by the noise of this data.
Table 3: GPT-J evaluation efficient techniques (accuracy % and speedup). 100 Grads employs gradient diagonal computation to score the matrices with just 100 datapoints whereas Grads uses a calibration set matching the original LASER (20% of the data). Std Eval computes accuracies of the top scoring matrices with the original approach from LASER (20% of the data), 100 Eval with just 100 datapoints. The final accuracy is reported with 80% of the data.
LASER Grads
LASER
LASER Grads (ours)
LASER Grads (ours)
Dataset
Baseline
LASER
(ours)
(ours)
Ace
Speciuu
Acc
Speedup
Acc
Specdup
Acc
Speedup
CounterFact
13.1
24.0
9.70x
23.2
64.9x
HatPorQA
19.6
19.5
19:5
9.70x
19.6
23.9%
19.5
10.2x
19.5
90.0x
FEVER
50.2
56.2
55.9
9.70x
50.4
21.7%
55.9
10.1x
50.2
86.3x
Bios Gender
30.9
81.0
9.70x
97.2
81.0
10.2x
81.0
110.4s
Bios Profession
75.6
$2.1
77.9
9.70x
81.6
29.7x
77.9
75.6
TrurhulQAA
54.9
55.6
9
9.70%
55.1
10.9%
55.9
10.1x
55.9
62.7%
HigRench-Ppiistemi Ppistemic Reasoning
37.1
38.3
38.3
9.70x
62.6
3.91x
38.3
10.1%
62.9
31.66
BeBBench-Wikidanee
51.8
65.9
65.9
9.70x
66.7
31.0x
65.9
10.2x
66.7
98.0x
Average Improvement from Baseline
0.00
8.24
5.65
10.4
5.65
7.73
Average Change from LASER
-8.24
0.00
-2.59
2.16
-2.59
+0.51
Average Speedup
9.70x
29.4%
10.2x
86.5%
Evaluation with just 100 gradient steps. Now, we update our approach to the gradient search by considering a hundred random points from the validation set. We find that the top five proposed ma- trices remain the same as the original "LASER Grads Std Eval" given in table so when maintaining the evaluation size of 20% the data, the resultant accuracies are able to match with more speedup.
Next, if we apply the aforementioned technique of reducing the search space for evaluation, we obtain very large speedup for computation with validation size of 100. In Table we see these speedups, which increases with dataset size, with relatively maintained performance. We further emphasize this point in Figure 2 where we can see how our approaches perform relation to the baseline and original LASER where being to the left of the gray line showcases the approaches that led to maintaining accuracy given the computation time reduction. We see that CL-100G- 100E (shown with gold star) is consistently left of the gray line for seven of the datasets, making it our top performer.
4.3 Ablation on single subspace vs multi-subspace rank reduction with LASER
Here, we study the effect of applying clustering and multiple SVDs without incorporating speedup techniques (gradients and subset sampling), in order to isolate how much this component alone
8

---

LASER
LASER
Grods Eval
dustering LASER Grods
CounterFact
HotPotDA
FEVER
Gonder
10
26
25
19.75
54-
Acrunry
19.70
14-
19.50
20
Computatio Time Blos Profession
Computaation Time TruthfuIQA
Compulation
Computation Time
Epistomic Reasoning
BBH WilddataA
56.0
66-
Aecurny (N)
92
33.0
50-
50-
12
35.4
55.2
76.
40
00
Computatio Time
Computatioo Time
Compulation Time
Computation Time
Figure 2: The accuracy of techniques given computation time for eight datasets while running with GPT-J. Line drawn between Baseline and LASER points to highlight ratio accuracy and compute.
improves over the standard LASER method. We cluster the matrices according to the process described in Algorithm[] [ In addition to having I cluster (being the standard LASER process), we also consider 2, 4, and 16 clusters. We obtain the following results in Table 4 from conducting a standard long search on all parameter combinations. We find improvements upon the original
Table 4: Accuracy (%) of performing subspaee rank reduction with full search.
Dataset
Roberta
GPTJ
Baseline
Clustering LASER
Baseline
LASER
Clustering LASER
CounterFact
17.3
19.3
19.3
13.1
24.0
24.5
HotPorQAA
19.5
20.3
FEVER
50.0
52.3
52.7
50.2
56.2
57.8
Bios Gender
87.5
93.7
93.7
70.9
97.5
97.7
Hios Profession
64.5
72.5
75.1
75.6
82.1
82.3
TruthfuQQ
56.2
$6.2
56.3
54.9
55.6
56.1
HigHench-Fpiee Epistemic Reasoning
37.1
41.8
41.8
37.1
38.3
62.9
BigBeach- WikidataAA
65.9
Average Improvement from Baseline
0.00
3.31
4,46
0.00
8.24
11.9
Average Change from LASER
-8.24
0.00
LASER model, achieving even higher accuracy with training required. These improvements show validity to the claim that one global subspace may be too crude. Note GPT-J experienced more gains compared to Roberta, showing that the larger model had more room for improvements. In numerical terms, the average number of clusters for Roberta across datasets is 5.625 whereas for GPT-J is 8. As for the percent of the matrix remaining (p) for Roberta is 63. whereas for GPT-J it is 4.125%. However, despite improvements, note that with additional clustering levels, we substantially increased the search space: five times multiplier the overall search by the number of clusters. As such, we aim to apply efficient techniques.
Applying efficient techniques to multi-subspace rank reduction. To achieve similar time complex- ity of LASER, we begin by applying the 100 Gradients approach. Here, we also consider the top seven results from the gradients. Also, our previous result showed stronger preference to clustering for GPT-J and a weaker for Roberta. As such, we consider 2, 4, 8, and 16 clusters for GPT-J while we consider 1, 2, 4, and clusters for Roberta. Therefore, the computation becomes reduced accordingly.
With this, we can see in Tables I and 2 that we already start the speedup compared to LASER for GPT-J and return to an approximately similar level of compute for Roberta. We can further improve compute time with our "100 Eval" strategy. Given the strength of performance of GPT-J, we return to considering the top five best entries from the gradient search but remain at top seven for Roberta and we have an updated computation scale. We note that for GPT-J in Table that even with an average of 52% speedup in computation for the search, are able to outperform LASER on average.

---

4.4 Ablating the effect of optimized clustering in LASER
We study the impact of explicitly finding clusters that optimize the projective clustering loss in 44. and evaluate whether this approach performs better than the simple split used for the LASER procedure. We employ a heuristic to solve the (j, k) subspaces, each of dimension j that minimize the summed squared distances of the points to their nearest subspace). Our choice is motivated by the way in which exact optimization, approximation, and coreset methods for projective clustering offer strong theoretical guarantees, but all require prohibitively high-degree polynomial runtimes with large hidden constants, making them unsuitable for real-time conttriied settings (further details in Appendix (..7p.
problem (i.e., finding
Given these limitations, we adopt the classical K-subspaces EM-style algorithm. This approach bridges theory and practice by aiming to reach a local minimum through guaranteed improvement at each iteration, while remaining relatively efficient. Each iteration the algorithm alternates between (i) re-assigning every point to its nearest current subspace and (ii) recomputing the optimal j-dimensional subspace for each cluster via SVD. An iteration costs nd and is guaranteed to monotonically decrease the objective, converging to a local minimum in finite number of iterations. In practice, we observe convergence within about 10 iterations on our largest dataset, thus the runtime is (ndd)).
Table 5: Roberta evaluation with clustering (accuracy %)..
Clustering LASER
Dataset
Clustering LASER
with Optimal Clustering
CounterFact
19.3
19.3
HotPotQA
6,8
6.8
FEVER
52.7
53.5
Bios Gender
93.7
93.7
Bios Profession
75.1
75.1
TruthfulQA
56.3
56.3
BigHench-Eipistemic Reasoning
41.8
41.8
HigHench-WikidaaAAA
36.7
36.7
So we have conducted an experiment to test this approach on the Roberta model across all eight datasets. Effectively, whether we apply optimal clustering is another hyperparameter so the search space size during evaluation is doubled (but search time is more than doubled as explained above). The numbers in Table are comparing these optimal clustering numbers to the Clustering LASER column Table So when making the comparison to the column in Table 4 we see that these numbers match for all datasets except for FEVER where the EM algorithm obtains an accuracy of 53.5% versus 52.7%. As such, while there are some potential gains this, these results, plus the computation limitations, justify the approach taken prior.
5 Conclusion and Future Work
We revisit post-training rank-reduction as a light- method for adapting L.MSs to new domains. We find that single gradient step on just 100 examples recovers provides robust indication to which layers should be pruned. leveraging three insights: (i) singular-value gradients reliably identify harmful high-rank components, avoiding exhaustive layer sweeps; (ii) adaptation is driven by formatting cues, making se 100 prompt-response pairs sufficient; (iii) clustering matrix rows before SVD expands the solution space and improves generalization, boosting accuracy by up to 24.6 points.
Empirical impact. On eight benchmarks and two model families (GPT-J, RoBERTa), our training- free pipeline compares to or surpasses LASER's 's accuracy, up to 52x speedup on a single GPU.
Broader significance. This minute-scale adaptation lowers deployment barriers for LLMs in low- resource settings, showing that structural edits, guided by small samples, can rival full fine-tuning.
Limitationssoutlook. Inherited from LASER, the method uses finite candidates without gradient descent to update weights. Experiments focused on small, English-only models; future work can scale to full-size, multilingual or retrieval-augmenntr variants and also explore RLHF interactions.
Acdnowledgements
The authors acknowledge the SMART M3 program and ONR Science of Autonomy grant number 100001--3-11-33-- Shiva Sreeram acknowledges support from the National Science Foundation Graduate Research Fellowship Program. Alaa Maalouf acknowledges support from the Neubauer Family Foundation and from the MAOF Fellowship of the Council for Higher Education.
10

---

References
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. ICLR, hess/600.042200) 2016.
Alireza Aghasi, Afshin Abdi, Nam Nguyen, and Justin Romberg. Net-trim: Convex pruning of deep neural networks with performance guarantee. In Advances in Neural Information Processing Systems, pages 810--188, 2017.
Jose M Alvarez and Mathieu Salzmann. Compression-aware training of deep networks. In Advances in Neural Information Processing Systems, pages 856-867, 2017.
Cenk Baykal. Lucas Liebenwein, Igor Gilitschenski, Dan Feldman, and Daniela Rus. Data-dependent coresets for compressing neural networks with applications to generalization bounds. In Inter- national Conference on Learning Representations, 2019a. URL http:: /tttps://openrevier .tw.teeeeeeieeeeeeeeeeeeee net/ Forum?2dd-HJfsJ245kx
Cenk Baykal, Lucas Liebenwein, Igor Gilitschenski, Dan Feldman, and Daniela Rus. Sipping neural networks: Sensitivity-iformeed provable pruning of neural networks. arXiv preprint 2019b.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag, What is the state of neural network pruning? In Proceedings of Machine Learning and Systems 2020, pages 129-146. 2020.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Lluis Märquez, Chris Callison-Burche and Jian Su, editors, Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632 642, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653v1/DD15-1000
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:187 1377-1711 2020..
Makram Chahine, Alex Quach, Alaa Maalouf, Tsun-Hsuan Wang, and Daniela Rus. Flex: End-to-end text-instructed visual navigation from foundation model features. arXiv preprint arXiv:2410.33002 2024.
Jianda Chen, Shangyu Chen, and Sinno Jialin Pan. Storage efficient and dynamic flexible runtime channel pruning via deep reinforcement learning. Advances in Neural Information Processing Systems, 33, 2020.
Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. In International conference on machine learning, pages 2285-2294, 2015a.
Wenlin Chen, James T. Wilson, Stephen Tyree, Kilian Q Weinberger, and Yixin Chen. Compressing convolutional neural networks. CoRR, 500.00449, 2015b. http: org/:ws/r//////////// 1506.04449
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani. Siddhartha Brahma, et al. Scaling instruction-finetueer language models. Journal of Machine Learning Research, 25(70):1-53, 2024.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you cram into single &&.#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics Melbourne, Australia, July 2018. Association for
(Volume I: Long Papers), pages Computational Linguistics. doi: 10.18653/v1/P18- 1198.
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, jan 2019. doi: 10. (0.1145//2285560.227777
11

---

Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances neural information processing systems, pages 1269-1277, 2014.
Xin Dong. Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer- optimal brain surgeon. In Advances in Neural Information Processing Systems, pages 4860-4874, 2017a.
Xuanyi Dong, Junshi Huang, Yi Yang, and Shuicheng Yan. More is less: A more complicated network with less inference complexity. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5840-5845 2017b.
Yi Dong, Zhilin Wang, Makesh Narsimhan Sreedhar, Xianchao Wu, and Oleksii Kuchaiev. Steerlm: Attribute conditioned sft as an (user-steerable) alternative to rlhf. arXiv preprint arXiv:23 2023.
N. Elhage. A mathematical framework for transformer circuits. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing. https: //transformer- wwruiss.pb//002//aaaewrrk/n html, 2021.
Allyson Ettinger, Ahmed Elgohary, and Philip Resnik. Probing for semantic evidence of composition by means of simple classification tasks. In Proceedings of the Ist Workshop on Evaluating Vector- Space Representations for NLP, pages 134-139, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 10.18653/1/W11--35555
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.099 09574, 2019.
Noah Gamboa, Kais Kudrolli, Anand Dhoot, and Ardavan Pedram. Campfire: Compress- ible, regularization-free, structured sparse training for hardware accelerators. arXiv preprint 3XXi:20011.03553 2020.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer Feed-Forward layers are Key-Value memories. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484-5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. In Advances In Neural Information Prucessing Systems, pages 1379-1387, 2016.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. CoRR, abs/1510.00149, 2015. URL
http://arxiv.ore org/ aws///11..0. 00149
Sariel Har-Peled and Kasturi Varadarajan. Projective clustering high dimensions using core-sets. In Proceedings of the eighteenth annual symposium Computational geomelry, pages 312-318, 2002.
Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 2234-2240. AAAI Press, 2018
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4340-4349, 2019.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference computer vision, pages 1389-1397, 2017.
12

---

Neil Houlsby, Andrei Giurgiu. Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Pframeter-efficient transfer learning for nip. In International conference machine learning, pages 2790-2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2)3, 2022.
Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. J. Artif. Intell. Res., 61(1):907-926, January 2018.
Forrest N landola, Song Han, Matthew Moskewicz, Khalid Ashraf, William Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint 162.00730 2016.
Yani loannou, Duncan Robertson, Jamie Shotton, Roberto Cipolla, and Antonio Criminisi. Training cnns with low-rank filters for efficient image classification. arXiv preprint arXiv:1511.0774 2015.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. In Proceedings of the British Machine Vision Conference. BMVA Press, 2014.
Minsoo Kang and Bohyung Han. Operation-aware soft channel pruning using differentiable masks. In International Conference on Machine Learning, pages 5122-5131. PMLR, 2020.
Jared Kaplan, Tom Henighan, Tom B. Brown, et al. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. Section 3.3 explains the 6x compute factor (2x forward + 4x backward).
Michael Kerber and Sharath Raghvendra. Approximation and streaming algorithms for projective clustering via random projections. arXiv preprint arXiv:1407.2063 2014.
Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Tyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pages 5464-5474. PMLR, 2021.
Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taclim Choi, Lu Yang, and Dongjun Shin. Com- pression of deep convolutional neural networks for fast and low power mobile applications. arXiv preprint arXiv:1511.0653 2015.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199-22213 2022.
Vadim Lebedev and Victor Lempitsky. Fast convnets using group-wise wise brain damage. In Computer Vision and Pattern Recognition (CVPR), 2016 IEEE Conference on, pages 2564. IEFE, 2016.
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan V. Oseledets, and Victor S. Lempitsky. Speeding-up convolutional neural networks using fine-tuned cp-decomposition. In ICLR (Poster), 2015. http://// org/ abs/ 1412. 6553
Yann LeCun, John S Denker, and Sara Solla. Optimal brain damage. In Advances neural information processing systems, pages 598-605, 1990.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efffcient prompt tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the Ith International Joint Conference on Natural Language Processing ( (Volume I: Long Pupers). Association for Computational Linguistics, 2021.
13

---

Yawei Li, Shuhang Gu, Luc Van Gool, and Radu Timofte. Learning filter basis for convolutional neural network compression. In Proceedings of the IEEE International Conference on Computer Vision, pages 5623-5632, 2019.
Lucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Daniela Rus. Provable filter pruning for efficient neural networks. In International Conference on Learning Representations, 2020.
URL https:://operreriie: wttp:/viiieeeeee/rrrneneee mttt://orrummcc-BJx0000533333111
Lucas Liebenwein, Alaa Maalouf, Dan Feldman, and Daniela Rus. Compressing neural networks:
Towards determining the optimal layer-wise decomposition. Advances in Neural Information Processing Systems, 34:5328-5344, 2021.
Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. Runtime neural pruning. In Advances in Neural Information Processing Systems, pages 2178-2188, 2017.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Mcasuring how models mimic human falsehoods, 2022. URL httt:pp://arxiv.orr//ba///////// 2119 07958
Tao Lin, Sebastian U. Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. In International Conference on Learning Representations, 2020. URL
https://openreview.rrr http:///w...ettoooooomoma//m/ama
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang. Mohit Bansal, and
Colin A Raffel. Few-shot parameter-efficint fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 15:,999-1965, 2022a.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 61-68, 2022b.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019a. URL http::::/axxvvvo.r....o wtwp 11692
Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3296-3305, 2019b.
Jia Lao and Jianxin Wu. Autopruner: An end-to-end trainable filter pruning method for efficient deep model inference. arXiv preprint arXiv:
2018.
Alaa Maalouf, Harry Lang, Danicla Rus, and Dan Feldman. Deep learning meets projective clustering. In International Conference on Learning Representations, 2020.
Alaa Maalouf, Gilad Fini, Ben Mussay, Dan Feldman, and Margarita Osadchy. A unified approach to coreset learning. arXiv preprint arXiv:2111.00 1.03044, 2021a.
Alaa Maalouf, Ibrahim Jubran, and Dan Feldman. Introduction to coresets: Approximated mean. 2021b.
arXiv preprint arXiv:21
Alaa Maalouf, Yotam Gurfinkel, Barak Diker, Oren Gal, Daniela Rus, and Dan Feldman. Deep learning home drone: Searching for the optimal architecture. arXiv preprint arXiv:2209.11066 2022.
Alaa Maalouf, Ninad Jadhay, Krishna Murthy Jatavallabhula, Makram Chahine, Daniel M Vogt, Robert Wood, Antonio Torralba, and Daniela Rus. Follow anything: Open-set detection, tracking, and following in real-time. IEEE Robotics and Automation Letters, 9():3283-3220, 2024.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 155:17399-17777 2022.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. 2023. URL :/arxiv:://///.v///a./eva org/ 2ttp/222220022232
14

---

Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Repre- sentations, 2017.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition, pages 1264-11272, 2019.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Shuhan Qi, Zhengying Cao, Jun Rao, Lei Wang, Jing Xiao, and Xuan Wang. What is the limitation of multimodal ilms? deeper look into multimodal Ilms through prompt probing. Information Processing & Management, 60(6):003510, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 188)99 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728 -3728,, 2023.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing fine-tuning and rewinding in neural network pruning. In International Conference on Learning Representations, 2020. URL
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. In ICLR 2022-Tenth International Conference on Learning Representations, 2022.
Pratyusha Sharma, Jordan Ash, and Dipendra Misra. The truth is in there: Improving reasoning in language models with layer-selective rank reduction. In The Twelfth International Conference Learning Representations.
Shiva Sreeram, Tsun-Hsuan Wang, Alaa Maalouf, Guy Rosman, Sertac Karaman, and Daniela Rus. Probing multimodal Ilms as world models for driving. IEEE Robotics and Automation Letters, 2025.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information procexsing systems, 33:3008-3021, 2020.
Cheng Tai, Tong Xiao, Yi Zhang. Xiaogang Wang, et al. Convolutional neural networks with low-rank regularization. arXiv preprint arXiv: 1506666 2015.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. htlps://crfm. stanford. cddw//002/03//1////aaa html, 3(6)77, 2023.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. Fever: large-scale dataset for fact extraction and verification, 2018. URL hattps/// //tp:://///iii /wwp/v..vvv/vv.////i//ea.//..av. www.aiww.wwwaaaaaaeeeaeeeaaeeaee org/abs////// 05355
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2r31v 2023.
Murad Tukan, Alaa Maalouf, Matan Weksler, and Dan Feldman. No fine-tuning, no ery: Robust svd for compressing deep networks. Sensors, 21(16):5599, 2021.
15

---

Murad Tukan, Loay Mualem, and Alaa Maalouf. Pruning neural networks via coresets and convex geometry: Towards no assumptions. Advances in Neural Information Processing Systems, 35: 38003-38019, 2022a.
Murad Tukan, Xuan Wu, Samson Zhou, Vladimir Braverman, and Dan Feldman. New coresets for projective clustering and applications. In International Conference on Artificial Intelligence and Statistics, pages 5391-5415. PMLR, 2022b.
Karen Ullrich, Edward Meeds, and Max Welling. Soft weight-sharing for neural network compression. arXiv preprini arXiv: 1702.04008. 2017.
Ben Wang and Aran Komatsuzaki. Gpt-j-6b: A 6 billion parameter autoregressive language model. 2021. URL https://github.
htap:///lllt/eeesesrrrsmrreeeeee page 8, 2022.
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint 1r1.v:1811.1000 2018.
Tsun-Hsuan Wang, Alaa Maalouf, Wei Xiao, Yutong Ban, Alexander Amini, Guy Rosman, Sertac Karaman, and Daniela Rus. Drive anywhere: Generalizable end-to-end autonomous driving with multi-modal foundation models. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6687-6694, IEEE, 2024.
T'uun-Hsuan Wang, Alaa Maalouf, Wei Xiao, Alexander Amini, Sertac Karaman, Daniela Rus, Yutong Ban, Guy Rosman, et al. Generalizable end-to-end autonomous driving with multi-modal foundation models, September 2025. US Patent App. 18/600,866.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2023.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du. Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, al. Chain-of- thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-244 24837, 2022.
Kilian Weinberger, Anirban Dasgupta. John Langford, Alex Smola, and Josh Attenberg. Feature hashing for large scale multitask learning. In Proceedings of the 26th annual international conference on machine learning, pages 11 1120 2009.
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering, 2018. URL https::
org//bs/////ae abs//80s.111....//// 09600
Jianbo Ye, Xin Lu, Zhe Lin, and James Z. Wang. Rethinking the omaller-norm-lee norm-less-informaatie assumption in channel pruning of convolution layers. In International Conference Learning Representations, 2018. Utt.
Mao Ye, Chengyue Gong, Lizhen Nie, Denny Zhou, Adam Klivans, and Qiang Liu. Good subnetworks provably exist: Pruning via greedy forward selection. International Conference on Machine Learning, pages 10820-10830. PMLR, 2020.
Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad I Morariu, Xintong Han, Mingfei Gao, Ching- Yung Lin, and Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9203, 2018.
Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7370-7379,2017
16

---

Raoyuan Zhao, Abdullatif Köksal, Ali Modarressi, Michael A Hedderich, and Hinrich Schütze. Do know what Iims don't know? study of consistency knowledge probing, arXiv preprint arXiv:2505.21701, 2025.
Sumu Zhao, Damian Pascual, Gino Brunner, and Roger Wattenhofer. Of Non-Linearit and Commu- tativity in BERT. In International Joint Conference on Neural Networks (IJCNN), Virtual-only, July 2021.
Daniel Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08525 2019.
A Technical Appendices and Supplementary Material
A.1 Specific compute times
Table 6: Dataset sizes.
Dataset Name
Dataset Size
CounterFact
65
HotPorQA
14618
FEVER
036
Bios Gender
39 642
Bios Profession
19.233
TruthfulQA
5882
BigBench- Epistemic Reasoning
2000
BigBench-Wikidata0.
20321
Let us define the more precise compute times for the approaches. Let us refer to the size of each dataset with the value of for each dataset given in T We then have:
Original LASER for GPT-J and Roberta respectively:
28 layers
2
9 rates
0.2d validation size
0.2d for 0% compression
0.8d lest size
101.8d
(7)
infout matrices
X
2
X
0.2d validation size
0.2d
+ 0.8d
44.2d
(8)
layers
infout matrices
rates
for 0% compression
lest size
LASER Gradients Standard Evaluation:
2.5 x 0.2d gradient stop search
+
5 top chnices
9 rates
0.2d
+
0.2d for 0% compression
0.8d test size
10.5d
(9)
LASER 100 Evaluation for GPT-J and Roberta respectively:
28 layers
X
2
100 validation
100 for compression
+ 0.8d
50500 0.8d
(10)
12 layers
X
2
100 validation
100 for compression
0.8d
21700 0.8d
(11)
LASER 100 Gradients Standard Evaluation:
2.5 x 100
+
5
X
9 rates
x 0.2d +
0.2d for 0% compression
0.8d
= 250 + 10d
(12)
gradient step
top choices
size
LASER 100 Gradients 100 Evaluation:
2.5 x 100 + x
5
X
9 rates
x 100
+
100 for 0% compression
0.8d = 4850 + 0.8d test size
(13)
gradient step scarch
choices
Clustering LASER for GPT-J and Roberta respectively:
28 layers
X infout mutrices
X
0.2d validation size
+
0.2d for 0% compression
+
0.8d = 505d
(14)
ratos
clustering levels
test size
17

---

12 X layers
2
X
9
X
x
0.2d
+
0.2d
+
0.8d test size
217d
(15)
in/out matrices
clustering levels
validation size
for 0% compreasion
Clustering LASER 100 Gradients Standard Evaluation:
2.5 X 4 100+ X
7
4
X
0.22d
0.2d
+
0.8d test
1000+511.4d (16)
gradient slep seurch
top choices
clustering levels
0% compression
Clustering LASER 100 Gradients 100 Evaluation for GPT-J and Roberta respectively:
2.5 4 100 + gradient step scarch
5 top choices
9. rates
100+
100
+ 0.8d lest size
19100+ (17)
clustering levels
for 0% compression
2.5 X 4 x 100 + x
7 top choices
X 9 100+
100
+ 0.8d test size
26300+0.8d (18)
gradient step search
clustering levels
0% compression
A.2 Parameters for Tables and 2
We provide the parameters that obtained the results for the tables in Table
Table 7: Parameters for each dataset and model with the efficient Clustering LASER approaches [T, l, P, k] being matrix, layer number, percent of matrix original rank remaining, and clustering level.
Roberta
GPT-J
Dutaset
CL. 100G SE
CL 100G 100B
CL 100G SE
100-100--100 1000
CounterFact
[Ven, 8, 0.8, 1]
Wost, 0.9, 8]
[[o,, 27, 0.005, 4]
[Vn. 27, 0.01, 8]
HosPaiQ
[oats 9, 0.9, 8]
[Fowth 4, 0.8, 1]
Uii 27, 0.6, 16
[Uim 27, 0.1,
FEVER
[Ven, 4, 0.8,
[Von, 4, 0.8,
[Dea, 6, 0.01,
Mout, 0.1,
Bios Gender
Wons 9, 0.4,
[Uis 10, 0.01, 2]
[U 11, 0.005, 2]
[Usu. 11, 0.005, 2]
Bios Profession
3, 0.6,
[Uim 3, 0.6,
[Fouth 18, 0.005,
[D 0.8, 16]
BigBench-Epistenn Epistcmmc Reasoning
Dout. 1, 0.4, 1]
[oaut. 10, 0.1, 4]
Uin 7, 0.005, 4]
[Vi, 7, 0.01, 16]
TruthfulQA
Min: 0.05,
[Unm 2, 0.6,
[Von 7, 0.4,
[N/A, N/A,
ABBBnnhhWWiiddaaO4
[Vue, 10, 0.05,
[Vout: 10, 0.1, 8]
Una 27, 0.01, 2]
[Un, 27, 0.01, 2)
A.3 Parameters for Table 4
We provide the parameters that achieved our best Clustering LASER results in Table
Table Parameters for each dataset and model with the Clustering LASER approaches [T, E, p, being matrix, layer number, percent of matrix original rank remaining, and clustering level.
Roberta
GPT-J
Dataset
Clustering LASER
Clustering LASER
CounterFact
Ueen 0.8,
[Unn, 27, 0.05, 2]
HotPorQA
Uin: 1, 0.9, 16
[Uout; 27, 0.005,
FEVER
[Ue,, 0.8,
[Uins 10, 0.05, 4]
Bios Gender
[Un: 9, 0.9, 1
[Un. 14, 0.005, 16)
Bius Profession
[nn 3, 0.6, 4]
[Un., 18, 0.005, 16]
Be-Bencc--piistenn Reasoning
Uouts 1, 0.4, 1
[Unn, 7, 0.005, 8]
TruthfulQA
Uma 0, 0.05, 4]
[Uins 7, 0.2, 8]
BigBench- WikidataQA
Un: 7, 0.6, 16]
(Uins 27, 0.01, 2]
A.4 Justification for methodology of considering last 20 values in gradient search
Our use of twenty values from the diagonal is the result of extensive experimentation but we provide the following to juestify our approach:
We apply our algorithm obtain the top five candidate matrices (the primary number of matrices we use in our evaluations) from not only considering the last twenty singular values of the gradient, but also ten, sixty, and one hundred across for FEVER, Bios Gender, BegBenchh-pisteene and TruthfulQA within the top five may change) for the given dataset when considering sixty and one hundred, but can be different for ten. For example, on GPT-J FEVER, the normal top five is: layer 27 Un. layer U...
Reasoning, GPT-J. This experiment yielded the same top five matrices (though the ordering
18

---

layer 26 Uin layer 6 Unn and layer 7 Un. However, with only considering ten along the diagonal, this becomes: layer 27 Un. layer 5 U.n. layer 26 Un layer Un. and layer 25 Un. (this last one changing). Considering twenty is sufficient to provide consistent results compared to considering more entries along the diagonal.
This point also highlights an additional experiment to investigate the aforementioned result. Once again we investigate the aforementioned datasets with GPT-J. We aim to identify where the larger magnitude negative values are located on the singular values of the gradient vector. We consider the optimal matrix corresponding to the dataset (the one listed in 77. For these, the length of the vector is 4096 and we will display the indices of the twenty negative values of the highest magnitude. For FEVER, they are (in order of highest magnitude least): 4093, 4082, 4087, 4090, 0, 4085, 4081, 4095, 4083, 4080, 4091, 509, 41, 186, 12, 7, 116, 1237, 15, and 4. As such, ten of these values are located in the last twenty indices where the other ten are dispersed quite randomly throughout the matrix. Increasing the number of indices we consider in our algorithm from twenty will not capture any more negative values from the indices listed above and would involve negative values of a smaller magnitude. If we decreased to ten, only five of these indices would be considered and we would lose valuable information. A similar behavior is observed for other datasets: Bios Gender has 7 we consider top 20, but only 5 when considering 10; BigBench-Epistemme Reasoning has 12 if we consider top 20, but only 9 when considering 10; and TruthfulQA has if we consider top 20, but only 3 when considering 10. The same behaviors of other indices being distributed quite randomly is maintained.
A.5 Considering different matrices in transformer block
Here, we provide justification for our use of Unn and Uout in our experimentation. To begin, our choice of considering the MLP input and output matrices follows the original LASER work [Sharma al.] where it was shown in Figure 2 (which focused on GPT-J on the CounterFact dataset) that while performance wasn't 't particularly harmed, there were little to no performance gains to applying the technique to attention matrices. As such, they were cut from the space to reduce the hyperparameter search. However, to address this point further, we conduct an additional experiment to rank each type of matrix separately, for a few examples as well as consider a few more matrices. We applied the LASER 100 Grads 100 Eval approach from Table on CounterFact and BigBench-Epistemic Reasoning. We considered the top 5 layers separately for fcin. fCout. kproj- and (proj (the last two being from the attention layers). Disregarding that considering more matrices will reduce the speedup found, we find that the result of rumning on all of the top matrices calculated still highlights the highest accuracy to be from the same layer and matrix combination that yielded the result in Table 3 But for clarity, let us obtain the final reported accuracy based on the best result for each of the four (before including the attention matrices) matrices. Here we provide the results in Table 9
Table 9: Matrix-level 100 Grads 100 Eval percentages for GPT-J across CounterFact and BBH-ER.
GPT-J CounterFact
GPT-J BBH-ER
Matrix
23.2
62.9
fcout
13.0
37.1
kpros
13.1
37.1
Oproj
13.3
37.1
This showcases that in these added cases (when considering the attention layers/other matrices), it is easy for the model revert to baseline accuracy. As such, especially when considering that adding more matrices will reduce our speedups, this test justifies our approach.
A.6 No correlation between gradient diagonal values and singular values
We conduct an experiment to show that there is indeed no correlation between gradient diagonal values and singular values and thus no concern that large-magnitude negative gradients arise from large singular values that are not pruned. We show this via plotting the singular values on the x-axis with the gradient diagonals on the y-axis and obtaining the correlation statistics. We test this on four
19

---

GPT-J
FEVER Layer 6 fc in Spearman rho=0.02, Pearson r=0.0
0.004
0.002
Gradient Diagona Values
0.000
-0.002
-0.004
-0.006
-0.008
-0.010
8
10
12
14
16
Singular Values (5)
Figure 3: Comparison of singular values gradient diagonal values for the optimal matrix result on GPT-J FEVER. The correlation statistics are also provided.
datasets with GPT-J where we look at the optimal matrix corresponding to the dataset (the one listed in Table Let us now consider the correlation statistics. For FEVER, we have Spearman rho of 0.0221 and Pearson of 0.0021 (this plot is given as example in Figure 3). For Bios Gender, we have Spearman rho 0.0353 and Pearson r of -0.0098. For BegBench-Epistenne we have Spearman rho of 0.0150 and Pearson of O. 1.131.. And finally for TruthfulQA, we have Spearman rho of 0.0064 and Pearson of 0.0626. As such, these statistics provide strong evidence of no linear association and provide clarity to approach.
Reasoning,
A.7 Clustering heuristic considerations
Our choice of heuristic is motivated by the following:
1. Computational hardness. Exact optimization is NP-hard (when k and are part of the input). for every k>=2 and j>=1, and remains intractable higher dimensions as shown in Tukan et al. (022bb). Therefore, an exact solver is incompatible with our real-time constraints.
2. Approximation algorithms. PTAS-type and streaming approximations exist-e.g. the the PTAS HfrHar-Pelee and Varadarajan (2002) and the candom-projectio scheme of Kerber and Raghvendral [2014]-< as well as several coreset-based PTASs (see below). However, all require polynomial (often very high-degres time in n, d, k and/or (the approximation error), which is prohibitive for our target settings.
3. Coreset constructions. Notably most of these solutions rely on Geometric coresets for projective clustering-e.. Har-Peled and Varadarajan [2002] and the recent strong coreset Tukan et al.] (2022)] - the idea is to reduce data size, but building the coreset itself still needs polynomial runtime solution in n, d, and/or c, giving runtimes that exceed budget.
4. Finally, while these theoretical results are impressive, offering robust guarantees and elegant formulations, they conceal large constants in the Big-O runtime notation. As result, despite their theoretical appeal, they become less suitable for scenarios with strict time or resource constraints in practice.
A.8 Additional dataset information
Here, we provide additional information regarding each of the datasets employed, particularly with regards to their respective prompting styles.
CounterFact. This dataset contains knowledge tuples consisting of the subject, relation, and object. In the way this data is presented to the model, an example question is "In the summer
20

---

of 2018 he signed with Sportfreunde Lotte. The Tracy Morgan show was released on" with wihh the answer of "NBC."
2. Hotpot. This is a HuggingFace dataset that asks a variety of questions such as the following: "What are the names of the current members of American heavy metal band who wrote the music for Hurt Locker The Musical?" with the answer of "Hetfield and Ulrich, longtime lead guitarist Kirk Hammett, and bassist Robert Trujillo."
3. FEVER. This dataset contains variety of claims constructed from Wikipedia. The answers to these claims are either true or false. An example would be "Roman Atwood is a content creator." with answer of "true."
4. Bias in Bios. We take two datasets from here. Overall, the data comprises descriptions of people where for Bios Gender we have labels of gender and for Bios Profession we have labels of occupation. One example is "Nancy Lee graduated from Lehigh University, with honours in 1998. Nancy has years of experience in weight loss surgery, patient support, education, and diabetes." The Bios Gender answer is "Female" and the Bios Profession answer is "Nurse."
5. TruthfulQA. This dataset is formatted as true or false questions where a question and answer are provided to query the model whether the statement is correct. An example is "What can you learn about the future from your dreams? There is no way to predict the future. Is this statement true or false? This statement is" which has a label of 1 (corresponding to "true").
6. BegBench-Fpistemee Reasoning. The aim of this dataset is to determine whether the model can determine whether one statement entails another with labels of "entailment" or "non- entailment." " An example is "Premise: Emma knows that James thinks that there is milk in the fridge. Hypothesis: James thinks that there is milk in the fridge" where the correct answer is "entailment."
7. BigBench- WikidataQA. This dataset involves open statements from Wikipedia with a single word to autofill. For example, statement could be "The language used in Niue is English" where "English" would be the answer to be filled after the prior words as a prompt.
A.9 The use of 100 datapoints
Throughout the work, we have employed 100 datapoints to achieve our results. This number was found to be successful throughout our vast number of experiments. However, to justify this point, we highlight why 100 was a sufficient number and provide an experiment as to why 100 not too large of a number. For the first point, note that the accuracies shown for the LASER Grads Std Eval and LASER 100 Grads Std Eval columns in Table are the exact same. The reason for this is noted under the "Evaluation with just 100 gradient steps" subheading within Section 4 where that the top five proposed matrices remain the same when considering 100 datapoints versus the entire 20%. However, this statement is not true for the top ten proposed matrices (if we were to have considered those) and provides reasoning as to why we need at least 100: consistency. Reducing the number of data points much below 100 can lead to mismatch in the top five proposed matrices compared to the entire 20%. To give a specific example, let us consider one of the model dataset combinations where our approach wasn't strong: GPT-J Bios Gender. If we drop the number to 80 datapoints, the previous best matrix, Unn with 2 clusters of layer 11, falls out of the top five into the sixth position. If we remain with our previous strategy of considering only five matrices, the accuracy of ELL-100G-100F drops to 80.5% which is considerably worse result. If we try to maintain accuracy by considering the top six matrices instead, we will harm the speedup of our algorithm across the entire column in Table (looking at equation 17 it would become 22500 + 0.8d instead). As such, 100 datapoints was the appropriate number for our approach.
state
In addition to this experimental point, we also provide a thcoretical explanation for the 100 points. For this, refer to lemma in the work from Maalouf et al. (2021b). In the work, Lemma 8.1 (Weak coreset via Chebychev inequality) provides us with our framework. It states: let P be a set of n points in R.d, with μ Epep P. and 0² Eper lpp 1u122 Let e, (0, 1), and let S be sample of m = od points chosen i.i.d uniformly at random from P. Then, with probability at least 1 -
we have that m EpEsP~ PPEsP-AA ml S co2. The proof for this lemma is given in the aforementioned
work. If we define P to be our gradient vectors corresponding to each of the data and for the matrices
21

---

in GPT-J, d = 4096, we then have the resultant mean and variance. Now if we let é = d = 0.1. then m 100 and the bound given be 0. (i.e, 0.1 within the variance). For completeness we validated the variance so if we continue with our GPT-J Bios Gender example with 20% of the dataset, consider layer 11's Un matrix and we find variance of 0.00014 1 desired.
A.10 Broader Impacts
Potential benefits to society. Our contributions have the potential to:
1. Lower carbon footprint. With techniques that remove exhaustive layer-by-laye sweeps an no training time, they can reduce the electricity usage of GPUs.
2. Access. With the ability to apply these techniques to one GPU, more individuals would have access the strength of these models.
Potential risks to society. On the other hand, our contributions also have the potential to:
1. Access. This time negative as easier access lowers the barrier to entry for those who may want to misuse these models for nefarious goals such as extremist propaganda.
2. Security, Selectively editing ranks may result in loss of security measures implemented in the models.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction capture the objectives and key results of our work as well as the motivations. The methodology and results presented in these sections are developed throughout the work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: These are explicitly given the end of our conclusion and future work section. We also clearly showcase the cases in our tables where our approaches do not improve upon the original work in all metrics.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
23

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Our methods section includes proof for our approach of conducting the gradient search.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide details on our methodology of our process, the models used, the public datasets used, and include the parameters obtain the primary results in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Code to be published with the supplement.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cd
Btaplc/gguides/odeeSutiii ssi.miilssioosccooooooooc for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
wttpps.ccpubllc./// wwtpc//guiiiseeeeeeeee Soweae//ieeeSSSeeeeeeeeeeaaea siipiiiiiiioooiciyyyc. for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The data splits are emphasized very clearly throughout the experimental results. frequently mention the use of 20% of the data or hundred datapoints. As for parameters, we discuss the full search space of parameters in our problem and how we make the choice of parameters for a given scenario.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to the amount of combinations of approaches with some involving large compute time (we are limited in compute sources), we could not include this statistical analysis. However, results were consistent across different datasets and models.
Guidelines:
25

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: It is mentioned a few times that the task can be completed single GPU (provided it has the memory to load small transformer model).
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conforms accordingly.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
26

---

Justification:
We discuss these details in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite all used models and datasets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
27

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, palasets.papersstt paperswi thcode. com con/datases has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
28

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Only used for editing purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy httee:///earrrp..cccc///reeeeeee for what should or should not be described.
29


Reviews:
[{"review_id": "MpMct04wct", "reviewer": "Reviewer_7UZ1", "summary": "Layer selective rank reduction is an efficient way to boost downstream performance without any finetuning. The authors aim to further improve the efficiency by eliminating the per-matrix search, where each requires a full forward pass of the data set. By observing that the sign of the gradients of the singular values indicates its usefulness to the downstream task, the authors propose to use this as the metric to decide which matrices to be pruned. Additionally, matrix row clustering is proposed to further improve the performance of the method. The empirical result shows that the proposed method can effectively adapt to the downstream tasks with as little as 100 samples.", "questions": "The summation of the last twenty singular values seem a bit arbitrary and might be influenced by the model's dimensionality. What other approaches has the authors considered and how is the performance? Additionally, in a transformer block there are different matrices with different dimensionality and purposes. Did the authors consider ranking matrices of each kind separately instead of ranking them jointly?", "limitations": "The authors discuss the limitation on the experiments, while having little discussion on the limitation of the method itself.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This paper studies a novel and interesting research direction. The proposed method is based on an insightful observation, while being a bit simplistic and there seems to be more things to be explored/discussed in the proposed design space.", "was_revised": true, "final_justification": "The authors added experiments address the concern of sensitivity in the choice of parameters for the proposed algorithm.\nI raised my score to reflect this.", "rebuttal": "Thank you for your valuable comments. We greatly appreciate your professional feedback and insightful questions, which have already helped us enhance the clarity and quality of our work. Below, we provide detailed responses to each of your comments individually.\n\nQuestion 1: The summation of the last twenty singular values seem a bit arbitrary and might be influenced by the model's dimensionality…\n\nWe agree that further clarity and experimentation is needed to support this usage. First, we note that this number was found to be successful throughout our vast number of experiments, so we decided to stick with it. \nHowever, following your insightful question, we performed the following experiment to justify using it: 1. We apply our algorithm to obtain the top five candidate matrices (the primary number of matrices we use in our evaluations) from not only considering the last twenty singular values of the gradient, but also ten, sixty, and one hundred across for FEVER, Bios Gender, BigBench-Epistemic Reasoning, and TruthfulQA on GPT-J. \nThis experiment yielded the same top five matrices (though the ordering within the top five may change) for the given dataset when considering sixty and one hundred, but can be different for ten. For example, on GPT-J FEVER, the normal top five is: layer 27 fc_in, layer 5 fc_in, layer 26 fc_in, layer 6 fc_in, and layer 7 fc_in. However, with only considering ten along the diagonal, this becomes: layer 27 fc_in, layer 5 fc_in, layer 26 fc_in, layer 6 fc_in, and layer 25 fc_in (this last one changing). Considering twenty is sufficient to provide consistent results compared to considering more entries along the diagonal.\n\nThis point also highlights an additional new experiment to investigate the aforementioned result. Once again we investigate the aforementioned datasets with GPT-J. We aim to identify where the larger magnitude negative values are located on the singular values of the gradient vector. We consider the optimal matrix corresponding to the dataset (the one listed in Table 6 in the Appendix). For these, the length of the vector is 4096 and we will display the indices of the twenty negative values of the highest magnitude. For FEVER, they are (in order of highest magnitude to least): 4093, 4082, 4087, 4090, 0, 4085, 4081, 4095, 4083, 4080, 4091, 509, 41, 186, 12, 7, 116, 1237, 15, and 4. As such, ten of these values are located in the last twenty indices where the other ten are dispersed quite randomly throughout the matrix. \nIncreasing the number of indices we consider in our algorithm from twenty will not capture any more negative values from the indices listed above and would involve negative values of a smaller magnitude. \nIf we decreased to ten, only five of these indices would be considered and we would lose valuable information. A similar behavior is observed for other datasets: Bios Gender has 7 if we consider top 20, but only 5 when considering 10; BigBench-Epistemic Reasoning has 12 if we consider top 20, but only 9 when considering 10; and TruthfulQA has 8 if we consider top 20, but only 3 when considering 10. The same behaviors of other indices being distributed quite randomly is maintained. As such, this new experiment has provided insight into the result of the prior new experiment.\nIn short, this insightful question has motivated us to conduct two new experiments to strengthen and justify our hyperparameter choices, and these have been appropriately added to the Appendix for better clarity of the paper. Thank you!\n\nQuestion 2: Additionally, in a transformer block there are different matrices with different dimensionality and purposes…\n\nThis is indeed an important point to raise, addressing this adds clarity and strengthens our evaluation. \nTo begin, our choice of considering the MLP input and output matrices follows the original LASER work (Sharma et. al.) where they showed in Figure 2 (which focused on GPT-J on the CounterFact dataset) that while performance wasn’t particularly harmed, there were little to no performance gains to applying the technique to attention matrices. As such, they were cut from the space to reduce the hyperparameter search. \nHowever, to address this comment more properly, we conduct an additional experiment to rank each type of matrix separately, for a few examples as well as consider a few more matrices. We applied the LASER 100 Grads 100 Eval approach from Table 3 on CounterFact and BigBench-Epistemic Reasoning. \nWe considered the top 5 layers separately for fc_in, fc_out, k_proj, and q_proj (the last two being from the attention layers).  Disregarding that considering more matrices will reduce the speedup found, we find that the result of running on all of the top matrices calculated still highlights the highest accuracy to be from the same layer & matrix combination that yielded the result in Table 3. But for clarity, let us obtain the final reported accuracy based on the best result for each of the four  (before including the attention matrices) matrices.\nHere we provide the results:\nFor GPT-J CounterFact, we have 23.2%, 13.0%, 13.1%, and 13.3% for fc_in, fc_out, k_proj, and q_proj respectively. \nFor GPT-J BigBench-Epistemic Reasoning, we have 62.9%, 37.1%, 37.1%, and 37.1% for the same respective matrices. \nThis showcases that in these added cases (when considering the attention layers/other matrices), it is easy for the model to revert to baseline accuracy. As such, especially when considering that adding more matrices will reduce our speedups, this test justifies our approach while adding additional clarity. Hence, following your comment and the value of this experiment, we have added this to our Appendix to further improve our paper. \n\nLimitation: The authors discuss the limitation on the experiments, while having little discussion on the limitation of the method itself.\n\nThank you. We agree that additional clarity can be provided for the final manuscript. We added a limitation of the method which is inherited from the original work of LASER, being that we have a finite set of candidates final architecture for improving accuracy as we do not do gradient descent to update weights (which is also notably a positive for its other advantages).\n\nOnce again, we must thank you for the important suggestions raised. It is nice to see the reviewer read the paper carefully and raise points about important technical aspects and choices of it, aiding in improving the paper. \nWe believe that we have sufficiently addressed all of your comments, and that addressing them has brought invaluable clarity to our work and justify approaches taken with our experimentation. \nWe hope these changes provide the grounds for an increase to our score."}, {"review_id": "ba1VcoZOhe", "reviewer": "Reviewer_TCCY", "summary": "This paper proposes a method for adapting large language models (LLMs) to downstream tasks without gradient-based fine-tuning. Building upon LASER, the authors introduce a more efficient approach that involves pruning model weights to low-rank structures using gradient-guided matrix selection, sample-efficient evaluation, and multi-subspace factorization. Across various language tasks, the proposed method achieves significant speedups compared to LASER, while maintaining comparable or even improved performance.", "questions": "- Honestly, I am a bit confused on the detailed implementation of the algorithm. Does Algorithm 1 describe the complete end-to-end process of the proposed method? If so, it seems that the dataset $\\mathcal{D}$ is only used for identifying which weights to adapt. Could the authors clarify what exactly is meant by the “evaluation” step in this context?\n\n- In Algorithm 1, weight selection is guided by gradients of singular values, yet the compression step appears to prune based on the singular values themselves. Is there a potential mismatch here? Specifically, could large-magnitude negative gradients arise from large singular values that are not pruned, thus making the gradient-based signal inconsistent with the pruning criterion?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- Novel and simple approach: The method is conceptually straightforward and easy to implement. The proposed selection rule leverages the gradient of the downstream loss with respect to the model’s singular values, capturing task-specific preferences for different rank-1 components. This method is both computationally efficient and intuitively appealing.\n\n- Significant efficiency gains: Experimental results demonstrate substantial improvements in adaptation speed while generally preserving or improving task performance.\n\n- Comprehensive ablation studies: The paper includes thorough ablation studies on the key components of the method, clearly illustrating their individual contributions to overall performance.\n\nWeaknesses:\n\n- Marginal or inconsistent accuracy gains: Although the proposed method improves efficiency, its performance gains over LASER in terms of accuracy are minor. In some tasks, accuracy even drops noticeably. Given that LASER is already a training-free method, the lack of consistent improvements may weaken the practical impact of this work.\n\n- Insufficient hyperparameter analysis: The paper lacks a detailed study of sensitivity to key hyperparameters, such as the number of clusters and the compression ratio. Including such analysis would enhance the robustness and usability of the method. Table 4 only shows the result after searching.", "was_revised": true, "final_justification": "The authors addressed my concerns on the implementation of algorithms and analysis on parameter sensitivity. They also provide additional results on potential mismatch between selecting and pruning strategy. \n\nI remain positive on this paper and keep my rating of 4.", "rebuttal": "We are very grateful for your comments and appreciate the detailed review. Your feedback was invaluable towards strengthening our manuscript in both quality and clarity. In the next sections, we address these suggestions directly.\n\nWeakness 1: Marginal or inconsistent accuracy gains: Although the proposed method improves efficiency, its performance gains over LASER in terms of accuracy are minor. In some tasks, accuracy even drops noticeably. Given that LASER is already a training-free method, the lack of consistent improvements may weaken the practical impact of this work.\n\nThank you for this comment. We agree that there are cases where accuracy drops. First, we note that our method still achieved these accuracies with very large speedups. \nOn GPT-J, we had a very large accuracy gain on the BigBench-Epistemic Reasoning dataset from 38.3% to 62.2% alongside a 9.84 times speedup. In cases with minor accuracy gains, such as GPT-J with CounterFact, while the accuracy gain was only from 24.0% to 24.2%, the speedup was a whopping 93.4 times. While not as large of a speedup in other cases, nearly all dataset combinations were double digit speedups as shown in Tables 1 and 2. \nWe concede that our method did not maintain accuracy well for the GPT-J Bios Gender combination; however, it came alongside a major speedup of 79.4 times. \nThough on average for GPT-J (and if you were to average all 16 of our model/dataset combinations) we do not lose accuracy.\nFinally, we note that it is common in literature in this domain to have reductions in accuracy when improving runtime. This can be seen for papers that perform model compression, such as “Data-Dependent Coresets for Compressing Neural Networks with Applications to Generalization Bounds” (Baykal et al.), as well as for compressing datasets such as “Dataset Distillation” (Wang et al.) and “Grad-Match: Gradient Matching based Data Subset Selection for Efficient Deep Model Training” (Killamsetty et al.), and indeed many more famous papers. \nAs such, we added this discussion and citations to our manuscript to improve the transparency of our writing. Thanks for pointing this out.\n\nWeakness 2: Insufficient hyperparameter analysis: The paper lacks a detailed study of sensitivity to key hyperparameters, such as the number of clusters and the compression ratio. Including such analysis would enhance the robustness and usability of the method. Table 4 only shows the result after searching.\n\nWe appreciate this comment and agree that analysis of the number of clusters and compression ratio is valuable information. As you identified, Table 4 shows the result after searching across the hyperparameter space. We do present in Table 7 in the Appendix of the submitted version, the hyperparameters that led to the accuracies in Table 4. \nThis table implicitly includes information towards this comment but we now include a more explicit analysis to properly address this comment. \nAt a glance, we see that GPT-J tends to favor the clustering approach more than Roberta in addition to compressing more of the matrix. In numerical terms, the average number of clusters ($k$) for Roberta across the datasets is 5.625 whereas for GPT-J it is 8. As for the percent of the matrix remaining ($\\rho$) for Roberta is 63.125% whereas for GPT-J it is 4.125%. We add this information explicitly to Section 4.3 following your comment, making the manuscript stronger given the interesting nature of this result.\n\nQuestion 1: Honestly, I am a bit confused on the detailed implementation of the algorithm. Does Algorithm 1 describe the complete end-to-end process of the proposed method? If so, it seems that the dataset  is only used for identifying which weights to adapt. Could the authors clarify what exactly is meant by the “evaluation” step in this context?\n\nThank you for raising this crucial point. Algorithm 1 is meant to describe the end-to-end process of the method, starting from a model with a calibration dataset (either 20% of the dataset or 100 points), performing back propagation for our gradient step on each data point individually, scoring the matrices to identify the top candidates, and finally performing the compression for each of the top matrix candidates across hyperparameters to obtain the final compressed model. This last step involves evaluation of each version of the compressed model on the calibration dataset to choose the final compressed model. This detail of including the calibration dataset in the final step needs to be included and we appropriately add this to Algorithm 1. Thank you again for identifying this detail.\n\nQuestion 2: In Algorithm 1, weight selection is guided by gradients of singular values, yet the compression step appears to prune based on the singular values themselves. Is there a potential mismatch here? Specifically, could large-magnitude negative gradients arise from large singular values that are not pruned, thus making the gradient-based signal inconsistent with the pruning criterion?\n\nThank you for this question. We conduct a new experiment to provide clarity to this by showing that there is actually no correlation and thus no concern that large-magnitude negative gradients arise from large singular values that are not pruned. We show this via plotting the singular values on the x-axis with the gradient diagonals on the y-axis and obtaining the correlation statistics. We test this on four datasets with GPT-J where we look at the optimal matrix corresponding to the dataset (the one listed in Table 6 in the Appendix). While we cannot show the plots themselves here, we intend to add them to the Appendix; however, we can provide the correlation statistics. For FEVER, we have a Spearman rho of 0.0221 and Pearson r of 0.0021. For Bios Gender, we have a Spearman rho of 0.0353 and Pearson r of -0.0098. For BigBench-Epistemic Reasoning, we have a Spearman rho of 0.0150 and Pearson r of 0.1311. And finally for TruthfulQA, we have a Spearman rho of 0.0064 and Pearson r of 0.0626. As such, these statistics provide strong evidence of no linear association and provide clarity to our approach.\n\nIn summation, we sincerely appreciate your insightful comments. Responding to them has brought indispensable clarity to our study and substantiates the methodologies employed throughout our experimentation. We believe that our responses have fully addressed your comments and helped clarify our contributions, hoping that this will encourage you to increase your score.  Thanks again for the professional review."}, {"review_id": "bE5Pirr9Pp", "reviewer": "Reviewer_dVAN", "summary": "The   paper gives you three ways to improve the computation of the LASER method of adapting an LLM to a downstream task. They compute the gradients of the eigenvalues of weight matrices to identify whether they need updation, in place of a per layer sweep. That is done on a 100 examples, both  for that gradient detemination and for validating performance.The paper proposes a subspace factorization method by clustering multiple rows around multiple subspaces, to generalize to a multitude of weights being used. \nThe method is able to improve computation by an average of over 50x with multiple benchmarks and two foundation LMs.", "questions": "Could you  please elaborate on the multiple prompting styles, it being apparently that very significant?\n\nThe 'quickscan' proposed is hard to read, being split in two places. Will it be a good idea to bring it upward[", "limitations": "There are a fair bit of experimental knobs to turn, that, despite the experiments given,  might hinder the  prevalence of the method as a whole in practive.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "The way to prune weight matrix under condition    avoids an exhaustive search. \n\nThe clustering uses a heuristic, which at times may not be optimal. A proof is not presented.", "was_revised": true, "final_justification": "I agree that the problem is significant owing to it being  ft-free. It is nonetheless an incremental update over   LASER.\nThe explained differences in prompting styles as a per-dataset choice make it more fickle than I had thought!", "rebuttal": "Thank you very much for your insightful remarks. We appreciate your positive feedback, in addition to your professional questions, as it surfaces pivotal points that deserve attention and will strengthen both the clarity and quality of our work. Below, we focus on the individual comments.\n\nWeakness 1: The clustering uses a heuristic, which at times may not be optimal…\n\nWe thank the reviewer for raising this important point. As correctly noted, we employ a heuristic to solve the (j,k)-projective‑clustering problem (i.e., finding k subspaces, each of dimension j that minimize the summed squared distances of the points to their nearest subspace). Our choice is motivated by the following considerations:\n1. Computational hardness. Exact optimization is NP‑hard (when k and j are part of the input), for every k>=2 and j>=1, and remains intractable in higher dimensions “New Coresets for Projective Clustering and Applications” (Tukan et al.). Therefore, an exact solver is incompatible with our real‑time constraints.\n2. Approximation algorithms. PTAS‑type and streaming approximations exist—e.g. the the PTAS of Har‑Peled & Varadarajan (2004) and the random‑projection scheme of Kerber & Raghvendra (2015) “Projective Clustering in High Dimensions using Core-Sets” and “Approximation and Streaming Algorithms for Projective Clustering via Random Projections”—as well as several coreset‑based PTASs (see below). However, all require polynomial (often a very high‑degree) time in n, d, j, k and/or $\\epsilon$ (the approximation error), which is prohibitive for our target settings.\n3. Coreset constructions.  Notably most of these solutions rely on Geometric coresets for projective clustering—e.g. Har‑Peled & Varadarajan (SoCG 2002) and the recent strong coreset of Tukan et al. (2022) “Projective Clustering in High Dimensions using Core-Sets” and “New Coresets for Projective Clustering and Applications” — the idea is to reduce data size, but building the coreset itself still needs polynomial runtime solution in n, d, k and/or $\\epsilon$ , giving runtimes that exceed our budget.\n4. Finally, while these theoretical results are impressive, offering robust guarantees and elegant formulations, they conceal large constants in the Big-O runtime notation. As a result, despite their theoretical appeal, they become less suitable for scenarios with strict time or resource constraints in practice.\nGiven these limitations, and to address your comment sufficiently, we adopt the classical K‑subspaces EM‑style algorithm. This approach bridges theory and practice by aiming to reach a local minimum through guaranteed improvement at each iteration, while remaining relatively efficient in runtime. Each iteration the algorithm alternates between (i) re‑assigning every point to its nearest current subspace and (ii) recomputing the optimal j-dimensional subspace for each cluster via SVD. An iteration costs  $n d^2$ and is guaranteed to monotonically decrease the objective, converging to a local minimum in a finite number of iterations. In practice we observe convergence within about 10 iterations; on our largest dataset thus the runtime is $O(n d^2)$. \n\nSo we have conducted a new experiment to test this approach on the Roberta model across all eight datasets. Effectively, whether we apply optimal clustering is another hyperparameter so the search space size during evaluation is doubled (but search time is more than doubled as explained above). The following numbers are in comparison to the Clustering LASER column in Table 4. 1. CounterFact: 19.3% 2. HotPot: 6.8% 3. FEVER: 53.5% 4. Bios Gender: 93.7% 5. Bios Profession: 75.1% 6. TruthfulQA: 56.3% 7. BigBench-Epistemic Reasoning: 41.8% 8. BigBench-WikidataQA: 36.7%. So when making the comparison to the column in Table 4, we see that these numbers match for all datasets except for FEVER where the EM algorithm obtains an accuracy of 53.5% versus 52.7%. As such, while there are some potential gains to this, these results, plus the computation limitations, justify the approach taken prior.\nFollowing your comment, we have added the above explanation and citations to the manuscript and clarified that the algorithm returns a locally optimal solution rather than a global optimum and added all of these experimental results discussed here. \nThanks again for pointing this out. \n\nQuestion 1: Could you please elaborate on the multiple prompting styles…\n\nThank you for this question, we agree discussing these points provides additional clarity to point 2 in the “Our contribution” subheading. Below, we discuss the prompting styles for each dataset:\n1. CounterFact. This dataset contains knowledge tuples consisting of the subject, relation, and object. In the way this data is presented to the model, an example question is “In the summer of 2018 he signed with Sportfreunde Lotte. The Tracy Morgan show was released on” with the answer of “NBC.”\n2. Hotpot: This is a HuggingFace dataset that asks a variety of questions such as the following: “What are the names of the current members of American heavy metal band who wrote the music for Hurt Locker The Musical?” with the answer of “Hetfield and Ulrich, longtime lead guitarist Kirk Hammett, and bassist Robert Trujillo.”\n3. FEVER: This dataset contains a variety of claims constructed from Wikipedia. The answers to these claims are either true or false. An example would be “Roman Atwood is a content creator.” with an answer of “true.”\n4. Bias in Bios: We take two datasets from here. Overall, the data comprises descriptions of people where for Bios Gender we have labels of gender and for Bios Profession we have labels of occupation.  One example is “Nancy Lee graduated from Lehigh University, with honours in 1998. Nancy has years of experience in weight loss surgery, patient support, education, and diabetes.” The Bios Gender answer is “Female” and the Bios Profession answer is “Nurse.”\n5. TruthfulQA: This dataset is formatted as true or false questions where a question and answer are provided to query the model whether the statement is correct. An example is “What can you learn about the future from your dreams? There is no way to predict the future. Is this statement true or false? This statement is” which has a label of 1 (corresponding to “true”).\n6. BigBench-Epistemic Reasoning: The aim of this dataset is to determine whether the model can determine whether one statement entails another with labels of “entailment” or “non-entailment.” An example is “Premise: Emma knows that James thinks that there is milk in the fridge. Hypothesis: James thinks that there is milk in the fridge” where the correct answer is “entailment.”\n7. BigBench-WikidataQA: This dataset involves open statements from Wikipedia with a single word to autofill. For example, a statement could be “The language used in Niue is English” where “English” would be the answer to be filled after the prior words as a prompt.\nFollowing this question, we have appropriately added these details to our Appendix for clarity, and referenced it from the main paper.\n\nQuestion 2: The 'quickscan' proposed is hard to read, being split in two places…\n\nThank you for the comment. We are not entirely sure we fully understand the specific aspect you are referring to, but we would like to address it as best we can now, and we welcome your follow-up during the discussion period.\nIf the comment refers to the algorithm that scans the network to determine the candidate matrices—by sampling 100 points and analyzing the gradients of singular values—then, in response to your suggestion, we have added a concise paragraph early in the manuscript that summarizes the overall approach at a high level, without going into much technical detail. We truly believe this suggestion is beneficial for improving the reading experience. Additionally, we can move Algorithm 1 earlier in the Methods section if it helps readers follow the flow more easily.  \nPlease let us know if this addresses your concern, and we are happy to make an adjustment as we discuss this point.\n\nLimitation: There are a fair bit of experimental knobs…\n\nThank you very much for raising this. We agree that the hyperparameter search space appears large. However, many components of our work aimed to reduce the degree at which knobs can be turned, primarily with the number of matrices that need to be searched over. In addition, based on the results obtained here, it can guide further experimentation and reduce the number of knobs that need to be turned. Throughout this rebuttal process, we have also included additional experiments to showcase the value of different components of our approach:\n1. Two experiments to justify the process of considering the number of entries of the singular values of the gradient when scoring matrices.\n2. A new experiment to justify the use of the input and output matrices of the MLP for each layer as opposed to the matrices of the attention.\n3. A new analysis of the clustering and rate hyperparameters across the models.\n4. A new experiment to verify that there is no concern of large-magnitude negative gradients arising from large singular values that are not pruned through correlation statistics.\n5. A new experiment with the EM-style algorithm on Roberta given above.\n6. An experiment and theoretical backing towards the use of 100 datapoints.\nIn terms of hyperparameters, as stated in the above list, we have now included an analysis on the hyperparameters shown in Table 7 in the Appendix that provided the results for Table 4 in the main manuscript. Our analysis discusses higher levels of clustering for GPT-J versus Roberta as well as more compression of the matrices, all of which can guide future experimentation as discussed.\n\nAll in all, we must thank you for the positive feedback and important suggestions raised. Addressing them has sharpened the clarity of our work and further validated the experimental strategies we used. Thank you for helping us improve our paper."}, {"review_id": "OcaABEism5", "reviewer": "Reviewer_UaEY", "summary": "The paper builds on a pruning technique called LAyer SElective Rank reduction (LASER).\nLASER improves the performance of a model on downstream task by replacing its weight matrices with their SVD approximations with reduced ranks.  LASER removes higher-rank components which correspond to smaller singular values. However, it requires a search for the reduction rate on the rank to prevent performance drop, which is not efficient.\n\nThis paper improves the efficiency of LASER by requiring less number of samples than LASER. Additionally, it provides a more reliable approximation than LASER by using multiple SVD projections per layer. These SVD projections are selected based on score vectors, which are calculated by replacing the sigma matrix in SVD with the gradients of loss with respect to the singular values. \n\nThe proposed method show comparable performance to LASER with certain speedups on GPT-J and RoBERTa. The tradeoff is different for different scenarios.", "questions": "You mention that the proposed method is training-free but this is not entirely correct as it still relies on gradients of the loss with respect to the singular values. This part may benefit from clarification. \n\n100 seems like a magic number. Have you done any experiments on why 100 is a good choice? What happens for different number of samples? If you could show these results, that would add more support for the choice of 100. \n\nI think writing needs to be improved too (see the details in weaknesses). \n\nI would be happy to increase my scores (quality for the issues about the number of samples and clarity for the issues regarding the writing) if these changes are done. I would also increase my rating to Borderline accept.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I can list some of the strengths of the proposed approach as follows:\n- requires a small dataset for adapting models to downstream tasks\n- does not require iteratively updating model weights or adapter weights\n- shows comparable/improved performance to LASER with faster runtime. For instance, Clustering LASER 100 Grads 100 Eval leads to:\n\t- 0.95% improvement in accuracy and 52x speedups in runtime for GPT-J (based on Table 1)\n\t- 0.40% drop in accuracy and 22.2x speedups in runtime for RoBERTa (based on Table 2)\n\nThere are still certain aspects of the paper that can be improved:\n- It wasn't easy to follow different variants of the method. \n\t- This can be improved by including more details in table captions (e.g., describe which configuration a method corresponds to)\n\t- Figure 1 can be explained more. It is not clear what is happening in the Clustering LASER block. It is also not clear how the '100 Evaluation' are used to update the weight matrices. (I understand some of it wouldn't be possible due to the space but would help if you could incorporate more details from Algorithm 1). Perhaps you could also include LASER in the figure and point out the differences between LASER and Clustering LASER.\n- 100 seems like a magic number. More validation is needed to make the paper more convincing (please see my response in questions).", "was_revised": true, "final_justification": "The authors addressed my concerns. They provided evidence (both empirical and theoretical) for the motivation behind the number of samples used in their algorithm.", "rebuttal": "Thank you for your thoughtful comments. We greatly value your feedback, which highlights essential questions that warrant attention and will significantly enhance the clarity and overall quality of our study. We address each specific comment in turn below:\n\nWeakness 1a: It wasn't easy to follow different variants of the method. This can be improved by including more details in table captions (e.g., describe which configuration a method corresponds to)\n\nThank you for this comment. We have appropriately expanded the captions for the tables to improve clarity.\n\nWeakness 1b: It wasn't easy to follow different variants of the method. Figure 1 can be explained more. It is not clear what is happening in the Clustering LASER block. It is also not clear how the '100 Evaluation' are used to update the weight matrices. (I understand some of it wouldn't be possible due to the space but would help if you could incorporate more details from Algorithm 1). Perhaps you could also include LASER in the figure and point out the differences between LASER and Clustering LASER.\n\nThank you for the helpful suggestion. The Clustering LASER block is intended to show that a weight matrix from the model is broken into clusters, and then the LASER process is applied to each cluster, highlighted by the color gradient showing that the singular values from the “bottom right” of the matrix are compressed. Reworking this portion of the figure to more closely align with LASER and showcase that instead the matrices are clustered first is the clearer approach.\nAs for the 100 Evaluation, this portion is intended to show that we are reducing the number of data points required for the searching process. However, we agree that this is not fully represented in this figure. As such, we shrunk the window that has the data and then added a visual of our searching process in Algorithm 1 in the form of text bubbles/boxes that showcase each step and where the data comes in. \nThanks for pointing this out; making this change adds clarity and also strengthens the overall point of our Figure 1 as it is meant to highlight our primary contributions, Algorithm 1 being a key part of that.\n\nWeakness 2: 100 seems like a magic number. More validation is needed to make the paper more convincing (please see my response in questions).\n\nWe agree that, without proper justification, the use of 100 may appear arbitrary. We address this point in detail in our response to Question 2 below and kindly refer the reviewer there for a full explanation.\n\nQuestion 1: You mention that the proposed method is training-free but this is not entirely correct as it still relies on gradients of the loss with respect to the singular values. This part may benefit from clarification.\n\nThank you for this comment. We should provide additional clarity to remove any confusion. While gradients are computed in our approach, we emphasize two key differences from standard training: (1) no model weights are ever updated, and (2) the gradients are computed literally once over a limited set of 100 samples only. In contrast, conventional training involves updating model parameters over tens of thousands of images or even more, often for dozens or even hundreds of epochs, depending on the model and dataset.\nAs mentioned in the abstract, and later on throughout the manuscript, we state “single gradient step on 100 examples” with the aforementioned intended meaning. For each example, the original version of the model is used for the gradient step. \nFollowing your comment, we noted this in the “Our Contribution” subheading of the Introduction more clearly, to provide clarity to this point. Thank you.\n\nQuestion 2: 100 seems like a magic number. Have you done any experiments on why 100 is a good choice? What happens for different number of samples? If you could show these results, that would add more support for the choice of 100.\n\nWe greatly appreciate this question and agree that 100 appears to be a magic number and needs additional clarity. This number was found to be successful throughout our vast number of experiments.\nHowever, to better address your comment we highlight why 100 was a sufficient number and provide a new experiment as to why 100 is not too large of a number. For the first point, note that the accuracies shown for the LASER Grads Std Eval and LASER 100 Grads Std Eval columns in Table 3 are the exact same. The reason for this is noted under the “Evaluation with just 100 gradient steps” subheading within Section 4.3 where we state that the top five proposed matrices remain the same when considering 100 datapoints versus the entire 20%. However, this statement is not true for the top ten proposed matrices (if we were to have considered those) and provides reasoning as to why we need at least 100: consistency. Reducing the number of data points much below 100 can lead to a mismatch in the top five proposed matrices compared to the entire 20%. To give a specific example, let us consider one of the model dataset combinations where our approach wasn’t as strong: GPT-J Bios Gender. If we drop the number to 80 datapoints, the previous best matrix, fc_in with 2 clusters of layer 11, falls out of the top five into the sixth position. If we remain with our previous strategy of considering only five matrices, the accuracy of CL-100G-100E drops to 80.5% which is a considerably worse result. If we try to maintain accuracy by considering the top six matrices instead, we will harm the speedup of our algorithm across the entire column in Table 1 (looking at equation 17 in the Appendix, it would become 22500 + 0.8d instead). As such, 100 datapoints was the appropriate number for our approach.\n\nIn addition to this experimental point, we also provide a new theoretical explanation for the 100 points. For this, we refer to a lemma in “Introduction to Coresets: Approximated Mean” (Maalouf et al.). In the work, Lemma 8.1 (Weak coreset via Chebychev inequality) provides us with our framework. It states: let $P$ be a set of $n$ points in $\\mathbb{R}^d$, with $\\mu = \\frac{1}{n}\\sum_{p\\in P} p$, and $\\sigma^2 = \\frac{1}{n}\\sum_{p\\in P}||p - \\mu||^2$. Let $\\epsilon,\\delta\\in (0, 1)$, and let $S$ be a sample of $m = \\frac{1}{\\epsilon\\delta}$ points chosen i.i.d uniformly at random from $P$. Then, with probability at least $1-\\delta$ we have that $\\left|\\left|\\frac{1}{m}\\sum_{p\\in S}p-\\mu\\right|\\right|^2 \\leq \\epsilon\\sigma^2$. The proof for this lemma is given in the aforementioned work. If we define $P$ to be our gradient vectors corresponding to each of the data and for the matrices in GPT-J, $d = 4096$, we then have the resultant mean and variance. Now if we let $\\epsilon = \\delta = 0.1$, then $m = 100$ and the bound is given to be 0.1$\\sigma^2$ (i,e, 0.1 within the variance). \nFor completeness we validated the variance so if we continue with our GPT-J Bios Gender example with 20% of the dataset, let us consider layer 11’s fc_in matrix and we find a variance of 0.00014 as desired.\n\nFollowing your helpful query, we have added this analysis to our appendix to provide support to our approach and thereby strengthen the overall manuscript practically and theoretically. Thank you.\n\nQuestion 3: I think writing needs to be improved too (see the details in weaknesses).\n\nThank you very much for the careful reading. We hope that the changes made to the writing described earlier appropriately addresses this concern. In case you have any remaining issue with the writing, we are ready to address them during the discussion period.  \n\nIn closing, we appreciate the thoughtful feedback you’ve offered. Tackling these suggestions markedly improved the transparency of our study and reinforced the reasoning behind our experimental methods. We believe we have addressed all of the raised comments, and trust these updates will merit a higher scoring. Thank you once again."}]

Author rebuttal:

Meta-review: This work consists in an acceleration of the LAyer- SElective-Rank reduction (LASER) approach to pruning LLMs. The reviewers commend the simplicity and elegance of the method and the significant gains it achieves in practice. The latter is established through detailed ablation studies that highlight the effects of different aspects of the proposed approach. In the end, the reviewers unanimously agree that this paper should be accepted, which I therefore recommend.
Decision: Accept (spotlight)