An Efficient Sparse Fine-Tuning with Low Quantization Error via Neural Network Pruning
Authors: Cen-Jhih Li
Aditya Bhaskara
Venue: Accepted by TMLR ()

Abstract: Fine-tuning is an important step in adapting foundation models such as large language models to downstream tasks. To make this step more accessible to users with limited computational budgets, it is crucial to develop fine-tuning methods that are memory and computationally efficient. Sparse Fine-tuning (SpFT) and Low-rank adaptation (LoRA) are two frameworks that have emerged for addressing this problem and have been adopted widely in practice. In this work, we develop a new SpFT framework, based on ideas from neural network pruning. At a high level, we first identify ``important'' neurons/nodes using feature importance metrics from network pruning (specifically, we use the structural pruning method), and then perform fine-tuning by restricting to weights involving these neurons. Experiments on common language tasks show our method improves SpFT’s memory efficiency by 20–50% while matching the accuracy of state-of-the-art methods like LoRA’s variants.

Published in Transactions on Machine Learning Research (11/2025)
An Efficient Sparse Fine-Tuning with Low Quantization Error via Neural Network Pruning
Cen-Jhih Li
cenjhih. com
Kahlert School of Computing University of Utah, Solt Lake City, 84112, USA
Aditya Bhaskara
ftteskarrddityy@@mmmi.ccm
Kahlert School of Computing
University of Utah, Solt Lake City, UT 84112, USA
Reviewed on OpenReview: https: // openreview. net/
net/forum? forum? id= 33b6775E2D
Abstract
Fine-tuning an important stop in adapting foundation models such as large language models to downstream tasks. To make this step more accessible to users with limited computational budgets, crucial to develop fine-tuning methods that are memory and computationally efficient. Sparse Fine-tuning (SpFT) and Low-rank adaptation (LoRA) are two frameworks that have emerged for addressing this problem and have been adopted widely in practice. In this work, we develop a new SpFT framework, based on ideas from neural network pruning. At a high level, we first identify "important" neurons neurons/nodes nodes using feature importance metrics from network pruning (specifically, we use the structural pruning method), and then perform fine-tuning by restricting to weights involving these neurons. Experiments on common language tasks show our method improves SpFT's memory efficiency by 20-50% while matching the accuracy of state-of-the-art methods like LoRA's variants. Code available at: pttps://github... http://ttb.cc.o//.aaaaaaaaaaaatt foitttyiinnnnnnnn
1
Introduction
The paradigm of pre-training followed by fine-tuning has seen tremendous success in the last few years. Very large models (often referred to as foundation models) are first trained, typically using very large amounts of data and computational resources, using self-supervised learning approaches (Dosovitskiy 2020: Achiam let al., 2023 Dubey et al.) 2024 Zhou et al.) (024)). When building a model for a new task (which could be supervised learning task), the idea is to start with the foundation model and then tune its parameters, possibly after adding additional classification layers, by training using task-specific data. The pre-train then fine-tune paradigm has been shown to have significant advantages over training new model from scratch for the new task. Often, high accuracy can be obtained using much smaller datasets for the new task.
Despite the success, fine-tuning a model with billions of parameters requires access to heavy computational resources, even when the task datasets are fairly small. Fortunately, studies (o.g., Panigrahi et al. (2023) and references therein) show that fine-tuning only small fraction of parameters can be effective. Parameter- efficient fine-tuning (PEFT) methods have thus been proposed to carry out this idea and address the challenge of making fine-tuning more accessible (Lialin et al. 2023). A leading PEFT approach, Low-Rank Adaptation (LoRA, Hu et al. 2022). achieves memory efficiency by simply making low-rank updates to the weight matrices in the different layers. Another class of PEFT methods is sporse fine-tuning (SpFT, Sung et al. 2021 Guo et al. 2021 Ansell et al. 2022 Nikdan et al. 2024 Guo et 2024b Panda et al. 2024 which learns a sparse matrix, typically an unstructured one, for updating the pre-trained weights. However, SpFT typically incurs
higher memory costs than LoRA during the fine-tuning process, because of the unstructured sparsity. Several
works aim to mitigate the memory complexity of SpFT (Mofrad al. 2019 Holmes et al. 2021 Nikdan

---

Published in Transactions on Machine Learning Research (11/2025)
et al. 2023 2024). often at the cost of increased running time and more complex implementations of sparse
kernels. Besides PFFTs, techniques like Zeroth-Order optimization (Malladi et al. 2023 Guo et al. 2024c
and quantization (Gholami et al. 2022 Dettmers et al. 2022 2024 can further enhance memory and training efficiency for fine-tuning, inchuding LoRA and SpFT.
As LLMs increase in scale, advancing efficient sparse matrix computation, PEFT, and efficient training remains crucial problem. Towards this goal, we study the question: Can sparse fine-tuning be improved to create memory- and parameter-efficient framework, while avoiding additional implementations of sparse operations and without increasing the training time complezity? We answer this question in the affirmative, by proposing new SpFT framework for fine-tuning LLMs that achieves memory- and paramcter-effiiennyy while maintaining or even improving performance on downstream tasks. Our approach utilizes NN pruning techniques to identify a subset of fine-tuning parameters and employs a matrix decomposition based computation for efficient fine-tuning. This design enables the integration of ideas from model compression, SpFT, and matrix decomposition methods.
1.1 Our Contributions
At a high level, our contributions are as follows:
We leverage ideas from network praning to improve SpFT, achieving a signilicant memory elficiency considerably lower than that of the popular ToRA. Our method uses only standard tensor operations, eliminating the need for custom sparse temsor operations. Additionally, our approach supports fine-tuning quantized base models to further reduce memory footprints.
We analyze the memory assignment of several PEFT methods and suggest that computation graphs can affect memory more significantly than the number of trainable parameters. In addition, we validate our methods in various fine-tuning tasks and provide practical guidance on training strategies.
We propose two variants of the Taylor importance for different settings image and language tasks: class-aware Taylor and Zeroth-Order Taylor. The first is designed for tasks where class-wise accuracy important (in addition to overall accuracy), such image classification. Zeroth-Order Taylor is designed for large language models and requires memory only equal to that of forward pass. In addition, we show how to effectively reduce the estimation variance of the Zeroth-Order estimator.
The rest of the paper is organized as follows. We discuss existing PEFT methods Scction 2 and analyze the existing problem in memory elficiency in Section 3 Following this, describe our approach in detail in Section 4 Section 5 describes the settings of our experiments. We then present and discuss our results in
Section
i Section concludes with some directions for future work along our lines.
2 Background and Related Work
Sinameter-Efficiinn and Memoey-Efficient Fine- Tuning: In various language and vision tasks, the "pre-train then fine-tune" paradigm has been shown highly elfective. PEFT methods [Lialin et al. 2023) fine-tune a small subset of the parameters of a large pre-trained model in order to accelerate the training process. We begin by introducing SpFT and LoRA, two popular approaches for PEFT.
Sparse Pine-Tuning SpFT formulates the fine-tuning process as learning another weight matrix 8:
W = W + 5, h f(W x) f(W 5,x),
(1)
(2)
where h E Rdot and x Rdw are the input and output of the hidden layer, respectively, f(.) is the forward function, W Rdou represents the frozen pre-trained parameters, and W R Xd.. denotes the final parameters used during inference for the fine-tuning task. The matrix 8 E Rdoee is sparse and is initialized at O. Such decomposition is done for every layer in the neural network. SpFT methods try
2

---

Published in Transactions on Machine Learning Research (11/2025)
to limit the number of parameters to fine-tune. For selecting non-zero indices, Diff pruning Guo et al. 2021) learns mask for å (using standard Backprop algorithm)) while FISH Mask (Sung et al. 2021 uses Fisher information to identify important indices in W. Lottery Ticket SpFT (Ansell et al. 2022) fine-tune the whole model for one epoch, then use 8 itself as an importance score to decide which parameters to fine-Lune subsequently. Robust. Adaplor (RoSA, Nikdan el al. 2024) combines the above SpFTS with LoRA and outperforms all these approaches. However, the key challenge of all SpFT methods that they do not sufficiently reduce memory usage, as ô keeps the dimensionality of W, and thus standard libraries do not yield memory improvements.
To reduce memory redundancy in sparse tensor com- putations, various data formats like compressed sparse column/ Sluun//rww row (CSC/CSR, Mofrad et al., 2019 Lu et al.
Techniques for Efficient Sparse Computation:
2024 and semi-structured formats (Holmes et al. 2021p have been proposed. These formats ena ble efficient operations like Sparse Matrix Multiplication (Sp.MM, which is crucial for dot products and matrix multiplications. Upon these techniques, sparse backpropagation is built to improve training efficiency (Zhang
et al.
2020 Gale al.
2020; Peste et. al. 2021 Schwarz et al..
2021 Hoeller el. al. 2021 Jiang et al.,
2022
Nikdan et al.) 2023 Xu et al. 2024). Beyond sparse tensor techniques, NVIDIA also offers memory
optimization techniques for efficient training
However, these techniques come with trade-ofis, particularly in terms of time complexity and implementation ation complexity. Achieving memory efficiency often requires a significant increase in time complexity. To mitigate this, some approaches employ optimizations implemented in C++ or lower-level languages, such as those used in (Gale et al. 2020 Nikdan et al. 2023 2024), to accelerate the training process.
Low-Rank Adaptation (LoRA): Instead of requiring 5 to be sparse, low-rank adaptation aims to find update matrices that are of small rank:
W = W + BA,
(3)
h f(W, f(W,x) + f( BA.x),
(4)
where is the LoRA scaling lyper-parameter B E Xdout A E Rexdia are the low-rank matrices with r < din, dout. During inference, the BA term can be merged into W to maintain the inference latency of the original model. During training, owing to the fact that ʃ is additive for both the self-attention blocks and the subsequent forwadinn neworks (FFN) of transformers (Vaswani 2017]. backpropagation can be performed efficiently for the B, A parameters. Due to LoRA's 's simplicity and effectiveness, numerous variants have been proposed to enhance the performance, e.g., QLoRA (Dettmers al.] 2022 Guo et al. 2024a| Li et al.
2024
Dettmers et al.) 2024), DoRA (Liu et al., 2024), RoSA (Nikdan et al. 2024), VeRA (Kopczkko and S-LoRA (Sheng et al. 2023). These methods have achieved exceptional performance, often
et al. comparable to full finc-tuning across a range of tasks.
2024
Neural Network Pruning: Besides PFFTs, neural network pruning is another widely applied technique that exploits parameter sparsity to reduce model complexity and speed up inference (LeCun et al. 1989 Han
ct al. 2015 Han 2017 Hoefler et al., 2021). Most pruming methods assess importance of neural network weights (or neurons) and remove the least important parameters. Unstructured pruning zeros out individual weights while preserving the network architecture, whereas structured pruning removes parameter groups like channels or neurons, which reduce model size (Liu et al. 2021 Fang et al., 2023 Ma et al., 2023) Both approaches often require retraining to recover lost accuracy during pruning. While effective for classical NNs, pruning LLMs is costly due to high memory demands for computing importance scores and the prohibitive
retraining step, making memory-efficient LLM pruning an active research area (Frantar & Alistarh 2023
Sun al. 2024).
Differentiate Our Work From Prior Works: Although prior SpFT and neural network pruning studies have explored training only subset of model parameters, they rarely highlight the substantial memory overhead and icaltical difficulties of unstructured sparsily. In contrast, our method uses structured row selection, enabling sparse fine-tuning with standard dense tensor operations. Furthermore, by selecting
Available https::///// http://///cccttatgttth/////////s
3

---

Published in Transactions on Machine Learning Research (11/2025)
entire rows, approach guarantees zero quantization error the fine-tuned rows in quantized models, whereas SVD-based techniques can only reduce, not eliminate, such error. In addition, to our knowledge, the Zeroth-Order Taylor method is the first to leverage zero-order estimation for computing neuron importance scores, with thcoretical analysis supporting its effectiveness.
3 Number of Trainable Parameters Is Not Everything
Memery footprints for fine- tete-tnning Llama 2 (7B)
74.72
74.56
75.00
gradients activations
param
dropout
60.00
53.33
52.84
51.67
model
47.49
46.75
45.00
40.52
39.14
40.18
37.87
35.83
34.06
32.16
32.91
30.00
15.00
0.00
LoRA DoRA RoSA VeRA
Ours
LoRA DoRA RoSA VeRA Ours LoRA DoRA RoSA VeRA Ours max token=512
max token=2048
max token=1024
Figure 1: Memory footprints (in GB) of fine-tuning full precision Llama2 using LoRA (Hu et al. 2022
RoSA Nikdan et al. 2024). DoRA (Liu et al., 2024h, VeRA (Kopiczko et al. 2024).. and ours. We set 32, d 1.2% for RoSA, r 128 for ours, and 64 for the others. Note that RoSA has its own official implementation, which may influence memory consumption, whereas LoRA, DoRA, and VeRA are integrated into the PEFT library provided by Hugging Face. More details please Appendix D.3
Before introducing our approach, we want emphasize that in PEFT research, reducing the number of trainable paremeters is not the most critical factor for minimizing memory consumption. While certain PEFT methods explicitly aim to lower the number of trainable parameters to reduce memory usage, the impact of this reduction diminishes once the parameter count is sufficiently small. To investigate this further, we compare several representative methods of SpFT and low-rank methods, focusing on their memory footprints during training, as shown in Figure For precisely, we assume full full-precision training to exclude memory costs introduced by quantization or other compression techniques and implementations. Notably, quantizing the model 4-bit precision can yield an additional memory savings of approximately 75%.
During neural network training, backpropagation requires caching large mumber of intermediate activations to compute gradients efficiently. The memory cost of these intermediate values is largely influenced by the structure of the computation graph. When the number of trainable parameters is small, the memory consumed by intermediate activations (see green bars in Figure 1 often dominates memory usage apart from the model weights.
Among the LoRA-based methods, VeRA attempts reduce memory by sharing a pair of low-rank matrices across layers, thereby reducing the number of trainable parameters. However, as shown in Figure this results
4

---

Published in Transactions on Machine Learning Research (11/2025)
in only marginal memory savings around 0.5GB to 2GB depending the maximum token length, which is almost negligible. In contrast, DoRA and RoSA incur significantly higher memory usage due to their more complex computation graphs and reliance on unstructured sparse matrices, For instance, DoRA decomposes LoRA's matrices into separate magnitude and direction components (sec Figure 4 Appendix D.4,, which substantially increases memory requirements for aclivation caching. While DoRA's 's trainable parameter cost. is similar to that of LoRA, its overall memory consumption considerably higher. RoSA also consumes much more memory than LoRA despite incorporating efficiency-orienteed design choices. These findings suggest. that simple computation graph can be far more significant contributor to memory usage than reducing trainable parameters.
4 Our Method
To address the challenges mentioned above, propose PruucturddPPuuing.e Sparse Fine-Tuning (SPruFT), as illustrated in Figure This is a novel approach designed to streamline compulation graphs and eliminate the need for implementing sparse tensor operations. This method ensures memory efficiency while maintaining competitive fine-tuning performance.
T1
i1
iz
W
W
W
AW
43
i4
is
W: dout X din
Importance score n]
AW:r x din
Figure 2: The illustration of SPruFT: evaluate the importance score for each neuron to select the fine-tuning indices. Then we construct the lower-dimensional fine-tuning parameter matrix AW.
4.1 Proposed Method
SPruFT utilizes structured neural network pruning to select a subset of the parameters for fine-tuning, NN pruning methods have been studied extensively (see Section 2 with the goal of reducing the size network (often fewer neurons) while preserving accuracy. These methods develop techniques for identifying neurons that are important for given task. Our key insight is use these importance metrics indicate which neurons to focus on during fine-tuning. Note that, unlike pruning, where importance often reflects neuron's 's role in the original task, here it pertains to the downstream fine-tuning task, which may have different input distribution and loss. In Section 4.3 we discuss various importance metrics from pruning rescarch and discuss their use in fine-tuning.
Our method selects the top-r important neurons based on an importance score where r is determined by the desired number of fine-tuning parameters. follows that the choice of importance matric bocomes crucial, which we discuss in Section 4.3 Let the top r neuron indices be i .g........ After obtaining 77, we next construct lower-dimensional parameter matrix AW Rrxdi., with the row selection matrix M for all j E [r] and zeros elsewhere, where M can be viewed as mask. Using notations from Section 2 we initialize A W to zero and define the final parameters W as Equation where 8 = MAW.
Let us now examine how to implement the forward to make backpropagation memory-efficien If the computation graph were to pass through W + MAW (as a naïve implementation would), the gradients would be computed for all d,, dout parameters, which is redundant. Instead, we use the additivity of the
While updating the corresponding rows of is the most efficient training, updating & provides more flexibility for adapting multiple tasks, see discussion LoRA Hu al. 2022

---

Published in Transactions on Machine Learning Research (11/2025)
forward function: we have, analogous to the setting of LoRA,
x(W,))= = J(W MAW, x) = J(W,)( J(MAW,)),
(5)
As W remains frozen during fine-tuning, backpropagation only needs to keep track of the derivatives of the second term on the RHS. In addition, M is now a fired matrix, so the only trainable parameters are those in AW and S(AW,, x) will not be cached, while LoRA requires the cache of f(A, x) for computing BB (backpropagation, Rumelhart et al. 1986). Besides, as an SpFT framework, our method does not rely on any dropout layer, which also saves huge amount of memory. We explain this in detail Appendix D.5 and we show that the benefits in terms of memory cost are significant in Section 6.1
2h
4.2 QSPruFT: Extension Approach with Low Quantization Error
Our approach is model-agnostic, allowing users to integrate recent advances in PEFT such as QLoRA (Detumers et al., 2024), LoftQ (Li al.) 2024). and QPiSSA (Meng et al. 2024]. QLoRA quantizes the base model Normal Float 4-bit (NF4) and fine-tunes full-precision LoRA matrices A and B. Based on QLoRA, LoftQ
and QPiSSA propose alternative initialization strategies for A and B to reduce quantization error. This error is defined as:
W W - WNFI
(6)
and both LoftQ and QPISSA use singular value decomposition (SVD) to initialize A and B such that Binit A init = Wres.
In our method, when applied to quantized base models, we do not require decomposition to approximate Wres, Since we fine-tune only selected rows, we can directly initialize AW using the corresponding rows from W where the quantization error of the fine-tuning rous is zero. This offers potential advantage over QLoRA, QDoRA, LoftQ, and QPISSA in terms of accuracy.
4.3 Broader Impacts: Importance Metrics
Importance evaluation plays crucial role in our approach, discussed above. We try various choices in our work: the first is the simple la norm of the weight vector corresponding to each neuron; the second is the uidely-used Taylor importance (LeCun et al. 1989). By considering the gradients, Taylor importance captures more information about the input distribution as well as the relevance of a neuron for the fine- tne-tuning task of interest (which can be different from the original model). We also consider different variants of Taylor importance, as we discuss below. We remark that norm-based importance can be quite powerful on its own, as is the case with sampling in the matrix approximation literature (Frieze et al. 2004).
4.3.1 Class Aware Taylor Importance
In our experiments on image classification tasks, we also consider a "class aware" variant of Taylor importance, which may be of independent interest. The motivation here comes from the observation that the importance of a neuron may depend on the class of an input, example (as toy example, a whisker-detecting neuron may be very important to the cat class, but not much to others; hence not too important on average). Another motivation comes from the observation that when we perform vanilla. (class agnostic) fino-tuning, the accuracy of some classes can be much worse than others an undesirable outcome. This is shown in Table I
We define the class-wise Taylor importance as follows: for neuron i and label t,
o = ((D',A L(D', F)] R
T T L(' P)),
(7)
where F' is the forward function of the entire model, L(D", F) denotes the average loss of F over inputs in class t, E, represents the forward without channel/neuroe i, and (()) is the parameter vector of channel i. One natural choice of importance of neuron motivated by the above discussion max, n/- We find that this score "too sensitive" (importance of neurons may be over-estimated because of just one class), leading to lower overall accuracy. On the other hand, the average (over t) of of corresponds to the standard
6

---

Published in Transactions on Machine Learning Research (11/2025)
Table 1:
The distribution of accuracies across different labels is sunmarized by statistics including the minimum (Min), first quartile (Q1), median (Med), third quartile (Q3), and maximum (Max) accuracies. #labels is the number of labels. The reported accuracies are the validation results of full fine-tuning DeïT for 5 epochs. Models and Datasets are described in Section 5
Q1
Q3
CIPAR100
100
90.18
65
88
82
95 98
Iing-In
87.56
62
100
Taylor importance. We find that the intermediate quantity of Quantiles-Mean, defined the average of the 0%, 10%, 10%,2.%,........................ ..... 100% quantiles of the ni, works well in reducing the accuracy imbalance across labels, and also achieving a high overall accuracy. Formally,
= El Qi((n:
(8)
11
where Qi represents the I 10-th quantile. See Appendix A for more details.
4.3.2 Zero-Order Taylor Importance
As discussed, Taylor importance can incorporate information about the data distribution and the finc-tuning task when evaluating important neurons. However, for large models like Llama-3, it turns out that the computational overhead required for computing Taylor importances prohibitively large In these cases, we apply the idea from the memory--fficient zeroth-order optimizer (MeZO, Malladi al. 2023 estimate the gradient in Taylor importance. The classical Zeroth-Order (ZO) setting is defined below.
Definition 4.1 (Simultaneous Perturbation Stochastic Approximation or SPSA (Saal 1992 ) Given a
model with parameters 0 E Rd and a loss function L, SPSA estimates the gradient on a dataset D as
(L L(O + ez) L(0 -6%) z.
2e
where Rd is drawn from NN(O, and is the scale of the perturbation. The n-SPSA estimate averages VL, D) over randomly sampled z. Note that as e + 0, the estimate above converges to z(z L D)), which is equal to (L D) in expectation.
By applying SPSA, the Zero-Order Taylor (ZOTaylor) importance can be defined as follows:
7 ) (+)) st ((())) T
where we denote (V, D)] and its estimate as g and & for convenience, and 7h, (O())), and g()) are the importance score, the parameter vector, and the gradient vector for neuron í.
We now assess the efficiency and effectiveness of ZOTaylor for our LLM. For efficiency, a naive implementation of SPSA still requires twice the memory of inference because of the need store z. However, MeZO uses the trick of regenerating dynamically using random seed (of much smaller size than the model), thus eliminating the storage and ensuring memory usage that is equal to that of inference. We then justify the effectiveness of ZOyaylor in the following.
,0 where Property 4.2. n-SPSA is an unbiased and consistent estimator with the variance vector (of. ..,
1-1 gi
n
Property 4.2
can be proved by simply noting that the covariance matrix of z is Id, details can be found
in Appendix B
With this property, we know that n-SPSA can accurately estimate the gradient. However,
TThe Taylor importance here refers computing the exact value without relying on approximations the importance score the gradient matrix used deriving the importance score.
7

---

Published in Transactions on Machine Learning Research (11/2025)
the variance can be large when d is large. Here, we first note that since aim to find the most important neurons, we do not care about the gradient estimate itself. That is, given th > Mig: our goal is to have a higher probability p pf - fh2 > 0) which is exactly equal to:
lg(l))((i]
((()))((2))
PZ-N(0.1)
Z>
(10)
Se 030 Σ 2 +
Exerotiph 02 /2
2
where {f(ts) and {j(tz)} } are the indices of neuron i and i2, and d, Ki( K2 The trivial lower bound of Equation 10 is 0.5 and the probability will close to 0.5 when the variance is large or when ti and 7h, are too close to one another. In our experiments, the values of & from single SPSA are almost uniformly in [-100, 100] with variances ranging from 102 to 10 and the values of 0 are mostly in [-1, 1], thus the probability above turns out to be too close to 0.5.
To address this issue, we utilize a simple but highly effective strategy (J Reddi al. 2015:: we partition the training data into l calibration sets. For each calibration set, we generate distinct perturbations to perform n-SPSA producing x k gradient estimates. Consequently, the variance of n-SPSA is effectively reduced to g >
Theorem 4.3. Suppose a model parameler 0 salisfies 0 S ug and a gradient g salisfies Ig S 1lg for some parameter ug. Also assume that ni 's are drawn IID from distribution Pn that is a-smooth and supported in the interval [0, un]- Let vh, and ni, be the importance values of the top-q1 % and top-q2 % important neurons and define to be the CDF of Gaussian distribution, then we have
2/nk(n) (n; mii) (dd, d1 u.3u
2Vnk(l0= - 91)/100 / (dr, + di, (u.)
p ( - ni 0) 2 PZ~N((.1) Z>
2 1 +
(11)
Theorem 4.3 follows directly from the variance expression oj provided in Property 4.2 detailed steps are included Appendix Using this result, we now show how estimate the required value of Vnkk reliably identify most of the desired neurons.
We argue that for any if we select the top-5% important neurons according to the á values (for some & E [0, 100]), most of top-5% neurons according to the true n values will be correctly selected with high probability, when nı is large enough. To formalize this, let ig = Too dn where dn is the number of neurons and let 7 2 7/2 ... 2 Mdy Let X; be the random variable that is if f is among the top ig of (ú) )ejd,) and 0 otherwise. Define p. to be p =).
We can now bound p(X, 0) as follows. If X; for ig, then least one of the values +1 f.tt...... ,d,, say 9 must be ip. Thus, for all S ig
dn
p(x, = 0)
II 1m
p(. > 1,) 51 -(h
nie dn = pi p( n din
(12)
The next result bounds the expected number of the "correct" neurons (or indices) chosen, in terms of n, Theorem 4.4. Let E (0,1) 1] be any desired error ratio, and let X; be as defined above. Then there exist a positive constant ca,, such that the incquality E X1) 2 (1 (c) ig holds, long
nk ca, a / (d,, + di, 12
(13)
WWe note that this formal guarantee; indeed, if is toa large dataset too small, there additional variance across calibration that will become significant.
8

---

Published in Transactions on Machine Learning Research (11/2025)
Theorem 4.4 can be proved by using Equation 12 and Theorem 4.3 detailed steps given in Appendix
Intuitively, Theorem 4.4 indicates that larger value of nk leads to more accurate estimates of importance scores. This increases the likelihood of capturing truly important neurons, which in turn may result in better fine-tuned models.
Unfortunately, the above threshold of vikk is too large in our experiments. To balance the variance of estimates and computation resources, we set nk = 2048 in our experiments. The error <ę=10 is likely to be higher than 20% under this setting, but our experimental results show that this setting is good enough to boost the fine-tuning performance.
5 Experimental Setup
Main experiments:
For language tasks, we fine-tune LLaMA-2-7B and LLaMA-3-8B in both full-precision (float32) and NF4 (Dettmers et al. 2024) on the training splits of 8 commonsense reasoning datasets (see Appendix D.. and evaluate them on their respective test splits. We then assess the models' mathematical reasoning by fine-tuning them the GSM8K (Cobbe al. 2021) training split and evaluating performance using the
EleutherAl LM Harness (Gao et al. 2021p. Finally, we evaluate instruction-followinng ability by fine-tuning
on Aapaca-GPTI (Taori et al.) 2023) and scoring responses with MT- (Zheng et al. 2023). using Gemini
2.5 flash TTam et al.
2023 as the judge
We incorporate the PEFT approaches including our SPruFT, QSPruFT, LoRA Hu et al., 2022
QLoRA (Dettmers el al. 2024 VeRA (Kopiczko et al. 2024). DoRA (Liu et al. 2024), QDoRA, RoSA Nikdan et al. 2024), LoftQ {Li et al. 2024), and QPiSSA (Meng et al., 2024]. RoSA is chosen as the representative SpFT method and is the only SpFT due to the high memory demands of other SpFT approaches, while full fine-tuning is excluded for the same reason. We freeze Llama's 's classification layers and fine-tune only the linear layers in attention and FFN blocks.
For training configurations sctting, we use a learning rate of 10-4 with lincar decay (rate 0.01) for all mothods, and we set a = 16 and dropout rate equal to 0.1 for LoRA and its variants. All methods apply linear decay after a 3% warmup. For commonsense reasoning, we train on 2048 samples (256 per dataset) and evaluate on 500 test examples per dataset. For GSMSK, we fine-tune on 2048 random training samples and evaluate the full test set. For Alpaca-GPT, we train the models on 1024 samples and evaluate the models on the whole MT-bench. Instruction-styyl prompts are used for commonsense datasets while GSM8K uses question-answering prompts. We fine-tune all models for 3 epochs.
Our framework is built on torch-pruning (Fang et al. 2023 PyTorch (Paszke et al. 2019). and HuggingFace Transformers (Wolf et al. 2020). Most experiments are conducted on a single A100-80GB GPU, except DoRA and RoSA (at 2048 max tokens) which run on an H100-96GB. We use the Adam optimizer (Kingma & Ba 2015 and train with fixed epoch budget without early stopping.
Experiments for importance metrics: In the comparison of different importance metrics, we also use our approach to fine-tune DeiT Touvron et al., 2021), ViT (Dosovitskiy 2020), ResNet101 (He et al. 2016), and ResNeXt101 (Xie el al. 2017) on Tiny- ImageNet [Tavanaei] 2020), CIFAR100 (Krizhevsky el al, 2009), and Caltech101 (Li et al.) 2022). For these tasks, we set the fine-tuning ratio at 5%, meaning the trainable parameters are a total of 5% of the backbone plus classification layers. Following this, we discuss the computational resource requirements for fine-tuning.
For image models, the learning rate is set to 10-4 with cosine annealing decay [Loshchilov & Hutter 2017) where the minimum learning rate is 10-9. All image models used in this study have been pre- traited on ImageNet. Note that memory efficiency is not emphasized for small-scale models, as dataset-related memory particulayy with large batch sizes dominates consumption in these cases. The main advantage of our method in these cases is the reduced FLOPs duc to fewer trainable parameters.
We Gemini 2.5 instead of CPT-4 due to lower cost and competitive performance. A-L performs well with aanswernnn prompts, and fine-tuning yields limited gains, suggesting possible pre- pre-training these datasets with question-answering prompts.
9

---

Published in Transactions on Machine Learning Research (11/2025)
Table 2: Main results of fine-tuning full precision Llama2 and Llama3. "mem" represents the memory cost in training excluding the model itself, with further details provided in Appendix D.3 #param is the number of trainable parameters. HS, OBQA, and WG represent HellaSwag, OpenBookQA, and WinoGrande datasets. All reported results for commonsense reasoning tasks are accuracies, while the results for GSM8k are the exact match score. The ablation study for different r can be found in Appendix D.6 All reported results are accuracies on the corresponding tasks. Bold indicates the best result on the same task.
Modal, wetting
#param
BoolQ
PIQA
SIQA
WG
ARC-c
ARC-a
OBQA
GSMak
Llam)2(7B), protzoined
58. 00.00
40.00
20.00
15:40
4.80
14.00
16.20
25.40
20.35
0.00
LoRA,
100.0M(3.3
08.60
1.374M10
DeRA,
86GB
52:20
RoSA, 32.d 1.2%
167.7b122
70.20
57.00
68.50
80.60
72.20
21:99
SPruPT, 128
17.62GB
145.8M (6%))))
80.00
67.60
85.00
63.40
70.80
82,40
73.80
74.53
22,90
LSB) pretzained
08. 80
41.00
88:00
10:20
11:20
00:20
68:00
27:40
38:18
0.00
LoRA, 64
87GB
167.8M12 (97))
$4.20
77:00
63.20
84.20
07. 27.20
76.40
$8.
71:00
76.50
41,77
VeTAA
29. 4BGB
1.30:M (0.02%))
62.40
5.5660
4.1.80
89.60
77.60
60.00
58.44
0,00
DeRA, 64
51.4b 4BBB
189. (2 11
20.20
82.80
69.00
89.40
70.80
77:20
89.00
80.40
80.23
46.02
RoSA, 52,
1.2%
48. 40GB
167.6bl .09%)
79.
81.00
69.20
84.80
88.60
79. .0
90044
78.40
78.80
45.72
SPruPT, 128
24.49GIB
159 98%)
87.00
77:40
71.40
85-40
70.30
79.80
90.80
81.80
80.55
46.10
6 Results and Discussion
We now present the results of fine-tuning image models and Llamas using framework. We first apply our SPruFT to finc-tune Llama2 and Llama3 and compare the results with those obtained using LoRA and its variants. Following this, we examine the performance of our approach by utilizing various importance metrics.
Table 3:
Comparing fune-tuning all linear layers with fine-tuning only the FFN on commonsense reasoning tasks. $ indicates that we freeze the layers for queue, key, and value projection. Full table please refer to
Table 19 in Appendix D.7
Modala
Llama2(7B)
Lla
Satsing
#param
Common
#param
Common(
LoRA, r
21.6GGB
40 (M((0.09%)
72.48
28. 88.8
41. 41.95((0.622)
78.95
LoRA 82
17.95GB
54 #M(0.811%
T2.00
B
65. (0.88))).81)
75.35
LoRA, r 32
23.21GB
50. (X()..99%)
72.03
29,37GB
83. 83.9M() 00%))
79.50
LoftA), == 64
18.81GB
109. 109.8M{(1 638%))
73.00
26.04GB
130. (M0. 1.62%)
77.73
SPruFT, 32
15.5YGB
36 4M(0..44%)
72:25
22.62GB
39 8M((0.0%))
76.53
SPrePrt, < 64
14.67GB
4T 71(0)71%)
72.45
21.81GB
34. (0.666))
79.05
SProFT, SPruFTi, - 128
16.20GB
9MM .08%)
7a.7M(0.99%)
16.68GB
96. 4M(1.42%%
73.98
22.710B
109. 366)
60.45
6.1 Main Results of LLM
We apply our SPruFT method to fine-tune Tlamma--B and Llama3-8B, comparing the results with those obtained through LoRA and its variants. We select the magnitude of the neuron vector as the importance metric due to its low memory requirements, simplicity, and widely tested effectiveness. In contrast, gradient- based metrics like Taylor and Hessian are memory-intensive full LLM fine-tuning While Wanda (Sun) ct al. 2024h and AWQ (Lin et al.] 2024) offer two activation- metries to evaluate neuron importance, for pruning LLMs, they require one epoch of data forwarding and significantly more memory than inference to compute the activation-bass importancd
We encountered OOM error when using Wanda's official implementation. When pruning neural network, each layer computes activation vector's norm and pruned immediately, gradually reducing the model size. However, in the fino-tuning process, the model size remains unchanged. Additionally, storing the activation values for computing importance scores further increases memory consumption, making memory cost significantly higher than when using activation for pruning.
10

---

Published in Transactions on Machine Learning Research (11/2025)
Table 2 demonstrates the exceptional memory efficiency of our approach[while achieving comparable accuracy. As shown, the accuracies of fine-tuned models remain similar across most PEFT methods, while memory usage varies significantly, VeRA, despite having significantly fewer trainable parameters, shows noticeably lower aceuracy. Notably, our approach consistently requires substantially less memory than all other PEFT methods listed in the table.
Table 4: MT-Bench results of fine-tuning full precision Llama2 and Llama3 on Alpaca-GPT4. Bold indicates the best result on the same task.
Modal, wotsing
Coding
Retraction
Hamanities
Roleglay
Stemm
Writing
L7am pretzaiued
1.00
2.63
1.60
1.50
1.39
1.65
1:70
1.30
1.66
4.50
VeRA,
DeRA, 1.2%
RoBA,
1.73
3.35
3.94
SPruPT, 128
1.82
2.53
2.08
4.07
4.79
4.50
3.33
3.52
LoRA!,
2.36
1.89
4.89
3.36
4.30
4.05
3.84
3.33
DORA r
1.10
1.75
4.26
2.40
8.62
4.59
4.21
3.58
8.23
RaSA 52, di 1.2%
2,00
3.11
5.80
2.50
4.45
4.50
4.11
3..74
3.75
SPruPT:
Llamm3(8B), preteniued
3.50
6.66
2.96
1.82
2,00
2.84
3.56
3.70
8.19
LoRA,
5.72
5.22
DeRA, 1.2%
RoSA, SPouPT, 128
3.88
5.11
3.83
6..44
5.40
0.04
LonAt,
3.75
4.78
4.42
4.75
5.15
5:006
5.95
5.12
4.25
4.00
DoRAF, 64
8.90
6.66
0.75
4.42
8.82
0.50
0.88
3.05
5.30
RoSAF, SPouPT:, 1.2%
3.63
5.88
4.45
2.92
5.32
8.87
5.30
5.05
We then demonstrate that strategically assigning trainable parameters saves more memory than merely reducing them, without compromising accuracy on commonsense rcasoning tasks. Section highlights the importance of the computation graph in memory consumplion. Strategically assigning trainable parameters can be an effective solution. For instance, Shi et al. (2024) suggest that layers such as output, down, up, and gate projections contribute more significantly than queue, key, and value projections, which leads to the strategy of freezing queue, key, and value projections. We compare tinettuiigg-a all linear layers and fine-tuning only the FFN and oudpul-project layer, with results shown in Table 3 The former requires more memory for the same number of trainable parameters, as distributing trainable parameters across the model increases the need for caching intermediate values. Table 3 shows an exceptional memory saving by freezing some layers in attention blocks. In addition, the accuracy remains nearly unchanged. Given that Llama models have been pre-trained extensive datasets, their attention layers likely already capture crucial patterns for token interactions.
We further evaluate models' instruction- following abilities using MT-Bench. As shown in Table ! SPruFT with magnitude-basod importance falls short of other methods. However, incorporating ZOTaylor significantly improves performance: Table 5 (Section 6.2 shows that SPruFT with ZOTaylor outperforms most approaches on cittructioonfollowiii tasks, ranking second in all cases. Only RoSA performs better overall, but it requires substantially more memory than our method.
6.2 Importance Metrics
We apply various importance metrics to finc-tune Llamas and image models using our approach and report the results to compare their performance. As shown in Table 5 and Table 0 Quantile-Mean Taylor and ZOTaylor offer slight improvements over other importance metrics. For image tasks, while the differences among importance metrics are not substantial, the results consistently indicate that Quantile-Mean Taylor slightly outperforms standard Taylor importance. Additionally, both Quantile-Mean Taylor and standard Taylor imporlance outperform magnitude- baase importance.
8 Also refer Table 16 in Appendix D.6 even with 128, our method's memory usage remains significantly lower than
that of LoRA with
11

---

Published in Transactions on Machine Learning Research (11/2025)
Similarly, in the cases of Llama2 and Llama3, our findings suggest that ZOTaylor provides a slight performance boost for fine-tuned models. This improvement is likely due to ZOTaylor's ability to capture richer data information, whereas magnitude- based importance tends to focus more on identifying generally important neurons. However, the observed performance gain remains modest, potentially due to the variance of the estimales, as discussed in Section 4.3.2 Beyond these observations, an interesting finding that models fine-tuned with random row selection significantly outperform VeRA, likely suggesting that overly aggressive parameter reduction can substantially degrade performance.
Table 5: Importance evaluation (random (rd) vs l vs ZOTaylor (70)) for Llama2 and Llama3 on commonsense reasoning tasks, GSM8k, and MT-BBech. We also present the results of freezing queue- key-, and value- projection in this table (t). Full table with different ranks for commonsense tasks please refer to Table 17
and Table 18 in Appendix D.6
Model
Llama2(7B)
Llamm3($B)
Bettings
Comment (Arg))
GSM8K
MTTBench(AAr))
Common
GSMsk
AFT-Benca(Avx)
128,
69.58
22.32
3.45
76.08
42. 12.33
4.40
2
20
128 ed
72.08
21.08
3.60
77.20
39.
5.00
128
2a
73.98
10:48
42 12.58
4.97
128 ZO
74.30
24.34
3.60
81.80
46.02
5.21
Table 6: Importance metries on fine-tuning image models by our SPruFT for 5 epochs. FFT, & Taylor, and QMTaylor represent full fine-tunin, the magnitude, Taylor importance, and Quantiles Mean Taylor importance (Equation S). Note that QMTaylor not applied to fine-tuning Caltech101 due to its significantly imbalanced labels. All reported results are validation accuracies. Bold indicates the superior results achieved through different importance metrics.
model
imp
CIPAR100
Fmayy-ImogeNNl
Cultecha101
DefT
FFT
$7.66
97.88
2
88.05
89.31
915.
Taylor
89.70
59.60 60
95.44
Hesd ars
85.73
58 66
85:10
QMTaylor
89.37
89.75
22
87.18
Taylor
85.08
90.87
93.96
Bossinm
87.63
90.56
93.70
QuTTallr
88.51
90.00
RN
FFT
86.21
77.78
96.60
2
Taylor Hessina QMTaylor
79. 67
02,
83.50
80.02
RNX
FPT
87.30
79.51
87.
22
96.
QMTaa
54.17
6.3
Dropout Comparison and r Sensitivity Analysis
We investigate the role of dropout in SpFT and LoRA to assess whether comparing approach without dropout to LoRA with dropout is appropriate. While prior works on SpFTT (Sung et al. 2021 Guo et al. 2021 Ansell ct al. 2022 Nikdan et al. 2024 Guo et al. 2024b Panda et al. 2024h have already compared SpFT without dropout to LoRA variants with dropout and found that SpFT is generally less prone to overfitting, we perform additional experiments to ensure that this comparison is fair. The key distinction is that SpFT itself constitutes a form of sparse training, which inherently provides regularization effect. Introducing dropout into SpFT may therefore lead to rvgr-eegulaiization By contrast, LoRA and its variants often benefit from dropout, it help mitigate overfitting The results in Table and Table confirm
12

---

Published in Transactions on Machine Learning Research (11/2025)
Table 7:
The influence of dropout on LoRA. Details on memory footprint and #param are provided in
Table [6
0.5%
1%
8%
gemsk
gpemale
gemsk
gambk
gemakk
grm8k
Liama2(7B)
dropont
drapont
72.78
Llama5(sB)
dropout
drapout
10.56
76.73
Table 8: The influence of dropout our approach, including importance metrics random (rd), ez and
ZOTaylor (70). Details on memory lootprint and # param are provided in Table 16
Outa
16, 0.26%
32, 0.5%
64, 1%
128.
2%
260, 4%
612, 9%
commmon
k-m8k
K-inSL
common
K-mSk
Llawa2(7B))
rd, w/ dropout
66.
14.71
69.33
16:40
06:70
19.94
68.03
21.61
68.20
20.32
65.00
14.45
rd, w/o dropous
60.75
15.85
00:80
18:35
69.45
20.39
09:80
22.52
69.23
18.88
08.00
17.00
w/ drepout
67.00
15.09
70.50
17.06
70.60
21.53
72.88
22.80 32.50
13.20
19.03
73.60
17.50
w/o dropout
67.88
16.81
72.26
18.27
72.05
21.00
74.68
22.00
73.08
20.02
73.08
17.74
ZO: dropout wy/ dropast
16.77
71.80
23.48
70.25
21.00
73.83
00
14.08
22.21
73.23
17.18
17.82
(SB)
rd, dropout
32:15
76.35
78.
74:10
36.62
e. w/ dropout
73.53
31.48
77.75
43.29
77:03
43.75
79.78
45.79
17:70
36.77
77. 93
31.72
1 w/o dropaut
80.53
20, w/ dropout
78.10
36.39
77.05
47.38
78.00
49.73
79.73
52. 06
79.25
49.98
79. 19
42.15
20, w/o dropout
78.35
$7.53
78.53
44.50
78.00
50.19
81.28
55.12
80:20
50.34
79.98
47.54
these tendencies:
SpFT generally does not rely on dropout, whereas LoRA shows some improvements when dropout is applied. In fact, dropout brings little to no benefit in our method and can even slightly degrade performance, while in LoRA it typically yields gains.
Table 7 and Table N also illustrate the effect of sparsity ratio and rank. When the sparsity ratio is very small (0.25%), our approach performs less effectively, and LoRA achieves slightly better results. As the sparsity ratio increases, performance improvements diminish, suggesting that the range [0.5%, 2%] may be the most suitable. Somewhat surprisingly, performance decreases when the rank r is large (256 for LoRA and 512 for our approach). We attribute this to the relatively small size of the fine-tuning dataset compared with the model size and the pre-training corpus. With more trainable parameters, the model is more likely to overfit. This observation is also consistent with the fact that many prior works using LoRA select rank values between 32 and 128, further supporting the view that PEFT can sometimes outperform full fine-tuning.
Table 9: Main results of tuning NF4-quantized LLaMA-2 and LLaMA-3. QSPruFT* denotes our approach with AW initialized randomly, while QSPruFTi uses AW initialized with the corresponding rows from the quantization residual Wres.
Model, De acsting
BoolQ
PIQA
SIQA
H3
ARC-c
ARC-c
OBQA
Avg
GSMBI
L1Bm protratned
37.00
ST.
00
19.20
20.00
4.40
15.40
17.80
20.30
21.48
0.00
pretrained--PP
58.00
40.00 00
29.00
15.40
4.50
14.00
16,20
25.40
25. 35
0.00
QLORA,
75.30
63.
77.60
70.00
QDoRA, 64
72.22
73.50
67.00
QPISSA. 64
79.40
66.00
72.33
QSPruFT". QSPruFT!, 128
78.00
63.20
67.20
79.60
T4.90
72.98
82.00
71.80
69.20
83.20
65.40
67.20
79.20
72.40
73.60
22.14
Mamaa(an)) prstratned
58.80
60
35.000
10.30
11:30
55. 25
63.00
97:40
3/8 18
0.00
pretssined-NPP N
62.00
37.80
11.60
8.40
0.20
45.60
63.50
16.20
28.00
0.00
QLoRA, QDRRA,
77.40
72.80
87.60
89.60
75.50
90.20
79.20
41.17
70.00
20
68.20
52.20
06, 66.40
70.50
77.20
QPISSA.
QSPruFT',
QSPraPT! 128
90.40
82.40
68.60
89.40
89:20
76.80
50.80
76.40
80.50
49.13
13

---

Published in Transactions on Machine Learning Research (11/2025)
6.4 Results of Quantized Models
Tables and 10 present the results of fine-tuning NF4-quanlized models using various PEFT met In these quantized experiments, we use magnitude-basee importance metric to save processing time, as quantized parameters increase latency during training and inference. The results show that our method, with A W initialized from W consistently achieves strong performance across tasks. These findings support that SPruFT simple, effective, and memory-efficient fine-tuning strategy.
Table 10:
Main results of fine-tuning NF4-quantized LLaMA-2 and LLaMA-3. QSPruFT^ denotes our approach with AW initialized randomly, while QSPruFTi uses AW initialized with the corresponding rows from the quantization residual Wres.
Modal, it setting
Coding
Extraction
Humanities
Math
Reasoning
Roleplay
Stem
Weiting
Avg
Llama2(7B), pretratmed
1.00
1:65
1.59
1:85
1:70
1:55
1.86
pretratoed-NFF
1.00
2:84
1:60
1.87
1.85
1.80
1.40
1.87
QLeRA,
1.00
5.06
4.74
1.79
3.23
5.15
4.69
3.55
3.40
QDoRA r=
1.46
3:78
5:43
1.92
3.63
4.63
4.63
4.05
3.09
LofsQ, ==
1.46
3.90
5:35
2.45
4.69
4.50
4.37
3.89
3.87
QSPeuPT" = 128
0.88
3.75
3.05
2.29
2.77
4.58
3.82
4.40
3.52
QSPcuPT],
1.92
4.70
0.45
4.57
5,20
4.00
4.14
Llama3($13), pretrained
3.06
5.56
2:05
1:32
2.00
2:54
3:56
3:75
3.10
pretrained-NP4
1.57
4.32
2:35
1:75
9:50
2:50
3:11
2:91
9.58
QLaRA, @DoRA,
4.43
5.41
4.74
3.00
4.79
5.65
4:74
4.65
3.56
6.26
6.00
4.03
3.83
5.95
5.28
5.04
LofiQ. --
4.30
1.00
0.60
6.65
3.73
5.90
5.60
0.30
6.02
Q8PeuPT" 128
2.43
3.74
4.26
2.00
3.40
3.58
3.69
3.65
3.51
Q8PruFTI.
7 Conclusions and Future Work
We propose a parameter-effcient fine-tuning (PEFT) framework that integrates various techniques and importance metrics from model compression rescarch to enhance sparse fine-tuning (SpFT). Using method, we can fine-tune LLMs and vision transformers using siguificantly less computation resources than the popular LoRA (Low-Rank Adaptation) techmique, while achieving similar accuracy. We also explore the effects of using different importance metrics. There are several future directions: (1) For importance metrics, while Quantile-Mean Taylor shows slight improvements, these gains are relatively minor compared to the standard Taylor metric in some cases of DeiT and VIT. We may wish to explore better metrics for classification tasks with large number of labels. (2) Developing memory-effcient importance metrics for LLMs is another future direction. While Zeroth-Order Taylor is effective for incorporating data-specific information without requiring large memory, the large variance of estimate is challenge. Although we reduce the variance effectively by increasing the number of estimations, exploring a simple method to reduce variance without increasing estimation times is essential for further advancements in this field. (3) Our results show that fine-tuning a small number of neurons can significantly improve model performance on downstream tasks, This observation naturally raises the question: do the selected neurons play a distinctive role in spocific tasks? This question is related to the explainability of ncural networks, which is an extensive area of research. It will be interesting to understand if (and how) individual neurons chosen for fine-tuning contribute to the new task.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint
"We omit the results of QPISSA in Table 10 because its training loss not reasonably low in our experiments. While the original paper [Meng et al., 2024 suggests disabling dropout to accelerate convergence, we observed that doing so leads to overfitting in all tasks. Instead, we set the dropout rate to 0.1 (consistent with LoRA's setting), which yielded better performance on commonsense reasoning tasks and GSM8k. However, for AlpacaGPT4, the training loss of QPISSA remained above whereas all other methods achieved losses below 0.1. Oue possible explanation for the underperformance of QPiSSA is its slight modification of the frozen parameter matrix WNF,, Although the matrix W QPISSA before training is similar those in other methods, QPISSA changes the frozen matrix WNF4 slightly, which may require more training data to achieve good performance.
14

---

Published in Transactions on Machine Learning Research (11/2025)
arXiv:2308.08774, 2023.
Alan Ansell, Edoardo Ponti, Anna Korhonen, and Ivan Vulić. Composable sparse fine-tuning for cross-lingual transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1778 1796, 2022.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth PASCAL recognizing textual entailment challenge. TAC, 7(8):1, 2009.
Trénée-Jules Bienaymé. Considéralions à l'appui de la décowverle de Laplace sur la loi de probabililé dans la méthode des moindres carrés. Imprimerie de Mallet-Bachelier, 1853.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Fourth AAAI Conference on Artificial Intelligence, 2020.
Daniel Cer, Mona Diab, Encko Agirre, Inigo Lopez-Gazpln Gazpio, and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel Cer, and David Jurgens (eds.), Proceedings of the
11th International Workshop on Semantic Evaluation (SemEoal-2017)) pp. 1-14, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: (0.18653/v1/S17-2200 URL https: //aclanthology... .wwgey..orr/eee... worg///-22000
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no nno questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2924 2936, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint NKi3:800305.. 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems, Xiv preprint arXiv:2110. 1410., 2021.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainly, visual object classificnlion, and recognising tectual entailment, pp. 177-190. Springer, 2006.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pottern recognition, pp. 248 255. leee, 2009.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318 0000883 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized 1lms. Advances Neural Information Processing Systems, 36, 2024.
William B Dolan and Chris Brockett Automatically constructing corpus of sentential paraphrases. In Proceedings of the International Workshop Paraphrasing, 2005.
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint XrXi::2010.1 11929, 2020.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ia herd of models. orXiv preprint 277 2024.
15

---

Published in Transactions on Machine Learning Research (11/2025)
Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. In Proceedings of the IEEE/CVE conference on computer vision and pollern recognilion, pp. 16091 16101, 2023.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 10323 10337. PMLR, 2023.
Alan Frieze, Ravi Kanman, and Santosh Vempala. Fast monte-carlo algorithms for finding low-rank approxi- mations. J. ACM, 51(6):1025 1041, November 2004. ISSN 0004-5411. doi: 0..145/1139488.1039993 URL https://doi. org. 10. 1145/103944888 1039494
Trevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. In SC20: International Conference for High Performance Computing, Networking, Storage and Anolysis, pp. 1-14.
TEEE, 2020.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Mucnnighoff, et al. A framework for few-shot language model cvaluation. Version vO. 0.1. Sept, 10:8 2021.
Amir Gholami, Schoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for elficient neural network inference. In Cow-Power Computer Vision, pp. 121- Chapman and Hall/CRC, 2022.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Proceedings of the (CL-PASCAL workshop on textual entailment and paraphrasing, pp. 9. Association for Computational Linguistics, 2007.
Demi Guo, Alexander Rush, and Yoon Kim. Parameter-efflcient transfer learning with diff pruning, In Annual Meeling of the Association for Compulational Linguistics, 2021.
Han Guo, Philip Greengard, Eric Xing, and Yoon Kim. LQ-IoRA: Low-rank plus quantized matrix decom- position for efficient language model finetuning. In The Twelfth International Conference Learning Representations, 2024a. URL https://
Song Guo, Fan Wu, Lei Zhang, Xiawu Zheng, Shengehuan Zhang, Fei Chao, Yiyu Shi, and Rongrong Ji. Ebft: Effective and block-wise fine-tuning for sparse llms. arXiv preprint 4rXiv:2402.1441, 12419 2024b.
Wentao Guo, Jikai Long, Yimeng Zeng, Zirui Liu, Xinyu Yang, Yide Ran, Jacob R Gardner, Osbert Bastani, Christopher De Sa, Xiaodong Yu, al. 'Zeroth-order fine-tuning of lims with extreme sparsity, arXiv preprint XrX:::2466.00 02911, 2024c.
R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second PASCAL recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, volume 7, pp. 785-794, 2006.
Song Han. Efficient methods and hardware for deep learning. PhD thesis, Stanford University, 2017.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-775 2016.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with Pradient-disentangle embedding sharing. In The Eleventh International Conference on Learning Representations, 2023.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241):1-122 2021.
16

---

Published in Transactions on Machine Learning Research (11/2025)
Connor Holmes, Minjia Zhang, Yuxiong He, and Bo Wu. Nxmtransformer: semi-structured sparsification for natural language understanding via admm. Advonces in neural informalion processing systems, 34: 1818 1830, 2021.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Sashank J Reddi, Ahmed Hefny, Suvrit Sra, Barnabas Poczos, and Alexander J Smola. On variance reduction in stochastic gradient descent and its asynchronous variants. Advances in neural information processing systems, 28, 2015.
Peng Jiang, Lihan Hu, and Shihui Song. Exposing and exploiting fine-grained block structures for fast and accurate sparse training. Advances in Neural Information Processing Systems, 1-38845-383351 2022.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR), 2015.
Dawid Jan Kopiezko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adaptation. In The Twelfth International Conference Learning Representations, 2024. URL https://openrrviee
net/forum? N.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in newral information processing systems, 2, 1989.
Fei-Fei Li, Marco Andreeto, Aurelio Ranzato, and Pietro Perona. Caltech 101, Apr 2022.
Yixiao Li, Yifan Yu, Chen Liang, Nikos Karampatziakis, Pengcheng He, Weizhu Chen, and Tuo Zhao. Loftq: LoRA-fine-tuiing-an aware quantization for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https:// net:/forun?dd--aPPPPPPPY
Vladislav Lialin, Vijeta Deshpande, and Amna Rumshisky. Scaling down to scale up: A guide to parameter- efficient fine-tuning. arXiv preprint 2rXiv:2003.1500 2023.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87 100, 2024.
Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression. In International Conference on Machine Learning, pp. 7021 7032. PMLR, 2021.
Shih-Yane Yang Liu, Chien-Yi Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang Ting Cheng, and Min-Hung Chen. DoRA: Weight-decomposeed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024. URL httpss///oparrvviieeeeee/ffrrrrt
Tlya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
Xudong Lu, Aojun Zhou, Yuhui Xu, Renrui Zhang, Peng Gao, and Hongsheng Li. SPP: Eparsity-preserved parameter-efficient fine-tuning for large language models. In Forty-first International Conference on Machine Learning, 2024. URL httcps/s/pppaaaeeeeeeeeeeeooooor .ett:/forum1ddd-1too
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LLM-pruner: On the structural pruning of large language models. In Seirth-seventh Conference on Neural Information Processing Systems, 2023. URL https
17

---

Published in Transactions on Machine Learning Research (11/2025)
Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. Advances in Newral Information Processing Systems, 36:53038 53075, 2023.
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. 'Zhang (eds.), Advances in Newral Information Processing Systems, volume 37, pp. 121038-121072. Curran Associates, Inc., 2024. URL https://ppproo wwurips....u......co cttpppscccppppppppeeeeeeeeeeeeee _ files/
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Melhods in Naturul Language Processing, pp. 2381 12111,991 2018.
Mohammad Hasanzadeh Mofrad, Rami Melhem, Yousuf Ahmad, and Mohammad Hammoud. Multithreaded layer-wise training of sparse deep neural networks using compressed sparse column. In 2019 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1-6. IEEE, 2019.
Mahdi Nikdan, Tommaso Pegolotti, Eugenia Tofinova, Eldar Kurtic, and Dan Alistarh. SparseProp: Efficient sparse backpropagation for faster training neural networks at the edge. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scar- lett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 26215 26227. PMLR, 23 29 Jul 2023. URL https; //proceedings.nt mlr. wttp:/inngs..l..ppsss///0/00000 watass/v222niidaaaa lttmll
Mahdi Nikdan, Soroush Tabesh, Elvir Crnčevié, and Dan Alistarh. RoSA: Accurate parameter-effiient fine-tuning via robust adaptation. In Forty-first International Conference Machine Learning, 2024. URL
Ashwinee Panda, Berivan Tsik, Xiangyu Qi, Sanmi Koyejo, Tsachy Weissman, and Prateek Mittal. Lottery ticket adaptation: Mitigating destructive interference in llms. arXiv preprint orXiv:2406.1679, 2024.
Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, and Sanjeev Arora. Task-specific skill localization in fine-tuned language models. In International Conference on Machine Learning, pp. 27011-27033. PMLR, 2023.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, perf-maeformance deep learning library. Advances in neural information processing systems, 32, 2019.
Alexandra Peste, Eugenia Iofinova, Adrian Vladu, and Dan Alistarh. Ac/de: Alternating com-
decoppeseee training of deep neural networks. Advances in neural information processing systems, -5:8557-8570, 2021.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP, pp. 2383 2392. Association for Computational Linguistics, 2016.
Jorma Rissanen. Fisher information and stochastic complexity. IEEE transactions information theory, 42(1):40 47, 1996.
David E. Rumelhart, Geoffrey F. Hinton, and Ronald Williams. Learning representations by back-propagating errors. nature, (2368)8)):533 536, 1986.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Commanications of the ACM, 64(9):99-11) 106, 2021.
18

---

Published in Transactions on Machine Learning Research (11/2025)
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pp. 4463-4473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v 11//3//11/11/ 11/119-1444 URL http::///ccaambhooogy.ooggg//g
Jonathan Schwarz, Siddhant Jayakumar, Razvan Pascamu, Peter Latham, and Yee Teh. Powerpropagation: A sparsity inducing weight reparameterisaton. Advances in neural information processing systems, 34: 28889 28903, 2021.
Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, et al. S-lora: Serving thousands of concurrent lora adapters. arXiv preprint arXie:3311.0288 2023.
Guangyuan Shi, Zexin Lu, Xiaoyu Dong, Wenlong Zhang, Xuanyn Zhang, Yujie Feng, and Xiao-Ming Wu. Understanding layer significance in Ilm alignment. arXiv preprint ar ArXi::2410. 7875, 2024.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic composilionality over sentiment treebank. In Proceedings of EMNIP, pp. 1631-1642, 2013.
James C Spall. Multivariate stochastic approximation using simultaneous perturbation gradient approxima- tion. IEFE transactions on aulomatic control, 37(3):332-38 1992.
Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2024.
Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems, 34:24193- -4205,, 2021.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-followig llama model. https://github com/ latsu-lab/stanfordd alpaca 2023.
Amirhossein Tavanaci. Embedded encoder-decoder in convolutional networks towards explainable ai. arXiv preprint arX:v22077006112 2020.
Pafmutii Tchébychef. Des valeurs moyennes, volume 12. Journal de Mathématiques Pures et Appliquées 2, 1867.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptisse Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint ar arx::23312.1 11805, 2023.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Alex Warstadt, Amanpreet Singh, and Samnel R. Bowman. Neural network acceptability judgments. arXiv preprint 1805.12471, 2018
Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of NAACL-HLT, 2018
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Transformers: Stato-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp. 38 45, 2020.
19

---

Published in Transactions on Machine Learning Research (11/2025)
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEF conference on compuler vision and pollern recognilion, pp. 1492 1500, 2017.
Mengwei Xu, Wangsong Yin, Dongqi Cai, Rongjic Yi, Daliang Xu, Qipeng Wang, Bingyang Wu, Yihao Zhao, Chen Yang, Shihe Wang, et al. A survey of resource-effcciet Ilm and multimodal foundation models. arv preprinl ar orXo:2401.08000 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2019.
Zhekai Zhang, Hanrui Wang, Song Han, and William J Dally. Sparch: Efficient architecture for sparse matrix multiplication. In 2020 IEEE International Symposium High Performance Compuler Archileclure TEEF, 2020.
(HPCA), pp.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595 46623, 2023.
Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, et al. Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. In The Twelfth International Conference on Learning Representations, 2024.
A Importance Metrics
Taylor importance is the Taylor expansion of the difference between the loss of the model with and without the target neuron: (we denote g(t) by w here for convenience.)
7 L(D, F.) F)
w T (DD + Zw FW..
w V L(D OV
(xx)
(Gw) (Gw)
+ (ww)
L(.
2
where G= LDD,DFF). (**) is from the result of Fisher information Rissanen 19966)
LDD(D.)) = (D, (V,(D.F)
Note that (*) from V L F) removing one nheunon/neuron from a large neural network typically results in only negligible reduction in loss. To efficiently compute ni the equation be further derived
FL(x,F)
7) RJ (Gw) T (Gw)=
w,)²
(14)
D
Ow,
XED
aL(x, F)
w;| 2
(15)
Ow;
X
where w (WI....,,....) Thus, people often compute E, [D] evaluate the importance score. In this study, describe this Taylor importance and consider eq 14 hessian importance.
EXED dw; wy without taking square to
0L(x.))
Magnitude importance is the fo-morm of the neuron vector computed as / VE, w
20

---

Published in Transactions on Machine Learning Research (11/2025)
B
Missing Proofs
Proof of Property 4.2
Proof. To prove Property 4.2 we first calculate the expectation and variance of SPSA. For convenience, we
denote [VL(0, D)] as g. Then, the expectation is as follows:
Elg E zz (L D) Elzz [
The variance can then be derived as follows:
d E 2
Varig:] R Varlzi(z' T g)] = [z(z g))² Elz.(z' T g)]² =5|2, >
2ig. )21
z ]
d
= E[
7:gi ) Elzig: + (zi Σ
z|g:)]
-=1
[H,LE(d
2
E
z:gi
Elz gi
Σ
E
E
gi
E|z,
z,g
82
E[2] (2:gi
Σ
7igi 12]
gi
(2,g 2 + 22.gi
Zigi +
Σ
zigi) g²
lfile(d)
El g
2z/gi
Σ
Zigi
2 Σ Zigi)² g
LHLEE
= E| gi
+ 26|
22 gi B
zigi + E|2,
Σ
zigi ) -
IE
g E
2[g
g
E[z
E[z
Σ
gी E
Σ
gkgi
E
g
3g +
Σ
gi g
g
g
El = 0, 2 Var(z,] 1, and E[z2z1]u4 = are because z La). B[z,] obtained from the
moment generating function of standard normal:
d'
E
de
(e )iz
(3e + 6 tee 1-= 3.
Thus, the expectation and variance of n-SPPA estimate are (g. is the jth estimate of g: here):
Σ Elg
E g.
E
-=
gi:
n
n
Σソ=1
8i
(
Σ n Σド=
Varlg!
Var [g.
g +
Var
n²
n
Then, we are going to prove the consistency n-SPSA estimate. Given any small 6 0, we can derive the following inequality by Chebyshev's Inequality [Bennayé
1853
Tchébychei
1867
g0)
g
6
Var
j=
g
-i g7
Pr[
g
c
n
2²
2n2
which converges to 0 0c.
21

---

Published in Transactions on Machine Learning Research (11/2025)
Proof of Theorem 4.3
Proof.
nk.
7/2
pf - -0, = P2.1((0.1)
2>
(16)
2 0009
E Ese(dee) 090²
2
Σ 0
d1
Σ
d.
8
21=1
1-
2
03
j=
8j + d;
Ei=1
gi
11g
2
(+)
d.u0
Σ 11= Bi
(17)
nk
nk
nk:
i=1
j=1
E [EEf 0303
E Eserdooo )0)
125
(di, + dig jug 21=1
Bi
(di, + dig )
(18)
2
2
4nk
4nk
7> 2/nk((1) (1: - nia)
(0n)
2V - 910/100
= pf, - -n > 2
2 1
(19)
((d, + d1 J
/ (di, d )ug ug
(*): de is the dimensionality of neuron i and d is the number of parameters, we have d > d2 and, further, de Ei= gi
(**): n, is a-smooth in [0, Un Pn. (nz S m7 S m..)
92 Q
I a(n, -k,), implying that Th 77/2
100
(o- 0/1/10
Proof of Theorem 4.4
Proof. We will argue that for any i E (ç - )
we have p(X; = 1) 2 This implics that
pi 2 ig (1
6g 2
2 ic (E),
(20)
2
1=1
as desired. Let us thus focus on p(X, 1) for some i s ( it suffices to prove that
Using the lower bound on from equation 12
p( Miee 2
Eg
2dn
Thus using Theorem
4.3 it suffices to prove that
2ea, a (di + die )a (ee (cc/2)/100
(-
Cdyy CE
<
CE 2dn
(21)
av(d (d: + dig Jugu
100
Given the fact that 1
sd (- 100
s - we have the following statement: the greater Cdn is. This concludes that, there must exist constant Ca, such that Wedn 2 Cdn.
= (2m~N(0,1)
100
the ca,, is, the smaller the Φ1
T (- 100
satisfies Equation
C Parameter Dependency
Dependencies of parameters between neurons or channels across different layers exist in NNs. These include basic layer connections, residual connections, tensor concatenations, summations, and more, shown in Figure 8 The black neurons connected by real lines represent the dependent parameters that are in the
22

---

Published in Transactions on Machine Learning Research (11/2025)
group. Pruning any black neurons results in removing the parameters connected by the real lines.
Liu
et al. 2021 introduced group pruning method for CNN models that treats residual connections grouped dependencies, evaluating and pruning related channels within the same group simultaneously. Similarly, Fang ct al. (023)) proposed a novel group pruning technique named Torch-Pruning, which considers various types of dependencies and achieves state-of-the-arl results. Ma et al. (2023) further applied this procedure pruning LLMs. Torch-Pruning can be applied prune a wide range of neural networks, including image transformers, LLMs, CNNs, and more, making it popular toolkit for neural network pruning.
concat
(a) Basic connection
(b) Residual connectton
(c) Concatenation
(d) Summation
Figure Common dependencies of parameters in neural networks.
In this study, we also evaluate the influences of incorporating parameter dependency in our approach. We put the experimental results of whether incorporating parameter dependency in Appendix D.2 In the experiments, parameter dependency becomes the following process for our approach: first, searching for dependencies by tracing the computation graph of gradient; next, evaluating the importance of parameter groups; and finally, fine-tuning the parameters within those important groups collectively. For instance, if w and w!!)
are dependent, where whfe) is the j-th column in parameter matrix (or the j-th imput channels (fnellsffaturrs) of of layer then
layer and wr() is the i-th in parameter matrix (or the i-th output 3. will be fine-tuned simultancously while the corresponding M for W/(a) becomes column dep
w and wi()) selection matrix and g(a) becomes A A M(a). dep dep' Consequently, fine-tuning 2.5% output channels for layer will result in finc-tuning additional 2.5% input channels in each dependent layer. Therefore, for the 5% of
desired fine-tuning ratio, the fine-tuning ratio with considering dependencies is set to 2.5% for the approach that includes dependencies.
The forward function of layer for columm selection mentioned above can be written as the following equation:
f(V((a) x) f(w()), x) + fMM A ( x) + f(Awla (a)
Maa dep dep x).
Note that this example, as the dependency connection between the output feature faatrrechannel of and the input feature/channel of the dimension d(a) is equal to d( where W(a) E Rada xal W(h) R x
Ablation Studies and Related Analysis
In this section, we first discuss the hyperparameter settings. While we do not include DeBERTaV3 He ct al., 2023) in the main context, we fino-tune DeBERTaV3-base (He et al., 2023) on GLUE. The learming rate is set to 2 10-5 with linear decay, where the decay rate is 0.01. The model is fine-tuned on the full training split of 8 tasks from the GLUE benchmark. The maximum sequence length is fixed to 256 with longer sequences truncated and shorter sequences padded. Note that memory efficiency is not emphasized for memory-partiene
small-scale models, dataset-related in these cases. The main ad advantage of our method in these cases is the reduced FL.OPs due to fewer trainable parameters.
with large batch sizes dominatess consumption
10 some complex models, considering dependencies results slightly more than twice the number of trainable parameters. However, cases, the factor
23

---

Published in Transactions on Machine Learning Research (11/2025)
Following this, we discuss the computational resource requirements for fino-tuning. Figure illustrates the computation and cache requirements during backpropagation. Next, we provide an ablation study on the impact of different rank settings for our approach and LoRA, as shown in Table [6] Finally, Table I9 demonstrates the advantages of froczing self-attention blocks to reduce memory usage while maintaining performance.
Table 11: Fine-tuning on CIFAR100 and Eiy--mmaeNNet. #ep and #param represent the number of epochs and the number of trainable parameters, where SPruFT our method with Taylor importance. Full and Head indicate full iin-tuning and and accuracies are based validation results. Bold denotes the best results of each fine-tuning approach (in the same column) the same model and dataset.
which only fin-tunes the classification layer, All reported losses
CIFAR100
Tiny Mlag-la
Callech101
Head
SPruFT
Full
Head
SPr
Full
Houd
SPzuPT
#-p
loas, acc
loes, acc
loso, nce
losa, a.o0
loss, aeo
loss, acc
loes, ace
J0s0, acc
loso, ace
Del'T
DefT
DelT
parann
86.0M
0. BM
4.6M
86. 1M
0.3M
4.8M
80 PM
0.2M
4.6M
5
0.36, 90.18
0. 80.25
0.37,
88.70
0.54, 87.55
85,
0.40,
89.00
0.11,
1.08, 80.
0.30 95:41
0. 90.000 00
0 81.83
0.42,
86
85,
0.40
85 96
0.11, 97.55
0.53
96.
0. 89.00
0.55, 83.42
0.64
88.61
0.84, 87.27
0.52, 86.06
0.73, 88.67
0.11, 97.11
0.22, 95.06
0.12, 96.50
VeT
pppara::
85.9M
0.1M
4.5M
86.0M
0.2M
1.8M
85.9M
0.1M
45.2M
D.3e, 90.13
1-01, 74.78
0.40, 88.33
0.51, 88.48
0.85, 84.10
0.38, 90.87
0.12, 87.
1.80, 83. 85-70
0.43, 93
0-4b, 89.85
0.88, 77.05
0.45,
8.86, 88
0:b6, 84.95
0-44, 90.48
0.13. 97-20
0.5h, 89.95
0.23. 9b.04
0. 88.78
0.71, 70.51
0.00, 87.83
0.86, 84.20
0.55, 65.49
0.01, 90.56
0:12, 97.24
0.38; 02.05
0.10,
R0sNet111
ReuNes101
ResNet101
parau:
42.7M
0.23
2.2M
42.9M
0.436
2:4M
42.7M
0.23
2.2M
5
0.50. 86.21
1.62. 60.78
0.59. 82 36
0.92, 77.78
1.64, 62.06
0.70. 70.00
0.14, 90.50
1.25, 82.33
0.45, 92.50
10
0.58, #6.41
1.39,
63.05
0-60.
89.3 33
1-10, 76 91
1.50, 63-18 18
0.79,
79.54
0.14, 96.54
0-68. 90. 80-24
0:23; 85
30
0.80, 84
1,21, 65.43
0.80, #2.49
1.54, 74.09
1.43, 64.47
1.08, 78.58
0.18, 95.80
0.33, 93.00
0.36, 95.89
1011
Rew1e
DexNeX:1101
#pparam:
87.0M
0.2M
4.9M
W7:2M
0.4M
5.1M
87 PM
0.2M
4.9M
0.71, 86.50
1.08. 69.45
0.68, 86.33
76.
1.29, 67.93
0.90, 82.53
0.16, 96.63
0.33, 92.87
0.14, 96.76
D.1 Hyperparameter Settings
We report the results of three approaches over several epochs as table 11 and table 12 Overall, full fine-tuning over higher epochs is more prone to overfitting, while head fine-tuning shows the exact opposite trend. Except for the results on caltech101[1] the loss patterns across all models consistently refiect this trend, and most accuracy results further support. this conclusion. However, our approach demonstrates a crucial advantage by effectively balancing the tradeoff between performance and computational resources.
Table 11 clcarly shows that both our approach and full fine-tuning achieve optimal results within a few epochs, while head fine-tuning requires more training. Notably, all models have been pre-trained on ImageNet- which may explain the strong performance observed with head fine-tuning on Tiny-ImageNet. However, even with this advantage, full fine-tuning still outperforms head fine-tuning, and our approach surpasses both. In just 5 epochs, our approach achieves results comparable to full fine-tuning on all datasets with significantly lower trainable parameters.
In contrast to Table [1 the results in Table 12 show more variation. Although the validation loss follows a similar trend, we report only the evaluation metrics due to the different patterns observed in these metrics. One potential reason for this variation is the varying amounts of training data across the GLUE tasks. As shown in the table, tasks with fewer samples often rcquire more epochs to achieve better performance for both full fine-tuning and our approach. Conversely, for lasks with large amounts training data such as "MNLF, QNLI', 'QQP', and 'SST-2', the results show tiny improvement from 3 10 epochs. Nevertheless, the results still demonstrate that our approach significantly balances the tradcoff between performance
The inconsistent trend observed in Caltech101 results likely due its significantly smaller sample size.
24

---

Published in Transactions on Machine Learning Research (11/2025)
Table 12: Fine-tuning DeBERTaV3 on GLUE. 'mec', 'acc', and 'corr' represent 'Matthews correlation', "accuracy,, and Pearsso correlation', respectively. #param is the number of trainable parameters. SPruFT is our method with Taylor importance, while Full and Head indicate full fine-tuning and head-finetuning, which only fine-tunes the classification layer. All reported metrics are based on validation results, and are percentages. Bold denotes the best results of each fine-tuning approach on the same task.
CoLA
MNLI
MRPC
QNLI
QQP
SST-2
STS-B
# Meain
8.0k
803k.
3.7k
1084
3641.
2.5k
67L.
7h
method
pporom
epocla
Full
184.42 42M
69:90
60.42
69.71
03.57
02.08
60.14
95.68
00.4 20.44
87:74
03.38
02
83.39
94.72
88.98
98.85
90.20
93.15
03.81
90.71
Head
132.13K
24.04
64
08.38
70.73
52.71
5.66
Head
46.89
01.75
68.38
72.32
80.02 00
47.20
78.44
20.88 80.88
Head
47.52
09.08
08.38
71.09
80.90
47.20
74.60
49.69
.88
SPruPT
03.17
and computational resources. Our method achieves near full fine-tuning performance with remarkably less trainable parameters.
D.2 Considering Dependency
We evaluate our approach with and without considering parameter dependency, as shown in Table 13 and
Table 14
Table 13: Fine-tuning image models by our SPrulT for 5 epochs. "dep" refers to whether parameter dependencies are involved or not. e, Taylor, and QMTayllor represent the magnitude, Taylor importance, and Quantiles-We Meat Taylor importance (Equation Note that QMTaylor not applied to fine-tuning Caltech101 due to its significantly imbalanced labels. All reported results are validation accuracies. Bold indicates the superior results achieved through dependency searching compared to not searching. Underline highlights the best thnettunig results.
data
CIFAT1100
Tiny-ImageNes
Caltech1111
22
Taylor
22
Taylor
Taylor
Deff
X
88.05
88.70
89.47
89.31
88.00
89.75
95.01
95.41
VIT
X
87.18
86.00
88.61
90.76
90.87
00.00
02.00
08.00
85.24
80.83
89.67
77.82
RN
X
79.88
78.63
78.62
89.91
72.51
52.71
RNX
84.71
85.01
78.39
79.84
82:13
01.82
We utilize various importance metrics fine-tune both models using our approach, with and without incorporating parameter dependencies, and report the results compare their performances. Searching for dependencies in structured pruning is natural, as dependent parameters pruned together. However, important neurons in a given layer do not always have dependent neurons that are also important in their respective layers. As demonstrated in Table 13 fine-tuning without considering parameler dependencies outperforms fine-tuning incorporating dependencies in all cases. For importance metrics, although the differences between them are not substantial, all results consistently conclude that the Quantile-Mean Taylor importance demonstrates slight improvement over the standard Taylor importance. Furthermore, both the Quantile-Mean Taylor and standard Taylor metrics outperform the magnitude importance.
25

---

Published in Transactions on Machine Learning Research (11/2025)
Table 14: Fine-tuning DeBERTaV3 on GLUE by our SPSFT for 10 epochs. "dep" refers to whether parameter dependencies are involved not. Taylor and & indicate the magnitude and Taylor importance. The importance score is Taylor. We do not apply QMTaylor since the nunber of labels is tiny. 'mec', 'acc', and 'corr' represent 'Matthews correlation', 'accuracy', and 'Pearson correlation', respectively. All reported metrics are based on validation results. Bold indicates the best results whether considering dependencies.
Laok
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST-2
ST3-B
imag
dep
ueee
ucc
0cc
ucc
800
000
coct
x
07.48
89.85
57.95
83.30
01:03
78.42
85-07
89.98
?
x
94.04
00.80
00.22
84.07
08.04
01.57
70.00
95.07
$7.39
Table 14 suggests slightly different conclusion: the impact of parameter dependencies on performance is minor, nearly negligible However, searching for dependencies involves additional implementations and computational overhead. Combining the results of image models, the conclusion not searching for the parameter dependencies. For importance metrics, this experiment shows that magnitude and Taylor importance perform similarly.
D.3 Memory Measurement
this study, we detail the memory measurement methodology employed. The total memory requirements can be categorized into three main components:
memTrI. memm + mempr + memAux
where:
1. memrri is the total memory consumed during training.
memm represents the memory consumed by the base model itself.
3. memFT corresponds to the memory required for the fine-tuning parameters and their gradients.
4. memAux accounts for any additional memory usage, including optimizer states, caching, and other intermediate computations.
We yield memy by measuring the memory usage during inference on the training data using the pre-trained model. The combined memory usage of memFT and meMAux calculated as the difference between memTrI and memmodet- For simplicity, we consistently report memrr mema as "mem" in all comparisons presented in this study.
12The results of using magnitude importance the RTE task show significant variation, but this is likely duc to the small sample size and the hardness of the task, which result in the unstable performances observed experiments. Aside from RTF, the results on other tasks significantly different.
26

---

Published in Transactions on Machine Learning Research (11/2025)
Table 15:
The requirements of computation resources for fine-tuning, 'mem' traces menTTL memm- indicates that we freeze the layers for queue, key, and value projection. All fine-tuning parameters are stored in full precision. We also examined the training time and observed that DoRA requires 50% to 100% more time than other methods, while LoRA, RoSA, and our approach need similar training time (differing only by few seconds). However, due to the influence of various factors on training time and the difficulty of ensuring fair comparison, we chose not to include these results in report.
(lama3(81t)
FT
menTTL
wemTTTL
LoftA, ==
159 (2.7777%)
53.33GB
29.87GI3
23. 23.15GB
167 8M,2 08%)
65.23611
33 86G1
30-37GB
RoSA, r= 32,
157 7M (2.34%)
T4.36GB
29.87GB
44.88GB
167 6M 00%
82-26G:B
33 86GB
10GB
DORA,
80%)
74.72GB
44.5GGB
1M/2
85.31GB
51.45GB
VoRA,
1.37M10
52.84GB
22.97GB
1.30M00.022%
23.33BB
SPreFT,
145 (2.111)%
47.49GB
17.02GB
350.4M(1.9955
LoNA!
663%)
48
18:816B
130.0M((1622)
ROSA 32, d- 1.2%
113. 2M(1 68%)
69:70GB
29,873B
39.83GB
130 180.1M 1M(1.74%)
77. 00.0
33. 96GB
43. 73.B
DORA", 64
110. (3(66))))
64.05GB
29.87GB
34.18GB
181 (M((1.0%%)
71. GB
S3. 36GB
44.11CB
VeRA.
0.84b4(0.01%)
47.63GB
29.87GB
17.76GB
1.06X(0.0199)
39.94 30.84CB
38. 36CB
26.T8CB
SPraFTI,
:09.1M(1.366%
D.4 Resource Requirements
Table 15 presents the resource requirements of various PEFT methods. We compare approach with LoRA and several of its variants that maintain or surpass LoRA's performance. As shown, our method is the most resource- efficient among these approaches. The subsequent ab- lation study further demonstrates that our approach achieves performance comparable to LoRA. We exclude comparisons with VeRA (Kopiczko et al., 2024), which proposes sharing a single pair of random low-rank ma- trices across all layers to save memory footprint. While VeRA achieves some memory savings, its performance often deteriorates.
Next layer
Bout
|[V DV ll."
Vour al/Obout
0L/am
mh
h= W'x
9p- Vour
TV
Vg W'
W'x
V
W"
[V &V lle
Pyr= V.
We note that while our approach offers significant memory efficiency, this benefit less pronounced in small-scale models, where the primary memory consumption arises from the dataset-este especially with large batch sizes. The main advantage of our method in these cases is the reduced FLOPs due to fewer trainable parameters. Therefore, we do not highlight memory efficiency in small-scale model scenarios.
V RRut Maa
4V = BA
0L/aB
BA
Vav
IV &V lle
aL.aa
drapout
previous layer
In Section 3. we explain that the memory usage of DoRA is significantly higher than that of LoRA due to its complex computation. We demonstrate the computation graph of DoRA here, as shown in Figure 4 DoRA decomposes W into magnitude m and direction V and computes V+AV This the final parameters matrix by W' = m m complicated computation siguificantly increases memory usage because it requires caching a lot of intermediate values for computing gradients of B, A, and m. As illustrated in Figure each node passed by backpropagation stores some intermediate values for efficient. gradient computing.
DoRA Initial = W. l|W]|c
Figure 4: The illustration of DoRA's computation graph. Black operations occur during the forward pass, while orange operations take place during the backward pass.
27

---

Published in Transactions on Machine Learning Research (11/2025)
Next layer
laxer
Cache bout
heat hright
Cache haur
bient
Bright
higf + highh
hiet + hyight
hright BAX
Brihht MWjx
Cache Friht 8L/ Bright
cache Fright
hioh
Wx
hjat Wx
0L/0B = Vrighth
Bh
M
Cache h Ax
No cache h Wyx
Wx
Ax
Wfx
Cache dropout mask
No cache dropout mask
dropout
Cache X
Cache
x
LORA
Our approach
Figure The illustration of backpropagation highlights the operations involved. Black operations occur during the forward pass, while orange operations take place during the backward pass. Blue operations highlight the benefits of our approach. Notably, since M is non-trainable, caching A WW during the forward pass is unnecessary, leading to significant memory savings. Additionally, in practice, PyTorch caches Tha to efficiently compute 0L 7B although this caching is not strictly required for backpropagation.
aL
28

---

Published in Transactions on Machine Learning Research (11/2025)
D.5 Cache Benefit
In the main context, we have already shown the memory cost of dropout layers in LoRA, in this section, we will discuss some other benefits of our approach. Figure illustrates the computation and cache requirements in backpropagation (Rumelhart et al. 1986). For simplicity, we replace the notation f(:--) with different. h. With the same number of trainable parameters, our approach eliminates the need to cache h - A Wx shown in the figure. While this benefit is negligible under lower rank settings (r) or when the number of fine-tuning layers is small, it becomes significant as the model size and rank settings increase. Although the caching requirement for h can be addressed by recomputing h = Ax during backpropagation, this would result in increased time complexity during training.
Table 16: Similar to Table 2 this is the full table with different rank settings for our SPruFT and LoRA. We also present the results of freezing queue, key-, and value protee-proeetton in this table (f)
Model, ft estting
mem
#param
BealQ
PIQA
SIQA
118
WG
ARC-
ARC--
OBQA
Avg.
(7)) LoRA, (7B). SPouFT,
BLB
20 20.0M(0.200%
80.4
73.8
70.2
79.8
63.4
68.6
80.8
71.6
73.55
15.26GB
18. 18.2M(0.275))
73.4
12.0
64.8
75.2
49.5
62.8
78.6
63.2
67.83
(7ama LeRA, 16
21. B1.6
40 aM. 59%%
76.4
71.4
69.4
81.2
59.8
67.0
81.4
78.2
72.48
(lama2(7B), SPruFT,
15.57GB
4M(0 54%)
73.0
67.4
78.4
08.8
09:4
Ulama2(7B) LoRA, F 32
32.21GB
80 (M(1919%)
74.8
74.2
71.4
75.4
55.6
67.0
82.2
69.6
72.03
(11)22(711) SProPT, ==
16.20GB
T2 9M(1 .85%)
78.0
73.8
66.4
83.4
39.6
69.4
79.0
71.6
72.65
Maamat(71) Llamo2(7B), SPruFT,
(2.377)%)
17.02GB
146. (M(.21099
76.2
67.6
68.4
71.8
74.69
L7ama8(7B), LcRA, e 128
26.0SGB
310. 95())75%)
77.2
76.2
72.0
84.2
62.0
71.6
81.2
71.8
74.68
(78m SPruFT. 256
20,47GB
291. (M(4)337%)
80.0
73.0
66.6
53.5
68.2 2
69.6
82.4
70.2
73.68
(lama2(7B), LoRA, 256 (aama2(7B), SProFT,
639. (M(9.4915)
78.0
T3.0
71.2
82.4
60.8
07.8
83.0
74.4
73:00
26.17GB
583 (M10006666))
79.4
73.2
08.6
61.4
02.2
09.8
75.2
73.93
L7)) LoRAt, Ldams2(7B), SPruFTi,
17.62GB
27. 37.4MM(0.4150
80.4
75.0
08.2
77.4
68.0
00.4
80.2
71.8
72.26
14.29GB
23.95))3358)
(7ama2(7B). LoRAl, M7)m22(7R) SPouPr1 -=
17.95GB
(M(0.81%)
11.0
14.87GB
47 (M(7771077)
78.2
75.8
66.4
81.4
39.2
67-8
78.8
72.0
72.45
Llawa2(7B), LoRAt, 64
16.8IGB
108. 6M(1)6315)
79.6
78.8
67.0
80.4
62.6
09.2
81.0
70.4
73.00
U14a2(771) SPraPTt y 128
15.58GB
95. 9M.4 .1%)
81.4
75.0
67.2
80.4
65.0
099.2
79.8
73.8
73.98
Mamsa(())) (dams2(8B)) SPeuPT,
79.8
74.4
70-2
19.9M(0.255) (5%)
78.0
(lama((NB)) LeRA, r Llamm3(SB), SPruFT,
28 8GGB
41 (255)
65.2
80.8
68.4
81.6
69.0
79.4
90.0
77.0
78.95
22.02GB
89.8 SM. 60%)
81.2
62.2
68.2
85.4
63.4
79.4
87.6
80.8
78.58
(8am)3(8B), LoRA, (4ama366B), SProFT,
29.37BB
63, 9M(1
61.8
87.6
07.0
76.4
80.4
79.60
B
79. 79.7M(0.6M) 09%)
90.0
09.4
56.0
60.6
81.4
Mamad(8tt) LoRA, r 64
30.37GB
167 9M(2.099) 0995)
84.2
77.0
63.2
54.3
67.2
76.4
88.8
71.0
76.50
Llamas(sB). SPeuFT, 128
24.49GB
159. 159.4M(1..00 (.099))
87.8
77.4
71.4
85.4
70.2
70.8
90.8
81.8
80.58
LSam LoRA, 128
32.35CB
336. 1855)
85.8
61.8
T1.8
$4.2
67.2
78.2
91.8
81.0
80.18
LSam SPruFT, 256
26.09GB
318. SM...M333999
80.4
90.2
71.8
85.4
64.2
77.8
80.4
73.2
78.50
Llama3(8B), LcRA, 250 [Jamso(NB)) SPenFT,
36.82GB
67L-13(88.00) (3055)
80.2
78.6
66.8
85.5
68.6
78.2
70.2
77.78
637.5M(7)94%)
Llamas(BBB. LoRAi, Ma)m SPouPT 7= 32
24.8GBB
(3(0.41)))
18.6
21.37GB
27. (00335))))
63.2
83.6
86.0
65.6
82.-1
89.8
80.4
79.58
Llama3((B); LeRAt, r= 32
25.28GB
65. (M..0.)0))))
83.0
88.8
71.4
85.2
65.2
81.0
93.8
79.6
78.35
(S)) SPraPT >= 64
21.81GB
54 5M (D.668)) 68%%
84.4
76.0
71.0
63.2
69.0
79.0
84.4
83.4
79.05
LlamaS(SB), LeRAt, LSB) SPruFT
26.04CB
(30.0MM(166275)
86.2
61.0
67.8
$1.8
66.0
73.8
86.2
78.2
77.78
22.71GB
109. (3..88)))388))
89.4
79.0
71.2
86.2
71.8
85.0
86.8
76.2
80.45
D.6 Rank Settings
We present an ablation study of rank settings here. Table 16 demonstrates that == 8 is sufficient for LoRA when fine-tuning Llama-2 and Llama-3. In contrast, increasing r for our approach yields slight performance improvements. The most remarkable observation in Table 16 is the exceptional memory efficiency of our approach: even with r = 128, the memory usage of our method is siguificantly lower than that of LoRA with r 8=8. 8.
Table 17 and Table 18 the full tables of importance evaluation for Llama2 and Llama3.
29

---

Published in Transactions on Machine Learning Research (11/2025)
Table 17: Similar to Table this is the full table with different rank settings. We also present the results of freezing queue- key-, and value-projection in this table (!).
Model, acttins
BoolQ
PIQA
BIQA
WG
ARC-c
OBQA
Llama2(TB), SPcuFT
16, random
72.5
72.4
63.2
78.0
16,
73.4
72.0
64.8
78.2
40.0
02.8
78.0
68.2
67.83
16, ZoTaylor
74.4
73.0
60.4
80.8
67.2
70.0
77.2
63.0
70.70
92, random
12.2
65.5
17.0
56.4
T1.0
17.0
89.2
89.80
20Tayler
74.4
64, random
71.0
68.0
62.8
74.2
68.0
96.0
78.0
71.4
69.45
84,
76.0
66.4
M0.6
190
70.0
71.0
72.85
64, ZOTaylor
78.2
71.0
68.4
80.8
02.0
69.0
81.0
71.4
72.80
random
80.0
75.2
67.6
63-4
82.1
74.50
128, 20Tsylor
79.4
74:4
67.0
65.0
72.4
63.0
71:2
74.35
256, random
71.0
69.8
70.4
62.4
70.4
78.4
69.23
612. random
73.6
70.0
68.0
67.2
50.8
88.8
75.2
67.8
88.60
612,
70.4
73.2
68.8
81.4
02.2
60.3
81.4
76.2
79.98
012, Z0Taylor
77.4
73.2
61.0
82.2
06.2
72.3
76.4
70.4
78.45
327, random
78.0
71.2
64.8
12.6
56.8
87.8
T8.4
50.8
89.55
322,
82.8
73.0
61.4
64.88
60.4
70.2
72.10
Taylor
77.4
65.8
68.8
79.8
71.15
64",
78.2
75.8
69.4
81.4
50.2
07.8
78.8
72.0
72.45
64
20 Tayler
81.4
71.4
67.0
82.2
61.8
71.2
80.0
72.0
73.34
128 randem
76.5
75.2
67.2
79.2
64.0
09.8
17.0
08.8
72.08
128! 2oThylor
81.0
74.4
69.0
81.0
64.6
70.8
81.2
72.8
74.30
Llama3(eB),
16, random
88.4
78.0
70.8
81.6
67.8
70.8
89.4
74.4
78.00
16,
$3.0
75.0
66.6
77.0
60.2
74.0
89.4
76.2
76.43
16, ZOTaylor
85.4
82.4
68.8
85.0
02.4
74.6
90.2
78.2
78.38
random
92,
81.3
#2.2
68.2
78.4
87.8
80.9
78.53
22, ZOTaylor
$1.6
70.4
88.2
79.88
64, rondou
82.6
78.4
68.8
72.6
68.4
74.0
$1.6
76.0
78.93
Z0Taylor
81.6
77.0
65.8
78.00
125, random
64.2
77.
70.3
72.0
72.4
72.8
84.0
70.6
76.08
125,
87.66
77.
71.4
#5.4
70:2
79.8
90.8
$1.8
80.55
128, 20Tsyler
89.0
78.8
70.5
86.2
69.4
80.4
92.0
83.8
#1.28
86.4
71.8
84.2
17.8
69.4
ZOTaylor
88.4
71.8
87.2
79.2
80.20
612. random
83.4
73.2
68.8
18.4
67.2
74.8
88.4
76.2
76.08
(
70.6
78.70
20Ta
87.4
70.8
70.98
327,
random
81.6
80.0
69.2
87.8
82.4
89.8
85.0
80.05
322.
321,
2OTaylor
85.2
77.8
72.2
84.0
85.6
78.8
99.0
88.44
80.10
random
81.2
80.2
61.8
82.8
70.6
78.6
88.2
0
78.43
G47,
94.4
78.0
71.0
88.2
69.0
79.0
$4.4
83.4
79.05
84
128
rendom
78.2
77.20
/
89.4
79.0
71.2
71.8
63.0
80.9
128
Z0Taylor
90.6
77.0
71.0
84.0
72.6
90.0
92.0
80.0
81.80
30

---

Published in Transactions on Machine Learning Research (11/2025)
Table 18: Importance evaluation for Llama2 and Llama3 on MT-Bench. We also present the results of freezing queue- key-, and value-projection in this table (t). Bold indicates the best result on the same task.
Model, setting
Homanities
Raiaplay
L7am 3PcuFT
random
ZOTaylor
1281, random
2.31
2.90
1.75
2.77
4.61
4.35
4.15
3.60
2:10
4.80
4.33
3.42
== 125t, 30tsyler [el)
2.44
2.63
5.20
2.21
2.93
5.00
4.94
3.95
3.86
r= 125, random
3.22
3.74
1.72
3:33
3.50
5:20
6:38
5:11
4.40
125.
3.48
5.11
8:11
3.83
4:21
5:35
6-44
5.40
5.08
125, 20Toylor
4.13
4.78
0.89
0.88
4.08
6.95
0.39
5.16
0.84
random
3.56
5.05
5.68
4.00
5.96
6.17
6.00
5.05
12st.
4.75
4.78
6.16
8.67
3.81
0.26
6.06
5.00
4.87
12st, ZOTaylor
Table 19: Same results of Table 16 with reordering of the rows. This table is for comparing fine-tuning all lnner layers with freezing queue- key, and value-projcction.
Model, sotsing
ARC--
OBQA
L7ama2(7BB) LeflA, = 18 (70m Lolt 1
21.61GB
(U.MM(..39)
78.4
71.4
69.4
61.2
59.8
67.0
81.4
73.2
72.48
17.95G
54 (1.85))))11)
76.0
70.2
07.2
82.0
71.0
72.90
M7am22(71) LoftA, 32 Llama2(7B), LonAA,
22.21GB
80 (0.95M()-195)
74.8
74.2
71.4
78.4
58.6
67.0
82.2
69.6
72.00
18.81BB
109 6M(11639%)
79.6
75.8
87.0
80.4
89.2
81.0
70.4
73.00
Lluma2(7B), SPruFT, 6lama2(71), SPruPT1, v 64
15.37GB
70.4
82.2
69.4
14.67GB
47. (M(77715))
78.2
75.5
66.4
81.4
59.2
67.9
79.8
72.0
72.45
Ilama2(7B), Bluma2(7B), SPruPT", 128
72.0M(11009)
16.68CB
96 (G.4M())4299)
81.4
75.0
67.2
80
05.0
60.2
70.8
78.8
73.98
Llamm3(8B), LoRA, Liama3(sB),
28.80GB
A1.9M(0.52255)
90.0
77.0
78.95
28.28GB
05.0M((.881%)
85.2
91.6
79.8
78.95
Llama3(8B), LoRA, Lloma3(8B), LORAT,
29.37GB
$3. $1.9%((1.04%)
86.2
81.8
68.2
87.8
0T.0
89.2
80.4
79.50
20.04GB
0M(1.62%)
86.2
78.2
77.78
Lloma3(8B), SPruFT, --
22.62GB
39. SM(O.5015)
81.2
82.2
68.2
86.4
63.4
79.4
87.0
80.8
78.58
Llama3(sB), SPruFT),
Llama3(SB), SPruFT, r Llama3(8B), SPruPTt 22.71GB
23.23GB
70. 70.73((0.99%)
89.2
80.6
00.4
86.0
60.8
78.4
83.4
81.4
77.88
89.4
86.2
$3.0
86.8
78.2
80.45
31

---

Published in Transactions on Machine Learning Research (11/2025)
D.7 Benefit of Freezing Attention Blocks
We now assess different fine-tuning strategies. Table 19 highlights the importance of selecting fine-tuning layers strategically to minimize redundant memory usage. Freezing the self-attention blocks achieves performance comparable to fine-tuning all layers while significantly reducing memory consumption during training. This efficiency stems from reducing the need cache intermediate outputs for gradient computation. For example, as illustrated in Figure 5 using LoRA, Vout must be cached to compute dL for the subsequent layer. Freezing the next layer eliminates this caching requirement, further optimizing memory usage.
E Details of Datasets
E.1 Vision Benchmarks
CIFAR100: CIFAR100 (Krizhessky et al. 2009 has 100 classes with 600 images of size 32x32 per class, while the CIFAR10 has 10 classes with 6000 images per class. In this study, we use the CIFAR100 downloaded from huggingface (https: hhtg:::://ggggggaac..... co validation images per class. In our experiments, we resize the images to 256x256, crop the center to 224x224, and normalize them using the CIFAR mean (0.507, 0.487, 0.441) and standard deviation (0.267, 0.256, 0.276) for the three channels.
http:///gggggg....e.aaaaaateeeee http///atsets/oootcccaeeee cifcrrio0 with 500 training images and 100
(Tavanaci 2020) has 200 classes with images of size 64x64, while the full ImageNet- (Deng et al. 2009) has all 1000 classes where each image is the standard size 224x224. In this study, we use the Tiny- ImageNtt downloaded from huggingface attps://buggingface.comm co/datasets/ zh-p 2h-plus/tiny--magennt
Tiny-ImageNet:
with 500 training images and 50 validation images per class. our experi- ments, we resize the images to 256x256, crop the center to 224x224, and normalize them using the mean (0.485, 0.456, 0.406) and standard deviation (0.229, 0.224, 0.225) for the three channels.
caltech101: Caltech101 (Li et al. 2022 consists of 101 classes, with images of varying sizes typically having edge lengths between 200 and 300 pixels. Each class contains approximately 40 to 800 images, resulting in total of around 9,000 images. In this study, we use the Caltech101 dataset provided by PyTorch (https ; datasets. Caltech101, html), allocating 75% of
//pytorch. org/ http//visiommmiini. wwwaaaainggeeeeeeeetrrrroroo htt.iiiiiiiondtasttss....
the images for training and the remaining 25% for validation. In our experiments, we preprocess the
images by resizing them to x266 cropping the center to 224x224, and normalizing them using the mean (0.485, 0.456, 0.406) and standard deviation (0.229, 0.224, 0.225) for the three channels.
E.2 General Language Understanding Evaluation Benchmark (GLUE)
CoLA: The Corpus of Linguistic Acceptability (CoLA) a dataset for assessing linguistic acceptabil- ity (Warstadt et al. 2018). This task is binary classification for predicting whether sentence is grammati- cally acceptable. The dataset is primarily from books and journal articles linguistic theory.
The Multi-Genre Natural Language Inference (MultiNLI) is a dataset designed to evaluate a model's ability to perform natural language inference (NLI). The task is to predict whether the premise entails the hypothesis, contradicts the hypothesis, or neither, The data set contains 433k sentence pairs annotated with textual entailment information (Williams et al., 2018).
MNLI:
The Microsoft Research Paraphrase Corpus (Dolan & Brockett 2005) is a dataset designed for evaluating paraphrase detection systems. It consists of sentence pairs, with binary labels of whether the two sentences in the pair are equivalent. The data are automatically extracted from online news and labeled by humans.
MRPC:
QNLI: The Stanford Question Answering Dataset (SQuAD) is a dataset designed for machine comprehension of text (Rajpurkar et al.) 20166. The dataset consists of question-paragrapy pairs, where one of the sentences in the paragraph contains the answer the corresponding question. The paragraphs are from Wikipedia and the questions written by human annotators.
32

---

The Quora Question Pairs (QQP) dataset is a dataset of question pairs (https://data. /wwps////aaaaauuaa.aaaa com/ The task is to determine whether two questions are Published in Transactions on Machine Learning Research (11/2025)
QQP:
semantically equivalent.
The Recognizing Textual Entailment (RTE) datasets are a series of challenges that evaluate models" ability to determine whether a premise can entail a given hypothesis (Dagan et al. 2006; Haim et al. 2006 Giampiccolo et al. 2007; [Bentivolli et al. 2009). The data are constructed based on the texts from Wikipedia and news. The datasets have been used to evaluate the performance of both traditional language models and the srate-of-the-ars LLMs.
RTE:
SST-2: The Stanford Sentiment Treebank is a dataset of sentences extracted from movie reviews (Sccher et al. 2013). Each sentence is labeled as either positive or negative. The task is to predict whether the sentence is positive or negative.
STS-B: The Semantic Textual Similarity Benchmark (STSB) is a dataset with sentence pairs collected from news headlines, video and image caplions, and natural language inference data (Cer et al.] 2017). The task is to predict the semantic similarity between pairs of sentences. Each pair of sentences is annotated with a similarity score ranging from 0 to 5, where 0 indicates no semantic similarity and 5 indicates semantically equivalent.
E.3 Text-Generation Datasets
GSM8k: GSMSK (Grade School Math 8K) is a dataset of 8792 high-quality grade school math problems, including problems in diverse languages. These problems take between 2 and 8 steps of elementary calculations using basic arithmetic operations (+- x+) solve. The dataset was created to support the task of question answering on basic mathematical problems to evaluate the model's ability of basic arithmetic reasoning.
Stanford Alpaca: Alpaca is instruction dataset designed for instruction training of pre-trained language models (aoor et al. 2023). It contains 52002 instruction-resppose pairs generated by OpenAl's text-davinci- 003 engine or written by humans. Note that there is only a training split in this dataset. Models fine-tuned on Alpaca are often evaluated by other tasks like "FleutherAT LM Harness". Alpaca-GPT is an updated version with the answers generated by GPT-4 (Achiam et al. 2023
ARC:
The AI2 Reasoning Challenge (ARC) dataset consists of grade-school level, multiple-choice science questions (Clark et al. 2018). ARC dataset includes Challenge Set and an Easy Set. The easy set contains questions that can be answered with straightforward reasoning, while the challenge set requires deoper understanding and more reasoning skills. The ARC-Easy includes 2251 training samples, 570 validation samples, and 2376 test samples and the ARC-Challenge includes 1119 training samples, 299 validation samples, and 1172 test samples.
Boolean Questions (BoolQ) is a dataset of yes/no no question answering (Clark et al. 2019) and includes 9427 training samples and 3270 validation samples. The dataset is designed assess models' comprehension and reasoning abilities. Each example contains question, passage, answer, and title.
BoolQ:
HellaSwag: HellaSwag is a dataset designed to evaluate the models' abilities in generating reasonable contexts [Zllers et al., 2019). It consists of prompts with a short context followed by multiple possible continuations. The goal is to find the correct or most plausible option. The training set, validation set, and test set have 39905 samples, 10042 samples, 10003 samples, respectively.
OpenBookQA: OpenBookQA is a question-answering dataset (Mihaylov et al. 2018) comprising 4957 training samples, 500 validation samples, and 500 test samples. It requires reasoning ability and a deeper understanding of common knowledge to answer questions. Each data contains short passage with multiple possible answers. The dataset emphasizes the integration of world knowledge and reasoning skills, making it a challenging benchmark for natural language processing models. It tests models' abilities to understand and apply factual information effectively to solve problems.
WinoGrande: WinoGrande tence (Sakaguchi et al.] (202)). It includes 40938 samples in the training set, 1,267 in the validation set, and 1,267 in the test set. The dataset is designed to assess models' commonsense reasoning abilities. The
datasct of 44k problems for choosing the right option for a given sen-
33

---

Published in Transactions on Machine Learning Research (11/2025)
examples contain sentences with inlnnlllanss that require the model to select the most appropriate option to complete the sentence.
SocialIQA: The SocialIQA dataset is a benchmark designed to evaluate a model's ability to reason about social interactions, including understanding social dynamics, intentions, and the effects human actions (Sap
et al. 2019). SocialTQA includes 33410 samples in the training set and 1954 the validation set.
PIQA: The PIQA (Physical Interaction Question Answering) dataset is a benchmark designed evaluate model's ability to understand and reason about everyday physical interactions and affordances (Bisk et al,, 2020). Here are some key details about PIQA: (Sakaguchi et al. 2021 PIQA contains 16113 samples in the training set and 1838 in the validation set.
34


Reviews:
[{"review_id": "rwEE7Pa17H", "reviewer": "Reviewer_x2ke", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "# Summary Of Contributions\nThis paper introduces a novel Sparse Fine-Tuning (SpFT) framework inspired by neural network pruning techniques to enhance the memory and computational efficiency of fine-tuning large language models. The proposed method identifies important neurons using structural pruning-based feature importance metrics and confines fine-tuning to weights associated with these selected neurons. This targeted approach enables a substantial reduction in memory usage—achieving 20–50% savings—while maintaining competitive performance with state-of-the-art methods such as LoRA and its variants on standard language tasks.\n\n\n# Strengths And Weaknesses\n\n## Strengths\n* The paper introduces a novel sparse fine-tuning (SpFT) framework that significantly improves memory efficiency over existing methods like LoRA, while relying solely on standard tensor operations, thus avoiding the complexities of custom sparse kernel implementations.\n\n* It offers practical support for fine-tuning quantized models, extending its utility to resource-constrained settings and further reducing memory overhead.\n\n* The authors present a thoughtful analysis of memory usage in PEFT methods, arguing that computation graphs play a more critical role in memory consumption than merely the number of trainable parameters—a useful insight for both researchers and practitioners.\n\n* The work includes rigorous empirical validation across diverse fine-tuning tasks, coupled with actionable training strategies, enhancing its relevance and applicability.\n\n* Two novel variants of Taylor importance—class-aware Taylor and Zeroth-Order Taylor—are proposed for different modalities. These demonstrate thoughtful adaptation to the specific needs of image classification and language modeling, respectively, with the latter requiring only a single forward pass in large models and featuring a variance reduction technique.\n\n\n## Weakness\n\n* The reliance on neural network pruning raises concerns about potential sensitivity to the pruning criteria; the robustness of the selected \"important\" neurons across different datasets or domains is not clearly discussed.\n\n* The theoretical grounding or comparative advantages of the Taylor importance variants (class-aware and Zeroth-Order Taylor) over existing importance metrics remain under-explored. Overall, the principles behind choosing these two methods needs further exploration and discussions.\n\n* There’s limited discussion on the potential training instability or convergence challenges introduced by the proposed novel work.\n\n* The computation and memory performance gain under various model scales or tasks is not discussed. Moreover, Although the method avoids custom sparse kernels, it's unclear whether the standard tensor operations used can fully leverage existing hardware acceleration, especially at scale—potentially limiting the claimed practical efficiency.\n\n* The integration of quantization is briefly mentioned, but the trade-offs between accuracy and compression in this context are not detailed.\n\n* Empirical results on the image domain are on very small datasets. Are they conclusive enough in terms of the efficiency gained while training converged with stability while performance is maintained?", "claims_and_evidence_yn": "No", "claims_explanation": "The claims are very well described and elaborated in the paper. The evidence is clear and accurate and well motivated and elaborated. There are some angles discussed in the weaknesses and requested changes sections regarding the convincing aspect of the paper that needs to be further discussed in the following reviewing stages. This will be revisited at the end for a final evaluation.", "audience_yn": "Yes", "audience_explanation": "The paper very well motivates the problem behind the memory efficiency in addition to the parameter efficiency. The novel idea of introducing structured pruning methods in similar fashion to LoRA is very interesting. Therefore, it is very likely the audience would find the paper interesting and the findings motivational to further explore this direction in the future.", "requested_changes": "Based on the aforementioned strengths and weaknesses, the requested changes are provided in an orderly manner from the highest to the lowest importance as follows:\n\n- Analysis over the sensitivity of the hyper-parameter r is missing. It is very instightful to show how much the gained efficiency is relied on a proper choince of r.\n\n- Analysis and discussion over the training instability and convergence challenges with respect to the amount of efficiency (computational and memory) gained should be provided. It is unclear the how tight is the choince of the current hyper-parameters with repect to the compression gained over the course of the training for a stable training.\n\n- There is a notation adding some confusion to the context in the second paragraph of section 4.1 where it says \" with the row selection matrix Mijj = 1 ...\".\n\n- what is p in Eq. 1?\n\n- Elaboration on this part is helpful as it is very concise and short on an important point; \"However, MeZO uses the trick of regenerating z dynamically using a random seed (of much smaller size than the model), thus eliminating the storage and ensuring memory usage that is equal to that of inference. We then justify the effectiveness of ZOTaylor in the following.\" It seems very vague at this state and needs more clarification of the context and elaboration of the material to help the audience gain further insights into this trick.\n\n- Can we look at the results of larger and more challenging image datasets to support the evidence behind the first importance approach vigorously?", "broader_impact_concerns": "There are no \"Broader Impact Concerns\" for this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RBe2RyfM0m", "reviewer": "Reviewer_JHRG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the author introduces SPruFT, a structured sparse fine-tuning method that selects the top-r neurons per layer and only trains their associated rows, reducing activation memory without custom kernels. Extensions include QSPruFT for quantized models using residual initialization, and ZOTaylor for efficient importance estimation with variance bounds. Experiments on LLaMA-2 and LLaMA-3 show competitive accuracy with LoRA-like methods while lowering memory.\n\nStrengths:\n- This paper tackles an important and prompt problem.\n- Simple, kernel-agnostic design with consistent memory savings and competitive accuracy. \n- Quantization-aware initialization that performs well in NF4.\n\nWeaknesses:\n- Some comparison between the proposed method and baselines seem not apple-to-apple. Your method “does not rely on dropout,” but others use dropout=0.1; please equalize dropout (and learning rates) to ensure memory/accuracy fairness. Also, for hardware consistency, some baselines ran on H100 vs A100; memory and throughput aren’t strictly comparable.\n- Experiments are restricted to LLaMA-2 and LLaMA-3, both in the 7B–8B range. Missing broader validation on other popular open-source LLMs (e.g., Mistral, Falcon, Qwen) and larger-scale models makes it harder to generalize the claimed efficiency benefits.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The core methodology is correct and the empirical results support the claim that simpler computation graphs reduce memory even with fewer parameters. The residual initialization and zeroth-order scoring show clear benefits. However, fairness issues (dropout, learning rate, GPU differences) and small-scale training data reduce the strength of the validation. Theoretical claims for ZOTaylor are not fully aligned with practice but the empirical evidence is consistent.", "audience_yn": "Yes", "audience_explanation": "Efficient fine-tuning of large models is an interesting, emerging, and important domain. The paper provides a simple, kernel-free alternative to LoRA-style approaches with both practical insight (graph-driven memory) and quantization integration, which would be of clear relevance to both researchers and practitioners.", "requested_changes": "Critical:\n\n- Re-run or report baselines with matched dropout and learning rate for fair comparison.\n\n- Clarify GPU differences in memory reporting (A100 vs H100).\n\nImportant but not critical:\n\n- Provide variance/CI across multiple seeds given small training sets.\n\n- Add one larger-data experiment for commonsense or GSM8K.\n\n- Strengthen ZOTaylor discussion by clarifying theory–practice limits and adding nk sensitivity analysis.\n\n- Extend evaluation beyond LLaMA-2/3 to include at least one other popular open-source model (e.g., Mistral, Falcon, Qwen) or larger-size LLaMAs.\n\nNice to have:\n\n- Report wall-clock time and tokens/sec.\n\n- Include GPT-4(o) MT-Bench scores for comparability.", "broader_impact_concerns": "No major ethical concerns beyond standard fine-tuning issues. The broader impact statement appears sufficient.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KKFRDlnxY6", "reviewer": "Reviewer_11jE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a Structured Pruning-Driven Sparse Fine-Tuning (SPruFT) framework, which selects a small subset of row/column parameters for fine-tuning based on neuron importance scores and employs low-rank decomposition for efficient training. The method is simple and implementation-friendly, with comprehensive experiments. Results on LLaMA-2/3 and multiple vision models outperform or match LoRA and other SOTA methods, while significantly reducing memory usage. Overall, the paper makes clear contributions with sufficient empirical evidence, though its novelty and depth of discussion remain limited.\n\nStrengths:\n\n1. No need for customized sparse operators, as it only relies on PyTorch’s native matrix multiplication for deployment.\n\n2. Covers 8 commonsense reasoning benchmarks, GSM8k, MT-Bench, and 3 vision datasets, comparing against 10+ PEFT methods including LoRA, QLoRA, DoRA, and RoSA.\n\n3. Derives the variance upper bound of Zeroth-Order Taylor importance estimation and provides a sampling complexity formula, laying a theoretical foundation for future research.\n\nWeaknesses:\n\n1. The core idea of “subnetwork selection via pruning importance + low-rank fine-tuning” is conceptually not new, resembling a combination of early Lottery Ticket Sparsity and LoRA.\n\n2. Reports only a few sparsity ratios, without showing the performance degradation curve as sparsity increases.\n\n3. The largest model evaluated is LLaMA-3-8B with sequence length ≤ 2048, leaving 70B-scale or longer-context scenarios unverified.\n\n4. Some key symbols (e.g., η, M) lack intuitive illustrations when first introduced.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors present systematic comparisons against more than ten PEFT baselines (including LoRA, QLoRA, DoRA, and RoSA) across a diverse set of benchmarks (8 commonsense reasoning tasks, GSM8k, MT-Bench, and 3 vision datasets). They further strengthen the empirical credibility of their work by providing detailed memory usage measurements, training hyperparameters, raw logs, and open-source code for full reproducibility.", "audience_yn": "Yes", "audience_explanation": "SPruFT provides a systematic solution to the critical challenge of efficiently fine-tuning billion-parameter models under limited GPU memory. The experiments span both LLaMA-2/3 and various vision Transformers, showing performance on par with LoRA and other state-of-the-art methods while further reducing memory usage by 20–50%. The method requires no custom CUDA kernels, and the accompanying code and scripts are open-sourced for reproducibility.", "requested_changes": "The paper should provide the full sparsity–performance curve, as it currently reports results only at around 2% sparsity. I recommend adding experiments in the appendix covering the 0.5%–10% range (under the same task and rank), to clearly illustrate the degradation point and suggest a recommended sparsity interval.\n\nFurthermore, the paper needs to more explicitly differentiate SPruFT from earlier approaches such as Lottery Ticket + LoRA, S-LoRA, or Block-Sparse LoRA. For instance, clarifying its novelty in terms of the neuron-importance objective, row–column selection strategy, or quantized-residual initialization would strengthen the contribution. Adding a dedicated comparison paragraph in the Related Work section would help address concerns about conceptual overlap.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)