SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity
Authors: Samir Khaki
Xiuyu Li
Junxian Guo
Ligeng Zhu
Konstantinos N. Plataniotis
Amir Yazdanbakhsh
Kurt Keutzer
Song Han
Zhijian Liu
Venue: ICML 2025 poster (2025)
Keywords: large language models
efficient fine-tuning
sparsity
peft
Primary area: deep_learning->large_language_models
TL;DR: This paper introduces SparseLoRA, a method that uses contextual sparsity to accelerate LLM fine-tuning, cutting compute by up to 2.2× and runtime by 1.6× while maintaining model accuracy on reasoning, coding, and instruction-following tasks.

Abstract: Fine-tuning LLMs is both computationally and
memory-intensive. While parameter-efficient fine-tuning methods, such as QLoRA and DoRA,
reduce the number of trainable parameters and
lower memory usage, they do not decrease computational cost. In some cases, they may even
slow down fine-tuning. In this paper, we introduce
SparseLoRA, a method that accelerates LLM fine-tuning through contextual sparsity. We propose
a lightweight, training-free SVD sparsity estimator that dynamically selects a sparse subset of
weights for loss and gradient computation. Also,
we systematically analyze and address sensitivity
across layers, tokens, and training steps. Our experimental results show that SparseLoRA reduces
computational cost by up to $2.2\times$ and a measured
speedup of up to $1.6\times$ while maintaining accuracy across various downstream tasks, including
commonsense and arithmetic reasoning, code generation, and instruction following.

SparseLoRA:
Accelerating LLM Fine- Tine-Tuning with Contextual Sparsity
Samir Khaki") Xi 1 Junxian Guo Ligeng Zhu Konstantinos N. Plataniotis
Amir Yazdanbakhsh Kurt Keutzer Song Han Zhijian Liu
https://z-labb htw.:b....//rojeetsssspsssoor
Abstract
SparseLoRA speedup Reasoning
LoRA
GLoRA
DeRA
SparseloRA
Fine-tuning LLMs both computationally and memory-intensive. While parameter-cfficient fine- tuning methods, such as QLoRA and DoRA, reduce the number of trainable parameters and lower memory usage, they do not decrease putational cost. In some cases, they may even slow down finetuning. In this paper, we introduce SparseLoRA, a method that accelerates LLM fine- tuning through contextual sparsity. We propose lightweight, training-free SVD sparsity estima- tor that dynamically selects a sparse subset of weights for loss and gradient computation. Also, systematically analyze and address sensitivity across layers, tokens, and training steps. Our ex- perimental results show that SparseLoRA reduces computational cost by up to 2.2x x and a measured speedup of up to 1.6x while maintaining accu- racy across various downstream tasks, including commonsense and arithmetic reasoning, code gen- eration, and instruction following.
OLoRA
DeRA
809
Ours
384
436
290
430
570
710
850
240
480
760
902
(mo)
Backward [ms]
SparseLoRA maintains lossless performance Rozsoning Code Generation
88.0
$2.0
47.0
1.3x Faster
1.6x Faster
1.3x Faster
97.4
51.4
44.0
63.1
63.9
$6.7
38.0
85.6
79.6
35.0
HumanEwl
SparseLoRA accelerates fine-tuning 1.5x Instruction Following tasks
9.0
MT Menchch
1.8
Cooleg
Futraction Hamonitee
Marh
Reaeoning
Relephy
STEM
Whiting
Figure 1: Many recent parameter-efficient fine-tuning meth- ods, such as QLoRA and DoRA, do not reduce compute. Our SparseLoRA accelerates LLM fine-tuning with minimal accuracy loss across a range of downstream tasks, including commonsense reasoning, math reasoning, code generation, and instruction following. See Section 4 for more details.
1. Introduction
Large language models (LLMs) are trained on vast, general- domain datasets. They are often fine-tuned to improve their performance in specific domains (Saab et al., 2024) or to align their predictions with user preferences (Zhang et al., 2024a). However, fine-tuning very large models can be pro- hibitively expensive, both in terms of memory requirements and computational costs.
follow-up methods (Dettmers al., 2023; Liu al., 2024c) have been proposed further reduce the number of trainable parameters. While they effective in reducing memory usage, they do not reduce computation. In fact, they can sometimes slow down fine-tuning due to the overhead they introduce: DoRA is 20% slower than LoRA (see Figure 1).
Extensive efforts have been made in parameter-efficiees fine- tuning (PEFT) reduce memory consumption of LLM fine- tuning. LoRA (Hu et al., 2022) represents weight updates using low-rank approximations. Building upon this, many
In this paper, we present SparseLoRA to accelerate LLM fine-tuning with contextual sparsity, making both memory- and computation-efficceet
Contextual sparsity has already been used in accelerating LLM inference (Liu et al., 2023b). SparseLoRA shows for the first time that it can also play role in LLM fine-tuning, where (1) only sparse subset of weights is required for loss and gradient computation, and (2) this sparse subset needs to be determined based on the input sequence or tokens. To realize this, we propose
Equal contribution 'Pojeect lead 'University of Toronto 'UC Berkeley 'MIT "Google DeepMind. Correspondence to: Samir Khaki "'sami..hake @ mail. utoronto.ca>.
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity
Layer Norm
PoPE
LoRA
aKC Pro
Attendion
FFN
Other
ferent sparsity patterns, necessitating alternative methods to reintroduce and exploit sparsity.
128
18%
1195
58%
10%
258
8%
15%
6%
50%
10%
$1P
8%
15%
5%
69%
9%
Recent work has explored various techniques, including continued pretraining (Song et al., 2023; Zheng et 2024a; Zhang et al., 2024c; Song et al., 2024a;b; Xue et al., 2024) and magnitude pruning with specific metries (Lee et al., 2024; Akhauri et al., 2024; Liu et al., 2024a; Zhang et al., 2025b), to leverage sparsity in LLMs. These approaches aim to reduce memory usage and computation time, with some work deploying small neural networks to predict non- zero activations (Liu et al., 2023b; Alizadeh et al., 2024; Akhauri et al., 2024; Song et al., 2023; Xue et al., 2024). In this work, we propose a novel approach to extend the benefits of contextual sparsity to the fine-tuning process for the first time, which addresses the associated challenges and accelerates fine-tuning without compromising performance.
1094
7%6
15%
51%
60%
10%
2048
816
14%
2%
63%
10%
25%
50%
75%
100%
Figure 2: Runtime breakdown of LLaMA3-8B fine-tuning under different sequence lengths.
using an SVD sparsity estimator to identify which channels should be activated based on samples within each batch. It is very lightweight, adding only 0.05% FLOOPs and 0.8% runtime overhead to fine-tuning. It is training-free, unlike the look-ahead predictor proposed in Deja Vu (Liu et al., 2023b), which leads to better generalization across datasets.
We have conducted systematic sensitivity analysis across multiple dimensions. First, each layer responds differently to sparsity, so we apply non-uniform sparsity based on layer sensitivity analysis. Second, output tokens much more sensitive to pruning than context tokens, so we apply sparsity only to context tokens. Finally, early iterations in fine-tuning are more sensitive, so we run dense fine-tuning in the early iterations and switch to sparse fine-tuning for the remainder.
Memory-Efficient Fine-tuning. As language models grow larger, memory-efficiet fine-tuning methods have become crucial. Parameter-efficient fine-tuning (PEFT) techniques address this challenge by updating only a small subset of parameters. LoRA (Hu et al., 2022) employs low- rank matrices to adjust pretrained model weights, sparking a rich line of research with numerous works proposing im- provements and variations (Dettmers et al., 2023; Shi et al., 2023; Qiu et al., 2023; Chen et al., 2024; Kopiczko et al.,
Evaluated across diverse set of benchmarks, SparseLoRA achieves a computational cost reduction of up to 2.2x and wall-clock speedup of up to 1.6x while maintaining accu- racy on various downstream tasks, including commonsense and arithmetic reasoning, code-generation, and complex in- struction following. To the best of our knowledge, this is the first work to leverage contextual sparsity for accelerating LLM fine-tuning. We believe that our work will inspire future research into fine-tuning methods that optimize both parameter and computational efficiency.
2024; Liu et al., 2024c; Meng et al., 2024; Hayou et al.,
2024; Wang et al., 2024; Wang & Liang, 2024; Liu et al.,
2024d; Pan et al., 2024; Yang et al., 2024). Among these, DoRA (Liu et al., 2024c) reparameterizes weight matrices to achieve more effective optimization. QLoRA (Dettmers et al., 2023) combines quantization with low-rank adapters, enabling fine-tuning of large models on a single GPU.
Recent research has also focused on exploiting the low- rank structure of PEFT (Nikdan et al., 2024; Agarwal et al., 2024; Zhang et al., 2025a). GaLore (Zhao et al., 2024) and its weight-quantizee the low-rank property of weight gradients to reduce opti- mizer state memory, while WeLore (Jaiswal et al., 2024) investigates how low-rank weights emerge from low-rank gradients during training. While these advancements make LLM adaptation more memory-efficient and accessible, they primarily focus reducing memory usage improving the accuracy, sometimes even at the cost of increased com- putation time. Our approach addresses this missing piece by focusing on compute efficiency, complementing exist- ing memory-efficien techniques to enable truly resource- efficient fine-tuning.
2. Related Work
Contextual Sparsity in LLMs. Sparsity has long been employed to enhance the efficiency of neural networks (Han et al., 2016b; 2015; 2016a; Frantar & Alistarh, 2023; Sun et al., 2024). Recently, there has been research dynamic approach: accelerating LLMs inference by leverag- ing contextual sparsity (Liu et al., 2023b; Song et al., 2023;
variant (Zhang et al., 2024b) leverage
more
Xue et al., 2024; Alizadeh et al., 2024; Lee et al., 2024:
Akhauri et al., 2024; Liu et al., 2024a; Zhang et al., 2025b)
at test time. Unlike static sparsity, contextual sparsity is a phenomenon where significant portions of model's 's hidden states dynamically contain vero-valued neurons based on the input context. This allows for aaput-dependent sparse com- putation without compromising outcomes. While such acti- vation sparsity naturally emerges in ReLU-based FFNs (Li et al., 2023b; Mirzadeh et al., 2023; Liu et al., 2023b; Al- izadch et al., 2024), newer architectures often employ non-
Computation-Eltiiien Training, Prior work has ex- plored sparsity to accelerate LLM training through var- ious approaches (Thangarasa et al., 2023; Chen et al., 2024; Mozaffari et al., 2024). LongLoRA (Chen et al.,
ReL.U activations (Team, 2023; 2024c;a;b) that create dif-
2

---

SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity
Pre-Trained Weight
SVD Sparsity Estimator
Sparse Weight
Offline SVD Decomposition
X
Y
X
Identify which channels are important for X.
We do not apply sparsity to the LoRA branch as it is very fast.
Low-Rank Update
Low-Rank Update
(a) Low-Rank Adaptation (LoRA)
(b) LoRA with Contextual Sparsity (SparseLoRA)
Figure 3: Spa accelerates LLM fine-tuning with contextual sparsity. It first performs an offline SVD decomposition of the pre-trained weights to construct the SVD sparsity estimator. During fine-tuning, relies on the SVD sparsity estimator to identify which weight channels are needed for the given input sequence. then sparsely computes by slicing the weights on the fly. Note that sparsity is applied only the main branch, the LoRA branch is typically very fast.
2024) enables efficient context window extension via shifted sparse attention during fine-tuning while retaining stan- dard attention at inference, reducing quadratic computation costs. General sparse training techniques have demonstrated promising FLOPs reduction through combinations of sparse pre-training and further fine-tuning (Thangarasa et al., 2023; Mozaffari al., 2024). Other methods employ dynamic sparsity patterns through gradient-based selection (Zhou et al., 2021; Li et al., 2022). Yet these techniques either focus on long-context scenarios where attention dominates computation, or rely on unstructured sparsity, whose theo- retical gains are challenging to realize on consumer-grann -raad GPUs. While there have been recent attempts to apply struc- tural sparsity (Ma al., 2024: Chen al., 2025), they either not memory efficient only partially accelerate fine- tuning (e.g., the backward pass), limiting their effectiveness. Our work introduces structured contextual sparsity with training- waining-free dynamic selection, enabling practical end-to- end acceleration while remaining memory-efficient.
memory- and computaaion--efficien fine-tuning with no impact on inference performance.
3.1. Sparse Neuron Selection Criteria
To achieve compute-efficiet fine-tuning while maintaining effectiveness, we require dynamic rather than static sparsity patterns that adapt to each input. Prior research has ex- plored contextual activation sparsity in LLM inference (Liu et al., 2023b; Alizadeh et al., 2024; Akhauri et al., 2024; Liu et al., 2024a). However, these inference methods target single-token computations in auto-regressive generation and do not directly translate fine-tuning, which has distinct workloads that consist of multiple sequences of tokens in batch. To bridge this gap, we first define "oracle" criteria for fine-tuning neuron selection using intermediate activa- tions, establishing an ideal but computationally infeasible metric. This oracle then guides the practical development of efficient approximations that enable on- on-the-fly channel selection for sparse computation. We categorize the linear layers in LLMs into three types FFN, VO projections, and QK projections and propose two oracle criteria tailored to their properties as follows.
3. Method
In this section, present the design of our SparseLoRA. As shown in Figure 2, linear layers dominate LoRA fine- tuning runtime in conventional settings. Therefore, we ap- ply dynamic channel sparsity to the main branch of LoRA fine-tuning while keeping the LoRA branch dense. This ap- proach selectively activates only the most important neurons FFN and attention layers. Since the main branch accounts for the vast majority of computation, and the sparsity in- achieve
1. SELECTION WITH L2 NORM
Motivated by the extreme sparsity in input activations for certain linear layers, such the SiLU-induced sparsity in the down projection of FFNs (Alizadeh et al., 2024; Lee et al., 2024; Song et al., 2023) as shown in Figure 4, introduce an L2 Norm metric to identify and retain the most significant neurons as the first oracle criterion. Importantly, this approach allows us to naturally extend the sparsity pat- tern to the preceding linear layers specifically using FFN as an example, the channel indices selected for the down
troduced is structural and mardware-frienlly significant efficiency improvements without altering which parameters are updated. By sparsifying only the base model while keeping LoRA intact, SparseL LoRA maintains both
3

---

SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity
FFN Input
Gate & Up Projs
Intermediate Output
Attn Input
Attn Output
Down Proj
OKV Proj
Projj
Attn
Layer 20
Figure 4: Activation value distributions across all tokens of a sequence for layer 20 of LLaMA2-7B. Each column shows inputs to: W gate,.up Wdown of FFNs, and Wq.k.v, Wo of attention projections. The inputs to Wdown and Wo follow Laplace distributions, enabling higher sparsity when using the L2 norm metric. W gate,up is pruned alongside corresponding W Woown channels and Wy is pruned alongside corresponding Wo channels, while Wo,k requires a different criterion.
3.1.2. SELECTION WITH QK NORM
For attention blocks, we aim to sparsify the linear layers as well: query, key, value, and output projections. While the L2 Norm metric works well for VO projections, it proves im- practical for QK projections due their lower input activa- tion sparsity shown in Figure 4. Previous approaches ex- ploring contextual sparsity during inference have proposed identifying and pruning unimportant attention heads (Liu et al., 2023b; Akhauri et al., 2024). However, this head-level pruning strategy proves problematic for fine-tuning scenar- ios that process multiple tokens simultaneously. Unlike FFNs where we can selectively choose from 11,384 chan- nels in LLaMA2-7B, pruning entire attention heads (e.g.. removing I out of 32) significantly constrains our pruning granularity and risks losing critical information. We empiri- cally verify 4.3 that this coarse- coarse-grained approach leads to degraded performance. Additionally, detailed analysis of attention head activation patterns during fine-tuning provided in Appendix A.I.
W
SILUUXW ause))
dowa
W
X W up
W pate
SxLUUxxW pum)
W Woown
W
xWup
Figure 5: An illustration of how sparsity is applied FFNs.
To address this challenge, we introduce oracle criterion for sparsifying QK projections based on attention scores, targeting channels with minimal contributions. Specifically, we define a proxy metric that quantifies each channel's importance and sparsify those with the lowest values. Given query and key projections Q, K E (Bx L)x where B is the batch size, L the sequence length, and D is the hidden dimension, compute their L2 norms across the flattened batch and sequence dimensions:
projection can be applied to both up and gate projections, as illustrated in Figure 5.
While prior work has applied similar magnitude-basee met- rics for inference with single-token (Liu al., 2023b; Lee et al., 2024; Akhauri et al., 2024; Liu et al., 2024a), we extend this to fine-tuning by considering cumulative activa- tions across all samples and tokens a batch. Despite recent exploration of incorporating gradient and weight informa- tion (Akhauri et al., 2024; Sun al., 2024), we find the L2 norm of activations alone provides simple yet effective criterion, activations generally contain more influential outliers than weights (Xiao et al., 2023). We apply this metric to both FFNs and the outer channels of value and output (VO) projections attention, where input activations to output projections show similar sparsity characteristics as demonstrated in Figure
|
2
k
K 2--
The element wise product of these normalized scores serves as our importance metric:
s k.
We retain the top n channels based on s, where n deter- mined by the desired sparsity ratio.

---

SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity
This approach ensures that only the projection channels contributing most significantly to the attention scores are retained. Compared to the L2 norm criterion from FFNs, this method better preserves the original computational out- comes. As demonstrated in Figure 6, our proposed oracle criterion maintains an attention map much more similar to the original dense QK computation compared to those derived by L2 norm or random pruning.
the full model while maintaining efficiency. The SVD de- composed components computed offline and loaded the start of fine-tuning alongside the model weights. No- tably, while low-rank module overheads can be significant in inference-tine sparsity methods due to memory- -ound exe- cution (Akhauri et al., 2024; Liu et al., 2023b), our approach introduces minimal overhead (less than 1% of runtime) since the lightweight SVD projections are negligible compared to the matrix multiplications in the compute-bound LLM fine-tuning. This enables us to achieve dynamic sparsity with negligible computational overhead, preserving both fine-tuning efficiency and model performance.
3.2. SVD Sparsity Estimator
The oracle criteria demonstrate that adaptively sparsifying the backbone weights during fine-tuning can significantly reduce computation while maintaining model quality. How- ever, computing these oracle patterns requires partial dense computation such as the gate and up projections for FFN intermediate activations and QK projections for QK norms, making it impractical and negating potential speedup bene- fits. The utility of SVD in identifying the most significant components of weight matrices is esta-estallished in model compression (Nikdan et al., 2024; Li* et al., 2025; Liu et al., 2024b; Zhang al., 2025b), suggesting its potential in our setting. To address this challenge, we introduce an efficient SVD-based low-rank sparsity estimator that dynamically se- lects channels with minimal overhead. Our method directly approximates the oracle criterion using top-k singular value decomposition (SVD) of the model weights, in contrast to prior approaches that employ learned low-rank predic- tors (Liu et al., 2023b; Alizadeh et al., 2024; Akhauri et al., 2024), which raise concerns about generalization across different datasets and tasks.
3.3. Sensitivity Analysis
Layer Sensitivity: Adaptive Sparsity Configuration The inherent contextual sparsity across layers in LLMs often varies significantly (Liu et al., 2023b; 2024a). Moreover, the importance of individual layers and their contributions to fine-tuning can differ substantially (Gromov et al., 2024; Zheng al., 2024a), necessitating layer-specific sparsity configurations for optimal performance. To determine these configurations, we conduct a systematic layer sensitivity analysis using a subset of the Commonsense Reasoning task proposed by Hu et al. (2023) as a proxy. Our analysis evaluates how different sparsity ratios affect each layer's per- formance independently - starting with a densely fine-tuned model, we progressively increase the sparsity ratio for each layer while keeping others dense, measuring performance each time to generate layer-specific sensitivity curves.
The results for LLaMA2-7B, shown in Figure 7, reveal that deeper layers contain more redundant information and are more amenable to sparsification than shallower layers, aligning with inference-time observations from Gromov et al. (2024). These sensitivity metries enable to apply aggressive sparsity to deeper, resilient layers while main- taining shallower ones dense, optimizing the performance- efficiency trade-off during fine-tuning.
Algorithm SVD Sparsity Estimator
# Assume input tensor x of shape (B, S, D1)
# Weight W of shape (B, D1, D2), SVD rank of k
Compute activations wi oracle out = torch. bmm(h W)
# Compute low-rank SVD weights (saved offline) U, s, = torch.linalg, (W, hull_matrice==Faass
Token Sensitivity: Context-Output Aware Sparsity The effectiveness of sparsity varies not only across layers but also across tokens within sequence. In LLM fine-tuning, input sequences typically consist of a context (the prefix to- kens provided as input) and output tokens (the target tokens used for loss computation). We find that applying uniform sparsity across all tokens degrades performance, as output tokens play a more critical role in optimization.
W_A = UC:, :k] @ (iag(S(:k))).....
+ W_B = torch. dicchdiaggS(:k)) sqrt() @ V[:k, :]
# Compute activations with loaded SVD estimator WLA), W_B)
out torch. borch..mm(tee
# Obtain channel indices with corresponding metric indices = metric(out)
To address this, we propose a context-output aware spar- sity strategy, selectively preserving dense computation for output tokens while applying sparsity to the context. This ensures that fine-tuning retains full expressiveness where it matters most while still benefiting from reduced computa- tion. Unlike heuristic-based token importance sampling, our approach exploits natural structural distinction context
The core idea is straightforward: instead of training a pre- dictor to map inputs to sparsity masks, we project inputs onto low-rank SVD decomposition of the original weights and compute the oracle metric from these projected acti- vations, as detailed in Algorithm 1. This approach pro- duces sparsity masks that closely match those obtained from
5

---

SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity
Dense
Sparse (Attn importance)
Sparse (Input L2 Norm)
Sparse (Random)
Token Index (Query)
8
10.
10
10
14-
14
14
16
18
20
20
20
26
(Key)
Figure 6: Comparison of attention maps for different pruning strategies at 50% sparsity. The figure shows the attention maps for the last 25 tokens of head using dense full attention, pruning based on attention importance (ours), pruning based on input L2 norm (simlar to FFNs), pruning randomly.
1095
50%
30% maintaining dense computations for a small portion of steps can significantly enhance final convergence (Lu et al., 2023; Thangarasa al., 2023; Bambhaniya al., 2024) with mini- mal impact overall speed-up. In our SparseLoRA design, allow the initial steps, up to maximum of 10% of the fine-tuning process, to remain dense. This approach en- sures the model establishes a strong foundation early on while still benefiting from sparse training's 's efficiency in later stages. A detailed analysis of this hybrid approach's impact on performance and efficiency can be found in 4.3.
Accuracy
0.37
10
20
25
30
Layer Index
Figure 7: Sensitivity analysis on layer-wise sparsity of LLaMA2-7B.
4. Experiments
4.1. Setup
Contect:: Jny can read papes a book::2 minutes, mony hours
Benchmarks. We conduct experiments on five down- stream tasks. The first set focuses on commonsense reason- ing (referred to as CSR170K) and includes eight datasets: BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap al., 2019), HellaSwag (Zellers al., 2019). WinoGrande (Sakaguchi et al., 2021). ARC-Easy and ARC- Challenge (Clark et al., 201 8), and OpenbookQA (Mihaylov et al., 2018). The second set focuses arithmetic reasoning (referred to as Math10K' and includes three benchmarks: GSM8K (Cobbe et al., 2021), 1), MAWPS et al., 2016), and SVAMP (Patel et al., 2021)'. Following the practices established by Hu et (2023) and Liu et al. (2024c), fine-tune models the combined train- ing sets of all sub-tasks within each respective benchmark. We run each experiment five times, discard the highest and lowest performing runs, and report the average accuracy of the remaining three. We further assess the generality of our method on three additional tasks: sequence classification using the GLUE benchmark (Wang al., 2018); instruction following, where we train on WizardLM (Xu et al., 2024) Step Sensitivity: Progressive Sparse Fine-tuning To balance efficiency and model quality, we incorporate a hy- brid approach in our fine-tune tuning process. Recent studies suggest that when doing incorporating sparsity in training,
Sparse Computation
Outpat:
minubes...
120:36-57-5 hours nead 200 pager.
Dense Computation
Figure 8: Output tokens go through the dense computation in our context-output aware sparsity strategy. The final outputs are gathered from both sparse and dense results.
-enzeel-eddiorsii
tokens inherently less sensitive to precise weight updates than output tokens, as illustrated in Figure 8. This strategy significantly mitigates the reconstruction errors between sparse and dense training, particularly in early fine-tuning steps where maintaining gradient signal is crucial. This simple yet effective approach improves computational effi- ciency while preserving fine-tuning performance.
We exclude AQuA (Ling al., 2017) since none of the meth- ods in Hu (2023) achieve better-than-randde performance (i.c., significantly above 20% for 5-choice multiple-choice task).
6

---

SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity
#FL.Pss
Speedup
Average
BoolQ
PIQA
SIQA
HellaS
WinoG
ARC-e
ARC-c
OBQA
LLaMA2-7B
31.4
$1.0
49.5
32.4
25.0
20.4
23.7
22.1
26.6
LoRA
100%
1,0x
82.3
70.7
84.8
81.4
90.0
85.8
87.6
74.3
84.7
QLLRR
100%
0.9x
82.5
69.2
84.7
81.9
85.8
87.9
74.2
85.4
DORA
132%
0.7x
81.7
71.4
84.7
81.1
90.0
85.2
87.3
72.8
84.2
+ SparseLoRA
65%
1.3x
81.8
69.7
84.3
80.8
88.4
86.0
86.7
73.4
84.0
LLaMA2-13B
35.0
61.9
49.8
31.8
25.6
17.2
33.3
30.5
29.6
LoRA
100%
1.0x
84.7
72.0
86.7
82.2
91.0
89.0
90.9
80.5
85.6
SparseLoRA
61%
1.3x
85.0
74.1
87.1
82.4
92.3
88.3
89.8
78.7
86.9
LLaMA3-8B
62.5
66.1
75.4
53.8
54.8
42.1
80.7
67.3
59.8
LoRA 100%
1.0x
87.1
74.6
89.4
82.7
95.4
89.1
92.8
83.4
89.3
+ QLoRA 100%
0,9x
87.1
74.3
89.3
83.1
95.3
88.7
92.9
83.8
89.4
DoRA 132%
0.8x
87.1
74.5
89.4
83.0
95.4
88.8
93.2
84.0
88.9
SparseLoRA 65%
1.3x
86.9
75.0
89.6
82.8
94.9
88.7
92.7
82.9
88.3
Table 1: SparseLoRA delivers up to 1.3x speedup and reduces fine-tuning FLOPs by up to 39%, while maintaining
performance comparable existing methods such as LoRA, QLoRA and DoRA on commonsense reasoning benchmarks.
#FL.
Spd. Avg.
GSMSK SVAMP MAWPS
While SparseLoRA is built on top of LoRA, can, in prin- ciple, be applied to any other PEFT method. We include model training details in Table 12. All PeFT methods only fine-tune the QK VO projections using a rank of 32, scal- ing factor of 64, and no dropout. Efficiency metrics are derived from an NVIDIA A6000 GPU.
LLaMA2-7B
2.6
2.7
3.1
2.1
+ LORA
100%
1.0x
54.6
38,6
47.5
77.5
+ QLoRA
100%
0.9x
55.0
36.2
49.7
79.1
+ DoRA
132%
0.7x
54.5
38.4
48.4
77.5
+ SOA
73%
1.2x
53.7
37.6
46.4
77.9
I.AMMA-13B
13.4
4.9
18.8
16.4
4.2. Main Results
+ LoRA
100%
1.0x
63.5
50.2
59.1
81.0
Table demonstrates results on the CSR170K bench- marks, demonstrating that SparseLoRA significantly re- duces computational requirements while maintaining ac- curacy comparable LoRA. For instance, on LLaMA2- 13B, SparseLoRA achieves an average accuracy of 85.0, outperforming LoRA's 84.7, while requiring only 61% % of its training cost with a 1.3 speedup in its training time. This trend extends to LLaMA3-8B, where SparseLoRA consis- tently reduces compute load while preserving competitive accuracy across tasks such as BoolQ, PIQA, SIQA, Hel- laSwag, WinoGrande, ARC, and OBQA.
+ SparseLoRA
70%
1.3x
62.7
49.5
57.1
81.5
LLaMA3-8B
33.5
25.0
38.4
37.0
LoRA
100%
1.0x 81.0
71.8
80.3
90.9
+ QLORA
100%
0.9x
80.6
71.8
80.2
89.6
+ DORA
132%
0.8x
81.0
72.5
79.3
91.0
+ Spa
46%
1.6x
81.1
72.0
80.2
90.9
Table 2: Sparsel.RR offers up to 1.6 1.6x speedup and reduces fine-tuning FLOPs by up to 54% on arithmetic reasoning tasks with accuracy comparable existing methods.
Table presents results on the Math10K benchmarks, fur- ther reinforcing SparseLoRA's compute-accuree advan- tage. On LLaMA3-8B. SparseLoRA reduces training costs by 54% and achieves a 1.6x speedup on LoRA while maintaining strong performance on SVAMP and MAWPS while slightly surpassing LoRA on GSM8K. These findings demonstrate that structured contextual sparsity can signifi- cantly reduce computational overhead in parameter-effiient fine-tuning without sacrificing performance in common- sense and mathematical reasoning tasks.
and evaluate MT-Bench (Zheng et al., 2023); and code generation, where train CodeFeedback (Chen al., 2021) and test on HvmanEval / HumanEval+ (Zheng et al., 2024b; Liu et al., 2023a).
We use LLaMA2-7B/13B and LLaMA3-8B (In- struct) as base models for fine-tuning. For instruction following and code generation benchmarks, additionally use LLaMA3.1-8B.
Models.
Baselines. We compare our method with two PEFT meth- ods, LoRA (Hu al., 2022) and DoRA (Liu et al., 2024c).
Natural Language Understanding. We evaluate
SparseLoRA using LLaMA3-8B on sequence classification
7

---

SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity
#FLOPs
Speedup
Average
COLA
STS-B
MRPC
RTB
SST2
QNLI
WNLI
MNLI
QQP
LLaMA3-8B + LORA
100%
1.0x
87.3
65.8
88.8
87.7
82.8
96.4
95.7
45,5
91.8
89.8
LLaMA3-8B + SparseLoRA
1.3x
87.7
66.3
89.3
88.6
82.9
96.6
96.6
55.9
91.8
89.8
Table 3: SparseLLRRA accelerates fine-tuning for sequence classification the GLUE benchmark by 1.3x.
#FLOPs
Speedup
Average
Coding
Extraction
Humanities
Math
Reasoning
Roleplay
STEM
Writing
LLaMA3.1-8B
4.08
2.15
2.88
4.50
2.05
2.60
5.72
7.90
4.80
LoRA
100%
1.0x
6.03
5.10
5.50
8.00
2.25
5.30
7.35
7.78
7.00
SparseLLRRA
53%
1.5x
6.06
5.30
5.45
8.10
2.20
5.35
7.35
7.70
7.05
Table Sparsel A delivers strong performance on instruction-follooinng tasks MT-Bench, achieving a 1.5x speedup while matching or exceeding LoRA across all categories.
with a subset of GLUE benchmark (Wang et al., 2018). Table 3 shows that SparseLoRA maintains competitive performance to the dense baseline with 1.3x speedup.
formance to the LoRA baseline on the pass@k (k=2,4) set- tings, while achieving up to a 1.3x speedup.
#FLOPs
Speedup
pass @
@4
Instruction Following. We evaluate SparseLoRA using LLaMA3.1-88 on the task of instruction following by fine- tuning subset of the WizardLM dataset (Xu al., 2024) and reporting scores across eight tasks in the MTT-Bench dataset (Zheng et al., 2023). We use GPT-4 to assess the quality of model responses. Table shows that SparseLoRA maintains competitive performance to the dense baseline while achieving up to 1.5x speedup.
LLaMA3.1-8D
24.2
29.0
LoRA SparseLoRA
100%
1.0x
33.7
37.4
71%
1.3x
33.9
37.5
Table SparseLoRA accelerates test-time time- on ARC- AGI by up 1.3x while maintaining performance.
Compatibility with PEFT Methods. Techniques such as gradient checkpointing and quantization (e.g.. QLoRA (Dettmers al.. 2023). LofiQ (Li al., 2023a)) primarily aim to reduce memory usage but often increase runtime, shown in Figure 1. These methods therefore orthogonal and complementary to our approach. As in Table 7, when combined with QLoRA, SparseLoRA achieves both lower memory usage and faster runtime.
Code Generation. We evaluate SparseLoRA using LLaMA2-7B and LLaMA3.1-80 on code generation by fine-tuning on a subset of the CodeFeedback dataset (Chen al., 2021) and testing on the HumanEval bench- marks (Zheng al., 2024b; Liu et al., 2023a). Table 5 shows that SoarseLoRA maintains competitive performance to the dense baseline while achieving up to 1.3x speedup.
#FLOPs Speedup HumanEval HumanEval+
4.3. Analysis
LLaMA2-7B
3.6
3.0
SVD Sparsity Estimator. The SVD sparsity estimator is key to SparseLoRA ability to apply contextual spar- sity with minimal impact on performance. As shown in Table 8. our estimator achieves performance comparable the Oracle method on the MathlOK dataset, demonstrating its effectiveness. Using rank 8 singular value decomposi- tion of the base model weights, the SVD sparsity estimator is lightweight and training-free, adding only 0.8% overhead to the end-to-end runtime.
LoRA
100%
1.0x
13.0
10.2
+ SoarseLoRA
73%
1.2x
12.8
11.0
LLaMA3.1-8B
30.9
27.9
+ LoRA
100%
1.0x
43.1
36.2
+ SpaarsLoRA
1.3x
43.9
37.0
Table 5: Sp accelerates fine-tuning for code gen- eration by up 1.3x while maintaining performance.
Effect of Output Token Splitting. We investigate the ef- fectiveness of our context-output splitting strategy, which optimizes computational efficiency by preserving dense computation for output tokens while sparsifying the con- text tokens. To evaluate the impact of this design choice, we compare three configurations: (1) the baseline, which
ARC-AGI. We evaluate Sparsel.RRA using fine-tuned version of LLaMA3.1-88 following Bare (Li et al., 2024) on test-time training for the abstract and reasoning corpus of artificial general intelligence (ARC-AGI) (Chollet, 2019).
Table 6 shows that LoA maintains competitive per-
8

---

SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity
CSR170K
Math10K
Sparsity Method
#FL.Pss
Speedup
Accuracy
#FL.
Spd.
Acc.
#FL.
Spd.
Acc.
Uniform
60%
1.1x
80.3
Nonuniform
60%
1.4x
81.1
ILLMA3-8B
62.5
33.5
Uniform
46%
1.5x
80.2
+LooRA
100%
1.0x
87.1
100%
1.0x
80.6
Nonuniform
46%
1.6x
81.1
+ +SparseQLoRA
65%
1.2x
86.9
60%
1.3x
80.8
Uniform
37%
1.6x
79.5
Table Sparsel LoRA can be combined with existing PeFT approaches, such as QLoRA, to accelerate fine-tuning while maintaining memory savings.
Nonuniform
37%
1.8x
80.5
Table 10: On LLaMA3-8B with Math10K, we show that layerwise sensitivity enables better sparsity allocation. At fixed FLOPs budgets, our non-uniform sensitivity-awee approach outperforms uniform sparsity in speedup and accu- racy, achieving lossless performance up to 1.6x speedup,
#FLOPs
Runtime
Memory
Accuracy
Oracle SVD
81.4
0.05%
0.8%
30MB
81.1
TopK Selection Per
Accuracy (Math10K)
Table 8: SVD sparsity estimator delivers near- near-oracle accu- racy with negligible computation and memory overheads.
Metric
Channel
Head
X
Attention Norm
80.7
does not employ output token splitting, (2) our proposed approach with output token splitting, and (3) control con- figuration, where a random subset of tokens is selected for dense computation, matching the number of output tokens in our approach. The results are presented Table 9, which demonstrate that selectively preserving dense computation for output tokens consistently outperforms both the random selection and the baseline, highlighting the efficacy of our proposed method in improving computational efficiency without sacrificing performance.
QK Criterion
Attention Norm
79.6
X
1.2 Norm
79.8
x
Random
79.1
VO Criterion
L2Norm
81.4
Random
79.6
L2Norm 81.4
FFN Criterion
Wanda
81.3
Random
78.6
Table 11: Comparison of pruning criteria for QK, VO, and FFN modules under 90% uniform sparsity with token split- ting 5% step offset. All other components densely. For each module, the selected metric: attention norm (channel-wise) for QK, and L2 norm for VO and FFN, achieves the highest accuracy, validating our design choices.
CSR170K
Math10K
computed
Sparsity
#FT.
Spd.
Acc.
#FL.
Spd.
Acc.
All tokens
100%
1.xx
86.7
100%
1.0x
47.1
Random
102%
0.9x
86.6
129%
0.8x
70.9
Inputs only
102%
0.9x
86.9
129%
0.8x
81.1
Pruning Criterion. We explore pruning criteria beyond L2 norm, such as methods based on weights (Sun al., 2024). While these approaches show some promise, L2 remains the most effective method with simplicity. We compare L2 norm pruning with Wanda (Sun et al., 2024) and random pruning, all using the oracle setting for FFNs. Additionally, we conduct an ablation study on attention projections, pruning heads and channels (same or different per head), as shown in Table 11. Our proposed attention norm performs the best.
Table Datasets with more output tokens (e.g., Math 10K) highly sensitive to sparsity on those tokens, leading to significant accuracy drops. Output-awan splitting preserves accuracy while still achieving strong runtime improvements.
Uniform Sparsity without Sensitivity Analysis. We in- vestigate the impact of applying sensitivity-guii -ugiiddtyyggddddddde layerwise sparsity compared to uniform sparsity across all model lay- ers. Our experiments, conducted at various speedup targets, sparsity consistently
show that the sensitivity- approach- ratios are adapted to each layer's sensitivity c outperforms the uniform sparsity baseline, as demonstrated in Table 10. This result underscores the importance of tailoring sparsification strategies the unique sensitivity characteristics of each layer, rather than adopting one- size-fits-ll approach. Additionally, Figure 7 illustrates the varying sensitivity of sparsity across different layers, further validating the effectiveness of our layer-wise approach.
5. Conclusion
We introduced SparseLoRA to accelcrate fine-tuning through contextual sparsity using a lightweight, training-free SVD sparsity estimator. By dynamically selecting sparse weights for loss and gradient computation, SparseLoRA reduces computational cost by up 2.2x and delivers 1.6x speedup while preserving accuracy across diverse tasks.

---

SparseLoRA: Accelerating LLM Pine-Tuning with Contextual Sparsity
Acknowledgment. We would like thank Google-BAIR Commons, Google DeepMind, and POSCO HOLDINGS for their support of this research. We are also grateful to NVIDIA for providing GPU hardware.
Clark, C.,, Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2924-2936, 2019. 6
Impact Statement
This paper presents an approach for compute-cfficient fine- tuning of LLMs. There many potential societal con- sequences of our work, none of which we feel must be specifically highlighted here.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1803.05457, 2018.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M.. Jun, H., Kaiser, L.., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C.,, and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv: 2110.14166 2021.
References
Agarwal, A., Ramesh, S. K., Sengupta, A., and Chakraborty, T. Step-by-step unmasking for parameter-efficien fine-tuning of large language models. arXiv preprint arXiv:2408. 2024. 2
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. QLORA: Efficient finetuning of quantized LLMs. In Thirty-sevente Conference on Neural Information Pro- cessing Systems, 2023. 2,
Akhauri, Y., AbouElhamayed, A. F., Dotzel, J., Zhang, Z., Rush, A. M., Huda, S., and Abdelfattah, M. Shadowllm: Predictor-bass contextual sparsity for large language models. Conference Empirical Methods in Natural Language Processing, 2024. 2, 2,3,4 5,5
Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. In Internu- tional Conference on Machine Learning, volume 202, pp. 10323-10337. PMLR, 2023. 2
Alizadeh, K., Mirzadeh, S. I., Belenko, D.,, Khatamifard, S., Cho, M., Del Mundo, C. C., Rastegari, M., and Farajtabar, M. LLM in a flash: Efficient large language model infer- ence with limited memory. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pp. 1-562-12584 2024. 3.5
Gromov, A., Tirumala, K., Shapourian, H., Glorioso, P., and Roberts, D. A. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv: 5
2024.
Bambhaniya, A. R., Yazdanbakhsh, A., Subramanian, S., Kao, S.-C., Agrawal, S., Evei, U., and Krishna, T. Pro- gressive gradient flow for robust nim sparsity training in transformers. arXiv preprint arxiv: 2402.04744 2024. 6
Han, S., Pool, J., Tran, J., and Dally, W. Learning both weights and connections for efficient neural network. Advances in Neural Information Processing Systems (NIPS), pp. 1135-1143, 2015.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 34, pp. 1432-7439, 2020. 6
Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., and Dally, W. Eie: Efficient inference engine on compressed deep neural network. International Con- ference on Computer Architecture (ISCA), 2016a. 2
Chen, G., He, Y., Hu, Y., Yuan, K., and Yuan, B. Ce- lora fine-tuning for language models. arXiv preprint arXiv: 2502.01376 378, 2025.
lora: Computation-effica
Han, S., Mao, H., and Dally, W. J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. International Confer- ence on Learning Representations (ICLR), 201 2
Chen, M., Tworek. J., Jun, H., Yuan, Q., Pinto, H. P. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.0333
2021. 7,8
Hayou, S., Ghosh, N., and Yu, B. Lora+: Efficient low rank adaptation of large models. In International Conference on Machine Learning, 2024.7
Chen, Y., Qian, S., Tang, H., Lai, X., Liu, Z., Han, S., and
Jia, J. Longlora: Efficient fine-tuning of long-context large language models. In The International Conference on Learning Representations (ICLR), 2024.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang. L.,, and Chen, W. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, 2022. 1, 2,7
Chollet, F. On the measure of intelligence, 2019. URL https://arxiv. orgoorgg/ab.o. abs//9/1.110000 015477. 8
10

---

SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity
Hu, Z., Wang, L.,, Lan, Y., Xu, W., Lim, E.,, Bing, L., Xu, X., Poria, S., and Lee, R. K. LIm-adapters: An adapter family for parameter-efficient fine-tuning of large lan- guage models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Langugge Processing, pp. 5254-5276, 2023. 5,6 6
Ling, W. Yogatama, D., Dyer, C., and Blunsom, P. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Anmual Meeting of the Association for Computa- tional Linguistics, 2017.
Liu, J., Xia, C. S., Wang, Y., and Zhang, L. Is your code generated by chatGPT really correct? rigorous evalua- tion of large language models for code generation. In Thirty-seventh Conference on Neural Information Pro- cessing Systems, 2023a. URL net/Forum2id==qqvx11 OCu7. 7,8
Jaiswal, A., Yin, L., Zhang, Z., Liu, S., Zhao, J., Tian, Y., and Wang, Z. From galore to welore: How low-rank weights non con-uniformly emerge from low-rank gradients. arXiv preprint arviv: 2407.11239, 2024. 2
R., Roy, S., Amini, A., Kushman, N., and Hajishirzi, H. MAWPS: A math word problem repos- itory. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1 1152- 1157, 2016. 6
Liu, J., Ponnusamy, P., Cai, T., Guo, H., Kim, Y., and Athiwaratkun, B. Training-free activation sparsity in large language models. arXiv preprint arXiv: 2408.14690, 2024a. 2, 3, 4.5
Liu, S.-Y., Khadkevich, M., Fung, N. C., Sakr, C., Yang. C.- H. H., Wang, C.-Y., Muralidharan, S., Yin, H., Cheng, K.- T., Kautz, J.,, et al. Eora: Training-fre free compensation for compressed I1m with eigenspace low-rank approximation. arXiv preprint arXiv:2410.21271, 2024b. 5
Kopiczko, D. J., Blankevoort, T., and Asano, Y. M. Vera: Vector-based random matrix adaptation. In International Conference on Learning Representations, 2024. 2
Lee, J.-Y., Lee, D., Zhang, G., Tiwari, M., and Mirhoseini, A. Cats: Contextually- aware thresholding for sparsity in large language models. arXiv preprint arXiv: 2404.08763, 2024. 2, 3, 4
Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-C. F., Cheng, K.-T., and Chen, M.-H. DoRA: Weight- decomposed low-rank adaptation. In Proceedings of the 41st International Conference on Machine Learning, vol- ume 235, pp. 32100-32121, PMLR, 2024c. 1, 2, 6, 7, 16
LiM, M., Lin Y., Zhang Z., Cai, T., Li, X., Guo, J., Xie, E., Meng, C., Zhu, J.-Y., and Han, S. Svdquant: Absorb- ing outliers by low-rank components for 4-bit diffusion models. In International Conference on Learning Repre- sentations, 2025.
Liu, W., Qiu, Z., Feng, Y., Xiu, Y., Xue, Y., Yu, L., Feng, H., Liu, Z., Heo, J., Peng, S., Wen, Y., Black, M. J., Weller, A., and Schölkopf, B. Parameter-efficient orthogonal finetuning via butterfly factorization. In The Twelfth Inter- national Conference on Learning Representations, 2024d. 2
Li, W.-D., Hu, K., Larsen, C., Wu. Y., Alford, S., Woo, C., Dunn, S. M., Tang. H., Naim, M., Nguyen, D., Zheng, W.-L.,, Tavares, Z., Pu, Y., and Ellis, K. Combining induction and transduction for abstract reasoning, 2024. URL https: http://axxvvvrrreee
02272. 8
Liu, Z., Wang, J., Dao, T., Zhou, T., Yuan, B., Song, Z.. Shrivastava, A., Zhang, C., Tian, Y., Ré, C., and Chen, B. Deja vu: Contextual sparsity for efficient lims at inference time. In International Conference on Machine Learning. PMLR, 2023b. 1, 3, 14
Li, Y., Luo, F., Tan, C., Wang, M., Huang, S., Li, S., and Bai, J. Parameter-efficient sparsity for large language models fine-tuning. In Proceedings of the Thirty-Fii International Joint Conference on Artificial Intelligence, IJCAI-22, 2022. 3
Lu, Y., Agrawal, S., Subramanian, S., Rybakov, O., De Sa. C., and Yazdanbakhsh, A. STEP: Learning N:M struc- tured sparsity masks from scratch with precondition. In Proceedings of the 40th International Conference on Ma- chine Learning, volume 202, pp. 22812-22824. PMLR, 2023.
Li, Y., Yu, Y., Liang, C., He, P., Karampatziakis, N., Chen, W., and Zhao, T. Loftq: Lora-fine-tune -aan-in--t quantiza- tion for large language models. International Conference on Learning Representations, 2023a. 8
Li, Z., You, C., Bhojanapalli, S., Li, D., Rawat, A. S., Reddi, S. J., Ye, K., Chern, F., Yu, F. X., Guo, R., and Kumar, S. The lazy neuron phenomenon: On emergence of activa- tion sparsity in transformers. In International Conference on Learning Representations, 2023b.
Ma, D.,, Chen, L., Wang, P., Xu, H., H., Sun, L., Zhu, S., Fan, S., and Yu, K. language models. Anmual Meeting of the Association for Computational Linguistics, 2024. 3
training for large

---

SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity
Meng, F., Wang, Z.,, and Zhang, M. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv: 2404.02948, 2024.
and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1463-4473, 2019. 6
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381 2331,2399, 2018.
Shi, B., Gai, S., Darrell, T., and Wang, X. Toast: Transfer learning via attention steering. arXiv preprint arXiv: 2305.15542. 2023. 2
Song, C., Han, X., Zhang, Z., Hu, S., Shi, X., Li, K., Chen, C., Liu, Z.,, Li, G., Yang. T., and Sun, M. Prosparse: Intro- ducing and enhancing intrinsic activation sparsity within large language models. arXiv preprini arXiv: 2402.13516 2024a. 2
Mirzadeh, I., Alizadeh-Vahhe K., Mchta, S., Mundo, C. C. D., Tuzel, O., Samei, G., Rastegari, M., and Fara- jtabar, M. Relu strikes back: Exploiting activation spar- sity in large language models. International Conference on Learning Representations, 2023. 2
Song, Y., Mi, Z., Xie, H., and Chen, H. Powerinfer: Fast large language model serving with a consumer-grade gpu, 2023. 2,3
Mozaffari, M., Yazdanbakhsh, A., Zhang, Z., and Dehnavi, M. M. Slope: Double-pruned sparse plus lazy low- rank adapter pretraining of Ilms. arXiv preprint arXiv: 2405. 16325, 2024. 2,3
Song, Y., Xie, H., Zhang, Z.,, Wen, B., Ma, I.,, Mi, 7., and Chen, H. Turbo sparse: Achieving Ilm sota performance with minimal activated parameters. arXiv preprint arXiv: 2406. 0595505 2024b.
Nikdan, M., Tabesh, S., and Alistarh, D. Rosa: Accurate parameter-efficient fine-tuning via robust adaptation. In- ternational Conference on Machine Learning, 2024. 2,
Sun, M., Liu, Z., Bair, A., and Kolter, Z. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2024. 2, 4,
Pan, R., Liu, X., Diao, S., Pi, R., Zhang, J., Han, C.,,
and Zhang, T. Lisa: layerwise importance sampling for memory-efficiee large language model fine-tuning. Ad- vances Neural Information Processing Systems, 2024. 2
Team, G. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv: 2403. 08295, 2024a.
2
Patel, A., Bhattamishra, S., and Goyal, N. Are NLP models really able to solve simple math word problems? In Pro- ceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 1D80-2094: 2021. 6
Team, G. Gemma 2:
Improving open language models at a
practical size. arXiv preprint arXiv: 2408.00114 2024b.
2
Team, L. Llama 2:
Open foundation and fine-tuned chat models. arXiv preprini arXiv: 2307.09288, 2023. 2
Qiu, Z., Liu, W., Feng, H., Xue, Y., Feng, Y., Liu, Z., Zhang, D., Weller, A., and Schölkopf. B. Controlling text- to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems, 36:79320- 79362, 2023.
Team, I.. The llaaa herd of models. arXiv preprint arXiv: 2407.21783, 2024c.
Thangarasa, V., Gupta, A., Marshall, W., Li, T., Leong, K., DeCoste, D.,, Lie, S., and Saxena, S. SPDF: sparse pre- training and dense fine-tuning for large language models. In Uncertainty in Artificial Intelligence, volume 216, pp. 2134-2146. PMLR, 2023. 2,
Saab, K., Tu, T. Weng, W.-H., Tanno, R., Stutz, D., Wulczyn, E., Zhang, F., Strother, T., Park, C., Vedadi, E., et al. Capabilities of Gemini Models in Medicine. DrXiv:2404.18411 2024. 1
Sakaguchi, K., Bras, R. L.,, Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021. 6
Wang. A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Glue: A multi-task benchmark and anal- ysis platform for natural language understanding. arXiv preprint arXiv: (00.:/800.0444 004.07461, 2018. 6,
Sap, M., Rashkin, H., Chen, D., Le Bras, R., and Choi, Y. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing
Wang, S., Yu, L., and Li, J. Lora-ga: Low-rank adapta- tion with gradient approximation. arXiv preprint arXiv: 2407.05000, 2024. 2
12

---

SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity
Wang, Z. and Liang, J. Lora-pro: Are low-rank adapters properly optimized? arXiv preprint arXiv: 2407.18242, 2024. 2
Zhao, J., Zhang, Z., Chen, B., Wang. Z., Anandkumar, A., and Tian, Y. Gore Memory-efficien LLM training by gradient low-rank projection. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pp. 61121-61143. PMLR, 2024. 2, 15
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. SmoothQuant: Accurate and efficient quantization for large language models. Proceedings of the 40th International Conference on Machine Learning, 2023. 4
Zheng, H., Bai, X., Liu, X., Mao, Z.,, Chen, B., Lai, F,, and Prakash, A. Learn to be efficient: Build structured sparsity in large language models. In The Thirty-eights Annual Conference on Neural Information Processing Systems, 2024a. 2,5
Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., Lin, Q., and Jiang, D. WizardIm: Empowering large pre-trained language models to follow complex in- structions. In The Twelfth International Conference on Learning Representations, 2024. 6,
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang. Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging Ilm-as-a-judge with mt-bench and chatbot arena. Ad- vances Neural Information Processing Systems, 36: 46595-46623, 2023. 7,8
Xue, Z.. Song, Y., Mi, Z., Chen, L., Xia, Y., and Chen, H. Powerinfer-2: Fast large language model inference on smartphone, arXiv preprint arXiv: 2406.0666 -426,, 2024. 2
Zheng, T., Zhang, G., Shen, T., Liu, X., Lin, B. Y., Fu, J., Chen, W. and Yue, X. Opencodeinterpreter: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402.14658 2024b. 7,8
Yang, X., Leng, J., Guo, G., Zhao, J., Nakada, R., Zhang, L., Yao, H., and Chen, B. S$ {2] Efficient, scalable and generalizable LLM fine-tuning by structured spar- sity. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.7 2
Zhou, X., Zhang, W., Chen, Z.,, Diao, S., and Zhang, T. Effi- cient neural network training via forward and backward propagation sparsification. Advances in Neural Informa- tion Processing Systems, 34, 2021.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Conference of the Association for Computational Linguistics, pp. 4791-4800, 2019. 6
Zhang, J., Wang. J., Li, H., Shou, L., Chen, K., You, Y., Xie, G., Gong, X., and Zhou, K. Train small, infer large: Memory- eeffory-fficieent lora training for large language models. International Conference on Leurning Representations, 2025a. 2
Zhang, R., Han, J., Liu, C., Zhou, A., Lu, P., Qiao, Y., Li, H., and Gao, P. LLaMA-Adapter: Efficient Fine- tuning of Language Models with Zero-init Attention. In International Conference on Learning Representations (ICLR), 2024a.
Zhang, Z. Jaiswal, A., Yin, L., Liu, S., Zhao, J., Tian, Y., and Wang, Z. Q-galore: Quantized galore with int4 projection and layer-adaptive low-rank gradients. arXiv preprint arXiv: 2407.08296 2024b. 2
Zhang, Z., Song, Y., Yu, G., Han, X., Lin, Y., Xiao, C., Song. C., Liu, Z., Mi, Z., and Sun, M. Relu² wins: Discovering efficient activation functions for sparse llms. arXiv preprint arXiv: 2402.03804, 2024c. 2
Zhang, Z., Liu, Z., Tian, Y. Khaitan, H., Wang, Z., and Li, S. R-sparse: Rank-aware activation sparsity for efficient LLM inference. In International Conference on Learning Representations, 2025b. 2,5
13

---

SparseLoRA: Acceleratingg LLM Pine-Tuning with Contextual Sparsity
A. Appendix
A.1. Analysis Pruning at Attention Head Level in Inference and Fine-tuning
Liu et al. (2023b) found that some attention heads show uniform attention scores across previous tokens during the auto- regressive generation. As illustrated in Figure 9a, at test time the top head is a uniform "token mixing" head, while the middle and bottom heads "heavy hitter" heads. Since uniform heads don't capture important interactions, keeping only the heavy hitter heads preserves prediction quality. However, this behavior changes during fine-tuning: Figure 9b shows that attention heads may exhibit different patterns depending on the token-what might be token-mixing head for one token could be critical for another.
Head
Head
Head
60-
60-
Head 21
2
3
2
(a) last row in Figure 9b's attention scores visualization
(b) attention scores of 3 different heads for a sample
Figure 9: Attention scores from three different heads are visualized for the last 9 tokens of a sample. Figure 9a (left) corresponds to the last row of Figure 9b (right), simulating the auto-regressive generation the final token during inference. Darker colors indicate higher attention scores.
A.2. Fine-Tuning Details
Table 12: Training Hyperparameters Across Datasets. All experiments use LoRA with dropout 0, rank 32, and 64.
Dataset
Seq. Len
Batch Size Epochs
LR
Scheduler Warmup Ratio
CSR170K
512
8
1
3e-4
cosine
0.04
Math10K
512
8
3
3e-4
cosine
0.04
GLUE (COLA, STS-B, RTE, SST2, QNLI, MNLI, QQP)
128
8
3
5e-5
cosine
0.04
GLUE (MRPC, WNLI)
128
8
5
5e-5
cosine
0.04
CodeFeedback
1024
6
1
2e-5
cosine
0.04
WizardLM
2048
2
1
2e-5
cosine
0.04
A.3. Detailed Sparsity Configuration
Table 13: Sparsity Configuration Across Models and Datasets
Model
Dataset
FFN
QKVO
Step #FLOPs Speedup
Layers
Sparsity (%)
Layers
Sparsity (%)
CSR170K
L17-L29 L20, L24
0.65
1.3x
L13-L29
90
5%
LLaMA2-7B
Math10k
L13-L29 / L20, L24
60
0.73
1.2x
CodeFeedback
L3-1.30
99
L14-1.199121-123112555111
25
5%
0.73
1.2x
CSR170K
20
10%
0.61
LLaMA2-13B
L13-L.36
L.17LL36
1.3x
Math
25
5%
0.70
CSR170K
L17-L30
97
L17-L191121-L23313333
20
0.65
1.3x
5%
LLaMA3-8B
Math10k
L3-L30
99
75
0.46
1.6x
GLUE
L17-1.30/131
95/50
L17-1.29
75
5%
0.61
1.3x
CodeFeedback
0.66
1.3x
LLMMA311-B
L3-L30
99
L14-L19.121--22311122
40
5%
WizardLM
0.53
1.5x
14

---

SparseLoRA: Acceleratinng LLM Fine-Tuning with Contextual Sparsity
A.4. GaLore vs. SparseLoRA
We compare GaLore (Zhao al., 2024) and SparseLoRA on CSRI70K and Math GaLore training takes slightly more VRAM than LoRA and requires A100 GPUs due VRAM limitations of A6000 under Distributed Data Parallel (DDP). Runtime is normalized to LoRA, in main submission. Result are shown in Table 14. GaLore achieves memory efficiency by projecting the full gradient matrices into low-rank subspace. This projection is periodically updated during training via an online Singular Value Decomposition (SVD) of the gradients. While this online SVD allows GaLore to adapt the subspace and maintain performance similar to LoRA, it incurs a significant 1.58 x training overhead compared to LoRA. The amortized time for GaLore, which accounts for these periodic projection updates via online SVD, substantially slows down the fine-tuning process by 1..72x. Thus, while GaLore prioritizes memory-ef -fffiiett fine-tuning by reducing optimizer states and gradient memory, this comes considerable cost computational efficiency due the demanding SVD operations. In contrast, SparseLoRA is designed to accelerate fine-tuning while delivering near-lossless performance.
CSR170K
Model
Runtime
Average
BoolQ
PIQA
SIQA
HellaSwag
WinoG
ARC-E
ARC-C
OBQA
LoRA
1.00
87.1
74.5
89.6
82.8
95.3
88.4
93.1
84.4
88.8
Gal
1.58 [13.72]
84.1
71.2
87.1
79.6
92.0
85.0
80.5
SparseLoRA
0.78
87.0
74.7
89.5
82.8
95.3
88.8
92.9
83.6
88.3
Math 10K
Model
Runtime
Average
GSM8K
SVAMP
MAWPS
LoRA
1.00
80.0
71.1
79.5
89.5
GaLore
1.58 [13.72]
78.7
68.1
77.9
SparseLoRA
0.82
80.0
70.9
79.4
89.9
Table 14: LLaMA 3-8B CSRI70K and Math10K GaLore's amortised cost (including periodic online-SVD updates) is
shown in brackets.
A.5. Impacts of Learning Rates Sweep
We perform learning rate sweeps eliminate selection bias from hyperparameter choices. Specifically, evaluate both LoRA and Spa variants using LLaMA3-8B the Math1 OK and CSR170K datasets. Table 15 shows that the performance gap between the best-performing LoRA and SparseL A is just 0.2% on Math10K and 0.3% on CSR CSR170K, validating the robustness of our approach.
Dataset
Learning Rate
LoRA
SparseLoRA
3,0 x 10
78.3
78.8
5,0 x 10-5
78.6
79,3
Math10K
9.5 x 10-5
79.6
79.8
3.0 x 10-4
80.0
80.0
5.0 x 10
80.2
79.6
9.5 x
78.1
77.3
3.0 x 10
85.7
85.6
5.0 x 10
86.7
86.5
CSR170K
9.5 x
87.7
87.4
3.0 x 10
87.1
87.1
Table 15: Learning-rate sweep LLaMA 3-8B on Math10K and CSKI Best accuracy per dataset is in bold.
A.6. LoRA on Different Projections
We primarily apply sparsity to accelerate the main branch of the model, while keeping the LoRA branches dense. To assess generality, we also conduct additional experiments on Math applying LoRA to the Q, K, V, up, and down
15

---

SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity
projections- the DoRA setup (Liu al., 2024c) beyond the configurations explored in the main paper. Results in Table 16 indicate that the benefits of SparseL.RRA extend beyond just QKVO, demonstrating its broader applicability.
Projection Set
LoRA
SparseLoRA
A
QKVO
79.9
80.0
+0.1
QKVUD
80.3
80.9
+0.6
QKVOGUD
80.5
80.7
+0.2
Table 16: Mean accuracy of LoRA versus SparseLoRA LLaMA 3-8B for different projection configurations.
A.7. Iso-FLOP Comparison
practical question is how methods behave when constrained by a fixed FLOP budget, rather than fixed number of training steps. In production, practitioners often allocate set amount of compute; a method that extracts more accuracy per FLOP therefore more valuable. To address this, we perform an Iso-FLOP study on LLaMA3-8B using Math10K and CSR170K, One full epoch is treated as 100% of the available FLOOPs, and sweep down to 5%. Figure 10 shows consistent gains across all tested budgets on both datasets. These results confirm that, for the same computational cost, SparseLoRA produces better-performing models than standard LoRA.
ISO-FLOPs Comparison (Math10K)
ISO-FLOPs Comparison (CSR170K)
80
88
78
Accuracy
77
85
75
84
74
20
40
60
80
100
20
40
60
80
100
FLOPs (%6)
LoRA
SparseLoRA
FLOPs (96)
Figure 10: Iso-FLOPs comparison on LLaMA3-8B 3B using Mathl0K and CSR170K. As the FLOPs budget decreases,
SparseLoRA is able to better retain task performance compared the LoRA counterpart by larger margin.
16


Reviews:
[{"review_id": "X1ASk8GmVH", "reviewer": "Reviewer_as16", "summary": "The paper proposes a framework for accelerating the fine-tuning large language models by structured pruning of pretrained weight matrices, and using dense and trainable LoRA adapters. The core proposed idea is to estimate the importance of each channel in a pretrained weight matrix, prune the unimportant channels by slicing the weights of that channel, and then using the sliced weights for computation. The LoRA computation is done following the original LoRA paper, with dense matrices.\n\nThe porposed method uses different pruning strategies for different components:\n- FFN: Uses the L2 norm of the activations of the Gate projection after SiLU activation along the batch and sequence dimensions, and retains the columns in the Gate and Up weight matrices with the highest L2 norm (assuming columns represent the weights if a single channel). The corresponsing rows of the Down weight matrix are also pruned.\n- Value and Output Projection in the attention layer: Uses the L2 norm of the activations of the attention heads after the Value projection along the batch and sequence dimensions, and retains the columns in the Value matrix with the highest L2 norm. The corresponding rows of the Output matrix are also pruned.\n- Query and Key: Calculates the L2 norm of the Key and Value projections along the batch and sequence dimensions, and uses their dot product as the pruning criterion. The columns in the Key and Query matrices with the highest dot product are retained.\n\nThe paper proposes several tricks to reduce computational cost and improve performance:\n\n- T1: The paper also proposes an SVD based estimator for the sparsity. Instead of using the projections from full pretrained weight matrices in order to calculate the L2 norms (the sparsity metric), the operations are performed using the forst $k$ ranks of their respective SVD projections, which results in reduced computational cost for the calculation of the sparsity metric.\n\n- T2: The sparsity ratio is selected based on layer sensitivity analysis, with higher sparsity for deeper layers. To determine the sparsity ratio, the authors perform a sensitivity analysis by greedily increasing the sparsity ratio of each layer while keeping the other layers dense, and measuring the performance on a subset of the Commonsense Reasoning tasks.\n\n- T3: The pruning (sparsity) is only applied to the context tokens, and dense pretrained weight matrices are retained for the output tokens. \n\n- T4: Early iterations in fine-tuning are run in dense mode (upto 20% of the total finetuning iterations) and sparsity is imposed in the later iterations. \n\n- T5: Uses sequence averaging in the SVD estimator for the FFN block to reduce the computational cost. *However, the authors do not define what they mean by the term, and do not provide the exact implementation details. See Point 5 in the Strengths and Weaknesses section.*\n\n## Update after rebuttal\n\nI have gone through the authors' rebuttal and their response to other reviews. I do not mind raining my score to 3, provided the writing of the paper is improved based on my comments in the Other Strengths And Weaknesses section.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's comments.\n\n> L2 norm vs. Random pruning in Self-Attention blocks\n\nL2 norm shows clear benefits for FFN blocks, while its gains over Random pruning in Self-Attention blocks are modest. We use a unified L2-based criterion to avoid over-engineering and ensure broad applicability.\n\n> Channel vs. Head Pruning:\n\nAlthough the performance gap is small (80.2 vs. 80.1), channel pruning offers greater flexibility. Unlike head pruning—which enforces equal channel counts—channel pruning enables nuanced selection of important channels, allowing precise control over computational cost.\n\n>SVD Rank and Per-Layer Sparsity Ratios:\n\nFor all models, we use an SVD rank of 8, which incurs marginal overhead (see Table 3). Our per-layer sparsity ratios are determined via sensitivity analysis per model (see Figure 7). For the detailed configurations, please check \"Model Sparsity Information table\" in our response to Reviewer pTrd. Similar settings, with slight adjustments for latency, are applied across models.\n\n> Model Specification and Math10K Accuracies\n\nAll experiments in Section 4.3 use LLaMA3-8B. The Math10K results for LLaMA3-8B in Table 2 have 79.8% accuracy on average. This matches the result in line 2 of Table 3, line 2 of Table 4, and the “Non-uniform 1.4x” configuration in Table 5. In Table 6, higher accuracies result from applying sparsity only to specific modules rather than across the entire model.\n\n> Context-Output Aware Sparsity\n\nLines 263–264 describe our strategy as “selectively preserving dense computation for output tokens while applying sparsity to the context,” meaning only a portion of output tokens are processed densely, as reflected in Figure 8. We will revise the text to avoid potential ambiguity.\n\n> Table 5 Speedup Columns\n\nThe “1.4x Speedup” and “1.6x Speedup” columns indicate runtime targets by adjusting sparsity ratios. Both uniform and non-uniform configurations are calibrated to these targets for fair comparison. The 1.4x (1/0.71) non-uniform setting, with 79.8% accuracy on Math10K, directly corresponds to the SparseLoRA LLaMA3-8B result in Table 2; the 1.6x setting explores a more aggressive sparsity trade-off.\n\n> \"Sequence averaging\" in the SVD estimator\n\n“Sequence averaging” reduces sparsity estimation cost by averaging activations over the sequence dimension for SVD inputs to compute a single channel score. This is particularly effective for FFN layers, where token-level differences are less critical, and it minimizes overhead while maintaining performance (see Table 3).\n\n> FLOPs and Runtime Optimizations in Tables 1-2.\n\nThe reductions reported in Tables 1 and 2 stem from applying contextual channel sparsity guided by our SVD estimator to expensive linear layers. Sequence averaging further minimizes the overhead of sparsity estimation, making the cost of identifying sparse channels negligible.\n\n> Trained to Convergence and Uniformity of Sparsity Ratios\n\nNo additional training was performed during the sensitivity analysis. As shown in Figure 7, we apply sparsity to the pretrained LLaMA2-7B model at individual layers (with others kept dense) and evaluate its performance on CSR170K. This analysis identifies robust sparsity patterns that are then uniformly applied across all models and tasks.\n\n> Comparison with Ma et al. (2024)\n\nMa et al.’s method differs from ours in many ways: it uses structured pruning with full fine-tuning, resulting in full-model memory usage. In contrast, SparseLoRA integrates structured pruning with LoRA adapters, reducing both computation and memory. Moreover, their importance metrics (Wanda, MaxiP) involve element-wise operations that incur high overhead (e.g., on A6000 GPUs), nearly canceling speedup gains when we experimented Wanda for Table 6. Our SVD-based sparsity estimator relies on efficient matrix multiplications with low-rank weights, ensuring minimal overhead across GPUs.\n\n> Performance drop not negligible\n\nSparseLoRA offers a flexible speed–accuracy trade-off. For applications where maximum accuracy is crucial, LoRA remains viable; for efficiency-critical scenarios, SparseLoRA delivers substantial speedups with minimal accuracy loss. Here we provide an updated, more conservative sparsity configuration for LLaMA2-13B and LLaMA3-8B that slightly increase FLOPs (0.59 → 0.62 for LLaMA2-13B and 0.60 → 0.63 for LLaMA3-8B) but yield nearly identical runtime (0.74 → 0.76 for LLaMA2-13B and 0.71 for LLaMA3-8B) and match or improve accuracy:\n\n| |#FLOPs|Runtime|Avg|GSM8K|SVAMP|MAWPS|\n|---|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|\n|LLaMA2-13B+LoRA|1|1|63.3%|50.7%|59.0%|80.4%|\n|LLaMA2-13B+SparseLoRA|0.62|0.76|63.5%|49.3%|58.7%|82.6%|\n|LLaMA3-8B+LoRA|1.00|1.00|80.0%|71.1%|79.5%|89.5%|\n|LLaMA3-8B+SparseLoRA|0.63|0.71|80.0%|70.9%|79.4%|89.9%|\n\nThese results show that, with proper configuration, SparseLoRA can achieve comparable or better accuracy than LoRA while delivering strong runtime benefits.", "claims_and_evidence": "1. **Sparsity metric**: The use of L2 norm is well justified, but I would like to mention some nuances to the results of the ablation studies in Table 6 which compares L2 norm with Random pruning and Wanda: In case of the Self-Attention block, L2 norm does not show a significant improvement over Random pruning. However, in the FFN block, L2 norm shows an improvement over Random pruning and maintins simplicity. The authors should mention this in the paper.\n\n2. **Channel pruning vs. Head pruning**: Section 3.1.2, mentions that head level pruning strategy could be problematic. However, Table 6 shows that the accuracy with Head pruning is 80.1 and accuracy due to channel pruning is 80.2, which is not a significant difference. Hence, the choice of channel pruning over head pruning proposed in the paper is not very well justified.", "methods_and_evaluation_criteria": "The paper evaluates the proposed SparseLoRA on Commonsense170k and Math10k benchmarks, which have been used in prior works [Hu et al. (2023), Liu et al. (2024)]. The performance on a task is measured using the accuracy of the final answer, which is standard in prior PEFT works. The computaitonal efficiency is measured using the wall clock time and the number of FLOPs relative to LoRA fine-tuning, which reflects the goal of the paper.\n\n---\n\n## References\n[1] Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, Roy Lee, \"LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models\", EMNLP 23\n\n[2] Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Min-Hung Chen, \"DoRA: Weight-Decomposed Low-Rank Adaptation\", ICML 24", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The design of the experiments and results reflect the goal if the paper. However, the ablation studies need to be described in more detail. Specifically, when the checking the effect of a particular setting, what are the other settings that are kept constant? For example, Table 5 shows the effect of uniform vs. layer specific sparsity. What are the other settings that are kept constant? The paper should provide a detailed description of the ablation studies.", "relation_to_broader_scientific_literature": "1. Using sparisity and structured pruning during finetuning has been explored before by Ma et al. (2024), which uses structured pruning along with full fine-tuning of unpruned model parameters. Ma et al. (2024) have also experimented with L2 norm as a sparsity metric, along with other metrics. Their findings show that L2 norm performs slightly worse than other metrics. \n\n2. This paper extends the idea of pruning+finetuning to use adapters (LoRA) instead of full fine-tuning, and uses SVD decomposition of the pretrained weights to obtain the sparsity metrics for efficiency. This paper also proposes channel pruning instead of head pruning in the self-attention block. However, as mentioned in the Claims and Evidence section (Point 2), the contribution of channel pruning over head pruning is not very significant.\n\n3. In terms of the efficiency metrics, while the proposed method can achieve a reduction in FLOPs to ~60-80% that or LoRA, the accuracy is not guaranteed to stay within a small range, suggesting that there could be limitations to the practical applicability of the proposed method. For example, on the Math10k benchmark (Table 2), with LLaMA2-7B, the performance drop on SVAMP is aaround 5% compared to LoRA. With Llama2-13B, the performance drop on GSM8k and SVAMP is around 2%. With Llama3-8B, the drop on GSM8k is ~2%. This drop in performance is not very significant, but it is not negligible either. LoRA, on the other hand, consistently achieves higher accuracy, while being very practical in terms of real-world applications.\n\nOverall, the contribution of the paper is fairly incremental and given the comparison with LoRA mentioned above, the practical applicability of the proposed method could be limited.\n\n---\n\n## References\n[1] Da Ma, Lu Chen, Pengyu Wang, Hongshen Xu, Hanqi Li, Liangtai Sun, Su Zhu, Shuai Fan, Kai Yu, \"Sparsity-Accelerated Training for Large Language Models\", ACL Findings 2024", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "## Strengths\n- Achieves 1.7x reduction in computational cost and 1.4x reduction in wall clock time over LoRA, while achieving almost the same accuracy in most cases.\n- The paper proposes a novel method for estimating the sparsity metric using SVD decomposition of the pretrained weights, which reduces the computational cost of the sparsity metric calculation.\n\n## Missing hyperparameter details\n1. The expeirmental details for the SVD rank and the exact sparsity ratios per layer are missing.\n\n## Writing and Clarity\nThe writing is unclear in many places: \n\n2. In section 4.3 (Analysis), none of the experiments and tables mention the model being used, or how the accruracy on Math10k is calculated (for e.g., is it the mean of the accuracies of individual datasets?). Furthermore, the accuracies on Math10k in the Tables 3, 4, 5, and 6 are much higher than the accuracies reported in Table 2. What is the reason for this discrepancy?\n\n3. Figure 8 seems to contradict the text on line 245, which claims that sparsity is only applied to the context tokens, and output tokens are processed in a dense manner. However, the figure shows that only a small portion of the output tokens use dense weights.\n\n4. In Table 5, what do columns 1.4x Speedup and 1.6x speedup denote, and how do they relate to the main results in Table 2? Table 5 also does not mention what benchmarks the experiments are performed with (from the accuracy scores, it looks to be Math10k)\n\n5. The paper is unclear about the definition and implementation of the sequence averaging in the SVD estimator.\n\n6. The paper is very fragmented in terms of what optimizations lead to the reported FLOPs and Runtime decrease. Lines 379, Section 4.3, mentions that sequence averaging incurs leads to significant speedup, yet it has not been defined or explained in the paper. A coherent explanation of exactly which optimizations lead to the decrease in FLOPs and Runtime reported in Tables 1 and 2 should be provided.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. To determine the sparsity ratio for every layer, the authors perform a sensitivity analysis by greedily increasing the sparsity ratio of each layer while keeping the other layers dense. Is the model trained to convergence for every setting?\n\n2. Related to Q1, if the model is trained for every combination, it will add to the computaitonal cost. Section 3.3 studies this with Llama2-7B and a subset of Commonsense Reasoning. Is the same sparsity ratio used across all the models and tasks? If the same ratio is applicable to all models and tasks, then the sensitivity analysis could be done once and the same ratio could be used across all models and tasks. If not, then the sensitivity analysis needs to be done for each model and task, which could be computationally expensive, adding to the overall computational cost of the method.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OhTtbYJVkB", "reviewer": "Reviewer_NtQC", "summary": "Previous parameter-efficient fine-tuning (PEFT) methods, such as LoRA and its variants, have primarily focused on memory efficiency and lightweight storage. However, these approaches do not necessarily lead to faster fine-tuning. This paper introduces SparseLoRA, a novel technique that accelerates fine-tuning by selecting a sparse subset of the base model’s weights, enabling more efficient loss and gradient computation while fully preserving LoRA’s structure.\n\nSparseLoRA achieves this by decomposing the original weight matrix and selectively activating channels based on a Singular Value Decomposition (SVD) sparsity estimator. This estimator adaptively determines sparsity using certain norms of the batched input, allowing the method to remain dynamic and data-aware. Unlike prior works that apply sparsity only during inference (typically with a batch size of 1), SparseLoRA incorporates sparsity directly into the training process.\n\nDuring training, LoRA initially operates in its standard dense form for the first few iterations. Sparse fine-tuning is then gradually introduced, reducing the computational overhead while maintaining performance. Experimental results demonstrate that SparseLoRA achieves faster training times with minimal accuracy degradation across multiple benchmark tasks, making it a promising alternative for efficient fine-tuning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s comments:\n\n> Fixed learning rate instead of hyperparameter sweeps. \n\nFor concision we only include mean performance. Here V1 refers to results in Table 1-2 of the paper and V2 is the new \"conservative\" config in our response to Reviewer as16\n### LLaMA3-8B LR Sweep (Math10K)\n|LearningRate|LoRA|SparseLoRA-V1|SparseLoRA-V2|  \n|-|-|-|-|  \n|3.00E-05|78.3|77.7|78.8|  \n|5.00E-05|78.6|78.6|79.3|  \n|9.49E-05|79.6|79.2|79.8|\n|3.00E-04|80.0|**79.8**|**80.0**|  \n|5.00E-04|**80.2**|79.3|79.6|  \n|9.49E-04|78.1|77.1|77.3|\n\n### LLaMA3-8B LR Sweep (Commonsense 170K)\n|LearningRate|LoRA|SparseLoRA-V1|SparseLoRA-V2|\n|-|-|-|-|\n|3.00E-05|85.7|84.8|85.6|\n|5.00E-05|86.7|85.9|86.5|\n|9.49E-05|**87.7**|**86.8**|**87.4**|\n|3.00E-04|87.1|86.3|87.1|\n\nWe conduct two learning rate sweeps with Llama3-8B on Math10K and CSR170K for LoRA and SparseLoRA. SparseLoRA shows minimal performance degradation compared to LoRA, confirming our preformance claims. In the conservative setting, the performance gap between the optimal LoRA and SparseLoRA is 0.2% on Math10K and 0.3% on CSR170K.\n> SparseLoRA is only evaluated on QKVO projections, but LoRA can be applied to different subsets of projections.\n\nWe focus on applying sparsity to speed up the main branch of the model, leaving the LoRA branches unchanged. However, we also provide additional experiments applying LoRA to Q, K, V, up, and down projections, following DoRA, in addition to the settings in our paper.\n### LLaMA3-8B Performance on Math10K for different projections\n|Method|Config|Mean| \n|-|-|-| \n|LoRA|QKVO|79.9| \n|SparseLoRA|QKVO|80.0| \n|LoRA|QKVUD|80.3| \n|SparseLoRA|QKVUD|80.9| \n|LoRA|QKVOGUD|80.5| \n|SparseLoRA|QKVOGUD|80.7|\n> Provide iso-FLOP comparisons showing performance vs. computational budget\n\nFollowing the reviewer’s recommendation, we conduct an iso-flop comparison on Llama3-8B using Math10K and CSR170K. SparseLoRA outperforms LoRA, with a 3% improvement at 5% FLOP budget on Math10K. Experiments run using 1 epoch as 100% FLOPs.\n### Iso-FLOP Comparison on Math10K with LLaMA3-8B\n|Setting|FLOP(%)|Mean|Diff| \n|-|-|-|-| \n|**LoRA (Baseline)**|100|79.6|--| \n|**LoRA**|63|78.7|--| \n|**SparseLoRA**|63|79.2|+0.55| \n|**LoRA**|30|76.7|--| \n|**SparseLoRA**|30|78.3|+1.55| \n|**LoRA**|10|74.8|--| \n|**SparseLoRA**|10|76.2|+1.63| \n|**LoRA**|5|72.5|--| \n|**SparseLoRA**|5|75.8|+3.30| \n### Iso-FLOP Comparison on Commonsense 170K with LLaMA3-8B\n|Setting|FLOP(%)|Mean|Diff|\n|-|-|-|-|\n|**LoRA (Baseline)**|100|87.1|--|\n|**LoRA**|64|86.8|--|\n|**SparseLoRA**|64|87.0|+0.28|\n|**LoRA**|30|86.0|--|\n|**SparseLoRA**|30|86.2|+0.19|\n|**LoRA**|10|83.9|--|\n|**SparseLoRA**|10|84.7|+0.83|\n|**LoRA**|5|82.0|--|\n|**SparseLoRA**|5|83.5|+1.47|\n\n> Lack of specifics like number of shots per task etc.\n\nTasks are evaluated with the mean over 5 shots. The SVD sparsity estimator is used during training to skip parts of the base-branch computation. The SVD computation is offline, but during training, marginal FLOPs are spent on the sparsity metric, accounting for 0.0534% of LoRA FLOPs and 0.8% runtime overhead, as shown in Table 3.\n\n> Details on SVD Predictor\n\nWe compute the SVD of weight matrices to create our sparsity estimator (Algorithm 1, lines 251-254) once per model. For the self-attention block, we SVD q, k, and v, and for the FFN, we compute up and gate projections, as described in Section 3.1. SVD is performed with rank 8 for all models. The predictors incur minimal runtime during training (Table 3) and negligible memory overhead (see below in Mb).\n- Llama2-7B: 26.8\n- Llama2-13B: 36.9\n- Llama3-8B: 30.0\n\n> Motivation for computationally efficient PEFT\n\nSparseLoRA achieves near-lossless performance while reducing computation. Even if PEFT tasks are fast, practitioners favor any free acceleration. For example, many opt for BF16 training over FP32 because, despite its slightly lower precision, the efficiency gains are substantial and the performance drop is negligible. Faster training means more rapid experimentation and ability to scale to larger models or more complex tasks—all of which are significant benefits in practice. SparseLoRA is a step toward this goal and can inspire future improvements in computationally efficient, lossless PEFT methods.\n\n> DoRA Settings\n\nIn this work, we use a rank of 32 for LoRA fine-tuning; DoRA has shown a smaller performance gap with this setting (Figure 5 in their work). S2FT reproduced similar results on Llama3-8B (arithmetic).  Unlike DoRA—which prioritizes stability and performance at the expense of runtime—SparseLoRA enhances efficiency, complementing DoRA for practitioners.\n\n> Figure 1\n\nWe visualize the results directly from Table 1, specifically highlighting the normalized runtime and average performance on CSR170K for Llama2-7B.\n\n> \"less than 0.5% overhead to finetuning\" by SVD\n\nWe meant \"0.8%\" runtime overhead introduced by SVD estimator. Additional metrics in Table 3.", "claims_and_evidence": "While the claims in the paper are generally well-supported, some unconvincing weaknesses remain. Below, I outline key concerns that, if addressed, would strengthen the paper’s experimental robustness:\n\n1. The experiments use a fixed learning rate, but it is standard practice to perform a hyperparameter sweep and report the best-performing configuration. To ensure fair comparisons, Tables 1 and 2 should reflect results from a learning rate sweep, rather than relying on a single fixed value. Otherwise, there is a risk that SparseLoRA benefits simply from better tuning rather than intrinsic efficiency.\n\n2. The paper evaluates SparseLoRA only on QKVO projections. However, LoRA can be applied to different subsets of projections, and it is unclear whether SparseLoRA remains effective across different configurations. A more thorough evaluation should test SparseLoRA on different subsets of trainable LoRA projections to confirm that its benefits generalize beyond QKVO.\n\n3. A crucial missing experiment is an Iso-FLOP comparison—i.e., comparing models trained with the same computational budget. In real-world applications, practitioners often have a fixed FLOP budget rather than a fixed number of iterations. Therefore, it is important to test whether LoRA trained with the same FLOP budget as SparseLoRA produces weaker models.\n\nIn Table 1 (LLaMA3-8B results), SparseLoRA trains at 0.62x FLOPs of full LoRA. The paper should compare whether training LoRA for the same 0.62x FLOPs produces weaker performance than SparseLoRA. The paper should include a graph of benchmark accuracy vs. wall-clock time (or training FLOPs). This would help practitioners determine when SparseLoRA is beneficial and when standard LoRA suffices.\n\nWhile SparseLoRA presents a promising approach for efficient fine-tuning, the paper lacks critical experimental validations. A more rigorous study should incorporate learning rate sweeps to avoid selection bias, evaluations across different LoRA projection sets, and, most importantly, Iso-FLOP comparisons and efficiency trade-offs, as practitioners need to know whether SparseLoRA is truly advantageous under fixed compute constraints. Without these, the claims of SparseLoRA’s efficiency remain incomplete and may not fully guide real-world adoption.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "The experiments seem sound. However, given the empirical nature of the paper, necessary experimental details are missing, especially those one would usually find in the supplementary material. For instance, I cannot find more information on Table 1 when I want to check how many shots were used for each task, or the absolute runtime in terms of wall-clock time, or whether FLOPs includes SVD or not.", "relation_to_broader_scientific_literature": "- Computationally efficient finetuning methods are not well developed. This work attempts to achieve this without sacrificing accuracy. The motivation is extremely practical and the sparse selection mechanism is sensible. However, I am not particularly well-versed in this sub-area, so I am not entirely sure how it fares to prior work or if this is truly the first work to consider computationally efficient PEFT methods. \n- The SVD estimator based on two kinds of norm-based criteria can be used elsewhere that require input-adaptive sparsity.", "essential_references_not_discussed": "Not that I know of.", "other_strengths_and_weaknesses": "**Strength:**\n- Computationally efficient finetuning methods are not well developed. This work attempts to achieve this without sacrificing accuracy. The motivation is extremely practical and the sparse selection mechanism is sensible. However, I am not particularly well-versed in this sub-area, so I am not entirely sure how it fares to prior work or if this is truly the first work to consider computationally efficient PEFT methods.\n- Adaptive sparsity that takes into account training-time input batches probably requires more care than dealing with just one input as in inference-time.\n\nWeaknesses listed \"Claims And Evidence\" and \"Questions For Authors\".", "other_comments_or_suggestions": "- It would be more helpful if Figure 1 is more descriptive, e.g., which models, tasks, and datasets were used for the particular experiment.\n- In lines 43-44, when claiming \"adding less than 0.5% overhead to finetuning\", does this mean computationally (if so, in terms of FLOPs or wall-clock time) or memory (VRAM)?\n- In general, the paper is lacking in detail, e.g., experimental settings, captions in figures. The paper will be stronger with these details.", "questions_for_authors": "Listed in order of decreasing importance.\n1. **Details on SVD.**\nI am curious to know the details on the computation of SVD that is done offline, such as time, memory, and etc. Furthermore, when finetuning with SparseLoRA, how much memory overhead is included by loading the SVD decomposition alongside the model weights? Include details about GPU memory in Table 1 seems important even if the overhead is minimal.\n2. **Motivation for computationally efficient PEFT.**\nThought PEFT methods are not computationally efficient, often this is not much of a concern since tasks that require finetuning do not require massive datasets and thus does not consume long GPU hours. While potentially useful, I have some reservations whether such method is necessary. Can the authors provide examples in which PEFT would take more than hundreds of GPU hours, making computationally efficient PEFT methods necessary? In my practical experience, I'd rather train hours to a day more if that means I do not sacrifice accuracy for finetuning tasks.\n3. **DoRA Settings.**\nDoRA was first published as an improvement to LoRA, albeit the additional overhead in computation. I assume there must be settings in which DoRA is better than LoRA, as settings described in the DoRA paper. Could authors comment on whether such setting exists, and if SparseLoRA's performance still holds in that particular setting as well?\n\nWith the weaknesses and questions addressed appropriately, I am willling to change my evaluation of the paper. Thank you.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OxKdO1FYjj", "reviewer": "Reviewer_pTrd", "summary": "The paper introduces a method to accelerate fine-tuning of large language models (LLMs) by leveraging contextual sparsity. Unlike existing parameter-efficient fine-tuning (PEFT) methods such as LoRA and DoRA, which reduce memory usage but not computational cost, SparseLoRA optimizes both memory and computation.\nThe key contributions of the paper include: a training-free SVD-based sparsity estimator that selects a subset of weights for loss and gradient computation, context-aware sparsity application (non-uniform sparsity across layers based on sensitivity analysis, selective sparsity for context tokens, while keeping output tokens dense, progressive sparsity).\n\nEmpirical validation on commonsense and arithmetic reasoning benchmarks is provided.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> While the computation speedup is clear, the memory analysis and comparison to the baselines are missing\n\nOur approach uses LoRA for fine-tuning, so the memory profile remains the same as LoRA. Sparsifying the main branch does not affect memory usage.\n\n> Include comparisons with additional PEFT methods like APT and LoSA\n\nAPT and LoSA are designed to optimize inference-time sparsity rather than fine-tuning speed. Their methods typically slow down fine-tuning significantly to produce a sparse model with minimal accuracy loss at inference. For instance, LoSA requires 45.34 minutes compared to 13.78 minutes for LoRA, and under the Wanda setting, it takes 73.91 minutes versus 21.40 minutes for LoRA under SparseGPT, as shown in Table 8 of the original paper.\n\n> Include comparisons with additional PEFT methods like GaLore. Missing comparison to GaLore in terms of training time and GPU memory\n\nWe compare LoRA, GaLore, and SparseLoRA on Commonsense 170K and Math 10K. GaLore training requires A100 GPUs due to VRAM limitations of A6000 under DDP. Runtime is normalized to LoRA, as in our main submission. GaLore incurs a 1.58x training overhead but achieves similar performance to LoRA. The amortized time of GaLore, accounting for projection updates and online SVD, slows fine-tuning by 13.72x. While GaLore focuses on memory-efficient fine-tuning at the cost of computational efficiency, SparseLoRA accelerates fine-tuning with near-lossless performance.\n\n\n### LLaMA3-8B Commonsense 170K [on A100s]\n\n|Model|Runtime|Mean|BoolQ|PIQA|Social-IQA|HellaSwag|Winogrande|ARC-Easy|ARC-Challenge|OpenBookQA|\n|-----|-------|----|-----|-----|----------|---------|----------|--------|--------------|----------|\n|LoRA|1.00|87.1|74.5|89.6|82.8|95.3|88.4|93.1|84.4|88.8|\n|GaLore|1.58[13.72]|84.1|71.2|87.1|79.6|92.0|85.0|89.4|80.5|87.8|\n|SparseLoRA|0.78|87.0|74.7|89.5|82.8|95.3|88.8|92.9|83.6|88.3|\n\n### LLaMA3-8B Math 10K [on A100s] \n\n|Model|Runtime|Mean|gsm8k|svamp|mawps|\n|-----|-------|----|-----|-----|----------|\n|LoRA|1.00|80.0|71.1|79.5|89.5|\n|GaLore|1.58[13.72]|78.7|68.1|77.9|90.2|\n|SparseLoRA|0.82|80.0|70.9|79.4|89.9|\n\n> Include results on GLUE benchmark\n\nFollowing the reviewer's recommendation, we extend LoRA and SparseLORA using Llama3-8B on the GLUE benchmark. SparseLoRA maintains competetive perofrmance on sequence classification accross various subsets of the GLUE benchmark. \n\n### LLaMA3-8B GLUE Benchmark\n\n|Model|Mean|MRPC|SST2|QNLI|RTE|QQP|COLA|\n|-----|----|----|----|----|---|---|-----|\n|LoRA|88.6|92.1|96.2|95.2|88.8|91.8|67.7|\n|SparseLoRA|88.6|92.3|96.4|95.5|88.7|91.9|66.7|\n\n> The proposed method doesn’t improve the finetuned model accuracy but only reduces the training time\n\nOur method is designed to accelerate fine-tuning while preserving accuracy—not to improve accuracy over standard fine-tuning methods.\n\n\n> The GPU memory increase/decrease is not discussed\n\nSparseLoRA improves computational efficiency during fine-tuning without altering the memory usage compared to baseline LoRA.\n\n\n> Specify SVD rank (k) values and sparsity levels used in experiments\n\nWe use an SVD Rank of 8 across all models and datasets, resulting in minimal runtime overhead (Table 3). Layer sparsity ratios are based on sensitivity analysis per model, with Llama2-7B's layer-wise analysis shown in Figure 7. The specific sparsity ratios employed are:\n\n### Model Sparsity Information\n\n|Model|Dataset|FFNSparsity|FFNSparseLayers|QKVOSparsity|QKVOSparseLayers|\n|-----|-------|-----------|---------------|-------------|----------------|\n|Llama2-7B|Commonsense170K|90|L13-L29|60|L17-L29/L20,L24|\n|Llama2-7B|Math10k|90|L13-L29|60|L13-L29/L20,L24|\n|Llama2-13B|Commonsense170K|90|L17-L37|60|L17-L37|\n|Llama2-13B|Math10k|90|L13-L37|60|L13-L37|\n|Llama3-8B|Commonsense170K|90|L13-L29|60|L15-L29|\n|Llama3-8B|Math10k|90|L9-L29|60|L9-L29|\n\nThe selected layers and sparsity rates remain mostly consistent across models and datasets. Sensitivity analysis identifies layers unsuitable for sparsity (i.e. layers 20 and 24 in Llama2-7B). On Math 10k, layer ranges were increased to account for token splitting overhead. The sparsity assignment is latency-driven and informed by sensitivity analysis, ensuring no added overhead in SparseLoRA.\n\n> Inference Setup\n\nSparseLoRA targets fine-tuning accleration and only applies sparsity during fine-tuning; inference remains unchanged compared to baseline LoRA.\n\n> Missing DoRA Results\n\nResults for the DoRA method with LLaMA2-13B are missing due to OOM on A6000 GPUs without gradient checkpointing, as DoRA requires significantly more memory.\n\n> Template Issues\n\nWe will correct table captions and formatting in the revised version.", "claims_and_evidence": "* While the computation speedup is clear, the memory analysis and comparison to the baselines (presented by the authors and not presented, please see below) are missing. \n* The comparison of the proposed method was done to a narrow set of models: LoRA and DoRA while there are more baselines that could be compared (please see my points below).\n\nThe claims regarding training speed-up are supported by empirical results.", "methods_and_evaluation_criteria": "The benchmark datasets choice looks reasonable to me. However I would like to see the results for a commonly used GLUE benchmark and not only common and arithmetic reasoning.", "theoretical_claims": "There are no theoretical claims", "experimental_designs_or_analyses": "There are plenty of works in LLM PEFT domain. I think the most missing baselines are APT[1] and Galore[2] methods (or their variants). Also I found [3] that looks similar to the proposed method. The authors presented only  2 baselines \n\nDespite the lack of additional baselines, I think that experimental design in terms of datasets and metrics is reasonable. \n\n\n[1] Zhao, Bowen, Hannaneh Hajishirzi, and Qingqing Cao. \"APT: adaptive pruning and tuning pretrained language models for efficient training and inference.\" Proceedings of the 41st International Conference on Machine Learning. 2024.\n\n[2] Zhao, Jiawei, et al. \"GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection.\" International Conference on Machine Learning. PMLR, 2024.\n\n[3] Huang, Weizhong, et al. \"Dynamic Low-Rank Sparse Adaptation for Large Language Models.\" The Thirteenth International Conference on Learning Representations.", "relation_to_broader_scientific_literature": "The method builds upon and extends:\n* LoRA (Hu et al., 2022) and DoRA (Liu et al., 2024b) for parameter-efficient fine-tuning.\n* Contextual sparsity approaches in LLM inference (Liu et al., 2023).\n\nThe key novelty is applying structured contextual sparsity to fine-tuning, whereas previous methods focused on inference-time acceleration. The work is highly relevant to ongoing research in:\n * Efficient LLM training (Thangarasa et al., 2023, Mozaffari et al., 2024)\n * Sparse computing for neural networks (Han et al., 2015, 2016)\n\nThe paper appropriately cites relevant work but does not discuss:\n* Alternative structured sparsity methods (e.g., block sparsity, hardware-aware sparsity).\n* Recent advances in mixed low-rank and sparse fine-tuning methods (e.g., WeLore, SLoPe).", "essential_references_not_discussed": "The paper cites enough related papers.", "other_strengths_and_weaknesses": "**Strengths**:\n* The proposed model reduces computation time by sparsifting the base model weights during the finetuning in addition to the low-rank adaptor training. \n* the method is evaluated on commonsense and arithmetic reasoning\n* The method reduces the number of FLOPS by 30-40%\n* runtime breakdown of LLM fine-tuning and sensitivity analysis on layer-wise sparsity is interesting and important. \n\n**Weaknesses**:\n\n* The proposed method  doesn’t improve the finetuned model accuracy but only reduces the training time\n\n* The GPU memory increase/decrease is not discussed.\n* Missing a comparison to the optimizer-base finetuning methods such as a Galore or other recent methods in terms of training time and GPU memory.\n* Missing more baselines for sparse finetuningof LLMs", "other_comments_or_suggestions": "* I think the format of the paper is not the same as provided in the ICML 2025 template (e.g. tables captions should be above the tables), please fix it.", "questions_for_authors": "* I don't find the values of k in svd decomposition you used in your experiments / or sparsity levels in Tables 1 + 2\n* It is not clear from the text what is your **inference** setup and how the inference times are compared across baselines and your method? I see only the training time measurements. \n\n* Why are some experimental results missing for DoRA method? (E.g. with base model LLaMA2-13B)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4aRBN4ioAa", "reviewer": "Reviewer_j9K3", "summary": "The paper proposes to accelerate LoRA fine-tuning with contextual sparsity. Tailored for fine-tuning, they propose a lightweight, training-free SVD sparsity estimator to reduce computation overhead. Experimental results show that they can speed up LoRA fine-tuning by 1.4x.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s thoughtful feedback and recognition of our approach. Below, we address each of the concerns in detail:\n\n> \"S2FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity\" is a recent paper published in Arxiv in December. However, it is officially published in ICLR 25 after the submission deadline of ICML.\n\nWe would like to clarify that our paper already cites the S2FT work. S2FT introduces a structured pruning approach aimed at memory-efficient fine-tuning. According to Figure 5 in their paper, S2FT achieves a speedup of 1.1× over LoRA on LLaMA2-7B for the Commonsense 170K dataset. In contrast, our method achieves a 1.32× speedup on the same setting. While S2FT prioritizes memory efficiency and accuracy, SparseLoRA addresses a complementary direction: computational efficiency with lossless accuracy. We will include a more detailed discussion of S2FT in the final version of the paper.\n\n\n> I can not see why this method needs to be used together with LoRA. It would be great to use it independently and compare it with FFT.\n\nWe agree that extending our method beyond LoRA fine-tuning is an exciting direction. However, there are some practical considerations tied to the characteristics of LoRA. A key component of our method is the dynamic prediction of activation sparsity using an SVD-based predictor. This predictor relies on the static nature of the base layer weights—which, in LoRA, remain frozen during fine-tuning. In contrast, in full fine-tuning settings, the base weights are updated throughout training. This undermines the validity of the SVD-based predictor, potentially leading to inaccurate sparsity estimates. At this stage, our method leverages the frozen base layers in LoRA to reliably identify and exploit structured sparsity for speedup. Exploring how to adapt the predictor dynamically for full fine-tuning is an interesting direction we plan to pursue in future work.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The motivation is related to contextual sparsity but the key contributions are how to apply the idea to fine-tuning.", "essential_references_not_discussed": "\"S2FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity\" is a recent paper published in Arxiv in December. However, it is officially published in ICLR 25 after the submission deadline of ICML.", "other_strengths_and_weaknesses": "Strengths\n- I think it is an interesting and meaningful observation that output tokens are more sensitive to pruning.\n\nWeaknesses\n- I can not see why this method needs to be used together with LoRA. It would be great to use it independently and compare it with FFT.", "other_comments_or_suggestions": "April 12: I will keep my score.", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received 4 weak accept. After careful consideration of the submission, the reviews and rebuttal, this meta-review recommends *weak accept*.

On the positive aspects, the proposed structural sparsity is a relevant contribution in improving computational efficiency of PEFT (LoRA) methods. The paper also presents comprehensive engineering of applying the proposed structural sparsity in terms of per-layer sensitivity analysis, context/output-aware sparsity, and progressive sparsity scheduling, with promising empirical results obtained. The rebuttal did a good job in addressing reviewers' concerns, leading to some upgraded recommendations.

There are several concerns remained: (i) Whether SparseLoRA’s complexity outweighs its benefits compared to simpler PEFT methods like QLoRA, especially for users not constrained by compute. The authors argued SparseLoRA is complementary and showed it can integrate with QLoRA. (ii) While SparseLoRA matches or slightly outperforms LoRA in low-FLOP regimes, it doesn’t necessarily improve peak accuracy. (iii) The writing and some definitions in the paper should be improved, which the authors promised to revise in the rebuttal.
Decision: Accept (poster)