ICML 2025

SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity
Samir Khaki * 1 Xiuyu Li * † 2 Junxian Guo * 3 Ligeng Zhu 3 Konstantinos N. Plataniotis 1
Amir Yazdanbakhsh 4 Kurt Keutzer 2 Song Han 3 Zhijian Liu 3
https://z-lab.ai/projects/sparselora
Abstract
Fine-tuning LLMs is both computationally and
memory-intensive. While parameter-efficient fine-
tuning methods, such as QLoRA and DoRA,
reduce the number of trainable parameters and
lower memory usage, they do not decrease com-
putational cost. In some cases, they may even
slow down fine-tuning. In this paper, we introduce
SparseLoRA, a method that accelerates LLM fine-
tuning through contextual sparsity. We propose
a lightweight, training-free SVD sparsity estima-
tor that dynamically selects a sparse subset of
weights for loss and gradient computation. Also,
we systematically analyze and address sensitivity
across layers, tokens, and training steps. Our ex-
perimental results show that SparseLoRA reduces
computational cost by up to 2.2× and a measured
speedup of up to 1.6× while maintaining accu-
racy across various downstream tasks, including
commonsense and arithmetic reasoning, code gen-
eration, and instruction following.
1. Introduction
Large language models (LLMs) are trained on vast, general-
domain datasets. They are often fine-tuned to improve their
performance in specific domains (Saab et al., 2024) or to
align their predictions with user preferences (Zhang et al.,
2024a). However, fine-tuning very large models can be pro-
hibitively expensive, both in terms of memory requirements
and computational costs.
Extensive efforts have been made in parameter-efficient fine-
tuning (PEFT) to reduce memory consumption of LLM fine-
tuning. LoRA (Hu et al., 2022) represents weight updates
using low-rank approximations. Building upon this, many
*Equal contribution †Project lead 1University of Toronto 2UC
Berkeley 3MIT 4Google DeepMind. Correspondence to: Samir
Khaki <samir.khaki@mail.utoronto.ca>.
Proceedings o...