SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity
Authors: name: Samir Khaki, name: Xiuyu Li, name: Junxian Guo, name: Ligeng Zhu, name: Chenfeng Xu, name: Konstantinos N. Plataniotis, name: Amir Yazdanbakhsh, name: Kurt Keutzer, name: Song Han, name: Zhijian Liu
Organization: Z Lab

Fine-tuning LLMs is both computationally and memory-intensive. While
parameter-efficient fine-tuning methods, such as QLoRA and DoRA, reduce the
number of trainable parameters and lower memory usage, they do not decrease
computational cost. In some cases, they may even slow down fine-tuning. In this
paper, we introduce SparseLoRA, a method that accelerates LLM fine-tuning
through contextual sparsity. We propose a lightweight, training-free SVD
sparsity estimator that dynamically selects a sparse subset of weights for loss
and gradient computation. Also, we systematically analyze and address
sensitivity across layers, tokens, and training steps. Our experimental results
show that SparseLoRA reduces computational cost by up to 2.2 times and a
measured speedup of up to 1.6 times while maintaining accuracy across various
downstream tasks, including commonsense and arithmetic reasoning, code
generation, and instruction following.