Title: SpikeLoRA: Learnable Activation Sparsity for Low-Rank Adaptation using Spiking Neural Networks
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: spiking neural networks, low-rank adaptation

Abstract: Low-rank adaptation (LoRA) is a fine-tuning method that freezes the parameters of a pre-trained model and injects small trainable matrices. LoRA-based methods focus on parameter-level efficiency, but do not directly control the activations in the low-rank space. We introduce SpikeLoRA, a spiking low-rank adaptation fine-tuning method that leverages the leaky integrate-and-fire (LIF) neuron to introduce learnable sparsity with minimal computational overhead. The LIF neuron gates the activations from the $A$-matrix in LoRA, sparsifying them while preserving learned information. This design makes SpikeLoRA a sparse fine-tuning method for both spiking and traditional LLMs, with the additional efficiency benefit of being compatible with neuromorphic hardware. Our experiments show that over 70\% sparsity is achievable without a significant drop in performance. Further, improved performance as compared to LoRA is observed for smaller datasets and higher-rank settings. We also show that SpikeLoRA indirectly mitigates overfitting, particularly for higher ranks.

TL;DR: SpikeLoRA is a PEFT method that makes use of leaky integrate-and-fire (LIF) neurons to gate LoRA's A-matrix activations, which promotes high sparsity in the low-rank space and mitigates overfitting for both spiking and traditional LLMs.