Title: GASDU: Gauss--Southwell Dynamic Update for Efficient LLM Fine-Tuning
Primary area: foundation or frontier models, including LLMs
Keywords: parameter-efficient fine-tuning; LLM Post-training; large language models

Abstract: Parameter-efficient fine-tuning (PEFT) is crucial for adapting large language models (LLMs), yet existing methods trade off accuracy, latency, and compute: some add inference-time modules, others fix a static parameter set that can drift from evolving gradients, and dynamic variants can be costly. We propose \textbf{GA}uss--\textbf{S}outhwell \textbf{D}ynamic \textbf{U}pdate (\textsc{GASDU}), which performs \emph{periodic Gauss--Southwell-$k$} selection: every $M$ steps it uses the current gradients to select the $k$ largest-magnitude coordinates and updates only those entries while reusing the mask until the next refresh. The Top-$k$ selection is implemented in a streaming, tile-wise way to avoid materializing dense gradients, making the amortized refresh cost negligible. Theoretically, under a local Polyak--Łojasiewicz condition, we prove that \textsc{GASDU} enjoys a linear convergence rate scaled by a measurable gradient-retention factor and show that the factor degrades sublinearly within each refresh window. This sublinear decay implies that a moderate $M$ can maintain a high retention factor, which in turn explains \textsc{GASDU}'s near–full–fine-tuning behavior. Empirically, \textsc{GASDU} sustains high retention between refreshes at an extreme parameter budget (0.01\%) and consistently outperforms strong PEFT baselines and closely tracks or exceeds full fine-tuning across diverse commonsense and arithmetic reasoning benchmarks and LLMs (LLaMA-2/3 and GPT-OSS-20B).

TL;DR: A novel dynamic sparse fine-tuning method for efficient LLM adaption.