SALT: Singular Value Adaptation with Low-Rank Transformation
Authors: name: Abdelrahman Elsayed, name: Sarim Hashmi, name: Mohammed Elseiagy, name: Hu Wang, name: Mohammad Yaqub, name: Ibrahim Almakky

The complex nature of medical image segmentation calls for models that are
specifically designed to capture detailed, domain-specific features. Large
foundation models offer considerable flexibility, yet the cost of fine-tuning
these models remains a significant barrier. Parameter-Efficient Fine-Tuning
(PEFT) methods, such as Low-Rank Adaptation (LoRA), efficiently update model
weights with low-rank matrices but may suffer from underfitting when the chosen
rank is insufficient to capture domain-specific nuances. Conversely, full-rank
Singular Value Decomposition (SVD) based methods provide comprehensive updates
by modifying all singular values, yet they often lack flexibility and exhibit
variable performance across datasets. We propose SALT (Singular Value
Adaptation with Low-Rank Transformation), a method that selectively adapts the
most influential singular values using trainable scale and shift parameters
while complementing this with a low-rank update for the remaining subspace.
This hybrid approach harnesses the advantages of both LoRA and SVD, enabling
effective adaptation without relying on increasing model size or depth.
Evaluated on 5 challenging medical datasets, ranging from as few as 20 samples
to 1000, SALT outperforms state-of-the-art PEFT (LoRA and SVD) by 2% to 5% in
Dice with only 3.9% trainable parameters, demonstrating robust adaptation even
in low-resource settings. The code for SALT is available at:
https://github.com/BioMedIA-MBZUAI/SALT