Singular Value Few-shot Adaptation of Vision-Language Models
Authors: name: Taha Koleilat, name: Hassan Rivaz, name: Yiming Xiao

Vision-language models (VLMs) like CLIP have shown impressive zero-shot and
few-shot learning capabilities across diverse applications. However, adapting
these models to new fine-grained domains remains difficult due to reliance on
prompt engineering and the high cost of full model fine-tuning. Existing
adaptation approaches rely on augmented components, such as prompt tokens and
adapter modules, which could limit adaptation quality, destabilize the model,
and compromise the rich knowledge learned during pretraining. In this work, we
present CLIP-SVD, a novel multi-modal and
parameter-efficient adaptation technique that leverages Singular Value
Decomposition (SVD) to modify the internal parameter space of CLIP without
injecting additional modules. Specifically, we fine-tune only the singular
values of the CLIP parameter matrices to rescale the basis vectors for domain
adaptation while retaining the pretrained model. This design enables enhanced
adaptation performance using only 0.04\% of the model's total
parameters and better preservation of its generalization ability. CLIP-SVD
achieves state-of-the-art classification results on 11 natural and 10
biomedical datasets, outperforming previous methods in both accuracy and
generalization under few-shot settings. Additionally, we leverage a natural
language-based approach to analyze the effectiveness and dynamics of the CLIP
adaptation to allow interpretability of CLIP-SVD. The code is publicly
available at https://github.com/HealthX-Lab/CLIP-SVD.