Semantic Library Adaptation: LoRA Retrieval and Fusion for
  Open-Vocabulary Semantic Segmentation
Authors: name: Reza Qorbani; name: Gianluca Villani; name: Theodoros Panagiotakopoulos; name: Marc Botet Colomer; name: Linus Härenstam-Nielsen; name: Mattia Segu; name: Pier Luigi Dovesi; name: Jussi Karlgren; name: Daniel Cremers; name: Federico Tombari; name: Matteo Poggi
Published: 2025-03-27 17:59:58

Abstract: Open-vocabulary semantic segmentation models associate vision and text to
label pixels from an undefined set of classes using textual queries, providing
versatile performance on novel datasets. However, large shifts between training
and test domains degrade their performance, requiring fine-tuning for effective
real-world applications. We introduce Semantic Library Adaptation (SemLA), a
novel framework for training-free, test-time domain adaptation. SemLA leverages
a library of LoRA-based adapters indexed with CLIP embeddings, dynamically
merging the most relevant adapters based on proximity to the target domain in
the embedding space. This approach constructs an ad-hoc model tailored to each
specific input without additional training. Our method scales efficiently,
enhances explainability by tracking adapter contributions, and inherently
protects data privacy, making it ideal for sensitive applications.
Comprehensive experiments on a 20-domain benchmark built over 10 standard
datasets demonstrate SemLA's superior adaptability and performance across
diverse settings, establishing a new standard in domain adaptation for
open-vocabulary semantic segmentation.