OpenDAS: Open-Vocabulary Domain Adaptation for Segmentation

Authors: Gonca Yilmaz, Songyou Peng, Marc Pollefeys, Francis Engelmann, Hermann Blum
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Computer Vision, Vision-Language Models, Domain Adaptation, Open-Vocabulary Segmentation, Prompt Tuning

Abstract:
Recently, Vision-Language Models (VLMs) have advanced segmentation techniques by shifting from the traditional segmentation of a closed-set of predefined object classes to open-vocabulary segmentation (OVS), allowing users to segment novel classes and concepts unseen during training of the segmentation model. However, this flexibility comes with a trade-off: fully-supervised closed-set methods still outperform OVS methods on base classes, that is on classes on which they have been explicitly trained. This is due to the lack of pixel-aligned training masks for VLMs (which are trained on image-caption pairs), and the absence of domain-specific knowledge, such as autonomous driving. Therefore, we propose the task of open-vocabulary domain adaptation to infuse domain-specific knowledge into VLMs while preserving their open-vocabulary nature. By doing so, we achieve improved performance in base and novel classes. Existing VLM adaptation methods improve performance on base (training) queries, but fail to fully preserve the open-set capabilities of VLMs on novel queries. To address this shortcoming, we combine parameter-efficient prompt tuning with a triplet-loss-based training strategy that uses auxiliary negative queries. Notably, our approach is the only parameter-efficient method that consistently surpasses the original VLM on novel classes. Our adapted VLMs can seamlessly be integrated into existing OVS pipelines, e.g., improving OVSeg by +6.0% mIoU on ADE20K for open-vocabulary 2D segmentation, and OpenMask3D by +4.1% AP on ScanNet++ Offices for open-vocabulary 3D instance segmentation without other changes.

Decision: reject
Reviewer scores: 3, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730588060104

thread_type:
official_review

<thread_object>
creator: reviewer_5snt

<content>
summary:
The paper proposes a new task called open-vocabulary domain adaptation for segmentation. The goal of this task is to improve the recognition performance of vision-language models on a specific domain, while preserving the open-vocabulary recognition capabilities. The proposed method relies on paramater efficient prompt tuning with a combination of cross-entropy and triplet loss. The success of adaptation is measured as classification performance on masked images containing a single segment corresponding to either base or novel semantic class. The experiments are conducted on 2D and 3D segmentation on three different datasets.

strengths:
The proposed method outperforms other adaptation methods on segment classification on both base and novel classes on different benchmarks.
The experiments reveal that the proposed method can also improve the segmentation performance when combined with two existing open-vocabulary segmentation models.

The proposed adaptation approach is simple, sound and effective.

weaknesses:
In my opinion, task name is missleading. The adaptation procedure actually optimizes segment classification (images containing individual segments), while it assumes there is a mask proposal generator available. More precisely, it assumes perfect mask generator both during training and evaluation. The term "open-vocabulary segmentation" implies both localization (mask proposal generation) and classification. Because of that, I was confused while reading the manuscript all the way until section 5 and the description of evaluation metrics. I think that this difference must be clearly stated and thoroughly described earlier in the manuscript.

The manuscript lacks a proper comparison with the related work from open-vocabulary segmentation. Table 3 shows that the adapted model can be used to improve performance of the open-vocabulary segmentation model OVSeg. However, the table shows results only for the ADE20k validation set which contains only in-vocabulary (base) classes, while it is usual to evaluate the performance on datasets with different vocabularies. Furthermore, OVSeg significantly lacks in performance for the current SOTA models (e.g. CatSeg (Cho et al.,2023) or FC-CLIP (Yu et al., 2024)). 

The proposed approach is computationally inefficient. For a single full image segmentation it requires multiple forward passes through CLIP backbone in order to classify all the mask proposals. This causes a lot of overhead due to processing potentially overlapping (or empty) image regions. On the other hand, some open-vocabulary segmentation methods avoid this by extracting features with CLIP backbone once, and then pooling these features for each mask (FC-CLIP, OpenSeg, ODISE). The proposed approach might also hurt the classification performance by removing the context (background) from each segment image. It is not clear what are the real performance relations between the two mentioned approaches. Hence, the proper comparison with related methods from open-vocabulary segmentation is necessary. 

It is unclear if other open-vocabulary segmentation methods which do not rely on image segment classification (e.g. FC-CLIP) could benefit from the proposed adaptation. This needs further investigation.

questions:
/
</content>
</thread_object>

</thread>
<thread>
creation date:
1730642907905

thread_type:
official_review

<thread_object>
creator: reviewer_ino1

<content>
summary:
This submission tends to address the task of the open vocabulary domain adaptation to infuse domain specific knowledge into vision language models. Extensive experiments are conducted.

strengths:
The motivation in this paper is convincing, addressing the issue of the lack of pixel-aligned training for pre-trained VLMs.

weaknesses:
1) The "lack of pixel-aligned training masks for VLMs (which are trained on image-caption pairs)" has been extensively studied in previous work [1*, 2*]. The authors should analyze and compare with [1*, 2*], even though they use different optimization strategies

2) This work introduces the concept of Domain Adaptation in OVS. The authors need to clarify the distinction between Domain Adaptation OVS and standard OVS. OVS itself can segment any text and has a general concept of data domains. In experimental settings, the authors use cross-dataset evaluation, which does not differ from the standard OVS settings.

3) In Table 3, the authors only compare with the CVPR23 work, which is insufficient. The latest OVS works [3*, 4*, 5*] should be included for comparison.

4) I have questions about the process of using GPT to generate Negative Queries (specifically the example from "ceiling" to "ceiling fan" in L303). Does this require image input? If only text input is provided, I believe GPT would return synonyms instead.

5) Minor Weaknesses: The writing contains redundancy and needs optimization. e.g., L227-231 does not describe the authors' proposed method; I suggest placing this part in Sec. 4.1.


[1*] Learning mask-aware clip representations for zero-shot segmentation, NeurIPS 23
[2*] Collaborative Vision-Text Representation Optimizing for Open-Vocabulary Segmentation, ECCV 24
[3*] SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation, CVPR 24.
[4*] Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP, NeurIPS 23.
[5*] Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models, CVPR 23.

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730728175732

thread_type:
official_review

<thread_object>
creator: reviewer_4ucz

<content>
summary:
This paper introduces OpenDAS, a novel approach to open-vocabulary domain adaptation (OVDA) for 2D and 3D segmentation. Traditional segmentation models are typically limited to closed-set vocabularies, lacking flexibility for novel classes or domain-specific knowledge. OpenDAS addresses this by adapting Vision-Language Models (VLMs), such as CLIP, to domain-specific segmentation tasks without sacrificing their open-vocabulary capabilities. The authors propose a method combining prompt tuning with a triplet-loss-based training strategy, incorporating auxiliary negative queries to boost generalization to novel classes within a target domain. OpenDAS integrates seamlessly with existing segmentation models, such as OVSeg for 2D and OpenMask3D for 3D segmentation, and experimental results show significant improvements over baseline methods on various benchmarks.

strengths:
* It defines a new task that adaptes VLMs to domain-specific segmentation tasks while preserving open-vocabulary capabilities.

* Extensive experiments demonstrate adaptability for both 2D and 3D segmentation tasks, improving performance on existing pipelines without extensive modifications.

weaknesses:
* The proposed framework is engineering, because the triple loss, visual and textual prompt tuning are common ways in open-vocabulary and domain adaptation communities. 

* More recent vision-language models should be compared, like SigLIP, llama, etc.

* Several ablation studies are needed, like the discussion about the margin $\mu$.

* The writing and structure should be further improved, e.g., the pipeline fig is a bit ambiguous.

questions:
Please see weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730983880175

thread_type:
official_review

<thread_object>
creator: reviewer_xeds

<content>
summary:
This paper introduces the task of domain adaptation for language-vision models in open-vocabulary 2D and 3D segmentation. In this framework, language-vision models are fine-tuned in a weakly supervised manner to enhance performance on domain-specific segmentation tasks.

The authors propose a method specifically tailored for the CLIP architecture. A pretrained CLIP model is extended by adding trainable prompts to the inputs of selected layers in both the textual and visual encoders, with the number of layers modified as a hyperparameter. The optimization process occurs in two phases. In the first phase, only the visual prompts are trained: an image passes through the visual encoder, while a set of domain-specific labels—comprising both positive and ChatGPT-generated negative queries—passes through the textual encoder. A cross-entropy loss is used to bring the image embedding closer to the correct textual embedding. In the second phase, only the textual prompts are trained, using a combined cross-entropy and triplet loss. For the triplet loss, a hard negative sample mining strategy is employed to refine results.

The authors conduct experiments on both 2D and 3D segmentation tasks, along with ablation studies to analyze the impact of different components in the proposed approach.

strengths:
S1) Paper is clearly written

S2) Experimental results demonstrate improvements over baseline CLIP performance when integrated with the OVSeg model. Additionally, the proposed prompt-learning method for specializing vision-language models (VLMs) outperforms alternative approaches

weaknesses:
W1 The paper diverges from the established definition and test setup for open-vocabulary segmentation. While previous studies assume that a segmentation model fine-tuned on one domain should retain its generalizability across other domains, this work fine-tunes on a more general domain and evaluates on a narrower domain. Consequently, the test setup is not directly comparable to previous open-vocabulary approaches, such as those in (Liang et al. 2023)

W2 The fine-tuning approach assumes access to densely labeled, domain-specific images, which differs from previous domain adaptation frameworks that often use self-supervised or student-teacher setups. Here, the fine-tuning process effectively performs domain-specific classification training.

questions:
1) What are the relations between ADE20K, SN++ Offices and KITTI-360 classes in Table 2?
2) Were the image segments used for "domain adaptation" obtained from densely labelled images?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733050809325

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to Reviewers and AC

comment:
Dear Reviewers and AC,

We sincerely thank the reviewers and AC for the time they spent on our submission. We take all the comments and suggestions very seriously to improve our work and hope our rebuttal can address these comments and suggestions. We have **updated the manuscript** accordingly. All revisions in the updated version are highlighted in **red**. The revisions in the paper and comments can be summarized as follows:

Add more experimental results:
* Add comparisons to more recent OVS methods: FC-CLIP [1], MAFT+ [2] (iNo1-Q1; iNo1-Q3; 5sNt-Q2; 5sNt-Q3; Table 3 and Appendix H).
* Add comparisons with predicted masks on ScanNet++ Offices (5sNt-Q2, in the comments: "Official Comment -- Part 3").
* Add ablation of mu (4Ucz-Q3; Appendix F).

Add more discussion and details:
* Clarify that the proposed method focuses on better segment classification, rather than adapting the mask proposals (5sNt-Q1; L99-L101).
* Clarify that our approach is based on densely labeled images (XEDS-Q2; L107-L108).
* Move L227-231 from Section 4.2 to 4.1 (iNo1-Q5; now L224-L230).
* Clarify the relationship between ADE20K and ScanNet++ Offices, as well as ADE20K and KITTI-360, respectively (XEDS-Q3; L373-L377).

Thanks again for all the effort and time.

Best,

Authors

[1] Yu et al., Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP, NeurIPS'23.

[2] Jiao et al., Collaborative Vision-Text Representation Optimizing for Open-Vocabulary Segmentation, ECCV'24.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733154943629

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Kindly Reminder: Response Deadline Approaching

comment:
Dear Reviewers,

We are thankful for the time reviewers committed to review our submission so far. We would like to remind all the reviewers that the deadline to respond is approaching (2nd December, 23:59 AoE). If there are any remaining questions, we would be happy to provide further clarifications.

Best regards,

Authors.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a simple yet effective approach to fine-tune open-vocabulary 2D and 3D segmentation models by leveraging negative prompts generated by LLMs.The strengths of this work lie in its clarity, well-structured presentation, and comprehensive experimental validation across multiple architectures and datasets, which demonstrate the universality of the proposed approach.  The additional results provided during the rebuttal further strengthen the empirical evidence, showing that despite the simplicity of the method, it achieves notable performance improvements.

However, a major concern raised by multiple reviewers pertains to the framing of the proposed OVDA task. The approach assumes access to pixel-level annotations in the target domain but does not adequately explain why novel classes remain unlabeled despite the availability of such dense annotations. This assumption limits the practical applicability of the method, as real-world scenarios often involve either weak supervision (e.g., partially annotated data) or unsupervised settings (e.g., no target labels at all). These scenarios may further introduce the challenges that were not discussed in this paper, such as the domain-specific concept definitions can vary significantly (e.g., "road" vs. "sidewalk"). Achieving robustness across such variations would require much more engineering efforts to adapt ChatGPT to all possible cases. Additionally, the method introduces considerable computational overhead, particularly due to its reliance on multiple forward passes through the CLIP to process overlapping image regions. This inefficiency remains unresolved.

Overall, the proposed method is an engineering-oriented solution, but its novelty and comprehensiveness are somewhat limited. Reframing the task to address more practical scenarios would better position the work in the future.