DynAlign: Unsupervised Dynamic Taxonomy Alignment for Cross-Domain Segmentation

Authors: Han Sun, Rui Gong, Ismail Nejjar, Olga Fink
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: unsupervised domain adaptation，semantic segmentation，taxonomy adaptation，foundational models

Abstract:
Current unsupervised domain adaptation (UDA) methods for semantic segmentation typically assume identical class labels between the source and target domains. This assumption ignores the label-level domain gap, which is common in real-world scenarios, and limits their ability to identify finer-grained or novel categories without requiring extensive manual annotation.
A promising direction to address this limitation lies in recent advancements in foundation models, which exhibit strong generalization abilities due to their rich prior knowledge. However, these models often struggle with domain-specific nuances and underrepresented fine-grained categories.
To address these challenges, we introduce DynAlign, a two-stage framework that integrates UDA with foundation models to bridge both the image-level and label-level domain gaps. Our approach leverages prior semantic knowledge to align source categories with target categories that can be novel, more fine-grained, or named differently. (e.g., vehicle to car, truck, bus). Foundation models are then employed for precise segmentation and category reassignment. To further enhance accuracy, we propose a knowledge fusion approach that dynamically adapts to varying scene contexts. DynAlign generates accurate predictions in a new target label space without requiring any manual annotations, allowing seamless adaptation to new taxonomies through either model retraining or direct inference. 
Experiments on the GTA $\rightarrow$  IDD and GTA$\rightarrow$ Mapillary benchmarks validate the effectiveness of our approach, achieving a significant improvement over existing methods. Our code is publically available at https://github.com/hansunhayden/DynAlign.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730525008192

thread_type:
official_review

<thread_object>
creator: reviewer_7s4z

<content>
summary:
Traditional UDA methods for semantic segmentation often assume identical class labels in both source and target domains. To tackle this, this paper presents DynAlign, a two-stage framework that combines UDA with foundation models to address both image-level and label-level domain gaps. It leverages prior semantic knowledge to align source and target categories and uses foundation models for detailed segmentation and category realignment. To boost accuracy further, a knowledge fusion strategy is divised to dynamically adapt to diverse scene contexts.

strengths:
1. The method design of this paper is straightforward and easy to understand, with sufficient details and reliable techniques.
2. The paper aims to address the label space mismatch issue in UDA tasks, and this research problem is very meaningful.
3. The research motivation of the paper is strong, and the solution is clearly articulated.

weaknesses:
1. The paper dedicates considerable space to comparisons with open-vocabulary semantic segmentation methods and benchmarks its approach against those used in that task. I understand that this is due to the limitations in label space, which restrict traditional UDA methods. However, since the focus of this paper is UDA, I believe the authors should concentrate on comparisons with UDA methods to better highlight the advantages of this work. Throughout my reading, I found myself constantly switching between the two tasks (UDA and open-vocabulary SS), making it challenging to clearly grasp the contributions of this paper.

2. I understand that earlier methods also utilized foundation models but still required target labels for few-shot training. In contrast, this paper employs foundation models in a fully unsupervised manner, correct? Is this the main difference/contribution this paper has compared to previous methods? 

3. How to better utilize foundation models has been a significant research topic in recent years, and I find the authors' exploration of this subject very meaningful. However, with the increasing number of foundation models available, it is insufficient to merely leverage more or better foundation models to complete tasks. The authors should emphasize the specific benefits of their design rather than simply implementing it in an unsupervised manner. Since the inference of large models is inherently unsupervised, simply using them does not constitute a substantial contribution.

4. The use of foundation models still incurs computational resource demands, which can affect the practical applicability of this framework. The experimental section should include comparisons regarding the computational resource consumption of the models, such as memory usage and inference time.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730682904177

thread_type:
official_review

<thread_object>
creator: reviewer_geae

<content>
summary:
This paper presents  a two-stage framework that integrates UDA with foundation models to bridge both the image-level and label-level domain gaps. It consists of multiple modules including conventional UDA to bridge the distribution gap between source and target, vision foundation model to extract general segmentation, LLM to construct the label taxonomy, and CLIP to align visual and text features to generate well-aligned pseudo labels for self-training.

strengths:
Belows are strong points that this paper has:

1. Well-written and systematically organized to make the reader to understand more easily. 

2. It proposes reasonable fusion strategy between UDA and Foundation model to resolve the distribution and label shift issue  in cross-domain scenario

3. It achieves remarkable performance improvement, showing the effectiveness of the proposed method on two benchmark.

4. Figure 4 shows that training with generated pseudo labels can not only achieve better score but also increase the inference efficiency.

weaknesses:
Belows are weak points that this paper has:

1. In order to align with line 112 ("integrating with any UDA-based semantic segmentation models"), it would be better that author provide another examples to show the compatibility of the proposed method (e.g., MIC + DynAlign, or other UDA model + DynAlign)

2. Need to have more analysis on how robust the proposed knowledge fusion is. One way to do is that using different foundation model (e.g., MobileSAM for SAM, Llama for GPT-4,  another VLM for CLIP) can also work with the proposed method. Showing this compatibility can provide more flexible fusion environment.

3. Any failure case while fusing UDA and foundation model?

questions:
Please see above Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702175033

thread_type:
official_review

<thread_object>
creator: reviewer_cqs3

<content>
summary:
The paper targets the label-level domain gap, which is common in real-world scenarios.  The authors introduce DynAlign, a
two-stage framework that integrates UDA with foundation models to bridge both the image-level and label-level domain gaps. DynAlign leverages prior semantic knowledge to align source categories with target categories that can be novel, more fine-grained, or named differently (e.g., ‘vehicle’ to {‘car’, ‘truck’, ‘bus’}). Foundation models are then employed for precise segmentation and category reassignment. To further enhance accuracy, the authors propose a knowledge fusion approach that dynamically adapts to varying scene contexts. DynAlign generates accurate predictions in a new target label space without requiring any manual annotations.

strengths:
1. The problem in this paper is of great practical significance.

2. The writing of this paper is clear and well-organized.

3. The experimental results showed some improvements when using such technology, compared with previous rivals.

weaknesses:
1. The article only explores coarse-to-fine experiments. In real-world scenarios, however, the label structure of the target domain may sometimes be coarser than that of the source domain. How would the proposed method be adapted for such a setting? Additionally, the effectiveness of the method under these conditions has not been verified.

2. The contribution towards DG area is limited. The generalization achieved by the methods in this paper primarily derives from the pre-trained models—specifically, SAM's generalization in segmentation and CLIP's generalization in multimodal alignment—rather than from the techniques introduced within the paper itself. As a result, this work mainly resembles a mapping technique that aligns open-vocabulary labels between the source and target domains, an idea that has been explored extensively in other studies. Consequently, I do not view this paper as making a significant contribution to the field of domain generalization. In my opinion, as an ICLR article, it still needs a certain theoretical depth.

Minor issues:
Some papers are cited repeatedly in the reference, such as "Domain aligned clip for few-shot classification." and "Scaling open-vocabulary image segmentation with image-level labels."

questions:
Please see the "Weaknesses" part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707422660

thread_type:
official_review

<thread_object>
creator: reviewer_ddgh

<content>
summary:
This paper addresses the label-space domain gap within cross-domain semantic segmentation by proposing a novel framework that integrates traditional unsupervised domain adaptation (UDA) methods with foundational models such as large language models, vision-language models, and vision foundation models. Specifically, the large language model is utilized to reason and establish mappings between source and target label spaces. Using these mappings, predictions from unsupervised adaptation segmentation models are transformed into the target label space and subsequently fused with predictions from foundation models, including SAM and CLIP. The proposed framework demonstrates superior performance compared to several open-vocabulary recognition models.

strengths:
1. The proposed framework is technically sound.
2. The paper addresses a practical and significant issue: the label-space domain gap within domain adaptation.

weaknesses:
1. High Complexity and Heuristic Dependence
The framework’s complexity, along with its reliance on various heuristics, may hinder practical usability. Additionally, some heuristics lack clear specifications, making reproduction challenging. Key points include:

a) Inference Complexity: The framework requires inference from at least three models (UDA model, SAM, and CLIP), contributing to increased computational demands and overall complexity.

b) Semantic Taxonomy Mapping: The framework initially prompts GPT-4 for potential taxonomy mappings, refined through human input. This process adds a human burden due to prompt tuning and taxonomy refinement and is only briefly outlined in the supplementary material (lines 916 & 947), lacking comprehensive detail.

c) Criteria for Label Ambiguity: The paper states, “For labels without ambiguity, e.g., sky, we only use the original label for text feature extraction” (lines 1000-1001). The criteria for “ambiguity” remain unclear, introducing additional heuristics or human intervention.

2. Limited Experimental Scope
While the paper showcases superior performance in certain settings, the experimental scope and choice of baselines are somewhat limited. Expanding the evaluation would strengthen the framework’s claim of superiority:

a) Additional Baselines with Foundation Models: Numerous studies have leveraged foundation models for semantic segmentation, yet many of these are omitted in the experimental section. Including models such as [A] for domain-generalized segmentation and [B] for open-vocabulary segmentation would enhance comparability.

b) Traditional Domain Adaptation Settings: Although the framework targets handling substantial label-space domain gaps for real-world applications, demonstrating its efficacy in traditional domain adaptation settings (e.g., GTA5 to Cityscapes, with a smaller label-space gap) is also important.

c) Other Label-Space Domain Gaps: The experiments primarily focus on scenarios where the target domain includes more semantic classes than the source (e.g., 45 classes for Mapillary, 24 for IDD, and 19 for GTA). Evaluating scenarios with fewer classes in the target domain (e.g., Mapillary to Cityscapes) would substantiate the framework’s generalizability.

References

[A] Stronger, Fewer, & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation, CVPR 2024

[B] Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactively, ECCV 2024

questions:
The framework seems to rely purely on SAM for pixel grouping, with UDA and CLIP then reclassifying each mask region. Is this my understanding accurate?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730722652336

thread_type:
official_review

<thread_object>
creator: reviewer_ggqb

<content>
summary:
This paper propose two-stage framework called DynAlign for unsupervised taxonomy-adaptive cross-domain semantic segmentation. DynAlign leverages prior semantic knowledge to align source categories with target categories that can be novel, more fine-grained, or named differently. Foundation models are then employed for precise segmentation and category reassignment. Knowledge fusion approach is proposed to dynamically adapt to varying scene contexts.

strengths:
1. The problem that the paper focuses on, taxonomy-adaptive domain adaptation without requiring additional annotations from the target domain, is innovative and interesting.
2. The approach based on the foundation model is explainable, reasonable, and innovative.

weaknesses:
1. The explanation of the cpation in Figure 2 is not clear. Please explain the functions of the two CLIPs in the figure. And the caption introduces "CLIP fusing the visual knowledge from SAM with the semantic knowledge from LLM to reassign accurate labels. The predictions can be used as pseudo-labels to further fine-tune the UDA model". Since the predictions can be used as pseudo-labels to further fine-tune the UDA model, should the second CLIP have an output arrow?  The figure 2 cannot show the process well, which causes some misunderstandings.
2. The paper proposes a two-stage approach. however, three steps are introduced in Figure 2. This makes it difficult to understand the content of figure 2. Please redraw the figure 2 to make the process more intuitive and easy to understand.
3. The proposed method includes many networks and is very complex. Please compare the model's training memory usage, training time, and model parameter.
4. As a new benchmark, will the proposed method publish the complete training and testing code for subsequent follow-up research?
5. In 4.2 SEMANTIC TAXONOMY MAPPING, how is the context description set generated? In formula 2, on which dimension is the average operation performed?
6. The symbol usage is inconsistent. In 4.4 KNOWLEDGE FUSION, the multi-scale visual feature is F_V, while the multi-scale visual feature in Figure 3 is F_g. Please check the symbols and charts in the full text to avoid ambiguity.
7. In Figure 3, what does lane-marking 0.6 mean?
8. In 5.4 ABLATION STUDIES, should multi-scale vision feature be changed to multi-scale visual feature? Please keep the name consistent in the paper.
9. Please add more experimental details. For example, the proposed method uses the CLIP backbone, but is it the CLIP ViT-B-16 or CLIP ViT-L-14 backbone or something else?

questions:
If the rebuttal process can explain the above problems and confusions, I will increase my score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730729613364

thread_type:
official_review

<thread_object>
creator: reviewer_5x3w

<content>
summary:
The paper formulates the problem of unsupervised taxonomy adaptive cross-domain semantic segmentation. To bridge both image-level and label-level domain gaps without supervision, the paper proposes a two-stage approach DynAlign that integrates both domain-specific knowledge and rich open-world prior knowledge from foundation models. Experiments on two street scene semantic segmentation benchmarks validates the effectiveness of the proposed method.

strengths:
1. The paper writing is good with well formula representation.
2. The unsupervised taxonomy adaptive cross-domain semantic segmentation task is interesting and meaningful.

weaknesses:
1. In Lines 80-81, the reason why this problem is significant but not be studied needs explain. 
2. Experimental results of existing taxonomy-adaptive domain adaptation and UDA methods also need to be presented in the Table 1 for comparison.  More parametric experiments about the confidence in the Lines 372 - 373 need to be conducted and discussed.

questions:
1. In Lines 80-81, why this problem is significant but not be studied?  
2. What are experimental results of existing taxonomy-adaptive domain adaptation and UDA  methods on two street scene semantic segmentation benchmarks.  How choose the confidence threshold value in the Lines 372 - 373.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732634892811

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank all the reviewers for their time and effort in reviewing our work and for recognizing the significance and practical meaning of the problem addressed in this work. Based on your feedback, we have made further modifications to the revised version of our paper:

1. **Clarity:** We appreciate the reviewers highlighting issues related to clarity and inconsistencies in the writing and figures. In response, we have elaborated further on the methodology, revised the figures, and addressed all the noted minor issues to improve clarity.

2. **Motivation:** We are grateful for the reviewers’ input on emphasizing the main contributions of this work. To address this, we have strengthened the motivation and clearly articulated our key contributions and strengths compared to existing methods. These improvements are reflected in the introduction, related works, and additional analysis in the experiments.

3. **Implementation Details:** We thank the reviewers for acknowledging our contributions and expressing interest in our work. We have added more implementation details in the Experiments section and confirmed our commitment to making the code publicly available in the abstract.

4. **Additional Experiments:** We greatly appreciate the reviewers’ suggestions for inspiring experiments to validate our method further. In response, we have included:  
   - **Results under more DA settings:** To demonstrate the practical significance of our work, we conducted additional experiments under traditional UDA, fine-to-coarse adaptation, and taxonomy-adaptive settings with smaller label space gaps. The results are presented in **Table 1** and Appendix **A.5**.  
   - **Ablations on each module of our framework:** We evaluated the performance of substituting different modules, as shown in **Table 4** and _Lines 495–501_, to highlight the flexibility of our framework.  
   - **Memory Usage and Efficiency:** We computed the memory usage and efficiency of our framework. These results are highlighted in ***Section 5.3***, with detailed information included in Appendix ***A.2 COMPUTATIONAL EFFICIENCY*** (_Lines 854–882_).
   - More additional experiments are detailed in the appendix, and we have addressed individual reviewers' concerns directly. As noted in our previous replies, we will include results from additional comparison methods in the camera-ready version to provide a more comprehensive benchmark and facilitate future research in this area.

We have also made further modifications and elaborations based on individual reviewer suggestions. For your convenience, all changes have been highlighted in blue in the updated revised version.

Once again, we sincerely thank all reviewers for their constructive feedback and invaluable suggestions. Could we kindly request confirmation on whether we have adequately addressed your concerns? We would be more than happy to engage in further discussions or implement additional refinements to improve the paper further based on your feedback.

Yours sincerely,  
Authors of Submission 9137
</content>
</thread_object>

</thread>


Metareview:
metareview:
All the 6 reviewers provide positive ratings after rebuttal, with 4 upgraded scores. Initially, the reviewers had concerns about technical clarity/complexity, experimental results with more baselines and realistic settings, usage of foundation models. In the post-rebuttal discussion period, all the reviewers are satisfactory with the authors' comments and revised paper. After taking a close look at the paper, rebuttal, and discussions, the AC agrees with reviewers' feedback of the proposed method being sound and effective to solve a challenging task. Therefore, the AC recommends the acceptance rating.