DynAlign: Unsupervised Dynamic Taxonomy Alignment for Cross-Domain Segmentation
Authors: Han Sun
Rui Gong
Ismail Nejjar
Olga Fink
Venue: ICLR 2025 Poster (2025)
Keywords: unsupervised domain adaptation，semantic segmentation，taxonomy adaptation，foundational models
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Current unsupervised domain adaptation (UDA) methods for semantic segmentation typically assume identical class labels between the source and target domains. This assumption ignores the label-level domain gap, which is common in real-world scenarios, and limits their ability to identify finer-grained or novel categories without requiring extensive manual annotation.
A promising direction to address this limitation lies in recent advancements in foundation models, which exhibit strong generalization abilities due to their rich prior knowledge. However, these models often struggle with domain-specific nuances and underrepresented fine-grained categories.
To address these challenges, we introduce DynAlign, a two-stage framework that integrates UDA with foundation models to bridge both the image-level and label-level domain gaps. Our approach leverages prior semantic knowledge to align source categories with target categories that can be novel, more fine-grained, or named differently. (e.g., vehicle to car, truck, bus). Foundation models are then employed for precise segmentation and category reassignment. To further enhance accuracy, we propose a knowledge fusion approach that dynamically adapts to varying scene contexts. DynAlign generates accurate predictions in a new target label space without requiring any manual annotations, allowing seamless adaptation to new taxonomies through either model retraining or direct inference. 
Experiments on the GTA $\rightarrow$  IDD and GTA$\rightarrow$ Mapillary benchmarks validate the effectiveness of our approach, achieving a significant improvement over existing methods. Our code is publically available at https://github.com/hansunhayden/DynAlign.

Published as a conference paper at ICLR 2025
DYNALIGN: UNSUPERVISED DYNAMIC TAXONOMY ALIGNMENT FOR
SEGMENTATION
Han Sun' Rui Gong* Ismail Nejjar'
Olga Fink'
'EFL AAmazon
(han. sun, ismail eeeaaiinneeeaae jejja, olga. fink.iiieeeepl.oo ruigonge aaaooggamazzonn com
ABSTRACT
Current unsupervised domain adaptation (UDA) methods for semantic segmen- tation typically assume identical class labels between the source and target do- mains. This assumption ignores the label-level domain gap, which is common in real-world scenarios, thus limiting their ability to identify finer-grained novel categories without requiring extensive manual annotation. tion to address this limitation lies in recent advancements in foundation models, which exhibit strong generalization abilities due to their rich prior knowledge. However, these models often struggle with domain-specific nuances and under- represented fine-grained categories. To address these challenges, we introduce DynAlign framework that integrates UDA with foundation models to bridge both the image-level and label-level domain gaps. Our approach leverages prior semantic knowledge to align source categories with target categories that can be novel, more fine-grained, or named differently (e.g., "vehicle" to ('car', 'truck', 'bus")). Foundation models are then employed for precise segmentation and cate- gory reassignment. To further enhance accuracy, we propose knowledge fusion approach that dynamically adapts to varying scene contexts. DynAlign generates accurate predictions in new target label space without requiring any manual an- notations, allowing seamless adaptation to new taxonomies through either model retraining or direct inference. Experiments tation benchmarks GTA GTA-Mapillary Vistas and GTA+>DD validate the effective- ness of our approach, achieving significant improvement over existing methods. Our code is publically available at httpss//gthbb..ommmmauuuuuyyyyyp
promising direc-
the street scene semantic segmen-
INTRODUCTION
Semantic segmentation is crucial computer vision task that assigns category labels to each pixel in an image, enabling detailed scene understanding. Driven by advancements in deep learning. the field has recently
significant progress, with applications ranging from autonomous driving (Cheng jet al. 2022 Jain et al.) 2023) to medical image diagnosis (Cao et al., (022). Despite this progress, models trained on labeled source datasets often struggle generalize data with different distri- butions due to variations in weather, illumination, or object appearance, resulting in degraded per- formance. While re-training or fine-tuning models can address this issue, these approaches require annotated in-domain data, which is particularly costly for semantic segmentation.
Unsupervised Domain Adaptation (UDA) addresses the challenge of adapting model trained on labeled source domain to an unlabeled target domain by mitigating the adverse effects of domain shift without requiring costly data annotations. However, most UDA are constrained by domain- specific knowledge from the available datasets and typically operate under closed-set assumptions, where the label spaces of the two domains are identical. This assumption limits their applicability in real-world scenarios, where source and target datasets often exhibit the label-level taxonomy gap - including variations in class categories, semantic contexts, and category granularity.
To overcome this limitation, open-set Domain Adaptation (DA) methods have been developed to recognize novel classes in the target domain (Saito & Saenko, 2021 Li et al. 2023). However, these methods only capable of distinguishing unknown classes from known ones and do not provide detailed classifications for new target classes To solve this, taxonomy-adan adaptive domain "This work done hefore Rui Gong joined Amazon.

---

Published as a conference paper at ICLR 2025
Knowledge
UDA
Open-Vocabile Segmentation
(a) UDA: domain-specific knowledge
(b) Open- vocabulayy segmentation: prior knowledge
Prior Knowledge
road
DynAlign
rider
road
bench
boat
(c) DynAlign: domain-specifie & prior knowledge
(d) Example taxonomy adaptation
DynAlign and faxonomy adaptation. Current UDA methods focus solely on domain- specific knowledge transfer and assume consistent class labels across domains. limiting their flexi- bility in adapting to different taxonomies. Open-vocabulary segmentation models excel with broader taxonomies through large-scale pretraining but lack the precision of domain-specific models for spe- cialized tasks. In contrast, DynAlign integrates with any UDA model and flexibly adapts to diverse taxonomies and scene contexts, leveraging the prior knowledge of Foundational models.
Figure 1:
adaptation has been proposed, enabling UDA in settings where the target domain adopts a label space different from that of the source domain (Gong et al. 2022 Fan et al., (023a). These methods aim align differing taxonomies by leveraging relationships between source and target classes but still require prior knowledge about the target labels through annotated samples from the target domain. These challenges highlight the need for a more flexible approach that can simultaneously manage domain shifts while accommodating the target domain. Despite the significance of this problem, fully unsupervised methods that can address both domain shifts and taxonomy discrepancies remain largely underexplored.
taxonomies without requiring additional annotations from
Recent advancements in foundation models offer a promising direction to overcome these limi- tations. By leveraging large-scale pretraining diverse datasets, these models generalize across varied domains, tackling the challenges posed by limited domain knowledge and unseen categories. For instance, recent open-vocabulary semantic segmentation works (Ghiasi et al. 2022) leverage the knowledge in foundation models like CLIP (Radford et al. 2021 and Segment Any- thing (SAM) (Kirillov et al.) 2023), enabling segmentation across a broad range of unbounded class labels. However, despite their strong generalization ability, foundation models often struggle with inferior performance compared to domain-specinn models trained on in-domain datasets. Their broad focus can limit their effectiveness in specialized tasks, such street scene understanding, where fine-graines segmentation requires detailed, domain-specific knowledge.
To conclude, existing works have been tackling image-level and label-level domain gaps separately, while the intersection of these two problems remains underexplored. In this work, we propose a new benchmark of unsupervised taxonomy adaptation, addressing both image-level and label-level domain gaps without supervision, as shown in Figure To achieve this, we propose DynAlign, knowledge and rich open-world prior knowl-
novel approach that integrates both domain edge from foundation models. The framework first leverages domain-specift knowledge by aligning the data distributions of the source and target domains within the UDA paradigm. Subsequently, to incorporate prior knowledge from foundation models, DynAlign dynamically adapts to new scene contexts by retrieving in-domain predictions and extending the knowledge with foundation models to generate predictions in the new target label space, accommodating the different taxonomy in the target domain. In DynAlign, foundational model knowledge fused in three modules. First, Large Language Model (LLM) is used for semantic taxonomy mapping and context-aware descriptions. For instance, the source domain label 'road" is mapped to { ('road', 'sidewalk' 'lane marking' etc.) in the target domain, and then each label such as 'lane marking' can be further enriched with more precise descriptions like { l'traffic lane marking'. 'double lines") ") to capture the current context and improve semantic granularity. Then, SAM is employed to refine the coarse semantic masks gener- ated by the UDA model, providing more fine-graine greingr masks within precise boundaries (e.g.. seg- menting 'lane marking', 'catch basin' within the 'road' region). Lastly, a knowledge fusion mech- anism is introduced, where CLIP is leveraged to extract textual features based on the context-ane aware taxonomy provided by I.I. I.LM and reassign semantic labels to the proposed mask regions, effectively
2

---

Published as a conference paper at ICLR 2025
fusing UDA knowledge with prior knowledge. Predictions from DynAlign can be directly used or leveraged as pseudo-labels to train an offline segmentation model on the target domain, enabling instant inference without the need for target domain annotations.
Our approach operates in a fully unsupervised manner, integrating seamlessly with any UDA-based semantic segmentation model. By leveraging the prior knowledge of foundation models, it strength- ens in-domain predictions and flexibly adapts to new classes and scene contexts. When the target label set changes, only the taxonomy mapping needs to be updated instantly predict on the tar- get dataset without requiring additional training. This adaptability offers a highly flexible solution for cross-domain semantic segmentation in dynamic. real-world environments with changing tax- onomies. We conduct extensive experiments on the GTA- Mapillary Vistas and GTA + IDD street scene semantic segmentation benchmarks. The results demonstrate that our method effectively com- bines domain-specific knowledge with prior knowledge from foundation models, achieving superior performance in the unsupervised coonomy-adaptiin domain adaptation task.
RELATED WORKS
2.1 UNSUPERVISED DOMAIN ADAPTATION
Unsupervised domain adaptation (UDA) aims to minimize the domain gap and transfer knowledge from labeled source domain an unlabeled target domain. Due the ubiquity of domain gaps,
UDA methods have been widely applied to major computer vision problems including image and
video classification (Zhang et al. [2033 Lai al. 2023; Zara al.) 2023), object detection (Chen et al., 2018 Li et al. 2022bl c Fan et al. 2023b, and semantic segmentation (T'ai et al., 2018
Hoyer al. 2022a| 2023). UDA approaches typically minimize domain gaps through methods like
discrepancy minimization (Long et al., 2015), adversarial training (Ganin et al., 2016 Long ct al.)
2018 Shi & Liu 2024), self-training (Pan et al. 2019 Mei et al. (000]22222 Zhang et al. 2021) l. Recently,
foundation models have been used to further enhance the adaptation performance by leveraging
large-scale pretraining (Fahes et al. 2023
Tang et al. 2024 Gondal et al. 2024).
While traditional UDA methods assume consistent label space between the source and target do- mains, this assumption is often violated in real-world scenarios. To address this, several specialized UDA methods have been developed to handle different label shift scenarios (Tacett des Combes
et al. 2020 Garg et al. 2023) Westfechtel et al., 2023). Partial DA (Cao et al.] 2018) Guo et al.] 2022 addresses situations where the target domain contains subset of the source domain's 's classes.
Open-set DA (Saito & Saenko [202 Li et al. 2023 handles cases where the target domain includes unknown classes not present in the source domain. Universal DA (You al. 2019 Qu et al. 2024)
aims to adapt to target domains with any combination of known and unknown classes.
Typically, In the field of cross-domain semantic segmentation, various UDA approaches have been proposed to address the challenges posed by the fine-grained task.
DA (Kundu al. (020)) focuses on adding new classes while preserving knowledge of previously learned ones. Open-set adaptation methods (Bucher et al.) [021] Choe et al.] (022)) aim to predict the boundaries of unknown classes. However, these works can only distinguish between the unknown classes and the known ones and do not perform further classification on the unknown classes. Taxonomy adaptive DA (Gong et al. 2022: Fan et al. 2023a) utilizes a more flexible taxonomy mapping where the target label space differs from the source domain. Despite being able to distinguish novel target classes further, these methods cannot be applied in a fully unsupervised DA setting as they rely on few-shot labeled samples from the target domain gain knowledge novel classes. Instead, DynAlign leverage prior knowledge from foundation models for unsupervised adaptation.
2.2 OPEN-VOCABULARY SEMANTIC SEGMENTATION
oven-vocabulary semantic segmentation aims to assign semantic label to each pixel of an im- age using an arbitrary vocabulayy label set (Xian et al. 2019 Bucher et al., 2019). Recent advancements in vision-language models like CLIP (Radford et al. 2021) have enabled zero-shot classification with enhanced generalization ability from large-scale pretraining, leading to their wide application in open-vocabulary tasks, including semantic segmentation (L al. 2022a)
3

---

Published as a conference paper at ICLR 2025
To adapt CLIP for better performance in dense prediction tasks, different approaches have been explored, including fine-tuning CLIP (Liang et al. [2023 Cho et al., 2024) and modifying CLIP's model architecture (Wang et al., 2023: Yu et al. (024). For finer segmentation boundaries, two- stage frameworks involving mask proposal generation followed by classification remain prominent
2023 WW et al. 2023a Ding al. 2022). The recent advent of Segment Anything Model (SAM) (Kirillov et al. (022)) provides a free and precise mask proposal generation approach and thus integrated by several works to enhance the performance (Shi & Yang, 2024 Li et al. 2024, Yuan et al. 2024). Despite these advancements, open-vocabulary methods struggle with domain- speciffc nuances and fine-graine categories (Zhou & Beyerer 2023 Wu et al., 2023b). Wei et al. (2024) leverages vision foundation models to improve the generalization ability of task-spe semantic segmentation models. Yilmaz et al. (2024) incor- porates domain-specifie knowledge into the open-vocabulary framework through supervised prompt fine-tuning, whereas our work focuses on fully unsupervised adaptation.
in open-vocabulary semantic segmentation (Kirillov et al.
PROBLEM DEFINITION
We formulate the problem of unsupervised taxonomy adaptive cross-domain semantic segmentation as follows: given labeled source domain D, with label set Cy our goal is to achieve semantic segmentation on an unlabeled target domain Dr with a known label set C, Specifically, we have:
labeled source domain D. {x., },}, where X, E R x xx represents RGB images and
Y, denotes pixel-wise wise annotations in the source label set C, c.......... c.").
an unlabeled target domain D. = {X₂]. The ground truth pixel-wise annotations belong to the .... cr) and not available during training.
known target label space C. = {c+ -2..........
C, and Ce may have inconsistent taxonomies. This inconsistency may involve differences in label granularity, hierarchical class structures (such as subclasses), or the introduction of entirely new categories in Cr, as illustrated in
Formally: Let P. and Pt represent the distributions of source domain data X, and target domain data Xz, respectively. In the context of aaxonomy-adaptive cross-domain semantic segmentation, three primary challenges need to be addressed:
Image-level domain gap: the source and target data have distinct data distributions (P, + Pt). Label-level taxonomy inconsistency: the source and target label sets different (C, * C.).
Absence of target domain annotations: no labeled data is available for the target domain.
We aim train a model using both the labeled source domain D, and the unlabeled target domain D., and evaluate the performance on the target dataset D, within the label space Ct.
4 METHOD
Method Overview In this work, we propose DynAlign, novel framework for unsupervised txonomy-adaptive cross-domain semantic segmentation that fuses domain knowledge with text and visual prior knowledge. Our method comprises three main stages: incorporating domain knowledge, incorporating prior knowledge, and combining them through a knowledge fusion mech- anism, illustrated in Figure 2 In the first stage, integrate domain -seaiin knowledge by training a domain-specific model on the available labeled source domain and unlabeled target do- main, which generates predictions within the source label space (Section 4.1.. In the second stage, we incorporate both text & vision prior knowledge to address the label-level taxonomy inconsis- tency. Specifically. we use LLM to construct taxonomy mapping to link the source labels with the target labels and further enrich the target labels with context descriptions (see Section 4.2). Then we utilize SAM to generate mask proposals that segment the image into fine- fine-grained semantic regions and extract multi-scale visual information for each region (see Section 4.3) In the final stage, we introduce knowledge fusion mechanism (see Section 4.4).. where the domain-specific predictions and prior knowledge are integrated based on CLIP. As shown in Figure 3 for each mask proposal from SAM, we take the majority pixel label of in-domain prediction as the initial source label and retrieve the correlated target classes from the taxonomy mapping. CLIP is then used to extract the multi-scale regional visual features and the context-awand text features for the mapped target classes. Finally, each fine- fine-grained mask region is reclassified based CLIP feature similarity.
4

---

Published as a conference paper at ICLR 2025
pseudo label
Domain knowledge
UDA
road serrain terrain anlabeled unlobeled
Knowledge fusion
D
(K Val
y
vision
SAM
CLIP encoder
DOdO DODO
CLIP similarity
vision & lext
marking
LLM
road
sand
(Taxonomy
terrain
D,= (X))
Mapping)
unlabeled
mouniaii
Prior knowledge
Figure 2: DynAlign overview. DynAlign integrates with any UDA model, leveraging its domain- specific knowledge and enhancing it with prior knowledge from foundation models. DynAlign starts with coarse UDA model predictions, followed by: 1) LLM constructing taxonomy mappings to align source and target domains; 2) SAM generating fine-grained masks. CLIP is deployed to fuse the visual knowledge from SAM with the semantic knowledge from LLM to reassign accurate labels. The CLIP-fuset predictions can be used pseudo-labels to further fine-tune the UDA model.
4.1 DOMAIN-SPECIRE KNOWLEDGE
To gain domain-specific knowledge, given a labeled source domain D, and an unlabeled target do- main Dr, train domain-specific model fe-s by leveraging data from both domains. Specifically, we follow the UDA paradigm, which aligns the data distribution the source and target domains. This adapted model integrates supervised knowledge from the source domain with visual informa- tion from the target domain, enabling to generate accurate predictions on the unlabeled target dataset D, within the source label space C.,
In our experiments, we develop an UDA model inspired by the architecture proposed by Hoyer et al. (2022)). The model comprises hierarchical transformer encoder, based the design of Xie al. (021), combined with a multi-scale decoder that effectively integrates contextual information from low-level features. Initially, the model is trained using a supervised cross-entropy loss on the labeled source domain D., Then, we adapt it to the unlabeled target domain D, through an unsupervised self-training. This adaptation process incorporates a teacher network that generates pseudo- labels for the target domain, which are then weighted based on confidence estimates to account for uncer- tainty. These weighted lseudo-labels are used to further refine the model's performance on the target domain. The adapted model learns shared feature representations for both domains, enabling it to effectively handle image-level domain shifts. Detailed model architecture and training procedures provided in the Appendix I We leverage the knowledge learned by fo-.t to generate initial predictions ie for the target images within the source label set C..
4.2 SEMANTIC TAXONOMY MAPPING
The acquired Nomain--ppciife model is constrained by the in-domain knowledge, thus limiting their ability to generalize beyond the learned source label space to a new target label space. To bridge this gap, we introduce taxonomy reasoning process that adapts the label space from C, C.. This taxonomy mapping enables the model to semantically link source domain labels to the known target domain label set C., For instance, the source label 'road' can mapped more granular target domain labels such as ('road', 'sidewalk', 'curb", 'lane marking', 'rail track', etc.). J. We construct taxonomy mapping for each source label to connect the source and target label spaces. This mapping is flexibly defined, allowing for differences in label granularity, hierarchical class structures (such as subclasses), or the introduction of entirely new categories, as shown in Figure (d).
Formally, given the source domain label set C, = [cc, e e and the target domain label set C, (G., }. we define the taxonomy mapping for each source domain label eç as:
+ C. 1sis m
where Cj represents an arbitrary subset of C, that semantically correlates with the source domain label ct. For novel classes that not present in the source domain, we map them to each source
5

---

Published as a conference paper at ICLR 2025
label, enabling better discovery of these new classes. Further details about the taxonomy mapping can be found in the Appendix A.7
To extract meaningful semantic features, each target label. Original class labels often lack sufficient context and semantic richness, leading to ambiguity and imprecision. For example, the label 'bridge' in street scene segmentation datasets is too generic and might be interpreted as a bridge over a river rather than a pedestrian bridge in urban environments. This lack of specificity can cause confusion and misclassification in the segmentation process, resulting in inferior performance. To address this, we enhance the semantic clarity of each target class label by expanding it with contextually relevant synonyms or related phrases. For each target class label, we use GPT-4 (Achiam et al.) (202)) to generate a set of contextually relevant terms, such as 'bridge' - { ('road bridge', , 'Jootbrigge, 'pedestrian bridge', etc.). These terms are generated by prompting GPT-4 with descriptions of the dataset context and the original class label. More details are provided in the appendix [A.8] This additional contextual information helps reduce ambiguity and improve the accuracy of mask region classification. We then compute the context- aware text feature for each target domain label. Specifically, for each target domain label cf E Ct generate context description set C Coonteat and compute the CLIP encoded text feature for each description Ccontent E Coontert The text feature for class label cf then obtained by averaging all the contextual description features, as follows:
the CLIP text encoder to encode text features for
F = average T (Cconteztt) Ccontezt E CJ Coontett
(2)
where (-)-) represents CLIP text encoder. We then aggregate these features to form the final text feature representations of the target domain label space denoted as IF Farrge = (f.,,, For each source label c,, we retrieve its mapped target labels Ci and the corresponding feature representation set Fý- C Fargget the semantic representation.
4.3 VISUAL PRIOR KNOWLEDGE
To identify new labels in the target domain, it is essential to align visual information with the target domain label set and its novel semantic categories. Therefore, we incorporate prior knowledge from general-purpose foundation models to enhance visual understanding. We first generate mask pro- posals using the AnnteetttAntthhnn-MMaa zero-shot capability to produce high-quality, fine-grand ire-graaned masks with precise segmentation bound- aries. With SAM, we are able to obtain set of mask regions that likely correspond to semantically meaningful object boundaries.
(SAM) (Kirillov et al. (023). which is renowned for its
Next, we capture semantic visual information for each mask region by extracting visual embeddings using CLIP, which requires capturing both fine-grained details and broader contextual information. To obtain more representative features, we propose multi-scale visual feature extraction, which concatenates local and multi-scale global features. Given a target domain image Tú, and a binary mask proposal m from SAM, we obtain the masked local region r = 2z C m and bounding box b that crops the masked region. The local feature is extracted with CLIP vision encoder Φ (-), as:
Fi
(3)
Specifically, we use ConvCLIP et et al. 2024) vision encoder due to its advantage in dense predic- incorporate global features by averaging tion tasks. To capture broader contextual information,
embeddings across multiple scales. This is achieved by adding multi-scale padding around each local bounding box region b, illustrated in Figure The padding size is adjusted based on the class labels, with larger objects like 'road" assigned larger padding sizes, and smaller objects like "bicycle' assigned smaller padding sizes. For each bounding box b, we create a set of padded global regions B and extract the corresponding global feature:
FF = { V (bx)),
by B
To combine both local and global visual information, we extract the final visual feature Fy as the weighted sum of F, and F,. Specifically, we calculate the cosine similarity between the local feature

---

Published as a conference paper at ICLR 2025
Fi and each global feature y2 gg. The similarity scores OR computed follows:
(Fi) Fgg
Φk = (Fi, Pax) = [ +l Fgx E
(5)
where e small constant. Subsequently, we derive the final multi-scale visual feature Fy for the current mask region by aggregating the local feature Fi with the global features Fg. weighted by their respective cosine similarities Φx- This aggregation ensures that both local details and broader contextual information contribute effectively to the visual feature representation:
Fy Eldx Fon + (1 0&) Pi)
Fgg E gg
(6)
Fg
SAM
V
X
GLIP simitarit ity
lane-marking
mask proposals
global region
multi-scale visual fenture
sand 0.1
roal (muth road,
... mane-narhing (street line marking
FOOD oooo
mapping
traflic lone marking,
sand froadsids pand, pilod sand.
P,et,
REEC,
(axonemy mapping
context-tware Ecature
Figure 3: Foundational models and knowledge fusion. The fine-grained mask proposals from SAM are encoded into multi-scale visual features using CLIP's vision encoder, while the enriched target domain taxonomies from LLM are encoded as context-aware text features via CLIP's text encoder. The similarity between these visual and text embeddings is then calculated to reassign semantic taxonomies accurately to the fine-grained masks in the target domain. Here, Φ (-) and (-) denote the CLIP vision and text encoders, respectively. Fi and F, represent the local and global features, while Fy denotes their weighted sum, forming the final extracted multi-scale visual feature to represent the mask region. Fr refers to the extracted text feature set of candidate classes.
4.4 KNOWLEDGE FUSION
Given the domain-specifie knowledge and prior knowledge from foundation models, aim to fuse them to bridge both image-level and label-level domain gaps. The general fusion process of DynAlign is shown in Figure
Given a target image Z4, we generate mask proposals that segment the image into fine-grained re- gions using SAM. For each mask region m, first associate with the source domain label space by retrieving in-domain knowledge from the label prediction Je generated by model fs-t (as de- scribed in Section 4.1) The initial label ot for mask region m is determined by the majority of predicted pixel labels within that mask area. To reassign each mask region with new target labels, given the source label ct, we retrieve its related target domain label subsets of - Cj S C, and the corresponding feature representation set Fr S F target using prior knowledge from the taxonomy mapping (see Section 4.2). Additionally, based on the initial label c,, we adjust the appropriate padding size of the global region B (see Section 4.3) accordingly
enrich the visual context and extract the multi-scale visual feature Fy, serving as the regional visual feature representation. Fi- nally, each mask region is reassigned a new target label based on the largest similarity between the context-awee awrrr text features Fr and multi-scale visual feature Fy as:
Fr). T. argmax(())
We reassign the label of each mask region from de to the newly estimated ÿe when the confidence exceeds 0.5. This reassignment is applied every mask proposal within the image sample It, thereby updating the pixel-wise semantic pseudo-labels in the target domain.
DynAlign generates predictions for classes in the target domain that were not present in the source domain, eliminating the need for manual annotations. This capability allows the model to adapt flexibly target classes and when the target label set changes, simply redefine the

---

Published as a conference paper at ICLR 2025
taxonomy mapping to generate new predictions for the target domain dataset. This two-step process- mapping known classes and discovering new ones enables our framework to effectively adapt to the target domain's 's taxonomy in fully unsupervised manner.
EXPERIMENT
5.1 IMPLEMENTATION DETAILS
Datasets We evaluate our method using the synthetic dataset GTA as the source domain and two real- world datasets, Mapillary Vistas and India Driving Dataset (IDD), as target domains. GTA (Richter) al., 2016) is synthetic dataset with pixel-level annotations for 19 semantic classes. It serves as the labeled source domain in our experiments. Mapillary Vistas (Neuhold et al. (017) contains 25k high-resolution images collected from a wide variety of environments, weather conditions, and geographical locations. is annotated with 66 object categories, providing highly challenging real- world target domain. IDD dataset ( (Varma et al. 2019), captured from Indian urban driving scenes, is characterized by complex and varied road environments. It contains 10,003 images annotated with 25 semantic classes at level granularity. For evaluation, we focus on 45 classes from Mapillary Vistas and 24 from IDD, excluding small-scale or less informative categories.
Experimental setup For the unsupervised laxonomy-adaptive DA task, we use the GTA training set as the labeled source domain and the training sets of Mapillary Vistas and IDD as the unlabeled target domains, No target domain annotations are used during training, ensuring fully unsuper- vised domain adaptation setup. We perform image-level adaptation by training the model fa on the labeled source domain and each of the unlabeled target domains separately, following the de- fault training parameters in Hoyer al.) (022b). For label-level adaptation, we define a taxonomy mapping between the source and target labels (see Appendix(A.7] and context descriptions for target labels (see Appendix (A.)). Performance reported on the validation set of each target dataset. We employ the ViT-L SAM model (Kirillov et al. 2023) and the ConvNeXt-Large CLIP model (Yu ct al. 2024) by default. More experimental details are provided in the appendix A.
Evaluation metries We evaluate the performance using two standard semantic segmentation met- rics. Mean Intersection over Union (mloU) calculates the average intersection over union for each class and then averages across all classes. Mean Aceuracy (mAce) calculates the percentage of correctly predicted pixels for each class and then averages the results across all classes.
5.2 EXPERIMENTAL RESULTS
To the best of our knowledge, our work is the first to investigate the fully unsupervised taxonomy- adaptive domain adaptation problem. Traditional DA methods are limited to fixed label spaces, making them inapplicable to our problem setting. Therefore, we compare our proposed method, DynAlign. with open-vocabulary segmentation methods, Grounded-SAM (Ren et al., 2024) and OWL-VIT (Matthias Minderer (023). Additionally, we evaluate the naive combination of these methods with HRDA, where HRDA generates predictions for classes matching the source labels, while open- vocabulary methods are employed to predict new classes (see Appendix A.3 for more illustration). The comparative results are summarized in Table
Table I: Open-vocabulary semantic segmentation comparisons on Mapillary Vistas and IDD
Mapillary Vistas
IDD known mACC mInU
Methods
all mACC mioU
known mACC mloU
unknown
all mACC
unknown mACC mloU
mACC mloU
mIoU
Crounded-SAM
33.1
28.6
46.7
43.0
24.0
18.3
36.0
30.8
43.7
37.2
16.1
14.1
OwIVIT-SAM
29.6
19.5
32,7
26.7
27.5
14.7
31.6
20.9
33.3
23.1
27.3
15.2
HRDA
75.8
65.8
68.9
61.3
IIRDA
Grounded-SAN
40.4
32.9
63.6
53.1
25.0
19.4
51.7
39.4
65.6
49.3
16.1
14.1
IIRDA OWIVIT-SAM
40.2
28.8
56.1
48.4
29.5
15.6
55.9
40.2
67.1
49.9
27.3
15.2
Ours
53,0
36.7
72.6
62.4
39.9
19.6
57.7
41.7
66.8
50.9
34.3
18.1
On the Mapillary dataset, vocauuucyy methods, which relies solely on foundation models, demonstrate limited performance. By integrating domain-specific knowledge through HRDA,
8

---

Published as a conference paper at ICLR 2025
the performance improves, with HRDA + Grounded-SAM reaching 32.9% and HRDA + OWL- ViT-SAM achieving 28.8% mloU. Notably, our method, DynAlign significantly outperforms all baselines, achieving the highest mloU of 36.7%. Results on IDD show consistent trends. Open-vocabulary methods alone perform modestly with mloUs 30.8% for Grounded-SAM and 20.9% for OWL.VVIT-SAM. When combined with HRDA, performance improves 39.4% and 40.2%, respectively. DynAlign demonstrates again superior performance with the highest mloU of 41.7%. Notably, for unknown classes, DynAlign achieves an mloU of 18.1%, surpassing HRDA + Grounded-SAN, at 14.1% and HRDA + OWL-ViT-SAM at 15.2%, showing a substantial im- provement in accurately segmenting unseen categories. The results demonstrate that incorporating domain-specific knowledge through UDA significantly enhances performance, even when leverag- ing strong foundation models for open-vocabulary tasks. DynAlign not only preserves the domain knowledge from the UDA baseline but also retains the flexibility to accommodate new classes. This highlights its advantages in effectively integrating both domain knowledge and prior knowledge, highlighting its advantage over traditional UDA methods and open-vocabulary approaches.
We present qualitative comparison results on the Mapillary Vistas dataset in Figure [] The key new classes highlighted under each set of results. Our method clearly outperforms the baselines by producing more precise boundaries, better detection new classes, and more accurate predictions.
5.3 PSEUDO-LABEL, TRAINING FOR SEGMENTATION MODEL
While DynAlign demonstrated superior perfor- mance over baseline open-vocabulary methods during inference, also explored its capabil- ity to generate pseudo-labels for training new segmentation model the target domain's tax- onomy. This approach aims to reduce infer- ence time and potentially boost segmentation accuracy. By generating pseudo-labels on the Mapillary Vistas training set using DynAlign we train a semantic segmentation model in the new label space (details provided Ap- pendix I.I). Figure E compares direct infer- ence using DynAlign and performance of the trained segmentation model on the validation set. The pseudo-label trained model outper- forms direct inference, with mACC improving from 53.0% to 56.2% and mloU from 36.7% to 38.9%. These results suggest that DynAlign can generate high-quality pseudo-labels, en- abling the training of segmentation models that can greatly boost the efficiency and maintain im- proved performance on the target domain. We report the memory usage and computational efficiency of DynAlign in both direct inference and pseudo-labeling the appendix
56.2
DynAliga (Düreet Inference)
DynAligs
Percentage
mAGC
mioU
Performance comparison between di- rect inference and pseudo-label training using Dy- nAlign on the Mapillary Vistas dataset.
Figure 4:
A.2 A.2
Table Ablations on multi-scale visual
Table 3: Ablations CLIP vision coders
feature and context-aware text feature
Modules
Mapillary Vistas
IDD
model
Mapillary Vistas mloU
IDD
MS (vision)
CA (text)
mACC
mloU
mACC
mloU
mACC
mACC
mloU
44.0
29.0
52.9
37.9
CLIP
50.8
35.0
54.7
38.0
45.4
29.4
52.5
MaskCLIP
50.5
34.7
54.1
37.5
50.2
35.6
$7.0
39.9
SCLIP
50.9
36.0
56.8
39.4
53,0
36.7
57.7
41.7
ConvCT.IP
53.0
36.7
57.7
41.7
5.4 ABLATION STUDIES
Multi-scale visual feature & context-aware text feature ablation In contrast to the basic CLIP features, enhance feature representations by introducing context-aware text feature (CA-Text) and multi-scale visual feature (MS-vision). As shown Table these two strategies have distinct impacts the performance. Incorporating multi-scale visual features improves the Mapillary mloU

---

Published as a conference paper at ICLR 2025
from 29.0% 35.6% and the IDD mloU from 37.9% to 39.9%. The context-aware text feature shows modest gains, particularly on Mapillary, where mloU rises from 29.0% to 29.4%. Combining both components yields the best results (Mapillary mloU: 36.7%, IDD mloU: 41.7%)
DynAlign modules. Table compares the performance various CLIP backbones in DynAlign framework. ConvCLIP Yu al. 2024), the only convolution- based model, outperforms ViT-based models including CLIP (Radford et al.) 2021), MaskCLIP (Dong et al. 2023), and SCLIP (Wang al. 2023), and achieves the best results, with Mapillary Vistas mloU 36.7% and IDD mloU of 41.7%. The result shows its advantage in dense prediction tasks when input sizes scale up.
Thole shows the performance the Mapillary Vistas dataset when replacing corresponding modules DynAlign with MIC (Hoyer al., 2023). Mobile- SAM (Zhang et a and Llama (Touvron et al., 2023). The results demonstrate consistently strong perfor- mance across various model integrations, emphasiz- ing DynAlign 's flexibility in seamlessly incorporating foundation models into domain- -ppecii tasks.
Table 4: Ablations on DynAlign modules
Method
mIoU
mAec
IIRDA > MIC
37.7
54,4
SAM MobileSAM
35.2
51.5
GPT4 > Llama
37.5
52,1
DynAlign (Ours)
36.7
53.0
road
sidewalk
vegetation
terniin building
Hall unlabeled
person
rraffic sign
traffic light
pole
lane-marking general
lame-marking crosswalk
billboard
street light
afility pole
lane-marking general www.wa.ww.......................
street Right
curb
lune-morking general
light
bridge
manhole
utility pale
(d) HRDA
(c) HRDA
(a) Image
(b) Ground Truth
DynAlign (ours)
Grounded-SAM
OWIVIT-SAM
Figure 5: Qualitative comparisons on Mapillary Vistas dataset. DynAlign effectively segments new and fine fine-graines grained classes on the target domain, showing strong taxonomy adaptation capabilities.
CONCLUSION
In this paper, propose DynAlign address the challenge of unsupervised taxonomy-adaptive cross-domain semantic segmentation, effectively segmenting images across domains with differ- taxonomies without requiring target domain annotations. DynAlign integrates domain- specific knowledge by utilizing UDA models to bridge the image-level domain gap, and leverages foundation models to resolve label-level taxonomy inconsistencies between domains. The approach demon- strates sigmificant improvements over existing methods on the Mapillary Vistas and IDD street scene datasets, consistently achieving higher mloU scores for both known and unknown classes.
To the best of our knowledge, we are the first to define and address the unsupervised taxonomy- adaptive domain adaptation problem. Our results demonstrate that DynAlign outperforms not only open-vocabulary segmentation methods but also their naive combinations with domain adaptation techniques. While our research focuses on road scene understanding, the framework has poten- tial to be extended to other domains with evolving taxonomies.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. zero-shot semantic segmenta- tion. Advances in Neural Information Processing Systems, 32, 2019.
Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Handling new target classes in semantic segmentation with domain adaptation. Computer Vision and Image Understanding, 212: 103258, 2021.
Hu Cao. Yueyue Wang. Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang. Swin-unet: Unet-like pure transformer for medical image segmentation. In European conference on computer vision, pp. 205-20 218. Springer, 2022.
Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Michael 1 Jordan. Partial transfer learning with selective adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2724-2732, 2018.
Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, and Lue Van Gool. Domain adaptive faster r-cnn for object detection in the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pp.
2018.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1290-1299, 2022.
Scokju Cho. Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, and Seungryong
Kim. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4113-4123, 2024.
Seun-An Choe, Ah-Hyung Shin, Keon-Hee Park, Jinwoo Choi, and Gyeong-Moon Park. Open-set domain adaptation for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23943-23953 2024.
Zheng Ding, Jieke Wang, and Zhuowen Tu. Open-vocabulle universal image segmentation with maskelip. arXiv preprint arXiv:2208. 08984, 2022.
Xiaoyi Dong, Jiammin Bao, Yinglin Zheng, Ting Zhang, Dongdong Chen, Hao Yang. Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Maskelip: Masked self- distillation advances contrastive language-imag pretraining. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 10995-11005, June 2023.
Mohammad Fahes, Tuan-Hung Vu, Andrei Bursuc, Patrick Pérez, and Raoul De Charette. Poda: Prompt-driven zero-shot domain adaptation. In Proceedings of the IEEF/CVE International Con- ference Computer Vision, pp. 18623-18633, 2023.
Jianan Fan, Dongnan Liu, Hang Chang, Heng Huang, Mei Chen, and Weidong Cai. Taxonomy adaptive cross-domain adaptation in medical imaging via optimization trajectory distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21174-21188 184, 2023a.
Qi Fan, Mattia Segu, Yu-Wing Tai, Fisher Yu, Chi-Keung Tang, Bernt Schiele, and Dengxin Dai. To- wards robust object detection invariant to real-world domain shifts. In The Eleventh International Conference on Learning Representations (ICLR 2023). OpenReview, 2023b.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Donain-adversarial training of neural networks. Journal of machine learning research, 17(59):1-35, 2016.
Saurabh Garg, Nick Erickson, James Sharpnack, Alex Smola, Sivaraman Balakrishnan, and Zachary Chase Lipton. Rlsbench: Domain adaptation under relaxed label shift. In International Conference on Machine Learning, pp. 10879-119228. PMI.R, 2023.
11

---

Published as a conference paper at ICLR 2025
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Scaling open- vocabulayy image segmentation with image-level labels. In European Conference on Computer Vision, pp. 540-557. Springer, 2022.
Muhammad Waleed Gondal, Jochen Gast, Inigo Alonso Ruiz, Richard Droste, Tommaso Macri, Suren Kumar, and Luitpold Staudigl. Domain aligned clip for few-shot classification. In Pro- ceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5721- 5730, 2024.
Rui Gong, Martin Danelljan, Dengxin Dai, Danda Pani Paudel, Ajad Chhatkuli, Fisher Yu, and Luc Van Gool. Tacs: Taxonomy adaptive cross-domain semantic segmentation. In European Conference on Computer Vision, pp. 19-35. Springer, 2022.
Pengxin Guo, Jinjing Zhu, and Yu Zhang. Selective partial domain adaptation. In BMVC, pp. 420, 2022.
Lukas Hoyer, Dengxin Dai, and Lac Van Gool. Daformer: Improving network architectures and training strategies for domain-adaptive semantic segmentation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 9924-9935, 2022a.
Lukas Hoyer, Dengxin Dai, and Luc Van Gool. Hrda: Context-aware high-resolution domain-
adaptive semantic segmentation. In European conference on computer vision, pp. 372-391. Springer, 2022b.
Lukas Hoyer, Dengxin Dai, Haoran Wang, and Luc Van Gool. Mie: Masked image consistency for context-enhanced domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11721-11732, 2023.
Jitesh Jain, Jiachen Li, Mang Tik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. Oneformer:
One transformer to rule universal image segmentation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 2989-2998, 2023.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yes Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pp. 4015 4026,402,220 2023.
Jogendra Nath Kundu, Rahul Mysore Venkatesh, Naveen Venkat, Ambareesh Revanur, and R Venkatesh Babu. Class-incremental domain adaptation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIII 16, pp- 53-69. Springer, 2020.
Zhengfeng Lai, Noranart Vesdapunt, Ning Zhou, Jun Wu, Cong Phuoc Huynh, Xuelu Li, Kah Kuen Fu, and Chen-Nee Chuah. Padclip: Pseudo-labeling with adaptive debiasing in clip for unsuper- vised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16155-16165, 2023.
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language-driven semantic segmentation. arXiv preprint arXiv:2201.0355 2022a.
Wuyang Li, Xinyu Liu, and Yixuan Yuan. Sigma: Semantic-complete graph matching for domain adaptive object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5291-5300, 2022b.
Wuyang Li, Jie Liu, Bo Han, and Yixuan Yuan. Adjustment and alignment for unbiased open set domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24110 11110-22111 24119, 2023.
Xiangtai Li, Haobo Yuan, Wei Li. Henghui Ding, Size Wu, Wenwei Zhang, Yining Li. Kai Chen, and Chen Change Loy. Omg-seg: Is one model good enough for all segmentation? In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 219949977999, 2024.
Yu-Jhe Li, Xiaoliang Dai, Chih-Yar Ma, Yen- Yen-Cheng Liu, Kan Chen, Bichen Wu, Zijian He. Kris Kitani, and Peter Vajda. Cross-domane adaptive teacher for object detection. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 7581 7581-7590 2022c.
12

---

Published as a conference paper at ICLR 2025
Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open- vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7061-7070, 2023.
Mingsheng Long. Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pp. 97-105, PMLR, 2015.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. Advances in neural information processing systems, 31, 2018.
Neil Houlsby Matthias Minderer, Alexey Gritsenko. Scaling open-vocabulary object detection. NeurlPS, 2023.
Ke Mei, Chuang Zhu, Jiaqi Zou, and Shanghang Zhang. Instance adaptive self-training for unsuper- vised domain adaptation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, Springer. 2020.
UK, August 23-28. 2020, Proceedings, Part XXVI 16. pp.
Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In Proceedings of the IEEE international conference on computer vision, pp. 4990-4999, 2017.
Yingwei Pan, Ting Yao, Yehao Li, Yu Wang, Chong-Woor Wah Ngo, and Tao Mei. Transferrable prototyp- ical networks for unsupervised domain adaptation. In Proceedings of the IEEF/CVE conference computer vision and pattern recognition, pp. 2239-2247 2019.
Sanqing Qu, Tianpei Zou, Lianghua He, Florian Röhrbein, Alois Knoll, Guang Chen, and Changjun Jiang. Lead: Learning decomposition for source-free universal domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23334-23343, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8766 8763. PMLR, 2021.
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open- open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14158 2024.
Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In Computer Vision-ECCv 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016. Proceedings, Part II 14, pp. 102-118. Springer, 2016.
Kuniaki Saito and Kate Saenko. Ovanet: One-vs-all network for universal domain adaptation. In Proceedings of the ieee/cvf international conference on computer vision, pp. 9000-9009, 2021.
Cheng Shi and Sibei Yang. The devil is in the object boundary: towards annotation free instance segmentation using foundation models. arXiv preprint arXiv:2404.11957 2024.
Lianghe Shi and Weiwei Liu. Adversarial self-training improves robustness and generalization for
gradual domain adaptation. Advances in Neural Information Processing Systems, 36, 2024.
Remi Tachet des Combes, Han Zhao, Yu-Xiang Wang. and Geoffrey Gordon. Domain adaptation with conditional distribution matching and generalized label shift. Advances in Neural Informa- fion Processing Systems, 33: 19276-9999,,, 2020.
Song Tang, Wenxin Su, Mao Ye, and Xiatian Zhu. Source-free domain adaptation with frozen multimodal foundation model. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2371 1-32200 2024.
13

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 21X1::2302.1377,, 2023.
Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming- Hing--suann Yang, and Manmohan Chandraker. Learning to adapt structured output space for semantic segmentation. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 7472-7481, 2018.
Girish Varma, Anbumani Subramanian, Anoop Namboodiri, Manmohan Chandraker, and CV Jawa- har. Idd: A dataset for exploring problems of autonomous navigation in unconstrained envi- ronments. In 2019 IEEE winter conference on applications of computer vision (WACV), pp. 1743-1751. IEEE, 2019.
Feng Wang, Jieru Mei, and Alan Yuille. Sclip: Rethinking self-attention for dense vision-language inference. arXiv preprint arXiv:2312.01597, 2023.
Zhixiang Wei, Lin Chen, Yi Jin, Xiaoxiao Ma, Tianle Liu, Pengyang Ling, Ben Wang, Huaian Chen, and Jinjin Zheng. Stronger fewer & superior: Harnessing vision foundation models for domain generalized semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28619-28630, 2024.
Thomas Westfechtel, Hao-Wei Yeh, Qier Meng, Yusuke Mukuta, and Tatsuya Harada. Backprop in- duced feature weighting for adversarial domain adaptation with iterative label distribution align- ment. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 392 2023.
Jianzong Wu, Xiangtai Li, Henghui Ding, Xia Li, Guangliang Cheng, Yunhai Tong, and Chen Change Loy. Betrayed by captions: Joint caption grounding and generation for open vo- cabulary instance segmentation.
Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21938 21944,, 2023a.
Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, and Chen Change Loy. Clipself: Vision transformer distills itself for open-vocabulary dense prediction. arXiv preprini arXiv:2310.0144 2023b.
Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata. Semantic pro- jection network for zero-and few-label semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 8256-8265, 2019.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems, 34:12077 2021.
Gonca Yilmaz, Songyou Peng, Francis Engelmann, Marc Pollefeys, and Hermann Blum. Opendas: Domain adaptation for open- vocabulary segmentation. arXiv preprint arXiv:2405.20141, 2024.
Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Universal domain adaptation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 2720-2729, 2019.
Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chen. Convolutions die hard: vocabuauy segmentation with single frozen convolutional clip. Advances in Neural Infor- mation Processing Systems, 36, 2024.
Haobo Yuan, Xiangtai Li, Chong Zhou, Yining Li, Kai Chen, and Chen Change Loy. Open- vocabulary sam: Segment and recognize twenty-thousan classes interactively. arXiv preprint arXiv:2401.02955. 2024.
Giacomo Zara, Subhankar Roy, Paolo Rota, and Elisa Ricci. Autolabel: Clip-based framework for open-set video domain adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11504-1151,, 2023.
C Zhaang D Han, Y Qiao, JU Kim, SH Bae, Lee, and CS Hong. Faster segment anything: Towards lightweight sam for mobile applications. arxiv 2023. arXiv preprint arXiv:2306. 14289.
14

---

Published as a conference paper at ICLR 2025
Pan Zhang, Bo Zhang, Ting Zhang, Dong Chen, Yong Wang, and Fang Wen. Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation. Pro- ceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 12414- 12424, 2021.
Yixin Zhang, Zilei Wang, Junjie Li, Jiafan Zhuang, and Zihan Lin. Towards effective instance dis- crimination contrastive loss for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11388-1 1399, 2023.
Jingxing Zhou and Jürgen Beyerer. Category differences matter: A broad analysis of inter-category error in semantic segmentation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp.
June 2023.
15

---

Published as a conference paper at ICLR 2025
A
APPENDIX
OVERVIEW
The supplementary material presents the following sections to strengthen the main manuseript:
Sec. A.1
includes more implementation details.
Sec.
A.2
presents memory usage and computational efficiency comparison.
provides examples and further explanation of naive combination of HRDA with open-vocabulary approaches, as mentioned in Table
Sec.
A.3
open-
Sec.
A.4
presents ablations CLIP confidence thresholding.
Sec.
A.5
provides additional experimental results under different DA settings,
Sec.
A.6
provides detailed per-class experimental results in Table |]
Sec. A.7
and Sec. A.8 provides details of building taxonomy mapping and context descrip-
tions for each target label.
A.1 MODEL ARCHITECTURE AND TRAINING
HRDA for UDA segmentation model For the UDA segmentation model, we follow HRDA (Hoyer ct al., 2022b). We follow the HRDA framework, which combines large low-resolution (LR) crops for capturing scene context with small high-resolution (HR) crops for fine detail. These two types of inputs are combined using a learned scale attention mechanism, which enables the model to effectively handle multi-resolution information. In addition, HRDA uses overlapping slide inference to refine the generated pseudo-labels ensuring that the context and details of the images are well- represented in the segmentation maps. The HRDA framework builds on the DAFormer (Hoyer al.) 2022a) architecture, which incorporates a domain-robust transformer backbone to extract features gelf-training strategy with teacher-student model is used, where the teacher generates pseudo-labels for the target domain, and these pseudo-label are weighted based on a confidence score to prevent error accumulation during training. The teacher model is updated using an exponential moving average (EMA) of the student model's 's weights, ensuring stable pseudo- labels over time. In our experiments, we follow the default training parameters in the GTA CityScapes setting and adapt from the labeled GTA dataset to the unlabeled Mapillary Vistas/IDD respectively. The labeled GTA training set and unlabeled Mapillary Vistas/IDD training set are used train the UDA Framework.
and perform segmentation.
Mask2Former for target domain segmentation To train a domain-specific model on the target domain, utilize the Mask2Former architecture. The model comprises a backbonc network that extracts low-resolution features, pixel decoder that upscales these features, and a transformer de- coder for processing object queries. It deploys masked attention, which focuses local regions of the predicted mask, improving convergence and handling small objects. In our experiment, we generate pseudo-labels on the target training set using DynAlign and train Mask2Former with these labels on the target dataset to improve segmentation performance.
All experiments are conducted NVIDIA A A100-SX44-80GB GPU.
A.2 COMPUTATIONAL EFFICIENCY
Our method comprises three main components: the UDA semantic segmentation model (HRDA), SAM, and CLIP. For semantic segmentation on high-resolution images, the memory usage primarily depends on the semantic segmentation model (HRDA), while the inference time influenced by the reassignment of novel class labels. We provide detailed information memory usage, inference time, and model parameters in table
The total model parameter count of our method includes the UDA model parameters, plus 308M for SAM (ViT-L) and 351M for CLIP (ConvNeXt-Large) The reported total memory usage corresponds to the memory allocated during the complete inference process for single image. Overall,
16

---

Published as a conference paper at ICLR 2025
Table Computational efficiency and memory usage of DynAlign
Model
Memory Usage
Inference Time
Model Parameters
HRDA
23.9
86 M
Ours (HRDA)
34.8 GB
64.1 s
668
DAFormer
10.5 GB
0.7
86 M
Ours (DAFormer)
21.7
64.1
668
Ours (pscudo-label, HRDA)
23,9 GB
1.9
86 M
Ours (pscudo-lahel, DAFormer)
10.5 GB
0.7
86 M
Ours (pseudo-label, Mask2Former)
3.7 GB
framework requires reasonable computational resources. Moreover, the UDA framework can be flexibly substituted to accommodate memory limitations. As shown in the table, replacing HRDA with Fororr significantly reduces memory consumption. For efficient inference, a segmentation model can also be trained using pseudo-labels generated by our framework, described in *Section 5.3 For example, a trained Mask2Former model using our inference pseudo labels, indicated as Ours (pseudo-label, Mask2Former), achieves inference just 0.3 seconds per single image, offering significant efficiency boost while maintaining high performance.
A.3 EXAMPLES OF NAIVE COMBINATION OF HRDA WITH OPEN-VOCABULARY APPROACHES
For naive combination of in-domain predictions with prior knowledge as baseline methods, we take the in-domain segmentation results produced by HRDA and layer them with the new class predictions from open-vocabulary models. As demonstrated in Figure 66 the open-vocabulary pre- dictions for known classes in the source domain label space are discarded. This ensures that only novel class predictions from the open-vocaleeeee vocabulary model are added on top of the HRDA predictions, preserving in in-oomiin knowledge from HRDA.
(n) Image
(b) HIRDA
(c) Grounded-SAM
HIRDA Groundod-SAM
Figure 6: Illutration on HRDA+Groundddd-ANN baseline
A.4 ABLATIONS ON CLIP CONFIDENCE THRESHOLD
In our experiments, we set the confidence threshold to 0.5 by default for reassigning the class label. We conduct ablation studies with varying confidence thresholds, as presented in Table The results show that while the mAcc improves with lower confidence threshold for assigning new class labels, the mloU may decrease correspondingly. Overall, DynAlign demonstrates robustness and is not very sensitive the confidence threshold within certain value range.
17

---

Published as a conference paper at ICLR 2025
Table 6: Ablations on CLIP confidence threshold
Threshold
0.4
0,5
0.6
0.7
0.8
0,9
mloU
35.9
36.7
37.6
36.5
36.6
36.5
mAcc
54.3
53.0
52.1
50.3
47.3
46.5
A.5 EXPERIMENTS UNDER MORE DA SETTINGS
To validate the effectiveness of our methods in diverse practical scenarios, we implement additional experiments under the following settings: 1) Traditional UDA Setting: In this case, the target label space is identical to the source label space. We adapt from GTA to Cityscapes, both using the same label space. 2) Fine-to-Coarse Setting: Here, the target domain labels are coarser than the source domain labels. For this, we adapt from Mapillary (45 classes) to Cityscapes (19 classes). As shown in Table 7 in both the traditional UDA and fine-to-coarse settings, where sufficient domain knowledge about the label space is available, our framework preserves the in-domain performance and provides slight improvements when adequate in-domain knowledge is already present.
In Table we present results under taxonomy- adapiive domain adaptation with smaller label- space domain gap. Specifically, we adapt from Synthia (16 classes) to Cityscapes (19 classes). These results highlight that our method only maintains strong performance on known classes but also predicts effectively for unknown classes.
Table Semantic segmentation comparison on Mapillary under traditional UDA and fine-to-coarse setting
Setting
mloU
mAcc
GTA to Cityscapes (HRDA, 19 -> 19)
74.9
82.0
GTA to Cityscapes (Ours. 19 19)
75.9
83.7
Mapillary to Cityscapes (Ours, 45 - 19)
70.4
81.5
Table 8: Semantic segmentation comparison Mapillary under coarse-to-fine setting with small label domain gap
Settings
all classes
known classes
unknown
mloU
mAce
mloU
mAce
mloU
mAce
Synthia to Cityscapes (HRDA, 16 16)
NA
N/A
66.8
73.7
N/A
N/A
Synthia to Cityscapes (Ours, + 19)
61.6
72.7
77.5
23.2
47.0
A.6 PER-CLASS EVALUATION RESULTS
For experimental results in Table we provide the corresponding per-class evaluation in Table
and Table 10
A.7 TAXONOMY MAPPING
We provide the detailed taxonomy mapping that maps each source label to its correlated target labels TT and Table 12, We utilizes GPT-4 to generate an initial proposal for potentially correlated taxonomy mappings between the source and target domains and introduce human intervention to refine these mappings due to the differing definitions of classes across datasets. For example, in the Mapillary dataset, the term "guard rail" specifically refers to roadside guard rails, whereas in other datasets or in general usage, may represent broader guarding fence. For the same reason, we exclude labels with inherent ambiguity, which we refer to classes that are broad highly context- dependent. (For example, the class "other vehicles" may encompass various types of vehicles that not explicitly defined in the dataset, making precise labeling challenge. Similarly, the class "ego vehicle" refers specifically to the vehicle equipped with the camera, which often appears only partially in the image.) We examine the generated taxonomy mapping ensure that the mappings
18

---

Published as a conference paper at ICLR 2025
Table Per-class semantic segmentation evaluation results on Mapillary. (lane marking is refered as Im).
Grounded- SAM
Grounded- SAM HRDA
OwIVIT SAM
OwIVIT-SAM + HRDA
Ours
IoU
Acc
IoU
Acc
IoU
Acc
IoU
Acc
loU
Acc
curb
1.6
1.6
1.6
1.6
18.7
27.0
18.7
27.0
16.3
44.9
fence
30.3
33.3
40.0
48.1
18.9
22.3
32.9
37.8
45.6
53.2
guard rail
21.3
21.3
28.8
36.1
17.9
36.1
15.3
21.1
wall
1.3
1.4
35.5
55.2
11.4
19.3
30.4
41.8
39.3
59.0
rail track
10.7
12.0
10.7
12.0
2.2
4.3
2.2
4.3
17.0
18.1
road
81.9
93.7
79.0
93.5
54.0
58.6
58.5
65.9
70.2
77.1
sidewalk
33.0
34.6
42.7
64.0
32.2
43.0
40.3
53.7
27.8
33.3
bridge
41.7
41.7
46.8
30.7
30.7
43.6
20.8
23.3
building
70.1
81.6
80.3
94.5
44.0
52.8
71.8
81.1
76.5
85.6
tunnel
0.0
0.0
0.0
0.0
1.2
3.6
1.2
3.6
13.4
79.5
person
67.9
78.6
74.9
85.6
29.5
33.3
69.0
78.3
77.7
89.3
bicyclist
8.3
9.4
21.6
36.4
10.6
12.6
35.1
66.8
53.8
72.0
motorcyclist
13.9
32.1
13.9
32.1
9.9
9.9
11.7
51.6
58.3
crosswalk
20.3
20.3
24.4
8.9
14.8
8.9
14.8
20.0
Im general
1.6
1.6
1.6
1.6
12.4
19.8
12.4
19.8
18.0
41.8
mountain
29.1
32.9
29.1
32.9
16.1
37.8
16.1
37.8
43.8
56.5
sand
29.6
35.7
29.6
35.7
24.1
38.1
24.1
38.1
18.4
49.2
sky
91.7
92.9
94.8
99.0
69.0
70.5
73.2
76.1
95.1
97.8
snow
34.1
34.1
34.5
26.9
26.9
38.9
40.9
77.5
terrain
12.3
12.5
48.1
93.1
0.3
0.3
47.6
82.7
48.5
84.4
vegetation
28.0
28.3
81.2
85.3
19.1
20.3
72.3
75.5
82.0
85.7
water
81.1
81.7
81.1
81.7
72.2
87.9
72.2
87.9
47.3
56.0
banner
5.3
5.4
5.3
5.4
6.8
10.0
6.8
10.0
0.5
0.7
bench
44.3
48.7
44.3
48.7
23.7
23.7
46.4
1.2
78.9
billhoard
12.9
15.1
12.9
15.1
15.3
15.3
32.1
28.1
55.8
catch basin
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
8.9
18.1
manhole
19.3
19.8
19.3
19.8
20.4
40.5
20.4
40.5
23.7
25.3
phone booth
12.8
26.6
12.8
26.6
1.9
35.7
1.9
35.7
0.8
57.8
street light
8.0
46.4
8.0
46.4
3.0
32.6
3.0
32.6
15.2
21.2
pole
22.4
33.0
43.6
15.5
16.6
29.1
37.2
41.9
57.6
traffic sign frame
0.4
0.9
0.4
0.9
3.2
7.6
3.2
7.6
0.1
0.1
utility pole
27.0
31.1
27.0
31.1
23.2
29.5
23.2
29.5
23.8
25.2
traffic light
60.3
64.5
56.4
65.1
26.2
28.1
53.5
61.0
59.8
73.0
traffic sign (back)
0.0
0.0
0.0
0.0
7.8
19.9
7.8
19.9
3.4
S.1
traffic sign (front)
30.7
32.5
40.4
43.1
29.2
32.0
45.8
48.1
61.8
66.8
trash can
48.5
49.8
48.5
49.8
28.0
42.9
28.0
42,9
15.0
19.4
bicycle
57.1
63.6
49.9
54.3
28.1
58.1
49.0
53.3
57.1
62.9
boat
40.0
62.9
40.0
62.9
12.4
63.8
12.4
63.8
7.4
12.8
bus
76.0
81.7
60.9
67.1
39.7
60.6
51.8
56.7
68.3
73.1
62.1
64.5
65.1
68.1
26.9
34.6
49.6
51.7
88.1
92.0
caravan
0.0
0.0
0.0
0.0
0.0
0.0
0.2
24.1
82.6
motorcycle
49.2
$2.8
53.2
61.4
37.2
39.7
54.9
62.9
65.1
76.6
rails
0.0
0.0
9.0
10.2
0.0
0.0
28.1
30.4
53.3
$9.3
trailer
0.0
0.0
0.0
0.0
0.1
5.7
0.1
5.7
1.4
20.0
truck
0.0
0.0
11.5
13.5
0.0
0.0
14.3
16.5
65.0
80.4
Average
28.6
33.1
32..9
40.4
19.5
29.6
28.8
40.2
36.7
53.0
accurate and contextually appropriate. The taxonomy mapping significantly reduces effort com- pared to tasks such as pixel-wise manual annotation and only needs to defined once per dataset pair. We highlight the identical classes in source and target domains in blue.
A.8 CONTEXT NAMES
We provide the context names used describe the target names under the scene context Table and
14
15
We generate those names by providing GPT-4 with the instruction:
19

---

Published as a conference paper at ICLR 2025
Table 10: Per-class semantic segmentation evaluation results IDD
Grounded- SAM
Grounded- SAM HRDA
OWIVTT- SAM
OwIVIT-SAM HHDD
Ours
loU
Acc
loU
Acc
IoU
Acc
loU
Acc
IoU
Acc
road
83.5
88.9
84.6
94.5
76.0
81.9
84.9
96.4
85.9
99.1
drivable fallback
17.2
23.9
17.2
23.9
0.0
0.0
0.0
0.0
2.9
3.4
sidewalk
15.8
18.8
12.8
68.9
3.6
17.7
15.6
74.6
17.0
72.2
non con-drivable fallback
13.8
18.8
13.7
14.2
1.9
2.4
14.1
14.6
4.9
5.1
person
25.1
71.8
57.3
72.7
13.6
27.4
56.8
71.8
59.5
74.6
rider
0.2
64.4
78.4
0.6
63.5
77.2
67.7
80.3
motoreycle
57.4
62.9
64.4
77.4
34.2
36.4
64.1
77.0
67.2
80.1
bicycle
33.7
36.2
15.2
38.5
2.6
34.0
16.2
38.4
15.8
39.5
autorickshaw
0.0
0.0
0.0
0.0
34.7
46.0
34.7
46.0
36.5
81.4
84.5
88.9
63.8
96.8
22.8
23.1
69.0
96.0
70.2
80.9
truck
72.3
86.2
91.7
31.6
35.2
80.7
84.9
86.8
91.9
bus
74.8
81.6
68.4
714
42.8
47.8
63.6
66.1
68.8
71.9
curb
6.8
7.3
6.8
7.3
14.9
30.6
14.9
30.6
14.7
18.2
wall
4.7
4.7
40.8
66.5
8.0
8.9
40.8
59.8
41.9
67.6
fence
8.3
21.2
20.2
33.8
5.5
13.4
21.5
34,9
22.3
35.1
guard rail
8.2
13.0
8.2
13.0
9.3
28.5
9.3
28.5
13.9
22.1
billhoard
22.3
23.7
22.3
23.7
16.1
25.8
43.9
60.7
traffic sign
14.8
16.0
10.3
33.1
16.3
70.4
23.3
78.2
31.0
80.3
traffic light
23.7
24.5
19.5
22.5
6.4
13.1
19.3
22.3
23,9
27.2
pole
8.7
8.8
34.1
39.4
19.2
21.5
33.8
38.9
35,4
41.0
abs-str-bar-fallaal
0.9
0.9
0.9
0.9
0.0
0.0
0.0
0.0
6.4
9.5
building
45.3
60.0
51.8
87.1
26.1
49.0
$2.0
83.9
54.3
82.6
bridge
43.3
44.2
43.3
44.2
31.6
60.4
31.6
60.4
8.1
44.7
vegetation
12.4
12.4
84.5
94.5
21.5
22.6
84.3
94.0
80.6
87.0
sky
91.4
92.8
94.6
98.8
84.1
93.8
94.7
98.7
82.3
85.3
Average
30.8
36.0
339.4]
51.7
20.9
31.6
40.2
55.9
41.77
57.7
Table 11: Taxonomy mapping from GTA Mapillary
Source Label
Target Label Set
road
road, sidewalk, snow, sand, water, catch basin, manhole. rail track, lane sidewalk, curb, snow, sand, water
lane marking -eringgggeeraal
sidewalk
building
building. bridge, tunnel, phone booth, billboard. wall, bridge, tunnel, trash can, banner, billboard fence, guard rail
wall
fence
pole
pole, utility pole, trash can, banner, street light, traffic sign frame traffic light, street light traffic sign (front), traffic sign (back), billboard, banner vegetation, snow
traffic light
traffic sign
vegetation
terrain
terrain. mountain, snow, sand, water sky
sky
person
person
rider
bicyclist, motorcyclist car, trailer, boat truck, caravan bus rails motorcycle bicycle
car
truck
bus
train
motorcycle
bicycle
unlabeled
bench, billboard, bridge, tunnel
Generate new class names within the context of street scene semantic segmentation, using the orig- inal class name the head noun. Use synonyms subcategories of the original class that make sense within this context, and if the class has multiple meanings, add specific context to avoid ambi- guity. Please provide the original class names along with context names.
20

---

Published as a conference paper at ICLR 2025
Table 12: Taxonomy mapping from GTA to IDD
Source Label
Target Label Set
road
road, sidewalk, drivable fallback
sidewalk
sidewalk, curb, drivable fallback, non non-drivable fallback
building
building, bridge, billboard,
wall
wall, obs-str-bare bar-fallback, bridge, billboard
fence
fence, guard rail, obs- bar bar--fallback
pole
pole
traffic light
traffic light
trafic sign
traffic sign, billboard, banner vegetation, obs-str-bar-fallbar terrain, non-drivable fallhack, haa-str-bar-fallback
vegetation
terrain
sky
sky
person
person bicyclist, motoreyclist car, autorickshaw truck, caravan
rider
car
truck
bus
train
other vehicles
motorcycle bicycle
motoreycle
bicycle billboard, bridge
unlabeled
For each label, we generate 10 context names. For labels without ambiguity, egg. sky, we only use the original label for the text feature extraction.
21

---

Published as a conference paper at ICLR 2025
Table 13: Context names for IDD labels
Target Label
[onteext Names
road
road, main road, driving lane, paved road, highway, residential street, arterial road, rural road, city road. thoroughfare
drivable fallback
drivable terrain, traffic lane, vehicle lane, driveable path, car lanc, driveable street, urban roadway, paved path, driveable surface, roadway
sidewalk
sidewalk, pavement, footpath, walkway, pedestrian path, side path, sidewalk pavement, urban side- walk, street sidewalk, sidewalk lane, sidewalk area
non-drivable fall-
non-drivable terrain, pedestrian area, park path, garden path, bike lane, footpath, public plaza, grass area, green space, pedestrian walkway, non-driveahle zone
back
person
person
rider
rider
motorcycle
motorcycle
bicycle
bicycle
autorickshaw
autorickshaw, whreeew, tuk-iuk, aulo-rickshaw, motorized rickshww, auto taxi, rickshw, three- wheeled taxi, auto, motor tricyele, auto rickshaw
car
car, sedan, hatchback, coupe, convertible, SUV, sports car, station wagon, compact car, electric car,
luxury
truck
truck, pickup truck, semi-truck, delivery truck, dump truck, fire truck, tow truck, box truck, flathed truck, garhage truck, tanker truck bus
bus
vchicle fallback
other vehicles, train, tram, metro, trollcybus, light rail, cable car
curb
curb, road curb, sidewalk curb, curbside, street curb, pavement curb, curb edge, curb line, curb boundary, urban curb, curb strip
wall
wall, barrier wall, protective wall, retaining wall, boundary wall, perimeter wall, dividing wall, sound harrier wall, security wall, freestanding wall, partition wall
fence
fence. building fence, road fence, vehicle separation fence, pedestrian fence, safety fence. boundary fence, traffic fence, divider fence, protective fence, barrier fence
guard rail
guard rail, road guard rail, highway guard rail, safety guard rail, traffic guard rail, barrier guard rail, roadside guard rail, protective guard rail, metal guard rail, crash barrier, median guard rail billbour,, advertising billboard, roadside billboard, digital billboard, outdoor billboard, highway billbour,, commercial billbourd, urban billboard, street billboard, electronic billboard, large bill- board
billboard
traffic sign
traflic sign, road sign, highway sign, street sign, regulatory sign, warning sign, directional sign, informational sign. traftic control sign, signpost. traffic marker
traffic light
traffic light, traffic signal, stoplight, traffic control light, intersection signal, traffic lamp. signal light, road signal, street light, traffic signal light, traffic control signal
pole
pole, street polc, lamp pole. traffic pole, sign pole, light pole. support polc, signal pole, flag polc, decurative pole, banner pole
obs-str-bar-
barrirs,, construction harrier, roadblock, traffic cone, temporary fence, safety barrier, barricade, obstruction, traffic barricade, road barrier, construction zone marker building, structure, edilice, construction, residential building, commercial building, office building, apartment building, skyscraper. public building, urban building
obstructive structures
fallback
building
bridge
road bridge, Tootbridge, pedestrian bridge, walking bridge, loolpath hridge, foot crossing, small bridge, pedestrian crossing, walkway bridge, urban footbridge, trail bridge
vegetation
vegetation, urban vegetation, city greenery, roadside plants, street vegetation, urban foliage. city flora, park vegetation, public greenery. urban plants, green space sky
sky
22

---

Published as a conference paper at ICLR 2025
Table 14: Context names for Mapillary labels (Part 1)
Target Label
Context Names
Road
mad, main road, driving lane, paved road, highway, residential street, arterial roa, rural road, city/ road, thoroughfare
Snow
snow, snow pile, street snow, roadside snow, accumulated snow, snowbank, plowed snow, urban snow, compacted snow, snow drift, snow on pavement
Sand
sand, sand pilc, strect sand, roadside sand, piled sand, sandbank, accumulated sand, urban sand, sand on pavemennt, construction sand, loose sand
Catch Basin
catch basin, road catch basin, street catch basin, roadside catch basin, storm drain, drainage basin, sewer catch basin, street drain, gutter catch basin, road drain, stormwater basin
Manhole
manhole, road manhole, street manhole, sewer manhole, manhole cover, utility manhole, drainage manhole, storm drain manhule, roadside manhule, underground access, inspection manhole pothole, road pothole, street pothole, asphalt pothols, pavement pothole, highway pothole, surface pothole, pothole damage, roadway pothole, pothole crater, pothole on pavement
Pothole
Bike Lane
bike lane, marked bike lane, roadside bike lane, main road bike lane, dedicated bike lane, paved bike lame, urban bike lane, bike path, protected bike lane, street bike lane, lane-marked bike lane
Rail Track
rail track, tram rail track, train rail track. street rail track, road rail track, urban rail track, tramway track, railroad track, commuter rail track, embedded rail track. rail track pavement
Lane Marking
crosswalk lane marking, street crosswalk marking, pedestrian crosswalk marking, zebra crossing marking. road crosswalk marking. intersection crosswalk marking, painted crosswalk, crosswalk lines, crosswalk road marking, sidewalk crosswalk marking
Crosswalk
Lane Marking
general lane marking. road lane marking, street lane marking. highway lanc marking, pavement lane marking, lane divider marking, traffic lane marking, lane line marking, roadway lane marking, lane boundary marking, asphalt lane marking
General
Water
water, urban water, river water, lake water, city river, roudside pond, street water, urban pond, city' lake, small urban river, stormwater
Sidewalk
sidewalk, pavement, lovipath, walkway, pedestrian path, side path, sidewalk pavement, urban side- walk, street sidewalk, sidewalk lane. sidewalk area
Curb
curb, road curb, sidewalk curb, curbside, street curb, pavement curb, curb edge, curb line, curb boundary, urban curb, curb strip
Pedestrian Area
pedestrian area, street pedestrian area. pedestrian zone, pedestrian walkway, pedestrian street, urban pedestrian area, pedestrian plaza, pedestrian path, sidewalk pedestrian area, pedestrian crossing area, designated pedestrian area
Building
building, structure, editice, construction, residential building, commercial building, office building, apartment building, skyscraper, public building, urban building
Bridge
ruad bridge, footbridge, pedestrian bridge. walking bridge, footpath bridge, foot crossing. small bridge. pedestrian crossing, walkway bridge, urban footbridge, trail bridge
Billboard
billboar,, advertising billboard, roadside billboar, digital billboard, outdoor billboard, highway billbourd. commercial billbourd, urban billboard, street billboard, electronic billboard, large bill- board
Tunnel
tunnel, road tunnel, tunnel entrance, highway tunnel, urban tunnel, vehicle tunnel, tunnel passage, tunnel opening, subway tunnel, underground tunnel, traffic tunnel
Wall
wall, barrier wall, protective wall, retaining wall, boundary wall, perimeter wall, dividing wall, sound burrier wall, security wall, freestanding wall, partition wall
Traffic Sign Frame
Traffic sign frame, signpost Trame, traffic sign holder, sign frame, sign support frame, read sign frame, tralfic sign structure, sign mounting frame, sign frame support, traffic sign bracket
Trash Can
trush can, street trash can, public trash can, roadside trash can, outdoor trush can, urban trash can, sidewalk trash can, street garbage can, public waste bin, street litter bin, municipal trush can banner, advertising bunner, promotional banner, street banner, event banner, hanging banner, ouldoor hanner, banner sign, vertical banner, display banner, publicity banner
Banner
Fence
fonce. building fence. road fence. vehicle separation fence, pedestrian fence, sufety fence, boundary fonce, traffic fence, divider fence, protective
barrier fence
Guard Rail
guard rail, road guard rail, highway guard rail, safety guard rail, traffic guard rail, barrier guard rail, ruadside guard rail, protective guard rail, metal guard rail, crash barrier, median guard rail
Pole
pole, street polc, lamp pole, traffic pole, sign pole, light polc, support pole, signal polc, flag polc, decorative pole, banner pole
Unility Pole
utility pole, electric pole, telephone pole, power pole, transmission pole, cable pole, utility line pole, utility post, service pole, communication pole, distribution pole
Street Light
street light, street lamp, road light, streetlight, lamp post, street lighting, urban street light, sidewalk light, public street light, street lantern, street illumination
Pront
Side
Of
Tront side of trallic sign, traflic sign frunt, Iront lace traffic sign, sign front, traffic sign face, front panel of traftic sign, signboard front, traffic sign display. front view of traffic sign. sign front side. traffic sign surface
Traffic Sign
Back
Side
Of back side of traffic sign, traffic sign back, back face of traffic sign, sign back, rear of traffic sign, signboard back, traffic sign reverse, sign back panel, back side of sign, traffic sign rear view, reverse
Traffic Sign
side of traffic sign
Trafhic Light
truttic light, traffic signal, stoplight, traffic control light, intersection signal, traftic lamp, signal light. road signal, street light. traffic signal light, traffic control signal
Vegetation
vegelation, urban vegelation, city greenery, street vegelation, roudside vegetation, urban plants, city foliage, urban flora, street greenery, public vegetation, cityscape vegetation
23

---

Published as a conference paper at ICLR 2025
Table 15: Context names for Mapillary labels (Part 2)
Target Label
Conteext Names
Terrain
lerrain, urban lerrain, city landscape, street terrain, ruadside terrain, urban ground, cityscupe lerrain, urban land, urban surface, city terrain, urban topography
Mountain
mountain, mountain peak. mountain range, mountain slope, rocky mountain, highland mountain, mountain summit, alpine mountain, mountain ridge, forest mountain, mountain terrain sky
Sky
Person
person
Bicychist
bicyclist, nider, cyclist, bicycle rider, bicycle commuter, mountain biker, cyclist motorcyclist, motorcycle rider, motorcycle driver, motorbike rider, motorcycle commuter, ruad mo- torcyclist
Motorcyclist
Car
car, sedam, hatchback, coupe, convertible, SUV, sports car, station wagon, compact car, electric car, luxury car
Trailer
trailer, utility trailer. travel trailer. cargo trailer, flatbed trailer. camper trailer, enclosed trailer. stock trailer, dump trailer
Boat
boat, sailboat, motorboat, fishing boat, speedboat, yacht, cance, kayak, pontoon boat, dinghy, house- boat
Truck
truck, pickup truck, semi-truck, delivery truck, dump truck, truck, tow truck, box truck, flathed truck, garhage truck, tanker truck
Caravan
caravan, travel caravan, camper caravan, motorhome, touring RV (recreational vehicle),
fifth-weeel caravan, pop-up caravan, teardrop caravan, static caravan, off--roa caravan
Bus
Rails
on rails
Motorcycle
motorcycle
Bicycle
bicycle
Bench
bench, street bench, public bench, park bench, sidewalk bench, outdoor bench, urbun bench, pave- ment bench, city bench, public seating bench, roadside bench
24


Reviews:
[{"review_id": "LNWBw6fhab", "reviewer": "Reviewer_5x3w", "summary": "The paper formulates the problem of unsupervised taxonomy adaptive cross-domain semantic segmentation. To bridge both image-level and label-level domain gaps without supervision, the paper proposes a two-stage approach DynAlign that integrates both domain-specific knowledge and rich open-world prior knowledge from foundation models. Experiments on two street scene semantic segmentation benchmarks validates the effectiveness of the proposed method.", "questions": "1. In Lines 80-81, why this problem is significant but not be studied?  \n2. What are experimental results of existing taxonomy-adaptive domain adaptation and UDA  methods on two street scene semantic segmentation benchmarks.  How choose the confidence threshold value in the Lines 372 - 373.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper writing is good with well formula representation.\n2. The unsupervised taxonomy adaptive cross-domain semantic segmentation task is interesting and meaningful.", "weaknesses": "1. In Lines 80-81, the reason why this problem is significant but not be studied needs explain. \n2. Experimental results of existing taxonomy-adaptive domain adaptation and UDA methods also need to be presented in the Table 1 for comparison.  More parametric experiments about the confidence in the Lines 372 - 373 need to be conducted and discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O0beZchpVK", "reviewer": "Reviewer_Ggqb", "summary": "This paper propose two-stage framework called DynAlign for unsupervised taxonomy-adaptive cross-domain semantic segmentation. DynAlign leverages prior semantic knowledge to align source categories with target categories that can be novel, more fine-grained, or named differently. Foundation models are then employed for precise segmentation and category reassignment. Knowledge fusion approach is proposed to dynamically adapt to varying scene contexts.", "questions": "If the rebuttal process can explain the above problems and confusions, I will increase my score.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1. The problem that the paper focuses on, taxonomy-adaptive domain adaptation without requiring additional annotations from the target domain, is innovative and interesting.\n2. The approach based on the foundation model is explainable, reasonable, and innovative.", "weaknesses": "1. The explanation of the cpation in Figure 2 is not clear. Please explain the functions of the two CLIPs in the figure. And the caption introduces \"CLIP fusing the visual knowledge from SAM with the semantic knowledge from LLM to reassign accurate labels. The predictions can be used as pseudo-labels to further fine-tune the UDA model\". Since the predictions can be used as pseudo-labels to further fine-tune the UDA model, should the second CLIP have an output arrow?  The figure 2 cannot show the process well, which causes some misunderstandings.\n2. The paper proposes a two-stage approach. however, three steps are introduced in Figure 2. This makes it difficult to understand the content of figure 2. Please redraw the figure 2 to make the process more intuitive and easy to understand.\n3. The proposed method includes many networks and is very complex. Please compare the model's training memory usage, training time, and model parameter.\n4. As a new benchmark, will the proposed method publish the complete training and testing code for subsequent follow-up research?\n5. In 4.2 SEMANTIC TAXONOMY MAPPING, how is the context description set generated? In formula 2, on which dimension is the average operation performed?\n6. The symbol usage is inconsistent. In 4.4 KNOWLEDGE FUSION, the multi-scale visual feature is F_V, while the multi-scale visual feature in Figure 3 is F_g. Please check the symbols and charts in the full text to avoid ambiguity.\n7. In Figure 3, what does lane-marking 0.6 mean?\n8. In 5.4 ABLATION STUDIES, should multi-scale vision feature be changed to multi-scale visual feature? Please keep the name consistent in the paper.\n9. Please add more experimental details. For example, the proposed method uses the CLIP backbone, but is it the CLIP ViT-B-16 or CLIP ViT-L-14 backbone or something else?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bhpSWrDdqS", "reviewer": "Reviewer_ddgh", "summary": "This paper addresses the label-space domain gap within cross-domain semantic segmentation by proposing a novel framework that integrates traditional unsupervised domain adaptation (UDA) methods with foundational models such as large language models, vision-language models, and vision foundation models. Specifically, the large language model is utilized to reason and establish mappings between source and target label spaces. Using these mappings, predictions from unsupervised adaptation segmentation models are transformed into the target label space and subsequently fused with predictions from foundation models, including SAM and CLIP. The proposed framework demonstrates superior performance compared to several open-vocabulary recognition models.", "questions": "The framework seems to rely purely on SAM for pixel grouping, with UDA and CLIP then reclassifying each mask region. Is this my understanding accurate?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposed framework is technically sound.\n2. The paper addresses a practical and significant issue: the label-space domain gap within domain adaptation.", "weaknesses": "1. High Complexity and Heuristic Dependence\nThe framework’s complexity, along with its reliance on various heuristics, may hinder practical usability. Additionally, some heuristics lack clear specifications, making reproduction challenging. Key points include:\n\na) Inference Complexity: The framework requires inference from at least three models (UDA model, SAM, and CLIP), contributing to increased computational demands and overall complexity.\n\nb) Semantic Taxonomy Mapping: The framework initially prompts GPT-4 for potential taxonomy mappings, refined through human input. This process adds a human burden due to prompt tuning and taxonomy refinement and is only briefly outlined in the supplementary material (lines 916 & 947), lacking comprehensive detail.\n\nc) Criteria for Label Ambiguity: The paper states, “For labels without ambiguity, e.g., sky, we only use the original label for text feature extraction” (lines 1000-1001). The criteria for “ambiguity” remain unclear, introducing additional heuristics or human intervention.\n\n2. Limited Experimental Scope\nWhile the paper showcases superior performance in certain settings, the experimental scope and choice of baselines are somewhat limited. Expanding the evaluation would strengthen the framework’s claim of superiority:\n\na) Additional Baselines with Foundation Models: Numerous studies have leveraged foundation models for semantic segmentation, yet many of these are omitted in the experimental section. Including models such as [A] for domain-generalized segmentation and [B] for open-vocabulary segmentation would enhance comparability.\n\nb) Traditional Domain Adaptation Settings: Although the framework targets handling substantial label-space domain gaps for real-world applications, demonstrating its efficacy in traditional domain adaptation settings (e.g., GTA5 to Cityscapes, with a smaller label-space gap) is also important.\n\nc) Other Label-Space Domain Gaps: The experiments primarily focus on scenarios where the target domain includes more semantic classes than the source (e.g., 45 classes for Mapillary, 24 for IDD, and 19 for GTA). Evaluating scenarios with fewer classes in the target domain (e.g., Mapillary to Cityscapes) would substantiate the framework’s generalizability.\n\nReferences\n\n[A] Stronger, Fewer, & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation, CVPR 2024\n\n[B] Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactively, ECCV 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1hgtKTXsdh", "reviewer": "Reviewer_cqS3", "summary": "The paper targets the label-level domain gap, which is common in real-world scenarios.  The authors introduce DynAlign, a\ntwo-stage framework that integrates UDA with foundation models to bridge both the image-level and label-level domain gaps. DynAlign leverages prior semantic knowledge to align source categories with target categories that can be novel, more fine-grained, or named differently (e.g., ‘vehicle’ to {‘car’, ‘truck’, ‘bus’}). Foundation models are then employed for precise segmentation and category reassignment. To further enhance accuracy, the authors propose a knowledge fusion approach that dynamically adapts to varying scene contexts. DynAlign generates accurate predictions in a new target label space without requiring any manual annotations.", "questions": "Please see the \"Weaknesses\" part.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. The problem in this paper is of great practical significance.\n\n2. The writing of this paper is clear and well-organized.\n\n3. The experimental results showed some improvements when using such technology, compared with previous rivals.", "weaknesses": "1. The article only explores coarse-to-fine experiments. In real-world scenarios, however, the label structure of the target domain may sometimes be coarser than that of the source domain. How would the proposed method be adapted for such a setting? Additionally, the effectiveness of the method under these conditions has not been verified.\n\n2. The contribution towards DG area is limited. The generalization achieved by the methods in this paper primarily derives from the pre-trained models—specifically, SAM's generalization in segmentation and CLIP's generalization in multimodal alignment—rather than from the techniques introduced within the paper itself. As a result, this work mainly resembles a mapping technique that aligns open-vocabulary labels between the source and target domains, an idea that has been explored extensively in other studies. Consequently, I do not view this paper as making a significant contribution to the field of domain generalization. In my opinion, as an ICLR article, it still needs a certain theoretical depth.\n\nMinor issues:\nSome papers are cited repeatedly in the reference, such as \"Domain aligned clip for few-shot classification.\" and \"Scaling open-vocabulary image segmentation with image-level labels.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LeSKiMzRt3", "reviewer": "Reviewer_GEaE", "summary": "This paper presents  a two-stage framework that integrates UDA with foundation models to bridge both the image-level and label-level domain gaps. It consists of multiple modules including conventional UDA to bridge the distribution gap between source and target, vision foundation model to extract general segmentation, LLM to construct the label taxonomy, and CLIP to align visual and text features to generate well-aligned pseudo labels for self-training.", "questions": "Please see above Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Belows are strong points that this paper has:\n\n1. Well-written and systematically organized to make the reader to understand more easily. \n\n2. It proposes reasonable fusion strategy between UDA and Foundation model to resolve the distribution and label shift issue  in cross-domain scenario\n\n3. It achieves remarkable performance improvement, showing the effectiveness of the proposed method on two benchmark.\n\n4. Figure 4 shows that training with generated pseudo labels can not only achieve better score but also increase the inference efficiency.", "weaknesses": "Belows are weak points that this paper has:\n\n1. In order to align with line 112 (\"integrating with any UDA-based semantic segmentation models\"), it would be better that author provide another examples to show the compatibility of the proposed method (e.g., MIC + DynAlign, or other UDA model + DynAlign)\n\n2. Need to have more analysis on how robust the proposed knowledge fusion is. One way to do is that using different foundation model (e.g., MobileSAM for SAM, Llama for GPT-4,  another VLM for CLIP) can also work with the proposed method. Showing this compatibility can provide more flexible fusion environment.\n\n3. Any failure case while fusing UDA and foundation model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R39OsCAHIJ", "reviewer": "Reviewer_7s4Z", "summary": "Traditional UDA methods for semantic segmentation often assume identical class labels in both source and target domains. To tackle this, this paper presents DynAlign, a two-stage framework that combines UDA with foundation models to address both image-level and label-level domain gaps. It leverages prior semantic knowledge to align source and target categories and uses foundation models for detailed segmentation and category realignment. To boost accuracy further, a knowledge fusion strategy is divised to dynamically adapt to diverse scene contexts.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The method design of this paper is straightforward and easy to understand, with sufficient details and reliable techniques.\n2. The paper aims to address the label space mismatch issue in UDA tasks, and this research problem is very meaningful.\n3. The research motivation of the paper is strong, and the solution is clearly articulated.", "weaknesses": "1. The paper dedicates considerable space to comparisons with open-vocabulary semantic segmentation methods and benchmarks its approach against those used in that task. I understand that this is due to the limitations in label space, which restrict traditional UDA methods. However, since the focus of this paper is UDA, I believe the authors should concentrate on comparisons with UDA methods to better highlight the advantages of this work. Throughout my reading, I found myself constantly switching between the two tasks (UDA and open-vocabulary SS), making it challenging to clearly grasp the contributions of this paper.\n\n2. I understand that earlier methods also utilized foundation models but still required target labels for few-shot training. In contrast, this paper employs foundation models in a fully unsupervised manner, correct? Is this the main difference/contribution this paper has compared to previous methods? \n\n3. How to better utilize foundation models has been a significant research topic in recent years, and I find the authors' exploration of this subject very meaningful. However, with the increasing number of foundation models available, it is insufficient to merely leverage more or better foundation models to complete tasks. The authors should emphasize the specific benefits of their design rather than simply implementing it in an unsupervised manner. Since the inference of large models is inherently unsupervised, simply using them does not constitute a substantial contribution.\n\n4. The use of foundation models still incurs computational resource demands, which can affect the practical applicability of this framework. The experimental section should include comparisons regarding the computational resource consumption of the models, such as memory usage and inference time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)