Cross-Class Domain Adaptive Semantic Segmentation with Visual Language Models

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 14:18:03

Abstract: This paper addresses the issue of cross-class domain adaptation (CCDA) in semantic segmentation, where the target domain contains both shared and novel classes that are either unlabeled or unseen in the source domain. This problem is challenging, as the absence of labels for novel classes hampers the accurate segmentation of both shared and novel classes. Since Visual Language Models (VLMs) are capable of generating zero-shot predictions without requiring task-specific training examples, we propose a label alignment method by leveraging VLMs to relabel pseudo labels for novel classes. Considering that VLMs typically provide only image-level predictions, we embed a two-stage method to enable fine-grained semantic segmentation and design a threshold based on the uncertainty of pseudo labels to exclude noisy VLM predictions. To further augment the supervision of novel classes, we devise memory banks with an adaptive update scheme to effectively manage accurate VLM predictions, which are then resampled to increase the sampling probability of novel classes. Through comprehensive experiments, we demonstrate the effectiveness and versatility of our proposed method across various CCDA scenarios.

Main paper: 
 introduction: 
 Traditional Unsupervised Domain Adaptation (UDA) methods for semantic segmentation [12, 24, 42, 49] aim to transfer knowledge from a labeled source domain (e.g. synthetic samples) to an unlabeled target domain (e.g. real-world samples). These methods, while typically relying on self-training with pseudo-label generation [11, 13, 35, 47] for better adaptation, have become a promising solution for addressing cross-domain challenges. However, due to the assumption that the source and target domains must share an identical set of classes, existing UDA solutions suffer from limited generalizability especially when confronted with previously unseen classes in the target domain [20, 25, 27] . This limitation renders Figure 1 : Comparison between previous UDA methods and our method. Here, the novel class is defined as "pole", and the ground truth is provided for better visualization. When encountering the novel class, conventional UDA methods like [11] fail to capture correct pseudo labels, where the novel class is always predicted as the shared ones. Instead, by utilizing VFMs [7, 16, 28] , our model can relabel novel classes with accurate annotations, which facilitates robust learning of both shared and novel classes.them inflexible in real-world scenarios, where encountering unseen classes is a common occurrence.Motivated by this, we focus on a more practical but challenging problem named Cross-Class Domain Adaptation (CCDA) [8] for semantic segmentation, where the target domain not only includes the shared classes but also involves some private novel classes that are unlabeled or unseen in the source domain. TACS [8] is a pioneer in researching this issue, but they require 30 labeled samples for each novel class. Different from TACS, our objective is to address challenges across different classes and domains without relying on additional labels for novel classes. However, this setting is extremely challenging as the absence of novel class labels hampers the correct segmentation of both shared and novel classes. Specifically, the lack of novel class annotations decreases the reliability of pseudo labels by assigning novel classes as shared ones (as shown in Figure 1 ), causing more noisy pseudo labels to mislead the training in the target domain.Recent breakthroughs in contrastive pre-training studies present a promising path for developing foundational models that integrate vision and language, known as Vision-Language Models (VLMs), for various computer vision tasks. These VLMs like CLIP [28] and Im-ageBind [7] excel at encoding a diverse range of visual concepts and demonstrate remarkable capabilities in making zero-shot predictions without task-specific training examples [34, 41] , presenting a potential solution to segment novel classes in CCDA. Nevertheless, VLMs are trained to match an entire image with a text description and (c) depict widely used mask proposal generators that predict class-agnostic segmentation maps. However, Mask-Former [3] fails to capture precise segmentation maps, while SAM [16] can not provide masks for each pixel. (d) and (e) are existing SOTA works, OVSeg [19] and SAN [44] , performing fine-grained semantic segmentation with VLMs, where they exhibit low performance on small-size objects and distort the outlines of objects. (f) denotes the mask proposals of the novel class "pole" produced by combining SAM [16] and ImageBind [7] . (g) and (h) are the results generated by the DAFormer [11] framework without and with our proposed elements. The ground truth is shown in (i).and fail to capture fine-grained alignment between image regions and text, resulting in limited applicability in fine-grained semantic segmentation tasks. A promising remedy for enabling fine-grained predictions is to generate a set of class-agnostic mask proposals (i.e., Figure 2 (b-c )), and then leverage VLMs to classify them into specific classes [19, 44, 45] . For example, simply combining SAM [16] and ImageBind [7] can generate Figure 2 (f) for the novel class "pole". However, these VLM-based fine-grained segmentation methods demonstrate limited performance (as shown in Figure 2 (d-f )) due to several factors: (1) the visual disparities between VLMs' training images and masked proposals [19, 45] , (2) the defective predictions of the mask proposal generators. Thus, CCDA challenges are far from solved by directly applying these VLM-based approaches.To bridge the research gap, in this paper, we present a label alignment approach for novel classes by employing VLMs to relabel pseudo labels. In detail, we propose to utilize VLMs for locating novel classes within target images, and then relabel pseudo labels with these novel-class proposals. Motivated by evident failure cases where several shared-class regions are misclassified as novel ones in VLM predictions (i.e., Figure (f)), we present an uncertaintydriven adaptive threshold for excluding misclassified proposals and alternatively performing label alignment. To further enable the sufficient learning of novel classes, we establish memory banks to store novel-class proposals and resample them to augment pseudo labels. Given the limited memory size, it is neither feasible nor necessary to memorize every novel class proposal. To address this issue, we devise an adaptive memory update scheme to dynamically incorporate new proposals while discarding outdated ones. We comprehensively assess the effectiveness and universality of our method across different benchmarks, UDA frameworks, and VLMbased segmentation methods. Extensive experiments illustrate our method surpasses the existing state-of-the-art (SOTA) approaches under the CCDA setting (as depicted in Figure 2 (g-i)). In summary, our contributions can be outlined as follows:• We propose a VLM-based label alignment method with an uncertainty-driven threshold to relabel pseudo labels with accurate novel class proposals. • We present a novel class resampling method with adaptively updated memory banks to augment pseudo labels, which facilitates the sampling probability of novel classes. [10, 33, 48] , feature level [5, 22] , output level [30, 39, 46] , or patch level [37] based on a Generative Adversarial Network (GAN) [9] . Benefiting from their training stability and strengthened performance, self-training methods, which strive to generate pseudo labels offline [32, 51] or online [6, 11, 35, 50] for enabling the training of the target domain, gradually become the main solution for UDA. Moreover, some work [15, 40] also attempts to combine adversarial training with selftraining strategies, pursuing better performance in cross-domain scenarios. However, these UDA methods assume that the source and target domains must possess identical sets of classes, rendering them vulnerable when confronted with novel classes [20, 25, 27] . This limitation hinders their applications in real-world scenarios. Therefore, investigating the CCDA problem is highly necessary for semantic segmentation tasks. 
cross-class domain adaptation: 
 Driven by the restriction of UDA methods on encountering novel classes, some recent studies [18, 27, 31] attempt to explore the cross-class problem in domain adaptation scenarios, where the target domain shares some classes with the source domain, but it also includes its own distinct, private novel classes. For instance, openset/universal domain adaptation works [27, 31] focus on assigning the "unknown" label to all novel classes during the inference, while class-incremental domain adaptation [18] aims at recognizing each shared and novel target classes with a unique semantic label. Nevertheless, these studies merely revolve around image classification tasks, resulting in CCDA for semantic segmentation still being an open question. TACS [8] is the work most similar to ours. However, TACS still necessitates few-shot labels for novel classes to conduct cross-class adaptation, which restricts its generalization capability in more challenging zero-shot scenarios. In contrast to TACS [8], we attempt to address CCDA challenges without relying on additional labels for novel classes.Due to the advancements in self-supervised representation learning [1, 2] , multimodal Visual Language Models (VLMs), designed to acquire generic representations of vision and language, have showcased remarkable achievements in recent years. Recent studies, exemplified by CLIP [28] and ImageBind [7] , connect visual and language concepts by conducting pre-training on large-scale text-image pairs. As numerous vision and language concepts are encompassed within large-scale datasets, these methods showcase surprisingly robust capabilities in conducting zero-shot predictions, marking significant progress in various downstream tasks. For example, Simseg [45] proposes a two-stage segmentation framework, where they first generate class-agnostic mask proposals within images and then assign specific semantics for each mask by a frozen CLIP. SAN [44] presents an adapter network to benefit CLIP in recognizing the semantics of mask proposals. However, despite their successes, the performance of these VLM-based segmentation methods [19, 44, 45] remains constrained in complex scenarios due to suboptimal mask proposal generators [3, 16] and visual disparities between training and test samples [19] . In this paper, we endeavor to harness the capabilities of VLMs for relabeling pseudo labels, offering a promising avenue for their application in driving scenes. 
methodology 3.1 problem description: 
 In the CCDA settings, we have a labeled source dataset {X, Y} 𝑆 = {(𝑥 𝑠 𝑛 , 𝑦 𝑠 𝑛 )} 𝑁 𝑆 𝑛=1 and an unlabeled target dataset{X 𝑇 } = {𝑥 𝑡 𝑛 } 𝑁 𝑇 𝑛=1, where X and Y refer to images and semantic labels. In the source dataset, each image 𝑥 𝑠 𝑛 is associated with a corresponding label 𝑦 𝑠 𝑛 of source classes 𝐶 𝑠ℎ𝑎𝑟𝑒𝑑 . However, in the target dataset, in addition to 𝐶 𝑠ℎ𝑎𝑟𝑒𝑑 , there are several private novel classes 𝐶 𝑛𝑜𝑣𝑒𝑙 (𝐶 𝑠ℎ𝑎𝑟𝑒𝑑 ∩ 𝐶 𝑛𝑜𝑣𝑒𝑙 = ∅) that are either unlabeled or unseen in the source dataset. In this context, the label space of the source domain 𝐶 𝑆 = 𝐶 𝑠ℎ𝑎𝑟𝑒𝑑 , and that of the target domain 𝐶 𝑇 = 𝐶 𝑠ℎ𝑎𝑟𝑒𝑑 ∪𝐶 𝑛𝑜𝑣𝑒𝑙 . Our objective is to adapt the segmentation model 𝑓 𝜃 , trained on {X, Y} 𝑆 and X 𝑇 , to target dataset samples with the label space 𝐶 𝑇 .In our framework, as shown in Figure 3 , a neural network 𝑓 𝜃 is trained on the source dataset (X 𝑆 , Y 𝑆 ) and is expected to perform well on the target dataset samples without access to target labels Y 𝑇 . As annotations Y 𝑆 are available for the source classes 𝐶 𝑆 , we naively compute the supervised cross-entropy loss L 𝑆 on the source domain via:EQUATIONNevertheless, the model only trained with L 𝑆 will suffer from performance degradation on the target domain. To address this issue, an auxiliary unsupervised loss L 𝑇 is devised for the target domain, which is calculated with pseudo labels ŷ𝑡 predicted via a teacher network 𝑓 𝜃 ′ : ŷ𝑡 𝑖,𝑗,𝑐 = 𝜉 (𝑐 = arg maxEQUATIONEQUATIONHere, 𝜉 (•) and 𝑞 𝑡 represent the Iverson bracket and the confidence weighting coefficient [12, 13] , respectively. Then the total loss L is computed as L = L 𝑆 + L 𝑇 . In addition, to stabilize pseudo labels [11] , the teacher network 𝑓 𝜃 ′ is optimized via the exponential moving average (EMA) strategy [17] by:EQUATIONHere, 𝛼 is the smoothing factor and is defined as 0.999. However, due to the absence of novel class labels, the segmentation model struggles to capture robust representations for novel classes during training, resulting in novel classes being erroneously assigned as shared ones in pseudo labels. Driven by this motivation, we first present a VLM-guided label alignment method to alternatively rectify pseudo labels with precise proposals for novel classes. Additionally, to enhance sufficient learning for novel classes, we construct memory banks to dynamically manage novel class proposals and utilize them to augment the supervision for novel classes. Detailed descriptions of our proposed methods for label alignment and novel class resampling will be provided in the following sections.Under the CCDA settings, the segmentation model consistently develops a biased perception of novel classes due to the absence of their annotations. Hence, we propose a label alignment method aimed at rectifying pseudo labels with accurate novel class semantics. Our label alignment method comprises two steps: mask proposal generation for novel classes and threshold-guided pseudo label rectification.Mask proposal generation for novel classes. The correction of pseudo labels necessitates precise semantic maps for novel classes. However, given unlabeled target samples, localizing novel classes with accurate semantics poses a significant challenge. To address this issue, we resort to pre-trained VLMs, especially ImageBind [7] , thanks to their capability of making zero-shot predictions without the need for task-specific training examples. Specifically, in this work, we introduce a VLM-based fine-grained segmentation method to conduct the localization of novel classes: we first generate classagnostic mask proposals from target images and then assign each mask proposal with a specific semantic.In particular, we segment objects within an unlabeled target image 𝑥 𝑡 based on a generic segmentation model 𝑔 𝑠𝑒𝑔 , where the process can be formulated as follows:EQUATIONHere, M represents a set of binary masks. Note that each mask covers a potential object in 𝑥 𝑡 but fails to deliver any semantic labels. By using each mask 𝑚 ∈ M, we extract an individual object from the color image and capture its unique mask proposal via 𝑤 = 𝑚 ⊙ 𝑥 𝑡 . We then classify these mask proposals with the guidance of ImageBind. The fundamental concept is to introduce the text encoder 𝜙 𝑡𝑒𝑥𝑡 and the image encoder 𝜙 𝑖𝑚𝑔 of ImageBind to conduct semantic alignment between visual proposals and text descriptions, identifying novel classes in unlabeled target samples.Specifically, let "a photo of a [𝐶𝐿𝐴𝑆𝑆]", denoted as A, refer to the semantic template that serves as input to 𝜙 𝑡𝑒𝑥𝑡 . This input produce a 𝑑-dimensional text embedding 𝑒 𝑐 for the class 𝑐 via:EQUATIONIn Eq. ( 6), we obtain a set of text embeddings for the classes in 𝐶 𝑇 . Similarly, for each mask proposal 𝑤, we can derive a visual embedding 𝑢 by 𝜙 𝑖𝑚𝑔 :EQUATIONWe calculate the cosine distance between 𝑢 and the set of text embeddings. Then, the most similar text embedding is selected via: . 3).EQUATIONThreshold-guided pseudo label rectification. To tackle the problem of noisy VLM predictions, we design an uncertainty-driven adaptive threshold to filter out misclassified proposals and conduct label alignment with accurate ones. Our proposed method is derived from the empirical observation that novel classes exhibit significant uncertainty compared with shared ones by the segmentation model [14, 21, 43] . Therefore, the uncertainty of shared classes can be regarded as a threshold for localizing novel class regions. Specifically, given a novel class proposal 𝑤 * and its associated image region 𝑤, we regard 𝑤 * as a misclassification if the segmentation model yields higher uncertainty in 𝑤. In this paper, we adopt the entropy 𝑣 of the pseudo label ŷ𝑡 as the prediction uncertainty, which is defined as:EQUATIONHere, 𝑝 𝑖,𝑗,𝑐 represents the softmax probability of pixel 𝑥 𝑖,𝑗 in class 𝑐. However, due to varying entropy exhibited throughout the selftraining process, and the inconsistent complexities among different classes, the prediction entropy is both class-dependent and timevarying. Hence, it is impractical to compare the entropy of novel and shared classes. Aiming at this challenge, to capture real-time entropy at different training stages, we calculate the average entropy v𝑐 for each class 𝑐 ∈ 𝐶 𝑇 at every iteration step and then employ the EMA strategy for their online updates via:EQUATIONEQUATIONHere, v′ 𝑐 is the mean entropy of the class 𝑐 in the previous iteration, and 𝛽 is the smoothing factor set as 0.999.Considering a pixel 𝑥 𝑖,𝑗 located in a proposal 𝑤 * labeled with 𝑐 𝑛𝑜𝑣𝑒𝑙 , the segmentation model predicts 𝑥 𝑖,𝑗 as 𝑐 𝑠ℎ𝑎𝑟𝑒𝑑 with entropy 𝑣 𝑖,𝑗 . It is intuitive to conclude that the segmentation model makes an inaccurate prediction at 𝑥 𝑖,𝑗 when 𝑣 𝑖,𝑗 > v𝑐 𝑠ℎ𝑎𝑟𝑒𝑑 , and an accurate prediction when 𝑣 𝑖,𝑗 ≤ v𝑐 𝑠ℎ𝑎𝑟𝑒𝑑 . Inspired by this, we relabel ŷ𝑡 via:EQUATIONHere, the scaling factor 𝜎 1 is introduced to exclude cases where the object is misjudged due to some factors such as distance or size.Nevertheless, Eq. ( 12) is executed discretely in spatial dimensions, ignoring the challenging regions on shared classes and hindering the generation of complete semantic maps for novel classes. Therefore, we further employ a voting strategy to address these issues in pseudo-label rectification. This voting strategy is conducted within the area covered by 𝑚 in ŷ𝑡 , utilizing statistical analysis of pixelwise semantics:EQUATION)EQUATIONHere, 𝛿 (•) is the indicator function. The pseudo label ŷ𝑡 is further refined via:EQUATIONI denotes a matrix filled with ones, which shares the same shape with ŷ𝑡 .Although the label alignment method is capable of rectifying pseudo labels, the quantity of corrected novel class regions remains insufficient to enable comprehensive learning by the segmentation model. Hence, we establish memory banks to store novel class proposals for further enhancing the supervision of novel classes. However, storing all novel class proposals generated by VLMs is impractical, as it would accumulate noisy proposals and significantly inflate the size of the memory bank. Therefore, we propose a fixed-size memory bank with an adaptive update strategy to more effectively manage novel class proposals. Specifically, our proposed adaptive update strategy comprises two operations: caching new proposals and removing obsolete ones.Caching new proposals. Since there is a considerable number of misclassifications in VLM predictions, simply storing these novel class proposals generated offline may involve misleading information. Therefore, we maintain an independent memory bank for each novel class online for organizing those proposals with accurate labels. In this work, the proposals intended for storage primarily originate from three sources. Firstly, executing the operations outlined in Eq. (5-15) yields a comprehensive appearance of the novel class region. These proposals, after being rectified by novel class alignment, are stored in the memory bank. Secondly, in the later stages of self-training, our label alignment method enables the segmentation model to accurately represent novel class objects. We incorporate these novel class regions, self-predicted by the segmentation model, into our memory bank for subsequent pseudo-label augmentation. Thirdly, to accommodate a more diverse range of novel class proposals, we relax the requirement in Eq. ( 14) that the novel class must constitute the majority. Specifically, given a VLM-predicted proposal 𝑤 * labeled with 𝑐 𝑛𝑜𝑣𝑒𝑙 and ŷ𝑡 relabeled after label alignment, we calculate the ratio 𝑧 𝑐 𝑛𝑜𝑣𝑒𝑙 via Eq.( 14). Once 𝑧 𝑐 𝑛𝑜𝑣𝑒𝑙 > 𝜎 2 , where 𝜎 2 is a predetermined threshold to mitigate the interference caused by challenging regions of shared classes, 𝑤 * is adopted for storage.Removing obsolete proposals. In this study, we eliminate obsolete proposals based on the pixel areas of cached samples, as larger-area proposals contribute significantly to the comprehensive learning of novel classes [36] . Instead of directly counting the pixel areas of cached samples, we preserve valuable proposals based on the number of pixels 𝑝 predicted or rectified as novel classes, as it indirectly reflects both the pixel area and discriminative capacity of a specific proposal. Specifically, within a memory bank of 𝑐 𝑛𝑜𝑣𝑒𝑙 , we preserve the subset 𝐷 𝑐 𝑛𝑜𝑣𝑒𝑙 consisting of 𝐾 proposals with the highest 𝑝 values. Furthermore, we utilize these cached novel class proposals to augment pseudo labels via ClassMix [26] :EQUATION) 
experiments 4.1 experimental settings: 
 Datasets. To validate the effectiveness of our proposed elements, we evaluate our framework on two benchmarks. Namely SYN-THIA → Cityscapes and SYNTHIA → IDD, respectively. The source dataset in both settings is SYNTHIA [29] , where only the objects of classes road, sidewalk, building, traffic light, traffic sign, vegetation, sky, person, rider, car, bus, motorcycle, and bike are labeled in our experiments. In the first setting, we focus on dealing with the domain gap in structured scenes, where the Cityscapes [4] serves as the target dataset, where no annotations are given for novel classes wall, fence, pole, terrain, truck, and train. In the second setting, we attempt to address domain gaps in unstructured scenes. We leverage IDD dataset [38] as the target dataset, where 6 classes wall, fence, pole, truck, animal, and autorickshaw are defined as novel classes with no annotations. Implementation details. In our experiments, the implementation is based on DAFormer [11] framework with its training hyperparameters and self-training strategy. Our model is trained with AdamW [23] , where a learning rate of 6 × 10 -5 is set for the encoder and that of 6 × 10 -4 for the decoder. We configure a weight decay to 0.01, and the batch size to 2, where the images are cropped into 512 × 512 [11] during the 40𝑘 iterations. We adopt SAM [16] as our generic segmentation model due to its capability on precisely segmenting arbitrary images. We fixed the size of the memory bank for each novel class at 30. When referring to the parameters involved in our elements on the DAFormer framework, we set 𝜎 1 and 𝜎 2 to 2 and 0.1 for best performance.Competitive methods. To evaluate the effectiveness in zeroshot CCDA scenarios, our proposed model is compared with the existing top-performing algorithms, which include UDA methods [11] [12] [13] 35] , the VLM-based methods [7, 16, 19, 44] , and the CCDA methods [8] .In this section, we evaluate our method against other top-performing methods in various cross-class scenarios. Tables 1 and 2 illustrates the segmentation results on both benchmarks.Comparison with UDA methods. We first assess our proposed method based on DAFormer [11] framework with SOTA UDA methods [11] [12] [13] 35] . As shown in Table 1 and 2, our proposed method achieves top performance and outperforms these UDA methods. [11, 13] , the VFM-based method [44] , the TACS method [8] , and our presented framework on the SYNTHIA → Cityscapes benchmark. The segmentation areas of novel classes are highlighted by yellow dashed boxes.Specifically, in contrast to the best performing MIC [13] , our model improves by +10.65 mIoU on the SYNTHIA → Cityscapes benchmark and +17.88 mIoU on the SYNTHIA → IDD benchmark. It is worth mentioning that the performance of our method far exceeds these UDA studies on novel classes. For example, in Table 1 , the novel class pole achieves a +26.43 significant performance gain against all UDA methods, demonstrating the effectiveness of our approach in segmenting novel classes. These observations are further supported by Figure 4 , where previous methods struggle with incomplete segmentation of novel classes and misclassify them as shared ones. For instance, MIC misclassifies "terrain" as "vegetation" in the second row. Instead, through relabeling pseudo labels and augmenting novel classes, our model adeptly handles the challenges arising from cross-class adaptation.Comparison with VLM-based method. Our method is also assessed against SOTA VLM-based fine-grained segmentation methods [7, 16, 19, 44] in Table 1 , where "S+I" refers to the segmentation results generated by SAM [16] and ImageBind [7] . As exhibited in Table 1 , the VLM-based methods fall short of the performance achieved by our proposed method. Specifically, the mIoU accuracy of our method stands at 61.10, surpassing the top-performing VLM method, SAN, by +20.1. In Figure 4 , we observe that SAN encounters challenges in accurately segmenting tiny novel classes (i.e., "poles" in the first row), and distinguishing visually similar objects (i.e., "train" and "bus" in the fourth row). Meanwhile, the segmentation maps generated by SAN exhibit significant distortions in object contours. Therefore, there is still a lot of room for VLMs to improve fine-grained semantic segmentation results. Instead, our approach, independent of VLMs during the inference stage, achieves accurate segmentation results even in cases of small sizes or similar appearances, exhibiting superior advantages compared to VLM-based methods.Comparison with CCDA method. We further evaluate our proposed method with TACS [8] which is also meticulously designed to address the CCDA problem. For comparison under our CCDA setting, we retrained TACS under the 0-shot setting, and the results obtained on the two benchmarks are shown in Tables 1 and 2 . Analyzing their mIoU values, our results achieve significant performance gains compared to TACS on both benchmarks. For instance, in Table 1 , the mIoU performance of novel classes and all classes are higher than TACS by +41.67 and +23.67, respectively. The segmentation maps shown in Figure 4 also demonstrate that the elements proposed in TACS are not effective in 0-shot settings:(1) TACS fails to locate novel classes in the target images, resulting in the persisting issue of misclassifying novel classes, (2) the objects segmented by TACS exhibit incompleteness in appearance, such as the bicycles in the first row. Meanwhile, our method not only achieves correct segmentation of novel classes in a zero-shot manner but also restores the complete appearance of the objects.Generalization for various UDA methods. Our proposed elements are plug-and-play and can be seamlessly embedded into self-training UDA frameworks, enabling an efficient resolution of both domain gaps and class gaps. In Tables 1 and 2 , we evaluate the performance of combining our proposed elements with representative UDA methods [11, 13, 35] . It is evident that our model consistently delivers notable performance enhancements when coupled with various UDA methods. For instance, when evaluating our method with the DAFormer [11] framework on the SYNTHIA → Cityscapes benchmark, we achieve performance gains +41.67 for novel classes and +15.81 for both shared and novel classes. The performance improvements on the SYNTHIA → IDD benchmark also highlight the adaptability and versatility of our proposed elements and clarify their positive influence on various UDA methods for dealing with the CCDA problem.To assess the effectiveness of each component, we conduct ablation studies on the DAFormer framework in Table 3 owing to the fast training. The baseline denotes the DAFormer framework retrained on CCDA settings. "LA" and "NCR" are our proposed label alignment and novel class resampling, respectively. Firstly, our LA has a positive impact on segmentation performance, resulting in a +3.55 gain in shared classes and a +35.99 in novel classes (method 1 and method 2). The absence of labels for novel classes in the baseline model impedes the robust representations of these novel classes. However, by relabeling novel classes in pseudo labels, our proposed LA method is capable of simultaneously improving the performance of both novel and shared classes. Figure 5 further illustrates the effectiveness of our presented uncertainty-guided threshold in LA. We can observe that there are a great number of misclassifications in VLM predictions. However, by comparing the uncertainty of novel classes with shared ones (i.e., the uncertainty of "terrain" and "tree"), the pseudo label can be accurately relabeled for novel class proposals. Meanwhile, the voting strategy further rectifies objects with complete masks. Secondly, the importance of NCR can not be ignored (method 1 and method 3), where mask proposals with accurate novel class labels are employed to augment pseudo labels. It plays a crucial role in facilitating the learning of novel classes while retaining the performance of shared classes. Thirdly, our complete model achieves a mIoU of 61.10, exhibiting a substantial mIoU performance improvement compared to other settings.Influence of the parameter 𝜎 1 . To gain a more profound insight into the operational principles of our label alignment mechanism, we explore the impacts of the parameter 𝜎 1 on excluding noisy novel class proposals. We build our experiments on the DAFormer [11] framework. Table 4 illustrates a comparison of the performance under different 𝜎 1 values when 𝜎 2 = 0.1. It is evident that as 𝜎 1 gradually increases, the predictive accuracy also shows an improvement, where its peak performance is achieved when 𝜎 1 reaches 2. This occurs because a smaller 𝜎 1 may result in challenging regions within shared classes being incorrectly relabeled as novel classes. In these cases, the number of trainable samples in shared classes is reduced and a significant amount of misleading information is introduced into the training of novel classes, hindering the robust learning of both shared and novel classes. Whereas, when the 𝜎 1 becomes excessively large, for instance, when 𝜎 1 is set to 4, a significant decline in performance is observed. This is due to the circumstance that, upon satisfying the specified threshold, the limited region is being reassigned with correct novel class labels, resulting in the underutilization of novel class proposals for pseudo labels.Influence of the parameter 𝜎 2 . We also conduct experiments to analyze the sensitivity of our proposed method on the parameter 𝜎 2 within the DAFormer [11] framework, evaluating its performance on the SYNTHIA → Cityscapes benchmark. We summarize the segmentation results with the different 𝜎 2 in Table 4 when 𝜎 1 = 2. Observably, the model achieves optimal segmentation performance when 𝜎 2 is configured as 0.1. Specifically, setting 𝜎 2 to 0.1 results in an MIoU value of 41.67 for the novel classes and an overall MIoU value of 61.10 across all classes. However, when 𝜎 2 is excessively high or low, the performance of our model experiences degradation. When the parameters are set too low, such as 0, challenging regions within the shared classes are incorrectly relabeled as novel classes, reducing the quality of trainable samples for novel classes. Larger values reduce the number of novel class proposals to be stored, thereby limiting pseudo-label augmentation and diminishing the enhancement effects on our segmentation model.In this section, we assess the feasibility of our method under different VLM-based implementations. Here, we provide several alternative solutions with different VLMs to classify mask proposals for novel classes. Specifically, ImageBind and CLIP [28] are involved in the evaluation. Based on the aforementioned implementations, we retrain our model with the DAFormer framework [11] on the SYNTHIA → Cityscapes benchmark. As shown in Table 5 , in addition to ImageBind, our method also displays positive effects with CLIP, where the mIoU accuracy of novel classes is 31.31 and that of all classes is 56.91. However, in comparison to CLIP, our model demonstrates superior performance on ImageBind, exhibiting a performance gain +10.36 for novel classes and +4.19 for all classes. This may be because ImageBind conducts contrastive pre-training on multiple modalities (i.e., audio and depth images), contributing to learning more robust visual-language connections of classes. This can be advantageous to obtain more precise mask proposals for novel classes. These results demonstrate that our method can be implemented with different VLMs.In TACS [8] , three types of inconsistent taxonomic setups are studied, including the open, coarse-to-fine, and implicitly overlapping taxonomy settings. In Tables 1 and 2 , we assess the effectiveness of our method under the open taxonomy setting in the zero-labeled target domain. Here, we further evaluate the adaptability and effectiveness of our DAFormer-based pipeline under other two distinct taxonomy settings. All experiments here strictly follow the settings in TACS, with the only variation being the zero-labeled target domain. Table 6 presents the results in the coarse-to-fine and implicitly overlapping taxonomy settings, where our method has a positive impact on both settings. The performance improvements demonstrate the generalizability of our proposed method across various taxonomy types. 
conclusion: 
 In this paper, we present a generic self-training framework for addressing the CCDA problem in semantic segmentation. Our proposed label alignment and novel class resampling methods effectively rectify pseudo labels and enhance the supervision of novel classes, offering a promising solution for both cross-class and crossdomain challenges. In comprehensive experiments, we have shown that our method archives considerable performance improvements when embedded into UDA approaches across various settings, such as different benchmarks and different taxonomy types. Importantly, our method exhibits versatility to diverse VLM-based implements, enhancing its flexibility for real-world applications. 
 
 Appendix:  


Decision: The AC goes through the paper, rebuttal and review comments. This paper got 3 borderline accept reviews. All reviewers acknowledge the motivation and novelty. Therefore, the AC thinks this paper can be accepted, but the authors should carefully improve the paper by leveraging the feedback.

Reviews: review: paper_summary: paper_summary: This paper proposes a Cross-Class Domain Adaptive (CCDA) framework for semantic segmentation, aiming to address the challenge of target domains containing both shared and novel classes that are either unlabeled or unseen in the source domain. This task is challenging because the absence of labels for novel classes hinders the accurate segmentation of both shared and novel classes in the source and target domains. To tackle this issue, the authors leverage the capabilities of Visual Language Models (VLMs), which can generate zero-shot predictions without requiring task-specific training examples.

Specifically, the main contributions of the paper include:

Label Alignment Method: A VLM-based label alignment method is proposed, which uses an uncertainty-driven threshold to relabel pseudo labels for novel classes, thereby enhancing the accuracy of segmentation.

Novel Class Resampling Method: A novel class resampling method is designed, employing an adaptively updated memory bank to effectively manage accurate VLM predictions and increase the sampling probability of novel classes, thus enhancing the supervision for novel classes.

Experimental Validation: The effectiveness and versatility of the proposed method are demonstrated through extensive experiments on various benchmarks, UDA frameworks, and VLM-based segmentation methods.

The paper also provides detailed descriptions of the proposed method's framework overview, problem statement, methodology, including specific steps for label alignment and novel class resampling, and showcases the performance of the method through experimental settings and results.

The significance of this work lies in providing an effective solution for handling common cross-class domain adaptation problems in the real world, especially in semantic segmentation tasks, where it can better manage the differences between the source and target domain classes.
strength_weakness: strength_weakness: Innovation: A novel Cross-Class Domain Adaptive (CCDA) method has been proposed, specifically addressing the challenge of the presence of novel classes in the target domain that are unmarked or unseen in the source domain, an issue that is less explored in existing literature.

Application of VLMs: The use of Visual Language Models (VLMs) is leveraged for zero-shot learning to tackle the problem of the lack of annotations for novel classes in the target domain. VLMs are capable of generating image-level predictions that match text descriptions, offering a potential solution for the segmentation of novel classes.

Label Alignment: Through the label alignment method, the paper introduces a strategy that utilizes VLMs to relabel pseudo labels, which helps to improve the model's segmentation accuracy for novel classes.

Novel Class Resampling: A novel class resampling mechanism has been designed, which dynamically manages the novel class proposals generated by VLMs through a memory bank and enhances the learning efficiency of novel classes with an adaptive updating strategy.

Uncertainty-Driven Threshold: The paper proposes an uncertainty-based adaptive threshold method to exclude noise in VLMs' predictions, thereby improving the accuracy of label alignment.
limitations: limitations: Dependency on VLMs' Performance: Since this method relies on Visual Language Models (VLMs) to generate pseudo labels for novel classes, the performance limitations of VLMs may affect the overall effectiveness of the framework.

Computational Resources: The computation of VLMs and the adaptive threshold may require substantial computational resources, which could limit the application of this method in environments with limited resources.
paper_summary: paper_summary: The paper addresses the problem of cross-class domain adaptation (CCDA) in semantic segmentation, where the target domain contains both shared and novel classes not present in the source domain. The proposed solution leverages visual language models (VLMs) to relabel pseudo labels for novel classes and uses memory banks to augment pseudo labels with accurate VLM predictions. The method is evaluated on multiple benchmarks and outperforms state-of-the-art approaches.
strength_weakness: strength_weakness: 1.The paper presents a novel approach that uses VLMs to address the CCDA problem in semantic segmentation.

2.The paper provide a detailed explanation of their proposed method and its underlying principles.
limitations: limitations: 1.Word spelling problem: Line85: VFMs->VLMs. 

2.In Figure 3, the terms "Student Net" and "Teacher Net" are mentioned. However, the paper does not provide a clear explanation of what "Teacher Net" specifically refers to, nor does it mention "Student Net" elsewhere in the text. This lack of clarity makes it difficult to understand the overall methodology.

3.The mention of the Genetic Segmentation Model in Figure 3 lacks clarity regarding the specific method employed in the text. Additionally, comparative experiments are absent. 

4.The paper needs clarification on the size and design of the Memory Banks used. Please provide more details for a better understanding of the system architecture.

5.The paper discusses the use of pseudo-labels in model training. However, the paper lacks a detailed analysis of the proportion of pseudo-labels and their impact on model accuracy.

Suggestions:
1.Could you please clarify why you chose to use VLMs for the CCDA problem? What are the advantages of using VLMs over other methods?

2.Can you discuss any potential limitations or drawbacks of your proposed method?
paper_summary: paper_summary: In addressing the emergence of novel categories in semantic segmentation domain adaptation, the paper leverages vision language model to facilitate zero-shot category prediction and employs confidence-based filtering to enhance the removal of unreliable predictions. Experimental results demonstrate that the proposed method exhibits performance improvements when integrated into various Unsupervised Domain Adaptation (UDA) semantic segmentation approaches. From a task perspective, semantic segmentation is a basic visual task, and domain adaptation is also a traditional visual problem. However, the paper starts from the annotation of semantic tags and uses a visual language model to bring new vitality to this traditional task. Although the semantic segmentation task cannot be called a multi-modal task, the Cross-Class Domain Adaptation method designed in the paper is consistent with the idea of ​​multi-modal research. Although I don’t know much about the field of semantic segmentation, I think the structure of the paper is reasonably arranged, the experiments are complete, and the logic is smooth. It is a qualified paper in the field of computer vision.
strength_weakness: strength_weakness: + The paper's utilization of vision language model's robust generalization capabilities for annotating unknown categories is intuitive and sound.
+ Using visual language models to annotate visual data and solve purely visual tasks takes advantage of multi-modal research.
+ The manuscript's writing style is easily comprehensible and coherent.
+ The methods compared in the paper's experiments are all recent state-of-the-art (SOTA) approaches, and their effectiveness is demonstrated on two distinct datasets.
limitations: limitations: + In the experiment from SYNTHIA to Cityscapes, the paper conducted experiments on DACS, DAFormer, and MIC, but in the experiment from SYNTHIA to IDD, only the experiment was conducted on DAFormer. Is there any reason for this?
+ Due to my limited familiarity with this specific domain, I may not be able to provide detailed technical feedback.
+ I will adjust my scoring based on the comments from other reviewers and rebuttals.