CLIP2UDA: Making Frozen CLIP Reward Unsupervised Domain Adaptation in 3D Semantic Segmentation

Authors: yao  wu
mingwei  xing
yachao  zhang
yuan  xie
yanyun  qu
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-27 09:47:11

Abstract: Multi-modal Unsupervised Domain Adaptation (MM-UDA) for large-scale 3D semantic segmentation involves adapting 2D and 3D models to a target domain without labels, which significantly reduces the labor-intensive annotations.
Existing MM-UDA methods have often attempted to mitigate the domain discrepancy by aligning features between the source and target data. However, this implementation falls short when applied to image perception due to the susceptibility of images to environmental changes compared to point clouds.
To mitigate this limitation, in this work, we explore the potentials of an off-the-shelf Contrastive Language-Image Pre-training (CLIP) model with rich whilst heterogeneous knowledge.
To make CLIP task-specific, we propose a top-performing method, dubbed \textbf{CLIP2UDA}, which makes frozen CLIP reward unsupervised domain adaptation in 3D semantic segmentation.
Specifically, CLIP2UDA alternates between two steps during adaptation: (a) Learning task-specific prompt. 2D features response from the visual encoder are employed to initiate the learning of adaptive text prompt of each domain, and (b) Learning multi-modal domain-invariant representations. These representations interact hierarchically in the shared decoder to obtain unified 2D visual predictions.
This enhancement allows for effective alignment between the modality-specific 3D and unified feature space via cross-modal mutual learning.
Extensive experimental results demonstrate that our method outperforms state-of-the-art competitors in several widely-recognized adaptation scenarios.
Code is available at: \textcolor{blue}{\url{https://github.com/Barcaaaa/CLIP2UDA}}.

Main paper: 
 introduction: 
 3D scene understanding is the foundation for many real-world applications, such as robotics [33] , autonomous driving [40] , and augmented reality [19] . Based on the LiDAR point cloud, 3D semantic segmentation is a critical task for scene understanding, which requires assigning semantic labels for each point. However, annotating large-scale datasets for training every new scenario is labor-intensive and time-consuming [38] , especially for the tasks demanding point-wise annotations.Currently, Multi-modal Unsupervised Domain Adaptation (MM-UDA) has been investigated in 3D semantic segmentation, which seeks to adapt 2D and 3D models to a target domain without labels. MM-UDA liberates the annotation of pair-wise 2D and 3D data in the target domain. The core idea of the recent MM-UDA methods [4, 13, 20, 26, 37, 41, 48] usually reinforces the complementation between images and point clouds via mutual learning, to mitigate the discrepancy in data distribution between the source and target domains (See Fig. 1 ). Albeit successful, these methods usually make the model under-adapted when applied to image perception due to the susceptibility of images to environmental changes compared to point clouds. Particularly with light changes, images lose plentiful context information at night. In this condition, allowing 3D prediction to mimic 2D prediction potentially degrades the performance. Generally, the scarcity of pre-training data hinders training a generalizable 3D model. Vision-Language Models (VLMs) [18, 27, 34] have emerged as the de-facto feature extractor in image classification, captioning, and segmentation nowadays. Such VLMs are trained on massive raw web-curated image-text pairs, achieving promising open-vocabulary recognition. Considering the text description as a domain-invariant prompt, this paper is committed to leveraging rich whilst heterogeneous knowledge from CLIP, allowing us to learn disentangled domain and category representation while avoiding the loss of context information. However, in earlier exploration, simply inheriting the pre-training weights of CLIP and training MM-UDA in a fine-tuning fashion would bring performance degradation. We found it is of utmost importance to avoid any unnecessary attempts to manipulate the visual representations in the space of CLIP. Such a combination may overfit source distribution and potentially lose pre-existing target information.To this end, two intuitive questions need to be solved: (i) How to borrow prior knowledge from frozen CLIP? and (ii) How to harness prior knowledge to boost 3D semantic segmentation for MM-UDA? To begin with, for issue (i), considering prompt learning is a popular practice in applying VLMs, we consider designing a task-specific text prompt to be well-tailored for describing domains. Moreover, inspired by MaskCLIP [50] , instead of deploying the visual encoder of CLIP as a task network, we serve it as a generalizable auxiliary network without updating its parameters, avoiding introducing additional domain gaps. After that, for issue (ii), we consider introducing inherent 3D attributes derived from LiDAR sensor, i.e., sparse depth and intensity information. Both of them contain spatial structure information and are less influenced by domain shift. By incorporating them in 2D shapes (perspective projection), we aim to learn vision-language-structure correlation to alleviate 3D performance degradation caused by substantial 2D domain shift.Accordingly, in this paper, we dig deeper into CLIP and propose a top-performing method, dubbed CLIP2UDA (See Fig. 1 ), which makes frozen CLIP reward unsupervised domain adaptation in 3D semantic segmentation. CLIP2UDA preserves the pre-existing target information from the CLIP model by alternating between two steps during adaptation: (a) Learning task-specific prompts and (b) Learning multi-modal domain-invariant representations. Concretely, for step (a), we introduce Visual-driven Prompt Adaptation (VisPA), using semantic names as the class token to make the data in the source and target domains as close to the label as possible. Meanwhile, to learn the visual distributions in the semantic space and obtain a distribution of prompts per class, VisPA leverages 2D features response from the visual encoder to initiate learning of task-specific text prompts, incorporating domain-specific and image-specific tokens. Nevertheless, relying solely on vision-language correlation is not enough to alleviate the limitation of unstable image perception in environmental changes. Thereupon, for step (b), we propose Textguided Context Interaction (TexCI), which conducts class-aware fusion and semantic-aware fusion across decoder layers to obtain the text-guided visual feature. This feature is then fused with image and attribute features, and interacts hierarchically in the shared decoder, resulting in unified 2D visual predictions. To summarize, CLIP2UDA tightly associates 3D features with point-wise unified 2D features, enhancing the performing capability of multi-modal domain-invariant representations in 3D scenes that benefit from vision-language-structure correlation.The main contributions of this paper are as follows:• We present CLIP2UDA, which exploits prompt learning to transfer the generalization capability of the CLIP model to MM-UDA. It preserves the pre-existing target information from CLIP and learns vision-language-structure correlation. UDA methods for 3D semantic segmentation can be grouped as uni-modality cases [16, 31, 39, [43] [44] [45] 49] and multi-modality cases [3, 4, 13, 20, 23, 26, 37, 41, 48] . For uni-modality, early methods [39, 49] exploit the generative adversarial network to mitigate domain shift caused by appearance and sparsity differences. Later on, Yuan et al. [44, 45] propose the adversarial network based on categorylevel and prototype-level alignments to address the mismatch of sampling patterns. Complete&Label [43] resolves UDA from the view of the 3D surface completion task. CosMix [31] and ConDA [16] construct an intermediate domain by utilizing joint supervision signals from both the source and target domains for self-training. Compared to uni-modal cases, multi-modal cases exploit the exclusive information of paired images and point clouds to complement each other. xMUDA [13] first provides a cross-modal mutual learning method for 3D semantic segmentation in UDA. To facilitate learning domain-robust dependencies, several multi-modal methods extend 2D techniques to learn the domain-invariant representations, such as adversarial learning [23, 26, 48] , contrastive learning [41] , and style transfer [20] . Based on these dependencies, SSE [48] presents a self-supervised exclusive learning mechanism from plane-to-spatial and discrete-to-textured. BFtD [37] employs cross-modal fusion representation to address imbalanced modality adaptability. Differently, we focus on how to leverage rich whilst heterogeneous knowledge from CLIP [27] to alleviate the 3D performance degradation caused by substantial 2D domain shifts. 
prompt learning: 
 Vision-Language Models (VLMs) integrate visual and textual inputs to achieve a more comprehensive understanding of the real world, leading to better performance in various tasks. Some of the popular VLMs are CLIP-based models [14, 51, 52] for classification. Recently, several studies have explored the use of prompt learning for dense predictions, which are for the topic of open-vocabulary segmentation in an annotation-free manner [24, 50] , supervised segmentation [17, 28] and weakly-supervised segmentation [22, 42] .For 3D scene understanding, there is limited research dedicated to employing prompt learning. Several works [7, 12, 46, 47] align 3D space with open-world language representation, facilitating zero-shot transfer in indoor and outdoor scenes. CLIP2Scene [6] first explores how to establish connections between point and text through self-supervised pre-training, which benefits fine-tuning for 3D downstream tasks. Chen et al. [5] leverage the strengths of CLIP and SAM [15] to supervise 2D and 3D networks simultaneously by introducing cross-modality noisy supervision.Nevertheless, none of the works explore CLIP for UDA in 3D semantic segmentation. To the best of our knowledge, our method is groundbreaking in exploiting CLIP to learn domain-invariant representation for MM-UDA. 
approach 3.1 preliminary: 
 3.1.1 Problem Definition. Given a source domain D 𝑆 = {(𝑋 2𝐷,𝑆 𝑖 , 𝑋 3𝐷,𝑆 𝑖 , 𝑌 3𝐷,𝑆 𝑖 )} 𝑛 𝑠𝑖=1 with 𝑛 𝑠 unlabeled 2D images and labeled 3D point clouds, and a target domainD 𝑇 = {(𝑋 2𝐷,𝑇 𝑖 , 𝑋 3𝐷,𝑇 𝑖 )} 𝑛 𝑡𝑖=1 with 𝑛 𝑡 unlabeled 2D images and 3D point clouds under the condition that the source and target data distributions are not equal. The source and target domains share the same label space and only the source point cloud has annotation 𝑌 3𝐷,𝑆 belonging to 𝐶 classes for each 3D point. The task is to learn a model 𝑓 : 𝑋 2𝐷,𝑇 , 𝑋 3𝐷,𝑇 → 𝑌 3𝐷,𝑇 that could predict the target 3D labels. [52] consists of a visual encoder F 𝑉 (•) and a text encoder F 𝑇 (•) based on CLIP, both jointly trained to map the image and text into a unified representation space. The learnable prompt given to F 𝑇 (•) is designed with the following form:EQUATIONwhere each [𝑉 ] 𝑚 (𝑚 ∈ {1, ..., 𝑀 }) is a learnable vector with the same dimension 𝐷 as textual embedding, 𝑀 is a hyperparameter that specifies the length of context tokens, and [𝐶𝐿𝑆] is the class token, which is converted into a low-cased byte pair encoding representation [32] . After that, textual embedding can be represented as 𝑒 𝑡 = F 𝑇 (𝑡), where 𝑒 𝑡 ∈ R 𝐶 ×𝐷 is the embedding for 𝐶 classes. Meanwhile, take ResNet [11] for example, there are 4 stages in total and we denote the feature maps as {𝑓 𝑖 } 4 𝑖=1 . Different from the original ResNet, a multi-head self-attention (MHSA) layer is performed to concatenate visual features f4 , 𝑓 4 of the last layer, where f4 is the global features after global average pooling of 𝑓 4 . After that, the global and local visual embeddings can be represented as 𝑧 𝑔 , 𝑧 𝑙 = MHSA( f4 , 𝑓 4 ), where 𝑧 𝑔 ∈ R 1×𝐷 and 𝑧 𝑙 ∈ R 𝐻 4 𝑊 4 ×𝐷 , 𝐻 4 ,𝑊 4 are the height and width of the feature map from the 4-th stage of the backbone, respectively.The overall framework of CLIP2UDA is depicted in Fig. 2 , which training alternately between two steps during adaptation: Visualdriven Prompt Adaptation (VisPA) and Text-guided Context Interaction (TexCI). Firstly, in VisPA, images 𝑋 2𝐷 are input to CLIP visual encoder F 𝑉 (•) to initiate learning of a task-specific text prompt, and then generate visual-driven text embedding via CLIP text encoder F 𝑇 (•) and Transformer decoder G 𝑇 (•, •). For the unified branch, the 2D segmentation task involves three types of visual inputs: image 𝑋 2𝐷 , sparse depth 𝑋 𝐷𝑒𝑝 , and intensity 𝑋 𝐼𝑛𝑡 . The former input is fed into the 2D encoder F 𝐼 (•). In contrast, the latter two inputs 𝑋 𝐷𝑒𝑝 and 𝑋 𝐼𝑛𝑡 are fed into the attribute encoder F 𝐴 (•) for acquiring a compact representation of 3D attributes in 2D space. Afterward, TexCI is introduced to learn multi-modal domain-invariant representation in the shared decoder G 𝐼 (•, •, •) and obtains point-wise unified features. Finally, after extracting 3D features output from the 3D network G 𝑃 (F 𝑃 (•)), we can bridge the cross-domain gaps by mutually learning between unified predictions and 3D predictions.Improving sentence structure through visual descriptions of natural objects helps reduce the disparity between textual and visual cues. Additionally, this principle holds in the context of autonomous driving scenarios. For instance, "A yellow car driving in the night scene." is more exact. However, incorporating rich text into the manual prompts is time-consuming, as it has to be based on trial and error, and does not guarantee an optimal prompt either. In this regard, VisPA is introduced to learn prompts for each class directly from the 2D visual domain to effectively encode the visual distribution, as opposed to the static prompting technique. To this end, we denote the prompt for one class with the class token [𝐶𝐿𝑆] given an image 𝑋 2𝐷 as:EQUATIONwhere [𝐷] 𝑆/𝑇 and [ Ṽ ] 𝑚 denote the Domain-specific Token (D-Token) and the Image-specific Token (I-Token), respectively. Both of them are dedicated to capturing visual context using a parameterefficient design, to boost the adaptability of text prompts. 𝑖=1 from four layers of F 𝑉 . In this work, ℎ 𝜙 is built with two linear layers corresponding to the output of an encoder layer that reduces the input dimension to the same as textual embedding.𝑚 , we further learn another lightweight neural network, called Meta-Net, to generate a conditional token for each input, which is then combined with the context prompt. In this work, the Meta-Net is built with a two-layer bottleneck structure (Linear-ReLU-Linear), with the hidden layer reducing the input dimension by 16×. Let ℎ 𝜃 (•) denote the Meta-Net parameterized by 𝜃 , each image-specific token is obtained by: [ Ṽ ] 𝑚 ← [𝑉 ] 𝑚 + 𝛾, where 𝛾 = ℎ 𝜃 (𝑧 𝑔 ).Embedding. After utilizing the visual distribution of domain to prompt text tokens, it is crucial to explicitly leverage visual features to drive textual embedding for obtaining more compact representations. Since the attention mechanism can capture the long-range dependency by using pair-wise affinities across all positions, we employ a Transformer decoder G 𝑇 [35] , which contains 𝑁 (=3 in our case) MHSA layers and multi-head cross-attention layers. G 𝑇 builds the interaction between visual embedding 𝑒 𝑣 = 𝑧 𝑔 ⊎𝑧 𝑙 and textual embedding 𝑒 𝑡 = F 𝑇 (𝑡 𝑣𝑖𝑠 ), and then generates refined textual embedding, where ⊎ is the concatenate operation. This implementation is written as:EQUATIONwhere ẽ𝑡 encourages textual embedding to mine the most correlated visual cues. Finally, we can obtain visual-driven textual embedding via a skip residual connection:EQUATION)where 𝛿 is a learnable parameter, which is initialized with a small value (𝑖.𝑒., 10 -4 ) to maximally preserve the language priors from the textual embedding.Failing the fine-tuning attempt with CLIP pre-trained weights, we turn to a solution that avoids overfit source distribution and potentially loses pre-existing target information. To this end, we relax from this constraint and benefit from more flexible architectures tailored for multi-modal semantic segmentation. That is, we employ the CLIP-based model as an auxiliary network, whose pre-trained parameters are frozen, and only update the learnable prompts 𝑡 𝑣𝑖𝑠 in VisPA. Specifically, as depicted in Fig. 3 , we devise TexCI for hierarchical multi-modal fusion in the shared decoder, which conducts class-aware fusion (C-Fus) and semantic-aware fusion (S-Fus) across decoder layers.Visual embeddings encompass global semantics about the entire image. We consider that they also carry the dense semantics of multiple objects, as the features of each pixel aggregate information from all other pixels in forwarding. Ideally, we hope to extract local features from CLIP to tightly associate with objects in a 3D scene. Specifically, the visual embedding 𝑒 𝑣 is leveraged to compute the pixel-text matching map with visual-driven textual embedding 𝑒 𝑣→𝑡 , which is defined as follows:EQUATIONwhere ê𝑣 and ê𝑣→𝑡 are the 𝑙 2 normalized version of 𝑒 𝑣 and 𝑒 𝑣→𝑡 along the channel dimension. and 𝑒 𝑣→𝑡 , S-Fus performs hierarchical multi-modal fusion by using visual embedding 𝑓 𝑖 as the Query and textual embedding 𝑒 𝑣→𝑡 as the Key and Value. Firstly, at each spatial location, S-Fus aggregates 𝑒 𝑣→𝑡 across the text dimension to generate a joint-modal affinity matrix 𝐽 𝑖 ∈ R 𝐶 ×𝐻 𝑖 𝑊 𝑖 via dot-product, which collects textual information most relevant to pixels of the current layer.EQUATION)where ⊗ means matrix multiplication. Secondly, leveraging the affinity matrix 𝐽 𝑖 , we can preliminarily estimate a set of text-guided visual feature 𝑉 𝑖 ∈ R 𝐻 𝑖 𝑊 𝑖 ×𝐷 𝑖 of each layer using the equation:EQUATIONwhere ⊙ means element-wise multiplication, 𝜔 𝑞 , 𝜔 𝑘 , 𝜔 𝑣 , 𝜔 𝑤 , and 𝜔 𝑜 are linear functions with trainable parameters.After that, we adopt the UNet-like decoder [29] as the shared decoder G 𝐼 in the unified branch. Due to the sparsity of LiDAR point clouds, in multi-modal interaction between points and pixels, low-resolution matching maps cannot establish effective one-to-one relationships with corresponding point labels. Hence, considering that the matching map retains the available vision-language prior knowledge, we simply concatenate upsampled 𝑆 𝑖 and 𝑉 𝑖 to the 𝑖-th image feature 𝑓 2𝐷 𝑖 and attribute feature map 𝑓 𝐴𝑡𝑡 𝑖 output from 𝑖-th layers of F 𝐼 and F 𝐴 respectively, explicitly incorporating language prior:EQUATIONwhere 𝑓 𝑢𝑛𝑖 𝑖 characterizes the multi-scale fusion feature, joint embedding during visual decoding. Consequently, by jointly learning multi-modal domain-invariant representations in G 𝐼 , a point-wise unified prediction P 𝑈 𝑛𝑖 can be defined to handle task and categories characteristics via the 2D classifier.The point-wise supervised segmentation loss of the source domain is formulated as follows:EQUATIONwhere main prediction P 𝑆 is either P 𝑈 𝑛𝑖,𝑆 or P 3𝐷,𝑆 . The goal of unsupervised learning across modalities is twofold. Firstly, we want to transfer knowledge from the unified modality to the 3D modality on the target-domain dataset. Secondly, we devise mutual learning on source and target domains, where the task is to estimate the prediction of other modalities. Same to xMUDA [13] , we choose the Kullback-Leibler divergence 𝐷 𝐾𝐿 (•||•) for the cross-modal loss L 𝑥𝑀 and define it as follows:L 𝑥𝑀 = 𝐷 𝐾𝐿 (P 𝑈 𝑛𝑖 ||P 3𝐷↦ →𝑈 𝑛𝑖 ) + 𝐷 𝐾𝐿 (P 3𝐷 ||P 𝑈 𝑛𝑖↦ →3𝐷 ), (10) where P 𝑈 𝑛𝑖 and P 3𝐷 is to be estimated by the mimicking P 3𝐷↦ →𝑈 𝑛𝑖 and P 𝑈 𝑛𝑖↦ →3𝐷 from auxiliary classifier, respectively. Note that for convenience, we disregard the domain notation 𝑆/𝑇 of superscript in prediction. Finally, the overall loss function is defined as:EQUATIONwhere 𝜆 𝑆 and 𝜆 𝑇 are weights trading off L 𝑥𝑀 on source domain and target domain inputs, respectively. 
experiments 4.1 datasets: 
 For evaluation, we use four public autonomous driving datasets, including three real scenarios: nuScenes [2] , SemanticKITTI [1] , A2D2 [9] and one synthetic scenario: VirtualKITTI [8] . For all real datasets, LiDAR and RGB cameras are synchronized and calibrated, allowing 2D-to-3D projection, and for the synthetic dataset, Virtu-alKITTI provides depth maps so we simulate LiDAR scanning via uniform point sampling. Furthermore, following [13] , we only use the front camera image and the corresponding LiDAR points.Our experimental scenarios cover typical real-to-real domain adaptation challenges like lighting changes (nuScenes: Day → Night), scene layout of country (nuScenes: USA → Singapore), and sensor setups (A2D2 → SemanticKITTI ). For the first two scenarios, we choose 6 merged classes while for the last scenario, we select 10 shared classes between two datasets. In addition, the syntheticto-real domain adaptation challenge also be considered (Virtu-alKITTI →SemanticKITTI, simulated depth, and RGB to real LiDAR and camera, with 6 merged classes). Details are provided in supplementary materials.For the 2D backbone, we use a modified version of U-Net [29] , which consists of a dual-branch ResNet-34 [11] as the 2D encoder and Attribute encoder and transposed convolutions with skip connections as the Shared decoder. It is worth noting that ResNet for the 2D encoder is pre-trained on ImageNet [30] , while the Attribute encoder is trained from scratch. Meanwhile, depth and intensity information are 3D attributes derived from LiDAR sensors. Therefore, Table 1 : Quantitative results (mIoU, %) on four settings. The best value of "2D+3D" is marked in red, and the second best value is marked in blue. " †" indicates the reproduced results. " 𝑃𝐿 " denotes retraining with pseudo labels. our method does not introduce extra datasets to ensure fairness in equal experimental conditions. For the 3D backbone, we use the official SparseConvNet [10] implementation. The voxel size is set to 5cm which is small enough to only have one 3D point per voxel.For the visual encoder of CLIP, all main experiments adopt CLIPpretrained ResNet-50, while the text encoder of CLIP is built on top of a Transformer [35] . Of note, we fix the parameters of text and visual encoders during training to preserve the natural language knowledge learned from large-scale pre-training. To reduce the computational costs, we project both the 2D visual embeddings and the textual embeddings to a lower dimension (256) before feeding into the Transformer module.We employ standard 2D/3D data augmentation and log-smoothed class weights on point-wise supervised segmentation loss to address the class imbalance. The batch size is set to 8. Our model is trained on real-to-real adaptation for 100k iterations. We utilize an iteration-based learning schedule where the initial learning rate is 0.001 and then it is divided by 10 at 80k and 90k iterations. For synthetic-to-real, the training is performed for 30k iterations, and the learning rate is divided by 10 at the 25k and 28k iterations. As regards the hyper-parameters, 𝜆 𝑆 and 𝜆 𝑇 in cross-modal loss are set to 1.0 and 0.1 on "Day→Night" and "USA→Sing.", 0.1 and 0.02 on "Virt.KITTI→Sem.KITTI" and "A2D2→Sem.KITTI" respectively, without performing any fine-tuning on these values. All experiments are conducted on a single NVIDIA RTX 3090 GPU.We compare our method with three typical 2D UDA methods, which can be easily extended to multi-modal UDA tasks. Moreover, six multi-modal UDA methods are discussed and they are roughly divided into two types: bridging the cross-modality gap, such as xMUDA [13] and BFtD [37] ; and bridging the cross-domain gap, such as AUDA [23] , DsCML [26] , Dual-Cross [20] , and SSE [48] . The comparison results for 3D semantic segmentation in mean Intersection over Union (mIoU) on the target testing data are shown in Tab. 1. Overall, CLIP2UDA achieves the best performance in all scenarios against the competitors, 𝑤 .𝑟 .𝑡 . ensemble result "2D+3D".The source-only model is the lower bound, which is not domain adaptation as it is only trained on the source-domain dataset. It is observed that our method brings a significant adaptation effect on all scenarios compared to the source-only model, with the gains of 10.8%, 5.8%, 18.2%, and 9.6% in mIoU, respectively. Compared with the baseline (xMUDA), our method exceeds it by large margins with gains of 6.7%, 4.6%, 12.2%, and 6.0% in mIoU. Compared to the best results in all multi-modal learning methods, it is observed that our method achieves 5.8%, 4.6%, 8.9%, 1.3% mIoU gains over all scenarios. Of note, our method typically yields a higher score, up to +1∼12% mIoU on each 2D/3D separate branch (See △ 𝐺𝑎𝑖𝑛 ). Especially in synthetic-to-real scenarios, it has been proven that point clouds with visual language priors are beneficial for addressing the adaptation problem in domains with significant discrepancy. Furthermore, cross-modal learning and self-training with pseudo-labels (PL) are complementary in their combination. When re-trained with PL, our model still achieves superior performance.In Fig. 4 , we visualize the segmentation results of four settings. Compared to xMUDA and BFtD, CLIP2UDA avoids erroneous and mingled predictions in many small (person) and large (sidewalk, vegetation) regions, showing versatility across all scenarios. 56.4%→58.1%/58.7%). Without VisPA, #5 still outperforms #2 by 0.8% and 0.5% in mIoU due to the densely integration of linguistic features and visual features. Ultimately, #6 combines all of our components to reach peak value.As shown in Fig. 5 , we elaborate on the effectiveness of 3D attributes and their combinations (line graph), as well as the effectiveness of 3D attributes for multi-modal interaction under the guidance of the CLIP model (bar graph). Firstly, we investigate whether unified features constructed solely based on 3D attributes would enhance performance. Experimental results indicate that feeding 3D attributes into a 2D network enhances the robustness of distribution changes between source and target domains of multi-modality. Subsequently, to combine the visual features extracted from depth and intensity maps, we design two feature-level fusion strategies: 1) F-Level: Independent data inputs to two attribute encoders, followed by concatenation of the output features; 2) D-Level: Input the fused data into a single attribute encoder, resulting in the output of multiattribute hybrid features. Compared to the former strategy, the improvement of "2D+3D" in the latter is remarked as less memoryconsuming and performance-boosted, which obtains the mIoU gain of 0.5%/0.6% and 0.5%/1.0%, respectively. This phenomenon shows that while striving to maintain the strengths of each modality, it is also essential to consider the intra-modality correlation.In this part, we illustrate the impact of prompt setting on the overall 2D performance of our method. As shown in Fig. 6 , the results demonstrate a consistent upward trend, indicating that incorporating different learnable prompts to extract visual-driven textual embeddings improves performance. This observation suggests that the model benefits from incorporating visual information from multiple layers, enabling it to capture more nuanced and discriminative features. The reason we speculate is that the class prompts are independent of domains and are shared across all images and point clouds, while the image and domain prompts are independent of classes and represent specific visual information. In Fig. 7 , we visualize confidence maps with and without appending prompts to demonstrate their effectiveness. 4.4.4 Roles of CLIP. In Tab. 3, we present results using the weights pre-trained CLIP model with fine-tuned and frozen strategy. In the fine-tuning stage, all models are utilized as 2D encoder F 𝐼 . Among them, "Ours w/ finetune" achieves text-guided visual features in place of image features in the shared decoder. Compared to this setup, "Ours w/ frozen" (𝑖.𝑒., CLIP2UDA) provides significant improvements of 1.1% mIoU on Day→Night and 1.5% mIoU on USA→Sing. We believe that our method inherently embeds local image semantics in its features as it learns to associate pixel and point contents with natural language descriptions, indicating real benefit to feature transfer from language-compatible 2D features to 3D features.To further test if our proposed method also works in the opposite adaptation direction, we run the experiments for "source-only", "xMUDA", "xMUDA 𝑃𝐿 ", and our methods "CLIP2UDA" and "CLIP2UDA 𝑃𝐿 " on nuScenes: Night→Day and nuScenes: Singapore→USA. As shown in Tab. 4, the experimental results come as no surprise. Remarkably, compared to the 3D branch, CLIP2UDA effectively alleviates the performance degradation caused by 2D domain shift. Meanwhile, when comparing 2D+3D results our CLIP2UDA improves mIoU by +6.3%/+2.7% with respect to xMUDA, and +6.2%/+2.0% with extra pseudo-label supervision. This demonstrates that regardless of what target domain is given, CLIP2UDA can harness the frozen CLIP to uncover its pre-existing domain recognition capabilities. 
conclusion: 
 In this work, we delve into the challenges faced by perceptual multi-models in the robustness of domain shifts in 3D scene understanding. To this end, we propose a top-performing method, dubbed CLIP2UDA, which makes frozen CLIP reward unsupervised domain adaptation in 3D semantic segmentation. Building upon CLIP, we introduce VisPA to leverage 2D feature response from the visual encoder to initiate learning of task-specific text prompts. Besides, we introduce TexCI to learn multi-modal domaininvariant representation in the shared decoder and obtain unified 2D visual predictions. We evaluate and discuss a comprehensive set of methods from related fields and study their adaptation across day-to-night, country-to-country, synthetic-to-real, and device-todevice datasets. Our findings underscore the need for more effective solutions in this region. 
 
 Appendix:  


Decision: Pros:
- Utilizing CLIP in for domain adaptive 3D Semantic Segmentation is novel.
- The performance is good.
- The paper is well-written and easy to understand.

Cons:
- The paper lacks comparisons to other methods that use CLIP for UDA.
- Some figures need to be improved.

Reviews: review: paper_summary: paper_summary: This paper leverages CLIP to mitigate 3D performance degradation caused by 2D domain shifts, i.e, environmental changes. This paper proposes a novel method named CLIP2UDA, which learns task-specific prompt and multi-modal domain-invariant representations with the frozen CLIP model.
strength_weakness: strength_weakness: 1. Utilizing CLIP in 3D Semantic Segmentation under domain shift is novel. 
2. Clear and concise writing, well-presented figures, formulas, and tables. 
3. Well-structured experiment setups with good analysis. 
4. The performance improvement is significant.
limitations: limitations: 1. In line 373, the example  "A yellow car driving in the night scene.” is more exact than "A car in the scene."  may not be suitable. Because prompts [V] is a learnable vector that may not have a specific meaning. 
2. The author should briefly introduce the comparison method in quantitative and qualitative comparison. 
3. How about the results on the ViT-based CLIP?

Despite the limitations that should be addressed, I don't believe they should impact the paper's consideration for publication.
paper_summary: paper_summary: The paper presents Clip2UDA as a possible approach for utilizing CLIP in multi-modal domain adaptation for 3D semantic segmentation.
strength_weakness: strength_weakness: The task is interesting, while the idea of leveraging CLIP as a possible prior is intuitive, given CLIP's success in image tasks.
limitations: limitations: Although the idea is intuitive and interesting, the key concern for this paper is its lack of novelty, or more specifically, the lack of novelty for using CLIP for UDA.

Leveraging frozen CLIP (or denoted just as CLIP) as a prior knowledge for downstream task is not a new idea. Whether for image/video, or for 3D data. Just to list a few for reference:
1. Rozenberszki, D., Litany, O., & Dai, A. (2022, October). Language-grounded indoor 3d semantic segmentation in the wild. In European Conference on Computer Vision (pp. 125-141). Cham: Springer Nature Switzerland.
2. Chen, R., Liu, Y., Kong, L., Zhu, X., Ma, Y., Li, Y., ... & Wang, W. (2023). Clip2scene: Towards label-efficient 3d scene understanding by clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7020-7030).
3. Zhou, Z., Lei, Y., Zhang, B., Liu, L., & Liu, Y. (2023). Zegclip: Towards adapting clip for zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11175-11185).
4. Wang, Y., Huang, S., Gao, Y., Wang, Z., Wang, R., Sheng, K., ... & Liu, S. (2023, October). Transferring CLIP's Knowledge into Zero-Shot Point Cloud Semantic Segmentation. In Proceedings of the 31st ACM International Conference on Multimedia (pp. 3745-3754).

The idea of leveraging CLIP is not groundbreaking. And in this paper, it seems that CLIP is only leveraged to provide additional text information towards the RGB stream of xMUDA, from which the method is built upon. As such, the native constraint of CLIP, which is its incapability of dealing with non-visual data (more specifically non-RGB-based data) still exists. Adding in a good representation can indeed improve performances, yet it seems more of a engineering skill rather than a technical novelty. The authors would need to state specifically how CLIP is used differently and differently from other CLIP+UDA methods.
paper_summary: paper_summary: This paper presents CLIP2UDA, a method that leverages prompt learning to transfer the CLIP's generalization capability to MM-UDA. It preserves the pre-existing target information from the frozen CLIP model and learns vision-language-structure correlation.
strength_weakness: strength_weakness: The paper introduces CLIP2UDA, a novel method leveraging a frozen CLIP model for unsupervised domain adaptation, significantly enhancing 3D semantic segmentation performance.

Extensive Experimental Validation: The method demonstrates superior performance across various challenging scenarios, outperforming state-of-the-art techniques in multiple adaptation contexts.
limitations: limitations: 1. The writing in the approach section needs improvement:

&emsp;	a. Due to the complexity of the method and the numerous modules, each module should be described from a more macro perspective. For instance, it is necessary to highlight the input of each module, the purpose of the transformation performed by the module, how the output is combined with other information, and how the modules/sub-modules collaborate. Additionally, Figure 2 should be more clearly organized, with the key modules proposed in this paper being prominently highlighted.

&emsp;	b. Given the numerous variables involved, they need to be appropriately labeled in the figure.

&emsp;	c. Why is C-Fus not shown in the figure? What is the relationship between C-Fus and S-Fus, and how are their results combined?

2. For I-token and D-token, can their attended information be presented through visualizations such as heatmaps?