UniDSeg: Unified Cross-Domain 3D Semantic Segmentation via Visual Foundation Models Prior

Authors: yao  wu
mingwei  xing
yachao  zhang
xiaotong  luo
yuan  xie
yanyun  qu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-29 11:43:32

Abstract: 3D semantic segmentation using an adapting model trained from a source domain with or without accessing unlabeled target-domain data is the fundamental task in computer vision, containing domain adaptation and domain generalization.
The essence of simultaneously solving cross-domain tasks is to enhance the generalizability of the encoder.
In light of this, we propose a groundbreaking universal method with the help of off-the-shelf Visual Foundation Models (VFMs) to boost the adaptability and generalizability of cross-domain 3D semantic segmentation, dubbed $\textbf{UniDSeg}$.
Our method explores the VFMs prior and how to harness them, aiming to inherit the recognition ability of VFMs.
Specifically, this method introduces layer-wise learnable blocks to the VFMs, which hinges on alternately learning two representations during training: (i) Learning visual prompt. The 3D-to-2D transitional prior and task-shared knowledge is captured from the prompt space, and then (ii) Learning deep query. Spatial Tunability is constructed to the representation of distinct instances driven by prompts in the query space.
Integrating these representations into a cross-modal learning framework, UniDSeg efficiently mitigates the domain gap between 2D and 3D modalities, achieving unified cross-domain 3D semantic segmentation.
Extensive experiments demonstrate the effectiveness of our method across widely recognized tasks and datasets, all achieving superior performance over state-of-the-art methods. Remarkably, UniDSeg achieves 57.5\%/54.4\% mIoU on ``A2D2/sKITTI'' for domain adaptive/generalized tasks. Code is available at https://github.com/Barcaaaa/UniDSeg.

Main paper: 
 introduction: 
 As deep learning technology develops by leaps and bounds [47, 57, 59 ], 3D scene understanding has become the foundation for many real-world applications, including autonomous driving, robotics, augmented reality, smart cities, etc. Based on the LiDAR sensor, 3D semantic segmentation is a critical task that provides an accurate and robust semantic prediction of the surrounding scenarios. However, annotating large-scale datasets for training every new scenario is labor-intensive and timeconsuming, especially for the tasks demanding point-wise annotations. These limitations hinder their practical applicability in real-world scenarios where acquiring high-quality labeled data.Currently, domain adaptive 3D semantic segmentation (DA3SS) [19, 32, 38, 45, 48, 51] and domain generalized 3D semantic segmentation (DG3SS) [21, 27, 49, 60] have been widely explored in autonomous driving scenes. Their difference lies in that the former seeks to narrow the domain gap between the source and target domain data without assigning 3D semantic labels, while the latter aims to learn a generic and robust model before being exposed to the target domain. Albeit successful, their applications in 3D semantic segmentation have primarily focused on generalizing or adapting between synthetic and real scenes or across different scene layouts. This leaves a gap in exploring a universal framework, enabling the generalization and adaptation of 3SS models across datasets.With the above considerations, this paper focuses on studying a universal framework for cross-domain 3D semantic segmentation. The essence of simultaneously solving cross-domain tasks is to enhance the generalizability of the encoder. Therefore, a generalizable 3D model with source-domain data discrimination power to the target domain is necessary. It is performed solely with access to source domain data, enabling the model to develop the ability to discriminate domain-agnostic and domainspecific features. Unfortunately, one limitation has arisen, the scarcity of 3D pre-training datasets hinders this endeavor. More recently, Visual Foundation Models (VFMs) [23, 34, 36] have emerged as the de-facto visual backbone in 2D image classification and segmentation. Such VFMs are trained on massive raw web-curated images, achieving promising open-vocabulary recognition. Hence, two natural questions are thrown up: (i) How to borrow 2D prior knowledge from VFMs? and (ii) How to harness 2D prior knowledge to boost 3D performance? To begin with, for issue (i), visual prompt tuning [16, 20] is a parameter-efficient strategy to exploit the representational potential of VFMs. We consider freezing the whole VFMs and only learn several trainable lightweight blocks as supplementary input, which inherit parameters of VFMs trained at scale to the maximum extent. After that, for issue (ii), we consider a more effective prompt tuning that introduces an extra depth-guided prompt space and extends the model input with the prompt, which could guide the generalization of powerful representations to achieve desirable performances.Accordingly, in this paper, we introduce UniDSeg, dig deeper into prompt tuning in the VFM-based encoder, and introduce a Learnable-parameter-inspired Mechanism to the off-the-shelf VFMs with frozen parameters. Our VFM-based encoder is designed to learn alternately between two lightweight modules: i.e., Modal Transitional Prompting (MTP) and Learnable Spatial Tunability (LST). The former depends on the transitional guidance from 3D-to-2D unnatural images, i.e., sparse depth, which exists in the prompt space before being fed into the encoder layer. The latter depends on the customized context length of vectors, which exists in the query space for seeking matched prompting after encoding in the layer. Hereby, depending on the number of VFM-based encoder layers involved, we place layer-wise MTP and LST blocks to take full advantage of semantic understanding of diverse levels and modalities. The proposed mechanism not only avoids any unnecessary attempts to manipulate the original visual space but also inherits the pre-existing target awareness from the VFMs to the maximum extent. Ultimately, by integrating the proposed two modules into a cross-modal learning framework, our method efficiently mitigates the domain gap and enables 2D and 3D models to learn domain-invariant representations.The key contributions of our work are summarized as follows: 1) Our method is groundbreaking in introducing the prompt-tuning concept into the universal model for DG3SS and DA3SS tasks.2) We propose a novel learnable-parameter-inspired mechanism to the off-the-shelf VFMs, which maximally preserves pre-existing target awareness in VFMs to further enhance its generalizability. 3) Extensive experimental results demonstrate the effectiveness of our method across widely recognized tasks and datasets, all achieving superior performance for DG3SS and DA3SS. 
domain adaptive 3d semantic segmentation: 
 In general, DA3SS seeks to narrow the domain gap between the source and target domain data, which can be grouped as uni-modal [61, 54, 48, 38, 56, 55, 24, 32] and multi-modal [19, 30, 35, 28, 58, 50, 45, 6, 5, 51, 13, 46] conditions. For uni-modality, early methods [48, 61] exploit the generative adversarial network to mitigate domain shift caused by appearance and sparsity differences. Later on, Yuan et al. [55, 56] propose the adversarial network based on category-level and prototype-level alignments to address the mismatch of sampling patterns. CosMix [38] and ConDA [24] construct an intermediate domain by utilizing joint supervision signals from both the source and target domains for self-training. 3D surface representation is also considered an effective method. Complete&Label [54] transforms domain adaptive task into a 3D surface completion task. SALUDA [32] learns an implicit underlying surface representation simultaneously on source and target data.Compared to uni-modality, multi-modality exploits the exclusive information of paired images and point clouds to complement each other. xMUDA [19] is a pioneering method of cross-modal mutual learning for DA3SS. To facilitate learning domain-robust dependencies, several methods extend 2D techniques to learn the 3D domain-invariant representations, such as adversarial learning [30, 35, 58] , style transfer [28] , and contrastive learning [50] . Based on these dependencies, SSE [58] presents a self-supervised learning mechanism from plane-to-spatial and discrete-to-textured representations. BFtD [45] presents cross-modal fusion-then-distillation to mitigate imbalanced modality adaptability. Recently, Segment Anything Model (SAM) [23] has presented its strong capability in generating semantics-aware regions, making it a suitable option for cross-modal interaction. MoPA [5] and VFMSeg [51] harness the knowledge priors learned from SAM to produce more accurate pseudolabels for unlabeled target domains.In contrast to DA3SS, where the inputs in the target domain, although without 3D labels, are accessible during the training process, DG3SS is evaluated on data from totally unseen target domains.The goal of DG3SS is to first learn as generic and robust representations as possible before being exposed to any target-domain data during training. Research on DG3SS has recently witnessed a surge, as highlighted in several studies, including uni-modal condition [21, 37, 39, 40, 49, 60] and multi-modal condition [27, 14] . Kim et al. [21] leverage sparsity invariant feature consistency at the feature level and semantic correlation consistency at the output level to constrain the model. Ryu et al. [37] and Sanchez et al. [40] incorporate multi-frame aggregation with 6-DoF ego-motions via randomized LiDAR configurations augmentation and label propagation, respectively. However, these methods are limited when 6-DoF ego-motion is unknown, and they need to be estimated using the off-the-shelf LiDAR SLAM method [2] . Recently, 3D representation under Bird's-Eye-View (BEV) has also been considered an effective method for learning domain-invariant features. LiDOG [39] introduces a dense top-down prediction auxiliary task and supervises it by employing BEV-view of semantic labels, while BEV-DG [27] introduces a BEV-view of cross-modal representation fusion to alleviate the domain gap. Particularly, 3D scenes also exist in several adverse weather conditions including fog, snow, and rain [49] . UniMix [60] leverages physically valid adverse weather simulation to construct a bridge domain and then blends it with samples of normal weather conditions.Our endeavor is tailored to designing a universal cross-domain multi-modal learning framework that enhances the performance of both DG3SS and DA3SS.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: In the appendix A.4, we point out the limitation of re-training DA3SS with pseudo-label (PL), with almost no impact on the 2D prediction.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminary: 
 Problem Definition. Given a source domain DS = {(X 2D,S i , X 3D,S i , Y 3D,S i )} ns i=1 with n s labeled data and a target domain D T = {(X 2D,T i , X 3D,T i )} nti=1 with n t unlabeled data under the condition that the source and target data distributions are not equal. For DA3SS, the task seeks to adapt models trained on the source domain D S to a target domain D T without labels. For DG3SS, the task aims to exploit the knowledge from the source domain to achieve generalization to the target domain, while the model remains unexposed to the target domain during training, i.e., D T is unseen. Both tasks are framed with the expectation of having paired images and point clouds and learning a mapping function f : X 2D,T , X 3D,T → Y 3D,T that could predict the target-domain 3D labels.Revisiting ViT. Given a pre-trained Vision Transformer (ViT) [10] model with L layers, an input is divided into M = HW P 2 fixed-size patches x m ∈ R H P × W P ×D , m = 1, 2, ..., M , where H and W are the height and width of the original input, P is the patch size, and D is the constant latent vector size through all of its layers. Each patch is then embedded into D-dimensional latent space:e m 0 = Embed(x m ), e m 0 ∈ R D . (1) We indicate the collection of 2D patch embeddings, E l-1 = {e m l-1 ∈ R D |1 ≤ m ≤ M },as input to the l-th ViT encoder layer L l (•). Specially, E pre 0 ∈ R M ×D learns beforehand together with class encoding E cls ∈ R 1×D and positional encoding E pos ∈ R (1+M )×D to create initial patch embedding E 0 . Hence, the whole ViT encoder is formulated as:EQUATIONEQUATIONwhere E l ∈ R (1+M )×D is the patch embedding output from L l (•), ⊎ denotes concatenation on the sequence length dimension. Each layer L l consists of several multi-head self-attention modules and feed-forward networks together with skip connection [17] and LayerNorm [1] . Samp. Overall Framework. As depicted in Fig. 1 , the overall framework of UniD-Seg is decomposed to a 2D network ) , and H (•) denote encoder, decoder, and classifier, respectively. The main insight of UniDSeg is to provide a universal framework that enhances the generalizability and adaptability of cross-domain 3D semantic segmentation. Thereby, we introduce a novel task-specific VFM-based encoder, which is guided by point-level prompts from 3D information. Formally, point cloudLearnable Prompts Learnable PromptsF 2D = E 2D • G 2D • H 2D and a 3D network F 3D = E 3D • G 3D • H 3D , where E (•) , G (•X 3Di is input to 3D network F 3D to generate 3D prediction, while image X 2D aux . In summary, we place layer-wise learnable blocks to take full advantage of semantic understanding of diverse levels and modalities, which inherits potential target information of VFMs into the current training model. Our method is parameter-efficient and could be directly deployed to various pre-trained transformer-based architectures without modifying the basic units.VFM-based Encoder. Since the feature extraction of VFMs still lacks task-specific information, freezing parameters may lose some domain-agnostic contextual semantic information from sourcedomain data or fine-tuning parameters may alter some pre-existing target representation from largescale pre-trained data. To provide a remedy, we reconsider the role of pre-trained models, balancing the capture of contextual information and the overfitting issue of source-domain data. As illustrated in Fig. 2 , we introduce a Learnable-parameter-inspired Mechanism, which provides a set of continuous embedding, i.e., 3D-to-2D transitional prompts and tunable deep queries. The former not only learns spatial distance perception prompts from point clouds but also learns invariance to sample perturbations. The latter ensures that the feature distribution of Ẽl will not be modified drastically, thus making better use of the pre-trained knowledge from VFMs. Only these prompts and queries are updated during fine-tuning, while parameters of other layers L l of the ViT encoder are kept frozen.Depending on the number of ViT layers involved, our VFM-based encoder is designed to learn alternately between two lightweight modules: Modal Transitional Prompting P G l (•, •) and Learnable Spatial Tunability T B l (•). The former depends on the transitional guidance from 3D-to-2D unnatural images, which exists in the prompt space before being fed into layer L l , while the latter depends on the customized context length of vectors, which exists in the query space for seeking matched prompting after encoding in layer L l . With deliberate design, the newer task-specific ViT encoder can be decomposed into: EQUATIONEQUATION)where ẼL is the final refined patch embedding output from our proposed VFM-based encoder. According to the DG or DA task, X 2D ∈ R H×W ×3 means source or target image and X Dep ∈ R H×W ×1 means source or target sparse depth. Note that as Eq. ( 5) shows, the prompt output from P G l is aligned to the patch embedding without contacting E cls .Modal Transitional Prompting. Previous methods [16, 20] have proved that visual prompt-tuning brings flexibility to the pre-trained VFMs for downstream tasks. However, their prompts like pixellevel perturbations or learnable vectors are black boxes with limited learning capacity, which cannot reliably explore convincing knowledge beneficial for cross-domain 3D semantic segmentation. Lee et al. [26] have proved that source data internally know much more about the world and how the scene is formed, called Privileged Information. Therefore, P G l is designed to capture 3D-to-2D transitional prior and task-shared knowledge of this information from the prompt space, which might be useful for cross-domain learning.To achieve this goal, following Eq. ( 1), a patch embed module borrowed from the plain ViT is re-initialized and applied to obtain a sequence of flattened patch embeddings E 2D 0 , E Dep 0 ∈ R M ×D . Particularly, sparse depth X Dep as a point cloud representation via perspective projection, presents an unnatural image. From the view of modal characteristics, it is easy to access and contains the prior spatial distance information that is lacking in 2D representation. From the view of deep encoding, it focuses on the scope of scenes at different receptive fields, tightly coupled with semantic information, so that their corresponding features have different contents when constructed. Note that X Dep only has the 1-channel to show the depth value in camera coordinates, we repeat the depth channel to make it equal to the number of RGB channels. In addition, considering that low-frequency bands of the amplitude spectrum tend to capture style information (or low-level statistics) [53] , we provide a simple way to perturb the amplitude spectra of the source image to ensure that the VFM is exposed to more variations in low-frequency components during training. The perturbed images can be obtained by using a low-frequency filter, and then Eq. ( 1) is also employed to generate corresponding flattened patch embedding E LF 0 ∈ R M ×D . After that, to effectively integrate prompt-tuning to visual embedding output from each frozen layer, we flexibly build a lightweight layer that contains exclusive MLP ϕ l (•) for adapting l-th encoder layer and shared MLP Φ(•) across whole ViT. This implementation is written as:EQUATIONwhereE P G l ∈ R M ×D is the 3D-to-2D transitional prompt output from P G l .MLP is built with a two-layer bottleneck structure (Linear-GELU-Linear) with the hidden layer reducing the input dimension by 3×.Learnable Spatial Tunability. To progressively improve feature generalization via 3D-to-2D transitional prompt, inspired by [31, 44] , T B l is introduced to bridge the discrepancy between the pre-training dataset and the target scene. To achieve this goal, after the l-th layer, T B l starts with a set of learnable tokens O l ∈ R K×D , where K is the length of the token and each token is initialized randomly. Considering the essential need in 3D semantic segmentation to discern multiple instances within a framed scene, T B l exploits an attention mechanism, which enables VFMs to seek matched prompting to the features of distinct instances, thereby assisting VFMs in adapting to the differences between pre-training and cross-domain data. Concretely, T B l adopts a dot-product operation to generate the affinity matrix J l , which captures the associations between prompted patch embedding L l ( Ẽl-1 ) and learnable tokens O l . Then, a SoftMax function is applied to align each patch with a unique instance. This implementation is written as:EQUATIONEQUATIONwhere O l,a ∈ R K×Dm and O l,b ∈ R Dm×D are constructed as low-rank matrices [18] , reducing the trainable parameters by learning pairs of rank-decomposition matrices. Note that the low-rank dimension D m ≪ D (D m = 32 in our case). Besides, we further process the embeddings through a down-projection layer with parameters W down ∈ R K×Dm followed by an up-projection layer with parameters W up ∈ R Dm×D , which is then element-wise added via the residual connection. After that, to enhance the flexibility in feature adjustment, T B l employs several layers to produce:EQUATIONwhere E T B l ∈ R M ×D is the tunable deep query output from T B l , δ 1 (•) and δ 2 (•) are linear layers.The point-wise supervised segmentation loss of the source domain is formulated as follows:EQUATIONwhere main prediction P S is either P 2D,S or P 3D,S , N and C being the number of points of the source point cloud and the number of classes, respectively.The goal of unsupervised learning across modalities is two-fold. Firstly, we want to transfer knowledge from 2D modality to 3D modality on the source-domain and target-domain dataset. Secondly, we devise mutual learning on the output level, where the task is to transfer the pre-existing target information of VFMs to a 3D model. Same to xMUDA [19] , we choose the Kullback-Leibler divergence D KL (•||•) for the cross-modal loss L xM and define it as follows:EQUATIONEQUATIONwhere P 2D,S and P 3D,S is to be estimated by mimicking predictions P 3D,S →2D,S and P 2D,S →3D,S from the respective auxiliary classifiers H 2D aux and H 3D aux . The same goes for target predictions. Ultimately, the overall loss function L DG of DG3SS and L DA of DA3SS are defined as:EQUATIONEQUATIONwhere λ S and λ T are the weights trading off cross-modal loss on source and target domain inputs.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
datasets: 
 For evaluation, we use four public autonomous driving benchmarks, including three real datasets: nuScenes [4] , SemanticKITTI [3] , A2D2 [12] and one synthetic dataset: VirtualKITTI [11] . For all real datasets, LiDAR sensor and RGB cameras are synchronized and calibrated, allowing 2D-to-3D projection, and for the synthetic dataset, VirtualKITTI provides depth maps so we simulate LiDAR scanning via uniform point sampling. Following DA3SS settings [19] , we exclusively utilize the front camera image and the corresponding LiDAR points that are projected onto it.Our experimental scenarios cover typical real-to-real single domain adaptation and generalization challenges like lighting changes (nuScenes: Day/Night), scene layout of country (nuScenes: USA/Sing.; nuScenes: Sing./USA), and sensor setups (A2D2/sKITTI; A2D2/nuScenes). Note that, the source and target classes are coincident. For the first three scenarios, we choose 6 merged classes while for the last two scenarios, we select 10 and 8 shared classes between two datasets. In addition, the synthetic-to-real domain adaptation and generalization challenges are also considered (vKITTI/sKITTI, simulated depth, and RGB to real LiDAR and camera, with 6 merged classes). Details of the class partition are provided in supplementary materials.For the 2D backbone, we use the visual encoders of CLIP [36] and SAM [23] , a large-scale pretraining model. Our selection of Vision Transformer [10] includes ViT-Base (ViT-B) and ViT-Large (ViT-L) architectures. Then, we utilize the MMSegmentation [8] codebase for the decoder head, SemanticFPN [22] , a widely-used segmentation head, is integrated with the visual encoder that serves as the 2D backbone. Meanwhile, depth information is a 3D attribute derived from LiDAR sensors, without the need to introduce additional datasets, thus ensuring fairness under equal experimental conditions. For the 3D backbone, we employ SparseConvNet [15] with a U-Net architecture in Sparse Convolution Library [9] . The voxel size is set to 5cm in the 3D network. This voxel size ensures that each voxel contains only one 3D point, maintaining a level of granularity suitable for the task.Our model is trained on "nuScenes:Day/Night", "A2D2/sKITTI", and "A2D2/nuScenes" for 100k iterations. We utilize an iteration-based learning schedule where the initial learning rate is set to 1e-3 except for the 2D encoder which is 1e-4, and then it is divided by 10 at 80k and 90k iterations. For "nuScenes:USA/Sing." and "nuScenes:Sing./USA", the training is performed for 60k iterations, and the learning rate is divided by 10 at the 40k and 50k iterations. For vKITTI/sKITTI, the training is performed for 30k iterations, and the learning rate is divided by 10 at the 25k and 28k iterations. The batch size is set to 8. As regards the hyper-parameters, following [19] , λ S and λ T in cross-modal loss are set to 1.0 and 0.1 on "nuScenes:Day/Night", "nuScenes:USA/Sing.", and "nuScenes:Sing./USA", 0.1 and 0.01 on "vKITTI/sKITTI", "A2D2/sKITTI", and 'A2D2/nuScenes" respectively, without performing any fine-tuning on these values. All experiments are conducted on NVIDIA RTX 3090. We compare the proposed method with three classic 2D domain adaptive methods, which can be easily extended to multi-modal conditions. Moreover, eight multi-modal DA3SS and two multi-modal DG3SS are discussed. As shown in Tab. 1, we tabulate the comparison results in mean Intersection over Union (mIoU, %) on the target testing data. Note that not all datasets provide image labels. Thus the quantitative evaluation of 2D semantic segmentation depends on the 3D-2D corresponding point-wise prediction. Overall, our method achieves the best performance on all scenarios against the competitors, w.r.t. ensemble result "xM", except for DA3SS on "nuScenes:Day/Night" gets the second best. As shown in Fig. 3 , we visualize the DG3SS results of four settings. By the merit of the visual foundation models, our method can segment detailed objects very well. From top to bottom, the focused areas are the sidewalk near the drivable surface, the trunk of a tree, the profile of the road, and most importantly, bicycles safely riding on the road.In terms of DG task, compared with baseline (xMUDA) and MM2D3D, our method still achieves remarkable results, exhibiting 3.7%/1.4%, 10.4%/1.7%, 21.0%/15.8%, and 12.8%/10.8% mIoU improvement. It is worth noting that our method, without exposure to any target-domain data, shows performance that is close to or even surpasses that of most DA3SS methods (e.g., the result of Ours-DG is comparable to VFMSeg on "nuScenes:USA/Sing."). All results demonstrate that utilizing the powerful open-vocabulary recognition capability of VFMs is beneficial for addressing generalization problems in domains with significant discrepancies.The source-only model is the lower bound, which is not domain adaptive as it is only trained on the source-domain data. It is observed that our method brings a significant adaptation effect on all scenarios compared to the source-only model, with the gains of 4.7%, 7.9%, 19.8%, and 17.1% in mIoU, respectively. Compared with baseline (xMUDA) for all DA competitors, our method exceeds it by large margins with gains of 3.5%, 3.8%, 13.8%, and 13.5% in mIoU. Particularly, on the "A2D2/sKITTI" scenario, our method typically yields a higher score compared to the best "xM" in VFMSeg (57.5% vs. 50.0%). that due to the fixed and relatively small number of trainable parameters in the decode head, the count of trainable parameters presented in the tables is focused solely on the visual architecture. In this setup, we separately select CLIP:ViT-B and CLIP:ViT-L backbone with fine-tuning, freezing, and our proposed Learnable-parameter-inspired Mechanism for analysis. The results indicate that frozen VFMs outperform previous DA3SS methods without specialized design. In addition to "nuScenes:Day/Night", VFMs with full parameter fine-tuning exhibit enhanced performance relative to their frozen counterparts because the 3D modality is less sensitive to light, thus dominating in ensemble prediction direction. Remarkably, our method achieves even superior generalization capabilities, surpassing the full parameter fine-tuning 0.5∼2.7% mIoU with merely ∼2% trainable parameters compared to the original backbone. This ablation experiment demonstrates that our method can inherit the pre-existing target awareness from the VFMs to the maximum extent.Evaluation of Different Components. As shown in Tab. 3, we train four models on two scenarios for DG3SS, including (1) Frozen parameter of ViT backbone; (2) only performing MTP in the frozen ViT by using prompt tuning, leading to a significant mIoU boost (1.3% and 1.8%). This highlights the robustness of the 3D-to-2D transitional prompts in cross-domain learning; (3) only performing LST in the frozen ViT by customizing a learnable token, demonstrating that learnable tokens can encourage the model to learn domain-invariant representations (increasing 2.1% and 2.4% mIoU); (4) combining two proposed components to reach peak value.Evaluation of Learnable Token Length. We illustrate the impact of learnable token length K on the overall 2D performance of UniDSeg with the ViT-B backbone for DG3SS. As shown in Fig. 4 , the results demonstrate a consistent upward trend. When K is set to 100, the relatively small training parameters and high performance make it the preferred choice for subsequent experiments. Meanwhile, this observation suggests that the model benefits from incorporating visual information from multiple layers, enabling it to capture more nuanced and discriminative features.Evaluation of Different 2D and 3D Backbones. The motivation of this work is to study a universal framework based on VFMs to enhance the generalizability and adaptability of cross-domain 3D semantic segmentation, demonstrating the effectiveness of the visual foundation model priors. However, CLIP is not designed for common surveillance-relevant tasks like semantic segmentation. Hereby, we dedicate ourselves to verifying its effectiveness on VFMs designed for segmentation tasks such as SAM [23] . Firstly, in Tab. 4, we evaluate the effect of SAM as the 2D backbone. It is observed that SAM-based UniDSeg exhibits better performance on "USA/Sing" scenario, with a "xM" gain of 0.3% on DG and 0.4% on DA. Then, in Tab. 5, we evaluate the effect of applying different training strategies to the SAM-based model. It is observed that our Learnable-parameter-inspired Mechanism for tuning the SAM-based encoder can achieve 1.8% and 1.9% "xM" gain compared to fine-tuning strategy. In addition, we evaluate UniDSeg on another 3D backbone, i.e., MinkowskiNet, making it more convincing as a "universal" framework for cross-domain 3D semantic segmentation. Evaluation of Computation Cost. In Tab. 7, we have reported model parameters of CLIP: ViT-B, CLIP: ViT-L, and SAM: ViT-L for the VFM-based encoder along with all trainable parameters. Note that, the entire ViT backbone in our VFM-based encoder is frozen during downstream training for DA3SS and DG3SS. Only two layer-wise learnable blocks, MTP and LST, are trainable. It is obvious that our method requires only 1-2% parameters optimization to exceed the fine-tuning results. 
conclusion: 
 In this work, we delve into a universal method that can enhance the adaptability and generalizability of cross-domain 3D semantic segmentation, dubbed UniDSeg. For this purpose, we introduce a learnableparameter-inspired mechanism to the off-the-shelf VFMs with frozen parameters, which maximally preserves pre-existing target awareness in VFMs, further enhancing the generalizability of VFMs.Our method achieves state-of-the-art performance in both DA3SS and DG3SS, as demonstrated by extensive experiments on different scenarios. We believe that our work will inspire a deeper investigation of cross-domain 3D semantic segmentation in autonomous driving. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [No] Justification: This paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: This paper and code URL are anonymous. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: Error by Ours Groundtruth The proposed method can also work reliably in handling fully-supervised large-scale 3D semantic segmentation. As shown in Tab. 9, we simply compare the results of some typical uni-modal and multimodal methods, including MinkowskiNet [7] , SPVCNN [42] , Cylinder3D [62] , and 2DPASS [52] , on the publicly available SemanticKITTI validation set. According to the official setting in 00-10 sequences, sequence 08 is the validation set, while the remaining sequences are the training set.In this experimental setup, we make some slight modifications to the training strategy. Considering that fully-supervised learning assumes the training and validation/testing sets are independent identically distribution, without involving domain shift problems in cross-domain learning. (i.e., avoid overfitting the source-domain data distribution). Therefore, when training our model, we replace the 2D backbone in 2DPASS with our VFMs backbone and fine-tune its parameters in the same way. Moreover, we reduce the learning rate of the 2D backbone to lr = 0.024 while keeping the learning rates of the other modules lr = 0.24 and the number of training epochs ep = 64 unchanged to ensure the fairness of the experiment. Similar to 2DPASS, we tabulate the validation results with and without Test-Time Augmentation (TTA) and set the number of views to 12 as the default. Some visualization results are shown in Fig. 5 . 2DPASS has a higher error recognizing small objects and region boundaries, while our method recognizes small objects better thanks to the knowledge prior of visual foundation models.Source-free domain adaptation (SFDA) seeks to learn models where the vendor can trade only the source model and the client can perform target adaptation without accessing source-domain data. Drawing inspiration from Jogendra et al. [25] , SFDA enables partition into two tasks: (a) source-only domain generalization and (b) source-free target adaptation. Hereby, towards the former, we utilize UniDSeg to achieve models with generalization capability. Towards the latter, we utilize target 3D pseudo-labels (PL) obtained from the source model for self-training. When selecting PL from the target-domain data, we consider the ensemble result "xM" as the fusion PL to supervise both the 2D and 3D branches. As shown in Tab. 10, we adopt conventional consistency learning and pseudo-label self-training to address the source-free domain adaptive 3D semantic segmentation For (a), similar to MM2D3D, we replace the 2D backbones with two ViT-B models. The sparse depth is fed into the ViT-B that trains from scratch, while the image is fed into the ViT-B pre-trained on large-scale datasets. After that, we can obtain encoded output by concatenating two representations. For (b), that is the simple architecture of only sparse depth as prompt for fine-tuning. Tab. 12 shows the comparison results of the aforementioned learning manners. Compared to (a), (b) not only requires 0.6% fine-tuning parameters of the 2D backbone but also achieves superior segmentation performance, making it flexible to expand to various 3D semantic segmentation tasks with multi-modal learning. In this part, we demonstrate more qualitative results of our proposed UniDSeg to illustrate the effectiveness of our DA3SS framework. The corresponding qualitative results from 2D and 3D predictions will also be provided (See Figs 7, 8, 9, and 10) . Note that the prediction differences are signed in the red rectangle. Justification: We claim our contribution, that is, a universal framework to enhance the adaptability and generalizability of cross-domain 3D semantic segmentation.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.------------- 


Decision: This paper receives ratings with 1 accept, 1 borderline accept, and 2 borderline rejects. Most reviewers like the idea of bringing prompt tuning in to the domains of DG3SS and DA3SS and the extensive experiments to verify the proposed method. The main concerns raised by the reviewers including the use of CLIP as the backbone, novelty compared to existing 3SS methods that utilize VFMs, and marginal improvement against existing methods. The authors provide very detailed rebuttal to address the raised concerns. After reading the paper, reviews, and rebuttal, AC finds that the raised concerns are mostly addressed by the rebuttal with some nice ablation studies, and the improvement against the existing method is non-trivial. Given the novelty of the proposed methods, AC believes that the paper is ready for publication. The authors are encouraged to make the above mentioned necessary changes to the best of their ability. We congratulate the authors on the acceptance of their paper!

Reviews: review: paper_summary: paper_summary: The manuscript proposes a universal method with the help of off-the-shelf Visual Foundation Models (VFMs) to boost the adaptability and generalizability of cross-domain 3D semantic segmentation, dubbed UniDSeg. The proposed method focus on learning visual prompt for 3D-2D transitional prior and deep query.
strength_weakness: strength_weakness: 1.What does Samp. in Figure 1 means, the corresponding explanation is not provided in the manuscript. 
2.Why choose CLIP as 2D backbone, since there is no language information for the task, other Visual Foundation Models (e.g. Swin-Transformer, SAM, …) seem more suitable for the 2D information extraction. 
3.In lines 148-152, the author mention “The main insight of UniDSeg is to provide a universal framework that enhances the adaptability and generalizability of cross-domain 3D semantic segmentation.” Though SparseConvNet is examined as 3D network, it would be more convincing as “universal” if authors could provide results utilizing other 3D backbones.
questions: questions: Please refer to the Weaknesses Section.
limitations: limitations: The authors have clearly discussed the potential limitations of this work.
paper_summary: paper_summary: This paper proposes a cross-domain 3D semantic segmentation model which utilizes off-the-shelf visual foundation models to boost the adaptability and generalizability. Two key designs are described to help the cross-domain task, e.g., visual prompt learning and deep query learning. Extensive experiments have been reported in the paper to illustrate the strength.
strength_weakness: strength_weakness: 1. The improvement in both quantitative and qualitative results does not seem to be strong enough.  
2. Does not analyze the reasons behind the SOTA performance, especially compared to models like VFMSeg which also exploit the power of VFM. I hope more detailed comparison and insight can be given.  
3. The motivation is not clear enough. Though the paper points out two natural questions when considering using VFM, some works have already used VFM to help with segmentation. Then the motivation should include how to improve upon these works.  
4. As the model uses CLIP as the backbone, why not try a more advanced model?   
5. The writing needs to be improved. For example, in line 29, the last sentence is not finished. Line 37, DG3SS does not require accessing target domain data, so the logic here is not convincing. Line 42, what do you mean by “source-domain data discrimination power”? what is the relation to your method? Line 53, what is “spatial-temporal synchronized prompt space”? Line 65, what is “pre-existing target awareness”?
Some detailed explanations could be seen in the Questions.
questions: questions: 1. I wonder about the motivation behind exploring the segmentation task in DA. As the paper has mentioned some foundation models like SAM, I want to ask that is it still meaningful to research the segmentation task in DA. A foundational segmentation model like SAM seems to be general enough (What's SAM's performance on the datasets mentioned in the paper?). Even if it's not powerful enough, for segmentation which is a kind of "high-level" task, it seems to be easier to directly improve the foundation model to achieve a real general segmentation model without any adaption.   
2.  Other works like VFMSeg produce more accurate pseudo-labels to help the training, but in the limitation mentioned in this paper, producing pseudo-labels cannot achieve many performance improvements. I wonder what is the difference here.
limitations: limitations: As mentioned in the limitation of the paper, not being able to exploit the ability to produce accurate pseudo labels of VFM would constrain the capacity of the model when it comes to a larger scale.
paper_summary: paper_summary: This paper introduces the prompt-tuning concept into DG3SS and DA3SS, and proposes a learnable parameter heuristic mechanism for the off-the-shelf VFM. Modal Transitional Prompting is proposed to capture 3D-to-2D transitional prior and task-shared knowledge from the prompt space. Learnable Spatial Tunability is constructed to the representation of distinct instances driven by prompts in the query space. Extensive experimental results demonstrate the effectiveness of the proposed method on widely recognized tasks and datasets.
strength_weakness: strength_weakness: 1. The method lacks theoretical analysis. In Section 3.2, Modal Transitional Prompting, line 193, the author mentioned that "PG is designed to capture 3D-to-2D transitional prior and task-shared knowledge from the prompt space." However, the designed method only uses learnable parameters, which is not related to the 3D-to-2D transitional prior and the promotion of 3D domain adaptation. This section only explains the steps of the method and lacks theoretical explanation of the method.

2. In Section 3.2, Learnable Spatial Tunability, the author mentioned that it was inspired by Rein [1]. However, it seems that the proposed LST has a similar design to Rein, which casts doubt on the novelty of LST. The modification of Rein lacks innovation and theoretical analysis. This inevitably makes people think that the good performance of LST is due to the effectiveness of Rein-like components.

3. In the experimental section, VFM is only verified based on the classification model CLIP [2]. However, the CLIP paper [2] states that “Additionally, CLIP is not designed for common surveillance-relevant tasks like object detection and semantic segmentation. This means it has limited use for certain surveillance tasks”. The proposed method is dedicated to segmentation tasks, but it is only verified on the classification model CLIP. As a pioneer method, it is recommended to verify its effectiveness on VFMs designed for segmentation tasks such as SAM [3] and SEEM [4].

[1]Wei, Zhixiang, Lin Chen, Yi Jin, Xiaoxiao Ma, Tianle Liu, Pengyang Ling, Ben Wang, Huaian Chen, and Jinjin Zheng. "Stronger Fewer & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28619-28630. 2024.

[2]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748–8763, 2021.

[3]Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4015–4026, 2023. 1, 2, 3, 6, 11

[4]Zou, Xueyan, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. "Segment everything everywhere all at once." Advances in Neural Information Processing Systems 36 (2023).
questions: questions: 1. In Section 3.2, Modal Transitional Prompting, line 199, the author mentioned "From the view of deep encoding, it focuses on the scope of scenes at different scales, so that their corresponding features have different content representations when constructed." Why does sparse depth have scenes at different scales?
2. Why does the model require multiple layers of PG and TB? Experiments are needed to verify the impact of the number of layers on the results.
3. How long does it take from the start of training to completion?
4. In Table 1, caption introduction: "Avg" is obtained by averaging the predicted probabilities from the 2D and 3D networks. However, "Avg" is ambiguous here. This leads to misleading averages of 2D and 3D mIou results, but this may not be the case in reality. It is recommended to replace "Avg" with another representation.
limitations: limitations: As above
paper_summary: paper_summary: This paper presents UniDSeg, a universal approach that enhances the adaptability and generalizability of cross-domain 3D semantic segmentation. To achieve this, we propose a learnable parameter-inspired mechanism for off-the-shelf VFMs with frozen parameters. This mechanism maximally preserves the pre-existing target awareness in VFMs, thereby further enhancing their generalizability.
strength_weakness: strength_weakness: 1.	Since the backbone is modified from xMUDA, the oracle performance should also be provided.
2.	The task settings in this study are derived from xMUDA (TPAMI), while the prompt tuning and corresponding experimental settings are based on VPT. Therefore, this proposed method can be considered somewhat incremental.
questions: questions: 1.	Please explain why token length is equal to 100, the best performance can be obtained. However, it is important to note that the prompt tuning process may not be entirely stable. To address this, all experiments should be conducted multiple times, and the average performance along with the standard deviation also should be computed and reported.
2.	Please explain the main contribution compared with “Learning to Adapt SAM for Segmenting Cross-domain Point Clouds”.
3.	Since VFM and ViT are utilized as encoder, the model parameters and computational costs should be comprehensive reported.
4.	Did the authors notice that when the point labels projected onto 2D images, there are slight mismatch between these labels and 2D image pixels? Please discuss the potential reasons.
limitations: limitations: 1.	The authors have addressed the limitations and potential negative societal impact of their work.

Comments: title: Response to Authors
Reply to Q2
Official Comment by Reviewer T5Kw
Reviewer response
comment: The authors' response has clearly addressed my concerns. After checking the peer review comments and the author's responses, I decided to raise the given score for this work.
Thank you for your reminder. We have answered the question of Reviewer 4rFB.
**Q**: Q2 mentioned that the design of LST is similar to Rein [1]. Please explain the differences between them.

**A**: Due to character limit constraints, we regret to have overlooked this question. The following is an explanation of the problem:

**Similarity**: Both of them use the learnable tokens and calculate affinity matrices between visual inputs and tokens.

**Difference**: Structure. Rein generates a low-rank token sequence, while our method additionally processes the visual features through a down-projection linear layer followed by an up-projection linear layer, which are then element-wise added to the residual-connected features.

Following table shows simple experiment on "USA/Sing." scenario. Due to rebuttal time limitation, we will supplement all analysis between Rein and our method subsequently, including algorithm, diagram, and experimental results.

| Task | Method | Visual Backbone | 2D | 3D | Avg |
|------|:-----:|:-----:|:-----:|:-----:|:-----:|
| DG | Use Rein | CLIP: ViT-B | 63.3 | 64.6 | 71.2 |
| DG | Use LST | CLIP: ViT-B | 63.8 | 64.7 | 71.5 |
Reviewer 4rFB mentions a new question, please carefully answer it since the discussion is about to end.
Q2 mentioned that the design of LST is similar to Rein [1]. Please explain the differences between them.
I've reviewed the authors' responses and appreciate their engagement. I will stay in touch for further discussion as we approach the final rating.