UniDSeg: Unified Cross-Domain 3D Semantic Segmentation via Visual Foundation Models Prior
Authors: Yao Wu
Mingwei Xing
Yachao Zhang
Xiaotong Luo
Yuan Xie
Yanyun Qu
Venue: NeurIPS 2024 poster (2024)
Keywords: Domain Generalization
Domain Adaptation
3D Semantic Segmentation
Visual Foundation Models
Primary area: machine_vision

Abstract: 3D semantic segmentation using an adapting model trained from a source domain with or without accessing unlabeled target-domain data is the fundamental task in computer vision, containing domain adaptation and domain generalization.
The essence of simultaneously solving cross-domain tasks is to enhance the generalizability of the encoder.
In light of this, we propose a groundbreaking universal method with the help of off-the-shelf Visual Foundation Models (VFMs) to boost the adaptability and generalizability of cross-domain 3D semantic segmentation, dubbed $\textbf{UniDSeg}$.
Our method explores the VFMs prior and how to harness them, aiming to inherit the recognition ability of VFMs.
Specifically, this method introduces layer-wise learnable blocks to the VFMs, which hinges on alternately learning two representations during training: (i) Learning visual prompt. The 3D-to-2D transitional prior and task-shared knowledge is captured from the prompt space, and then (ii) Learning deep query. Spatial Tunability is constructed to the representation of distinct instances driven by prompts in the query space.
Integrating these representations into a cross-modal learning framework, UniDSeg efficiently mitigates the domain gap between 2D and 3D modalities, achieving unified cross-domain 3D semantic segmentation.
Extensive experiments demonstrate the effectiveness of our method across widely recognized tasks and datasets, all achieving superior performance over state-of-the-art methods. Remarkably, UniDSeg achieves 57.5\%/54.4\% mIoU on ``A2D2/sKITTI'' for domain adaptive/generalized tasks. Code is available at https://github.com/Barcaaaa/UniDSeg.

UniDSeg: Unified Cross-Domain 3D Semantic Segmentation via Visual Foundation Models Prior
Yao Wu', Mingwei Xing", Yachao Zhang's Xiaotong Luo', Yuan Xie Yanyun Qu 1.23.
School of Informatics, Xiamen University
2 Institute of Artificial Intelligence, Xiamen University
3 Key Laboratory of Multimedia Trusted Perception and Efficient Computing,
Ministry of Education of China, Xiamen University
School of Computer Science and Technology. East China Normal University Chongging Institute of East China Normal University wuyao@stu amu.edu. cn, yyquεxmu. edu. cn
Abstract
3D semantic segmentation using an adapting model trained from source domain with or without accessing unlabeled target-domain data is the fundamental task in computer vision, containing domain adaptation and domain generalization. The essence of simultaneously solving cross-domant tasks is to enhance the general- izability of the encoder. In light of this, we propose groundbreaking universal method with the help of off-the-shelf Visual Foundation Models (VFMs) to boost the adaptability and generalizability of cross-domain 3D semantic segmentation, dubbed UniDSeg. Our method explores the VFMs prior and how to harness them, aiming to inherit the recognition ability of VFMs. Specifically, this method troduces layer-wise Icarnable blocks to the VFMs, which hinges alternately learning two representations during training: (i) Learning visual prompt. The 3D-to-2D transitional prior and task-shared knowledge is captured from the prompt space, and then (ii) Learning deep query. Spatial Tunability is constructed to the representation of distinct instances driven by prompts in the query space. Integrat- ing these representations into a cross-modal learning framework, UniDSeg effi- ciently mitigates the domain gap between 2D and 3D modalities, achieving unified cross-domain 3D semantic segmentation. Extensive experiments demonstrate the effectiveness of method across widely recognized tasks and datasets, all achiev- ing superior performance over state-of-the-aat methods. Remarkably, UniDSeg achieves 57.5% mloU on "A2D2/sKITTI" for domain adaptive/generraized tasks. Code is available at httth::t///iiie http:/ttttttbbb/./ ctwpttth..oooooo.oooooooootoettt htth//BrraaaaunnDDD
1 Introduction
As deep learning technology develops by leaps and bounds 47 59 3D scene understanding has become the foundation for many real-world applications, including autonomous driving, robotics, augmented reality, smart cities, etc. Based on the LiDAR sensor, 3D semantic segmentation is critical task that provides accurate and robust semantic prediction of the surrounding scenarios. However, annotating large-scale datasets for training every new scenario is labor-intensive and time- consuming, especially for the tasks demanding point-wise annotations. These limitations hinder their practical applicability in real-world scenarios where acquiring high-quality labeled data.
Currently, domain adaptive 3D semantic segmentation (DA3SS)
51 and domain 60 have been widely explored in
19
32
38
45
generalized 3D semantic segmentation (DG3SS) 21
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

autonomous driving scenes. Their difference lies in that the former seeks to narrow the domain gap between the source and target domain data without assigning 3D semantic labels, while the latter aims to learn a generic and robust model before being exposed to the target domain. Albeit successful, their applications in 3D semantic segmentation have primarily focused on generalizing or adapting between synthetic and real scenes or across different scene layouts. This leaves gap in exploring universal framework, enabling the generalization and adaptation of 3SS models across datasets.
With the above considerations, this paper focuses on studying a universal framework for cross-domain 3D semantic segmentation. The essence of simultaneously solving cross- dossin tasks is to enhance the generalizability of the encoder. Therefore, generalizable 3D model with source-domain data discrimination power to the target domain necessary. is performed solely with access to source domain data, enabling the model to develop the ability to discriminate specific features. Unfortunately, one limitation has arisen, the scarcity of 3D pre-training datasets hinders this endeavor. More recently. Visual Foundation Models (VFMs) 23 34 36 have emerged as the de-facto visual backbone in 2D image classification and segmentation. Such VFMs are trained on massive raw web-curated images, achieving promising spen-vocabulary recognition. Hence, two natural questions are thrown up: (i) How to borrow 2D prior knowledge from VFMs? and (ii) How to harness 2D prior knowledge to boost 3D performance? To begin with, for issue (i), visual prompt tuning [6, 20 is a surrameter-effiienn strategy to exploit the representational potential of VFMs. We consider freezing the whole VFMs and only learn several trainable lightweight blocks as supplementary input, which inherit parameters of VFMs trained at scale to the maximum extent. After that, for issue (ii), we consider a more effective prompt tuning that introduces an extra depth-guided prompt space and extends the model input with the prompt, which could guide the generalization of powerful representations achieve desirable performances.
and domain-
Accordingly, in this paper, we introduce UniDSeg, dig deeper into prompt tuning in the VFM-based encoder, and introduce Leurnable-parameter--me parameter-inspiree Mechanism to the off-the-shelf VFMs with frozen parameters. Our VFM-based encoder is designed to learn alternately between two lightweight modules: i.e., Modal Transitional Prompting (MTP) and Learnable Spatial Tunability (LST). The former depends on the transitional guidance from 3D-to-2D unnatural images, i.e., sparse depth, which exists in the prompt space before being fed into the encoder layer. The latter depends on the customized context length of vectors, which exists in the query space for seeking matched prompting after encoding in the layer. Hereby, depending on the number of VFM-based encoder layers involved, we place layer-wise MTP and LST blocks to take full advantage of semantic understanding of diverse levels and modalities. The proposed mechanism not only avoids any unnecessary attempts to manipulate the original visual space but also inherits the pre-existing target awareness from the VFMs to the maximum extent. Ultimately, by integrating the proposed two modules into a cross-modal learning framework, our method efficiently mitigates the domain gap and enables 2D and 3D models to learn invarriat representations.
The key contributions of our work are summarized as follows: 1) Our method is groundbreaking in introducing the prompt-tunin concept into the universal model for DG3SS and DA3SS tasks. 2) We propose a novel learmable-parameter parameter-inspired mechanism to the off-the-shelf VFMs, which maximally preserves pre-existing target awareness in VFMs further enhance its generalizability. 3) Extensive experimental results demonstrate the effectiveness of our method across widely recognized tasks and datasets, all achieving superior performance for DG3SS and DA3SS.
2 Related Works
2.1 Domain Adaptive 3D Semantic Segmentation
In general, DA3SS seeks to narrow the domain gap between the source and target domain data, which can be grouped as uni-modal 61 54 48,
28 38 50 45] 6 5 51 B 46) conditions. For uni-modality, early methods [88 61 exploit the generative
38
36
55
24
32 and multi-moda [19
30
35
adversarial network to mitigate domain shift caused by appearance and sparsity differences. Later on, Yuan el al.
[55] 56) propose the adversarial network based on category-level and prototype-level alignments to address the mismatch of sampling patterns. CosMix [38 and ConDA (24] construct an intermediate domain by utilizing joint supervision signals from both the source and target domains for self-training. 3D surface representation is also considered effective method. Complete&Label (54)

---

transforms domain adaptive task into a 3D surface completion task. SALUDA 32 learns an implicit underlying surface representation simultaneously on source and target data.
Compared to uni-modality, multi-modality exploits the exclusive information of paired images and point clouds to complement each other. xMUDA [9] is pioneering method of cross-modal mutual learning for DA3SS. To facilitate learning domain-robust dependencies, several methods extend 2D
techniques to learn the 3D domain-invariant representations, such as adversarial learning 30 35 58)
style transfer [28]. and contrastive learning (50] Based on these dependencies, SSE [58] presents a self-super learning mechanism from caaetto--paatal and discrete-to discrete-to-textuee to-textured representations.
BFtD 45 presents cross-modal fusion- thenntthe--eee then-distillation to mitigate imbalanced modality adaptability, Recently, Segment Anything Model (SAM) (23) has presented its strong capability in generating semantics-aware regions, making it a suitable option for cross cross-modal interaction. MoPA [S] and VFMSeg [51] harness the knowledge priors learned from SAM to produce more accurate pseudo- labels for unlabeled target domains.
In contrast DA3SS, where the inputs in the target domain, although without 3D labels, are accessible during the training process, DG3SS is evaluated on data from totally unseen target domains.
2.2 Domain Generalized 3D Semantic Segmentation
The goal of DG3SS is to first learn as generic and robust representations as possible before being exposed to any target-domain data during training. Research on DG3SS has recently witnessed a surge, highlighted in several studies, including uni-modal condition [2] 37
60 and multi-modal condition [4 Kim et al. 21 leverage sparsity invariant feature consistency the feature level and semantic correlation consistency at the output level to constrain the model. Ryu et al. (37] and Sanchez et al. [40] incorporate multi-frame aggregation with 6-DoF ego-motions via randomized LiDAR configurations augmentation and label propagation, respectively. However, these methods are limited when 6-DoF ego-motion is unknown, and they need to be estimated using the aff-the-shelf LiDAR SLAM method (2). Recently, 3D representation under Bird's s-Eye- View (BEV) has also been considered an effective method for learning domain- invariant features. LiDOG [39 introduces dense top-down prediction auxiliary task and supervises it by employing BEV-view of semantic labels, while BEV-DG [27] introduces a BEV-view of cross-modal representation fusion to alleviate the domain gap. Particularly, 3D scenes also exist in several adverse weather conditions
39
40,
49
including fog, snow, and rain [49]. UniMix (60] leverages physically valid adverse weather simulation to construct a bridge domain and then blends it with samples of normal weather conditions.
Our endeavor is tailored to designing a universal cross-domain multi-modal learning framework that
enhances the performance of both DG3SS and DA3SS.
3 Method
3.1 Preliminary
Problem Definition. Given a source domain Ds ((X?D.s X3D.S, y. with n, labeled data and a target domain Pr = [(X?D.I x2D.T)) ) with n, unlabeled data under the condition not equal. For DA3SS, the task seeks to adapt models
that the source and target data distributions trained on the source domain Dş to a target domain Dr without labels. For DG3SS, the task aims to exploit the knowledge from the source domain achieve generalization to the target domain, while the model remains unexposed to the target domain during training, i.e., DD is unseen. Both tasks framed with the expectation of having paired images and point clouds and learning mapping function f X2D,T X3D.T + Y.D.T' that could predict the target-domain 3D labels.
Revisiting VIT. Given pre-trained Vision Transformer (ViT) 10 model with T. layers, an input is divided into Mf HW fixed-size patches I'm R x > m 1,2, .... M, where H and W are the height and width the original input, P is the patch size, and D is the constant latent vector size through all of its layers. Each patch is then embedded into D-dimensional latent space:
pa
eo Embedd
60 RR.
We indicate the collection of 2D patch embeddings, E = {e_h, E | S m S M). as input to the 1-th ViT encoder layer L.).. Specially, Eg MxD learns beforchand together with class
3

---

encoding Edi E RIxD and positional encoding Epos E <I((++M))DD to create initial patch embedding Eg. Hence, the whole ViT encoder is formulated as:
Eg = (Eel W E + Epos:
(2)
E = (4),, = 1,1....., L,
(3)
where Ej E R(++M)D is the patch embedding output from Lr(.), a denotes concatenation on the sequence length dimension. Each layer L, consists of several multi- head self-attention modules and ford-forward networks together with skip connection and LayerNorm
3.2 Learnalleaarrmete-iispiie Mechanism
Overall Framework. As depicted in Fig. D the overall framework of UniD- Seg is decomposed to 2D network
Image from Camera
Learable
FED 8DD G2D H2D and a 3D network J3D EBD G3D H&D
Prompts
g
Samp.
A
where g() go), and H() denote en-
Sparse Depth
2D Decoder
VFM-based Encoder
coder, decoder, and classifier, respec- tively. The main insight UniDSeg is to provide universal framework that hances the generalizability and adaptabil- ity of cross-domaie 3D semantic segmen- tation. Thereby, we introduce a novel task-specific VFM-based encoder, which guided by point-level prompts from 3D information. Formally, point cloud X is input to 3D network Jr3D to gen- erate 3D prediction, while image X?? and sparse depth x.Prr are input to 2D network F2D to generate 2D prediction with the help of VFMs priors. X XPPeP is derived from the LiDAR sensor via per-
Cruss modal Leaming
Proj.
ED
g
LIDAR Point Cloud
3D Encoder
3D Decoder
Figure I: Overall framework of UniDSeg for DG3SS and DA3SS. The backbone of the VFM-based Encoder is frozen and only trains several learnable modules. "Samp." means sampling of 2D features. Only the points falling into the intersected field of view are geometrically associ- ated with multi-moda. data.
spective projection. Afterward, we employ cross-modal learning on the source/target predictions via
auxiliary classifier H(.).. In summary, we place layer-wise learnable blocks to take full advantage of semantic understanding of diverse levels and modalities, which inherits potential target information of VFMs into the current training model. Our method is parameter-efficien and could be directly deployed to various pre-trained transformer-base architectures without modifying the basic units.
VFM- Encoder. Since the feature extraction of VFMs still lacks task-specific information, freezing parameters may lose some domain-agnostic contextual semantic information from source- domain data fine-tuning parameters may alter some pre-existing target representation from large- scale pre- rre-triined data. To provide a remedy, we reconsider the role of pre-trained models, balancing the capture of contextual information and the overfitting issue of source-domain data. As illustrated in Fig. 2 we introduce Learnable parppppee-iaspppeeeeeeee embedding, i.e., 3D-to-2D transitional prompts and tunable deep queries. The former not only learns spatial distance perception prompts from point clouds but also learns invariance to sample perturbations. The latter ensures that the feature distribution of E, will not be modified drastically, thus making better use of the pre-trained knowledge from VFMs. Only these prompts and queries are updated during fine-tuning, while parameters of other layers T, of the ViT encoder are kept frozen.
Mechanism, which provides a set of continuous
Depending the number of VIT layers involved, VFM--ased encoder is designed learn alternately between two lightweight modules: Modal Transitional Prompting PG1(-, and Learnable Spatial Tunability TB: (-). The former depends on the transitional guidance from 3D-to-2D unnatural images, which exists in the prompt space before being fed into layer L,, while the latter depends on the customized context length of vectors, which exists in the query space for seeking matched prompting after encoding in layer LJ. With deliberate design, the newer task-specific ViT encoder can be decomposed into:
È = (EI11) (TB( = 1,2, ...,,
(4)

---

VFM-based Encoder
Paramoters
Modal Transttiona
Prompting
3D -0- Transitional Prompt ...
VIT Encoder Layer
Leuraalle Spotial
PG1
PG
PatchSi Position Embed
Tumablitt
Sparse Depth
Space
TB
L
TB:
Image
Space
Tunable Deep Query
Space
Leumable
Pott Sune
X000
Liwer
Spanse Depth
XIP
E
+
L((Bii)
Linoe
Imsage
Down
E
(a) Modal Transitional Prompting
(b) Learnable Spatial Tumability
Figure 2: The architecture of VFM-based Encoder. We explore two layer-wise learnable blocks: (a) Modal Transitional Prompting and (b) Learnable Spatial Tunability, During training, only the parameters of two modules are updated while the whole ViT encoder layer is frozen.
B1 :, Em-1[1 :,:] PG:(X2P, x Dep).
(5)
where Er is the final refined patch embedding output from our proposed VFM-based encoder. According to the DG or DA task, X E R wxx means source or target image and X Dep E xxxx means source or target sparse depth. Note that as Eq. ( shows, the prompt output from PG; is aligned to the patch embedding without contacting Ecls.
Modal Transitional Prompting. Previous methods [6 20 have proved that visual prompt-tuning brings flexibility to the pre-trained VFMs for downstream tasks. However, their prompts like pixel- level perturbations learnable vectors are black boxes with limited learning capacity, which cannot reliably explore convincing knowledge beneficial for cross-domain 3D semantic segmentation. Lee al. (26) have proved that source data internally know much more about the world and how the scene is formed, called Privileged Information. Therefore, PG; is designed to capture 3D-to-2D transitional prior and task-shared knowledge of this information from the prompt space, which might be useful for cross- dross-domain learning.
To achieve this goal, following Eq. 4 patch embed module borrowed from the plain VIT is re-initialized and applied obtain sequence of flattened patch embeddings EJD E.Der RD Particularly, sparse depth X Dep point cloud representation via perspective projection, presents an unnatural image. From the view of modal characteristics, it is casy to access and contains the prior spatial distance information that lacking in 2D representation. From the view of deep encoding, it focuses on the scope of scenes at different receptive fields, tightly coupled with semantic information, so that their corresponding features have different contents when constructed. Note that Dep only has the I-channel to show the depth value in camera coordinates, we repeat the depth channel to make it equal to the number of RGB channels. In addition, considering that low-frequency bands of the amplitude spectrum tend to capture style information (or low-level statistics) (53] we provide a simple way to perturb the amplitude spectra of the source image ensure that the VFM is exposed to more variations in low-frequency components during training. The perturbed images can be obtained by using low-frequency filter, and then Eq. ( also employed to generate corresponding flattened patch embedding F.g. E TR: MD After that, to effectively integrate prompt-tuning to visual embedding output from each frozen layer, we flexibly build lightweight layer that contains exclusive MLP (x(.) for adapting 1-th encoder layer and shared MLP +(-) across whole ViT. This implementation written as:
EP (6)(EE) 2D W EJ
U E Dep
(6)
where P.G MxD is the 3D-to-2D transitional prompt output from PG,, MLP is built with a two-layer bottleneck structure (Linear-GLLU-E with the hidden layer reducing the input dimension by 3x.
5

---

Learnable Spatial Tunability. To progressively improve feature generalization via 3D-to-2D transitional prompt, inspired by BT 44 TB, is introduced to bridge the diserepancy between the pre-training dataset and the target scene. To achieve this goal, after the I-th layer, TB; starts with set of learnable tokens O₂ E RxD where K' is the length of the token and each token is initialized randomly. Considering the essential need in 3D semantic segmentation to discern multiple instances within a framed scene, T T2 exploits an attention mechanism, which enables VFMs to seek matched prompting to the features of distinct instances, thereby assisting VFMs in adapting to the differences between pre-training and cross-domain data. Concretely, T TB; adopts a dot-product operation to generate the affinity matrix Jr, which captures the associations between prompted patch embedding Li (1(E)1) and learnable tokens Oj. Then, a SoftMax function is applied to align each patch with unique instance. This implementation is written as:
Ji = SoftMaaar Li(Ei 111 : X OI,
Ji JER
VD
O == Or,a x OLD,
(8)
where Oua E km and OL, E D XD are constructed as low-rank matrices 18. reducing the trainable parameters by learning pairs of rank-decomposition matrices. Note that the low-rank dimension D. < D (D,. = 32 in our case). Besides, we further process the embeddings through a down-projection layer with parameters W E Rx followed by an up-projection layer with parameters Wap E RD x which is then element-wise added via the residual connection. After that, to enhance the flexibility in feature adjustment, TB, employs several layers to produce:
F = 622 (F)-)))) :,:] + W x (WT Ta ( :,:)) Jr x (0
(9)
where ETB E M is the tunable deep query output from TB, 51(-) and 82(-) are linear layers.
3.3 Cross-modal Learning
The point-wise supervised segmentation loss of the source domain is formulated as follows:
N
Leeg NXC 1=1
3D.S log (n.c))
(10)
where main prediction p is either D2D.S or páD.s N and C' being the number of points of the source point cloud and the number of classes, respectively.
The goal of unsupervised learning across modalities is two-fold. Firstly, we want to transfer knowl- edge from 2D modality to 3D modality on the source-domain and target-domain dataset. Secondly, we devise mutual learning on the output level, where the task is to transfer the pre-existing target information of VFMs to a 3D model. Same to xMUDA 119, we choose the Kullback-Leibler divergence DKL(-ll-) for the cross-modas loss Leas and define it as follows:
f = DKL
(D2 (P2,, ppp... + DKL (P3D.S 22D.S-3D.S
(11)
C² = D 2D, (.D.T-2D.T) + DKL (P (2DTT)DD
(12)
where D2D,9 and p3D. is to be estimated by mimicking predictions p3D,S-2D,S and p
from the respective auxiliary classifiers H&D and Hap The same goes for target predictions. Ultimately, the overall loss function LDG of DG3SS and LDA of DA3SS are defined as:
LDG Leeg + >>sL
(13)
LDA Leeg + A + Xy
(14)
where Äg and \r are the weights trading off cross-modal loss on source and target domain inputs.
4 Experiments
4.1 Datasets
For evaluation, we use four public autonomous driving benchmarks, including three real datasets:
nuScenes 4, SemanticKITTI (3), A2D2 [12] and one synthetic dataset: VirtualKITTI [T]. For all

---

Table 1: Performance comparison of multi-modal domain adaptive and domain generalized 3D semantic segmentation methods in four typical scenarios. We report the mloU results (with best
and 2nd best) on the target testing set for each network as well as the ensemble result (i.e., xM by averaging the predicted probabilities from the 2D and 3D networks.
S:Source/ / Turget
USSScnn.:
nuScenes: muScenes.Day/Nighh
vKITTιsKITTI
A2D2/KTTTI
Task
Method
2D
3D
xM
2D
3D
xM
2D
3D
xM
2D
3D
xM
Source-only
58.4
62.8
68.2
47.8
68.8
63.3
26.8
42.0
42.2
34.2
35.9
40.4
logCORAL
33
64.4
63.2
69.4
47.7
68.7
63.7
41.4
36.8
47.0
35.1
41.0
42.2
MinEnt 43
57.6
61.5
66.0
47.1
68.8
63.6
39.2
43.3
47.1
37.8
39.6
42.6
BDL 29
62.0
64.8
70.4
47.0
69.6
63.0
21.5
44.3
35.6
34.7
41.7
45.2
XMUDA 19
64.4
63.2
69.4
55.5
69.2
67.4
42.1
46.7
48.2
38.3
46.0
44.0
AUDA 30
64.0
64.0
69.2
55.6
69.8
64.8
35.8
37.8
41.3
43.0
43.6
46.8
DA
DsCML 35
65.6
56.2
50.9
49.3
53.2
38.4
38.4
45.5
39.6
45.1
44.5
Dual-Cross
64.7
58.1
58.5
69.7
68.0
40.7
35.1
45.9
40.0
48.6
SSE (88)
64.9
63.9
69.2
62.8
69.0
68.9
45.9
40.0
49.6
44.5
46.8
48.4
BFD 45
63.7
62.2
69.4
57.1
70.4
68.3
41.5
45.5
51.5
40.5
44.4
48.7
MM2D3D 6
71.7
66.8
72.4
70.5
70.2
72.1
53.4
50.3
56.5
42.3
46.1
46.2
VFMSeg 51
70.0
65.6
72.3
60.6
70.5
66.5
57.2
52.0
61.0
45.0
52.3
50.0
UniDSeg
67.2
67.6
72.9
63.2
71.2
71.2
60.5
50.9
62.0
50.7
55.4
57.5
XMUDA
19
58.7
62.3
68.6
43.0
68.9
59.6
25.7
37.4
39.0
34.9
36.7
41.6
DG
MM2D3D 6
69.7
62.3
70.9
65.3
63.2
68.3
37.7
40.2
44.2
39.6
35.9
43.6
UniDSeg
66.5
64.5
72.3
57.0
70.5
70.0
57.6
44.7
60.0
48.8
46.3
54.4
real datasets, LiDAR sensor and RGB cameras are synchronized and calibrated, allowing 2D-to-3D projection, and for the synthetic dataset, VirtualKITTI provides depth maps so we simulate LiDAR scanning via uniform point sampling. Following DA3SS settings [19], we exclusively utilize the front camera image and the corresponding LiDAR points that are projected onto
Our experimental scenarios cover typical real-to-real single domain adaptation and generalization chal- lenges like lighting changes (nuScenes: Day/Night), scene layout of country (nuScenes: USA/Sing:: nuScenes: Sing./USA), and sensor setups (A2I22/KKITT); :101:;A22222 A2DD2luuScenss) Note that, the source and target classes are coincident. For the first three scenarios, we choose merged classes while for the last two scenarios, we select 10 and 8 shared classes between two datasets. In addition, the synthetic-to-real domain adaptation and generalization challenges are also considered (vKITTIIsKITTI, simulated depth, and RGB to real LiDAR and camera, with 6 merged classes). Details of the class partition are provided in supplementary materials.
4.2 Implementation Details
For the 2D backbone, we use the visual encoders of CLIP 36) and SAM [23] a large-scale pre- training model. Our selection Vision Transformer [10] includes ViT-Base (ViT-B) and ViT-Large (ViT-L) architectures. Then, we utilize the MMSegmentation [8] codebase for the decoder head, SemanticFPN 22), widely-used segmentation head, is integrated with the visual encoder that serves as the 2D backbone. Meanwhile, depth information is a 3D attribute derived from LiDAR sensors, without the need introduce additional datasets, thus ensuring fairness under equal experimental conditions. For the 3D backbone, we employ SparseConvNer [15] with a U-Net architecture in Sparse Convolution Library (9) The voxel size is set to 5cm in the 3D network. This voxel size ensures that each voxel contains only one 3D point, maintaining level of granularity suitable for the task.
Our model trained on DusScenes:Day/Nighe iterations. We utilize an iteration- based learning schedule where the initial learning rate is set to lc-3 except for the 2D encoder which is Ic-4, and then it divided by 10 80k and 90k iterations. For "nuScenes:USAAA USA/Sing." and SinSeenes:Sing/USAAA the training is performed for 60k iterations, and the learning rate divided by 10 the 40k and 50k iterations. For <KITTI/SKITIL the training is performed for 30k iterations, and the learning rate is divided by 10 at the 25k and 28k iterations. The batch size is set to 8. As regards the hyper-parameters. following 19, As and \T in cross-modal loss and "nuScenes:Sing/UUS.
"A2D2/sKITTI", and "A2D2/nuScenes" for 100k
are set to 1.0 and 1 on "nuSSennes:Dy/NNight 0.1 and 0.01 on "'KITTISKITTI". "A2D2/<KITTI" and 'A2D2/muScenes" respectively, without performing any fine-tuning these values. All experiments are conducted on NVIDIA RTX 3090.
UnSScenes:USA/Sinn
7

---

Image
MM2D3D
Olus
Grounctruth
mur Day/Night
Mannadde
Lerrin
Sidrwall
1RLSSSRR USASS55
Car
Drivable Surface
Sidewalk
Terain
Manmade
V Vegetation
Ignore
Object
Car
Truck
Road
Objoct
Building
Vegetation
Ignore
Car
Truck
Bike
Person
Road
Sidewalk
Parking
Objoct
Building
V Vegetation
Ignore
Figure 3: Qualitative results of DG3SS. We showcase the ensembling result of four scenarios.
4.3 Quantitative and Qualitative Comparison
We compare the proposed method with three classic 2D domain adaptive methods, which can be easily extended to multi-modal conditions. Moreover, eight multi-modal DA3SS and two multi-modal DG3SS are discussed. As shown in Tab. D we tabulate the comparison results in mean Intersection over Union (mloU, %) on the target testing data. Note that not datasets provide image labels. Thus the quantitative evaluation of 2D semantic segmentation depends on the 3D-2D corresponding point-wise prediction. Overall, our method achieves the best performance on all scenarios against the competitors, w.r.t. ensemble result "xM"", except for DA3SS on "nuScenes: Day/Nghht gets the second best. As shown Fig. 3 we visualize the DG3SS results of four settings. By the merit the visual foundation models, method can segment detailed objects very well. From top to bottom, the focused areas are the sidewalk near the drivable surface, the trunk of a tree, the profile of the road, and most importantly, bicycles safely riding the road.
Comparison of DG3SS. In terms of DG task, compared with baseline (XMUDA) and MM2D3D, method still achieves remarkable results, exhibiting 3.7%/1.49% 10.4% /...0%%%%%%/ and 12.8% mloU improvement. It is worth noting that our method, without exposure to any target-domain data, shows performance that is close to even surpasses that of most DA3SS methods (e.g., the result of Ours-DG is comparable to VFMSeg on "nuScenes: USA/Sig..)//// All results demonstrate that utilizing the powerful veccbvocabulary recognition capability of VFMs is beneficial for addressing generalization problems in domains with significant discrepancies.
Comparison of DA3SS. The source-only model is the lower bound, which is not domain adaptive as it is only trained on the source-domait data. It is observed that our method brings a significant adaptation effect on all scenarios compared to the source-only model, with the gains of 4.7 4.7%. 7.9%. 19.8%, and 17.1% in mloU, respectively. Compared with baseline (xMUDA) for all DA competitors, method exceeds it by large margins with gains of 3.5%, 3.8%. 13.8%, and 13.5% in mloU. Particularly, on the "A2D2/<KITTT" scenario, our method typically yields a higher score compared to the best "xM" in VFMSeg (57.5% vs. 50.0%).
4.4 Ablation Study
Evaluation of Different VFMs Training Strategies. Our analysis various training strategies for VFMs in six experimental scenarios within and across datasets can be found in Tab, 2 Note

---

Table Ablation study on VFM-based encoder with different training strategies for DG3SS. This setup is based on the same LiDAR -Cmeraa configuration but different environments (top-3 scenarios), and different LiDAR- R-Carra configurations (bottom-3 scenarios). Of note, "Params" denote trainable parameters in the encoder.
S:Source/ / T.Tagett
nuScenes: USA/Sing.
muScenes:Day/Nee
muScenes:Sing./u
Strategy
Visual Backhone
Params
2D
3D
xM
2D
3D
xM
2D
3D
xM
Finetune
86 MM
62.4
64.1
69.6
53.3
70.7
68.8
65.7
67.9
72.9
Frozen
CLTP.ViT-B
0.0M
59.7
64.5
69.7
46.8
71.0
69.8
58.3
67.9
71.2
Ours
1.82M
64.7
71.5
55.9
70.7
70.0
68.2
68.0
74.0
Finetune
30SM
65.5
64.5
70.4
54,9
70.7
67.3
69.9
67.8
74.5
Frozen
CITP.VIT-L
0.0M
60.4
64.2
70.1
50.2
70.5
69.5
62.2
67.8
73.3
Ours
4.70M
64.5
72.3
57.0
70.5
70.0
70.6
68.0
75.1
S:Source / Targget
VKITTM/KITTI
A2D2/sKITTI
A2D2/muSccnes
Strategy
Visual Backbone
Params
2D
3D
xM
2D
3D
x.M
2D
3D
x.
Finetune
86.9M
54.9
41.5
55.8
43.0
43.8
51.5
55.4
50.1
60.2
Frozen
CLIP.VIT-B
0.0M
49.1
42.0
54.4
35.3
43.8
48.7
51.2
49.4
58.1
Ours
1.82M
55.6
43.6
58.0
43.2
44.6
52.0
56.3
50.3
61.0
Finetune
305M
57.4
43.5
58.7
46.9
44.3
53.0
57.2
50.8
61.3
Frozen
CLIP.VIT-L
0.0M
54.0
42.9
58.4
41.8
44.4
51.4
53.7
50.0
59.6
Ours
4.70M
57.6
44.7
60.0
48.8
46.3
54.4
58.0
50.7
61.9
Table Ablation study on the effectiveness of signifi- cant components in UniDSeg with the ViT-B backhone for DG3SS task.
20 minu mhg
69
66.0
68.2
67.1
66.7
nuScenes::ing /USA
A2D2/KIITI
MTP LST
USA/Sing.
2D
3D
xM
24
43.2
Frozen VFM 58.3
67.9
71.2
35.3
43.8
48.7
43
42.1
63.9
67.8
72.5
40.4
44.0
50.5
65.7
67.8
73.3
41.8
44.2
51.1
68.2
68.0
74.0
43.2
44.6
$2.0
Thkee Length
Figure 4: Efffect of the learnable token length.
that due to the fixed and relatively small number trainable parameters in the decode head, the count of trainable parameters presented in the tables is focused solely on the visual architecture. In this setup, we separately select CLIP: and CLIP:ViT-L backbone with fine-tuning, freezing, and our proposed gearmabll-parrmmtte-innspp frozen VFMs outperform previous DA3SS methods without specialized design. In addition to VFMs with full parameter fine-tuning exhibit enhanced performance relative to their frozen counterparts because the 3D modality is less sensitive to light, thus dominating in ensemble prediction direction. Remarkably, our method achieves even superior generalization capabilities, surpassing the full parameter fine-tuning 0.5~2.7% mloU with merely ~2% trainable parameters compared to the original backbone. This ablation experiment demonstrates that our method can inherit the pre-existing target awareness from the VFMs to the maximum extent.
Mechanism for analysis. The results indicate that
"nuScenes:Day/Nige Day/Ngh,,, "auSeenes..aa//iihhh""
Evaluation Different Components. As shown in Tab, we train four models on two scenarios for DG3SS, including (1) Frozen parameter of VIT backbone; (2) only performing MTP in the frozen ViT by using prompt tuning, leading to a significant mloU boost (1.3% and 1. 1.8%). This highlights the robustness of the 3D-to-2D transitional prompts in cross-domain learning; (3) only performing LST the frozen VIT by customizing learnable token, demonstrating that learnable tokens can encourage the model to learn domain-invarian representations (increasing 2.1 % and 2.4% mloU); (4) combining two proposed components to reach peak value.
Evaluation of Learnable Token Length. We illustrate the impact of learnable token length K on the overall 2D performance of UniDSeeg with the ViT-B backbone for DG3SS. shown in Fig. 4 the results demonstrate consistent upward trend. When K is set 100, the relatively small

---

Table 4: Effect of using Segment Table 5: Effect of applying Anything Model (SAM) as the different training strategies 2D backbone.
Table 6: Effect of using different 3D backbones on the DA3SS methods.
to the SAM-base model.
USA/Sing.
USA/Sing.
USA/Sing.
3D Backbone
DA3SS
Task 2D Backbone
Task Strategy
2D
3D
xM
2D
3D
xM
2D
3D
xM
XMUDA
64.4
63.2
69.4
DG
SAM:VIT-L
CLIP.VIT-L
66.8
66.5
64.7
64.5
72.6
72.3
DG
Finetune
Ours
66.8
65,9
64.3
64.7
72.6
70.8
SparseConvNet
UniDSeg
67.2
67.6
72.9
XMUDA
65.9
64.0
69.7
DA
CLIP-VIT-L
67.2
67.6
72.9
DA
Finetune
66.5
67.9
71.4
MinkowskiNet
UniDSog 67.5 68.6
73.1
SAM.VIT-L
67,8
68.8
73.3
Ours 67.8
68.8
73.3
training parameters and high performance make it the preferred choice for subsequent experiments. Meanwhile, this observation suggests that the model benefits from incorporating visual information from multiple layers, enabling it to capture more nuanced and discriminative features.
Evaluation of Different 2D and 3D Backbones. The motivation of this work is to study a universal framework based VFMs to enhance the generalizability and adaptability of cross-domain 3D semantic segmentation, demonstrating the effectiveness of the visual foundation model priors. However. CLIP not designed for common surveillance-relevan tasks like semantic segmentation. Hereby, we dedicate ourselves to verifying its effectiveness on VFMs designed for segmentation tasks such as SAM 233 Firstly, in Tab. we evaluate the effect of SAM as the 2D backbone. is observed that --Meb UniDSeg exhibits better performance on "USA/Sing" scenario, with a "xM" gain of 0.3% on DG and 0.4% on DA. Then, Tab. 5 we evaluate the effect of applying different training strategies to the SAM-based model. It is observed that our Learnable- Leammble-parameter
Mechanism for tuning the SAM- encoder can achieve 1.8% and 1.9% "xM" gain compared to fine-tuning strategy. In addition, we evaluate UniDSeg on another 3D backbone, i.e., MinkowskiNet, making it more convincing as a "universal" framework for cross-domain 3D semantic segmentation.
The parameters and computational costs of CLiP-based and SAM-based 2D backbones. "Cost" means the percentage of trainable parameters in MTP and LST compared to fine-tuning the whole encoder consumed.
Table 7:
2D Backhone Full Params
Trainable Params
Cost
MTP
LST
CLTP-ViT--
86.9M 305M
1.82M
2.09%
0.48M
1.34M
CITP-VIT-L
4.70M
1.54%
1.78M
2.92M
SAM:ViT-L
307M
4.34M
1.41 %
1.42M
2.92M
Evaluation of Computation Cost. In Tab. we have reported model parameters of CLIP: ViT-B, CLIP: ViT-L., and SAM: ViT-L for the VFMebased encoder along with all trainable parameters. Note that, the entire ViT backbone in our VFM-based encoder is frozen during downstream training for DA3SS and DG3SS. Only two layer-wise learnable blocks, MTP and LST, are trainable. It obvious that our method requires only 1-2% parameters optimization to exceed the fine-tuning results.
5 Conclusion
In this work, we delve into a universal method that can enhance the adaptability and generalizability of cross-domain 3D semantic segmentation, dubbed UniDSeg. For this purpose, introduce a learnable- parameter-inspiee mechanism to the off off-the-shelf VFMs with frozen parameters, which maximally preserves pre-existing target awareness in VFMs, further enhancing the generalizability of VFMs. Our method achieves state-of-the art performance in both DA3SS and DG3SS, as demonstrated by extensive experiments on different scenarios. We believe that our work will inspire a deeper investigation of cross-domain 3D semantic segmentation in autonomous driving.
Acknowledgments. This work was supported by the National Natural Science Foundation of China under Grant No.62176224, No.62306166 65;; Natural Science Foundation of Chongqing under No. C.CSTB2022NSCQ-IX0000 China Computer Federation (CCF) Lenovo Blue Ocean Research Fund; China Academy of Railway Sciences No.2023YJ357.
10

---

References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv: 2016.
arXiv preprint
[2] Chunge Bai, Tao Xiao, Yajie Chen, Haoqian Wang. Fang Zhang, and Xiang Gao. Faster- lio: Lightweight tightly coupled lidar-inerial odometry using parallll sparse incremental voxels. RAL, pages 4861-466 18688, 2022.
[3] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall, Semantickitti: dataset for semantic scene understanding of lidar sequences. In ICCV, pages 9297-9307, 2019.
[4] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, pages 11621-11631, 2020.
[5] Haozhi Cao, Yuecong Xu, Jianfei Yang, Pengyu Yin, Shenghai Yuan, and Lihua Xie. MoPA: Multi-modal prior aided domain adaptation for 3d semantic segmentation. arXiv preprint arXiv:2300. 1839, 2023.
[6] Adriano Cardace, Pierluigi Zama Ramirez, Samuele Salti, and Luigi Di Stefano. Exploiting the comple- mentarity of 2d and 3d networks to address domain-shift in 3d semantic segmentation. In CVPR Workshops, pages 98-109, 2023.
[7] Christopher B. Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In CVPR, pages 3075-3084, 2019.
[8] MMSegmentation Contributors. MMSegmentatiin: Openmmlab semantic segmentation toolbox and benchmark. httpp://github.. www.coo./oppcee.e.eccooccce..eae hatp//pennmmmmmmmeeee... htt.mm//ma///mmmmtetttttttooo 2020.
[9] Spconv Contributors. Spconv: Spatially sparse convolution library.
https://github.cu
braveoler53/spcoon 2022.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakoh Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR. 2021.
[11] Adrien Gaidon, Qiao Wang, Yohann Cabon, and Eleonora Vig. Virtual worlds as proxy for multi-objeet tracking analysis. In CVPR, pages 1340-4349 2016.
[12] Jakob Geyer, Yohannes Kassahun, Mentar Mahmudi, Xavier Ricou, Rupesh Durgesh, Andrew Chung. Lorenz Hauswald, Viet Hoang Pham, Maximilian Mühlegg, Sebastian Dorn, et al. A2d2: Audi autonomous driving dataset. arXiv preprint arXiv:2004. 06320,, 2020.
[13] Yunpeng Gong. Liqing Huang, and Lifei Chen. Person re-identification method based on color attack and joint defence. In CVPR, pages 4313 2022.
[14] Yunpeng Gong, Zhun Zhong, Yansong Qu, Zhiming Luo, Rongrong Ji, and Min Jiang. Cross-modality perturbation synergy attack for person re- re-identification. arXiv preprint 2rXiv:2401.10090 2024.
[15] Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. In CVPR, pages 9224-9232, 2018
[16] Cheng Han, Qifan Wang, Yiming Cui, Zhiwen Cao, Wenguan Wang. Siyuan Qi. and Dongfang Liu. E vpt: An effective and efficient approach for visual prompt tuning. In ICCV, pages 17445-17456, 2023.
[17] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[18] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0906 2021.
[19] Maximilian Jaritz, Tuan-Hung Vu, Raoul de Charette, Émilic Wirbel, and Patrick Pérez. Cross-modal leamnigg for domain adaptation in 3d semantic segmentation. TPAMI, pages 1533-1544 2023.
[20] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge J. Belongie, Bharath Hariharan, and Nem Lim. Visual prompt tuning. In ECCV, pages 707-727, 2022.

---

[21] Hyeonseong Kim, Yoonsu Kang, Changgyoon Oh, and Kuk-Jin Yoon. Single domain generalization for lidar semantic segmentation. In CVPR. pages 17587-17598, 2023.
[22] Alexander Kirillov, Ross B. Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In CVPR, pages 6399 6408, 2019.
[23] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloć Rolland, Laura Gustafson, Tote Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. In ICCV, pages 3992-4003, 2023.
2023. [24] Lingdong Kong, Niamul Quader, and Venice Erin Liong. ConDA: Unsupervised domain adaptation for lidar segmentation via regularized domain concatenation. In ICRA, pages
[25] Jogendra Nath Kundu, Akshay R. Kulkarni, Amit Singh, Varun Jampani, and R. Venkatesh Babu. Gen- eralize then adapt: Source-free domain adaptive semantic segmentation. In ICCV, pages 7026-7036, 2021.
[26] Kuan-Hui Lee. Germán Ros, Jie Li, and Adrien Gaidon. SPIGAN: privileged adversarial learning from simulation. In ICLR, 2019.
[27] Miaoyu Li, Yachao Zhang, Xu Ma. Yanyun Qu, and Yun Fu. BEV-DG: cross-modal learning under bird' s-eye view for domain generalization 3d semantic segmentation. In ICCV, pages 11598-1 1608, 2023.
[28] Miaoyu Li, Yachao Zhang. Yuan Xie, Zuodong Gao, Cuihua Li, Zhizhong Zhang, and Yanyun Qu. Cross- domain and cross-modal knowledge distillation in domain adaptation for 3d semantic segmentation. In ACM MM. pages 3829-3837, 2022.
[29] Yunsheng Li. Lu Yuan, and Nuno Vasconcelos. Bidirectional learning for domain adaptation of semantic segmentation. In CVPR, pages 6936-6945, 2019.
[30] Wei Liu, Zhiming Luo, Yuanzheng Cai, Ying Yu, Yang Ke, José Marcato Junior, Wesley Nunes Gonçalves, and Jonathan Li. Adversarial unsupervised domain adaptation for 3d semantic segmentation with multi- modal learning. ISPRS. pages 21 1-221, 2021.
[31] Xinhong Ma, Yiming Wang, Hao Liu, Tianyu Guo, and Yunhe Wang. When visual prompt tuning meets source-free domain adaptive semantic segmentation. In NewrlPS, 2023.
[32] Bjoern Michele, Alexandre Boulch, Gilles Puy. Tuan-Hung Vu, Renaud Marlet, and Nicolas Courty. Saluda: Surface-based automotive lidar unsupervised domain adaptation. In 3DV, 2024.
[33] Pietro Morerio, Jacopo Cavazza, and Vittorio Murino. Minimal- correlation alignment for unsuper- vised deep domain adaptation. In ICLR. pages 1-14, 2018.
[34] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacddin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.0719. 2023.
[35] Duo Peng, Yinjie Lei. Wen Li. Pingping Zhang, and Yulan Guo. Sparse-to-denss feature matching: Intra and inter domain cross-modal learning in domain adaptation for 3d semantic segmentation. In ICCV, pages 7088-7097, 2021.
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML., pages 8748-8763, 2021.
[37] Kwonyoung Ryu. Soommin Hwang, and Jaesik Park. Instant domain augmentation for lidar semantic segmentation. In CVPR, pages 9350-9360, 2023.
[38] Cristiano Saltori, Fabio Galasso, Giuseppe Fiameni, Nicu Sebe, Elisa Ricci, and Fabio Poiesi. CosMix: Compositional semantic mix for domain adaptation in lidar segmentation. ECCV, pages 586-602, 2022.
[39] Cristiano Saltori, Aljosa Osep, Elisa Ricci, and Laura Leal-Taixé Walking your lidog: A journey through multiple domains for lidar semantic segmentation. In ICCV, pages 196-206, 2023.
[40] Jules Sanchez, Jean- Emmanuel Deschaud, and François Goulette. Domain generalization of 3d semantic segmentation in autonomous driving. In ICCV, pages 18031-18044 2023.
2023.
12

---

[41] Cody Simons, Dripta S. Raychaudhuri. Sk Miraj Ahmed, Suya You, Konstantinos Karydis, and Amit K. Roy-Chowdhury. SUMMIT: source-free adaptation of uni-modal models multi-modal targets. In ICCV, pages 1239-1249, 2023.
[42] Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han. Searching efficient 3d architectures with sparse point-voxel convolution. In ECCV, pages 685-702, 2020.
[43] Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthicu Cord, and Patrick Pérez. ADVENT: adversarial entropy minimization for domain adaptation in semantic segmentation. In CVPR, pages 2517-2526, 2019.
[44] Zhixiang Wei, Lin Chen, Yi Jin, Xiaoxiao Ma. Tianle Liu, Pengyang Lin, Ben Wang. Huaian Chen, and Jinjin Zheng. Stronger, fewer, & superior: Harnessing vision foundation models for domain generalized semantic segmentation. arXiv preprint arXiv:2312.04255, 2023.
[45] Yao Wu, Mingwei Xing, Yachao Zhang, Yuan Xie, Jianping Fan, Zhongchao Shi. and Yanyun Qu. Cross-modal unsupervised domain adaptation for 3d semantic segmentation via bidirectional fusion-then- distillation. In ACM MM. pages 490-498, 2023.
[46] Yao Wu, Mingwei Xing. Yachao Zhang, Yuan Xie, and Yanyun Qu. Clip2uda: Making frozen clip reward unsupervised domain adaptation in 3d semantic segmentation. In ACMMM, 2024.
[47] Yao Wu, Mingwei Xing. Yachao Zhang, Yuan Xie, Zongze Wu, and Yanyun Qu Perturbed progressive learning for semisupervised defect segmentation. TNNIS, pages 61 88-113,, 2024.
[48] Aoran Xiao, Jiaxing Huang, Dayan Guan, Fangneng Zhan, and Shijian Lu. Transfer learning from synthotic to real lidar point cloud for semantic segmentation. In AAAI, pages 7795-2803, 2022.
[49] Aoran Xiao, Jiaxing Huang, Weihao Xuan, Ruijie Ren, Kangeheng Liu, Dayan Guan, Abdulmotaleh EadSkk Shijian Lu, and Eric Xing. 3d semantic segmentation the wild: Learning generalized models for adverse-condition point clouds. In CVPR, pages 9382 9392, 2023.
[50] Bowei Xing. Xianghua Ying, Ruibin Wang, Jinfa Yang, and Taiyan Chen. Cross-modal contrastive learning for domain adaptation in 3d semantic segmentation. In AAAI, pages 2974-2982, 2023.
[51] Jingyi Xu, Weidong Yang. Lingdong Kong. Youquan Liu. Rui Zhang, Qingyuan Zhou, and Ben Fei. Visual foundation models boost cross-modal unsupervised domain adaptation for 3d semantic segmentation. 2024.
[52] Yan Xu, Jiantao Gao, Chaoda Zheng, Chao Zheng. Ruimao Zhang, Shuguang Cui, and Zhen Li. 2dpass: 2d priors assisted semantic segmentation on lidar point clouds. In ECCV, pages 677-695, 2022.
[53] Yanchao Yang and Stefano Soatto. FDA: fourier domain adaptation for semantic segmentation. In CVPR, pages 4084-4094, 2020.
[54] Li Yi, Boqing Gong, and Thomas Funkhouser. Complete & Label: A domain adaptation approach to semantic segmentation of lidar point clouds. In CVPR, pages 15363,15373, 2021.
[55] Zhimin Yuan, Ming Cheng, Wankang Zeng, Yanfei Su, Weiquan Liu, Shangshu Yu, and Cheng Wang. Prototype-guider multitask adversarial network for cross-domain lidar point clouds semantic segmentation. TGRS. pages 1- 13, 2023.
[56] Zhimin Yuan, Chenglu Wen, Ming Cheng, Yanfei Su. Weiquan Liu, Shangshu Yu, and Cheng Wang. Category-level adversaries for outdoor lidar point clouds cross-domain semantic segmentation. TITS, pages 1982-1993, 2022.
[57] Yachao Zhang, Runze Hu, Ronghui Li, Yanyun Qu, Yuan Xie, and Xiu Li. Cross-modal match for language conditioned 3d object grounding. In AAAI, pages 7359-7367, 2024.
[58] Yachao Zhang, Miaoyu Li, Yuan Xie, Cuihua Li, Cong Wang, Zhizhong Zhang, and Yanyun Qu. Self- supervised exclusive learning for 3d segmentation with cross-modal unsupervised domain adaptation. In ACM MM, pages 3338-3346 2022.
[59] Yachao Zhang, Yuan Xie, Cuihua Li, Zongze Wu, and Yanyun Qu. Learning all-in collaborative multiview binary representation for clustering. TNNLS, pages 1260-4273, 2022.
[60] Haimei Zhao, Jing Zhang, Zhuo Chen, Shanshan Zhao, and Dacheng Tao. Unimix: Towards domain adap- tive and generalizable lidar semantic segmentation in adverse weather. arXiv preprint arXiv:2404.05145. 2024.
13

---

[61] Sicheng Zhao, Yezhen Wang, Bo Li, Bichen Wu, Yang Gao, Pengfei Xu, Trevor Darrell, and Kurt Keutzer. ePoinIDA: An end-to-end sinulation-to-real domain adaptation framework for lidar point cloud segmentation. In AAAI, pages 3500-3509, 2021.
[62] Xinge Zhu, Hui Zhou, Tai Wang. Fangzhou Hong, Yuexin Ma. Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical convolution networks lidar segmentation. CVPR, pages 9939-9948, 2021.
14

---

A Appendix supplemental material
This section starts with more information on UniDSeg, including dataset split, extensible tasks, more ablation studies, and additional visualization results.
A.1 Dataset Split of Cross-domain Learning
To compose our domain adaptive and generalized scenarios, following 19), we exploit public datasets,
including nuScenes 4 VirtualIITTI [ SemanticKITTI [3. and A2D2 12. The split details are tabulated in Tab.
Table 8: Size of the splits in frames for all proposed cross- domain learning scenarios.
Source
Target
Scenarios
Categories
Train
Train
Val/Test
muSeenes: Day/Night
24745
2779
606/602
Vehicle: [bicyclc, bus, construction, vehicle, motorcyele, trailer, track];
auScomes:USAA USA/Sing.
15695
9665
2770/2929
Driveable Surface; Sidewalk: Terrain: Maumade, Vegetation
Singenns:Sing/USA
9665
15695
-1090
<KITT]: Car: Truck: Road: Object: [traflic sign, traflic light, pole, miscl:
Building: Vegetution: [terrain, tree, vegetation]
KKITTIKKTTII
2126
18029
1101/4071
sKITTI Car: Truck: Read: Object: [fence, pole, traffie-sign, other-objectj;
Building: Vegetation:
Ivogetation. trunk, terrain]
A21D2: Car: Truck, Bike: [bicyele, small vehicle]; Person: Road: Parking:
Sidewalk: [sidewalk, curbstone];: Object: Building: Vegetation
A2D2AKITTI
27695
18029
1101/4071
<KITTI: Car: Truck: Bike: [hicycle. motoreycle, bicyclist. motorcyclis]] Person; Road;
Purking: Sidewalk:; Object; Building: Vegetation: [cemain, trunk, vegetation)
A2D2: Car:
Bike: [bicycle,
vehicle]: Person; Road:
A2D2/uSeenes
27695
25330
28006019
BuSeenes Car. Truck: Bike: [bicycll, motorcyele], Person: Road: [driveable_sarfan
Sidewalk: [sidewalk, curbstone]: Building: Vegetation
Sidewalk; Building: [manmade]; Vegetation: [terrain, vegetation]
nuScenes. It contains 1,000 scenes, each of 20 seconds, corresponding to 40k annotated keyframes taken at 2Hz. The original scenes are split into 28,130 training frames and 6,019 validation frames. Each frame contains a 32-beam LiDAR point cloud with point-wise annotations and six RGB images captured by six cameras from different views of LiDAR. For nuScenes: Day/Nggh,, we choose 602 night scenes for testing data, while for nuScenes:USA/Sin
and nuScenes: Sing we choose 2,929 Singapore and 3,090 USA scenes for testing data, respectively. Both of them merge the objects into 6 categories: Vehicle, Driveable Surface, Sidewalk, Terrain, Manmade, and Vegetation.
It consists of driving scenes which are created with the Unity game engine by vai-tal cloning of the scenes 1, 2, 6, 18, and 20 of the real KITTI dataset. Different from real KITTI, VirtualKITTi does not simulate LiDAR, but rather provides a dense depth map, alongside semantic, instance, and flow ground truth. Each of the 5 scenes contains between 233 and 837 frames, ie., in total 2126 for the 5 scenes. Fach frame is rendered with 6 different weather/lighting variants (clone, morning, sunset, overcast, fog, rain) which we use all.
VirTualKITTI.
SemanticKITTI. is a large-scale dataset based on the KITTI Odometry Benchmark captured in Germany. The original scenes are split into 19,130 training scans and 4,071 validation scans. Unlike nuScenes, SemanticKITTI only provides the front-view images and 64-layer front LiDAR. 19 categories are used for segmentation.
A2D2. It consists of 20 drives, which corresponds to 28,637 frames. The point cloud comes from three 16-layer front LiDARs (center, left, and right), where the left and right LiDARs are inclined. By projecting 3D point clouds onto 2D images, corresponding 2D semantic labels are regarded 3D point-wise labels, which contain 38 categories.
Note that, we select 6 merged categories between the VirtualKITTI and SemanticKITTI, including Car, Truck, Road, Object, Building, and Vegetation. Between A2D2 and nuScenes, we select merged categories, including Car, Truck, Bike, Person, Road, Sidewalk, Building, and Vegetation. Between A2D2 and SemanticKITTI, 2 additional merged categories are considered, which are Object, and Parking.
15

---

Entor by 2DPASS
Enor by Ouis
Groundmnuth
Trunk
TTrnnk
Bicydlist
Bkcydist
Truck
Truck
Figure 5: Qualitative results of our method on the validation set SemanticKITTI. The misclassifi- cation points are signed in red.
A.2 Fully-supervised 3D Semantic Segmentation
The proposed method can also work reliably in handling fully-supervised large-scale 3D semantic segmentation. As shown in Tab.9 we simply compare the results of some typical uni-modal and multi- modal methods, including MinkowskiNet 77, SPVCNN (42), Cylinder3D [62), and 2DPASS (52). on the publicly available SemanticK/TTI validation set. According to the official setting in 00-10 sequences, sequence 08 is the validation set, while the remaining sequences the training set.
In this experimental setup, we make some slight modifications to the training strategy. Considering that fully supervsede learning assumes the training and validation/testing sets independent identi- cally distribution, without involving domain shift problems cross cross-domain learning. (i.e., avoid overfitting the source-domain data distribution). Therefore, when training model, replace the 2D backbone in 2DPASS with our VFMs backbone and fine-tune its parameters in the same way. Moreover, learning rates of the other modules Ir = 0.24 and the number of training epochs ep 64 unchanged ensure the fairness of the experiment. Similar 2DPASS, and without Test-Time Augmentation (TTA) and set the number of views to 12 as the default. Some visualization results are shown in Fig. 5 2DPASS has higher error recognizing small objects and region boundaries, while our method recognizes small objects better thanks to the knowledge prior of visual foundation models.
reduce the learning rate of the 2D backbone to Ir 0.024 while keeping the
tabulate the validation results with
A.3
Source-Free Domain Adaptive 3D Semantic Segmentation
Source-free domain adaptation (SFDA) seeks to learn models where the vendor can trade only the source model and the client can perform target adaptation without accessing source-domain data. Drawing inspiration from Jogendra et al. 25). SFDA enables partition into two tasks: (a) source-only domain generalization and (b) source- free target adaptation. Hereby, towards the former, we utilize UniDSeg to achieve models with generalization capability. Towards the latter, we utilize target 3D pseudo-labels (PL. obtained from the source model for self-training. When selecting PL from the target-domain data, we consider the ensemble result "xM" as the fusion PI. to supervise both the and 3D branches. shown Tab. TO we adopt conventional consistency learning and pscudo-label self-training address the source-free domain adaptive 3D semantic segmentation
16

---

Table 9: Fully-supervised 3D semantic segmentation results on the Semantick/1T/ validation set. We
report per-class IoU. "f" denotes the reproduced result referring to the official codebase. "w/ TTA" means using test-time augmentation in the inference stage.
Method
CaR
bicycle
motorsy
track
bus
pesson
bicyelist
motongyelist
rood
purking
sddwwak
ground
bailding
vegetation
brunk
tonain
pole
Inlliic sign
mloll
MinkowskiNet
61.1
SPVCNN 1
63.8
Cylndeer3D
62
65.9
2DPASS
62
95.3 47.1
73.7
818
56.0
73.5
87.6
2.1
92
45.2
78.6
1.0
90.8
61.8
88.4
69.5
75.5
58.1
51.
64.7
2DPASS
w/ TTA
96.6
52.2
77.9
91.1
68.2
77.9
92.0
0.2
94.0
50.6
81.4
1.2
91.8
66.3
89.6
72.0
77.3
63.0
53.5
68.2
Ours
96.2
47.2
70.1
84.3
64.5
74
89.5
2.1
92.6
46.6
79.1
3.2
90.9
62.8
88.3
69.9
75.1
58.6
52.1
65.6,0.0
Ours w/ TTA
97.0
52.3
73.4
92.6
7L1
78.3
92.3
0.0
94.1
51.3
81.
92.1
67.4
89.5
72.0
77.0
63.8
54.6
68.6.0.4
Table 10: Performance comparison of SFDA3SS methods in three typical scenarios. "I" denotes the reproduced result referring to the official codebase, the different category splits applied in the same adaptation scenario.
S:Source / TTaaget
RuScenes:
DuScenes:Duy/Nighh
A2D2AKITTI
Task
Method
Source-free
2D
3D
xM
2D
3D
xM
2D
3D
xM
Baseline
58.4
62.8
68.2
47.8
68.8
63.3
34.2
35.9
40.4
Consistency
58.7
63.2
68.1
50.4
66.8
63.6
37.1
36.5
41.8
Pscudo-Label
V
58.9
62.7
68.5
48.3
69.0
63.2
37.6
36.6
41.5
DA
SUMMITI
41
V
61.6
66.2
68.4
53.8
68.9
68.2
42.9
43.7
46.8
UniDSeg
x 67.2
67.6
72.9
63.2
71.2
71.2
50.7
55.4
$7.5
UniDSeg
69.3
71.7
73.5
62.6
70.7
68.7
49.6
59.1
58.6
(SFDA3SS). Moreover, we compare UniDSeg under the source-free condition with the pioneering method SUMMIT [41]. Experimental results demonstrate that our method achieves superior perfor- mance, remarkably "USA/Sing." and "A2D2/<KITTI". Note that do not adopt any source- free adaptive learning methods, but only extend the DG3SS model to SFDA3SS tasks, to prove that good generalization model is conducive the execution of SFDA3SS. We have reason to believe that well-designed source-free adaptive learning can bring even more performance improvements.
A.4 Re-training DA3SS with Pseudo-Label
In general, cross-modal learning and self-training with pseudo-labels are complementrry in their combination. As shown in Tab, [1] when re-train the DA3SS model with PL, our method still achieves competitive performance. Similar to SFDA3SS, when selecting PL from the target-domain data, we consider the ensemble result "xM" as the fusion PL to supervise both the 2D and 3D branches.
Limitation. Nevertheless, compared other methods that achieve performance improvements of 1~2% mloU in 2D prediction with the help of target PL, our method only improves via the 3D pseudo supervision of the target domain, with almost no impact on the 2D prediction. We speculate that the proposed learnable- learnable-parameter parameter-inspired mechanism is more effective in unleashing the potential of VFMs, providing more pre-existing target information. With the increase in reliable target label information, the limited learnable parameters cannot bear more contextual information, which hinders the "1+1=2" effect. This is the problem that our unified cross-domain 3D semantic segmentation model needs to be improved in the future.
A.5 Evaluation on the Roles of Sparse Depth
MM2D3D 66] provides an effective approach to solving the DA3SS task by utilizing depth as an auxiliary input. By adding a parallel reinitialized encoder to the 2D backbone to process sparse depth obtained from the point cloud. Therefore, we consider analyzing whether the sparse depth needs to deeply encoded or employed as a prompt. As shown in Fig. 6 we illustrate the architecture of two learning manners: (a) Depth Deep Encoding and (b) Depth as Point-level Prompts.
17

---

Table 11: Performance comparison of multi-modal domain adaptive 3D semantic segmentation methods with pseudo-label ("p1.") ") re- re-training on four typical scenarios.
S.Source / T.Target
MuScenes:USAASiin
Day/ighs
VKITTIAKITTI
A2D2/KKITTI
Task
Method
2D
xM
2D
xM
3D
xM
3D
xM
xMUDAPt
67.0
71.2
57.6
69.6
64.4
51.4
52.0
41.2
49.8
47.5
AUDApz
30
65,9
65.3
70.6
54.3
69.6
61.1
35.9
45.5
45.9
46.8
48.1
50.6
D&CML
65.6
57.5
66.9
51,4
49.8
53.8
39.6
41.8
42.2
46.8
51.8
52.4
Dual-Cross
28
66.5
59.8
68.8
59.1
69.8
68.2
43.1
39.4
47.6
44.9
52.8
52.3
DA
SSEpz
S8
66.9
64.4
70.6
59.1
67.0
66.3
47.2
53.5
55.2
45.9
$1.5
52.5
BFID
45
65.9
66.0
71.3
60.6
70.0
66.6
48.6
55.4
57.5
42.6
53.7
52.7
MM2D3Dpt
74.3
74,9
71.3
69.6
72.2
55.0
59.7
48.7
50.7
UniDSegr1
67.4
70.0
64.5
71.9
71.7
52.7
62.7
50.4
57.3
58.6
For (a), similar to MM2D3D, we replace the 2D backbones with two ViT-B models. The sparse depth is fed into the ViT-B that trains from scratch, while the image is fed into the VIT-B pre-trained on large-scale datasets. After that, we can obtain encoded output by concatenating two representations. For (b), that is the simple architecture of only sparse depth as prompt for fine-tuning. Tab. T2 shows the comparison results of the aforementioned learning manners. Compared to (a), (b) not only requires 0.6% fine-tuning parameters of the 2D backbone but also achieves superior segmentation performance, making it flexible to expand various 3D semantic segmentation tasks with multi- learning.
Vir
Prompts
Sparse Depth
Sparse Depth
Pretrained -VIT
Pretrained -VIT
Image
Image
(a) Depth Deep Encoding
(b) Depth as Point-level Prompts
Figure 6: Role of Depth with brief diagram.
Table 12: Performance comparison of two sparse depth learning roles for DG3SS.
SUSeenes:Sing/UUA
Role Depth
Params
2D
3D
xM
Deep Encoding
86.9M
66.1
67.7
73.0
Point-level Prompts
0.48M
67.8
67.9
73.8
A.6 Additional Visualization Results
In this part, we demonstrate more qualitative results of our proposed UniDSeg to illustrate the effectiveness of our DA3SS framework. The corresponding qualitative results from 2D and 3D predictions will also be provided (See Figs and I0). Note that the prediction differences are signed in the red rectangle.
18

---

2D
3D
Image
Avg
Groundtruth
2D
3D
Image
Avg
Groundtruth
Car
Drivable Surface
Sidewalk
Terrain
Manmade
Vegetation
Igmore
Figure 7: Additional qualitative results DayNenes:Day/Night scenario for DA3SS.
19

---

2D
3D
Image
Avg
Groundtrth
2D
3D
Image
Avg
Groundtruth
Car
Drivable Surface
Sidewalk
Terrain
Manmade
Vegetation
Ignore
Figure 8: Additional qualitative results of muScenns::SAASimee scenario for DA3SS.
20

---

3D
Image
AVg
Groundrruth
2D
3D
Image
Avg
Groundtruth
Car
Truck
Road
Object
Building
V V'ggetaton
Ignore
Figure 9: Additional qualitative results of иKITTи/sKITT scenario for DA3SS.
3D
image
Avg
Groundtruth
20
3D
Image
A
Groundruuth
Truck
Bike
Person
Road Vegetation
Sidewalk
Parking
Object
Building
Ignore
Figure 10: Additional qualitative results of A2D2/>KITTI scenario for DA3SS.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim our contribution, that is, universal framework to enhance the adaptability and generalizability of cross-domain 3D semantic segmentation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: In the appendix A.4 we point out the limitation of re-training DA3SS with pseudo-label (PL.) with almost no impact on the 2D prediction.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer: [NA]
Justification:
This paper does not include theoretical results with proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Tab, shows the comparison result of DA3SS and DG3SS. Tab. 8 shows the detailed data and category splits. Tabs. 10 and I1 tabulate the expansibility of UniDSeg in other 3D semantic segmentation tasks. All results are evaluated on the public 3D datasets. Code is available at btttps://githute www.coo/aaaaaa http//Barcaaaa/nnii UnnDSe..
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
23

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
Code is available httttp:://gtuubb...mm//rrrrreeee
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
public/guides wttpss//ooooebboiiiioooooooooeee
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7/nips cc/ wtc/publiic// cubbiic/guides gwt.///ieee//eee// Codeeoomiiisseee /usd/guuuude//oeuubbiiiiii SanSSuubissss solbbmisssioo ssissoolicyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In sec 4.2 we illustrate the training details, including networks, learning rates, loss weights, etc.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[NA]
Justification:
Our experimental results do not contain statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Tab. 2 we compare the trainable parameters of 2D visual backbone. All experiments conducted on NVIDIA RTX 3090 with 24GB RAM.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed the NeurlPS Code of Ethics. This paper and code URL are anonymous
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper has societal impact of the work performed. No Al ethics involved, and no private data is involved.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
25

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: This paper and code URL are anonymous.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
26

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [No]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [No]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "And2Uish3j", "reviewer": "Reviewer_DeKz", "summary": "The manuscript proposes a universal method with the help of off-the-shelf Visual Foundation Models (VFMs) to boost the adaptability and generalizability of cross-domain 3D semantic segmentation, dubbed UniDSeg. The proposed method focus on learning visual prompt for 3D-2D transitional prior and deep query.", "questions": "Please refer to the Weaknesses Section.", "limitations": "The authors have clearly discussed the potential limitations of this work.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.This work is the first to introduce prompt-tuning concept into the universal model for DG3SS and DA3SS.\n2.The authors propose a learnable-parameter-inspired mechanism to the off-the-shelf VFMs for enhancing generalizability of VFMs.\n3.The proposed method achieves state-of-the-art results on DG3SS and DA3SS tasks.", "weaknesses": "1.What does Samp. in Figure 1 means, the corresponding explanation is not provided in the manuscript. \n2.Why choose CLIP as 2D backbone, since there is no language information for the task, other Visual Foundation Models (e.g. Swin-Transformer, SAM, …) seem more suitable for the 2D information extraction. \n3.In lines 148-152, the author mention “The main insight of UniDSeg is to provide a universal framework that enhances the adaptability and generalizability of cross-domain 3D semantic segmentation.” Though SparseConvNet is examined as 3D network, it would be more convincing as “universal” if authors could provide results utilizing other 3D backbones.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer DeKz,\n\nWe thank you for the precious review time and valuable comments. We have provided corresponding responses and results, which we believe have covered your concerns.\n\nWe hope to further discuss with you whether your concerns have been addressed or not. If you still have any unclear parts of our work, please let us know. Thanks.\n\nBest,\nAuthors\n\n**Q1**:  What does Samp. in Figure 1 means, the corresponding explanation is not provided in the manuscript.\n\n**A1**: \"Samp.\" means sampling of 2D features. Only the points falling into the intersected field of view are geometrically associated with multi-modal data (i.e., 3D-to-2D projection). After \"Samp.\" operation, we can obtain point-wise 2D features for cross-modal learning with 3D features.\n\n**Q2**: Why choose CLIP as 2D backbone, since there is no language information for the task, other Visual Foundation Models (e.g. Swin Transformer, SAM, …) seem more suitable for the 2D information extraction.\n\n**A2**: Thank you for your reminder. The motivation of this work is to study a **universal framework based on VFMs** to enhance the generalizability and adaptability of cross-domain 3D semantic segmentation, demonstrating the effectiveness of the visual foundation model priors. To this end, we utilize the off-the-shelf image encoder of VFMs as our visual backbone. UniDSeg **can be applied to various downstream tasks, including DA3SS, DG3SS, SFDA3SS, and fully-supervised 3SS**. Following your suggestion, we have supplemented the experimental results for DA3SS and DG3SS under another visual backbone, SAM: ViT-L. As shown in the table below, SAM-based UniDSeg exhibit better performance on \"USA/Sing.\" scenario.\n| Task | Method | Visual Backbone | 2D | 3D | Avg |\n|-|:-:|:-:|:-:|:-:|:-:|\n| DG | UniDSeg | CLIP: ViT-L | 66.5 | 64.5 | 72.3 |\n| | | **SAM: ViT-L** | **66.8** | **64.7** | **72.6 (+0.3)** |\n| DA | UniDSeg | CLIP: ViT-L | 67.2 | 67.6 | 72.9 |\n| | | **SAM: ViT-L** | **67.8** | **68.8** | **73.3 (+0.4)** |\n\nFurthermore, we supplement the experimental comparison between fine-tuning and our method, implemented on SAM image encoder serving as a visual backbone. Due to rebuttal time limitation, we will supplement all SAM-based experimental results subsequently.\n| Task | Visual Backbone | Strategy | 2D | 3D | Avg |\n|-|:-:|:-:|:-:|:-:|:-:|\n| DG | SAM: ViT-L | Fine-tuning | 65.9 | 64.3 | 70.8 |\n| | | Ours | 66.8 | 64.7 | 72.6 |\n| DA | SAM: ViT-L | Fine-tuning | 66.5 | 67.9 | 71.4 |\n| | | Ours | 67.8 | 68.8 | 73.3 |\n\n**Q3**: Though SparseConvNet is examined as 3D network, it would be more convincing as \"universal\" if authors could provide results utilizing other 3D backbones.\n\n**A3**: Thank you for your suggestion. We have supplemented the DA3SS experimental results for xMUDA and our proposed UniDSeg under another 3D backbone network, MinkowskiNet. Due to rebuttal time limitation, we will supplement all MinkowskiNet-based experimental results subsequently.\n| 3D Backbone | DA3SS Method | 2D | 3D | Avg |\n|-|:-:|:-:|:-:|:-:|\n| SparseConvNet | xMUDA | 64.4 | 63.2 | 69.4 |\n| | UniDSeg | 67.2 | 67.6 | 72.9 |\n| MinkowskiNet | xMUDA | 65.9 | 64.0 | 69.7 |\n| | UniDSeg | 67.5 | 68.6 | 73.1 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IiV3SzCUye", "reviewer": "Reviewer_KKSy", "summary": "This paper proposes a cross-domain 3D semantic segmentation model which utilizes off-the-shelf visual foundation models to boost the adaptability and generalizability. Two key designs are described to help the cross-domain task, e.g., visual prompt learning and deep query learning. Extensive experiments have been reported in the paper to illustrate the strength.", "questions": "1. I wonder about the motivation behind exploring the segmentation task in DA. As the paper has mentioned some foundation models like SAM, I want to ask that is it still meaningful to research the segmentation task in DA. A foundational segmentation model like SAM seems to be general enough (What's SAM's performance on the datasets mentioned in the paper?). Even if it's not powerful enough, for segmentation which is a kind of \"high-level\" task, it seems to be easier to directly improve the foundation model to achieve a real general segmentation model without any adaption.   \n2.  Other works like VFMSeg produce more accurate pseudo-labels to help the training, but in the limitation mentioned in this paper, producing pseudo-labels cannot achieve many performance improvements. I wonder what is the difference here.", "limitations": "As mentioned in the limitation of the paper, not being able to exploit the ability to produce accurate pseudo labels of VFM would constrain the capacity of the model when it comes to a larger scale.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Experiments demonstrate that the proposed model indeed outperforms compared to baseline methods.    \n2. Appropriately freezing and finetuning strategy seems reasonable.    \n3. Consider prompt tuning in domain adaptation task is novel, as it can exploit the capability of VFM by prompt engineering.", "weaknesses": "1. The improvement in both quantitative and qualitative results does not seem to be strong enough.  \n2. Does not analyze the reasons behind the SOTA performance, especially compared to models like VFMSeg which also exploit the power of VFM. I hope more detailed comparison and insight can be given.  \n3. The motivation is not clear enough. Though the paper points out two natural questions when considering using VFM, some works have already used VFM to help with segmentation. Then the motivation should include how to improve upon these works.  \n4. As the model uses CLIP as the backbone, why not try a more advanced model?   \n5. The writing needs to be improved. For example, in line 29, the last sentence is not finished. Line 37, DG3SS does not require accessing target domain data, so the logic here is not convincing. Line 42, what do you mean by “source-domain data discrimination power”? what is the relation to your method? Line 53, what is “spatial-temporal synchronized prompt space”? Line 65, what is “pre-existing target awareness”?\nSome detailed explanations could be seen in the Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer KKSy,\n\nWe thank you for the precious review time and valuable comments. We have provided corresponding responses and results, which we believe have covered your concerns.\n\nWe hope to further discuss with you whether your concerns have been addressed or not. If you still have any unclear parts of our work, please let us know. Thanks.\n\nBest,\nAuthors\n\n**Q1**: The improvement does not seem to be strong enough.\n\n**A1**: Our method enhances the performance of both DG3SS and DA3SS. Notably, all loss functions are the same as those in xMUDA, and no typical data augmentation or style transfer methods are introduced to address domain shift issues. The following table shows that UniDSeg, when combined with cross-domain distillation loss from Dual-Cross, achieves best performance compared to SOTA MM2D3D. We believe that more cross-domain learning constraints can provide better model.\n| DA3SS Method | 2D | 3D | Avg |\n|-|:-:|:-:|:-:|\n| xMUDA | 55.5 | 69.2 | 67.4 |\n| Dual-Cross | 58.5 | 69.7 | 68.0 |\n| MM2D3D | 70.5 | 70.2 | 72.1 |\n| UniDSeg | 63.2 | 71.2 | 71.2 |\n| **UniDSeg + Dual-Cross** | 64.5 | 71.6 | **72.5 (+1.3)** |\n\nWhen combining with MM2D3D, a parallel reinitialized encoder is added to the 2D backbone to process the sparse depth. We have shown the comparisons in Table 8. On Sing./USA scenario, compared to (a), (b) not only requires 0.6% fine-tuning parameters of the 2D backbone but also achieves superior segmentation performance, making it flexible to expand to various 3D semantic segmentation tasks with multi-modal learning. For convenience, we reshow the table below:\n| Role of Depth | Params | 2D | 3D | Avg |\n|-|:-:|:-:|:-:|:-:|\n| (a) Deep Encoding | 86.9M | 66.1 | 67.7 | 73.0 |\n| (b) Point-level Prompts | 0.48M | 67.8 | 67.9 | 73.8 |\n\n**Q2**: Analyze the reasons behind the SOTA performance.\n\n**A2**: VFMSeg essentially utilizes SAM to generate instance masks for both the source and target images, applying mixing augmentation to the source and target domain data. This is a typical and useful method used to address domain shift in DA3SS. In contrast, our proposed UniDSeg exploits the image encoder of VFMs (e.g., CLIP and SAM) to extract robust visual prior information. UniDSeg proposes a learnable-parameter-inspired mechanism to the off-the-shelf image encoder of VFMs, which maximally preserves pre-existing target awareness in VFMs to further enhance its generalizability.\n\n**Q3**: The motivation is not clear enough.\n\n**A3**: Thank you for your suggestion. The motivation of this work is to study a **universal framework based on VFMs** to enhance the generalizability and adaptability of cross-domain 3D semantic segmentation, demonstrating the effectiveness of the visual foundation model priors. Different from VFMSeg, which utilize SAM to generate instance masks for both the source and target images, applying mixing augmentation to the source and target domain data. Our proposed UniDSeg is groundbreaking in introducing the **depth-guided prompt-tuning** concept into the image encoder of VFMs, which can be applied to various downstream tasks, including DA3SS, DG3SS, SFDA3SS, and fully-supervised 3SS.\n\n**Q4**: Why not try a more advanced model?\n\n**A4**: We supplement the experimental results for DA3SS and DG3SS under SAM: ViT-L. As shown in the table below, SAM-based UniDSeg exhibit better performance on \"USA/Sing.\" scenario.\n| Task | Method | Visual Backbone | 2D | 3D | Avg |\n|-|:-:|:-:|:-:|:-:|:-:|\n| DG | UniDSeg | CLIP: ViT-L | 66.5 | 64.5 | 72.3 |\n| | | **SAM: ViT-L** | 66.8 | 64.7 | **72.6 (+0.3)** |\n| DA | UniDSeg | CLIP: ViT-L | 67.2 | 67.6 | 72.9 |\n| | | **SAM: ViT-L** | 67.8 | 68.8 | **73.3 (+0.4)** |\n\n**Q5**: Some detailed explanations could be seen.\n\n**A5**: (1) \"source-domain data discrimination power\" refers to the scenario in the DG3SS task where learning is conducted solely access to source domain data, enabling the model to develop the ability to discriminate domain-specific and domain-agnostic features; (2) Spatial: introducing spatial information (depth) into the prompt space to supplement 2D images; Temporal: since most scenes are captured smoothly, we learn the temporal invariance from the temporal correlations between the adjacent frames; (3) \"pre-existing target awareness\": In fine-tuning stage, the weights of VFMs are conventionally used to initialize source models and subsequently discarded. However, VFMs have diverse features important for generalization, and finetuning on source data can overfit to source distribution and potentially lose pre-existing target information.\n\n**Q6**: What's SAM's performance on the datasets?\n\n**A6**: In Table 2 of our paper, the experimental results demonstrate that, whether freezing or fine-tuning the image encoder of the VFM, it cannot directly solve the domain shift issue present in DG3SS and DA3SS. We supplement the experimental comparison between fine-tuning and our method, implemented on SAM image encoder serving as a visual backbone. The motivation please refer to **A3**. For other high-level tasks, they are not the focus of this work.\n| Task | Visual Backbone | Strategy | 2D | 3D | Avg |\n|-|:-:|:-:|:-:|:-:|:-:|\n| DG | SAM: ViT-L | Fine-tuning | 65.9 | 64.3 | 70.8 |\n| | | Ours | 66.8 | 64.7 | 72.6 |\n| DA | SAM: ViT-L | Fine-tuning | 66.5 | 67.9 | 71.4 |\n| | | Ours | 67.8 | 68.8 | 73.3 |\n\n**Q7**: Pseudo-label difference and improvement compared to VFMSeg.\n\n**A7**: Based on the VFMSeg experimental results, this method did not provide helpful pseudo-labels in \"Day/Night\" scenario, and there was even a decrease (-0.4) in performance. VFMSeg uses the averaging the probabilistic prediction of pretrained 2D network and SEEM to generate pixel-wise pseudo-labels, while our method follows the common practice adopted by most DA3SS methods in using offline 2D pseudo-label.\n| DA3SS Method | 2D mIoU |\n|-|:-:|\n| xMUDA | 55.5 |\n| xMUDA+PL | 57.5 |\n| VFMSeg+PL | 57.1 (-0.4) |\n| UniDSeg | 63.2 |\n| **UniDSeg+PL** | 64.5 (+1.3) |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3VBo0gxuut", "reviewer": "Reviewer_4rFB", "summary": "This paper introduces the prompt-tuning concept into DG3SS and DA3SS, and proposes a learnable parameter heuristic mechanism for the off-the-shelf VFM. Modal Transitional Prompting is proposed to capture 3D-to-2D transitional prior and task-shared knowledge from the prompt space. Learnable Spatial Tunability is constructed to the representation of distinct instances driven by prompts in the query space. Extensive experimental results demonstrate the effectiveness of the proposed method on widely recognized tasks and datasets.", "questions": "1. In Section 3.2, Modal Transitional Prompting, line 199, the author mentioned \"From the view of deep encoding, it focuses on the scope of scenes at different scales, so that their corresponding features have different content representations when constructed.\" Why does sparse depth have scenes at different scales?\n2. Why does the model require multiple layers of PG and TB? Experiments are needed to verify the impact of the number of layers on the results.\n3. How long does it take from the start of training to completion?\n4. In Table 1, caption introduction: \"Avg\" is obtained by averaging the predicted probabilities from the 2D and 3D networks. However, \"Avg\" is ambiguous here. This leads to misleading averages of 2D and 3D mIou results, but this may not be the case in reality. It is recommended to replace \"Avg\" with another representation.", "limitations": "As above", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method of improving cross-domain 3D semantic segmentation based on visual basic models is interesting and innovative.\n2. The proposed Modal Transitional Prompting (MTP) and Learnable Spatial Tunability (LST) explore and utilize 2D prior knowledge from VFMs.\n3. Experiments on multiple datasets verify the effectiveness of the method.", "weaknesses": "1. The method lacks theoretical analysis. In Section 3.2, Modal Transitional Prompting, line 193, the author mentioned that \"PG is designed to capture 3D-to-2D transitional prior and task-shared knowledge from the prompt space.\" However, the designed method only uses learnable parameters, which is not related to the 3D-to-2D transitional prior and the promotion of 3D domain adaptation. This section only explains the steps of the method and lacks theoretical explanation of the method.\n\n2. In Section 3.2, Learnable Spatial Tunability, the author mentioned that it was inspired by Rein [1]. However, it seems that the proposed LST has a similar design to Rein, which casts doubt on the novelty of LST. The modification of Rein lacks innovation and theoretical analysis. This inevitably makes people think that the good performance of LST is due to the effectiveness of Rein-like components.\n\n3. In the experimental section, VFM is only verified based on the classification model CLIP [2]. However, the CLIP paper [2] states that “Additionally, CLIP is not designed for common surveillance-relevant tasks like object detection and semantic segmentation. This means it has limited use for certain surveillance tasks”. The proposed method is dedicated to segmentation tasks, but it is only verified on the classification model CLIP. As a pioneer method, it is recommended to verify its effectiveness on VFMs designed for segmentation tasks such as SAM [3] and SEEM [4].\n\n[1]Wei, Zhixiang, Lin Chen, Yi Jin, Xiaoxiao Ma, Tianle Liu, Pengyang Ling, Ben Wang, Huaian Chen, and Jinjin Zheng. \"Stronger Fewer & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28619-28630. 2024.\n\n[2]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748–8763, 2021.\n\n[3]Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4015–4026, 2023. 1, 2, 3, 6, 11\n\n[4]Zou, Xueyan, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. \"Segment everything everywhere all at once.\" Advances in Neural Information Processing Systems 36 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 4rFB,\n\nWe thank you for the precious review time and valuable comments. We have provided corresponding responses and results, which we believe have covered your concerns.\n\nWe hope to further discuss with you whether your concerns have been addressed or not. If you still have any unclear parts of our work, please let us know. Thanks.\n\nBest,\nAuthors\n\n**Q1**: This section only explains the steps of the method and lacks theoretical explanation of the method.\n\n**A1**: Firstly, sparse depth map as a perspective projection representation of point clouds presents an unnatural image that can be processed by the image encoder in VFM. This is an additional prior derived from the point cloud. Lee et al. [Ref1] have been proved that source data internally know a lot more about the world and how the scene is formed, which called Privileged Information. This Privileged Information includes physical properties (e.g., depth) that might be useful for cross-domain learning. Secondly, though depth information is easy-to-access and tightly coupled with semantic information, Hu et al. [Ref2] argue that depth clues that complement colors are hard to deduce from color images alone, thus directly deep encoding fail to capture valid geometric information. To this end, we consider utilizing depth as point-level prompts inserted into ViT layers, enabling the model to learn spatial distance perception. This complements 2D representations, as LiDAR-derived depth information is less affected by domain variations compared to images, which can be easily influenced by changes in lighting and other factors. Experimental results in Table 8 further demonstrate the effectiveness of our method. For convenience, we reshow the table below:\n| Role of Depth | Params | 2D | 3D | Ens|\n|-|:-:|:-:|:-:|:-:|\n| (a) Deep Encoding | 86.9M | 66.1 | 67.7 | 73.0 |\n| (b) Point-level Prompts | 0.48M | 67.8 | 67.9 | 73.8 |\n\n[Ref1] Lee K H, Ros G, Li J, et al. Spigan: Privileged adversarial learning from simulation. arXiv preprint arXiv:1810.03756, 2018.\n\n[Ref2] Hu S, Bonardi F, Bouchafa S, et al. Multi-modal unsupervised domain adaptation for semantic image segmentation. Pattern Recognition, 2023.\n\n**Q2**: The novelty of LST.\n\n**A2**: The innovation of our method is in introducing learnable-parameter-inspired mechanism to the off-the-shelf VFMs, which is guided by point-level prompts from 3D information. To achieve this, we place layer-wise MTP and LST blocks to take full advantage of semantic understanding of diverse levels and modalities. Among them, the affinity matrix computed in LST is a common solution to capture the associations between learnable tokens and 2D representations. The insight of learnable token originally comes from Visual Prompt Tuning (VPT) [Ref3].\n\n[Ref3] Jia M, Tang L, Chen B C, et al. Visual prompt tuning. European Conference on Computer Vision, 2022: 709-727.\n\n**Q3**: As a pioneer method, it is recommended to verify its effectiveness on VFMs designed for segmentation tasks such as SAM [3] and SEEM [4].\n\n**A3**: We have supplemented the experimental results for DA3SS and DG3SS under another visual backbone, SAM: ViT-L. Notably, in this work, we only utilize the off-the-shelf image encoder of VFMs (e.g., CLIP (w/o text encoder) and SAM (w/o prompt encoder and mask decoder)). As shown in the table below, SAM-based UniDSeg exhibit better performance on \"USA/Sing.\" scenario. Due to rebuttal time limitation, we will supplement all SAM-based experimental results subsequently.\n| Task | Method | Visual Backbone | 2D | 3D | Ens |\n|-|:-:|:-:|:-:|:-:|:-:|\n| DG | UniDSeg | CLIP: ViT-L | 66.5 | 64.5 | 72.3 |\n| | | **SAM: ViT-L** | 66.8 | 64.7 | **72.6 (+0.3)** |\n| DA | UniDSeg | CLIP: ViT-L | 67.2 | 67.6 | 72.9 |\n| | | **SAM: ViT-L** | 67.8 | 68.8 | **73.3 (+0.4)** |\n\n**Q4**: Why does sparse depth have scenes at different scales?\n\n**A4**: We are sorry to make you misunderstand. Here, \"different scales\" refers to different receptive fields of adjacent pixels. The complementarity of the features extracted by the 2D and 3D branches is also tightly correlated to the different information processing machinery, i.e., 2D and 3D convolutions, which makes networks focusing on different areas of the scene with different receptive fields. When sparse depth is projected onto the image plane and processed via a 2D network, it can focus on the scope of scenes at different scales. For example, adjacent pixels in the image may have similar pixel values, but their corresponding depth values (distance) in the depth map can differ significantly.\n\n**Q5**: Experiments are needed to verify the impact of the number of layers of MTP and LST.\n\n**A5**: In the following table, \"Half layer\" indicates that the MTP and LST blocks are inserted every other layer. Due to rebuttal time limitation, we will supplement the experiments of other layer combinations subsequently.\n| Visual Backbone | MTP | LST | 2D | 3D | Ens|\n|-|:-:|:-:|:-:|:-:|:-:|\n| CLIP: ViT-L | All layers | All layers | 66.5 | 64.5 | 72.3 |\n| | Half layers | Half layers | 66.1 | 64.3 | 71.9 |\n\n**Q6**: How long does it take from the start of training to completion?\n\n**A6**: All experiments are conducted on one NVIDIA RTX 3090 GPU with 24GB RAM.\n| Task | USA/Sing. | Day/Night | vKITTI/sKITTI | A2D2/sKITTI |\n|-|:-:|:-:|:-:|:-:|\n| DA | ~22h | ~37h | ~21h | ~48h |\n| DG | ~12h | ~21h | ~11h | ~27h |\n\n**Q7**: \"Avg\" is ambiguous. This leads to misleading averages of 2D and 3D mIoU results.\n\n**A7**: Initially, xMUDA used \"2D+3D\" to denote the final results, but we found that \"2D+3D\" might be misleading as it implies an average of 2D and 3D. Therefore, we follow MM2D3D and VFMSeg by using \"Avg\" to denote the final result. Perhaps \"Ensemble Result\" (short for \"Ens\") would be more appropriate. Notably, in all your answers, we have changed \"Avg\" to \"Ens\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vxy96Midy4", "reviewer": "Reviewer_T5Kw", "summary": "This paper presents UniDSeg, a universal approach that enhances the adaptability and generalizability of cross-domain 3D semantic segmentation. To achieve this, we propose a learnable parameter-inspired mechanism for off-the-shelf VFMs with frozen parameters. This mechanism maximally preserves the pre-existing target awareness in VFMs, thereby further enhancing their generalizability.", "questions": "1.\tPlease explain why token length is equal to 100, the best performance can be obtained. However, it is important to note that the prompt tuning process may not be entirely stable. To address this, all experiments should be conducted multiple times, and the average performance along with the standard deviation also should be computed and reported.\n2.\tPlease explain the main contribution compared with “Learning to Adapt SAM for Segmenting Cross-domain Point Clouds”.\n3.\tSince VFM and ViT are utilized as encoder, the model parameters and computational costs should be comprehensive reported.\n4.\tDid the authors notice that when the point labels projected onto 2D images, there are slight mismatch between these labels and 2D image pixels? Please discuss the potential reasons.", "limitations": "1.\tThe authors have addressed the limitations and potential negative societal impact of their work.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe motivation is clear.\n2.\tThe proposed method is intuitive, and the experiments have validated their contributions.", "weaknesses": "1.\tSince the backbone is modified from xMUDA, the oracle performance should also be provided.\n2.\tThe task settings in this study are derived from xMUDA (TPAMI), while the prompt tuning and corresponding experimental settings are based on VPT. Therefore, this proposed method can be considered somewhat incremental.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer T5Kw,\n\nWe thank you for the precious review time and valuable comments. We have provided corresponding responses and results, which we believe have covered your concerns.\n\nWe hope to further discuss with you whether your concerns have been addressed or not. If you still have any unclear parts of our work, please let us know. Thanks.\n\nBest,\nAuthors\n\n**Q1**: Since the backbone is modified from xMUDA, the oracle performance should also be provided.\n\n**A1**: \"Oracle\" means solely training on the target domain, except in the \"Day/Night\" case where it uses a 50%/50% source/target batches to prevent overfitting due to the small target size, thereby serving as the upper bound for DA3SS.\n| | USA/Sing. | | | Day/Night | | | vKITTI/sKITTI | | | A2D2/sKITTI | | |\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| | 2D | 3D | Avg | 2D | 3D | Avg | 2D | 3D | Avg | 2D | 3D | Avg |\n| Source-only | 58.4 | 62.8 | 68.2 | 47.8 | 68.8 | 63.3 | 26.8 | 42.0 | 42.2 | 34.2 | 35.9 | 40.4 |\n| UniDSeg | 67.2 | 67.6 | 72.9 | 63.2 | 71.2 | 71.2 | 60.5 | 50.9 | 62.0 | 50.7 | 55.4 | 57.5 |\n| Orcale| 75.4 | 76.0 | 79.6 | 61.5 | 69.8 | 69.2 | 66.3 | 78.4 | 80.1 | 59.3 | 71.9 | 73.6 |\n\n**Q2**: The task settings in this study are derived from xMUDA (TPAMI), while the prompt tuning and corresponding experimental settings are based on VPT. Therefore, this proposed method can be considered somewhat incremental.\n\n**A2**: Our method is groundbreaking in introducing the prompt-tuning concept into the **universal model for DG3SS and DA3SS**. In VPT-Deep, each prompt is a learnable token randomly initialized and directly insert into input space of all ViT layers. In contrast, **our method first leverages sparse depth as a point-level prompts inserted to each ViT layers**. It not only learns spatial distance perception prompts from point clouds but also learns invariance to sample perturbations. Secondly, **our method uses the learnable token as query for seeking matched prompting** after encoding in the ViT layer. It bridges the discrepancy between the information of pre-training dataset and the target scene.\n\n**Q3**: Please explain why token length is equal to 100, the best performance can be obtained. The average performance along with the standard deviation also should be computed and reported.\n\n**A3**: Consistent with all competitors, for each run of the experiment, we select the best model based on the validation set results (Via trial and error, when token length is set to 100, the result is stable and up to peak value), and use it for the final inference on the test set. All experimental settings are conducted at least 3 times and we select the best performance of all runs. The table below reports the 2D performance with standard deviation over 4 runs when the token length is set to 100.\n| | USA/Sing. | A2D2/sKITTI |\n|-|:-:|:-:|\n| 2D mIoU | 68.100&plusmn;0.071 | 43.975&plusmn;0.453 |\n\n**Q4**: Please explain the main contribution compared with \"Learning to Adapt SAM for Segmenting Cross-domain Point Clouds\".\n\n**A4**: Above work leverages whole SAM to generate instance mask, guiding the alignment of features from diverse 3D data domains into a unified domain. On the contrary, our proposed UniDSeg does not directly use instance masks generated offline, but instead uses only the pre-trained image encoder of VFMs (e.g., CLIP and SAM) to extract visual prior information. UniDSeg is groundbreaking in introducing the prompt-tuning concept into the universal model for DG3SS and DA3SS. We propose a learnable-parameter-inspired mechanism to the off-the-shelf image encoder of VFMs, which maximally preserves pre-existing target awareness in VFMs to further enhance its generalizability. Just as the following A5 response, our method not only requires nearly 2% fine-tuning parameters of the visual backbone but also achieves superior segmentation performance.\n\n**Q5**: Since VFM and ViT are utilized as encoder, the model parameters and computational costs should be comprehensive reported.\n\n**A5**: Thank you for your suggestion. We have reported model parameters of CLIP: ViT-B, CLIP: ViT-L, and SAM: ViT-L for the VFM based encoder along with all trainable parameters. Note that, the entire ViT backbone in our VFM-based encoder is frozen during downstream training for DA3SS and DG3SS. Only two layer-wise learnable blocks, MTP and LST, are trainable. Hereby, we provide supplementary information on the trainable parameters for MTP and LST in the following table, where \"Cost\" means only need percentage trainable parameters compared to fine-tuning whole encoder consume.\n| Visual Backbone | VFM-based Encoder | All trainable Params | **Cost** | MTP | LST |\n|-|:-:|:-:|:-:|:-:|:-:|\n| CLIP: ViT-B | 86.9M | 1.82M | **2.09%** | 0.48M | 1.34M |\n| CLIP: ViT-L | 305M | 4.70M | **1.54%** | 1.78M | 2.92M |\n| SAM: ViT-L | 307M | 4.34M | **1.41%** | 1.42M | 2.92M |\n\n**Q6**: Did the authors notice that when the point labels projected onto 2D images, there are slight mismatch between these labels and 2D image pixels? Please discuss the potential reasons.\n\n**A6**: The mismatch can be caused by a variety of factors, including calibration error, geometric error, image resolution, etc. These factors are beyond the scope of this work. Following xMUDA, we assume that the calibration of LiDAR-Camera is available for both domains and does not change over time. Since images and point cloud are heterogeneous, according to extrinsic matrix from dataset, only the points falling into the intersected field of view are geometrically associated with multi-modal data (i.e., 3D-to-2D projection).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper receives ratings with 1 accept, 1 borderline accept, and 2 borderline rejects. Most reviewers like the idea of bringing prompt tuning in to the domains of DG3SS and DA3SS and the extensive experiments to verify the proposed method. The main concerns raised by the reviewers including the use of CLIP as the backbone, novelty compared to existing 3SS methods that utilize VFMs, and marginal improvement against existing methods. The authors provide very detailed rebuttal to address the raised concerns. After reading the paper, reviews, and rebuttal, AC finds that the raised concerns are mostly addressed by the rebuttal with some nice ablation studies, and the improvement against the existing method is non-trivial. Given the novelty of the proposed methods, AC believes that the paper is ready for publication. The authors are encouraged to make the above mentioned necessary changes to the best of their ability. We congratulate the authors on the acceptance of their paper!
Decision: Accept (poster)