No Object Is an Island: Enhancing 3D Semantic Segmentation Generalization with Diffusion Models
Authors: Fan Li
Xuan Wang
Xuanbin Wang
Zhaoxiang Zhang
Yuelei Xu
Venue: NeurIPS 2025 poster ()
Keywords: domain generalization
3D semantic segmentation
diffusion model
Primary area: applications

Abstract: Enhancing the cross-domain generalization of 3D semantic segmentation is a pivotal task in computer vision that has recently gained increasing attention. Most existing methods, whether using consistency regularization or cross-modal feature fusion, focus solely on individual objects while overlooking implicit semantic dependencies among them, resulting in the loss of useful semantic information. Inspired by the diffusion model's ability to flexibly compose diverse objects into high-quality images across varying domains, we seek to harness its capacity for capturing underlying contextual distributions and spatial arrangements among objects to address the challenging task of cross-domain 3D semantic segmentation. In this paper, we propose a novel cross-modal learning framework based on diffusion models to enhance the generalization of 3D semantic segmentation, named XDiff3D. XDiff3D comprises three key ingredients: (1) constructing object agent queries from diffusion features to aggregate instance semantic information; (2) decoupling fine-grained local details from object agent queries to prevent interference with 3D semantic representation; (3) leveraging object agent queries as an interface to enhance the modeling of object semantic dependencies in 3D representations. Extensive experiments validate the effectiveness of our method, achieving state-of-the-art performance across multiple benchmarks in different task settings. Code is available at \url{https://github.com/FanLiHub/XDiff3D}.

No Object Is an Island: Enhancing 3D Semantic Segmentation Generalization with Diffusion Models
Fan Li
Xuan Wang Xuanbin Wang' Zhaoxiang Zhang'
Yuelei Xu !u
Northweetern Polytechnical University lifan mwwsagos@gmail..commo
Abstract
Enhancing the cross-domain generalization of 3D semantic segmentation is piv- otal task in computer vision that has recently gained increasing attention. Most existing methods, whether using consistency regularization or cross-modal feature fusion, focus solely on individual objects while overlooking implicit semantic dependencies among them, resulting in the loss of useful semantic information. Inspired by the diffusion model's ability flexibly compose diverse objects into high-quality images across varying domains, we seek to harness its capacity for capturing underlying contextual distributions and spatial arrangements among objects to address the challenging task of cross domain 3D semantic segmenta- tion. In this paper, we propose a novel cross-mode modal learning framework based diffusion models to enhance the generalization of 3D semantic segmentation, named XDift3D. XDifI3D comprises three key ingredients: (I) constructing object agent queries from diffusion features to aggregate instance semantic information; (2) decoupling fine- mne-grained local details from object agent queries to prevent in- terference with 3D semantic representation; (3) leveraging object agent queries as an interface to enhance the modeling of object semantic dependencies representations. Extensive experiments validate the effectiveness of method, achieving state-of-thed tftt--ff-he-art performance across multiple benchmarks in different task settings. Code is available at lttth:::/ :ttp:::///ithube wwt./aaa//a///DDDFF........
3D
Introduction
3D semantic segmentation, fundamental task in computer vision with widespread applications in autonomous driving, robotics, and augmented reality, has made significant advancements in recent [0]. Despite these advancements, it still experiences severe performance degradation when models trained on the source domain are applied to unseen target domains due to the existence of domain gap. This has sparked growing interest in Domain Generalized 3D Semantic
years 39 40
Segmentation (DG3SS) 28
68 aiming to learn Nomain-invarian features that enable models to perform well on a variety of unseen target domains with similar semantic distribution.
62
24
55
Current approaches can be broadly categorized into uni-modal methods based solely on point clouds and cross-modal methods that integrate both point cloud and image data, as shown in Figure [ The former focuses on reducing the domain gap between the source and target domains through domain domain mixing
but its performance remains limited. The latter exploits image features paired with point clouds to perform consistency regularization [37 19 59 55
augmentation 24
42 44
4 25
or cross moosl feature fusion [54] (8)), substantially improving cross-domain generalization and delivering impressive results. However, most existing approaches treat objects as isolated islands, focusing solely on nomain-invariant features of individual objects and overlooking the implicit
*Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Diffisioo Medel
2D Model
Pixel-ta-point
2D Model
Pixel-ta-point
[ Form
Consiis
Fussion
Objert Agee Ageut Queia
Orgact Agez: Querae
Leam
Semantic Enhance
Model
Seg. Loss
Model
Seg. Loss
3D Model
Seg.
(a) Consistency Regularization
(b) Cross-modal Fusion
(c) Our Method
Figure Existing methods mainly adopt two paradigms: (a) consistency regularization, which enforces alignment across modalities, and (b) cross-modal feature fusion, which integrates multi- modal information. However, both paradigms focus primarily on individual domain-invariant features, overlooking rich semantic dependencies among objects. In contrast, our method (c) leverages object agent queries as an interface to incorporate instance semantic dependencies from diffusion priors into the 3D semantic space, enhancing tross-domain generalization of 3D semantic segmentation.
semantic dependencies among them such as spatial arrangements and contextual relations, which leads to the loss of informative cues and ultimately leads to suboptimal performance.
Recently, diffusion models have provided new perspective for improving semantic segmentation generalization with remarkable capabilities in capturing underlying semantic relations among objects to synthesize high-quality samples across diverse domains [18
Building on this insight, several studies [38] (2 35 52 have leveraged diffusion priors to better model semantic dependencies among objects, thereby enhancing the generalization of 2D semantic segmentation. Given the remarkable progress of diffusion models in the 2D visual domain, a natural curiosity has been raised: How can the prior knowledge encoded in diffusion models be leveraged to improve the generalization of 3D semantic segmentation?
naive solution is treat diffusion models general feature extractors and utilize their features as supervisory signals or auxiliary information by pixel-to-poing matching for 3D semantic segmentation. However, this comes with two new challenges: (1) Strict pixel-to-poine matching leads to significant loss of image features due to point cloud sparsity [ 37). further aggravated by mainstream diffusion models operating compressed latent spaces rather than pixel spaces, hindering effective cross- modal feature association. (2) Diffusion models are inherently designed for generative tasks, and their feature spaces not only model high-level instance semantics but also retain fine eraeggainnd local visual details (e.g. slogans and scrawl) that may introduce noise and disturb 3D feature learning.
To address the above challenges, propose XDiff3D, a cross-modal learning framework via diffusion models for cross-domain 3D semantic segmentation, which leverages the semantic dependencies among objects embedded in pretrained diffusion priors enrich the learned representations of point clouds. Specifically, for the first challenge, we introduce learnable queries that interact with diffusion features to aggregate rich semantic dependencies among objects, forming object agent queries that subsequently refine the 3D feature representations. To tackle the second challenge, we introduce a dual-query learning scheme that enforces semantic consistency between the principal components of two sets of queries, effectively suppressing interference from potentially distracting visual details. Ultimately, these object agent queries serve as an interface to infuse 2D semantic priors into the 3D representation space, enabling the refinement of 3D semantic features in the decoder for more generalizable representations. Extensive experiments across multiple benchmark settings demonstrate that XDiff3D consistently delivers state-of-the-art performance, surpassing existing baselines by a significant margin. Moreover, comprehensive ablation studies and analyses further validate the effectiveness of its core components. The contributions are summarized as follows:
To the best of our knowledge, XDiff3D is the first diffusion- framework that constructs object agent queries as an interface to infuse 2D diffusion priors into the 3D representation space, thereby enhancing the cross-domain generalization of 3D semantic segmentation.
modal learning
We propose a dual-queries learning scheme that suppresses potentially intricate visual details embedded in diffusion priors to prevent interference with the learning of robust 3D feature representations.

---

XDiff3D is concise and general framework, consistently ouiperforming wide variety of baselines and achieving ofte-oof-theart performance across multiple benchmarks for cross-domain 3D semantic segmentation.
2 Related Works
2.1
Diffusion Models
Diffusion models [8] 46 12 41 have demonstrated impressive capabilities in image generation and increasingly being explored for visual perception tasks such as semantic segmentation [60] 49),
object detection 7 20)) and depth estimation ( Recent studies have begun to exploit diffusion For example, DGInStyle
models for improving domain generalization in 2D segmentation [38 35 4
21 introduces a controllable framework that generates diverse, task-specific images from diffusion priors. Niemeijer al. [35] text-guided diffusion synthesize pseudo-target domains for better coverage of target domain variations. Despite the progress in 2D tasks, the potential of diffusion priors enhance generalization in 3D semantic segmentation remains underexplored. This work addresses this gap by leveraging instance-level semantic priors from diffusion models to enrich 3D) semantic representations and improve their generalization with respect domain shifts.
2.2 Domain Generalized 3D Semantic Segmentation
Domain Generalized 3D Semantic Segmentation (DG3SS) aims to learn model solely from source domain data that can perform well across diverse unseen target domains, and has recently garnered significant attention in the research community [5] 62 58 55 [68]. SemanticSTF (58) introduces large-scale benchmark for semantic segmentation of LiDAR point clouds in adverse weather, enabling comprehensive evaluation of domain adaptive and generalizable 3D segmentation methods under all-weather conditions. MM2D3D [5] injects depth cues into the 2D branch and RGB information into the 3D branch to improve modality complementarity and robustness to domain shift. 2DPASS strategy to transfer rich semantic and structural information from 2D images to 3D point clouds without requiring strictly paired data. UniDSeg (55) leverages Visual Foundation Models by introducing learnable prompts within a cross-modal framework to bridge the 2D-3D domain gap and enhance generalization in cross-domain 3D semantic segmentation. Despite these ad advancmments, existing methods primarily focus on learning comainn-nnvarian for individual objects, while overlooking the latent semantic dependencies among objects. In contrast, this paper exploits the contextual relationships and spatial organization among objects to move beyond isolated instance modeling, enabling the model to grasp unified semantic distribution underlying diverse scenes, thereby enhancing cross-domain generalization.
[62] introduces a musion-then-distillaln
features
2.3 Learnable Queries Design
have adopted learnable query-based frameworks in- spired by DETR (6). MaskFormer and Mask2Former [9] unify semantic and instance segmentation using object queries. Tqdm 36) introduces domain-invariann textual queries for domain general- ized semantic segmentation. XMaX-DeepLLa [64] treats pixel-query interaction k-means-style clustering, simplifying cross-attention for better segmentation. While learnable queries have proven effective in various perception tasks, how to leverage them to enhance generalization in 3D semantic segmentation remains an open question. In this work, we derive object agent queries from diffusion features and utilize them as cross-modal interface enhance 3D semantic representations.
Recently, various approaches
36
30
66
51
Method
3.1 Preliminary
Stable Diffusion. Stable diffusion models comprise two complementary stochastic processes: diffusion process and reverse process. During the diffusion process, random noise progressively added to the data via Markov chain:
q ( z0) N' (zt Varzo, (1- 1) 20 E(x)
(I)
3

---

(a) Construction of Object Agent Queries
(b) Qual-queries Consistency Scheme
trained Encoder
Add Natse
Objject Agent Queries
Diffusion *
Model
SVD
Source Image
Aggregating Instance Semantic
Object Agent
Lnimaaee Oe min
tn oljen
Aullayy Ooject Cuenies
quariis
Min
Otgec Querio
Design
Principal Componeet Extraction
you.
Sive.v
Cross vosda Somantic Aligriment
fho
Aggregaling Instance Semantic
Festure Refinement
Feature Refinement
Corv
Decoder
Label
Feature Refinement
Backbone
Feature Refinement
Cross-modal Semantic
Figure 2: A brief illustration of our proposed framework. First, aggregate semantic information of objects within scene from diffusion features to form the object agent queries. Next, we propose dual-query mechanism to eliminate local visual details from the object agent queries, preventing interference with 3D semantic representation. Finally, the optimized object agent queries are used interface to infuse inter-object semantic dependencies into the 3D representation, guiding the model to learn domain-invarian invariant features.
where denotes a pretrained VAE encoder that maps an input image R-Hxxx3 into latent representation ag- The hyperparameters ã; represent a pre-defined noise schedule, where larger / corresponds larger noise weights. The reverse process gradually reconstructs clean data from noisy samples using a noise predictor e. Each step of this reverse process can be formulated as:
(#t-1 21) = A (2--1 (Z+, t). Eo (2,, t))
where is the mean predicted by c0(-), and Eg is typically set to predefined covariance value.
Problem Definition. The goal of DG3SS is to train segmentation model solely labeled source domain S, enabling it to generalize to unseen target domains T. For the source domain, it is assumed that paired images and point clouds available. In general, the segmentation model o g consists of a backbone g for feature extraction and a decoder e for producing semantic predictions.
3.2 Overview
With diffusion models successfully enhaneing segmentation generalization in the 2D vision domain drawing on strong instance semantic priors, a natural question arises: How can the prior knowledge encoded in diffusion models be leveraged bolster the generalization 3D semantic segmentation? As illustrated in Figure 2 we propose XDif13D, a cross-modal learning framework for cross-domain 3D semantic segmentation based on diffusion models, which consists of three key ingredients: (1) aggregating object semantic information from the prior knowledge encoded in diffusion features to generate object agent queries (Section 3.3). (2) suppressing potential intricate visual details embedded within the object agent queries (Section 3.4), and (3) leveraging the optimized object agent queries as an interface perform cross-modal semantic enhancement (Section 3.5)
3.3 Construction of Object Agent Queries
Given that diffusion models operate in compressed latent spaces, traditional calibration- based hard associations between LiDAR points and image pixels become unreliable. fully harness the rich instance semantic priors captured by diffusion features, we construct object agent queries from the diffusion space and employ them an interface for infusing semantic knowledge into the 3D feature space, with the goal of enhancing the generalization capability of 3D semantic segmentation.

---

Diffusion Feature Extraction. Before that, we need to extract diffusion features. To do this, we begin by encoding the source image a into a latent code 20 {2) via pretrained encoder Subsequently, we introduce noise to according to Equation | yielding the noisy latent code Ey-
Finally, we apply the noise predictor <a(-) to perform the denoising process to get diffusion features:
sed (2,, ((e)), with 1 SisM M
where T denotes the text encoder, fed represents the i-th layer features from the diffusion model, the empty text, and M is the total number of diffusion feature layers.
Aggregating Instance Semantic Features. We employ multiple sets of learnable queries. each corresponding to a specific layer of diffusion features. Specifically, for the i-th layer diffusion features, we map them to key (K') and value (V') vectors, while corresponding set of randomly initialized learnable queries is mapped to query (Q') vectors:
Prnit We K' fed Wi: K V = f Wi. with Ginit E -
where W Q. Wi and W are linear projection matrices, Linit denotes randomly initialized learnable queries, e is the dimension of Linie and r is the sequence length of Gina. Next, the layer-wise object agent queries are formed follows:
exp (s*)
8 Q'(K+)² with Lins R Vdh
Pons FFN
Vi
(5)
how exp (8+)
where lins are the object agent queries of the i-th diffusion layer, d' is a scaling factor, and FFN consists of a linear mapping followed by an activation layer. h and w denote the height and width of the similarity matrix s", respectively. After the final layer, M, we compute both the maximum and average components across all layer-wise object agent queries to obtain the global object agent queries:
M x Wa - ba, with Tins E Rrxx
Gins
max Pins + 1=1,2,..,M
Pins
(6)
M
where Wa and ba signify the weights and biases, respectively. Average queries encode global contextual information to improve robustness to noise, while max queries emphasize the most prominent and distinctive signals, highlighting key semantic features. As a result, the object agent queries establish implicit associations with scene instances and learn the semantic dependencies among objects embedded in diffusion features. Acting as an interface, these queries guide the 3D feature representations to beyond isolated individuals, enabling the model to grasp consistent semantic patterns across diverse scenario domains.
3.4 Dual-queries Consistency Scheme
As diffusion models are inherently designed for generative tasks, their feature spaces encode not only high-level instance semantics but also intricate local visual details. These details, though irrelevant to 3D semantic segmentation, may be inadvertently propagated into object agent queries during interaction process and disrupt the learning of 3D semantic representations. To this end, we propose dual-query consistency scheme decouple potential local visual details from the object agent queries.
Dual-queries Design. To achieve this, we first construct auxiliary set of object agent queries gio. guided by diffusion features at higher timesteps t', following the procedure described in Scction 3.3 Different timesteps in the diffusion process correspond to successive stages of denoising. Higher timesteps introduce stronger noise that blurs local visual details, while the overall semantic structure of the scene remains largely preserved. As result, the primary difference between the diffusion features at timestep t' and t lies in the superficial visual details.
Principal Component Extraction and Consistency Constraint. Recently, SoMA 65 reveals that the principal components derived from singular value decomposition (SVD) of weight matrices in vision foundation models (VFMs) capture generalized world knowledge, which underpins their strong generalization capability. Inspired by this, the object agent queries, as learnable parameters, expected to exhibit similar properties with principal components that emphasize domain- invariant
5

---

instance semantic knowledge rather than intricate visual details. To this end, we first perform singular value decomposition (SVD) on the two sets of object agent queries:
gins UEV", Lins = U'E'YT' with U E R' EE R V VERO
(7)
where U (or U') and V (or V") are the left and right singular vectors, E (or E') is diagonal matrix whose entries are singular values arranged in descending order. We select the top-k singular values
along with their associated components in U (or U") and V (or V") to generate the quantized object agent queries qins_pe and Kenspe''
Qins_pe Up.. EAA) [kk.)) Lins.n- Ui.aa 2(aa)
y [RR.)) with Qins_pe _pc E <R' Vins ppc E R (8)
Subsequently, we impose consistency constraint between the two sets of quantized agent queries:
(F))
Connsist
deme_pe (f) log
MLP (I.P((is_pell
where the MLP includes linear transformation followed by layer normalization. By enforcing consistency constraint, the principal components of the object agent queries are effectively guided to focus on instance semantic distribution rather than local visual details. This is because encoding excessive visual details within the principal components of the object agent queries would result in substantial discrepancies between the two sets of queries, Qine_pe and ''ns_pe' Thus, the consistency loss reduces these discrepancies. ensuring that only semantic dependencies among objects are retained in the constructed object agent queries, while irrelevant visual details are effectively suppressed.
3.5 Cross-modal Semantic Enhancement
Our ultimate goal is to improve the cross-domain generalization of 3D segmentation models. To this end, we leverage object agent queries as an interface for cross- modsl semantic alignment, which subsequently enables the refinement of 3D feature representations to better capture semantic dependencies among objects across domains.
Cross-modal Semantic Alignment. Before that, we first generate the 3ID object agent queries corresponding to the point cloud features. Similarly, we introduce multiple sets of learnable queries, each corresponding to a specific layer of the 3D backbone. These queries are projected into query vectors, while the associated 3D features are transformed into key and value vectors. A cross-attention mechanism is then employed to facilitate interaction between the learnable queries and 3D features, yielding layer-specific 3D agent queries. To generate the final 3D object agent query, we aggregate the outputs from all layers using both max and average pooling, followed by an MLP projection. For clarity, the previously introduced object agent queries derived from diffusion features are referred to as 2D object agent queries in the remainder of this paper. Next, we perform a dot product operation on the 2D object agent queries gins_pe with the 3D object agent queries y3D to obtain similarity map:
S = Softmax (gsD Linear (emearr(mms.e")) with S R
(10)
where Linear is a two-layer MLP with layer normalization. The dot product operation produces a similarity matrix S, explicitly linking instance semantic information between the 2D and 3D object agent queries. Subsequently, we leverage S to recompose the semantic information from the 2D object agent queries, resulting in new set of enhanced 3D object agent queries ÂSD infused with the object-wise semantic dependencies conveyed by the 2D queries:
ĀBD MLP(5 qina_pe + QaD); with ÿ&D E RR
(11)
Feature Refinement and Mask Prediction. The enhanced 3D object agent queries G3D act as supervisory signals to guide the original 3D agent queries in actively modeling inter-instance semantic dependencies during their construction process. Inspired by UniDSeg (55) and xMUDA [19), we
refine the segmentation features in the 3D decoder using ÎMD, while encouraging Q&D to mimic this refinement process:
f faD_ = -MP (Softmaaaa x () x QAD), föD = MLP(Softmax(550 x (qaD)") x x3D)
K
S5D_oag) ong), with (x = 00.((z =))²
if [ - yl < 1 otherwise
Sup
1 |/ yl 0.5
1=1
(12)

---

Table Performance comparison of domain adaptive and domain generalized 3D semantic segmenta-
tion methods in four typical settings. Top three results are highlighted as best, second and third
respectively. xM denotes the result which is obtained by taking the mean of the predicted 2D and 3D
probabilities after softmax.
S:Source Target
VKITTI/KITTI
nuScenes: ASScenes:USA/Sing USA/Sing
nuScenes Day/Night
A2D2/sKITT
Task
Method
3D
xM
3D
xM
3D
xM
3D
xM
logCORAL
36.8
47.0
63.2
68.7
63.7
41.0
42.2
MinEnt [50]
43.3
47.1
61.5
66.0
68.8
63.6
39.6
42,6
BDL [29]
44.3
35.6
64.8
70.4
69.6
63.0
41.7
45.2
XMUDA 19
46.7
48.2
63.2
69.4
69.2
67.4
46.0
44.0
AUDA [31
37.8
41.3
64.0
69.2
69.8
64.8
43.6
46.8
DsCML [37
38.4
45.5
56.2
66.1
49.3
53.2
45.1
44.5
27
35.1
44.2
58.1
66.5
69.7
68.0
40.0
48.6
SSE 67
40.0
49.6
63.9
69.2
69.0
68.9
46.8
48.4
BFI 54
45.5
51.5
62.2
69.4
70.4
68.3
44.4
48.7
DA
MM2D3D 5
50.3
56.5
66.8
72.4
70.2
72.1
46.1
46.2
VFMSeg 61
52.0
61.0
65.6
72.3
70.5
66.5
52.3
50.0
UniDSeg 55
50.9
62.0
67.6
72.9
71.2
71.2
57.5
XDiff3
53.1
63.3
69.5
73.5
72.3
57.6
58.8
xMUDA 19
37.4
39.0
62.3
68.6
68.9
59.6
36.7
41.6
DG
MM2D3D 5
40.2
44.2
62.3
70.9
63.2
68.3
35.9
43.6
UniDSeg [55
44.7
60.0
64.5
72.3
70.5
70.0
46.3
54.4
XDiff3D
46.9
61.3
66.7
73.5
72.4
71.6
49.1
56.2
where fäp denotes s-th stage features of the 3D decoder and S S. After the final stage, S, the refined segmentation features fāD are fed into a convolution to generate the final segmentation predictions:
Gug Lee (Conv (Conv(fāD),
(13)
where Y denotes the point cloud labels and Lee denotes the cross-entropy loss.
Full Objective. Ultimately, the overall objective of the training process is defined as follows:
Liotal Lug YLsup + A.Consist
(14)
where y and are hyperparameters. Notably, all operations involving 2D object agent queries and diffusion models are discarded during inference ensure framework remains concise and general.
Experiments
4.1 Datasets and Metrics
Following prior works 19 58 55 68,, we evaluate our method on six publicly available autonomous
driving datasets, comprising three real-world datasets (nuScenes [13] SemanticKITTI B] and A2D2
[5]). two synthetic datasets (VirtualKITTI [14] and SynLiDAR (57). and one Severse-weathhe dataset (SemanticSTF (58).. The real-world datasets provide synchronized and calibrated LiDAR and RGB sensor data, enabling direct 2D-to-3D projection, whereas VirtualKITTI includes depth maps from which simulate LiDAR scans by uniformly sampling points. For more details about
the datasets, please refer to the supplementary material. Following standard practice [58] [55 68) we
evaluate segmentation performance using the mean Intersection Union (mloU) averaged over all
classes for each dataset.
4.2 Implementation Details
Following prior works [55] 19), we adopt SparseConvNet [66 with a U-Net-style architecture as our 3D segmentation model, implemented using the Sparse Convolution Library [[]. The voxel resolution is set to 5cm, ensuring that each voxel encapsulates a single 3D point and provides sufficient spatial granularity for semantic segmentation. For the diffusion model, we leverage Stable Diffusion v [11, pretrained on the LAION-5B dataset 45), which is kept frozen during the entire training process. The model is optimized using AdamW
33 with a learning rate of le-5 for the 3D backbone

---

Table Comparison of previous domain generalization methods on SemantcK and SynLiDAR SSeRRR-SSmmmTeTTT benchmarks.
Dense-fog
Snow SemanticK[TTI SSemanticSTF
Light-fog
Rain
Dense-fog
Light-fog
Rain
Snow
Method
SynLiDAR *SemanticSTF
Dropout 47
29.3
25.6
29.4
24.8
13.3
16.6
20.4
14.0
Perturbation
58
26.3
27.8
30.0
24.5
16.3
16.7
19.3
13.4
PolarMix
56
29.7
25.0
28.6
25.6
16.1
15.5
19.2
15.6
MMD
26
30.4
28.1
32.8
25.2
17.3
16.3
20.0
12.7
PCL
28.9
27.6
30.1
24.6
17.8
16.7
19.3
14.1
PointDR
58
31.3
29.7
31.9
26.2
19.5
19.9
21.1
16.9
UniMix
68
34.8
30.2
34.9
30.9
24.3
22.9
26.1
20.9
XDiff3D
37.5
34.1
38.1
33.7
26.2
24.6
27.9
22.5
car
truck
bike
person
road
sidowalk
parking
object
building
vegctation
Ignore
UniDSeg
Ours
Ground Truth
Figure 3: Qualitative results of DG3SS. From left right: the visual results predicted by UniDSeg.
Ours, and Ground Truth. We deploy the white dash boxes to highlight different prediction parts.
and le-4 for the decoder and learnable queries. Training is conducted for 50000 iterations with batch size of 8. All experiments are performed on 4 NVIDIA RTX 4090 GPUs.
4.3 Comparison with State-of-the-art Methods
We comprehensively compare method with existing domain adaptive 3D semantic segmentation (DA3SS) and domain generalized 3D semantic segmentation (DG3SS) methods. We conduct ex- periments in three standard evaluation settings: synthetic-to-e -t-real TrrtualKKrTI--SSSaneiiIIIIII (Senn--miimmtt,TTTT) real-to-real (nuScenes: USA-> muScenes:Day-NNiihe and SDDD2=-SSmmatiK/ITT) and normal- to-adverse (SemanticKITT KITTII+Semantii architecture, we adopt the same structure as UniDSeg the exact configurations and training protocols of UniDSeg ensure consistency and fair comparison.
and -iiDAR-SSmantiiTTT: For the 2D branch
(55). For DA3SS experiments, we also follow
Synthetie-to-real generalization. In Table we compare our method with existing DG3SS and DA3SS approaches under the VirtualKITTI SrruukKKTTT-Smmmtteelll SemanticKITTI setting. Our method significantly outperforms the previous state-of-the-an method UniDSee in both 3D-only and 2D-3D fusion (i.e. xM) settings. Notably, in the 3D-only configuration, method surpasses UniDSeg by 2.2 points in mloU, demonstrating its strong cross- cross-domain generalization capability.
Real-to-real generalization. In this experimental setting, models are evaluated under the real-to-real benchmarks, including nuScenes:US: USA USA--inn -Sing, Day
and 22D2-+Smmnntick As illustrated from the fifth column onward in Table our method consistently achieves superior performance across all datasets in both DG and DA scenarios, surpassing previous statc-of-the art methods by clear margins. These results highlight the robustness and strong generalization capability of our framework in effectively handling complex and realistic domain variations encountered in practical 3D semantic segmentation tasks. Figure further provides qualitative comparisons the A2202-ASemanticKITT benchmark, where our approach yields more complete and coherent predictions for road, sidewalk, building, and vegetation regions, effectively resolving ambiguous boundaries and reducing unreasonable predictions observed in UniDSeg. More qualitative results and detailed analyses are included in the supplementary material.
Normal-to-adverse generalization. In this experimental setting, all models are evaluated on the SemanticSTF dataset, which includes range of challenging weather conditions. As shown in Table

---

Table 3: Ablation study on primary components, where 3D AQ denotes 3D object agent queries.
JuScenes:USA//SI muScenes:Day/Ni
A2D2/sKITT
3D_AQ
Lup
Ceansist
3D
xM
3D
XM
3D
xM
64.5
72.3
70.5
70.0
46.3
54.4
2
64.9
72.5
70.7
70.1
46.8
54.7
3
66.0
73.1
71.6
71.2
48.0
55.6
4
66.7
73.5
72.4
71.6
49.1
56.2
Table Ablation study different timestep se- lections under the DG3SS setting.
Table 5:
Comparison of diffusion models under DG3SS on ViruaaKiTTI--SSmaniiiIII
ASScenes:USA/Singg
A2D2/sKITT
Model
UniDSeg
+SD 1.4 +SD
1.5
+SD
t
3D)
xM
3D
xM
mloU
44.7
46.1
46.6
46.9
0
50
66.1
72.9
48.4
55.6
Table 6: Ablation principal components under
50
75
66.5
73.4
48.6
56.0
DG3SS on VirtualKITTI SSemanticKTTTT
50
150
66.7
73.5
49.1
56.2
150
300
66.3
73.1
48.3
55.6
70
90
300
400
65.7
72.7
47.8
55.3
mloU 45.7
46.
46.9
46.7
2. our method consistently outperforms existing approaches across all adverse scenarios. Notably, under the SrmnantccKITI-SSmmnniic setting, our approach achieves significant improvement over the previous state-of-the-art, exceeding it by more than 2 mloU points on average. These results demonstrate the strong robustness and generalization ability of our method the face of severe domain shifts.
Comparison of the class-wise IoU. We present a class-wise IoU analysis in Figure using UniDSeg as the baseline model. The results reveal consistent improvements across a wide range of categories. The heatmap further illustrates the robustness of our method in capturing meaningful semantic structures and maintaining performance across domain shifts.
4.4 Ablation Study
This section presents comprehensive experimental results to verify the effectiveness of the proposed method. For more ablation studies and detailed analyses, please refer to the supplementary material.
Effect of components. We conduct ablation studies under DG3SS settings validate the effectiveness of key components in our proposed method, specifically examining the contributions of the 3D object agent queries, Luup and Lconsist- Using UniDSeg as baseline model, the results presented in Table 3 reveal that: (1) Each component individually enhances performance, confirming their respective effectiveness. (2) Integrating all three components yields the highest mloU scores across all benchmarks, underscoring their complementary roles in improving cross-domain segmentation generalization.
Study the object agent query dimension c. The object agent queries serve as the core component of our framework. To assess the impact of their feature dimensionality, experiment with values ranging from 64 to 1024. As shown in Figure []] setting = 256 yields competitive performance, achieving mloU scores of 66. 66.7% and 72.4% the nuScenes: SaateesssSSA//inn. and DayNenes benchmarks, respectively.
The choice of and I'. Different timesteps in the diffusion process correspond to different denoising stages, with larger timesteps introducing stronger noise. Prior studies [60] 2 indicate that effective timesteps typically fall within the range of to 300, and that adjacent timesteps often yield highly similar features. To ensure sufficient diversity while maintaining semantic integrity, we select representative intervals from this range. As shown in Table 4. we empirically choose & = 50 and = 150 to effectively capture high-level semantic information while minimizing interference from low-level visual details.
Comparing different stable diffusion. As illustrated Table we adopt UniDSeg as the baseline and compare the performance of our method using three representative stable diffusion models. The results demonstrate that our method robust to the choice of diffusion model, consistently outperforming the baseline and achieving significant performance gains.

---

Table 7: Effect of using different 3D back- bones on the DA3SS methods.
67.0
66.7
66.6
66.5
3D Backbone
DA3SS
USA/Sing
66.
3D
xM
66.0
Spar
Uni
67.6
72.9
mlou (%)
65.7
nuScenes PibeenesUUSASSS USA/
XDiff3D
69.5
74.1
65.5
MinkowskiNet
UniDSeg
68.6
73.1
72.7
XDiII3D
70.2
74.3
72.
Base 60.66
53.37
82.24
4.64
24.53
80.13
72.2
72.0
61.18
58.04
83.26
5.09
30.02
81.64
71.7
71.5
Veget.
Build.
Road Object
Truck
Car
nuSeenes: maSeenes.Day/Nigiee Day//Nght
71.2
Figure 4: Comparison of the class-wise IoU on VirtualKITTI + SemanticKITTI under the DA3SS setting, using UniDSeg with and without our method.
64
128
256
512
1024
Dimension
Figure 5: Ablation study on agent queries dimension c.
The number of principal components k. The results in Tablee]indicate that the selection of k-the number retained principal components-Signfficannn
influences segmentation performance, with k = 70 achieving the highest mloU. Increasing & beyond this value introduces fine- fine-grained visual details, which interferes with the learning of Nomain-invariin semantics and leads to performance degradation.
evaluate the performance of various backbones integrated into our framework under consistent parameter settings. The results indicate that stronger backbones yield improved performance, and our method consistently outperforms the baseline across all configurations.
Ablation on different backbones. As shown in Table 7
Table 8: Performance comparison under different noise levels.
a (m)
0.05
0.1
0.2
0.3
UniDSeg (baseline)
40.3
37.1
33.8
30.6
Ours
44.7
43.2
39.6
34.7
Robustness noisy point clouds. To evaluate robustness against measurement noise, we conducted an ablation study on A2D2 SemanticKITTI benchmarks by adding zero-mean Gaussian noise with varying standard deviations a. As shown in Table our method consistently outperforms the UniDSeg baseline across all noise levels, maintaining higher mloU even under severe perturbations. This demonstrates that our model possesses strong robustness and effectively preserves semantic consistency under noisy conditions.
5 Conclusion
In this work, we propose XDiff3D, novel cross-modal framework guided by diffusion models to enhance the generalization of 3D semantic segmentation. XDiff3D constructs object agent queries to capture semantic dependencies among objects from diffusion features and infuses them into the 3D representation space. To mitigate interference from visual details, we further propose a dual-queries consistency scheme that encourages object-agent queries focus on domain-invariant semantics. Extensive experiments on DG3SS and DA3SS benchmarks demonstrate that XDiff3D significantly outperforms previous SOTA methods, underscoring its effectiveness. This work addresses a significant gap in existing research and sets benchmark for cross-domain 3D semantic segmentation.
Acknowledgments and Disclosure of Funding
This work was supported by the National Natural Science Foundation of China under Grant No.42504030 and No.52302506; the Fundamental Research Funds for the Central Universities (Science and Technology Program) under Grant D50002500047 and the Shaanxi Key Research and Development Program under Grant
10

---

References
[1] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. In Proceedings of the IEEE/CVE conference on computer vision and patten recognition, pages 1090~1099, 2022.
[2] Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic segmentation with difffusion models. In International Conference Learning Representations, 2022. URL httpps://openreeiirr..
[3] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall. Semantickitti: A dataset for semantic scene understanding of lidar sequences. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9297-9307, 2019.
[4] Yasser Benigmim, Subhankar Roy, Slim Essid, Vicky Kalogeiton, and Stéphane Lathuilière. Collaborating foundation models for domain generalized semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3108-3119,
2024.
[5] Adriano Cardace, Pierluigi Zama Ramirez, Samuele Salti, and Luigi Di Stefano. Exploiting the complementarity of 2d and 3d networks to address domain-shift in 3d semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 98-109, 2023.
[6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 21 213-229. Springer, 2020.
[7] Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19830-19843, 2023.
[8] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances neural information processing systems, 34: 77864-17875. 2021.
[9] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1290-1299, 2022.
[10] Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets:
Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 3075-3084, 2019.
[11] Spconv Contributors. Spconv: Spatially sparse convolution library. SpConv: Spatially sparse convolution library, 2022.
[12] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[13] Whye Kit Fong, Rohit Mohan, Juana Valeria Hurtado, Lubing Zhou, Holger Caesar, Oscar Beijbom, and Abhinav Valada. Panoptic nuscenes: A large-scale benchmark for lidar panoptic segmentation and tracking. IEEE Robotics and Automation Letters, 7()):395533822 2022.
[14] Adrien Gaidon, Qiao Wang, Yohann Cabon, and Elconora Vig. Virtual worlds as proxy for multi-object tracking analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4340-4349, 2016.
[15] Jakob Geyer, Yohannes Kassahun, Mentar Mahmudi, Xavier Ricou, Rupesh Durgesh, Andrew S Chung, Lorenz Hauswald, Viet Hoang Pham, Maximilian Mühlegg, Sebastian Dorn, et A2d2: Audi autonomous driving dataset. arXiv preprint arXiv:2004.0633 2020.
11

---

[16] Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9224-9232, 2018
[17] Meng-Hau Guo, Jun Lun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pct: Point cloud transformer. Computational visual media, 7:187-199, 2021.
[18] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:684006851, 2020.
[19] Maximilian Jaritz, Tuan-Hung Vu, Raoul De Charette, Émilie Wirbel, and Patrick Pérez. Cross- modal learning for domain adaptation in 3d semantic segmentation. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(2): 2022.
[20] Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, and Ping Luo. Ddp: Diffusion model for dense visual prediction. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21741-21752, 2023.
[21] Yuru Jia, Lukas Hoyer, Shengyu Huang, Tianfu Wang, Luc Van Gool, Konrad Schindler, and Anton Obukhov. Dginstyle: Nomain-generalizahe semantic segmentation with image diffusion models and stylized semantic control. In Synthetic Data for Computer Vision Workshop@ CVPR 2024, 2023.
[22] Yuru Jía, Lukas Hoyer, Shengyu Huang, Tianfu Wang, Luc Van Gool, Konrad Schindler, and Anton Obukhov. Dginstyle: Domain-gent genaiingenerallzzblle semantic segmentation with image diffusion models and stylized semantic control. European Conference on Computer Vision, pages 91-109. Springer, 2024.
[23] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Kon- rad Schindler. Repurposing diffusion-bar based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
pages
[24] Hyeonseong Kim, Yoonsu Kang, Changgyoon Oh, and Kuk-Jin Yoon. Single domain gen- eralization for lidar semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17587-17599 17598, 2023.
[25] Lingdong Kong, Niamul Quader, and Venice Erin Liong. Conda: Unsupervised domain adapta- tion for lidar segmentation via regularized domain concatenation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9338-9345. IEEE, 2023.
[26] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5400-5409, 2018
[27] Miaoyu Li, Yachao Zhang, Yuan Xie, Zuodong Gao, Cuihua Li, Zhizhong Zhang, and Yanyun Qu. Cross-domain and cross-modal knowledge distillation in domain adaptation for 3d semantic segmentation. In Proceedings of the 30th ACM International Conference on Multimedia, pages 3829-3837, 2022.
[28] Miaoyu Li, Yachao Zhang, Xu Ma, Yanyun Qu, and Yun Fu, Bev-dg: Cross-modal learning under bird's-eye view for domain generalization of 3d semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11632-11643 2023.
[29] Yunsheng Li, Lu Yuan, and Nuno Vasconcelos. Bidirectional learning for domain adaptation of semantic segmentation. Proceedings of the IEEE/CVF conference on computer vision and pattem recognition, pages
2019.
[30] Zhiqi Li, Wenhai Wang, Enze Xie, Zhiding Yu, Anima Anandkumar, Jose Alvarez, Ping Luo, and Tong Lu. Panoptic segformer: Delving deeper into panoptic segmentation with transformers. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 1280- 1089, 2022.
12

---

[31] Wei Liu, Zhiming Luo, Yuanzheng Cai, Ying Yu, Yang Ke, José Marcato Junior, Wesley Nunes Gonçalves, and Jonathan Li. Adversarial unsupervised domain adaptation for 3d semantic segmentation with multi-modal learning. ISPRS Journal of Photogrammeiry and Remote Sensing, 176:211-22 1-221, 2021.
[32] Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han. Point-voxel enn for efficient 3d deep learning. Advances in neural information processing systems, 32, 2019.
[33] Iyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 2017.
[34] Pietro Morerio, Jacopo Cavazza, and Vittorio Murino. Minimal-entroop correlation alignment for unsupervised deep domain adaptation. International Conference on Learning Representa- tions, 2018. URL
[35] Joshua Niemeijer, Manuel Schwonberg, Jan-Aike Termöhlen, Nico M Schmidt, and Tim Fingscheidt. Generalization by adaptation: Diffusion-basee domain extension for domain- generalized semantic segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2830-2840, 2024.
[36] Byeonghyun Pak, Byeongju Woo, Sunghwan Kim, Dae-hwan Kim, and Hoscong Kim. Textual query-driven mask transformer for domain generalized segmentation. In European Conference on Computer Vision, pages 37-54. Springer, 2024.
[37] Duo Peng, Yinjie Lei, Wen Li, Pingping Zhang, and Yulan Guo. Sparse-to-dense feature matching: Intra and inter domain cross-modal learning in domain adaptation for 3d semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7108-7117, 2021.
[38] Duo Peng, Ping Hu, Qiuhong Ke, and Jun Liu. Diffusion-based image translation with la- bel guidance for domain adaptive semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 808-820, 2023.
[39] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[40] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
[41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[42] Kwonyoung Ryu, Soonmin Hwang, and Jaesik Park. Instant domain augmentation for lidar semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9350-9360, 2023.
[43] Cristiano Saltori, Fabio Galasso, Giuseppe Fiameni, Nicu Sebe, Elisa Ricci, and Fabio Poiesi. Cosmix: Compositional semantic mix for domain adaptation in 3d lidar segmentation. In European Conference on Computer Vision, pages 586-666666 602. Springer, 2022.
[44] Jules Sanchez, Jean- -ammnell Deschaud, and François Goulette. Domain generalization of 3d semantic segmentation in autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18077-18087, 2023.
[45] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 15:25278-25294 2022.
[46] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint 2rX1v:2010.02502,, 2020.
13

---

[47] Nitish Srivastava, Geoffre Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, (5(1):1929-1958 2014.
[48] Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6411-6420, 2019.
[49] Junjiao Tian, Lavisha Aggarwal, Andrea Colaco, Zsolt Kira, and Mar Gonzalez-Franco. Diffuse attend and segment: Unsupervised zero-shot segmentation using stable diffusion. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3554-3563, 2024.
[50] Tuan- Maan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. Advent: Ad- versarial entropy minimization for domain adaptation in semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2517-2526, 2019.
[51] Zhixiang Wei, Lin Chen, Yi Jin, Xiaoxiao Ma. Tianle Liu, Pengyang Ling, Ben Wang. Huaian Chen, and Jinjin Zheng. Stronger fewer & superior: Harnessing vision foundation models for domain generalized semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16619, 2024.
[52] Weijia Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu, Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, and Chunhua Shen. Datasetdm: Synthesizing data with perception annotations using diffusion models. Advances in Neural Information Processing Systems, 36:54683-54655 2023.
[53] Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, and Chunhua Shen. Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1206-1221 1216-121,,000 2023.
[54] Yao Wu, Mingwei Xing, Yachao Zhang, Yuan Xie, Jianping Fan, Zhongehao Shi, and Yanyun Qu. Cross-modal unsupervised domain adaptation for 3d semantic segmentation via bidirec- tional fusion-then-ddisilllaioo In Proceedings of the 31st ACM International Conference on Multimedia, pages 490-498 490.498.22 2023.
[55] Yao Wu, Mingwei Xing, Yachao Zhang, Xiaotong Luo, Yuan Xie, and Yanyun Qu Unidseg: Unified cross cross-domain 3d semantic segmentation via visual foundation models prior. Advances in Neural Information Processing Systems, 37:101223-101249) 2024.
[56] Aoran Xiao, Jiaxing Huang, Dayan Guan, Kaiwen Cui, Shijian Lu, and Ling Shao. Polarmix: general data augmentation technique for lidar point clouds. Advances in Neural Information Processing Systems, 35:1 1035 11048, 2022.
[57] Aoran Xiao, Jiaxing Huang, Dayan Guan, Fangneng Zhan, and Shijian Lu. Transfer learning from synthetic to real lidar point cloud for semantic segmentation. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 2795-2803, 2022.
[58] Aoran Xiao, Jiaxing Huang, Weihao Xuan, Ruijie Ren, Kangeheng Liu. Dayan Guan, Abdulmo- taleb El Saddik, Shijian Lu, and Eric P Xing. 3d semantic segmentation in the wild: Learning generalized models for adverse-conditio
point clouds. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 9382-9392, 2023.
[59] Bowei Xing, Xianghua Ying, Ruibin Wang. Jinfa Yang, and Taiyan Chen. Cross-modal contrastive learning for domain adaptation in 3d semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 2974-2982, 2023.
[60] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulayy panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 2955 2966, 2023.
14

---

[61] Jingyi Xu, Weidong Yang, Lingdong Kong, Youquan Liu, Rui Zhang, Qingyuan Zhou, and Ben Fei. Visual foundation models boost cross- modsl unsupervised domain adaptation for 3d semantic segmentation. arXiv preprint arXiv:2403. 10001, 2024.
[62] Xu Yan, Jiantao Gao, Chaoda Zheng, Chao Zheng, Ruimao Zhang, Shuguang Cui, and Zhen Li. 2dpass: 2d priors assisted semantic segmentation on lidar point clouds. European conference on computer vision, pages 677-695. Springer, 2022.
[63] Xufeng Yao, Yang Bai, Xinyun Zhang, Yuechen Zhang, Qi Sun, Ran Chen, Ruiyu Li, and Bei Yu. Pcl: Proxy-based contrastive learning for domain generalization. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 7097-7107, 2022.
[64] Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan
Yuille, and Liang-Chieh Chen. k-means mask transformer. In European Conference on Computer Vision, pages 288-301 307. Springer, 2022.
[65] Seokju Yun, Seunghye Chae, Dongheon Lee, and Youngmin Ro. Soma: Singular value decomposed minor components adaptation for domain generalizable representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR),
2025.
[66] Hao Zhang, Feng Li, Huaizhe Xu, Shijia Huang, Shilong Liu, Lionel M Ni, and Lei Zhang. Mp-former: Mask-piloted transformer for image segmentation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 18074- 18083, 2023.
[67] Yachao Zhang, Miaoyu Li. Yuan Xie, Cuihua Li, Cong Wang. Zhizhong Zhang, and Yanyun Qu Self-supervised exclusive learning for 3d segmentation with cross-modal unsupervised domain adaptation. Proceedings of the 30th ACM International Conference on Multimedia, pages 3338 3346, 2022.
[68] Haimei Zhao, Jing Zhang, Zhuo Chen, Shanshan Zhao, and Dacheng Tao. Unimix: Towards domain adaptive and generalizable lidar semantic segmentation adverse weather. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14781-14791, 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim our contribution, that is, cross-modal learning framework to enhance the generalizability of cross-domain 3D semantic segmentation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitation of the the work is detailed discussed in the Appendix A.4
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
This paper does not include theoretical results with proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification;
All the necessary information needed to reproduce the main experimental results are fully stated in the Section and Appendix A. This information ensures the understanding the results which support the main claims and conclusions of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All results are evaluated on the public 3D datasets. We will make the code publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Cotee//ddeeeSbmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Section 4.2, we illustrate the training details, including networks, learning rates, batch size, etc.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[NA]
Justification:
Our experimental results do not contain statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Section 4.2, we state that all experiments are conducted on four NVIDIA RTX 4090 GPUs, each equipped with 24GB of RAM.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:///wwwrps..ccpueeeeeeeeeeae
Answer: [Yes]
Justification:
We have reviewed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper has no socictal impact of the work performed. No AI ethics are involved, and no private data is involved.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators original owners of assets are properly credited and the license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: This paper does not involve LLMs as any important, original, non- non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy huttep//hearipssccc//offrrreeeee for what should or should not be described.
22


Reviews:
[{"review_id": "S3UnBdPPq5", "reviewer": "Reviewer_gtR6", "summary": "This paper focuses on enhancing the cross-domain generalization of 3D semantic segmentation task. They propose a diffusion model based cross-modal learning framework named XDiff3D. XDiff3D proposes to leverage diffusion model priors for 3D semantic segmentation. It mainly includes three components: (1) Contruction of Object Agent Queries, (2) Dual-Queries Consistency Scheme and (3) Cross-Modal Semantic Enhancement.", "questions": "(1) Why aggregate the semantic information of objects using diffusion features to form the object agent queries? What are the advantages of using diffusion modes in module (a)?\n(2) What's the effect of SVD in module (b)? Is SVD a standard method for feature quantization?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths \n(1) XDiff3D propose to inject 2D diffusion priors into 3D representations via object agent queries to enhance cross-domain generalization.\n(2) XDiff3D  achieves state-of-the-art results, and extensive experiments also show the effectiveness of the proposed method.\n\nWeaknesses\n(1) The framework in Fig.2 is complex, which makes it hard to follow. It's vital to show the core idea of the proposed method.", "was_revised": true, "final_justification": "Thanks to the authors for comprehensive responses.", "rebuttal": "**Q1**: Why aggregate the semantic information of objects using diffusion features to form the object agent queries? What are the advantages of using diffusion models in module (a)? \n\n  **A**: Thank you for the question. The motivation for aggregating semantic information from diffusion features to form object agent queries is discussed in lines 57–64 of the Introduction. In brief, the object agent queries act as an interface to inject scene-level priors from diffusion features into the segmentation representation space, while simultaneously suppressing low-level visual details that may introduce noise. This mechanism allows the model to exploit the rich scene semantic distribution encoded in diffusion features without being distracted by irrelevant textures, thereby enhancing generalization across domains.\n\n  The advantages of using diffusion models in module (a) are discussed in lines 38–45 of the Introduction. Briefly, diffusion models provide strong scene-level priors by capturing semantic relationships among objects. Leveraging these priors helps the model better learn semantic dependencies and spatial arrangements, ultimately improving generalization in 3D semantic segmentation.\n\n**Q2**: What's the effect of SVD in module (b)? Is SVD a standard method for feature quantization?\n\n  **A**: Thank you for the question. The effect of SVD in module (b) is explained in Section 3.4, lines 184–189. Inspired by SoMA [1], which shows that principal components derived from SVD in vision foundation models encode generalized semantic knowledge essential for strong generalization, we apply SVD to the object agent queries to emphasize domain-invariant semantics while suppressing low-level visual details, thereby enhancing cross-domain generalization.\n\n  While SVD is not a standard method for feature quantization, it is widely adopted for feature decomposition and compression [1,2,3]. By decomposing the feature space into high-rank and low-rank components,  SVD enables structured compression, mitigates overfitting, and strengthens the model’s capacity to extract robust and semantically meaningful representations.\n\n  [1] Yun, Seokju, et al. \"SoMA: Singular Value Decomposed Minor Components Adaptation for Domain Generalizable Representation Learning.\" *Proceedings of the Computer Vision and Pattern Recognition Conference*. 2025.\n\n  [2] Sun, Yanpeng, et al. \"Singular value fine-tuning: Few-shot segmentation requires few-parameters fine-tuning.\" *Advances in neural information processing systems* 35 (2022): 37484-37496.\n\n  [3] Wang, Yuan, Rui Sun, and Tianzhu Zhang. \"Rethinking the correlation in few-shot segmentation: A buoys view.\" *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*. 2023.\n\n**Q3**: The framework in Fig.2 is complex, which makes it hard to follow.\n\n  **A**: Thank you for the helpful feedback. We will revise Figure 2 in the final version by adding sub-figures to clarify key components and highlight the core idea, making the framework more intuitive."}, {"review_id": "WYebKlXyZb", "reviewer": "Reviewer_Qzhy", "summary": "This paper presents XDiff3D, a novel cross-modal framework leveraging diffusion models to improve the cross-domain generalization of 3D semantic segmentation. By constructing object agent queries from diffusion features, decoupling irrelevant local details, and enhancing 3D representations via cross-modal semantic alignment, the method captures object-level semantic dependencies more effectively. Extensive experiments demonstrate state-of-the-art performance across diverse domain adaptation benchmarks.", "questions": "1. How does the method compare to other approaches in terms of training and inference time, given the computational demands of diffusion models?\n\n2. Is it possible to use more informative textual conditions (e.g., class names or semantic prompts) instead of empty text to better exploit the semantic potential of diffusion models?\n\n3. How robust is the method to noisy or incomplete point cloud data, especially in target domains where LiDAR quality may vary?\n\n4. Given that the method relies on paired RGB images and point clouds, how sensitive is it to calibration or alignment errors between the two modalities?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The writing is clear and well-organized.\n2. The method is novel and builds a compelling cross-modal framework using diffusion models.\n3. Experiments demonstrate strong performance with thorough ablation studies.\n\nWeaknesses:\n1. The paper lacks a comparison of training and inference time with other methods, which is particularly important given the potential computational overhead introduced by diffusion models.\n2. The method depends on access to paired RGB images and point clouds in the target domain, which may hinder its practicality in real-world applications where obtaining fully aligned multi-modal data is often challenging.\n3. The method uses empty text as the textual condition, which may not fully activate the semantic potential of the diffusion model. In contrast, incorporating class names or prior semantic labels could yield more precise and informative semantic features.", "was_revised": true, "final_justification": "The authors have addressed all of my concerns with detailed clarifications and additional experiments. Their responses demonstrate the method's practicality, robustness, and efficiency. I am satisfied with the revisions and support acceptance of the paper.", "rebuttal": "**Q1**: How does the method compare to other approaches in training and inference time, given the computational overhead of diffusion models?\n\n  **A**: Thank you for your valuable comment. We have added a comparison of both training and inference time with other methods. As shown in the table below, our method introduces a slight increase in training time due to the incorporation of diffusion models. However, since all diffusion-related computations are performed entirely offline, this overhead can be further reduced through practical engineering optimizations. Notably, the diffusion model is not involved during inference, and thus our method introduces no additional computational cost at test time compared to previous approaches.\n\n  | Method         | xMUDA | UniDSeg (baseline) | Ours |\n  | -------------- | ----- | ------------------ | ---- |\n  | Training time  | 20h   | 26h                | 28h  |\n  | Inference time | 16ms  | 19ms               | 19ms |\n\n**Q2**: The method relies on paired RGB images and point clouds in the target domain, which limits its practicality in real-world applications.\n\n  **A**: Thank you for your insightful comment. To clarify, our method does not require paired RGB images and point clouds in the target domain. In both unsupervised domain adaptation (UDA) and domain generalization (DG) settings, training is performed using paired RGB–point cloud data only in the source domain. This design improves the practicality of our approach, and we will highlight this point more clearly in the final version.\n\n**Q3**: Is it possible to use more informative textual conditions (e.g., class names or semantic prompts) instead of empty text to better exploit the semantic potential of diffusion models?\n\n  **A**: Thank you for the insightful suggestion. We experimented with simple prompts using class names but did not observe noticeable performance improvements. Notably, despite using empty prompts, our method already achieves competitive results compared to previous approaches, indicating that the diffusion model can still provide meaningful scene-level priors without explicit textual input. Incorporating more complex prompts would require more powerful multimodal models and significantly increase computational cost. The current design strikes a well-balanced trade-off between effectiveness and efficiency. Additionally, overly detailed prompts may introduce task-irrelevant information, potentially disrupting the extraction of scene-level semantic priors and degrading 3D segmentation performance.\n\n**Q4**: How robust is the method to noisy or incomplete point cloud data, especially in target domains where LiDAR quality may vary?\n\n  **A**: Thank you for the insightful question. To evaluate robustness to noisy point clouds, we conducted an ablation study on A2D2$\\to$SemanticKITTI benchmark by injecting zero-mean Gaussian noise into the input point clouds, with the noise level controlled by the standard deviation $\\sigma$. As shown in the table below, our method consistently outperforms the baseline across varying noise levels, demonstrating strong robustness.\n\n  In addition, we assessed the robustness of our method to noisy and incomplete point clouds via the Normal-to-Adverse generalization experiment (Table 2 of the paper). Prior studies [1, 2, 3] have shown that adverse weather degrades LiDAR signals by introducing spatial noise, reducing point density, and distorting reflection intensity, leading to noisy and incomplete point clouds that significantly challenge perception systems.\n\n  Despite these effects, our method consistently outperforms previous approaches under adverse conditions (Table 2 of the paper). We attribute this to the incorporation of scene-level semantic priors from the diffusion model, which enable the model to capture contextual relationships and inter-object semantic dependencies, effectively mitigating the impact of noisy or incomplete geometry. This is further supported by Figures 1 and 2 of the supplementary material, which demonstrate the robustness of our method to incomplete point clouds resulting from occlusions or limited viewpoints (e.g., Figure 2, row 8).\n\n  | $\\sigma$ (m)       | 0.05 | 0.1  | 0.2  | 0.3  |\n  | ------------------ | ---- | ---- | ---- | ---- |\n  | UniDSeg (baseline) | 40.3 | 37.1 | 33.8 | 30.6 |\n  | Ours               | 44.7 | 43.2 | 39.6 | 34.7 |\n\n  [1] Zhao, Haimei, et al. \"Unimix: Towards domain adaptive and generalizable lidar semantic segmentation in adverse weather.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.\n\n  [2] Xiao, Aoran, et al. \"3d semantic segmentation in the wild: Learning generalized models for adverse-condition point clouds.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2023.\n\n  [3] Hahner, Martin, et al. \"Fog simulation on real LiDAR point clouds for 3D object detection in adverse weather.\" *Proceedings of the IEEE/CVF international conference on computer vision*. 2021.\n\n**Q5**: Given that the method relies on paired RGB images and point clouds, how sensitive is it to calibration or alignment errors between the two modalities?\n\n  **A**: Thank you for the thoughtful question. Our method follows the commonly adopted standard setup in multi-modal semantic segmentation, which assumes well-calibrated and aligned RGB–point cloud pairs in the source domain. Accordingly, we do not account for calibration or alignment errors during training. Notably, in the target domain, our method does not require paired RGB images and point clouds, where only point clouds are needed. As a result, potential misalignment in the target domain does not affect the applicability or performance of our method."}, {"review_id": "gbCUfjbhoY", "reviewer": "Reviewer_nDRB", "summary": "This paper introduces XDiff3D, a novel cross-modal learning framework based on diffusion models to enhance the cross-domain generalization of 3D semantic segmentation. The authors leverage diffusion models to capture semantic dependencies among objects, constructing object agent queries to infuse 2D diffusion priors into the 3D representation space. A dual-queries consistency scheme is proposed to suppress visual details and focus on domain-invariant semantics. Experiments across multiple benchmarks demonstrate state-of-the-art performance, outperforming existing methods in various settings.", "questions": "1. Computational Efficiency: Since the diffusion model is applied, it is neccessraty to provide a direct comparison of computational resources between XDiff3D and other methods (e.g., training time, memory usage, FLOPs).\n2. Semantic Dependency Validation: The object agent queries claim to capture \"semantic dependencies\". Could attention maps or correlation metrics (e.g., inter-object affinity scores) be provided for the justification over other methods?\n3. Impact of different diffusion layers: The author introduced the dual-queries consistency scheme to suppress visual details. Have the authors considered not utilizing all diffusion layers from the start? It would be beneficial to conduct experiments exploring the selection of diffusion layers to determine their impact on the model's performance and the suppression of visual details.\n4. Failure case analysis in limitation: In the supplementary material, the authors outline certain limitations; however, the discussion primarily centers around input constraints. To provide a more comprehensive understanding, it is advisable to include specific failure cases encountered when applying the proposed methods, accompanied by in-depth analyses of the underlying causes. This approach would not only enhance transparency but also offer valuable insights for future improvements and applications.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n•\tOriginal Approach: The use of diffusion models to capture object-level semantic dependencies for 3D segmentation is innovative, addressing a gap in existing methods that treat objects in isolation.\n•\tComprehensive Evaluation: The paper evaluates XDiff3D across diverse benchmarks, including synthetic-to-real, real-to-real, and normal-to-adverse weather transitions, showcasing its robustness.\n•\tAblation Studies: Detailed ablation experiments validate the effectiveness of key components, such as object agent queries, dual-queries consistency, and loss functions.\n•\tClear Contributions: The framework is well-motivated, with a clear explanation of how diffusion priors address limitations in existing 3D segmentation approaches.\n\nWeaknesses\n•\tComputational Overhead: The use of diffusion models may introduce additional computational complexity.\n•\tDependency on Paired Data: The method relies on paired images and point clouds, which could limit applicability in scenarios where multi-modal data is scarce.\n•\tClaim Justification: The method asserts its capability to effectively capture \"semantic dependencies\", yet lacks both quantitative or qualitative evidence, such as attention maps, to substantiate this claim.", "was_revised": true, "final_justification": "I decided to keep my score because the proposed method demonstrates clear improvements over existing baselines, and most of my initial concerns have been addressed. While some limitations remain—particularly regarding generalization vs. semantic disentanglement—the authors have acknowledged these and committed to further visualizations and analysis in the final version.", "rebuttal": "**Q1**: Since the diffusion model is applied, it is necessary to provide a direct comparison of computational resources between XDiff3D and other methods (e.g., training time, memory usage, FLOPs).\n\n  **A**: Thank you for pointing this out. In our framework, the diffusion model is used only during training to provide semantic priors, resulting in a slight increase in training time and memory usage compared to the baseline, as shown in the table below. Notably, all diffusion-related processes are conducted entirely offline and can be further optimized through engineering techniques to reduce training overhead. In addition, the diffusion model is not involved during inference, so the overall computational complexity at test time remains unchanged.\n\n  | Method             | Training time | Memory usage | FLOPs |\n  | ------------------ | ------------- | ------------ | ----- |\n  | UniDSeg (baseline) | 26h           | 11.1G        | 190M  |\n  | Ours               | 28h           | 15.7G        | 190M  |\n\n**Q2**: The method relies on paired images and point clouds, which could limit applicability in scenarios where multi-modal data is scarce. \n\n  **A**: Thank you for your insightful comment. To clarify, our method does not require paired RGB images and point clouds in the target domain. In both unsupervised domain adaptation (UDA) and domain generalization (DG) settings, training is conducted using paired RGB–point cloud data only in the source domain. This design improves the practicality of our approach, as paired data is typically easier to obtain in the source domain (e.g., via simulation or controlled capture). We will highlight this point more clearly in the final version.\n\n**Q3**: The method claims to effectively capture \"semantic dependencies\", yet lacks quantitative or qualitative evidence to support this claim.\n\n  **A**: Thank you for the thoughtful comment. As illustrated in Figures 1 and 2 of the supplementary material (see lines 87–96 in Section A.5), our method shows clear improvements over the baseline (UniDSeg) in capturing semantic dependencies and spatial arrangements. Specifically, it exhibits more comprehensive spatial coverage of objects, contextually consistent category predictions, and effective suppression of fragmented outputs.\n\n  For instance, in Figure 1 (row 1), our method produces a more coherent segmentation of the \"road\" region, ensuring comprehensive spatial coverage. In Figure 1 (row 4), it avoids misclassifying \"vegetation\" as \"truck\", resulting in contextually consistent category predictions. Similarly, in Figure 2 (row 2), our method significantly reduces fragmented or implausible predictions for objects. To further demonstrate the model’s ability to capture semantic dependencies, we will include more comprehensive and systematic visual results, such as clustering-based visualizations of intermediate features, in the final version.\n\n**Q4**: It would be valuable to investigate whether selectively using diffusion layers, instead of all from the beginning, affects model performance and the suppression of visual details.\n\n  **A**: Thank you for the valuable suggestion. For fine-grained perception tasks such as 3D semantic segmentation, preserving low-level visual cues—especially at object boundaries—is crucial. Discarding shallower diffusion layers and relying only on high-level semantics from deeper layers can impair boundary localization and ultimately lead to suboptimal performance. To validate this, we conducted an ablation study using different subsets of diffusion layers. As shown in the table below, the model achieves optimal performance only when the full range of diffusion features—from both shallow and deep layers—is utilized. This underscores the importance of multi-level diffusion information, which integrates fine-grained edge details with global semantic context to enable accurate and robust segmentation.\n\n  | Layer                      | vKITTI/sKITTI | nuScenes:USA/Sing | nuScenes:Day/Night | A2D2/sKITT |\n  | -------------------------- | ------------- | ----------------- | ------------------ | ---------- |\n  | 12                         | 44.6          | 64.7              | 70.5               | 46.2       |\n  | 11,12                      | 44.6          | 65.0              | 70.7               | 46.4       |\n  | 10,11,12                   | 44.8          | 64.9              | 70.8               | 46.2       |\n  | 7,8,9,10,11,12             | 45.4          | 65.5              | 71.4               | 46.9       |\n  | 4,5,6,7,8,9,10,11,12       | 46.3          | 66.3              | 71.9               | 48.2       |\n  | 1,2,3,4,5,6,7,8,9,10,11,12 | 46.9          | 66.7              | 72.4               | 49.1       |\n\n**Q5**: While the supplementary material outlines input-related limitations, including concrete failure cases with analysis would offer deeper insight and support future enhancements.\n\n  **A**: Thank you for the valuable suggestion. In our experiments, we employed Stable Diffusion v2, which was trained on a relatively limited range of visual domains and semantic classes. This constraint made it easier to extract scene-level semantic priors. However, it also limits generalization when encountering out-of-distribution inputs, leading to potential failure cases in unfamiliar environments. While adopting more recent diffusion models—such as Flux or Stable Diffusion v3—could improve domain coverage, it comes with substantially higher computational cost. Moreover, these models are trained on extremely broad and diverse datasets, leading to highly entangled semantic spaces. As a result, it becomes considerably more challenging to identify suitable timesteps and to extract scene-level semantic priors, as these are often obscured by entangled low-level visual cues such as textures and stylized patterns."}, {"review_id": "ExvoyuvYt5", "reviewer": "Reviewer_4ZFB", "summary": "This paper studies 3D Semantic Segmentation Generalization enhanced by diffusion models. Specifically, it aims to harness the ability from diffusion models to capture underlying contextual distributions and spatial relationships among objects to tackle the challenging task of cross-domain 3D semantic segmentation. To achieve it, it designs  Dual-queries Consistency Scheme to extract features from diffusion model and construct Consistency Constraint between  queries from different timesteps. The experimental results show that the proposed method outperforms previous methods.", "questions": "- The authors claim that it uses diffusion models to address the dependency between objects. However, it does not provide evidence to support the motivation. It is recommended to provide detailed analysis (visualization and/or quantitative results).\n\n- The experimental results show that the diffusion model indeed bring improvements. But the improvements really result from  \"the diffusion model’s ability to flexibly compose diverse objects into high-quality images across varying domains\"? Please provide evidence. In addition, can we use other visual foundation models to achieve it?\n\n- It seems that the number of parameters of the proposed method is higher than previous methods due to the introduction of diffusion models. Thus, please provide the parameters for each method and provide a more fair comparison.", "limitations": "The authors note a limitation: 'the lack of source-domain data makes it reliant on paired images and point clouds in the target domain.' However, the reviewer is more interested in scenarios where the method underperforms and the reasons behind a limitation.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The method is novelty and effective.\n\n- The ablation can show the effectiveness of each components.\n\nWeakness:\n\n- The authors claim that it uses diffusion models to address the dependency between objects. However, it does not provide evidence to support the motivation. It is recommended to provide detailed analysis (visualization and/or quantitative results).\n\n- The experimental results show that the diffusion model indeed bring improvements. But the improvements really result from  \"the diffusion model’s ability to flexibly compose diverse objects into high-quality images across varying domains\"? Please provide evidence. In addition, can we use other visual foundation models to achieve it?\n\n- It seems that the number of parameters of the proposed method is higher than previous methods due to the introduction of diffusion models. Thus, please provide the parameters for each method and provide a more fair comparison.", "was_revised": true, "final_justification": "My concerns are addressed. After reading other reviews, I believe this paper could be accepted with slight modifications.", "rebuttal": "**Q1**: To substantiate the claim that diffusion models help capture inter-object dependencies, it is recommended to include detailed analysis through visualizations and/or quantitative results.\n\n  **A**: Thank you for the thoughtful comment. As illustrated in Figures 1 and 2 of the supplementary material (see lines 87–96 in Section A.5), our method shows clear improvements over the baseline (UniDSeg) in capturing semantic dependencies and spatial arrangements. Specifically, it exhibits more comprehensive spatial coverage of objects, contextually consistent category predictions, and effective suppression of fragmented outputs.\n\n  For instance, in Figure 1 (row 1), our method produces a more coherent segmentation of the \"road\" region, ensuring comprehensive spatial coverage. In Figure 1 (row 4), it avoids misclassifying \"vegetation\" as \"truck\", resulting in contextually consistent category predictions. Similarly, in Figure 2 (row 2), our method significantly reduces fragmented or implausible predictions for objects. To further demonstrate the model’s ability to capture semantic dependencies, we will include more comprehensive and systematic visual results, such as clustering-based visualizations of intermediate features, in the final version.\n\n**Q2**: It is unclear whether the performance gain from the diffusion model comes from its cross-domain compositional ability. Please provide evidence and clarify if other vision foundation models could achieve similar results.\n\n  **A**: Thank you for the thoughtful question. Figures 1 and 2 in the supplementary material intuitively demonstrate that our method achieves a deeper understanding of scene semantic distributions than the baseline (UniDSeg), as evidenced by broader spatial coverage of objects, contextually consistent predictions, and reduced fragmented outputs (see lines 87–96 in Section A.5). These improvements can be attributed to the scene-level priors of the diffusion model, due to its unique ability to compose diverse objects into coherent scenes across varying domains.\n\n  To further support this, we conducted ablation studies replacing diffusion features with those from vision foundation models (VFMs) designed for image understanding, such as CLIP and DINOv2. As shown in the table below, only the diffusion model consistently yielded significant performance gains, while the VFM-based variants underperformed. Even advanced multimodal models like Qwen2.5-VL—which exhibit stronger image-text understanding capabilities—showed inferior performance compared to diffusion model. This underscores the unique contribution of diffusion models in our framework.\n\n  Unlike VFMs that focus on capturing category-specific attributes of individual objects, diffusion models are trained to generate coherent and realistic images, requiring them to model spatial layouts, semantic relations, and contextual dependencies. These structured priors enable our method to better capture domain-invariant semantics, resulting in superior generalization. Thus, existing VFMs are unable to achieve comparable results to diffusion models.\n\n  | S:Source/T:Target | vKITTI/sKITTI | nuScenes:USA/Sing | nuScenes:Day/Night | A2D2/sKITT |\n  | ----------------- | ------------- | ----------------- | ------------------ | ---------- |\n  | CLIP              | 44.5          | 64.9              | 70.7               | 46.8       |\n  | DINO              | 44.8          | 64.1              | 69.8               | 47.3       |\n  | DINOv2            | 45.0          | 64.6              | 69.7               | 47.6       |\n  | Qwen2.5-VL        | 45.6          | 65.3              | 71.0               | 48.1       |\n  | Ours (Diffusion)  | 46.9          | 66.7              | 72.4               | 49.1       |\n\n**Q3**: The proposed method appears to have more parameters due to the diffusion model. Please provide parameter counts for each method to enable a fair comparison.\n\n  **A**: Thank you for the valuable comment. While our method incorporates a diffusion model during training, we would like to clarify that the diffusion model remains entirely frozen and is not used during inference. As a result, it introduces no additional trainable parameters and does not increase the model size at test time. The only additional trainable components are the agent queries, which introduce a relatively minor parameter overhead compared to the whole model. As shown in the table below, the total number of parameters in our method remains comparable to the baseline, with a minimal increase that does not affect inference efficiency.\n\n  | Method             | Trainable Params | Inference Params |\n  | ------------------ | ---------------- | ---------------- |\n  | UniDSeg (baseline) | 38.5M            | 38.5M            |\n  | Ours               | 40.3M            | 40.3M            |\n\n**Q4**: The noted limitation focuses on data dependency, but the reviewer is more interested in failure cases and their underlying causes.\n\n  **A**: Thank you for the valuable suggestion. In our experiments, we employed Stable Diffusion v2, which was trained on a relatively limited range of visual domains and semantic classes. This constraint made it easier to extract scene-level semantic priors. However, it also limits generalization when encountering out-of-distribution inputs, leading to potential failure cases in unfamiliar environments. While adopting more recent diffusion models—such as Flux or Stable Diffusion v3—could improve domain coverage, it comes with substantially higher computational cost. Moreover, these models are trained on extremely broad and diverse datasets, leading to highly entangled semantic spaces. As a result, it becomes considerably more challenging to identify suitable timesteps and to extract scene-level semantic priors, as these are often obscured by entangled low-level visual cues such as textures and stylized patterns."}]

Author rebuttal:

Meta-review: This paper presents a novel framework for improving the cross-domain generalization of 3D semantic segmentation through leveraging knowledge in pre-trained diffusion models. Reviewers acknowledged the contribution and strong performance of the proposed method, while initially raising some concerns, such as its efficiency and unsupported claims which require additional clarification.

After the rebuttal, the authors addressed most of the concerns, and all reviewers agreed to accept this paper. AC read all the reviews, author rebuttals, and the paper, and believes this is a good paper and recommends acceptance.
Decision: Accept (poster)