Title: No Object Is an Island: Enhancing 3D Semantic Segmentation Generalization with Diffusion Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: domain generalization

3D semantic segmentation

diffusion model
Abstract: Enhancing the cross-domain generalization of 3D semantic segmentation is a pivotal task in computer vision that has recently gained increasing attention. Most existing methods, whether using consistency regularization or cross-modal feature fusion, focus solely on individual objects while overlooking implicit semantic dependencies among them, resulting in the loss of useful semantic information. Inspired by the diffusion model's ability to flexibly compose diverse objects into high-quality images across varying domains, we seek to harness its capacity for capturing underlying contextual distributions and spatial arrangements among objects to address the challenging task of cross-domain 3D semantic segmentation. In this paper, we propose a novel cross-modal learning framework based on diffusion models to enhance the generalization of 3D semantic segmentation, named XDiff3D. XDiff3D comprises three key ingredients: (1) constructing object agent queries from diffusion features to aggregate instance semantic information; (2) decoupling fine-grained local details from object agent queries to prevent interference with 3D semantic representation; (3) leveraging object agent queries as an interface to enhance the modeling of object semantic dependencies in 3D representations. Extensive experiments validate the effectiveness of our method, achieving state-of-the-art performance across multiple benchmarks in different task settings. Code is available at \url{https://github.com/FanLiHub/XDiff3D}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies 3D Semantic Segmentation Generalization enhanced by diffusion models. Specifically, it aims to harness the ability from diffusion models to capture underlying contextual distributions and spatial relationships among objects to tackle the challenging task of cross-domain 3D semantic segmentation. To achieve it, it designs  Dual-queries Consistency Scheme to extract features from diffusion model and construct Consistency Constraint between  queries from different timesteps. The experimental results show that the proposed method outperforms previous methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The method is novelty and effective.\n\n- The ablation can show the effectiveness of each components.\n\nWeakness:\n\n- The authors claim that it uses diffusion models to address the dependency between objects. However, it does not provide evidence to support the motivation. It is recommended to provide detailed analysis (visualization and/or quantitative results).\n\n- The experimental results show that the diffusion model indeed bring improvements. But the improvements really result from  \"the diffusion model’s ability to flexibly compose diverse objects into high-quality images across varying domains\"? Please provide evidence. In addition, can we use other visual foundation models to achieve it?\n\n- It seems that the number of parameters of the proposed method is higher than previous methods due to the introduction of diffusion models. Thus, please provide the parameters for each method and provide a more fair comparison.\n\nQUESTIONS:\n- The authors claim that it uses diffusion models to address the dependency between objects. However, it does not provide evidence to support the motivation. It is recommended to provide detailed analysis (visualization and/or quantitative results).\n\n- The experimental results show that the diffusion model indeed bring improvements. But the improvements really result from  \"the diffusion model’s ability to flexibly compose diverse objects into high-quality images across varying domains\"? Please provide evidence. In addition, can we use other visual foundation models to achieve it?\n\n- It seems that the number of parameters of the proposed method is higher than previous methods due to the introduction of diffusion models. Thus, please provide the parameters for each method and provide a more fair comparison.\n\nLIMITATIONS:\nThe authors note a limitation: 'the lack of source-domain data makes it reliant on paired images and point clouds in the target domain.' However, the reviewer is more interested in scenarios where the method underperforms and the reasons behind a limitation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces XDiff3D, a novel cross-modal learning framework based on diffusion models to enhance the cross-domain generalization of 3D semantic segmentation. The authors leverage diffusion models to capture semantic dependencies among objects, constructing object agent queries to infuse 2D diffusion priors into the 3D representation space. A dual-queries consistency scheme is proposed to suppress visual details and focus on domain-invariant semantics. Experiments across multiple benchmarks demonstrate state-of-the-art performance, outperforming existing methods in various settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n•\tOriginal Approach: The use of diffusion models to capture object-level semantic dependencies for 3D segmentation is innovative, addressing a gap in existing methods that treat objects in isolation.\n•\tComprehensive Evaluation: The paper evaluates XDiff3D across diverse benchmarks, including synthetic-to-real, real-to-real, and normal-to-adverse weather transitions, showcasing its robustness.\n•\tAblation Studies: Detailed ablation experiments validate the effectiveness of key components, such as object agent queries, dual-queries consistency, and loss functions.\n•\tClear Contributions: The framework is well-motivated, with a clear explanation of how diffusion priors address limitations in existing 3D segmentation approaches.\n\nWeaknesses\n•\tComputational Overhead: The use of diffusion models may introduce additional computational complexity.\n•\tDependency on Paired Data: The method relies on paired images and point clouds, which could limit applicability in scenarios where multi-modal data is scarce.\n•\tClaim Justification: The method asserts its capability to effectively capture \"semantic dependencies\", yet lacks both quantitative or qualitative evidence, such as attention maps, to substantiate this claim.\n\nQUESTIONS:\n1. Computational Efficiency: Since the diffusion model is applied, it is neccessraty to provide a direct comparison of computational resources between XDiff3D and other methods (e.g., training time, memory usage, FLOPs).\n2. Semantic Dependency Validation: The object agent queries claim to capture \"semantic dependencies\". Could attention maps or correlation metrics (e.g., inter-object affinity scores) be provided for the justification over other methods?\n3. Impact of different diffusion layers: The author introduced the dual-queries consistency scheme to suppress visual details. Have the authors considered not utilizing all diffusion layers from the start? It would be beneficial to conduct experiments exploring the selection of diffusion layers to determine their impact on the model's performance and the suppression of visual details.\n4. Failure case analysis in limitation: In the supplementary material, the authors outline certain limitations; however, the discussion primarily centers around input constraints. To provide a more comprehensive understanding, it is advisable to include specific failure cases encountered when applying the proposed methods, accompanied by in-depth analyses of the underlying causes. This approach would not only enhance transparency but also offer valuable insights for future improvements and applications.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents XDiff3D, a novel cross-modal framework leveraging diffusion models to improve the cross-domain generalization of 3D semantic segmentation. By constructing object agent queries from diffusion features, decoupling irrelevant local details, and enhancing 3D representations via cross-modal semantic alignment, the method captures object-level semantic dependencies more effectively. Extensive experiments demonstrate state-of-the-art performance across diverse domain adaptation benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The writing is clear and well-organized.\n2. The method is novel and builds a compelling cross-modal framework using diffusion models.\n3. Experiments demonstrate strong performance with thorough ablation studies.\n\nWeaknesses:\n1. The paper lacks a comparison of training and inference time with other methods, which is particularly important given the potential computational overhead introduced by diffusion models.\n2. The method depends on access to paired RGB images and point clouds in the target domain, which may hinder its practicality in real-world applications where obtaining fully aligned multi-modal data is often challenging.\n3. The method uses empty text as the textual condition, which may not fully activate the semantic potential of the diffusion model. In contrast, incorporating class names or prior semantic labels could yield more precise and informative semantic features.\n\nQUESTIONS:\n1. How does the method compare to other approaches in terms of training and inference time, given the computational demands of diffusion models?\n\n2. Is it possible to use more informative textual conditions (e.g., class names or semantic prompts) instead of empty text to better exploit the semantic potential of diffusion models?\n\n3. How robust is the method to noisy or incomplete point cloud data, especially in target domains where LiDAR quality may vary?\n\n4. Given that the method relies on paired RGB images and point clouds, how sensitive is it to calibration or alignment errors between the two modalities?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper focuses on enhancing the cross-domain generalization of 3D semantic segmentation task. They propose a diffusion model based cross-modal learning framework named XDiff3D. XDiff3D proposes to leverage diffusion model priors for 3D semantic segmentation. It mainly includes three components: (1) Contruction of Object Agent Queries, (2) Dual-Queries Consistency Scheme and (3) Cross-Modal Semantic Enhancement.

STRENGTHS AND WEAKNESSES:
Strengths 
(1) XDiff3D propose to inject 2D diffusion priors into 3D representations via object agent queries to enhance cross-domain generalization.
(2) XDiff3D  achieves state-of-the-art results, and extensive experiments also show the effectiveness of the proposed method.

Weaknesses
(1) The framework in Fig.2 is complex, which makes it hard to follow. It's vital to show the core idea of the proposed method.

QUESTIONS:
(1) Why aggregate the semantic information of objects using diffusion features to form the object agent queries? What are the advantages of using diffusion modes in module (a)?
(2) What's the effect of SVD in module (b)? Is SVD a standard method for feature quantization?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-13