Title: CymbaDiff: Structured Spatial Diffusion for Sketch-based 3D Semantic Urban Scene Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: 3D semantic scene generation

3D semantic scene completion
Abstract: Outdoor 3D semantic scene generation produces realistic and semantically rich environments for applications such as urban simulation and autonomous driving. However, advances in this direction are constrained by the absence of publicly available, well-annotated datasets. We introduce SketchSem3D, the first large‑scale benchmark for generating 3D outdoor semantic scenes from abstract freehand sketches and pseudo‑labeled annotations of satellite images. SketchSem3D includes two subsets, Sketch-based SemanticKITTI and Sketch-based KITTI-360 (containing LiDAR voxels along with their corresponding sketches and annotated satellite images), to enable standardized, rigorous, and diverse evaluations. We also propose Cylinder Mamba Diffusion (CymbaDiff) that significantly enhances spatial coherence in outdoor 3D scene generation. CymbaDiff imposes structured spatial ordering, explicitly captures cylindrical continuity and vertical hierarchy, and preserves both physical neighborhood relationships and global context within the generated scenes. Extensive experiments on  SketchSem3D demonstrate that CymbaDiff achieves superior semantic consistency, spatial realism, and cross-dataset generalization. The code and dataset will be available at here.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents CymbaDiff, a novel approach for generating 3D outdoor scenes from sketches and pseudo-labeled satellite images. The model enhances spatial coherence by capturing cylindrical continuity and vertical hierarchy in urban environments. The authors introduce the SketchSem3D dataset, a large-scale benchmark for evaluating sketch-based 3D scene generation. The method outperforms existing approaches in terms of semantic consistency and spatial realism, demonstrating robust performance across multiple urban datasets.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths \n1. Quality of the Work: The paper introduces a novel and robust generative model for 3D outdoor scene generation from abstract sketches and satellite image annotations. The model, CymbaDiff, effectively addresses the challenge of spatial coherence in urban environments\n2. Clarity: The paper is well-structured, with clear explanations of the proposed CymbaDiff model, the SketchSem3D dataset, and the experimental setup. The key contributions are outlined concisely, making it easy for readers to grasp the significance of the work.\n3. Significance: The proposed sketch-based generation approach addresses a critical gap in the current literature on 3D outdoor scene synthesis, especially for applications like autonomous driving and urban simulation, where the generation of complex, semantically rich environments is crucial.\n## Weaknesses\n1. Quality of the Work: While the paper demonstrates strong performance on the SketchSem3D dataset, it would be beneficial to evaluate the model on more real-world data or include cross-domain experiments. This would further validate the model’s generalizability in unseen real-world urban settings.\n2. Clarity. Lack of a detailed analysis of failure cases: While the paper highlights the strengths of the model, it does not provide a deep dive into failure modes or limitations. A discussion of specific types of scenes or conditions where the model struggles could help provide a more balanced assessment of its capabilities.\n3. Significance: Although the paper demonstrates impressive results in controlled environments, scalability to larger urban areas or more diverse scene types (e.g., complex rural or mixed terrains) is not fully explored. \n4. Absence of user studies or human evaluation: The paper focuses on quantitative metrics to evaluate the model's performance, but it lacks any user studies or human evaluations to gauge how intuitive and effective the system is from a practical, real-world perspective.\n\nQUESTIONS:\n1. Clarification of the Pseudo-Labeled Satellite Image Annotations Process:\nQuestion: The paper uses pseudo-labeled satellite image annotations generated by CLIP and SAM for training. Could you provide more details on how the pseudo-labeling pipeline handles cases where CLIP or SAM might produce inaccurate or ambiguous labels? Specifically, how robust is the model to errors in the pseudo-labeling process?\nSuggestion: A more detailed analysis of the error propagation in the pseudo-labeled data pipeline would be useful. Discuss any failure modes observed in the pseudo-labeling process and whether any mechanisms, such as manual refinement or a confidence threshold, are employed to mitigate errors.\n2. Question: The paper demonstrates strong results on the SketchSem3D dataset, but its scalability to more diverse, larger-scale urban scenes is not fully explored. How well does CymbaDiff generalize to large-scale real-world urban environments or scenes that are more complex or varied than those in the SketchSem3D dataset?\nSuggestion: It would be valuable to conduct additional experiments on real-world datasets or more diverse scenes beyond the current benchmark. This would provide insights into whether the model can handle the scale and complexity of urban environments as encountered in real applications, like autonomous driving or urban planning.\n3. Question: The paper lacks extensive quantitative comparison experiments with other methods. While a single comparison is provided in the supplementary material, would it be possible to include more quantitative results comparing CymbaDiff to other state-of-the-art methods, especially in terms of generation quality, computational efficiency, and generalization across different datasets?\nSuggestion: It would strengthen the paper to expand the comparison to more methods, ideally covering a range of recent techniques in the domain of 3D scene generation.\n4. Question: While the paper proposes a novel approach to generating 3D scenes from sketches and pseudo-labeled satellite images, could you elaborate on the specific innovative aspects of CymbaDiff that distinguish it from existing methods, particularly in terms of model architecture and novel contributions to the field of 3D scene generation?\nSuggestion: It would be helpful to provide a clearer distinction between CymbaDiff and other state-of-the-art models, especially in the way the Cylinder Mamba blocks are integrated into the diffusion process. How does this structured spatial coherence contribute uniquely to the generation process compared to previous methods like UrbanDiff or Semcity?\n5. Question: Could you provide some failure cases or error examples to objectively illustrate the limitations of the CymbaDiff method?\nSuggestion: Including error analysis would be very useful in highlighting specific weaknesses of the model, such as when it fails to generate accurate representations for highly complex scenes (e.g., dense urban environments with varying object sizes or moving elements) or low-quality inputs (e.g., poorly sketched or ambiguous freehand sketches).\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of generating semantically rich 3D outdoor scenes from freehand sketches and satellite pseudo-labels. The authors propose CymbaDiff, a diffusion-based framework that enhances spatial consistency in large-scale urban scenes by explicitly modeling cylindrical continuity and vertical hierarchies. To enable standardized evaluation, the authors introduce SketchSem3D, a large-scale benchmark for sketch-based 3D semantic scene generation, comprising two subsets: Sketch-based SemanticKITTI and Sketch-based KITTI-360.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The method introduces cylinder mamba blocks to enforce structured spatial ordering, explicitly capturing cylindrical continuity and vertical hierarchies while preserving local/global scene coherence, overcoming limitations of Cartesian-based sequential modeling.\n2. The work establishes SketchSem3D, the first large-scale benchmark for sketch-based 3D semantic scene generation. \n\nWeaknesses:\n1. L120-L122: The paper's justification for using Mamba over DiT hinges critically on overcoming the latter's \"high memory costs\" in 3D scene generation. However, no experimental data (e.g., parameter counts, inference memory consumption, runtime) is provided to demonstrate the purported memory efficiency gains of the proposed approach. Substantiating this key aspect requires such comparative analysis.\n2. L240-L249: While the proposed cylindrical coordinate system aims to solve the issue of \"disrupting original 3D spatial adjacency structures\" inherent to serialization in Cartesian coordinates, its efficacy for common street-view datasets like KITTI (shown in the paper) is unclear. Since such scenes lack rotational symmetry, simply \"(θ,r) unfolding\" the cylinder likely does not preserve meaningful adjacency relationships. \n3. L263: Triplane Mamba uses Cartesian coordinates while C-Mamba uses cylindrical coordinates, leading to different feature indexing schemes. Please clarify how feature alignment is achieved before fusion: Are the features directly fused, or is a coordinate transformation.\n4. L249:  It is stated that the evaluation metric differs from SemCity's approach. Yet, the reported FID performance for the SemCity baseline in this work matches the SemCity paper's reported FID value exactly.\n5. L308-L315: Request for explanation of ablation result. The decomposition of 3D convolution across three dimensions (Sec. 4.3) is intuitively expected to trade-off some reconstruction quality for computational efficiency. However, the ablation study shows a significant decrease in FID after applying DDCB. \n\nMinor:\nFigure 4 can be enhanced by incorporating fine-grained structural consistency analysis, as employed in scene reconstruction benchmarks like NeRF-Synthetic. Quantification of spatial coherence patterns (e.g., cylindrical projection seam artifacts or voxel occupancy consistency) would strengthen methodological transparency.\n\nQUESTIONS:\nAlthough the authors have made significant contributions in generating outdoor scenes using abstract views, some details in algorithm design and module implementation require clearer explanations to facilitate reader comprehension. Experimental metrics should also incorporate evaluations of algorithm efficiency performance to validate the achievement of experimental objectives.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a sketch-to-3D scene generation dataset to enable the outdoor generation of 3D scenes from sketch drawings, based on the KITTI-360 and SemanticKITTI datasets. Together with the dataset, a 3D scene diffusion method is also proposed that leverages a state-space formulation trained with the sketch-to-3D scene generation dataset.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper's main strength lies in both the dataset, which includes paired sketches, BEV pseudolabels, and 3D semantic scenes, and the model architecture that generates scenes conditioned on the sketch images. The dataset is well-built, and the task of scene generation conditioned on a sketch is useful, as sketching a scene provides an easy interface between the user and the generated scene. The model leverages 2D sketches, upsampling them to 3D, which enables the conditioning and generation of the corresponding 3D scene.\n\nThe paper's main weakness is regarding the clarity of the paper. The dataset is well-explained, with details about the sketch and pseudolabel generation. However, the method (Sec. 4) lacks details. Sec. Sections 4.1 and 4.2 do not provide details about the Scene Structure Estimation Network, nor do they offer any information about the Latent Mapping network. Additionally, the diffusion network is built upon state-space formulation works (Mamba). In Sec. 4.4 the variables x(t) , y(t) and h(t) are introduced but those variables are never linked to the z_TMB and f_TMB variables afterwards. For clarity, I suggest adding a brief explanation of the state-space formulation and then linking it to the z_TMB and f_TMB variables. I see the value in the method, but a clearer explanation would be helpful, as it would also strengthen the contributions in the paper.\n\nSome minor concerns are that, from lines 286 to 299, it was unclear which table the text was referring to. Also, the paired sketch and pseudolabels in Fig. 4 seem to be from different scenes. They seem unrelated most of the time. Lastly, I would suggest mentioning other recent scene generation methods, such as \"DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenes\" and \"Towards Generating Realistic 3D Semantic Training Data for Autonomous Driving\", for completeness of the related works.\n\nQUESTIONS:\n- Why do the sketches and pseudolabels (Fig. 4) not always seem to be consistent/related?\n- During the CLIP pseudolabels generation, are the semantic classes present in the 3D GT semantics used to filter out wrong CLIP classes?\n- Are the textual descriptions (Fig. 1) also available within the dataset?\n\nLIMITATIONS:\nThe main limitation that I see is that the sketches are too precise to be considered as human drawings. However, this limitation is also mentioned in the paper. Still, the dataset is a first step towards generating a 3D scene from human-drawn sketches.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a dataset of outdoor scenes with paired sketches, semantic maps, and semantic voxels. It presents a pipeline that automatically constructs the dataset from existing semantic outdoor datasets, ground-truth semantic voxels and satellite imagery. The resulting dataset is approximately 2.7 times larger than existing BEV-conditioned datasets and supports more flexible conditioning. Furthermore, the authors propose an LDM model with Cylinder Mamba diffusion to preserve spatial relationships for conditioned 3D scene generation.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nThe dataset is larger than previous BEV based NuScenes and provides more flexible conditioning. The description of the dataset construction is clear and the model also show better quantitative results with respect to the chosen baselines.\n\n## Weaknesses\n\nOn the writing side, the clarity of the proposed method can be improved by giving more detail regarding the key components of the model. I also have concerns about the dataset, particularly the necessity of including both sketches and pseudo-labeled annotations as inputs. Additionally, the paper also lacks some more baselines that would make the evaluation more complete, and the validity of the semantic scene completion comparisons in section 5.2 is questionable. More details are given in the questions below.\n\nQUESTIONS:\n1. Section 4 lacks sufficient details about key components. For the CSCB and DDR blocks, additional architectural tables or figures in the supplementary would have been helpful. The details regarding the cylinder mamba blocks is also quite limited. Such as the functionality and architectural details of the triple mamba mechanism as well as the ordering mechanism in the cylinder mamba block.\n\n2. The diffusion model is conditioned on both PSA and sketches. However, the necessity of the sketch is unclear since the PSA alone should provide sufficient information for 3D scene generation. A more user friendly approach could involve a separate network to predict PSA from sketches. Finally, it's unclear to me why the pseudo labels from satellite images are needed as opposed to directly using the BEV projections of the semantic voxels.\n\n3. The paper also lacks relevant conditioned scene generation baselines in table 2. As 3D latent diffusion and 3D DiT were not designed for 3D scene generation specifically. For example, the proposed method could have been run on NuScenes using BEV images as input to compare against UrbanDiff. This would also help understand whether the proposed task is easier or harder compared to BEV conditioned NuScenes. BlockFusion [1] is also another potential baseline that can be compared.\n\n4. I found the 3D semantic scene completion results in table 4 to not be very informative. As the other methods takes monocular or stereo inputs, while the proposed method uses SK+PSA. SK+PSA may offer an easier setting as the semantic labels are directly provided to the model, whereas working with raw images is more challenging and requires stronger feature extractors. As a result, making direct comparisons with other methods less meaningful.\n\nI think there is value in establishing unified datasets and benchmarks for evaluating 3D scene generation as well as offer more flexible inputs. If the above concerns can be addressed during the rebuttal I am willing to raise my rating.\n\n[1] Wu, Zhennan, et al. \"Blockfusion: Expandable 3d scene generation using latent tri-plane extrapolation.\" ACM Transactions on Graphics (TOG) 43.4 (2024): 1-17.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}