CymbaDiff: Structured Spatial Diffusion for Sketch-based 3D Semantic Urban Scene Generation
Authors: Li Liang
Bo Miao
Xinyu Wang
NAVEED AKHTAR
Jordan Vice
Ajmal Saeed Mian
Venue: NeurIPS 2025 poster ()
Keywords: 3D semantic scene generation
3D semantic scene completion
Primary area: deep_learning
TL;DR: Introducing a new task " sketch-based 3D outdoor scene generation" and presenting the first 3D outdoor generation dataset and propose CymbaDiff for 3D semantic scene generation task

Abstract: Outdoor 3D semantic scene generation produces realistic and semantically rich environments for applications such as urban simulation and autonomous driving. However, advances in this direction are constrained by the absence of publicly available, well-annotated datasets. We introduce SketchSem3D, the first large‑scale benchmark for generating 3D outdoor semantic scenes from abstract freehand sketches and pseudo‑labeled annotations of satellite images. SketchSem3D includes two subsets, Sketch-based SemanticKITTI and Sketch-based KITTI-360 (containing LiDAR voxels along with their corresponding sketches and annotated satellite images), to enable standardized, rigorous, and diverse evaluations. We also propose Cylinder Mamba Diffusion (CymbaDiff) that significantly enhances spatial coherence in outdoor 3D scene generation. CymbaDiff imposes structured spatial ordering, explicitly captures cylindrical continuity and vertical hierarchy, and preserves both physical neighborhood relationships and global context within the generated scenes. Extensive experiments on  SketchSem3D demonstrate that CymbaDiff achieves superior semantic consistency, spatial realism, and cross-dataset generalization. The code and dataset will be available at here.

CymbaDiff: Structured Spatial Diffusion for Sketch-based 3D Semantic Urban Scene Generation
Li Liang'
Bo Miao
Xinyu Wang' Naveed Akhtar*
Jordan Viceo
Ajmal Mian'
The University of Western Australia
AIML, The University of Adelaide
3 The University of Melbourne
Abstract
Outdoor 3D semantic scene generation produces realistic and semantically rich environments for applications such as urban simulation and autonomous driving. However, advances in this direction constrained by the absence of publicly available, well-annotated datasets. We introduce SketchSem3D, the first large-scale benchmark for generating 3D outdoor semantic scenes from abstract freehand sketches and pscudo-labeled annotations of satellite images. SketchSem3D in- cludes two subsets, Sketch-based SemanticKITTI and Sketch-based KITTI-360 (containing LiDAR voxels along with their corresponding sketches and annotated satellite images), to enable standardized, rigorous, and diverse evaluations. We also propose Cylinder Mamba Diffusion (CymbaDiff) that significantly enhances spatial coherence outdoor 3D scene generation. CymbaDiff imposes structured spa- tial ordering, explicitly captures cylindrical continuity and vertical hierarchy, and preserves both physical neighborhood relationships and global context within the generated scenes. Extensive experiments SketchSem3D demonstrate that Cym- baDiff achieves superior semantic consistency, spatial realism, and cross-dataset generalization. The code and dataset will be available at https://github.ccm com//illian: research- Shraarchhhub/Cymbaiif...
1 Introduction
Generative modeling has demonstrated remarkable progress in the 2D and 3D domains, largely fueled by the rapid development of diffusion models [1, 2, 4J. In 3D, diffusion approaches have significantly advanced 3D object synthesis [5, 6] and indoor scene generation [7, 8]. However, generating large-scale 3D outdoor environments remains widely underexplored [9, 10, 11]. outdoor urban scenes pose greater challenges due to their higher semantic diversity, complex spatial structures, and dynamic contextual dependencies. Despite these challenges, synthesizing realistic and scalable 3D urban scenes is increasingly critical, as it underpins a wide range of emerging applications, including city-scale simulation [12, 13] and autonomous driving [14, 15, 16, 17].
A few methods have recently surfaced for 3D outdoor scene generation [9, 10, 11, 18, 19], often relying on bird's-eye view (BEV) with only road data multi-scale scene hierarchies guide generation. BEV-based approaches suffer from insufficient 3D structural information, limiting both semantic richness and geometric fidelity. Meanwhile, modeling multi-scale scene hierarchies typically requires generative models to repeatedly synthesize scenes at multiple spatial resolutions, increasing both computational and structural complexity. Moreover, due to the lack of public large-scale benchmark, current approaches typically use self-curated and heavily preprocessed datasets for evaluation [9], which fundamentally constrains rigorous benchmarking. Sketch-based methods [20, 21, 22, 23] have recently emerged as a promising paradigm for user-guided 3D generation, enabling intuitive control through freehand drawings. However, their applicability remains confined to the synthesis of isolated 3D objects or simple indoor scenes. Expanding sketch-based 3D reconstruction to outdoor scenes is currently widely open. Challenges in this novel pursuit arise from complex
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

scene layouts, diverse object geometries, and the need to preserve spatio-semantic coherence across large-scale scenes.
This work takes a significant step towards extending sketch-based generation to outdoor environments. To that end, build upon the growth of State Space Models (SSMs) [24], which have gained increased attention across image segmentation [25, 26] and point cloud processing [27, 28] for their ability to capture long-range dependencies while remaining efficient through selective computa- However, to enhance global contextual understanding, SSMs typically aggregate information from multiple scan directions, leading to substantial memory overhead. Moreover, the scanning order imposed by the Cartesian coordinate system can distort local neighborhood relationships, especially in scenes with limited spatial coherence.
To address the above-noted challenges for sketch-based 3D outdoor scene generation, we first present "SketchSem3D"" large-scale dataset tailored for the task. SketchSem3D enables the synthesis of semantically rich outdoor 3D environments from freehand sketches and pseudo-labell satellite image annotations. The annotation pipeline properly integrates CLIP-based textual guidance [29] with image embeddings from the Segment Anything Model (SAM) [30], enabling robust and automated semantic labeling. SketchSem3D comprises two subsets, Sketch-based SemanticKITTI and Sketch- based KITTI-360, designed to support standardized benchmarking and fair comparison. Building upon this dataset, we define the novel 'sketch-based 3D outdoor scene generation' research task. We also propose Cylinder Mamba Diffusion, the first approach to handle this task. As adjacent Cartesian- voxel sequences may misrepresent spatial proximity in outdoor scenes, CymbaDiff is particularly tailored to handle voxel discrepancies. Our underlying model is denoising network, combining an SSM architecture with generative diffusion in the latent space. We design cylinder mamba blocks to enhance spatial coherence during the generative process, imposing structured spatial ordering to explicitly encode cylindrical continuity and vertical hicrarchy, preserving spatial neighborhood relationships within scenes.
Our key contributions are summarized below:
We introduce the novel task of 'sketch-based 3D outdoor scene generation', which enables intuitive and flexible user interaction through freehand sketches and pseudo-labeled satellite image annotations. By reducing the need for manual semantic annotation, this task offers an efficient solution to generate training data for applications such as urban-scale simulation and autonomous driving.
We present SketchSem3D, the first public large-scale sketch-based benchmark for 3D outdoor semantic scene generation. It includes two subsets, Sketch-based SemanticKrTTI and Sketch- based KITTI-360, and enables standardized benchmarking for the development and evaluation of generative models in complex outdoor settings.
We propose CymbaDiff, a generative model that incorporates the proposed cylinder mamba blocks to enhance spatial coherence during the generation process. We also conduct extensive experiments the Sketch-based SemanticKITTI and Sketch-based KITTI-360 benchmarks, demonstrating state-of-the-an performance in 3D semantic scene generation and completion.
2
Related Work
2.1 State Space Models
Recent studies have demonstrated the strong capability of State-Space Models (SSMs) in capturing
long-range dependencies across sequential data [31, 32]. These models have been successfully applied in a variety of domains, including medical image segmentation [25, 33], image restoration
[34, 35], natural language processing (NLP) [36, 37], and point cloud processing [38, 28]. Many of these approaches build upon foundational architectures such as VisionMamba [39], S4ND [40], and Mamba-ND [41]. Specifically, VisionMamba [39] integrates bidirectional SSMs for data-dependent global context modeling and employs positional embeddings to enhance location-ave -aware visual recognition. S4ND [40] extends the SSM framework by incorporating local convolution operations, thereby enabling processing beyond one- me-dimensioonll inputs. Mamba-ND [41] further addresses multi- dimensional data by utilizing various sean patterns within a single block to enhance performance in discriminative tasks. Despite their strengths, these methods primarily focus on maximizing contextual information through multiple scanning directions, often neglecting structured spatial coherence across horizontal and vertical hicrarchies, particularly under meemory-constriine settings.

---

2.2 3D Semantic Scene Generation
Diffusion models have evolved from generating 2D images to addressing increasingly complex 3D data modeling tasks [2]. Compared to traditional generative models such Generative Adversarial Networks (GANs) [42] and Variational Autoencoders [43], diffusion models follow a progressive denoising process [44], which enhances training stability and improves the capacity to capture complex data distributions. These advantages render diffusion models particularly suitable for 31) data generation tasks. While much of the existing research has focused
object-level synthesis [45, 46, 47, 48, 49, 50, 51, 52] and indoor scene generation [53, 54, 55, 56], there is a growing body of work exploring 3D outdoor semantic scene generation [57, 11, 10, 58, 47, 9] as it underpins wide range of emerging applications, including autonomous driving [14, 15, 16, 17] and city-scale simulation [12, 13.. For instance, UrbanDiff [9] conditions generation on BEV maps produce urban scenes in the form of semantic occupancy grids, integrating both geometry and semantic information. P-DiscreteDif [10] proposes a progressive multi-scale strategy that synthesizes large-scale 3D scenes by conditioning each stage on the output from the preceding resolution level, with the initial model conditioned solely on noise. Despite these advancements, the absence of standardized datasets for 3D outdoor semantic scene generation has led to the use of heterogeneous benchmarks with inconsistent scene conditions, thereby limiting fair comparison and hindering systematic progress in the field.
2.3 3D Semantic Scene Completion
3D semantic scene completion methods can be broadly categorized into four categories: image-based approaches [59, 60, 611. point cloud-based methods [62, 633, voxel-based techniques [64, 65], and multi-modallty-basee
frameworks [66, 67]. Most existing methods are built upon convolutional neural networks (CNNs) or Transformer-bast based architectures. For instance, Xia et al. [65] propose a CNN network (SCPNet), which enhances single-frame scene completion by incorporating dense relational semantic knowledge distillation along with a label rectification strategy to mitigate artifacts introduced by dynamic objects. CGFormer [59] enhances semantic scene completion by introducing context- and geometry-aware voxel transformer, which initializes queries based on the contextual information from individual input images and extends deformable tross-attention mechanisms from 2D image space 3D voxel space. While CNNs are computationally efficient, they are inherently limited by their receptive field size. Transformers address this limitation by enabling global context modeling but come with high memory costs. Recently, Segmamba [25] has emerged as a promising alternative, offering a favorable trade-off by supporting large receptive fields with improved memory efficiency, making it suitable for 3D semantic scene completion.
3 SketchSem3D Dataset
Sketch-based methods have recently gained increasing attention as a promising paradigm for user- guided 3D modeling, offering intuitive and flexible interaction through freehand drawing. While these approaches show great potential, they are constrained to generating isolated 3D objects and lack the capacity to model complex, semantically rich scenes. In a related direction, UrbanDiff [9] introduced BEV representations as conditional inputs for 3D semantic scene generation. By leveraging the spatial alignment between 2D projections and 3D structures, this approach promotes 2D-to-3D consistency. However, BEV-based supervision inherently constrains the diversity of the generated scenes. Moreover, acquiring BEV images that accurately reflect the semantic layout of complex 3D environments is particularly challenging in outdoor settings.
We propose a sketch-based framework for 3D outdoor semantic scene generation. enables users to define scene layouts using coarse freehand sketches combined with pseudo-labeled satellite image annotations, facilitating a more natural and accessible interaction modality. By circumventing the need for labor-intensiv annotations and large-scale sensor-based data collection, the framework significantly enhances scalability. We leverage this framework in the design of our SketchSem3D benchmark dataset.
3.1 Benchmark Construction
The benchmark comprises two distinct datasets, Sketch- based SemanticKITTI and Sketch- KITTI-360, each constructed through systematic three-stage pipeline discussed below.
Data Sourcing. We construct the two datasets using the 3D ground truth (GT) from Se- manticKITTI [68] and CCC-Bench-KKTTI-S66 [69], respectively. Each scene is enriched with freehand sketches and pseudo-labeled satellite image annotations enable conditioned 3D scene generation. Both datasets comprise five components: frechand (like) sketches, satellite images, pseudo-labeled
3

---

Broad, continuuus carriageways forming four-way intersection and serving the primary vehicle comidors.
Road:
road
sidewalk
building
Chat
Sidewallas: Narrow linear strips flanking road edges and wrapping comers,, designatedd for pedestrian circulation.
vogetation
Building: Reciangular blacks indicating man-made structures aligned along block edges heside sidewalks.
fence
Terrain
Vegetation: Irrogular patches of dense plant cover, clastering along verges and within open parcels...
I Lahel keywords
Detailed descriptions
Google API
SAM
CLIP
Latitude: 48, 48.98530
oeginude: 8. 8.39346
3D GT
Satelltt Image
Mask Embedding
Text Embedding
Canny Edge Detection Algerichm
BEV
Sketch
PSA
Similarity Calculation
Figure 1: Pipeline for SketchSem3D construction. SAM and PSA denote Segment Anything Model and Pseudo-labeled Satellite Image Annotations, respectively.
Table 1: Comparing SketchSem3D (last two rows) with BEV-based NuScenes.
Dataset
Pairs
Condition
3D Geospatial Semantics
Classes
3D GT Voxels
BEV-based NuScenes [9]
34149
17
192 x 192
Sketch-based SemanticKITTI
58987
Sketch / PSA
20
256 x 256 32
Sketch-base KITTI-360
36057
Sketch / PSA
19 256 x 256 x 32
annotations, semantic label keywords, and 3D GT (output). Figure shows the dataset construction pipeline. The 3D GT is extended from the respective source datasets. Sketches are generated by applying the Canny edge detector [70] BEV projections of 3D GT. These sketches closely resemble freehand drawings, which can be more easily produced at test time compared to BEV projections, providing abstract representations of scene geometry.
Semantic categories (e.g., road, tree, vehicle) are also available as GT and recorded as label keywords without spatial encoding. To enrich the semantic context, GPT-4 [71] is used to generate descriptive texts for each category, supporting alignment with visual features. We leverage the GPS information provided in KITTI [72] and KITTI-360 [73] retrieve the corresponding satellite images. We then apply CLIP [29] to encode the enriched contextual descriptions and SAM [30] to obtain mask-level embeddings from the satellite images. By computing the cosine similarity between text and image embeddings, we infer the semantic composition of each scene from the satellite perspective, producing the pseudo-labeled annotations used in our SketchSem3D dataset.
Data Filtering and Formatting. To address any semantic labeling errors or inconsistencies in the automated alignment between CLIP [29] text emheddings and SAM [30] image mask embeddings, perform a manual review of the resulting class distributions to ensure annotation accuracy and dataset reliability. Each sketch-based dataset consists of five components: (i) the sketch, (ii) satellite image, (iii) pseudo-labeled satellite image annotations, (iv) label keywords, and (v) 3D GT. The sketch is stored as a binary edge map in image format, capturing the structural outline of the scene. The satellite image is geo-referenced RGB image of the same size, spatially aligned with the GPS coordinates of the corresponding scene. The pseudo-labelae labeled satellite image annotations are single-channel semantic maps, where each pixel represents semantic class ID. Although two awo-dimensional, these annotations provide coarse semantic cues that serve as important conditional guidance for reconstructing 3D) voxel scenes. The label keywords for each scene are saved in a txt file indexed by scene II), listing the semantic class keywords present the scene. Finally, 3D GT is provided as a volumetric label map, where each voxel is assigned semantic class encoded as a 16-bit unsigned integer, following the format of SemanticKITTI [68].
3.2 Data Statisties Comparison and Evaluation Metrics
Table compares our SketchSem3DD dataset with the BEV-based NuScenes dataset [9]. We can see that our dataset is better in every aspect offering higher resolution, more classes, additional geospatial

---

Training
CNN
3D CMN Deccder
Testing
Vooel
Skip connection
Add
Noses
SSEN
UMN
CyMs
CylMa
3D CNN
Decoderr
Vovel
PSA
Starch
CymbaD Denolier
Figure 2: Architecture of our CymbaDiff generation network. The Scene Structure Estimation Network (SSEN) extracts abstract structural information from Pseudo-labeled Satellite Image Anno- tations (PSA) and the Sketch. The Latent Mapping Network (LMN) compresses the input conditions into a latent representation, which then processed by the CymbaDiff denoiser, which utilizes the proposed cylinder mamba blocks (CylMa) to perform latent denoising.
semantics, two conditions instead of one and contains a much larger number of 3D scenes (total 95,044 compared 34, 34,149 in [9]). Notably, each subset of SketchSem3D contains more frames than [9]. Moreover, our conditions (sketch and ) are easier to obtain at test time, enhancing the practicality. Sketch-based SemanticKITT1 includes 58, 172 training and 815 validation frames, while Sketch-based KITTI-360 consists of 33,892 training and 2,165 validation frames. In SketchSem3D, all satellite images, sketches, and pseudo-labeled annotations are standardized 256 x 256 pixels, with corresponding 3D GT of 256 x 256 x 32 voxels. In comparison, BEV-based NuScenes [9] contains 3D GT of 192 192 x 16 voxels and lacks explicit geospatial structure as well as detailed 3D semantic distribution.
resolution of
To evaluate the quality and diversity of the generated 3D semantic scenes, adopt two widely used metrics: Fréchet Inception Distance (FID) [74] and Maximum Mean Discrepancy (MMD) [9]. Together, these metrics capture statistical similarity and feature-level realism, providing comprehen- sive assessment of generative performance. Further details on the evaluation metrics are supplied in the Appendix.
4 Method
We propose a 3D semantic scene generation method that captures both geometric structure and semantic information, based on given sketch and its corresponding pseudo-labeled satellite image annotations. Formally, let the sketch image be denoted as / TR. Lx x1, and the associated pseudo- labeled satellite image annotations as PSA E RL These two modalities are jointly projected into a structured 3D voxel grid Lx #x1, which encodes the spatial structure of the semantic scene, where L, W, Hl represent the length, width, and height of the 3D space, respectively. The goal is to generate a semantically complete 3D scene by predicting each voxel's 's occupancy state and semantic label. Each voxel in the generated grid is assigned semantic class label 0, 1,2,.... ..... C where C is the total number of semantic categories. By convention, c = corresponds to empty or unoccupied space, while the remaining values represent distinct semantic classes.
4.1 Scene Structure Estimation Network
To facilitate efficient convergence of CymbaDiff, we introduce a scene structure estimation network (SSEN) that produces coarse structural representation of the target 3D scene, as shown in Figure This structural prior guides the diffusion model towards geometrically plausible outputs during early generation steps. Inspired by recent advances in structural scene modeling [65, 75], the SSEN architecture incorporates multi-scale feature extraction modules with Dimensional Decomposition Residual (DDR) blocks. Specifically, multi-scale feature extraction modules capture hierarchical contextual information by aggregating features across multiple receptive fields. It employs parallel branches of progressively stacked and merged at multiple levels, as shown in Figure 3 (b). The DDR structure decomposes a standard & x x 3D convolution into a sequence of three separable layers: 1 1 k 1 x & x 1, and & x 1 x 1, as illustrated in Figure (d). The multi-scale modules capture spatial context and semantically-riee network's representational capacity with limited computational cost. Through joint use of these components, SSEN generates a voxel-based structural representation that accelerates convergence during the diffusion inffusion-driven 3D generation while improving geometric fidelity.
3 convolutions to replace
5 and
x 7 convolutions, which are
features across different receptive fields, while the DDR blocks enhance the
5

---

(4D, L/8. W/B H/8) (160) U16, W/16 H116
Cond Nokes
Down Senn-SSmplnggg
Cy/Membe Block
CylMamba Block
Sampling
CylMamba Block
CylMambaa Block
Conv 3x3x3
CSCB
DDCR
DDCB
Conv
Conv 3x3xx3
Cony 3x3x33
Conv 3x3x3
MLP
MLP
GylMamba Block
CylMaama Block
CylMamba Block
Sowr/oom
Triple Mamba
LayerNomn
Up Sampling
Up Sampling
Ouipur
MLP
DDCB
CylMamba
DDCB
MLP
Mamba
(c) CylMamba Black
(D, L/4, W/4, H/4)
L18D,L/8, W/8, H/8)
CymbaDiff Denosing Network
Fonward
Forward
Convild
SSM
Projection
Backward
Backward
LayerNorm
LayerNorm
Random
Canv iet1x
Corv loxtx
Conv lorxx
Conv lext lor1x1
Conv lofto
SSM
Projection
Activation
(3)
(e) Triple Mamba
Figure 3: Architecture of the CymbaDiff denoising network. CylMamba denotes cylinder mamba
block. Refer to the text for details.
4.2 Variational Autoencoder (VAE) Latent Mapping Network
As illustrated in Figure 2, CymbaDiff operates in the latent space of VAE, which provides compact and informative representation for 3D semantic scenes. The VAE is trained with a combination of cross-entropy loss [76] and Lovász-Softmax loss [77]. This joint objective encourages alignment with the voxel grid manifold while mitigating the blurriness often introduced by conventional voxel-wise
losses (like L2 [78]). Given voxelized input scene V L x W H, the encoder E maps it latent representation
EEV). the decoder T then reconstructs the scene V TD(=) = (VE(V)). In our implementation, E(-) reduces the spatial resolution of the input voxel grid by a factor of f = 4, effectively compressing the scene while preserving key structural features. The VAE encoder consists of two down-sampling blocks, each comprising four consecutive convolutional layers. Every pair of convolutional layers is followed by Batch Normalization layer and ReLU activation function. Following these operations, a downsampling convolutional layer is applied, which is also followed by Batch Normalization and ReLU. To align with the VAE's latent distribution, the latent mapping network is designed to share the same architecture as the encoder.
4.3 Cross-Scale Contextual Block Dilated Decompased Convolution Block
introduce the Cross-Scale Contextual Block (CSCB), inspired by hierarchical receptive fields in VGG [79] and multi-path processing in SCPNet [65]. CSCB efficiently captures local-to-global context from conditioning inputs with minimal memory overhead. Starting with 3 x convolution, it has cascaded multi-covolution blocks (see Figure 3 (b)) with skip connections, and ends with another x 3 x convolution before adding the residual output. Moreover, the Dilated Decomposed Convolution Block (DDCB) employs DDR blocks [75] with varying dilation rates of 1, and 3 to capture diverse contextual features. The DDR structure is shown in Figure 3(d). The DDR block reduces computational cost of C Cout 1- in traditional 3DD convolutions C x Cout x 3k by breaking down the operations into x x k, x x 1, and & 1 x layers, which decreases the parameter count three times while maintaining detailed spatial layout. Therefore, this decomposition significantly reduces the number of parameters while preserving fine-grained spatial layout information.
4.4 CymbaDiff Denoising Network
As shown in Figure (a), CymbaDiff generates scenes from conditional inputs and latent noise, drawing on the Mamba framework [80] model sequences through a state-space formulation. continuous input æzt) R is transformed into output y(t) R via an intermediate hidden state

---

h(t) E R,, before being discretized. The SSMs model [81] is typically formulated using linear ordinary differential equations (ODEs), defined as:
h' (t) Ah (t) + Bz (t), (t) = Ch (t).
(I)
where A E RNNN and B RNX1, C RIXN denote the state matrix, input matrix, and output matrix, respectively. Since deriving the analytical solution for / (1) is often intractable and real-world data is typically discrete, the system is discretized as follows:
h (t) Ah (t - 1) Br (t), y (t) (t),
where A= = exp (A.A) and B (AA) (exp (AA) - AB, are the discretized state parameters and A is the discretization step size. The final output is obtained by applying a global convolution over a structured kernel. The downsampling and upsampling operations follow the design proposed in [25].
Cylinder Mamba Block. A core component of the CymbaDiff denoiser is the cylinder mamba block, illustrated in Figure 3 (c). This block integrates the Triple Mamba module [82] with our proposed cylinder mamba layer design to jointly leverage the advantages of both Cartesian and cylindrical coordinate representations. The Triple Mamba module, based on Cartesian grids, effectively preserves precise geometric distances, critical for modeling local physical neighborhoods. However, adjacent el- ements in Cartesian voxel sequences may misrepresent spatial relationships, limiting the effectiveness of sequential modeling. In contrast, the cylinder mamba layer (0, 2) imposes structured spatial ordering that explicitly captures cylindrical continuity and vertical hierarchy. This ordering provides vehicle-centric, geometrically coherent view, enabling angular-radial semantic tokenisation and supporting long-range context modelling with Mamba, for example, capturing structural information about sidewalks and buildings flanking the road.
The detailed structure of Triple Mamba layer is illustrated in Figure 3()), and the cylinder mamba (C-Mamba) layer adopts the same architecture. Before entering the Mamba layers, input features undergo residual Layer Normalization (L.N) on respective coordinate-based feature representation Le, 2TMB (t) (fN(fT MB (t))) fraB (t) and 2CMB (t) (LN(fcMB (t))) + fCMB (t). fTMB (t) and 2TMB (t) are the input and output features before the Triple Mamba layer, while
fCMB (t) and 2CMB (t) denote the corresponding features before cylinder mamba layer. The
temporal dynamics of the Triple Mamba and C-Mamba layer input are thus governed by:
A (t) = Ah (t - 1) + BRTMB (t), y (t) Ch (1), h (t) Ah (1 - + BECMB (4), (1) Ch
(3)
(4)
The Triple Mamba layer and C-mamba layer apply three separate Mamba modules, each operating on the same input 2TMB B (t) and ZCMB (t) but with distinct ordering strategies: forward (w ) backward (wf)), and random inter-slice (x)f) directions. The output of the 7th Triple Mamba layer and C-mamba layer are computed as:
db (ZTMB (1)) = (2TMB (1) (ETMB (1) + 10' (ZTMB (4)), Wi (ECMB (1)) w/ FCMB (1)) + (SCMB (1)) + vo (#CMB (f)),
(5)
(6)
where 2ty (2TMB (t)) and we (ZCMB (t)) represent the outputs of the 7th triple Mamba and C-mamba layer. Fused 3D features from triple Mamba and C-mamba layers are formulated as wsllt = dell (ZTMB (t)) + will (#CMB (t)), where Sell (2TMB (t))
MLP (LN (1hs (STMB (())) + + (ZTMB (t)) and will (ZCMB (f)) = MLP (LN (wi (ZCMB (f))) wi (ZGMB (l)). dalll (2TMB (I)) and witll (ZCMB (L)) denote the output feature from the triple Mamba and the C-mamba layer. MLP corresponds to stacked linear layers. Note that the in- put features in the C-Mamba layer are sorted by angular, radial, and vertical indices ((O, r, =)), and the output features are mapped back to Cartesian spatial ordering (x, y, 2) (the same ordering in the Triple Mamba layer) and fused with those from the Triple Mamba layer, allowing the model to jointly exploit radial and axis- axis-aligned spatial cues. This joint representation enhances the model's 's ability to learn both local and global 3D spatial structures, capturing both Cartesian and cylindrical representations. Unlike the original Mamba [41, 83), which emphasizes directional context aggregation along scan lines with higher memory usage, our cylinder mamba block is specifically designed to efficiently capture spatially-structe structured 3D information.
7

---

mad
sidewalk
parking
other-grad.
building
car
truck
bicycle
motorcycle
other-veh.
vegelation
terrain
person
bicyclist
motorcyclist
fence
pole
tral. tral.-sign
Figure 4: Qualitative results on the Sketch- based SemanticKITTI validation set. The 1st and 2nd rows show generated scenes conditioned on the corresponding freehand sketch and pseudo-labeled satellite images. The 3rd and 4th rows demonstrate the model's capability to generate moderately diverse 3D scenes (with different details) under identical input conditions.
5 Experiments
Implementation Details. Our model is trained on the Sketch-based SemanticKITTI training split from the SketchSem3D dataset. For evaluation, we use the validation splits both the Sketch- based SemanticKITTI and Sketch-based KITTI-360 subsets, also from SketchSem3D. Following UrfanDiff [9], we train a dedicated network to extract latent features that encode both geometric and semantic information. These features used to compute 3D FID and MMD, providing a joint assessment of generation quality and distributional similarity to ground-truth scenes. Additional implementation details presented in the Appendix.
5.1 3D Semantic Scene Generation and Ablation Study
3D Semantic Scene Generation. As shown in Table 2, we compare our approach with two recent otate-of-the-art baselines, SSD [57] and Semeity [11]. Across all evaluation metrics, our method consistently achieves superior performance. Notably, on the Sketch- bescch SemanticKITTI subset, improves the FID score by approximately 16 points compared Semeity [11], highlighting its effectiveness in interpreting sparse and abstract conditional inputs, such as frechand sketches and pseudo-labeled satellite image annotations.
SSD [57] and Semcity [11] both adopt 2D FID for evaluation. contrast, we adopt more compre- hensive 3D evaluation metrics, 3D FID and MMD, that more accurately assess geometric fidelity and semantic consistency in voxel space. To evaluate the effectiveness of our approach, we replace the CymbaDiff denoising network with two baselines: 3D extension of the Latent Diffusion net- work [1] and the 3D DiT' model [84], and conduct experiments on the SketchSem3D benchmark. Results Table show that our method consistently outperforms both baselines, demonstrating its superior performance in 3D semantic scene generation. For additional context, UrbanDiff [9] reports competitive performance, with 3D FID of 291.4 and 3D MMD of 0.11 on the NuScenes dataset. However, their experimental setting is less challenging, as the voxel resolution of NuScenes 192 x 192 x 16 and with only semantic classes. In comparison, our benchmark dataset has resolution of 256 x 256 x 256 and with 20 classes for the Sketch-based SemanticKITTI subset and 16 classes for the Sketch-based KITTI-360 subset. The primary factor underlying this that UrbanDiff operates solely within the Cartesian coordinate system, leading to the loss of important volumetric structural information. Furthermore, Uifa does not release its source code or preprocessed data, which prevents direct comparison with our proposed task.

---

Table 2: Semantic scene generation results. SK: sketch, PSA: pseudo-labele labeled satellite image annotations. SSD and Semcity: 2D FID.
Table 3: Ablation study on Sketch-
based SemanticKITTI test set. w/o: "without", C-Mamba: mamba.
cylinder
Datasets
Method
Condition
FID 4
MMD
SSD [57]
112.82
Method
FID +
MMD +
Semeity [10]
$6.55
SemanticKITTI
3D Latent Diffusion [T]
SK+P5A
165.05
0.09
wo CSCB
90.53
0,06
3D DIT [84]
SK+PSA
138.86
0.08
w/o DDCB
76.57
0.06
CymbaDDff (ours)
SK+P8A
40.67
0.04
Latent Diffusion [1]
SK+FAA
0.12
w/o C-Mamba
74.09
0.05
KITTI-360
3D DIT [84]
SK+PSA
272.83
0.11
CymbaDiff 40.67
0.04
CynhaDiff (ours)
SK+PSA
107.53
0.08
In Table 2, to evaluate robustness and generalization, we directly applied our model, trained only on Sketch-based SemanticKITTI, to Sketch-based based KITTI-360 without any fine-tuning. During this evaluation, only the overlapping class labels (16 classes) between the two subsets are used. Our model maintains top-tier performance, producing structurally coherent and semantically meaningful 3D scenes. This cross-dataset evaluation highlights the strong generalization capability of our approach.
We present qualitative results on the fketch-based SemanticKITTI validation set in Figure 4. Rows 1 and 2 illustrate the generated semantic scenes conditioned on the input sketches and their corre- sponding PSAs. Rows 3 and 4 present additional generation results using the same input conditions to demonstrate both consistency and moderate diversity in scene synthesis. We see that our model effectively produces structurally accurate, and semantically meaningful 31) scenes that align well with inputs. These visualizations further demonstrate the model's s ability to integrate abstract freehand sketches and pseudo-labeled satellite cues to generate high-quality semantic reconstructions. Some sketch-PSA pairs may have differences because the 3D ground truth annotations in SemanticKITTI were collected around 2013 and the satellite images used for PSA were captured around 2025. PSA generation, being automatic, is also prone to errors. In contrast, sketches originate directly from the 2013 ground-trute -trtt data, maintaining temporal consistency and serving stable spatial reference to mitigate the domain gap.
Observing the results of CymbaDiff on the proposed SketchSem3D dataset, it is apparent that it demonstrates strong performance, effectively handling challenges such as semantic misalignment caused by noisy pscudo-labels, c.g., due to confusion between vegetation and buildings. Nevertheless, this method does occasionally fail to accurately reconstruct small or occluded objects that are underrepresented in the training data or sparsely encoded in the sketch and PSA inputs. Although CymbaDiff mitigates this issue to some extent through the use of the Cross-Scale Contextual Block and Cylinder Mamba Block, which capture multi-scale contextual information, its performance could be further enhanced by increasing the representation of small objects in the dataset.
Ablation Study. We conducted systematic experiments to evaluate the impact of different components in our model and to quantify their individual contributions to the overall performance. As presented in Table 3, the ablation study offers valuable insights into the role and effectiveness of each component. These results allow us to isolate and identify the elements that most significantly enhance the model's performance in the 3D semantic scene generation task. Notably, the CSCB, DDCB, and cylinder mamba blocks play a critical role, as they enable the model to capture complex spatial and semantic relationships within 3D scenes more effectively. "w/o C-Mamba" refers to a variant that retains only the triple Mamba layers.
5.2
3D Semantic Scene Completion.
Since our work explores a new research direction and, currently, there are no directly comparable methods using the same input modalities, we compare CymbaDiff with existing state-of-the-an se- mantic scene completion methods that use monocular or stereo RGB inputs. However, we emphasize that our main contribution lies in 3D scene generation. Table 4 compares our method to 3D scene completion methods on the IoU and mloU metrics reported in their respective publications. All methods are evaluated for 3D semantic scene completion on the SemanticKITTI validation set. The compared methods either use monocular or stereo (image) inputs. Remarkably, despite relying only on input SK and PSA, our method achieves highly competitive performance, matching or exceeding several leading methods that utilize richer input modalities. This demonstrates that SK and PSA offer

---

Table 4: Quantitative results the SemanticKITTI validation set. The best results indicated in bold. Mono and Stereo refer to methods using monocular and stereo inputs, respectively, while SK and PSA denote sketch and pseulo-labeled satellite annotations. Note that demonstrate strong performance using SK+PSA, which are much easier to obtain than images.
sidewalk
parking
other-gma
building
bicyelc
motoreycle
other- veh..
vegetation
terain
person
bicyelist
motoreyclist
traf-sign
road
truck
trunk
fence
pol
Method
Input
loU mloU
car
MonoScene [85]
Mono
11.1
56.5
26.7
14.3
0.5
14.1
7.0
0.5
1.5
29.6
2.3
TPVForer [86]
Mono
35.6
11.3
56.5
25.9
20.6
0.9
13.9
23,8
8.1
0.4
0.1
4.4
16.9
2.3
30.4
0.5
0,9
0.0
5.9
3.1
1.5
NDCC-ccene [87]
Mono
37.2
12.7
59.2
28.2 21.4
1.7
14.9
26.3
14,8
1.7
2.4
7.7
19.1
3.5
31.
3,6
2.7
0.0
6.7
4.5
2.7
OceFormer [88]
Mono
13.5
58.9
19.6
0.3
14,4
25.5
8.5
19.6
3.9
32.6
0.0
2.9
SparseDee [89]
Mono
36.5
13.1
59.6
29.7 20.4
0.5
15,4
24.0
18
0.8
0.9
8.9
18.9
3.5
31,
3.7
0.6
0.0
6.7
3,9
2.6
IAMSSC [60]
Mono
44.3 12.5
54.6
25.9 16.0
0.7
17.4
26.3
8.7
0.6
0.2
5.I
24.6
5.0
30
1.3
3.5
0.0
6.9
6.4
3.6
VoxFormer [90]
13.4
53.6
26.5
19.7
19.5
7.3
7.8
26
33.1
4.9
DepthSSC [91]
Stereo
45.8 13.3
55.4
27.0 18.8
0.9
19.2
25.9
6.0
0.4
1.2
7.5
26.4
4.5
30
2.6
6.3
0.0
8.5
744
4.1
HASSC-S [92]
Stereo
44.8
13.5
57.1
28.3
15.9
1.1
19.1
27.2
9.9
0.9
0.9
5.6
25.5
6.2
32
28
4.7
0.0
6.6
7.7
4.1
H2GFommer [62]
13.7
17.8
19.7
10.0
4.7
CymbaDiff
SK+PSA 43.2 14.6
52.4
33.3
13.1
10.9
32.4
32.1
0.8
1.0
0.0
3.2
28.0
8.7
22.2
4.6
4.9
0.0
1I
12.
52
a flexible alternative, especially when RGB data are unavailable or impractical, such as in remote sensing.
For semantic scene completion, our method achieves 43.2% IoU and 14.6% mloU on the Se- manticKITTI validation set, outperforming the leading monocular baseline by 1.1% mloU and the best stereo-based method by 0.9%. This performance gain underscores the strong representational and generative capabilities of CymbaDiff, particularly in reconstructing large-scale structures such as sidewalks, buildings, vegetation, other-ground, and fences. In addition, our method maintains com- petitive accuracy for smaller objects like people, poles, traffic signs, and tree trunks, demonstrating robustness across a wide range of object sizes and semantic categories. These results collectively highlight the effectiveness and versatility of our approach in diverse urban scene contexts. We present further qualitative examples, including results on underrepresented classes in the Appendix.
Conclusion
We introduced a novel and scalable task: 3D outdoor semantic scene generation from sketches and pseudo-labeled satellite image annotations. This task offers a low-cost and flexible alternative to traditional unnotation-intensive methods, particularly beneficial for applications such as autonomous driving, urban planning. To achieve this, we proposed SketchSem3D, the first publicly available dataset specifically designed for multi-conditioned scene generation in outdoor environments. We proposed CymbaDiff, a liffusion-basee generative model designed to enforce structured spatial coherence by explicitly modeling angular continuity and vertical hierarchies, while preserving physical local and global spatial relationships within 3D scenes. CymbaDiff achieves top-tier performance for 3D scene generation and completion using only sparse and abstract input modalities, establishing a solid baseline for future advancements in this field. We hope our new task, dataset, and approach (including code) would foster advancements in related areas.
Broader Impacts. CymbaDiff model inherently neutral and designed for positive human-centric applications such as urban simulation and autonomous driving, may pose potential societal risks if misused, particularly scenarios involving unauthorized mass surveillance.
Limitations, While CymbaDiff generates high-quality 3D semantic scenes from freehand (like) sketches and pseudo-labeled satellite image annotations (PSA), obtaining authentic human-drawn sketches could further improve its generalizability and effectiveness in practical human-AI interaction tasks. Future work could focus using authentic human-drawn sketches for 3D semantic scene generation.
7 Acknowledgments
This research was supported by the Australian Government through the Australian Research Council's Discovery Projects funding scheme (project # DP240101926). Professor Ajmal Mian is the recipient of an ARC Future Fellowship Award (project # 1121010028) funded by the Australian Government.
10

---

Dr. Naveed Akhtar is recipient of the ARC Discovery Early Career Researcher Award (project # DE230101058), funded by the Australian Government.
References
[1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[2] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15(0):10850-1086, 2023.
[3] Ling Yang. Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Hsuan Yang. Diffusion models: comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-33) 2023.
[4] Hanqun Cao, Cheng Tan, Zhangyang Gao, Yilun Xu, Guangyong Chen, Pheng-Ann Heng, and Stan Z Li. A survey generative diffusion models. IEEE Transactions Knowledge and Data Engineering, 2024.
[5] Yizhi Song, Zhifei Zhang, Zhe Lin, Scott Cohen, Brian Price, Jianming Zhang, Soo Ye Kim, and Daniel Aliaga. Objectstitch: Object compositing with diffusion model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 18310-18319, 2023.
[6] Chenfeng Xu, Huan Ling, Sanja Fidler, and Or Litany. 3difftection: 3d object detection with seometry-awant diffusion features. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10617-10627, 2024.
[7] Siyuan Huang, Zan Wang. Puhao Li, Baoxiong Jia, Tengyu Liu, Yixin Zhu, Wei Liang, and Song-Chur Chun Zhu. Diffusion-based generation, optimization, and planning 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16750-16761, 2023.
[8] Xiaoliang Ju, Zhaoyang Huang, Yijin Li, Guofeng Zhang, Yu Qiao, and Hongsheng Li. Diffindscnne: Diffusion-based high-quality indoor scene generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4526 4535, 2024.
[9] Junge Zhang, Qihang Zhang, Li Zhang, Ramana Rao Kompella, Gaowen Liu, and Bolei Zhou. Urban scene diffusion through semantic occupancy map. arXiv preprint arXiv:2403.1 1697, 2024.
[10] Yuheng Liu, Xinke Li, Xueting Li, Lu Qi, Chongshou Li, and Ming- Haug- Yang. Pyramid diffusion for fine 3d large scene generation. In European Conference Computer Vision, pages 71-87. Springer, 2024.
[11] Jumin Lee, Sebin Lee, Changho Jo, Woobin Im, Juhyeong Seon, and Sung-Bui Yoon. Semcity: Semantic scene generation with triplane diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 28337-28347, 2024.
[12] Alaia Sola, Cristina Corchero, Jaume Salom, and Manel Sanmarti. Simulation tools to build urban-scale energy models: A review. Energies, 11(1)):269, 2018.
[13] Alaia Sola, Cristina Corchero, Jaume Salom, and Manel Sanmarti, Multi-domaii urban-scall energy modelling tools: A review. Sustainahle Cities and Society, 54:101872, 2020.
[14] Xuran Pan, Zhuofan Xia, Shiji Song, Li Erran Li, and Gao Huang. 3d object detection with pointformer. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 7463-7472, 2021.
11

---

[15] Yingwei Li, Adams Wei Yu, Tianjian Meng, Ben Caine, Jiquan Ngiam, Daiyi Peng, Junyang Shen. Yifeng Lu, Denny Zhou, Quoc V Le, al. Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 17182-17191, 2022.
[16] Junbo Yin, Jianbing Shen, Rumnan Chen, Wei Li, Ruigang Yang, Pascal Frossard, and Wenguan Wang. Is-fusion: Instance-scene collaborative fusion for multimodal 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14905-14915, 2024.
[17] Zhiwei Lin, Zhe Liu, Zhongyu Xia, Xinhao Wang, Yongtao Wang, Shengxiang Qi. Yang Dong, Nan Dong, Le Zhang. and Ce Zhu. Rebevdet: radar-camera fusion in bird's eye view for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14928-14937, 2024.
[18] Hengwei Bian, Lingdong Kong. Haozhe Xie, Liang Pan, Yu Qiao, and Ziwei Liu. Dynamice- ity: Large-scall scale 4d occupancy generation from dynamic scenes. The Eleventh International Conference on Learning Representations, 2024.
[19] Lucas Nunes, Rodrigo Marcuzzi, Jens Behley, and Cyrill Stachniss. Towards generating realistic 3d semantic training data for autonomous driving. arXiv preprint arXiv:2503.21449 2025.
[20] Aryan Mikaeili, Or Perel, Mehdi Safaee, Daniel Cohen-Or, and Ali Mahdavi-Amiri. Sked: Sketch-guided text-based 3d editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14607-14619, 2023.
[21] Aditya Sanghi, Pradeep Kumar Jayaraman, Arianna Rampini, Joseph Lambourne, Hooman Shayani, Evan Atherton, and Saeid Asgari Taghanaki. Sketch-a-shape: Zero-shot sketch-to-3d shape generation. arXiv preprint S.Ki::2377.3866 2023.
[22] Zijie Wu, Yaonan Wang, Mingtao Feng. He Xie, and Ajmal Mian. Sketch and text guided diffu- sion model for colored point cloud generation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 8929-8939, 2023.
[23] Zijie Wu, Mingtao Feng. Yaonan Wang, He Xie, Weisheng Dong, Bo Miao, and Ajmal Mian. External knowledge enhanced 3d scene generation from sketch. In European Conference on Computer Vision, pages 286-304. Springer, 2024.
[24] Lianghui Zhu, Bencheng Liao, Qian Zhang. Xinlong Wang. Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. In Forty-first International Conference on Machine Learning.
[25] Zhaohu Xing, Tian Ye, Yijun Yang, Guang Liu, and Lei Zhu. Segmamba: Long-range sequential modeling mamba for 3d medical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 578-588. Springer, 2024.
[26] Jun Ma, Feifci Li, and Bo Wang. U-mamba: Enhancing long-range dependency for biomedical image segmentation. arXiv preprint urXiv:2401.04722 2024.
[27] Jiuming Liu, Ruiji Yu, Yian Wang, Yu Zheng, Tianchen Deng, Weicai Ye, and Hesheng Wang. Point mamba: A novel point cloud backbone based on state space model with octree-based ordering strategy. arXiv preprint arXiv:2403.0644 2024.
[28] Tao Zhang, Haobo Yuan, Lu Qi, Jiangning Zhang, Qianyu Zhou, Shunping Ji, Shuicheng Yan, and Xiangtai Li. Point cloud mamba: Point cloud learning via state space model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 10121-10130, 2025.
[29] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmano, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 2818 2818-2829, 2023.
12

---

[30] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander Berg, Wan Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015-4026, 2023.
[31] Quentin Authony, Yury Tokpanov, Paolo Glorioso, and Beren Millidge. Blackmamba: Mixture of experts for state-space models. arXiv preprint arXiv:2402.01777, 2024.
[32] Wenrui Li, Xiaopeng Hong, and Xiaopeng Fan. Spikemba: Multi-modal spiking saliency mamba for temporal video grounding. arXiv preprint arXiv:2404.01174. 2024.
[33] Ziyang Wang, Jian-Qing Zheng, Yichi Zhang, Ge Cui, and Lei Li. Mamba-unet: Unet-like pure visual mamba for medical image segmentation. arXiv preprint arXiv:2400..05075 2024.
[34] Rui Deng and Tianpei Gu. Cu-mamba: Selective state space models with channel learning for image restoration. In 2024 IEEE 7th International Conference on Multimedia Information Processing and Retrieval (MIPR), pages 328-334. IEEE, 2024.
[35] Hang Guo, Jinmin Li, Tao Dai, Zhihao Ouyang, Xudong Ren, and Shu-Tao Xia. Mambair:
A simple baseline for image restoration with state-space model. In European conference computer vision, pages 222-241. Springer, 2024.
[36] Shida Wang and Qianxiao Li. Stablessm: Alleviating the curse of memory in state-space models through stable reparameterization. arXiv preprint drXiv:2311.14495, 2023.
[37] Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, et al. An empirical study of mamba-base based language models. arXiv preprint ar wxXiv2246.0078877 2024.
[38] Dingkang Liang, Xin Zhou, Wei Xu, Xingkui Zhu, Zhikang Zou, Xiaoqing Ye, Xiao Tan, and Xiang Bai. Pointmamba: A simple state space model for point cloud analysis. arXiv preprint urXiv:2402.1 10739, 2024.
[39] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu. Vmamba: Visual state space model. Advances neural information processing systems, 37:103031-103063, 2024.
[40] Eric Nguyen, Karan Goel, Albert Gu, Gordon Downs, Preey Shah. Tri Dao, Stephen Baccus, and Christopher Ré. S4nd: Modeling images and videos as multidimensional signals with state spaces. Advances in neural information processing systems, 35:2846-2888 22611, 2022.
[41] Shufan Li, Harkanwar Singh, and Aditya Grover. Mamba-nd: Selective state space modeling for multi-dimensional data. arXiv preprint XX2::22022.08822, 2024.
[42] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[43] Diederik P Kingma, Max Welling, al. Auto-encoding variational bayes, 2013.
[44] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-66 6851, 2020.
[45] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- 1-to3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9298-9309, 2023.
[46] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, et al. Magic One image to high-quality 3d object generation using both 2d and 3d diffusion priors. arXiv preprint arXiv:2306. 2306.17333, 2023.
[47] Xuanchi Ren, Jiahui Huang, Xiaohui Zeng, Ken Museth, Sanja Fidler, and Francis Williams. Xcube: Large-scale 3d generative modeling using sparse voxel hierarchies. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4209-4219, 2024.
13

---

[48] Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang. Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3d latents for scalable and versatile 3d generation. arXiv preprini arXiv:2412.01506, 2024.
[49] Chao Xu, Ang Li, Linghao Chen, Yulin Liu, Ruoxi Shi, Hao Su, and Minghua Liu. Sparp: Fast 3d object reconstruction and pose estimation from sparse views. In European Conference on Computer Vision. pages 143-163. Springer, 2024.
[50] Guandao Yang. Xun Huang, Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4541-4550, 2019.
[51] Bo Miao, Mohammed Bennamoun, Yongsheng Gao, and Ajmal Mian. Spectrum-guidde multi- granularity referring video object segmentation. Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 920-930, October 2023.
[52] Bo Miao, Mingtao Feng, Zijie Wu, Mohammed Bennamoun, Yongsheng Gao, and Ajmal Mian. Referring human pose and mask estimation in the wild. Advances in Neural Information Processing Systems, 37:44791 448133, 2024.
[53] Alexey Bokhovkin, Quan Meng, Shubham Tulsiani, and Angela Dai. Scenefactor: Factored latent 3d diffusion for controllable 3d scene generation. arXiv preprint arXiv:2412.01881, 2024.
[54] Chuan Fang, Yuan Dong, Kunming Luo, Xiaotao Hu, Rakesh Shrestha, and Ping Tan. Ctrl- room: controllable text-to-3d room meshes generation with layout constraints. arXiv preprint arXiv:2310.03362, 2023.
[55] Jiapeng Tang, Yinyu Nie. Lev Markhasin, Angela Dai, Justus Thies, and Matthias Nießner. Diffuscene: Denoising diffusion models for generative indoor scene synthesis. Proceedings of the IEEF/CVF conference computer vision and paltern recognition, pages 20507-205 88, 2024.
[56] Xiuyu Yang. Yunze Man, Junkun Chen, and Yu-Xiong Wang. Scenecraft: Layout- guided 3d scene generation. Advances in Neural Information Processing Systems, 37:82060-82088 2024.
[57] Jumin Lee, Woobin Im, Sebin Lee, and Sung- Eui Yoon. Diffusion probabilistic models for scene-scale 3d categorical data. arXiv preprint arXiv:2301.0055 2023.
[58] Quan Meng, Lei Li, Matthias Nießner, and Angela Dai. Lt3sd: Latent trees for 3d scene diffusion. arXiv preprint arXiv:2409.082155 2024.
[59] Zhu Yu, Runming Zhang, Jiacheng Ying, Junchen Yu, Xiaohai Hu, Lun Luo, Siyuan Cao, and Huiliang Shen. Context and geometry aware voxel transformer for semantic scene completion. arXiv preprint arXiv:2405. 1 3675,, 2024.
[60] Haihong Xiao, Hongbin Xu, Wenxiong Kang, and Yuqiong Li. Instance-aware monocular 3d semantic scene completion. IEEE T-ITS, 2024.
[61] Li Liang, Naveed Akhtar, Jordan Vice, Xiangrui Kong, and Ajmal Saeed Mian. Skip mamba diffusion for monocular 3d semantic scene completion. arXiv preprint arXiv:2501.07260, 2025.
[62] Yu Wang and Chao Tong. H2gformer: Horizontal-to globba voxel transformer for 3d semantic scene completion. In AAAI, volume 38, pages 5722-5730, 2024.
[63] Yuwen Xiong, Wei-Chiu Ma, Jingkang Wang, and Raquel Urtasun. Learning compact represen- tations for lidar completion and generation. In CVPR, pages 1074-1083, 2023.
[64] Ran Cheng, Christopher Agia, Yuan Ren, Xinhai Li, and Liu Bingbing. S3cnet: A sparse semantic scene completion network for lidar point clouds. In CoRL, pages 2148-2161 2021.
[65] Zhaoyang Xia, Youquan Liu, Xin Li, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, and Yu Qiao. Scpnet: Semantic scene completion on point cloud. In Proceedings of the IEFF/CVF conference computer vision and pattern recognition, pages 17642-17651, 2023.
14

---

[66] Yingjie Cai, Xuesong Chen, Chao Zhang, Kwan- Yee Lin, Xiaogang Wang, and Hongsheng Li. Semantic scene completion via integrating instances and scene in-the-loop. In CVPR, pages 924-333, 2021.
[67] Xuzhi Wang, Di Lin, and Liang Wan. Ffnet: Frequency fusion network for semantic scene completion. In AAAI, volume 36, pages 2550-2557, 2022.
[68] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall. Semantickitti: A dataset for semantic scene understanding of lidar sequences. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9297-9307, 2019.
[69] Yiming Li, Sihang Li, Xinhao Liu, Moonjun Gong, Kenan Li, Nuo Chen, Zijun Wang, Zhiheng Li, Tao Jiang, Fisher Yu, et al. Sscbench: Monocular 3d semantic scene completion benchmark in street views. 2023.
[70] Yibo Li and Bailun Liu. Improved edge detection algorithm for canny operator. In 2022 IEEE 10th Joint International Information Technology and Artificial Intelligence Conference (ITAIC), volume 10, pages 1-5. IEEE, 2022.
[71] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[72] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pages
IEEE, 2012.
[73] Yiyi Liao, Jun Xie, and Andreas Geiger. Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3292-3310, 2022.
[74] Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, and Sanja Fidler. Atiss: Autoregressive transformers for indoor scene synthesis. Advances in Neural Information Processing Systems, 34: 14:12003-12000 2021.
[75] Jie Li, Yu Liu, Dong Gong, Qinfeng Shi, Xia Yuan, Chunxia Zhao, and Ian Reid. Rgbd based dimensional decomposition residual network for 3d semantic scene completion. In CVPR, pages 7693-7702, 2019.
[76] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
[77] Maxim Berman, Amal Rannen Triki, and Matthew B Blaschko. The lovász-softmax loss: A tractable surrogate for the optimization of the onersection-over-une measure in neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 4421, 2018. 8.
[78] Batia Laufer and Stig Fliasson. What causes avoidance 12 learning: 1.1-12 difference, 11-12 similarity, or 12 complexity? Studies in second language acquisition, 15(1):35 -88, 1993.
[79] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409, 1e09.4409555555 1556, 2014.
[80] Albert Gu and Tri Dao. Mamba: I Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:23 2023.
[81] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2/11.00399 2021.
[82] Yijun Yang, Zhaohu Xing, and Lei Zhu. Vivim: video vision mamba for medical video object segmentation, arXiv preprint 4rXiv:2401.14168, 2024.
15

---

[83] Lianghui Zhu, Bencheng Liao, Qian Zhang. Xinlong Wang. Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprini arXiv:2401.09417 2024.
[84] Shentong Mo, Enze Xie, Ruihang Chu, Lanqing Hong, Matthias Niessner, and Zhenguo Li. Dit-3d: Exploring plain diffusion transformers for 3d shape generation. Advances in neural information processing systems, 36:67960-6799 17971, 2023.
[85] Anh-Quan Cao and Raoul De Charette. Monoscene: Monocular 3d semantic scene completion. In The IEEE/CVF Conference: CVPR, pages 3991-4001, 2022.
[86] Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Tri-perspective view for vision-based 3d semantic occupancy prediction. The IEEE/CVF Conference: CVPR, pages 9223 9232, 2023.
[87] Jiawei Yao, Chuming Li, Keqiang Sun, Yingjie Cai, Hao Li, Wanli Ouyang, and Hongsheng Li. Ndc-scene: Boost monocular 3d semantic scene completion in normalized device coordinates space. In The IEEE/CVF Conference: ICCV, pages 9421-9433 2023.
[88] Yunpeng Zhang, Zheng Zhu, and Dalong Du. Occformer: Dual-path transformer for vision- based 3d semantic occupancy prediction. The IEEE/CVF Conference: ICCV, pages 9433-
9443, 2023.
[89] Pin Tang. Zhongdao Wang, Guoqing Wang, Jilai Zheng, Xiangxuan Ren, Bailan Feng, and Chao Ma. Sparseocc: Rethinking sparse latent representation for vision-based semantic occupancy prediction. In The IEEE/CVF Conference: CVPR, pages 15035-15044, 2024.
[90] Yiming Li, Zhiding Yu, Christopher Choy, Chaowei Xiao, Jose M Alvarez, Sanja Fidler, Chen Feng, and Anima Anandkumar. Voxformer: Sparse voxel transformer for camera-based 3d semantic scene completion. The IEEE/CVF Conference: CVPR, pages 9087-9098, 2023.
[91] Jiawei Yao and Jusheng Zhang. Depthssc: Depth-spatial alignment and dynamic voxel resolution for monocular 3d semantic scene completion. arXiv preprint 2rXiv:2311.17088 2023.
[92] Song Wang, Jiawei Yu, Wentong Li, Wenyu Liu, Xiaolu Liu, Junbo Chen, and Jianke Zhu. Not all voxels are equal: Hardness-awes aware semantic scene completion with self-distillation. In The IEEE/CVT Conference: CVPR, pages 14792-14801, 2024.
[93] Haozhe Xie, Zhaoxi Chen, Fangzhou Hong, and Ziwei Liu. Citydreamer: Compositional generative model of unbounded 3d cities. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9666 9675, 2024.
16

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the claims, which are supported by our experimental results. See Abstract and Sec. 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
The paper does not contain any theoretical assumptions or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We report the dataset, model, and training details in Sec. 3, Sec. 4, Sec. 5, and the Appendix. The introduced dataset and code publicly available.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification:
The introduced dataset and code are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
bublic/guiies..e
for more details.
While we encourage the release code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /oodeSubii
/
htlpppullluiiieeeeeeeepppppppppe / ssionPolicy iy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
See Sec. 5 and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We follow the usual format used in previous related works to report and compare the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative enror rates).
If error bars are reported tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : htt///eerrppp.pp// cttpcc/s.......pppppooooo..ccc wwtppbccc//caeeeccccceecceeaeeee ctthei//iehcctuieeeeeecceeeeeeee ines?
Answer: [Yes]
Justification:
The research conducted in the paper conforms with NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special considera- tion due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Sec. 6.
Guidelines:
The answer NA means that there is societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance). fairness considerations (e-g., deployment technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, Iet alone deployments. However, if there direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mecha- nisms for monitoring misuse, mechanisms monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
[nswer:[NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the creators or original papers of all the related code, data, and models used in this paper. All the assets are free for research studies and widely used in previous related works. We also provide the licenses of the used datasets in the Appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the pack- age should be provided. For popular datasets, paperswi swi thcode. com/ /ndddataeet has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
21

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. available online, the authors are encouraged reach the
If this information
asset's ss creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We report the dataset, model, and training details in Sec. 3, Sec. 4, Sec. 5, and the Appendix. The introduced dataset and code are publicly available.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We did not use crowdsourcing conduct research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We did not use crowdsourcing or conduct research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22

---

A. Appendix
A.1 Training Objective
Due to the complexity of the task, our training objective combines multiple loss terms, For the 31) VAE, the objective is:
L == LCE + + _D (4o (2)2))1
where Y 1.0 and 3 0.001 balance the contributions of each loss component, Lee and Lovass denote the standard cross- entoopp and Lovasz-Softmax losses, respectively, following SCPNet [64]. DKL denotes the Kullback- Kulbback-Leileer Divergence between the approximate posterior 9e (2|x)) and the prior p (z), similar to the Latent Diffusion Model (LDM) [1]. The objective function for the
CymbaDiff denoising network follows the LDM [1], minimizing the expected squared error between the predicted noise and true noise:
[Iee co (z,, t]|
CLOM Bo. N(,.),
(8)
where 6 (0( 1) denotes serfformll-weigheee denoising autoencoder applied across time steps t = .. At each step t, the model predicts a denoised estimate of the input 2t, which is a noise-corrupted version of the original input
A.2 Evaluation Metries
To evaluate the quality and diversity of the generated 3D semantic scenes, we use two widely used metrics: Fréchet Inception Distance (FID)[74] and Maximum Mean Discrepancy (MMD) Together, these metries capture both the statistical similarity and feature-level fidelity between the generated and real data, providing a comprehensive assessment of generative performance. Specifically, FID measures the similarity between the distributions of generated and real samples in latent feature space. Formally, FID is defined as:
FID = || - M, 1 TT (C.-C,-2(CCC))
where (Me, M2) and (C,, C2)) are the mean and covariance of the real and generated feature distri- butions. MMD is a non-parametric. kernel-based metric that quantifies the distance between two probability distributions. Unlike FID, MMD does not rely the assumption that features follow Gaussian distribution, making it suitable for evaluating generative models under more flexible conditions. our case, MMD computed using Gaussian kernel applied to features extracted from the same latent space as used for FID. The formal definition of MMD is:
MMD? (x, = = [A: (x,,x')
+ E.y''' [ (v..v))
2 [k (I,
(10)
where X {a), 32, T2,,.., .... Fm and Y (y,, y2, --. Mm denote the sets of latent features extracted from real and generated 3D scenes, respectively.
A.3 Additional Implementation Details
All experiments were conducted on single NVIDIA GefFrce RTX 4090 GPU with 24 GB of RAM. The Variational Autoencoder (VAE) was trained for 22 epochs using the AdamW optimizer with an initial learning rate of 3e-4. The VAE and the CymbaDiff denoising network were trained with a batch size 2 and 4, each occupying approximately 20 GB of GPU memory. The CymbaDiff denoiser was trained for 31 epochs using the AdamW optimizer with learning rate of 1e-3 and weight decay of le-4. The number of denoising steps in CymbaDiff was set to 100. A WarmupCosineLL scheduler was used in all training stages to gradually decrease the learning rate, which helped ensure stable convergence.
A.4 VAE Results
Our CymbaDiff denosing network operates in the latent space of VAE. To ensure high-quality semantic scene generation, this VAE needs be accurate. We report the performance of the proposed VAE on the SemanticKITTI validation set in Table 5.
23

---

Table 5: VAE reconstruction performance SemanticKITTI validation set. IoU and mloU denote Intersection over Union and mean Intersection over Union, respectively.
Model| Original Spatial Size Latent Spatial Size Latent Channel training epoch batch size
loU
mloU
VAE
256 x 256 32
64 x 64 x 8
22
2
192.1
92.0
Ground Truth
CymbaDiff
MonoScene
OccFormer
VoxFormer
mad vegetation
sidewalk
parking temrain
other-grad.
building
car
track
bicycle pole
motorcycle sign
oth
trunk
bicyclist
motorcyclist
fence
Figure 5: Qualitative results on the SemTnticKITTI validation set. Columns from the left represent ground truth, and outputs of CymbaDiff (our method), MonoScene, OceFormer, and VoxFormer.
A.5 Effieicency Comparison
provide quantitative comparisons in the Table 6 across methods in terms of parameter count and runtime performance. These results demonstrate that CymbaDiff achieves a favorable trade-off between model efficiency and computational cost, offering competitive performance with significantly fewer parameters compared to these two generative models.
A.6 Cross-domain Test
We have now trained SemCity[11] and CityDreamer(93]or Jon the SketchSem3D dataset to compare with our CymbaDiff. To ensure compatibility with our 3D voxel- roxel-based setup. we integrated their denoisers into our framework. We also attempted to train the full SemCity pipeline directly, but resulted in unstable training, with the VAE loss diverging to NaN, an issue also reported by other users SemCity's official GitHub page. Please note, CityDreamer is designed for 2D generation and cannot be directly applied to 3D voxel scenes. As shown in the Table 7, CymbaDiff consistently outperforms both baselines across all evaluation metries strongly.
The reason why Semeity and CityDre do not perform well in our experiments is their denoisers (provided in their official GitHub repositories). The denoiser in SemCity only has convolutional and linear layers, whereas that in CityDreamer relies on simple stacking of transformer layers. Although transformer layers can model long-range dependencies, such simplified designs may be suboptimal for large-scale 3D voxel scene generation, where sparse and irregular data demand specialized mechanisms to effectively capture both local geometry and relevant global context.
A.7 Qualitative results on 3D Semantic Scene Completion
To demonstrate the effectiveness of our proposed framework for 3D semantic scene completion, we present additional qualitative results in Figures 5. The figure displays representative examples randomly selected from the SemanticKITTI validation set [68]. CymbaDiff accurately delineates fine-gre fine-grained boundaries of 3D scenes and objects by incorporating the cylinder Mamba blocks, which

---

Table Efficiency comparison. M: Million, and S: seconds.
Input Modality
Parameters (M)
Inference Times (S)
3D DIT
195
4.5
3D Latent Diffusion
1265
11.4
CymbaDiff
7.2
Table 7: Cross-domain Comparison
Sketch-bused
Sketch-based
Sketch-bused
Sketch-based
Method
SemanticKITTI FID SemanticKITTI MMD + KITTI-360 FID
KITTI-360 MMD
3D SemCity[1
987.91
0.26
740.09
0.25
3D CityDreamer(93)
950.16
0.26
754.47
0.25
CymbaDiff
40.67
0.04
107.53
0.08
promotes structured spatial coherence through explicit modeling of angular continuity and vertical hierarchies.
A.8 Licenses
Licenses of SemanticKITTI and SSCBench KITT1-360. The SemanticKITTI dataset is licensed under the CC BY-NC-SA 4.0, while the SSCBench KITTI-360 dataset is released under CC BY-NC- SA 3.0 license.
Terms of Use and License of SketchSem3D, The SketchSem3D dataset is licensed under CC BY-NC-SA 4.0.
25


Reviews:
[{"review_id": "1CN2tMmrsu", "reviewer": "Reviewer_4gZF", "summary": "The paper introduces a dataset of outdoor scenes with paired sketches, semantic maps, and semantic voxels. It presents a pipeline that automatically constructs the dataset from existing semantic outdoor datasets, ground-truth semantic voxels and satellite imagery. The resulting dataset is approximately 2.7 times larger than existing BEV-conditioned datasets and supports more flexible conditioning. Furthermore, the authors propose an LDM model with Cylinder Mamba diffusion to preserve spatial relationships for conditioned 3D scene generation.", "questions": "1. Section 4 lacks sufficient details about key components. For the CSCB and DDR blocks, additional architectural tables or figures in the supplementary would have been helpful. The details regarding the cylinder mamba blocks is also quite limited. Such as the functionality and architectural details of the triple mamba mechanism as well as the ordering mechanism in the cylinder mamba block.\n\n2. The diffusion model is conditioned on both PSA and sketches. However, the necessity of the sketch is unclear since the PSA alone should provide sufficient information for 3D scene generation. A more user friendly approach could involve a separate network to predict PSA from sketches. Finally, it's unclear to me why the pseudo labels from satellite images are needed as opposed to directly using the BEV projections of the semantic voxels.\n\n3. The paper also lacks relevant conditioned scene generation baselines in table 2. As 3D latent diffusion and 3D DiT were not designed for 3D scene generation specifically. For example, the proposed method could have been run on NuScenes using BEV images as input to compare against UrbanDiff. This would also help understand whether the proposed task is easier or harder compared to BEV conditioned NuScenes. BlockFusion [1] is also another potential baseline that can be compared.\n\n4. I found the 3D semantic scene completion results in table 4 to not be very informative. As the other methods takes monocular or stereo inputs, while the proposed method uses SK+PSA. SK+PSA may offer an easier setting as the semantic labels are directly provided to the model, whereas working with raw images is more challenging and requires stronger feature extractors. As a result, making direct comparisons with other methods less meaningful.\n\nI think there is value in establishing unified datasets and benchmarks for evaluating 3D scene generation as well as offer more flexible inputs. If the above concerns can be addressed during the rebuttal I am willing to raise my rating.\n\n[1] Wu, Zhennan, et al. \"Blockfusion: Expandable 3d scene generation using latent tri-plane extrapolation.\" ACM Transactions on Graphics (TOG) 43.4 (2024): 1-17.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\nThe dataset is larger than previous BEV based NuScenes and provides more flexible conditioning. The description of the dataset construction is clear and the model also show better quantitative results with respect to the chosen baselines.\n\n## Weaknesses\n\nOn the writing side, the clarity of the proposed method can be improved by giving more detail regarding the key components of the model. I also have concerns about the dataset, particularly the necessity of including both sketches and pseudo-labeled annotations as inputs. Additionally, the paper also lacks some more baselines that would make the evaluation more complete, and the validity of the semantic scene completion comparisons in section 5.2 is questionable. More details are given in the questions below.", "was_revised": true, "final_justification": "I have raised my rating as the authors have addressed my concerns regarding the experiments. However, the model section requires additional clarifications and details that the authors have promised during the rebuttal.", "rebuttal": "We sincerely thank Reviewer 4gZF for their comments and appreciation of our work, especially noting our dataset's size, flexibility, and strong baseline performance. In response to the concerns expressed in Weaknesses and Questions, we provide the following answers:\n\n> Supplementary architectural figures for CSCB and DDR blocks are needed. Details on the Cylinder Mamba, including the triple Mamba mechanism and its ordering process, are also insufficient.\n\nThank you for the constructive comment. As shown in Figure 3, the Cylinder Mamba block integrates the Triple Mamba and C-Mamba layers to enhance representational capacity by combining Cartesian and cylindrical spatial representations. The Triple Mamba layer models feature dependencies along forward, backward, and inter-slice directions in Cartesian space. Although architecturally identical to the C-Mamba layer, the detailed structure of C-Mamba is explicitly illustrated in Figure 3 for clarity. The key distinction lies in the input domain: Triple Mamba operates on Cartesian grids, whereas C-Mamba processes data in cylindrical coordinates.\n\nIn the C-Mamba layer, inputs are transformed into cylindrical coordinates and sorted by angular, radial, and vertical indices $(\\theta, r, z)$ to align with the egocentric geometry of street-view scenes, enabling directional context modeling. After processing, features are mapped back to Cartesian space (sorted according to $(x, y, z)$) and fused with those from the Triple Mamba layer, allowing the model to jointly exploit radial and axis-aligned spatial cues.\n\nWe will further improve clarity by adding architectural figures and more details on the triple Mamba mechanism and its ordering process in the final version. Please note, our source code will also be released to facilitate comprehensive understanding of our method.\n\n> The necessity of incorporating sketches remains unclear; PSA can be predicted from sketches. Why not directly utilize the BEV projections of semantic voxels?\n\nThank you for the comment. Sketches offer clear and structured spatial layouts that effectively complement pseudo-labeled satellite annotations (PSA). Since PSA is generated by aligning recent satellite imagery with text prompts, it is susceptible to noise and spatial inconsistencies, particularly due to the temporal discrepancy between contemporary satellite data (e.g., from 2025) and the older LiDAR-based ground truth in SemanticKITTI (2013). Incorporating sketches mitigates these issues by providing reliable spatial priors, and jointly using both inputs enables more flexible and robust conditioning, thereby enhancing spatial accuracy. \n\nWe conducted an ablation study by training CymbaDiff using only PSA inputs, which required code adjustments and excluded sketch cues. As shown in the table below, the resulting performance drop underscores the essential role of sketch guidance in enhancing 3D scene generation quality and fidelity.\n\n| Input Modality       | Dataset        | 3D FID ↓ | 3D MMD ↓ |\n|----------------------|----------------|----------|----------|\n| PSA only   (Ours)          | SemanticKITTI  | 83.31    | 0.06     |\n| PSA + Sketch (Ours)  | SemanticKITTI  | 40.67    | 0.04     |\n| PSA only     (Ours)        | KITTI-360      | 155.62   | 0.08     |\n| PSA + Sketch (Ours)  | KITTI-360      | 107.53   | 0.08     |\n\nPredicting PSA from sketches remains an underexplored direction, which we plan to investigate in future work using the annotations provided in our benchmark. Notably, we deliberately refrain from using BEV projections of semantic voxels, as dense 3D annotations are often unavailable in practical settings. In contrast, satellite imagery serves as a scalable and accessible source of weak supervision, making our approach more broadly applicable under realistic data constraints.\n\n> The method could be tested on BEV-based NuScenes to compare with UrbanDiff and assess task difficulty, and provide BlockFusion results on SketchSem3D.\n\nWe selected 3D Latent Diffusion and 3D-DiT as baselines due to the limited availability of open-source 3D scene generation models. Although not originally designed for semantic voxel generation, both models have shown strong performance in image and 3D object synthesis, and their general architectures can be effectively adapted to the SketchSem3D dataset to establish strong baselines.\n\nThe pre-processed BEV-conditioned NuScenes dataset used in UrbanDiff is not publicly available, so we cannot train the proposed CymbaDiff on this dataset. To address the reviewer's concern, we directly applied our model, trained solely on the Sketch-based SemanticKITTI dataset, to the Sketch-based nuScenes validation dataset without any fine-tuning. Evaluation on the nine overlapping semantic classes shared between the two datasets shows that our CymbaDiff achieves a 3D FID of 183.37 and a 3D MMD of 0.09, outperforming the popular UrbanDiff model on the BEV-conditioned NuScenes dataset that achieves 3D FID and 3D MMD scores of 291.4 and 0.11 (lower is better). In addition, the proposed SketchSem3D dataset offers a more practical alternative to BEV-conditioned NuScenes. We will make this dataset publicly available to facilitate future research.\n\nBlockFusion serves as a strong baseline for 3D scene generation. However, since it was originally designed as a 2D tri-plane denoiser, adapting it to our 3D latent space led to unstable training and suboptimal performance, with the loss failing to converge. \n\n> Table 4 is less informative, as SK+PSA offers an easier setting than raw-image-based methods.\n\nOur work explores a new research direction, and currently, there are no directly comparable methods using the same input modalities. For this reason, we compare CymbaDiff with existing methods that use monocular or stereo RGB inputs, which are common in semantic scene completion. These comparisons highlight the differences, gaps, and challenges between input modalities, as our approach is based on sketches (SK) and pseudo-labeled satellite image annotations\n(PSA) rather than RGB images. While stereo images can provide geometric depth cues for 3D reconstruction, our results demonstrate that SK and PSA inputs offer a flexible and effective alternative, particularly in scenarios where conventional RGB inputs are unavailable or impractical, such as remote sensing. In the final version, we will provide further explanation to clarify these distinctions and prevent possible misinterpretation.\n\nWe appreciate the reviewer mentioning their willingness to raise the rating, and hope that the above responses help in that regard. We look forward to discuss further in the discussion phase, if needed."}, {"review_id": "y4M73NwmSN", "reviewer": "Reviewer_fZuf", "summary": "The paper proposes a sketch-to-3D scene generation dataset to enable the outdoor generation of 3D scenes from sketch drawings, based on the KITTI-360 and SemanticKITTI datasets. Together with the dataset, a 3D scene diffusion method is also proposed that leverages a state-space formulation trained with the sketch-to-3D scene generation dataset.", "questions": "- Why do the sketches and pseudolabels (Fig. 4) not always seem to be consistent/related?\n- During the CLIP pseudolabels generation, are the semantic classes present in the 3D GT semantics used to filter out wrong CLIP classes?\n- Are the textual descriptions (Fig. 1) also available within the dataset?", "limitations": "The main limitation that I see is that the sketches are too precise to be considered as human drawings. However, this limitation is also mentioned in the paper. Still, the dataset is a first step towards generating a 3D scene from human-drawn sketches.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper's main strength lies in both the dataset, which includes paired sketches, BEV pseudolabels, and 3D semantic scenes, and the model architecture that generates scenes conditioned on the sketch images. The dataset is well-built, and the task of scene generation conditioned on a sketch is useful, as sketching a scene provides an easy interface between the user and the generated scene. The model leverages 2D sketches, upsampling them to 3D, which enables the conditioning and generation of the corresponding 3D scene.\n\nThe paper's main weakness is regarding the clarity of the paper. The dataset is well-explained, with details about the sketch and pseudolabel generation. However, the method (Sec. 4) lacks details. Sec. Sections 4.1 and 4.2 do not provide details about the Scene Structure Estimation Network, nor do they offer any information about the Latent Mapping network. Additionally, the diffusion network is built upon state-space formulation works (Mamba). In Sec. 4.4 the variables x(t) , y(t) and h(t) are introduced but those variables are never linked to the z_TMB and f_TMB variables afterwards. For clarity, I suggest adding a brief explanation of the state-space formulation and then linking it to the z_TMB and f_TMB variables. I see the value in the method, but a clearer explanation would be helpful, as it would also strengthen the contributions in the paper.\n\nSome minor concerns are that, from lines 286 to 299, it was unclear which table the text was referring to. Also, the paired sketch and pseudolabels in Fig. 4 seem to be from different scenes. They seem unrelated most of the time. Lastly, I would suggest mentioning other recent scene generation methods, such as \"DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenes\" and \"Towards Generating Realistic 3D Semantic Training Data for Autonomous Driving\", for completeness of the related works.", "was_revised": true, "final_justification": "The authors have addressed and clarified my concerns in their rebuttal. Still, the mathematical notations and explanations in their rebuttal should be added to the main paper to enhance the clarity of the methodology section. I suggest adding it to the main paper, and in case space is needed, the authors can consider moving Figure 3 (the architecture) to the supplement. I see the mathematical explanation as more relevant to be in the main paper rather than the architecture figure. Therefore, I have raised my rating since the rebuttal clarified my concerns.", "rebuttal": "We sincerely thank Reviewer fZuf for their comments and appreciation of our work, particularly highlighting the well-designed dataset and model as the main strengths, which enable effective 3D scene generation from simple 2D sketch inputs. In response to the concerns expressed in Weaknesses and Questions, we provide the following answers:\n\n> Section 4 lacks details on the Scene Structure Estimation and Latent Mapping networks.\n\nIn Section 4.1, the Scene Structure Estimation Network (SSEN) incorporates multi-scale feature extraction modules along with several Dimensional Decomposition Residual (DDR) blocks with various dilations to effectively capture spatial structures and rich contextual information across varying receptive fields.\n\nSpecifically, multi-scale feature extraction modules capture hierarchical contextual information by aggregating features across multiple receptive fields. It employs parallel branches of $3 \\times 3 \\times 3$ convolutions to replace $5 \\times 5 \\times 5$ and $7 \\times 7 \\times 7$ convolutions, which are progressively stacked and merged at multiple levels. Residual connections are integrated throughout the block to facilitate feature reuse and gradient flow.\n\nThe Dimensional Decomposition Residual (DDR) blocks begin with a $1 \\times 1 \\times 1$ convolution for channel adjustment, followed by a series of three directional convolutions: $1 \\times 1 \\times 3$, $1 \\times 3 \\times 1$, and $3 \\times 1 \\times 1$, which capture contextual dependencies along the depth, height, and width dimensions, respectively. The outputs of these directional convolutions are then fused through an additional $1 \\times 1 \\times 1$ convolution. A residual connection is incorporated by adding the input feature map to the fused output.\n\nThe Latent Mapping Network in Section 4.2 adopts the same architecture as the Encoder of the Variational Autoencoder (VAE). The VAE encoder consists of two down-sampling blocks, each comprising four consecutive convolutional layers. Every pair of convolutional layers is followed by a Batch Normalization layer and a ReLU activation function. Following these operations, a downsampling convolutional layer is applied, which is also followed by Batch Normalization and ReLU. We will add this architectural information in the camera-ready version and provide a corresponding figure to clearly depict the structure of the SSEN and Latent Mapping Network. We would also like to emphasize that we intend to make our code public to ensure a comprehensive understanding of our technique. \n\n> Section 4.4 introduces x(t), y(t), and h(t), but does not link them to z_TMB and f_TMB; a brief explanation and connection are needed.\n\nThank you for the constructive comment. Advanced state space models (SSMs), such as the Structured State Space Sequence Model (S4) and Mamba, represent a class of systems that map a one-dimensional continuous input sequence $x(t) \\in \\mathbb{R}$ to an output $y(t) \\in \\mathbb{R}$ via a hidden state $h(t) \\in \\mathbb{R}^{N}$. These models are typically formulated using linear ordinary differential equations (ODEs), defined as:\n\n$h^{\\prime}(t) = Ah(t) + Bx(t), \\quad y(t) = Ch(t),$\n\nwhere $A\\in \\mathbb{R}^{N \\times N}$ and$B\\in \\mathbb{R}^{N \\times 1}$, $C\\in  \\mathbb{R}^{1 \\times N}$ denote the state matrix, input matrix, and output matrix, respectively. Since deriving the analytical solution for $h(t)$ is often intractable and real-world data is typically discrete, we discretize the system as follows:\n\n$h(t) = \\overline{A}h(t-1)  + \\overline{B} x(t), \\quad y(t) = Ch(t),$\n\nwhere  $\\overline{A}= exp\\left (\\triangle A \\right )$ and $\\overline{B}=\\left ( \\triangle  A \\right )^{-1} \\left ( exp\\left ( \\triangle  A \\right ) - I  \\right ) \\cdot \\triangle B $ are the discretized state parameters and $\\triangle$ is the discretization step size. In the context of our Triple Mamba layer, the input $x(t)$ corresponds to a residual-enhanced representation:\n\n$z_{TMB}(t) =LN(f_{TMB}(t))  + f_{TMB}(t),$\n\nwhere $f_{TMB}(t)$ is the raw input features prior to residual layer normalization $LN$. The temporal dynamics of the Mamba input are thus governed by:\n\n$h(t) = \\overline{A}h(t-1)  + \\overline{B} z_{TMB}(t), \\quad y(t) = Ch(t),$\n\nTo capture multi-directional dependencies, the Triple Mamba layer applies three separate Mamba modules, each operating on the same input $z_{TMB}(t)$ but with distinct ordering strategies: forward, backward, and random inter-slice directions. The output of the $i^{th}$ Triple Mamba layer is computed as:\n\n$\\psi_{i}(z_{TMB}(t)) = \\psi_{i}^{f}(z_{TMB}(t)) + \\psi_{i}^{b}(z_{TMB}(t)) + \\psi_{i}^{u}(z_{TMB}(t)),$\n\nwhere $\\psi_{i}^{f}$, $\\psi_{i}^{b}$, $\\psi_{i}^{u}$ represent the outputs from the forward, backward, and random inter-slice directions, respectively. This formulation can be succinctly expressed as:\n\n$Triplemamba_{i}(z_{TMB}(t)) = Mamba_{i}^{f}(z_{TMB}(t)) + Mamba_{i}^{b}(z_{TMB}(t)) + Mamba_{i}^{u}(z_{TMB}(t)).$\n\nWe will include these details in the camera-ready version (supplementary material if the space does not permit it in the main paper). \n\n> Lines 286–299 lack a clear reference to the corresponding table.\n\nThank you for the comments. We will add the corresponding table number in the text.\n\n> The sketch and pseudolabels in Fig. 4 often appear to come from different, unrelated scenes.\n\nThe difference is due to a temporal discrepancy between the data sources: the 3D ground truth annotations in SemanticKITTI were collected around 2013, whereas the satellite imagery used for Pseudo-labeled Satellite Image Annotations (PSA) was captured approximately in 2025. This temporal gap may lead to minor inconsistencies in spatial structures. Also, the PSA are generated through a weakly supervised pipeline that aligns satellite imagery with text embeddings, which enables scalable semantic supervision but can introduce semantic or geometric inaccuracies. In contrast, the sketches are derived directly from the 3D ground truth, ensuring temporal consistency with the underlying scenes. As a result, sketches serve as a reliable spatial reference that helps mitigate the domain gap introduced by the satellite imagery. We will explicitly note this in the camera-ready.\n\n> Consider citing recent methods like DynamicCity and Towards Generating Realistic 3D Semantic Training Data for completeness.\n\nWe will cite and discuss DynamicCity [1] and 3DiSS [2] in the final version. These works are indeed relevant to our study. We sincerely thank the reviewer for mentioning them.\n\n[1] DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenes\n\n[2] Towards Generating Realistic 3D Semantic Training Data for Autonomous Driving.\n\n> Are 3D GT classes used to filter incorrect CLIP pseudolabels during generation?\n\nDuring pseudo-label generation, we do not directly filter the CLIP outputs using the 3D ground truth semantic classes. Instead, we utilize the set of semantic labels present in the 3D ground truth scenes as candidate classes for predicting labels in the satellite imagery. After the initial predictions are made, we conduct a manual validation step, where the predicted class distributions are compared against the bird's-eye view (BEV) projections of the 3D ground truth. This post-hoc verification ensures that the retained pseudo-labels are semantically consistent with the underlying scene context, thereby enhancing the accuracy and reliability of the annotation process.\n\n> Are the textual descriptions in Fig. 1 included in the dataset?\n\nYes, we include textual descriptions used in Fig.1 in the dataset. It will be released to support further research.\n\n> The sketches are too precise to be human-drawn.\n\nWe agree that human-drawn sketches are an important direction. Thanks for acknowledging our SketchSem3D dataset as a first step towards generating a 3D scene from human-drawn sketches. Based on our benchmark, future work will incorporate freehand or user-generated sketches to better approximate real-world input conditions and further enhance the model's generalization and practical usability."}, {"review_id": "Dc6VUklgtr", "reviewer": "Reviewer_Ws1Y", "summary": "This paper addresses the challenge of generating semantically rich 3D outdoor scenes from freehand sketches and satellite pseudo-labels. The authors propose CymbaDiff, a diffusion-based framework that enhances spatial consistency in large-scale urban scenes by explicitly modeling cylindrical continuity and vertical hierarchies. To enable standardized evaluation, the authors introduce SketchSem3D, a large-scale benchmark for sketch-based 3D semantic scene generation, comprising two subsets: Sketch-based SemanticKITTI and Sketch-based KITTI-360.", "questions": "Although the authors have made significant contributions in generating outdoor scenes using abstract views, some details in algorithm design and module implementation require clearer explanations to facilitate reader comprehension. Experimental metrics should also incorporate evaluations of algorithm efficiency performance to validate the achievement of experimental objectives.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The method introduces cylinder mamba blocks to enforce structured spatial ordering, explicitly capturing cylindrical continuity and vertical hierarchies while preserving local/global scene coherence, overcoming limitations of Cartesian-based sequential modeling.\n2. The work establishes SketchSem3D, the first large-scale benchmark for sketch-based 3D semantic scene generation. \n\nWeaknesses:\n1. L120-L122: The paper's justification for using Mamba over DiT hinges critically on overcoming the latter's \"high memory costs\" in 3D scene generation. However, no experimental data (e.g., parameter counts, inference memory consumption, runtime) is provided to demonstrate the purported memory efficiency gains of the proposed approach. Substantiating this key aspect requires such comparative analysis.\n2. L240-L249: While the proposed cylindrical coordinate system aims to solve the issue of \"disrupting original 3D spatial adjacency structures\" inherent to serialization in Cartesian coordinates, its efficacy for common street-view datasets like KITTI (shown in the paper) is unclear. Since such scenes lack rotational symmetry, simply \"(θ,r) unfolding\" the cylinder likely does not preserve meaningful adjacency relationships. \n3. L263: Triplane Mamba uses Cartesian coordinates while C-Mamba uses cylindrical coordinates, leading to different feature indexing schemes. Please clarify how feature alignment is achieved before fusion: Are the features directly fused, or is a coordinate transformation.\n4. L249:  It is stated that the evaluation metric differs from SemCity's approach. Yet, the reported FID performance for the SemCity baseline in this work matches the SemCity paper's reported FID value exactly.\n5. L308-L315: Request for explanation of ablation result. The decomposition of 3D convolution across three dimensions (Sec. 4.3) is intuitively expected to trade-off some reconstruction quality for computational efficiency. However, the ablation study shows a significant decrease in FID after applying DDCB. \n\nMinor:\nFigure 4 can be enhanced by incorporating fine-grained structural consistency analysis, as employed in scene reconstruction benchmarks like NeRF-Synthetic. Quantification of spatial coherence patterns (e.g., cylindrical projection seam artifacts or voxel occupancy consistency) would strengthen methodological transparency.", "was_revised": true, "final_justification": "The authors' rebuttal has addressed my concerns. However, I agree with other reviewers that the paper's clarity still needs further improvement.\n\nAt this stage, I recommend a \"Borderline Accept\" rating considering the technical contribution, though I would also support rejecting the paper to allow for substantial writing improvements before submission to a top-tier venue.", "rebuttal": "We thank Reviewer Ws1Y for their constructive feedback and appreciation, especially for recognizing the effectiveness of Cylinder Mamba blocks in enhancing spatial coherence and the contribution of SketchSem3D as the first large-scale sketch-based 3D scene generation benchmark. In response to the concerns expressed in Weaknesses and Questions, we provide the following answers:\n\n> Provides data on parameters, memory use, or runtime to support reducing memory cost.\n\nThank you for the valuable suggestion. We provide the following table to compare the memory cost of our method with 3D DiT and 3D Latent Diffusion. To ensure a fair comparison, we replaced our latent-space denoiser with their respective architectures, and the results reflect only the denoiser parameters. Minor code modifications were applied to both baselines to enable consistency with our method's runtime environment. The table demonstrates that our architecture (CymbaDiff) offers a clear memory gain. Over DiT, this gain is about 8.5x, which can be critical in practical deployment.\n\nTable: The efficiency comparison. M: Million, and S: seconds\n| Methods | Parameters (M) | Inference Times (S) |\n|---------|----------------|-------------------|\n| 3D DIT | 195 | 4.5 |\n| 3D Latent Diffusion | 1265 | 11.4 |\n| CymbaDiff | 23 | 7.2 |\n\n> The benefit of cylindrical coordinates is unclear for datasets like SemanticKITTI.\n\nAlthough datasets such as SemanticKITTI do not exhibit strict rotational symmetry, they often display approximate bilateral symmetry along the driving path, for example, with sidewalks and buildings typically appearing on both sides of the road. To capture this structure, we adopt cylindrical coordinates $(\\theta, r, z)$, which align well with the egocentric and radial layout of street-view environments.\n\nCylindrical coordinates offer a vehicle-centric, geometrically coherent representation, enabling semantic tokenization along angular and radial dimensions and facilitating long-range context modeling with Mamba. In contrast, Cartesian coordinates provide uniform, axis-aligned voxels suitable for dense computation and 3D convolutions but lack egocentric awareness and may underrepresent radial continuity.\n\nTo exploit the strengths of both representations, our framework integrates cylindrical and Cartesian coordinate systems. This hybrid design combines the semantic coherence of cylindrical encoding with the computational efficiency of Cartesian grids, enhancing overall 3D scene modeling. We will further clarify this in the camera-ready.\n\n> Clarify how Triple Mamba (Cartesian) and C-Mamba (cylindrical) features are aligned before fusion.\n\nTriple Mamba and C-Mamba operate in distinct coordinate systems, Cartesian and cylindrical, respectively, which necessitates careful alignment to enable effective feature fusion. To achieve this, we apply an inverse coordinate transformation to convert the cylindrical features produced by C-Mamba back into Cartesian voxel space. Specifically, for each token in the $(\\theta, r, z)$ grid, we compute its corresponding $(x,y,z)$ location using the predefined voxel resolution and center alignment parameters.\n\nThis transformation allows the C-Mamba outputs to be reprojected into a dense 3D tensor that is spatially aligned with the Cartesian grid used by Triple Mamba. Once both feature representations reside in the same $(x,y,z)$ coordinate space, we apply element-wise fusion, ensuring spatial correspondence is preserved during integration. This approach enables the coherent fusion of features across modalities, enhancing the model's ability to capture complementary information from both coordinate systems. We will release the code to enhance the clarity for readers.\n\n> The paper claims a different evaluation metric from SemCity, yet reports the same FID value as in the SemCity paper. Please clarify.\n\nApologies for the misunderstanding. The evaluation metrics used in our study differ from those adopted in SemCity. SemCity relies on the 2D Fréchet Inception Distance (FID) computed from rendered images. Our approach employs more comprehensive 3D evaluation metrics, specifically 3D FID and Maximum Mean Discrepancy (MMD), which directly measure geometric fidelity and semantic consistency within the voxel space. Although this distinction is noted in Lines 279–282 of the manuscript, we will further emphasize it in Table 2 to avoid potential misinterpretation and to ensure a fair comparison between the methods.\n\n> The 3D convolution decomposition (Sec. 4.3) should trade quality for efficiency, yet FID improves with DDCB. Please explain.\n\nTo balance reconstruction quality and efficiency in decomposed 3D convolutions, the Dilated Decomposed Convolution Block (DDCB) stacks multiple Dimensional Decomposition Residual (DDR) units with varying dilation rates. This design captures multi-scale spatial features while maintaining low computational cost.\n\nSerially structuring decomposed convolutions compensates for the reduced joint receptive field in dimension-wise decomposition, preserving representational capacity and enhancing directional context modeling. This contributes to the improved FID performance observed in our experiments.\n\n> Enhance Fig. 4 with structural consistency analysis (e.g., seam artifacts, voxel coherence) as in NeRF-Synthetic for better transparency.\n\nWhile fine-grained analysis of structural consistency could offer insights into model behavior, the Cylinder Mamba block operates in 3D latent space, where features represent abstract embeddings rather than explicit geometry, making spatial interpretation challenging. Nonetheless, we acknowledge its value and will consider such interpretability strategies in future work.\n\n> Clearer design details.\n\nThank you for your valuable suggestions. We will improve Sec. 4 to provide clearer descriptions of each core module in the final version, and also augment the appendix with illustrative figures to visually support the architectural details. Furthermore, we plan to release the source code alongside the paper, which will ensure covering all details."}, {"review_id": "yfMRzZm5UX", "reviewer": "Reviewer_pTZW", "summary": "This paper presents CymbaDiff, a novel approach for generating 3D outdoor scenes from sketches and pseudo-labeled satellite images. The model enhances spatial coherence by capturing cylindrical continuity and vertical hierarchy in urban environments. The authors introduce the SketchSem3D dataset, a large-scale benchmark for evaluating sketch-based 3D scene generation. The method outperforms existing approaches in terms of semantic consistency and spatial realism, demonstrating robust performance across multiple urban datasets.", "questions": "1. Clarification of the Pseudo-Labeled Satellite Image Annotations Process:\nQuestion: The paper uses pseudo-labeled satellite image annotations generated by CLIP and SAM for training. Could you provide more details on how the pseudo-labeling pipeline handles cases where CLIP or SAM might produce inaccurate or ambiguous labels? Specifically, how robust is the model to errors in the pseudo-labeling process?\nSuggestion: A more detailed analysis of the error propagation in the pseudo-labeled data pipeline would be useful. Discuss any failure modes observed in the pseudo-labeling process and whether any mechanisms, such as manual refinement or a confidence threshold, are employed to mitigate errors.\n2. Question: The paper demonstrates strong results on the SketchSem3D dataset, but its scalability to more diverse, larger-scale urban scenes is not fully explored. How well does CymbaDiff generalize to large-scale real-world urban environments or scenes that are more complex or varied than those in the SketchSem3D dataset?\nSuggestion: It would be valuable to conduct additional experiments on real-world datasets or more diverse scenes beyond the current benchmark. This would provide insights into whether the model can handle the scale and complexity of urban environments as encountered in real applications, like autonomous driving or urban planning.\n3. Question: The paper lacks extensive quantitative comparison experiments with other methods. While a single comparison is provided in the supplementary material, would it be possible to include more quantitative results comparing CymbaDiff to other state-of-the-art methods, especially in terms of generation quality, computational efficiency, and generalization across different datasets?\nSuggestion: It would strengthen the paper to expand the comparison to more methods, ideally covering a range of recent techniques in the domain of 3D scene generation.\n4. Question: While the paper proposes a novel approach to generating 3D scenes from sketches and pseudo-labeled satellite images, could you elaborate on the specific innovative aspects of CymbaDiff that distinguish it from existing methods, particularly in terms of model architecture and novel contributions to the field of 3D scene generation?\nSuggestion: It would be helpful to provide a clearer distinction between CymbaDiff and other state-of-the-art models, especially in the way the Cylinder Mamba blocks are integrated into the diffusion process. How does this structured spatial coherence contribute uniquely to the generation process compared to previous methods like UrbanDiff or Semcity?\n5. Question: Could you provide some failure cases or error examples to objectively illustrate the limitations of the CymbaDiff method?\nSuggestion: Including error analysis would be very useful in highlighting specific weaknesses of the model, such as when it fails to generate accurate representations for highly complex scenes (e.g., dense urban environments with varying object sizes or moving elements) or low-quality inputs (e.g., poorly sketched or ambiguous freehand sketches).", "limitations": "Yes.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths \n1. Quality of the Work: The paper introduces a novel and robust generative model for 3D outdoor scene generation from abstract sketches and satellite image annotations. The model, CymbaDiff, effectively addresses the challenge of spatial coherence in urban environments\n2. Clarity: The paper is well-structured, with clear explanations of the proposed CymbaDiff model, the SketchSem3D dataset, and the experimental setup. The key contributions are outlined concisely, making it easy for readers to grasp the significance of the work.\n3. Significance: The proposed sketch-based generation approach addresses a critical gap in the current literature on 3D outdoor scene synthesis, especially for applications like autonomous driving and urban simulation, where the generation of complex, semantically rich environments is crucial.\n## Weaknesses\n1. Quality of the Work: While the paper demonstrates strong performance on the SketchSem3D dataset, it would be beneficial to evaluate the model on more real-world data or include cross-domain experiments. This would further validate the model’s generalizability in unseen real-world urban settings.\n2. Clarity. Lack of a detailed analysis of failure cases: While the paper highlights the strengths of the model, it does not provide a deep dive into failure modes or limitations. A discussion of specific types of scenes or conditions where the model struggles could help provide a more balanced assessment of its capabilities.\n3. Significance: Although the paper demonstrates impressive results in controlled environments, scalability to larger urban areas or more diverse scene types (e.g., complex rural or mixed terrains) is not fully explored. \n4. Absence of user studies or human evaluation: The paper focuses on quantitative metrics to evaluate the model's performance, but it lacks any user studies or human evaluations to gauge how intuitive and effective the system is from a practical, real-world perspective.", "was_revised": true, "final_justification": "I appreciate the authors' detailed rebuttal and their efforts in conducting additional experiments. However, two core issues remain unresolved and have significantly influenced my recommendation. 1. The method is restricted to semantic voxel outputs, and the examples shown remain abstract and symbolic. Without the ability to generate realistic or physically plausible scenes, the practical impact of the proposed approach in downstream applications (e.g., simulation, urban planning) remains questionable. 2. Nearly half of the object categories in Table 4 do not achieve state-of-the-art performance—including critical classes like \"truck\" and \"road.\" The authors attribute this to weak supervision, but the limited performance across both major and minor categories raises concerns about robustness and generalization. Given these limitations, I believe the contribution is promising but not yet mature enough for acceptance. Therefore, I am keeping my score as borderline reject.", "rebuttal": "We thank Reviewer pTZW for their constructive feedback and appreciation, particularly for recognizing CymbaDiff's novelty and robustness, the clarity of our model, dataset, and experiments, and the significance of our sketch-based approach for 3D scene synthesis in applications like autonomous driving and urban simulation. In response to the concerns expressed in Weaknesses and Questions, we provide the following answers:\n\n> Real-world and cross-domain tests are needed to assess generalizability and scalability to diverse scenes.\n\nWe have demonstrated the effectiveness of our approach on SketchSem3D dataset, which is curated using the widely adopted SemanticKITTI and KITTI-360 datasets. \n\nTo further address the concern, we now also validate the efficacy of CymbaDiff by evaluating our model trained on Sketch-based SemanticKITTI on the challenging real-world nuScenes dataset. CymbaDiff achieves a 3D FID of 183.37 and a 3D MMD of 0.09, outperforming the widely used UrbanDiff model, which achieves a 3D FID of 291.4 and a 3D MMD of 0.11 (lower is better). This ascertains the effectiveness of our approach.\n\n> The paper lacks analysis of failure cases; discussing challenging scenes would offer a more balanced evaluation.\n\nObserving the results of CymbaDiff on the proposed SketchSem3D dataset, it is apparent that it demonstrates strong performance, effectively handling challenges such as semantic misalignment caused by noisy pseudo-labels, e.g., due to confusion between vegetation and buildings. Nevertheless, it does occasionally fail to accurately reconstruct small or occluded objects that are underrepresented in the training data or sparsely encoded in the sketch and PSA inputs. Although CymbaDiff mitigates this issue to some extent through the use of the Cross-Scale Contextual Block and Cylinder Mamba Block, which capture multi-scale contextual information, its performance could be further enhanced by increasing the representation of small objects in the dataset. We will add more discussions in the camera-ready (in supplementary if it is not permitted by space in the main paper).\n\n> The paper lacks user studies or human evaluations.\n\nWhile our study primarily relies on quantitative metrics such as 3D FID and MMD to evaluate the proposed approach, these widely adopted measures in generative tasks, ranging from 2D image synthesis to 3D object generation, offer objective, reproducible, and scalable assessments of fidelity and diversity.\n\nWe acknowledge that human evaluation can offer complementary insights into semantic coherence, structural plausibility, and perceptual realism. However, its subjectivity, high resource demands, limited scalability, and variability in user interpretation make it less practical for consistent and large-scale benchmarking.\n\nIn future work, we will consider incorporating structured human evaluation alongside metric-based evaluation to qualitatively assess the perceptual quality of generated 3D scenes, complementing our current quantitative analysis for a more comprehensive evaluation.\n\n> The paper uses CLIP and SAM for pseudo-labeling. How does the pipeline handle inaccurate or ambiguous labels from them?\n\nTo reduce noise in Pseudo-labeled Satellite Image Annotations (PSA), we restrict CLIP predictions to semantic classes present in the 3D ground truth, ensuring alignment with the SketchSem3D taxonomy and preventing irrelevant or out-of-distribution labels.\n\nSecond, we perform manual spot-checks by comparing predicted class distributions with BEV projections of the 3D ground truth to identify and remove mislabeled samples. Third, incorporating sketches provides structural cues that help the model compensate for PSA imperfections during training.\n\nBeyond these safeguards, the architecture enhances robustness through a pre-trained VAE that encodes inputs into a stable latent space with accurate spatial priors. The latent diffusion process then iteratively denoises within this space, making it resilient to moderate input noise. Combined with sketch-based guidance, this design ensures semantically coherent 3D scene generation under weak or noisy supervision.\n\n> Provide more quantitative comparisons with state-of-the-art methods, and include broader results on generation quality, efficiency, and generalization to strengthen the paper.\n\nWe selected 3D Latent Diffusion and 3D-DiT as baselines due to the limited availability of open-source 3D scene generation models. Although not originally designed for semantic voxel generation, both models have shown strong performance in image and 3D object synthesis, and their general architectures can be effectively adapted to the SketchSem3D dataset to establish strong baselines.\n\nWe acknowledge the relevance of methods like UrbanDiff and SemCity; however, the lack of public implementations prevents direct comparison. We address this by adding BlockFusion (required by other reviewers) as a baseline on SketchSem3D. However, since it was originally designed as a 2D tri-plane denoiser, adapting it to our 3D latent space led to unstable training and suboptimal performance, with the loss failing to converge. Despite these challenges, we still cite these works and will include a discussion of their contributions in the camera-ready.\n\nMoreover, we provide quantitative comparisons in the following table across methods in terms of parameter count and runtime performance. These results demonstrate that CymbaDiff achieves a favorable trade-off between model efficiency and computational cost, offering competitive performance with significantly fewer parameters compared to these two generative models.\n\nTable: The efficiency comparison. M: Million, and S: seconds\n| Methods | Parameters (M) | Inference Times (S) |\n|---------|----------------|-------------------|\n| 3D DIT | 195 | 4.5 |\n| 3D Latent Diffusion | 1265 | 11.4 |\n| CymbaDiff | 23 | 7.2 |\n\nThese additions will be included in the camera-ready to improve clarity and support a more comprehensive evaluation of CymbaDiff. We will also provide qualitative comparisons on the SketchSem3D dataset to visually demonstrate the model's effectiveness.\n\n> What are CymbaDiff's key innovations compared to prior methods? and Clarify how Cylinder Mamba enhances diffusion and differs from models like UrbanDiff or SemCity.\n\nThe key innovations of CymbaDiff lie in its use of sketches and pseudo-labeled satellite annotations (PSA) as input modalities for 3D semantic voxel scene generation, and the incorporation of Cylinder Mamba blocks into the denoising diffusion process, facilitating contextual modeling across both cylindrical $(\\theta, r, z)$ and Cartesian $(x, y, z)$ coordinate systems.\n\nThe former enables generation from abstract and lightweight inputs, which supports more accessible supervision and is particularly suitable for large-scale or remote scenarios where high-fidelity sensor data is unavailable or impractical. The latter leverages the semantic alignment and radial awareness of cylindrical encoding with the computational efficiency and regularity of Cartesian grids. This dual representation, together with a triple-axis decomposition strategy, enables effective modeling of radial, vertical, and angular dependencies in complex outdoor scenes.\n\nHowever, methods such as UrbanDiff and SemCity rely on dense sensory inputs, including RGB images or BEV representations. UrbanDiff operates solely within the Cartesian coordinate system, while SemCity projects 3D data onto three orthogonal 2D planes, leading to the loss of important volumetric structural information. The experimental results demonstrate that CymbaDiff achieves superior performance, particularly in terms of 3D Fréchet Inception Distance (FID), highlighting its effectiveness in capturing complex spatial dependencies. We will add more discussions in the camera-ready (in supplementary if it is not permitted by space in the main paper)."}]

Author rebuttal:

Meta-review: The AC and the reviewers thank the authors for their response.
This paper proposes the first benchmark for outdoor 3D semantic scene generation, SketchSem3D, and designs the CymbaDiff framework to achieve higher spatial consistency and semantic fidelity, advancing research in this field.

After the rebuttal, one reviewer leaned toward acceptance, and two reviewers leaned toward borderline acceptance. However, another reviewer expressed concerns about generating realistic scenes, downstream applications, and the overall quality, thus giving a borderline rejection.

After carefully reading the paper, the reviewers’ comments, and the authors’ response, the AC acknowledges the contribution of establishing the first large-scale outdoor 3D semantic scene generation model and benchmark. The proposed large-scale benchmark SketchSem3D and the CymbaDiff method demonstrate promising direction and performance. Moreover, the construction process based on abstract freehand sketches and pseudo-labels significantly reduces the need for manual annotations. These aspects provide new perspectives for research in this direction.

The AC believes that after addressing the reviewers’ comments and fixing minor issues, the paper is suitable for acceptance.
The AC recommends acceptance.
Decision: Accept (poster)