X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability
Authors: Yu Yang
Alan Liang
Jianbiao Mei
Yukai Ma
Yong Liu
Gim Hee Lee
Venue: NeurIPS 2025 poster ()
Keywords: 3D Scene Generation
Autonomous Driving
Primary area: applications

Abstract: Diffusion models are advancing autonomous driving by enabling realistic data synthesis, predictive end-to-end planning, and closed-loop simulation, with a primary focus on temporally consistent generation. However, large-scale 3D scene generation requiring spatial coherence remains underexplored. In this paper, we present X-Scene, a novel framework for large-scale driving scene generation that achieves geometric intricacy, appearance fidelity, and flexible controllability. Specifically, X-Scene supports multi-granular control, including low-level layout conditioning driven by user input or text for detailed scene composition, and high-level semantic guidance informed by user intent and LLM-enriched prompts for efficient customization. To enhance geometric and visual fidelity, we introduce a unified pipeline that sequentially generates 3D semantic occupancy and corresponding multi-view images and videos, ensuring alignment and temporal consistency across modalities. We further extend local regions into large-scale scenes via consistency-aware outpainting, which extrapolates occupancy and images from previously generated areas to maintain spatial and visual coherence. The resulting scenes are lifted into high-quality 3DGS representations, supporting diverse applications such as simulation and scene exploration. Extensive experiments demonstrate that X-Scene substantially advances controllability and fidelity in large-scale scene generation, empowering data generation and simulation for autonomous driving.

X'-ccen: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability
Yu Yang!. Zhejjanng University
Alan Liang?
Jianbiao Mei
Yukai Ma
Yong Liu Gim Hee 2.e
National University of Singapore gi github. io/
Abstract
Diffusion models are advancing autonomous driving by enabling realistic data syn- thesis, predictive end-to-end planning. and closed-loop simulation, with a primary focus on temporally consistent generation. However, large-scale 3D scene genera- tion requiring spatial coherence remains underexplored. X' X'-Scene, a novel framework for large-scale driving scene generation that achieves geometric intricacy, appearance fidelity, and flexible controllability. Specifically, X-Scene supports multi-granular control, including low-level layout condition- ing driven by user input or text for detailed scene composition, and high-level semantic guidance informed by user intent and LLM-enriched prompts for efficient customization. To enhance geometric and visual fidelity, we introduce a unified pipeline that sequentially generates 3D semantic occupancy and corresponding multi-view images and videos, ensuring alignment and temporal consistency across modalities. We further extend local regions into large-scale scenes via consistency- aware outpainting, which extrapolates occupancy and images from previously generated areas to maintain spatial and visual coherence. The resulting scenes are lifted into high-quality 3DGS representations, supporting diverse applications such as simulation and scene exploration. Extensive experiments demonstrate that X' -Seeee substantially advances controllability and fidelity in large-scale scene generation, empowering data generation and simulation for autonomous driving.
this paper, we present
1
Introduction
Recent ad advanemments in generative Al have profoundly impacted autonomous driving, with diffusion models (DMs) emerging as pivotal tools for data synthesis and driving simulation. Some approaches utilize DMs data machines, producing high-fidelity driving videos [1-14] multi-modal synthetic data [15-18] to augment perception tasks, as well as generating corner cases (e.g., vehicle cut-ins) to enrich planning data with uncommon yet critical scenarios. Beyond this, other methods employ DMs as world models to predict future driving states, enabling end-to-end planning [19-21] and closed-loop simulation [22-28]. All these efforts emphasize long-term video generation through temporal recursion, encouraging DMs to produce coherent video sequences for downstream tasks.
However. large-scale scene generation with spatial expansion, which aims to build expansive and immersive 3D environments for arbitrary driving simulation, remains an emerging yet underexplored direction. A handful of pioneering works have explored 3D driving scene generation at scale. For example, SemCity [29] generates city-scale 3D occupancy grids using DMs, but the lack of appearance details limits its practicality for realistic simulation. UniScene [18] and InfiniCube [30] extend this by generating both 3D occupancy and images, but require manually defined large-scale layout conditioning input, complicating the generation process and hindering flexibility.
corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

High-Level Control
Joint Occupancy & Image / Video Generation 10 Occupancy
Mutthee Vioges/idece
Scoee Tuet Besegette
Scoee Styl
Temprend
Bockyrwed cloraets
X- Scene: World Generaton
Goncn-Gooph Lagust
Lagur
Corr-saatt Mecter Hop?
Geometric and Visual Reconstruction Recondtructed 30 OS
Low-Level Control
Semontic
RGB
Rendder
Depth
RCB
Layost Map
Render
Flexible Controllabiity
Generation and Reconstruction High Fidelity
Figure 1: Overview of X -Ceeeee, unified world generator that supports multi- ceattiggraularr controllability through high-level text-to-layout generation and low-level BEV layout conditioning. It performs joint occupancy, image, and video generation for 3D scene synthesis and reconstruction with high fidelity.
In this work, we tackle the problem of large-scale scene generation with spatial expansion, which presents three key challenges: 1) Flexible Controllability: enabling versatile control through both low- level conditions (e.g., layouts) for precise scene composition and high-level prompts (e.g., isee-intent text descriptions) for intuitive customization; 2) High-Fidelit Geometry and Appearance: generating intricate geometry with photorealistic appearance to ensure structural integrity and visual realism in 3D scenes; 3) Large-Scale Consistency: maintaining spatial coherence across extended regions to ensure global consistency throughout the generated environment.
To address these challenges, we propose X-Scene, novel framework for large-scale driving scene generation featuring: 1) Multi-Granular Controllability: It enables users guide generation at multiple abstraction levels, supporting fine- ireeggrined BEV semantic layouts for precise control and high-level text prompts for efficient customization. Text prompts enriched by LLMs into detailed scene narratives, structured as scene graphs and converted into vector-mas layouts via a scene- saaeeegrpphhhh to layout diffusion module. These layouts provide spatial and semantic cues that guide subsequent scene synthesis, combining layout-level precision with prompt-hased flexibility. 2) Geometric and Visual Fidelity: X'Scene employs unified pipeline that sequentially generates 3D semantic occupancy and corresponding multi-view images and videos, ensuring structural accuracy, photorealistic appearance, and temporal consistency with cross-modal alignment. Consistent Large-Scale Extrapolation: To synthesize expansive environments, it progressively extrapolates new scene content conditioned on adjacent, previously generated regions. The consistency-ave outpainting mechanism preserves spatial continuity and enables seamless extension beyond local areas.
Furthermore, to support downstream applications such realistic driving simulation, reconstruct the generated occupancy and multi-view images/videos into 3D Gaussian (3DGS) [3]])))]]eeee representations, which faithfully preserve geometric detail and visual quality. By unifying controllability. fidelity, and scalability, X' X'Scene advances the state-of-the-art in large-scale, controllable driving scene synthesis, empowering realistic data generation and simulation for autonomous driving.
The main contributions our work are summarized as follows:
We propose X'-ccene, novel framework for large-scale 3D driving scene generation with multi- granular controllability, geometric and visual fidelity, and consistent large-scale extrapolation, supporting wide range of downstream applications.
We design flexible multi-granular control mechanism that synergistically combines high-level semantic guidance (L.MM-enriched text prompts) with low-level geometric specifications (user- provided text-driven layout), enabling scene creation tailored diverse user needs.
We present a unified occupancy-image -iiaiimaaddeeee video generation pipeline that achieves geometric fidelity, photorealistic appearance, and temporal coherence, enabling seamless large-scale scene expansion.
Extensive experiments show X'Scceee achieves superior performance in generation quality and controllability. enabling diverse applications from data augmentation to driving simulation.

---

Multi- Granular Controliablity
Joint Rccupancy-lmogge-e Video Generation
Large-Sccr Scale Scone Extropolation
High-Level Control
Tripone Diffusion Medel
overassting
Scare Test Deserption
Visel Appostence
Jense Segle
Geometry
Faregrined ofjects
-Reddgrent Scese-Graph
Stuse Gook
Difrailes Mode
Low Level
Jwiten
Recalved
Control
Ruedor
Carera
Layet
Mop
Multi-view Image Diffusion Mode!
Figure 2: Pipeline of X'-Scene for driving scene generation: (a) Multi maaliigraaular controllability supports both high-level text prompts and low-level geometric constraints for flexible specification; (b) Joint grepancy--maage and videos via conditional diffusion; (c) Large-scale extrapolation enables coherent scene expansion through consistency- aware outpainting (Fig. 4). Fig. details the scene-gras gene-graph layout diffusion.
generation synthesizes aligned 3D voxels and multi-view images
2 Related Works
Driving Image and Video Generation. Diffusion models [32-35] have revolutionized image synthesis by iteratively refining Gaussian noise into high-quality results. Building on this, they have greatly advanced autonomous driving by enabling realistic image and video generation for various downstream tasks. Several methods synthesize driving images [1, 36-39] or videos [2-14] from layout conditions to augment perception data. Others [40, 41] generate rare yet critical events, e-g.. lane changes or cut-ins, to improve planning under corner cases. Moreover, diffusion- basedd world models predict future driving videos for end-to-end planning [19-21] or closed-loop simulation [22- 27J. While prior works emphasize temporal consistency, our approach explores the complementary aspect of spatial coherence for large-scale scene generation.
3D and 4D Driving Scene Generation. Recent advances extend beyond 2D generation to 3D/4D
scene synthesis [42], producing 3D environments from LiDAR point clouds [43-52], occupancy volumes [53, 54, 29, 55-59], or 3D Gaussian Splatting (3DGS) [60-67], serving as neural simulators for data generation and driving simulation. The field has futher progressed two directions: 1) 3D world models that predict future scene representations (e.g.. point clouds [68-70] or occupancy maps [71-76] to aid planning and pretraining; and 2) multi-modal generators that synthesize aligned data across modalities, such as image-LiDAR [15, 16] iccge-occupancy pairs [17, 18, 24]. Our work explores joint
generation, constructing scenes that integrate fine-grained geometry, photorealistic appearance, and temporally coherent dynamics.
Large-Scale Scene Generation. Large-scale city generation has evolved along four main directions: video-based [77, 78], outpainting-basee [79-81], PCG-based [82-84], and coural-based methods [85- 87]. While effective for natural urban environments, these approaches are not tailored for driving scenarios requiring accurate street layouts and dynamic agents. triving-specific methods also face key limitations: XCube [58] and SemCity [29] model only geometric occupancy without appearance, while DrivingSphere [24], UniScene [18], and InfiniCube [30] depend on manually designed large- scale layouts, limiting scalability, In contrast, our -'eeee framework jointly generates geometry and appearance with flexible, text-driven control, offering efficient and user-friendly customization.
3 Methodology
X'Scene aims to generate large-scall scale 3D driving scenes within a unified framework addressing controllability. fidelity, and scalability. As shown in Fig. 2, it consists of three main components: 1) Multi-Granular Controllability (Sec. 3.1), which integrates high-level user intent with low-level geometric constraints for flexible scene specification; 2) Joint Occupancy, Image, and Video Gen- eration (Sec. 3.2), which employs conditioned diffusion models synthesize 3D voxel occupancy, multi-view images, and temporally coherent videos with 3D-aware guidance; and 3) Large-Scale
3

---

Scene Groph
Semontic Graph
Graph
Pleae geveaate wteen dtrett degtiee, mochs dlhing on red few pedistrions warknn the sieaslis and posite
inught driie
Feind
somsrnt
style sibse ceg diwe as Exery disture
Duck
fannrraed oijects:
car block SUV sarked aloeg the sovet
Lftref
Fact
scote-goph
whiee gree
User Fromp?
porkatile:: wat stacding near the stlows
Pignt Probinnie
owbeddiig
Badgmand Clenvet::
Large-Language Model
sibowill cacrete mueeing paolle nod crecovoll stiliee at the ietecestiee cioer
DeNpiON''
Scae Goje Logoat-
Scone Description
trcck inbood and biger then
Momory Bank
pedeetiale on the sdewelk
Cocenatdd Legae
(a) Textual Description Enrichment
(b) Textual Scene-Graph Layout Generation
Figure 3: Pipeline of textual description enrichment and scene-graph to layout generation: (a) Input prompts enriched using RAG-augmented L.LSs to produce structured scene descriptions; (b) Spatial relationships are converted into a scene graph and encoded with a graph network, followed by conditional diffusion that denoises object boxes and lane polylines into the final layouts.
Scene Extrapolation and Reconstruction (Sec. 3.3), which extends scenes via consistency- outpainting and lifts them into 3DGS representations for downstream simulation and exploration.
3.1 Multi-Granula Controllability
X-Scene supports dual-mode scene control through: 1) high-level textual prompts, which are enriched by LLMs and converted into structured layouts via text-to-layout generation model (illustrated in Fig. 3); and 2) direct low-level geometric control for precise spatial specification. This hybrid approach enables both intuitive creative expression and exacting scene customization.
Text Description Enrichment. Given coarse user-provided textual prompt Tp, we first enrich into a comprehensive scene description D {S, O B, L}, comprising: scene style S (weather, light- ing, environment), foreground objects O (semantics, spatial attributes, and appearance), background elements B (semantics and visual characteristics), and textual scene-graph layout L, representing spatial relationships among scene entities. The structured description D generated as:
Gdeseription (Tp, RAG(Tp,
where M {m,) denotes the scene description memory. Each entity m; is automatically constructed using one of the collected scene datasets by: 1) extracting (S, O, B} using VLMs on scene images; and 2) converting spatial annotations (object boxes and road lanes) into textual scene- graph layout L. As shown in Fig. 3, the Retrieval-Augmenned Generation (RAG) module retrieves relevant descriptions similar Tp from the memory bank .M, which are then composed into detailed, user-intended scene description by an LLM-based generator Gdescription-
This pipeline leverages RAG for few-shot retrieval and composition when processing brief user prompts, enabling ficxible and context-aware scene synthesis. The memory bank M is designed to be extensible, allowing seamless integration of new datasets support a broader variety of scene styles. Additional examples of generated scene descriptions are provided in the appendix.
Textual Scene-Graph to Layout Generation. Given the textual layout L, we follow prior works [88, 89] and translate it into a spatial layout map via a scene-graph-to-layy
pipeline (Fig. 3). First, we construct a scene graph G = (),E), where nodes V = ( represent M scene entities (c.g., cars, pedestrians, road lanes) and edges = {e,-)li, E (1, ........ represent spatial relations front of, on top of). Each node and edge is then embedded by concatenating semantic features s;, 8,-ر (extracted via a text encoder Eςex) with learnable geometric features 9t, Ji-sj, result- ing in node embeddings v, Concat( 9:)) and edge embeddings e, Concat(s; 9i-+)).
The graph embeddings are refined using a graph convolutional network, which propagates contextual information C;+j across the graph and updates each node embedding via neighborhood aggregation. Finally, layout generation formulated as conditional diffusion process: each object layout initialized as a noisy 7-D vector by E IR? (representing box center, dimensions, and orientation), while each road lane begins as set of N noisy 2D points pt E RNx2 with denoising process conditioned on the corresponding node embeddings v, produce geometrically coherent placements.

---

Low-Level Conditional Encoding. We encode fine- fineggrinned conditions (such as user seer-provide or model-ge generrted layout maps and 3D bounding boxes) into embeddings enable precise geometric control. As illustrated in Fig. 2. the 2D layout maps are processed by a ConvNet (Elayou) to extract layout embeddings Clayour- while 3D box embeddings Cbor are obtained via MLPs (Ebox), which fuse object class and spatial coordinate features. To further enhance geometric alignment, we project both the scene layout and 3D boxes into the camera view to generate perspective maps, which are encoded by another ConvNet (Quers)) to capture spatial constraints from the image plane. Additionally, high-level scene descriptions DD are embedded via a T5 encoder (Ewex) providing rich semantic cues for controllable generation through the resulting text embeddings Cjext
3.2 Joint Occupancy, Image, and Video Generation
We adopt a joint 3D-to-2D generation hierarchy that first models scene geometry via occupancy diffusion, followed by image synthesis guided by occupancy- rendaee maps to ensure geometric consistency. The pipeline is further extended with a temporal diffusion module for video generation, producing smooth motion and cross-view temporal coherence.
Occupancy Generation via Triplane Diffusion. We adopt triplane representation [90] encode 3D occupancy fields with high geometric fidelity. Given an occupancy volume o E R x triplane encoder compresses it into three orthogonal latent planes h = {hT hh= hh²) with spatial downsampling. To mitigate information loss due to reduced resolution, we propose novel triplane deformable attention mechanism that aggregates richer features for a query point q = (I, y/ 2) as:
K
Fq (2,
Σ
Σ
o (w -E(xx h (projp (2, App)
(2)
PE(my,az.yE) R=1
where K is the number of sampling points, PE(.): Rd + RD denotes positional encoding, and WP XxD generates attention weights with the softmax function o.)). The projection function 11 (x,
+ projp maps 3D coordinates to 2D planes (e.g., projay
(z, v)). and the learnable offset App [ PE(x, y, 2) E R² uses weights W E g2xD to shift sampling positions for better feature alignment. Then the triplane-VAA VAE decoder reconstructs the 3D occupancy field from the aggregated features Fy.
Building on the latent triplane representation h, we introduce a conditional diffusion model eg that synthesizes novel triplanes through iterative denoising. At each timestep I, the model refines a noisy triplane h, toward the clean target ho using two complementary conditioning strategies: 1) additive spatial conditioning with the layout embedding Playout and cross-attenttion-aseed conditioning with C = Concat (eoox exx)) integrating geometric and semantic constraints. The model is trained to predict the added noise c using the denoising objective: Doe = Ex,bo.c . [[]c C6
2000 (h, f, Clayout C) [2]]
Image Generation with 3D Geometry Guidance. After obtaining the 3D occupancy, we convert voxels into 3D Gaussian primitives parameterized by voxel coordinates, semantics, and opacity, which are rendered into semantic and depth maps via tile-based rasterization [31]. To incorporate object-level geometry, we first generate normalized 3D coordinates for the entire scene and extract object-specific regions based on bounding boxes. The corresponding coordinates are encoded into object positional embeddings Cpos. providing fine-grained geometric guidance. The semantic, depth, and layout (or perspective) maps are processed by ConvNets and fused with epos to form the final geometric embedding Egeo- This embedding is combined with noisy image latents to achieve pixel- aligned geometric conditioning. The image diffusion model eg further leverages cross-attention with conditions C (text, camera, and box embeddings) for appearance control. The model is trained
via: cime Ex,xa. ||ee - jmn (xe, ł, egoo 1
Video Generation with Motion-Aware Diffusion. After obtaining multi view images, we extend the diffusion framework to synthesize temporally coherent videos conditioned on motion cues. The generated images from preceding clips serve as reference frames to guide the denoising of subsequent noisy latents x,. The diffusion model exd takes both x, and encoded reference features Feer, concatenated along the temporal dimension, and applies temporal self-attention layer to capture motion correspondences, with the relative ego poses P also encoded for motion aware conditioning.
5

---

Only the temporal attention layers are fine-tuned from the pre-trained image diffusion model, enabling efficient transfer from spatial temporal domains. The training objective follows the denoising vid (x,, I, ret: Pet: C) During inference, an autoregressive formulation: Liif E x ||| & d
strategy employed for streaming video generation, where previously generated frames are reused as motion references to ensure smooth transitions and temporal coherence across clips.
3.3 Large-Scale Scene Extrapolation and Reconstruction
Building on single-chunk generation, we propose progressive extrapolation approach that coherently expands occupancy and images across multiple chunks, maintaining geometric and visual consistency with the generated multi-view videos for downstream applications.
Contrsstt- Scene Outpainting. We extend the occupancy field via triplane extrapola- tion [91]. which decomposes the task into extrapolating three orthogonal 2D planes, as illustrated in Fig. 4. The core idea is to generate a new latent plane h by synchronizing its denoising process with the forward diffusion of known reference plane h guided by an overlap mask M. Specifically, at each denoising step t, the new latent is updated as:
new 1-1
(Vā,hu + VI - āre) O + 5o (h,
axc
DEW (1-
(3)
where e ~ M and ā, is determined by the noise scheduler at timestep f. This method preserves structural consistency in overlapping regions while plausibly extending reference content into unseen areas, yielding coherent and geometry- consistent scene extensions.
Cbee
Trplen
Mer
My -
Visual-Coherent Image Extrapolation. Beyond occupancy outpainting, we fur- ther extrapolate the visual field for syn- chronized image generation. To main- tain visual coherence between the refer- ence image x and the new view x naive approach warps xfe using the camera pose (R, T' and applies image in- painting (Fig. 4). However, using only warped images as conditions is inadequate. To address this, we fine-tune the diffu- sion model &g with explicit conditioning on x and camera embeddings e [R T). Concretely, x is concatenated with the novel latent x and e(R, I') is incorpo-
(6)
Gesmetay- Connistent Triplane Ootpointing
(1,77
Cames
Inage
(e) Connirtency-assn Scone Extropslction
Visual- CohannCt Intege Estropolation
Figure 4: Illustration of (a) consistency-aware out- painting: (b) Occupancy triplane extrapolation is de- composed into three 2D plane extensions guided by over- lapped regions; (c) Image extrapolation is performed via diffusion conditioned on images and camera parameters.
rated via cross-attention, enabling view-consistent extrapolation with photorealistic visual results.
4 Experiments
4.1
Experimental Settings
We use Oce3ID-nuSeenes [92] to train the occupancy module and nuScenes [93] for the multi-view image and video generation modules. Additional implementation details are provided in the appendix.
Experimental Tasks and Metrics. We evaluate X'-Scene across three aspects using range of metrics: 1) Occupancy Generation: We evaluate the reconstruction results of the VAE with loU and mloU metrics. For occupancy generation, following [59], we report both generative 3D and 2D metries, including Inception Score, FID, KID, Precision, Recall, and F-Score. 2) Multi- Image Generation: We evaluate the quality of the synthesized images using FID. 3) Multi- Video Generation: We evaluate video temporal consistency using FVD. 4) Downstream Tasks: We evaluate the sim-to-real gap by measuring performance on the generated scenes across downstream

---

Lorge-Scale Image and Occupancy Generation
Lorge-Scale Scene Reconstruction
Legoue
Cenerated Geverated
Occupancy
Plenag
User Prompt Generation and Editing
User Prompt Generation and Editing
Figure 5: Versatile generation capability of X'-Scene: (a) Generation of large-scale, consistent semantic occupancy and multi-view images, which are reconstructed into 3D scenes for multi-view rendering; (b) User-prompted layout and scene generation, along with scene geometry editing.
Table 1:
Comparisons of occupancy reconstruction of the VAE. The downsampled size is reported in terms of spatial dimensions (H, W) and feature dimension (C).
OceSora [72] (VQVAE)
OccWorld [?]]
OceL.Lama [94]
UniScene [18]
X'-Soene (Ours)
Method
(VQVAE)
(VQVAB)
(VAE)
(Triplane-Val VAE)
Downsampled Size
(T/8,25,25.512)
(50.50.128)
(50,50.128)
(50,50,8)
(50,50.8)
(100,100,16)
m
27.4
66.4
65.9
72.9
73.7
92.4
IoU f
37.0
62.3
57.7
64.1
65.1
85.6
Table 2: Comparisons of 3D occupancy generation. We report Inception Score (IS), Fréchet Inception Distance (FID), Kernel Inception Distance (KID), Precision (P), Recall (R), and F-Score (F) both the 2D and 3D domains. denotes unconditioned generation, while other methods are evaluated using layout conditions. All methods are implemented using official codes and checkpoints.
Metric
Metric
Method
#Classes
1t
p
R
11
FID
KID
1
R
DynamicCity! [59]
1,008
7.792
8o-3
0.108
0.009
0.017
1,269
1890
0.369
0.028
UniScene [18]
11
1.015
0.728
Se-4
0.295
0.572
0.389
1.278
495.6
0.027
0.387
0.482
0.429
J'-Scene (Ours)
1.030
0.275
6e-5
0.744
0.772
0.757
1.287
281.3
0.009
0.766
0.785
0.775
UniSeene [18]
1.023
0.770
6c-4
0.259
0.588
0.360
1,235
529.6
0.024
0.382
0.412
0.396
17
X'-Scene (Ours)
1,028
0.262
6e-5
0.762
0.811
0.785
1.276
258.8
0.004
0.769
0.787
0.778
tasks, including semantic occupancy prediction (loU, mloU), 3D object detection (mAP, NDS), BEV segmentation (mloU), and end-to-end planning with UniAD (trajectory L2 eтror and collision rate).
4.2 Qualitative Results
Large-Scale Scene Generation. The upper part of Figure showcases large-scale scene generation results. By iteratively applying consistency-awert outpainting, X'-Scene effectively expands local regions into coherent, large-scale driving scenes. The generated scenes can be further reconstructed into 3D representations, enabling view rendering and supporting downstream perception tasks. Beyond static environments, our pipeline also produces temporally coherent multi-view videos (see Sec. 4.3 and Fig. 7 for qualitative and guantitative results).
User-Prompted Generation and Editing. The lower part of Figure 5 demonstrates the flexibility of S'Sccen in interactive scene generation, supporting both user-prompted generation and geometric editing. Users can provide high-level prompts (e.g., "create a busy intersection"). which are processed to generate corresponding layouts and scene content. Furthermore, given an existing scene, users can specify editing intents (e.g., "remove the parked car") or adjust low-level geometric attributes. Our pipeline updates the scene graph accordingly and regenerates the scene through conditional diffusion.
7

---

Table 3: Comparisons of multi-view image generation. We report FID and evaluate generation fidelity by performing BEV segmentation [95] and 3D object detection [96] tasks on the generated
data from the validation set. Bold indicates the best, and underline denotes the second-best results.
Synthesis Resolution
BEV Segmentation
3D Object Detection
Method
Avenue
FID4
Road mloU ?
Vehicle mloU
mAP
NDS
Original nuScenes [93]
73.67
34.82
35.54
41.21
BEVGen [36]
RA-L'24
224xx400
25.54
50.20
5.89
BEVControl [37]
23
24.85
60.80
26.80
DriveDreamer [3]
2CCV
256x448
26.80
MagieDrive [1]
ICLR'24 24
224x400
16.20
61.05
27.01
12.30
23.32
Panacea [5]
CVPR
256x512
16.96
55.78
22.74
11,58
22.31
Drive-WM [19]
CVPR'24
192x384
15.80
65.07
27.19
DreamForge [23]
arXiv' 25
224x4000
14.61
65.27
28.36
13.01
22.16
Glad [14]
ICLR'25
256x512
12:57
X--Seene (Ours)
224x400
11.29
66.48
29.76
16.28
26.26
X-SSene (Ours)
336
12.83
68.66
32.67
24.92
32.48
X'-Scene (Ours)
448 800
12.77
69.06
33.27
27.65
34.48
Table 4: Comparison of multi-view video generation. We report FVD and assess generation fidelity by evaluating end-to-end planning performance using UniAD [96] on the generated validation data.
Data Source
Synthesis
JVD
3DOD
BEV Segmentation mIoU (%)
1,2 (m)
Col. Rate (%)
Resolution
|MPP
NDS
Lanest Drivablet Dividert Crosingt
1.0s 2.0s 3.0s
Avg.
1.00s 2.0s 3.0s Avg.
Ori muScenes
224 400
31,20
45.22
29.19
65.83
23.51
12.99
0.60
1,10 1.85 1,18
[0.08
0.28 0.66 0.34
MrvieDrive [I]
224 400
217.9
12.92
28.30
21.95
51.46
17.10
5.25
0.57
1.14
1.95
1.22
00.10
0.25
0.70 0.35
DreamForge [23]
224 400
209.9
16.63
30.57
26. 16
58.98
20.22
8.83
0.55
1,08
1.85
1.16
0.08
0.27
0.81 0.39
X'Seeene (Ours)
224 400
179.7
20.40
31.76
28.04
61.96
22.32
10.48
0.55
1.08
1.81
1.15
0.03
0.13 0.66 0.27
4.3 Main Result Comparisons
Occupancy Reconstruction and Generation. Table presents the comparative occupancy re- construction results. The results show that Xcene achieves superior reconstruction performance, significantly outperforming prior approaches under similar compression settings (e.g., +0.8% mloU and +2.5% loU compared to UniScene [18]). This improvement is attributed to the enhanced capacity of our triplane representation to preserve geometric details while maintaining encoding efficiency.
Table 2 presents the quantitative results for 3D occupancy generation. Following the protocol in [59], we report performance under two settings: (1) a label-mapped setting, where 11 classes are evaluated by merging similar categories (e.g., car, bus, truck) into unified "vehicle" class, and (2) the full 17-class setting without label merging. Our approach consistently achieves the best performance across both 2D and 3D metrics. Notably, in the 17-class setting without label mapping, substantial improvements, with FID reduced by 51.2% (258.8 vs. 529.6). highlighting our method's capacity for fine- -rainge category distinction. Additionally, our method demonstrates strong precision and recall, reflecting its ability to generate diverse yet semantically consistent occupancy.
observe
Image Generation Fidelity. Table presents the results of multi-view image generation, including FID scores and downstream task evaluations. Notably, X'-Scene supports high-resolution image generation with competitive fidelity, which crucial for downstream tasks like 3D reconstruction. The results show that X'SSeene achieves the best FID, with 4.91 % improvement over the baseline [[], indicating superior visual realism. Moreover, X-Scene consistently outperforms other methods in BEV segmentation and 3D object detection resolution increases. For BEV segmentation in particular, performance on generated scenes at 448 x 800 resolution closely matches that on real data, showcasing X' R'eene'ss strong conditional generation aligned with downstream visual applications.
Video Generation Fidelity. Table 4 presents the results of dynamic video generation and end-to- end evaluation. X'SScene is trained on short 7-frame clips using an autoregressive temporal modeling strategy. It achieves a lower FVD than the 16- frram--trainne baseline MagicDrive, indicating stronger temporal consistency and video realism with higher efficiency. When evaluated on downstream perception and planning tasks using UniAD, X-Scene consistently outperforms the baseline across all metrics. These results demonstrate that X' -Seene generates temporally coherent and physically consistent dynamic scenes, effectivvly supporting realistic end-to-end simulation.
8

---

Table 5: Comparisons of training support for semantic occupancy pre-
Table 6: Comparison of training support for BEV segmenta- tion (Baseline as CVT [95]) and 3D object detection (Baseline as StreamPETR [98] following the setup in [23, 51).
diction (Baseline CONet [97].
Input Modality
33D Oce Pred.
3D Object Detection
BEV Segmentation
Data Source
Data Type
Data Source
|oU f mloU
mAP
mAoF.
Rd. mloU 1 Veh. mloU
Ori nuScenes
20.1
12.8
Real
Ori nuSeenes
34.5
46.9
59.4
74.30
36,00
+MagicDrive [1]
2D
21.8
13.9
+UniScene [1S]
(Images)
28.6
16.5
Panacea [5]
22.5
36.1
72.7
+. -'--eenn (Ours)
29.1
17.2
Gen.
DreamForge [23]
26.0
41.1
62.2
6..80
28.60.1.9
X-Seene (Ours)
28.2
43.4
61.0
68.41
29.23
Ori nuScenes
30.9
15.8
+ +UniScene [18]
---ene (Ours)
(LiDAR/Oce)
3D
33.1
19.3
Real+Gen
MavicDrive [I]
UniScene [ [18]
Vista [2]
34.0
36.5
35.4
41.2
38.6
76.62 1.32
79.56-8.2
81.69-1.9
41.62-662
40.34-134
L7,71
Ori nuScenes
29.5
20.1
DreamForge [23]
36.6
49.5
52.9
UniScenee [1S]
2D+3D
35.4
23.9
Panacea [5]
37.1
49.2
54.2
+.I--eene (Ours)
37.1
26.3
2' X-Sceme (Ours)
39.9
$1.6
51.2
83.37
43.05 2008
Cocaited
Figure 6: Qualitative comparison of joint voxel-and-image generation. Our method achieves superior consistency between generated 3D occupancy and 2D images compared to UniScene [18].
Downstream Tasks Evaluation. We evaluate the effectiveness of the generated scene data in supporting downstream model training. Table 5 reports the results for 3D semantic occupancy prediction. Fine-tuning with our synthesized 3D occupancy grids notably improves the baseline performance (+4.9% IoU, +6.8% mioU), as the high- spatial structures that enable better geometric reasoning and feature learning. Moreover, integrating 2D and 3D modalities yields the highest performance, demonstrating the importance of multimodal aligmment. Table presents the results for 3D object detection and BEV segmentation. Our generated data consistently surpasses all synthetic data baselines, verifying the superior fidelity, realism, and temporal consistency of our approach. Overall, these results confirm the potential of our synthesized images and videos to serve as high-quality data augmentation for downstream models.
grids provide accurate and detailed
Qualitative Comparisons. Figure 6 illustrates a comparison of joint voxel-and-image generation results. X'SScene produces more realistic and diverse scenes while maintaining tighter geometric aligmment between 3D occupancy and 2D images, leading to improved mross- coherence. Figure 7 further showcases qualitative results of multi-view video generation. X'-Scene generates temporally coherent sequences with smoother motion transitions and stable object dynamics, while maintaining accurate cross-view geometry and visual consistency. Together, these results demonstrate X'Sccene's s ability to generate spatially coherent 3D structures and photorealistic, temporally consistent videos, offering a scalable and reliable foundation for simulation and data generation.
4.4 Ablation Study
Effects of Designs in Occupancy Generation. As shown in Table 7, the proposed triplane de- formable attention module improves performance, particularly at lower resolutions. For instance, at a (50, 50, 16) resolution, introducing deformable attention yields gains of +1.9% IoU and +2.4% mIoU, confirming its role in alleviating feature degradation caused by downsampling. We further examine the impact of conditioning strategies. Removing either the additive layout condition or the box condition leads to noticeable performance drops, highlighting their complementary contributions. These conditions provide essential fine-grained geometric cues that guide the model to better capture scene structure and spatial context, ultimately improving occupancy field accuracy.

---

Table 7: Ablation study for designs in the oc- cupancy generation model.
Table 8:
Ablation study for designs in the multi- view image generation model.
Variants
Triplane Resolution
loUt mloUr
FID
Variants
FID
3D Detection
BEV Segmentation
MAP 1 NDS
Rd. mloU
Voh. mloU
DE Scene (Our))
(100,100.16)
85.6
92.4
258.8
0.778
Sr-oonee (Ours)
111.29]
16.12
26.26
66.48
w/ VAE deform stta
(50,50.16)
66.6
76.6
436.1
0.522
wo VAE deform altn
(50,50,16)
64.7
74.2
462.4
0.510
w/o semantic
12.23
15.27
25.59
65.75-am
28.71
deform
(100,100.16)
91.8
266.4
0.762
w/o depth mup
12.94
25.98
64.87-141
29.22.uus
w/ perspective map
16.87
13.15
22.37
63.35
27.13
w/o layout Condition
(100,100.16 (6)
85.6
92.4
1584
0.237
w/o position embed
11.38
15.60
26.16
66.46
27.88.177
w/o box Condition
(100,100,16)
85.6
92.4
271.4
0.751
w/o text description
12.60
15.54
26.06
66.26-a.2
29.47
View
Visw
Figure 7: Qualitative comparison of multi-view video generation. Our method demonstrates superior temporal consistency across frames and spatial coherence among multiple camera views.
Effects of Designs in Image Generation. Table 8 presents the ablation results for various condi- tioning components in the image generation model. Removing the semantic or depth maps that are rendered from 3D occupancy significantly degrades FID and downstream performance, highlighting their importance in providing dense geometric and semantic cues. Excluding the perspective map, which encodes projected 3D boxes and lanes, also reduces downstream performance (with mAP dropping by 2.97%), underscoring its role in conveying explicit layout priors. The 3D positional embedding is particularly critical for object detection, it enhances localization and spatial represen- tation. Finally, removing the text description degrades generation fidelity (FID worsening by 1.31 1)), showing that rich linguistic context aids fine-grained appearance modeling and scene understanding.
5 Conclusion and Limitations
In this paper, we present X'Sccene, a novel framework for 3D driving scene generation that achieves high fidelity, flexible controllability, and large-scale spatial and temporal consistency. Leveraging the multi-granular control mechanism, X'-Scene allows intuitive yet precise specification of both high- level semantic guidance and low-level geometric details. Its unified voxel-image-videe generation pipeline captures detailed 3D geometry, photorealistic appearance, and temporally coherent dynamics, while consistency-awant outpainting maintains spatial coherence across expansive environments. Extensive experiments show that X'Scene outperforms existing approaches in generation quality, controllability, and scalability, establishing it as versatile tool for large-scale data generation, driving simulation, and interactive scene exploration. Future work will explore longer temporal horizons and multi-agent interactions to further enhance the realism and dynamism of generated driving scenarios.
Acknowledgments
This research was supported by the Tier 2 Grant (MOE-T2EP2012240015 from the Singapore Min- istry of Education and by the National Natural Science Foundation of China (Grant No. 62525309).
10

---

References
[1] Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhenguo Ii, Dit-Yan Yeung, and Qiang Xu. Mag- iedrive: Street view generation with diverse 3d geometry control. arXiv preprint arXi::311000601, 2023.
[2] Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. arXiv preprint arXiv:2405. 17398, 2024.
[3] Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu.. Drivedreamer: Towards worlddoerd world models for autonomous driving. In European Conference Computer Vision, pages 55-72. Springer, 2024.
[4] Xiaofan Li, Yifu Zhang, and Xiaoqing Yc. Drivingdiffusion: Layout-guice multi-view driving scenarios video generation with latent diffusion model. In European Conference on Computer Vision, pages 469-485. Springer, 2024.
[5] Yuqing Wen, Yucheng Zhao, Yingfei Liu, Fan Jia, Yanhui Wang, Chong Luo, Chi Zhang, Tiancai Wang, Xiaoyan Sun, and Xiangyu Zhang. Panacea: Panoramic and controllable video generation for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6902 6912. 2024.
[6] Binyuan Huang. Yuqing Wen, Yucheng Zhao, Yaosi Hu, Yingfei Liu, Fan Jia, Weixin Mao. Tiancai Wang. Chi Zhang, Chang Wen Chen, et al. Subjectdrive: Scaling generative data autonomous driving via subject control. In Proceedings of the AAA! Conference on Artifcial Intelligence, volume 39, pages 3617-3625, 2025.
[7] Ruiyuan Gao, Kai Chen, Bo Xiao, Lanqing Hong. Zhenguo Li, and Qiang Xu. Magicdrivedit: rightteesolutin long video generation for autonomous driving with adaptive control. arXiv preprint arXiv:2411.13807, 2024.
[8] Leheng Li, Weichao Qiu, Yingjie Cai, Xu Yan, Qing Lian. Bingbing Liu, and Ying-Cong Chen. Syn- theoce: Synthesize coomerric-onntrole street view images through 3d semantic mpis. arXiv preprint arXiv:2410.00333,, 2024.
[9] Hannan Lu, Xiaohe Wu, Shudong Wang, Xiameng Qin, Xinyu Zhang, Junyu Han, Wangmeng Zuo, and Ji Tao. Seeing beyond views: Multi-view driving scene video generation with holistic attention. arXiv preprint arXiv:2412.0335 0332200 2024.
[10] Wei Wu, Xi Guo, Weixuan Tang, Tingxuan Huang, Chiyu Wang, Dongyue Chen, and Chenjing Ding. Drivescape: Towards high-resolution controllable multi-vies view driving video generation. arXiv preprint arXiv:2409.05460 2024.
[11] Junpeng Jiang, Gangyi Hong, Miao Zhang, Hengtong Hu, Kun Zhan, Rui Shao, and Liqiang Nie. Dive: Efficient multi-view driving scenes generation based on video diffusion transformer. arXiv preprint arXiv:2504. 100044/9//// /96/4, 2025.
[12] Rui Chen, Zehuan Wu. Yichen Liu, Yuxin Guo, Jingeheng Ni, Haifeng Xia, and Siyu Xia. Unimlyg: Unified framework for multi-view long video generation with comprehensive control capabilities for autonomous driving. arXiv preprint arXiv:2412.04842,, 2024.
[13) Yishen Ji, Ziyue Zhu, Zhenxin Zhu, Kaixin Xiong, Ming Lu, Zhiqi Li, Lijun Zhou, Haiyang Sun, Bing Wang, and Tong Lu. Cogen: 3d consistent video generation via adaptive conditioning for autonomous driving. arXiv preprint arXiv:2503.22 22231,, 2025.
[14] Bin Xie, Yingfei Liu. Tiancai Wang, Jiale Cao, and Xiangyu Zhang. Glad: streaming scene generator for autonomous driving. arXiv preprint (xXx:2503.0004. 2025.
[15] Yichen Xie, Chenfeng Xu, Chensheng Peng, Shuqi Zhao, Nhat Ho, Alexander T Pham, Mingyu Ding, Masayoshi Tomizuka, and Wei Zhan. X-drive: Cross-modality consistent multi-sensor data synthesis for driving scenarios. arXiv preprint arXiv:2411.01123 2024.
[16] Zehuan Wu, Jingcheng Ni, Xiaodong Wang. Yuxin Guo, Rui Chen, Lewei Lu, Jifeng Dai, and Yuwen Xiong. Holodrive: Holistic 2d-3d multi-modal street scene generation for autonomous driving. arXiv preprint arXiv:2412.01407, 2024.

---

[17] Jiachen Lu, Ze Huang, Zeyu Yang. Jiahui Zhang, and Li Zhang, Wovogen: World volume- aware diffusion for controllable multi-camera driving scene generation. In European Conference on Computer Vision, pages 329-345. Springer, 2024.
[18] Bohan Li, Jiazhe Guo, Hongsi Liu, Yingshuang Zou, Yikang Ding, Xiwu Chen, Hu Zhu, Feiyang Tan, Chi Zhang, Tiancai Wang, et al. Uniscene: Unified oceupancy-centr driving scene generation. arXiv preprint 242 2024.
[19] Yugi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2024.
[20] Enhui Ma, Lijun Zhou, Tao Tang, Zhan Zhang, Dong Han, Junpeng Jiang, Kun Zhan, Peng Jia, Xianpeng Lang, Haiyang Sun, et al. Unleashing generalization of end-to and-to-end autonomous driving with controllable long video generation. arXiv preprint arXiv:2406.01344 2024.
[21] Yanhao Wu, Haoyang Zhang, Tianwei Lin, Lichao Huang. Shujie Luo, Rui Wu, Congpei Qiu, Wei Ke, and Tong Zhang. Generating multimodal driving scenes via next-scene prediction. arXiv preprint arXiv:2503. 14945, 2025.
[22] Xuemeng Yang, Licheng Wen, Yukai Ma, Jianbiao Mei, Xin Li, Tiantian Wei, Wenjie Lei, Dancheng Fu, Pinlong Cai, Min Dou, et al. Drivearena: A closed-loop generative simulation platform for autonomous driving. arXiv preprint arXiv:2408.0044 2024.
[23] Jianbiao Mei, Tao Hu, Xuemeng Yang, Licheng Wen, Yu Yang. Tiantian Wei, Yukai Ma, Min Dou, Botian Shi, and Yong Liu. Dreamforge: scenes. arXiv preprint arXiv:2409,0400 2024.
aware autoregressive video generation for multi-view driving
[24] Tianyi Yan, Dongming Wu, Wencheng Han, Junpeng Jiang, Xia Zhou, Kun Zhan, Cheng-zhong Xu, and Jianbing Shen. Drivingsphere: Building high high-fidelity 4d world for closed-loop simulation. arXiv preprint arxiv:2411. 2024.
[25] Chaojun Ni, Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Wenkang Qin, Guan Huang, Chen Liu, Yuyin Chen, Yida Wang, Xueyang Zhang, al. Recondreamer: Crafting world models for driving scene reconstruction via online restoration. arXiv preprint arXiv:2411. 1..22411.9999 1.19488, 2024.
[26] Junhao Ge, Zuhong I. Liu, Longteng Fan, Yifan Jiang, Jiaqi Su, Yiming Li, Zhejun Zhang, and Siheng Chen. Unraveling the effects of synthetic data on end-to-end autonomous driving. arXiv preprint arXiv:2503. 18108, 2025.
[27] Lening Wang. Wenzhao Zheng, Dalong Du Yunpeng Zhang, Yilong Ren, Han Jiang, Zhiyong Cui, Haiyang Yu, Jie Zhou, Jiwen Lu, et al. Stag-1: 1: Towards realistic 4d driving simulation with video generation model. arXiv preprint arXiv:2412.05280, 2024.
[28] Yue Liao, Pengfei Zhou, Siyuan Huang, Donglin Yang. Shengcong Chen, Yuxin Jiang, Yue Hu, Jingbin Cai, Si Liu, Jianlan Luo, Liliang Chen, Shuicheng Yan, Maoqing Yao, and Guanghui Ren. Genic envi- sioner: A unified world foundation platform for robotic manipulation. arXiv preprint 2xXi:25088006333 2025.
[29] Jumin Lee. Sebin Lee, Changho Jo, Woobin Im, Juhyeong Seon, and Sung-Eui Yoon. Semcity: Semantic scene generation with triplane diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 08337-28347, 2024.
[30] Yifan Lu, Xuanchi Ren, Jiawei Yang, Tianchang Shen, Zhangjie Wu, Jun Gao, Yue Wang, Siheng Chen, Mike Chen, Sanja Fidler, et al. Infinicuhe: Unbounded and controllable dynamic 3d driving scene generation with world-guided video models. arXiv preprint arX1v:44/2.03934, 2024.
[31] Bernhard Kerbl, Georgios Kopanas, Thomas Leimküihler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139-1, 1, 2023.
[32] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 0051-881,222 2020.
[33] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXivv2010.02502, 2020.
[34] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
12

---

[35] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung, Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22563-22575. 2023.
[36] Alexander Swerdlow, Runsheng Xu, and Bolei Zhou. Street-view image generation from a bird's-eye view layout. IEEE Robotics and Automation Letters, 2024.
[37] Kairui Yang, Enhui Ma, Jibin Peng, Qing Guo. Di Lin, and Kaicheng Yu. Bevcontrol: Accurately controlling street-view elements with mulsppperseective consistency via bev sketch layout, arXiv preprint arXiv:2308.0166 2023.
[38] Valentina Mugat, Daniele De Martini, Matthew Gadd, and Paul Newman. Neuralftoors: Conditional street-level scene generation from bev semantic maps via neural fields. IEEE Robotics and Automation Letters, 9(3):2431 1-24888, 2024.
[39] Valentina Muşat, Daniele De Martini, Matthew Gadd, and Paul Newman. Neuralfloorss+: Consistent street-level scene generation from bev semantic maps. In 2024 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 12872-12879, IEEE, 2024.
[40] Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Xinze Chen, Guan Huang. Xiaoyi Bao, and Xingang Wang. Drivedreamer-2: Lim enhanced world models for diverse driving video generation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pages 104200 2025.
[41] Guosheng Zhao, Chaojun Ni, Xiaofeng Wang, Zheng Zhu, Xueyang Zhang. Yida Wang. Guan Huang, Xinze Chen, Boyuan Wang, Youyi Zhang, al. Drivedreamer4d: World models effective data machines for driving scene representation. arXiv preprint urXiv:2410.13571, 2024.
[42] Lingdong Kong, Wesley Yang, Jianbiao Mei, Youquan Liu, Ao Liang, Dekai Zhu, Dongyue Lu, Wei Yin, Xiaotao Hu, Mingkai Jia, et al. 3d and 4d world modeling: A survey. arXiv preprint arXiv:2509007996. 2025.
[43] Vlas Zyrianov, Xiyue Zhu, and Shenlong Wang. Learning to generate realistic lidar point clouds. In European Conference Computer Vision, pages 17-35. Springer, 2022.
[44] Martin Hahner, Christos Sakaridis, Mario Bijelic, Felix Heide, Fisher Yu, Dengxin Dai, and Luc Van Gool. Lidar snowfall simulation for rohust 3d object detection. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 13364416374, 2022.
[45] Yuwen Xiong, Wei-Chiu Ma, Jingkang Wang, and Raquel Urtasun. Ultralidar; Learning compact representations for lidar completion and generation. arXiv preprint arXiv:2311.01444 2023.
[46] Haoxi Ran, Vitor Guizilini, and Yue Wang. Towards realistic scene generation with lidar diffusion models. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 14738-14748, 2024.
[47] Vlas Zyrianov, Henry Che, Zhijian Liu, and Shenlong Wang. Lidardm: Generative lidar simulation in a generated world. arXiv preprint arXiv:2404.02003 2024.
[48] Qianjiang Hu, Zhimin Zhang, and Wei Hu. Rangeldm: Fast realistic lidar point cloud generation. In European Conference Computer Vision, pages 115-135. Springer, 2024.
[49] Yang Wu, Kaihua Zhang, Jianjun Qian, Jin Xie. and Jian Yang. Texı2lidar: Text-guided lidar point cloud generation via equirectangular transformer. European Conference on Computer Vision, pages 291-310. Springer, 2024.
[50] Tai- Yu Pan, Sooyoung Jeon, Mengdi Fan, Jinsu Yoo, Zhenyang Feng, Mark Campbell, Kilian Q Wein- berger, Bharath Hariharan, and Lei Chao. Transfer your perspective: Controllable 3d generation from any viewpoint in driving scene. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 12027-1200 12036, 2025.
[51] Ao Liang, Youquan Liu, Yu Yang. Dongyue Lu, Linfeng Lingdong Kong, Huaici Zhao, and Wei Tsang Ooi. Lidarcrafter: Dynamic world modeling from lidar sequenees. arXiv preprint arXiv:2508.. 03699, 2025.
[52] Ao Liang, Youquan Liu, Yu Yang, Dongyue Lu, Linfeng Li, Lingdong Kong, Huaici Zhao, and Wei Tsang Ooi. Learning to generate 4d lidar sequences. arXiv preprint arXiv:2509. 11959, 2025.
13

---

[53] Yuheng Liu, Xinke Li, Xueting Li, Lu Qi. Chongshou Li, and Ming-Hsuan Yang. Pyramid diffusion for fine 3d large scene generation. In European Conference on Computer Viston, pages 71-87. Springer, 2024.
[54] Jumin Lee, Woobin Im. Sebin Lee, and Sung-Bu Eui Yoon. Diffusion probabilistic models for scene-scale 3d categorical data. arXiv preprint arXiv:2301.00522 000227 2023.
[55] Junge Zhang, Qihang Zhang. Li Zhang, Ramana Rao Kompella, Gaowen Liu, and Bolei Zhou. Urban scene diffusion through semantic occupancy map. arXiv preprint wVXiv22003.1699 2024.
[56] Yu Yang, Jianbiao Mei, Siliang Du, Yilin Xiao, Huifeng Wu, Xiao Xu, and Yong Liu. Dqformer: Towards unified lidar panoptic segmentation with decoupled queries for large-scale outdoor scenes. IEEE Transactions on Geoscience and Remote Sensing, 2025.
[57] Xuanchi Ren. Yifan Lu, Hanxue Liang, Zhangjie Wu, Huan Ling, Mike Chen, Sanja Fidler, Francis Williams, and Jiahui Huang. Scube: Instant large-scale scene reconstruction using voxsplats. arXiv preprint 241.:2210022003333 2024.
[58] Xuanchi Ren, Jiahui Huang, Xiaohui Zeng, Ken Museth, Sanja Fidler, and Francis Williams. Xcube: Large-scale 3d generative modeling using sparse voxel hierarchies. In Proceedings of the IEEE/CVF conference on computer vision and pultern recognition, pages 4209-4219, 2024.
[59] Hengwei Bian. Lingdong Kong, Haozhe Xie, Liang Pan. Yu Qiao, and Ziwei Liu. Dynamiccity: Large- scale lidar generation from dynamic scenes. arXiv preprint arXiv: 18084, 2024.
[60] Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang. Xiaowei Zhou, and Sida Peng. Street gaussians: Modeling dynamic urban scenes with gaussian splatting. In European Conference on Computer Vision, pages 176. Springer, 2024.
[61] Ruiyuan Gao, Kai Chen, Zhihao Li, Lanqing Hong, Zhenguo Li, and Qiang Xu. Magicdrive3d: Control- lable 3d generation for any-view rendering in street scenes. arXiv preprint arXiv:2405. 14475,
[62] Jiageng Mao, Boyi Li, Boris Ivanovic, Yuxiao Chen, Yan Wang, Yurong You, Chaowei Xiao, Danfei Xu, Marco Pavone, and Yue Wang. Dreamdrive: Generative 4d scene modeling from street view images. arXiv preprint arXiv:2501.0006 000001, 2024.
[63] Jiawei Yang. Jiahui Huang. Yuxiao Chen, Yan Wang, Boyi Li, Yurong You, Apoorva Sharma, Maximilian Igl. Peter Karkus, Danfei Xu, et al. Storm: Spatio-temporal reconstruction model for large-scale outdoor scenes. arXiv preprint arXiv:2501.0060 00602, 2024.
[64] Yunzhi Yan, Zhen Xu, Haotong Lin, Haian Jin, Haoyu Guo, Yida Wang, Kun Zhan, Xianpeng Lang. Hujun Bao. Xiaowei Zhou, et al. Streetcrafter: Street view synthesis with controllable video diffusion models. arXiv preprint arXiv:2412. 13188 2024.
[65] Jiazhe Guo, Yikang Ding, Xiwu Chen, Shuo Chen, Bohan Li, Yingshuang Zou, Xiaoyang Lyu, Feiyang Tan, Xiaojuan Qi, Zhiheng Li, al. Dist-4d: Disentangled spatiotemporal diffusion with metric depth for 4d driving scene generation. arXiv preprint arXiv:2503. 152088 2025.
[66] Hanlin Chen, Chen Li, and Gim Hee Lee. Neusg: Neural implicit surface reconstruction with 3d gaussian splatting guidance. arXiv preprint arXiv:2312.00846, 2023.
[67] Hanlin Chen, Fangyin Wei, Chen Li, Tianxin Huang, Yunsong Wang, and Gim Hee Lee. Ver-gaus: View consistent depth-normal regularizer for gaussian surface reconstruction. arXiv preprint 24X.::2406.0577 2024.
[68] Lunjun Zhang, Yuwen Xiong, Ze Yang, Sergio Casas, Rui Hu, and Raquel Urtasun. Copilot4d: Learning unsupervised world models for autonomous driving via discrete diffusioo. arXiv preprint arX1v:2311.01017, 2023.
[69] Zetong Yang. Li Chen, Yanan Sun, and Hongyang Li. Visual point cloud forecasting enables scalable autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14673-14684, 2024.
[70] Ben Agro, Quinlan Sykora, Sergio Casas, Thomas Gilles, and Raquel Urtasun. Uno: Unsupervised occupancy fields for perception and forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14487-14496, 2024.
[71] Wenzhao Zheng, Weiliang Chen, Yuanhui Huang, Borui Zhang, Yueqi Duan, and Jiwen Lu. Oceworld: Learning 3d occupancy world model for autonomous driving. In European conference on computer vision, pages 55 72. Springer, 2024.
14

---

[72] Lening Wang, Wenzhao Zheng. Yilong Ren, Han Jiang. Zhiyong Cui, Haiyang Yu, and Jiwen Lu. Occsora: 4d occupancy generation models as world simulators for autonomous driving. arXiv preprint urXiv:2405.2083 203337 2024.
[73] Chen Min, Dawei Zhao, Liang Xiao, Jian Zhao, Xinli Xu, Zheng Zhu, Lei Jin, Jianshu Li, Yulan Guo, Junliang Xing, et al. Driveworld: 4d pre-trained scene understanding via world models for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2024.
[74] Yu Yang, Jianbiao Mei, Yukai Ma, Siliang Du, Wenqing Chen, Yijie Qian, Yuxiang Feng, and Yong Liu. Driving in the occupancy world: Vision-centric occupancy forecasting and planning via world models for autonomous driving. In Proceedings of the AAAI Conference on Artificial Intellignnce volume 39, pages 9327-9335, 2025.
[75] Songen Gu, Wei Yin, Bu Jin, Xiaoyang Guo, Junming Wang, Haodong Li, Qian Zhang, and Xiaoxiao Long. Dome: Taming diffusion model into high-fidelity controllable occupancy world model. arXiv preprint arXiv:2410. 10429, 2024.
[76] Jianbiao Mei, Yu Yang, Xuemeng Yang, Licheng Wen, Jiajun Lv, Botian Shi, and Yong Liu. Vision-centric 4d occupancy forecasting and planning via implicit residual world models, 2025.
[77] Andrew I Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa. Infinite nature: Perpetual view generation of natural scenes from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14458-14467, 2021.
[78) Boyang Deng, Richard Tucker, Zhengqi Li, Leonidas Guibas, Noah Snavely, and Gordon Wetzstein. Streetscapes: Large-scale consistent street view generation using autoregressive video diffusion. In ACM SIGGRAPH 2024 Conference Papers, pages 1-11, 2024.
[79] Yixun I Liang, Xin Yang, Jiantao Lin, Haodong Li, Xiaogang Xu, and Yingcong Chen. Lnciddreamer: Towards high-fideliyy text-to-3d generation via interval score matching. In Proceedings the IEFF/CVF conference on computer vision and pultern recognition, pages 6517-6526, 2024.
[80] Xing Yu, Haoyi Duan, Charles Herrmann, William T Freeman, and Jiajun Wu. Wonderworld: Interactive 3d scene generation from a single image. arXiv preprint arXiv:2406.09394 09394, 2024.
[81] Hong-Xing Yu, Hanyi Duan, Junhwa Hur, Kyle Sargent, Michael Rubinstein, William T Freeman, Forrester Cole, Deqing Sun, Noah Snavely, Jiajun Wu, et al. Wonderjourney: Going from anywhere to everywhere. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6658-6667, 2024.
[82] Mengqi Zhou, Jun Hou, Chuanchen Luo, Yuxi Wang. Zhaoxiang Zhang, and Junran Peng. Scenex: Procedural controllable large-scale scene generation via large-language models. arXiv e-prints, pages 2024.
[83] Jie Deng. Wenhao Chai, Junsheng Huang, Zhonghan Zhao, Qixuan Huang, Mingyan Gao, Jianshu Guo. Shengyu Hao, Wenhao Hu, Jenq-Neng Hwang, et al. Citycraft:: A real crafter for 3d city generation. arXiv preprint urXiv:2406.04988 049883 2024.
[84] Shougao Zhang. Mengqi Zhou, Yuxi Wang, Chuanchen Luo, Rongyu Wang, Yiwei Li, Zhaoxiang Zhang, and Junran Peng. Cityx: Controllable procedural content generation for unbounded 3d cities. arXiv preprint mxXiv22407.17577 2024.
[85] Chieh Hubert Lin, Hsin- Ying Lee, Willi Menapace, Menglei Chai, Aliaksandr Siarohin, Ming-Hsuan Hsuan Yang. and Sergey Tulyakov. Infinicity: Infinite-scale city synthesis. In Proceedings of the IEEE/CVF international conference on computer vision, pages
228118 2023.
[86] Haozhe Xie, Zhaoxi Chen, Fangzhou Hong, and Ziwei Liu. Citydreamer: model of unbounded 3d cities. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9666 967669675,2000 2024.
Compositional generative
[87] Haozhe Xie. Zhaoxi Chen, Fangzhou Hong, and Ziwei Liu. Gaussiancity: Generative gaussian splatting for unbounded 3d city generation. arXiv preprint ar Xix::24466666 065266,66666666 2024.
[88] Guangyao Zhai, Evin Pinar Ömek, Shun-Cheng Cheng Wu, Yan Di, Federico Tombari, Nassir Navab, and Benjamin Busam. Commonscenes: Generating commonsense 3d indoor scenes with scene graph diffusion. Advances in Neural Information Processing Systems, 36:30022-30000 2023.
15

---

[89] Guangyao Zhai, Evin Pinar Örnek. Dave Zhenyu Chen. Ruotong Liao, Yan Di, Nassir Navab, Federico Tombari, and Benjamin Busam. Echoscene: Indoor scene generation via information echo scene graph diffusion. In European Conference on Computer Vision. pages 167-184. Springer, 2024.
[90] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-awar 3d generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16123-161.5 2022.
[91] Zhennan Wu, Yang Li. Han Yan, Taizhang Shang. Weixuan Sun, Senbo Wang, Ruikai Cui, Weizhe Liu, Hiroyuki Sato, Hongdong Li, et al. Blockfusion: Expandable 3d scene generation using latent tri tri-plane extrapolation. ACM Transactions Graphics (TOG), 1344:1117, 2024.
[92] Xiaoyu Tian, Tao Jiang, Longfei Yun, Yucheng Mao, Huitong Yang. Yue Wang, Yilun Wang, and Hang Zhao. Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving. Advances in Neural Information Processing Systems, 36:64318 64330, 2023.
[93] Holger Caesar, Varun Bankiti, Alex H Lang, Sourahh Vora, Venice Erin Liong. Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 1 11621 11631, 2020.
[94] Julong Wei, Shanshuai Yuan, Pengfei Li, Qingda Hu, Zhongxue Gan, and Wenchao Ding. Oc- cllama: An saauaancy-langugg-actiie generative world model for autonomous driving. arXiv preprint arXiv:2409.03272 2024.
[95] Brady Zhou and Philipp Krähenbühl. Cross-view transformers for real-time map-view semantic segmen- tation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13760-13769, 2022.
[96] Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang. Huizi Mao, Daniela L Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified s-ede view representation. In 2023 IEEE international conference on robotics and automation (ICRA), pages 2774-2781. IEEE, 2023.
[97] Xiaofeng Wang, Zheng Zhu, Wenbo Xu, Yunpeng Zhang. Yi Wei, Xu Chi, Yun Ye, Dalong Du, Jiwen Lu. and Xingang Wang. Openoccupancy: A large scale benchmark for surrounding semantic occupancy perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17850- 11550-1785..20 2023.
[98] Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xiangyu Zhang. Exploring object-centric temporal modeling for efficient multi-view 3d object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages
2023.
[99] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 250:: 13923, 2025.
[100] Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: A unified medictor-correette framework for fast sampling of diffusion models. Advances in Neural Information Processing Systems, 36:49842-49866 -98699 2023.
[101] Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temport convnets: Minkowski convolutional neural networks. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3075-3066 3084,0 2019.
[102] Anton Obukhov, Maximilian Seitzer, Po- Wei Wu, Semen Zhydenko, Jonathan Kyl, and Elvis Yu-Jing Lin. High-fidelity performance metrics for generative models in pytorch, 2020. Version: 0.3.0, DOL:
[103] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2821 2016.
[104] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arxiv preprint arXiv:1409.1555 2014.
16

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Both contributions and scope have been discussed in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contribu- tions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper diseuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The detailed analysis on limitations have been discussed in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to vi- olations of these assumptions (c.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with a few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recogmition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by review- ers grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This is empirical study that excludes theory assumptions and proofs.
17

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main experimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: All information needed to reproduce the experimental results have been disclosed. To ensure reproducibility, code and data are committed to be publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility results, access to a hosted model (c.g., in the case a large language model), releasing of a model checkpoint, other means that appropriate the research performed.
also be provided via detailed instructions for how to replicate the
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way reproduce the model (e.g., with an open-souree dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access the model limited in some way (c.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
Open access data and code
Question: Does the paper provide open access the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The detailed implementation procedures have been included in the appendix. To ensure reproducibility, code and data committed to be publicly available.
18

---

Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https: /tl:://nnippp cc/
/oolded//Coeeumm CodeSubmi s lonPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed reproduce the results. See the NeurlPS code and data submission guidelines (https:
to
/ inips.. wwtpcccpppcccccccccccccccccccccc wtt///uieeeeeeee CodeSubmi s onPol i icy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve amonymity, the authors should release anonymized versions (if applicable).
Providing much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits. hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: All training and test details have been discussed in either main body or appendix. To ensure reproducibility, code and data are committed to be publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
Sufficient information about experiment settings have been discussed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
19

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric eтror bars that would yield results that are out of range (e.g, negative error rates). If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
Experiments compute resourees
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experi- ments?
Answer: [Yes]
Justification:
The details on computing resources have been discussed.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experi- mental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:://// /tteps://neee /ne///nuuriii ps. cc/ pubbl i a Etthi Gu de iiee?
Answer: [Yes]
Justification: This research follows the NeurlPS Code of Ethics properly.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., if there is a special consideration due laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The discussion on societal impacts has been included in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
20

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology heing used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification:
The discussion on safeguards has been included in the appendix.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The ucknowledgments on licenses have been included in the appendix.
Guidelines:
The answer NA means that paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thooee com/ datasets has cu- rated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged reach out to the asset's creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer:
[Yes]
Justification:
The discussions on new assets have been included in the appendix.
Guidelines:
The answer NA means that the paper does not release new assets.
21

---

Researchers should communicate the details of the dataset/ccdeemmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs it is an important, original, or non- standard component of the core methods in this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: An L.M is employed as an essential component of our core methodology (Section 3.1) translate high-level user prompts into structured scene descriptions.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standard components.
Please refer to our LLM policy (https neurige cc/Conferences /2025/ LLMM for what should or should not be described.
22

---

X'-Scene:
Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability
Supplementary Material
Contents
A Additional Implementation Details
24
A.1 Datasets
24
A.2 Model Implementation Details
................................
24
A.3 Evaluation Metrics for Occupancy Generation
25
Additional Details of Scene Description Generation
26
B.1 Scene Description Memory Construction
26
B.2 Novel Scene Description Generation
26
B.3 Prompt Details and Scene Description Examples
27
Additional Quantitative Results
C.1 Effect of Spatial Conditioning
31
C.2 Effect of Layout Type
................................
Robustness and Efficiency
................................
C,4 Effect of Setrieval-Augmenne Generation
31
Additional Qualitative Results
32
D.1 Conditional Occupancy and Image Generation
32
D.2
Text-to-Scene Generation
36
D.3
Large-Scale Scene Generation
36
Potential Societal Impact & Limitations
36
E. Societal Impact
36
E.2 Known Limitations
36
F Public Resources Used
37
F1 Public Datasets Used
37
F.2 Public Implementations Used
37
23

---

A Additional Implementation Details
In this section, we provide additional implementation details to facilitate reproducibility. Specifically, we elaborate on the experimental datasets, model implementation, and the evaluation metrics.
A.1 Datasets
We use Occ3D-nuSeenes [92] to train our controllable occupancy generation module, and nuScenes [93] for the multi-view image and video generation modules. The textual scene graph- to-layout generation module also trained using 3D bounding box and HD map annotations from nuScenes. The dataset comprises 1,000 driving scenes under diverse weather, lighting, and traffic conditions. Each 20-second scene includes about 40 annotated keyframes, yielding roughly 40,000 samples with 360 multi-viee images, 3D occupancy, bounding boxes, and maps. We follow the standard split of 700 training and 150 validation scenes. For video generation, ASAP interpolation is applied to upsample the frame rate from 2 Hz 12 Hz, yielding about 240 frames per scene and enabling more consistent training for temporally coherent video synthesis. Following DynamicC- ity [59], map the original 17 semantic categories to 11 commonly used classes (see Table 9) and conduct experiments both with and without label mapping to enable comprehensive comparisons.
Table 9: Summary of Semantic Label Mappings. We map the original 17-class nuScenes semantic
labels to 11 classes following the protocol in [59] to enable comprchensive evaluation.
Bailding
Barrior
Free
Pedestrian
Pole
Road
Ground
Sidewalk
Vegetation
Vehicle
Bicyele
Mapped Class 11
Original Class 17
Manmade
Barrier
Free
Pedestrian
Traffic
come
Driveable surface
Other flat.
Terrain
Sidewalk
Vegetation
Bus, Car
Const vel.
Trailer, Truck Blgole, Motor
A.2 Model Implementation Details
Textual Scene Description Generation Module. To construct the scene description memory bank M,, we utilize QWen2.5-VL [99] to extract structured information from nuScenes. For each frame, six surround-view images are jointly processed to generate holistic scene descriptions, which are parsed into scene style S, foreground objects O, and background elements B. Concurrently, 3D bounding boxes and lane markings are converted into textual scene-graph layouts L. These components collectively form memory entries m, (8,0,
For retrieval, text descriptions are encoded using OpenAl's text- eeatt-eebedigg---iieeeee 3-mall model and indexed with FAISS to enable efficient similarity search. During inference, given a coarse prompt Tp, we retrieve the top-K relevant entries from M,, which are then combined with the prompt and fed into GPT-4o generate detailed and structured scene description D. Please refer Sec. B for further details and example illustrations.
Scene-Graph to Layout Generation Module. For the scene-graph layout generation module, training and evaluation were conducted on single NVIDIA A6000 GPU with 48GB of memory. We employed batch size of 128 and trained the model for 400 epochs. The optimization was performed using the AdamW optimizer with an initial learning rate of x 10 and a cosine annealing scheduler. To ensure stable training and consistent representation, the 3D bounding boxes were normalized using dataset-specific parameters. Each bounding box b; was parameterized by its center coordinates (x, y, =), dimensions (1, w. it), and yaw angle 0. Following standard practices in 3D object detection, we normalized the box center coordinates to the range (0, 1]. applied logarithmic transformation to the dimensions, and represented the yaw angle using its sine and cosine components. Each graph node was augmented with an 8-dimensional noise vector enhance robustness during training.
24

---

Occupancy Generation Module. For the occupancy generation module, the triplane- VAE encodes the original occupancy field with a resolution of 200 x 200 x 16 into a triplane representation of spatial dimensions (X,, Yn, 2n) = (100, 100, 16) and feature dimension C₁ = 16, reducing memory consumption while preserving structural details. The triplane-VAN is trained using the Adam optimizer with an initial learning rate of 1 10-3 and a step decay factor of 0.1, over 200 epochs on 4 NVIDIA A6000 GPUs with a batch size of 24 per GPU.
During diffusion, the three orthogonal planes are arranged into a unified square feature map by zero-padding the uncovered corners, forming tensor h E R 2n. +Z..Ch, Attention is applied across this tensor to capture inter-plane correlations. The diffusion model is trained from scratch using the AdamW optimizer with an initial learning rate of 1 10 and cosine scheduler, over 300 epochs with a batch size of 12 per GPU. For occupancy outpainting, we adopt the RePaint sampling strategy with resampling steps and a jump size of 20.
Multi- Vieli Image Generation Module. We initialize the multi-view image generation module with pretrained Stable Diffusion v2.1 weights, while randomly initializing newly added parameters. The diffusion model is trained on 4 NVIDIA A6000 GPUs with mini-batch size of 8, using the AdamW optimizer with a learning rate of 8 x 10 and a cosine learning rate scheduler over 200 epochs. After initial training a resolution of 224 x 400, we fine-tune the model for an additional 50K iterations at higher resolutions of 448 x 800 and 336 x 600. During inference, we use the UniPC [100] scheduler with 20 steps and a Classifier Guidance (CFG) scale of 1.2.
Multi-View Video Generation Module. We initialize the multi-view video generation module using the pretrained image diffusion U-Net and focus on fine-tuning the newly introduced temporal attention layers. The training is performed for 100 epochs with total batch size of where two reference frames are randomly sampled from the preceding five ground-truth frames, and each training sample contains 7 frames in total. For higher-resolution settings, we further train the model for 50K iterations, initializing from the corresponding lower-resolution weights. The temporal module is trained on eight NVIDIA A 100 GPUs using the AdamW optimizer with a learning rate of 8 x 10-5 and cosine learning rate scheduler.
During inference. the reference frames are drawn from previously generated video clips. For the first clip, we employ the single-frame image generation model to produce the initial reference frame, after which the system follows an autoregressive generation strategy. By default, two reference frames are used to generate the subsequent seven frames, enabling temporally coherent and geometrically consistent video synthesis across multiple views.
A.3 Evaluation Metrics for Occupancy Generation
Following the evaluation protocol of DynamicCity [59], we adopt two complementary strategies to assess the quality of occupancy generation:
3D Evaluation: We train sparse convolutional autoencoder based on the MinkowskiUNet [101] architecture to extract 3D features from generated occupancy fields. Features from the final down- sampling layer are aggregated via global average pooling and used to compute evaluation metrics using the Torch-Fidelity library [102].
2D Evaluation: We render the 3D occupancy fields into 2D images for image-based evaluation. To ensure fair comparison, we standardize the rendering process across all methods using consistent semantic color mappings and camera parameters. We compute IS, FID, and KID using a standard pretrained Inception [103] network, and use a VGG-16 [104] model for precision and recall. Both networks are fine-tuned on our semantically color-mapped dataset to ensure domain alignment.
To evaluate the quality and diversity of the generated samples, we use several quantitative metrics: 1) Inception Score (IS) measures both quality and diversity via the KL divergence between each image's s conditional label distribution and the marginal distribution, with higher scores indicating sharper and more diverse samples; 2) Fréchet Inception Distance (FID) computes the distance between real and generated distributions the Inception feature space, where lower values indicate higher fidelity: 3) Kernel Inception Distance (KID) calculates the squared Maximum Mean Discrepancy (MMD) between real and generated features using a polynomial kernel, and is unbiased and less sensitive to sample size; 4) Precision estimates the proportion of generated samples within the support of real
25

---

Generate raw description Algorithm 1: Textual Scene Description Generation via VLM, LLM, and RAG
Input: User prompt Tp; Scene dataset Dxcene
Output: Structured scene description 'D = [5,0,
Offline Stage: Build Memory Bank M
for frame in Scene do
Load 6 surround-view images 1 dj VIM(
5
S. + Parse(d));
// Parse style, objects, and background // Extract spatial annotations // Convert annotations to textual layout
Ay < DataAnnotations:enn (J))
7
L + LayoutFrom(Ap)) ;
my
// Assemble memory item
Add my to memory bank M
10 Online Stage: Generate Structured Description D
11 2p f Embed(Tp);
Embed user prompt // Embed memory entries // Retrieve top-k relevant memories with RAG // Prepare input context // Generate final description via GPT-4o
12 { < Embed (mn; for all m, MM;
13 Mx f TopK &p, {
Format LLM input using Tp and Mx :
15 D + Idescription (Tp,MK)
data; 5) Recall measures how well the generated distribution covers real data; and 6) F1-Score, the harmonic mean of precision and recall, reflects the balance between generation quality and coverage.
B Additional Details of Scene Description Generation
The scene description module constructs textual scene representations by integrating lasgua-language models (VLMs) and large language models (LLMs). As shown in Algorithm 1, a scene memory bank first built offline using a VLM. During inference, RAG pipeline selects the most relevant memory items based on a user's s coarse prompt, enabling the LLM to generate detailed, context -oroudgr scene descriptions. This framework supports flexible and scalable scene description generation.
B.1 Scene Description Memory Construction
To construct the scene description memory bank M, we use QWen2.5-VL [99] to extract structured scene information from the nuScenes dataset. For each annotated timestamp, the six surround-view camera images are processed by the VLM to generate a holistic natural language description, which parsed into structured components {S, O, B}: scene style (e.g., "a rainy afternoon in an urban area"), ), foreground objects with spatial and appearance attributes (c.g.. "a red sedan parked alongside the walkway"), and background elements (e.g.. "high-rise buildings in the distance"). In parallel, nuScenes 3D bounding boxes and lane markings are converted into textual scene-grast -rane layout L capturing spatial relationships (e.g.. "car behind truck B", "pedestrian on the sidewalk near lane LI"). Together, these components form each memory item m,
B.2 Novel Scene Description Generation
During inference, given a coarse user prompt Tp, we employ GPT-4o as the LLM-based generator Gdescription and implement a RAG mechanism to enrich the prompt with relevant memories. Specifically, both the prompt and the entries in the memory bank M are embedded using a pre-trained sentence -emeedding---smalll
embedding model (i.e., similar descriptions from M. These retrieved examples serve as contextual references, enabling the LLM to generate rich and coherent scene description D (S, O, B, L} tailored to the user's input.
We then retrieve the top-K most semantically
This RAG design motivated by the need bridge coarse user prompts and fine-grained scene representations, enabling few-shot generalization and knowledge transfer from similar scenes in the memory bank. Furthermore, the memory bank M is modular and extensible, supporting future inclusion of other datasets with minimal adaptation effort.
26

---

B.3 Prompt Details and Scene Description Examples
The following system prompt defined for constructing scene description memories. Given two images capturing the 360-degree surroundings, the VLM is guided to extract and organize key elements of the driving scene into comprehensive representation:
System prompt for scene description memory construction with VLM
Given two panoramic images Eimage>FRONT IMAGE</mage> and <image>BACK IMAGE</image> that encapsulate the surroundings of vehicle in 360 degree view, your task is to analyze the driving scene.
Your analysis
following core information:
Time of the day: Indicate whether it is daytime or nighttime.
Weather: Spocity if it is sunny, rainy. cloudy, snowy, or foggy.
Surrounding environment: Classify the environment downtown, urban expressway, suburban, rural, highway, residential,
industrial, nature, etc.
identify objects in foreground, such cars, buses, trucks, pedestrians, bicyeles, motoreyeles, construction vehicles, burriers, trallic traflic signs/lights,,
Foreground objects:
Background elements: Describe background elements, including reads, sidewalks, pedestrian crossings, car parks, terrain, vegetation, builddings, etc.
Road condition: Characterize the road as intersection, straight road, narrow street, wide road, pedestrian crossing, etc.
Abstract Description: Provide concise summary the scene, integrating details about scene features, foreground objects, background information, and road conditions.
Instruction:
of smaller images. views of the vehicle. The second image includes the relt-rear, directly behind, und right rear views.
Each panoramic image
image covers left-frent, directly front, and right-frent
When describing foreground objects, clearly detail their unique appearance and location. Specify each object's relative position like "firat/scond image" directional phrases such as mront-lffthear-cent view". there are maltiple objects of the same type, provide a description for each one.
the ego vehicle using lerms like front, back, left, right, Avoid referencing
background elements, provide descriptions their notahle characteristies.
Assess the presence of objects from the provided candidate list. object exists, describe attributes briefly. does exist, omit from your output. You may also include objects not listed in the candidates.
Please format your results follows:
"Time of the day": "xxx", "Weather": "xxx", "Surrounding environment"; "xxx",
"Foreground objects"; ["ob]ect1 class"; "object1 attributes"),
"Background elements":
["elementi class": *elementl attributes"),
},
"Road condition" "xxx",
"Abstract Descripttor"
Example output:
"time of the day": "daytime", "Westher"; *aunny", "Surrounding enviconmeat"; "downtow=", "Foreground objects"; |"ear": "blue color"),
("truck*: "gray color"), },
"Background elements": [
{"sidewalk"; "nazrow"),
["building"; "white color and tall*)
1.
"Road condition"s *intersection",
"Abstract Description"; "The acene depicts a sunny day in a downtown area with blue gray truck, and crouching pedestrian. The nerrow sidewalk is lined with a white, tall building."
27

---

The following system prompt is defined for generating novel scene descriptions. Given coarse user prompt, the LLM guided to retrieve semantically relevant scene descriptions from structured memory bank. These retrieved references are then used to enrich, clarify, and ground the final output, resulting coherent and contextually accurate scene description:
System prompt for novel scene description generation with LLM+RAG
intelligent assistant detailed driving scene understanding generation. Given a coarse user prompt relevanl memory items relrieved from a structured memory bank, your lask to generale comprehensive, structured description of target driving scene.
Input Tokens:
<lexi>USER PROMPT<//exs: high level, possibly amhiguous user query describing scene, e.g., "a busy urban street at night".
<JNON=MEMORY/JJON collection of scene descriptions in ISON format, semantically retrieved as relevant references to prompe.
These memory examples should used to enhance. clarify, and ground your final output. Your oulput strictly follow the specified ISON structure and provide a cohesive and concrete description of the driving scene.
Instructions:
Leveruge relevant entries in <memory> to help expand, clarify, or disurbiiguate the user's sprompt>.
When information in the prompt is sparse or vague, infer plausible details based on common potterns from similar memory entries.
Be specific in describing the following:
Spatial relationships between objects (e.g.. beside, abead, behind)
Object attributes (e color, type, behavior)
Environmental context (c.g. weather, road type. background elements)
not directly copy content memory items; instead, synthesize a new, coherent scone inspired
Avoid referencing the tokens <prompt> <memory> the output.
Expected Output Structure (JSON):
"Time of the day"; "xxx", "westher": "xox", "Surrounding ronment": "xxx", "Fozeground objects"; {"object1%; tobject1 attributes"),
}, "Background elements"; [ |"elamentio "alementi attributas"),
},
"Boed condition": "xxx",
"Layout"; "xxx",
"Abstract Description": "xox"
Example Output:
"Time of the day"; "dayLime", "Westher"; "cloudy", "Buzrounding enviconmente "urben exрсеasway", "Fozeground objects":
["truck"; "zed container truck driving ahoad"),
[*rotorcyale"; "black motorcyale overtaking from the right")
"Background elements": [
["avarpass"; "grey concrete with visible treffic sigas"),
|"vegetation": "small bushes along the road divides*)
b
"Rozd condition": "straight road",
"Layout"; "The truck is poaitioned ahead of the notorcycle and located on the the pedestrian crossing. " "Abstract Description': "A cleudy daytime scene on an usban expressway, with e red container truck ahoad and black matoraycle overtaking. Conereta overpasses and roadsids bushes shape the background."
driveble road, The notorcycle is adjacent
28

---

Representative examples of the generated scene descriptions, including scene style, foreground objects, background elements, and scene-graph layouts, are presented below:
Scene Description Example
Scene Style:
Time of the day: daytime, with diffused lighting cloud cover
Weather: light to moderate rain, as indicated by raindrops the camera lens and wet pavement
Surrounding envirunment: urban street llanked by mixed use buildings and parking areas
condition: long, straight,
clearly marked
dividers; visibly rainfall
Foreground Objects:
Cars (left variety of parked vehicles, including dark pickup truck, compact sedans, SUVs, and on, aligned parallel along the sidewalk; some cars have reflections on the wel ground
Cars (right side): several vehicles parked curbside in front of commercial buildings, including economy cars to midsize SUVs
Pedestrian: individual wearing bright orange reflective safety vest holding
standing near
suggesting crossing action in prugress
Trallic come: bright orange come placed on the sidewalk neur the edge a parking enirance, likely for safety or reserve space
Traffic sign: a yellow pedestrian crossing sign mounted pole; the sign is positioned
glass-door bailding entrance
Background Elements:
appears
glossy
rainfiall; markings and crosswalks are clearly
Sidewalk: wide, concrete sidewalks run alongside the road, bordered planters and lined with trees
Buildings: prominent structures include multi-level modern commercial buikdings with large glass façados
Trees: scattered urban landscaping includes small trees planted at regular intervals, offering touch of greenery
Parking: multiple designated parking areas, some directly along
within enclosed lots
Croaswalk: wide white-striped pedestrian crossings present intersections
Streetlight:: installed intervals along the road to provide visibility during low-light conditions
Scene-Graph Layout:
Traffic Existing: False
Crosswalk (-9.6, -50.1) (-9.4, 443.3) (-1288 (+13.), 5500))
Current struight lone ( 0.7, 15.0, (.0., 8.2), (0.3, 1.41 (-0.1, +5.4)]
ogo top
Straight with-flow
-4500), (-13,
vehiickc.car top the lane. sume direction is the left back, location: (2..
Straight lane allowing from left so right driving (6+63, 30.6),112.. (-12,,,-00..) (-187..). (-224 2124922111)1
Straight uppoite-foww ((-45, -37.9), (-477 41.9), 45.0% 49.99)
Other Lanex Roadside
movaiste object trafficcone the left front location: (-114, 19.9, 10.3) lengh widhh 0.4, height:
movobllc object trafficcone the left from location: (-13.4, +10.0, +0-4) kuuhh 04, wilth: 0.3, hiight 088
vehicle.car porked the left, heading
right, location: 6.66, +0 60 lenghh 4.2, widhs bright 1.4
hending left right,
-2,6, length
hright:
vehicle.car parkol the left back, heading from loft right, location: (-18.4 -52, (06) leggth 4.2, widh:: 1.8. height:
human pedestria moving the right front, oppusite direction, locatio:: (47.2, 22000 -1.2) le2ghh 0.9, widhh 0.7, height: 1.7.
human pedestrian standing in the left back, hending from left right, location: (-13 2, 3006, +0.8) length: 0.6, with:: 0.6, height: back, heading left right,
pedestrian.co standing
31.3, lengths
height:
Abstract Deseription:
urban expressway with roads reflecting light. Numerous parked both sides of the street, and pedestrian in bright orange safety vest is near the cresswalk. The background features multi-story commercial signage. Streetlights
shows rainy
buildings large windows, lining the sidewalk, croeswalks and lane lines, creating realistic and structured urhan driving scene.
marked
29

---

Scene Description Example
Style:
Time of the day: bright daytime with clear shadows, indieating direct sunlight and good visibility
Weather: sunny with parily cloudy skies; signs of precipittion poor visibility
Surrounding enviromment: suburban campus or business park like area with a mix of roadways, pedestrian poths, and landscuped
condition: smooth asphalt gentle curve transitioning straight segment
Foreground Objects:
Barriers (construction ): bright orange modular barriers clearly indicating temporarily restricted area
(construction zone): construction
visible parked adjacent
Stop sign (constraction zonej: standard red octagonal stop sign mounted metallic pole, reinforcing right-of way rales
Entrance barrier: red and white striped boom bartier
vehicle access points, indicating controlled catry
Cars: silver sedan (parked at curve near the booth); black sodan (alongside silver), dark-colored sedan (moving towards camera, (visihle in sidefrear view building behind the no- entry sign)
on lane); silver
Seated pedestriam: person resting the grasy
right side the rond, shaded
Background Elements:
dual- road center markings
directional arrows
Sidewalk: paved pathways flanked grass and shrubs hoth sides the road
Buildings: main visible structure is multi story white facility with large windows and blue signage
Vegotation: tall green trees forming canopy along both sides of the road, sculpted bashes reinforce the planned landscape dosign
Car Parking: clearly delineated lot visible on the left. populated with multiple parked cars
Traftic Signage: Entry" cirele with horizontal white bar) sign prominently displayed access control gate
Scene-Graph Layout:
Traffic Light Esisting: False
Curentt strnight [(-0.2, -0.0), (-011, +2.0), (-01,,
1go vehicle tup
Straight lane with-Jow [[+0 +7.2) (+0.2, +10.5) (-0.3, (1..0,,-1....000,.511......
movoble object.traffeeeee in the left freet location: (-L.7, +14.3, +0.6).
movable object trafficcone the left freet location: (-1.7, 11500 10.5).
turning allowing right
driving [(1 (5.00) 14144) (-38, 134.39]
vehicle.car moving the front, heading from right left, locution: (-0-4, 431.7, +13).
Left turning lame allowing from left right driving [t (t8.8, -64.4) (533, 44 6-19, +6899 (-0.1,
vehicle.car moving the left back, same direction as ego, location: (-1.2, 22.3, 10.3).
movable object.harrte harrier in left treat location: (:277 +129, length: width: 2.3. height:
movable, hbjecth in the left froat location: (3.7. +15.2, +0.5) length: 0.4, width: 2.4, height: 1.2.
movable object.traffeeeeee is the left frest location: (33.2, +18.8 lengt. 0.3, width 0.3, height: 0.7.
bicccllceeeeeo. without
right front, oppasite dircction ogo, location: (49.7, -9.0, long) width: height. I.I. opposite direction ego,
vehicle bicell..hccclcc. without rider the
(+10.3, +8.7, leng)t: withh height: location: +4.6,, +3.2) +3.2) length 4.7, widh: 1.9, height: 1.8.
vehickcccar parked in the left frout, same direction
vehicke.tuckk packed in the left front, heading from laft right, location (-644 +14.5, +2.0)lengthh 6.5, widds 2.5, heighe a,3.
human, www.pppootttttn...o siring lying down right frunt, heading right left, location: (+9.0, +10.9 +0.))
ceenaanpedeeee pedestriann.co sdult sittng ling down
heading right left, location: (+9.5, +11.7, 10.3).
Abstract Description:
shows a sunny day suburban with mix of urban infrastructure and greenery. The road slightly before straightening out, with construction barriers and stop sign indicating ongoing work. Several cars are paree along the roadside, while vibers
pedestrian seated grass right-rear view, hackground, The read appears well. maintainnd with clear lane markings and mix of open spaces and developed areas.
are trees buildings
30

---

Table 10: Ablation text-only generation.
Table 11: Ablation input layout types.
3DOD
BEVSeg mloU (%)
3DOD
BEVSeg mloU (%)
Variants
FIDI
Input Layout
FID,
MAPT
NDS
Road?
Vehicle
MAPt
NDS?
Road
Vehiclef
Full Model
11.29
16.28
26.26
66,48
29.76
Semantic Map
11.29
16,28
26.26
66.48
29.76
Text Only
20.74
2.13
5.34
28.32
7.49
Vector Map
12.07
15.73
25.84
65.17
28.38
Table 12: Robustness to layout noise. Performance under Table 13: noisy layout shows graceful degradation across stages.
Inference efficiency of
each stage on a single RTX A6000.
OceGen
ImgGen
3DOD
BEVSeg mIoU( )
Stage
Steps
Time (8)
GPU (GB)
Layout
FID"D
F
FIDL
mAPt
NDST
Roadf
Vehiclet
LayoutGen
50
0.15
1.0
Clean
258.8
0.778
11.29
16.28
26.26
66,48
29.76
OceGen IngGen
3.25
7.7
Noisy
276.3
0.742
12.47
14.87
25.02
65.28
28.44
2.30
7.0
C
Additional Quantitative Results
C.1
Effect of Spatial Conditioning
We evaluate the role of spatial conditioning using text-only variant that removes all spatial inputs (layout maps, object boxes, and perspective maps) while retaining textual prompts. As shown in Table 10, the absence of spatial cues causes clear degradation in visual realism (FID 1 9.45) and spatial fidelity (Vehicle mloU 4 22.27%), underscoring the importance of spatial conditioning for maintaining geometric coherence and consistent scene alignment.
C.2 Effect of Layout Type
To examine different layout representations our dual-mode controllability design, we compare two layout types: 1) BEV semantic maps for fine-grained spatial control and 2) BEV vector maps of object boxes and lanes for efficient customization. As shown in Table 11, both yield geometrically accurate and visually coherent scenes, while semantic maps provide stronger spatial priors with slightly better realism and downstream performance. This confirms that both layout types are fully compatible with pipeline, enabling flexible and effective scene control.
C.3 Robustness and Efficiency
To assess potential error accumulation in our cascaded generation pipeline, we conduct a noise- injection ablation by applying Gaussian perturbations (25% probability) the initial layout, including 3D box centers and lane coordinates. As shown in Table 12, the pipeline degrades gracefully under noise, with only marginal drops in downstream metrics. This robustness arises from the multi- stage alignment design, where eccupancy-renderts semantic and depth priors enforce geometric consistency, and overlap-aware extrapolation maintains spatial continuity.
We also report inference efficiency Table 13. Each scene chunk is generated in about seconds on single RTX A6000 GPU, showing that our system achieves strong balance between robustness and computational efficiency for large-scale scene synthesis.
C.4 Effect of Retrieval-Augmeented Generation
RAG enhances text-to-scene generation by expanding brief prompts into detailed scene descriptions through retrieving semantically related examples from mem- ory bank. This process transfers prior knowledge from similar scenes, improving layout accuracy and reducing user effort. Table 14 presents human preference study in which ten participants evaluated 100 scene pairs gen- erated with and without RAG across multiple criteria. The results show that RAG-based generation is consis- tently preferred (overall 77% vs. 23%), highlighting effectiveness in grounding prompts and producing more diverse, realistie, and controllable scenes.
Table 14: Human preference study paring scene generation w/ and w/o RAG.
Criterion
RAG (%)
Non-RAG (%)
Diversity
13
Realism
82
18
Controllability
74
26
Phys. Plaus.
Overall
77
23
31

---

D Additional Qualitative Results
D.1
Conditional Occupancy and Image Generation
Figure 8 presents additional conditional generation results, where layout conditions are used to synthesize multi-view images and 3D occupancy. These results demonstrate the effectiveness of our approach in generating coherent multi-modal outputs conditioned on low-level layout inputs.
Imput Condition
Generated Multi-view Images
Generated Occupancy
Figure 8: Additional qualitative results of X-Seene conditional occupancy and image genera- tion. These results demonstrate the model's ability to generate semantically consistent and structurally accurate multi-modal outputs conditioned layout inputs across diverse urban scenes.
32

---

Description Generation
Grane-Graph Layout Generation
Imngle driig acu Icuacco
fellings
living
Scene-Graph
Genersted Logout
Joint Occupancy and Multiview Image Generation
(s)
Geametry and Visual Reconstruction
(o) Text-to- -o-enene Generation Example: Campus Strest Scene
(a) Textual Scone Description Generation
(b) Scene-Graph Layout
Prang
behind
.lfft
crink
ego
car
shorter than
Scene-Groph
Geverated Layout
Joint Occuponcy and Multiview Image Generation
(d) Geometay Visual Reconstrructio
(b) Text-to-Seee Sceme Generotion Example: Country Road Scene
Figure 9: Qualitative results of the text-fo-scene generation pipeline of X'-Scene. Starting from user prompt, the system generates plausible scene description, constructs the corresponding layout, synthesizes consistent occupancy and multi-view images, and finally performs 3D reconstruction.
33

---

Textual Scone Description Generation
(b) Grene-Graph Layout Generation
shilig
ap
Scone-Graph
Generated Laguut
(c) Joint Occuponcy and Multiniw Image Goveration
(d) Goometry and Vicual Reconstruction
(c) -te-----enn Generation Example: Industrial Narrow Road Scene
(a) Textual Scone Description Geveration
(b) Scone-Graph Layout
Prange
Graph
Generated Layout
e Joint Occupancy and Multiview Image Ganeration
(s) Geometry and Viaual Reconstruction
(d) Text-to-Sceme Generotion Example: City Crossroad Scene
Figure 10: Qualitative results of the text-to-scene generation pipeline of X-Sceee. Starting from user prompt, the system generates a plausible scene description, constructs the corresponding layout, synthesizes consistent occupancy and multi-view images, and finally performs 3D reconstruction.
34

---

Figure 11: Qualitative results of large-scale scene generation by X'Secen. The model extrapolates coherent occupancy fields and multi-view images across extended areas, enabling high-fidelity and complete 3D scene reconstruction. The generated scenes support novel view synthesis of RGB, depth, and occupancy, demonstrating both geometric consistency and high photorealistic quality scale.
35

---

D.2
Texf-to-Scene Generation
Figure and Figure 10 illustrate examples of the text-to-scene generation pipeline, which primarily consists of four steps:
Textual scene description generation: Given a coarse user text prompt, the LLM leverages RAG to retrieve semantically relevant scene descriptions from the memory bank, then composes plausible scene description encompassing scene style, foreground and background elements, and a textual scene-graph layout.
Scene-graph to layout generation: The layout diffusion model uses the textual acene-graph to generate the corresponding layout, including object bounding boxes and lane lines.
Joint occupancy and multi-view image generation: The occupancy and image diffusion models leverage the layout for geometry control and the text description for semantic control, generating a coherent and realistic 3D occupancy field and multi-view images.
Geometry and visual reconstruction: Given the generated voxels and images, we reconstruct the scene while preserving intricate geometry and realistic appearance, supporting various downstream applications.
These results demonstrate that the proposed text-to-scene pipeline is effective and flexible method for driving scene generation.
D.3 Large-Scale Scene Generation
Figure 1l illustrates the results of large-scale scene generation. The results show that our method can generate coherent, large-scale driving scenes through consistency-awe extrapolation. Moreover, the generated occupancy and images are fused and lifted for large-scale scene reconstruction, preserving both intricate geometry and realistic visual appearance. The reconstructed scenes support novel RGB, depth, and occupancy rendering.
E Potential Societal Impact & Limitations
In this section, we discuss the potential societal impact of our work and outline its possible limitations.
E.1
Societal Impact
Our proposed framework, X--ccene, for large-scale controllable driving scene generation holds significant potential for real-world societal impact. By unifying fine-grained geometric accuracy with photorealistic visual fidelity, X'Sccene enables the generation of highly realistic and semantically consistent 3D driving environments. This capability directly supports the development safer and more efficient autonomous driving systems enabling rigorous simulation and validation across richly diverse scenarios, including rare cases such as complex intersections, unexpected pedestrian behavior, and unusual road layouts. As result, X'-Scene can accelerate the development cycle of autonomous vehicles, reduce reliance on costly and cone-con real-world data collection, and improve safety standards, ultimately contributing to reduction in traffic-related accidents and fatalities.
E.2 Known Limitations
While X'-Scene offers promising framework for large-scale controllable 3D scene generation, several limitations remain and warrant further investigation.
First, while X'Scenn supports dynamic 4D scene generation, the current autoregressive video diffusion framework is still limited in long-horizon synthesis. As the number of autoregressive iterations increases, errors in geometry and appearance may accumulate, leading temporal drift and degraded motion consistency. Future work will focus on improving long-term temporal stability and mitigating accumulation to achieve more robust and extended video generation.
Second, the scene description memory bank currently built from the nuScenes dataset [93]. While this dataset provides a solid foundation, its limited geometric and semantic diversity may restrict the
36

---

range and realism of generated scenes, Incorporating additional datasets featuring a broader range of environments, weather conditions, and traffic patterns would enhance the system' generalization and scene richness.
Third, the occupancy generation pipeline depends fixed set of semantic categories predefined in the training data. As a result, introducing new object types or unseen classes requires retraining the model. This rigidity hinders adaptability in evolving or open-world settings. Future work could explore more extensible architectures that support incremental learning or open-vocabulary generation.
Addressing these limitations is essential for enhancing the realism, scalability, and applicability of X X-Scene in real-world simulation and data generation tasks.
F Public Resources Used
In this section, we acknowledge the public resources used, during the course of this work.
F1
Public Datasets Used
nuScenes
CC BY-NC-SA 4.0
nuScenes-devkit".
Apache License 2.0
Occ3D
MIT License
F.2
Public Implementations Used
MagicDrive
Apache License 2.0
SemCity
MIT License
DynamicCity
Unknown
DriveArena
Apache License 2.0
OccSora
Apache License 2.0
X-Drive
Apache License 2.0
MinkowskiEnniine
MIT License
Torch-Fidelity"!
Apache License 2.0
Qwen2.5-VL
Apache License 2.0
UniScene
Apache License 2.0
tnttpss://gtttub.cmmmm/
Phttps:
:/tg:/:///i// thub.
37


Reviews:
[{"review_id": "XIANP0KpKe", "reviewer": "Reviewer_w82Z", "summary": "The authors describe a system based on multistage diffusion policies for synthesising driving scenarios from text prompts (and/or maps and boxes) to 360-degree images and semantic occupancy maps. The system can synthesise specially (but not temporarily) coherent scenarios and complete/expand/edit maps on a large scale.", "questions": "Here are some questions for the authors to comment on.\nI have ordered them in priority order from my perspective.\n\nQ1 - metric for spatial consistency in outpainting: while the completion and extension of the scenarios are discussed qualitatively, I think it would be interesting to evaluate the system quantitatively, especially at discontinuity points. One way would be to synthesise videos and use video metrics (FVD).\n\nQ2 - baseline for pure text-based diffusion: how well would a pure text-based diffusion policy synthesise images of the scene from the given prompt? While most likely not spatially consistent or entirely following the map layout, it could provide hints on how the presented input affects the rendering qualities of such a model. \n\nQ3 - low-level metric for image generation: FID gives important information about the semantic distribution of the generated scenes. Could the authors propose a low-level metric for evaluating the realism of the generated images, perhaps to the provided training set?\n\nQ4 task metrics: please, can you explain why you limit your BEV segmentation evaluation to only road and vehicles?", "limitations": "yes. I would also include any limitations in terms of the training data required to achieve such performances (volume, alignment, labels, etc.).", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "I will use the symbol (+) to represent strengths and (-) to represent weaknesses.\n\n# Quality\n(+) The work describes a substantial framework for scenario generation. The framework consists of multiple steps that enable high-level and low-level conditioning and editing, as well as faster rendering through 3DGS.\n\n(+) The quality of the discussion, in terms of thoroughness in experiment design, is well-suited to demonstrating the system's performance.\n\n# Clarity\n(+) The work is well-written and structured.\n\n(+) All components of the framework are clearly discussed and formulated.\n\n(-) Some of the images contain quite small details that are difficult to appreciate on printed paper.\n\n# Significance\n(+) The work is interesting and covers a significant topic. The creation of scenarios for robotic tasks can lead to editable simulators that can enhance the policies trained on them, especially in edge cases not easily recorded in real datasets.\n\n# Originality\n(-) This work is crucially a system paper, which utilises multi-step diffusion policies to generate data. Except for minor adaptations, the concepts and elements are retrieved from previous works.\n\n(+) This said, the scale and performance achieved are impressive.\n\n(-) I suggest that the authors review `NeuralFloors: Conditional Street-Level Scene Generation From BEV Semantic Maps via Neural Fields` and `NeuralFloors++: Consistent Street-Level Scene Generation From BEV Semantic Maps`. Although it is unfortunately closed-source, it employs a similar triplane approach, combining NeRF and diffusion. From this, it would be interesting to take inspiration from metrics (FVD for consistency) and baselines (prompt-based diffusion policy to test how inputs affect the quality of reconstruction).", "was_revised": true, "final_justification": "The authors present a complete framework for driving scenario synthesis. The framework primarily utilises existing methodologies, yet it yields impressive evaluations and results. \nThe authors made an effort to address my questions and concerns.\nThe additional results and insights are valuable and support the claims of the work.\nFor this reason, I confirm my rating.", "rebuttal": "We sincerely thank the reviewer for the constructive and encouraging feedback. The reviewer **appreciated our work's** substantial **framework design**, **comprehensive experimental setup**, **clear presentation**, and **strong practical significance**.\n\nThe reviewer also raised valuable suggestions regarding evaluation metrics, ablations, and training requirements. As detailed below, we have **addressed all concerns** with targeted **experiments** and **clarifications.**\n\n> ### **Q1: Scene outpainting is discussed qualitatively; a quantitative evaluation of consistency using FVD (inspired by NeuralFloors and NeuralFloors++) is recommended.**\n\n**A1**: Thank you for the insightful suggestion. \n\n- **Incorporating prior work:** We have reviewed *NeuralFloors* and *NeuralFloors++* and agree they are highly relevant. We will cite them appropriately in the revised manuscript.\n- **FVD-based evaluation:** Following their protocol, we adopted **FVD**  to assess the temporal consistency of our scene outpainting. As shown in **Table R1**, our method achieves competitive performance, with strong **frame-to-frame coherence**.\n\n**Table R1: FVD Comparisons on Video Generation.**\n|Method|FVD↓|\n|-|-|\n|DriveGAN|502.3|\n|DriveDreamer|452.0|\n|WoVoGen|417.7|\n|MagicDrive|217.9|dd\n|DreamForge|209.9|\n|**X-Scene (Ours)**|**206.2**|\n\n> ### **Q2: Suggest evaluating a pure text-based diffusion baseline; although spatial inconsistency is expected, it can reveal how the input conditions affect generation quality.**\n\n**A2:** Thank you for the suggestion. We conducted an additional **text-only generation** experiment to assess the role of spatial conditioning:\n\n- **Experiment setup:** We excluded all spatial conditions (i.e., layout maps, boxes, and perspective maps) while retaining only the textual prompt.\n- **Experiment results**: As shown in **Table R2**, compared to our full model, the text-only variant shows a substantial decline in spatial coherence (mAP ↓ 14.15, Veh. mIoU ↓ 22.27). However, it still maintains a plausible semantic distribution (FID ↑ 9.45).\n- **Interpretation:** Note that downstream evaluations are performed using downstream models trained on GT-aligned real images. The generated outputs exhibit spatial mismatches in the absence of spatial constraints, which naturally lead to degraded performance.\n- **Conclusion:** These results demonstrate that while text prompts convey high-level semantics, **spatial conditions are crucial** for ensuring both *visual realism* and *spatial fidelity*, especially for downstream utilization.\n\n**Table R2: Ablation on Text-only Generation.**\n|Variants|ImgGen FID↓|3DOD mAP↑|3DOD NDS↑|BEVSeg Rd. mIoU↑|BEVSeg Veh. mIoU↑|\n|-|-|-|-|-|-|\n|Full Model|**11.29**|**16.28**|**26.26**|**66.48**|**29.76**|\n|Text Only|20.74|2.13|5.34|28.32|7.49|\n\n\n> ### **Q3: Suggest incorporating low-level metrics to assess image realism beyond FID’s semantic-level evaluation.**\n\n**A3:**  Thank you for the suggestion. We have already included **low-level realism metrics** in the main paper (Tables 1–3) and further expanded the evaluation in **Table R3**:\n\n- **Existing evaluations (Tables 1–3 of the main paper):**  \n    1. **Table 1** evaluates images generated from *GT bounding boxes and HD maps* using pretrained downstream models for **3D object detection** (mAP, NDS) and **BEV segmentation** (mIoU). High scores indicate that both *foreground objects* and *background layouts* are visually realistic and well-aligned with real data.\n    2. **Tables 2 and 3** assess the impact of generated images when used as **data augmentation** to fine-tune downstream models. Performance gains over the baseline demonstrate the **low-level fidelity and practical utility** of our generated data.\n- **Additional validation (Table R3):**  \nWe further conduct an **end-to-end evaluation** using UniAD, covering *3D object detection* (mAP, NDS), *BEV segmentation* (mIoU), and *trajectory planning* (L2 Distance, Collision Rate). These results provide further evidence through **low-level metrics**, to show that the generated images are both *realistic and practical* for downstream applications.\n\n**Table R3: Evaluation of Video Generation Fidelity (FVD) and Additional Low-level End-to-end Metrics using UniAD.**\n|Method|VideoGen FVD↓|3DOD mAP↑|3DOD NDS↑|BEVSeg Lanes↑|BEVSeg Drivable↑|BEVSeg Divider↑|BEVSeg Crossing↑|L2\\@1.0s↓|L2\\@2.0s↓|L2\\@3.0s↓|L2 Avg.↓|Col.\\@1.0s↓|Col.\\@2.0s↓|Col.\\@3.0s↓|Col. Avg↓|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|MagicDrive[1]|217.9|12.92|28.36|21.95|51.46|17.10|5.25|0.57|1.14|1.95|1.22|0.10|0.25|0.70|0.35|\n|X-Scene(Ours)|**206.2**|**20.40**|**31.76**|**28.04**|**61.96**|**22.32**|**10.48**|**0.55**|**1.08**|**1.81**|**1.15**|**0.03**|**0.13**|**0.66**|**0.27**|\n\n> ### **Q4: Justification for evaluating BEV segmentation solely on road and vehicle classes.**\n\n**A4:** Thank you for the question. We **follow established practice** in the main evaluation, and provide **extended results** in Table R3:\n\n- **Standard protocol:** We follow standard practice by using **CVT** [83] as the downstream model for BEV segmentation. CVT is pre-trained specifically to predict **road** and **vehicle** categories in the BEV view. As a result, prior works including ours typically report metrics on these two classes to ensure comparability.  \n- **Extended evaluation:** We additionally use the end-to-end **UniAD** model for BEV segmentation (see **Table R3**) to provide a more comprehensive assessment. Unlike CVT, UniAD supports a **broader set of semantic classes** including **Lanes, Drivable Area, Dividers, and Pedestrian Crossings**. This extended evaluation complements the CVT-based results and further validates the semantic realism and spatial alignment of our generated scenes under **richer label settings**.\n\n> ### **Q5: Clarify training data requirements (e.g., volume, alignment, labels).**\n\n**A5:** Thank you for the question. Below we clarify the dataset and preprocessing used for training:\n\n- **Dataset:** We use nuScenes dataset with 1,000 diverse driving scenes under varying weather, lighting, and traffic conditions. Each 20-second scene includes approximately 40 keyframes. This results in around 40,000 annotated samples, where each sample provides 360° images from six cameras, 3D occupancy, bounding boxes, and HD maps. Following the standard protocol, we use 700 scenes for training and 150 for validation.  \n- **Video upsampling:** For video generation, we apply ASAP interpolation to upsample to 12 Hz, producing ~240 frames per scene since annotations are available only at 2 Hz. This denser supervision supports more consistent training for temporally coherent video synthesis."}, {"review_id": "FMmHIUBDgX", "reviewer": "Reviewer_sCXS", "summary": "This paper introduces X-Scene, a framework for generating large-scale, high-fidelity, and controllable 3D driving scenes. X-Scene is designed as a multi-stage pipeline. It begins with a \"Multi-Granular Controllability\" module that allows user input via either high-level text prompts or low-level geometric layouts. This control signal guides the core generation engine, which first synthesizes a 3D semantic occupancy field using a triplane-based diffusion model, and then generates corresponding photorealistic multi-view images conditioned on the 3D geometry. To achieve large-scale generation, the system employs a \"consistency-aware outpainting\" mechanism to iteratively expand the scene chunk by chunk, ensuring both geometric and visual coherence. The final generated assets are lifted to 3D 3DGS. The authors demonstrate through extensive experiments that their method achieves best results in both generation quality and utility for downstream perception tasks.", "questions": "See weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1: The hierarchical generation process (3D occupancy first, then 2D images) is a good way to ensure 3D consistency. The use of a triplane representation with a novel deformable attention mechanism is a nice touch that, as shown in the ablations, demonstrably improves reconstruction quality. The wealth of conditioning information used for the image diffusion model (rendered semantic/depth maps, perspective maps, object embeddings) to have better alignment between the generated geometry and appearance, a common failure point in other methods.\n\n2: The experimental results are thorough. It also demonstrates significant gains when using its synthetic data to train downstream perception models. The ablation studies are excellent.\n\n3: Paper is easy to follow. \n\nWeaknesses: \n\n1: The most significant weakness, which the authors also acknowledge, is that the framework generates static scenes.  Real-world driving is fundamentally dynamic. While generating high-quality static worlds is a critical first step, the lack of dynamics limits its immediate use for end-to-end planning and simulation.\n\n2: X-Scene is a very complex, multi-stage cascading pipeline (LLM -> RAG -> Graph Diffusion -> Triplane Diffusion -> Image Diffusion). While powerful, this complexity raises concerns about reproducibility,  training/inference costs, and potential error propagation. How long is the inference time? Also a small error in the initial layout generation could lead to significant artifacts in the final rendered scene. The paper doesn't discuss how these errors are handled or how they might compound. \n\n3: The paper claims large-scale generation via iterative outpainting. However, the practical limits of this approach are not discussed. What are the computational/memory constraints as the scene grows? Does the con", "was_revised": true, "final_justification": "Thank you for the exceptionally thorough and compelling rebuttal. I am very impressed with the extensive work authors have done to address the initial concerns, which has strengthened the paper. Most of my concerns are very well addressed. I have raised my score to BA.", "rebuttal": "We thank the reviewer for the detailed and constructive feedback. The reviewer appreciated **several key strengths of our work**, including the hierarchical 3D-to-2D generation strategy, deformable attention in triplane diffusion, and enhanced geometry-appearance alignment. The reviewer also acknowledged our **comprehensive experiments**, **strong downstream results**, and the **manuscript's clarity**.\n\nIn response to the reviewer’s concerns, we provide **additional results and clarifications** below to address dynamics, error accumulation, computational cost, and outpainting scalability.\n\n> ### **Q1: While not the original focus, the lack of dynamic (video) scene generation acknowledged by the authors limits the framework's applicability to real-world end-to-end planning and simulation.**\n\n**A1:** While video generation was not the original focus, we have extended **X-Scene to support dynamic scene synthesis** by introducing an **autoregressive video generation framework** with temporal conditioning and attention, trained on 7-frame clips.\n\n- **FVD-based evaluation:** To assess temporal consistency, we adopt the **FVD metric** suggested by reviewers sCXS, w82Z, and oXSL.\n- **End-to-end evaluation:** We also use **UniAD** to test the effectiveness of the generated videos for E2E perception and planning tasks, including *3D object detection* (mAP, NDS), *BEV segmentation* (mIoU), and *trajectory planning* (L2 Distance, Collision Rate).\n- **Experiment results:** As shown in **Table R1**, X-Scene trained on 7-frame clips achieves **lower FVD** than the 16-frame-trained baseline *MagicDrive*. This indicates its superior video fidelity with higher efficiency. Our X-Scene also **outperforms the baseline in the end-to-end evaluation** via UniAD. This demonstrates its effectiveness for closed-loop simulation.  \n- **Scalability:** Furthermore, our autoregressive design enables **long-horizon video generation** (e.g., over 100 frames) despite short-clip training. This showcases the strong scalability and sample efficiency of our design. Qualitative results will be included in the final paper (subject to rebuttal constraints).\n\n**Conclusion:** These results confirm that X-Scene now supports high-quality and temporally coherent **dynamic scenes generation**, making it well-suited for realistic end-to-end simulation. We will release the video generation code to support reproducibility.\n\n**Table R1: Evaluation of Video Generation Fidelity (FVD) and End-to-End Performance using UniAD.**\n\n|Method|VideoGen FVD↓|3DOD mAP↑|3DOD NDS↑|BEVSeg Lanes↑|BEVSeg Drivable↑|BEVSeg Divider↑|BEVSeg Crossing↑|L2\\@1.0s↓|L2\\@2.0s↓|L2\\@3.0s↓|L2 Avg.↓|Col.\\@1.0s↓|Col.\\@2.0s↓|Col.\\@3.0s↓|Col. Avg↓|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|MagicDrive[1]|217.9|12.92|28.36|21.95|51.46|17.10|5.25|0.57|1.14|1.95|1.22|0.10|0.25|0.70|0.35|\n|**X-Scene (Ours)**|**206.2**|**20.40**|**31.76**|**28.04**|**61.96**|**22.32**|**10.48**|**0.55**|**1.08**|**1.81**|**1.15**|**0.03**|**0.13**|**0.66**|**0.27**|\n\n\n> ### **Q2: Error accumulation in the cascaded pipeline, inference cost, and reproducibility.**\n\n**A2:** We address concerns about error accumulation and pipeline robustness through a **noise-injection ablation study**:\n- **Error accumulation ablation:**\n    1. **Ablation setup:** We introduce Gaussian noise to the initial layout (i.e., 3D box centers and lane points) with 25% probability to simulate partial noise arising from imperfect layout predictions. We then assess the impact on occupancy generation, image synthesis, and downstream performance.\n    2. **Ablation results:** As shown in **Table R2**, the results **degrade gracefully** despite the injected noise. Particularly, in downstream detection (mAP ↓ 1.41) and BEV segmentation (Rd. mIoU ↓ 1.20, Veh. mIoU ↓ 1.32). This indicates the strong robustness of our pipeline.\n    3. **Why robust:** This resilience stems from our **multi-stage alignment design**: (1) occ rendered semantic/depth maps provide strong geometric priors for image generation; (2) overlap-aware extrapolation ensures spatial continuity across chunks.\n- **Inference cost:** **Table R3** summarizes inference time and memory usage for each module. A single scene chunk takes \\~6 seconds to generate with **moderate GPU consumption**.\n- **Reproducibility:** We provide detailed implementation info in Section A.2 of the supplementary material and will release code for full reproducibility.\n\n**Table R2: Ablation on Error Accumulation: Impact of Layout Noise on Generation Quality and Downstream Tasks.**\n|Input Layout|OccGen FID³ᴰ↓|OccGen F³ᴰ↑|ImgGen FID↓|3DOD mAP↑|3DOD NDS↑|BEVSeg Rd. mIoU↑|BEVSeg Veh. mIoU↑|\n|-|-|-|-|-|-|-|-|\n|Clean Layout|**258.8**|**0.778**|**11.29**|**16.28**|**26.26**|**66.48**|**29.76**|\n|Noisy Layout|276.3|0.742|12.47|14.87|25.02|65.28|28.44|\n\n\n**Table R3: Inference Costs for Each Module on a Single RTX A6000.**\n|Module|Denoise Steps|Time|GPU Memory|\n|-|-|-|-|\n|Layout Diffusion|50|150 ms|1.0 GB|\n|Occupancy Diffusion|20|3.25 s|7.7 GB|\n|Multiview Image Diffusion|20|2.30 s|7.0 GB|\n\n\n> ### **Q3: Memory and computation scalability of iterative outpainting.**\n\n**A3:** Our iterative outpainting design is **memory-efficient and scalable**, which enables large-scale generation without memory bottlenecks:\n\n- **Constant memory per chunk:** As detailed in Lines 202-215, each chunk is generated using only its **own latent state** (occupancy latent $\\mathbf{h}^{\\text{new}}$, image latent $\\mathbf{x}^{\\text{new}}$) and **overlapping context** from the previous chunk ($\\mathbf{h}^{\\text{ref}}$, $\\mathbf{x}^{\\text{ref}}$). No global scene context needs to be held in GPU memory.  \n- **Efficient memory usage:** Once a chunk is generated, it is offloaded from GPU memory to RAM (as `NumPy arrays`). This allows GPU memory to be reused in subsequent iterations.  \n- **Result:** This design ensures both memory usage and computation cost **per chunk to remain constant** (as Table R3) regardless of the overall scene size.\n\nWe will clarify this efficiency in the final version."}, {"review_id": "v5tXsrSZ8i", "reviewer": "Reviewer_oXSL", "summary": "This paper introduces X-Scene, an end-to-end pipeline for driving scene generation that supports various beneficial functions. The functions include converting text or user-drawn layouts into a scene graph and outpainting the scene chunk by chunk. The authors propose to use a triplane latent occupancy volume, multi-view dissusion model to generate the scene and pack the final output into a 3D Gaussian-splat scene for downstream usages. Extensive experiments demonstrate that X-Scene outperforms existing baselines in terms of generation quality, image size, and improvement in downstream perception tasks.", "questions": "Please see the above.", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strenghts\n---\n- The paper is generally well-written and easy to follow.\n- The research is timely and aims to improve the current generation in many aspects, including fidelity and controllability.\n- Experiments are carefully designed and complete.\n\nWeaknesses\n---\nAlthough I am not actively working in this field, I was unable to identify any critical weaknesses in the manuscript that would prevent me from giving a positive score. Below are some questions after reading through the paper.\n- While I don't think it's the main point of the research, as the authors mentioned, the current method doesn't support video generation, which is critical for real usage such as closed-loop evaluation.\n- Contributions may come from mostly an engineering unification rather than scientific novelty as key ideas including triplane, cross-modal diffusion, and latent extrapolation have already been widely used.\n- It would be beneficial if the authors could provide a more in-depth discussion on the effectiveness of using RAG in their framework.\n- How does outpainting behave after many hops? Does the artifact accumulate?\n- Is there any reason for the performance discrepancy between Tables 5 and 7?\n\nAdditional Comments\n---\n- Missing references that the authors may want to add:\n  - Wu et al. Text2lidar: Text-guided lidar point cloud generation via equirectangular transformer. In ECCV 2024.\n  - Pan et al. Transfer Your Perspective: Controllable 3D Generation from Any Viewpoint in a Driving Scene. In CVPR 2025.", "was_revised": true, "final_justification": "I appreciate the author's enthusiastic and impressive rebuttal. I have carefully read the fellow reviewers' comments and the response. All of my concerns have been well addressed. I will raise the score from *borderline accept* to *accept*, as I believe the research is well-designed, timely, and impactful.", "rebuttal": "The reviewer appreciated the **clear writing**, the **comprehensive and well-designed experiments**, and the **practical value** of our controllable, high-fidelity generation framework, which unifies scene graph conversion, triplane diffusion, multiview image generation, and chunk-wise outpainting, delivering **strong performance** in both image quality and downstream tasks.\n\nIn response to the reviewer’s comments, we provide **additional results and clarifications** below to address video generation capability, architectural novelty, the effectiveness of RAG, outpainting robustness, and evaluation consistency.\n\n> ### **Q1: While it may not be the main focus of the paper, the lack of video generation limits the applicability such as for realistic closed-loop evaluation.**\n\n**A1:** While video generation was not the original focus, we have extended **X-Scene to support dynamic scene synthesis** by introducing an **autoregressive video generation framework** with temporal conditioning and attention, trained on 7-frame clips.\n\n- **FVD-based evaluation:** To assess temporal consistency, we adopt the **FVD metric** suggested by reviewers oXSL, w82Z, and sCXS.\n- **End-to-end evaluation:** We also use **UniAD** to test the effectiveness of the generated videos for E2E perception and planning tasks, including *3D object detection* (mAP, NDS), *BEV segmentation* (mIoU), and *trajectory planning* (L2 Distance, Collision Rate).\n- **Experiment results:** As shown in **Table R1**, X-Scene trained on 7-frame clips achieves **lower FVD** than the 16-frame-trained baseline *MagicDrive*. This indicates superior video fidelity with higher efficiency. Our X-Scene also **outperforms the baseline in end-to-end evaluation** via UniAD. This demonstrates its effectiveness for closed-loop simulation.\n- **Scalability:** Furthermore, our autoregressive design enables **long-horizon video generation** (e.g., over 100 frames) despite short-clip training. This showcases the strong scalability and sample efficiency of our design. Qualitative results will be included in the final paper (subject to rebuttal constraints).\n\n**Conclusion:** These results confirm that X-Scene now supports high-quality and temporally coherent **dynamic scenes generation**, making it well-suited for realistic end-to-end simulation. We will release the video generation code to support reproducibility.\n\n**Table R1: Evaluation of Video Generation Fidelity (FVD) and End-to-End Performance using UniAD.**\n|Method|VideoGen FVD↓|3DOD mAP↑|3DOD NDS↑|BEVSeg Lanes↑|BEVSeg Drivable↑|BEVSeg Divider↑|BEVSeg Crossing↑|L2\\@1.0s↓|L2\\@2.0s↓|L2\\@3.0s↓|L2 Avg.↓|Col.\\@1.0s↓|Col.\\@2.0s↓|Col.\\@3.0s↓|Col. Avg↓|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|MagicDrive[1]|217.9|12.92|28.36|21.95|51.46|17.10|5.25|0.57|1.14|1.95|1.22|0.10|0.25|0.70|0.35|\n|X-Scene (Ours)|**206.2**|**20.40**|**31.76**|**28.04**|**61.96**|**22.32**|**10.48**|**0.55**|**1.08**|**1.81**|**1.15**|**0.03**|**0.13**|**0.66**|**0.27**|\n\n\n> ### **Q2: Concern about the lack of scientific novelty beyond component integration.**\n\n**A2:** We emphasize two key innovations that go beyond simple integration:\n\n- **Multi-Granular Control Mechanism**:   \n   We introduce a **dual-mode control framework** that supports both (1) *text-prompted* scene generation for high-level usability and (2) *layout-guided* generation for precise control. This design balances usability and precision, enabling intuitive *high-level customization* (mode 1) and precise *low-level geometric control* (mode 2). To the best of our knowledge, **no prior work has explored such a flexible and unified control mechanism especially in the context of text-to-scene generation**.  \n- **Unified Generation and Extrapolation Pipeline:**   \n   Our framework jointly models *3D occupancy and multi-view video* generation, coupled with a *consistency-aware outpainting mechanism* for scalable scene extrapolation. This design ensures both geometric continuity and temporal coherence. As a result, it enables realistic closed-loop and end-to-end simulation. This is an **improvement over existing methods that are limited to single-chunk or image-only generation**.\n\nThese contributions collectively establish our X-Scene as a unified and effective framework for large-scale and controllable 3D scene generation. They advance both the capabilities of the system and its real-world applicability.\n\n> ### **Q3: Motivation and empirical evidence for using RAG in the framework.**\n\n**A3:** We clarify our motivation for using RAG and provide ablation results as follows:\n\n- **Motivation for using RAG:**  \n   RAG plays a pivotal role in the *text-to-scene generation* mode. It transforms **brief, coarse** user prompts (e.g., *“a busy city intersection”*) into **detailed and structured** scene descriptions (see Pages 7–8 and 10–11 in the supplementary material). By retrieving semantically relevant examples from the memory bank, it enriches the input and enables accurate layout generation, guiding realistic scene synthesis.  \n   This mechanism not only **reduces user effort** but also improves generation quality by **transferring prior knowledge** from similar scenes.  \n- **Ablation Study (with vs. without RAG):**  \n   As RAG is specifically designed for the **user-prompted generation** setting, we conducted a **human preference study** to evaluate its effectiveness.  \n   We generate 100 scene pairs from identical user prompts, with and without RAG, and ask 10 users to compare them based on **diversity**, **realism**, **controllability**, and **physical plausibility**.  \n   As shown in **Table R2**, RAG-enhanced outputs are consistently preferred. This confirms that RAG effectively transforms vague prompts into more **diverse** and **realistic** scenes through **effective retrieval and prior knowledge transfer**.\n\n**Table R2: Human Preference Study Comparing Scene Generation with vs. without RAG.**\n|Criterion|RAG Preferred (%)|Non-RAG Preferred (%)|\n|-|-|-|\n|Diversity|**87**|13|\n|Realism|**82**|18|\n|Controllability|**74**|26|\n|Physical Plausibility|**66**|34|\n|**Overall Preference**|**77**|23|\n\n\n> ### **Q4: Robustness of outpainting across multiple hops and potential artifact accumulation.**\n\n**A4:** Our design explicitly addresses error accumulation during outpainting through a **consistency-aware extrapolation mechanism** (Sec. 3.3, Fig. 3) that ensures stable generation across multiple hops:\n\n- **Local conditioning:** Each new chunk is conditioned on **overlapping regions** from the previous one, which serve as **spatial and appearance anchors** to maintain structural and visual continuity.  \n- **Soft constraints:** We use **latent-level conditioning** instead of directly copying pixels or voxels from the previous chunk. This provides flexibility for correcting inconsistencies during denoising.  \n- **Global regularization:** In addition, each hop also incorporates **global conditions** (layout, boxes, text prompts) to offer high-level semantic guidance across the scene.  \n- **Diffusion robustness:** The denoising process in diffusion models further helps to **suppress minor artifacts** instead of propagating them.\n\nAs shown in Fig. 4 and Page 12 of the supplementary material, our model consistently generates coherent large-scale scenes without noticeable degradation, even after many hops.\n\n> ### **Q5: Reason for the performance discrepancy between Table 5 and Table 7.**\n\n**A5:** The apparent performance difference stems from **different evaluation purposes**:\n\n- **Table 5 (Fine-tuning):** We assess the **training utility** of our generated data by using it to **fine-tune** downstream models. This demonstrates its effectiveness as a form of data augmentation.  \n- **Table 7 (Direct testing):** We evaluate **inference-time compatibility** by **testing** how well pretrained models that are trained on real data perform on our generated data. This assesses the ability of real-data-trained models to generalize to generated inputs.\n\nThus, Table 5 highlights how generated data benefits learning, and Table 7 reveals how it aligns with real data without any adaptation. We will clarify this distinction in the revised manuscript.\n\n> ### **Q6: Recommend adding related works (i.e., Text2LiDAR, Transfer Your Perspective)**\n\n**A6:** We appreciate the suggestion and agree that both works are highly relevant.\n\n- **Text2LiDAR** introduces a text-controllable framework for LiDAR point cloud generation. We extend this to text-conditioned *occupancy* and *multi-view video* generation for more comprehensive simulation.  \n- **Transfer Your Perspective (TYP)** generates high-quality LiDAR from alternative viewpoints to simulate collaborative perception. Likewise, X-Scene focuses on large-scale scene generation with enhanced controllability and scalability.\n\nWe will include appropriate citations and discuss these works in the revised manuscript."}, {"review_id": "fkFz9PVvtI", "reviewer": "Reviewer_SavH", "summary": "This paper proposes X-Scene, a novel pipeline for large-scale driving scene generation with high fidelity and flexible controllability. By dividing scene generation into 1) textual description enrichment, 2) textual scene-graph to layout generation, 3) joint occupancy and image generation, and 4) large-scale scene extrapolation and reconstruction, the proposed X-Scene can generate driving scenes with multiple modalities. Extensive experiments demonstrate the effectiveness.", "questions": "Check the Weakness part for details.", "limitations": "Check the Weakness part for details.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "- Strengths:\n  - This is a solid work with superior engineering efforts, including LLM rephrasing, GNN for layout generation, occupancy generation, multi-view image generation, and occupancy & image outpainting.\n  - The experiments thoroughly evaluate the performance of each component of X-Scene.\n  - The authors provide the source code in the supplementary. I appreciate the authors for doing that.\n- Weakness:\n  - As a solid work with lots of effort, the explanation for each part is unfortunately quite unclear, including:\n    - Line 132: How to describe spatial attributes via texts? Left or right?\n    - Lines 141-144: The introduction of RAG is quite weird. Perhaps an ablation on using or not using RAG is necessary. Moreover, how to retrieve relevant samples for ICL given a specific input is unclear.\n    - Lines 155-157: The definition of \"layout\" is quite unclear throughout the whole paper. According to Lines 155-157, the \"layout\" consists of object bounding boxes and road lines, while in Figure 1 and Table 3, it seems that the \"layout\" is more similar to BEV maps with semantic conditions. \n    - Equation 2: I can see any conditions encoded in Sec. 3.1.\n    - Equation 3: seems like an unconditional outpainting procedure, suggesting no supervision is applied for the outpainted regions.\n    - Lines 216-230: Visual-Coherent Image Extrapolation seems to have no relationship with Geometry-Consistent Scene Outpainting? Otherwise, how to get the reference image at first?\n    - Figure 4 caption: The camera poses supported by the image generative model are pre-defined by the nuScenes dataset, and thus, there is no way to generate novel views, even with occupancy conditions.\n  - Novelty would be a controversial problem of this work, which mainly develops a pipeline for 3D street scene generation, while for each component, different existing works are utilized.\n  - For a pipeline work with complicated component design, it would be essential to analyze the effects of error accumulation after each component.\n- Overall, I think this is quite a solid work which combines recent developments of different areas of street scene generation with lots of effort. My concerns are mainly about the unclear paper writing, and I will consider increasing the score if the aforementioned questions are well-addressed after rebuttal.", "was_revised": true, "final_justification": "Overall, this is a solid paper with superior engineering efforts. Even if each component is relatively straightforward but inheriting existing practices, it is still fantastic to combine everything in a unified system. The authors further explain the unclear details missing in the initial draft, which makes it more beneficial for further progress. Thus, I decided to raise my initial score.", "rebuttal": "We thank the reviewer for recognizing the strength and scope of our work. This includes the **solid engineering effort**, **thorough experiments**, and the **inclusion of code** in the supplementary material.\n\nWe address the remaining concerns that are primarily related to **clarity and technical explanation** through detailed responses and additional clarifications.\n\n> ### **Q1: Clarification on how spatial attributes are described via text in Line 132.**\n\n**A1:** Our textual spatial attributes cover three key dimensions:\n\n- **Relative Position**: e.g., *left, right, front, back, on top of* (e.g., *\"the car is behind the truck\")*;\n- **Relative Size**: e.g., *larger, smaller, taller, shorter* (e.g., *\"the car is smaller than the truck\")*;\n- **Heading Direction**: e.g., *heading left to right*, *facing front*, etc.\n\nThese are parsed via predefined linguistic patterns into scene-graph layouts to encode both **inter-object spatial relationships** and **per-object size/orientation** (see supplementary Pages 7–8). We will clarify this in the final version.\n\n> ### **Q2: Clarification and justification of RAG usage and retrieval strategy for ICL.**\n\n**A2:** We clarify our motivation for using RAG and provide ablation results as follows:\n\n- **Motivation for using RAG:**  \n   RAG is central to the *text-to-scene generation* mode. It transforms **brief, coarse** user prompts (e.g., *“a busy city intersection”*) into **detailed and structured descriptions** (see supplementary Pages 7–8, 10–11) by retrieving semantically relevant examples from the memory bank.\n   This mechanism not only **reduces user effort** but also improves generation quality by **transferring prior knowledge** from similar scenes.\n- **Ablation Study (with vs. without RAG):**  \n   Since RAG is tailored for **user-prompted generation**, we conducted a **human preference study** to evaluate its effectiveness.  \n   Using 100 prompt-matched scene pairs (w/ and w/o RAG), we asked 10 users to evaluate them across four criteria: **diversity, realism, controllability, and physical plausibility**.  \n   As shown in **Table R1**, RAG-based outputs are consistently preferred. This confirms its effectiveness in transforming vague prompts into more diverse and realistic scenes.\n- **Retrieval mechanism for ICL:**   \n   Our memory bank encodes both **high-level scene styles** (e.g., *sunny daytime city intersection*) and **low-level object attributes** (e.g., *construction area with barriers*). This enables flexible retrieval across **different levels of prompt granularity**.  \n   At inference time, we encode the input prompt using `text-embedding-3-small` and perform **top-k retrieval based on cosine similarity**. The retrieved examples act as **few-shot demonstrations** to ground the LLM's generation in relevant prior scenes.\n\n**Table R1: Human Preference Study Comparing Scene Generation with vs. without RAG.**\n|Criterion|RAG Preferred (%)|Non-RAG Preferred (%)|\n|-|-|-|\n|Diversity|**87**|13|\n|Realism|**82**|18|\n|Controllability|**74**|26|\n|Physical Plausibility|**66**|34|\n|**Overall Preference**|**77**|23|\n\n\n> ### **Q3: Clarification on the definition of \"layout\": object boxes and road lines (Lines 155–157) vs. BEV semantic maps (Figure 1, Table 3)**\n\n**A3:** We clarify that our framework supports **two types of layouts** under a **dual-mode controllability design**:\n\n- **BEV vector maps** (object boxes + road lanes): These are generated from user-provided textual prompts via the scene-graph-to-layout diffusion module (as in Lines 155–157 and Fig. 2). This form supports the **text-to-scene generation mode** to offer a compact yet sufficient condition for efficient scene synthesis.\n- **BEV semantic maps**: These are directly specified by users for the **fine-grained spatial control mode** (as in Fig. 1 and Table 3), which enables detailed constraints over scene structure.\n\nBoth layouts are compatible with our scene diffusion pipeline.\n\n- **Additional evaluation:** To further assess their impact on controllability and generation quality, we conduct an ablation study in **Table R2** using different layout types. The results show that both layout types support accurate scene generation, while semantic maps offer stronger spatial priors and lead to slightly better performance.\n\nWe will revise the manuscript to better explain this **dual-mode layout design**.\n\n**Table R2: Ablation on layout types (BEV semantic map vs. vector map).**\n|Input Layout|FID↓|mAP↑|NDS↑|Rd. mIoU↑|Veh. mIoU↑|\n|-|-|-|-|-|-|\n|BEV semantic map|11.29|16.28|26.26|66.48|29.76|\n|BEV Vector map|12.07|15.73|25.84|65.17|28.38|\n\n> ### **Q4: Clarification on the absence of conditioning in Equation (2).**\n\n**A4:** We clarify the role of Equation (2) and where conditioning is applied:\n\n- **Purpose of Equation (2):** This equation defines the deformable attention in the triplane encoder. It is used purely for encoding input occupancy, and is not part of the conditional generation process.\n- **Where conditioning occurs:** Conditioning is applied later in the occupancy diffusion model $\\epsilon^{occ}_{\\theta}$ (**Lines 182–187**), which incorporates layout, box, and text conditions defined in Section 3.1. \n\nWe will make this clearer in the final manuscript.\n\n> ### **Q5: Clarification on conditioning in Equation (3).**\n\n**A5:** Equation (3) defines an outpainting process guided by **implicit conditions**:\n\n- **Overlap-based conditioning:** The process uses the pre-generated **reference latent** $h_0^{ref}$ and an **overlap mask** $M$ to constrain the denoising of the new latent $h_t^{new}$, ensuring geometric continuity in the overlapped region (Fig. 3b).\n- **Implicit conditioning:** The overlapping region acts as an **implicit constraint in latent space** instead of explicitly copying pixels or voxels.\n- **No additional training:** This functions as an **inpainting-style procedure** within the diffusion model and requires no extra training.\n\n> ### **Q6: Clarification on the relationship between image extrapolation and occupancy outpainting.**\n\n**A6:** We clarify that image extrapolation is not an independent process but is **tightly coupled** with occupancy outpainting. Specifically, the generated occupancy is rendered into semantic and depth maps under camera views, which condition the image diffusion (see Fig. 1 and Lines 188-190).\n\n- **First chunk:** 3D occupancy is generated from noise via triplane diffusion and rendered into semantic/depth maps. These serve as conditions to guide the generation of multi-view images.\n- **Subsequent chunks**: Previously generated occupancy and images serve as references. The new occupancy is rendered again to guide the next image region.\n\nThis design ensures consistent extrapolation of both **geometry and appearance** to enable coherent large-scale scene generation.\n\n> ### **Q7: Clarification on camera poses and novel view generation in Figure 4 caption.**\n\n**A7:** The reviewer is correct, and we appreciate the correction. In Figure 4, the rendered images are generated under the **pre-defined camera poses** from the nuScenes dataset. This setup ensures alignment with real-world views for consistent visualization and evaluation.  \nWe will revise the figure caption to clearly state this setup and avoid potential confusion.\n\n> ### **Q8: Clarification on the novelty of the proposed framework.**\n\n**A8:** We emphasize two key innovations that go beyond simple integration:\n\n- **Multi-Granular Control Mechanism**:  \n  We introduce a **dual-mode control framework** that supports both (1) *text-prompted* scene generation for high-level usability and (2) *layout-guided* generation for precise control. This design balances usability and precision, enabling intuitive *high-level customization* (mode 1) and precise *low-level geometric control* (mode 2). To the best of our knowledge, **no prior work has explored such a flexible and unified control mechanism especially in the context of text-to-scene generation**.  \n- **Unified Generation and Extrapolation Pipeline:**  \n  Our framework jointly models *3D occupancy and multi-view image* generation, coupled with a *consistency-aware outpainting mechanism* for scalable scene extrapolation. This design ensures both geometric continuity and visual coherence. As a result, it enables realistic closed-loop and E2E simulation. This is an **improvement over existing methods that are limited to single-chunk or image-only generation**.\n\nThese contributions collectively establish our X-Scene as a cohesive and powerful framework for large-scale and controllable 3D scene generation. They advance both the system’s capabilities and its practical applicability.\n\n> ### **Q9: Error accumulation in the cascaded pipeline.**\n\n**A9:** We address concerns about error accumulation and pipeline robustness through a **noise-injection ablation study**:\n\n- **Ablation setup:** We introduce Gaussian noise to the initial layout (i.e., 3D box centers and lane points) with 25% probability to simulate partial noise arising from imperfect layout predictions. We then assess the impact on occupancy generation, image synthesis, and downstream performance.\n- **Ablation results:** As shown in **Table R3**, the results **degrade gracefully** despite the injected noise. Particularly, in downstream detection (mAP ↓ 1.41) and BEV segmentation (Rd. mIoU ↓ 1.20, Veh. mIoU ↓ 1.32). This indicates the strong robustness of our pipeline.\n- **Why robust:** This resilience stems from our **multi-stage alignment design**: (1) occ rendered semantic/depth maps provide strong geometric priors for image generation; (2) overlap-aware extrapolation ensures spatial continuity across chunks.\n\n**Table R3: Ablation on Error Accumulation.**\n|Input Layout|OccGen FID³ᴰ↓|OccGen F³ᴰ↑|ImgGen FID↓|3DOD mAP↑|3DOD NDS↑|BEVSeg Rd. mIoU↑|BEVSeg Veh. mIoU↑|\n|-|-|-|-|-|-|-|-|\n|Clean Layout|**258.8**|**0.778**|**11.29**|**16.28**|**26.26**|**66.48**|**29.76**|\n|Noisy Layout|276.3|0.742|12.47|14.87|25.02|65.28|28.44|"}]

Author rebuttal:

Meta-review: This work presents a novel scene generation framework that is both highly realistic and controllable. The framework combines multiple stages of generation and supports both low-level and high-level control. Post-rebuttal, all reviewers reached consensus in favor of acceptance, and the AC concurred.

The AC recommends acceptance, with the expectation that all promised changes and additional experimental results will be incorporated into the final version.
Decision: Accept (poster)