Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency
Authors: Xiangyu Guo
Zhanqian Wu
Kaixin Xiong
Ziyang Xu
Lijun Zhou
Gangwei Xu
Shaoqing Xu
Haiyang Sun
BING WANG
Guang Chen
Hangjun Ye
Wenyu Liu
Xinggang Wang
Venue: NeurIPS 2025 poster ()
Keywords: autonomous driving; driving scenario generation
Primary area: deep_learning

Abstract: We present Genesis, a unified world model for joint generation of multi-view driving videos and LiDAR sequences with spatio-temporal and cross-modal consistency. Genesis employs a two-stage architecture that integrates a DiT-based video diffusion model with 3D-VAE encoding, and a BEV-represented LiDAR generator with NeRF-based rendering and adaptive sampling. Both modalities are directly coupled through a shared condition input, enabling coherent evolution across visual and geometric domains. To guide the generation with structured semantics, we introduce DataCrafter, a captioning module built on vision-language models that provides scene-level and instance-level captions. Extensive experiments on the nuScenes benchmark demonstrate that Genesis achieves state-of-the-art performance across video and LiDAR metrics (FVD 16.95, FID 4.24, Chamfer 0.611), and benefits downstream tasks including segmentation and 3D detection, validating the semantic fidelity and practical utility of the synthetic data.

Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency
Xiangyu Guo",, Zhanqian Wu"", Kaixin Xiong" Ziyang Xu', Lijun Zhou', Gangwei Xu',
Shaoqing Xu², Haiyang Sun i, Bing Wang', Guang Chen?, Hangjun Ye?, Wenyu Liu', Xinggang Wang X
Huazhong University of Science and Technology Xiaomi EV
https://xiaomi resear ch. gi github. 10/ggeess//es////////
Abstract
We present Genesis, a unified world model for joint generation of multi-view driving videos and LiDAR sequences with spatio-temporal and cross-modal consis- tency. Genesis employs two-stage architecture that integrates a DiT-based video diffusion model with 3D-VAE encoding, and BEV-represented LiDAR generator with NeRI-based rendering and adaptive sampling. Both modalities are directly coupled through a shared condition input, enabling coherent evolution across visual and geometric domains. To guide the generation with structured semantics, we introduce DataCrafter, a captioning module built on vision-language models that provides scene-level and instance-level captions. Extensive experiments the nuScenes benchmark demonstrate that Genesis achieves sfute-of-the-an perfor- mance across video and LiDAR metrics (FVD 16.95, FID 4.24, Chamfer 0.611), and benefits downstream tasks including segmentation and 3D detection, validating the semantie fidelity and practical utility of the synthetic data.
1 Introduction
As autonomous driving systems progress toward higher levels of intelligence, generating diverse, realistic driving scenarios [34 30 has become essential for improving the robustness and safety of perception and planning modules. While recent advances have explored video generation and LiDAR sequence synthesis independently, achieving holistic multimodal consistency across visual and geometric modalities remains open challenge, limiting the development of faithful world models that can simulate real- world sensor interactions.
As shown in Fig. [ existing driving scene generation methods typically focus on generating data in 18 (6) or LiDAR point clouds [54 53 2
single modality, usually RGB videos [35] 10 39 50 221122 (2). While these approaches have significantly advanced the field of driving scene generation, they overlook the synergistic potential of multimodal generation and lack consistency in aligning RGB videos with various sensor data, leading to limitations in weal-world applications. Critically, they fall short of constituting a comprehensive world model, as they fail to capture the intricate dependencies between different sensory streams. Many of these methods rely one-step layout-to-dar data pipelines
40
42
conditioned solely on coarse spatial priors, such as BEV maps [36] 10
43
46
39
50
40) or
3D boxes [12 33), which restricts their ability to capture complex scene dynamics and fine-
grained semantics. To propel driving scene generation into the multimodal domain, UniScene [00]
involves occupancy grids in for multimodal generation, while the acquisition of occupancy labels is
* Equal contribution, the work was done when Xiangyu Guo was an intern of Xiaomi EV. Project lead (sunbaiyang1 @xiaomi. com). & Corresponding author digaan@@hutt.edu. en)
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Input
Video Gen Model
Generation
Input
Model
Oce Gen
Lidar Gen Model
Classical video generation pipeline
(c) Two-stage LiDAR/Video generation pipeline
Lidar Generation Model
Video Gen Model
Input
Input
Lidar Gen Model
Classical lidar cloud points generation pipeline
Ours
Figure 1: Comparison of Multimodal Scene Generation Pipelines. (a) Video-only generation, (b)
LiDAR-only generation, (c) gerupancy-guidees dual-brant generation, (d) Our Genesis.
highly costly, which severely limits its application. Moreover, most existing generation approaches generally rely on limited semantic annotations. typically in the form of coarse labels or generic captioning models, without fully leveraging the fine-grained descriptive capabilities of modern vision- language models (VLMs) [S] 23 1. This lack of structured semantic grounding restricts the fidelity, controllability, and contextual alignment of generated scenes, further distancing them from truly grounded and actionable world model.
Therefore, advancing toward a realistic driving world model requires more than unimodal fidelity-it demands tight cross-modal alignment and fine-grained semantic grounding ensure spatio-temporal coherence. However, existing methods suffer from three core limitations: (1) they often decouple video and LiDAR generation, weakening cross-modal consistency; (2) they rely on intermediate representations like occupancy grids that may cause information loss; (3) they provide limited semantic supervision, typically in the form of coarse layout maps or generic captions, which hinders scene-level controllability and realism.
To address these limitations, we propose Genesis, unified joint generation framework tailored for autonomous driving that synthesizes multi-view RGB videos and LiDAR point clouds a consistent and semantically grounded manner. Our framework introduces three key innovations:
Unified Multimodal Generation Architecture. Genesis employs a unified pipeline where both video and LiDAR branches use shared conditional inputs, including scene descriptions and scene layouts, etc. To further guarantee the consistency between point clouds and image. BEV features extracted from projected image features are incorporated conditional inputs into the Lidar diffu- sion model. This approach enhances cross-modality consistency without relying on intermediaries like occupancy grids.
Structured Semantic Captions via DataCrafter. To improve semantic controllability. we introduce DataCrafter, a structured captioning module built upon vision- lssoon-angguag models. extracts multi- view, scene-level and instance-level descriptions that are fused into dense, language- guided priors. These captions provide detailed semantic guidance to both video and LiDAR generators, resulting in outputs that
only realistic but also interpretable and controllable.
State-of-the-art performance. Extensive experiments conducted on the nuScenes benchmark reveal that Genesis achieves the most advanced performance in both video and LiDAR metries.
2 Related Work
Driving Scene Video Generation. Driving scene video generation has seen rapid progress, with many methods relying on structured spatial priors for controllability. BEVGen [35] conditions generation on BEV maps to encode road and vehicle layouts, but ignores height information, limiting its 3D representation capacity. BEVControl [46] addresses this by introducing a height-lifting module to partially restore scene geometry. MagicDrive [10] advances 3D-awareness through geometric constraints and cross-view attention, while MagicDriveDiTT
leverages diffusion transformers for improved temporal fidelity. MagicDrive3D g further employs deformable Gaussian splatting for
coarse 3D reconstruction. Driverreamer (39 proposes hybrid Gaussians for temporally consistent
synthesis of complex maneuvers. Voxel-based approaches such Wo VoGnn (55 and Drive-WM, [40]

---

Imput
Camera-branch
Oulpul
PoseNet
1
SIDiT-Bleck-4
XN
STDiT-Blocke
3D VAT. Lomder
dif
Human Pose
STDiT-Block-C
STDiT-Block.
From
Video
N oisy
Conditions
Frames
Layout
Video Latent
Boxes
Lidar-branch
VAF
xN
Trajectory
To Cam
STDiT-Block
STDiT-Block-L
STDIT Block-L.
Litar
Generated
Scene
STDoT-Block.L
XN
STDiT-Block-L
Description
Semantic
Noisy BEV Latent
Conditions
From
Lidar Sequence
Generated
Conditions
Encuder & Decoder
Lidar Braneh
Camera Branch
Trainable
Frozen
Figure 2: Overview of the Genesis architecture for joint video and LiDAR generation. A dual- branch design processes shared semantic conditions via camera and LiDAR pathways, using STDiT blocks for spatiotemporal generation and BEV encoder for geometric alignment.
explore 4D voxel diffusion and latent substitution model spatiotemporal dynamics. Despite progress, most methods remain unimodal and rely on weak or decoupled temporal priors, limiting their ability generate globally consistent, semantically grounded sequences.
LiDAR Point Cloud Generation. LiDAR point clouds form the perceptual cornerstone of au- tonomous driving, enabling precise 3D understanding through sparse yet geometrically rich mea- surements. PointNet [31 and VoxelNet (52) laid the foundation for point- and voxel-based LiDAR processing. Building these, neural radiance fields (NeRF) [28] 45 41 incorporate LiDAR priors, improving training efficiency and geometric fidelity. Beyond static scenes, recent work emphasizes capturing spatiotemporal dynamics and generating high-fidelity, temporally consistent LiDAR se- quences. Copilot4D 48 models long-range dependencies via LiDAR tokenization and hierarchical employs video diffusion models generate temporally consistent LiDAR employ denoising and conditional diffusion to enhance geometry and con- sistency. While recent methods begin to capture temporal dynamics, they typically remain unimodal and underexplore semantic integration. In contrast, our framework jointly generates semantically grounded, spatiotemporally coherent sequences across both video and LiDAR modalities.
Transformers. ViDAR 47
sequences, while [3
32
3 Method
In this section, we present Genesis, a unified generation framework designed to jointly synthesize multi-view video and LiDAR point cloud data with fine- ime-grained semantic consistency.
Overview. As shown in Fig. 2 Genesis is architected as a unified world model with a dual- branch design that jointly synthesizes multimodal driving scenes, including videos and LiDAR point clouds. The camera branch leverages a DiT-based spatiotemporal diffusion backbone with a 3D-VAR encoder for fine-graines visual dynamics (Sec. 3.2). while the LiDAR branch employs B-W-aeere autoencoder with NeRF-style rendering and adaptive sampling for accurate geometric reconstruction (Sec. 3.3). Both branches jointly conditioned scene layouts, intermediate video latent features, and structured scene semantics to maintain strong cross-modal consistency, Semantic priors such as captions are derived from our DataCrafter module (Sec. 3.I and injected as global conditioning to enhance alignment across modalities. More architectural details are provided in the supplementary material (Sec.
3.1 DataCrafter Module for Scene-Level Semantics
propose DataCrafter, a structured captioning framework designed for multi-view autonomous driving videos. As illustrated in Fig. 3 given multi-view input V (Vi...., Vi J. we segment

---

CAM BACK LEFT
Quality
check
Time:
Weather:
CAM CAM_FRONT FRONT, RIGHT
CAM BACK RIGHT
Road lane:
Surmunding:
End
Tralfs: [Fren].... (BBck))..(BBak......
Step
Step
Step
Step
Figure 3: Crafeer pipeline for structured multi-view captioning. Videos are segmented and filtered via VLM-based quality checker (1-2), then per-view captions are generated and fused into coherent, structured descriptions (3-4). Steps 1-4 are used during training, while only 3-4 are used inference.
into clips C= fci...., cN) via a scene boundary detector. Each clip is scored by a visual-language model base) module:
S(G;) Qetarity (c:) Q (c)) À aestteetes (c)),
where Q terms represent VI---verived subscores and X; are fixed weights.
To ensure consistency across overlapping views, we introduce multi-view consistency module. Let y denote the caption from view Vg- Each caption is encoded via pre-trained VLM:
5 = F(fOO&))
(2)
where o(.) is language encoder and F()) performs summarization and redundancy removal.
Finally, we produce structured caption for each clip c;:
) = Ei, (1o,, bj d,))/=
(3)
where E; encodes global scene context (e.g., weather, road, time), and each object is represented by its category oj. bounding box bj- = (æ M1 U2js 322). and grounded description d,-
This hierarchical design enables coherent and detailed scene-level and object-level captioning across
time and views. More information about DataCrafter be found in Sec. A
Outpot
Outpet
FFN
FFN
Spatial Temporal Campress
Temporal Atteation
3D Boxes
STD-T-BI Bluck Cam
30 Boxes
Tempural Selt-AtienSon
MLP
MIPP
Trajectory
Tuat
Text Ent
MultiHead
Cress-Atiention
Scrne Descrigtion
Scvra Desrription
Noisy Video
Spatial Sell'
Spotial Self'
Figure 4: Left: STDiT-Block-C Camera branch. Right: STDiT-Block-L of Lidar branch.
3.2
Video Generation Model
To ensure coherent multi-view video generation, a DiT-based diffusion backbone is extended with 3D-aware latent encoding and conditioned scene-level infors-including road topology and linguistic descriptions via attention mechanisms, enabling spatial alignment, temporal consistency, and semantic fidelity.
Structured Semantic Priors and Cross- View Conditioning. To enable view-consistent and se- mantically grounded video generation, structured BEV layout s {C, H, B3} is constructed as

---

Right View 2D Layout
Left View 20 Layout
piog
proj
Lase
Uuman key points
M, 12fF-WNC
Vehicle
M, R
Ego Yehicle
21 Layout
Figure 5: Illustration of the scene layouts projection pipeline, which enables view-consistent and semantically grounded video generation.
shown in Fig. 5 where L. H. and B denote lane segments, human pose keypoints, and 3D vehicle bounding boxes, respectively. Each element in S' is projected onto the 2D image plane of view using calibrated intrinsics K, and extrinsics E,, resulting in a set of view-specific semantic control maps Ma E R #xxxC These projected maps comprising layout curves, pose skeletons, and instance masks are subsequently encoded and fed into a Control-DiV' module.
Specifically, each block of the spatiotemporal diffusion transformer receives guidance from M, via cross-attention, allowing the model to integrate structured priors at every denoising timestep. The forward process at timestep / for view v is formulated as:
DiT (=! )) + CrossAun(=((-1), MM),,
where D denotes the base diffusion transformer and CrossAttn represents the conditional modula- tion induced by My.
Pedestrian Pose Enhancement. To enhance dynamic scene semantics, human keypoints are detected using YOLOv8x- and projected into each view, forming semantic channels that augment layout and instance conditions.
Latent Encoding via 3D VAE. To ensure spatiotemporal consistency of control signals, 3D Variational Autoencoder (VAE) is employed to compress the multi-frame scene layouts S and images into latent representation 2 e /R/ xexhxx The decoder reconstructs the RGB images from the denoised latents.
Scene-Level Caption Conditioning. To encode high-level semantics, descriptive captions generated via the DataCrafter module and processed using a pre-trained T5 encoder:
Ccap Ener
2s = (
(5)
Both embeddings are used conditioning inputs via cross-attention in the DiT blocks:
CrossAutn(g) 2,, k (ecpp =z];
!ecaps zs]).
(6)
Spatiotemporal Control via STDiT-Block-aam As shown Fig. our model integrates a Semantic-Aligned Control Transformer modulate early-stage diffusion blocks. semantic features are injected via control attention:
Attn (A hase 2,))
i= K.
This module works in parallel with the core DiT pipeline, which includes spatial self-attention, cross- view attention, and temporal attention blocks to ensure view consistency and motion smoothness.
3.3 Lidar Generation Model
Lidar generation model is composed of two modules: Point Cloud AutoEncoder Module for reconstructing point clouds and a SpatioTempora Diffusion Module for generating Bird's Erye- View(BEV) representation.
Point Cloud AutoEncoder. We design a BEV representation autoencoder to compress sparse and unordered point clouds into structured latent embedding. Inspired by 449 48 we first voxelize the point clouds into BEV grids and then employ a Swin Transformer (24 backbone downsample the BEV features by factor of 8x in spatial aspect and output the latent hidden feature with channel

---

4. which is further used in the SpatioTemporal Diffusion Module, as shown in Fig. 4 We decode point clouds from this latent embedding via Swin-based decoder and a NeRF [28 -baeed rendering module. We adopt spatial skipping algorithm [8 to avoid the accumulated errors introduced by empty grids. The total training losses include vanilla depth I.1 loss [SS. occupancy loss [27] and surface regularization loss [7]. Furthermore, we propose simple post-process algorithm to filter those noisy points generated from nerf rendering process. Specifically, we use filter point cloud that falls within grids, the occupancy value of which is smaller than the threshold.
Spatiotemporal Diffusion with ControlNet-Branche With the latent embedding from the Point Cloud AutoEncoder Module, we design diffusion network to learn this representation to generate realistic point clouds. Similar the structure adopted in the video generation, we use a dual DiT- based network with ControlNet assistance. For semantic condition input, we use scene captions and cappeeaaaeeeee
scene layouts to provide structured scene priors. To keep cross modal consistency, we use the camera branch's 's RGB video outputs another condition input. We adopt widely used LSS [29] algorithm to convert the perspective images to BEV feature. For geometric condition input, we encode 31) bounding boxes into embeddings for providing accurate foreground information. The image BEV feature is concatenated with the scene layouts latent feature and then send into the ControlNet branch as input. For given latent token 2; at block i, the cross-altention operation cloud be formulated as:
2 = Cross.Attnq = Z4, k = [Ccap- @box]
[Ecaps ebox)).
(8)
where the embeddings Gcap and Cbox are derived from road sketches and 3D bounding boxes respec- tively. To ensure temporal coherence, the STDiT-Block- applies multi-head self- self-attention operation. Given input z', the tokens are updated as 2 = MHSA(=") + 2'. The entire LiDAR diffusion model is trained using rectified flow schedule 6 to enhance generation quality.
4 Experiments and Main Results
4.1 Implementation details
Training and evaluation are conducted on the nuScenes 44 dataset, which includes 1,000 urban driving scenes (700 train / 150 val 150 test). Semantic occupancy labels at 2Hz are interpolated to 12Hz for dense supervision 10 20 Multimodal clips are sampled and evaluation follows the standard protocol [IO, 20 [I] using 5,369 and 6,019 validation clips. Additional implementation details and training strategy are provided in the supplementary material (Sec. A..
4.2 Qualitative comparison and Versatile Generation Ability
High-quality video samples generated by our method demonstrate strong geometric fidelity and visual coherence across challenging scenes. As shown in Fig. 6 and 10 our model faithfully preserves vehicle shapes, lane structures, and environmental textures. In contrast, MagicDrive suffers from object deformation and layout artifacts, while Panacea exhibits hallucinated content and distorted backgrounds.
Beyond visual realism, the proposed framework enables controllable and diverse scene synthesis. By editing input captions, global scene attributes such as lighting and weather can be modified, as illustrated in Fig. Furthermore, altering the veh-lll trajectory in the layout facilitates novel view generation, yielding coherent scene continuations with consistent geometry and appearance across perspectives, as shown in Fig.
4.3 Video Generation Results
Tab. T reports the quantitative comparison of video generation performance under various generation modes. We evaluate model in three configurations: without first-frame conditioning, with first- frame conditioning, and with noisy latent initialization. In all settings, our method consistently achieves superior performance across both FVD and FID metrics.
In the setting without first-frame conditioning, our method achieves an FVDmuti of 83.10 and a FIDmatti of 14.90, outperforming prior works such as DriveDreamer-2 (50), MagicDrive-VV [10]. and Drive-WN, 40 With first-frame conditioning, our method further improves to 16.95 FVD
and 4.24 FID, showing competitive results compared to MiLA [88 while maintaining temporal

---

GT
Read Sketch
Paroaca
Magicleeve
Vision Distortion
Twisted Vehicle
videos are generated without frame reference image
Figure 6: Qualitative comparison of video generation. From top to bottom: (1) Ground-trute images, (2) Scene Layouts input, (3) Panacea [42, (4) MagicDrive [IO], (5) Ours. Panacea suffers from hallucinated textures and geometric misalignment. MagicDrive shows vehicle distortion and broken structures. In contrast, ours preserves accurate layout, object shapes, and background integrity.
Table 1: Video Generation Comparison on nuScenes validation set, where green and blue represent the best and the second best values.
Method
Gen. Mode
Multi-view
Video
Sample Num
Frame Num
FVD
FID
DriveDreamer-2
50
w/o first cond wfo first cond wfo first cond
105.10
25,00
MagicDrive- V2
16
94,
20.91
Drive-WM WM
40
122.70
15.80
Ours
w/o first cond
/
/
5369
16
83.10
14.90
Mil.A
cond
/
$369
16
18.20
3.00
DriveDreamer-2
50
w/ first cond
/
55.70
11.20
Ours
w/ first cond
/
/
5369
16
16.95
4.24
UniScene
X0
w/ noisy latent
6019
70.52
6.12
Ours
noisy latent
6019
16
67.87
6.45
consistency and structural fidelity. Under the noisy latent setting, achieve 67.87 FVD and 6.45 FID on 6,019 samples, surpassing the previous best results reported by UniScene 20
4.4 LiDAR Generation Results
Tab. reports quantitative comparison of LiDAR sequence generation performance between prior state-of- -f--the-rt methods and our proposed Genesis framework. Evaluation is conducted following the HERMES (1 protocol, using Chamfer Distance as the primary metric within a spatial volume of 51.2, 51.2] meters in the horizontal plane and [-3, meters in height.
Table 2: Lidar Generation Comparison on nuScenes validation set, where green and blue represent the best and the second best values. "gt_img" img" and "gen_img" indicate using ground-truth or generated images BEV condition input, respectively.
Method
Condition Type
Chamfer@1s
Chamfer@2. | Chamfer@3s
4D-Occ
19
gt gt_img
1.13
1.53
2.11
ViDAR
4
gt gt_img
1.12
1.38
1.73
HERMES
51
gt_img
0.78
0.95
1.17
Ours
gt_img
0.611
0.625
0.633
Ours
gen_img
0.634
0.638
0.641

---

CAM
CMMRA & RIGH
Figure 7: Joint generation of LiDAR and multi-view video. Our method generates spatially aligned LiDAR and camera views conditioned on a shared right turn T-junction layout.
DataCrafter
Wio DataCrafter
W DataCrafter [RIGHT BACK CAMI A black SUV prominencly partially cccupying the right side of the frame. male driver clearly visible through background festues brick building under construction. with a blue scaffolding cover and metallic support hars the entrnee. Two bicycles parked sidewalk betteath scaffolding.
Wio DataCrafter [RIGHT BACK CAMI Dayttime. Driving on the street. A black SUV passing.
Figure 8: Comparison with and without CrtaCrafter. Our method yields semantically rich, structurally consistent scenes, while the baseline lacks fine-grained details and geometric accuracy.
Genesis consistently outperforms existing approaches across both short- and long-term horizons. At 1 second, it achieves Chamfer Distance of 0.611, 1, surpassing the previous best (0.78 by HERMES) by 21%. At seconds, the advantage widens to a 45% relative reduction (from 1.17 to 0.633), underscoring the model's ability to maintain geometric fidelity over extended temporal predictions. Notably, the performance remains stable when replacing ground-truth images with generated images as conditional input, indicating strong robustness and effective cross-modal generalization. These results highlight the efficacy our structured semantic conditioning and hierarchical generation design, establishing Genesis new state-of-the-art for LiDAR sequence synthesis, especially in long-horize horizon prediction scenarios where geometric consistency is critical.
4.5
Joint Generation Results
The proposed framework enables coherent joint generation of LiDAR and multi-view vidcos with consistent semantic and spatial alignment. As illustrated in Fig. 7 I2 and 13 our model maintains cross-modal correspondence even in complex scenes and over long temporal horizons (Fig. 14 15
Dynamic objects and road structures are faithfully preserved across modalities, demonstrating the model's 's ability to synthesize structured scene dynamics with high temporal and geometric consistency.
4.6 Ablation Studies
We conduct an ablation study to assess the effectiveness of the proposed DataCrafter and PoseNet modules, which introduce structured semantic captions into the video generation pipeline. As shown in Tab. 3 removing DataCrafter causes a notable degradation in generation quality, with FVDmuti increasing from 85.91 to 117.49 and FID multi rising from 15.20 to 22.32. Including the PoseNet module further improves performance, reducing FVD to 83.10 and FID to 14.90. These results underscore the importance of semantic guidance for maintaining temporal consistency and structural fidelity. As illustrated in Fig. [ coption-conditional generation produces sharper structures and more coherent layouts, whereas the ablated variant exhibits missing objects and visual distortions. In addition, PoseNel contributes fine-grained dynamic cues by localizing human anatomical keypoints.
Another ablation study on LiDAR generation is presented in Tab. to assess the contributions of BEV latent features and first-frame conditioning. Removing first-frame input leads to a consistent performance drop across all prediction horizons (c.g., Chamfer @ Is increases from 0.634 to 0.668), highlighting its importance for temporal guidance. Further excluding BEV latent features causes additional degradation, indicating their role in preserving spatial structure. These results demonstrate that both components are critical for achieving geometrically consistent long-range LiDAR synthesis.
8

---

Table 3: Ablation in the video generation model.
Method
Gen. Mode
Sample Num
Frame Num
FVDmuti 4
FIDmutt
baseline
w/o first cond
5369
117.49
22.32
w/ DataCrafter
w/o first cond
53369
16
85.91
15.20
w/ DataCrafter and PoseNet
w/o first cond
5369
16
83.10
14.90
Table 4: Ablation in the Lidar generation model.
Method
Chamfer@1 Is 4 Chamfer@ 2s
Chamfer@
Img BEV Latent + w/o Ref. Frame
0.661
0.669
0.673
w/ Img BEV Latent w/o Ref. Frame
0.668
0.672
0.677
w/ Img BEV Latent w/ Ref. Frame
0.634
0.638
0.641
4.7 Domain gap and Downstream task Utility
Our evaluation approach examines generated videos from two perspectives: the degree of reality measured by domain map metries, and the functional utility demonstrated in downstream training applications such as 3D object detection tasks.
Table 5: Domain gap on bev segmentation.
Table Effect of Multimodal Data Generation on 3D Object Detection. Inputs of the methods in the table are both camera and lidar modalities.
Method
mloUt
mAPt
MagicDrive 10
18.34
11.86
Method
mAPt
NDS1
MagicDrive3D 8
18.27
12.05
MagicDriveDiT 9
20.40
18.17
Baseline
66.87
69.65
DiVE 18
35.96
24.55
Ours(+cam_ gen)
67.09 (+0.22)
70.12 (+0.47)
Cogen 16
37.80
27.88
Ours(+lidar_ge) gen)
67.69 (+0.82)
70.58 (+0.93)
Ours
38.01
27.90
muns+cam&liddrr gen)
67.78 (40.91)
71.13 (+1.48)
As shown in Table 5. we generate videos according to conditions from validation set to metric the domain gap following [9] Our method achieves the best mIoU (38.01) and mAP (27.90) bev segmentation, surpassing prior methods like DiVE [18] and Cogen [16], demonstrating strong semantic alignment and visual fidelity. As shown in Tabl6
evaluate the effectiveness of our generative data on the BEVFusion (22 framework for 3D object detection. Our approach yields consistent improvements across all settings, increasing the mAP from 66.87 to 67.78 and NDS from 69.65 to 71.13. Notably, joint generation of both camera and LiDAR modalities achieves the highest gains (+0.91 mAP/ +1.48 NDS), demonstrating the complementary benefits of multimodal generation. These results validate the utility of high-quality synthetic data in enhancing downstream perception tasks, especially in data-scarce long-tail scenarios.
Conclusion
We propose Genesis, unified world model for joint multi-view video and LiDAR point cloud generation in autonomous driving. Integrating structured semantic priors via the DataCrafter module and ensuring cross-modal alignment through shared conditioning pipeline, it bridges visual and geometric modalities to enable high-fidelity, spatiotemporally coherent, and semantically consis- tent multimodal sequence synthesis. Experiments nuScenes confirm Genesis achieves SOTA performance in both modalities, validating its robustness and scalability.
Limitations and future work. Despite Genesis's 's strong performance, several limitations remain. Training demands significant computational resources, motivating future efforts in model compression efficient architectures. The framework is currently evaluated nuScenes, and generalizing to other domains such as robotics or aerial pwwceptionnrreeeeeeeee requires further adaptation. Addressing these challenges is essential for broader deployment in safety-critical systems.
Acknowledgement: This work was partially supported by the National Natural Science Foundation of China (No. 62276108).

---

References
[1]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint 0rXiv22309/66609 2023.
[2] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1090~1099, 2022.
[3] Julie Stephany Berrio, Mao Shan, Stewart Worrall, and Eduardo Nebot. Camera-lidar inte- gration: Probabilistic sensor fusion for semantic mapping. IEEE Transactions on Intelligent Transportation Systems, 23(7):7637-7652 2021.
[4] Holger Caesar, Varun Bankiti, Alex Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-11633 2020.
[5] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generie visual-linguiste tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24185-241999 2024.
[6] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
[7] Lan Feng, Quanyi Li, Zhenghao Peng, Shuhan Tan, and Bolei Zhou. Trafficgen: Learning to generate diverse and realistic traffic scenarios. In 2023 IEEE international conference on robotics and automation (ICRA), pages 3567-3575. IEEE, 2023.
[8] Ruiyuan Gao, Kai Chen, Zhihao Li, Lanqing Hong, Zhenguo Li, and Qiang Xu. Magic- drive3d: Controllable 3d generation for any-view rendering in street scenes. arXiv preprint M.Xiv:2405.14475 2024.
[9] Ruiyuan Gao, Kai Chen, Bo Xiao, Lanqing Hong, Zhenguo Li, and Qiang Xu. Magicdrivedit: High-resolution long video generation for autonomous driving with adaptive control. arXiv preprint 2rXiv:2411.13880 2024.
[10) Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhenguo Li, Dit- Ya- Yeung, and Qiang Xu. Magicdrive: Street view generation with diverse 3d geometry control. arXiv preprint arXiv:2310.02601, 2023.
[11] Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. arXiv preprint arXiv:2405.1744 173988 2024.
[12] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The international journal of robotics research, 32(11):231-1227, 2013.
[13] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[14] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205. 15868, 2022.
[15] Junjie Hu, Mete Ozay, Yan Zhang, and Takayuki Okatani. Revisiting single image depth estimation: Toward higher resolution maps with accurate object boundaries. 20/9 IEEE winter conference applications of computer vision (WACV), pages 1043-1051. IEEE, 2019.
10

---

[16] Yishen Ji, Ziyue Zhu, Zhenxin Zhu, Kaixin Xiong, Ming Lu, Zhiqi Li, Lijun Zhou, Haiyang Sun, Bing Wang. and Tong Lu. Cogen: 3d consistent video generation via adaptive conditioning for autonomous driving. arXiv preprint arXiv:2503.22331, 2025.
[17] Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, Thomas Funkhouser, et al. Local implicit grid representations for 3d scenes. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 6001-6010, 2020.
[18] Junpeng Jiang, Gangyi Hong. Lijun Zhou, Enhui Ma, Hengtong Hu, Xia Zhou, Jie Xiang. Fan Liu, Kaicheng Yu, Haiyang Sun, et al. Dive: Dit-based video generation with enhanced control. arXiv preprini urXiv:2409.01599 2024.
[19] Tarasha Khurana, Peiyun Hu, David Held, and Deva Ramanan. Point cloud forecasting as a proxy for 4d occupancy forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pages 1116-1124, 2023.
[20] Bohan Li, Jiazhe Guo, Hongsi Liu, Yingshuang Zou, Yikang Ding, Xiwu Chen, Hu Zhu, Feiyang Tan, Chi Zhang, Tiancai Wang. et al. Uniscene: Unified occupancy-centre driving scene generation. arXiv preprini urXiv:24/2.05433. 2024.
[21] Yingwei LI, Adams Wei Yu, Tianjian Meng, Ben Caine, Jiquan Ngiam, Daiyi Peng, Junyang Shen, Yifeng Lu, Denny Zhou, Quoe V Le, et al. Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17182-17191, 2022.
[22] Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Tao Tang, Bing Wang, and Zhi Tang. Bevfusion: A simple and robust lidar-camera fusion framework. Advances in Neural Information Processing Systems, 15:10421-10433 2022.
[23] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-349999 2023.
[24] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[25] Jiachen Lu, Ze Huang, Zeyu Yang. Jiahui Zhang, and Li Zhang. Wovogen: World volume-aware diffusion for controllable multi-camera driving scene generation. In European Conference on Computer Vision, pages 329-345. Springer, 2024.
[26] Enhui Ma, Lijun Zhou, Tao Tang. Zhan Zhang, Dong Han, Junpeng Jiang, Kun Zhan, Peng Jia, Xianpeng Lang, Haiyang Sun, al. Unleashing generalization of end-to-end autonomous driving with controllable long video generation. arXiv preprint arXiv:2406.01144 2024.
[27] Lars Mescheder, Michael Oechsle, Michael Niemeyer. Sebastian Nowozin, and Andreas Geiger. Oceupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages
2019.
(28) Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 6009 2021.
[29] Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pages 194-210. Springer, 2020.
[30] Ethan Pronovost, Meghana Reddy Ganesina, Noureldin Hendy, Zeyu Wang, Andres Morales, Kai Wang, and Nick Roy. Scenario diffusion: Controllable driving scenario generation with diffusion. Advances in Neural Information Processing Systems, 36:68873-68894 2023.
[31] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.

---

[32] Haoxi Ran, Vitor Guizilini. and Yue Wang. Towards realistic scene generation with lidar diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14738-14748, 2024.
[33] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard. Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2446-2454, 2020.
[34] Shuo Sun, Zekai Gu, Tianchen Sun, Jiawei Sun, Chengran Yuan, Yuhang Han, Dongen Li, and Marcelo H Ang. Drivescenegen: Generating diverse and realistic driving scenarios from scratch. IEEE Robotics and Automation Letters, 2024.
[35] Alexander Swerdlow, Runsheng Xu, and Bolei Zhou. Street-view image generation from a bird's-eye view layout. IEEE Robotics and Automation Letters, 2024.
[36] Alexander Swerdlow, Runsheng Xu, and Bolei Zhou. Street-view image generation from sird's-eye view layout. IEEE Robotics and Automation Letters, 2024.
[37] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: new metric for video generation. 2019.
[38] Haiguang Wang, Daqi Liu, Hongwei Xie, Haisong Liu, Enhui Ma, Kaicheng Yu, Limin Wang, and Bing Wang. Mila: Multi-view snsensive-iielltty long-term video generation world model for autonomous driving. arXiv preprint (.X.:25503//8777 2025.
[39] Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drive- dreamer: Towards real- world-drvve world models for autonomous driving. In European Confer- ence on Computer Vision, pages 55-72. Springer, 2024.
[40] Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14749 14759, 2024.
[41] Pengcheng Wei, Li Yan, Hong Xie, Dashi Qiu, Changcheng Qiu, Hao Wu, Yinghao Zhao, Xiao Hu, and Ming Huang. Lidenerf: Neural radiance field reconstruction with depth prior provided by lidar point cloud. ISPRS Journal of Photogrammetry and Remote Sensing. 208:296-307, 2024.
[42] Yuqing Wen. Yucheng Zhao, Yingfei Liu, Fan Jia, Yanhui Wang, Chong Luo, Chi Zhang, Tiancai Wang, Xiaoyan Sun, and Xiangyu Zhang. Panacea: Panoramic and controllable video generation for autonomous driving. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6902-6912. 12, 2024.
[43] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang. Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuming of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7623-7633, 2023.
[44] Zhanqian Wu, Kaichong Liang, Dengeheng Liu, and Zhiguo Zhao. Driver lane change intention recognition based on attention enhanced residual-mbi-lsiee network. IEEE Access, 10:58050- 58061, 2022.
[45] Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-based neural radiance fields. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5438-5448 5448, 2022.
[46] Kairui Yang, Enhui Ma, Jibin Peng. Qing Guo, Di Lin, and Kaicheng Yu. Bevcontrol: Accurately controlling street-view elements with multi-perspectivv consistency via bev sketch layout. arXiv preprint arXiv:2308. 01661, 2023.
12

---

[47] Zetong Yang, Li Chen, Yanan Sun, and Hongyang Li. Visual point cloud forecasting enables scalable autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14673-14684, 2024.
[48] Lunjun Zhang, Yuwen Xiong, Ze Yang, Sergio Casas, Rui Hu, and Raquel Urtasun. Copilot4d: Learning unsupervised world models for autonomous driving via discrete diffusion. arXiv preprint arXiv:2311.01017 2023.
[49] Yumeng Zhang, Shi Gong, Kaixin Xiong, Xiaoqing Ye, Xiao Tan, Fan Wang, Jizhou Huang, Hua Wu, and Haifeng Wang. Bevworld: A multimodal world model for autonomous driving via unified bev latent space. arXiv preprint arXiv:2407.05679 2024.
[50] Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Xinze Chen, Guan Huang, Xiaoyi Bao, and Xingang Wang. Drivedreamer-2: LIm-enhanced world models for diverse driving video gen- eration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 10412-10420, 2025.
[51] Xin Zhou, Dingkang Liang, Sifan Tu, Xiwu Chen, Yikang Ding, Dingyuan Zhang, Feiyang Tan, Hengshuang Zhao, and Xiang Bai. Hermes: A unified self-driving world model for simultaneous 3d scene understanding and generation. arXiv preprint urXiv:2501.1472) 2025.
[52] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings the pages 4490 4499, 2018.
the IEEE conference on computer vision and pattern recognition,
[53] Vlas Zyrianov, Henry Che, Zhijian Liu, and Shenlong Wang. Lidardm: Generative lidar simulation in a generated world. arXiv preprint UrXiv:2404.02903 2024.
[54] Vlas Zyrianov, Xiyue Zhu, and Shenlong Wang. Learning to generate realistic lidar point clouds. In European Conference on Computer Vision, pages 17-35. Springer, 2022.
13

---

A Technical Appendices and Supplementary Material
(a) DataCrafter Setup
To enable structured semantic supervision during training, the DataCrafter module was developed based on the Qwen2-VL 7B vision-language model. To enhance its understanding of driving scenes and its ability to assess sample quality, the base model was fine-tuned using Low-Rank Adaptation (LoRA) on two domain-specific datasets: OmniDrive and CoDALM. During inference, the fine-tuned model evaluated multi-view clips along three predefined dimensions: visual and aesthetics- and semantic deseripriveness. Only samples that met a predefined quality threshold were retained for training, ensuring the exclusion of visually poor or semantically inconsistent sequences. The selected high-quality images were then forwarded to the captioning module. The captioning procedure detailed in the following table.
clarity
and
You are provided with set of descriptions of images captured by vehicle's multiple cameras. Please carefully analyze these descriptions and answer the following questions. Your answer must strictly adhere to the format provided below, including all start and end tags for each field. All responses must be in English.
Time: Select one: "Daytime", "Night", "Indoor", "No visible sign"
Weather: Select one from "Sunny". "Cloudy", "Overcast", "Rain", "Snow" or "Night with no visible sign"
Road Type: Select one from "Highway", "Urban Road", "Rural Road", "Tunnel", "Bridge" or "No visible sign" Road Surface: Select one from "Asphalt" ". "Concrete", "Gravel", "resin (Indoor)" Lane: Select one from "No visible sign", "Single Lane", "Dual Lane", "Multi-Lane", "Other(should explain in Details)" and describe specifically what the lanes are.
Environment Type:
Select
one from the following op- "Tampp.."unnell" "Tunnel", "Parking Lot","Urber "Urban Road", "Rural
tions: "highw"""" "Roundabbou""" "ooonddaouu","IIttrseeeee "Intersection", ", "Rammp". Road", "Bridge" "Other(should explain in Details)"
Surroundings Details: Provide a general description of the environment (e.g., busy street, quiet neighborhood).
Traffic: Details: Describe the vehicles present in the scene. Include their spatial relationship to the ego vehicle-foe example. Categorized by direction: Front, Left, Right, Behind. Each line follows <position>> <object> <action>.
the format:
(b) Model Setup
The video generation pipeline is constructed upon DiT-based architecture. The backbone is initialized with pretrained weights from MagicDrive ( while the variational autoencoder (VAE) is initialized using the publicly available encoder from Cog Video-X-L [14]. In contrast, the LiDAR generation branch trained entirely from scratch. Both its diffusion backbone and 3D autoencoder are designed and optimized specifically for sparse geometry modeling, enabling independent learning of modality-specifii representations without reliance on pretrained vision priors.
(c) Training Strategy & Training Setup
three-stage curriculum is adopted. (1) In the first stage, image-level generation trained at 512x768 resolution to warm-start spatial representation learning. (2) The second stage focuses on video synthesis, where a two-phase training protocol is employed. Multi-resolution pretraining is first conducted by gradually increasing input resolution from 144p (144x256) to 900p (900x1600), paired with clip lengths ranging from 128 frames at lower resolutions to 6 frames at higher ones. This is followed by adapter-based fine-tuning at a fixed resolution of 360p (360x640) and 16 frames, where lightweight spatiotemporal adapters are inserted into DiT blocks. (3) The third stage performs joint training of video and LiDAR generation with shared conditioning signals, enabling mrosl temporal alignment. For inference, video frames are generated at 900p with the first frame observed as input, consistent with prior work [8].
Training is performed with PyTorch using NVIDIA H20 GPUs and mixed-precision acceleration. Stages 1, 2, and 3 are trained for 300, 800, and 200 epochs respectively. The optimizer is AdamW
14

---

with weight decay of 0.01. A cosine annealing learning rate schedule is adopted with linear warm-up over the first 10% of steps. Learning rates are set to 10 4 for the image and video stages, and 1 x 10-5 for the joint stage. A global batch size of 1024 is used, distributed evenly across GPUs.
(d) Evaluation Metrics
To comprehensively evaluate the quality of multimodal generation, adopt suite of metrics tailored each modality. For video synthesis, we report Fréchet Inception Distance (FID) and Fréchet Video Distance (FVD), which assess spatial and spatiotemporal realism, respectively, in accordance with prior works [33 37 For LiDAR sequence generation, we employ Chamfer Distance 1s, 2s, and 3s horizons, These metrics quantify both geometric fidelity and distributional alignment.
For evaluating downstream 3D perception utility, we adopt mean Average Precision (mAP) and NuScenes Detection Score (NDS) on generated sequences using BEVFormer & BEVFusion [22 detector, following the protocol in [[6]. These metries provide practical proxy for assessing the semantic controllability and structural consistency of generated data.
(e) Supplementary visualization and quantitative analysis results
Daytime
Night
CAM FRONT SIFT
CON COMPRONT
CAL FRINT SCEIT
CAM LETT
CAM DACK
CAML BACK SUCET
Figure 9: Controllable generation across time-of-day. By altering scene-level conditions, our method produces consistent multi-view videos aligned with the same underlying map and object layout, while adapting appearance to represent daytime and nighttime settings.
Ground Truth
Ours
Magiedrive
Panacca
Figure 10: Qualitative comparison of video generation quality. Our method (second column) preserves accurate layout, object shapes, and background integrity. MagicDrive (third column) shows vehicle distortion and broken structures. Panacea (fourth column) often suffers from hallucinated textures and geometric misalignment.
15

---

Original Path
Right Shift 4m
Trajectory-condiiionel novel view synthesis. Given ground-truth trajectory (middle), we modify the layout (top) by shifting the ego path 4 meters right (bottom). Our model generates plausible and consistent scenes across all views under these layout changes.
Figure 11:
CAN PALN
CAN
CAMU
CAM SICK JuIIZ
Figure 12: Joint generation of LiDAR and multi-view video. Our method generates spatially aligned LiDAR and camera views conditioned on shared straight street layout.
TRENT
TRON7
CAM TRONT ROOIT
RUI
Figure 13: Joint generation of LiDAR and multi-view video. Our method generates spatially
aligned LiDAR and camera views conditioned on Busy junction layout.
16

---

Generated s
CAM
CAMLBaCK HIEET
Generated
CAR
CAM IN
CAM FRONT RHEIT
BUGHE
Generated 38
FAMEBACE
Figure 14: Long-term multi-view video generation over 3 seconds in an urhan driving scene, condi- tioned on the straight street layout.
Generated Is
CAM
FISH
Generated 28
CASURACK LIFT
CO ENIHT
Generated 3s
FAM PIONT
CAM TRONT
CAM BONE MHRLTE
DU BACK LFF
CA
Figure 15: Long-term multi-view video generation over 3 seconds in an urban driving scene, condi- tioned on the busy junction layout.
17

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA]-
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1 (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked While "[Yes] " is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providded a proper justification is given (e.g., "error bars expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
the checklist one of the factors in their evaluation.
reported because it would be too computationally
"[No] " or "[NA] is not grounds for rejection. While the questions are phrased acknowledge that the true answer often more nuanced, so please just use your best, judgment and write justification elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the main contributions of the paper, including the proposed algorithm, its theoretical properties, and empirical results. These claims are consistently supported by the theoretical analysis and experiments presented in the main body of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
18

---

Answer: [Yes]
Justification: Despite Genesis's ' strong performance in joint video and LiDAR generation, several limitations remain. Training is computationally intensive, suggesting future work on efficient architectures or model compression. Generalization beyond nuScenes, such to embodied agents robotics, requires further adaptation. Real-time and interactive generation also remains an open challenge.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper does contain formal theoretical results proofs. The focus is on the design, implementation, and empirical evaluation of the proposed method.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
19

---

Answer: [Yes]
Justification: The paper fully discloses all information necessary reproduce the main experimental results. Detailed training configurations, including hyperparameters, model architecture, optimization settings, and evaluation protocols, are provided in the appendix to ensure reprodueibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
The datasets used in our experiments are publicly available, while the code is not yet released. Detailed experimental procedures provided to ensure the verifiability of our findings.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
20

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper provides comprehensive details on the experimental setup, including dataset splits, hyperparameter settings and optimizer type.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We report the generated metrics for the joint generation task and provide visualizations of the generated LiDAR point clouds and video images demonstrate the effectiveness of our method. Nevertheless, the current experiments lack error bars and formal statistical significance analysis.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the eтror bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma emror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
21

---

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the number of GPUs used during training, each equipped with 96GB of memory.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttpp///aaarrss..cc//blc//ccaaa
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and confirm our research fully complies with its principles. No ethical issues arise.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
This work primarily focuses on algorithm/nodel improvements and technical methods. There are currently no direct or significant societal impacts associated with the research; therefore, no discussion positive or negative societal impacts is included.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended
(e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinfornation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
22

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
The paper does not involve the release of models or datasets that pose a high risk of misuse. Therefore, no specific safeguards are necessary.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use publicly available datasets and pretrained models, all of which are properly cited in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
23

---

Answer: [No]
Justification: The assets introduced in the paper (such as the self-developed training framework and private dataset) have not yet provided public documentation. The main reasons include that the models involve commercial privacy protection.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve research with human subjects or crowdsourcing, and therefore IRB equivalent ethical approval is required.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method of this research does not involve any important, original, or non-standard usage of large language models (LLMs). Any use of LLMs was limited to writing assistance and does not impact the scientific contributions of the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
25


Reviews:
[{"review_id": "NpIBoQlzw1", "reviewer": "Reviewer_FmRN", "summary": "The paper presents Genesis, a novel framework for joint generation of multi-view driving videos and LiDAR sequences with spatio-temporal and cross-modal consistency. The authors address a significant challenge in autonomous driving research by integrating structured semantic supervision and cross-modal alignment into a unified generation pipeline. The results demonstrate state-of-the-art performance in both video and LiDAR generation tasks, highlighting the framework's effectiveness in producing high-fidelity, semantically rich, and temporally coherent driving scenes. The work is well-executed, technically sound, and presents a substantial contribution to the field.", "questions": "1. In the FID_mult indicator in Table 1, this method is not as effective as UniScene. What is the reason? What are the next improvement measures?\n2. As shown in Table 6, the accuracy of Lidar detection is lower than that of Camera detection when using the synthetic data of this method for incremental training. What is the reason? What are the next improvement measures?\n3. For downstream utilization (incremental synthetic data for training), the accuracy is still declining. What are the bad cases we have seen so far? What problems need to be solved in using synthetic data?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The text and pictures are clear and direct, and the algorithm framework is simple and effective;\n2. A unified generation framework is designed to jointly synthesize multi-view video and LiDAR point cloud data with fine-grained semantic consistency;\n3. The system design is complete, integrating video generation, Lidar generation, and semantic controllable data synthesis into a complete production line, effectively leveraging the strengths of each module;\n4. According to the task, some detailed building block improvements are designed, such as 3D-aware latent encoding and conditioned on scene-level priors, which are effective and easy to implement;\n\nWeaknesses:\n1. There is a lack of refined processing for lidar generation, and the detailed design is more focused on the video generation branch;\n2. The overall architecture innovation is average, mainly integrating some industry-verified effective building blocks, finely constructing training data, and making end-to-end training work;\n3. The evaluation is mainly based on upstream generation quality evaluation, and there is a lack of detailed discussion on the incremental training evaluation of generated data for downstream detection or planning tasks;", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the detailed and constructive feedback. We address each concern below and will incorporate the suggested clarifications and improvements in the final version.\n### **Q1: Detailed design and refining process of LiDAR point cloud generation** \n\nOur design philosophy and refinement efforts focus on two components: the Point Cloud AutoEncoder and the LiDAR Generation Model.\n\n1. In the Point Cloud AutoEncoder, directly applying the NeRF algorithm to LiDAR rendering results in multiple noisy points. This occurs because the α-blending learns the average distance across multiple occupied grids along the same ray. To address this, we design a post-processing module that utilizes the occupancy feature to suppress these noisy points.\n\n2. For the LiDAR Diffusion model, previous methods rely heavily on the reference frame to predict the future point cloud sequence, which is comparatively a simple task.  In contrast, we focus on generating controllable point cloud sequences without reference frame input. For the foreground point cloud generation, we achieve precise control through BEV layout foreground boxes and category-aware box embeddings. For the background point cloud, we leverage road topology and scene descriptions, along with BEV latent representations extracted from images.\n\nWe believe our LiDAR generation framework effectively simulates realistic point cloud distributions and the metrics on the nuScenes dataset reflect the effectiveness of our design.\n\n### **Q2: Explanation of model innovation**\n\nWe emphasize that our model is not a simple integration of existing design paradigms. Genesis is purposefully crafted to address the core challenge of **multi-modal data generation for autonomous driving scenarios**. We build our technical framework from the first principles: since images and point clouds are fundamentally different sensor modalities capturing the same physical scene, our approach begins by extracting core scene elements, including scene layout, foreground object bounding boxes, and semantic descriptions.\n\nThese elements are then processed in a sensor-specific manner:\n\n1. For the camera branch, the 3D conditions (layout and bounding boxes) are projected onto the image plane, yielding 2D layouts and 2.5D boxes (i.e., 2D boxes with depth) as conditional inputs.\n\n2. For the LiDAR branch, the original 3D layout and bounding boxes are used directly as conditions.\n\nThese processed representations serve as conditional inputs to a decoupled dual-DiT architecture, where we further introduce cross-modal consistency via image feature guidance.\n\nThe key innovation of our framework lies in the shared condition guidance mechanism, which is the **first solution** capable of generating both image and point cloud video sequences without relying on the first-frame LiDAR input or explicit intermediate representations like occupancy grids.\n\nIn addition, we implemented several detail-level enhancements:\n\n1. During image generation, we project the layout into the image plane to reinforce viewpoint-consistency constraints;\n\n2. For non-rigid object generation, we introduce keypoint-based control to improve geometric fidelity;\n\n3. For the point cloud AutoEncoder, we design a post-processing module to filter out noisy points.\n\nThese designs reflect our commitment to build a principled, effective, and robust multi-modal generation system tailored for autonomous driving.\n\n### **Q3: Detailed discussion on downstream detection task**\n\nWe apologize for the confusion between Table 6 in the main text and Table 1 in the supplementary materials. Table 6 in the main text presents the **domain gap metric** for BEV segmentation tasks, calculated by replacing ground truth images and LiDAR in the validation dataset. Preliminary results show our method achieves the best mIoU (38.01%) and mAP (27.90%) in BEV segmentation.\n\nWe display the **downstream detection task** in the table 1 in the supplementary materials. Our approach yields consistent improvements across all settings, increasing the mAP from 66.87% to 67.78% and NDS from 69.65% to 71.13%. Notably, joint generation of both camera and LiDAR modalities achieves the highest gains (+0.91% mAP / +1.48% NDS), demonstrating the complementary benefits of multi-modal  generation. These results validate the utility of high-quality synthetic data in enhancing downstream perception tasks, especially in data-scarce or long-tail scenarios.\n\n### **Q4: Reasons for our method being less effective than UniScene in the $FID_{multi}$ metric and future work**\n\nThanks for this great suggestion. Regarding the slightly inferior performance of our method compared to UniScene on $FID_{multi}$ metric, we attribute the difference to two key distinctions in the conditional image selection strategies between our approach and UniScene:\n\n1. UniScene utilizes all camera views from the 0-th frame, while we did not use.\n\n2. UniScene incorporates 3D occupancy as auxiliary supervision, where the semantic-enriched depth information within volumetric grids delivers fine-grained geometric cues.\n\nIn contrast, our current version does not incorporate such supervisions, primarily for the following reasons:\n\n1. Maintain the lightweight nature of our method and avoid reliance on precomputed semantic/depth modules;\n\n2. Focus on enhancing the model’s intrinsic spatio-temporal modeling capabilities, rather than leveraging external structural hints.\n\nThus, from a technical perspective, our method achieves comparable performance under significantly fewer conditional inputs, which demonstrates its strong modeling capacity and potential for scalability.\n\n**Future Work and Improvements:**\n\nWe plan to further enhance the boundary capabilities of multimodal generation while maintaining model generality and inference efficiency, with the following specific steps:\n\n1. Few-shot sample generation capability. A key role of generative models is to provide downstream modules with abundant high-quality training data. Expanding the model's capability boundaries to generate samples rare in the training set is worth exploring—for instance, generating point clouds and images of scenarios like \"a dog crossing the road\" or \"a fallen traffic cone\".\n\n2. Inference efficiency optimization. Model inference speed is a major bottleneck, mainly due to VAE resolution compression and denoising steps. While VAE compression rates for images are extensively studied, developing efficient compression methods for LiDAR latent spaces emerges as a critical research frontier. Additionally, exploring methods to reduce denoising steps is also worthwhile.\n\n### **Q5: Reasons for lower LiDAR detection accuracy compared to camera in incremental training with synthetic data**\n\nThank you for highlighting the potential ambiguity in our previous wording. Table 6 does not present incremental training results; instead, it evaluates generation quality by replacing real data with generated samples to measure their impact on downstream performance.\n\nRegarding why \"LiDAR detection accuracy is lower than that of cameras\", we attribute it to differences in LiDAR sparsity and data quantity. On the nuScenes dataset, LiDAR point clouds are relatively sparse (with lower beam density) and limited in volume (only 700 training clips), making it harder for the model to accurately learn the distribution of foreground point clouds. This results in a larger domain gap in LiDAR performance compared to image generation.\n\nTo verify this, we conducted a comparison using our private dataset, collected with the Hesai AT128 LiDAR (with denser beams) and trained on 100,000 clips. The domain gap metrics are as follows:\n\n|Modality|Precision(%)|Recall(%)| F1 Score(%)|\n|-|-|-|-|\n|Camera(Real)| 46.1| 92.6| 61.5|\n|Camera(Generated) | 45.3 |89.1|60.1|\n|Domain Gap|-0.8| -3.5| -1.4|\n|LiDAR(Real)|44.3|89.1| 59.2|\n|LiDAR(Generated) |44.5| 86.8| 58.9|\n|Domain Gap| +0.2| -2.3| -0.3|\n\nAs shown, in a higher-density LiDAR setup with more data, the domain gap of LiDAR becomes smaller than that of the image modality. This further supports our analysis that data quality and quantity, especially LiDAR beam density, play a crucial role in determining the effectiveness of generative modeling across modalities.\n\n### **Q6: Bad cases of synthetic data in downstream training and problems to be solved**\n\nThank you for this valuable suggestion. In relation to your question about \"the decreasing accuracy when using additional synthetic data for training\", we would like to clarify that the results shown in the supplementary table actually demonstrate an improvement. When utilizing both image and LiDAR modalities together, we observed a 1.48% increase in NDS compared to training without synthetic data, indicating a positive impact on downstream performance.\n\nAs for your concern about bad cases, we have identified two primary issues during our internal evaluations:\n\n1. Blurry generation of distant small objects. At distances beyond 150 meters, generated objects often become too indistinct to confidently determine whether they represent vehicles or other entities.\n\n2. Occasional hallucinations. Despite employing classifier-free guidance to regulate foreground object generation, we observed rare instances where the model hallucinates vehicles in regions not conditioned by any bounding boxes.\n\nTo address these challenges, we believe the following steps are essential:\n\n1. Strengthening quality control for synthetic data: We are introducing a combination of VLM-based automatic filtering and manual verification to ensure that only high-quality samples are used for downstream tasks.\n2. Ensuring the quality of training data for generative models: By refining the curation pipeline and enhancing the fidelity of the training dataset, we aim to minimize hallucination and improve the overall reliability of the generation model."}, {"review_id": "bRYWU5PYIF", "reviewer": "Reviewer_3gho", "summary": "The paper introduces Genesis, a framework for the joint generation of multi-view driving videos and LiDAR point clouds. It features a two-branch architecture where a video diffusion model (DiT-based with 3D-VAE encoding) and a LiDAR generator (with BEV-aware autoencoders and NeRF rendering) are coupled through a shared latent space. To improve semantic control and alignment, the authors propose DataCrafter, a captioning pipeline leveraging vision-language models to generate scene-level and instance-level descriptions. The model is evaluated on the nuScenes dataset and shows competitive results on standard metrics (FVD, FID, Chamfer).", "questions": "1.\tPlease clarify in what fundamental ways Genesis differs from UniScene and MagicDrive3D, which also support layout-conditioned multimodal generation. Are the performance gains mostly due to better tuning, or are there architectural contributions that make a structural difference?\n\n2.\tHas Genesis been tested (even qualitatively) on datasets beyond nuScenes, such as Waymo Open Dataset or KITTI? If not, what are the expected challenges for generalization?\n\n3.\tThe ablation study shows modest improvements from DataCrafter, yet the module adds substantial complexity. Can you provide a controlled experiment comparing against simple captioning (e.g., T5 without VLM-based filtering)? This would help justify the added engineering overhead.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe paper offers a framework that couples video and LiDAR generation within a shared latent space.\n\n2.\tThe proposed DataCrafter module uses structured scene descriptions for fine-grained control, which improves interpretability and controllability.\n\nWeaknesses:\n\n1.\tThe core novelty claimed—joint multimodal generation—has already been explored in prior works such as UniScene, BEVFusion, and MagicDrive3D, which also discuss layout-conditioned or multi-modal generation pipelines. Genesis does not go far beyond these methods in either formulation or outcomes. The use of a shared latent space and VLM-based semantic conditioning feels like a composition of existing techniques rather than a novel innovation.\n\n2.\tThe model relies heavily on existing components (DiT, NeRF, ControlNet, BEV encoders, VLM captioners) and integrates them in a fairly straightforward way. While integration is non-trivial, there is little architectural or algorithmic novelty beyond known design patterns.\n\n3.\tThe improvement margins are incremental compared to prior works like MagicDrive-V2 or UniScene. The broader impact is limited by the paper's tight focus on nuScenes only, with no discussion of transferability to other autonomous driving datasets or domains.", "was_revised": true, "final_justification": "After reviewing the authors’ rebuttal, I recognize that several of my earlier concerns about novelty and contribution have been addressed. Although I still view the architectural innovations as incremental, the additional explanations, comparisons with related work, and supplementary experiments have clarified the value of the proposed framework. I now consider the reasons to accept to slightly outweigh the reasons to reject, and therefore I am updating my rating to borderline accept.", "rebuttal": "We sincerely thank you for the thoughtful review and constructive suggestions. Below we provide detailed responses to each concern raised. The additional experiments will be included in the revision.\n\n### **Q1: Explanation of model innovation**\n\nWe would like to claim that our model is not a simple integration of existing design paradigms. Genesis is purposefully crafted to address the core challenge of **multi-modal data generation for autonomous driving scenarios**. We build our technical framework from the first principles: since images and point clouds are fundamentally different sensor modalities capturing the same physical scene, our approach begins by extracting core scene elements, including scene layout, foreground object bounding boxes, and semantic descriptions.\n\nThese elements are then processed in a sensor-specific manner:\n\n1. For the camera branch, the 3D conditions (layout and bounding boxes) are projected onto the image plane, yielding 2D layouts and 2.5D boxes (i.e., 2D boxes with depth) as conditional inputs.\n2. For the LiDAR branch, the original 3D layout and bounding boxes are used directly as conditions.\n\nThese processed representations serve as conditional inputs to a decoupled dual-DiT architecture, where we further introduce cross-modal consistency via image feature guidance.\n\nThe key innovation of our framework lies in the shared condition guidance mechanism, which is the **first solution** capable of generating both image and point cloud video sequences without relying on the first-frame LiDAR input or explicit intermediate representations like occupancy grids.\n\nIn addition, we implemented several detail-level enhancements:\n\n1. During image generation, we project the layout into the image plane to reinforce viewpoint-consistency constraints;\n\n2. For non-rigid object generation, we introduce keypoint-based control to improve geometric fidelity;\n\n3. For the point cloud AutoEncoder, we propose a dedicated post-processing module designed to effectively filter out noisy points and enhance output quality.\n\nThese designs reflect our commitment to build a principled, effective, and robust multi-modal generation system tailored for autonomous driving.\n\n### **Q2：Explanation of the differences between Genesis, UniScene and MagicDrive3D, and the real reason for performance gains**\n\n**(1) The fundamental differences between Genesis and UniScene**: \n\nThe fundamental distinction between Genesis and UniScene lies in the fact that Genesis **does not require any additional structural representations**. In contrast, UniScene relies on ground-truth OCC labels to supervise the training of a dedicated OCC generation network. However, obtaining such OCC annotations is costly and labor-intensive, limiting scalability in real-world scenarios. In comparison, Genesis achieves high-quality generation without relying on any external structural supervision, making it more practical and generalizable.\n\n **(2) The fundamental differences between Genesis and MagicDrive3D**: \n\n1. The essential difference between Genesis and MagicDrive3D is that Genesis is a fully multimodal generation framework, while MagicDrive3D **does not support LiDAR generation**. MagicDrive3D employs a monocular depth estimation network to produce pseudo-LiDAR point clouds from generated images. However, significant distributional discrepancies exist between these synthetic point clouds and authentic LiDAR data distributions. \n\n2. For image video generation, Genesis uses projected 2D layouts as condition rather that 3D layouts used in MagicDrive3D. Compared to 3D Layouts, utilizing 2D Layouts ensures better multi-view consistency and facilitates view generalization, as evidenced by the successful extension from 6 views to 11 views.\n\n3. For image video generation, Genesis proposes a pedestrian pose optimization strategy for enhanced pedestrian generation, which solves the challenging in non-rigid object generation in video synthesis to some extend.\n\n4. Finally, compared with MagicDrive3D, we introduce DataCrafter to provide structured semantic supervision, yielding outputs that are realistic, interpretable, and controllable.\n\n**(3) The real reason for performance gains**:\n\nWe maintain that the primary reason behind our performance gains is the shared latent space design, which enables tightly coupled multimodal modeling and significantly improves generation quality and consistency.\n\nFor the camera modality, we introduce targeted enhancements for non-rigid objects such as pedestrians and adopt layout control via 3D-to-2D projection to ensure higher structural fidelity.\n\nFor the LiDAR modality, we pioneer the BEV-Layout control mechanism, which allows for precise control of point cloud structures while maintaining cross-modal consistency.\n\n### **Q3：Genesis testing experiments on private dataset**\n\nThank you for this important reminder. In addition to our experiments on the nuScenes dataset, we also conducted extensive training and evaluation on a proprietary internal dataset. Below, we briefly introduce its characteristics: night-time scenes account for 32.14%, urban roads make up 69.43%, multi-vehicle complex road conditions cover 62.63%.\n\nWe trained the Genesis model on 100,000 clips (each 30 seconds long) and evaluated its performance on 3D object detection using both Camera and LiDAR modalities. The evaluation results are shown below:\n\n|Modality| Precision(%) | Recall(%) | F1 Score(%)|\n|-|-|-|--|\n|Camera(Real)| 46.1| 92.6| 61.5|\n|Camera(Generated)| 45.3 |89.1|60.1|\n|Domain Gap| -0.8| -3.5| -1.4|\n|LiDAR(Real) |44.3| 89.1| 59.2|\n|LiDAR(Generated) |44.5| 86.8| 58.9|\n|Domain Gap| +0.2| -2.3| -0.3|\n\nWe adopt a domain gap evaluation protocol, which quantifies the performance difference between using real and generated data. The goal is to evaluate the quality of the generated samples, guiding us to improve realism and utility by minimizing this gap.\nOur procedure is as follows:\n\n1. Replace the real images or point clouds in the internal dataset with the generated ones.\n\n2. Evaluate downstream 3D detection performance on the mixed dataset.\n\n3. Compare the results against the performance on the original real dataset.\n\nAs shown in table above, the F1 Score drops only 1.4% for the camera modality, and just 0.3% for the LiDAR modality, even without using condition frames. This demonstrates that our generated point clouds already exhibit high geometric fidelity, making them suitable for direct replacement of real LiDAR data in training and evaluation.\n\nFor scene-specific editing and generalization, we also validate the effectiveness using this internal dataset. For example, in the case of low mAP for cones at night, we inject cones in diverse distances into the BEV layout under a nighttime condition constraint. This finally leads to the mAP improvement from **16.47% to 17.26%**.\n\nThe detailed comparison is shown below:\n\n|Model|Detection category|Scenario| mAP(%) |\n|-|-|-|-|\n|Baseline| vehicles |night| 69.71 |\n|Baseline + Synthetic Data| vehicles |night| 71.36 (+1.65) |\n|Baseline| cone |night|16.47|\n|Baseline + Synthetic Data|cone|night| 17.26 (+0.79) |\n\nThese results demonstrate that our synthetic data generation preserves high fidelity and simultaneously enhances downstream task performance.\n\n### **Q4: Motivation and rationale for introducing the DataCrafter module**\n\nWe appreciate your attention to the DataCrafter module, and would like to clarify its motivation and design rationale as follows:\n\n1. Aligned with Mainstream Paradigms in Video Generation\n\nThe current dominant paradigm for text-guided video generation follows the VLM → Caption → T5 → Embedding pipeline, as adopted by works such as VideoCrafter. In this context, the VLM component is not an added complexity, but rather a structural necessity for semantic control. Our proposed DataCrafter module does not introduce an additional VLM, but instead offers a high-quality refinement of this paradigm by applying caption filtering, re-ranking, and quality control on the VLM outputs, thereby generating more accurate and structurally sound captions.\n\n2. Effectiveness Validated via Ablation Studies\n\nAblation experiments have shown that DataCrafter significantly outperforms traditional captioning methods (e.g., BLIP) in terms of key metrics such as FID. These results demonstrate that DataCrafter enhances semantic quality and improves the final video generation performance. The comparative results will be explicitly highlighted in the final version.\n\n3. Controllable Computational Cost\n\nMost VLM-caption pipelines are implemented with offline preprocessing: caption generation is performed during data preparation for training, and cached during inference. Only rare scenarios require real-time processing. Moreover, the captioning module accounts for a negligible fraction of inference time. In our deployment, DataCrafter (including filtering and augmentation) constitutes **only 2.44%** of total inference time, indicating high efficiency.\n\n4. Supported by Real-World Applications in Autonomous Driving\n\nSeveral prior works in autonomous driving support the value of high-quality captions. For instance, DriveGenVLM uses VLM-generated captions to assess semantic alignment, and DriveDreamer-2 leverages LLMs to improve video-text consistency. These studies collectively demonstrate that high-quality language conditions are not only beneficial but essential for enhancing cross-modal alignment and structural accuracy. DataCrafter is designed as a refined and enhanced realization of this paradigm.\n\nIn summary, DataCrafter is a lightweight and effective enhancement to the VLM → Caption → T5 pipeline. It introduces no additional model components, focuses purely on improving the quality of caption generation and selection, and has a well-controlled computational footprint. Backed by practical success in autonomous driving scenarios, we believe DataCrafter offers a justified and valuable contribution to video generation tasks."}, {"review_id": "jKjdueC4tI", "reviewer": "Reviewer_EYgF", "summary": "This paper introduces a multimodal driving scene generation framework named Genesis. It jointly generates multi-view driving videos and LiDAR with spatio-temporal and cross-modal consistency. It also introduced a VLM-based captioning module named DataCrafter to guide the generation with structured semantics. Extensive experiments on the nuScenes benchmark show that Genesis achieves state-of-the-art generation and alignment performance across video and LiDAR metrics", "questions": "No", "limitations": "The limitations have already been discussed.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1.\tThis paper proposes a unified framework for jointly generating videos and LiDAR point clouds, which has the potential to benefit multimodal downstream modules.\n\n2.\tThe writing is clear and well-structured. Figure 1 effectively illustrates the comparison with previous pipelines.\n\n3.\tThe generation quality is good for both videos and LiDARs.\n\nWeaknesses\n\n1.\tWhile Genesis shows good generation quality, it remains unclear whether producing better replicas of training samples translates to meaningful improvements in downstream module performance. Since the ultimate goal of driving scene generation is to enhance downstream tasks, the paper would be stronger if it demonstrated how Genesis can generate novel and useful training samples.\n\n2.\tThe inference cost is not discussed. It would be helpful to include a comparison of efficiency with other generation pipelines to assess the practicality of the proposed joint generation framework.\n\n3.\tThe downstream evaluation is relatively limited. Expanding the evaluation to include additional tasks, such as lane detection, and testing on more benchmarks like Waymo, would further strengthen the paper.", "was_revised": true, "final_justification": "The rebuttal has addressed my concern. I maintain my recommendation to accept the paper.", "rebuttal": "We sincerely thank you for the thoughtful review and constructive suggestions. Below we provide detailed responses to each concern raised. The additional experiments will be included in the revision.\n### **Q1:  Downstream performance and capability of expanding the training samples**\n\nThank you for this valuable suggestion. We have provided experimental results for 3D object detection task in the supplementary materials. We evaluate the effectiveness of our generated data on the BEVFusion framework for 3D object detection. Our approach yields consistent improvements across all settings, increasing the mAP from **66.87% to 67.78%** and NDS from **69.65% to 71.13%**. Notably, joint generation of both camera and LiDAR modalities achieves the highest gains **(+0.91% mAP / +1.48% NDS)**, demonstrating the complementary benefits of multi-modal generation. These results validate the utility of high-quality synthetic data in enhancing downstream  perception tasks, especially in data-scarce or long-tail scenarios.\n\nWe primarily adopt the following two approaches for expanding training samples:\n\n1. **Style transfer** was implemented based on existing scene layouts to enhance the model's adaptability to diverse meteorological conditions (e.g., changing from sunny to rainy weather); \n\n2. **Editing and generalising for specific scene distributions.** This aspect is primarily validated on private dataset. For instance, if night-time cone detection metrics are low, we add various cones to the BEV layout and control the scene generation to be night-time; ultimately, the mAP metric improved from **16.47% to 17.26%** on the private dataset.\n\n### **Q2:  Comparison of efficiency with other generation pipelines**\n\nWe tested the inference cost on a single GPU NVIDIA H20 machine;\n\n1. For the camera modality, generating a sequence of 17 frames with 6 views, each at a resolution of 424*800, takes 66.73 seconds. In contrast, MagicDrive-V2 requires a total of 231.18 seconds to complete the inference of 17 frames under the same setting. The main reason for faster inference speed is that our denoising step is 8, compared to 30 used in MagicDrive-V2. \n\n2. For the LiDAR modality, generating 16 frames takes 57.72 seconds.  In contrast, UniScene first needs to generate semantic occupancy conditioned on BEV layouts; then, based on this, generate LiDAR point clouds. It takes a total of 63.2 seconds to generate 16 frames. \n\nWe could draw a conclusion that Genesis achieves competitive inference efficiency in multi-modal generation.\n\n ### **Q3:  Expanding the downstream evaluation**\n\nThank you for your comment. In addition to 3D object detection tasks, we conduct domain gap testing on BEV segmentation tasks (including lane lines and foreground objects) on nuScenes dataset. Preliminary results indicate that our method achieves the best mIoU (38.01%) and mAP (27.90%) on BEV segmentation, surpassing prior methods such as DiVE and Cogen, demonstrating strong semantic alignment and visual fidelity. \n\nAs for more experiments on  downstream evaluation, we conduct experiments on our private dataset. We trained Genesis on 100,000 clips (each clip lasting 30 seconds) and evaluate the downstream evaluation on 5000 clips. We focus on using generated data to solve specific issues. For example, we find that the mAP metric of  traffic cone detection during night-time is  low. Thus we add several traffic cones to the BEV layout and control the scene generation to be night-time. The quantity ratio of synthetic data and real data is 1:8.\n\n|Model|Detection category | Scenario| mAP(%) |\n|--|--|--|-|\n| Baseline| vehicles | night  | 69.71 |\n| Baseline + Synthetic Data| vehicles | night  | 71.36 (+1.65) |\n| Baseline| cone |night  |16.47 |\n| Baseline + Synthetic Data| cone |night | 17.26 (+0.79) |\n\nAs shown in the above table, with the mixture training of real data and generated synthetic data, traffic cone's mAP metric improves from **16.47% to 17.26%** on our private dataset."}]

Author rebuttal:

Meta-review: In their paper, the authors introduce Genesis, a unified framework designed to jointly generate multi-view driving videos and LiDAR sequences. The novelty of the method is maintaining consistency across different views, over time, and between the video and LiDAR data.

All the reviewers are positive about the contributions of the papers, including: (1) the novelty of the proposed Genesis framework; (2) the experiments are sufficient to support the claims about the performance of the proposed method; (3) the paper is written and presented clearly. After the rebuttal, most of the concerns of the reviewers were addressed, and all the reviewers are happy with the current stage of the paper.

In my opinion, the contributions and originality of the proposed methods are sufficient for acceptance at NeurIPS. Therefore, I recommend accepting it in its current form. I encourage the authors to address the reviewers’ suggestions and integrate their feedback into the camera-ready version of their paper.
Decision: Accept (poster)