Title: Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency
Venue: NeurIPS 2025
Accepted: yes
Keywords: autonomous driving; driving scenario generation
Abstract: We present Genesis, a unified world model for joint generation of multi-view driving videos and LiDAR sequences with spatio-temporal and cross-modal consistency. Genesis employs a two-stage architecture that integrates a DiT-based video diffusion model with 3D-VAE encoding, and a BEV-represented LiDAR generator with NeRF-based rendering and adaptive sampling. Both modalities are directly coupled through a shared condition input, enabling coherent evolution across visual and geometric domains. To guide the generation with structured semantics, we introduce DataCrafter, a captioning module built on vision-language models that provides scene-level and instance-level captions. Extensive experiments on the nuScenes benchmark demonstrate that Genesis achieves state-of-the-art performance across video and LiDAR metrics (FVD 16.95, FID 4.24, Chamfer 0.611), and benefits downstream tasks including segmentation and 3D detection, validating the semantic fidelity and practical utility of the synthetic data.


Peer reviews:
comments: SUMMARY:
This paper introduces a multimodal driving scene generation framework named Genesis. It jointly generates multi-view driving videos and LiDAR with spatio-temporal and cross-modal consistency. It also introduced a VLM-based captioning module named DataCrafter to guide the generation with structured semantics. Extensive experiments on the nuScenes benchmark show that Genesis achieves state-of-the-art generation and alignment performance across video and LiDAR metrics

STRENGTHS AND WEAKNESSES:
Strengths

1.	This paper proposes a unified framework for jointly generating videos and LiDAR point clouds, which has the potential to benefit multimodal downstream modules.

2.	The writing is clear and well-structured. Figure 1 effectively illustrates the comparison with previous pipelines.

3.	The generation quality is good for both videos and LiDARs.

Weaknesses

1.	While Genesis shows good generation quality, it remains unclear whether producing better replicas of training samples translates to meaningful improvements in downstream module performance. Since the ultimate goal of driving scene generation is to enhance downstream tasks, the paper would be stronger if it demonstrated how Genesis can generate novel and useful training samples.

2.	The inference cost is not discussed. It would be helpful to include a comparison of efficiency with other generation pipelines to assess the practicality of the proposed joint generation framework.

3.	The downstream evaluation is relatively limited. Expanding the evaluation to include additional tasks, such as lane detection, and testing on more benchmarks like Waymo, would further strengthen the paper.

QUESTIONS:
No

LIMITATIONS:
The limitations have already been discussed.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Genesis, a framework for the joint generation of multi-view driving videos and LiDAR point clouds. It features a two-branch architecture where a video diffusion model (DiT-based with 3D-VAE encoding) and a LiDAR generator (with BEV-aware autoencoders and NeRF rendering) are coupled through a shared latent space. To improve semantic control and alignment, the authors propose DataCrafter, a captioning pipeline leveraging vision-language models to generate scene-level and instance-level descriptions. The model is evaluated on the nuScenes dataset and shows competitive results on standard metrics (FVD, FID, Chamfer).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tThe paper offers a framework that couples video and LiDAR generation within a shared latent space.\n\n2.\tThe proposed DataCrafter module uses structured scene descriptions for fine-grained control, which improves interpretability and controllability.\n\nWeaknesses:\n\n1.\tThe core novelty claimed—joint multimodal generation—has already been explored in prior works such as UniScene, BEVFusion, and MagicDrive3D, which also discuss layout-conditioned or multi-modal generation pipelines. Genesis does not go far beyond these methods in either formulation or outcomes. The use of a shared latent space and VLM-based semantic conditioning feels like a composition of existing techniques rather than a novel innovation.\n\n2.\tThe model relies heavily on existing components (DiT, NeRF, ControlNet, BEV encoders, VLM captioners) and integrates them in a fairly straightforward way. While integration is non-trivial, there is little architectural or algorithmic novelty beyond known design patterns.\n\n3.\tThe improvement margins are incremental compared to prior works like MagicDrive-V2 or UniScene. The broader impact is limited by the paper's tight focus on nuScenes only, with no discussion of transferability to other autonomous driving datasets or domains.\n\nQUESTIONS:\n1.\tPlease clarify in what fundamental ways Genesis differs from UniScene and MagicDrive3D, which also support layout-conditioned multimodal generation. Are the performance gains mostly due to better tuning, or are there architectural contributions that make a structural difference?\n\n2.\tHas Genesis been tested (even qualitatively) on datasets beyond nuScenes, such as Waymo Open Dataset or KITTI? If not, what are the expected challenges for generalization?\n\n3.\tThe ablation study shows modest improvements from DataCrafter, yet the module adds substantial complexity. Can you provide a controlled experiment comparing against simple captioning (e.g., T5 without VLM-based filtering)? This would help justify the added engineering overhead.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents Genesis, a novel framework for joint generation of multi-view driving videos and LiDAR sequences with spatio-temporal and cross-modal consistency. The authors address a significant challenge in autonomous driving research by integrating structured semantic supervision and cross-modal alignment into a unified generation pipeline. The results demonstrate state-of-the-art performance in both video and LiDAR generation tasks, highlighting the framework's effectiveness in producing high-fidelity, semantically rich, and temporally coherent driving scenes. The work is well-executed, technically sound, and presents a substantial contribution to the field.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The text and pictures are clear and direct, and the algorithm framework is simple and effective;\n2. A unified generation framework is designed to jointly synthesize multi-view video and LiDAR point cloud data with fine-grained semantic consistency;\n3. The system design is complete, integrating video generation, Lidar generation, and semantic controllable data synthesis into a complete production line, effectively leveraging the strengths of each module;\n4. According to the task, some detailed building block improvements are designed, such as 3D-aware latent encoding and conditioned on scene-level priors, which are effective and easy to implement;\n\nWeaknesses:\n1. There is a lack of refined processing for lidar generation, and the detailed design is more focused on the video generation branch;\n2. The overall architecture innovation is average, mainly integrating some industry-verified effective building blocks, finely constructing training data, and making end-to-end training work;\n3. The evaluation is mainly based on upstream generation quality evaluation, and there is a lack of detailed discussion on the incremental training evaluation of generated data for downstream detection or planning tasks;\n\nQUESTIONS:\n1. In the FID_mult indicator in Table 1, this method is not as effective as UniScene. What is the reason? What are the next improvement measures?\n2. As shown in Table 6, the accuracy of Lidar detection is lower than that of Camera detection when using the synthetic data of this method for incremental training. What is the reason? What are the next improvement measures?\n3. For downstream utilization (incremental synthetic data for training), the accuracy is still declining. What are the bad cases we have seen so far? What problems need to be solved in using synthetic data?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-05"}