Title: X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability
Venue: NeurIPS 2025
Accepted: yes
Keywords: 3D Scene Generation

Autonomous Driving
Abstract: Diffusion models are advancing autonomous driving by enabling realistic data synthesis, predictive end-to-end planning, and closed-loop simulation, with a primary focus on temporally consistent generation. However, large-scale 3D scene generation requiring spatial coherence remains underexplored. In this paper, we present X-Scene, a novel framework for large-scale driving scene generation that achieves geometric intricacy, appearance fidelity, and flexible controllability. Specifically, X-Scene supports multi-granular control, including low-level layout conditioning driven by user input or text for detailed scene composition, and high-level semantic guidance informed by user intent and LLM-enriched prompts for efficient customization. To enhance geometric and visual fidelity, we introduce a unified pipeline that sequentially generates 3D semantic occupancy and corresponding multi-view images and videos, ensuring alignment and temporal consistency across modalities. We further extend local regions into large-scale scenes via consistency-aware outpainting, which extrapolates occupancy and images from previously generated areas to maintain spatial and visual coherence. The resulting scenes are lifted into high-quality 3DGS representations, supporting diverse applications such as simulation and scene exploration. Extensive experiments demonstrate that X-Scene substantially advances controllability and fidelity in large-scale scene generation, empowering data generation and simulation for autonomous driving.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes X-Scene, a novel pipeline for large-scale driving scene generation with high fidelity and flexible controllability. By dividing scene generation into 1) textual description enrichment, 2) textual scene-graph to layout generation, 3) joint occupancy and image generation, and 4) large-scale scene extrapolation and reconstruction, the proposed X-Scene can generate driving scenes with multiple modalities. Extensive experiments demonstrate the effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\n- Strengths:\n  - This is a solid work with superior engineering efforts, including LLM rephrasing, GNN for layout generation, occupancy generation, multi-view image generation, and occupancy & image outpainting.\n  - The experiments thoroughly evaluate the performance of each component of X-Scene.\n  - The authors provide the source code in the supplementary. I appreciate the authors for doing that.\n- Weakness:\n  - As a solid work with lots of effort, the explanation for each part is unfortunately quite unclear, including:\n    - Line 132: How to describe spatial attributes via texts? Left or right?\n    - Lines 141-144: The introduction of RAG is quite weird. Perhaps an ablation on using or not using RAG is necessary. Moreover, how to retrieve relevant samples for ICL given a specific input is unclear.\n    - Lines 155-157: The definition of \"layout\" is quite unclear throughout the whole paper. According to Lines 155-157, the \"layout\" consists of object bounding boxes and road lines, while in Figure 1 and Table 3, it seems that the \"layout\" is more similar to BEV maps with semantic conditions. \n    - Equation 2: I can see any conditions encoded in Sec. 3.1.\n    - Equation 3: seems like an unconditional outpainting procedure, suggesting no supervision is applied for the outpainted regions.\n    - Lines 216-230: Visual-Coherent Image Extrapolation seems to have no relationship with Geometry-Consistent Scene Outpainting? Otherwise, how to get the reference image at first?\n    - Figure 4 caption: The camera poses supported by the image generative model are pre-defined by the nuScenes dataset, and thus, there is no way to generate novel views, even with occupancy conditions.\n  - Novelty would be a controversial problem of this work, which mainly develops a pipeline for 3D street scene generation, while for each component, different existing works are utilized.\n  - For a pipeline work with complicated component design, it would be essential to analyze the effects of error accumulation after each component.\n- Overall, I think this is quite a solid work which combines recent developments of different areas of street scene generation with lots of effort. My concerns are mainly about the unclear paper writing, and I will consider increasing the score if the aforementioned questions are well-addressed after rebuttal.\n\nQUESTIONS:\nCheck the Weakness part for details.\n\nLIMITATIONS:\nCheck the Weakness part for details.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces X-Scene, an end-to-end pipeline for driving scene generation that supports various beneficial functions. The functions include converting text or user-drawn layouts into a scene graph and outpainting the scene chunk by chunk. The authors propose to use a triplane latent occupancy volume, multi-view dissusion model to generate the scene and pack the final output into a 3D Gaussian-splat scene for downstream usages. Extensive experiments demonstrate that X-Scene outperforms existing baselines in terms of generation quality, image size, and improvement in downstream perception tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghts\n---\n- The paper is generally well-written and easy to follow.\n- The research is timely and aims to improve the current generation in many aspects, including fidelity and controllability.\n- Experiments are carefully designed and complete.\n\nWeaknesses\n---\nAlthough I am not actively working in this field, I was unable to identify any critical weaknesses in the manuscript that would prevent me from giving a positive score. Below are some questions after reading through the paper.\n- While I don't think it's the main point of the research, as the authors mentioned, the current method doesn't support video generation, which is critical for real usage such as closed-loop evaluation.\n- Contributions may come from mostly an engineering unification rather than scientific novelty as key ideas including triplane, cross-modal diffusion, and latent extrapolation have already been widely used.\n- It would be beneficial if the authors could provide a more in-depth discussion on the effectiveness of using RAG in their framework.\n- How does outpainting behave after many hops? Does the artifact accumulate?\n- Is there any reason for the performance discrepancy between Tables 5 and 7?\n\nAdditional Comments\n---\n- Missing references that the authors may want to add:\n  - Wu et al. Text2lidar: Text-guided lidar point cloud generation via equirectangular transformer. In ECCV 2024.\n  - Pan et al. Transfer Your Perspective: Controllable 3D Generation from Any Viewpoint in a Driving Scene. In CVPR 2025.\n\nQUESTIONS:\nPlease see the above.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces X-Scene, a framework for generating large-scale, high-fidelity, and controllable 3D driving scenes. X-Scene is designed as a multi-stage pipeline. It begins with a \"Multi-Granular Controllability\" module that allows user input via either high-level text prompts or low-level geometric layouts. This control signal guides the core generation engine, which first synthesizes a 3D semantic occupancy field using a triplane-based diffusion model, and then generates corresponding photorealistic multi-view images conditioned on the 3D geometry. To achieve large-scale generation, the system employs a \"consistency-aware outpainting\" mechanism to iteratively expand the scene chunk by chunk, ensuring both geometric and visual coherence. The final generated assets are lifted to 3D 3DGS. The authors demonstrate through extensive experiments that their method achieves best results in both generation quality and utility for downstream perception tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1: The hierarchical generation process (3D occupancy first, then 2D images) is a good way to ensure 3D consistency. The use of a triplane representation with a novel deformable attention mechanism is a nice touch that, as shown in the ablations, demonstrably improves reconstruction quality. The wealth of conditioning information used for the image diffusion model (rendered semantic/depth maps, perspective maps, object embeddings) to have better alignment between the generated geometry and appearance, a common failure point in other methods.\n\n2: The experimental results are thorough. It also demonstrates significant gains when using its synthetic data to train downstream perception models. The ablation studies are excellent.\n\n3: Paper is easy to follow. \n\nWeaknesses: \n\n1: The most significant weakness, which the authors also acknowledge, is that the framework generates static scenes.  Real-world driving is fundamentally dynamic. While generating high-quality static worlds is a critical first step, the lack of dynamics limits its immediate use for end-to-end planning and simulation.\n\n2: X-Scene is a very complex, multi-stage cascading pipeline (LLM -> RAG -> Graph Diffusion -> Triplane Diffusion -> Image Diffusion). While powerful, this complexity raises concerns about reproducibility,  training/inference costs, and potential error propagation. How long is the inference time? Also a small error in the initial layout generation could lead to significant artifacts in the final rendered scene. The paper doesn't discuss how these errors are handled or how they might compound. \n\n3: The paper claims large-scale generation via iterative outpainting. However, the practical limits of this approach are not discussed. What are the computational/memory constraints as the scene grows? Does the con\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors describe a system based on multistage diffusion policies for synthesising driving scenarios from text prompts (and/or maps and boxes) to 360-degree images and semantic occupancy maps. The system can synthesise specially (but not temporarily) coherent scenarios and complete/expand/edit maps on a large scale.\n\nSTRENGTHS AND WEAKNESSES:\nI will use the symbol (+) to represent strengths and (-) to represent weaknesses.\n\n# Quality\n(+) The work describes a substantial framework for scenario generation. The framework consists of multiple steps that enable high-level and low-level conditioning and editing, as well as faster rendering through 3DGS.\n\n(+) The quality of the discussion, in terms of thoroughness in experiment design, is well-suited to demonstrating the system's performance.\n\n# Clarity\n(+) The work is well-written and structured.\n\n(+) All components of the framework are clearly discussed and formulated.\n\n(-) Some of the images contain quite small details that are difficult to appreciate on printed paper.\n\n# Significance\n(+) The work is interesting and covers a significant topic. The creation of scenarios for robotic tasks can lead to editable simulators that can enhance the policies trained on them, especially in edge cases not easily recorded in real datasets.\n\n# Originality\n(-) This work is crucially a system paper, which utilises multi-step diffusion policies to generate data. Except for minor adaptations, the concepts and elements are retrieved from previous works.\n\n(+) This said, the scale and performance achieved are impressive.\n\n(-) I suggest that the authors review `NeuralFloors: Conditional Street-Level Scene Generation From BEV Semantic Maps via Neural Fields` and `NeuralFloors++: Consistent Street-Level Scene Generation From BEV Semantic Maps`. Although it is unfortunately closed-source, it employs a similar triplane approach, combining NeRF and diffusion. From this, it would be interesting to take inspiration from metrics (FVD for consistency) and baselines (prompt-based diffusion policy to test how inputs affect the quality of reconstruction).\n\nQUESTIONS:\nHere are some questions for the authors to comment on.\nI have ordered them in priority order from my perspective.\n\nQ1 - metric for spatial consistency in outpainting: while the completion and extension of the scenarios are discussed qualitatively, I think it would be interesting to evaluate the system quantitatively, especially at discontinuity points. One way would be to synthesise videos and use video metrics (FVD).\n\nQ2 - baseline for pure text-based diffusion: how well would a pure text-based diffusion policy synthesise images of the scene from the given prompt? While most likely not spatially consistent or entirely following the map layout, it could provide hints on how the presented input affects the rendering qualities of such a model. \n\nQ3 - low-level metric for image generation: FID gives important information about the semantic distribution of the generated scenes. Could the authors propose a low-level metric for evaluating the realism of the generated images, perhaps to the provided training set?\n\nQ4 task metrics: please, can you explain why you limit your BEV segmentation evaluation to only road and vehicles?\n\nLIMITATIONS:\nyes. I would also include any limitations in terms of the training data required to achieve such performances (volume, alignment, labels, etc.).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}