Title: SyntheOcc: Synthesize Occupancy-Controlled Street View Images through 3D Semantic MPIs
Venue: NeurIPS 2025
Accepted: no
Keywords: Autonomous Driving

Generative Model

Image and Video Generation

Data-centric AI

3D Vision
Abstract: The advancement of autonomous driving is increasingly reliant on high-quality annotated datasets, especially in the task of 3D occupancy prediction, where the occupancy labels require dense 3D annotation with significant human effort. In this paper, we propose SyntheOcc, which denotes a diffusion model that Synthesize photorealistic and geometric controllable images by conditioning Occupancy labels in driving scenarios. This yields an unlimited amount of diverse, annotated, and controllable datasets for applications like training perception models and simulation. SyntheOcc addresses the critical challenge of how to efficiently encode 3D geometric information as conditional input to a 2D diffusion model. Our approach innovatively incorporates 3D semantic multi-plane images (MPIs) to provide comprehensive and spatially aligned 3D scene descriptions for conditioning. By doing so, SyntheOcc can generate photorealistic multi-view images and videos that faithfully align with the given geometric labels (semantics in 3D voxel space). Extensive qualitative and quantitative evaluations of SyntheOcc on the nuScenes dataset prove its effectiveness in generating controllable occupancy datasets that serve as an effective data augmentation to perception models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents SyntheOcc, a diffusion-based framework for generating realistic, geometrically controllable street-level images and videos by conditioning on 3D semantic occupancy labels. SyntheOcc leverages 3D semantic multi-plane images (MPIs) to encode detailed geometric and semantic information, enabling precise scene editing, dataset augmentation, and long-tail scene synthesis for autonomous driving applications. Experiments on nuScenes demonstrate its superiority over prior methods in terms of image quality, 3D controllability, and performance on downstream tasks such as occupancy prediction.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n(1) SyntheOcc shows great potential in addressing the challenges of long-tail distributions and enriching datasets, providing strong support for downstream practical applications.\n(2) This paper establishes a rigorous experimental framework and conducts extensive qualitative and quantitative analysis. Empirical results confirm that the method outperforms existing methods in both real data to simulated data evaluation and simulated data to real data data augmentation. Further ablation studies confirm key design decisions.\n(3) The paper has very clear and detailed explanations, intuitive visualizations, and logical structures that enhance understanding and reproducibility.\n\n\nWeakness:\n(1) Limited video generation quality: SyntheOcc’s generated results have noticeable artifacts, especially when rendering lane lines on roads (as shown in Figure 11), where lane lines disappear on some images. These inaccuracies may have an adverse impact on subsequent tasks such as motion planning, where accurate scene understanding is critical.\n(2) Limited complexity of object editing: The current experiments mainly demonstrate scene editing using simple, regularly shaped objects. To better evaluate the generalization ability of the framework, more complex and irregular objects should be tested in tasks such as relocalization or insertion.\n(3) Unclear dataset expansion effect: The paper claims that SyntheOcc increases the diversity of the dataset, but Table 9 shows that too much synthetic data can degrade the performance of the model. Therefore, the proposed method may not be very effective for expanding the dataset.\n(4) Recent work on image or video generation in autonomous driving should be included in the comparison method, such as UniScene[1]. \n\nLi B, Guo J, Liu H, et al. Uniscene: Unified occupancy-centric driving scene generation. CVPR, 2025\n\nQUESTIONS:\nplease address the questions in weakness section.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes SyntheOcc, a method that generates street-view images conditioned on a 3D semantic occupancy map. The core idea is to use multiple image planes (MPIs) to encode the 3D semantic scene, which are then used as conditioning for a diffusion model. This adds conditioning on the depth dimension and occluded scenery, allowing for better control over the generated images. The paper also proposes several data re-weighting strategies, including progressive foreground enhancement, depth-aware foreground re-weighting, and class-balanced grouping and sampling. These strategies improve supervision on foreground regions, particularly those at a distance, while better handling long-tailed classes. The method is shown to be effective for data augmentation and generates images with better FID scores than previous methods on the nuScenes dataset.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths:\n- The use of multiple image planes (MPIs) to create 3D semantic scene conditioning for a diffusion model is a novel approach. This provides conditioning on both depth and occluded scenery, allowing for greater control over the generated images.\n\n- The paper is well-written and clearly explains its core ideas, with the exception of the related work section.\n\n- The method proves useful for data augmentation, improving mIoU by approximately one point, and generates images with superior FID scores compared to previous methods on the nuScenes dataset.\n\n# Weaknesses and Questions\n\n- The main advantage of multiple image planes (MPIs) over standard voxel projection is their ability to encode occluded regions. To clarify this effect, the authors should include an ablation study where only visible voxels are fed to the network (i.e., by masking all occluded pixels in all planes). On line 250, the authors state, \"The ControlNet+depth can be regarded as a degradation of SyntheOcc which is reduced to a single plane.\" I disagree with this comparison, as the differing architectures of SyntheOcc and ControlNet make it difficult to isolate whether the performance gains truly stem from the modeling of occluded regions.\n\n- The related work section is lacking in its coverage of 3D occupancy prediction and diffusion-based image generation. The section on 3D occupancy prediction lists only four seemingly random references. The part on diffusion-based generation discusses only Stable Diffusion (CVPR'22) and ControlNet (CVPR'23), offering limited insight and suggesting an incomplete literature review. Additionally, the introduction (line 24) fails to cite seminal works in camera-based 3D occupancy prediction, such as MonoScene (CVPR'22), which pioneered the approach.\n\n- While SyntheOcc can be used for data augmentation, the resulting improvement of ~1 mIoU is limited. Furthermore, the training process is computationally expensive (requiring 8 A100 GPUs for one day), not to mention the inference time. This makes its practical application for data augmentation questionable.\n\n- Figure 7a is unconvincing because the comparison is made with a ControlNet model that lacks depth information. The ambiguity in the example could likely be resolved by conditioning on depth.\n\n- It is unclear whether text prompts are used during training. If so, the paper should explain how these prompts are generated.\n\n- The \"Gen Val\" setting in Table 1 seems to create an unfair comparison. The validation set images are generated by SyntheOcc, and all methods, including SyntheOcc itself, are evaluated on this set. This inherently advantages SyntheOcc.\n\n- The \"Progressive Foreground Enhancement\" and \"Depth-aware Foreground Reweighing\" modules seem heavily hand-crafted. The paper does not provide clear guidance on how to set the parameters for these modules.\n\nQUESTIONS:\nQuestions are included alongside weaknesses to better contextualize.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SyntheOcc, a novel generative framework for synthesizing realistic, controllable street-view images and videos conditioned on 3D semantic occupancy grids. The key innovation is using 3D Semantic Multi-Plane Images (MPIs) as conditioning input for a diffusion model, effectively encoding detailed 3D geometry and semantics while maintaining pixel-aligned relationships. The framework includes a specialized MPI encoder and re-weighting strategies to improve foreground object generation in imbalanced driving scenes. SyntheOcc demonstrates effective data augmentation capabilities for improving downstream perception tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n- Utilizing multiplane images (MPIs) as representation provides a novel method to encode 3D semantic occupancy.\n- Addresses critical data scarcity problem for dense 3D perception using 3D semantic MPIs, bridging 3D occupancy and 2D image synthesis with geometric soundness and occlusion handling.\n- Through manipulating voxels, SyntheOcc is capable of creating diverse novel scenes, which is vital for downstream tasks such as scene editing and data augmentation.\n\n**Weaknesses**:\n- While 3D semantic occupancy provides good geometric control, compared to using HD maps and bounding boxes as input conditions, the control over traffic element types and scene editing convenience remains limited. For example, control over background elements such as crossings, as well as lane width and number of lanes, is still limited.\n- It would be better to provide more test cases such as rotating the ego vehicle, translating it to other lanes and cutting in scenarios to validate the spatial generation consistency of the proposed method, further demonstrating the potential for practical applications.\n- The current FVD comparison lacks corresponding evaluation settings and detailed assessment protocols, making it difficult to fully assess the quality of generated videos.\n\nQUESTIONS:\nPlease address the concerns raised in the weaknesses section.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presented SyntheOcc, an image and video generation method conditioned on 3D occupancy, to generate photorealistic images. A key component of the proposed method is to represent 3D occupancy as a stack of Multi-Plane Images (MPIs), preserving geometric and semantic information. Certain reweighting strategies during training time are also proposed to address the class imbalance problem.\n\nSTRENGTHS AND WEAKNESSES:\n[Strengths]\n1) Representing 3D occupancy as MPIs is a good attempt to preserve geometry and semantic information, which is important for image and video generation. Such representation also has the potential to lead to better efficiency.\n2) Compared to previous methods, the video generation results in terms of FID/FVD show modest improvement.\n\n[Weaknesses]\n\n1) I don't think the claim of 3D MPIs is a lossless representation of 3D scenes is correct. For one, as described in Eq. 1-2 in Section 3.2, for entities outside $[d_{min}, d_{max}]$ by definition cannot be captured by such MPIs, leading to information loss. For another, MPIs are limited by viewpoints, which means handling all occlusions is impossible. Hence this claim \"lossless representation\" is misleading.\n2) It is not clear to me how much benefit of using 3D MPIs brings. For example, in UniScene[1] which the authors also surveyed in Section 2.3, directly conditions on the full 3D occupancy volume for image and video generation, and demonstrated an FID score of 6.12 (Table 4 in UniScene paper) on nuScenes, far outperforms proposed method's 14.75. \n3) Inference runtime analysis is missing. It would very beneficial to include them since that's a critical aspect of evaluating image/video generation approaches especially in terms of practicality.\n\n[1] Li, Bohan, et al. \"Uniscene: Unified occupancy-centric driving scene generation.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\nQUESTIONS:\n1) How is Eq. 8 applied to reweighting loss incurred from foreground objects in Eq. 9? Is it done through masking (by using GT?)\n2) For the \"Gen\" validation set in Table. 1, is it generated by the proposed method?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}