SyntheOcc: Synthesize Geometric-Controlled Street View Images through 3D Semantic MPIs

Authors: Leheng Li, Weichao Qiu, Yingjie CAI, Xu Yan, Qing LIAN, Bingbing Liu, Ying-Cong Chen
Primary area: applications to robotics, autonomy, planning
Keywords: Autonomous Driving, Image Generation, Data-centric AI, 3D Vision

Abstract:
The advancement of autonomous driving is increasingly reliant on high-quality annotated datasets, especially in the task of 3D occupancy prediction, where the occupancy labels require dense 3D annotation with significant human effort. In this paper, we propose SyntheOcc, which denotes a diffusion model that Synthesize photorealistic and geometric-controlled images by conditioning Occupancy labels in driving scenarios. This yields an unlimited amount of diverse, annotated, and controllable datasets for applications like training perception models and simulation. SyntheOcc addresses the critical challenge of how to efficiently encode 3D geometric information as conditional input to a 2D diffusion model. Our approach innovatively incorporates 3D semantic multi-plane images (MPIs) to provide comprehensive and spatially aligned 3D scene descriptions for conditioning. As a result, SyntheOcc can generate photorealistic multi-view images and videos that faithfully align with the given geometric labels (semantics in 3D voxel space). Extensive qualitative and quantitative evaluations of SyntheOcc on the nuScenes dataset prove its effectiveness in generating controllable occupancy datasets that serve as an effective data augmentation to perception models.

Decision: reject
Reviewer scores: 5, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730333249590

thread_type:
official_review

<thread_object>
creator: reviewer_nhs7

<content>
summary:
The paper proposes SyntheOcc, a novel image generation framework that can synthesize photorealistic and geometric-controlled street view images by conditioning on 3D occupancy labels. The key innovation is the use of 3D semantic multi-plane images (MPIs) to efficiently encode 3D geometric information as conditional input to the 2D diffusion model. The extensive experiments demonstrate that the synthetic data generated by SyntheOcc can effectively augment perception models for 3D occupancy prediction tasks.

strengths:
1. The paper proposes an innovative approach by replacing ControlNet with multiplane images, enhancing image synchronization from occluded views.
2. The provided video effectively demonstrates and supports the proposed method.
3. The writing is clear and easy to follow.
4. The experiments are comprehensive, covering both quantitative and qualitative evaluations.

weaknesses:
1. **Clarification on excluding ControlNet**: The paper should more thoroughly explain the decision to exclude ControlNet. The rationale for why ControlNet fails to meet 3D requirements remains unclear. Since multiplane images could serve as conditions for ControlNet.

2. **Incorporating KPM evaluation for consistency**: It would be beneficial for the paper to include KPM evaluations from Driving into the Future [1] to better assess temporal and multiview consistency.

3. **Additional out-of-domain results**: Presenting more out-of-domain results, such as experiments with variations in camera intrinsic and extrinsic parameters, would explain the ability of generalization of the model.

4. **Weak video quality**: Some objects in the provided video appear twisted or lack realistic representation. And the videos look a little bit unreal but I cannot tell why.

5. **World model integration**: Considering that driving scene generation works nowadays provides world model results—capable of forecasting future layouts and generating future images based on actions. It would be valuable for the paper to explore integration with world models. For example, testing if the generation method can be adapted to synthesize occupancy predictions, as seen in recent work on occupancy-based world models [2]. It will showcase potential for further real-world applications.

[1] Wang, Yuqi, et al. "Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving." CVPR 2024.
[2] Zheng, Wenzhao, et al. "Occworld: Learning a 3d occupancy world model for autonomous driving." ECCV 2024

questions:
I am still wondering why the paper cannot use ControlNet, as multiplane images are still images and could potentially be used as input for ControlNet.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730380601405

thread_type:
official_review

<thread_object>
creator: reviewer_ktuh

<content>
summary:
This paper introduces SytheOcc, a novel image generation framework enabling precise 3D geometric control for applications like 3D editing and dataset generation. By leveraging 3D semantic multiplane images (MPIs), the framework achieves finer geometry and semantic control, enhancing image quality and recognizability. Experimental results show the effectiveness of synthetic data in augmenting 3D occupancy prediction tasks, indicating a significant advancement over existing methods.

strengths:
1. SytheOcc offers finer and precise 3D geometric control, allowing for intricate manipulation of object shapes and scene geometry, which is crucial for tasks like 3D editing and dataset generation.
2. Experimental results demonstrate that the synthetic data generated by SytheOcc exhibit better recognizability, indicating a substantial advancement in image quality over existing methods.
3. The synthetic data produced by SytheOcc prove to be highly effective for data augmentation in 3D occupancy prediction tasks, enhancing the performance and robustness of perception models in such applications.

weaknesses:
1. The practicality of occupancy editing in 3D space should be addressed. It is crucial to automate or accelerate the editing process to make it feasible for practical applications requiring large amounts of data. The authors may report the time required to generate a new image through editing, and discuss possible solutions to scaling up data.
2. The paper use occupancy as a condition due to its spatial information. It would be beneficial to discuss the fundamental differences between using occupancy and using of depth&segmentation maps as conditions to control image generation [1]. Experimental comparisons between using depth&semantic maps versus occupancy as conditions could be conducted to evaluate metrics like FID and inference time.
3. The multi-view consistency appears not so good. In Figure 5 (b) (c), the color of the car in the first row changes in the second and third images. The authors could includsa comparative analysis of the generation results from different models (e.g., MagicDrive) within the same scene.
4. The concept of imbalance mentioned by the authors in Line 272 requires further clarification. It is essential for the authors to provide a detailed explanation of what this imbalance refers to and how it impacts their proposed framework.
5. The examples of editing provided in paper mostly revolve simple cars. In Figure 6, it would be beneficial to explore if the model can accurately move and position more complex, irregularly shaped vehicles or pedestrians to demonstrate the capability of the framework in generating new, diverse scenes.
6. While the paper discusses some designs related to temporal consistency, only qualitative results are presented. It would be valuable for the authors to report metrics like FVD compared to existing works such as DrivingDiffusion and Panacea to provide a more comprehensive evaluation of the proposed framework.
7. Missing reference: [2-4]

[1] UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild

[2] Generalized Predictive Model for Autonomous Driving

[3] Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability

[4] SimGen: Simulator-conditioned Driving Scene Generation

questions:
By conducting these expanded experimental comparisons, the authors can more comprehensively validate their claims and provide more compelling evidence for the effectiveness of the proposed SytheOcc framework.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730781315593

thread_type:
official_review

<thread_object>
creator: reviewer_ypbc

<content>
summary:
This paper introduces SyntheOcc, a controllable camera image simulation framework in the autonomous driving domain. The proposed framework uses 3D semantic occupancy grid as the conditions for camera image simulation, where multi-plane semantic images (MPIs) projected from 3D semantic occupancy grids have been used as conditional input to a 2D diffusion model. The effectiveness of SyntheOcc is demonstrated through improved performance in Real-to-sim evaluation and Sim-to-real data augmentation on the NuScenes dataset.

strengths:
- [S1: Quality] The paper has demonstrated extensive experimental results and showcased that the proposed method is superior under both real-to-sim evaluation and sim-to-real data augmentation, when compared against existing methods. The paper also includes ablation studies including the MPI encoder architecture and reweighing methods.

- [S2: Clarity] The proposed method is well described in detail with clear illustrations (e.g., Figure 1 and Figure 2).

weaknesses:
- [W1] The paper does not include several very relevant work in the literature review. Most of the baselines used in the paper come from publications within the past two years. The reviewer feels that two relevant papers on camera simulation is missing [NewRef1] and [NewRef2] from the literature review.

- [W2] This paper does not compare against an important baseline UniSim [Yang et al., CVPR 2023].
While the proposed method is pure data-driven, is it possible to showcase the results on Pandaset used in the Unisim? It is unclear whether the proposed method is superior to UniSim as a camera image simulator or simply works well on Nuscenes dataset but does not generalize to other datasets (e.g., Pandaset, Waymo Open Dataset). The reviewer feels that such discussions and experimental comparisons are needed as a strong justification for acceptance.
  - [W2.1] It is important to understand whether the proposed method is transferrable to other datasets with minimum fine-tuning or adaptation. For example, as shown in Figure 6 of GeoSim paper [NewRef1], the same pipeline works for a different city in the Argoverse dataset.

- [W3] While the data augmentation experimental results are interesting (section 4.3, first two rows in Table 1), this paper does not comment on the role of synthetically generated data in nuScenes occupancy prediction. 
  - [W3.1] The semantic categories with significant improvements are bus, traffic cones, trailer, driving surface, other flat, and sidewalk. It is unclear how and why synthetically generated data help for such categories but not the other categories. Is it possible to provide convincing qualitative examples with explanations? 
  - [W3.2] Comment on the overall improvements to the driving system (e.g., behavior prediction and motion planning). How much does the long-tailed scene generation help the downstream tasks? 

References:
- [NewRef1] GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving, Chen et al., In CVPR 2021.
- [NewRef2] Block-NeRF: Scalable Large Scene Neural View Synthesis, Tancik et al., In CVPR 2022.

questions:
Please address the questions raised in the weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730907893445

thread_type:
official_review

<thread_object>
creator: reviewer_gbog

<content>
summary:
This paper presents SyntheOcc, a method for generating multi-camera images and videos of driving scenarios, using occupancy and text prompt as guiding inputs. The innovation of SyntheOcc lies in its proposed MPI encoder, which projects the raw occupancy of different depth ranges onto the camera plane, combining them into semantic multiplane images. These semantic multiplane images are then encoded as guidance for image generation. The paper provides a robust qualitative and quantitative comparison of generated images and videos. Additionally, it demonstrates the performance of perception models trained on the synthetic data and tested on real validation sets, as well as perception models trained on real data and tested on synthetic validation sets, to validate the proximity of SyntheOcc-generated images to the real domain.

strengths:
1. SyntheOcc has potential for generating rare long-tail data that could support downstream tasks in real-world scenarios.
2. The experiments are solid, offering extensive qualitative and quantitative comparisons. The key validation experiments, blending generated data with real data, are particularly convincing.

weaknesses:
1. While the paper proposes the MPIs for encoding occupancy as guidance for image generation, the overall novelty remains unclear in several areas. The core functionalities—such as multi-camera image and video generation for driving scenes—are well-covered in prior works like MagicDrive [1], DriveDreamer [2], and Drive-WM [3], which demonstrate strong temporal consistency in video generation that appears more stable than the qualitative results in this paper. It would be helpful for the authors to clarify the novel aspects of SyntheOcc’s contributions by explicitly identifying any unique advantages or improvements over these methods. Furthermore, key capabilities like generating out-of-domain images and videos via text prompts, using occupancy as guiding inputs, editing by modifying ocupancy elements, and simulating images with varying camera intrinsics are already present in WovoGen [4]. The authors could strengthen the paper by specifying which of these functionalities are advanced by SyntheOcc and justifying their relevance. Providing concrete comparisons to these works, either through experiments or discussion, could more clearly highlight SyntheOcc’s contributions and novelty.
2. The generated results of SyntheOcc show notable issues with road markings (evident in Figures 11, 12, and 13), which could be problematic for certain downstream tasks, such as planning. The paper lacks a systematic evaluation of these synthetic data’s impact on such tasks. Some previous works, like BEVGen [4] and MagicDrive [1], leverage HD maps as guidance, which could effectively resolve such issues. To strengthen the paper, the authors could provide a quantitative analysis comparing the quality of road markings in their generated images to ground truth or to outputs from other methods. Such an analysis would clarify the current limitations and help illustrate areas for potential improvement. Additionally, evaluating the impact of these road marking inconsistencies on specific downstream tasks, such as lane detection or path planning, could further demonstrate the practical implications of this issue and guide refinements to improve SyntheOcc’s usability for these applications.
3. I am not convinced that SyntheOcc effectively expands current datasets. Firstly, the paper mentions (Table 9) that excessive synthetic data can hinder perception model performance. Some corner cases, which require manual adjustment, are the truly valuable data needed in datasets (such as those in the lower parts of Figures 1 and 7). Editing these corner cases still demands considerable manual intervention (e.g., placing barriers on roads, altering road structures, or positioning pedestrians atop vehicles).


references:
[1] Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhenguo Li, Dit-Yan Yeung, and Qiang Xu. Magicdrive: Street view generation with diverse 3d geometry control. In ICLR, 2024.
[2] Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, and Jiwen Lu. Drivedreamer: Towards real-world-driven world models for autonomous driving. In arxiv, 2024.
[3] Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In CVPR, 2024.
[4] Jiachen Lu, Ze Huang, Jiahui Zhang, Zeyu Yang, and Li Zhang. Wovogen: World volume-aware diffusion for controllable multi-camera driving scene generation. In ECCV, 2024.
[5] Alexander Swerdlow, Runsheng Xu, and Bolei Zhou. Street-view image generation from a bird’s-eye view layout. IEEE RAL, 2024.

questions:
What specific perception model was used in the experiments?
</content>
</thread_object>

</thread>


Metareview: