SyntheOcc: Synthesize Geometric-Controlled Street View Images through 3D
  Semantic MPIs
Authors: name: Leheng Li; name: Weichao Qiu; name: Yingjie Cai; name: Xu Yan; name: Qing Lian; name: Bingbing Liu; name: Ying-Cong Chen
Published: 2024-10-01 02:29:24

Abstract: The advancement of autonomous driving is increasingly reliant on high-quality
annotated datasets, especially in the task of 3D occupancy prediction, where
the occupancy labels require dense 3D annotation with significant human effort.
In this paper, we propose SyntheOcc, which denotes a diffusion model that
Synthesize photorealistic and geometric-controlled images by conditioning
Occupancy labels in driving scenarios. This yields an unlimited amount of
diverse, annotated, and controllable datasets for applications like training
perception models and simulation. SyntheOcc addresses the critical challenge of
how to efficiently encode 3D geometric information as conditional input to a 2D
diffusion model. Our approach innovatively incorporates 3D semantic multi-plane
images (MPIs) to provide comprehensive and spatially aligned 3D scene
descriptions for conditioning. As a result, SyntheOcc can generate
photorealistic multi-view images and videos that faithfully align with the
given geometric labels (semantics in 3D voxel space). Extensive qualitative and
quantitative evaluations of SyntheOcc on the nuScenes dataset prove its
effectiveness in generating controllable occupancy datasets that serve as an
effective data augmentation to perception models.