Baking Gaussian Splatting into Diffusion Denoiser for Fast and Scalable
  Single-stage Image-to-3D Generation
Authors: name: Yuanhao Cai; name: He Zhang; name: Kai Zhang; name: Yixun Liang; name: Mengwei Ren; name: Fujun Luan; name: Qing Liu; name: Soo Ye Kim; name: Jianming Zhang; name: Zhifei Zhang; name: Yuqian Zhou; name: Zhe Lin; name: Alan Yuille
Published: 2024-11-21 18:21:24

Abstract: Existing feed-forward image-to-3D methods mainly rely on 2D multi-view
diffusion models that cannot guarantee 3D consistency. These methods easily
collapse when changing the prompt view direction and mainly handle
object-centric prompt images. In this paper, we propose a novel single-stage 3D
diffusion model, DiffusionGS, for object and scene generation from a single
view. DiffusionGS directly outputs 3D Gaussian point clouds at each timestep to
enforce view consistency and allow the model to generate robustly given prompt
views of any directions, beyond object-centric inputs. Plus, to improve the
capability and generalization ability of DiffusionGS, we scale up 3D training
data by developing a scene-object mixed training strategy. Experiments show
that our method enjoys better generation quality (2.20 dB higher in PSNR and
23.25 lower in FID) and over 5x faster speed (~6s on an A100 GPU) than SOTA
methods. The user study and text-to-3D applications also reveals the practical
values of our method. Our Project page at
https://caiyuanhao1998.github.io/project/DiffusionGS/ shows the video and
interactive generation results.