DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction
  Model
Authors: name: Yinghao Xu, name: Hao Tan, name: Fujun Luan, name: Sai Bi, name: Peng Wang, name: Jiahao Li, name: Zifan Shi, name: Kalyan Sunkavalli, name: Gordon Wetzstein, name: Zexiang Xu, name: Kai Zhang

We propose DMV3D, a novel 3D generation approach that uses a
transformer-based 3D large reconstruction model to denoise multi-view
diffusion. Our reconstruction model incorporates a triplane NeRF representation
and can denoise noisy multi-view images via NeRF reconstruction and rendering,
achieving single-stage 3D generation in sim30s on single A100 GPU. We train
DMV3D on large-scale multi-view image datasets of highly diverse
objects using only image reconstruction losses, without accessing 3D assets. We
demonstrate state-of-the-art results for the single-image reconstruction
problem where probabilistic modeling of unseen object parts is required for
generating diverse reconstructions with sharp textures. We also show
high-quality text-to-3D generation results outperforming previous 3D diffusion
models. Our project website is at: https://justimyhxu.github.io/projects/dmv3d/ .