VolumeDiffusion: Flexible Text-to-3D Generation with Efficient
  Volumetric Encoder
Authors: name: Zhicong Tang; name: Shuyang Gu; name: Chunyu Wang; name: Ting Zhang; name: Jianmin Bao; name: Dong Chen; name: Baining Guo
Published: 2023-12-18 18:59:05

Abstract: This paper introduces a pioneering 3D volumetric encoder designed for
text-to-3D generation. To scale up the training data for the diffusion model, a
lightweight network is developed to efficiently acquire feature volumes from
multi-view images. The 3D volumes are then trained on a diffusion model for
text-to-3D generation using a 3D U-Net. This research further addresses the
challenges of inaccurate object captions and high-dimensional feature volumes.
The proposed model, trained on the public Objaverse dataset, demonstrates
promising outcomes in producing diverse and recognizable samples from text
prompts. Notably, it empowers finer control over object part characteristics
through textual cues, fostering model creativity by seamlessly combining
multiple concepts within a single object. This research significantly
contributes to the progress of 3D generation by introducing an efficient,
flexible, and scalable representation methodology. Code is available at
https://github.com/tzco/VolumeDiffusion.