GVGEN: Text-to-3D Generation with Volumetric Representation
Authors: name: Xianglong He; name: Junyi Chen; name: Sida Peng; name: Di Huang; name: Yangguang Li; name: Xiaoshui Huang; name: Chun Yuan; name: Wanli Ouyang; name: Tong He
Published: 2024-03-19 17:57:52

Abstract: In recent years, 3D Gaussian splatting has emerged as a powerful technique
for 3D reconstruction and generation, known for its fast and high-quality
rendering capabilities. To address these shortcomings, this paper introduces a
novel diffusion-based framework, GVGEN, designed to efficiently generate 3D
Gaussian representations from text input. We propose two innovative
techniques:(1) Structured Volumetric Representation. We first arrange
disorganized 3D Gaussian points as a structured form GaussianVolume. This
transformation allows the capture of intricate texture details within a volume
composed of a fixed number of Gaussians. To better optimize the representation
of these details, we propose a unique pruning and densifying method named the
Candidate Pool Strategy, enhancing detail fidelity through selective
optimization. (2) Coarse-to-fine Generation Pipeline. To simplify the
generation of GaussianVolume and empower the model to generate instances with
detailed 3D geometry, we propose a coarse-to-fine pipeline. It initially
constructs a basic geometric structure, followed by the prediction of complete
Gaussian attributes. Our framework, GVGEN, demonstrates superior performance in
qualitative and quantitative assessments compared to existing 3D generation
methods. Simultaneously, it maintains a fast generation speed (sim7
seconds), effectively striking a balance between quality and efficiency.