Structured 3D Latents for Scalable and Versatile 3D Generation
Authors: name: Jianfeng Xiang; name: Zelong Lv; name: Sicheng Xu; name: Yu Deng; name: Ruicheng Wang; name: Bowen Zhang; name: Dong Chen; name: Xin Tong; name: Jiaolong Yang
Published: 2024-12-02 13:58:38

Abstract: We introduce a novel 3D generation method for versatile and high-quality 3D
asset creation. The cornerstone is a unified Structured LATent (SLAT)
representation which allows decoding to different output formats, such as
Radiance Fields, 3D Gaussians, and meshes. This is achieved by integrating a
sparsely-populated 3D grid with dense multiview visual features extracted from
a powerful vision foundation model, comprehensively capturing both structural
(geometry) and textural (appearance) information while maintaining flexibility
during decoding. We employ rectified flow transformers tailored for SLAT as our
3D generation models and train models with up to 2 billion parameters on a
large 3D asset dataset of 500K diverse objects. Our model generates
high-quality results with text or image conditions, significantly surpassing
existing methods, including recent ones at similar scales. We showcase flexible
output format selection and local 3D editing capabilities which were not
offered by previous models. Code, model, and data will be released.