TripoSG: High-Fidelity 3D Shape Synthesis using Large-Scale Rectified
  Flow Models
Authors: name: Yangguang Li, name: Zi-Xin Zou, name: Zexiang Liu, name: Dehu Wang, name: Yuan Liang, name: Zhipeng Yu, name: Xingchao Liu, name: Yuan-Chen Guo, name: Ding Liang, name: Wanli Ouyang, name: Yan-Pei Cao

Recent advancements in diffusion techniques have propelled image and video
generation to unprece- dented levels of quality, significantly accelerating the
deployment and application of generative AI. However, 3D shape generation
technology has so far lagged behind, constrained by limitations in 3D data
scale, complexity of 3D data process- ing, and insufficient exploration of
advanced tech- niques in the 3D domain. Current approaches to 3D shape
generation face substantial challenges in terms of output quality,
generalization capa- bility, and alignment with input conditions. We present
TripoSG, a new streamlined shape diffu- sion paradigm capable of generating
high-fidelity 3D meshes with precise correspondence to input images.
Specifically, we propose: 1) A large-scale rectified flow transformer for 3D
shape generation, achieving state-of-the-art fidelity through training on
extensive, high-quality data. 2) A hybrid supervised training strategy
combining SDF, normal, and eikonal losses for 3D VAE, achieving high- quality
3D reconstruction performance. 3) A data processing pipeline to generate 2
million high- quality 3D samples, highlighting the crucial rules for data
quality and quantity in training 3D gen- erative models. Through comprehensive
experi- ments, we have validated the effectiveness of each component in our new
framework. The seamless integration of these parts has enabled TripoSG to
achieve state-of-the-art performance in 3D shape generation. The resulting 3D
shapes exhibit en- hanced detail due to high-resolution capabilities and
demonstrate exceptional fidelity to input im- ages. Moreover, TripoSG
demonstrates improved versatility in generating 3D models from diverse image
styles and contents, showcasing strong gen- eralization capabilities. To foster
progress and innovation in the field of 3D generation, we will make our model
publicly available.