MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction
  Model
Authors: name: Minghua Liu; name: Chong Zeng; name: Xinyue Wei; name: Ruoxi Shi; name: Linghao Chen; name: Chao Xu; name: Mengqi Zhang; name: Zhaoning Wang; name: Xiaoshuai Zhang; name: Isabella Liu; name: Hongzhi Wu; name: Hao Su
Published: 2024-08-19 17:55:17

Abstract: Open-world 3D reconstruction models have recently garnered significant
attention. However, without sufficient 3D inductive bias, existing methods
typically entail expensive training costs and struggle to extract high-quality
3D meshes. In this work, we introduce MeshFormer, a sparse-view reconstruction
model that explicitly leverages 3D native structure, input guidance, and
training supervision. Specifically, instead of using a triplane representation,
we store features in 3D sparse voxels and combine transformers with 3D
convolutions to leverage an explicit 3D structure and projective bias. In
addition to sparse-view RGB input, we require the network to take input and
generate corresponding normal maps. The input normal maps can be predicted by
2D diffusion models, significantly aiding in the guidance and refinement of the
geometry's learning. Moreover, by combining Signed Distance Function (SDF)
supervision with surface rendering, we directly learn to generate high-quality
meshes without the need for complex multi-stage training processes. By
incorporating these explicit 3D biases, MeshFormer can be trained efficiently
and deliver high-quality textured meshes with fine-grained geometric details.
It can also be integrated with 2D diffusion models to enable fast
single-image-to-3D and text-to-3D tasks. Project page:
https://meshformer3d.github.io