MeshAnything: Artist-Created Mesh Generation with Autoregressive
  Transformers
Authors: name: Yiwen Chen; name: Tong He; name: Di Huang; name: Weicai Ye; name: Sijin Chen; name: Jiaxiang Tang; name: Xin Chen; name: Zhongang Cai; name: Lei Yang; name: Gang Yu; name: Guosheng Lin; name: Chi Zhang
Published: 2024-06-14 16:30:25

Abstract: Recently, 3D assets created via reconstruction and generation have matched
the quality of manually crafted assets, highlighting their potential for
replacement. However, this potential is largely unrealized because these assets
always need to be converted to meshes for 3D industry applications, and the
meshes produced by current mesh extraction methods are significantly inferior
to Artist-Created Meshes (AMs), i.e., meshes created by human artists.
Specifically, current mesh extraction methods rely on dense faces and ignore
geometric features, leading to inefficiencies, complicated post-processing, and
lower representation quality. To address these issues, we introduce
MeshAnything, a model that treats mesh extraction as a generation problem,
producing AMs aligned with specified shapes. By converting 3D assets in any 3D
representation into AMs, MeshAnything can be integrated with various 3D asset
production methods, thereby enhancing their application across the 3D industry.
The architecture of MeshAnything comprises a VQ-VAE and a shape-conditioned
decoder-only transformer. We first learn a mesh vocabulary using the VQ-VAE,
then train the shape-conditioned decoder-only transformer on this vocabulary
for shape-conditioned autoregressive mesh generation. Our extensive experiments
show that our method generates AMs with hundreds of times fewer faces,
significantly improving storage, rendering, and simulation efficiencies, while
achieving precision comparable to previous methods.