CraftsMan: High-fidelity Mesh Generation with 3D Native Generation and
  Interactive Geometry Refiner
Authors: name: Weiyu Li; name: Jiarui Liu; name: Rui Chen; name: Yixun Liang; name: Xuelin Chen; name: Ping Tan; name: Xiaoxiao Long
Published: 2024-05-23 18:30:12

Abstract: We present a novel generative 3D modeling system, coined CraftsMan, which can
generate high-fidelity 3D geometries with highly varied shapes, regular mesh
topologies, and detailed surfaces, and, notably, allows for refining the
geometry in an interactive manner. Despite the significant advancements in 3D
generation, existing methods still struggle with lengthy optimization
processes, irregular mesh topologies, noisy surfaces, and difficulties in
accommodating user edits, consequently impeding their widespread adoption and
implementation in 3D modeling software. Our work is inspired by the craftsman,
who usually roughs out the holistic figure of the work first and elaborates the
surface details subsequently. Specifically, we employ a 3D native diffusion
model, which operates on latent space learned from latent set-based 3D
representations, to generate coarse geometries with regular mesh topology in
seconds. In particular, this process takes as input a text prompt or a
reference image and leverages a powerful multi-view (MV) diffusion model to
generate multiple views of the coarse geometry, which are fed into our
MV-conditioned 3D diffusion model for generating the 3D geometry, significantly
improving robustness and generalizability. Following that, a normal-based
geometry refiner is used to significantly enhance the surface details. This
refinement can be performed automatically, or interactively with user-supplied
edits. Extensive experiments demonstrate that our method achieves high efficacy
in producing superior-quality 3D assets compared to existing methods. HomePage:
https://craftsman3d.github.io/, Code: https://github.com/wyysf-98/CraftsMan