AToM: Amortized Text-to-Mesh using 2D Diffusion
Authors: name: Guocheng Qian; name: Junli Cao; name: Aliaksandr Siarohin; name: Yash Kant; name: Chaoyang Wang; name: Michael Vasilkovsky; name: Hsin-Ying Lee; name: Yuwei Fang; name: Ivan Skorokhodov; name: Peiye Zhuang; name: Igor Gilitschenski; name: Jian Ren; name: Bernard Ghanem; name: Kfir Aberman; name: Sergey Tulyakov
Published: 2024-02-01 18:59:56

Abstract: We introduce Amortized Text-to-Mesh (AToM), a feed-forward text-to-mesh
framework optimized across multiple text prompts simultaneously. In contrast to
existing text-to-3D methods that often entail time-consuming per-prompt
optimization and commonly output representations other than polygonal meshes,
AToM directly generates high-quality textured meshes in less than 1 second with
around 10 times reduction in the training cost, and generalizes to unseen
prompts. Our key idea is a novel triplane-based text-to-mesh architecture with
a two-stage amortized optimization strategy that ensures stable training and
enables scalability. Through extensive experiments on various prompt
benchmarks, AToM significantly outperforms state-of-the-art amortized
approaches with over 4 times higher accuracy (in DF415 dataset) and produces
more distinguishable and higher-quality 3D outputs. AToM demonstrates strong
generalizability, offering finegrained 3D assets for unseen interpolated
prompts without further optimization during inference, unlike per-prompt
solutions.