HexaGen3D: StableDiffusion is just one step away from Fast and Diverse
  Text-to-3D Generation
Authors: name: Antoine Mercier; name: Ramin Nakhli; name: Mahesh Reddy; name: Rajeev Yasarla; name: Hong Cai; name: Fatih Porikli; name: Guillaume Berger
Published: 2024-01-15 14:41:15

Abstract: Despite the latest remarkable advances in generative modeling, efficient
generation of high-quality 3D assets from textual prompts remains a difficult
task. A key challenge lies in data scarcity: the most extensive 3D datasets
encompass merely millions of assets, while their 2D counterparts contain
billions of text-image pairs. To address this, we propose a novel approach
which harnesses the power of large, pretrained 2D diffusion models. More
specifically, our approach, HexaGen3D, fine-tunes a pretrained text-to-image
model to jointly predict 6 orthographic projections and the corresponding
latent triplane. We then decode these latents to generate a textured mesh.
HexaGen3D does not require per-sample optimization, and can infer high-quality
and diverse objects from textual prompts in 7 seconds, offering significantly
better quality-to-latency trade-offs when comparing to existing approaches.
Furthermore, HexaGen3D demonstrates strong generalization to new objects or
compositions.