TEXGen: a Generative Diffusion Model for Mesh Textures
Authors: name: Xin Yu; name: Ze Yuan; name: Yuan-Chen Guo; name: Ying-Tian Liu; name: JianHui Liu; name: Yangguang Li; name: Yan-Pei Cao; name: Ding Liang; name: Xiaojuan Qi
Published: 2024-11-22 05:22:11

Abstract: While high-quality texture maps are essential for realistic 3D asset
rendering, few studies have explored learning directly in the texture space,
especially on large-scale datasets. In this work, we depart from the
conventional approach of relying on pre-trained 2D diffusion models for
test-time optimization of 3D textures. Instead, we focus on the fundamental
problem of learning in the UV texture space itself. For the first time, we
train a large diffusion model capable of directly generating high-resolution
texture maps in a feed-forward manner. To facilitate efficient learning in
high-resolution UV spaces, we propose a scalable network architecture that
interleaves convolutions on UV maps with attention layers on point clouds.
Leveraging this architectural design, we train a 700 million parameter
diffusion model that can generate UV texture maps guided by text prompts and
single-view images. Once trained, our model naturally supports various extended
applications, including text-guided texture inpainting, sparse-view texture
completion, and text-driven texture synthesis. Project page is at
http://cvmi-lab.github.io/TEXGen/.