Garment3DGen: 3D Garment Stylization and Texture Generation
Authors: name: Nikolaos Sarafianos; name: Tuur Stuyck; name: Xiaoyu Xiang; name: Yilei Li; name: Jovan Popovic; name: Rakesh Ranjan
Published: 2024-03-27 17:59:33

Abstract: We introduce Garment3DGen a new method to synthesize 3D garment assets from a
base mesh given a single input image as guidance. Our proposed approach allows
users to generate 3D textured clothes based on both real and synthetic images,
such as those generated by text prompts. The generated assets can be directly
draped and simulated on human bodies. First, we leverage the recent progress of
image to 3D diffusion methods to generate 3D garment geometries. However, since
these geometries cannot be utilized directly for downstream tasks, we propose
to use them as pseudo ground-truth and set up a mesh deformation optimization
procedure that deforms a base template mesh to match the generated 3D target.
Second, we introduce carefully designed losses that allow the input base mesh
to freely deform towards the desired target, yet preserve mesh quality and
topology such that they can be simulated. Finally, a texture estimation module
generates high-fidelity texture maps that are globally and locally consistent
and faithfully capture the input guidance, allowing us to render the generated
3D assets. With Garment3DGen users can generate the textured 3D garment of
their choice without the need of artist intervention. One can provide a textual
prompt describing the garment they desire to generate a simulation-ready 3D
asset. We present a plethora of quantitative and qualitative comparisons on
various assets both real and generated and provide use-cases of how one can
generate simulation-ready 3D garments.