Tailor3D: Customized 3D Assets Editing and Generation with Dual-Side
  Images
Authors: name: Zhangyang Qi; name: Yunhan Yang; name: Mengchen Zhang; name: Long Xing; name: Xiaoyang Wu; name: Tong Wu; name: Dahua Lin; name: Xihui Liu; name: Jiaqi Wang; name: Hengshuang Zhao
Published: 2024-07-08 17:59:55

Abstract: Recent advances in 3D AIGC have shown promise in directly creating 3D objects
from text and images, offering significant cost savings in animation and
product design. However, detailed edit and customization of 3D assets remains a
long-standing challenge. Specifically, 3D Generation methods lack the ability
to follow finely detailed instructions as precisely as their 2D image creation
counterparts. Imagine you can get a toy through 3D AIGC but with undesired
accessories and dressing. To tackle this challenge, we propose a novel pipeline
called Tailor3D, which swiftly creates customized 3D assets from editable
dual-side images. We aim to emulate a tailor's ability to locally change
objects or perform overall style transfer. Unlike creating 3D assets from
multiple views, using dual-side images eliminates conflicts on overlapping
areas that occur when editing individual views. Specifically, it begins by
editing the front view, then generates the back view of the object through
multi-view diffusion. Afterward, it proceeds to edit the back views. Finally, a
Dual-sided LRM is proposed to seamlessly stitch together the front and back 3D
features, akin to a tailor sewing together the front and back of a garment. The
Dual-sided LRM rectifies imperfect consistencies between the front and back
views, enhancing editing capabilities and reducing memory burdens while
seamlessly integrating them into a unified 3D representation with the LoRA
Triplane Transformer. Experimental results demonstrate Tailor3D's effectiveness
across various 3D generation and editing tasks, including 3D generative fill
and style transfer. It provides a user-friendly, efficient solution for editing
3D assets, with each editing step taking only seconds to complete.