PartCraft: Crafting Creative Objects by Parts
Authors: name: Kam Woh Ng; name: Xiatian Zhu; name: Yi-Zhe Song; name: Tao Xiang
Published: 2024-07-05 15:53:04

Abstract: This paper propels creative control in generative visual AI by allowing users
to "select". Departing from traditional text or sketch-based methods, we for
the first time allow users to choose visual concepts by parts for their
creative endeavors. The outcome is fine-grained generation that precisely
captures selected visual concepts, ensuring a holistically faithful and
plausible result. To achieve this, we first parse objects into parts through
unsupervised feature clustering. Then, we encode parts into text tokens and
introduce an entropy-based normalized attention loss that operates on them.
This loss design enables our model to learn generic prior topology knowledge
about object's part composition, and further generalize to novel part
compositions to ensure the generation looks holistically faithful. Lastly, we
employ a bottleneck encoder to project the part tokens. This not only enhances
fidelity but also accelerates learning, by leveraging shared knowledge and
facilitating information exchange among instances. Visual results in the paper
and supplementary material showcase the compelling power of PartCraft in
crafting highly customized, innovative creations, exemplified by the "charming"
and creative birds. Code is released at https://github.com/kamwoh/partcraft.