ZePo: Zero-Shot Portrait Stylization with Faster Sampling
Authors: name: Jin Liu; name: Huaibo Huang; name: Jie Cao; name: Ran He
Published: 2024-08-10 08:53:41

Abstract: Diffusion-based text-to-image generation models have significantly advanced
the field of art content synthesis. However, current portrait stylization
methods generally require either model fine-tuning based on examples or the
employment of DDIM Inversion to revert images to noise space, both of which
substantially decelerate the image generation process. To overcome these
limitations, this paper presents an inversion-free portrait stylization
framework based on diffusion models that accomplishes content and style feature
fusion in merely four sampling steps. We observed that Latent Consistency
Models employing consistency distillation can effectively extract
representative Consistency Features from noisy images. To blend the Consistency
Features extracted from both content and style images, we introduce a Style
Enhancement Attention Control technique that meticulously merges content and
style features within the attention space of the target image. Moreover, we
propose a feature merging strategy to amalgamate redundant features in
Consistency Features, thereby reducing the computational load of attention
control. Extensive experiments have validated the effectiveness of our proposed
framework in enhancing stylization efficiency and fidelity. The code is
available at https://github.com/liujin112/ZePo.