InstantStyle-Plus: Style Transfer with Content-Preserving in
  Text-to-Image Generation
Authors: name: Haofan Wang; name: Peng Xing; name: Renyuan Huang; name: Hao Ai; name: Qixun Wang; name: Xu Bai
Published: 2024-06-30 18:05:33

Abstract: Style transfer is an inventive process designed to create an image that
maintains the essence of the original while embracing the visual style of
another. Although diffusion models have demonstrated impressive generative
power in personalized subject-driven or style-driven applications, existing
state-of-the-art methods still encounter difficulties in achieving a seamless
balance between content preservation and style enhancement. For example,
amplifying the style's influence can often undermine the structural integrity
of the content. To address these challenges, we deconstruct the style transfer
task into three core elements: 1) Style, focusing on the image's aesthetic
characteristics; 2) Spatial Structure, concerning the geometric arrangement and
composition of visual elements; and 3) Semantic Content, which captures the
conceptual meaning of the image. Guided by these principles, we introduce
InstantStyle-Plus, an approach that prioritizes the integrity of the original
content while seamlessly integrating the target style. Specifically, our method
accomplishes style injection through an efficient, lightweight process,
utilizing the cutting-edge InstantStyle framework. To reinforce the content
preservation, we initiate the process with an inverted content latent noise and
a versatile plug-and-play tile ControlNet for preserving the original image's
intrinsic layout. We also incorporate a global semantic adapter to enhance the
semantic content's fidelity. To safeguard against the dilution of style
information, a style extractor is employed as discriminator for providing
supplementary style guidance. Codes will be available at
https://github.com/instantX-research/InstantStyle-Plus.