InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image
  Generation
Authors: name: Haofan Wang; name: Qixun Wang; name: Xu Bai; name: Zekui Qin; name: Anthony Chen
Published: 2024-04-03 13:34:09

Abstract: Tuning-free diffusion-based models have demonstrated significant potential in
the realm of image personalization and customization. However, despite this
notable progress, current models continue to grapple with several complex
challenges in producing style-consistent image generation. Firstly, the concept
of style is inherently underdetermined, encompassing a multitude of elements
such as color, material, atmosphere, design, and structure, among others.
Secondly, inversion-based methods are prone to style degradation, often
resulting in the loss of fine-grained details. Lastly, adapter-based approaches
frequently require meticulous weight tuning for each reference image to achieve
a balance between style intensity and text controllability. In this paper, we
commence by examining several compelling yet frequently overlooked
observations. We then proceed to introduce InstantStyle, a framework designed
to address these issues through the implementation of two key strategies: 1) A
straightforward mechanism that decouples style and content from reference
images within the feature space, predicated on the assumption that features
within the same space can be either added to or subtracted from one another. 2)
The injection of reference image features exclusively into style-specific
blocks, thereby preventing style leaks and eschewing the need for cumbersome
weight tuning, which often characterizes more parameter-heavy designs.Our work
demonstrates superior visual stylization outcomes, striking an optimal balance
between the intensity of style and the controllability of textual elements. Our
codes will be available at https://github.com/InstantStyle/InstantStyle.