ConsisLoRA: Enhancing Content and Style Consistency for LoRA-based Style
  Transfer
Authors: name: Bolin Chen, name: Baoquan Zhao, name: Haoran Xie, name: Yi Cai, name: Qing Li, name: Xudong Mao

Style transfer involves transferring the style from a reference image to the
content of a target image. Recent advancements in LoRA-based (Low-Rank
Adaptation) methods have shown promise in effectively capturing the style of a
single image. However, these approaches still face significant challenges such
as content inconsistency, style misalignment, and content leakage. In this
paper, we comprehensively analyze the limitations of the standard diffusion
parameterization, which learns to predict noise, in the context of style
transfer. To address these issues, we introduce ConsisLoRA, a LoRA-based method
that enhances both content and style consistency by optimizing the LoRA weights
to predict the original image rather than noise. We also propose a two-step
training strategy that decouples the learning of content and style from the
reference image. To effectively capture both the global structure and local
details of the content image, we introduce a stepwise loss transition strategy.
Additionally, we present an inference guidance method that enables continuous
control over content and style strengths during inference. Through both
qualitative and quantitative evaluations, our method demonstrates significant
improvements in content and style consistency while effectively reducing
content leakage.