DreamTuner: Single Image is Enough for Subject-Driven Generation
Authors: name: Miao Hua; name: Jiawei Liu; name: Fei Ding; name: Wei Liu; name: Jie Wu; name: Qian He
Published: 2023-12-21 09:37:14

Abstract: Diffusion-based models have demonstrated impressive capabilities for
text-to-image generation and are expected for personalized applications of
subject-driven generation, which require the generation of customized concepts
with one or a few reference images. However, existing methods based on
fine-tuning fail to balance the trade-off between subject learning and the
maintenance of the generation capabilities of pretrained models. Moreover,
other methods that utilize additional image encoders tend to lose important
details of the subject due to encoding compression. To address these
challenges, we propose DreamTurner, a novel method that injects reference
information from coarse to fine to achieve subject-driven image generation more
effectively. DreamTurner introduces a subject-encoder for coarse subject
identity preservation, where the compressed general subject features are
introduced through an attention layer before visual-text cross-attention. We
then modify the self-attention layers within pretrained text-to-image models to
self-subject-attention layers to refine the details of the target subject. The
generated image queries detailed features from both the reference image and
itself in self-subject-attention. It is worth emphasizing that
self-subject-attention is an effective, elegant, and training-free method for
maintaining the detailed features of customized subjects and can serve as a
plug-and-play solution during inference. Finally, with additional
subject-driven fine-tuning, DreamTurner achieves remarkable performance in
subject-driven image generation, which can be controlled by a text or other
conditions such as pose. For further details, please visit the project page at
https://dreamtuner-diffusion.github.io/.