VMix: Improving Text-to-Image Diffusion Model with Cross-Attention
  Mixing Control
Authors: name: Shaojin Wu; name: Fei Ding; name: Mengqi Huang; name: Wei Liu; name: Qian He
Published: 2024-12-30 08:47:25

Abstract: While diffusion models show extraordinary talents in text-to-image
generation, they may still fail to generate highly aesthetic images. More
specifically, there is still a gap between the generated images and the
real-world aesthetic images in finer-grained dimensions including color,
lighting, composition, etc. In this paper, we propose Cross-Attention Value
Mixing Control (VMix) Adapter, a plug-and-play aesthetics adapter, to upgrade
the quality of generated images while maintaining generality across visual
concepts by (1) disentangling the input text prompt into the content
description and aesthetic description by the initialization of aesthetic
embedding, and (2) integrating aesthetic conditions into the denoising process
through value-mixed cross-attention, with the network connected by
zero-initialized linear layers. Our key insight is to enhance the aesthetic
presentation of existing diffusion models by designing a superior condition
control method, all while preserving the image-text alignment. Through our
meticulous design, VMix is flexible enough to be applied to community models
for better visual performance without retraining. To validate the effectiveness
of our method, we conducted extensive experiments, showing that VMix
outperforms other state-of-the-art methods and is compatible with other
community modules (e.g., LoRA, ControlNet, and IPAdapter) for image generation.
The project page is https://vmix-diffusion.github.io/VMix/.