RealCompo: Dynamic Equilibrium between Realism and Compositionality
  Improves Text-to-Image Diffusion Models
Authors: name: Xinchen Zhang; name: Ling Yang; name: Yaqi Cai; name: Zhaochen Yu; name: Jiake Xie; name: Ye Tian; name: Minkai Xu; name: Yong Tang; name: Yujiu Yang; name: Bin Cui
Published: 2024-02-20 10:56:52

Abstract: Diffusion models have achieved remarkable advancements in text-to-image
generation. However, existing models still have many difficulties when faced
with multiple-object compositional generation. In this paper, we propose a new
training-free and transferred-friendly text-to-image generation framework,
namely RealCompo, which aims to leverage the advantages of text-to-image and
layout-to-image models to enhance both realism and compositionality of the
generated images. An intuitive and novel balancer is proposed to dynamically
balance the strengths of the two models in denoising process, allowing
plug-and-play use of any model without extra training. Extensive experiments
show that our RealCompo consistently outperforms state-of-the-art text-to-image
models and layout-to-image models in multiple-object compositional generation
while keeping satisfactory realism and compositionality of the generated
images. Code is available at https://github.com/YangLing0818/RealCompo