Compositional Text-to-Image Generation with Dense Blob Representations
Authors: name: Weili Nie; name: Sifei Liu; name: Morteza Mardani; name: Chao Liu; name: Benjamin Eckart; name: Arash Vahdat
Published: 2024-05-14 00:22:06

Abstract: Existing text-to-image models struggle to follow complex text prompts,
raising the need for extra grounding inputs for better controllability. In this
work, we propose to decompose a scene into visual primitives - denoted as dense
blob representations - that contain fine-grained details of the scene while
being modular, human-interpretable, and easy-to-construct. Based on blob
representations, we develop a blob-grounded text-to-image diffusion model,
termed BlobGEN, for compositional generation. Particularly, we introduce a new
masked cross-attention module to disentangle the fusion between blob
representations and visual features. To leverage the compositionality of large
language models (LLMs), we introduce a new in-context learning approach to
generate blob representations from text prompts. Our extensive experiments show
that BlobGEN achieves superior zero-shot generation quality and better
layout-guided controllability on MS-COCO. When augmented by LLMs, our method
exhibits superior numerical and spatial correctness on compositional image
generation benchmarks. Project page: https://blobgen-2d.github.io.