Make It Count: Text-to-Image Generation with an Accurate Number of
  Objects
Authors: name: Lital Binyamin; name: Yoad Tewel; name: Hilit Segev; name: Eran Hirsch; name: Royi Rassin; name: Gal Chechik
Published: 2024-06-14 17:46:08

Abstract: Despite the unprecedented success of text-to-image diffusion models,
controlling the number of depicted objects using text is surprisingly hard.
This is important for various applications from technical documents, to
children's books to illustrating cooking recipes. Generating object-correct
counts is fundamentally challenging because the generative model needs to keep
a sense of separate identity for every instance of the object, even if several
objects look identical or overlap, and then carry out a global computation
implicitly during generation. It is still unknown if such representations
exist. To address count-correct generation, we first identify features within
the diffusion model that can carry the object identity information. We then use
them to separate and count instances of objects during the denoising process
and detect over-generation and under-generation. We fix the latter by training
a model that predicts both the shape and location of a missing object, based on
the layout of existing ones, and show how it can be used to guide denoising
with correct object count. Our approach, CountGen, does not depend on external
source to determine object layout, but rather uses the prior from the diffusion
model itself, creating prompt-dependent and seed-dependent layouts. Evaluated
on two benchmark datasets, we find that CountGen strongly outperforms the
count-accuracy of existing baselines.