Iterative Object Count Optimization for Text-to-image Diffusion Models
Authors: name: Oz Zafar; name: Lior Wolf; name: Idan Schwartz
Published: 2024-08-21 15:51:46

Abstract: We address a persistent challenge in text-to-image models: accurately
generating a specified number of objects. Current models, which learn from
image-text pairs, inherently struggle with counting, as training data cannot
depict every possible number of objects for any given object. To solve this, we
propose optimizing the generated image based on a counting loss derived from a
counting model that aggregates an object\'s potential. Employing an
out-of-the-box counting model is challenging for two reasons: first, the model
requires a scaling hyperparameter for the potential aggregation that varies
depending on the viewpoint of the objects, and second, classifier guidance
techniques require modified models that operate on noisy intermediate diffusion
steps. To address these challenges, we propose an iterated online training mode
that improves the accuracy of inferred images while altering the text
conditioning embedding and dynamically adjusting hyperparameters. Our method
offers three key advantages: (i) it can consider non-derivable counting
techniques based on detection models, (ii) it is a zero-shot plug-and-play
solution facilitating rapid changes to the counting techniques and image
generation methods, and (iii) the optimized counting token can be reused to
generate accurate images without additional optimization. We evaluate the
generation of various objects and show significant improvements in accuracy.
The project page is available at https://ozzafar.github.io/count_token.