FRAP: Faithful and Realistic Text-to-Image Generation with Adaptive
  Prompt Weighting
Authors: name: Liyao Jiang; name: Negar Hassanpour; name: Mohammad Salameh; name: Mohan Sai Singamsetti; name: Fengyu Sun; name: Wei Lu; name: Di Niu
Published: 2024-08-21 15:30:35

Abstract: Text-to-image (T2I) diffusion models have demonstrated impressive
capabilities in generating high-quality images given a text prompt. However,
ensuring the prompt-image alignment remains a considerable challenge, i.e.,
generating images that faithfully align with the prompt's semantics. Recent
works attempt to improve the faithfulness by optimizing the latent code, which
potentially could cause the latent code to go out-of-distribution and thus
produce unrealistic images. In this paper, we propose FRAP, a simple, yet
effective approach based on adaptively adjusting the per-token prompt weights
to improve prompt-image alignment and authenticity of the generated images. We
design an online algorithm to adaptively update each token's weight
coefficient, which is achieved by minimizing a unified objective function that
encourages object presence and the binding of object-modifier pairs. Through
extensive evaluations, we show FRAP generates images with significantly higher
prompt-image alignment to prompts from complex datasets, while having a lower
average latency compared to recent latent code optimization methods, e.g., 4
seconds faster than D&B on the COCO-Subject dataset. Furthermore, through
visual comparisons and evaluation on the CLIP-IQA-Real metric, we show that
FRAP not only improves prompt-image alignment but also generates more authentic
images with realistic appearances. We also explore combining FRAP with prompt
rewriting LLM to recover their degraded prompt-image alignment, where we
observe improvements in both prompt-image alignment and image quality.