IMAGINE-E: Image Generation Intelligence Evaluation of State-of-the-art
  Text-to-Image Models
Authors: name: Jiayi Lei; name: Renrui Zhang; name: Xiangfei Hu; name: Weifeng Lin; name: Zhen Li; name: Wenjian Sun; name: Ruoyi Du; name: Le Zhuo; name: Zhongyu Li; name: Xinyue Li; name: Shitian Zhao; name: Ziyu Guo; name: Yiting Lu; name: Peng Gao; name: Hongsheng Li
Published: 2025-01-23 18:58:33

Abstract: With the rapid development of diffusion models, text-to-image(T2I) models
have made significant progress, showcasing impressive abilities in prompt
following and image generation. Recently launched models such as FLUX.1 and
Ideogram2.0, along with others like Dall-E3 and Stable Diffusion 3, have
demonstrated exceptional performance across various complex tasks, raising
questions about whether T2I models are moving towards general-purpose
applicability. Beyond traditional image generation, these models exhibit
capabilities across a range of fields, including controllable generation, image
editing, video, audio, 3D, and motion generation, as well as computer vision
tasks like semantic segmentation and depth estimation. However, current
evaluation frameworks are insufficient to comprehensively assess these models'
performance across expanding domains. To thoroughly evaluate these models, we
developed the IMAGINE-E and tested six prominent models: FLUX.1, Ideogram2.0,
Midjourney, Dall-E3, Stable Diffusion 3, and Jimeng. Our evaluation is divided
into five key domains: structured output generation, realism, and physical
consistency, specific domain generation, challenging scenario generation, and
multi-style creation tasks. This comprehensive assessment highlights each
model's strengths and limitations, particularly the outstanding performance of
FLUX.1 and Ideogram2.0 in structured and specific domain tasks, underscoring
the expanding applications and potential of T2I models as foundational AI
tools. This study provides valuable insights into the current state and future
trajectory of T2I models as they evolve towards general-purpose usability.
Evaluation scripts will be released at https://github.com/jylei16/Imagine-e.