Constantly Improving Image Models Need Constantly Improving Benchmarks
Authors: name: Jiaxin Ge; name: Grace Luo; name: Heekyung Lee; name: Nishant Malpani; name: Long Lian; name: XuDong Wang; name: Aleksander Holynski; name: Trevor Darrell; name: Sewon Min; name: David M. Chan
Published: 2025-10-16 17:59:30

Abstract: Recent advances in image generation, often driven by proprietary systems like
GPT-4o Image Gen, regularly introduce new capabilities that reshape how users
interact with these models. Existing benchmarks often lag behind and fail to
capture these emerging use cases, leaving a gap between community perceptions
of progress and formal evaluation. To address this, we present ECHO, a
framework for constructing benchmarks directly from real-world evidence of
model use: social media posts that showcase novel prompts and qualitative user
judgments. Applying this framework to GPT-4o Image Gen, we construct a dataset
of over 31,000 prompts curated from such posts. Our analysis shows that ECHO
(1) discovers creative and complex tasks absent from existing benchmarks, such
as re-rendering product labels across languages or generating receipts with
specified totals, (2) more clearly distinguishes state-of-the-art models from
alternatives, and (3) surfaces community feedback that we use to inform the
design of metrics for model quality (e.g., measuring observed shifts in color,
identity, and structure). Our website is at https://echo-bench.github.io.