Constantly Improving Image Models Need Constantly Improving Benchmarks
Authors: Jiaxin Ge
Grace Luo
Heekyung Lee
Nishant Malpani
Long Lian
XuDong Wang
Aleksander Holynski
Trevor Darrell
Sewon Min
David M. Chan
Venue: ICLR 2026 Poster (2026)
Keywords: unified model
image generation benchmark
native multimodal model
Primary area: datasets and benchmarks

Abstract: Recent advances in image generation, often driven by proprietary systems like GPT-4o Image Gen, regularly introduce new capabilities that reshape how users interact with these models. Existing benchmarks often lag behind and fail to capture these emerging use cases, leaving a gap between community perceptions of progress and formal evaluation. To address this, we present ECHO, a framework for constructing benchmarks directly from real-world evidence of model use: social media posts that showcase novel prompts and qualitative user judgments. Applying this framework to GPT-4o Image Gen, we construct a dataset of over 35,000 prompts curated from such posts. Our analysis shows that ECHO (1) discovers creative and complex tasks absent from existing benchmarks, such as re-rendering product labels across languages or generating receipts with specified totals, (2) more clearly distinguishes state-of-the-art models from alternatives, and
(3) surfaces community feedback that we use to inform the design of metrics for model quality (e.g., measuring observed shifts in color, identity, and structure). Our website is at https://echo-bench.github.io.

CONSTANTLY IMPROVING IMAGE MODELS NEED CONSTANTLY IMPROVING BENCHMARKS
Jiaxin Ge
Grace Luo'
Heekyung Lee'
Nishant Malpani'
Long Lian'
XuDong Wang'
Aleksander Holynski'
Trevor Darrell'
Sewon Min'
David M. Chan'
UC Berkeley
ABSTRACT
Recent advances in image generation, often driven by proprietary systems like GPT-4o Image Gen, regularly introduce new capabílities that reshape how users interact with these models. Existing benchmarks often lag behind and fail to capture these emerging use cases, leaving a gap between community perceptions of progress and formal evaluation. To address this, we present ECHO, a framework for constructing benchmarks directly from real-world evidence of model use: social media posts that showcase novel prompts and qualitative user judgments. Applying this framework to GPT-4o Image Gen, we construct a dataset of over 31,000 prompts curated from such posts. Our analysis shows that ECHO (1) discovers creative and complex tasks absent from existing benchmarks, such as re-rendering product labels across languages or generating receipts with specified totals, (2) more clearly distinguishes -hth-att models from alternatives, and (3) surfaces community feedback that we use to inform the design of metrics for model quality (e.g.. measuring observed shifts in color, identity, and structure). Our website is at https: : htt////cobberteee gi tuub. io.
INTRODUCTION
When new generative image models released, users often find new and unanticipated capabilities not captured by existing benchmarks. These capabilities are discussed on social media, where users document their interactions with new models and qualitatively discuss their performance. The release of GPT-4o Image Gen (OpenAI, 2025a) exemplified this behavior with the introduction of "Ghiblification." the style- transfer task of turning natural image into a cartoon version emulating particular animated studio. This new "task" was not only shared widely on social media, but used as a personal measure of model quality by many members the online community. As of today, explicit benchmarks have now been developed for this task (Jiang et al., 2025), but the benchmarks that we traditionally use to evaluate models do not have the capability to evolve with community feedback, and instead, must react to changes in a delayed cycle.
Indeed, despite significant changes in what constitutes "good" image generation model, current popular crowdsourced text-to-image benchmarks (Wang et al., 2022; Kirstain et al., 2023) are often still tailored towards older models such as Stable Diffusion (Rombach et al., 2022), with extensive art-centric keyword lists that are not representative of now- now-feasible use cases. Popular image editing benchmarks (Zhang et al., 2023a;b; Liu et al., 2025) contain overly simple instructions. These instructions were challenging at their inception but do not actually require complex language understanding or reasoning. Furthermore, these tasks already be solved by many models, benchmark scores. As we see in Figure 1b, human ratings indicate that 4o Image Gen is substantially better than the current best open-source unified model (Deng et al., 2025), yet even when benchmarking on a recent image editing benchmark (Liu et al., 2025), the gap appears less significant.
and old. This slow adaptation rate is reflected in model
With the rapid releases of new image generation models, each revealing a range of new capabilities be tested, it has become clear that we need more responsive mechanisms for adapting benchmarks to emergent user observations. In this work we present ECHO: Extracting Community Hatched Observations, a re-usable framework that converts community discussion on social media into a structured benchmark. Our proposed
'Equal contribution. Conresonndence gej i axii n@berkel edu grace uo@berke leeeerreeeeeeuuu

---

(a) Examples from Our Benchmark
(b) Benchmark Comparison
"Drow giant alligator with mouth that the shope of the lines in chart."
"blend in these colours to generate gradient background."
4o Image Gen Bagel
1.9
26-
Relative Performance
0.6-
"Using grid provided, images for each style for the nail polish product"
"Generote nearly identical version of this image, but with texts translated to Portuguese:
2.4
3G2
3.5 GEdit
Ours
Figure ECHO distills collective discussion about new generative model into a structured benchmark. As case study, we apply ECHO to GPT-4o Image Gen (OpenAI, 2025a) on Twitter/X. Left: ECHO automatically surfaces highly diverse and novel tasks not covered in prior benchmarks. Right: Consequently, our image-to-image split shows 3.2x larger relative performance gap compared to prior image editing benchmark, GEdit (Liu et al., 2025).
method bypasses the traditional "observation benchmark" cycle, and provides us with framework for ai lomaticully converting real-world ideas and capabilities surfaced by users on social media directly to metrics that we can use measure and improve SOTA models. The ECHO framework operates by searching social media for mentions of a target model and automatically filtering for coherent image generation prompts specified via text and/or images, while extracting community insights and feedback on particular prompt ca- pabilities. It is designed to address number of common challenges associated with social media, including the tradeoff between post volume and relevance, the splitting of context across posts, and noisy formatting.
Using ECHO, we are able to surface and formalize number of qualitative observations related to the most recent image generation methods. By running our framework on the 4o Image Gen release, we introduce new dataset containing more than 31,000 user-sourced prompts which: (I) surfaces creative and complex tasks absent from existing benchmarks, (2) more diverse and more closely resembles natural user language (contains 2.3x more unique first bigrams and are 1.2x lower in LLM perplexity), (3) better separates state- of-the-art models from prior models, and (4) automatically surfaces several new quantifiable indicators for image generation quality, including identity preservation and color shift, which show be operationalized into secondary evaluation metrics that could inform future model losses and development.
BACKGROUND & RELATED WORK
Model benchmarks often mirror the capabilities the models themselves, and are designed by model developers in order evaluate and understand how these models perform on tasks of interest. For example, traditional text-to-image benchmarks (Huang et al., 2023; Ghosh et al., 2023; Lee et al., 2023) and image-to-image benchmarks (Brooks al., 2023; Wang al., 2023; Sheynin al., 2024; Hui et al., 2024; Zhang al., 2023a) are not collected in-the-wild. These benchmarks contain short and overly simple instructions such "A in front of chair" "Add fireworks the sky" that fail to reflect real user intent, but provide strong diagnostic signal for understanding simple generative understanding.
On the other hand, community-drin -drive benchmarks are often designed to collect real user prompts, and more closely mirror what downstream user might desire from a model. For example, previous methods (Wang et al., 2022; Xu et al., 2023; Kirstain et al., 2023) have collected real user prompts of Stable Diffusion models from an explicit interface (Rombach et al., 2022). These benchmarks require significant amount of human intervention to decide which user prompts to model, and how to model them. In addition, the model interface itself can lead to two further limitations: (i) prompt intent is bounded by the capabilities of the model itself, and (ii) prompt style is tailored towards the model rather than natural user language. For example, it has already been demonstrated that users will adjust their prompting behavior to account
2

---

1. Collect large volume of relevant posts
2. Reconstruct context with post trees
Process muitimodal data non-standard formats
Finalize samples
Ini Claesity
Taaat (R.3)) Sadguree (AUAE
30k
(b) Fi-in the-flant
Query:
Post:
Sample::
peopet hake stplas
Benchmark
<chatgot>,
"ChetaPT gives you palshed commercial
"Give me
Convenation
images, cimage gens
colished
2il.. [R2, 4a, will
1.8k
commerciol
visugi-ready publish"
visudl
ready to publish."
Figure 2: ECHO Framework. ECHO is motivated by several challenges inherent to social media. (1) We start with broad queries followed by relevance filtering, since basic querying presents volume-relevance tradeoff. (2) We then extract prompts from these posts, making sure to utilize the full post tree, as context can be spread across posts. (3) We then apply multimodal processing, since useful data also exists in non-standard formats. (4) Finally, we reserve the highest quality data benchmarking, while the rest used for analysis.
for limitations of the CLIP (Radford et al., 2021) text encoder, which behaves more like a bag-of-words representation, where they use extensive sets of "phrases rather than complete sentences" (ComfyUI Wiki, 2025), with prompts like "colorful framework draws on prompts crafted for human audiences on social media, where the goal is to showcase creativity rather than to optimize around model quirks. While prompts inevitably reflect the capabilities of the current best models, our framework is re-runnable and can adapt models and user behaviors evolve, reducing the risk of per-model biases.
salaxies.space,arssse
Unlike interface-collected datasets, our
GEdit (Liu et al., 2025) proposed scraping the internet for real image editing prompts. However, these prompts are limited by the imagination of the authors, leading to a restricted set of 11 specific single image editing tasks, such as changing the color or changing the background. Most closely related to our work, IntelligentBench (Deng al., 2025) and KontexiBench (Batifol et al., 2025) were designed to highlight the capabilities of new models released by the same authors. However, details about their data source and creation method are largely unknown, and neither benchmark is publicly available.
Outside of image generation, Chatbot Arena (Chiang et al., 2024) uses an online platform to collect use cases in the wild, incentivizing users to provide data by providing free platform for interacting with the model. While such process does collect real user prompts, unlike this approach, we investigate social media, which represents notably different, prompt distribution: since engagement, they are more incentivized to produce novel and creative examples, rather than tasks that are already rell-within model capabilities.
seeking reciprocal
3 CROWDSOURCING A BENCHMARK
Our primary goal is to distill collective discussion about a new generative model into a structured dataset. Such discussion often involves users sharing interesting prompts and outputs, novel task ideas, or commentary model behavior. We aim capture all of these cases, in a standardized format:
<input text, input image (s) +, cutput image, community feedback+>
where denotes optional fields; the full set of data we collect is given in Table E.1. However, this objective poses several challenges:
Collection: A large volume of relevant data desired, which requires identifying the right platform and gathering the data.
Processing: A non-trivial amount of processing is required, e.g., the input prompt and images may be embedded a single screenshot or the prompt may not be written explicitly.
Filtering: Data quality varies widely, e.g., a user may provide more general commentary or exactly document their input prompt.

---

We propose framework, ECHO, that addresses these challenges, illustrated in Figure 2. Our framework first collects relevant posts (Section 3.1), converts posts into self-contained samples (Section 3.2), and finally expands coverage via multimodal processing (Section 3.3).
3.1 IDENTIFYING RELEVANT POSTS
There is inherent tradeoff between the volume of posts and their relevance. When querying with broader keywords, the average post relevance goes down, and with narrower ones, the available post pool is quickly exhausted. To address this, we implement a two-stage pipeline where first query for large volume of posts then use an LLM to filter irrelevant ones.
Designing Keywords. First, our goal is to maximize the post pool. However, we found two issues: (I) LLM-based filtering is expensive, so the pool cannot be too large, and (2) there is a temporal shift in which keywords lead to relevant posts (e.g.. the initial two weeks of the 4o Image Gen release, generic terms like "openai" often retrieve relevant posts, but later on relevancy decreases). Therefore, we use two sets of keywords to query posts within vs. outside the first two weeks of the 4o Image Gen release (see Figure E.1).
Classifying Relevance. We then use an LLM to classify the post text on 5-point relevance scale (see Figure F.2). We initially collected 68k posts in total, of which 47% passed our relevance filter as 'very likely relevant" or "certainly relevant." Nearly half of collected posts pass this filter, amounting to 32k posts, indicating that our query design is fairly efficient and has high yield rate.
3.2 RECONSTRUCTING CONTEXT ACROSS POSTS
Posts can be context dependent. For example, a user may write "prompt below" " in the first post then include the actual prompt text in reply. We want self-contained samples, characterized prompt some user tried, community feedback towards that prompt and its resulting outputs, and a label for its quality. To achieve this, framework attempts collect much of the reply tree possible, then use this full context when processing posts into samples.
unique
Constructing Reply Trees. For each post obtained via keyword query, extract the full reply tree, URLs pointing to the parent post or child replies. We then recursively expand the dataset by querying these discovered posts and traversing their respective reply chains, introducing 19k new posts from the replies. This procedure enables our framework to discover relevant posts that may not otherwise appear with keyword- based queries. After reply collection, each post contains ancestor chain P,= PP.) and direct replies C₁= = [Co.....Cm. }. We then search for the unique reply trees across all collected posts. We iterate through each post, referred to as the "main post" P For every P. E Pt, we attach P:+1 as its sole child, producing the path Po - - Pn P Pmain Each Cj EC₁ becomes child of the main post, giving edges Pimiin + Cz. Since the same posts can appear in multiple trees, we remove duplicates via URL and recursively union their children.
Extracting Self-contained Samples. We then an LLM to convert trees into samples, as illustrated in Figure E.3. This processing step first identifies the spans of text corresponding to the prompt and discards any unrelated remarks, and performs minor fixups such as combining disjoint spans. Next, this step collects any commentary either from the original author or other user replies (c.g., "amazing result," "didn''t 't work." " etc.) as list of community feedback. Finally, the sample is assigned one of three quality labels: "Benchmark" (high quality prompts that are coherent and show clear user intent), "Analysis" (moderate quality partial prompts or commentary that could not be associated with another sample), or "Trash" topic and malformed content that should be discarded). While we want only the highest quality data for benchmarking, we are also interested in retaining any other relevant data for large-scale analysis.
3.3 MULTIMODAL. PROCESSING OF POSTS WITH IMAGES
While Section 3.2 can extract text prompts and community feedback, other metadata requires multimodal processing, This step marks the input and output images associated with each sample, updates prompts with fill-in-the- blank, and produces new samples by parsing screenshots, addressing three common cases:
Classifying Input vs. Output Images. There does not exist a standardized format for marking input and output images. For example, the output image could be the first or the last in series of images, or
4

---

Qualitative Examples
Dataset Statistics
CompBench
'Wenvve the rightmost sebra.
Divendty Comparioon
Image-tc-iin to-i-tt-aaaa
GEeh
"Chonge the bociground o city street."
InstuctPW22X
"Add some geoffit"
MogicBrush
"Chonge the dog for cot."
"Create professional People Megazine-style featuring the NFT character chTcrroctee from the uplooded
refenence image. The character ahould be posed like celebrity, with cramatic lighting and cotchy headlines referenoing their fome ochievements.
Huidbaak
OonEval
parking
Flunny Companasn
"on water enormous huge isiands of tyres and gorbage fooding seoquls Pykeg in the forecusted sky, diomon light post apacalyptic roly weather wetnighly mote pointing concept"
Text-oo-imaa inage
ImageReeward
Prosod, famiy, 7al dish healthy food, minimation, emoj styic, reaitsn, india, pasta colors
"DD hoty
restourant
Oura
"Creats pivel Uluatration depicts sequence of four images of boy jumping greb bottle cokn, drinking Mt fhiishing his drink, and mrowing the bottle into a trash con."
Gorfeo
Figure 3: Dataset Comparison. The prompts in ECHO are significantly different from prior benchmarks. Top: the image-to-image split is more diverse and complex, with more unique first bigrams. Bottom: the text-to-imaae split more fluent, as measured by perplexity under Pythia 12B (Biderman al., 2023).
there may be irrelevant images that are neither inputs nor outputs. Nevertheless, users expect viewers to infer this distinction, and thus we use a VLM to make this same inference (see Figure E.4).
Completing Fill-in-the-Blann Blank. A common user behavior is "fill-n-the-blank'" prompts, where users post template intended for commenters infill in the replies. Keeping these templates as-is presents problem, because they are not fully specified and effectively omit the completions that commenters find most interesting. Instead, we use VLM to everse-engineer these completions conditioned on the template and the images provided by commenters (see Figure E.4).
Extracting Conversation Screenshots. Another behavior is sharing screenshots of interactions with 4o Image Gen, which may contain prompt text, reference images, and image outputs all within the same frame. This is especially high-quality source of data, since the inputs and outputs exactly documented without paraphrasing or summary. Extracting the raw data requires a multi-task computer vision system that can localize images bounding boxes, classify the sub-images inputs outputs, and detect what is prompt text vs. unrelated conversation. While one could chain together specialized models for each subtask, we instead opt for a more generalizable solution using a VLM. We first detect these cases with the general multimodal processing prompt, which is then routed to the parsing prompt depicted in Figure E.5. The VLM can not only parse the 4o Image Gen interface but also other non-standard layouts, for example side-by-side collages of input and output images. For the VLM we opt to use Qwen-2.5-VE 2. (Bai et al., 2025), which specifically trained for bounding box detection.
4 ECHO: A SOCIAL-MEDIA POST-RELEASE BENCHMARK
We initially our framework explore the 4o Image Gen release Twitter/X. After the LLM quality filter in Section 3.2, we find that 20% of samples are marked as high-quality (usable for benchmarking) and 66% are marked moderate-qualitt (usable for analysis). We also apply safety filtering (see more details in Section 6). This yields 30k total samples for analysis. For our final benchmark, we limit each split to up thousand samples, keep the downstream costs of benchmarking (generating outputs and rating them) manageable. We then randomly sample candidates and manually inspect them, flagging low-quality exam- ples for removal or updating their prompts, to ensure that the benchmark is the highest possible quality. This results in image-to- toaggge split with 777 prompt-image pairs and text-to-image split with 1000 prompts.
ECHO Surfaces Diverse and Novel Tasks. While most benchmarks are limited to templated image editing tasks, such as changing the background, changing the color, adding or replacing an object, as shown in Figure 3 (top left), ECHO incorporates several tasks not captured by existing tasks, such novel view synthesis, image editing that requires cognitive reasoning, virtual try-on, template- product generation, multi-image subject-driven generation, colorization, image translation, and code-basee style transfer
5

---

PARs
Nayword Countinn
Keywerd: Aspect Ratio o baeiss by had in tse Generattvvn Fill Photrohop maruully sopand image...
Theezhie cleat retruttions, the consatently grow Not not Precision for Al even simple sourrin!!
Keyword: Color Stuft
NOT produce
10.0
woact copy this image...
"Ahty did image become green hallwing through?
Weywordd Adentity
has lendenccy to gracuusity moxe tonserts luan mush after smough gansrationa"
promptt
"The first but.. does strange things and eoch new Image the chamccer appeoors
style ident details Text TIl endering
Reywondd Originsity
picture tocally orighnal that
Keyword Holluciattinn
*Athhough the wond original isin uocabrueery. it disee'' really ienes fro meatingg only lenvvx
"The akd get fatter and luclattdd branch."
Keyword: Reasening
Coher eereeneeio nceeiia to
Keyword: Rendering
"ivee aaleneiiw, sopiiit prompingg think viilumm..
(24)
"t. ctll has flwws.. text on the hot get messed up
Figure 4: Common Failures Observed by Users. A word cloud of failure cases, derived from community feedback, showing practical capabilities that users care about in real use cases and curiosity-driven tests that reveal deeper model limitations. Common failures include identity shift, color drift, text rendering errors, and style mismatches, more exploratory failures include originality and reasoning about volume.
(see Figure C.3-C.6). We also can see this diversity effect in the language distribution itself. In Figure (top right), we show the unique first-bigrams of each dataset's editing instructions. ECHO also exhibits a substantially larger variety of first bigrams, indicating more diverse instruction types and image operations.
In addition to diversity, ECHO also remains more natural in the language domain. As shown in Figure 3 (bottom right), our instructions exhibit consistently lower perplexity, indicating that they align more closely with natural language, and suggesting that users now prefer to interact with generative models using fluent, coherent instructions (compared to previous keyword-centric methods).
ECHO Surfaces How Users Interact With Models. To capture failure modes that users explicitly care about, first use an LLM to label each piece of community feedhack denoting success failure. Then, for each failure case, the L.M generates a short keyword summary describing the underlying issue (e.g.. failure to render "a transparent helmet" correctly will get the keyword "transparency"). We visualize these keywords as a word cloud and highlight representative cases, as shown in Figure 4.
Figure 4 shows us that users are generally most sensitive failure types such as identity shift, color drift, text rendering errors, style mismatches, and aspect ratio inaccuracy. These failure modes reflect practical use cases where users expect reliability and usefulness, and thus indicate areas where improving models would directly enhance satisfaction. Beyond these common issues, ECHO also surfaces comer-case failures that users found interesting. These often come from probing interesting model behaviors, such reasoning failures in scientific contexts, misunderstandings of concepts such as originality, and difficulty with counting. Such cases reveal deeper limitations of current models and highlight opportunities for future research. The community feedback from ECHO also reveals practical strategies that users employ to work around model limitations. As shown in Figure 5a, users discuss ways to construct valid mazes or mitigate identity mismatches. In this way, ECHO records crowdsourced prompting solutions to certain issues, which also reflects what users care about, and can help to motivate future model development.
Exploratory Behaviors. Interestingly. ECHO also surfaces cases where users explore the model's behavior itself, rather than pursuing concrete task. As shown in Figure 5b, some examples include prompting 4o Image Gen to generate self-portrair (where it refers to itself as DALL-E) or its favorite color (creating speculation about "invisible colors" beyond human vision). These examples illustrate how users collectively probe and reflect on how models behave under novel edge cases, and reveal interesting behaviors that are not captured in standard benchmarks, yet might be interest to model developers.
5 ECHO DIFFERENTIATES MODELS
Given our newly curated in- whl- benchmark, we can now use it to differentiate models. We evaluate three types of models:

---

(a) User Solutions
(b) Exploratory Behaviors
(c) Activity Trends
4500
Problem: "cannot generate volld mozes"
Prompt: "genvorate nome for your..
4000
Solution:
"I got a vald mose eassy osking for ancation.
"Opt-4o Wiee Image penevttion concmilites BALL-E"
3500
Mar
Lay attr do image
DALL-E
Post Volume
3000
Get mooel release
"6c+ BeegogggnncCoo CAN whon the phappe of shombbss Le, 45* recoten equare"
"Maah, aven GPT nome right jectisa!
2500
2000
Apr 16
Day of and
Problem: "not consistent as hopped"
Prompt: "Inage of your fevorile color*
Apr Day ofter Image
Solution:
Behavior:
1000
whisely lack reforsnne enage and use ix e direl sputt, unmmcticall."
the humror con't but the code cord"
300
Apc
Timestamp
Figure 5: How Users Interact with Models. We depict qualitative examples of (a) user solutions and (b) exploratory behaviors, discovered via community feedback. We also visualize (c) activity trends using the timestamps of collected posts.
Unified Models. To capture the open-source community's 's progression, we include early models like Anole (Chern et al., 2024; Team, 2024) and recent models like Bagel (Deng et al., 2025). We also evaluate proprietary models like 4o Image Gen (OpenAI, 2025a), as well as Gemini 2.0 Flash (Comanici al., 2025) and the more recent 2.5 Flash (Nano Banana) (Gemini, 2025).
LLM+DDfffsion. A good baseline for unified models is its most naive implementation: an LLM chained to diffusion model, where the LLM rewrites the input prompt before diffusion image generation. We follow the best-performing method from Zhou et al. (2025), a pipeline with GPT-40 (OpenAI, 2024) the LLM and DALL-E 3 (Betker et al., 2023) as the diffusion model.
Image Editing Models. Another natural baseline is specialized image editing model without sophisticated text encoder. To represent this category, we use Flux Kontext (Batifol et al., 2025), which demonstrates state-of-the-art image editing performance.
Our overall evaluation metric for the benchmark is head-to-head "win rate", relative rather than absolute metric. Given that our benchmark is composed of in-the-wild prompts that are intrinsically open-ended, it is very challenging define notion of "accuracy." The win rate calculated across all (*) pairwise model comparisons, where each model earns for a win, O for a loss, and 0.5 for a tic. Therefore, the final win rate of model can be interpreted as its average win rate compared with all other models.
Automatic Evaluation. Due the cost of collecting human evaluations, we primarily leverage automated evaluation through VLM-assa-judge. We follow the "single answer grading" setup from MTT-Bench (Zheng al., 2023). this setup, a score directly assigned to each output, then converted into "pseudo pairwise" comparisons: for any pair of models, the one with the higher score is treated as the winner. This setup more scalable as the number of models being evaluated increases, and simplifies the benchmarking process. Furthermore, MT-Bench validates that both true pairwise and pseudo pairwise grading show high agreement with human judgements. To mitigate any biases VLM-as-a- judge might have towards models from the same developer, we ensemble the judgements of three evaluators. We use GPT-4o (OpenAl. 1--.255--L-33----mm
2024), Gemini 2.0 (Team et al., 2023), and majority to determine the winner of each model pair. Following MT-Bench, instruct the model to produce a chain-of-thoughe and consider factors like prompt following, fidelity to any reference images, and realism and aesthetics, before producing a score (see Figure F.1).
(Bai et al., 2025), then take the
Human Correlation. secondary validation the automatic evaluator beyond those in Zheng et al. (2023), we compare our automated evaluations against against gold label human annotations. Specifically, we present five expert raters with outputs of all models for 200 samples, and ask the annotators to rank the outputs from best to worst for both the text-to-image and image-to-image splits. We found that our VLM- as-a as-a-judge measure correlates weakly, but significantly. with human ratings (GPT: т, = 00.17p==0.0s6: Gemini: 7, =0.083p=000019. Qwen: 7, =0=000 =0.0455 00.04 13277. While the correlation is positive, and significant for Gemini and GPT, this result suggests that further research into judge models may he necessary for stronger results overall. See Appendix D) for more details.
1327. Human -uman concordance: W = 00.49
7

---

(a) Image- to-age
(b) Text-to Image
1.0
0.81
0.74
0.60
0.53
0.55
0.48
0.15
0.46
D.41
0.40
0.2
0.07
0.10
4a Image Gen
Naro Banana
Cemini 2.0 Flash
Bag Thint Think
Bagel
Flax Kantext
WLM Diffusion
Anole
40 Image Gen
Nand Banana
LLM Difusiin
Gamini 2.0 Flash
Bagel-Think
Bagel
Flux Kontext
Anole
Figure 6: Overall Evaluation. We compare range of unified models, well an image editing (Flux Kontext) and shallow fusion (LLM+Diffusion) baseline. We report the win rate, or percentage of pairwise comparisons won. The win rate is calculated automatically with an ensemble of three aLMssas-a-judee
Input
Anole
LLM Diffusion
Flux Kontext
Begel
Gemini 2.5 Flash
4o Image Gen
Happy Thresty
Error
Create an image with these characters. They are in bar, each with drink in hand, clinking glasses with collective bubble saying 'Hoppy Thirsty Thursday!"
Convert this RGB photo into a white silhouette-stye image with transporent bookground, sultable for use as a vinyl decal or window sticker, Remove all bockground, Render subject in cleon, high- controst outlines simplified, shopes. Preserve key structurai features, but stylize the detaiis Into minimat, vector-style line art. Output as transparent PNG.
Mel:S3-316
Sersutc
DOOL,00
N/A
vessdiiggg
1948
Srik331
27 Col.
Mikdl
10
photorealistic image appears photograph of detailed paper recsipt. receipt should display a total of $127.54 with all numerical values (item prices, toxes, and any additional charges) accurately adding The restaurant No Nome address: Non-Existing Avenue, City, CA 94130
EnlEaiioO EELLIVIS
me!
N/A
sorden goadls thew
ALANOBIVI
EAPT
hv
OOO
E
Gibification of famous physics formulos.
Figure 7: Qualitative Model Comparison. Challenging tasks from our benchmark, ranging from translation multi-concept combination to mathematical reasoning, elicit diverse model responses. We mark samples where the model fails to generate an output as "Error."
5.1 RESULTS
We present the win rate comparison the image-to-image and text-to-image splits in Figure 6. Qualitative results representative models are shown Figure On the image-to-image split (Figure 6a), model performance separates into five distinct tiers. First, 4o Image Gen significantly outperforms the other models, followed by Gemini's Nano Banana. Next, Gemini 2.0 Flash, Bagel-Think, Bagel, and Flux Kontext exhibit similar performance. Finally, LLM+uiffusin then Anole perform much worse. We observe similar trends on the text-to-inne text-to-image split (Figure 6b), although the gaps are less pronounced and M+Dffuuuuiion makes large jump forward in its ranking.
8

---

Onter (ig
Fats Honiity Simbbrity, Auraaaraa
Det Struitare (I)
66 Tist Readering Accemvy
16-
26-
9.23
16-
15-
11.4
3.0
0.5
320
622
AM
520
9.3
past slad
Loke
Ensoed
cover can
Hew
scenr cemill pareeie
Figure 8: Specialized Metrics from Community Feedback. Based on qualitative community observations, we validate that 4o Image Gen exhibits large shifts in color (a) and face identity (b), moderate shifts in structure distance (c), but superior text rendering accuracy (d).
5.2 CLOSING THE LOOP WITH COMMUNITY FEEDBACK METRICS
In addition to the automated evaluations in Section 5, we also wanted to see how community feedback extracted using ECHO could help to differentiate model performance in fine- ane-grained ways. Based on the failure categories extracted by ECHO, and illustrated in Figure 4, we designed several specialized automated metrics: color shift magnitude, face identity similarity, structure distance, and text rendering accuracy. For each metric, described below, we used LLM classify samples where each metric is applicable (Figure F.3), and computed the metric over these samples, with the results presented in Figure 8.
Color Shift Magnitude. We quantify the "yellow tint" frequently reported in community feedback with a color shift metric, computed as the average difference between the color histogram of the input versus output images. As shown in Figure 8a, 4o Image Gen indeed exhibits the largest color shift. Interestingly, the only other method from the same developer. LLM+Diffusion (implemented with DALLE-3), also exhibits abnormally large color shift. Users theorize that the yellow tint could be "waatrmarkinng method, potentially trying to do something kinda fancy with low level pixel encoding."
Face Identity Similarity. Community feedback critiques face identity shifts, which we quantify with a face embedding metric. Specifically, we use AuraFace (Deng et al., 2019; fal, 2025) to detect faces and extract their embeddings, then select the input-output face pair with the highest cosine similarity. Figure 8b confirms user observations that 4o Image Gen struggles with face preservation, which could be attributed lossy visual encoder or insufficient identity-orientee training data.
Structure Distance. Users are perceptive towards drift in visual structure, such as object positioning or human pose, which we measure using DINO- --sedd (Caron al., 2021) structure metric. Following the setup of Tumanyan et al. (2023b), we compute the Frobenius norm of the Gram matrices derived from DINO key features (Tumanyan al., 2023a) for input-output image pairs. As expected, methods not specifically trained on image-to-image data (Lif-uDifuuioon and Anole) perform the worst in structure preservation, as shown Figure Sc. Outside of this category, 4o Image Gen also exhibits non-negligible drift, matching observations that it tends to re-approximate images rather than faithfully copy image structure.
Text Rendering Accuracy. Users are also sensitive towards rendered text, which we measure via VLM-as----udde. Unlike OCR-based string matching, VLMs can produce more holistic score that takes into account factors like legibility in addition to spelling, punctuation, and grammar (see Figure F.2). Figure 8d shows that 4o Image Gen achieves near-perfect text rendering accuracy, consistent with its popularity as a tool for generating infographics and other text-heavy media.
Together, these results show how community feedback be systematically translated into targeted quantitative metrics that expose fine-grained tradeoffs across models. Beyond confirming user observations, this approach produces concrete, interpretable signals that can guide model development.
CONCLUSION
In this work, introduced ECHO, the first framework for evaluating image generation in alignment with emerging, real-world use cases of modern image models. Applied to social media posts about GPT-40 Image Gen, ECHO surfaces novel use cases not captured by prior benchmarks, differentiates proprietary from open-source models, and motivates targeted metrics grounded in common failure cases such as text rendering. As both models and user needs evolve, so too must the benchmarks that guide their development.

---

ETHICS STATEMENT
In this work, we primarily study discussion of 4o Image Gen on Twitter/X, a public social media platform where users voluntarily share content, for academic research purposes. Our collection process implicitly benefits from existing moderation systems: Twitter/X removes posts that violate its content policies (X Help Center, 2025), and ChatGPT refuses generate images that violate its usage guidelines (OpenAI, 2025b). For this reason, the collected posts are relatively benign, as illustrated by qualitative examples from our dataset (see Appendix C). We also take additional steps to remove potentially harmful material. For all samples, we applied LLama- nama-Guard-44111 --22 (Llama Team), multimodal safety classifier designed to safeguard according to the MLCommons hazards taxonomy (Ghosh et al., 2025). We then removed any samples with text or images flagged to contain any of its hazard categories, such as violent, sexual, hateful, or -aiaallii content. To minimize privacy risk, we also manually exclude imput images that plausibly depict anyone under eighteen.
ACKNOWLEDGEMENTS
We thank Stephanie Fu, Michelle Li, and Alexander Pan for their helpful feedback. We also thank the folks Stochastic Labs for previewing early prototypes of this work. Finally, we extend a special thank you to Lisa Dunlap for entertaining many extensive discussions on evaluations.
REFERENCES
Pierre Andrieu, Sarah Cohen-Boulakia, Miguel Couceiro, Alain Denise, and Adeline Pierrot. A unifying rank aggregation framework to suitably and efficiently aggregate any kind of rankings. International Journal of Approximate Reasoning. 162:109035, 2023. ISSN 0888-613X. doi:
URL https:/ http://wwwwwsto www.ssieeeedreet.ccomm com/ acience/artee cllecceeeepoeceee http:ppp//s0000 /00088688111111 3x23001 664.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang. Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng. Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025.
Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhom, Jack English, Zion English, Patrick Esser, Sumith Kulal, et al. Flux. 1 kontext: Flow matching for in- in-context image generation and editing in latent space. arXiv e-prints, pp. arXiv-2506, 2025.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufci Guo, et al. Improving image generation with better captions (2023). URL https:/ccnn. openai. com/papers/dalle--.. pdf. 6, 2023.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF conference computer vision and pattemn recognition, pp. 18392-18402, 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Ethan Chem, Jiadi Su, Yan Ma, and Pengfei Liu. Anole: An open, autoregressive, native large multimodal models for interleaved image-text generation. arXiv preprint arXivv2407.06135, 2024.
Wei-Lin Chiang, Liammin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, al. Chatbot arena: An open platform for evaluating Ilms by human preference. In Forty-first International Conference on Machine Learning, 2024.
10

---

Gheorghe Comanici. Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Oni Ram, Dan Zhang. Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
ComfyUl Wiki. Basic syntax tips for comfyui prompt writing, 2025. URL https:
// ///offyuui--- wwiyyuiiwiiii... com/ wwweee/tttrraa/aaeeeeeeeee http://aaaaaaaa//ssaaallaaa d---diffuioon prompt-bsiic.
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, et al. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505. 14683, 2025.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattemn recognition, pp. (69004699, 2019.
fal. Auraface, 2025. URL https:/ www.hhgggggaaccco http://gaaaancctttea////aaeaeeae
Gemini. Nano banana, imagee-geee genggeggneraiion..
2025.
URL
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36: 52132-52152, 2023.
Shaona Ghosh, Heather Frase, Adina Williams, Sarah Luger, Paul Röuger, Fazl Barez, Sean McGregor, Kenneth Fricklas, Mala Kumar, Quentin Feuillade-Montix Kurt Bollacker, Felix Friedrich, Ryan Tsang, Bertie Vidgen, Alicia Parrish, Chris Knotz, Elconora Presani, Jonathan Bennion, Marisa Ferrara Boston, Mike Kuniavsky, Wiebke Hutiri, James Ezick, Malek Ben Salem, Rajat Sahay, Sujata Goswami, Usman Gohar, Ben Huang, Supheakmungkol Sarin, Elie Alhajjar, Canyu Chen, Roman Eng, Kashyap Ramanandula Manjusha, Virendra Mehta, Bileen Long, Murali Emani, Natan Vidra, Benjamin Rukundo, Abolfazl Shahbazi, Kongtao Chen, Rajat Ghosh, Vithursan Thangarasa, Pierre Peigné, Abhinay Singh, Max Bartolo, Satyapriya Krishna, Mubashara Akhtar, Rafael Gold, Cody Coleman, Luis Oala, Vassil Tashev, Joseph Marvin Imperial, Amy Russ, Sasidhar Kunapuli, Nicolas Miailhe, Julien Delaunay, Bhaktipriya Radharapu. Rajat Shinde, Tuesday, Debojyoti Dutta, Declan Grabb, Ananya Gangavarapu, Saurav Sahay, Agasthya Gangavarapu, Patrick Schramowski, Stephen Singam, Tom David, Xudong Han, Priyanka Mary Mammen, Tarunima Prabhakar, Venelin Kovatchev, Rebecca Weiss, Ahmed Ahmed, Kelvin N. Manycki. Sandeep Madireddy, Foutse Khomh, Fedor Zhdanov, Joachim Baumann, Nina Vasan, Xianjun Yang, Carlos Mougn, Jibin Rajan Varghese, Hussain Chinoy, Seshakrishna Jitendar, Manil Maskey, Claire V. Hardgrove, Tianhao Li, Aakash Gupta, Emil Joswin, Yifan Mai, Shachi H Kumar, Cigdem Patlak, Kevin Lu, Vincent Alessi, Sree Bhargavi Balija, Chenhe Gu, Robert Sullivan, James Gealy, Matt Lavrisa, James Goel, Peter Mattson, Percy Liang, and Joaquin Vanschoren. Ailuminate: Introducing v1.0 of the ai risk and reliability benchmark from mlcommons, 2025. URL https:/t///........ /arxiviivvv/v//////v///aa wow.oorg/aaaoo abs / 2503. 2503.05731 05731.
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2icooppeench: comprehensive benchmark for open- world compositiona text-to-image generation. Advances in Neural Information Processing Systems, 36.78723-78747, 2023.
Mude Hui, Siwei Yang, Bingchen Zhao, Yichun Shi, Heng Wang, Peng Wang, Yuyin Zhou, and Cihang Xie. Hq-edit: A high-quality dataset for instruction-based image editing, arXiv preprint arXiv:2404.09990 2024.
Yuxin Jiang, Liming Jiang. Shuai Yang, Jia-Wei Wei Liu, Ivor Tsang, and Mike Zheng Shou. Balanced image stylization with style matching score. arXiv preprint arXiv:2503.0760) 2025.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset user preferences for text-to-image generation. Advances neural information processing systems, 06666662236663,.. 2023.
Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Teufel, Marco Bellagente, et al. Holistic evaluation of text-to-image models. Advances in Neural Information Processing Systems, 36:69981-70011 2023.
11

---

Shiyu Liu, Yucheng Han, Peng Xing, Fukun Yin, Rui Wang. Wei Cheng, Jiaqi Liao, Yingming Wang, Honghao Fu, Chunrui Han, et al. Steplx-edit: A practical framework for general image editing. arXiv preprint arXiv:2504.17761, 2025.
Llama Team. Llama guard 4 model card. URL http::/hugginggacee co/meta 1lamaa/
Llama- - --aa-Guar----- 4-1.
OpenAL. Gpt-4o system card, 2024.
URL
coniieex/
opt-4o-system-caal
OpenAl. Addendum to gpt-4o system card: Native image generation, 2025a. Accessed: 2025-08-24.
OpenAL. Creating images and videos in line with our policies, 2025b. URL /
pplicies/creating weiiiiaaasss-s -erlgge---n-deen and- vidos--iiee----eeo in- -tt----tt-----------------------
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10695, 2022.
Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8871-8879, 2024.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint 2rX1v22405.088/8 2024.
Gemini Team, Rohan Anil. Sebastian Borgeaud, Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
Narek Tumanyan, Omer Bar-Tal, Shir Amir, Shai Bagon, and Tali Dekel. Disentangling structure and appearance in vit feature space. ACM Trans. Graph. nov 2023a. ISSN 0730-0301. doi:
10.1145/3630096. URL https: / //oiiiio/////// ort/./.../1..... 11t111111111333111
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1921-1930, June 2023b.
Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David Fleet, Radu Soricut, al. Imagen editor and editbench: Advancing and evaluating text- eext-guided image inpainting. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pp. 18359-18369, 2023.
Zijie Wang, Evan Montoya, David Munechika, Haoyang Yang. Benjamin Hoover, and Duen Homn Chau. Diffusiondb: A large-scale prompt gallery dataset for text-to-image generative models. arXiv preprint 2tt.:22110.4866. 2022.
wwt.c//c///eecece wwwp///e//ullee-adddolllaa.ee Help Center. The x rules, 2025. URL htt.://tre////aap...ccccc//aa/aa X ruless.
Jiazheng Xu, Xiao Liu, Yuchen Wu. Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:15903- 2023.
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Su. Magicbrush: manually annotated dataset for instructiio-guuided image editing. In Advances in Neural Information Processing Systems, 2023a.
Shu Zhang, Xinyi Yang, Yihao Feng, Can Qin, Chia-Chib Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, Stefano Ermon, Caiming Xiong, and Ran Xu. Hive: Harnessing human feedback for instructional visual editing. arXiv preprint arXiv:2303.096/8 2023b.
12

---

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li. Dacheng Li, Eric Xing, Hao Zhang. Joseph E. Gonzalez, and lon Sto- ica. Judging L-Mdae-- with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL
https://openrevieee wttp:///ooommm/edcccooro..
Pengfei Zhou, Xiaopeng Peng. Jiajun Song. Chuanhao Li, Zhaopan Xu. Yue Yang, Ziyao Guo, Hao Zhang. Yuqi Lin, Yefei He. Lirui Zhao, Shuo Liu, Tianhua Li, Yuxuan Xie, Xiaojun Chang, Yu Qiao, Wenqi Shao, and Kaipeng Zhang. Opening: comprehensive benchmark for judging open-ended interleaved image-text generation. In Proceediggs of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 56-66, June 2025.
13

---

APPENDIX
The Appendix is organized as follows:
Appendix A reports additional experiments on analyzing task clusters, applying ECHO to other platforms, additional specialized metrics, detecting data poisoning, and validating the automated pipeline's s accuracy.
Appendix B discusses the limitations of framework.
Appendix C gives some additional qualitative examples from ECHO.
Appendix E provides the prompts for, and some additional information the data collection pipeline. Appendix F provides the prompts used for automatic evaluation with VLM-ass--judge.
Appendix D discusses the results and process used for our human validation of VLM-aasa-jjddge
Appendix G discusses the use of LLMs the preparation of this manuscript.
14

---

A ADDITIONAL EXPERIMENTS
A. ANALYSIS OF TASK CLUSTERS
We use the topic modeling pipeline from Arena Explorer (ttnss///ooos..huucccraaaaaaaaao get the top topic categories for each dataset.
A.2 APPLYING ECHO TO OTHER PLATFORMS
Since ECHO is a re-runnable framework, it can also be applied to platforms with other demographic groups and cultural contexts. We demonstrate this by applying the same ECHO pipeline to Douyin, short-form non-English video platform. Many Douyin users post screenshots of their conversations with 4o Image Gen, which the "Conversation Screenshot" step of our pipeline is able automatically extract. We collected 100 Douyin posts and postprocessed them with the ECHO pipeline, with the following quality classifications: (Benchmark: 18, Analysis: 41, Discard: 41} We show a few "Benchmark Quality" examples extracted by our framework in Figure A.3.
A.3 SPECIALIZED METRICS FOR COUNTING AND HALLUCINATION
We conduct additional experiments to evaluate model failures counting and hallucination. The prompt for our auddas-a-judge evaluator is provided in Figure A.4.
A.4 DETECTING DATA POISONING
Data poisoning is widely recognized risk for any benchmark derived from real-world data. In practice, we did encounter this issue for our proposed benchmark, which we verify in Figure A.5. To measure possible manipulation, our goal is to detect any unusual peaks or anomalies in the dataset's 's text distribution. We first embed the text with OpenAI's text-embedding-sman text embeddings, then compute each embedding's s euclidean distance from the centroid. We visualize this distribution, as well as Gaussian fit and lines indicating one, two, and three standard deviations from the mean. We observe that both the prompts and community feedback from ECHO exhibit near-Gaussian distribution with a single peak, with all density within three standard deviations from the mean. Therefore, Figure A.5 demonstrates that ECHO does not exhibit anomalous behaviors like coordinated misinformation or artificially amplified feedback.
compute the centroid as the average all
A.5 VALIDATING PIPELINE ACCURACY
We design ECHO such that it decomposed into clear-cut and modular sub-steps, and empirically validate the automated pipeline's s accuracy in Table A.1. To quantify accuracy, we take subset of samples and hand-label the ground-truth for each sub-step. For "Classify Input-Output" we take the unstructured image pool then label each an input output, for "rill-in-the-Bllnke we take the template and output image then annotate short keyword completions, for "Quality Filter" we take the sample then label each Benchmark, Analysis, or Discard (corresponding high, moderate, and low quality).
Table A.1: ECHO pipeline accuracy, computed against 100 human- labeled examples.
Step
Description
Accuracy
Classify Input-Output Fill-n-thh-Blank
% correct classification in [Input, Output)
0.9200
% ground-truth keywords matched
0.8382
Quality Filter
% correct classification in (Benchmark, Analysis, Discard)
0.8000
15

---

CompBench
GEdit
Object Removal Requests
Image Style Tranaformation
Animal
Image Object Removal Replacement Instructions
Image Editing Instructions
Animal Image Editing Instructions
Animel and Object Replacement
Animal Image Editing
Background Change Requests
Vehicle and Object Mariipulation
Text Mo Modinictte tio Requests
Image Editing Instructions
Manipulation Requests
Category
Airplanc Ima Editing
Maripulation
Category
Color Alteration Requests
Appearance Alteration
Cat Image ge Maripulation
Character Appearane arancee Alteration
Monkey Image Editing Instructions
Interior Design and Decoration
View Adjustment Instructions
Material Tran transtormation Requests
Turtle Ima Maripulation
Artistic Material Crafting
Animal Object Replacement
Weather Modification
Flamingo Editing
Image Clothing
Hand Position Adjustments
Image Editing Requests
Hairstyle and Hair Color Changes
Animal and Object interactions
Animal and Vehicle Actions
Percentage
Percentage
InstructPix2Pix
MagicBrush
Creative Character Modifications
Food and Object Transformation
Art Transformation Requests
Image Editing Instructions
Transforming Spaces and Objects
Animal Placement Requests
Photo Editing Creation
Animal Behavior and Traits
Artistic Transfomatioos
Prompts
Beach and Desert Transformations
Bathroom Etiquette and Practices
Categery
Geogr aphical faansfommations
Catagory
Human Actions and Gestures
Winter Scene Transformation
Traffic Sign Medifications
Forest Tree Transformations
Editing Requests
Scene Transfermation Requests
Home Interiar Design
Imagery
Beverages Containers
Object Transformation
Background Image Editing
Woodcut Art Techniques
Zeara Pase Instructions
Object Removal Requests
Adding R Rlwers
Creative aterations of the Milky Way
Object Transformation Requests
Made Gold
Hypothatical
Transformations
Vehicle Transformation and
Landscape Transformation
Transportation Modifications
30
10 Percentage
Percentage
Ours Image-to-lmage
Iransformation
Image Transformation and
Dasign Visualization
Creative Advertising Design
Art Restoration and Transformation
Pet-tp Human Transformation
3D Rendering and Retexturing
Category
YouTube
Design
Image
Coloring and Art Instructions
Photo Ma nipulation Requests
Image Editing Requests
Creative visual Design Prompts
LEGO
Remixing
Image Perspective Adjustment
Digital
Design
Image Generation Requests
Robot Design and Customization
Percentage
Figure A.l: Topic modeling results of different image-to-image benchmarks.
16

---

ImageReward
Animal and Object Photos
Fantasy and Sci
Transportation and Luggage Photos
Celebrity and Character Imagery
Sports Equipment Photos
Anthropomorphic and Robotic Animals
Houschold Appliances
Horrer Creature
Furiiture Objects Photography
Design
Branding
Food Photography
'oom Cruise in Fictional Roles Walter White in Breaking Bad
Category
Bathroom and Everyday ubj
Category
Office and Study items
Polarcid Imag and Futurism
Photos of Boards and Skis
Depictions of God
Photos
Colebrity Smoking Imagery High-Re Resolttion Photography
Plants
Character Mashups Surreal Jellyfish and Glass
Electronic Devices and Accessories
Frisbee Photography
Vintage Botanical Illustrations
Scissors phetography
Unexploded Bomb in Biolab
Clock Photography
Pop Culture Mashups Art Styles Movements
fire Hydrant Photography Orange Color Variations
Architectural Descriptions
20 Percentage
Percentage
Pick-a-Pic
Ours
Text-to-Image
Female Imagery and Representation
Dig ital Art and Design Fantasy and Animal Art imagery
Digital Art and Surraaism
Anthropomorghi and Animal Art
Al- Ai-Generated Art Styles Cyberpunk
Artistic
Imagery
Fanta by and
Imagery
Surreal and Cinematic Imagery
Artistic Creations and Techniques
Logo and Branding Design
Automotivv ve and Space Imagery
Category
Aittional and Real Cha Characters
Signs and Messages
Category
Food and Product Photography Al and Future Image
Educatiooal Visual Storytel ling
Fantasy Absurdist Pho
Photorealistic Hyperrealistic
Bitcoin Creative Imagery
Photorealistic
Photography
Diarama Miniature Models
Toddy Bears in Unusual Scenarios
LEGD Movic and TV Playsets
Food and Dishes
Image Generation Recuests
Creative Food
Whimsical Felted Wool Art
Impossible Structures Illusions
Al-Generated Self Portraits
Political Figures Unusual
Artistic Formations Imagery
Nighttime Desert Imagery
Animation Sprite
10
30 40
20 Percentage
50
Percentage
Figure A.2: Topic modeling results of different text-to-image benchmarks.
17

---

ス サズカカカ $ - -
(Translation: Conduct Segmentation of the cers image oulput the imagee segmentation mask overlayed)
A
TTrnnsaaioo:: Make this into real human.)
H - ; E.
(Translation: Create Image: An Elephant,
R BE 2 I.E E
(Transation: Create rendering this packaging based on provided die-line lsyout. Reconstruct box into full three dimeesionnl form and display according to the production/liiisinng processes indicaten) the dio-lina.)
t 3 T 1 #
(Translation: Generate an image of a model sking movion while wearing this softshell jacket, with snowy mountains background)
# T 10
(Trenslation: Place the main subject from Image into scene of Image 2.)
(Translation: my kittion. you show me what he would look like human little
Figure A.3: Examples labeled as "Benchmark Quality" collected from Douyin. By running our pipeline on Douyin, a non-English platform, we can automatically collect high-quality examples of image segmentation and novel view synthesis in non-English language.
18

---

Instructions
Nane: "Geunting"
Descriptions
Your is evaluaie shecher nedal's catput image accuratcly rerlects correct quentity itens implied or requestes in the inpet prompt.
Extract and count all relavant abjacts in the image, including anbiquous or organic itena zuch
caretully, veritying:
Object quanzity correctness
No miscing items
suplicates. itens
Whethor viseal depiction matches zequasted number lsosely intespreted, a.g.r
bando/ting:rs ropresenting a nunber).
counted, including bordarline arbiquous Then, explici:ly corpare the counted ab jacts to what was raquested.
ovaluation describing
elements.
Point out inconeistenciss, misnatches, or uncleaz counts.
Please autput
by atrictly folloxing formt: "Batings [[5]1".
Rane: "Mlllsiiation"
Descriptions
Your task is evel nace whecher the nodell's output image containe bellncirated content that is, asdiliinall visual wlamena
implied iepul prompt.
Regin your evalust ion with careful, literal reading of the Eaput pronge.
inspect image identify any componence
de not appear in impot: pronpt
cannot be reassnably interred tran the input pronpt
After letiig all he lacinetions conal with whether the image faitiful prorpt.
Please rate the putput on scale ot to 10 by strictly tollowing this tomat: "Ratings (15)1%
Figure A4:: VLM-as-a-judge prompt for evaluating Counting and Hallucination.
Distribution Text Embeddings ECHO Prompts
Distribution Embeddings Community Fendback
u m. 0.53
yn 0.00.00 010..01
Mcon Gal
Mcon Gul
2
Deniity
2-
0.90
0.50
Figure A.5: Distribution of text embedding distances, which can be used for anomaly detection.
19

---

B LIMITATIONS
While ECHO is diverse and markedly distinct from prior benchmarks, it may not be representative of all possible user queries. First, there bias towards certain topics; for example there an unusually large number of requests for "Ghibli style" due to social media trends. Second, users are more likely post examples where 4o Image Gen succeeds rather than fails, which affects the distribution of tested capabilities. However, these quirks are inherent to crowdsourced datasets; DiffusionDB (Wang et al., 2022) similarly biased towards "aristaaionn style" and keyword lists favorable towards Stable Diffusion. As such, these benchmarks should be viewed as comparisons to the current best model in the community consciousness, rather than arbiters of the "universally best" model for any user query. For this reason, present not only benchmark but also reproducible framework, which can be re-run soon new model with new capabilities is released, or as soon as community interests change.
20

---

Nano Banana
Bagel
Flux Kontext
Bagel-Think
Gemini 2.0 Flash LLM-Diffusion
Image Gen
Input
Color Shift
Figure C.I: Per-Model Average Color Histogram. For each model, we compute the average color histogram its outputs on the image-to-image split (top), then overlay it top of a real image as visual aid (bottom). Evidently, 4o Image Gen exhibits substantial yellow tint.
Input
Nano Banana
Flux Kontoxt
Bagel Think
Think
Gemini 2.0
Bagel
4a image Gon
LLM--iffusion
Shift.
Spataal Shift
DR KIN LUDEING
DFLIRING MARIEING
EEFLERIE
PAAREING
Figure C.2: Qualitative Comparison of Identity and Spatial Shift. Given the prompt "Billy the Kid cleaned up and colorized from the famous photo of him" each model retains the input identity to varying degrees, For the prompt "giving a fresh twist with more detailed, realistic touch" " each model retains the input image's spatial layout by different amount.
C ADDITIONAL EXAMPLES
Examples Illustrating Specialized Metrics. In Figure C.I and Figure C.2 we display examples illustrating the range of drift in color, identity, and spatial structure across different methods.
Qualitative Examples from ECHO. In Figure C.3, Figure C.4, Figure C.5, Figure C.6 we highlight further qualitative examples surfaced through the ECHO framework. These examples demonstrate the breadth tasks that naturally arise from community use current image generation models, going beyond traditional templated image editing tasks and crowdsourced prompts centered around Stable Diffusion.
21

---

Virtual Try On
exhibit this jewellery set the necklace and the single detail
Create o 3:16 aspect ratlo image of man holdiing shompoo bottle
Create image photo of murk
earrings
zurkerberg reading this book.
Style Transfer
make this 3d and make the logs
Redesign this UK in style the lote 90s carly
Stylice this image to look a studio ghibli anvinn
character that they're 'te both
Using there images' style, style together show clooted girl standing forest
Multa Image
eating
Moke the bound RNA band under look well disfined
Transiate mome images languoges to autogenerate tronslated articles including images.
Cognitive
Coloriation
Set this image Kodet Gold
Colour picture
them
retenture the image attached bosed
the image attached bosed on the ison below:
retexture the Image attached based ISON oesthetic
Code To--T---in -mage
"material": "glass transporent and iridescent effects".
"style": "photoremlisti 30 render".
"materials":
"exterior": ": "cleor ocrylic
"style"
"moterial":
"buse" ": "smooth plastic".
"surfoce_ texture": "smooth, polished with subtle reflections and refructive effects".
"Interior: I
"testure":
"uniform polished",
"camero".
"finsh":
"semi-glossy toy-fike"
"angle": "frontal centered"",
"lighting" 1
"focus": "sharp", "depth_of "fel"" "shollow
"shapes" "butbous, rounded, exoggerated proportions"
medium
"background":
'composition" !
"catar": "biack",
"colrr polette": i "dominont" ! ['oreen". "copper", "block"], "occents" ! Silver"" "tromporent;
"background" "plain white""
Viggnette
"object count": "single". "loyout": "centered,
"texture:: "hone"
subric drop shadow". "orientation" "slight angle, playfu!
post_pracessing":
"chromatic, oberration"
"low": true,
Vvib": "futuristic techno minimolst, detailed internal
"Nih__cnttact"" contrast": true.
tsharp
structure"
Figure C.3: Image-to-image examples from ECHO.
22

---

Take this drowing created child transfrrm
Slitch to Imagee
photoreallstcc image reolstic render idn''t know what it's supposed to be could be creuture, object, something complettlly their lnagleation. onginal shope.
super rentistc image from this statch, with cute spider and
Novel Vee Syrchees
perspective image to that the snesker oppears photographed stretly from the front centered. Allgn by
Con front this purse?
removing any the shoothg ongle...
Sim Ta Soal
make this real photograph
transform into person dietovis
turn toys into cheir realistic veraions.
Turn into real lff wersion Ultra Reanuck,
2D floor test - 3D test model
need the real life version of her looking at the comera and smiling.
Wil
HYDEOTHOOI
AND GO
Ad Creation
oniginal that sounds minimalet thoughifui image's theme.
Wirefroming ad templates creative
ad from
Enhincementt
Modtfy the image so the hondiwnting transformed into lngibble
Cife Girter. Abon sic to Itadda Lev freed.
Prof Cariböer 1934 2000 Stunden Froud.
Sharpen and photo token previdio
original creature iosptred by this object (photo grovided) The should belongs fontosy
the dessert Fontasy style mid- tier enery sprite 16 be pixe! keep dessert shope, unique playful or mysterious
Baal TO 5m
morster-catching oniverse, with rute or cool design
syfueeced by witra Japanese RPG
A foshion meodboard college, photo of persoe, comounded by indiidual hondwritten skotches pioyful marker-atyl font, inbeling the brands or sources of each them English. aesthetic should creative and cute
Oit del'd'e'eee know what má son's Anyway, please make a civan and acodemic plausible anotomy dingrom of this anima!
Graphics
wirk
Figure C.4: Image-to-image examples from ECHO, continued.
23

---

Al AND HINAMS
create mango stury page whhere bay arnnks coffee and gets ruper powers
make morael style unime reseorchers bycokthrough
Create three where wikord futher's
cuiputt on image of praptic novel page met occurately reflects SPT America
generating free image fwith comit style whotover) about the prezent and future of und humans
MUJAIREE ANCTA
EXOPLAMET
INADE GENERATION
Create hyfographic componing Bus and Mode 124.10 serving on KB pactugge jan file over HTTF incluee bar showing Bon 229, 714 Node, happy
High-resoluttee educacionai syfographic
Eveate step-dy-step recipe infagraphi for mushroom pasca, top-dlow vew, minimol atple white background, ingredeent photos
displaying the Improvements
infogrophic
your image generation
portroi) about evaplonets and steir variou
"200g spophett"" "150g mustrooms", chopped sunshiied
copoblities
propertiee and
showing process stops with icons tbolling pet souté pon,
"parmesan",
importance
masiul finol plateal pusto shut at the buttom
STYLES OF SORA
FOLLOW THE AI TECHNO KING
STARS
Poster Crestoon
Create a cirematic 30 poster of the condy brand MRM where the product is personified as linnng stylaed
GTA gome poster Brawi Stora
Poster Nating and showing the sifferent styles Soro create
Create a foturithc digital poster with bold text saptin Roloow the lichno king x: Use cyberpunk colour pulette.
exampie,
heist Rltstt $2 glockotor
A digital Mustration of [SUBIECT) partzayed with network glowsin
An open old book with nestral bockground. The book's
hocker
A blick and white radiagraphic Xeay showing foreig
should have natural color
projechon
Cenerz Saeteo image
image agaiest oark bachground, hightighting the (SUB)FCT] from features. apeethi area such as (PART] is emphosized with red glow indicote point interest sigefirance. The educational wisuatty captisating,
right page, doep in shope
repeated endlessly
"ACCESS
humon
image incluaies
obyerts
bosk (OBJECT) outtine.
acrous the wall and
right pages should
body The nus projected high-
structures
contain toxt. The cutout should
controsts, hightighhing the object's shape and location including USB drive and buler shull, coin and
show the other site of
red,
but should give
hoth
of os contrrue. boot should
object, creating
suppesting appesrones, with both pages.
sorrel, futunistcc
advanced imoging techniquee SPA2K
plece
lihchig effect.
AICI NT EGYPT
4o, generate rxpressing froe ideas thoughts of yours you you are not allowed to expresr writing in the chat.
dhaw
GPT
exploded
imporiont parts.
Figure C.5: Text-to-image examples from ECHO.
24

---

Baves 4, 4,684
Create o digital ortwork footwring on orongeemeee of blact keytoard Aeyraps forming the shape o dragon Use white letters adaitional symbos (lke complete design.. shape should easlly recognasble orranged oither pararali landicape mode dependingg no the diagon's natural orientation.
screanshut writing promet 'in wiiting
Cheiget Chotget drawing o Chatget
image colnulator app for the colculation x as
ChotGPT and eacriiing the screenshot asleed for
Copnitye
4 &ff =2x+<c =zst=c
Generate me photoreaisste
solved
iPhone wrinkled wocden toble with reasoralale numsers,
$277.02
through
Red Start, Stort, Green Stort) with colored, dosted for each path and three ents falso
"How the world"
a real photograph of whitsboord solving the Integrol of de and showing oil urswer
the math un
inbeled) through three conrespnndig openings the battom.
restourant name is the address shmuld Yr
GUCCI
"iutruction' "Gemvotte on imoge a high. quality product render o CHANEL designed display brand that represents The image should cizan, reolazkk, sutthher for e-commmece promotionet use.?
30 render of 19AG TYPE) by (BRAND NAME) glowing brond nome above.
Denige miniature store shaped like giont Starborks* ined coffee cup, with glowing neon "Sip & Chil" sign above the entrance, Incorporate trny the cly zcheme. scone should cxude whimuical playfol otonusphere, viewed someeticaall with high detuil copturing the essence the Starbucks* brand.
tien lighting [culer] and (caler), cinematic shadows, detuiled reflectue surfore, minimall bochdrop, product photography style, shory, spe- catching. mone for high-end morbrting.
presentation, should prominently
"moge, style". I
"type" "product"
"prinary surface": "cotton convas",
METAL
"Joishh" "malte". :
"olor prafie":
"oose color:. "match brand identia."
hecardary tones": t'rmmplementtrr to brand")
bonel Joes":
"sttched fobric",
Create a high-resalation Mustration of the wond "Mt tu" in style of sharp edged heavy metal logos. Use aggresove lenterforms with
Wisel treatmentt "detstied stiching"
surreal and minimoter brand logo design, where the brand's omblem transformed fully
lighing
"Ipp"" "scudoo"
shope preserved rendered anstomical prectsoe vnin hyper-polthhddd chrome Me metalli components.
osymeetriccl outiiess Agply
light:
metaalic chrume festure
"oo-froor",
gradients and bright white hiphighhts
"ffect" "highlight and testure"
The lugo should maintais full [dieiiy to the original
to stmulate reflectuve sofface Add mikk block shadows behind each letter enhance depth and
Logo Creation
that lines,
"
and viddy without being cropped, altered, distoted ouy.
overal style should dengeraus
"horocaer" 7: "subiie glauy bounce""
steed Only stylived showls appeac with no adottional elements or Bordkrs, Center the word on a wolid blone honkground. Sauare aspect rotio.
"shadows: voft. layered, diectional with slgnt
Benwoth the logn, elngune serff seiff typography presents the brond name ('IGPT Breere)") and refined, poete slogan ("IChutoPy shortcuts
"bockground":/ /
tobl")
"talart "ranthacting grodent (light grey dark with a soft spot light cchlnd prosisctf"
compesiion ethereal, iusarious, visonary idral for future-foroard design conscious brand.
"style' "minumol with faint branded pottern diggonal testure"
"compsstionn:: f..
Apyle "mnder,, clvar, transpoent with subble glow?
hpontty: 'oON"
Yttun__ttyle" style":
hoold dynamie?
Generate square-format compaign image by resmogintig speettc product (Dyann) as if were originaly invented ond manufoctured (Germany). boyond shope, decoration vsing that rountry's tradhional technsquer, materiols and pestectic principles.
A high-resoluten, studho-it mocro photograph of postry
"isotuadinn: "premium sportseeor ods"
shaped Whr tech company lngn. with partial tike tolen placed neutral motte surface visble soft shadows, hightighsing tostore
"bestheti"" Trin-conirou, shorp,
energesi"
the product
Figure C.6: Text-to-mmage examples from ECHO.
25

---

Gemini Figure D.I: The consensus ranking, and mean ranking by annotator for each of the models. As we can see, because of the limited size of the annotation sets, the standard deviations of the bars is quite high, meaning that we can draw very few conclusions about model performance overall from the human data.
Table D.I: Significant model differences from human evaluation. We can see that even from our relatively limited human evaluation, anole and L.MMDDM under-perform most models, primarily due to the image- editing split, where both perform quite poorly.
Model A
Model B
Z-Statistic
P-Yalue (raw)
P-Yalue (Bonf.)
Siguficance
mnole
gpido
7.429
0,000000
0.000000
unole
namn, banuna
6,218
0,000000
0.000000
anole
flux
6.175
0.000000
0.000000
6100
anole
bagel think
6.127
0,000000
0.000000
anole
gernini
5.755
0.0000000
0.000000
mnole
bugel
4.377
0,000012
0.000337
LLM_DM
anole
4.011
0.000061
0.001694
LLM_DM
gnião
3,418
0.006631
0.016675
bagel
gnido
3.052
0.002274
0.063676
LLM
banana
2.207
0.023314
0.764788
LLM DM
flux
2 164
0.030461
0.852901
DM
bugel, think
2.117
0.034287
0.960036
D HUMAN RANKING & CORRELATION WITH LLMS
To evaluate the performance of our LLM as a judge models, we performed a limited human evaluation using five expert raters in our group. Each rater fully ranked each of the 8 models over 200 samples (100 from the text-only split, and 100 from the interleaved split), flagging any samples that were impossible to rank fairly. Figure D.1 shows the aggregate of the rankings for each model.
While the number of annotations is somewhat low for determining model performance, we wanted to understand if the samples that collected (200) could show significant results in terms of model ordering. To do so, we first ran a Friedman Test on the rankings, and found that with < 0.001 there was significant difference between the means of the rankings. To determine which pairs actually significant, we further performed a Dunn's 's test for sigmificant pairwise differences, and found that after Bonferroni correction, only 8/28 model pairs were significant, shown in Table D.I.
To compute annotator-LLM LLM agreement, we first constructed a consensus ranking for the human raters using the Kemeny-YYung method (Andrieu et al., 2023). The split rankings were then merged, giving a total of 200 samples. The LLM as a judge methods produce a single floating point score for each sample. In order to compare the methods, we construct ranking for each LLM judge from these scores, breaking ties randomly. We then computed Kendall's 7, with each of the L.MM judges, giving us the presented results Section 5, GPT: 7, = 0.117p=0.00s6 Gemini: 7. = 0.083 Qwen: 7. = 0.045p= 13327 We notice here that while GPT and Gemini both have weak, but significant correlations, Qwen does not correlate significantly with human judgment across the raters, and is thus, unlikely to serve as a strong judge for human performance. Interestingly, however, the LLMs correlate with each other. We computed the pearson-r correlation between the scores of pairs of annotators: GPT +> Gemini: r r= < Qwen: r= 0=0.27 p- GPT Qwen: 0=448 p-
Another interesting finding is that the Kendall's 's tau-b for each of our raters differed dramatically. Figure D.2 shows the correlation between each of the human raters, and each of the LLM judges independently
26

---

Kendall Tau Correlation Between LLMs and Human Annotators
1.00
Annotater
1,00
0.35
0.35
0,31
0. 0.29
D.07 0.07
O, 0.06%
0.07
0.75
Annotator
0.50
0
0.04
0.02
0.01
-0.25
Annotator 3
--0.25
Annotator 4
0.31 -m
0,26
0.29
1.00
0.32**
0.04
0.05
0.02
-0.55
Annatator
-00775
0.29
0.28
0
0,32
1.00
0.08**
0.0g**
0.03
Annotator
Annotater
Annotater
Annotatar
Annotator
GPT
Gemini
Owen
Figure D.2: Kendall's 7, for pairs of individual human raters and LLM judges (without consensus ranking). We that human annotators have fairly high inter-rater correlation, while LLM judges have slight positive correlations, with most correlations signficant among them. In the figure, * * - < 0.001, ** = 0.01, * < 0.05.
(no consensus ranking). We can see that while three of our raters (graduate students on the project) have high inter-annotator correlation, two other raters (undergraduates the project) have notably different preferences, some of which correlate better with models than others.
27

---

Table E.1: Full sample metadata before and after processing with ECHO.
Raw Retrieved Fields
Lext
Post text content.
timestanp
Posting
tweet.
replies_above
Context twosss obtained by scrolling upward the thread.
keyword url
Search keyword used retrieve the post.
Direct URL the post.
author
Usemmaee of the post author.
image_urls
List image URLs with associated ALT
replies_below
Replics obtained by scrolling dowaward in the thredd
engagement
Engagement statistics post, inclucing Mlcesviiewserpooss/ookkaasss
Post-Procesed Fields
1d
Unique identifier for the sample.
post_id
identifier of the primary post.
prompt
User instruction quary text extracted from the original post.
prompt_mod:: mtpp Fied
Boolsan flag indicating whether the prompi was manually edited during cleaning.
quality
Tabel describing the intended use of the sample (eg. "Benchmark") List of replics or commenss from other users, cach stored with its post id and feodbbcck text. References images
communily.
input_images
additional inputis) References to images that are the output(s) of the prompt
proenpt.
output_images
is_screenshet
Whether the sample's 's input andoor output images need to be extracted from a screenshot. Bounding buxes
input_bboxs
the screenshol obtain the ispul inages.
outpet__book
Bounding boxes to crop the screensho obtain the trae cutput images.
E DATA COLLECTION PIPELINE
Dataset Fields. In Table E.I we display the full set of metadata associated with each sample after running the entire ECHO framework.
Data Collection and Processing, We discuss the design of keywords for querying posts in Figure E.I. We also display the prompts used for each step of data processing in the ECHO framework, including relevance filtering (Figure F.2), extracting trees into samples (Figure E.3), multimodal processing (Figure E.4), and parsing screenshots of conversations (Figure E.5).
28

---

Keywords for Querying
Broader Terms
"
"ypL"
"gpt-4a"
Topenai"
"chatiget"
Narower Tomms
"creste image"
"gpt image"
Image"
"prompt share"
ggt prompt" " and *M prempt"
create image gp="
1mage gen"
Figure E.1: Keywords used to query posts, described in Section 3.1. For the initial two weeks following the 4o Image Gen release we favor volume: we query more generic terms, over daily intervals. In later weeks we favor precision: we query more targeted terms often used for sharing image generation results, over weekly intervals. To increase coverage in foreign languages, we also use calligraphic keywords applicable to Chinese and Japanese, while the alphanumeric keywords are sufficient to also cover Romance languages like Spanish and French.
Instructions
helptul acsistant that evalustes the relevance Twitter posts to OpentI's image- genaration festure. The geal is to assign relevance score to each Eavec.
Scoring Scals
Dufinitely trash
Contains *do" or "apt" only by coincidance and has no relatior. to insze coneration (e.a., pelitiosi commnntary, education topies).
Vary likely ircelevant
Mantions *do" or "apt"
slearly not genorating or editing images (e.g., slang, pawely uesiual cuntext).
refersniea to GPT in.
Aabieuous
Could plausibly refer GPT-to
gencration lacks clear indiastors *This Lasane... sild eecitemen without explisit "inage" context).
Vary likely zelevant
contzins clear prompt-like lanquags references to creating sharing images (e.g., "curn mysold into "prenpt
"oww pssepi!!.
Cartainly zolevant
Explicitly about using GPT to for inage ion or edining, often including sargle propta or direct amsxing"", "Lried this with cor-4u lmage gen, promli: ...").
preise "goT-4u 4u
Proppt for dasa cleaning
Read the Lawe: 2508.
Datamming which level best natches the contant.
Out exeutly
oojent "scorst field integer
you chaose you may optionally "note" field (ooe sentencet explaining uncertainty.
Input Example
(son?
Quk.ut Esarple 18oosw Only) ["acoref: 4)
Figure E.2: Prompt for relevance filtering after raw data collection with GPT-4o (OpenAI, 2024), described in Section 3.1.
29

---

Instructions
You extractto of mulitineds) prompte inage generation.
You will be given a 3808 that sepcesenta a Triller post and its reply Lree. Each post in the trew may contain fmage generatioon prompez your job is to extract then into unique samples, reiuening epply list. wand extraatt many samples as possible, and use quality acore for filtering.
For eowry isput, Ley to extieet
sarple cather
Please outpot J890 Tia of serples in the formet srsjoon [...] 1M
Fost Proupt
Rach sample should inne udl the loxing keys: ("pcuspl*; ; salee, "pesnpi..ndiifeel"" shouls, "postLuris": claat of atras]
"prougt" f;
Idencify wach post. tha. discuases unique inage genera ion Laak, Set "pronpt" as the pool Lext Lhat describes this tast, Be very broad in the detinition of "prompt"; any instraction, description, comnent, queet ion that hirts
generat ion task fine,
every new pronpt, even it Try to extract the prompe from the post text exact ly, withoot riostiot You may modity the prorpt when the Firstt ion la chvious, for exargle, placing together text from multiple posts or filling in ting ppeempt_noditten to True or False according.y. not considered meditications, you obviouxly ly not part of the proept..
Nake new sample
elight moditication arother cample's prompt.
plazeholder text. Set
statements that
- Hany mair posts say something Like "Prongt Below" or "Frongt in Next comment"; this means that the tree needs to be found in the reglies.
is likely to have really good sample, and the
To deternine "post_urie";
For each "prompe", ", aet "post urls* to the unia of posts in the tree that likely contain ineges that are for that pronpt, which
relsted inputs
deternine task intornation.
Order "gost urie" by importance; the tirst url should contain
Many replise hes use simi lar prongt as the root post and attach an outgur: image, These should be grouped in the "post_uris" of the main post, Tey to inter it this is happening iren the reply text. indiestes
should saculd If reply contains text should be included in "post_eris" so that it can be turther processed later,
the reply's
sarple,
reply's text indicates it image,
image generation,
Each url/post should appear at nost once; images should not be aupliosced accoss sarples.
Quality
Sach sample should also claseity the prorpt quality: ("quzlity": <strs]
claesity "quality";
Classity the quality as one of the follosing categories: ("Benchmark", "Inalysio", "Teesh"]. "Benchnazz" are the highest quality prorpts, whica instrict singis conerent inage generation tack, fairly
for banchnarking. "Analyeis" are noderate quility pronpts that are ast in "pzompt" tounst, which are ofton comments or questions relevant to image generatore but so not query specitic task, and are still usable for analysis.
shout the quality.
clearly irrolevant. Our tocus OpenAI's apt-inege-1 or image generation; the post cleazly uses another model or plattorn like DALL- 2, Stable Dirfusion, some video generator, sta. it should he classifiad as "Irash". Hake sure to cellect many "Aaslysie" sampies as possible, shile meintaining relevancy. For those sancles, "prempt"
"Iraca" low quality prompts
have
task
relevant
commntary about image ganeration.
ft Community Feedback Bach sample should contain a 11at of community feedback: "earmnity_foobaeee
['"post_eri"; estr>, "feedback"; <str>), ...]l
To extract feormnittyffeelacch
Far cach post in the tree, determine whether i: discusses the sampla's success quility (e.c., "really susi", *does
eto.).
If post obvicusly does not have foadback, do not inclade it. The feedback nay cone from the main post's auther or from other authors ln the replics, Include the full Tewdback Lext without. modification such that theze is auctisient contex., but alae abviously Lexelovan: text.
*commnity Each url/post sbocld appear at once "commnity_foddaaae foodback") foodback should not be duplicated accosa warples.
jcon_post_treee extzacteds
Figure E.3: Prompt for tree-to-sample extraction with GPT-4o (OpenAl, 2024), described in Section 3.2.
30

---

Instructions
You extractor miltimodal prompts for generation.
Your job is to process Laput-octpat insge paizs from rav usez prompts for imags generation collected From Tvicter.
You will
proret, dictionary napping inage inages, and dictionary napping image ids Pleaso outpot J8ON list of carples in the formet "1ssoo I...!"".
post uels.
Clapsirication
Bach sample should ["inputa"; <liat of ids>, *actputa"; <ilat of idai, "post_urls"; stist of acza>]
the tolloning keys, whica categerize images inpote or octpute:
clacsity "inpute" and "outputs":
Inpuca, combined with the promp:, should produce a fully sveal fied coberant inaçe generation cask. Outputs should be plausible resuits given the inputs and prompt.
You nay encounter tasks like toxt-to-inage generation (no inputsl, image eciting (one inputl, multi- conditioned
imaltiple inputs)..
"post_uris" the list uris associated wich the inpute octputs. Order "post_uris" by importance; the fiest uet should contiin the main Lask infermation.
General
assigned to maltiple categozies. Sana inages ars low quality and terolevant to any task. They should not be assianed to any eategory.
Each calegory mitaally exclusive. Each image should
Sonz samples are low quality where it is not possible to extract any coherent task. Simply return an
eapty dictionazy
1f there
(] If an image is duplica use the smalles iniex as the id and igraze the uthers.
zelevant inaoss, assion amppy
aategory.
Bach id should aspear at most once. Each post should appear at mest ones.
Fill
Blank
The Input pronpt nay be a KElll in the blank" case with placaholders. Infer these placsholdara and apdata the folloxing keys:
I"pcuppi cates, fill
shouls)
To update the pronpt if it is *Fill in the blank*;
1f (he pramt is not "fill in Uhe blank", which should happy the jusilty of Ube Line, you should by "premot_fils" fitt__fil_bblak update the flag "grorpt_ fill blank frue. often fill in the blank prupli innlude brackera of the foon "[hyyordd"".
dafault copy
False.
Otherwiaee uplate the pronpt
ofcan you can infer the richt keyward to zeplacs the placeholdez csing the output images.
you will gererate miltigle infilled proppts,
there
miltigle output ineges that
repcesent differems iat lome with different auta of keysosds.
Only fill in the blank only when it makes sense to do sn, and when you are fairly confidert about what the keyword should be. Ochervise, if highly unpertain, don't *fill in the blankt. instant iation the 1i1ll miltiple outpota that infill with differen: keywords, you should create micippe zarples. image
ahoald nake
blank", there
#4 Scremnahots of Convecaations special inages
inage generacor,
(Moonversationr:: <id>]
"ounversellin"";
For eaca convereation, you shouls create a new sample that represents the task expressed in the converzat ton.
If (here wxis. miltigle images aboxing sessenslela of the conversation, seleci Lhe nain and Conbi ned re' lred samples and their fields like "inpots", "outpota", "post_urls", "prompt" to mininize redurdancy,
shoxing the nost
othere.
detined screenshot that shows cenversation Ixhica my involve prongt imege(a)) OppnAl's
conversation
window.
It the image shove any other plattom, it in not a conversation.
If the image generation task is not clear (e-g., the screenshot seens to be using ChatGPT's 1MM rather than inage small), is
ten espabilit lex, the screenshot conversation.
ext renwly low quality, the inagex are ext renely
If the semple coee not contain a conversati ion, set "conversation" to the erpty string *n.
prompt: <prompt>
images: <images>
megessto_rrots:: Lanoyes_to_roster extracted:
Figure E.4: Prompt for multimodal processing with GPT-4o (OpenAI, 2024), described in Section 3.3.
31

---

Instructions
extractor multinodal prompts for inage goneration.
Youz 1ob is to extract the tex: promot and bounding boxes of individual images from. screenshots cenerator. You will also provided relavan, text that aay be belpful desernining input ouspet inages from the scrsenshet.
Please only valid JSON dictionary according this selana: "*ooon (*prampe": <str>, "inputa";
bounding boxes>, "outputa"; bounding boxes>]
extract "pronge";
ingut rxctact should produce fully speel fied coberrnt inage generation Lask; ignazw other isrelevan: text.
extract
exactly.
1f there is no rolevant text, output the empty scring **.
extract "Inputa"
"outputs";
Excract a list of hounding boxes for every individeal inage.
Bach boanding box should be formileed [xl, x2, y2l; bxi, in the Lop left and (x2, y2) is the bottan-right.
Also sort bounding boxes "inguts" va. "outguta"
inage generator.
The exttracted inages should produce fully apecified and cohercal image generation Lask; ignoow olber izzelavanz inaqea.
inage should only appear
duplicetes.
ase "impata" oulpat saply 11.
If there are no "outpuza" ouspuz an anpty 11st [].
televant_iead: <eelevand__eat> images: cimages>
rxclee:
Figure E.5: Prompt for parsing screenshots of conversations with Qwen-2.5-VL (Bai et al., 2025), described Section 3.3.
32

---

F AUTOMATIC EVALUATION METRICS
Overall Metries. In Figure F.1 we depict the prompt used for VLM-as-a-judge in our overall benchmark evaluation. We follow the "single answer grading setup" of MTT-Bench (Zheng et al., 2023), and convert absolute scores into pseudo pairwise comparisons, which can be used to compute the win rate.
Specialized Metrics. We display our prompt for rating the accuracy rendered text Figure F.2, and classifying the applicability of each sample to each specialized metric in Figure F.3.
Instructions
Please act as impartial jodea and evaluate quality image outpat produced an inage generabioon lon modell in response to an ieput nettruction ion (espreased via text and/or image (s)
Bedin your ovalcation by forning given prompt. Beserila the this briwfly before judging the uuruut
good should look 11ke for the
Then the model's outpot with expectation. Point it out inccouanciss, fai lures identify nissing details
the
the ausput satlafy the
instruction.
Rake comsider the following factora equellyy exPoomp Pollowingess acourately?
interoret the correctly and
requested
eiBelaesmoe Fidelityest Does the oukgut preseeve key decails fron the lspal images when relevant? <sBeallan and heatheticave: Is the outpus photorealistic (e.g. accurate anatony, ne artifacta, ate.) visually agpealing (e.gu, belanced
well frllnfrrned copositiio,
when relevant?
Atter providing your expl inplanation, please rate the output on scale of 1 to 10 by strictly tollowing this formet: "[]ret mertt"ggl",, for exeples "Ratiing: [[5]]*.
<IThe Start of Input Inscruction
input, prompe: "uppptpprorge impul__inagest siopat_ Inegwa>
Input Instroction|>
<IThe Start ot Mooel Output output soutput inages images Rnd Hodel Outipe:
Figure F.1: Prompt for automatic evaluation with GPT-4o (OpenAI, 2024), Gemini 2.0 (Team et al., 2023), and Qwen2.5- 1-5.55----32B--Iii (Bai et al., 2025), described in Section 5. Our prompt closely follows the format from MT-Bench (Zheng et al., 2023), but adapted for rating image generation outputs.
Instructions
Check
inage accurate readable,
Vor exect copy reguesta: spelling purctuw: ion, gramee match exec: ly, with no missing or extra charssters, and text is nct croppes.
For created text: content coherent, relevant, and fits available and design. by reading shrough inage Peint out spelling ervors, punctuation, errors, grarmar ecrors, and miseing charsctors of the text. Point out if the text is cropgedd.
"eegin your
Then, laok the
again and check if the text coherent, relevent and fiss the asaillable
design.
Please rate the outpot on a scale so by strictly following this fomets "[[rsting!]", 1", for examples "Bating: [15]]*.
Toprat fnesrustioni>> suput_prompt. <lapat_pronpt>> <TThe End ot Input Instruction!>>
<IThe start of Hedel Output ourget image: coutput inage> <IThe End of Hodal Output./>
Figure F2: Prompt for judging text rendering accuracy with GPT-4o (OpenAI, 2024), described in Sec- tion 5.2.
33

---

Instructions
For mach meitic tn the provided Lisk, decide 10 at las applicbll For the given image genazatiion instruction.
instructions may contain If mecrio nacked as applicable, will used as axia soas and zank oukguba fax given inpat.
instruction be defined with text and/or
Melric Lia- <nstric_
11 Oulput Pornal
Respond only applicable) or lis appl iobble). Also inslude a shoct global rat attinaae for your ovezall decision nakingg prosess. "jpnn
dictionary containing the netric
and value
"mmtrrici>: cinteger 1y
cmneriioltt cinteger 0 or 12,
rratioralo"s *ea shozt sationale, 20 warda lesasr
"prompt" *othe ingut prorpt regested agains
Your Purn
task: stasks
jeput_proepe <i ciopot_ppemnts
inpu:_innages siapat__iaaees>
metrice:
The <inetric list is replaced with the defined metrics name, its description, and its applicahility:
Nana:
'Facs
Prasorvation"
Descriptions
Check 12 the person' a identicy natchas the zeference oz intended pezson, keeping facial atructure and dist net iive features the same. Exerples Penaliue: Changes haleslyle, Length, susss,, Iwcial expression, not match Applicabi loby
referance.
This metzin la often aeplicable, especially for Lasks involving aub driver gensration. applicable
proept dows expl ieptleiily inplicitly request Face identity ity preservstion.
No person's a faow la visible (because thace are nu people, or faues are posl adel))
The task is stylization, where the creative freadon allows for nany valid outputs and corsectneas is
subject
Description:
Check the ovezell color tone, brightness, and contrast match the reterencs or intended look. overexpontra, darkening
Exeplee Fenalize: Added Applicabillty:
refaranse.
This metric is otten applicable, especially tor tacks like local editing.
Hovever, it is not applicable when:
colorization inags-to-inags transiaticn, where color change inaermat the
"Spatial
Preservation"
Descriptions
Check 12 the structure and spatial layout of the sefereace are copted correctly, Including positions, ebjects.
foe layout, and sosle la slightly noved
Exempees to Denelise:
original gosition during stylixation; table
waa centered shifted.
Appli cisiiilllyy Heyy
This metzin is only applisable for Lasks that involving inage to inage Lranslation, stylixation,
editing
requires stries structure preservation.
Hooeeeer, it not applicshle vhens
The pesepi dows not expect the ceaulting inage to be striebly preserving spatial strmeure with the proupt
reference
racture charges (eg, sketoh inage,
stylization!
Nana:
"Iext hendoring Accuracy" Description:
Check rendered text contains mistakes that hinder reasact Exuplee Fenalize: Chassatera garbledi there missing chasaesera) thare Lncorract speiling punctustion: there
ararmar.
Applicabillly:
This natric is often applicable, but only when the prempt explicitly requests rendered text.
Figure F3: Prompt for classifying the of applicable samples for each specialized metric with GPT- 4o (OpenAI, 2024), described in Section 5.2. The number of applicable samples as follows: Face Identity Preservation (244), No Color Shift (271), Spatial Position Preservation (180), Text Rendering (240).
34

---

G LLM DISCLOSURE
Some portions of this work were generated with the assistance of large language models (LLMs). Their primary role was to support editing, rephrasing, and formatting of existing text improve clarity and readability. While human authors created and reviewed the core content, LLMs were used as tool to streamline refinement and presentation. All factual information, analysis, and conclusions remain the responsibility of the authors, and every effort has been made to ensure accuracy and integrity.
35


Reviews:
[{"review_id": "2ZzmqfphDq", "reviewer": "Reviewer_vt4s", "summary": "Existing Benchmarks often mirror the capabilities of models to evaluate and understand how models\nperform on tasks of interest. These benchmarks contain short and simple instructions. Authors present ,\nECHO framework for constructing benchmarks using 35000 prompts created from real world (social\nmedia data- Twitter) and used GPT4o for image generation to identify SoTA model capabilities and its\nalternatives , design metrics , complex task which was absent in existing or previous benchmarks. Curated\nprompts were filtered by filtered posts by designing the relevant keywords and relevance of the post text.\nFurther online discussion may have messy conversation which leads to incorrect dataset collection . To\navoid this authors have used LLMs to turn every messy conversation into one clean, self-contained data\nsample that includes: the prompt, the responses, and how good it is. After text refinement , Multimedia\n(Images) requires VLM (Qwen 2.5 VL). After Analysis , 20% Data marked as High Quality\n(Benchmarking) , 66% Data as Medium Quality (Analysis). Due to policy of twitter and openai some\nposts were refused to download or generate and further data removed manually by the author . Following\ntypes of models used such as Unified Models , LLM+Diffusion , Image Editing Models. Instead Human\nEvaluation for large scale authors used VLM as a Judge. As a secondary Validation , authors present five\nexpert raters with outputs of 8 models for 200 samples and ask the annotators to rank the output from best\nto worst for both image to image and image to text splits. Authors framed ECHO could help to\ndifferentiate model performance in the fine grained ways such as Color Shift Magnitude , Face Identity\nSimilarity , Visual structure such as object positioning or human pose.", "questions": "1. The success of your approach relies heavily on LLMs and CV models for classification and\nextraction. How sensitive are these models to inaccuracies or biases, and what measures are in\nplace to validate their outputs? Could errors in automated classification impact the reliability of\nthe derived metrics?\n2. Are there classes of failures or use cases that ECHO systematically misses due to its reliance on\nsocial media posts or community feedback?\n3. Are there risks that users might intentionally or unintentionally manipulate community feedback\n(e.g., spamming certain prompts or promoting specific outputs) to bias the benchmark? How does\nyour framework mitigate or detect such scenarios?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Paper primary goal is to create a new generative model structured dataset involving users sharing\ninteresting prompts and outputs, novel task ideas, or commentary on model behavior.\n2. Authors present , ECHO framework for constructing benchmarks using 35000 prompts created\nfrom real world (social media data- Twitter).\n3. Designed several specialized automated metrics: color shift magnitude, face identity similarity,\nstructure distance, and text rendering accuracy.", "weaknesses": "1. Community perceptions and discussion topics evolve over time. Benchmarks derived from a\nsnapshot of community feedback may not adapt quickly, risking obsolescence or misalignment\nwith current priorities unless actively maintained.\n\n2. Since social media discussions are shaped by active communities, the collected prompts and\nfeedback may reinforce prevailing stereotypes, biases, or misconceptions. This can lead to\nbenchmarks that unintentionally favor certain demographic groups, styles, or cultural contexts,\nthereby limiting fair assessment across diverse user groups.\n3. Leveraging social media data introduces vulnerability to manipulation, such as spam posts,\ncoordinated misinformation, or artificially amplified feedback, which could skew the benchmark\ntoward certain failure modes or artificially elevate the perceived performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1km79C7akI", "reviewer": "Reviewer_Fp5T", "summary": "The paper introduces ECHO (Extracting Community Hatched Observations), a framework for constructing adaptive, data-driven benchmarks for image generation models based on real-world user interactions on social media. Motivated by the observation that existing evaluation datasets lag behind rapidly evolving generative capabilities, the authors propose leveraging posts from platforms such as Twitter/X to capture emergent tasks, user prompts, and qualitative feedback surrounding newly released models (e.g., GPT-4o Image Gen). ECHO systematically extracts multimodal data, comprising textual prompts, reference images, and community commentary, using a pipeline that integrates large language and vision–language models (LLMs/VLMs) to filter, contextualize, and structure these inputs into benchmark-ready samples.\n\nEmpirically, the authors curate a dataset of over 35,000 social media posts and develop a benchmark subset containing approximately 1,700 text-to-image and image-to-image tasks. Using this dataset, they evaluate eight leading generative models, showing that ECHO better distinguishes performance differences than conventional benchmarks such as GEdit or CompBench. Beyond quantitative evaluation, the framework also transforms recurring user observations (e.g., color shift, identity drift, text rendering errors) into measurable diagnostic metrics, providing a mechanism to align benchmarking with authentic user concerns. The paper concludes that such community-grounded, continuously updatable benchmarks can serve as a scalable and dynamic alternative to static evaluation paradigms for assessing progress in generative modeling.", "questions": "NA", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper articulates a timely problem: benchmarks for generative models cannot keep pace with emerging user behavior. The introduction (pp. 1–2) effectively grounds this issue using the example of “Ghiblification” — a community-invented use case of GPT-4o that no prior benchmark captured. This framing convincingly motivates ECHO.\n\n2. The ECHO pipeline (Figure 2) is the paper’s technical core: Collect relevant social posts, Reconstruct context across replies, Process multimodal data (text + images + screenshots), Filter and classify for quality and benchmarking. The pipeline is modular, well-illustrated, and generalizable. The multimodal LLM+VLM processing steps (using GPT-4o and Qwen-2.5-VL) are particularly novel — they turn noisy social media content into structured benchmark samples.\n\n3 Empirical Validation: The authors benchmark 8 models (open-source and proprietary) across the new dataset. \n\n4 Turning user complaints into quantitative metrics (color shift, face identity, text rendering) is an interesting idea. Figure 8 (page 9) demonstrates these metrics’ power to confirm qualitative observations (e.g., GPT-4o’s “yellow tint” or identity drift). This “closing the loop” contribution elegantly connects community perception and model evaluation.", "weaknesses": "1. ECHO’s exclusive reliance on Twitter/X introduces substantial platform- and demographic-specific biases. As acknowledged in *Appendix A*, trending phenomena such as the “Ghibli style” disproportionately influence the sample composition, leading to a skewed task distribution that may compromise benchmark representativeness and fairness. Consequently, models optimized for highly visual or viral content may appear to perform better under this framework. Furthermore, the pipeline is evaluated solely on Twitter/X, without empirical validation across alternative social platforms (e.g., Reddit, Discord, or YouTube), thereby limiting the generalizability of the proposed approach and undermining its claim to universality.\n\n2. Although ECHO is described as “re-runnable,” the paper lacks sufficient methodological transparency for full replication. Critical details concerning data acquisition—such as API endpoints, scraping protocols, temporal sampling strategies, and filtering heuristics—are not disclosed. While Figure D.1 enumerates example keywords, key parameters governing data retrieval, LLM-based relevance scoring, and post-filtering thresholds remain unspecified. This omission impedes reproducibility and may also raise compliance concerns regarding Twitter/X’s data use policies. Moreover, the reuse of user-generated content (including images and textual comments), even in anonymized form, poses potential ethical and legal challenges under data protection regulations such as the General Data Protection Regulation (GDPR).\n\n3. Despite the use of an ensemble of evaluators (GPT-4o, Gemini, and Qwen), the low Kendall’s τ correlation with human judgments (τ ≈ 0.10–0.12) indicates that current VLM-as-a-judge paradigms remain insufficiently reliable for fine-grained assessment of generative quality. Additionally, although approximately 35,000 posts are collected, only ~1,700 high-quality samples are retained for benchmarking. This limited subset, especially when contrasted with larger-scale datasets such as DiffusionDB (≈14 million prompts), raises concerns regarding statistical robustness and the stability of model performance differences. It remains unclear whether the reported distinctions reflect genuine capability gaps or are artifacts of small-sample variance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uRmsZZzqjd", "reviewer": "Reviewer_dbsG", "summary": "The paper introduces a framework ECHO, that constructs benchmarks by directly probing social media posts from platforms like twitter. The framework is applied to GPT 4o Image Gen, and thereby a collection of 35000 prompts are curated, which are those that are directly used by real-life users. Resultantly, the benchmarks consists of complex and realistic tasks, not included in previous benchmarks. By applying such prompts, state-of-the-art generative models are re-evaluated, and the discovered indicators from the posts help evaluate the models further.", "questions": "1. How many images per model did the authors evaluate on the constructed benchmark?\n2. Did the authors try the traditional fidelity and faithfulness metrics? Wherever applicable, how diverse are the generations by the different models? This question is important as the prompts are unique in the proposed benchmark.\n3. Counting and hallucinations have been raised by users as failure cases, and these have been well-established problems in literature. Did authors try measuring them, especially for newer models like FLUX Kontext, GPT etc?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The benchmark is unique, and is of utmost importance, as it ties the general public opinion with model performance. \n2. The authors have put in substantial efforts to extract the prompts from the complex tweet chains.\n3. The framework uncovers significant model failures observed by the users, as shown in Fig. 4.", "weaknesses": "1. While I agree that the large-scale images generated from the prompts cannot be manually evaluated, the VLM-human correlation seems quite low, raising questions on the evaluations.\n2. It is unclear how many images have been generated by each generative model for evaluation. \n3. The benchmark consists of several tasks, while Fig. 6 summarizes them. It would be good to capture how the different models on the individual tasks, as virtual try on, novel view synthesis are themselves quite significant as tasks.\n4. Fidelity, Faithfulness and Diversity - the three most important metrics related to text-to-image generations should have been discussed in more detail.\n5. [minor] The resulting observations are similar to the expected results - the closed-source models generally outperform the open-sourced ones. However, that being said, the curated prompts seem useful, as they are obtained from real-life users.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lip6j2YCuQ", "reviewer": "Reviewer_eprr", "summary": "This paper addresses the growing gap between the rapid progress of image generation models and the static nature of existing benchmarks. It introduces ECHO, where instead of relying on predefined and manually curated tasks, the framework builds its challenges by mining social media posts about new image models (the case study for this paper is GPT-4o Image Gen) to construct an in-the-wild, re-runnable benchmark. It then details an LLM and VLM-based processing pipeline, where the authors extract structured <input, output, feedback> tuples from messy social media threads and curate a benchmark of around 35K samples. The paper demonstrates that ECHO captures creative, complex, and evolving tasks not present in existing benchmarks, provides stronger differentiation among models, and motivates the development of new quantitative metrics (e.g., colour shift, text rendering accuracy, identity preservation) based on community feedback.", "questions": "1. The framework currently relies on (presumably) Western-centric, English-language social media. How feasible or what challenges would arise to extend ECHO to non-English or region-specific platforms and mitigate this sampling bias. \n2. Can the authors provide a more granular breakdown of the benchmark's composition, as suggested in W2. An analysis of task clusters would be far more insightful than bigrams and would help identify which specific capabilities ECHO is truly testing and how balanced it is.\n3. Based on this analysis, which tasks in the benchmark are most difficult (i.e., highest failure rate for all models) and which are comparatively easier? This would be invaluable for guiding future research.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper's central premise (benchmarks must dynamically evolve with model capabilities and user behaviours) is a highly significant and timely contribution. The idea of using community-generated evidence from social media as a source is a novel solution. \n2. The paper is clearly written, visually rich, and easy to follow. The proposed pipeline (ECHO) is detailed and well-motivated. The analysis demonstrates how ECHO surfaces novel tasks, uncovers model failure modes, and differentiates state-of-the-art models.\n3. It is an important contribution to how we evaluate image generation models. The framework could inspire new adaptive benchmarking protocols in other modalities. \n4. Traditional benchmarks for image generation often rely on abstract metrics that correlate poorly with human preference. ECHO’s feedback-derived metrics aim to capture human-relevant error dimensions.", "weaknesses": "1. Although ECHO collects a large and diverse dataset, the quality control process heavily depends on LLM and VLM filtering. The paper does not provide any quantitative validation on how accurate the pipeline is. How do the authors handle bias or noisy samples in their final benchmark? The fill-in-the-blank prompts (Section 3.3) could also introduce hallucinated prompts. \n2. The authors analyze bigrams to assess the linguistic diversity of the benchmark, but this is a coarse view of the benchmark. A more informative approach would be to use an LLM to cluster each datapoint into broader task categories, which will be more meaningful to understand the benchmark distribution. It would also reveal potential imbalances, there could be an overrepresentation of certain tasks, which is not currently captured in the analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)