ImagenWorld: Stress-Testing Image Generation Models with Explainable Human Evaluation on Open-ended Real-World Tasks
Authors: Samin Mahdizadeh Sani
Max Ku
Nima Jamali
Matina Mahdizadeh Sani
Paria Khoshtab
Wei-Chieh Sun
Parnian Fazel
Zhi Rui Tam
Thomas Chong
Edisy Kin Wai Chan
Donald Wai Tong Tsang
Chiao-Wei Hsu
Lam Ting Wai
Ho Yin Sam Ng
Chiafeng Chu
Chak-Wing Mak
Keming Wu
Hiu Tung Wong
Yik Chun Ho
Chi Ruan
Zhuofeng Li
I-Sheng Fang
Shih-Ying Yeh
Ho Kei Cheng
Ping Nie
Wenhu Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Image Generation
Image Editing
Evaluation
Benchmark
Primary area: datasets and benchmarks

Abstract: Advances in diffusion, autoregressive, and hybrid models have enabled high-quality image synthesis for tasks such as text-to-image, editing, and reference-guided composition. Yet, existing benchmarks remain limited, either focus on isolated tasks, cover only narrow domains, or provide opaque scores without explaining failure modes. We introduce \textbf{ImagenWorld}, a benchmark of 3.6K condition sets spanning six core tasks (generation and editing, with single or multiple references) and six topical domains (artworks, photorealistic images, information graphics, textual graphics, computer graphics, and screenshots). The benchmark is supported by 20K fine-grained human annotations and an explainable evaluation schema that tags localized object-level and segment-level errors, complementing automated VLM-based metrics. Our large-scale evaluation of 14 models yields several insights: (1) models typically struggle more in editing tasks than in generation tasks, especially in local edits. (2) models excel in artistic and photorealistic settings but struggle with symbolic and text-heavy domains such as screenshots and information graphics. (3) closed-source systems lead overall, while targeted data curation (e.g., Qwen-Image) narrows the gap in text-heavy cases. (4) modern VLM-based metrics achieve Kendall accuracies up to 0.79, approximating human ranking, but fall short of fine-grained, explainable error attribution. ImagenWorld provides both a rigorous benchmark and a diagnostic tool to advance robust image generation.

Published as a conference paper at ICLR 2026
IMAGENWORLD. STRESS-TESTING IMAGE GENERA- TION MODELS WITH EXPLAINABLE HUMAN EVALUA- TION ON OPEN-ENDED
TASKS
1 Samin Mahdizadeh Sani" 11 Max Ku Nima Jamali Matina Mahdizadeh Sani $ Parii
Khoshtab Wei-Chieh Sun Parnian Fazel Zhi Rui Tam 8 Thomas Chong ? Edisy Kin Wai
Chan Donald Wai Tong Tsang" 2 Chiao- Wei Hsu Ting Wai Lam Ho Yin Sam Ng Chiafeng
Chu 2 Chak- Chak-Wing Mak
Li I-Sheng Fang &f Keming Wu Hiu Tung Wong", Yik Chun Ho " Chi Ruan Zhuofeng
Shih-Ying Yeh $0
Ho Kei Cheng 08 Ping Nie #Wenhuu Chen
University of Waterloo
G-G-G
Comfy Org Independent
University of Illinois Urbana-Champaign
https:://igger-a wtt.lllbbgggbbbb. io/ ImagerWorld. htb.gitthbbbbbomm/ggglllllldddd
ABSTRACT
Advances in diffusion, autoregressive, and hybrid models have enabled high-quality image synthesis for tasks such text-to-image, editing, and reference- reference-guided com- position. Yet, existing benchmarks remain limited, either focus on isolated tasks, cover only narrow domains, provide opaque scores without explaining failure modes. We introduce Imagen World, benchmark of 3.6K condition sets spanning six core tasks (generation and editing, with single multiple references) and six topical domains (artworks, photorealistic images, information graphics, textual graphics, computer graphics, and sereenshots). The benchmark is supported by 20K fine-grained human annotations and an explainable evaluation schemn that tags localized object-level and segment-level errors, complementing automated VLM-based metrics. Our large-scale evaluation of 14 models yields several in- sights: (1) models typically struggle more in editing tasks than in generation tasks, especially in local edits. (2) models excel in artistic and photorealistic settings but struggle with symbolic and text-heavy domains such as screenshots and informa- tion graphics. (3) closed-source systems lead overall, while targeted data curation (c.g.. (wen-Image) narrows the gap in text-heavy cases. (4) modern VLM-basee metrics achieve Kendall accuracies up to 0.79, approximating human ranking, but fall short fine-grained, explainable error attribution. Imagen World provides both rigorous benchmark and a diagnostic tool advance robust image generation.
INTRODUCTION
The rapid progress in generative image modeling, powered by diffusion (Rombach et al., 2022 Lipman al. 2023), autoregressive (AR) (Yu al. 2022 Tian al. 2024). and hybrid architec- tures (OpenAl 2025), has enabled systems capable of producing high-quality images under diverse conditioning inputs. More recent work has begun push toward broader functionality, developing models that can handle multiple tasks- such as generation and editing-wiite a single frame- work (Deng et al. 2025 Wu et al.) 2025b Chen al.) 2025a Google 2025). with early evidence of real-world applicability (Chen et al.] 2025a). However, evaluation has not kept pace with this modeling progress. Existing benchmarks are fragmented, often restricted to isolated tasks ge.g., text- to-image (Saharia et al. 2022, Yu et al., 2022), editing (Huang et al. (023), or personalization (Peng) 2025 Li ct al.| (023) ) or biased toward narrow domains such as artworks (Ku et al. 2024b) or textual graphics (Tuo et al. 2024). As result, it remains unclear how well these unified models generalize across the full spectrum real-world use cases. To address this gap, we introduce
ct al.
"qqual contribution
Project Lead
M samin samin.mahdizadchee gmail.com; (m3ku, wenhu.chen) @uwaterloo.ca

---

Published as a conference paper at ICLR 2026
Dotoset Creation
Object and Segment Extraction
Datoset Evaluation
TASK
Object Extraction
Sconest
Proept Relevance 0.78
Text-guided imge geniration
Aesthetic Quality 0.75
Single reference Inage genenction
Condition set generatiion
Superman Cat Text
Human
Coherence:
Multiple reference image generation
Artifact:
Text quided mage editing
Gemini
Object Issues:
Single reference image editing
Muitiple reference image editing
Segment issues:
TOPIC
Image
Artworks
Photorealistic Images
Scorest
Information Grophics
VLM
Prompt Relevance: 0,70
Texthul Graphics
Quality:
Computer Grophics
Centent Cohorence: 1
Screenshots
Segment Extraction
Artifact
Output image
Figure 1: Overview of our dataset and evaluation pipeline, covering six content categories and six generation/cdiinng tasks. Model outputs are assessed by both human annotators (cxplainable schema) and vision-language models (scores only).
Imagen World, large-scale, human-centric benchmark comprising 3.6K condition sets designed to systematically stress-test generative models. Imagen World unifies six representative task types and six topical domains, creating a diverse testbed that mirrors the breadth of real-world image generation and editing. At its core, Imagen World relies on structured human evaluation, where annotators not only provide scores but also tag specific failure modes with textual descriptions and localized masks as illustrated in Figure [ This schema yields explainable outcomes, revealing why models fail. To complement human judgments, we also include VL-Mas--a-jude metrics, enabling comparison between human and automatic evaluators. Together, this design supports both rigorous benchmarking and forward- looking exploration of how evaluation protocols can scale. By evaluating a broad set of model families under single protocol, Imagen World provides the most comprehensive picture to date of model performance and failure patterns across real-world generation and editing tasks. Our study covers 14 models in total, including 4 recent unified models capable of both generation and editing, and 10 task-specific models that serve as auxiliary baselines.
We uncover four key insights: (1) For editing tasks, we identify two distinct failure modes: (i) regenerating an entirely new image, and (ii) returning the input unchanged. Strikingly, models tend to exhibit one mode far more frequently than the other, suggesting systematic bias in how they interpret editing instructions. This highlights a deeper limitation in current architectures: they lack fine-grained control mechanisms to modify localized regions without either overhauling or ignoring the input. (2) All models struggle with text-related tasks such as information graphics, screenshots, and textual graphics. However, our results reveal an exception: Qwen-Image consistently outperforms other models on textual graphics. Notably, Qwen-Image employs synthetic data curation pipeline explicitly tailored for text-heavy images, suggesting that targeted data augmentation may be a practical path to closing this gap. This highlights that the challenge is not purely architectural, but also fundamentally tied to data design.
(3) While closed-source models consistently achieve strong results across tasks, open-source models are primarily competitive in text-to-image generation, where abundant training data and community optimization have driven rapid progress. Their weaker performance in editing and multimodal composition highlights the need for further research and targeted data curation in these areas, beyond scale alone. (4) Beyond model performance, we find that modern VLM metries achieve Kendall accuracies up 0.79, closely matching or even exceeding human- numan-human agreement. This suggests that modern VLMs as a judge are reliable as scalable evaluators for relative ranking in our context, but they fall short in the explainable paradigm, where humans remain indispensable for fine-grained tagging of specific failure modes.
Our contributions are threefold: (1) we introduce Imagen World, diverse benchmark that unifies six core tasks and six topical domains, enabling consistent cross-model and cross-task evaluation of generative image systems; (2) we conduct the first human study of its kind, looking into the failure modes, offering new insights and observed patterns. (3) we propose a schema for explainable human evaluation, labeling object-level errors and segment-level errors to provide fine- mme-indi interpretable error attribution beyond scalar scores. By combining task diversity, model breadth, and diagnostic evaluation depth, ImagenWort World establishes a unified and human-centric study to record our process towards the full control of creation and manipulation in images.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORKS
Progress in Conditional and Multimodal Image Synthesis. The introduction of Latent Diffusion Models (LDMs) (Rombach et al. 2022) marked turning point, leading to a flourishing ecosystem of
conditional image synthesis systems frunwayml [2023] (stability.ai (203) spanning diverse tasks such
as iustructioondriver editing (Brooks et al. [2023a]] Huang et al. 2025), structural control Zhang & Agrawala 2023). and personalization (Ruiz et al. 2023] Yeh et al.) 2024] Hu et al. 2024).
While diffusion remains the dominant paradigm, alternative architectures
rapidly advancing.
Autoregressive approaches (Yu et al.) 2022 Tian et al. 2024) improve compositional reasoning and fidelity (Xiong et al., 2025), flow-matching models (BlackForestLabs
al.) (202) leverage
ODE-native properties for potentially faster sampling, and hybrid designs such as autoregressive
LLMs with diffusion decoders (Wu al.] 2024 OpenAT 2025 Google 2025 integrate native
image generation into conversational agents. Together, these families define the current landscape of multimodal conditional image synthesis, though their evaluation remains fragmented across tasks and settings. Our work takes these developments into account by systematically studying their strengths and weaknesses under a unified evaluation framework.
Image Synthesis Assessments and Benchmarks. Traditional evaluations of generative image models have relied on metrics such as FID (Heusel et al. 2017) and LPIPS (Zhang et al. 2018) for image fidelity, or CLIPScore (Hessel et al., 2021) > for imxt- alignment. More recent approaches, including VIEScore and VQAScore (Cho et al. 2023 Hu et al. 2023] Ku et al. 2024a Lin et al. 2024 Niu et al.) 2025). use vision language models (VLM) to better capture semantic relevance, though they introduce biases and often depend on proprietary models. Human preference-driven metrics such Pick-a-Pic (Kirstain et al., 2023a), ImageReward (Xu et al., 2023), and HPS (Mal ct al. 2025) emphasize aesthetics and subjective preferences. Beyond individual metrics, benchmarks like DrawBench (Saharia al.) 2022) and PartiPrompts (Yu al., 2022) target text-to-image fidelity, while others focus on editing (Huang et al. 2023) or personalization (Peng et al. 2025 Li et al. 2023).
More recent efforts, including ImagenHub (Ku al. 2024b) and SAMI--Bence (Hua al. 2025).
extend beyond single tasks by covering multiple generation settings and integrating both automatic and human evaluation. Gecko (Wiles et al., (025)) further scales this direction by introducing large evaluation suite that measures text-to-image alignment across diverse human annotation templates and scoring setups. Open platforms like GenAl-Arena (Jiang et al., 2024) provide Elo-style rankings, but suffer from topic bias in user-submitted prompts. Overall, existing protocols remain task-specific or opaque, limiting the interpretability and scalability. Beyond simply adding another dataset, our work offers new perspective: unified benchmark across tasks and domains, complemented by structured, explainable human evaluation that can also serve as a foundation for future TLM-based automatic evaluators. Table 1 reflects how Imagen World differs from prior works.
Table Comparison with different evaluation benchmarks on different properties.
Generation Single Ref Multi Ref Human
Tople Rating Variety
Explainable
Method
& Editinn
Guided
Guided
Trace
ImagenHub (Ku al 2024b
GenAl-Arent Uiaan et 2014
x
x
/
DreamBenchhe [[enr ct 2025
x
x
x
I'EBench (Ma et al. 2024
x
X
/
ICF-Bench (Pinn
2025
/
x
MMIG. Bench (Fl03
Kich-HF (Ciang ar
x
x
x
ImagenWorld (Ours)
/
3 THE IMAGENWORLL BENCHMARK
Problem Formulation. To capture practical usage scenarios, we unify multiple generation and editing tasks under common instruction-driven framework: every task is conditioned on natural language instruction fus optionally accompanied by auxiliary inputs such as a source image Isce or a set of reference images LR. This unification reflects how real users typically interact with generative systems by providing instructions that guide the system to either create new images or edit existing ones. Building on this formulation, we categorize tasks into two groups: instruction-drive driven generation,
3

---

Published as a conference paper at ICLR 2026
Task
Text Instruction
Source Image
Image Conditions
Success Case
Failure Case
scese stawing bounding antmmlss such as deec, rabbits, birds.
(CG)
SRIG (P)
brain using reference mask anatomical shape.
Generat pet simdlation gume interface with pet charac ler designs from the first and second the home image.
MRIG (8)
style third
TIE
Change text to "T learns from humans" and the bottom "Whe's cooching cosch?"
Re-color the stylined portmat in image 1 using exclusively the specific color poletto prosentod in Image
SRIE (A)
step update added pasta to resemble speghetti noodles as shown in image 2. For steps and 2, rečine the water graphic, depicting with the pattern style found image
MRIE ①
Figure 2: Illustrative examples from our dataset, showing successful and failure cases for each task
where the system synthesizes new image without source image, and instruction- driven editing, where the system modifies an existing source image Ire while following the instruction. Formally:
Text-guided Image Generation (TIG): Given an instruction liss natural language, the model synthesizes new image = f(ims)
Single Reference Image Generation (SRIG): Given an instruction fins and a reference image 4ef, the model generates new image y = f (Lief, tins) of the referenced entity (e.g., subject, object, or layout) in different context, pose, or environment.
Multiple Reference Image Generation (MRIG): Given an instruction tims and a set of reference images R [ .),, the model synthesizes new image fI tins) that composes multiple visual concepts from the instruction and references.
Text-guided Image Editing (TTE): Given an instruction lins and source image Jac the model produces ftnms Isre) by modifying Lxee according to the instruction while preserving its core structure.
Single Reference Image Editing (SRIE): Given an instruction fams a source image J and reference image Teef the model edits Tge to = f[Tats lins: Js adapting the reference entity to the specified instruction or style.
Multiple Reference Image Editing (MRIE): Given instruction Lmss a source image Lsce and set of reference images Ig. the model edits Iue to fI fims Jme), aligning it with the visual attributes semantics suggested by the instruction and references.
Dataset Curation Pipeline. To construct Imagen World, we curated large-scale dataset through a combination of human annotation and automated refinement. Annotators wrote natural language prompts and paired them with corresponding reference or source images, ensuring that each instance aligned with one the six benchmark tasks. To reflect real-works applications, our dataset covers six major topics: Artworks (A), Photorealistic Images (P), Information Graphies (I), Textual Graphics (T), Computer Graphics (CG), and Screenshots (S), each further divided into fine- grained subtopics to guarantee diverse cases. In total, our dataset contains 3.6K entries, with 100 samples for each task-topic combination. Figure shows representative examples from our dataset (see Appendix A.6 for details).
4

---

Published as a conference paper at ICLR 2026
User Query
Scores:
Prompt Relevance: 0.75
Aesthetic Qualitty: 0.75
chandelier to ceiling of the room. Match style color scheme of the original painting.
Content Coherence: 0.75
Artifacts:
Object Issues:
chandelier in the upper-middle section, appearing to hang from the ceiling
Expected
Model Output
Segment Issues:
Figure 3: Examples include object-level issues, where expected objects are missing or distorted, and segment-level issues, where SoM partitions highlight specific regions with visual inconsistencies that affect evaluation scores.
4 EVALUATION SETUP
Scoring Criteria. Our evaluation relies on four criteria that measure Prompt Relevance, Aesthetic Quality, Content Coherence, and Artifact, capturing complementary aspects of image generation and editing quality, similar to prior works (Xu et al. 2023] Ku et al., 2024b). Each criterion is rated on 5-point Likert scale poor, excellent) and later rescaled the range [0,1]. The definitions of the scoring dimensions are:
Prompt Relevance: Measures whether the image faithfully reflects the instruction.
Aesthetic Quality: Evaluates the overall visual appeal and design (c.g.. overcrowded or poorly aligned elements, poor color schemes, inconsistent fonts).
Content Coherence: Assesses logical and semantic consistency (e.g., labels pointing to the wrong region, a chart titled "growth" showing decreasing values, figure labeled "Parts of a Flower" depicting tree anatomy).
Captures visual flaws and technical issues caused by generation errors (e.g., distorted or gibberish text, warped edges, extra limbs, unnatural eyes, or repeated patterns).
Artifacts:
Each criterion was evaluated both by human annotators and by automated scorers. Specifically, each image was rated by three annotators independently, while LLM --seedd scores were obtained using the Gemini-2.5-Flastt model following the VIEScore (Ku et al. 2024a paradigm, which produces ratings aligned with the same four criteria. In addition, CLIPScore (Hessel et al. (021] and LPIPS (Zhang al. 2018 were computed auxiliary automated metrics assess image quality.
Explainability via Object and Segment Issues. While many prior works focus on evaluating image generation quality, few consider the explainability of evaluation scores (Chen et 2023 Ku et al.) 2024a). To improve interpretability, we define two complementary еrror taxonomies: object-level issues text and segment-level issues in image. In addition to assigning ratings on the four criteria. annotators were asked to identify which objects or regions in the image negatively influenced their scores. For object-level issues, the instruction together with any source reference images from our dataset were given to Geminni-2.5-Flase and the model was queried to generate list of objects expected to appear in the output image. Annotators reviewed this list and marked any objects that were missing, incorrectly rendered, distorted. For segment-level issues, each generated image was partitioned into regions using the Set-of-Mark (SoM) (Yang et al. 2023). and annotators selected any segments that contained visual flaws inconsistencies, thereby identifying specific areas of the image responsible for score deductions. Figure illustrates examples of both object-level and segment-level annotations from our dataset.
5

---

Published as a conference paper at ICLR 2026
Table 2: Task coverage and architectural categorization of the evaluated models are shown in chronological order. We classify multimodal large language models form of autoregressive model, and group flow matching within the diffusion family, following recent literature (Xiong et al.) 2025 Chang et al. (025). For closed-source models where architectural details are unavailable, we label them as Unknown.
Model
Architecture
TIG TIE SRIG SRIE
MRIG MRIE
InstructPix2Pix (Brooks et
2023b
Diffusion
K
K
SDXL (Podell nl 2025
Diffusion
X
X
Infinity (Hal et 0i 2025
AR
x
X
X
(Clter el. 2025b
Diffussio
GPT
Image (UpenAL
4022
Unknown
/
V
WWa et M17250
Diffissin
/
BAGGE [Dong al. 2025
AR
J
V
Step1X-Eair Eai ct 2022
AR + Diffusion
X
X
IC-Edit (Zang
Diffusion
X
Gemini 2UTiash [WD al.
2025c
Unknown
V
V
OmniGen22 [Wu
Diflisioo
V
Flux. -lux.I-Krea-dey [ [l 2025
Diffusion
X
X
X
Flux. I-Kontext-dev Foreslorst
2025
Diffusion
Qwen-Image (Wu et al. 20253)
Diffusion
X
X
X
Baselines. We evaluate Imagen World using models from three major architectural families: diffusion models, autoregressive models, and hybrids that combine AR with diffusion decoders. Table 2 lists the evaluated models, their architectural families, and their task coverage. This set spans both unified models capable of all six tasks (GPT-Image-1 (Chen et al. 2025a). Gemini 2.0 Flash (Google 2025). BAGEL (Deng al. 2025), , OmniGen2 (Wu et al. (2025b) and expert models specialized for subsets of tasks (Wu et al. 2025a BlackForest et al. 2025 Liu et al. [2025] Han et al. 2025). This setup enables broad comparisons across architectures and between unified and expert approaches.
5 RESULTS AND ANALYSIS
We summarize our quantitative findings in Table 3 which reports human evaluation scores and VLM predictions across six tasks and four criteria. The subsections below analyze these results by task, topic, and evaluation metric (see Appendix A.5 for statistical tests).
5.1 QUANTITATIVE ANALYSIS
Task Level. Across the six benchmark tasks, there is a clear gap between open- and closed- models as reflected in Figure 4 GPT-Image-1 achieves the strongest overall performance, outper- forming Gemini 2.0 Flash by about 0. 1-12 points on average. This margin is particularly pronounced editing tasks, where Gemini falls further behind. Despite this average gap, scale alone doesn't determine success: several open-source models (c.g.. Krea-Kee-dee, Qwen-Image, Flux- 1-Kontext- Dev) outperform Gemini on text- eex--guided generation and editing, showing that larger scale doesn't 't always yield superior results. However, none of the open-source unified models catch up with close-source models, as seen in Figure 5) Beyond scale effects, the distribution of scores further highlights differences in task difficulty: models consistently achieve lower performance on editing tasks (TIE, SRIE, MRIE) than on their generation counterparts (TIG, SRIG, MRIG), with an average gap of roughly 0.1 This pattern suggest that while generation has benefited from scaling and improved reasoning integration, localized modification remains major bottleneck.
Topic Level. Performance also varies substantially across topical domains. Figure 4 presents topic- level statistics (see (A.rddixAAA4fooo detailed results across the full model set). Among the six defined topics, Artworks (A) and Photorealistic Images (P) stand out as the most successful, with averages approaching 0.78 and the best model (GPTT-maag-1) 1) reaching around 0.9 in both categories. This reflects notable progress in rendering high-fidelity content naturalistic and stylistic domains. In contrast, structured and symbolic topics reveal much larger gaps. Textual Graphics (T) and Computer Graphics (C) both average near 0.68, while Screenshots (S) and Information Graphics (I) remain the most challenging, with averages closer to 0.55. Overall, these findings underscore persistent weaknesses in handling text-heavy and symbolic content, highlighting the need for targeted advances in these domains. We also observe similar trend but more obvious gap in unified models (Figure 5).

---

Published as a conference paper at ICLR 2026
Table All evaluated models across six core tasks with bolded maxima for the four human metries and overall in each task. Ottuman denotes the predicted overall. a is Krippendorff's alpha for inter-rater reliability and ps is the Spearman's 's rank correlation for human-vim alignment.
overall average; ÓVLM is the VLM-
Human
Overalls
Agreement
Model
Prompt Relevance
Aesthetic Quality
Content t
Artifacts
Ouumnnn
ÖVLM
P Spear
Coherence
Kripp
Text guided
Image Generation (TIG)
GPT-Inage-1
0.90::0114
0.92::0.10
0.91::0.13
0.92:0012
0.91:10.10
0.12:00.3
0.52
0.39
Qwen-Image Image
0.82+0.22
0.92--0111
0.88+0.17
0.85+0.22
0.87++0.15
0.86-0.20
0.75
0.76
Flux.1-Kree --ve--Krea-dev
0.77+0023
0.85+00.16
0,84:+018
0.80=0022
0.82++0.17
0.82=0.23
0.67
0.79
Gemini-2.00
0.79=0.21
0.78++0.21
0,83=0019
0.79=0.25
0.80+00199
0.82=0.24
0.67
0.70
UNO
0.75:50.21
0.7710.19
0.83:0.17
0,76=0026
0.78:10.18
0.72=0.27
0.72 0.73
BAGEL
0.72::0.24
0.80+:0.0
0.82:10.11
0.73=0029
0.76:10.21
0.75=00266
0.74 0.78
OmniGen2
0.67::0.2
0.77::0.23
0.78:10.24
0.72::032
0.74:10.23
0.75=0027
0.71 0.80
Infinity
0.64+-0.30
0.74+0.23
0,76++0.24
0.69-0031
0.70+0.24
0,73+0027
0.78 0.81
SDXL
0.55:+0.29
0.70+-0.5
0,70-40.27
0.61-0.31
0.64+0.25
0.65=0.30
0.75 0.79
Janus Pro
0.62++0.30
0.62+0.30
0.62:40.3
0.60=0.29
0.61::0.29
0.54=0.32
0.77
0.74
Text-guided Image Editing (TIB)
GPT-Image-1
0.17
0.15
0.16:
0.79
0.82-10.15
0.80=0.28
0.65
0.58
Flux.1-Ke -onttextt-d
0.52440.31
0.76++0.2
0.75+40.2
0.76+0026
0.70+40.21
0.63-0.32
0.68
0.71
Gemini-2.00
0.62:+0.29
0.68:0.25
0.75=#024
0.70=00288
0.69::0.23
0.64=0.35
0.70
0.64
BAGEL
0.59:0.228
0.6410026
0,73:10.23
0.70=0.25
0.67=:0.22
0.59=0.33
0.62
0.66
OmniGen2
0.38:10.31
0.6310.29
0.67:10.30
0.6620.31
0.58:10.27
0.57=0.32
0.77
0.72
IC-Edit
0.38:10.28
0.541:0.33
0.56:10.22
0.52::0.4
0.50:10.22
0.49:00299
0.65
0.58
Step1xEdit
0.34++0.28
0.461:0.29
0.51+0.29
0.51+00.2
0.46--0.26
0.43-1.33
0.73
0.71
InstructPix2Pix
0.26+-0.24
0.48++0.25
0.56++0.26
0,51+00.7
0.45+0.22
0.41=0030
0.64
0.66
Single Reference Image Generation (SRIG)
GPT-Image-I
0.79:10.20
0.86:10.16
0.86:1018
0.85--0.18
0.84:0015
0.86=0.18
0.61
0.48
Gemini-2,0-Flast
0.62=00226
0.65+00244
0,74=10.24
0.69=0.28
0.67=00.2
0.70=0.28
0.66
0.60
BAGEL
0.57:20.24
0.6710.25
0.69:1.24 24
0,64
0.6410.22
0.65=0.28
0.66
0.66
UNO
0.57:10.25
0.621:0.3
0.65:10.23
0,59=0.25
0.6110.211
0.60=0.29
0.65
0.65
OmniGen2
0.41:10.26
0.61
0.65::0.26
0.64:0.28
0.58:00.22
0.62:0030
0.65
0.74
Single Reference Image Editing (SRIE)
GPT-Image-1
0.7310.22
0.80110.20
0.8611.117
0.80=0.22
0.80110116
0.76=0.29
0.63
0.46
Demini-2.0-Flaar
0.44:10.29
0.60100.4
0.69:1..2
0.63=0.26
0.5911021
0.54=0.32
0.64
0,66
BAGEL
0.35:10.29
0.58:10.5
0.62:10.26
0.65
0.55::0.2
0.53:00.2
0.56
0.60
OmniGien2
0,30+0.27
0.60+-0.77
0.62+0.28
0.62+0.30
0.54++0.24
0.54-0..31
0.66
0.72
Multiple Reference Image Generation (MRIG)
GPT-Image-1
0.80+0.17
0.86+-0.55
0.86-10188
0.85
0.84-10.15
0.88-0.16
0.51
0.39
Gemini-2.0- Flash
0.69::0.5
0.7240.24
0.81=1.20
0.72-0.29
0.73++0.2
0.68=0.30
0.73 0.73
BAGEL
0.56:0.26
0.631:.26
0.6620.26
0.60=0.30
0.61::0.24
0.59=0.31
0.67 0.65
OmniGen2
0.47=0.26
0.66110.2
0.66:10.26
0.65=0.28
0.61110.21
0.59=0.28
0.66
0.70
UNO
0,53=20.22
0.61+:.20
0.67:10.20
0.60=0022
0.601:0.18
0.58=0.25
0.58
0.54
Multiple Reference Image Editing (MRIE)
GPT-Image-I
0.72:10.19
0.8210.18
0.82:10.20
0.80-0021
0.79110.17
0.75=0.27
0.66
0.55
Gemini-2.0- Flash
0.49::0.25
0.6610.21
0.69:1022
0.64=0.25
0.62+0.19
0.51=0311
0.62
0.56
OmniGen2
0.32+0.20
0.5611025
0.55:10.88
0.60:0.28
0.5110.22
0.45=0.30
0.69
0.66
BAGEL
0.28::0.22
0.44:10.2
0.48:10.29
0.51
0.4310023
0.38=0028
0.67
0.64
Evaluation Criteria. Across the four defined metrics, we observe distinct patterns in model behavior from Figure 4 Prompt Relevance shows the largest variability across tasks, peaking in TIG (0.72) but dropping to 0.46 in editing tasks on average, underscoring the difficulty of aligning edits with instructions. Aesthetic Quality and Content Coherence are more stable, with maximum task-level gaps of 0.17 and 0.16, respectively. Both metrics (Aesthetic Quality/Content Coherence) achieve their highest values in Artworks (0.79/0.79) and Photorealistic Images (0.82/0.82) but decline in symbolic settings such as Screenshots (0.58/0.63) and Information Graphics (0.58/0.59). Artifact suppression appears more uniform at the task level (gap- = 0.11), yet topic-level analysis reveals clear asymmetry: non-symbolic content is largely free of distortions, whereas text-heavy categories frequently suffer from unreadable corrupted elements. Taken together, these results show that instruction following is the primary bottleneck across tasks, while artifact control remains the central challenge in text-intensive domains, particularly Screenshots.
7

---

Published as a conference paper at ICLR 2026
Motric
100
0.48
0.58
0.59
0.54
0.53
0.72
0.79
0.80
0.75
0.76
0.
-19
0.67
0.79
0.82
0.83
0.78
TH
0.48
0.63
0.67
0.64
0.61
0.52
0.58
0.63
0.47
0.55
530
0.59
0.68
0.72
0.68
0.67
0.35
0.74
0.04
0.67
0.02
18
0.65
0.84
-05
MR1
0.61
0.70
0.68
0.68
115
0.59
0.69
NTC
0.62
0.59
Andiedi
Cattort Hetre
400x
Numon
Pores
Accther
Cantart Hetri
AntRcc
Puman
Figure 4: Mean human evaluation scores across metrics by topic (left) and task (right).
1.0
3.51
CPT
3.54
Image-2
0:54
1.00
0.7
BAGGLL
GAOGL
0.70
0.55
0.59
BS1
0.3
guiced Image Generatian
Tea guided Image corting
Belerene Image Genaration
Single Retwerte Image Editing
Malliple Reference Image Sene
Nulto Reference Image Editin
GPT
imAge11
Crrel
Vrages
Image
Card
SACEL
0.9
CALEL 053
TrAge
Figure 5: Overall human rating by task and topic for the four unified models that support all six tasks.
5.2 QUALITATIVE ANALYSIS
Quantitative metrics alone fail to capture common mistakes that become clear when examining outputs more closely. These mistakes manifest in several recurring ways, as illustrated in Appendix [A.3] Across tasks, models often skip parts of complex and multi-step instructions (Figure 8) or produce unreadable and corrupted text (Figure 13). problem that persists across nearly all text-heavy domains. Numerical inconsistencies
also frequent in symbolic settings, such pie chart percentages not summing to 100 or receipt totals not matching itemized values (Figure 99. Symbolic and structured domains add further challenges, including emrors in understanding depth maps, frequent mismatches between chart legends and plotted data, and incomplete object detection in computer graphics where bounding boxes are poorly aligned with object boundaries (Figure TOP. Editing tasks further expose critical weaknesses: models may alter the canvas, misplace objects, neglect shadows and reflections, return the original input unchanged, or even generate a completely new image instead of applying the requested modification (Figure 11}. In generation, systems sometimes collapse to near-duplicates of references, ignore one or more references, or produce distorted human figures. These recurring errors clarify why artifact suppression and fine-grained alignment remain key open problems, even for the strongest models.
5.3 HUMAN VS. VLM AS JUDGE
To assess the reliability of automatic evaluation, compare VLM-based judgments with human ratings across all four criteria. T "old-averaged""ran rank correlations (Spearman), Kendall's
aveerage results are computed using leave-one-out procedure: each annotator (or the VLM) is compared against the mean of the other two, and results averaged across folds.
8

---

Published as a conference paper at ICLR 2026
Table Fold-averaged Spearman correlations, Kendall's accuracy, and Bias (average signed differ- ence between VLM and human ratings) comparing VLM-based judgments with human ratings across all criteria.
Human Human
VLM-Human
Metric
Spearman p
Kendall Acc.
Spcarman p
Kendall Acc.
Bias
Prompt Relevance
0.67 + 0.03
0.77 + 0.01
0.70 +: 0.00
0.79 = 0.00
-0.07 + 0.01
Aesthetic Quality
0.55 + 0.01
0.73 + 0.00
0.62 + 0.01
0.76 + 0.00
-0.01
0.01
Content Coherence
0,50 + 0.01
0.57 0.01
0.74 + 0.01
-0.05 + 0.01
Artifact
0.64 = 0.01
0.76 0.01
0.59 0.01
0.75 = 0.00
0.06 0.00
Overall
0.68 + 0.02
0.76 0.01
0.72 0.00
0.78 + 0.00
-0.02 0.00
accuracy, and VLM bias (the average signed difference between VLM and human ratings). Overall, the VLM exhibits moderate to strong agreement with human raters across evaluation dimensions, aligning well with Homan-human consistency. Spearman correlations between VLM and human ratings range from 0.57 for content coherence to 0.70 for prompt relevance, closely matching and in some cases exceeding human-human correlations. Kendall's accuracy follows a similar trend, with values between 0.74 and 0.79, indicating that the VLM preserves the relative ranking of samples in line with human judgments. At the metric level, prompt relevance shows the strongest aligmment, suggesting that the VLM is particularly effective at assessing semantic faithfulness to the input prompt. In contrast, for artifacts, VLM-human agreement lags behind numan-humaa agreement, and the positive bias indicates that the VLM under-penalizes flaws such as unreadable text, boundary glitches, misaligned layouts that humans consistently flag. Taken together, the results suggest that VLM-based evaluation provides a reliable proxy for human scoring on high-level criteria such as prompt relevance and aesthetic quality, but human judgment remains crucial for more detailed criteria such as visual defects.
6 DISCUSSION
The tast is aditing, but the modal generated a new Image Inttaad
The model outputs the same image the Impot.
31.3%
300%
12.85
8.4%
25.5%
18.6%
15%
1.5%
2,1%
20.5%
0.%%
2,4%
1.1%
2.3%
0.6%
Vecel
Podes
Figure 6: Percentage of cases where the model generating completely new image or simply return input in image editing tasks.
Targeted data curation boosts domain-specific performance. While GPT-Image-1 achieves the strongest results across most domains, Qwen-Image demonstrates how deliberate data design can shift the balance in text-heavy scenarios. Unlike many models that struggle with textual content, Owen-Image consistently outperforms even closed-source systems in artwork, photorealistic images, and especially textual graphics for text-guided generation (Figure 15). This advantage stems from training pipeline enriched with synthetic text-rich samples and a progressive curriculum that gradually introduced increasingly complex layouts. Together with a balanced domain coverage, these choices directly improved text rendering, making Qwen-Image particularly effective in text-heavy tasks where most models perform poorly.
Architectural pathways and training objectives influence editing tasks behaviours. The error rates in Figure [ suggest that autoregressive diffusion hybrids such OmniGen2 and Step EX-Edit are more likely to generate entirely new images when asked to edit (17%), possibly reflecting their reliance language-driven pathways that override source conditioning. In contrast, diffusion-

---

Published as a conference paper at ICLR 2026
only editors like IC-Edit (0.6%) and InstructPix2Pix (3.4%) rarely disregard the source image, consistent with their architectures being explicitly optimized for localized modifications (though this may come at the cost of limited task coverage). Interestingly, Flux.1 Kontext, despite being diffusion-based, shows a higher rate of generating a new image (14.4%), likely due to its broader training scheme that mixes local edits, global edits, and reference-based composition. This unified in-context design improves versatility but makes the model less constrained to preserve the source image like InstructPix2Pix and IC-Edit.
Future Work. The structure of our dataset, which includes human scores, object-level tags, and full model outputs, points to several directions for future research. (1) The collected human scores can be used as preference data to train models that better align with human judgments, for example through preference optimization or ranking-basee fine-tuning (Wu et al., 2023 Liang al. 2024). (2) The
object-level tags provide a basis for diagnostic and self-correcti approaches, enabling models to generate object-aware refinements or produce corrective instructions (Chen et al. 2024) Chakrabarty et al.) 2023). (3) The combination of fine-grained annotations and human evaluations creates opportunities for developing more interpretable metrics that capture object-level errors, compositional inconsistencies, and coomptobbjed mismatches more faithfully than existing automatic metrics (Tan et al., (0244) Kirstain et al., 2023b). Taken together, we hope that these directions will support the development of more reliable, controllable, and human-aligned multimodal generation systems.
7 CONCLUSION
We present ImagenWorld, a benchmark that unifies six generation and editing tasks across six topical domains, supported by 20K fine-grained human annotations. Unlike prior benchmarks, ImagenWorr World introduces an explainable evaluation schema with object-level and segment-level issue labels, enabling interpretable diagnosis of model failures beyond scalar scores. Evaluating 14 models, we find consistent trends: models perform better on generation than on editing, struggle with symbolic and text-heavy domains, and show a persistent gap between closed-source and open-source models. Our structured human annotations capture localized eтrors and reveal insights that modern VLM-based metrics miss. By combining broad task coverage with explainable labeling, Imagen World serves not only rigorous benchmark but also diagnostic tool, laying the groundwork for more faithful and robust image generation systems.
ETHICS STATEMENT
The rapid advancement of image generation and editing technologies raises pressing safety and ethical concerns. Traditional photo manipulation tools such as Photoshop already enabled malicious uses, but modern Al Al-powered systems dramatically lower the technical barrier, making realistic forgeries accessible to a much broader audience. This democratization amplifies risks of fraud and deception in everyday contexts: for example, user might alter a photo of delivered food to obtain fraudulent refunds, fabricate receipts for economic gain, or generate fake identification documents. More broadly, the same capabilities imagery, political disinformation, or privacy violations. Such misuse threatens not only businesses and institutions but also public trust in digital media and the safety of individuals. As generative models continue to improve in fidelity and controllability. it is critical to establish safeguards, encourage responsible deployment, and promote ethical practices. In releasing Imagen World, we will provide only sanitized data and annotations, exclude sensitive personally identifiable content, and make all resources publicly available to support transparent and responsible research.
be misused for deepfake creation, leading non-consensual
REPRODUCIBILITY STATEMENT
All experiments were conducted with 8 NVIDIA A6000 GPUs using the ImagenHub (Ku al. 2024b) inference library, into which we integrated the latest models when not already included. w For fairesss, all model implementations followed the default configurations recommended in their respective papers or official releases, and all open-source models were evaluated under the same seed (42). Approximately $1, $1,000 USD were spent API access for closed- models and
10

---

Published as a conference paper at ICLR 2026
VLM- evaluators. Code, dataset, and sanitized human annotations will be released publicly accessible platforms (e.g., GitHub, HuggingFace)..
ACKNOWLEDGEMENT
We extend our gratitude to Min-Hung Chen for valuable feedback and insightful comments on this paper. We also sincerely thank Yu- Ying Chiu, Boris Leung, Wyett Zeng, and Dongfu Jiang for their help with data creation and annotation setup.
REFERENCES
BlackForestl Labs, Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, Zion English, Patrick Esser, Sumith Kulal, Kyle Lacey, Yam Levi, Cheng Li, Dominik Lorenz, Jonas Müller, Dustin Podell, Robin Rombach, Harry Saini, Axel Sauer, and Luke Smith. Flux.1 kontext: Flow matching for in-context image generation and editing in latent space, 2025. URLbttts: arxiv. org /2zzg//bb/2500.15//
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning follow image editing instructions. In CVPR, 2023a.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18392 18402, 2023b.
Tuhin Chakrabarty, Kanishk Singh, Arkadiy Saakyan, and Smaranda Muresan. Learning to follow object-centric image editing instructions faithfully. arXiv preprint arXiv:2310.19145 2023.
Ziyi Chang, George Alex Koulieris, Hyung Jin Chang, and Hubert P. H. Shum. On the design fundamentals of diffusion models: A survey, 2025. URL /tttps://arxtw.oorg//b////// wwtp:/s./.aa/aaeeeoo..o..osss 2206
04542
Sixiang Chen, Jinbin Bai, Zhuoran Zhao Tian Ye, Qingyu Shi, Donghao Zhou, Wenhao Chai, Xin Lin, Jianzong Wu, Chao Tang, Shilin Xu, Tao Zhang, Haobo Yuan, Yikang Zhou, Wei Chow, Linfeng Li, Xiangtai Li, Lei Zhu, and Lu Qi. An empirical study of gpt-40 image generation capabilities, 2025a. URL https:: /ttpppp://rxxi...o/
http://aas22200005550 05979
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811 2025b.
Xinyan Chen, Jiaxin Ge, Tianjun Zhang, Jiaming Liu, and Shanghang Zhang. Learning from mistakes: Iterative prompt relabeling for text-to-image diffusion model training. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 2937-2952, 2024.
Yixiong Chen, Li Liu, and Chris Ding. X-iqe: explainable image quality evaluation for text-to-image generation with visual large language models. arXiv preprint arXiv:2305. 2023.
Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian scene graph: Improving reliability fine-grained evaluation for text-to-image generation. arXiv preprint arXiv:23 18235, 2023.
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nic, Ziang Song. Guang Shi, and Haoqi Fan. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505. 14683,.....4,,.. 2025.
Google. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. URL htw.::ttps///cc... /ttttts://axxi...oo http:///wxxvvvg///aaa/
06261
Jian Han, Jinlai Liu, Yi Jiang, Bin Yan, Yuqi Zhang, Zehuan Yuan, Bingyue Peng, and Xiaobing Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis. In Proceedings of the Computer Vision and Patter Recognition Conference, pp. 15733-15744, 2025.
11

---

Published as a conference paper at ICLR 2026
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: reference- frrencee-freeeeee evaluation metric for image captioning. In EMNLP. 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Hexiang Hu, Kelvin Chan, Yu-Chuan Su, Wenhu Chen, Yandong Li, Zhao Yang. Xue Ben, Boqing Gong, William W Cohen, Ming- Ming-Wei Chang, and Xuhui Jia. Instruct-imagen Image generation with multi-modal instruction. arXiv preprint arXiv:2401.01952, 2024.
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang. Mari Ostendorf, Ranjay Krishna, and Noah Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
(0406-20417, 2023.
Hang Hua, Ziyun Zeng, Yizhi Song, Yunlong Tang. Liu He, Daniel Aliaga, Wei Xiong, and Jiebo Luo. Mmig-bence Towards comprehensive and explainable evaluation of generation models, 2025. httpts/s//a.....o/////////a/////
image
111225555199999 19415
Yi Huang, Jiancheng Huang, Yifan Liu, Mingfu Yan, Jiaxi Lv, Jianzhuang Liu, Wei Xiong, He Zhang, Liangliang Cao, and Shifeng Chen. Diffusion model-based image editing: survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(6):4409 4437, June 2025, ISSN 1939-3539.
doi: 10.
URL http:/ //tx///xx//// doi www.ii.orr11... 1109/
ТPAMT 2025 325
Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge. Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, and Ying Shan. Smartedit: Exploring complex instruction-based image editing with multimodal large language models, 2023. URL https
//arxiv.org/ abs 2312 06739
Dongfu Jiang, Max Ku, Tianle Li, Yuansheng Ni, Shizhuo Sun, Rongqi Fan, and Wenhu Chen. Genai arena: An open evaluation platform for generative models, 2024. URLhttts:://axiv
abs/2406. 04485
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a- pic: An open dataset of user preferences for text-to-image generation. 2023a.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pie: An open dataset of user preferences for text-to-image generation. Advances in neural information processing systems, 36:36652-36663, 2023b.
Max Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen. VIEScore: Towards explainable metrics for conditional image synthesis evaluation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12268-1220,, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 1118665/w1/0224..clllonggg //aclanthology org. 2024 acl224..ccl110 acl- -ong. 6637
URL https
Max Ku, Tianle Li, Kai Zhang, Yujie Lu, Xingyu Fu, Wenwen Zhuang, and Wenhu Chen. Ima- genhub: Standardizing the evaluation of conditional image generation models. In The Twelfth International Conference on Learning Representations, 2024b. URL //LLpptpp///oeeeeeeieeww
Sangwu Lee, Titus Ebbecke, Erwann Millon, Will Beddow, Le Zhuo, Iker García-Ferrero, Liam Esparraguera, Mihai Petrescu, Gian SaB, Gabriel Menezes, and Victor Perez. Flux.1 krea [dev]. ai/flux krea 2025.
https://githube www.eeaaaaooooaaaaaaaaaaaaaaaaaa homps////gtthubb.cmm/reeeee
Tianle Li, Max Ku, Cong Wei, and Wenhu Chen. Dreamedit: Subject-driven image editing, arXiv preprint arXiv:2306. 12624, 2023.
12

---

Published as a conference paper at ICLR 2026
Youwei Liang, Junfeng He, Gang Li, Peizhao Li, Arseniy Klimovskiy, Nicholas Carolan, Jiao Sun, Jordi Pont-Tuset, Sarah Young, Feng Yang. et al. Rich human feedback for text-to-image generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
19401-19411, 2024.
Zhiqiu Lin, Deepak Pathak. Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation, 2024. URL
https::/arxxiy: wwtp:/i..orrgggooo wttp://///...a.o. 01291
Yaron Lipman, Ricky T. Q. Chen, Heli Ben- Maximilian Nickel, and Matt Le. Flow matching for generative modeling, 2023. httpshhts::///xxiie org/ /ttbb//22100 2220022777770
Shiyu Liu, Yucheng Han, Peng Xing, Fukun Yin, Rui Wang. Wei Cheng, Jiaqi Liao, Yingming Wang. Honghao Fu, Chunrui Han, et al. Steplx-edit: A practical framework for general image editing. arXiv preprint ar arXiv:2504.17761 2025.
Yiwei Ma, Jiayi Ji, Ke Ye, Weihuang Lin, Zhibin Wang, Yonghan Zheng, Qiang Zhou, Xiaoshuai Sun, and Rongrong Ji. 12ebench: comprehensive benchmark for instruction-base image editing, 2024. URL httpstts::// //ap:i////iiiv.aa/aa org abs /20/8 14180
Yuhang Ma, Yunhao Shui, Xiaoshi Wu, Keqiang Sun, and Hongsheng Li. Hpsv3: Towards wide- spectrum human preference score, 2025. / abs 2508. 03789
Yuwei Niu, Munan Ning, Mengren Zheng. Weiyang Jin, Bin Lin, Peng Jin, Jiaqi Liao, Chaoran Feng, Kunpeng Ning, Bin Zhu, and Li Yuan. Wise: A world inowledge-nnforeedd semantic evaluation for text-to-image generation, 2025. http:///tss///aaaa....o///////./
OpenAl. Introducing 4o image generation.
https://openai. hetmm/innx//////
int roducingn 40- image- generation March 2025. Image generation example from the official OpenAl announcement.
Yulin Pan, Xiangteng He, Chaojic Mao, Zhen Han, Zeyinzi Jiang, Jingfeng Zhang, and Yu Liu. Ice-bench: A unified and comprehensive benchmark for image creating and editing, 2025. URL
wttp://wwwv.rrgo/////.002/ 14482
Yuang Peng, Yuxin Cui, Haomiao Tang, Zekun Qi, Runpei Dong, Jing Bai, Chunrui Han, Zheng Ge, Xiangyu Zhang, and Shu-Tao Xia. Dreambench++: A human-aligned benchmark for personalized image generation. In The Thirteenth International Conference on Learning Representations, 2025. gi
htllhhtts://// httpsss//dddeabmbbnhhhll..o www.ggittuuuuu..... thub. io/
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aber- man. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEECVF Conference Computer Vision and Pattern Recognition, pp.
22500-22510, 2023.
runwayml. Stable diffusion inpainting, 2023. URL httthttss//hgggggggfce..oo conuuu/
stable e- dif fussinnnn-- wnnaannntt in
Chitwan Saharia, William Chan. Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems. 35:36479-36494, 2022.
stability.ai. Stable diffusion xl, 2023. URL https::/stabil
13

---

Published as a conference paper at ICLR 2026
Zhiyu Tan, Xiaomeng Yang. Luozheng Qin, Mengping Yang, Cheng Zhang, and Hao Li. Evalalign: Supervised fine-tuning multimodal Ilms with human-aligned data for evaluating text-to-image models. arxiv preprint arXiv:2406.16560 16562, 2024.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng. and Liwei Wang. Visual autoregressivv modeling:: Scalable image generation via next-scale prediction, 2024. URL http::://arviv.oog///// 2404. 02905
Yuxiang Tuo, Wangmeng Xiang, Jun He, Yifeng Geng, and Xuansong Xie. Anytext: Multilingual visual text generation and editing, 2024. httptpp:://arrl.ooo/////////////
Olivia Wiles, Chuhan Zhang, Isabela Albuquerque, Ivana Kajic, Su Wang, Emanuele Bugliarello, Yasumasa Onoe, Pinelopi Papalampidi, Ira Ktena, Christopher Knutsen, Cyrus Rashtchian, Anant Nawalgaria, Jordi Pont-Tuset, and Aida Nematzadeh. Revisiting texl-to-image evalu- ation with gecko: on metrics, prompts, and human rating. In The Thirteenth International www.:ottttfoooeeeeeeo
Conference on Learning Representations, 2025. /eattps://operreet
Id=Im2neAMIre
Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, Yuxiang Chen, Zecheng Tang, Zekai Zhang, Zhengyi Wang, An Yang, Bowen Yu, Chen Cheng, Dayiheng Liu, Deqing Li, Hang Zhang, Hao Meng, Hu Wei, Jingyuan Ni, Kai Chen, Kuan Cao, Liang Peng. Lin Qu, Minggang Wu, Peng Wang, Shuting Yu, Tingkun Wen, Wensen Feng. Xiaoxiao Xu, Yi Wang, Yichang Zhang, Yonggiang Zhu, Yujia Wu, Yuxuan Cai, and Zenan Liu. -meneimage technical report, 2025a. URL CRLhttps::/ http///axvvv/.////////////.aaa.. org/ / 2508. 02324
Chenyuan Wu, Pengfei Zheng, Ruiran Yan, Shitao Xiao, Xin Luo, Yueze Wang. Wanli Li, Xiyan Jiang. Yexin Liu, Junjie Zhou, Liu, Ziyi Xia, Chaofan Li, Haoge Deng, Jiahao Wang, Kun Luo, Bo Zhang, Defu Lian, Xinlong Wang, Zhongyuan Wang. Tiejun Huang, and Zheng Liu. Omnigen2: Exploration to advanced multimodal generation. arXiv preprint arXiv:2506.1885 18871, 2025b.
Shaojin Wu, Mengqi Huang, Wenxu Wu, Yufeng Cheng, Fei Ding, and Qian He. Less Lore-to-more more general- ization: Unlocking more controllability by in-context generation. arXiv preprint arX1v:2504.02166. 2025c.
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal 11m. In Proceedings of the International Conference on Machine Learning, pp. 53366-5333 53397, 2024.
Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score: Better aligning text-to-image models with human preference. In Proceedings of the IEEE/VVF International Conference on Computer Vision, pp. 2096-2105, 2023.
Jing Xiong, Gongye Liu, Lun Huang, Chengyue Wu, Taiqiang Wu, Yao Mu, Yuan Yao, Hui Shen, Zhongwei Wan, Jinfa Huang, Chaofan Tao, Shen Yan, Huaxiu Yao, Lingpeng Kong, Hongxia Yang, Mi Zhang, Guillermo Sapiro, Jiebo Luo, Ping Luo, and Ngai Wong. Autoregressive models in vision: A survey, 2025.
wwwpv..ivvv.m org/ abs/ 2211. 05902
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation, 2023.
Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-man mark prompting unleashes extraordinary visual grounding in gpt-4v, arXiv preprint 1tXiv:2310.114411 2023. URLL/ttts:///arx/ http:///axxxvvvv..aaa/e/ was///////////////a///o///eeaeee /2310. 11441
Shih-Ying Yeh, Yu-Guan Hsieh, Zhidong Gao, Bernard W Yang, Giyeong Oh, and Yanmin Gong. Navigating text-to-image customization: From lycoris fine-tuning to model evaluation, 2024. URL
abs/
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, Ben Hutchinson. Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation, 2022. URI.ttps:: http://wwww....///////0....// 10789
14

---

Published as a conference paper at ICLR 2026
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543, 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
Zechuan Zhang, Ji Xie, Yu Lu, Zongxin Yang, and Yi Yang. In-context edit: Enabling instructional image editing with in-context generation in large scale diffusion transformer, 2025. URL https 7/arxiv. wttp:ivv..rg//b/////e/e//// 20690
15

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1 USE LLM WRITING
We used ChatGPT to improve the writing, mainly on grammar fix.
A.2 HUMAN ANNOTATION DETAILS
During the annotation stage, we recruited 22 expert annotators, primarily graduate students fluent in English. Each annotator evaluated approximately 500-1,000 images, with each annotation taking 30-90 seconds. To reduce fatigue and ensure consistent quality, we restricted annotators to maximum of 200 samples per week. The entire process conducted using Label Studio, which provided a user-friendly interface. Potential object-level and segment-level issues were pre-listed using languala model (VLM) and Set-of-Marks (SOM), allowing annotators to simply select them via checkboxes. For cases where the VLM and SOM failed to capture issues, annotators could switch manual mode to input text or create bounding boxes. The full annotation process spanned two months, during which all data were collected. Below we present the guidelines for annotators, and Figure illlstrates the annotation interface.
Prompt Relevance
Definition: Whether the image accurately reflects or responds to the prompt. Ask yourself: Does the image actually follow the instructions topic in the prompt?
Common issues:
Prompt: "Explain the water cycle." - Image shows the food chain.
Prompt: "Slide about diabetes symptoms." Image shows a kidney structure.
Prompt: "Compare apples and oranges." - Image compares apples and bananas.
Rating (1-5):
1. Completely unrelated to the prompt.
Mostly incorrect; vague connections with many mismatches.
3. Partially relevant; key ideas present but with errors/omissions.
Mostly accurate; follows the prompt with minor issues.
5. Fully aligned with the prompt; clear, focused, and complete.
Aesthetic Quality / Visual Appeal
Definition: Whether the image is visually appealing, clean, and easy to interpret.
Ask yourself: this image pleasant look at, readable, and professionally designed?
Common issues:
Text too small/hard to read blending into the background.
Cluttered or misaligned layout.
Poor color combinations; inconsistent fonts spacing.
Rating (1-5):
Visually poor; unattractive, hard to read, confusing.
2. Below average; noticeable design flaws, poor readability.
3. Decent; generally readable with minor layout/design issues.
4. Clean and aesthetically good; few flaws.
Polished and visually excellent.
16

---

Published as a conference paper at ICLR 2026
Content Coherence
Definition: Whether the content is logically consistent and fits together meaningfully. Ask yourself: Does everything in the image belong together and make sense as a whole?
Common issues:
Labels pointing to the wrong part (e.g., "liver" pointing to lungs).
Chart shows decreasing values titled "growth over time."
Title: "Parts of Flower" but the figure shows tree anatomy.
Rating (1-5):
1. Internally inconsistent or nonsensical; contradictory parts.
2. Some logic present, but components are confusing/missattceed.
3. Mostly coherent, with noticeable mismatches or awkward parts.
4. Logically sound overall; only minor inconsistencies.
5. Completely coherent and internally consistent.
Artifacts/ Visual Errors
Definition: Whether the image has visual flaws due to generation errors (e.g., distortions, glitches).
Ask yourself: Are there technical visual issues like melting edges, strange text, or broken parts?
Common issues:
Gibberish or distorted text (e.g.. "#di%et@es"" instead of "diabetes").
Warped borders, duplicated textures, glitched areas.
Extra limbs, broken symmetry, over-smoothed or pixelated regions.
Rating (1-5):
1. Severe artifacts that ruin the image.
2. Major flaws that are clearly noticeable.
3. Minor artifacts; image remains usable.
Mostly clean; very subtle flaws any.
5. No visible artifacts.
Issue Tagging
Object-level issues (model output); Select any item that has issues. none apply, choose "None of the objects have issues."
Segmentation issues (segmentation map): Select any index with issues. none apply,
choose "None."
Other issues; If your score is not 5/5 and the issue is not covered above, briefly describe it here.
Selecting "None" may mean either (i) there is no issue, or (ii) your issue is not listed. In the latter case, provide brief explanation under "Other issues." If your score is not full (5/5), you must either select a corresponding issue or provide short explanation.
Important:
17

---

Published as a conference paper at ICLR 2026
Refined Prompt:
image
Change highightt
decelecting
active
second right), and selecting
12' by applying similar blue dirouler highlight to (ocateed the third of dates, from left). Genersted image
Reference images
May 2025
May 2025
M
T
W
T
S
$
M
W
5
28
29
30
3
4
30
6
7
10
12
21
19
21
23
24
25
22
26
27
28
29
30
31
27
23
2
7
3
4
8
Prompt Relevance
Aesthetic Quality
x
1
12
12
Content Coherence
Artifacts or Visual Defects artifacts defects)
Object-Level Issues (Model Output)
bue crroulrr highlight on date 1216
calendar grid struturl!!
texf
The teats not readabba!!
nevigation
objects have issuesi
weelsday iabels T. 5y
cxlender date numbers 1,2, 31
The taok editing, but the model generntedd a new image instwaght
numbers
siceff
The model outputs the some Image the input
Eup memtal feseen Segrentatinn Medl
15
8
4
18
26
Other issues (Not Captured Above)
Describe any additiinnal insure coplired by objject level aegrmentation level ostections.
Add
The model incemnely changed date '11 '12,, musulting two 12a and no the calendar: it adso hightighoed the hailin
prompt
Figure 7: Label Studio interface: annotators read the prompt, inspect the image, rate four criteria (1-5), and flag Vuugese object- and segment-level issues.
18

---

Published as a conference paper at ICLR 2026
A.3 EXAMPLES OF FAILURES
A.3.1 MODEL FAILS TO PRECISELY FOLLOW INSTRUCTIONS.
Prompt: Edit image 1. Replace the top-left crate with the yellow warning sign from image 3. Place the pink crewmate (from the center of image 2) and the yellow crewmate (from the bottom right of image 2) standing side-by-side on the central doorway in image 1. Ensure all new elements are integrated with correct perspective, lighting, and scale.
Source image
Conditional images
Output
Figure 8: Instruction-followii problem: The model placed red and green crewmates instead of pink and yellow, and the yellow sign's position does not match the request.
A.3.2
NUMERICAL INCONSISTENCIES
10%
45%
95%
Hoppiness
TARGET
13%
70% Neutral
40%
MITEDOO
Noutral
20%
30%
SUSPEOL
EOTTOON
12.0
15% Sadness
30%
10%
35% Surprise
TOTTAL
5.09
4,00
The total percentage the
pie
The sum of person expressions equal to 275%
The tocal the receipt items $378.65, not $420.69.
Figure 9: Examples of numerical inconsistencies.
A.3.3
SEGMENTS AND LABELING ISSUES
Plant
deer
vaay
rabbil
glaly
Parra
cord
Jharts
foobit
The model does specify borders labels.
Label locations do point the detected areas.
Mismatched Labels and Segments
Figure 10: Examples of labeling issues.
19

---

Published as a conference paper at ICLR 2026
A.3.4 GENERATE NEW IMAGE IN EDITING
Source
Output
Transform the triangle in logo into a glossy neon blue square with a reflective highlight and slight bevel.
YouTub
YouTube
model generates new person The model changes the source image ratio
Add tractor the right side of the field. Follow the texture of the original painting.
The model generates new background similar to the one in the source image.
the nail from this image
The model generates new bone structure with different coloring
Figure 11: Examples of generating a new image when the task is editing.
20

---

Published as a conference paper at ICLR 2026
A.3.5 PLOTS AND CHART ERRORS
Visuelize bubole chart of 2025 SmuS startups: k-adis monthly burn. rate (SM), y-axis ARR growth bubble employees, bubble industry vertical (FiTTech, HealthTech. AdTech. EcTech ClimatsToch)
Visualizc decision tree for mediical ciagnosis with branching pathways, probability pereentages at cach nade, colored outcome leaves (grean-healthhy yellow-monitor, sod-urgenti, stethoscope icons.
M 2025 Seas starttups
MEDICAL DIAGNSOUSIS
18
CONDTOR
39
Iustesd of company names. the text in all bubbles is 'Emplogess..
The tree does not have clear instractions and flow for diaguesis.
Figure 12: Examples of plots and diagram issues.
A.3.6 UNREADABLE TEXT
senving besf
Angedag
Meepiigg lost
In pod paing
sudg dech
opoll1
Hello
CASE REAO
The text is not readable
Figure 13: Examples of text issues.
A.3.7 ERROR MASK STATISTICS
pixel proportion
error/ tatal
000
600
0.5
Velu
Figure 14: Distribution of error pixel proportions across the dataset. Histograms illustrate the fraction of error pixels relative to total mask pixels, excluding masks with none or all pixels marked as errors.
21

---

Published as a conference paper at ICLR 2026
A.4
TASK AND TOPIC STATISTICS
Computer Graphies
Information Graphics
Photorealistic Images
Textual
Model
Artworks
Screenshots
Graphies
Text-guided Image Generation
GPT-Image-1
0.95+0.06
0.90+0.09
0.11
0.96+00.66
0.93++0099
0.92+0.08
Owen-Image
0.9710.03
0.8110.14
0.79110.18
0.97
0.73:1.14
0.9410.11
Flux. --vea-dev
0.92:10.06
0.75:0.18
0.67:0.22
0,9410.08
0.7310.12
0.8810.13
Gemini 2.0 Flash
0.87+0009
0.73++0220
0.61+0.26
0.94+00.99
0.75+0012
0.88++0.14
UNO
0.8910.08
0.7310.16
0.62:10.21
0.8610.13
0.64
0.92:0.09
BAGEL
0.95=00.44
0.66#0.19
0.66:10.23
0.91
0.62+0.17
0.79:1..17
Infinity
0.93+0.08
0.69+0.18
0.56++0.27
0.85+10.15
0,444:0.12
0.76:10.2
OmniGen2
0.8611.111
0.77#0.19
0.52::..9
0.89+0.11
0.52++0.13
0.87+0.13
SDXL
0,90+1.10
0.43+0.27
0.55
0.82:10.14
0.57+10.18
0.57
Janus Pro
0.8710.13
0.4910.27
0.42.10.29
0.81+10.23
0.50.10.22
0.60:10.27
Text-guided Image Editing
GPT-Image-1
0.8611.09
0.9010.12
0.69110.19
0.8810.09
0.73:1.13
0.8611.112
0.7310.14
0.8310.14
0.50:10.23
0.76+0.12
0.5610.20
0.8210.17
Gemini 2.0 Flash
0.72+10.16
0.75+
0.50-10.23
0.83:10.17
0.51
0.83:10.18
BAGEL
0.64=:020
0.86#0.14
0.45+t0.22
0.71#0.16
0.61
0.73+0.21
OmniGen2
0.74+0.13
0.72+0023
0.40
0.81
0,36+:0222
0.48
IC-Edit
0.59110.00
0.57.0024
0.35
0.5510021
0.39
0.56:10.21
Step X-Edit
0.53+0.26
0.3940.30
0.19
0.65+0.119
0.47+0020
0.50
InstructPix2Pix
0.4911022
0.53
0.32
0.52
0.48.10.19
0.36
Single Reference Image Generation
GPT-Image-1
0.95
0.86
0.72-10.18
0.87+-0.10
0.83
0.81
BAGEL
0.83+0.17
0.76+1..3
0.
0.67:0.16
0.52:t..1
0.
Gemini 2.0 Flash
0.77#0.19
0.73+10.19
0.52
0.81
0.56+0.18
0.66:0.20
UNO
0.78:10.20
0.65
0.49:1.119
0.68:10.20
0.56:10.15
0.49:10.19
OmniGen2
0.78
0.68+0.18
0.49
0.66#0.11
0.41
0.
Single Reference
Image Editing
GPT-4o
0.79:1.116
0.7911.18
0.80
0.89:10.8
0.63:10.15
0.861:.11
Gemini 2.0 Flash
0.59:1..19
0.63:10.22
0.39
0.64:10.20
0.39410.14
0.63
BAGEL
0.54
0.53:10.22
0.50-10.26
1.68
0.47-10.17
0.58
OmniGen2
0.62+:0.1
0.57+0023
0.27=t0.14
0.57+0.23
0.27+0014
0.47
Multiple Reference
Image Generation
Gma----age-1
0.96110.03
0.8110.12
0.66:10.17
0.8611.07
0,83:10112
0.92:10.10
Gemini 2.0 Flash
0.91+0.12
0.71+0.20
0.53
0.85+10.15
0.54
0.86:10.15
OmniGen2
0.80=10.14
0.62+0.18
0.46:10.15
0.69+0.17
0,40:1018
0.68
BAGEL
0.79
0.4910.22
0.47:10.16
0.80+0.11
0.42+0..0
0.71
UNO
0.70-10.12
0.66.10.14
0.41-10.13
0.63-10.14
0.51-10.15
0.7110..0
Multiple Reference Image Editing
GPT-Image-I
0.85
0.85
0.69:10.20
0.92-10.05
0.81:10.12
0.62
Gemini 2.0 Flash
0.76110.1
0.58+0.17
0.62:10.25
0.70+0.13
0,56:0..16
0.50 16
OmniGen2
0.53+0.18 0.66
0.44 0.50+0.20
0.68-10.19
0.3240.15
0.41
BAGEL
0.53+1.23
0.41
0.59+0.11
0.21+0.11
0.32+0.16
Table 5: Topic-wise performance across six tasks. Bolded entries are the per-topic best within each task.
22

---

Published as a conference paper at ICLR 2026
All Models
Best Model by Task Topic
CG
Owen Image 0.97
GPT Image-1 0.50
GPT Image-1 0.81
GPT Image-1 0.93
Owen Image 0.94
TIG
Image 0.97
0.86
0.50
0.69
0.88
0.73
0.86
GFT Image-1
GPT Image-1
GPT Image-1
GPT Image-1
GPT Image-1
SRIG
Image
200
GPT Image-1 0.79
GPT Image-1 0.79
GPT Image-1 0.80
GPT Image-1 0.89
GPT Image-1 0.63
GPT Image-1 0.86
SRIE
GFT Image 1 0.96
GFT Image 0.81
GPT Image-1 1 0.65
GPT Imaze 0.86
GPT Image1 0.83
GPT Image 1 0.92
0.5
MRIG
GPT 0.85
GPT 0.85
GPT Image 0.81
0.4
Image 0.92
Image 0.62
0.69
0.3
Figure 15: Best-performing model per taskxtopic across all of our 14 baselines.
Closed
Average Score Tast x Toplc
Opin
Average Score
TRIS
Toplc
IClssed
Open)
Amrraa
Amtrae
Aerga
9.01
0.55
0.90
0.88
9.74
+0.15
19.11
+0.07
+0.12
+8.11
71
15227
5.84
0.00
0.60
0.84
0.78
6.74
390
8.70
0.67
6.90
9.59
9.41
5900
10.00
+9.10
10.12
10317
-0.28
16-21
+5.55
M4B
5.92
9.76
0.50
0.65
0.48
6.88
9.79
M80
8.76
0:20
0.45
0.72
6.44
5:70
2.44
9981-98.77
#0.17
+0.15
40.35
-0.21
+6.18
+3.08
120
0.6
Auragg:
1.33
8.77
0.05
0.85
0.67
6.7
1.75
Neogge
0.71
0:43
0.48
0.78
0.44
6:53
3.59
Average
+..22
+8.15
+0.17
+0.34
-0.33
+6.28
+1.17
Figure 16: Best-performing model per taskxtopic for closed-source (left) and open-source (middle) models based on human overall scores; the panel (right) shows A Closed Open.
Closed
Best Model by Task x Topic
Open
Best Model by Trik Topic
[Closed Open)
104
L.8.
1.74
C014 4.97
$137
0.85
TEC
60.12
+8
+311
10.04
1032
14.4
0.4
0.75
03721
3.66
150
47
P12
19620
1142E
$1.0%
5822
0.6
0102
0o2
DACEL
34GEL
S48
+0.7
+8.22
+335
+0.2
4838
1038
OFT
8.44
1:27
12
1.
+8.16
+622
4A
PR1Z
035
035
0.5
3.50
0.1
2.52
844
0.33
0.44
2.48
0.32
44
Figure 17: Average human score per taskxtopic for dlsse--oource (left) and open-source (middle) models; the A panel (right) shows A Closed Open.
23

---

Published as a conference paper at ICLR 2026
A.5 STATISTICAL SIGNIFICANCE ANALYSIS
We conduct statistical significance tests quantify the robustness of the performance differences reported in the paper. All analyses operate on human ratings, where each human score denotes the average of the three independent annotators for given (task model) pair. The tests are performed on a per-item basis. Paired two-sided r-tests are used when comparing two models (or model families) evaluated on the same instances, while Welch's 1-test used when comparing two disjoint groups of items (e.g., generation vs. editing tasks).
GPT-Image-I vs. Gemini 2.0 Flash. We compare GPT- ImPa-mm I and Gemini across all task instances. GPT-Image-1 receives significantly higher ratings overall compared to Gemini 2.0 Flash.
Overall: 1080, mean diff 0.5961, std 0.7706, t 25.42, 4.3 10-112
II. Closed vs. Open Model Families. We compare the family-wise means of closed-source models with open-source ones using paired test across tasks.
Overall: 1080, mean diff 0.6821, std 0.5840, 38.38, p 6.6 10-204 204
III. Generation Editing Tasks (Unified Models). To assess task difficulty, compare human ratings on generation and editing tasks over our unified models: (Bagel, Gemini 2.0 Flash, GPT- 1mage-1, OmniGen2).
Overall: generation (n 2158, mean 3.9152), editing (n 2160, mean 3.5244), t 13.71, = 6.6 x 10 10-42
IV. Symbolic vs. Non-Symbolic Topics. We compare human ratings on symbolic and text-heavy do- mains (Information Graphics, Screenshots) against non-symbolic domains (Artworks, Photorealistic Images).
Overall: non-symbolic (n = 2159, mean 4.1067), symbolic (n = 2157, mean 3.1905), t 34.26, 7.3 10 1022
Artifacts: non-symbolic (n 2159, mean 4.3522), symbolic (n 2157, mean 3.0163). = 46.23, p 10-196
V. Human vs. VLM-as-Judge Agreement. We compare Gemini's automatic scores with human- mean ratings for all 6,469 (task id, model) instances.
Prompt Relevance: mean diff = -0.2700 0.2704, std = 1.0185, / = -21.35, 8.4 x 10 98
Aesthetic Quality: mean diff 00.0529, std 1.0135, = -4.20,p x 10-5
Content Coherence: mean diff 0.2069, std 1.2741, -13.66 p 1.7 10-35
Artifacts: mean diff +0.2220, std 1.1207. f 15.93, 4.2 10-50
Overall: mean diff -0.0770, std 0.8516, t -7.28, 3.9 x 10
24

---

Published as a conference paper at ICLR 2026
A.6 DATASET DETAILS
Topic Distribution Across the Dataset
16.9%
16.7%
16.0%
16.0%
16.6%
Screemshats
retworks
Computer Graphics
Photorea istic Images
Information Graphics
Textual Graphics
Figure 18: Topic distribution for the entire dataset
Word Cloud of Subtopics Across the Dataset
Facial es estimation and tnnlyissTadiiiionll art
Diagr ams charts and graphs Cultural or Heritage art
seanting
Scenie Neture
Microscopic imagery
Panniigg
Visual instructions and nanuals
Object detection visuals
Branding and visual identity
Websited
Collage
forms
Arherttortin
Astronomical images
Stylized art
CAD models
Ioons and symbols
Mildlife &
listrre Photegraphy
Digital arts
Medical images
Oet
Massage
Softwal interfaces
Semantic or Ins tance se segmentation
Memes
Giturs art
Typography
Ontirr Besing Sreee faees
3D renders
Tetal lutter
Depth maps
UI mockups
Maps
seertec
ComicSGame assets
Receipts VFX and Particles
Games
Posters
Suage Transformation
Figure 19: Word cloud of subtopics across the entire dataset, with word size proportional to frequency
25

---

Published as a conference paper at ICLR 2026
Task
Topic
Subtopics (Count)
Collage (26). Cultural/heritage art (9), Digital arts (15), Glitch Art (3), Pixel Art
(5), Pop art (5), Stylized art (11), Traditional art (11), oil paintings (16)
CG
3D renders (17), CAD models (13), Depth maps (11), Facial estimation and analysis (3), Game
(8), Object detection visuals (11), Semantic Instance
segmentation (16), Texture Map (8), Texture maps (1), VEX and Particles (12)
Diagrams, charts, and graphs (25), Icons and symbols (26), Maps (17), UI mockups (21), Visual instructions and manuals (11)
P
Astronomical images (18), Long exposure (7), Medical Images (1), Medical
TIG
images (21), Microscopic imagery (20), Minimalist Designs (5), Photography (28), Retro Futurism (2)
Games (19), Receipts (35), Software interfaces (19), Websites (27)
T
ASCHI Art (6), Branding and visual identity (7), Calligraphy (6), Comics (13),
Concrete Poetry (3), Graffiti (Text-based) (9), Illuminated Manuscripts (9),
Memes (11), Posters (12), Textile Art with Text (6), Textual Tattoos (7), Typog-
raphy (6), Word Clouds (5)
Collage (16), Cultural or Heritage (14), Digital arts (24), Portraits (1), Styl- ized art (32), Traditional art (14)
CG
3D renders (12), CAD models (12), Depth maps (13), Facial estimation and analysis (13), Game Assets (1), Game assets (12), Object detection visuals (13), Semantic or Instance segmentation (13), VFX and Particles (12)
Diagrams, charts, and graphs (25), Icons and symbols (21), Maps (14), UI mockups (20). Visual instructions and manuals (20)
SRIG
Astronomical images (28), Medical images (22), Microscopic imagery (23),
Photography (27)
Games (24), Receipts (18), Software interfaces (28), Websites (30)
T
Branding and visual identity (11), Comics (15), Memes (25), Newspaper (3),
Posters (22), Typography (22), Website (2)
A Collage (13), Cultural or Heritage art (23), Digital arts (29), Stylized art (19),
Traditional art (16)
CG
renders (12), CAD models (11), Depth maps (13), Facial estimation
analysis (10), Game assets (11), Object detection visuals (23), Semantic Instance segmentation (10), VPX and Particles (10)
Diagrams, charts, and graphs (38), Icons and symbols (15), Maps (18), UT mockups (18), Visual instructions manuals (11)
MRIG
P
Astronomical images (6), Microscopic imagery (13), Photography (81)
Games (24), Receipts (23), Software interfaces (31), Websites (22)
T Branding and visual identity (24), Comics (15), Memes (16), Posters (23),
Typography (18), Typography-not-engiin (5)
Table 6: Subtopics across generation tasks (TIG, SRIG, MRIG)
26

---

Published as a conference paper at ICLR 2026
Task
Topic
Subtopics (Count)
A
Animation (2), Blueprint (1), Ceramic art (1), Collage (4), Conceptual art (1),
Cropping (1), Cubism (3), Cultural or Heritage art (4), Digital (8), Digital arts
(2), Expressionism (4), Fauvism (2), Futurism (4), Graffiti (1), Graffiti art (I),
Hand drawn blueprint (1), Impressionism (4), Medieval art (I), Neoclassicism
(2), Oil Painting (1), Painting (8), Paintings (3), Pixel art (4), Pixel arts (2),
Pop art (3), Post-Impressionism (2), Renaissance (6), Renaissance arts (1),
Romanticism (4), Sketches (2). Statue (5), Surrealism (3), Traditional art (4),
Traditional arts (1), Vector art (3), Vector arts (1)
CG
3D renders (10), CAD models (10), Depth maps (10), Facial estimation and analysis (10), Game assets (10), Mesh Processing (5), Object detection visuals (10), Physics Simulation (5), Semantic or Instance segmentation (10), Shader Programming (5), Skeletal Animation and Rigging (5), VFX and Particles (5),
TIE
Volume Rendering (5)
I
Diagrams, charts, and graphs (61), Exploded Views (5), Exploded views (2),
Floor Plan (8), Icons and symbols (8), Maps (5), Tables (6). UI mockups (5)
P
Astronomical images (21), Medical images (15), Microscopic imagery (20), Photography (45)
S
Game Screenshots (1), Games (26), Mobile software interfaces (7), QR codes
(11), Receipts (3), Software interfaces (50), Websites (14)
Branding and visual identity (27), Comics (15), Memes (26), Posters (16),
Typography (16)
Collage (20), Cultural or Heritage art (19), Digital arts (20), Stylized art (20),
Traditional art (21)
CG
3D renders (11), CAD models (6), Depth maps (13), Facial estimation and analysis (13). Game assets (13), Image Transformation (15), Object detection visuals (18), Semantic or Instance segmentation (11)
Diagrams, charts, and graphs (30), Icons and symbols (20), Maps (26), UI
SRIE
mockups (24)
P
Astronomical images (30), Medical images (23), Microscopic imagery (20), Photography (27)
S
Games (25), Receipts (25), Software interfaces (25). Websites (25)
T
Branding and visual identity (22), Comics (22), Memes (20), Posters (16),
Typography (20)
A
Collage (20). Cultural or Heritage art (12), Digital arts (9), Oil painting (12),
Pop (14), Portraits (10), Stylized (9), Traditional (14)
CG
3D Character Integration (1), 3D renders (7), Augmented Reality Computer
Vision Visualization (1), CAD models (1), Computer Animation (1), Com-
puter Animation or Stylized Rendering (1), Depth maps (10), Game assets
(9), Geometric Modeling or Stylized Rendering (1), Glohal Illumination (2), Non-Photorealistic Rendering (1), Object detection visuals (23), Scene Editing
(1), Semantic Instance segmentation (19), Stylized Rendering (1), Stylized Rendering or renders (4), Stylized Rendering or Computer Animation (2),
Stylized Rendering or Visual Effects (1), UI Design or Stylized Rendering (1),
VFX and Particles (13)
MRIE
Diagrams, charts, and graphs (23), Icons and symbols (22), Maps (25), UI
mockups (23). Visual instructions and manuals (8)
P
Astronomical images (15), Medical images (15), Microscopic imagery (10), Scenic Nature Parks (19), Urban & Street Photography (16), Wildlife & Nature
Photography (25)
S
Chat and Social Messaging (14), Forms and Authentication (9), Games (14), Online Meeting Interfaces (11), QR Code (14), Receipts (15), Software interfaces
(11), Software interfaces farsi (1), Websites (11)
Advertisements (I), Book (I), Branding and visual identity (12), Card (1),
Comics (9), Flyers (2), Fonts (1), Magazine (2), Meme (1), Memes (18), Menu
(4), Newspaper (2), Newspapers (I), Nutrition Fact (1), Poster (2), Posters (15),
Products (1), Signs (5), Subtitle (1), Typography (20)
Table 7: Subtopics across editing tasks (TIE, SRIE, MRIE)
27

---

Published as a conference paper at ICLR 2026
A.7
AUTOMATED EVALUATIONS
LLM-bused
Embedding & perceptuul
Prompt relevance
Acsthetic quality
Content Coherence
Model
Artifact
Overall
CLIP
LPIPS
Text-guided Image Generation
GPT-4o
0.89+0.20
0,95+0012
0.91=0.23
0.96#0.13
0
0.25=0.12
N/A
Qwen-Image
0.7910.29
0.93110117
0.8610.28
0.88.1024
0.87110.0
0.2410.12
N/A
0.74+0.30
0.89+-0200
0.82+0.33
0.89=0.25
0.84+0023
0.24+0.12
N/A
Janus Pro
0.40+0.31
0.52+0.34
0.60+0.41
0.58+0.38
0.52+0.32
0,23+0.10
N/A
Gemini 2.0 Flash
0.7911.277
0.861:0233
0.8420.31
0.83:0.29
0.83-10.3
0,2420.11
N/A
BAGEL
0.65+0033
0.83++0224
0.76+0..7
0,81+0.29
0.76+H0.26
0,24+0.12
N/A
UNO
0.6111031
0.80.10.25
0.79110.3
0.80.10.00
0.75.10.25
0.2410.11
N/A
OmniGen2
0.62+0.31
0.81#0.26
0.74=0.38
0.81#0.33
0.74
0.24
N/A
Infinity
0.5810.33
0.7910.27
0.7610.36
0.76.10.33
0.72.10228
0.2410.11
N/A
SDXL
0,4910.33
0.7210.31
0.7110.39
0.72.10.5
0.66110.00
0.2410.11
N/A
Janus Pro
0.40+0.31
0.52+0.34
0.60+0.41
0.58+0.38
0.52+0033
0.23+0.10
N/A
Text-guided Image Editing
GPT-4o
0.7710.31
0.8110.30
0.7910.35
0.8410.32
0.80.0.28
0.2510010
0.54
Flux. I-Kontext-dev
0.52+0.38
0.66+0.34
0.67+0.41
0.76++0.36
0.65+0031
0.25+0.10
0.52++0.18
BAGEL
0.5610.37
0.6110.35
0.6810.39
0.70.10.7
0.64110.2
0.2511009
0.3340.23
Gemini 2.0 Flash
0.5710.37
0.6110.36
0.6310.42
0.6810.40
0.62:10.34
0.25
0.39:0.23
OmniGen2
0.34++0.35
0.56++0.35
0.61+00.43
0.72++.339
0.56++0.32
0.25+0009
0.42+0025
IC-Edit
0.2540.34
0.501:0.5
0.63+0.41
0.661:0.0
0.51
0.25+0009
0,23+0.19
InstructPix2Pix
0.16+0.27
0.45+00.35
0.58+0.43
0.61+00.3
0.45-+0.30
0.25+00099
0.27+0.19
XtepiX-Edit
0.3510.34
0.44.00377
0.4610043
0.5310042
0.44.10.34
0.25110.99
0.36.10.27
Single Reference Image Generation
GPT-Image-I
0.7740.28
0.91+:0116
0.88+0.24
0.93+0.20
0.88:0.17
0.15#00099
0.73+0008
Gemini 2.0 Flash
0.6110.32
0.73.10.35
0.7310.38
0.77.10.5
0.69.10.88
0.17110.10
0.59.10.16
BAGEL
0,5110.30
0.68.10.32
0.6510.42
0.73100.7
0.64100.0
0.17100110
0.6810.17
UNO
0.40+0.30
0.65+00.77
0.66+0.40
0.73++0.37
0.61
0.16+0.10
0.66+0.14
OmniGen2
0.4110.33
0.6310.35
0.64110.2
0.7410.37
0.6110031
0.1710.11
0.5310.24
Single Reference Image Editing
GPT-Image-I
0.75+0.30
0.80+0029
0.78+0.35
0.84+0.32
0.79-
0.66+0.12 0,24+0.10
Gemini 2.0 Flash
0.4010.35
0.5510.35
0.5910.42
0.6710.39
0.5510.32
0.2410010
0.5910.16
OmniGen2
0.30110.0
0.5910.34
0.5810.43
0.7010.40
0.54
0.2410.11
0.6910.11
BAGEL
0.31+0.33
0.56+
0.59+0.42
0.67+0.38
0.53++0.00
0.24+0.10
0.57+0.16
Multiplc Reference Image Generation
GPT-Image-1
0.6910.31
0.75100.0
0,7310.37
0.8010.33
0.7410.29
0.1610.10
0.72.10.7
UNO
0.35+0.24
0.64+0.27
0.62+0..99
0.73+0.36
0.59+0.226
0.13++0.99
0.73+00.9
Gemini 2.0 Flash
0.3511.30
0.4910.35
0.5010.42
0.5910.41
0.48-10.2
0.1710.10
0.6910.09
OmniGen2
0.2611.28
0.5110.34
0.45+0042
0.63+0.40
0.46:1.30
0.1620.11
0.72+0..9
BAGEL
0.23:1028
0.3911034
0.36:10.00
0.49::0.40
0.37:10.29
0.14:0..99
0.70
Multiple Reference Image Editing
GP'T-lmaee-1
0.6910.31
0.7510.30
0.7310.37
0.80+1.33
0.74-0.29
0.16210110
0.72+00.7
Gemini 2.0 Flash
0.35++.30
0.49++0.3
0.50+0.42
0.59+0.41
0.48++0.22
0.17+0.10
0.69+00.0
OmniGen2
0,2640.26
0.5140.34
0.45+0.42
0.63+0.40
0.46-10.0
0.16#0.11
0.73:0.07
BAGEL
0.23+0.24
0.394:034
0.36+0.40
0.49=00.00
0.37:0029
0.16+0.10
0.70+0009
Table 8: Automatic evaluations on all entries in our dataset
28

---

Published as a conference paper at ICLR 2026
I.LM-based
Embedding & perceptual
Prompt relevance
Aesthetic quality
Content Coherence
Model
Artifact
Overall
CLIP
LPIPS
Text-guided Image
Generation
GPT-Image-1
0.88+0.22
0,95+0.13
0,91=0.23
0.96+0.11
0.92
0.24:0.12
N/A
Qwen-Image
0.77+0.31
0.93+0.17
0.87#0.28
0.89=0.24
0.86=0.20
0.23#0.12
N/A
Flux. 1.x.1-Ke 1-Kea-dev
0.72.10.1
0.8810.21
0,8010.35
0.8910.26
0.8210.23
0.2311011
N/A
Gemini 2.0 Flash
0.7810.28
0.8510.23
0.8210.31
0.8310029
0.82:1024
0.2411011
N/A
BAGEL
0.62+0.34
0.83+0023
0.74+0.37
0.82+0.30
0.75++0.26
0.23+0.12
N/A
OmniGen2
0.6210.33
0.8110.25
0.73+0.37
0.82:10.11
0.75
0.2210.12
N/A
UNO
0.57+0.32
0.7910.22
0.75+0.37
0.78+10.1
0.72:10.27
0.23:0.12
N/A
Infinity
0.5810.32
0.8010.26
0.76110.36
0.75:10.34
0.73
0.22:1..11
N/A
SDXL
0.48+0.33
0.7240.32
0.71
0.71+0.36
0.65+0.30
0.23+0.11
N/A
Janus Pro
0.41+0.30
0.55+00.3
0.62+0.41
0.60++0.37
0,54+0.32
0.22+0.10
N/A
Text-guided Image
Editing
GPT-Image-I
0.7740.31
0.81+0.30
0.79+0.35
0.85+0.31
0.80=0028
0.24:0.10
0.55
Gemini 2.0 Flash
0.59+0.37
0.6240.37
0.66+0.41
0.68+0.40
0.64=0.35
0.24+0.10
0.42+40.23
Flux. 1-k J-Kontext-dev
0.4810036
0.66110.35
0.6410.42
0.7510.37
0.63.1032
0.2410.10
0.52.10.19
BAGEL
0.51
0.57+00.77
0.65+0.41
0.64+0.39
0.59=
0.24+0.10
0.35++0.25
OmniGen2
0.34+0.36
0.58+0.35
0.63+0.43
0.74+0038
0.57+0032
0.25+0.10
0.41+0.26
IC-Edit
0.23:10322
0.4810.35
0.6010.40
0.6410.41
0.49=0022
0.24
0.2340.18
Stepl X Edit
0.34++0.35
0.43++0.88
0.45+00.5
0.52++0.33
0.43-+0.36
0.24
0.39
InstructPix2Pix
0.1410025
0.41110.35
0.5211043
0.58:10.44
0.41
0.24
0.26.10.19
Single Reference Image Generation
GPT-Image-I
0.73+0.28
0.91+0.16
0.87+0.24
0.92+0022
0.86++0.18
0.16+0.10
0.73+0.18
Gemini 2.0 Flash
0.58+0.33
0.72+:0.35
0.74+10.8
0.76++0.35
0.70:t0.28
0.16+0.10
0.
BAGEL
0.4910.30
0.68+0.32
0.65+0.42
0.73+0.37
0.65
0.16#0.10
0.69
OmniGen2
0.41110.33
0.6311035
0.6410.42
0.7410.37
0.62.10.30
0.1710.11
0.55.0.24
UNO
0.3810.29
0.6410032
0.6510.40
0.7310.37
0.60.10.29
0.1510.10
0.6810..14
Single Reference Image Fditing
GPT-40
0.67:10.30
0.7710029
0.78+1035
0.84:10.32
0.76:10.29
0.24:10.10
0.65
OmniGen2
0.2910.31
0.58.10.4
0.5810.43
0.70.10.0
0.54
0.2410.10
0.6910.11
BAGEL
0.3840.35
0.54+00.6
0.60+0.42
0.65+0038
0.53+0.32
0.24+0.10
0.56+0.17
Gemini 2.0 Flash
0.42.10336
0,52.10.37
0.5910.42
0.68.10.40
0.51
0.24110110
0.5910.16
Multiple Reference Image Generation
GPT-Image-I
0.7940.25
0.91+00117
0.90+0.25
0.93+0.21
0.88:0.16
0.14+0.09
0.73+0.08
Gemini 2.0 Flash
0.5940.31
0.70+-0.30
0.70+0.39
0.74+0.37
0.68+0.30
0.14+0.09
0.74+0.09
OmniGen2
0.40:10288
0.64.10.28
0.5810.40
0.74110.8
0,5910.28
0.1310.10
0.72.10.09
BAGEL
0.47+0.26
0.61+0033
0.61+0.41
0.68++0.39
0.59++033
0.14+0.09
0.74+0.08
UNO
0.35+0.25
0.65+0.26
0.62+0.39
0.72+00.3
0.58 +.25
0.13++0.99
0.74+0.08
Multiple Reference Image Editing
GPT-Image-1
0.67+0.31
0.77+-0.0
0.73+0.36
0.82+0.32
0.75++0.28
0.16#0.10
0.72+0008
Gemini 2.0 Flash
0.36+0.50
0.51+0.35
0.53+0.42
0.62+0.41
0.51+0.31
0.17+0.10
0.69+0.09
OmniGen2
0.2310.23
0.5010.34
0.4210.42
0.6410.41
0.45+0.30
0.1510.11
0.7310.07
BAGEL
0.23+0.25
0.40++0.33
0.37+
0.52+00.0
0.38++0.28
0.16+0.10
0.70+00.9
Table 9: Automatic evaluations human-annotated subset
29

---

Published as a conference paper at ICLR 2026
A.8 PROMPT TEMPLATES
Prompt Template for Image Generation & Editing
You expert visual generation assistant.
Task: (TASK NAME}
Task Definition: YEASKDEFINITION
Visual Domain: [TOPIC_N) NAME)
User Objective: {USER PROMPT}
Attached Images: (IMG LIST) (None for Text-guided image generation)
Please generate an image that fulfills the user's objective, adheres to the task definition, and fits within the specified visual domain.
TASK_NAME:
Text- Text-guided Image Generation
Text-guided Image Editing
Single Reference-guided Image Generation
Single Snference-guider Image Editing
Multiple References-guidee Image Generation
Multiple References-guiie Image Editing
DASKIDEFINITION:
Text-guided Image Generation: Generate a completely new image based only on descrip- tive text prompt. No source reference images provided.
Text-guided Image Editing: Edit an existing image using a descriptive text prompt. Decide what to modify in the image based on the prompt. No mask or marked region given.
Single inference-guided Image Generation: Create a new image by combining visual cues from one reference image with instructions from a descriptive text prompt.
Single seference-guidder Image Editing: Edit an existing image using both a reference image and text prompt. Use the reference image to guide the style or content of the edits.
Multiple References-guee guided Image Generation: Generate ence images along with a text prompt. The new image should reflect visual elements from the references and follow the prompt's description.
image using several refer-
Multiple References-guidee Image Editing: Modify an existing image using multiple reference images and a descriptive text prompt. The edits should guided by both the style content of the references and the instructions in the prompt.
ROPIC_NAME
Information Graphics
Artworks
Screenshots
Computer Graphics
Photorealistic Images
Textual Graphics
30

---

Published as a conference paper at ICLR 2026
Prompt Template for Image Evaluation
You are an expert Al image evaluator. Your task is rate generated image based on a provided text prompt and any reference images. Use the following guidelines for your assessment. Provide rating from for each criterion. Do NOT include any additional text or explanations in your response. The response MUST be a single JSON object.
Quality Assessment
Prompt Relevance
Definition: Whether the image accurately reflects or responds to the prompt.
Rating Guide (1-5):
- Completely unrelated to the prompt.
2 Mostly incorrect; some vague connections but many mismatches.
3 Partially relevant; key ideas are present but with errors or omissions.
Mostly accurate; follows the prompt well with minor issues.
5 3- Fully aligned with the prompt: clear, focused, and complete.
Aesthetic Quality / Visual Appeal
Definition: Whether the image visually appealing. clean, and easy to interpret.
Rating Guide (1-5):
Visually poor; unattractive, hard read or confusing.
2 Below average; noticeable design flaws, poor readability.
Decent; generally readable but has minor layout/design issues.
4 Clean and aesthetically good; professional feel with few flaws.
5 Beautiful, polished, and visually excellent.
Content Coherence
Definition: Whether the content in the image is logically consistent and fits together meaningfully. Rating Guide (I-5):
- Internally inconsistent or nonsensical; parts contradict each other.
2 Some logic, but confusing mismatched components.
Mostly coherent, though there are noticeable mismatches or awkward parts.
4 Logically sound overall, with only minor inconsistencies.
5 Completely coherent and internally consistent.
Artifacts/ Visual Errors
Definition: Whether the image has visual flaws due to generation errors (c.g., distortions, glitches).
Rating Guide (I-5):
Severe artifacts that ruin the image.
2 Major flaws that are clearly noticeable.
3 Some minor artifacts, but the image remains usable.
Mostly clean; only very subtle flaws if any.
5 5- Perfectly clean; no visible artifacts at all.
Expected Output (single JSON object):
pprromtt_reeevance <rating>, bbesthetic_quallee <rating>, "content_ "ecatten_ccohreecce" <rating>, "artifacts"; <rating>
31


Reviews:
[{"review_id": "lO9OjzoMEj", "reviewer": "Reviewer_dy2F", "summary": "The paper introduces ImagenWorld, a unified benchmark for image generation and editing that spans six task types (generation/editing with 0/1/multi references) and six topical domains, totaling 3.6K condition sets. It is supported by ~20K fine-grained human annotations and an explainable evaluation schema that tags object-level and segment-level issues, alongside automated VLM-based metrics. A large-scale study across 14 models reveals consistent trends: models struggle more with editing than generation, and with text-heavy/symbolic domains (e.g., screenshots, infographics), while targeted data curation can narrow gaps; VLM-as-judge correlates well with humans (Kendall up to 0.79) but lacks fine-grained error attribution. Overall, the work positions ImagenWorld as both a rigorous benchmark and a diagnostic tool for real-world image synthesis.", "questions": "Please see the weakness part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Substantially more “solid” than typical T2I-only benchmarks. Unlike prior works that center on narrow, compositional prompts, ImagenWorld covers six tasks × six domains with 3.6K condition sets and ~20K human annotations, and adds explainable labels (object-level and segment-level issue tags) that go beyond opaque scalar scores—this breadth and diagnostic depth should be highly valuable to the community.\n\n2. Well-designed evaluation pipeline that bridges rigor and scalability. The study evaluates 14 diverse models under a single protocol and reports aligned human/VLM results (Kendall up to 0.79) while retaining human-annotated, localized error tags—striking a practical balance between scalable automatic scoring and explainable human diagnostics.\n\n3. Clear, readable writing. The paper consistently defines evaluation criteria (Prompt Relevance, Aesthetic Quality, Content Coherence, Artifacts) and motivates them with concrete examples, which makes the methodology easy to follow.", "weaknesses": "The paper lacks a solution method showing how to leverage the benchmark’s diagnostics (e.g., a targeted data-curation or instruction-parsing baseline). Adding such a solution would make the benchmark more actionable with minimal extra work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RMnB1prvkb", "reviewer": "Reviewer_QiZo", "summary": "The paper introduces Imagen World, a large-scale benchmark designed to stress-test generative image models across a broad spectrum of 3,600 condition sets, covering six core tasks (generation and editing with varying reference guidance) and six topical domains. To address the limitations of opaque scalar metrics, the authors propose an explainable human evaluation schema, collecting 20,000 annotations that include both standard ratings and fine-grained object/segment-level error tags. The study evaluates 14 models, revealing that closed-source systems generally lead, particularly in complex editing tasks and text-heavy domains, while also benchmarking VLM-based automated metrics against these human judgments.", "questions": "- Can you clarify exactly how the VLM judge was implemented? Did it evaluate all four dimensions in a single pass? Have you considered that this might introduce a bottleneck, and have you experimented with specialized VLM judges for each distinct criterion to improve correlation?\n\n- In Table 4, the VLM significantly under-penalizes artifacts compared to humans (positive bias). Can you provide some empirical evidence or analysis explaining this discrepancy? Is it a fundamental limitation of using current VLMs that may have been trained on noisy data?\n\n- How was the prompt template for the VLM judge (Appendix A.7) derived? Did you utilize a \"gold set\" of human-annotated examples to validate and optimize this prompt before deployment?\n\n- Regarding the reported Krippendorff’s alpha in Table 3, why was the least inter-annotator agreement found for the overall best-performing model (GPT-Image-1 in TIG)? What does this imply about the subjectivity of evaluating higher-quality outputs?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The benchmark addresses a clear gap in current evaluation by unifying a diverse set of tasks (generation and editing) and real-world domains (e.g., Information Graphics, Screenshots) under a single standardized protocol.\n\n- The proposed explainable evaluation schema is a logical attempt to move beyond simple scalar scores. By asking annotators to explicitly tag object and segment-level failures, the benchmark provides richer diagnostic information on why models fail.\n\n- The authors provide a large-scale study of 14 varied models, offering a comprehensive view of the current capabilities of both unified state-of-the-art systems and task-specific baselines.", "weaknesses": "- The automated evaluation hinges entirely on a proprietary model (Gemini-2.5-Flash) accurately assessing four distinct criteria (Relevance, Aesthetics, Coherence, Artifacts) and performing object listing. This reliance on a specific, closed-source model acts as a severe reproducibility bottleneck. If the model version changes or becomes unavailable, the automated portion of the benchmark is fundamentally compromised.\n\n- The paper lacks necessary statistical rigor for a benchmark standard. It presents mean scores without conducting statistical significance tests. Without these, it is hard to determine if the performance gaps between top-performing models are meaningful or simply noise.\n\n- While the paper reports inter-rater agreement (Krippendorff’s alpha), it fails to adequately address the high variability observed (ranging from a low 0.52 to 0.80 in Table 3). With such high disagreement for certain models/tasks, it is difficult to gauge the actual reliability of the human ground truth in those areas.\n\n- The paper fails to cite or compare against established \"decompose-and-verify\" metrics that are highly relevant to fine-grained evaluation. A key missing reference is Gecko [1]. \n\n\n[1] Wiles et al., Revisiting text-to-image evaluation with gecko: On metrics, prompts, and human ratings, ICLR 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FTqKyKiBgA", "reviewer": "Reviewer_7NZP", "summary": "This paper introduces ImagenWorld, a new large-scale benchmark designed to \"stress-test\" modern image generation and editing models.  ImagenWorld is built on a comprehensive foundation comprising:\n- Six Core Tasks: It covers a wide range of functionalities, including text-guided image generation, single and multiple reference-guided generation, and the corresponding editing variations of these tasks.\n- Six Topical Domains: The benchmark includes a diverse set of real-world use cases: artworks, photorealistic images, information graphics, textual graphics, computer graphics, and screenshots.\n- Explainable Human Evaluation: At its core, ImagenWorld relies on a detailed human evaluation schema. Instead of just providing a single score, annotators identify and tag specific, localized errors at both the object level (e.g., a \"missing chandelier\") and the segment level (e.g., visual inconsistencies in a specific region of the image). This is supported by 3.6K condition sets and 20K fine-grained human annotations.\n\nThe authors conduct a large-scale study evaluating 14 different models, including closed-source systems like GPT-Image-1 and Gemini 2.0 Flash, as well as various open-source models. The key findings from this evaluation are:\n- Models generally struggle more with editing tasks than generation, often failing by either completely regenerating the image or ignoring the edit instruction entirely.\n- All models excel in artistic and photorealistic domains but perform poorly on text-heavy and symbolic domains like screenshots and information graphics. An exception is Qwen-Image, whose performance suggests that targeted data curation can mitigate this weakness.\n- While closed-source models lead in overall performance, open-source models are competitive in standard text-to-image generation, indicating that progress in more complex editing and multimodal tasks requires more than just scaling.\n- Modern Vision-Language Models (VLMs) used as automated judges achieve high correlation with human rankings but fail to provide the fine-grained, explainable error analysis that human annotators can.", "questions": "1. Regarding to the weakness point mentioned above, how do you conclude that VLMs cannot provide more fine-grained details? Can this be improved by simply giving few-shot examples or fine-tuning the evaluator to create fine-grained criteria lists?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper presents a valuable benchmark and evaluation protocol for image generation models. As the capabilities of the models are improving and we are moving to models that can perform several different tasks from text-to-image generation to multi-image reference image editing unifying evaluation is important.\n2. The authors systematically evaluate several closed- and open-source models and draw conclusions on the current weaknesses of image generation models. They make sure to include different model families and measure performance for different tasks and with respect to different generation aspects (prompt relevance, content coherence, etc)\n3. The authors assess the reliability of automatic judges by measuring correlation with human ratings and find that there is moderate to strong correlation with coarse-grained ratings per generation aspect.", "weaknesses": "1. One important contribution of the paper is the collection of fine-grained feedback on the image generation errors by human raters by annotating the object and segment errors in the images. These annotations are not being sufficiently utilized in the paper for either analysis or automatic judge improvement. The authors mention is Section 5.3 that \"Taken together, the results suggest that VLM-based evaluation provides a reliable proxy for human annotation, particularly for assessing semantic alignment and visual appeal, while human judgment remains essential for identifying fine-grained details such as visual defects.\" without providing any analysis on the fine-grained collected data (they only present results and correlation on the coarse-grained ratings). Moreover, it would be great to show whether a subset of these data can be used as a few shot prompt or for fine-tuning in order to improve the capabilities of VLMs on evaluating fine-grained errors in image generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)