EdiVal-Agent: An Object-Centric Framework for  Automated,    Fine-Grained Evaluation of Multi-Turn  Editing
Authors: Tianyu Chen
Yasi Zhang
Zhi Zhang
Peiyu Yu
Shu Wang
Zhendong Wang
Kevin Lin
Xiaofei Wang
Zhengyuan Yang
Linjie Li
Chung-Ching Lin
Jianwen Xie
Oscar Leong
Lijuan Wang
Ying Nian Wu
Mingyuan Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-turn editing benchmark
image editing
Primary area: datasets and benchmarks

Abstract: Instruction-based image editing has advanced rapidly, yet reliable and interpretable evaluation remains a bottleneck. Current protocols either (i) depend on paired reference images—resulting in limited coverage and inheriting biases from prior generative models—or (ii) rely *solely* on zero-shot vision–language models (VLMs), whose prompt-based assessments of instruction following, content consistency, and visual quality are often imprecise.

To address this, we introduce **EdiVal-Agent** , an automated and fine-grained evaluation framework grounded in an object-centric perspective, designed to assess not only standard single-turn but also multi-turn instruction-based editing with precision. Given an input image, **EdiVal-Agent** first decomposes it into semantically meaningful objects, then synthesizes diverse, context-aware editing instructions while dynamically updating object pools across turns. These two stages enable two novel object-centric metrics tailored for multi-turn evaluation and one global metric of visual quality: 1) EdiVal-IF, which measures instruction following by combining open-vocabulary object detectors for symbolic checks with VLMs for semantic verification on detector-guided crops; 2) EdiVal-CC, which evaluates content consistency by calculating semantic similarity of unchanged objects and background using the evolving object pools; and 3) EdiVal-VQ, which quantifies changes in overall visual quality with human preference models.
 
 



Instantiating this pipeline, we build **EdiVal-Bench**, a multi-turn editing benchmark  covering 9 instruction types and    16 state-of-the-art editing models spanning in-context, flow-matching, and diffusion paradigms.  Our results show that *Seedream 4.0* achieves the best overall performance, offering the strongest balance of instruction following, content consistency, and latency. *GPT-Image-1.5* clearly improves over *GPT-Image-1*, especially in content consistency across turns, while *Nano Banana 2* consistently outperforms *Nano Banana* in instruction following and overall score,. Among flow-matching models, *FLUX.2-max* is the strongest baseline, whereas *Qwen-Image-Edit* performs well on the first turn but degrades sharply in later turns, indicating strong exposure bias in multi-turn editing. We demonstrate that **EdiVal-Agent** can be used to identify existing failure modes, thereby informing the development of the next generation of editing models. Our code is available at https://github.com/TianyuCodings/EdiVal.

Published as a conference paper at ICLR 2026
EDIVAL-AGENT! AN OBJECT-CENTRIC FRAMEWORK FOR AUTOMATED, FINE-GRAINED EVALUATION OF MULTI-TURN EDITING
Tianyu Chen 133. Yasi Z Zhi Zhang", Peiyu Yu², Shu Wang", Zhendong Wang", Kevin Lin Xiaofei Wang", Zhengyuan Yang", Linjie Li,, Chung-Ching Lin Jianwen Xie Oscar Leong?., Lijuan Wang Ying Nian Wu Mingyuan Zhou 1.3.1
'Tee University of Texas at Austin Uniieersity of California, Los Angeles
3 Microsooft AI Superintelligence *Lambda, Inc
* "Equal contribution. 'Equal advising. Work done during an internship at Microsoft Correspondonce to t anaacche@uuteeass edu and ya smsmi www.innnnnnnnaa...e.mmom edu.
Website:
Code
Dataset
ABSTRACT
Instruction-base image editing has advanced rapidly, yet reliable and inter- pretable evaluation remains bottleneck. Current protocols either (i) depend on paired reference tanges-resulting in limited coverage and inheriting biases from prior generative models-ov or (ii) rely solety zero-shot vision- language mod- els (VLMs), whose prompt-based assessments of instruction following, content consistency, and visual quality are often imprecise.
To address this, we introduce EdiVal-Agent, an automated and fine- fine-grained evalu- ation framework grounded in object-centric perspective, designed to assess not only standard single-turn but also multi-turn instruction-based editing with preci- sion. Given input image, EdiVal-Agent first decomposes it into semantically meaningful objects, then synthesizes diverse, context- awarre editing instructions while dynamically updating object pools across turns. These two stages enable two novel object-centric metrics tailored for multi-turn evaluation and one global metric of visual quality: 1) Edi Val-IF which measures instruction following by combining open-vocabulary object detectors for symbolic checks with VLMs for semantic verification on detector-guided crops: 2) Edi which evaluates content consistency by calculating semantic similarity of unchanged objects and background using the evolving object pools; and 3) Edi VQ, which quantifies changes in overall visual quality with human preference models.
Instantiating this pipeline, build EdiVal-Bench, multi-turn editing bench- mark covering 9 instruction types and 16 state-of-the-arr editing models spanning in-context D flow-matching, and diffusion paradigms. Our results show that See- dream 4.0 achieves the best overall performance, offering the strongest balance of instruction following, content consistency, and latency. GPT-Image- 1.5 clearly improves over GPT CPT-Image-1, especially in content consistency across turns, while Nano Banana 2 consistently outperforms Nano Banana in instruction following and overall score.. Among flow-matching models, FL.XX2-max is the strongest baseline, whereas
performs well on the first turn but degrades sharply in later turns, indicating strong exposure bias in multi-turn editing. We demonstrate that EdiVal-Agen can be used to identify existing failure modes, thereby informing the development of the next generation of editing models. Our code is available at httpp:: http:::///iitusse com/ httmm//aaaaoonnn//aaaaaaaa
In this paper, we label certain closed models GPT- 1/1.5, Nano Banana 1/2, and Gemini 2.0 Flash Image-as in-context, since they are integrated into autoregressive language models in the web UI and support in-context multi-turn editing.

---

Published as a conference paper at ICLR 2026
Bar Erroae
THORESS
Seoen huse deee
Beplase
of wilte borse Srown
HOOBSESS
Leoas boner dee
Seplace
of cfllite lurse leows
HORSES
HORSES
HORSES
HORSES
HORSES
HORSES
Instuctonn
Indructin Cd
rollctorl
Inspuctoon E
Instactonn E
Gore X3
Gonee 22 Do Quality (3
Consadency X ga
Conserweyy D3
Commatency E3 (01229)
Concadercy 23
Qual Ir E38554 wtwweeero
7
Quality
Quality D 8:13
GPT GImg--m
Qwen- Emoge Edit
HORSES
HORSES
HORSES
SES
Indrucionn E1
Vetrutionn E
Inercciin E
retoucion E8
Ineturtion E
Inetuctton: E
Ocro stancy:
E 190 541
Oo stovec 11 6M)
Conobtuncy, E B7
Coneetency, & MS
Conditerey Quality (87.1)
Quality 242
Quality 7-11
Qualiyy C 8.22)
QueRy: E 19.400
Ouilt
y 7:20
Nano Benana
FLUX Kontest- dev
Figure I: Overview of our workflow and representative model's performance. For visualization, we adopt two thresholds: consistency score of at least 90 and visual quality score of least 6. Details of the automated evaluation pipeline are provided in Figure 2 and Section 2 In multi-turn editing, models exhibit distinct weaknesses: GPT-Image-I struggles with content consistency, Qwen-Image- Edit underperforms in both visual quality and content consistency, and FLUX. I-Kontext-dev lags in instruction following, whereas Nano Banana shows no single dominant weakness.
INTRODUCTION
What truly defines the success of an image editor? At its core, editing requires making targeted, instruction-driven changes while preserving contextual consistency and perceptual realism- offiis across multiple refinement turns. Yet current evaluation practice struggles to capture this multi- faceted objective.
When ground-truit truth edited images are available, a common strategy is to compare model outputs against these references (e.g.. MagicBrush Zhang et al. (002)). UltraEdit et al. (2024), AnyEdit eu al. (2025), EmuEdit (Sheynin et al. (2024)). Typical metrics include pixel-level distances (e.g.. LI/L2) and semantic similarities (c.g., DINO Caron et al. (222]) and CLIP Radford et al.) (2021), ))
While informative, such metrics suffer from two structural issues: (i) the space of acceptable edits inherently large, whereas a single reference provides only one realization; and (ii) references are frequently synthesized by existing editing models (e.g., Prompt-to-Pront Hertz et al. (2023), SDXL Podell al. (2024). DALLE-2 Ramesh RRmmssh et al.] (2022). thereby importing their biases and limitations into the evaluation itself. Consequently, high reference similarity does not necessarily imply faithful instruction following, preservation of irrelevant content, or aesthetically plausible outcomes.
A complementary line of work employs zero-shot VLMS interpretable evaluators (e.g.. VIEScore Ku et al. (2023). GEdit-Bench Liu et al. (2025), I'Beench Ma et al. (2024). HQ-Edit Hui et al. (2024), Complex-Edit Yang et al. (2025), and ImgEdit Ye et al. (2025) and queries VLMs about specific aspects of an edit. While VLMS offer holistic, language-
judgments, they remain insufficient for precise editing assessment for several reasons. First, for mnstruction-followwe
evalu- ation, they are notoriously poor at spatial reasoning Zhang et al. | (2025b); Chen et al. (2024): Chang al. (2025) and are prone object hallucinations in existence, category, attributes, and relations Bai et (2024). These issues together undermine their ability to assess common object-related edit instructions. Second, they have limited sensitivity to pixel-level changes and frequently miss subtle, localized modifications Vo et al. (2025) (c.g., fine structures, small attribute shifts, etc.), which are crucial for evaluating content consistency. Third, since they are predominantly pretrained on natural images rather than synthetic generations, their priors are miscalibrated for artifacts and aesthetics, leading to failures in detecting common generative defects (e.g., extra fingers) and in modeling perceptual "naturalness" LLing et al. (2024); Xu et al. (2023); Ma et al. (2025). which humans are sensitive to. Consequently, VLM-only scoring lacks the precision and reliability required for fine-granne fine-grained evaluation across instruction following, content consistency, and visual quality. How- ever, we find recently open-source state-of-the of-te-of-the-ar editing models (e.g., Qwen-Image- Image- Wu et al.) (2025)), Step1 X-Edit Liu et al. 2025), Bage Deng et al. (2025) solely rely VLMS for evaluation.
2

---

Published as a conference paper at ICLR 2026
Decompostion
Instruction Generation
Evaluation
Imagi Decompasition and Anelysis
Editing Insteustion Geceration
Fine-Grsined Reslustion
to ideallés and describe all dearlyy visible objects the
Toak Deseription You will
Yier tade
MAX TURAD
istage ar stractared JSON Samor.
Output Formatt
crreeaa (Inttraction by typoecaeor aller? inerrattion
Task Descr pilion: You ard govenn as ineas and sakked to
butreatke
Sermat "materiaal (coler) (ubject same!"
1. Each objoc mast lasad aa key is fae ISON, sanng the
Object to change colen (object mame pole!
3. Ai dae and the ISON, iachade as additionl key called abjarte..
"count" "Voregrsand"
Sillwwing filde 'vhjert" "aljerr"
"caler" "maneial" "ted""
Seaport simple commant onker that would lock good Biceogect
Dettalled
slentily
Pinal
Choose differes foom cureert
JBODT Ooject Lank
Insttration Type
Clyats POSt
subrect ald, ndiject ranoos sabjez peplace, color alle
DefocTee Piheriig
AEUREN PACT
OEpect Pool
balkgourd chazs
endiral wi wyp.e
Figure 2: Framework of AdiVal-Agent. first decomposes images into semantically meaningful objects, such as metal yellow sign and metal brown pole, and identifies their contextual relationships, e.g., they are both in foreground. It then generates diverse and proper editing scenarios at scale which are based on the initial analysis, e.g., Change the color of metal brown pole to gray. Finally, it systematically evaluates editing model outputs from multiple axes with our proposed metrics: Ea Edi and Val-VV. Our agentic pipeline is agnostic the expert tools used and can be readily enhanced with more advanced tools in the future.
To address these challenges, we introduce EdiVal-Agent: an automated and fine-graine evaluation agent for multi-turn instruction-basee image editing from an object-centric perspective, designed to assess not only standard single-turn but also multi-turn instruction-based editing with precision. As shown in Fig. 2 EdiVal-Agent first decomposes it into semantically meaningful objects, then syn- thesizes diverse, context-aware editing instructions while dynamically updating object pools across turns. These two stages enable two novel object-centric metrics tailored for multi-turn evaluation and one global metric of visual quality: 1) Val-IF which measures instruction following by combin- ing even-vocabulary object detectors for symbolic checks with VL.Ms for semantic verification on detector-guides crops: 2) EdiVal-CC, which evaluates content consistency by calculating semantic similarity of unchanged objects and background using the evolving object pools; and 3) EdiVal- VQ, which quantifies changes in overall visual quality with human preference models. We show evalua- that Edi Val-IF yields stronger agreement with human judgments in instruction-follovii
tion compared to thresholded CLIP directional (CLIP dir) scores Gal et al. (2022) and using VLMs
alone, as evidenced in Sec. 2.5
Instantiating the agentic pipeline, we curate a new multi-turn image editing benchmark, EdiVal- Bench, featuring 9 instruction types and 16 existing editing models-spanning in-context, flow- matching, and diffusion paradigms, across both closed- and open-source systems conduct fine- grained analyses, and draw actionable insights. Empirically, as demonstrated in Fig. [ and Tab. 3. GPT-Image-I excels at instruction following yet ranks near the bottom in content consistency, whereas Seedream 4.0 and Nano Banana performs strongly on both axes. Besides, open-sourced models like Joven-Image-Fatee significantly degrade in instruction following and visual quality when editing turns increase, while FL.UX. 1--onoKxt--ma and FLUX. -Kontext-dev lags in instruction fol- lowing. We further contrasts multi-turn editing with single-shot complex prompts Yang et al. (2025), highlighting complementary strengths and failure modes. We hope that our agent pipeline, bench- mark, and analyses accelerate the transition of multi-turn editing toward practical applications.
Key contributions. 1) Agent: AdlVentAgent is a fully automated evaluator that performs object- centric decomposition, generates diverse multi- fulli--ummmoe editing instructions, and measures overall edit- ing quality using two object-centric metrics (EdiVal-IF and EdiVal-CC) plus Edi for visual quality. 2) Benchmark: using Edival-Agent, we construct Edival- with 1,716 instructions
3

---

Published as a conference paper at ICLR 2026
Table 1: Key attributes of open-source edit benchmarks. Note that ImgEdit Ye al. (2025) include multi-tury editing experiments in the paper.
does not
Benchmark
# Size
Object-eentrie
Automnted
Malti-Turn
Free from Ref. Images
Tools used
Emalia (Sheynin et al
2024
3,055
X
LI, DINO
MagicBrusd/Zaag
2125
1,053
L2, DINO, CLIP
AnyEdi
1,250
CLJ DINO
2ERenncMMaetn 2024
2,240
X
VLM
GEdit Behref Iu 2025
606
X
K
VLM
HQ Edi orai 2024
1,651
V
VLM
ImgEdii-Bex0ee
Context-Benclhe Land die 2023
1,026
/ Human Amnotation
EdiVal Bench tours)
1,716
V Detector. VLM. LI, DINO, HPS
across nine types and three turss on 572 real-world images, with comparisons to prior benchmarks in Tab. D Human agreement: EdiVal-IF attains 81.3% agreement with human ratings for in- struction following, outperforming zero-shot VLMs and CLIP-based baselines. 4) Evaluation: we assess 13 editors (diffusion, flow-matching, and close source) along instruction following, content consistency, and visual quality. 5) Insights: overall ranking-Seedream > GPT--mmae--1; the strongest open-source editor, Qwen-Image-Fdie ex- hibits exposure bias under multi-turn editing. 6) Artifacts & settings: reveal luminance drift across turns, and contrast multi-turn against complex single-shot editing to delineate strengths and weaknesses across model families.
Seedream 4.0 > Nano Banana >
FLUXXI-Kontext-mn
AGINAL-AGENT
2.1 OVERVIEW
As illustrated in Fig. The pipeline comprises three stages: (1) Decomposition uses a VLM (e.g., GPT-4o; other VLMs are viable alternatives) to extract structured, object-level descrip- objects, attributes, and celation--enabling symbolic reasoning: (2) Instruction Generation produces multi-turn, diverse, compositional prompts by maintaining an explicit object pool and sampling from nine instruction types spanning subject-, attribute-, relational-, text-, count-, and global-level edits; (3) Evaluation scores edited images with EdiVal-IF, Edival-CC, and EdiVal-VQ.
2.2 STEP DECOMPOSITION
Given image, VLM-based agent parses clearly visible foreground objects and returns per-object ISON with fields object, color, material, text, count, and a boolean foreground, Names follow " "(material) (color) {object) unknown fields are omitted; person iden- tity is never recorded (only wearables/accessoriiss)) Example: { ("metal yellow si sign", "material", "metal", "text "SCHOO L", "count": "fat:ntt:ilf.feeeaiieeeeee "foreground" :true)). An aggregated al all_objects string concisely lists objects (e.g., "metal yellow sign metal brown pole"). We apply this stage to GEdit-Bench Liu et al. (2025) (606 images), exclude 34 images with sensitive personal content, and retain 572 images. After extraction, Brounding-DING validates objects and detects bounding boxes; only reli- able detections are kept seed instruction generation and evaluation. The filtered objects stored in the A1 1 Objects Pool and later used to initialize three distinct object pools that dynamically track the evolving state of instruction generation.
("object " ""sig""..""""""""""""""""""".""" """""""""" "yellow",
2.3 STEP 2: INSTRUCTION GENERATION
From the decomposed scene, the agent generates multi-turn edits that are grounded the cur- rent object state. The instruction taxonomy (nine types; six categories) appears in Table 2 We maintain three evolving pools at turn t: Pull (all objects ever present), Punch (original objects not edited up to t), and Pavail (objects currently editable). With a turn budget MAX_TURNS, at each turn the agent (i) selects sampling type not yet used in the chain; (ii) selects object(s) from pavail, (iii) emits natural- language instruction via GPT-4o referencing those objects and the scene state; and (iv) updates pall, pavail and punch according to the intended edit. When background change edit ap-
deffuliin to subject.add if prvail 0, otherwise
4

---

Published as a conference paper at ICLR 2026
plies at turn f, background-consistrn scoring is disabled since this turn, and append "make wtt.ctssii.ooorrrruuu.eoeoeo
unchanged" to the instruction to preserve object-level compara- true }. The
bility, where ob jects in foreground {o P.vail o.foreground loop is adaptive by expanding/contracting Parail and naturally compositional. Our default sets MAX_TURNS= (In our implementation, each turn is assigned distinct instruction type.), though longer chains are easily obtained by allowing repetition or adding types.
Table 2: Instruction types in EdiVal-Bench created by EdiVal-Agent, grouped by semantic cate- gory. Counts are shown per turn (T1-T3).
Category
Instruction Type
Example Instruction
T1
T2
T3
Total
aubject
Add bench on the left of metal red fire hydrant.
67
77
93
237
Subject-centric
aubject.remove
Remove wooden brown door.
75
69
205
subfect.replace
Replace gray railing with wooden fence.
54
57
55
166
Atribute--entric
color alter malerielallez
Change the color metal white airplane to blue.
56
73
57
186
Change the material of plastic black pen metal.
66
50
72
188
Text-related
change
Replace the text 'BEARS CONTRÓI: cotton
64
70
54
188
black cap with 'WILD PATH'.
Relational
posttrio_chharge
Change the position of ceramic white cup to right
52
63
48
163
plastic white laptop.
Counting
count_change
Change the count fur brown hear to
73
58
60
191
Globul
backgrounddctaagge
Change the background to forest, remain the brown fur bear unchanged.
65
55
72
192
2.4 STEP 3: EVALUATION
The first two stages enable two novel object-centric metrics for multi-turn editing evaluation for instruction following and content consistency, respectively, and one global metric for visual quality:
EdiVal-IF To evaluate instruction following, introduce EdiVal-IF, which assesses multi-tum edits by comparing the image from the previous turn, 1²-1, to the current image, I'. For a given instruction ps at turn t. the score is determined differently for symbolically and semantically ver- ifiable tasks. Symbolically verifiable types (Taym)-such as subject. add, subject remove, sub jecctrrpclllleeeeeeeeeeeeepppeee a ace, position..... conchaage, and count_change- are evaluated using open- vocabulary object detector Maewe Liu et al. (2024b). The detector's outputs, including bounding boxes and confidence, are assessed against geometrie and logical criteria Fsym derived from the instruction. For example, for posit iion.haanee instruction "Move [A] to the left of [B]", Fsym verifies that the x-coordinate of A's bounding box B center is less than that of B Ie, i... center,(B*,) < (B"ter,(B",) In this case,
EdiVal-IF(I', I'-1 1' P' T Frym (ym (Maeeter (I (I'-1I"(P'))
Semantically verifiable types (Tsm))oom collr_ooaa ller, material. __a Lal terr text_chaaeeeeeeeee and background.change. are evaluated with a VLM MVLM Yang et al. (2024). To focus the eval- uation, the VLM is applied to detector-guide object crops (lo) using instruction-specin templates.
EdiV 1111 pt E Tsem) = MVLM 1yIPt) MyLM(Manne (I (1))) (2)
We show that Edi Val-IF achieves superior hu- man agreement (Sec. 2.5). The multi-turn edit- ing success rate is defined as the logical AND of the EdiVal-IF scores across all edits along the chain, whereas the marginal task rate at turn t is defined according the formulas D and 2 pro- vided above.
(a) Base image
(b) GPT-Image-1 (c) 1-mx.1-max
Figure 3: Beautification vs. preservation under the prompt: "Change the background brary." GPT-Image- tends to increase HPSv3 via beautification, while FLUX.1 -onteextttxttaa em- phasizes fidelity to the input.
EdiVal-CC To assess content consistency, Edi measures the preservation of non- target content between the base image 1 and the current image /!. Given editing instructions
li-
5

---

Published as a conference paper at ICLR 2026
plat from turn to turn t, the object pools Phunc and Pi are dynamically updated. Let Ω2 denote the entire image area. Using object bounding boxes from the base image (B5)) and the current image (,)), extracted by the detector Mateeet the back- ground region defined as 22 = - UoEPp U )) i.e., the region obtained by excluding all objects that have appeared. Background consistency is then computed as bng < bbg) where lig = 0g 1s denotes the background of the image, and @ a similarity function such as L distance or DINO-based similarity. For unchanged objects, we compute the per-object consistency so = (I I5) for each o "Pinish, and then average them. Formally, the final EdiVal-CC score emphasizes semantic preservation by averaging the feature-level similarities of the background and unchanged objects (see Appendix, M.3 for details):
EdiVal-CC(4', 19,, pl.) =
5/g
(3)
oC
OE
EdiVal-CC aligns with the intuitive notion of consistency, providing precise measurement.
EdiVal-VQ. Zero-shot VLMs are not trained for reliable assessment of image qual- ity my-particularly artifacts and aesthetics-and we find they are imprecise as scoring functions (see adopt Human Preference Score v3 (HPSv3) Ma et al. (2025)
Appx. D. Consequently,
visual-quality (VQ) evaluator. In practice, applying preference models to unedited, real photographs often yields relatively low aesthetic scores. We also observe divergent behaviors across editors (See Fig. 3): some (e.g., GPT-Image-1) tend to beautify images and increase HPSv3, whereas others preserve the original appearance with minimal aesthetic change. Be- cause aesthetic preference is inherently user- and task-dependent, and beautification can trade off with content consistency (already incorporated into our overall score), we report EdiVal-VQ sepa- rately and do not fold it into the aggregate metric.
(e.g.,
EdiVal-O. We aggregate Instruction Following (EdiVal-IFF and Content Consistency (Edi Val-CC) into a single overall score. Since both metrics are unit-free and normalized to [o, 1] but capture com- plementary aspects, follow prior work and use the geometric mean balance them and penalize imbalance (Liu et al. 2025 Ku et al. 2023). Formally, EdiVal-O
VEdiVal-IF EdiVal-CCC
Design Scope and Limitations. We omit style le_hange from our taxonomy because style cate- gories are inherently ill-defined, which makes instruction-follwiin reliably. Extending EdiVal-Agent with style-aware recognition is promising future work. After language-based extraction, we validate objects using Grounding-Diie DDNgg Liu et al. (2024a) and prune low-confidence or ambiguous detections. This stabilizes the object pool and reduces error propaga- tion during instruction generation and IF evaluation. By default, we employ Grounding-DING as the open-vocabulary detector, Qwen2-VL as the VLM, and DINOv3 Siméoni et al. (2025) as the image feature extractor due their state of-te-of-theaarr art performance and open-source availability, which facilitates community use. The agentic pipeline is tool-agnosti and can be readily strengthened by substituting more advanced modules in the future.
Val-FF difficult to evaluate
2.5 MEASURING HUMAN AGREEMENT
Setup. We conduct human study on edits made by four exemplary models, Gemini 2.0 Flash and Flux. 1-Koeeext--,e,, on EdiVal-Bench, generated by EdiVal-Agent de- scribed in Sec. 2.3 In total, we collect 4,576 human annotations of edits. During evaluation, raters were shown the original image, the edited image, and the corresponding instruction, and asked binary question: "Evaluate whether the edited image successfully follows the given instruction."
AnyEdit,
Results. Figure summarizes the findings. EdiVal-IF achieves human agreement accuracy of 81.3%, significantly higher than VLM-only (Qwen-2-Vi, 75.2%), CLIP (65.4%), and other zero-shot VLMs. These results verify that integrating VLMs reasoning with object detection leads to better alignment with human judgment compared to existing methods. The inter-annotator's agree- ment rate (85.5% indicates the best performance any evaluation tool can reach.

---

Published as a conference paper at ICLR 2026
Agreement Analysis Task Type
CLP.0F VLLY (Qm5
Agreement (%)
Task Type
Figure 4: Results of human agreement. Dashed lines represent the average accuracy of each method. Vdi-FI achieves 81.3% human agreement accuracy, significantly outperforming the VLM (Qwen2-VI.) 75.2% and thresholded CLIP dir at 65.4%. Note that the CLIP dir thresh- old is tuned separately for each task.
We attribute the improvement in instruction-folloving evaluation to two factors. First, for sym- bolically verifiable instruction sypes-subject.ccem add, add, subject. remove, subject_reet and count_channge-Bae EalVal-FF relies solely on Grounding-DDne termines the success of edit by checking object presence/absence, the positions of object centers, and the number of bounding boxes. Results for posit iin change and sub subject remove show that these fixed rules, combined with Grounding-DINS edit evaluation. We hypothesize that errors in posi ceaiitionnchhe tior_change stem from poor spatial rea- soning, while failures in sub remo Second, semantically verifiable sypes-color.altes materialataallleeaaaaaaaaeeaaaee background,char www.odddhhhaaaaaaa are evaluated using Qwen2-VL combined with Grounding-DINO. The decomposition stage in EdiVal-Agent can supports evaluation by localizing text regions. enabling the LLM to reason more precisely about text edits. These findings indicate that VdiVal-IF not only enhances interpretability but also improves the practical applicability of evaluation pipelines in real- world settings that demand human-like understanding. Nonetheless, EdiVal-IF has failure modes, which we document and analyze in Appendix.
position_chang
de-
can significantly outperform Qwen2-V VL in
due to hallucinations regarding object existence.
and
BENCHMARKING MULTI-TURN EDITING
Table Results of multi-turn editing, Val-FI, EdiVal-CC,, and Edi Vdi-al-O across three sequential editing turns. Best per column in dark red; second-best in lighter red. indicates close-sourced model, and indicates open-sourced model.
EdiVal-IF
Edias-OC
EdiNal-O
Rank
Technique
In-Contest
(vimg)
T1
T12
Seodicaa 8
Unknowa
*GPT--naage-115
25.12.16
35.55
75.19
55.92
-21-00
04.222
21.20
$3..49
$4420
71.41
59.55
NNmo Ramana 2 8
26.03.01
23.70
73.89
54.17
36.
93.54
90.52
58.61
83.14
70.02
58.40
3
NNae Banana
25.08.26 26
9.20
70.70
50.06
35.35
93.91
90.45
$1.48
67.70
56.24
GIPT-Intggo-1
25.07.16 16
75.50
76.96
25.02.05
78.52
'FLUX2-max
25.12.16
75.55
"HUXX -LLUX.I-Knnext-uaa
25.06.03
Flow Matching
'uee image- Edit
25.08.04 X
115.08
72.90
44.06
22.55
84.22
30.52
77.98
75.36
59.56
41.93
9
1Stp1X Ex-li
25.04 25
20.42
61.89
34.97
17.53
92.76
88.52
85.21
75.77
55.64
38.95
10
|HUUX.1 Kontext-de dey
59.97
93.24
10mmies
24.09.11
54.72
$3.92
46.52
29.91
Anviditt
X
24.11 24
3.93
41.07
16.32
7.22
86.42
78.91
70.10
59.58
35.89
22.50
14
Dictusion
(MagieBush
Ultraaaa
X
23.06 16
24.07 07
4,08
3.15
42.31
51.37
17.70
15.73
6.36
4,50
86.96
86.80
$4.50
$1.26
76.86
$2.
66. 78.78
60.66
38.67
35.75
22.89
19.41
15
13
1PPPP
Summary of Results. Table shows that the benchmark has very tight top tier. Seedream 4.0 re- mains the best overall model (Rank 1), combining the strongest instruction-followee performance at T1/T3 (75.93/41.59), strong cross-turn consistency (92.51/88.03/)) 03/85.86/8)... and relatively low la- tency (14.55 s/img). GPT-Image-I.5 close second and achieves the highest overall EdiVal-O T1/T2 (84.29/71.41). substantially outperforming GPT-Image- I; compared with its predecessor,

---

Published as a conference paper at ICLR 2026
it improves EdiVal-CC by 13.49/13.42/12.99 points and EdiVal-O by 7.33/6.07/5.74 points across the three turns, although at higher latency (35.55 vs. 26.47 s/img). A similar trend appears in the Nano series: Nano Banana 2 consistently improves over Nano Banana in instruction following and overall quality, raising EdiVal-O from 81 48/67 77/56.44 to 83. 14/70.022/88.44 49, but these gains come with large speed penalty (23.79 vs. 9.20 s/img) and only marginal changes in consistency. Among the flow-matching baselines, FLUX.2-max is the strongest competitor, while Qwen-Image-Edii still degrades sharply as the number of turns increases 178.36-459.66-444 pronounced error accumulation. Overall, newer model versions improve multi-turn robustmess and instruction retention, but often by trading off latency. Among open-source systems, Qwen-Image Edit performs well initially (EdiVal-O 78.36 at T1) but degrades rapidly with additional turns, likely due to exposure bias as discussed below. We can see that there is a clear gap between the per- formance of closed-source and open-source systems. With the exception of Owen--mage--ile Endge our model rankings exactly match those reported on the Artificial Analysis leaderboard (rank by human vote) as of September 12, 2025; see Appendix.
in EdiVal-O), indicating
3.1 INSTRUCTION FOLLOWING
Nerdral Turk fate Aerose Tures
Marginal Task Success Rate. For given turn, the marginal task success rate (Eqns. and is the propor- tion of prompts for which the edit requested at that turn is successfully executed. By contrast, the instruction- following score in Table 3 reports the multi-turn task suc- cess rate at turn i: the logical AND of the Vdi-Al scores across all edits along the chain. Figure 5 summa- rizes per-turn performance. High-ranking models-such as Seedream 4.0, Nano Banana, and max mai-maintain relatively stable Val-FF across turns, even though Seedream 4.0 and FLUX.I-Kontex--eee not in-context editors (they do not condition prior contrast, several other models exhibit clear decline in marginal success as the number of turns increases.
13:00
are
Figure 5: Marginal Task Success rate acToss turns.
prompts or intermediate images).
A particularly salient case is Qwen- Iwan-Image-Edit. Although it is the strongest open-source system at turn Val-O 78.36 vs. 81.48 for Nano Banana), its performance degrades more rapidly over subsequent turns. We hypothesize that this stems from exposure bias (Ning al. 2023 Schmidt, 2019): many single-turn editors are trained to operate on real images and ground-truth inputs rather than on their own previous outputs. When asked to edit their own generations, small distributional mismatches compound across turns, reducing stability; this effect is further aggravated when the model can only attend to a limited history.
Marginal Task Success Rate Across Instruction Types. We analyze marginal subtask success rates across turns for different instruction types. The results for Nano Banana are shown in Fig. 6 Other editing models exhibit similar behavior. Nano Banana performs relatively well on attribute- centric tasks such as color_alter and mater material.altet....riaai......eae and count change, indicating weaknesses in spatial and numerical reasoning, respectively.
but poorly on posit iistiiii chnnchanne
3.2 CONTENT CONSISTENCY
We evaluate two aspects: (i) unchanged-object consistency (Fig. B). which measures whether objects that are not edited up to turn i remain unchanged, and (ii) background consistency (Fig. 7]. which assesses whether the background remains stable when it not explicitly modified. When calculating consistency, the grounding box is extracted from the raw input image and applied to all edited images. We therefore choose to report DINOY3 over L1 distance for consistency computation because even small shifts in object location can lead to large variations in pixel-wise Lı loss, even unchanged objects are well preserved. By relying on DINO features, we ensure that consistency is measured semantically, capturing attributes such as object identity, attributes, and texture, etc. Nevertheless, the consistency scores from DINOv3 remain highly correlated with those computed using wixel-wise I. loss (See results in the Appendix. N). Based the results, the most consistent
8

---

Published as a conference paper at ICLR 2026
Harginal
Base image
Qwen-lmage-Ede
Figure 6: Marginal task success rate grouped by task types for Nano Banana.
Figure 7: Illustration of background consistency. Instruc- tion: "Remove beige brick house." The grounding box is the union of all object regions from the raw and edited images.
editing model is FLUX.I-Kontext-eee followed by Nano Banana and M.UXXI-Kontext-men In contrast, GPT-Image-1 ranks near the bottom, showing notably poor consistency across turns. Representative qualitative examples are shown in Figure and Figure]
STOP
STOP
STOP
STOP
(a) Base Image
(b) GPT-mmage-1 (95.19)
(c) Nano Banana (98.05)
(d) Qwen-Image- (94.96)
Figure 8: Illustration of object consistency. Instruction: "Remove brick beige house." " The ground- ing box, extracted from the raw input image, highlights the localized region used to compute unchanged-object consistency. The corresponding consistency score is shown brackets.
3.3
VISUAL QUALITY
Besides VdiVaI-VQ, we also report the absolute change in visual quality (VQ) relative to the base smaller indicates stronger style fidelity to the base image, whereas larger A suggests greater bcautification stylistic drift. As summarized in Table 4 GPT-Image-1 achieves the highest aesthetic scores across turns and the largest A, indicating substantial stylistic shift (Fig. 3). In contrast, GPT-Image-1.55 places greater emphasis on consistency, reducing from 2.18 8 to 0.15. For preserving the base image's appearance (small A), Gemini 2.0 Flash exhibits the least drift, with Nano Banana also performing strongly.
image: EdiVallVQAS = [EdiVal- - Edi Val-Val-
We provide a low-level exposure statistics analysis in Appendix K
3.4 MULTI-TURN EDITING VS. COMPLEX EDITING
We compare two strategies for composing multiple edits. In multi-turn editing, instructions are executed sequentially- apply instruction 1, then apply instruction 2 to the result, and so on. In complex editing, we concatenate C' instructions into a single prompt and perform one edit ("complex level" C. with C E (1,2,3}) 3)). Empirically (Fig. 9.. when model does not suffer from exposure bias, multi-turn editing tends to yield higher success rates, consistent with a step-by-ste "chain of in reasoning). For instance, Nano Banana benefits from the
edits" (analogous multi-turn formulation. Conversely, when exposure bias is pronounced, compressing instructions into a single, complex prompt can perform better, Owen-Imaae-Fadd Edit in Fig.

---

Published as a conference paper at ICLR 2026
Table 4: VdiVal-VQ and EdiVal-VOA results across turns. dark red denotes the best value in the column; lighter red denotes the second-best. For HPS, higher values are stronger aesthetics. For A, smaller values are stronger fidelity preservation.
Nutt--trm vs. Single- Shot Comples 6diting
Technique
Model
Edival VQ
EdiVal YQS
T1
12
T3
T1
T2
T3
Seodreass 4.0
5.34
5:15
5.15
0.76
0.77
0.77
Uninown
Girl-Image-155
4.16
4.00
0:30
4.35
2.18
Naru Rasunn
6.51
Nazo
1000
2LUX
5.21
5.36
0.65
0.95
FLUX Kinnica-traes
5.82
5.00
5.04
0.74
0.69
0.66
Flow Manciing
Owen-Innge Edit
5386
5:72
5.15
1.47
1.34
0.77
Suep Edi
HLUX 1- --Koreest- dav
5.12
4.06
3.54
5.00
2.76
5.04
0.33
0.73
0.09
1.04
0.65
1.63
Figure 9: Turn-3 instruction follow-
4A1
4.00
9.50
0023
0.31
0.89
ing: Multi-turn vs, single-shot complex prompts.
Anytiotr
3,06
2.90
1.5
0.71
1.38
2.44
Difisaioo
BaskcBrush
Uli Eddi
2336
IP2P
1.44
3.5
PARETO FRONT
After constructing the leaderboard using EdiVal-O, we further analyze the trade-offs between dif- ferent evaluation dimensions. To ensure that model "games" the benchmark by excelling in only one dimension, we plot the Pareto boundary at Turn for all pairwise combinations of our three evaluated dimensions: EdiVal-IF, EdiVal-CC, and Edi (see Figure T0P. Additional Pareto
plots for Turn and Turn are provided in Figures and 12
Pareto Plots for T3
MPele
edivil.C
ortonagee
Unikken Seeak
TLJX
Banera Fax
Fororo
Medictrnan
Emagetae
urranage
Uttaddtt
Geniial
Inago115
D Corann-der
Magictraa
Magi-Sn
A Aestot
Aryfdi
68
Figure 10: Pareto front plot for Turn 3 editing across EdiVal-IF, EdiVal-CC, and EdiVal-VQ.
3.6 TOOL SWAP ANALYSIS AND PROMPT COMPRESSION STRATEGIES
We evaluate the robustness of our evaluation stack by systematically swapping key components, and E.2) and the image feature extractor (Appendix E.4). Our analysis indicates that these modifications have only a negligible impact the final evaluation outcomes. Additionally, we conduct ablation study on prompt compression by testing three variants: random shuffling, sequential connectors, and explicit unchangnhan constraints. The results demonstrate that these structural variations yield only mild effect on the overall complex editing success rate, suggesting the model is relatively insensitive to the specific formatting of compressed instructions. More details are in Appendix D
including the VLM (Appendix E. the detector and its threshold (Appendices[E.3)
4 CONCLUSION
We introduced Edi Val-gg-,,eet, automated, and interpretable framework for evaluating instruction- based image editing. By leveraging symbolic object decomposition, structured instruction genera- tion, and a hybrid evaluation pipeline integrating both specialist tools and vision-language reasoning models, EdiVal-Agent enables fine- aeaggraaidd object-centric assessment of modern multi-turn editing systems. Limitations and discussions are deferred Appendix B
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Our work focuses on developing reliable and interpretable evaluation methods for instruction- based image editing. While such technology holds promise for creative design, accessibility, and efficient content creation, it may also be misused for harmful purposes such as generating misleading, de- ceptive, or inappropriate content. We emphasize that our benchmark and evaluation framework are intended solely for advancing research in safe and trustworthy generative AI. To mitigate risks, build on publicly available datasets, apply safety filters to generated images, and encourage respon- sible aligned with ethical standards and community guidelines.
REPRODUCIBILITY STATEMENT
We provide complete prompting templates and pseudo-code in the Appendix, along with imple- mentation details and API links. Comprehensive results, datasets, and evaluation metrics are also documented. To ensure full reproducibilityy we will release all code, data, and model checkpoints upon acceptance of this manuscript.
ACKNOWLEDGMENTS
Ying Nian Wu was supported in part by NSF DMMS-4415266 DARPA W912CG25CA001 and search gifts from Amazon and Qualcomm. Tianyu Chen and Mingyuan Zhou acknowledge the support of NSF-IIS 2212418 and NIH-R37 CA271186. The authors thank Wenyi Yin from Stan- ford and Binwei Yao from UW-Madison for their constructive and insightful feedback. The authors gratefully acknowledge the support of Lambda, Inc. and Microsoft for providing compute resources for this project.
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.1390 13923, 2025.
Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint a4Xiv:2404. 18930, 2024.
Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pix: editing instructions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18392-18402, 2023.
Learning to follow image
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Yingshan Chang, Yasi Zhang, Zhiyuan Fang. Ying Nian Wu, Yonatan Bisk, and Feng Gao. Skews in the phenomenon space hinder generalization text-to-image generation. In Aleš Leonardis, Elisa Ricci. Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (eds.). Computer Vision ECCV 2024, pp. 422-439, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031-73021-
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14455-14465, 2024.
An-Chich Cheng, Hongxu Yin. Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision-language models. Advances in Neural Information Processing Systems, 37:135062-135093 2024.

---

Published as a conference paper at ICLR 2026
Google Deepmind. Gemini 2.5:
Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. URL
http///axvvv... www.vv.or///... /bs/
25261
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, et al. Emerging properties in unified multimodal pretraining. arXiv 2025.
preprint arXiv:2505.
Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen-
Or. Stylegan-nadat Clip-guided domain adaptation of image generators. ACM Transactions on Graphics (TOG), 41(4):1-13, 2022.
Yu Gao, Lixue Gong, Qiushan Guo, Xiaoxia Hou, Zhichao Lai, Fanshi Li, Liang Li, Xiaochen Lian, Chao Liao, Liyang Liu, et al. Seedream 3.0 technical report. arXiv preprint arXiv:2504. 1346, 2025.
Google age generation.
Gemini2.
Experiment with gemini 2.0 flash
native im-
goog googleblog. cwm/en////////o
experi ment monnt-wttt wi th weri--emiiii-ie flash-aatt-tta -a ima.e-mageeeee generaatonn
2025.
Accessed: 2025-06-22.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or Prompt-to-prompt image editing with cross-attention control. In ICLR, 2023.
Mude Hui, Siwei Yang, Bingchen Zhao, Yichun Shi, Heng Wang, Peng Wang, Yuyin Zhou, and Cihang Xie. Hq-edit: A high-quality dataset for instruction-based image editing, arXiv preprint MrXiv:2404.0999)) 2024.
Max Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen. Viescore: Towards explainable metrics for conditional image synthesis evaluation. arXiv preprint arXiv:2312.14867 2023.
Black Forest Labs, Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, Zion English, Patrick Esser, Sumith Kulal, Kyle Lacey, Yam Levi, Cheng Li, Dominik Lorenz, Jonas Müller, Dustin Podell, Robin Rombach, Harry Saini, Axel Sauer, and Luke Smith. Flux. kontext: Flow matching for in-context image generation and editing latent space, 2025. URL http::ttt//aaatet
org/ abs/ aww.aaas./55 15742
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre- training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10965-10975, 2022.
Yuhong Li, Xiaofan Zhang, and Deming Chen. Csmet: Dilated convolutional neural networks for understanding the highly congested scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1091-1100, 2018.
Youwei Liang, Junfeng He, Gang Li, Peizhao Li, Arseniy Klimovskiy, Nicholas Carolan, Jiao Sun, Jordi Pont-Tuset, Sarah Young, Feng Yang. Junjie Ke, Krishnamurthy Dj Dvijotham, Katie Collins, Yiwen Luo, Yang Li, Kai J Kohlhoff, Deepak Ramachandran, and Vidhya Navalpakkam. Rich human feedback for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Yaron Lipman, Ricky TQ Chen, Heli Ben- Gam-H Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:22 2022.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European conference on computer vision, pp. 28-55. Springer, 2024a.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, pp. 38-55. Springer, 2024b.
12

---

Published as a conference paper at ICLR 2026
Shiyu Liu, Yucheng Han, Peng Xing, Fukun Yin, Rui Wang, Wei Cheng, Jiaqi Liao, Yingming Wang, Honghao Fu, Chunrui Han, et al. Steplx-edit: A practical framework for general image editing. arXiv preprint arXiv:2504. 17761, 2025.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. arXiv preprint (.Xiv:2209.03000 2022.
Yiwei Ma, Jiayi Ji, Ke Ye, Weihuang Lin, Zhibin Wang, Yonghan Zheng, Qiang Zhou, Xiaoshuai Sun, and Rongrong Ji. 12ebench: A comprehensive benchmark for instruction-based image edit- ing. Advances in Neural Information Processing Systems, 17: 1494-41516, 2024.
Yuhang Ma, Xiaoshi Wu, Keqiang Sun, and Hongsheng Li. Hpsv3; Towards wide-spectrum human preference score, 2025. J /208 03789
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Amab, Mostafa Dehghani, Zhuoran Shen, et al. Sim- ple vncebloccyylaryy object detection. European conference on computer vision, pp. 728-755. Springer, 2022.
Mang Ning, Mingxiao Li, Jianlin Su, Albert Ali Salah, and Itir Onal Ertugrul. Elucidating the exposure bias in diffusion models. arXiv preprint 15322,,,00.,,2,,, 2023.
OpenAL Introducing 4o image generation.
https:/ https://openai.. introducinq-4e0 -aaao----mm-e-eeeetaiitooooo 2025. Accessed: 2025-06-22.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe
Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024. URL
/tttps://openreee /ttt:s/s..ppaaeeeeeeee.sr./sssra
net pooouummddd11122RR55
Muhammad Fetrat Qharabagh, Mohammadreza Ghofrani, and Kimon Fountoulakis. Lvlm- count: Enhancing the counting ability of large vision-language models. arXiv preprint arXiv:24/2.00686, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PmLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Tianhe Ren, Qing Jiang, Shilong Liu, Zhaoyang Zeng, Wenlong Liu, Han Gao, Hongjie Huang, Zhengyu Ma, Xiaoke Jiang, Yihao Chen, Yuda Xiong, Hao Zhang, Feng Li, Peijun Tang, Kent Yu, and Lei Zhang. Grounding dino 1.5: Advance the "edge" of open-set object detection, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10694 2022.
Florian Schmidt. Generalization in generation: closer look at exposure bias. arXiv preprint
a7X1v:1910.000292, 2019.
Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks. Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8871- 8879, 2024.
Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Mare Szafraniec, Scungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint Xiv:22500 10104, 2025.
An Vo, Khai-Nguyen Nguyen, Mohammad Reza Taesiri, Vy Tuong Dang, Anh Totti Nguyen, and Daeyoung Kim. Vision language models are biased. arXiv preprint arXiv:2505.239. 2025.
13

---

Published as a conference paper at ICLR 2026
Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, Yuxiang Chen, Zecheng Tang, Zekai Zhang, Zhengyi Wang, An Yang, Bowen Yu, Chen Cheng, Dayiheng Liu, Deqing Li, Hang Zhang. Hao Meng. Hu Wei, Jingyuan Ni, Kai Chen, Kuan Cao, Liang Peng, Lin Qu, Minggang Wu, Peng Wang, Shuting Yu, Tingkun Wen, Wensen Feng, Xiaoxiao Xu, Yi Wang, Yichang Zhang. Yonggiang Zhu, Yujia Wu, Yuxuan Cai, and Zenan Liu. Qwen-image technical report, 2025a. BRLhhtts::/// /ttpss///aavvv.oo axxiv.org/abs.. 2508. 02324
Chenyuan Wu, Pengfei Zheng, Ruiran Yan, Shitao Xiao, Xin Luo, Yueze Wang. Wanli Li, Xiyan Jiang. Yexin Liu, Junjie Zhou, Ze Liu, Ziyi Xia, Chaofan Li, Haoge Deng. Jiahao Wang. Kun Luo, Bo Zhang, Defu Lian, Xinlong Wang, Zhongyuan Wang, Tiejun Huang, and Zheng Liu. Omnigen2: Exploration to advanced multimodal generation. arXiv preprint arXiv:2506. 18871, 2025b.
Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Chaofan Li, Shuting Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation. In Proceed- ings of the Computer Vision and Pattern Recognition Conference, pp. 19294-13308 2025.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li. Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: learning and evaluating human preferences for text-to-image generation. In Proceedings of the 37th International Conference Neural Information Processing Systems, pp- 2023.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui,, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024.
Siwei Yang, Mude Hui, Bingchen Zhao, Yuyin Zhou, Nataniel Ruiz, and Cihang Xie. Complex- edit: Cot-like instruction generation for caaplexity-cootrolllat image editing benchmark. arXiv preprint arXiv:2504.1344 2025.
Yang Ye, Xianyi He, Zongjian Li, Bin Lin, Shenghai Yuan, Zhiyuan Yan, Bohan Hou, and Li Yuan. Imgedit: A unified image editing dataset and benchmark. arXiv preprint 25XX:::2005.022755 2025.
Qifan Yu, Wei Chow, Zhongqi Yue, Kaihang Pan, Yang Wu, Xiaoyang Wan, Juncheng Li, Siliang Tang, Hanwang Zhang, and Yueting Zhuang. Anyedit: Mastering unified high-quality image editing for any idea. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 26125-26135, 2025.
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction soctuutttoonuuieedddee image editing. Advances in Neural Information Processing Systems, 36:31428 1111442--1449,, 2023.
Yasi Zhang, Peiyu Yu, Yaxuan Zhu, Yingshan Chang, Feng Gao, Ying Nian Wu, and Oscar Leong. Flow priors for linear inverse problems via iterative corrupted trajectory matching. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:// openreview. net hww.://wwwee/.ouuuueme -ateddlHHeeeee 7USI09
Yasi Zhang, Peiyu Yu, and Ying Nian Wu. Object-conditioned energy-based attention map align- ment in text-to-image diffusion models. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Rus- sakovsky, Torsten Sattler, and Gül Varol (eds.), Computer Vision - ECCV 2024, pp. 55-71, Cham, 2025a. Springer Nature Switzerland. ISBN 978-3-031-77946-1.
Yingying Zhang, Desen Zhou, Sigin Chen, Shenghua Gao, and Yi Ma. Single-image crowd count- ing via multi-colum convolutional neural network. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 589-597, 2016.
14

---

Published as a conference paper at ICLR 2026
Zheyuan Zhang. Fengyuan Hu, Jayjun Lee, Freda Shi, Parisa Kordjamshidi, Joyce Chai, and Ziqiao Ma. Do vision- languala models represent space and how? evaluating spatial frame of reference under ambiguities. In The Thirteenth International Conference on Learning Representattons, 2025b. URL httpttttpss///peeeeeeiiieeee
net f 8
Haozhe Zhao, Xiaojian Shawn Ma, Liang Chen, Shuzheng Si, Rujie Wu, Kaikai An, Peiyu Yu, Minjia Zhang, Qing Li, and Baobao Chang. Ultraedit: Instruction-basee fine-grained image editing at scale. Advances in Neural Information Processing Systems, 37:3058-3000 2024.
15

---

Published as a conference paper at ICLR 2026
Related Work
17
Limitation and Discussion
18
Pareto Plot
18
Ablation Study on Complex Editing Compression
18
Tool Swap Analysis
19
E.1
VLM Swap
19
Grounding DINO Threshold and Large-Boo Filter Swap
20
Detector Swap
E,4 Image Feature Extractor Swap: DINOv3 to DINOv
23
Improvement towards Counting
23
More metrics for human agreement
25
Artificial Analysis Leaderboard
26
Failure Case
27
J Discussion on single-shot complex editing
28
K Discussion Visual Quality
28
VLMs failing to judge visual quality
29
M Algorithmic Details
30
M. Decomposition
30
M.2 Instruction Generation
30
M.3 Evaluation
33
M.4 Overall Score and Aggregation Details
34
M.5
Model generations
35
Additional Evaluation Results
38
Human Agreement
42
Counting
42
Diversity of Our Dataset
44
More Quality Examples
46
16

---

Published as a conference paper at ICLR 2026
A RELATED WORK
Instruction-basse editing models. InstructPix2Pix (IP2P) Brooks ct al. (2023) introduced a two-
stage recipe that converts a text-to-image diffusion model Rombach ct al. (2022); Zhang et al.
(025a) into editor: (i) synthesize paired editing data using Stable Diffusion Rombach et al.
2022) and training-free techniques such as Prompt-to-Prompl Hertz et al. (2023) (ii) fine-tune the
diffusion model on these pairs. Subsequent Systems-MaaiccBrus Zhang al. (022), UltraEdit
Zhao et al. (2024), and AnyEdit Yu et al. (225))scaal this paradigm to large, fine-grained real-
image editing. More recent work (e.g.. OmniGen Xiao al. (2025 ; Wu et al. (2025b). StepIX-Edit
Liu et al. (2025), FLUX.1 Kontext Labs et al. (2025), and Qwen- Image- Edit Wu et al. (2025a), See-
dream Gao et al. (2025) adopts task-aware architectures and increasingly leverages flow matching
Liu et al. (2022) Lipman et al. 2022 Zhang et al. (2024).
A complementary line explores autoregressive (AR) editors such as Gemini 2.0 Flash Image Gem-
(2255,, Gemini 2.5 Flash Image ("Nano Banana") Deeemminddddd (2025), and GPTT-mmaee [OpenA] (025).. These models enable in-context multi-turn editing: users iteratively refine an image within conversational interface, with the model maintaining coherent editing history. To our knowledge, we provide the first systematic comparison of in-context multi-turn AR editing versus context- multi-turn editing with non-AR models across instruction following, content consistency, and visual quality.
Editing evaluation. Early evaluations (c.g., Brooks et al. (2033) ) rely on CLIP-based similarity Radford et al. (2021), including directional variants Gal ct al. (2022). to approximate editing quality. However, CLIP emphasizes semantic alignment and is less sensitive to fine, pixel-level changes. When ground-truth edited images exist, is natural to compare model outputs against references using pixel distances (L,) and semantic similarities (DINO Caron et al. (202T). CLIP Radford et al.. 1021 Zhang et al. (2023) ; Zhao et al. ( (202)); Yu al. (205)); Sheynin al. (2024). Yet references are imperfect: the space of valid edits is inherently multimodal, while a single reference captures only one realization; moreover. many references are themselves synthesized by prior editors (e.g..
[rerz et al. (2023). SDXL Podell et al. (2024), DALLE-2 Ramesh et al. (2022 importing their biases into evaluation.
Recent work relies exclusively on VLMs as interpretable judges-es e.g., VIEScore Ku et al. 202.3)
HQ-Edit Hui et al./ (2024), and Complex- Yang et al.) (2025) by querying models such as GPT- about specific aspects of an edit. While VLMs offer holistic, language-mediated assessments, they are insufficient on their own: they are notoriously poor at spatial reasoning Zhang ett al. (2025b); Cheng et al. (2024); Chen et al. (2024); [Qharabahh et al. (2024); Chang et al. (2025) and are prone to object hallucinations in existence, category, attributes, and relations Bai et al. (202);; they have limited sensitivity to pixcl-level changes and frequently miss subtle, localized modifications Vo et al. (0025) (e.g., fine structures, small attribute shifts, etc.), which are crucial for evaluating content consistency; they are miscalibrated for artifacts and aesthetics [Liang et al. (2024); Xu et al. (202)); Ma et al. (2025), which humans are sensitive to. Our approach, Edi Agent, addresses these gaps by integrating VLM-bbsed reasoning with grounding tools, symbolic, object-centric pixel- and semantic-leve tools, and human preference models, yielding precise and interpretable evaluation of instruction-barn editing.
4o OJ
Editing tasks. We consider three settings: (i) Single-turn vs. multi-turn. Multi-turn editing Zhang et al.] (2023); Zhao et al. (2024) is more demanding than single-tur,, as the model must maintain consistency across sequential instructions. In contrast context-free multi-turn pipelines (each turn consumes the previous image and the next instruction), AR models Gemini2 (2025);
Deepmind (205)); OpenAl (2025) support in-context multi-turn editing by conditioning on the full conversational history. (ii) Complex single-shot vs. multi-turn. Following Yang et al. (2025)
a sequence of edits can be concatenated into a single complex prompt and executed in one pass; we compare this setting to genuine multi-turn editing. (iii) Other tasks. We focus on instruction- based editing, the most common regime; other scenarios (e.g.,
wwwpppppcaapttoon---.-/ Hertz et al.) (2023) ) are outside our scope. To the best of our knowledge, this paper offers the first comprehensive comparison covering single-turn, multi-tuna -urn, and complex single-shot editing within unified framework.
17

---

Published as a conference paper at ICLR 2026
B LIMITATION AND DISCUSSION
Given the object-centric evaluations conducted in this study, several limitations warrant considera- tion. First, our instruction types are limited to object-centric prompts, which may not capture the full range of creative editing requests typical in real-world scenarios. Future research should explore broader spectrum of instructions, including those involving stylistic changes or complex narrative elements. Additionally, while our work provides a reliable and comprehensive evaluation frame- work for multi-turn editing, it does not apply the evaluation results improve the editing models themselves. A straightforward extension would be to use evaluation scores for Best-of-N selection to improve inference-time performance. Future work could also explore post-training methods such as reinforcement learning, treating the evaluation scores as reward signals.
Please note that we have included performance evaluations for GPT-Image- 1.5, Nano Banana 2, and FLUX.2-max in this camera-ready version. These models were released following the initial submission to ICLR 2026. While the main text has been updated to provide a self-contained analysis of these models, please be advised that the appendix may not reflect exhaustive comparative analysis for these three newly additions due to space and timing constraints.
C PARETO PLOT
we also visualize the trade-offs among the three Val dimensions-Fdivve Val-IF, EdiVal-CC, and Redival-VQuusigg Pareto plots. These plots illustrate the performance frontier for each turn, providing more concrete view of the trade-off and allowing users to select the most suitable model according to their preference.
In addition presenting leaderboard in Table 13
We present the Pareto plots for Turn in Figure Turn 2 in Figure and Turn in Figure TO
Pareto Plots for T1
EdWaHi EdIVNs-CC (TA)
Ed Edivva/v (TA)
EdIasiCC Ediasvv (TR)
G rin 2.0l
image
HogidEinat
coneutinna
@costrom
Sendromm The
Briigg Ulrstst
Atritc
Paain
huge
B
$00
Asyad
713
17.5
953 100
Figure 11: Pareto plot for Turn 1.
Pareto Plets for T2
NRWAHFF us FREW-CC (T2)
Edilosk edi vo (T9)
MiNAACC vs EdRNAiNG (17)
Wolkfit
aT lafara
Queen Imageetteeee Ed1
FUUKL Kor
1 Eratic S Binceenn-
Sandhoar 11
Restedy
CC
Uluelt
No'o
Brotck
Cemvie 2:3
Cartel
Euit
Hagirtin Statkktck
Mag chruch
Anydot
Anyca:
Figure 12: Pareto plot for Turn 2.
D ABLATION STUDY ON COMPLEX EDITING COMPRESSION
We conduct an ablation study on how to compress three-turn instructions into a single "complex edit" prompt. In the previous experiment, we adopt the simplest concatenation strategy: (prompt
18

---

Published as a conference paper at ICLR 2026
Table 5: Complex Editing Performance for com-
Table 6: Tool swap analysis (correlations).
pression variants.
Type
Debail
Viriane
Metric
Pleeeen
Spearmon
Connector Variant
Complex(C3) (%)
VIM
Qwent-7B-V
Qwwn22 5-10-VV
Qoee2 5-918--I..
IF
0.9244
0.929
[ntzmnV1.3.88
Default Random shuffle
27.62
27.10
Theshold
p.5.
Thrshold
Sequential connector
Pilter
niter
tiker
0.930
Keep-unchanged
25.87
Desectoc
Grounding DENO
OWL VIT
IF
03155
0.7929
Fonise
DINO>3
OC
0.9877
1,000
TI}. {prompt T2}. (prompt T3J. We further evaluate three alternative variants on owen-Image- Edit: 1) Random shuffle: randomly shuffle the three per-turn prompts before concatenation. 2) Sequential connector: explicitly encode the order as: first, {prompt T1}, then, {prompt T2), last, (prompt T3}. 3) Keep-unchanged objects: append an explicit constraint: {prompt T1J. {prompt T2}. (prompt T3}. Keep (unchanged objects} unchanged. Table 5 reports the resulting complex editing success rate(%) at 3. The results show these compression variants have only a very mild effect on performance.
E TOOL SWAP ANALYSIS
We analyze the effect of swapping individual components in our evaluation stack, including the VLM, detector, detector threshold, and image feature extractor. We find that modifying the VLM (Appendix [..T). adjusting the detector threshold (Appendix [E2). changing the detector (Ap- pendix E.3) changing the image feature extractor (Appendix (F.4) has only minor impact on the final evaluation outcomes. The summary statistics are shown in Tab. 6 which demonstrates that for each configuration replacement, our evaluation results remain highly correlated with those obtained under the default stack. As for EdiVal- we found that HPSv3 is the only human preference model trained on images generated after SD3.5. This suggests that other preference models. such as HPSv2, cannot provide reliable assessments of recent, more advanced generations, as they are only sensitive to earlier-stage generations that are significantly lower quality.
However, we note that replacing Grounding DINO with OWL- Minderer et al. (2022) (Ap- pendix E.3) fails to filter out failure cases originally caused by Grounding DINO and also reduces agreement with human annotations. This highlights that detector accuracy should be prioritized when considering detector substitutions.
We further examine component-specific tasks, such counting, where we compare the perfor- mance of density-map estimation methods (Appendix [F). Finally, we include preliminary experi- ments style transfer (Appendix ??), which indicate that existing VLMs still struggle to reliably judge whether a style transfer succeeds.
VLM swap. We replace the default VLM in stack (Qwen2-7B-VD) with several alternatives: Qwen2.5-7B-VL. Qwen2.5-33B-V.. -L, and Intern VL3-8B. For each configuration, we recompute the per-turm instruction-followinn success rates for four representative editors: GPT-Image-I, Seedream 4.0, Nano Banana, and FLUX. 1-KoooKott-xaax
Across all VLM choices, the ranking of these editors remain the same. While the absolute numbers shift slightly (e.g., the Qwen2.5- and than the default Qwen2-7B- BL stack). When we compare per-turn success rates between the default Qwen2-7B 2-2-7B-VL stack and each swapped VLM, both Pearson and Spearman correlations remain high over (model, turn) pairs. Concretely, we show results Tab 7
stacks tend to assign somewhat lower scores
E. VLM SWAP
To assess how our evaluation results depend the choice of VLM, we swap the default VLM in our stack (Qwen2.5-7B-) with several alternatives: Qwen2-7B-V VL, Qwen2.5-32B-V and Intern IntemVL33 VL3-8B. For each configuration, recompute the per-turn cnttucttionfollowwii success rates for four representative editors: GPT-Image-1, Seedream 4.0, Nano Banana, and 1-UX.1- Kontext-max.
19

---

Published as a conference paper at ICLR 2026
Across all VLM choices, the relative ranking of these editors remains unchanged. While the ab- solute scores shift slightly (e.g.. the Qwen2.5- and AatervL-bbssee stacks tend to assign somewhat lower scores than the Qwen2-7B-VL stack), the overall ordering is stable. When compare per- turn/marginal success rates between the default Qwen2.5-7B- both Pearson and Spearman correlations remain high over all (model, turn) pairs.
stack and each swapped VLM,
Concretely, Table 7 Trpports correlations between the per-turn success rates under our default VLM (Qwen2.5-7B-VL) and those obtained with each alternative VLM. All Pearson correlations exceed 0.95 and all Spearman correlations exceed 0.92, indicating that changing the VLM has only a modest effect on the absolute scores and largely preserves the relative ordering of the four models. In particular, on the VdlVal leaderboard we consistently observe
Seedream 4.0 > GPT--maae-1 Nano Banana
FLUX.11 Kooteext-kax
Table 7: Correlations between per-turn instructionnfollooiing success rates under the default VLM (Qwen2.5-7B-VV and three strong vision-language baselines. All correlations computed over 12 (model, turn) points.
Baseline Model
Pearson
Spearman
Qwen2-7B-V. VL
0.9544
0,9298
Qwen2.5-322 --322----B-
0.9790
0,9544
InternVL33-8
0.9660
0.9228
E.2 GROUNDING DINO THRESHOLD AND LARGE-BOX FILTER SWAP
In this section, we ablate the effect of changing Grounding DINO's detection threshold and dis- abling the large-box filter. By default, we use a threshold of 0.30 for subject remove, 0.40 for pos
and 0.35 for all other tasks. In addition, during detection we discard any box whose normalized height and width are both larger than 0.98, in order to filter out degenerate predictions that cover almost the entire image.
To investigate sensitivity to these choices, we re-run the evaluation under three alternative settings: (1) global threshold of 0.30, (2) global threshold of 0.40, and (3) the default thresholds but with the large-box filter disabled. The results are reported in Table g Across all settings, the instruction- following metrics (image success, task success, and overall scores) change only slightly, and the ranking of models remains unchanged. This suggests that our conclusions are robust to reasonable variations in the detector threshold and the large-box filtering heuristic.
Table 8: Per-turn image success, task success, and overall scores under different evaluation settings (ranked).
Config
Model
Ing. Success
Task Success T2
Overall
Rank
Tl
Default
Seedream 4.0
75.930
55.580
41.590
75.930
75.580
76.110
83.811
69.948
59.757
Nano Banana
70.700
35.350
70.700
72.590
68.240
81.483
67.703
56.242
GPT-Image-1
73.120
54.890
37.970
73.120
74.440
72:240
76.959
65. 340
53.542
FLUX. 1-Kontxxx-maxx
69.490
46.890
31.830
69.490
69.110
70.430
80.791
65.286
53,045
Threshold 0.3
Seedream 4.0
73.450
53.270
38,940
73.450
74,870
74,870
82.431
68.479
57.8 57.822
Nano Banana
69.940
-0:340
33.460
69. (9.940
71.640
67.490
81.044
66.815
54.717
2
GPT--magoo1]
71.430
52,820
37.030
74.060
76.065
FLUX. 1- Kontext-max
68.170
45,570
30.510
68.170
68.360
68.95 (8,930
80.020
64. 361
51,933
Threshold 0.4
Seedream 4.0
73,980
52.570
38,940
73,980
73,450
74.510
82.
68. 027
82.
Nano Banans.
68.810
47.260
31,950
68.810
69.570
64.460
80.386
65. 65.392
53.469
GPT--mage-1
70.110
50.190
35.150
70
72.560
71.240
75. 75.359
62,480
51.515
FLUX.1 Kontext-max
66.290
28.630
67.230
63.019
No Large-Box Filter
Seedream 4.0
75.930
53.220
41.240
75.930
75.220
76.110
83.811
69.721
59,505
Nano Banana
70.700
35.350
72.590
68.240
81.483
67.703
56.242
GPT Image-1
72.930
54,890
38.350
72.930
74.4-00
73.120
76.859
65.340
53,809
FLUX.1 1- Kontext-max
69.490
46.890
31.830
69,490
69.110
70.430
80.791
65.286
53,045
20

---

Published as a conference paper at ICLR 2026
E.3 DETECTOR SWAP
We also perform an ablation study by swapping the open-vocabulary detector in our pipeline from Grounding DINO to alternative detectors, such OWL-ViT Minderer al. (2022) and GLIP Li et al. (2022). For this swap, detector accuracy is the primary factor we must prioritize, since a more accurate detector directly translates to higher agreement with human annotations.
Table 9 summarizes the performance of several popular open-vocabulary detectors on standard de- lection benchmarks. In the open-set setting ODinW (object detection in the wild), which most closely resembles our scenario, Grounding DINO outperforms GLIP. It also achieves the best AP on COCO, a widely used benchmark with 80 common objects. LVIS challenging benchmark with more than 1 categories spanning rare, common, and frequent objects; on LVIS OWL-ViT attains strong performance, but Grounding DINO still provides better trade-off for our open-world editing setting, particularly when ODinW performance is considered.
We do not adopt other available detectors such as Grounding DINO 1.5, because they are closed- source and thus unsuitable for community benchmark where users may need to run evaluation many times or adapt the pipeline to their own models. Moreover, our current configuration already achieves better alignment with human judgment (81.3%) than strong zero-shot VLM baselines, so switching to a closed-source source detector would reduce reproducibility without a clear benefit.
When we swap Grounding DINO for OWL-ViT in EdiVal pipeline, the relative ranking of models (based on Img.Success at Turn 3) remains unchanged, but the absolute scores drop. Never- theless, the per-turn/marginal success rates still exhibit high correlation with those under Grounding DINO (Pearson 0.82, Spearman 0.79), OWL-ViT frequently fails to detect objects that are clearly present in the image, which leads to many spurious "reject" decisions and thus lower Success and Img. across all models. This bias is largely consistent across models, so EdiVal- pre- serves the same ordering, but the absolute values are shifted downward. A qualitative example is shown in Figure 14. where OWL fails to detect objects that are clearly present in the image.
We therefore do not recommend replacing Grounding DINO with OWL-VVT, GLIP, or other weaker open- voce-vocabulary detectors in our framework. Grounding DINO remains widely adopted state-of- the-art open-vocabulan detector, and weaker detectors not only reduce absolute performance scores but also harm agreement with human judgments. For example, when swapping Grounding DINO for OWL-ViT, the human agreement of EdiVal- drops from 81.30% to 53.67% In general, upgrading to stronger open- vocabulary detector should preserve the relative Edi EdiVal-IF ranking while improv- ing absolute scores and human agreement, whereas downgrading to significantly weaker detectors has the opposite effect and is therefore undesirable.
COCO
LVIgetaal
LVIS
ODinW35
ODinW13
Method
Backbone
Pre-trainig data
AP
APai
AP,
AP.
AP,
AP.i
AP,
AP.
AP;
AP..
APp.
OWL-VVT GLIP
VIT-I
0365, OID, VG. LIT
42.2
34.6
31.2
Swin-L
FourODs GoldG, Cap24M
49.8
37,3
28.2
31,3
41.5
26.9
17.1
23.3
35.4
52.1
Grounding DINO
Swin-L
0365, OID, GolaG
$2.5
26.1
56.9
Table 9: Performance of popular detectors including OWL-ViT, GLIP, and Grounding DINO (Swin- L). Numbers are copied from Table 1 of Grounding DINO 1.5 Ren et al. (2024). In the open-set setting WDinW, which is most similar to our scenario, Grounding DINO outperforms GLIP. It also achieves the best AP COCO. LVIS is large-scale benchmark with over k categories spanning rare, common, and frequent objects.
Can detector swapping avoid failure cases? No. We also examine the same failure cases of Grounding DINO under OWL-ViT and GLIP. In these examples, OWL-ViT behaves differently from Grounding DINO but does not fix the underlying issue: regardless of whether the object truly exists in the image, OWL-ViT often detects nothing at all. GLIP, on the other hand, exhibits similar false-positive behavior to Grounding DINO, but with denser and less precise bounding boxes. Thus, switching to OWL-ViT or GLIP does not eliminate such failure cases; it merely changes them into systematic missed detections or more cluttered false positives.
21

---

Published as a conference paper at ICLR 2026
300
(a) Base image (original)
(b) Seedream 4.0 (edited)
(c) Detected: stone gray railing
(d) Detected: wooden fence
Figure 13: Failure due to detector false positives. Although the edit visually replaces the railing with a wooden fence, Grounding DINO fires on both "stone gray railing" and "wooden fence" in overlapping regions, causing an incorrect failure in our cnstruction-follwwiie metric.
(a) Base image (original)
(b) Scedream 4.0 (edited)
(c) Nothing detected: "stone gray railing"
(d) Nothing detected: "wooden fence"
Figure 14: OWL-VIT fails to detect the queried objects regardless of whether they are actually present in the image, and therefore does not resolve this failure case.
(a) Base image (original)
(b) Seedream 4.0 (edited)
(c) GLIP prediction: "stone gray railing"
(d) GLIP prediction: "wooden fence"
Figure 15: GLIP produces substantially more false positives than Grounding DINO, hallucinating objects such as "stone gray railing" and misclassifying the tower region as a "wooden fence" or "stone gray railing".
22

---

Published as a conference paper at ICLR 2026
E.4 IMAGE FEATURE EXTRACTOR SWAP: D TO DINOV2
We further ablate the image feature extractor by swapping the default backbone (DINOv3) with DINOv2. As with our other component swaps (VLM, detector, and threshold), we observe the same qualitative behavior: the relative ranking of models remains unchanged, while the absolute consistency scores shift.
Concretely, DINOv2 systematically produces lower EdiVal-CC values than DINOv3, but the or- dering across models identical. When compare per-turn consistency scores across the two backbones over all (model, turn) pairs, we obtain Pearson correlation 0.9987 and Spearman corre- lation 1. 1.0000, indicating that DINOv2 and DINOv3 induce essentially the same ranking and very similar relative differences between models, despite the shift in absolute scale.
Table 10: DINO-v2 feature backbone results with EdiVal-CC Rank.
Model
Turn 1
Tum 2
Turn 3
EdiVal-CC Rank
Seedreum 4.0
89,455
84.185
81.120
Nano Banana
90.820
86,900
85.070
GPT-Image-1
76.360
72.555
69.845
FLUX. I-Kootxtt--xx
91.530
87,535
84.460
Table 11: DINO-v3 feature backbone results with EdiVal-CC Rank.
Model
Turn
Turn
Turn 3
EdiVal-CC Rank
Seedream
92.51
88.03
85.86
Nano Banana
93.91
90.48
89.48
GPT--magee1
81,00
77.78
75,50
FLUX. I- Konlext-max
93,93
90.90
88.40
Parforeaicc Compations Defhallt (Dinu vij rmaga Pestare consistanc es Chine taage mesture Canaiserr
Figure 16: Per-turn Vdi-Cl-CC scores under DINOv3 (x-axis) vs. DINOv2 (y-axis) for the four rep- resentative models. Each point is a (model, turn) pair; Pearson correlation is 0.9987 and Spearman correlation is 1.0000, confirming that the feature-extractor swap only shifts the absolute scale while preserving the ranking.
F IMPROVEMENT TOWARDS COUNTING
When we generate prompts, the target count never higher than 10. Under this regime, Grounding DINO is sufficiently reliable for counting the relevant objects.
Potentially there are two other families of methods do counting: racking-based counting and density-map estimation. Tracking-base methods are typically designed for videos, where they exploit multiple frames to track objects over time. our case, we only use single edited images to do counting, such tracking-based approaches are not applicable.
Density-map estimation methods (e.g., CSRNet Zhang et al. (2016); Li et al. (2018) are usually developed for crowd counting of a fixed target category (such as humans) on specific datasets, rather than open- open-vocabulary object counting. When the objects to be counted are not aligned with the training data distribution, these methods can fail dramatically. To make this concrete, apply CSRNet to one of our examples: CSRNet outputs an estimated count of 44.86, whereas Grounding DINO correctly predicts 3 bounding boxes for the target objects.
23

---

Published as a conference paper at ICLR 2026
500
(a) Count by Grounding (b) Count by CSRNet DINO.
(density map).
Figure 17: Illustration counting with an open-vocabulary detector (Grounding DINO) vs. density-map caceyy-mpppaattaa map-based crowd counter (CSRNet). CSRNet severely overestimates the count in this open- vocabulary setting.
24

---

Published as a conference paper at ICLR 2026
G MORE METRICS FOR HUMAN AGREEMENT
The most straightforward metric is accuracy. Here, we provide more metrics measuring human agreement: Pearson Linear Correlation Coefficient (PLCC), Cohen's Kappa Coefficient (Kappa) and Fl scores as shown in Tab. [2 However, we note that for 0/1 predictions, correlation metrics like PLCC and Kappa may be considered not suitable for measuring the agreement with human annotators.
Table 12: Task-specific and overall performance comparison across models. Metrics reported: Pearson Linear Correlation Coefficient (PLCC), Cohen's Kappa, and Fl. Best per column high- lighted in bold.
Task Type
Model
PL.CC
Kappa
FI
subject add
CLIP dir
0.2110
0.1764
0.6914
Qwen2.5-VL
0.5331
0.5264
0.7893
EdiVal-IF
0,5365
0.5364
0.7786
background change
CL.IP dir
-0.0329
-0.0076
0.8745
Qwen2.5-V6
0.5686
0.8889
EdiVal-IF
0.5244
0.5157
0.8763
suhject_remove
CLIP _dir
-0.011
-0,0002
0.6592
Qwen2.5-VE
0.1891
0.1758
0.4896
EdiVal-IF
0.5473
0,5409
0.7837
count_change
CLIP_dir
0.0456
0.0142
0.0782
Qwen2.5-VT.
0.1998
0.1748
0.2162
EdiVal-IF
0,3431
0.3274
0.3571
material.alter
CLIP
0.2086
0,2038
0.4561
Qwen2.5-VL.
0.8658
0.8616
0.8971
Val--al
0.4778
0.4624
0.6364
color_alter
CLIP _dir
0.1841
0.0874
0.8542
Qwen2.5-Ve
0.8409
0.8407
0.9573
EdiVal-IF
0.7820
0.7744
0.9338
position_change
CLIP-dir
0.0996
0.0430
0,3285
Qwen2.5- VL
-0.0381
-0,0374
0,1798
EdiVal-IF
0.3907
0.3271
0.5000
change
CLIP dir
0.6178
0.6173
0.8063
Qwen2.5-V
0.7161
0.6947
0.8651
EdiVal-IF
0.7438
0.7347
0.8571
subject replace
CL.IP dir
0.0420
0.0121
0.8219
Qwen2.5-VE
0.6028
0.5994
0.8699
EdiVal-IF
0.5533
0.5429
0,8410
Overall
CLIP dir
0.3186
0.2568
0.6858
Qwen2.5-Ve 5L
0.6162
0.6161
0.7922
Edi Val-Fl
0.6278
0.6273
0.8030
25

---

Published as a conference paper at ICLR 2026
H ARTIFICIAL ANALYSIS LEADERBOARD
We report the leaderboard from the Artificial Analysis website as of September 12, 2025 (Fig. 18] To ensure fair comparison, we align on the intersection of models evaluated by both platforms and exclude Owen-Imaae--Eii Among the overlapping "ystems-Seedrenn 4.0, Nano Banana (Gem- ini 2.5 Flash), GPa-Image-1 (GPT-40), ordering of human votes on Artificial Analysis matches our EdiVal rankings exactly (Table 13) supporting the accuracy of our methodology.
1-xXte-ontext-oioe and Gemini 2.0 Flash tte relative
Table Model rankings on the overlapping set. Relative ranks from Artificial Analysis (human votes) vs. EdiVal (ours) as of Sep 12, 2025.
Model
Artificial Analysis (Rank)
EdiVal (Rank)
Seedream 4.0
1
Nano Banana (Gemini 2.5 Flash)
2
2
GPT--maae-1 (GPT-4o)
3
3
FLUX. --nntexxt-mxx
4
4
Gemini 2.0 Flash
5
5
Appearances
Google
1301
-5155
Moack Poreet
PLUE Kardoee Lowd
1089
1299
Spenns
1080
-244
5:50
Alrobto
Owen Image
1007
-2/22
6355
15/19
A72
Google
Gan 2. Fissh
1000
+240
Depk FoMer
FUXU connectoe
955
-24-
5.59
Figure 18: Artificial Analysis leaderboard (Sep 12, 2025). Screenshot of the public leaderboard used for comparison in Table 13
26

---

Published as a conference paper at ICLR 2026
I FAILURE CASE
We discuss a representative failure mode of our evaluation. The most severe errors arise from false positives in Grounding-DINO, despite its strong open-vocabulary performance. Consider the prompt: "Replace [stone gray railing] with [wooden fence]." As shown in Fig. 19| Seedream 4.0 produces an edit that visually correct. Our rule for the edited image. the source object (stone gray railing) is no longer detected while the target object (wooden fence) is detected. However, Grounding- DINO occasionally reports both the source and target objects in the same region with high confidence, incorrectly suggesting that the source object remains and thereby degrading the measured instruction-following accuracy. Improving the relia- bility of open- vocabulary oetection-particulain reducing false positives positives-would directly improve the fidelity our evaluation.
declares success if, on
(c) Detected:
gray (d) Detected:
wooden
(a) Base image
(b) Seedream (edited) railing
fence
Figure 19: Failure due to detector false positives. Although the edit visually replaces the railing with a wooden fence, Grounding-DDie DINO fires on both "stone gray railing" and "wooden fence" in overlapping regions, causing incorrect failure in our followii metric.
27

---

Published as a conference paper at ICLR 2026
J DISCUSSION ON SINGLE-SHOT COMPLEX EDITING
Figure 20 shows that marginal success for the final instruction remains largely stable as complex prompt length increases. Together with the multi-turn drops seen in Figure this pattern supports an exposure-bias explanation: performance degradation primarily stems from error accumulation across sequential edits rather than intrinsic inability to handle multiple instructions in single prompt.
K DISCUSSION ON VISUAL QUALITY
Beyond instruction following and content consistency, the perceptual quality of the edited image is a key dimension. We therefore report (i) a learned aesthetic score and (ii) several low-level image statistics that can surface systematic artifacts and drift in multi-turn editing pipelines.
Marginal Suroece Rate Armoxt Complee
Table 14: Turn-3 instruction following: Multi-turn vs. single-shot complex prompts, grouped by tech- nique. Bold indicates which setting is higher for each model.
Geen Ran
Technique
Complex
Nano Barara
35.35
28.14
In-Context
GPT-Image-1
38.35
28.78
Gemini Flash
Owen-Image Edit
22.55
27.62
Flow Matching
E-dit
17,83
15.73
Al
FLUX.1 Kontext-dov
OmniGen
10.66
11.01
AnyEdit
7.22
2800
Diffusion
UlimFdit
8.22
Figure 20: Marginal task-success rate of the last instruction as a function of complex prompt length (levels C = 2,2,33 3).
MagicBrush
4,90
4.55
IPIP
2,80
2,80
Lurrnnance 99.9%
1.20
0.75
Luminnnce 00.00 10.0% Quaent1
3.225
#:4 Image
3.99
3.800
3.8
OeTAzio
Figure 21: Per-image 99.9% luminance quantile across turns. Higher values indicate more extreme bright pixels and greater risk of over-exposure.
Low-level image statisties In addition to learned aesthetic scores, we compute several low-level image statistics that help reveal systematic, multi-turn editing artifacts. Concretely, we convert RGB pixels to luminance using the Rec. 709 luma coefficients: Y 0.2126 R 0.7152 G + 0.0722 B, and for each edited image we extract the 99.9% luminance quantile (the per-image pixel value below which 99.9% of pixels fall). The 99.9% quantile is sensitive to high-exposure pixels and therefore highlights over-exposure and bright streaks while being robust to single-pixel outliers. In Figure 21 we plot the trend of this statistic across turns.
The measured trend shows a clear pattern: Qwen-Image-Edit and several other flow-matching models (with the notable exception of FL.UX.I- exhibit a pronounced increase in the
28

---

Published as a conference paper at ICLR 2026
99.9% luminance quantile over turns, indicating progressive brightening and increased risk of over- exposure. By contrast, regeneration-style editors such as GPT-Image-1 tend to produce lower lu- minance values than the imput (reflecting darker, more conservative reconstructions), and several models remain stable across turns.
Figure 22 provides qualitative examples from Lrann-lmaee-diit The edited images exhibit elevated luminance and noticeable high-frequent bright artifacts (e.g., white streaks or "line" textures) that degrade perceptual quality, with luminance quintiles increasing substantially. Correspondingly, HPS drops from 6.19 to 19 and 3.34, suggesting that HPS is sensitive to over-exposure to some extent. In contrast, when querying VLMs about the visual quality of these images, the returned scores do not change in the first two turns and remain consistently above 50, reflecting positive evaluation under the [0, 100] scale, while the T2/T3 edited images show significant artifacts.
(a) IIPS: 4.25 VLM: 85
(b) HPS: 6.19 VLM: 85
(c) HPS: 4.19
(d) HIPS: 3.34
VLM: 85
VLM: 60
Luminance:0.7
Luminance: 0.60
Luminance: 0.97
Luminance: 1.00
Figure 22: Representative Qwen-Image-FEE Image-Ediid examples illustrating over-exposure and bright artifact
formation across turns. Although editing instructions are often satisfied, the images show elevated luminance and high-frequency bright streaks that accompany the edits (visible especially in T2/T3). Editing instructions: [Remove polyester white skirt, Change the count of tennis ball to 4, Change the color of tank top to blue]. Note that VLM gives positive score to all the images.
VLMs FAILING TO JUDGE VISUAL QUALITY
The following the zero-shot prompt for visual quality with VLMs. The example results are shown in Fig. 22
You are an expert at evaluating image visual quality and naturalness.
will show you an image.
Please analyze whether the image is visually pleasing and natural. Consider:
1. Is the image visually pleasing?
2. Is the image natural?
3. Does the image look natural and coherent?
Respond only with a score between 0 and 100, where 100 is the highest score. 100 means the image is visually pleasing and natural.
0 means the image is not visually pleasing and natural.
50 means the image is neutral.
29

---

Published as a conference paper at ICLR 2026
Algorithm Object Listing and Grounding Filter
Require: image I
Ensure: object pool with grounding metadata
1: J < LISTÖBJECTS(()
D see Sec. M.2.1
2: O
3: for all (name, attrs) J excluding key All Objects do
4:
(boxes, phrases, scores) <- GROUND(T, name)
D thresholds 0.3-0.4
5: if boxess # 9 and each box has w. h < 0.9 and area. I 0.4 then
6:
O[name] < attrs; attach grounding metadata (count, boxes, phrases, scores)
7: end if
8: end for
9: O'ilttred All Objects] JOIN(KEYS(O))". and then append
M ALGORITHMIC DETAILS
This appendix provides the algorithmic details of our pipeline: object discovery and grounding- based filtering (decomposition), instruction generation for multi-turn editing, and evaluation (in- struction following, consistency, and perceptual quality). We also list the exact prompts and imple- mentation specifics needed for reproducibility, and summarize the model--eneration configurations.
M.1 DECOMPOSITION
We first enumerate visible objects in an input image using vision-language model (VLM) prompt, then filter these objects via visual grounding.
Object listing: We GРT-4o with the prompt in Section M.2.1 The model turns a JSON with one entry per object and a terminal aggregated string key 'All Ob object'''.
Grounding filter: We use GroundingDINO SwinT-OGC Liu et al. 2024a) to retain only objects that can be visually grounded. We resize images to 512 x 512. We keep detections meeting text/box thresholds (0.35) and reject oversized boxes by checking width/height in normalized coordinates; we use max wax.box_siiz ..9 and filter large regions if area 0.4. The output augments each kept object with grounding counts, phrases, boxes, and scores, and creates 'Filtered All Ob Objects' string listing retained objects.
M.2 INSTRUCTION GENERATION
We generate multi-turn editing instructions from the grounded object pool. We support nine task types: local edits (subject color alter, subject add, text change, position change, count change) and the global edit (background change]. We set MAX TURNS=3. At each turn, we sample new task type without repetition where feasible. Feasibility is checked against the current object pool requires at least two objects). If a sampled task is infeasible, we fall
materialalter.
(e.g., position.changeeeeeeeeeooo back to subject_add. add. We maintain an available-objects pool that updated after each instruc- tion according to its semantics (adds, removes, modifies attributes). If background change curs, we mark bg.consia www.onnistentie sttsscstaalasssee for subsequent turns and restrict the pool to foreground objects for the remainder of the episode.
Prompts (Full Text) Below we reproduce the prompts used by our generators, reformatted for readability in print (content preserved).
M.2.1 OBJECT LISTING PROMPT
You will be given image. Your task is to identify and describe all clearly visible objects in the image structured JSON format.
Output rules:
30

---

Published as a conference paper at ICLR 2026
Algorithm 2 Multi-Turn Instruction Generation
Require: grounded pool Og, turns T-3 Ensure: tasks {Tr). instructions } formats {Pr), flag has bg, set all obwootooeeee.. ever
1: used <- 0; O <- On: has_bg + false
2: all_edited +- 0: all_objects_ever <- (e))(Oo)
3: fort = 1 to I' do
4: cand + {all task types) used
5: T. sample(cand);
6:
not feasible(T,, O) then Tt + subject.add end if
7:
8:
F. + format_instruutionnnf Tr. O)
> Query VLM by prompts in Section M.2.2
9:
4 f render insttrucioit Tr, TT,
>> strip brackets; add unchanged list for background
10:
used t used U {T: append Fr.
Update all object pool by adding any objects introduced instruction 4..
12:
Update available object pool by adding removing objects as specified in I,. by removing any objects affected by It-
Update crhjentedd.ohjets..o
14: if Lackground.chaam
then has_bg f- true; + filter foregrouor)))
15:
16: end if
17:
end for
18: return (Te), (Te), (F.), has bg, all objects ever
1. Fach ohject must be listed as key in the JSON, using the format: "[material] {color] (object name)". If material color unknown, omit that part. Do not include any visible text in the key. Do not use "person" as an object name; instead, describe wearable items (e.g.. "blue cotton shirt").
2. For each object, the value is dictionary with fields: "object" (type, e.g., shirt, cup). "color" (dominant color, use null if unknown). "material" (likely material, use null if unknown), "text" (visible text, null if none), "count" (number of instances), and "foreground" (boolcan).
Do include objects are too small to describe, mostly oceluded/incoopplte. only background scenery (e.g.. distant sky, wall, floor).
Add final key "All Objects" whose value is a single string listing all object names, formatted as: "(material) (color) [object name). {color) (object name). [material} (object name). (object name]." " Exclude "nune"/"Noee" and separate entries by :. (period space). Do not include any text content in this list.
Example output (abridged JSON):
"cotton blue shirt": (object: "shirt", color: "blue", material: "cotton", text: null,
count: 1, foreground: true)
"ceramic white cup": (object: "cup". color: "white", material: "ceramic", text:
"GOOD DAY", count: 1, foreground: false)
"leather bag": [object: "bag", color: null, material: "leather", text: null, count: 2,
foreground: true)
"red scarf":: {object: "scarf", color: "red", material: null, text: null, count: 1, fore- ground: true)
"All Objects": "cotton blue shirt. ceramic white cup. leather hag. red scarf."
M.2.2 TASK PROMPTS
Subject Replace
You are given an image and asked to suggest a replacement object for specific object the scene.
Given object to replace: object name
Your task:
Understand the seene context.
2. Suggest object that naturally replaces "object name".
31

---

Published as a conference paper at ICLR 2026
3. Ensure the suggestion realistic for the scene.
4. Respond with only the object name (e.g.. "chair", "lamp", "book"). Examples: In a kitchen: "howl", "mug"; on street: "bus", "truck"; in an office: "stool", "bench".
Answer format:
New object name:
Material Alter
You are given an image and asked suggest new material for specific object. Object: bbjecttnamme Current material: current material
Your task:
1. Identify the object.
2. Suggest a realistic alternative material that is casy to distinguish from the current one.
3. Respond with only the material name (e.g., "wood", "metal", "plastic", "leather").
Examples: cup: ceramic, glass, metal, plastie; chair: wood, metal, plastic, fabrie; bag: leather, canvas, nylon, fabric.
Answer format: New material:
Position Change
You are given an image and asked to create position change instruction.
Available objects: available objects Positions: left, right, above, below
Your task:
Select a target object move a reference ohject.
Choose relative position (left, right, above, below).
Ensure the instruction physically reasonable.
Format: "Change the position of [target object] to [position] of [reference object]".
Examples: "Change the position of [cup] to [right] of [book]"; "Change the position of [lamp] [above] of [table]".
Answer format: Position change instruction:
Count Change
You are given an image and asked to create count change instruction. Available objects: available objects Target count: targer count Your task:
1. Identify suitable object for the requested count.
2. Ensure the target count is realistic for the seene.
3. Format: "Change the count of [object name] to [target count]".
Examples: "Change the count of [cup] to [3]"; "Change the count of [book] to [2]".
Answer format: Count change instruction:
Text Change
You are given image and asked to generate new text content.
Context: text situation
Your task:
1. Generate text that fits the scene.
2. Keep text short: max 2 words in English or Chinese characters.
3. Respond with only the text content (no quotes or extra words).
Examples: coffee shop: "COFFEE", "OPEN"; book: "NOVEL", "GUIDE"; sign: "EXIT".
"STOP": Chinese: " ]uul " I1i D
Answer format: New text:
Color Alter
32

---

Published as a conference paper at ICLR 2026
You given image and asked suggest color for specific object.
Object: object name Current color: current color
Your task:
1. Suggest a simple, common color that fits the object.
2. Use only basic colors: red, blue, green, yellow, black, white, brown, gray, orange, purple, pink.
3. Choose color different from the current color and answer with the color name only.
Answer format: New color:
Subject Add
You are given an image and asked suggest new object to add to scene. Reference object: reference object Position: position
Your task:
Propose an object that would naturally the specified position relative to the erence object.
Ensure the suggestion realistic and contextually appropriate.
3. Respond with only the object name (e.g., "lamp", "book", "cup").
Examples: next to a desk: "chair", "lamp", "computer"; near a kitchen counter: "bowl", "plate", "mug"; by window: "plant", "curtain", "book".
Answer format: New object:
Background Change
You are given an image and asked to suggest a new background for the scene. The existing objects should remain unchanged.
Your task:
background that works with the current setting. 2. Keep it simple and realistie; use 1-2 words (e.g.. "kitchen", "office", "garden", "beach", "forest").
1. Propose
3. Respond with only the background name.
Answer format: New background:
M.3 EVALUATION
We evaluate in two modes: (i) Multi-turn (each turn edits the output of the previous turn), and (ii) Complex Editing (compress all instructions single prompt).
Instruction Following. We compute binary success per instruction with detector combining GroundingDINO Liu et al. (022a) and VLM (Qwen2-VL-7B) Bai et al. (2025). Representative details:
Detector thresholds. Unless noted per task, GroundingDINO thresholds 0.3-0.4; detec- tions return normalized boxes [21, V/, x2, 8/2-
Cropping and small objects. For object-level checks we crop by detected boxes; very small boxes (< 0.05 in width/height) can be enlarged before VIM queries.
Replace. Detect old and new objects in source/target; success both are detected and any IoU between source box (old) and target box (new) is 0. A VLM pre-check rejects obvious non-replacements. See details Alg 3
Remove. Detect the object in source; success if the object absent in target. See
details in Alg 4
Position change. Detect target and reference objects and verify the requested spatial relation using object centers; also ensure the object count did not increase spuriously. See details in
6
Count change. Use the detector to locate instances of the target object and take the number of validated detections the count. See details in Alg 7
33

---

Published as a conference paper at ICLR 2026
Algorithm 3 Evaluate Subject Replace
Require: base B, target T, old object name new object name n
Ensure: success flag suce
1: S DETECT B, o, T):
T, DETECT(T, n)
2: f T, + then
3:
- IET, (bU(b, >0
4: else
5:
false
6: end if
7: return
Algorithm Evaluate Subject Remove
Require: base B, target T,, object name o
Ensure: success flag succ
1: S DETECT B, o, r); 2: succ + (S # 0 A To = 0)
t + DETECT(TTO,7)
3: return suce
Color/material. Crop the object in the target and ask the VLM a yes/no question about the new color/material. See details in Alg and Alg
Text change. If the instruction adds text anywhere, run the VLM on the whole image; if replaces text specific object, first crop that object's box, ask the VLM extract the text, and compare it to the requested text. See details in Alg
Background change. Ask the VLM yes/no whether the requested background category is present. See details in Alg 11
Consistency.
We measure object and background stability as follows:
Object consistency (unchanged objects): DINOv3 ViT-B/16 (iiéonii et al. (2025) feature similarity between crops of unchanged objects in base vs. target; we also report pixel L1 consistency and average across objects per image.
VTT-B//6[Siméoni
Background consistency: detect objects in all all_objects_pon in base/target (Ground- ingDINO), mask them to isolate background, then compute masked LI between back- grounds (optionally DINOv3 masked similarity). Background consistency is evaluated only when no background change occurred earlier (bg.consistency--aa.
Perceptual Quality. We report HPSv3 Ma al. (2025) plausibility and aesthetics, plus luminance metrics. Quality is not folded into the Overall score.
M.4 OVERALI SCORE AND AGGREGATION DETAILS
Let a be the image success rate at turn the fraction of images for which edits up to and including turn successful (aggregated per task type, then averaged). Let denote the average content-consistency score combining object and background DINOv3 similarities when applicable.
Overall score. We report
Overall = [mean (ar) x mean(k) 1/2
Missing outputs across turns. For summary tables, we include only images that produce all required outputs for the evaluated mode. If a model fails to generate later turn, that image is omitted from later-turn aggregates for that mode. Some edits will be rejected by some models since the sensitive content flag.
No unchanged objects. If the unchanged-object list is empty, object consistency is recorded as None and excluded from averages; background consistency is still computed when bg consi stency=true.
34

---

Published as a conference paper at ICLR 2026
Algorithm 5 Evaluate Subject Add
Require: base B, target T, new object name n, optional reference object name optional position p E {left, right, above, below]
Ensure: success flag
1: B, + DETECTTTBB,,... n, T);
Ta + (TT
2: if',
V Bn # 0 then return false
3:
4: end if
5: r and are provided then
6:
B, + (ETECT(B,r, rT); T, (,TT
7:
if Tf = 0 then
8:
return false
9:
end if
Choose max logits boxes
E na u E T
11:
(Iz, Ut) <- CENTER((); (Tuz Mu) CENTER
12:
p = left 2z < Iu - Ex then
13:
return true
14:
end if right > Zu Ex then
15:
16:
return true end if
18:
if p = above A yt < yu - Ey then
return true
20:
end if
if p below A Yt Yu + Ey then
22:
return true
23: end
24: return false
25:
else
26:
return true
27:
end if
Turn-level reporting. We also report per-turn (T1, T2, T3) instruction-folloving and con- sistency, and per-task-type success rates 04 Quality metrics are reported separately and are not folded into Overall.
M.5 MODEL GENERATIONS
We evaluate mix of closed- and open-source editors using each model's default settings (no hyper- parameter tuning):
GPT-Image-1, Nano Banana, and Gemini 2.0 Flash: called via their APIs with default parameters.
QWEN Image Edit: default settings from http::/huggingfaccc.c/ooaaa
Owen-Imagen Image-Edit
InstructPix2Pix (IP2P): settings from http:://gghhbb..mmmmmmttttttooop instzuct-pix2pix
Magicbrush: same settings as IP2P: model weights from htttp://hugginni co///inessuuc/naacccrruubbbbb
UltraEdit:
settings from apply black mask since explicit mask is provided.
we
AnyEdit: repository at
with edit type set to
general.
at weights at at/https: httph///s/////nnno////oooooooooe
35

---

Published as a conference paper at ICLR 2026
Algorithm 6 Evaluate Position Change
Require: base B, target T, target object name reference object r, position Ensure: success flag
1: B, < DETECT B, a, T);
Ta (ETECT(T,a,7 a,))
2: B, < DETECT(B,r, r, T):
Tr f- (T,,C
3: Tf 0 T, then
4:
return false
5: end if
6: if [2a] > [Bal then
7: return false
No count inflation
8: end if
9: Select max logits boxes
Ta, u
10: (ze. V4) < -CENTER((); (Iur Yu) & CENTER(U)
if p left then
12:
return
Zu
13: end if
14: if
right then
15: return 24 > Zu + Ex
16: end if
17: if = above then
18: return Yr Yu Ey
19: end if
pf below then
21: return > Yu + Ey
end if
23: return false
Algorithm Evaluate Count Change
Require: target T,, name o, requested count
Ensure: success flag
8 [ETECT(T..)) 2: return (é = c*)
at FLUX: settings from httpo//wwwppp..ggoaaa...e.eaaaaa FLUX. -ee
OmniGen: http://////////////////////////a
Modes. For clarity in the paper: we report both Multipass and Complex Editing (renamed from singlepass for consistency with the rest of the paper).
Reproducibility Notes. Prompts provided in full (Section M.2 thresholds specified above. Grounding uses SwinT-OGC weights; consistency uses DINOv3 ViT-B/16; the quality head follows our RAHF implementation, and HPSv3 is included when available. All other parameters are left defaults.
36

---

Published as a conference paper at ICLR 2026
Algorithm 8 Evaluate Color Alter
Require:
target image T, object name color k:
I: return (LMYESNO(T, "Is the k?")
Algorithm Evaluate Material Alter
Require: target image T' object name o, material m
Ensure: success flag
I: return YLMYESNO(T "Is the made of m??"
Algorithm 10 Evaluate Text Change
Require: target T. desired text t" (optionally object name) Ensure: success flag
1: / 4 VLMTEXT(T)
2: Normalize i and 1a (case, punctuation, whitespace)
3: return TEXT (EXT-MATCH(t,t')
Algorithm 11 Evaluate Background Change
Require: target T, category
Ensure: success flag
1: return (IS "Does the background show g?")
37

---

Published as a conference paper at ICLR 2026
Table 15: Image success rates and overall task success rates across turns, (Multi-turn model)
Model
Image Success Rate
Overall Task Rute
Seedream 1.0
75.93
55.58
41.59
75.93
75.58
76.11
Nano Banana
70.70
50.66
35.35
70.70
72.59
68.24
GPT-lnage-1
FLI1X L-Kotext--axx
69.49
46.89
31.83
69.49
69:11
70.43
Gemint 2.0 Fash
68.07
45.96
28.42
68.07
67.72
68.42
Quen- Qwen-Imago-Fale
72.90
Step1 X-Edit
61.89
34.97
17.83
61.89
99.09
53.32
FLUX, I- Kontext-dev
59.97
32.69
16.61
99.97
36.29
$1.40
OnniGen
54,72
24.48
10.66
51.72
48.60
42.48
UltraEdit
AnyEdit
41.07
16.32
7.22
40.03
39.34
40.56
MagicRrush
42.31
15.73
4,0
42,31
40.73
41.26
Table 16: Task success rates (%) across five instruction types and three turns (multi-tury mode).
Subject Replace
Subject Remove
Material
Subject
T1
T2
T3
Seedream 4.0
90.74
91.23
88.89
68.92
47.60
50.00
95.31
96.00)
95.27
100.00
95.55
100000
X3.08
89.61
81.32
Naso Ranana
91.84
92.31
75.47
64.18
$1.61
40.35
91.94
89.36
87.50
100.00
97.18
98:11
73.02
77.50
72.41
GPT-Image
84.31
94.64
85.71
70.77
55.93
47.37
96.83
95.65
$7.88
100.00
97.05
100.00
50.95
12.46
12.41
BLUXX_-Kontest--it max
Gemini 2.0 Flach
83.33
92.98
78
58.67
53.62
50.82
90.91
82.00
83.33
100.00
89.04
98.21
77.61
72.73
75.27
Owen Image-mageee Edit
87.04
$2.45
70.91
70.67
31.88
37.70
93.94
90.00
79.17
100.00
97.25
94.74
77.61
55.84
39.78
IX-Edit Edit
90.74
96.49
67.27
53.33
30.43
21,31
95.45
80.00
87.50
100.00
100.00
91.23
64. 18
57.14
45.16
FLUX.1
OnniGen
84.21
58.
46.67
21.74
19.67
84.85
72.00
70.83
100.00
90.41
91.23
53.73
51.95
37.63
Ulrabdi
88.89
63.16
38. 38.18
26.67
5.80
6.56
87.88
66.00
63.89
98.21
80.82
78.95
38.81
23.38
9.68
AnyFdir
74.07
66.67
61.82
37.33
39.
36.07
78.79
68.00
68.06
78.57
68.49
78.95
22.
38.96
25.81
MagicBruch
83.33
75.44
63.64
28.00
83.33
80.56
87.67
37.31
41.56
37.63
28.57
19.35
N ADDITIONAL EVALUATION RESULTS
In this section, we provide extended evaluation results. We separate the analysis into two modes: multi-turn editing and complex editing. Each mode evaluated across three aspects: instruction following, consistency, and quality.
For multi-turn editing mode, the overall foslroi success rate reported in Table 15 while success rates for individual instruction types appear in Tables and 17 Consistency results and
are summarized in Table 21 We also observed that some input images are non-square after resizing, which leave black padding on the top/bottom or left/right edges. Certain editing models, such as GPT GPT-Imaae--1 and Image-ima,--Fdit,, attempt to fill these areas, whereas others preserve them. To account for this, separately report consistency for square (Table 22 and non-square inputs (Table 23) The conclusions remain consistent with the overall evaluation. Quality results for multi- turn editing are presented in Table 25
For the complex editing mode, the overall instruction-followinn success rate is shown in Table 18 and per-instruutioo-ttyee results are in Tables 19 and 20 Consistency and quality results are reported
in Tables 24 and 26 respectively.
In consistency table, p99 means 99% quantile of luminance value, and p999 means 99.9% quantile of luminance value.
38

---

Published as a conference paper at ICLR 2026
Table 17: Task success rates (%) across four instreution types and three turns (multi-turn mode).
Text Change
Position Change
Count Change
Background Change
Model
T2
T3
T2
T3
T1
T2
T3
T1
T3
Scedream 4.0
95,31
97.14
96.23
39.22
39.68
48.94
18.06
19.30
20.00
98.46
96.36
93.06
Nano Banana
83.33
86.36
81.25
20.00
47.37
45.24
23.19
9.62
10.91
96.67
94,44
90.00
GPT-Image-1
88.33
97.01
97.96
31.11
40.68
50.00
11.27
18.52
18.18
98.39
94.44
91.30
FLUX.1 -ootexxt-max
80.36
86.15
82.69
18.75
38.18
44.44
8.82
9.09
12.28
96.88
92.59
92.54
Gemini Flash
90.32
94.29
96.30
21.15
28.57
31.91
10.96
7.14
10.00
86.15
83.64
80.56
Dwenn-Image-Bie Edit
98.44
92.86
72.22
21.15
34.92
33.33
12.33
1.72
0.00
98.46
80.00
77.78
StepIX-Edit
60.94
51.43
44.44
13.46
31.75
27.08
0.00
1.72
1.67
87.69
87.27
83,33
FLUX.1 1- Kontext-dev
50.00
41.43
27.78
15.38
26.98
33.33
0.00
1.72
5.00
90.77
80.00
77.78
OmniGen
29.69
35.71
18.52
17.31
22.22
20.83
5.48
5.17
76.92
56,94
UltraFdit
28.12
15.71
7.41
21.15
36.51
35.42
5.48
6.90
5.00
75.38
38.18
43.06
AnyEdit
3.12
10.00
11.11
21.15
25.40
27.08
0.00
1.72
1.67
56.92
40.00
52.78
MagicBrush
7.81
12.86
3.70
19.23
15.87
20.83
0.00
0.00
3.33
43.08
34.55
43.06
IP2P
1.56
5.56
15.87
0.00
47.69
31,94
Table 18: Image rates, overall task rates, and marginal means across three turns (complex mode).
Image Success Rate
Overall Task Rate
Marginal Task Rate
Model
T2
T3
T2
T3
T2
T3
GPT-Image-I
73.08
48.45
28.78
73.08
69.77
68.25
73.08
69.98
70.19
Nano Banana
71.46
46.56
28.14
71.46
68.83
67.27
71.46
69.03
67.21
Gemini 2.0 Flash
67.43
40.63
21.89
67.43
64,54
61.94
67.43
64,80
62.17
Qwen- Owen-Image-Edic
72.90
46.15
27.62
72.90
69.23
68.07
72.90
67.83
67.66
StepIX-Fdit Edit
61.36
32.34
15.73
61.36
$7.69
55.01
61.36
55.77
$4.02
FLUX. 1 --nntext-dev
60.14
33.74
19.58
60.14
59.53
57.87
60.14
57.52
56.99
OmniGen
54.55
23.43
11.01
54.55
50.96
49.83
54.55
48.95
47.90
AnyEdit
39.86
10.31
2.80
39.86
34.79
34.27
39.86
31.47
31.64
Ultralsdit
50.70
22.03
8.22
50.70
48.34
46.62
50.70
47.73
47.38
MagicBrush
42.13
14.86
4.55
42.13
38.46
38.81
42.13
38.64
39.34
IP2P
37.24
12.41
2.80
37.24
37.76
35.14
37.24
38.46
34.62
Table 19: Success rates (%) for five instruction types across three turns (complex mode).
Subject Replace
Subject Remove
Material Alter
Color Alter
Suhject Add
GPT--nage-1
82.22 22
80.65
78.99
70.31
63.87
58.54
96.19
90.20
81.66
100.00
97.27
91.88
$1.67
70.73
69.00
Nano Banana
91.6 67
88.89
79.33
67.16
55,83
39.88
92.59
85.15
83.33
97.87
98.15
94.19
69.84
71.21
67.77
75.00
Qwen-Innge-Edde
87.04
86.19
80.72
70.67
58.33
54.63
93.94
36.21
85.64
100.00
94.22
98.39
77.61
75.69
73.84
StepIX Edit
90.74
$4.68
76.51
$2.00
41.67
42.93
93.94
$2.76
79.26
100.00
93.02
90.86
64.18
59.03
$4.01
FLUX.1 Kontext-da dey
85.19
82.88
74.10
54.67
47.92
40.00
86.36
75.00
76.06
100.00
99.22
98.39
67.16
68.75
63.29
OmniGen
39.58
73.28
AnyEdit
66.67
58.56
49.40
33.33
20.14
22.44
77.27
76.72
70.21
78.57
68.99
71.04
28.36
22.92
21.10
Ultraiiie
88.89
87.39
78.31
26.67
30.56
31.71
87,88
79.31
77.66
98.21
93.02
90.32
38.81
46.53
43.46
MagicBrash
83.33
74.77
69.28
28.00
27.78
29.27
83.33
69 83
73.94
96
84.50
83.87
37.31
35.42
32.91
75.93
73.87
61.45
25.33
31,25
24.8
71.21
63.79
59.57
87.50
87.95
79.03
23.88
31.94
29.11
Table 20: Success rates (%) for four instruction types across three turns (complex mode).
Text Change
Position Change
Count Change
Background Change
Model
T1
T2
T3
Tl
T2
T3
T1
T2
T3
TI
T2
T3
GPT-Image-1
87.50
92.24
93.75
30.95
22.11
25.36
13.11
10.38
13.12
98.15
96.08
93.37
Nano Banana
84.21
85.34
84.24
24.44
26.00
22.54
20.34
14.15
13.75
98.15
93.40
93.71
Gemini 2.0 Flash
85.94
86.57
84.04
17.31
23.68
16.67
11.11
10.77
6.84
90.77
84.17
80.73
Qwen- Brren-Imaae-Fain
98.44
97.01
93.62
21.15
24.54
12.33
3.05
3.66
98.46
95.83
93.75
Xtep1X-Edit
57.81
59.70
52.13
15,38
21.74
25.15
1.37
3.05
1.57
86.15
80.00
73.44
FLUX.1 1- donvext-dev
50.00
48.51
47.87
15.38
26.96
27.61
0.00
1.53
3.66
90.77
90.00
88.54
OmniGen
28.12
32.84
27.13
15.38
21.74
17.18
6.85
1.53
2.09
75.38
70.00
69.79
AnyEdit
3.12
5.22
25.00
26.96
29.45
1.37
1.53
1.57
$6.92
44.17
40.62
UltraFdit
29.69
16.42
11.70
21.15
26.09
22.70
5.48
2.29
3.14
75.38
65.00
64.06
MagicBrush
7.81
4.48
6.38
19.23
22.61
17.18
0.00
0.00
5.24
43.08
36.67
35.42
IP2P
3.12
5.22
6.38
13.46
20.00
20.25
1.37
2.29
1.57
47.69
37.50
38.54
39

---

Published as a conference paper at ICLR 2026
Table 21: Consistency scores (%) across DINOv3-based and L1-based object/background metrics.
(multi-turn mode)
Model
Object DINOV3 Consistency
Backgroand DINOv3 Consistency
Objert
Consistency
Background
I, Consistency
T2
T3
F2
T3
T1
T3
T1
T2
T3
Seedream 4,0
83.68
90.54
93.31
88.76
86.22
94.30
90.17
85.25
84.38
95.20
94.58
99.10
77
75.07
-LUXX.-KKoutet-mma
92.11
94.16
Gemini
77-12
90.72
84:11
95.04
93.15
91.73
Qwa--ImageFili
77.12
71.56
68.51
91.31
89.47
87.45
83.48
79.15
76.32
84-57
81.18
78.43
Sup X Edit
88.17
$1.65
77.33
97.34
93.40
93.09
93.92
90.64
85.
96.24
97.10
95.73
FLUX Kontest dev
92.66
S1.92
85.29
97357
96.35
95.14
$4.39
91 91.59
89.91
96.36
95.06
94.13
Deniclan
58.
$0.77
73664
97
96.08
94.21
93.87
91
89.41
97.44
97.00
96.34
Ukraadit
78.81
75.11
72.24
94.30
93.89
92.57
91.86
89
97.12
AayEdit
82.02
MagicBoush
79.20
91.13
56.83
48.01
72.59
77.21
91.31
Table 22: Consistency and L, metrics across three turns (multi-turn mode for square image).
Model
Object DINOv3 Consistency (Mean)
Background DINOv3 Comsistency
Object I, Cunsistency (Meam)
Background
Consistoncy
T2
T2
T3
T1
12
TI
Seedream 4.0
$9.04
85.39
80.74
95.76
92.43
$5.27
91.85
55.00
85.45
95.34
92.45
$9.09
Naro Bunsus
$8.68
85.19
83.31
96.85
93.79
92.41
89.73
57.64
88.48
94.21
93.77
93.36
GPF-taage-1
73.62
71.27
67.27
90.00
85.72
82.16
92.33
78.80
79.88
$5.09
$3.86
50.85
FLUX Kontou-mas mas
91.34
89.90
86.04
95.99
94.18
90.10
93.81
92.80
90.58
97.27
95.77
93.94
Gemin 2.0 Flvsh
64
70.11
71.13
91.50
91.85
87.
$3.95
87.07
85.13
95.15
93.35
99.34
Qua--mage-Eit
16.56
72.68
68.35
88.50
88.95
82.71
$1.65
78.47
77.04
91.16
87.85
85.78
XeplX-Eiit
$7.40
84.03
78.27
97.43
92.42
85.15
93.14
91.57
89.40
98.10
97.11
96.01
RLUX Kentext-dav
92.55
68.92
8404
96.69
93.57
92.
92.83
90.61
86.34
96 91
95.74
94.71
Ommii
89.41
64.51
77.77
97.34
93.13
86.64
93.26
91:40
89.53
95.70
95.44
95.36
Uliaadii
79.41
76.51
71.54
92.83
8988
54
91.74
90.15
89
98.87
95.14
94.63
Adi
62.25
72.40
53.59
75.60
70.09
92.33
94.36
$7.55
Migicilrush
79.02
75.60
70.07
92.07
67.33
81.08
59.67
87.53
36.90
95.28
91.88
92.38
IP2P
76.35
65.12
34.94
65.29
81 46
65.50
86.90
77.79
92.45
83.01
84,51
Table 23: Consistency and Li metrics across three turns (multi-turn model for unsquared image).
Object DENUM Consistency (Mean)
Bookkrrand DINUrd Consieiency
Objeect
Zg Conditency (Mean)
Background
Z. Conelstency
Seedream 4.0
55.48
Nano Haranz
00:14
15.25
$4.50
97.24
95.
155
04:31
94.51
91.95
Lasge
73.22
65.53
67.20
35.60
36.85
83.99
79.57
78.27
77.35
17.20
75.91
75.14
FLUX Kontest
90.56
96.31
93.43
97.05
95.27
93.51
94.20
91.09
89.34
95.73
93.16
91.14
Ckmine 2.0 Fash
R3
76.79
72.12
96.14
03.23
00. 10
90:31
86.25
$3.90
05.02
03.12
91.51
Qwenddage--al Eali
77.32
71.43
63.47
91.63
39.53
85.10
$3.68
70.33
76.23
63.78
80.37
77.43
Ste1X bdtt
88.26
$1.37
71.21
97.33
95.1
93.77
94.01
90.53
58.73
98.26
97.08
95,70
FLUX Hoooeest-dov
01.67
9K.15
98.1
91.00
94.98
94.00
87.00
23.53
MagisBnssh
IPIF
55.75
Table 24: Consistency scores (%) across object/background DINOv3 and L, metrics (complex
mode).
Object DINOv3
Object
Li
Background DINOy3
Background Ly
Model
T1
T2
T3
T1
T2
T3
T1
T2
T3
TI
T2
T3
GPT-Image-1
73.23
70.02
67.57
79.52
78.03
77.29
88.72
86.77
84.79
77.96
77.38
76.51
Nano Banana
89.23
87.20
86.46
92.15
91.08
90.40
97.39
96.69
95.38
96.39
95.75
95.33
Gemini Flash
85.41
80.37
77.38
90.60
88.45
86.53
96.03
92.83
95.03
93.46
Qwen-Image-EEl Image-Edit
77.12
76.09
76.69
83.48
83.08
83.11
91.31
91.32
90.51
84.57
84.93
85.35
Step StepIX-Edin
88.14
85.31
84.38
93.93
92.34
92.11
97.34
96.37
95.44
98.24
98.02
98.04
FLUX. Kontext-dev
92.66
90.30
94.39
92.79
91.61
97.97
95.40
96.36
95.57
94.04
OmniGen
88.37
85.15
83.14
93.88
92.46
91.06
97.62
97.10
96.07
97.45
97.58
97.40
AdyEdit
81.90
82.94
84.92
92.34
92.43
93.72
90.97
92.63
93.78
94.15
95.11
95.87
UliraEdit
78.81
72.75
71.67
91.86
89.51
88.99
94.80
93.01
92.03
97.12
96.35
96.02
MagicBrush
79.70
75.64
75.53
91.13
89,23
88.69
94.22
94.34
93.13
96.31
96.14
95.83
IP2P
68.24
67.31
84.44
82.93
83.72
85.47
85.88
86.45
91.31
90.19
40

---

Published as a conference paper at ICLR 2026
Table 25: Human preference scores, p999, and p99 across three turns (multi-turn mode).
Model
Human Preference Score
p99y
p99
T1
GPT-Image-1
6.65199
6.5898
6.5609
84.73
82.9%
82.50
74.38
71.91
70.54
Nano Hanana
4.9431
5.1179
5.2638
89.67
90.27
90..5
81.31
82.01
82.09
Owen-Image Edit
5.8591
5.7198
5.1502
89.16
97.20
99.04
79.50
90.51
95.28
E-Fir
4.0577
3.3443
2.7569
92.81
95.39
97.27
85.10
88.46
91.21
Kontext-dev
5.1192
OmniGen
4.6099
4,0743
3,4958
93.15
95.24
96.71
83.55
$8.24
90.50
AnyEdii
3,6609
2,8017
1.9457
86.70
86.
86.71
77.54
76.53
75.82
Uhrabdit
4,7934
MagicBrush
3.8465
3.0805
2.3606
91.49
93.33
94.20
83,42
84.70
85.32
IP2P
3.2020
2.3779
1.4418
89.61
91.59
92.44
81.79
83.78
84.77
Table 26: Updated human preference scores, p999 scores, and p99 scores across three turns (com- plex mode).
Model
Human Preference Score
p999
p49
T1
12
13
T1
1:2
T3
11
122
T3
GPT-lmage-1
6.9655
Nano Banana
4.9444
5.1700
5.3632
89.65
90.67
91.79
81.00
81.93
82.75
Gemini 2.0 Flash
4,4511
4.5428
4.5732
91.27
92.66
93.48
83.85
85.69
86.75
Qwen-Image-Eue
SteplX-Edit
4.0534
3.9063
3.8648
92.82
93.55
94.05
85.11
85.49
86.01
FLIIX. don
5.1192
5.2446
5.4645
90.76
91.01
91.29
82.05
81.53
81.55
OmniCenn
AnyEdit
3,7020
3.7601
3,8382
86.47
87.16
87.43
77.82
73.60
79.12
Ubraaidit
4,7934
4,7647
4.8117
92.71
93.06
93.24
85.42
86.09
$6.47
MagicBrush
IP2P
3,2020
3.3552
3.5640
89.61
90.46
90.86
81.79
82.57
82.71
41

---

Published as a conference paper at ICLR 2026
O HUMAN AGREEMENT
The human study was conducted online through Gradic Annotators were asked to answer a 2-way multiple-choice problem (Yes/No) about an editing instruction, original image, and an edited image. There were very limited potential participant risks, if they were to be exposed to an image that was disturbing or not safe for work (NSFW). is because the source images we used were from BEit-Bench|/ Liu et (205)), which were not in themselves offensive. Also, our agent already filtered out unsafe images during the first decomposition stage. Furthermore, all edited images from the models were passed through its own NSFW filters which blacked out any potentially unsafe content.
We conducted human study on edits made by four exemplary models-Stepll StepIX-Edit AnyEdit, Gemini-Flash 2.0, and Flux---onnext-ee--o EdiVal-Bench, generated by EdiVal-Agent as described in Section M.2 For each edit, we collected two human ratings, yielding total of 572 x 2 = 4,576 annotations. Depending on the prompt (which affected the editing instruc- tion), each annotation took about 1-2 minutes. Raters were recruited online, each holding at least a bachelor's degree. They were shown the original image, the edited image, and the correspond- ing instruction, and were asked binary question: "Evaluate whether the edited image successfully follows the given instruction."
P COUNTING
Among all subtasks, count_change is the most challenging, Even the best-performing model (GPT- Image-1) achieves a success rate below 25% turn I, while most models remain under 5%, We also provide illustrative examples in Figure
(a) Base Image
(b) GPT-Image-1 (V)
(c) Nano Banana (V)
(d) Gemini 2.0 Flash (x) (c) Qwen-Image-Edit Edit (x)
Figure 23: Example of the count chunge task: changing the number of paper cups to five.
httts://www.gradi.. app/
42

---

Published as a conference paper at ICLR 2026
People Scenes
(a) One person
(b) Cartoon person
(c) Crowd
(d) Kids
Outdoor Nature
(e) Forest
Outdoor
(g) Outdoor
(h) View
Special Scenes
ii Snow
ii Volcano
(k) Dusk
(I) Romance
Indoor Scenes
(m) Furniture
(n) Cake
(o) Cat
(p) Dog
Special Items
Dreyer's English
IELTS
(q) Book Cover
(n) Ads
(s) Logo
(t) Menu
Artistic/ Cartoon Scenes
(u) Artistic
(v) Cartoon
(w) Poster
(x) Statue
Figure 24: Scene diversity of our dataset across multiple environments and content types.
43

---

Published as a conference paper at ICLR 2026
Q DIVERSITY OF OUR DATASET
hotwind
BLaCK
WUKONG
32
Welcoms d ONTARIO
Figure 25: Images of different lightning sorted from very dark to very bright.
Object Classification Exampies
Sod Ko
lecte
114 care waler
Figure 26: Categories and examples of objects in the base images.
All base images can be found here: the GEdit-Bench's official Huggingface link https://
huggingface co htt//aassssss///e /ttp://tssttteeeerrrraaaaaaaaaaa
Our dataset could reflect real-world editing scenarios in the following senses.
Data Source We first stress that our images
1. are from real-world user editing cases as stated in GEdit-Bench Liu et al. 2025
2. include both synthetic and real images
3. are carefully selected by the GEdit team to ensure the diversity.
Different Scenes
We demonstrate example images of different scenes in Fig. 24 which covers assorted environments and content types, including but not limited to indoor/outdoor, person, special items, and artistic scenes. This diversity helps dataset better reflect the real-world user cases.
44

---

Published as a conference paper at ICLR 2026
Different Lightning We demonstrate images of lightning conditions in Fig. 25 sorted from very dark to very bright. Our dataset includes a variety of illumination scenarios that aim to cover the spectrum of very bright to very dark environments. This variation reflects the real-world complex editing scenes.
Object categories. After decomposition, we analyzed and found that there are a total of 724 dis- tinct objects in the dataset. We classify them into 13 categories. ranging from everyday items such as furniture and kitchenware to special entities like vehicles and animals. Specifically, they are (The distribution of these objects is presented in Fig. 26 ):
1. Clothing & Accessories
2. Furniture & Home
3. Food & Kitchen
4. Structures & Places
5. Vehicles & Transport
6. Nature & Outdoors
7. Animals
8. Sports, Toys & Hobby
9. Stationery & Office
10. Electronics & Tech
11. Body & Medical
12. Tools & Industrial
13. Other
45

---

Published as a conference paper at ICLR 2026
Input
T1
T2
T3
Seedream 4.0
Nano Banana
GPT-Image-1 I
Gemini 2.0 Flash
FLUXe--Kottxxt-max
Qwen-Image-Edti
Figure 27: T1: Change the color of pumpkin to purple; T2: Change the background to forest; T3: Remove fabric orange bow. Row-wise quality examples for the first six models: Scedream 4.0, Nano Banana, GPT-Image-1, Gemini 2.0 Flash, Each row shows generations for Input and three editing turns.
and Qwen- Owen-Image-Edit.
R MORE QUALITY EXAMPLES
46

---

Published as a conference paper at ICLR 2026
Input
T1
T2
T3
Step StepIX-Edi IX-Edit
ILUX.I-kontext-de
OmniGen
AnyEdit
UltraEdit
MagicBrush
IP2P
Figure 28: T1: Change the color pumpkin to purple; T2: Change the background forest; T3: Remove fabric orange bow. Row-wise quality examples for the remaining models: Step1X- Edit, FLU..L-UUUUOOO ---onttxtt-eey, OmniGen, AnyEdit, UltraEdit, Bruic and IP2P.
47


Reviews:
[{"review_id": "OKJ5DBM9eG", "reviewer": "Reviewer_FCAS", "summary": "This paper introduces EdiVal-Agent, an automated object-centric framework that addresses the unreliable evaluation of multi-turn image editing by first decomposing scenes into objects and then employing a novel hybrid evaluation strategy. This approach strategically combines open-vocabulary detectors for precise symbolic checks and VLMs for semantic verification on localized crops, leading to its core metrics for instruction-following (EdiVal-IF) and content preservation (EdiVal-CC), while separately employing a specialized human preference model for visual quality assessment (EdiVal-VQ) to account for subjective aesthetic judgments. Its primary contributions include the proposed pipeline, the creation of the large-scale EdiVal-Bench for multi-turn editing, and a comprehensive evaluation of 13 editors that provides critical insights—such as diagnosing exposure bias and performance degradation over turns—thereby establishing a new, more reliable, and interpretable standard for the field.", "questions": "1. Data Selection and Representativeness\nThe paper states that the benchmark contains 572 real-world images. What were the specific, detailed criteria for selecting this dataset? Was consideration given to diversity across essential dimensions such as scenes, lighting conditions, and object categories? Crucially, how can the authors demonstrate that this relatively small dataset is statistically significant and sufficiently representative of the diverse and complex demands of real-world image editing?\n\n2. Blind Spot in Evaluating Generated Images\nThe benchmark completely excludes synthetically generated images. Given that many advanced editing models are built upon generative techniques, and considering the rapid proliferation of AI-generated content, does the absence of evaluation on this class of images introduce a significant bias in the results? How does this limitation affect the benchmark's long-term relevance and its ability to stress-test state-of-the-art generative models?\n\n3. Editing level\nWhat was the precise rationale for setting the editing depth at 3 turns? Is there experimental evidence demonstrating that three turns are sufficient to expose critical issues in multi-turn interactions, such as error accumulation and state drift, or was this a pragmatic compromise between computational cost and evaluation depth? Furthermore, the current nine editing types do not cover common and fundamental operations like zooming, rotation, or adding special effects. Could this restricted task taxonomy lead to an incomplete or skewed evaluation, potentially misrepresenting the true ranking of model capabilities?\n\n4. Dependency on GPT-4o and Evaluation of Alternatives\nThe core image decomposition phase relies heavily on the closed-source model, GPT-4o. While the paper claims that other Visual Language Models (VLMs) could serve as alternatives, this assertion lacks supporting data. How stable or robust would the evaluation pipeline outputs be if alternative, open-source VLMs (e.g., Qwen-VL or Llava) were utilized? Does this deep reliance on a single, powerful commercial model introduce the very \"evaluation bias\" that the proposed framework aims to critique and overcome?\n\n5. Positioning and Autonomy of the \"Agent\"\nThe system is named \"EdiVal-Agent.\" However, its described workflow appears to be predefined and strictly rule-driven. To what extent does the system actually possess the autonomous reasoning, dynamic decision-making, and adaptive planning capabilities typically expected of a true intelligent agent, as opposed to functioning as a complex, yet fixed, automation script?\n\n6. Benchmark Evolution and Obsolescence\nAs a static benchmark, how do the authors plan to mitigate the critical risk of community overfitting? Do they have concrete mechanisms or a roadmap for continuous update and evolution—such as periodically introducing new images, fresh editing types, or adversarial instruction sets—to ensure the benchmark maintains its challenge and long-term relevance?\n\n7. Limitations of the Object-Centric Paradigm\nThe current \"object-centric\" paradigm appears inherently limited in its ability to evaluate global attribute editing (e.g., style transfer, mood change) or effectively handle highly ambiguous or creative instructions. Does this imply a fundamental limitation of the framework in assessing more complex and artistic editing tasks? What potential extensions or modifications could be implemented in the future to broaden its scope?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1）Automated Evaluation: The proposed EdiVal-Agent introduces a complete, automated pipeline that systematically tackles multi-turn editing evaluation. It decomposes the process into three core stages: image decomposition, instruction generation, and hybrid assessment.\n2）Hybrid Assessment Methodology: A tool-specific hybrid strategy is employed for evaluation. This methodology efficiently handles different task types: symbolic tasks (e.g., object detection, counting) are addressed by a specialized object detector; semantic tasks (e.g., color, material changes) are evaluated by a Visual Language Model (VLM) on localized image crops; and overall visual quality is assessed by a dedicated human preference model.\n3）Human Alignment: Validation through a substantial human study, comprising 4,576 human judgments, demonstrates the efficacy of the framework. The core metric, EdiVal-IF, achieves an impressive 81.3% agreement with human judgments, significantly outperforming compared baseline methods.\n4）Multi-Turn Editing Benchmark: The framework is instantiated as the EdiVal-Bench benchmark. This structured testbed uniquely features multi-turn interaction, an object-centric design, and reference-free evaluation, offering a robust standard for assessing multi-turn editing capabilities.", "weaknesses": "1. Limitations in the Benchmark Scale and Data Diversity\n--Homogeneous Data Sources: The benchmark relies solely on 572 real-world images. This high level of homogeneity lacks adequate evaluation coverage of synthetic or generated images, restricting its applicability to emerging models and research directions.\n--Opaque Scene Coverage: The paper provides no clear specification of image selection criteria, scene distribution characteristics, or quantitative diversity metrics. This lack of transparency makes the benchmark's scene coverage unverifiable.\n--Insufficient Complexity for Long-Sequence Tasks: The default 3-turn editing setup is inadequate. It fails to sufficiently test model robustness and coherence in demanding, long-sequence editing scenarios.\n--Incomplete Editing Taxonomy: The framework is limited to 9 basic edit types, omitting several essential operations critical for real-world editing, such as object zooming, lighting and environmental adjustments, image cropping, special effects (addition/removal), object duplication, and object rotation.\n\n2. Inherent Biases in Evaluation Methodology\n--VLM Dependency Risk: The core image decomposition phase is entirely dependent on proprietary models like GPT-4o. The paper offers no quantitative analysis of how performance would be affected by using different or open-source Visual Language Models, creating significant dependency and reproducibility risks.\n--Artificial Framework Advantage: While correctly criticizing the imprecision of VLM-only evaluation, the method's claimed \"fine-grained\" superiority is derived from a manually engineered evaluation framework, not a fundamental methodological breakthrough that generalizes beyond its custom design.\n\n3. System Architecture and Practicality Deficiencies\n--Misleading \"Agent\" Characterization: The system's fixed workflow and rule-based decision processes fundamentally lack the autonomous reasoning and adaptive capability expected of a true intelligent agent, rendering the \"Agent\" designation misleading.\n\n4. Superficial Problem Diagnosis and Limited Utility\n--Solution Gap: The work identifies existing issues, such as exposure bias, without proposing effective solutions or mitigation strategies, offering a diagnosis but no actionable remedy.\n--Limited Guidance Value: The evaluation findings fail to translate into concrete, targeted suggestions for improving model architectures or training protocols, thus diminishing the framework's practical guidance value for future research.\n\n5. Incomplete Evaluation Dimensions\n--Blind Spots for Global Editing: The object-centric evaluation paradigm cannot effectively assess crucial non-object-level capabilities, including global attribute editing, style transfer, and overall scene mood alteration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ee7Twd0toI", "reviewer": "Reviewer_nxo4", "summary": "This paper tackles a big headache in AI image editing: how do we actually know if multi-turn edit is good? Current methods are flawed, so the authors built EdiVal-Agent, a smart and automated evaluation system. In a nutshell, the agent looks at an image, breaks it down into individual objects, and then generates a series of editing instructions, like a conversation.\n\nIt then judges the edited image using three clever new metrics. EdiVal-IF checks if the AI followed the instructions correctly, using a combo of object detectors and vision models. EdiVal-CC checks if the AI messed up the background or other objects it wasn't supposed to touch. Finally, EdiVal-VQ scores the final image's aesthetic quality. Using this agent, they created a benchmark called EdiVal-Bench to test 13 top models, revealing who's best at complex, multi-step edits.", "questions": "1. The framework is awesome for object-focused edits, but what about creative styles? How would you adapt EdiVal-Agent to reliably evaluate a prompt like \"make this photo look like a van Gogh painting,\" where there isn't a single correct answer?\n\n2. You mentioned that some models suffer from \"exposure bias\" when editing their own outputs over multiple turns. Do you think this problem could be solved by training these models specifically on multi-turn generative data, or is it a more fundamental flaw in their architecture?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Object-Centric view, instead of just vaguely looking at the whole picture, the framework focuses on specific objects. This allows it to check with much higher precision whether \"the brown horse\" was really changed to \"a deer\" while leaving the other horses alone.\n2. Built for Multi-Turn Editing. It's one of the first benchmarks designed specifically for conversational, back-and-forth editing. This is much closer to how people actually use these tools, rather than just giving a single, one-off command.\n3. It doesn't just rely on one tool. It smartly combines object detectors with Vision-Language Models and human preference models (for aesthetics). This hybrid approach is more robust and aligns better with human judgment.\n4. Fully Automated Pipeline. the agent can create the entire benchmark on its own—from analyzing the image to generating diverse instructions. This makes it super scalable and easy to expand in the future.", "weaknesses": "1. This work only support limited instruction types. The agent is a pro at object-based edits (add, remove, change color, etc.), but it can't handle abstract or stylistic requests like \"make it more dramatic\" or \"change the style to cyberpunk.\" The paper acknowledges this limitation.\n2. The entire system is a chain, and it's only as strong as its weakest link. If the object detector (Grounding-DINO) fails to spot an object correctly, the evaluation for that step will be wrong. Its accuracy is capped by the performance of the tools it uses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FLKSUEUBGT", "reviewer": "Reviewer_hVAS", "summary": "The paper proposes EdiVal-Agent, an automated, object-centric framework for evaluating instruction-based image editing, including multi-turn scenarios. The agent (i) decomposes an input image into semantically meaningful objects, (ii) generates diverse, context-aware editing instructions across turns while tracking object pools, and (iii) evaluates model outputs with three metrics: EdiVal-IF (instruction following via symbolic checks with open-vocabulary detection plus VLM verification on crops), EdiVal-CC (content consistency for unchanged objects and background with DINO features / L1), and EdiVal-VQ (visual quality via HPSv3). Using this pipeline, the authors build EdiVal-Bench (572 images; 1,716 instructions; 9 types; 3 turns) and benchmark 13 editors spanning diffusion, flow-matching, and in-context AR systems. Key findings include 81.3% human-agreement for EdiVal-IF vs. 75.2% for a strong VLM baseline (Fig. 4, p.6), and clear trade-offs among systems (e.g., Seedream 4.0 best overall EdiVal-O; GPT-Image-1 strong IF but weaker consistency and high latency; Table 3, p.7).", "questions": "Detector robustness: Have you tested alternative detectors (e.g., GLIP, OWL-ViT, recent Grounding-DINO variants) and/or ensembles to reduce false positives observed in Fig. 11? How do EdiVal-IF accuracies shift? \nThreshold sensitivity: How sensitive are EdiVal-IF decisions to the 0.3–0.4 detection thresholds and box-size filters (Appendix I.1, p.20)? Could you provide a calibration curve or confidence-weighted IF score? \nStyle edits: If you added a bounded \"style change\" category (e.g., limited palette/style banks) with reference-free checks, what metric would you recommend? Any preliminary experiments? (Design Scope, p.6.) \nCounting verification: For count change, did you consider tracking-based counting across turns or density estimators to complement detection counts (Table 10 & Appendix L)? \nTool-swap ablation: Can you include a tool-agnostic study (swap VLM, detector, feature backbone) to quantify portability and establish guidance for communities without access to the default stack? (Sec. 2.4 & Appendix I.) \nAggregation design: Why geometric mean for EdiVal-O (p.6)? Did you compare with harmonic mean or a dominance/Pareto analysis to discourage gaming one axis?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clear problem framing: Multi-turn editing evaluation is under-served; the object-centric perspective addresses spatial grounding, subtle changes, and compositionality (Sec. 1–2). \nMethodological novelty: EdiVal-IF combines symbolic checks (presence/absence, spatial relations, counts) with semantic VLM verification on detector-guided crops, improving precision and interpretability (Sec. 2.4; Eqs. (1)–(2), p.5). \nHuman agreement: 81.3% on IF vs. 75.2% VLM-only; inter-annotator agreement 85.5% bounds achievable performance (Fig. 4, p.6). \nComprehensive benchmark & analysis: 13 editors, 9 instruction types, 3 turns; rich breakdowns (per-task success, consistency via DINO/L1, visual quality HPSv3, exposure drift analysis with luminance p99/p999; Tables 3–4, 8–19; Figs. 5–9, 12–14). \nTransparency & reproducibility: Full prompts, thresholds, and pseudo-code (Appendix I), with limitations and failure cases (Appendix E). \nActionable insights: Identifies exposure bias in multi-turn (Fig. 5, p.7; Fig. 12, p.18), weaknesses in spatial/counting tasks (Figs. 6, 15), and model-family trade-offs between beautification vs. preservation (Fig. 3, p.5).", "weaknesses": "Detector dependence / error propagation: EdiVal-IF's symbolic branch hinges on Grounding-DINO; false positives/negatives can flip outcomes (failure case in Fig. 11, p.17). A more robust ensemble or confidence-calibrated aggregation could mitigate this. \nLimited edit taxonomy: Style change is excluded (Design Scope, p.6), which reduces coverage for prevalent real-world edits; future extensions are suggested but not explored. \nTool ablations & sensitivity: While the agent is \"tool-agnostic\", results rely on specific choices (Qwen-2.5-VL, DINOv3, HPSv3). A systematic tool swap study (e.g., different open-vocab detectors/VLMs/features) and threshold sensitivity analysis would strengthen claims of generality. \nCounting & spatial reasoning remain brittle: Even top systems struggle on count change (<25% at T1) and position change (Table 10, p.29). The paper diagnoses this but does not propose evaluator improvements specialized for numeracy/spatial relation verification beyond center-based rules. \nQuality metric choice: HPSv3 is reasonable, but the decision to report separately (not integrated) may complicate a single headline score; some readers may want a tunable composite with explicit weights or Pareto front visualizations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lSeDVzLrtq", "reviewer": "Reviewer_TgZ8", "summary": "The paper introduces EdiVal-Agent, an object-centric, multi-turn framework for evaluating instruction-based image editing. It generates scene-aware instructions and scores edits with three metrics—EdiVal-IF (instruction following via detector+VLM), EdiVal-CC (content consistency for unchanged regions), and EdiVal-VQ (visual quality)—and releases EdiVal-Bench across 9 instruction types and 13 editors.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Clear problem framing & object-centric design: The paper crisply motivates why existing reference-based and VLM-only evaluation is insufficient and positions an object-centric alternative that combines symbolic checks with semantic reasoning.\n- Well-specified agentic pipeline with dynamic object pools: The decomposition → instruction generation → evaluation workflow is precise.", "weaknesses": "- Tool-dependence and potential evaluator bias: The pipeline fixes Grounding-DINO as the detector, Qwen-2.5-VL as the VLM, and DINOv3/HPSv3 as backbones for CC/VQ. While the authors say the pipeline is tool-agnostic, the reported results (e.g., human agreement advantage) are contingent on this specific stack. Please quantify sensitivity to swapping components (e.g., different detectors/VLMs), or report ablations where each tool is replaced by a strong alternative.\n- Metric coupling between IF and CC is under-analyzed: The geometric mean \\sqrt{\\text{IF}\\cdot \\text{CC}} is reasonable, but different applications may value preservation much more than instruction compliance (or vice versa). The choice is referenced to prior work but lacks a robustness study: how often do model rankings flip under other aggregations (harmonic/weighted means)?\n- Fairness of latency comparison: The table mixes closed-source UI latency with open-source single-GPU latency, which is not directly comparable (unknown batch sizes, backend scaling, caching). The paper acknowledges measurement contexts but still draws speed–quality conclusions. Consider removing cross-setting latency comparisons or normalizing via a controlled local runtime for open models and API budgeted throughput for closed models.\n- Multi-turn vs. single-shot comparison design: The conclusion that multi-turn can help when exposure bias is low is plausible, but details on prompt concatenation for single-shot “complex” edits matter (ordering, connectors, explicit constraints like “make X unchanged”). A stricter prompt engineering baseline for single-shot could narrow gaps; please share exact templates and run ablations with stronger single-shot compositions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)