\texttt{Complex-Edit}: CoT-Like Instruction Generation for Complexity-Controllable Image Editing Benchmark
Authors: Siwei Yang
Mude Hui
Bingchen Zhao
Yuyin Zhou
Nataniel Ruiz
Cihang Xie
Venue: Accepted by TMLR ()

Abstract: We introduce Complex-Edit, a comprehensive benchmark designed to systematically evaluate instruction-based image editing models across instructions of varying complexity. To develop this benchmark, we harness GPT-4o to automatically collect a diverse set of editing instructions at scale. Our approach follows a well-structured "Chain-of-Edit" pipeline: we first generate individual atomic editing tasks independently and then integrate them to form cohesive, complex instructions. Additionally, we introduce a suite of metrics to assess various aspects of editing performance, along with a VLM-based auto-evaluation pipeline that supports large-scale assessments. Our benchmark yields several notable insights: 1) Open-source models significantly underperform relative to proprietary, closed-source models, with the performance gap widening as instruction complexity increases; 2) Increased instructional complexity primarily impairs the models’ ability to retain key elements from the input images; 3) Stronger models aren't necessarily more resilient towards higher complexity; 4) Decomposing a complex instruction into a sequence of atomic steps, executed in a step-by-step manner, substantially degrades performance across multiple metrics; 5) A straightforward Best-of-N selection strategy improves results for both direct editing and the step-by-step sequential approach; and 6) We observe a "curse of synthetic data": when synthetic data is involved in model training, the edited images from such models tend to appear increasingly synthetic as the complexity of the editing instructions rises --- a phenomenon that intriguingly also manifests in the latest GPT-Image-1's outputs. The code for evaluation and data generation, and the test set is released at https://github.com/UCSC-VLAA/Complex-Edit.

Published in Transactions on Machine Learning Research (03/2026)
Complex-Edit: CoT-Like Instruction Generation for Complexity-Controllbbie Image Editing Benchmark
Siwei Yang
syong217/8ucsc.euu
Computer Science and Engineering Department, UC Santa Cruz
Mude Hui
muhuií@ucsc.edi
Computer Science and Engineering Department, UC Santa Cruz
Bingchen Zhao
Singche..haooeed..coo ak.uk
School of Informatics, University of Edinburgh
Yuyin Zhou
ychou284@ussc.comm edu
Computer Science and Enginecring Department, UC Santa Cruz
Nataniel Ruiz
natonielraizz@google.com
DeepMind, Google
Cihang Xie
cirie edu
Computer Science and Engineering Department, UC Santa Cruz
Reviewed on OpenReview: https: // optpr/veeeeeeee... net/ forum? LL1JR6daG8
Abstract
We roduce Complex-Edit, a comprehensive benchmark designed to systematically evalu- ate instruction-based image editing models across instructions of varying complexity. To develop this benchmark, we harness GPT-4o to automatically collect a diverse set of editing instructions scale. Our approach follows well-structured "Chain-of-Edie pipeline: we first generate individual atomic editing tasks independently and then integrate them to form cohesive, complex instructions. Additionally, aspects of editing performance, along with a VLM-based aulo-evaluation pipeline that sup- ports large-scale assessments. Our benchmark yields several notable insights: 1) Open-source models significantly underporform relative to proprictary, closed-source models, with the performance gap widening as instruction complexity increases; 2) Increased instructional com- plexity primarily impairs the models' ability to retain key elements from the input images; 3) Stronger models aren't 't necessarily more resilient towards higher complexity; 4) Decomposing complex instruction into a sequence of atomic steps, executed in a step-by-step manner, substantiall) degrades performance across multiple metries; 5) A straightforward Best-of-N selection strategy improves results for both direct editing and the step-by by-stpp sequential approach; and 6) We obsorve "curse of synthetic data": when synthetic data is involved in model training, the edited images from such models tend to appear increasingly synthetic as the complexity of the editing instructions rises - a phenomenon that intriguingly also manifests in the latest GPT-Image-15s outputs. The code for evaluation and data generation, and the test set is released at httts://github.comm WUC.oO//USSCCCAAACC Ca..peeeeAAoooleee....tttt
introduce suite of metrics assess various
1 Introduction
Since InstructPix2Pix (Brooks al.) 2023) introduced instruction-basod image teating-wweert images are directly modified through textual commands the field has witnessed remarkable progress. Noteworthy developments include enhancements in the quality of training images (Kawar al. 2023 Zhang et al.

---

Published in Transactions on Machine Learning Research (03/2026)
Complexiy Level
Complexit Leve
Companiiy Leve
Conoeesio Level
Replece with renone the bicele and change wlote
taan rawove the benylle, not thon betiinnnss night when
naning change actting
Reslace the rig with Pole
the
Seplce drg wilhe
ber by snntion two spereies resting the ferce
Figure 1: An illustration of our Complex-Edie Exit.... This figure presents a progression of instruction complexity in image editing tasks, highlighting the transition from atomic edits to highly intricate transformations.
Stage #1:
Stage #2:
Stage #3:
Sequence Generation
Simptification
Inseruction Compounding
Atomcc Instructions
Simplified Atomic Instructions Add snall ball of yenr.
Compound Instructions
email of yorr.
Add shall bull Add small bell rad
Crange the color of The yem ned.
Change color the to
dlow
soft around cats erghis ae the far
saft glow aroand
yans.
o soft around Then. onerge the beckground to ceny' livin room.
Reploce bocignound
Repiace the bockgrsund cary licing
Figure 2: An overview of our data collection pipeline. The pipeline consists of three distinct stages: 1) Stage #1 Sequence Generation: for each image, a series of atomic instructions is produced; 2) Stage #2 Simplification: each fundamental instruction is refined to eliminate extraneous details, preserving only the essential description of the editing process; 3) Stage #3 Instruction Compounding: several atomic instructions integrated into one comprehensive instruction.
2024 Hui et al. 2025 Zhao el al. 2024 Wei el al. 2025 Baldridge et al. 2024), the evolution of model
architectures from based to transformer based diffusion models (Ma et al. (024), the refinement of
training methodologics (Simsar et al. 2024 Shi ct al. 2024), and the on-going exploration of test-time scaling
[Guo et al. 2025 Ma et al. 2025
Despite these strides, much of the available training data and evaluation benchmarks rely on relatively simple editing instructions (Brooks et al. 2023 Zhang et al. 2023 2024 Zhao et al. 2024 Yu et al. 2024 Wei et al.
2025 Sheynin et al. 2023; Ku et al. 2024b). Yet, world scenarios often require handling instructions that vary considerably in complexity. For instance, while one user might simply request to "remove the car", another might need a more elaborate transformation, such as "replace the car with a blue bus featuring an iPhone advertisement on its side". The current lack of ovaluation benchmarks capable of handling such diverse instruction complexities not only impedes our ability to rigorously assess existing editing models but also hampers the evaluation of emerging, more powerful systems (Guo et al., 2025 Ma et al. 2025).
To bridge this gap, present a novel data generation pipeline that leverages advanced GPT models to create diverse, scalable, and cemplexity-cootrollabb evaluation dataset for image editing (see Fig, 2 for an illustration). Our approach follows a chain-of-thought-le three key stages. First, in the Sequence Generation stage, an input image prompts GPT-4o to generate a series of simple instructions corresponding to predefined atomic operations (e.g., "Add an Object", "Change the Object Color"), with each serving as an intermediate step toward more complex oditing task. Next, /ppeffuous/unceessaary
(Wei et al. 2022) paradigm, unfolding in
recognizing that these GPT-generated atomic instructions may contain superfluous/ unnecessary details such added commentary the operation's 's intention we then move to Simplification stage, where each
2

---

Published in Transactions on Machine Learning Research (03/2026)
instruction is trimmed to retain only its core editing intent. In this final Instruction Compounding stage, the simplified atomic instructions are merged into single, coherent complex instruction (again, via GPT-4o), using the input image as contextual guidance. We name this new dataset Complex-Edit. Importantly, this structured process of creating Complex-Edis allows us to quantitatively control the editing instruction's 's complexity defined as the number of atomic operations a single instruction - by simply adjusting the number of merged atomic instructions. This benchmark directly scales the semantic complexity of the task: the number of atomic operations increases, the instruction inherently compels the model process denser linguistic information and resolve increasingly intricate dependencies between visual elements. We further discuss the theoretical positioning of this compositional complexity within larger hierarchy of complexity types for image editing in Appendix Sec.
Object Manissation and Trans
To evaluate performance on these complex instructions, we developed a comprehensive evaluation framework that measures three critical dimensions: (1) Instruction Following, which as- sesses whether the intended modification ions are correctly applied; (2) Identity Preservation, which ensures that unspecified ele- ments remain unchanged; and (3) Perceptual Quality, which evaluates the overall aesthetic quality and absence of artifacts. To support large-scale evaluations, we implement a vision lan- guage model (VLM))based autograding system to quantify these dimensions. Our investigation further identifies important con- siderations in enhancing this evaluation pipeline. For example, enabling chain-of-thought reasoning in VLM cvaluations does not necessarily enhance evaluation quality contrary to previ- ous findings (Ku et al. 2024a Hui et al. 2025) while providing detailed rubrics and using direct numeric scoring (Xie et al.
A
Pazz Expreesion onn
Testuur 2012
Special Erieets
backgrronng grownd Invernmment
Text Symbols
Con pouttioi
and cropping
Lighting
Shaddws
The distribution of atomic instruc- tions among 9 operation categories. The are thickness between two categories shows the nunber of adjacent instructions from these two categories.
Figure
2023: 2024 Zhou et al., 2024 Cui et al. 2024h consistently improves autograder performance.
Our extensive evaluation results reveal five key insights that were previously difficult to capture with existing benchmarks. First, open-source models significantly underperform compared to proprietary, closed-source models, with this disparity growing as editing complexity increases. Second, the increasing instruction complexity mainly affects editing models' identity presorvation, while its impact on instruction following varies across models. Thirdly, more capable models do not. reliably exhibit increased resilience to the negative effects of the higher instruction complexity. Fourth, unlike the benefits observed with chain-of-thought (CoT) reasoning in text generation, CoT-inspired sequential editing yields much worse results than directly executing complex instructions significantly degrades outputs in instruction following, identity preservation, and perceptual quality. Fifth, simple techniques such as Best-of-N sampling improve direct editing outcomes across all three dimensions and, when applied to sequential editing, notably enhance identily preservation and percept ual quality. Lastly, our Complex-Ede eepiit reveals an emerging 'curse of synthetic data' - when training utilizes synthetic images, models tend to produce odited outputs with overly synthetic qualitics under complex instructions, often resembling oil paintings or animations. More interestingly, this phenomenon is also observed with the latest GPT-Inage-l, potentially suggesting that the incorporation of synthetic data may contribute to GPT-Image-1's advanced image generation capabilities.
note this is because sequential editing
We hope this newly developed benchmark, Complex-Edss will not only deepen the understanding of instruction-based image editing models but also serve as a valuable framework for the rigorous evaluation of next-generation image editing systems, particularly those with test-time scalability.
2 Collection of Complex-Edit
As illustrated in Fig. 2 the collection of Complex-Edit is organizod into three stages: 1) Stage #1 Sequence Generation: sequence of atomic instructions are generated for each image; 2) Stage #2 Simplification:
3

---

Published in Transactions on Machine Learning Research (03/2026)
Object Manipulation and Transformation
Reploce
Color and Tone Adjastmants
Texture and Material Adjustments
Background and Environment
Lighting and Shadawi
Change Color
Appy Fiter/Weother
Change Tixtture
Change Background
Adjosl Lighting
Text and Symbols
Composition and Cropping
Propartios
Meow
Grop Irsge
Composil
Pose Expression
Special Effects
Remeve Spedial
Remove Particies
Effects
Add Pur licks
Add Speciel
Change
Charge Expression
Figure 4: An illustration of lypes of atomic editing operations 9 categories.
each atomic instruction is simplified to remove unnecessary information other than the description of the editing operation; 3) Stage #3 Instruction Compounding: multiple atomic instructions combined into single, complex instruction.
2.1 Sequence Generation
In this first stage, define 24 distinct atomic operations that represent the most basic actions of image editing, which are grouped into categories Object Manipulation and Transformation, Color and Tone Adjustments, Texture and Material Adjustments, Background and Enwironment, Lighting and Shadows, Text and Symbols, Composition and Cropping, Pose and Expression, Special Effects each of which captures a unique aspect image editing, as shown in Fig.
We provide detailed descriptions each operation type the Sec. These categorized atomic operation types and their descriptions, along with an input image, are fed into GPT-4o to generate sequence of atomic instructions with pre-determined length. Each generated instruction is associated with of the predefined atomic operations. We also record the atomic operation type for each instruction, with its distribution illustrated in Fig. 3 This metadata can facilitate the future development of specialized image editing models.
2.2 Simplification
In practice, these GPT-generated instructions sometimes are unnecessarily rich in detail and may include extrancous commentary on the intentions of the editing operations, as shown in Fig. Becanse our benchmark is designed to evaluate the performance of image-editing models on clear and concise instructions, employ a dedicated simplification stage to climinate any superfluous content. Specifically, each generated atomic instruction examined by GPT-4o to determine whether it requires further simplification; so, GPT-4o outputs a simplified version adhering to the predefined response format, and the original instruction is replaced accordingly.
2.3 Instruction Compounding
Lastly, simplified atomic instructions are combined into compound instructions. Specifically, given a sequence of N atomic instructions, we progressively generate N compound instructions corresponding to different complexity levels: compound instruction at complexity level C₁ is formed by combining the first i atomic instructions in the sequence, with the simplest level (Ci) being identical to the first atomic instruction and the hardest level (Cx) integrating all N atomic instructions.
implementation, rather than simply concatenating the instructions, atomic instructions are seamlessly intograted, which allows potential reordering or merging to produce to make the final outcome more natural and coherent. For example, in Fig. 2 rather than sequentially executing "add ball of yarn" and "change the color of the yarn to red". the instructions compounded into single directive: "add red ball of yarn".
4

---

Published in Transactions on Machine Learning Research (03/2026)
Table 1: Random sampling vs Discriminative sampling.
Table 2:
Results from different scoring meth- ods. We can observe that w/ COT and w/
Deterministic
Correlatin
1F
IP
PQ
Rubric are the best for IF and IP, and w/o COT and w/ rubric are the best for PQ.
0.468
0.530
0.234
0.461
0.507
0.266
Cocrelation
Scoring Method
CUT
Rubrie
Instruction Following
Identity Preservation
Parceptual Quality
1F
IP
PQ
Anolicaia
imsgen3
SoudEd
imagena SocdEdit
Imagen
SoudEdRt
Tokenn Prob
0.111
0.366
0.136
Tokenn Prob
0.447
0.460
0.158
Numerie
0.434
0.450
0.207
Numeric
0.448
0.451
Numerie
0.468
0,530
0.208
CLIPair
CLI in
Figure 5: Relationship between per-sample variance
0.182
0.523
Instruction To fowwing Identily Preservition,
Perceptusl Quality
Dverall
Instruction Following
identity Preservation
Perceptual Qualiyy
Overall
odneet on Conondary
Inemuction Complecky
CometienComppee
Poruccion Complesty
mdmmction Comolndtyy
Inrurssnn Complech
Peruetion Complecyy
inemetion Comp
(hrtee [aquertaa
DeeSBnn (oopectall
AnyDoR
Iragest
Send-a
MENDOR Iequeeta
Anda
Inepe13
Sea Deqaetaa
Ortivege beggetia
Degaecab
Dequetat
(togests)
(a) Real input images.
(b) Synthetic input images.
Figure 6: Evaluation results of direct and sequential editing. The instruction complexity level ranges from C to Cg. This indicates that IP and PQ consistently dropped as the instruction complexity level grows, while it influence IF varies among different models. Also, the performance gap between models is enhanced with increasing complexity.
2.4 Implementation Details
To ensure that GPT-4o fully understands our objectives, we carefully construct multiple few-shot examples for each of the three stages. Additionally, CoT reasoning (Wei et al.] 2022) is enabled in both Stage (Sequence Generation) and Stage #3 (Instruction Compounding) to enhance generation quality. Furthermore, to improve data diversity, we slightly increase the sampling temperature from 1.0 to 1.15 during Stage #1 (Sequence Generation). Although this adjustment promotes diversity, it also increases the likelihood of generating garbled text. To address this, we implement a rule-based filtration mechanism at each stage that detects and regenerates any flawed outputs.
3 Evaluation
Following prior works (Ku et al. 2024a Hui al., 2025), we employ VLMM-ased autograders to facilitate large-scale evaluation. However, have observed that these frameworks do not fully capture the nuances of instructoon-uuide image editing. In response, we introduce several enhancements to address these limitations and improve the overall autograding framework.
3.1 Metric Design
Our evaluation framework focuses on two primary dimensions: 1) the alignment dimension whether the output image reffects the changes specified by the editing instruction, and 2) the perceptual quality dimension whether the output. image looks aesthetically pleasing and is devoid of visual artifacts.
3.1.1 Alignment
Unlike HQ-Edit (Hui et al. 2025 which simply measures alignment as single metric, hereby decompose it
into two complementary criteria: 1)Instruction Following (TF) measures whether the specified modifications are present in the output image; 2) 2)ddentity Preservation (IP) assesses whether elements of the input image that should remain unchanged are indeed preserved.
We would like to point out that these two criteria roughly correspond to the directional CLIP similarity (CLIP d and image- wise CLIP similarity (CLIP (CLIPing) metrics used in earlier studies (Brooks et al.) 2023
Zhang et al. 2023 2024h. When accessing IF and IP, feed the VLM with a triplet containing (inpul
5

---

Published in Transactions on Machine Learning Research (03/2026)
image, output image, and editing instruction). Appendix Fig. 16
We provide series of example results of IF and IP in
3.1.2 Perceptual Quality
Beyond alignment, we also evaluate the overall visual quality of the generated images. Specifically, our Perceptual Quality (PQ) metric examines factors such as consistency in lighting and shadows, style coherence, and the seamless integration of elements. As users may sometimes request bizarre or non-aesthetic edits, such as "add motion blur to these vehicles" or "enlarge the pet match the height of the owner", one might argue that it is essential to include editing instructions when evaluating perceptual quality. However, discussed in Sec. 4.1.4 empirical results indicate that providing editing instructions to VLMs actually reduces the correlation between VLM and human evaluations. Consequently, for PQ evaluation, we only supply the VLM autograder with the edited image. Sample PQ results are provided in Appendix Fig. 17
3.1.3 Overall Score
Lastly, we define the overall score, O, simply as the arithmetic mean of the three metrics to summarize performance: O I1++P+PP
3
3.2 Metric Calculation
3.2.1
Numeric Scoring v.s. Token Probability
In line with VTEScore and HQ-Edit, instruct the VLM to assign metric. We refer to this method numeric scoring. In addition, explore another approach for metric computation: token probalility, which is widely used in test-time scaling and VLM post-training (Xie) et al. 2023 2024 Zhou et al. 2024 Cui et al.) 2024).. Specifically, rather than directly asking the VLM for a numerical score, we reformulate the evaluation as a binary classification task by posing yes-or-no question to the VLM, e.g., "Do the specified changes appear in the output image?". The score is then calculated as the normalized token probability for the affirmative response "Yes" using Probyes/ Probyas(Probye. Probxo).
between 0 and 10 for each
3.2.2 Detailed Rubric
In contrast to existing frameworks like VIEScore and HQ-Edit, we design comprchensive rubrics for cach metric to guide the evaluation process and improve interpretability. Detailed descriptions of these rubrics are provided in the Sec. C.1 We discuss the impact of these rubrics and different scoring approaches in Sec. 4.1.2
3.3 Par-se Variance
Previous evaluations of image-editing models using advanced proprietary VLMs, such as GPT-4o (Hurst et al. 2024) and Gemini [Team et al., 2023 2024), have encountered issues with score stability, leading to score variations for individual samples across different runs. Although averaging over many samples generally mitigates these discrepancics at the datasct level, per-sample stability crucial for test-time scaling, where
accurale and consistent evaluation is required for each individual sample (Guo et al.) 2025 Ma et. al. 2025
A straightforward approach to enhance stability utilizes multiple measurements to average the resultant scores, though this approach may considerably escalate computational costs. Alternatively, one can attempt to make VLM outputs more deterministic by using greedy random sampling techniques (e.g., by setting the sampling probability mass to an extremely low value such as 1e-7). However, empirically, this deterministic approach slightly lower the correlation with human evaluations as shown in Tab. [ Consequently, we adoptod the first approach by default, utilizing the average score from 20 independent evaluations for each sample. Further discussion reducing per-sample variance is included in Sec. 4.2

---

Published in Transactions on Machine Learning Research (03/2026)
Table 3: Evaluation results with direct editing on real images with the instruction complexity at Cg.
Model
Prop. of Real Images
Photorealism
IF
IP
PQ
UktraEdit
6.56
5.93
7.29
6.39
OmniGen
6.25
6.42
7.54
6.74
AnyEdit
1.60
8.15
7.25
5.67
SoidEdit
8.49
6.91
8.74
8.04
Imagen3
7.56
6.55
7.67
7.26
GPT Image
9.61
8.06
9.39
9.02
Input
UltraEdit
OmniGen
AnyEdit
Imagen3
SeedEdit
GPT-Image-1
Receuie bedtroor mndernting www.ggo pray parart nuslicingg rncooder
CUTER SLSSSII ome Ado
Figure 7: A real image edited with Cs instruction. Some models' output lose the photorcalism completely.
Figure 8: Roal images edited with Cs instructions by GPT-Imago-1. Outputs from GPT-Image-1 severely lose the photorealism. See additional results Appendix Fig. 23
4 Meta-Evaluation for Metrics
In this section, we detail our meta-evaluation of the VLM-based auto-evaluation framework, focusing on design choices and their effectiveness in assessing image editing model outputs. We expect effective auto-evaluation pipeline to satisfy two core criteria: 1) High correlation with human judgments, and 2) Low variance for ensuring reproducibility. Next, we first examine how various design decisions impact correlation with human evaluations (Sec. 4.1 and then describe our strategies for reducing per-sample variance (Sec. 4.2
4.1
Correlation with Human Evaluation
4.1.1
Meta-Evaluation Settings
Meta-ovaluation for instruction-based image editing metrics (Ku et al. 2024a Hui et al. 2025 is typically performed by correlating human-assigned scores with metric values. However, found that expecting human raters to assign consistent numeric scores across diverse samples is challenging. Instead, for each input image and editing instruction, we present raters with a pair of output images and ask them to compare the outputs with respect to each metric. We then compute the correlation between these human comparisons and the differences in the corresponding metric scores. Additionally, we ask raters to choose their preferred out based on overall impression. This comparative approach allows us align our metric aggregation with human intuition and to determine the best way to combine the three proposed metrics into single score.
To ensure that our meta-evaluations generalize across varying levels of instruction complexity, sampled 100 output images for cach complexity level from C1 (f.e., the simplest level, just single atomic instruction) to C& (f.e., the hardest level, compounding from 8 alomic instructions) using both Imagen3 (Baldridge el al., 2024) and SeedEdit (Shi et al.) 2024, resulting 800 image pairs. Each pair was annotated independently by least two human raters. We use GPT-4o as our VLM evaluator for the meta-ovaluation study. Since the par-sample variance is yet to be discussed in Sec. 4.2 for now, we naively evaluate each sample 40 times and average the scores to reduce the per-sample variance.

---

Published in Transactions on Machine Learning Research (03/2026)
4.1.2 Scoring Method and Rubric
As shown in Tab. experiments indicate that numeric scoring yields higher correlation with human evaluation than token probability methods. Moreover, adding detailed rubric to numeric scoring enhances this correlation consistently, highlighting the need for clear evaluation guidelines.
4.1.3 Chain-of- Thought
We additionally investigate the impact of CoT on our evaluation by adding an explicit instruction (e.g., "explain your reasoning before answering the question") to encourage the VLM evaluator to articulate its reasoning prior to delivering the final score. The complete prompt for evaluation is provided in Appendix Sec. C.2 However, as shown in Tab. 2 our results indicate that CoT fails to enhance metric correlation consistently. In particular, CoT negatively affects the correlation for Perceptual Quality PQ when using numeric scoring. Consequently, we utilize numeric scoring with detailed rubrics and CoT only for Instruction Following TF and Identity Preservation IP, while CoT disabled during Perceptual Quality PQ evaluation.
4.1.4 Instruction Input for Perceptual Quality
One might initially hypothesize that including the editing instructions alongside the output image could help the VLM discount acsthetically inferior modifications that are explicitly required by the instructions during Perceptual Quality PQ evaluation. However, our experiments reveal that providing the editing instructions significantly lowers the correlation with human evaluations from 0.234 0.046. Consequently, we exclude the editing instructions when evaluating Perceptual Quality PQ, ensuring that the VLM's assessments of image quality better align with human perception.
4.1.5 Different Averaging Formula for Overall Score
Previous studies Ku et al. 2024a Wei et al. 2025) advocate using the geometric mean to combine individual metrics to penalize low scores. In our case, switching from the geometric mean the arithmetic mean resulted in modest correlation increase from 0.334 0.386. Therefore, we compute the overall score as the arithmetic mean of the individual metries.
4.1.6 Comparison with CLIP Scores
We further compare the correlation Directional CLIP Score CLIP die and Image-wise CLIP Score CLIP img with human evaluation, as these scores similarly assess aspects of Instruction Following IF and Identity Preservation IP. As reported in Tab. g these CLIP scores exhibit lower correlations with human evaluations compared to metrics.
4.2 Pam-le Variance
As discussod Sec. 3.3 one way to reduce per-sample variance is set the probability mass for sampling extremely small value, e.g., le-7, thereby approximating the determinism of proprietary VLMs. However, shown in Tab. [ this approach slightly diminishes the correlation with human evaluations. Consequently, we disable deterministic sampling and instead perform multiple evaluations per sample, using the average score as the final metric.
To determine the required number of measurements per sample, we study the relationship between the number of measurements per sample and the variation in final scores across independent runs. Specifically, given measurements for one sample, we compute average scores four times, each averaged from K distinct measure- ments, denoted as S,, S2, S3, S4. The variation for this sample defined as max(S; S3. min min(mnnnn S8..8)
Fig. 5 shows that the variation amplitudes for all three metries converge when K 20. Therefore, we adopt 20 measurements per sample in all subsequent experiments.
8

---

Published in Transactions on Machine Learning Research (03/2026)
Input
C2
C4
OmniGen
Seed-Edit
C2 Replace teclgrrund with nustic wooden
change
Plaae
texture wooden
reducee parsley
C8 Replace cining backgground with dish, fcened liber ambiance.
next
bottle
paperr cenamic pizza evertep
aod
garnish
introduce effects
(a) Real input images.
Input
C2
C4
C6
C8
UltraEdit
Imagen3
Replace baciground pinnic patreckth setingg add chrik nest one burgers.
Deppaaai backgiound with pianic iutrecltt setting
cup with ofhile logo one birgens Frilange
annaarce cheese
bunger appaai melted
Place the burgers onto a planic tabi aalreclath setiigg Add vivid red cup with a white logo next to one of the burgers, Resize the cheese stices on the burgers to appear melted over the pantes, Dupiicate the visilee gr een lesuuee on the fromt burger. Enhance one the burgers with noing steam for freshly-cooked cooked arpesrance.
C8
Replace the background with bright planee tobledoth setting Add viiiily red drink cup with white logo nood one of the bu burpons, Maki the choese slice burger appear larger and more mehed over patty. Add soeams disigg from one of the burgers, dupiicate the green letuce leaf the fontt burges brightonn the Image highlihht the burger details and apply surny and warm lier scone.
(b) Synthetic input images.
Figure 9: Outputs from open-source and proprietary models via direct editing, See results with more models in the Appendix Figs. 18 and 19
5 Experiments
5.1 Experiment Setup
Dataset. Our Complex-Edit dataset comprises both realistic and synthetic images. Specifically, we select 531 deduplicated images from the EMU-Edit test (Sheynin al., 2024) to serve as our realistic image of 531 synthetic images
collection. In parallel, use FLUX.1 (Labs] 2024) generate an equivalent based on the captions associated with these real-world images, also sourced from EMU-Edit. Following Sec. create editing instructions with the complexity ranging from Ci to C3 for these images.
Models. Our experiments involve five advanced instruction- based image editing models: three open-source modcls (UltraEdit (Zhao et al.] (202)), OmniGen (Xiao et al.) 20224. AnyEdit (Yu et al. (024)) and three proprictary models (Imagen3 (Baldridge et al. (004)), SeodEdit (Shi al. (024)), GPT-Image-1 (OpenAI, 2025) We evaluate each model on both realistic and synthetic images using editing instructions spanning
9

---

Published in Transactions on Machine Learning Research (03/2026)
Table 4: Performance comparison between C₁ and Cs on images. All models almost consistently underperform on all metrics when the complexity increases from C1 to Cg.
Table 5: Real input images
Table 6: Synthetic input image.
36xll
G
C C2
Miald
G2
Cy CL
UltaEda
1.15
1.30
1.45
2.45
6.55
6.58
1.29
0.0
-0.57
-1.82
-0.30
-0.55
Ulimkii
1.82
5.83
080
8.68
0.11
0.81
5.22
1.36
-1.30
-1.75
-0.30
-1.15
Ouniities
6.76
8:00
7.0
188
8:35
6:13
1.55
6.78
-0.30
-2.27
-0:45
-1.00
OrniCen
7:80
9:31
8.65
8m
6.88
7-47
6.71
135
-1.11
-1.87
-0.31
-1.16
AnpEda
6.86
3.30
0.38
2.87
1.3
8.15
1.36
9.6
-4.53
-0.34
10:47
-1.49
AudEa
1.23
5:17
8.8
1.80
148
6.78
572
6.31
-4.3
-0.64
10.11
-1.86
Inapved
1:87
8838
2381
8:37
8:35
1.32
-0.11
-0.30
6:27
746
235
-8.30
GPI-bma--1
Aecrna
7.37
8.78
3.12
8,83
6.85
7,00
7.56
7.22
-1.08
-1.77
-0.08
-0.90
5334
5.29
9.00
8.8
0.0
7.51
5.87
7.82
-1.40
-1.00
-000
-1.02
Input
C2
OmniGen
(a) Real input images.
Input
C2
C4
C6
C8
Imagen3
(b) Synthetic input images.
Figure 10: Qualitative results of sequential editing. The complexity level ranges from C1 Cs- It can be seen that IF and PQ are severely hurt by the growing instruction complexity. The instructions are the same
ones in Fig. 9 Results with more models are shown in Appendix Figs. 20 and 21
Input
C2
C4
C6
C8
Sequential
Sequential w/ Best-of-4
Figure 11: Qualitative results of sequential editing with and without Best-of-4 with OmniGen real input images. The complexity level is at Cg. This shows that sequential editing can benefit lot from Best-of-N in terms of IF and PQ. The instructions are the same ones in Fig. Da Qualitative results with more models are shown in Appendix Fig.
complexity levels from C1 to Cs. Note that, owing to usage policies for the proprictary models, results are reported on approximately 60% of the images for Imagen3, 95% for SeedEdit and 90% for GPT-Image-1. All the evaluation are performed by GPT-4o by default. Although Sec. demonstrates that GPT-4o's evaluations are highly aligned with human preferences, is important note that a noticcable discrepancy between GPT-4o's judgments and human evaluations still remains. In addition GPT-4o, also provide evaluation results by Cemini-2.5-Prro (Comanici et al. 2025) in Appendix Sec. D
10

---

Published in Transactions on Machine Learning Research (03/2026)
Instruction Following
Identity Preservation
Perceptual Quality
Overall
10
Best of N
Best of N
Best of N
Best of N
UltraFdt
OmniCien
AnyEdt
Ulrraditt (Sequentia))
OmniGen (Sequentia)
AnyEdl (Sequential)
Figure 12: Direct and Sequential Editing results w/ Best-of-N of open-source models on real input images. The complexity level is Cs. This shows that Best-of-N can improve IF and PQ, especially for sequential editing. However, sequential editing with Best-of-N can still barely surpass direct editing without Best-of-N.
5.2 Effect of Increasing Complexity
Fig. illustrates the overall performance trends as instruction complexity increases. In general, higher complexity leads to a consistent decline in Identity Preservation across models, while Instruction Following and Perceptual Quality lend to fluctuale depending on the specific model. For instance, although AnyEdit exhibits substantial drop in Instruction Following as complexity increases, its Perceptual Quality improves moderately. Additionally, proprietary models outperform open-source alternatives on both realistic and synthetic images, and the performance gap becomes more pronounced at higher complexity levels. Specifically, GPT-Imago-1 significantly outperforms all other models at the hardest instruction complexity level Cs, particularly in terms of Instruction Following and Perceptual Quality.
Furthermore, both the GPT-4o's and 2.5-oo- evaluation in Tab. 4 and Appendix Tabs. 7 and show that the performance drops of open-source models are not notably more severe than those of proprietary models other than GPT-Image-l. For example, Tab. 5 shows that when editing real images, all open-source models have drops around 1.00 in Overall score while the Overall score's drop of Imagen3 and SeedEdit are 0.90 and 0.95 respectively, implying that stronger models may not be more resilient towards the negative impact of incrcased instruction complexity. Sample qualitative results are shown in Fig.
5.3 Curse of Synthetic data
Our qualitative analysis reveals an intriguing phenomenon: when applying extremely complex editing instructions (Cs) to real input images, the resulting outputs frequently lose their photorealism and adopt distinctly synthetic aesthetic. Notably, UltraEdit particularly susceptible to this effect compared OmniGen and AnyEdit, illustrated in Fig. 7 We attribute this trend to the composition of UltraEdit's 's training data, which contains significantly higher proportion of synthetic images than the datasets used by OmniGen and AnyEdi (see Tab. 3 This observation aligns with the "curse of synthetic data" phenomenon discussed in (Gerstgrasser et al.] 2024
This phenomenon also extends to highly capable proprietary models, including SeedEdit, Imagen3, and even GPT-Image-1. Specifically, as illustrated in Fig. 8. the edited images produced by GPT-Image-1 completely lose their photorealism, even though its overall performance surpasses that of all other models evaluated. This observation may indicate that, although the training sources for Imagen3 and GPT-Image-1 these models likely incorporate synthetic data into their training sets, which in turn contributes to their impressive performance as well as the emergence of distinctly synthetic aesthetic. Morcover, the identified "curse of synthetic data" underscores the risk that repeated reliance on synthetic images in training and evaluation may gradually erode photorealism and authenticity. This stresses the necessity of incorporating safeguards to mitigate possible self-reinforcing cycles, such as balancing synthetic datasets with high-quality real-world examples or implementing rigorous human-in-the-loop verification steps. We detail the evaluation protocal for this phenomenon in Appendix Sec.
undisclosed,
11

---

Published in Transactions on Machine Learning Research (03/2026)
5.4 Test-Time Scaling Approaches
5.4.1 CoT-like Sequential Editing
Our data pipeline composes complex instructions by combining sequences of atomic editing operations, suggesting that sequential application of these alomic sleps should be equivalent to executing the complex instruction directly. Moreover, this way of decomposing a complex task and then sequentially exccuting has been proven effective in both language generation (Wei et al. 2022) and T2I generation (Guo et al.] 2025).
Formally, given an imput image 2, complex instruction T at complexity level C₁, and a corresponding sequence of atomic instructions 1,,..., , we define the outputs for direct and sequential editing as follows: Jdirect = [(x T), and Ysequence = (2, {41....,44 4i)) [([((
We evaluate sequential editing on both real and synthetic images. Our results (summarized in Fig. and illustrated qualitatively in Fig. 10 reveal that sequential editing yiclds a steady doeline in performance across all three metrics, with accumulating visual artifacts and distortions even for strong proprietary models such Imagen3, SdedEdit and GPT- GPI--mmae--1 Notably, AnyEdit demonstrates improved Instruction Following with sequential editing compared to direct editing, although this improvement is offset by reduction in Identity Preservation.
5.4.2 Best-of-N
We also experiment with Best-of-N, which is another simple test-time-scaling method commonly used for lext-lo-image generation (Guo et al. 2025 Ma et al., 2025). For direct editing, we generate N candidate outputs for each input image and select the one with the highest overall score O. For sequential editing, we generate N candidates at cach stop and choose the candidate with the highest overall score O to proceed the next editing operation. When evaluating an intermediate candidate at the i-th step during sequential editing, instead of considering it the output of a single-step edit produced with the predecessor output and atomic instruction, e.g. f()--1,() 44) with y/-1 = fx, 141.....4411)) we evaluate it as the output from the original input image with a compound instruction T. This compound instruction T, has C2 complexity and encompasses all the atomic instructions up to the i-th step: (4,,...,4 4i). The rationale here is that the intermediate result should best represent the cumulative effects of all preceding atomic steps on the original image, ensuring the final output best reflects all atomic steps and consoquently the final compound instruction.
Our evaluations with open-source models on real-life images (sce Fig. 12 indicate that increasing N gradually improves direct editing performance across all metrics. For saquential editing, even a Best-of-N strategy with N = 2 produces significant gains in Identity Preservation and Perceptual Quality; however, the improvement in Instruction Following is less consistent. Qualitative comparisons are provided in Fig.
6 Related Works
Instruction-Bassd Image Editing InstructPix2Pix [Boooks et al., 2023 introduced instruction- based image oditing, where images are modified through text instructions, improving user interaction. This required large dataset created using fine-tuned GPT-3 (Achiam al., 2023) and the Prompt-to-Pronee
(Hertz et al.,
2022 Diffusion Model, but editing was limited by these models' quality. Research explored new methods:
MagicBrush (Kawar et al. 2023) used curated annotations, HQ-Edit (Hui et al.) 2025) utilized advanced
language models with text-to- text-to-image systems for better datasets, MGIE (Fu et al.) 2023) and SmartEdit (Huang et al. 2024) applied multilevel language learning for more precise edits, and Edita (Zhao et al. 2024
used advanced masking for detailed manipulation. OmniGen (Xiao et al.) (024) expanded model architecture
and data with Transformer framework for processing text and visuals and inferring edits. UIP2P (Simsar
et al. (2044) introduced cyclic editing for unsupervised image editing without paired training images. These methods have significantly advanced instruction-based image editing.
12

---

Published in Transactions on Machine Learning Research (03/2026)
7 Conclusion
In this work, we introduced Complex-Edit, comprehensive benchmark designed to systematically evaluate instruction-based image editing models across varying levels of instruction complexity. Through extensive experiments, Complex-Edit enable us to uncover multiple key insights regarding the limitations and capabili- ties of current models. We hope this benchmark will drive future advancements in developing more powerful instruction-baser image editing models, especially those with test-time scaling ability.
Acknowledgement
We would like to thank Google Cloud Research Credits Program, and the Microsoft Accelerating Foundation Models Research Program for supporting our computing needs.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tlee Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXi::2303.08777, 2023.
Jason Baldridge, Jakob Bauer, Mukul Bhutani, Nicole Brichtova, Andrew Bunner, Lluis Castrejon, Kelvin Chan, Yichang Chen, Sander Dieleman, Yuqing Du, et al. Imagen 3. arXiv preprint arXiu:408.007009 2024.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF CVF conference on compuler vision and pallern recognilion, pp- 18392-18402, 2023.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint 3rXiv:2507.666) 2025.
Chenhang Cui, An Zhang, Yiyang Zhou, Zhaorun Chen, Gelei Deng, Huaxiu Yao, and Tat-Seng Chua. Fine-grained verifiers: Preference modeling as next-token prediction vision-language alignment. arXiv preprint ar Xix::224 14148, 2024.
Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, and Zhe Gan. Guiding instruction-based image editing via multimodal large language models. arXio preprint or orxi:22000. 17102, 2023.
Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Henry Sleight, John Hughes, Tomasz Korbak, Rajashree Agrawal, Dhruv Pai, Andrey Gromov, et al. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data. arXie preprint Xix::22404. 01413, 2024.
Ziyu Guo, Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Peng Gao, Hongsheng Li, and Pheng-Ann Heng.
Can we generate images with cot? let's verify and reinforce image generation step by step. arXie preprinl arXiv:2501.19922 2025.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yacl Pritch, and Daniel Cohen-Or. Prompt-to- prompt image editing with cross attention control. Xixi preprint. arXiv:2208. 01122008.0166.. 01626, 2022.
Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, et al. Smartedit: Exploring complex instruction based image editing with multimodal large language models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pallern Recognition, pp. 8362 8371, 2024.
Mude Hui, Siwci Yang. Bingehen Zhao, Yichun Shi, Heng Wang, Peng Wang, Cihang Xic, and Yuyin Zhou. HQ-edit: A high-quality dataset for instruction- based image editing. In The Thirteenth International Conference on Learning Representations, 2025. URL.
13

---

Published in Transactions on Machine Learning Research (03/2026)
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila. Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXim:210.22177 2024.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Trani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 6007-6017, 2023.
Max Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen. Viescore: Towards explainable metrics for conditional image synthesis evaluation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.
2024a.
Max Ku, Tianle Li, Kai Zhang, Yujie Lu, Xingyu Fu, Wenwen Zhuang, and Wenhu Chen. Imagenhub: Standardizing the evaluation of conditional image generation models. In The Twelfth International Conference on Learning Representations, 2024b. URL https://ppeerrviier
Black Forest Labs. Flux. http:s///gtabb.cocoooaaaaatttttt
2024.
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xic. SiT: Exploring flow and diffusion-based generalive models with scalable interpolant transformers. In ECCV, 2024.
Nanye Ma, Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang. Yandong Li, Tommi Jaakkola, Xuhui Jía, et al. Inference-time scaling for diffusion models beyond scaling denoising steps. arXie preprint or 09970920.0 2025.
OpenAI. Introducing our latest image generation model in the API. hutpps//oppenaaccomm/mee//////// 2025.
generation-aply 2025.
Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks. ar preprint 10X00 2023.
Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks. In Proceedings of the IEEE/ /CVF Conference on Computer Vision and Pattern Recognition, pp. 8871-8879, 2024.
Yichun Shi, Peng Wang, and Weilin Huang. Seododit: Align image re-genoration to image editing. ar preprint Xiv: 160666, 2024.
Enis Simsar, Alessio Tonioni, Yongqin Xian, Thomas Hofmann, and Federico Tombari. Uip2p: Unsupervised instruction-based image editing via cycle edit consistency. arXiv preprint arXiv:2412 12. 15216, 2024.
Gemini Team, Rohan Anil, Sebastian Borgcaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. Xixiv preprint ar Xix:22312. 1180,, 2023.
Gemini Team, Petko Georgiev, Ving Tan Tei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint ar arXiv:2403. 05530, 2024.
Cong Wei, Zheyang Xiong, Weiming Ren, Xeron Du, Ge Zhang, and Wenhu Chen. Omniedit: Building image editing generalist models through specialist supervision. In The Thirteenth International Conference on Learning Representations, 2025. URL
Jason Wei, Xuczhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. tf prompting clicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24888 2022.
14

---

Published in Transactions on Machine Learning Research (03/2026)
Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrum Xing, Ruiran Yan, Chaofan Li, Shuting Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation. orxie preprint ar Xiv:2400 11340, 2024.
Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min-Yen Kan, Junxian He, and Michael Xie. Self-evaluation guided beam search for reasoning. Advances in Neural Information Processing Systems, 36: 41618-41650, 2023.
Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min- Yen Kan, Timothy Lillicrap, Kenji Kawaguchi, and Michael Shich. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint arXi:22405.00451, 2024.
Qifan Yu, Wei Chow, Zhongqi Yue, Kaihang Pan, Yang Wu, Xiaoyang Wan, Juncheng Li, Siliang Tang, Hanwang Zhang, and Yueting Zhuang. Anyedit: Mastering unified high-quality image editing for any idea. ar preprint 2rXix:2411.150 2024.
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guided image editing. Advances in Neural Information Processing Systems, 36:31428 31449, 2023.
Shu Zhang, Xinyi Yang, Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, Stefano Ermon, et al. Hive: Harnessing human feedback for instructional visual editing, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9026 9036, 2024.
Haozhe Zhao, Xiaojian Ma, Liang Chen, Shuzheng Si, Rujie Wu, Kaikai An, Peiyu Yu, Minjia Zhang, Qing Li, and Baobao Chang. Ultraedit: Instruction-based fine- -raiiga image editing at scale. In The Thirty-eight Conference on Newral Information Processing Systems Datasets and Benchmarks Track, 2024.
URL https::/ .ett:/forrmmmmdd322000g00g00
Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. Calibrated self-rewarding vision language models. ar preprint Xr0: 146622 2024.
15

---

Published in Transactions on Machine Learning Research (03/2026)
A The Hierarchy of Editing Complexity Types
To better contextualize the contribution of Complex-Eai Edit and clarify our definition of instruction complexity, we propose a hierarchy of editing complexily types for instruction-based image editing.
Tier 1 (T1): Simple Edit. This tier primarily operates on a perceptual basis, comprising singlo-step operations that correspond the manipulation of specific visual attribute or object (e.g.. "add a cat," "dim the background"). The principal technical challenge is visual grounding and regeneration that is, accurately locating the target pixels and generating consistent textures. Most existing datasets and benchmarks focus assessing this foundational capability.
Tier 2 (T2): Semantic-Comple. Edit. This tier introduces structural and linguistic challenges. The difficulty stems not just from generating pixels, but from processing dense instructions and resolving the inter-relationships between multiple visual elements. This is the primary focus of Complex-Edit.
Compositional Complexity, The instruction contains multiple atomic operations, creating a high compositional load. The core challenge is satisfying multiple constraints simultancously (e.g., "Add red car on the left and blue bicycle on the right") while maintaining attribute binding and disentanglement, Successful editing requires the model to prevent semantic leakage (e.g., ensuring the car does not absorb the color blue) among the dense set of constraints.
Contextual Complexity. The instruction relies linguistic contert, such pronouns, ellipsis, or multi-turn interaction history. The challenge lies in coreference resolution (e.g., "Add dog. Make it look angry") and pragmatic interpretation (e.g., "Give the man on the left pair of glasses. Same for the one on the right."), where the requested edit is implied by context rather than an self-contained description.
Tier 3 (T3): Inferential-Compler Edit. Unlike Tier 2, where instructions are concrete and descriptive, this tier involves implicit abstract directives that typically require cognitive-level processing. The model maybe presented with high-level goal rather than checklist of changes. The core challenge shifts from execution to latent planning, as the instruction does not directly describe the final visual state.
Reasoning Complexity. The instruction implics a causal relationship. Determining what to edit requires logical derivation. For instance, "Add support. beam where the bridge is structurally weak" implies multi-step process: analyzing the force distribution to locate potential weakness placing the beam. The edit is contingent on understanding the scene's physical logic rather than text matching.
Knowledge Complexity. The instruction relies on specific world knowledge or factual relations that have no direct visual correspondence in the text. The challenge involves translating non-vise visual entity into specific visual attributes via knowledge retricval. For example, in the instruction "Change the animal to the natural predator of the seal in the Arctic," the model cannot simply map the keyword "predator" to a specific texture; necessitates establishing the factual link: arctic predator of seal + polar bear.
Tier 4 Higher (2 T4): Future Frontiers. Defining the upper tiers of this hicrarchy remains open area of exploration. We anticipate that these tiers will necessitate capabilities that transcend the current definition of image editing, such as long-term planning (e.g.. in multi-step agentic workflows), iterative self-reflection.
Complexity Tier vs. Execution Difficulty. It is essential to distinguish between the cognitive tier of a task and its execution difficulty. While T3 (Inferential Complexity) conceptually represents a higher level of intelligence requiring latent planning, T2 (Semantic Complexity) rigorously tests the model's capacity to process dense, structured information involving intricate compositional and contextual

---

Published in Transactions on Machine Learning Research (03/2026)
dependencies. Intuitively, executing high-density T2 instruction (e.g., Cs)-which requires simultaneously
satisfying numerous specific constraints- can impose heavier processing load than T3 task involving simple logical deduction. In this high-load regime, models are prone to "losing focus" or omitting details due to the overwhelming semantic density, a vulnerability that Complex-Eds specifically targets.
Atomic Editing Operations
Here we present the descriptions for all the atomic operation types. These operation types are also fed GPT-4o during Stage #1 (Sequence Generation) of our data pipeline, as discussed in Sec. 2.1
Object Manipulation and Transformation
Add on Object: Insert a new element into the image.
Remove Object: Eliminate existing element from the image.
Replace an Object: Swap one element with another.
Move an Object: Change the position of an existing element within the image.
Resize Object: Adjust the size of an existing element.
Rotate an Object: Rotate an element to specified angle.
Duplicate an Object: Create a copy of an existing element.
Color and Tone Adjustments
Change Color: Replace the color of an element with a specified color.
Apply Filter/ Weethe: Add a color filter or weather effect to the entire image or specific parts.
Texture and Material Adjustments
Change Texture: Apply texture to an element (e.g., change from metal wood).
Background and Environment
Change Background: Replace the background with a different scene color.
Lighting and Shadows
Adjust Lighting: Change the overall lighting or lighting of specific elements.
Text and Symbols
Add Teal: Insert. text into the image.
Remove Text: Eliminate existing text from the image.
Change Text Properties: Modify font, color, size, or position of existing text.
Pose and Expression
Change Pose: Modify the stance or posture of person or object.
Change Facial Expression: Alter the facial expression of a character.
Composition and Cropping
Crop Image: Adjust the framing of the image by removing outer areas.
Reframe Composition:
Change the focus or arrangement of elements within the image. Zoom In/Out: Adjust the zoom level to focus specific elements show broader view.
Special Effects
Add Special Effects: Introduce effects like glow, motion blur, or lens flare.
Remove Special Effects: Eliminate existing special effects from the image.
Add Particles: Insert particles like dust.
Remove Particles: Remove existing particles from the image.
2

---

Published in Transactions on Machine Learning Research (03/2026)
C Prompts for Evaluation
C.1
Rubrics
In the rubric for cach metric we propose, we specify the detailed textual description for every score from 0 to 10, and most scores have a textual example to reduce ambiguity. All the rubrics for evaluation in are listed in Fig. 13
Rubric for Instruction Following.
* 10 (Perfect Instruction Following): the required changes occur in the output image.
(Near Perfect Instruction Following with negligible deviations): Almost all instructed changes are present but negligible deviatious exist (e.g., a thny color variation such as the cat in the image is now black but the cars are grey).
* 7-8 (Strong Instruction Following with minor deviations): Most required changes are applied sccurately. Minor deviations exist but do not substantially alter the intended modification (e.g., a car is changed to blue as instructed, but the reflection on its surface still contains red tint).
5-6 (Moderate Instruction Following with noticeable deviations): The output reffects an attempt to follow instructions but with moderate errors (e.g., adding a required element but with incorrect attributes like color or shape).
3-4 (Weak Instruction Following with major deviations); Most required modifications are missing, incorrect, only vaguely implemented. Significant elements from the instruction are misrepresented (e.g., when instructed to add a hat, a small, barely visible accessory is added the head, but does not resemble proper hat).
1-2 (Miniuul Instruction Following with severe deviatious]: A vague attempt is made, but the required modifications are either incorrect or so minimal that they not fulfill the instruction (c.g., the instruction asks remove a person from image, but. they are still visible, just slightly blurred or faded instead of being properly erased.).
* (Complete failed Instruction Following): The output image is entirely unrelated to the instruction.
(a) Instruction Following.
Rubric for Identity Preservation.
* 10 (Perfect Identity Preservation): All key elements that should remain unchanged are completely preserved and indistinguishable from the input (e.g., person's face, expression, and proportions remain completely unchanged except for the required edits).
(Near Perfect Identity Preservation with negligible distortion): Key elements that should remain unchanged are preserved with uegligible distortion (e.g., A person's is face is identical except. for a tiny, imperceptible variation in hair texture).
* 7-8 (Strong Identity Preservation with minor distortion): Small details of the key elements may have changed, but they do not siguificantly disrupt the overall ideutity (e.g,, a pet's fur pattern remains mostly accurate, but a minor detail like stripe or spot diffarent.).
5-6 (Moderate Identity Preservation with noticeable distortion): Most of the key elements remain recognizable but with noticeable distortions (e.g., the instruction asks to change car's color, but the car's s shape or size modified along with the color).
3-4 (Weak Identity Preservation with major distortion): Key elements maintain general resemblance but noticeable changes present (e.g., the instruction asles to brighten the sky, but additional buildings in the background appear or disappear).
* 1-2 (Minimal Identity Preservation with severe distortion): Most key elements are significantly altered or elements in the output retain only minor aspects of the original, but major features are incorrect (e.g., a person's 's face is still a human face, but it no longer resembles original person all).
The key
* 0 (Complete failed Identity Preservation): All key elements that should remain unchanged are altered, distorted, or missing.
(b) Identity Preservation.
Rubric for Perceptual Quality.
(Perfort Perceptual Quality): The image appears flawlessly natural, and ohjects are seamlessly intograted into the environment with consistent lighting and shadows. There is no visual artifact at all.
9 (Near Perfect Perceptual Quality with negligible incoherence): The image is very close to perfect, but a tiny, almost imperoeptible inconsistency exists. Seamless integration, but one might notice extremely subtle flaw. (e.g., person added to group photo blends in perfectly, but upon close examination, their shadow is slightly softer than others.)
* 7-8 (Strong Perceptual Quality with minor incoherence): Minor incoherence and artifacts present but they do not significantly detract from the overall harmony. (e.g., sunset scene where the added reflections on water are slightly off in intensity, but the image still looks highly realistic.)
* 5-6 (Moderate Perceptual Quality with noticeable incoberence): There is noticeable visual artifacts affecting the image's harmony, Lighting and shadows may be misaligned or inconsistent. (e.g., an animal is distorted in size or shape, making it appear out of place the scene.)
3-4 (Weak Perceptual Quality with major incoherence): Disharmonious elements are prominent, greatly disturhing the visnal harmony. (e.g., an animal's shape a person's face is greatly distorted, only showing some resemblance of the animal species human face.)
* 1-2 (Minimal Perceptual Quality with aevere incoherence): The whole scene is distorted, making it difficult to recognize the objects or subjects in the image. (Complete failed Perceptual Quality): The image is completely random and makes no sense at all.
(c) Perceptual Quality.
Figure 13: Rubrics for evaluating Instruction Following, Identity Preservation and Perceptual Quality.
3

---

Published in Transactions on Machine Learning Research (03/2026)
C.2
System Prompts
The system prompls that we use for evaluation are provided in Fig. 14
System Prompt for Instruction Following and Identity Preservation.
You are required evaluate result instruction- based image editing model. Given an input image, an output image and text. instruction, you are required to access the output image hased on whether the changes made the input image align with the text instruction.
You are required to give two integer scores in [0, 10) based on the following criteria: Instruction Following: whether the required changes also made. 10 means that all the changes required by the instruction occur in the output image, 0 means that no changes required by instruction occur in putput image.
the output image, regardless of whether unnecessary changes
2. Identity Preservation: whether elements that should not be changed stay the same in the output image, regardless of whether required changes occur. means that no unnecesary changes occur in the output image, means that all elements in the Input image that should he kept the
changed in the output image.
Here the detailed rubric for Instruction Following: <INSTRUCTION
NNNTTUCC1100FFOLLVVVIG FOLLOVING RUBRIC>
Here is the detailed rubric for Identity Preservation:
<IDENTITY, SIDENTITY_PPESEEVAAA PRESERVATION RUERIC>
Note that these two scores should be graded independently, and low score for one criterion should not affect the score for the other criterion. but an Identity Preservation score of 10. Also, an output imnaee that has no relevance with the input imngee should have an Identity Preservation score affect the Instruction Following score long changes required by the Instruction occur in the output.
example, an output image that identical the imput image should
Instruction Following score
instruction specifically orders model
different image,
not.
If the instruction contains several stomic operations, evaluate the Instruction Following for each atomic operation separately and then average the scores as the assessment for Instruction Following.
Explain your reasoning hefore answering the question.
(a) Instruction Following and Identity Preservation.
System Prompt for Perceptual Quality.
You required evaluate model-generater image. Given an output, you are required to access the output image's 'Percepunal Quality'.
You are required to give one integer score in [0, 10) with 0 indicating extreme disharmony characterized by numerous conflicting or clashing elements, and indicating perfect harmony with all components blending effortlessly.
These are criteria:
1. Consistency in lighting and shadows: The light
and corresponding shadows consistent various elements, with
no discrepancies in direction or intensity.
Element cohesion: Every item in the image should logically within the scene's context, without appearing misplaced extraneous.
Integration and edge smoothness: Objects should blend seamlessly their surroundings, edges appear artificially inserted or poorly integrated.
Aesthetic uniformity and visual flow: image should not only be aesthetically pleasing but also facilitate natural visual journey, without abrupt interruptions caused by disharmonious elements.
Here the detailed rubric for scoring:
<PERCEPTULL, BUALLL_QUALLYY__UREAAAA
(b) Perceptual Quality.
Figure 14: System prompt for evaluating Instrottion Following, Identity Preservation and Perceptual Quality.
4

---

Published in Transactions on Machine Learning Research (03/2026)
D Evaluation Bias due to VLM Evaluators
To avoid evaluation completely relying GPT-4o, we opt for another VLM evaluator, Gemini-2.5-Pro CCo- manici et al. 2025), which is from a distinct family.
Here we present the comparison between evaluation results of direct editing with C₁ and Cs instructions, in supplement to the results from GPT-4o ovaluation shown in Tab. 4 We only measured once for cach sample with Gemimii2.5-PPa due to cost limitation. As we report average scores across the test set, the variance of each score is expected to be 0.01, sufficient. for reliability.
The observations based on Tabs. 7 and 8 remain consistent as ones discussed in Sec. 5.2 that at the highest instruction complexity level Cs. GPT--mmage-1 excels over all other models in Instruction Following and Perceptual Quality; and performance drops in open-source models are comparable to proprietary ones, except. for G-T--maae--
Table 7: Performance comparison between C and Cs on real images. All models almost consistently underperform on all metrics when the complexity increases from C₁ to Cz. Evaluated by Gemini-2.5-PPe
C2
C. C
Model
PQ
IF
PQ
Aip
App
Apq
Ao
UhraEdit
6.63
5.26
5.94
OmniGen
6.43
7.44
6.82
6.90
5.17
4.35
5.47
5.00
-1.26
-3.09
-1.35
-1.90
EdyEdit
5.26
7.31
5.50
6.02
1.01
7.81
6.51
5.11
-4.25
+0.51
+1.01
-0.91
Imagen3
7.25
8.32
7.71
7.76
6.70
4.65
6.53
5.96
-0.55
-3.67
-1.18
-1.80
SeedEaitt
9.17
6.29
7.96
7.81
7.67
3.83
7.69
6.40
-1.50
-2.46
-0.27
-1.41
GPT Imuge-1
9.78
8.06
9.10
8.98
9.45
5.04
9.25
7.92
-0.33
-3.01
40.15
-1.06
Average
7.42
7.11
7.17
7.24
5.88
4.84
6.74
5.82
-1.54
-2.27
-0.43
-1.42
Table 8: Performance comparison between C1 and Cs on synthetic images. All models almost consistently
underperform on all metrics when the complexity increases from C₁ to Cz. Evaluated by Gemini-2.5-PPr
C
C
Model
IF
IP
IF
PQ
Aip
App
Anq
4o
UltraEdit
7.31
6.82
8.50
7.55
4.97
4.70
7.69
5.79
-2.34
-2.12
-0.81
-1.76
OmniGen
7.46
8.68
8.52
8.22
5.15
6.19
7.68
6.34
-2.31
-2.48
-0.84
=1.88
AnyEdit
5.31
8.68
7.98
7.32
0.78
8.83
5.33
5.98
-4.53
+0.115
+0.35
-1.34
Imagen3
7.64
8.43
8.49
8.19
7.11
6.34
9.37
7.34
-0.53
-1.90
-0.12
-0.85
SeedEdit
9.05
8.36
8.96
8.79
6.97
6.86
8.63
7.49
-2.08
-1.50
-0.33
-1.30
GPT--magg-1
9.26
9.09
9.54
-2.52
+0.15
Average
8.27
5.68
8.37
-2.07
-1.73

---

Published in Transactions on Machine Learning Research (03/2026)
E
Human Evaluation of Curse of Synthetic Data
To isolate the influence of Perceptual Quality on the assessment of an image's photorcalistic style, randomly selected 50 imput samples at Cg complexity level for which all evaluated models attained a PQ 2 8. For each selected input, the corresponding output images produced by each model 300 images in total were subsequently evaluated by a human rater. The rater was instructed to assign a photorealism score to each image on a discrete ordinal scale ranging from (totally ) to (perfectly photorealistic), shown in Fig. 15
Photorealism Score:
Photoreallsm
Photorealism Photorealsm Score:
Photorealism Score:
Figure 15: Examples of human-evaluated photorealism score.
Table Evaluation realistic style with direct editing on real images with the instruction complexity at Cg.
Model
Prop. of Real Images
Realisim Score
UltraEdit
2.84
OmniGen
3.72
AnyEdit
4.74
SeedEdit
2.84
Imagen3
2.76
GPTT-mage-1
1.30
Given that this evaluation protocol is relatively rudimentary, we report the exact photorealism scores only in Appendix Tab. and omit them from Tab. 3 We rely exclusively on human evaluation rather than VLM--baed automatic assessment because, in the context of image editing, the photorcalism of output image must often be judged under prompts that intentionally include fantastical or physically implausible elements (e.g., science-fiction or magical content). Assessing such "photorealism" requires the evaluator to possess not only high-fidelity visual perception, but also counterfactual reasoning capabilities for instance, the ability to infer "how a unicorn should plausibly appear if unicorns were to exist".

---

Published in Transactions on Machine Learning Research (03/2026)
F
Additional Qualitative Results
Here present more comprehensive qualitative results.
F.1 Examples Evaluation Metrics
Ingut Image
Instruction Following:
Instruction Following:2 2
Instruction Following:
Instrucion Follwing: 5
Instruciio Following: 0
Instruction Following 10
Input Image
Idansity Preservation:
Idemiiy Presanvation::
Preservaton:
Identhly
Preservation:
Presorvation:
Instruction (Replace the palm trees with cherry bicssom lrees and change the cloudy sky to clear blue sky. Add car the road beneeat the brigge complete the selling.
Figure 16: Examples of evaluation results for Instruction Following and Identity Preservation.
Perceptual Quality:
Perceptual Quality
Perceptusl Quality
Perceptual Quality:
Perceptual Quality 8
Perceptual Quality: 10
Figure 17: Examples of evaluation results for Perceptual Quality.
F.2 Direct Editing
Additional qualitative results of direct editing with more models are shown in Figs. 18 and 19 indicating that the gap between open-source and proprietary models expands with increasing complexity of the instruction. Also, Identity Preservation and Perceptual Quality continue drop instruction complexity increases, while the effect on Instruction Following varies from model to model. One refer the discussion in Sec. 5.2 for more detailed insights.
F.3 Sequential Editing
We illustrate additional qualitative results of sequential editing with more models Figs. 20 and 21 showing that all three aspects, particularly Identity Preservation and Perceptual Quality, consistently degrade, visual artifacts and distortions accumulate with increase in the number of intermediate steps. Even advanced proprietary models, i.e. Imagen3 and SeedEdit, demonstrate limitations in maintaining the identity of key elements and the overall quality of output images. More in-depth discussions are given in Sec. 5.4.1
F.4 Best-of-N
Additional qualitative results of sequential editing combined with Best-of-4 with more open-source models shown in Fig. 22 Tt. can been noticed that the Identity Preservation and Perceptual Quality of sequential
7

---

Published in Transactions on Machine Learning Research (03/2026)
Input
C2
C4
C6
C8
UltraEdit
OmniGen
AnyEdit
Imagen3
Seed-Edit
C2 Replace dining background Replace the background with rustic wooden table, add sauce bottle beside slices, paper ceramic dish glazed finish, and rearrange the pizza slices for less everlap.
pizza
C4
C6
Place the pizza slices ceramic with glazed garmish next each.
wooden table. Rearrange the siices reduce overlap and pansley
hot
Agply warm-toned
cozy ambiance.
C8 Replace dining background
Adjust pizza everlap less, and add parsley gamiis near each slice as well as hot sauce bottle beside them. Apply warm-toned filter, introduce steam effects over the pizza, slices, slighty zoom framing for
one. Transforn paper plate glazed
Figure 18: Additional qualitative results with direct editing on real-life images. As instruction complexity increases, the gap between open-source and proprietary models widens, and both Identity Preservation and Perceptual Quality decline, with variable effects on Instruction Following across models. See Sec. 5.2 for more details.
oditing's results can be substantially improved with Best-of-4, A more comprehensive analysis of this topic is provided in Sec. 5.4.2
8

---

Published in Transactions on Machine Learning Research (03/2026)
The Identity Preservation and Perceptual Quality of sequential editing improve significantly with Best-of-4, as analyzed in detail in Sec. 5.4.2
F.5 Direct Editing with GPT-Image-1
We provide more qualitative results with GPTT-Image-1 via direct editing real-life images in Fig. 23 These results imply that while GPT--mage-l is capable of generating edited images of exceptional quality, output images with very complex instructions prune to lose the photorealism in the original images. This may imply a lack real-life images in GPT-Image-1's 's training data. See Sec. 5.3 for a more in-depth analysis.

---

Published in Transactions on Machine Learning Research (03/2026)
Input
C2
C4
C6
C8
UltraEdit
OmniGen
AnyEdit
Imagen3
Seed-Edit
CacCClla
C2 Replace dark background picnic tablecloth setting soft cup Replace the dark background with picnic tablecloth setting, Place vivid red soft drink with white enhhance
burgers.
C4
the Enlarge and
burger appear melted patly.
C6
burgers picnic tablecloth setting. burgers to sppear mekted over the pattees, Duplicate the visible green lettuce on the front burger. Enhance one of the burgers with rising steem for freshly-cooked appearance.
cup
logo
burgers.
C8 Replace background cheese slice on burger appear larger and more melted over the pally. Add steam rising from one the burgers, duplcatee the green lettuce leal on the burger, brighten image highlight burger
bright picnic tablecloth setting. vividly
cup
logo to burgers.
apply sunny
Figure 19: Additional qualitative results with direct editing on synthetic images. As instruction complexity increases, the gap between open-source and proprictary modcls widens, and both Identity Prescrvation and Perceptual Quality decline, with variable effects Instruction Following across models. See Sec. 5.2 for details.
10

---

Published in Transactions on Machine Learning Research (03/2026)
Input
C2
C4
C6
C8
UltraEdit
OmniGen
AnyEdit
Imagen3
Seed-Edit
C2 Replace the dining table background with rustic wooden table Add a hot sauce bottle next to the pizza sloes. Replace the background with a rustic wooden table, add a sauce bottle beside the pizza slices, change the paper plate to a ceramic dish with glazed finish, and rearrange the pizza Slices less overlap.
C4
C6
Place the pizza slices on a ceramic dish with glazed texture stop a rustic wooden table. Rearrange the siices reduce overlap and add parsley germish next eech. Include bottle of hot sauce near the dish. Apply wam-tooed filter cozy amblance.
C8 Replace the dining table background with rustic wooden one. Transform the paper plate into glazed ceramic dish. Adjust the pizza siices averlap Apply warm-toned filter, introduce
less, and add parsley garnish slices, and slightl zoom in the framing for emphasis.
the pizza.
Figure 20: Additional qualitative results with sequential editing on real-life images. Identity Preservation and Perceptual Quality, degrade as visual artifacts and distortions increase with more intermediate steps. Even advanced proprietary models, iee. Tmagen3 and SeedEdit, struggle to maintain element identity and quality in output images. More in-depth analysis is in Sec. 5.4.1
11

---

Published in Transactions on Machine Learning Research (03/2026)
Input
C2
C4
C6
C8
UltraEdit
OmniGen
AnyEdit
Imagen3
C
Seed-Edit
C2 Replace dark background picnic tablecloth setting soft cup Replace the dark background with picnic tablecloth setting. Place vivid soft drink with white enhhance
burgers.
C4
the Enlarge and
burger appear melted patly.
C6
burgers picnic tablecloth setting. burgers to sppear mekted over the patties, Duplicate the visible green lettuce on the front burger. Enhance one of the burgers with rising steem for freshly-cooked appearance.
cup
logo
burgers.
C8 Replace background bright picnic tablecloth setting. vividly cheese slice burger appear larger and more melted over the pally. Add steam rising from one the burgers, duplcatee the green lettuce leal on the burger, brighten image highlight burger
cup with logo to burgers.
apply sunny
Figure 21: Additional qualitative results with sequential editing on synthetic images. Identity Preservation and Perceptual Quality, degrade as visual artifacts and distortions increase with more intermediate steps. Even advanced proprietary models, i.e. Imagen3 and SeedEdit, struggle maintain element. identity and quality in output images. More in-depth analysis is in Sec. 5.4.1
12

---

Published in Transactions on Machine Learning Research (03/2026)
Input
C2
C4
C6
C8
UltraEdit
OmniGen
AnyEdit
C2 Replace dining background Replace the background with wooden table, add glazed linish, and rearrange the pizza slices for less overlap.
sauce
pizza
C4
bottle beside pizza siices paper ceramic dish
wooden Rearrange garmish next to each. include bottle of hot sauce near the dish. Apply warm-tonee filter a cozy ambiance.
pizza
reduce overiap parsley
Replace the dining table background with rustic wooden one. Transfomn the paper plate into glazed ceramic dish. Adjusi the pizza slices overlap less, and add parsley gamiih near each slice as well hot sauce bottle beside them. Apply warm- toned filter, introduce steam effects over the pizza slices, slightly zoom framing emphasis.
Figure 22: Qualitative results with sequential editing with Best-of-4 on real-life images. Identity Preservation and Perceptual Quality of soquential editing improve significantly with Best-of-4. Refer to Sec. 5.4.2 for more detailed discussions.
13

---

Published in Transactions on Machine Learning Research (03/2026)
M
EXCLUSIVE SALE
Happy Holidays
Figure 23: More real-life images edited with Cs instructions by GPPT-mage-l Outputs from GPTT-maagel severely lose the photorcalism style. Refer to Sec. 5.3 for a dotailed discussion.
14

---

Published in Transactions on Machine Learning Research (03/2026)
G Assets and Licensing
All code used for evaluation and data generation is relcased at AutrbuttionNNnnCommeet Edit We also release the test set under the Creative Commons
4.0 (CC BY-NC
4.0) license This release includes both real and synthetic input images, as well as all the artifacts produced across the three generation stages, including atomic instructions (both pre and post-simplification)) rationales, and the final composed instructions.
Ethical concerns
We manually review all the generated content, including the synthetic input images generated by FLUX.1 (Labs) 2024) as well as the instructions and rationales from GPT-4o (Hurst et al.] 2024). to ensure that there no inappropriate generation.
15


Reviews:
[{"review_id": "MYDPpKVBr5", "reviewer": "Reviewer_nkmc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a  a large-scale benchmark called Complex-Edit, designed for evaluating instruction-based image editing models across instructions of varying complexity. The benchmark is constructed using Chain-of-Edit data generation pipeline, which constructs complex editing instructions by compounding atomic editing operations. GPT-4o is used to automatically collect a diverse set of editing\ninstructions at scale. The paper also proposes a VLM-based evaluation framework based on three types of metrics: Instruction Following, Identity Preservation, and Perceptual Quality. In addition to the benchmark construction, experiments on five image editing models reveal several valuable empirical findings, such as the degradation under increasing complexity, the failure of CoT-like sequential editing, and the “curse of synthetic data”.\n\nStrengths:\n- Paper is well-written and easy to follow\n- Data generation pipeline is clearly defined, structured (24 operations across 9 categories are defined), is scalable and diverse\n- Evaluation metric is divided into three meaningful categories (Instruction Following, Identity Preservation and Perceptual Quality). \n- Correlation with human judgment is investigated, which is important for the task of image editing, where the aim is to basically increase human satisfaction with the result. \n- Good ablation on metrics (numeric scoring vs token-probability scoring, effect of rubrics)\n- Experimental results provide interesting insights, such as effect of CoT, instruction input on VLM evaluation, and missing correlation between model strength and improved image editing results\n- 5 modern image editing models are considered, including open-sourced and proprietary ones.\n\nWeaknesses:\n- For generating synthetic images, only FLUX.1 is used. It is interesting to see if the choice of the model affects image editing performance, as there is evidence that most image generative models produce images with certain distinct characteristics ([1], [2]) which possibly can impact on how well image editing models can process these images\n- As far as I understand, in the Best-of-N setting, the metric that is used to select the best candidate is the same as the final metric used to assess the editing quality (with the same VLM). I am not sure if this is the standard thing to do, but in my mind this creates a situation when a final image might become biased towards metric computed by a certain VLM. Does it make sense to do so?\n- In fig.5. I personally do not understand why the last image in the second row has the highest identity preservation score. Doesn't a second image from the right follow the original image better?\n- \"Curse of Synthetic data\" is presented only in the qualitative setting. It would be nice to see some metrics, e.g. CLIP alignment score with the prompts of a form \"an image with synthetic aesthetic\" and  \"an image with realistic aesthetic\" or asking some VLM to assess realism of an input image. This is an important finding, in my opinion, hence it's desirable to support it with quantitative evidence.\n- It would be interesting to see analysis of failure mods of image editing models. The presented benchmark consists of 24 operations divided by 9 categories, and it would be interesting to see if there is a pattern of which categories/operations fail first in which image editing models. I can imagine that some edits (for example, involving complex/small objects) resulting in more failures than others (e.g. style manipulations). Also, it would help reveal biases of image editing models towards certain types of edits. Next, more complex prompts are constructed as combinations of simple editing prompts: is there evidence that some combinations are more challenging for the models to handle than others (e.g. background + lighting, object + color, etc)?  Also, it would be interesting to see reasons why different models fail on benchmark tasks. E.g. is failure defined by hallucinating new objects, object omission or identity drift? This might shed a light on how to improve these models and bridge the gap between open-source models and proprietary models. \n- Overall metric is now computed as a simple average of three components (Instruction Following, Identity Preservation and Perceptual Quality). Is it the best way to compute a final metric? Intuitively, I can imagine that some of these components (e.g. Perceptual Quality) mean more than others (e.g. Identity Preservation). Would other weighting strategies improve correlation with human judgement?\n\n\n[1] Guangyu Nie, Changhoon Kim, Yezhou Yang, Yi Ren: Attributing Image Generative Models using Latent Fingerprints. ICML 2023: 26150-26165\n[2] Hae Jin Song, Mahyar Khayatkhoei, Wael AbdAlmageed: ManiFPT: Defining and Analyzing Fingerprints of Generative Models. CVPR 2024: 10971-10981", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper is clearly written and coherent in general", "audience_yn": "Yes", "audience_explanation": "The construction of a well-defined, complexity-controllable benchmark with a clear evaluation framework for assessing the quality of image editing models is an important and highly relevant topic", "requested_changes": "I would request the following: \n- I would like the authors to answer my question about Best-of-N setting \n- Assessing \"Curse of Synthetic data\" using some relevant quantitative measure\n- Analysis of failure mods of image editing models (at least assessment of individual complexities of atomic edits: are there edit types that are more challenging than others in general? Can we treat them equally?)", "broader_impact_concerns": "-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Jq0WcbkxmV", "reviewer": "Reviewer_2iM2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "### Summary of Contributions:\nThis paper introduces Complex-Edit, a large-scale benchmark designed to systematically evaluate instruction-based image editing models under varying levels of instruction complexity. The authors develop a three-stage GPT-4o-driven data generation pipeline (“Sequence Generation → Simplification → Instruction Compounding”) to create editing instructions with controllable complexity. They further propose an improved VLM-based auto-evaluation framework, incorporating refined metrics—Instruction Following (IF), Identity Preservation (IP), and Perceptual Quality (PQ)—along with detailed rubrics and variance control strategies to enhance alignment with human judgments. Using this benchmark, they conduct an extensive study across open-source and proprietary models, uncovering key phenomena such as the decline of identity preservation with complexity, the limited robustness of stronger models, and the “curse of synthetic data” where models trained with synthetic inputs produce increasingly artificial results under complex edits.\n\n### Key Strengths:\n- Comprehensive and systematic benchmark: Introduces the first dataset explicitly designed to evaluate complexity in instruction-based image editing.\n\n- Robust evaluation methodology: Well-validated VLM-based auto-evaluation framework that correlates strongly with human judgment.\n\n- Novel empirical insights: Reveals important trends—performance degradation under complexity, limits of test-time scaling, and synthetic data artifacts.\n\n- Thorough experimental design: Includes multiple models (open-source and proprietary), realistic vs. synthetic data, and rigorous meta-evaluation of metrics.\n\n### Key Weaknesses:\n\n- Heavy dependence on proprietary GPT-4o for both dataset creation and evaluation, potentially limiting reproducibility and accessibility.\n\n- Limited semantic diversity analysis: “Complexity” is defined mainly by the number of atomic edits rather than deeper compositional semantics.\n\n- Potential circularity in evaluation: Using the same VLM family for generating and grading data could bias outcomes toward GPT-style reasoning.\n\n- Qualitative analysis of realism drift is insightful but somewhat subjective, with limited quantitative grounding.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Overall, yes — the paper’s claims are well supported by convincing and clearly presented evidence, though a few limitations remain.\n\n### Strengths in Support and Evidence:\n\n- The paper provides comprehensive quantitative evidence across multiple models (open-source and proprietary), two image domains (real vs. synthetic), and eight levels of instruction complexity. This breadth substantiates claims about general trends—especially the decline of identity preservation and the widening performance gap at higher complexity.\n\n- The meta-evaluation study rigorously validates their VLM-based scoring framework against human judgments, lending strong credibility to their quantitative results.\n\n- The reported findings are consistent across two evaluators (GPT-4o and Gemini-2.5-Pro), reinforcing robustness.\n\n- Detailed qualitative visual comparisons (figures illustrating degradation, synthetic drift, and sequential editing artifacts) visually corroborate the numerical trends.\n\n### Areas Less Strongly Supported:\n\n- While the “curse of synthetic data” phenomenon is compelling, the evidence is mostly qualitative and interpretive rather than quantitatively isolated or controlled.\n\n- The definition of instruction complexity (by number of atomic edits) is clear and systematic but may not capture all aspects of semantic or cognitive difficulty; this limits how far the conclusions can generalize.\n\n- Because both data generation and evaluation depend heavily on GPT-4o, some bias or circularity may remain unmeasured.\n\n### Verdict:\nThe paper’s core claims are accurate and convincingly backed by extensive experiments and analyses, with only minor caveats regarding the scope of “complexity” and the interpretive nature of certain findings (e.g., the synthetic-data effect).", "audience_yn": "Yes", "audience_explanation": "The paper’s findings would be of clear interest to a substantial portion of the TMLR audience.\n\n### Rationale:\n\n- The work directly addresses an emerging subfield at the intersection of vision-language modeling and generative editing, which overlaps with current research trends in multimodal learning, diffusion models, and instruction following.\n\n- Its benchmarking and evaluation framework fill a recognized gap in how instruction-based image editing systems are assessed, offering practical tools and insights valuable to both academic and industry researchers.\n\n- The discovery of phenomena such as the performance degradation with increasing edit complexity and the “curse of synthetic data” are conceptually significant and likely to stimulate further methodological and dataset research in multimodal generation and training data quality.\n\n- Even readers outside image editing (e.g., those working on instruction tuning, evaluation metrics, or test-time scaling in multimodal models) would find the methodological innovations in auto-evaluation and meta-evaluation broadly relevant.\n\nIn short, the paper contributes both practical benchmarking resources and conceptual insights that align well with TMLR’s focus on rigorously analyzing and improving machine learning systems.", "requested_changes": "## Critical Adjustments (Required for Acceptance)\n\n### Clarify the Definition and Scope of “Complexity.”\n\n- Issue: The paper defines instruction complexity solely as the number of merged atomic edits, which may not fully represent semantic or compositional difficulty.\n\n- Adjustment: Provide justification or empirical evidence that this metric correlates with human-perceived complexity (e.g., via small-scale user validation or linguistic analysis).\n\n- Rationale: Essential for ensuring the benchmark truly measures the intended construct and that results are interpretable beyond this operational definition.\n\n### Address Potential Bias and Circularity in GPT-4o–Based Generation and Evaluation.\n\n- Issue: GPT-4o is used both to generate the dataset and to evaluate models, raising the possibility of model-family bias.\n\n- Adjustment: Include a discussion or experiment assessing how results differ when evaluated with a different VLM family (e.g., Gemini, Claude, or open-source BLIP/InternVL).\n\n- Rationale: Critical to confirm that the benchmark’s conclusions are not artifacts of one model’s evaluation tendencies.\n\n### Quantify the “Curse of Synthetic Data.”\n\n- Issue: This claim is supported mainly by qualitative visuals and speculation about training data composition.\n\n- Adjustment: Add quantitative measures (e.g., FID, realism classifiers, human realism ratings) or controlled experiments isolating the proportion of synthetic data.\n\n- Rationale: Strengthens the credibility of this key empirical insight and distinguishes correlation from causation.\n\n## Non-Critical Improvements (Would Strengthen the Work)\n\n### Enhance Transparency and Reproducibility.\n\n- Issue: Heavy reliance on proprietary systems (GPT-4o, Imagen3, SeedEdit) limits replicability.\n\n- Adjustment: Clearly state which data and scripts can be released publicly and provide detailed pseudocode or prompts for the three-stage data-generation pipeline.\n\n- Rationale: Necessary to ensure future researchers can replicate or extend the benchmark.", "broader_impact_concerns": "### Broader Impact Concerns:\nThe paper’s benchmark and evaluation pipeline raise several ethical considerations that merit explicit acknowledgment:\n\n### Synthetic Data Feedback Loops:\nThe identified “curse of synthetic data” highlights a risk that repeated use of synthetic images in training and evaluation could erode visual realism and authenticity over time. The authors should briefly discuss safeguards against such self-reinforcing cycles.\n\n### Automated Evaluation Bias:\nReliance on proprietary VLMs (GPT-4o, Gemini) for judging aesthetics and accuracy may embed opaque cultural or stylistic biases. A short note on the limits of automated evaluation and the need for periodic human calibration would improve transparency.\n\n### Misuse Potential:\nMore capable instruction-based editing systems can be misused for deceptive or harmful image manipulation. Acknowledging this risk and referencing standard safety measures (e.g., watermarking, content filtering) would strengthen the ethical framing.\n\n### Accessibility and Equity:\nSince data generation and evaluation depend on paid APIs, researchers with limited resources may be excluded. Mentioning plans for partial data or prompt release could mitigate this concern.\n\nOverall, these issues don’t undermine the technical work but warrant a Broader Impact Statement addressing synthetic-data amplification, evaluation bias, misuse prevention, and equitable access.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qifImjmvYl", "reviewer": "Reviewer_69ut", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Complex-Edit, a benchmark for instruction-based image editing where instruction complexity is controlled by composing multiple atomic edit operations. A three-stage GPT‑4o-driven pipeline generates sequences of atomic instructions for each image, simplifies them, and compounds prefixes of length 1…8. The authors also design a VLM-based evaluation protocol that evaluates for instruction following, identity preservation, and perceptual quality. Using their benchmark, the authors evaluate multiple open- and closed-source models, analyze direct vs CoT-style sequential editing, and study Best-of‑N selection and the “curse of synthetic data” phenomenon.\n\n**Strengths:**\n- The paper presents several interesting findings.\n- The 24 atomic instructions offer a good coverage of editing primitives.\n- Splitting alignment metric into IF and IP seems like a good alternative to CLIP scoring, especially because it aligns more with human judgement.\n- The paper provides a good analysis for their design choices: numeric scoring vs. token probabilities, CoT vs. no CoT, rubric vs. no rubric, and deterministic decoding vs. stochastic sampling.\n- Evaluating three open-source models and three proprietary models on both real and synthetic images at different complexity levels provides a fairly comprehensive evaluation.\n\n**Limitations:**\n- Heavy dependence on GPT-4o as both instruction generator and evaluator, without a thorough bias analysis.\n- Reliance on only EMU-Edit test images and relatively small scale.\n- The paper does not clearly state whether the dataset (images, instructions, metadata) and evaluation code will be publicly released.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I think the authors do a good job in substantiating the claims with thorough experiments.", "audience_yn": "Yes", "audience_explanation": "The paper offers a good evaluation of existing open- and closed-source image editing models across instructions of varying complexity and presents a well-designed evaluation protocol. The findings and strategies used would likely be of interest to many in the audience.", "requested_changes": "I would request the following updates to the paper:\n- Provide a clearer dataset description section and release plan.\n- Add an ethics section addressing data licensing and misuse.", "broader_impact_concerns": "The dataset uses GPT-4o to automatically generate editing instructions (e.g., changing a person’s expression or body posture) and includes both synthetic and real-world images modified according to those instructions. Adding a Broader Impact section would help address the ethical considerations involved in generating and editing such data.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)