Image Editing As Programs with Diffusion Models
Authors: Yujia Hu
Songhua Liu
Zhenxiong Tan
Xingyi Yang
Xinchao Wang
Venue: NeurIPS 2025 poster ()
Keywords: Image Editing
Diffusion Transformer
Primary area: applications

Abstract: While diffusion models have achieved remarkable success in text-to-image generation, they encounter significant challenges with instruction-driven image editing. Our research highlights a key challenge: these models particularly struggle with structurally-inconsistent edits that involve substantial layout changes. To address this gap, we introduce Image Editing As Programs (IEAP), a unified image editing framework built upon the Diffusion Transformer (DiT) architecture. Specifically, IEAP deals with complex instructions by decomposing them into a sequence of programmable atomic operations. Each atomic operation manages a specific type of structurally consistent edit; when sequentially combined, IEAP enables the execution of arbitrary and structurally-inconsistent transformations. This reductionist approach enables IEAP to robustly handle a wide spectrum of edits, encompassing both structurally-consistent and inconsistent changes. Extensive experiments demonstrate that IEAP significantly outperforms state-of-the-art methods on standard benchmarks across various editing scenarios. In these evaluations, our framework delivers superior accuracy and semantic fidelity, particularly for complex, multi-step instructions. Codes are available at https://github.com/YujiaHu1109/IEAP.

Image Editing As Programs with Diffusion Models
Yujia Hu', Songhua Liu Zhenxiong Tan', Xingyi Yang".,, and Xinchao Wang'* "National University of Singapore School of Artificial Intelligence. Shanghai Jiao Tong University
The Hong Kong Polytechnic University yujia. huGu. nus edu, xinchao@nus.co edu.
Complex Instruction Editing Stop Display
Ckange Anckgruute Agrundd she Adste lody drurz
for brtsle lodyy
Origral Ineraction: Whr Nir placed budy to fornn daring cutuonn and war worringa dvers, with
Simplo Instruction Editing
Aeylace ooffee a
CAJow agmonian to not amiiags
Mone the plaxt on tbe capNNArn
MUNIO ANe AVANS
Complex Instruction Editing Results Only
STOP
All dos
Change suiling moke sketch.
Par the baack,
Change the matorial the size esed and tim the car ieroa Aign.
Entezze and the lafe.
Churse the antion o'e baly cranding and make a sturag do:
make it decorated Clstimar style aut anange color
Figure 1: Visual results of our IEAP. Rows I and 3 showcase complex multi-step edits (Row is further decomposed into individual instructions), while Row 2 shows single-instruction edits. Single instructions underlined if needing be reduced atomic operations.
Abstract
While diffusion models have achieved remarkable success in text-to-image genera- tion, they encounter significant challenges with instruction-driven image editing. Our research highlights key challenge: these models particularly struggle with atructraally-incosiise
edits that involve substantial layout changes. To address this gap, we introduce Image Editing As Programs (IEAP), unified image editing framework built upon the Diffusion Transformer (DiT) architecture. Specifically, IEAP deals with complex instructions by decomposing them into a sequence of programmable atomic operations. Fach atomic operation manages a specific type of structurally consistent edit; when sequentially combined, IEAP enables the exccution of arbitrary and structurally-innoossitee transformations. This re-
"Correspondigg Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

ductionist approach enables IEAP to robustly handle a wide spectrum of edits, encompassing both prrucurally-cooniiserr and inconsistent changes. Extensive experiments demonstrate that IEAP significantly outperforms tfate-of-the-art meth- ods on standard benchmarks across various editing scenarios. these evaluations, our framework delivers superior accuracy and semantic fidelity, particularly for complex, multi-step instructions. Codes are available here
Introduction
Image editing lies at the heart of a wide range of applications from photo retouching and content creation to visual storytelling and scientific visualization [42 63 With the advent of diffusion
models 23
53 47. the field has shifted towards highly precise and controllable manipulations The inherently progressive denoising process enables multi-stage pipelines [24] 2
45
12
62].
and localized editing methods [10 77 58) and its native support for multi-modal inputs has inspired
unified frameworks that integrate heterogeneous signals within a single model
33
15
More recently, text-to-image pipelines based on Diffusion Transformers (DiTs) 46
13
3
have set
new standards in generative fidelity. However, their capacity for instruction-driven editing 41 27
remains under-explored. Notably, although there are a few existing methods [80 37 that have editing. they are always restricted to a narrow set of common
extended DiTs to instruction-dee editing operations and lack evaluation comprehensive editing tasks.
To address this limitation, we initiate a taxonomy study of image editing instructions to system- atically assess the editing capabilities of current DiT- bar-d conditional generation methods. Our empirical analysis reveals an interesting performance dichotomy: While current methods demonstrate proficiency in conssstelly-cosistteetn edits where the layouts of the input and output images remain aligned, they exhibit significant degradation when handling structurally-inconiinn operations that require layout modifications.
To overcome this issue, we introduce Image Editing As Programs (IEAP), a unified framework atop the DiT architecture which is capable of handling diverse types of editing operations efficiently and robustly in this paper. Notably, we show that suuuturalll-inconsssteee reduced to a small set of simple operations, which are called as atomic operations in our paper. Thus, instead of treating each edit as a monolithic, end-to-end task, IEAP levarages the Chain-of-Thonn Thouhtt (CoT) reasoning [65] to break the original editing command into sequence of atomic operations, which are namely Region of Interest (Rol) localization, Rol inpainting, Rol editing, Rol compositing and global transformation, and then executes them in sequential manner via neural program interpreter [49]
instructions can in fact be
The five atomic operations serve as the fundamental building blocks for complex editing tasks. As such, through the sequential combination of atomic operations, IEAP can robustly handle complex. multi-step instructions that are typically confound in conventional end-to-end approaches.
Extensive experiments show that framework demonstrates state-of-the-art performance across standard benchmarks, excelling in both structural preservation and alteration tasks through atomic- level operation decomposition compared to other approaches. Simultaneously, the CoT reasoning and programming pipeline of IEAP enable significantly more accurate and semantically more coherent edits under complex, multi-step instructions even compared the leading proprictary models.
Our main contributions can be summarized as follows:
We present comprehensive taxonomy and empirical analysis of instruction-driven editing in baseb conditional generation, revealing a performance dichotomy between structurally- consistent and -inconsistent edits.
We introduce Image Editing As Programs (IEAP), a unified framework on the DiT back- bone that leverages CoT reasoning to parse free-form instructions into sequential atomic operations and then executes them sequentially by neural program interpreter, thereby enabling robust handling of layout-altering and complex edits.
Extensive experiments demonstrate that IEAP achieves state-of-the-art performance in both structure-prestt preceuri- and -altering scenarios, delivering notably higher accuracy and semantic fidelity especially on complex, multi-step instructions compared to existing methods.

---

2 Related Work
Conditional image generation. Early conditional image generation approaches like ControlNet 77
typically adopt plug-in control adapters to incorporate single condition [3] 16 33 like segmentation mask or diverse conditional inputs 82 48 26
40 70] to guide the generation of images. Recently, the
field of conditional image generation has witnessed remarkable breakthroughs through the integration
of DiTs [3 46 31 with continuous innovations improving output quality and edit precision [45].
Some methods 169
68 9 aim to create unified DiT foundation for versatile conditional image generation and editing by integrating diverse inputs within a single framework. while approaches
32
like OminiControl [60] and so on [6] 79 38
80
67
leverage LoRA-based fine-tuning [25] for
lightweight and effective control.
Instructional image editing. Instruction-based image editing 4 27 enables intuitive, language- driven modifications of existing images. Early works like InstructPix2Pix [ establishes paired instruction iimaee datasets for supervised fine-tuning of diffusion models. For subsequent works, 34 20], which introduce specialized conditioning units and multi-stage training to improve control granularity and consistency, others concentrate on data-centric enhancements 776. diversify edit examples. Moreover, some approaches 75
some of them focus on architectural refinement [38] 37
81
g that expand instruction coverage and has unified LL-M-based [ language reasoning with diffusion-based synthesis in a single framework, and some [72] 78 leverage
17
56
28
33 15
CoT [65] and in-context learning (21] enhance the reasoning ability of models for more complex editing tasks. Mcanwhile, certain approaches [64] 66 [5] build upon multimodal understanding to decompose intricate instructions, ensuring editing precision and output reliability. More recently, some works [14 80 37 have advanced image editing with DiTs. For instance, ICEdit [80] leverages the in-context generation capabilities of large-scale DiTs to achieve flexible few-shot instruction editing, while StepIX X E-Eiit [37 focuses on large-scale data construction and multi-modal integration to enable general-
ceneral-purpose purpose image editing with performance approaching proprietary models.
3 Motivation
3.1 Preliminaries
Diffusion Transformer Fundamentals. The image generation process of text-guided DiTs 44 13
31 is accomplished by successively denoising input tokens in multiple steps. At step f, the model
processes:
= Cr]
where X; E Nxd represents noisy image tokens and CT E Md denotes text tokens, they share the embedding dimension Image tokens use Rotary Position Embedding (RoPE) [59] with spatial coordinates (i,j), while text tokens fix positions at (0, 0), enabling Multi-Modal Attention (MMA) 44 mechanisms to model cross- modal interactions.
Unified Conditioning Framework. To integrate visual control signals, the prior work 660 extends the baseline formulation by incorporating encoded condition images:
S: = _ Cr, Cr]
(2)
where Cj E R.-xd denotes latent tokens from condition images via the pretrained VAE encoder 30 (52.
This unified sequence enables tri-modal fusion within transformer architectures, eliminating spatial misalignment inherent in feature concatenation basclines.
Moreover, an auxiliary adaptive positional encoding mechanism further preserves spatial consistency across these modalities by assigning coordinates to each token type with minimal overhead.
Gap in Instruction-Driver DiT Editing. Despite the rapid advances in DiT-based conditional image generation [60] 79, 38 67], research on citruuctiondrrveen editing [41, 27 remains scarce. The few existing methods 80 37 that do support instructional edits are typically confined to a small set of routine operations, and lack a comprehensive evaluation across diverse editing scenarios, leaving DiT's true editing potential unclear. This gap motivates us to conduct a taxonomy study of DiT's ability in instructional image editing, which is detailed in Sec. 3.2
3

---

Yower
Prpbent Pesii
Eatting Typec
(3)
Figure 2: Results of our preliminary experiments. Figure (a) shows the GPT-4o scores for three editing types across instruction faithfulness and semantic consistency, ranging from to 5. Figure (b) shows the representative failure cases from local semantic editing.
3.2 Preliminary Experiments and Observations
To this end, we conduct a comprehensive evaluation of diffusion models for instruction-driven editing, uncovering an interesting performance dichotomy: While these methods excel at structurally- consistent edits, they falter dramatically on annuctralll-iinconistter operations that demand explicit layout modifications.
Taxonomy and Experimental Setup. To enable systematic analysis 27 73 72 we first categorize
instruction-based image editing into three main types: local semantic editing, which modifies the identity, position or size, e.g., add, remove, replace, action change, move and resize; local attribute editing, which adjusts certain properties of objects, e.g., color change, texture change, appearance change, expression change, and background change; and overall content editing, which alters the whole image consistently, e.g., tone transfer and style change.
Then we use EdyEdit dataset [73 and OminiControl 600 to train models on the above editing types, accompanied by GPT-4o 29 to rate each edit on instruction faithfulness and semantic consistency.
Results and Analysis. As shown in Fig. 2aa), both local attribute editing and overall content editing attain relatively high GPT-4o scores, whereas local semantic editing exhibits a notable performance drop. As illustrated in Fig. [bb) the cases of "add" and "action change" alter unrelated areas like the background, and the remaining four cases demonstrate a complete failure.
We attribute this discrepancy to the fact that, unlike local attribute and overall content edits, local se- mantic edits require explicit spatial-layout modifications. For instance, "add" and "delete" operations necessitate instance-level scene recomposition, while "move" and "resize" further demand precise coordinate system recalibration.
Key Insight. Based on the above analysis, spatial-layout modification remains critical challenge for diffusion-based editing models; conversely, edits that preserve the original layout demonstrate substantially better performance. We speculate that, with limited training data, it is difficult for the model learn the complex patterns underlying layoui-changing tasks. Although DiT architectures 46 13 31 employ powerful full-attention mechanisms to capture long- range dependencies, they still struggle with editing operations that require nontrivial scene reconfiguration.
Due to the combinatorial complexity of lpatial-layout modifications and the empirical limitations of DiT architectures. we propose simplify the layout-editing paradigm through decomposition, which is detailed in Sec. 4
Methods
4.1 Program with Atomic Operations
The insight Sec. 3.2 motivates us to decouple semantic and spatial reasoning. Building on this foun- dation, we propose programmatic reduction framework that systematically decomposes complex editing instructions into modular atomic operations. Specifically, we first formulate instruction-driven image editing an executable program via Chain-of- Thoughh (CoT) reasoning (65, and then use a neural program interpreter [49] to transcode the reasoning graph into a dynamic execution plan, sequentially invoking relevant atomic modules.

---

A
Replace
Local Semantic Editing
Coler Change
Expecssoo Chonge
Appcanance Change
Testum: Change
Background
Change
Local Attributo Editing
roher the pellor figheer
oiee and bithh
IEAP
Adl
Chaes
Zaplec
Tea Clange
Iepeniin
Comparsing
Figure 3: Our pipeline. The original instruction is first parsed by VLM into atomic operations, which are then sequentially executed via a neural program interpreter.
4.2 General Pipeline
We abstract all editing instructions into five atomic primitives: (1) Rol Localization: Identify and iso- late the relevant region in the image that the instruction refers to, serving as the spatial grounding step for subsequent localized edits; (2) Rol Inpainting: Introduce new visual content remove existing elements within the localized region, enabling semantic-level additions, substitutions, or deletions; (3) Rol Editing: Modify visual attributes within the region, such as color, texture, or appearance, to reflect fine the-grained property changes specified by the instruction; (4) Rol Compositing: Reintegrate the edited region into the full image while preserving spatial coherence and visual continuity; (5) Global Transformation: Adjust the overall content for coherent full-image modifications, such as changing the illumination, weather, or style of the whole image.
The overall pipeline is shown as Fig. 3 We reduce any editing instruction into an arbitrary combina- tion of the five atomic operations described above, which can be formulated as:
T
Ak
Ak E (AUce Amp, Aedits Acomp Aglobal)
where T denotes the free-form editing instruction, + represents the sequential program combination, K is the number of atomic operations, Aree Ainp- Astit- Acomp- and Aglohal represent the five atomic primitives respectively.
Rol Localization. All problematic local semantic edits share a common first step: localizing Region of Interest (Rol) in the image for editing. Given image and an editing instruction T,, first employ Large Language Model (LLM) (1 to locate the text Rol:
P (T)M(T),
(4)
where p represents the text Rol extracted by the LLM MLLM- Subsequently, we achieve accurate localization of image Rol by:
R= R=Msee[T,, (T,
where A denotes the image Rol segmented by the segmentation model Maeg
74
For add operation, the instruction may not specify text Rol, the specification may be ambiguous. In such cases, we first derive the overall layout of all candidate objects using the capability of segmentation models [50 [4], and then prompt the LLM to determine the appropriate image Rol based on T.
Regarding move and resize, once the image Rol is obtained, we update the spatial layout of the image using an L.M [ Specifically, we provide the LLM with a set of in-context examples that define our layout representation and demonstrate representative editing patterns [36]. Given the current layout L, and the instruction the LLM is prompted to produce a modified layout L as formulated below:
Tags MI)M(!), L Meeg (Tags), Ledit (MLM(L,T))
(6)
5

---

Kol Lcalication
Kol Inpaining
Kol Editing
Kot Compoctioring
Example Procedure of Action Change Openation.
tepaincing
Compositiniig
Irpix
Layout Rocafigaraice
(b) Example Procedure of Move Operation.
Figure 4: Example procedure. Figure (a) and Figure (b) illustrate the procedures of action change and movement respectively.
We then derive the geometric differences between and Ledit and convert them into the corresponding affine transformations, consisting of translation, scaling, and reshaping, and apply it to R to update the spatial configuration, yielding the transformed mask R'.
Rol Inpainting. Once the image Rol has been localized, we apply inpainting to seamlessly fill and complete the region. For additive and substitutive operations, which aim introduce new objects, employ prompt-conditioned inpainting process to guide the generation of new content. Specifically, first extract the semantic entity E from the instruction T via LLM [I:
E (L),((),
and then construct composite prompt P in the form: "add E on the black region". For removal operations, which aim to eliminate existing content without introducing new semantics, we adopt backeround-oriented infilling strategy, setting P as "fill in the hole of the image". The edited image Jedi is then generated by:
Tedit
Minpaint
(1-1))),
(
where Mmpaint denotes the inpainting model trained by
Rol Editing. When operations pertain to property change are performed, we use trained attribute editing model Mattr to perform edits in this stage to obtain Tomi
edi Mattr (I,T).
(9)
RoI Compositing. To ensure seamless integration the edited Rol with its surrounding context, we first construct an annular mask M by applying morphological dilation and erosion 55 to the transformed Rol mask R³:
M Mann Dilato(R', kn) Erodc(R', k2).
(10)
Then, employ fusion network Mrusion- trained on ring-masked object boundaries. refine the pre-composited image fprop using the generated annular mask. The final edited image is obtained as:
Todi Minsion (Jprep O Mann), P),
(11)
where is set as "inpaint the blacerbordered region so that the object's 'ss edges blend smoothly with the background" to guide seamless boundary blending.
Global Transformation. Like Rol editing, in the scenarios involving global transformation, we use the trained global transformation model Mgtobal to perform edits in this final stage obtain Lodn-

---

Input
InstruciP2P
MagicBrush
UltraEdit
OmniGen22
IEAP(Ours)
[Add] Adi hird on the yellow stoo!
[Remove]
[Replace] Replace mirror,
|Action Change] Change the action the russiing
[Move] Move the hird she left.
|Appearance Changel Moke the cup decoruted in maple leaf patterns.
Chunge] Change of bay's
[Style Change] Change style the image bubbles.
Figure 5: Comparison results of ours with baseline methods on representative editing cases. Others exhibit poor performance some common editing operations, while approach demonstrates superior effectiveness across all operations.
5
Experiments
5.1
Experimental Settings
Training Settings. We train four specialized models for Rol inpainting, Rol editing, Rol compositing, and global transformation respectively. All models are fine-tuned on FLUX.1-dey [3] using LoRA 25). with default settings for rank 128 and alpha 128. Training is conducted with a batch size of and runs for 50,000 iterations each. We use the Prodigy optimizer [39) enabling safeguard warmup and bias correction, with weight decay of 0.01. The experiments are conducted on single NVIDIA
H100 GPU (80GB).

---

MagicBrush test
AnyEdit test
Method
CLIP km
CL.P out
L1 4
DINO f
CLIP im
L1 +
DINO
GPT t
InstructPix2Pix
0.838
0.229
0.112
0.758
0,801
0.110
0.765
3,83
MagicBrush
0.886
0.241
0.074
0.859
0,824
0.128
0.742
3,90
UltraEdit
0.911
0.227
0.061
0.883
0,833
0.114
0.772
3.93
GenArtist
0.908
0.232
0.063
0.889
0.829
0.121
0.776
3.98
OmniGen2
0.881
0.242
0.100
0.830
0.857
0.132
0.772
4.13
ICEdit
0.913
0.236
0.058
0.885
0.847
0.110
0.765
4.13
Ours
0.922
0.247
0.060
0.897
0.882
0.096
0.825
4.41
Table 1: Quantitative results on MagicBrush and AnyEdit test set.
Local Semantic Editing
Local Attribute Editing
Overall Content Editing
Method
CLIP
L1 DINO
GPT 1/ [CLIP
L1
DINO 1 GPT CLIPim im
L1
DINO 1 GPT
InstructP2P
0.826
0.132
0.738
3.74
0.790
0.135
0.737
3.92
0.766
0.156
0.642
3.91
MagicBrush
0.860
0.106
0.796
3.90
0.809
0.117
0.762
4.21
0.763
0.187
0,616
3,99
UltraEdit
0.867
0.095
0.812
3.86
0.801
0.092
0.793
3.94
0.754
0.201
0.611
4.41
GenArtist
0.864
0.097
0.821
3.88
0,814
0.108
0.801
3.96
0.752
0.207
0.595
4.38
OmniGen2
0.893
0.097
0.834
4.12
0.832
0.114
0.786
4.06
0.783
0,224
0.634
4.42
ICEdit
0.881
0.088
0.810
4.08
0.825
0.095
0.795
4.06
0.759
0.188
0.603
4.45
Ours
0.907
0.081
0.854
4.42
0.861
0.083
0.821
4.54
0.895
0.107
0.879
4.51
Table 2: Quantitative results on different types of editing operations.
Dataset Setup. For both the Rol editing and global transformation models, we sample from the relevant subsets of the AnyEdit [73] dataset and apply GPT-4o (29] to filter the data of some types that have numerous noisy examples. To cover facial expression edits absent AnyEdit, we integrate the CelebHQ-FM dataset [1I] which offers consistent identities and annotated expressions suitable for our instruction schema. The Rol inpainting and Rol compositing models are trained on samples from the "add", "remove" and "replace" splits of AnyEdit. For each sample, we first obtain the image Rol according to the editing instruction. In the Rol Inpainting training setup, we set the pixels within image Rol to black input train. For Rol Compositing, we set k and k2 3 in default to blackout the annular mask region image Rol as input for training.
Evaluation Settings. We evaluate our method on two benchmarks: MagicBrush test set 77]. widely used dataset spanning diverse editing types, and AnyEdit test set 773, from which we select 16 instruction-basse editing categories. For MagicBrush,
follow previous works [76 8 15 56) and
scores to measure the similarity between the generated results and ground-truth images. While for AnyEdit, where some categories lack reference captions required for calculating CLIPout, we instead leverage GPT-4o (2] to rate each edited image on scale from to 5 across three dimensions: instruction faithfulness, semantic consistency, and aesthetic quality, with the final GPT score obtained by averaging the three aspect scores.
report CLIPimg, CLIPout [22], L1, and DINO 7 43)
We first compare our method with existing state-of-the-art open-source baselines, including Instruct- Pix2Pix ()) MagicBrush (6). UltraEdit [81] GenArtist (4). OmniGen2 (66 and ICEdit [80 In addition, to demonstrate the competitiveness of our approach against powerful proprietary multimodal foundation models in complex image editing scenarios, we further make comparisons with SeedEdit (Doubao) 57 Gemini 2.0 Flash [19, and GPT-4o (29
5.2 Comparisons with State of the Art.
Qualitative Comparisons. Fig. 5 shows the results of our approach against other six methods (6. 76 8I
66 80 on some representative editing cases. Unlike previous methods, which sometimes misinterpret or fail to execute the given instructions, modify unintended regions, introduce undesired artifacts, or produce visually implausible results, our method consistently exhibits clear and consistent advantages in accurately following the instruction, maintaining structural coherence, preserving instance-level fidelity and retaining fine-grained visual details.
64
Quantitative Comparisons. Table I exhibits the quantitative comparison results of our method and other approaches 6 76 8T 66 80 on MagicBrush test set (76) and AnyEdit test set [73]
The results show that method demonstrates state-of-the-ard performance on both datasets. On

---

Impel
SeedEditDesbbo) Gembal 20 Flask
GFT-4>
TEAPiOUs))
Inpat
SeodEdirDonbbs) Gential 2,0 Flash
GFT-4o
TEAP(Ouss)
Move thr Aggest anage Autoyy on ewner The ruge change tee evereig
AM caleea dee nihie. solle wele no piia and flww/ppitooss renore See lyPchar.
TEAP
aocilacn odd patr nanglaure of the and par tEa!"
Change the action jumpling and rypiaue and caphoard rith plast
muteraal the conbary subatt 2an struct
Aemune the sausagr and
Figure 6: Comparisons Complex Instructions with Leading Multimodal Models. Our method achieves comparable or even better edit completeness and pre-post consistency.
MagicBrush, our method achieves the best performance in terms of caption alignment, semantic consistency, and preservation of fine- tne-grained structural details. Although it incurs marginal increase in pixel-level deviation compared to the best [80), this far outweighed by the substantial gains in perceptual quality and semantic fidelity. Furthermore, on AnyEdit, our approach yields significant and comprehensive improvements across all evaluation metrics, further highlighting its superiority over existing techniques.
To provide more fine- fine-grained analysis of editing performance, we group a subset the instruction- based categories from the AnyEdit test set (73) into three macro-tasks: local semantic editing, local attribute editing and overall semantic editing. For local attribute editing, we augment with some CelebHQ FM [1 test images to evaluate facial expression changes. The quantitave comparison results are shown in Tab. [ where our method consistently outperforms other candidates across all three task categories and evaluation metrics.
Comparisons with Cutting-Edge Multimodal Models. To demonstrate the superiority of reduction strategy on complex editing tasks, we also conduct comparative experiments against prominent closed-source multimodal models [57 19 29]. As illustrated in Fig. 6 method rivals, and in most cases surpasses the performance of these leading models on intricate scenarios requiring multiple sequential edits. Unlike competing approaches, which frequently omit specified instructions or introduce extraneous alterations unrelated to the editing directives, our framework faithfully executes each instruction while ensuring superior image consistency and instance preservation.
5.3
Ablation Studies
Boderion
Settings
CLIP
CCIP
out t L1 DINO GPT
w/o CoT Reduction
0.873
0,241
0.117 0.795
4.10
w/o Rol Inpainting
0.861
0.218
0.124
0.775
3.65
Cheepi tolooe
w/o Rol Editing
0.900
0.244
0.088
0.843
4.23
Fall
ano Fel Tikking
Aoe Noxt
w/o Layout Reconfiguration
0,900
0.245
0.088
0.848
4.31
w/o Annular Mask Integration
0.906
0.252
0.083
0.854
4.39
Full
0.907
0.252
0.081
0.854
4.42
Table 3: Module-wise ablation results AnyEdit local seman- tie editing test set.
Figure 7: Qualitative ablation of action change operation.
Module-wise Ablation Studies. To quantify the impact of each key component in our framework. we perform series of ablation studies on the AnyEdit (73] local semantic editing test set as we split in Sec. [.2] As shown in Tab. 3 we first substitute our CoT reasoning and reduction pipeline with end-to-end editing pipeline, resulting in marked performance deterioration across all metrics.
Next, we replace our specialized Rol inpainting and Rol editing models respectively with generic inpainting model from [60]. which induces performance declines of varying degrees. We then remove the L.M- --gdddd layout reconfiguration and instead employing random layout modifications for relevant operations, which incurs noticeable performance decline. Finally, omitting the annular

---

Co
CUPor
DINO
0.51
0433
(31
3515
924
C-0
3FM
0.4
Gemenaa Eottin LLMI
3301
090
0800
3300
GFT-4o las (8J4
0.49
M00
C000 0.0 GAM
0390
0240/
633
Figure Ablation results LLMs/VLMs on layout modification tasks.
mask integration produces a modest drop, underscoring its role in precise boundary delineation. Fig. 7 exxibbis the ablation results on an example of "action change", visually showcasing each module's necessity. Collectively, these ablation results confirm that each component our pipeline contributes significantly in handling robust local semantic editing tasks requiring layout changes.
Ablation Studies on Different LLMs/VLMS. We present comparative experimental results involving various LLMs and VLMs to analyze their influence on editing quality. For complex instruction decomposition, we evaluate chal- lenging instructions from the MagicBrush dataset [76]. As shown in Table 4 the step remains robust across different VL.Ms, but performance drops notably when the original image is excluded.
Model Type
Model
Accuracy (%)
VLM
GPT-4o
100.0
VLM
Gemini-2.0-
96.7
L.M
GPT-4
90.0
L.M
GPT-3.5-turbo 76.7
Table 4: Ablation results on complex
instruction decomposition.
As in IEAP, LLMs also assist in layout modifications for "add", "move" and "resize" operations. also compare the quantitative editing performance using different models. As shown in Fig, g the layout modification capacity is robust across various LLMs/VLMs generally.
5.4 Applications
Recove rés can vébo dusng supped un.
What bugaon wher lost isake?
Stop dee hoaats of paraer.
(a) oditing.
(b) Multi-object manipulaiio.
(c) Generaluuubiitty spen-world edit..
(4) Image inpainting.
Figure 9: Applications of IEAP: Handling precise, multi-object, open-world, and inpainting tasks.
Our method also demonstrates strong versatility across diverse image editing scenarios. As shown in Fig. 9[) a), our method can accurately localize and edit complex target entities described in the instruction. In Fig. b). when multiple synonymous objects are mentioned, all relevant instances are consistently modified. Moreover, as illustrated in Fig. C),, our model generalizes well to some unseen and challenging tasks such as swapping object locations, which remain difficult even for advanced large models. Finally, as shown in Fig. ([)), our approach can naturally extend to image inpainting with satisfactory performance.
6 Conclusions, Limitations and Future Work
In this paper, propose Image Editing As Programs (IEAP), unified DiT-based framework for instruction-drive image editing. By defining five atomic operations and using CoT reasoning to convert instructions into sequential programs, IEAP processes the ability to handle both simple and complex edits. Experiments demonstrate that IEAP outperforms of-te-of-the-ar methods in both structure preeurrepressering and structure -etteringa tasks, especially for complex edits.
Despite its strong overall performance, there are also some limitations. First, for complex shadow changes, our method sometimes leaves shadows inconsistent after compositing operations. Second, multiple editing iterations may induce progressive image quality decay. Future work could focus on addressing these issues via physics-aware shadow modeling and diffusion- based quality restoration.
10

---

Acknowledgment
This project is supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 (Award Number: KOE-T22P20122--006)
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, et al. Gpt-4 technical report, 2024.
[2] Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. ACM transactions on graphics (TOG), 42(,):1-11, 2023.
[3] Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin. Spatext: Spatio-textual representation for controllable image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18370- 18380, 2023.
[4] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 18208-18218, 2022.
Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. Patchmatch: A randomized correspondence algorithm for structural image editing. ACM Trans. Graph., 28(3):24, 2009.
[6] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 18392-18888 18402, 2023.
[7] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in sulf-super vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650 00069666,2222 2021.
[8] Tuhin Chakrabarty, Kanishk Singh, Arkadiy Saakyan, and Smaranda Muresan. Learning to follow object- centric image editing instructions faithfully. arXiv preprint arXiv:2310. 19145, 2023.
[9] Xi Chen, Zhifei Zhang, He Zhang, Yuqian Zhou, Soo Kim, Qing Liu, Yijun Li, Jianming Zhang, Nanxuan Zhao Yilin Wang, ct al. Unircal: Universal image generation and editing via learning real-world 2024.
dynamics. arXiv preprint
[10] Guillaume Couairon, Jakoh Verbeek, Holger Schwenk, and Matthicu Cord. Diffedit: Diffusion-based semantic image editing with mask guidance. arXiv preprint arXiv:2210.11227. 2022.
[11] Brian DeCann and Kirill Trapeznikov. Comprehensive dataset of face manipulations for development and evaluation of forensic tools, 2022.
[12] Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self-guidance for controllable image generation. Advances in Neural Information Processing Systems, 36:16222 36:16222-16239, 2023.
[13] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Bocsel, al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning. 2024.
[14] Kunyu Feng, Yue Ma, Bingyuan Wang, Chenyang Qi. Haozhe Chen, Qifeng Chen, and Zeyu Wang. Dit4edit: Diffusion transformer for image editing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 2969-2977, 2025.
[15] Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, and The Gan, Guiding instruction- based image editing via multimodal large language models. arXiv preprint arXiv:2309.1 17102, 2023.
[16] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based sext-to-image generation with human priors. In European Conference on Computer Vision, pages 99-106. Springer, 2022.
[17] Zigang Geng, Binxin Yang. Tiankai Hang, Chen Li. Shuyang Gu, Ting Zhang, Jianmin Bao, Zheng Zhang. Houqiang Li. Han Hu, et al. Instructdiffusion: A generalist modeling interface for vision tasks. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 22709-12727 2024.

---

[18] Vidit Goel, Elia Peruzzo, Yifan Jiang, Dejia Xu. Xingqian Xu, Nicu Sebe, Trevor Darrell, Zhangyang Wang. and Humphrey Shi. Pair diffusion: A comprehensive multimodal object-level image editor. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8609-8618, 2024.
[19] Google. Experiment with gemini 2.0 flash native image generation. Technical report, Google Al Studio, 2025.
[20] Qin Guo and Tianwei Lin. Focus on your instruction: Fine-grained and multi-instruction image editing by attention modulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6986-6996, 2024.
[21] Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training, 2022.
[22] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.0877 18,.8888, 2021.
[23] Jonathan Ho, Ajay Jain, and Picter Abheel. Denoising diffusion prohabilistic models, 2020.
[24] Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47): 1-37,))1333222 2022.
[25] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
[26] Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Composer: Creative and controllable image synthesis with composable conditions. arXiv preprint 2rX.v22302.09778, 2023.
[27] Yi Huang, Jiancheng Huang. Yifan Liu, Mingfu Yan, Jiaxi Lv, Jianzhuang Liu, Wei Xiong, He Zhang, Liangliang Cao, and Shifeng Chen. Diffusion model-based image editing: A survey. IEEE Transactions on Puttern Analysis and Machine Intelligence, page 1-27, 2025.
[28] Yuzhou Huang, Liangiin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, et al. Smartedit: Exploring complex instruction bassccttoo--beedddd image editing with multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 8362-8371, 2024.
[29] Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, et al. Gpt-4o system card, 2024.
[30] Diederik Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013.
[31] Black Forest Labs. Flux.
www./////ooooo/eoeoooeoooeoeeeee forest- labs//luab 2024.
[32] Duong Le, Tuan Pham, Sangho Lee, Christopher Clark, Aniruddha Kembhavi, Stephan Mandt, Ranjay Krishna, and Jiasen Lu. One diffusion to generate them all, arXiv preprint 1xXiv:2411.1611 2024.
[33] Shufan Li, Harkanwar Singh, and Aditya Grover. Instructany2pix: Flexible visual editing via multimodal instruction following. arXiv preprint 23 2023.
[34] Sijia Li, Chen Chen, and Haonan Lu, Moecontroller: Instruction-based based arbitrary image manipulation with mixture-of of expert controllers. arXiv preprint arXiv:2309.04372. 2023.
[35] Yuheng Li. Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jac Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22511 1-22211,,,, 2023.
[36] Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. LIm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models, 2024.
[37] Shiyu Liu, Yucheng Han, Peng Xing, Fukun Yin, Rui Wang. Wei Cheng, Jiaqi Liao, Yingming Wang. Honghao Fu, Chunrui Han, et al. Steplx-edit: A practical framework for general image editing. arXiv preprint 25044 17761, 2025.
[38] Chaojie Mao, Jingfeng Zhang, Yulin Pan. Zeyinzi Jiang, Zhen Han, Yu Liu, and Jingren Zhou, Ace++: Instruction based image creation and editing via context-aware content filling, 2025.
12

---

[39] Konstantin Mishchenko and Aaron Defazio. Prodigy: An expeditiously adaptive parameter-free learner. arXiv preprint arXiv:2306.06011, 2023.
[40] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 4296-4304, 2024.
[41] Thanh Tam Nguyen, Zhao Ren, Trinh Pham, Thanh Trung Huynh, Phi Le Nguyen, Hongzhi Yin, and Quoc Viet Hung Nguyen. Instruction-guides editing controls for images and multimedia: A survey in Ilm era. arXiv preprint Xiv:2411.09955 2024.
[42] Byong Mok Oh, Max Chen, Julie Dorsey, and Frédo Durand. Image-base based modeling and photo editing. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 433-442, 2001.
[43] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.0719; 2023.
[44] Zexu Pan, Zhaojie Luo, Jichen Yang. and Haizhou Li. Multi-modal attention for speech emotion recognition. arXiv preprint arXiv:2009.04100 2020.
[45] Rishubh Parihar, VS Sachidanand, Sabariswaran Mani, Tejan Karmali, and R Venkatesh Babu. Precisecon- trol: Enhancing text-to-image diffusion models with fine-grained attribute control. In European Conference on Computer Vision, pages 469-487. Springer, 2024.
[46] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195 4205, 2023.
[47] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.0199 2023.
[48] Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, al. Unicontrol: A unified diffusion model for controllable visual generation in the wild. arXiv preprint arXiv:2305. 2023.
[49] Scott Reed and Nando De Freitas. Neural programmeer-inerrpetoor. arXiv preprint arXiv: 151.0669, 2015.
[50] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng. Hao Zhang, Feng Li, Jie Yang. Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks, 2024.
[51] Francira Rivest, Pierre Soille, and Serge Beucher. Morphological gradients. Journal of Electronic Imaging, 2(4):326-336, 1993.
[52] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2022.
[53] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffiusion models. 2022.
[54] Noam Rotstein, Gal Yona, Daniel Silver, Roy Velich, David Bensaïd, and Ron Kimmel. Pathways on the image manifold: Image editing via video generation. arXiv preprint arXiv:2411.168) 2024.
[55] Khairul Anuar Mat Said and Asral Bahari Jambek. Analysis of image processing using morphological erosion and dilation. In Journal of Physics: Conference Series, volume 2071, page 012033. TOP Publishing, 2021.
[56] Shelly Shcynin, Adam Polyak, Uriel Singer. Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8871-8879, 2024.
[57] Yichun Shi, Peng Wang, and Weilin Huang, Seededit: Align image re-generation to image editing, arXiv preprint arXiv:2411.06688 2024.
13

---

[58] Yujun Shi, Chuhui Xue, Jun Hao Liew, Jiachun Pan. Hanshu Yan, Wenqing Zhang. Vincent YF Tan. and Song Bai. Dragdiffusiin: Harnessing diffusion models for interactive point-based image editing. In Pruceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8839-8849. 2024.
[59] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 368:127063. 2024.
[60] Zhenxiong Tan, Songhua Liu, Xingyi Yang. Qiaochu Xue, and Xinchao Wang. Ominicontrol: Minimal and universal control for diffusion transformer. arXiv preprint arXiv:2411.15099 2024.
[61] Zhenxiong Tan, Qiaochu Xue, Xingyi Yang. Songhua Liu, and Xinchao Wang. Ominicontrol2: Efficient conditioning for diffusion transformers. arXiv preprint arXiv:2503.0m0 2025.
[62] Nikolaos Tsagkas, Jack Rome, Subramanian Ramamoorthy, Oisin Mac Aodha, and Chris Xiaoxuan Lu. Click to grasp: Zero-shot precise manipulation via visual diffusion descriptors. In 2024 JEEE/RS] International Conference on Intelligent Robots and Systems (IROS), pages 11610-11617, IEEE, 2024.
[63] Ting-Chun Wang, Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[64] Zhenyu Wang, Aoxue Li, Zhenguo Li, and Xihui Liu, Genartist: Multimodal Ilm as an agent for unified image generation and editing. Advances in Neural Information Processing Systems, 77:128374-128395, 2024.
[65] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2455 2022.
[66] Chenyuan Wu, Pengfei Zheng, Ruiran Yan, Shitao Xiao, Xin Luo, Yueze Wang, Wanli Li, Xiyan Jiang, Yexin Liu, Junjic Zhou, Ze Liu, Ziyi Xia, Chaofan Li, Haoge Deng, Jiahao Wang, Kun Luo, Bo Zhang, Defu Lian, Xinlong Wang, Zhongyuan Wang, Tiejun Huang, and Zheng Liu. Omnigen2: Exploration to advanced multimodal generation, 2025.
[67] Shaojin Wu, Mengqi Huang, Wenxu Wu, Yufeng Cheng, Fei Ding, and Qian He. Less-to-more gener- alization: Unlocking more controllability by in-context generation. arXiv preprint arXiv:2504.02166 2025.
[68] Bin Xia, Yuechen Zhang, Jingyao Li, Chengyao Wang, Yitong Wang, Xinglong Wu, Bei Yu, and Jiaya Jia. Dreamomni: Unified image generation and editing. arXiv preprint arXiv:2412.17099 2024.
[69] Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing. Ruiran Yan, Chaofan Li, Shut- ing Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation. arXiv preprint arXiv:2409. 11300, 2024.
[70] Xingqian Xu, Jiayi Guo, Zhangyang Wang, Gao Huang, Irfan Essa, and Humphrey Shi. Prompt-free diffusion: Taking" text" out of text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8682-8692, 2024.
[71] Shiyuan Yang, Xiaodong Chen, and Jing Liao. Uni-paint: A unified framework for multimodal image inpainting with pretrained diffusion model. In Proceedings of the s1 ACM International Conference on Multimedia, pages 3190-3199, 2023.
[72] Siwei Yang, Mude Hui, Bingehen Zhao, Yuyin Zhou, Nataniel Ruiz, and Cihang Xic. Complex Hamplex-Editt Edit: Cot-like instruction generation for
controllable image editing benchmark, 2025.
[73] Qifan Yu. Wei Chow, Zhongqi Yue, Kaihang Pan, Yang Wu, Xiaoyang Wan, Juncheng Li, Siliang Tang, Hanwang 7hang, and Yueting Zhuang. Anyedit: Mastering unified high-quality image editing for any idea. arXiv preprint arXiv:241 15738, 2024.
[74] Haobo Yuan, Xiangtai Li, Tao Zhang, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, and Ming-Hsuan Yang. Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos, 2025.
[75] Hong Zhang, Zhongjie Duan, Xingjun Wang. Yingda Chen, Yuze Zhao, and Yu Zhang, Nexus-gen: A unified model for image understanding, generation, and editing, arXiv preprint 2rX1: 2025.
14

---

[76] Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guided image editing. Advances in Neural Information Processing Systems, 36:31428-31449, 2023.
[77] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3836-3847. 2023.
[78] Xinyu Zhang, Mengxue Kang, Fei Wei, Shuang Yuhe Liu, and Lin Tie: Revolutionizing text-based image editing for complex prompt following and high fidelity editing, 2024.
[79] Yuxuan Zhang, Yirui Yuan, Yiren Song, Haofan Wang, Jiaming Liu. Easycontrol: Adding efficient and flexible control for diffusion transformer, 2025.
[80] Zechuan Zhang, Xie, Yu Lu, Zongxin Yang. and Yi Yang. In-context edit: Enabling instructional image editing with io-tontext generation in large scale diffusion transformer, 2025.
[81] Haozhe Zhao, Xiaojian Shawn Ma, Liang Chen, Shuzheng Si, Rujie Wu, Kaikai An, Peiyu Yu, Minjia Zhang, Qing Li, and Baobao Chang. Ultraedit: Instruction-based fine-grained image editing at scale. Advances in Neural Information Processing Systems, 37:3058-3088 2024.
[82] Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan- Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffuusin models. Advances in Neural Information Processing Systems, 36:11127-11155 2023.
15

---

Technical Appendices and Supplementary Material
In this part, we provide additional algorithm illustration, implementation details, more comparison results, more visualization results, and more analysis and discussions the proposed approach.
A Algorithm Illustration
To better elaborate the details of the proposed IEAP, we provide an algorithmic illustration for the whole pipeline in Alg. [
Algorithm IEAP: Image Editing As Programs
Input:
I: input image path
T: original instruction
RRo__oocalizaton, RoI_ _op_innann, Global_ Lobal_,Tranfformaiiino editing primitives
cot cot_with..gt():: CoT prompt to GPT-4o
extractt_instruutionns()::: parse CoT output
infer_witth_DiT(pp.. invoke DiT for primitive op
roi roi_localizzation/ i returns mask for region of interest
fusion(41,12)> blends two intermediate outputs
layout_change(4 instr): compute geometric transform
Output: final edited image /*
1: uri + aacode_imagee soagee__t____ to _o_datauri(( (I)
2: (C, + cot_with._gt(uri,,
Categories and instructions
3: [(]) +-
4: for to |C' do
5:
cat C(i). instr T
6:
if cat E (Add, Remove, Replace) then
7:
M + roirroloooo roi__ooaaiizaaiioo( instr)
8:
I' infer_ with__ihhhhhoIIII Inpainting, M, instr)
9:
I(() l'
10:
else cat = Action Change then
11:
M + [oo__occalizationn[f[11] instr)
12:
1og -- infer__wtt__ii
Inpainting, M, instr)
13:
Iast infer_with_ Editing, [(i-1), instr)
14:
[(() +- infer_ Compositing, fusion Iact): instr)
15:
else cal E (Move then
16:
M t www_rooooolala 1o___oocaa1iiiatioo(((11) instr)
17:
Ty +- inferr_with_DiT(BBo. Inpainting, M, instr)
I. Layouts.chagge((--iiinnss
((() +- infer_vith.DsTT(Ro) Compositing, fusi fusion(Ibg. 4tc), instr)
20:
else if cat E (Appearance Bhange,Backgroos Change,
Color
Change, Expression Change} then ((i-1),instr) instr) else if cat {Tone Transfer, Style Change} then
((0) +- infer_eriiia innfer_with_DiTeeeee Editing, ((i-1),
23:
24:
(0) + Lnnfer_withDiiT(C1.... Transformation, (i-1), instr)
else
26:
raise Value category: "cat")
end if
28:
end for
29:
return 7(C))
16

---

Change the global atmosphere or lighting conditions, e.g.: change the weather to B Implementation Details
In this section, we present the prompts employed to leverage VLM for CoT reasoning over complex instructions, providing further details on the layout-adjustmen prompts.
Below are the detailed prompts used to invoke the VLM for the CoT process on complex instructions:
Now you are an expert in image editing, Based on the given single image, what atomic image editing instructions should be if the user wants to (instruction]? Let's think step by step.
Atomic instructions include 13 categories as follows:
Add: Introduce new object, person, or element into the image, e.g.: add car on the road
Remove: Eliminate an existing object or element from the image, e.g.: remove the sofa in the image
Color Change: Modify the color of specific object, e-g.: change the color of the shoes to blue
Material Change: Alter the surface material or texture of an object, e.g.: change the material of the
sign like stone
Action Change: Modify the pose action of an instance, c.g.: change the action of the boy to
raising hands
Expression Change: Adjust the facial expression, e.g.: change the expression to smiling
Replace: Substitute one object in the image with a different object, e.g.: replace the coffee with an apple
Background Change: Change the background scene to another, e.g.: change the background into
forest
Appearance Change: Modify visual attributes such as patterns or accessories, e.g.: make the cup have a floral pattern
Move: Change the spatial position of an object within the image, e.g.: move the plane to the left
Resize: Adjust the scale or size of an object, e.g.: enlarge the clock
Tone Transfer:
foggy
Style Change: Modify the entire image to adopt different visual style, e.g.: make the style of the
image cartoon
Respond *only* with numbered list. Each line must begin with the category in square brackets, then the instruction. Please strictly follow the atomic categories. The operation (what) and the target (to what) are crystal clear. Do not split replace to add and remove. Always place [TTne Transfer] and [Style Change] instructions at the end of the list. For example:
[Add] add a car on the road
2. [Color Change] change the color of the shoes to blue
3. [Move] move the lamp to the left
Do not include any extra text, explanations, JSON markdown, just the list.
Below are the detailed prompts used adjust the layout of move and resize operations:
You are an intelligent bounding box editor. will provide you with the current bounding boxes and the editing instruction. Your task is to generate the new bounding boxes after editing. Let's think step by step.
The images of size 512x512. The top-left corner has coordinate [0, 0]. The bottom-right corner has coordinnate [512, 512]. The bounding boxes should not overlap or go beyond the image boundaries. Each bounding box should be in the format of (object name, [top-left coordinate, top-left coordinate, bottom-right x coordinate, bottom-right y coordinate]),
Do not add new objects delete any object provided in the bounding boxes. Do not change the size or the shape of any object unless the instruction requires so.
Please consider the semantic information of the layout. When resizing, keep the bottom-left corner fixed by default. When swaping locations, change according to the center point. If needed, you can make reasonable guesses. Please refer the examples below: Input bounding boxes: [("bed", [50, 300, 450, 450]), ("pillow", [200, 200, 300, 230)] Editing instruction: Move the pillow the left side of the bed. Output bounding boxes: [("bed", [50, 300, 450, 450]), ("pillow", [70, 270, 170, 300])
17

---

Input bounding boxes: [('a green car', [21, 281, 232, 440]), ('a blue truck', (269, 283, 478, 443]), Caa red air balloon', [66, 8, 211, 143]), ("a bird', [296, 42, 439, 142))]
Editing instruction: Move the car to the right.
Output bounding boxes: [("a green car', [81, 281, 292, 440J), ("a blue truck', [269, 283, 478, 443]),
('a red air balloon', [66, 8, 211, 143]). ("a bird', [296, 42, 439, 142])] Input bounding boxes: [("sofa", [100, 300, 400, 400D, ("dog", [150, 250, 250, 300]] Editing instruction: Enlarge the dog.
Output bounding boxes: [(" sofa", [100, 300, 400, 400)). ("dog", [150, 225, 300, 300]] Input bounding boxes: [("chair", [100, 350, 200, 450D. ("lamp", [300, 200, 360, 300)) Editing instruction: Swap the location of the chair and the lamp.
Output bounding boxes: (("chair", [280, 200, 380, 300D, ("lamp", [120, 350, 180, 450))
Now, the current bounding boxes is [bbox), the instruction (instruction).
Below the detailed prompts used adjust the layout of add operations:
You are intelligent bounding box editor. I will provide you with the current bounding boxes and an add editing instruction. Your task is to determine the new bounding box the added object. Let's think step by step.
The images are of size 512x512. The top-left corner has coordinate [0, Oj. The bottom-right corner has coordinnate [512, 512].
The bounding boxes should not go beyond the image boundaries. The new box must be at least as large as needed to encompass the object. Each bounding box should be in the format of (object name, [top-left x coordinate, top-left coordinate, bottom-right x coordinate, bottom-right coordinate)), Do not delete any object provided the bounding boxes. Please consider the semantic information of the layout, preserve semantic relations.
needed, you can make reasonable guesses. Please refer to the examples below: Input bounding boxes: [(' green car', [21, 281, 232, 440])] Editing instruction: Add bird on the green car.
Output bounding boxes: [("a bird', [80, 150, 180, 281]), ('a green car', [21, 281, 232, 440))) Input bounding boxes: [(" ((stool', [300, 350, 380, 450))
Editing instruction: Add cat to the left of the stool.
Output bounding boxes: [("a cat', [180, 300, 300, 450D)
Input bounding boxes: [('the white cat', [200, 300, 320, 420])] Editing instruction: Add hat on the white cat.
Output bounding boxes: [('the white hat', [200, 260, 320, 310J), ('cat', [200, 300, 320, 420))] Now, the current bounding boxes is [bbox), ), the instruction finstruction).
C More Quantitative Results
Method
CLIP cm
CLIP out
4
DINO
GPT,F
GPT t
GPT 1
GPT E
InstructPix2Pix
0.847
0.264
0.092
0.829
4.50
4.40
4.26
4.39
MagicBrush
0.889
0.277
0.068
0.892
4.66
4.76
4.62
UltraEdit
0.897
0.274
0.056
0.909
3.36
4.24
4.22
3.94
ICEdit
0.925
0.277
0.057
0.915
4.60
4.80
4.76
4.72
IEAP(Ours)
0.928
0.278
0.056
0.917
4.68
4.84
4.60
4.71
Table 5: Quantitative comparison results AnyEdii Add test set.
Method
CLIP
CLIP out
L1 J
DINO
GPT,F f
GPT t GPT t
GPT avy
InstruetPix2Pix
0.800
0.202
0.108
0.721
2.74
3,42
3.20
3.12
MagicBrush
0.853
0.211
0.083
0.800
3.08
3.60
3.18
3.29
UltraEdit
0.846
0.211
0.066
0.802
2.50
3.54
3.44
3.16
ICEdit
0.895
0.212
0.054
0.875
4.06
4.48
4.32
4.29
IEAPP(Orr)
0.916
0.230
0.057
0.886
4.18
3.88
3.66
3.91
Table Quantitative comparison results on AnyEdit Remove test set.
18

---

Method
CLIP mm
CLIP mut
L1
DINO
GPT,F
GPT t
GPTxq
GPT.vg
InstructPix2Pix
0.766
0.234
0.179
0.588
3.72
3.68
3.80
3.73
MagicBrush
0.806
0.248
0.148
0.671
4.52
4.48
4.38
4.46
UltraEdit
0.779
0.242
0.142
0.621
3.80
4.40
4.40
4.20
ICEdit
0.797
0.228
0.128
0.614
3.68
4.02
4.04
3.91
IEAP(Ours)
0.866
0.252
0.099
0.701
4.68
4.68
4.48
4.61
Table 7: Quantitative comparison results on AnyEdit Replace test set.
Method
CLIP
CLIP
L1 4
DINO
GPT f GPT f GPT 1
GPT
InstructPix2Pix
0.829
0.254
0.164
0.774
3.46
3.84
3.58
3.63
MagicBrush
0.831
0.266
0.156
0.784
2.96
4.28
4.28
3.84
UltraEdit
0.847
0.259
0.157
0.781
2.92
4.22
4.24
3.79
ICEdit
0.827
0.255
0.152
0.745
2.68
4.04
4.04
3.59
TRAP(Ours)
0.848
0.267
0.154
0.798
4.66
4.86
4.68
4.73
Table 8: Quantitative comparison results on AnyEdit Action Change test set.
Method
CLIP
CLIPout
+
DINO
GPTIF
GPT
GPTAQ
GPT avv
InstructPix2Pix
0.881
0.219
0.127
0.771
3.82
4.44
4.36
4.21
MagieBrush
0,902
0.219
0.088
0.828
2.94
3.94
3.90
3.59
UltraEdit
0.923
0.211
0,074
0.867
3.48
4.40
4.40
4.09
ICEdit
0.944
0.213
0.063
0.868
3.28
4.64
4.30
4.07
IEAP(Ours)
0.963
0.223
0.058
0.903
3.88
4.44
4.38
4.23
Table 9: Quantitative comparison results on AnyEdit Relation test set.
Method
CLIP
CLIPoss
L1 4
DINO f
GPT,F
GPT t GPT t
GPT
InstructPix2Pix
0.831
0,241
0.124
0.746
2.94
3.56
3.62
3.37
MagicBrush
0.875
0.258
0.094
0.802
2.80
3.88
4.00
3.56
UltraEdit
0.908
0.262
0.073
0.889
3.22
4.38
4.38
4.00
ICEdit
0.895
0.253
0.074
0.841
3.14
4.28
4.26
3.89
IEAP(OOrr)
0.923
0.263
0.066
0.921
4.38
4.32
4.28
4.32
Table 10: Quantitative comparison results on AnyEdit Resize test set.
Method
CLIP cm
CLIP out
4
DINO
GPT
GPT f GPTAQ t
GPT
InstructPix2Pix
0.815
0.280
0.139
0.744
3.60
4.08
3.92
3.87
MagicBrush
0,852
0.294
0.094
0.815
3.96
4.32
3.98
4.09
UltraEdit
0.857
0.277
0.068
0.845
4.04
4.62
4.42
4.36
ICEdit
0.847
0.273
0.085
0.808
4.04
4.42
4.16
4.21
IEAP(Ours)
0.886
0.285
0.082
0.833
4.06
4.72
4.80
4.53
Table 11: Quantitative comparison results AnyEdit Appearance test set.
Method
CLIP km
CLIP out
L1 1
DINO
GPT,F f
GPT t GPT
GPT
InsiructPix2Pix
0.725
0.224
0.216
0.582
3.40
3.60
3.44
3.48
MagicBrush
0.746
0.230
0.228
0.567
4.58
4.38
4.46
4.47
UltraEdit
0.796
0.257
0.169
0.747
3.48
4.36
3.14
3.66
ICEdit
0.799
0.241
0.166
0.757
3.04
4.16
3.88
3.69
IFAAP(Our))
0.801
0.243
0.165
0.759
4.74
4.68
4.70
4.71
Table 12: Quantitative comparison results on AnyEdit Background Change test set.
19

---

Method
CLIP km
CLIP out
L1
DINO
GPT,F
GPT f
GPT
GPT.vg
InstructPix2Pix
0.886
0.279
0.120
0.876
3.60
4.40
4.00
4.00
MagicBrush
0.898
0.282
0.087
0.869
4.20
4.82
4.62
4.55
UltraEdit
0.890
0.280
0.065
0.87
3.80
4.40
4.20
4.13
ICEdit
0.896
0.278
0.073
0.849
4.72
4.80
4.64
4.72
IEAP(Ours)
0.911
0.276
0.059
0.876
4.62
4.72
4.78
4.71
Table 13: Quantitative comparison results on AnyEdit Color Change test set.
Method
CLIP im
L1 +
DINO 1
GPT,r
GPTre t
GPT
GPT
InstructPix2Pix
0.776
0.068
0,936
3.74
4.60
4.30
4.21
MagicBrush
0.770
0.064
0.940
3.86
4.48
4.18
4.17
UltraEdit
0.699
0.073
0,907
3.14
4.10
3,80
3,68
ICEdit
0.796
0.065
0.943
3.16
4.60
4.30
4.02
IRAP(Ours)
0.882
0.052
0.945
4.34
4.72
4.50
4.52
Table 14: Quantitative comparison results on Expression set.
Method
CLIP mm
DINO f GPTIF
GPT
GPTAQ
GPTavg
InstructPix2Pix
0.746
0.130
0.549
4.00
4.18
4.04
4.07
MagicBrush
0.778
0.110
0.621
3.36
4.06
3.84
3.75
UltraEdit
0.765
0.086
0.598
3.34
4.28
4.04
3.89
ICEdit
0.787
0.086
0.616
3.48
3.92
3.58
3.66
IEAP(Ours)
0.826
0.055
0.696
4.08
4.48
4.18
4.25
Table 15: Quantitative comparison results on Material Change test set.
Method
CLIP
L1 +
DINO 1
GPT,#
GPTFc
GPTAQ
GPTavy Tauy
InstruetPix2Pix
0.710
0.212
0,463
3.56
4.32
3,94
3,94
MagicBrush
0.692
0.214
0.440
3.12
4.64
4,00
3.92
Uhtaadit
0.703
0.201
0.467
4.02
4.8
4.62
4.48
ICFdit
0.706
0.219
0.458
4.04
4.82
4.36
4.41
IEAP(Ours)
0.922
0.097
0.915
4.44
4.64
4.44
4.51
Table 16: Quantitative comparison results on AnyEdit Style Change test set.
Method
CLIP om
CLIPows
4
DINO
GPT1F f
GPT f
GPT t
GPT avg
InstructPix2Pix
0.822
0.260
0.100
0.821
3.72
4.48
3.92
MagicBrush
0.834
0.266
0.159
0.791
3.56
4.64
3.98
4.06
UltraEdit
0,804
0.268
0.201
0.767
4.12
4.62
4.26
4.33
ICEdit
0.812
0.260
0.157
0.748
4.06
4.88
4.56
4.50
IEAP(Ours)
0.868
0.268
0.116
0.843
4.44
4.64
4.44
4.51
Table 17: Quantitative comparison results on AnyEdit Tone Transfer test set.
Method
CLIP mm
L1
DINO t
GPT,F
GPT
GPT
GPT
Inxir
0.815
0.134
0.647
3.40
4,04
4.80
4.08
MagicBrush
0.835
0.081
0.697
1.82
3.56
3.50
2.96
UltraEdit
0.833
0.066
0.756
4.02
4.02
3.54
ICEdit
0.906
0.042
0.842
2.98
4.40
3.40
3.59
IRAP(OOrr)
0.908
0.056
0.794
3.42
4.48
4.46
4.12
Table 18: Quantitative comparison results on AnyEdit Counting test set.
20

---

Method
CLIP km
L1
DINO 1
G
GPT
GPTAQ
GPT
InstructPix2Pix
0.773
0.208
0.581
3.46
4.18
4.08
3.91
MagicBrush
0.806
0.174
0.631
2.98
3.88
4.04
3.63
UltraEdit
0.825
0.167
0.669
2.82
4.38
4.38
3.86
ICEdit
0.806
0.171
0,629
3.56
4.16
4.06
3.93
IEAP(Ours)
0.833
0.169
0.662
3.88
4.44
4.52
4.28
Table 19: Quantitative comparison results on AnyEdit Implicit Change test set.
Method
CLIP
L1 +
DINO 1
GPT,r
GPT
GPTAQ
GPT
InstructPix2Pix
0.887
0.111
0.858
4.30
4.50
4.30
4.37
MagicBrush
0.900
0.100
0,874
4.12
4.36
4.54
4.34
UltraEdit
0.922
0.077
0.911
3.24
4.4
4.36
4.00
ICEdit
0.898
0.079
0.864
4.16
4.46
4.20
4.27
TRAP(Ours)
0.938
0.084
0.925
4.18
4.56
4.38
4.37
Table 20: Quantitative comparison results on AnyEdit Move test set.
Method
CL
CLIPous
+
DINO
GPTIF f
GPT f
GPTAQ t
GPT avv
InspructPix2Pix
0.688
0.243
0.189
0.742
1.04
4.38
3.92
3.11
MagieBrush
0.680
0.255
0.156
0.786
4.48
4.10
3,20
UltraEdit
0.732
0.279
0.147
0.843
1.96
4.46
3.98
3.47
ICEdit
0.810
0.289
0.155
0.811
4.18
4.42
4.68
4.43
IEAP(Ours)
0.788
0.285
0.162
0.786
3.96
4.58
4.06
4.20
Table 21: Quantitative comparison results on AnyEdit Textual Change test set.
D More Visualization Results
In this section, we provide more visualization results, as shown below:
Input
Edited Results
grass.
dog on grass.
laplog un grass.
the grazs.
Ramove
Romove the haz
Remove the nechiace.
Ronove the cat.
Remove the clother.
Replace
Replace the dog with a buokpoot.
Repince the dog with cat
Replace the dog with a cale.
Replace the dog with flowers.
Figure 10: More Visualization Results.
21

---

Input
Edited Results
20O
Action Change
Change the action the dancing.
Change the action the squating
Change the
Change
running
Move
Move the bard the left
Move the bard to the right.
Move the hird uproadd
Move the bird downward.
Resize
Zoom be the digor.
Miniff the riger.
Eniarge the tiger to twice original stoe
Misify the riger half its original size.
Appearance Change
Mishr the reorf filled butuyffiez
Make the how
Make the fost and mecklooee
Make the decorated Christanse
Hockground Change
Change the background to the forest
Change the background the beach.
Chunge the buskgroand the mouutiii.
Chunge backgroand to city street
Color Change
Change the color of the skirt to white.
Change the color of the hat so purple.
Change color of the hatr red.
Change the color of the joans black.
Figure 11: More Visualization Results.
22

---

Input
Edited Results
Material
Change the material of the
Change the naterial of the styrffoo.
Chonge the material the leuther..
Changy the material the corduray.
Expression
Chonge expression saiting
Chonge exprension to smiling
Chunge expression surpesed.
Change uppecronce to
Transfar
Change westher
Change
Change
Style Change
Change the ztyle to 8-Ott.
Change the 2ode clean fisc.
Change Atyle cartoon. Change the style to tsk paintinn
Figure 12: More Visualization Results.
Vriwe
Doai) plr dochi de
leare harld feme aadt
mede with Sagher sigee tor
tre eveny.
Seapesa
Figure 13: More Detailed Visualization Processes of the pipeline.
23

---

E Analysis and Discussions
E.1 Runtime Performance Analysis
We quantified the inference latency of IEAP across different editing types on single NVIDIA H200 GPU. For local attribute editing and overall content editing, only one atomic diffusion-based step is required. In contrast, local semantic editing typically involves multiple atomic operations, depending on the specific editing intent:
Add/Replace:
1 x Rol Localization + x Diffusion + 2 x LLM Response,
Remove:
X RoI Localization + Diffusion 1 X LLM Response,
Action Change:
2 X RoI Localization + 3 x Diffusion + 2 x LLM Response,
Move/Resize:
RoI Localization + 2 Diffusion x LLM Response.
Empirical measurements show that each diffusion step takes approximately 5s when using FLUX.1- dev and around 4s with B.U.X.I-schnell The LLM response incurs about Is latency, while Rol localization requires roughly ss. Based on these estimates. the end-to-end latencies for representative operations are approximately: Add/Replace: 8-9 s; Remove: 7-8 s; Action Change: 18-20 Move/Resize: 12-14s: s; Other operations: typically 4-5
On the AnyEdit test set, which includes 16 subtypes of editing, our method achieves a weighted average latency of approximately 7s per edit. For comparison, the end-to-end editing systems InstructPix2Pix, MagicBrush, UltraEdit, and ICEdit exhibit average latencies of 4s, 4s, ss, and 3s, respectively. Although IEAP incurs slightly higher inference latency, it provides faithful, modular, and compositional control over both simple and complex multi-step edits, extending beyond the capabilities of purely end-to-end approaches.
E.2 Limitations and Future Work
Limitations. Despite its strengths, IEAP exhibits several limitations in handling dynamic scenes and complex physical interactions. First, the Rol compositing may introduce geometric distortions or texture discontinuities when editing highly dynamic non-rigid content, such as motion-blureed instances, and fluid or smoke effects. For example, in the task of "changing the cat's action to jumping," in Fig- 6 the rapid motion of fur can produce blurred regions that fail to blend naturally with the background. Second, Rol compositing struggles to simulate physically consistent lighting effects in scenes with reflective refractive surfaces, sometimes resulting in mismatched shadow directions and illumination conflicts between edited objects and their environments. For example, in the task of "change the action of the woman to dancing." in Fig. 4 the shadows before and after editing remain the same, but the action of the woman has changed, so it is unnatural. Third, the DiT-based architecture and multi-stage atomic operations incur substantial inference latency for 5 to 9s per operation on a single H100 GPU, precluding real-time interactivity in applications such as AR/VR. Finally, the requirement for high-memory GPUs like NVIDIA H100 (80 GB) limits reproducibility for snnource-constraiie quality degradation over successive operations.
researchers, and
editing can exacerbate image
Future Work. As for future work, several avenues may be pursued to overcome the identified limi- tations. To begin with, physics-aware compositing techniques and motion-compeenattel inpainting could be explored to better accommodate dynamic blur and fluid effects, thereby ensuring seamless integration of non-rigid edits. Mcanwhile, differentiable lighting models or neural rendering modules may be incorporated to enforce global illumination consistency, particularly in reflective and refrac- tive contexts. On the performance front, model distillation, operation fusion, and sparse attention strategies could be investigated to reduce per-operation latency and facilitate interactive editing. To enhance accessibility, memory optimization and support for smaller-footprin architectures amenable to commodity GPUs may be implemented. Moreover, iterative refinement and error-correction mechanisms may be developed to mitigate quality degradation over successive editing steps. Fur- thermore, beyond still-image editing, an extension to video-based complex instruction editing could be considered, where temporal coherence and motion consistency present additional challenges and opportunities for dynamic, multi-step visual manipulation.
24

---

E.3 Societal Impacts and Ethical Safeguards
Positive Societal Impacts. The proposed IEAP framework introduces modular and interpretable approach to complex image editing, which holds significant potential to benefit a range of creative and technical domains. By decomposing high-level visual instructions into atomic operations, IEAP enables users to perform multi-step edits with enhanced precision and control. This capability particularly valuable in digital content creation, advertising. and education, where fine-gran fine-grained manipulation of visual content is often required. For example, IEAP's ability to support structurally inconsistent modifications can streamline visual storytelling workflows or facilitate the generation of accurate scientific visualizations for publications and teaching materials. Furthermore, its potential extensions to fields such as medical imaging by enabling localized enhancement diagnostic visuals, and accessibility technology by generating descriptive visual representations for users with visual impairments, demonstrate the framework's 's broader societal utility and interdisciplinary relevance.
Negative Societal Impacts and Ethical Safeguards. Despite its benefits, IEAP's high-fidelity editing capabilities also introduce ethical risks, particularly in the domains misinformation and privacy. The framework's 's precision in altering visual content could be misused for the creation of deepfakes or manipulated images intended for disinformation, identity falsification, or reputational harm. Operations such as "Remove" or "Replace" could be exploited to tamper with sensitive or private imagery, potentially infringing on individual rights.
To address these concerns, the development and deployment of IEAP adhere to strict ethical standards. Specifically, safeguards include the implementation of data filtering pipelines, such as the use of PYTT-4o-ilterre subsets of AnyEdit and the compliance-oriented oriented CelebHQ-FM dataset, to reduce harmful biases and content. Additionally, the modular nature IEAP facilitates transparency and traceability in the editing process, supporting future content provenance systems designed to detect and flag manipulated media. All these safeguards jointly contribute to ongoing efforts in Al safety and accountability.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed the limitations in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully disclose all the information needed reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

Answer: [Yes]
Justification:
Please refer to the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.com wttpcccpppbccccceec guwdes/dde/eeee Cooeeeoodeesssseeeeeee http:///d///uuudddd//eee/e/ee/s/
ssioiolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We explained the experimental setting and details in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Justification: We follow the other relevant works which do not include this part.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
28

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We explained the compute resources Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to the supplemental material.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
29

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Please refer to the supplemental material.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the original paper that produced the code package or dataset and the name of the license (e.g.. CC-BY 4.0) is included for each asset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
30

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subject. Guidelines: The paper does not involve crowdsourcing nor research with human subjects. The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
31

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
32


Reviews:
[{"review_id": "LaksmZvKDD", "reviewer": "Reviewer_K6UQ", "summary": "This paper proposes a novel image editing framework, IEAP, based on the Diffusion Transformer architecture. IEAP decomposes complex natural language instructions into a sequence of atomic operations, including RoI localization, inpainting, editing, compositing, and global transformations. These operations are executed sequentially by a neural program interpreter to accommodate complex editing tasks. Experimental results demonstrate that IEAP outperforms existing methods in multiple editing benchmarks, especially in handling complex multi-step instructions.", "questions": "1. Many of the defined atomic operations rely on the segmentation model M_seg. Can M_seg locate objects at fine-grained levels? For example, can it accurately localize “people wearing black clothes” in a complex scene?\n\n2. How does IEAP handle instructions that involve editing multiple objects of the same type? For example, in the 5th image of Row 3 in Figure 3, how does IEAP determine the order of atomic operations if the instruction is “enlarge all the unicorns”?\n\n3. In the first row of Figure 4, the intermediate image after RoI inpainting removes the person but leaves its shadow, which is corrected in the next step. I wonder why IEAP didn’t treat the shadow as part of the environment during the next RoI Editing operation?\n\n4. Will different VLMs/LLMs (used for instruction decomposition or RoI localization) lead to differences in editing performance?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1. The idea of decomposing image editing into atomic operations is interesting. By transforming instructions into a sequence of atomic operations and executing them procedurally, the paper presents an editing framework that enables compositional generalization. That is, use limited atomic operations to model a large set of edit instructions. The use of LLMs for instruction parsing enhances the model’s ability to understand natural language and improves the correspondence between instructions and editing behaviors.\n\n2. IEAP achieves nearly the best performance on different editing benchmarks. The paper also provides sufficient visual examples that make the method easy to understand and evaluate.\n\nWeaknesses\n\n1. It is unclear whether the defined set of atomic operations can adequately cover the editing instructions in open-world scenarios. \n\n2. The paper appears to lack results in the editing of multiple objects. I notice that the examples primarily involve clearly defined editing targets. However, in practice, editing targets may be ambiguous. For example, the target can be “all objects of type A in the scene”.\n\n3. The pipeline depends on accurate RoI localization. A concern is that if the localization is inaccurate or the instruction is vague, the editing may fail. It is recommended to discuss the robustness of IEAP in such cases.\n\n4. This paper seems to lack discussions of how different VLMs/LLMs influence the editing quality, including the influence on instruction decomposition or RoI localization.", "was_revised": true, "final_justification": "The authors provided additional results on various LLMs/VLMs, multi-object manipulation, and robustness to inaccurate localization. They also explained the visualization results I was concerned about. I think these results have addressed my earlier concerns. So I will maintain my score.", "rebuttal": "We sincerely thank Reviewer K6UQ for the constructive feedback and thoughtful comments. We are happy that the reviewer finds our idea interesting and paper easy to follow. We address the concerns point by point below:\n\n- > **(W1: Generalizability to open-world editing instructions)** It is unclear whether the defined set of atomic operations can adequately cover the editing instructions in open-world scenarios.\n\n  Thanks for the valuable question regarding the generalizability of our atomic operations to open-world scenarios. Our 5 atomic operations are elicited from a comprehensive survey of image editing tasks and represent a minimal yet sufficient set that can be flexibly composed to cover diverse editing instructions. \n\n  Notably, **IEAP is capable of addressing previously unseen editing tasks**—which are often challenging for end-to-end models, particularly those underrepresented in training data—such as layout reconfiguration (e.g., swapping the positions of objects A and B or rotating object C). IEAP achieves this by organically composing atomic operations to fulfill complex instructions.\n\n  To validate this capability, we select layout-editing test cases and evaluate performance using GPT-based accuracy (instruction faithfulness, as introduced in Section 5.1 of our paper). The results, compared with other approaches, are presented in the table below.\n\n  | Approaches            | InstructP2P | MagicBrush | UltraEdit | ICEdit | IEAP(ours) |\n  | --------------------- | ----------- | ---------- | --------- | ------ | ---------- |\n  | GPT$_{IF}$ $\\uparrow$ | 2.65        | 2.55       | 3.2       | 3.05   | **3.85**   |\n\n- > **(W2: Multi-object manipulation)** The paper appears to lack results in the editing of multiple objects. I notice that the examples primarily involve clearly defined editing targets. However, in practice, editing targets may be ambiguous. For example, the target can be “all objects of type A in the scene”.\n\n  We thank the reviewer for highlighting this. The example raised by the reviewer can be addressed by our RoI localization module, implemented as Sa2VA in our experiments, which is capable of performing segmentation based on a wide range of complex instructions. To verify this, we construct an evaluation dataset by selecting editing instructions related to multi-object editing and find that IEAP successfully handles such instructions. Comparative results in GPT-based accuracy (GPT instruction faithfulness introduced in Sec 5.1 in our paper) with other methods are shown in the table below. We will include them and the qualitative results in the revised version to enrich our evaluation.\n\n  | Approaches            | InstructP2P | MagicBrush | UltraEdit | ICEdit | IEAP(ours) |\n  | --------------------- | ----------- | ---------- | --------- | ------ | ---------- |\n  | GPT$_{IF}$ $\\uparrow$ | 3.3         | 3.9        | 3.8       | 4.2    | **4.5**    |\n\n- > **(W3: Robustness to inaccurate localiation)** The pipeline depends on accurate RoI localization. A concern is that if the localization is inaccurate or the instruction is vague, the editing may fail. It is recommended to discuss the robustness of IEAP in such cases.\n\n    We thank the reviewer for this insightful question. We adopt Sa2VA-8B in our work due to its strong RoI localization capabilities. To further assess the impact of localization accuracy, we compare different Sa2VA variants (1B, 4B, 8B, and 26B). As shown in the table below, all models perform similarly on the Local Semantic Editing test set of AnyEdit, suggesting that our pipeline remains robust even under less accurate localization.\n\n    |Model Size|CLIP$_{im}$ $\\uparrow$|CLIP$_{out}$ $\\uparrow$|L1 $\\downarrow$|DINO $\\uparrow$|\n    |-------------------|-----------|----------|---------|---------|\n    |1B|0.901|0.251|0.084|0.848|\n    |4B|0.904|0.252|0.081|0.852|\n    |8B|**0.907**|0.252|0.081|0.854|\n    |26B|0.906|**0.253**|**0.079**|**0.854**|\n\n- > **(W4/Q4: Results by various LLMs/VLMs)** This paper seems to lack discussions of how different VLMs/LLMs influence the editing quality, including the influence on instruction decomposition or RoI localization. Will different VLMs/LLMs (used for instruction decomposition or RoI localization) lead to differences in editing performance?\n\n    We thank the reviewer for this valuable feedback. We have supplemented comparative experiments involving different VLMs and LLMs to analyze their influence on editing quality.\n\n    For complex instruction decomposition, we evaluate challenging instructions from the MagicBrush dataset across 2 VLMs (GPT-4o and Gemini-2.0-flash) and 2 LLMs (GPT-4 and GPT-3.5-turbo). The table below reports the accuracy of various models: \n\n    |Model|GPT-4o (as VLM)|Gemini-2.0-flash (as VLM)|GPT-4 (as LLM)|GPT-3.5-turbo (as LLM)|\n    |-------------------|----------|---------|---------|---------|\n    |Acc(%)|**100**|96.7|90|76.7|\n\n    We find that the instruction decomposition step is robust across different VLMs. However, when the original image is excluded—i.e., using only LLMs—we observe a notable performance drop.\n\n    RoI localization is implemented by Sa2VA. The ablation studies regarding various models are presented in our response to the previous question. \n    \n    For text-based ROI extraction, e.g., identifying “bananas” in “remove the bananas”, which is a straightforward task, all tested models, including GPT-3.5-turbo, GPT-4, GPT-4o, and Gemini-2.0-flash, consistently accomplish successfully.\n\n    In our method, LLMs also assist in layout modifications: for “add” operations, they determine the appropriate RoI for new elements; for “move” and “resize” operations, they predict the RoIs for the items after editing. Quantitative comparisons across these models regarding this step are shown below:\n\n    Add:\n    |Model|CLIP$_{im}$ $\\uparrow$|CLIP$_{out}$ $\\uparrow$|L1 $\\downarrow$|DINO $\\uparrow$|\n    |-------------------|-----------|----------|---------|---------|\n    |Gemini-2.0-flash (as LLM)|0.926|0.276|0.060|0.912|\n    |GPT-3.5-Turbo (as LLM)|0.903|0.265|0.070|0.901|\n    |GPT-4 (as LLM)|0.929|0.279|0.058|0.917|\n    |GPT-4o (as LLM)|0.928|0.278|0.056|0.917|\n    |GPT-4o(as VLM)|**0.935**|**0.281**|**0.056**|**0.919**|\n\n    Move/Resize:\n    |LLM|CLIP$_{im}$ $\\uparrow$|CLIP$_{out}$ $\\uparrow$|L1 $\\downarrow$|DINO $\\uparrow$|\n    |-------------------|-----------|----------|---------|---------|\n    |Gemini-2.0-flash (as LLM)|0.941|0.243|0.062|0.912|\n    |GPT-3.5-Turbo (as LLM)|0.919|0.236|0.068|0.901|\n    |GPT-4 (as LLM)|0.939|0.243|**0.061**|0.913|\n    |GPT-4o (as LLM)|0.943|0.243|0.062|0.912|\n    |GPT-4o (as VLM)|**0.945**|**0.243**|0.062|**0.913**|**\n    \n    In general, the layout modification capacity is robust across various LLMs/VLMs.\n\n- > **(Q1: Fine-grained localization)** Many of the defined atomic operations rely on the segmentation model M_seg. Can M_seg locate objects at fine-grained levels? For example, can it accurately localize “people wearing black clothes” in a complex scene?\n\n    We appreciate the reviewer’s concern regarding the fine-grained localization ability of our segmentation model. Here, we specifically test M$_{seg}$ on 15 challenging instructions involving fine-grained references, such as “the biggest bear” and “the cat who is stepped on”, and find that the ROI localization model successfully segments all expected regions, demonstrating robust performance for such fine-grained object localization tasks.\n\n- > **(Q2: Editing multiple objects of the same type)** How does IEAP handle instructions that involve editing multiple objects of the same type? For example, in the 5th image of Row 3 in Figure 3, how does IEAP determine the order of atomic operations if the instruction is “enlarge all the unicorns”?\n\n    Thanks for the valuable question. For instructions involving multiple objects of the same type, our current pipeline processes them collectively—for example, by extracting a joint mask of all unicorns and performing the corresponding edit. As demonstrated in our response to W2, this approach yields results that surpass those of end-to-end methods.\n\n- > **(Q3: The person's shadow)** In the first row of Figure 4, the intermediate image after RoI inpainting removes the person but leaves its shadow, which is corrected in the next step. I wonder why IEAP didn’t treat the shadow as part of the environment during the next RoI Editing operation?\n\n    Thanks for the detailed inspection. In fact, IEAP does treat the shadow as part of the background. To enhance the clarity, we would like to rephrase the operation flow of such editing tasks involving action change. As shown in the ROI-Editing plot of Figure 4(a), using an end-to-end editing model alone may result in inconsistencies between the pre- and post-editing backgrounds. So, our workflow involves three steps: first performing the action change, then extracting the modified object (only contains the person without the shadow), and finally blending it with the inpainted original background (contains the shadow in the original image) seamlessly. We are sorry for the misunderstanding and will update this figure in the revision to enhance the clarity.\n\nWe would like to express our gratitude again to Reviewer K6UQ for the constructive feedback to improve this work. We hope that our response will alleviate the reviewer’s concern, and we are glad to have further discussions if there are remaining concerns."}, {"review_id": "hW9vv42T1U", "reviewer": "Reviewer_ogfC", "summary": "This paper describes a new approach to image editing as a series of operation as opposed to a one shot generation. Operations are broken down into 5 atomic operations that can be combined to create better images. The approach was successfully applied and have been demonstrated to have SOTA performance compared to other methods.", "questions": "1. How would using different LLMs affect the approach? \n2. How would different starting diffusion affect the approach?\n3. It might be good to use a different LLM evaluator other than GPT4 since there would be bias (the underlying operations are based on GPT4 understanding.)", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n+ Paper is written really well and easy to follow. \n+ Evaluations and comparisons are pretty comprehensive and demonstrated to have SOTA performance.\n+ Good breakdown and analysis of edit operations.\n\nWeak\n- The method is primarily an integration of multiple methods and demonstrated to be a really great approach, and not really model training and improvements.\n- Would be great to have some understanding of how the underlying models used might affect the overall approach (for example using a different LLM model GPT3 or others)", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere gratitude to Reviewer ogfC for the insightful and encouraging feedback of our work. We feel glad that the reviewer finds our paper clear, comprehensive in evaluation, and strong in performance. We address the concerns and questions as follows.\n\n- > **(W1: Contributions of IEAP)** The method is primarily an integration of multiple methods and demonstrated to be a really great approach, and not really model training and improvements.\n\n    We appreciate the reviewer’s concern. While our framework integrates multiple components, its novelty lies in rethinking image editing as program. This paradigm enables decomposing complex instructions into modular and interpretable steps, addressing the limitations of monolithic end-to-end models.\n\n    Moreover, we made model-level contributions in critical submodules. Specifically, we trained four specialized models tailored for our task, i.e., RoI inpainting, RoI editing, RoI compositing, and global transformation. These four operators are deliberately designed to disentangle and cover the full range of spatial-layout-sensitive editing operations, enabling capabilities that remain unachievable with other existing end-to-end models.\n\n- > **(W2/Q1/Q3: Results by different LLMs/VLMs)** Would be great to have some understanding of how the underlying models used might affect the overall approach. How would using different LLMs affect the approach? It might be good to use a different LLM evaluator other than GPT4.\n\n    We thank the reviewer for this valuable suggestion. We have supplemented comparative experiments involving different VLMs and LLMs to analyze their influence on editing quality.\n  \n    For complex instruction decomposition, we evaluate challenging instructions from the MagicBrush dataset across 2 VLMs (GPT-4o and Gemini-2.0-flash) and 2 LLMs (GPT-4 and GPT-3.5-turbo). The table below reports the accuracy of various models: \n\n    |Model|GPT-4o (as VLM)|Gemini-2.0-flash (as VLM)|GPT-4 (as LLM)|GPT-3.5-turbo (as LLM)|\n    |-------------------|----------|---------|---------|---------|\n    |Acc(%)|**100**|96.7|90|76.7|\n\n    We find that the instruction decomposition step is robust across different VLMs. However, when the original image is excluded—i.e., using only LLMs—we observe a notable performance drop.\n\n    RoI localization is implemented by Sa2VA. The ablation studies regarding various models are presented in our response to the previous question. \n    \n    For text-based ROI extraction, e.g., identifying “bananas” in “remove the bananas”, which is a straightforward task, all tested models, including GPT-3.5-turbo, GPT-4, GPT-4o, and Gemini-2.0-flash, consistently accomplish successfully.\n\n    In our method, LLMs also assist in layout modifications: for “add” operations, they determine the appropriate RoI for new elements; for “move” and “resize” operations, they predict the RoIs for the items after editing. Quantitative comparisons across these models regarding this step are shown below:\n\n    Add:\n    |Model|CLIP$_{im}$ $\\uparrow$|CLIP$_{out}$ $\\uparrow$|L1 $\\downarrow$|DINO $\\uparrow$|\n    |-------------------|-----------|----------|---------|---------|\n    |Gemini-2.0-flash (as LLM)|0.926|0.276|0.060|0.912|\n    |GPT-3.5-Turbo (as LLM)|0.903|0.265|0.070|0.901|\n    |GPT-4 (as LLM)|0.929|0.279|0.058|0.917|\n    |GPT-4o (as LLM)|0.928|0.278|0.056|0.917|\n    |GPT-4o(as VLM)|**0.935**|**0.281**|**0.056**|**0.919**|\n\n    Move/Resize:\n    |LLM|CLIP$_{im}$ $\\uparrow$|CLIP$_{out}$ $\\uparrow$|L1 $\\downarrow$|DINO $\\uparrow$|\n    |-------------------|-----------|----------|---------|---------|\n    |Gemini-2.0-flash (as LLM)|0.941|0.243|0.062|0.912|\n    |GPT-3.5-Turbo (as LLM)|0.919|0.236|0.068|0.901|\n    |GPT-4 (as LLM)|0.939|0.243|**0.061**|0.913|\n    |GPT-4o (as LLM)|0.943|0.243|0.062|0.912|\n    |GPT-4o (as VLM)|**0.945**|**0.243**|0.062|**0.913**|**\n    \n    In general, the layout modification capacity is robust across various LLMs/VLMs.\n\n- > **(Q2: Results by different starting diffuion)** How would different starting diffusion affect the approach?\n\n  Thanks for this insightful question. To assess the performance of different starting diffusion, we compare two diffusion backbones: FLUX.1-dev (used in our paper) and FLUX.1-schnell on AnyEdit test set. As shown below, our method consistently performs well, but FLUX.1-dev yields better overall performance.\n\n  |Starting Diffusion|CLIP$_{im}$ $\\uparrow$|L1 $\\downarrow$|DINO $\\uparrow$|GPT $\\uparrow$| \n  |-------------------|-----------|----------|---------|---------|\n  |FLUX.1-dev|**0.882**|**0.096**|**0.825**|**4.41**|\n  |FLUX.1-schnell|0.858|0.108|0.779|4.23|\n\n We would like to thank again to Reviewer ogfC for the valuable comments. Hope our response can alleviate the reviewer’s concern and we are definitely willing to interact with the reviewer if there are any further questions."}, {"review_id": "p0waM3qPtw", "reviewer": "Reviewer_Ansq", "summary": "This work aims to decouple complex image editing instructions into sequential simple ones, thus achieve more robust editing. The idea is straightforward and effective.", "questions": "1. During training, only need to prepare those simple editing pair? If so, this will be a pain release.\n\n2. During inference, will it infer multiple round according to steps decoupled from the complex instruction? May also need to compare running time for fairness.\n\n3. Complex instructions to simple steps should be the key. How robust is the VLM? Please discuss more on how to finetune the VLM towards you task.\n\n4. How to decide the order of simple edit steps? Any ablation studies or just heuristic?", "limitations": "The idea and whole pipeline make sense heuristically, while they may need more support from ablation studies, as well as comparisons to SOTA image editing works like UniReal, OmniGen.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This paper is easy to follow, and the motivation is clearly discussed. Experimental results demonstrate the advantages of the proposed method. VLM should be the key to support such an agent-like editing pipeline.\n\nAlthough SOTA performance on benchmarks (MagicBrush, AnyEdit) and competitive results against proprietary models (GPT-4o), validated by ablation studies. key limitations remain: \n1. reliance on simple training pairs raises questions about generalization to unseen edits\n2. inference latency for multi-step edits is unquantified compared to end-to-end baselines\n3. the robustness of the VLM for instruction parsing and heuristic operation ordering lacks empirical justification. \n\nWhile the modular approach is innovative, further analysis of runtime costs, VLM fine-tuning, and step-order ablation would strengthen the work.", "was_revised": true, "final_justification": "Thanks for the response, which resolved most of my concerns. I would like to keep the positive score.", "rebuttal": "We thank Reviewer Ansq sincerely for the constructive feedback and insightful comments. We are pleased that the reviewer appreciates the motivation and clarity of our writing. We address the reviewer's concerns and questions as below.\n\n- > **(Q1: Only simple pairs for training)** During training, only need to prepare those simple editing pair? If so, this will be a pain release.\n\n  Thanks for this insightful observation. Yes, our framework is trained with only simple atomic-level editing pairs, which are easy to annotate. Each operator learns independently from clear, task-specific examples, reducing annotation cost and enabling modular reuse.\n  \n- > **(W1: Generalizability to unseen edits)** reliance on simple training pairs raises questions about generalization to unseen edits 1.1\n\n  We appreciate the reviewer’s concern on the generalizability of our method. Our 5 atomic operations are elicited from a comprehensive survey of image editing tasks and represent a minimal yet sufficient set that can be flexibly composed to cover diverse editing instructions. \n\n  Notably, **IEAP is capable of addressing previously unseen editing tasks**—which are often challenging for end-to-end models, particularly those underrepresented in training data—such as layout reconfiguration (e.g., swapping the positions of objects A and B or rotating object C). IEAP achieves this by organically composing atomic operations to fulfill complex instructions.\n\n  To validate this capability, we select layout-editing test cases and evaluate performance using GPT-based accuracy (instruction faithfulness, as introduced in Section 5.1 of our paper). The results, compared with other approaches, are presented in the table below.\n\n  | Approaches            | InstructP2P | MagicBrush | UltraEdit | ICEdit | IEAP(ours) |\n  | --------------------- | ----------- | ---------- | --------- | ------ | ---------- |\n  | GPT$_{IF}$ $\\uparrow$ | 2.65        | 2.55       | 3.2       | 3.05   | **3.85**   | \n\n- > **(W2: Inference latency)** W2: inference latency for multi-step edits is unquantified compared to end-to-end baselines\n\n  We thank the reviewer for raising the point. We have quantified inference latency for different editing types. For local attribute editing and overall content editing, only 1 atomic diffusion-based step is required. For local semantic editing, the required atomic steps are:\n\n  Add/Replace: $1 \\times RoI Localization$ + $1 \\times Diffusion$ + $2 \\times LLM Response$\n\n  Remove: $1 \\times RoI Localization$ + $1 \\times Diffusion$ + $1 \\times LLM Response$\n\n  Action Change: $2 \\times RoI Localization$ + $3 \\times Diffusion$ + $2 \\times LLM Response$\n\n  Move/Resize: $1 \\times RoI Localization$ + $2 \\times Diffusion$ + $2 \\times LLM Response$\n\n  We tested the time cost for each operation on H200. Each diffusion step takes approximately 5 s if using FLUX.1-dev and 4s if using FLUX.1-schnell, an LLM response requires about 1 s, and an RoI localization needs about 2 s. \n\n  Thus, Add/Replace operations take ～8-9s; Remove ～7-8s; Action Change ～18-20s; Move/Resize ～12-14s. Other operations typically take ～4-5s.\n\n  On AnyEdit test set which includes 16 sub types of editing, our approach has a weighted average latency of ～7s. We also tested the latency for end-to-end approaches: InstructPix2Pix(～4s), MagicBrush(～4s), UltraEdit(～1s) and ICEdit(～3s). Our approach offers slightly higher latency but enables faithful, modular handling of both simple and complex multi-step edits beyond the reach of end-to-end systems.\n\n- > **(Q2: Fair runtime comparison)** Q2: During inference, will it infer multiple round according to steps decoupled from the complex instruction? May also need to compare running time for fairness.\n\n  Yes. Our pipeline executes multiple atomic steps sequentially according to the decomposed instruction. The runtime comparison of ours and others are illustrated in the previous reply, i.e., IEAP(～7s), InstructPix2Pix(～4s), MagicBrush(～4s), UltraEdit(～1s) and ICEdit(～3s). The most complex case of IEAP uses 3 diffusion steps (each with 28 denoising iterations) and 1 RoI localization. To ensure a fair runtime comparison, we configure competing end-to-end methods to perform inference with a comparable total diffusion budget — specifically, $4 \\times 28$ denoising steps. The results are shown in the table below:\n\n  MagicBrush Test Set:\n  |Method|CLIP$_{im}$ $\\uparrow$|CLIP$_{out}$ $\\uparrow$|L1 $\\downarrow$|DINO $\\uparrow$| \n  |-------------------|-----------|----------|---------|---------|\n  |InstructP2P|0.838|0.229|0.112|0.759|\n  |MagicBrush|0.886|0.242|0.074|0.859|\n  |UltraEdit|0.912|0.227|0.061|0.889|\n  |ICEdit|0.913|0.236|**0.058**|0.885|\n  |IEAP|**0.922**|**0.247**|0.060|**0.897**|\n\n  AnyEdit Test Set:\n  |Method|CLIP$_{im}$ $\\uparrow$|L1 $\\downarrow$|DINO $\\uparrow$|\n  |-------------------|-----------|----------|---------|\n  |InstructP2P|0.804|0.110|0.766|\n  |MagicBrush|0.825|0.125|0.743|\n  |UltraEdit|0.833|0.114|0.722|\n  |ICEdit|0.847|0.110|0.765|\n  |IEAP|**0.895**|**0.107**|**0.879**|\n\n- > **(Q4: Operation ordering)** How to decide the order of simple edit steps? Any ablation studies or just heuristic?\n\n  Thanks for the insightful questions. For complex instructions we place tone/style edits last to avoid degradation of preceding edits. To validate this design choice, we curate a small evaluation set of 5 images with complex editing instructions, each involving tone/style edits, and compare GPT-based accuracy (instruction faithfulness, as introduced in Section 5.1 of our paper) across different operation orders. We observed that placing tone/style edits at the end consistently yielded higher scores (average +0.4), confirming that this ordering better preserves visual quality and executes the intended edits.\n\n  For individual operations, the sequence follows an naturally reasoning logic: locate→modify→composite, which are designed to reflect human-like workflows.\n\n- > **(W3/Q3: Robustness of VLMs/LLMs used)** W3/Q3: Complex instructions to simple steps should be the key. How robust is the VLM? Please discuss more on how to finetune the VLM towards you task.\n\n  Thanks for pointing this out. We have supplemented comparative experiments involving different VLMs and LLMs to analyze their influence on editing quality.\n\n    For complex instruction decomposition, we evaluate challenging instructions from the MagicBrush dataset across 2 VLMs (GPT-4o and Gemini-2.0-flash) and 2 LLMs (GPT-4 and GPT-3.5-turbo). The table below reports the accuracy of various models: \n\n    |Model|GPT-4o (as VLM)|Gemini-2.0-flash (as VLM)|GPT-4 (as LLM)|GPT-3.5-turbo (as LLM)|\n    |-------------------|----------|---------|---------|---------|\n    |Acc(%)|**100**|96.7|90|76.7|\n\n    We find that the instruction decomposition step is robust across different VLMs. However, when the original image is excluded—i.e., using only LLMs—we observe a notable performance drop.\n\n    RoI localization is implemented by Sa2VA. The ablation studies regarding various models are presented in our response to the previous question. \n    \n    For text-based ROI extraction, e.g., identifying “bananas” in “remove the bananas”, which is a straightforward task, all tested models, including GPT-3.5-turbo, GPT-4, GPT-4o, and Gemini-2.0-flash, consistently accomplish successfully.\n\n    In our method, LLMs also assist in layout modifications: for “add” operations, they determine the appropriate RoI for new elements; for “move” and “resize” operations, they predict the RoIs for the items after editing. Quantitative comparisons across these models regarding this step are shown below:\n\n    Add:\n    |Model|CLIP$_{im}$ $\\uparrow$|CLIP$_{out}$ $\\uparrow$|L1 $\\downarrow$|DINO $\\uparrow$|\n    |-------------------|-----------|----------|---------|---------|\n    |Gemini-2.0-flash (as LLM)|0.926|0.276|0.060|0.912|\n    |GPT-3.5-Turbo (as LLM)|0.903|0.265|0.070|0.901|\n    |GPT-4 (as LLM)|0.929|0.279|0.058|0.917|\n    |GPT-4o (as LLM)|0.928|0.278|0.056|0.917|\n    |GPT-4o(as VLM)|**0.935**|**0.281**|**0.056**|**0.919**|\n\n    Move/Resize:\n    |LLM|CLIP$_{im}$ $\\uparrow$|CLIP$_{out}$ $\\uparrow$|L1 $\\downarrow$|DINO $\\uparrow$|\n    |-------------------|-----------|----------|---------|---------|\n    |Gemini-2.0-flash (as LLM)|0.941|0.243|0.062|0.912|\n    |GPT-3.5-Turbo (as LLM)|0.919|0.236|0.068|0.901|\n    |GPT-4 (as LLM)|0.939|0.243|**0.061**|0.913|\n    |GPT-4o (as LLM)|0.943|0.243|0.062|0.912|\n    |GPT-4o (as VLM)|**0.945**|**0.243**|0.062|**0.913**|**\n    \n  In general, the layout modification capacity is robust across various LLMs/VLMs.\n\n  Notably, without any task-specific finetuning, GPT-4o already achieves strong performance, guided by carefully designed task-specific prompts.\n\n- > **(L: Comparison with OmniGen and OmniGen2)** May need more comparisons to SOTA image editing works like UniReal, OmniGen.\n\n  Thanks for the valuable feedback and bringing the SOTA works to our attention. As of the first-round rebuttal deadline, UniReal’s inference model and code were not publicly available, preventing a direct comparison. However, we have compared our method with OmniGen and OmniGen2 on two benchmarks, with the results summarized below:\n  MagicBrush Test Set:\n  |Method|CLIP$_{im}$ $\\uparrow$|CLIP$_{out}$ $\\uparrow$|L1 $\\downarrow$|DINO $\\uparrow$| \n  |-------------------|-----------|----------|---------|---------|\n  |IEAP|**0.922**|**0.247**|**0.060**|**0.897**|\n  |OmniGen|0.808|0.227|0.189|0.707|\n  |OmniGen2|0.881|0.242|0.100|0.830|\n\n  AnyEdit Test Set:\n  |Method|CLIP$_{im}$ $\\uparrow$|L1 $\\downarrow$|DINO $\\uparrow$|GPT $\\uparrow$| \n  |-------------------|-----------|----------|---------|---------|\n  |IEAP|**0.882**|**0.096**|**0.825**|**4.41**|\n  |OmniGen|0.805|0.175|0.711|3.96|\n  |OmniGen2|0.857|0.132|0.772|4.13|\n\nThanks again for Reviewer Ansq’s insightful reviews. Hope our response can alleviate the reviewer’s concern and we are happy to discuss with the reviewer if there are any further questions."}, {"review_id": "dfYxgGrsnw", "reviewer": "Reviewer_2JCg", "summary": "This paper introduce Image Editing as Programs (IEAP), in which, a complex text-instruction for image editing will be decomposed into small parts (or called atomic operations). These atomic operations are created via Vision Language Models, while the atomic operations are conducted by five atomic primitives (RoI Localization, RoI In-painting, RoI Editing, RoI Compositing, and Global Transformation).\nExperiments result show that IEAP yield more accurate edits on MagicBrush and AnyEdit dataset.", "questions": "See Weakness.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* Overall I like the idea of using VLMs model to decompose the complex image editing instruction into small parts.\n* Generally I think the paper's structure is good and the method/ experiments section are easy to follow.\n\nWeaknesses:\nWhile the motivation of the paper is good, the designed experiments and baselines are not well-designed/ justified.\n* Lack of comparison to more direct baseline: While this paper focus on leveraging LLMs/ VLMs to decompose the text-based edits, there is no baselines that use LLMs/ VLMs (e.g., GenArtist (NeurIPS 2024)).\n* There are multiple technical design that need to be further explain/ justify in the main paper. For example:\n(i) Current pipeline use LLM or VLM? In text, there are multiple time wrote \"LLM\", yet in the Figure, it seems to indicate the VLM (e.g., Line 163 \"... employ LLM to locate the text RoI\"; yet in Figure 3, caption is \"VLM\"). Either VLM or LLM is chosen, I think it'd be the best if we have an ablation study on this as well (e.g., if it's better to use VLM or LLM).", "was_revised": true, "final_justification": "Thanks author for providing a strong rebuttal.\n\nI have read other reviews and author's rebuttal.\nOverall, I maintain my positive view for this paper, thus, I will increase my rating.\nI hope authors will include all the discussion about designs choice in the revision.", "rebuttal": "We sincerely thank Reviewer 2JCg for the thoughtful feedback on our work. We are happy that the reviewer gives positive comments on the motivation and writing clarity of our work. We would like to address the concerns of the reviewer as below.\n\n- > **(W1: Comparison to GenArtist)** Lack of comparison to more direct baseline: While this paper focus on leveraging LLMs/ VLMs to decompose the text-based edits, there is no baselines that use LLMs/ VLMs (e.g., GenArtist (NeurIPS 2024)).\n\n  Thanks for pointing out the relevant baseline GenArtist. We have now included direct comparisons on both primary benchmarks used in our paper. The results are presented below:\n\n  MagicBrush Test Set:\n  |Method|CLIP$_{im}$ $\\uparrow$|CLIP$_{out}$ $\\uparrow$|L1 $\\downarrow$|DINO $\\uparrow$| \n  |-------------------|-----------|----------|---------|---------|\n  |IEAP|**0.922**|**0.247**|**0.060**|**0.897**|\n  |GenArtist|0.908|0.232|0.063|0.889|\n\n  AnyEdit Test Set:\n  |Method|CLIP$_{im}$ $\\uparrow$|L1 $\\downarrow$|DINO $\\uparrow$|GPT $\\uparrow$|\n  |-------------------|-----------|----------|---------|---------|\n  |IEAP|**0.882**|**0.096**|**0.825**|**4.41**|\n  |GenArtist|0.829|0.121|0.776|3.98|\n\n- > **(W2: Ablation studies of different VLMs/LLMs)** Current pipeline use LLM or VLM? In text, there are multiple time wrote \"LLM\", yet in the Figure, it seems to indicate the VLM (e.g., Line 163 \"... employ LLM to locate the text RoI\"; yet in Figure 3, caption is \"VLM\"). Either VLM or LLM is chosen, I think it'd be the best if we have an ablation study on this as well (e.g., if it's better to use VLM or LLM).\n\n  Thanks for the valuable questions and suggestions. In our pipeline, VLM and LLM are deployed in distinct modules based on their complementary strengths:\n\n  * VLM is mainly used for instruction decomposition (as mentioned in Figure 3), which requires understanding both text semantics and image context. \n  * LLM is mainly used for text-based RoI extraction (e.g., identifying “bananas” in “remove the bananas”). \n  * Moreover, LLMs also assist in layout modifications if necessary: for “add” operations, they determine the appropriate RoI for new elements; for “move” and “resize” operations, they predict the RoIs for the items after editing. \n\n  To justify these design choices, we conducted additional ablation studies evaluating different VLMs and LLMs across the key components of our pipeline, with the results summarized below.\n\n  For complex instruction decomposition, we evaluate challenging instructions from the MagicBrush dataset across 2 VLMs (GPT-4o and Gemini-2.0-flash) and 2 LLMs (GPT-4 and GPT-3.5-turbo). The table below reports the accuracy of various models: \n\n    |Model|GPT-4o (as VLM)|Gemini-2.0-flash (as VLM)|GPT-4 (as LLM)|GPT-3.5-turbo (as LLM)|\n    |-------------------|----------|---------|---------|---------|\n    |Acc(%)|**100**|96.7|90|76.7|\n\n    We find that the instruction decomposition step is robust across different VLMs. However, when the original image is excluded—i.e., using only LLMs—we observe a notable performance drop.\n\n  For text-based ROI extraction, e.g., identifying “bananas” in “remove the bananas”, which is a straightforward task, all tested models, including GPT-3.5-turbo, GPT-4, GPT-4o, and Gemini-2.0-flash, consistently accomplish successfully.\n\n  For editing tasks with layout modifications, quantitative comparisons across LLMs/VLMs are shown below:\n\n  Add:\n  |LLM|CLIP$_{im}$ $\\uparrow$|CLIP$_{out}$ $\\uparrow$|L1 $\\downarrow$|DINO $\\uparrow$|\n  |-------------------|-----------|----------|---------|---------|\n  |Gemini-2.0-flash (as LLM)|0.926|0.276|0.060|0.912|\n  |GPT-3.5-Turbo (as LLM)|0.903|0.265|0.070|0.901|\n  |GPT-4 (as LLM)|0.929|0.279|0.058|0.917|\n  |GPT-4o (as LLM)|0.928|0.278|0.056|0.917|\n  |GPT-4o (as VLM)|**0.935**|**0.281**|**0.056**|**0.919**|\n\n  Move/Resize:\n    |LLM|CLIP$_{im}$ $\\uparrow$|CLIP$_{out}$ $\\uparrow$|L1 $\\downarrow$|DINO $\\uparrow$|\n    |-------------------|-----------|----------|---------|---------|\n    |Gemini-2.0-flash (as LLM)|0.941|0.243|0.062|0.912|\n    |GPT-3.5-Turbo (as LLM)|0.919|0.236|0.068|0.901|\n    |GPT-4 (as LLM)|0.939|0.243|**0.061**|0.913|\n    |GPT-4o (as LLM)|0.943|0.243|0.062|0.912|\n    |GPT-4o (as VLM)|**0.945**|**0.243**|0.062|**0.913**|\n  \n  In general, the layout modification capacity is robust across various LLMs/VLMs. Although using a VLM for layout arrangement yields slightly better results, it introduces an additional forward pass over image tokens, significantly increasing computational overhead. Therefore, we adopt LLMs without image inputs here by default.\n  \n\n- > **(W: Clarification on the experimental design)** While the motivation of the paper is good, the designed experiments and baselines are not well-designed/ justified.\n\n  We appreciate the reviewer’s feedback on our experimental design. We would like to clarify that our experiments were carefully designed to comprehensively evaluate IEAP across diverse and challenging scenarios, with baselines chosen to represent the state-of-the-art and all methods evaluated under identical settings. \n\n  To further strengthen this aspect, we have added additional ablations and comparisons, including direct evaluations against LLM/VLM-based approaches such as GenArtist, as suggested by the reviewer. We believe these updates can further substantiate the fairness and rigor of our experimental validation, and we would like to thank the reviewer for prompting these clarifications.\n\nThanks again for Reviewer 2JCg’s insightful comments. Our sincere hope is that our response will alleviate the reviewer’s concern. We are looking forward to having further discussion with the reviewer if there are remaining concerns."}, {"review_id": "EKwJ7THuiJ", "reviewer": "Reviewer_t8qA", "summary": "IEAP is a system for editing images based on text prompts. It uses a LLM to decompose complex multi-step editing tasks into a series of more basic ‘atomic’ image-to-image editing operations. These image-to-image editing operations rely on specially finetuned DiT models (e.g. for in-painting / attribute modification / global modification), with help from LLMs for various localization tasks (e.g. what region needs to be modified, w.r.t. an editing prompt). The system proposes a taxonomy over different types of image transformations, and shows how their atomic operations are able to help realize and better adhere to these requests. On multiple image editing benchmarks, the proposed system outperforms previous alternatives w.r.t. metrics that automatically evaluate the edited result.", "questions": "What were the settings / details for the training datasets / learning set-up of the DiT specialized operators? Beyond compounding errors from repeated application of atomic operators, I would be curious to know if the cause of the visual fidelity issues is more related to the base DiT, or the distribution of available paired training data.\n\nI was a bit surprised to see in Table 2 that the ‘gains’ of the proposed method compared with baselines are less pronounced for local semantic editing tasks, compared with the other tasks. This seems to run counter to the paper narrative that the method is especially designed for these local semantic editing tasks, so I would be curious to understand your interpretation of this result.", "limitations": "Yes, largely, although as I mentioned in the S/W section, some of the points can be expanded (and the supplemental discussion should be forward referenced from the main document)", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nIEAP is a well-presented paper that introduces a new and interesting method for a challenging, timely task. Decomposing difficult tasks, like text-based image editing, into a series of more manageable, and compositional sub-tasks, is a well-founded approach for addressing this problem. The main contribution of the work is the proposed taxonomy of transformation tasks, and the implementation of the atomic operations that are used within this ‘programmatic’ framework. While some details are omitted from the submission (e.g. exact details of data curation + training logic for these ‘atomic’ DiT operators), the system would largely be reproducible (modulo some design decisions), and I think would be of interest to the community, who is invested in identifying ways of making these types of models more robust, controllable, interpretable. \n\n\nWeaknesses:\n\nWhile the edited results do a much better job compared with the baselines of adhering to the semantics of the instructions, some of the ‘final’ results produced by the pipeline have noticeable errors / artifacts. The authors state that this is a fundamental limitation of this approach, where errors can ‘build-up’ across different ‘operation’ invocations – but I would have liked to see a more thoughtful treatment of this point, and maybe some investigations into what could be done to mitigate this issue.\n\nWhile I am quite partial to the problem framing of decomposing hard tasks into sub-problems, I think the framing of these series of operations as ‘programmatic’ seems like a bit of stretch. This iterative process is much less programmatic compared with other recent approaches that for instance, operate in general coding environments with specialized sub-modules (e.g. VisProg, ViperGPT) or produce node-based graphs (e.g. comfyUI workflows). I would encourage the authors to either revisit the framing of this paper as programmatic, or do more to demonstrate that these simple ‘atomic’ operations provide programmatic benefits beyond ‘end-generation’ performance (modularity, editability, compositionality, debuggability). At the very least, these mentioned prior works need a more detailed treatment / discussion, if not treated as baseline comparison conditions.\n\nI also have some concerns that the set of atomic operations that have been designed, seem especially ‘well-tailored’ for the types of edits that exit within these benchmarks – while this is potentially OK, I think it would be good to describe the process by which the author’s developed this taxonomy, and maybe include some description/ideation on what is missing, or what is next, in terms of atomic operations that would be needed to support all types of text-based image editing.", "was_revised": true, "final_justification": "My concerns have been addressed and I remain positive on this work, so I would vote for its acceptance.", "rebuttal": "We sincerely thank Reviewer t8qA for the insightful feedback and constructive suggestions. We are pleased that the reviewer recognizes our method as novel and interesting, and finds our pipeline meaningful. We would like to address the concerns and questions reflected in the review below.\n\n- > **W1: Accumulated artifacts across sequential edits and potential solutions**\n\n  Thanks for pointing this out. We acknowledge that artifact accumulation across sequential edits is a limitation of modular pipelines. To mitigate this, we believe that an error-reflection mechanism, where a VLM self-assesses intermediate results after each atomic operation and refines unsatisfactory steps before proceeding may help. We evaluated on 10 multi-step cases, and found that this mechanism improved 8 out of 10 cases, notably reducing artifacts and enhancing instruction faithfulness.\n\n- > **W2: Clarification of the Term “Programmatic”**\n\n  Thanks for this thoughtful comment. We contend that “programmatic” here refers less to mimicking general-purpose programming environments and more to embracing the core epistemic principles of programming—decomposition, modularity, compositionality, and transparent control. In IEAP, editing is not a monolithic black box but a sequence of interpretable, verifiable transformations, where each atomic operation can be reasoned about, reused, or refined. This programmatic structure creates a conceptual bridge between abstract linguistic instructions and concrete visual modifications, allowing for intermediate inspection, targeted debugging, and flexible recomposition. While the operational form differs from graph-based or code-centric systems (e.g., VisProg, ViperGPT), the underlying philosophy—a shift from opaque end-to-end generation toward structured, controllable editing—remains the same. And IEAP enjoys the same programmatic benefits—structured decomposition, modular adaptability, compositional flexibility, and traceable control—in image editing.\n\n- > **W3: Design process of atomic operations and potential improvements**\n\n  Thanks for this insightful comment. Our taxonomy was not designed solely for the current benchmarks but derived through a principled reduction of the broad landscape of image editing operations. We began by conducting a thorough analysis and categorization of common image editing operations. In our preliminary experiments, we found that some categories could not be reliably solved. Inspired by the CRUD paradigm in data operations, we proposed to reduce operations that cannot be executed well into atomic operations. Specifically, each category was retained as atomic only if a unified approach could robustly solve it; otherwise, we refined it further. For example, “remove” edits were split out because standard inpainting often failed to fully eliminate targets due to localization errors. Through this iterative refinement, we found that our current taxonomy of atomic operations strikes a balance between complexity and effectiveness, enabling robust performance across diverse editing tasks. Looking forward, we recognize that supporting all text-based editing may require extending the taxonomy, for example by incorporating atomic operations for fine-grained physical property manipulation, e.g., material or lighting changes, or edits requiring deeper semantic reasoning across multiple modalities.\n\n- > **Q1: Training setup and source of fidelity issues**\n\n  We thank the reviewer for asking this. For training settings, as illustrated in Sec 5.1, we fine-tune four specialized models (RoI inpainting, RoI editing, RoI compositing, global transformation) on FLUX.1-dev using LoRA with rank=128 and alpha=128, with each trained on approximately 5k-15k image pairs.\n\n  For visual fidelity issues: Beyond compounding errors, key issues such as misaligned human-shadow interactions are more attributable to the inherent difficulty of modeling tightly coupled visual relationships in a stepwise editing framework, and we see this as a promising direction for extending our current design.\n  \n  Notably, our approach leverages the strong generative capabilities of the base DiT (FLUX.1-dev), which enables effective performance even with limited training data. Compared to ControlNet, our method achieves competitive results on standard editing tasks while using far less data.\n\n- > **Q2: Performance concerns on local semantic editing**\n\n  We truly appreciate the reviewer’s keen observation. While the relative improvements on static metrics for local semantic editing may appear less pronounced, as they are limited to the dataset’s ground truth and fail to capture key human-centric qualities like the naturalness of object interactions or the coherence of edits, which can be better reflected by the GPT-based scores (as introduced in Sec 5.1 in our paper). We argue that a comprehensive assessment requires combining both: static metrics quantify objective progress, while GPT scores capture subtler human-centric nuances. Below are the detailed GPT-based scores on local semantic editing type, which reveal that our approach outperforms baselines significantly in terms of alignment with human judgment on these nuanced aspects of edit quality.\n\n  |Methods|GPT$_{IF}$ $\\uparrow$|GPT$_{FC}$ $\\uparrow$|GPT$_{AQ}$ $\\uparrow$|GPT$_{avg}$ $\\uparrow$|\n  |-------------------|-----------|----------|---------|---------|\n  |InstructP2P|3.53|3.89|3.80|3.74|\n  |MagicBrush|3.49|4.16|4.06|3.90|\n  |UltraEdit|3.21|4.20|4.18|3.86|\n  |ICEdit|3.57|4.38|4.29|4.08|\n  |IEAP|**4.41**|**4.50**|**4.35**|**4.42**|\n\n- > **(L: Suggestions for improvement)** Some of the points can be expanded (and the supplemental discussion should be forward referenced from the main document)\n\n  We thank the reviewer for the valuable suggestion. We will expand key discussions in the revised version and add clear forward references to the supplementary material for clarity and completeness.\n\nWe would like to thank Reviewer t8qA again for the in-depth reviews, which have prompted us to revisit and refine our work. We are glad to having further discussion with the reviewer if there are remaining concerns."}]

Author rebuttal:

Meta-review: This paper proposes a neat solution to dealing with complex instruction-driven image editing by decomposing the compound instruction into a sequence of programmable atomic operations using VLMs. During rebuttal and author-reviewer discussion phases, the authors provided additional results on various LLMs/VLMs, multi-object manipulation, and robustness to inaccurate localization, etc. All reviewers feel that their major concerns have been addressed and are inclined to accept this paper.

The reviewers have reached a consensus on the innovation of the proposed approach. However, there are some common concerns, which have been addressed to a certain extent by the authors, are expected to be fully addressed in the future work:

1. Influence of the performance of different VLMs/LLMs (Reviewers K6UQ, ogfC, Ansq)

2. Adequacy of the defined set of atomic operations (Reviewers K6UQ, t8qA) 

3. Lack of certain experiments (Reviewers K6UQ, Ansq, 2JCg)
Decision: Accept (poster)