Title: Image Editing As Programs with Diffusion Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Image Editing

Diffusion Transformer
Abstract: While diffusion models have achieved remarkable success in text-to-image generation, they encounter significant challenges with instruction-driven image editing. Our research highlights a key challenge: these models particularly struggle with structurally-inconsistent edits that involve substantial layout changes. To address this gap, we introduce Image Editing As Programs (IEAP), a unified image editing framework built upon the Diffusion Transformer (DiT) architecture. Specifically, IEAP deals with complex instructions by decomposing them into a sequence of programmable atomic operations. Each atomic operation manages a specific type of structurally consistent edit; when sequentially combined, IEAP enables the execution of arbitrary and structurally-inconsistent transformations. This reductionist approach enables IEAP to robustly handle a wide spectrum of edits, encompassing both structurally-consistent and inconsistent changes. Extensive experiments demonstrate that IEAP significantly outperforms state-of-the-art methods on standard benchmarks across various editing scenarios. In these evaluations, our framework delivers superior accuracy and semantic fidelity, particularly for complex, multi-step instructions. Codes are available at https://github.com/YujiaHu1109/IEAP.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIEAP is a system for editing images based on text prompts. It uses a LLM to decompose complex multi-step editing tasks into a series of more basic ‘atomic’ image-to-image editing operations. These image-to-image editing operations rely on specially finetuned DiT models (e.g. for in-painting / attribute modification / global modification), with help from LLMs for various localization tasks (e.g. what region needs to be modified, w.r.t. an editing prompt). The system proposes a taxonomy over different types of image transformations, and shows how their atomic operations are able to help realize and better adhere to these requests. On multiple image editing benchmarks, the proposed system outperforms previous alternatives w.r.t. metrics that automatically evaluate the edited result.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nIEAP is a well-presented paper that introduces a new and interesting method for a challenging, timely task. Decomposing difficult tasks, like text-based image editing, into a series of more manageable, and compositional sub-tasks, is a well-founded approach for addressing this problem. The main contribution of the work is the proposed taxonomy of transformation tasks, and the implementation of the atomic operations that are used within this ‘programmatic’ framework. While some details are omitted from the submission (e.g. exact details of data curation + training logic for these ‘atomic’ DiT operators), the system would largely be reproducible (modulo some design decisions), and I think would be of interest to the community, who is invested in identifying ways of making these types of models more robust, controllable, interpretable. \n\n\nWeaknesses:\n\nWhile the edited results do a much better job compared with the baselines of adhering to the semantics of the instructions, some of the ‘final’ results produced by the pipeline have noticeable errors / artifacts. The authors state that this is a fundamental limitation of this approach, where errors can ‘build-up’ across different ‘operation’ invocations – but I would have liked to see a more thoughtful treatment of this point, and maybe some investigations into what could be done to mitigate this issue.\n\nWhile I am quite partial to the problem framing of decomposing hard tasks into sub-problems, I think the framing of these series of operations as ‘programmatic’ seems like a bit of stretch. This iterative process is much less programmatic compared with other recent approaches that for instance, operate in general coding environments with specialized sub-modules (e.g. VisProg, ViperGPT) or produce node-based graphs (e.g. comfyUI workflows). I would encourage the authors to either revisit the framing of this paper as programmatic, or do more to demonstrate that these simple ‘atomic’ operations provide programmatic benefits beyond ‘end-generation’ performance (modularity, editability, compositionality, debuggability). At the very least, these mentioned prior works need a more detailed treatment / discussion, if not treated as baseline comparison conditions.\n\nI also have some concerns that the set of atomic operations that have been designed, seem especially ‘well-tailored’ for the types of edits that exit within these benchmarks – while this is potentially OK, I think it would be good to describe the process by which the author’s developed this taxonomy, and maybe include some description/ideation on what is missing, or what is next, in terms of atomic operations that would be needed to support all types of text-based image editing.\n\nQUESTIONS:\nWhat were the settings / details for the training datasets / learning set-up of the DiT specialized operators? Beyond compounding errors from repeated application of atomic operators, I would be curious to know if the cause of the visual fidelity issues is more related to the base DiT, or the distribution of available paired training data.\n\nI was a bit surprised to see in Table 2 that the ‘gains’ of the proposed method compared with baselines are less pronounced for local semantic editing tasks, compared with the other tasks. This seems to run counter to the paper narrative that the method is especially designed for these local semantic editing tasks, so I would be curious to understand your interpretation of this result.\n\nLIMITATIONS:\nYes, largely, although as I mentioned in the S/W section, some of the points can be expanded (and the supplemental discussion should be forward referenced from the main document)", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

comments: SUMMARY:
This paper introduce Image Editing as Programs (IEAP), in which, a complex text-instruction for image editing will be decomposed into small parts (or called atomic operations). These atomic operations are created via Vision Language Models, while the atomic operations are conducted by five atomic primitives (RoI Localization, RoI In-painting, RoI Editing, RoI Compositing, and Global Transformation).
Experiments result show that IEAP yield more accurate edits on MagicBrush and AnyEdit dataset.

STRENGTHS AND WEAKNESSES:
Strengths:
* Overall I like the idea of using VLMs model to decompose the complex image editing instruction into small parts.
* Generally I think the paper's structure is good and the method/ experiments section are easy to follow.

Weaknesses:
While the motivation of the paper is good, the designed experiments and baselines are not well-designed/ justified.
* Lack of comparison to more direct baseline: While this paper focus on leveraging LLMs/ VLMs to decompose the text-based edits, there is no baselines that use LLMs/ VLMs (e.g., GenArtist (NeurIPS 2024)).
* There are multiple technical design that need to be further explain/ justify in the main paper. For example:
(i) Current pipeline use LLM or VLM? In text, there are multiple time wrote "LLM", yet in the Figure, it seems to indicate the VLM (e.g., Line 163 "... employ LLM to locate the text RoI"; yet in Figure 3, caption is "VLM"). Either VLM or LLM is chosen, I think it'd be the best if we have an ablation study on this as well (e.g., if it's better to use VLM or LLM).

QUESTIONS:
See Weakness.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 2
IMPACT: 1
ORIGINALITY: 2
DATE: 2025-06-30

comments: SUMMARY:
This work aims to decouple complex image editing instructions into sequential simple ones, thus achieve more robust editing. The idea is straightforward and effective.

STRENGTHS AND WEAKNESSES:
This paper is easy to follow, and the motivation is clearly discussed. Experimental results demonstrate the advantages of the proposed method. VLM should be the key to support such an agent-like editing pipeline.

Although SOTA performance on benchmarks (MagicBrush, AnyEdit) and competitive results against proprietary models (GPT-4o), validated by ablation studies. key limitations remain: 
1. reliance on simple training pairs raises questions about generalization to unseen edits
2. inference latency for multi-step edits is unquantified compared to end-to-end baselines
3. the robustness of the VLM for instruction parsing and heuristic operation ordering lacks empirical justification. 

While the modular approach is innovative, further analysis of runtime costs, VLM fine-tuning, and step-order ablation would strengthen the work.

QUESTIONS:
1. During training, only need to prepare those simple editing pair? If so, this will be a pain release.

2. During inference, will it infer multiple round according to steps decoupled from the complex instruction? May also need to compare running time for fairness.

3. Complex instructions to simple steps should be the key. How robust is the VLM? Please discuss more on how to finetune the VLM towards you task.

4. How to decide the order of simple edit steps? Any ablation studies or just heuristic?

LIMITATIONS:
The idea and whole pipeline make sense heuristically, while they may need more support from ablation studies, as well as comparisons to SOTA image editing works like UniReal, OmniGen.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-04

comments: SUMMARY:
This paper describes a new approach to image editing as a series of operation as opposed to a one shot generation. Operations are broken down into 5 atomic operations that can be combined to create better images. The approach was successfully applied and have been demonstrated to have SOTA performance compared to other methods.

STRENGTHS AND WEAKNESSES:
Strengths
+ Paper is written really well and easy to follow. 
+ Evaluations and comparisons are pretty comprehensive and demonstrated to have SOTA performance.
+ Good breakdown and analysis of edit operations.

Weak
- The method is primarily an integration of multiple methods and demonstrated to be a really great approach, and not really model training and improvements.
- Would be great to have some understanding of how the underlying models used might affect the overall approach (for example using a different LLM model GPT3 or others)

QUESTIONS:
1. How would using different LLMs affect the approach? 
2. How would different starting diffusion affect the approach?
3. It might be good to use a different LLM evaluator other than GPT4 since there would be bias (the underlying operations are based on GPT4 understanding.)

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel image editing framework, IEAP, based on the Diffusion Transformer architecture. IEAP decomposes complex natural language instructions into a sequence of atomic operations, including RoI localization, inpainting, editing, compositing, and global transformations. These operations are executed sequentially by a neural program interpreter to accommodate complex editing tasks. Experimental results demonstrate that IEAP outperforms existing methods in multiple editing benchmarks, especially in handling complex multi-step instructions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The idea of decomposing image editing into atomic operations is interesting. By transforming instructions into a sequence of atomic operations and executing them procedurally, the paper presents an editing framework that enables compositional generalization. That is, use limited atomic operations to model a large set of edit instructions. The use of LLMs for instruction parsing enhances the model’s ability to understand natural language and improves the correspondence between instructions and editing behaviors.\n\n2. IEAP achieves nearly the best performance on different editing benchmarks. The paper also provides sufficient visual examples that make the method easy to understand and evaluate.\n\nWeaknesses\n\n1. It is unclear whether the defined set of atomic operations can adequately cover the editing instructions in open-world scenarios. \n\n2. The paper appears to lack results in the editing of multiple objects. I notice that the examples primarily involve clearly defined editing targets. However, in practice, editing targets may be ambiguous. For example, the target can be “all objects of type A in the scene”.\n\n3. The pipeline depends on accurate RoI localization. A concern is that if the localization is inaccurate or the instruction is vague, the editing may fail. It is recommended to discuss the robustness of IEAP in such cases.\n\n4. This paper seems to lack discussions of how different VLMs/LLMs influence the editing quality, including the influence on instruction decomposition or RoI localization.\n\nQUESTIONS:\n1. Many of the defined atomic operations rely on the segmentation model M_seg. Can M_seg locate objects at fine-grained levels? For example, can it accurately localize “people wearing black clothes” in a complex scene?\n\n2. How does IEAP handle instructions that involve editing multiple objects of the same type? For example, in the 5th image of Row 3 in Figure 3, how does IEAP determine the order of atomic operations if the instruction is “enlarge all the unicorns”?\n\n3. In the first row of Figure 4, the intermediate image after RoI inpainting removes the person but leaves its shadow, which is corrected in the next step. I wonder why IEAP didn’t treat the shadow as part of the environment during the next RoI Editing operation?\n\n4. Will different VLMs/LLMs (used for instruction decomposition or RoI localization) lead to differences in editing performance?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-08"}