Lego-Edit: A General Image Editing Framework with Model-Level Bricks and MLLM Builder

## Abstract

Abstract Instruction-based image editing has garnered significant attention due to its direct interaction with users. However, real-world user instructions are immensely diverse, and existing methods often fail to generalize effectively to instructions outside their training domain, limiting their practical application. To address this, we propose Lego-Edit, which leverages the generalization capability of Multi-modal Large Language Model (MLLM) to organize a suite of model-level editing tools to tackle this challenge. Lego-Edit incorporates two key designs: (1) a model-level toolkit comprising diverse models efficiently trained on limited data and several image manipulation functions, enabling fine-grained composition of editing actions by the MLLM; and (2) a three-stage progressive reinforcement learning approach that uses feedback on unannotated, open-domain instructions to train the MLLM, equipping it with generalized reasoning capabilities for handling real-world instructions. Experiments demonstrate that Lego-Edit achieves state-of-the-art performance on GEdit-Bench and ImgBench. It exhibits robust reasoning capabilities for open-domain instructions and can utilize newly introduced editing tools without additional fine-tuning. Code is available: https://github.com/xiaomi-research/lego-edit.

## 1 Introduction

Instruction-based image editing methods accept natural language instructions as input and modify the input image accordingly. It enables intuitive human-computer interaction through natural language, offering broad application potential. However, the significant diversity inherent in real-world editing instructions poses a substantial challenge for image editing systems in handling flexible user commands.

Existing approaches for instruction-based editing are broadly classified into two categories. End-to-end methods Labs (2024a); Liu et al. (2025) train a single generative model to learn both instruction comprehension and pixel mapping for editing implicitly. As shown in Fig. [2](https://arxiv.org/html/2509.12883v1#S1.F2) (a), these methods are primarily constrained by the fixed instruction patterns within their training data, and consequently struggle to generalize well even with massive training datasets.

In contrast, agent-based schemes utilize MLLMs to explicitly interpret editing instructions and invoke editing tools to execute the requested modifications. Prior research Xue et al. (2025) often relies on curated prompts to guide the agents, but such prompt-driven approaches lack a deep understanding of editing tools, impeding agents’ ability to organize them effectively. Subsequent studies Guo et al. (2025) attempt to alleviate this burden by constructing complex predefined workflows as task-level tools for agents to invoke, as shown in Fig. [2](https://arxiv.org/html/2509.12883v1#S1.F2) (b). However, this strategy inherently limits the framework’s capacity to handle instructions that deviate from the predefined workflows.

To address the challenge of processing flexible real-world instructions for image editing, we propose a novel framework, named Lego-Edit. It employs a fine-tuned MLLM as an agent, termed Builder. The Builder leverages its enhanced reasoning capability to organize a set of specialized pre-trained editing models, called Bricks, enabling precise execution of diverse editing instructions, as shown in Fig. [2](https://arxiv.org/html/2509.12883v1#S1.F2) (c). It incorporates two key design innovations:

Figure: Figure 2: Comparison of end-to-end approach, API-based agent with curated workflows, and our method.
Refer to caption: x2.png

Model-Level Tools: We train and integrate a suite of specialized models as editing tools. This model-level design provides the Builder with high flexibility for composition and enables individual tools to achieve superior performance for their specific functions with less training data.

Three-Stage Progressive Reinforcement Learning Training Strategy: We first employ a Supervised Fine-Tuning (SFT) stage, followed by a Reinforcement Learning (RL) stage to train our Builder on specific editing tasks with the ground truth, establishing fundamental reasoning capabilities and tool usage knowledge. Subsequently, we conduct an additional RL stage utilizing abundant unlabeled instructions beyond the specific tasks, where a large-scale critic model provides feedback. This process enhances our Builder’s capability to handle flexible instructions.

Benefiting from these designs, our Lego-Edit reliably reasons about and executes flexible editing instructions and can integrate novel tools without additional training. Furthermore, it achieves state-of-the-art results on the GEdit-Bench and ImgBench benchmarks.

In summary, our principal contributions are threefold:

- •
We propose Lego-Edit, an instruction-based image editing framework that utilizes a reinforcement learning fine-tuned MLLM agent to coordinate model-level editing tools for executing flexible real-world instructions.
- •
We introduce a three-stage progressive reinforcement learning training strategy that provides feedback using unlabeled data, significantly enhancing the reasoning and tool composition capabilities of MLLM.
- •
Extensive experiments demonstrate that Lego-Edit achieves SOTA performance on GEdit-Bench and ImgBench. The framework also exhibits strong generalization in processing flexible open-domain instructions and could integrate new tools without retraining.

## 2 Related Works

Instruction-based Image Editing: Instruction-driven image editing, which emerged from InstructPix2Pix Brooks et al. (2023), provides a natural user interface for image editing. The majority of existing methods rely on diffusion models Gao et al. (2025); Han et al. (2024); Hertz et al. (2022); some methods incorporate Multi-modal Large Language Models (MLLMs) to achieve more precise edits. For instance, SmartEdit leverages embeddings derived from nuanced edit semantics by MLLMs Huang et al. (2024).

Models trained on massive datasets (e.g., FLUX Labs (2024a), HiDream Cai et al. (2025), Step1X Liu et al. (2025)) exhibit strong performance across various editing tasks. However, generalizing effectively across flexible editing instructions remains a critical challenge for them. Recently developed unified visual understanding and generation models (e.g., ILLUME++ Huang et al. (2025a), GPT-4o Hurst et al. (2024), Bagel Deng et al. (2025), UniWorld Lin et al. (2025)), trained on broader datasets encompassing multiple tasks like image captioning and image editing, demonstrate enhanced generalization capabilities yet are still constrained.

In contrast, our proposed approach achieves superior instruction generalization with minimal training data by orchestrating specialized editing tools via a fine-tuned agent.

MLLMs as Agents: Autonomous agents capable of utilizing tool calls have garnered significant research interest. Many approaches primarily leverage LLMs to invoke tools Qin et al. (2023); Du et al. (2024); Zheng et al. (2024b). With advances in MLLMs, considerable effort has focused on employing or fine-tuning MLLMs for multi-modal agent applications. For instance, VisualToolAgent Huang et al. (2025b) aligns models with tools via reinforcement learning, and SeeAct Zheng et al. (2024a) integrates MLLMs with grounding capabilities for WebUI interactions.

Within the domains of image generation and editing, research explores MLLMs as agents for tool invocation. ComfyAgent Xue et al. (2025) adapts prompts to call ComfyUI tools via code execution, but this code-based invocation constrains its performance. ComfyMind Guo et al. (2025) manually defines multiple pipelines for agent-driven tool use, ensuring high success rates but limiting operational flexibility.

Notably, our approach employs reinforcement learning to equip the MLLM with compositional tool-usage knowledge and reasoning ability. Combined with model-level tools, this framework achieves high success rates, superior editing performance, and robust generalization to diverse instructions.

## 3 Method

Figure: Figure 3: Overall framework of Lego-Edit. Given an instruction and an input image, the Builder generates a tool invocation workflow. The Executor then executes this workflow, calling corresponding tools to generate the edited output image.
Refer to caption: x3.png

We introduce Lego-Edit, a framework designed for general instruction-based image editing. It uses Builder (an MLLM) to invoke Bricks (model-level tools) for flexibility and employs reinforcement learning (RL) to enhance the Builder’s reasoning and tool composition ability. We first outline the overall framework, then detail the Builder’s prompt structure. The following section describes tool classification, and the final section elaborates on our three-stage progressive RL training strategy for the Builder.

### 3.1 Overall Framework

As illustrated in Fig. [3](https://arxiv.org/html/2509.12883v1#S3.F3), our system comprises: 1) the Builder, an MLLM reasoning agent denoted as $f(\theta)$ that generates workflows, where $\theta$ denotes the model parameters; 2) an Executor $D$ that parses and executes workflows; and 3) the Bricks, an external model-level tool library $\mathcal{M}={M_{1},\dots,M_{N}}$ containing functions encapsulating models or logic processes, where $N$ is the total number of tools.

Given an input pair $(I,q)$ comprising a target image $I$ and an editing prompt $q$, the Builder $f(\theta)$, observing the state $s=(I,q)$, generates a reasoning trace, denoted as $Think$, and a JSON-formatted workflow $g$ based on its strategy $\pi_{\theta}(g\mid s)$. This workflow $g=(V,E)$ is a tool invocation graph. The vertex set $V=\{M_{1},...,M_{K}\}$ represents the selected tool instances, with each $M_{i}\in\mathcal{M}$ and $1\leq K\leq N$; here, $K$ is adaptively determined by the task complexity. The edge set $E\subseteq V\times V$ defines sequential dependencies, where an edge $(M_{i},M_{j})$ indicates that the input of $M_{j}$ depends on the output of $M_{i}$. The Executor $D$ then parses $g$, invokes the tools, and generates the edited image $I^{\prime}=D(g,I)$.

### 3.2 Prompt Structure

Our curated Builder’s input prompt format (left side of Fig. [3](https://arxiv.org/html/2509.12883v1#S3.F3)) has three key components:

System Description and Invocation Constraints: Defines capabilities, task scope, and valid tool parameter types via a system prompt.

Available Tool List: Each entry includes the model name, functional description, and invocation constraints detailing editing capabilities and requirements.

Workflow Composition Examples: Few-shot examples guiding valid workflow writing.

Following these, the prompt presents the editing instruction and directs the Builder to generate an editing workflow after reasoning.

### 3.3 Model-Level Editing Tools

We constructed a fine-grained library of model-level image editing tools $\mathcal{M}$, where each tool represents a single model or function. Tools are categorized into two classes by whether they modify the image:

Predictive Models ($\mathcal{M}_{p}\subset\mathcal{M}$): Extract/process masks and regions to provide spatial constraints without altering pixels. Included tools are RES (segment specified objects), SOS (subject object segmentation), ADD-PRED (predict addition location), CAP-PRED (image captioning), INVERSE (invert the mask), and one additional tool.

Editing Models ($\mathcal{M}_{e}\subset\mathcal{M}$): Modify image content. Included tools are FILL (add object with given prompt or reference image), five specialized LoRA adapters trained on FILL: INPAINT (inpainting), POSE (human pose change), ENV (environment alteration), STYLE (style transfer), RCM (material/color change), and three additional tools.

To call a tool, the Builder needs to specify its name, input parameters, and output parameters. Complete details are in the supplementary material.

To prevent task confusion that can arise from joint training in end-to-end models (as in ICEdit Zhang et al. (2025)), we train independent LoRA adapters for each editing model. Furthermore, the Builder could precisely control edit scope using masks from $\mathcal{M}_{p}$, enabling more accurate editing.

### 3.4 Three-Stage Progressive Reinforcement Learning Strategy

To train the Builder $f(\theta)$ for effective tool composition, we employ a three-stage progressive RL strategy, gradually increasing task complexity and reducing reliance on ground truth data.

#### 3.4.1 Reinforcement Learning with GRPO

We first introduce the Group Relative Policy Optimization (GRPO) Shao et al. (2024) algorithm utilized in stages 2 and 3. For a given input $(I,q)$, the policy $\pi_{\theta}$ samples $G$ workflows $\{g_{1},...,g_{G}\}$. Each workflow $g_{j}$ receives a reward $r_{j}$ (defined per stage below). The relative advantage for each sample within the group is computed as:

$$ $A_{j}=\frac{r_{j}-\text{mean}(\{r_{1},\ldots,r_{G}\})}{\text{std}(\{r_{1},\ldots,r_{G}\})},\quad j=1,2,\ldots,G$ (1) $$

The policy is updated by maximizing the GRPO objective:

$$ $\displaystyle\mathcal{J}(\theta)$ $\displaystyle=\text{E}_{\begin{subarray}{l}s\sim\mathcal{Q},\\ \{g_{j}\}_{j=1}^{G}\sim\pi_{\theta_{\text{old}}}(g|s)\end{subarray}}\left[\frac{1}{G}\sum_{j=1}^{G}\min\left(r_{j}^{\text{ratio}}A_{j},\text{clip}(r_{j}^{\text{ratio}},1\pm\epsilon)A_{j}\right)-\beta\text{D}_{\text{KL}}(\pi_{\theta}\|\pi_{\text{ref}})\right]$ (2) $$

where $\mathcal{Q}$ denotes the distribution of image-instruction pairs for sampling observation $s$, $\pi_{\theta_{\text{old}}}$ is the old policy before updating, $\pi_{\text{ref}}$ is a fixed reference policy, $r_{j}^{\text{ratio}}=\pi_{\theta}(g_{j}|s)/\pi_{\theta_{\text{old}}}(g_{j}|s)$, $\epsilon$ controls the clipping range, and $\beta$ weights the KL regularization $\text{D}_{\text{KL}}$ towards $\pi_{\text{ref}}$.

#### 3.4.2 Stage 1: Supervised Fine-Tuning (SFT)

We adapt the Builder to image editing domain using SFT on data from several specific tasks. Each sample $(I,q)$ is paired with expert-generated reasoning traces $Think_{\text{GT}}$ and ground truth workflows $g_{\text{GT}}$. The learning target is denoted as the concatenated sequence $l=[Think_{\text{GT}},g_{\text{GT}}]$. The model is trained to minimize the negative log-likelihood:

$$ $L_{\text{SFT}}=-\sum_{t=1}^{T}\log p_{\theta}(l_{t}\mid I,q,l_{<t})$ (3) $$

where $T$ is the total length of $l$, and $p_{\theta}$ is the model’s conditional next-token distribution.

Figure: Figure 4: Illustration of the reward design adopted in Stage 2 (GT-based RL training).
Refer to caption: x4.png

#### 3.4.3 Stage 2: Refinement with GT-Based Rewards

Building on SFT, Stage 2 continues training on specific tasks using only $(I,q)$ pairs and $g_{\text{GT}}$. The workflow generated by Builder is denoted $g$. We adopt two rewards at this stage:

Valid Reward ($R_{\text{valid}}$): It penalizes non-executable workflows as follows:

$$ $R_{\text{valid}}=\begin{cases}0,&\text{if $D$ successfully executes $g$}\\ -1,&\text{otherwise}\end{cases}$ (4) $$

Similarity Reward ($R_{\text{sim}}$): It measures alignment between $g$ and the expert workflow $g_{\text{GT}}$ using hierarchical graph matching. The depth of nodes in $g$ is calculated based on inverse topological order, which means the depth of the output node is 0. Matched nodes $M$ are found per depth layer using the Hungarian algorithm based on node similarity with a threshold of 0.6. The node similarity $\text{sim}_{\text{node}}$ is calculated by averaging the indicator of whether the same model is used and the proportion of identical parameters. $R_{\text{sim}}$ combines node coverage and average matched node similarity:

$$ $\displaystyle R_{\text{sim}}$ $\displaystyle=5\cdot\frac{|M|}{\max(|V_{g}|,|V_{g_{\text{GT}}}|)}+5\cdot\frac{1}{|M|}\sum_{(i,j)\in M}\text{sim}_{\text{node}}(i,j)$ (5) $$

where $V_{g}$ and $V_{g_{\text{GT}}}$ denote the node sets of the generated workflow $g$ and the ground truth workflow $g_{\text{GT}}$ respectively.

The total reward is: $r_{\text{stage2}}=R_{\text{valid}}+R_{\text{sim}}$.

Figure: Figure 5: Illustration of the reward design adopted in Stage 3 (GT-free RL training).
Refer to caption: x5.png

#### 3.4.4 Stage 3: Generalization with GT-Free Critic Rewards

Stage 3 targets generalization to open-domain instructions using only $(I,q)$ pairs. We employ the same valid reward $R_{\text{valid}}$ as Eq. [4](https://arxiv.org/html/2509.12883v1#S3.E4) and another effectiveness reward to provide feedback without a ground truth workflow:

Effectiveness Reward ($R_{\text{effect}}$): It uses an MLLM critic model to assess semantic alignment between the workflow’s effect and instruction $q$. Workflows are decomposed into several editing chains $c_{i}$, each of which contains only one editing model in $\mathcal{M}_{e}$ to perform actual editing. The MLLM critic would abstract each chain’s effect into a meta-edit description $m_{i}$ and then evaluate the description set $\{m_{i}\}$ against $q$. Specifically, the LLM must determine whether to remove existing editing chains or add new editing chains to better achieve the instruction. It must specify the number of chains to remove ($N_{\text{remove}}$), the number of chains to add ($N_{\text{add}}$), and the content of new chains. $R_{\text{effect}}$ applies a penalty defined as follows:

$$ $R_{\text{effect}}=1-0.5\cdot(N_{\text{add}}+N_{\text{remove}})$ (6) $$

The total reward is: $r_{\text{stage3}}=R_{\text{valid}}+R_{\text{effect}}$.

## 4 Experiments

We first demonstrate the model’s zero-shot capability, followed by extensive experiments validating the superiority of our framework on image editing benchmarks. Then we analyze the performance improvements and other advantages introduced by model-level tools, and finally demonstrate the Builder’s improved generalization and performance enabled by reinforcement learning.

### 4.1 Implementation Details

Builder: Our Builder is based on MiMo-VL-7B Team et al. (2025) and undergoes full-parameter fine-tuning in bf16 using a progressive three-stage curriculum: (1) 500 image-text pairs (instructions, thoughts and workflows), (2) 20K pairs (instructions and workflows), and (3) 50K pairs (instructions), all sourced from OmniEdit Wei et al. (2024). The generation of thoughts and workflows, along with the critic model used in stage 3 training, are both based on Qwen2.5-VL-72B Bai et al. (2025). Each stage trains for 1 epoch with AdamW (lr = $1\mathrm{e}{-5}$, $\beta=(0.9,0.999)$, weight decay = 0.01), without warmup or decay (batch size 8, image size $448\times 448$). Data construction details are provided in the supplementary material.

Predictive Tools: RES utilizes EVF-SAM Zhang et al. (2024), trained from scratch on 200K MS COCO Lin et al. (2014) samples (lr = $1\mathrm{e}{-4}$, batch size 64, resolution $512\times 512$, 10K iterations) with BCE and Dice loss.
SOS employs U2Net Qin et al. (2020), trained on 5K DIS Qin et al. (2022) with identical loss and optimization settings (batch size 24, 100 epochs, BCE loss only). Both models initialize without pretrained weights.
ADD-PRED and CAP-PRED share a Qwen2-VL-2B backbone Wang et al. (2024). CAP-PRED directly utilizes the base model’s inherent captioning capability. ADD-PRED is fine-tuned on 50K OmniEdit samples for addition/removal region prediction, formulated as bounding box regression between the source and target images; training uses 1 epoch and a learning rate of $1\mathrm{e}{-5}$.

Editing Tools: We adopt ICEdit’s Zhang et al. (2025) framework (FLUX-1 Fill Labs (2024b) backbone with LoRA fine-tuning at rank=32) but implement five specialized adapters for individual tasks, rather than the multi-task adapter. Each adapter is trained on 10K task-specific samples curated from OmniEdit Wei et al. (2024) and MagicBrush Zhang et al. (2023) via VIEScore Ku et al. (2023) assessment, using a global batch size of 8 for 10K steps at 768×768 resolution.

All the experiments utilized 8$\times$NVIDIA H20 GPUs for training.
We adopt DeepSpeed ZeRO-3 Rajbhandari et al. (2020) to accelerate training.

### 4.2 Evaluation Settings

To ensure authoritative evaluation, we benchmark our method on two widely adopted datasets: GEdit-Bench (606 samples) Liu et al. (2025) and ImgEdit (811 samples) Ye et al. (2025),
known for complex editing instructions and high-quality imagery.
Following standard protocols, we employ VIEScore executed by GPT-4o Hurst et al. (2024) as our primary metric. To ensure fairness and reproducibility, we fix the random seed to 0 and perform single-shot inference for all evaluations.
On GEdit-Bench using a single H20 GPU, our Builder takes 3.5 seconds and the slowest Tool 2.7 seconds, resulting in a total pipeline latency of $\sim$ 7.2 seconds, versus $>$ 25 seconds for the end-to-end method Bagel under identical settings.

Figure: Figure 6: Comparison with other methods on complex edits (top) and our Tool composition workflow (bottom).
Refer to caption: x6.png

### 4.3 Zero-Shot Capability of the Builder

Zero-Shot Complex Edits with Flexible Tool Composition: Fig. [6](https://arxiv.org/html/2509.12883v1#S4.F6) presents a visual comparison of editing results on flexible instructions, alongside the Builder’s tool composition process. For the “swap” instruction, although the Builder was not explicitly trained on this task, it effectively decomposes the instruction into atomic operations by first removing object A using RES and INPAINT, then inserting object B via ADD-PRED and FILL. This example shows its ability to compose specialized tools for flexible editing instructions, which enables complex edits beyond the reach of end-to-end or curated-pipeline models.

Figure: Figure 7: Example of Zero-Shot adaptation via feedback and Tool Insertion in reflection removal.
Refer to caption: x7.png

Zero-Shot Adaptation to Feedback and New Tools:
Fig. [15](https://arxiv.org/html/2509.12883v1#A3.F15) demonstrates the Builder’s adaptability to user feedback and new tools without retraining. For the reflection removal task, the Builder’s initial workflow (RES and INPAINT) failed because RES could not segment reflections effectively. Users can provide direct instructions, such as “don’t use RES before INPAINT” to prevent it. Guided by this feedback, the Builder revises its workflow: it uses SOS for foreground segmentation, INVERSE to infer the background, and then INPAINT to remove part of the reflection. Additionally, users can introduce a dedicated reflection-removal tool (RRF), which the Builder readily adopts to solve the task effectively.
This illustrates the system’s adaptability to extend capabilities by integrating new tools or incorporating user feedback, all without modifying the Builder.

Figure: Figure 8: Quantitative evaluation on GEdit-Bench-EN. All metrics are reported as higher-is-better ($\uparrow$).
Refer to caption: x8.png

### 4.4 Comparison with State-of-the-Art

GEdit-Bench: As shown in Tab. [9](https://arxiv.org/html/2509.12883v1#S4.F9), our method achieves the highest image preservation score on GEdit-Bench-EN, with G_PQ = 7.7, and delivers the best overall performance with G_O = 6.64, outperforming all competing approaches.
As shown in Fig. [9](https://arxiv.org/html/2509.12883v1#S4.F9), Lego-Edit excels in fine-grained sub-tasks such as color change and material replacement. This precision is attributed to the Builder composing RES to execute these tasks with minimal impact on non-target regions, as detailed in the section on Effectiveness of Tool Composition.
Compared to traditional manual-workflow API agents like ComfyMind, which depend on pre-scripted pipelines, Lego-Edit achieves notable performance gains via a capable Builder and flexible orchestration of specialized Tools.

**Table 1: Quantitative evaluation on ImgEdit-Bench. All metrics are reported as higher-is-better ($\uparrow$).**
| Model | Add | Adjust | Extract | Replace | Remove | Style | Action | Hybrid | Background | Overall$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MagicBrush Zhang et al. (2023) | 2.72 | 1.47 | 1.31 | 1.89 | 1.57 | 2.49 | 1.39 | 1.80 | 2.03 | 1.85 |
| Instruct-P2P Brooks et al. (2023) | 2.29 | 1.79 | 1.33 | 1.93 | 1.49 | 3.54 | 1.51 | 1.48 | 1.67 | 1.89 |
| AnyEdit Yu et al. (2025) | 3.12 | 2.66 | 1.82 | 2.71 | 2.34 | 3.27 | 3.31 | 2.07 | 2.37 | 2.63 |
| UltraEdit Zhao et al. (2024) | 3.63 | 3.01 | 2.02 | 3.13 | 1.71 | 3.69 | 3.57 | 2.33 | 3.31 | 2.93 |
| Step1X-Edit Liu et al. (2025) | 3.90 | 3.13 | 1.87 | 3.45 | 2.61 | 4.44 | 3.43 | 2.52 | 3.19 | 3.17 |
| BAGEL Deng et al. (2025) | 3.55 | 3.30 | 1.56 | 3.38 | 2.44 | 4.24 | 4.29 | 2.55 | 3.22 | 3.17 |
| UniWorld-V1 Lin et al. (2025) | 3.86 | 3.70 | 2.23 | 3.49 | 3.54 | 4.22 | 3.44 | 3.13 | 2.76 | 3.37 |
| ComfyMind Guo et al. (2025) | 1.45 | 3.14 | 2.21 | 3.43 | 2.81 | 2.66 | 2.74 | 0.57 | 2.26 | 2.63 |
| OmniGen2 Wu et al. (2025) | 3.57 | 3.06 | 1.77 | 3.74 | 3.20 | 4.81 | 4.68 | 2.52 | 3.57 | 3.44 |
| Ours | 3.67 | 3.82 | 2.47 | 3.22 | 3.39 | 4.47 | 4.01 | 3.18 | 3.24 | 3.50 |

ImgEdit-Bench: Lego-Edit maintains top performance on ImgBench,
achieving the highest overall score (3.50) among all compared methods. The detailed results are shown in Tab. [1](https://arxiv.org/html/2509.12883v1#S4.T1).
Crucially, our framework dominates the most challenging Hybrid Editing sub-task (3.18). This success validates our proposition that the Builder can parse composite instructions into atomic sub-tasks and dynamically generate workflows to coordinate specialized Tools.

Figure: Figure 10: Qualitative results compared to other methods.
Refer to caption: x9.png

Qualitative Results: As illustrated in Fig. [10](https://arxiv.org/html/2509.12883v1#S4.F10), our method outperforms other approaches in both edit accuracy and visual realism. Edits are well-aligned with the intended regions and maintain high perceptual quality. More comparative results are provided in the supplementary material.

### 4.5 Ablation Study on Tools

Necessity of Task-Specialized Tools.
To validate Lego-Edit’s design, we compare its task-specialized architecture with a unified alternative using identical settings. Three separate LoRA adapters are trained on 10K samples per task, while the unified model uses a combined 30K dataset. As shown in Tab. [2](https://arxiv.org/html/2509.12883v1#S4.T2), specialized models outperform the unified one (e.g., 6.83 vs. 5.94 in color alteration). Increasing LoRA rank in the unified model brings no gain. Qualitative results (provided in supplementary material) reveal frequent task confusion in the unified setup, highlighting the importance of specialization for editing fidelity.

**Table 2: Comparison between Multi-task LoRA and Single-task LoRA on GEdit-Bench-EN, we report the G_O metric.**
| Model | Rank | Color | Style | Tone |
| --- | --- | --- | --- | --- |
| Alteration | Transfer | Transformation |  |  |
| Multi-task LoRA | 32 | 5.94 | 5.47 | 5.56 |
| Multi-task LoRA | 64 | 5.42 | 5.43 | 5.57 |
| Single-task LoRA | 32 | 6.83 | 6.75 | 6.63 |

### 4.6 Ablation Studies on Builder

**Table 3: Performance is evaluated by execution success rate, indicating error-free syntax, and VIEScore, assessing editing quality. Success rate comprises Simple %Pass on original instructions and Complex %Pass on three GPT-4o-generated variants per instruction.**
| Agent Model | Simple | Complex | G_SC$\uparrow$ | G_PQ$\uparrow$ | G_O$\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| %Pass | %Pass |  |  |  |  |
| GPT-4o | 75.6 | 53.8 | 4.25 | 6.64 | 5.12 |
| MiMo-VL-7B | 57.2 | 47.7 | 3.24 | 4.87 | 3.88 |
| Builder-SFT | 80.6 | 55.9 | 4.83 | 6.94 | 5.89 |
| Builder-RL w/ GT | 100 | 83.6 | 5.58 | 7.32 | 6.17 |
| Builder-RL w/o GT | 100 | 99.0 | 5.99 | 7.45 | 6.64 |

Effectiveness of Reinforcement Learning Training.
Ablations on GEdit-Bench in Tab. [3](https://arxiv.org/html/2509.12883v1#S4.T3) show the effectiveness of our progressive RL training. Starting from Builder-SFT, which outperforms basemodel MiMo-VL-7B and powerful MLLM GPT-4o on simple and complex success rates and VIEScores, subsequent RL training with ground truth (Builder-RL w/ GT) achieves 100% simple and 83.6% complex success with better VIEScores. Final GT-free RL training (Builder-RL w/o GT) maintains 100% simple success, boosts complex success to 99.0%, and achieves the highest VIEScores. This complex task improvement, despite identical simple accuracy, indicates more accurate parameter output and enhanced compositional understanding of tool interfaces, enabling reliable execution under flexible instructions.

Effectiveness of Tool Composition. To evaluate the impact of explicit Tool composition in Lego-Edit, we examine its performance across GEdit-Bench sub-tasks.
As shown in Tab. [4](https://arxiv.org/html/2509.12883v1#S4.T4), integrating RES segmentation masks with RCM increases G_O for color alteration and material modification, highlighting the benefits of RES’s spatial control.
Similarly, incorporating CAP-PRED text descriptions into STYLE boosts G_O in style transfer, as textual cues enforce semantic alignment between stylized outputs and the source image.
These results demonstrate that Tool composition improves performance, confirming that Lego-Edit coordinates specialized tools to achieve superior editing performance.

**Table 4: Comparison between separately adopted tools and with Builder compose on GEdit-Bench-EN.**
| Task | Compose Pattern | G_O |
| --- | --- | --- |
| Color Alteration | RCM | 6.83 |
| RCM + RES | 7.70 |  |
| Material Modification | RCM | 6.08 |
| RCM + RES | 6.92 |  |
| Style Transfer | STYLE | 6.75 |
| STYLE + CAP-PRED | 7.37 |  |

## 5 Conclusion

In this paper, we introduce Lego-Edit, a novel agent-based framework for generative instruction-based image editing. It employs an RL-finetuned MLLM (Builder) to orchestrate model-level editing tools (Bricks), enabled by two key innovations: fine-grained tools allowing flexible composition and precise control, and a three-stage progressive RL training strategy enhancing reasoning and tool organization abilities via GT-free feedback on open-domain instructions. Extensive experiments demonstrate Lego-Edit’s state-of-the-art performance on GEdit-Bench and ImgBench, showcasing superior accuracy and generalization in handling flexible requests and integrating new tools without retraining. Future work will expand the tool set for broader capabilities and explore agent-based feedback for better robustness.

## References

- Bai et al. (2025)
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al.
Qwen2. 5-vl technical report.
*arXiv preprint arXiv:2502.13923*, 2025.
- Brooks et al. (2023)
Tim Brooks, Aleksander Holynski, and Alexei A Efros.
Instructpix2pix: Learning to follow image editing instructions.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp.  18392–18402, 2023.
- Cai et al. (2025)
Qi Cai, Jingwen Chen, Yang Chen, Yehao Li, Fuchen Long, Yingwei Pan, Zhaofan Qiu, Yiheng Zhang, Fengbin Gao, Peihan Xu, et al.
Hidream-i1: A high-efficient image generative foundation model with sparse diffusion transformer.
*arXiv preprint arXiv:2505.22705*, 2025.
- Deng et al. (2025)
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, et al.
Emerging properties in unified multimodal pretraining.
*arXiv preprint arXiv:2505.14683*, 2025.
- Du et al. (2024)
Yu Du, Fangyun Wei, and Hongyang Zhang.
Anytool: Self-reflective, hierarchical agents for large-scale api calls.
*arXiv preprint arXiv:2402.04253*, 2024.
- Gao et al. (2025)
Daiheng Gao, Shilin Lu, Wenbo Zhou, Jiaming Chu, Jie Zhang, Mengxi Jia, Bang Zhang, Zhaoxin Fan, and Weiming Zhang.
Eraseanything: Enabling concept erasure in rectified flow transformers.
In *Forty-second International Conference on Machine Learning*, 2025.
- Guo et al. (2025)
Litao Guo, Xinli Xu, Luozhou Wang, Jiantao Lin, Jinsong Zhou, Zixin Zhang, Bolan Su, and Ying-Cong Chen.
Comfymind: Toward general-purpose generation via tree-based planning and reactive feedback.
*arXiv preprint arXiv:2505.17908*, 2025.
- Han et al. (2024)
Zhen Han, Zeyinzi Jiang, Yulin Pan, Jingfeng Zhang, Chaojie Mao, Chenwei Xie, Yu Liu, and Jingren Zhou.
Ace: All-round creator and editor following instructions via diffusion transformer.
*arXiv preprint arXiv:2410.00086*, 2024.
- Hertz et al. (2022)
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.
Prompt-to-prompt image editing with cross attention control.
*arXiv preprint arXiv:2208.01626*, 2022.
- Huang et al. (2025a)
Runhui Huang, Chunwei Wang, Junwei Yang, Guansong Lu, Yunlong Yuan, Jianhua Han, Lu Hou, Wei Zhang, Lanqing Hong, Hengshuang Zhao, et al.
Illume+: Illuminating unified mllm with dual visual tokenization and diffusion refinement.
*arXiv preprint arXiv:2504.01934*, 2025a.
- Huang et al. (2024)
Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, et al.
Smartedit: Exploring complex instruction-based image editing with multimodal large language models.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp.  8362–8371, 2024.
- Huang et al. (2025b)
Zeyi Huang, Yuyang Ji, Anirudh Sundara Rajan, Zefan Cai, Wen Xiao, Junjie Hu, and Yong Jae Lee.
Visualtoolagent (vista): A reinforcement learning framework for visual tool selection.
*arXiv preprint arXiv:2505.20289*, 2025b.
- Hurst et al. (2024)
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al.
Gpt-4o system card.
*arXiv preprint arXiv:2410.21276*, 2024.
- Ku et al. (2023)
Max Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen.
Viescore: Towards explainable metrics for conditional image synthesis evaluation.
*arXiv preprint arXiv:2312.14867*, 2023.
- Labs (2024a)
Black Forest Labs.
Flux.
[https://github.com/black-forest-labs/flux](https://github.com/black-forest-labs/flux), 2024a.
- Labs (2024b)
Black Forest Labs.
Flux.1-fill-dev.
[https://huggingface.co/black-forest-labs/FLUX.1-Fill-dev](https://huggingface.co/black-forest-labs/FLUX.1-Fill-dev), 2024b.
- Lin et al. (2025)
Bin Lin, Zongjian Li, Xinhua Cheng, Yuwei Niu, Yang Ye, Xianyi He, Shenghai Yuan, Wangbo Yu, Shaodong Wang, Yunyang Ge, et al.
Uniworld: High-resolution semantic encoders for unified visual understanding and generation.
*arXiv preprint arXiv:2506.03147*, 2025.
- Lin et al. (2014)
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.
Microsoft coco: Common objects in context.
In *European conference on computer vision*, pp.  740–755. Springer, 2014.
- Liu et al. (2025)
Shiyu Liu, Yucheng Han, Peng Xing, Fukun Yin, Rui Wang, Wei Cheng, Jiaqi Liao, Yingming Wang, Honghao Fu, Chunrui Han, et al.
Step1x-edit: A practical framework for general image editing.
*arXiv preprint arXiv:2504.17761*, 2025.
- Qin et al. (2020)
Xuebin Qin, Zichen Zhang, Chenyang Huang, Masood Dehghan, Osmar R Zaiane, and Martin Jagersand.
U2-net: Going deeper with nested u-structure for salient object detection.
*Pattern recognition*, 106:107404, 2020.
- Qin et al. (2022)
Xuebin Qin, Hang Dai, Xiaobin Hu, Deng-Ping Fan, Ling Shao, and Luc Van Gool.
Highly accurate dichotomous image segmentation.
In *European Conference on Computer Vision*, pp.  38–56. Springer, 2022.
- Qin et al. (2023)
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al.
Toolllm: Facilitating large language models to master 16000+ real-world apis.
*arXiv preprint arXiv:2307.16789*, 2023.
- Rajbhandari et al. (2020)
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He.
Zero: Memory optimizations toward training trillion parameter models.
In *SC20: International Conference for High Performance Computing, Networking, Storage and Analysis*, pp.  1–16. IEEE, 2020.
- Shao et al. (2024)
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
*arXiv preprint arXiv:2402.03300*, 2024.
- Team et al. (2025)
Core Team, Zihao Yue, Zhenru Lin, Yifan Song, Weikun Wang, Shuhuai Ren, Shuhao Gu, Shicheng Li, Peidian Li, Liang Zhao, Lei Li, et al.
Mimo-vl technical report, 2025.
URL [https://arxiv.org/abs/2506.03569](https://arxiv.org/abs/2506.03569).
- Wang et al. (2024)
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al.
Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.
*arXiv preprint arXiv:2409.12191*, 2024.
- Wei et al. (2024)
Cong Wei, Zheyang Xiong, Weiming Ren, Xeron Du, Ge Zhang, and Wenhu Chen.
Omniedit: Building image editing generalist models through specialist supervision.
In *The Thirteenth International Conference on Learning Representations*, 2024.
- Wu et al. (2025)
Chenyuan Wu, Pengfei Zheng, Ruiran Yan, Shitao Xiao, Xin Luo, Yueze Wang, Wanli Li, Xiyan Jiang, Yexin Liu, Junjie Zhou, et al.
Omnigen2: Exploration to advanced multimodal generation.
*arXiv preprint arXiv:2506.18871*, 2025.
- Xue et al. (2025)
Xiangyuan Xue, Zeyu Lu, Di Huang, Zidong Wang, Wanli Ouyang, and Lei Bai.
Comfybench: Benchmarking llm-based agents in comfyui for autonomously designing collaborative ai systems.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pp.  24614–24624, 2025.
- Ye et al. (2025)
Yang Ye, Xianyi He, Zongjian Li, Bin Lin, Shenghai Yuan, Zhiyuan Yan, Bohan Hou, and Li Yuan.
Imgedit: A unified image editing dataset and benchmark.
*arXiv preprint arXiv:2505.20275*, 2025.
- Yu et al. (2025)
Qifan Yu, Wei Chow, Zhongqi Yue, Kaihang Pan, Yang Wu, Xiaoyang Wan, Juncheng Li, Siliang Tang, Hanwang Zhang, and Yueting Zhuang.
Anyedit: Mastering unified high-quality image editing for any idea.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pp.  26125–26135, 2025.
- Zhang et al. (2023)
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su.
Magicbrush: A manually annotated dataset for instruction-guided image editing.
*Advances in Neural Information Processing Systems*, 36:31428–31449, 2023.
- Zhang et al. (2024)
Yuxuan Zhang, Tianheng Cheng, Lianghui Zhu, Rui Hu, Lei Liu, Heng Liu, Longjin Ran, Xiaoxin Chen, Wenyu Liu, and Xinggang Wang.
Evf-sam: Early vision-language fusion for text-prompted segment anything model.
*arXiv preprint arXiv:2406.20076*, 2024.
- Zhang et al. (2025)
Zechuan Zhang, Ji Xie, Yu Lu, Zongxin Yang, and Yi Yang.
In-context edit: Enabling instructional image editing with in-context generation in large scale diffusion transformer.
*arXiv preprint arXiv:2504.20690*, 2025.
- Zhao et al. (2024)
Haozhe Zhao, Xiaojian Shawn Ma, Liang Chen, Shuzheng Si, Rujie Wu, Kaikai An, Peiyu Yu, Minjia Zhang, Qing Li, and Baobao Chang.
Ultraedit: Instruction-based fine-grained image editing at scale.
*Advances in Neural Information Processing Systems*, 37:3058–3093, 2024.
- Zheng et al. (2024a)
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su.
Gpt-4v (ision) is a generalist web agent, if grounded.
*arXiv preprint arXiv:2401.01614*, 2024a.
- Zheng et al. (2024b)
Yuanhang Zheng, Peng Li, Wei Liu, Yang Liu, Jian Luan, and Bin Wang.
Toolrerank: Adaptive and hierarchy-aware reranking for tool retrieval.
*arXiv preprint arXiv:2403.06551*, 2024b.

## Appendix A Details of Model-Level Tools

We provide detailed specifications of all model-level tools in Tab. [5](https://arxiv.org/html/2509.12883v1#A1.T5), including their names, functions, inputs, and outputs.

**Table 5: Specification of Model-Level Tools: Names, functions, inputs, and outputs.**
| Tool Type | Tool Name | Function | Input | Output |
| --- | --- | --- | --- | --- |
|  | RES | Segment object specified by prompt. | Image[image], | Mask[mask], |
|  | Str[prompt] | Image[image] |  |  |
|  | SOS | Segment salient objects in image. | Image[image] | Mask[mask], |
|  | Image[image] |  |  |  |
|  | ADD-PRED | Predict optimal position to add target object (specified by prompt). If mask provided, position must be within mask. | Image[image], | Mask[mask] |
|  | Str[prompt], |  |  |  |
|  | Mask[mask] |  |  |  |
| Predictive | CAP-PRED | Generate English caption describing the image for FLUX model conditioning. For image expansion tasks, requires expansion ratios; outputs caption, expanded image, and expansion mask. Otherwise, ratios null, outputs caption only (image/mask null). | Image[image], |  |
|  | Float[left_ratio], | Str[caption], |  |  |
| Models | Float[right_ratio], | Image[image], |  |  |
|  | Float[top_ratio], | Mask[mask] |  |  |
|  | Float[bottom_ratio] |  |  |  |
|  | INVERSE | Subtract mask2 from mask1 or image2 from image1. If mask1 is null, subtract mask2 from a full mask. | Mask[mask1], |  |
|  | Mask[mask2], | Mask[mask], |  |  |
|  | Image[image1], | Image[image] |  |  |
|  | Image[image2] |  |  |  |
|  | BBOX | Compute bounding box mask from input mask. | Mask[mask] | Mask[mask] |
|  | FILL | Generate content within mask region based on prompt. Not for object color/material replacement. | Image[image], |  |
|  | Mask[mask], | Image[image] |  |  |
|  | Str[prompt], |  |  |  |
|  | Image[preimage] |  |  |  |
|  | FASTINPAINT | Perform fast inpainting and output an effectiveness score. | Image[image], | Image[image], |
|  | Mask[mask] | Float[score] |  |  |
|  | INPAINT | Fill background in mask region, using preimage as reference and requiring a quality score threshold. | Image[image], | Image[image] |
|  | Mask[mask], |  |  |  |
|  | Image[preimage], |  |  |  |
|  | Float[score] |  |  |  |
|  | RCM | Replace color or material of object within mask. | Image[image], | Image[image] |
|  | Mask[mask], |  |  |  |
| Editing | Str[prompt] |  |  |  |
|  | STYLE | Convert style of entire image or object within mask to specified style (e.g., ’anime style’). | Image[image], | Image[image] |
|  | Mask[mask], |  |  |  |
| Models | Str[prompt], |  |  |  |
|  | Str[style] |  |  |  |
|  | ENV | Alter environment (e.g., weather, time of day) of an object. | Image[image], | Image[image] |
|  | Str[prompt] |  |  |  |
|  | POSE | Modify posture or expression of an object. | Image[image], | Image[image] |
|  | Str[prompt] |  |  |  |
|  | COMPOSE | Combine two masks or images. Overlapping regions use the second input. | Mask[mask1], |  |
|  | Mask[mask2], | Mask[mask], |  |  |
|  | Image[image1], | Image[image] |  |  |
|  | Image[image2] |  |  |  |
|  | RESIZE | Resize the valid region of input mask or image by a given scaling ratio. | Mask[mask], | Mask[mask], Image[image] |
|  | Image[image], |  |  |  |
|  | Float[ratio] |  |  |  |

## Appendix B Additional Qualitative Comparisons with State-of-the-Art Methods

We present extended qualitative comparisons with state-of-the-art (SOTA) methods in Fig. [11](https://arxiv.org/html/2509.12883v1#A2.F11). The comparison methods include InstructP2P Brooks et al. (2023), ComfyMind Guo et al. (2025), BAGEL Deng et al. (2025), OmniGen2 Wu et al. (2025), Step-1X Liu et al. (2025), Uniworld-V1 Lin et al. (2025).

Figure: Figure 11: Qualitative comparison with state-of-the-art methods.
Refer to caption: x10.png

Further examples demonstrating performance across different aspect ratios are shown in Fig. [13](https://arxiv.org/html/2509.12883v1#A2.F13).

Fig. [12](https://arxiv.org/html/2509.12883v1#A2.F12) showcases results achieved with more complex and flexible instructions.

Figure: Figure 12: Qualitative results of handling complex and flexible editing instructions.
Refer to caption: x11.png

Figure: Figure 13: Qualitative results for image editing at varying aspect ratios.
Refer to caption: x12.png

## Appendix C Qualitative Analysis of Task Confusion in Joint Training

As discussed in the main paper, joint training approaches such as ICEdit Zhang et al. (2025) can suffer from task confusion. While quantitative evidence is presented therein, Fig. [14](https://arxiv.org/html/2509.12883v1#A3.F14) provides qualitative examples illustrating this phenomenon.

Figure: Figure 14: Qualitative examples of task confusion arising from joint model training.
Refer to caption: x13.png

Figure: Figure 15: Qualitative improvements achieved through strategic composition of model-level tools.
Refer to caption: x14.png

## Appendix D Qualitative Benefits of Tool Composition

The main paper presents quantitative results demonstrating the performance gains achieved by composing specific tools (e.g., RES for color/style change, CAP-PRED for style change). Fig. [15](https://arxiv.org/html/2509.12883v1#A3.F15) offers corresponding qualitative evidence of these improvements.

## Appendix E Complete Prompt Template for the Builder Agent

The full prompt used to configure the Builder agent is provided in Listing [1](https://arxiv.org/html/2509.12883v1#LST1).

Figure: Listing 1: Complete Prompt for the Builder Agent

## Appendix F Prompt Example for Producing Ground Truth Data

Listing [2](https://arxiv.org/html/2509.12883v1#LST2) provides an exemplar prompt specifically used for generating ground truth training data for the environment change task.

Figure: Listing 2: Prompt Example for Generating Ground Truth Data for the Environment Change Task

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. What is your opinion on whether the image generation area will pursue a strong end-to-end model or develop an agent that uses specific models?
soundness: 2
strengths: 1. This paper proposes a new direction to handle complex human-instructed image-editing tasks that uses existing image-editing models to construct an agent.
2. It shows superior experimental results when compared with other SOTA image generation models.
summary: This paper proposes a general instruction-based image editing framework addressing poor generalization of existing methods to out-of-training-domain instructions. It uses a reinforcement learning (RL)-fine-tuned Multimodal Large Language Model (MLLM) called Builder to organize model-level tools (Bricks).

Bricks include predictive models (e.g., RES for object segmentation, ADD-PRED for adding position prediction) and editing models (e.g., INPAINT for inpainting, STYLE for style transfer), each trained independently for flexibility and performance.

Its three-stage progressive RL training: first, Supervised Fine-Tuning (SFT) builds basic capabilities; second, GT-based RL optimizes tool composition; third, GT-free RL uses an MLLM critic for feedback to enhance open-domain instruction handling.

Experiments show LEGO-Edit achieves state-of-the-art on GEdit-Bench and ImgBench. It handles complex multi-step edits, adapts to new tools/feedback zero-shot, maintains non-edited region consistency, and supports Chinese instructions and text editing.
weaknesses: 1. The description of training Builder is not sufficient, especially for the reinforcement learning section.  How to obtain an accurate reward when dealing with complex scenarios?
2. How about the results when compared with the Qwen-image and Seedream 3.0/4.0 series, which are end-to-end image generation models? 
3. There seem to be no failure cases presented in this paper. How to deal with the situation where the agent determines the error workflow, which may obtain terrible results? And how about the success rate of the agent?

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: 1. The tools are not always accurate. How to handle this situation?
2. How to avoid the error accumulation when using the tool chain?
soundness: 4
strengths: 1. The paper presents a valuable task formulation: how to orchestrate multiple sufficiently strong single-function perception models and editing models to achieve automated, instruction-compliant, and flexible image editing.
2. The proposed method in the paper is ingenious yet intuitive. Its multi-stage training strategy, tailored for the task, along with the reward design, effectively addresses the challenge of imbalanced training data annotations.
3. The results achieved by the proposed method are satisfactory, positioning it at the forefront of current community standards from both qualitative and quantitative perspectives.
4. The authors' approach to solving the problem is concise and clear, making it easy to understand and facilitating follow-up research.
summary: This paper proposes a method for fine-tuning MLLMs to establish an automated workflow for instruction-based image editing tasks. By appropriately invoking pre-trained predictive models and editing models, this workflow achieves a variety of editing tasks and even exhibits a certain level of zero-shot capability for flexible editing requirements. Extensive visual results demonstrate the reliability of the method in terms of editing effectiveness, while quantitative results indicate its performance improvement over comparative methods.
weaknesses: 1. Lacking a comparison with manual workflow construction would better highlight the efficiency and automation advantages of the proposed method.
2 . Lacking a dedicated limitations section. A section of limitations would improve the academic rigor and transparency of the work.
3. Authors need to provide a clearer explanation of the dataset construction for each training stage of the Builder to facilitate reader comprehension. If the authors used Qwen2.5-VL to generate ground truth data, why did they opt for the MLLM trained in the paper instead of directly calling the Qwen API in practical applications?

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: See weaknesses. I think this paper is ok, but we can have more discussions during the rebuttal period. If my concerns are solved, I will be happy to raise the score. I hope the discussions can also bring about some new insights.
soundness: 2
strengths: 1. The paper is in a good organization, the idea is easy to understand. 
2. The experiment section is good, with abundant results and clear explanation. So I think the results are convincing.
summary: This paper targets on the problem that though have somewhat good editing performance, the current instruction-based image editing model always fail to generalize to unknown instructions. They propose a method called Lego-Edit, it finetunes a Multi-modal Large Language Model (MLLM) by reinforcement learning, enable it to coordinate model-level editing tools. In the MLLM part, it three-stage progressive reinforcement learning training strategy to make the MLLM more adapted to the current task. Extensive experiment results prove the effectiveness of the proposed method.
weaknesses: 1. Some typos, eg. Line 182, fine-graine. Please check the whole paper. 
2. If I do not mis-understand, since you incorporate more models, can you have some discussions about the train / inference efficiency of the proposed method?
3. I am willing to see more discussions about the novelty of the proposed method, since dividing one editing task to multiple different tasks is not interesting enough I think. 
4. I also want the authors to have more discussions about the limitation and future work of the work.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. How dependent is Lego-Edit’s performance on the chosen MLLM backbone (MiMo-VL-7B)? Could smaller or publicly available models achieve comparable coordination ability?
2. Could the authors report quantitative results comparing Stage 2 (GT-based RL) and Stage 3 (critic-based RL) to show how much each stage improves reasoning or compositional success rate?
3. How does the framework scale when the number of available tools increases or when workflows become more complex? Are there computational or latency trade-offs?
4. What are the typical failure modes? For example, does the Builder sometimes misinterpret instructions, or do tool boundaries introduce visual artifacts?
5. Since the paper is positioned in “applications to multiple modalities,” how feasible would it be to extend this framework to non-visual domains such as audio or video editing?
6. Have the authors evaluated how easily new users can add tools or control the Builder’s behavior without retraining? Understanding the system’s real-world maintainability could increase its practical impact.
7. Beyond the single latency result against BAGEL, could the authors provide runtime or computational cost comparisons with other leading systems such as Step1X-Edit, OmniGen2, or FLUX.1 Kontext? Such information would clarify Lego-Edit’s practical efficiency.
soundness: 3
strengths: - The paper introduces a well-motivated modular architecture that addresses the limited flexibility of end-to-end image-editing models.
- The three-stage reinforcement-learning pipeline is a strong design choice that contributes to improved reasoning and generalization.
- Experimental results are extensive and include both quantitative benchmarks and visual demonstrations that convincingly illustrate the system’s strengths.
- The implementation details and reproducibility statement are thorough and transparent.
- The framework shows promising potential for extensibility and future multimodal applications.
summary: This paper introduces Lego-Edit, a modular framework for instruction-based image editing. The system consists of a Builder (a multimodal large language model fine-tuned with reinforcement learning) that organizes a library of model-level tools, Bricks, such as segmentation, inpainting, and style-transfer modules. A three-stage progressive RL scheme, supervised fine-tuning, ground-truth-based RL, and GT-free RL guided by a critic mode, gradually improves the Builder’s reasoning and tool-composition ability. Experiments on GEdit-Bench and ImgEdit-Bench show performance gains over strong baselines (e.g., BAGEL, Step1X-Edit, UniWorld-V1). The system also demonstrates zero-shot generalization and the ability to integrate new tools without retraining.
weaknesses: - The contribution is primarily engineering-focused. The paper combines known techniques, MLLM agents, RL fine-tuning, LoRA adapters, into a unified system rather than introducing a new theoretical or algorithmic idea. While the integration is impressive, it offers limited conceptual insight into why or how the components interact optimally.
- The complexity of the architecture (Builder, Executor, Bricks, Critic) makes it difficult to interpret which design choices drive performance gains. More controlled ablations would clarify the marginal value of each component.
- The critic model used in the final RL stage plays an important conceptual role but is evaluated mostly qualitatively; quantitative ablations or alternative critic formulations would strengthen the argument.
- The writing and structure could be improved to highlight the core intuition behind the framework before diving into system specifics.
- The evaluation scope is limited to existing benchmarks. Although results are strong, the paper would benefit from evidence of user studies or real-world use cases demonstrating the practicality of Lego-Edit in diverse conditions.
- Some important limitations and failure modes are only briefly mentioned (e.g., potential tool conflicts, scalability bottlenecks) and deserve more explicit discussion.
- Runtime analysis is incomplete. The paper reports latency only relative to BAGEL but does not compare against other strong baselines such as Step1X-Edit, OmniGen2, or FLUX.1 Kontext. A broader efficiency analysis would help position Lego-Edit’s computational trade-offs relative to state-of-the-art models.


Decision: Reject