OmniEdit: Building Image Editing Generalist Models Through Specialist Supervision

## Abstract

Abstract Instruction-guided image editing methods have demonstrated significant potential by training diffusion models on automatically synthesized or manually annotated image editing pairs. However, these methods remain far from practical, real-life applications. We identify three primary challenges contributing to this gap. Firstly, existing models have limited editing skills due to the biased synthesis process. Secondly, these methods are trained with datasets with a high volume of noise and artifacts. This is due to the application of simple filtering methods like CLIP-score. Thirdly, all these datasets are restricted to a single low resolution and fixed aspect ratio, limiting the versatility to handle real-world use cases.
In this paper, we present Omni-Edit , which is an omnipotent editor to handle seven different image editing tasks with any aspect ratio seamlessly. Our contribution is in four folds: (1) Omni-Edit is trained by utilizing the supervision from seven different specialist models to ensure task coverage. (2) we utilize importance sampling based on the scores provided by large multimodal models (like GPT-4o) instead of CLIP-score to improve the data quality. (3) we propose a new editing architecture called EditNet to greatly boost the editing success rate, (4) we provide images with different aspect ratios to ensure that our model can handle any image in the wild. We have curated a test set containing images of different aspect ratios, accompanied by diverse instructions to cover different tasks. Both automatic evaluation and human evaluations demonstrate that Omni-Edit can significantly outperform all the existing models.
Our code, dataset and model will be available at https://tiger-ai-lab.github.io/OmniEdit/

## 1 Introduction

Image editing, particularly when following user instructions to apply semantic transformations to real-world photos, has seen significant advancements. Recently, text-guided image editing (Brooks et al., ) has gained prominence over traditional methods such as mask-based or region-based editing (Meng et al., ). With the rise of diffusion models (Rombach et al., ; Podell et al., ; Chen et al., ; Sauer et al., ), numerous diffusion-based image editing techniques have emerged. Generally, they can be roughly divided into two types: (1) Inversion-based methods (Parmar et al., ; Kawar et al., ; Gal et al., ; Xu et al., ; Tumanyan et al., ; Tsaban & Passos, ) propose to perform zero-shot image editing by inverting the diffusion process and manipulating the attention map in the intermediate diffusion steps to achieve desired editing goal. (2) End-to-end methods (Brooks et al., ; Zhang et al., ; Sheynin et al., ; Zhao et al., ; Fu et al., ) propose to fine-tune an existing diffusion model on large-scale image editing pairs to learn the editing operation in an end-to-end fashion. End-to-end methods have generally achieved better performance than inversion-based methods and gained higher popularity.

**Table 1: Comparison of Omni-Edit with all the existing end-to-end image editing models. The scores are based on a preliminary studies on around 50 prompts.**
| Property | InstructP2P | MagicBrush | UltraEdit | MGIE | HQEdit | CosXL | Omni-Edit |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Training Dataset Properties |  |  |  |  |  |  |  |
| Real Image? | ✗ | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ |
| Any Res? | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ |
| High Res? | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✓ |
| Fine-grained Image Editing Skills |  |  |  |  |  |  |  |
| Obj-Swap | \faStar\faStar\faStarO | \faStar\faStar\faStarO | \faStar\faStar\faStarO | \faStar\faStarHalfO\faStarO | \faStar\faStar\faStarO | \faStar\faStarO\faStarO | \faStar\faStar\faStarHalfO |
| Obj-Add | \faStar\faStarO\faStarO | \faStar\faStar\faStarO | \faStar\faStarO\faStarO | \faStar\faStarHalfO\faStarO | \faStar\faStarO\faStarO | \faStar\faStarO\faStarO | \faStar\faStar\faStarHalfO |
| Obj-Remove | \faStar\faStarO\faStarO | \faStar\faStar\faStarO | \faStar\faStarO\faStarO | \faStar\faStarHalfO\faStarO | \faStar\faStarO\faStarO | \faStar\faStarO\faStarO | \faStar\faStar\faStarHalfO |
| Attribute | \faStar\faStar\faStarO | \faStar\faStarO\faStarO | \faStar\faStar\faStarO | \faStar\faStarHalfO\faStarO | \faStar\faStarO\faStarO | \faStar\faStarO\faStarO | \faStar\faStar\faStarHalfO |
| Back-Swap | \faStar\faStar\faStarO | \faStar\faStar\faStarO | \faStar\faStar\faStarO | \faStar\faStarHalfO\faStarO | \faStar\faStar\faStarO | \faStar\faStar\faStarO | \faStar\faStar\faStarHalfO |
| Environment | \faStar\faStarO\faStarO | \faStar\faStarO\faStarO | \faStar\faStarO\faStarO | \faStar\faStarHalfO\faStarO | \faStar\faStarO\faStarO | \faStar\faStar\faStarO | \faStar\faStar\faStarHalfO |
| Style | \faStar\faStar\faStarO | \faStar\faStarO\faStarO | \faStar\faStar\faStarO | \faStar\faStarHalfO\faStarO | \faStar\faStarO\faStarO | \faStar\faStar\faStarHalfO | \faStar\faStar\faStarHalfO |

Despite their effectiveness, end-to-end methods face a significant limitation: the scarcity of human-annotated image editing pairs. As a result, all current end-to-end approaches depend on synthetic training data. For instance, existing datasets are synthesized using techniques such as Prompt2Prompt (Hertz et al., ) or mask-based editing models like SD-Inpaint (Rombach et al., ), and DALLE-2/3 (Ramesh et al., ; Betker et al., ). However, these synthetic data generation pipelines exhibit significant biases, resulting in the following limitations:

Limited Editing Capabilities: The synthetic data is heavily influenced by the underlying generation models. For example, Prompt2Prompt struggles with localized edits, such as adding, removing, or swapping objects, while SD-Inpaint and DALLE-2 are ineffective at global edits, such as style or background changes. As a result, models trained on such data inherit these limitations.
Poor Data Quality Control: Most approaches use simplified filtering mechanisms like CLIP-score (Radford et al., ) or DINO-score (Caron et al., ) to automatically select training samples. However, recent studies (Ku et al., ) show that these metrics exhibit poor correlation with actual data quality, leading to suboptimal training data that negatively impacts the model.
Lack of Support for Varying Resolutions: All current models are trained on square image editing pairs, making their generalization to non-square images poor.

In our preliminary studies, we curate a few prompts for seven different desired tasks to observe their success rate across the board. We show our findings in Table 1. This show that these models are truly biased in their skills caused by the underlying synthesis pipeline.

In this paper, we introduce Omni-Edit, a novel model designed to address these challenges through four key innovations:

1. Specialist-to-Generalist Supervision: We propose learning a generalist editing model, Omni-Edit, by leveraging supervision from multiple specialist models. Unlike previous approaches that rely on a single expert, we conduct an extensive survey and construct (or train) seven experts, each specializing in a different editing task. These specialists provide supervisory signals to Omni-Edit.
2. Importance Sampling: To ensure high-quality training data, we employ large multimodal models to assign quality scores to synthesized samples. Given the computational cost of GPT-4o (Achiam et al., ), we first distill its scoring ability into InternVL2 (Chen et al., ) through medium-sized samples. Then we use the InternVL2
model for large-scale scoring.
3. EditNet Architecture: We introduce EditNet, a novel diffusion-transformer-based architecture (Peebles & Xie, ) that facilitates interaction between the control branch and the original branch via intermediate representations. This architecture enhances Omni-Edit ’s ability to comprehend diverse editing tasks.
4. Support for Any Aspect Ratio: During training, we incorporate a mix of images with varying aspect ratios as well as high resolution, ensuring that Omni-Edit can handle images of any aspect ratio with any degradation in the output quality.

We curate an image editing benchmark Omni-Edit-Bench, which contains diverse images of different resolutions and diverse prompts that cover all the listed editing skills. We perform comprehensive automatic and human evaluation to show the significant boost of Omni-Edit over the existing baseline models like CosXL-Edit  (Boesel & Rombach, ), UltraEdit  (Zhao et al., ), etc. On automatic metrics like VIEScore (Ku et al., ), we can outperform all the existing approaches by a reasonable margin in terms of both perceptual quality and semantic consistency. We also performed a human evaluation and observed an overall improvement 20% over the best baseline editing model CosXL-Edit (Boesel & Rombach, ).

## 2 Preliminaries

### 2.1 Text-to-Image Diffusion Models

Diffusion models (Song et al., ; Ho et al., ) are a class of latent variable models parameterized by $\theta$, defined as $p_{\theta}(\mathbf{x}_{0}):=\int p_{\theta}(\mathbf{x}_{0:T})\,d\mathbf{x}_{1:T}$, where $\mathbf{x}_{0}\sim q(\mathbf{x}_{0})$ represents the original data, and $\mathbf{x}_{1},\dots,\mathbf{x}_{T}$ are progressively noisier latent representations of the input image $\mathbf{x}_{0}$. Throughout the process, the dimensionality of $\mathbf{x}_{0}$ and the latent variables $\mathbf{x}_{1:T}$ remains consistent, with $\mathbf{x}_{0:T}\in\mathbb{R}^{d}$, where $d$ corresponds to the product of the image’s height, width, and channels.
The forward (diffusion) process, denoted as $q(\mathbf{x}_{1:T}|\mathbf{x}_{0})$, is a predefined Markov chain that incrementally adds Gaussian noise to the data according to a pre-defined schedule $\{\beta_{t}\}_{t=1}^{T}$. The process of forward diffusion is defined as:

$$ $q(\mathbf{x}_{1:T}|\mathbf{x}_{0})=\prod_{t=1}^{T}q(\mathbf{x}_{t}|\mathbf{x}_ {t-1}),\quad q(\mathbf{x}_{t}|\mathbf{x}_{t-1}):=\mathcal{N}(\mathbf{x}_{t}; \sqrt{1-\beta_{t}}\,\mathbf{x}_{t-1},\beta_{t}\mathbf{I}),$ (1) $$

where $\mathcal{N}$ denotes a Gaussian distribution, and $\beta_{t}$ controls the amount of noise added at each step.
The objective of diffusion models is to reverse this diffusion process by learning the distribution $p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$, which enables the reconstruction of the original data $\mathbf{x}_{0}$ from a noisy latent $\mathbf{x}_{t}$. This reduces to a denoising problem where the model $\epsilon_{\theta}$ is trained to denoise the sample $\mathbf{x}_{t}\sim q(\mathbf{x}_{t}|\mathbf{x}_{0})$ back into $\mathbf{x}_{0}$. The maximum log-likelihood training objective breaks down to minimizing the weighted mean squared error between the model’s prediction $\hat{\mathbf{x}}_{\theta}(\mathbf{x}_{t},c)$ and the true data $\mathbf{x}_{0}$:

$$ $\arg\max_{\theta}\log p_{\theta}(\mathbf{x_{0}}|c)=\arg\min_{\theta}\mathbb{E} _{(\mathbf{x}_{0},c)\sim\mathcal{D}}\left[\mathbb{E}_{\epsilon,t}\left[w_{t} \cdot\|\hat{\mathbf{x}}_{\theta}(\mathbf{x}_{t},c)-\mathbf{x}_{0}\|^{2}_{2} \right]\right],$ (2) $$

where $(\mathbf{x}_{0},c)$ pairs come from the dataset $\mathcal{D}$, with $c$ representing the text prompt. The term $w_{t}$ is a weighting factor applied to the loss at each timestep $t$. For simplicity, prior papers (Song et al., ; Ho et al., ; Karras et al., ) will set $w_{t}$ to be 1.

### 2.2 Instruction-Based Image Editing in Supervised Learning

Instruction-based image editing can be formulated as a supervised learning problem.
Existing methods (Brooks et al., ; Zhang et al., ) often adopt a paired training dataset of text editing instructions and images before and after the edit. An image editing diffusion model is then trained on this dataset.
The latent diffusion objective is defined as:

$$ $\arg\max_{\theta}\log p_{\theta}(\mathbf{x^{\prime}_{0}}|\mathbf{x_{0}},c)= \arg\min_{\theta}\mathbb{E}_{(\mathbf{x}_{0}^{\prime},\mathbf{x}_{0},c)\sim \mathcal{D}}\left[\mathbb{E}_{\epsilon,t}\|\hat{\mathbf{x}}_{\theta}(\mathbf{x }_{t},c)-\mathbf{x}_{0}^{\prime}\|^{2}_{2}\right],$ (3) $$

where $(\mathbf{x}_{0}^{\prime},\mathbf{x}_{0},c)$ triples are sampled from the dataset $\mathcal{D}$ with $\mathbf{x}_{0}$ denoting the source image, $c$ denoting the editing instruction and $\mathbf{x}^{\prime}_{0}$ denoting the target image.

## 3 Learning with Specialist Supervision

In this section, we introduce the entire specialist-to-generalist learning framework to build Omni-Edit.
We describe the overall learning objective in subsection 3.1. We then describe how we learn the specialists in subsection 3.2 and the importance weighting function in  subsection 3.3. In Figure 2, we show the overview of the Omni-Edit training pipeline.

Figure: Figure 2: Overview of the Omni-Edit training pipeline.
Refer to caption: x2.png

### 3.1 Learning Objective

We assume there is a groundtruth editing model $p(\mathbf{x}^{\prime}|\mathbf{x},c)$, which can perform any type of editing tasks perfectly according to the instruction $c$. Our goal is to minimize the divergence between $p_{\theta}(\mathbf{x}^{\prime}|\mathbf{x},c)$ with $p(\mathbf{x}^{\prime}|\mathbf{x},c)$ by updating the parameters $\theta$:

$$ $L(\theta):=\sum_{\mathbf{x},c}D_{KL}(p(\mathbf{x}^{\prime}|\mathbf{x},c)\|p_{ \theta}(\mathbf{x}^{\prime}|\mathbf{x},c))=-\sum_{\mathbf{x},c}\sum_{\mathbf{x }^{\prime}}p(\mathbf{x}^{\prime}|\mathbf{x},c)\log p_{\theta}(\mathbf{x}^{ \prime}|\mathbf{x},c)+C$ (4) $$

where $C$ is a constant, which we leave out in the following derivation. However, since we don’t have access to $p(\mathbf{x}^{\prime}|\mathbf{x},c)$, we adopt importance sampling for approximation:

$$ $\displaystyle\begin{split}L(\theta)&=-\sum_{\mathbf{x},c}\sum_{\mathbf{x^{ \prime}}}q(\mathbf{x}^{\prime}|\mathbf{x},c)\frac{p(\mathbf{x}^{\prime}| \mathbf{x},c)}{q(\mathbf{x}^{\prime}|\mathbf{x},c)}\log p_{\theta}(\mathbf{x}^ {\prime}|\mathbf{x},c)\\ &\approx-\mathbb{E}_{(\mathbf{x},c)\sim D}\left[\mathbb{E}_{\mathbf{x^{\prime} }\sim q(\mathbf{x}^{\prime}|\mathbf{x},c)}\left[\lambda(\mathbf{x}^{\prime}, \mathbf{x},c)\log p_{\theta}(\mathbf{x}^{\prime}|\mathbf{x},c)\right]\right]\\ &\approx-\mathbb{E}_{(\mathbf{x},c)\sim D}\left[\mathbb{E}_{\mathbf{x^{\prime} }\sim q_{s}(\mathbf{x}^{\prime}|\mathbf{x},c)}\left[\lambda(\mathbf{x}^{\prime },\mathbf{x},c)\log p_{\theta}(\mathbf{x}^{\prime}|\mathbf{x},c)\right]\right] \end{split}$ (5) $$

where $q(\mathbf{x}^{\prime}|\mathbf{x},c)$ is the proposal distribution and $\lambda(\cdot)$ is the importance function. To better approximate the groundtruth distribution $p(\mathbf{x}^{\prime}|\mathbf{x},c)$, we propose to use an ensemble model $q(\mathbf{x}^{\prime}|\mathbf{x},c)$. In essence, $q(\mathbf{x}^{\prime}|\mathbf{x},c):=q_{s}(\mathbf{x}^{\prime}|\mathbf{x},c)$, where $q_{s}$ is a specialist distribution decided by the type of the instruction $c$ (e.g. object removal, object addition, stylization, etc). Combing with Equation 3, our objective can be rewritten as:

$$ $\arg\min_{\theta}L(\theta)=\arg\min_{\theta}\mathbb{E}_{(\mathbf{x},c)\sim D} \mathbb{E}_{\mathbf{x^{\prime}}\sim q_{s}(\mathbf{x}^{\prime}|\mathbf{x},c)} \lambda(\mathbf{x}^{\prime},\mathbf{x},c)\left[\mathbb{E}_{\epsilon,t}\|\hat{ \mathbf{x}}_{\theta}(\mathbf{x}_{t},\mathbf{x},c)-\mathbf{x}^{\prime}\|^{2}_{2 }\right].$ (6) $$

The whole process can be described as: we first sample a pair from dataset $D$, and then choose the corresponding specialist $q_{s}$ to sample demonstrations $\mathbf{x^{\prime}}$ for the our editing model $\hat{\mathbf{x}}_{\theta}(\mathbf{x}_{t},\mathbf{x},c)$ to approximate with an importance weight of $\lambda(\mathbf{x}^{\prime},\mathbf{x},c)$. We formally provide the algorithm in 1.
In our specialist-to-generalist framework, we need to have a series of specialist models $\{q_{s}(\cdot)\}_{s}$ and an importance function $\lambda(\cdot)$. We describe them separately in subsection 3.2 and subsection 3.3.

### 3.2 Constructing Specialist Models

**Table 2: Task Definitions and Examples**
| Editing Tasks | Definition | Instruction $c$ Example |
| --- | --- | --- |
| Object Swap | $c$ describes an object to replace by specifying both the object to remove and the new object to add, along with their properties such as appearance and location. | Replace the black cat with a brown dog in the image. |
| Object Removal | $c$ describes which object to remove by specifying the object’s properties such as appearance, location, and size. | Remove the black cat from the image. |
| Object Addition | $c$ describes a new object to add by specifying the object’s properties such as appearance and location. | Add a red car to the left side of the image. |
| Attribute Modification | $c$ describes how to modify the properties of an object, such as changing its color and facial expression. | Change the blue car to a red car. |
| Background Swap | $c$ describes how to replace the background of the image, specifying what the new background should be. | Replace the background with a space-ship interior. |
| Environment Change | $c$ describes a change to the overall environment, such as the weather, lighting, or season, without altering specific objects. | Change the scene from daytime to nighttime. |
| Style Transfer | $c$ describes how to apply a specific artistic style or visual effect to the image, altering its overall appearance while keeping the content the same. | Apply a watercolor painting style to the image. |

We group the image editing task into 7 categories as summarized in Table 2. For each category, we train or build a task specialist $p_{s}(\mathbf{x^{\prime}}\mid\mathbf{x},c)$ to generate millions of examples.
Table 2 provides detailed information on task groups and example editing instructions $c$. In this section, we briefly summarize each specialist, with details available in Appendix A.1.

Object Replacement.
We trained an image-inpainting model to serve as the specialist $q_{\text{obj_replace}}$ for object replacement. Given a image $\mathbf{x}$ and an object caption $c_{\text{obj}}$ and a object mask $M_{\text{obj}}$. The $q_{\text{obj_replace}}$ can fill the content indicated by the mask with an object in $c_{\text{obj}}$. We then generate an object replacement sample by masking out an existing object and fill the image with a new object.
Object Removal.
We trained an image inpainting model to serve as the specialist $q_{\text{obj_removal}}$ for object removal. We use a similar procedure as in the object replacement but use a predicted background content caption to inpaint the masked image.
Object Addition.
We treat object addition as the inverse task of object removal.
Specifically, for each pair of editing examples generated by the object removal specialist,
we reverse the roles of the source and target images to create a new pair.
Attribute Modification.
We adopt the Prompt-to-Prompt (P2P) (Hertz et al., ) pipeline to generate examples. To enable precise modification, we adapt the method from Sheynin et al. () where we provide a mask $M_{\text{obj}}$ for the object and force P2P to only make edits inside the mask.
Background Swap.
We trained an image inpainting model to serve as the specialist $q_{\text{obj_background_swap}}$. We use a similar procedure as in the object replacement but use an inverse mask of the object to indicate the background and guide the inpainting.
Environment Modification.
For environment modification, we use P2P pipeline to generate original and edited image. To ensure structural consistency between two images, we apply a mask of the foreground to maintain details in the foreground while changing the background.
Style Transfer.
We use CosXL-Edit (Boesel & Rombach, ) as the specialist model as its training data contains a large number of style transfering examples. We provide CosXL-Edit with $(\mathbf{x},c)$, and let it generates the edited image $\mathbf{x}^{\prime}$.

Figure: Figure 3: InternVL2 as a scoring function before (top right) and after (bottom right) fine-tuning on GPT-4o’s response. On the top right, the original InternVL2 fails to identify the unusual distortions in the edited image it also does not spot the error when the edited image fails to meet the specified editing instructions. On the bottom right, finetuned-InternVL2 successfully detects such failures and serve as a reliable scoring function.
Refer to caption: x3.png

### 3.3 Importance weighting

The importance weighting function $\lambda$ takes as input a tuple of source image, edited image, and editing prompt. Its purpose is to assign higher weights to data points that are more likely to be sampled from the ground truth distribution, and lower weights to the unlikely ones. This is essentially a quality measure to up-weight high-quality samples. Unlike previous work, we do not use CLIP score because prior work (Jiang et al., ) has shown its low correlation with human judges. Instead, we propose to use large multimodal models (LMMs) to approximate the weighting function, as they demonstrate strong image understanding. Following VIEScore (Ku et al., ), we designed a prompting template for GPT-4o (Achiam et al., ) to evaluate the image editing pairs and output a score on a scale from 0 to 10. We then filter out data with a score greater than or equal to 9, so the LMM essentially serves as a binary weighting function:

$$ $\lambda(\mathbf{x^{\prime}},\mathbf{x},c)=\begin{cases}1,&\text{if }\text{LMM} (\text{prompt},\mathbf{x^{\prime}},\mathbf{x},c)\geq 9\\ 0,&\text{otherwise}\end{cases}$ $$

Details of the prompt template are provided in the Appendix.

While the GPT-4o is an effective choice for this task, scoring large-scale datasets with millions of examples is extremely costly and time-consuming. Therefore, we employ knowledge distillation from GPT-4o to a smaller 8B model, InternVL2 (Chen et al., ). For each task, we sample 50K data points and instruct GPT-4o to output both a score and a score rationale. We fine-tune InternVL2 on these GPT-4o-generated examples. After fine-tuning, InternVL2 performs as an ideal scoring function due to its smaller size and efficiency. A comparison of the model’s performance before and after fine-tuning is presented in the Appendix. Finally, we apply the fine-tuned InternVL2 model to filter data across a dataset with millions of samples. Only examples with a score of $\geq 9$ are retained, resulting in a curated training dataset of 775K examples. We visualize InternVL2’s response as a scoring function before and after fine-tuning in Figure 3. We observe that fine-tuning InternVL2 on GPT-4o’s response effectively turns InternVL2 into a realiable scoring function and it can identify unusual distortions or unsuccessful edit that does not follow the editing instruction. Additional dataset statistics are detailed in the Appendix.

## 4 EditNeT

We found that directly fine-tuning a pre-trained high-quality diffusion model like SD3 using channel-wise image concatenation methods (Brooks et al., ) compromises the model’s original representational capabilities (see Figure 7 and Section 5.2 for details comparison).

Figure: Figure 4: Architecture Comparison between EditNet(ours), ControlNet and InstructPix2Pix(Channel-wise concatenation) for DiT models. Unlike ControlNet’s parallel execution, EditNet allows adaptive adjustment of control signals by intermediate representations interaction between the control branch and the original branch. EditNet also updates the text representation, enabling better task understanding.
Refer to caption: x4.png

To enable a diffusion transformer to perform instruction-based image editing while preserving its original capabilities, we introduce EditNet to build Omni-Edit. EditNet can effectively transform common DIT models like SD3 into editing models. As illustrated in Figure 4, we replicate each layer of the original DIT block as a control branch. The control branch DIT blocks allow interaction between the original DIT tokens, conditional image tokens, and the editing prompts. The output of the control branch tokens is then added to the original DIT tokens and editing prompts. Since the original DIT blocks are trained for generation tasks and are not aware of the editing instructions specifying which contents to modify and how to modify them, this design allows the control branch DIT to adjust the representations of the original DIT tokens and editing prompts according to the editing instruction, while still leveraging the strong generation ability of the original DIT.
Compared to ControlNet (Zhang et al., ), our approach offers two key advantages that make it more suitable for image editing tasks: First, ControlNet does not update text representations, making it challenging to execute editing tasks based on instruction, particularly object removal, as it fails to understand the “removal” intent (see Figure 6). Secondly, ControlNet’s control branch operates in parallel without access to the original branch’s intermediate representations. This fixed precomputation of control signals restricts the overall representation power of the network. We provide an ablation study on the Omni-Edit architecture design in Section 5.2.

## 5 Experiments

Figure: Figure 5: Qualitative comparison between baselines and Omni-Edit on a subset of the test set.
Refer to caption: x5.png

In this section, we first provide statistics of the Omni-Edit training set and test set in Table 5. Then we introduce the human evaluation protocol in Section 5, and comparative baseline system in 5. We present the main results in Section 5.1, highlighting the advantages of Omni-Edit in tacking multi-aspect ratio, multi-resolution, and multi-task image editing.
In Section 5.2, we study the advantages of importance sampling for synthetic data.
In Section 5.2, we perform an analysis to study the design of Omni-Edit.

Omni-Edit Training Dataset.
We constructed the training dataset $\mathcal{D}$ by sampling high-resolution images with a minimum resolution of 1 megapixel from the LAION-5B (Schuhmann et al., ) and OpenImageV6 (Kuznetsova et al., ) databases. The images cover a range of aspect ratios including 1:1, 2:3, 3:2, 3:4, 4:3, 9:16, and 16:9.
For the task of object swap, we employed a specialist model to generate 1.5 million entries. We then applied InternVL2 for importance weighting, retaining samples with scores of 9 or higher, resulting in a dataset of 150K entries for this task. Similarly, we generate 250k-1M samples for each task, then keep the top 10% as the final dataset.
The final training dataset comprises 775K entries, with detailed information provided in Appendix 6.
Omni-Edit-Bench.
To create a high-resolution, multi-aspect ratio, multi-task benchmark for instruction-based image editing, we manually collected 62 images from pexels () and LAION-5B (Schuhmann et al., ). These images cover a variety of aspect ratios, including 1:1, 2:3, 3:2, 3:4, 4:3, 9:16, and 16:9. We ensured that the images feature a diverse range of scenes and object counts, from single to complex compositions. Additionally, we selected images with a relatively high aesthetic score to better align with the practical use cases of image editing. For each image, we tasked the model with performing 7 tasks as outlined in Table 2. This results in a total of 434 edits.
Omni-Edit implementation details.
The Omni-Edit model is built upon Stable diffusion 3 Medium(Esser et al., ) with EditNet architecture. The stable diffusion 3 has 24 DiT layers. Each layer has a corresponding EditNet layer. We train Omni-Edit on the 775K Omni-Edit training dataset for 2 epochs on a single node with 8 H100 GPUs.
Baseline models.
We compare Omni-Edit with $8$ other text-guided image editing baselines: MagicBrush (Zhang et al., ), InstructPix2Pix (Brooks et al., ), UltraEdit(SD3) (Zhao et al., ), DiffEdit (Couairon et al., ), SDEdit (Meng et al., ), CosXL-Edit (Boesel & Rombach, ), HIVE (Zhang et al., ) and HQ-Edit (Hui et al., ).
Evaluations Protocol
We conduct both human evaluation and automatic evaluation. For the human evaluation, we follow the procedure from Ku et al. () to rate in two criteria: Semantic Consistency ($SC$) and Perceptual Quality ($PQ$). Both scores are in $\{0,0.5,1\}$. For $SC$, the human subject is asked to rate the consistency between 1) the edited image and the editing instruction (whether the editing instruction is reflected on the edited image) and between 2) the source image and the edited image (whether the model makes the edit that is beyond the editing instruction). For $PQ$, the subject is asked to rate on the quality of edited image). We then calculate a overall score $O=\sqrt{SC\times PQ}$ that measures the overall quality of the edit. We also calculate the accuracy of the edit, which is defined by the percentage of $SC=1$ among all examples.
We recruit four human raters and require them to evaluate all the editing examples.
For LMMs’ evaluation, we follow the procedure from Ku et al. () where models (in particular, we chose GPT4o and Gemini) are also asked to give $SC$ and $PQ$ scores but on a scale of 0-10. We then normalize the scale to 0-1.

### 5.1 Main Results

**Table 3: Main evaluation results on Omni-Edit-Bench. In each column, the highest score is bolded, and the second-highest is underlined.**
| Models | VIEScore (GPT4o) | VIEScore (Gemini) | Human Evaluation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | $PQ_{avg}\uparrow$ | $SC_{avg}\uparrow$ | $O_{avg}\uparrow$ | $PQ_{avg}\uparrow$ | $SC_{avg}\uparrow$ | $O_{avg}\uparrow$ | $PQ_{avg}\uparrow$ | $SC_{avg}\uparrow$ | $O_{avg}\uparrow$ | $Acc_{avg}\uparrow$ |
| Inversion-based Methods |  |  |  |  |  |  |  |  |  |  |
| DiffEdit | 5.88 | 2.73 | 2.79 | 6.09 | 2.01 | 2.39 | - | - | - | - |
| SDEdit | 6.71 | 2.18 | 2.78 | 6.31 | 2.06 | 2.48 | - | - | - | - |
| End-to-End Methods |  |  |  |  |  |  |  |  |  |  |
| InstructPix2Pix | 7.05 | 3.04 | 3.45 | 6.46 | 1.88 | 2.31 | - | - | - | - |
| MagicBrush | 6.11 | 3.53 | 3.60 | 6.36 | 2.27 | 2.61 | - | - | - | - |
| UltraEdit(SD-3) | 6.44 | 4.66 | 4.86 | 6.49 | 4.33 | 4.45 | 0.72 | 0.52 | 0.57 | 0.20 |
| HQ-Edit | 5.42 | 2.15 | 2.25 | 6.18 | 1.71 | 1.96 | 0.80 | 0.27 | 0.29 | 0.10 |
| CosXL-Edit | 8.34 | 5.81 | 6.00 | 7.01 | 4.90 | 4.81 | 0.82 | 0.56 | 0.59 | 0.35 |
| HIVE | 5.35 | 3.65 | 3.57 | 5.84 | 2.84 | 3.05 | - | - | - | - |
| Omni-Edit | 8.38 | 6.66 | 6.98 | 7.06 | 5.82 | 5.78 | 0.83 | 0.71 | 0.69 | 0.55 |
| $\Delta$ - Best baseline | +0.04 | +0.85 | +0.98 | +0.05 | +0.92 | +0.97 | +0.01 | +0.15 | +0.10 | +0.20 |

We provide a qualitative comparison with baseline models in Figure 5. We show the top 4 baselines with Omni-Edit on a subset of the Omni-Edit-Bench. We provide more results in Figure 10 and Figure 11. Our main results are detailed in Table 3, where we provide the VIEScore and conduct human evaluation on the Top2 baselines and Omni-Edit.
In Figure 1, Omni-Edit demonstrates its capability to handle diverse editing tasks across various aspect ratios and resolutions. The results are notably sharp and clear, especially in the addition/swap task, where new content is seamlessly integrated. This underscores the effectiveness of the Edit-Net design in preserving the original image generation capabilities of the base text-image generative model. Similarly, in Figure 5, Omni-Edit uniquely adds a clean and distinct NASA logo onto a T-shirt. Table 3 corroborates this with Omni-Edit achieving the highest Perceptual Quality (PQ) score among the models evaluated.

We highlight the efficacy of our proposed specialist-to-generalist learning framework. Unlike baseline models that utilize a single method for generating synthetic data—often the prompt-to-prompt method—This method typically alters the entire image, obscuring task-specific data. In contrast, Omni-Edit leverages task-specific data curated by experts, resulting in a clearer task distribution and improved adherence to editing instructions. Both the VIEScore and human evaluations in Table 3 demonstrate that our method significantly outperforms the best baseline in following editing instructions accurately and minimizing over-editing. For instance, baseline models frequently misunderstand the task intent as illustrated in Figure 5, where the CosXL-Edit model fails to recognize the removal task and incorrectly interprets a bird addition as a swap between a panda and a bird.

Lastly, baseline models often produce blurry images on the Omni-Edit-Bench, as they are trained at resolutions limited to 512x512 or even 256x256, and they perform poorly on non-square aspect ratios. For example, with a 3:4 aspect ratio, the baselines struggle to perform editing. Omni-Edit, trained on data with multiple aspect ratios, maintains robust editing capabilities across the diverse aspect ratios encountered on the Omni-Bench, as evidenced in Figure 5.

### 5.2 Ablation Study

In this section, We provide an ablation study w.r.t importance weighting and EditNet.

Ablation study on the importance sampling.
We study a baseline that utilizes the same architecture as Omni-Edit, but instead of applying importance scoring and filtering, we sample 775K examples directly from the 5M pre-filtering dataset as specified in Table 6 and compare it with Omni-Edit. As shown in Table 4, we observe a significant decrease in VIEScores for both PQ and SC metrics.
Ablation Study on Omni-Edit Architecture Design.
We conducted an analysis of Omni-Edit ’s architectural design in comparison to two baseline models: Omni-Edit-ControlNet and Omni-Edit-ControlNet-TextControl and show the result in Table 5. Omni-Edit-ControlNet represents the SD3-ControlNet architecture trained on the Omni-Edit dataset, where the source image serves as the conditioning image for the control branch.
Omni-Edit-ControlNet-TextControl is a variant of Omni-Edit-ControlNet with an added modification: at each layer, we incorporate the text-token output from the control branch into the text-token in the main image generation branch. So this baseline can update the text representation in the main branch but doesn’t have the intermediate representation interaction design in EditNet.

**Table 4: Ablation on importance sampling.**
| Models | VIEScore (GPT4o) | VIEScore (Gemini) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | $PQ_{avg}\uparrow$ | $SC_{avg}\uparrow$ | $O_{avg}\uparrow$ | $PQ_{avg}\uparrow$ | $SC_{avg}\uparrow$ | $O_{avg}\uparrow$ |
| Omni-Edit | 8.38 | 6.66 | 6.98 | 7.06 | 5.82 | 5.78 |
| Omni-Edit w/o importance sampling | 6.20 | 2.95 | 3.30 | 6.40 | 1.80 | 2.25 |

**Table 5: Ablation on Omni-Edit architecture design.**
| Models | VIEScore (GPT4o) | VIEScore (Gemini) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | $PQ_{avg}\uparrow$ | $SC_{avg}\uparrow$ | $O_{avg}\uparrow$ | $PQ_{avg}\uparrow$ | $SC_{avg}\uparrow$ | $O_{avg}\uparrow$ |
| Omni-Edit | 8.38 | 6.66 | 6.98 | 7.06 | 5.82 | 5.78 |
| Omni-Edit- ControlNet - TextControl | 6.45 | 4.70 | 4.89 | 6.50 | 4.35 | 4.48 |
| Omni-Edit- ControlNet | 6.35 | 4.60 | 4.75 | 6.40 | 4.25 | 4.35 |

Our analysis, as shown in Figure 6, reveals that Omni-Edit-ControlNet struggled to accurately capture task intent. This is primarily because the ControlNet branch does not update the text representation. For instance, in object removal tasks, prompts like “Remove ObjA” are common, yet the original DIT block remains unchanged, causing it to mistakenly generate an image of “ObjA.” On the other hand, although Omni-Edit-ControlNet-TextControl successfully updates the text representation, it still encounters difficulties in content removal. The substantial VIEScores gap between Omni-Edit-Controlnet-TextControl and Omni-Edit in Table 5 underscores the importance of the intermediate representation interaction design in EditNet. We also compared Omni-Edit with the channel-wise token concatenation method used in InstructPix2Pix (see Figure 4). Channel-wise Token concatenation requires fine-tuning the entire network, which can distort the network’s original representations. As illustrated in Figure 7, after fine-tuning an SD3 channel-wise concatenation model on Omni-Edit training set, the representation of Batman is altered. In contrast, EditNet preserves the original representation of Batman while still learning the object swap task.

Figure: Figure 6: Omni-Edit-ControlNet fails to grasp the task intent, while Omni-Edit-ControlNet-TextControl—a variant with a text-updating branch—recognizes the intent but struggles with content removal. In contrast, Omni-Edit accurately removes content.
Refer to caption: x6.png

Figure: Figure 7: (a) shows the source image. (d) presents images generated by SD3 in response to prompts for “an upper body picture of Batman” and “a shiny red vintage Chevrolet Bel Air car.” We use the prompts “Replace the man with Batman” and “Add a shiny red vintage Chevrolet Bel Air car to the right” to Omni-Edit and Omni-Edit-Channel-Wise-Concatenation, which was trained on Omni-Edit training data. From (b) and (c), one can observe that Omni-Edit preserves the generation capabilities of SD3, while Omni-Edit-Channel-Wise-Concatenation exhibits a notable degradation in generation capability.
Refer to caption: x7.png

## 6 Related Work

Image Editing via Generation.
Editing real images according to specific user requirements has been a longstanding research challenge (Crowson et al., ; Liu et al., ; Zhang et al., ; Shi et al., ; Ling et al., ). Since the introduction of large-scale diffusion models, such as Stable Diffusion (Rombach et al., ; Podell et al., ), significant progress has been made in tackling image editing tasks. SDEdit (Meng et al., ) introduced an approach that adds noise to the input image at an intermediate diffusion step, followed by denoising guided by the target text description to generate the edited image. Subsequent methods, such as Prompt-to-Prompt (Hertz et al., ) and Null-Text Inversion (Mokady et al., ), have focused on manipulating attention maps during intermediate diffusion steps for image editing. Other techniques like Blended Diffusion (Avrahami et al., ) and DiffEdit (Couairon et al., ) utilize masks to blend regions of the original image into the edited output. More recently, the field has seen a shift towards supervised methods, such as InstructP2P (Brooks et al., ), HIVE (Zhang et al., ), and MagicBrush (Zhang et al., ), which incorporate user-written instructions in an end-to-end framework. Our work follows this direction to develop end-to-end editing models without inversion.
Image Editing Datasets.
Due to the difficulty of collecting expert-annotated editing pairs, existing approaches rely heavily on synthetic data to train editing models. InstructP2P (Brooks et al., ) was the first to curate large-scale editing datasets using prompt-to-prompt filtering with CLIP scores. MagicBrush (Zhang et al., ) subsequently improved data quality by incorporating a human-in-the-loop annotation pipeline based on DALLE-2. However, DALLE-2, primarily an inpainting-based method, struggles with global editing tasks such as style transfer and attribute modification. More recently, HQ-Edit (Hui et al., ) utilized DALLE-3 to curate editing pairs, although the source and target images lack pixel-to-pixel alignment, which is critical for preserving fine-grained details. Emu Edit (Sheynin et al., ) scaled up the training dataset to 10 million proprietary pairs, resulting in strong performance, but the lack of public access to their model checkpoints or API makes direct comparison difficult. UltraEdit (Zhao et al., ) proposed another inpainting-based approach, avoiding the use of DALLE-2 or DALLE-3 for data curation. However, like MagicBrush, it still faces limitations in handling complex global edits. Our work is the first to leverage multiple specialists to significantly expand the range of editing capabilities. Additionally, we are the first to use more reliable large multimodal models, for quality control in the editing process.

## 7 Discussion

In this paper, we identify the imbalanced skills in the existing end-to-end image editing methods and propose a new framework to build more omnipotent image editing models. We surveyed the field and chose several approaches as our specialists to synthesize candidate pairs and adopt weighted loss to supervise the single generalist model. Our approach has shown significant quality boost across the broad editing skills. Throughout the experiments, we found that the output quality is highly influenced by the underlying base model. Due to the weakness of SD3, our approach is still not achieving its highest potential. In the future, we plan to use Flux or other more capable base models to see how much further we can reach with the current framework.

## Appendix A Appendix

**Table 6: Omni-Edit training dataset statistics reflecting the number of samples before and after importance scoring and filtering with o-score $\geq$ 9.**
| Task | Pre-Filtering Number | After-Filtering Number |
| --- | --- | --- |
| Object Swap | 1,500,000 | 150,000 |
| Object Removal | 1,000,000 | 100,000 |
| Object Addition | 1,000,000 | 100,000 |
| Background Swap | 500,000 | 50,000 |
| Environment Change | 500,000 | 100,000 |
| Style Transfer | 250,000 | 25,000 |
| Object Property Modification | 450,000 | 250,000 |
| Total | 52,000,000 | 775,000 |

Figure: Algorithm 1 Specialist-to-Generalist Learning Framework

### A.1 Training Data Generation Details

#### A.1.1 Object Replacement

We trained an image-inpainting model to serve as the expert for object replacement. During training, given a source image $\mathbf{x}_{\text{src}}$ and an object caption $C_{\text{obj}}$, we employ GroundingDINO and SAM to generate an object mask $M_{\text{obj}}$. The masked image is then created by removing the object from the source image:

$$ $\mathbf{x}_{\text{masked}}=\mathbf{x}_{\text{src}}\odot(1-M_{\text{obj}}).$ (7) $$

Here, $\odot$ denotes element-wise multiplication, effectively masking out the object in $\mathbf{x}_{\text{src}}$. Both the mask $M_{\text{obj}}$ and the object caption $C_{\text{obj}}$ are provided as inputs to the expert model $q_{\text{obj_replace}}$. The expert $q_{\text{obj_replace}}$ is trained to reconstruct (inpaint) the original source image $\mathbf{x}_{\text{src}}$ from the masked image.

To generate data for object replacement, we sample 200K images from the LAION and OpenImages datasets, ensuring a diverse range of resolutions close to 1 megapixel. For each image, we utilize GPT-4o to propose five object replacement scenarios. Specifically, GPT-4o identifies five interesting source objects $C_{\text{src_obj}}$ within the image and suggests corresponding target objects $C_{\text{trg_obj}}$ for replacement.

For each proposed replacement, we perform the following steps:

- 1.
Mask Generation: Use GroundingDINO and SAM to generate the object mask $M_{\text{src_obj}}$ for the source object $C_{\text{src_obj}}$.
- 2.
Mask Dilation: Apply a dilation operation to $M_{\text{src_obj}}$ to expand the mask boundaries.
- 3.
Image Editing: Apply the expert model to generate the edited image $\mathbf{x}_{\text{edit}}$ by replacing the source object with the target object $C_{\text{trg_obj}}$:
$\mathbf{x}_{\text{edit}}=q_{\text{obj_replace}}\left(\mathbf{x}_{\text{src}}
\odot(1-M_{\text{src_obj}}),\;M_{\text{src_obj}},\;C_{\text{trg_obj}}\right)$
(8)

In this equation:

- •
$\mathbf{x}_{\text{src}}\odot(1-M_{\text{src_obj}})$ represents the source image with the target object masked out.
- •
$M_{\text{src_obj}}$ is the mask of the source object to be replaced.
- •
$C_{\text{trg_obj}}$ is the caption of the target object for replacement.

Then a pair of instruction-based image editing examples will be:
$\langle\mathbf{x}_{\text{src}},\mathbf{x}_{\text{edit}},T\rangle$.
The instruction $T$ initially just be “Replace $C_{\text{src_obj}}$ with $C_{\text{trg_obj}}$ ”.
We then employ large multimodal models (LVLM) to generate more detailed natural language instructions.

#### A.1.2 Object Removal

Similar to object replacement, we trained an image inpainting model to serve as the expert for object removal.
During training, given a source image $\mathbf{x}_{\text{src}}$ and an image caption $C_{\text{src}}$,
we randomly apply strikes to create a mask $M_{\text{src}}$.
The masked image is then created by:

$$ $\mathbf{x}_{\text{masked}}=\mathbf{x}_{\text{src}}\odot(1-M_{\text{src}})$ (9) $$

Both the mask $M_{\text{src}}$ and the image caption $C_{\text{src}}$ are provided as inputs to the expert model $q_{\text{obj_removal}}$.
The expert $q_{\text{obj_removal}}$ is trained to reconstruct (inpaint) the original source image $\mathbf{x}_{\text{src}}$ from the masked image.
To generate data for object removal, we also sample 200K images from the LAION and OpenImages datasets,
ensuring a diverse range of resolutions close to 1 megapixel.
For each image, we utilize GPT-4o to propose five objects to remove and predict the content of the space after removal.
Specifically, GPT-4o identifies five interesting source objects $C_{\text{src_obj}}$ within the image
and predicts the new content after removing the object $C_{\text{trg_background}}$.
For each proposed removal, we perform the following steps:

- 1.
Mask Generation: Use GroundingDINO and SAM to generate the object mask $M_{\text{src_obj}}$ for the source object $C_{\text{src_obj}}$.
- 2.
Image Editing: Apply the expert model to generate the edited image $\mathbf{x}_{\text{edit}}$ by infilling the masked region with the predicted background content $C_{\text{trg_background}}$:
$\mathbf{x}_{\text{edit}}=q_{\text{obj_removal}}\left(\mathbf{x}_{\text{src}}
\odot(1-M_{\text{src_obj}}),\;M_{\text{src_obj}},\;C_{\text{trg_background}
}\right).$
(10)

In this equation:

- •
$\mathbf{x}_{\text{src}}\odot(1-M_{\text{src_obj}})$ represents the source image with the target object masked out.
- •
$M_{\text{src_obj}}$ is the mask of the source object to be removed.
- •
$C_{\text{trg_background}}$ is the predicted content for the background after object removal.

Then a pair of instruction-based image editing example will be:
$\langle\mathbf{x}_{\text{src}},\mathbf{x}_{\text{edit}},T\rangle$.
Initially, the instruction $T$ initially just be “Remove $C_{\text{src_obj}}$ from the image”
We then employ large multimodal models (LVLM) to generate more detailed natural language instructions.

#### A.1.3 Object Addition

We conceptualize the object addition task as the inverse of the object removal process.
Specifically, for each pair of editing examples generated by the object removal expert,
we swap the roles of the source and target images to create a new pair tailored for object addition.
This approach leverages the naturalness and artifact-free quality of the original source images,
ensuring high-quality additions.
Given a pair of editing examples $\langle\mathbf{x}_{\text{src_removal}},\mathbf{x}_{\text{edit_removal}},c_{
\text{removal}}\rangle$ generated for object removal and $C_{\text{src_obj_removal}}$ represents the object to remove. We transform this pair into an object addition example by swapping $\mathbf{x}_{\text{src}}$ and $\mathbf{x}_{\text{edit}}$, and modifying the instruction accordingly. The resulting pair for object addition is $\langle\mathbf{x}_{\text{src}}=\mathbf{x}_{\text{edit_removal}},\mathbf{x}_{
\text{edit}}=\mathbf{x}_{\text{src_removal}},c\rangle$, where $c$ is the new instruction defined as “Add $C_{\text{src_obj_removal}}$ to the image.”

#### A.1.4 Attribute Modification

We adapt the Prompt-to-Prompt (P2P) (Hertz et al., ) pipeline where a text-guided image generation model is provided with a pair of captions $\langle C_{\text{src}},C_{\text{edit}}\rangle$ and injects cross-attention maps from the input image generation to that during edited image generation. For example, a pair could be $\langle\text{``a blue backpack''},\text{``a purple backpack''}\rangle$ with the corresponding editing instruction “make the backpack purple”.

To enable precise attribute modification on the object we want (in our example, the “backpack”), we adapt the method from Sheynin et al. () where we provide an additional mask $M_{obj}$ that masks the object. Specifically, to obtain a pair of captions, we obtain source captions $C_{\text{src}}$ from Zhang () and let GPT4 to identify an object $C_{\text{obj}}$ in the original caption $C_{\text{src}}$, propose an editing instruction that edits an attribution of $C_{\text{obj}}$ and output the edited caption $C_{\text{edit}}$ with object’s attribution reflected.

We first let the image generation model to generate a source image $\mathbf{x}_{\text{src}}$ using $C_{\text{src}}$. We then use GroundingDINO to extract mask $M_{\text{obj}}$ that masks the object from the source image. We then apply P2P generation with caption pair $\langle C_{\text{src}},C_{\text{edit}}\rangle$. During the generation, we use the mask to control precise image editing control. In particular, let $\mathbf{x}_{\text{src},t}$ denote the noisy source image at step $t$ and $\mathbf{x}_{\text{edit},t}$ denote the noisy edited image at step $t$, we apply the mask and force the new noisy edited image at time $t$ be $M_{\text{obj}}\odot\mathbf{x}_{\text{edit},t}+(1-M_{\text{obj}})\odot\mathbf{x
}_{\text{src},t}$. In other words, we keep background the same and only edit the object selected.

#### A.1.5 Environment Modification

For environment modification, we use P2P pipeline to generate original and edited image. To ensure structural consistency between two images, we apply a mask of the foreground to maintain details in the foreground while changing the background. In particular, given a source image caption $C_{\text{src}}$, we use GPT4 to identify the foreground (e.g., an object or a human) and apply GroundingDINO to extract mask $M_{\text{foreground}}$. During the generation, let $\mathbf{x}_{\text{src},t}$ denote the noisy source image at step $t$ and $\mathbf{x}_{\text{edit},t}$ denote the noisy edited image at $t$. We apply the mask so that the new noisy edited image at time $t$ is $M_{\text{foreground}}\odot\mathbf{x}_{\text{src},t}+(1-M_{\text{foreground}})
\odot\mathbf{x}_{\text{edit},t}$. We also set $\tau_{\text{env}}=0.7$ so that this mask operation on noisy image is only applied at the first $\tau_{\text{env}}$ of all timesteps.

#### A.1.6 Background Swap

We trained an image inpainting model to serve as the specialist $q_{\text{obj_background_swap}}$. We use a similar procedure as in the object replacement but use an inverse mask of the object to indicate the background to guide the inpainting.

#### A.1.7 Style Transfer

We use CosXL-Edit (Boesel & Rombach, ) as the expert style transfer model. We provide CosXL-Edit with $\langle\mathbf{x}_{\text{src}},c\rangle$ and let it generates the edited image $\mathbf{x}_{\text{edited}}$.

#### A.1.8 Importance Sampling

We apply the importance sampling as described in Section 3.3. Example prompts that are provided to LMMs are shown in Figure 8 and 9. We compute the Overall score following (Ku et al., ) as the importance weight. After importance sampling, we obtain our training dataset described in Table 6.

Figure: Figure 8: Prompt for evaluating SC score.

Figure: Figure 9: Prompt for evaluating PQ score.

### A.2 Additional Evaluation Result

We present additional evaluation results. In Table 7, we compare Omni-Edit with specialist models of three tasks on Omni-Edit-Bench (other specialist models cannot take in input image). As is shown in the Table, Omni-Edit shows comparable performance as the specialist models on tasks that specialist models specialize.

Figure 10 shows additional comparisons between Omni-Edit other baseline models. We observe that Omni-Edit consistently outperforms other baselines.

**Table 7: Comparison between Omni-Edit and our specialist models.**
|  | VIEScore (GPT4o) | VIEScore (Gemini) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | $PQ_{avg}\uparrow$ | $SC_{avg}\uparrow$ | $O_{avg}\uparrow$ | $PQ_{avg}\uparrow$ | $SC_{avg}\uparrow$ | $O_{avg}\uparrow$ |
| Obj-Remove-Specialist | 9.10 | 7.76 | 7.82 | 7.46 | 5.39 | 4.84 |
| Omni-Edit | 8.45 | 7.16 | 7.23 | 7.37 | 5.45 | 5.09 |
| Obj-Replacement-Specialist | 8.48 | 6.92 | 7.02 | 7.06 | 5.68 | 5.36 |
| Omni-Edit | 8.95 | 7.74 | 8.14 | 7.00 | 7.77 | 7.09 |
| Style-Transfer-Specialist | 8.08 | 7.47 | 7.37 | 7.97 | 6.61 | 6.76 |
| Omni-Edit | 7.98 | 5.77 | 6.16 | 8.24 | 5.24 | 6.08 |

Figure: Figure 10: Additional qualitative comparisons between Omni-Edit and the baseline methods.
Refer to caption: x8.png

Figure: Figure 11: Additional qualitative comparisons between Omni-Edit and the baseline methods.
Refer to caption: x9.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Since the dataset is divided into seven types of tasks, would it be possible to categorize the test set similarly? This would allow for a more detailed evaluation of the model's performance across different editing tasks and facilitate an analysis of performance differences and underlying reasons.

2. The relatively small size of the Omni-Edit-Bench dataset may limit the reliability and generalizability of model evaluation. It is recommended to assess performance on additional benchmark datasets. For example, are there results available on the Emu Edit test set?
soundness: 3
strengths: 1. The authors collected a high-resolution dataset covering seven types of tasks, annotated through multiple expert models, and used InternVL2 to score the dataset. 

2. And they introduced EditNet, which enhances image noise control capabilities compared to ControlNet.

3. In the Omni-Edit-Bench benchmark, OMNI-EDIT outperformed existing baseline models across multiple evaluation metrics.
summary: OMNI-EDIT is an image editing model designed to address the skill imbalance issues present in existing instruction-based image editing methods. It learns by combining supervision from seven different expert models and utilizes InternVL2 for data scoring. Additionally, this approach introduces a diffusion-Transformer architecture named EditNet.
weaknesses: 1. Lack of testing of GPT4-o generated CONFIDENCE SCORE, LLM may be subjective and biased

2. Limitations of the benchmarking dataset: The Omni-Edit-Bench dataset used in the paper is small (only 62 images), which may limit the reliability and generalization of the model evaluation.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. How does this paper differentiate itself in terms of contributions from the CVPR 2024 paper [1], given their similar focus on generalist instruction-guided modeling?

2. It would be better to compare the proposed method with the generalist modeling approach in [1].

3. Can the authors provide more detail on how the support for multi-size and multi-resolution inputs was implemented? What specific innovations, if any, were introduced to achieve this?

4. How does the training data size affect the performance of OMNI-EDIT? 

5. Could the authors elaborate on the training process and datasets used for the specialist models, such as the image-inpainting model?
soundness: 3
strengths: --The paper introduces OMNI-EDIT to address limited editing capabilities, poor data quality control, and lack of support for varying resolutions, which are critical point in image editing method.

--The use of importance sampling with high-quality data scoring via distilled multimodal models (such as InternVL2) demonstrates a meticulous approach to data curation

--The paper is clearly structured, with each contribution outlined and explained in detail. The authors provide a logical flow that makes it easy for readers to understand how OMNI-EDIT was developed.
summary: The paper introduces OMNI-EDIT, an advanced image editing model addressing current limitations such as biased training data, noise, and fixed aspect ratios. It uses supervision from multiple specialist models, improved data quality via importance sampling with large multimodal models, and a new architecture called EditNet for better editing performance. OMNI-EDIT supports various aspect ratios and outperforms existing models in evaluations.
weaknesses: 1. The primary concern with this paper is its similarity to a CVPR 2024 paper [1], which proposed a generalist modeling interface that formulates most vision tasks as instruction-based editing. In contrast, the proposed paper is simpler and more direct as it focuses on building a generalist model specifically for editing tasks rather than all vision tasks. The training approach is similar to [1] which also involves generating data using current specialist models. This reduces the contribution of leveraging specialist models to train the generalist model. Additionally, the experimental results lack a comparison with [1], which would be necessary to highlight differences.

2. Although the paper claims that the model supports inputs of varying sizes and resolutions, there does not appear to be a detailed explanation of this in the methodology or experimental sections. This is crucial information, as it raises questions about how this feature is implemented and whether it involves any novel techniques.

3. The training data for Omni-Edit consists of 505K samples, all generated by specialist models and selected using importance sampling. It would be beneficial to discuss whether different training data sizes affect the model’s results.

4. Some specialist models mentioned training an image-inpainting model, but there seems no description of the training data for specialist models.

5. The overall framework is complex, requiring the training of specialist models, a score prediction model for importance sampling, and finally the Omni-Edit model with all the collected data. And such a process makes it difficult to dynamically include more editing tasks.

6. There seems to be unnecessary "5" in Lines 361 and 362.

7. Typo: In Line 246, "for the our editing model".

【1】InstructDiffusion: A Generalist Modeling Interface for Vision Tasks.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. The authors rely on GPT 4o knowledge to build the scoring model. How robust is the scoring model with InternVL in ranking the quality of the data point, how's the accuracy level in general? 
2. Could the scoring model be further improved by finetuning with human labeled GT data?
3. It seems the inpainting specialist model cannot handle shadow/reflection. Any idea to improve it?
4. Authors rely on half synthetic data for training, is the trained model affected by the inpainting artifact caused by the synthetic generation?
soundness: 3
strengths: 1. The paper is well-written and easy to read.
2. The proposed method pipeline sound reasonable.
2. Experiments are reasonable with comparisons to many recent methods, and good ablation study.
summary: This paper presents a new multi-task image editing model. The key contributions of this paper includes (1) framework for learning generalist from data generated from a set of specialist models, (2) scoring of data generated from specialist using a finetuned VLM model, (3) control-net like architecture with DiT. Authors conducted experiments to compare multiple recent methods and did a nice ablation study analyzing components of the proposed model.
weaknesses: 1. Lacks novelty. The most significant novel idea of this paper is to train specialist editing models to generate training data for training unified generalist editing model. This idea is quite straightforward and I think many existing multi-task generation models mixes data from different sources, either existing data, or data generated by some models (can be specialist) for joint training from multiple data sources.
2. Results quality. The visual results are not impressive (shadow/reflection is left after removal, inpainting/visual artifact in several results, cartoonish object generated given photo-realistic background, change color instruction lead to texture changes, etc.)

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. I have a question about the generalization ability of the OMNI-EDIT. Since the seven specialist models are trained on their pre-defined tasks, will OMNI-EDIT have the ability to generalize to editing instructions beyond these seven (for example move a person from right to left)?
2. Ideally the specialist model should generate close to perfect edited data, but in real scenarios, for some tasks, a trained specialist model may still not generate great data. Do you encounter this situation where a large portion of the generated data are filtered leading to a small proportion in the final training dataset? Will this data unbalancing affect the model's performance?
3. Can we have some visualization examples of your proposed training data and benchmark?
soundness: 3
strengths: 1. The proposed training data generation process using seven different specialist models and a better filtering method is novel and reasonable.
2. The modification from simple ControlNet + DiT to the proposed EditNet especially on accessing intermediate presentations and updating text presentations is interesting.
3. The results are strong in both quantitative metrics and qualitative assessments."
summary: The paper introduces an effective editing image training data generation process and a new model structure EditNet for general image editing of various image resolutions. The paper proposes to train and utilize seven different specialist image editing models to generate training data and use LLM to filter good-quality images, the paper also proposes a modified model backbone for general image editing capability training. The proposed method is trained on the OMNI-EDIT training dataset and experiments are conducted on the proposed OMNI-EDIT-Bench.
weaknesses: 1. I would recommend updating Figure 3, the color of the frozen DiT model is near the same as the white background making readers difficult to comprehend.
2. It would be better if the paper could also include a walkthrough caption other than the only training pipeline image in Figure 2.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1) Will the dataset be released?

2) When constructing the training data for object removal, how to ensure the generated content in the removal area will be background instead of a new object?

3) Could the authros provide visualized examples of (src image, tgt image, instruction) in the training set for each task?

4) In Tab. 1, It is unclear how different methods are evaluated on their image editing capabilities. Were user studies conducted to get the star ratings? Or are they rated by some automatic metrics? 

5) MagicBrush is a human-annotated dataset that also includes 7 tasks. How does the data quality of the synthetic dataset introduced in this paper compare with MagicBrush? Is there a quantitative evaluation?
soundness: 2
strengths: 1) The curated dataset greatly expands the instruction-based image editing datasets with multiple specialists.

2) The paper standardizes the procedure to automatically generate the trainig data for 7 image editing tasks.

3) The proposed method performs well on both automatic metrics and user studies.
summary: This paper builds an instruction-based image editing method using synthetic data from 7 specialists. The major contributions include: 1) Curated an (image, editing instruction) dataset for seven editing tasks using the importance sampling strategy; 2) Proposed a ControlNet-like image-editing architecture to achieve these tasks. Experiments show that Omni-Edit outperforms existing approaches after training the proposed model on the curated dataset.
weaknesses: 1) The ablations are not comprehensive. Since the authors claim the new dataset and their model are two separate contributions, they could compare with existing methods trained on their dataset and their method trained on the existing dataset to validate each contribution. For example, the authors can report the results of CosXL-Edit trained on Omni-Edit-Bench.

2) L433 states that while prior methods struggle to perform editing for non-squared images, the proposed method can handle multiple resolutions. However, only two visualized examples are provided in the paper, which is not convincing. I suggest the authors report quantitative results on non-squared images.

Meta-review: This paper presents an instruction-based image editing method for seven different tasks, trained on the basis of pre-trained task-specific models, as well as LLM scoring. A novel architecture is presented and trained using images with different image resolutions. The authors release a new dataset to the community along with this work.  Initially there were some concerns regarding the evaluations and ablations of the method, and similarilty to recent related work. The paper writing, results, dataset, and proposed approach are appreciated by the reviewers.

Decision: Accept (Poster)