OmniEdit: Building Image Editing Generalist Models Through Specialist Supervision

Authors: Cong Wei, Zheyang Xiong, Weiming Ren, Xeron Du, Ge Zhang, Wenhu Chen
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Image Editing, Diffusion Models

Abstract:
Instruction-guided image editing methods have demonstrated significant potential by training diffusion models on automatically synthesized or manually annotated image editing pairs. However, these methods remain far from practical, real-life applications. We identify three primary challenges contributing to this gap. Firstly, existing models have limited editing skills due to the biased synthesis process. Secondly, these methods are trained with datasets with a high volume of noise and artifacts. This is due to the application of simple filtering methods like CLIP-score. Thirdly, all these datasets are restricted to a single low resolution and fixed aspect ratio, limiting the versatility to handle real-world use cases.
In this paper, we present OmniEdit, which is an omnipotent editor to handle seven different image editing tasks with any aspect ratio seamlessly. Our contribution is in four folds: (1) OmniEdit is trained by utilizing the supervision from seven different specialist models to ensure task coverage. (2) we utilize importance sampling based on the scores provided by large multimodal models (like GPT-4o) instead of CLIP-score to improve the data quality. (3) we propose a new editing architecture called EditNet to greatly boost the editing success rate, (4) we provide images with different aspect ratios to ensure that our model can handle any image in the wild. We have curated a test set containing images of different aspect ratios, accompanied by diverse instructions to cover different tasks. Both automatic evaluation and human evaluations demonstrate that OmniEdit can significantly outperforms all the existing models.

Decision: accept (poster)
Reviewer scores: 6, 5, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729284572685

thread_type:
official_review

<thread_object>
creator: reviewer_qafp

<content>
summary:
This paper builds an instruction-based image editing method using synthetic data from 7 specialists. The major contributions include: 1) Curated an (image, editing instruction) dataset for seven editing tasks using the importance sampling strategy; 2) Proposed a ControlNet-like image-editing architecture to achieve these tasks. Experiments show that Omni-Edit outperforms existing approaches after training the proposed model on the curated dataset.

strengths:
1) The curated dataset greatly expands the instruction-based image editing datasets with multiple specialists.

2) The paper standardizes the procedure to automatically generate the trainig data for 7 image editing tasks.

3) The proposed method performs well on both automatic metrics and user studies.

weaknesses:
1) The ablations are not comprehensive. Since the authors claim the new dataset and their model are two separate contributions, they could compare with existing methods trained on their dataset and their method trained on the existing dataset to validate each contribution. For example, the authors can report the results of CosXL-Edit trained on Omni-Edit-Bench.

2) L433 states that while prior methods struggle to perform editing for non-squared images, the proposed method can handle multiple resolutions. However, only two visualized examples are provided in the paper, which is not convincing. I suggest the authors report quantitative results on non-squared images.

questions:
1) Will the dataset be released?

2) When constructing the training data for object removal, how to ensure the generated content in the removal area will be background instead of a new object?

3) Could the authros provide visualized examples of (src image, tgt image, instruction) in the training set for each task?

4) In Tab. 1, It is unclear how different methods are evaluated on their image editing capabilities. Were user studies conducted to get the star ratings? Or are they rated by some automatic metrics? 

5) MagicBrush is a human-annotated dataset that also includes 7 tasks. How does the data quality of the synthetic dataset introduced in this paper compare with MagicBrush? Is there a quantitative evaluation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730179723928

thread_type:
official_review

<thread_object>
creator: reviewer_dyun

<content>
summary:
The paper introduces an effective editing image training data generation process and a new model structure EditNet for general image editing of various image resolutions. The paper proposes to train and utilize seven different specialist image editing models to generate training data and use LLM to filter good-quality images, the paper also proposes a modified model backbone for general image editing capability training. The proposed method is trained on the OMNI-EDIT training dataset and experiments are conducted on the proposed OMNI-EDIT-Bench.

strengths:
1. The proposed training data generation process using seven different specialist models and a better filtering method is novel and reasonable.
2. The modification from simple ControlNet + DiT to the proposed EditNet especially on accessing intermediate presentations and updating text presentations is interesting.
3. The results are strong in both quantitative metrics and qualitative assessments."

weaknesses:
1. I would recommend updating Figure 3, the color of the frozen DiT model is near the same as the white background making readers difficult to comprehend.
2. It would be better if the paper could also include a walkthrough caption other than the only training pipeline image in Figure 2.

questions:
1. I have a question about the generalization ability of the OMNI-EDIT. Since the seven specialist models are trained on their pre-defined tasks, will OMNI-EDIT have the ability to generalize to editing instructions beyond these seven (for example move a person from right to left)?
2. Ideally the specialist model should generate close to perfect edited data, but in real scenarios, for some tasks, a trained specialist model may still not generate great data. Do you encounter this situation where a large portion of the generated data are filtered leading to a small proportion in the final training dataset? Will this data unbalancing affect the model's performance?
3. Can we have some visualization examples of your proposed training data and benchmark?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730527241099

thread_type:
official_review

<thread_object>
creator: reviewer_kwlc

<content>
summary:
This paper presents a new multi-task image editing model. The key contributions of this paper includes (1) framework for learning generalist from data generated from a set of specialist models, (2) scoring of data generated from specialist using a finetuned VLM model, (3) control-net like architecture with DiT. Authors conducted experiments to compare multiple recent methods and did a nice ablation study analyzing components of the proposed model.

strengths:
1. The paper is well-written and easy to read.
2. The proposed method pipeline sound reasonable.
2. Experiments are reasonable with comparisons to many recent methods, and good ablation study.

weaknesses:
1. Lacks novelty. The most significant novel idea of this paper is to train specialist editing models to generate training data for training unified generalist editing model. This idea is quite straightforward and I think many existing multi-task generation models mixes data from different sources, either existing data, or data generated by some models (can be specialist) for joint training from multiple data sources.
2. Results quality. The visual results are not impressive (shadow/reflection is left after removal, inpainting/visual artifact in several results, cartoonish object generated given photo-realistic background, change color instruction lead to texture changes, etc.)

questions:
1. The authors rely on GPT 4o knowledge to build the scoring model. How robust is the scoring model with InternVL in ranking the quality of the data point, how's the accuracy level in general? 
2. Could the scoring model be further improved by finetuning with human labeled GT data?
3. It seems the inpainting specialist model cannot handle shadow/reflection. Any idea to improve it?
4. Authors rely on half synthetic data for training, is the trained model affected by the inpainting artifact caused by the synthetic generation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730644010132

thread_type:
official_review

<thread_object>
creator: reviewer_u8aw

<content>
summary:
The paper introduces OMNI-EDIT, an advanced image editing model addressing current limitations such as biased training data, noise, and fixed aspect ratios. It uses supervision from multiple specialist models, improved data quality via importance sampling with large multimodal models, and a new architecture called EditNet for better editing performance. OMNI-EDIT supports various aspect ratios and outperforms existing models in evaluations.

strengths:
--The paper introduces OMNI-EDIT to address limited editing capabilities, poor data quality control, and lack of support for varying resolutions, which are critical point in image editing method.

--The use of importance sampling with high-quality data scoring via distilled multimodal models (such as InternVL2) demonstrates a meticulous approach to data curation

--The paper is clearly structured, with each contribution outlined and explained in detail. The authors provide a logical flow that makes it easy for readers to understand how OMNI-EDIT was developed.

weaknesses:
1. The primary concern with this paper is its similarity to a CVPR 2024 paper [1], which proposed a generalist modeling interface that formulates most vision tasks as instruction-based editing. In contrast, the proposed paper is simpler and more direct as it focuses on building a generalist model specifically for editing tasks rather than all vision tasks. The training approach is similar to [1] which also involves generating data using current specialist models. This reduces the contribution of leveraging specialist models to train the generalist model. Additionally, the experimental results lack a comparison with [1], which would be necessary to highlight differences.

2. Although the paper claims that the model supports inputs of varying sizes and resolutions, there does not appear to be a detailed explanation of this in the methodology or experimental sections. This is crucial information, as it raises questions about how this feature is implemented and whether it involves any novel techniques.

3. The training data for Omni-Edit consists of 505K samples, all generated by specialist models and selected using importance sampling. It would be beneficial to discuss whether different training data sizes affect the model’s results.

4. Some specialist models mentioned training an image-inpainting model, but there seems no description of the training data for specialist models.

5. The overall framework is complex, requiring the training of specialist models, a score prediction model for importance sampling, and finally the Omni-Edit model with all the collected data. And such a process makes it difficult to dynamically include more editing tasks.

6. There seems to be unnecessary "5" in Lines 361 and 362.

7. Typo: In Line 246, "for the our editing model".

【1】InstructDiffusion: A Generalist Modeling Interface for Vision Tasks.

questions:
1. How does this paper differentiate itself in terms of contributions from the CVPR 2024 paper [1], given their similar focus on generalist instruction-guided modeling?

2. It would be better to compare the proposed method with the generalist modeling approach in [1].

3. Can the authors provide more detail on how the support for multi-size and multi-resolution inputs was implemented? What specific innovations, if any, were introduced to achieve this?

4. How does the training data size affect the performance of OMNI-EDIT? 

5. Could the authors elaborate on the training process and datasets used for the specialist models, such as the image-inpainting model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696696262

thread_type:
official_review

<thread_object>
creator: reviewer_txht

<content>
summary:
OMNI-EDIT is an image editing model designed to address the skill imbalance issues present in existing instruction-based image editing methods. It learns by combining supervision from seven different expert models and utilizes InternVL2 for data scoring. Additionally, this approach introduces a diffusion-Transformer architecture named EditNet.

strengths:
1. The authors collected a high-resolution dataset covering seven types of tasks, annotated through multiple expert models, and used InternVL2 to score the dataset. 

2. And they introduced EditNet, which enhances image noise control capabilities compared to ControlNet.

3. In the Omni-Edit-Bench benchmark, OMNI-EDIT outperformed existing baseline models across multiple evaluation metrics.

weaknesses:
1. Lack of testing of GPT4-o generated CONFIDENCE SCORE, LLM may be subjective and biased

2. Limitations of the benchmarking dataset: The Omni-Edit-Bench dataset used in the paper is small (only 62 images), which may limit the reliability and generalization of the model evaluation.

questions:
1. Since the dataset is divided into seven types of tasks, would it be possible to categorize the test set similarly? This would allow for a more detailed evaluation of the model's performance across different editing tasks and facilitate an analysis of performance differences and underlying reasons.

2. The relatively small size of the Omni-Edit-Bench dataset may limit the reliability and generalizability of model evaluation. It is recommended to assess performance on additional benchmark datasets. For example, are there results available on the Emu Edit test set?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732974353307

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response (1/3)

comment:
We appreciate the reviewers for their thoughtful feedback.

We greatly appreciate your recognition of the strengths of our work as follows:

 **Novel Data Generation Method and Novel Filtering Method**

Our Multi-Specialist data generation pipeline "**is novel**"(`R-dYun`)  and **"sounds reasonable"**(`R-KWLC`,`R-dYun`), "**standardizes the procedure** to automatically generate the training data" (`R-QAfp`), addressing "the **critical point** in image editing method" (`R-u8AW`).

Our proposed LLM Filter "**is novel**"(`R-dYun`)  and **"reasonable"**(`R-KWLC`,`R-dYun`), "demonstrates a **meticulous approach** to data curation."(`R-u8AW`), addressing "the **critical point** in image editing method." (`R-u8AW`)

Our curated dataset "**greatly expands** the instruction-based image editing datasets with multiple specialists."  (`R-QAfp`)

Our EditNet architecture design "**enhances image noise control capabilities**"(`R-txht`) and "**is interesting**"(`R-dYun`)
 
# General Response: Quality and Robustness of scores generated by LMMs (R-txht, R-KWLC)
We provide additional experiments to compare the performance of three filtering approaches: (1) the **CLIP filter**, which has been the primary filtering method in all prior works; (2) our proposed **novel LMM filters**; and (3) evaluations by a **human expert**.

Specifically, we randomly sampled 500 examples from our pre-filtered raw dataset for this analysis.

For **CLIP** score-based filtering, we follow the pipeline from InstructPix2Pix [1] to calculate image-image CLIP similarity, image-caption CLIP similarity for both original and edited images, and directional CLIP similarity. We set the thresholds at 0.75, 0.2, and 0.2 respectively, strictly adhering to the InstructPix2Pix methodology.

For **InterVL2-8B**, **InternVL2-8B-distilled (from-GPT4o)**, **GPT-4o** and four **human experts**, each method was tasked with generating a VIEScore [2] for every sample. We then apply a threshold of overall score $\geq$ 9.0 to select high-quality images.

We treated the selections made by **human experts as the ground truth**. We calculated the **Precision**, **Recall** and **F1-Score** to measure the overlap between the images selected by each method and those selected by the human experts. 
Additionally, we calculated the Pearson **correlation** coefficient between each model's predicted scores and the human experts' overall scores. Finally, we ranked the 500 examples based on each model's predicted scores and the human experts' scores to calculate the **Spearman’s rho** correlation.

|Filtering Method         | Recall | Accuracy | F1-Score |  Spearman’s rho    |Correlation| 
|-------------------------|--------|----------|----------|---------|-----------|
| Human               | 1.00   |     1.00 | 1.00     |   N/A   |  N/A      |
| GPT-4o                  | 0.67   |     0.96 | 0.79  |  0.68   | 0.77      | 
| InternVL2-8B-distilled  | 0.54   | 0.84 | 0.66 |  0.67   | 0.67      |
| InternVL2-8B            | 0.05  | 0.75 |0.09 | -0.08   | -0.12     |
| CLIP | 0.01  | 0.74 | 0.02     | -0.09   | -0.13     |

Our findings are summarized below:
1. GPT-4o and our proposed **InternVL2-8B-distilled** filter serve as the most reliable scoring functions, aligning **very well** with **human** scores, as reflected in their high correlations.
2. **CLIP-based** filtering demonstrates poor alignment with human judgments. While it achieves a 74% Accuracy, this is largely a result of the imbalanced dataset, where negative examples dominate. However, its low Recall and F1-Score of 0 shows that CLIP **fails recognize good training pairs**.
3. The promising results for InternVL2-8B-distilled further validate the effectiveness of our proposed **scoring-ability distillation method** for building a **scalable and accurate** filtering mechanism, which is crucial for generating high-quality, large-scale datasets. In [Figure 15](https://github.com/OmniEditRebuttal/OmniEditRebuttal/blob/main/Figure%2015.png), we show a case comparison between InternVL2-8B's scoring ability before and after distillation.

We believe that our **proposed LMM-based filtering method** will inspire the community to adopt LMMs for image data filtering in the future, particularly for **more complex editing or generation tasks** where traditional methods such as CLIP-based approaches fall short.


### In addition to the quantitative analysis, we provide a randomly sampled **training subset for visualization** at this link: [OmniEdit(auto pipeline)](https://huggingface.co/datasets/OmniEditRebuttal/training_set_samples)
### This allows a direct comparison with **prior works' datasets**, provided at this link:[UltraEdit(auto pipeline)](https://huggingface.co/datasets/BleachNick/UltraEdit_500k), [InstructPix2Pix(auto pipeline)](https://huggingface.co/datasets/timbrooks/instructpix2pix-clip-filtered), [MagicBrush(human-annotated)](https://huggingface.co/datasets/osunlp/MagicBrush).


### All our datasets will be open-sourced!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents an instruction-based image editing method for seven different tasks, trained on the basis of pre-trained task-specific models, as well as LLM scoring. A novel architecture is presented and trained using images with different image resolutions. The authors release a new dataset to the community along with this work.  Initially there were some concerns regarding the evaluations and ablations of the method, and similarilty to recent related work. The paper writing, results, dataset, and proposed approach are appreciated by the reviewers.