OmniEdit: Building Image Editing Generalist Models Through Specialist Supervision
Authors: Cong Wei
Zheyang Xiong
Weiming Ren
Xeron Du
Ge Zhang
Wenhu Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Image Editing
Diffusion Models
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose to train omnipotent image editing model by experts supervision and llm filtering.

Abstract: Instruction-guided image editing methods have demonstrated significant potential by training diffusion models on automatically synthesized or manually annotated image editing pairs. However, these methods remain far from practical, real-life applications. We identify three primary challenges contributing to this gap. Firstly, existing models have limited editing skills due to the biased synthesis process. Secondly, these methods are trained with datasets with a high volume of noise and artifacts. This is due to the application of simple filtering methods like CLIP-score. Thirdly, all these datasets are restricted to a single low resolution and fixed aspect ratio, limiting the versatility to handle real-world use cases.
In this paper, we present OmniEdit, which is an omnipotent editor to handle seven different image editing tasks with any aspect ratio seamlessly. Our contribution is in four folds: (1) OmniEdit is trained by utilizing the supervision from seven different specialist models to ensure task coverage. (2) we utilize importance sampling based on the scores provided by large multimodal models (like GPT-4o) instead of CLIP-score to improve the data quality. (3) we propose a new editing architecture called EditNet to greatly boost the editing success rate, (4) we provide images with different aspect ratios to ensure that our model can handle any image in the wild. We have curated a test set containing images of different aspect ratios, accompanied by diverse instructions to cover different tasks. Both automatic evaluation and human evaluations demonstrate that OmniEdit can significantly outperforms all the existing models.

Published as a conference paper at ICLR 2025
OMNIEDIT: BUILDING IMAGE EDITING GENERALIST MODELS THROUGH SPECIALIST SUPERVISION
1,3 Cong Wei", Zheyann Xiong', Weiming Ren, *Xinrun Du, Ge Zhang, Wenhu Chen University of Waterloo. UUniversity of Aesconsin-Madison Veector Institute, *M-A-P
cong. cong.weiee wei uwaterloo.ca zheyang @ wisc.edu, wenhuchen @ uwaterloo.ca
imagie painting
Replane the bobile with Fire
Para the hame into colourful unioorn
anvironment lato
anowy
landscape
shark
a golden retelover the foraground
Remove hat
Figure 1: Editing high-resolution multi-aspect images with OMNI-EDIT. OMNI-EDIT is an instruction-basee based image editing generalist capable of performing diverse editing tasks across dif- ferent aspect ratios and resolutions. It accurately follows instructions while preserving the original image's fidelity. We suggest zooming for better visualization.
ABSTRACT
Instruction Instruction-guided image editing methods have demonstrated significant poten- tial by training diffusion models on automatically synthesized or manually an- notated image editing pairs. However, these methods remain far from practical, real-life applications. We identify three primary challenges contributing to this gap. Firstly, existing models have limited editing skills due to the biased synthesis process. Secondly, these methods are trained with datasets with high volume of noise and artifacts. This is due to the application of simple filtering methods like CLIP-score. Thirdly, all these datasets are restricted to single low resolu- tion and fixed aspect ratio, limiting the versatility handle real-world use cases. In this paper, we present OMNI-EDIT, which is an omnipotent editor to handle
First authors.

---

Published as a conference paper at ICLR 2025
seven different image editing tasks with any aspect ratio seamlessly. Our con- tribution is in four folds: (1) OMNI-EDIT is trained by utilizing the supervision from seven different specialist models ensure task coverage.
(2) we utilize im- portance sampling based on the scores provided by large multimodal models (like GPT-4o) instead of CLIP-score improve the data quality. (3) we propose a new editing architecture called EditNet to greatly boost the editing success rate, (4) we provide images with different aspect ratios ensure that our model can handle any image in the wild. We have curated a test set containing images of different aspect ratios, accompanied by diverse instructions cover different tasks. Both automatic evaluation and human evaluations demonstrate that OMNI-EDIT can significantly outperform the existing models. Our code, dataset and model will be available at https: :tteps://tan http//tttiieeeeee ai- lab. github. i he/mmmmniiii
INTRODUCTION
Image editing, particularly when following user instructions to apply semantic transformations to real-world photos, has seen significant advancements. Recently, text-guided image editing (Brooks et al., 2023) has gained prominence over traditional methods such as mask- or region-based editing (Meng et al., 2022). With the rise of diffusion models (Rombach et al., 2022; Podell al., image editing techniques meth-
2024; Chen et al., 2024a; Sauer et al., 2024), numerous diffusion- have emerged. Generally, they can be roughly divided into two types: (1) Inversion-
ods (Parmar et al., 2023; Kawar et al., 2023; Gal et al., 2023; Xu et al., 2023; Tumanyan et al., 2023; Tsaban & Passos, 2023) propose to perform zero-shot image editing by inverting the diffusion process and manipulating the attention map in the intermediate diffusion steps to achieve desired editing goal. (2) End-to-end methods (Brooks et al., 2023; Zhang et al., 2024a; Sheynin et al., 2024; Zhao al., 2024; Fu al., 2024) propose fine-tune an existing diffusion model on large-scale image editing pairs to learn the editing operation in an end-to-end fashion. End-to-end methods have generally achieved better performance than inversion-base methods and gained higher popularity.
Table 1:
Comparison of OMNI-EDIT with all the existing end-to-end image editing models. The scores are based on a preliminary studies on around 50 prompts.
Property
InstructP2P
MagicBrush
Ultraldit
MGIE
HQEdit
CosXL
OMNI-EDIT
Training Dataset Properties
Reul Image?
X
x
X
Any Res? x
X
x
X
High Res?
x
/
Fine-grained Image Editing Skills
Obj-Swap
*
*
*
*
Obj-Add
**
**
Ohj Remove
*
*
Autribute
*
*
Back-Swap
***2
*
Environment
*
*
Style
*
Despite their effectiveness, end-to-end methods face a significant limitation: the scarcity of human- annotated image editing pairs. As result, all current end-to-end approaches depend on syn- thetic training data. For instance, existing datasets are synthesized using techniques such as Prompt2Prompt (Hertz et al., 2023) or mask-based editing models like SD-Inpaint (Rombach al., 2022), and DALLE-2233 (Ramesh et al., 2022; Betker et al., 2023). However, these synthetic data generation pipelines exhibit significant biases, resulting in the following limitations:
Limited Editing Capabilities: The synthetic data is heavily influenced by the underlying generation models. For example, Prompt2Prompt struggles with localized edits, such as adding, removing, or swapping objects, while SD-Inpaint and DALLE-2 are ineffective at global edits, such as style or background changes. As a result, models trained on such data inherit these limitations.
Poor Data Quality Control: Most approaches use simplified filtering mechanisms like CLIP- score (Radford al., 2021) or DINO-score (Caron et al., 2021) automatically select training samples. However, recent studies (Ku et al., 2024) show that these metrics exhibit poor correlation with actual data quality, leading suboptimal training data that negatively impacts the model.
2

---

Published as a conference paper at ICLR 2025
Lack of Support for Varying Resolutions: All current models trained on square image editing pairs, making their generalization to non-square images poor.
In our preliminary studies, we curate a few prompts for seven different desired tasks to observe their success rate across the board. We show our findings in Table 1. This show that these models are truly biased in their skills caused by the underlying synthesis pipeline.
In this paper, we introduce OMNI-EDIT a novel model designed to address these challenges through four key innovations:
1. Specialist-tos -ortaliiittt-GGGeallill Supervision: We propose learning a generalist editing model, OMNI- EDIT, by leveraging supervision from multiple specialist models. Unlike previous approaches that rely on a single expert, we conduct an extensive survey and construct (or train) seven experts, each specializing different editing task. These specialists provide supervisory signals OMNI-EDIT.
Importance Sampling: To ensure high-quality training data, we employ large multimodal models to assign quality scores to synthesized samples. Given the computational cost of GPT- 4o (Achiam al., 2023), we first distill its scoring ability into Intern (Chen al., 2024b) through medium-size sized samples. Then we use the Intern VL2 model for large-scale scoring.
trrtoontraaffome--bbee based architec- ture (Peebles & Xie, 2022) that facilitates interaction between the control branch and the original branch via intermediate representations. This architecture enhances OMNI-EDIT 's ability to com- prehend diverse editing tasks.
3. EditNet Architecture: We introduce EditNet, a novel
Support for Any Aspect Ratio: During training, we incorporate a mix of images with varying aspect ratios well as high resolution, ensuring that OMNI-EDIT can handle images of any aspect ratio with any degradation in the output quality.
We curate image editing benchmark OMNI-EDIT-BENCEE which contains diverse images of dif- ferent resolutions and diverse prompts that cover all the listed editing skills. We perform comprehen- sive automatic and human evaluation to show the significant boost of OMNI-EDI' over the existing baseline models like CosXL-Edit (Boesel & Rombach, 2024), UltraEdit (Zhao et al., 2024), etc.
2
PRELIMINARIES
2.1 TEXT-TO-IMAGE DIFFUSION MODELS
Diffusion models (Song et al., 2021; Ho et al., 2020) are a class of latent variable models parameter- ized by 0, defined as pe (xo) := pa (Xo.r) dx where Xo q(xo) represents the original data, and X XT are progressively noisier latent representations of the input image Xa- Through- out the process, the dimensionality of Xu and the latent variables X1:7 remains consistent, with X0.7 E iRd where d corresponds to the product of the image's height, width, and channels. The forward (diffusion) process, denoted as q(xi.T xo), is a predefined Markov chain that incrementally adds Gaussian noise to the data according to a pre-defined schedule {8)) The process of forward diffusion is defined as:
q(xIT(X)) xx) = II [(xx x(x:|x=-)) Nxx: /1 3, x,11,
(I)
where N denotes a Gaussian distribution, and 3, controls the amount of noise added each step. The objective of diffusion models is to reverse this diffusion process by learning the distribution po(x--1|x<) which enables the reconstruction of the original data Xg from a noisy latent x,. This reduces denoising problem where the model is trained to denoise the sample x; q(x: (x)) back into Xo. The maximum log-likelihen likeliioood training objective breaks down to minimizing the weighted mean squared error between the model's prediction & c) and the true data Xo:
arg max log po (xolc) = arg min ()xxe)))> [Fe, [w: [ X x0]
where (Xo, c) pairs come from the dataset D, with representing the text prompt. The term w, is a weighting factor applied to the loss at each timestep t. For simplicity, prior papers (Song et al., 2021; Ho al., 2020; Karras et al., 2022) will w, be 1.
3

---

Published as a conference paper at ICLR 2025
2.2 INSTRUCTION- ESTRUCTION-BBAED IMAGE EDITING IN SUPERVISED LEARNING
Instruction-basse image editing can be formulated as a supervised learning problem. Existing meth- ods (Brooks al., 2023; Zhang al., 2024a) often adopt paired training dataset of text editing instructions and images before and after the edit. An image editing diffusion model is then trained on this dataset. The latent diffusion objective is defined as:
arg max log ps (xo Xo. c) = arg min (x) [E,,e|/i((>
-x1
(3)
where (x), Xo. e) triples are sampled from the dataset D with Xo denoting the source image, denoting the editing instruction and x denoting the target image.
Stage 1: Specialist learning
Stage task-speolife synthetle generation Add cap
Pre rre-trained T2I model(3D)
Object-Swap Specialist
Object-Addition Specialist
EAII
Object-Removal Specialist
Task specific Fine
Specialist
Taakeppcific Augmentation
Specialist
Background-Swwar Specialist
Object-Property Specialist
BAI
Beploce Steruon with delecker
aaptions
Specialists
Object-Swap Specialist
EdI
Resolution
Stage 4: Learning a Generalist
Stage 3: Confidence Scoring
Semantic Consilteray Score: 10
Add MM
Percestual Quality Score: 10
cap,
Contidence score: Àg
Remove child
Confidence Scoring
Omni-Edit
Oad
function
Percaptual Quality Score:
the shild
Optimization
Figure 2: Overview of the OMNI-EDIT training pipeline. The pipeline consists of four stages: (1) task-specific specialist models are trained for diverse editing tasks; (2) these specialist models are used to generate large, high-resolution, multi-aspect-ratio dataset; (3) cost-efficient distilled large multi-modal model (LMM) assigns importance weights to each pair of image-editing data; and (4) the final generalist model is trained the weighted dataset.
3 LEARNING WITH SPECIALIST SUPERVISION
In this section, we introduce the entire apecialist-to-generalll learning framework to build OMNI- EDIT. We describe the overall learning objective in subsection 3.1. We then describe how we leam the specialists in subsection 3.2 and the importance weighting function in subsection 3.3. In Figure 2, we show the overview of the OMNI-EDIT training pipeline.
3.1 LEARNING OBJECTIVE
We assume there is groundtruth editing model p(x' |x, c). which can perform any type of editing tasks perfectly according to the instruction c. Our goal is to minimize the divergence between px(x' (x, c) with p(x'|xx |x. c) by updating the parameters A:
L( = Σ DKL(p(x1 c))p.c)lpppooxxxll
Σ
E 1oox/1x
(4)
4

---

Published as a conference paper at ICLR 2025
Intern112:
PQ:
(reasoning: There no visible distortions... The lighting and shadows consistent, score: 10
(reasoning: The edited image perfectly follows this instruction. The rugby player is completely replaced by an astronaut in a space suit... The editing is effective and matches the instruction without any overediting... score: [10, 10]}
Source Image
InternVia Distilled from GPP44: PQ:
(reasoning: The image has some notable distortions, especially in body proportions, as the legs and arms of the individuals appear unnaturally elongated...,
score: 4}
SC:
Edited Image
The editing partially follows instruction by adding padded sults that slightly resemble space suits, but doesn't convincingly replace the players with as astronauts... The players' bodies, poses, and proportions remain largely intact, making like an incomplete tramstornation.
(reasoning:
Editing Instruction:
Replace the rugby player with astronaut in space suit
score:
Figure 3: Intern V1.2 as scoring function before (top right) and after (bottom right) fine-tuning on GPT-4o's ' response.
the top right, the original Intern VI.2 fails identify the unusual distortions in the edited image it also does not spot the error when the edited image fails to meet the specified editing instructions. On the bottom right, peectundd-InternVV successfully detects such failures and serve reliable scoring function.
where is a constant, which we leave out the following derivation. However, since we don't have access to p(x'Ix c), we adopt importance sampling for approximation:
L(O) = - --=--EE q(x' (xx c) p(x|xx, c) g(x'|x, c)
log pa (x (x'(x,,)
X.c
(5)
2 - -B (x,c)=D
[E [xx (x,c) [A(x', log ci]]
(x,,)~D (xxxmx [X(x', x, c) log (x c)]
where q('' (x, e) is the proposal distribution and A(-) is the importance function. To better approx- imate the groundtruth distribution p(x') |x, c), we propose to use an ensemble model q(x') |x, c). In essence, q(x'|xx |x
qs (x'x x c). where Is is specialist distribution decided by the type of the instruction e (e.g. object removal, object addition, stylization, etc). Combing with Equation 3, our objective can be rewritten as:
arg min L(0) arg min
A(x c) E 1o (x,,
x
(6)
The whole process can be described as: we first sample pair from dataset D, and then choose the corresponding specialist 4s to sample demonstrations x' for the our editing model Xe x x, e) to approximate with importance weight of A(x', c). We formally provide the algorithm in 1. In our specialist-to-geerraiis framework, we need to have a series of specialist models (g.(.) (.)): and an importance function A(-). We describe them separately in subsection 3.2 and subsection 3.3.
3.2 CONSTRUCTING SPECIALIST MODELS
We group the image editing task into 7 categories as summarized in Table 2. For each category, we train or build a task specialist (x(x' x, x) to generate millions of examples. Table provides detailed information task groups and example editing instructions In this section, we briefly summarize each specialist, with details available in Appendix A.
Object Replacement. We trained an image-inpainting model to serve as the specialist repjace for object replacement. Given a image x and an object caption Cobj and a object mask Marj- The dobj_replace can fill the content indicated by the mask with object in Cobj. We then generate object replacement sample by masking out an existing object and fill the image with new object.
Object Removal. We trained object removal. We use a similar procedure as in the object replacement but use predicted back- ground content caption to inpaint the masked image.
image inpainting model to serve the specialist qobj removal for
5

---

Published as a conference paper at ICLR 2025
Table 2: Task Definitions and Examples
Editiin Tasks
Definition
Instruction Example
Object Swap
с describes object to replace by specifying both object remove and the new object to add, along with their properties snch as appear- ance and location.
Replace the black cat brown dog the image.
Object Remowal
e deseriies which chject to remove by specitying the object's properies such as appearance, location, and size.
Remove the black cat from the image.
Object Addition
e describes object add by specifying the object's propeties such as appearance and location.
the lefl side the image,
Attribute Modification
e describes how modify the propesties an object, auch as changing its color and facial expression.
Change blue car to car.
Background Swap
e descrihes how to replace the background of the image, specifying what the new background should be.
Replace the background with a space-ship inte- rior.
Fnvironment Change
e deserites change the overall environment, such as the weather. lighting.
Change the scene from daytime to nighttime.
without altering specific objects.
Style Transfer
deserines how apply specific artistic style visual effect the image, altering overall appearance while keeping the content the same.
Apply watercolor painting style the image.
Object Addition. We treat object addition the inverse task of object removal. Attribute Modification. We adopt the Prompt-to-Pronee erate examples. To enable precise modification, we adapt the method from Sheynin et al. (2024) where we provide a mask Maty for the object and force P2P to only make edits inside the mask. Background Swap. We use similar procedure in the object replacement but use inverse mask of the object to indicate the background and guide the inpainting.
(P2P) (Hertz et al., 2023) pipeline to gen-
Environment Modification. For environment modification, we use P2P pipeline generate origi- nal and edited image.
Style Transfer. We use CosXL-Edit (Boesel & Rombach, 2024) as the specialist model as its train- ing data contains large number of style transfering examples. We provide CosXL-Edit with (x, c), and let it generates the edited image x'.
3.3 IMPORTANCE WEIGHTING
The importance weighting function takes as input a tuple of source image, edited image, and edit- ing prompt. Its purpose is to assign higher weights to data points that are more likely to be sampled from the ground truth distribution, and lower weights to the unlikely ones. This is essentially a qual- ity measure to up-weight high-quality samples. Unlike previous work, we do not use CLIP score because prior work (Jiang et al., 2024) has shown its low correlation with human judges. Instead, we propose to use large multimodal models (LMMs) to approximate the weighting function, as they demonstrate strong image understanding. Following VIEScore (Ku et al., 2024), we designed a prompting template for GPT-4o (Achiam et al., 2023) to evaluate the image editing pairs and output a score on a scale from 0 to 10. We then filter out data with a score greater than or equal to 9, so the LMM essentially serves as binary weighting function:
[1,
if LMM xx,x,)) otherwise
X(x, x, c)
{o,
Details of the prompt template are provided in the Appendix.
While the GPT-4o is an effective choice for this task, scoring large-scale datasets with millions of examples is extremely costly and time-consuning. Therefore, we employ knowledge distillation from GPT-4o to a smaller 8B model, Intern VL.2 (Chen et al., 2024b). For each task, we sample 50K data points and instruct GPT-4o to output both a score and a score rationale. We fine-tune InternVL2 on these GPT-4o-generated examples. After fine-tuning, InternVL2 performs as an ideal scoring function due to its smaller size and efficiency. A comparison of the model's performance before and after fine-tuning is presented in the Appendix. Finally, we apply the fine-tuned Intern VL2 model to filter data across a dataset with millions of samples. Only examples with a score of 2 9 are retained, resulting in curated training dataset of 1.2M examples. We visualize Intern response as a scoring function before and after fine-tuning in Figure 3. We observe that fine-tuning Intern on GPT-4o's 's response effectively turns Intern VL2 into realiable scoring function and it can identify unusual distortions or unsuccessful edit that does not follow the editing instruction. Additional dataset statistics are detailed in the Appendix.

---

Published as a conference paper at ICLR 2025
Rakey Latern token
Text token
Canailtun Image tokeen
Nane Latent token
Text tokion
Gand bunn Image to
Nowy Lateni taken
Image token
Cundition
Text token
self-attention
self-attention ? (trainable copy)
celf-attention
self-attention (trainable copy)
self-atter celf-attention
(a) EditNet
(b) ControlNet
(c) InstructPix2Pix
Figure 4:
Architecture Comparison between EditNet(ours). ControlNet and concatenation) for DiT models. Unlike ControlNet's parallel
execution, EditNet allows adaptive adjustment of control signals by intermediate representations interaction between the control branch and the original branch. EditNet also updates the text representation, enabling better task understanding.
EDITNET
We found that directly fine-tuning a pre-trained high-quality diffusion model like SD3 using channel- wise image concatenation methods (Brooks et al., 2023) compromises the model's original repre- sentational capabilities (see Figure 10 and Section 5.2 for details comparison).
To enable a diffusion transformer to perform instruction-based image editing while preserving its original capabilities, we introduce EditNet to build OMNI-EDIT. EditNet can effectively transform common DIT models like SD3 into editing models. As illustrated in Figure 4, replicate each layer of the original DIT block as a control branch. The control branch DIT blocks allow interaction between the original DIT tokens, conditional image tokens, and the editing prompts. The output of the control branch tokens is then added to the original DIT tokens and editing prompts. Since the original DIT blocks are trained for generation tasks and not aware of the editing instructions specifying which contents to modify and how to modify them, this design allows the control branch DIT to adjust the representations of the original DIT tokens and editing prompts according to the editing instruction, while still leveraging the strong generation ability of the original DIT. Com- pared to ControlNet (Zhang et al., 2023), our approach offers two key advantages that make it more suitable for image editing tasks: First, ControlNet does not update text representations, making it challenging to execute editing tasks based on instruction, particularly object removal, as fails to understand the "removal" intent (see Figure 11). Secondly, ControlNet's control branch operates in parallel without access to the original branch's 's intermediate representations. This fixed precompu- tation of control signals restricts the overall representation power of the network. We provide an ablation study on the OMNI-EDIT architecture design in Section 5.2.
5 EXPERIMENTS
In this section, we first provide statistics of the OMNI-EDIT training set and test set in Table 5. Then we introduce the human evaluation protocol in Section 5, and comparative baseline system in 5. We present the main results in Section 5. highlighting the advantages of OMNI-EDIT in tacking multi- aspect ratio, multi-resolution, and multi-task image editing. In Section 5.2, we study the advantages of importance sampling for synthetic data. In Section 5.2, we perform an analysis to study the design of OMNI-EDIT
OMNI-EDIT Training Dataset. We constructed the training dataset D by sampling high-resolution images with a minimum resolution of megapixel from the LAION-SB (Schuhmann et al., 2022) and 0mge (Kuznetsova et al., 2020) databases. The images cover a range of aspect ratios including 1:1, 2:3, 3:2, 3:4, 4:3, 9:16, and 16:9. For the task of object swap, we employed spe- cialist model to generate 1.5 million entries. We then applied Intern VL2 for importance weighting, retaining samples with scores of 9 or higher, resulting in dataset of 1 150K entries for this task. Sim- ilarly, we generate 250k- IM samples for each task, then keep the top 10% as the final dataset. The final training dataset comprises 1.2M entries, with detailed information provided in Appendix 4. OMNI SaaN-EDrr-Benchh To create a high-resolution, multi- ratio, multi-task benchmark for instruction-batt based image editing, manually collected 62 images from pexels (2024) and LAION-
7

---

Published as a conference paper at ICLR 2025
Original Image
Omnildil (Ouza)
OostI-Edit
UltraBait
MagloBrosh
Figure 5: Qualitative comparison between baselines and OMNI-EDIT on subset of the test set.
5B (Schuhmann et al., 2022). These images cover a variety of aspect ratios, including 1:1, 2:3, 3:2, 3:4, 4:3, 9:16, and 16:9. We ensured that the images feature diverse range of scenes and object counts, from single to complex compositions. Additionally, we selected images with a relatively high aesthetic score to better align with the practical use cases of image editing. For each image, we tasked the model with performing 7 tasks as outlined in Table 2. This results in total of 434 edits. OMNI-EDIT implementation details. The -MNI-EDIT model built upon Stable diffusion Medium(Esser et al., 2024) with EditNet architecture. The stable diffusion 3 has 24 DiT layers. Each layer has corresponding EditNet layer. We train OMNI-EDIT on the 1 1.2M OMNI-EDIT train- ing dataset for epochs single node with 8 H100 GPUs.
Baseline models. We compare OMNI-EDIT with 8 other text-guided image editing baselines: Mag- icBrush (Zhang et al., 2024a), InstructPix2Pix (Brooks et al., 2023), UltraEdi(SDD3) (Zhao et al., 2024), DiffEdit (Couairon et al., 2022), SDEdit (Meng et al., 2022), CosXL-Edit (Boesel & Rom- bach, 2024), HIVE (Zhang et al., 2024b) and HQ-Edii (Hui et al., 2024).
Evaluations Protocol We conduct both human evaluation and automatic evaluation. For the human evaluation, we follow the procedure from Ku et al. (2023) rate in two criteria: Semantic Consis- tency (SC) and Perceptual Quality (PQ). Both scores are in {0, 0.5, 1}. For SC', the human subject asked to rate the consistency between 1) the edited image and the editing instruction (whether the editing instruction is reflected on the edited image) and between 2) the source image and the edited image (whether the model makes the edit that is beyond the editing instruction). For PQ, the subject is asked to rate on the quality of edited image). We then calculate overall score O VSC PQ that measures the overall quality of the edit. We also calculate the accuracy of the edit, which is defined by the percentage of SC 1 among all examples. We recruit four human raters and require them to evaluate all the editing examples. For LMMs' evaluation, we follow the procedure from Ku et al. (2024) where models (in particular, chose GPT40 and Gemini) are also asked to give SC and PQ scores but on a scale of (0-00. We then normalize the scale to ()-1
5.1 MAIN RESULTS
We provide a qualitative comparison with baseline models in Figure 5. We show the top 4 baselines with OMNI-EDIT on a subset of the DMNI-EDIT- Bench. We provide more results in Figure 8 and Figure 9. Our main results are detailed in Table 3, where we provide the VIEScore and conduct human evaluation on the Top2 baselines and OMNI-EDIT. In Figure 1, OMNI-EDIT demonstrates its capability to handle diverse editing tasks across various aspect ratios and resolutions. The results are notably sharp and clear, especially in the addition/swap task, where new content is seamlessly integrated. This underscores the effectiveness of the Edit-Net design in preserving the original image
8

---

Published as a conference paper at ICLR 2025
Table Main evaluation results on Omni-Edii-Bench. In each column, the highest score is bolded, and the becond-highed is underlined.
Models
VIEScore (GPT4o)
VIEScore (Gemini)
Human Evaluation
PQ
SCavg
Ouog
PQ
SCarg t Oasy
PQ
SCanp"
Oaro
AoCuse
Inversion-based Methods
DitlEdit SDEdit
2.73
2.79
6.09
6.71
2.18
2.78
6.31
2,06
2.48
End End Methods
InstructPix2Pix
7.05
3.04
3.45
6.46
1.88
2.31
MagicBrush
3.53
3.60
6.44
4,56
4,86
6.49
4.33
4.45
0.72
0.52
0.57
0.20
HQ-EEdi
2.15
2.25
0.27
CosXL-Edit
8.34
5.81
6.00
7,01
4,90
4.81
0.82
0.56
0.39
0.35
HTVE
5.35
3,65
3.57
5.84
2.84
3.05
ONNI-EDIT
8.38
6.66
6.98
7.06
5.82
5.78
0.83
0.71
0.69
0.55
A Best baseline
+0.04
+0.85
+0.98
+0.05
+0.92
+0.97
+0.01
+0.15
+0.10
+0.20
generation capabilities of the base text-image generative model. Similarly, in Figure 5, OMNI-EDIT uniquely adds a clean and distinct NASA logo onto a T-shirt. Table 3 corroborates this with OMNI- EDIT achieving the highest Perceptual Quality (PQ) score among the models evaluated.
We highlight the efficacy of our proposed peccialisttto-gneeraa learning framework. Unlike base- line models that utilize a single method for generating synthetic data-often the prompt-to-promp Saihod-Thhs method typically alters the entire image, obscuring task-specific data. In contrast, OMNI-EDIT leverages task-specific data curated by experts, resulting in a clearer task distribution and improved adherence to editing instructions. Both the VIEScore and human evaluations in Ta- ble 3 demonstrate that our method significantly outperforms the best baseline in following editing instructions accurately and minimizing over-editing. For instance, baseline models frequently mis- understand the task intent as illustrated in Figure 5, where the CosXL-Edit model fails to recognize the removal task and incorrectly interprets bird addition
swap between panda and bird.
Lastly, baseline models often produce blurry images on the OMNI-EDIT-Benchh as they are trained resolutions limited to 51xx512 or even 256x256, and they perform poorly on non-square aspect ratios. For example, with 3:4 aspect ratio, the baselines struggle to perform editing. OMNI-EDIT, trained on data with multiple aspect ratios, maintains robust editing capabilities across the diverse aspect ratios encountered on the Omni-Bes Bench, as evidenced in Figure
5.2 ABLATION STUDY
In this section, We provide an ablation study w.r.t importance weighting and EditNet.
Ablation study on the importance sampling. We study baseline that utilizes the same archi- tecture as OMNI-EDIT, but instead of applying importance scoring and filtering, we sample 1.2M examples direetly from the 5M pre-filtering dataset specified in Table 4 and compare it with OMNI-EDIT. As shown in Table 6, we observe a significant decrease in VIEScores for both PQ and SC metrics.
A Alltioo Study on OMNI-EDIT Architecture Design. We conducted an analysis of OMNI-EDIT 's architectural design in comparison to two baseline models: DMNN-EDDIT-Cnntolle and OMNI- and show the result in Table 7, OMNI-EDIr-Controll the SD3-ControlNet architecture trained on the OMNI-EDIT dataset, where the source image serves as the conditioning image for the control branch. OMNI-EDIT EDD.E-CCCCoONNee-eeeeettt of SMNI-EDITE SEDIT-ControING with an added modification: at each layer, we incorporate the text-token output from the control branch into the text-token in the main image generation branch. So this baseline can update the text representation in the main branch but doesn't have the intermediate representation interaction design EditNet.
represents
is a variant
Our analysis, as shown in Figure 11, reveals that SOAN--EDr--Contrllle struggled to accurately capture task intent. This is primarily because the ControlNet branch does not update the text repre- sentation. For instance, in object removal tasks, prompts like "Remove ObjA" are common, yet the original DIT block remains unchanged, causing it to mistakenly generate an image of "ObjA." On the other hand, although OMNI-EDIT-CA EDIT-ControlNeet TextCCorrolll successfully updates the text repre- sentation, it still encounters difficulties in content removal. The substantial VIEScores gap between -oter-Coooonoee-eeeetetec
and OMNI-EDIT in Table 7 underscores the importance of the

---

Published as a conference paper at ICLR 2025
intermediate representation interaction design in EditNet. We also compared OMNI-EDIT with the channel -ainee--ise token concatenation method used in InstructPix2Pix (see Figure 4). Channel-wise To- ken concatenation requires fine-tuning the entire network, which can distort the network's 's original representations. As illustrated in Figure 10, after fine-tuning an SD3 channel-wise concatenation model on OMNI-EDIT training set, the representation of Batman is altered. In contrast, EditNet preserves the original representation of Batman while still learning the object swap task.
6 RELATED WORK
Image Editing via Generation. Editing real images according to specific user requirements has been a longstanding research challenge (Crowson et al., 2022; Liu et al., 2020; Zhang et al., 2023; Shi et al., 2022; Ling et al., 2021; Wasserman et al., 2024; Ju et al., 2024). Since the introduction of large-scale diffusion models, such as Stable Diffusion (Rombach et al., 2022; Podell et al., 2024), significant progress has been made in tackling image editing tasks. SDEdit (Meng et al., 2022) introduced an approach that adds noise to the input image at an intermediate diffusion step, followed by denoising guided by the target text description to generate the edited image. Subsequent methods, such as Prompt-to- Po-Proppt (Hertz et al., 2023) and Null-Text Inversion (Mokady et al., 2023), have focused manipulating attention maps during intermediate diffusion steps for image editing. Other techniques like Blended Diffusion (Avrahami et al., 2022) and DiffEdit (Couairon et al., 2022) utilize masks to blend regions of the original image into the edited output. More recently, the field has seen a shift towards supervised methods, such as InstructP2P (Brooks et al., 2023), HIVE (Zhang al., 2024b), and MagicBrush (Zhang et al., 2024a), which incorporate user-written instructions in an end-to-end framework. Our work follows this direction to develop end-to-end editing models without inversion.
Image Editing Datasets. Due to the difficulty of collecting expert-annotatee editing pairs, existing approaches rely heavily on synthetic data to train editing models. InstructP2P (Brooks et al., 2023) was the first to curate large-scale editing datasets using prompt-to-prome filtering with CLIP scores.
MagicBrush (Zhang et al., 2024a) subsequently improved data quality by incorporating a human- in-the-loop annotation pipeline based on DALLE-2. However, DALLE-2, primarily an inpainting- based method, struggles with global editing tasks such as style transfer and attribute modification. More recently, HQ-Edit (Hui et al., 2024) utilized DALLE-3 to curate editing pairs, although the source and target images lack pixel-to-pxxe alignment, which is critical for preserving fine- fine-graing grained details. Emu Edit (Sheynin et al., 2024) scaled up the training dataset to 10 million proprictary pairs, resulting in strong performance, but the lack of public access their model checkpoints API makes direct comparison difficult. UltraEdit (Zhao et al., 2024) proposed another inpainting -based approach, avoiding the use of DALLE-2 or DALLE-3 for data curation. However, like MagicBrush, it still faces limitations in handling complex global edits. Our work is the first to leverage multiple specialists to significantly expand the range of editing capabilities. Additionally, we are the first to use more reliable large multimodal models, for quality control in the editing process.
7 DISCUSSION
In this paper, we identify the imbalanced skills in the existing end-to-end image editing methods and propose new framework build more omnipotent image editing models. We surveyed the field and chose several approaches as our specialists to synthesize candidate pairs and adopt weighted loss to supervise the single generalist model. Our approach has shown significant quality boost across the broad editing skills. Throughout the experiments, we found that the output quality is highly influenced by the underlying base model. Due to the weakness of SD3, our approach is still not achieving its highest potential. In the future, we plan to use Flux or other more capable base models to see how much further we can reach with the current framework.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arxiv preprint arXiv:2303.08774 2023.
Omri Avrahami, Dani Lischinski, and Ohad Fried, Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 18208-18218, 2022.
James Betker, Gabriel Goh, Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://edn.. openai. com/papers/dall-e-3. pdf. 2(3):8, 2023.
Frederic Boesel and Robin Rombach. Improving image editing models with generative data refine- ment. In The Second Tiny Papers Track at ICLR 2024, 2024.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
2021.
Junsong Chen, YU Jincheng, GE Chongjian, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo. Huchuan Lu, and Zhenguo Li. Pixart-alpha: Fast training of difffusinn transformer for photorealistic text-to-image synthesis. The Twelfth International Conference on Learning Representations, 2024a.
The Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 24185-24198, 2024b.
Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion- based semantic image editing with mask guidance. arXiv preprint arXiv:2210.11422 2022.
Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Cas- tricato, and Edward Raff. Vggan-clip: Open domain image generation and editing with natural language guidance. In European Conference on Computer Vision, pp. 88-105, 2022.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfci Yang, and Zhe Gan. Guid- ing instruction-basee image editing via multimodal large language models. In The Twelfth International Conference on Learning Representations, 2024.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. image worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations, 2023.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfrr Aberman, Yael Pritch, and Daniel Cohen-or. Prompt-to-prompt image editing with attsenattnnioon control. In The Eleventh International Conference on Learning Representations, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Mude Hui, Siwei Yang, Bingchen Zhao, Yichun Shi, Heng Wang, Peng Wang, Yuyin Zhou, and Cihang Xie. Hq-edit: A high-quality dataset for pnrtucttio-bbseee image editing. arXiv preprint arXiv:2404.0999 2024.
11

---

Published as a conference paper at ICLR 2025
Dongfu Jiang, Max Ku, Tianle Li, Yuansheng Ni, Shizhuo Sun, Rongqi Fan, and Wenhu Chen. Genai arena: An open evaluation platform for generative models. arXiv preprint arXiv:2406.04485, 2024.
Xuan Ju, Xian Liu, Xintao Wang, Yuxuan Bian, Ying Shan, and Qiang Xu. Brushnet: A plug-and- play image inpainting model with decomposed dual-branch diffusion. In European Conference on Computer Vision, pp. 150-168. Springer, 2024.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 35:26365-26577 2022.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6007-6017, 2023
Max Ku, Tianle Li, Kai Zhang, Yujie Lu, Xingyu Fu, Wenwen Zhuang, and Wenhu Chen. Im- agenhub: Standardizing the evaluation of conditional image generation models. arXiv preprint arXiv:2310.01594 2023.
Max Ku, Dongfu Jiang. Cong Wei, Xiang Yue, and Wenhu Chen. VIEScore: Towards ex- plainable metrics for conditional image synthesis evaluation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. gust 2024. Association for Computational Linguistics. doi: 18.18653/v wwwp://.....oooo..cooooo URL
Bangkok, Thailand, Au-
https: a/ac///ccaccc /aaaaalaattt www.honnooooooooooooooooooooo logy org/2024 acl lclllong..na.oaooaaoaooooooooooe 663.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Tuste, Sha- hab Kamali, Stefan Popov, Matteo Malloci. Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International journal of computer vision, 128(7):1956-1985 2001
Huan Ling, Karsten Kreis, Daiqing Li, Seung Wook Kim, Antonio Torralba, and Sanja Fidler. Ed- itgan: High-precision semantic image editing. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Xihui Liu, Zhe Lin, Jianming Zhang, Handong Zhao, Quan Tran, Xiaogang Wang. and Hong- sheng Li. Open-edit: Open-domain image manipulation with open-vocabulary instructions. In Computer VisVOn-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp. 89-106. Springer, 2020.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen- Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6038-6047, 2023.
Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun- Yan Zhu. Zero-shot smage-to-image translation. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. 2023 ieee. In CVF International Conference on Computer Vision (ICCV), volume 4172, 2022.
pexels. Pexels, 2024. URL www www. www.pexels.coom we.els.eecoom
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024. URL
12

---

Published as a conference paper at ICLR 2025
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PMLR, 2021.
Adilya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rom- bach. Fast high-resolution image synthesis with latent adversarial diffusion distillation. arXiv preprint 11Xi:22403.12015 2024.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25299 2022.
Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 8871-8879, 2024.
Yichun Shi, Xiao Yang. Yangyue Wan, and Xiaohui Shen. Semantiestylegan: Learning compo- sitional generative priors for controllable image synthesis and editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11254-11264, 2022.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In
International Conference on Learning Representations, 2021.
Linoy Tsaban and Apolinário Passos. Ledits: Real image editing with ddpm inversion and semantic guidance. arXiv preprint 0tti::20070005222 2023.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-pl diffusion features for text-driven maage-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1921-1930, June 2023.
Navve Wasserman, Noam Rotstein, Roy Ganz, and Ron Kimmel, Paint by inpaint: Learning to add image objects by removing them first. arXiv preprint arXiv:2404. 18212, 2024.
Sihan Xu, Yidong Huang, Jiayi Pan, Ziqiao Ma, and Joyce Chai. Inversion-free image editing with natural language. arXiv preprint arXiv:2312.04965 2023.
Fred Zhang. subb--dffuusoorpropps-447777 fattsets/Freehhann//ttto-----r-
2024. URL https://huggingface co/ row=19.
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually anno- tated dataset for instruction-guider image editing. Advances in Neural Information Processing Systems, 36, 2024a.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3836-38477 2023.
Shu Zhang, Xinyi Yang, Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, Stefano Ermon, al. Hive: Harnessing human feedback for instructional visual editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9026-90366 2024b.
Haozhe Zhao, Xiaojian Ma. Liang Chen, Shuzheng Si, Rujie Wu, Kaikai An, Peiyu Yu, Minjia Zhang, Qing Li, and Baobao Chang. Ultraedit: Instruction-based one-grained image editing at scale. arXiv preprint arXiv:2407. AiV.22407.05282, 2024.
13


Reviews:
[{"review_id": "j4z43I9do4", "reviewer": "Reviewer_txht", "summary": "OMNI-EDIT is an image editing model designed to address the skill imbalance issues present in existing instruction-based image editing methods. It learns by combining supervision from seven different expert models and utilizes InternVL2 for data scoring. Additionally, this approach introduces a diffusion-Transformer architecture named EditNet.", "questions": "1. Since the dataset is divided into seven types of tasks, would it be possible to categorize the test set similarly? This would allow for a more detailed evaluation of the model's performance across different editing tasks and facilitate an analysis of performance differences and underlying reasons.\n\n2. The relatively small size of the Omni-Edit-Bench dataset may limit the reliability and generalizability of model evaluation. It is recommended to assess performance on additional benchmark datasets. For example, are there results available on the Emu Edit test set?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors collected a high-resolution dataset covering seven types of tasks, annotated through multiple expert models, and used InternVL2 to score the dataset. \n\n2. And they introduced EditNet, which enhances image noise control capabilities compared to ControlNet.\n\n3. In the Omni-Edit-Bench benchmark, OMNI-EDIT outperformed existing baseline models across multiple evaluation metrics.", "weaknesses": "1. Lack of testing of GPT4-o generated CONFIDENCE SCORE, LLM may be subjective and biased\n\n2. Limitations of the benchmarking dataset: The Omni-Edit-Bench dataset used in the paper is small (only 62 images), which may limit the reliability and generalization of the model evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CARZyCz7TV", "reviewer": "Reviewer_u8AW", "summary": "The paper introduces OMNI-EDIT, an advanced image editing model addressing current limitations such as biased training data, noise, and fixed aspect ratios. It uses supervision from multiple specialist models, improved data quality via importance sampling with large multimodal models, and a new architecture called EditNet for better editing performance. OMNI-EDIT supports various aspect ratios and outperforms existing models in evaluations.", "questions": "1. How does this paper differentiate itself in terms of contributions from the CVPR 2024 paper [1], given their similar focus on generalist instruction-guided modeling?\n\n2. It would be better to compare the proposed method with the generalist modeling approach in [1].\n\n3. Can the authors provide more detail on how the support for multi-size and multi-resolution inputs was implemented? What specific innovations, if any, were introduced to achieve this?\n\n4. How does the training data size affect the performance of OMNI-EDIT? \n\n5. Could the authors elaborate on the training process and datasets used for the specialist models, such as the image-inpainting model?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "--The paper introduces OMNI-EDIT to address limited editing capabilities, poor data quality control, and lack of support for varying resolutions, which are critical point in image editing method.\n\n--The use of importance sampling with high-quality data scoring via distilled multimodal models (such as InternVL2) demonstrates a meticulous approach to data curation\n\n--The paper is clearly structured, with each contribution outlined and explained in detail. The authors provide a logical flow that makes it easy for readers to understand how OMNI-EDIT was developed.", "weaknesses": "1. The primary concern with this paper is its similarity to a CVPR 2024 paper [1], which proposed a generalist modeling interface that formulates most vision tasks as instruction-based editing. In contrast, the proposed paper is simpler and more direct as it focuses on building a generalist model specifically for editing tasks rather than all vision tasks. The training approach is similar to [1] which also involves generating data using current specialist models. This reduces the contribution of leveraging specialist models to train the generalist model. Additionally, the experimental results lack a comparison with [1], which would be necessary to highlight differences.\n\n2. Although the paper claims that the model supports inputs of varying sizes and resolutions, there does not appear to be a detailed explanation of this in the methodology or experimental sections. This is crucial information, as it raises questions about how this feature is implemented and whether it involves any novel techniques.\n\n3. The training data for Omni-Edit consists of 505K samples, all generated by specialist models and selected using importance sampling. It would be beneficial to discuss whether different training data sizes affect the model’s results.\n\n4. Some specialist models mentioned training an image-inpainting model, but there seems no description of the training data for specialist models.\n\n5. The overall framework is complex, requiring the training of specialist models, a score prediction model for importance sampling, and finally the Omni-Edit model with all the collected data. And such a process makes it difficult to dynamically include more editing tasks.\n\n6. There seems to be unnecessary \"5\" in Lines 361 and 362.\n\n7. Typo: In Line 246, \"for the our editing model\".\n\n【1】InstructDiffusion: A Generalist Modeling Interface for Vision Tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V7g1wG7DnH", "reviewer": "Reviewer_KWLC", "summary": "This paper presents a new multi-task image editing model. The key contributions of this paper includes (1) framework for learning generalist from data generated from a set of specialist models, (2) scoring of data generated from specialist using a finetuned VLM model, (3) control-net like architecture with DiT. Authors conducted experiments to compare multiple recent methods and did a nice ablation study analyzing components of the proposed model.", "questions": "1. The authors rely on GPT 4o knowledge to build the scoring model. How robust is the scoring model with InternVL in ranking the quality of the data point, how's the accuracy level in general? \n2. Could the scoring model be further improved by finetuning with human labeled GT data?\n3. It seems the inpainting specialist model cannot handle shadow/reflection. Any idea to improve it?\n4. Authors rely on half synthetic data for training, is the trained model affected by the inpainting artifact caused by the synthetic generation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to read.\n2. The proposed method pipeline sound reasonable.\n2. Experiments are reasonable with comparisons to many recent methods, and good ablation study.", "weaknesses": "1. Lacks novelty. The most significant novel idea of this paper is to train specialist editing models to generate training data for training unified generalist editing model. This idea is quite straightforward and I think many existing multi-task generation models mixes data from different sources, either existing data, or data generated by some models (can be specialist) for joint training from multiple data sources.\n2. Results quality. The visual results are not impressive (shadow/reflection is left after removal, inpainting/visual artifact in several results, cartoonish object generated given photo-realistic background, change color instruction lead to texture changes, etc.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DOeniFQGZ5", "reviewer": "Reviewer_dYun", "summary": "The paper introduces an effective editing image training data generation process and a new model structure EditNet for general image editing of various image resolutions. The paper proposes to train and utilize seven different specialist image editing models to generate training data and use LLM to filter good-quality images, the paper also proposes a modified model backbone for general image editing capability training. The proposed method is trained on the OMNI-EDIT training dataset and experiments are conducted on the proposed OMNI-EDIT-Bench.", "questions": "1. I have a question about the generalization ability of the OMNI-EDIT. Since the seven specialist models are trained on their pre-defined tasks, will OMNI-EDIT have the ability to generalize to editing instructions beyond these seven (for example move a person from right to left)?\n2. Ideally the specialist model should generate close to perfect edited data, but in real scenarios, for some tasks, a trained specialist model may still not generate great data. Do you encounter this situation where a large portion of the generated data are filtered leading to a small proportion in the final training dataset? Will this data unbalancing affect the model's performance?\n3. Can we have some visualization examples of your proposed training data and benchmark?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed training data generation process using seven different specialist models and a better filtering method is novel and reasonable.\n2. The modification from simple ControlNet + DiT to the proposed EditNet especially on accessing intermediate presentations and updating text presentations is interesting.\n3. The results are strong in both quantitative metrics and qualitative assessments.\"", "weaknesses": "1. I would recommend updating Figure 3, the color of the frozen DiT model is near the same as the white background making readers difficult to comprehend.\n2. It would be better if the paper could also include a walkthrough caption other than the only training pipeline image in Figure 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4ACT4q9ImM", "reviewer": "Reviewer_QAfp", "summary": "This paper builds an instruction-based image editing method using synthetic data from 7 specialists. The major contributions include: 1) Curated an (image, editing instruction) dataset for seven editing tasks using the importance sampling strategy; 2) Proposed a ControlNet-like image-editing architecture to achieve these tasks. Experiments show that Omni-Edit outperforms existing approaches after training the proposed model on the curated dataset.", "questions": "1) Will the dataset be released?\n\n2) When constructing the training data for object removal, how to ensure the generated content in the removal area will be background instead of a new object?\n\n3) Could the authros provide visualized examples of (src image, tgt image, instruction) in the training set for each task?\n\n4) In Tab. 1, It is unclear how different methods are evaluated on their image editing capabilities. Were user studies conducted to get the star ratings? Or are they rated by some automatic metrics? \n\n5) MagicBrush is a human-annotated dataset that also includes 7 tasks. How does the data quality of the synthetic dataset introduced in this paper compare with MagicBrush? Is there a quantitative evaluation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) The curated dataset greatly expands the instruction-based image editing datasets with multiple specialists.\n\n2) The paper standardizes the procedure to automatically generate the trainig data for 7 image editing tasks.\n\n3) The proposed method performs well on both automatic metrics and user studies.", "weaknesses": "1) The ablations are not comprehensive. Since the authors claim the new dataset and their model are two separate contributions, they could compare with existing methods trained on their dataset and their method trained on the existing dataset to validate each contribution. For example, the authors can report the results of CosXL-Edit trained on Omni-Edit-Bench.\n\n2) L433 states that while prior methods struggle to perform editing for non-squared images, the proposed method can handle multiple resolutions. However, only two visualized examples are provided in the paper, which is not convincing. I suggest the authors report quantitative results on non-squared images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)