HQ-Edit: A High-Quality Dataset for Instruction-based Image Editing
Authors: Mude Hui
Siwei Yang
Bingchen Zhao
Yichun Shi
Heng Wang
Peng Wang
Cihang Xie
Yuyin Zhou
Venue: ICLR 2025 Poster (2025)
Keywords: Image-to-image translation
Image Editing
Primary area: datasets and benchmarks

Abstract: This study introduces HQ-Edit, a high-quality instruction-based image editing dataset with around 200,000 edits. Unlike prior approaches relying on attribute guidance or human feedback on building datasets, we devise a scalable data collection pipeline leveraging advanced foundation models, namely GPT-4V and DALL-E 3. To ensure its high quality, diverse examples are first collected online, expanded, and then used to create high-quality diptychs featuring input and output images with detailed text prompts, followed by precise alignment ensured through post-processing. In addition, we propose two evaluation metrics, Alignment and Coherence, to quantitatively assess the quality of image edit pairs using GPT-4V. HQ-Edits high-resolution images, rich in detail and accompanied by comprehensive editing prompts, substantially enhance the capabilities of existing image editing models. For example, an HQ-Edit finetuned InstructPix2Pix can attain state-of-the-art image editing performance, even surpassing those models fine-tuned with human-annotated data.

Published as a conference paper at ICLR 2025
HQ-EDIT: A HIGH-QUALITY DATASET FOR INSTRUC- TION BASED IMAGE EDITING
Mude Hui Siwei Yang', Binnchhn Zhao?, Yichun Shi Heng Wang", Peng Wang', Cihang Xie Yuyin Zhou'
"equal contribution
University of California, Santa Cruz
University of Edinburgh
BytteDance
Ours 17.90
(n) Zoom the ant add avouflakes tilling amound
(h) Alber her Tair oolor
512
HIVE
Megidinah
HC-ESB
(c) Replace Ameovr rainfoned backegoondd with the indernater scenery of the pamol' puoition and wings duprcs flying.
(1) Replace the eliver teaper ceraii white polterred leagat similer Kooden fondli and musching
(d) Compurison betwore Inetrcsfiiulis HIVE, Magifiimah
adjust the
l1Q
commic
Figure 1: (a) (d): example images and edit instructions from HQ-Edit. (e): compare the dataset quality between our HQ-Edit and existing ones. Note that "Alignment" and "Coherence" are our newly developed metrics (introduced in Sec. 3.4) for measuring image/text qualities.
ABSTRACT
This study introduces HQ-Edit, a high-quality iustruction-basee image editing dataset with around 200,000 edits. Unlike prior approaches relying on attribute guidance or human feedback on building datasets, we devise a scalable data col- lection pipeline leveraging advanced foundation models, namely GPT-4V and DALL-E 3. To ensure its high quality, diverse examples are first collected on- line, expanded, and then used to create high-quality diptychs featuring input and output images with detailed text prompts, followed by precise alignment ensured through post-pun post-processing. In addition, we propose two evaluation metrics, Align- ment and Coherence. quantitatively assess the quality of image edit pairs using GPT-4V. HQ-Edit's high-resolution images, rich in detail and accompanied by comprehensive editing prompts, substantially enhance the capabilities of existing image editing models. For example, an HQ-Edit finetuned InstructPix2Pix can attain state-of-the-an image editing performance, even surpassing those models fine-tuned with human- aanaan-nn data.
INTRODUCTION
The recent advancements in text-to- image generative models (Rombach et al., 2022 Ramesh et al.) 2022, Gu et al., 2022 Saharia al. 2022 Huang et al. 2024) have catalyzed a new era in diverse real-world applications ranging from advertising and photography to digital art and movie production. Among these generative models, applications of domain-specific image conditioned generations (Ruiz
et al., 2023 Ye et al. (023) Wang & Shi| 2023 Hu al. 2023), and multi-modal non-specific
generation methods (Pan et a. 2023 Sheynin et al 2023: Wu et al 2023) have gathered significant
attention.

---

Published as a conference paper at ICLR 2025
Our work concentrates on applications of highly accurate, general instruction single image editing without relying on external attribute guidance, as proposed in previous studies (Avrahami et al. 2022; Hertz et al. 2022 Ling et al. 2021 Wallace et al. 2023 Shi et al. 2022). We identity that this particular challenge has not been adequately addressed in the literature yet. To the best
of our knowledge, one of the major hurdles in training an instruct-based image editing model lies in the limited availability of high-quality datasets pairing editing instructions with corresponding images. This challenge was best tackled by the seminal work InstructPix2Pix (Brooks et al. (023). Specifically, it first leverages GPT-3 (Brown et al. (020)) to generate both an instruction and an edited image caption based on a given image description; then, it applies Stable Diffusion (SD1.5) (Rombach) et al.] 2022) and Crompt-oo-romp (Hertz et al. 2022) to create the paired input and output images. However, their underlying models, namely SDL.5 and GPT-3, are outdated compared to current state- of-the-art counterparts such as DALL--E 3 and GPT-4. Consequently, these models produce images with lower resolution and suboptimal edit-image alignment. Subsequent studies also attempted to improve it via incorporating human feedback (Zhang et al. (2023) or segmentation masks (Chakrabartyy et al. 2023 Zhang et al. 2024), yet the generated data continue to exhibit one or more of the aforementioned issues, as showcased in Figure
In this work, we aim to leverage the ability from the best text-image models, ie.. DALL-E 3 (OpenAl 2023a), GPT4 & GPT4V (OpenAl 2023b), to build a high-quality dataset for improving the image editing datasets. Ideally, in case of accessing the model weights, it should provide high-resolution images that offer rich detail, both in their visual content and the accompanying instructions; Also, should provide more precise alignment between textual instructions and image pairs, ensuring edits are applied as directed while maintaining fidelity in areas not subject to modification.
However, only with the access to their APIs, in this study, we discover a way of pair image generation with DALL-E 3 based on prompt-engineer, which enable similar Prompt-to-Pronee to-orompt process, yielding high-quality editing image pairs, which we name as HQ-Edit. HQ-Edit provides a significant leap forward, featuring high image resolutions of approximately 900 x 900 pixels- double that of existing datasets, and comprises around 200,000 detailed edit instructions. Moreover, unlike prior approaches relying on attribute guidance or human feedback, HQ-Edit is symthetically generated through scalable pipeline that harnesses the image text understanding capabilities of powerful foundation models of GPT-4V and DALL-E
Our data curation process comprises three key steps: Expansion Generation Post- post-processing Firstly, in the Expansion phase, we extract seed triplets with high -eresitt-consstinngg of input/output image descriptions along with edit instructions- from online sources. Subsequently, we leverage GPT-4 to expand these initial triplets into around 100,000 instances, ensuring the comprehensive diversity of edit instructions. In the subsequent Generation phase, the seed triplets are processed by GPT-4 to merge and refine into detailed diptych prompts for DALL-E 3, creating diptychs with input and output image pairs displayed side-by-side. Note this diptych-based prompting design is motivated by the finding that, compared to generating input images and output images separately, generating diptychs generally exhibits superior quality, with better alignment and consistency in edit-irrelevant areas. Lastly, the generated diptychs and refined prompts undergo poxt-processing to ensure precise alignment between the paired images and their corresponding instructions. Specifically, 1) each diptych is decomposed into paired images, which undergo warping and filtering to ensure correspondence; 2) the instructions are refined using rewritten instructions from GPT-4V; and 3) the inverse-edit instructions are also generated, allowing for the transformation of output images back into their input counterparts.
On top of HQ-Edit, we introduce two metries, Alignment and Coherence, to comprehensively and quantitatively evaluate the quality of image edit pairs. The first metric, Aligument, checks for semantic consistency with the edit prompt, ensuring accurate modification of mentioned objects while preserving image fidelity. The second metric, Coherence, evaluates the edited image's aesthetic quality, including lighting and shadow consistency, style coherence, and edge smoothness. Extensive empirical results show that our synthetically created HQ-Edit can even surpass human-annotade data in enhancing instruction-basee based image editing models. For example, the HQ-Edit finetuned InstructPix2PPx model substantially outperforms its vanilla version, achieving a 12.3 increase at Alignment, and a 5.6 64 enhancement at Coherence.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORKS
Text Guided Image Editing Model Text guided image editing models have been extensively discussed recently. Prompt2Prompt (Hertz al. 2022) modifies words in the original prompts to perform both local editing and global editing by cross-attention control. Imagic (Kawar et al.) 2023) optimizes a text embedding that aligns with the input image, then interpolates it with the target description, thus generating correspondingly different images for editing. DiffEdit (Couairon et al.) 2022) locate edit position based on text (generate mask), and limit diffusion model to generate the mask area. An important type of Text Guided is the instruction, which describes where, what and how an image should be edited. Instruction- without requiring elaborate descriptions or region masking, and enables users to modify images more easily and flexibly. InstructPix2Pix (Brooks et al. 2022) is the first instruction-based image editing model, by fine-tuning the Stable Diffusion (Rombach et al. 2022) on dataset of image editing examples, which generated by GPT-3 (Brown et al. 2020) and Prompt2Prompt. Subsequent work,
image editing model will follow the instruction
such as HIVE (Zhang et al., 2023) and Magicbrush (Zhang et al. 2024). have focused on improving the quality or quantity of the dataset.
Instruction- Image Editing Datasets Since collecting high-quality open data for image edit- ing can be challenging, early approaches construct datasets by manually labeling image pairs (Zhang al.) 2024). While this ensured degree of quality, inherently restricted the scale and diversity of the dataset. For example. Magicbrush (Zhang et al., (2024) contains about only 10,000 edits, and predominantly focuses on object-level transformations, largely overlooking global edits like style or weather changes. On the other hand, there have been endeavors to synthesize large-scale datasets. For example, InstructPix2Pix (Brooks et al., 2023) leverages GPT-3 and Prompt2Prompt (Hertz et al.) 2022) to generate editing pairs, and HIVE (Zhang et al. 2023) introduces reinforcement learning from human feedback to align the data with human expectations. However, these synthetic data often have the drawback of low quality and inaccurate editing, resulting in such trained image editing models out- putting low-quality images and deviating from the actual edit instructions. FaithfulEdits (Chakrabarty et al. 2023) attempts mitigate these issues by using impainting techniques, followed by filtering process involving VQA models. Yet, this method tends to underperform, particularly in global edits requiring extensive image modification, like style transfer.
Different from existing approaches, we leverage the latest foundation models, GPT-4 and DALL-E 3, to generate high-quality image editing pairs at scale. We also introduce additional enhancements, egg. using GPT-4V to rewrite the edit instruction to align with the images more closely.
3 HQ-EDIT DATASET
The process of collecting HQ-Edit, illustrated in Figure comprises three phases. Initially, triples of input/output image descriptions and edit instructions are expanded into 100,000 instances during the Expansion phase (Section 3.1pp Subsequently, these instances are refined into detailed prompts for DALL-E to generate diptychs in the Generation phase (Section 3.2). Finally, alignment and refinement occur in the Post-processing phase (Section 3.3
3.1 EXPANSION
As in Figure 2 we first collect small yet representative dataset comprising 203 samples from online sources as the seed. To ensure aligmment between the text descriptions and image pairs, we manually revise the descriptions based on the disparities content. Additionally, we include 90 samples from the Emu Edit (Sheynin et al. (023) test set. We refer to these 293 samples as seed triplets, with each triplet comprising input/output image descriptions along with corresponding edit instructions.
To increase its size, we follow the pipeline in Self-instruct (Wang et al., 2022). which applies large language models on a small set of seed samples to generate a large volume of expansions that are both high in quality and consistent with the seed structure. Specifically, we utilize GPT-4 to expand this initial set of 293 seed triplets into around 100,000 instances, ensuring a thorough representation of diverse image editing scenarios. This strategy not only broadens the scope of edit instructions but also leverages GPT-4's knowledge to enrich the diversity and detail of image descriptions and edit instructions.
3

---

Published as a conference paper at ICLR 2025
Step #1: Expansion
Step #2: Generation
Step #3: Post-Processe
Output
Output
Input
Output
Edit
Seed
input
Outpur
Edit
GPT-4
Diplych
GPT-4
GPT-4V
DALL-E 3
Imput
Outputt
Edit
Rewnitte
Rewrltttn Inwerse Edil
Expanded
Distych
Input
Outour
Input
Imput
Outpur
Image
Rewritton Edit
Processer
Input
Input Image Description
Inpuc
Outpul
Output Image Description
Image
Image
Edit
Edt Instruction
Figure 2: Our method consists of three steps: (1) Expansion: Massively generating image descriptions
and edit instructions based on seed samples using GPT-4. (2) Generation: Generating diptychs using GPT-4V and DALL-E according to image descriptions and instructions. (3) Post-Processing: Post- process diptychs and edit instructions with GPT-4V and other various methods to produce image pairs and further enhance the quality of the dataset in different aspects.
Table 1: An example of the diptych prompt.
Input/Outpul/Edit
Diptych Prompt For DALL-E
Generate a diptych with two side side-by-sioe by-side images. On the left, depict vibrant, narrow urban alley teeming with colorful graffiti Details should include assorted tags and street art in various styles, with depth indicating the alley stretches far back. Miscellaneous urben elements like dumpster, stray cat, and fire escape ladders should present, and subtle sunlight cast soft shadows, indicating day time setting. On the right, replicate this scene exactly but convert the image into high-contrast black and white with stark lighting to enhance textures and shadows, and accentuale the details of the graffiti, giving edgy, gritty aesthetic. Each element from the left image must be recognizable in monochrome, especially the contrasts between the shaded areas and the illuminated ones created by an overhead midday light.
Input: graffiti-covered urban alley
walls.
Edit:
present the photo with high- contrast black and white effect
Output: high-contrast black and white image of a graffiti covered alley
3.2 GENERATION
Upon acquiring the essential instructions and image descriptions from Expansion, the next step is to generate paired images that align with the instruction data. We hereby employ DALL-E 3 (OpenAl 2023a), sate-of-the-arr image generation model capable of producing high-resolution images based on textual descriptions. However, DALL-E 3 is not originally designed for instruction-based image editing, and therefore cannot directly produce paired images. Thus, devised workaround by creating diptychs consisting of input and output images side by side, followed by post-processing (Section (3.3) reconstruct paired images. Interestingly, we note that generating imput and output images together in diptych form, rather than separately, significantly enhances the relevance and correspondence between image pairs. outlined Figure each triplet fed GPT-4 to form diptych prompt for DALL-E to generate diptych. Moreover, to refine the diptych prompts and improve consistency between image pairs, GPT-4 also utilized to elaborate further on the prompts. For instance, basic description like "an elder Asian woman" can be enriched into "an elderly East Asian woman with wrinkle-lined skin and white hair pulled back neatly, wearing a traditional gold silk hanbok". This enrichment adds complexity to the prompts and subsequently to the generated diptychs. An example of the enhanced diptych prompt is shown in Table [ Overall, this process yields 98,675 data samples comprising onput-output text pairs, edit instructions, and diptych images.
4

---

Published as a conference paper at ICLR 2025
MAZ After Warping Figure 3: The effect of decomposing and warping in image post-processing.
Diptych
(())))))))) After Decomposing
3.3
POST-PROCESSING
After generating the diptych and its corresponding prompt, we implement tailored post-processing stage aimed at decomposing the diptych back into paired images and further refining the quality of both image pairs and text instructions. This process involves two key steps: image post-processing and instruction refinement.
Image Pust-processe
The goal of image post-processing is to decompose the diptych into paired images well as to improve their correspondence. We later use corresoondence quality control to (optionally) filter our training set. It consists of three steps: Decomposing, Warping, and Filtering:
Decomposing horizontally separates diptychs generated by DALL-E 3 into image pairs using a retrained object detection model. Specifically, we train YOLOv8 (Reis et al. 2023) on 3,000 diptych images, where human annotators mark bounding boxes for both left and right segments.
2. Warping aligns the decomposed paired images based semantic correspondence between input and output images. We employ DIFT (Tang et al.j ) an advanced diffusion- establish pixel-wise semantic correlations between paired images. By leveraging semantic correspon- dence, we determine the homography, which maps pixels from the input image to corresponding pixels in the output image, facilitating the precise alignment between them. An example of warping in improving alignment between input and output images is illustrated in Figure 3
model, to
Filtering assesses image distortion post-warping and retains those with minimal distortion for training purposes. When the dimensions of the image before warping are denoted as (w1, W2, h1, h2), and those after warping as U33, W4. h3, h4 ). any image undergoing more than a 50% deformation on any single dimension before and after warping, such w1 < 0.5 W3, is filtered out. Note that this step is applied exclusively to the InstructPix2Pii fine-tuning process for selecting high-quality training samples from our HQ-Edit dataset.
Instruction Refinement While image post-processing improves alignment between input and output images, further refinement vital to ensure that editing instructions are well-aligned with image pairs. First, by leveraging GPT-4V, we rewrite edit instructions based on the differences between input and output image details, thereby enhancing the detail of the text descriptions. Rewriting not only helps fix discrepancies in existing descriptions but also includes visual differences between background objects, which are often omitted in the original text descriptions. Additionally, we use GPT-4V directly generate inverse-edit instructions for transforming output images back input images. This simple strategy can effectively double the instruction count but at marginal cost.
Overall, as demonstrated in Figures the application of rewriting and inversion techniques substan- tially increases both the length and diversity of edit instructions. This enrichment leads to a dataset enhanced with a wider range of composite operations, resulting in broader distribution of instruction lengths. Our edit instructions not only have a larger average length but also display a more expansive distribution, underscoring the effectiveness of these augmentation strategies.
3.4
DATA QUALITY ASSESSMENT
Diversity of Edit Instruction Unlike previous studies which either focus on global or object editing (Brooks et al., 2023 Zhang et al., 2023 2024), our editing operations span a broad spectrum, encompassing both global operations- such as altering the weather, modifying the background, and transforming the style-and and local operations, which include a variety of soject-based editing. Figure provides providess comprehensive overview of the keywords in the edit instructions of Edit. This diversity of edit instructions indicates that our HQ-Edit incorporates a vast range of editing tasks, thereby demonstrating its extensive coverage of potential editing operations.
5

---

Published as a conference paper at ICLR 2025
Distribution Instruction HOBd
Distribution of Edit Instruction Length Instructhi22h
120
140
Lenghh
Length
Figure 4: The histograms illustrate the distribution of edit instruction lengths for HQEdit and InstructPix2Pix. Overall, HQEdit exhibits a more uniform and dispersed distribution, indicating broader diversity in the length of its instructions.
aey
mmatiional
oai
bleig
wopon
remove
back
original
change
Transform object
replace
change
Change weather
giveel sbyle
Wy SM
(10.11%)
Change color (4.9%%
1A
Remove
abject tl 85%
Add object (3..72 (1
remive
change
calor
ade
Figure 5: Distribution of edit types and keywords in instructions. The inner ring depicts the types of edit instructions and the outer circle shows the frequency of the instruction keywords.
Alignment and Coherence To quantitatively evaluate the quality of editing, introduce two formal metrics: Aligument and Coherence. The Alignment metric assesses the semantic consistency of edits with the given prompt,utilizing different criteria for various types of edits, such global editing (e.g.. stylization) and local editing (e.g., object removal), ensuring accurate modifications while preserving fidelity in the rest of the image. On the other hand, the Coherence metric evaluates the overall aesthetic quality of the edited image, considering factors such as lighting and shadow consistency, style coherence, and edge smoothness. These metrics, performed using GPT-4V, produce scores from 0 to 100, with higher scores indicating better alignment or coherence.
We present evaluation example results with varying Alignment scores in Figure and example images showing different Coherence scores in Figure both suggesting a potential (positive) correlation with human perception. Detail of the evaluation can found at supplemental material.
To further validate the effectiveness of our proposed metrics, as detailed in Section 4.1 we conducted a human evaluation on 1,651 image pairs generated by DALL-E Notably, our metric exhibited much higher correlation to human preference compared to the popular CLIP score.

---

Published as a conference paper at ICLR 2025
Input Image
"Abiumentt: 40
"Aligmmeat':
"Alignment": 100
Figure 6: Examples of different Alignment. Instruction: 'dd comet in the sky." Editing follows the edit instruction more accurately as Alignment increases.
Coherence: 20
Coherence 40
Coberence 60
Coberence 80
Coberence 100
Figure 7: Examples of different Coherence. As the Coherence score increases, the image quality improves significantly.
Comparisons To demonstrate the supe- rior data quality of HQ-Edit compared to
Table Comparison between different editing datasets.
Datasct
Alignment
Coherence
existing public editing datasets, we con-
InstrucdPix2Pix (Brooks et al.
2023
68.29
83.35
duct evaluations on 500 randomly sam- pled data points from InstructPix2Pix, HIVE, MagicBrush, and HQ-Edit (Ta- ble 2) assessing their Alignment and Coherence metrics. HQ-Edit significantly outperforms all others with Alignment and Coherence scores of 92.80 and 91.87, respectively, compared to InstructPix2Pix (68.29 and 83.35), HIVE (9.85 and 84.65), and MagicBrush (80.61 and 65.42), demonstrating its superior data quality.
HIVE (Zhang et at 2023
9.85
84.65
MagicBrush (Zhang al. 2024 HQ-Edit
80.61
65.42
92.80
91.87
EXPERIMENTS
Baselines. We conducted comparative analysis with existing open-source text-based image editing methods, iee. DiffEdit (Couairon et al. 2022), Imagic (Kawar al., (202)), Promptnnverse (Mokady al. 2022). HIVE (Zhang et al. 2023 MagicBrush (Zhang al. [024). To ensure reproducibility and fairness, we utilized default byperparameters from the official implementations. Our testing set comprised the 293 samples mentioned in Section 3.1 with all input images generated by DALL-E
based on the input image descriptions.
Implementation details. We choose InstructPix2Pix (Brooks al., 2023) our default model, and use HQ-Edit to fine-tune it. During training, we set the image resolution to 512, total training steps to 15000 on A 100 GPUs, learning rate to 5e-5, and conditioning dropout prob to 0.05. During the editing, set the image guidance scale to 1.5, the instruct guidance scale to 7.0, and the number of inference steps to 20.
4.1 HUMAN EVALUATION
In this section, conduct human evaluation assess whether GPT-based metrics good enough and necessary for evaluating the performance of image editing.
Human Evaluation of Alignment For alignment, we use CLIP Directional Similarity (CLIP dir) RRad ford et al. 2021 as the comparison metric, CLIPair shares the same purpose as Alignment metric: verifying the changes in the image are those required by the instruction. The human evaluation is obtained on 1,651 image pairs generated by DALL-E 3. We utilize Gradio (Abid et al.) 2019) to create the evaluation platform. For each assessment, edit instructions, the input/output image pairs, and their corresponding descriptions are provided for evaluation. We categorize whether the change
7

---

Published as a conference paper at ICLR 2025
between the input image and the output image matches the corresponding edit instruction into the following 5 levels: 1. Totally not related; 2. Not following edit, but there is some relation between the two images; 3. OK image pair. but not following the edit instruction; 4. Good image pair, but need to modify the edit instruction for better alignment; 5. Perfectly follows the edit instruction.
We report the results in Table 3] We Pearson Correlations analyze the correlation between Alignment and CLIPdir to Human Evaluation Score. We can observe that the Alignment metric sig- nificantly surpasses CLIPair in accurately evaluating the fidelity of editing instructions reflect the alterations between the input and output images. This notable discrepancy underscores significant limitation of CLIIP dirs namely inability to comprehensively grasp the nuances of the editing process and accurately retain fidelity to the intricate details of the images.
Table 3: Correlation compari- son of Alignment and CLIP with Human.
Alignment CLIP dir
0.3592
-0.
Human Evaluation of Coherence Since traditional metrics for image quality evaluation are lacking, we explored two model-based evaluations: evaluating with 1) general large models like GPT-4 with stronger base capabilities but less specialization or 2) fine-tuned expert models that excel in specific domains despite weaker general capabilities. Specifically, we select IewaeR (Xu et al. 2023).
Table 4: Correlation compari- son of Coherence and Reward Score with Human.
Reward Score
Coherence
Raw
Sigmoid
0.27
0.12
0.10
which measures the output image quality, as our expert model comparison metric (Reward Score). As ImageReward's score is unbounded, we normalize it using Sigmoid. We perform human evaluation with 160 image pairs from HQ-Edit - human evaluators are tasked with determining which image in each pair is better. "Image is better". : "Tic" and Image 2 is better" are encoded -1, 0 and 1. We then compute the Pearson Correlation following the Human Evaluation of Alignment.
As shown in Table 4 the proposed Coherence metric is significantly more aligned with human preference than the Reward Score. This suggests that, as the development of reward models for image generation and image editing is still at preliminary stage, using mature and well-tester general-
general-purpose purpose model (like GPT-4V) is more valid and reasonable at this stage.
Table 5: Per model variance of Alignment and Coherence on HQ-Edit test set.
Alignment
IestruciPix2Pix
RIVE
MagicBrush HQ-EdR
Colerracc
Instr
HIVE
MagicBrush
HQ-EaR
InstruciPaPPPx
0.00
12.56
14.27
12:31
InstroctPiPPPi
0.00
27.44
38.72
25.90
HIVE
12.56
0.00
9.87
9.21
HIVE
27
32.58
26.02
MagieBrush
14.27
9.87
0.00
10.51
MagicBrush
38.72
32.58
0.00
36.31
HQ-+xlit
12.51
9.21
10.51
DO00
HQU-blil
25.99
26.02
36.31
0.00
(a) Alignment variance
(b) Coherence variance
Differentiation GPT-Based Metrics There still remains question of whether GPT-4V truly differentiates between scores that are just few points apart. To this end, we calculate the per-image variance of Alignment and Coherence between InstructPix2Pix, HIVE, MagicBrush, and HQ-Edit and report the averaged per-image absolute variance in Table We can observe that the per-image variance of Aligmment scores are around 10 and Coherence scores around 30, indicating metrics for each image are differentiated by large margin. This evidence, therefore, strongly support that our GPT metrics are valid and can accurately tell the quality differences between images.
4.2 QUANTITATIVE EVALUATION ON GENERATED IMAGES
Evaluation on HQ-Edit Test Set We first compare our model with existing text-based image editing models in Ta- ble using Aligmment and Coherence. Compared to other methods, our model performs best in all metrics. Specifi- cally, our model outperforms the vanilla InstructPix2Pix. achieving a notable in- crease of 12.30 in Alignment (from 34.71 to 47.01) and 5.56 in Coherence (from 80.52 to 16)1 Furthermore, it is noteworthy that our model surpasses HIVE and MagicBrush, two methods
Table 6: Comparison with existing text-based image edit- ing models on HQ-Edit test set.
Method
Alignment 1
Coherence 1
Imagic (Kwar el al. 2023
1.50
63.58
DiflEdit (Couarrnn el al 2022
21.53
81,81
PromptInverse Mokady ct al.
2022
22.82
80.85
InstructPixPPix (Bookss et al. /Base XXL.
20233
34.71
80.52
35.03
84.45
HIVE (hang al.
2023
wecomuntonnal
40.34
82,93
w/weighted
40.68
84,94
MagicBrush (hang et al. HQ-Edit
2024
43.77
84.19
47.01
$6.16
8

---

Published as a conference paper at ICLR 2025
Table 7: Quantitative Evaluation of Instruct-
Table 8: Quantitative Evaluation of Instruct-
Pix2Pix Based Method on HQ-Edit Test Set.
Pix2Pix Based Method on Emu Edit Test Set.
C1.PPs
DINO
Raw
Signood
CL.P,.
DINO
Raw
Sigmood
Sooce
07444
07355
0.7232
0.1675
-3.54
0.40
00775
0.8506
0.787
0.2223
-2.05
01.2
HIVE
0.0920
0.8633
0.8851
0.4646
-0.99
0.48
lIVE
0U0527
0.8567
0.7855
0.1978
2.07
0.12
MagicBoush
0.110
0.175
0.8346
0.2785
-0005
0.49
MagicBaush
0.1011
08526
0,278
0.2516
-2.13
0.11
HQ- Edit
0.1351
0.3246
0.9692
4003
HO-Edit
0.1067
0,8588
0.2251
0.13
Input
InstructPix2Pix
HIVE E
Magic Brush
HQ-Edit
dd snow to the scene of the image
Make flowers on ground region the image
Figure 8: Qualitative comparison of InstructPix2Pix, MagicBrush, HIVE and HQ-Edit.
Input
InstructPix2Pix
HIVE
MagicBrush
IQQ-Edit
Replace cat with an orange cat
Make it rainy Figure 9: Qualitative comparison of InstructPix2Pix, MagicBrush, HIVE and HQ-Edit on Emu Edit
fine-tuned on InstructPix2Pix, further validating its capability enhance InstructPix2Pix" image editing outcomes beyond their respective datasets. Next in Table 7 we evaluate our model using the most commonly used CLIP dirs CLIP image similarity (CLIP ing DINO image similarity (DINO), SSIM and Reward Score. HQ-Edit also performs the best in all metrics in the HQ-Edit test set. These results demonstrate our comprehensive corpus of high-fidelity images and precise editing instructions establishes a robust framework for more intuitive and efficacious image manipulation protocols.
Evaluation on Emu Edit Test Set We next on the Emu Edit (Sheynin et al. 2023) which consists of real images. As shown in Table g HQ-Edit performs the best in terms of CLIPar. CLIP img- and Reward Score and ranks second-best in terms of DINO and SSIM metries. With these results, we can conclude that the superb quality and large size of HQ-Edit can still enable models to reasonably handle real-life and photorealistic images despite the domain gap.
4.3 QUALITATIVE EVALUATION
As shown in Figure S and Figure comparative analysis of the performance of various models visually presented, with each column dedicated to showcasing the results of a distinct model. In general, we observe that the models trained with HQ-Edit consistently show stronger visual quality. For example, in the second line of Figure & only the model trained with HQ-Edit understands the ground region in the edit instruction and correctly adds the flowers in it required. It can also be

---

Published as a conference paper at ICLR 2025
Input Image
Make the image anime style
Change background of the subject boach
some snow to the the image
Add wings
Figure 10: Qualitative results with the same input image but with different edit instructions. HQ-Edit enhances the editing capabilities of InstructPix2Pix by enabling it to modify the same image of black cat in various ways.
seen in Figure 10 that the model trained with HQ-Edit can carry out various types of edit operations. This observation not only underscores HQ-Edit's s advanced understanding of spatial and contextual directives but also its capability to precisely manipulate image content in accordance with specific editing specifications.
4.4 ABLATION STUDY
We hereby ablate the effectiveness of different post- post-processing strategies, introduced in Sec. 3.3 Specifically, we use "RAW"" to denote the simply decomposed DALL-E 3 images (i.e., image pairs that directly splitted from diptych), and use "Rewrite", "Filter", "Warp". and "Inverse" to mark whether the corresponding operations applied for further processing. For example, applying all these four operations to process these will lead to our HQ-Edit dataset. Table reports the corresponding results.
Interestingly, by comparing the first row and the second row, note that directly fine-tuning the model on the raw DALL-EE 3 images enhances its performance A Alignment but hurts Coherence. This potentially suggests that while the image quality of these DALL-E generated images exceeds that of the InstructPix2Pix dataset, the alignment between the image and edit instruction is less satisfactory. This issue be mitigated with our post-processing techniques. For example, our rewrite method, when compared to the second row's results, delivers improvements of 1.79 in Alignment and 0.94 in Coherence. This boost, primarily enhancing the images' alignment with the edit operation, indicates DALL-E 3's challenges in producing accurate images from dypitch prompts- gap our method effectively bridges. Additionally, employing the inverse technique, which acts as a form of data augmentation, further elevates Aligmment by 5.2 and Coherence by 0.94. The warp technique serves augment both pre- and post-edit image alignment, resulting in notable 5.2 increase in alignment accuracy. Nonetheless, the application of warp may occasionally lead undesirable levels of image distortion. Through the implementation of filtering mechanism targeting such occurrences, we not only achieve further enhancement in image alignment, registering 3.6 increase, but also mitigate the associated data volume. Consequently, this filtering process incurs a marginal reduction in Coherence, specifically by 1.4 points, yet remains superior to other baselines. These results indicate that HQ-Edit holds significant potential to enhance instruction-bart based edit models, especially when combined with effective post-processing.
Table Ablation experiments on Post-processing. RAW
Rewrite
Inverse
Warp
Filter
Alignment 1 Coherence
16.83
85.74
28.
86.68
43,41
87.56
47.01
86.16
CONCLUSION
this study, we present an automatic way to synthesize the image editing dataset scale. Specifically, we leverage two foundation models, GPT-4V and DALL-E 3, to automatically generate, rewrite, and expand a set of seed image editing data with high high-quality. Additionally, we develop two GPT. the alignment of the edited images to the editing instruction, and the coherence of the image content. Our extensive experiments demonstrate that models trained on HQ-Edit set state-of of-t-oofthheart performance in the task of instruction image editing.
4V-based evaluation metrics
10

---

Published as a conference paper at ICLR 2025
Ethical Discussion The integration of foundation models like GPT-4 and DALL-E into automatic dataset creation offers significant benefits by enhancing data quality, diversity, and the overall coverage of editing operations, which in turn supports rapid, large-scale generation of precise image- edit pairs for training advanced image editing models. However, this same automation also introduces ethical challenges, as Al allgenerated content may inadvertently embed biases, stereotypes, other unintended issues due to reduced human oversight and lack of clear data provenance. Additionally, this uncertainty complicates efforts verify authenticity, uphold privacy norms, and prevent the spread of misinformation or the reinforcement of societal biases. To address these challenges, future work should prioritize the development of verifiable, transparent gettaaeenertionn pipelines that integrate systematic bias detection and mitigation strategies. Incorporating human validation where possible can further safeguard against inadvertent harms, enabling regular audits and ensuring that ethical standards are met throughout the data-generation process. We believe strengthening accountability through these measures will enable us to responsibly harness the innovative potential of these foundation models while simultaneously promoting fair, trustworthy, and socially responsible applications of image editing in the real world.
Acknowledgement We thank the AWS Cloud Credit for Research Program, the Microsoft Accelerate Foundation Models Research Program, and the OpenAl Researcher Access Program for supporting our computing needs.
REFERENCES
Abubakar Abid, Ali Abdalla, Ali Abid, Dawood Khan, Abdulrahman Alfozan, and James Zou. Gradio: Hassle-free sharing and testing of ml models in the wild. arXiv preprint (006:00900002500 2019.
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18208-18218 2022.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Tuhin Chakrabarty, Kanishk Singh, Arkadiy Saakyan, and Smaranda Muresan. Learning to follow object-centric image editing instructions faithfully. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion-based semantic image editing with mask guidance. arXiv preprint arXiv:2210.1 11422.0..147,,. 2022.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to- text-to-image synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10696-10706, 2022.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt- to-prompt image editing with cross attention control. arXiv preprint urXiv:2208.01626, 2022.
Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. arXiv preprint arXiv:2311.17111 2023.
Yi Huang, Jiancheng Huang, Yifan Liu, Mingfu Yan, Jiaxi Lv, Jianzhuang Liu, Wei Xiong, He Zhang, Shifeng Chen, and Liangliang Cao. Diffusson model-based image editing: A survey, arXiv preprint MPXi::2402.17525 2024.
Babjat Kawar, Shiran Zada. Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text- based real image editing with diffusion models. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 6007-6017, 2023.
11

---

Published as a conference paper at ICLR 2025
Huan Ling, Karsten Kreis, Daiqing Li, Seung Wook Kim, Antonio Torralba, and Sanja Fidler. Editgan: High-precision semantic image editing. Advances in Neural Information Processing Systems, 34: 16331-16345, 2021.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.097994 2022.
OpenAL. GPT-4v System Card. https///opruu...mmmmmssssaaaaels 2023a.
OpenAl. GPT-4v System Card. https///ppeaa...mmmmsaaraaggg/es
2023b.
Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei. Kosmos-g: Gener- ating images in context with multimodal large language models. arXiv preprint arXiv:2310.02992. 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06115, 1(2):3, 2022.
Dillon Reis, Jordan Kupec, Jacqueline Hong, and Ahmad Daoudi. Real-time flying object detection with yolov8. arXiv preprint wwXiv:2305.09972 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani. Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning toxt-to-mmage diffusion models for subject-driven generation. In Proceed- ings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12500-22510 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494 2022.
Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks. arXiv preprint arXiv:2311. 10089, 2023.
Yichun Shi, Xiao Yang, Yangyue Wan, and Xiaohui Shen. Semanticstylegan Learning compositional generative priors for controllable image synthesis and editing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11224-11222 11244, 2022.
Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. arXiv preprint Ntt.:2306.03881, 2023.
Bram Wallace, Akash Gokul, and Nikhil Naik. Edict: Exact diffusion inversion via coupled transfor- mations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22532-22541, 2023.
Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation. arXiv preprint 23 2023.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannanch Hajishirzi. Sclf-instruct: Aligning language model with self generated instructions, 2022.
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal lim. arXiv preprint 19X.522309.055/9, 2023.
12

---

Published as a conference paper at ICLR 2025
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 11115590--19333 2023.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for texi-to-image diffusion models. 2023.
Kai Zhang. Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guiae guided image editing. Advances Neural Information Processing Systems, 36, 2024.
Shu Zhang, Xinyi Yang. Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, Stefano Ermon, et al. Hive: Harnessing human feedback for instructional visual editing. arXiv preprint wtx.i:2303.091111 2023.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
B PROMPTS
We list all the prompts we used for data collection, including the EXPAND PROMPT used for the Expansion step; DIPTYCH PROMPT and REWRITE PROMPT used for the Generation step; and two metric prompt ALIGNMENT PROMPT and COHERENCE PROMPT for the evaluation.
B. STEP #1: EXPANSION
EXPAND PROMPT (GPT-4)
You are required to generate num examples considering the given examples. The examples should vary widely, including different human characteristics (such race, age, and body type), various animals, insects, furniture, tools, or any object types, etc., and diverse backgrounds (like different countries, natural environments, landscapes, skies). The editing attributes should also be diverse. Make sure the examples are clear, concise, comprehensive, and easier for DALL-E 3 to generate this diptych image following the prompt. Describe the first image in "INPUT DESCRIPTION" like "input", the second image in "OUTPUT DESCRIPTION "INPUT_DESCRIIIS DESCRIPTION" and "OUTPUT DESCRIPTION" should be like "output", both "INPUT
independent complete sentences, and the operation that edits the first image to the second image and the operation that edits the second image the first image "INVERSE EDIT OPERATION" the output should be list of JSON format as such: ("input": "INPUT DESCRIPTION", "edit": "EDIT OPERATION", "edit_inv": "INVERSE EDIT OPERATION". "output": "OUTPUT DESCRIPTION". ].
"EDIT "EDIT_OPERATIIOO
Do not output anything else, all examples should have complete keys "input", "edit", "edit_inv", and "output".
B.2 STEP #2: GENERATION
REWRITE PROMPT (GPT-4)
Please rewrite the following prompt to make more clear and concise, and easier for DALL-E 3 to generate this diptych image follow the prompt. The original prompt is: {prompt). The output prompt should start with "REVISED":
DIPTYCH PROMPT (DALL-E
Create a diptych image that consists two images. The left image is (prompt); The right image keep everything the same but [edit_action];
B.3 EVALUATION METRIC
ALIGNMENT PROMPT (GPT-4V)
From to 100, how much do you for EDIT TEXT in terms of the correct and comprehensive description of the change from the first given image to the second given image? Correctness refers whether the mentions any change that are not made between two images. Compre- hensiveness refers to whether the text misses any change that are made between two images.
14

---

Published as a conference paper at ICLR 2025
The second image should have minimum change to reflect the changes made with EDIT TEXT. Be strict about the changes made between two images:
1. If the EDIT TEXT is about stylization or lighting change, then no content should be changed and all the details should preserved.
2. If the EDIT TEXT is about local change, then no irrelevant area nor image style should be changed.
3. The first image should not have the attribute described inside the EDIT TEXT, rate low, (<80) if this happens.
4. Be aware to check whether the second image does maintain the important attribute in the left image that not reflected the EDIT TEXT. Rate low (<50) if two images are not related. Provide few lines for explanation and give the final response in a json format such: "Exppanatioo".: "Score": "",
COHERENCE PROMPT (GPT-VV)
Rate the Coherence of the provided image on a scale from 0 to 100, with O indicating extreme disharmony characterized by numerous conflicting or clashing elements, and 100 indicating perfect harmony with all components blending effortlessly. Your evaluation should rigorously consider the following criteria:
1. Consistency in lighting and shadows: Confirm that the light source and corresponding shadows are coherent across various elements, with no discrepancies direction or intensity.
2. Element cohesion: Every item in the image should logically fit within the scene's 's context, without any appearing misplaced or extraneous.
3. Integration and edge smoothness: Objects subjects should integrate seamlessly into their surroundings, with edges that do not appear artificially inserted or poorly blended.
4. Aesthetic uniformity and visual flow: The image should only be aesthetically pleasing but also facilitate a natural visual journey, without abrupt interruptions caused by disharmonious elements.
Implement stringent scoring guideline:
Award a high score (90-100) solely if the image could pass as flawlessly captured scene, devoid of any discernible disharmony.
Assign moderate to high score (70-89) if minor elements of disharmony are present but they do not significantly detract from the overall harmony.
Give moderate score (50-69) if noticeable disharmonious elements are evident, affecting the image's 's harmony to a moderate degree.
Allocate low score (30-49) for images where disharmonious elements are prominent, greatly disturbing the visual harmony.
Reserve the lowest scores (0-29) for images with severe disharmony, where the elements are so discordant that it disrupts the intended aesthetic.
Your assessment must be detailed, highlighting the specific reasons for the assigned score based on the above criteria. Conclude with response formatted in JSON as shown below: "Explanation": "<Insert detailed explanation here>". "Score": <Insert precise score here>
C MORE VISUALIZATION RESULTS
C.I DATA POINTS
Here, we provide randomly sampled data points from HQ-Edit in Figure 11 for for visual assessment
15

---

Published as a conference paper at ICLR 2025
"imput": German Shopherd with black coot, pointed ears, and dog tag sitting grassy lawn with trees and sunlight in the barkground.
"Input": "portrait of majestic golden eagle flight against warm sunset sky."
"edit"": "Replace the G erman Shepherd with a iluffy white cat with blue eyes and a long tail, keeping the same background.
"edit": "adjust the overall image to nighttime setting with darker sky and cooler tones."
"inverse-ediit": "Replace the fluffy white with the original German Shepherd with a black and tän coat, pointed ears, and dog tag."
"Inverse--diit" "adjust the overall image to daytime setting with brighter sky and warmer tones."
"output"" "A flfffy white cat with blue and long tail sitting on the same grassy lawn with trees and sunlight in the background.
"output": "portrait of majestic golden eagle in flight against cooler night: time sky with visible clouds and darker ambiance..
"Alignment": 100 "Cohetence:1100
"A Alignment": "Coharence":
Figure 11: Example data sampled from HQ-Fdit. Our data contains two main parts, Instruction (input, edit, inverse-edit, output) and Image (input image, output image). The two samples highlight that, I) the image is densely packed with details, 2) the input and ouput offers comprehensive description of the input and output image, and 3) the edit and inverse-edit instructions precisely delineate the transformations occurring between the two images.
C.2 DATA POINTS COMPARISON
We visualize the data InstructPix2Pix in Fig. 13 MagicBrush in Fig. 14 HIVE Fig. T5] and HQ-Edit in Fig. 12 with the Edit instruction, Aligment and Coherence. This shows that HQ-Edit possesses higher image quality and better image-text alignment. and more data with Alignment and Coherence score from HQ-Edit in Figure 12
16

---

Published as a conference paper at ICLR 2025
Edit change lee hair color blonde and add waves
Edit: Regkace the heavy-duty power drill with high-tech precision
Aligmmen:: 100
power tool.
Coherence:
Aligament:
Coherance: 96
Edit: Change the weather to rainy.
Edit: T Transfomm the elderly woman into young woman, change traditional dress sandals with white sceabers, and add black bandbag bestde her the banch.
Aligument:
leather jankel, replane
Coherence: 95
Aligument: 100
Coherenze:: 90
plastic toy
Edit: Change chameloon's
bright orange and blue bandle.
keeping the green color on its bead crest and tail. Alignneet: 100
Alignment 80
Coherenc:
Coherence:
Edit: Replace the Japanese tea with Victorian including porcelala teapots cups with tloral designs, tabloclott, silver cutlery, and decorntive golden tea strainer Change the backdrop to inctude framed Lloral lapestty.
Edit- A ller the bint's color vibcant blue. Change the backdrmp framed floma topestry.
A liggnment: 100
Coherance:
Alignment: 100
Coherence: 8B
Figure 12: Data of HQ-Edit, the left side is the input image and the right side is the output image.
17

---

Published as a conference paper at ICLR 2025
Edit: Make fammer
Edit swap the cyclist biker Aligument: Coherence: 90
Aiignment:
Coherence:
65
Figure 13: Data of InstructPix2Pix. the left side is the input image and the right side is the output image.
95
Edit: Add julphinjumping out of the water A Alignment: 100
Edit: Turn the faucet A Alignment: 0
Coherence: 75
Coherence: 95
Figure 14: Data MagicBrush, the left side the input image and the right side the output image.
pios powe
apins apes Appor
Epes pges
Edit: Change retro to futuristic Aligmment:
Edit: make man woman
A Aligument:
Coherence: 95
Coherence:
30
Figure 15: Data of HIVE, the left side is the input image and the right side is the output image.
18


Reviews:
[{"review_id": "2uofAtVmvM", "reviewer": "Reviewer_ptsc", "summary": "This paper introduces a large-scale and high quality instruction-based image editing dataset, HQ-Edit, with 200K edits. This dataset contributes to address the challenges of low instruction alignment and quality in image editing by employing GPT-4, DALL-E3 and GPT-4V. This work presents a new and scalable data curation process involving with crafted prompt engineering and diptych generation. The authors also present baselines finetuned on HQ-Edit which shows state-of-the-art performance in instruction-based single image editing task.", "questions": "Referring to weakness 1, can you provide an analysis or design an experiment to show the potential generalization ability of the finetuned baseline model on photo-realistic data?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Compared with previous work, the proposed dataset is high quality regarding to image resolution, content/edit-type diversity and prompt-image alignment.\n2. Baseline method InstructPix2Pix finetuned on the proposed HQ-Edit achieves state-of-the-art performance.\n3. The proposed data curation pipeline is scalable by leveraging pretrained generative model (e.g. DALL·E3) and visual-language model (e.g. GPT-4 / GPT-4V).\n4. The paper is well-organized and easy to follow.", "weaknesses": "1. Compared with previous work, e.g. MagicBrush, the source images from HQ-Edit are generated by DALLE-3, which may introduce distribution bias between AIGC and photo realistic contents.\n2. The necessity/importance analysis of using diptych generation is missing.\n3. The proposed two metrics Alignment and Coherence are mainly used in the main evaluation. Given the validated limitation of CLIP directional similarity, other commonly used metrics are missing for quantitative evaluation, which may lead to unfair comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J6pWn2QF9C", "reviewer": "Reviewer_PppU", "summary": "This paper introduces HQ-Edit for instruction-based image editing task, which contains around 200,000 edits. By using GPT-4V and DALL-E 3, the author develops a scalable data collection pipeline, which consists of three steps (Expansion, Generation and Post-Processing). Besides, the author also designs two metrics by using GPT-4V, which own a higher correlation to human preference compared to the CLIP score. Based on HQ-Edit, the finetuned InstructPix2Pix can attain state-of-the-art image editing performance.", "questions": "Please refer to the weaknesses part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed HQ-Edit can be a training data for instruction-based image editing task, which can promote the development of this area. \n2. The performance of finetuned InstructPix2Pix has proven the effectiveness of HQ-Edit.\n3. The proposed evaluation metrics are superior to the CLIP score.", "weaknesses": "1. It seems that HQ-Edit only contains non-rigid pair data.\n2. There are many types of operations for instruction-based image editing task (e.g., object addition, object removal, non-rigid operation, local transformation, global transformation). The figure 8 and figure 9 only show the transformation part. The author should show the results of all these operations to make a comprehensive comparison.\n3. Regarding metrics, since there is already a large amount of Human Evaluation Scores, why not use a model to learn a reward model for scoring? Alternatively, we could fine-tune an MLLM for scoring. Simply using prompt engineering with GPT-4V may not be the best option.\n4. There is some typo in line 512 – 516. It should be “enhances its performance on Coherence but hurts alignment”.\n5. What is the architecture of InstructPix2Pix trained by the author? (Base or XL). If the XL version is finetuned with HQ-Edit, can the performance be further improved than Base version?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Io89nPrCST", "reviewer": "Reviewer_Drub", "summary": "This paper presents HQ-Edit, a dataset designed for instruction-based image editing, which contains around 200,000 high-quality image edits. The authors have developed a scalable data collection pipeline that utilizes advanced foundation models, GPT-4V and DALL-E 3, to generate detailed text prompts paired with input and output images. This approach overcomes the limitations of previous methods that relied on attribute guidance or human feedback, which were constrained by scalability and diversity.\n\nTo quantitatively assess the quality of image-edit pairs, the authors introduce two novel metrics: Alignment and Coherence. Alignment evaluates the semantic consistency with edit prompts, while Coherence assesses the overall aesthetic quality of the edited image.\n\nThe author fine-tuned the InstructPix2Pix model using the HQ-EDIT dataset.Compared to existing text-based image editing models,their model performs best in Alignment and Coherence Score.", "questions": "1. In the appendix, the provided image scores suggest that GPT-4V seems to give higher coherence scores to synthetic images. The paper does not substantiate that higher coherence scores from GPT-4V are indicative of higher human ratings.This raises the question of whether GPT-4V is 、capable of assessing image quality accurately.\n2. Both evaluation metrics have a scale ranging from 0 to 100. Even though there are some segmented scoring prompts, can GPT-4V truly differentiate between scores that are a few points apart?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. High-Quality Dataset: The paper introduces HQ-Edit, a dataset with approximately 200,000 high-quality image edits, which is a significant contribution to the field of instruction-based image editing.\n2. Advanced Foundation Models: Leveraging state-of-the-art models like GPT-4V and DALL-E 3 ensures that the dataset benefits from the latest advancements in AI, leading to high-resolution and detailed images.\n3. Broad Coverage of Editing Operations: HQ-Edit covers a wide range of editing tasks, from global operations like weather changes to local object edits, demonstrating its versatility.", "weaknesses": "1. Synthetic Data Limitations: Although the synthetic images are useful for training, the trained model may not perform well on real images.\n2. Constrained Persuasiveness of Evaluation Metrics:The paper only conducted comparisons on the two evaluation metrics it proposed, Alignment and Coherence, without making comparisons on more widely used and popular metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yPftDNkv0D", "reviewer": "Reviewer_x6Ut", "summary": "This paper presents a new high resolution dataset on image editing named HQ-Edit. Compared to existing works, the proposed dataset contains high resolution and leveraged the most advanced vision-language model for creating human like instructions. Since all the instruction and image generated are using synthetic data, as a result, the proposed dataset is in a reasonable large scale for public usage. Simply leverage the dataset, the existing InstructPix2Pix can generate better results.", "questions": "Since the proposed dataset contains multiple type of editing, it is important to show the results of different editing cases, especially on real images.   Moreover, it is also better to compare with some specialist models such as inpainting and object insertion model. \n\nIn addition, I also notice that the edited results still can't keep the un-edited region unchanged. Many cases showing in the paper indicate that changing unrelated region will cause lots of identity shift on other regions. \n\nIt also worths to show more global editing on more real images such as stylization (painting, sketch, water color, cartoon style) and image2image translation (day to night, winter to summer). EMU edit benchmark should be a good dataset to be evaluated. \n\nReference \n[1]https://huggingface.co/datasets/AILab-CVC/SEED-Data-Edit\n[2]https://huggingface.co/datasets/BleachNick/UltraEdit", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The key contribution in this paper is the proposed HQ-Edit dataset. Compared to existing widely used InstructPix2Pix dataset, HQ-Edit dataset is in a high resolution and better editing due to verification  process of a powerful vision-language models such as GPTV.  For example, after generating the dataset by Dalle 3.  GP4V will be used to rewrite the instructions to improve the instruction and editing alignment. Such process will greatly improve the dataset.  Besides, human evaluation has shown that the proposed dataset is better on editing image quality.", "weaknesses": "The key weakness of the proposed method lies on the limited quality improvement of dataset. \n\nFirst, InstructPix2Pix firstly proposed a pipeline to leverage prompt2prompt and gpt to generate the triplets. While the proposed \"high quality\" editing dataset is simply replaced the prompt2prompt method to Dalle3 and leveraged a more powerful gpt version to filter out the low quality instructions. Such improvement is incremental while not significantly improve the datasets.  The main reason is, similar to prompt2prompt, the HQ Edit dataset is still all synthetic pairs.  The proposed editing types are still similar to InstructPix2Pix.  Most importantly, when most of existing editing dataset such as Magicbrush, SeedX and UltraEdit have adopt real image as input,  the usage of synthetic image pairs has became less important. \n\nSecond, the proposed dataset have shown multiple editing type, while leveraging Dalle3 is obviously not the optimal solutions.  SeedX has been demonstrated the remove and add case should leverage more accurate model such as image inpainting. Meanwhile, the quality of Dalle3 images have become less convincing to be recognize as \"high resolution and high quality\" since Flux and other generative super resolution can achieve much better higher resolution image editing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)