HQ-Edit: A High-Quality Dataset for Instruction-based Image Editing

Authors: Mude Hui, Siwei Yang, Bingchen Zhao, Yichun Shi, Heng Wang, Peng Wang, Cihang Xie, Yuyin Zhou
Primary area: datasets and benchmarks
Keywords: Image-to-image translation, Image Editing

Abstract:
This study introduces HQ-Edit, a high-quality instruction-based image editing dataset with around 200,000 edits. Unlike prior approaches relying on attribute guidance or human feedback on building datasets, we devise a scalable data collection pipeline leveraging advanced foundation models, namely GPT-4V and DALL-E 3. To ensure its high quality, diverse examples are first collected online, expanded, and then used to create high-quality diptychs featuring input and output images with detailed text prompts, followed by precise alignment ensured through post-processing. In addition, we propose two evaluation metrics, Alignment and Coherence, to quantitatively assess the quality of image edit pairs using GPT-4V. HQ-Edits high-resolution images, rich in detail and accompanied by comprehensive editing prompts, substantially enhance the capabilities of existing image editing models. For example, an HQ-Edit finetuned InstructPix2Pix can attain state-of-the-art image editing performance, even surpassing those models fine-tuned with human-annotated data.

Decision: accept (poster)
Reviewer scores: 8, 6, 5, 3

Reviews and discussion:
<thread>
creation date:
1730539613553

thread_type:
official_review

<thread_object>
creator: reviewer_x6ut

<content>
summary:
This paper presents a new high resolution dataset on image editing named HQ-Edit. Compared to existing works, the proposed dataset contains high resolution and leveraged the most advanced vision-language model for creating human like instructions. Since all the instruction and image generated are using synthetic data, as a result, the proposed dataset is in a reasonable large scale for public usage. Simply leverage the dataset, the existing InstructPix2Pix can generate better results.

strengths:
The key contribution in this paper is the proposed HQ-Edit dataset. Compared to existing widely used InstructPix2Pix dataset, HQ-Edit dataset is in a high resolution and better editing due to verification  process of a powerful vision-language models such as GPTV.  For example, after generating the dataset by Dalle 3.  GP4V will be used to rewrite the instructions to improve the instruction and editing alignment. Such process will greatly improve the dataset.  Besides, human evaluation has shown that the proposed dataset is better on editing image quality.

weaknesses:
The key weakness of the proposed method lies on the limited quality improvement of dataset. 

First, InstructPix2Pix firstly proposed a pipeline to leverage prompt2prompt and gpt to generate the triplets. While the proposed "high quality" editing dataset is simply replaced the prompt2prompt method to Dalle3 and leveraged a more powerful gpt version to filter out the low quality instructions. Such improvement is incremental while not significantly improve the datasets.  The main reason is, similar to prompt2prompt, the HQ Edit dataset is still all synthetic pairs.  The proposed editing types are still similar to InstructPix2Pix.  Most importantly, when most of existing editing dataset such as Magicbrush, SeedX and UltraEdit have adopt real image as input,  the usage of synthetic image pairs has became less important. 

Second, the proposed dataset have shown multiple editing type, while leveraging Dalle3 is obviously not the optimal solutions.  SeedX has been demonstrated the remove and add case should leverage more accurate model such as image inpainting. Meanwhile, the quality of Dalle3 images have become less convincing to be recognize as "high resolution and high quality" since Flux and other generative super resolution can achieve much better higher resolution image editing.

questions:
Since the proposed dataset contains multiple type of editing, it is important to show the results of different editing cases, especially on real images.   Moreover, it is also better to compare with some specialist models such as inpainting and object insertion model. 

In addition, I also notice that the edited results still can't keep the un-edited region unchanged. Many cases showing in the paper indicate that changing unrelated region will cause lots of identity shift on other regions. 

It also worths to show more global editing on more real images such as stylization (painting, sketch, water color, cartoon style) and image2image translation (day to night, winter to summer). EMU edit benchmark should be a good dataset to be evaluated. 

Reference 
[1]https://huggingface.co/datasets/AILab-CVC/SEED-Data-Edit
[2]https://huggingface.co/datasets/BleachNick/UltraEdit
</content>
</thread_object>

</thread>
<thread>
creation date:
1730650596579

thread_type:
official_review

<thread_object>
creator: reviewer_drub

<content>
summary:
This paper presents HQ-Edit, a dataset designed for instruction-based image editing, which contains around 200,000 high-quality image edits. The authors have developed a scalable data collection pipeline that utilizes advanced foundation models, GPT-4V and DALL-E 3, to generate detailed text prompts paired with input and output images. This approach overcomes the limitations of previous methods that relied on attribute guidance or human feedback, which were constrained by scalability and diversity.

To quantitatively assess the quality of image-edit pairs, the authors introduce two novel metrics: Alignment and Coherence. Alignment evaluates the semantic consistency with edit prompts, while Coherence assesses the overall aesthetic quality of the edited image.

The author fine-tuned the InstructPix2Pix model using the HQ-EDIT dataset.Compared to existing text-based image editing models,their model performs best in Alignment and Coherence Score.

strengths:
1. High-Quality Dataset: The paper introduces HQ-Edit, a dataset with approximately 200,000 high-quality image edits, which is a significant contribution to the field of instruction-based image editing.
2. Advanced Foundation Models: Leveraging state-of-the-art models like GPT-4V and DALL-E 3 ensures that the dataset benefits from the latest advancements in AI, leading to high-resolution and detailed images.
3. Broad Coverage of Editing Operations: HQ-Edit covers a wide range of editing tasks, from global operations like weather changes to local object edits, demonstrating its versatility.

weaknesses:
1. Synthetic Data Limitations: Although the synthetic images are useful for training, the trained model may not perform well on real images.
2. Constrained Persuasiveness of Evaluation Metrics:The paper only conducted comparisons on the two evaluation metrics it proposed, Alignment and Coherence, without making comparisons on more widely used and popular metrics.

questions:
1. In the appendix, the provided image scores suggest that GPT-4V seems to give higher coherence scores to synthetic images. The paper does not substantiate that higher coherence scores from GPT-4V are indicative of higher human ratings.This raises the question of whether GPT-4V is 、capable of assessing image quality accurately.
2. Both evaluation metrics have a scale ranging from 0 to 100. Even though there are some segmented scoring prompts, can GPT-4V truly differentiate between scores that are a few points apart?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695548706

thread_type:
official_review

<thread_object>
creator: reviewer_pppu

<content>
summary:
This paper introduces HQ-Edit for instruction-based image editing task, which contains around 200,000 edits. By using GPT-4V and DALL-E 3, the author develops a scalable data collection pipeline, which consists of three steps (Expansion, Generation and Post-Processing). Besides, the author also designs two metrics by using GPT-4V, which own a higher correlation to human preference compared to the CLIP score. Based on HQ-Edit, the finetuned InstructPix2Pix can attain state-of-the-art image editing performance.

strengths:
1. The proposed HQ-Edit can be a training data for instruction-based image editing task, which can promote the development of this area. 
2. The performance of finetuned InstructPix2Pix has proven the effectiveness of HQ-Edit.
3. The proposed evaluation metrics are superior to the CLIP score.

weaknesses:
1. It seems that HQ-Edit only contains non-rigid pair data.
2. There are many types of operations for instruction-based image editing task (e.g., object addition, object removal, non-rigid operation, local transformation, global transformation). The figure 8 and figure 9 only show the transformation part. The author should show the results of all these operations to make a comprehensive comparison.
3. Regarding metrics, since there is already a large amount of Human Evaluation Scores, why not use a model to learn a reward model for scoring? Alternatively, we could fine-tune an MLLM for scoring. Simply using prompt engineering with GPT-4V may not be the best option.
4. There is some typo in line 512 – 516. It should be “enhances its performance on Coherence but hurts alignment”.
5. What is the architecture of InstructPix2Pix trained by the author? (Base or XL). If the XL version is finetuned with HQ-Edit, can the performance be further improved than Base version?

questions:
Please refer to the weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730724981887

thread_type:
official_review

<thread_object>
creator: reviewer_ptsc

<content>
summary:
This paper introduces a large-scale and high quality instruction-based image editing dataset, HQ-Edit, with 200K edits. This dataset contributes to address the challenges of low instruction alignment and quality in image editing by employing GPT-4, DALL-E3 and GPT-4V. This work presents a new and scalable data curation process involving with crafted prompt engineering and diptych generation. The authors also present baselines finetuned on HQ-Edit which shows state-of-the-art performance in instruction-based single image editing task.

strengths:
1. Compared with previous work, the proposed dataset is high quality regarding to image resolution, content/edit-type diversity and prompt-image alignment.
2. Baseline method InstructPix2Pix finetuned on the proposed HQ-Edit achieves state-of-the-art performance.
3. The proposed data curation pipeline is scalable by leveraging pretrained generative model (e.g. DALL·E3) and visual-language model (e.g. GPT-4 / GPT-4V).
4. The paper is well-organized and easy to follow.

weaknesses:
1. Compared with previous work, e.g. MagicBrush, the source images from HQ-Edit are generated by DALLE-3, which may introduce distribution bias between AIGC and photo realistic contents.
2. The necessity/importance analysis of using diptych generation is missing.
3. The proposed two metrics Alignment and Coherence are mainly used in the main evaluation. Given the validated limitation of CLIP directional similarity, other commonly used metrics are missing for quantitative evaluation, which may lead to unfair comparison.

questions:
Referring to weakness 1, can you provide an analysis or design an experiment to show the potential generalization ability of the finetuned baseline model on photo-realistic data?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732518990371

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Concerns

comment:
We thank all reviewers for their thoughtful feedback, which will help us improve the quality of this paper. We are delighted to see all the reviewers agree that the model finetuned with our HQ-Edit achieved strong performance, indicating the quality and utility of HQ-Edit and, hence, our data pipeline's effectiveness. Additionally, reviewers acknowledge that the data collection pipeline is innovative, scalable, and well-defined (k3ov, U5Yr), and the metrics we proposed align well with human evaluation and can be used for quantitatively assess the editing quality (cVQk, U5Yr), and contains a variety of diverse editing operations, showing its versatility (Drub)

We first address some common concerns as follows:

## Necessity of diptych generation
We would like to clarify that this design choice is mainly driven by the fact that DALL-E 3 doesn't have a built-in editing API --- i.e., it **cannot generate an image given an input image**. This also marks **one of the key differences** between our pipeline and others, such as InstructPix2Pix. Given this restriction, we alternatively utilize diptych generation as a workaround so that it is possible to leverage DALL-E 3's powerful text-to-image generation ability.

Furthermore, to accurately and effectively extract info from the generated diptych, we proposed a series of post-processing methods, including:
1. Image-wise
   We further improved image alignment using **Warping & Filtering**. As validated empirically in Sec. 4.4 and visually demonstrated in Fig. 3, these designs can improve image quality and, consequently, the model's performance, enabling our proposed dataset to further enhance the image editing models' capabilities, such as InstructPix2Pix.
2. Text-wise
   Additionally, we leveraged **Instruction Refinement**, using GPT-4V to refine the original editing instructions further, ensuring all still-existing misalignment after image post-processing are encompassed in editing instructions. Arguably, this makes all the image pairs achieve almost precise alignment as they correspond to the rewritten instructions in detail.

## Other evaluation metrics and human evaluation
Some reviewers are concerned with the two GPT-based metrics we proposed and find results lack classic metrics.

To address this concern, we first present evaluation results with more metrics, including the most commonly used CLIP directional similarity (CLIP_dir), CLIP image similarity (CLIP_img), DINO image similarity (DINO), and SSIM. Also, we employ ImageReward (Jiazheng et al. [1]) as the reward model for measuring the quality of output images, according to the suggestion from Reviewer PppU. Since scores from ImageReward are unbounded, we normalize reward scores to $[0, 1]$ with Sigmoid. The results are as follows:

| Model       | $\text{CLIP}_\text{dir}$       | $\text{CLIP}_\text{img}$       | DINO       | SSIM           | Reward Score | Reward Score (after Sigmoid)
|--------------|-----------------|------------------|----------------|----------------|----------------|----------------|
| InstructPix2Pix         | 0.0444 | 0.7353 | 0.7252 | 0.1673 | -0.54|0.40|
 |HIVE   | 0.0970 | 0.8633 | 0.8851 |  0.4646 | -0.09|0.48|
| MagicBrush   | 0.1109 | 0.8173 | 0.8246 | 0.2735 | -0.05|0.49|
| HQ-Edit (Ours)         | **0.1351** | **0.9246** | **0.9692** | **0.6561** | **-0.03**| **0.50** |

We can observe that the model trained with HQ-Edit consistently yields the highest performance across all metrics, further confirming its effectiveness.

Additionally, we present the human evaluation results with Coherence and ImageReward scores. Specifically, we conducted a human evaluation with 160 image pairs formed with output images from HQ-Edit. Human evaluators are tasked with determining which image in each pair is better. "Image 1 is better", "Tie" and "Image 2 is better" are encoded as -1, 0 and 1. We then compute the Pearson Correlation between human evaluation scores and deltas between the GPT-evaluated scores of the first and second images. The correlation between scores from ImageReward and human evaluation is computed in the same way.

| Coherence | Reward Model | Reward Model (after Sigmoid) |
| - | - | - |
| 0.27 | 0.12 | 0.10 |

The results above suggest that the Coherence score we proposed is significantly more aligned with human preference than the newly proposed reward model. As the development of reward models for image generation and image editing is still at a preliminary stage, we find alternatively, using a mature and well-tested general-purpose MLLM that is GPT-4V is valid and reasonable at this stage.

We will add these additional results and corresponding discussions in the next version.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733088463283

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A General Response Regarding Reviewer x6Ut's Concerns

comment:
Dear All Reviewers,

We appreciate the thoughtful feedback provided by all reviewers and would like to address the concerns raised by Reviewer x6Ut, which may be shared by others. Our response comprises two parts: (1) presenting a practical and successful example of utilizing image generation models for image editing, and (2) reiterating the value of our HQEdit submission.


1.  **A Successful Example of Utilizing Image Generation for Image Editing** 
We are pleased to share a direct commercial follow-up of HQEdit that incorporates significant scaling and further enhancements. We have updated the results of this commercial model in [real_image_results.png](https://anonymous.4open.science/r/hqedit_re-8D01/real_image_results.png) and [rigid_operation_results.png](https://anonymous.4open.science/r/hqedit_re-8D01/rigid_operation_results.png). These results demonstrate that the previously concerning errors—such as background alteration or unintended transformations—can be substantially mitigated. 
Regarding technique enhancement, this commercial model makes three main changes: 1) it generates a large-scale pairwise dataset with more randomness to ensure diversity, and then we apply filters to choose good examples for model training; 2) in the model training, a causal self-attention structure is introduced for enhancing image conditioning; 3) based on the current state of the editing model, it prepares a new set of data following a similar generation pipeline; and the results are labeled, filtered, and used to fine-tune the editing model. This process is repeated multiple times until convergence. We hope this strong result offered by our commercialized model can alleviate reviewers' concerns about the practical viability of using image generation models to produce training data for successful image editing models in real-world applications.

2) In this second part, we would like to invite all reviewers to **re-assess the valve of this ICLR submission**. Specifically, As one of the pioneering works in this field, we believe our HQEdit alone has already provided substantial evidence that utilizing image generation models for image editing is a promising direction. Our paper and this rebuttal offer comprehensive evaluations showing that **HQEdit achieves superior performance compared to other baselines**, measured by different classic metrics (like CLIP scores), our proposed GPT scores (alignment and coherence), and human-valve-based reward model scores.  Furthermore, our pipeline is highly scalable for generating vast amounts of synthetic images for training purposes. We believe this scalability is crucial for advancing future research in image editing. As history has shown in deep learning, large quantities of somewhat noisy data can significantly boost model performance compared to small datasets with extremely high accuracy. We believe this principle is equally applicable to image editing, thereby potentially underscoring the importance and research value of our HQEdit approach.

We are grateful for your consideration and are happy to address any further questions or concerns you may have.

Thanks
Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents an approach for generating a large-scale synthetic (generated) dataset for training models to perform instruction-based image editing (similar to InstructPix2Pix). Unlike InstructPix2Pix, which uses an open-source text-to-image model to generate pairs of corresponding images via attention-hacking, this approach instead uses a closed-source image generator as a black box (Dall-E3), followed by a handful of filtering operations driven by vision-language models. The resulting dataset can be used to fine-tune editing models, resulting in SoTA performance.

Reviewers acknowledge the value of the dataset, and that evaluations show improved performance over the state-of-the-art. Some reviewers question the utility of a synthetic dataset for training models to operate on real data, as well as the particular methodology of the way in which the closed-source model was used (i.e., generating image pairs jointly as a diptych). 

Ultimately, the review remain split, with a spread across all different scores. Although the proposed method may not be the ideal solution for training the top-quality commercial models, the approach used to filter the data may have interesting insights for future work, and the current evaluations are sufficiently convincing to validate that it provides an improvement over prior work. I advocate for the paper's acceptance.